Never Release to Test: Why we had to shift CI and AI agent governance left

During the second week of August 2026, our GitLab CI infrastructure was running hot: 16,300 runner minutes in seven days. Of those, 2,809 minutes (18.9%) were burned in pipelines that never delivered any value, failing on avoidable errors. For a single web application, we recorded 22 release attempts across six days just to get a working build into production.

The reflexive diagnosis in moments like these is almost always: “Our pipeline is flaky.”

Our forensic audit across every pipeline run that week revealed the exact opposite: Only 10.3% of failures were true flakes (network timeouts, runner race conditions). A staggering 78.4% were genuine errors — syntax blunders, missing imports, misplaced configs, and broken builds — that engineers and autonomous AI agents only discovered after a 17-minute remote CI roundtrip.

This article is Part 1 of a two-part retrospective. It covers how three architectural levers — enforcing AI agent governance before Turn 0, eliminating validation blind spots in merge requests, and establishing an evidence-based preflight gate before tagging — cut 1,150 minutes of weekly CI waste.

The Anatomy of Failure: What the Data Showed

A detailed breakdown of all 46 failed pipelines between August 8 and 15 identified three structural epicentres:

The most damaging dynamic was the compounding friction between autonomous AI coding agents and incomplete merge request pipelines.

The “Silent Absence” Problem in AI Coding Agents

In our monorepo, engineers work side-by-side with autonomous LLM agents (Claude Code, OpenCode, Codex) operating in isolated Git worktrees. All company engineering standards — from commit message conventions to strict TDD mandates (“Never call test runners directly; always use mise run <task>”) — live in a shared submodule under .agents/.

When auditing active worktrees, we uncovered a critical breakdown: in 19 out of 19 active agent worktrees, the .agents/ submodule had never been initialised.

/worktrees/feat-auth/
├── .agents/          <-- EMPTY (submodule uninitialised)
├── AGENTS.md -> .agents/rules/AGENTS.md  <-- Dead symlink!

Because git worktree add does not check out submodules recursively by default, the symlinks referenced by agent runtimes pointed into a void. Without an error or prompt, models entered Turn 0 with generic default prompts, violated internal conventions, generated non-compliant commit headers, and committed unverified code directly into the repository.

The Fix: A Root-Owned, Fail-Closed Git Shim (Layer −1)

Relying on agents or developers to remember git submodule update --init after creating a worktree fails reliably. We moved the enforcement down to the operating system level.

Using Ansible, we placed a service-scoped wrapper ahead of the real git binary to intercept every checkout:

#!/usr/bin/env bash
set -euo pipefail

REAL_GIT="/usr/bin/git"

# Execute the intended Git command
"$REAL_GIT" "$@"
EXIT_CODE=$?

# After every checkout or worktree command: enforce .agents synchronisation
if [[ "$EXIT_CODE" -eq 0 && ("$*" =~ "checkout" || "$*" =~ "worktree add") ]]; then
  if [[ -f ".gitmodules" ]] && grep -q "\.agents" .gitmodules; then
    "$REAL_GIT" submodule update --init --recursive .agents >/dev/null 2>&1 || {
      echo "[FATAL] .agents submodule failed to initialise. Failing closed." >&2
      exit 1
    }
  fi
fi

exit $EXIT_CODE

This shim is fail-closed: if the governing rules cannot be synchronously and completely loaded, the worktree command aborts immediately. An agent cannot write a single character of code without the monorepo guardrails present in its context.

The Anti-Pattern: “Release to Test”

The second major lever addressed our release mechanics.

Historically, expensive CI stages — building Docker images, starting dependencies with health checks, and running delivery probes — were restricted to Git release tags (v*.*.*). The original rationale was resource conservation: avoid building heavy container images on every feature branch.

The result was disastrous. Developers and agents pushed feature branches, saw green merge request pipelines (which only checked unit tests), merged into main, and immediately tagged a release. Only on the tag did the Docker build fail — often due to an outdated COPY path in a Dockerfile following a directory restructure.

Because a failed release tag cannot simply be rewritten, teams were trapped in a loop: commit a fix, merge, tag v1.2.1, wait 17 minutes, and hope.

That is how 22 release tags were burned in six days.

Layer 0: Exhaustive Reachability in Merge Requests

To break this cycle, we restructured our CI trigger matrix:

1. Downstream Bridges in MRs: All 10 client frontends and services trigger their full validation pipelines inside merge requests whenever relevant paths change.
2. Immutable Candidate Images: Docker builds run inside merge requests and are validated against real registry health checks. To avoid polluting shared caches, MR pipelines publish immutable candidate tags (mr-<iid>-<sha>), reserving the mutable :latest tag strictly for successful main builds.
3. Read-Only Smoke Tests: On main, smoke tests run strictly read-only against staging environments to prevent cross-pipeline locking.

This guarantees a core invariant: Nothing that fails in an MR can ever reach main.

The Final Barrier: The release/preflight Gate (Layer 1)

Even with comprehensive MR pipelines, residual uncertainty remained: in a monorepo containing 17 distinct deployables (frontends, APIs, parsers, background workers), subtle merge order interleavings on main can still introduce integration drift.

Previously, this was discovered only after pushing a tag. Today, a zero-side-effect preflight gate on main prevents premature releases before tagging can begin.

# Retrieve all 17 deployable monorepo projects
DEPLOYABLES=$(nx show projects --with-target=build)

for app in $DEPLOYABLES; do
  echo "Checking CI evidence for $app..."
  
  # Verify that the latest changes to this deployable 
  # have an unbroken green pipeline record on main
  if ! verify_deployable_evidence "$app"; then
    echo "[BLOCK] $app has unverified changes or failing checks!" >&2
    exit 1
  fi
done

echo "[READY_TO_TAG] All 17 deployables verified against CI evidence."
exit 0

The release/preflight gate verifies purely on evidence:

• Does every one of the 17 deployables have a passing build and test record since its last modified commit?
• Are there unverified changes that were merged but have not yet completed their pipelines?

Only when this gate exits with code 0 is tagging permitted.

The Outcome

Implementing Layers −1, 0, and 1 transformed our pipeline reliability:

• 0 failed release tags since deploying the preflight gate.
• 1,151 minutes (41% of weekly CI waste) eliminated immediately.
• 100% agent compliance: The T3 Git shim guarantees no agent operates without governance rules and active typechecks.

In Part 2 of this series, we tackle the remaining 1,377 minutes: how we used cgroup-based CPU budgeting in Vitest, streamlined Playwright containers, and automated weekly DX scorecards to eliminate frontend test contention.