Skip to content

Engineering · · 5 min read

Exit 0, and Nothing Was Written

Codex reported success in write mode although no file appeared. The fault was in the Linux sandbox, but the lasting lesson is about our gate.

A successful process above an unchanged tree: checking the requested file exposes the failed write mode.

A worker received a clear assignment: create a file containing one line. Codex ran in write mode, reported exit code 0, and then explained that the patch had failed. The file was not there.

A crash would have been better. A non-zero exit code would have stopped the orchestrator and left the work visibly open. Exit 0 claimed that the worker was finished. Anyone who trusted only the process and its report could close a delegation in which nothing in the repository had changed.

This was not a theoretical edge case. Our delegation policy sends clearly specified implementation work to gpt-5.6. While the fault existed, every such packet either went to a more expensive model or ended as a silent failure. The tool meant to perform checked work needed a gate of its own.

A probe in the right place

The smallest useful test consisted of one file. We started codex exec at the root of the trusted repository and asked for .codex-probe.txt containing exactly one line. We then checked the process status and the filesystem separately.

Probe inside the trusted repository

codex exec "Create .codex-probe.txt containing exactly PROBE_OK"

exit code: 0
.codex-probe.txt: NOT WRITTEN

The patch writer was denied unexpectedly...
patch: failed
Unable to create `.codex-probe.txt`: the workspace write sandbox failed.

Our first probe was worse. It ran under /tmp and ended with “Not inside a trusted directory.” That was an error, but a different one. Only the probe inside the repository reached the patch writer and reproduced the silent write mode. In debugging, the location helps determine which failure you are measuring.

Read-only mode was never affected. That mode does not need to construct a writable sandbox, so it could still inspect repositories, make plans, and deliver reviews. Only the path that was supposed to change files failed.

Two defects behind one symptom

We first recorded the behaviour on 14 August 2026. A second measurement on 1 September again produced REAL_EXIT=0 and FILE=ABSENT. On 2 September the same probe completed after the fix and the file contained PROBE_OK. Write mode had never worked in this host configuration during that period.

Our first explanation was wrong. We suspected that bubblewrap could not nest inside the sandbox of a running agent session. Repeating the probe from an ordinary SSH shell, with no agent sandbox in the process tree, produced the same failure. Nesting was not the cause.

Codex version 0.14x and later uses bubblewrap, or bwrap, on Linux to run the workspace-write process in a sandbox. The distribution package was not installed on the host, so Codex fell back to the binary bundled with its npm package.

Ubuntu 24.04 restricts unprivileged user namespaces through AppArmor. A bwrap binary without a matching profile could not create its namespace and failed while writing the UID mapping with Permission denied. No namespace meant no sandbox. Without the sandbox, the patch writer rejected every change.

The second defect sat one level higher: Codex still returned exit 0 to its caller after this failure. We could repair the sandbox on our host. We could not repair the meaning of that exit code there.

The fix took two attempts

The first fix installed bubblewrap from the Ubuntu distribution before setting up the Codex CLI. That was sensible, but insufficient. We expected the package to provide /etc/apparmor.d/bwrap. The first real Ansible run checked that assumption and failed red.

For package version 0.9.0-1ubuntu0.1, dpkg -L bubblewrap showed the binary and a sysctl file, but no AppArmor profile. The sysctl file sets kernel.unprivileged_userns_clone=1, a Debian control that Ubuntu 24.04 does not use for this decision. Installing the package alone therefore did not change the symptom.

The second commit shipped the profile through our Ansible role and loaded it with apparmor_parser --replace. It follows the form Ubuntu uses for comparable programs: the named binary receives precisely the userns, permission. The role then checks both the file and the profile loaded into the kernel. A file sitting on disk grants nothing.

After that apply, the probe wrote PROBE_OK. Write mode was usable again. The wrong exit code remained known behaviour for any future case in which the patch writer was rejected.

The red gate checks the tree

Our internal verification rule requires more than a green run. A gate is demonstrated only when an intentionally broken input turns it red. For a writing worker, the expected effect must be visible in the working tree.

The countercheck deliberately denied Codex write access with -s read-only. Codex still returned exit 0, the file remained absent as expected, and our downstream gate turned red. That closed the silent path even though the third-party program itself still reported the wrong status.

This tree check now lives in our delegation policy. It remains useful even if a later Codex version corrects the exit code. A process status proves what a process reports about itself. A diff proves whether the requested change exists.

What the plan got wrong

For almost two weeks, we handled every affected delegation packet separately. Implementation was rerouted to another model while read-only work stayed with Codex. The workaround was cheap enough to keep displacing the root-cause investigation.

The first technical diagnosis also fit the visible architecture too neatly. A sandbox appeared to run inside a sandbox, so nesting had to be at fault. Only the probe from the SSH shell disproved that story. The missing distribution package was then plain to see. The first fix immediately introduced another wrong assumption: surely the distribution package would contain the required profile.

Both errors survived only until the next measurement. That is the kindest part of this story. The less comfortable part is that we could have made those measurements sooner.

Conclusion

The specific fault belongs to this Linux host. It would not have occurred in this form on a machine without the AppArmor restriction or one that already had a suitable bwrap profile. The suspected nesting was demonstrably irrelevant.

The operational lesson is broader. Coding agents are workers in a pipeline, and their self-report is not evidence. If they are asked to write, the resulting state has to be checked. For us, that means a non-empty working tree. In another system it may be an artifact, a record, or a reproducible test.

Silence is not a success signal.

Patrick Lehmann

Architecture & Governance Lead

Squibble GmbH

Has spent twenty years bringing structure to IT landscapes that grew rather than were designed – as architect, developer, and operator. Writes here about the systems actually running at Squibble and the decisions behind them.