Only the Mail Ports Went Quiet
From home, every mail port ran into a timeout while HTTPS on the same host answered normally. The cause was a second [DEFAULT] ignoreip in jail.d that replaces FAIL2BAN_IGNOREIP from docker-mailserver.
· 6 min
Blog · 11 posts
From the workshop
Engineering write-ups about the systems actually running at Squibble – architecture, infrastructure, and the decisions behind them.
Exit 0, and Nothing Was Written
Codex reported success in write mode although no file appeared. The fault was in the Linux sandbox, but the lasting lesson is about our gate.
· 5 min
A webhook is not a data source
Workbench brings GitLab changes into an open browser in under five seconds. The webhook accelerates the route, while truth stays in GitLab and the local projection stays disposable.
· 12 min
357 Seconds per Push, and the Tests Were Not the Problem
Our pre-push gate took almost six minutes. The container read the repository through a bind mount, and that alone accounted for 324 of the 357 seconds.
· 6 min
CI/CD Analytics: Where Pipeline Time Really Goes
A slow or failed pipeline is only a symptom. Workbench connects wasted runner time, job trends, and pipeline waterfalls in one investigation – from the aggregate signal to the exact merge request attempt.
· 7 min
Never Release to Test: Why we had to shift CI and AI agent governance left
16,300 CI minutes in a single week, 18.9% complete waste, and 22 release attempts for a single app: Why releases were abused as test environments, how AI agents silently ignore uninitialised standards, and how a three-tiered filter solved the problem.
· 5 min
Boards: a kanban that doesn't know where its cards are
Every board eventually fails the same way – it remembers which column a card is in, and from then on that memory has to be maintained. Workbench never records it: columns are rules, evaluated on read.
· 10 min
Special vdev: 113 TB hanging on two files
The special vdev of a 113 TB pool sat on two qcow2 files – both on the same consumer SSD, with 11% of its endurance left. Moving it meant operating on a live mirror with no room to be wrong. And it surfaced two checks that had been pointed at the wrong thing the whole time.
· 9 min
Workbench: a delivery index that is allowed to be wrong
We built a multi-tenant view across several GitLab instances without becoming a second source of truth. The design rule that made it work: the local database is disposable, and reconciliation – not webhooks – is what makes it correct.
· 6 min
Env Manager: a secrets store that cannot read its own secrets
A security review pointed out that our secrets manager trusted its own administrators. The fix was zero-knowledge end-to-end encryption – and the honest part of the design is the list of three places where the server still sees plaintext, and why we accepted each one.
· 6 min
Observability: the read model that acts
Workbench could drop its database and rebuild it. This one polls Graylog and files GitLab issues – and a side effect has no upstream. Why dry-run became a database column, why the dashboard has no delete button, and why the safety guard sits at the HTTP boundary rather than in configuration.
· 6 min