Workbench: a delivery index that is allowed to be wrong

Workbench answers a question that sounds trivial until you have more than one GitLab instance: what is actually in flight right now? Work items, merge requests and milestones live across several instances and several customers. Nobody wants to open six tabs to find out whether a release is blocked.

The obvious build is a dashboard that queries every instance on page load. That version works for a week and then dies — one slow instance, and the whole page is slow. So we did what everyone does: we put a PostgreSQL read model in front of it.

That is where the interesting decision was, and it was not a technical one. It was a rule about what the database is allowed to know.

GitLab is authoritative. The index is not.

Every projection table in Workbench stores identifiers, titles, URLs, states, dates, people, labels and freshness metadata. It stores no descriptions, no comments, no code, no diffs, and no complete webhook payloads.

This is a hard rule, not a guideline, and it is enforced in review. The reason is not storage cost — it is that the moment your index holds content, someone will read it, and the day it drifts you have shipped a lie with a timestamp on it. Titles and states drift too, of course, but they drift visibly: a stale state is obviously stale. A stale paragraph of prose is not.

The practical consequence is that the whole database is disposable. We can drop every projection table and rebuild it from GitLab. That is not a disaster-recovery story we wrote down and never tested — it is the normal path, because it is also how a new customer is onboarded and how a bug in the sync is fixed.

Reconciliation is the mechanism. Webhooks are an optimisation.

Everyone reaches for webhooks first. Signed project hooks arrive in milliseconds and feel like the real answer.

They are not, for a boring reason: webhooks are a delivery guarantee you do not control. They get dropped, they get replayed, they arrive out of order, they stop silently when someone rotates a token, and they do not exist at all for the window between “project added to scope” and “hook installed”.

So Workbench treats hooks as an accelerator over a system that is already correct without them. A full reconciliation pass reads GitLab and rewrites the projection; repeated syncs are idempotent; removing a project from scope hides it immediately and purges it later. If every webhook in the system stopped arriving tonight, the index would get slower to update and stay correct.

Writing that sentence into an ADR before writing the code changed the code. Every time we were tempted to handle a state transition only in the webhook path, the rule said no — it has to be derivable from a full read as well.

Writes go to GitLab first, and cannot be atomic

Reads are the easy half. Workbench also lets you change things: retitle a work item, move a milestone, merge a request. The order is fixed:

1. Authorize locally.
2. Write to GitLab.
3. Update the projection from the successful GitLab response — not from what we sent.
4. Audit, then enqueue a targeted refresh.

Step three matters more than it looks. Updating the projection from the request body means your index reflects your intent; updating it from the response means it reflects reality, including whatever GitLab normalised, rejected or expanded along the way.

We wrote that limitation into the ADR rather than into a ticket nobody would read. It is the difference between a known trade-off and a bug waiting to be discovered by a customer.

Tenancy: forced RLS, and a context you cannot forget

A workspace is a customer, and it is the security boundary. Every tenant table carries a non-null workspace_id with tenant-aware uniqueness and forced PostgreSQL row-level security — forced, so the table owner does not quietly bypass it.

Application code never filters by workspace by hand. It enters a context:

WorkspaceContext.with(workspace_id) do
  MergeRequestProjection.where(state: 'opened').count
end

WorkspaceContext.with opens a transaction and issues SET LOCAL app.workspace_id. The RLS policies read that setting. Outside a context there is no setting, and every tenant query fails closed — an exception, not an empty result set. Background jobs receive the workspace ID explicitly as an argument and establish their own context; a job may never inherit one.

The failure mode we were designing against is not “someone writes a query without a WHERE”. It is subtler: connection pool reuse leaking a setting into the next request, or a savepoint rollback restoring a context you thought you had left. Those are the cases the test suite exercises, alongside the obvious cross-workspace read and insert.

Three table groups sit deliberately outside RLS, because they are what establishes context in the first place: users, workspaces, memberships, invitations, approved GitLab instances and user GitLab identities. Platform administrators still have to select a single workspace to reach ordinary tenant data, the UI keeps that selection visible, and entering it is audited.

Two connection modes, and no fallback between them

A workspace reaches GitLab either through the current user’s delegated OAuth identity or through a workspace service token. Which one is in play is explicit per connection.

What is explicitly forbidden is falling back from one to the other. It is a tempting two lines of code — if the user’s token is expired, use the service token — and it silently converts “this user may not see this project” into “this project is visible to everyone in the workspace”. Authorization at the GitLab boundary only means something if the identity making the request is the identity you think it is.

What I would keep

Three things from this build are the ones I would carry into any system shaped like it:

• Write the constraint down before the code. “Reconciliation is the correctness mechanism” is one sentence in an ADR that shaped a dozen files.
• Name the gap you did not close. The audit hole on GitLab-first writes and the KMS table outside RLS are both in the record. Neither is a surprise waiting to happen.
• Make the disposable path the normal path. A rebuild you run every week is a rebuild that works.

Observability, earlier in this series, starts from the same shape and then does one thing Workbench never does: it acts on the world. That single verb changes what “disposable” is worth.