A webhook is not a data source

Workbench is a delivery index across multiple GitLab instances. The first article described its most important property: GitLab is authoritative, every local row is derived, and the PostgreSQL projection is allowed to be wrong. Reconciliation, another read from GitLab, repairs it.

That property remained. Its time window became the problem.

Someone could close a work item in GitLab, switch back to an already-open Workbench board, and still see the old state. The page knew nothing about the change until the next reconciliation pass or a manual reload. It became correct later. For a tool where teams watch active work, later was too late.

The new target was therefore precise: a work item or merge request change should appear in an already-open browser in under five seconds at p95. The measurement starts when Workbench receives the webhook and ends after the browser completes its refetch. The index may still be wrong. It just cannot stay wrong for as long.

Why polling gets more expensive with every workspace

Polling would have been the straightforward solution. Every open browser could ask periodically whether anything had changed. Alternatively, the server could read every GitLab project at a fixed interval.

Both versions pay for elapsed time instead of changes. A workspace with many projects produces reads even when nobody has touched a work item. Every additional workspace adds to that baseline. The more useful Workbench becomes, the more scope it has to query for no particular reason.

Polling still has a role: repair. The existing incremental reconciliation reads GitLab every five minutes and repairs missed or delayed events. A full pass runs nightly. Those jobs buy correctness. They do not need to make an interaction feel fast.

Webhooks buy freshness. GitLab sends one only when something changes. The recurring work therefore shifts from “read everything again” to “read the affected record now”.

Installing the smallest complete hook set

A webhook is an HTTP request initiated by GitLab and sent to Workbench. The first naive topology installed one project hook per project. That works until a workspace covers a whole group with many projects. At that point dozens of hooks describe the same scope separately and each one must be installed, checked, and removed.

Workbench therefore calculates the smallest complete hook set for each GitLab connection. A group hook covers its group and subgroups. It suppresses redundant project hooks even when a project is also configured as an explicit scope. Overlapping groups retain only the ancestor hook.

GitLab Free does not provide group hooks in this model. A group scope there falls back deliberately to project hooks for the projects Workbench discovers. Coverage is complete in both cases; only the number of remote hooks differs.

Reconciliation of this topology is idempotent: repeating it creates no duplicates. A drift sweep checks every 15 minutes for hooks deleted or automatically disabled outside Workbench. The full hook reconciliation runs nightly and retries group hooks that the tier previously refused. Both install missing hooks and remove redundant ones. If GitLab is unavailable during teardown, the registration stays in error and removal can be retried later. A failed remote call must not erase the local record of unfinished work.

The receiver verifies the raw body

A publicly reachable receiver is a trust boundary. It may understand what a request claims only after verifying who sent it.

From GitLab 19.1 onwards, Standard Webhooks sign the exact raw body: the unmodified bytes of the request. Workbench calculates an HMAC-SHA256 over those bytes. An HMAC is a cryptographic check derived from a message and a shared secret. JSON is parsed only after the signature and timestamp are valid.

The order matters. If Rails parsed and serialized the body first, whitespace or JSON key ordering alone could change the bytes. The signature belongs to the received message, not to a semantically similar reconstruction.

An unknown registration, invalid signature, expired timestamp, and suspended registration all receive the same response: 404. The endpoint does not reveal whether a particular public ID exists. From the outside, there is no oracle for valid registrations.

After successful verification, the receiver writes a delivery to PostgreSQL and enqueues the job. Only then does it respond with 202. Deduplication also lives in PostgreSQL: the delivery ID is unique per registration, while the GitLab event UUID suppresses the same event arriving through accidentally overlapping hooks. Restarting the Rails process does not clear this memory.

The event is only a routing envelope

The tempting next step would be writing a title, state, or labels from the webhook payload straight into the projection. The fast path would then own a second data source. Different event types supply different fields, payload versions change, and a delayed event could roll a newer state backwards.

Workbench therefore retains only a routing envelope from the payload: event type, project, resource kind, and IID. The envelope answers where to read. It never answers what is currently there.

A work item triggers a targeted GitLab read through GLQL, GitLab’s query language for work items. A merge request triggers one REST read. Only that response is projected. If safe coordinates are missing, the job falls back to the existing full project sync.

Webhook payloads provide routing coordinates only. Workbench then reads current state from GitLab.

The fast path therefore looks like this:

1. 01WebhookReceive the routing envelope
2. 02HMACVerify the raw body and deduplicate
3. 03GitLab readRead a work item via GLQL or an MR via REST
4. 04CommitUpdate the projection transactionally
5. 05BroadcastSend workspace coordinates through Action Cable
6. 06InvalidationInvalidate the TanStack Query prefix and refetch

Old reads cannot overwrite new data

Webhooks can arrive twice, late, or out of order. Even two correctly routed jobs can finish their GitLab reads in an unexpected order. source_updated_at therefore decides whether a targeted upsert may write. An older read cannot overwrite a projection carrying a newer GitLab timestamp.

A targeted upsert must also never deactivate its siblings. A read that returns exactly one work item knows nothing about every other work item in the project. Only a full authoritative snapshot may infer from absence that a record has left the scope.

This uncovered an older defect. The projector incremented projection_version on every sync, even when the persisted bytes were identical. Such a no-op looked like a change and would have emitted an unnecessary broadcast. The version now changes only after a material change.

Together, the timestamp comparison, bounded scope, and no-op detection form the stale-write guard. More speed does not create new forms of drift.

After commit, the browser receives coordinates only

The broadcast happens after the database commit. Broadcasting earlier could make a browser refetch immediately and still read the old state. Afterwards, Workbench sends only the workspace, project, resource kind, and ID or IID through Action Cable. Titles, labels, and other GitLab content never travel through the channel.

Action Cable uses its PostgreSQL subscription adapter. A commit therefore reaches the web process even when the job ran in another process. The workspace channel checks current membership when a client subscribes. A client cannot subscribe to another workspace by guessing its ID.

In the React frontend, the message invalidates every matching TanStack Query key through their common prefix. The active query then reads from the local API again. One coordinate can update the list, detail view, and board without recreating their concrete cache keys in Rails.

Connections break. That is normal. On reconnect, or when the browser window becomes active again, the client invalidates every projection for the current workspace. Missed broadcasts therefore require no server-side replay buffer. The existing read path heals the client.

The E2E proof crosses the real boundaries

MR !480’s Ruby suite ran 2,748 examples with no failures using mise run applications:workbench:test. The same suite measured 8,670 of 8,670 lines and 1,479 of 1,479 branches covered. The frontend coverage suite applications:workbench:test:frontend:coverage ran 2,369 tests. mise run verify --changed ran 1,246 CI tests for the affected paths.

Those figures establish breadth. The E2E process mattered more for the central claim. applications:workbench:e2e:process ran 45 scenarios and measured the route across real process boundaries.

The asserted title exists only in Fake GitLab. It never appears in the webhook payload. If it reaches the browser, Workbench must have performed the targeted read. The projection commits in a different process from the browser connection. If the invalidation arrives, it must have travelled through Action Cable’s PostgreSQL adapter. The spec additionally verifies that the page was never reloaded.

In that setup, projection commit to visible browser change took 52 ms. Delivery receipt to visible change took 2,196 ms. Both values come from applications:workbench:e2e:process; the latter measures the agreed 5,000 ms SLO.

This is not a production p95. It is a reproducible E2E proof of the complete mechanism and its 5,000 ms budget. Operational measurements must confirm the SLO over many real deliveries.

How a workspace turns real-time on

Everything above happens once a hook is installed. Nobody installs one by hand. Workbench creates, updates and deletes every hook itself through the GitLab API. What a maintainer owns are the conditions under which that succeeds.

They live in the settings, on a card per GitLab connection, under “Before you enable”. The list is ordered because the items build on each other. Each carries a state: Ready, Action needed, Not checked or Check yourself. The state is a word, not only a colour. Someone reading the page on a projector, or with a colour vision deficiency, still has to be able to tell “ready” from “something is missing here”.

1. A base URL GitLab can call. Without APP_URL installation aborts. A wrong URL is worse than a missing one: GitLab accepts the hook and delivers into the void. That looks like working synchronization, except nothing is ever fresh. This is why the card prints the receiver URL back instead of assuming it is known.
2. A workspace service token on the connection. Connections that sign in as a person through delegated OAuth are excluded from all unattended work (ADR 0003). Their access ends the moment that person signs out. An installed hook keeps firing. Such connections cannot arm real-time synchronization at all.
3. GitLab 19.1 or newer. Below that the Standard Webhooks headers are missing, and there is deliberately no legacy path over X-Gitlab-Token. The card shows the version recorded by the last connection test, and Not checked before that.
4. Token scope and roles. The api scope, plus Maintainer on every project in scope. For a single group hook, additionally Owner on the group and an instance on Premium or Ultimate.
5. Outbound requests to the local network. This applies as soon as the receiver URL resolves to a private address. It then needs the admin setting “Allow requests to the local network from webhooks and integrations”. Without it GitLab refuses the hook at creation time, and the card translates the error for you.

The sixth item stays manual, and the card says so: Check yourself. A curl from the GitLab host against the receiver URL proves reachability, and the expected answer is 404. The receiver refuses every delivery whose signature it cannot verify. A timeout, a TLS error or a proxy’s 502 proves the opposite.

Only then does the button appear. Installation runs in a background reconciliation pass, so coverage fills in over the following seconds rather than with the click. The card then shows two blocks: how many of the intended hooks are installed, including the pending, failed and GitLab-suspended ones, and how fresh the last 24 hours of deliveries were, with median and 95th percentile against the 5,000 ms budget.

The two timestamps below them are deliberately separate: last delivery received, and last broadcast. When receipts keep coming while broadcasts stop, the fault is not on GitLab’s side. The browser’s Action Cable connection is gone.

Four actions sit below the status rather than above it, because each one is a response to something the status claimed: request a test delivery, retry reconciliation, disable real-time synchronization, and force-remove hooks from GitLab. That last button is the only destructive one. It changes GitLab, not the workspace’s setting, and it exists for hooks left behind by a failed teardown.

The error message is always the same; the category is not

GitLab answers every refused hook operation with the same sentence: “GitLab request failed”. As a diagnosis that is worthless. The card shows it verbatim anyway and puts the category beside it, because the category is the part an action follows from.

One case deserves its own explanation, because it looks like a failure and is not. GitLab never announces whether an instance offers group hooks at all. Workbench learns it as a 403 at installation time and then falls back to project hooks silently. The result is a correct installation, just a larger one.

“Retry reconciliation” can be pressed as often as you like. The pass compares the remote hook list against each registration’s stable URL, so it converges without duplicates, and it forgets an earlier group-hook refusal. A role granted afterwards therefore takes effect on the next attempt.

That leaves the harder case: coverage reads fully active and still nothing arrives. GitLab’s own record is authoritative then, not Workbench’s. Under Settings → Webhooks → Edit → Recent Deliveries every attempt appears with its response status.

The 404 in that table is the interesting one. Against a hand-made curl it is the right answer; against GitLab’s own signed delivery it is a defect. The same number means opposite things in two places. What that cost us is in the next section.

What the plan got wrong

The first version of ADR 0013 proposed cross-workspace streaming. One channel would have served several workspaces. The authorization spec showed that the model undermined the required isolation, so it was removed from the ADR. Every subscription now belongs to exactly one workspace.

The projection_version no-op bug did not originate in this project either. It was already there. One passage in the boards article made the system sound cleaner than it was: repeated syncs changed no domain data, but they still changed the version. Realtime exposed that inaccuracy because every false version change could have become a false broadcast.

The most expensive mistake only surfaced in operation, after this article was published. The reconciler sent the signing secret under GitLab’s legacy token attribute. That is a different field from signing_token: token makes GitLab echo the value back in a plaintext X-Gitlab-Token header, and the receiver never checks that scheme. Every installed hook was therefore unsigned, and every real delivery failed signature verification.

None of it was visible. The hooks existed in GitLab, coverage reported them active, and the receiver answered 404. The same answer denies an attacker an oracle over valid registrations. It also hides from the operator whether a signature just failed or a registration was missing. It surfaced on a hook whose signing_token_present read false and token_present read true, and it was reproduced through GitLab’s synthetic test delivery, the same button that now sits on the card.

One Rack Attack spec remains flaky. It can cross the minute boundary of its throttle bucket. We deliberately did not pull that repair into this change. It is open test work, not evidence against the data path described here.

Conclusion

The full project sync remains. It is the fallback for ambiguous events, the repair path for missed deliveries, and the final authority when drift appears. Webhooks do not replace it. They only shorten the time until the next authoritative read.

GitLab versions before 19.1, system hooks, event sourcing, and retained full webhook payloads remain deliberately out of scope. None is required to bring a change into an open browser quickly and safely.

The projection may still be wrong. Now it notices sooner.