Observability

The problem

Collecting logs centrally is the easy part. What follows is a stream in which the same error appears a thousand times over and a new one drowns. Whoever turns that into tickets does it by hand and eventually stops. Whoever does not has a search box instead of an overview.

How it solves it

Observability queries Graylog on a schedule, clusters log events into incidents and aggregates their severity. It keeps those clusters in step with GitLab issues in both directions, under a label lifecycle that is the same for every project. A React SPA shows the overview, the metrics, the trend and the list of what demands action.

What it does

A schedule instead of a dashboard you have to look at. Every configured application is polled every five minutes, the daily digest goes out at 08:00 Europe/Zurich, and at 08:10 quiet clusters close themselves.

Seven mandatory label mappings per GitLab connection. Type, opened and closed, priority P0 to P3. Integrations may add project-specific labels, but type::, priority:: and status:: are centrally governed. Updates are applied as a delta so foreign labels survive; the unscoped blocked label is never touched by the sync.

A preflight that cannot create shadow labels. It loads every label visible to the target project, inherited group labels included, and checks all seven mappings. If one is missing, or GitLab is unreachable, the writing call aborts in the preflight.

Outbound calls in production only. Traffic to Graylog and GitLab requires a production environment and an exactly-set switch. The check sits at both Faraday boundaries, so a copied production token does not route around it. An integration’s dry-run flag is binding per run and cannot be overridden by any global variable.

Backfill over time windows, dry by default. A historical backfill works in five-minute buckets and initially produces clusters, observations and issue previews only. The real sync requires an explicit fourth argument.

What it does not do

It does not collect logs. Graylog stays the source; Observability is the triage above it.

It knows no configuration files and no tokens from environment variables. The only source is the integration, Graylog connection and GitLab connection records, with ActiveRecord-encrypted tokens that never go back to the client.

It runs as an internal production system for ourselves today and has no external users.