Field notes · 11 Aug 2026 · AI code review
Intercom says AI approves their PRs.
We read the fine print.
Intercom's engineering post — AI is approving our pull requests: here's how we made it safe — is the most-cited AI code-review case study of the year. It deserves to be: real guardrails, real numbers, unusually honest admissions. We dissected it line by line, took every number at face value, and asked what each one would look like in Sober's reality — where we are genuinely stronger, and where we will not pretend.
What Intercom actually built
Intercom ships to production hundreds of times a day, with a 12-minute average merge-to-production time. More than 93% of PRs across their two main codebases are now agent-driven, and more than 19% are auto-approved with no human reviewer. A CTO goal to double R&D productivity in twelve months was hit in nine. Deployments doubled while downtime from breaking changes dropped 35%. In the first four weeks of rollout, 497 PRs went fully autonomous: Claude wrote the code, their system reviewed, approved, and shipped.
The mechanism: a model-based reviewer split into independent sub-agents — one judges problem-description quality, one checks the diff against stated intent, one hunts safety concerns, one checks logic, one reviews against best practices. It "traces execution paths" through the codebase, is grounded in internal standards, will not approve large PRs, and logs everything for auditors (SOC 2, HIPAA, ISO 27001 / 42001, AIUC-1, via Schellman). Any engineer can force a human review at any time. The human who ships remains accountable for the outcome.
The numbers, read honestly
Every figure below is Intercom's, taken at face value — and each gets a Sober counterpart. Where we have real data, it is real. Where we do not, we say so instead of inventing a stat.
| Intercom's number | What it actually proves | Sober's honest counterpart |
|---|---|---|
| 12 min merge → prod | Pipeline speed — their CI/CD, not the reviewer. |
Sober has no production pipeline to brag about. Its gate runs
at a different layer: preflight in the
pre-commit / pre-push hook — milliseconds,
offline, no model. The diff is checked before it
exists as a commit.
|
| >93% agent-driven PRs | Agent adoption in their org, not a review metric. | Sober is agent-agnostic. It does not care whether the author is Claude, Copilot, or a tired human — the guardrails and the review are the same. Adoption is your team's decision, not our KPI. |
| >19% auto-approved, no human | Throughput — and a governance decision. | Structurally impossible in Sober. There is no auto-merge path in the binary. "Never auto-merges, never closes, never deletes" is a source-level guarantee, not a config flag. One in five PRs shipping with zero human review is a number we refuse to offer, not a number we cannot reach. |
| 6–16× faster time-to-approval (75th pct) | Review latency under queue pressure. | Sober review is advisory: it never blocks the queue, because it never merges. Preflight is milliseconds. Model review is seconds to minutes on a branch tip. Human merges on their schedule — there is no queue to back up. |
| 0 reverts in a 100+ PR pilot | A thin, self-selected pilot. Intercom itself: "we don't expect that to hold at zero forever." |
We publish no revert claims, because we have not measured
revert rates at scale and we will not fake them. What we can
prove: the deterministic class — secrets,
private-file leaks, unsafe eval()/exec(),
curl|sh — either matches a pattern or it
does not. Zero variance, zero false-negative drift.
|
| 497 autonomous PRs / 4 weeks | Their rollout scale. |
A fully-autonomous PR count is a number for your
pipeline, not ours. Sober's Raccoon posts advisory labels
(Sober clean / Sober findings /
Sober error) and stops. If you build an
auto-merge pipeline on top of that evidence, the count is
yours to own — and to audit.
|
| Reverts: AI 0.53% vs human 5.39% (backend); 0.22% vs 2.00% (frontend) | The real headline: AI-authored code reverted roughly 10× less — in their environment, with their agents, their review, their codebase. |
A provable, narrower equivalent: the deterministic class
cannot cause a revert, because it never ships. A hardcoded
key or an eval() caught by preflight never
reaches the forge. That is not a claim about revert
rates; it is a claim about a class of failure being
removed, and it holds everywhere.
|
| −35% downtime, deployments doubled | Correlated with the program, not proven caused by it. |
We claim no org-level productivity numbers. Our claim is
smaller and airtight: guardrails run in milliseconds, cost
nothing, work offline, and leave evidence in
.sober/store.sqlite. Measure the rest yourself
— we would rather you measure than we advertise.
|
The guardrails that make it safe — and what Sober does instead
Intercom's safety story is not the model. It is the structure around the model. Guardrail by guardrail:
| Intercom guardrail | Why it matters | Sober today |
|---|---|---|
| Sub-agent decomposition | Each PR gets multiple specialist lenses — the equivalent of a panel, not one tired reviewer. | Model review is single-pass today — one lens, not a sub-agent panel. Honest gap. What we do have is a deterministic multi-detector preflight covering a class their model review does not even address: secrets, private-file leaks, unsafe execution, shell-pipe slop. |
| Execution tracing | Follows a change's implications through the codebase; caught a one-line change contradicting a validation elsewhere. | No cross-file semantic tracing yet. Our evidence is diff-local; we do not pretend otherwise. |
| Grounding + feedback flywheel | Review is grounded in internal standards; engineers flag helpful/unhelpful comments, sharpening the system. | You ground the system yourself — custom rules, tiered rule packs, reviewer personas per group. Disagreements are recorded as false-positive verdicts. The model-sharpening flywheel does not exist yet (gap); the evidence half does. |
| Human-in-the-loop, optional | Anyone can force a human review. Approval is never forced. | Stronger by construction: in Sober the human is the merge gate. Review is advisory by default and no configuration can opt out of human merge authority. It is not a safety valve; it is the valve. |
| Strict size gate | The agent "won't approve large PRs" — forcing small, well-scoped changes. | Same intent, different mechanism: no size refusal, but the hook + advisory findings make large unreviewable PRs uncomfortable, not impossible. The discipline is yours; the signal is ours. |
| Auditability | Every approval labelled, logged, queryable; independently audited against SOC 2 / HIPAA / ISO / AIUC-1. |
Every run, finding, and verdict lands in a per-repo
.sober/store.sqlite — queryable,
exportable, deletable, yours. Honest gap: no
third-party audit yet. The evidence model is
audit-shaped; the audit itself is not done. That is where we
are, and we will not dress it up.
|
Where Sober is genuinely stronger
Stated plainly, without marketing posture:
A deterministic floor the model never reaches
Intercom's safety story starts at model review. Ours starts before
a model is ever called: pattern-exact guardrails for secrets,
private-file leaks, unsafe eval()/exec(),
curl|sh, and AI-slop shapes — milliseconds,
offline, deterministic. A model can hallucinate a missed secret;
a pattern match cannot half-match. This is the layer that makes
"the agent got it wrong" a non-event for a defined class.
No rubber-stamp drift — structurally
Intercom's own admission: when humans cannot keep up, queues back up or humans start rubber-stamping, and some companies are drifting into that silently. In Sober there is no auto-approve path to be gamed or lazily accepted. The reviewer advises; the human merges. The failure mode cannot exist, because the permission does not.
Local-first data flow by default
Intercom's reviewer ingests PR diffs into Claude — a
third-party model pipeline. Sober's default route is
local (Ollama / LM Studio / llama.cpp); remote
routing is explicit opt-in per route, with a
local_only privacy class. For teams handling regulated
data, the diff-vs-model data flow is the whole ballgame, and ours
does not leave the machine unless you say so.
Evidence you own, not a vendor database
Intercom's audit trail lives inside their platform. Yours lives in
.sober/store.sqlite, in the repo, on your disk:
inspectable, portable, deletable. The "labelled, logged, queryable"
property they describe is exactly what Sober gives you — plus
the right to delete it, which a vendor database never offers.
Cost and sovereignty
Sober is free under LSL-1.0; the hardened enterprise container is free; the Raccoon has no per-seat subscription; the binary makes no call-home and ships no keys. Intercom's system is an internal build at a company that can absorb the spend. Ours is the part of that story you can run on a two-person team today.
Where we are honest about the gaps
Intercom shipped a decade of infrastructure ahead of us. Pretending otherwise serves nobody.
No sub-agent decomposition
Their reviewer is a panel of specialist lenses; ours is a single-pass model review. The deterministic preflight covers a different (exact) class, but we do not yet give you the equivalent of a dozen senior reviewers per PR.
No execution tracing
They follow a change's implications through the codebase and caught a real cross-file contradiction that way. Sober's review is diff-local. A change that breaks a contract in another file is something we flag via rules, not by tracing.
No autonomous approve-and-merge pipeline
By design — but a design stance is still a feature you do not have. Teams that want Intercom-style "497 autonomous PRs" must build that pipeline on top of Sober's advisory evidence, with their own merge policy, their own telemetry, their own liability.
No production-diagnosis agent
Intercom's article is blunt that most of their worst outages were not product-code changes at all — they are building a production-diagnosis agent. Sober has nothing on the runtime side; the Raccoon is forge-side only. If you need post-merge runtime diagnosis, that is your stack, not ours — yet.
No third-party compliance audit
Schellman has signed off on their pipeline against SOC 2, HIPAA, ISO 27001/42001, and AIUC-1. Sober is a stable self-hosted project: the evidence model is audit-shaped, the audit is not done. Regulated teams should treat us as "bring your own evidence and your own assessment" until that changes.
No org-scale telemetry of our own
Their numbers come from a large R&D org. Ours are real but modest — live download counters, not productivity studies. We will not extrapolate "2× R&D productivity" from a public download count, and neither should you believe anyone who does.
The uncomfortable truths we agree with
The best parts of Intercom's post are the admissions. We quote them and take them as doctrine:
"The baseline we've been comparing against — human review — was weaker than most of us assumed."
Which is why a deterministic floor under both human and agent review is not optional."When humans can't keep up, queues back up or humans start rubber-stamping."
Every auto-approve feature you ship is a rubber-stamp you are building on purpose. Sober refuses the permission, not the pace."Only in production do you discover the unknown unknowns."
True. Which is why the merge moment is the last cheap control point — and why it should be a human with evidence, not a model alone."The human who ships the code remains accountable for the outcome."
In Sober this is not a policy, it is the architecture: the binary cannot merge. Accountability is enforced by construction.What this means for your team
-
Put a deterministic floor under every agent workflow.
The sub-agents, the tracing, the flywheel — all of that
assumes a diff already exists. Intercom's story never addresses the
hardcoded key that should never have become a diff. Sober's hooks
kill that class at the commit boundary, in milliseconds, offline.
sober hooks install all. -
Keep merge authority human until your data says otherwise
— and maybe after. Sober's review is advisory:
Sober clean,Sober findings,Sober error. There is no--auto-mergeflag to accidentally enable. - If you want Intercom-level autonomy, build it on evidence you own. Wire your own pipeline on top of Sober's advisory labels and SQLite evidence, and measure your own revert rates before and after — the way they did. If the numbers justify it, the pipeline is yours to own and audit. If they do not, you have lost nothing but a flag.
- If your code touches regulated data, notice the data flow. Intercom's reviewer feeds PRs to a third-party model. Sober's default route never leaves the machine. That difference is worth more than any latency number when the auditor arrives.