Review as the Control Point

Sources#

3100 Opinions on Code Review in an AI World: Building Causal Theory from Practitioner Discourse

Summary#

Agarwal, Miller, Kästner & Vasilescu (Carnegie Mellon, arXiv 2607.07980, July 2026) synthesize practitioner discourse at scale into an explanatory causal theory of how coding agents reshape code review: 26 constructs and 67 relationships (64 directed, 3 contested), built from 3,100 coded grey-literature documents. Its organizing claim: review is the control point through which a coding agent's effect on software is decided, and AI does not fix the sign of that effect — the team does, through the expertise its reviewers bring and how it adapts the review process. "AI is changing code review" is turned into falsifiable propositions with named constructs and moderators.

The paper is the vault's first non-vendor, mechanism-level account of AI's effect on review dynamics, and a direct foil to Faros's vendor telemetry (see the contradiction below).

Evidence note. empirical, with a load-bearing distinction. The theory is a proposed explanatory theory grounded in what practitioners argue, filtered through the researchers — "we confirm none of these relationships." Its constructs and edges are hypotheses to test, not measured effects; treat P1–P17 as falsifiable claims, not findings. The genuinely measured part is a motivating observational GitHub study (2.5M+ PRs across 2,860 agent-exposed public repos) — real telemetry, but the authors' headline is that it is direction-unstable under defensible analysis choices. So this page does not out-measure Faros on quality outcomes (it measures none); what it adds is non-vendor over-time telemetry and a rigorous argument that surface trends can't be read without a causal model.

The three moderators (the paper's punchline)#

The core claim is that agents don't monotonically help or harm review; they pressure the review system without fixing any single outcome, and three moderators set which way it breaks:

Reviewer expertise and disposition — the skill a reviewer brings and whether they read AI code with a trusting or a skeptical frame.
Automated-reviewer capability — how good the AI reviewer folded into the loop actually is.
Process adaptation — the review governance policy the team adopts in response.

This is the review-side mirror of Returns to Expertise in Agentic Coding: what amplifies (or protects against) a coding agent is the human's understanding and the team's practices, not the technology.

The shape of the theory: drivers → review dynamics → outcomes#

The theory (Figure 2) reads top-to-bottom. Node shape encodes role, edge style encodes sign (solid = increases, dashed = decreases, bold = contested, sign set by a moderator):

Drivers (exogenous properties of agent code, rectangles): increased volume & velocity of code production ("reviewer throughput grows linearly with headcount; AI generation grows multiplicatively per developer"), surface plausibility (reads clean and idiomatic), code opacity / lost intent (no human formed the rationale), output inconsistency (run-to-run variance), and data & IP exposure risk.
Review dynamics (the steerable middle, ellipses): skepticism, use of automated review & gating, review governance policy, review budget, focus on high-level intent, review depth, comprehension debt, reviewer skill & experience, coding skill, collective ownership, review effectiveness, review efficiency, review motivation.
Outcomes (what practitioners care about, hexagons): code quality, code security, review throughput, review latency, maintainability, knowledge transfer & mentorship.

The central core is review depth + reviewer skill — the busiest constructs, through which surface plausibility, ownership, and coding skill reach throughput and quality. The core feeds back on itself (see the loops below).

The key mechanisms (falsifiable propositions)#

Volume → shallower review or burnout. Higher review load decreases review depth (P1: skimming not reading — "reviewing 200 AI-generated PRs a sprint… they're skimming, not reading") and decreases review motivation (P2), and shifting relative time toward reviewing further demotivates (P3: "after a few hours reading AI code, I'm more drained than after twice as long writing it"). The endpoint practitioners fear: review becomes "shallow or abandoned… performative."
Surface plausibility disarms the reviewer. Polish lowers the reviewer's guard (P4: "polish can lower the reviewer's guard"; "seniors rubber-stamping code that looks idiomatic but hides subtle bugs") — a defect "dressed as correct code" clears faster because it's scrutinized less. This is Polish No Longer Signals Readiness at the review layer.
Skepticism is the mirror. Treating AI code as suspect — "like a junior dev who's super confident but only moderately competent, I read every line like it's a puzzle" — raises depth and effectiveness, lowers efficiency (P5), and can compensate for surface plausibility; it's fed by awareness of output inconsistency (P6).
Opacity degrades review three ways (P7): reconstructing intent lowers efficiency, the missing yardstick lowers effectiveness even at full effort, and the drudgery lowers motivation.
Automated review: throughput yes, quality contested. Full automation raises combined throughput and cuts latency (P8), but its effect on code quality/security is genuinely contested (P9) — one camp cites CodeRabbit catching ~82% of planted bugs; the other says it "catches style, misses the race-condition-under-load kind of problems" and manufactures false confidence.
The comprehension trap (P10/P11). AI-assisted review climbs the abstraction ladder ("you are no longer reading every line, you are now an auditor") and raises reviewer skill — but focusing on high-level intent accrues comprehension debt even when review depth is high: "we are not reviewing the recipe, we are tasting the dish."
Governance policy (P16/P17). A policy that mandates a careful pass raises depth (effectiveness up, efficiency down); its effect on latency is contested and set by calibration — a risk-tiered policy that gates only material changes lowers latency, a blanket policy that gates everything raises it.

The feedback loops (the under-attended human costs)#

The paper's structural contribution is showing that the core loops back on itself, so decisions that buy throughput today erode the capacity for good review tomorrow:

Low review depth limits growth in reviewer skill (P12) and, compounded by less time coding + low-quality reviews received, undermines the coding skill that review skill is downstream of (P13) — "how can you review what you don't yet know how to build?"
Review depth builds collective ownership and knowledge transfer; automated review undercuts both by lowering depth and by shortcutting the human arrangement that carried them (P14) — "when no human has deeply reviewed the logic, nobody owns it."
Low depth + opacity increase comprehension debt, which erodes maintainability, ownership, knowledge transfer, and future review skill (P15) — "if you're merging PRs you don't fully understand, the bus factor climbs to dangerous levels."

Whether the loop turns virtuous (skepticism → deeper review → sharper reviewers) or vicious (rubber-stamping → skill atrophy → still shallower review) depends on many interacting decisions — the paper frames this as a system-dynamics question, not a monocausal one.

The observational study: real telemetry, unstable trends#

The theory grew out of the authors' failure to interpret their own GitHub data. Re-scraping full PR histories (Jan 2020–Feb 2026) for 2,860 agent-exposed repos from the Agents in the Wild corpus, they found agent-authored PRs are, at a point in time, reviewed less independently (author-only review 40.1% vs 21.5% for human PRs; paired Wilcoxon p<0.001), merged several times faster (median minutes-to-hours vs hours), and discussed less (50.2% get no review comments vs 37.9%; thinner even per line changed). This matches prior snapshot studies.

But two things break the easy reading:

The no-review rate converges, not diverges. The share of merged agent PRs receiving no human review falls from >50% in mid-2025 toward the stable human baseline of ~14% by early 2026 — an initial willingness to merge agent output unchecked gives way to reviewing it much as human PRs are reviewed. (This runs opposite to Yu et al.'s within-reviewer habituation, where oversight weakens over time — so studies disagree even on the basic direction of whether oversight is consolidating or eroding.)
The sign flips with a definitional choice. Whether "independent review" of agent PRs is above or below the human rate depends on whether you count the developer who invoked the agent as an independent reviewer (agent-as-author) or as the author reviewing their own work (agent-as-tool). Same data, opposite conclusion. Figures 5 and 6 support different headlines from the same rows.

The lesson (Pearl: "data are profoundly dumb"): surface traces establish what is changing, not why, and can't be read without a theory of what review is for.

Contradiction: this paper vs. Faros's Acceleration Whiplash#

Faros 2026 (vendor-claim) argues the throughput/quality gap widens with adoption and hits even high-maturity orgs ("even the strongest foundations are buckling"; 31.3% of PRs merged with no review is "the most urgent finding"), i.e., AI's downstream damage has a roughly fixed, worsening sign. This paper tensions that on two axes:

Direction over time. Faros reads under-review as a deepening crisis; this paper's non-vendor GitHub telemetry finds the agent no-review rate converging down toward the human baseline as orgs learn to review agent code. Caveat both ways: different populations (open-source GitHub here vs Faros's largely-enterprise telemetry), different metric (coverage over calendar time vs adoption-depth cross-section), and this study is observational with no non-adopter control.
Is the sign fixed? Faros's framing is deterministic ("maturity doesn't protect"); this paper's whole thesis is that AI does not fix the sign — the team's expertise and process do, which is closer to DORA's "strong foundations protect you" than to Faros. But it doesn't measure a maturity effect; it argues the moderators exist, grounded in discourse.

Weighting: neither out-measures the other on quality outcomes (this paper measures none; Faros's are genuine but vendor-selected telemetry). The durable takeaways are (a) non-vendor evidence that under-review of agent code is a transient early-adoption phase, not a widening gap, at least in open source, and (b) a well-argued case that Faros's fixed-sign framing skips the moderators that decide the outcome. A full reconciliation — the "divergence" dissolves into four axes of non-comparability (delta-in-count vs level-of-share, enterprise vs open source, adoption-depth cross-section vs calendar trend, all-PRs vs agent-PRs), leaving only the volume forecast genuinely contested — is filed at The Under-Review Divergence: Faros's Widening Crisis vs. CMU's Convergence.

Connections#

Acceleration Whiplash — the direct foil: Faros's vendor telemetry says the gap widens and maturity doesn't protect; this non-vendor theory says the sign is team-set and the no-review rate converges (see the contradiction above)
Telemetry vs. Survey Measurement — sharpens the methodology debate: this is the non-vendor telemetry that OQ asked for, but its headline is that telemetry is direction-unstable without a causal model — telemetry beats surveys on latency yet can't adjudicate alone
AI as Primary Author — supplies this theory's drivers: volume/velocity, surface plausibility, and code opacity (lost intent) are exactly Faros's authorship shift seen as inputs to review; the "is the invoking human an independent reviewer or the author?" ambiguity is the review-side of "what does acceptance mean"
Verification as the New Bottleneck — the control-point claim is the bottleneck thesis with a mechanism map; answers "how far do you push automated reviews?" — throughput/latency yes (P8), quality/security contested (P9), latency sign set by policy calibration (P17)
Returns to Expertise in Agentic Coding — the review-side mirror: reviewer expertise + disposition is the first of the three moderators that set the sign; expertise amplifies (and protects) here as it does at the authoring keyboard there
Outsource Your Thinking, Not Your Understanding — comprehension debt is Karpathy's "you can't outsource understanding" measured as a review construct; "tasting the dish, not reading the recipe" is the erosion mechanism
Agentic Technical Debt — comprehension debt is the cognitive twin of architectural drift; both compound through reinforcing loops at agentic-coding speed
Polish No Longer Signals Readiness — surface plausibility at the review layer: polish disarms the reviewer's defenses (P4), so a prod-looking diff signals less than it seems
AI Brain Fry — review load → fatigue → rubber-stamping (P2/P3) is the oversight-fatigue mechanism, sourced here from practitioner discourse rather than a controlled experiment
Loop Engineering — "fold human review into the agent's own loop / a second sub-agent checks the first" is one of the paper's enumerated positions; the maker/checker split is automated review as a review-dynamics construct
LLM-Assisted Grey-Literature Theory Building — the method that produced this theory (the paper's secondary contribution)

Open Questions#

Every one of the 67 relationships is a hypothesis, not a finding — the paper's explicit call is for causal-estimand studies (controlling for the other constructs) to confirm, reverse, or drop each edge. Which of P1–P17 survive measurement?
Does the no-review convergence hold as agentic authoring crosses from <1% of PRs toward double digits, or does the early-adoption discipline break down under volume the way Faros predicts?
The paper's own question: which decisions, under which conditions, push the system toward the virtuous loop rather than the vicious one? — the system-dynamics leverage-point analysis it gestures at but doesn't run.
The three contested edges (automated review → quality/security, P9; governance → latency, P17; and one more) are contested because their sign is moderator-set — what are the moderator thresholds that flip them?

Sources#

3100 Opinions on Code Review in an AI World: Building Causal Theory from Practitioner Discourse — Agarwal, Miller, Kästner & Vasilescu (CMU, arXiv 2607.07980, 2026-07-08), empirical. §IV (the theory, constructs + P1–P17), §II-B & Appendix 1 (motivating observational study, finding-instability), §VI (discussion: control point, three moderators, system-dynamics framing), §VII (conclusion)