Part of Mechanisms of Vibe Coding

Mechanism: Confidence Calibration

One-sentence definition

Confidence calibration is matching how much you trust agent output — skim, standard review, line-by-line audit — to risk, novelty, and verify evidence, instead of blanket trust or blanket distrust.

The problem

You merge agent auth changes without reading because “it usually works.” Or you re-read every import in a typo fix and burn an hour. Both are miscalibrated: under-review ships risk; over-review kills vibe coding benefits.

Calibration is a skill: same agent, different scrutiny per task.

Symptoms:

• Security bugs in “small AI PRs”
• Reviewing comment-only changes like payment logic
• No written risk tier for agent work
• Team disagrees on what AI code needs

How it works

Tier tasks before merge:

  Low risk (docs, copy, test-only you wrote)
       → skim diff + spot verify

  Medium (feature in fenced scope, tests pass)
       → full diff + checkpoint questions

  High (auth, money, data, infra, deps)
       → line audit + second human; maybe no agent merge

Factors that raise tier: new dependencies, new public API, permissions, deletes, unfamiliar domain.

Factors that lower tier (not to zero): PR-sized diff, all AC green, exemplar pattern match, playground already exercised.

Calibration pairs with checkpoint — checklist depth follows tier.

When to use it

• Before every merge of agent-generated code
• Team policy for AI-assisted PRs
• Choosing pair vibe vs solo agent
• After incidents — adjust tiers with evidence

When not to use it

• Calibration as excuse to skip verify on high risk — verify is mandatory; depth of diff review still scales
• Trusting tier because “agent used Opus” — model is not risk tier

Failure modes

Uniform trust — All agent PRs rubber-stamped. Fix: Risk tier label on each PR.

Uniform paranoia — No agent value. Fix: Low-risk fast path with verify.

Wrong tier — Payment change labeled docs. Fix: Tag auth/payments/PII as high always.

Calibration without verify — Deep read of code that never ran tests. Fix: Verify loop first.

Stale tier — Small PR grew via drift. Fix: Re-tier if diff grows (vibe drift).

Minimal example

Context: Two agent PRs same day.

• PR A: README documents verbose=1 — low tier: skim + link check.
• PR B: Bootstrap changes startup order — medium: full diff, run full test suite, trace startup once.

Same agent, different review depth — intentional.

Done when: You labeled tier before review and matched effort to label.

Tool instances (optional deep-dive)

Portable idea above; this section is tool-specific. Date: June 2026.

Cursor

• PR template: risk tier checkbox (low / medium / high).
• High tier: require Plan approval + second reviewer in rules.
• Use diff stat to bump tier if budget exceeded.

Related mechanisms

• Checkpoint — review gate scaled by tier
• Completion bias — under-calibration symptom
• Pair vibe — high tier often needs human driver

Try it yourself

Exercise: Label low / medium / high on three recent agent changes. Did your actual review time match?

Done when: You adjust one habit — more depth on high, less ceremony on low (with verify).