Benchmarks
Proof, not promises.
Many reviewers say they find bugs. Here is how we test Forseti, what it caught, what it missed and what it cost, so you can judge it before you connect a repository.
Measured 28 Oct 2026Standard depthour own test set
Headline results
- 24 of 29 planted bugs caught
- 83%, at Standard depth
- 20 of 24 serious bugs caught
- the ones rated high or critical
- 0 blocking findings on clean PRs
- two PRs with no planted bug
- £0.43 a Standard review
- for a PR of about 1,000 lines
We ran the set before and after this week's cost changes and the score did not move: the same bugs caught and the same bugs missed on every PR. The model cost of all sixteen reviews together, on a cold run with nothing cached, was $1.30 on the engine as it ships now, about eight cents a PR. A re-run with the prompt cache warm cost less ($0.69 once), but that is not the cost to expect: repeat reviews are cheaper than a first one (see the limits). No Glance, Standard or Deep review costs more than £2 (Molecular, in early access, shows its own price first), and on the large PRs the ceiling cut nothing: no file went unreviewed.
Method
Plant a bug, then see if it is found.
The same four steps every time. The key and the scoring script are kept in our repository.
We plant the bugs
Sixteen pull requests in a private sandbox repository, from 44 to 3,391 changed lines. We hide 29 bugs in them: a missing await, a tenant taken from the URL, an off-by-one page, a locking migration, a timing-unsafe compare. Two PRs have none, to catch false alarms. Four are large drafts with decoys.
We write the answer key first
Each bug is written down before any review runs: its file, its lines and how serious it is. The key is in the repository and does not change after the run.
Forseti reviews them, for real
Standard depth, the real models and the real engine, with the review held back from GitHub so nothing is posted.
A fixed rule scores it
A bug counts as caught when a finding is on its file and its lines overlap the bug's lines, give or take four. A finding that matches no planted bug is counted as an extra, not as a catch. A script applies the rule, so the score does not depend on who reads it.
By depth
Deeper finds more, and costs more.
Planted bugs caught at each depth, from the depth comparison run on 20 October 2026. Where a depth was run more than once, every run is listed. Small PRs: 15 bugs. Large PRs: 14 bugs.
| Depth | Small PRs caught | Large PRs caught | Model cost per PR (small / large) |
|---|---|---|---|
| Glancea fast summary, no line comments | 8, 10, 8 of 15 | 6, 6, 7 of 14 | $0.002 / $0.012 |
| Standard | 13 of 15 | 11, 12 of 14 | $0.040 / $0.156 |
| Deep | 13 of 15 | 13 of 14 | $0.060 / $0.305 |
Cost is the model cost of the run, in US dollars, not the price you pay. Deep ran with a second-opinion model until 20 October; the row shows Deep without it, which is how Deep runs now. Molecular is in early access and is not scored here.
What it costs you
The price is on the PR before it runs.
Our price for a PR of about 1,000 changed lines. Smaller PRs cost less. No Glance, Standard or Deep review costs more than £2; Molecular (early access) shows its price before it runs.
Work out your own month with the pricing calculator.
Glance
£0.04
per medium PR
Standard
£0.43
per medium PR
Deep
£0.58
per medium PR
Misses
What it did not catch.
- 9a, a break in webhook.ts, a file the PR does not change. Every depth missed it.
- A tenant read from the URL (6b) at Standard and Deep.
- On the large PRs, Standard missed an off-by-one (1c) and one other medium bug (2c) in both runs.
- Glance, by design, misses most subtle bugs: a race on credits, a status renamed across files, a locking migration.
Limits
What this does not show.
- It is our own test set. We planted the bugs and chose the PRs, so a real codebase will differ. Read it as a method you can check, not a promise about your code.
- It is not a head-to-head. We have not run the same PRs through other reviewers, so this page says nothing about how they score.
- It is small: 16 PRs, 29 bugs, mostly one run each. One more catch or miss moves the percentage by about three points, and results vary from run to run.
- Extra findings are not graded. Forseti raised 19 findings that match no planted bug. Some are real issues we did not plant, such as a CSV formula injection; others may be noise. None was blocking on the clean PRs.
- Costs here are measured, and prompt caching makes a repeat review cheaper than a first one. Prices on this site are quoted from first-review cost.
Judge it on your own pull request.
Connect GitHub in four clicks. Add credit, and your first review runs on your latest small open PR, priced before it runs.