↓Skip to main content
  1. Posts/

How to Judge an 'AI Found N Vulnerabilities' Claim

·2841 words·14 mins·
Security Ai Vulnerabilities Cve Analysis
Dave Amit
Author
Dave Amit
Principal Architect with nearly two decades designing distributed systems and cloud-native platforms. I bring deep systems expertise, hands-on AI-assisted engineering workflows, and a track record of shipping things that actually scale. Currently writing Go and Rust, running on Kubernetes, and exploring what happens when strong architectural judgment meets modern AI tooling.
Table of Contents

So the headline making the rounds is that Anthropic’s Project Glasswing produced 37,966 findings and only 217 fixes, and both of those numbers are real. They come from a November 9 post by Patrick Garrity at VulnCheck, “Tracking Anthropic-discovered vulnerabilities”, and both appear on Anthropic’s own public dashboard and ledger. There’s a third number on that same Anthropic page, 629 patched upstream, and the article’s text doesn’t mention it — though I didn’t read the chart image embedded in the article, so I can’t tell you what’s in there.

Before going further, the disclosure, once. I use Claude, made by Anthropic, to make my videos, and Anthropic is the party being audited. VulnCheck sells vulnerability intelligence, runs the KEV list (a catalogue of vulnerabilities seen exploited) that the post cites for “exploited in the wild”, and is the CNA that assigned one of the two CVEs it cites. Those are facts about who’s holding which number, and I’m going to stick to the numbers and the definitions.

This post is not a verdict on either side. What I want out of it is a method: five checks that any “AI found N vulnerabilities” claim should survive, each one introduced by the real number that forced it when I went through VulnCheck’s audit.

What I rebuilt, and what matched
#

Anthropic’s ledger is a list of hash commitments — one SHA-3-512 hash per finding, with the details revealed only after a disclosure window closes. VulnCheck lines that ledger up against public CVE data and against one maintainer’s triage of a batch of reports. To see how far its figures hold, I pulled the ledger’s data file (payload.json) and three of Anthropic’s archived snapshots (May 22, Aug 26, Oct 2), and rebuilt what could be rebuilt.

From the ledger rows and snapshots I reproduced 7,349 ledger rows, 217 with status fixed, 1,654 duplicates, 419 withdrawn, 235 revealed, the launch figures of 23,019 candidates and a 90.8% true positive rate, and the whole Aug 26 column of the post’s table (26,153 / 2,736 / 202 / 245 / 91.4%). The rest of what VulnCheck quotes from Anthropic, 37,966 candidates and the 94.2% rate, matches the headline figures on Anthropic’s dashboard, though those can’t be recomputed from the public rows, so I won’t call them “reproduced”.

One exception: the post’s table says 1,670 duplicates while its text says 1,654, and the ledger says 1,654. The figures that aren’t Anthropic’s to begin with (442 CVEs, 93,450, the two exploited CVEs, the curl counts) get their own treatment below.

If the numbers match, where’s the problem? It’s in what the words next to the numbers count.

Check 1: What is being counted?
#

Start with 37,966. Anthropic’s About page defines “Candidates” as “Every distinct crash or vulnerability hypothesis that Claude produced across the program, before any triage.” The dashboard uses “findings” as the unit label for every stage, so an unreviewed hypothesis and a confirmed bug get the same word, which is a lot to ask of one noun. Then the stages narrow, per the dashboard as of November 2:

  • 37,966 candidates
  • 6,376 reviewed by external security firms
  • 6,009 confirmed valid, which is where the 94.2% comes from (6,009 / 6,376)
  • 5,087 disclosed, of which 1,862 are labeled “Reported to maintainers” and 3,225 “Reported direct to maintainers by Anthropic”, the second group marked “May contain false positives”
  • 629 patched upstream, defined as “A patch has landed after the report was sent”

Two phrases in VulnCheck’s post sit awkwardly against that list. It writes “Of the 37,966 findings Anthropic reports, only 7,349 (19.4%) have been made public.” But 7,349 is the number of ledger rows, and each row is a hash commitment; before notification the page shows only a hash and a date. Anthropic’s own count of findings revealed with details is 235 and its count of disclosed is 5,087, so “made public” is VulnCheck’s label and doesn’t match either. Separately, the post says it tracked “more than twice as many as Anthropic has disclosed itself” while setting 442 CVEs against the ledger’s 217 fixed; Anthropic’s “disclosed” is 5,087, and 217 is a ledger status.

Duplicates need a read of the definition too. The About page says a duplicate is a finding that was “later found it was the same bug as another of our findings”, meaning one of Anthropic’s own findings matching another, explicitly not a match against an externally known issue. The jump from 2 to 1,654 came with the November 2 update, which Anthropic explains in its own words: “We found that many reported findings were duplicates of our other findings. We marked them duplicate in the ledger and stopped counting them.”

Then the number that forces this check in the first place, which is “fixed”. The ledger says 217. The dashboard says 629 patched upstream. The public rows flag 234 as patched, a third value. The About page doesn’t say why these differ, and “Fixed” has no entry in its glossary.

I do have a reading, and it’s mine rather than Anthropic’s. Every one of the 217 fixed rows is a revealed row — the headline says 235 revealed, 237 rows carry the revealed tier, and no row outside that tier is ever fixed. The snapshots point the same direction: revealed rows went 212, 235, 235 across Aug 26, Oct 2 and Nov 2 while ledger-fixed went 202, 218, 217, so fixed looks like something recorded after a reveal. The dashboard’s patched-upstream figure went 421, 516, 629 over the same three dates. If that reading is right, the two numbers measure different things, but only Anthropic can say whether it’s right, so treat it as a question for them rather than a finding.

Check 2: What is the denominator?
#

VulnCheck’s next sentence is “Just 217 of those have been marked fixed.” The trouble with putting 217 over 37,966 (or over 7,349, or over 5,087) is that, if the reading above holds, the numerator only exists on revealed rows — so each of those divisions uses stages the numerator can’t reach. I’m not computing a fix rate here for the same reason, because I’d be doing the thing I’m pointing at. And 629 can’t be put over a comparable denominator at all from the public rows.

Now a denominator that mostly doesn’t matter, as a contrast. VulnCheck says its 442 CVEs are “only about 0.5% of all CVEs” out of “93,450 CVEs published this year”. I counted NVD entries by published date for 2026-01-01 through 2026-11-09 and got 96,462 (22,140 + 37,096 + 37,226 across three windows). “Published this year” depends on the source and on the day you pull it, and the count keeps growing while you write the sentence that uses it: NVD’s count through November 1 is 92,417, so 93,450 sits between that and today’s figure and reads as an earlier pull. 442 over 93,450 is 0.47% and 442 over 96,462 is 0.46%, so “about 0.5%” holds either way and nothing here suggests VulnCheck made an error. (My NVD window also includes some CVE-2025-* ids that were published in 2026.)

The check is cheap in both directions: ask what the number is a share of, and whether the thing on top can exist in the thing underneath.

Check 3: Same population, same definition?
#

The 94.2% is the share of firm-reviewed findings confirmed as real, and Anthropic’s glossary says it includes “duplicates and ‘won’t fix’ findings”, so a real bug nobody intends to patch still counts as a true positive. VulnCheck sets it against curl: “Of the 34 reports published so far, 19 have been marked as not a vulnerability, 6 have been confirmed as low severity, and 9 are still awaiting assessment. That puts the false positive rate at 55.9% so far”, followed by “It’s a far cry from the 94.2% true positive rate Anthropic currently claims on its ledger.”

The count itself reproduces. curl’s GitHub issue tracker has 45 issues filed between November 5 and 9, and each one says it was forwarded from Anthropic’s OSS Scanner. 19 of 34 is 55.9%. The differences are in everything around it:

  • The label. curl’s vuln:NONE is described as “Vulnerability assessed and deemed insignificant”. VulnCheck’s wording is “not a vulnerability” and then “false positive”. On issue #23845, which was filed after the 34, Daniel Stenberg’s comment ends “A bug, not a vulnerability.”
  • The pipeline. Anthropic describes OSS Scanner output as “fully model-generated, without human review or triage”, and says “it is possible reports will be incorrect or invalid”. The 94.2% comes from candidates that external firms reviewed. Those are two different pipelines with two different reviewers.
  • The overlap. Each of the 45 carries an ANT-2026- id, and none of those ids appears in the public ledger rows, but the rows only show ids after a reveal. So I can’t say whether any of the 45 are in the ledger or inside the 94.2%, and I haven’t found anything public that says.
  • The size. One project, one maintainer, 9 of the 34 still undecided.

So two measurements, taken on different populations with different definitions, are being compared with each other, which is a bit like grading two different exams on one curve. I’d put it that way and stop there. It doesn’t show that the 94.2% is wrong, and it doesn’t show that the scanner’s output is mostly noise. What it does show — going by one maintainer’s labels — is that findings judged insignificant exist in that batch, which Anthropic warned about in advance.

Check 4: When was it counted?
#

The ledger doesn’t update continuously. It updates in batches, and Anthropic archives the snapshots: May 22, Aug 26, Oct 2, Nov 2. The “September 8” column in VulnCheck’s table matches the Aug 26 snapshot exactly, which was the page live on September 8, so the column is labeled correctly. It’s still a handy example of what a date label is worth when the data only moves four times in six months: it tells you which batch you were looking at, and not much more.

The November 2 update also lowered three of Anthropic’s own totals (total reported to maintainers, acknowledged by maintainer, and reported direct to maintainers). Disclosed fell from 6,157 on Oct 2 to 5,087, because by Anthropic’s account earlier counts had included internal duplicates. A comparison between any two dates in that stretch needs to know which counting rule each date used.

curl is the faster-moving example. The counts in the post, 19 / 6 / 9, match the issue tracker as of about 07:18 UTC on November 9, within seconds of the 34th issue being filed (it was created at 07:18:37 UTC). I don’t know when VulnCheck pulled its numbers. By 09:00 UTC that morning the same 34 issues read 22 / 6 / 6, and by 12:00 UTC they read 26 / 6 / 2, because the maintainer was working through them. Same 34 issues, three different tallies by noon. When I re-pulled on November 11 (around 15:00 UTC) one more label had changed and they read 27 / 5 / 2 — and 22 of the 45 issues were still open, so today’s tally wouldn’t settle anything in either direction. A rate computed while nine of 34 items are undecided is a progress report, and the post itself says “so far”.

VulnCheck’s strongest line on timing is worth having in its own words: “In two months, Anthropic added nearly 12,000 findings and fixed just 15 more.” The arithmetic is right (37,966 minus 26,153 is 11,813; 217 minus 202 is 15). What the check raises is which “fixed” series that 15 comes from, which takes us back to check 1.

Check 5: Who counts, and what can’t be checked?
#

442 is VulnCheck’s own count. It lives in a public GitHub tracker, Anthropic-Credited-CVEs, that VulnCheck builds by keyword-matching CVE records — names like Anthropic, partner firms and certain researchers, anywhere in the record — plus a manual audit. It includes collaboration credits, and the README describes entries as “possibly discovered by Project Glasswing”. Anthropic’s dashboard counts 230 CVE records and 503 GitHub Security Advisories, and publicly lists 102 CVE ids, since it lists them only after each disclosure window closes. All 102 of those ids are in VulnCheck’s file, which counts in VulnCheck’s favor. So 442 and 230 are different counting rules, and nothing about them is a contradiction.

The “exploited in the wild” figure rests on a list. VulnCheck says only 2 of the 442 “have been exploited in the wild, just 0.45%” (2 / 442 is 0.452%), measured against VulnCheck KEV. I checked CISA’s KEV catalog (version 2026.11.10, 1,779 entries) and neither CVE-2026-61500 nor CVE-2026-26980 is in it. That says nothing about whether they’re exploited, since a CVE can be exploited without ever landing on CISA’s list. I couldn’t query VulnCheck KEV myself because it needs a token, so for exploitation the honest summary is “VulnCheck KEV lists two”.

On the two CVEs: CVE-2026-61500, a rejetto HFS bug, was published 2026-07-13, and the CNA that assigned it is VulnCheck, the same company that runs the KEV counting it. That’s just where the number comes from; I’m not reading anything into it. Its credit names Zach Hanley of Horizon3.ai, “in collaboration with Claude and Anthropic Research”, and it isn’t among the 102 ids Anthropic lists (Anthropic counts 230, so that isn’t the same as being absent from Anthropic’s total). CVE-2026-26980 is a Ghost bug published 2026-02-20 and appears on Anthropic’s ledger as ANT-2026-69D8H6RP, status fixed, revealed 2026-06-03.

One hard edge on each side
#

Anthropic’s pages carry two “fixed” numbers, 217 on the ledger and 629 on the dashboard, and the ones I read don’t explain the gap; my structural reading above is only a reading. Anthropic also says, on the dashboard, that the true-positive count “should only be taken as one proxy for impact. Another, more reliable one is the number of patches created, though this is only a lagging indicator of progress, since patches take a long time to create.” It says there are “many additional bugs that we or our security partners have investigated and confirmed are real but that we have not yet reported to maintainers, due to capacity limitations”, and about the OSS Scanner, “we remain bottlenecked on our human capacity to validate these findings.”

VulnCheck’s post, going by the definitions above, compares figures that don’t share a definition: “made public” for ledger rows, 217 fixed against 442 CVE records, a ledger-wide rate against a single project’s labels. Its conclusion is its own, and it says it plainly: “I don’t know how anyone can read these numbers as anything but damning.” It also says, and this deserves equal billing: “AI is a valuable tool for finding vulnerabilities, and I’m not discounting that. But seven months in, the evidence still doesn’t match the launch narrative.”

I looked for a public response from Anthropic to the post and found none as of November 11, which says something about my search and nothing about whether one exists.

What the data leaves open
#

It shows false positives, or at least findings judged insignificant, in curl’s batch, going by one maintainer’s labels. It also shows real fixes exist, 217 on the ledger and 629 on the dashboard by Anthropic’s own count. Nobody in this exchange has measured what the scanners missed, and the net cost and benefit to the maintainers receiving the reports is unmeasured too. I can’t tell you from this data whether this is going well or badly.

What I couldn’t check: VulnCheck KEV membership, the HackerOne reports, the chart image in VulnCheck’s article, and the exact time of VulnCheck’s data pull. The one question I’d put to Anthropic is the 217 versus 629 one.

When one of these claims lands in your inbox
#

If you maintain software, the claim you’re likely to meet isn’t a dashboard but a single AI-generated report, and the five checks shrink to a short list:

  1. What is being counted? Reproduce it yourself. A report that can’t be turned into a failing test or a crash on your machine is still a hypothesis, which is the word Anthropic’s own glossary uses for candidates.
  2. What is the denominator? Check whether it’s a duplicate, of an issue you already track or of another report in the same batch. Anthropic’s own ledger had to merge 1,654 of its own findings after the fact.
  3. Same population, same definition? Ask which population the vendor’s true-positive rate was measured on, and what counts as true. A rate over firm-reviewed candidates that counts “won’t fix” as true says little about an unreviewed scan of your repository.
  4. When was it counted? Ask for the date and the rule: a share of what, counted when, under which definition. Labels on a batch of 34 moved from 19 / 6 / 9 to 27 / 5 / 2 in two days.
  5. Who counts? Ask who assigned the label and what list it was measured against, and note what you can’t check from your side of the report.

Related

GitHub Issues as an AI Agent Task Manager
·2087 words·10 mins
Ai Go Automation Github Devops
The ceiling is the same. The floor just moved.
·1536 words·8 mins
Devlog Ai Software-Engineering Economy Future-of-Work
Go 1.27: What Actually Matters
·1959 words·10 mins
Go Release Runtime Pprof Json
The Container Debugging Workflow: A Practical Guide
·2053 words·10 mins
Go Debugging Kubernetes Observability
GC Pauses and Latency: Where the Time Actually Hides
·2553 words·12 mins
Go Kubernetes Containers Garbage-Collection Latency