Reference Checker Accuracy

Results

The sets come from HALLMARK, a public benchmark of real and fabricated references under the MIT licence.

How a reference counts as flagged

• A reference counts as flagged when its verdict is anything but Verified: Mismatch or Not found.
• Detection is the share of fabricated references that were flagged.
• The wrongly flagged share is the share of real references that were flagged.
• An entry is left out of the answered counts when its verdict is Unparseable.
• It is also left out when the run rejected it or failed on it, or when a registry still failed after 3 attempts.
• In both runs only Unparseable entries were left out. No entry was rejected, failed, or still failing after 3 attempts.

Where it is wrong

These come from reading the wrong rows of the run.

• A real title with a word or two changed still reads as the same work. This was 17 of the 28 fabrications missed in dev_public and 18 of the 19 fabrications missed in test_crossdomain.
• An author list with a changed or added name passes when the first author's family name still matches the record. This was 5 of the 28 fabrications missed in dev_public.
• A preprint cited with a conference and a changed year can pass when the registry holds only the preprint. This was 3 of the 28 fabrications missed in dev_public.
• A reference that names a publisher as its venue is read as a book, and a book is checked by title and author only. One dev_public entry passed with the year 2022 for a book of 1998.
• A real reference is flagged when its author list is cut short without et al. This was 4 of the 14 false flags in dev_public and 2 of the 6 false flags in test_crossdomain.
• A real reference is flagged when its DOI resolves to a record with another title. This was 6 of the 14 false flags in dev_public. In 5 of them the DOI printed in the data set belongs to a different paper. In 1 the record writes the title in LaTeX.
• In test_crossdomain, 1 of the 6 false flags was a bioRxiv work whose record holds a shorter title than the reference.
• An arXiv DOI cited with a conference name is flagged when the arXiv record names no venue. This was 2 of the 14 false flags in dev_public, and one of them is also among the 4 with a short author list.
• A real paper with no DOI is flagged when no source holds its title. This was 2 of the 14 false flags in dev_public and 1 of the 6 false flags in test_crossdomain.
• A real reference is flagged when its year is one off from the matched record. This was 1 of the 14 false flags in dev_public and 1 of the 6 false flags in test_crossdomain.
• An entry with no venue, or with a title that ends in a question mark, cannot be read. This was 98 entries in dev_public (83 without a venue, 15 with a question mark) and 3 entries in test_crossdomain (all with a question mark).
• Counting every unreadable entry as passed lowers detection to 0.840 in dev_public and 0.927 in test_crossdomain.

What these numbers do not say

• Each entry was written as an APA reference and checked on its own. A student's pasted list is messier than a benchmark entry.
• We ran the check ourselves. The benchmark's own scorer did not score it, and nobody outside CiteTools reviewed it.
• Every entry in both sets is typed as an article or a conference paper. Books, theses, reports and web pages are not measured.
• The databases change, so a new run can give other numbers. The date and the commit say which code was measured.
• The run gave each reference up to 120 seconds and up to 3 attempts when a registry failed. The site gives each reference 15 seconds, makes no second attempt, and shows Not checked when a lookup fails.
• The Crossref client of the checker changed after the measured commit. That change was not measured.
• A flagged reference needs a look at its source. The checker does not judge honesty.

Repeat the run

Each entry is written as Authors (Year). Title. Venue. https://doi.org/DOI, then sent to the checker alone. An entry with no DOI ends with its url field instead, when it has one.

1. Download dev_public.jsonl and test_crossdomain.jsonl from the HALLMARK tag v1.2.3 into an empty folder. The script refuses files whose SHA-256 sums differ from the tag's.
2. Check out main and run pnpm install. The benchmark script does not exist at the measured commit.
3. Compare the checker's files with the measured commit. The word changed means the run measures newer code, and its numbers replace these. git diff --quiet 11e9e02 HEAD -- server/services/citation/referenceVerifier.ts server/services/citation/referenceClaimParser.ts server/services/citation/venueAgreement.ts server/services/citation/workRegistry.ts server/services/citation/similarity.ts server/services/citation/wordMatch.ts server/services/citation/titleEnding.ts server/services/citation/crossref.ts server/services/citation/datacite.ts server/services/citation/dblp.ts server/services/citation/openalex.ts server/services/citation/pubmed.ts && echo unchanged || echo changed
4. Set OPENALEX_API_KEY_BENCH in .env to a key from another OpenAlex account than the site's. OpenAlex counts its daily budget per account.
5. Run the script with the two files. pnpm run benchmark:checker -- --run --dev <dev_public.jsonl> --cross <test_crossdomain.jsonl>
6. The run takes about 20 minutes and spends part of that account's daily OpenAlex budget. It rewrites shared/config/accuracy-benchmark.ts and keeps the known failures, which are written by hand.