Accuracy: what we measured and how
On a held-out synthetic test set of 317 transactions from two sample businesses, LedgerReady extracted 100.0% of transactions exactly, reconciled 10 of 10 statements, categorised 89.6% correctly first time and flagged 14.2% for confirmation; 2.2% were wrong without a flag.
| Test set | Transactions | Extracted exactly | Reconciled | Category right | Flagged for review | Wrong, unflagged | Receipts matched |
|---|---|---|---|---|---|---|---|
| Tuning set, rules only | 340 | 100.0% | 10/10 | 95.0% | 9.4% (32) | 0.0% (0) | 15/15 |
| Held-out set, rules + model | 317 | 100.0% | 10/10 | 89.6% | 14.2% (45) | 2.2% (7) | 15/15 |
How we tested
- We wrote the categorisation rules first, before any test data existed.
- We generated a tuning set: synthetic statements for two fictional businesses (a design studio and a candle seller) in three statement layouts plus a CSV export, and 15 receipt photos with blur, tilt and noise. Each merchant's correct category was assigned by hand. We fixed the rules once based on its errors, then froze them.
- After freezing, we wrote a held-out set: two new fictional businesses (a tutoring company and a ceramics seller) with about 70 merchants the rules had never seen. We did not change the rules after seeing its results.
- "Extracted exactly" means date, amount and description all match the answer key. "Reconciled" means opening balance + extracted transactions = closing balance to the cent.
Limits, plainly
- All test data is synthetic. Real statements come in many more layouts than the three we tested; a new layout can need a parser adjustment, which we do as part of the job.
- Scanned or photographed statements were not tested; they are slower and more lines get flagged.
- Real crumpled thermal receipts read worse than our synthetic photos.
- Some things no software can know (who a cheque was paid to, whether an Amazon order was stock or personal). We always flag those rather than guess silently.
- A known rule bug from the held-out run (a phone bill paid by "AUTOPAY" filed as a card payment) is not fixed yet. When it is, we will measure the new rules on a fresh test set and post the result on the updates page.
What the numbers mean for your books
- Every flagged row needs a human review. On the held-out set, 45 of 317 rows (14.2%) were put on the "please confirm" list. You (or your bookkeeper or accountant) must check each of those before importing; we do not guess them for you.
- Unflagged rows can still be wrong. 7 of 317 rows (2.2%) were filed in the wrong category with no flag. A quick skim of the whole ledger before import is worth it.
- Rules alone were weaker on new merchants. On the same held-out set with rules only (no model stage), 222 of 317 categories were right (70.0%), 108 rows were flagged and 3 were silent errors. The model stage was added after we saw that result, with a generic prompt that was not tuned row by row.
- Extraction and reconciliation (317 of 317 rows, 10 of 10 statements) were measured on text-based synthetic PDFs in three layouts and one CSV format only.
The test generator, answer keys and scoring script are kept with our records; the held-out results above were produced on 2026-10-14.
Last updated · What changed