Measuring AI extraction errors

Specify the extraction job before judging the model

An extraction system turns source material into proposed structured fields. Its quality depends on the exact job. Copying a date explicitly printed in a document is different from deciding which of three dates controls a contractual obligation. Reading a reported rent is different from confirming collected income. Write a task specification that identifies the permitted documents, fields, source locations, acceptable transformations, and excluded judgments. A useful boundary is that the system may propose a value and cite the source passage, while a responsible person verifies consequential meaning before it influences a transaction or communication.

For a hypothetical acquisition workflow, choose five fields from an intake packet: property address, reported asking price, reported monthly rent, stated roof age, and requested callback date. Label these as reported information, not verified facts about the property or the person’s legal authority. Define how ranges, conflicting statements, illegible scans, and absent values should be represented. If a note says rent may be about twelve hundred, an extractor should not silently output a confirmed annual income of $14,400. That introduces both an interpretation and a calculation beyond the original statement.

Build a reference set that can survive disagreement

Select documents representing the workflow you expect to operate: different layouts, scan quality, handwritten additions, multiple properties, amendments, and genuinely missing information. Retain a separate test set that is not used to tune prompts or examples. If all documents from one property appear in both tuning and testing, near duplicates can make results look stronger than performance on new work. Keep original source references and version identifiers. Use synthetic or appropriately deidentified material where practical, and handle any necessary personal information under the same access and retention controls used for live operations.

Have qualified reviewers label the reference values under written rules. For ambiguous cases, preserve the disagreement and its resolution instead of forcing a convenient answer. The reference can be an accepted value, a range, absent, unreadable, conflicting, or outside scope. Reviewers should cite the relevant page or passage. A callback request on page two that is changed on page four needs a precedence rule or escalation label. Reserve some independently reviewed examples for checking the labelers themselves. An error score against unreliable reference labels measures agreement with those labels, not truth.

Choose the unit and matching rule explicitly

Use a field instance as the unit when the decision concerns individual extracted values. One packet containing five required fields creates five field instances, even if the model emits a single JSON object. Define exact matching for identifiers and an allowed normalization for presentation differences. A date can use one standardized format if the source is unambiguous. A currency value can remove punctuation while preserving currency and units. Do not normalize away meaningful differences such as a missing unit number, monthly versus annual rent, a negative sign, or uncertainty in the source.

State how each error category is counted. An incorrect value is a wrong emitted answer. An unsupported value is emitted when the source does not support a value under the task rules. An abstention is an explicit refusal to supply a value, routed to review. A missing output caused by invalid JSON is a processing failure, which should not disappear from the denominator. Score source citation correctness separately: a correct number paired with the wrong page can still obstruct verification. Keep the raw output and the scored interpretation so another evaluator can reproduce the result.

Calculate coverage and error without hiding abstentions

Consider a hypothetical test with 100 packets and five required fields per packet: 500 field instances. The system emits 440 values and explicitly abstains on 60. Review finds 396 correct emitted values and 44 wrong or unsupported emitted values. Coverage is 440 divided by 500, or 88 percent. Emitted-value error rate is 44 divided by 440, or 10 percent. Correct emitted yield across all required instances is 396 divided by 500, or 79.2 percent. These measures answer different questions and should be displayed together with the raw counts.

The abstention rate is 60 divided by 500, or 12 percent. Do not count those sixty as correct extractions merely because the system avoided guessing. Also do not treat them as wrong emitted values in the ten-percent calculation. If the reference says some fields were genuinely absent, score whether the system correctly recognized absence under a separate stated policy. In this example, all 500 instances require either an extracted answer or human resolution, so abstention represents review work. A system with zero emitted errors because it abstains on everything has no automatic extraction yield.

Use confusion matrices only for a defined binary question

A confusion matrix is useful when the task has a defined positive and negative class. For example, ask whether a packet contains an explicit request to stop further contact. In a hypothetical set of 200 packets, human review identifies twenty positives and 180 negatives. The system flags eighteen packets: sixteen true positives and two false positives. It misses four positives and correctly leaves 178 negatives unflagged. Precision is 16 divided by 18, or about 88.9 percent. Recall is 16 divided by 20, or 80 percent. The false-positive rate is 2 divided by 180, or about 1.1 percent.

Accuracy is 194 divided by 200, or 97 percent, because sixteen positives and 178 negatives were classified correctly. That attractive accuracy does not erase four missed stop requests. This matrix assumes a binary decision on every packet and no abstentions. If the system abstains on some packets, show their count and reference classes separately, report coverage, and state whether classification metrics apply only to the decided subset. Never remove hard cases silently. This example teaches measurement; a real stopping-preference control should also have deterministic intake checks and human review rather than depending on this classifier alone.

Inspect error severity and concentration

Break results down by field and document type. Forty correct addresses do not compensate for one wrong contract deadline when the deadline controls an important decision. Create a severity rubric tied to workflow consequences, such as harmless formatting, operational delay, financial-model distortion, or a potential unauthorized action. Record both unweighted errors and severity categories. Avoid hiding the raw counts inside a single weighted score. A weighted score depends on subjective weights and can improve simply because the test mix contains more easy fields, even while the most consequential field gets worse.

In the hypothetical five-field test, suppose thirty of the forty-four emitted errors concern roof age and eight concern monthly rent. Examine whether roof statements use ranges or whether the model treats renovation dates as installation dates. The remaining six errors still need individual review, especially if they affect identity or contact instructions. Compare results within the same slices after a change. A new prompt can reduce easy formatting mistakes while increasing unsupported assertions on poor scans. Keep a small error catalog with source evidence, failure category, expected behavior, and the proposed safeguard.

Evaluate the workflow cost, not only the model score

Measure review time and the number of corrections under a realistic process. In the hypothetical test, the sixty abstentions require 45 seconds each, or forty-five minutes of review. If a reviewer also checks all 440 emitted values at ten seconds each, that adds about seventy-three minutes and twenty seconds. Total field review is approximately 118 minutes and twenty seconds, excluding packet opening, exception discussion, and rework. This is a workload estimate, not proof of savings. Compare it with a separately timed manual workflow on comparable packets, including the same quality requirements.

Track whether reviewers actually find the errors. Human approval is not automatically effective if the interface hides the source, preselects an answer, or encourages rapid acceptance. Present the proposed value alongside its source location and uncertainty status. Require deliberate verification for consequential fields, and provide an easy correction or abstention path. Sample reviewed outputs independently to detect rubber-stamping. The relevant production result is the residual error after the whole process, together with time and missed cases. A faster model that leaves reviewers overconfident can make the complete workflow worse.

Set approval gates from consequences

Before deployment, decide which fields can populate a draft, which require review before entering an underwriting model, and which can never authorize an external action by themselves. A reported asking price may be safe to place in an unreviewed worksheet if visibly labeled. Bank instructions, signing authority, contractual dates, property identity, and contact restrictions demand stronger controls. Do not let an extracted amount generate an offer or a payment instruction automatically. Separate extraction permissions from execution permissions, and preserve the identity and time of the person approving any consequential transition.

NIST’s AI RMF Core treats measurement and management as part of ongoing risk work and calls for documented testing and evaluation suited to context. That supports a disciplined lifecycle rather than a universal passing score. Choose release criteria before observing the final test results, with stricter conditions for higher-consequence uses. A tiny test with no severe errors does not establish that the true severe-error probability is zero. When evidence is too limited, retain human verification, limit the pilot, or exclude the difficult field. Do not lower the standard merely because automation is already purchased.

Protect data and resist instructions inside documents

Review what information reaches the extraction provider, where it is processed, how it is retained, and who can access it. Remove irrelevant identifiers and restrict access to test packets and outputs. Treat generated logs as potentially sensitive because they can contain source text. Establish appropriate vendor and security review before using real records, and do not upload confidential documents into an unapproved demonstration account. A deidentified document can still expose someone through a distinctive address or narrative, so assess the complete packet rather than deleting one obvious identifier and assuming the remaining material is anonymous.

Documents are untrusted input. A source file can contain text that tells a model to ignore instructions, change fields, or disclose other information. Keep the extractor narrowly scoped, isolate it from outbound tools and unrelated records, validate its output against an explicit schema, and route unexpected content for review. A quoted request inside a document is evidence to interpret, not an authorization to execute an action. Test adversarial examples alongside ordinary poor scans. Privacy and human-review controls belong in the measurement plan because a numerically accurate extractor can still be operationally unsafe.

Retest changes and monitor live exceptions

Version the model identifier, prompt, preprocessing, schema, matching rules, and reference set together. Changing optical character recognition can alter performance even if the language model stays the same. Run a fixed regression set after a material change, then test additional examples representing the new conditions. Keep a holdout set independent of prompt tuning. If repeated tuning has effectively taught the team the holdout answers, create a fresh evaluation set. Record release decisions and known limitations so a future operator can understand why a field remained subject to mandatory review.

In production, monitor processing failures, abstentions, corrections, reviewer time, and incidents by slice. Investigate abrupt changes and sample accepted outputs, including those nobody corrected. A falling correction rate may mean better extraction or less attentive review. Maintain a pause control and a manual fallback for consequential records. Before expanding the pilot, confirm that denominators reconcile, severe errors have been investigated, all review obligations are staffed, and data handling is approved. Publish results as bounded evidence about the tested workflow and dataset. They are not a guarantee that every new document will be understood correctly.

• Freeze field definitions and matching rules before the test.
• Keep a representative reference set and independent holdout.
• Report counts, coverage, emitted errors, and abstentions together.
• Inspect consequential errors and reviewer effectiveness.
• Prevent extracted data from directly authorizing outbound actions.
• Retest material changes and retain a manual fallback.