Who Reviews What the Machine Extracted: Staffing the Exception Queue With Contractors

Last updated: November 3, 2026

By Khurram Tehseen, FSA, Founder and Principal at Operating Alpha

Route only the fields the extractor tags as uncertain to contract reviewers and leave the rest of the document flow automated. Two mid-level financial analysts plus one data analyst who tunes the routing rules run about $134,400 for three months at KORE1 Contract Rate Index 2026-Q4 national modeled medians.

Here is the output. I made this one up, but it is shaped like the real ones.

Twenty-one fields went straight through. Two did not. The first is a table that broke across a page, so the machine can’t be sure the 3.50x belongs to the row it thinks it does. The second is a date written as a sentence, and the sentence depends on a fiscal year end that lives in some other document, probably the disclosure schedules.

Both are honest flags. Good. Now somebody has to open page 112. Then find the schedules.

Who? How fast? And what do they write down when they’re done? Those three questions decide whether the extraction project turns into a working process or a backlog with a dashboard on top. Most teams answer the first one with whoever is free, which is usually the credit analyst who was supposed to be freed up by the project, and who now spends the week after quarter-end opening PDFs to check numbers a machine read for them, on top of the compliance memos they were already late on. The work is still done by hand. It has just moved to a queue.

This piece is about staffing that queue while you still don’t know how accurate the extractor is. I’d call it a review desk. KORE1, whose site you’re reading, fills these seats through its contract staffing practice, so weigh what follows with that in mind. The method works the same if you staff it from your own bench. If you haven’t scored the extractor yet, start with the test in can AI read a credit agreement, because a desk is how you run it in production, not a substitute for running it once.

Which Fields Should a Human Review, and Which Should Not?

An extraction review desk is a small human-in-the-loop review team that settles the fields a document extraction system marks as uncertain, records why each one was uncertain, and checks a sample of the fields the system was sure about. It exists while accuracy is being measured, and it shrinks one field class at a time as the measurements come in.

Not every field earns a person. Most don’t. The test I use is plain. Does a wrong value change a decision somebody makes with money?

• Covenant levels and test dates. Yes, always, because a misread step-down moves a breach by a quarter.
• The pricing grid and the margin in force. Billing runs on it.
• Definitions that feed a covenant, Consolidated EBITDA above all, plus the add-back caps inside them.
• On a borrowing base certificate, eligibility exclusions, advance rates, and concentration limits. Availability gets funded off those, the same day sometimes.
• Cure rights and reporting deadlines, when tagged. A missed deadline is a default notice somebody has to decide whether to send.

Then the other list, which is longer than people expect. Party names, notice addresses, governing law, the agent’s wire instructions copied from a schedule nobody uses. A person can review those, but I wouldn’t spend a reviewer’s hour on them. Send them to the sample. Move on.

And arithmetic never goes to the desk. If a certificate’s aging doesn’t sum to the gross receivables line, that’s a validation rule firing, and a rule should catch it every time without a human retyping the column. I’ve put the borrowing base version of that split in contract analysts versus automation, and I won’t repeat it. Reviewers read. Formulas add.

How Do Uncertainty Tags Decide What Goes to the Review Desk?

The tag is the routing rule. Every field leaves the extractor as trusted, needing review, or unreadable, with a reason attached, and only the second and third reach a person. I wrote out what a tag has to carry in uncertainty tags and data quality. The short version is state, reason, owner, and what settles it.

Machine learning has a name for this. A reject option. Yonatan Geifman and Ran El-Yaniv framed it for deep networks in a 2017 paper on selective classification, where the system declines the inputs it is least sure of, trading coverage for a lower error rate on what it keeps. Their line for it is “Our method allows a user to set a desired risk level.” That is your desk’s job description, written by two computer scientists. You pick the error rate you can live with on each field class, and the desk absorbs whatever the machine declines to hold at that rate.

Two things go wrong here. Both are common. The first is routing on the model’s confidence score alone. A score is the model grading its own reading, and models grade themselves generously. Chuan Guo and coauthors put it bluntly in their 2017 ICML paper on calibration: “modern neural networks, unlike those from a decade ago, are poorly calibrated.” Language models have changed a lot since. Some are better calibrated. I still don’t trust a threshold of 0.9 on a field I haven’t measured, and you shouldn’t either. Route on the score plus the validation rules plus the field class, and expect the rules to do more of the work than the score.

The second is quieter. The desk only ever sees what the machine flagged, so it can’t see a confident wrong answer. Ever. A reviewer who clears 300 tags a week and finds most of them were real errors will come away feeling the extractor is well behaved and knows its limits, and that feeling, however sincere, says nothing at all about the 3.50x it didn’t flag on some other agreement. So every desk I set up has a second job, which is pulling a random sample of trusted fields each week and checking them against the page. That sample is your production measurement of silent errors. Without it, you’re measuring how good the machine is at doubting itself.

How Many Reviewers Do You Need While Accuracy Is Still Being Measured?

Size from your own exception rate, never from a ratio somebody quoted you. I haven’t seen a published documents-per-reviewer figure I’d trust for credit files, and the ones that circulate tend to come from invoice processing, where a field takes seconds.

Four inputs. Documents a month. Fields extracted per document. The share of fields tagged. Minutes to settle one tag, timed on your own people for two weeks with a stopwatch, not estimated in a meeting. Then add the audit sample. Here is a worked example with invented inputs, so you can see the arithmetic, not borrow the answer.

Illustrative inputs, not a benchmark. Replace every number with your own before sizing anything.

Call it 120 productive review hours per person per month once meetings, training, questions to the credit team, and the odd sick day come out of the roughly 173 hours a full-time contractor bills in an average month. That’s about one and a half people on average. Averages lie on a credit desk, though, because compliance certificates arrive in a wave after quarter-end and borrowing base certificates bunch around month-end. If 35% of the month lands in one week, that week carries about 62 hours. Two reviewers cover most of it. The third seat covers the rest.

Now the third seat. Somebody owns the routing rules, and I’d keep that person off the queue, since a reviewer who sets thresholds sets them to the queue they can clear by Friday. Human nature. Mine too. Give it to a data analyst, who reads what the reviewers recorded, moves reasons between routes, sets the sample rate, reruns last week’s documents through any changed rule, and sends two numbers each week to whoever signed off on the extraction project. Two people read pages, and one person reads what those two wrote down about the pages, which is the part most desks never staff.

Smaller book? One reviewer and a part-time analyst may be plenty. Run the arithmetic. Then run it again in a month.

What Should Reviewers Record So the Extractor Gets Better?

A review desk that only fixes values is an expensive way to type. The corrections have to come back out as data, or the extractor makes the same mistake on the next agreement and the desk never gets smaller.

Six fields per settled tag, one row. The value the reviewer landed on. Page and clause, so the next person doesn’t go hunting. A reason code from the list; free text turns into forty spellings of “late certificate” by spring. Minutes. Did it feed a decision, yes or no, meaning a covenant result or an availability number. And a sixth, which three desks I’ve seen left off.

Was the machine actually wrong? Often not.

Lots of tags turn out to sit on a good value. Maybe the 3.50x up top is fine and the machine just lost its place at the page break. Count it anyway. I want two percentages out of the desk. What share of tags held a wrong value? That’s flag precision, and when it’s low your reviewers spend their week confirming things that were right, which is a router problem, not a people problem. And what share of the sampled trusted fields were wrong? Silent-error rate. Above zero on a covenant field means the router is too relaxed. By field class. One page. Mondays.

Some things I’d expect the record to show in the first month, from what I’ve seen on loan books. Page-break tables generate a lot of tags with right values, which is a layout fix and an engineer’s problem. Dates written as sentences generate tags that are hard to settle from the document in hand, because the fiscal year end sits in another file; that’s a document-stacking fix upstream. Neither needs a smarter model. Neither needs more reviewers either. The record tells you which fix to fund, and that is most of what I mean by why OCR projects stall. The desk is where the evidence piles up, if somebody writes it down.

One habit I insist on. Reviewers never retrain prompts or edit rules themselves, however obvious the fix looks on a Thursday afternoon. They record. The rules analyst changes the rules, logs the change with a date and a reason, and the next week’s flag precision and silent-error numbers say whether it helped or just moved the problem to another field class.

When Does the Review Desk Shrink?

Field class by field class, on evidence, not on a calendar. Slowly at first.

Before the desk opens, write down what lets a field class leave it. My version is eight weeks without a silent error in the sample, a quarter-end inside those weeks, and flag precision that hasn’t drifted; a class that clears that moves to sample-only. Cure limits clear it fast. Pricing grids, slower. Definitions, slowest, since amendments restate defined terms years after closing. Covenant levels and availability inputs never leave the sample at all. Too expensive to miss, and the model under them changes whenever the vendor says so.

As classes drop out, hours fall, and the seats follow. The second reviewer usually goes first. Then the first reviewer moves from full weeks to the busy weeks around month-end and quarter-end, which is the reason to buy these seats hourly in the first place. The rules analyst stays longest. When the next document type comes on, say compliance certificates after credit agreements, the cycle starts over on new fields, and the person who already knows your reason codes is the one you want running it.

And if the numbers don’t move after a quarter? That happens. Then the desk has done its other job. It has told you, with a record, that the extractor isn’t improving on your documents, which is worth knowing before you renew the license.

Before You Staff the Desk

Realistically, what does a contract mid-level financial analyst cost for extraction review?

About $72 to $94 an hour nationally, median $82, per KORE1’s modeled 2026-Q4 index cell. At the median, three months of full-time review runs about $42,400 and six months about $84,800. Those are bill rates, so the contractor’s wages and benefits, the payroll taxes and insurance, and the staffing firm’s own recruiting and account management are all inside the hourly number you see. New York runs higher; Dallas-Fort Worth runs a little lower.

Can contractors review documents with borrower data?

Yes, if each reviewer has a named account, sees only the documents in the queue, and works in a logged workspace with exports turned off. A tagged field rarely needs a borrower’s personal identifiers to settle, so the queue can usually carry the page and nothing more. The fuller design is in contractor access to borrower data.

Who owns the decision on an exception?

A named employee owns any decision the value feeds; the reviewer owns the value and the record behind it. If a settled tag changes a covenant result or an availability number, it goes to your credit analyst or portfolio manager before anything moves. KORE1 partner Devin Hornick describes the contract arrangement this way: “Hourly, which we prefer. It’s really more of a staffing model to do the hourly billing and you manage the person.” You manage the person. That suits a desk working under your reason codes.

Why audit fields the machine was sure of?

Because a confident wrong value never reaches the queue, and a weekly random sample of trusted fields is how you measure, in production, how often that happens. One or two percent of trusted fields is usually enough to see a pattern within a month.

Do we need the data analyst seat, or can a reviewer tune the rules?

Keep them separate where you can, even on a small desk, even if it means the analyst seat is only twenty hours a week for the first quarter. A reviewer who sets the thresholds tends to set them to fit the queue. On a small book, a part-time analyst works, and if the reason codes and routing rules are already written, a junior data analyst can run the weekly numbers.

Count Last Month’s Tags

Pull every field your extractor tagged last month and, beside each, write who settled it and how many minutes it took. If you can’t fill in the second column, nobody owns the queue yet. That is the first seat to fill, before a second reviewer and before anyone argues about which vendor’s confidence threshold to use. If you can, divide the minutes by sixty, add a sample of trusted fields on top, and you have most of your sizing without a single vendor benchmark in it.

I’m happy to look at a tag log if you have one. Connect with me on LinkedIn.