Last updated: October 14, 2026
When an AI tool is not working in a credit or insurance operation, the cause is usually the data under it, not the model. Trace the outputs your team corrected, then run the tool on records you have checked by hand before you replace anything.
The override log had 214 rows.
Every one was a covenant flag an analyst had changed by hand during the quarter, with a free-text reason. Most said “wrong.” A few said “see email.” The COO had already drafted the note to the vendor. He wanted the monitoring tool gone before the next renewal date, and he wanted a second opinion so the note would carry more weight.
I asked for the last forty rows. And the source file behind each one.
It took two of us most of a Thursday. For every correction we opened the borrower’s compliance certificate, the administrator’s file, the loan system record, and whatever the tool had actually read, then wrote down in one plain line where the wrong number was born and who had touched it on the way. Six of the forty were the tool misreading a page that was fine.
Six.
The other thirty-four were wrong before the tool ever saw them.
He did not send the note.
I hear a version of this most weeks. The tool you bought is a car with no road under it. Nobody blames the road. Nobody bought the road.
Most of my work comes through finance and accounting operations consulting for direct lenders, ABL shops, and life and specialty insurers, so the examples here are credit examples. The method works on any tool that reads your records and hands back an answer.
What “the AI Tool Is Not Working” Usually Means
An AI tool that is not working is one whose outputs your team corrects often enough that nobody trusts it unsupervised. Most of those corrections trace to the input, which means a value that was wrong, late, contradicted, or never captured, rather than to the model reading it.
Pedantic? Maybe. It decides whether you spend the next six months on a replacement or on the records.
Practitioners have a name for the second kind of failure. In a 2021 study presented at CHI, Sambasivan and colleagues at Google Research interviewed 53 AI practitioners working on high-stakes systems and found data cascades, compounding downstream failures that start with a data issue nobody valued at the time, at 92% prevalence. The title of the paper is a quote from one of them. Everyone wants to do the model work, not the data work.
Finance is not exempt. When the Bank of England and the FCA surveyed 118 UK financial firms in 2024, 75% said they were already using AI, and four of the top five risks those firms perceived were data risks, covering privacy and protection, data quality, data security, and bias and representativeness.
So the firms know. They still blame the car. Mostly.
Trace the Last Forty Corrections
Here is the Thursday exercise. You can run it without me.
Pull the most recent forty outputs your team overrode, corrected, or quietly re-keyed into a spreadsheet somewhere else, including the ones nobody logged because the analyst simply fixed the number in the pack and moved on. Forty is enough to see a pattern and small enough to finish in a day. For each one, find the right answer and the input the tool used.
Then one question. Was the input correct when the tool received it?
If yes, the tool misread a good record. Vendor’s problem.
If no, keep going until you find where the wrong value entered. That location is the useful part.
Below is a typical first trace. It is a composite, built from a handful of firms, and it lands about where that Thursday did.
| Where the wrong value came from | Corrections (of 40) | Whose problem | What fixes it |
|---|---|---|---|
| Tool misread a clean page or field | 6 | Vendor | A ticket with the six examples attached |
| Two sources disagreed and no rule said which one wins | 11 | Yours | A written precedence rule, per field |
| Value was wrong at the source, in the certificate or the administrator file | 9 | Yours, then the borrower’s | A validation check that catches it on arrival |
| Value arrived late, so the tool used last quarter’s | 8 | Yours | A freshness tag the tool can see |
| The term only existed in an amendment PDF nobody keyed | 6 | Yours | Reading the amendments into fields, once |
Look at the second column before the fourth. Thirty-four of forty.
The largest single bucket holds two good numbers and no decision about which one counts. A model cannot make that decision for you. You would not want it to.
Not every firm lands there. Others come out closer to even. I have seen a trace where the vendor owned more than half. That is the point of running it.
You find out which kind of firm you are instead of arguing about it on a renewal call.
Every bucket except the first is road. Each one has a shape, and I have written about most of them separately, from the same borrower appearing five ways in five systems to marking a number your data cannot support. If you want the whole surface in one place, that is what a data foundation actually means in a credit fund.
Then Run It Twice
The trace tells you where past errors came from. It does not tell you how good the tool is. For that you need a second test. A cheap one.
Take twenty-five records the tool handles, such as borrowers, certificates, or claim files. Have the person who would do the work check every input on those twenty-five against the source documents by hand, fix what is wrong, and write down the right answer for each. That is the clean set.
Now run the tool twice on the same twenty-five records. Once on the clean set. Once on the inputs exactly as they arrived last month, warts included.
Score both. Same answer key.
Three readings come out of it.
The tool scores well on the clean inputs and badly on the live ones, often by twenty or thirty points on the same records, which is a gap no amount of vendor tuning was ever going to close. That gap is the road, measured in your own units. It is the most common result I see. By a distance.
Fix the inputs and the tool you already bought starts earning.
It scores badly on both? Then it really is the car. Put the clean-set results in front of the vendor in writing, because now you have evidence instead of a feeling, and if they cannot close the gap you have a defensible reason to switch.
Sometimes both runs score well and the team still overrides it. Rarer. Stranger.
It usually means the tool is answering a question nobody on the desk is actually asking, and the overrides are people doing the job the old way out of habit or out of a reasonable suspicion that nobody will back them if the tool is wrong. Read the record of who made the final call on a dozen of those cases and the reason tends to show up.
Twenty-five records is a small sample. Small is fine. I have run this on credit agreements, borrowing base certificates, and medical files, and twenty-five has been enough to show the direction of the gap. If you want the more formal version for documents, I wrote up a field-by-field scoring of AI on credit agreements.
Why the Car Gets Blamed
A purchase has a date, an approver, and an invoice. The road has none of those.
Nobody signs off on the decision about which of two loan tapes ties, or on reading forty amendments into fields, or on telling a borrower that their certificate keeps arriving with last quarter’s aging attached. That work is still done by hand, usually by whoever is closest to it, and usually without anyone noticing it is the actual product.
The pilot itself has the same blind spot, which is why I put a named financial owner on every AI pilot before it starts.
So when the outputs go wrong, the only visible object in the room is the tool. It has a logo. It has a renewal date. It even has a person whose job is to answer your emails about it, which is more than the road ever had.
Replacing it feels like action. I get the appeal.
A second tool fed the same inputs produces the same thirty-four corrections with a different interface, and now you have spent a year and a migration learning that.
The platform-level version of this, where the system is live and almost nobody uses it, has its own symptoms and I covered it in why a platform you bought sits unused. Running the trace before a purchase rather than after is the point of the AI readiness scorecard, which walks one decision through seven links before any vendor demo. For a pilot whose contract is not signed yet, the gates to clear before an AI contract put the clean-set test in writing ahead of the invoice.
Who Builds the Road
Road work splits into two kinds of hours. Rarely one person.
Precedence rules, validation checks, and freshness tags get built by data engineers who can read a credit agreement well enough to know which field matters. The ongoing part, clearing the exceptions those checks raise, belongs to an analyst who knows the asset class and has protected time to do it. Neither ran the procurement.
Usually neither seat exists yet, which is why the corrections ended up in a free-text column.
Match the contract to the shape of the work. A build of a few months suits contract data engineers who leave a working pipeline behind. The exception queue is permanent and wants a permanent owner. If you need either seat, KORE1’s data engineer staffing practice averages 17 days to fill IT roles, which matters because the trace usually lands a quarter before a renewal date.
Sequence matters more than headcount. Pick the bucket with the most corrections, fix that one properly, and rerun the twenty-five.
Scorecard now, ML when your data can support it. Same logic for the road. You do not need a perfect foundation to see the gap close on one bucket.
Questions I Get Once the Vendor Has Been Blamed
Our fund administrator says its platform handles the data. Isn’t that their job?
Their platform automates their work, and your decisions sit downstream of it.
An administrator is accountable for the books it keeps. It is not accountable for which of its figures your covenant test should prefer when your own loan system says something different. That precedence rule is yours. Ask them for the rule they apply when two sources disagree, and you will usually find there isn’t one written down.
Are forty corrections really enough to trace?
Forty is plenty for the first pass.
The buckets settle fast. By the twentieth correction you can usually guess the shape, and the last twenty confirm it. If your team only corrected a handful of outputs all quarter, you may not have a problem. That is rare.
What if the clean run fails too?
Then you have a car problem, documented, and the vendor should hear about it in writing.
Send the twenty-five records, the right answers, and both scores. A serious vendor engages within a week.
One that answers with a training offer has told you how the renewal conversation will go.
Should we start looking at replacement tools in parallel?
Not until the two runs are scored.
Demos run on the vendor’s clean data. Your current tool passes that too. Probably.
If you must look, bring your twenty-five live records to the demo and ask them to run those.
Who should run the trace, us or the vendor?
You, with the person who would do the work sitting next to whoever is typing.
The vendor can help pull outputs. It cannot tell you which of your sources is authoritative, because that is a governance decision your firm has to make and own, and it has an obvious interest in which way the answer falls. An outside reviewer is useful mostly for keeping the reasons honest when a correction turns out to be somebody’s own spreadsheet.
How long does building the road take?
Weeks for one decision, longer for a portfolio.
A precedence rule for one covenant test, plus validation on the three fields that feed it, is a few weeks of work with an owner who has the hours. The full surface for a loan book takes quarters, and nobody should start there. Start with the largest bucket from the trace and measure the second run again when it is done.
Pull Forty Corrections This Week
Ask whoever owns the tool for the last forty outputs your team changed, with the source file behind each. Block a day. Sort them into the five rows above before anyone writes to the vendor.
Then send me the split. Find me on LinkedIn and say how many of the forty were the tool. If the bucket that won needs a person to own it, KORE1 can staff that seat.
