Last updated: October 12, 2026
Kill any AI pilot that has no financial owner, cannot name the credit decision it changes, or has never been scored against a result that already happened. Most AI implementation failure in private credit is a pilot that nobody had the standing to stop.
The list arrived as an attachment called pilots_current.xlsx. Eleven rows.
A COO at a direct lending manager wanted to know which ones to stop. There were columns for vendor, start month, monthly cost, and sponsor. Four sponsors said “Ops.” Two named a portfolio analyst who had left in the spring. One said “IC.”
Committees do not cancel things.
There was no column for the decision each pilot was supposed to change, so I added one and we filled it in together on a call that ran about forty minutes. Seven of the eleven cells stayed blank.
That blank column is what this piece is about. Over six years I shipped 34 data and AI solutions into production for a specialty finance investor. The shipped count is honestly the less useful number. What decided the outcome was how fast we stopped everything else.
Most of this work reaches me through finance and operations consulting engagements with private credit managers, so the examples below are credit examples. The review assumes the pilots already exist. They always do.
Still weighing whether to sign the first contract? Then the four gates to clear before signing an AI vendor cover the earlier conversation.
What AI Implementation Failure Looks Like in a Credit Shop
AI implementation failure is a pilot that consumes budget and attention without changing a decision anybody signs. In private credit that usually means a model that reads credit agreements, borrowing base certificates, or monitoring packages, produces output, and leaves the covenant test, the availability figure, and the watchlist exactly where they were.
The research version is blunter. RAND’s 2024 study, The Root Causes of Failure for Artificial Intelligence Projects, interviewed 65 data scientists and engineers with at least five years of experience each and repeats the estimate that over 80% of AI projects fail, about double the failure rate of technology projects without AI. Its most common root cause was stakeholders misunderstanding, or miscommunicating, what problem the AI was supposed to solve.
That is the blank column, more or less word for word.
The pilot set at a mid-sized direct lender looks about the same wherever I go. Somebody is extracting covenant terms from credit agreements and their amendments. Somebody else has a borrowing base certificate reader.
There is a copilot that drafts investment committee memos, and a chat window over the loan document library that IT stood up because it was easy. Often an early-warning model on monthly borrower financials. Lately, a drafter for LP due diligence questionnaires. Every firm has that one now.
Six pilots. Maybe two of them touch a number someone signs.
Four Questions, Asked in This Order
I ask the same four questions about every row, and I stop at the first no. The order matters because each question is cheaper to answer than the one after it, and the first two retire most pilots before anyone in the room has to argue about model accuracy.
Who would notice if it stopped on Monday?
Ask for the financial owner. That means the person whose budget pays for the pilot and whose operating number is supposed to move because of it, which at most firms is a different person from the sponsor, because the sponsor is whoever liked the demo.
Where does the cost sit? If it lives in an innovation line or a general IT budget, nobody’s number moves when it works and nobody’s number moves when it doesn’t. So try a test. Ask the named owner to defend the monthly cost to the CFO in one sentence.
Silence is an answer.
Which decision does it change?
Name it as a verb with a signature attached. Approve availability on the March borrowing base. Set the Q2 covenant test result for a borrower. Move a name onto the watchlist.
If the output is a summary or a report, ask the person who would do the work what they did differently last month because of it. Not the sponsor. The analyst. Their answer is usually “I read it, then I checked it against the PDF,” which means the pilot added a step.
Has it been scored on something that already happened?
This is the replay. Run the pilot on last quarter’s certificates or last year’s amended agreements, then compare its output field by field against what the team actually booked and signed. Read the record. Not the demo.
I set out the test I run before trusting AI on a credit agreement separately, and the same logic applies to any document a pilot claims to read. A pilot that has only ever run on live data has never been graded, because nobody knows the right answer for this month yet.
What does it write, and where?
Last question, and the one that separates a pilot from a product. If the output lands in an email, a PDF, or a separate screen that someone then re-keys into the loan system, the work is still done by hand and the pilot cannot become production without a second project nobody scoped. I wrote about what happens when a platform asks for inputs the firm cannot produce, and a pilot with no write path is the same failure at a smaller price.
Here is how those four questions sorted a composite register, assembled from several portfolio reviews rather than one firm’s list.
| Pilot | Financial owner | Decision it changes | Scored on a past period? | Verdict |
|---|---|---|---|---|
| Covenant extraction from credit agreements | Head of portfolio operations | Quarterly covenant test results | Yes, 20 agreements against booked tests | Fund to production |
| Borrowing base certificate reader | None named | Monthly availability | No | Assign an owner, then replay three months |
| IC memo drafting copilot | Investment team, informally | None; analysts rewrite the draft | No | Kill |
| Chat over the loan document library | IT | None anyone could name | No | Kill, keep the document index |
| Early-warning model on monthly financials | Chief risk officer | Watchlist additions | Two years of history, nine credit events | Replace with a scorecard for now |
| LP due diligence questionnaire drafter | Head of investor relations | Which answer goes to an LP | Partly, against last year’s questionnaire | Keep, and keep it small |
Look at the early-warning row. Nine credit events in two years is not enough history to train anything you would defend to a committee, and the CRO knew it before the vendor did.
Scorecard now, ML when your data can support it. Nobody had to be wrong for that pilot to stop. The data just was not there yet.
The memo copilot was harder to kill than it looked. People liked it. Liking is not a decision.
The Pilot With No Financial Owner
This is the one that survives every review, and the reason is boring. The cost was approved once, inside an annual innovation budget, at a monthly figure small enough that renewing it never needed a meeting. Status updates are demos. Demos go well.
Cancelling it would require someone to stand up and say they had been paying for something they could not describe, and there is no budget line that rewards that.
Gartner’s June 2025 forecast that over 40% of agentic AI projects will be canceled by the end of 2027 named escalating costs, unclear business value, and inadequate risk controls as the reasons. It also called out “agent washing,” older chatbots and automation relabeled as agents. Three of the eleven rows on that COO’s sheet called themselves agents. One was a scheduled script.
Unclear business value is a polite phrase. In my experience it nearly always means nobody owns the number.
Sometimes the fix is to assign one.
The certificate reader in the register had a real decision behind it, monthly availability across about forty ABL borrowers, and nobody had been asked to own it because the vendor had been talking to IT.
Then the head of portfolio operations took it. The questions changed overnight.
Which certificates? Which fields? Whose sign-off on a mismatch? The pilot got smaller and more useful in the same week.
If nobody will take it, that is your answer too.
What You Keep When You Kill One
A killed pilot is rarely a total loss, and treating it as one is how firms end up paying for the same groundwork twice. Before the contract lapses, pull out four things.
- The schema the pilot was built against, meaning the list of fields, their definitions, and the rules someone argued about for a month. That argument is the expensive part.
- Any ground truth. If an analyst hand-checked twenty agreements to grade the output, that labeled set is worth more than the model that failed it.
- Validation rules.
- The exception list, every case the pilot could not handle, which is a map of where your data is weakest and almost always the honest version of the project plan nobody wrote.
The first thing I build is never the dashboard, and a stopped pilot often leaves behind exactly the pieces I would have built first. At one firm the document index from a dead chat project became the intake for the covenant extraction that replaced it.
Nobody planned that. It was just sitting there. I describe the order those pieces belong in, schema before extraction and validation before either, in my document-to-decision build sequence.
The money side of keep versus kill is its own exercise, and the line items that appear once a pilot leaves its proof of concept are broken out in what AI implementation actually costs after the POC. Do that arithmetic for the survivors, not the casualties.
Running the Review
One hour. The register on a screen, every financial owner in the room, and at least one analyst who would inherit the pilot if it went live.
No vendors. They will want to come. Say no.
Go row by row and write one sentence per verdict. For a kill, the sentence says what the pilot was meant to change and why it did not. For a survivor, it says who owns it, what it is scored against, and the date of the next replay.
The NIST AI Risk Management Framework already expects this kind of decision, in fairly plain language. Under NIST AI 100-1, MANAGE 1.1 calls for a determination of whether a system achieves its intended purposes and whether its development or deployment should proceed, and MANAGE 2.4 asks that responsibilities be assigned to supersede, disengage, or deactivate systems whose performance is inconsistent with intended use. The governance process exists. Most firms just never run the stop half of it.
Keep the sentences. An operational due diligence team asking about AI will get more out of four stopped pilots with written reasons than out of eleven live ones without owners, and your reporting pack is fundraising infrastructure whether or not anyone treats it that way.
The survivors still need people. Where one has to be built rather than bought, the build vs buy decision for pilots headed to production is the next call. Where the gap is an engineer to wire the write path, most managers I work with bring one in on a contract engagement for the build and decide on a permanent seat once the thing has run through two reporting cycles.
Questions From the Portfolio Review
The numbers look good. Should we still stop it?
Good numbers on a pilot nobody owns usually describe a good demo.
Ask where the numbers came from. If they were produced by the vendor on a sample the vendor chose, they describe the sample. Run it on a period you already closed and see what survives.
Who should the financial owner actually be?
The person whose signature sits on the decision the pilot changes.
For a borrowing base reader that is whoever approves availability, usually the head of portfolio operations. For an early-warning model it is the CRO or whoever runs the watchlist meeting. It is rarely the CTO, and it is never an innovation committee, because neither one signs anything the pilot touches.
How long should a pilot run before it faces this review?
Ninety days, or one full reporting cycle, whichever is longer.
A monthly certificate process gives you three cycles in that window, which is enough to see whether the exceptions are shrinking. A quarterly covenant process gives you one live quarter, so pair it with a replay of the previous two. Running longer than that without a review is how a pilot quietly becomes a subscription.
We have a year left on the contract. Is stopping it still worth it?
Usually, because the contract is spent either way and your analysts’ hours are not.
The hidden cost of a weak pilot is the checking. Every output somebody has to verify against the source document is time taken from the work it was supposed to save. Stop using it, keep the schema and the ground truth, and let the term run out.
What do we tell LPs or the board about the pilots we stopped?
One sentence per pilot, covering what it was meant to change and why you stopped it.
That answer reads as control. It is control. The alternative, a long list of live pilots with no owners and no scores, reads as a firm that buys things and hopes. Investors who run operational due diligence can tell the difference quickly.
Is most AI implementation failure really a data problem?
About half the time, and the other half is a problem nobody wrote down.
RAND lists insufficient data among its five root causes, right after a misunderstood problem. In credit the two travel together. A pilot aimed at a vague problem never finds out its data was wrong, because nobody defined what right would look like. Write the decision first and the data question gets much easier to answer, sometimes in an afternoon.
Fill In Two Columns Before Friday
Open your pilot list. Add a column for the decision each one changes and another for its financial owner, and fill both in without asking the vendors. Whatever is still blank on Friday goes first on the agenda.
If the review leaves you with two survivors and nobody to run them, talk to KORE1 about staffing the build. If you want a second read on your own register, connect with me on LinkedIn.
