Can AI Read a Credit Agreement Reliably? Here Is the Test I Run

Last updated: September 17, 2026

By Khurram Tehseen, FSA, Founder and Principal at Operating Alpha

AI reads a credit agreement reliably only after it has been scored against agreements your team already closed, on the fields that set covenant tests and pricing, with silent errors counted apart from flagged ones. Contract data extraction breaks on this document in three predictable places. Amendments, definitions that point somewhere else, and tables. The test below is built to find all three before a credit decision does.

The extraction came back in under a minute. Forty-one fields from a 187-page credit agreement. Forty matched the file.

The margin said LIBOR plus 4.75%.

That loan had not priced off LIBOR for more than two years. U.S. dollar LIBOR panels ended after June 30, 2023, and the Federal Reserve’s final rule under the LIBOR Act named SOFR-based replacements for contracts that had no workable fallback. This loan never needed it. Amendment No. 3 had already moved it to Term SOFR with a credit spread adjustment on top, and Amendment No. 3 was a fourteen-page PDF saved in a different folder, under a different naming convention, by an associate who left the firm in 2024.

Nothing about the extraction was wrong except the answer.

Credit teams ask me some version of this question every few weeks. Can AI read our agreements? My answer doesn’t change. I don’t know yet, and neither do you, until it has been scored on agreements you have already closed. That scoring is the equivalence step, the last of the six in the document-to-decision method, and a credit agreement needs its own version of it, because the document is unusually good at hiding where the answer lives.

You should know where you’re reading this. KORE1 is in the placement business, and its data science and data engineering recruiting desk fills the engineering seat a test like this needs. I’m a guest author here. Nothing below requires buying software.

What Contract Data Extraction Has to Get Right in a Credit Agreement

Contract data extraction is turning agreement text into typed fields a downstream system can use, such as amounts, dates, rates, covenant levels, basket sizes, and cure limits, each tied to the clause and the version of the document that governs it. On a credit agreement, the version and the cross-references matter as much as the reading.

Most published work on the problem was built around commercial contracts. The Atticus Project’s CUAD dataset, labeled under the supervision of experienced lawyers, covers 41 clause types across 510 contracts, and it does a good job of measuring what it set out to measure. Find the governing law clause. Find the termination right. Find the cap on liability.

A credit agreement asks a harder question. What is the maximum ratio of total debt to EBITDA at the next test date? The level sits in a covenant table in Article VII. It depends on a definition of Consolidated EBITDA in Section 1.01 that runs four pages and caps its own add-backs. And an amendment may have restated that definition in its entirety eighteen months after closing.

Three reads. Three places. One of them may be in another file.

Length compounds it. Nelson Liu and his co-authors put long inputs in front of language models for a 2023 paper, later printed in Transactions of the Association for Computational Linguistics. The models tended to do best with the useful passage near the start or the end, and accuracy “significantly degrades when models must access relevant information in the middle.” Definitions come first. Covenants come in the middle. The carve-out that changes the answer is wherever the last markup put it. Newer models handle long inputs better than the ones in that study, and I test for it anyway, because learning otherwise on a live facility is the expensive way to learn it.

A Credit Agreement Is a Stack, and the Test Has to Read All of It

The first thing I check on any extraction project is what the model was handed. Usually the executed agreement. Sometimes the conformed copy. Rarely the full stack, meaning the agreement, every numbered amendment, the joinders, the incremental facility notices, and the side letter that quietly moved the annual audit deadline from 90 days to 120.

The governing text is all of those, applied in the order they were signed. Read the record, the whole record, or what you extract is history.

Amendments are hard for a model in a particular way. Most don’t restate anything. They say that clause (c) of the definition of Applicable Margin is amended by deleting “4.75%” and inserting “5.25%,” and the page holding that sentence contains an instruction and no value. To get the value, the pipeline has to apply the edit to a document it may not have been given, in a sequence nobody wrote down in one place.

So the test set is built from stacks. I also plant a gap. Hand the pipeline Amendment No. 4 and hold back No. 2. A later amendment’s recitals usually list the earlier ones by name and date, so a pipeline worth trusting notices a document is missing and says so. Plenty don’t, on the first run, and that is worth knowing before a funding decision rides on it.

The Test, Built From Agreements You Already Closed

A replay proves a pipeline on something that already happened, scored against the real outcome, before anyone is asked to trust a forecast. For credit agreements, what already happened is your own book.

Twenty facilities. I don’t go lower, because under twenty a single strange agreement moves the score by more than five points and the meeting turns into an argument about that agreement instead of the result. Make them uneven.

• A handful with three or more amendments, including one where an amendment restated the whole agreement.
• ABL facilities, since the borrowing base definition feeds availability and availability gets funded.
• Deals with a pricing grid keyed to the debt-to-EBITDA ratio.
• One agreement drafted on your own form and one on the sponsor’s. They hide things in different places.
• Two or three simple ones, so a bad score can’t be blamed on hard documents alone.

Build the answer key from what your team concluded at the time, and build it before the model runs. The loan administration system gives you pricing and dates. Last quarter’s covenant compliance memo gives you the levels and the EBITDA the team accepted. The amendment memos give you what changed and when. Don’t have a new analyst reread the agreements to make the key. You would be scoring the model against a second reading, and a second reading has no track record either.

Where the memo and the loan system disagree, and some will, you have a finding before the model has run anything.

Then run it blind. Same configuration for all twenty, same prompts, no hand-fixing a document that came back strange. Freeze the model version and write down the date. A test you adjusted halfway through measures you.

The Fields I Score

About thirty fields per agreement, so roughly six hundred scored values across the set. That is enough to show a pattern and few enough that a credit analyst can build the key in about a week.

Two fields need more explanation. The pricing grid and the step-down schedule are both tables inside running text, and a model reading them as prose tends to return one row, usually the first or the one in force at closing. A grid with levels at 4.50 to 1.00 and 3.75 to 1.00 gives a different margin every time the ratio crosses a line. The field is the whole table, and it gets scored that way.

Scoring With a Column for Silent Errors

Every scored value lands in one of three buckets. Correct. Flagged, meaning the pipeline returned a value it marked uncertain or declined to return one. Or wrong, unflagged.

That bucket is the test.

A flagged miss costs a reviewer a few minutes. A silent miss costs whatever got decided on top of it, which on a credit agreement can be a funded draw, a breach nobody raised, or a margin billed at the wrong rate for three quarters. Nobody enjoys the call that follows.

Here is what a silent miss looks like when every individual field is technically right. A borrower reports $40.0 million of EBITDA for the test period and claims $12.0 million of run-rate cost savings. The definition allows that add-back up to 25% of Consolidated EBITDA, “calculated before giving effect to such adjustments.” Total debt is $290 million. The covenant maximum is 5.75x.

The extraction returned 25%, correctly. It dropped the seven words saying what the 25% is a percentage of. Field score, perfect. Covenant result, backwards.

So I score decisions too. For each agreement, run last quarter’s financials through the extracted terms and check whether the covenant result and the margin come out the way the compliance memo concluded. Field scores tell the engineer what to fix. Decision scores tell the credit team whether to use it. When a late covenant read does slip through, the bill is bigger than it looks, and I priced that out in a separate piece on death by PDF in private credit.

My pass bar, by field class, for what it’s worth. On covenant levels, test dates, pricing, and anything that feeds availability, zero silent errors across all twenty stacks. Flags are fine there. You’ll get plenty. On reporting deadlines and cure limits, a few flagged misses are tolerable. On parties and notice addresses I care less than most people expect. Writing those bars into policy is governance work, and Colin Boothe’s piece on AI governance guardrails for mid-market firms covers where they should live.

For scale, the strongest result I have from this kind of replay is 99.9% decision accuracy on more than 50 million pages, and those were regulated medical files rather than loan documents. It took a validation library and a long run of scored history to reach. Don’t expect a first run on twenty credit agreements to look like that. If yours does, check the answer key.

Where the Misses Cluster

Greater-of baskets come back half read. “The greater of $15,000,000 and 20% of Consolidated EBITDA” returns as $15,000,000, which is the number that matters least once the borrower grows, and it is the grower that decides whether a new debt incurrence was permitted.

Defined terms drift. One agreement can carry Consolidated EBITDA, Adjusted EBITDA, and Pro Forma EBITDA, which differ by a single add-back or a look-back period, and the covenant uses exactly one of them. Pipelines swap them more often than you’d guess, because the surrounding sentences are nearly identical.

Cure rights lose their structure. The clause allows no more than two equity cures in any four consecutive fiscal quarters and no more than five over the life of the facility. The pipeline returns five.

And the stale version, which I’ve covered, and which alone justifies building the test around stacks.

Rehearse on Public Agreements First

If you’d rather not start with your own book, public filings work for a dry run. Regulation S-K Item 601(b)(10) requires public companies to file material contracts as exhibits, and those filings often include credit agreements and their later amendments, which EDGAR full-text search finds in minutes. There is no compliance memo to score against, so it rehearses field scoring only. It is still a quick way to learn whether a pipeline notices a missing amendment.

The Analyst and the Engineer

This test needs two people for about six weeks, and the first one already works for you.

The credit analyst who wrote the compliance memos builds the answer key and settles every disagreement. That is the person who would do the work, and their calendar is the real constraint, because the key has to come from what was concluded then rather than from a fresh reading now. If they can’t come off the book for a week, backfill the seat for that week rather than let the test slide a quarter. That is what credit analyst staffing is for.

The second is the engineer who builds the harness. Load every stack, run the pipeline, store each value with its page reference and the document version it came from, score it against the key, and produce a sheet a credit committee can read. The work sits between NLP engineering and ETL development, and the people who are good at it can read a pricing grid without asking what a basis point is. If you’re writing that requisition, the NLP engineer job description template has a section on contracts and other text with rules attached.

Few credit funds have that engineer on staff, and a six-week test is a poor reason to open a permanent requisition. It’s contract staffing by shape. KORE1 fills it through data engineering staff augmentation, where a technology search averages about 17 days from kickoff to hire. When the test turns into a rebuild of how covenant monitoring runs, that is the ground KORE1’s private credit data operations consulting covers.

Objections After the Scoring Sheet Lands

If We Paste the Agreement Into a General-Purpose Chatbot, What Comes Back?

A fluent summary that is right about most of what it covers and says nothing about what it skipped. That helps with orientation before a call. As extraction it falls short three ways: nothing is typed, nothing points back to a page, and nothing says which amendment it assumed was current.

What Counts as Contract Data in a Credit Agreement?

Any term a credit decision depends on, including commitments, maturity, margins and floors, covenant levels by test date, the definitions feeding those covenants, basket sizes, reporting deadlines, and cure limits. Governing law matters once, at closing. The definition of Consolidated EBITDA matters every quarter.

Is Twenty Agreements Enough?

For a first read, yes. Twenty keeps one strange agreement from swinging the result by double digits. More helps if your book mixes ABL, unitranche, and sponsor-drafted deals heavily, but in my experience variety beats volume. Three agreements with long amendment histories will teach you more than thirty clean ones.

What Silent-Error Rate Should Stop a Go-Live?

One, if it lands on a covenant level, a test date, the margin, or a term that feeds availability. On those fields the pipeline goes live flagged-or-right or it doesn’t go live. Elsewhere I tolerate a few, provided each one is understood and the fix gets written into the validation rules rather than into a prompt.

Two Vendors Both Quote 95% Accuracy. How Do We Compare Them?

Ask each what the 95% was measured on, and then run both on your twenty stacks against your key anyway. Accuracy per field on clean documents and accuracy on your amended agreements are different measurements. Compare silent errors on decision fields. Everything else tends to come out close.

Does the Result Hold After the Model Is Updated?

Only until the next version ships. Keep the twenty stacks and the key as a fixed regression set and rerun it whenever the model, the prompts, or the extraction configuration changes. Once the harness exists it takes an afternoon, and it leaves the record an auditor tends to ask for.

Twenty Agreements and One Answer Key

Before the next vendor call, pick the twenty facilities and ask the analyst who wrote their last compliance memos to build the key. Thirty fields each, taken from what was concluded at the time. Then give the same twenty stacks to whatever you’re evaluating, the tool you already bought included.

If the key takes longer than a week to build, that tells you something about your book before it tells you anything about AI.

I’m glad to look at a scoring sheet if you have one half built. Connect with me on LinkedIn. When the missing piece is the engineer, KORE1 can staff the person who builds the harness.