Last updated: July 22, 2026
By Kris Drouet, Engineering Executive, in partnership with KORE1
Before you sign an AI vendor contract, run the pilot through four gates. Proof, integration, exit, and kill. Each one is a go or no-go you clear before the signature, not after. Clear all four and the pilot has a real shot at production. Miss one and you are paying full price for a demo.
A vendor once ran me a pilot that was genuinely good. The model did what they said it would do. My engineers liked it. The numbers in the room were real. I still walked away, and the person who owned the budget thought I had lost my mind for about a week. Then the integration estimate came back at four times what the vendor had implied, the data export clause turned out to be a paragraph nobody had read, and the deal quietly fell apart on someone else’s desk two quarters later. I did not walk because the demo was bad. I walked because the demo was the only thing anybody had checked.
That is the whole problem with buying AI right now. The pilot is designed to be persuasive. The contract is designed to be signed. Neither one is designed to tell you whether the thing survives contact with your production systems, and by the time you find out, the money is already spent. This piece sits underneath our larger guide on build versus buy in AI and why most pilots die before production. That one is about the decision. This one is about the diligence you run once you have decided to buy, in the narrow, dangerous window between a good demo and your signature.
Here is the part that took me years to see clearly. When a bought AI pilot dies after a strong demo, it almost never dies at the model. In my own count, across more of these vendor deals than I want to admit to, the failures cluster at two gates that nobody checks before signing. The integration, and the way out. Not the algorithm. The plumbing around it and the exit nobody negotiated.
The research says the same thing in colder language. RAND put a hard number on it. More than 80% of AI projects fail, roughly twice the rate of IT work with no AI in it, in a 2024 report built on interviews with 65 data scientists and engineers. The top causes were not technical. They were leadership and framing. The most expensive mistakes land before anyone writes a line of code. So write a checklist. Run it every time. Here is mine.
What an AI Pilot Production Checklist Actually Is
An AI pilot production checklist is the set of go/no-go tests you run against a vendor and a contract before you buy, to judge whether a pilot can survive real production instead of just a slick demo. Mine has four gates. Proof, integration, exit, and kill. You run all four during diligence, before the signature commits you to anything, because after the signature your bargaining power is gone and your options narrow to whatever the vendor already agreed to in writing.
Gates, not a score. A checklist that averages nicely is useless here, because one hard no should stop the deal no matter how well the other three go. One no ends it. A vendor can ace proof, integration, and kill, and if there is no clean way to get your data back out, that single failure is enough. Treat each gate as a wall. You clear it or you stop. That framing alone will save you from the most common trap in AI procurement, which is talking yourself into an average when one of the numbers was a zero.
Gate One: The Proof Gate
A demo proves the easy twenty percent. It proves the model can do the thing once, on clean data, on the vendor’s own infrastructure, with their best engineer in the room. That is a rehearsal. It is not evidence. It is theater. With a good production budget.
The proof gate asks one question. Show me this running in production, at a company that looks like mine, under load, and let me talk to the engineer who keeps it alive.
Not the champion who bought it. The champion will tell you it changed their life, because their reputation is now attached to that decision and nobody wants to admit they backed a lemon. You want the operator. The person who gets paged when it breaks. Ask that person three things. What breaks most often. What the real uptime looks like, not the number on the slide. And whether they would buy it again knowing what they know now. The pause before that last answer tells you more than the entire sales cycle did.
For a regulated shop this gate gets sharper. I have spent twenty-five years in mortgage tech and fintech, where a bad quarter is measured in failed audits and not just missed features, so my version of the proof gate includes a reference customer in my own regulatory world. An AI vendor that has only ever shipped to unregulated startups has not proven anything about whether it holds up when an examiner asks how a decision was made. If they cannot produce that reference, the answer is not no. The answer is not yet. Come back when you can.
Gate Two: The Integration Gate
This is the gate where most of the bodies are buried, and it is the one the pilot is specifically built to hide from you.
The vendor demoed against a clean sample of your data. Production is not clean. Production is your actual authentication, your actual data residency rules, your actual field mappings that three different teams defined over eight years and nobody fully documented. The integration gate forces the real surface area into the open before you pay for it. How does it authenticate? Where does the data physically live, and does that survive your compliance review? Does it push events with webhooks, or make you poll for changes? If the answer is polling, your latency story just got worse in a way the demo never showed.
Then the question almost nobody asks. Who absorbs the work when the vendor changes their API. Because they will. A vendor deprecates an endpoint, moves to a new auth model, sunsets the SDK you built against, and suddenly your team is doing an unplanned migration on the vendor’s schedule instead of yours. I have watched exactly that turn a clean integration into a source of quiet dread on a team, the kind where every release cycle somebody asks if this is the sprint the vendor finally breaks us. That is not hypothetical. That is a Tuesday.
I wrote a whole separate piece on how badly this specific cost gets underestimated, using a real integration I lived through, in the real cost of an Encompass integration. The short version for the checklist is this. Get the integration scoped by your own senior engineer, not the vendor’s solutions consultant, and take the vendor’s timeline estimate as the floor, never the ceiling. If your engineer and their consultant are more than a little apart on the number, believe your engineer. Every time.
Gate Three: The Exit Gate
You should know how you leave before you agree to enter. That sentence sounds obvious and almost nobody lives by it, because the exit is the least exciting thing in the room during a deal that everyone is excited to close.
The exit gate is three plain questions. Who owns the data you put in and the outputs the model produces? In what format can you get all of it back, and has anyone ever tested that export against a real migration? And what does it cost, in money and in months, to move off this vendor to another one? If the honest answer to that last question is that leaving would be so painful you never realistically could, you are not buying a tool. You are renting a dependency, and the rent goes up whenever the vendor decides it should.
The ground is shifting toward the buyer now. Use that. The EU Data Act entered into force in January 2024 and has applied since September 2025, and it sets an actual framework for customers to switch between cloud and data-processing providers and to move their data out, described on the European Commission’s Data Act page. Whether or not you fall under it directly, serious vendors are already writing portability into their contracts to stay ahead of it, and you can ask for the same language. The place to win the exit is at the negotiating table, while the vendor still wants your signature. After you sign, portability is a favor. Before you sign, it is a term.
Gate Four: The Kill Gate
Decide what failure looks like before you pay for success. Write it down. That is the entire gate, and it is the one that separates the teams that recover from a bad AI bet from the teams that keep funding one for eighteen months because nobody wanted to be the person who called it.
Before you sign, write down the one business number this pilot has to move, and the date by which it has to move it. One number. Not a dashboard. A single metric that someone in finance already tracks and already cares about. A threshold. A deadline. Nothing softer than that. Cut manual review time by thirty percent by the end of Q2. Specific enough to pass or fail. Then, and this is the part people skip, make the contract honor that decision. A pilot-to-production milestone. A termination clause for non-performance. An off-ramp that does not cost you the full annual commitment when the thing does not hit the number you agreed on together.
This is where the market data quietly backs the discipline. MIT’s NANDA researchers found that buying AI from specialized vendors succeeded around 67% of the time while internal builds succeeded only about a third as often, even as 95% of enterprise generative AI pilots delivered no measurable impact on the bottom line, reported by Fortune. Sit with both numbers at once. Buying is usually the smarter play, and most pilots still fail. The resolution to that apparent contradiction is the kill gate. Buying wins when you buy with an exit and a deadline. Buying loses when you buy on faith and hope the demo was the whole story.
| Gate | The question you ask | A good answer sounds like | Where it kills you if you skip it |
|---|---|---|---|
| Proof | Is it running in production somewhere like us? | A reference operator, in your regulatory world, who takes your call | You buy a demo and inherit an experiment |
| Integration | What does wiring it into our core actually take? | A surface your own engineer scoped, with a real number | The vendor’s API change becomes your unplanned migration |
| Exit | How do we leave, and what does leaving cost? | Tested export, clear ownership, portability written into the contract | You rent your advantage and the rent keeps rising |
| Kill | What number, by what date, ends this? | One finance metric, a deadline, a termination clause that matches | You fund a failure for a year because nobody defined failure |
Running the Whole Thing in One Diligence Cycle
Four gates sounds like a lot of process for a team that just wants to ship something with AI in it. It is not. Run correctly, the whole checklist fits inside a single diligence cycle, and it mostly replaces meetings you were already going to have with sharper versions of the same meetings.
Give it a single owner. One senior person who runs all four gates and has the authority to stop the deal at any one of them. Not a committee. Committees average. We already covered why that is the enemy here. That owner needs enough technical depth to tell a good vendor from a confident one, which is the real reason I argue so often that engineering leaders have to stay technical even after they stop writing code every day. You cannot run the integration gate from a slide deck. You have to read the vendor’s reality yourself. Not their story.
If you do not have that person in the seat, get one before you buy, not after the integration is on fire. Sometimes that means bringing in a senior engineer on a contract basis specifically to pressure-test the vendor and scope the real integration, which is cheap insurance against a bad multi-year commitment. Sometimes it means the gates tell you to build the thing after all, in which case you are now making a very different hiring decision, and our guide on build versus buy in AI is the right place to start that conversation.
When the Checklist Says Walk Away
Sometimes all four gates come back clean and you sign with real confidence, which is a good feeling and the entire point of doing the work. Sometimes a gate comes back a hard no, and the checklist just saved you from a mistake that would have shown up on your desk twelve months from now wearing a much bigger price tag. Both outcomes are wins. A no that arrives before the signature is the cheapest no you will ever get.
Now the obvious bias, disclosed plainly. KORE1 places engineering and AI talent for a living, so we benefit when your gates point toward a build and you need people to staff it. With that said, the number I would put in front of you is retention, because for this kind of work it is close to everything. KORE1 fills IT and engineering roles in about 17 days on average and holds a 92% twelve-month retention rate on its direct hire placements, and retention is what actually keeps a bought-then-extended AI system alive after the launch excitement fades and the original owner moves on. A tool with nobody to run it is a liability with good branding. If your checklist lands you on the build side, or you want a second read on a vendor before you commit, KORE1’s engineering staffing team has scoped this kind of decision before.
And if you just want to argue with the four gates or tell me which one I am underweighting, I am genuinely up for that. Connect with me on LinkedIn. If you would rather have KORE1 help you get the right person into the seat to run this diligence, you can reach their recruiting team here.
What Executives Ask Me Before Signing an AI Contract
How is this different from a normal software procurement checklist?
The exit and kill gates carry far more weight with AI. A normal SaaS tool degrades gracefully and you can usually rip it out. An AI system gets wired into decisions, trains on your data, and creates a dependency that is genuinely hard to reverse, so the way out and the kill criteria are not paperwork here. They are the whole risk.
The proof gate changes too. With ordinary software a demo is a fair preview of the product. With AI a demo shows you the model on its best day, on clean data, and hides the drift, the edge cases, and the maintenance that decide whether it survives. You are buying the boring eighty percent the demo never touches, so you have to go look for it yourself.
The vendor will not give me a production reference. Is that a dealbreaker?
Usually, yes, at least for now. No production reference means one of two things. Either nobody is running this in production yet, or the people who are will not vouch for it. Both are answers, and neither is the one you want before you sign a multi-year commitment.
There is an honest exception. If you are knowingly buying from an early-stage vendor because the upside is worth the risk, that can be a fine bet, as long as you name it out loud and price the risk into the deal. What you cannot do is let a slick demo stand in for evidence that does not exist and call it diligence. Early-stage is a choice. Pretending is a mistake.
Who should actually own this checklist?
One senior technical person with the authority to kill the deal. Not procurement alone, not a committee, and not the executive whose budget is riding on a yes. The owner needs enough depth to scope the integration and enough standing to say no to a vendor the whole room already likes.
That combination is rarer than it should be, which is exactly why so many AI deals sail through on enthusiasm. If the only people in the room are a business sponsor who wants it and a salesperson who is selling it, there is no gate. There is just momentum. Put someone in the room whose job is to be unmoved by the demo.
We already signed and skipped a gate. What now?
Run the gate late, because a late gate still beats no gate. Go find your production references now, scope the real integration now, and test the data export now, before renewal instead of before the original signature. You have less bargaining power than you would have had, but renewal is a second, smaller window where some of it comes back.
Then make the kill gate real for next time. Write down the one number and the date retroactively, and hold the tool to it at renewal. The worst outcome is not a skipped gate. The worst outcome is a tool that quietly failed its purpose two quarters ago and keeps drawing budget because no one ever defined what failing would look like.
Does this slow everything down when we are trying to move fast on AI?
It speeds up the part that matters and slows down the part that ruins you. The four gates fit inside one diligence cycle and mostly sharpen meetings you were already having. What they add is a few days. What they save is the quarter you would have lost integrating a tool you could not exit.
Speed on AI is not signing fast. It is reaching production without a detour through a failed pilot, an angry finance team, and a migration nobody budgeted for. The teams that actually move fast are the ones that spent four honest days at the gates and then never had to slam the brakes later. Slow is smooth. Smooth is fast.
