AI Data Cleaning: Using AI to Fix the Data Your AI Needs

Last updated: September 11, 2026

By Colin Boothe, CIO at Foretopia

AI data cleaning uses machine learning and language models to find duplicate records, standardize formats, and flag missing or wrong values, then proposes fixes for a person to approve. It matches fast and decides badly. So write the rules first, and clean only what your next AI project reads.

Meet one customer.

• BIRCHLINE INDUSTRIAL SUPPLY, made by the Shopify connector
• Birchline Industrial Supply, Inc., added by a sales rep in 2019
• ZZ Birchline Indust. Supply DO NOT USE
• Birchline Industrial Supply (NEW), which somebody in AR created during a collections call
• Birchline Ind Supply Co (Reno), really a ship-to
• Birchline Industrial Suply, courtesy of a 2021 CSV import
• Birchline Industrial Supply Co., the one finance actually invoices

Seven records. I made up the name. I didn’t make up the shape, because some version of that list has been sitting in nearly every customer master I’ve opened in eight years of ERP work, usually one tab over from a vendor list with the same disease.

Now point an AI assistant at it and ask for your top twenty customers by revenue. You’ll get an answer in about four seconds, nicely formatted, with a confident little summary on top. Birchline, your second-biggest account, lands at number nine and number fourteen and nowhere else, because its revenue is split across four records and nothing told the model they’re one company. The assistant didn’t malfunction. It did exactly what you asked, on exactly what you gave it.

The AI-ready pitch decks skip this bit. A model is only as smart as your customer master, and yours was assembled by a connector, three departments, and a guy named Rick who left in 2022.

Conflict of interest, stated once so I can stop hedging. My group does ERP and systems consulting, so a company that reads this and decides its data needs adult supervision might end up calling us. KORE1, whose site you’re on, places the data and ERP people who keep a master clean after consultants like me leave. We both do fine if you conclude this is hard. It mostly isn’t. Most of what follows costs afternoons rather than invoices, and if you haven’t decided where AI belongs in your operation yet, back up to where practical AI starts in a mid-market operation, because this piece assumes you’ve already picked a use case and are now staring at the data it needs.

The Definition, Before a Vendor Hands You Theirs

AI data cleaning is the use of machine learning and large language models to detect and correct problems in business records, such as duplicate customers and vendors, inconsistent names and addresses, blank required fields, and miscategorized items. The software proposes matches and corrections at scale. A person with authority decides which ones become true.

That last sentence is the one the landing pages leave off. Search the term and you’ll mostly find product pages from companies selling master data platforms, plus tutorials that assume a data science team and a Python notebook already open. Neither was written for a controller at a $200 million distributor with 38,000 customer records and one overworked NetSuite admin.

Under the buzzword sit four separate jobs. They aren’t equal. Each one breaks in a different place, and each needs a different person to sign off.

Rows one and two are where AI earns its keep. Row four is where it quietly hurts people, and I’ll get to why, because it’s the most dangerous line on this page.

Row two has a cheat code, by the way. Address formatting is a solved argument. USPS Publication 28 already decided that Suite 200 and Ste. 200 are both STE 200, uppercase, no punctuation. Nobody needs to debate it. Hand the model the standard and let it grind.

Your Chatbot Inherits Every Duplicate You Own

Gartner put a number on this in February 2025. Through 2026, it expects companies to abandon 60% of the AI projects that don’t have AI-ready data under them. In the same release, 63% of the 248 data management leaders it surveyed in the third quarter of 2024 said their organization either lacks the data practices AI needs or can’t tell whether it has them.

Can’t tell. From people whose actual job is data.

The money version is older and blunter. Gartner research from 2020 found that poor data quality costs the average organization at least $12.9 million a year. I’d take that figure with some salt at mid-market scale, since the average organization in a Gartner sample skews big. The mechanism holds at any size. Size doesn’t save you. A duplicate vendor is how one invoice gets paid twice. A customer split across four records is how a credit limit gets blown by $80,000 while each record, on its own screen, looks perfectly fine.

I’ve called AI a high speed idiot for years, and I mean it fondly. The idiot part isn’t that it’s dumb. It never stops to ask whether the input makes sense. Hand a new hire that Birchline list and by the third record they’ll lean back and ask whether these are all the same company. The model won’t. It ranks them.

Then there’s gap-filling, which is the one I’d tape to your monitor. Ask a language model to fill in 300 missing ZIP codes and it will. All 300. Plenty will even be right, because a model can infer a lot from a city and a street name. The rest will be plausible fabrications, formatted perfectly and indistinguishable from the real ones. You won’t spot them.

That’s worse than a blank. A blank at least admits it’s a blank.

Generative models are built to produce the likely answer, and cleaning needs the opposite instinct, which you have to supply in writing. My rule for clients is short. The model may flag a missing value, and it may suggest where the real one lives, whether that’s a W-9, a past invoice, or a phone call. It may never invent the value itself. I don’t bend on that.

Matching Is the Cheap Part

Deduplication is older than most people assume. The statistical approach nearly every modern matching tool still builds on comes from a 1969 paper by Ivan Fellegi and Alan Sunter in the Journal of the American Statistical Association, written to work out which records in two files describe the same person, object, or event. Fifty-plus years old. Not a startup in sight.

The tooling built on it is mostly free now. Splink, an open-source Python library from the UK Ministry of Justice’s analytics team, runs a customized version of that model and claims it can link a million records on a laptop in about a minute. The UK’s Office for National Statistics used it on its 2021 census work. Your 38,000 customers? It won’t notice.

Below that sits software you probably already pay for. Excel’s Power Query can fuzzy-merge two lists, and Microsoft’s documentation says it defaults to a similarity threshold of 0.8 on a scale where 1.00 allows only exact matches. OpenRefine, also free, clusters messy text with fingerprint, phonetic, and edit-distance methods, and its own documentation admits the clustering works only at the syntactic level. It matches spellings. Companies are a separate question.

That 0.8 default is where I’d start the argument. Wherever you set the line, some pair like Summit Fabrication and Summit Fabricators will land right on it, and that pair might be one shop with a typo or two unrelated shops forty miles apart that both bought from you last quarter. Drop the threshold and you catch more real duplicates along with more false ones. Raise it and you get a tidy, confident list that misses half the mess. I’ve seen both. No tool can pick that number for you, because it’s a trade between two kinds of wrong, and the person choosing should know which kind costs your business more.

If you run NetSuite, matching is built in and worth knowing at the edges. The Duplicate Detection and Merge feature covers customers, vendors, partners, and contacts, and out of the box it matches on email address. If your people routinely skip the email field, the default setup has very little to go on. Near Match Detection catches typos and small variations, which is nice, except Oracle’s documentation is explicit that it only checks records being created in the NetSuite UI. The customer your ecommerce connector creates at two in the morning never gets asked. That surprises people. For the backlog already in the account there’s a mass update called Entity Duplicate Resolution, which most admins I meet have never opened.

The options, sized for a company like yours.

The Record That Survives Is a Business Call

Matching tells you two records might be the same thing. Merging is different. Merging can hurt you. When two records become one, somebody decides which one survives and, field by field, whose version of the truth wins. Data people call those survivorship rules. I call them the part of the project nobody volunteers for.

Back to Birchline. The AI will flag all seven. Six are duplicates. The Reno record isn’t. It’s a branch with its own ship-to address, maybe its own tax exemption on file, maybe its own buyer who gets loud when invoices go to headquarters. Merge it blindly and you haven’t cleaned anything. You’ve deleted a fact about how you do business with that customer, and the warehouse finds out when a pallet goes to the wrong state.

So the rules get written before anything runs. They fit on a page. Which record survives, usually the one finance invoices, since that’s where the transaction history and the approved terms live. Which fields come from where, meaning billing address from the survivor, every legitimate ship-to kept as an address on it, and payment terms and credit limit from whatever AR last approved rather than from whichever record happens to be newest. And who approves, by name.

Vendors get their own paragraph because the stakes change. Same name with a different tax ID is a different legal entity, and merging those is how a 1099 ends up under the wrong company. Same tax ID under two names is almost always one payee, and also the classic setup for paying one invoice twice. The IRS runs a free service for exactly this. Payers who file information returns can use TIN Matching through IRS e-Services to check name and taxpayer ID combinations, up to 25 at a time interactively or up to 100,000 in a bulk file with results inside 24 hours. Run the vendor master through it before the merge. After is too late.

NetSuite, to its credit, agrees with most of the above. It refuses to merge two entities carrying different tax registration numbers, it won’t merge across subsidiaries in OneWorld, and a merge pulls both transaction histories onto the survivor. One detail catches people. System notes from the record you merge away don’t follow it to the survivor, so that record’s change history stops at the merge. If your auditors care, keep your own log. A spreadsheet is fine. Record IDs, survivor, approver, date, and one sentence on why.

Permissions matter too. Oracle recommends limiting merge rights to specific roles or an administrator, and merging two vendors requires full permissions on vendors. Don’t hand that to whatever integration role your AI tool logs in under. An AI client connected to NetSuite can do precisely what its login role allows and nothing past that, which I walked through in the piece on how the NetSuite MCP connector handles permissions. A read-only role can propose merges all day. It can’t execute one. Good.

Clean the Slice, Not the Company

The most expensive sentence in this topic is “let’s clean up all our data first.” It sounds responsible. I’ve seen it park AI work for a year. Whole-company cleanup has no finish line, no owner, and no number anybody reports, so it drifts until someone new asks why nothing shipped.

Scope it to the project. This is the sequence I use.

1. Name the AI project before you touch a record. “Prioritize collections calls” is a project. “Better data” is a wish with a budget attached.
2. List every field that project reads. For a collections model that’s customer, parent account, terms, credit limit, open balance, and a working contact, and it’s almost always shorter than the list people fear.
3. Profile those fields and write the baseline down, dated. Duplicate candidates, blanks, how many formats of the same thing.
4. One page of rules. What counts as a match, which record survives, which field wins, who approves.
5. Let the tool propose. People approve in batches, and every merge gets logged with a name.
6. Close the front door that let the mess in, then re-run the same profile thirty days later to see whether it held.

If the project is an agent rather than a report, step two comes straight out of the spec you wrote when you built your first agentic workflow. Same fields, same exceptions.

Step five is where the budget lives, and it isn’t software. Say the tool proposes 2,400 candidate pairs out of a 30,000-record customer master. If someone who knows the accounts can clear a pair in thirty seconds, that’s twenty hours of review. Twenty. Spread over two weeks it’s roughly two hours a day for your AR lead, which is annoying and completely doable, and nothing like the six-month initiative somebody pitched at the offsite.

What isn’t doable is skipping review and letting the tool merge on its own confidence score. Handing an AI write access to a system of record, with nobody in between, lands in the top risk class of the one-page AI governance standard I keep pushing on people. And a bad merge is miserable to undo, because by the time anyone notices, invoices, payments, and credit memos from both companies have piled onto one record and somebody in AR spends a week pulling them apart by hand. Ask anyone who’s tried.

Cleaning ahead of an ERP move instead of an AI project? Same logic, harder deadline. I covered the tiering and reconciliation side of that in the clean ERP data migration playbook.

Every Mess Has a Front Door

Duplicates don’t breed. Somebody creates each one, usually for a reasonable-sounding reason, and a cleanup only sticks if you find the door they came through.

Reread the Birchline list, because the notes after each name are the diagnosis. The connector created one because it was set up to make a new customer for any email it hadn’t seen. A sales rep made another because searching for the existing account took longer than typing a new one. AR made one mid-call. A CSV import made one in 2021 and nobody read the log.

Vendors walk in through the AP inbox. A PDF invoice arrives from a company that isn’t set up, the payment run is Thursday, and the vendor gets created from whatever’s printed on the invoice. No W-9. No tax ID. Sometimes a remit-to address that turns out to be a lockbox in another state.

Items are worse. Five people creating items with no naming convention gets you “Widget, Blue, 10pk” next to “10PK BLUE WIDGET” next to “Blue Widget (10).” An AI tool will pull those into one cluster in seconds, and three weeks later there are two more, because the people and the form haven’t changed.

Multichannel growth is where this gets ugly fastest. When we rebuilt the stack at justingredients.com, the business had grown past $150 million in sales on disparate systems across ecommerce and wholesale. The new stack carries it past $250 million, and it works because both channels now roll into one view of the financials. One view, one customer record. There’s no version of that where two systems both get to create customers.

Closing the doors is mostly configuration. Pick the single system allowed to create a customer and make every other system look customers up instead. Require a tax ID and a W-9 before a vendor record can be saved. Put a naming convention on item creation and narrow who’s allowed to do it. In NetSuite, add the fields your people actually fill in to the duplicate detection criteria instead of living with the email-only default. None of that is AI. All of it decides whether the cleanup you paid for is still clean in March.

The Seat That Keeps It Clean

A cleaned master with no owner starts decaying the week the project ends. I’d bet on it. The duplicate count creeps back a little each month, and within a couple of quarters someone proposes another cleanup project, which is how companies end up buying the same scrub every eighteen months.

Somebody has to own the rules page, approve the ambiguous merges, and watch the one number that says whether the doors are still shut. At fifty million in revenue that’s a slice of your NetSuite administrator’s week, assuming they have a week to slice. Somewhere past a few hundred million it turns into a real role, usually titled data steward or data governance analyst, and it belongs closer to finance than to IT, because finance is who bleeds when the master is wrong.

KORE1 fills both versions. A one-time scrub before a big AI project is a defined window, so contract staffing is usually the honest structure, and they keep a bench of ERP data migration specialists who’ve done this kind of cleanup before. The permanent owner is a different hire, through their data governance analyst staffing desk or a NetSuite administrator search if the seat is really an admin with more authority. A typical search there closes in about seventeen days, and twelve months on, 92% of the people they’ve placed haven’t left. That second number is the one I’d care about, for a job whose entire point is still being around next year.

What People Ask Once They’ve Seen Their Own Duplicate Count

Can I just paste the customer list into ChatGPT and ask it to dedupe?

For a few hundred rows on a business account whose data terms you’ve actually read, sure, and the first pass will be decent. Past that it fails quietly. Long lists get truncated or summarized without warning, the same prompt groups things differently on different days, and nothing about the run is repeatable. The better move is asking it to write the Power Query steps or the Python that does the matching, because then the logic sits somewhere you can read, rerun, and hand to an auditor.

How accurate is AI at spotting duplicates, honestly?

Accurate enough to propose matches, and nowhere near accurate enough to merge records unsupervised. A missed duplicate costs you a messy report. A false merge welds two companies’ receivables together, and pulling them apart is slow manual work for whoever in AR drew the short straw. Those errors aren’t the same size, so don’t tune your threshold as if they were.

Do we need a master data management platform for this?

Probably not yet. MDM platforms earn their price when four or five systems all create the same customer and none of them can be the boss. If your ERP can be the single system of record, a rules page and one accountable person get you most of the way.

Who should be approving the merges?

Whoever takes the angry phone call when an invoice goes to the wrong address. For customers that’s usually the AR lead, for vendors the AP lead, for items whoever runs purchasing. Not IT. IT owns the tool and the permissions, but it doesn’t own the truth about which Birchline you ship to, and asking it to guess is exactly how the Reno branch disappears.

Should an AI tool be allowed to write merges straight into NetSuite?

Reading, yes. Merging, not in year one, and for vendors maybe never. Let the tool read under a restricted role and produce a proposal list. A person reviews it, and the approved merges go in through NetSuite’s own merge screen or the duplicate resolution mass update, under a named user. You give up some speed. What you keep is a human name on every change to a record that money moves through, and since a merge doesn’t carry the losing record’s system notes forward, the name in your own log may be the only trail you’ve got. I’ll take slow and traceable on vendor records every single time, and I’ve yet to meet an auditor who disagreed.

What stops the duplicates from creeping back?

Fix the form, not the data. Require the fields your matching depends on, make one system the only place a customer can be born, and re-run the same profile monthly. If the duplicate count climbs, a door is open somewhere.

Sort the Vendor List by Tax ID Tonight

A ten-minute test tells you more than most AI strategy decks I’ve been handed. Export your vendor list with the tax ID column. Sort by tax ID. Count every ID that shows up more than once.

Each one is a payee you might be paying through two doors. If the count is small, go build the AI project you were planning, because your data can probably take it. If it isn’t small, you’ve just found your first project, and it’s a better one than whatever the demo promised, since it pays for itself even if the AI part never ships.

Want the wider picture before you commit to anything? The AI readiness scorecard for mid-market operations walks one real decision through your stack and shows where it breaks. It’s usually the data.

Run the sort, then hit me up on LinkedIn with the count. I’m genuinely curious. And if the number looks fine but nobody in the building has the hours to keep it that way, it stops being a project and becomes a job description, which is the point where KORE1’s recruiters can scope the data steward seat with you.