Compliance Automation Starts in the Pipeline: Audit Evidence as a Byproduct of Delivery

Last updated: September 26, 2026

By Kris Drouet, Engineering Executive, in partnership with KORE1

“How do you know this list is complete?”

It was week three of fieldwork on a SOC 2 Type II, at a lender where I ran engineering, and the auditor had been pleasant and quick about everything up to that point, which in my experience is exactly when to start worrying. Our compliance platform showed 96 percent of its automated tests passing. She had asked for every production change in the review period and received a clean export, a little over 1,400 merged pull requests, each one approved by someone other than its author, each one with a green build. She sampled twenty-five. All clean.

Then she asked the question. The room went quiet in that particular way rooms do when everyone realizes the honest answer is “because the tool said so.”

It wasn’t complete. The payment-posting batch for our servicing platform deployed from a Jenkins server older than half the team, straight to production. Every night. Nobody had connected it to the compliance platform, because nobody thought of what it did as a deploy. Thirty-one releases went out through that box during the window. We checked afterward. Every one of them turned out fine. None of them was provable in the time we had.

That was the finding. Not a bad change. A population nobody could vouch for. And once an auditor can’t trust the population, a clean sample of twenty-five stops meaning much of anything.

I told the change-management half of this in the three patterns that keep fintech velocity high without audit pain. Have the pipeline produce the trail, sort changes into risk lanes, and define the break-glass path before anyone needs it. This is the follow-on. It covers everything outside the merge queue, and the harder problem underneath all of it, which is whether anyone can trust what your machines write down. It’s also the most practical version I have of the argument I keep making about engineering leadership in regulated industries. The audit trail is part of the product.

What Compliance Automation Actually Is

Compliance automation is having your production systems produce, store, and check their own control evidence continuously, instead of having people assemble it before an audit. It works in three layers: collecting evidence, running controls without a human in the loop, and monitoring whether each control is still working. Most tools sold under the label cover the first layer well.

The third layer is the one that matters, and it isn’t new. NIST’s guide to information security continuous monitoring, Special Publication 800-137, described a program giving an organization “visibility into the effectiveness of deployed security controls” in September 2011. Fifteen years ago. The federal government is only now rebuilding cloud authorization around that idea. Its FedRAMP 20x program says Key Security Indicators “can demonstrate security posture in near real time, replacing static yearly manual assessments.”

So the concept is settled. What most teams get wrong is the order they build it in.

They start with collection. It demos well. Connect Okta, connect GitHub, connect AWS, and watch a hundred checks turn green before lunch. I understand the appeal. But collection without monitoring is just a faster way to gather evidence nobody has checked, and monitoring without completeness is a dashboard that only knows about the systems someone remembered to plug in. The Jenkins box taught me that one. If you haven’t been through a single audit yet, start with what SOC 2 actually changes for an engineering team day to day and automate afterward.

The Question Your Dashboard Can’t Answer

Auditors have a name for evidence your own systems generate. Information produced by the entity, IPE for short. Their training says don’t trust it on sight. Good training.

The PCAOB, which sets the standards for auditors of public companies, spells the rule out in AS 1105, its standard on audit evidence. When an auditor relies on information the company produced, they’re expected to “test the accuracy and completeness of the information, or test the controls over the accuracy and completeness of that information.” SOC 2 examiners work under the AICPA’s attestation standards rather than the PCAOB’s. In my experience they ask the same question almost word for word.

Read that as an engineer and it gets concrete fast. Every automated evidence feed you build is itself a report, and the auditor has to trust the report before trusting anything in it. Three properties get you there.

• It comes from the system of record. Not a copy, not a cache inside a vendor tool that refreshes when it feels like it, and definitely not an export someone saved to a shared drive in February.
• Every run records how it was pulled, meaning the query, the time range, when it ran, and how many rows came back. A population that can’t describe its own pull can’t be re-pulled, and re-pulling is exactly what a careful auditor tries first.
• Reconciliation.

That last one deserves more than a bullet. Deploys in your evidence should equal the deploys your runners actually executed. Terminations in the HR system should equal deactivations in the identity provider. Restores tested should equal backups taken for the systems in scope. If the only proof of completeness is the same tool that produced the list, you have proved nothing at all, and a good auditor will say so, politely and in writing.

After the Jenkins finding we wrote a nightly job that pulled deploy events from every runner capable of touching production, including two that weren’t listed anywhere, and matched each one to an approved change. Unmatched deploys opened a ticket. At the next fieldwork the same auditor asked the same question, and the answer was a one-page report with a count from the runners on one side, a count from the evidence on the other, and a zero in the middle. She moved on in about a minute.

That’s what good evidence buys. A boring minute.

Write the Evidence Contract Before the Code

Before anyone opens an editor, I want one page per control that answers six questions. What’s the system of record? What event proves the control ran? How often does it run? Who owns it? What does failure look like, stated precisely enough that a script can detect it? Where does the evidence land? I call that page the evidence contract, and most of the real work of compliance automation is writing it. The code is the easy part.

Here’s the short version for the controls I instrument first in a fintech or mortgage shop. Change management is in there, but it gets one row, since I’ve covered it elsewhere.

Look at the last column. No opinions in it. Not one. Every failure condition is something a script can evaluate at midnight on a Sunday without asking anyone how they feel about the control. If you can’t fill in that column for a control, it isn’t ready to automate, and I’d argue it isn’t ready to be audited either.

Start With Leavers

If you can automate only one control this quarter, make it deprovisioning. It’s where I find the most real exposure, and it’s where manual evidence lies most convincingly.

The identity provider report always looks perfect. Someone leaves. IT deactivates them in Okta that afternoon, and the report shows a same-day timestamp. Done. Except SSO was never the whole story. At one mortgage company I worked with, a contract database administrator’s local PostgreSQL role outlived his contract by five months. Five months. His Okta account died on his last day, right on schedule. The database role had never been in Okta. Neither had two SFTP accounts on a document vendor’s server, or the service account he’d created under his own name to run a nightly reporting job. Nobody did anything wrong, exactly. The evidence covered only the systems the evidence knew about.

The job that fixed it isn’t sophisticated. Two lists and a join. Every night it pulls terminations and contract end dates from the HRIS and the vendor management system. Then it enumerates accounts in every system that holds regulated data, SSO-connected or not: databases, the loan origination system’s admin console, AWS IAM users, vendor portals. It joins the two lists on a person identifier and opens a ticket for every account that belongs to someone who’s gone. The nightly report plus the ticket trail is the evidence. An auditor sees the population, the exceptions, and how fast each exception closed, which is roughly everything they were going to ask for anyway.

If you hold a license in New York, this stops being optional housekeeping. The Department of Financial Services’ Part 500 requirement checklist says covered entities must limit and review access privileges, third-party service providers included, “at least annually,” and terminate access that’s no longer necessary. By April 15 each year, the company reviews data and documentation for the prior calendar year and files either a certification of compliance or a written acknowledgment that it didn’t materially comply. Somebody puts their name on that. I’d rather they were signing off on a nightly reconciliation than on a spreadsheet a manager filled out in March.

Access Reviews Are the Easiest Control to Fake

Nobody sets out to fake one. A manager gets a spreadsheet of 400 entitlements on a Thursday, has a quarter-end deadline, and approves all 400 in six minutes. The review happened, technically. It’s in the evidence folder. It proves a person clicked. That’s all it proves.

Automation can’t make anyone care. It can make rubber-stamping visible, though, which works better than you’d think. Generate the review from each application’s live entitlement list. Capture every decision as data, with a timestamp. Then read that data the way an auditor eventually will. Median seconds per decision. Share of reviews closed with zero changes. Reviewers who have never revoked anything, ever. None of that is a finding by itself, but all of it tells you which reviews were real before somebody outside the company works it out.

Close the loop, too. A revoke decision is an intention. The second entitlement export, pulled after the review window closes, is the proof it happened. I’ve seen more access-review exceptions come from revocations that were approved and never executed than from bad decisions. Far more.

The Ticket Is Not the Fix

Vulnerability management is where “closed” means the least.

A team marks the Jira ticket done because the fix merged. The release carrying it slips two weeks. Nobody reopens the ticket. The scanner keeps seeing the vulnerable package in production that entire time, while the SLA report, built from Jira, shows a clean close inside the window. Two systems, two truths. Only one of them matters to an attacker.

So the evidence comes from the scanner. First-seen date, severity, an SLA clock that starts at first-seen, and closure confirmed by a rescan that no longer finds it. Tickets coordinate people. They aren’t evidence of anything except coordination.

For a lot of mortgage lenders there’s a regulatory reason to care, and it’s more specific than most people expect. The FTC’s Safeguards Rule, which I came at from a different direction in my earlier piece on mortgage tech compliance, gives non-bank financial institutions a choice in 16 CFR 314.4(d)(2). For information systems, monitoring and testing “shall include continuous monitoring or periodic penetration testing and vulnerability assessments.” Without effective continuous monitoring, the rule requires annual penetration testing plus vulnerability assessments at least every six months. That little word “or” is doing a lot of work. Continuous monitoring is a named path in the regulation itself. Done well, the instrumentation in this post is one of the two ways the rule lets you comply, not a favor you’re doing for your auditors or a line item your CFO gets to cut next spring.

Where the Evidence Lives

Evidence your own engineers can edit is evidence an auditor has to discount, so it goes somewhere nobody on the team can change it, including the people who built the pipeline. On AWS that usually means an S3 bucket with Object Lock in compliance mode, where a protected object “can’t be overwritten or deleted by any user, including the root user in your AWS account,” in Amazon’s own words. Write access belongs to a single service identity and nobody else. Not even you. Retention covers the full observation period plus the next audit, and runs longer wherever a regulator keeps its own clock. Each daily bundle gets a hash, and that hash rides along inside the next day’s bundle, so a missing or altered day shows up as a broken chain instead of a gap somebody has to happen to notice. The format is plain JSON that carries its own query and parameters, readable two years from now by an auditor, an examiner, or a new hire who has never seen your code and never will. NIST maintains a machine-readable standard for this kind of material, OSCAL, and it’s where federal programs are heading. Skip it for now. You need files a stranger can read.

Build vs Buy for Compliance Automation

I hear this one more than any other. Usually from a CFO holding a quote. Fair enough.

The platforms are good at what they’re built for. Vanta, Drata, Secureframe, and their competitors connect to AWS, Okta, GitHub, Google Workspace, and your device management tool in an afternoon. They track policies and training, answer security questionnaires, and give the auditor a portal. For a cloud-native startup whose entire control environment lives in those systems, a platform can carry most of a first SOC 2 on its own. In that seat I’d buy one without much hesitation.

Regulated companies aren’t built only out of those systems. What makes you a lender or a payments company lives in the loan origination system, the servicing core, a pricing engine a vendor hosts, an SFTP server your document provider insists on, and at least one Jenkins box nobody wants to discuss. The platform sees what it’s connected to. The auditor asks about everything.

So my usual build vs buy answer, reached with the same framework I bring to every vendor decision, is both. Buy the platform for the commodity integrations. Build the reconciliation jobs for the systems that carry your regulatory risk, because nobody outside your company knows what’s under the hood of your servicing core, or why the pricing engine’s vendor still drops a flat file on an SFTP server at four in the morning. Then push those homegrown results into the platform through its API, so the auditor still gets one place to look. It’s rarely the answer either side of the sales call wants. It’s usually the one that survives fieldwork.

The Failing-Controls Report Is the Real Product

Green is a claim, not a status.

The single most useful artifact in a mature program is a daily report of which controls failed yesterday, who owns each one, and how many days each has been open. Not the dashboard with the percentage on it. The failures. Auditors expect exceptions in a Type II period, because real systems have them. What’s hard to defend is an exception nobody noticed until the auditor did. A control that fails loudly on Tuesday and gets fixed by Thursday is better evidence than one that reported green all year because nothing was checking it.

Which makes the evidence pipeline a control in its own right, and it fails like one. Jobs die quietly. An API token expires over a holiday weekend. A log source stops sending and nothing errors, since silence isn’t an error. Alert on missing evidence, not just bad evidence. PCI DSS 4.0 turned a version of this into a requirement on March 31, 2025, according to the PCI Security Standards Council’s summary of the v4.0 changes. Requirement 10.7.2 says failures of critical security control systems are “detected, alerted, and addressed promptly,” and requirement 10.4.1.1 calls for audit log reviews to be automated.

Who Actually Builds This

A disclosure before this part. KORE1 partners with me on these posts, and recruiting engineers is how KORE1 makes its money. Discount the next two paragraphs by whatever you think that deserves.

The person you want has lived through a Type II from the engineering side and knows what an auditor does with a population. That’s a narrower profile than “security engineer” or “platform engineer” on a job board. Much narrower. Look for someone who has written a reconciliation job, sat through a walkthrough, and been asked the completeness question personally. Usually that’s a DevSecOps engineer or a platform engineer who drifted toward controls work. The initial build is a defined project, ninety days or so, which is why I often start it through contract staffing and hire the permanent owner once the evidence contracts exist. KORE1’s recruiters average 17 days to fill IT roles, and 92 percent of its placements make it past their first year. That second number matters more in this seat, because the value compounds once someone learns where your particular gaps are.

Sometimes the missing piece sits higher up. If nobody in the building can decide which controls matter, argue evidence formats with an auditor, and tell the CFO plainly what the platform won’t cover, that’s a leadership gap rather than a tooling gap. A fractional VP of engineering who has run regulated shops before can own it for a few quarters without a permanent executive search.

What Fintech Leaders Ask Before They Automate the Evidence

Can a Compliance Automation Platform Get Us Through SOC 2 on Its Own?

It can carry most of the load for a small cloud-native company, because nearly every control lives in systems the platform already connects to.

The math changes with every system it can’t see. A lender running a loan origination system, a vendor-hosted pricing engine, and a legacy batch server will still need homegrown evidence for those three, and the auditor asks about them whether a connector exists or not. A connector isn’t coverage.

Will Auditors Accept Automated Evidence Instead of Screenshots?

Usually more readily than screenshots, provided they can see the source system, the exact query, and a reconciliation against an independent count.

Screenshots are about the weakest evidence there is. Auditors know it. They’re frozen, easy to fake, and show exactly one moment. Hand an auditor a pull they can rerun and a count that ties out, and most of them look relieved. Ask for a walkthrough of one automated control before fieldwork starts, not during it, and you’ll find out early what they need to see.

How Long Should We Keep the Evidence?

Two full audit cycles at a minimum, and longer anywhere a regulator sets its own retention period.

The observation window alone doesn’t cover it. Next year’s auditor may compare against this year’s populations, and a regulatory exam can reach back further than either. Storage is cheap. Rebuilding a year of evidence you already deleted isn’t possible at any price.

We Have Three Platform Engineers. What Do We Automate First?

Leaver deprovisioning, reconciled across every system that holds regulated data, because it surfaces real exposure fastest and manual evidence hides those gaps best.

Then the vulnerability SLA feed straight from your scanner, then access reviews. Change management may already be in decent shape if branch protection is enforced. Small teams have a quiet advantage here, honestly. Fewer systems to enumerate, fewer people to chase.

Does Any of This Help With SOX and PCI, or Only SOC 2?

All three benefit, because access, change, logging, and vulnerability controls show up in every framework a fintech is likely to face, even where the numbering differs.

Write the evidence contract once per control and map it to every framework that needs it. Collect once, map many times. The one caution is SOX. An external auditor testing controls over financial reporting applies the AS 1105 completeness test with real rigor, and if your evidence satisfies that auditor, the others tend to follow.

What Happens When the Evidence Pipeline Itself Breaks?

Then you have a control failure, so treat it like one, with an alert, an owner, and a record of how quickly it got fixed.

Alert on silence. A job that stops running throws no errors. It just produces no evidence, and nobody notices until an auditor asks for a date that isn’t there. A heartbeat per feed, checked daily, catches it. Let the gap and its fix sit in the record, too. A broken feed caught within a day is fine. One the auditor finds for you is not.

Make the Question Boring

That auditor will ask again next year. They always do. The goal isn’t to stop the question. It’s to make the answer so dull she writes it down and moves on. A count from the source system, a count from the evidence, and a zero between them.

Start with one control. Write its contract, reconcile it against a second source, and store the result somewhere nobody can edit. Then do the next one. Then the one after that. If you need the engineer who has built this before, talk to a KORE1 recruiter while the next observation window is still months away rather than weeks.

And if you want to compare evidence contracts, find me on LinkedIn. Bring your ugliest legacy system. Everybody has one. Mine was a Jenkins server.