Last updated: September 25, 2026
A data foundation is the set of trusted records every report and model reads from, built as raw sources kept untouched, one checked record per entity, and a few tables published only after written checks pass. The dashboard sits on top. I build it last.
The request came as a slide. Nineteen tiles.
It was my first month running data science at a specialty finance investor, and the slide was what the board had been promised for the next quarter. Policies in force. Premium due in the next ninety days. Expected maturities by year. Average days from a medical record request to a completed file. Nineteen boxes, each with a placeholder chart drawn in grey.
I did one thing with it before anything else. Next to each tile I wrote the systems it would have to read from.
Twelve of the nineteen needed three or more. Some needed five. Policies in force alone touched the policy admin extract, the premium ledger, the servicer’s monthly file, and a spreadsheet that tracked which carriers had confirmed a change of ownership. Those four sources did not agree on how many policies the firm owned, and the reason they disagreed was different for each pair. The slide showed none of it. One grey chart.
So the dashboard waited. What got built first was the layer underneath, across 11 data domains, on a medallion architecture. The board did get its tiles. They came last, and they were the quickest part.
A note on the venue. This runs on KORE1’s site, and KORE1 recruits for the seats I describe near the end through its data engineering and data science staffing group. Worth knowing before you get there. Nothing below requires you to buy software.
What a Data Foundation Is, in Build Order
A data foundation is one canonical record per borrower, policy, or claim, a validation library that says which fields are trusted, and an uncertainty tag on everything that isn’t. In build order, that means three layers. Raw data kept exactly as it arrived, a matched and checked record, and a small number of published tables that answer named questions.
The longer definition of a single source of truth covers the record itself, the grain decision, and who owns it. This piece is about sequence. What I build in week one, what waits, and why the thing everyone asked for waits longest.
Medallion is the name the industry settled on for the layers. Microsoft’s Azure Databricks documentation on the medallion lakehouse architecture describes bronze as raw, silver as validated, and gold as enriched, and then says something most vendor decks leave out. Following it is “a recommended best practice but not a requirement.” Good. The names matter much less than the rule for getting from one layer to the next, and that rule is the part nobody puts on a slide.
| Layer | What it holds | Rule for getting in | Who signs off |
|---|---|---|---|
| Bronze | Every source file as it arrived, dated, never edited | It arrived. That’s the only rule. | Nobody. Engineering keeps it complete. |
| Silver | One matched record per person, policy, or borrower, with every field checked and tagged | Passes the validation library, or carries a tag saying why it didn’t | The person who does the work, on the definitions |
| Gold | A few tables, each answering one named business question | A written definition and a named owner exist before the table does | The owner of the question |
| Dashboard | Pictures of gold tables | Reads gold only, never silver or bronze | Whoever presents it |
Read the third column top to bottom. The rules get stricter as you go up, and the last one is the easiest to write. Everything hard is already done by the time a chart is drawn.
Week One Is Bronze, and Bronze Is Boring
Land everything. Change nothing.
That was the first deliverable. Every feed the firm received, from the admin system, the servicers, the carriers, the medical record vendors, landed in one place on the day it arrived, stamped with the date and left exactly as it came. No cleaning, no joins, no fixing the obvious typo in a carrier’s file. The typo is evidence. When a number in a report looks wrong two years from now, somebody has to be able to open the file it came from.
The temptation in week one is to start matching, because matching looks like progress and landing files looks like plumbing. I have made that mistake once. We cleaned a servicer feed on the way in, the servicer restated three months of history, and we could not tell which of our corrections had been right and which had just been reversed at the source. We re-landed a year of raw files. Two weeks gone, or three if you count the week spent explaining it.
Bronze has no owner in the business sense, only an engineer whose job is to make sure nothing is missing, which is less trivial than it sounds when a vendor quietly changes its file layout on a Tuesday and the pipeline keeps running.
Silver Takes the Longest
Most of the build lives here, and most of silver is decisions rather than code.
Eleven domains. Insured lives, policies, premiums, carriers, medical records, and valuations were six of them. Each one needed a key, a field list, a named source of authority for every field, and a set of rules that said when a value could be trusted on its own, without a phone call to somebody. Order mattered, more than I’d planned for. We matched people before policies, because one person could hold several policies and a policy could change hands between owners, and a policy key alone kept splitting one life into two.
That matching ran across hundreds of thousands of individuals and policies. Not a small job. How it works deserves its own piece, and Colin Boothe’s guide to cleaning and matching messy records covers the customer-and-vendor version. The short version is that the rule for which source wins a disputed field is a business decision, written down once, and the matching engine only enforces it.
The checks sat in silver too. Premium paid cannot exceed premium billed by more than a set tolerance. A maturity cannot have a date before the policy was issued. A medical record cannot be attached to a life without a request that asked for it. Plain rules. Every one of them written with the analyst who had been checking the same thing by hand for years, still done by hand the week I arrived, because that analyst was the only person who knew which exceptions were real.
Anything that failed a rule stayed in silver with a tag and a reason. Nothing was deleted. Nothing was quietly corrected.
Gold Has a Waiting List
Gold is small on purpose.
A gold table exists because someone asked a question, the question was written as a sentence, and a person agreed to own the answer, in writing, with their name next to it. No sentence, no table. The nineteen tiles turned into fewer gold tables than tiles. Several were one question, asked by different departments in different words.
Here is how part of that slide translated. Five of the tiles. I’m simplifying a little. I’ve also rewritten the tiles in private credit terms, since that’s where most of the people reading this work now, and the pattern carries over without changes.
| Tile on the slide | The question underneath | Gold table it needs | Silver decision it waits on |
|---|---|---|---|
| Exposure by sponsor | Which borrowers roll up to one sponsor? | Exposure by ownership group, monthly | How borrower entities are matched into groups |
| Covenant headroom | Which EBITDA definition governs this loan? | Covenant test results by facility and quarter | Defined terms captured per agreement and amendment |
| Availability trend | Which receivables were eligible this month? | Borrowing base availability by borrower and month | Ineligible tests written as rules, not as memory |
| Watchlist count | What puts a loan on the watchlist, and who decides? | Watchlist entries with trigger and date | A written trigger list with an owner |
| Days to close | When does month-end start and when is it done? | Close events with timestamps | Which system’s timestamp counts as the event |
The last column is the work. Every entry in it is a conversation with the person who would do the work, and none of them can be settled by a better charting tool.
Look at the watchlist row. The tile looked like the simplest thing on the slide, a single number. The question under it was political. Two people believed they owned the decision to add a loan, and the count on the dashboard would have depended on whose spreadsheet it read. Settled in one meeting. The table took an afternoon.
Why the Dashboard Keeps Winning the Budget Meeting
It’s visible. Mostly that.
A foundation produces nothing a board can look at for weeks, and a dashboard produces a screenshot on day three. Steering committees fund screenshots. I understand why. From the other side of the table, without having watched what comes next, I might fund them too.
Then the dashboard does its job, faithfully. It shows the data exactly as it is. In 2017, Tadhg Nagle, Thomas Redman, and David Sammon had 75 executives each pull the last 100 records their own departments had created and mark the errors, and reported the results in Harvard Business Review. On average, 47% of newly created records had at least one critical error. Only 3% of the scores rated acceptable, and that was on the loosest standard the authors used.
Build the tiles on top of that and you get the 47% in a nicer font.
Then someone who knows the numbers spots a wrong one, says so in a meeting, and after that every tile gets checked against a spreadsheet before anyone quotes it, which means the old manual process is still running, next to the new one, forever. I get called into a lot of those. The fix doesn’t vary. Take the dashboard off the agenda, go down a layer, and build what should have come first. Sometimes the restart has an extraction project in the middle of it, and why a stalled OCR project reports the wrong accuracy number is the same failure seen from the document side.
There is one case where I build the dashboard early. One source system, one owner, and a question that was already settled before I arrived. A pipeline report pulled straight from a single loan system, owned by the head of originations, counting deals by stage. No matching needed and no dispute about what a deal is. Build it that week. Move on. It is a narrow case, and it comes up less often than the people requesting dashboards believe.
What the Order Bought
Across those eleven domains, data preparation time fell 84%, weeks to days, and infrastructure costs fell 35%. The definition piece has that part, and it doesn’t need telling twice.
The part I’d add is what came afterward. The automated valuation that later ran across the whole managed portfolio read from gold tables, and its runtime fell 97.7%. Hours to minutes. The medical record work and the maturity monitoring read from the same silver records. None of those projects had to start by asking which list of policies was the real one. That question had been answered a year earlier. Once. In writing.
And the nineteen tiles? Built once, in about two weeks, on top of gold. The chart work was the fastest part of the whole program. Charts are.
Who Builds Each Layer
Bronze and silver pipelines are engineering. The gate into gold is governance. Mixing up the two is how firms end up with a platform nobody trusts, and if you’re unsure which seat comes first, the difference between a data engineer and a data architect is a good place to start.
The engineering seat keeps bronze complete and runs the matching. It turns the validation library into tests that run every night, and it builds a gold table only after somebody has defined it, which takes some nerve when a sponsor wants the thing by the end of the month. That is the job KORE1’s data warehouse engineer staffing is set up to fill, and the strongest candidates have built a medallion design on Databricks or Snowflake before and can explain why the bronze layer is never edited. The guide to hiring Databricks engineers covers the screen for the Databricks-heavy version of that seat.
The governance seat owns the gate. Definitions, authority for each field, the rule list, and the refusal to publish a gold table that has no owner. That seat is covered by data governance analyst staffing for regulated firms, and it’s the one firms skip, because governance sounds like a policy binder and the work is closer to editing. If you’re writing that job spec yourself, the data governance analyst hiring guide covers the screen.
For a build with a start and an end, project-based staffing fits better than two permanent requisitions, since the run state afterward is lighter than the build. KORE1 reports 92% twelve-month retention across its placements, which matters mostly if you decide one of those seats should stay.
The dashboard gets built by whoever is best at dashboards, at the end, in a couple of weeks. Often that’s a Power BI developer brought in for exactly that stretch.
What Sponsors Ask Before the First Tile
So What Actually Counts as a Data Foundation?
Records you can read without asking anyone what they mean. Raw sources kept untouched, one checked record per entity, and a handful of published tables that each answer one written question.
If you still need a particular person in the room to interpret a report, you have reports. The foundation is what lets you stop calling that person.
Is Medallion Architecture Only a Databricks Thing?
No, it’s a pattern, and it runs on Snowflake, Microsoft Fabric, BigQuery, or a well-organized Postgres database. Databricks popularized the bronze, silver, and gold names, and Microsoft’s own documentation calls it a recommended practice rather than a requirement.
Microsoft’s Fabric guidance on medallion layers in OneLake describes the same three tiers. What matters is the admission rule between layers. I’ve seen a textbook medallion setup where analysts wrote straight into gold, which made the layering decorative.
How Long Before the Board Sees Anything?
Six to ten weeks for the first gold table on one domain, if the person who does the work can give it a few hours a week.
That first table should answer the question the board asks most often. Show it as a table rather than a chart. A plain table that matches the finance team’s number on the first try earns more trust than a well-designed dashboard that doesn’t, and it buys you the time for the next domain.
Can We Build the Dashboard in Parallel and Swap the Data In Later?
You can, and the dashboard will quietly set your definitions for you. Whatever logic someone writes to make the first version look right becomes the definition people remember.
If there’s political pressure for something visible, publish the tile list with a status on each one, ready or waiting on a named decision. In my experience sponsors trust that list more than they would have trusted an early dashboard, because it tells them what is actually blocking each number.
Where Do the Validation Rules Actually Live?
In silver, as code, next to the tables they check. Each rule has a plain-language version the business signed off on and a test that runs on every load.
Keep the plain-language version where a non-engineer can read it. The day a rule changes, it’s the analyst who notices first, and they should be able to find the rule without filing a ticket.
We Already Bought a Platform. Do We Start Over?
Almost never. The tool you already bought can usually hold all three layers, and what it lacks is the rules for moving data between them.
Start by landing raw feeds into it untouched and writing the first domain’s checks. Most of the platforms sold to private credit and insurance firms in the last five years are fine at storage. Storage was never the gap.
Take the Tile List Off Your Desk
Find the dashboard request that’s waiting on someone right now. Under each tile, write the sources it would read from and the name of the person who decides what the number means. Count the tiles where either line is blank. Start there.
That count is your foundation work, in order. Read the record before you draw it.
If you’re having this conversation with a board or an investment committee, connect with me on LinkedIn. If it has turned into a hiring conversation, KORE1’s recruiters can start the search.
