Building an AI Team Structure: Roles, Reporting Lines, and Headcount Benchmarks

Last updated: June 20, 2026 | By Mike Carter

An AI team structure groups six functions: product, research, machine learning engineering, MLOps, data engineering, and infrastructure, all under a single owner, usually a Head of AI or a VP who reports to the CTO. Most companies start with three to five people and add specialists only as the work proves out.

That one sentence hides a pile of expensive decisions. Here is how the org actually comes together, function by function and stage by stage.

We staff AI and machine learning teams across tech, fintech, and healthcare IT, and the call that comes in most often is not “find me an ML engineer.” It is some version of “we hired two ML engineers and nothing is in production, what did we do wrong?” The honest answer is almost never the engineers. It is the shape of the team around them.

A Series B health-tech company in the Austin corridor called us last fall with that exact problem. Three strong modelers, all from good programs, a year of runway gone, and not a single model serving a customer. No data engineer. Nobody owning deployment. The scientists were hand-cleaning CSVs in notebooks, arguing about feature stores on Slack, and quietly refreshing their own resumes, because nobody had hired the data engineer who should have owned that pipeline from the first week. So we didn’t place a fourth scientist. We placed a data engineer and an MLOps hire, and the first model shipped that quarter. The team they needed was the mirror image of the team they kept trying to hire.

Get the structure right before you post another req. This guide covers the whole AI organization, the cross-functional version that runs from product through infrastructure. If you only need the engineering sub-team mapped, the modelers and the platform people, we go deep on that in our ML engineering team structure guide. This one sits a level up. One disclosure worth stating plainly: KORE1 gets paid when you hand us the search. So when parts of this piece tell you to make a hire yourself and skip the fee, that’s me arguing against my own commission. Fair warning, and fair advice. If you’d rather have a recruiter pressure-test the plan, that’s what our AI/ML engineer staffing desk is for.

The Six Functions an AI Team Is Actually Built From

An AI team is the cross-functional group that takes a business problem, turns it into a working model or AI product, and keeps that thing alive in production. It spans six functions: product, research, machine learning engineering, MLOps, data engineering, and infrastructure. Not every function is a separate human early on. One strong hire usually covers two or three of them at the start.

Here is the full cast, what each one owns, and the base bands we are actually closing in 2026. These are placement numbers from our own desk, not survey averages, so they run a little hot against the public aggregators.

Read that table top to bottom and you’ll notice the cheapest-sounding rows, data and platform, are the ones that decide whether anything ever ships. Most companies read it the other way. They spend first on the research scientist with the impressive resume, then wonder six months later why the person with the impressive resume is writing Airflow DAGs at two in the morning instead of doing the research they were actually hired to do.

Early on, these roles blur. A single senior AI engineer can pick the stack, write the model, stand up a basic pipeline on Snowflake or BigQuery, and still find time to talk to a customer. That person is rare and worth overpaying for. What you should not do is assume the title on the offer letter equals the job. The line between an AI engineer and an ML engineer trips up half the hiring managers we talk to, and it is worth getting straight before you write the description. We pulled that distinction apart in a separate piece on AI engineer vs ML engineer if you want the long version.

The data engineer deserves a paragraph of its own, because it’s the hire companies skip and then regret. The U.S. Bureau of Labor Statistics projects data scientist roles to grow 34% through 2034, with roughly 23,400 openings a year, against an average near 4% across all jobs. Everyone read that number and went hunting for scientists. The roads those scientists drive on, the pipelines and warehouses, got built by nobody. So the scientists became the road crew. Hire the road crew first.

For where each of these lands by city and seniority, our salary guides break it down further: the ML engineer salary guide and the MLOps engineer salary guide both run the real ranges, not the LinkedIn fantasy ones.

Who Reports to Whom

Roles are the easy part. The fights happen over reporting lines.

Three operating models have settled out, and which one fits depends almost entirely on how central AI is to what you sell.

• Centralized. One AI team, often a center of excellence, owns every model in the company. Clean standards, one budget, one place to point. The trap is that it becomes a bottleneck, and the business units learn to wait in line instead of building intuition of their own. Common in early AI maturity, and a fine place to start.
• Hub-and-spoke. A central group sets the standards, the tooling, and the governance, while practitioners sit embedded inside product or business units with a dotted line back to the hub. For most mid-size and larger companies, this is the one that holds up. The hub stops everyone from reinventing the same broken pipeline. The spokes keep the work close to the actual problem.
• Embedded, or fully decentralized. Every business unit runs its own AI people, and the center, if it exists at all, mostly writes policy. Innovation moves fast. Standards drift, tooling forks, and two teams build the same churn model without ever knowing. Works for big, mature organizations with strong engineering discipline already in place. It punishes everyone else.

Reporting lines should move with scale, not vanity. At a small company, the AI lead reports to the CTO and that’s plenty. Cross roughly twenty AI and data people and you want a dedicated Director or VP of AI sitting under the CTO, because nobody can do that job as a side quest. Only at real scale, or when AI is the core of the business, does a Chief AI Officer reporting to the CEO start to earn its seat.

The leadership layer matters more than the org-chart hobbyists admit. McKinsey’s State of AI research found that nearly 30% of organizations now put the CEO directly on the hook for AI governance, double the prior year, and that the high performers are three times more likely to have senior leaders actively championing the work rather than rubber-stamping it. When AI is run as a side project of the tech team, it underperforms. When someone senior owns it out loud, it doesn’t. If you’ve reached the point where that owner needs to be a real executive, we walk through the whole search in our guide to hiring a Head of AI or Chief AI Officer.

Headcount Benchmarks by Company Stage

Every founder asks the same thing. How many people, and in what order? There is no universal number, but there are honest benchmarks by stage. Here is what we see actually work.

A few ratios worth committing to memory. Across a benchmark of hundreds of data teams compiled by data analytics firm SYNQ, the median data team runs about 13% the size of the broader engineering org, with fintech the heaviest at around 3.5% of total headcount and B2B the lightest near 2.4%. Inside the AI group itself, plan on roughly one MLOps or platform engineer for every four to six people building models. And early on, your data engineers should outnumber your data scientists, not the reverse. The companies that flip that ratio are the ones calling us in month nine.

Across twenty years and more than 30 U.S. metros, our placements carry a 92% twelve-month retention rate, and the AI and data teams that hold their people best almost all share one boring trait. They hired the data and platform people before the third modeler, not after. The exciting hire feels like progress. The plumbing hire is what makes the exciting hire productive.

Where AI Teams Go Wrong, the Patterns We See on Repeat

Most broken AI orgs break the same handful of ways. None of them are about talent.

The first is the one from Austin. Hiring research before infrastructure. A PhD who wants to publish, parachuted into a company with no warehouse and no pipelines, will spend half a year doing data engineering they never signed up for, and then they leave. Every time.

Then there’s the three-modelers-no-MLOps shape. Plenty of model code, nothing in production, because building a model and running that model reliably in front of real customers are genuinely different jobs, and this company had staffed exactly one of the two. A model that lives in a notebook is a science project. A model that survives a Tuesday outage is a product.

Copying the frontier-lab org chart is another favorite. A Series A startup does not need the structure OpenAI or a Google DeepMind runs. You need three people who can ship something a customer touches this quarter, not a research division with a publication calendar, a reading group, and an internal conference nobody outside the team will ever attend. Borrow the ambition. Skip the org chart.

The quiet killer is no single owner. AI gets split between a CTO and a product VP who disagree about what it’s for, and the team spends its energy refereeing instead of shipping. One owner, one roadmap. And the last one McKinsey already named. Treat AI as a tech-team errand instead of a business priority with executive air cover, and it underperforms the companies that don’t. We’ve seen both. The difference shows up in the product, not the press release.

What Hiring Managers Ask Us About AI Teams

What’s the smallest team that can actually ship AI?

Two people, if they’re the right two. A senior AI or ML engineer who can build and deploy, paired with a data engineer who keeps the inputs clean, can put a real model in front of customers. Add a product manager the moment priorities start colliding.

Do we hire a data scientist or a data engineer first?

The data engineer, almost always. A data scientist with no clean data to work from spends their first months building pipelines they resent, and they don’t stay. If your data already lives in a proper warehouse and you have a real prediction problem, then a scientist earns the first seat. Most companies aren’t there yet.

Where should the AI team report?

Under the CTO at first. One lead, reporting straight to the CTO, covers you until you pass roughly twenty AI and data people. After that, a dedicated Director or VP of AI keeps it from drifting. A Chief AI Officer to the CEO is a real-scale move, not a Series A one.

When does a Head of AI or Chief AI Officer make sense?

When AI stops being a feature and becomes a function. If it spans multiple business units, carries regulatory weight, or sets company strategy, you want a senior executive owning it end to end. Before that, a strong VP of AI under the CTO does the job without the title inflation. Our Head of AI hiring guide covers the line.

How many MLOps engineers do we need per ML engineer?

Roughly one for every four to six people building models. Push the ratio thinner than that and deployment becomes the bottleneck, with expensive modelers idling while one platform engineer drowns. The first MLOps hire should land before your second model goes live, not as a cleanup crew afterward.

Should AI sit inside engineering or stand on its own?

Inside engineering early, on its own later. At small scale, keeping AI close to the engineers who ship everything else avoids a silo you’ll have to dismantle. Once the group passes a dozen people and serves multiple product lines, a hub-and-spoke structure with its own leader usually beats both extremes.

Build the Org, Not the Title

The teams that win with AI in 2026 aren’t the ones with the flashiest research hire. They’re the ones that built a balanced org: someone to decide what’s worth building, people to build it, and the data and platform crew that keeps it alive after launch. Get that shape right and the individual hires get a lot easier. Get it wrong and no resume saves you.

If you’re staring at a headcount plan, a finite budget, and a board that wants AI in the product by next quarter, and you still aren’t sure which seat to fill first, that’s the conversation we have just about every week. Talk to a KORE1 recruiter and we’ll pressure-test the structure before you spend a dollar on the search. Even when the answer is that you can run it yourself, we’ll tell you. It’s the honest version, and it’s usually the cheaper one too.