AI/ML Team Org Chart 2026: How to Structure Your AI Org

Last updated: September 2, 2026

By Mike Carter, Director of Partnership Success, KORE1

An AI/ML org chart works when the ratios do: roughly one platform engineer per three modelers, one data engineer per two, and no more than eight people reporting into any single lead.

Argue about the rest of it all you want. Those three numbers hold, and the companies that wave them off because their situation is different end up with a diagram that photographs beautifully in a board deck and a team that has not put a model in front of a paying customer in nine months.

I run a tech desk. Most weeks somebody sends me a slide with boxes on it and asks whether the shape is right. The boxes are usually fine. It is the empty space between them that tells me what is about to go wrong, because the space is where the ratios live, and the ratios are what nobody drew.

One example from last spring. A fintech in the Dallas market had eleven people under a newly promoted Head of AI. Seven of them were modelers. One data engineer, one analyst, two contract infrastructure people who reported somewhere else entirely and had never once been in an AI planning meeting. On paper it read like a real org. In practice, seven people were competing for the attention of one data engineer, and the data engineer had started declining meetings. Their models were good. Their retraining pipeline was a cron job somebody wrote in a hurry. Nothing about the chart said any of that out loud, which is exactly the problem with charts.

This piece is about the chart as an artifact. The lines, the ratios, the widths, the labels on the boxes. If you want the wider version of the argument, roles and headcount and when to add each function, we covered it separately under AI team structure and headcount benchmarks, and this piece sits one level underneath that one. For the reqs themselves, our AI/ML engineer staffing desk fills these seats across 30-plus U.S. metros.

The Ratios That Decide Whether the Chart Works

Start here, before you name a single person. A chart is a set of ratios wearing rectangles.

Below are the ones we see hold up across real teams. They are not laws. They are the point at which, in our experience placing into these orgs, something audibly starts to creak.

The 3-to-1 platform ratio is the one people fight me on hardest, and it is the one I would defend the longest. Model code is a small slice of a production machine learning system. The rest is ingestion, feature storage, serving, monitoring, retraining triggers, rollback, cost control, and the unglamorous work of noticing that a feature went stale nine days ago. Somebody owns that road. If nobody owns it, your most expensive people pave it themselves, badly, in hours they were supposed to spend modeling. The profiles that actually cover this seat are a blend of two disciplines, which we broke down in DevOps and MLOps hiring profiles.

I am overstating the 2-to-1 data engineering ratio slightly for teams that already sit on a mature warehouse. Slightly. If your analytics org already runs clean dbt models on Snowflake and the AI team is consuming rather than building, you can stretch to 3 to 1. Nobody else can.

Notice what is missing from that table. No ratio for headcount to revenue, none for engineers to GPUs, nothing about budget. Those get argued in board decks. They do not belong on an org chart, and putting them there is how you end up hiring to a number instead of to a gap.

Solid Lines, Dotted Lines, and the One Nobody Draws

Every org chart shows one relationship. Who your manager is.

Real AI orgs run three at once, and the gap between the one you drew and the three you actually have is where most of the friction comes from.

The solid line is administrative. This person writes your review, argues for your raise, and decides whether you get promoted. It is the line the chart is actually about, and it should be boring. One solid line per person. Always.

The dotted line is work assignment. In a hub-and-spoke org, an ML engineer might sit inside the payments product team every day while the Head of AI owns their career. The dotted line runs to the product lead. It is real, it consumes most of their week, and it appears on maybe a third of the charts I get sent.

The third line has no standard name and it is the one that matters at 2 a.m. Who is accountable when a model degrades in production? Not who deployed it. Who answers for it. On most charts, nobody. The model was built by a research scientist who has moved on to the next project, deployed by a platform engineer who does not know what the output is supposed to look like, and consumed by a product surface owned by a third team that assumed somebody was watching the drift metrics.

Draw that third line. Write a name next to every model in production, and put that name on the chart even if it means an ugly annotation off to the side. Ugly and true beats clean and fictional.

A quick test for whether your dotted lines are real or decorative. Ask a spoke engineer who decides what they work on next week, then ask their solid-line manager the same question. If the answers do not match, your chart is describing an organization that does not exist.

Four Charts, Drawn Out

Here is what the shape actually looks like at four sizes. These are composites of orgs we have staffed into, not templates from a management book, and the titles are the ones our clients actually use rather than the ones that sound tidiest.

Chart One: The First Four

CTO
 |
 +-- Lead AI Engineer  (player-coach, still ships code)
       |
       +-- Data Engineer        pipelines, warehouse, feature tables
       +-- ML Engineer          model to production
       +-- ML Engineer          model to production

     dotted line ---> Product (borrowed, not owned)

Four boxes. One of them is a lead who still writes code. Right at this size. It stops being right around the time a fourth engineer joins and that lead works out, usually on a bad Wednesday, that reviewing three people’s work and owning the roadmap conversation with the CTO and shipping their own feature do not fit inside the same week. No MLOps seat yet, because the lead is covering it. No research seat at all unless the model is literally the product you sell.

The data engineer sits first in that list on purpose. Companies at this stage almost always hire the modeler first and the pipeline person third, then spend six months confused about why the modeler is unhappy. If you are weighing which of those two seats opens first, we pulled the comparison apart in data scientist vs ML engineer.

Chart Two: Around Fifteen

CTO
 |
 +-- Director of AI/ML
       |
       +-- Applied ML Manager
       |     +-- ML Engineer  x4
       |     +-- Applied Scientist  x1
       |
       +-- ML Platform Manager
       |     +-- MLOps Engineer  x2
       |     +-- Data Engineer  x2
       |
       +-- AI Product Manager   (solid to Director, dotted to Head of Product)
       +-- Analytics Engineer   (shared with the data org)

This is where the first management layer appears, and where most orgs get it wrong by six months in either direction. Too early and you have a manager with three reports and no one to manage. Too late and the Director has eleven directs, no time to think, and a slow leak of senior people who feel unmanaged.

Note the platform manager sitting at the same level as the applied manager. Equal. Not a service function reporting into modeling. That parity is a deliberate signal, and teams that demote platform under applied ML get a platform team that behaves like a ticket queue.

Chart Three: Fifty and a Real Platform

CTO
 |
 +-- VP of AI/ML   [ THE HUB ]
       |
       +-- Head of ML Platform
       |     +-- Infrastructure / GPU  x3
       |     +-- MLOps  x4
       |     +-- Data Platform  x4
       |
       +-- Head of Applied ML
       |     +-- Pod leads  x3
       |
       +-- Head of AI Research      (only if models are the product)
       +-- AI Governance / Risk     (1 to 2, often part-time legal)

  ----------------- THE SPOKES -----------------
  Payments Product  --> 2 embedded ML engineers  (dotted to Applied ML)
  Risk Product      --> 2 embedded ML engineers  (dotted to Applied ML)
  Growth Product    --> 1 embedded ML engineer   (dotted to Applied ML)

Governance shows up as a box here for the first time. It is usually one person, frequently borrowed part-time from legal or security, and in almost every org we have watched go through this the need gets discovered exactly one quarter after the moment it would have been useful, normally during a customer security review that nobody scheduled time for. Put the box on the chart before you can staff it. An empty box is a plan. An undrawn box is a surprise.

The embedded engineers are the part people get wrong. They are drawn under the product teams they sit with, which is honest about where their day goes, but their solid line runs back to Applied ML. Career, calibration, and comp stay in the hub. Work comes from the spoke.

Chart Four: Past a Hundred

CEO
 |
 +-- Chief AI Officer  (or CTO, if AI is not the product)
       |
       +-- VP, AI Platform            ~35 people
       +-- VP, Applied AI             ~45 people, 6 to 8 pods
       +-- VP, AI Research            ~15 people
       +-- Director, AI Governance    ~6 people
       +-- Chief of Staff / AI Ops    1 to 2

     Every pod: 1 PM + 4 to 6 engineers + 1 designer where there is a UI

A Chief AI Officer with a direct line into the CEO earns that seat here and almost nowhere smaller. Below roughly a hundred people, that title is usually a retention play for somebody you were about to lose, and everyone in the building knows it.

Compare that shape to the equivalent chart in a data org or a product org and the family resemblance is obvious. Same hub-and-spoke logic, different center of gravity. We drew both: the data team org chart and the product team org chart run the same exercise for those functions.

How Wide Should Each Box Be?

Span of control is the least interesting thing on a chart and the fastest way to break one.

Gallup’s 2025 workplace research found the average manager now oversees 12.1 direct reports, up from 10.9 the year before, and close to a 50% increase in team size since they started measuring in 2013. The median has held near five or six. That gap between the mean and the median tells you a small number of managers are carrying enormous teams, and Gallup’s own finding is that what saves those teams is not size but whether the manager gives meaningful weekly feedback. With it, roughly seven in ten employees stayed highly engaged regardless of team size. Without it, one in four.

For AI specifically I would run tighter than the general benchmark. Six to eight. Here is why, and it is not about the manager’s calendar.

AI work has an unusually high rate of quiet failure. A model that is 4% worse than last month does not page anybody. A pipeline that silently drops a feature category looks exactly like a pipeline that works. Catching that requires a manager who has enough context on each person’s work to notice the thing that is not being said in standup, and past eight people, nobody has that context. They have status updates instead.

One more wrinkle worth naming. Gallup found 97% of managers still carry individual contributor work, spending a median of 40% of their time on it. In AI orgs that number runs higher, because the manager was usually the best engineer on the team eighteen months ago and the team still routes hard problems to them. Budget for it. A manager with eight reports who also owns the inference service does not have eight reports. They have a second job.

The Labels on the Boxes Are Doing More Damage Than the Lines

Titles are the part of the chart people treat as decoration, something to tidy up after the real decisions are made, and they are in fact load-bearing in a way that shows up six months later in the form of an offer you cannot make to a candidate you already want.

Three specific title problems come up over and over on our desk.

The first is title inflation at the front of the chart. A four-person team with a “Head of AI,” a “Principal ML Engineer,” and a “Staff Data Engineer” has spent its entire leveling budget before hire number five. Now every subsequent offer has to come in at or above those bands, or the new person reads the chart and does the math. We have watched searches stall for a full quarter over exactly this, because the company could not offer a title that made sense next to what the second employee already had.

The second is the AI engineer and ML engineer collision. These are different jobs in 2026. One builds on top of foundation models, the other trains and serves their own. Put both titles on the same chart without defining them and you will interview one pool for a job the other pool is qualified for. We wrote the distinction out properly in AI engineer vs ML engineer, and it is worth settling internally before the req goes out rather than after the third rejected shortlist.

Third, and this one is subtle. Naming a box after a technology dates the chart to the quarter you drew it. “LLM Engineer” made sense in 2024. “RAG Engineer” made sense for about eight months. “Agent Engineer” is having its turn right now. Name boxes after outcomes the box owns, not the framework it currently uses, or you will be redrawing and renegotiating comp bands every time the stack moves. It moves constantly.

None of this is theoretical for pay. Titles anchor bands, and bands anchor whether your offer closes. If you want to sanity-check what a given box should cost before you write it down, our ML engineer salary guide, AI engineer salary guide and MLOps engineer salary guide carry the real 2026 ranges, and the salary benchmark assistant will give you a market read by role and metro in about a minute.

A general software org runs the same exercise with different numbers. We broke those out in the engineering org chart and its ratios.

Charts That Look Right and Aren’t

Four failure shapes, all of which pass a casual glance.

The inverted pyramid. Six senior modelers, one data engineer, no platform seat. Everyone on the chart is expensive and the chart cannot ship. This is the single most common shape we are handed, and it comes from hiring in order of who is most fun to interview.

The orphaned platform team. Platform exists, reports to infrastructure, and has never had a conversation about model serving requirements with anyone who trains models. On the chart it looks like coverage. In practice, two organizations are optimizing against each other and calling it a roadmap dependency.

Research with no downstream. An applied research function with no pod committed to productionizing anything it produces. The output is a stream of promising results that nobody has capacity to ship, and eventually a very good scientist leaves for a company that will ship their work. That departure is usually the first time anyone notices the structural problem.

The reorg that only moved boxes. Somebody redraws the chart, everyone gets a new manager, the ratios are identical, and six months later the same complaints show up in the same engagement survey with slightly different wording. If a reorg does not change a ratio or add a missing function, it did not do anything. It cost you two months of productivity to accomplish that.

What all four have in common: the diagram was optimized to look balanced instead of to describe how work moves. Symmetrical charts are a warning sign. Real orgs are lumpy, because the work is lumpy.

Six Questions We Get Once the Chart Is on the Whiteboard

Can one person hold two boxes at once?

One person can hold two boxes routinely, three briefly, and four never. A senior AI engineer covers modeling and platform, or modeling and data, for a few quarters without damage, and the calendar shows the strain before the output does.

The pairings that hold up are modeling plus platform, or data plus platform. The one that reliably fails is modeling plus product, because the same person ends up deciding what to build and then grading their own work on whether it was worth building. Split those two the moment you have a second use case competing for time.

Are dotted lines real, or just a diagram thing?

Dotted lines are real, and they fail in one specific way: the dotted-line manager controls the work, the solid-line manager controls the review, and the two never speak. The engineer optimizes for whoever they see more often. That is always the spoke.

The fix is procedural and dull. Written input from the dotted-line lead in every review cycle, and a standing monthly conversation between the two managers. Skip either one and your embedded engineers will drift toward whatever the product team rewards, which is usually shipping fast and monitoring never.

Who carries the pager when a model degrades?

The team that owns the serving surface carries the pager, with a named escalation to whoever owns the model. Split the pager between platform and modeling without that escalation and you get a twenty-minute ownership argument while the metric keeps sliding.

Write it down per model, not per team. A recommendation model with an obvious business metric behind it deserves a different response tier than an internal classification job nobody looks at until Friday. Most orgs discover this distinction during their first bad incident, which is a fine time to learn it and an expensive one.

How often should we redraw this thing?

Redraw an AI org chart every two quarters, or whenever headcount moves by more than about a third. Redrawing on a schedule keeps it a maintenance task instead of an event, and events are what make people update their resumes.

The trigger nobody watches for is the second model in production. That is a genuine phase change, not a milestone, and the chart that carried one model into production almost never carries two. Retraining, monitoring, and rollback stop being somebody’s side project on the day the second model ships.

Does the ML platform team belong under AI or under infrastructure?

The ML platform team belongs under AI until the AI org passes roughly fifty people. Platform engineers reporting into general infrastructure get prioritized against database migrations and network work, and model serving loses that argument every single time.

Past fifty, a federated model starts making sense: platform sits under infrastructure for career and standards, with a hard-committed capacity allocation to AI. Hard-committed meaning written down, in a doc, with a number in it. A verbal promise of infrastructure support is worth precisely nothing during an outage in a different part of the business.

We drew a box we cannot fill. Now what?

Fill an unfillable box on contract while the search continues, but only for platform, MLOps, and data engineering seats. Modeling roles tied to your specific domain rarely work as short contracts, because the context-loading cost eats most of the engagement.

Time to name my incentive, since this is the section where it starts to matter. My firm collects a fee on the searches you give us, so read the next sentence knowing it costs me money. For a first data engineer at a four-person startup, run the search yourself. Your network is better than you think at that level, and a fee on a foundational hire you could have made through a warm intro is a bad trade for both of us. Where we earn the fee is the seat you have already failed to fill once, or the seat you cannot describe well enough to post. Our average time-to-hire on IT roles sits at 17 days and 92% of our placements are still there at the twelve-month mark, and those two numbers are the whole argument. Modeling seats run longer than that, and we benchmarked how much longer in time to fill an ML engineer role. When speed on a bridge hire is the constraint, contract staffing is usually the faster path, and the MLOps hiring guide covers the screening side of the seat that goes unfilled most often.

Redraw It Every Two Quarters

The chart is not documentation. It is a hypothesis about how work should move through your company, and hypotheses expire.

Most of the AI org design advice circulating right now assumes you are scaling. Worth remembering that most companies are not, yet. McKinsey’s State of AI survey found 40% of large organizations now report scaling AI agents, up from 27% the year before, while the figure for smaller organizations sat flat at 22%. The gap between those two numbers is mostly structural, not technical. Big companies built the platform function. Small ones are still asking their best modeler to do it on weekends.

Underneath all of it, the talent supply stays tight. Twenty percent growth from 2024 to 2034 is what the Bureau of Labor Statistics puts on computer and information research scientists, against roughly 3,200 openings a year and a median wage of $140,910 as of May 2024. Three thousand two hundred. Nationally. Their broader read on AI and IT employment puts the demand in the platform and data roles that most charts under-staff, which is the same finding arriving from a second direction.

Then there is the verification problem, which almost nobody puts on a chart at all. The 2025 Stack Overflow Developer Survey found 84% of developers using or planning to use AI tools while trust in the output fell to 29%, with more of them actively distrusting accuracy than trusting it, and the most experienced developers came out the most skeptical of the whole sample. Everyone is shipping AI. Almost nobody trusts it unchecked. Checking it costs hours that come out of somebody’s week, which makes verification a headcount question rather than a cultural one, and headcount questions belong on the chart as a named box instead of an assumption somebody makes quietly on a Thursday.

So draw the ratios first. Then the three lines, including the ugly one. Then the labels, chosen for what the box owns rather than what it currently runs on. If a box has no name next to it in ninety days, either kill the box or admit the plan changed.

If you would rather have somebody argue with your chart before you commit headcount to it, send it over to our staffing team. We read these diagrams most weeks. We will tell you which box to open first, and we will tell you plainly on the weeks when the answer is that you do not need us at all.