Big Data Engineer Job Description Template 2026

Last updated: September 24, 2026

By Mike Carter, Managing Director, KORE1

A big data engineer job description has to state the daily data volume, the platform with its Spark, Hadoop, and Java versions, and whether the hire will run the cluster or move off it. Our 2026 salary guide puts mid-level base pay between $134,000 and $152,000, and seniors land from $152,000 to $180,000. The version line carries more weight this year than it used to. Between March and June 2026, Cloudera, Apache Hadoop, and Amazon EMR all made Java 17 their default.

Sixteen tools. Two of them retired.

That was the requirements list an electric utility in the Mountain West sent us in March. It started at Hadoop and MapReduce, wandered through Pig, Sqoop, Flume, and Oozie, and only reached Spark and Kafka about halfway down. The title said Big Data Engineer. The band was fair. Three weeks on the job boards had produced forty-one applicants and exactly one phone screen anybody wanted to repeat.

I asked the data platform manager what this person would actually spend the first six months doing. The answer had almost nothing to do with the list. Their Cloudera cluster, 38 worker nodes on a release from 2021, takes in interval reads from about 1.9 million smart meters. Each meter reports every fifteen minutes, which works out to roughly 182 million rows a day before anyone adds voltage and outage events. The Spark jobs were written in Scala against 2.12. The scheduler was Oozie, around 400 coordinator jobs, and nobody who wrote them still worked there. Cloudera’s newest release, 7.3.2, had just dropped support for Java 8 and 11, and the utility’s security team wanted the cluster current before its next audit. That audit was in October.

So the real job was an upgrade and a scheduler migration, carried out while meter data kept landing every fifteen minutes. Nothing in the posting said so. It asked for Sqoop, which the Apache Software Foundation retired in June 2021, and for Oozie, which followed it into retirement in February 2025. Plenty of the right people saw it. The engineers who could have done the work read that list, took it for a maintenance seat on a museum piece, and kept scrolling.

We rewrote it in an afternoon. Same title, same band, eleven lines shorter, with the meter count, the row count, the Java 17 upgrade, and the Oozie retirement in the first paragraph. The first qualified applicant showed up nine days later, an engineer who had moved a natural gas distributor’s cluster to Java 17 the year before.

I’m Mike Carter, one of KORE1’s managing directors. Reqs like that one cross our big data engineer staffing desk most weeks, occasionally before the posting goes live and more often after it has sat for a month. The employer pays KORE1 when someone we sent over gets hired, and that bias is worth keeping in view as you read. You can use every piece of this without calling us.

A quick fork before the rest. If your data fits in a warehouse and a nightly batch, you’re probably hiring a data engineer, and the template I wrote for general data engineer postings will fit you better. What follows is for Spark and Hadoop work at volumes that need a cluster.

Java 17 Arrived Everywhere at Once

Every major way to run Spark or Hadoop raised its Java floor inside about thirteen months. Here is the sequence, pulled from each project’s or vendor’s own release documentation.

Sources: the Spark 4.0.0 release notes, Cloudera’s 7.3.2 release summary, the Hadoop 3.5.0 documentation, the AWS announcement of Spark 4.0 on EMR, and Google’s image 3.0 component list.

Read that table as a hiring manager and one conclusion falls out of it. Whatever you run, somebody on your team is upgrading it soon, or ought to be. Upgrade plans slip.

The Spark project saw this coming. Its versioning policy gives the 3.5 line an extended support period “to allow for migrations to be completed,” and that period ends in November 2027, with security fixes only until then. If your jobs run on 3.5 today, you have roughly fourteen months of patches and no new features. Plenty of time, if the work starts this year. Not much, if it starts after the person who knows the jobs has left.

So put the versions in the posting. Engineers read these lines closely. I’d take something close to this, straight from a req we helped rewrite in July: “Today we run Cloudera 7.1.9 with Spark 3.3, Java 8, and Scala 2.12, and by next summer we’ll be on 7.3.2, Spark 3.5, and Java 17. You’ll lead that upgrade.” It screens harder than any adjective you could write. An engineer who has done it recognizes the job instantly. An engineer who hasn’t knows exactly what to go learn before the interview, which is also fine.

What Actually Changes Between Spark 3.5 and Spark 4

Good candidates will ask about this in specifics, so it helps to know the shape of it before they do.

• Scala 2.13 only. Builds break. Every Scala job and every internal library compiled against 2.12 has to be rebuilt, and some older third-party jars have no 2.13 build at all. A team with a homegrown UDF library from 2017 feels this first.
• ANSI SQL mode is on by default. An invalid cast or an arithmetic overflow that used to come back as a null now fails the query. AWS put the upside plainly in its EMR announcement, saying ANSI mode “catches invalid operations at query time rather than silently propagating nulls downstream.” The data gets better. It also makes for a rough week on any job that has leaned on those nulls for five years without anyone noticing.
• JDK 17, with every JVM flag, garbage collector setting, and reflective hack that went with the old runtime.
• Smaller removals that still bite someone: Python 3.8 support is gone, SparkR is deprecated, and Mesos support was dropped.

None of that is exotic. It’s just a lot of it, touching every job at once, and the output has to match before anyone turns the old cluster off. AWS ships an upgrade agent for EMR that scans applications for Spark 4 incompatibilities. It helps. Somebody still reads every diff and signs off on every table.

Nobody writes “survived a Scala 2.13 upgrade” on a resume. More people should.

If Your Cluster Lives on Azure HDInsight

Azure deserves its own section this year. According to Microsoft’s HDInsight retirement table, versions 4.0 and 5.0 were retired on March 31, 2025, which leaves 5.1 as the supported line. The Enterprise Security Package, the add-on that handles Active Directory sign-in and Apache Ranger authorization on secure clusters, hit its retirement date on July 31, 2026, and Microsoft points customers toward alternatives such as Microsoft Fabric.

A posting that says “HDInsight” and nothing about where the platform goes next reads, to a strong Azure candidate, like a team that hasn’t decided. Decide first. Staying on HDInsight 5.1, moving to Fabric, and moving to Azure Databricks are three different jobs with three different candidate pools. If it’s Databricks, our Databricks engineer job description covers that seat in more depth than I will here.

Retired Apache Projects in the Requirements List

Old requirements lists don’t die. They get copied into the next req, then the one after that, and a decade later a 2016 stack is still recruiting for a 2026 job.

The Apache Software Foundation moves projects it no longer maintains to the Apache Attic. A surprising number of Hadoop-era names now live there. I check that list more than I expected to.

Retirement dates are from each project’s Attic page, for example Sqoop and Oozie.

Two cautions, because this can be taken too far.

Retired upstream does not mean gone from your cluster. If Oozie still runs 400 of your jobs, it belongs in the posting, framed as work to be done instead of a skill to screen for. “We run about 400 Oozie coordinators and want them on Airflow by the third quarter” is a sentence a strong candidate respects, because it’s honest about the mess and it puts a finish line on it. “Expert in Oozie required” attracts a very different, much smaller crowd. We’ve watched it happen.

And plenty of the old names are fine. Hive is alive and shipping. Google’s newest image carries Hive 4.2. The point isn’t to scrub every tool older than your intern. It’s to stop asking for expertise in software the maintainers themselves have walked away from.

Six Numbers a Big Data Candidate Checks First

Adjectives don’t recruit. “Massive datasets” could mean 200 gigabytes or 20 petabytes, and experienced engineers stopped believing the word years ago. Numbers they believe. Here are the six we push hiring managers to put in writing, with the kind of line that works.

You don’t need all six. Three is usually enough, and the three depend on the seat. A streaming role lives or dies on the event rate. A migration role wants the job count and the history. An operations-heavy role wants the freshness promise more than anything, because that’s the thing that wakes people up.

One caution on the first row. Say compressed or raw. A team that means 4 TB of compressed Parquet and a candidate who pictures 4 TB of raw JSON are imagining different jobs, and the candidate will find out which one it is in the second interview, which is a late and expensive place to find out anything.

Your platform lead has most of the six in a monitoring dashboard somewhere, and the rest are one query away. Ask for them in the same meeting where you settle the band. It takes twenty minutes, and it’s the single most useful twenty minutes in the whole req.

Scala, Java, or PySpark

Last fall an ad-tech company in Playa Vista posted a big data role with “Scala required, 5+ years” near the top. Their Spark code had been about 85% PySpark since 2023. The Scala line was a leftover from the 2019 version of the req, and before the platform lead caught it, the recruiter screen had turned away two strong PySpark engineers on that line alone. One of them took a job at a competitor two blocks away. The platform lead still brings it up.

Say which language the jobs are written in. Percentages are fine. “Roughly 70% PySpark, 30% Scala, and new work is Python” tells a candidate far more than “Scala or Python.” Candidates respect that. If the Scala share is legacy code you’re keeping alive through a 2.13 upgrade, say that too, since reading and fixing someone else’s Scala is a different skill from writing fresh Scala, and the engineers who enjoy the first one are rarer.

Java still matters in two places. Banks and older enterprise shops that wrote their Spark jobs in Java tend to know it and should say it. And the streaming side, Kafka Streams and Apache Flink in particular, leans on the JVM regardless of what the batch jobs use. When streaming is a big enough share of the work to be its own seat, our Kafka engineer staffing team treats it as one, and so should the posting.

Salary Ranges to Publish

Post a range. The engineers you want are comparing your posting against three others that include one, and a posting without a number usually loses that comparison before anyone reads the responsibilities.

These bands come straight from our big data engineer salary guide, which builds its experience bands from Glassdoor data and checks them against what we negotiate on real offers.

One adjustment we see again and again. A version upgrade with a hard date on it prices toward the senior band even when the title says mid-level, because the pool of people who have done one recently is small and they know it. They negotiate accordingly. If the seat carries the Spark 4 or Java 17 move, budget for that up front instead of discovering it at the offer.

Metro and scope move these numbers more than the title does, and our salary benchmark assistant will price both for a specific req.

Big Data Engineer Job Description Template

Swap your own facts into the square brackets. Parenthetical notes are coaching for the person drafting the req and should come out before it goes on any job board. A section that doesn’t apply to your team gets deleted outright, not padded.

Job Title

[Big Data Engineer / Senior Big Data Engineer / Big Data Engineer, Spark Platform / Big Data Engineer, Hadoop Migration] (Add a short qualifier when the work has one center of gravity. “Hadoop Migration” pulls in people who have done one and quietly warns off people who only want greenfield work, which is the point.)

The Team and the Data

[Company] runs [what the data does for the business, such as meter-to-bill processing, card fraud scoring, ad attribution, or claims adjudication]. The platform takes in about [daily volume, compressed or raw] from [the handful of source systems people care about] and keeps [N years] of history. You’ll join a team of [N] engineers reporting to [role], working alongside [analytics, data science, SRE, or whoever depends on the output].

Platform and Versions Today

(Experienced candidates hunt for this block first. Most postings leave it out entirely.)

• Where it runs: [Cloudera Base on premises / Amazon EMR / Google Managed Service for Apache Spark / Azure HDInsight / Databricks / a mix, with rough shares]
• Versions: Spark [x.y], Hadoop [x.y], Java [8 / 11 / 17 / 21], and [Scala 2.12 / Scala 2.13 / PySpark]
• Table format and storage: [Hive tables / Apache Iceberg / Delta Lake / Apache Hudi] on [HDFS / S3 / GCS / ADLS]
• Scheduling: [Airflow / Oozie / Dagster / the cloud’s own scheduler], about [N] scheduled jobs
• Streaming: [Kafka / Flink / Spark Structured Streaming / none yet], peaking near [N] events per second
• Next twelve months: [the version upgrade, the platform move, or nothing planned, and if it’s nothing, say that]

What You’ll Own

• Build and run the batch and streaming pipelines that turn [sources] into [the tables people depend on], at [volume]
• Keep [the freshness promise, such as billing tables ready by 5 a.m.] and share pager duty on a [weekly / biweekly] rotation with [N] other engineers
• [Upgrade seat] Lead the move from [current versions] to [target versions], including rebuilding [Scala 2.12 jobs and internal libraries] and proving each job’s output matches before cutover
• [Migration seat] Move [N] workflows off [Oozie / the on-prem cluster] to [target], running old and new side by side until the numbers agree
• Keep job runtimes inside the batch window as volume grows
• Work with [analysts / data scientists / the ML platform team] on table design, data contracts, and what “done” means for a dataset
• Write things down well enough that the next engineer can run the platform without calling you

Required

• [4+] years building pipelines that ran in production on Spark[, with HDFS and YARN if you’re on premises]
• Strong [PySpark / Scala / Java], meaning the language your jobs are actually written in (pick one, or state the split)
• Hands-on work with [your platform] at a volume in the same range as ours
• Time on call for pipelines other teams depended on, and a specific incident you can walk us through
• [Upgrade seat] A Spark or Java version upgrade you took from plan to production

Helpful

• Kafka or Flink in production, if streaming is on the roadmap
• Maintenance of [Iceberg / Delta Lake / Hudi] tables at scale
• Airflow, especially moving another scheduler’s workflows onto it
• Background with [your industry’s data] or [the regulation you work under]
• Infrastructure as code for the cluster itself, in [Terraform / Cloudera Manager templates / bootstrap scripts]

Where, When, and How Much

Work location: [fully remote, U.S. time zones [list] / [city] office [N] days weekly / [city] on site, including trips to the data center about [frequency]]. Pager: [rotation], which fired about [N] times last quarter. Base salary range: [$low] to [$high], with [bonus], [equity], and [benefits] listed separately. (Our 2026 guide puts mid-level at $134,000 to $152,000 and senior at $152,000 to $180,000 nationally. If this seat leads an upgrade or a platform move, say so in this paragraph and price toward the top.)

Req Questions We Hear on Big Data Searches

Can we ask for five years of Spark 4 experience?

Nobody has it, since Spark 4.0 shipped on May 23, 2025, so the most anyone could claim this fall is about sixteen months, and on the clusters we hear about, the move came much later.

Amazon EMR only made Spark 4.0 generally available in June 2026. Ask for years of Spark in any version, then name the specific 4.x experience you care about, such as “has run production jobs on Spark 4 with ANSI mode on.” That’s a real filter. Five years of Spark 4 is a typo waiting to embarrass somebody. I’ve seen it posted twice this year.

We still run Oozie. Does it belong in the posting?

Put it in as the migration, never as a requirement, because Apache retired Oozie in February 2025, and the engineers you want aren’t looking to be hired for Oozie expertise.

Write the count and the destination. “About 400 Oozie coordinators moving to Airflow” is a project with an end. The candidates who respond to that line have usually done a scheduler migration before, and they’ll ask good questions about how you plan to test it. Testing is where these projects stall.

What if our jobs are in Scala and every applicant writes PySpark?

For new work it’s usually fine, but keeping old Scala jobs alive through a 2.13 upgrade takes someone who reads Scala comfortably, even if they write most new code in Python.

Split the difference in the posting. Require the language new work happens in, and list “able to read and modify existing Scala” as its own line. We’ve seen that one sentence double the qualified pool on a search where “Scala required” had been choking it. It also tells candidates the truth about the codebase, which they’d find out in week one anyway. Honesty costs nothing here.

How is this posting different from a Databricks engineer posting?

A Databricks posting starts from a platform that’s already chosen, while a big data posting often has to say where the work runs today and whether that’s about to change.

That second question is the one that decides who applies. An engineer who wants to run a stable Cloudera estate for five years and one who wants to lead the move off it are both big data engineers, and both will answer the same vague posting. If the destination really is Databricks, write for that seat from the start.

The upgrade has a deadline. Contract help or a permanent hire?

Fourteen months is the window if you’re on Spark 3.5, because its security-only support runs through November 2027, and that’s usually enough to run a contract upgrade and a permanent search side by side.

The upgrade is a project with a finish line, and the people best at it have done three or four of them and would get bored running the platform afterward. Contract big data engineers fit that work well. Write into the agreement that the runbooks, the test harness, and the version notes stay in your repository. Then open the permanent search for whoever owns the platform once it lands, as a direct hire search. And when the contract engineer is plainly the one you want to keep, contract-to-hire covers that case, and it happens more often than people expect.

What should the first phone screen ask about?

Ask about versions: what they last upgraded from and to, what broke along the way, and how they proved the output matched afterward.

People who have really done it answer with version numbers and dates. Most can name the month. People who have read about it answer with the words “it depends.” Beyond that first screen, the full interview loop in our big data engineer hiring guide takes it from there, and the scale-focused interview question set we just published gives your panel material built for this seat.

A Ten-Minute Audit of Your Current Posting

Pull up the req you have now. Four checks.

• Search it for Sqoop, Sentry, Oozie, Crunch, and Tajo. If any of them shows up as a requirement, rewrite it as migration work or delete it.
• Look for a version line. If there isn’t one, ask whoever owns the cluster for the output of spark-submit --version and java -version from an edge node. It takes them a minute.
• Count the numbers. Fewer than three of the six above, and the posting is still describing an adjective.
• Check the band against the salary table above. Plenty of the bands we see were set two budgets ago and never touched since.

That’s the whole audit. Most of the postings we see fail at least two of the four, and fixing them costs an afternoon, not a new search.

If the req has already been open a while, or the upgrade date is closer than the hiring plan, talk to a KORE1 recruiter about it. KORE1’s IT staffing practice has been placing engineers since 2005, in thirty-plus metros around the country. Seventeen days is our average time to fill an IT seat, and one year out, 92 of every 100 people we have placed are still there. Big data searches run a little longer than that average. I won’t pretend otherwise. The version line is usually why, and it’s the easiest part to fix.