Big Data Engineer Interview Questions 2026

Last updated: September 24, 2026

By Jennifer Burdick, Recruiting Manager, KORE1

Big data engineer interview questions worth asking test the physics of scale, such as where a shuffle’s partition count came from, what fills a NameNode’s memory, and why a Kafka consumer group stalls on every deploy. Ask for definitions of the four Vs or lazy evaluation during a phone screen if you must, and then move on. The engineer worth hiring answers with numbers from a cluster they actually ran, and usually with a bad night attached.

In January a consumer-goods supplier in Bentonville, Arkansas, hired a senior engineer to build streaming sales feeds for its largest retail account. He interviewed beautifully. He could define the four Vs, explain lazy evaluation, sketch the Lambda architecture from memory, and tell you exactly why a DataFrame beats an RDD. Three weeks after his start date, the company’s on-premises Hadoop cluster stopped accepting writes on a Saturday morning. Of course it was Saturday.

Nothing was broken, exactly. His Spark Structured Streaming job wrote a new file every thirty seconds for each of about three hundred store partitions, which works out to roughly 864,000 files a day, most of them a few kilobytes. After three weeks the NameNode was tracking more than eighteen million extra files and had run out of heap. Getting the cluster back, compacting the mess, and paying an outside Hadoop consultant for the weekend cost them a little over $41,000. The data itself was fine the whole time.

Every question he had been asked has a printed answer on a dozen websites. Not one of them asked how many files his job would leave behind.

That is most of what is wrong with the big data engineer interview questions you can find online. They were written for the candidate, to be memorized, and they test whether someone has read about distributed systems. The job is running them. Those are different skills, and the space between them is where the Bentonville weekend came from.

Data platform searches have crossed my desk at KORE1 since Hadoop clusters were something companies bragged about at conferences. Most of the big data searches that reach us now come in through our big data engineer staffing practice after an interview process like the one above has already produced a hire, and the hire has already produced an incident.

One thing to know before you read further. We earn a fee when a client hires an engineer we introduced, so I have a stake in how your search goes. The questions below work the same either way, and the Bentonville team used them to hire the next engineer on their own, from a referral their platform lead brought in. We did not bill for that one. It went well anyway.

Some ground is already covered elsewhere on our site, so it is not repeated here. Data skew, salting, and broadcast joins already have their own question in our Databricks engineer interview questions. SQL, modeling, and pipeline design are in the general data engineer interview questions. How to structure the loop is in our guide to hiring a big data engineer, and what belongs in the posting itself is in our big data engineer job description guide. What follows is the part of the interview that only exists once the data no longer fits on one machine.

Nobody Pages You at 3 A.M. About the Four Vs

The usual list goes something like this. Name the four Vs of big data. What is commodity hardware? What does a NameNode do, and what does a DataNode do? RDD or DataFrame? Transformations versus actions? Draw the Lambda architecture. Each one is fair to ask a student. Each one can be answered perfectly by someone who has never been responsible for a cluster at two in the morning, which is the only situation where the answer matters.

Scale itself stopped being exotic a long time ago. When Jeffrey Dean and Sanjay Ghemawat described MapReduce at OSDI in 2004, their paper said a typical computation processed many terabytes of data on thousands of machines. That was twenty-two years ago. Terabytes are ordinary now. A regional insurer has them. So does a grocery distributor.

What has not changed is that the hard parts of the job are physical. Memory on one node. The number of files a metadata server has to remember. Bytes crossing the network during a shuffle. The state a streaming job has to carry from one hour into the next. A data engineer interview checks whether someone can model data and build a pipeline that produces the right answer, and it should. A big data engineer interview has to find out whether the candidate knows where each of those resources runs out, what it looks like when it does, and what it costs.

Resumes will not tell you. Everybody who has touched Spark lists Spark.

So every question below asks for a number, a setting, or a date from the candidate’s own work. Anyone can look up what a NameNode does. Nobody can look up how many files were in their own biggest table last spring.

Seven Questions That Only Make Sense Above a Terabyte

No trick questions here. Each one starts from something a production cluster actually did, and each has a weak answer that sounds perfectly reasonable to a panel that has never been paged. Listen hardest to the second sentence of every answer. The first sentence is usually a definition.

Your last big shuffle ran with 200 partitions. Who picked that number?

Usually nobody did.

Spark’s performance tuning documentation lists 200 as the default for spark.sql.shuffle.partitions, the number of partitions used when data is shuffled for a join or an aggregation. On a 50 GB shuffle that is about 250 MB per task. Fine. On a 3 TB shuffle it is about 15 GB per task. Not fine at all. The job spills to disk, slows to a crawl, and eventually fails in a way that looks like a memory problem to anyone reading the error instead of the stage.

Listen for the word default. A strong candidate knows the 200 is one before you finish asking. Then they bring up adaptive query execution, which the same documentation says has been on by default since Spark 3.2.0 and which merges small shuffle partitions at runtime toward an advisory size of 64 MB. The better ones add the catch without being prompted. Coalescing only merges. It cannot turn 200 oversized partitions into 2,000 sensible ones, so they set a high initial partition count and let Spark shrink it.

The weak answer is “Spark handles that now.” Half true. That half is the problem.

This one takes four minutes, and in my experience it separates a room of finalists more cleanly than the other six.

YARN killed your executor for exceeding memory limits, and the heap graph looked fine. Where did the memory go?

This is the Hadoop-era version of a lesson every PySpark shop learns eventually. I learned the arithmetic from a candidate, honestly. On YARN each executor runs inside a container with a hard memory ceiling. The heap is only part of what counts against it. Spark’s configuration reference adds a separate overhead allowance, plus anything off-heap or reserved for PySpark, and by default that overhead is a tenth of the heap with a small floor. Give an executor 16 GB and it gets roughly 1.6 GB of overhead. That overhead is where native allocations and interned strings live, and unless somebody set PySpark memory separately, it is also where the Python worker processes have to fit. A pandas UDF that pulls a fat batch of rows into Python can blow straight through 10 percent while the JVM heap never looks stressed. YARN sees a container over its limit and kills it. The heap never knew. The Spark UI shows a lost executor and a retried task, and the heap chart stays calm the entire time, which is exactly why so many people get this wrong.

Candidates who have lived through it name the overhead setting, ask whether the job runs Python UDFs, and talk about Arrow batch sizes or moving the logic back into Spark SQL where it runs inside the JVM. Candidates who have not lived through it raise spark.executor.memory. That makes the heap bigger. It does nothing for Python, and on a shared cluster it also means fewer executors fit on each node, so the job gets slower on top of still failing.

How many files are in your biggest table, and what is the average size?

This is the Bentonville question.

A 2009 Cloudera engineering post on the small files problem gave the rule of thumb that still gets repeated in most Hadoop shops. Every file, directory, and block in HDFS is an object in the NameNode’s memory at roughly 150 bytes each, so ten million single-block files cost about 3 GB of heap. The HDFS design document confirms the NameNode keeps the entire namespace and block map in memory, and it describes a typical block as 128 MB. Put those side by side. A four-kilobyte file costs the NameNode as much memory as a file that fills an entire 128 MB block.

Object storage moved the pain rather than removing it. On S3 or Azure Data Lake Storage there is no NameNode to exhaust, but millions of tiny files still mean millions of requests, slower query planning, and a table format spending its evenings on compaction jobs that somebody has to pay for.

Strong candidates know their numbers. Roughly, anyway. “About forty thousand files, averaging a couple hundred megabytes, compacted every night” is a real answer. So is “far too many, and here is how I found out.” Then ask which settings in a streaming job decide how many files it writes. The trigger interval and the output partitioning are the two they should name without help. To be fair to the engineer in Bentonville, he could have answered every part of this after his Saturday. Nobody asked him before it.

You moved a job from Spark 2 to Spark 3 and some old dates shifted. What happened?

A Hartford, Connecticut, insurer that writes commercial property found this one in March. Its policy history lived in Hive tables written years earlier by Spark 2.4, and records with no known effective date carried a placeholder of January 1 in the year 1. During the upgrade, the first run on Spark 3 stopped with an upgrade exception about ancient dates. An engineer with a deadline set the Parquet rebase mode to CORRECTED. The error went away. The job finished. The placeholder dates now came back two days early. Nobody noticed. A downstream filter that looked for the exact placeholder matched nothing, and about 11,400 active policies dropped out of a reserving feed for four business days before an actuary asked where they had gone. She was not pleased.

The cause is documented. Spark’s SQL migration guide explains that Spark 3.0 switched from the hybrid Julian and Gregorian calendar to the Proleptic Gregorian calendar, and that the change affects dates before October 15, 1582. Files that Spark 2 wrote need LEGACY rebasing when Spark 3 reads them. CORRECTED reads the stored value exactly as written, which is right only for files Spark 3 wrote itself.

I like this question because it is not really about Spark. It asks whether the candidate reads a migration guide before an upgrade, and whether they treat a silenced error as a finding or as a fix. It keeps coming back, too. Spark 4.0 dropped the old spark.sql.legacy names for these rebase settings, so any shop still carrying Hadoop-era Parquet will meet the same problem again under new configuration names.

Consumer lag on your main topic jumps for a minute or two after every deploy. Why?

A card processor in Sioux Falls, South Dakota, ran fraud scoring off a 64-partition Kafka topic with twelve consumer instances behind it. Rolling deploys restarted the instances one at a time, and each restart set off a consumer group rebalance. Under the older eager protocol, every consumer in the group stops, gives up its partitions, and waits for a new assignment. Twelve restarts meant twelve stalls in a row. Scoring fell several minutes behind on every release, and the team released three times a day. Thirty-six stalls a day.

The engineer they eventually hired answered in about twenty seconds. Give each instance a stable identity with static membership, which came from KIP-345, so a restart inside the session timeout does not trigger a rebalance at all. Use cooperative incremental rebalancing from KIP-429 so the consumers that are not moving keep working while partitions change hands. Then she brought up the newer option on her own. Kafka 4.0, released on March 18, 2025, made the next-generation consumer rebalance protocol from KIP-848 generally available, and the announcement says goodbye to stop-the-world rebalances by name. Clients opt in with group.protocol=consumer.

Weak answers blame the deploy tooling, or suggest deploying less often. One finalist suggested adding partitions. That only gives the rebalance more to move.

Your Flink checkpoints now take longer than the checkpoint interval. What do you look at first?

Flink’s guide to tuning large state describes what happens next. When a checkpoint takes longer than the interval, the next one waits for it. Checkpoints stretch out, recovery after a failure means replaying more data, and the job turns fragile in a way nobody notices until the day it restarts.

The strong answer starts with a question back. Is the state growing, or is the job under backpressure? Those have different fixes. Growing state usually means keyed state with no time-to-live, deduplication sets and session windows that were supposed to expire and never did, and a look at state size over the past month settles it. Backpressure means a slow operator or a slow sink. Incremental checkpoints on RocksDB help with the first problem, because they record only what changed since the last completed checkpoint. Unaligned checkpoints help barriers move under backpressure, and the Flink documentation is blunt that they do not fix whatever is causing the backpressure in the first place. A candidate who says that unprompted has read the docs and has also been on call.

Weak candidates raise the interval. Sometimes that is even correct. It is never the first thing to try.

Tell me about a job you made slower on purpose.

Every cluster has neighbors. This question is about them, and they are easy to forget when the cluster is the part you own.

A grocery distributor in Louisville, Kentucky, pulled order history out of its production Oracle ERP every morning with a Spark JDBC read. Someone had set numPartitions to 200 to make the extract faster. It was faster. Spark’s JDBC data source documentation notes that the same option also sets the maximum number of concurrent JDBC connections, so every morning at six the extract opened up to 200 sessions against the database the warehouse used to take orders. Order entry dragged through the first shift, and for months everyone blamed the ERP vendor. Wrong suspect.

The engineer who fixed it cut the read to sixteen partitions, pointed it at a read replica, and added an hour to the extract. Nobody downstream noticed the extra hour. The warehouse noticed the fix the first morning.

Good answers to this question sound like that. A throttle on an API sink. Speculative execution switched back off for a job whose sink cannot tolerate a duplicate write. Spark ships with it off. Someone had turned it on. A lower parallelism to stay under a vendor’s rate limit. Candidates who cannot think of a single example have usually treated the cluster as the whole world.

What Each Question Is Actually Measuring

I hand this table to clients to keep beside their interview notes. The second column is what you are really testing. The question itself is just the way in.

Patterns show up fast. Strong on the first three rows and blank on the fifth and sixth almost always means a batch engineer. That is not a flaw. It is a different hire, and I have watched a streaming platform end up owned by someone learning Flink on the job because a panel assumed one person could cover both. When a team genuinely needs both at the senior level, we usually run them as two senior data engineer searches rather than one.

A Napkin, Three Table Sizes, and No Laptop

The last round I recommend takes twenty minutes and needs nothing but paper.

Hand the candidate a scenario. Nine terabytes of compressed Parquet click events land every day. They need to be joined to a 40 GB customer table and a 300 MB product table, then rolled up into daily sales by store and product. No laptop, no documentation, no rush. Then ask for four things.

• Roughly how many tasks the first stage will run. Spark’s default maximum partition size for file reads is 128 MB, which puts the answer somewhere around 70,000, and anything in that neighborhood with the reasoning shown is a pass.
• Which of the two joins moves more data across the network, and whether the product table should be broadcast, given that 300 MB is thirty times Spark’s 10 MB default broadcast threshold.
• What one run costs, in dollars, at their last company’s prices.
• What they would measure before changing anything.

You are not grading arithmetic. You are grading whether they think in units. The people worth hiring turn terabytes into tasks, tasks into minutes, and minutes into dollars without being asked, and the cost question is where the field splits in two. Engineers who have owned a platform budget can tell you what an hour of their old cluster cost. Engineers who have only ever submitted jobs to someone else’s cluster usually cannot, and that is a fair thing to learn before you hand them yours.

One candidate this spring got the task count right to within a few thousand and then asked whether the click data arrived sorted by store. It did not. She pointed out that sorting it once at ingestion would make the nightly rollup cheaper forever, and the client made her an offer the following week.

Pay is its own conversation. Bands by level and city are in KORE1’s big data engineer pay guide, and the salary benchmark assistant gives a quick first read for a single metro. The rest of the bill, fees and ramp time included, is laid out in what a data engineer costs to hire. What these questions change is how confidently you can pay a strong candidate what they are worth, because you will know what you are buying.

What Came Up After the Candidate Left

Do the “four Vs” questions from prep sites tell us anything at all?

Very little, since any candidate can memorize volume, velocity, variety, and veracity in an evening, and knowing the words says nothing about whether they have kept a real cluster alive.

Use them as a vocabulary check on a phone screen if you like. Skip them on the panel. The same goes for definitions of commodity hardware, transformations versus actions, and the Lambda architecture diagram. A good engineer knows all of it and will be mildly bored.

We already run a data engineer loop. What should change for the big data seat?

Keep the SQL and modeling rounds short, and replace the pipeline design round with two or three of the scale questions above plus the napkin exercise.

A general loop tests correctness and modeling well. It was never built to find out whether someone understands executor memory or consumer group behavior, and those are the things that wake a big data engineer up at night. Many seats are both. If yours is a hybrid, say so in the requisition and weight the rounds to match. Our data engineer recruiters run both kinds of search and can tell you which one your req actually describes.

The candidate’s biggest Spark job read 200 GB. Does that count as big data experience?

Sometimes, because the size of the job matters less than whether the candidate hit and fixed the limits that come with scale, such as spills, file counts, or memory kills.

Two hundred gigabytes processed carelessly on a huge cluster teaches very little. Two hundred gigabytes on a tight, shared YARN queue with a four-hour window teaches a lot. Ask what broke. Someone who has never had anything break at their scale has not really been at scale yet, whatever the resume says.

Our platform is all Databricks now. Should the Hadoop questions stay in?

Keep the ones about files, memory, and old data, since those problems followed teams onto the cloud even though the NameNode stayed behind.

Small files still slow down a Delta or Iceberg table and still cost money in requests. Parquet written by Spark 2 is still sitting in a lot of object storage. Drop the YARN wording and ask about the equivalent on your platform instead. The physics did not move. Shops on a warehouse rather than a lakehouse will find the sizing and cost version in our Snowflake engineer interview guide.

Can one set of questions cover a batch engineer and a streaming engineer?

Partly, since the file and memory questions apply to both, but streaming candidates need the rebalance and checkpoint questions, and a batch candidate can reasonably be weak on them.

Decide which one you are hiring before the first interview. Not after. A strong batch engineer can learn Flink. It takes months rather than weeks, and senior streaming experience is priced accordingly.

What stops a candidate from reading answers off a second screen?

The questions themselves, mostly, because a chatbot can explain the shuffle partition setting but cannot tell a candidate how many files were in their own biggest table last year.

Follow-ups do the rest. Ask what the number was, what they changed, and what happened the week after. Real answers get more specific with every follow-up. Borrowed ones get more general.

Ask Who Picked the Number

The Bentonville supplier kept its streaming feed, for the record. The engineer they hired in the spring runs compaction every night and moved the trigger to five minutes, and the retail account never noticed the difference. She now asks the file-count question in every interview she sits in on, including for analyst roles, which I think is overdoing it a little. Only a little. She has a point.

The difference shows up in the second half of every answer. Somebody who has run the cluster names the setting, the date, and the dollar figure. Somebody who has read about it names the concept and stops there.

If you would rather have someone else run the first rounds, talk to our data recruiting team. Migrations and fixed builds usually go through our contract staffing desk, and a Hadoop exit with a hard deadline sometimes needs a small team through data engineering staff augmentation instead of one person. The engineer who will own the platform for the next five years is a direct hire search, or contract-to-hire when you want to watch someone run your cluster for a quarter first. We have run both kinds since 2005, and ninety-two percent of the engineers we place make it past their first anniversary with the client. Start by asking who picked the 200.