How it works
A Spark application has one driver and multiple executor processes. The driver builds execution plans, schedules work, and tracks progress; executors run tasks and hold cached data and shuffle files. A cluster manager, such as Kubernetes, YARN, or Spark Standalone, allocates resources, but the driver decides which application tasks to run.
Transformations are evaluated lazily. An action, such as writing a dataset or collecting results, triggers execution. For SQL and DataFrames, Spark uses the Catalyst optimizer to optimize a logical plan and choose a physical plan, with runtime execution optimizations such as code generation where supported. Adaptive Query Execution can revise parts of the SQL plan using runtime statistics.
Spark divides work into jobs, stages, and tasks. Operations that can process partitions independently are pipelined within a stage. Operations that redistribute data, such as many joins, groupings, and global sorts, introduce shuffle boundaries between stages. Within a stage, a task generally processes one partition.
A shuffle redistributes records across executors according to keys or ranges. It consumes network bandwidth and local disk, and sorting or aggregation can spill when memory is insufficient. Spark can cache reusable datasets in memory or on disk, but caching is optional, and Spark does not require the entire dataset to fit in memory.
Structured Streaming runs incremental queries using the Spark SQL engine, normally as micro-batches. Stateful operators retain information such as per-key aggregates or join state across batches. Durable checkpoints record progress and state needed for recovery; replayable sources and a compatible sink are also necessary for end-to-end correctness.
What it gives you in a design
- Distributed SQL and DataFrames
- Express filters, joins, aggregations, and transformations without manually scheduling distributed tasks. Built-in expressions give the optimizer more visibility than opaque user-defined functions.
- Large joins and aggregations
- Spark supports shuffle joins and broadcast joins, among other strategies. Broadcasting a genuinely small input can avoid shuffling a large input, but the broadcast data must fit safely in executor memory.
- Adaptive Query Execution
- For supported SQL queries, Spark can coalesce shuffle partitions, change join strategies, and split skewed join partitions using runtime statistics. This helps with imperfect estimates, but it does not eliminate the need to understand data distribution.
- Reusable intermediate datasets
- Persisting an expensive intermediate result can avoid repeated scans and computation across actions. Choose a storage level deliberately and release cached data when it is no longer useful.
- Stateful stream processing
- Structured Streaming supports event-time windows, watermarks, aggregations, and supported stream joins. The design must bound retained state and define what happens to events that arrive too late.
- Storage and format integration
- Spark reads and writes systems such as object storage, distributed filesystems, and Kafka through appropriate connectors. Columnar formats such as Parquet support column pruning and predicate pushdown; transactional table behavior comes from the table format and catalog, not Spark alone.
Scaling, replication and failure
Parallelism comes from partitions, not just executor count. Too few partitions leave cores idle and create large tasks; too many add scheduling overhead and small outputs. Input partitioning, shuffle partitioning, and output file layout are separate choices, so explain each rather than proposing one partition count for the whole pipeline.
Skew often determines completion time. A popular key can send disproportionate work to one task while other executors finish early. Depending on the operation, use partial aggregation, key salting with a later merge, broadcast joins, or adaptive skew handling. Adding workers alone does not divide an indivisible hot-key task.
Spark recovers many executor failures by retrying tasks and recomputing lost partitions from lineage. Cached data is not a durable source of truth, and lost shuffle files may require upstream work to run again. Driver failure usually requires an external restart mechanism; a Structured Streaming query can resume from a compatible durable checkpoint.
Replication and transaction guarantees primarily belong to the source, storage layer, and sink. Retried or speculative tasks can repeat external side effects, so arbitrary database calls inside tasks are not exactly-once. Use supported commit protocols, transactional table writes, or idempotent sink operations, and budget catch-up capacity so streaming lag can fall after an outage.
Limits and defaults worth knowing
Check the current documentation before relying on an exact value; many of these are configurable.
| SQL shuffle partitions | spark.sql.shuffle.partitions defaults to 200 for SQL joins and aggregations. It is configurable, and Adaptive Query Execution can coalesce post-shuffle partitions for supported queries. |
|---|---|
| Automatic broadcast join threshold | spark.sql.autoBroadcastJoinThreshold defaults to 10 MiB. It is configurable, depends on available size statistics, and is not a guarantee that the broadcast will fit safely in memory. |
| File scan partition sizing | spark.sql.files.maxPartitionBytes defaults to 128 MiB for supported file-based sources. It guides input partition planning, not output file size, and actual splits depend on file formats and other settings. |
| Task failure tolerance | spark.task.maxFailures defaults to 4 continuous failures of a particular task before Spark gives up on the job. It is configurable; increasing retries does not fix deterministic failures or unsafe side effects. |
| Default streaming trigger | In the default micro-batch mode, when no trigger interval is specified, Structured Streaming starts the next batch as soon as the previous batch completes. This provides no fixed end-to-end latency guarantee. |
Use it when
- The design needs recurring ETL or backfills over large datasets in object storage or a data lake.
- Large joins, sorts, or aggregations exceed a practical single-machine working set.
- The same team needs SQL-oriented batch processing and throughput-focused streaming transformations.
- Historical recomputation is important for correcting analytics, rebuilding aggregates, or generating training datasets.
- Results can be published to a separate database or index for low-latency application reads.
Avoid it when
- The component must serve individual API requests with tight tail-latency requirements or perform transactional record updates.
- The dataset fits comfortably in a single database or process, and distributed scheduling would add unnecessary complexity.
- The main requirement is consistently very low event-processing latency, and micro-batch scheduling or large state checkpoints would consume too much of the latency budget.
- The job mostly dispatches independent external API calls with strict per-call rate limits and side-effect handling; a durable worker queue is usually simpler.
What interviewers ask about it
Follow-ups you can expect once Spark is on your whiteboard, and the core of a strong answer.
- Where does Spark sit in the system, and how are results served?
- Place Spark between durable inputs and derived outputs, such as Kafka or object storage on the input side and an analytics table or serving database on the output side. Keep interactive requests off the Spark execution path unless a separate query-serving design meets the latency requirement.
- Why is this job slow even after adding executors?
- Inspect the execution plan and stage metrics for skew, insufficient partitions, shuffle volume, spills, and slow input reads. If one task dominates a stage, more executors will not help until the partitioning or algorithm changes.
- How do you prevent a large join from exhausting memory?
- Project and filter inputs early, verify statistics, and avoid broadcasting an input merely because its compressed file size looks small. Choose an appropriate join strategy, allow enough shuffle parallelism, and account for skew and multiple concurrent tasks sharing executor memory.
- Does Structured Streaming guarantee exactly-once results?
- It can provide exactly-once processing with replayable sources, durable checkpoints, deterministic computation, and a sink with the required commit semantics. That does not make arbitrary external writes exactly-once: foreachBatch is at-least-once by default, so use its batch identifier or another stable key to make writes idempotent.
- How do watermarks affect late events and state size?
- A watermark tracks event-time progress with an allowed lateness delay and lets supported operators eventually discard old state. Events within the configured delay are protected from being dropped as too late, while older events may or may not be processed; exact eviction and output behavior depend on the operator and output mode.
- What happens when a streaming query cannot keep up?
- Monitor source lag, batch duration, state size, and sink throughput, then identify whether the bottleneck is compute, skew, input reads, or output writes. Source-specific admission controls can limit intake per batch, but they do not create capacity, so also plan retention and enough spare throughput to catch up.
Common mistakes
- Calling Spark an in-memory database or assuming all data must fit in RAM. It is a compute engine that can scan external storage, spill to disk, and recompute partitions.
- Saying more executors always make a job faster without checking partition count, skew, shuffle costs, and source or sink bottlenecks.
- Claiming checkpointing makes all external writes exactly-once. Recovery can replay work, and arbitrary side effects still need idempotency or transactional coordination.
- Collecting a large dataset to the driver or converting it to a local dataframe without a size bound. Distributed executor capacity does not protect the driver's memory.
Alternatives
- Apache Flink
- Prefer it when the central requirement is continuous, low-latency stateful event processing with detailed event-time and state-management needs. Compare operational complexity and connector guarantees rather than choosing only by latency labels.
- Cloud data warehouse
- Prefer a managed warehouse for primarily SQL analytics when reducing cluster operations, supporting concurrent users, and using warehouse-managed tables matter more than custom distributed computation.
- Trino
- Prefer it for interactive distributed SQL over data lakes and federated sources when the main task is querying data rather than running general-purpose processing pipelines.
- Single-node SQL or dataframe engine
- Prefer tools such as DuckDB or Polars when the workload fits on one machine and parallel local execution meets the processing window. This avoids distributed scheduling, shuffle, and cluster operations.
Where it comes up
Interview questions where Spark is a natural fit, and why.
- Design a system to sort huge datasetsSpark illustrates range partitioning, shuffle, external sorting, skew handling, and recomputation after executor failures.
- Design an ad click aggregatorStructured Streaming can build event-time aggregates, while batch jobs reconcile live counts against durable event history.
- Design Google AnalyticsSpark fits large event transformations, sessionization, historical backfills, and partitioned analytics tables.
- Design a top K trending systemSpark can compute windowed popularity aggregates and merge partition-local candidates before publishing rankings to a serving store.
- Design an A/B testing platformSpark supports large exposure-to-outcome joins and repeatable experiment analysis over historical events.
- Design a metrics monitoring system like DatadogSpark can produce rollups and historical recomputations, with explicit trade-offs around aggregation state and alerting latency.
Questions people ask
What are the main components of Apache Spark architecture?
The main runtime components are the driver, executors, and cluster manager. The driver plans and schedules jobs, executors process partition-level tasks, and the cluster manager allocates resources. Storage systems remain separate from these compute components.
What is the difference between a job, stage, and task in Spark?
An action triggers execution that can involve one or more jobs. Each job is divided into stages, generally separated by shuffle boundaries. A stage runs tasks in parallel, with each task generally handling one partition.
What is the difference between Apache Spark and Hadoop?
Spark is a distributed processing engine, while Hadoop commonly refers to an ecosystem that includes HDFS, YARN, and MapReduce. Compared with MapReduce, Spark can pipeline operations within stages and reuse cached intermediate data across computations. Spark can run on YARN and read HDFS, but it does not require either.