How it works
HDFS separates metadata from file contents. The NameNode manages directories, file permissions, file-to-block mappings, and replica placement. DataNodes store the actual blocks on local disks, so file bytes do not pass through the NameNode.
The NameNode keeps namespace metadata in memory and persists it through an FsImage checkpoint plus an edit log. DataNodes send heartbeats and block reports, allowing the NameNode to track live workers and block locations. Namespace checkpoints do not replace replication or high availability.
For a read, the client asks the NameNode for block locations, then reads directly from a nearby DataNode. The client verifies checksums and can switch to another replica if a disk, node, or network path fails. Processing frameworks can use block locations to schedule work near the data.
For a replicated write, the NameNode chooses DataNodes and the client streams packets through a replication pipeline. Each DataNode forwards packets downstream, and acknowledgments travel back through the pipeline. HDFS permits one active writer per file and supports append rather than arbitrary in-place updates; packet acknowledgment alone should not be described as a physical-disk sync guarantee.
In a common high-availability deployment, active and standby NameNodes share namespace edits through a quorum of JournalNodes. The standby follows those edits and receives DataNode reports so it can take over. Automatic failover commonly uses ZooKeeper-based coordination, with fencing to prevent an old active from continuing to act as the metadata authority.
What it gives you in a design
- Large sequential file access
- Large blocks reduce metadata overhead and support long streaming reads and writes. HDFS works well for scans, distributed sorting, and batch ETL, not frequent tiny random operations.
- Rack-aware replication
- Placement policies distribute replicas across nodes and racks to reduce correlated failure risk. State the failure domain explicitly: three copies on one rack do not provide the same protection as copies across racks.
- Data locality
- Compute engines can place tasks on or near machines holding input blocks. This reduces network traffic, although locality matters less when compute and storage are deliberately separated.
- Checksums and automatic repair
- Checksums detect corrupted data, while heartbeats and block reports help identify missing replicas. The NameNode schedules replacement copies from healthy replicas.
- Erasure-coded storage
- Supported HDFS versions can use erasure coding to reduce storage overhead compared with three-way replication. Reconstruction adds CPU and network work, making it more attractive for large, colder datasets than latency-sensitive data.
- Namespace operations and snapshots
- Within a namespace, rename provides an atomic metadata operation that batch jobs can use to publish completed output. Directory snapshots preserve point-in-time views, but they are not an independent disaster-recovery copy.
Scaling, replication and failure
File contents scale across DataNodes, but metadata remains a separate bottleneck. Every file, directory, and block requires NameNode state, so millions of tiny files can exhaust metadata capacity while disk space remains available. Compact small records into larger files and choose formats suited to the query workload.
HDFS Federation scales metadata through multiple independent namespaces and NameNodes sharing a DataNode fleet. It does not automatically turn one namespace into a transparently sharded metadata service. High availability protects a namespace from NameNode failure; federation addresses a different problem.
Completed files have coherent filesystem semantics rather than an eventually consistent object listing model. For a file still being written, distinguish data accepted into the pipeline, data made visible through hflush, and durability requested through hsync. On client failure, lease recovery is needed before another writer can safely take over.
DataNode failures trigger re-replication, consuming network bandwidth and disk I/O while the cluster is already degraded. Losing enough replicas makes a block unavailable, and losing every valid copy loses its data. Metadata operations require the active NameNode; cached block locations can allow some existing reads to continue during a metadata outage.
Limits and defaults worth knowing
Check the current documentation before relying on an exact value; many of these are configurable.
| Default block size | Modern Apache Hadoop releases commonly default to 128 MiB per block through dfs.blocksize. It is configurable, and a short final block does not reserve a full block of disk space. |
|---|---|
| Default replication factor | The default replication factor is 3 through dfs.replication. It is configurable, and the target factor is not a promise that every block always has that many healthy replicas. |
| Writer concurrency | A file has one active writer at a time. HDFS supports append but does not provide arbitrary concurrent in-place writes. |
| Common erasure-coding policy | RS-6-3-1024k uses 6 data units and 3 parity units with 1 MiB cells. A full stripe has 1.5 times raw storage overhead and can tolerate up to 3 missing internal blocks; deployments need enough DataNodes for placement. |
Use it when
- The workload consists mainly of large sequential scans, batch transformations, or distributed external sorting.
- The design already includes a Hadoop-compatible compute cluster and benefits from scheduling work near stored data.
- Files are written once or appended by one writer, then read many times.
- You need distributed storage for large analytical datasets and can operate the storage cluster and its metadata services.
Avoid it when
- The application needs low-latency point reads, frequent row updates, or transactions across mutable records.
- The dataset consists mostly of small files that cannot be compacted.
- You need concurrent writers updating arbitrary offsets within the same file.
- A cloud-native design benefits more from managed object storage, independent compute scaling, and reduced storage operations.
What interviewers ask about it
Follow-ups you can expect once HDFS is on your whiteboard, and the core of a strong answer.
- What happens when the NameNode fails?
- Without high availability, namespace operations stop until the NameNode is recovered, even though DataNodes still hold file contents. With HA, a sufficiently caught-up standby can become active using shared edits and failover coordination. Fencing is essential to prevent two active metadata authorities.
- Does an acknowledged write mean three durable replicas exist?
- Not necessarily: acknowledgment reflects the current write pipeline, and failures can leave fewer replicas than the configured target. Also distinguish packet acknowledgment from hsync durability semantics. Explain the required failure tolerance and recovery behavior rather than treating the replication setting as an unconditional write guarantee.
- Why are small files a problem?
- Each file and block consumes NameNode metadata and creates extra lookup, open, and scheduling work. A small file does not consume an entire block of physical storage, so the main problem is metadata and per-file overhead. Compaction into larger files usually helps more than merely increasing block size.
- How does HDFS recover from a DataNode failure?
- After failure detection, the NameNode identifies under-replicated blocks and schedules copies from surviving replicas to other DataNodes. Reads can retry against healthy replicas while repair runs. Recovery needs spare capacity and bandwidth, and it cannot recreate a block if every valid copy is gone.
- Why choose replication instead of erasure coding?
- Replication provides simpler reads and recovery, at a higher storage cost. Erasure coding reduces storage overhead but requires reconstruction work when pieces are missing and has placement constraints. Choose based on data temperature, recovery traffic, file size, and supported write operations.
- How is HDFS related to Google File System?
- HDFS follows a similar architecture to the published Google File System design: centralized metadata, large distributed blocks or chunks, and direct client access to storage workers. They are separate systems with different APIs and write semantics. In particular, GFS atomic record append should not be attributed to HDFS.
Common mistakes
- Calling the Secondary NameNode a hot standby. Its traditional role is checkpointing; it is not the HA standby NameNode.
- Treating NameNode HA as metadata sharding, or assuming adding DataNodes removes the namespace memory bottleneck.
- Claiming a 1 KiB file consumes a full 128 MiB block on disk. Small files primarily hurt metadata capacity and per-file efficiency.
- Using HDFS as a database or queue without addressing random access, indexing, concurrent updates, and application-level processing guarantees.
Alternatives
- Object storage
- Wins for cloud data lakes, durable bulk objects, and independent scaling of compute and storage. Check the chosen service's consistency and publication semantics rather than assuming HDFS rename behavior applies.
- Apache HBase
- Wins when a Hadoop-oriented system needs indexed row-key access and mutable records. It adds a database serving layer and commonly uses HDFS underneath.
- CephFS
- Wins when applications need a more general-purpose shared filesystem with POSIX-style access rather than HDFS's batch-oriented, single-writer file model.
- Apache Kafka
- Wins for continuously arriving ordered events with consumer offsets and replay. HDFS can retain compacted analytical output, but it does not replace the event-log interface.
Where it comes up
Interview questions where HDFS is a natural fit, and why.
- Design a system to sort huge datasetsHDFS can hold large input datasets and durable sort output while workers use block locations to improve read locality.
- Design Google AnalyticsHDFS can store compacted event files for batch sessionization, historical queries, and recomputation.
- Design a web crawlerHDFS can store large crawl archives and feed batch parsing or indexing pipelines, while URL-frontier state remains in a separate serving system.
- Design an ad click aggregatorHDFS can retain raw click events for batch reconciliation and recomputation independently of the low-latency aggregation path.
Questions people ask
What is HDFS architecture?
HDFS uses a NameNode for filesystem metadata and DataNodes for file blocks. Clients contact the NameNode to locate blocks, then transfer data directly with DataNodes. Replication or erasure coding protects file contents, while NameNode HA protects metadata-service availability.
What is the difference between HDFS and Google File System?
HDFS is an Apache open-source distributed filesystem; Google File System is the system described in Google's original GFS paper. Both separate centralized metadata from distributed file contents, but they are not interchangeable implementations. The original GFS paper describes 64 MiB chunks and atomic record append, while modern HDFS commonly defaults to 128 MiB blocks and one active writer per file.
Is HDFS still useful with cloud object storage?
Yes, particularly in established Hadoop deployments or environments where operating a storage cluster and using data locality make sense. For many new cloud analytics systems, object storage is simpler to operate and lets compute scale independently. Compare access semantics, operational cost, and recovery behavior rather than choosing solely by throughput.