Components failure and recovery mechanisms

Hadoop is designed to be a fault-tolerant system. Hadoop keeps track of active nodes with the help of heartbeat message. When the Namenode does not receives heartbeat messages from a datanode, it treats it as dead or unavailable.

If any or some of the datanodes becomes dead,  hdfs may still be able to serve the requests due to replication of files which are already distributed across the cluster. So hadoop is considered a fault-tolerant system in partial-failure.

If datanode(s) becomes available again the Namenode checks the disk capacity & storage to replicate and distribute the files on new added nodes to ensure desired replication of files evenly distributed across all nodes of a cluster. You can also run the rebalancer command explicitly to ensure evenly distribution of data blocks over the cluster. In small / single node cluster the rebalancer work is not very crucial.

If Namenode goes down, then secondary Namenode plays a vital role in hadoop cluster. But in real, it is not a backup node, rather it’s a checkpointing node. So, if Namenode fails, it needs to restart wherein it will get the checkpointed fsimage.

fsimage is the state of filesystem metadata information when Namenode starts. All the changes made after starting Namenode are kept in editlogs.

We can look at the fsimage and editlogs residing inside current directory of Namenode as shown in the snapshot below. As per our cluster configuration the absolute path of Namenode location is

/usr/local/hadoop_store/hdfs/nameode

Secondary Namenode keeps the fsimage updated by adding editlogs on regular intervals and gives this back to Namenode in case of failure which works as a quick recovery point for the cluster and reduces Namenode’s restart time.