Question - Define HDFS and YARN, and talk about their respective components.
Answer -
Now that we’re in the zone of Hadoop, the next Big Data interview question you might face will revolve around the same.
The HDFS is Hadoop’s default storage unit and is responsible for storing different types of data in a distributed environment.
HDFS has the following two components:
- NameNode – This is the master node that has the metadata information for all the data blocks in the HDFS.
- DataNode – These are the nodes that act as slave nodes and are responsible for storing the data.
- YARN, short for Yet Another Resource Negotiator, is responsible for managing resources and providing an execution environment for the said processes.
- The two main components of YARN are –
- ResourceManager – Responsible for allocating resources to respective NodeManagers based on the needs.
- NodeManager – Executes tasks on every DataNode.