hadoop源码阅读-shell启动流程

open the bin/hadoop file,you will see the there is a config file to load:

  either libexec/hadoop-config.sh or bin/hadoop-config.sh

and the previor is loaded if exists,else the load the later.

 

you will see the HADOOP_HOME is same as HADOOP_PREFIX at last:

export HADOOP_HOME=${HADOOP_PREFIX}

 

ok,now start to have a grance of shell-starting flow of distributed-mode:

  namenode format -> start-dfs -> start-mapred

 

step 1-namenode format

 appropriate cmd is: "hadoop namenode -format",and the related class entry is:

 org.apache.hadoop.hdfs.server.namenode.NameNode

 

well ,what is NameNode(NN) responsible for? description copied from code:

 * NameNode serves as both directory namespace manager and
 * "inode table " for the Hadoop DFS.  There is a single NameNode
 * running in any DFS deployment.  (Well, except when there
 * is a second backup/failover NameNode.)
 *
 * The NameNode controls two critical tables:
 *   1)  filename->blocksequence (namespace )
 *   2)  block->machinelist ("inodes ")
 *
 * The first table is stored on disk and is very precious.
 * The second table is rebuilt every time the NameNode comes
 * up.
 *
 * 'NameNode' refers to both this class as well as the 'NameNode server'.
 * The 'FSNamesystem' class actually performs most of the filesystem
 * management.  The majority of the 'NameNode' class itself is concerned
 * with exposing the IPC interface and the http server to the outside world,
 * plus some configuration management.
 *
 * NameNode implements the ClientProtocol interface, which allows
 * clients to ask for DFS services.  ClientProtocol is not
 * designed for direct use by authors of DFS client code.  End-users
 * should instead use the org.apache.nutch.hadoop.fs.FileSystem class.
 *
 * NameNode also implements the DatanodeProtocol interface, used by
 * DataNode programs that actually store DFS data blocks.  These
 * methods are invoked repeatedly and automatically by all the
 * DataNodes in a DFS deployment.
 *
 * NameNode also implements the NamenodeProtocol interface, used by
 * secondary namenodes or rebalancing processes to get partial namenode's
 * state, for example partial blocksMap etc.

 

 

the formated files list are here:

hadoop@leibnitz-laptop:/cc$ ll data/hadoop/hadoop-1.0.1/cluster-hadoop/mapred/local/

hadoop@leibnitz-laptop:/cc$ ll data/hadoop/hadoop-1.0.1/cluster-hadoop/dfs/name/current/
-rw-r--r-- 1 hadoop hadoop    4 2012-05-01 15:41 edits
-rw-r--r-- 1 hadoop hadoop 2474 2012-05-01 15:41 fsimage
-rw-r--r-- 1 hadoop hadoop    8 2012-05-01 15:41 fstime
-rw-r--r-- 1 hadoop hadoop  100 2012-05-01 15:41 VERSION

 

 

hadoop@leibnitz-laptop:/cc$ ll data/hadoop/hadoop-1.0.1/cluster-hadoop/dfs/name/image/
-rw-r--r-- 1 hadoop hadoop  157 2012-05-01 15:41 fsimage

 

 

ok.let's to see what does these files to keep.

edits: FSEditLog maintains a log of the namespace modifications .(same as transactional logs)

(these files belong to FSImage listed below)

fsimage : FSImage handles checkpointing and logging of the namespace edits .

fstime : keep last checkpoint time

VERSION: File VERSION contains the following fields:

1. node type
2. layout version
3. namespaceID
4. fs state creation time
5. other fields specific for this node type

   The version file is always written last during storage directory updates. The existence of the version file indicates that     all other files have been successfully written in the storage directory, the storage is valid and does not need to be     recovered.

 

a dir named 'previous.checkpoint ' wil be occured when :

     * previous.checkpoint is a directory, which holds the previous
     * (before the last save) state of the storage directory .
     * The directory is created as a reference only, it does not play role
     * in state recovery procedures, and is recycled automatically,
     * but it may be useful for manual recovery of a stale state of the system.

content like this:
hadoop@leibnitz-laptop:/cc$ ll data/hadoop/hadoop-1.0.1/cluster-hadoop/dfs/name/previous.checkpoint/
-rw-r--r-- 1 hadoop hadoop  293 2012-04-25 02:26 edits
-rw-r--r-- 1 hadoop hadoop 2934 2012-04-25 02:26 fsimage
-rw-r--r-- 1 hadoop hadoop    8 2012-04-25 02:26 fstime
-rw-r--r-- 1 hadoop hadoop  100 2012-04-25 02:26 VERSION

 

 

yes, i found a import class named "Lease" which will do as:

A Lease governs all the locks held by a single client.
   * For each client there's a corresponding lease , whose
   * timestamp is updated when the client periodically
   * checks in.  If the client dies and allows its lease to
   * expire, all the corresponding locks can be released.