hadoop 2.x-HDFS short circuit read

  this feature is a considerable performance improvement,as per the idea 'moves the computation to data side',this is a further usage of it.

 

Outline

 1.what is 

 2.how to 

 3.setup to run

 4.one more thinking

 

1.what is

  about ten years ago,i usually heard some ad words,maybe belong  of intell,"the technology of moving computation".of course ,hadoop,uses it as one of the key features to advance hdfs reads:read on local node.u will see some info like "Launched local maps" from the results of issuing a job.

  or,maybe u will occur similar scenarios like rebuilding indexes from hbase by MR,and u want to index to local host directly by local fs without crossing the solr webserver if the maps are co-located in the same node with solr .sure,if u adjust the map slots to one per node,this will run fine,else that will cause multi-threads competition.

  in the opposite direction of data stream with front case,the so called 'short circuit reads' is issued by mr mappers,and the datanode lied in same host is hardly to do  nothing:maybe only pass file descirptiors etc,and most part of reading files are based on client to act on,so this maybe say that 'reading data without crossing datanode'.so something costs like threads usage,tcp sockets are eliminated by client to increase performance.

  

 2.how to 

  through comaring the three modes of reading data ,then will get a clear impress

mode	target version	data-flow	cache	other features	safe protection on local files
common tcp	before 1.0 or before 0.23.1	local datanode->client	-	hdfs build-in before 1.x	y
hdfs-2246(short circuit reads )	1.x,0.23.1+	read local directly without dn	block path	complex to setup, through changing user attributes and data dir permissions; pass file offset ,length etc to client,then the later reas file using them	n,but can be fixed by setuping some properties
hdfs-347(short circuit reads with secure)	2.10-beta	read local directly without dn	file descriptor (higheffect)	simple to setup; using system domain socket to pass fd[1]	y,passing fd only without other fils needed not

 

 

 3.setup to run

  here are some properties in hdfs-site.xml from setuping a hdfs-347 configs,check it here for hdfs-2246

<!--short circuit read -->
<property>
    <name>dfs.client.read.shortcircuit</name>
    <value>true</value>
</property>
<property>
    <name>dfs.domain.socket.path</name>
    <value>/usr/local/hadoop/data-2.5.1/dfs_dn_socket</value>
</property>

    so the last hdfs-347 is more effective and safe,from my test,this is the result from reading 1 GB file from hdfs

 

time hadoop fs -get /user/tmp.dd 2

  tcp:33s

 

  hdfs-347:31s

  yes,this is not obvious,as the bottleneck of the test is stuck on suboptimal hard disk.but u see, the difference between them is also reasonable.

 

 

 4.one more thinking

  in according to current common hard-disks(not ssd),if enable short circuit reads,may be broken the io trade off if some other services are deployed in same cluster since this feature will swallow nearly all the io resouces?maybe 

 

 

 

Ref:

[1]http://troydhanson.github.io/misc/Unix_domain_sockets.html

HDFS Short-Circuit Local Reads

How Improved Short-Circuit Local Reads Bring Better Performance and Security to Hadoop

jira of this topic