1. submitting a MR job
2.flow from nodemanager to resourcemanager:
so from this figure,we know that the yarn use WritableRPCEnginge.java as rpc engine by default,but here of course,hadoop use ProtobufRpcEngine.java instead ,and the serialization/deserialization of parameters are used the ProtocolBuffer protocol.
for more detail,check in YARNServiceProto.java to see how does yarn transition a common param to PB's one.
some important classes:
ApplicationClientProtocolPBClientImpl--this is the client proxy to remote ResourceManager
RpcClientFactoryPBImpl--supplys which client proxy implemention of certain protocols ,eg. ApplicationClientProtocol.that is it determines which class to react to the appropriate api protocol.for example transition package path to impl.pb.client,convert protocol class with subfix PBClientImpl etc.
HadoopYarnProtoRPC--provide rpc proxy to remote server,so the above proxy is generated by this.
DefaultFailoverProxyProvider--failover provider when the retry proxy provider warrants failure over.
RetryInvocationHandler--this is the entry of proxy,it will deliver invocation to underlying impl e.g. ApplicationClientPBClientImpl
ApplicationClientPBClientImpl--the client proxy class. with this proxy,any methods invocations have been supplied the retry machanism,yep,this is the meaning of 'proxy'
Client --the final rpc service in client side.
YarnServiceProtos--used to convert common request & response relative params to PB's one,corresponding to 'yarn_service_protos.proto'
YarnProtos--similar to YarnServiceProtos,this is responsible to conert certain fields of a param generated by YarnServiceProtos.corresponding to 'yarn_protos.proto'
相关推荐
Hadoop 2.7.3 Windows64位 编译bin(包含winutils.exe, hadoop.dll),自己用的,把压缩包里的winutils.exe, hadoop.dll 放在你的bin 目录 在重启eclipse 就好了
hadoop-eclipse-plugin-2.6.0.jar
ambari-2.7.5 编译过程中四个大包下载很慢,所以需要提前下载,包含:hbase-2.0.2.3.1.4.0-315-bin.tar.gz ,hadoop-3.1.1.3.1.4.0-315.tar.gz , grafana-6.4.2.linux-amd64.tar.gz ,phoenix-5.0.0.3.1.4.0-315....
Hadoop 3.x(MapReduce)----【Hadoop 序列化】---- 代码 Hadoop 3.x(MapReduce)----【Hadoop 序列化】---- 代码 Hadoop 3.x(MapReduce)----【Hadoop 序列化】---- 代码 Hadoop 3.x(MapReduce)----【Hadoop ...
Hadoop是大数据技术中最重要的框架之一,是学习大数据必备的第一课,在Hadoop平台之上,可以更容易地开发和运行其他处理大规模数据的框架。尚硅谷Hadoop视频教程再次重磅升级!以企业实际生产环境为背景,增加了更...
flink1.14.0与hadoop3.x的兼容包,放在flink的lib目录下
文件名: spark-3.4.1-bin-hadoop3.tgz 这是 Apache Spark 3.4.1 版本的二进制文件,专为与 Hadoop 3 配合使用而设计。Spark 是一种快速、通用的集群计算系统,用于大规模数据处理。这个文件包含了所有必要的组件,...
hadoop-annotations-3.1.1.jar hadoop-common-3.1.1.jar hadoop-mapreduce-client-core-3.1.1.jar hadoop-yarn-api-3.1.1.jar hadoop-auth-3.1.1.jar hadoop-hdfs-3.1.1.jar hadoop-mapreduce-client-hs-3.1.1.jar ...
hadoop.dll-winutils.exe-hadoop2.7.x,本人环境是2.7.2,使用hadoop.dll和winutils.exe之后不再报错
Flink-1.11.2与Hadoop3集成JAR包,放到flink安装包的lib目录下,可以避免Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Hadoop is not in the classpath/dependencies.这个报错,实现...
hadoop2.X新特性介绍
flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar
Hadoop 3.x(MapReduce)----【MapReduce 概述】---- 代码 Hadoop 3.x(MapReduce)----【MapReduce 概述】---- 代码 Hadoop 3.x(MapReduce)----【MapReduce 概述】---- 代码 Hadoop 3.x(MapReduce)----...
hadoop.dll与winutils.exe 64位windows下使用hadoop2.9.0必要插件,解压后放在hadoop-2.9.0\bin下
编译环境: hadoop2.5.2 win7 32位系统 eclipse-luna-x86
Hadoop 3.x(HDFS)----【HDFS 的 API 操作】---- 代码 Hadoop 3.x(HDFS)----【HDFS 的 API 操作】---- 代码 Hadoop 3.x(HDFS)----【HDFS 的 API 操作】---- 代码 Hadoop 3.x(HDFS)----【HDFS 的 API 操作】--...
- `hadoop2x-eclipse-plugin-master/ivy/library.properties` - `hadoop2x-eclipse-plugin-master/src/contrib/eclipse-plugin/build.xml` 开源源地址: https://github.com/winghc/hadoop2x-eclipse-plugin
spark-3.0.0-bin-hadoop2.7.tgz 官网下载不了的,需要资源的,可以到这里下载哦
kettle 9.1 连接hadoop clusters (CDH 6.2) 驱动
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory). log4j:WARN Please initialize the log4j system properly. log4j:WARN See ...