YARN-HA 配置 - 最码农

YARN-HA工作机制

1）官方文档：

http://hadoop.apache.org/docs/r3.1.3/hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html

2）YARN-HA工作机制

配置YARN-HA集群

1）环境准备

（1）修改IP
（2）修改主机名及主机名和IP地址的映射
（3）关闭防火墙
（4）ssh免密登录
（5）安装JDK，配置环境变量等
（6）配置Zookeeper集群

2）规划集群

hadoop102	hadoop103	hadoop104
NameNode	NameNode	NameNode
JournalNode	JournalNode	JournalNode
DataNode	DataNode	DataNode
ZK	ZK	ZK
ResourceManager	ResourceManager	ResourceManager
NodeManager	NodeManager	NodeManager

3）具体配置

（1）yarn-site.xml


<!-- 启用resourcemanager ha -->
		<property>
			<name>yarn.resourcemanager.ha.enabled</name>
			<value>true</value>
		</property>
	 
		<!-- 声明三台resourcemanager的地址 -->
		<property>
			<name>yarn.resourcemanager.cluster-id</name>
			<value>cluster-yarn1</value>
		</property>
		<!--指定resourcemanager的逻辑列表-->
		<property>
			<name>yarn.resourcemanager.ha.rm-ids</name>
			<value>rm1,rm2,rm3</value>
		</property>
		
		<!-- ========== rm1的配置 ========== -->
		<!-- 指定rm1的主机名 -->
		<property>
			<name>yarn.resourcemanager.hostname.rm1</name>
			<value>hadoop102</value>
		</property>
		<!-- 指定rm1的web端地址 -->
		<property>
			<name>yarn.resourcemanager.webapp.address.rm1</name>
			<value>hadoop102:8088</value>
		</property>
		<!-- 指定rm1的内部通信地址 -->
		<property>
			<name>yarn.resourcemanager.address.rm1</name>
			<value>hadoop102:8032</value>
		</property>
		<!-- 指定AM向rm1申请资源的地址 -->
		<property>
			<name>yarn.resourcemanager.scheduler.address.rm1</name>  
			<value>hadoop102:8030</value>
		</property>
		<!-- 指定供NM连接的地址 -->  
		<property>
			<name>yarn.resourcemanager.resource-tracker.address.rm1</name>
			<value>hadoop102:8031</value>
		</property>
		
		<!-- ========== rm2的配置 ========== -->
		<!-- 指定rm2的主机名 -->
		<property>
			<name>yarn.resourcemanager.hostname.rm2</name>
			<value>hadoop103</value>
		</property>
		<property>
			<name>yarn.resourcemanager.webapp.address.rm2</name>
			<value>hadoop103:8088</value>
		</property>
		<property>
			<name>yarn.resourcemanager.address.rm2</name>
			<value>hadoop103:8032</value>
		</property>
		<property>
			<name>yarn.resourcemanager.scheduler.address.rm2</name>
			<value>hadoop103:8030</value>
		</property>
		<property>
			<name>yarn.resourcemanager.resource-tracker.address.rm2</name>
			<value>hadoop103:8031</value>
		</property>
		
		<!-- ========== rm3的配置 ========== -->
		<!-- 指定rm3的主机名 -->
		<property>
			<name>yarn.resourcemanager.hostname.rm3</name>
			<value>hadoop104</value>
		</property>
		<property>
			<name>yarn.resourcemanager.webapp.address.rm3</name>
			<value>hadoop104:8088</value>
		</property>
		<property>
			<name>yarn.resourcemanager.address.rm3</name>
			<value>hadoop104:8032</value>
		</property>
		<property>
			<name>yarn.resourcemanager.scheduler.address.rm3</name>
			<value>hadoop104:8030</value>
		</property>
		<property>
			<name>yarn.resourcemanager.resource-tracker.address.rm3</name>
			<value>hadoop104:8031</value>
		</property>
	 
		<!-- 指定zookeeper集群的地址 --> 
		<property>
			<name>yarn.resourcemanager.zk-address</name>
			<value>hadoop102:2181,hadoop103:2181,hadoop104:2181</value>
		</property>

		<!-- 启用自动恢复 --> 
		<property>
			<name>yarn.resourcemanager.recovery.enabled</name>
			<value>true</value>
		</property>
	 
		<!-- 指定resourcemanager的状态信息存储在zookeeper集群 --> 
		<property>
			<name>yarn.resourcemanager.store.class</name>     
			<value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
		</property>

4）启动hdfs


[hadoop@hadoop102 ~]$ start-dfs.sh

5）启动YARN

（1）在hadoop102或者hadoop103中执行：


[hadoop@hadoop102 ~]$ start-yarn.sh

（2）查看服务状态


[hadoop@hadoop102 ~]$ yarn rmadmin -getServiceState rm1

（3）可以去zkCli.sh客户端查看ResourceManager选举锁节点内容：


[hadoop@hadoop102 ~]$ zkCli.sh
[zk: localhost:2181(CONNECTED) 16] get -s /yarn-leader-election/cluster-yarn1/ActiveStandbyElectorLock

cluster-yarn1rm1
cZxid = 0x100000022
ctime = Tue Jul 14 17:06:44 CST 2020
mZxid = 0x100000022
mtime = Tue Jul 14 17:06:44 CST 2020
pZxid = 0x100000022
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x30000da33080005
dataLength = 20
numChildren = 0

（4）web端查看hadoop102:8088和hadoop103:8088的YARN的状态，和NameNode对比，查看区别

Hadoop 教程