分享知识，分享快乐

0%

使用Yarn标签机制实现任务资源隔离

发表于

使用Yarn标签机制实现任务资源隔离

官网参考地址： https://docs.cloudera.com/runtime/7.2.0/yarn-allocate-resources/topics/yarn-configuring-node-labels.html

参考地址： https://zhuanlan.zhihu.com/p/674882366

1
2
3
4
[root@bigdata-1 ~]# sudo -u hdfs hadoop fs -mkdir -p /yarn/node-labels
[root@bigdata-1 ~]# sudo -u hdfs hadoop fs -chown -R yarn:yarn /yarn
[root@bigdata-1 ~]# sudo -u hdfs hadoop fs -chmod -R 700 /yarn
[root@bigdata-1 ~]# hadoop fs -ls /yarn

YARN Service Advanced Configuration Snippet (Safety Valve) for yarn-site.xml

add the following:

  • Set the following property to enable Node Labels:

    1
    2
    Name: yarn.node-labels.enabled
    Value: true

  • Set the following property to reference the HDFS node label directory

    1
    2
    Name: yarn.node-labels.fs-store.root-dir
    Value: hdfs://:/

    For example,

    1
    2
    Name: yarn.node-labels.fs-store.root-dir
    Value: hdfs://node-1.example.com:8020/yarn/node-labels/

使用命令方式给yarn集群添加标签

1
sudo -u yarn yarn rmadmin -addToClusterNodeLabels "x(exclusive=true),y(exclusive=false)"

使用命令方式节点关联标签：

1
sudo -u yarn  yarn rmadmin -replaceLabelsOnNode "bigdata-2.baofoo.cn=x bigdata-3.baofoo.cn=y"

提交MR任务指定标签： 测试下来不能用

1
hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-cdh6.0.1-tests.jar TestDFSIO -D test.build.data=/tmp/benchmark -Dmapreduce.job.node-label-expression=x -write -nrFiles 10 -fileSize 1000