分享知识,分享快乐

0%

CDH Disk Balancer 磁盘数据均衡

CDH Disk Balancer 磁盘数据均衡

1.设置dfs.disk.balancer.enabled 为true , 可以单个datanode设置,重启单个datanode生效。

1
2
3
4
<property>
<name>dfs.disk.balancer.enabled</name>
<value>true</value>
</property>
image-20210429175922996

1.创建均衡任务并生成计划任务配置文件

sudo -u hdfs hdfs diskbalancer -plan cdh85-73

1
2
3
4
5
6
7
8
[root@cdh85-73 tmp]# sudo -u hdfs hdfs diskbalancer -plan cdh85-73
21/04/29 17:02:45 INFO planner.GreedyPlanner: Starting plan for Node : cdh85-73:9867
21/04/29 17:02:45 INFO planner.GreedyPlanner: Disk Volume set f4da1504-1dfc-42d0-ac61-020dd7b02620 Type : DISK plan completed.
21/04/29 17:02:45 INFO planner.GreedyPlanner: Compute Plan for Node : cdh85-73:9867 took 7 ms
21/04/29 17:02:46 INFO command.Command: Writing plan to:
21/04/29 17:02:46 INFO command.Command: /system/diskbalancer/2021-Apr-29-17-02-45/cdh85-73.plan.json
Writing plan to:
/system/diskbalancer/2021-Apr-29-17-02-45/cdh85-73.plan.json

注意: 这个路径是HDFS的路径,不是本地路径

2.查看配置文件

1
2
3
4
[root@cdh85-73 tmp]# hdfs dfs -ls  /system/diskbalancer/2021-Apr-29-16-45-51
Found 2 items
-rw-r--r-- 3 hdfs supergroup 192586 2021-04-29 16:45 /system/diskbalancer/2021-Apr-29-16-45-51/cdh85-73.before.json
-rw-r--r-- 3 hdfs supergroup 4546 2021-04-29 16:45 /system/diskbalancer/2021-Apr-29-16-45-51/cdh85-73.plan.json

3.启动均衡任务

sudo -u hdfs hdfs diskbalancer -execute /system/diskbalancer/2021-Apr-29-16-51-42/cdh85-73.plan.json

注意:这里执行后 立马结束了 看不到效果,实际上是生效的

4.查看状态

sudo -u hdfs hdfs diskbalancer -query cdh85-73

1
2
3
4
5
[root@cdh85-73 tmp]# sudo -u hdfs hdfs diskbalancer -query cdh85-73
21/04/29 17:42:24 INFO command.Command: Executing "query plan" command.
Plan File: /system/diskbalancer/2021-四月-29-16-40-04/cdh85-73.plan.json
Plan ID: bb2087b0d6dfbe65835e2831836fd814cf70e605
Result: PLAN_UNDER_PROGRESS

完成后的状态,耗时大约20几个Hours