分享知识，分享快乐

0%

部署Alluxio集群

发表于

准备
下载文件：wget https://downloads.alluxio.io/downloads/files/2.9.2/alluxio-2.9.2-bin.tar.gz

解压目录：/opt/servers/alluxio

1
2
wget https://downloads.alluxio.io/downloads/files/2.9.2/alluxio-2.9.2-bin.tar.gz
tar -xzf alluxio-2.9.2-bin.tar.gz -C /opt/servers/alluxio

创建目录：mkdir -p /mnt/ramdisk

机器：10.0.21.190（master，worker）10.0.21.191（master，worker）10.0.21.192（worker）

HA 配置

搭建高可用集群前的准备:

  • ①确保Zookeeper服务已经运行

  • ②一个单独安装的可靠的共享日志存储系统(可用HDFS或S3等系统)

  • ③这个配置针对Alluxio 2.x版本，不适用于1.x版本

  • ④需要事先创建好ramdisk挂载目录

配置hive alluxio文件系统：core-site.xml

1
2
3
4
5
6
7
8
9
10
11
12
<configuration>
  <property>
      <name>fs.alluxio.impl</name>
      <value>alluxio.hadoop.FileSystem</value>
      <description>The Alluxio FileSystem (Hadoop 1.x and 2.x)</description>
  </property>
  <property>
      <name>fs.AbstractFileSystem.alluxio.impl</name>
      <value>alluxio.hadoop.AlluxioFileSystem</value>
      <description>The Alluxio AbstractFileSystem (Hadoop 2.x)</description>
  </property>
</configuration>

在所有机器上配置alluxio-env.sh

1
2
3
4
5
6
7
ALLUXIO_HOME=/opt/servers/alluxio
ALLUXIO_LOGS_DIR=/opt/servers/alluxio/logs
ALLUXIO_RAM_FOLDER=/mnt/ramdisk
JAVA_HOME=/usr/java/jdk1.8.0_141-cloudera
ALLUXIO_WORKER_JAVA_OPTS=" -XX:+PrintGCDetails -XX:+PrintTenuringDistribution -XX:+PrintGCTimeStamps"
ALLUXIO_MASTER_JAVA_OPTS=" -Xms2048M -Xmx4096M"
ALLUXIO_JAVA_OPTS+=" -Djava.library.path=/opt/cloudera/parcels/CDH/lib/hadoop"

在10.0.21.190机器上配置Master和Worker：alluxio-site.properties

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
# 10.0.21.190    Master Worker

# Common properties

alluxio.master.hostname=10.0.21.190
alluxio.underfs.hdfs.configuration=/etc/hadoop/conf/core-site.xml:/etc/hadoop/conf/hdfs-site.xml
alluxio.master.mount.table.root.ufs=hdfs://ns1/user/alluxio/ha/

# Worker properties

alluxio.worker.ramdisk.size=512MB
alluxio.worker.tieredstore.levels=1
alluxio.worker.tieredstore.level0.alias=MEM
alluxio.worker.tieredstore.level0.dirs.path=/mnt/ramdisk

# HA properties

alluxio.zookeeper.enabled=true
alluxio.zookeeper.address=10.0.21.190:2181,10.0.21.191:2181,10.0.21.192:2181
alluxio.master.journal.type=UFS
alluxio.master.journal.folder=hdfs://ns1/user/alluxio/journal/
alluxio.worker.block.heartbeat.timeout.ms=300000
alluxio.zookeeper.session.timeout=120s

# User properties

alluxio.user.file.readtype.default=CACHE_PROMOTE
alluxio.user.file.writetype.default=ASYNC_THROUGH
alluxio.user.metrics.collection.enable=true
alluxio.master.metrics.time.series.interval=1000

# Security properties

alluxio.security.authorization.permission.enabled=true
alluxio.security.authentication.type=SIMPLE
alluxio.master.security.impersonation.hive.users=*
alluxio.master.security.impersonation.hive.groups=*
alluxio.master.security.impersonation.yarn.users=*
alluxio.master.security.impersonation.yarn.groups=*
alluxio.master.security.impersonation.hdfs.users=*
alluxio.master.security.impersonation.hdfs.groups=*

在10.0.21.191机器上配置Master和Worker：alluxio-site.properties

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
# 10.0.21.191      Master Worker

# Common properties

alluxio.master.hostname=10.0.21.191
alluxio.underfs.hdfs.configuration=/etc/hadoop/conf/core-site.xml:/etc/hadoop/conf/hdfs-site.xml
alluxio.master.mount.table.root.ufs=hdfs://ns1/user/alluxio/ha/

# Worker properties

alluxio.worker.ramdisk.size=512MB
alluxio.worker.tieredstore.levels=1
alluxio.worker.tieredstore.level0.alias=MEM
alluxio.worker.tieredstore.level0.dirs.path=/mnt/ramdisk

# HA properties

alluxio.zookeeper.enabled=true
alluxio.zookeeper.address=10.0.21.190:2181,10.0.21.191:2181,10.0.21.192:2181
alluxio.master.journal.type=UFS
alluxio.master.journal.folder=hdfs://ns1/user/alluxio/journal/
alluxio.worker.block.heartbeat.timeout.ms=300000
alluxio.zookeeper.session.timeout=120s

# User properties

alluxio.user.file.readtype.default=CACHE_PROMOTE
alluxio.user.file.writetype.default=ASYNC_THROUGH
alluxio.user.metrics.collection.enable=true
alluxio.master.metrics.time.series.interval=1000

# Security properties

alluxio.security.authorization.permission.enabled=true
alluxio.security.authentication.type=SIMPLE
alluxio.master.security.impersonation.hive.users=*
alluxio.master.security.impersonation.hive.groups=*
alluxio.master.security.impersonation.yarn.users=*
alluxio.master.security.impersonation.yarn.groups=*
alluxio.master.security.impersonation.hdfs.users=*
alluxio.master.security.impersonation.hdfs.groups=*

在10.0.21.192机器上配置Worker：alluxio-site.properties

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
# 10.0.21.192      Worker

# Common properties

# Worker不需要写alluxio.master.hostname参数和alluxio.master.journal.folder参数

alluxio.underfs.hdfs.configuration=/etc/hadoop/conf/core-site.xml:/etc/hadoop/conf/hdfs-site.xml
alluxio.master.mount.table.root.ufs=hdfs://ns1/user/alluxio/ha/

# Worker properties

alluxio.worker.ramdisk.size=512MB
alluxio.worker.tieredstore.levels=1
alluxio.worker.tieredstore.level0.alias=MEM
alluxio.worker.tieredstore.level0.dirs.path=/mnt/ramdisk

# HA properties

alluxio.zookeeper.enabled=true
alluxio.zookeeper.address=10.0.21.190:2181,10.0.21.191:2181,10.0.21.192:2181
alluxio.worker.block.heartbeat.timeout.ms=300000
alluxio.zookeeper.session.timeout=120s

# User properties

alluxio.user.file.readtype.default=CACHE_PROMOTE
alluxio.user.file.writetype.default=ASYNC_THROUGH
alluxio.user.metrics.collection.enable=true
alluxio.master.metrics.time.series.interval=1000

# Security properties

alluxio.security.authorization.permission.enabled=true
alluxio.security.authentication.type=SIMPLE
alluxio.master.security.impersonation.hive.users=*
alluxio.master.security.impersonation.hive.groups=*
alluxio.master.security.impersonation.yarn.users=*
alluxio.master.security.impersonation.yarn.groups=*
alluxio.master.security.impersonation.hdfs.users=*
alluxio.master.security.impersonation.hdfs.groups=*

在所有机器上指定Master和Worker节点

1
2
3
4
5
6
7
8
#vim masters
10.0.21.190
10.0.21.191

#vim workers
10.0.21.190
10.0.21.191
10.0.21.192

测试部署是否成功

1
2
3
4
./alluxio format
./alluxio-start.sh all SudoMount
./alluxio fsadmin report
./alluxio runTests    # 如果出现Passed the test则说明部署成功

测试高可用模式的自动故障处理: (假设此时hadoop101位primary master)

1
2
3
4
ssh hadoop101
jps | grep AlluxioMaster
kill -9 <AlluxioMaster PID>
alluxio fs leader  # 显示新的primary Master(可能需要等待一小段时间选举)

补充：添加新的worker节点

动态添加worker到Alluxio集群就像通过适当配置启动新Alluxio worker进程一样简单。 在大多数情况下，新worker配置应与所有其他worker配置相同。 在新worker上运行以下命令，以将其添加到集群。

1
$ ./bin/alluxio-start.sh worker SudoMount # 启动本地 worker

一旦worker启动，它将在Alluxio master上注册，并成为Alluxio集群的一部分。

减少worker只需要简单停止一个worker进程。

1
$ ./bin/alluxio-stop.sh worker # 停止本地 worker

一旦worker被停止，master将在预定的超时值（通过master参数alluxio.master.worker.timeout配置）后将此worker标记为缺失。 主机视worker为“丢失”，并且不再将其包括在集群中。

实际步骤：

  • 1、修改workers文件，添加新服务器
  • 2、scp alluxio目录 到新的服务器节点
  • 3、在新服务节点手动挂载 sudo ./bin/alluxio-mount.sh SudoMount
  • 4、master节点统一启动

补充：配置多级存储

1、创建目录

1
mkdir /mnt/alluxio_data

2、对于与启动Alluxio服务的用户组同组用户应给予770权限。

1
sudo chmod 770 alluxio_data

3、修改配置文件

1
2
3
4
5
6
7
8
9
10
alluxio.worker.tieredstore.levels=2
alluxio.worker.tieredstore.level0.alias=MEM
alluxio.worker.tieredstore.level0.dirs.path=/mnt/ramdisk
alluxio.worker.tieredstore.level0.dirs.mediumtype=MEM
alluxio.worker.tieredstore.level0.dirs.quota=1GB

alluxio.worker.tieredstore.level1.alias=HDD
alluxio.worker.tieredstore.level1.dirs.path=/mnt/alluxio_data
alluxio.worker.tieredstore.level1.dirs.mediumtype=HDD
alluxio.worker.tieredstore.level1.dirs.quota=50GB