分享知识,分享快乐

0%

DolphinScheduler集群部署

官网部署文档

https://dolphinscheduler.apache.org/zh-cn/docs/latest/user_doc/cluster-deployment.html

1、基础软件安装(必装项请自行安装)

  • PostgreSQL (8.2.15+) or MySQL (5.7系列) : 两者任选其一即可, 如MySQL则需要JDBC Driver 5.1.47+
  • JDK (1.8+) : 必装,请安装好后在/etc/profile下配置 JAVA_HOME 及 PATH 变量
  • ZooKeeper (3.4.6+) :必装
  • Hadoop (2.6+) or MinIO :选装,如果需要用到资源上传功能,可以选择上传到Hadoop or MinIO上
1
注意:DolphinScheduler本身不依赖Hadoop、Hive、Spark,仅是会调用他们的Client,用于对应任务的提交。

2、下载二进制tar.gz包

  • 请下载最新版本的后端安装包至服务器部署目录,比如创建 /tmp/dolphinscheduler 做为安装部署目录,下载地址: 下载,下载后上传tar包到该目录中,并进行解压
1
2
3
4
5
6
7
# 创建部署目录,部署目录请不要创建在/root、/home等高权限目录 
mkdir -p /tmp/app/dolphinscheduler; #安装文件别放到/tmp/dolphinscheduler , 这是特殊路径 创建文件时会用到的临时路径
cd /tmp/app/dolphinscheduler;
# 解压缩
tar -zxvf apache-dolphinscheduler-incubating-1.3.5-dolphinscheduler-bin.tar.gz -C /opt/dolphinscheduler;

mv apache-dolphinscheduler-incubating-1.3.5-dolphinscheduler-bin dolphinscheduler-bin

3、创建部署用户和hosts映射

  • 所有部署调度的机器上创建部署用户,并且一定要配置sudo免密。假如我们计划在ds1,ds2,ds3,ds4这4台机器上部署调度,首先需要在每台机器上都创建部署用户
1
2
3
4
5
6
7
8
9
10
11
12
13
# 创建用户需使用root登录,设置部署用户名,请自行修改,后面以dolphinscheduler为例
useradd dolphinscheduler;

# 设置用户密码,请自行修改,后面以dolphinscheduler123为例
echo "dolphinscheduler123" | passwd --stdin dolphinscheduler

# 配置sudo免密
echo 'dolphinscheduler ALL=(ALL) NOPASSWD: NOPASSWD: ALL' >> /etc/sudoers
sed -i 's/Defaults requirett/#Defaults requirett/g' /etc/sudoers
注意:
- 因为是以 sudo -u {linux-user} 切换不同linux用户的方式来实现多租户运行作业,所以部署用户需要有 sudo 权限,而且是免密的。
- 如果发现/etc/sudoers文件中有"Default requiretty"这行,也请注释掉
- 如果用到资源上传的话,还需要在`HDFS或者MinIO`上给该部署用户分配读写的权限

4、配置hosts映射和ssh打通及修改目录权限

  • 备注:当然 通过sshpass -p xxx sudo scp -r /etc/hosts $ip:/etc/就可以省去输入密码了

    centos下sshpass的安装:

    1. 先安装epel

      yum install -y epel-release

      yum repolist

    2. 安装完成epel之后,就可以按照sshpass了

      yum install -y sshpass

  • 在ds1上,切换到部署用户并配置ssh本机免密登录

    1
    2
    3
    4
    5
     su dolphinscheduler;

    ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
    cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
    chmod 600 ~/.ssh/authorized_keys

注意:正常设置后,dolphinscheduler用户在执行命令ssh localhost 是不需要再输入密码的

  • 在ds1上,配置部署用户dolphinscheduler ssh打通到其他待部署的机器

    1
    2
    3
    4
    5
    su dolphinscheduler;
    for ip in bigdata-9.baofoo.cn bigdata-8.baofoo.cn bigdata-7.baofoo.cn ;
    do
    sshpass -p dolphinscheduler123 ssh-copy-id $ip
    done
  • 在ds1上,修改目录权限,使得部署用户对dolphinscheduler-bin目录有操作权限

5、数据库初始化

  • 进入数据库,默认数据库是PostgreSQL,如选择MySQL的话,后续需要添加mysql-connector-java驱动包到DolphinScheduler的lib目录下,这里以MySQL为例
1
mysql -h192.168.xx.xx -P3306 -uroot -p
  • 进入数据库命令行窗口后,执行数据库初始化命令,设置访问账号和密码。注: {user} 和 {password} 需要替换为具体的数据库用户名和密码
1
2
3
4
mysql> CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'%' IDENTIFIED BY '{password}';
mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'localhost' IDENTIFIED BY '{password}';
mysql> flush privileges;
  • 创建表和导入基础数据

    • 修改 conf 目录下 datasource.properties 中的下列配置
    1
    vi conf/datasource.properties
    • 如果选择 MySQL,请注释掉 PostgreSQL 相关配置(反之同理), 还需要手动添加 [ mysql-connector-java 驱动 jar ] 包到 lib 目录下,这里下载的是mysql-connector-java-5.1.47.jar,然后正确配置数据库连接相关信息
    1
    2
    3
    4
    5
    6
    7
    8
    #postgre
    #spring.datasource.driver-class-name=org.postgresql.Driver
    #spring.datasource.url=jdbc:postgresql://localhost:5432/dolphinscheduler
    # mysql
    spring.datasource.driver-class-name=com.mysql.jdbc.Driver
    spring.datasource.url=jdbc:mysql://10.0.20.107:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8&allowMultiQueries=true
    spring.datasource.username=cs_yangz 需要修改为上面的{user}值
    spring.datasource.password=xxx 需要修改为上面的{password}值
    • 修改并保存完后,执行 script 目录下的创建表及导入基础数据脚本
    1
    sh script/create-dolphinscheduler.sh

6、修改运行参数

  • 修改 conf/env 目录下的 dolphinscheduler_env.sh 环境变量(以相关用到的软件都安装在/opt/soft下为例)

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
       export HADOOP_HOME=/opt/cloudera/parcels/CDH/lib/hadoop
    export HADOOP_CONF_DIR=/etc/hadoop/conf
    #export SPARK_HOME1=/opt/soft/spark1
    export SPARK_HOME2=/opt/cloudera/parcels/CDH/lib/spark
    export PYTHON_HOME=/usr/bin/python
    export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
    export HIVE_HOME=/opt/cloudera/parcels/CDH/lib/hive
    #export FLINK_HOME=/opt/soft/flink
    #export DATAX_HOME=/opt/soft/datax/bin/datax.py
    export PATH=$HADOOP_HOME/bin:$SPARK_HOME2/bin:$PYTHON_HOME:$JAVA_HOME/bin:$HIVE_HOME/bin:$PATH

    `注: 这一步非常重要,例如 JAVA_HOME 和 PATH 是必须要配置的,没有用到的可以忽略或者注释掉`

修改一键部署配置文件 conf/config/install_config.conf中的各参数,特别注意以下参数的配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
# postgresql or mysql
dbtype="mysql"

# db config
# db address and port
dbhost="10.0.20.107:3306"

# db username
username="cs_yangz"

# database name
dbname="dolphinscheduler"

# db passwprd
# NOTICE: if there are special characters, please use the \ to escape, for example, `[` escape to `\[`
password="***"

# zk cluster
zkQuorum="10.0.19.130:2181,10.0.19.131:2181,10.0.19.132:2181"

# Note: the target installation path for dolphinscheduler, please not config as the same as the current path (pwd)
installPath="/opt/dolphinscheduler"

# deployment user
# Note: the deployment user needs to have sudo privileges and permissions to operate hdfs. If hdfs is enabled, the root directory needs to be created by itself
deployUser="dolphinscheduler"


# alert config
# mail server host
mailServerHost="smtp.qiye.163.com"

# mail server port
# note: Different protocols and encryption methods correspond to different ports, when SSL/TLS is enabled, make sure the port is correct.
mailServerPort="25"

# sender
mailSender="hadooper@baofoo.com"

# user
mailUser="hadooper@baofoo.com"

# sender password
# note: The mail.passwd is email service authorization code, not the email login password.
mailPassword="***"

# TLS mail protocol support
starttlsEnable="true"

# SSL mail protocol support
# only one of TLS and SSL can be in the true state.
sslEnable="false"

#note: sslTrust is the same as mailServerHost
sslTrust="smtp.qiye.163.com"


# resource storage type:HDFS,S3,NONE
resourceStorageType="HDFS"

# if resourceStorageType is HDFS,defaultFS write namenode address,HA you need to put core-site.xml and hdfs-site.xml in the conf directory.
# if S3,write S3 address,HA,for example :s3a://dolphinscheduler,
# Note,s3 be sure to create the root directory /dolphinscheduler
defaultFS="hdfs://bigdata-7.baofoo.cn:8020"

# if resourceStorageType is S3, the following three configuration is required, otherwise please ignore
s3Endpoint="http://192.168.xx.xx:9010"
s3AccessKey="xxxxxxxxxx"
s3SecretKey="xxxxxxxxxx"

# if resourcemanager HA enable, please type the HA ips ; if resourcemanager is single, make this value empty
yarnHaIps="bigdata-7.baofoo.cn"

# if resourcemanager HA enable or not use resourcemanager, please skip this value setting; If resourcemanager is single, you only need to replace yarnIp1 to actual resourcemanager hostname.
singleYarnIp="bigdata-7.baofoo.cn"

# resource store on HDFS/S3 path, resource file will store to this hadoop hdfs path, self configuration, please make sure the directory exists on hdfs and have read write permissions。/dolphinscheduler is recommended
resourceUploadPath="/dolphinscheduler"

# who have permissions to create directory under HDFS/S3 root path
# Note: if kerberos is enabled, please config hdfsRootUser=
hdfsRootUser="hdfs"

# kerberos config
# whether kerberos starts, if kerberos starts, following four items need to config, otherwise please ignore
kerberosStartUp="false"
# kdc krb5 config file path
krb5ConfPath="$installPath/conf/krb5.conf"
# keytab username
keytabUserName="hdfs-mycluster@ESZ.COM"
# username keytab path
keytabPath="$installPath/conf/hdfs.headless.keytab"


# api server port
apiServerPort="12345"


# install hosts
# Note: install the scheduled hostname list. If it is pseudo-distributed, just write a pseudo-distributed hostname
ips="bigdata-7.baofoo.cn,bigdata-8.baofoo.cn,bigdata-9.baofoo.cn"

# ssh port, default 22
# Note: if ssh port is not default, modify here
sshPort="22"

# run master machine
# Note: list of hosts hostname for deploying master
masters="bigdata-9.baofoo.cn,bigdata-8.baofoo.cn"

# run worker machine
# note: need to write the worker group name of each worker, the default value is "default"
workers="bigdata-7.baofoo.cn:default,bigdata-8.baofoo.cn:default,bigdata-9.baofoo.cn:default"

# run alert machine
# note: list of machine hostnames for deploying alert server
alertServer="bigdata-8.baofoo.cn"

# run api machine
# note: list of machine hostnames for deploying api server
apiServers="bigdata-9.baofoo.cn"

特别注意:

  • 如果需要用资源上传到Hadoop集群功能, 并且Hadoop集群的NameNode 配置了 HA的话 ,需要开启 HDFS类型的资源上传,同时需要将Hadoop集群下的core-site.xml和hdfs-site.xml复制到/opt/dolphinscheduler/conf,非NameNode HA跳过次步骤

7、一键部署

  • 切换到部署用户dolphinscheduler,然后执行一键部署脚本

    sh install.sh

    1
    2
    3
    注意:
    第一次部署的话,在运行中第3步`3,stop server`出现5次以下信息,此信息可以忽略
    sh: bin/dolphinscheduler-daemon.sh: No such file or directory
  • 脚本完成后,会启动以下5个服务,使用jps命令查看服务是否启动(jpsjava JDK自带)

1
2
3
4
5
MasterServer         ----- master服务
WorkerServer ----- worker服务
LoggerServer ----- logger服务
ApiApplicationServer ----- api服务
AlertServer ----- alert服务

如果以上服务都正常启动,说明自动部署成功

部署成功后,可以进行日志查看,日志统一存放于logs文件夹内

1
2
3
4
5
6
logs/
├── dolphinscheduler-alert-server.log
├── dolphinscheduler-master-server.log
|—— dolphinscheduler-worker-server.log
|—— dolphinscheduler-api-server.log
|—— dolphinscheduler-logger-server.log

8、登录系统

默认的用户是admin,默认的密码是dolphinscheduler123

修改 JVM 参数

  • 两个文件
  • /bin/dolphinscheduler-daemon.sh
  • /scripts/dolphinscheduler-daemon.sh
1
export DOLPHINSCHEDULER_OPTS="-server -Xmx16g -Xms1g -Xss512k -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:LargePageSizeInBytes=128m -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70"

复制代码

  • 一键部署
1
2
3
su  dolphinscheduler

sh install.sh

复制代码

  • 进程检查
1
su dolphinscheduler;jps;

复制代码

img

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# 一键停止
sh ./bin/stop-all.sh
# 一键开启
sh ./bin/start-all.sh
# 启停master
sh ./bin/dolphinscheduler-daemon.sh start master-server
sh ./bin/dolphinscheduler-daemon.sh stop master-server
# 启停worker
sh ./bin/dolphinscheduler-daemon.sh start worker-server
sh ./bin/dolphinscheduler-daemon.sh stop worker-server
# 启停api-server
sh ./bin/dolphinscheduler-daemon.sh start api-server
sh ./bin/dolphinscheduler-daemon.sh stop api-server
# 启停logger
sh ./bin/dolphinscheduler-daemon.sh start logger-server
sh ./bin/dolphinscheduler-daemon.sh stop logger-server
# 启停alert
sh ./bin/dolphinscheduler-daemon.sh start alert-server
sh ./bin/dolphinscheduler-daemon.sh stop alert-server

遇到的坑:

1.安装文件别放到/tmp/dolphinscheduler , 这是特殊路径

2.用root用户启动服务导致一系列问题,要用新建的用户操作所有的一切。