CDH6.3.2 升级 Spark3.3.0 版本
https://juejin.cn/post/7140053569431928845
根据上面的文档进行部署 还有下列操作需要补充
1
2
3
4
5
6
7
8
9
10
11
12
| cp -r /etc/hadoop/conf/*.xml /opt/cloudera/parcels/CDH/lib/spark3/conf/
cp /etc/hive/conf/hive-site.xml /opt/cloudera/parcels/CDH/lib/spark3/conf/
# 奇怪用快捷方式就不行
cd /opt/cloudera/parcels/CDH/lib/spark3/conf
ln -s /etc/hive/conf/hive-site.xml hive-site.xml
ln -s /etc/hive/conf/hdfs-site.xml hdfs-site.xml
ln -s /etc/hive/conf/core-site.xml core-site.xml
ln -s /etc/hive/conf/mapred-site.xml mapred-site.xml
ln -s /etc/hive/conf/yarn-site.xml yarn-site.xml
ln -s /etc/spark/conf/spark-defaults.conf spark-defaults.conf
ln -s /etc/spark/conf/spark-env.sh spark-env.sh
|
1
2
| cd /usr/local/bin
ln -s /opt/cloudera/parcels/CDH/lib/spark3/bin/pyspark pyspark3
|
新增一个spark3-shell的快捷方式
vim /opt/cloudera/parcels/CDH/bin/spark3-shell
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
| #!/bin/bash
# Autodetect JAVA_HOME if not defined
# Reference: http://stackoverflow.com/questions/59895/can-a-bash-script-tell-what-directory-its-stored-in
SOURCE="${BASH_SOURCE[0]}"
BIN_DIR="$( dirname "$SOURCE" )"
while [ -h "$SOURCE" ]
do
SOURCE="$(readlink "$SOURCE")"
[[ $SOURCE != /* ]] && SOURCE="$DIR/$SOURCE"
BIN_DIR="$( cd -P "$( dirname "$SOURCE" )" && pwd )"
done
BIN_DIR="$( cd -P "$( dirname "$SOURCE" )" && pwd )"
CDH_LIB_DIR=$BIN_DIR/../../CDH/lib
LIB_DIR=$BIN_DIR/../lib
export HADOOP_HOME=$CDH_LIB_DIR/hadoop
. $CDH_LIB_DIR/bigtop-utils/bigtop-detect-javahome
exec $LIB_DIR/spark3/bin/spark-shell "$@"
|
1
| chmod +x /opt/cloudera/parcels/CDH/bin/spark3-shell
|
用法：alternatives --install <链接> <名称> <路径> <优先度>
1
| alternatives --install /usr/bin/spark3-shell spark3-shell /opt/cloudera/parcels/CDH/bin/spark3-shell 1
|