分享知识，分享快乐

0%

发表于

doris部署

fe部署

1
2
3
4
5
第一次启动需指定--helper参数，后续再启动无需指定此参数
sh /opt/module/apache-doris-fe/bin/start_fe.sh --helper cdh192-150:19010 --daemon

ansible fe -m shell -a " sh /opt/module/apache-doris-fe/bin/stop_fe.sh "
ansible fe -m shell -a " sh /opt/module/apache-doris-fe/bin/start_fe.sh --daemon "
1
2
3
4
5
6
7
8
9
10
mysql -h 127.0.0.1 -P9030 -uroot -p
mysql> SHOW PROC '/frontends'\G
ALTER SYSTEM ADD FOLLOWER "cdh192-151:19010";
# ALTER SYSTEM DROP FOLLOWER "cdh192-151:19010";
ALTER SYSTEM ADD OBSERVER "cdh192-152:19010";
# ALTER SYSTEM drop OBSERVER "cdh192-152:19010";
ALTER SYSTEM ADD FOLLOWER "cdh192-153:19010";
ALTER SYSTEM ADD FOLLOWER "cdh192-146.hadoop.xy:19010";

netstat -nap |grep 19010

be 部署

1
2
ansible be -m shell -a " sh /opt/module/apache-doris-be/bin/stop_be.sh "
ansible be -m shell -a " sh /opt/module/apache-doris-be/bin/start_be.sh --daemon "
1
2
3
4
5
mysql> SHOW PROC '/backends'\G
ALTER SYSTEM ADD BACKEND "cdh192-150:9050";
ALTER SYSTEM ADD BACKEND "cdh192-151:9050";
ALTER SYSTEM ADD BACKEND "cdh192-152:9050";
ALTER SYSTEM ADD BACKEND "cdh192-146:9050";
1
2
ansible be -m shell -a "sh /opt/module/apache-doris-be/bin/stop_be.sh "
ansible be -m shell -a "sh /opt/module/apache-doris-be/bin/start_be.sh --daemon "

doris升级

重要！！在升级之前需要备份元数据（整个目录都需要备份）！！

阅读全文 »

发表于

群用到了那些组件？ 版本有什么要求？

trino386，这个是jdk11了，安装两个目录就行了， phoenix5， hive3.1.2, hadoop.3.1.3,spark2.4.8,spark3.2 flink1.12.7, flink和spark都想升级。 iceberg.0.13.2,

trino ui地址 http://cdh192-57:8089/ui/ 用户 presto jdbc 172.20.192.57 用户presto

软件都安装在/home/hadoop/bigdata/这个下面

[root@hadoop102 ~]$ xsync/home/hadoop/bigdata/hadoop/etc/hadoop/capacity-scheduler.xml

刷新yarn集群的队列

1
${HADOOP_HOME}/bin/yarn rmadmin -refreshQueues` 或`yarn rmadmin -refreshQueues

hbase

cdh192-[234-236]

/opt/local/

阅读全文 »

发表于

CDH6.3.2 升级 Spark3.3.0 版本

https://juejin.cn/post/7140053569431928845

根据上面的文档进行部署 还有下列操作需要补充

1
2
3
4
5
6
7
8
9
10
11
12
cp -r /etc/hadoop/conf/*.xml /opt/cloudera/parcels/CDH/lib/spark3/conf/
cp /etc/hive/conf/hive-site.xml /opt/cloudera/parcels/CDH/lib/spark3/conf/

# 奇怪用快捷方式就不行
cd /opt/cloudera/parcels/CDH/lib/spark3/conf
ln -s /etc/hive/conf/hive-site.xml hive-site.xml
ln -s /etc/hive/conf/hdfs-site.xml hdfs-site.xml
ln -s /etc/hive/conf/core-site.xml core-site.xml
ln -s /etc/hive/conf/mapred-site.xml mapred-site.xml
ln -s /etc/hive/conf/yarn-site.xml yarn-site.xml
ln -s /etc/spark/conf/spark-defaults.conf spark-defaults.conf
ln -s /etc/spark/conf/spark-env.sh spark-env.sh
1
2
cd /usr/local/bin
ln -s /opt/cloudera/parcels/CDH/lib/spark3/bin/pyspark pyspark3

新增一个spark3-shell的快捷方式

vim /opt/cloudera/parcels/CDH/bin/spark3-shell

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
#!/bin/bash
# Autodetect JAVA_HOME if not defined
# Reference: http://stackoverflow.com/questions/59895/can-a-bash-script-tell-what-directory-its-stored-in
SOURCE="${BASH_SOURCE[0]}"
BIN_DIR="$( dirname "$SOURCE" )"
while [ -h "$SOURCE" ]
do
SOURCE="$(readlink "$SOURCE")"
[[ $SOURCE != /* ]] && SOURCE="$DIR/$SOURCE"
BIN_DIR="$( cd -P "$( dirname "$SOURCE"  )" && pwd )"
done
BIN_DIR="$( cd -P "$( dirname "$SOURCE" )" && pwd )"
CDH_LIB_DIR=$BIN_DIR/../../CDH/lib
LIB_DIR=$BIN_DIR/../lib
export HADOOP_HOME=$CDH_LIB_DIR/hadoop
. $CDH_LIB_DIR/bigtop-utils/bigtop-detect-javahome
exec $LIB_DIR/spark3/bin/spark-shell "$@"
1
chmod +x /opt/cloudera/parcels/CDH/bin/spark3-shell

用法：alternatives --install <链接> <名称> <路径> <优先度>

阅读全文 »

发表于

anaconda部署文档

https://docs.anaconda.com/anaconda/install/linux/

1
bash ./Anaconda3-2022.10-Linux-x86_64.sh

退出默认环境

1
conda config --set auto_activate_base False

conda activate

conda deactivate #退出环境

1
tar -czvf anaconda3.tar.gz anaconda3/
1
sudo -u hdfs hadoop fs -put /opt/module/anaconda3.tar.gz /user/yarn/mapreduce/

anaconda已安装 在/opt/module/anaconda3/

阅读全文 »

发表于

FreeIPA 服务器在重启后不会启动

参考 https://redhatlinux.guru/2020/10/09/freeipa-server-will-not-start-after-reboot/

原因 证书过期：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
[root@freeipa ~]# ipa-getcert list 
Number of certificates and requests being tracked: 9.
Request ID '20190830074301':
	status: CA_UNREACHABLE
	ca-error: Server at https://freeipa.baofoo.cn/ipa/xml failed request, will retry: -504 (libcurl failed to execute the HTTP POST transaction, explaining:  Failed connect to freeipa.baofoo.cn:443; Connection refused).
	stuck: no
	key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-BAOFOO-CN',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-BAOFOO-CN/pwdfile.txt'
	certificate: type=NSSDB,location='/etc/dirsrv/slapd-BAOFOO-CN',nickname='Server-Cert',token='NSS Certificate DB'
	CA: IPA
	issuer: CN=Certificate Authority,O=BAOFOO.CN
	subject: CN=freeipa.baofoo.cn,O=BAOFOO.CN
	expires: 2021-08-30 07:43:01 UTC
	dns: freeipa.baofoo.cn
	principal name: ldap/freeipa.baofoo.cn@BAOFOO.CN
	key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
	eku: id-kp-serverAuth,id-kp-clientAuth
	pre-save command: 
	post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv BAOFOO-CN
	track: yes
	auto-renew: yes

问题
这更多是利基问题。但可能对其他人有用。我在家里有一个用于 DNS 的 FreeIPA 服务器设置。在简单的重新启动以向 VM 添加一些 RAM 后，服务器将无法启动。我收到如下错误。

IPA 服务器错误

1
systemctl status ipa
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
● ipa.service - Identity, Policy, Audit
Loaded: loaded (/usr/lib/systemd/system/ipa.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Fri 2020-10-09 14:57:15 EDT; 1s ago
Process: 1110 ExecStart=/usr/sbin/ipactl start (code=exited, status=1/FAILURE)
Main PID: 1110 (code=exited, status=1/FAILURE)

Oct 09 14:57:15 ipasrv.home.local ipactl[1110]: Aborting ipactl
Oct 09 14:57:15 ipasrv.home.local ipactl[1110]: Starting Directory Service
Oct 09 14:57:15 ipasrv.home.local ipactl[1110]: Starting krb5kdc Service
Oct 09 14:57:15 ipasrv.home.local ipactl[1110]: Starting kadmin Service
Oct 09 14:57:15 ipasrv.home.local ipactl[1110]: Starting named Service
Oct 09 14:57:15 ipasrv.home.local ipactl[1110]: Starting httpd Service
Oct 09 14:57:15 ipasrv.home.local systemd[1]: ipa.service: main process exited, code=exited, status=1/FAILURE
Oct 09 14:57:15 ipasrv.home.local systemd[1]: Failed to start Identity, Policy, Audit.
Oct 09 14:57:15 ipasrv.home.local systemd[1]: Unit ipa.service entered failed state.
Oct 09 14:57:15 ipasrv.home.local systemd[1]: ipa.service failed.
Apache 服务错误
1
systemctl status httpd -l
阅读全文 »

发表于

cdh 安装Pheonix5.12

Phoenix5.12-之安装部署（Hbase-2.1.1）
Phoenix与Hbase的版本兼容是很严格的，我们需要使用Phoenix去适配Hbase，有很严格的版本标准，具体的版本匹配如下图。

具体版本的选择请查阅官网下载页面：https://phoenix.apache.org/download.html

1 解压

tar -zxvf phoenix-hbase-2.1-5.1.2-bin.tar.gz -C /opt/module/

2 重新命名

mv phoenix-hbase-2.1-5.1.2-bin/ phoenix-hbase-2.1-5.1.2

3 将phoenix-server-hbase-2.1-5.1.2.jar拷贝到Hbase的Master和Slave节点的lib目录下

cp phoenix-server-hbase-2.1-5.1.2.jar …/hbase-2.1.1/lib/

4 配置环境变量 hbase和phoenix 在/etc/profile

阅读全文 »

发表于

Dbeaver连接phoenix

1.切换到/opt/cloudera/parcels/PHOENIX-5.0.0-cdh6.2.0.p0.1308267/lib/phoenix，下载

phoenix-5.0.0-cdh6.2.0-client.jar这个jar包

2.下载客户端配置

里面有个phoenix-5.0.0-cdh6.2.0-client.jar下载下来

3.执行下面命令:

jar -uf phoenix-5.0.0-cdh6.2.0-client.jar hbase-site.xml 将hbase-site.xml添加到phoenix-5.0.0-cdh6.2.0-client.jar里

4.打开Dbeaver，创建新链接->编辑驱动->添加文件(刚才的jar包)->找到类->确定

5.填写连接信息，点击测试连接，成功！

阅读全文 »

发表于

Docker设置容器开机自启

设置docker开机自启

1
systemctl enable docker

设置docker容器开机自启
创建docker容器时设置开机自启
#在使用docker run启动容器时，使用–restart参数来设置：

1
docker run --restart=always --name imagesName

修改已创建的docker容器开机自启
#如果创建时未指定 --restart=always ,可通过update 命令

1
docker update --restart=always imagesName

参数说明
  --restart具体参数值详细信息：

no - 容器退出时，不重启容器；
on-failure - 只有在非0状态退出时才从新启动容器；
always - 无论退出状态是如何，都重启容器；

还可以在使用on - failure策略时，指定Docker将尝试重新启动容器的最大次数。默认情况下，Docker将尝试永远重新启动容器。

阅读全文 »

发表于

vim /usr/bin/xsync

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
#!/bin/bash
#1. 判断参数个数
if [ $# -lt 1 ]
then
 echo Not Enough Arguement!
 exit;
fi

#2. 遍历集群所有机器
for host in usdp01 usdp02 usdp03
do
 echo ==================== $host ====================
 #3. 遍历所有目录，挨个发送
 for file in $@
 do
 #4. 判断文件是否存在
 if [ -e $file ]
 then
 #5. 获取父目录
 pdir=$(cd -P $(dirname $file); pwd)
 #6. 获取当前文件的名称
 fname=$(basename $file)
 ssh $host "mkdir -p $pdir"
 rsync -av $pdir/$fname $host:$pdir
 else
 echo $file does not exists!
 fi
 done
done

jpsall.sh 脚本：

vim /usr/bin/jpsall

1
2
3
4
5
6
7
8
9
10
11
#!/bin/bash

# 执行jps命令查询每台服务器上的节点状态
echo ======================集群节点状态====================

for i in usdp01 usdp02 usdp03
do
        echo ====================== $i ====================
        ssh $i "/opt/module/jdk1.8.0_202/bin/jps"
done
echo ======================执行完毕====================
阅读全文 »

发表于

StartRocks补数据

创建catalog

1
2
3
4
5
6
7
8
9
CREATE EXTERNAL CATALOG hive_catalog0 
PROPERTIES(
  "type"="hive", 
  "hive.metastore.uris"="thrift://172.20.85.29:9083,thrift://172.20.85.29:9083"
);

SHOW RESOURCES;

show databases from hive_catalog0

创建parquet文件格式的临时表

1
2
3
4
create table PAY_TRADECENTER.T_TC_BASE_P 
 STORED AS parquet  
 as 
SELECT  *  from  PAY_TRADECENTER.T_TC_BASE where pk_day >= '2022-07-16' and  pk_day < '2022-08-17'

导入数据

1
2
3
4
5
6
7
8
SET query_timeout = 259200;
set exec_mem_limit=21474836480;
SET enable_insert_strict = false; 


INSERT  into PAY_TRADECENTER.`T_TC_BASE` (CREATED_AT ,ID, TRADE_NO, OUT_TRADE_NO, PRODUCT_TYPE, SUB_PRODUCT_TYPE, PRODUCT_CHANNEL, TRADE_CHANNEL, REQUEST_DATE, REQUEST_SYSTEM, TRADE_AMT, SERVICE_FEE, CCY, MERCHANT_NO, MERCHANT_TRADE_NO, ORIG_TRADE_NO, TRADE_FINISH_DATE, MEMO, EXTEND, SUBJECT, TRADE_STATUS, TRADE_FAIL_STATUS, TRADE_TYPE, REFUND_AMT, RESULT_CODE, RESULT_DESC, TRACE_LOG_ID, SETTLE_ACCOUNT_NO, SETTLE_ACCOUNT_TYPE, FEE_ACCOUNT_NO, FEE_ACCOUNT_TYPE, SUB_MERCHANT_ID, AGENT_NO, STORE_ID, EXTEND1, EXTEND2, EXTEND3, CREATED_BY, UPDATED_AT, UPDATED_BY, USER_ID, EXTEND4, EXTEND5, EXTEND6, TOTAL_ORDER_AMT, UNION_INFO)
SELECT CREATED_AT ,ID, TRADE_NO, OUT_TRADE_NO, PRODUCT_TYPE, SUB_PRODUCT_TYPE, PRODUCT_CHANNEL, TRADE_CHANNEL, REQUEST_DATE, REQUEST_SYSTEM, TRADE_AMT, SERVICE_FEE, CCY, MERCHANT_NO, MERCHANT_TRADE_NO, ORIG_TRADE_NO, TRADE_FINISH_DATE, MEMO, EXTEND, SUBJECT, TRADE_STATUS, TRADE_FAIL_STATUS, TRADE_TYPE, REFUND_AMT, RESULT_CODE, RESULT_DESC, TRACE_LOG_ID, SETTLE_ACCOUNT_NO, SETTLE_ACCOUNT_TYPE, FEE_ACCOUNT_NO, FEE_ACCOUNT_TYPE, SUB_MERCHANT_ID, AGENT_NO, STORE_ID, EXTEND1, EXTEND2, EXTEND3, CREATED_BY, UPDATED_AT, UPDATED_BY, USER_ID, EXTEND4, EXTEND5, EXTEND6, TOTAL_ORDER_AMT, UNION_INFO
FROM  hive_catalog0.PAY_TRADECENTER.T_TC_BASE_P

删除临时表

1
drop table PAY_TRADECENTER.T_TC_BASE_P

PAY_AGGREGATE T_ORDER_EXTERNAL CREATED_AT
PAY_BENEFIT T_BIZ_CMD CREATED_AT
PAY_BENEFIT T_ORDER_BASE CREATED_AT
PAY_BENEFIT T_ORDER_BASE_BENEFIT CREATED_AT
PAY_GATEWAY T_ACQ_AGGREGATE CREATED_AT
PAY_GATEWAY T_ACQ_AGGREGATE_DYNAMIC CREATED_AT
PAY_GATEWAY T_ACQ_NOTIFY CREATED_AT
PAY_GATEWAY T_ACQ_REFUND CREATED_AT
PAY_GATEWAY T_RISK_INFO CREATED_AT
PAY_SETTLE T_SETTLE_ORDER_ACQUIRING TRADE_FINISH_DATE
PAY_SETTLE T_SETTLE_ORDER_RE·FUND TRADE_FINISH_DATE
PAY_TRADECENTER T_BIZ_CMD CREATED_AT
PAY_TRADECENTER T_TC_ACCOUNT_TRANS CREATED_AT
PAY_TRADECENTER T_TC_AGGREGATE_TRANS CREATED_AT
PAY_TRADECENTER T_TC_BASE CREATED_AT
PAY_TRADECENTER T_TC_CHANNEL_MSG CREATED_AT
PAY_TRADECENTER T_TC_CHANNEL_TRANS CREATED_AT
PAY_VERIFY T_VERIFY_BANK_TRANS VERIFY_DATE
PAY_VERIFY T_VERIFY_CHANNEL_COST VERIFY_DATE
PAY_VERIFY T_VERIFY_CHANNEL_TRANS VERIFY_DATE
PAY_VERIFY T_VERIFY_RESULT VERIFY_DATE

阅读全文 »