作者：櫰木按照上文中hadoop集群规划进行安装。 1 HADOOP集群安装在hd1.dtstack.com主机root权限下安装hadoop集群 解压[root@hd1.dtstack.com software]# tar -zvxf hadoop-3.2.4.tar.gz -C /opt/[root@hd1.dtstack.com software]# chown -R hdfs:hadoop /opt/hadoop-3.2.4[root@hd1.dtstack.com software]# ln -s /opt/hadoop-3.2.4 /opt/hadoop2 HADOOP Kerberos主体服务 所在主机 主体格式（Principal） keytab文件 NameNode hd1.dtstack.com、hd2.dtstack.com hdfs/_HOST@DTSTACK.COM /etc/security/keytab/hdfs..keytab DataNode hd3.dtstack.com、hadoop04、hadoop05 hdfs/_HOST@DTSTACK.COM /etc/security/keytab/hdfs..keytab JournalNode hd1.dtstack.com、hd2.dtstack.com、hd3.dtstack.com hdfs/_HOST@DTSTACK.COM /etc/security/keytab/hdfs.keytab Web UI hd1.dtstack.com、hd2.dtstack.com、hd3.dtstack.com HTTP/_HOST@DTSTACK.COM /etc/security/keytab/hdfs.keytab JobHistory Server hd1.dtstack.com、hd2.dtstack.com yarn/_HOST@DTSTACK.COM /etc/security/keytab/yarn..keytab ResourceManager hd1.dtstack.com、hd2.dtstack.com yarn/_HOST@DTSTACK.COM /etc/security/keytab/yarn.service.keytab NodeManager hd3.dtstack.com yarn/_HOST@DTSTACK.COM /etc/security/keytab/yarn..keytab 说明： 创建主体命令见上面kerberos票据创建_HOST表示配置文件变量，在实际使用过程会自动替换成主机名，如hd1.dtstack.comKeytab文件名每台主机文件名一样，但文件内容不一样，主要区别是主机名Keytab文件创建完成后分发到对应主机，且权限修改成600，权限修改命令如下：chown -R root:hadoop /etc/security/keytab/ chmod 660 /etc/security/keytab/*按照kerberos票据创建进行票据主体创建和keytab文件创建以及分发到对应主机目录上 生成keytab文件 bash /root/bigdata/getkeytabs.sh /etc/security/keytab/hdfs.keytab hdfs由于页面需要http的principal，给hdfs的keytab添加httpprincipalbash /root/bigdata/getkeytabs.sh /etc/security/keytab/hdfs.keytab HTTPbash /root/bigdata/getkeytabs.sh /etc/security/keytab/yarn.keytab yarnbash /root/bigdata/getkeytabs.sh /etc/security/keytab/yarn.keytab HTTP3、 HDFS使用HTTPS安全传输协议配置在hd1.dtstack.com主机root权限下执行 添加生成脚本[root@hd1.dtstack.com hadoop]# cd /opt/hadoop/[root@hd1.dtstack.com hadoop]# cd bin/ && vi on.sh#!/bin/bash path1=/opt/hadoop/binhosts="hd1.dtstack.com hd3.dtstack.com hd2.dtstack.com"echo "===========begine install ca ==========="sh $path1/ca_install.shecho "===========finish install ca ===========" echo "===========begine install https ==========="for host in $hostsdo ssh -t $host "$path1/keystore.sh" doneecho "===========finish install https ===========" 添加ca脚本vi ca_install.sh#! /bin/bash path=/data/kerberos/hdfs_ca#集群中安装httpshostnamess="hd1.dtstack.com hd3.dtstack.com hd2.dtstack.com"passwords=abc123hostname1=`hostname`#ca证书创建，只需要在一个节点上创建function make_CA(){ hostnames=$hostnamess password=$passwords echo 'make_CA begin ...' cd $path #删除之前可能产生的过期CA证书 rm -rf $path/hdfs_ca* #其中一台上生成CA，密码全部为abc123 /usr/bin/expect <<-eof 10 755 2048 9999 set timeout spawn openssl req -new -x509 -keyout hdfs_ca_key -out hdfs_ca_cert -days -subj c="CN/ST=zhejiang/L=hangzhou/O=dtstack/OU=dtstack/CN=$hostname1" expect { "*phrase*" {send "$password\r"; exp_continue} send }eof #将生成的ca证书hdfs_ca_key、hdfs_ca_cert分发到其他节点上 for host in $hostnames; do echo "copy hadoop ca to $host:$path" ssh root@$host "mkdir -p data kerberos hdfs_ca" scp hdfs_ca_* $host:$path done #rm -rf hdfs_ca* 'make_ca end ...'} make_ca添加keystore脚本vi keystore.sh#! bin bashpath="/data/kerberos/hdfs_ca#集群中安装https" keystorehostnamess="hadoop01.dtstack.com hadoop03.dtstack.com hadoop02.dtstack.com" passwords="abc123current_hostnames="`hostname`"export.UTF-8function" make_certificate(){ current_hostname="$current_hostnames" password="$passwords" cd $path #keytool需要使用java环境 source etc profile #生成keystore #name="CN=$current_hostname, OU=dtstack, O=dtstack, L=hangzhou, ST=zhejiang, C=CN" usr <<-eof keytool -keystore keystore -alias localhost -validity -genkey -keyalg rsa -keysize -dname "cn="$current_hostname," ou="dtstack," o="dtstack," l="hangzhou," st="zhejiang," "*password*" #添加ca到truststore truststore caroot -import -file "*certificate*" "yes\r"; #从keystore中导出cert -certreq cert #用ca对cert签名 x509 -req -ca -cakey -in cert_signed -cacreateserial #将ca的cert和用ca签名之后的cert导入keystore #将最终keystore,trustores放入合适的目录，并加上后缀jks security https && mkdir #chmod "install keystore、truststore hdfs_ca ..." cp keystore.jks truststore.jks} "[+] execute hlk_each_host_install_https.sh begin "hostnames:$hostnames" "current_hostname:$current_hostname" #每个节点获取ca证书签照 make_certificate ..."将脚本分发到每个节点的 opt 目录下，同时修改脚本权限 4、生成对应https证书（只需要在一个节点执行即可）mkdir hdfs_cacd bash on.sh更多技术信息请查看云掣官网云掣yunche - 可观测运维专家 | 大数据运维托管 云msp服务