该 Kubernetes 部署过程中,对于部署环节,涉及多个组件,主要有 kubeadm 、kubelet 、kubectl。
Kubeadm 为构建 Kubernetes 提供了便捷、高效的“最佳实践” ,该工具提供了初始化完整 Kubernetes 过程所需的组件,其主要命令及功能有:
更多参考:
Kubeadm介绍
kubelet 是 Kubernetes 集群中用于操作 Docker 、containerd 等容器运行时的核心组件,需要在每个节点运行。通常该操作是基于 CRI 实现,kubelet 和 CRI 交互,以便于实现对 Kubernetes 的管控。
kubelet 主要用于配置容器网络、管理容器数据卷等容器全生命周期,对于 kubelet 而言,其主要的功能核心有:
更多参考:
kubelet介绍
kubectl 控制 Kubernetes 集群管理器,是作为 Kubernetes 的命令行工具,用于与 apiserver 进行通信,使用 kubectl 工具在 Kubernetes 上部署和管理应用程序。
使用 kubectl,可以检查群集资源的创建、删除和更新组件。
同时集成了大量子命令,可更便捷的管理 Kubernetes 集群,主要命令如下:
更多参考:
kubectl介绍
本方案基于 kubeadm 部署工具实现完整生产环境可用的 Kubernetes 高可用集群,同时提供相关 Kubernetes 周边组件。
其主要信息如下:
提示:本方案部署所使用脚本均由本人提供,可能不定期更新。
节点主机名 | IP | 类型 | 运行服务 |
---|---|---|---|
master01 | 172.24.10.11 | Kubernetes master节点 |
kubeadm、kubelet、kubectl、KeepAlived、 containerd、etcd、kube-apiserver、kube-scheduler、 kube-controller-manager、calico、WebUI、metrics、ingress、Longhorn ui节点 |
master02 | 172.24.10.12 | Kubernetes master节点 |
kubeadm、kubelet、kubectl、KeepAlived、 containerd、etcd、kube-apiserver、kube-scheduler、 kube-controller-manager、calico、WebUI、metrics、ingress、Longhorn ui节点 |
master03 | 172.24.10.13 | Kubernetes master节点 |
kubeadm、kubelet、kubectl、KeepAlived、 containerd、etcd、kube-apiserver、kube-scheduler、 kube-controller-manager、calico、WebUI、metrics、ingress、Longhorn ui节点 |
worker01 | 172.24.10.14 | Kubernetes worker节点 | kubelet、containerd、calico、Longhorn存储节点 |
worker02 | 172.24.10.15 | Kubernetes worker节点 | kubelet、containerd、calico、Longhorn存储节点 |
worker03 | 172.24.10.16 | Kubernetes worker节点 | kubelet、containerd、calico、Longhorn存储节点 |
worker04 | 172.24.10.17 | Kubernetes worker节点 | kubelet、containerd、calico、Longhorn存储节点 |
Kubernetes集群高可用主要指的是控制平面的高可用,多个Master节点组件(通常为奇数)和Etcd组件的高可用,worker节点通过前端负载均衡VIP连接到Master。
Kubernetes高可用架构中etcd与Master节点组件混合部署方式特点:
提示:本实验使用Keepalived+Nginx架构实现Kubernetes的高可用。
需要对所有节点主机名进行相应配置。
[root@localhost ~]# hostnamectl set-hostname master01 #其他节点依次修改
生产环境通常建议在内网部署dns服务器,使用dns服务器进行解析,本指南采用本地hosts文件名进行解析。
如下hosts文件修改仅需在master01执行,后续使用批量分发至其他所有节点。
[root@master01 ~]# cat >> /etc/hosts << EOF
172.24.10.11 master01
172.24.10.12 master02
172.24.10.13 master03
172.24.10.14 worker01
172.24.10.15 worker02
172.24.10.16 worker03
EOF
为实现自动化部署,自动化分发相关文件,提前定义相关主机名、IP组、变量等。
[root@master01 ~]# wget http://down.linuxsb.com/mydeploy/k8s/v1.30.3/environment.sh
[root@master01 ~]# vi environment.sh #确认相关主机名和IP
#!/bin/sh
#****************************************************************#
# ScriptName: environment.sh
# Author: xhy
# Create Date: 2022-10-11 17:10
# Modify Author: xhy
# Modify Date: 2023-11-30 23:00
# Version: v1
#***************************************************************#
# 集群 MASTER 机器 IP 数组
export MASTER_IPS=(172.24.10.11 172.24.10.12 172.24.10.13)
# 集群 MASTER IP 对应的主机名数组
export MASTER_NAMES=(master01 master02 master03)
# 集群 NODE 机器 IP 数组
export NODE_IPS=(172.24.10.14 172.24.10.15 172.24.10.16)
# 集群 NODE IP 对应的主机名数组
export NODE_NAMES=(worker01 worker02 worker03)
# 集群所有机器 IP 数组
export ALL_IPS=(172.24.10.11 172.24.10.12 172.24.10.13 172.24.10.14 172.24.10.15 172.24.10.16)
# 集群所有IP 对应的主机名数组
export ALL_NAMES=(master01 master02 master03 worker01 worker02 worker03)
为了方便远程分发文件和执行命令,本方案配置master01节点到其它节点的 ssh信任关系,即免秘钥管理所有其他节点。
[root@master01 ~]# source environment.sh #载入变量
[root@master01 ~]# ssh-keygen -f ~/.ssh/id_rsa -N ''
[root@master01 ~]# for all_ip in ${ALL_IPS[@]}
do
echo ">>> ${all_ip}"
ssh-copy-id -i ~/.ssh/id_rsa.pub root@${all_ip}
done
[root@master01 ~]# for all_name in ${ALL_NAMES[@]}
do
echo ">>> ${all_name}"
ssh-copy-id -i ~/.ssh/id_rsa.pub root@${all_name}
done
提示:此操作仅需要在master01节点操作。
kubeadm本身仅用于部署Kubernetes集群,在正式使用kubeadm部署Kubernetes集群之前需要对操作系统环境进行准备,即环境初始化准备。
环境的初始化准备本方案使用脚本自动完成。
使用如下脚本对基础环境进行初始化,主要功能包括:
[root@master01 ~]# wget http://down.linuxsb.com/mydeploy/k8s/v1.30.3/k8sconinit.sh
[root@master01 ~]# vim k8sconinit.sh
#!/bin/sh
#****************************************************************#
# ScriptName: k8sconinit.sh
# Author: xhy
# Create Date: 2020-05-30 16:30
# Modify Author: xhy
# Modify Date: 2024-02-28 22:38
# Version: v1
#***************************************************************#
# Initialize the machine. This needs to be executed on every machine.
rm -f /var/lib/rpm/__db.00*
rpm -vv --rebuilddb
#yum clean all
#yum makecache
sleep 3s
# Install containerd
CONVERSION=1.6.32
yum -y install yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
sudo sed -i 's+download.docker.com+mirrors.aliyun.com/docker-ce+' /etc/yum.repos.d/docker-ce.repo
sleep 3s
yum -y install containerd.io-${CONVERSION}
mkdir /etc/containerd
cat > /etc/containerd/config.toml <<EOF
disabled_plugins = ["restart"]
[plugins.linux]
shim_debug = true
[plugin."io.containerd.grpc.v1.cri".registry.mirrors]
[plugin."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://dbzucv6w.mirror.aliyuncs.com"]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = true
[plugins.cri]
sandbox_image = "registry.k8s.io/pause:3.9"
EOF
cat > /etc/crictl.yaml <<EOF
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
EOF
systemctl restart containerd
systemctl enable containerd --now
systemctl status containerd
# Disable the SELinux.
sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config
# Turn off and disable the firewalld.
systemctl stop firewalld
systemctl disable firewalld
# Modify related kernel parameters & Disable the swap.
cat > /etc/sysctl.d/k8s.conf << EOF
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.tcp_tw_recycle = 0
vm.swappiness = 0
vm.overcommit_memory = 1
vm.panic_on_oom = 0
net.ipv6.conf.all.disable_ipv6 = 1
EOF
sysctl -p /etc/sysctl.d/k8s.conf >&/dev/null
swapoff -a
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
modprobe br_netfilter
modprobe overlay
# Add ipvs modules
cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
modprobe -- nf_conntrack
modprobe -- br_netfilter
modprobe -- overlay
EOF
chmod 755 /etc/sysconfig/modules/ipvs.modules
bash /etc/sysconfig/modules/ipvs.modules
# Install rpm
yum install -y conntrack ipvsadm ipset jq iptables curl sysstat libseccomp wget iproute-tc
# Update kernel
# rpm --import http://down.linuxsb.com/RPM-GPG-KEY-elrepo.org
# rpm -Uvh http://down.linuxsb.com/elrepo-release-7.el7.elrepo.noarch.rpm
# mv -b /etc/yum.repos.d/elrepo.repo /etc/yum.repos.d/backup
# wget -c http://down.linuxsb.com/myoptions/elrepo7.repo -O /etc/yum.repos.d/elrepo.repo
# yum --disablerepo="*" --enablerepo="elrepo-kernel" install -y kernel-ml
# sed -i 's/^GRUB_DEFAULT=.*/GRUB_DEFAULT=0/' /etc/default/grub
# grub2-mkconfig -o /boot/grub2/grub.cfg
# yum -y --exclude=docker* update
# Reboot the machine.
# reboot
提示:此操作仅需要在master01节点操作。
[root@master01 ~]# source environment.sh
[root@master01 ~]# chmod +x *.sh
[root@master01 ~]# for all_ip in ${ALL_IPS[@]}
do
echo ">>> ${all_ip}"
scp -rp /etc/hosts root@${all_ip}:/etc/hosts
scp -rp k8sconinit.sh root@${all_ip}:/root/
ssh root@${all_ip} "bash /root/k8sconinit.sh"
done
HAProxy是可提供高可用性、负载均衡以及基于TCP(从而可以反向代理kubeapiserver等应用)和HTTP应用的代理,支持虚拟主机,它是免费、快速并且可靠的一种高可用解决方案。
[root@master01 ~]# wget https://mirrors.huaweicloud.com/haproxy/3.0/src/haproxy-3.0.3.tar.gz
[root@master01 ~]# for master_ip in ${MASTER_IPS[@]}
do
echo ">>> ${master_ip}"
ssh root@${master_ip} "yum -y install gcc gcc-c++ make libnl3 libnl3-devel libnfnetlink openssl-devel wget openssh-clients systemd-devel zlib-devel pcre-devel"
scp -rp haproxy-3.0.3.tar.gz root@${master_ip}:/root/
ssh root@${master_ip} "tar -zxvf haproxy-3.0.3.tar.gz"
ssh root@${master_ip} "cd haproxy-3.0.3/ && make ARCH=x86_64 TARGET=linux-glibc USE_PCRE=1 USE_ZLIB=1 USE_SYSTEMD=1 PREFIX=/usr/local/haprpxy && make install PREFIX=/usr/local/haproxy"
ssh root@${master_ip} "cp /usr/local/haproxy/sbin/haproxy /usr/sbin/"
ssh root@${master_ip} "useradd -r haproxy && usermod -G haproxy haproxy"
ssh root@${master_ip} "mkdir -p /etc/haproxy && mkdir -p /etc/haproxy/conf.d && cp -r /root/haproxy-3.0.3/examples/errorfiles/ /usr/local/haproxy/"
done
提示:Haproxy官方参考:
https://docs.haproxy.org/
。
KeepAlived 是一个基于VRRP协议来实现的LVS服务高可用方案,可以解决静态路由出现的单点故障问题。
本方案3台master节点均部署并运行Keepalived,一台为主服务器(MASTER),另外两台为备份服务器(BACKUP)。
Master集群外表现为一个VIP,主服务器会发送特定的消息给备份服务器,当备份服务器收不到这个消息的时候,即主服务器宕机的时候,备份服务器就会接管虚拟IP,继续提供服务,从而保证了高可用性。
[root@master01 ~]# wget https://www.keepalived.org/software/keepalived-2.3.1.tar.gz
[root@master01 ~]# for master_ip in ${MASTER_IPS[@]}
do
echo ">>> ${master_ip}"
ssh root@${master_ip} "yum -y install curl gcc gcc-c++ make libnl3 libnl3-devel libnfnetlink openssl-devel"
scp -rp keepalived-2.3.1.tar.gz root@${master_ip}:/root/
ssh root@${master_ip} "tar -zxvf keepalived-2.3.1.tar.gz"
ssh root@${master_ip} "cd keepalived-2.3.1/ && LDFLAGS=\"$LDFAGS -L /usr/local/openssl/lib/\" ./configure --sysconf=/etc --prefix=/usr/local/keepalived && make && make install"
ssh root@${master_ip} "systemctl enable keepalived"
done
提示:如上仅需Master01节点操作,从而实现所有节点自动化安装。若出现如下报错:undefined reference to `OPENSSL_init_ssl’,可带上openssl lib路径:
LDFLAGS="$LDFAGS -L /usr/local/openssl/lib/" ./configure --sysconf=/etc --prefix=/usr/local/keepalived
提示:KeepAlive官方参考:
https://www.keepalived.org/manpage.html
。
创建集群部署所需的相关组件配置,采用脚本自动化创建相关配置文件。
[root@master01 ~]# wget http://down.linuxsb.com/mydeploy/k8s/v1.30.3/k8sconfig.sh #拉取自动部署脚本
[root@master01 ~]# vim k8sconfig.sh
#!/bin/sh
#****************************************************************#
# ScriptName: k8sconfig
# Author: xhy
# Create Date: 2022-06-08 20:00
# Modify Author: xhy
# Modify Date: 2024-02-25 23:57
# Version: v3
#***************************************************************#
#######################################
# set variables below to create the config files, all files will create at ./kubeadm directory
#######################################
# master keepalived virtual ip address
export K8SHA_VIP=172.24.10.100
# master01 ip address
export K8SHA_IP1=172.24.10.11
# master02 ip address
export K8SHA_IP2=172.24.10.12
# master03 ip address
export K8SHA_IP3=172.24.10.13
# master01 hostname
export K8SHA_HOST1=master01
# master02 hostname
export K8SHA_HOST2=master02
# master03 hostname
export K8SHA_HOST3=master03
# master01 network interface name
export K8SHA_NETINF1=eth0
# master02 network interface name
export K8SHA_NETINF2=eth0
# master03 network interface name
export K8SHA_NETINF3=eth0
# keepalived auth_pass config
export K8SHA_KEEPALIVED_AUTH=412f7dc3bfed32194d1600c483e10ad1d
# kubernetes CIDR pod subnet
export K8SHA_PODCIDR=10.10.0.0/16
# kubernetes CIDR svc subnet
export K8SHA_SVCCIDR=10.20.0.0/16
# kubernetes CIDR pod mtu
export K8SHA_PODMTU=1450
##############################
# please do not modify anything below
##############################
mkdir -p kubeadm/$K8SHA_HOST1/{keepalived,haproxy}
mkdir -p kubeadm/$K8SHA_HOST2/{keepalived,haproxy}
mkdir -p kubeadm/$K8SHA_HOST3/{keepalived,haproxy}
mkdir -p kubeadm/keepalived
mkdir -p kubeadm/haproxy
echo "create directory files success."
# wget all files
wget -c -P kubeadm/keepalived/ http://down.linuxsb.com/mydeploy/k8s/common/k8s-keepalived.conf.tpl
wget -c -P kubeadm/keepalived/ http://down.linuxsb.com/mydeploy/k8s/common/check_apiserver.sh
wget -c -P kubeadm/haproxy/ http://down.linuxsb.com/mydeploy/k8s/common/k8s-haproxy.cfg.tpl
wget -c -P kubeadm/haproxy/ http://down.linuxsb.com/mydeploy/k8s/common/k8s-haproxy.service
wget -c -P kubeadm/ http://down.linuxsb.com/mydeploy/k8s/v1.30.3/kubeadm-config.yaml.tpl
wget -c -P kubeadm/calico/ http://down.linuxsb.com/mydeploy/k8s/calico/v3.28.1/calico.yaml.tpl
wget -c -P kubeadm/ http://down.linuxsb.com/mydeploy/k8s/v1.30.3/k8simage.sh
echo "down files success."
# create all kubeadm-config.yaml files
sed \
-e "s/K8SHA_HOST1/${K8SHA_HOST1}/g" \
-e "s/K8SHA_HOST2/${K8SHA_HOST2}/g" \
-e "s/K8SHA_HOST3/${K8SHA_HOST3}/g" \
-e "s/K8SHA_IP1/${K8SHA_IP1}/g" \
-e "s/K8SHA_IP2/${K8SHA_IP2}/g" \
-e "s/K8SHA_IP3/${K8SHA_IP3}/g" \
-e "s/K8SHA_VIP/${K8SHA_VIP}/g" \
-e "s!K8SHA_PODCIDR!${K8SHA_PODCIDR}!g" \
-e "s!K8SHA_SVCCIDR!${K8SHA_SVCCIDR}!g" \
kubeadm/kubeadm-config.yaml.tpl > kubeadm/kubeadm-config.yaml
echo "create kubeadm-config.yaml files success."
# create all keepalived files
chmod u+x kubeadm/keepalived/check_apiserver.sh
cp kubeadm/keepalived/check_apiserver.sh kubeadm/$K8SHA_HOST1/keepalived
cp kubeadm/keepalived/check_apiserver.sh kubeadm/$K8SHA_HOST2/keepalived
cp kubeadm/keepalived/check_apiserver.sh kubeadm/$K8SHA_HOST3/keepalived
sed \
-e "s/K8SHA_KA_STATE/BACKUP/g" \
-e "s/K8SHA_KA_INTF/${K8SHA_NETINF1}/g" \
-e "s/K8SHA_IPLOCAL/${K8SHA_IP1}/g" \
-e "s/K8SHA_KA_PRIO/102/g" \
-e "s/K8SHA_VIP/${K8SHA_VIP}/g" \
-e "s/K8SHA_KA_AUTH/${K8SHA_KEEPALIVED_AUTH}/g" \
kubeadm/keepalived/k8s-keepalived.conf.tpl > kubeadm/$K8SHA_HOST1/keepalived/keepalived.conf
sed \
-e "s/K8SHA_KA_STATE/BACKUP/g" \
-e "s/K8SHA_KA_INTF/${K8SHA_NETINF2}/g" \
-e "s/K8SHA_IPLOCAL/${K8SHA_IP2}/g" \
-e "s/K8SHA_KA_PRIO/101/g" \
-e "s/K8SHA_VIP/${K8SHA_VIP}/g" \
-e "s/K8SHA_KA_AUTH/${K8SHA_KEEPALIVED_AUTH}/g" \
kubeadm/keepalived/k8s-keepalived.conf.tpl > kubeadm/$K8SHA_HOST2/keepalived/keepalived.conf
sed \
-e "s/K8SHA_KA_STATE/BACKUP/g" \
-e "s/K8SHA_KA_INTF/${K8SHA_NETINF3}/g" \
-e "s/K8SHA_IPLOCAL/${K8SHA_IP3}/g" \
-e "s/K8SHA_KA_PRIO/100/g" \
-e "s/K8SHA_VIP/${K8SHA_VIP}/g" \
-e "s/K8SHA_KA_AUTH/${K8SHA_KEEPALIVED_AUTH}/g" \
kubeadm/keepalived/k8s-keepalived.conf.tpl > kubeadm/$K8SHA_HOST3/keepalived/keepalived.conf
echo "create keepalived files success. kubeadm/$K8SHA_HOST1/keepalived/"
echo "create keepalived files success. kubeadm/$K8SHA_HOST2/keepalived/"
echo "create keepalived files success. kubeadm/$K8SHA_HOST3/keepalived/"
# create all haproxy files
sed \
-e "s/K8SHA_IP1/$K8SHA_IP1/g" \
-e "s/K8SHA_IP2/$K8SHA_IP2/g" \
-e "s/K8SHA_IP3/$K8SHA_IP3/g" \
-e "s/K8SHA_HOST1/$K8SHA_HOST1/g" \
-e "s/K8SHA_HOST2/$K8SHA_HOST2/g" \
-e "s/K8SHA_HOST3/$K8SHA_HOST3/g" \
kubeadm/haproxy/k8s-haproxy.cfg.tpl > kubeadm/haproxy/haproxy.conf
echo "create haproxy files success. kubeadm/$K8SHA_HOST1/haproxy/"
echo "create haproxy files success. kubeadm/$K8SHA_HOST2/haproxy/"
echo "create haproxy files success. kubeadm/$K8SHA_HOST3/haproxy/"
# create calico yaml file
sed \
-e "s!K8SHA_PODCIDR!${K8SHA_PODCIDR}!g" \
-e "s!K8SHA_PODMTU!${K8SHA_PODMTU}!g" \
kubeadm/calico/calico.yaml.tpl > kubeadm/calico/calico.yaml
echo "create calico file success."
# scp all file
scp -rp kubeadm/haproxy/haproxy.conf root@$K8SHA_HOST1:/etc/haproxy/haproxy.cfg
scp -rp kubeadm/haproxy/haproxy.conf root@$K8SHA_HOST2:/etc/haproxy/haproxy.cfg
scp -rp kubeadm/haproxy/haproxy.conf root@$K8SHA_HOST3:/etc/haproxy/haproxy.cfg
scp -rp kubeadm/haproxy/k8s-haproxy.service root@$K8SHA_HOST1:/usr/lib/systemd/system/haproxy.service
scp -rp kubeadm/haproxy/k8s-haproxy.service root@$K8SHA_HOST2:/usr/lib/systemd/system/haproxy.service
scp -rp kubeadm/haproxy/k8s-haproxy.service root@$K8SHA_HOST3:/usr/lib/systemd/system/haproxy.service
scp -rp kubeadm/$K8SHA_HOST1/keepalived/* root@$K8SHA_HOST1:/etc/keepalived/
scp -rp kubeadm/$K8SHA_HOST2/keepalived/* root@$K8SHA_HOST2:/etc/keepalived/
scp -rp kubeadm/$K8SHA_HOST3/keepalived/* root@$K8SHA_HOST3:/etc/keepalived/
echo "scp haproxy & keepalived file success."
chmod u+x kubeadm/*.sh
[root@master01 ~]# bash k8sconfig.sh
解释:如上仅需Master01节点操作。执行k8sconfig.sh脚本后会生产如下配置文件清单:
[root@master01 ~]# vim kubeadm/kubeadm-config.yaml #检查集群初始化配置
---
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
networking:
serviceSubnet: "10.20.0.0/16" #设置svc网段
podSubnet: "10.10.0.0/16" #设置Pod网段
dnsDomain: "cluster.local"
kubernetesVersion: "v1.30.3" #设置安装版本
controlPlaneEndpoint: "172.24.10.100:16443" #设置相关API VIP地址
apiServer:
certSANs:
- 127.0.0.1
- master01
- master02
- master03
- 172.24.10.11
- 172.24.10.12
- 172.24.10.13
- 172.24.10.100
timeoutForControlPlane: 4m0s
certificatesDir: "/etc/kubernetes/pki"
imageRepository: "registry.k8s.io"
#clusterName: "example-cluster"
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs
提示:如上仅需Master01节点操作,更多config文件参考:
kubeadm 配置 (v1beta3)
默认kubeadm配置可使用kubeadm config print init-defaults > config.yaml生成。
启动keepalive和HAProxy服务,从而构建master节点的高可用。
[root@master01 ~]# cat /etc/keepalived/keepalived.conf #所有节点确认相关keepalive配置文件
! Configuration File for keepalived
global_defs {
router_id LVS_DEVEL
script_user root
enable_script_security
}
vrrp_script check_apiserver {
script "/etc/keepalived/check_apiserver.sh"
interval 5
weight -60
fall 2
rise 2
}
vrrp_instance VI_1 {
state BACKUP
interface eth0
mcast_src_ip 172.24.10.11
virtual_router_id 51
priority 102
advert_int 5
authentication {
auth_type PASS
auth_pass 412f7dc3bfed32194d1600c483e10ad1d
}
virtual_ipaddress {
172.24.10.100
}
track_script {
check_apiserver
}
}
[root@master01 ~]# cat /etc/keepalived/check_apiserver.sh #所有节点确认相关keepalive监测脚本文件
#!/bin/bash
# if check error then repeat check for 12 times, else exit
err=0
for k in $(seq 1 12)
do
check_code=$(curl -k https://localhost:6443)
if [[ $check_code == "" ]]; then
err=$(expr $err + 1)
sleep 5
continue
else
err=0
break
fi
done
if [[ $err != "0" ]]; then
# if apiserver is down send SIG=1
echo 'apiserver error!'
exit 1
else
# if apiserver is up send SIG=0
echo 'apiserver normal!'
exit 0
fi
[root@master01 ~]# for master_ip in ${MASTER_IPS[@]}
do
echo ">>> ${master_ip}"
ssh root@${master_ip} "systemctl enable haproxy.service --now && systemctl restart haproxy.service"
ssh root@${master_ip} "systemctl enable keepalived.service --now && systemctl restart keepalived.service"
ssh root@${master_ip} "systemctl status keepalived.service | grep Active"
ssh root@${master_ip} "systemctl status haproxy.service | grep Active"
done
[root@master01 ~]# for all_ip in ${ALL_IPS[@]}
do
echo ">>> ${all_ip}"
ssh root@${all_ip} "ping -c1 172.24.10.100"
done #等待10s执行检查
提示:如上仅需Master01节点操作,从而实现所有节点自动启动服务。
需要在每台机器上都安装以下的软件包:
kubeadm不能安装或管理 kubelet 或 kubectl ,因此在初始化集群之前必须完成kubelet和kubectl的安装,且能保证他们满足通过 kubeadm 安装的 Kubernetes控制层对版本的要求。
如果版本没有满足匹配要求,可能导致一些意外错误或问题。
具体相关组件安装见;
附001.kubectl介绍及使用书
提示:Kubernetes 1.29.2版本所有兼容相应组件的版本参考:
https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.29.md
。
快速安装所有节点的kubeadm、kubelet、kubectl组件。
[root@master01 ~]# for all_ip in ${ALL_IPS[@]}
do
echo ">>> ${all_ip}"
ssh root@${all_ip} "cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.30/rpm/
enabled=1
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.30/rpm/repodata/repomd.xml.key
EOF"
ssh root@${all_ip} "yum install -y kubelet-1.30.3-150500.1.1 kubectl-1.30.3-150500.1.1 --disableexcludes=kubernetes"
ssh root@${all_ip} "yum install -y kubeadm-1.30.3-150500.1.1 --disableexcludes=kubernetes"
ssh root@${all_ip} "systemctl enable kubelet"
done
[root@master01 ~]# yum search -y kubelet --showduplicates #查看相应版本
提示:如上仅需Master01节点操作,从而实现所有节点自动化安装,同时此时不需要启动kubelet,初始化的过程中会自动启动的,如果此时启动了会出现报错,忽略即可。
说明:同时安装了cri-tools, kubernetes-cni, socat三个依赖:
socat:kubelet的依赖;
cri-tools:即CRI(Container Runtime Interface)容器运行时接口的命令行工具。
初始化过程中会pull大量镜像,并且镜像位于国外,可能出现无法pull的情况导致Kubernetes初始化失败。建议提前准备镜像,保证后续初始化。
[root@master01 ~]# kubeadm --kubernetes-version=v1.30.3 config images list #列出所需镜像
[root@master01 ~]# vim kubeadm/k8simage.sh
#!/bin/sh
#***************************************************************#
# ScriptName: v1.30.3/k8simage.sh
# Author: xhy
# Create Date: 2024-08-08 22:00
# Modify Author: xhy
# Modify Date: 2024-08-08 22:00
# Version: v1
#***************************************************************#
KUBE_VERSION=v1.30.3
KUBE_PAUSE_VERSION=3.9
ETCD_VERSION=3.5.12-0
CORE_DNS_VERSION=v1.11.1
K8S_URL=registry.k8s.io
UCLOUD_URL=uhub.service.ucloud.cn/imxhy
LONGHORN_URL=longhornio
CALICO_URL='docker.io/calico'
CALICO_VERSION=v3.28.1
METRICS_SERVER_VERSION=v0.7.1
INGRESS_VERSION=v1.11.1
INGRESS_WEBHOOK_VERSION=v1.4.1
LONGHORN_VERSION=v1.6.2
LONGHORN_VERSION2=v0.0.37
CSI_ATTACHER_VERSION=v4.5.1
CSI_NODE_DRIVER_VERSION=v2.9.2
CSI_PROVISIONER_VERSION=v3.6.4
CSI_RESIZER_VERSION=v1.10.1
CSI_SNAP_VERSION=v6.3.4
CSI_LIVE_VERSION=v2.12.0
mkdir -p k8simages/
# config node hostname
export ALL_IPS=(master02 master03 worker01 worker02 worker03)
kubeimages=(kube-proxy:${KUBE_VERSION}
kube-scheduler:${KUBE_VERSION}
kube-controller-manager:${KUBE_VERSION}
kube-apiserver:${KUBE_VERSION}
pause:${KUBE_PAUSE_VERSION}
etcd:${ETCD_VERSION}
)
for kubeimageName in ${kubeimages[@]} ; do
echo ${kubeimageName}
ctr -n k8s.io images pull ${UCLOUD_URL}/${kubeimageName}
ctr -n k8s.io images tag ${UCLOUD_URL}/${kubeimageName} ${K8S_URL}/${kubeimageName}
ctr -n k8s.io images rm ${UCLOUD_URL}/${kubeimageName}
ctr -n k8s.io images export k8simages/${kubeimageName}\.tar ${K8S_URL}/${kubeimageName}
done
corednsimages=(coredns:${CORE_DNS_VERSION}
)
for corednsimageName in ${corednsimages[@]} ; do
echo ${corednsimageName}
ctr -n k8s.io images pull ${UCLOUD_URL}/${corednsimageName}
ctr -n k8s.io images tag ${UCLOUD_URL}/${corednsimageName} ${K8S_URL}/coredns/${corednsimageName}
ctr -n k8s.io images rm ${UCLOUD_URL}/${corednsimageName}
ctr -n k8s.io images export k8simages/${corednsimageName}\.tar ${K8S_URL}/coredns/${corednsimageName}
done
calimages=(cni:${CALICO_VERSION}
node:${CALICO_VERSION}
kube-controllers:${CALICO_VERSION})
for calimageName in ${calimages[@]} ; do
echo ${calimageName}
ctr -n k8s.io images pull ${UCLOUD_URL}/${calimageName}
ctr -n k8s.io images tag ${UCLOUD_URL}/${calimageName} ${CALICO_URL}/${calimageName}
ctr -n k8s.io images rm ${UCLOUD_URL}/${calimageName}
ctr -n k8s.io images export k8simages/${calimageName}\.tar ${CALICO_URL}/${calimageName}
done
metricsimages=(metrics-server:${METRICS_SERVER_VERSION})
for metricsimageName in ${metricsimages[@]} ; do
echo ${metricsimageName}
ctr -n k8s.io images pull ${UCLOUD_URL}/${metricsimageName}
ctr -n k8s.io images tag ${UCLOUD_URL}/${metricsimageName} ${K8S_URL}/metrics-server/${metricsimageName}
ctr -n k8s.io images rm ${UCLOUD_URL}/${metricsimageName}
ctr -n k8s.io images export k8simages/${metricsimageName}\.tar ${K8S_URL}/metrics-server/${metricsimageName}
done
ingressimages=(controller:${INGRESS_VERSION}
kube-webhook-certgen:${INGRESS_WEBHOOK_VERSION}
)
for ingressimageName in ${ingressimages[@]} ; do
echo ${ingressimageName}
ctr -n k8s.io images pull ${UCLOUD_URL}/${ingressimageName}
ctr -n k8s.io images tag ${UCLOUD_URL}/${ingressimageName} ${K8S_URL}/ingress-nginx/${ingressimageName}
ctr -n k8s.io images rm ${UCLOUD_URL}/${ingressimageName}
ctr -n k8s.io images export k8simages/${ingressimageName}\.tar ${K8S_URL}/ingress-nginx/${ingressimageName}
done
longhornimages01=(longhorn-engine:${LONGHORN_VERSION}
longhorn-instance-manager:${LONGHORN_VERSION}
longhorn-manager:${LONGHORN_VERSION}
longhorn-ui:${LONGHORN_VERSION}
backing-image-manager:${LONGHORN_VERSION}
longhorn-share-manager:${LONGHORN_VERSION}
)
for longhornimageNameA in ${longhornimages01[@]} ; do
echo ${longhornimageNameA}
ctr -n k8s.io images pull ${UCLOUD_URL}/${longhornimageNameA}
ctr -n k8s.io images tag ${UCLOUD_URL}/${longhornimageNameA} ${LONGHORN_URL}/${longhornimageNameA}
ctr -n k8s.io images rm ${UCLOUD_URL}/${longhornimageNameA}
ctr -n k8s.io images export k8simages/${longhornimageNameA}\.tar ${LONGHORN_URL}/${longhornimageNameA}
done
longhornimages02=(support-bundle-kit:${LONGHORN_VERSION2})
for longhornimageNameB in ${longhornimages02[@]} ; do
echo ${longhornimageNameB}
ctr -n k8s.io images pull ${UCLOUD_URL}/${longhornimageNameB}
ctr -n k8s.io images tag ${UCLOUD_URL}/${longhornimageNameB} ${LONGHORN_URL}/${longhornimageNameB}
ctr -n k8s.io images rm ${UCLOUD_URL}/${longhornimageNameB}
ctr -n k8s.io images export k8simages/${longhornimageNameB}\.tar ${LONGHORN_URL}/${longhornimageNameB}
done
csiimages=(csi-attacher:${CSI_ATTACHER_VERSION}
csi-node-driver-registrar:${CSI_NODE_DRIVER_VERSION}
csi-provisioner:${CSI_PROVISIONER_VERSION}
csi-resizer:${CSI_RESIZER_VERSION}
csi-snapshotter:${CSI_SNAP_VERSION}
livenessprobe:${CSI_LIVE_VERSION}
)
for csiimageName in ${csiimages[@]} ; do
echo ${csiimageName}
ctr -n k8s.io images pull ${UCLOUD_URL}/${csiimageName}
ctr -n k8s.io images tag ${UCLOUD_URL}/${csiimageName} ${LONGHORN_URL}/${csiimageName}
ctr -n k8s.io images rm ${UCLOUD_URL}/${csiimageName}
ctr -n k8s.io images export k8simages/${csiimageName}\.tar ${LONGHORN_URL}/${csiimageName}
done
allimages=(kube-proxy:${KUBE_VERSION}
kube-scheduler:${KUBE_VERSION}
kube-controller-manager:${KUBE_VERSION}
kube-apiserver:${KUBE_VERSION}
pause:${KUBE_PAUSE_VERSION}
etcd:${ETCD_VERSION}
coredns:${CORE_DNS_VERSION}
cni:${CALICO_VERSION}
node:${CALICO_VERSION}
kube-controllers:${CALICO_VERSION}
metrics-server:${METRICS_SERVER_VERSION}
controller:${INGRESS_VERSION}
kube-webhook-certgen:${INGRESS_WEBHOOK_VERSION}
longhorn-engine:${LONGHORN_VERSION}
longhorn-instance-manager:${LONGHORN_VERSION}
longhorn-manager:${LONGHORN_VERSION}
longhorn-ui:${LONGHORN_VERSION}
backing-image-manager:${LONGHORN_VERSION}
longhorn-share-manager:${LONGHORN_VERSION}
support-bundle-kit:${LONGHORN_VERSION2}
csi-attacher:${CSI_ATTACHER_VERSION}
csi-node-driver-registrar:${CSI_NODE_DRIVER_VERSION}
csi-provisioner:${CSI_PROVISIONER_VERSION}
csi-resizer:${CSI_RESIZER_VERSION}
csi-snapshotter:${CSI_SNAP_VERSION}
livenessprobe:${CSI_LIVE_VERSION}
)
for all_ip in ${ALL_IPS[@]}
do
echo ">>> ${all_ip}"
ssh root@${all_ip} "mkdir /root/k8simages"
scp -rp k8simages/* root@${all_ip}:/root/k8simages/
done
for allimageName in ${allimages[@]}
do
for all_ip in ${ALL_IPS[@]}
do
echo "${allimageName} copy to ${all_ip}"
ssh root@${all_ip} "ctr -n k8s.io images import k8simages/${allimageName}\.tar"
done
done
[root@master01 ~]# bash kubeadm/k8simage.sh #确认版本,提前下载镜像
提示:如上仅需Master01节点操作,从而实现所有节点镜像的分发。
注意相关版本,如上脚本为v1.30.3 Kubernetes版本所需镜像。
[root@master01 ~]# ctr -n k8s.io images ls #确认验证
[root@master02 ~]# crictl images ls
IMAGE TAG IMAGE ID SIZE
docker.io/calico/cni v3.28.1 f6d76a1259a8c 94.6MB
docker.io/calico/kube-controllers v3.28.1 9d19dff735fa0 35MB
docker.io/calico/node v3.28.1 8bbeb9e1ee328 118MB
docker.io/longhornio/backing-image-manager v1.6.2 9b8cf5184bda1 133MB
docker.io/longhornio/csi-attacher v4.5.1 ebcde6f69ddda 27.5MB
docker.io/longhornio/csi-node-driver-registrar v2.9.2 438c692b0cb6d 10.8MB
docker.io/longhornio/csi-provisioner v3.6.4 cc753cf7b8127 28.7MB
docker.io/longhornio/csi-resizer v1.10.1 644d77abe33db 28.1MB
docker.io/longhornio/csi-snapshotter v6.3.4 eccecdceb86c0 26.9MB
docker.io/longhornio/livenessprobe v2.12.0 38ae1b6759b01 13.4MB
docker.io/longhornio/longhorn-engine v1.6.2 7fb50a1bbe317 142MB
docker.io/longhornio/longhorn-instance-manager v1.6.2 23292e266e0eb 272MB
docker.io/longhornio/longhorn-manager v1.6.2 6b0b2d18564be 112MB
docker.io/longhornio/longhorn-share-manager v1.6.2 f578840264031 81.1MB
docker.io/longhornio/longhorn-ui v1.6.2 b1c8e3638fc43 75.6MB
docker.io/longhornio/support-bundle-kit v0.0.37 df2168e6bf552 89.3MB
registry.k8s.io/coredns/coredns v1.11.1 cbb01a7bd410d 18.2MB
registry.k8s.io/etcd 3.5.12-0 3861cfcd7c04c 57.2MB
registry.k8s.io/ingress-nginx/controller v1.11.1 5a3c471280784 105MB
registry.k8s.io/ingress-nginx/kube-webhook-certgen v1.4.1 684c5ea3b61b2 23.9MB
registry.k8s.io/kube-apiserver v1.30.3 1f6d574d502f3 32.8MB
registry.k8s.io/kube-controller-manager v1.30.3 76932a3b37d7e 31.1MB
registry.k8s.io/kube-proxy v1.30.3 55bb025d2cfa5 29MB
registry.k8s.io/kube-scheduler v1.30.3 3edc18e7b7672 19.3MB
registry.k8s.io/metrics-server/metrics-server v0.7.1 a24c7c057ec87 19.5MB
registry.k8s.io/pause 3.9 e6f1816883972 319kB
Master01节点上执行初始化,即完成单节点的Kubernetes,其他节点采用添加的方式部署。
提示:kubeadm init过程会执行系统预检查,预检查通过则继续init,也可以提前执行如下命令进行预检查操作:
kubeadm init phase preflight
[root@master01 ~]# kubeadm init --config=kubeadm/kubeadm-config.yaml --upload-certs #保留如下命令用于后续节点添加
[init] Using Kubernetes version: v1.30.3
[preflight] Running pre-flight checks
……
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of the control-plane node running the following command on each as root:
kubeadm join 172.24.10.100:16443 --token 5ogx63.mjfb2mvyyebp30v6 \
--discovery-token-ca-cert-hash sha256:35332dd14dac287b35b85af9fc03bd45af15d14248aa3c255dfc96abb1082021 \
--control-plane --certificate-key 2a3eea130eb22d945cfee660c40a250731a1853e54bbf25ee13c0400d4a04ad1
Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 172.24.10.100:16443 --token 5ogx63.mjfb2mvyyebp30v6 \
--discovery-token-ca-cert-hash sha256:35332dd14dac287b35b85af9fc03bd45af15d14248aa3c255dfc96abb1082021
注意:如上token具有默认24小时的有效期,token和hash值可通过如下方式获取:
kubeadm token list
如果 Token 过期以后,可以输入以下命令,生成新的 Token:
kubeadm token create
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
创建相关Kubernetes集群配置文件保存目录。
[root@master01 ~]# mkdir -p $HOME/.kube
[root@master01 ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@master01 ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config
[root@master01 ~]# cat << EOF >> ~/.bashrc
export KUBECONFIG=$HOME/.kube/config
EOF #设置KUBECONFIG环境变量
[root@master01 ~]# echo "source <(kubectl completion bash)" >> ~/.bashrc
[root@master01 ~]# source ~/.bashrc
附加:初始化过程大致步骤如下:
提示:初始化仅需要在master01上执行,若初始化异常可通过
kubeadm reset -f kubeadm/kubeadm-config.yaml && rm -rf $HOME/.kube /etc/cni/ /etc/kubernetes/
重置。
采用 kubeadm join 将其他Master节点添加至集群。
[root@master02 ~]# kubeadm join 172.24.10.100:16443 --token 5ogx63.mjfb2mvyyebp30v6 \
--discovery-token-ca-cert-hash sha256:35332dd14dac287b35b85af9fc03bd45af15d14248aa3c255dfc96abb1082021 \
--control-plane --certificate-key 2a3eea130eb22d945cfee660c40a250731a1853e54bbf25ee13c0400d4a04ad1
[root@master02 ~]# mkdir -p $HOME/.kube
[root@master02 ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@master02 ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config
[root@master02 ~]# cat << EOF >> ~/.bashrc
export KUBECONFIG=$HOME/.kube/config
EOF #设置KUBECONFIG环境变量
[root@master02 ~]# echo "source <(kubectl completion bash)" >> ~/.bashrc
[root@master02 ~]# source ~/.bashrc
提示:master03也如上操作,添加至当前集群的controlplane。
若添加异常可通过
kubeadm reset -f kubeadm/kubeadm-config.yaml && rm -rf $HOME/.kube /etc/cni/ /etc/kubernetes/
重置。
提示:本方案使用Calico插件。
确认相关配置,如MTU,网卡接口,Pod的IP地址段。
calico原文件可参考官方:
https://raw.githubusercontent.com/projectcalico/calico/v3.27.2/manifests/calico.yaml
[root@master01 ~]# vim kubeadm/calico/calico.yaml #检查配置
……
data:
……
veth_mtu: "1450"
……
- name: CALICO_IPV4POOL_CIDR
value: "10.10.0.0/16" #配置Pod网段
……
- name: IP_AUTODETECTION_METHOD
value: "interface=eth.*" #检查节点之间的网卡
……
[root@master01 ~]# kubectl apply -f kubeadm/calico/calico.yaml
[root@master01 ~]# kubectl get pods --all-namespaces -o wide #查看部署的所有Pod
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system calico-kube-controllers-77d59654f4-ld47h 1/1 Running 0 63s 10.10.59.194 master02 <none> <none>
kube-system calico-node-5x2qt 1/1 Running 0 63s 172.24.10.11 master01 <none> <none>
kube-system calico-node-hk9p6 1/1 Running 0 63s 172.24.10.12 master02 <none> <none>
kube-system calico-node-ttbr5 1/1 Running 0 63s 172.24.10.13 master03 <none> <none>
kube-system coredns-7db6d8ff4d-4swqk 1/1 Running 0 8m 10.10.59.195 master02 <none> <none>
kube-system coredns-7db6d8ff4d-zv4n9 1/1 Running 0 8m 10.10.59.193 master02 <none> <none>
kube-system etcd-master01 1/1 Running 0 8m12s 172.24.10.11 master01 <none> <none>
kube-system etcd-master02 1/1 Running 0 5m38s 172.24.10.12 master02 <none> <none>
kube-system etcd-master03 1/1 Running 0 5m46s 172.24.10.13 master03 <none> <none>
kube-system kube-apiserver-master01 1/1 Running 0 8m12s 172.24.10.11 master01 <none> <none>
kube-system kube-apiserver-master02 1/1 Running 0 5m47s 172.24.10.12 master02 <none> <none>
kube-system kube-apiserver-master03 1/1 Running 0 5m46s 172.24.10.13 master03 <none> <none>
kube-system kube-controller-manager-master01 1/1 Running 0 8m18s 172.24.10.11 master01 <none> <none>
kube-system kube-controller-manager-master02 1/1 Running 0 5m47s 172.24.10.12 master02 <none> <none>
kube-system kube-controller-manager-master03 1/1 Running 0 5m46s 172.24.10.13 master03 <none> <none>
kube-system kube-proxy-98dzr 1/1 Running 0 8m 172.24.10.11 master01 <none> <none>
kube-system kube-proxy-wcgld 1/1 Running 0 5m50s 172.24.10.13 master03 <none> <none>
kube-system kube-proxy-wf4tg 1/1 Running 0 5m50s 172.24.10.12 master02 <none> <none>
kube-system kube-scheduler-master01 1/1 Running 0 8m14s 172.24.10.11 master01 <none> <none>
kube-system kube-scheduler-master02 1/1 Running 0 5m47s 172.24.10.12 master02 <none> <none>
kube-system kube-scheduler-master03 1/1 Running 0 5m45s 172.24.10.13 master03 <none> <none>
[root@master01 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master01 Ready control-plane 8m25s v1.30.3
master02 Ready control-plane 4m52s v1.30.3
master03 Ready control-plane 4m49s v1.30.3
提示:官方calico参考:
https://docs.projectcalico.org/manifests/calico.yaml
Kubernetes默认的端口范围为30000-32767,为便于后期大量的应用,如ingress的80、443端口,可开放全端口。
同时开放全端口范围后,使用的时候需要注意避开公共端口,如8080。
[root@master01 ~]# vi /etc/kubernetes/manifests/kube-apiserver.yaml #追加端口开放配置
……
- --service-node-port-range=1-65535
……
提示:如上需要在所有Master节点操作。
[root@master01 ~]# source environment.sh
[root@master01 ~]# for node_ip in ${NODE_IPS[@]}
do
echo ">>> ${node_ip}"
ssh root@${node_ip} "kubeadm join 172.24.10.100:16443 --token 5ogx63.mjfb2mvyyebp30v6 \
--discovery-token-ca-cert-hash sha256:35332dd14dac287b35b85af9fc03bd45af15d14248aa3c255dfc96abb1082021"
ssh root@${node_ip} "systemctl enable kubelet.service"
done
提示:如上仅需Master01节点操作,从而实现所有Worker节点添加至集群,若添加异常可通过如下方式重置:
[root@worker01 ~]# kubeadm reset
[root@worker01 ~]# ifconfig kube-ipvs0 down
[root@worker01 ~]# ip link delete kube-ipvs0
[root@worker01 ~]# ifconfig tunl0@NONE down
[root@worker01 ~]# ip link delete tunl0@NONE
[root@worker01 ~]# rm -rf /var/lib/cni/
[root@master01 ~]# kubectl get nodes #节点状态
[root@master01 ~]# kubectl get cs #组件状态
[root@master01 ~]# kubectl get serviceaccount #服务账户
[root@master01 ~]# kubectl cluster-info #集群信息
[root@master01 ~]# kubectl get pod -n kube-system -o wide #所有服务状态
提示:更多Kubetcl使用参考:
https://kubernetes.io/docs/reference/kubectl/kubectl/
https://kubernetes.io/docs/reference/kubectl/overview/
更多kubeadm使用参考:
https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm/
Kubernetes的早期版本依靠Heapster来实现完整的性能数据采集和监控功能,Kubernetes从1.8版本开始,性能数据开始以Metrics API的方式提供标准化接口,并且从1.10版本开始将Heapster替换为Metrics Server。在Kubernetes新的监控体系中,Metrics Server用于提供核心指标(Core Metrics),包括Node、Pod的CPU和内存使用指标,对其他自定义指标(Custom Metrics)的监控则由Prometheus等组件来完成。
Metrics Server是一个可扩展的、高效的容器资源度量,通常可用于Kubernetes内置的自动伸缩,即自动伸缩可依据metrics的度量指标。
Metrics Server从Kubelets收集资源指标,并通过Metrics API将它们暴露在Kubernetes apisserver中,供Pod水平或垂直自动伸缩使用。
kubectl top也可以访问Metrics API,可查看相关对象资源使用情况。
提示:当前官方建议Metrics Server仅用于自动伸缩,不要使用它来当做对Kubernetes的监控解决方案,或者监控解决方案的上游来源,对于完整的Kubernetes监控方案,可直接从Kubelet的/metrics/resource endpoint收集指标。
使用Metrics Server的场景:
基于CPU/内存的水平自动缩放;
自动调整/建议容器所需的资源(了解有关垂直自动缩放的更多信息)
不建议使用Metrics Server的场景:
Metrics Server主要特点:
Metrics Server对集群和网络配置有特定的需求依赖,这些需求依赖并不是所有集群默认开启的。
在使用Metrics Server之前,需要确保集群支持这些需求:
有关聚合层知识参考:
https://blog.csdn.net/liukuan73/article/details/81352637
kubeadm方式部署默认已开启。
根据实际生产环境,对Metrics Server的部署进行个性化修改,其他保持默认即可。
主要涉及:部署副本数为3,追加–kubelet-insecure-tls配置。
[root@master01 ~]# mkdir metrics
[root@master01 ~]# cd metrics/
[root@master01 metrics]# wget https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
[root@master01 metrics]# vi components.yaml
……
apiVersion: apps/v1
kind: Deployment
……
spec:
replicas: 3 #根据集群规模调整副本数
……
spec:
hostNetwork: true #追加此行
containers:
- args:
- --cert-dir=/tmp
- --secure-port=10300 #修改端口 #修改端口
- --kubelet-insecure-tls #追加此行
- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname,InternalDNS,ExternalDNS #修改此args
- --kubelet-use-node-status-port
- --metric-resolution=15s
image: registry.k8s.io/metrics-server/metrics-server:v0.7.1
imagePullPolicy: IfNotPresent
……
ports:
- containerPort: 10300
……
提示:默认的10250会被kubelet当做服务监听的端口,因此建议修改端口。
[root@master01 metrics]# kubectl apply -f components.yaml
[root@master01 metrics]# kubectl -n kube-system get pods -l k8s-app=metrics-server -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
metrics-server-78bd46cc84-lm9r7 1/1 Running 0 42s 172.24.10.15 worker02 <none> <none>
metrics-server-78bd46cc84-qsxtf 1/1 Running 0 112s 172.24.10.14 worker01 <none> <none>
metrics-server-78bd46cc84-zjsn6 1/1 Running 0 78s 172.24.10.16 worker03 <none> <none>
可使用kubectl top查看相关监控项。
[root@master01 ~]# kubectl top nodes
[root@master01 ~]# kubectl top pods --all-namespaces
提示:Metrics Server提供的数据也可以供HPA控制器使用,以实现基于CPU使用率或内存使用值的Pod自动扩缩容功能。\
有关metrics更多部署参考:
https://kubernetes.io/docs/tasks/debug-application-cluster/resource-metrics-pipeline/
开启开启API Aggregation参考:
https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation/
API Aggregation介绍参考:
https://kubernetes.io/docs/tasks/access-kubernetes-api/configure-aggregation-layer/
Kubernetes中的应用通常以Service对外暴露,而Service的表现形式为IP:Port,即工作在TCP/IP层。
对于基于HTTP的服务来说,不同的URL地址经常对应到不同的后端服务(RS)或者虚拟服务器(Virtual Host),这些应用层的转发机制仅通过Kubernetes的Service机制是无法实现的。
从Kubernetes 1.1版本开始新增Ingress资源对象,用于将不同URL的访问请求转发到后端不同的Service,以实现HTTP层的业务路由机制。
Kubernetes使用了一个Ingress策略规则和一个具体的Ingress Controller,两者结合实现了一个完整的Ingress负载均衡器。
使用Ingress进行负载分发时,Ingress Controller基于Ingress策略规则将客户端请求直接转发到Service对应的后端Endpoint(Pod)上,从而跳过kube-proxy的转发功能,kube-proxy不再起作用。
简单的理解就是:ingress使用DaemonSet或Deployment在相应Node上监听80或443,然后配合相应规则,因为Nginx外面绑定了宿主机80端口(就像 NodePort),本身又在集群内,那么向后直接转发到相应ServiceIP即可实现相应需求。
ingress controller + ingress 策略规则 —-> services。
同时当Ingress Controller提供的是对外服务,则实际上实现的是边缘路由器的功能。
典型的HTTP层路由的架构:
建议对于非业务相关的应用,构建集群所需的应用(如Ingress),部署在master节点,从而复用master节点的高可用。
采用标签,结合部署的yaml中的tolerations,实现ingress部署在master节点的配置。
[root@master01 ~]# kubectl label nodes master0{1,2,3} ingress=enable
获取部署所需的yaml资源。
[root@master01 ~]# mkdir ingress
[root@master01 ~]# cd ingress/
[root@master01 ingress]# wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.11.1/deploy/static/provider/baremetal/deploy.yaml
提示:ingress官方参考:
https://github.com/kubernetes/ingress-nginx
https://kubernetes.github.io/ingress-nginx/deploy/
为方便后续管理和排障,对相关Nginx ingress挂载时区,以便使Pod时间正确,从而相关记录日志能具有时效性。
同时对ingress做了简单配置,如日志格式等。
[root@master01 ingress]# vi deploy.yaml
……
---
apiVersion: v1
data:
allow-snippet-annotations: "true"
client-header-buffer-size: "512k" #客户端请求头的缓冲区大小
large-client-header-buffers: "4 512k" #设置用于读取大型客户端请求标头的最大值number和size缓冲区
client-body-buffer-size: "128k" #读取客户端请求body的缓冲区大小
proxy-buffer-size: "256k" #代理缓冲区大小
proxy-body-size: "50m" #代理body大小
server-name-hash-bucket-size: "128" #服务器名称哈希大小
map-hash-bucket-size: "128" #map哈希大小
ssl-ciphers: "ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-R
SA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA
:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA" #SSL加密套件
ssl-protocols: "TLSv1 TLSv1.1 TLSv1.2" #ssl 协议
log-format-upstream: '{"time": "$time_iso8601", "remote_addr": "$proxy_protocol_addr", "x-forward-for": "$proxy_add_x_forwarded_for", "request_id": "$req_id","remote_user": "$remote_user", "bytes_sent": $bytes_sent, "request_time": $request_time, "sta
tus":$status, "vhost": "$host", "request_proto": "$server_protocol", "path": "$uri", "request_query": "$args", "request_length": $request_length, "duration": $request_time,"method": "$request_method", "http_referrer": "$http_referer", "http_user_agent":
"$http_user_agent" }' #日志格式
kind: ConfigMap
……
---
apiVersion: v1
kind: Service
metadata:
……
spec:
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- appProtocol: http
name: http
port: 80
protocol: TCP
targetPort: http
nodePort: 80 #追加此行
- appProtocol: https
name: https
port: 443
protocol: TCP
targetPort: https
nodePort: 443 #追加此行
selector:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
type: NodePort
externalTrafficPolicy: Local #追加此行
……
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.11.1
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
replicas: 3 #配置副本数
……
spec:
containers:
- args:
- /nginx-ingress-controller
……
image: registry.k8s.io/ingress-nginx/controller:v1.11.1 #修改image镜像
……
volumeMounts:
……
- mountPath: /etc/localtime #挂载localtime
name: timeconfig
readOnly: true
dnsPolicy: ClusterFirst
nodeSelector:
kubernetes.io/os: linux
ingress: enable
tolerations:
- key: node-role.kubernetes.io/control-plane
effect: NoSchedule #追加nodeSelector和tolerations
……
volumes:
- name: webhook-cert
secret:
secretName: ingress-nginx-admission
- name: timeconfig #将hostpath配置为挂载卷
hostPath:
path: /etc/localtime
……
image: registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.1 #修改image镜像
……
image: registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.1 #修改image镜像
……
[root@master01 ingress]# kubectl apply -f deploy.yaml
提示:添加默认backend需要等待default-backend创建完成controllers才能成功部署,新版本ingress不再推荐添加default backend。
查看Pod部署进度,是否成功完成。
[root@master01 ingress]# kubectl get pods -n ingress-nginx -o wide
[root@master01 ingress]# kubectl get svc -n ingress-nginx -o wide
提示:参考文档:
https://github.com/kubernetes/ingress-nginx/blob/master/docs/deploy/index.md
。
Longhorn是用于Kubernetes的开源分布式块存储系统。
当前Kubernetes 1.30.3版本建议使用Longhorn 1.6.2 。
提示:更多介绍参考:
https://github.com/longhorn/longhorn
。
后续业务应用可能运行在任意节点位置,挂载操作需要在任何节点可正常执行。
所有节点均需要安装基础以来软件。
[root@master01 ~]# source environment.sh
[root@master01 ~]# for all_ip in ${ALL_IPS[@]}
do
echo ">>> ${all_ip}"
ssh root@${all_ip} "yum -y install iscsi-initiator-utils &"
ssh root@${all_ip} "systemctl enable iscsid --now"
done
提示:所有节点都需要安装。
在Master节点上部署存储组件的图形界面。
[root@master01 ~]# kubectl label nodes master0{1,2,3} longhorn-ui=enabled
提示:ui图形界面可复用master高可用,因此部署在master节点。
Longhorn的分布式存储,建议独立磁盘设备专门作为存储卷,可提前挂载。
longhorn默认使用/var/lib/longhorn/作为设备路径,可提前挂载/dev/nvme0n2设备。
不同环境下裸磁盘的设备名不一样,根据实际环境为准。
[root@master01 ~]# source environment.sh
[root@master01 ~]# for node_ip in ${NODE_IPS[@]}
do
echo ">>> ${node_ip}"
ssh root@${node_ip} "mkfs.xfs -f /dev/nvme0n2 && mkdir -p /var/lib/longhorn/ && echo '/dev/nvme0n2 /var/lib/longhorn xfs defaults 0 0' >> /etc/fstab && mount -a"
done
根据实际生产环境,对Longhorn进行优化配置。
[root@master01 ~]# mkdir longhorn
[root@master01 ~]# cd longhorn/
[root@master01 longhorn]# wget https://raw.githubusercontent.com/longhorn/longhorn/v1.6.2/deploy/longhorn.yaml
[root@master01 longhorn]# vi longhorn.yaml
……
---
# Source: longhorn/templates/deployment-ui.yaml
apiVersion: apps/v1
kind: Deployment
……
spec:
replicas: 2
……
containers:
- name: longhorn-ui
image: longhornio/longhorn-ui:v1.6.0
……
nodeSelector:
longhorn-ui: enabled #追加标签选择
tolerations:
- key: node-role.kubernetes.io/control-plane #添加容忍
effect: NoSchedule
……
基于优化的yaml进行部署。
[root@master01 ~]# cd longhorn/
[root@master01 longhorn]# kubectl apply -f longhorn.yaml
[root@master01 longhorn]# kubectl -n longhorn-system get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
csi-attacher-57689cc84b-55vw6 1/1 Running 0 104s 10.10.19.82 worker03 <none> <none>
csi-attacher-57689cc84b-gxm62 1/1 Running 0 105s 10.10.30.82 worker02 <none> <none>
csi-attacher-57689cc84b-jtqdw 1/1 Running 0 104s 10.10.5.17 worker01 <none> <none>
csi-provisioner-6c78dcb664-cswfz 1/1 Running 0 104s 10.10.19.83 worker03 <none> <none>
csi-provisioner-6c78dcb664-kdnhc 1/1 Running 0 104s 10.10.5.18 worker01 <none> <none>
csi-provisioner-6c78dcb664-vqs4z 1/1 Running 0 104s 10.10.30.81 worker02 <none> <none>
csi-resizer-7466f7b45f-8lbhp 1/1 Running 0 104s 10.10.5.21 worker01 <none> <none>
csi-resizer-7466f7b45f-9g7rw 1/1 Running 0 104s 10.10.30.83 worker02 <none> <none>
csi-resizer-7466f7b45f-xzsgs 1/1 Running 0 104s 10.10.19.81 worker03 <none> <none>
csi-snapshotter-58bf69fbd5-5b59k 1/1 Running 0 104s 10.10.19.85 worker03 <none> <none>
csi-snapshotter-58bf69fbd5-7q25t 1/1 Running 0 104s 10.10.30.85 worker02 <none> <none>
csi-snapshotter-58bf69fbd5-rprpq 1/1 Running 0 104s 10.10.5.19 worker01 <none> <none>
engine-image-ei-acb7590c-9b8wf 1/1 Running 0 116s 10.10.30.79 worker02 <none> <none>
engine-image-ei-acb7590c-bbcw9 1/1 Running 0 116s 10.10.19.79 worker03 <none> <none>
engine-image-ei-acb7590c-d4qlp 1/1 Running 0 116s 10.10.5.15 worker01 <none> <none>
instance-manager-0cf302d46e3eaf0be2c65de14febecb3 1/1 Running 0 110s 10.10.30.80 worker02 <none> <none>
instance-manager-652604acb4423fc91cae625c664b813b 1/1 Running 0 2m21s 10.10.5.14 worker01 <none> <none>
instance-manager-6e47bda67fc7278dee5cbb280e6a8fde 1/1 Running 0 110s 10.10.19.80 worker03 <none> <none>
longhorn-csi-plugin-j92qz 3/3 Running 0 104s 10.10.19.84 worker03 <none> <none>
longhorn-csi-plugin-lsqzb 3/3 Running 0 104s 10.10.5.20 worker01 <none> <none>
longhorn-csi-plugin-nv7vk 3/3 Running 0 104s 10.10.30.84 worker02 <none> <none>
longhorn-driver-deployer-576d574c8-vw8hq 1/1 Running 0 4m53s 10.10.30.78 worker02 <none> <none>
longhorn-manager-2vfpz 1/1 Running 3 (2m16s ago) 4m53s 10.10.30.77 worker02 <none> <none>
longhorn-manager-4d5w9 1/1 Running 2 (2m19s ago) 4m53s 10.10.5.13 worker01 <none> <none>
longhorn-manager-r55wx 1/1 Running 3 (4m26s ago) 4m53s 10.10.19.78 worker03 <none> <none>
longhorn-ui-7cfd57b47d-brh89 1/1 Running 0 4m53s 10.10.59.198 master02 <none> <none>
longhorn-ui-7cfd57b47d-qndks 1/1 Running 0 4m53s 10.10.235.8 master03 <none> <none>
提示:若部署异常可删除重建,若出现无法删除namespace,可通过如下操作进行删除:
wget https://raw.githubusercontent.com/longhorn/longhorn/v1.6.0/uninstall/uninstall.yaml
kubectl apply -f uninstall.yaml
kubectl get job/longhorn-uninstall -n longhorn-system -w
kubectl delete -f uninstall.yaml #等待任务完成再次执行delete
rm -rf /var/lib/longhorn/*
若依旧无法释放,参考《附098.Kubernetes故障排查记录》。
部署Longhorn后,默认已创建一个名为longhorn的sc。
[root@master01 longhorn]# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
longhorn (default) driver.longhorn.io Delete Immediate true 5m53s
使用常见的Nginx Pod进行测试,模拟生产环境常见的Web类应用的持久性存储卷。
[root@master01 longhorn]# cat <<EOF > longhornpvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: longhorn-pvc
spec:
accessModes:
- ReadWriteOnce
storageClassName: longhorn
resources:
requests:
storage: 50Mi
EOF #创建PVC
[root@master01 longhorn]# cat <<EOF > longhornpod.yaml
---
apiVersion: v1
kind: Pod
metadata:
name: longhorn-pod
namespace: default
spec:
containers:
- name: volume-test
image: nginx:stable-alpine
imagePullPolicy: IfNotPresent
volumeMounts:
- name: volv
mountPath: /usr/share/nginx/html
ports:
- containerPort: 80
volumes:
- name: volv
persistentVolumeClaim:
claimName: longhorn-pvc
EOF #创建Pod
[root@master01 longhorn]# kubectl apply -f longhornpvc.yaml
[root@master01 longhorn]# kubectl apply -f longhornpod.yaml
[root@master01 longhorn]# kubectl get pods
[root@master01 longhorn]# kubectl get pvc
[root@master01 longhorn]# kubectl get pv
使用已部署完成的ingress将Longhorn UI暴露,以便于使用URL形式访问Longhorn图形界面进行Longhorn的基础管理。
[root@master01 longhorn]# yum -y install httpd-tools
[root@master01 longhorn]# htpasswd -c auth admin #创建用户名和密码
New password: [输入密码]
Re-type new password: [输入密码]
提示:也可通过如下命令创建:
USER=admin; PASSWORD=admin1234; echo "${USER}:$(openssl passwd -stdin -apr1 <<< ${PASSWORD})" >> auth
[root@master01 longhorn]# kubectl -n longhorn-system create secret generic longhorn-basic-auth --from-file=auth
[root@master01 longhorn]# cat <<EOF > longhorn-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: longhorn-ingress
namespace: longhorn-system
annotations:
nginx.ingress.kubernetes.io/auth-type: basic
nginx.ingress.kubernetes.io/auth-secret: longhorn-basic-auth
nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required '
spec:
ingressClassName: "nginx"
rules:
- host: longhorn.linuxsb.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: longhorn-frontend
port:
number: 80
EOF
[root@master01 longhorn]# kubectl apply -f longhorn-ingress.yaml
[root@master01 longhorn]# kubectl -n longhorn-system get svc longhorn-frontend
[root@master01 longhorn]# kubectl -n longhorn-system get ingress longhorn-ingress
[root@master01 longhorn]# kubectl -n longhorn-system describe svc longhorn-frontend
[root@master01 longhorn]# kubectl -n longhorn-system describe ingress longhorn-ingress
浏览器访问:longhorn.linuxsb.com ,并输入设置的账号和密码。
使用admin/[密码]登录查看。
Helm 是 Kubernetes 的软件包管理工具。包管理器类似 Ubuntu 中使用的apt、Centos中使用的yum 或者Python中的 pip 一样,能快速查找、下载和安装软件包。通常每个包称为一个Chart,一个Chart是一个目录(一般情况下会将目录进行打包压缩,形成name-version.tgz格式的单一文件,方便传输和存储)。
Helm 由客户端组件 helm 和服务端组件 Tiller 组成, 能够将一组K8S资源打包统一管理, 是查找、共享和使用为Kubernetes构建的软件的最佳方式。
在 Kubernetes中部署一个可以使用的应用,需要涉及到很多的 Kubernetes 资源的共同协作。
如安装一个 WordPress 博客,用到了一些 Kubernetes 的一些资源对象。包括 Deployment 用于部署应用、Service 提供服务发现、Secret 配置 WordPress 的用户名和密码,可能还需要 pv 和 pvc 来提供持久化服务。并且 WordPress 数据是存储在mariadb里面的,所以需要 mariadb 启动就绪后才能启动 WordPress。这些 k8s 资源过于分散,不方便进行管理。
基于如上场景,在 k8s 中部署一个应用,通常面临以下几个问题:
如何统一管理、配置和更新这些分散的 k8s 的应用资源文件;
如何分发和复用一套应用模板;
如何将应用的一系列资源当做一个软件包管理。
对于应用发布者而言,可以通过 Helm 打包应用、管理应用依赖关系、管理应用版本并发布应用到软件仓库。
对于使用者而言,使用 Helm 后不用需要编写复杂的应用部署文件,可以以简单的方式在 Kubernetes 上查找、安装、升级、回滚、卸载应用程序。
Helm 将使用 kubectl 在已配置的集群上部署 Kubernetes 资源,因此需要如下前置准备:
建议采用二进制安装helm。
[root@master01 ~]# mkdir helm
[root@master01 ~]# cd helm/
[root@master01 helm]# wget https://repo.huaweicloud.com/helm/v3.15.3/helm-v3.15.3-linux-amd64.tar.gz
[root@master01 helm]# tar -zxvf helm-v3.15.3-linux-amd64.tar.gz
[root@master01 helm]# cp linux-amd64/helm /usr/local/bin/
[root@master01 helm]]# helm version #查看安装版本
[root@master01 helm]]# echo 'source <(helm completion bash)' >> $HOME/.bashrc #helm自动补全
提示:更多安装方式参考官方手册:
https://helm.sh/docs/intro/install/
。
helm search:可以用于搜索两种不同类型的源。
helm search hub:搜索 Helm Hub,该源包含来自许多不同仓库的Helm chart。
helm search repo:搜索已添加到本地头helm客户端(带有helm repo add)的仓库,该搜索是通过本地数据完成的,不需要连接公网。
[root@master01 ~]# helm search hub #可搜索全部可用chart
[root@master01 ~]# helm search hub wordpress
类似CentOS添加yum源,可以给helm仓库添加相关源。
[root@master01 ~]# helm repo list #查看repo
[root@master01 ~]# helm repo add stable http://mirror.azure.cn/kubernetes/charts
[root@master01 ~]# helm repo add aliyun https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts
[root@master01 ~]# helm repo add jetstack https://charts.jetstack.io
[root@master01 ~]# helm search repo stable
[root@master01 ~]# helm search repo aliyun #搜索repo中的chart
[root@master01 ~]# helm repo update #更新repo的chart
dashboard是基于Web的Kubernetes用户界面,即WebUI。
可以使用dashboard将容器化应用程序部署到Kubernetes集群,对容器化应用程序进行故障排除,以及管理集群资源。
可以使用dashboard来查看群集上运行的应用程序,以及创建或修改单个Kubernetes资源(例如部署、任务、守护进程等)。
可以使用部署向导扩展部署,启动滚动更新,重新启动Pod或部署新应用程序。
dashboard还提供有关群集中Kubernetes资源状态以及可能发生的任何错误的信息。
通常生产环境中建议部署dashboard,以便于图形化来完成基础运维。
从7.0.0版本开始,社区已放弃了对基于manifest安装的支持,现在只支持基于helm的安装。 由于多容器设置和对Kong网关API代理的严重依赖,原有基于yaml清单安装的方式不可行的。
同时基于helm的安装,部署速度更快,并且可以更好地控制Dashboard运行所需的所有依赖项。并且已经改变了版本控制方案,并从Helm chart中删除了appVersion。
因为,使用多容器设置,每个模块现在都是单独的版本,Helm chart版本现在可以被视为应用版本。
基于最佳实践,非业务应用,或集群自身的应用都部署在Master节点。
[root@master01 ~]# kubectl label nodes master0{1,2,3} dashboard=enable
提示:建议对于Kubernetes自身相关的应用(如dashboard),此类非业务应用部署在master节点。
默认dashboard会自动创建证书,同时使用对应证书创建secret。生产环境可以启用相应的域名进行部署dashboard,因此需要将对于的域名制作为TLS证书。
本实验已获取免费一年的证书,免费证书获取可参考:
https://freessl.cn.
将已获取的证书上传至对应目录。
[root@master01 ~]# mkdir -p /root/dashboard/certs
[root@master01 ~]# cd /root/dashboard/certs
[root@master01 certs]# mv web.linuxsb.com.crt tls.crt
[root@master01 certs]# mv web.linuxsb.com.key tls.key
[root@master01 certs]# ll
total 8.0K
-rw-r--r-- 1 root root 3.9K Aug 8 16:15 tls.crt
-rw-r--r-- 1 root root 1.7K Aug 8 16:15 tls.key
提示:也可手动如下操作创建自签证书:
[root@master01 ~]# openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/C=CN/ST=ZheJiang/L=HangZhou/O=Xianghy/OU=Xianghy/CN=webui.linuxsb.com"
自定义证书的场景,建议提前使用对应的证书创建secret。
[root@master01 ~]# kubectl create ns kubernetes-dashboard #v3版本dashboard独立ns
[root@master01 ~]# kubectl create secret generic kubernetes-dashboard-certs --from-file=/root/dashboard/certs/ -n kubernetes-dashboard
[root@master01 ~]# kubectl get secret kubernetes-dashboard-certs -n kubernetes-dashboard -o yaml #查看证书信息
添加kubernetes-dashboard的repo仓库。
[root@master01 ~]# helm repo add kubernetes-dashboard https://kubernetes.github.io/dashboard/
[root@master01 ~]# helm repo list
NAME URL
……
kubernetes-dashboard https://kubernetes.github.io/dashboard/
根据实际情况修改默认的chart values,未配置的项表示使用默认值。
如下yaml主要做了几项自定义配置:
kubernetes-dashboard默认的values值参考
kubernetes-dashboard values
[root@master01 ~]# cd /root/dashboard/
[root@master01 dashboard]# vi myvalues.yaml
app:
mode: 'dashboard'
scheduling:
nodeSelector: {"dashboard": "enable"}
ingress:
enabled: true
hosts:
# - localhost
- web.linuxsb.com
ingressClassName: nginx
useDefaultIngressClass: false
annotations:
nginx.ingress.kubernetes.io/ssl-redirect: "true"
tls:
enabled: true
secretName: "kubernetes-dashboard-certs"
tolerations:
- key: node-role.kubernetes.io/control-plane
effect: NoSchedule
auth:
nodeSelector: {"dashboard": "enable"}
# API deployment configuration
api:
scaling:
replicas: 3
containers:
volumeMounts:
- mountPath: /tmp
name: tmp-volume
- mountPath: /etc/localtime
name: timeconfig
volumes:
- name: tmp-volume
emptyDir: {}
- name: timeconfig
hostPath:
path: /etc/localtime
nodeSelector: {"dashboard": "enable"}
# WEB UI deployment configuration
web:
role: web
scaling:
replicas: 3
revisionHistoryLimit: 10
containers:
volumeMounts:
- mountPath: /tmp
name: tmp-volume
- mountPath: /etc/localtime
name: timeconfig
volumes:
- name: tmp-volume
emptyDir: {}
- name: timeconfig
hostPath:
path: /etc/localtime
nodeSelector: {"dashboard": "enable"}
# Metrics Scraper
metricsScraper:
scaling:
replicas: 3
revisionHistoryLimit: 10
containers:
volumeMounts:
- mountPath: /tmp
name: tmp-volume
- mountPath: /etc/localtime
name: timeconfig
volumes:
- name: tmp-volume
emptyDir: {}
- name: timeconfig
hostPath:
path: /etc/localtime
nodeSelector: {"dashboard": "enable"}
kong:
nodeSelector: {"dashboard": "enable"}
根据生产环境最佳实践进行调优,调优完成后开始部署。
[root@master01 dashboard]# helm upgrade --install kubernetes-dashboard kubernetes-dashboard/kubernetes-dashboard --create-namespace --namespace kubernetes-dashboard -f myvalues.yaml
[root@master01 dashboard]# helm -n kubernetes-dashboard list
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
kubernetes-dashboard kubernetes-dashboard 1 2024-08-14 19:39:29.034973262 +0800 CST deployed kubernetes-dashboard-7.5.0
[root@master01 dashboard]# kubectl -n kubernetes-dashboard get all
[root@master01 dashboard]# kubectl -n kubernetes-dashboard get deployments.apps
[root@master01 dashboard]# kubectl -n kubernetes-dashboard get services
[root@master01 dashboard]# kubectl -n kubernetes-dashboard get pods -o wide
[root@master01 dashboard]# kubectl -n kubernetes-dashboard get svc
[root@master01 dashboard]# kubectl -n kubernetes-dashboard get ingress -o wide
建议创建管理员账户,dashboard默认没有创建具有管理员权限的账户,同时v7版本登录只支持token方式。
因此建议创建管理员权限的用户,然后创建此用户的token,然后使用此token进行登录。
[root@master01 dashboard]# cat <<EOF > dashboard-admin.yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: admin
namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: admin
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: admin
namespace: kubernetes-dashboard
---
apiVersion: v1
kind: Secret
type: kubernetes.io/service-account-token
metadata:
name: admin
namespace: kubernetes-dashboard
annotations:
kubernetes.io/service-account.name: "admin"
EOF
[root@master01 dashboard]# kubectl apply -f dashboard-admin.yaml
使用token相对复杂,可将token添加至kubeconfig文件中,使用KubeConfig文件访问dashboard。
[root@master01 dashboard]# ADMIN_SECRET=$(kubectl -n kubernetes-dashboard get secret | grep admin | awk '{print $1}')
[root@master01 dashboard]# DASHBOARD_LOGIN_TOKEN=$(kubectl describe secret -n kubernetes-dashboard ${ADMIN_SECRET} | grep -E '^token' | awk '{print $2}')
[root@master01 dashboard]# echo ${DASHBOARD_LOGIN_TOKEN}
提示:也可通过如下方式获取name为admin的secret的token。
kubectl -n kubernetes-dashboard get secret admin -o jsonpath={".data.token"} | base64 -d
将web.linuxsb.com.crt证书文件导入,以便于浏览器使用该文件登录。
将webui.linuxsb.com.crt证书导入浏览器,并设置为信任,可规避证书不受信任的弹出。
本实验采用ingress所暴露的域名:
https://web.linuxsb.com
使用对应admin用户的token进行访问。
登录后默认进入的是default命名空间,可切换至其他对应的namespace,对整个Kubernetes进行管理和查看。
提示:更多dashboard访问方式及认证可参考
附004.Kubernetes Dashboard简介及使用
。
dashboard登录整个流程可参考:
https://www.cnadn.net/post/2613.html
[root@master01 ~]# kubectl drain master03 --delete-emptydir-data --force --ignore-daemonsets
[root@master01 ~]# kubectl delete node master03
[root@master03 ~]# kubeadm reset -f && rm -rf $HOME/.kube
[root@master01 ~]# kubectl drain worker04 --delete-emptydir-data --force --ignore-daemonsets
[root@master01 ~]# kubectl delete node worker04
[root@worker04 ~]# kubeadm reset -f
[root@worker04 ~]# rm -rf /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf