Kubespray部署Kubernetes生产集群
服务器说明
节点要求
节点数 >=3台
CPU >=2
Memory >=2G
安全组：关闭（允许节点之间任意端口访问，以及ipip隧道协议通讯）
演示环境说明
| 系统类型 | IP地址 | 节点角色 | CPU | Memory | Hostname |
|---|---|---|---|---|---|
| centos-7.5 | 172.16.15.8 | master | >=2 | >=2G | node1 |
| centos-7.5 | 172.16.15.9 | master | >=2 | >=2G | node2 |
| centos-7.5 | 172.16.15.10 | worker | >=2 | >=2G | node3 |
系统设置（所有节点）
主机名
主机名必须合法，并且每个节点都不一样（建议命名规范：数字+字母+中划线组合，不要包含其他特殊字符）
# 查看主机名
$ hostname
# 修改主机名
$ hostnamectl set-hostname <your_hostname>
关闭防火墙、selinux、swap，重置iptables
# 关闭selinux
$ setenforce 0
$ sed -i '/SELINUX/s/enforcing/disabled/' /etc/selinux/config
# 关闭防火墙
$ systemctl stop firewalld && systemctl disable firewalld
# 设置iptables规则
$ iptables -F && iptables -X && iptables -F -t nat && iptables -X -t nat && iptables -P FORWARD ACCEPT
# 关闭swap
$ swapoff -a && free –h
# 关闭dnsmasq(否则可能导致容器无法解析域名)
$ service dnsmasq stop && systemctl disable dnsmasq
k8s参数设置
# 制作配置文件
$ cat > /etc/sysctl.d/kubernetes.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_nonlocal_bind = 1
net.ipv4.ip_forward = 1
vm.swappiness = 0
vm.overcommit_memory = 1
EOF
# 生效文件
$ sysctl -p /etc/sysctl.d/kubernetes.conf
移除docker相关软件包（可选）
$ yum remove -y docker*
$ rm -f /etc/docker/daemon.json
使用kubespray部署集群
这部分只需要在一个操作节点执行，可以是集群中的一个节点，也可以是集群之外的节点。甚至可以是你自己的笔记本电脑。我们这里使用更普遍的集群中的任意一个linux节点。
配置免密
使操作节点可以免密登录到所有节点
# 1. 生成keygen（执行ssh-keygen，一路回车下去）
$ ssh-keygen
# 2. 查看并复制生成的pubkey
$ cat /root/.ssh/id_rsa.pub
# 3. 分别登陆到每个节点上，将pubkey写入/root/.ssh/authorized_keys
$ mkdir -p /root/.ssh
$ echo "<上一步骤复制的pubkey>" >> /root/.ssh/authorized_keys
依赖软件下载、安装
# 安装基础软件
$ yum install -y epel-release python36 python36-pip git
# 下载kubespray源码
$ wget https://github.com/kubernetes-sigs/kubespray/archive/v2.15.0.tar.gz
# 解压缩
$ tar -xvf v2.15.0.tar.gz && cd kubespray-2.15.0
# 安装requirements
$ cat requirements.txt
$ pip3.6 install -r requirements.txt
## 如果install遇到问题可以先尝试升级pip
## $ pip3.6 install --upgrade pip
生成配置
项目中有一个目录是集群的基础配置，示例配置在目录inventory/sample中，我们复制一份出来作为自己集群的配置
# copy一份demo配置，准备自定义
$ cp -rpf inventory/sample inventory/mycluster
# 国内能否安装的关键
cp inventory/mycluster/group_vars/all/offline.yml inventory/mycluster/group_vars/all/mirror.yml
sed -i -E '/# .*\{\{ files_repo/s/^# //g' inventory/mycluster/group_vars/all/mirror.yml
tee -a inventory/mycluster/group_vars/all/mirror.yml <<EOF
gcr_image_repo: "gcr.m.daocloud.io"
kube_image_repo: "k8s.m.daocloud.io"
docker_image_repo: "docker.m.daocloud.io"
quay_image_repo: "quay.m.daocloud.io"
github_image_repo: "ghcr.m.daocloud.io"
files_repo: "https://files.m.daocloud.io"
EOF
由于kubespray给我们准备了py脚本，可以直接根据环境变量自动生成配置文件，所以我们现在只需要设定好环境变量就可以啦
# 使用真实的hostname（否则会自动把你的hostname改成node1/node2...这种哦）
$ export USE_REAL_HOSTNAME=true
# 指定配置文件位置
$ export CONFIG_FILE=inventory/mycluster/hosts.yaml
# 定义ip列表（你的服务器内网ip地址列表，3台及以上，前两台默认为master节点）
$ declare -a IPS=(10.155.19.223 10.155.19.64 10.155.19.147)
# 生成配置文件
$ python3 contrib/inventory_builder/inventory.py ${IPS[@]}
个性化配置
配置文件都生成好了，虽然可以直接用，但并不能完全满足大家的个性化需求，比如用docker还是containerd？docker的工作目录是否用默认的/var/lib/docker？等等。当然默认的情况kubespray还会到google的官方仓库下载镜像、二进制文件，这个就需要你的服务器可以上外面的网，想上外网也需要修改一些配置。
# 定制化配置文件
# 1. 节点组织配置（这里可以调整每个节点的角色）
$ vi inventory/mycluster/hosts.yaml
# 2. containerd配置（教程使用containerd作为容器引擎）
$ vi inventory/mycluster/group_vars/all/containerd.yml
# 3. 全局配置（可以在这配置http(s)代理实现外网访问）
$ vi inventory/mycluster/group_vars/all/all.yml
# 4. k8s集群配置（包括设置容器运行时、svc网段、pod网段等）
$ vi inventory/mycluster/group_vars/k8s-cluster/k8s-cluster.yml
# 5. 修改etcd部署类型为host（默认是docker）
$ vi ./inventory/mycluster/group_vars/etcd.yml
# 6. 附加组件（ingress、dashboard等）
$ vi ./inventory/mycluster/group_vars/k8s-cluster/addons.yml
一键部署
# -vvvv会打印最详细的日志信息，建议开启
$ ansible-playbook -i inventory/mycluster/hosts.yaml -b cluster.yml -vvvv
验证结果
[root@node1 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
node1 Ready control-plane 148m v1.25.6
node2 Ready control-plane 127m v1.25.6
node3 Ready <none> 126m v1.25.6
[root@node1 ~]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS MESSAGE ERROR
etcd-2 Healthy {"health":"true","reason":""}
scheduler Healthy ok
controller-manager Healthy ok
etcd-1 Healthy {"health":"true","reason":""}
etcd-0 Healthy {"health":"true","reason":""}
[root@node1 ~]# kubectl get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-856b65b4c6-grhcz 1/1 Running 0 124m
kube-system calico-node-h5zck 1/1 Running 0 125m
kube-system calico-node-r64x9 1/1 Running 0 125m
kube-system calico-node-z5kqt 1/1 Running 0 125m
kube-system coredns-9ffd646b7-2bzlf 1/1 Running 0 124m
kube-system coredns-9ffd646b7-vcpjr 1/1 Running 0 124m
kube-system dns-autoscaler-d9847d776-tbbpx 1/1 Running 0 124m
kube-system kube-apiserver-node1 1/1 Running 1 148m
kube-system kube-apiserver-node2 1/1 Running 0 127m
kube-system kube-controller-manager-node1 1/1 Running 1 148m
kube-system kube-controller-manager-node2 1/1 Running 1 127m
kube-system kube-proxy-4bhlm 1/1 Running 0 126m
kube-system kube-proxy-5pllk 1/1 Running 0 126m
kube-system kube-proxy-tlrdr 1/1 Running 0 126m
kube-system kube-scheduler-node1 1/1 Running 1 148m
kube-system kube-scheduler-node2 1/1 Running 1 127m
kube-system nginx-proxy-node3 1/1 Running 0 125m
kube-system nodelocaldns-f6r8f 1/1 Running 0 124m
kube-system nodelocaldns-prvkh 1/1 Running 1 (123m ago) 124m
kube-system nodelocaldns-vm549 1/1 Running 0 124m
[root@node1 ~]#
集群冒烟测试
创建nginx-ds
写入配置
[root@node1 ~]# cat > nginx-ds.yml <<EOF
> apiVersion: v1
> kind: Service
> metadata:
> name: nginx-ds
> labels:
> app: nginx-ds
> spec:
> type: NodePort
> selector:
> app: nginx-ds
> ports:
> - name: http
> port: 80
> targetPort: 80
> ---
> apiVersion: apps/v1
> kind: DaemonSet
> metadata:
> name: nginx-ds
> spec:
> selector:
> matchLabels:
> app: nginx-ds
> template:
> metadata:
> labels:
> app: nginx-ds
> spec:
> containers:
> - name: my-nginx
> image: docker.io/library/nginx:latest
> ports:
> - containerPort: 80
> EOF
创建ds
[root@node1 ~]# kubectl apply -f nginx-ds.yml
service/nginx-ds created
daemonset.apps/nginx-ds created
检查IP联通性
检查各Node上Pod IP连通性
[root@node1 ~]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-ds-7psqb 1/1 Running 0 4m31s 10.233.75.6 node2 <none> <none>
nginx-ds-nddxd 1/1 Running 0 4m31s 10.233.71.6 node3 <none> <none>
nginx-ds-x9jzf 1/1 Running 0 4m31s 10.233.102.135 node1 <none> <none>
在每个节点上ping pod ip
[root@node1 ~]# ping 10.233.75.6
PING 10.233.75.6 (10.233.75.6) 56(84) bytes of data.
64 bytes from 10.233.75.6: icmp_seq=1 ttl=63 time=1.09 ms
64 bytes from 10.233.75.6: icmp_seq=2 ttl=63 time=0.959 ms
^C
--- 10.233.75.6 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 0.959/1.026/1.094/0.074 ms
[root@node1 ~]# ping 10.233.71.6
PING 10.233.71.6 (10.233.71.6) 56(84) bytes of data.
64 bytes from 10.233.71.6: icmp_seq=1 ttl=63 time=4.58 ms
64 bytes from 10.233.71.6: icmp_seq=2 ttl=63 time=0.612 ms
^C
--- 10.233.71.6 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.612/2.598/4.584/1.986 ms
[root@node1 ~]# ping 10.233.102.135
PING 10.233.102.135 (10.233.102.135) 56(84) bytes of data.
64 bytes from 10.233.102.135: icmp_seq=1 ttl=64 time=1.49 ms
64 bytes from 10.233.102.135: icmp_seq=2 ttl=64 time=0.084 ms
64 bytes from 10.233.102.135: icmp_seq=3 ttl=64 time=0.083 ms
^C
--- 10.233.102.135 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2005ms
rtt min/avg/max/mdev = 0.083/0.553/1.492/0.663 ms
检查service可达性
[root@node1 ~]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.233.0.1 <none> 443/TCP 5h24m
nginx-ds NodePort 10.233.56.115 <none> 80:32177/TCP 5m46s
# 在每个节点上访问服务
[root@node1 ~]# curl 10.233.56.115:80
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
[root@node1 ~]# curl 10.233.75.6:80
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
检查dns可用性
创建一个nginx pod
[root@node1 ~]# cat > pod-nginx.yaml <<EOF
> apiVersion: v1
> kind: Pod
> metadata:
> name: nginx
> spec:
> containers:
> - name: nginx
> image: docker.io/library/nginx:1.19
> ports:
> - containerPort: 80
> EOF
创建pod
[root@node1 ~]# kubectl apply -f pod-nginx.yaml
pod/nginx created
进入pod，查看dns
[root@node1 ~]# kubectl exec -it nginx -- sh
# 查看dns配置
/ # cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 169.254.25.10
options ndots:5
# ping nginx-ds
/ # ping nginx-ds
PING nginx-ds (10.233.56.115): 56 data bytes
64 bytes from 10.233.56.115: seq=0 ttl=64 time=1.421 ms
64 bytes from 10.233.56.115: seq=1 ttl=64 time=0.121 ms
64 bytes from 10.233.56.115: seq=2 ttl=64 time=0.192 ms
日志功能
使用kubectl查看pod的容器日志
[root@node1 ~]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx 1/1 Running 0 12m 10.233.71.7 node3 <none> <none>
nginx-ds-7psqb 1/1 Running 0 27m 10.233.75.6 node2 <none> <none>
nginx-ds-nddxd 1/1 Running 0 27m 10.233.71.6 node3 <none> <none>
nginx-ds-x9jzf 1/1 Running 0 27m 10.233.102.135 node1 <none> <none>
[root@node1 ~]# kubectl logs nginx
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2023/02/03 09:13:17 [notice] 1#1: using the "epoll" event method
2023/02/03 09:13:17 [notice] 1#1: nginx/1.23.2
2023/02/03 09:13:17 [notice] 1#1: built by gcc 11.2.1 20220219 (Alpine 11.2.1_git20220219)
2023/02/03 09:13:17 [notice] 1#1: OS: Linux 4.18.0-448.el8.x86_64
2023/02/03 09:13:17 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 65535:65535
2023/02/03 09:13:17 [notice] 1#1: start worker processes
2023/02/03 09:13:17 [notice] 1#1: start worker process 30
2023/02/03 09:13:17 [notice] 1#1: start worker process 31
Exec功能
[root@node1 ~]# kubectl get pods -l app=nginx-ds
NAME READY STATUS RESTARTS AGE
nginx-ds-7psqb 1/1 Running 0 29m
nginx-ds-nddxd 1/1 Running 0 29m
nginx-ds-x9jzf 1/1 Running 0 29m
[root@node1 ~]# kubectl exec -it nginx-ds-x9jzf -- nginx -v
nginx version: nginx/1.23.2