01.K8S集群环境搭建

kubadm 安装

强烈建议参考官方文档: 官方文档 https://kubernetes.io/zh-cn/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

系统：Ubuntu 24.04.1 LTS

架构：1 master 3 worrd

安装成功版本：kubeadm 1.30.7 (最新 1.31.4 失败，无限重启)

一. 基础环境配置

1.1 关闭 swap

# 临时关闭：
swapoff -a
# 永久关闭：
sed -ri 's/.*swap.*/#&/' /etc/fstab

1.2 加载模块

#手动加载
modprobe overlay
modprobe br_netfilter
#验证
lsmod |grep "over"
#永久生效
cat << EOF | tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

1.3 开启IPv4 数据包转发

#永久配置
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
#立即生效
sudo sysctl --system

1.4 安装ipset ipvsadm

#安装
apt install -y ipset ipvsadm 

#添加ipvsadm模块
cat << EOF | tee /etc/modules-load.d/ipvs.conf
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack
EOF

#创建加载模块脚本文件
cat << EOF | tee ipvs.sh
#!/bin/sh
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack
EOF

#执行
bash ipvs.sh

1.5 追加增加主机名直接解析

cat << EOF >> /etc/hosts
192.168.31.150 k8smaster
192.168.31.151 k8snode1
192.168.31.152 k8snode2
EOF

1.6 各个节点配置主机名

#master节点配置
hostnamectl set-hostname k8smaster
#work1节点配置
hostnamectl set-hostname k8snode1
#work2节点配置
hostnamectl set-hostname k8snode2

1.7 安装cri-containerd

官网 https://github.com/containerd/containerd/releases 选择crixxx 下载啊，这个坑例如本次下载https://github.com/containerd/containerd/releases/download/v1.7.24/cri-containerd-1.7.24-linux-amd64.tar.gz

1.7.1 配置生成

mkdir /etc/containerd/
containerd config default > /etc/containerd/config.toml

1.7.2 配置修改

vim /etc/containerd/config.toml
#修改如下
#1. 3.8->3.9 主要看kubeam 依赖来改
#sandbox_image = "registry.k8s.io/pause:3.9"
#或阿里云仓库
#sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.9"

#2. 修改139行
#SystemdCgroup = true

1.7.3设置代理：

该配置会影响calico安装，导致启动失败，巨坑，了解calico原理后应该会明白

不会影响flanal 的安装

#vim /usr/lib/systemd/system/containerd.service
[Service]
Environment="HTTP_PROXY=http://10.30.61.24:1080"
Environment="HTTPS_PROXY=http://10.30.61.24:1080"
Environment="NO_PROXY=localhost"

1.7.4 启动和开启服务

 systemctl enable --now  containerd
 #检查
 #ls /var/run/containerd/

1.8 安装 kubeadm、kubelet 和 kubectl

你需要在每台机器上安装以下的软件包：

kubeadm：用来初始化集群的指令。
kubelet：在集群中的每个节点上用来启动 Pod 和容器等。
kubectl：用来与集群通信的命令行工具。

1.8.1 更新 apt 包索引并安装使用 Kubernetes apt 仓库所需要的包：

sudo apt-get update
#apt-transport-https 可能是一个虚拟包（dummy package）；如果是的话，你可以跳过安装这个包
sudo apt-get install -y apt-transport-https ca-certificates curl gpg

1.8.2 下载用于 Kubernetes 软件包仓库的公共签名密钥。

所有仓库都使用相同的签名密钥，因此你可以忽略 URL 中的版本：

#如果 `/etc/apt/keyrings` 目录不存在，则应在 curl 命令之前创建它，请阅读下面的注释。
#sudo mkdir -p -m 755 /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

1.8.3 添加 Kubernetes apt 仓库。

请注意，此仓库仅包含适用于 Kubernetes 1.30 的软件包；对于其他 Kubernetes 次要版本，则需要更改 URL 中的 Kubernetes 次要版本以匹配你所需的次要版本（你还应该检查正在阅读的安装文档是否为你计划安装的 Kubernetes 版本的文档）。

#此操作会覆盖 /etc/apt/sources.list.d/kubernetes.list 中现存的所有配置。
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

1.8.4 安装

更新 apt 包索引安装 kubelet、kubeadm 和 kubectl，并锁定其版本：

sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

1.8.5 查看安装

apt-cache policy kubeadm
kubeadm:
  Installed: 1.30.7-1.1
  Candidate: 1.30.7-1.1
  Version table:
 *** 1.30.7-1.1 500
        500 https://pkgs.k8s.io/core:/stable:/v1.30/deb  Packages
        100 /var/lib/dpkg/status
     1.30.6-1.1 500
        500 https://pkgs.k8s.io/core:/stable:/v1.30/deb  Packages
     1.30.5-1.1 500
        500 https://pkgs.k8s.io/core:/stable:/v1.30/deb  Packages
     1.30.4-1.1 500
        500 https://pkgs.k8s.io/core:/stable:/v1.30/deb  Packages
     1.30.3-1.1 500
        500 https://pkgs.k8s.io/core:/stable:/v1.30/deb  Packages
     1.30.2-1.1 500
        500 https://pkgs.k8s.io/core:/stable:/v1.30/deb  Packages
     1.30.1-1.1 500
        500 https://pkgs.k8s.io/core:/stable:/v1.30/deb  Packages
     1.30.0-1.1 500
        500 https://pkgs.k8s.io/core:/stable:/v1.30/deb  Packages

1.8.6 常用命令

# 查看以依赖
apt-cache showpkg kubeadm
# 可安装版本
apt-cache madision kubeadm
# 指定版本安装
apt install kubeadm=xxx kubectl=xxx
# 版本锁定
apt-mark hold kubet kubeadm kubectl
# 解锁
apt-mark unhold kubet kubeadm kubectl
# 生成配置文件
kubeadm config print init-defaults > kubeadm-config.yaml
# 使用生成的文件
kubeadm init --config kubeadm-config.yaml

1.8.7 修改kubelet配置

修改

 vim /etc/default/kubelet
 #修改如下
 #KUBELET_EXTRA_ARGS="--cgroup-driver=systemd"

1.8.8 依赖镜像查看和下载

#依赖镜像
kubeadm config images list
#提前拉去镜像
kubeadm config images pull
#阿里云镜像仓库
#kubeadm config images list --image-repository registry.aliyuncs.com/google_containers
#kubeadm config images pull --image-repository registry.aliyuncs.com/google_containers

#crictl images list
#registry.k8s.io/coredns/coredns           v1.11.3             c69fa2e9                                                                          cbf5f       18.6MB
#registry.k8s.io/etcd                      3.5.15-0            2e96e591                                                                          3fc06       56.9MB
#registry.k8s.io/kube-apiserver            v1.30.0             c42f1365                                                                          6d0b2       32.7MB
#registry.k8s.io/kube-apiserver            v1.30.8             772392d3                                                                          72035       32.7MB
#registry.k8s.io/kube-controller-manager   v1.30.0             c7aad438                                                                          36fa5       31MB
#registry.k8s.io/kube-controller-manager   v1.30.8             85333d41                                                                          dd3ce       31.1MB
#registry.k8s.io/kube-proxy                v1.30.0             a0bf559e                                                                          280cf       29MB
#registry.k8s.io/kube-proxy                v1.30.8             ce61fda6                                                                          7eb41       29.1MB
#registry.k8s.io/kube-scheduler            v1.30.0             259c8277                                                                          fcbbc       19.2MB
#registry.k8s.io/kube-scheduler            v1.30.8             eb53b988                                                                          d5e03       19.2MB
#registry.k8s.io/pause                     3.9                 e6f18168                                                                          83972       322kB

calico 依赖镜像

crictl pull docker.io/calico/cni:v3.29.1
crictl pull docker.io/calico/node:v3.29.1
crictl pull docker.io/calico/kube-controllers:v3.29.1
crictl pull docker.io/calico/pod2daemon-flexvol:v3.29.1
crictl pull docker.io/calico/typha:v3.29.1

二、master 节点安装

2.1 安装核心组件

2.1.1 配置文件生成和修改

#获取初始化文件
kubeadm config print init-defaults > kubeadm-config.yml
#修改节点地址
#advertiseAddress: 192.168.31.150
#name: k8smaster
#增加
#podSubnet: 10.244.0.0/16

2.1.2 开始安装

kubeadm init --config kubeadm-config.yml

输出详情

[init] Using Kubernetes version: v1.31.3
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action beforehand using 'kubeadm config images pull'
W1207 07:20:31.696722   21051 checks.go:846] detected that the sandbox image "registry.k8s.io/pause:3.8" of the container runtime is inconsistent with that used by kubeadm.It is recommended to use "registry.aliyuncs.com/google_containers/pause:3.10" as the CRI sandbox image.
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8smaster kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.1.12]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8smaster localhost] and IPs [192.168.1.12 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8smaster localhost] and IPs [192.168.1.12 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "super-admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests"
[kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s
[kubelet-check] The kubelet is healthy after 501.526721ms
[api-check] Waiting for a healthy API server. This can take up to 4m0s
[api-check] The API server is healthy after 3.501394768s
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8smaster as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node k8smaster as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: f84j06.ujmkv1ua46w4c4nc
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.1.12:6443 --token f84j06.ujmkv1ua46w4c4nc \
        --discovery-token-ca-cert-hash sha256:393f0be8c4763e348c1109591e38e10249e1380a40ca98cfbbdcc13750069835

2.1.3 安装配置

需要把$HOME/.kube/config 复制到各个work节点，否则节点执行kubectl命令失败
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

2.2 安装 calico 网络插件

官方文档 https://docs.tigera.io/calico/latest/getting-started/kubernetes/quickstart

2.2.1 安装

kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.29.1/manifests/tigera-operator.yaml


#kubectl get ns
#NAME              STATUS   AGE
#default           Active   95m
#kube-node-lease   Active   95m
#kube-public       Active   95m
#kube-system       Active   95m
#tigera-operator   Active   88s

2.2.2 资源安装

2.2.2.1 下载和修改资源

#1. 下载资源
wget https://raw.githubusercontent.com/projectcalico/calico/v3.29.1/manifests/custom-resources.yaml
#2. 修改pod网段修:10.244.0.0/16,
#具体可以通过kubectl get node -o yaml |grep "10."

2.2.2.2 安装资源

kubectl create -f custom-resources.yaml

2.2.3观察安装情况

#观察运行情况
watch kubectl get pods -n calico-system

三、workd 节点

3.1 加入集群

kubeadm join 192.168.1.12:6443 --token p9uink.ulunpc8evn4oxoql \
        --discovery-token-ca-cert-hash sha256:dcdf96c59c93766a48bc6221d90c00b5bd7325742b4125e7180a93566860fdcb

3.3 主节点的/etc/kubernet/admin.conf 拷贝到从节点，设置环境变量

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

四、测试

在Kubernetes集群中创建一个pod，验证是否正常运行：

$ kubectl create deployment nginx --image=nginx
$ kubectl expose deployment nginx --port=80 --type=NodePort
$ kubectl get pod,svc

五、附加问题解决

5.1 容器告警问题

WARN[0000] runtime connect using default endpoints: [unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead.

crictl config --set runtime-endpoint=unix:///run/containerd/containerd.sock

六、导出镜像

6.1 查看镜像

sudo ctr -n k8s.io images list

6.2 批量导出脚本

cat << EOF |tee k8s_export_images.sh 
#!/bin/bash
IMAGES=$(sudo ctr -n k8s.io images list --quiet)
for IMAGE in $IMAGES; do
  FILENAME=$(echo $IMAGE | tr '/' '-' | tr ':' '-')
  sudo ctr -n k8s.io images export ${FILENAME}.tar $IMAGE
done
EOF

# bash k8s_export_images.sh