PVE 安装 k8s

版本信息

• Proxmox Virtual Environment 8.4-1
• Ubuntu Server 20.04 LTS
• Kubernetes 1.26.3

PVE 制作 Cloud-Init 虚拟机模板

虚拟机初始化

环境准备

禁用交换分区

# 关闭
swapoff -a

# 禁用
sed -i "s/\/dev\/mapper\/centos-swap/# \/dev\/mapper\/centos-swap/g" /etc/fstab

关闭防火墙

# 关闭并禁用
systemctl stop firewalld && systemctl disable firewalld

禁用 SELinux

# 关闭
setenforce 0

# 禁用
sed -i "s/SELINUX=enforcing/SELINUX=disable/g" /etc/selinux/config

设置 IPV4 转发内核参数

cat>/etc/sysctl.d/kubernetes.conf<<'EOF'
net.ipv4.ip_forward=1
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
EOF

modprobe br_netfilter
sysctl -p /etc/sysctl.d/kubernetes.conf

安装 kubectl

安装 kubelet/kubeadm/kubectl：

这部分的内容存在时效性，如果失效，请前往 k8s 进行查阅

sudo apt-get update && sudo apt-get install -y apt-transport-https ca-certificates curl
sudo mkdir -p /etc/apt/keyrings/
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.33/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
sudo chmod 644 /etc/apt/keyrings/kubernetes-apt-keyring.gpg # allow unprivileged APT programs to read this keyring
# This overwrites any existing configuration in /etc/apt/sources.list.d/kubernetes.list
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.33/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo chmod 644 /etc/apt/sources.list.d/kubernetes.list   # helps tools such as command-not-found to work correctly
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

安装 containerd

 sudo apt-get update
 sudo apt-get install \
    ca-certificates \
    curl \
    gnupg \
    lsb-release

sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null


# 安装containerd
sudo apt-get update
sudo apt-get install -y containerd.io

# 查看运行状态
systemctl enable containerd
systemctl status containerd

# 配置 containerd
sudo mkdir -p /etc/containerd
sudo containerd config default | sudo tee /etc/containerd/config.toml

# 修改为 SystemdCgroup
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
cat /etc/containerd/config.toml | grep SystemdCgroup

# 配置 containerd 服务
sudo systemctl enable containerd
sudo systemctl restart containerd
sudo systemctl status containerd

配置 containerd 镜像源

Containerd 换源

配置 crictl

# 创建配置文件
sudo mkdir -p /etc
sudo tee /etc/crictl.yaml > /dev/null <<EOF
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
EOF

安装CNI插件

# 在所有节点上安装 CNI 插件
sudo mkdir -p /opt/cni/bin
cd /opt/cni/bin

# 下载并安装 CNI 插件
sudo wget https://github.com/containernetworking/plugins/releases/download/v1.3.0/cni-plugins-linux-amd64-v1.3.0.tgz
sudo tar -zxvf cni-plugins-linux-amd64-v1.3.0.tgz

# 确保 CNI 目录存在
sudo mkdir -p /etc/cni/net.d/

# 重启 containerd 和 kubelet
sudo systemctl restart containerd
sudo systemctl restart kubelet

kubeadm 镜像初始化

sudo containerd config default | sudo tee /etc/containerd/config.toml
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml
sudo sed -i 's|sandbox_image = "registry.k8s.io/pause:3.8"|sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.10"|g' /etc/containerd/config.toml

# 重启服务
sudo systemctl daemon-reload
sudo systemctl restart containerd
sudo systemctl restart kubelet

# 配置虚拟机
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

# 设置所需的 sysctl 参数，参数在重新启动后保持不变
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

# 应用 sysctl 参数而不重新启动
sudo sysctl --system

# 通过运行以下指令确认 br_netfilter 和 overlay 模块被加载
lsmod | grep br_netfilter
lsmod | grep overlay

# 通过运行以下指令确认 net.bridge.bridge-nf-call-iptables、net.bridge.bridge-nf-call-ip6tables 和 net.ipv4.ip_forward 系统变量在你的 sysctl 配置中被设置为 1
sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-ip6tables net.ipv4.ip_forward

制作虚拟机模板

之后用这个模板克隆五个 VM 出来（一个 control-plane 控制平面，四个 worker），如下，注意是完整克隆：

最终效果

创建内部网桥

因为这里主要是用于内部的 k8s 相互发现的，而且 k8s 有设置静态地址的需求，这里就不设置网关，也不搭建 dhcp 服务器了。

创建内部子网，用于 k8s 之间的相互通信

克隆虚拟机初始化

设置内网地址

重新获取ip地址

在所有虚拟机上执行下列命令，获取 vmbr0 的 ip 地址

sudo dhclient -v

重新生成 machine-id

这个比较笨，一个一个去敲命令吧

在所有虚拟机上执行下列命令

# 删除现有的machine ID
sudo rm -f /etc/machine-id
sudo rm -f /var/lib/dbus/machine-id

# 生成新的machine ID
sudo systemd-machine-id-setup

# 重启服务
sudo systemctl restart systemd-journald

# 查看当前machine ID
cat /etc/machine-id

# 验证dbus machine ID
cat /var/lib/dbus/machine-id

# 使用systemd命令查看
hostnamectl | grep "Machine ID"

拉取镜像

在所有虚拟机上执行下列命令，拉取镜像

sudo -E kubeadm config images pull --image-repository registry.aliyuncs.com/google_containers

k8s 组网

Control-Plane

init

在所有 control-plane 中执行下列命令：

拉取命令

sudo kubeadm config images list
sudo kubeadm config images pull --image-repository registry.aliyuncs.com/google_containers

请指定使用 --control-plane-endpoint=10.10.10.11来指定内部网卡

如果网络不好，请使用下列来拉去命令

# 手动拉取阿里云的pause镜像并重新标记
sudo crictl pull registry.aliyuncs.com/google_containers/pause:3.10

# 使用ctr重新标记（containerd原生命令）
sudo ctr -n k8s.io image tag registry.aliyuncs.com/google_containers/pause:3.10 registry.k8s.io/pause:3.10

# 修改containerd配置使用正确的sandbox镜像
sudo cp /etc/containerd/config.toml /etc/containerd/config.toml.backup

# 编辑配置文件，找到sandbox_image配置
sudo sed -i 's|sandbox_image = ".*"|sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.10"|' /etc/containerd/config.toml

如果需要重新初始化

sudo kubeadm reset --force --cri-socket unix:///var/run/containerd/containerd.sock

sudo kubeadm init \
  --control-plane-endpoint=10.10.10.11 \
  --pod-network-cidr=10.244.0.0/16 \
  --image-repository registry.aliyuncs.com/google_containers \
  --apiserver-advertise-address=10.10.10.11 \
  --kubernetes-version=v1.33.1 \
  --cri-socket unix:///var/run/containerd/containerd.sock \
  --ignore-preflight-errors=NumCPU

初始化成功后，配置 kubectl

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

安装 Flannel 插件

# 安装 flannel
kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

# 查看安装状态
kubectl get pods --all-namespaces

水平拓展

在 control-plane 上执行：

sudo kubeadm token create --print-join-command --certificate-key $(sudo kubeadm init phase upload-certs --upload-certs 2>/dev/null | tail -1)

获取加入命令

请注意自己补充--apiserver-advertise-address=10.10.10.12来指定使用内网网卡

kubeadm join 10.10.10.11:6443 --token lsbyes.aza8lh8y1n1eaiji --discovery-token-ca-cert-hash sha256:xxx --control-plane --apiserver-advertise-address=10.10.10.12

Worker

在 control-plane 上执行：

sudo kubeadm init phase upload-certs --upload-certs
sudo kubeadm token create --print-join-command

获取加入命令

kubeadm join 10.10.10.11:6443 --token lsbyes.aza8lh8y1n1eaiji --discovery-token-ca-cert-hash sha256:xxx

在 Control-Plane 使用下列命令查看 k8s 集群状态

# 查看 Flannel Pod 状态
kubectl get pods -n kube-flannel

# 查看节点状态
kubectl get nodes

常用批量操作命令

重启节点

pssh -h <(seq -f "10.10.10.%g" 111 116) -p 6 -i -t 0 -x "-p 36633" \
"sudo mv /etc/containerd/config.toml /etc/containerd/config.toml.bak_$(date +%F_%T) && \
sudo containerd config default | sudo tee /etc/containerd/config.toml > /dev/null && \
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml && \
sudo systemctl daemon-reexec && \
sudo systemctl restart containerd && \
sudo systemctl status containerd --no-pager"

参考资料

• 使用 PVE 搭建 Kubernetes 集群
• 年轻人折腾的第一台独服 – PVE 7.4 + NAT 内网 + DHCP + K8S 多节点集群 + Istio + CertManager(letsencrypt)
• https://6xyun.cn/article/k8s-manual-deploy