在裸金属集群上部署Kubernetes

容器运行时

使用docker作为容器运行时

安装docker

# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update

sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

由于当前国内无法拉取镜像，需配置代理，有两种方式

找到守护进程的代理配置/etc/systemd/system/docker.service.d/proxy.conf，进行如下修改

[Service]
Environment="HTTP_PROXY=http://proxy.example.com:8080/"
Environment="HTTPS_PROXY=http://proxy.example.com:8080/"
Environment="NO_PROXY=localhost,127.0.0.1,.example.com"

重新加载配置并重启服务

sudo systemctl daemon-reload
sudo systemctl restart docker

配置docker配置文件

{
  "proxies": {
        "default": {
          "httpProxy": "http://proxy.example.com:8080",
          "httpsProxy": "http://proxy.example.com:8080",
          "noProxy": "localhost,127.0.0.1,.example.com"
        }
  }
}

安装cri-docker

下载安装包，将可执行文件放到/usr/bin或者/usr/local/bin

下载systemd默认配置文件，并进行一些修改 https://github.com/Mirantis/cri-dockerd/tree/master/packaging/systemd

在ExecStart添加启动参数–pod-infra-container-image设置国产镜像，这个容器是用于创建每个pod中都存在的基础设施容器pause，他是实现pod中容器共享网络和命名空间的关键

ExecStart=/usr/bin/cri-dockerd --container-runtime-endpoint fd:// --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.10

（可选）配置docker代理

修改/etc/systemd/system/docker.service.d或者/usr/lib/systemd/system/docker.service.d下的http-proxy.conf

[Service]
Environment="HTTP_PROXY=http://your-proxy-server:port"
Environment="HTTPS_PROXY=http://your-proxy-server:port"
Environment="NO_PROXY=localhost,127.0.0.1,.example.com"

使用containerd作为容器运行时

根据containerd/docs/getting-started.md at main · containerd/containerd官方文档进行安装

安装kubelet、kubectl、kubeadm

sudo apt-get update
# apt-transport-https 可能是一个虚拟包（dummy package）；如果是的话，你可以跳过安装这个包
sudo apt-get install -y apt-transport-https ca-certificates curl gpg

# 如果 `/etc/apt/keyrings` 目录不存在，则应在 curl 命令之前创建它，请阅读下面的注释。
# sudo mkdir -p -m 755 /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.xx/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

# 此操作会覆盖 /etc/apt/sources.list.d/kubernetes.list 中现存的所有配置。
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.xx/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

系统配置

cgroups与交换空间

如果linux的版本默认支持cgroupsv2，那么支持开启交换空间，否则则要关闭交换空间。

// 临时关闭
sudo swapoff -a
// 永久关闭，在/etc/fstab设置
sed -ri 's/.swap./#&/' /etc/fstab

netfilter模块

确保Linux网桥(bridge)的流量能够被iptables/ip6tables规则处理，从而保障Kubernetes的网络功能

cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF

sysctl --system  # 生效

使用kubeadm创建集群

创建主节点

kubeadm init \
    # 该节点ip地址
    --apiserver-advertise-address=xx.xxx.xx.xxx \
    # k8s版本
    --kubernetes-version v1.32.2 \
    # service和pod所在ip域
    --service-cidr=10.96.0.0/12 \
    --pod-network-cidr=10.244.0.0/16 \
    # 选择cri-docker
    --cri-socket=unix:///var/run/cri-dockerd.sock \
    # 忽略所有检查错误
    --ignore-preflight-errors=all \
    # 国内镜像源, 设置代理可以不配置
    --image-repository=registry.aliyuncs.com/google_containers

创建成功后，会返回设置集群配置文件、添加子节点的命令。

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a Pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  /docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join <control-plane-host>:<control-plane-port> --token <token> --discovery-token-ca-cert-hash sha256:<hash>

如果返回错误，需要先删除已有配置：

kubeadm reset --cri-socket=unix:///var/run/cri-dockerd.sock

加入子节点

使用创建集群后返回的添加子节点命令将子节点添加到集群：

如果忘记命令可以再主节点执行如下命令获取：

kubeadm token create --print-join-command

配置CNI

CNI是容器网络插件，主要功能是实现节点间容器的网络通信，我们这里使用flannel配置集群

kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

如果pod网络不是默认的10.244.0.0/16，需要下载manifest进行配置，或者使用helm安装指定–set podCidr=“xx.xx.xx.xx"参数。

Nvidia Time-slicing

详情见官方配置文档

创建配置文件，为显卡划分配置 data中可以设定多个版本的配置，如果开启mig，可以在resources中设定每个分片的配置。

apiVersion: v1
kind: ConfigMap
metadata:
  name: time-slicing-config-fine
data:
  a100-40gb: |-
    version: v1
    flags:
      migStrategy: mixed
    sharing:
      timeSlicing:
        resources:
        - name: nvidia.com/gpu
          replicas: 8
        - name: nvidia.com/mig-1g.5gb
          replicas: 2
        - name: nvidia.com/mig-2g.10gb
          replicas: 2
        - name: nvidia.com/mig-3g.20gb
          replicas: 3
        - name: nvidia.com/mig-7g.40gb
          replicas: 7
  tesla-t4: |-
    version: v1
    flags:
      migStrategy: none
    sharing:
      timeSlicing:
        resources:
        - name: nvidia.com/gpu
          replicas: 4

应用配置

kubectl create -n gpu-operator -f time-slicing-config-fine.yaml

修改nvidia集群策略

kubectl patch clusterpolicies.nvidia.com/cluster-policy \
  -n gpu-operator --type merge \
  -p '{"spec": {"devicePlugin": {"config": {"name": "time-slicing-config-fine"}}}}'

将配置应用到节点指定节点名

kubectl label node <node-name> nvidia.com/device-plugin.config=tesla-t4

根据GPU型号筛选

kubectl label node \
  --selector=nvidia.com/gpu.product=Tesla-T4 \
  nvidia.com/device-plugin.config=tesla-t4

重启nvidia服务

kubectl rollout restart -n gpu-operator daemonset/nvidia-device-plugin-daemonset

Some Questions Q1.在修改了配置重新应用后，标签nvidia.com/gpu.replicas=还是之前的配置，但是节点的Allocatable和Capacity都修改成功 Q2.部署pod占用一个GPU后，Allocatable未发生变动 Allocatable是Capacity去除kubelet等必要组件资源的可分配资源，而pod分配后资源变化需要查看Allocated

部署prometheus

kube-prometheus是集成了Grafana dashboards和Prometheus的可以轻易部署在k8s上服务，部署后可以方便的使用其监听k8s集群相关状态。

准备阶段

为了保证prometheus能够收集kubelet的信息，kubelet服务需要保证如下参数

–authentication-token-webhook=true This flag enables, that a ServiceAccount token can be used to authenticate against the kubelet(s). This can also be enabled by setting the kubelet configuration value authentication.webhook.enabled to true.

–authorization-mode=Webhook This flag enables, that the kubelet will perform an RBAC request with the API to determine, whether the requesting entity (Prometheus in this case) is allowed to access a resource, in specific for this project the /metrics endpoint. This can also be enabled by setting the kubelet configuration value authorization.mode to Webhook.

查看kubelet进程是否带有这两个参数配置

systemctl status kubelet
       Docs: https://kubernetes.io/docs/
   Main PID: 3720162 (kubelet)
      Tasks: 33 (limit: 76746)
     Memory: 54.3M
        CPU: 1h 4min 40.277s
     CGroup: /system.slice/kubelet.service
             └─3720162 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --pod-infra-container-image...
             
sudo vim /var/lib/kubelet/config.yaml

apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 0s
    enabled: true   # 开启authen
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
  mode: Webhook     # 设置webhook鉴权
  webhook:
    cacheAuthorizedTTL: 0s
    cacheUnauthorizedTTL: 0s
cgroupDriver: systemd
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
containerRuntimeEndpoint: ""
cpuManagerReconcilePeriod: 0s
crashLoopBackOff: {}
evictionPressureTransitionPeriod: 0s
fileCheckFrequency: 0s
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 0s
imageMaximumGCAge: 0s
imageMinimumGCAge: 0s
kind: KubeletConfiguration
logging:
  flushFrequency: 0
  options:
    json:
      infoBufferSize: "0"
    text:
      infoBufferSize: "0"
  verbosity: 0
memorySwap: {}
nodeStatusReportFrequency: 0s
nodeStatusUpdateFrequency: 0s
resolvConf: /run/systemd/resolve/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 0s
shutdownGracePeriod: 0s
shutdownGracePeriodCriticalPods: 0s
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 0s
syncFrequency: 0s
volumeStatsAggPeriod: 0s

部署

下载配置文件。当前grafana和prometheus的接口只能在集群内部访问，可以通过修改配置文件将其暴露出来。

NOTE grafana默认账号密码为admin、admin，初次进入需要重设密码

git clone https://github.com/prometheus-operator/kube-prometheus.git
cd kube-prometheus/manifests

修改grafana-service.yaml

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/component: grafana
    app.kubernetes.io/name: grafana
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 11.5.2
  name: grafana
  namespace: monitoring
spec:
  type: LoadBalancer    # 添加type
  ports:
  - name: http
    port: 3000
    targetPort: http
  selector:
    app.kubernetes.io/component: grafana
    app.kubernetes.io/name: grafana
    app.kubernetes.io/part-of: kube-prometheus

修改prometheus-service.yaml

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/component: prometheus
    app.kubernetes.io/instance: k8s
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 3.2.1
  name: prometheus-k8s
  namespace: monitoring
spec:
  type: LoadBalancer    # 添加type
  ports:
  - name: web
    port: 9090
    targetPort: web
  - name: reloader-web
    port: 8080
    targetPort: reloader-web
  selector:
    app.kubernetes.io/component: prometheus
    app.kubernetes.io/instance: k8s
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/part-of: kube-prometheus
  sessionAffinity: ClientIP

按照官方文档命令进行部署

kubectl apply --server-side -f manifests/setup
kubectl wait \
	--for condition=Established \
	--all CustomResourceDefinition \
	--namespace=monitoring
kubectl apply -f manifests/

查看服务运行状况

kube@master:~/tools/kube-prometheus$ kubectl get pod -n monitoring
NAME                                   READY   STATUS    RESTARTS   AGE
alertmanager-main-0                    2/2     Running   0          17s
alertmanager-main-1                    2/2     Running   0          17s
alertmanager-main-2                    2/2     Running   0          17s
blackbox-exporter-78cc978f77-p2mf7     3/3     Running   0          21s
grafana-6db778c967-b6h4p               1/1     Running   0          20s
kube-state-metrics-5f96f94459-qrbbz    3/3     Running   0          20s
node-exporter-8mj69                    2/2     Running   0          20s
node-exporter-fmzb6                    2/2     Running   0          20s
prometheus-adapter-599c88b6c4-hr94r    1/1     Running   0          19s
prometheus-adapter-599c88b6c4-xdvp7    1/1     Running   0          19s
prometheus-k8s-0                       2/2     Running   0          17s
prometheus-k8s-1                       2/2     Running   0          17s
prometheus-operator-6f476fbbd5-q5s2q   2/2     Running   0          19s

查看接口ip

kube@master:~/tools/kube-prometheus/manifests$ kubectl get service -o wide -n monitoring
NAME                    TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)                         AGE     SELECTOR
alertmanager-main       ClusterIP      10.101.206.117   <none>          9093/TCP,8080/TCP               7m34s   app.kubernetes.io/component=alert-router,app.kubernetes.io/instance=main,app.kubernetes.io/name=alertmanager,app.kubernetes.io/part-of=kube-prometheus
alertmanager-operated   ClusterIP      None             <none>          9093/TCP,9094/TCP,9094/UDP      7m30s   app.kubernetes.io/name=alertmanager
blackbox-exporter       ClusterIP      10.110.74.236    <none>          9115/TCP,19115/TCP              7m34s   app.kubernetes.io/component=exporter,app.kubernetes.io/name=blackbox-exporter,app.kubernetes.io/part-of=kube-prometheus
grafana                 LoadBalancer   10.105.184.132   192.168.1.102   3000:31154/TCP                  7m33s   app.kubernetes.io/component=grafana,app.kubernetes.io/name=grafana,app.kubernetes.io/part-of=kube-prometheus
kube-state-metrics      ClusterIP      None             <none>          8443/TCP,9443/TCP               7m33s   app.kubernetes.io/component=exporter,app.kubernetes.io/name=kube-state-metrics,app.kubernetes.io/part-of=kube-prometheus
node-exporter           ClusterIP      None             <none>          9100/TCP                        7m33s   app.kubernetes.io/component=exporter,app.kubernetes.io/name=node-exporter,app.kubernetes.io/part-of=kube-prometheus
prometheus-adapter      ClusterIP      10.106.18.174    <none>          443/TCP                         7m32s   app.kubernetes.io/component=metrics-adapter,app.kubernetes.io/name=prometheus-adapter,app.kubernetes.io/part-of=kube-prometheus
prometheus-k8s          LoadBalancer   10.106.91.240    192.168.1.103   9090:32723/TCP,8080:30971/TCP   7m33s   app.kubernetes.io/component=prometheus,app.kubernetes.io/instance=k8s,app.kubernetes.io/name=prometheus,app.kubernetes.io/part-of=kube-prometheus
prometheus-operated     ClusterIP      None             <none>          9090/TCP                        7m30s   app.kubernetes.io/name=prometheus
prometheus-operator     ClusterIP      None             <none>          8443/TCP                        7m32s   app.kubernetes.io/component=controller,app.kubernetes.io/name=prometheus-operator,app.kubernetes.io/part-of=kube-prometheus

k8s拉取私有仓库

kubectl create secret docker-registry <secret_name> \
  --docker-server=harbor.example.com \
  --docker-username=your-username \
  --docker-password=your-password \
  --docker-email=your-email@example.com \
  --namespace=default

secret_name为密钥名称，用户自定义
docker-server为私有仓库地址
docker-username和docker-password为对应仓库用户账号密码

在创建完成后可以在pod生命时添加改secret，这样kubelet就可以有拉取私有镜像的权限了。

apiVersion: v1
kind: Pod
metadata:
  name: xxx
spec:
  containers:
  - name: xxx
    image: xxx
  imagePullSecrets:
  - name: <secret_name>

参考资料

[1] Docker Documentation. “Install Docker Engine on Ubuntu.” [Online]. Available: https://docs.docker.com/engine/install/ubuntu/

[2] Kubernetes Documentation. “安装 kubeadm | Kubernetes.” [Online]. Available: https://kubernetes.io/zh-cn/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

[3] Cri-docker Documentation. “Mirantis/cri-dockerd: dockerd as a compliant Container Runtime Interface for Kubernetes.” [Online]. Available: https://github.com/Mirantis/cri-dockerd

[4] Flannel Project. “flannel-io/flannel: Network fabric for containers, designed for Kubernetes.” [Online]. Available: https://github.com/flannel-io/flannel

[5] NVIDIA Documentation. “Time-Slicing GPUs in Kubernetes.” [Online]. Available: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-sharing.html

[6] 指月小筑(探索云原生). “GPU 环境搭建指南：使用 GPU Operator 加速 Kubernetes GPU 环境搭建.” [Online]. Available: https://www.lixueduan.com/posts/ai/02-gpu-operator/

[7] 指月小筑(探索云原生). “一文搞懂 GPU 共享方案：NVIDIA Time Slicing.” [Online]. Available: https://www.lixueduan.com/posts/kubernetes/25-gpu-share-time-slicing/

[8] Prometheus Operator. “kube-prometheus: Use Prometheus to monitor Kubernetes and applications running on Kubernetes.” [Online]. Available: https://github.com/prometheus-operator/kube-prometheus