测试环境搭建

  • kubernetes master + 2 node （使用cilium替代kube-proxy）

  • helm3

  • cilium+hubble

  • 切换superedge

  • 安装ISTIO

  • 安装Karmada

1
kubeadm init --pod-network-cidr=10.218.0.0/16 --skip-phases=addon/kube-proxy
1
kubeadm init --kubernetes-version=1.18.20  --image-repository registry.aliyuncs.com/google_containers  --service-cidr=10.10.0.0/16 --pod-network-cidr=10.218.0.0/16   --skip-phases=addon/kube-proxy
1
2
3
4
5
6
7
helm install cilium cilium/cilium --version 1.10.4 \
--namespace kube-system \
--set kubeProxyReplacement=strict \
--set k8sServiceHost=192.168.92.132 \
--set k8sServicePort=6443 \
--set hubble.relay.enabled=true \
--set hubble.ui.enabled=true
1
2
3
4
5
6
7
helm install cilium cilium/cilium --version 1.10.4 \
--namespace kube-system \
--set kubeProxyReplacement=strict \
--set k8sServiceHost=183.131.145.82 \
--set k8sServicePort=6443 \
--set hubble.relay.enabled=true \
--set hubble.ui.enabled=true
  • 将 Hubble UI 改为 NodePort 访问
1
kubectl -n kube-system patch svc hubble-ui -p '{"spec": {"type": "NodePort"}}'
1
kubectl -n kube-system get svc hubble-ui

升级内核

1
2
3
4
5
6
7
8
9
10
11
12
13
wget https://mirrors.nju.edu.cn/elrepo/kernel/el7/x86_64/RPMS/kernel-lt-5.4.151-1.el7.elrepo.x86_64.rpm

rpm -ivh kernel-lt-5.4.151-1.el7.elrepo.x86_64.rpm

cat /boot/grub2/grub.cfg | grep menuentry

grub2-set-default 'CentOS Linux (5.4.151-1.el7.elrepo.x86_64) 7 (Core)'

grub2-editenv list

grub2-mkconfig -o /boot/grub2/grub.cfg

reboot

遇到问题

image-20211012155844618

  • Cilium pod CrashLoopBackOff

    内核最低支持4.8.0

    image-20211012160038293

  • coredns ContainerCreating

多公网集群搭建

1
kubeadm init --kubernetes-version=1.18.20  --apiserver-advertise-address=机器Ip  --image-repository registry.aliyuncs.com/google_containers  --service-cidr=10.10.0.0/16 --pod-network-cidr=10.122.0.0/16

#在kubeadm-config.yml设置好镜像源，版本，集群网段，–apiserver-advertise-address为外网ip 等

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
cat <<EOF > kubeadm-init.yaml
apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.66wcf1rc5wk6637f            ## token 建议重新生成别用默认的
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: "IPADDRESS"     # 公网IP地址
  bindPort: 6443            # API 端口
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: master001
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
---
apiServer:
  extraArgs:
    advertise-address: "IPADDRESS"    ## 公网机器互联需要
  certSANs:
  - "IPADDRESS"
  timeoutForControlPlane: 10m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers       # image的仓库源
kind: ClusterConfiguration
kubernetesVersion: v1.16.0
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
  podSubnet: 192.168.0.0/16
scheduler: {}
EOF

以上配置文件中的IPADDRESS作为占位符,可以通过以下命令替换成公网IP

1
2
ip=$(curl -s -4 ip.sb)
sed -i "s/IPADDRESS/$ip/g" kubeadm-init.yaml
1
2
3
4
5
6
kubeadm init phase preflight --config myKubeadmConfig.yaml  --ignore-preflight-errors NumCPU
kubeadm init phase certs all --config myKubeadmConfig.yaml
kubeadm init phase kubeconfig all --config myKubeadmConfig.yaml
kubeadm init phase kubelet-start --config myKubeadmConfig.yaml
kubeadm init phase control-plane all --config myKubeadmConfig.yaml
kubeadm init phase etcd local --config myKubeadmConfig.yaml
1
2
vi /etc/kubernetes/manifests/etcd.yaml
#把--listen-client-urls 和 --listen-peer-urls 都改成0.0.0.0:xxx
1
kubeadm init --skip-phases=preflight,certs,kubeconfig,kubelet-start,control-plane,etcd,addon/kube-proxy --config myKubeadmConfig.yaml
1
2
kubeadm join 172.25.156.55:6443 --token y9ohc0.77zro9ruyyub4t0w \
    --discovery-token-ca-cert-hash sha256:b84e69459da3ee0e1c26a4b8ffa4f778040bd93511a0465d83596edef7bcb56a
1
iptables -t nat -A OUTPUT -d 172.25.156.55 -j DNAT --to-destination 120.79.141.184

高可用实验环境搭建

安装docker

安装所需的软件包。yum-utils 提供了 yum-config-manager ，并且 device mapper 存储驱动程序需要 device-mapper-persistent-data 和 lvm2

1
2
3
sudo yum install -y yum-utils \
  device-mapper-persistent-data \
  lvm2

设置国内仓库

1
2
3
sudo yum-config-manager \
    --add-repo \
    http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

安装最新版本的 Docker Engine-Community 和 containerd

1
sudo yum install docker-ce docker-ce-cli containerd.io

配置 Docker 守护程序，尤其是使用 systemd 来管理容器的 cgroup。

1
2
3
4
5
6
7
8
9
10
11
sudo mkdir /etc/docker
cat <<EOF | sudo tee /etc/docker/daemon.json
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF

启动docker

1
sudo systemctl start docker && systemctl enable docker
1
sudo systemctl start kubelet && systemctl enable kubelet

安装kubeadm

修改主机名

节点之中不可以有重复的主机名、MAC 地址或 product_uuid

允许 iptables 检查桥接流量

确保 br_netfilter 模块被加载。这一操作可以通过运行 lsmod | grep br_netfilter 来完成。若要显式加载该模块，可执行 sudo modprobe br_netfilter。

为了让你的 Linux 节点上的 iptables 能够正确地查看桥接流量，你需要确保在你的 sysctl 配置中将 net.bridge.bridge-nf-call-iptables 设置为 1。例如：

1
2
3
4
5
6
7
8
9
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sudo sysctl --system

开启包转发

1
2
3
vi /etc/sysctl.conf
添加：
net.ipv4.ip_forward = 1

开放指定端口

1
2
3
4
5
6
firewall-cmd --permanent --add-port=6443/tcp
firewall-cmd --permanent --add-port=2379-2380/tcp
firewall-cmd --permanent --add-port=10250/tcp
firewall-cmd --permanent --add-port=10251/tcp
firewall-cmd --permanent --add-port=10252/tcp
firewall-cmd --permanent --add-port=8443/tcp
1
systemctl restart firewalld
1
firewall-cmd --list-ports

控制平面节点

协议 方向 端口范围 作用 使用者
TCP 入站 6443 Kubernetes API 服务器 所有组件
TCP 入站 2379-2380 etcd 服务器客户端 API kube-apiserver, etcd
TCP 入站 10250 Kubelet API kubelet 自身、控制平面组件
TCP 入站 10251 kube-scheduler kube-scheduler 自身
TCP 入站 10252 kube-controller-manager kube-controller-manager 自身

工作节点

协议 方向 端口范围 作用 使用者
TCP 入站 10250 Kubelet API kubelet 自身、控制平面组件
TCP 入站 30000-32767 NodePort 服务† 所有组件

设置内核、K8S的yum源

1
2
3
4
5
6
7
8
9
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
1
2
3
4
5
6
7
# 将 SELinux 设置为 permissive 模式（相当于将其禁用）
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

sudo systemctl enable --now kubelet
1
2
3
#关闭swap
swapoff -a 
sed -i 's/.*swap.*/#&/' /etc/fstab

查找指定版本

1
yum list kubelet  --showduplicates | sort -r

安装kubernetes1.18.20

1
sudo yum install -y kubeadm-1.18.20-0 kubectl-1.18.20-0 kubelet-1.18.20-0

keepalived

Master 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
! /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
}
vrrp_script check_apiserver {
  script "/etc/keepalived/check_apiserver.sh"
  interval 3
  weight -2
  fall 10
  rise 2
}

vrrp_instance VI_1 {
    state MASTER
    interface em1.101@em1
    virtual_router_id 51
    priority 100
    authentication {
        auth_type PASS
        auth_pass keepalive
    }
    virtual_ipaddress {
        183.131.145.10
    }
    track_script {
        check_apiserver
    }
}

backup

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
! /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
}
vrrp_script check_apiserver {
  script "/etc/keepalived/check_apiserver.sh"
  interval 3
  weight -2
  fall 10
  rise 2
}

vrrp_instance VI_1 {
    state BACKUP
    interface ${INTERFACE}
    virtual_router_id 51
    priority 50
    authentication {
        auth_type PASS
        auth_pass keepalive
    }
    virtual_ipaddress {
        183.131.145.10
    }
    track_script {
        check_apiserver
    }
}

/etc/keepalived/check_apiserver.sh

1
2
3
4
5
6
7
8
9
10
11
12

#!/bin/sh

errorExit() {
    echo "*** $*" 1>&2
    exit 1
}

curl --silent --max-time 2 --insecure https://localhost:8443/ -o /dev/null || errorExit "Error GET https://localhost:8443/"
if ip addr | grep -q 183.131.145.10; then
    curl --silent --max-time 2 --insecure https://183.131.145.10:8443/ -o /dev/null || errorExit "Error GET https://183.131.145.10:8443/"
fi

haproxy

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
# /etc/haproxy/haproxy.cfg
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    log /dev/log local0
    log /dev/log local1 notice
    daemon

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 1
    timeout http-request    10s
    timeout queue           20s
    timeout connect         5s
    timeout client          20s
    timeout server          20s
    timeout http-keep-alive 10s
    timeout check           10s

#---------------------------------------------------------------------
# apiserver frontend which proxys to the control plane nodes
#---------------------------------------------------------------------
frontend apiserver
    bind *:8443
    mode tcp
    option tcplog
    default_backend apiserver

#---------------------------------------------------------------------
# round robin balancing for apiserver
#---------------------------------------------------------------------
backend apiserver
    option httpchk GET /healthz
    http-check expect status 200
    mode tcp
    option ssl-hello-chk
    balance     roundrobin
        server master01 183.131.145.85:6443 check
        server master02 183.131.145.84:6443 check
        server master03 61.153.100.147:6443 check
        # [...]

keepalive static pod

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  name: keepalived
  namespace: kube-system
spec:
  containers:
  - image: osixia/keepalived:2.0.17
    name: keepalived
    resources: {}
    securityContext:
      capabilities:
        add:
        - NET_ADMIN
        - NET_BROADCAST
        - NET_RAW
    volumeMounts:
    - mountPath: /usr/local/etc/keepalived/keepalived.conf
      name: config
    - mountPath: /etc/keepalived/check_apiserver.sh
      name: check
  hostNetwork: true
  volumes:
  - hostPath:
      path: /etc/keepalived/keepalived.conf
    name: config
  - hostPath:
      path: /etc/keepalived/check_apiserver.sh
    name: check
status: {}

haproxy

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
apiVersion: v1
kind: Pod
metadata:
  name: haproxy
  namespace: kube-system
spec:
  containers:
  - image: haproxy:2.1.4
    name: haproxy
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: localhost
        path: /healthz
        port: 8443
        scheme: HTTPS
    volumeMounts:
    - mountPath: /usr/local/etc/haproxy/haproxy.cfg
      name: haproxyconf
      readOnly: true
  hostNetwork: true
  volumes:
  - hostPath:
      path: /etc/haproxy/haproxy.cfg
      type: FileOrCreate
    name: haproxyconf
status: {}

/etc/kubernetes/manifests/kube-vip.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  name: kube-vip
  namespace: kube-system
spec:
  containers:
  - command:
    - /kube-vip
    - start
    - -c
    - /vip.yaml
    image: 'plndr/kube-vip:0.3.1'
    name: kube-vip
    resources: {}
    securityContext:
      capabilities:
        add:
        - NET_ADMIN
        - SYS_TIME
    volumeMounts:
    - mountPath: /vip.yaml
      name: config
  hostNetwork: true
  volumes:
  - hostPath:
      path: /etc/kube-vip/config.yaml
    name: config
status: {}
1
2
3
4
5
6
7
sudo docker run --network host --rm plndr/kube-vip:v0.3.7 manifest pod \
--interface em1.101 \
--vip 183.131.145.82 \
--controlplane \
--services \
--arp \
--leaderElection | sudo tee /etc/kubernetes/manifests/kube-vip.yaml
1
2
3
4
5
6
7
sudo kubeadm init \
    --kubernetes-version=1.18.20 \
    --control-plane-endpoint=183.131.145.82 \
    --image-repository=registry.aliyuncs.com/google_containers \
    --pod-network-cidr=10.218.0.0/16 \
    --skip-phases=addon/kube-proxy \
    --upload-certs
1
sudo kubeadm init --config kubeadm-config.yml --upload-certs --skip-phases=addon/kube-proxy

SUCCESS

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

  kubeadm join 183.131.145.82:6443 --token hxduyw.e7pm2u32wcffr652 \
    --discovery-token-ca-cert-hash sha256:53aea6bf8eeefca7a672021c80e203da459f1655a43127ab557ed5a7d3275eec \
    --control-plane --certificate-key 273b46dcea4ecff30ebed45812b99ecb7d02cc493a88ea60bb547e95c14a4ff8

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 183.131.145.82:6443 --token hxduyw.e7pm2u32wcffr652 \
    --discovery-token-ca-cert-hash sha256:53aea6bf8eeefca7a672021c80e203da459f1655a43127ab557ed5a7d3275eec
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
This node has joined the cluster and a new control plane instance was created:

* Certificate signing request was sent to apiserver and approval was received.
* The Kubelet was informed of the new secure connection details.
* Control plane (master) label and taint were applied to the new node.
* The Kubernetes control plane instances scaled up.
* A new etcd member was added to the local/stacked etcd cluster.

To start administering your cluster from this node, you need to run the following as a regular user:

	mkdir -p $HOME/.kube
	sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
	sudo chown $(id -u):$(id -g) $HOME/.kube/config

Run 'kubectl get nodes' to see this node join the cluster.

刚才加入集群的 master 节点，不要忘记将 kube-vip 的静态 pod 进行创建，这样才能确保集群拥有了 HA 属性 。执行上面的 kube-vip 部署步骤

其他节点部署kube-vip static pod

1
2
3
4
5
6
7
sudo docker run --network host --rm plndr/kube-vip:v0.3.7 manifest pod \
--interface bond0.101 \
--vip 183.131.145.82 \
--controlplane \
--services \
--arp \
--leaderElection | sudo tee /etc/kubernetes/manifests/kube-vip.yaml

ERROR1

1
2
3
4
5
6
7
Oct 21 10:44:01 master01-183-131-145-85 kubelet[23679]: E1021 10:44:01.664374   23679 kubelet.go:2292] node "master01-183-131-145-85" not found
Oct 21 10:44:01 master01-183-131-145-85 kubelet[23679]: E1021 10:44:01.764481   23679 kubelet.go:2292] node "master01-183-131-145-85" not found
Oct 21 10:44:01 master01-183-131-145-85 kubelet[23679]: E1021 10:44:01.864582   23679 kubelet.go:2292] node "master01-183-131-145-85" not found
Oct 21 10:44:01 master01-183-131-145-85 kubelet[23679]: E1021 10:44:01.964685   23679 kubelet.go:2292] node "master01-183-131-145-85" not found
Oct 21 10:44:02 master01-183-131-145-85 kubelet[23679]: E1021 10:44:02.064796   23679 kubelet.go:2292] node "master01-183-131-145-85" not found
Oct 21 10:44:02 master01-183-131-145-85 kubelet[23679]: E1021 10:44:02.164907   23679 kubelet.go:2292] node "master01-183-131-145-85" not found
Oct 21 10:44:02 master01-183-131-145-85 kubelet[23679]: E1021 10:44:02.265034   23679 kubelet.go:2292] node "master01-183-131-145-85" not found

ERROR2

1
Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")

本博客所有文章除特别声明外，均采用 CC BY-SA 4.0 协议 ，转载请注明出处！