官方文档: https://ubuntu.com/kubernetes/docs

开始之前

集群ip规划，所有机器系统都是ubuntu 20.04

hostname ip
juju-client 10.0.0.10
juju-controller-1 10.0.0.11
juju-master-1 10.0.0.21
juju-master-2 10.0.0.22
juju-master-3 10.0.0.23
juju-worker-1 10.0.0.31
juju-worker-2 10.0.0.32

juju-client 为juju客户端和haproxy机器
juju-controller-1为juju控制器节点(可以做高可用)

以下操作都是在juju-client上执行

安装juju

1
sudo snap install juju --classic

设置云类型

1
2
juju add-cloud
输入manual

添加机器(一共5台)

1
2
juju bootstrap
juju add-machine ssh:root@x.x.x.x

机器添加完成后, juju machines 能看到机器id

生成yaml

在https://jujucharms.com/new/画图，然后导出成yaml
etcd 还是 和 master 分开部署
etcd.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
description: Kubernetes Cluster Deploy.
series: focal
machines:
  '0':
    series: focal
  '1':
    series: focal
  '2':
    series: focal
  '3':
    series: focal
  '4':
    series: focal
applications:
  easyrsa:
    charm: cs:~containers/easyrsa-345
    num_units: 1
    resources:
      easyrsa: 5
    to:
    - '0'
  etcd:
    charm: cs:~containers/etcd-553
    num_units: 5
    options:
      channel: 3.4/stable
      bind_to_all_interfaces: false
    resources:
      core: 0
      etcd: 3
      snapshot: 0
    to:
    - '0'
    - '1'
    - '2'
    - '3'
    - '4'
relations:
- - etcd:certificates
  - easyrsa:client

core.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
description: Kubernetes Cluster Deploy.
series: focal
machines:
  '0':
    series: focal
  '1':
    series: focal
  '2':
    series: focal
  '3':
    series: focal
  '4':
    series: focal
  '5':
    series: focal
  '6':
    series: focal
  '7':
    series: focal
  '8':
    series: focal
applications:
  containerd:
    charm: cs:~containers/containerd-102
    resources: {}
  easyrsa:
    charm: cs:~containers/easyrsa-345
    num_units: 1
    resources:
      easyrsa: 5
    to:
    - '0'
  etcd:
    charm: cs:~containers/etcd-553
    num_units: 5
    options:
      channel: 3.4/stable
      bind_to_all_interfaces: false
    resources:
      core: 0
      etcd: 3
      snapshot: 0
    to:
    - '0'
    - '1'
    - '2'
    - '3'
    - '4'
  kubeapi-load-balancer:
    charm: cs:~containers/kubeapi-load-balancer-757
    expose: true
    num_units: 1
    resources: {}
    to:
    - '5'
  kubernetes-master:
    charm: cs:~containers/kubernetes-master-955
    expose: true
    num_units: 3
    options:
      channel: 1.20/stable
      service-cidr: 172.31.64.0/21
      enable-dashboard-addons: false
      proxy-extra-args: proxy-mode=ipvs
    resources:
      cdk-addons: 0
      core: 0
      kube-apiserver: 0
      kube-controller-manager: 0
      kube-proxy: 0
      kube-scheduler: 0
      kubectl: 0
    to:
    - '5'
    - '6'
    - '7'
  kubernetes-worker:
    charm: cs:~containers/kubernetes-worker-726
    expose: true
    num_units: 1
    options:
      channel: 1.20/stable
      proxy-extra-args: proxy-mode=ipvs
    resources:
      cni-amd64: 708
      cni-arm64: 699
      cni-s390x: 711
      core: 0
      kube-proxy: 0
      kubectl: 0
      kubelet: 0
    to:
    - '8'
  canal:
    charm: 'cs:~containers/canal-755'
    options:
      cidr: 172.31.0.0/18
      iface: eth0
      ignore-loose-rpf: true
    series: focal
relations:
- - kubernetes-master:kube-api-endpoint
  - kubeapi-load-balancer:apiserver
- - kubernetes-master:loadbalancer
  - kubeapi-load-balancer:loadbalancer
- - kubernetes-worker:kube-api-endpoint
  - kubeapi-load-balancer:website
- - kubernetes-master:kube-control
  - kubernetes-worker:kube-control
- - kubernetes-master:certificates
  - easyrsa:client
- - kubeapi-load-balancer:certificates
  - easyrsa:client
- - kubernetes-master:etcd
  - etcd:db
- - kubernetes-worker:certificates
  - easyrsa:client
- - etcd:certificates
  - easyrsa:client
- - canal:etcd
  - etcd:db
- - canal:cni
  - kubernetes-master:cni
- - canal:cni
  - kubernetes-worker:cni
- - containerd:containerd
  - kubernetes-worker:container-runtime
- - containerd:containerd
  - kubernetes-master:container-runtime

根据yml来部署

1
2
juju deploy ./etcd.yaml --map-machines=existing,0=0,1=1,2=2,3=3,4=4
juju deploy ./core.yaml --map-machines=existing,0=0,1=1,2=2,3=3,4=4,5=5,6=6,7=7,8=8

juju status 全部idle就算正常了

扩容worker节点

1
2
juju add-machine ssh:root@x.x.x.x
juju add-unit kubernetes-worker --to <machine_id>

kata容器

部署

1
2
3
4
juju deploy cs:~containers/kata
juju add-relation kata kubernetes-master
juju add-relation kata kubernetes-worker
juju add-relation kata:untrusted containerd:untrusted

使用

部署的时候加上io.kubernetes.cri.untrusted-workload: "true"

1
2
3
4
5
6
7
8
9
apiVersion: v1
kind: Pod
metadata:
  name: nginx-untrusted
  annotations:
    io.kubernetes.cri.untrusted-workload: "true"
spec:
  containers:
    image: nginx

删除kata运行时

1
2
3
4
juju remove-relation --force kata kubernetes-master
juju remove-relation --force kata kubernetes-worker
juju remove-relation --force kata:untrusted containerd:untrusted
juju remove-application kata

更换master节点

先删除节点

1
2
3
juju remove-unit etcd/1 --force --no-wait
juju remove-unit kubernetes-master/1 --force --no-wait
juju remove-machine 1 --force --no-wait

etcd集群删除这个member

https://ubuntu.com/kubernetes/docs/charm-etcd

需要先下载证书到本地(要解压)

1
2
juju run-action --wait etcd/0 package-client-credentials
juju scp etcd/0:etcd_credentials.tar.gz etcd_credentials.tar.gz

然后使用etcdctl删除节点

1
2
3
4
5
6
7
export ETCDCTL_KEY_FILE=$(pwd)/client.key
export ETCDCTL_CERT_FILE=$(pwd)/client.crt
export ETCDCTL_CA_FILE=$(pwd)/ca.crt
export ETCDCTL_ENDPOINT=https://10.0.0.21:2379
etcdctl member list

etcdctl member remove c2499df1988d1925

增加节点

1
2
3
juju add-machine ssh:root@100.64.1.167
juju machines
记住节点ID,假如是9

扩容master到节点9

1
juju add-unit kubernetes-master --to 9

扩容etcd到节点9

1
juju add-unit etcd --to 9

master节点高可用

方案1 搭建haproxy负载均衡

添加了参数proxy-extra-args: proxy-mode=ipvs表示使用lvs做负载均衡，可以不用haproxy

  1. haproxy配置文件

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    ...省略

    frontend http_ingress_traffic_fe
        bind 0.0.0.0:80
        mode tcp
        default_backend   http_ingress_traffic_be

    backend http_ingress_traffic_be
        mode tcp
        balance     roundrobin
        server      juju-worker-1 10.0.0.31:80 check
        server      juju-worker-2 10.0.0.32:80 check

    frontend https_ingress_traffic_fe
        bind 0.0.0.0:443
        mode tcp
        default_backend   https_ingress_traffic_be

    backend https_ingress_traffic_be
        mode tcp
        balance     roundrobin
        server      juju-worker-1 10.0.0.31:443 check
        server      juju-worker-2 10.0.0.32:443 check

    frontend k8s_api_fe
        bind 0.0.0.0:6443
        mode tcp
        default_backend   k8s_api_be

    backend k8s_api_be
        mode tcp
        balance     roundrobin
        server      juju-master-1 10.0.0.21:6443 check
        server      juju-master-2 10.0.0.22:6443 check
        server      juju-master-3 10.0.0.23:6443 check

  2. 修改负载均衡的ip

    1
    juju config kubernetes-master loadbalancer-ips="10.0.0.10"

方案2 使用keepalived

参考文章：https://ubuntu.com/kubernetes/docs/keepalived

juju-controller高可用

增加新的controller节点

首先切换到controller

1
2
juju switch controller
# juju switch default 切换回来
1
juju add-machine ssh:root@100.64.1.169

查看controller机器

1
juju machines

开启高可用

1
juju enable-ha --to 5,6

查看controller信息

1
2
juju controllers --refresh
juju controllers

其他常见问题

安装kubectl

  1. 安装

    1
    sudo snap install kubectl --classic

  2. 获取新的kubeconfig配置文件

    1
    juju scp kubernetes-master/0:config ~/.kube/config

操作etcd

1
2
3
4
juju run-action --wait etcd/10 package-client-credentials
juju scp etcd/25:etcd_credentials.tar.gz etcd_credentials.tar.gz

etcdctl --cacert=$(pwd)/ca.crt --cert=$(pwd)/client.crt --key=$(pwd)/client.key --endpoints="https://172.31.72.5:2379" member list

重新添加节点

比如已经添加过的机器，由于配置出错或者其他原因想重新添加进来初始化，可以先移除再添加。

1
juju remove-machine <machine-id> --force

在目标机器上执行

1
2
sudo /sbin/remove-juju-services
sudo rm -rf /root/cdk /var/lib/juju/ /opt/calicoctl