Kubeadm介绍

kubeadm是官方社区推出的一个用于快速部署kubernetes集群的工具,是一个K8s部署工具，提供kubeadm init和kubeadm join.

官方地址：https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm/

这个工具能通过两条指令完成一个kubernetes集群的部署：

# 创建一个 Master 节点
$ kubeadm init

# 将一个 Node 节点加入到当前集群中
$ kubeadm join <Master节点的IP和端口>

在开始之前，部署Kubernetes集群机器需要满足以下几个条件：

• 一台或多台机器，操作系统 CentOS7.x-86_x64
• 硬件配置：2GB或更多RAM，2个CPU或更多CPU，硬盘30GB或更多
• 集群中所有机器之间网络互通
• 可以访问外网，需要拉取镜像
• 禁止swap分区

资源规划

角色	ip	组件
k8s-master1	192.168.60.128	Docker 19.03.15 kubeadm 1.19.0 kubelet 1.19.0
k8s-node1	192.168.60.129	Docker 19.03.15 kubeadm 1.19.0 kubelet 1.19.0
k8s-node2	192.168.60.130	Docker 19.03.15 kubeadm 1.19.0 kubelet 1.19.0

单master架构图

基础环境配置

所有节点都操作

关闭防火墙

关闭防火墙：
systemctl stop firewalld
systemctl disable firewalld

关闭selinux

setenforce 0
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config

关闭swap

swapoff -a  # 临时
sed -ri 's/.*swap.*/#&/' /etc/fstab # 永久

主机名称修改

#每台主机修改对应的即可
hostnamectl set-hostname k8s-master1
hostnamectl set-hostname k8s-node1
hostnamectl set-hostname k8s-node2

修改host

cat >> /etc/hosts << EOF
192.168.60.128 k8s-master1
192.168.60.129 k8s-node1
192.168.60.130 k8s-node2
EOF

将桥接的IPv4流量传递到iptables的链

cat >> /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
#使生效
sysctl --system

时间同步

yum install ntpdate -y
ntpdate time.windows.com

Docker/kubeadm/kubelet安装所有节点

Kubernetes默认CRI（容器运行时）为Docker，因此先安装Docker,节点无需安装kubectl

docker安装

详情见我的另一篇文章https://www.wangjian.run/docker/

kubeadm/kubelet安装

添加阿里云YUM软件源

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

setenforce 0

#安装最新的
yum install -y kubelet kubeadm kubectl

#安装指定版本
yum install -y kubelet-1.18.0 kubeadm-1.18.0 kubectl-1.18.0
yum install -y kubelet-1.19.0 kubeadm-1.19.0 kubectl-1.19.0

#只下载不按照
mkdir -p /root/kubelet-kubeadm-kubectl-1.19.0-rpm
yum -y install --downloadonly --downloaddir=/root/kubelet-kubeadm-kubectl-1.19.0 kubelet-1.19.0 kubeadm-1.19.0 kubectl-1.19.0

ps: 由于官网未开放同步方式, 可能会有索引gpg检查失败的情况, 这时请用 yum install -y --nogpgcheck kubelet kubeadm kubectl 安装

设置Kubelet开机自启动 ( 如果启动失败无须管理，初始化成功以后即可启动 )
[ERROR Port-10250]: Port 10250 is in use kubelet用的就是这个端口

查看日志
journalctl -xefu kubelet

#开机启动 这步必须要做，不然后面检查不通过。
systemctl enable kubelet 

部署Master节点

先拉取镜像

1、根据需要修改对应的参数

• 先获取默认的初始化参数

kubeadm config print init-defaults > init.default.yaml

#默认参数如下:
apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 1.2.3.4
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: k8s-master1
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: k8s.gcr.io
kind: ClusterConfiguration
kubernetesVersion: v1.19.0
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
scheduler: {}

• 备份下原始文件，进行修改

cp init.default.yaml init-config.yaml

主要是修改：imageRepository: k8s.gcr.io

改为阿里源:imageRepository: registry.aliyuncs.com/google_containers

此文件是关键，主要是用此配置文件中的参数 去拉取相关镜像。

2、提前下载需要的镜像文件

查看有哪些镜像文件

• 列出所需的镜像列表

kubeadm config images list

• 下载镜像文件 两种下载方式都可以

docker pull #这个方式是去hubdocker中去拉去镜像

kubeadm config images pull –config=init-config.yaml #这个方式是根据配置文件中的参数去拉取镜像，因为我的安装的kubeadm等是1.19版本，所有拉取的也是此版本对应的镜像文件。我们配置的是阿里镜像，会去阿里仓库去拉取，所以此方式最合适。

在拉取前我们查看系统已有的镜像情况:

[root@k8s-master1 kubeadm-kubelet-1.19.16]# docker images

结果显示镜像为空。

拉取过程：

拉取之后查看镜像：

init初始化开始

在执行init之前系统会先预检查 ,用户可以先自行预检查，如果不希望检查可使用--ignore-preflight-errors
kubeadm init phase preflight 
检查完成后再执行(此操作只需在master操作即可)
kubeadm init \
  --apiserver-advertise-address=192.168.60.128 \
  --image-repository registry.aliyuncs.com/google_containers \
  --kubernetes-version v1.19.0 \
  --service-cidr=10.1.0.0/16\
  --pod-network-cidr=10.244.0.0/16

备注:
apiserver-advertise-address修改为master地址
由于默认拉取镜像地址k8s.gcr.io国内无法访问，这里指定阿里云镜像仓库地址。

如果执行报错，删除生产的配置文件在执行一次
rm -rf /etc/kubernetes/manifests/kube-apiserver.yaml
rm -rf /etc/kubernetes/manifests/kube-controller-manager.yaml
rm -rf /etc/kubernetes/manifests/kube-scheduler.yaml
rm -rf /etc/kubernetes/manifests/etcd.yaml

下面是执行成功后的实例：

[root@k8s-master1 kubelet-kubeadm-kubectl-1.19.0-rpm]# kubeadm init \
>   --apiserver-advertise-address=192.168.60.128 \
>   --image-repository registry.aliyuncs.com/google_containers \
>   --kubernetes-version v1.19.0 \
>   --service-cidr=10.1.0.0/16\
>   --pod-network-cidr=10.244.0.0/16
W1111 14:24:02.535537    1998 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[init] Using Kubernetes version: v1.19.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.1.0.1 192.168.60.128]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master1 localhost] and IPs [192.168.60.128 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master1 localhost] and IPs [192.168.60.128 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 18.505052 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.19" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s-master1 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s-master1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: 6r2zvx.jz6a9x3zb68ra8d4
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.60.128:6443 --token 6r2zvx.jz6a9x3zb68ra8d4 \
    --discovery-token-ca-cert-hash sha256:8e0eef06e2e1607438d091d25c5947688e7bca62cc327e376aefb1f14f6421b8 

此日志很重要，很重要，安装完成后，保存下来备用，每次安装后信息都不一样。日志内容需要进行三步操作。

第一步

拷贝kubectl连接集群用的配置文件(刚初始化日志里面有)master操作(非root执行下面的命令)

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

root用户执行这个命令就行
export KUBECONFIG=/etc/kubernetes/admin.conf

第二步执行网络部署master操作

You should now deploy a pod network to the cluster. Run “kubectl apply -f [podnetwork].yaml” with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/

第三步，加入节点 (节点操作)

kubeadm join 192.168.60.128:6443 –token 6r2zvx.jz6a9x3zb68ra8d4 \ –discovery-token-ca-cert-hash sha256:8e0eef06e2e1607438d091d25c5947688e7bca62cc327e376aefb1f14f6421b8

至此以上流程安装完成，结果查看(没有安装网络的情况下)

[root@k8s-master1 ~]# kubectl get nodes
NAME          STATUS     ROLES    AGE     VERSION
k8s-master1   NotReady   master   4m52s   v1.19.0

因为没部署网络，所有状态是notready

部署CNI网络

• 网络calico

https://github.com/projectcalico/calico

#要选择对应的版本，不然安装不上
wget -O calicov3.20.yaml https://docs.projectcalico.org/v3.20/manifests/calico.yaml

#选择最新的
wget https://docs.projectcalico.org/manifests/calico.yaml

#修改对应的参数

#提前下载需要镜像，查看需要的镜像
[root@k8s-master1 ~]# grep image calicov3.20.yaml 
          image: docker.io/calico/cni:v3.20.6
          image: docker.io/calico/cni:v3.20.6
          image: docker.io/calico/pod2daemon-flexvol:v3.20.6
          image: docker.io/calico/node:v3.20.6
          image: docker.io/calico/kube-controllers:v3.20.6
#下载
for i in calico/cni:v3.20.6 calico/pod2daemon-flexvol:v3.20.6 calico/node:v3.20.6 calico/kube-controllers:v3.20.6 ; do docker pull $i ; done

#安装,安装之前要修改两处，见下面（此操作只需在master操作即可）
kubectl apply -f calicov3.20.yaml
#删除  
kubectl delete -f calicov3.20.yaml

#删除dashboard时，几个其它几个节点也要开机，不要无法删除。
#3.20版本对应 k8s:1.19 1.20 1.21

安装之前记得修改两个地方

修改calico.yaml里的pod网段

把calico.yaml里pod所在网段改成kubeadm init时选项--pod-network-cidr所指定的网段，
直接用vim编辑打开此文件查找192，按如下标记进行修改：
# no effect. This should fall within `--cluster-cidr`.
# - name: CALICO_IPV4POOL_CIDR
#   value: "192.168.0.0/16"
# Disable file logging so `kubectl logs` works.
- name: CALICO_DISABLE_FILE_LOGGING
value: "true"
把两个#及#后面的空格去掉，并把192.168.0.0/16改成10.244.0.0/16
# no effect. This should fall within `--cluster-cidr`.
- name: CALICO_IPV4POOL_CIDR
value: "10.244.0.0/16"
# Disable file logging so `kubectl logs` works.
- name: CALICO_DISABLE_FILE_LOGGING
value: "true"

修改images名字，大概一共4处

• 网络flannel

wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
sed -i -r "s#quay.io/coreos/flannel:.*-amd64#lizhenliang/flannel:v0.12.0-amd64#g" kube-flannel.yml

默认镜像地址无法访问，修改为docker hub镜像仓库。

kubectl apply -f kube-flannel.yml

部署work节点

Docker/kubeadm/kubelet/网络插件, 前提都已安装完成，相关镜像都已导入完成。

部署好网络插件，Node准备就绪。

加入Kubernetes Node

向集群添加新节点，使用在Master执行kubeadm init最后输出的kubeadm join命令：

#此操作是在节点操作
kubeadm join 192.168.60.128:6443 --token 6r2zvx.jz6a9x3zb68ra8d4 \
    --discovery-token-ca-cert-hash sha256:8e0eef06e2e1607438d091d25c5947688e7bca62cc327e376aefb1f14f6421b8 

默认token有效期为24小时，当过期之后，该token就不可用了。这时就需要重新创建token，可以直接使用命令快捷生成：

#mast节点执行
kubeadm token create --print-join-command

在master执行kubectl get nodes，显示节点角色ROLES 为none

kubectl get node
NAME         STATUS     ROLES    AGE   VERSION
k8s-master   NotReady   <none>   7s    v1.18.3

需要在master执行以下语句

kubectl label node k8s-node1 node-role.kubernetes.io/worker=worker
kubectl label node k8s-node2 node-role.kubernetes.io/worker=worker

再次执行kubectl get nodes就显示work节点了

节点至少有这三个镜像，至少有cni镜像，不然节点的状态是NotReady

测试kubernetes集群nginx

在Kubernetes集群中创建一个pod，验证是否正常运行：

kubectl get pod,svc

#方式1
kubectl create deployment nginx --image=nginx
kubectl expose deployment nginx --port=80 --type=NodePort

#方式2,目前web页面再用这个
kubectl create deployment nginx --image nginx:1.19.0
kubectl expose deployment nginx --port=80 --type=NodePort 
kubectl get pod,svc
#为了让80端口对所有地址开放访问
nohup kubectl port-forward --address 0.0.0.0 pod/nginx-85b45874d9-n8rl2 80:80 &
#这个值 pod/nginx-85b45874d9-n8rl2 是通过kubectl get pod,svc获取的

#方式3：
kubectl run nginx --image nginx:latest
kubectl port-forward --address 0.0.0.0 pod/nginx 80:80
K8S会监听任何地址，不会再拒绝了，所以使用K8S节点的IP地址去访问，访问成功

访问地址：

http://192.168.60.128:80

部署 Dashboard

官方支持说明里面有具体的版本对应关系:点我查看官网。

k8s的1.19版本对应最新dashboard版本是2.0.5，点击我查看官网说明

Imagesv2.0.5命令如下：

Kubernetes Dashboard
kubernetesui/dashboard:v2.0.5

Metrics Scraper
kubernetesui/metrics-scraper:v1.0.6

Installation
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.5/aio/deploy/recommended.yaml

如果要升级版本：使用 kubectl delete -f ***之后,我又重启了所有服务器，才完成升级。

老规矩先导入相关镜像和配置文件，在master节点操作,我这此用的是2.0.0版本做的。

服务器上可能无法执行下载安装，直接用浏览器访问，然后把内容拷贝出来即可
https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta8/aio/deploy/recommended.yaml

https://raw.githubusercontent.com/kubernetes/dashboard/v2.6.1/aio/deploy/recommended.yaml

默认Dashboard只能集群内部访问，修改Service为NodePort类型，暴露到外部：

# 添加映射到虚拟机的端口,k8s只支持30000以上的端口

vim recommended.yaml
kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  ports:
    - port: 443
      targetPort: 8443
      nodePort: 30001
  type: NodePort
  selector:
    k8s-app: kubernetes-dashboard

[root@k8s-master1 dashboardv2.0.0-beta8]# kubectl apply -f recommendedv2.0.0.yaml 
namespace/kubernetes-dashboard created
serviceaccount/kubernetes-dashboard created
service/kubernetes-dashboard created
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created
secret/kubernetes-dashboard-key-holder created
configmap/kubernetes-dashboard-settings created
role.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
deployment.apps/kubernetes-dashboard created
service/dashboard-metrics-scraper created
deployment.apps/dashboard-metrics-scraper created

kubectl get pods,svc -n kubernetes-dashboard
NAME                                             READY   STATUS              RESTARTS   AGE
pod/dashboard-metrics-scraper-694557449d-z8gfb   1/1     Running             0          2m18s
pod/kubernetes-dashboard-9774cc786-q2gsx         1/1     Running		     0          2m19s

NAME                                TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE
service/dashboard-metrics-scraper   ClusterIP   10.1.0.141   <none>        8000/TCP        2m19s
service/kubernetes-dashboard        NodePort    10.1.0.239   <none>        443:30001/TCP   2m19s

访问地址

如果刚开机页面打不开的话，那就是慢，等会就好了。

https://192.168.60.128:30006

token:

eyJhbGciOiJSUzI1NiIsImtpZCI6InN0SlZ2UWM1WVplMHI5emhCanI1VWV5WXB2N0g4TTY5MnFmQ0FTMzVUT00ifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXNoYm9hcmQtYWRtaW4tdG9rZW4taDd3MmQiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGFzaGJvYXJkLWFkbWluIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiODNmMzRmMGItMWM1OS00NDA0LTkxMzEtNTM0NGMwOTgzNGVkIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOmRhc2hib2FyZC1hZG1pbiJ9.uiyleaUVRSpruaF2PHZXrh-CzcDf-L9ZKkx9VVMUpR9NGVQqi3zMEc3Kj3x4r3Zgb1qLY_xiEwrlTnF0NAblgDr3Bg_Othkn603ufFbD8FFTJBzM6IdGZoejvoFW91-5CMurfYg5a62LWq2_MFav6T2do-9WqzT2Yx0KdhDqB8ArcuRHoZMkL1L8XcyUW10XkKpj3x5NoW2lxN_11vHAzH45pOLEU6mbxYVPjGUwWP9U5pPa-I0gmFAkCMwIthraKTTj7JiJRYOi_zzChQ3SJAZgBNoLwxFxyVwBZBJvEDF0aNMGgrPr8U7OptOu3j7T_nTvpibJPLgHB7FxN6osXg

创建用户获取token

创建service account并绑定默认cluster-admin管理员集群角色：

#创建用户
kubectl create serviceaccount dashboard-admin -n kube-system
#用户授权
kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin
#获取用户Token
kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}')

使用输出的token登录Dashboard。

浏览器访问不安全设置

可能会出现以下错误

解决办法，修改浏览器配置

调整键盘为英文输入状态，刷新一下页面，鼠标点击当前页面任意位置，然后依次按键：thisisunsafe按完上面的按键，页面会自动刷新，然后就可以正常访问了。（也没搞懂这是啥操作，反正可以用了）

修改中文页面

我们看到现在 dashboard 的页面都已经被汉化了，当前支持英文、中文简体、中文繁体、日语，根据浏览器的语言自动切换的。如果想要强制设置 dashboard 中显示的语言，需要在 dahsboard 的 Deployment yaml 配置中增加如下配置：

value: english  是英文

这种方式是强制容器显示内容为中文，经过实验有效。

扩容node节点

1、基础系统参数配置修改，及hosts地址添加，参考其它节点配置

其它修改忽略，下面是hosts修改

#节点3操作
hostnamectl set-hostname k8s-node3
cat >> /etc/hosts << EOF
192.168.60.128 k8s-master1
192.168.60.129 k8s-node1
192.168.60.130 k8s-node2
192.168.60.131 k8s-node3
EOF

#其它节点操作加入这个节点
cat >> /etc/hosts << EOF
192.168.60.131 k8s-node3
EOF

2、把之前的其它node上跑的镜像都导入进去

3、开始加入,先在master上生成新的token

默认token有效期为24小时，当过期之后，该token就不可用了。这时就需要重新创建token，可以直接使用命令快捷生成：

#mast节点执行
kubeadm token create --print-join-command

然后在节点操作加入,在执行加入的时候，所有节点都开机了，新节点的网络镜像文件都提前导入完成了，不用执行calicov3.20.yaml 这个文件，

#此操作是在节点操作,这条命令是在上一个命令执行之后，出来的，直接复制即可。
kubeadm join 192.168.60.128:6443 --token r67pfg.rwn4apr50ge4iftx     --discovery-token-ca-cert-hash sha256:b982912e62b435bb1d65f935ee3716670834715f0d2b3c8e28ad81891566e363

验证，加入完成