in0va'S Blog JustLearnM0re

犯二

0%

Kubernetes Learning Notes - Installation Process Record

发表于 分类于

一、起源

云将会是未来技术的热点与风口，云安全也将逐渐体现它的重要性，技术栈开始慢慢从主机安全往云原生相关去靠拢，本章记录下k8s的安装流程（虚拟机部署，master和node各一台，master为centos7，node为ubuntu18.04）

二、过程

安装docker：

1
2
3
[root@centos7  ~]# yum install -y docker-ce docker-ce-cli containerd.io 
[root@centos7  ~]# systemctl start docker 
[root@centos7  ~]# systemctl enable docker

安装命令补全：

1
2
[root@centos7  ~]# yum -y install bash-completion 
[root@centos7  /]# source /etc/profile.d/bash_completion.sh

docker加速：

1
2
3
4
5
6
7
8
[root@centos7  ~]# mkdir -p /etc/docker 
[root@centos7  ~]# tee /etc/docker/daemon.json <<-'EOF' 
{
"registry-mirrors": ["https://v16stybc.mirror.aliyuncs.com"]
}
EOF
[root@centos7  ~]# systemctl daemon-reload 
[root@centos7  ~]# systemctl restart docker

修改主机名及关闭swap：

1
2
3
4
[root@centos7  ~]# vim /etc/hosts 
[root@centos7  ~]# hostnamectl set-hostname master 
[root@centos7  ~]# swapoff -a 
[root@master  ~]# sed -i.bak '/swap/s/^/#/' /etc/fstab

关闭防火墙命令：

1
2
3
4
5
6
[root@master  ~]# sysctl net.bridge.bridge-nf-call-iptables=1 
net.bridge.bridge-nf-call-iptables = 1
[root@master  ~]# sysctl net.bridge.bridge-nf-call-ip6tables=1 
net.bridge.bridge-nf-call-ip6tables = 1
[root@master  ~]# iptables --flush 
[root@master  ~]# iptables -tnat --flush

修改CgroupDriver：

1
2
3
4
5
[root@master  ~]# more /etc/docker/daemon.json 
{
"registry-mirrors": ["https://v16stybc.mirror.aliyuncs.com"],
"exec-opts": ["native.cgroupdriver=systemd"]
}

增加kubernetes源（centos）：

1
2
3
4
5
6
7
8
9
10
11
12
[root@master  ~]# cat <  /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
[root@master  ~]# yum clean all 
[root@master  ~]# yum -y makecache 
[root@master  ~]# yum install -y kubelet kubeadm kubectl

设置k8s开机自启动：

1
2
3
[root@master  ~]# systemctl enable kubelet && systemctl start kubelet 
[root@master  ~]# echo "source <(kubectl completion bash)" >> ~/.bash_profile 
[root@master  ~]# source .bash_profile

镜像下载及修改tag：

1
2
3
4
5
6
7
8
9
[root@master  ~]# more image.sh 
#!/bin/bash
url=registry.cn-hangzhou.aliyuncs.com/google_containers
version=v1.19.4
images=(`kubeadm config images list --kubernetes-version=$version|awk -F '/' '{print $2}'`)
for imagename in ${images[@]} ; do
	docker pull $url/$imagename  
  docker tag $url/$imagename k8s.gcr.io/$imagename  
  docker rmi -f $url/$imagenamedone

初始化命令：

1
2
3
kubeadm init --apiserver-advertise-address masterip --pod-network-cidr=10.244.0.0/16
[root@master  ~]# echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile 
[root@master  ~]# source .bash_profile

安装Pod网络：

1
[root@master  ~]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

查看令牌：

1
2
3
[root@master  ~]# kubeadm token list 
TOKEN                     TTL         EXPIRES                     USAGES                   DESCRIPTION                                                EXTRA GROUPS
j5eoyz.zu0x6su7wzh752b3      2019-06-04T17:40:41+08:00   authentication,signing   The default bootstrap token generated by 'kubeadm init'.   system:bootstrappers:kubeadm:default-node-token

生成新的令牌：

1
[root@master  ~]# kubeadm token create 1zl3he.fxgz2pvxa3qkwxln

生成新的加密串：

1
[root@master  ~]# openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

node节点加入集群：

1
[root@node01  ~]# kubeadm join 172.27.9.131:6443 --token 1zl3he.fxgz2pvxa3qkwxln  --discovery-token-ca-cert-hash sha256:5f656ae26b5e7d4641a979cbfdffeb7845cc5962bbfcd1d5435f00a25c02ea50

创建Dashboard面板：

1、在 master 节点执行

本例 k8s 是 v1.17.2，对应的 dashboard 是 v2.0.0-rc5 这个版本，具体去这里查看对应的版本 https://github.com/kubernetes/dashboard/releases

1
wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-rc5/aio/deploy/recommended.yaml

方式一：

2、创建pod，查看，成功创建

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# 创建 pod
kubectl apply -f recommended.yaml
[root@master1  ~]# kubectl get pods --all-namespaces 
NAMESPACE              NAME                                         READY   STATUS    RESTARTS   AGE
default                nginx-5578584966-ch9x4                       1/1     Running   1          8h
kube-system            coredns-9d85f5447-qghnb                      1/1     Running   38         6d13h
kube-system            coredns-9d85f5447-xqsl2                      1/1     Running   37         6d13h
kube-system            etcd-master1                                 1/1     Running   8          6d13h
kube-system            kube-apiserver-master1                       1/1     Running   9          6d13h
kube-system            kube-controller-manager-master1              1/1     Running   8          6d13h
kube-system            kube-flannel-ds-amd64-h2f4w                  1/1     Running   5          6d10h
kube-system            kube-flannel-ds-amd64-z57qk                  1/1     Running   1          10h
kube-system            kube-proxy-4j8pj                             1/1     Running   1          10h
kube-system            kube-proxy-xk7gq                             1/1     Running   7          6d13h
kube-system            kube-scheduler-master1                       1/1     Running   9          6d13h
kubernetes-dashboard   dashboard-metrics-scraper-7b8b58dc8b-5r22j   1/1     Running   0          15m
kubernetes-dashboard   kubernetes-dashboard-866f987876-gv2qw        1/1     Running   0          15m

3、 删除现有的dashboard服务，dashboard 服务的 namespace 是 kubernetes-dashboard，但是该服务的类型是ClusterIP，不便于我们通过浏览器访问，因此需要改成NodePort型的

1
2
3
4
5
6
7
8
9
10
11
# 查看 现有的服务
[root@master1  ~]# kubectl get svc --all-namespaces 
NAMESPACE              NAME                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                  AGE
default                kubernetes                  ClusterIP   10.96.0.1                443/TCP                  6d13h
default                nginx                       NodePort    10.102.220.172           80:31863/TCP             8h
kube-system            kube-dns                    ClusterIP   10.96.0.10               53/UDP,53/TCP,9153/TCP   6d13h
kubernetes-dashboard   dashboard-metrics-scraper   ClusterIP   10.100.246.255           8000/TCP                 61s
kubernetes-dashboard   kubernetes-dashboard        ClusterIP   10.109.210.35            443/TCP                  61s

# 删除
kubectl delete service kubernetes-dashboard --namespace=kubernetes-dashboard

4、创建配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
vim dashboard-svc.yaml
 
# 内容
kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  type: NodePort
  ports:
    - port: 443
      targetPort: 8443
  selector:
    k8s-app: kubernetes-dashboard
 
 
# 执行
kubectl apply -f dashboard-svc.yaml

方式二：

修改recommended.yaml文件，添加一个type,指定端口类型为NodePort，这样外界可以通过地址 nodeIP:nodePort访问dashboard

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
---

kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  ports:
    - port: 443
      targetPort: 8443
      nodePort: 30000
  type: NodePort
  selector:
    k8s-app: kubernetes-dashboard
    
# 执行
kubectl apply -f recommended.yaml

5、再次查看服务，成功

1
2
3
4
5
6
7
[root@master1  ~]# kubectl get svc --all-namespaces 
NAMESPACE              NAME                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                  AGE
default                kubernetes                  ClusterIP   10.96.0.1                443/TCP                  6d13h
default                nginx                       NodePort    10.102.220.172           80:31863/TCP             8h
kube-system            kube-dns                    ClusterIP   10.96.0.10               53/UDP,53/TCP,9153/TCP   6d13h
kubernetes-dashboard   dashboard-metrics-scraper   ClusterIP   10.100.246.255           8000/TCP                 4m32s
kubernetes-dashboard   kubernetes-dashboard        NodePort    10.110.91.255            443:30000/TCP            10s

6、想要访问dashboard服务，就要有访问权限，创建kubernetes-dashboard管理员角色

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
vim dashboard-svc-account.yaml

# 结果
apiVersion: v1
kind: ServiceAccount
metadata:
  name: dashboard-admin
  namespace: kube-system
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: dashboard-admin
subjects:
  - kind: ServiceAccount
    name: dashboard-admin
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io


# 执行
kubectl apply -f dashboard-svc-account.yaml

7、获取 token

1
2
3
4
5
6
7
[root@master1 ~]# kubectl get secret -n kube-system |grep admin|awk '{print $1}'
dashboard-admin-token-bwgjv
 
 
# 复制下面的 token,后面登陆的时候要用到
[root@master1 ~]# kubectl describe secret dashboard-admin-token-bwgjv -n kube-system|grep '^token'|awk '{print $2}'
eyJhbGciOiJSUzI1NiIsImtpZCI6IkJOVUhyRElPQzJzU2t6VDNVdWpTdzhNZmZPZjV0U2s1UXBFTzctNE9uOFEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXNoYm9hcmQtYWRtaW4tdG9rZW4tYndnanYiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGFzaGJvYXJkLWFkbWluIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiOTE5NGY5YWYtZDZlNC00ZDFmLTg4OWEtMDY4ODIyMDFlOGNmIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOmRhc2hib2FyZC1hZG1pbiJ9.kEK3XvUXJGzQlBI4LIOp-puYzBBhhXSkD20vFp9ET-rGErxmMHjUuCqWxg0iawbuOndMARrpeGJKNTlD2vL81bXMaPpKb4Y2qoB6bH5ETQPUU0HPpWYmfoHl4krEXy7S95h0mWehiHLcFkrUhyKGa39cEBq0B0HRo49tjM5QzkE6PNJ5nmEYHIJMb4U62E8wKeqY9vt60AlRa_Re7IDAO9qfb5_dGEmUaIdr3tu22sa3POBsm2bhr-R3aC8vQzNuafM35s3ed8KofOTQFk8fXu4p7lquJnji4yfC77yS3yo5Jo3VPyHi3p5np_9AuSNYfI8fo1EpSeMsXOBH45hu2w

8、访问https://192.168.2.102:30000（30000是kubectl get svc –all-namespaces拿到的port）把上面的 token粘贴到web上的令牌项就可以访问了

image

image-20210110140929023

三、坑

1、访问dashboard各种报forbidden禁止访问：

https://my.oschina.net/xxbAndy/blog/3103532

2、node节点无法成功加入到集群中，kubeadm reset后join显示端口占用：

1）kubeadm reset + kubeadm join

2）reset后join若还是显示端口占用则重启虚机再试一次

3、kubelet因无法创建/sys/fs/cgroup/pids路径启动失败（未解决）

https://www.gitmemory.com/issue/easzlab/kubeasz/575/497541914

4、pd虚拟机中止后重新开启访问dashboard失败：

删除dashboard的pod重新创建就好了（kubectl delete & apply recommended.yaml）

其它：https://www.jianshu.com/p/8e78e0abddf9

四、参考链接

https://zhuanlan.zhihu.com/p/31398416

https://zhuanlan.zhihu.com/p/92923128

https://segmentfault.com/a/1190000038420933

https://www.jianshu.com/p/a3d8028002f3

http://docs.kubernetes.org.cn/251.html

https://du2016.gitbooks.io/k8s-learning-notes/content/architecture/

https://edu.aliyun.com/course/1651/lesson/list?spm=5176.8764728.aliyun-edu-course-tab.2.408020beuUyCp3&previewAs=guest

https://v1-16.docs.kubernetes.io/docs/tasks/debug-application-cluster/falco/