k8s学习

k8s学习

  • 2020-03-18
  • 次

K8S

资料参考

集群的条件

  • 集群中所有机器之间网络互通
  • 可以访问外网，需要拉取镜像
  • 禁止swap分区

Mac 本地安装虚拟机

  • 配置: 2C4G 50G
  • 修改网络配置:
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    >cd /etc/sysconfig/network-scripts/

    > vi ifcfg-eno-33

    设置 :

    ONBOOT=yes 

    > service network restart

    > ping baidu.com   # 测试虚拟机是否可以进行连外网

    > ip addr  #  查看 ip 地址

环境的准备

  • 关闭防火墙：
1
2
3
4

>  systemctl stop firewalld

>  systemctl disable firewalld

  • 关闭selinux：

    1
    2
    3
    4

    >  sed -i 's/enforcing/disabled/' /etc/selinux/config 

    >  setenforce 0

  • 关闭swap：

    1
    2
    3
    4
    5
    > swapoff -a


    > vi /etc/fstab  
    把 swap 那一行进行注释掉  ： # /dev/mapper/centos-swap swap swap default 0
1
2
3
4
5
6

+ 添加主机名与IP对应关系（记得设置主机名）：
$ cat /etc/hosts
192.168.31.60 k8s-master
192.168.31.61 k8s-node1
192.168.31.62 k8s-node2
  • 将桥接的IPv4流量传递到iptables的链：
    1
    2
    3
    4
    5
    6
    7

    > cat > /etc/sysctl.d/k8s.conf << EOF
    net.bridge.bridge-nf-call-ip6tables = 1
    net.bridge.bridge-nf-call-iptables = 1
    EOF
      
    > sysctl --system
  • 时间同步 : yum install -y ntpdate ntpdate ntp.api.bz

安装的 docker : 使用阿里云镜像库来进行安装

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

# 下载工具的安装
> yum install wget

>  wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo

>  yum -y install docker-ce-18.06.1.ce-3.el7

>  systemctl enable docker && systemctl start docker

# 更改cgroup driver为systemd,并且为docker添加镜像加速

cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "registry-mirrors": ["https://c7lrovdh.mirror.aliyuncs.com"]
}
EOF
sudo systemctl daemon-reload
sudo systemctl restart docker


> docker version  #  校验 docker 版本

添加阿里云YUM软件源

1
2
3
4
5
6
7
8
9
10
> 
cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

安装kubeadm，kubelet和kubectl

  • 指定 1.15版本
    1
    2
    3
    > yum install -y kubelet-1.15.0 kubeadm-1.15.0 kubectl-1.15.0

    > systemctl enable kubelet

部署 k8s master

  • master node : 172.16.182.130
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    >
    kubeadm init \
      --apiserver-advertise-address=172.16.182.137 \
      --image-repository registry.aliyuncs.com/google_containers \
      --kubernetes-version v1.15.0 \
      --service-cidr=10.1.0.0/16 \
      --pod-network-cidr=10.244.0.0/16
      


    成功安装之后:

    Your Kubernetes control-plane has initialized successfully!
        
    To start using your cluster, you need to run the following as a regular user:

      mkdir -p $HOME/.kube
      sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
      sudo chown $(id -u):$(id -g) $HOME/.kube/config

    You should now deploy a pod network to the cluster.
    Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
      https://kubernetes.io/docs/concepts/cluster-administration/addons/

    Then you can join any number of worker nodes by running the following on each as root:

    kubeadm join 172.16.182.134:6443 --token 7p4yoc.np5kdlji6mtflldc \
        --discovery-token-ca-cert-hash sha256:a1f6d5775eaed1f13a1eb654416b0971b9b9d5e112138d2793e4f3ffd5c4d24c

使用kubectl工具

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

> mkdir -p $HOME/.kube

> sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

> sudo chown $(id -u):$(id -g) $HOME/.kube/config


# 查看节点内容进行测试 kubectl 是否安装成功

> kubectl get nodes 

# 显示内容  成功安装

NAME                    STATUS     ROLES    AGE     VERSION
localhost.localdomain   NotReady   master   4m26s   v1.15.0

安装Pod网络插件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

> kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/a70459be0084506e4ec919aa1c114638878db11b/Documentation/kube-flannel.yml

执行的会报错.被墙了

> wget https://raw.githubusercontent.com/coreos/flannel/a70459be0084506e4ec919aa1c114638878db11b/Documentation/kube-flannel.yml

# 本地就有相应的 kube-flannel.yml 文件, ls 查看

需要修改配置文件的 image 的地址:

quay.io/coreos/flannel:v0.11.0-amd64    >    lizhenliang/flannel:v0.11.0-amd64


scp ~/Desktop/kube-flannel.yml root@172.16.182.137:/root/kube-flannel.yml
scp ~/Desktop/kube-flannel.yml root@172.16.182.136:/root/kube-flannel.yml


> docker pull lizhenliang/flannel:v0.11.0-amd64

> kubectl apply -f ./kube-flannel.yml



还有一种方法:

kubectl apply -f  https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

整个安装下来就有相应的 k8s 安装包了，然后就可以进行虚拟机的复制操作。让其他的 node 都有相应的安装包，一样的环境

  • 关机操作

    1
    > shutdown -h now ：立即关机

  • 克隆镜像

    1
    2
    3
    4
    先关机操作 >  进行设置克隆  >  copy 出 k8s master node  2个节点:
    ip 地址:
    k8s-node-1 :  172.16.182.131
    k8s-node-2 :  172.16.182.132

  • 开机重新的初始化 k8s

  • 本地已经存在 kubeadm 服务了，需要重启 : kubeadm reset

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    >
    kubeadm init \
      --apiserver-advertise-address=172.16.182.134 \
      --image-repository registry.aliyuncs.com/google_containers \
      --kubernetes-version v1.15.0 \
      --service-cidr=10.1.0.0/16 \
      --pod-network-cidr=10.244.0.0/16 \
      --ignore-preflight-errors=all
      
    再执行:
      mkdir -p $HOME/.kube
      sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
      sudo chown $(id -u):$(id -g) $HOME/.kube/config
      
    执行的结果:

    kubeadm join 172.16.182.134:6443 --token vg18ba.q937acgy9ifamuog \
        --discovery-token-ca-cert-hash sha256:356e42541d06ac75084508e3db76a1f7902fa6cb88e5dfa139e1eeb9bc1674a7


    kubeadm join 172.16.182.130:6443 --token 52szaz.t3mr0nmth4e4trmb \
        --discovery-token-ca-cert-hash sha256:4341cb5fec285e14780ade60fa1935997fbce44a91baef05c50a5959896aca31 \
        --ignore-preflight-errors=all

还是有问题

  • 执行 kubectl get nodes

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    The connection to the server localhost:8080 was refused - did you specify the right host or port?


    将 master 的 admin 文件 copy 到 node 节点上:

    >  scp /etc/kubernetes/admin.conf root@172.16.182.131:/etc/kubernetes/admin.conf
    >  scp /etc/kubernetes/admin.conf root@172.16.182.132:/etc/kubernetes/admin.conf
    >  scp /etc/kubernetes/admin.conf root@172.16.182.133:/etc/kubernetes/admin.conf


    mkdir -p $HOME/.kube
    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    sudo chown $(id -u):$(id -g) $HOME/.kube/config

    在 node 节点:
    echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile

    source ~/.bash_profile

  • 在从节点会报 master 的节点的错误，因为我 master 执行过 kubeadm init 指令导致镜像还有证书信息.故需要执行: kubeadm reset 还有把 swap 关掉 swapoff -a

1
2
3
4
5
6
7
8
9

删除文件: rm -rf $HOME/.kube

重启服务: kubeadm reset

关掉swap : swapoff -a


iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X
  • 从节点就可以执行 kubeadm join 指令了,查看

添加的 node 节点的 host

172.16.182.134 k8s-master
172.16.182.135 k8s-node1
172.16.182.136 k8s-node2
172.16.182.137 k8s-node3

172.16.182.137 k8s-master

172.16.182.130 k8s-master
172.16.182.131 k8s-node1
172.16.182.132 k8s-node2
172.16.182.133 k8s-node3
172.16.182.134 k8s-node4
172.16.182.135 k8s-node5

创建 pod 节点内容

  • docker pull hub.c.163.com/public/nginx:1.2.1

  • kubectl create deployment nginx-test –image=hub.c.163.com/public/nginx:1.2.1

  • kubectl expose deployment nginx-test –port=80 –type=NodePort

  • kubectl get pod,

  • kubectl create deployment nginx-k –image=hub.c.163.com/public/nginx:1.2.1

  • kubectl expose deployment nginx-k –port=81 –type=NodePort

  • kubectl get pod,svc

1
zg8cgy.lljuzh1uyoktnrad

发现 pod 运行的时候报错

[root@localhost kubernetes]# kubectl get pod –namespace=kube-system
NAME READY STATUS RESTARTS AGE
coredns-bccdc95cf-4rwkm 0/1 ContainerCreating 0 13h
coredns-bccdc95cf-pzlhl 0/1 ContainerCreating 0 13h
kube-flannel-ds-amd64-8fljt 0/1 Error 2 13h
kube-flannel-ds-amd64-zfkst 0/1 CrashLoopBackOff 4 60m
kube-proxy-gpsd7 1/1 Running 0 13h
kube-proxy-lml9w 1/1 Running 0 60m

  • 查看的 k8s的全部服务的运行的情况 : kubectl get pods –all-namespaces
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    [root@localhost ~]# kubectl get pods --all-namespaces
    NAMESPACE     NAME                                            READY   STATUS     RESTARTS   AGE
    default       nginx-test-79f9c95576-vdm2d                     0/1     Pending    0          4m49s
    kube-system   coredns-bccdc95cf-dk9qn                         0/1     Pending    0          15m
    kube-system   coredns-bccdc95cf-wdmf6                         0/1     Pending    0          15m
    kube-system   etcd-localhost.localdomain                      1/1     Running    0          14m
    kube-system   kube-apiserver-localhost.localdomain            1/1     Running    0          14m
    kube-system   kube-controller-manager-localhost.localdomain   1/1     Running    0          14m
    kube-system   kube-flannel-ds-amd64-blfvw                     0/1     Init:0/1   0          10m
    kube-system   kube-proxy-n7n86                                1/1     Running    0          15m
    kube-system   kube-scheduler-localhost.localdomain            1/1     Running    0          14m
  • 部署的 pod 一直在 pending 中:
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    [root@localhost ~]# kubectl get pod
    NAME                          READY   STATUS    RESTARTS   AGE
    nginx-test-79f9c95576-vdm2d   0/1     Pending   0          6m50s

    # 查看原因详情
    [root@localhost ~]# kubectl describe pods nginx-k-64bf97c5c7-9s7wm
    Name:           nginx-test-79f9c95576-vdm2d
    Namespace:      default
    Priority:       0
    Node:           <none>
    Labels:         app=nginx-test
                    pod-template-hash=79f9c95576
    Annotations:    <none>
    Status:         Pending
    IP:
    Controlled By:  ReplicaSet/nginx-test-79f9c95576
    Containers:
      nginx:
        Image:        hub.c.163.com/public/nginx:1.2.1
        Port:         <none>
        Host Port:    <none>
        Environment:  <none>
        Mounts:
          /var/run/secrets/kubernetes.io/serviceaccount from default-token-6g6b8 (ro)
    Conditions:
      Type           Status
      PodScheduled   False
    Volumes:
      default-token-6g6b8:
        Type:        Secret (a volume populated by a Secret)
        SecretName:  default-token-6g6b8
        Optional:    false
    QoS Class:       BestEffort
    Node-Selectors:  <none>
    Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                     node.kubernetes.io/unreachable:NoExecute for 300s
    Events:
      Type     Reason            Age                 From               Message
      ----     ------            ----                ----               -------
      Warning  FailedScheduling  5s (x9 over 9m13s)  default-scheduler  0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.
      
      
      
    + 允许 master 节点可以运行:

    NAME                    STATUS     ROLES    AGE   VERSION
    localhost.localdomain   NotReady   master   36m   v1.15.0


    ## 主节点也可以进行工作而不是负载   没有解决问题
    kubectl taint nodes --all node-role.kubernetes.io/master-


    ## 查看日志内容
    journalctl -f -u kubelet.service



    错误日志内容:

    3月 19 16:07:11 k8s-master kubelet[4132]: E0319 16:07:11.163577    4132 kuberuntime_manager.go:688] createPodSandbox for pod "coredns-bccdc95cf-wdmf6_kube-system(e9f67903-ef2c-4ee3-b74b-08390a50ca85)" failed: rpc error: code = Unknown desc = failed to set up sandbox container "2e5707bd73138ed55e6561f650b1fadc52c1fc1f10a16829672306081f2406c7" network for pod "coredns-bccdc95cf-wdmf6": NetworkPlugin cni failed to set up pod "coredns-bccdc95cf-wdmf6_kube-system" network: open /run/flannel/subnet.env: no such file or directory
    3月 19 16:07:11 k8s-master kubelet[4132]: E0319 16:07:11.163627    4132 pod_workers.go:190] Error syncing pod e9f67903-ef2c-4ee3-b74b-08390a50ca85 ("coredns-bccdc95cf-wdmf6_kube-system(e9f67903-ef2c-4ee3-b74b-08390a50ca85)"), skipping: failed to "CreatePodSandbox" for "coredns-bccdc95cf-wdmf6_kube-system(e9f67903-ef2c-4ee3-b74b-08390a50ca85)" with CreatePodSandboxError: "CreatePodSandbox for pod \"coredns-bccdc95cf-wdmf6_kube-system(e9f67903-ef2c-4ee3-b74b-08390a50ca85)\" failed: rpc error: code = Unknown desc = failed to set up sandbox container \"2e5707bd73138ed55e6561f650b1fadc52c1fc1f10a16829672306081f2406c7\" network for pod \"coredns-bccdc95cf-wdmf6\": NetworkPlugin cni failed to set up pod \"coredns-bccdc95cf-wdmf6_kube-system\" network: open /run/flannel/subnet.env: no such file or directory"
    3月 19 16:07:11 k8s-master kubelet[4132]: E0319 16:07:11.240120    4132 remote_runtime.go:105] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to set up sandbox container "767300ea7c8db979280b1a0addea975c05654cdb041eb1f8c2c476cd70fede60" network for pod "nginx-k-64bf97c5c7-9s7wm": NetworkPlugin cni failed to set up pod "nginx-k-64bf97c5c7-9s7wm_default" network: open /run/flannel/subnet.env: no such file or directory
    3月 19 16:07:11 k8s-master kubelet[4132]: E0319 16:07:11.240154    4132 kuberuntime_sandbox.go:68] CreatePodSandbox for pod "nginx-k-64bf97c5c7-9s7wm_default(453a9e1c-2a96-4c24-a7d9-2ea8895c1637)" failed: rpc error: code = Unknown desc = failed to set up sandbox container "767300ea7c8db979280b1a0addea975c05654cdb041eb1f8c2c476cd70fede60" network for pod "nginx-k-64bf97c5c7-9s7wm": NetworkPlugin cni failed to set up pod "nginx-k-64bf97c5c7-9s7wm_default" network: open /run/flannel/subnet.env: no such file or directory
    3月 19 16:07:11 k8s-master kubelet[4132]: E0319 16:07:11.240172    4132 kuberuntime_manager.go:688] createPodSandbox for pod "nginx-k-64bf97c5c7-9s7wm_default(453a9e1c-2a96-4c24-a7d9-2ea8895c1637)" failed: rpc error: code = Unknown desc = failed to set up sandbox container "767300ea7c8db979280b1a0addea975c05654cdb041eb1f8c2c476cd70fede60" network for pod "nginx-k-64bf97c5c7-9s7wm": NetworkPlugin cni failed to set up pod "nginx-k-64bf97c5c7-9s7wm_default" network: open /run/flannel/subnet.env: no such file or directory
    3月 19 16:07:11 k8s-master kubelet[4132]: E0319 16:07:11.240207    4132 pod_workers.go:190] Error syncing pod 453a9e1c-2a96-4c24-a7d9-2ea8895c1637 ("nginx-k-64bf97c5c7-9s7wm_default(453a9e1c-2a96-4c24-a7d9-2ea8895c1637)"), skipping: failed to "CreatePodSandbox" for "nginx-k-64bf97c5c7-9s7wm_default(453a9e1c-2a96-4c24-a7d9-2ea8895c1637)" with CreatePodSandboxError: "CreatePodSandbox for pod \"nginx-k-64bf97c5c7-9s7wm_default(453a9e1c-2a96-4c24-a7d9-2ea8895c1637)\" failed: rpc error: code = Unknown desc = failed to set up sandbox container \"767300ea7c8db979280b1a0addea975c05654cdb041eb1f8c2c476cd70fede60\" network for pod \"nginx-k-64bf97c5c7-9s7wm\": NetworkPlugin cni failed to set up pod \"nginx-k-64bf97c5c7-9s7wm_default\" network: open /run/flannel/subnet.env: no such file or directory"
    3月 19 16:07:11 k8s-master kubelet[4132]: E0319 16:07:11.244832    4132 cni.go:331] Error adding kube-system_coredns-bccdc95cf-dk9qn/a3c254c236a377a21243c739164b3678ad43591095fc2bdd80511b54bb3ef735 to network flannel/cbr0: open /run/flannel/subnet.env: no such file or directory
    3月 19 16:07:11 k8s-master kubelet[4132]: E0319 16:07:11.334985    4132 remote_runtime.go:105] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to set up sandbox container "62d3f366a81cb3b7a9ea01311ba45c64723c92e38b1eb450387db5790f5fb89f" network for pod "nginx-test-79f9c95576-vdm2d": NetworkPlugin cni failed to set up pod "nginx-test-79f9c95576-vdm2d_default" network: open /run/flannel/subnet.env: no such file or directory
    3月 19 16:07:11 k8s-master kubelet[4132]: E0319 16:07:11.335069    4132 kuberuntime_sandbox.go:68] CreatePodSandbox for pod "nginx-test-79f9c95576-vdm2d_default(acd00deb-5e92-440f-834e-398348066f7a)" failed: rpc error: code = Unknown desc = failed to set up sandbox container "62d3f366a81cb3b7a9ea01311ba45c64723c92e38b1eb450387db5790f5fb89f" network for pod "nginx-test-79f9c95576-vdm2d": NetworkPlugin cni failed to set up pod "nginx-test-79f9c95576-vdm2d_default" network: open /run/flannel/subnet.env: no such file or directory
    3月 19 16:07:11 k8s-master kubelet[4132]: E0319 16:07:11.335090    4132 kuberuntime_manager.go:688] createPodSandbox for pod "nginx-test-79f9c95576-vdm2d_default(acd00deb-5e92-440f-834e-398348066f7a)" failed: rpc error: code = Unknown desc = failed to set up sandbox container "62d3f366a81cb3b7a9ea01311ba45c64723c92e38b1eb450387db5790f5fb89f" network for pod "nginx-test-79f9c95576-vdm2d": NetworkPlugin cni failed to set up pod "nginx-test-79f9c95576-vdm2d_default" network: open /run/flannel/subnet.env: no such file or directory
    3月 19 16:07:11 k8s-master kubelet[4132]: E0319 16:07:11.335140    4132 pod_workers.go:190] Error syncing pod acd00deb-5e92-440f-834e-398348066f7a ("nginx-test-79f9c95576-vdm2d_default(acd00deb-5e92-440f-834e-398348066f7a)"), skipping: failed to "CreatePodSandbox" for "nginx-test-79f9c95576-vdm2d_default(acd00deb-5e92-440f-834e-398348066f7a)" with CreatePodSandboxError: "CreatePodSandbox for pod \"nginx-test-79f9c95576-vdm2d_default(acd00deb-5e92-440f-834e-398348066f7a)\" failed: rpc error: code = Unknown desc = failed to set up sandbox container \"62d3f366a81cb3b7a9ea01311ba45c64723c92e38b1eb450387db5790f5fb89f\" network for pod \"nginx-test-79f9c95576-vdm2d\": NetworkPlugin cni failed to set up pod \"nginx-test-79f9c95576-vdm2d_default\" network: open /run/flannel/subnet.env: no such file or directory"
    3月 19 16:07:11 k8s-master kubelet[4132]: E0319 16:07:11.391284    4132 remote_runtime.go:105] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to set up sandbox container "a3c254c236a377a21243c739164b3678ad43591095fc2bdd80511b54bb3ef735" network for pod "coredns-bccdc95cf-dk9qn": NetworkPlugin cni failed to set up pod "coredns-bccdc95cf-dk9qn_kube-system" network: open /run/flannel/subnet.env: no such file or directory
    3月 19 16:07:11 k8s-master kubelet[4132]: E0319 16:07:11.391328    4132 kuberuntime_sandbox.go:68] CreatePodSandbox for pod "coredns-bccdc95cf-dk9qn_kube-system(533e9a3e-ae7f-4bff-a85e-8dc2b884462e)" failed: rpc error: code = Unknown desc = failed to set up sandbox container "a3c254c236a377a21243c739164b3678ad43591095fc2bdd80511b54bb3ef735" network for pod "coredns-bccdc95cf-dk9qn": NetworkPlugin cni failed to set up pod "coredns-bccdc95cf-dk9qn_kube-system" network: open /run/flannel/subnet.env: no such file or directory
    3月 19 16:07:11 k8s-master kubelet[4132]: E0319 16:07:11.391346    4132 kuberuntime_manager.go:688] createPodSandbox for pod "coredns-bccdc95cf-dk9qn_kube-system(533e9a3e-ae7f-4bff-a85e-8dc2b884462e)" failed: rpc error: code = Unknown desc = failed to set up sandbox container "a3c254c236a377a21243c739164b3678ad43591095fc2bdd80511b54bb3ef735" network for pod "coredns-bccdc95cf-dk9qn": NetworkPlugin cni failed to set up pod "coredns-bccdc95cf-dk9qn_kube-system" network: open /run/flannel/subnet.env: no such file or directory
    3月 19 16:07:11 k8s-master kubelet[4132]: E0319 16:07:11.391392    4132 pod_workers.go:190] Error syncing pod 533e9a3e-ae7f-4bff-a85e-8dc2b884462e ("coredns-bccdc95cf-dk9qn_kube-system(533e9a3e-ae7f-4bff-a85e-8dc2b884462e)"), skipping: failed to "CreatePodSandbox" for "coredns-bccdc95cf-dk9qn_kube-system(533e9a3e-ae7f-4bff-a85e-8dc2b884462e)" with CreatePodSandboxError: "CreatePodSandbox for pod \"coredns-bccdc95cf-dk9qn_kube-system(533e9a3e-ae7f-4bff-a85e-8dc2b884462e)\" failed: rpc error: code = Unknown desc = failed to set up sandbox container \"a3c254c236a377a21243c739164b3678ad43591095fc2bdd80511b54bb3ef735\" network for pod \"coredns-bccdc95cf-dk9qn\": NetworkPlugin cni failed to set up pod \"coredns-bccdc95cf-dk9qn_kube-system\" network: open /run/flannel/subnet.env: no such file or directory"

37机器的再次尝试

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167

iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X

rm -rf $HOME/.kube

kubeadm reset && \
kubeadm init \
  --apiserver-advertise-address=172.16.182.137 \
  --image-repository registry.aliyuncs.com/google_containers \
  --kubernetes-version v1.15.0 \
  --service-cidr=10.1.0.0/16 \
  --pod-network-cidr=10.244.0.0/16 \
  --ignore-preflight-errors=all
  
  
  
  

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:


kubeadm join 172.16.182.137:6443 --token 01ew9e.ejtqp3kd9gn41mq8 \
    --discovery-token-ca-cert-hash sha256:0e1d4e69f388bcfd7c8104baa213fd1e1cc4cb38f41a4211d26b655a8990fb83
    
    
    
export KUBECONFIG=/etc/kubernetes/admin.conf
  
scp flannel 文件

kubectl apply -f ./kube-flannel.yml





[root@common ~]# kubectl get pods --all-namespaces
NAMESPACE     NAME                             READY   STATUS                  RESTARTS   AGE
kube-system   coredns-bccdc95cf-s2jw7          0/1     Pending                 0          9m16s
kube-system   coredns-bccdc95cf-xqksb          0/1     Pending                 0          9m16s
kube-system   etcd-common                      1/1     Running                 0          9m34s
kube-system   kube-apiserver-common            1/1     Running                 0          9m34s
kube-system   kube-controller-manager-common   1/1     Running                 0          9m35s
kube-system   kube-flannel-ds-amd64-pjmcf      0/1     Init:ImagePullBackOff   0          6m48s
kube-system   kube-proxy-8zvkx                 1/1     Running                 0          9m17s
kube-system   kube-scheduler-common            1/1     Running                 0          9m34s

查看错误信息:

kubectl -n kube-system logs kube-flannel-ds-amd64-pjmcf



[root@common ~]# kubectl -n kube-system logs kube-flannel-ds-amd64-pjmcf
Error from server (BadRequest): container "kube-flannel" in pod "kube-flannel-ds-amd64-pjmcf" is waiting to start: PodInitializing



kubectl describe -n kube-system pod/coredns-bccdc95cf-s2jw7





kubectl taint nodes --all node-role.kubernetes.io/master-





kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/k8s-manifests/kube-flannel-legacy.yml





Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 172.16.182.137:6443 --token 3a45vi.tnn6ssxhrt92f3zn \
    --discovery-token-ca-cert-hash sha256:72d88308670db12e00ff425698ebcdb3ce49810b18cc7e906fcb845cae543072
    


+ docker pull hub.c.163.com/public/nginx:1.2.1

+ kubectl create deployment nginx-test --image=hub.c.163.com/public/nginx:1.2.1

+ kubectl expose deployment nginx-test --port=80 --type=NodePort

+ kubectl get pod,svc





删除某个 pod:

kubectl delete pod kube-flannel-ds-amd64-4gb49 -n kube-system

kubectl delete pod coredns-5df4c9fcf6-thd9m -n kube-system
kubectl delete pod coredns-bccdc95cf-mcrst7 -n kube-system
kubectl delete pod coredns-bccdc95cf-xqksb -n kube-system




kubectl get pods --all-namespaces


default       nginx-test-79f9c95576-6f4zz      0/1     Pending            0          18m
kube-system   coredns-5df4c9fcf6-thd9m         0/1     Pending            0          43m
kube-system   coredns-bccdc95cf-mcrst          0/1     Pending            0          21m
kube-system   coredns-bccdc95cf-xqksb          0/1     Pending            0          66m
kube-system   etcd-common                      1/1     Running            0          66m
kube-system   kube-apiserver-common            1/1     Running            0          66m
kube-system   kube-controller-manager-common   1/1     Running            0          22m
kube-system   kube-flannel-ds-amd64-pjmcf      0/1     CrashLoopBackOff   8          64m
kube-system   kube-proxy-8zvkx                 1/1     Running            0          66m
kube-system   kube-scheduler-common            1/1     Running            0          66m




查看日志内容
kubectl logs kube-flannel-ds-amd64-4gb49 -n kube-system


I0319 10:36:28.198574       1 main.go:514] Determining IP address of default interface
I0319 10:36:28.199846       1 main.go:527] Using interface with name ens33 and address 172.16.182.137
I0319 10:36:28.199863       1 main.go:544] Defaulting external address to interface address (172.16.182.137)





docker ps | grep flanneld


docker logs e79ca7012968


I0319 10:40:05.096184       1 main.go:514] Determining IP address of default interface
I0319 10:40:05.097347       1 main.go:527] Using interface with name ens33 and address 172.16.182.137
I0319 10:40:05.097444       1 main.go:544] Defaulting external address to interface address (172.16.182.137)
E0319 10:40:35.301107       1 main.go:241] Failed to create SubnetManager: error retrieving pod spec for 'kube-system/kube-flannel-ds-amd64-rmxz2': Get https://10.1.0.1:443/api/v1/namespaces/kube-system/pods/kube-flannel-ds-amd64-rmxz2: dial tcp 10.1.0.1:443: i/o timeout

再来一次的尝试操作

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X

rm -rf $HOME/.kube

kubeadm reset 

kubeadm init \
  --apiserver-advertise-address=172.16.182.137 \
  --image-repository registry.aliyuncs.com/google_containers \
  --kubernetes-version v1.15.0 \
  --service-cidr=10.1.0.0/16 \
  --pod-network-cidr=10.244.0.0/16 \
  --ignore-preflight-errors=all
  
  
## 主要是添加flannel网络



To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 172.16.182.137:6443 --token 5ckarf.hnhi2cgm2zi7w9aa \
    --discovery-token-ca-cert-hash sha256:f50feb41918e489c6b75b9dabc49d2039f870f4a995a478d9928860b5d2313f8

kubectl apply -f  https://raw.githubusercontent.com/coreos/flannel/a70459be0084506e4ec919aa1c114638878db11b/Documentation/kube-flannel.yml
 
 
 
 
 
查看详情:
kubectl describe pods kube-flannel-ds-amd64-rmxz2 -n kube-system


kubectl delete pod kube-flannel-ds-amd64-rmxz29 -n kube-system
感谢您的阅读，本文由 左之右 版权所有。如若转载，请注明出处：左之右（https://zuoyoulai.github.io/2020/03/18/k8s%E5%AD%A6%E4%B9%A0/）
mac操作虚拟机
k8s-kubeadm1.7安装