Kubernetes on CRI-O (CentOS)

container

发布日期: 2020-12-15

Kubernetes on CRI-O (CentOS)

Docker 现在在 Kubernetes 中已弃用。

kubelet 中的 Docker 支持现已弃用，并将在以后的版本中删除。Kubelet 使用一个名为 dockershim的模块，该模块实现了对 Docker 的 CRI 支持，并且在 Kubernetes 社区中看到了维护问题。

So，现在 Kubernetes 推荐使用不同的 Container Runtime Interface 而不是 docker。

为什么选择 CRI-O ?

CRI-O是 Kubernetes CRI（ Container Runtime Interface ）的实现，以启用使用OCI（ Open Container Initiative ）兼容的运行时。

它是将Docker，Moby 或 rkt 用作 Kubernetes 的运行时的轻量级替代方案。

CRI-O是主要由Red Hat员工开发的CRI运行时。实际上，该运行时现在已在 Red Hat OpenShift 中使用。并且，他们不再依赖 Docker。

有趣的是，RHEL 7 也不正式支持 Docker。相反，它们为容器环境提供 Podman，Buildah 和 CRI-O。

Let’s get started

one master node
two worker nodes

Installing CRI-O

(在所有节点上完成)

为OS创建一个环境变量。

$ export OS=CentOS_7

注意：如果您使用的是CentOS 8，则将变量名称命名为CentOS_8

为crio的VERSION创建另一个环境变量。

$ export VERSION=1.17

注意：如果您使用的是CentOS 8，则还可以安装crio版本1.18

为cri-o配置REPO。

$ sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/devel:kubic:libcontainers:stable.repo

$ sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.repo https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:$VERSION/$OS/devel:kubic:libcontainers:stable:cri-o:$VERSION.repo

Install cri-o.

$ sudo yum install cri-o -y

启动并启用cri-o服务。

$ sudo systemctl start cri-o

如果启动cri-o有任何问题，请参阅故障排除部分！

$ sudo systemctl enable cri-o

故障排除

无法启动cri-o服务：

$ sudo systemctl status cri-o

crio.service - Container Runtime Interface for OCI (CRI-O)
   Loaded: loaded (/usr/lib/systemd/system/crio.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2020-12-11 01:42:49 EST; 29s ago
     Docs: https://github.com/cri-o/cri-o
  Process: 1259 ExecStart=/usr/bin/crio $CRIO_CONFIG_OPTIONS $CRIO_RUNTIME_OPTIONS $CRIO_STORAGE_OPTIONS $CRIO_NETWORK_OPTIONS $CRIO_METRICS_OPTIONS (code=exited, status=1/FAILURE)
 Main PID: 1259 (code=exited, status=1/FAILURE)

Dec 11 01:42:49 ip-172-31-89-94.ec2.internal systemd[1]: Starting Container Runtime Interface for OCI (CRI-O)...
Dec 11 01:42:49 ip-172-31-89-94.ec2.internal crio[1259]: time="2020-12-11 01:42:49.409813214-01:00" level=fatal msg="Validating root config: failed to get store to set defaults: kernel does not support overlay fs: overlay: the backing xfs filesystem is formatted without d_type support, which leads to incorrect behavior. Reformat the filesystem with ftype=1 to enable d_type support. Running without d_type is not supported.: driver not supported"
Dec 11 01:42:49 ip-172-31-89-94.ec2.internal systemd[1]: crio.service: main process exited, code=exited, status=1/FAILURE
Dec 11 01:42:49 ip-172-31-89-94.ec2.internal systemd[1]: Failed to start Container Runtime Interface for OCI (CRI-O).
Dec 11 01:42:49 ip-172-31-89-94.ec2.internal systemd[1]: Unit crio.service entered failed state.
Dec 11 01:42:49 ip-172-31-89-94.ec2.internal systemd[1]: crio.service failed.

根本原因：

默认容器存储即 /var/lib/containers/storage 是以 ftype=0, 挂载的，因此d_type被禁用（d_type启用文件名到其inode的映射）。修改后，ftype=1！

$ sudo xfs_info /var/lib/containers/storage | grep ftype

naming   =version 2              bsize=4096   ascii-ci=0 ftype=0

解决方法:

创建一个逻辑卷（LV）。

$ sudo fdisk <disk_name>

$ sudo partprobe <disk_name>

$ sudo vgcreate vgname <partition>

$ sudo lvcreate -l 100%FREE -n lvname vgname

$ sudo lvs

LV     VG     Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
root   centos -wi-ao----  <6.67g
swap   centos -wi-ao---- 820.00m
lvname vgname -wi-a-----  <5.00g

使用XFS文件系统格式化LV

$ sudo mkfs.xfs /dev/mapper/vgname-lvname

meta-data=/dev/mapper/vgname-lvname isize=512    agcount=4, agsize=327424 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=1309696, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

现在ftype设置为1。
Mount LV on /var/lib/containers/storage permanently.
将 LV 永久挂载在/var/lib/containers/storage上。

$ sudo cat /etc/fstab | grep /var/lib/containers/storage

/dev/mapper/vgname-lvname /var/lib/containers/storage xfs defaults 0 0

$ sudo mount -a

$ sudo df -hT | grep /var/lib/containers/storage

/dev/mapper/vgname-lvname xfs       5.0G   33M  5.0G   1% /var/lib/containers/storage

更改runroot路径，并取消哈希 runroot 和 root 在 /etc/crio/crio.conf 中。

$ sudo cat /etc/crio/crio.conf | grep /var/lib/containers/storage

root = "/var/lib/containers/storage"
runroot = "/var/lib/containers/storage"

然后启动并启用cri-o服务。

$ sudo systemctl start cri-o

$ sudo systemctl enable cri-o

前期准备：

（应该在所有节点上完成）

加载 overlay 模块

$ sudo modprobe overlay

加载 br_netfilter 模块.

$ sudo modprobe br_netfilter

启用此模块，以便 iptables，ip6tables 和 arptables 可以过滤桥接的 IPv4/IPv6/ARP 数据包并使防火墙 transparent 。

Set up required sysctl params, these persist across reboots.
设置所需的 sysctl 参数，这些参数将在重新启动后还生效。

$ cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

重新加载 sysctl 。

$ sudo sysctl --system

禁用防火墙。

$ sudo systemctl stop firewalld && sudo systemctl disable firewalld

禁用 SELinux

$ sudo setenforce 0

$ sudo sed -i -e s/SELINUX=enforcing/SELINUX=permissive/g /etc/sysconfig/selinux

关闭 swap .

$ sudo swapoff -a

禁用 swap 在 /etc/fstab 处修改

$ sudo cat /etc/fstab | grep swap

#/dev/mapper/centos-swap                                swap                    swap    defaults        0 0

在 /etc/crio/crio.conf 更改 crio 的 cgroup_manager .

$ sudo cat /etc/crio/crio.conf | grep cgroup_manager

cgroup_manager = "cgroupfs"

配置 Kubernetes repo.

$ sudo cat /etc/yum.repos.d/kube.repo

[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg

部署 Kubernetes 集群

COMMANDS ON MASTER

Install kubeadm package.

$ sudo yum install kubeadm -y

Start and enable kubelet.

$ sudo systemctl start kubelet && sudo systemctl enable kubelet

初始化 Kubernetes

$ sudo kubeadm init

Note: If giving error of iptables does not exist, just load br_netfilter module again and run the command.

注意：如果有iptables does not exist 错误信息，只需再次加载 br_netfilter 模块并运行命令。

Output

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.46.157:6443 --token 7xlnp0.9uv4z0qr4wvzhtqn \
    --discovery-token-ca-cert-hash sha256:4a1a412d2e682556df0bf10dc380c744a98eb99e8c927fa58eb025d5ff7dc694

使用普通用户运行 kunectl ,请运行以下命令

$ mkdir -p $HOME/.kube

$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

$ sudo chown $(id -u):$(id -g) $HOME/.kube/config

这些命令将您的 Kubernetes 配置文件存储在主目录中。
记录 kubeadm init 输出的kubeadm join命令。需要使用此命令将节点加入集群。
安装CNI插件使 CoreDNS Pod 正常运行。

$ kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

Weave Net creates a virtual network that connects Docker containers across multiple hosts and enables their automatic discovery.
Weave Net创建了一个虚拟网络，该网络可以使 Docker 容器跨主机通信。

$ kubectl get nodes

NAME     STATUS   ROLES                  AGE   VERSION
master   Ready    control-plane,master   14m   v1.20.0

Now we need to add worker nodes to the cluster.
现在我们需要将nodes 添加到集群中。

Worker nodes 命令：

Install kubeadm package.

$ sudo yum install kubeadm -y

运行 kubeadm join命令。（在 kubeadm init 输出中的命令）

$ sudo kubeadm join 192.168.46.157:6443 --token 7xlnp0.9uv4z0qr4wvzhtqn \
    --discovery-token-ca-cert-hash sha256:4a1a412d2e682556df0bf10dc380c744a98eb99e8c927fa58eb025d5ff7dc694

Output

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

现在我们的工作节点已加入集群！

Master 命令

检查节点是否准备就绪。

$ kubectl get nodes

NAME      STATUS   ROLES                  AGE     VERSION
master    Ready    control-plane,master   15m     v1.20.0
worker1   Ready    <none>                 4m44s   v1.20.0
worker2   Ready    <none>                 4m18s   v1.20.0

Kubernetes CRI-O