Kubernetes环境搭建

目录

服务器说明系统设置主机名安装依赖包关闭防火墙、swap，重置iptables系统参数设置 软件安装、系统部署安装docker（所有节点）安装必要工具（所有节点）部署主节点部署flannel网络部署node节点 集群状态检测创建nginx ds检查各种ip连通性检查dns可用性 部署Dashboard 去年通过本地搭建虚拟机的方式实现了k8s环境的搭建和应用部署实践，但一直没有将部署方式记录下来。这次在公司的服务器上搭建了k8s集群，顺便将部署步骤进行记录 此次部署的k8s版本为1.14.0，非高可用版本，高可用部署有待研究。

服务器说明

系统类型IP地址节点角色CPUMemoryHostnamecentos-7.6xxx.xxx.xxx.xxxmaster>=2核>=2Gm1centos-7.6xxx.xxx.xxx.xxxworker>=2核>=2Gw1centos-7.6xxx.xxx.xxx.xxxworker>=2核>=2Gw2

系统设置

所有节点全部操作

主机名

主机名必须每个节点都不一样，并且保证所有点之间可以通过hostname互相访问。

# 查看主机名 $ hostname # 修改主机名 $ hostnamectl set-hostname <your_hostname> # 配置host，使主节点之间可以通过hostname互相访问 $ vi /etc/hosts # <node-ip> <node-hostname>

安装依赖包

# 更新yum $ yum update # 安装依赖包 $ yum install -y conntrack ipvsadm ipset jq sysstat curl iptables libseccomp

关闭防火墙、swap，重置iptables

# 关闭防火墙 $ systemctl stop firewalld && systemctl disable firewalld # 重置iptables $ iptables -F && iptables -X && iptables -F -t nat && iptables -X -t nat && iptables -P FORWARD ACCEPT # 关闭swap $ swapoff -a $ sed -i '/swap/s/^\(.*\)$/#\1/g' /etc/fstab # 关闭selinux $ setenforce 0 # 关闭dnsmasq(否则可能导致docker容器无法解析域名) $ service dnsmasq stop && systemctl disable dnsmasq

系统参数设置

# 制作配置文件 $ cat > /etc/sysctl.d/kubernetes.conf <<EOF net.bridge.bridge-nf-call-iptables=1 net.bridge.bridge-nf-call-ip6tables=1 net.ipv4.ip_forward=1 vm.swappiness=0 vm.overcommit_memory=1 vm.panic_on_oom=0 fs.inotify.max_user_watches=89100 EOF # 生效文件 $ sysctl -p /etc/sysctl.d/kubernetes.conf

软件安装、系统部署

安装docker（所有节点）

yum install -y yum-utils yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo yum list docker-ce --showduplicates | sort -r yum install docker-ce-18.06.0.ce-3.el7 docker-ce-cli-18.06.0.ce-3.el7 containerd.io systemctl start docker

安装必要工具（所有节点）

kubeadm: 部署集群用的命令kubelet: 在集群中每台机器上都要运行的组件，负责管理pod、容器的生命周期kubectl:集群管理工具（可选，只要在控制集群的节点上安装即可） # 配置yum源（科学上网的可以把"mirrors.aliyun.com"替换为"packages.cloud.google.com"） $ cat <<EOF > /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=0 repo_gpgcheck=0 gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF # 安装工具 # 找到要安装的版本号 $ yum list kubeadm --showduplicates | sort -r # 安装指定版本（这里用的是1.14.0） $ yum install -y kubeadm-1.14.0-0 kubelet-1.14.0-0 kubectl-1.14.0-0 --disableexcludes=kubernetes # 设置kubelet的cgroupdriver（kubelet的cgroupdriver默认为systemd，如果上面没有设置docker的exec-opts为systemd，这里就需要将kubelet的设置为cgroupfs） $ sed -i "s/cgroup-driver=systemd/cgroup-driver=cgroupfs/g" /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

部署主节点

kubeadm init --kubernetes-version=1.14.0 --apiserver-advertise-address=10.10.77.51 --image-repository registry.aliyuncs.com/google_containers --pod-network-cidr=172.22.0.0/16

注意：此处需要保存打印的join命令，在加入node节点时需要该命令

部署flannel网络

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/a70459be0084506e4ec919aa1c114638878db11b/Documentation/kube-flannel.yml

部署node节点

初始化集群时保存的join命令

kubeadm join 10.10.10.10:6443 --token zmo8rn.lfg2n5uj11nbfrss \ --discovery-token-ca-cert-hash sha256:a4a4f8c81524adfe85c105d2c1be2c8f9ee5fb4711ef96dff59e8c0297f34171

集群状态检测

kubectl get nodes kubectl get pods -n kube-system

创建nginx ds

# 写入配置 $ cat > nginx-ds.yml <<EOF apiVersion: v1 kind: Service metadata: name: nginx-ds labels: app: nginx-ds spec: type: NodePort selector: app: nginx-ds ports: - name: http port: 80 targetPort: 80 --- apiVersion: extensions/v1beta1 kind: DaemonSet metadata: name: nginx-ds labels: addonmanager.kubernetes.io/mode: Reconcile spec: template: metadata: labels: app: nginx-ds spec: containers: - name: my-nginx image: nginx:1.7.9 ports: - containerPort: 80 EOF # 创建ds $ kubectl create -f nginx-ds.yml

检查各种ip连通性

# 检查各 Node 上的 Pod IP 连通性 $ kubectl get pods -o wide # 在每个节点上ping pod ip $ ping <pod-ip> # 检查service可达性 $ kubectl get svc # 在每个节点上访问服务 $ curl <service-ip>:<port> # 在每个节点检查node-port可用性 $ curl <node-ip>:<port>

检查dns可用性

# 创建一个nginx pod $ cat > pod-nginx.yaml <<EOF apiVersion: v1 kind: Pod metadata: name: nginx spec: containers: - name: nginx image: nginx:1.7.9 ports: - containerPort: 80 EOF # 创建pod $ kubectl create -f pod-nginx.yaml # 进入pod，查看dns $ kubectl exec nginx -i -t -- /bin/bash # 查看dns配置 root@nginx:/# cat /etc/resolv.conf # 查看名字是否可以正确解析 root@nginx:/# ping nginx-ds

部署Dashboard

dashboard的yaml文件

# Copyright 2017 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ------------------- Dashboard Secret ------------------- # apiVersion: v1 kind: Secret metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard-certs namespace: kube-system type: Opaque --- # ------------------- Dashboard Service Account ------------------- # apiVersion: v1 kind: ServiceAccount metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kube-system --- # ------------------- Dashboard Role & Role Binding ------------------- # kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: kubernetes-dashboard-minimal namespace: kube-system rules: # Allow Dashboard to create 'kubernetes-dashboard-key-holder' secret. - apiGroups: [""] resources: ["secrets"] verbs: ["create"] # Allow Dashboard to create 'kubernetes-dashboard-settings' config map. - apiGroups: [""] resources: ["configmaps"] verbs: ["create"] # Allow Dashboard to get, update and delete Dashboard exclusive secrets. - apiGroups: [""] resources: ["secrets"] resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs"] verbs: ["get", "update", "delete"] # Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map. - apiGroups: [""] resources: ["configmaps"] resourceNames: ["kubernetes-dashboard-settings"] verbs: ["get", "update"] # Allow Dashboard to get metrics from heapster. - apiGroups: [""] resources: ["services"] resourceNames: ["heapster"] verbs: ["proxy"] - apiGroups: [""] resources: ["services/proxy"] resourceNames: ["heapster", "http:heapster:", "https:heapster:"] verbs: ["get"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: kubernetes-dashboard-minimal namespace: kube-system roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: kubernetes-dashboard-minimal subjects: - kind: ServiceAccount name: kubernetes-dashboard namespace: kube-system --- # ------------------- Dashboard Deployment ------------------- # kind: Deployment apiVersion: apps/v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kube-system spec: replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: k8s-app: kubernetes-dashboard template: metadata: labels: k8s-app: kubernetes-dashboard spec: containers: - name: kubernetes-dashboard image: lizhenliang/kubernetes-dashboard-amd64:v1.10.1 ports: - containerPort: 8443 protocol: TCP args: - --auto-generate-certificates # Uncomment the following line to manually specify Kubernetes API server Host # If not specified, Dashboard will attempt to auto discover the API server and connect # to it. Uncomment only if the default does not work. # - --apiserver-host=http://my-address:port volumeMounts: - name: kubernetes-dashboard-certs mountPath: /certs # Create on-disk volume to store exec logs - mountPath: /tmp name: tmp-volume livenessProbe: httpGet: scheme: HTTPS path: / port: 8443 initialDelaySeconds: 30 timeoutSeconds: 30 volumes: - name: kubernetes-dashboard-certs secret: secretName: kubernetes-dashboard-certs - name: tmp-volume emptyDir: {} serviceAccountName: kubernetes-dashboard # Comment the following tolerations if Dashboard must not be deployed on master tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule --- # ------------------- Dashboard Service ------------------- # kind: Service apiVersion: v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kube-system spec: type: NodePort ports: - port: 443 targetPort: 8443 nodePort: 30001 selector: k8s-app: kubernetes-dashboard # 创建服务 $ kubectl apply -f kubernetes-dashboard.yaml # 查看服务运行情况 $ kubectl get deployment kubernetes-dashboard -n kube-system $ kubectl --namespace kube-system get pods -o wide $ kubectl get services kubernetes-dashboard -n kube-system $ netstat -ntlp|grep 30001

访问dashboard 为了集群安全，从 1.7 开始，dashboard 只允许通过 https 访问，我们使用nodeport的方式暴露服务，可以使用 https://NodeIP:NodePort 地址访问 关于自定义证书 默认dashboard的证书是自动生成的，肯定是非安全的证书，如果大家有域名和对应的安全证书可以自己替换掉。使用安全的域名方式访问dashboard。 在dashboard-all.yaml中增加dashboard启动参数，可以指定证书文件，其中证书文件是通过secret注进来的。

–tls-cert-filedashboard.cer–tls-key-filedashboard.key

登录dashboard

Dashboard 默认只支持 token 认证，所以如果使用 KubeConfig 文件，需要在该文件中指定 token，我们这里使用token的方式登录

# 创建service account $ kubectl create sa dashboard-admin -n kube-system # 创建角色绑定关系 $ kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin # 查看dashboard-admin的secret名字 $ ADMIN_SECRET=$(kubectl get secrets -n kube-system | grep dashboard-admin | awk '{print $1}') # 打印secret的token $ kubectl describe secret -n kube-system ${ADMIN_SECRET} | grep -E '^token' | awk '{print $2}'