2: 2021-09-07 (火) 14:34:26 iseki |
現: 2021-09-15 (水) 13:03:38 iseki |
| - Podによるクラスター構築環境 | | - Podによるクラスター構築環境 |
| - 勉強中 | | - 勉強中 |
| + | |
| + | - [[kubectl>./kubectl]] |
| + | - [[helm>./helm]] |
| + | |
| + | - RANCHER https://qiita.com/suzukihi724/items/00b167c6f5f2ddeca718 |
| #br | | #br |
| - https://qiita.com/ishida0503/items/f3b62b02dec4f6fef42f | | - https://qiita.com/ishida0503/items/f3b62b02dec4f6fef42f |
| #br | | #br |
- | curl -L https://github.com/containernetworking/plugins/releases/download/v1.0.0/cni-plugins-linux-amd64-v1.0.0.tgz | tar -C /usr/local/cni/bin -xz | + | **** ''Check !!!! 後でチェックする.'' [#f5a64eb5] |
| + | - kubectl config set-context $(kubectl config current-context) --namespace k8sns |
| + | #br |
| + | |
| + | ** Install (CentOS8)[#zf601099] |
| + | - オンプレ用 |
| + | #br |
| + | |
| + | *** swap を止める [#i460600e] |
| + | # swapoff -a |
| + | - 再起動するとまた有効になるので,/etc/fstab でスワップ行をコメントアウト |
| + | #br |
| + | |
| + | *** firewalld を止める. [#k1b76d7b] |
| + | #br |
| + | *** routing の適用設定 [#k9224b53] |
| + | - /etc/sysctl.d/k8s.conf |
| + | net.bridge.bridge-nf-call-ip6tables = 1 |
| + | net.bridge.bridge-nf-call-iptables = 1 |
| + | -sysctl --system |
| + | #br |
| + | |
| + | *** 本体のインストール [#sabae243] |
| + | |
| + | **** K8sリポジトリ(RHEL7) [#o2437fdb] |
| + | - /etc/yum.repos.d/kubernetes.repo |
| + | [kubernetes] |
| + | name=Kubernetes |
| + | baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64 |
| + | enabled=1 |
| + | gpgcheck=1 |
| + | repo_gpgcheck=1 |
| + | gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg |
| + | exclude=kube* |
| + | |
| + | |
| + | - yum install -y ipvsadm iproute-tc |
| + | - yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes |
| + | #br |
| + | |
| + | **** RHEL7 ではなくて,最新版を手動で入れる. [#dae1dbdf] |
| + | - yum install -y socat iproute-tc ipvsadm conntrack-tools |
| + | - バイナリのダウンロード |
| + | cd /usr/bin |
| + | RELEASE="$(curl -sSL https://dl.k8s.io/release/stable.txt)" |
| + | curl -L --remote-name-all https://storage.googleapis.com/kubernetes-release/release/${RELEASE}/bin/linux/amd64/{kubeadm,kubelet,kubectl} |
| + | chmod a+rx /usr/bin/kube* |
| + | |
| + | - /usr/lib/systemd/system/kubelet.service |
| + | [Unit] |
| + | Description=kubelet: The Kubernetes Node Agent |
| + | Documentation=https://kubernetes.io/docs/ |
| + | Wants=network-online.target |
| + | After=network-online.target |
| + | |
| + | [Service] |
| + | ExecStart=/usr/bin/kubelet |
| + | Restart=always |
| + | StartLimitInterval=0 |
| + | RestartSec=10 |
| + | |
| + | [Install] |
| + | WantedBy=multi-user.target |
| + | |
| + | - /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf |
| + | [Service] |
| + | Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf" |
| + | Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml" |
| + | # This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically |
| + | EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env |
| + | # This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use |
| + | # the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file. |
| + | EnvironmentFile=-/etc/sysconfig/kubelet |
| + | ExecStart= |
| + | ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS |
| + | |
| + | |
| + | #br |
| + | |
| + | *** cgroup の設定 [#f418f5e8] |
| + | - /etc/sysconfig/kubelet |
| + | KUBELET_EXTRA_ARGS="--cgroup-driver=cgroupfs" |
| + | |
| + | - 設定しないと kubeadm が正常に動作しない |
| + | #br |
| + | |
| + | *** クラスター構築 [#n5846171] |
| + | |
| + | **** kubeadm [#e750df5b] |
| + | - kubeadm init --pod-network-cidr=10.128.0.0/16 --service-cidr 10.128.0.0/16 --control-plane-endpoint=172.22.1.75:6443 |
| + | -- --control-plane-endpoint は内部 DNSのIP:Port を指定する? |
| + | - 作られるファイル |
| + | -- /var/lib/etcd/ |
| + | -- /var/lib/kubelet |
| + | -- /etc/kubernetes |
| + | -- /etc/cni/net.d |
| + | #br |
| + | - もう一度設定する場合は,kubeadm reset を行う |
| + | -- ネットワークインターフェイスを作った場合は,削除して置く.ex) ip link delete flannel.1 |
| + | -- 何かリセットできない.... 関連ファイル削除,関連プロセス皆殺し... |
| + | #br |
| + | |
| + | **** minikube [#g2baeece] |
| + | - 別のクラスター構築ツール. |
| + | #br |
| + | |
| + | *** 環境設定 [#m6fea0a8] |
| + | - mkdir ~/.kube |
| + | - cp /etc/kubernetes/admin.conf ~/.kube/config |
| + | |
| + | - または |
| + | |
| + | - export KUBECONFIG=/etc/kubernetes/admin.conf |
| + | #br |
| + | |
| + | *** 仮想ネットワーク設定 CNI(calico) [#aa02daa6] |
| + | **** Create a single-host Kubernetes cluster [#h0381a92] |
| + | - https://docs.projectcalico.org/getting-started/kubernetes/quickstart |
| + | |
| + | kubectl create -f https://docs.projectcalico.org/manifests/tigera-operator.yaml |
| + | wget https://docs.projectcalico.org/manifests/custom-resources.yaml |
| + | vi custom-resources.yaml (correct IP) |
| + | kubectl create -f custom-resources.yaml |
| + | |
| + | watch kubectl get pods -n calico-system |
| + | kubectl taint nodes --all node-role.kubernetes.io/master- (マスタノードでも Podを実行できるようになる) |
| + | kubectl get nodes -o wide |
| + | |
| + | - /etc/NetworkManager/conf.d/calico.conf で以下の設定が必要になるかもしれない |
| + | [keyfile] |
| + | unmanaged-devices=interface-name:cali*;interface-name:tunl*;interface-name:vxlan.calico |
| + | |
| + | **** calicoctl [#t3e4c13e] |
| + | - 各ソフトのバージョンを合わせる. |
| + | - Install calicoctl as a binary on a single host |
| + | cd /usr/local/bin |
| + | curl -o calicoctl -O -L "https://github.com/projectcalico/calicoctl/releases/download/v3.20.0/calicoctl" |
| + | chmod a+rx calicoctl |
| + | ln -s calicoctl kubectl-calico |
| + | -- kubectl calico -h でチェック |
| + | |
| + | - Install calicoctl as a container on a single host |
| + | docker pull calico/ctl:v3.20.0 |
| + | - Install calicoctl as a Kubernetes pod |
| + | kubectl apply -f https://docs.projectcalico.org/manifests/calicoctl.yaml |
| + | - 確認 |
| + | kubectl exec -ti -n kube-system calicoctl -- /calicoctl get profiles -o wide |
| + | #br |
| + | |
| + | ***** 他の方法 [#dba04328] |
| + | - その1 |
| + | curl -L https://docs.projectcalico.org/manifests/calico.yaml | \ |
| + | sed '/ - name: CALICO_DISABLE_FILE_LOGGING/i\ # ADD' | \ |
| + | sed '/ - name: CALICO_DISABLE_FILE_LOGGING/i\ - name: FELIX_IPTABLESBACKEND' | \ |
| + | sed '/ - name: CALICO_DISABLE_FILE_LOGGING/i\ value: Auto' | \ |
| + | sed '/ - name: CALICO_DISABLE_FILE_LOGGING/i\ # ADD' | \ |
| + | sed '/ - name: CALICO_DISABLE_FILE_LOGGING/i\ - name: CALICO_IPV4POOL_CIDR' | \ |
| + | sed '/ - name: CALICO_DISABLE_FILE_LOGGING/i\ value: \"10.128.0.0\/16\"' | \ |
| + | cat - > calico.yaml |
| + | kubectl apply -f calico.yaml |
| + | |
| + | watch kubectl get pods -n calico-system |
| + | kubectl taint nodes --all node-role.kubernetes.io/master- |
| + | kubectl get nodes -o wide |
| + | |
| + | - その2 |
| + | cat <<EOF > /etc/NetworkManager/conf.d/calico.conf |
| + | [keyfile] |
| + | unmanaged-devices=interface-name:cali*;interface-name:tunl*;interface-name:vxlan.calico |
| + | EOF |
| + | wget https://docs.projectcalico.org/v3.15/manifests/calico.yaml |
| + | vi calico.yaml (correct IP) |
| + | kubectl apply -f calico.yaml |
| + | |
| + | watch kubectl get pods -n calico-system |
| + | kubectl taint nodes --all node-role.kubernetes.io/master- |
| + | kubectl get nodes -o wide |
| + | |
| + | *** Worker node [#w001f6b1] |
| + | - 初期化で kubeadm join コマンドを打つ. |
| + | -- ex) kubeadm join 172.22.1.75:6443 --token nmtraf.wfzxuqzqti5unh9f --discovery-token-ca-cert-hash sha256:13d681e6bd6466503666bbb... |
| + | - マスタの /etc/kubernetes/admin.conf をコピーすると,kubectl コマンドが打てるようになる. |
| + | #br |
| + | |
| + | *** MetalLB Load Balancer [#eae94dce] |
| + | - https://zaki-hmkc.hatenablog.com/entry/2020/07/10/235944 |
| + | |
| + | kubectl edit configmap -n kube-system kube-proxy |
| + | kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/namespace.yaml |
| + | kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/metallb.yaml |
| + | kubectl get ns |
| + | kubectl get pod -n metallb-system |
| + | kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)" |
| + | vi l2-configuration.yaml |
| + | kubectl apply -f l2-configuration.yaml |
| + | kubectl get service |
| + | |
| + | - [[l2-configuration.yaml>./l2-configuration.yaml]] |
| + | #br |
| + | |
| + | ** Getting first [#o1bbd83d] |
| + | - kubeadm はデフォルト状態では --type=LoadBalancer が使用できないという情報あり? |
| + | -- EXTERNA-IP が pending のままになる. |
| + | #br |
| + | |
| + | *** nginx [#f3a9efea] |
| + | - https://qiita.com/suzukihi724/items/241f7241d297a2d4a55c |
| + | #br |
| + | |
| + | **** MetalLB を使用した場合 [#l6b076d5] |
| + | kubectl run nginx --image=nginx:1.11.3 |
| + | kubectl get pod |
| + | kubectl expose pod nginx --port=80 --type=LoadBalancer --name=nginx |
| + | kubectl get service |
| + | |
| + | NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE |
| + | kubernetes ClusterIP 10.128.0.1 <none> 443/TCP 17m |
| + | nginx LoadBalancer 10.128.44.222 172.22.1.70 80:32606/TCP 3m42s |
| + | |
| + | - 172.22.1.70:80 へのアクセスで Webに接続可 |
| + | #br |
| + | |
| + | *** OLD: nginx [#fe881da9] |
| + | |
| + | **** Start [#x082d3f7] |
| + | kubectl run nginx --image=nginx:1.11.3 |
| + | kubectl get pod |
| + | kubectl expose pod nginx --external-ip=172.22.1.75 --port=80 |
| + | kubectl get service |
| + | |
| + | NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE |
| + | kubernetes ClusterIP 10.128.0.1 <none> 443/TCP 15m |
| + | nginx ClusterIP 10.128.175.104 192.168.27.43 80/TCP 3s |
| + | - access to http://192.168.27.43 |
| + | - IP を自分のもの以外にしても,アクセスポイントはできる.ルーティングしていないから外部からはアクセス不可. |
| + | #br |
| + | |
| + | **** Stop [#k5b28d74] |
| + | kubectl delete service nginx |
| + | kubectl get service |
| + | kubectl delete pod nginx |
| + | kubectl get pod |
| + | #br |
| + | |
| + | **** LoadBalancer [#meaea643] |
| + | - IPを指定した場合,--type=LoadBalancer を付けると,そのまま通る. |
| + | - その場合,ポートは通常の 80番 と LoadBalancerが指定した番号,どちらも使用できる? |
| + | |
| + | # kubectl expose pod nginx --external-ip=172.22.1.75 --port=80 --type=LoadBalancer |
| + | # kubectl get service |
| + | NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE |
| + | kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 80m |
| + | nginx LoadBalancer 10.107.28.138 172.22.1.75 80:32348/TCP 4s |
| + | - --external-ip を指定しない場合は,pending となる |
| + | |
| + | # kubectl expose pod nginx --port=80 --type=LoadBalancer --name=nginx2 |
| + | # kubectl get service |
| + | NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE |
| + | kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 95m |
| + | nginx LoadBalancer 10.107.28.138 172.22.1.75 80:32348/TCP 14m |
| + | nginx2 LoadBalancer 10.107.150.209 <pending> 80:31353/TCP 7s |
| + | - pending 状態のサービスには,kubectl edit service -n .... で pending 部分を編集できる! |
| + | |
| + | spec: |
| + | allocateLoadBalancerNodePorts: true |
| + | clusterIP: 10.128.90.50 |
| + | clusterIPs: |
| + | - 10.128.90.50 |
| + | externalIPs: |
| + | - 192.168.27.43 |
| + | externalTrafficPolicy: Cluster |
| + | |
| + | **** 自分のIP以外でも指定可能 [#i0993ac6] |
| + | # kubectl expose pod nginx --port 80 --external-ip=192.168.27.44 --name=nginx2 |
| + | # kubectl get service |
| + | NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE |
| + | kubernetes ClusterIP 10.128.0.1 <none> 443/TCP 67m |
| + | nginx ClusterIP 10.128.175.104 192.168.27.43 80/TCP 51m |
| + | nginx2 ClusterIP 10.128.175.124 192.168.27.44 80/TCP 4m53s |
| + | #br |
| + | *** JupyterHub [#t5f59cf9] |
| + | - [[JupyterHub>./JupyterHub]] |
| + | #br |