In this LSF458 lab (originally Excercise 4.3 and 4.4), we will explore how to manage maintenance modes of nodes in a Kubernetes cluster. For that, we will deploy a complex example, before we drain the node.
Step 0: Enter Kubernetes Playground
We will use the Katacode Kubernetes Playground again. Just click on launch.sh on the left pane, and we are ready to go.
Step 1: Deploy a complex Scenario
We deploy a complex example from Github like follows:
kubectl create namespace sock-shop kubectl apply -f https://raw.githubusercontent.com/microservices-demo/microservices-demo/master/deploy/kubernetes/complete-demo.yaml
In this example, we have applied the configuration from an HTTP resource on the Internet.
With the following command, we can explore the status of all resources in the sock-shop namespace:
kubectl get all -n sock-shop
After two minutes, all PODS should be up and running:
kubectl get pods -n sock-shop # output: NAME READY STATUS RESTARTS AGE carts-d646975bf-cb858 1/1 Running 0 113s carts-db-f5677f464-v7bsk 1/1 Running 0 113s catalogue-7db746bc5b-7lzbj 1/1 Running 0 113s catalogue-db-55965799b9-7cwgp 1/1 Running 0 113s front-end-55b99f8c59-56wb5 1/1 Running 0 113s orders-986494c98-srmdm 1/1 Running 0 113s orders-db-78568b65bb-jjvr5 1/1 Running 0 113s payment-7d4d4bf9b4-dm2t5 1/1 Running 0 113s queue-master-6b5b5c7658-p7nj7 1/1 Running 0 113s rabbitmq-f984b75c4-2wdcz 1/1 Running 0 113s shipping-7b8865d964-znngm 1/1 Running 0 112s user-54555fbb7-8mm6r 1/1 Running 0 112s user-db-7b9846c559-wplbv 1/1 Running 0 112s
Step 2: Drain the Worker Node
Our cluster consists of a master and a worker node:
kubectl get nodes # output: NAME STATUS ROLES AGE VERSION master Ready master 31m v1.14.0 node01 Ready <none> 31m v1.14.0
After having made a note of the worker node’s name, we can drain the worker node as follows:
# 1st try: kubectl drain node01 # output: node/node01 cordoned error: unable to drain node "node01", aborting command... There are pending nodes to be drained: node01 cannot delete DaemonSet-managed Pods (use --ignore-daemonsets to ignore): kube-system/kube-proxy-549d7, kube-system/weave-net-fk259 cannot delete Pods with local storage (use --delete-local-data to override): sock-shop/carts-d646975bf-cb858, sock-shop/carts-db-f5677f464-v7bsk, sock-shop/orders-986494c98-srmdm, sock-shop/orders-db-78568b65bb-jjvr5, sock-shop/shipping-7b8865d964-znngm, sock-shop/user-db-7b9846c559-wplbv
Let us try again with the suggested option to ignore the DaemonSets:
# 2nd try: kubectl drain node01 --ignore-daemonsets # output: node/node01 already cordoned error: unable to drain node "node01", aborting command... There are pending nodes to be drained: node01 error: cannot delete Pods with local storage (use --delete-local-data to override): sock-shop/carts-d646975bf-cb858, sock-shop/carts-db-f5677f464-v7bsk, sock-shop/orders-986494c98-srmdm, sock-shop/orders-db-78568b65bb-jjvr5, sock-shop/shipping-7b8865d964-znngm,sock-shop/user-db-7b9846c559-wplbv
This has failed again because of some PODs with local storage. Finally, we will succeed with the following command:
# finally: kubectl drain node01 --ignore-daemonsets --delete-local-data # output: node/node01 already cordoned WARNING: ignoring DaemonSet-managed Pods: kube-system/kube-proxy-549d7, kube-system/weave-net-fk259 evicting pod "user-db-7b9846c559-wplbv" evicting pod "queue-master-6b5b5c7658-p7nj7" evicting pod "orders-db-78568b65bb-jjvr5" evicting pod "orders-986494c98-srmdm" evicting pod "carts-d646975bf-cb858" evicting pod "carts-db-f5677f464-v7bsk" evicting pod "shipping-7b8865d964-znngm" evicting pod "rabbitmq-f984b75c4-2wdcz" evicting pod "payment-7d4d4bf9b4-dm2t5" evicting pod "user-54555fbb7-8mm6r" evicting pod "catalogue-db-55965799b9-7cwgp" evicting pod "front-end-55b99f8c59-56wb5" evicting pod "catalogue-7db746bc5b-7lzbj" pod/user-54555fbb7-8mm6r evicted pod/catalogue-db-55965799b9-7cwgp evicted pod/rabbitmq-f984b75c4-2wdcz evicted pod/catalogue-7db746bc5b-7lzbj evicted pod/orders-db-78568b65bb-jjvr5 evicted pod/carts-db-f5677f464-v7bsk evicted pod/payment-7d4d4bf9b4-dm2t5 evicted pod/user-db-7b9846c559-wplbv evicted pod/shipping-7b8865d964-znngm evicted pod/carts-d646975bf-cb858 evicted pod/orders-986494c98-srmdm evicted pod/front-end-55b99f8c59-56wb5 evicted pod/queue-master-6b5b5c7658-p7nj7 evicted node/node01 evicted
Note that all PODs are in Pending state now, since we have drained the only worker node available in our cluster:
kubectl get all -n sock-shop # output: NAME READY STATUS RESTARTS AGE pod/carts-d646975bf-qdh7h 0/1 Pending 0 8m32s pod/carts-db-f5677f464-dtmhm 0/1 Pending 0 8m32s pod/catalogue-7db746bc5b-njrz7 0/1 Pending 0 8m31s pod/catalogue-db-55965799b9-lp6h6 0/1 Pending 0 8m32s pod/front-end-55b99f8c59-7bv8p 0/1 Pending 0 8m32s pod/orders-986494c98-fvjh6 0/1 Pending 0 8m32s pod/orders-db-78568b65bb-d7lpt 0/1 Pending 0 8m32s pod/payment-7d4d4bf9b4-c8f7v 0/1 Pending 0 8m32s pod/queue-master-6b5b5c7658-k5bkr 0/1 Pending 0 8m32s pod/rabbitmq-f984b75c4-qtv7l 0/1 Pending 0 8m32s pod/shipping-7b8865d964-pphhf 0/1 Pending 0 8m32s pod/user-54555fbb7-cjzw8 0/1 Pending 0 8m32s pod/user-db-7b9846c559-264xt 0/1 Pending 0 8m31s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/carts ClusterIP 10.103.27.31 80/TCP 15m service/carts-db ClusterIP 10.109.74.181 27017/TCP 15m service/catalogue ClusterIP 10.103.60.110 80/TCP 15m service/catalogue-db ClusterIP 10.102.223.130 3306/TCP 15m service/front-end NodePort 10.110.176.33 80:30001/TCP 15m service/orders ClusterIP 10.103.218.142 80/TCP 15m service/orders-db ClusterIP 10.108.148.237 27017/TCP 15m service/payment ClusterIP 10.101.72.86 80/TCP 15m service/queue-master ClusterIP 10.99.36.110 80/TCP 15m service/rabbitmq ClusterIP 10.98.195.97 5672/TCP 15m service/shipping ClusterIP 10.96.235.40 80/TCP 15m service/user ClusterIP 10.105.80.42 80/TCP 15m service/user-db ClusterIP 10.96.150.174 27017/TCP 15m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/carts 0/1 1 0 15m deployment.apps/carts-db 0/1 1 0 15m deployment.apps/catalogue 0/1 1 0 15m deployment.apps/catalogue-db 0/1 1 0 15m deployment.apps/front-end 0/1 1 0 15m deployment.apps/orders 0/1 1 0 15m deployment.apps/orders-db 0/1 1 0 15m deployment.apps/payment 0/1 1 0 15m deployment.apps/queue-master 0/1 1 0 15m deployment.apps/rabbitmq 0/1 1 0 15m deployment.apps/shipping 0/1 1 0 15m deployment.apps/user 0/1 1 0 15m deployment.apps/user-db 0/1 1 0 15m NAME DESIRED CURRENT READY AGE replicaset.apps/carts-d646975bf 1 1 0 15m replicaset.apps/carts-db-f5677f464 1 1 0 15m replicaset.apps/catalogue-7db746bc5b 1 1 0 15m replicaset.apps/catalogue-db-55965799b9 1 1 0 15m replicaset.apps/front-end-55b99f8c59 1 1 0 15m replicaset.apps/orders-986494c98 1 1 0 15m replicaset.apps/orders-db-78568b65bb 1 1 0 15m replicaset.apps/payment-7d4d4bf9b4 1 1 0 15m replicaset.apps/queue-master-6b5b5c7658 1 1 0 15m replicaset.apps/rabbitmq-f984b75c4 1 1 0 15m replicaset.apps/shipping-7b8865d964 1 1 0 15m replicaset.apps/user-54555fbb7 1 1 0 15m replicaset.apps/user-db-7b9846c559 1 1 0 15m
Step 3: Perform Maintenance Actions
Now, we could update the worker node, as desired.
The worker node and the master node has the following taints:
kubectl describe node | grep -i taint # output: Taints: node-role.kubernetes.io/master:NoSchedule <---------- master Taints: node.kubernetes.io/unschedulable:NoSchedule <-------- worker
Step 4: Take Node into Service again
Now let us take the node into service again:
kubectl uncordon node01 # output: node/node01 uncordoned
The taint is removed from the worker node again:
kubectl describe node | grep -i taint # output: Taints: node-role.kubernetes.io/master:NoSchedule Taints: <none>
After some time, all PODs are up and running again:
kubectl get pods -n sock-shop # output: NAME READY STATUS RESTARTS AGE carts-d646975bf-qdh7h 1/1 Running 0 20m carts-db-f5677f464-dtmhm 1/1 Running 0 20m catalogue-7db746bc5b-njrz7 1/1 Running 0 20m catalogue-db-55965799b9-lp6h6 1/1 Running 0 20m front-end-55b99f8c59-7bv8p 1/1 Running 0 20m orders-986494c98-fvjh6 1/1 Running 0 20m orders-db-78568b65bb-d7lpt 1/1 Running 0 20m payment-7d4d4bf9b4-c8f7v 1/1 Running 0 20m queue-master-6b5b5c7658-k5bkr 1/1 Running 0 20m rabbitmq-f984b75c4-qtv7l 1/1 Running 0 20m shipping-7b8865d964-pphhf 1/1 Running 0 20m user-54555fbb7-cjzw8 1/1 Running 0 20m user-db-7b9846c559-264xt 1/1 Running 0 20m
Step 5: Delete the Deployment
Since we have chosen to deploy the sock-shop into its own namespace, we can easily delete all of its resources with a single command:
kubectl delete namespace sock-shop # output: namespace "sock-shop" deleted
The synchronous delete command will hang a while since it is waiting for all resources to be deleted.
Thanks for sharing. I read many of your blog posts, cool, your blog is very good.
Thanks for sharing. I read many of your blog posts, cool, your blog is very good.