In this LSF458 lab (originally Excercise 4.3 and 4.4), we will explore how to manage maintenance modes of nodes in a Kubernetes cluster. For that, we will deploy a complex example, before we drain the node.

Step 0: Enter Kubernetes Playground

We will use the Katacode Kubernetes Playground again. Just click on launch.sh on the left pane, and we are ready to go.

Step 1: Deploy a complex Scenario

We deploy a complex example from Github like follows:

kubectl create namespace sock-shop
kubectl apply -f https://raw.githubusercontent.com/microservices-demo/microservices-demo/master/deploy/kubernetes/complete-demo.yaml

In this example, we have applied the configuration from an HTTP resource on the Internet.

With the following command, we can explore the status of all resources in the sock-shop namespace:

kubectl get all -n sock-shop

After two minutes, all PODS should be up and running:

kubectl get pods -n sock-shop

# output:
NAME                            READY   STATUS    RESTARTS   AGE
carts-d646975bf-cb858           1/1     Running   0          113s
carts-db-f5677f464-v7bsk        1/1     Running   0          113s
catalogue-7db746bc5b-7lzbj      1/1     Running   0          113s
catalogue-db-55965799b9-7cwgp   1/1     Running   0          113s
front-end-55b99f8c59-56wb5      1/1     Running   0          113s
orders-986494c98-srmdm          1/1     Running   0          113s
orders-db-78568b65bb-jjvr5      1/1     Running   0          113s
payment-7d4d4bf9b4-dm2t5        1/1     Running   0          113s
queue-master-6b5b5c7658-p7nj7   1/1     Running   0          113s
rabbitmq-f984b75c4-2wdcz        1/1     Running   0          113s
shipping-7b8865d964-znngm       1/1     Running   0          112s
user-54555fbb7-8mm6r            1/1     Running   0          112s
user-db-7b9846c559-wplbv        1/1     Running   0          112s

Step 2: Drain the Worker Node

Our cluster consists of a master and a worker node:

kubectl get nodes

# output:
NAME     STATUS   ROLES    AGE   VERSION
master   Ready    master   31m   v1.14.0
node01   Ready    <none>   31m   v1.14.0

After having made a note of the worker node’s name, we can drain the worker node as follows:

# 1st try:
kubectl drain node01

# output:
node/node01 cordoned
error: unable to drain node "node01", aborting command...

There are pending nodes to be drained:
 node01
cannot delete DaemonSet-managed Pods (use --ignore-daemonsets to ignore): kube-system/kube-proxy-549d7, kube-system/weave-net-fk259
cannot delete Pods with local storage (use --delete-local-data to override): sock-shop/carts-d646975bf-cb858, sock-shop/carts-db-f5677f464-v7bsk, sock-shop/orders-986494c98-srmdm, sock-shop/orders-db-78568b65bb-jjvr5, sock-shop/shipping-7b8865d964-znngm, sock-shop/user-db-7b9846c559-wplbv

Let us try again with the suggested option to ignore the DaemonSets:

# 2nd try:
kubectl drain node01 --ignore-daemonsets

# output:
node/node01 already cordoned
error: unable to drain node "node01", aborting command...

There are pending nodes to be drained:
 node01
error: cannot delete Pods with local storage (use --delete-local-data to override): sock-shop/carts-d646975bf-cb858, sock-shop/carts-db-f5677f464-v7bsk, sock-shop/orders-986494c98-srmdm, sock-shop/orders-db-78568b65bb-jjvr5, sock-shop/shipping-7b8865d964-znngm,sock-shop/user-db-7b9846c559-wplbv

This has failed again because of some PODs with local storage. Finally, we will succeed with the following command:

# finally:
kubectl drain node01 --ignore-daemonsets --delete-local-data

# output:
node/node01 already cordoned
WARNING: ignoring DaemonSet-managed Pods: kube-system/kube-proxy-549d7, kube-system/weave-net-fk259
evicting pod "user-db-7b9846c559-wplbv"
evicting pod "queue-master-6b5b5c7658-p7nj7"
evicting pod "orders-db-78568b65bb-jjvr5"
evicting pod "orders-986494c98-srmdm"
evicting pod "carts-d646975bf-cb858"
evicting pod "carts-db-f5677f464-v7bsk"
evicting pod "shipping-7b8865d964-znngm"
evicting pod "rabbitmq-f984b75c4-2wdcz"
evicting pod "payment-7d4d4bf9b4-dm2t5"
evicting pod "user-54555fbb7-8mm6r"
evicting pod "catalogue-db-55965799b9-7cwgp"
evicting pod "front-end-55b99f8c59-56wb5"
evicting pod "catalogue-7db746bc5b-7lzbj"
pod/user-54555fbb7-8mm6r evicted
pod/catalogue-db-55965799b9-7cwgp evicted
pod/rabbitmq-f984b75c4-2wdcz evicted
pod/catalogue-7db746bc5b-7lzbj evicted
pod/orders-db-78568b65bb-jjvr5 evicted
pod/carts-db-f5677f464-v7bsk evicted
pod/payment-7d4d4bf9b4-dm2t5 evicted
pod/user-db-7b9846c559-wplbv evicted
pod/shipping-7b8865d964-znngm evicted
pod/carts-d646975bf-cb858 evicted
pod/orders-986494c98-srmdm evicted
pod/front-end-55b99f8c59-56wb5 evicted
pod/queue-master-6b5b5c7658-p7nj7 evicted
node/node01 evicted

Note that all PODs are in Pending state now, since we have drained the only worker node available in our cluster:

kubectl get all -n sock-shop

# output:
NAME                                READY   STATUS    RESTARTS   AGE
pod/carts-d646975bf-qdh7h           0/1     Pending   0          8m32s
pod/carts-db-f5677f464-dtmhm        0/1     Pending   0          8m32s
pod/catalogue-7db746bc5b-njrz7      0/1     Pending   0          8m31s
pod/catalogue-db-55965799b9-lp6h6   0/1     Pending   0          8m32s
pod/front-end-55b99f8c59-7bv8p      0/1     Pending   0          8m32s
pod/orders-986494c98-fvjh6          0/1     Pending   0          8m32s
pod/orders-db-78568b65bb-d7lpt      0/1     Pending   0          8m32s
pod/payment-7d4d4bf9b4-c8f7v        0/1     Pending   0          8m32s
pod/queue-master-6b5b5c7658-k5bkr   0/1     Pending   0          8m32s
pod/rabbitmq-f984b75c4-qtv7l        0/1     Pending   0          8m32s
pod/shipping-7b8865d964-pphhf       0/1     Pending   0          8m32s
pod/user-54555fbb7-cjzw8            0/1     Pending   0          8m32s
pod/user-db-7b9846c559-264xt        0/1     Pending   0          8m31s

NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
service/carts          ClusterIP   10.103.27.31             80/TCP         15m
service/carts-db       ClusterIP   10.109.74.181            27017/TCP      15m
service/catalogue      ClusterIP   10.103.60.110            80/TCP         15m
service/catalogue-db   ClusterIP   10.102.223.130           3306/TCP       15m
service/front-end      NodePort    10.110.176.33            80:30001/TCP   15m
service/orders         ClusterIP   10.103.218.142           80/TCP         15m
service/orders-db      ClusterIP   10.108.148.237           27017/TCP      15m
service/payment        ClusterIP   10.101.72.86             80/TCP         15m
service/queue-master   ClusterIP   10.99.36.110             80/TCP         15m
service/rabbitmq       ClusterIP   10.98.195.97             5672/TCP       15m
service/shipping       ClusterIP   10.96.235.40             80/TCP         15m
service/user           ClusterIP   10.105.80.42             80/TCP         15m
service/user-db        ClusterIP   10.96.150.174            27017/TCP      15m

NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/carts          0/1     1            0           15m
deployment.apps/carts-db       0/1     1            0           15m
deployment.apps/catalogue      0/1     1            0           15m
deployment.apps/catalogue-db   0/1     1            0           15m
deployment.apps/front-end      0/1     1            0           15m
deployment.apps/orders         0/1     1            0           15m
deployment.apps/orders-db      0/1     1            0           15m
deployment.apps/payment        0/1     1            0           15m
deployment.apps/queue-master   0/1     1            0           15m
deployment.apps/rabbitmq       0/1     1            0           15m
deployment.apps/shipping       0/1     1            0           15m
deployment.apps/user           0/1     1            0           15m
deployment.apps/user-db        0/1     1            0           15m

NAME                                      DESIRED   CURRENT   READY   AGE
replicaset.apps/carts-d646975bf           1         1         0       15m
replicaset.apps/carts-db-f5677f464        1         1         0       15m
replicaset.apps/catalogue-7db746bc5b      1         1         0       15m
replicaset.apps/catalogue-db-55965799b9   1         1         0       15m
replicaset.apps/front-end-55b99f8c59      1         1         0       15m
replicaset.apps/orders-986494c98          1         1         0       15m
replicaset.apps/orders-db-78568b65bb      1         1         0       15m
replicaset.apps/payment-7d4d4bf9b4        1         1         0       15m
replicaset.apps/queue-master-6b5b5c7658   1         1         0       15m
replicaset.apps/rabbitmq-f984b75c4        1         1         0       15m
replicaset.apps/shipping-7b8865d964       1         1         0       15m
replicaset.apps/user-54555fbb7            1         1         0       15m
replicaset.apps/user-db-7b9846c559        1         1         0       15m

Step 3: Perform Maintenance Actions

Now, we could update the worker node, as desired.

The worker node and the master node has the following taints:

kubectl describe node | grep -i taint

# output:
Taints:             node-role.kubernetes.io/master:NoSchedule  <---------- master
Taints:             node.kubernetes.io/unschedulable:NoSchedule  <-------- worker

Step 4: Take Node into Service again

Now let us take the node into service again:

kubectl uncordon node01

# output:
node/node01 uncordoned

The taint is removed from the worker node again:

kubectl describe node | grep -i taint

# output:
Taints:             node-role.kubernetes.io/master:NoSchedule
Taints:             <none>

After some time, all PODs are up and running again:

kubectl get pods -n sock-shop

# output:
NAME                            READY   STATUS    RESTARTS   AGE
carts-d646975bf-qdh7h           1/1     Running   0          20m
carts-db-f5677f464-dtmhm        1/1     Running   0          20m
catalogue-7db746bc5b-njrz7      1/1     Running   0          20m
catalogue-db-55965799b9-lp6h6   1/1     Running   0          20m
front-end-55b99f8c59-7bv8p      1/1     Running   0          20m
orders-986494c98-fvjh6          1/1     Running   0          20m
orders-db-78568b65bb-d7lpt      1/1     Running   0          20m
payment-7d4d4bf9b4-c8f7v        1/1     Running   0          20m
queue-master-6b5b5c7658-k5bkr   1/1     Running   0          20m
rabbitmq-f984b75c4-qtv7l        1/1     Running   0          20m
shipping-7b8865d964-pphhf       1/1     Running   0          20m
user-54555fbb7-cjzw8            1/1     Running   0          20m
user-db-7b9846c559-264xt        1/1     Running   0          20m

Step 5: Delete the Deployment

Since we have chosen to deploy the sock-shop into its own namespace, we can easily delete all of its resources with a single command:

kubectl delete namespace sock-shop

# output:
namespace "sock-shop" deleted

The synchronous delete command will hang a while since it is waiting for all resources to be deleted.

Previous:

Resource Management

Next:

Kubernetes API

Comments

Diese Website verwendet Akismet, um Spam zu reduzieren. Erfahre mehr darüber, wie deine Kommentardaten verarbeitet werden.