In part 3 of the Certified Kubernetes Administrator Labs Challenge, we will deploy a simple application via command line as well as by applying a YAML file. Then we will expose and access the service from within the Kubernetes Cluster. Now we will explore how Kubernetes Deployments helps us maintain the service by automatically restarting failed PODS. Last, but not least, we will discuss how to access the service from outside the Kubernetes cluster.

Note: LFS458 starts with the installation of a Kubernetes cluster. We will skip this part for now and we will make use of the Katakoda Kubernetes playground instead. If you want to install a cluster, you might want to check out our blog post on kubeadm-based cluster installation or any other resource in the internet. Please use kubeadm and create a two node cluster with a master and a worker node.

Create Deployment per CLI command

This is the simplest way of creating an app on Kubernetes,m even if it is not the recommended one. However, later, we will learn better ways. Those will involve YAML files. However, for now, let us start with the simple command:

kubectl create deployment nginx --image=nginx
# output:
deployment.apps/nginx created

Show Deployments

Show all deployments:

kubectl get deployments
# output
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
nginx   1/1     1            1           52s

Describe Details

Describe all the details of the deployment:

master $ kubectl describe deployment nginx

# output:
Name:                   nginx
Namespace:              default
CreationTimestamp:      Fri, 19 Jul 2019 08:22:58 +0000
Labels:                 app=nginx
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               app=nginx
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=nginx
  Containers:
   nginx:
    Image:        nginx
    Port:         <none>
    Host Port:    <none>
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   nginx-65f88748fd (1/1 replicas created)
Events:
  Type    Reason             Age    From                   Message
  ----    ------             ----   ----                   -------
  Normal  ScalingReplicaSet  3m22s  deployment-controller  Scaled up replica set nginx-65f88748fd to 1

Show Events

Can we get a list of events on the cluster? This is, how:

kubectl get events

# output:

4m32s       Normal   Scheduled                 pod/nginx-65f88748fd-5dp7b    Successfully assigned default/nginx-65f88748fd-5dp7b to node01
4m31s       Normal   Pulling                   pod/nginx-65f88748fd-5dp7b    Pulling image "nginx"
4m24s       Normal   Pulled                    pod/nginx-65f88748fd-5dp7b    Successfully pulled image "nginx"
4m24s       Normal   Created                   pod/nginx-65f88748fd-5dp7b    Created container nginx
4m24s       Normal   Started                   pod/nginx-65f88748fd-5dp7b    Started container nginx
4m32s       Normal   SuccessfulCreate          replicaset/nginx-65f88748fd   Created pod: nginx-65f88748fd-5dp7b
4m32s       Normal   ScalingReplicaSet         deployment/nginx              Scaled up replica set nginx-65f88748fd to 1

You can see that creating a deployment is performing many steps. They seem to be somehow unordered in time. Interesting.

Deployments are one of the most important objects of Kubernetes, but they consist of many other, simpler objects, which we will explore in more details below:

  • Docker container
  • POD consists of one or more Docker containers
  • Replicaset makes sure that x replicas of PODs are always up and running
  • Deployment is a Replicaset with more features like rolling updates of PODs etc.

So, when we have created a Deployment, the following things happen:

  • The images for the containers defined in the POD are pulled if they are not already running on the worker node.
  • A scheduler looks for a worker node the PODs can be run on. Thus us node01 in our case, as can be seen from the event log. Note, that different Containers of a POD must be located on the same worker node, i.e. they may not be distributed among different worker nodes.
  • The Scheduler distributes the desired number of POD replicas evenly among worker nodes. That is,  replicas of the PODs can be run on different worker nodes.

Export Kubernetes Objects as YAML File

Above, we have stated that there are better, recommended ways of creating Kubernetes deployments using YAML files. We have not created our deployment using a YAML file yet, but we can export the existing Deployment as a YAML file:

kubectl get deployment nginx -o yaml

# output:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
  creationTimestamp: "2019-07-19T08:46:30Z"
  generation: 1
  labels:
    app: nginx
  name: nginx
  namespace: default
  resourceVersion: "3326"
  selfLink: /apis/extensions/v1beta1/namespaces/default/deployments/nginx
  uid: b42c9957-aa01-11e9-9b44-0242ac11000d
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: nginx
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        imagePullPolicy: Always
        name: nginx
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
status:
<omitted>

Now let us redirect the content into a file:

kubectl get deployment nginx -o yaml > first.yaml

We will use this YAML file soon to create a new Deployment

Delete Deployment

Let us test the YAML file and re-create the deployment. For that let us delete the deployment first:

kubectl delete deployment nginx

# output:
deployment.extensions "nginx" deleted

The deployments list is empty in our namespace (we will come to that later):

kubectl get deployments

# output:
No resources found.

Create Deployment with YAML File

Now we can re-create the NginX deployment using the YAML file we have created before:

kubectl create -f first.yaml 
# or: kubectl apply -f first.yaml

# output:
deployment.extensions/nginx created

Note: there are two ways to install a non-existing Kubernetes object from a YAML file: kubectl create -f or kubectl apply -f. What is the difference? The difference is as follows:

  • create:
    • if the object does not exist yet, it will be created
    • if the object exists already, there will be an error
      Error from server (AlreadyExists): error when creating "first.yaml": deployments.extensions "nginx" already exists
  • apply:
    • if the object does not exist yet, it will be created
    • if the object exists already, kubectl will try to update the existing object with the data found in the YAML file.
      Note however, that re-applying first.yaml will create an error, since it contains information from the first installation that cannot be modified on the created object. We will get an error like Error from server (Conflict): error when applying patch:
      {"metadata":{"creationTimestamp":"2019-07-19T08:46:30Z", ... for: "first.yaml": Operation cannot be fulfilled on deployments.extensions "nginx": the object has been modified; please apply your changes to the latest version and try again

Getting rid of too much Info in the YAML File

We can see with a diff command, that the newly created object differs from the old object:

kubectl get deployment nginx -o yaml > second.yaml
diff first.yaml second.yaml

# output:
6,7c6,9
<   creationTimestamp: "2019-07-19T08:46:30Z"
<   generation: 1
---
>     kubectl.kubernetes.io/last-applied-configuration: |
>       {"apiVersion":"extensions/v1beta1","kind":"Deployment","metadata":{"annotations":{"deployment.kubernetes.io/revision":"1"},"creationTimestamp":null,"generation":1,"labels":{"app":"nginx"},"name":"nginx","namespace":"default","selfLink":"/apis/extensions/v1beta1/namespaces/default/deployments/nginx"},"spec":{"progressDeadlineSeconds":600,"replicas":1,"revisionHistoryLimit":10,"selector":{"matchLabels":{"app":"nginx"}},"strategy":{"rollingUpdate":{"maxSurge":"25%","maxUnavailable":"25%"},"type":"RollingUpdate"},"template":{"metadata":{"creationTimestamp":null,"labels":{"app":"nginx"}},"spec":{"containers":[{"image":"nginx","imagePullPolicy":"Always","name":"nginx","resources":{},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File"}],"dnsPolicy":"ClusterFirst","restartPolicy":"Always","schedulerName":"default-scheduler","securityContext":{},"terminationGracePeriodSeconds":30}}},"status":{}}
>   creationTimestamp: "2019-07-19T08:59:49Z"
>   generation: 2
12c14
<   resourceVersion: "3326"
---
>   resourceVersion: "6040"
<< output omitted >>

This is the reason, why the same YAML file cannot be re-applied as shown in the note above.

Lean YAML with Export Function (deprecated)

There is an option that will produce output that cleans the output from such information. However, it is deprecated, since it is considered buggy. We test it nevertheless:

# caution: --export is deprecated. We use it nevertheless:
kubectl get deployment nginx -o yaml --export > first-export.yaml

kubectl apply -f first-export.yaml
# output: deployment.extensions/nginx configured
kubectl apply -f first-export.yaml
# output: deployment.extensions/nginx configured

Idempotence

Here, we can demonstrate that the apply function is idempotent. That is, if we apply it twice, nothing will be changed:

# idempotence:
A x A = A

Lean YAML with dry-run

Another supported possibility to create a YAML that is stripped from unnecessary information is to run a dry run creation and combine it with the output as YAML option:

kubectl delete deployment nginx
# output: deployment.extensions "nginx" deleted

kubectl create deployment nginx --image=nginx --dry-run -o yaml > nginx-dry.yaml
kubectl apply -f nginx-dry.yaml
# output: deployment.apps/nginx created

kubectl apply -f nginx-dry.yaml
# output: deployment.apps/nginx configured

kubectl apply -f nginx-dry.yaml
# output: deployment.apps/nginx configured

Here we have created a YAML file that can be applied several times with no error.

Export as JSON

APIs love JSON more than YAML in most cases. Is JSON output supported as well? Yes, of course:

kubectl get deployment nginx -o json
{
    "apiVersion": "extensions/v1beta1",
    "kind": "Deployment",
    "metadata": {
        "annotations": {
            "deployment.kubernetes.io/revision": "1",
<output omitted>

Expose Service

NginX is a little webserver. However, how can we access it? Let us try to expose a port:

kubectl expose deployment nginx

# output:
error: couldn't find port via --port flag or introspection
See 'kubectl expose -h' for help and examples

We could specify the port with the –port option, but we want to make save the port configuration persistently. For that, we can open the YAML file with the vi: vi nginx-dry.yaml and add the 3 blue lines after the container name:

   spec:
      containers:
      - image: nginx
        name: nginx
        ports:
        - containerPort: 80
          protocol: TCP
        resources: {}

Now, we replace the deployment with the one defined in the edited YAML file:

kubectl replace -f nginx-dry.yaml
# output: deployment.apps/nginx replaced

Now let us try again:

kubectl expose deployment nginx
# output: service/nginx exposed

This time, it has worked. Let us view the service:

kubectl get services

# output:
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP   72m
nginx        ClusterIP   10.110.223.199   <none>        80/TCP    48s

The first entry is a service needed by the Kubernetes API and the second one is the one we have just created. We now can access the service:

curl 10.110.223.199

# output:
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

We can see that we can access the NginX welcome page.

The same should be possible on the POD endpoint:

kubectl get ep nginx

# output:
NAME    ENDPOINTS      AGE
nginx   10.44.0.2:80   6m32s

Accessing the POD endpoint leads to the same result:

curl 10.44.0.2:80

# output:
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Note that the service is using private ClusterIP addresses that are reachable only from within the Kubernetes cluster. To access the service from outside, you need to take other measures that are handled in a different lab:

  • quick and dirty: kubeclt port-forward service nginx from any device that has kubectl installed and configured to access your Kubernetes cluster
  • permanent: install a Kubernetes Ingress solution based on NginX or Traefik
  • optional: configure a cloud load balancer to access the cluster service or the Kubernetes Ingress

TODO: LINKS TO BE PROVIDED HERE; however, for now, you can also have a look at this blog post, which shows how to make a service available via a NodePort configuration. NodePort is not recommended, so it is better to use an ingress-based solution like found in the first paragraphs here.

Scaling your Deployment

Deployments create ReplicaSets and replicasets can be scaled. Let us try this out now:

…via command line

We will scale the application to three PODs:

kubectl scale deployment nginx --replicas=3
# output: deployment.extensions/nginx scaled

kubectl get deployment
# output:
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
nginx   3/3     3            3           7m51s

kubectl get pod
# output:
NAME                     READY   STATUS    RESTARTS   AGE
nginx-56db997f77-bf2kk   1/1     Running   0          17s
nginx-56db997f77-fvg5n   1/1     Running   0          4m50s
nginx-56db997f77-vxd6n   1/1     Running   0          17s

…via File

We just can edit the spec section of the deployment YAML file:

# nginy-dry.yaml
...
spec:
  replicas: 3

and apply it with

kubectl apply -f nginy-dry.yaml

# output: deployment.apps/nginx configured

The full file we are using can be found below as a reference:

---
# file: nginx-dry.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: nginx
  name: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        name: nginx
        ports:
        - containerPort: 80
          protocol: TCP
        resources: {}
status: {}

View Endpoints for more than one POD

Now, we can see three endpoints for the same service:

kubectl get ep nginx

# output:
NAME ENDPOINTS AGE
nginx 10.44.0.1:80,10.44.0.2:80,10.44.0.3:80 24m

Replicasets and Automatic Restarts

Under the hood, the Deployment has created a Replicaset:

kubectl get replicasets.apps

# output:
NAME               DESIRED   CURRENT   READY   AGE
nginx-56db997f77   3         3         3       12m

Let us delete one of the PODs now:

kubectl get pods

# output:
NAME                     READY   STATUS    RESTARTS   AGE
nginx-56db997f77-f579s   1/1     Running   0          7s
nginx-56db997f77-gwnjt   1/1     Running   0          7s
nginx-56db997f77-rl9wv   1/1     Running   0          15m

kubectl delete pod nginx-56db997f77-rl9wv
# output: pod "nginx-56db997f77-rl9wv" deleted

kubectl get pods

# output: 
NAME                     READY   STATUS    RESTARTS   AGE
nginx-56db997f77-f579s   1/1     Running   0          113s
nginx-56db997f77-gwnjt   1/1     Running   0          113s
nginx-56db997f77-xr7rg   1/1     Running   0          38s

We have deleted the oldest POD, but a new POD has been spun up immediately.

Let us review the details of the ReplicaSet. The name of the ReplicaSet can be extracted with jq, which is installed in Kodecata Kubernetes Playground:

RS=$(k get rs -o json | jq -r '.items[0].metadata.name') \
  && kubectl describe rs $RS

# output
Name:           nginx-56db997f77
Namespace:      default
Selector:       app=nginx,pod-template-hash=56db997f77
Labels:         app=nginx
                pod-template-hash=56db997f77
Annotations:    deployment.kubernetes.io/desired-replicas: 3
                deployment.kubernetes.io/max-replicas: 4
                deployment.kubernetes.io/revision: 1
Controlled By:  Deployment/nginx
Replicas:       3 current / 3 desired
Pods Status:    3 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:  app=nginx
           pod-template-hash=56db997f77
  Containers:
   nginx:
    Image:        nginx
    Port:         80/TCP
    Host Port:    0/TCP
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Events:
  Type    Reason            Age    From                   Message
  ----    ------            ----   ----                   -------
  Normal  SuccessfulCreate  30m    replicaset-controller  Created pod: nginx-56db997f77-k264n
  Normal  SuccessfulCreate  30m    replicaset-controller  Created pod: nginx-56db997f77-7n8dj
  Normal  SuccessfulCreate  30m    replicaset-controller  Created pod: nginx-56db997f77-rl9wv
  Normal  SuccessfulDelete  26m    replicaset-controller  Deleted pod: nginx-56db997f77-k264n
  Normal  SuccessfulDelete  26m    replicaset-controller  Deleted pod: nginx-56db997f77-7n8dj
  Normal  SuccessfulCreate  26m    replicaset-controller  Created pod: nginx-56db997f77-fnglx
  Normal  SuccessfulCreate  26m    replicaset-controller  Created pod: nginx-56db997f77-pbs4m
  Normal  SuccessfulDelete  15m    replicaset-controller  Deleted pod: nginx-56db997f77-pbs4m
  Normal  SuccessfulDelete  15m    replicaset-controller  Deleted pod: nginx-56db997f77-fnglx
  Normal  SuccessfulCreate  15m    replicaset-controller  Created pod: nginx-56db997f77-f579s
  Normal  SuccessfulCreate  15m    replicaset-controller  Created pod: nginx-56db997f77-gwnjt
  Normal  SuccessfulCreate  14m    replicaset-controller  Created pod: nginx-56db997f77-xr7rg
  Normal  SuccessfulCreate  4m55s  replicaset-controller  Created pod: nginx-56db997f77-p5mqv
  Normal  SuccessfulCreate  3m10s  replicaset-controller  Created pod: nginx-56db997f77-6548k

It is telling us that the ReplicaSet is controlled by the Deployment named „nginx“, which is no wonder since this ReplicaSet was created automatically by this Deployment. The number of desired replicas is set to 3.

Accessing the Service from outside the Cluster

Above, we have pointed out that the ClusterIP, as well as the endpoint IP addresses, are only accessible from within the Kubernets Cluster. This is nice for Kubernetes Cluster admins, but how about the rest of the world?

Port-Forwarding

This method is designed for temporary access from a system that has kubectl installed and is configured to access the Kubernetes Cluster. This cannot be done easily on a Katacoda system, but we can kind of simulate it.

Install kubectl

On the system, you want to access the cluster from, install kubectl. This can also be done on a Windows system. However, we will run our commands on the worker node, where kubectl is installed already.

Configure kubectl

After the installation, kubectl does not know, how to access the Kubernetes API of the cluster:

node01 $ kubectl get nodes

# output: The connection to the server localhost:8080 was refused - did you specify the right host or port?

The default configuration is found on ~/.kube/config. We will find the correct configuration on the master node

master $ cat ~/.kube/config

The output must be copied and pasted to ~/.kube/config on the worker node.

mkdir ~/.kube
vi ~/.kube/config
# copy and paste content from the corresponding file of the master 

node01 $ kubectl get nodes

# output:
NAME     STATUS   ROLES    AGE   VERSION
master   Ready    master   59m   v1.14.0
node01   Ready    <none>   59m   v1.14.0

The command now works on the kubectl client. Now let us create a port-forwarding:

# --address is only needed since we want to allow other machines to make use of this port-forward:

node01 $ kubectl port-forward svc/nginx 8888:80 --address 0.0.0.0
Forwarding from 127.0.0.1:8888 -> 80
Forwarding from [::1]:8888 -> 80
< the system seems to hang here, that is normal, since port-forward is run in the foreground >

The –address option is only needed since we will access the port from the master node since the terminal of node01 is blocked by the forwarding command.

Note: the service must be specified in the „svc/<servicename>“ notation. „service <servicename> is not supported here.

Now, we can access the service from the master using the worker node’s forwarding connection:

master $ curl node01:8888
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
< output omitted here .... >

In a real-world example with kubectl running on a developer’s or administrator’s notebook the –address option is not needed and you can access the service with

curl 127.0.0.1:8888

Load Balancer a Cloud System

We cannot test it on Katacoda, but for those, who have installed the Kubernetes Cluster in an Infrastructure-cloud like GCE, AWS, Azure are well off. Accessing the service is as simple as replacing the service type from ClusterIP to type LoadBalancer. The cloud provider will do the rest for you:

kubectl delete svc nginx
# output: service "nginx" deleted

kubectl expose deployment nginx --type=LoadBalancer
# output: 
service/nginx exposed
master $ kubectl get svc
NAME         TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP      10.96.0.1        <none>        443/TCP        35m
nginx        LoadBalancer   10.108.206.206   <pending>     80:31521/TCP   7s

If you are performing the tests on Katacoda and not on a cloud system, then the EXTERNAL-IP will remain <pending> and access from outside is not possible in this way. However, there are some alternatives below.

Other Possibilities not shown here

  • type=NodePort instead of type=ClusterIP or LoadBalander will open a high port on the node the service is running on. See the example on our hello kubernetes post. Drawback: if there is more than one worker node, you never know, which IP address to contact.
  • Install an Ingress. See e.g. here for minikube or here for Kubernetes clusters.

Next

Kubernetes Resource Management

One comment

Comments

Diese Website verwendet Akismet, um Spam zu reduzieren. Erfahre mehr darüber, wie deine Kommentardaten verarbeitet werden.