In part 3 of the Certified Kubernetes Administrator Labs Challenge, we will deploy a simple application via command line as well as by applying a YAML file. Then we will expose and access the service from within the Kubernetes Cluster. Now we will explore how Kubernetes Deployments helps us maintain the service by automatically restarting failed PODS. Last, but not least, we will discuss how to access the service from outside the Kubernetes cluster.
Note: LFS458 starts with the installation of a Kubernetes cluster. We will skip this part for now and we will make use of the Katakoda Kubernetes playground instead. If you want to install a cluster, you might want to check out our blog post on kubeadm-based cluster installation or any other resource in the internet. Please use kubeadm and create a two node cluster with a master and a worker node.
Create Deployment per CLI command
This is the simplest way of creating an app on Kubernetes,m even if it is not the recommended one. However, later, we will learn better ways. Those will involve YAML files. However, for now, let us start with the simple command:
kubectl create deployment nginx --image=nginx # output: deployment.apps/nginx created
Show Deployments
Show all deployments:
kubectl get deployments # output NAME READY UP-TO-DATE AVAILABLE AGE nginx 1/1 1 1 52s
Describe Details
Describe all the details of the deployment:
master $ kubectl describe deployment nginx
# output:
Name: nginx
Namespace: default
CreationTimestamp: Fri, 19 Jul 2019 08:22:58 +0000
Labels: app=nginx
Annotations: deployment.kubernetes.io/revision: 1
Selector: app=nginx
Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: app=nginx
Containers:
nginx:
Image: nginx
Port: <none>
Host Port: <none>
Environment: <none>
Mounts: <none>
Volumes: <none>
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
Progressing True NewReplicaSetAvailable
OldReplicaSets: <none>
NewReplicaSet: nginx-65f88748fd (1/1 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 3m22s deployment-controller Scaled up replica set nginx-65f88748fd to 1Show Events
Can we get a list of events on the cluster? This is, how:
kubectl get events # output: 4m32s Normal Scheduled pod/nginx-65f88748fd-5dp7b Successfully assigned default/nginx-65f88748fd-5dp7b to node01 4m31s Normal Pulling pod/nginx-65f88748fd-5dp7b Pulling image "nginx" 4m24s Normal Pulled pod/nginx-65f88748fd-5dp7b Successfully pulled image "nginx" 4m24s Normal Created pod/nginx-65f88748fd-5dp7b Created container nginx 4m24s Normal Started pod/nginx-65f88748fd-5dp7b Started container nginx 4m32s Normal SuccessfulCreate replicaset/nginx-65f88748fd Created pod: nginx-65f88748fd-5dp7b 4m32s Normal ScalingReplicaSet deployment/nginx Scaled up replica set nginx-65f88748fd to 1
You can see that creating a deployment is performing many steps. They seem to be somehow unordered in time. Interesting.
Deployments are one of the most important objects of Kubernetes, but they consist of many other, simpler objects, which we will explore in more details below:
- Docker container
- POD consists of one or more Docker containers
- Replicaset makes sure that x replicas of PODs are always up and running
- Deployment is a Replicaset with more features like rolling updates of PODs etc.
So, when we have created a Deployment, the following things happen:
- The images for the containers defined in the POD are pulled if they are not already running on the worker node.
- A scheduler looks for a worker node the PODs can be run on. Thus us node01 in our case, as can be seen from the event log. Note, that different Containers of a POD must be located on the same worker node, i.e. they may not be distributed among different worker nodes.
- The Scheduler distributes the desired number of POD replicas evenly among worker nodes. That is, replicas of the PODs can be run on different worker nodes.
Export Kubernetes Objects as YAML File
Above, we have stated that there are better, recommended ways of creating Kubernetes deployments using YAML files. We have not created our deployment using a YAML file yet, but we can export the existing Deployment as a YAML file:
kubectl get deployment nginx -o yaml
# output:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
creationTimestamp: "2019-07-19T08:46:30Z"
generation: 1
labels:
app: nginx
name: nginx
namespace: default
resourceVersion: "3326"
selfLink: /apis/extensions/v1beta1/namespaces/default/deployments/nginx
uid: b42c9957-aa01-11e9-9b44-0242ac11000d
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: nginx
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: nginx
spec:
containers:
- image: nginx
imagePullPolicy: Always
name: nginx
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
status:
<omitted>
Now let us redirect the content into a file:
kubectl get deployment nginx -o yaml > first.yaml
We will use this YAML file soon to create a new Deployment
Delete Deployment
Let us test the YAML file and re-create the deployment. For that let us delete the deployment first:
kubectl delete deployment nginx # output: deployment.extensions "nginx" deleted
The deployments list is empty in our namespace (we will come to that later):
kubectl get deployments # output: No resources found.
Create Deployment with YAML File
Now we can re-create the NginX deployment using the YAML file we have created before:
kubectl create -f first.yaml # or: kubectl apply -f first.yaml # output: deployment.extensions/nginx created
Note: there are two ways to install a non-existing Kubernetes object from a YAML file: kubectl create -f or kubectl apply -f. What is the difference? The difference is as follows:
- create:
- if the object does not exist yet, it will be created
- if the object exists already, there will be an error
Error from server (AlreadyExists): error when creating "first.yaml": deployments.extensions "nginx" already exists- apply:
- if the object does not exist yet, it will be created
- if the object exists already, kubectl will try to update the existing object with the data found in the YAML file.
Note however, that re-applying first.yaml will create an error, since it contains information from the first installation that cannot be modified on the created object. We will get an error likeError from server (Conflict): error when applying patch:
{"metadata":{"creationTimestamp":"2019-07-19T08:46:30Z", ... for: "first.yaml": Operation cannot be fulfilled on deployments.extensions "nginx": the object has been modified; please apply your changes to the latest version and try again
Getting rid of too much Info in the YAML File
We can see with a diff command, that the newly created object differs from the old object:
kubectl get deployment nginx -o yaml > second.yaml
diff first.yaml second.yaml
# output:
6,7c6,9
< creationTimestamp: "2019-07-19T08:46:30Z"
< generation: 1
---
> kubectl.kubernetes.io/last-applied-configuration: |
> {"apiVersion":"extensions/v1beta1","kind":"Deployment","metadata":{"annotations":{"deployment.kubernetes.io/revision":"1"},"creationTimestamp":null,"generation":1,"labels":{"app":"nginx"},"name":"nginx","namespace":"default","selfLink":"/apis/extensions/v1beta1/namespaces/default/deployments/nginx"},"spec":{"progressDeadlineSeconds":600,"replicas":1,"revisionHistoryLimit":10,"selector":{"matchLabels":{"app":"nginx"}},"strategy":{"rollingUpdate":{"maxSurge":"25%","maxUnavailable":"25%"},"type":"RollingUpdate"},"template":{"metadata":{"creationTimestamp":null,"labels":{"app":"nginx"}},"spec":{"containers":[{"image":"nginx","imagePullPolicy":"Always","name":"nginx","resources":{},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File"}],"dnsPolicy":"ClusterFirst","restartPolicy":"Always","schedulerName":"default-scheduler","securityContext":{},"terminationGracePeriodSeconds":30}}},"status":{}}
> creationTimestamp: "2019-07-19T08:59:49Z"
> generation: 2
12c14
< resourceVersion: "3326"
---
> resourceVersion: "6040"
<< output omitted >>This is the reason, why the same YAML file cannot be re-applied as shown in the note above.
Lean YAML with Export Function (deprecated)
There is an option that will produce output that cleans the output from such information. However, it is deprecated, since it is considered buggy. We test it nevertheless:
# caution: --export is deprecated. We use it nevertheless: kubectl get deployment nginx -o yaml --export > first-export.yaml kubectl apply -f first-export.yaml # output: deployment.extensions/nginx configured kubectl apply -f first-export.yaml # output: deployment.extensions/nginx configured
Idempotence
Here, we can demonstrate that the apply function is idempotent. That is, if we apply it twice, nothing will be changed:
# idempotence: A x A = A
Lean YAML with dry-run
Another supported possibility to create a YAML that is stripped from unnecessary information is to run a dry run creation and combine it with the output as YAML option:
kubectl delete deployment nginx # output: deployment.extensions "nginx" deleted kubectl create deployment nginx --image=nginx --dry-run -o yaml > nginx-dry.yaml kubectl apply -f nginx-dry.yaml # output: deployment.apps/nginx created kubectl apply -f nginx-dry.yaml # output: deployment.apps/nginx configured kubectl apply -f nginx-dry.yaml # output: deployment.apps/nginx configured
Here we have created a YAML file that can be applied several times with no error.
Export as JSON
APIs love JSON more than YAML in most cases. Is JSON output supported as well? Yes, of course:
kubectl get deployment nginx -o json
{
"apiVersion": "extensions/v1beta1",
"kind": "Deployment",
"metadata": {
"annotations": {
"deployment.kubernetes.io/revision": "1",
<output omitted>Expose Service
NginX is a little webserver. However, how can we access it? Let us try to expose a port:
kubectl expose deployment nginx # output: error: couldn't find port via --port flag or introspection See 'kubectl expose -h' for help and examples
We could specify the port with the –port option, but we want to make save the port configuration persistently. For that, we can open the YAML file with the vi: vi nginx-dry.yaml and add the 3 blue lines after the container name:
spec:
containers:
- image: nginx
name: nginx
ports:
- containerPort: 80
protocol: TCP
resources: {}Now, we replace the deployment with the one defined in the edited YAML file:
kubectl replace -f nginx-dry.yaml # output: deployment.apps/nginx replaced
Now let us try again:
kubectl expose deployment nginx # output: service/nginx exposed
This time, it has worked. Let us view the service:
kubectl get services # output: NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 72m nginx ClusterIP 10.110.223.199 <none> 80/TCP 48s
The first entry is a service needed by the Kubernetes API and the second one is the one we have just created. We now can access the service:
curl 10.110.223.199
# output:
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
We can see that we can access the NginX welcome page.
The same should be possible on the POD endpoint:
kubectl get ep nginx # output: NAME ENDPOINTS AGE nginx 10.44.0.2:80 6m32s
Accessing the POD endpoint leads to the same result:
curl 10.44.0.2:80
# output:
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
Note that the service is using private ClusterIP addresses that are reachable only from within the Kubernetes cluster. To access the service from outside, you need to take other measures that are handled in a different lab:
- quick and dirty: kubeclt port-forward service nginx from any device that has kubectl installed and configured to access your Kubernetes cluster
- permanent: install a Kubernetes Ingress solution based on NginX or Traefik
- optional: configure a cloud load balancer to access the cluster service or the Kubernetes Ingress
TODO: LINKS TO BE PROVIDED HERE; however, for now, you can also have a look at this blog post, which shows how to make a service available via a NodePort configuration. NodePort is not recommended, so it is better to use an ingress-based solution like found in the first paragraphs here.
Scaling your Deployment
Deployments create ReplicaSets and replicasets can be scaled. Let us try this out now:
…via command line
We will scale the application to three PODs:
kubectl scale deployment nginx --replicas=3 # output: deployment.extensions/nginx scaled kubectl get deployment # output: NAME READY UP-TO-DATE AVAILABLE AGE nginx 3/3 3 3 7m51s kubectl get pod # output: NAME READY STATUS RESTARTS AGE nginx-56db997f77-bf2kk 1/1 Running 0 17s nginx-56db997f77-fvg5n 1/1 Running 0 4m50s nginx-56db997f77-vxd6n 1/1 Running 0 17s
…via File
We just can edit the spec section of the deployment YAML file:
# nginy-dry.yaml ... spec: replicas: 3
and apply it with
kubectl apply -f nginy-dry.yaml # output: deployment.apps/nginx configured
The full file we are using can be found below as a reference:
---
# file: nginx-dry.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: nginx
name: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: nginx
spec:
containers:
- image: nginx
name: nginx
ports:
- containerPort: 80
protocol: TCP
resources: {}
status: {}View Endpoints for more than one POD
Now, we can see three endpoints for the same service:
kubectl get ep nginx # output: NAME ENDPOINTS AGE nginx 10.44.0.1:80,10.44.0.2:80,10.44.0.3:80 24m
Replicasets and Automatic Restarts
Under the hood, the Deployment has created a Replicaset:
kubectl get replicasets.apps # output: NAME DESIRED CURRENT READY AGE nginx-56db997f77 3 3 3 12m
Let us delete one of the PODs now:
kubectl get pods # output: NAME READY STATUS RESTARTS AGE nginx-56db997f77-f579s 1/1 Running 0 7s nginx-56db997f77-gwnjt 1/1 Running 0 7s nginx-56db997f77-rl9wv 1/1 Running 0 15m kubectl delete pod nginx-56db997f77-rl9wv # output: pod "nginx-56db997f77-rl9wv" deleted kubectl get pods # output: NAME READY STATUS RESTARTS AGE nginx-56db997f77-f579s 1/1 Running 0 113s nginx-56db997f77-gwnjt 1/1 Running 0 113s nginx-56db997f77-xr7rg 1/1 Running 0 38s
We have deleted the oldest POD, but a new POD has been spun up immediately.
Let us review the details of the ReplicaSet. The name of the ReplicaSet can be extracted with jq, which is installed in Kodecata Kubernetes Playground:
RS=$(k get rs -o json | jq -r '.items[0].metadata.name') \
&& kubectl describe rs $RS
# output
Name: nginx-56db997f77
Namespace: default
Selector: app=nginx,pod-template-hash=56db997f77
Labels: app=nginx
pod-template-hash=56db997f77
Annotations: deployment.kubernetes.io/desired-replicas: 3
deployment.kubernetes.io/max-replicas: 4
deployment.kubernetes.io/revision: 1
Controlled By: Deployment/nginx
Replicas: 3 current / 3 desired
Pods Status: 3 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
Labels: app=nginx
pod-template-hash=56db997f77
Containers:
nginx:
Image: nginx
Port: 80/TCP
Host Port: 0/TCP
Environment: <none>
Mounts: <none>
Volumes: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 30m replicaset-controller Created pod: nginx-56db997f77-k264n
Normal SuccessfulCreate 30m replicaset-controller Created pod: nginx-56db997f77-7n8dj
Normal SuccessfulCreate 30m replicaset-controller Created pod: nginx-56db997f77-rl9wv
Normal SuccessfulDelete 26m replicaset-controller Deleted pod: nginx-56db997f77-k264n
Normal SuccessfulDelete 26m replicaset-controller Deleted pod: nginx-56db997f77-7n8dj
Normal SuccessfulCreate 26m replicaset-controller Created pod: nginx-56db997f77-fnglx
Normal SuccessfulCreate 26m replicaset-controller Created pod: nginx-56db997f77-pbs4m
Normal SuccessfulDelete 15m replicaset-controller Deleted pod: nginx-56db997f77-pbs4m
Normal SuccessfulDelete 15m replicaset-controller Deleted pod: nginx-56db997f77-fnglx
Normal SuccessfulCreate 15m replicaset-controller Created pod: nginx-56db997f77-f579s
Normal SuccessfulCreate 15m replicaset-controller Created pod: nginx-56db997f77-gwnjt
Normal SuccessfulCreate 14m replicaset-controller Created pod: nginx-56db997f77-xr7rg
Normal SuccessfulCreate 4m55s replicaset-controller Created pod: nginx-56db997f77-p5mqv
Normal SuccessfulCreate 3m10s replicaset-controller Created pod: nginx-56db997f77-6548kIt is telling us that the ReplicaSet is controlled by the Deployment named „nginx“, which is no wonder since this ReplicaSet was created automatically by this Deployment. The number of desired replicas is set to 3.
Accessing the Service from outside the Cluster
Above, we have pointed out that the ClusterIP, as well as the endpoint IP addresses, are only accessible from within the Kubernets Cluster. This is nice for Kubernetes Cluster admins, but how about the rest of the world?
Port-Forwarding
This method is designed for temporary access from a system that has kubectl installed and is configured to access the Kubernetes Cluster. This cannot be done easily on a Katacoda system, but we can kind of simulate it.
Install kubectl
On the system, you want to access the cluster from, install kubectl. This can also be done on a Windows system. However, we will run our commands on the worker node, where kubectl is installed already.
Configure kubectl
After the installation, kubectl does not know, how to access the Kubernetes API of the cluster:
node01 $ kubectl get nodes # output: The connection to the server localhost:8080 was refused - did you specify the right host or port?
The default configuration is found on ~/.kube/config. We will find the correct configuration on the master node
master $ cat ~/.kube/config
The output must be copied and pasted to ~/.kube/config on the worker node.
mkdir ~/.kube vi ~/.kube/config # copy and paste content from the corresponding file of the master node01 $ kubectl get nodes # output: NAME STATUS ROLES AGE VERSION master Ready master 59m v1.14.0 node01 Ready <none> 59m v1.14.0
The command now works on the kubectl client. Now let us create a port-forwarding:
# --address is only needed since we want to allow other machines to make use of this port-forward: node01 $ kubectl port-forward svc/nginx 8888:80 --address 0.0.0.0 Forwarding from 127.0.0.1:8888 -> 80 Forwarding from [::1]:8888 -> 80 < the system seems to hang here, that is normal, since port-forward is run in the foreground >
The –address option is only needed since we will access the port from the master node since the terminal of node01 is blocked by the forwarding command.
Note: the service must be specified in the „svc/<servicename>“ notation. „service <servicename> is not supported here.
Now, we can access the service from the master using the worker node’s forwarding connection:
master $ curl node01:8888
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
< output omitted here .... >In a real-world example with kubectl running on a developer’s or administrator’s notebook the –address option is not needed and you can access the service with
curl 127.0.0.1:8888
Load Balancer a Cloud System
We cannot test it on Katacoda, but for those, who have installed the Kubernetes Cluster in an Infrastructure-cloud like GCE, AWS, Azure are well off. Accessing the service is as simple as replacing the service type from ClusterIP to type LoadBalancer. The cloud provider will do the rest for you:
kubectl delete svc nginx # output: service "nginx" deleted kubectl expose deployment nginx --type=LoadBalancer # output: service/nginx exposed master $ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 35m nginx LoadBalancer 10.108.206.206 <pending> 80:31521/TCP 7s
If you are performing the tests on Katacoda and not on a cloud system, then the EXTERNAL-IP will remain <pending> and access from outside is not possible in this way. However, there are some alternatives below.
Other Possibilities not shown here
- type=NodePort instead of type=ClusterIP or LoadBalander will open a high port on the node the service is running on. See the example on our hello kubernetes post. Drawback: if there is more than one worker node, you never know, which IP address to contact.
- Install an Ingress. See e.g. here for minikube or here for Kubernetes clusters.
Can you be more specific about the content of your article? After reading it, I still have some doubts. Hope you can help me.