This is a cheat sheet on how to perform backup&restore of the etcd server in kubernetes quickly.

Test this on

tl;dr

Find reference: https://kubernetes.io –> Documentation –> Search „etcd backup restore“

–> you will find: Operating etcd clusters for Kubernetes | Kubernetes

# get params
cat /var/lib/kubelet/config.yaml | grep static
cat /etc/kubernetes/manifests/etcd.yaml | grep -A 30 command | egrep '^ *-'

# create test pod (optional)
k run test --image nginx
k get pod

# backup (get params from output above):
ETCDCTL_API=3 etcdctl --endpoints=https://172.30.1.2:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save snapshotdb

# delete test pod (optional)
k delete pod test
k get pod

# stop kube-apiserver and etcd server:
mv /etc/kubernetes/manifests/kube-apiserver.yaml ./;
mv /etc/kubernetes/manifests/etcd.yaml ./;

# restore:
rm -rf /var/lib/etcd;
ETCDCTL_API=3 etcdctl snapshot restore --data-dir=/var/lib/etcd snapshotdb

# start etcd server and kube-apiserver:
mv etcd.yaml /etc/kubernetes/manifests/;
mv kube-apiserver.yaml /etc/kubernetes/manifests/

# check (optional; wait for 2 minutes or so to allow etcd and kube-apiser to get up)
k get pod

Step 1 (optional): Create a test POD

k run test-pod --image nginx
k get pod

# output (if you wait a minute or so):
NAME READY STATUS RESTARTS AGE
test 1/1 Running 0 3m40s

Step 2: Get parameters from the etcd.yaml file

cat /var/lib/kubelet/config.yaml | grep static 

# output:
staticPodPath: /etc/kubernetes/manifests

Use the staticPodPath to find the POD manifests for etcd (and kube-apiserver further down below):

cat /etc/kubernetes/manifests/etcd.yaml | grep -A 30 command | egrep '^ *-'

# output:
 - command:
- etcd
- --advertise-client-urls=https://172.30.1.2:2379
- --cert-file=/etc/kubernetes/pki/etcd/server.crt <----------------- cert-file
- --client-cert-auth=true
- --data-dir=/var/lib/etcd
- --experimental-initial-corrupt-check=true
- --experimental-watch-progress-notify-interval=5s
- --initial-advertise-peer-urls=https://172.30.1.2:2380
- --initial-cluster=controlplane=https://172.30.1.2:2380
- --key-file=/etc/kubernetes/pki/etcd/server.key <-------------------- key-file
- --listen-client-urls=https://127.0.0.1:2379,https://172.30.1.2:2379  <--- endpoints
- --listen-metrics-urls=http://127.0.0.1:2381
- --listen-peer-urls=https://172.30.1.2:2380
- --name=controlplane
- --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt
- --peer-client-cert-auth=true
- --peer-key-file=/etc/kubernetes/pki/etcd/peer.key
- --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt <-------- trusted-ca-file
- --snapshot-count=10000
- --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt

Step 3: Create an etcd Backup

You need to replace the endpoints, cacert, cert and key by the parameters found in step 2:

ETCDCTL_API=3 etcdctl --endpoints=https://172.30.1.2:2379   --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key   snapshot save etcd-backup

# output:
{"level":"info","ts":1669193343.0635,"caller":"snapshot/v3_snapshot.go:68","msg":"created temporary db file","path":"snapshotdb.part"}
{"level":"info","ts":1669193343.0680363,"logger":"client","caller":"v3/maintenance.go:211","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":1669193343.068161,"caller":"snapshot/v3_snapshot.go:76","msg":"fetching snapshot","endpoint":"https://172.30.1.2:2379"}
{"level":"info","ts":1669193343.1282105,"logger":"client","caller":"v3/maintenance.go:219","msg":"completed snapshot read; closing"}
{"level":"info","ts":1669193343.1477027,"caller":"snapshot/v3_snapshot.go:91","msg":"fetched snapshot","endpoint":"https://172.30.1.2:2379","size":"5.7 MB","took":"now"}
{"level":"info","ts":1669193343.148174,"caller":"snapshot/v3_snapshot.go:100","msg":"saved","path":"snapshotdb"}
Snapshot saved at snapshotdb

Step 4 (optional): Delete the test pod

k delete pod test

# output: pod "test" deleted

Step 5: Stop kube-apiserver and etcd

The kube-apiserver and the etcd server are static PODs. They are automatically stopped by moving them from the manifests directory:

mv /etc/kubernetes/manifests/kube-apiserver.yaml ./
mv /etc/kubernetes/manifests/etcd.yaml ./

Step 6: Restore etcd

Before we can restore the *etcd database, we need to remove the existing database from the data-dir path:

rm -rf /var/lib/etcd

We now restore the etcd database from the previously created snapshotdb file:

ETCDCTL_API=3 etcdctl snapshot restore --data-dir=/var/lib/etcd snapshotdb 

# output:
Deprecated: Use `etcdutl snapshot restore` instead.

2022-11-23T07:34:00Z    info    snapshot/v3_snapshot.go:251     restoring snapshot      {"path": "savedb", "wal-dir": "/var/lib/etcd/member/wal", "data-dir": "/var/lib/etcd", "snap-dir": "/var/lib/etcd/member/snap", "stack": "go.etcd.io/etcd/etcdutl/v3/snapshot.(*v3Manager).Restore\n\t/tmp/etcd-release-3.5.0/etcd/release/etcd/etcdutl/snapshot/v3_snapshot.go:257\ngo.etcd.io/etcd/etcdutl/v3/etcdutl.SnapshotRestoreCommandFunc\n\t/tmp/etcd-release-3.5.0/etcd/release/etcd/etcdutl/etcdutl/snapshot_command.go:147\ngo.etcd.io/etcd/etcdctl/v3/ctlv3/command.snapshotRestoreCommandFunc\n\t/tmp/etcd-release-3.5.0/etcd/release/etcd/etcdctl/ctlv3/command/snapshot_command.go:128\ngithub.com/spf13/cobra.(*Command).execute\n\t/home/remote/sbatsche/.gvm/pkgsets/go1.16.3/global/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:856\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/home/remote/sbatsche/.gvm/pkgsets/go1.16.3/global/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:960\ngithub.com/spf13/cobra.(*Command).Execute\n\t/home/remote/sbatsche/.gvm/pkgsets/go1.16.3/global/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:897\ngo.etcd.io/etcd/etcdctl/v3/ctlv3.Start\n\t/tmp/etcd-release-3.5.0/etcd/release/etcd/etcdctl/ctlv3/ctl.go:107\ngo.etcd.io/etcd/etcdctl/v3/ctlv3.MustStart\n\t/tmp/etcd-release-3.5.0/etcd/release/etcd/etcdctl/ctlv3/ctl.go:111\nmain.main\n\t/tmp/etcd-release-3.5.0/etcd/release/etcd/etcdctl/main.go:59\nruntime.main\n\t/home/remote/sbatsche/.gvm/gos/go1.16.3/src/runtime/proc.go:225"}
2022-11-23T07:34:00Z    info    membership/store.go:119 Trimming membership information from the backend...
2022-11-23T07:34:00Z    info    membership/cluster.go:393       added member    {"cluster-id": "cdf818194e3a8c32", "local-member-id": "0", "added-peer-id": "8e9e05c52164694d", "added-peer-peer-urls": ["http://localhost:2380"]}
2022-11-23T07:34:00Z    info    snapshot/v3_snapshot.go:272     restored snapshot       {"path": "savedb", "wal-dir": "/var/lib/etcd/member/wal", "data-dir": "/var/lib/etcd", "snap-dir": "/var/lib/etcd/member/snap"}

Step 7: Start etcd and kube-apiserver

The etcd and kube-apiserver are automatically started, if the corresponding YAML files are moved back to the manifests directory:

mv etcd.yaml /etc/kubernetes/manifests/ ;
mv kube-apiserver.yaml /etc/kubernetes/manifests/

Step 8 (optional): Check the status of the test POD

When the restore was successful, you should see the test POD up and running again:

k get pod

# output, if you do not wait long enough:
The connection to the server 172.30.1.2:6443 was refused - did you specify the right host or port?

# output after 2 minutes or so:
NAME   READY   STATUS    RESTARTS   AGE
test   1/1     Running   0          4m49s

References

30 comments

  1. Fantastic beat I would like to apprentice while you amend your web site how could i subscribe for a blog site The account helped me a acceptable deal I had been a little bit acquainted of this your broadcast offered bright clear concept

  2. of course like your website but you have to check the spelling on several of your posts A number of them are rife with spelling issues and I in finding it very troublesome to inform the reality on the other hand I will certainly come back again

  3. What i do not understood is in truth how you are not actually a lot more smartlyliked than you may be now You are very intelligent You realize therefore significantly in the case of this topic produced me individually imagine it from numerous numerous angles Its like men and women dont seem to be fascinated until it is one thing to do with Woman gaga Your own stuffs nice All the time care for it up

  4. certainly like your website but you need to take a look at the spelling on quite a few of your posts Many of them are rife with spelling problems and I find it very troublesome to inform the reality nevertheless I will definitely come back again

  5. My brother suggested I might like this blog He was totally right This post actually made my day You can not imagine simply how much time I had spent for this info Thanks

  6. I was recommended this website by my cousin I am not sure whether this post is written by him as nobody else know such detailed about my trouble You are amazing Thanks

  7. Thanks I have just been looking for information about this subject for a long time and yours is the best Ive discovered till now However what in regards to the bottom line Are you certain in regards to the supply

  8. Fantastic beat I would like to apprentice while you amend your web site how could i subscribe for a blog site The account helped me a acceptable deal I had been a little bit acquainted of this your broadcast offered bright clear concept

  9. ???????, ????? ???-?????? ? ??????????? ???????, ?????????? ??????????? ? ???????? ????? ??????? ? ??????. ?? ??????? ???????????? ????????, ????????? ? ?????????? ?????????? ? ?????????????. ????? ?? ?????? ????? ????? ???????? „Like a Prayer“, „Vogue“, „Material Girl“, „Into the Groove“ ? „Hung Up“. ??? ????? ?? ?????? ???????????? ?? ??????????? ??????, ?? ? ???????? ???????????? ???? ? ?????????? ? ???????????? ???????? ??????. ??????? ?? ?????? ??????, ?? ? ????? ?????, ??????? ? ???????????????, ??? ??????? ???????????? ?????? ?? ????? ??????????? ?????????. ??????? mp3 ?????? 2024 ???? ? ??????? ?????? ?????????.

  10. The unique set of AD0-E716 dumps is the easiest and the most rewarding content, you ever found on any web page. Your success is guaranteed! The questions and answers format of our dumps is rich with information and provides you also Adobe Commerce Developer with Cloud Add-on latest lab help, enhancing your exam skills. The content is approved by the most distinguished professionals and revised and updated by our experts on regular basis. With these brilliant features it is rated as the most worthwhile, informative and highly exam relevant. In all respects, you will find the AD0-E716 dumps compatible to your actual preparatory needs. The language is simple and the content is engaging and easy. No more, AD0-E716 exam is a nightmare.

  11. I became a fan of this phenomenal website earlier this week, they give informative content to visitors. The site owner has a knack for educating readers. I’m impressed and hope they keep sharing great material.

  12. I’m often to blogging and i really appreciate your content. The article has actually peaks my interest. I’m going to bookmark your web site and maintain checking for brand spanking new information.

  13. Nice blog here Also your site loads up fast What host are you using Can I get your affiliate link to your host I wish my web site loaded up as quickly as yours lol

  14. Nice blog here Also your site loads up fast What host are you using Can I get your affiliate link to your host I wish my web site loaded up as quickly as yours lol

  15. I do agree with all the ideas you have introduced on your post They are very convincing and will definitely work Still the posts are very short for newbies May just you please prolong them a little from subsequent time Thank you for the post

  16. What i do not understood is in truth how you are not actually a lot more smartlyliked than you may be now You are very intelligent You realize therefore significantly in the case of this topic produced me individually imagine it from numerous numerous angles Its like men and women dont seem to be fascinated until it is one thing to do with Woman gaga Your own stuffs nice All the time care for it up

  17. Its like you read my mind You appear to know a lot about this like you wrote the book in it or something I think that you could do with some pics to drive the message home a little bit but instead of that this is fantastic blog An excellent read I will certainly be back

  18. I was recommended this website by my cousin I am not sure whether this post is written by him as nobody else know such detailed about my trouble You are amazing Thanks

Comments

Diese Website verwendet Akismet, um Spam zu reduzieren. Erfahre mehr darüber, wie deine Kommentardaten verarbeitet werden.