This is a cheat sheet on how to perform backup&restore of the etcd server in kubernetes quickly.
Test this on
tl;dr
Find reference: https://kubernetes.io –> Documentation –> Search „etcd backup restore“
–> you will find: Operating etcd clusters for Kubernetes | Kubernetes
# get params cat /var/lib/kubelet/config.yaml | grep static cat /etc/kubernetes/manifests/etcd.yaml | grep -A 30 command | egrep '^ *-' # create test pod (optional) k run test --image nginx k get pod # backup (get params from output above): ETCDCTL_API=3 etcdctl --endpoints=https://172.30.1.2:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save snapshotdb # delete test pod (optional) k delete pod test k get pod # stop kube-apiserver and etcd server: mv /etc/kubernetes/manifests/kube-apiserver.yaml ./; mv /etc/kubernetes/manifests/etcd.yaml ./; # restore: rm -rf /var/lib/etcd; ETCDCTL_API=3 etcdctl snapshot restore --data-dir=/var/lib/etcd snapshotdb # start etcd server and kube-apiserver: mv etcd.yaml /etc/kubernetes/manifests/; mv kube-apiserver.yaml /etc/kubernetes/manifests/ # check (optional; wait for 2 minutes or so to allow etcd and kube-apiser to get up) k get pod
Step 1 (optional): Create a test POD
k run test-pod --image nginx k get pod # output (if you wait a minute or so): NAME READY STATUS RESTARTS AGE test 1/1 Running 0 3m40s
Step 2: Get parameters from the etcd.yaml file
cat /var/lib/kubelet/config.yaml | grep static # output: staticPodPath: /etc/kubernetes/manifests
Use the staticPodPath to find the POD manifests for etcd (and kube-apiserver further down below):
cat /etc/kubernetes/manifests/etcd.yaml | grep -A 30 command | egrep '^ *-' # output: - command: - etcd - --advertise-client-urls=https://172.30.1.2:2379 - --cert-file=/etc/kubernetes/pki/etcd/server.crt <----------------- cert-file - --client-cert-auth=true - --data-dir=/var/lib/etcd - --experimental-initial-corrupt-check=true - --experimental-watch-progress-notify-interval=5s - --initial-advertise-peer-urls=https://172.30.1.2:2380 - --initial-cluster=controlplane=https://172.30.1.2:2380 - --key-file=/etc/kubernetes/pki/etcd/server.key <-------------------- key-file - --listen-client-urls=https://127.0.0.1:2379,https://172.30.1.2:2379 <--- endpoints - --listen-metrics-urls=http://127.0.0.1:2381 - --listen-peer-urls=https://172.30.1.2:2380 - --name=controlplane - --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt - --peer-client-cert-auth=true - --peer-key-file=/etc/kubernetes/pki/etcd/peer.key - --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt <-------- trusted-ca-file - --snapshot-count=10000 - --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
Step 3: Create an etcd Backup
You need to replace the endpoints, cacert, cert and key by the parameters found in step 2:
ETCDCTL_API=3 etcdctl --endpoints=https://172.30.1.2:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save etcd-backup
# output:
{"level":"info","ts":1669193343.0635,"caller":"snapshot/v3_snapshot.go:68","msg":"created temporary db file","path":"snapshotdb.part"}
{"level":"info","ts":1669193343.0680363,"logger":"client","caller":"v3/maintenance.go:211","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":1669193343.068161,"caller":"snapshot/v3_snapshot.go:76","msg":"fetching snapshot","endpoint":"https://172.30.1.2:2379"}
{"level":"info","ts":1669193343.1282105,"logger":"client","caller":"v3/maintenance.go:219","msg":"completed snapshot read; closing"}
{"level":"info","ts":1669193343.1477027,"caller":"snapshot/v3_snapshot.go:91","msg":"fetched snapshot","endpoint":"https://172.30.1.2:2379","size":"5.7 MB","took":"now"}
{"level":"info","ts":1669193343.148174,"caller":"snapshot/v3_snapshot.go:100","msg":"saved","path":"snapshotdb"}
Snapshot saved at snapshotdbStep 4 (optional): Delete the test pod
k delete pod test # output: pod "test" deleted
Step 5: Stop kube-apiserver and etcd
The kube-apiserver and the etcd server are static PODs. They are automatically stopped by moving them from the manifests directory:
mv /etc/kubernetes/manifests/kube-apiserver.yaml ./ mv /etc/kubernetes/manifests/etcd.yaml ./
Step 6: Restore etcd
Before we can restore the *etcd database, we need to remove the existing database from the data-dir path:
rm -rf /var/lib/etcd
We now restore the etcd database from the previously created snapshotdb file:
ETCDCTL_API=3 etcdctl snapshot restore --data-dir=/var/lib/etcd snapshotdb
# output:
Deprecated: Use `etcdutl snapshot restore` instead.
2022-11-23T07:34:00Z info snapshot/v3_snapshot.go:251 restoring snapshot {"path": "savedb", "wal-dir": "/var/lib/etcd/member/wal", "data-dir": "/var/lib/etcd", "snap-dir": "/var/lib/etcd/member/snap", "stack": "go.etcd.io/etcd/etcdutl/v3/snapshot.(*v3Manager).Restore\n\t/tmp/etcd-release-3.5.0/etcd/release/etcd/etcdutl/snapshot/v3_snapshot.go:257\ngo.etcd.io/etcd/etcdutl/v3/etcdutl.SnapshotRestoreCommandFunc\n\t/tmp/etcd-release-3.5.0/etcd/release/etcd/etcdutl/etcdutl/snapshot_command.go:147\ngo.etcd.io/etcd/etcdctl/v3/ctlv3/command.snapshotRestoreCommandFunc\n\t/tmp/etcd-release-3.5.0/etcd/release/etcd/etcdctl/ctlv3/command/snapshot_command.go:128\ngithub.com/spf13/cobra.(*Command).execute\n\t/home/remote/sbatsche/.gvm/pkgsets/go1.16.3/global/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:856\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/home/remote/sbatsche/.gvm/pkgsets/go1.16.3/global/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:960\ngithub.com/spf13/cobra.(*Command).Execute\n\t/home/remote/sbatsche/.gvm/pkgsets/go1.16.3/global/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:897\ngo.etcd.io/etcd/etcdctl/v3/ctlv3.Start\n\t/tmp/etcd-release-3.5.0/etcd/release/etcd/etcdctl/ctlv3/ctl.go:107\ngo.etcd.io/etcd/etcdctl/v3/ctlv3.MustStart\n\t/tmp/etcd-release-3.5.0/etcd/release/etcd/etcdctl/ctlv3/ctl.go:111\nmain.main\n\t/tmp/etcd-release-3.5.0/etcd/release/etcd/etcdctl/main.go:59\nruntime.main\n\t/home/remote/sbatsche/.gvm/gos/go1.16.3/src/runtime/proc.go:225"}
2022-11-23T07:34:00Z info membership/store.go:119 Trimming membership information from the backend...
2022-11-23T07:34:00Z info membership/cluster.go:393 added member {"cluster-id": "cdf818194e3a8c32", "local-member-id": "0", "added-peer-id": "8e9e05c52164694d", "added-peer-peer-urls": ["http://localhost:2380"]}
2022-11-23T07:34:00Z info snapshot/v3_snapshot.go:272 restored snapshot {"path": "savedb", "wal-dir": "/var/lib/etcd/member/wal", "data-dir": "/var/lib/etcd", "snap-dir": "/var/lib/etcd/member/snap"}
Step 7: Start etcd and kube-apiserver
The etcd and kube-apiserver are automatically started, if the corresponding YAML files are moved back to the manifests directory:
mv etcd.yaml /etc/kubernetes/manifests/ ; mv kube-apiserver.yaml /etc/kubernetes/manifests/
Step 8 (optional): Check the status of the test POD
When the restore was successful, you should see the test POD up and running again:
k get pod # output, if you do not wait long enough: The connection to the server 172.30.1.2:6443 was refused - did you specify the right host or port? # output after 2 minutes or so: NAME READY STATUS RESTARTS AGE test 1/1 Running 0 4m49s
Fantastic beat I would like to apprentice while you amend your web site how could i subscribe for a blog site The account helped me a acceptable deal I had been a little bit acquainted of this your broadcast offered bright clear concept
of course like your website but you have to check the spelling on several of your posts A number of them are rife with spelling issues and I in finding it very troublesome to inform the reality on the other hand I will certainly come back again
What i do not understood is in truth how you are not actually a lot more smartlyliked than you may be now You are very intelligent You realize therefore significantly in the case of this topic produced me individually imagine it from numerous numerous angles Its like men and women dont seem to be fascinated until it is one thing to do with Woman gaga Your own stuffs nice All the time care for it up
Hi my loved one I wish to say that this post is amazing nice written and include approximately all vital infos Id like to peer more posts like this
certainly like your website but you need to take a look at the spelling on quite a few of your posts Many of them are rife with spelling problems and I find it very troublesome to inform the reality nevertheless I will definitely come back again
My brother suggested I might like this blog He was totally right This post actually made my day You can not imagine simply how much time I had spent for this info Thanks
I was recommended this website by my cousin I am not sure whether this post is written by him as nobody else know such detailed about my trouble You are amazing Thanks
Thanks I have just been looking for information about this subject for a long time and yours is the best Ive discovered till now However what in regards to the bottom line Are you certain in regards to the supply
Thank you I have just been searching for information approximately this topic for a while and yours is the best I have found out so far However what in regards to the bottom line Are you certain concerning the supply
Fantastic beat I would like to apprentice while you amend your web site how could i subscribe for a blog site The account helped me a acceptable deal I had been a little bit acquainted of this your broadcast offered bright clear concept
greate post i love it..
greate post i love very goood..
???????, ????? ???-?????? ? ??????????? ???????, ?????????? ??????????? ? ???????? ????? ??????? ? ??????. ?? ??????? ???????????? ????????, ????????? ? ?????????? ?????????? ? ?????????????. ????? ?? ?????? ????? ????? ???????? „Like a Prayer“, „Vogue“, „Material Girl“, „Into the Groove“ ? „Hung Up“. ??? ????? ?? ?????? ???????????? ?? ??????????? ??????, ?? ? ???????? ???????????? ???? ? ?????????? ? ???????????? ???????? ??????. ??????? ?? ?????? ??????, ?? ? ????? ?????, ??????? ? ???????????????, ??? ??????? ???????????? ?????? ?? ????? ??????????? ?????????. ??????? mp3 ?????? 2024 ???? ? ??????? ?????? ?????????.
The unique set of AD0-E716 dumps is the easiest and the most rewarding content, you ever found on any web page. Your success is guaranteed! The questions and answers format of our dumps is rich with information and provides you also Adobe Commerce Developer with Cloud Add-on latest lab help, enhancing your exam skills. The content is approved by the most distinguished professionals and revised and updated by our experts on regular basis. With these brilliant features it is rated as the most worthwhile, informative and highly exam relevant. In all respects, you will find the AD0-E716 dumps compatible to your actual preparatory needs. The language is simple and the content is engaging and easy. No more, AD0-E716 exam is a nightmare.
I became a fan of this phenomenal website earlier this week, they give informative content to visitors. The site owner has a knack for educating readers. I’m impressed and hope they keep sharing great material.
greate post i love very goood…
greate post i like very much..
wow men thats great very nice post..
wow men thats great very nice post..
wow men thats great very nice post..
men thats great very good..
men thats great very nice..
I’m often to blogging and i really appreciate your content. The article has actually peaks my interest. I’m going to bookmark your web site and maintain checking for brand spanking new information.
Nice blog here Also your site loads up fast What host are you using Can I get your affiliate link to your host I wish my web site loaded up as quickly as yours lol
Nice blog here Also your site loads up fast What host are you using Can I get your affiliate link to your host I wish my web site loaded up as quickly as yours lol
I do agree with all the ideas you have introduced on your post They are very convincing and will definitely work Still the posts are very short for newbies May just you please prolong them a little from subsequent time Thank you for the post
Experience Excellence with Bwer Pipes: Elevate your farming operations with Bwer Pipes‘ industry-leading irrigation solutions. Our cutting-edge sprinkler technology and durable pipes are engineered for superior performance, enabling Iraqi farmers to achieve optimal water management, crop health, and productivity. Explore Bwer Pipes
What i do not understood is in truth how you are not actually a lot more smartlyliked than you may be now You are very intelligent You realize therefore significantly in the case of this topic produced me individually imagine it from numerous numerous angles Its like men and women dont seem to be fascinated until it is one thing to do with Woman gaga Your own stuffs nice All the time care for it up
Its like you read my mind You appear to know a lot about this like you wrote the book in it or something I think that you could do with some pics to drive the message home a little bit but instead of that this is fantastic blog An excellent read I will certainly be back
I was recommended this website by my cousin I am not sure whether this post is written by him as nobody else know such detailed about my trouble You are amazing Thanks
Hello my loved one I want to say that this post is amazing great written and include almost all significant infos I would like to look extra posts like this
I do trust all the ideas youve presented in your post They are really convincing and will definitely work Nonetheless the posts are too short for newbies May just you please lengthen them a bit from next time Thank you for the post
Usually I do not read article on blogs however I would like to say that this writeup very compelled me to take a look at and do so Your writing taste has been amazed me Thanks quite nice post
I was suggested this web site by my cousin Im not sure whether this post is written by him as no one else know such detailed about my trouble You are incredible Thanks
you are in reality a good webmaster The website loading velocity is amazing It sort of feels that youre doing any distinctive trick Also The contents are masterwork you have done a fantastic job in this topic
I do agree with all the ideas you have introduced on your post They are very convincing and will definitely work Still the posts are very short for newbies May just you please prolong them a little from subsequent time Thank you for the post
Wow amazing blog layout How long have you been blogging for you made blogging look easy The overall look of your web site is magnificent as well as the content
What i do not understood is in truth how you are not actually a lot more smartlyliked than you may be now You are very intelligent You realize therefore significantly in the case of this topic produced me individually imagine it from numerous numerous angles Its like men and women dont seem to be fascinated until it is one thing to do with Woman gaga Your own stuffs nice All the time care for it up
Hi my loved one I wish to say that this post is amazing nice written and include approximately all vital infos Id like to peer more posts like this
Ive read several just right stuff here Certainly price bookmarking for revisiting I wonder how a lot effort you place to create this kind of great informative website
My brother recommended I might like this web site He was totally right This post actually made my day You cannt imagine just how much time I had spent for this information Thanks
hiI like your writing so much share we be in contact more approximately your article on AOL I need a specialist in this area to resolve my problem Maybe that is you Looking ahead to see you
I do agree with all the ideas you have introduced on your post They are very convincing and will definitely work Still the posts are very short for newbies May just you please prolong them a little from subsequent time Thank you for the post
you are truly a just right webmaster The site loading speed is incredible It kind of feels that youre doing any distinctive trick In addition The contents are masterwork you have done a great activity in this matter
I was recommended this website by my cousin I am not sure whether this post is written by him as nobody else know such detailed about my difficulty You are wonderful Thanks
Somebody essentially lend a hand to make significantly posts I might state That is the very first time I frequented your web page and up to now I surprised with the research you made to create this particular put up amazing Excellent job
of course like your website but you have to check the spelling on several of your posts A number of them are rife with spelling issues and I in finding it very troublesome to inform the reality on the other hand I will certainly come back again
Attractive section of content I just stumbled upon your blog and in accession capital to assert that I get actually enjoyed account your blog posts Anyway I will be subscribing to your augment and even I achievement you access consistently fast
O que eu não entendi é que, na verdade, você não é muito mais inteligente do que seria agora. Você é muito inteligente. Você sabe muito sobre esse assunto e me fez acreditar nisso de vários ângulos diferentes. É como se mulheres e homens fossem não estou interessado, exceto que é uma coisa a realizar com Woman gaga Suas próprias coisas são excelentes Sempre cuide disso
Your blog is a treasure trove of valuable insights and thought-provoking commentary. Your dedication to your craft is evident in every word you write. Keep up the fantastic work!
Your blog is a breath of fresh air in the often stagnant world of online content. Your thoughtful analysis and insightful commentary never fail to leave a lasting impression. Thank you for sharing your wisdom with us.
Batida fantástica, gostaria de aprender enquanto você altera seu site, como posso me inscrever em um blog? A conta me ajudou a um acordo aceitável. Eu estava um pouco ciente disso, sua transmissão ofereceu um conceito claro e claro
Live casino games for you to choose from Fun88 provides a full range of betting games
Wow superb blog layout How long have you been blogging for you make blogging look easy The overall look of your site is magnificent as well as the content
I do not even know how I ended up here but I thought this post was great I do not know who you are but certainly youre going to a famous blogger if you are not already Cheers
What i do not realize is in fact how you are no longer actually much more wellfavored than you might be right now Youre very intelligent You recognize thus considerably in relation to this topic made me in my view believe it from numerous numerous angles Its like men and women are not fascinated until it is one thing to do with Lady gaga Your own stuffs excellent All the time handle it up
Olá, acho que vi que você visitou meu site, então vim retribuir o favor. Estou tentando encontrar coisas para melhorar meu site. Suponho que não há problema em usar algumas de suas ideias
I loved as much as youll receive carried out right here The sketch is attractive your authored material stylish nonetheless you command get bought an nervousness over that you wish be delivering the following unwell unquestionably come more formerly again as exactly the same nearly a lot often inside case you shield this hike
helloI really like your writing so a lot share we keep up a correspondence extra approximately your post on AOL I need an expert in this house to unravel my problem May be that is you Taking a look ahead to see you
Somebody essentially lend a hand to make significantly posts I might state That is the very first time I frequented your web page and up to now I surprised with the research you made to create this particular put up amazing Excellent job
Nice blog here Also your site loads up very fast What host are you using Can I get your affiliate link to your host I wish my site loaded up as quickly as yours lol
certainly like your website but you need to take a look at the spelling on quite a few of your posts Many of them are rife with spelling problems and I find it very troublesome to inform the reality nevertheless I will definitely come back again
Somebody essentially lend a hand to make significantly posts I might state That is the very first time I frequented your web page and up to now I surprised with the research you made to create this particular put up amazing Excellent job
Simply wish to say your article is as amazing The clearness in your post is just nice and i could assume youre an expert on this subject Well with your permission let me to grab your feed to keep updated with forthcoming post Thanks a million and please carry on the gratifying work
you are truly a just right webmaster The site loading speed is incredible It kind of feels that youre doing any distinctive trick In addition The contents are masterwork you have done a great activity in this matter
Its like you read my mind You appear to know so much about this like you wrote the book in it or something I think that you can do with a few pics to drive the message home a little bit but other than that this is fantastic blog A great read Ill certainly be back
These Salesforce AI Associate Salesforce-AI-Associate questions and answers not only help you in your preparation but also familiarize you with the AI Associate exam situation. By attempting our Salesforce-AI-Associate AI Associate practice exam questions, you will also learn time management. The best thing is that most questions in our exam dumps are those questions that can be included in upcoming Salesforce-AI-Associate Salesforce AI Associate exam dumps. Get access to our test engine so that you can get more updated exam dumps.
I was suggested this web site by my cousin Im not sure whether this post is written by him as no one else know such detailed about my trouble You are incredible Thanks
Hi my family member I want to say that this post is awesome nice written and come with approximately all significant infos I would like to peer extra posts like this
Keep up the fantastic work! Kalorifer Sobas? odun, kömür, pelet gibi yak?tlarla çal??an ve ?s?tma i?levi gören bir soba türüdür. Kalorifer Sobas? içindeki yak?t?n yanmas?yla olu?an ?s?y? do?rudan çevresine yayar ve ayn? zamanda suyun ?s?nmas?n? sa?lar.
Can you be more specific about the content of your article? After reading it, I still have some doubts. Hope you can help me.
Can you be more specific about the content of your article? After reading it, I still have some doubts. Hope you can help me.
Nice blog here Also your site loads up fast What host are you using Can I get your affiliate link to your host I wish my web site loaded up as quickly as yours lol
Noodlemagazine Nice post. I learn something totally new and challenging on websites
Thank you I have just been searching for information approximately this topic for a while and yours is the best I have found out so far However what in regards to the bottom line Are you certain concerning the supply
Thank you for your sharing. I am worried that I lack creative ideas. It is your article that makes me full of hope. Thank you. But, I have a question, can you help me?