Elasticsearch Backup : Snapshot and Restore on AWS S3

(written by lawrence krubner, however indented passages are often quotes). You can contact lawrence at: lawrence@krubner.com, or follow me on Twitter.

Because Terraform will delete all of your databases when you rebuild your infrastructure, and that is a nasty shock. So you should manually back up ElasticSearch:

In Aws you have many options to allow [users|servers] access AWS resources. The idea we have is to setup a snapshot repository on Amazon Aws S3 and doing a restore from that specific location. In order to do that, the servers that have to access S3 must be authorised. Thankfully to Amazon we have many options :

using IAM (Identity Access Management) Roles
using a specific user with specific roles/policies
The first approach is the one I suggest to use, as it is more reliable, we don’t need to setup anything on the server than the AWS cli command, and it’s not mandatory at all because, the Elasticsearch plugin do the job pretty well. The difference between the two approaches is :

IAM Role is attached to the EC2 instance at the moment of starting it up.
USER needs its own credentials whenever you need to access S3.
So basically if you need to start a new EC2 or a new fleet of EC2’s use a IAM ROLE, if you want to setup an Elasticsearch “snapshotting” to an already existing ES cluster you have to use a USER with a specific policy.

AWS S3 : Policy
Before entering the details of IAM Roles or Users we need, both cases, to define a Policy. A Policy is a set of rules you can define and use for all the services AWS offer, and with many levels of granularity, in order to grant access to these resources.

“A policy is a document that formally states one or more permissions.”
If you want to understand what a policy is and how to use it, I suggest these links to the well written AWS documentation :

Access Policies
Access Policies Reference
There are many pre-defined Policies for the whole AWS service stack. In our case we want to grant access our EC2 instances to S3 services :

AmazonS3FullAccess

Post external references

  1. 1
    https://medium.com/@federicopanini/elasticsearch-backup-snapshot-and-restore-on-aws-s3-f1fc32fbca7f
Source