How to easily migrate a small load of objects to Bucketeer?

In a previous post, I outlined the advantages and disadvantages of using Bucketeer. Also, I described the process to automatically add buckets to review apps. But what happens when we already have objects in S3 buckets from existing apps?

There are various solutions that AWS provides for migrating objects. But most involve too many steps or configurations, like adding IAM or bucket policies. Even, some of them required the Account ID, which we didn't have for the Bucketeer ones. Having limited access to Bucketeer credentials makes those configurations a headache. We would need help from Bucketeer’s Support Team.

By far, using AWS CLI was one of the most straightforward ways to migrate objects between buckets. It allows you to copy objects from one bucket to another. Our former buckets had less than 100 objects by then, so the other alternatives would be overly time-consuming. We made a few tweaks to make it work since migrating to Bucketeer involves different accounts and credentials.We couldn’t pass the data from one location to the other with a single action. But we had access to manage objects from both buckets. Thus, all we needed was a mediator: a local machine. I’ll supply more details shortly, but the approach consisted of doing the transition in two steps:

Before proceeding with the steps, you must ensure that you meet certain requirements:

• Ensure AWS CLI is installed in the local machine that will serve as a mediator. Refer to the following tutorial for installation instructions: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html
• Access to the following information:some text
  1. Secret access key
  2. Access key id
  3. The bucket name
  4. The Region from the source and destination bucket. 

This information is accessible in the environment variables from the Heroku app.

• Consider the data you’re handling. If it’s a big amount or sensitive data, I recommend exploring alternative approaches.

Once you have made sure you have accomplished the requirements, the steps are as easy as follows:

1. Configure AWS CLI with the source bucket credentials using the following command:

aws configure

Then it will ask you to set the following data:

• Secret Access Key
• Access Key ID
• Region
• Default text

2. Create a directory to store the data
3. Download from the source bucket to your local directories using the following command:

aws s3 cp s3://bucket path/to/file --recursive

‍

4. Configure AWS CLI with Bucketeer credentials using the following command:

aws configure

Again, you will be asked to set the following data. Note that this information can be found within your Heroku app variables.

• Secret Access Key
• Secret Access Key ID
• Region
• Default text
  ‍

5. Upload files to Bucketeer using the following command: 

aws s3 cp path/to/file s3://bucketeer-bucket --recursive

Migration processes can be tricky and intimidating. Considering your needs and requirements can be key to finding the optimal solution.

‍