AWS S3 Sync Example

We have provided examples of how to interact with S3 using AWS CLI. In this post, we will show how we can synchronize our local directory with S3. Assume that in our local directory we have the following files.

• file_1.txt
• file_2.txt
• file_4.txt

And at the destination S3 bucket there are the following files:

• file_3.txt
• file_4.txt

Our goal is to synchronize the S3 bucket by recursively copying the new and updated files from the source directory to the destination. Let’s see how we can easily do it with the sync command from the AWS CLI.

aws s3 sync . s3://gpipis-test-bucket/aws_sync_example/

Output

upload: .\file_1.txt to s3://gpipis-test-bucket/aws_sync_example/file_1.txt
upload: .\file_2.txt to s3://gpipis-test-bucket/aws_sync_example/file_2.txt

As we can see, only the file_1.txt and file_2.txt files have been uploaded to the S3 bucket.

Note that a local file will require uploading if the size of the local file is different from the size of the s3 object, the last modified time of the local file is newer than the last modified time of the s3 object, or the local file does not exist under the specified bucket and prefix.

We could have worked the other way around, meaning to define as source the S3 bucket and as destination the local directory. For example:

aws s3 sync s3://gpipis-test-bucket/aws_sync_example/ .

Output

download: s3://gpipis-test-bucket/aws_sync_example/file_3.txt to .\file_3.txt

As we can see, the file_3.txt was added to our local directory.

We can synchronize 2 S3 buckets too, as follows:

aws s3 sync s3://mybucket s3://mybucket2

Finally, with the sync command, we can do more sophisticated tasks, like synchronizing folders by excluding or including specific types of files or by deleting filers that exist in the destination folder but not in the source one.

Example: We any files existing under the specified prefix and bucket but not existing in the local directory to be deleted. In this example, the user syncs the bucket mybucket to the local current directory. The local current directory contains the files test.txt and test2.txt. The bucket mybucket contains the object test3.txt:

aws s3 sync . s3://mybucket --delete

Output:

upload: test.txt to s3://mybucket/test.txt
upload: test2.txt to s3://mybucket/test2.txt
delete: s3://mybucket/test3.txt

Finally, if we want to exclude hidden files and directories (.dotfiles and .dotfolders)

aws sync --exclude "^\." ./ s3://mybucket