Takuma Yoneda

Boto3 client cheatsheet

Some notes on boto3 client. Prerequisite: ~/.aws/client and ~/.aws/credentials are properly set. For more details on how to set these, have a look at this page.

This will list up the files that starts with foo/bar. It seems this prefix does not support regex.

import boto3
bucket_name = 'my-awesome-bucket'
prefix = 'foo/bar'
bucket = boto3.Session().resource('s3').Bucket(bucket_name)
for obj in bucket.objects.filter(Prefix=prefix):
    print(obj.key)

Search, glob on S3 bucket

If you want to glob using ** keyword (e.g., foo/bar/**/baz.txt), you can do a trick like this:

import boto3
bucket_name = 'my-awesome-bucket'
prefix = 'foo/bar/**/baz.txt'
bucket = boto3.Session().resource('s3').Bucket(bucket_name)

if '/**/' in prefix:
    prefix, key = prefix.split('/**/', 1)
else:
    key = ''

bucket = boto3.Session().resource('s3').Bucket(bucket_name)
for obj in bucket.objects.filter(Prefix=prefix):
    if key and not obj.key.endswith(key):
        continue

Basically this splits the prefix into foo/bar and baz.txt. Then search for files with prefix foo/bar, while filtering those that don't end with baz.txt with simple if syntax.

Copy files / directories, move file / directory on S3 bucket

There's no notion of "directory" on S3. You always do some action on files. To move a file, you copy & remove the original one.

import boto3, botocore
bucket_name = 'my-awesome-bucket'
s3 = boto3.Session().resource('s3')

src_key = 'hoge/fuga/baz.txt'
dest_key = 'hoge/piyo/baz.txt'

try:
    # NOTE: This fails if the file is 5GB+
    copy_source = bucket_name + '/' + src_key
    s3.Object(bucket_name, dest_key).copy_from(CopySource=copy_source)
except botocore.exceptions.ClientError:
    # NOTE: This works however large the file is.
    copy_source_dict = {'Bucket': bucket_name, 'Key': src_key}
    s3.meta.client.copy(copy_source_dict, bucket_name, dest_key)

Some other links