how to check if aws s3 path exists with the help of python?

Question

I am trying to write python program that checks if path exists or not. For example, given the path /root/subfolder1/subfolder2/, I want to pass this path to the S3 API to check whether this path exists in AWS S3 or not.

I have tried this, but it is not full-fledged solution for my requirement:

import boto3
import botocore
client = boto3.client('s3',aws_access_key_id=AccessKey, aws_secret_access_key=SecretAccessKey,region_name='us-east-1')
result = client.list_objects(Bucket=full_poc", Prefix="sub_folder1/sub_folder2/full" )
print(result)
exist = False
if "Contents" in result:
    exist = True

print(exist)

With this code, even if I pass sub instead of sub_folder1 it prints True.

What are other ways to solve this problem?

Answer 1

S3 doesn't have folders:

In Amazon S3, buckets and objects are the primary resources, and objects are stored in buckets. Amazon S3 has a flat structure instead of a hierarchy like you would see in a file system. However, for the sake of organizational simplicity, the Amazon S3 console supports the folder concept as a means of grouping objects. Amazon S3 does this by using a shared name prefix for objects (that is, objects have names that begin with a common string). Object names are also referred to as key names.

The only way that /root/subfolder1/subfolder2/ can "exist" is if you have an object whose key begins with /root/subfolder1/subfolder2/. List the objects in your bucket and see if any begin with that prefix, e.g. something like

any((s.startswith("/root/subfolder1/subfolder2/") for s in bucket.objects.all()))

Answer 2

No such thing called Folder in S3. Folder is basically an empty file with name ending with '/'. We can check two things

• getObject results in empty body
• Make sure name of key ends with / before getObject. Reason for this check is, we don't want to get the actual object unless we know its a folder name, it will result in unnecessary data transfer.

If object doesn't exist getObject will result in error, we can just catch it.

s3 = boto3.client('s3')
key = 'myfolder1/subfolder/'
try:
    if(key.endswith('/')):
        obj = s3.get_object(Bucket='my-bucket',
                            Key=key)
        if(len(obj.get('Body').read()) == 0):
            folder = True
    else:
        folder = False
except Exception as e:
    folder = False
if(folder):
    print("yes its a folder")
else:
    print("No Its not")

Answer 3

import os
import tensorflow as tf
os.environ['AWS_REGION'] = 'us-west-2'
os.environ['S3_ENDPOINT'] = 's3-us-west-2.amazonaws.com'
print(tf.gfile.Exists('s3path'))#返回True or False