AWS S3 Presigned URLs

I have been working with AWS S3 from quite a while now. One of the great features provided by S3 is Presigned URLs. I had to use this feature a couple of times in recent times so I decided to write about it.

What are Presigned URLs?

S3 provides Presigned URLs to give temporary access to unauthorized users to GET or POST files to and from S3.

These URLs are generated by authorized AWS user who has access to the S3 resource. While creating the URL AWS user can set an expiry time for that URL after which the URL stops working.

Why I had to use Presigned URLs?

I have created a Python microservice using Flask which is deployed on AWS Lambda and is made accessible by AWS API Gateway. One of the APIs in the microservice allows user to upload multiple images for a resource and there I was stuck with API Gateway’s payload size limitation which is 10 MB and cannot be increased.

So, in the API multiple images were making the API payload size cross the limit of 10 MB and API Gateway was returning 413 status code which means Payload Too Large. And then I decided to use S3 Presigned URLs.

FYI: Another important thing to note, these payload errors won’t show up in Lambda’s CloudWatch logs. Because the error response is returned from the API Gateway itself. So you should find these logs in your API Gateway logs.

GET S3 files

There is a method available with S3 client object named generate_presigned_url. I have created the a python function to generate public url for an S3 key. You can set the expiry time when you are generating the url.

The url which is returned will be accessible publicly and can be rendered on webpages, if needed.

import boto3
from botocore.exceptions import ClientError

def get_public_url(bucket, key, expiry=3600):
    """
    Method to get presigned public url for an object in  s3
    :param bucket: name of s3 bucket
    :param key: key of the object
    :return: SimpleNamespace object with public url
    """
    try:
        url = boto3.client('s3').generate_presigned_url(
            ClientMethod='get_object',
            Params={
                'Bucket': bucket,
                'Key': key,
            },
            ExpiresIn=expiry
        )
        return SimpleNamespace(success=True, data=url)

    except ClientError as client_error:

        return SimpleNamespace(
            success=False,
            data='',
            error=client_error.response['Error']['Message']
        )

Upload files directly to s3

To get s3 URL to upload a file, the method available is generate_presigned_post.

Here is a python function that uses this method:

import boto3
from botocore.exceptions import ClientError

def get_s3_presigned_post_url(bucket, key, expiry=3600):
    """
    Method to get post url from s3
    :param bucket: name of s3 bucket
    :param key: key of the file
    :return:
    """
    try:

        presigned_post = boto3.client('s3').generate_presigned_post(
            bucket,
            key,
            ExpiresIn=expiry
        )

        return SimpleNamespace(success=True, data={
            'upload_url': presigned_post['url'],
            'fields': presigned_post['fields']
        })

    except ClientError:

        return SimpleNamespace(
            success=False,
            error='Unable to upload image'
        )

This method will return you and upload_url and some fields as key-value pairs which the unauthorized user will have to pass in the request when uploading the file.

{ 
   "upload_url":"url-of-your-s3-bucket",
   "fields":{ 
      "key":"path/to/your/file.jpg",
      "AWSAccessKeyId":"XXX",
      "x-amz-security-token":"XXX",
      "policy":"XXX",
      "signature":"XXX"
   }
}

Along with the fields, one more field will be added with the key name as file and an uploaded file as value of this key. See the example below:

In my case, my frontend application was making this request instead of sending it from PostMan.