AWS S3 Presigned URLs

I have been working with AWS S3 from quite a while now. One of the great features provided by S3 is Presigned URLs. I had to use this feature a couple of times in recent times so I decided to write about it. What are Presigned URLs? S3 provides Presigned URLs to give temporary access to unauthorized users to GET or POST files to and from S3. These URLs are generated by authorized AWS user who has access to the S3 resource. While creating the URL AWS user can set an expiry time for Read more [...]

CORS Configuration for AWS S3 and CloudFront files

I have been using S3 to host my images and delivering them using AWS CloudFront. But recently I faced an issue whiling accessing those images via a CORS request. I tried many solutions available for the same but couldn't get the issue to fix. Here I am sharing the configuration that I had to do in AWS to fix the issue. Update S3 bucket CORS configuration Go to you S3 bucket > Permissions > CORS Configuration and add the following in the CORS configuration editor: <?xml version="1.0" Read more [...]

How to scan your code (Github) for Passwords and Keys

If you are a programmer, there is a very good chance that you use Github to host your code. But do you have a habit (or maybe unintentionally) of committing sensitive and confidential information like Keys and Passwords to your public repositories. In some cases, it can be a complete disaster. It is more common than you may think: Over 100,000 GitHub repos have leaked API or cryptographic keys So how to prevent something like above from happening? Of course, the straight forward answer Read more [...]

The f-Strings in Python 3.6+

Python has multiple ways to format text strings. The most common being %-formattingĀ andĀ str.format(). These existing ways of formatting are either error prone, inflexible, or cumbersome. And then PEP 498 introduced the f-strings also known as Literal String Interpolation. PEP 498 now gives us a very simple and clear way of formatting strings. f-String is a more readable and concise way of formatting python strings. It is also less prone to error. But it is important to remember that these Read more [...]

Trigger AWS Lambda Asynchronously from API Gateway

AWS API Gateway
AWS Lambda & API Gateway are the perfect partners when it comes to serverless architecture. The default invocation type of AWS Lambda from API Gateway is synchronous. That means the lambda keeps the connection open until the function returns a response or times out. And that is usually required by most of the REST APIs. The API returns a response along with optional data as soon as the request processing is complete. But what if the API is doing batch operations for a longer than Read more [...]

AWS Lambda Function To Launch EMR with Hadoop Map-Reduce Python

AWS Lambda Function
Recently, I have been working with processing of large data sets on the daily basis. I decided to use Hadoop Map-Reduce and wrote mapper and reducer scripts to process the data. The whole process included launching EMR cluster, installing requirements on all nodes, uploading files to Hadoop's HDFS, running the job and finally terminating the cluster (Because AWS EMR Cluster is expensive). To eliminate the manual effort I wrote an AWS Lambda function to do this whole process automatically. Read more [...]