Uploading Images to AWS S3 and Storing S3 Key in DynamoDB using AWS Lambda - Streamlit Web App

ยท

4 min read

I recently worked on a Streamlit web application project that allows users to upload a photo and a completed form.

Streamlit is an open-source app framework primarily used by Machine-learning and Data Science communities to quickly develop and share their projects on web applications. Streamlit allows us to code in Python and has a limited but useful set of features to spin up a web application quickly.

Since I did not have much experience with front-end development especially in JavaScript, HTML, or CSS, I opted to use Streamlit.

To understand how easy it is to get a Streamlit app running, I will provide a sample code to create a form with the file upload functionality:

import streamlit as st

with st.form(key="my_form"):
    st.write("Test Form:")
    name = st.text_input("Enter your name: ")
    fav_color = st.text_input("Enter your favorite color: ")
    hobby = st.text_input("Enter your hobby: ")

    photo = st.file_uploader("Choose a photo:", type=['jpg', 'jpeg', 'png'])

    submitted = st.form_submit_button("Submit")
    if submitted:
        st.success("Form submitted.")

Install Streamlit pip install streamlit and run the application with the below example:

streamlit run app.py

Streamlit runs on localhost port 8501 by default:

The application is rendered in the following manner:

Once the user submits the form, we can perform some basic input validation and send the data as an HTTP POST request to the backend.

The text attributes of the form are stored in a DynamoDB table and the photo is stored in an S3 bucket.

To associate the user's data stored with the corresponding image in the S3 bucket, we created a unique "S3 Key" for each image and stored it in the DynamoDB table.

The backend in this project consists of an API Gateway's HTTP API integrated with an AWS Lambda function (also written in Python). The Lambda function extracts the user attributes and the image, creates the S3 Key, and stores the text values in the DynamoDB table and the image in the S3 bucket.

Image built using: online.visual-paradigm.com

The code in Streamlit to send the photo to the HTTP API is as follows:

# Include the code to display the form here.
    if submitted:

        import base64
        import json

        encoded_image = base64.b64encode(photo.read()).decode('utf-8')

        photo_upload_api_url = "https://test.execute-api.us-east-2.amazonaws.com/photo_upload"
        photo_payload = {
            "name": name,
            "image": encoded_image,
            "email": email
        }
        photo_headers = {
            'Content-Type': 'application/json',
            'Authorization': access_token # This is the access token for API Gateway if authorization is enabled.
        }
        try:
            photo_response = requests.post(photo_upload_api_url, data=json.dumps(photo_payload), headers=photo_headers)
            st.success("ID upload successful. ๐Ÿ‘")

        except Exception as err:
            st.error(f'An error occurred: {err}')

The Lambda integration attached to the HTTP API will trigger the Python code similar to the one below to create new records in the DynamoDB table and upload the image into the S3 bucket.

Example Lambda function:

import json
import boto3
import base64
from io import BytesIO

def lambda_handler(event, context):
    http_method = event['requestContext']['http']['method']

    # If the request is a POST, then we know that the user has submitted the form.
    if http_method == 'POST':
        photo_response = handle_dynamo_db(event)

        return {
            'statusCode': 200,
            'body': json.dumps(photo_response)
        }
    else:
        return {'statusCode': 200}


def handle_dynamo_db(event):
    """
    This function inserts the user's name, email, and S3 key into DynamoDB.
    """
    body = json.loads(event['body'])

    dynamodb = boto3.client('dynamodb') # This is the DynamoDB client.

    image = body['image']
    name = body['name']
    email = body['email']

    s3_key = f"{name}-{email}-id-key.jpg" # This is the S3 key for the image.

    upload_to_s3(image, s3_key) # Upload the image to S3.

    update_response = dynamodb.put_item(
        TableName="test-table",
        Key={
            'name': {'S': name},
            'email': {'S': email},
            's3_key': {'S': s3_key}
        }

    )

    return update_response

def upload_to_s3(base64_image, s3_key):
    """
    This function uploads the image to S3.
    """
    s3 = boto3.client('s3') # This instantiates the S3 client.
    bucket_name = 'test-bucket'

    image_bytes = base64.b64decode(base64_image)

    # Create a file-like object from the bytes
    image_file = BytesIO(image_bytes)

    # Upload the image to S3
    s3.upload_fileobj(image_file, bucket_name, s3_key)

Some notes for implementation:

  • Ensure that the Lambda function has the necessary permissions to 'put-item' to the DynamoDB table and upload objects to your S3 bucket.

  • If there are errors in the Lambda execution, you will see HTTP code 500 responses returned on the Streamlit app.

  • Use AWS Cloudwatch logs to monitor your lambda function execution and check for any error messages.

  • Check the DynamoDB table to ensure that the user data and the S3 key are stored as expected.

  • The S3 key is the file name of the image stored in the S3 bucket (hence we add the .jpg suffix to the key name).

ย