AWS Lambda in VPC: Intermittent S3 Timeouts & Slow Cold Starts Despite NAT Gateway for Public Access
We're developing a serverless backend for a mobile application, heavily leveraging AWS Lambda functions. One particular Lambda function (let's call it UserEventHandler) processes incoming user events. Its primary tasks are twofold: storing structured event data in an AWS RDS PostgreSQL instance and asynchronously uploading detailed raw event logs to an S3 bucket.
To ensure secure access to our RDS database, which resides in private subnets, we've configured our UserEventHandler Lambda to run within our VPC. This setup places the Lambda in private subnets, and we've deployed a NAT Gateway in public subnets with corresponding route table entries to allow the Lambda to make outbound calls to public internet services, specifically S3 and potentially external APIs.
Our environment details are as follows:
- AWS Lambda Runtime: Python 3.9 (latest Boto3 bundled)
- AWS Region:
ap-south-1 - Database: AWS RDS PostgreSQL 13.x (in private subnets within the same VPC)
- VPC Configuration: We have a standard VPC setup with 3 public subnets (where the NAT Gateway resides) and 3 private subnets (where the Lambda and RDS instances are placed).
- Security Groups: The Lambda's security group allows outbound traffic to the RDS security group on port 5432. The RDS security group allows inbound traffic from the Lambda's security group on port 5432. For S3, the Lambda's security group has an outbound rule allowing
All Traffic(0.0.0.0/0), which we assumed would handle S3 access via the NAT Gateway.
The RDS connection works reliably and relatively quickly after the initial connection pooling. However, we're consistently observing two critical issues related to S3 operations during Lambda cold starts:
- Elevated Cold Start Times: The
UserEventHandlerLambda takes significantly longer (8-12 seconds) to initialize and process the first request after a cold start, compared to warm invocations (under 200ms). - Intermittent S3 Timeouts/Delays: Specifically, the
boto3.client('s3').put_object()call experiences substantial delays (sometimes 5-10 seconds) or evenbotocore.exceptions.ConnectTimeoutErrorduring cold starts. After the cold start, subsequent S3 operations on warm invocations are fast and reliable.
Here's a simplified version of the Python code within our Lambda function:
import json
import os
import boto3
import psycopg2
import logging
logger = logging.getLogger()
logger.setLevel(os.environ.get('LOG_LEVEL', 'INFO'))
# Environment variables
DB_HOST = os.environ.get('DB_HOST')
DB_NAME = os.environ.get('DB_NAME')
DB_USER = os.environ.get('DB_USER')
DB_PASSWORD = os.environ.get('DB_PASSWORD')
S3_BUCKET = os.environ.get('S3_BUCKET_NAME')
s3_client = boto3.client('s3')
db_conn = None
def get_db_connection():
global db_conn
if not db_conn or db_conn.closed:
logger.info("Establishing new DB connection...")
try:
db_conn = psycopg2.connect(
host=DB_HOST,
database=DB_NAME,
user=DB_USER,
password=DB_PASSWORD,
connect_timeout=5
)
except Exception as e:
logger.error(f"Error connecting to database: {e}")
raise
return db_conn
def lambda_handler(event, context):
request_id = context.aws_request_id
logger.info(f"Processing event {request_id}")
try:
# 1. Store data in RDS
conn = get_db_connection()
with conn.cursor() as cur:
# Example: insert a record
user_id = event.get('userId')
event_type = event.get('eventType')
cur.execute("INSERT INTO events (user_id, event_type) VALUES (%s, %s);", (user_id, event_type))
conn.commit()
logger.info(f"Event for user {user_id} stored in RDS.")
# 2. Upload raw event data to S3
event_data_bytes = json.dumps(event).encode('utf-8')
s3_key = f"raw-events/{request_id}.json"
logger.info(f"Attempting to upload {len(event_data_bytes)} bytes to S3 bucket {S3_BUCKET} with key {s3_key}")
# This is where the delay/timeout happens during cold starts
s3_client.put_object(
Bucket=S3_BUCKET,
Key=s3_key,
Body=event_data_bytes,
ContentType='application/json'
)
logger.info(f"Raw event {request_id} uploaded to S3.")
return {
'statusCode': 200,
'body': json.dumps({'message': 'Event processed successfully!'})
}
except botocore.exceptions.ConnectTimeoutError as e:
logger.error(f"S3 Connect Timeout Error: {e}")
return {
'statusCode': 500,
'body': json.dumps({'message': 'S3 operation timed out during cold start.'})
}
except Exception as e:
logger.error(f"Unhandled error: {e}")
# Invalidate DB connection on unhandled error to force re-establishment
global db_conn
if db_conn:
db_conn.close()
db_conn = None
return {
'statusCode': 500,
'body': json.dumps({'message': 'Internal Server Error'})
}
Expected Behavior:
We expected S3 operations, like RDS connections, to be fast and reliable, even during cold starts, given the NAT Gateway provides outbound internet access. The s3_client.put_object() call should complete within milliseconds.
Actual Behavior:
During cold starts, the Lambda takes 8-12 seconds to become ready. The s3_client.put_object() call specifically introduces a significant portion of this delay, sometimes leading to botocore.exceptions.ConnectTimeoutError or even the entire Lambda timing out if its timeout is set too aggressively (e.e. 30 seconds). Warm invocations exhibit no such issues.
Error Message Example:
[ERROR] 2023-10-27T10:30:10.567Z <request-id> S3 Connect Timeout Error: Connect timeout on endpoint URL: "https://my-log-bucket-123.s3.ap-south-1.amazonaws.com/raw-events/abc-123.json"
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 75, in lambda_handler
ContentType='application/json'
File "/var/runtime/botocore/client.py", line 403, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/var/runtime/botocore/client.py", line 695, in _make_api_call
http, parsed_response = self._endpoint.make_request(request_dict, request_context)
File "/var/runtime/botocore/endpoint.py", line 105, in make_request
return self._send_request(request_dict, request_context)
File "/var/runtime/botocore/endpoint.py", line 170, in _send_request
raise ConnectTimeoutError(endpoint_url=request.url, error=e)
botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "https://my-log-bucket-123.s3.ap-south-1.amazonaws.com/raw-events/abc-123.json"
It seems like the NAT Gateway is not providing the expected performance for S3 access during cold starts, or there's some underlying network configuration causing this specific bottleneck. We're trying to understand the root cause and the most efficient way to resolve this without compromising security or significantly increasing costs.
Community Insights
1Unlock Global Discussions
The TechLab Forum is reserved for our premium engineering community. Upgrade now to participate in high-level architectural debates.
Get Pro Access