TL;DR: Prevent Node.js memory exhaustion and ALB timeouts by offloading gigabyte-scale file uploads directly to AWS S3. This guide demonstrates how to orchestrate client-side multipart uploads using AWS SDK v3 presigned URLs while enforcing strict data-at-rest encryption with AWS KMS. You will also learn how to dynamically batch URL generation to avoid blocking the event loop with heavy HMAC-SHA256 computations.

⚡ Key Takeaways

• Eliminate OOMKilled errors and ALB timeouts by removing Express and multer.memoryStorage() from the data plane, preventing file bytes from touching your Node.js backend.
• Initialize secure uploads using CreateMultipartUploadCommand and enforce enterprise-grade encryption at creation by passing ServerSideEncryption: 'aws:kms' with your SSEKMSKeyId.
• Adhere strictly to AWS S3 multipart constraints by ensuring every upload chunk (except the last) is at least 5MB, with a maximum limit of 10,000 parts per file.
• Generate presigned URLs for UploadPartCommand dynamically in small batches using Promise.all() to prevent synchronous HMAC-SHA256 signature calculations from blocking the Node.js event loop.
• Architect your Node.js API to act solely as a control plane that issues cryptographic signatures via @aws-sdk/s3-request-presigner, rather than a proxy buffering gigabytes of data.

Streaming large file uploads through your Node.js API works perfectly in local development. But deploy it to production, and the reality of handling gigabyte-scale documents quickly shatters the illusion.

When a user uploads a 5GB video or medical dataset, streaming it through a standard Express middleware pipeline creates massive backpressure. The V8 garbage collector kicks into overdrive, CPU utilization spikes, and event loop latency skyrockets. Soon, your Kubernetes pods are killed with OOMKilled errors, and your Application Load Balancer (ALB) drops connections due to 60-second idle timeouts. Every byte shuttled through your backend wastes expensive compute resources just to act as a dumb proxy to your storage bucket.

The solution is to remove Node.js from the data plane entirely.

By utilizing presigned URLs and S3 multipart upload, we can orchestrate a secure architecture where clients upload chunks directly to AWS S3. We pair this with AWS Key Management Service (KMS) to ensure strict, enterprise-grade data encryption at rest. In this guide, we will architect a production-ready, fault-tolerant document vault.

The Anti-Pattern: Memory Exhaustion via Naive Streams

Most Node.js applications handle file uploads using intermediate streams or buffers. Here is the common anti-pattern that brings production systems to their knees:

import express from 'express';
import multer from 'multer';
import { S3Client, PutObjectCommand } from '@aws-sdk/client-s3';

const app = express();
// Anti-pattern: Buffering the entire file in memory
const upload = multer({ storage: multer.memoryStorage() }); 

app.post('/api/upload', upload.single('document'), async (req, res) => {
  const s3 = new S3Client({ region: 'us-east-1' });
  
  await s3.send(new PutObjectCommand({
    Bucket: 'secure-document-vault',
    Key: req.file.originalname,
    Body: req.file.buffer, // V8 Heap is exhausted here
  }));

  res.status(200).send('Upload complete');
});

Production Warning: Even if you swap multer.memoryStorage() for multer.diskStorage() and pipe a read stream to S3, you are still bound by EBS disk IOPS limits, load balancer connection timeouts, and unnecessary network egress costs.

To build scalable healthcare document vaults or high-throughput media platforms, the file bytes must never touch your backend.

Phase 1: Initializing the Direct-to-S3 Multipart Vault

We will use the AWS SDK v3 to orchestrate the upload. The Node.js backend acts only as the control plane, issuing cryptographic signatures that permit the browser to interact directly with S3.

First, we initialize the multipart upload. This tells S3 to expect an incoming file broken into numbered parts. Crucially, we enforce server-side encryption using a customer-managed KMS key at the moment of creation.

import { S3Client, CreateMultipartUploadCommand } from '@aws-sdk/client-s3';

const s3Client = new S3Client({ region: process.env.AWS_REGION });

export async function initiateUpload(fileName: string, mimeType: string) {
  const command = new CreateMultipartUploadCommand({
    Bucket: process.env.S3_VAULT_BUCKET,
    Key: `uploads/${Date.now()}-${fileName}`,
    ContentType: mimeType,
    // Enforce KMS encryption at the moment of object creation
    ServerSideEncryption: 'aws:kms',
    SSEKMSKeyId: process.env.AWS_KMS_KEY_ID,
  });

  try {
    const { UploadId, Key } = await s3Client.send(command);
    if (!UploadId) throw new Error('Failed to obtain UploadId');
    
    return { uploadId: UploadId, objectKey: Key };
  } catch (error) {
    console.error('Multipart init failed:', error);
    throw new Error('Storage initialization failed');
  }
}

Phase 2: Generating Pre-Signed Chunk URLs

AWS S3 has strict rules for multipart uploads:

1. Every part (except the last) must be at least 5MB.
2. A single upload can have a maximum of 10,000 parts.

For a 50GB file utilizing 10MB chunks, we need to generate 5,000 presigned URLs. Generating these synchronously in Node.js requires computing thousands of HMAC-SHA256 signatures, which will block the event loop.

Instead, we expose an endpoint that dynamically generates presigned URLs in small batches as the client requests them.

import { S3Client, UploadPartCommand } from '@aws-sdk/client-s3';
import { getSignedUrl } from '@aws-sdk/s3-request-presigner';

const s3Client = new S3Client({ region: process.env.AWS_REGION });

interface PresignRequest {
  uploadId: string;
  objectKey: string;
  partNumbers: number[]; // e.g., [1, 2, 3, 4, 5]
}

export async function generatePartUrls(req: PresignRequest) {
  const { uploadId, objectKey, partNumbers } = req;
  const urls: Record<number, string> = {};

  // Execute in parallel, keeping batch sizes small to avoid CPU starvation
  await Promise.all(
    partNumbers.map(async (partNumber) => {
      const command = new UploadPartCommand({
        Bucket: process.env.S3_VAULT_BUCKET,
        Key: objectKey,
        UploadId: uploadId,
        PartNumber: partNumber,
      });

      // URL expires in 15 minutes, tightly scoping the blast radius
      urls[partNumber] = await getSignedUrl(s3Client, command, { expiresIn: 900 });
    })
  );

  return urls;
}

Performance Tip: Do not request 10,000 URLs in a single API call. Your backend CPU will spike processing the cryptography. Have your frontend request URLs in batches of 50-100 just-in-time as the upload progresses.

Phase 3: Client-Side Concurrency and Fault Tolerance

This is where the magic happens. We utilize the browser's native File.slice() API. It does not load the entire file into RAM; it merely creates a pointer to a specific byte range on the user's disk.

The frontend must orchestrate chunking, manage concurrent uploads, and capture the ETags returned by S3 in the response headers.

// browser-client.ts
const CHUNK_SIZE = 10 * 1024 * 1024; // 10MB

interface UploadedPart {
  PartNumber: number;
  ETag: string;
}

export async function uploadFileDirectToS3(file: File) {
  const totalParts = Math.ceil(file.size / CHUNK_SIZE);
  
  // 1. Initialize Upload via Backend
  const { uploadId, objectKey } = await api.post('/init-upload', {
    fileName: file.name,
    mimeType: file.type
  });

  const uploadedParts: UploadedPart[] = [];
  const activeUploads = new Set<Promise<void>>();
  const MAX_CONCURRENCY = 4; // Prevent browser network queue collapse

  // 2. Upload Orchestration
  for (let i = 0; i < totalParts; i++) {
    // Wait for a slot to open if we've hit our concurrency limit
    if (activeUploads.size >= MAX_CONCURRENCY) {
      await Promise.race(activeUploads);
    }

    const partNumber = i + 1;
    const start = i * CHUNK_SIZE;
    const end = Math.min(start + CHUNK_SIZE, file.size);
    const chunk = file.slice(start, end);

    const uploadPromise = (async () => {
      try {
        // Fetch presigned URL just-in-time
        const { url } = await api.post('/get-part-url', { uploadId, objectKey, partNumber });
        const eTag = await uploadChunk(url, chunk, partNumber);
        
        uploadedParts.push({ PartNumber: partNumber, ETag: eTag });
      } catch (err) {
        console.error(`Chunk ${partNumber} failed`, err);
        // Implementation note: Add retry logic with exponential backoff here
        throw err;
      }
    })();

    activeUploads.add(uploadPromise);
    uploadPromise.finally(() => activeUploads.delete(uploadPromise));
  }

  // Wait for all remaining active uploads to finish
  await Promise.all(activeUploads);

  // 3. Complete Upload
  await api.post('/complete-upload', {
    uploadId,
    objectKey,
    parts: uploadedParts.sort((a, b) => a.PartNumber - b.PartNumber) // Order is strictly required
  });
}

async function uploadChunk(url: string, chunk: Blob, partNumber: number): Promise<string> {
  const response = await fetch(url, {
    method: 'PUT',
    body: chunk,
    headers: { 'Content-Type': 'application/octet-stream' }
  });

  if (!response.ok) throw new Error(`HTTP ${response.status}`);
  
  // S3 returns the ETag wrapped in quotes. We MUST keep the quotes.
  const eTag = response.headers.get('ETag');
  if (!eTag) throw new Error('No ETag returned from S3');
  
  return eTag;
}

Phase 4: Validating ETags and Completing the Vault Entry

Once the frontend finishes uploading chunks, S3 still treats them as isolated, unlinked fragments. If you look in the AWS console, the file does not exist yet.

Your Node.js backend must issue a CompleteMultipartUploadCommand, passing the exact array of PartNumber and ETag combinations. S3 validates these ETags to ensure no data corruption occurred in transit, stitches the file together, and makes it available.

import { S3Client, CompleteMultipartUploadCommand } from '@aws-sdk/client-s3';

const s3Client = new S3Client({ region: process.env.AWS_REGION });

interface CompleteRequest {
  uploadId: string;
  objectKey: string;
  parts: { PartNumber: number; ETag: string }[];
}

export async function completeUpload(req: CompleteRequest) {
  // Parts array MUST be sorted by PartNumber in ascending order
  const sortedParts = req.parts.sort((a, b) => a.PartNumber - b.PartNumber);

  const command = new CompleteMultipartUploadCommand({
    Bucket: process.env.S3_VAULT_BUCKET,
    Key: req.objectKey,
    UploadId: req.uploadId,
    MultipartUpload: {
      Parts: sortedParts,
    },
  });

  try {
    const response = await s3Client.send(command);
    return { fileUrl: response.Location, eTag: response.ETag };
  } catch (error) {
    console.error('Failed to complete upload stitch:', error);
    // If completion fails due to invalid ETags, the upload is corrupted.
    throw new Error('Upload integrity verification failed');
  }
}

Production Note: The ETag returned by a completed multipart upload is not an MD5 hash of the entire file. It is a composite hash that typically looks like "hash-N", where N is the number of parts. Do not attempt to validate the final ETag against a local MD5 calculation of the original file.

Preventing Cost Leaks: Orphaned Multipart Uploads

What happens if a user starts uploading a 100GB dataset, closes their laptop at 50%, and never returns?

S3 keeps those raw parts sitting in storage indefinitely. You will be billed for storage of an invisible file. To prevent this, you must define an S3 Bucket Lifecycle rule that automatically aborts incomplete multipart uploads.

Here is a Terraform configuration snippet to enforce this lifecycle rule:

resource "aws_s3_bucket_lifecycle_configuration" "vault_lifecycle" {
  bucket = aws_s3_bucket.secure_vault.id

  rule {
    id     = "abort-incomplete-multipart-uploads"
    status = "Enabled"

    abort_incomplete_multipart_upload {
      days_after_initiation = 3
    }
  }
}

If you manage infrastructure manually, you can apply this via a JSON policy to ensure any orphaned parts are wiped after 72 hours.

Enforcing Zero-Trust with IAM and KMS Policies

Finally, issuing presigned URLs exposes a temporary vector to your bucket. To prevent a malicious actor from using a hijacked token to upload unencrypted malware, we enforce strict boundary policies at the bucket level.

We apply an S3 Bucket Policy that outright rejects any PutObject request (which covers multipart initiation) that doesn't explicitly request KMS encryption.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "RequireKMSEncryption",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::secure-document-vault/*",
      "Condition": {
        "StringNotEquals": {
          "s3:x-amz-server-side-encryption": "aws:kms"
        }
      }
    },
    {
      "Sid": "EnforceTLSRequestsOnly",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": "arn:aws:s3:::secure-document-vault/*",
      "Condition": {
        "Bool": {
          "aws:SecureTransport": "false"
        }
      }
    }
  ]
}

By layering client-side chunking, Node.js control-plane presigning, KMS encryption, and S3 lifecycle rules, we achieve a highly scalable, zero-trust document vault. Your backend memory stays flat regardless of file size, your bandwidth costs plummet, and your infrastructure remains resilient under heavy payload constraints.

Work With Us

Need help building this in production? SoftwareCrafting is a full-stack dev agency — we ship React, Next.js, Node.js, React Native & Flutter apps for global clients. When building scalable architectures at our backend development and API services division, handling massive data throughput securely is our bread and butter.

Get a Free Consultation →

Frequently Asked Questions

Why shouldn't I stream large file uploads directly through my Node.js backend?

Streaming large files through Express middleware like Multer creates massive backpressure, spikes CPU utilization, and exhausts the V8 garbage collector. This often leads to OOMKilled errors in Kubernetes and load balancer timeouts. Instead, the backend should only act as a control plane, keeping the actual file bytes off your server.

What are the AWS S3 limitations for multipart uploads?

AWS S3 requires every part of a multipart upload to be at least 5MB in size, with the exception of the final part. Additionally, a single file upload is strictly limited to a maximum of 10,000 total parts.

How can I prevent presigned URL generation from blocking the Node.js event loop?

Generating thousands of presigned URLs synchronously requires heavy HMAC-SHA256 computations that will quickly block the Node.js event loop. To avoid CPU starvation, you should expose an endpoint that dynamically generates these URLs in small, parallel batches as the client requests them.

How does SoftwareCrafting recommend securing direct-to-S3 file uploads?

SoftwareCrafting recommends enforcing server-side encryption at the exact moment of object creation using AWS Key Management Service (KMS). By passing ServerSideEncryption: 'aws:kms' and your KMS Key ID during the multipart upload initialization, you ensure enterprise-grade data encryption at rest.

Can SoftwareCrafting help architect scalable document vaults for healthcare or media platforms?

Yes, SoftwareCrafting specializes in building fault-tolerant, high-throughput backend architectures for enterprise use cases. We can help your team implement direct-to-S3 data planes that bypass Node.js memory limits, ensuring secure and scalable handling of gigabyte-scale medical datasets or media files.

Which AWS SDK commands are needed to initialize a direct-to-S3 multipart upload?

You need the AWS SDK v3 for Node.js, specifically utilizing the CreateMultipartUploadCommand from @aws-sdk/client-s3. This command tells S3 to expect an incoming file broken into numbered parts and returns the UploadId required for generating subsequent chunk URLs.

📎 Full Code on GitHub Gist: The complete upload-anti-pattern.ts from this post is available as a standalone GitHub Gist — copy, fork, or embed it directly.