Large Files & Memory: Streaming Without Buffering

A WebRTC file transfer has no server in the data path. The browser tab holds the file. That single fact decides how large a file you can move.

Send a 200 MB photo album and a naive implementation works. Send a 12 GB video and the tab dies. The transfer logic did not change. The memory budget did.

This article covers why buffering fails at scale, how to read the source lazily, how to write the received bytes to disk incrementally, and how to apply backpressure across the whole pipeline so neither side runs out of RAM. It assumes you already split the file into chunks (chunking), verify them (integrity), and understand channel delivery (reliability).

The problem: a multi-GB file will not fit in memory

A browser tab is not a server. It runs inside a memory budget the operating system and the browser enforce. On a 64-bit desktop Chrome that budget is often a few gigabytes per renderer process. On mobile it can be under a gigabyte. Cross it and the tab is killed with no recoverable error: the page just reloads or shows "Aw, Snap".

WebRTC moves the file peer-to-peer over RTCDataChannel. No cloud storage sits in the middle. The sending tab reads from local disk. The receiving tab writes to local disk. Between them, bytes pass through JavaScript heap memory. The danger is keeping too many of those bytes resident at once.

There are two independent ways to run out of memory. Both are easy to write by accident.

Failure 1: accumulating chunks on the receiver

The receiver gets chunks in onmessage. The obvious code collects them:

const parts = [];

channel.onmessage = (event) => {
  parts.push(event.data); // ArrayBuffer per chunk
};

channel.onclose = () => {
  const file = new Blob(parts); // assemble at the end
  download(file);
};

This holds the entire file in memory twice. First as an array of ArrayBuffer chunks. Then again when new Blob(parts) copies them into a single backing store before the originals can be freed. A 4 GB transfer needs roughly 8 GB resident at the moment of assembly. The tab dies long before onclose fires.

The Blob constructor can keep its backing store on disk in some engines, but you cannot rely on that, and the intermediate array of ArrayBuffers is always on the heap. The pattern is broken regardless.

There is a subtler version of the same bug that looks safe but is not. Suppose you grow a single Uint8Array and copy each chunk into it as it arrives:

let assembled = new Uint8Array(totalSize); // pre-allocate the whole file
let written = 0;

channel.onmessage = (event) => {
  const chunk = new Uint8Array(event.data);
  assembled.set(chunk, written); // copy into the big buffer
  written += chunk.byteLength;
};

This allocates the entire file in one contiguous heap buffer up front. For a 6 GB file the allocation itself fails: new Uint8Array(6e9) throws RangeError: Invalid array length or the allocation is refused by the engine. Pre-sizing the destination does not avoid the memory cost; it front-loads it. The only safe receiver never has the whole file resident at once.

Failure 2: reading the whole source at once

The sender has the same trap in reverse:

const buffer = await file.arrayBuffer(); // reads ALL bytes into RAM
for (let off = 0; off < buffer.byteLength; off += CHUNK) {
  channel.send(buffer.slice(off, off + CHUNK));
}

file.arrayBuffer() materializes every byte of the file in the heap before the first chunk is sent. A 12 GB file demands 12 GB up front. The slicing afterward is correct; the read is fatal.

The fix: never hold more than a few chunks

The principle is one sentence. At any instant, only a bounded window of the file is resident in memory: a handful of chunks in flight, not the whole file.

That requires three things working together:

1. Lazy reads on the sender: pull the next slice only when you are ready to send it.
2. Incremental writes on the receiver: push each chunk toward disk and forget it.
3. Backpressure: a feedback signal that slows the producer so the resident window stays bounded.

The rest of this article builds each part.

Reading the source lazily

The source on the sender is a File, which is a Blob with a name. Blob gives you two ways to read a window without touching the rest.

Option A: Blob.slice() on demand

Blob.slice(start, end) returns a new Blob describing a byte range. It does not read those bytes. It is a cheap view. The read happens when you call .arrayBuffer() on the slice.

const CHUNK = 256 * 1024; // 256 KiB

async function readSlice(file, offset, size) {
  const slice = file.slice(offset, offset + size);
  return await slice.arrayBuffer(); // reads only this window
}

This gives you random access. You decide when each window enters memory and you control the offset yourself. That matters for resumable transfers, where you may need to re-read from an arbitrary offset after a reconnect.

A send loop driven by slices:

async function sendFile(channel, file) {
  let offset = 0;
  while (offset < file.size) {
    const buffer = await readSlice(file, offset, CHUNK);
    channel.send(buffer);
    offset += buffer.byteLength;
    await drain(channel); // backpressure, defined below
  }
}

Each iteration holds exactly one chunk. The garbage collector reclaims the previous buffer once send() has copied it into the channel's send queue.

Option B: Blob.stream() / File.stream() with a reader

Blob.stream() returns a ReadableStream of Uint8Array chunks. The stream pulls from disk as you read it, so it is lazy by construction.

async function sendFileStreamed(channel, file) {
  const reader = file.stream().getReader();
  while (true) {
    const { value, done } = await reader.read();
    if (done) break;
    // value is a Uint8Array; its size is decided by the engine
    await sendInChunks(channel, value);
  }
}

The stream decides its own chunk size, which is usually larger than your wire chunk and not aligned to it. So sendInChunks must re-slice each value into wire-sized pieces:

async function sendInChunks(channel, bytes) {
  for (let off = 0; off < bytes.byteLength; off += CHUNK) {
    const piece = bytes.subarray(off, off + CHUNK);
    channel.send(piece);
    await drain(channel);
  }
}

subarray returns a view over the same buffer, not a copy, so this stays cheap. Note the difference from slice on a typed array: TypedArray.prototype.slice copies, subarray does not. Use subarray here because the underlying bytes stay valid until the next reader.read(), and send() copies them out anyway.

One caveat with streamed reads: the stream's chunk sizes are not stable. The engine may hand you a 64 KiB Uint8Array then a 1 MiB one, depending on disk timing and internal buffering. Your wire-chunk loop must not assume any particular value.byteLength. The re-slicing above handles that, but it means a single wire chunk can straddle two stream reads if you try to carry leftover bytes across iterations, so re-slice within each value and let the next value start fresh, accepting that wire chunks align to stream boundaries. If you need every wire chunk to be exactly CHUNK bytes (for example to match a fixed-size framing on the receiver), buffer leftovers across reads explicitly. For most transfers, variable trailing chunks are fine because the receiver writes whatever it gets.

Slice vs stream

For a peer-to-peer transfer that may survive an ICE restart or a tab refresh, slicing wins because you can resume from a known offset. Stream reading is simpler when the transfer is single-pass and you accept restarting on failure.

Writing the received data incrementally

The receiver is where buffering does the most damage, because chunks arrive faster than naive code disposes of them. The goal: take each chunk from onmessage and move it toward disk without ever holding the whole file.

The right tool depends on what the browser supports. Detect first, then choose.

The File System Access API (write straight to disk)

showSaveFilePicker() lets the user pick a destination file. createWritable() opens a stream to that file on disk. You write each chunk and the bytes leave the heap. This is the only approach that keeps receiver memory flat regardless of file size.

async function receiveToDisk(channel, fileName) {
  const handle = await window.showSaveFilePicker({
    suggestedName: fileName,
  });
  const writable = await handle.createWritable();

  channel.onmessage = async (event) => {
    await writable.write(event.data); // chunk goes to disk
  };

  channel.addEventListener('close', async () => {
    await writable.close(); // flush and finalize
  });
}

writable.write() accepts an ArrayBuffer, a TypedArray, a Blob, or a string. It returns a promise that resolves when the chunk is queued for the disk write. Each resolved write frees the chunk.

You can also write at an explicit offset, which lines up with resumable transfers:

await writable.write({ type: 'write', position: offset, data: chunk });

createWritable() writes to a temporary file by default and swaps it into place on close(). If the transfer aborts, call writable.abort() so the temp file is discarded rather than left half-written.

One detail trips people up: createWritable() opens with keepExistingData: false by default, which truncates the target file to zero length the moment it opens. For a resumable transfer where you want to keep already-written bytes and append from an offset, pass { keepExistingData: true } and write at an explicit position. For a fresh transfer the default is what you want.

Writes are not guaranteed durable until close() resolves. The intermediate write() calls queue data; the swap-into-place and final flush happen on close. So an interrupted transfer leaves no partial file in the user's chosen location: the temp file is discarded on abort() or on a crash. Treat await writable.close() as the commit point and surface failure there.

The receiver as a WritableStream consumer

A cleaner shape pipes incoming chunks through a WritableStream, because a WritableStream carries its own backpressure signal (covered in the next section). The file handle gives you one directly:

async function openSink(fileName) {
  const handle = await window.showSaveFilePicker({ suggestedName: fileName });
  const writable = await handle.createWritable();
  return writable; // already a WritableStream
}

You then feed chunks into a writer and respect its ready promise, which is the backpressure hook.

Fallback 1: assemble a Blob (small to medium files)

When the File System Access API is missing (Firefox and Safari do not expose showSaveFilePicker at the time of writing), fall back to assembling a Blob and triggering a download via an object URL.

async function receiveToBlob(channel) {
  const parts = [];

  channel.onmessage = (event) => {
    parts.push(event.data);
  };

  return new Promise((resolve) => {
    channel.addEventListener('close', () => {
      const blob = new Blob(parts);
      const url = URL.createObjectURL(blob);
      triggerDownload(url, 'received.bin');
      URL.revokeObjectURL(url); // free the URL; the click already started
      resolve();
    });
  });
}

function triggerDownload(url, name) {
  const a = document.createElement('a');
  a.href = url;
  a.download = name;
  a.click();
}

This is Failure 1 from earlier, and that is the point. The Blob fallback only works for files that fit in the memory budget twice over. Cap it. A practical ceiling is a few hundred megabytes on desktop and far less on mobile. Refuse larger transfers in this mode rather than crashing:

const BLOB_CEILING = 500 * 1024 * 1024; // 500 MB
if (declaredSize > BLOB_CEILING && !hasFileSystemAccess()) {
  throw new Error('File too large for this browser; use Chrome or Edge.');
}

URL.createObjectURL keeps the blob alive until you revoke the URL or the document unloads. Revoke it right after the download click starts so the memory can be reclaimed.

Fallback 2: a service-worker download stream

To stream large files to disk in browsers without the File System Access API, route the download through a service worker. The worker registers a virtual URL and answers the fetch for it with a ReadableStream whose body you feed chunk by chunk. The browser treats it as a normal download and writes to disk as bytes arrive, so the page never holds the whole file.

This is the technique the StreamSaver.js library popularized. The shape:

1. Register a service worker.
2. Navigate or open an iframe to a URL the worker intercepts.
3. The worker responds with a Response whose body is a ReadableStream.
4. The page posts chunks to the worker via postMessage; the worker enqueues them into the stream.
5. The browser's download manager pulls from the stream and writes to disk.

// page side, simplified
const fileStream = streamSaver.createWriteStream('big.bin', { size });
const writer = fileStream.getWriter();

channel.onmessage = async (event) => {
  await writer.write(new Uint8Array(event.data));
};

channel.addEventListener('close', () => writer.close());

writer.write() resolves when the worker accepts the chunk, giving you backpressure for free. The cost is a service worker registration and, in some setups, an HTTPS origin and a MITM iframe for cross-origin cases. Use this path only when the File System Access API is absent and files exceed the Blob ceiling.

Feature detection and the decision tree

Choose the sink once, up front, from capabilities and declared size.

function hasFileSystemAccess() {
  return typeof window.showSaveFilePicker === 'function';
}

async function chooseSink({ fileName, size }) {
  if (hasFileSystemAccess()) {
    return openSink(fileName); // streams to disk, any size
  }
  if (size <= BLOB_CEILING) {
    return blobSink(fileName); // assemble + object URL
  }
  if (serviceWorkerStreamingAvailable()) {
    return swStreamSink(fileName, size); // streamed download
  }
  throw new Error('No memory-safe sink for this file in this browser.');
}

Detect at runtime, not by user-agent string. The matrix above is guidance; capability checks are truth.

Applying backpressure end-to-end

Lazy reads and incremental writes bound memory only if the producer slows to match the slowest consumer. Without backpressure, the sender reads and sends as fast as the loop runs, the channel's send buffer grows without limit, and you are back to holding the file in memory, this time inside the WebRTC send queue.

There are two backpressure signals. A complete transfer respects both.

Channel backpressure: bufferedAmount

RTCDataChannel.bufferedAmount is the number of bytes queued for sending but not yet handed to the network. Every send() adds to it. The transport drains it as it transmits. If you send faster than the link drains, it climbs without bound, and that queue lives in memory.

Set a high-water mark and pause when you cross it. The channel fires bufferedamountlow when it drops back under bufferedAmountLowThreshold.

const HIGH_WATER = 8 * 1024 * 1024; // 8 MB queued max
const LOW_WATER = 1 * 1024 * 1024;  // resume under 1 MB

channel.bufferedAmountLowThreshold = LOW_WATER;

function drain(channel) {
  if (channel.bufferedAmount < HIGH_WATER) {
    return Promise.resolve(); // room to send now
  }
  return new Promise((resolve) => {
    channel.addEventListener('bufferedamountlow', resolve, { once: true });
  });
}

The drain(channel) call in the send loops earlier is this function. It returns immediately while there is room and blocks the loop until the queue drains when there is not. The resident send window stays under HIGH_WATER plus one chunk.

Do not poll bufferedAmount in a tight loop or with setTimeout. The bufferedamountlow event is the supported, efficient signal.

Sink backpressure: the WritableStream ready promise

The receiver has its own limit: how fast it can write to disk. A WritableStream writer exposes writer.ready, a promise that resolves when the stream can accept more without growing its internal queue. Await it before each write.

const writer = sink.getWriter();

channel.onmessage = async (event) => {
  await writer.ready;          // wait until the sink wants more
  await writer.write(event.data);
};

If the disk is slow, writer.ready stays pending, your onmessage handler stalls, incoming chunks are not consumed, and SCTP's own flow control eventually pushes back across the connection to the sender. That is the end-to-end chain: slow disk on the receiver → stalled writer → SCTP receive window shrinks → sender's bufferedAmount climbs → sender's drain() pauses the read loop. Memory stays bounded on both ends without either side knowing the other's speed.

Why both signals are needed

bufferedAmount protects the sender's memory. writer.ready protects the receiver's memory. Skip the first and the sender's queue grows. Skip the second and the receiver's write queue grows. The connection's transport propagates pressure between the two, but only if both ends actually pause when their local signal tells them to.

Negotiating the sink before bytes flow

The sender and receiver must agree on the transfer before any chunk moves, because the receiver's choice of sink depends on the file size and the receiver's capabilities, and the sender needs to know whether to proceed at all.

A minimal handshake exchanges metadata first, over the same channel, as a control message:

// sender announces the file
channel.send(JSON.stringify({
  t: 'offer-file',
  name: file.name,
  size: file.size,
  mime: file.type,
}));

The receiver inspects the size against its own capabilities and answers:

channel.onmessage = (event) => {
  if (typeof event.data !== 'string') return; // binary chunk, handled elsewhere
  const msg = JSON.parse(event.data);
  if (msg.t === 'offer-file') {
    const sink = chooseSink({ fileName: msg.name, size: msg.size });
    if (!sink) {
      channel.send(JSON.stringify({ t: 'reject-file', reason: 'too-large' }));
      return;
    }
    channel.send(JSON.stringify({ t: 'accept-file' }));
    // open the sink, then begin consuming binary chunks
  }
};

This is also where you mix string control messages and binary chunks on one channel. Test typeof event.data === 'string' to route them. The sender waits for accept-file before reading the first slice. If the receiver rejects (too large for its sink, user declined the save dialog), the sender never starts, and neither side wasted memory.

Carry the size into the receiver's progress display and into the streaming sink's expected length. The service-worker stream in particular wants a size hint so the browser's download manager can show a determinate progress bar.

Tuning chunk size for throughput

Chunk size is a tradeoff, and the right value differs for large transfers versus small messages. See chunking for the framing details; this section is about the memory and throughput angle.

Smaller chunks mean more send() calls, more per-message overhead, and more event-loop turns: lower throughput. Larger chunks mean fewer calls and higher throughput, but more memory per resident chunk and a hard ceiling set by the transport.

The transport ceiling matters. SCTP, the protocol under RTCDataChannel, has a maximum message size negotiated per connection (pc.sctp.maxMessageSize). Send a single message larger than it and the channel errors or closes. Historically 16 KiB was the safe interoperable maximum; modern browsers negotiate much higher, often 256 KiB or into the megabytes, but the value is per-connection and you must respect it.

function safeChunkSize(pc, preferred = 256 * 1024) {
  const max = pc.sctp?.maxMessageSize ?? 16 * 1024;
  return Math.min(preferred, max);
}

Practical guidance for large files:

Pick the largest chunk that (a) stays under maxMessageSize and (b) keeps your resident window (chunk size times the in-flight count) comfortably inside the memory budget. With an 8 MB HIGH_WATER and 256 KiB chunks, at most ~32 chunks plus a couple in hand are resident: a few megabytes, flat, for any file size.

There is a second reason not to oversize chunks beyond the message limit: a single oversized message is all-or-nothing. SCTP delivers a message whole or not at all. A 4 MB message that must be retransmitted retransmits all 4 MB on a single loss. Smaller messages limit the retransmission cost of any one loss, which on a lossy link can matter more than per-message overhead. The sweet spot for large transfers on a typical connection sits around 256 KiB: large enough to amortize overhead, small enough that a loss is cheap to recover and the resident window stays small.

The high-water mark is a second tuning knob, independent of chunk size. A larger HIGH_WATER lets more bytes sit in the send queue, which smooths over short link stalls and can raise throughput, at the cost of more resident memory and a longer pause when backpressure finally triggers. An 8 MB window is a reasonable default; raise it only if profiling shows the link starving for data, and never so high that the queue alone threatens the memory budget.

Progress and ETA

Streaming transfers run long enough that a progress indicator is not optional. You have the numbers already: bytes sent versus total size on the sender, bytes written versus declared size on the receiver.

function makeProgress(total) {
  let sent = 0;
  let lastBytes = 0;
  let lastTime = performance.now();

  return function update(chunkBytes) {
    sent += chunkBytes;
    const now = performance.now();
    const dt = (now - lastTime) / 1000;

    if (dt >= 0.5) { // sample rate over a window, not per chunk
      const rate = (sent - lastBytes) / dt; // bytes/sec
      const remaining = total - sent;
      const etaSeconds = rate > 0 ? remaining / rate : Infinity;
      lastBytes = sent;
      lastTime = now;
      return { fraction: sent / total, rate, etaSeconds };
    }
    return { fraction: sent / total };
  };
}

Compute the rate over a moving window of half a second or so, not from a single chunk. Per-chunk rates jump wildly because chunks send in bursts between backpressure pauses. A windowed rate gives a stable ETA.

Show bytes transferred, percentage, current rate, and ETA. The receiver should display progress from its own write count, since that reflects bytes actually committed to disk, not just bytes the sender pushed.

Memory caveats and browser limits

A few facts that change behavior at scale.

Per-tab memory is finite and not introspectable. performance.memory exists only in Chromium, is coarse, and is gated. You cannot reliably read the budget at runtime. Design to stay well under any plausible limit rather than measuring it.

The Blob constructor copies. new Blob(parts) allocates a new backing store and copies every part in. At the instant of construction you need the parts plus the new blob resident. This is why the Blob fallback has a hard ceiling.

Object URLs leak until revoked. URL.createObjectURL(blob) pins the blob in memory for the document's lifetime unless you call URL.revokeObjectURL. Revoke as soon as the download has started.

Mobile budgets are small and enforced harshly. iOS Safari in particular kills tabs that grow past a low threshold, often under a gigabyte, with no warning. Treat mobile as the constraining case. The Blob fallback ceiling there should be tens of megabytes, not hundreds.

The receiver controls the sink, the sender does not. The sender cannot know whether the receiver streams to disk or buffers a Blob. If the transfer must support arbitrary sizes, the receiver must use a streaming sink or refuse the file up front and tell the sender. Negotiate this in your handshake metadata.

createWritable may need a permission and a user gesture. showSaveFilePicker() must be called from a user activation (a click). You cannot open the sink automatically when a chunk arrives; prompt for the destination before the transfer starts.

Recap

A peer-to-peer file transfer holds the file in the tab, so memory is the binding constraint, not bandwidth or storage.

• Buffering the whole file (as an array of chunks, one big Blob, or via file.arrayBuffer()) fails at scale. Keep only a bounded window resident.
• Read the source lazily with Blob.slice() (random access, resumable) or Blob.stream() (forward-only, simpler).
• Write the received bytes incrementally. Prefer the File System Access API (showSaveFilePicker → createWritable → write → close) for flat memory at any size. Fall back to a capped Blob for small files, or a service-worker download stream for large files where the API is missing.
• Apply backpressure on both ends: bufferedAmount with bufferedamountlow on the sender, writer.ready on the receiver. The transport propagates pressure between them.
• Size chunks as large as pc.sctp.maxMessageSize and your memory window allow; 256 KiB is a good default when negotiated.
• Sample transfer rate over a time window for a stable ETA.

Going further

• Chunking: framing, sequence numbers, and reassembly order.
• Integrity: hashing chunks and the whole file so corruption is caught.
• Reliability: ordered vs unordered delivery and what RTCDataChannel guarantees.

Troubleshooting