Guide

Web Streams API explained

Harbor Commerce's admin dashboard offered a “Download all orders (CSV)” button that fetched the full export into a single string, concatenated rows in JavaScript, then triggered a blob download. On a merchant with 4.2 million line items, the tab allocated 2.4 GB of RAM and crashed on most corporate laptops before the save dialog appeared. Rewriting the path with the Web Streams API — piping the fetch() response body through a TransformStream that formats CSV rows on the fly, then into a Response consumed by the File System Access API fallback — held peak memory at 18 MB and let downloads start within two seconds of clicking export.

Browsers process enormous amounts of data: HTTP bodies, video frames, file uploads, and worker messages. Loading everything into one array or string does not scale. The Web Streams API models I/O as sequences of chunks flowing through ReadableStream, WritableStream, and TransformStream objects with explicit backpressure: slow consumers automatically slow producers. This guide explains stream types and reader modes, piping with pipeTo and pipeThrough, streaming fetch responses and uploads, built-in transform streams (compression, text decoding), the Harbor Commerce refactor, a technique decision table versus WebSockets and SSE, common pitfalls, and a production checklist alongside our Web Workers guide and service workers guide.

Why buffers break in the browser

The classic pattern — await response.text() or await response.arrayBuffer() — is fine for kilobyte JSON payloads. It fails when:

Exports and logs exceed available heap (multi-gigabyte CSV, NDJSON audit trails).
Video or audio should play before the full file downloads.
Uploads should start sending while the user still selects files, not after reading entire blobs into memory.
Progress UI needs byte counts during transfer, not after.

Streams solve this by processing chunks (typically Uint8Array for bytes or strings for text) as they arrive. Memory stays bounded by buffer high-water marks, not total payload size. The same abstraction powers fetch bodies, ReadableStream constructors in libraries, and the WHATWG-standard pipe operators that connect stages.

Backpressure in one sentence

When a WritableStream (or transform's writable side) is full, write() returns a promise that resolves only when space opens. Upstream producers await that promise before enqueueing more chunks. Without backpressure, fast networks flood memory faster than your parser can run.

Core stream types

ReadableStream

A source of data. Consumers acquire a reader:

Default reader (getReader()) — exclusive access; each read() returns { value, done }.
BYOB reader (getReader({ mode: 'byob' })) — “bring your own buffer” for zero-copy byte fills into a pre-allocated Uint8Array; useful for binary parsers.

response.body on a fetch result is a ReadableStream<Uint8Array> once the headers arrive. You can iterate with for await (const chunk of stream) in modern browsers if the stream is async-iterable (native fetch bodies are).

WritableStream

A sink. Call getWriter(), then writer.write(chunk) and writer.close(). Writers are exclusive — release locks with writer.releaseLock() before piping elsewhere. Service worker respondWith(new Response(stream)) and file downloads built from streams both terminate in writables provided by the platform.

TransformStream

A readable/writable pair with a transform hook invoked per chunk. Implement CSV row formatting, gzip compression, JSON line parsing, or encryption without materializing the full dataset. Chain transforms with pipeThrough().

const { readable, writable } = new TransformStream({
  transform(chunk, controller) {
    const rows = decoder.decode(chunk).split('\n');
    for (const row of rows) {
      if (row) controller.enqueue(formatCsvRow(row));
    }
  }
});
await fetch('/api/orders/export').then(r => r.body
  .pipeThrough(new TextDecoderStream())
  .pipeThrough(csvTransform)
  .pipeTo(fileWritable));

Piping: pipeTo, pipeThrough, tee

source.pipeTo(destination) connects a readable directly to a writable, handling backpressure and errors. It returns a promise that resolves when the source closes or rejects on failure. Prefer piping over manual read() loops — less boilerplate, correct cancellation via AbortSignal.

pipeThrough(transform) inserts a middle stage and returns a new readable you can continue chaining. Built-in transforms include:

TextDecoderStream / TextEncoderStream — bytes to UTF-8 strings and back.
CompressionStream / DecompressionStream — gzip or deflate on the fly (check browser support; polyfill or WASM fallback on older Safari).
IdentityTransformStream — debugging passthrough.

readable.tee() splits one source into two identical branches (e.g. hash while downloading). Both branches must keep pace or backpressure stalls the shared source — use only when both consumers are fast.

Cancellation and abort

Pass { signal: abortController.signal } to pipeTo. When the user cancels a download, abort the controller; the stream chain propagates cancellation upstream, allowing the server to stop generating rows if it respects closed connections.

Streaming fetch patterns

Consuming large downloads

Never call .text() on a multi-megabyte export. Instead:

Start fetch(url) and read response.headers.get('Content-Length') for progress bars.
Pipe response.body through parsers/transforms.
Terminate in WritableStream that writes to disk (File System Access API) or accumulates only a sliding window.

For progressive JSON arrays (newline-delimited JSON), pipe through a transform that buffers incomplete lines — do not JSON.parse the entire body. Libraries like stream-json in Node mirror this pattern; in the browser, a 64 KB line buffer is usually enough.

Streaming uploads

fetch('/upload', { method: 'POST', body: readableStream, duplex: 'half' }) sends chunks as they are produced. Useful for screen recordings and log tail uploads. Not every CDN or reverse proxy supports request bodies without Content-Length; test nginx proxy_request_buffering off and HTTP/2 end-to-end. When unsupported, fall back to multipart or chunked uploads via presigned S3 URLs.

SSE without EventSource

When you need custom headers (Bearer tokens) on server-sent events, use fetch plus a readable body parser instead of EventSource — the pattern mentioned in our WebSockets and SSE guide. Pipe bytes through TextDecoderStream, split on \n\n event boundaries, and parse data: lines manually.

Harbor Commerce export refactor

Requirements: merchants export up to five years of orders; exports must work on 8 GB RAM Windows laptops; progress indicator required; cancel must stop server work within five seconds.

Before

GET /api/orders/export?format=csv buffered entirely in Node, sent as one response.
Client await response.text(), split lines, re-encode as Blob.
Peak client memory 2.4 GB; 34% of export attempts OOM-crashed the tab.

After (streaming end-to-end)

Server: Node Readable converted to Web Stream via Readable.toWeb(); cursor-paginated SQL with 10k-row chunks.
Client: fetch body piped through TextDecoderStream → custom TransformStream adding CSV headers on first chunk → CompressionStream('gzip') when browser supports it.
Download: showSaveFilePicker when available; fallback URL.createObjectURL on a Response(readable) without buffering.
Progress: byte counter from TransformStream enqueue size vs Content-Length when known, else indeterminate spinner.
Cancel: AbortController on fetch and pipeTo; server detects closed socket and stops pagination.

Result: peak RAM 18 MB, time-to-first-byte 1.8 s, crash rate 0%. Gzip streaming cut transfer size 71% on repetitive CSV. The same pipe later powered a “live tail” fraud-alerts view without architectural changes.

Technique decision table

Approach	Best for	Memory profile	Complexity	Browser support
Buffer entire body (`.json()`, `.text()`)	Small API payloads (< few MB)	O(total size)	Low	Universal
Web Streams + `pipeThrough`	Large downloads, transforms, gzip	O(chunk buffer)	Medium	All modern browsers
`EventSource` SSE	Server push, text events, auto-reconnect	Low per event	Low	Universal; no custom headers
WebSockets	Bidirectional binary/text, games, chat	Depends on app	Medium–high	Universal
Web Worker + streams	Parse/decode off main thread	Isolated heap	Medium	High (transferable streams)
Service worker cache stream	Offline progressive cache	Disk + small RAM	High	PWA contexts

Default to buffering for small JSON. Reach for Web Streams when payload size or latency-to-first-byte matters. Pair streams with Web Workers when parsing would block the main thread for more than a few milliseconds per chunk.

Common pitfalls

Forgetting to release locks — piping fails if a reader or writer lock is still held; always releaseLock() before re-piping.
Mixing async iteration and readers — do not call getReader() and for await on the same stream.
Unhandled stream errors — a rejected pipeTo aborts the chain; wrap in try/catch and reset UI state.
Assuming Content-Length — chunked transfer encoding omits length; design progress UI for unknown totals.
Text split across chunks — UTF-8 multi-byte characters and CSV rows can span chunks; maintain a carry buffer in transforms.
Upload streaming blocked by proxy — test duplex fetch through production CDN; many buffer request bodies.
tee() stall — one slow branch blocks both; prefer single-consumer pipelines or duplicate at source.
Mobile tab suspension — background tabs throttle timers and network; warn users to keep export tab focused.

Production checklist

Measure payload P95 size; stream anything above ~10 MB or with unknown size.
Use pipeTo / pipeThrough instead of manual read loops when possible.
Wire AbortController to cancel buttons and route signal through fetch and pipeTo.
Handle cross-chunk boundaries in text transforms; add unit tests with split UTF-8 sequences.
Feature-detect CompressionStream and File System Access API; ship fallbacks.
Log stream errors with chunk index and bytes processed for support triage.
Server-side: paginate or cursor exports; stop SQL when client disconnects.
Test on 8 GB RAM devices and throttled “Slow 3G” network profiles.
Document maximum export row count and offer async email delivery for edge cases.

Key takeaways

Web Streams process data in chunks with automatic backpressure — memory stays bounded regardless of total payload size.
ReadableStream, WritableStream, and TransformStream compose via pipeTo and pipeThrough; fetch response bodies are readable streams out of the box.
Built-in TextDecoderStream and CompressionStream cover common transform stages without custom code.
Harbor-style large exports should stream from database cursor through HTTP to disk without ever calling .text() on the full body.
Pair streams with AbortSignal cancellation and Worker offloading when parsing cost threatens frame budgets.