Core ModulesIntermediate⏱ 7 min06 / 11

Streams & Buffers

Process data piece by piece instead of all at once — the key to handling big files and network data efficiently.

What if you need to read a 5 GB file? Loading it fully into memory would blow up. Streams let you process data in small chunks as it arrives — constant memory, no matter the size. They're everywhere in Node: files, HTTP requests/responses, compression, and more.

#Buffers: raw bytes

A Buffer is a fixed-length chunk of raw binary data (bytes). Streams hand you Buffers; you turn them into text with .toString() or an encoding. Binary data (images, video, network packets) lives in Buffers.

reading a file as a stream
const fs = require('node:fs');

const stream = fs.createReadStream('big.log', 'utf8');

stream.on('data', (chunk) => {
  console.log('got', chunk.length, 'chars');
});
stream.on('end', () => console.log('done'));
stream.on('error', (err) => console.error(err));

Instead of one giant read, you get many 'data' events, each a manageable chunk. 'end' fires when the source is exhausted; 'error' if something fails. This is a Readable stream — data flows out of it.

#Piping

pipe a readable into a writable
const fs = require('node:fs');

// copy a file chunk-by-chunk, with automatic backpressure
fs.createReadStream('input.txt')
  .pipe(fs.createWriteStream('output.txt'));
Tip

pipe() handles backpressure

.pipe() connects a Readable to a Writable and automatically manages backpressure — if the destination can't keep up, it pauses the source. It's the idiomatic, safe way to move stream data (a web server streaming a file to a client is exactly this).

Quick check

Why use a stream to read a very large file instead of fs.readFile?

Key takeaways

  • Streams process data in chunks over time, keeping memory usage flat for any size.
  • A Buffer is a fixed block of raw bytes; use .toString(encoding) to read it as text.
  • Readable streams emit 'data', 'end', and 'error' events.
  • `.pipe()` connects a readable to a writable and handles backpressure automatically.
  • Files, HTTP, and compression are all built on streams.
Practice challenges
Test yourself · earn XP
0/3
Predict the output#1

For a 5 GB file, which approach keeps memory usage low?

predict-output
// A: fs.readFile('big', cb)
// B: fs.createReadStream('big').on('data', chunk => ...)
Fill in the blank#2

Complete the idiomatic way to copy one file to another chunk-by-chunk.

fs.createReadStream('a').(fs.createWriteStream('b'));
Predict the output#3

What does a Readable stream emit when the source has no more data?

predict-output
stream.on('????', () => console.log('finished reading'));
Your turn
Practice exercise

Using streams, how would you copy 'a.txt' to 'b.txt' without ever holding the whole file in memory? Write the one-liner.

Try it yourself — a starting point to build on:

starter.js
# Write your solution here