# Stream
Stability: 2 Stable
A stream is an abstract interface for working with streaming data in Node.js.
The `stream` module provides a base API that makes it easy to build objects
that implement the stream interface.
There are many stream objects provided by Node.js. For instance, a
[request to an HTTP server][http-incoming-message] and [`process.stdout`][]
are both stream instances.
Streams can be readable, writable, or both. All streams are instances of
[`EventEmitter`][].
The `stream` module can be accessed using:
`js
const stream = require(stream);
`
While it is important for all Node.js users to understand how streams works,
the `stream` module itself is most useful for developers that are creating new
types of stream instances. Developers who are primarily *consuming* stream
objects will rarely (if ever) have need to use the `stream` module directly.
## Organization of this document
This document is divided into two primary sections and third section for
additional notes. The first section explains the elements of the stream API that
are required to *use* streams within an application. The second section explains
the elements of the API that are required to *implement* new types of streams.
## Types of Streams
There are four fundamental stream types within Node.js:
* [Readable][] streams from which data can be read (for example
[`fs.createReadStream()`][]).
* [Writable][] streams to which data can be written (for example
[`fs.createWriteStream()`][]).
* [Duplex][] streams that are both Readable and Writable (for example
[`net.Socket`][]).
* [Transform][] Duplex streams that can modify or transform the data as it
is written and read (for example [`zlib.createDeflate()`][]).
### Object Mode
All streams created by Node.js APIs operate exclusively on strings and `Buffer`
objects. It is possible, however, for stream implementations to work with other
types of JavaScript values (with the exception of `null` which serves a special
purpose within streams). Such streams are considered to operate in object
mode.
Stream instances are switched into object mode using the `objectMode` option
when the stream is created. Attempting to switch an existing stream into
object mode is not safe.
### Buffering
Both [Writable][] and [Readable][] streams will store data in an internal
buffer that can be retrieved using `writable._writableState.getBuffer()` or
`readable._readableState.buffer`, respectively.
The amount of data potentially buffered depends on the `highWaterMark` option
passed into the streams constructor. For normal streams, the `highWaterMark`
option specifies a total number of bytes. For streams operating in object mode,
the `highWaterMark` specifies a total number of objects.
Data is buffered in Readable streams when the implementation calls
[`stream.push(chunk)`][stream-push]. If the consumer of the Stream does not
call [`stream.read()`][stream-read], the data will sit in the internal
queue until it is consumed.
Once the total size of the internal read buffer reaches the threshold specified
by `highWaterMark`, the stream will temporarily stop reading data from the
underlying resource until the data currently buffered can be consumed (that is,
the stream will stop calling the internal `readable._read()` method that is
used to fill the read buffer).
Data is buffered in Writable streams when the
[`writable.write(chunk)`][stream-write] method is called repeatedly. While the
total size of the internal write buffer is below the threshold set by
`highWaterMark`, calls to `writable.write()` will return `true`. Once the
the size of the internal buffer reaches or exceeds the `highWaterMark`, `false`
will be returned.
A key goal of the `stream` API, and in particular the [`stream.pipe()`] method,
is to limit the buffering of data to acceptable levels such that sources and
destinations of differing speeds will not overwhelm the available memory.
Because [Duplex][] and [Transform][] streams are both Readable and Writable,
each maintain *two* separate internal buffers used for reading and writing,
allowing each side to operate independently of the other while maintaining an
appropriate and efficient flow of data. For example, [`net.Socket`][] instances
are [Duplex][] streams whose Readable side allows consumption of data received
*from* the socket and whose Writable side allows writing data *to* the socket.
Because data may be written to the socket at a faster or slower rate than data
is received, it is important each side operate (and buffer) independently of
the other.
## API for Stream Consumers
Almost all Node.js applications, no matter how simple, use streams in some
manner. The following is an example of using streams in a Node.js application
that implements an HTTP server:
`js
const http = require(http);
const server = http.createServer( (req, res) => {
// req is an http.IncomingMessage, which is a Readable Stream
// res is an http.ServerResponse, which is a Writable Stream
let body = ;
// Get the data as utf8 strings.
// If an encoding is not set, Buffer objects will be received.
req.setEncoding(utf8);
// Readable streams emit data events once a listener is added
req.on(data, (chunk) => {
body += chunk;
});
// the end event indicates that the entire body has been received
req.on(end, () => {
try {
const data = JSON.parse(body);
} catch (er) {
// uh oh!bad json!
res.statusCode = 400;
return res.end(`error: ${er.message}`);
}
// write back something interesting to the user:
res.write(typeof data);
res.end();
});
});
server.listen(1337);
// $ curl localhost:1337 -d {}
// object
// $ curl localhost:1337 -d foo
// string
// $ curl localhost:1337 -d not json
// error: Unexpected token o
`
[Writable][] streams (such as `res` in the example) expose methods such as
`write()` and `end()` that are used to write data onto the stream.
[Readable][] streams use the [`EventEmitter`][] API for notifying application
code when data is available to be read off the stream. That available data can
be read from the stream in multiple ways.
Both [Writable][] and [Readable][] streams use the [`EventEmitter`][] API in
various ways to communicate the current state of the stream.
[Duplex][] and [Transform][] streams are both [Writable][] and [Readable][].
Applications that are either writing data to or consuming data from a stream
are not required to implement the stream interfaces directly and will generally
have no reason to call `require(stream)`.
Developers wishing to implement new types of streams should refer to the
section [API for Stream Implementers][].
### Writable Streams
Writable streams are an abstraction for a *destination* to which data is
written.
Examples of [Writable][] streams include:
* [HTTP requests, on the client][]
* [HTTP responses, on the server][]
* [fs write streams][]
* [zlib streams][zlib]
* [crypto streams][crypto]
* [TCP sockets][]
* [child process stdin][]
* [`process.stdout`][], [`process.stderr`][]
*Note*: Some of these examples are actually [Duplex][] streams that implement
the [Writable][] interface.
All [Writable][] streams implement the interface defined by the
`stream.Writable` class.
While specific instances of [Writable][] streams may differ in various ways,
all Writable streams follow the same fundamental usage pattern as illustrated
in the example below:
`js
const myStream = getWritableStreamSomehow();
myStream.write(some data);
myStream.write(some more data);
myStream.end(done writing data);
`
#### Class: stream.Writable
##### Event: close
The `close` event is emitted when the stream and any of its underlying
resources (a file descriptor, for example) have been closed. The event indicates
that no more events will be emitted, and no further computation will occur.
Not all Writable streams will emit the `close` event.
##### Event: drain
If a call to [`stream.write(chunk)`][stream-write] returns `false`, the
`drain` event will be emitted when it is appropriate to resume writing data
to the stream.
`js
// Write the data to the supplied writable stream one million times.
// Be attentive to back-pressure.
function writeOneMillionTimes(writer, data, encoding, callback) {
let i = 1000000;
write();
function write() {
var ok = true;
do {
i;
if (i === 0) {
// last time!
writer.write(data, encoding, callback);
} else {
// see if we should continue, or wait
// dont pass the callback, because were not done yet.
ok = writer.write(data, encoding);
}
} while (i > 0 && ok);
if (i > 0) {
// had to stop early!
// write some more once it drains
writer.once(drain, write);
}
}
}
`
##### Event: error
* {Error}
The `error` event is emitted if an error occurred while writing or piping
data. The listener callback is passed a single `Error` argument when called.
*Note*: The stream is not closed when the `error` event is emitted.
##### Event: finish
The `finish` event is emitted after the [`stream.end()`][stream-end] method
has been called, and all data has been flushed to the underlying system.
`js
const writer = getWritableStreamSomehow();
for (var i = 0; i < 100; i ++) {writer.write(‘hello, #${i}!
‘);}writer.end(‘This is the end
‘);writer.on(‘finish’, () => {
console.error(All writes are now complete.);
});
`
##### Event: pipe
* `src` {stream.Readable} source stream that is piping to this writable
The `pipe` event is emitted when the [`stream.pipe()`][] method is called on
a readable stream, adding this writable to its set of destinations.
`js
const writer = getWritableStreamSomehow();
const reader = getReadableStreamSomehow();
writer.on(pipe, (src) => {
console.error(something is piping into the writer);
assert.equal(src, reader);
});
reader.pipe(writer);
`
##### Event: unpipe
* `src` {[Readable][] Stream} The source stream that
[unpiped][`stream.unpipe()`] this writable
The `unpipe` event is emitted when the [`stream.unpipe()`][] method is called
on a [Readable][] stream, removing this [Writable][] from its set of
destinations.
`js
const writer = getWritableStreamSomehow();
const reader = getReadableStreamSomehow();
writer.on(unpipe, (src) => {
console.error(Something has stopped piping into the writer.);
assert.equal(src, reader);
});
reader.pipe(writer);
reader.unpipe(writer);
`
##### writable.cork()
The `writable.cork()` method forces all written data to be buffered in memory.
The buffered data will be flushed when either the [`stream.uncork()`][] or
[`stream.end()`][stream-end] methods are called.
The primary intent of `writable.cork()` is to avoid a situation where writing
many small chunks of data to a stream do not cause an backup in the internal
buffer that would have an adverse impact on performance. In such situations,
implementations that implement the `writable._writev()` method can perform
buffered writes in a more optimized manner.
##### writable.end([chunk][, encoding][, callback])
* `chunk` {String|Buffer|any} Optional data to write. For streams not operating
in object mode, `chunk` must be a string or a `Buffer`. For object mode
streams, `chunk` may be any JavaScript value other than `null`.
* `encoding` {String} The encoding, if `chunk` is a String
* `callback` {Function} Optional callback for when the stream is finished
Calling the `writable.end()` method signals that no more data will be written
to the [Writable][]. The optional `chunk` and `encoding` arguments allow one
final additional chunk of data to be written immediately before closing the
stream. If provided, the optional `callback` function is attached as a listener
for the [`finish`][] event.
Calling the [`stream.write()`][stream-write] method after calling
[`stream.end()`][stream-end] will raise an error.
`js
// write hello, and then end with world!
const file = fs.createWriteStream(example.txt);
file.write(hello, );
file.end(world!);
// writing more now is not allowed!
`
##### writable.setDefaultEncoding(encoding)
* `encoding` {String} The new default encoding
* Return: `this`
The `writable.setDefaultEncoding()` method sets the default `encoding` for a
[Writable][] stream.
##### writable.uncork()
The `writable.uncork()` method flushes all data buffered since
[`stream.cork()`][] was called.
When using `writable.cork()` and `writable.uncork()` to manage the buffering
of writes to a stream, it is recommended that calls to `writable.uncork()` be
deferred using `process.nextTick()`. Doing so allows batching of all
`writable.write()` calls that occur within a given Node.js event loop phase.
`js
stream.cork();
stream.write(some );
stream.write(data );
process.nextTick(() => stream.uncork());
`
If the `writable.cork()` method is called multiple times on a stream, the same
number of calls to `writable.uncork()` must be called to flush the buffered
data.
`
stream.cork();
stream.write(some );
stream.cork();
stream.write(data );
process.nextTick(() => {
stream.uncork();
// The data will not be flushed until uncork() is called a second time.
stream.uncork();
});
`
##### writable.write(chunk[, encoding][, callback])
* `chunk` {String|Buffer} The data to write
* `encoding` {String} The encoding, if `chunk` is a String
* `callback` {Function} Callback for when this chunk of data is flushed
* Returns: {Boolean} `false` if the stream wishes for the calling code to
wait for the `drain` event to be emitted before continuing to write
additional data; otherwise `true`.
The `writable.write()` method writes some data to the stream, and calls the
supplied `callback` once the data has been fully handled. If an error
occurs, the `callback` *may or may not* be called with the error as its
first argument. To reliably detect write errors, add a listener for the
`error` event.
The return value indicates whether the written `chunk` was buffered internally
and the buffer has exceeded the `highWaterMark` configured when the stream was
created. If `false` is returned, further attempts to write data to the stream
should be paused until the `drain` event is emitted.
A Writable stream in object mode will always ignore the `encoding` argument.
### Readable Streams
Readable streams are an abstraction for a *source* from which data is
consumed.
Examples of Readable streams include:
* [HTTP responses, on the client][http-incoming-message]
* [HTTP requests, on the server][http-incoming-message]
* [fs read streams][]
* [zlib streams][zlib]
* [crypto streams][crypto]
* [TCP sockets][]
* [child process stdout and stderr][]
* [`process.stdin`][]
All [Readable][] streams implement the interface defined by the
`stream.Readable` class.
#### Two Modes
Readable streams effectively operate in one of two modes: flowing and paused.
When in flowing mode, data is read from the underlying system automatically
and provided to an application as quickly as possible using events via the
[`EventEmitter`][] interface.
In paused mode, the [`stream.read()`][stream-read] method must be called
explicitly to read chunks of data from the stream.
All [Readable][] streams begin in paused mode but can be switched to flowing
mode in one of the following ways:
* Adding a [`data`][] event handler.
* Calling the [`stream.resume()`][stream-resume] method.
* Calling the [`stream.pipe()`][] method to send the data to a [Writable][].
The Readable can switch back to paused mode using one of the following:
* If there are no pipe destinations, by calling the
[`stream.pause()`][stream-pause] method.
* If there are pipe destinations, by removing any [`data`][] event
handlers, and removing all pipe destinations by calling the
[`stream.unpipe()`][] method.
The important concept to remember is that a Readable will not generate data
until a mechanism for either consuming or ignoring that data is provided. If
the consuming mechanism is disabled or taken away, the Readable will *attempt*
to stop generating the data.
*Note*: For backwards compatibility reasons, removing [`data`][] event
handlers will **not** automatically pause the stream. Also, if there are piped
destinations, then calling [`stream.pause()`][stream-pause] will not guarantee
that the stream will *remain* paused once those destinations drain and ask for
more data.
*Note*: If a [Readable][] is switched into flowing mode and there are no
consumers available handle the data, that data will be lost. This can occur,
for instance, when the `readable.resume()` method is called without a listener
attached to the `data` event, or when a `data` event handler is removed
from the stream.
#### Three States
The two modes of operation for a Readable stream are a simplified abstraction
for the more complicated internal state management that is happening within the
Readable stream implementation.
Specifically, at any given point in time, every Readable is in one of three
possible states:
* `readable._readableState.flowing = null`
* `readable._readableState.flowing = false`
* `readable._readableState.flowing = true`
When `readable._readableState.flowing` is `null`, no mechanism for consuming the
streams data is provided so the stream will not generate its data.
Attaching a listener for the `data` event, calling the `readable.pipe()`
method, or calling the `readable.resume()` method will switch
`readable._readableState.flowing` to `true`, causing the Readable to begin
actively emitting events as data is generated.
Calling `readable.pause()`, `readable.unpipe()`, or receiving back pressure
will cause the `readable._readableState.flowing` to be set as `false`,
temporarily halting the flowing of events but *not* halting the generation of
data.
While `readable._readableState.flowing` is `false`, data may be accumulating
within the streams internal buffer.
#### Choose One
The Readable stream API evolved across multiple Node.js versions and provides
multiple methods of consuming stream data. In general, developers should choose
*one* of the methods of consuming data and *should never* use multiple methods
to consume data from a single stream.
Use of the `readable.pipe()` method is recommended for most users as it has been
implemented to provide the easiest way of consuming stream data. Developers that
require more fine-grained control over the transfer and generation of data can
use the [`EventEmitter`][] and `readable.pause()`/`readable.resume()` APIs.
#### Class: stream.Readable
##### Event: close
The `close` event is emitted when the stream and any of its underlying
resources (a file descriptor, for example) have been closed. The event indicates
that no more events will be emitted, and no further computation will occur.
Not all [Readable][] streams will emit the `close` event.
##### Event: data
* `chunk` {Buffer|String|any} The chunk of data. For streams that are not
operating in object mode, the chunk will be either a string or `Buffer`.
For streams that are in object mode, the chunk can be any JavaScript value
other than `null`.
The `data` event is emitted whenever the stream is relinquishing ownership of
a chunk of data to a consumer. This may occur whenever the stream is switched
in flowing mode by calling `readable.pipe()`, `readable.resume()`, or by
attaching a listener callback to the `data` event. The `data` event will
also be emitted whenever the `readable.read()` method is called and a chunk of
data is available to be returned.
Attaching a `data` event listener to a stream that has not been explicitly
paused will switch the stream into flowing mode. Data will then be passed as
soon as it is available.
The listener callback will be passed the chunk of data as a string if a default
encoding has been specified for the stream using the
`readable.setEncoding()` method; otherwise the data will be passed as a
`Buffer`.
`js
const readable = getReadableStreamSomehow();
readable.on(data, (chunk) => {
console.log(`Received ${chunk.length} bytes of data.`);
});
`
##### Event: end
The `end` event is emitted when there is no more data to be consumed from
the stream.
*Note*: The `end` event **will not be emitted** unless the data is
completely consumed. This can be accomplished by switching the stream into
flowing mode, or by calling [`stream.read()`][stream-read] repeatedly until
all data has been consumed.
`js
const readable = getReadableStreamSomehow();
readable.on(data, (chunk) => {
console.log(`Received ${chunk.length} bytes of data.`);
});
readable.on(end, () => {
console.log(There will be no more data.);
});
`
##### Event: error
* {Error}
The `error` event may be emitted by a Readable implementation at any time.
Typically, this may occur if the underlying stream in unable to generate data
due to an underlying internal failure, or when a stream implementation attempts
to push an invalid chunk of data.
The listener callback will be passed a single `Error` object.
##### Event: readable
The `readable` event is emitted when there is data available to be read from
the stream. In some cases, attaching a listener for the `readable` event will
cause some amount of data to be read into an internal buffer.
`javascript
const readable = getReadableStreamSomehow();
readable.on(readable, () => {
// there is some data to read now
});
`
The `readable` event will also be emitted once the end of the stream data
has been reached but before the `end` event is emitted.
Effectively, the `readable` event indicates that the stream has new
information: either new data is available or the end of the stream has been
reached. In the former case, [`stream.read()`][stream-read] will return the
available data. In the latter case, [`stream.read()`][stream-read] will return
`null`. For instance, in the following example, `foo.txt` is an empty file:
`js
const fs = require(fs);
const rr = fs.createReadStream(foo.txt);
rr.on(readable, () => {
console.log(readable:, rr.read());
});
rr.on(end, () => {
console.log(end);
});
`
The output of running this script is:
`
$ node test.js
readable: null
end
`
*Note*: In general, the `readable.pipe()` and `data` event mechanisms are
preferred over the use of the `readable` event.
##### readable.isPaused()
* Return: {Boolean}
The `readable.isPaused()` method returns the current operating state of the
Readable. This is used primarily by the mechanism that underlies the
`readable.pipe()` method. In most typical cases, there will be no reason to
use this method directly.
`js
const readable = new stream.Readable
readable.isPaused() // === false
readable.pause()
readable.isPaused() // === true
readable.resume()
readable.isPaused() // === false
`
##### readable.pause()
* Return: `this`
The `readable.pause()` method will cause a stream in flowing mode to stop
emitting [`data`][] events, switching out of flowing mode. Any data that
becomes available will remain in the internal buffer.
`js
const readable = getReadableStreamSomehow();
readable.on(data, (chunk) => {
console.log(`Received ${chunk.length} bytes of data.`);
readable.pause();
console.log(There will be no additional data for 1 second.);
setTimeout(() => {
console.log(Now data will start flowing again.);
readable.resume();
}, 1000);
});
`
##### readable.pipe(destination[, options])
* `destination` {stream.Writable} The destination for writing data
* `options` {Object} Pipe options
* `end` {Boolean} End the writer when the reader ends. Defaults to `true`.
The `readable.pipe()` method attaches a [Writable][] stream to the `readable`,
causing it to switch automatically into flowing mode and push all of its data
to the attached [Writable][]. The flow of data will be automatically managed so
that the destination Writable stream is not overwhelmed by a faster Readable
stream.
The following example pipes all of the data from the `readable` into a file
named `file.txt`:
`js
const readable = getReadableStreamSomehow();
const writable = fs.createWriteStream(file.txt);
// All the data from readable goes into file.txt
readable.pipe(writable);
`
It is possible to attach multiple Writable streams to a single Readable stream.
The `readable.pipe()` method returns a reference to the *destination* stream
making it possible to set up chains of piped streams:
`js
const r = fs.createReadStream(file.txt);
const z = zlib.createGzip();
const w = fs.createWriteStream(file.txt.gz);
r.pipe(z).pipe(w);
`
By default, [`stream.end()`][stream-end] is called on the destination Writable
stream when the source Readable stream emits [`end`][], so that the
destination is no longer writable. To disable this default behavior, the `end`
option can be passed as `false`, causing the destination stream to remain open,
as illustrated in the following example:
`js
reader.pipe(writer, { end: false });
reader.on(end, () => {
writer.end(Goodbye
);
});
`
One important caveat is that if the Readable stream emits an error during
processing, the Writable destination *is not closed* automatically. If an
error occurs, it will be necessary to *manually* close each stream in order
to prevent memory leaks.
*Note*: The [`process.stderr`][] and [`process.stdout`][] Writable streams are
never closed until the Node.js process exits, regardless of the specified
options.
##### readable.read([size])
* `size` {Number} Optional argument to specify how much data to read.
* Return {String|Buffer|Null}
The `readable.read()` method pulls some data out of the internal buffer and
returns it. If no data available to be read, `null` is returned. By default,
the data will be returned as a `Buffer` object unless an encoding has been
specified using the `readable.setEncoding()` method or the stream is operating
in object mode.
The optional `size` argument specifies a specific number of bytes to read. If
`size` bytes are not available to be read, `null` will be returned *unless*
the stream has ended, in which case all of the data remaining in the internal
buffer will be returned (*even if it exceeds `size` bytes*).
If the `size` argument is not specified, all of the data contained in the
internal buffer will be returned.
The `readable.read()` method should only be called on Readable streams operating
in paused mode. In flowing mode, `readable.read()` is called automatically until
the internal buffer is fully drained.
`js
const readable = getReadableStreamSomehow();
readable.on(readable, () => {
var chunk;
while (null !== (chunk = readable.read())) {
console.log(`Received ${chunk.length} bytes of data.`);
}
});
`
In general, it is recommended that developers avoid the use of the `readable`
event and the `readable.read()` method in favor of using either
`readable.pipe()` or the `data` event.
A Readable stream in object mode will always return a single item from
a call to [`readable.read(size)`][stream-read], regardless of the value of the
`size` argument.
*Note:* If the `readable.read()` method returns a chunk of data, a `data`
event will also be emitted.
*Note*: Calling [`stream.read([size])`][stream-read] after the [`end`][]
event has been emitted will return `null`. No runtime error will be raised.
##### readable.resume()
* Return: `this`
The `readable.resume()` method causes an explicitly paused Readable stream to
resume emitting [`data`][] events, switching the stream into flowing mode.
The `readable.resume()` method can be used to fully consume the data from a
stream without actually processing any of that data as illustrated in the
following example:
`js
getReadableStreamSomehow()
.resume()
.on(end, () => {
console.log(Reached the end, but did not read anything.);
});
`
##### readable.setEncoding(encoding)
* `encoding` {String} The encoding to use.
* Return: `this`
The `readable.setEncoding()` method sets the default character encoding for
data read from the Readable stream.
Setting an encoding causes the stream data
to be returned as string of the specified encoding rather than as `Buffer`
objects. For instance, calling `readable.setEncoding(utf8)` will cause the
output data will be interpreted as UTF-8 data, and passed as strings. Calling
`readable.setEncoding(hex)` will cause the data to be encoded in hexadecimal
string format.
The Readable stream will properly handle multi-byte characters delivered through
the stream that would otherwise become improperly decoded if simply pulled from
the stream as `Buffer` objects.
Encoding can be disabled by calling `readable.setEncoding(null)`. This approach
is useful when working with binary data or with large multi-byte strings spread
out over multiple chunks.
`js
const readable = getReadableStreamSomehow();
readable.setEncoding(utf8);
readable.on(data, (chunk) => {
assert.equal(typeof chunk, string);
console.log(got %d characters of string data, chunk.length);
});
`
##### readable.unpipe([destination])
Reviews
There are no reviews yet.