Tricks on writing & appending to a file in Node.js
Tags: node, filesystem, tutorial, fs
Published: 2020-01-29 08:37:04

---
title: Tricks on writing & appending to a file in Node.js
published: true
description: Some tricks while you're working with a filesystem in Node.js
tags: node, filesystem, tutorial, fs
cover_image: https://sergchr.netlify.com/images/Group%203.svg
---
 
This article covers the use of [`fs.appendFile`](https://nodejs.org/docs/latest-v11.x/api/fs.html#fs_fs_appendfile_path_data_options_callback) and [`fs.writeFile`](https://nodejs.org/docs/latest-v11.x/api/fs.html#fs_fs_writefile_file_data_options_callback) functions, how they work in details. Specifically, we'll investigate them in a practical case.

## Writing logs
Let's discover a use case where we want to write logs to a file. It seems like there is an obvious way to do this - call `fs.writeFile` each time we need it.
```javascript
fs.writeFile('log.txt', 'message', 'utf8', callback);
```
The problem is `writeFile` replaces the file data each time we use the function, so we can't just write to a file. We could use a different approach: read a file data via `fs.readFile`, then append to the existing logs a necessary data and new line.
```javascript
// we'll use callbacks in the article, but remember you always
//  can promisify those functions
// *we will not handle the errors in callbacks
const newLogs = `${Date.now()}: new logs`;
fs.readFile('log.txt', { encoding: 'utf8' }, (err, data) => {
  const newData = data + newLogs + '\n';
  fs.writeFile('log.txt', newData, 'utf8', callback);
});
```
But this method also has cons. Each time we want to write new logs, the program opens a file, load all file data to memory, then opens the same file again and writes new data. Imagine how much resources a script will need in case of a large file.

Node has another method to do this simpler - `fs.appendFile`.
```javascript
fs.appendFile('log.txt', 'new logs', 'utf8', callback);
```
This is much better, but what does this method do? Let's discover how `appendFile` is implemented.
[`lib/internal/fs/promises.js`](https://github.com/nodejs/node/blob/master/lib/internal/fs/promises.js#L504):
```javascript
async function appendFile(path, data, options) {
  // manipulations with the "options" argument, you can skip it
  // up to the return statement
  options = getOptions(options, { encoding: 'utf8', mode: 0o666, flag: 'a' });
  options = copyObject(options);
  options.flag = options.flag || 'a';
  return writeFile(path, data, options); // so, writeFile anyway?
}

// ...
async function writeFile(path, data, options) {
  options = getOptions(options, { encoding: 'utf8', mode: 0o666, flag: 'w' });
  const flag = options.flag || 'w';

  // in our case, the "path" isn't a FileHandle, it's a string
  if (path instanceof FileHandle)
    return writeFileHandle(path, data, options);

  // "fd" is a file descriptor (FileHandle instance)
  const fd = await open(path, flag, options.mode);
  return writeFileHandle(fd, data, options).finally(fd.close);
}
```
We discover what is the `FileHandle` a bit further.

Still, the `appendFile` does pretty much the same as we did earlier. In details, it:
- [opens](https://github.com/nodejs/node/blob/master/lib/internal/fs/promises.js#L217) a file, [gets a file handle](https://github.com/nodejs/node/blob/master/lib/internal/fs/promises.js#L222)
- [writes](https://github.com/nodejs/node/blob/master/lib/internal/fs/promises.js#L142) data to a file (calls ['write'](https://github.com/nodejs/node/blob/master/lib/internal/fs/promises.js#L257) which decide whether to write buffer or string(`C++` bindings)).

Is it okay to write logs like that? Not really. It's okay for occasional writes, here's why. 


`appendFile` opens a file each time we need to write logs. In some cases, it can cause `EMFILE` error which means an operating system denies us to open more files/sockets.  For example, if we need to write a new log entry every `5ms`, a Node script will open a file every `5ms`. Also, you need to wait for the callback to make `appendFile` again, otherwise, the function will append a file data in a conflicting way. Example:
```javascript
// Notice: `appendFile` is called asynchronously
fs.appendFile('log.txt', '1', 'utf8', callback);
fs.appendFile('log.txt', '2', 'utf8', callback);
fs.appendFile('log.txt', '3', 'utf8', callback);

// log.txt can be as follows:
1
3
2
```

## [File descriptors](https://en.wikipedia.org/wiki/File_descriptor)
In short, file descriptor or file handle it's a reference to an opened file. They are non-negative integers. For example, standard input uses `0` value as a file handle, standard output uses `1`, standard error output occupies `2` value. So, if you open a file programmatically, you'll get a file handle valued as `3` or more.
Node has its own wrapper for file handlers - [`FileHandle`](https://github.com/nodejs/node/blob/master/lib/internal/fs/promises.js#L65) to perform basic operations on them (such as read, write, close, etc.).

The less opened file handles we have, the better. It means, `fs.appendFile` not a suitable solution to write logs.

## Maybe streams?
Let's append to a file using writable streams:
```javascript
// 'a' flag stands for 'append'
const log = fs.createWriteStream('log.txt', { flags: 'a' });

// on new log entry ->
log.write('new entry\n');

// you can skip closing the stream if you want it to be opened while
// a program runs, then file handle will be closed
log.end();
```
What did we do here? We create a writable stream that opens `log.txt` in the background and queues writes to the file when it's ready. Pros:
- we don't load the whole file into RAM;
- we don't create new file descriptors each time a program writes to the file.
The purpose of streams here is to write small chunks of data to a file instead of loading the whole file into memory.

## Summaries
- Don't use `fs.appendFile` if you need to write to a file often.
- Use `fs.appendFile` for occasional writes.
- Don't use `fs.writeFile` (or `fs.write`) to write a large amount of data or when dealing with large files. Use writable streams instead.

[Source](https://sergchr.netlify.com/posts/tricks-on-writing-appending-to-a-file-in-node/)