Processing Multi-Gigabyte JSON Payloads: Streaming, Memory Leaks, and SAX/Pipeline Parsers in Node.js & Go

1. The V8 Memory Wall: Why JSON.parse() Fails

Every JavaScript developer eventually encounters the dreaded error: `text FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Why does parsing a 400 MB JSON file cause a 2 GB Node.js process to run out of memory?

1. String Duplication in Heap: The raw file string occupies 400 MB of heap memory.
2. Object Overhead Factor (4x - 8x): When V8 parses a string into an in-memory object, each key-value pair creates hidden classes (Shapes), pointer references, and property arrays. A 400 MB JSON string frequently balloons to 1.8 GB - 3.2 GB of heap objects.
3. Garbage Collection Pressure: Allocating millions of tiny short-lived objects simultaneously freezes V8's concurrent garbage collector (Mark-Sweep), leading to severe event loop latency spikes.

2. DOM vs. SAX Parsing Paradigm

• DOM (Document Object Model) Parsers: (JSON.parse(), json.loads()) Load the complete tree into memory. Ideal for small configuration files (< 10 MB), disastrous for data dumps.
• SAX / Token Stream Parsers: (stream-json, json.Decoder, ijson) Read chunks of bytes sequentially, emitting tokens (start-object, key, string, end-array) as they fly through the buffer. Memory consumption remains constant $O(1)$ regardless of whether the file is 10 MB or 50 GB.

3. Streaming JSON in Node.js with stream-json

By chaining transform streams, we can ingest massive array arrays without exceeding 40 MB of RAM:

import fs from "node:fs";
import { pipeline } from "node:stream/promises";
import { parser } from "stream-json";
import { streamArray } from "stream-json/streamers/StreamArray.js";
import { Batch } from "stream-json/utils/Batch.js";

async function processBigDataset(filePath) { let recordsProcessed = 0;

const dbBatchWriter = new Transform({ objectMode: true, async transform(batch, encoding, callback) { // batch is an array of { key: index, value: dataItem } await writeToDatabase(batch.map((b) => b.value)); recordsProcessed += batch.length; callback(); }, });

await pipeline( fs.createReadStream(filePath), parser(), streamArray(), new Batch({ batchSize: 500 }), dbBatchWriter );

console.log(Successfully processed ${recordsProcessed} records with O(1) memory.); }

4. Zero-Allocation Streaming in Go (json.Decoder)

Go's standard library provides a world-class streaming tokenizer with no external dependencies required:

import ( "encoding/json" "fmt" "io" "os" )

type UserRecord struct { ID string json:"id" Email string json:"email" Balance int64 json:"balance" }

func StreamLargeJSONFile(filePath string) error { file, err := os.Open(filePath) if err != nil { return err } defer file.Close()

decoder := json.NewDecoder(file)

// Read opening bracket '[' t, err := decoder.Token() if err != nil || t != json.Delim('[') { return fmt.Errorf("expected array start") }

for decoder.More() { var record UserRecord if err := decoder.Decode(&record); err != nil { return err } // Process record immediately without keeping reference processRecord(record) }

// Read closing bracket ']' _, err = decoder.Token() return err }

5. Iterative Parsing in Python with ijson

In Python, loading multi-gigabyte JSON into json.load() will trigger the OS OOM killer. The standard production solution is ijson:

def process_large_transactions(filename): with open(filename, 'rb') as f: # Stream elements under the top-level 'items' array records = ijson.items(f, 'items.item') for record in records: # Process single dictionary validate_and_persist(record)

6. Memory & Throughput Benchmarks

Tested on a 2.8 GB JSON file containing 10,000,000 telemetry objects:

7. Production Guidelines for 1GB+ Ingestion

1. Use NDJSON (Newline Delimited JSON) When Possible: If you control the data export, write each object on its own line. NDJSON can be split with standard line readers and processed concurrently across CPU workers.
2. Always Configure Stream Backpressure: Ensure your readable stream pauses while your database or downstream sink finishes processing batches.
3. Avoid String Buffering in Memory: Never collect chunks into a giant string before calling parsers.