How to Handle Huge JSON Files Without Crashing Your Application

Consider a small JSON file:

1{
2  "users": [
3    {
4      "id": 1,
5      "name": "Alice"
6    },
7    {
8      "id": 2,
9      "name": "Bob"
10    }
11  ]
12}

Reading this file into memory is not a problem. But imagine the same structure contains millions of objects:

1{
2  "users": [
3    ...
4    millions of records
5    ...
6  ]
7}

Now the application has to:

1. Read the file from storage.
2. Load the contents into memory.
3. Parse the JSON text.
4. Create objects representing the data.
5. Keep those objects in memory while processing them.

The actual memory consumption can be significantly larger than the original file size. That is where large JSON processing becomes challenging.

In JavaScript, a common way to parse JSON is:

1const data = JSON.parse(jsonString);

This works very well for normal-sized JSON documents. The problem is that JSON.parse() expects the JSON text to be available as a complete string before parsing. For a huge file, the workflow can look like this:

1Large JSON file
2      ↓
3Read entire file
4      ↓
5Large string in memory
6      ↓
7JSON.parse()
8      ↓
9Large JavaScript object
10      ↓
11More memory usage

A 500 MB JSON file does not necessarily mean the application only needs 500 MB of memory. It may need memory for:

• The original file buffer
• The JSON string
• Parsed JavaScript objects
• Temporary parsing structures
• Application logic
• Other running processes

The actual memory requirement depends on the runtime, data structure, encoding, and implementation.

One of the biggest mistakes when handling large JSON is assuming:

The file is 1 GB, so I only need around 1 GB of RAM.

In reality, processing can require considerably more memory. Consider:

1Disk
2  ↓
3JSON file
4  ↓
5Buffer
6  ↓
7String
8  ↓
9Parsed objects
10  ↓
11Application processing

The key technique for handling very large datasets is streaming. Instead of loading everything at once, you process data incrementally. This can dramatically reduce peak memory usage.

1File
2 ↓
3Everything into memory
4 ↓
5Parse everything

1File
2 ↓
3Small chunk
4 ↓
5Process
6 ↓
7Next chunk
8 ↓
9Process
10 ↓
11Next chunk

What Is JSON Streaming?

JSON streaming means processing JSON data incrementally rather than requiring the entire document to be loaded into memory at once. The application can process records as they arrive.

1Large data source
2       ↓
3   Small chunks
4       ↓
5 JSON parser
6       ↓
7 Individual records
8       ↓
9 Application logic

This is particularly useful for:

• Large datasets
• Logs
• Data pipelines
• ETL systems
• Imports
• Exports
• Network streams

Consider a huge JSON array:

1[
2  {"id":1},
3  {"id":2},
4  {"id":3},
5  {"id":4}
6]

For a very large dataset, parsing the entire array can be expensive. An alternative is JSON Lines, commonly called JSONL or NDJSON. Instead of one huge JSON array, JSONL stores one JSON object per line:

1{"id":1}
2{"id":2}
3{"id":3}
4{"id":4}

Now each line can be processed independently.

JSON vs JSONL for Large Data

JSON is still the right format for many applications. But when you have huge collections of independent records, JSONL can be much easier to process.

Example: Processing a JSONL File

1{"id":1,"name":"Alice"}
2{"id":2,"name":"Bob"}
3{"id":3,"name":"Charlie"}

Instead of loading the complete file, the application can process one line at a time. Conceptually:

1for await (const line of stream) {
2  const record = JSON.parse(line);
3  processRecord(record);
4}

The important idea is that the application does not need to keep every record in memory simultaneously.

Node.js provides streaming APIs that can be useful for large files. A normal approach might be:

1import fs from "fs";
2
3const data = JSON.parse(
4  fs.readFileSync("large.json", "utf8")
5);

This is simple but potentially expensive for a huge file. A streaming approach uses a readable stream:

1import fs from "fs";
2
3const stream = fs.createReadStream("large.json");
4
5stream.on("data", (chunk) => {
6  // Process incoming data
7});

Why You Cannot Simply Parse Every Chunk

Imagine this JSON, and a stream that delivers it in two pieces:

1{"name":"Alice","age":30}
2
3Chunk 1:
4{"name":"Ali
5
6Chunk 2:
7ce","age":30}

Calling the following on the first chunk would fail:

1JSON.parse(chunk) // fails on the first chunk

The parser needs to understand that the JSON document is incomplete and wait for more data. This is why proper streaming parsers maintain parsing state across chunks.

For large JSON documents, developers often use libraries that can parse JSON incrementally. These libraries can expose events or callbacks when complete objects or values are encountered. Conceptually:

1Chunk 1 ─┐
2Chunk 2 ─┤
3Chunk 3 ─┼──> Streaming parser
4Chunk 4 ─┤
5Chunk 5 ─┘
6               ↓
7          Complete object
8               ↓
9          Process object

The exact library depends on the programming language and application.

One of the most effective optimizations is to ask:

Do I really need the entire dataset in memory?

Often, the answer is no. Suppose you need to calculate the total value of millions of transactions. You do not necessarily need:

1const transactions = [...millions of records...];

Instead, you can process each record:

1let total = 0;
2
3for await (const transaction of transactionsStream) {
4  total += transaction.amount;
5}

The application only needs to retain the information required for the current operation.

Imagine a large dataset containing millions of users, and you only need users from a particular country.

1Load everything
2      ↓
3Parse everything
4      ↓
5Filter

1Read record
2      ↓
3Check condition
4      ↓
5Keep only matching record
6      ↓
7Discard everything else

This reduces memory usage.

Avoid Unnecessary Copies

Large JSON processing can become expensive when data is copied repeatedly. For example:

1const filtered = hugeArray.filter(...);
2const mapped = filtered.map(...);
3const sorted = mapped.sort(...);

Depending on the operation, multiple large intermediate arrays may exist. For large datasets, consider processing records incrementally or using a streaming pipeline when appropriate.

Huge JSON responses are not always caused by files. APIs can also return enormous JSON responses. Instead of this returning millions of users:

1GET /users

use pagination:

1GET /users?page=1&limit=100
2
3Request 1 → 100 records
4Request 2 → 100 records
5Request 3 → 100 records

This reduces:

• Response size
• Memory usage
• Network time
• Parsing cost
• Server load

Pagination is often one of the simplest solutions to large JSON responses.

Cursor-Based Pagination

For very large or frequently changing datasets, cursor-based pagination can be more reliable than traditional page numbers.

1GET /users?limit=100&cursor=abc123

The server returns:

1{
2  "data": [...],
3  "nextCursor": "def456"
4}

The client then requests:

1GET /users?limit=100&cursor=def456

This approach can work well for large datasets because the client only processes a manageable portion at a time.

If large JSON is being transferred over a network, compression can significantly reduce the amount of data transmitted. Common compression formats include:

• gzip
• Brotli
• zstd

1Uncompressed JSON
2       ↓
3   Compression
4       ↓
5Smaller response
6       ↓
7Network
8       ↓
9Decompression
10       ↓
11Application

Compression reduces network transfer size, but it does not eliminate the memory cost of parsing a huge JSON document after decompression.

You still need an appropriate processing strategy.

A JSON file's size on disk is only one measurement. A "100 MB JSON" does not tell you exactly how much memory the application will require. The processing cost depends on:

• Number of records
• Nesting depth
• String sizes
• Object structure
• Runtime implementation
• Parser implementation
• Temporary allocations
• Application logic

Therefore, large JSON workloads should be tested under realistic conditions.

Processing huge JSON in a browser can be even more challenging. Browsers have limited resources and must also keep the user interface responsive. Consider this:

1const response = await fetch("/large.json");
2const data = await response.json();

This is convenient for normal API responses. But if /large.json is hundreds of megabytes, the browser may experience:

• High memory usage
• Long parsing time
• UI freezing
• Slow garbage collection
• Browser tab crashes

For large browser workloads, consider pagination, server-side processing, streaming approaches, or moving heavy work into a Web Worker where appropriate.

Web Workers for Heavy JSON Processing

If a large JSON dataset must be processed in the browser, a Web Worker can move CPU-intensive work away from the main UI thread. Conceptually:

1Main Thread
2     │
3     │ data
4     ▼
5Web Worker
6     │
7     ├── Parse
8     ├── Filter
9     └── Transform
10     │
11     ▼
12Processed result

This can help keep the interface responsive. However, Web Workers do not magically remove memory limitations. The dataset still has to be represented somewhere, so streaming and incremental processing remain important.

Sometimes the best optimization is architectural. If your system regularly handles extremely large JSON documents, ask whether JSON is the correct storage or exchange format for that workload. A huge JSON file containing millions of independent records may be better represented using:

• JSONL
• A database
• Columnar storage
• Compressed data files
• Partitioned datasets
• Streaming APIs

The correct solution depends on how the data is consumed.

Start with the simplest solution and increase complexity only when necessary.

Before processing a huge JSON file, ask:

These questions can prevent many performance problems before they happen.

The difference can be summarized simply:

1Traditional parsing:
2
3Large file
4   ↓
5Load everything
6   ↓
7Parse everything
8   ↓
9Process
10
11
12Streaming:
13
14Large file
15   ↓
16Small chunk
17   ↓
18Parse
19   ↓
20Process
21   ↓
22Next chunk

Traditional parsing is easier. Streaming is more complex but can be much more appropriate for large datasets.