AI & Agent Dev Bug Sandbox logo
AI & Agent Dev Bug Sandbox
Back to Radar

RecursiveJsonSplitter Exceeds Max_chunk_size For Nested JSON Due To Missing Size Checks

The RecursiveJsonSplitter in langchain-text-splitters can produce chunks larger than max_chunk_size when splitting nested JSON, because the size measurement for nested containers does not include the full path overhead, and leaf values are inserted without any size validation. This can lead to downstream errors due to oversized chunks.

highConfidence 95%Langchain-Text-SplittersAffected V1.1.2

Origin Analysis

In the `_json_split` method, when processing a dict or list value, the code checks the size of the isolated key-value pair via `_json_size({key: value})`, which does not account for the nesting depth at which the value will be inserted. Additionally, for leaf values, the code directly calls `_set_nested_dict` without checking if adding the leaf would exceed `max_chunk_size`. The combination results in chunks that violate the configured size limit.
Run the following Python code: ```python from langchain_text_splitters import RecursiveJsonSplitter import json splitter = RecursiveJsonSplitter(max_chunk_size=50) data = {"container": {"a": {"x": 1, "y": 2, "z": 3}, "b": "test"}} chunks = splitter.split_json(data) for i, chunk in enumerate(chunks): size = len(json.dumps(chunk)) assert size <= 50, f"chunk {i} is {size}, exceeds max_chunk_size=50" ``` The assertion fails on the first chunk with size 59.

Fixing Code Block

Edge Case Audit

The `deepcopy`-based simulation introduces O(n) overhead per insertion, which may become slow for large JSON documents. This fix changes chunk boundaries and may produce a different number of chunks compared to the previous behavior, potentially impacting downstream consumers that rely on consistent chunking. If performance or compatibility issues arise, consider rolling back to the original version until an optimized implementation is available. The class is not thread-safe; concurrent use may lead to race conditions. Ensure to test thoroughly after applying, especially with deeply nested structures and edge cases like empty containers or very large leaves.

Ecosystem Topology