Framework
Reports that share this framework, grouped in one feed.
MarkdownHeaderTextSplitter corrupts chunk content by applying line.strip() and str.isprintable() filtering, which removes tabs, NBSP, and leading whitespace inside fenced code blocks, causing silent data loss during text splitting.
aggregate_lines_to_chunks() repeatedly appends to a growing string, causing quadratic time complexity when many adjacent lines share the same metadata. The issue is observable with large inputs (e.g., 16,000 paragraphs take ~5.8 seconds) and becomes progressively worse.
The max_length parameter is not applied when using the lightweight 'sentencizer' pipeline, causing spaCy's default limit of 1,000,000 characters to be used instead of the user-specified limit.
MarkdownHeaderTextSplitter incorrectly includes CommonMark optional closing hash sequences in header metadata values, e.g. '# Title ##' yields metadata 'Title ##' instead of 'Title'. This breaks filtering and grouping by header.
When a piece cannot be split further because no separators remain, it is appended directly to the output without applying strip_whitespace, causing inconsistent leading/trailing whitespace compared to merged chunks.
The constructor of HTMLHeaderTextSplitter sorts headers_to_split_on using a bare int(tag[1:]) conversion, which fails for any tag outside h1-h6 (e.g., div -> 'iv'). The _generate_documents method already contains a fallback level of 9999 for such tags, causing an internal inconsistency that crashes at initialization.
The splitter terminates fenced code blocks whenever it sees any line of three or more backticks or tildes, ignoring CommonMark rules that require a closing fence to match the opening character and have at least the opening length. This creates extra chunks and wrong `Code` metadata when the content includes an inner shorter fence or a mismatched fence.
When `ensure_ascii=False` is passed to `split_text`, `_json_size` still measures using `json.dumps` with default `ensure_ascii=True`, causing non-ASCII chunks to be far below `max_chunk_size` and producing more chunks than expected.