AI & Agent Dev Bug Sandbox Back to Radar

SpacyTextSplitter Ignores Max_length With Pipeline='Sentencizer'

The max_length parameter is not applied when using the lightweight 'sentencizer' pipeline, causing spaCy's default limit of 1,000,000 characters to be used instead of the user-specified limit.

mediumConfidence 95%Langchain-Text-Splitters Affected V1.1.2

Origin Analysis

In `_make_spacy_pipeline_for_splitting`, the line `sentencizer.max_length = max_length` is incorrectly indented inside the `else` branch, which only runs when the pipeline is not 'sentencizer'. For the 'sentencizer' branch, the limit is left at spaCy's default.

1. Mock spaCy's import and English class to avoid installing spaCy. 2. Instantiate SpacyTextSplitter with pipeline='sentencizer' and max_length=2_000_000. 3. Access `splitter._tokenizer.max_length` and observe it is 1_000_000 instead of 2_000_000, causing an assertion failure.

Fixing Code Block

def _make_spacy_pipeline_for_splitting(
    pipeline: str, max_length: int = 1_000_000
) -> Any:
    try:
        import spacy
    except ImportError:
        raise ImportError(
            "Spacy is not installed, please install it with `pip install spacy`."
        )
    if pipeline == "sentencizer":
        sentencizer = cast("Language", spacy.blank("en"))
    else:
        sentencizer = cast("Language", spacy.load(pipeline, exclude=["ner", "tagger"]))
    sentencizer.max_length = max_length
    return sentencizer

The assignment `sentencizer.max_length = max_length` is moved out of the conditional block, ensuring that the maximum length is set on the pipeline object regardless of whether the 'sentencizer' branch or the model-loading branch is taken. This applies the user-specified limit in both scenarios.

Edge Case Audit

The change is minimal and should not break existing behavior because setting `max_length` after pipeline construction is safe and intended. However, ensure that `max_length` is not `None`; if it can be `None`, add a guard to skip assignment. Rollback can be done by restoring the original indentation. Test both 'sentencizer' and model-loaded paths to confirm the limit is respected.