Validation Order In NLTKTextSplitter Raises Misleading Error When NLTK Is Missing
NLTKTextSplitter.__init__ validates argument combinations (e.g., use_span_tokenize with non-empty separator) before checking if NLTK is installed, causing a misleading ValueError instead of the expected ImportError when NLTK is missing.
The __init__ method of NLTKTextSplitter performs argument validation before checking for the availability of the NLTK dependency. This order results in a ValueError about separator usage being raised first, masking the actual missing dependency issue.
Ensure NLTK is not installed (pip uninstall nltk). Then run:
from langchain_text_splitters import NLTKTextSplitter
splitter = NLTKTextSplitter(use_span_tokenize=True)
Observe that a ValueError about separator is raised before the ImportError.
Fixing Code Block
def __init__(
self,
separator: str = "\n\n",
language: str = "english",
use_span_tokenize: bool = False,
**kwargs: Any,
) -> None:
"""Initialize the NLTK splitter."""
if not __HAS_NLTK__:
raise ImportError(
"NLTK is not installed, please install it with `pip install nltk`."
)
super().__init__(**kwargs)
if use_span_tokenize and separator != "":
raise ValueError(
"When use_span_tokenize is True, separator should be empty string"
)
self._separator = separator
self._language = language
self._use_span_tokenize = use_span_tokenize
try:
from nltk.tokenize import sent_tokenize, word_tokenize
except ImportError:
raise ImportError(
"NLTK is not installed, please install it with `pip install nltk`."
)
self._tokenizer = sent_tokenize if not use_span_tokenize else word_tokenize
Reorder the checks in __init__ so the dependency availability check (__HAS_NLTK__) executes before argument validation. This ensures users immediately see the missing dependency error when NLTK is absent, while preserving all existing validation logic and tokenizer setup.
Edge Case Audit
Low risk: the change only alters the order of error-raising conditions. Potential secondary effects: if NLTK is installed but cannot be imported due to a broken installation, __HAS_NLTK__ may be True but the subsequent import will still raise ImportError (handled by the try/except fallback). No concurrency or threading concerns. Rollback: restore the original order of validation before dependency check. Ensure tests cover both missing NLTK and invalid argument combinations.
