AI & Agent Dev Bug Sandbox Back to Radar

Unit Tests Fail On Windows With Non-UTF-8 Locale Due To Missing Explicit UTF-8 Encoding When Reading Pyproject.Toml

Two unit tests in the langchain package (tests/unit_tests/test_version.py and tests/unit_tests/test_dependencies.py) open pyproject.toml using Path.open() without specifying encoding. On Windows systems with a non-UTF-8 locale (e.g., cp936), Python defaults to the locale encoding, causing UnicodeDecodeError when encountering UTF-8 bytes. The fix is to pass encoding="utf-8" to the open calls.

mediumConfidence 95%Python Affected V1.4.1

Origin Analysis

The tests rely on Python's default file encoding, which is locale-dependent. On Windows, the default is typically the active code page (e.g., cp936 for Simplified Chinese), not UTF-8. Since pyproject.toml is always UTF-8 encoded per specification, opening it without an explicit encoding leads to decoding errors on such systems.

On a Windows machine with system locale set to cp936 (Simplified Chinese), clone the langchain repository, navigate to libs/langchain_v1, run `uv sync --group test` and then `uv run --group test pytest tests/unit_tests/test_version.py tests/unit_tests/test_dependencies.py -q`. The tests fail with UnicodeDecodeError: 'gbk' codec can't decode byte 0x93.

Fixing Code Block

# tests/unit_tests/test_version.py
# Replace the existing open call with an explicit encoding:
# Before:
# with Path("pyproject.toml").open() as f:
#     pyproject = tomllib.load(f)
# After:
with Path("pyproject.toml").open(encoding="utf-8") as f:
    pyproject = tomllib.load(f)

# tests/unit_tests/test_dependencies.py
# Replace the existing open call with an explicit encoding:
# Before:
# with Path("pyproject.toml").open() as f:
#     pyproject = tomllib.load(f)
# After:
with Path("pyproject.toml").open(encoding="utf-8") as f:
    pyproject = tomllib.load(f)

Adding encoding="utf-8" to the Path.open() call forces Python to read the file as UTF-8 regardless of the system locale, matching the encoding standard for pyproject.toml and the convention already used in scripts/check_version.py.

Edge Case Audit

This fix is limited to test code and does not affect runtime. Rollback is trivial by removing the encoding argument. No concurrency or cross-platform issues are introduced. The only theoretical risk is if the pyproject.toml ever contains non-UTF-8 content, but this is not allowed by the standard. The change is safe and recommended.