Blob.As_string() Leaks UTF-8 BOM Into Decoded Text
Blob.as_string() uses the default 'utf-8' encoding, which does not strip a UTF-8 byte order mark (BOM). Files written by Windows tools (Notepad, PowerShell, etc.) often include a BOM, and this BOM is returned as part of the string, contaminating downstream processing such as Markdown heading parsing.
The Blob class decodes bytes or reads files with encoding='utf-8' by default. Python's utf-8 codec preserves the BOM character (U+FEFF) when present, whereas utf-8-sig would strip it. Since the encoding is not automatically switched to utf-8-sig, the BOM leaks into the returned string.
1. Create a file with a UTF-8 BOM (e.g., using encoding='utf-8-sig' in Python, or saving from Windows Notepad).
2. Load the file with Blob.from_path().
3. Call as_string() on the Blob instance.
4. Observe that the returned string begins with '\ufeff'.
Fixing Code Block
def as_string(self) -> str:
"""Return content as string, stripping a UTF-8 BOM if present."""
# Use utf-8-sig only when the default utf-8 is in effect,
# so explicit encoding arguments are respected.
encoding = "utf-8-sig" if self.encoding == "utf-8" else self.encoding
if self.data is not None:
return self.data.decode(encoding)
with open(self.path, "r", encoding=encoding) as f:
return f.read()
The fix changes the decoding to use 'utf-8-sig' when the current encoding is the default 'utf-8'. utf-8-sig behaves identically to utf-8 for files without a BOM, but automatically removes a leading BOM when present. This preserves the behavior for non-UTF-8 encodings and only affects the default path.
Edge Case Audit
This change may break code that intentionally relies on the BOM being preserved (e.g., when later re-encoding or writing the string back to a file with a BOM). If such behavior is required, users should explicitly pass encoding='utf-8' to as_string() or use Blob with an explicit encoding. The change is safe for concurrent or multi-threaded usage as no shared state is modified. Rollback can be performed by pinning to the previous version of langchain-core.
