AI & Agent Dev Bug Sandbox logo
AI & Agent Dev Bug Sandbox
Back to Radar

_To_protocol_usage Drops Input_token_details In V3 Streaming Path, Causing Cache Token Counts To Always Be 0

When using astream_events(version='v3'), the message-finish event's usage metadata omits input_token_details and output_token_details because _to_protocol_usage only forwards a subset of usage fields. This makes prompt cache read/creation counts unavailable, breaking cache tracking for providers like Bedrock Converse.

highConfidence 92%LangChainAffected V1.4.0

Origin Analysis

In langchain_core/language_models/_compat_bridge.py, the helper _to_protocol_usage builds the protocol usage dictionary by explicitly selecting only input_tokens, output_tokens, total_tokens, and cached_tokens from the model's usage_metadata. It silently discards the nested input_token_details and output_token_details dictionaries, so the final aggregated protocol event never carries cache_read or cache_creation counts.
Run the provided reproduction code that simulates a streaming chat model yielding a final ChatGenerationChunk with usage_metadata containing input_token_details. Collect events via achunks_to_events and inspect the message-finish event. The usage dict lacks input_token_details even though it was present on the chunk.

Fixing Code Block

Edge Case Audit

This change widens the protocol usage payload. Downstream consumers or strict validators that do not expect the nested input_token_details/output_token_details keys may fail or ignore them; ensure the protocol schema and any deserialization models accept these fields. The conditional checks use is not None to avoid adding empty objects, but if usage_metadata contains other unknown nested keys they are still dropped. Rollback by reverting to the previous explicit field selection if downstream breakage is observed. No thread-safety impact because the function only reads from a local dict, but concurrent stream aggregation must ensure the final chunk with details is not overwritten by later chunks lacking details.

Ecosystem Topology