Standard-Tests: Chat Model Tests Never Check The Documented `Total_tokens == Input_tokens + Output_tokens` Invariant
The standard chat model integration tests in langchain-tests 1.1.9 verify that input_tokens, output_tokens, and total_tokens are integers but do not assert that total_tokens equals the sum of input_tokens and output_tokens. A model can report inconsistent usage metadata (e.g., input_tokens=10, output_tokens=105, total_tokens=15) and still pass both test_usage_metadata and test_usage_metadata_streaming.
ChatModelIntegrationTests.test_usage_metadata and test_usage_metadata_streaming only assert that each token count is an int, not the documented invariant from langchain-core UsageMetadata. The UsageMetadataCallbackHandler separately sums total_tokens via add_usage, so incorrect totals persist and can propagate through aggregated streaming results.
Run the provided pytest snippet with a ChatStale model that returns usage_metadata={'input_tokens': 10, 'output_tokens': 105, 'total_tokens': 15}. Both test_usage_metadata and test_usage_metadata_streaming pass on langchain-tests 1.1.9, despite the invariant violation.
Fixing Code Block
def _assert_usage_metadata_total(usage_metadata):
if not usage_metadata:
return
input_tokens = usage_metadata.get('input_tokens', 0)
output_tokens = usage_metadata.get('output_tokens', 0)
total_tokens = usage_metadata.get('total_tokens')
if total_tokens is None:
return
assert total_tokens == input_tokens + output_tokens, (
f"total_tokens {total_tokens} != input_tokens {input_tokens} + "
f"output_tokens {output_tokens}"
)
# In test_usage_metadata, after the integer type assertions:
self._assert_usage_metadata_total(message.usage_metadata)
# In test_usage_metadata_streaming, after aggregating and asserting integer types:
self._assert_usage_metadata_total(aggregated.usage_metadata)
Add a shared helper that checks the documented invariant and call it in both non-streaming and streaming tests. For streaming, the assertion is applied to the aggregated message only, because individual chunks may each satisfy the invariant but the independent summation of total_tokens in add_usage can produce an aggregate that violates it. The helper safely ignores missing total_tokens to accommodate providers like mistralai that may report 0 or omit the field.
Edge Case Audit
The assertion may fail for providers that legitimately report total_tokens=0 when the provider omits it (e.g., langchain-mistralai). This is a separate bug and may require provider-specific handling or a documented exception list. Rollback is straightforward: remove the added helper calls or conditionally skip the assertion for known providers. No runtime code in LangChain is modified, so there is no concurrency or upgrade risk beyond the test suite itself.
