AI & Agent Dev Bug Sandbox logo
AI & Agent Dev Bug Sandbox
Back to Radar

AsyncCallbackManager.On_llm_start Drops Non-Inline Handlers When An Inline Handler Is Registered

AsyncCallbackManager.on_llm_start dispatches to non-inline handlers only when there are zero inline handlers. Registering any inline handler (e.g., LangChainTracer) suppresses on_llm_start for all non-inline handlers, causing state-tracking custom handlers to receive on_llm_end without on_llm_start.

highConfidence 95%Langchain-Core

Origin Analysis

The on_llm_start method partitions handlers into inline and non-inline groups, then uses a single if/else branch: if any inline handler exists, only inline handlers are awaited and non-inline handlers are skipped entirely. It should instead iterate over all handlers and dispatch each according to its run_inline flag, as on_chat_model_start correctly does.
```python import asyncio from langchain_core.callbacks import AsyncCallbackManager, AsyncCallbackHandler received = [] class H(AsyncCallbackHandler): def __init__(self, name, run_inline): self.name = name self.run_inline = run_inline async def on_llm_start(self, *args, **kwargs): received.append(self.name) mgr = AsyncCallbackManager(handlers=[H("inline", True), H("non_inline", False)]) asyncio.run(mgr.on_llm_start({}, ["hi"])) print(received) # got: ['inline'] expected: ['inline', 'non_inline'] ```

Fixing Code Block

Edge Case Audit

This change makes non-inline handlers execute concurrently even when inline handlers are present, which was already the behavior when no inline handlers existed. However, applications that relied on the buggy behavior (inline handler suppressing non-inline on_llm_start) will now receive additional callbacks, which could trigger unintended side effects. Verify state management and logging after upgrading. If rollback is needed, reapply the previous if/else structure; no migration steps are required beyond testing.

Ecosystem Topology