AI & Agent Dev Bug Sandbox logo
AI & Agent Dev Bug Sandbox
Back to Radar

VideoContentBlock Not Supported In OpenAI Block Translator

The OpenAI block translator in langchain-core does not handle VideoContentBlock, causing a ValueError when sending video content blocks to OpenAI-compatible models via langchain-openai.

mediumConfidence 92%LangchainAffected Vlangchain-Core 1.4.0Affected Vlangchain-Openai 1.2.2

Origin Analysis

The function convert_to_openai_data_block in langchain_core/messages/block_translators/openai.py lacks a branch for block type 'video', so it falls into the generic else clause and raises ValueError.
1. Initialize a chat model using init_chat_model with an OpenAI-compatible base URL (e.g., DashScope). 2. Create a HumanMessage with content_blocks containing a TextContentBlock and a VideoContentBlock with url and mime_type. 3. Invoke the model with this message. The call fails with ValueError: Block of type video is not supported.

Fixing Code Block

def convert_to_openai_data_block(block: dict, api: str = 'openai') -> dict: if block['type'] == 'text': return {'type': 'text', 'text': block['text']} elif block['type'] == 'image': image_url = block['url'] if block.get('detail'): return {'type': 'image_url', 'image_url': {'url': image_url, 'detail': block['detail']}} return {'type': 'image_url', 'image_url': {'url': image_url}} elif block['type'] == 'audio': return {'type': 'input_audio', 'input_audio': {'data': block['data'], 'format': block['format']}} elif block['type'] == 'video': video_url = block['url'] mime_type = block.get('mime_type', 'video/mp4') return {'type': 'video_url', 'video_url': {'url': video_url, 'mime_type': mime_type}} else: raise ValueError(f"Block of type {block['type']} is not supported.")
Add a dedicated elif branch for block type 'video' that maps VideoContentBlock to the OpenAI video_url content part format, including url and mime_type. The mime_type defaults to 'video/mp4' if absent to avoid KeyError.

Edge Case Audit

This fix assumes the target OpenAI-compatible endpoint supports the video_url content type. Some models or providers may not support video input, causing a different API error. Additionally, if VideoContentBlock is used without a mime_type, the default may not match the actual video format, leading to rejection. Rollback advice: if you encounter provider-specific issues, revert this change and construct the video_url content part manually as a plain dict in your message content.

Ecosystem Topology