Fix: AssistantMessageItem Content Bug In OpenAI Realtime API

This article dives deep into a specific bug encountered when using OpenAI's Realtime API in TEXT mode. We'll explore why the AssistantMessageItem objects in RealtimeHistoryUpdated and RealtimeHistoryAdded events might show empty content, even when the assistant has successfully generated a text response. This is a critical issue for developers relying on real-time text interactions, and thankfully, the solution is relatively straightforward. We'll break down the problem, pinpoint the root cause within the SDK, and provide a clear fix, along with a workaround for those needing an immediate solution.

Understanding the Problem: Empty Content in Text Mode

When you're building applications that leverage real-time communication with AI models, the ability to seamlessly receive and process assistant responses is paramount. OpenAI's Realtime API offers a powerful way to achieve this, enabling dynamic, back-and-forth conversations. However, a peculiar bug can arise when you're specifically using the API in TEXT mode, where your expected modalities are set to ["text"]. In this scenario, you might observe that the AssistantMessageItem objects delivered through RealtimeHistoryUpdated and RealtimeHistoryAdded events have an empty content array. This is puzzling because, logically, if the assistant has responded with text, its content should be present.

To further illustrate, let's contrast this with VOICE mode, where the modalities include ["audio"]. When operating in voice mode, the AssistantMessageItem correctly populates its content with the audio transcript. This discrepancy highlights that the issue is specific to how text content is handled and delivered in the real-time stream for text-only interactions. For developers, this means that while the API might be functioning and receiving a response, the structured data meant to represent that text response is missing, forcing you to implement workarounds or spend valuable time debugging. The core of the problem lies in how the SDK interprets and processes the incoming data from the real-time API, specifically when that data pertains to text output.

The Root Cause: A Mismatch in Text Type Handling

To truly understand and resolve this bug, we need to dive into the code. The root cause of the empty content issue in TEXT mode lies within the openai_realtime.py file, specifically in the _handle_ws_event() method. This method is responsible for processing the events received from the WebSocket connection.

When the API signals that a response part is complete (specifically, response.output_item.done events), the SDK checks the type of the content part. The code snippet provided reveals the logic:

if part.get("type") == "audio":
 converted_content.append({
 "type": "audio",
 "audio": part.get("audio"),
 "transcript": part.get("transcript"),
 })
elif part.get("type") == "text":
 converted_content.append({"type": "text", "text": part.get("text")})

Here's the crucial point: the realtime API, when sending text content in TEXT mode, labels it with the type `