SeedRealtime – ByteDance’s Native Audio-Visual Full-Duplex Foundation Model Operational

SeedRealtime is a native audio-visual full-duplex foundation model developed by ByteDance’s Seed team. Built on a unified architecture that natively integrates audio, video, and text, it enables fully multimodal real-time interaction while seeing, listening, and speaking simultaneously. The model delivers three major breakthroughs: joint audio-visual understanding, proactive interaction, and smooth conversational pacing. End-to-end evaluations show that conversational pacing issues are reduced by half compared with cascaded models. It is now fully deployed in the Doubao app, becoming the industry’s first audio-visual full-duplex AI product to achieve large-scale deployment.

Joint audio-visual understanding: Natively and deeply integrates audio, visual, and temporal information. It can use visual context to resolve homophone ambiguity, accurately interpret temporal references, and provide unified perception of what is seen, heard, and said.
Proactive interaction: Continuously perceives its environment and can express itself proactively. It can alert users when the scene changes and call tools to enrich its responses, upgrading interaction from passive responses to proactive collaboration.
Smooth conversational pacing: Perceives the user’s conversational state and interaction rhythm in real time, naturally taking turns, pausing, and responding at appropriate moments. It is highly resistant to interference and can distinguish side conversations from background noise.
Multi-person scene recognition: In crowded and noisy environments, it simultaneously identifies faces, distinguishes sound sources, and understands content while continuously associating each person’s voice with their identity.

End-to-end unified modeling: Uses an end-to-end unified audio-visual modeling framework that incorporates perception, understanding, decision-making, and expression into a single model operating synchronously. This avoids the multi-stage information loss and accumulated latency of the ASR → VLM → TTS pipeline in cascaded systems.
Native full-duplex interaction: Does not rely on external VAD rules to determine conversational turns. Instead, the model continuously decides the conversational state and timing based on multimodal information streams, truly enabling “speaking while listening” rather than the question-and-answer pattern of half-duplex interaction.
Audio-visual temporal alignment: Uniformly models audio, visual, and temporal information. By combining the current scene, gestures, gaze, and historical actions, it understands user intent and performs cross-modal disambiguation and reference resolution.
Low-latency engineering optimization: Uses chunked audio-visual input and streaming generation, combined with efficient quantization and inference optimization, to continuously reduce end-to-end latency from “hearing” to “understanding” to “responding.”

Update the Doubao app: Update the Doubao app to the latest version.
Enter a voice call: Open the app and tap the “Call” button in any conversation.
Switch to video mode: After entering the call interface, enable camera and microphone permissions and switch to a video call.
Start real-time interaction: Show the scene while speaking naturally. The model will perceive the audio-visual stream synchronously and respond in real time.
Use proactive observation: Give the model an instruction to observe continuously. It will proactively alert you when the target appears or the scene changes.

End-to-end unified architecture: Uses a single model to natively integrate audio, video, and text, eliminating the information loss and accumulated latency caused by chaining multiple modules in cascaded systems.
Native full-duplex interaction: Does not rely on external VAD rules. The model independently determines conversational timing based on multimodal information streams, truly enabling “speaking while listening” rather than a question-and-answer exchange.
Joint audio-visual understanding: Deeply integrates audio, visual, and temporal information. It can use visual context to resolve homophone ambiguity and accurately interpret cross-modal references such as “this” and “there.”
Proactive environmental perception: Continuously observes the visual environment and proactively alerts users when the scene changes or a target appears, upgrading interaction from passive response to proactive collaboration.
Smooth, interference-resistant pacing: Perceives the user’s conversational state in real time and accurately distinguishes side conversations from formal questions in noisy environments. Stuttering and false triggers are reduced by half compared with cascaded models.

Project website: https://seed.bytedance.com/en/seedrealtime

Smart tour guide: Continuously observes exhibits in a museum and proactively explains their historical background and craftsmanship details when a target appears.
Foreign-language practice: Uses visual input to provide real-time pronunciation correction and example sentences while remaining focused on the target learner in noisy environments.
Equipment operation guidance: Provides real-time corrections and adjustment suggestions based on visual changes when users operate complex equipment.
Travel information assistant: Identifies flight information on airport display screens in noisy environments, answers questions about arrival times, and provides route guidance.
Children’s educational support: Observes learning materials in real time, corrects pronunciation, and remains unaffected by background voices while helping children study.