Build an AI Real-Time Livestream from Scratch: Digital Humans, Chat Moderation, and OBS Streaming

Project Overview

AI Real-Time Livestream is an open-source project that enables AI avatars to livestream autonomously. When viewers send comments, the system filters the content first, then queues requests according to the current voice and motion state, preventing a new instruction from interrupting the previous sentence.

The project uses Vidu S2 Real-Time Avatar to generate video, voice, and replies. This project handles comment filtering and scheduling, character profiles, wardrobe management, livestream overlays, the local workbench, and output to OBS or direct streaming.

The code is released under the MIT license. The repository does not include the author's own character appearance, persona, voice, or asset library. After deployment, you must prepare your own character assets and assume responsibility for content copyright and compliance.

Features and Limitations

Currently Supported

• Chat replies based on a custom persona, with configurable opening lines.
• Switch complete outfits from a pre-made wardrobe. Each change takes about 6 seconds.
• Trigger small motions such as forming a heart with the hands, waving, and blinking through keywords.
• When no viewers are speaking, automatically warm up the stream from the character's topic pool without repeating a topic before the current round is finished.
• Automatically filter vulgar or promotional comments. Gifts do not jump the queue.
• Optionally automatically restart the stream after Vidu disconnects mid-session.

Important Limitations

• Background changes, eating and drinking, and prop handling are disabled by default because real-time redrawing may deform the character's clothing.
• The project is currently unsuitable for singing, playing music, standing, full-body movements, large dance motions, or livestream shopping.
• It does not remember returning viewers across livestream sessions.
• Douyin comments are not captured internally. Use the platform's official open APIs; do not scrape livestream webpages.

Background and prop features are disabled by default. For private testing only, you can set {"safety":{"live_visual_edits":true}} in config.json. Enabling this for public livestreams is not recommended.

Prepare the Runtime Environment

Prepare the following environment and accounts before you begin:

• macOS or Linux.
• Python 3.11 or later. The author uses Python 3.13.
• ffmpeg with libx264 and aac.
• A Vidu Open Platform API Key beginning with vda_, with sufficient available credits.
• An optional Evolink API Key for generating character images, clothing, props, and backgrounds, and for uploading audio during voice cloning.
• Optional OBS 30 or later for cameras, overlays, and streaming.
• An optional YouTube stream key, required only when using built-in streaming.

Vidu Real-Time Avatar is billed by duration. The official price listed in the README is 1.5 credits per second, with 1 credit equal to 0.03125 yuan, or approximately 169 yuan per hour. It is recommended to verify the workflow first with dry-run mode and the local mock service.

Store Keys Securely

Keys should be stored only in local files and read by the server; they are never sent to the browser. Create the default directories first:

mkdir -p ~/.deepsea/vidu ~/.deepsea/evolink && chmod 700 ~/.deepsea/vidu ~/.deepsea/evolink

On macOS, you can save the Vidu Key from the clipboard to its default path:

pbpaste > ~/.deepsea/vidu/api.key && chmod 600 ~/.deepsea/vidu/api.key

Put the Evolink Key in ~/.deepsea/evolink/api.key. If you use built-in YouTube streaming, put the stream key in ~/.deepsea/youtube/stream.key; put the OBS WebSocket password in ~/.deepsea/obs/ws.password. The default paths can also be changed in config.json.

Install the Project

Clone the repository from GitHub and enter the project directory:

git clone https://github.com/garycaogan/ai-live-avatar.git
cd ai-live-avatar

Create a Python virtual environment and install the dependencies:

python3 -m venv .venv
.venv/bin/pip install -r requirements.txt

The stage page depends on the Alibaba Cloud RTC web SDK. The project does not redistribute this SDK; instead, a script downloads and verifies it from the official CDN:

./tools/fetch_vendor.sh

Finally, copy the example configuration file:

cp config.example.json config.json

Start the Workbench

Run the service:

.venv/bin/python server.py

The service listens only on the local machine by default. After starting it, open http://127.0.0.1:28900/workbench in your browser to enter the workbench.

If you do not want to consume Vidu credits yet, use Dry-Run Start on the livestream page. This mode does not call Vidu; it only demonstrates comment filtering, queuing, and overlays. The project also provides tests/dev_mock_stack.py, which connects to a locally mocked Vidu service to help you test the complete control flow.

Create Your Own AI Avatar

1. Prepare a Character Image

Upload a character image on the Assets page, or generate one with Evolink. The base image affects the character's appearance, clothing, room, and persistent expression. The README recommends a close-up image of one person, with both hands on the table and a natural expression.

Landscape compositions are placed in a small frame above a Douyin-style vertical livestream page and cropped at a 1.2:1 ratio, so the base image should be composed accordingly in advance.

2. Create a Character Profile

Create a character on the Characters page and select the character image as the base image. Then enter the name, persona, and opening line. Focus the persona on speaking style rather than writing complete example sentences, or the model may recite the examples verbatim.

3. Add a Wardrobe

Add several outfits on the Wardrobe page. Each outfit should be a complete photo, with the room, lighting, and composition kept as consistent with the base image as possible. Otherwise, the image may change noticeably during outfit switches.

4. Configure the Voice

Select a system voice on the Voice page, or upload 10 to 20 seconds of clean speech for cloning. Vidu voice cloning is paid. Confirm that you have authorization for the voice before uploading any material.

5. Configure Motions

You can edit the motions associated with comment keywords on the Motions page. Character settings also let you edit idle motions, making the image more natural when there are no new comments.

Character profiles are stored in assets/avatars/profiles.json, and personas are stored in personas/. These files are excluded by .gitignore and will not be committed to the repository.

Understand Comment Processing

The livestream page in the workbench supports manually sending simulated comments. Real comments can be submitted through the following endpoint:

curl -s -X POST http://127.0.0.1:28900/api/danmaku \
  -H 'Content-Type: application/json' \
  -d '{"user":"Viewer A","text":"Change your outfit","gift":0}'

After a comment enters the system, it is checked against the filter words in moderation.json. Then live/router.py identifies it as chat, an outfit change, a prop request, a background request, a motion, or a reset according to the rules. This identification does not call a large language model, so it adds no extra latency or cost.

By default, prop and background requests are handled as ordinary chat. Comment recognition works best with simple, explicit expressions, such as asking about something nearby, naming an outfit from the wardrobe, or making only one motion request at a time.

The following typically affect results: multiple tasks in one sentence, requests for real-time news or weather, and requests to sing or play music. It is recommended to split complex requests into several simple comments.

Connect OBS

On the Streaming page in the workbench, switch the output method to OBS, then add a browser source in OBS. Select one of the following URLs as needed:

• Avatar only: http://127.0.0.1:28900/stage?obs=1&role=obs&layer=avatar
• Transparent overlay only: http://127.0.0.1:28900/stage?overlay=1
• Avatar and overlay: http://127.0.0.1:28900/stage?obs=1&role=obs&layer=all

For the full frame, a width of 1080 and height of 1920 are recommended. Enable “Control audio via OBS” for the browser source. Because the service listens only on the local machine, OBS and the project service must run on the same computer.

To create an OBS scene automatically, use tools/obs_control.py. The overlay permanently displays the label “AI-generated live video,” which is required by the platform for AI avatar livestreams and should be retained.

Configure the Douyin Livestream Layout

Douyin's common AI avatar livestream layout uses a landscape frame. On mobile, the video appears in the upper area, with comments below. Run the following command to create a “Douyin Livestream” scene:

.venv/bin/python tools/obs_control.py setup douyin --camera FaceTime

This command sets the OBS canvas to 1296×1080, a 1.2:1 ratio, and configures the avatar, information card, subtitles, and real camera. After --camera, enter part of the camera name. If omitted, the first real camera found is used.

On the first run, macOS may ask OBS for permission to use the camera. Allow it, then run the command again so the camera crop can be applied correctly. Landscape and portrait scenes can be switched at any time. Both use the same avatar layer and do not consume additional meeting slots.

There are two ways to connect to Douyin Live Companion: enable the OBS virtual camera and select OBS Virtual Camera in Live Companion, or add a window source in Live Companion and capture the OBS projector window. The virtual camera transmits video only, not audio, so you must also confirm that Live Companion can capture the computer audio from the “Douyin Livestream” scene.

If the account has permission for third-party streaming, you can also enter the platform-provided streaming URL directly in OBS. Douyin comments will not automatically enter the project. You must apply for permission through the livestream features API of the Douyin Open Platform, then push the data to /api/danmaku.

Edit the Livestream Information Card

The information cards in the upper-left and upper-right corners of the stage page show what the current livestream can and cannot do, as well as how to send more effective comments. Their content comes from assets/stage_panel.json; if that file does not exist, assets/stage_panel.example.json is used.

Copy the example file and edit it with your own content. The stage page refreshes automatically within one minute, while overlays in built-in streaming use the new content immediately. outfit_hint can provide outfit-change prompts for different characters.

Enable Automatic Stream Recovery

Vidu may occasionally disconnect during a livestream. With automatic recovery enabled, the project waits 3 seconds before restarting the stream, with a maximum of 6 attempts per hour. Disconnects caused by insufficient credits or content review will not be retried automatically.

Enable it through the API:

curl -s -X POST http://127.0.0.1:28900/api/settings \
  -H 'Content-Type: application/json' \
  -d '{"auto_recover":true}'

You can also set it in config.json:

{"schedule":{"auto_recover":true}}

Before a long livestream, check your account balance, character assets, and network status. Verify the recovery process with dry-run mode or the local mock service first.

Testing and Project Structure

The project tests use a local Vidu mock and do not incur charges. Run:

python -m unittest tests.test_logic tests.test_workbench tests.test_e2e

The main directories and files are used as follows:

• server.py: Service entry point and default configuration.
• live/: Live streaming orchestration, the Vidu real-time API, chat message recognition, wardrobe, asset generation, built-in streaming, and workbench APIs.
• static/: Workbench, stage page, and legacy console.
• WORKBENCH_API.md: All APIs used by the workbench.
• tests/: Test code and local mock services.

Conclusion

This project connects Vidu real-time digital humans, chat rules, character assets, and live-stream output into a local workflow. We recommend starting in dry-run mode to become familiar with filtering, queuing, the wardrobe, and overlays. Then configure your own avatar, voice, and OBS scenes before connecting to a real platform.

For public livestreams, disable real-time background and prop editing that could distort the image, use the platform's official interaction APIs, and retain AI-generated content labels. You should also confirm authorization in advance for real-person voices, character likenesses, music, and other assets.