Creating Natural Bilingual Intimate Voice Audio with ElevenLabs, Whisper, and ffmpeg

Project Overview

This workflow is designed to produce natural, coherent bilingual intimate audio in Chinese and English. The core principle is not to make a text-to-speech model imitate every sound, but to split the finished piece into tracks with different responsibilities: ElevenLabs handles dialogue and breathing, real recordings provide kissing and ambient sounds, ffmpeg handles trimming, time-stretching, mixing, and assembly, while Whisper serves as a quality-control tool.

Most important conclusion: The voice model should only handle “speech” and breathing integrated into the speaking rhythm. Kissing sounds, wet sounds, and other nonverbal effects should be produced separately and then mixed with the voice.

The workflow in this README was completed on a Mac mini, but the methods do not depend on a specific Mac model. As long as the system can call the ElevenLabs API, run ffmpeg, and use Whisper for transcription, the main steps can be reproduced.

Core Tools and Responsibilities

• ElevenLabs: Create a voice through Voice Design and generate dialogue with a text-to-speech model.
• ffmpeg: Standardize sample rates, slow the speech, trim assets, apply fades, mix, and assemble audio.
• Whisper: Check whether nonverbal assets are being spoken as recognizable words by the model.
• Legally licensed real recordings: Provide separate ambient sound. The original workflow used only public-domain CC0 assets and recorded their sources.

Installation and Preparation

1. Set Up ElevenLabs

Sign up for ElevenLabs and obtain credentials that can call the Voice Design and text-to-speech interfaces. The interfaces actually needed are POST /v1/text-to-voice/design and POST /v1/text-to-voice.

First submit voice_description and preview text to receive multiple candidate previews. After selecting a candidate, save the production voice using its generated_voice_id. Once saving succeeds, record the voice_id; use this ID for all subsequent synthesis.

2. Install ffmpeg

Confirm that ffmpeg can be executed directly from the terminal. On macOS, it can be installed through a commonly used package manager. After installation, run:

ffmpeg -version

3. Prepare Whisper

Whisper is not used to generate audio in this workflow. It acts as an “audio verification probe.” It checks candidate kissing and nonverbal clips: if a clip can be transcribed into words consistently, that usually means the model is actually speaking onomatopoeia rather than producing a genuine nonverbal sound.

4. Create an Asset Directory

It is recommended to separate files by function in advance. At minimum, include voice outputs, kissing candidates, approved kissing assets, raw ambient recordings, trimmed ambient-sound windows, and final mixes. This prevents unreviewed candidate assets from being used directly in the finished output.

Step One: Design a Bilingual Chinese-English Voice

Use a Text Description Instead of Choosing Only from the Voice Library

According to the README tests, existing Chinese voices may have an accent when speaking English. Explicitly describing a bilingual background through Voice Design makes it easier to obtain natural results in both Chinese and English. The description should specify four categories of information: native-language ability, age range, relationship or character qualities, and target emotion.

The following is the description used in the original workflow and can be adjusted to suit the character:

Native bilingual speaker of Mandarin Chinese (standard putonghua) and American English,
both at native level, with no cross-language accent in either. Male, 35–40. Studio quality.
Persona: mature, dominant, doting. Emotion: composed, doting, low-burning.
Rich, chesty, slightly gravelly low voice; unhurried, even pacing with weight on key words
and a breathy softness at the ends of sentences.

Do not use advertising copy for preview text. Choose lines the character would genuinely say in everyday situations, and generate multiple candidates from the same description. During previewing, prioritize whether the voice sounds like the target character speaking naturally rather than simply comparing whether it sounds deep or pleasant.

Choose the Right Model

• Ordinary dialogue: The README recommends v3 for expressing everyday emotions from the text.
• Real-time calls or intimate scenes: v4 is recommended. It is better suited to handling Chinese, English, and mixed Chinese-English text in the same voice, and is more likely to recognize lightweight voice-direction tags.
• Long passages in English only: If v4 is not used, choose eleven_turbo_v2_5 and explicitly set language_code=en to reduce a Chinese accent in English.

Adjust Voice Parameters

The README records the following tested starting points:

• stability=0.35: More variation and dramatic expression.
• stability=0.5: Closer to everyday speech and less deliberately performed.
• similarity_boost=0.8: Helps maintain consistency with the target voice.
• style=0.3: The original notes state that v3 does not recognize this parameter, so do not rely on it to control v3.

Change only one variable at a time during debugging. Keep a baseline audio file that has already received positive feedback, then adjust only stability or one tag at a time so the source of each change can be identified accurately.

Step Two: Integrate Breathing into the Speaking Rhythm

Add Tags Only When the Vocal Delivery Changes

Do not add tags to every sentence. Two or three lightweight tags per group of six to ten sentences are usually enough; later sentences can continue the preceding vocal delivery. Common terms in the original workflow include low, close, quiet, soft, and unhurried.

[low and close to her ear, unhurried]
…Come here. Don’t hide.

…A little closer. That’s it.

[quiet laugh under the breath]
…What’s the rush?

Strong words such as intense, heavy, growl, strained, and rough are not entirely unusable, but in the original tests they easily made the voice exaggerated. A more reliable strategy is to ask for a lower, closer, slower delivery rather than forcing the model to perform intensely.

Shape Breathing with Punctuation

Instead of repeatedly writing heavy breathing, adding a moderate number of ellipses at the beginning, end, or middle of a sentence makes it easier to produce natural pauses. Arrange only two or three prominent breaths per group, varying how they are expressed, such as a gentle exhale, a pause midway through a line, a brief held breath, a smiling exhale, or a breath that is cut short.

Repeating the same breath sound makes the repetition obvious. Listen to the entire generated passage rather than checking only individual lines.

Generate Full Passages Whenever Possible

When a complete passage is generated at once, breathing, emotion, and pitch are more likely to remain continuous. Synthesizing eight sentences separately and then joining them makes each segment sound as if it is starting over. If a real-time call cannot be generated as one passage, use blocks of two sentences or about forty Chinese characters instead of one sentence per block.

Pass Along Context

v4 supports previous_text and next_text. When generating blocks one by one, passing the surrounding text to the model can reduce the mechanical reading quality. The README specifically notes that v3 does not support these two fields and may return an error if they are passed to v3, so requests must be constructed separately for each model.

Avoid Incorrect Language Routing

If the program determines the model or language based on the proportion of Chinese characters, it must strip English tags inside square brackets first. Otherwise, a short Chinese sentence may be incorrectly classified as English because it contains many English tags. With v4, the same model can be kept throughout, reducing Chinese-English switching.

Step Three: Control Speech Rate

In the original tests, v4’s speed parameter did not produce the expected effect. To make v4 slower, first add natural pauses to the text, then slow the audio with ffmpeg while preserving its pitch:

ffmpeg -i input.mp3 -filter:a "atempo=0.9" output.mp3

For English-only content using the turbo model, the README records speed=0.84 as a usable starting point. Pauses can also be added between sentences:

<break time="0.4s"/>

Do not stack too many pause tags in a single segment. The original tests recommend no more than about six; too many may make the output unstable.

Step Four: Build a Reusable Kissing-Asset Library

Why Not Write Onomatopoeia Directly?

When a text-to-speech model is asked to read kissing onomatopoeia, the common result is recognizable syllables or letter-like sounds rather than natural nonverbal audio. Writing only an action tag may also degrade into an ordinary hum. The original project therefore used an asset-library approach: generate many candidates first, then trim and filter them.

Production Steps

1. Using the same voice and the same passage containing kissing actions, generate five or six complete takes. The original workflow used v4 with stability=0.35.
2. Use silence detection or an audio editor to trim candidate nonverbal clips from each output.
3. Transcribe every clip with Whisper. Immediately discard clips that produce clear characters or words.
4. Check the ratio between differential energy and original energy. In the original tests, more credible clips were usually above 0.2; clips spoken as characters were around 0.12 to 0.16; ordinary humming was generally below 0.1. These figures came from specific assets and should be treated only as initial references.
5. Listen to the clips that pass the automated checks and organize them by category, such as light kisses, deeper kisses, and changes in position.
6. Select a recently unused clip at random when needed, applying slight pitch and speed variation of no more than about 3%. The original workflow allowed slowing only, not speeding up.

Automated filtering cannot replace human listening. The README records an instance in which a clip containing sustained hiss was transcribed by Whisper as something like Sssss, yet entered the final output because the checks were incomplete. The correct process should check the transcription, waveform metrics, and actual listening impression together.

Step Five: Complete the Two-Track Mix with Real Ambient Sound

Choose the Assets

The original workflow compared three options: having the voice model simulate the sound, generating AI sound effects, and using real recordings. It ultimately chose real recordings with no voices and clear CC0 licensing. Before using any external recording, verify its license and save a record of its source.

Do Not Simply Multiply by a Fixed Volume

The average loudness and peak level of an ambient recording can vary considerably, and it may also contain long quiet sections. Applying a fixed volume multiplier can easily make the sound inaudible or cause it to cover the voice. A more stable approach is:

1. Randomly select two windows from the original asset, each approximately as long as the voice track.
2. Measure the peaks of both windows and choose the one containing more effective sound.
3. Calculate the gain so that the ambient peak is approximately 12 dB below the current voice peak.
4. Apply a 0.5-second fade-in to the ambient sound.
5. Keep about 1 second of tail after the voice ends and apply a fade-out of about 1.4 seconds.
6. When the kissing asset already has the required texture, do not layer additional ambient sound.

ffmpeg Mixing Example

Assume the voice is v.mp3, the trimmed ambient window is w.wav, GAIN is the gain calculated from peak alignment, and END is the point where the fade-out begins:

ffmpeg -i v.mp3 -i w.wav -filter_complex \
 "[1:a]volume=GAIN,afade=t=in:st=0:d=0.5,afade=t=out:st=END-1.4:d=1.4[w];\
  [0:a]apad=pad_dur=1.0[v];[v][w]amix=inputs=2:duration=first:normalize=0[a]" \
 -map "[a]" -c:a libmp3lame -q:a 2 out.mp3

normalize=0 is important. According to the README tests, omitting it may cause amix to reduce the voice level noticeably.

Separate Mixing Markers from Performance Tags

During the writing stage, you can use tags such as [low, close, wet], with wet serving as a post-production mixing marker. Before requesting speech synthesis, the program must remove this word so the model does not attempt to perform the sound effect. After the voice is returned, the corresponding real asset can be layered according to the marker. When the next tag no longer contains the marker, the ambient sound should fade out.

Step Six: Produce the Complete Voice Track

Divide the script into three types of blocks: ordinary dialogue blocks are sent to text-to-speech; pure nonverbal blocks are selected from the asset library; blocks with an ambient-sound marker are first synthesized as voice and then processed through the two-track mixing workflow.

Audio from different sources may use different sample rates, so they should be standardized before concatenation. For example, convert them to 44100 Hz first:

ffmpeg -i input.wav -ar 44100 output.wav

After standardizing the sample rate, use ffmpeg's concat filter to join the clips into one track in script order. After concatenation, listen from beginning to end, paying close attention to sudden changes in timbre, background noise, loudness, and breathing between clips.

Step 7: For Real-Time Conversations

The key to real-time mode is reducing wait time while preserving context. Split the text at labels placed on separate lines, allowing subsequent sentences without new labels to inherit the current voice. Submit a generation request after accumulating two sentences or about forty Chinese characters.

1. While the current block is playing, generate the next block in the background.
2. Pass the sentence before and the sentence after the current block to v4.
3. Take purely nonverbal blocks directly from the asset library instead of requesting text-to-speech.
4. Turn on the microphone only after the final sentence finishes playing to prevent overlapping voices.
5. If synthesis fails, show only the text or remain silent briefly; do not switch to another voice at the last moment.

A character break caused by switching voices midway is usually more noticeable than a pause of around one second. Therefore, maintaining a consistent character voice should take priority.

Advanced Debugging Techniques

Establish a Fixed Reference Sample

Save a reference audio clip that has already been confirmed to sound natural. Change only one of the following at a time: stability, labels, punctuation, model, or post-processing speed. Then compare it with the reference version in a blind listening test.

Separate Generation Labels from Post-Processing Labels

The voice model should receive only the voice-character labels it is expected to perform. Ambient sounds, asset categories, and mix intensity are post-processing metadata and must be removed before making the request. This prevents the model from reading production instructions aloud or attempting to imitate sound effects.

Save Recent Usage History

Maintain a recent-use queue for the asset library, such as excluding the ten clips used most recently. This reduces repetition and is more consistent than selecting completely at random each time.

Automate Quality Gates

The following checks can be added to the output pipeline:

• Did Whisper transcribe words that should not be present?
• Is the peak difference between the voice and ambient sound close to the target value?
• Is the sample rate consistent?
• Does the output preserve a natural tail at the end?
• Does the requested model support the supplied parameters?
• Were the English labels removed before language detection?

Costs, Licensing, and Content Boundaries

The README records use of the Creator plan, priced at $22 per month with an allowance of approximately 120,000 characters. Designing a voice for the first time consumes about 3,000 characters, while daily use ranges from several hundred to 2,000 characters. The README also mentions a limited-time v4 promotion, so actual pricing, quotas, and billing methods should be verified on the official page at the time of use.

All content must involve clearly identified adults only. Do not use character settings that could be understood as minors or ambiguous age descriptions. The platform reviews content based on the text, and violations may result in account suspension.

Authorization must be confirmed when using recordings made by others. Even if an asset is available for free download, that does not mean it may be redistributed or used commercially. The original workflow uses only CC0 assets and records the source of every file.

Conclusion

Natural intimate speech does not come from piling on intense labels. A more reliable approach is to have the voice model focus on dialogue, subtle emotion, and breathing rhythm; separate nonverbal sounds into assets screened through transcription and manual listening; and then use ffmpeg for peak matching, fades, and concatenation.

Start with a minimal version: build stable bilingual voices first, then validate full-segment generation and context transfer, and add the asset library and ambient sounds afterward. Keeping a reference sample at each step and changing only one variable at a time usually makes it easier to achieve natural, consistent, and reusable results than continually adding more labels.