Build and Run a Local AI Phone Agent with ai-phone

What is ai-phone?

ai-phone is an AI phone agent that runs on your computer. It can receive incoming calls, place outgoing calls, conduct conversations from written instructions, and expose turn-by-turn commands for integration with an external agent.

The current release is designed for macOS on Apple Silicon. Speech recognition and synthesis run locally through MLX, even when the conversational language model is remote. The computer hosting ai-phone must be a supported Mac, but callers can use any telephone or SIP client.

Key features

• Incoming and outgoing calls: Configure either direction or enable both during setup.
• Local speech processing: Speech recognition and voice synthesis remain on the Mac.
• Flexible language models: Use a local MLX model, a Claude Code or Codex subscription, or an OpenAI-compatible endpoint.
• SIP and telephone connectivity: Test with a direct SIP address or connect a SIP trunk, Telnyx account, or Twilio account for telephone numbers.
• Command-line automation: Start calls with a prompt and receive the resulting transcript.
• Relay mode: Let another shell-enabled agent control each conversational turn.
• Voice choices: Use Phonon with its bundled English voice or Kokoro with American and British voices and adjustable speed.

ai-phone currently supports English and one active call at a time.

Requirements

Supported platform

You need a Mac with Apple Silicon. Intel Macs, Linux, and Windows are not currently supported because the speech backends rely on MLX and the installation includes platform-specific integration.

Memory and storage

• 8 GB RAM: Setup offers a smaller 2B model.
• 16 GB RAM: Recommended for the default 4B model.
• 32 GB RAM: Recommended for the 9B model.
• Storage: Allow several gigabytes for downloaded model weights. Setup checks available disk space.

Software and phone connection

Install Git, Homebrew, and uv before proceeding. The installer manages the required Python packages and native SIP library.

For calls, use a SIP address for direct connectivity or provide your own SIP trunk, Telnyx account, or Twilio account. Carrier fees and subscription limits may apply.

Install ai-phone

Clone the repository, enter its directory, run the installer, and launch the interactive setup:

git clone https://github.com/oxnw/ai-phone.git
cd ai-phone
./scripts/install.sh
uv run ai-phone setup

The setup process asks whether you want incoming calls, outgoing calls, or both. It then configures the language model, phone line, and voice. For a local model, setup recommends an option based on detected memory and downloads the required weights.

If your terminal does not work well with the standard interactive prompts, use plain mode:

uv run ai-phone setup --plain

Start the phone daemon

After setup, start the daemon from the repository directory:

uv run ai-phone run

Keep this process running whenever you want to receive calls. For outgoing-only use, start it before placing a call and stop it with Ctrl-C when finished.

Setup does not install a background service. You must start the daemon yourself after setup or after restarting the computer.

Place an outgoing call

Leave the daemon running and open another terminal in the repository. Use call with a destination number and a prompt describing the task:

uv run ai-phone call +15555550100 --json \
  --prompt "Ask whether they have a table for two at eight tonight."

Replace +15555550100 with the number you intend to call. When the call ends, the command returns its transcript. Every outgoing call begins with an AI disclosure.

Incoming calls instead use the greeting and instructions saved during setup.

Run a free local SIP test

You can test locally without placing a carrier call. Run setup, enable incoming calls, choose A SIP address, and retain the default name agent. With the ai-phone daemon running, execute:

pjsua --local-port 5070 sip:agent@127.0.0.1

This connects a local SIP client to the agent at the loopback address, allowing you to validate the basic incoming-call flow before configuring a telephone provider.

Control calls with relay mode

Relay mode is intended for an external agent with shell access. Instead of giving ai-phone one prompt for the entire conversation, the external agent can initiate a call, speak, listen, and decide what to do after each response.

Start a relay call

uv run ai-phone call +15555550100 --relay --json

Speak and listen turn by turn

uv run ai-phone say "Hi, I'd like to book a table for two." --json
uv run ai-phone listen --json
uv run ai-phone say "Eight o'clock, please." --json

End the conversation

uv run ai-phone hangup --say "Thank you. Goodbye." --json

Each command using --json writes one JSON object to standard output. The call command also writes progress events to standard error, so integrations should process those streams separately.

If there is no active call, listen --json waits for an incoming call and handles it in relay mode. An unanswered relay turn times out after 25 seconds by default. Consult the agent command reference for call IDs, transcript retrieval, configurable timeouts, and exit codes.

Choose models and voices

Speech recognition

ai-phone uses Parakeet locally through MLX for speech recognition.

Conversation model

The language model can be a local MLX model, your Claude Code or Codex subscription, or an OpenAI-compatible endpoint. Setup helps configure the selected option.

Voice synthesis

Phonon provides a bundled English voice. Kokoro offers American and British voices with adjustable speaking speed.

If you use a remote language model, it receives the call text, instructions, and conversation history. If an external agent uses a cloud provider, that provider can also receive any transcripts supplied to it. Speech recognition and synthesis still remain local, while phone audio travels over SIP and RTP.

Advanced configuration tips

Rerun setup when requirements change

Run setup again to change the voice, model, enabled call directions, or caller ID:

uv run ai-phone setup

Restart the daemon after changing the configuration so the new settings take effect.

Start with local SIP before adding a carrier

Validate the agent with the loopback SIP test first. Once local calling works, configure a SIP trunk, Telnyx, or Twilio if you need regular telephone numbers. For network-related issues, review the project's NAT checklist.

Select a model that fits available memory

Follow setup's RAM-based recommendation rather than choosing the largest model automatically. The 2B option is intended for lower-memory systems, while the default 4B and larger 9B models require more memory.

Account for current latency and testing status

The README reports a 1.50-second median first-audible response in the latest local run under heavy memory pressure, which missed the project's one-second target. Local call and interruption tests have run, but live carrier and end-to-end agent acceptance checks remain pending. Review the published measurements and latency history when evaluating deployment readiness.

Use the detailed references

• Usage guide for lines, forwarding, caller ID, models, and CLI commands.
• Example configuration for available settings.
• Architecture guide for components and call flow.
• NAT checklist for network configuration.

License considerations

ai-phone is licensed under AGPL-3.0-or-later. Its bundled PJSIP interface files use GPLv2-or-later; consult the project's license notes before distributing or integrating the software.

Conclusion

ai-phone provides a practical command-line workflow for running an AI phone agent on an Apple Silicon Mac. After installing it, configure a model and SIP line, start the daemon, and test locally before making carrier calls. For automation, relay mode gives an external agent structured, turn-by-turn control through JSON-producing shell commands.