/* BEGIN BLOG-CLOSE-CHAT */ /* END BLOG-CLOSE-CHAT */
View Categories

10. AI Voice Agent

2 min read

Table of Contents

10. AI Voice Agent #

Service Provider Edition only. The AI Voice Agent is an optional add-on that a server administrator installs separately.

The AI Voice Agent lets users manage the PBX in plain language by phone: just dial *99. Instead of clicking through the portal, you speak a request ("What's my balance?", "Read my last voicemail", "Create a ring group") and the agent carries it out through the same REST API the web portal uses.

It is an extra front-end, not a replacement: the web portal and softphone keep working exactly as before.

How it works #

When you dial *99, FreeSWITCH streams your call audio to the AI Gateway service running alongside ICTPBX:

Dial *99  →  FreeSWITCH  ──audio──▶  AI Gateway
                                        │
              Speech-to-Text ──▶ Language Model (understands + picks an action)
                                        │
              Text-to-Speech ◀── ICTPBX REST API (does the action, returns result)
  • Speech-to-Text: your speech is transcribed by OpenAI (gpt-4o-transcribe).
  • Language Model: an OpenAI model understands the request and decides which PBX action to run.
  • Text-to-Speech: spoken replies are generated locally on your server with Piper.
  • ICTPBX REST API: performs the action, exactly as if you had done it in the portal.

Installation (server administrators) #

The agent is opt-in and needs an OpenAI API key. Install it either way:

  • During a fresh install or upgrade: run the installer with WITH_AI=1 OPENAI_API_KEY=<your-key>.
  • On an existing server: run ai-gateway-install.sh.

If dialing *99 returns busy or silence, the agent is not installed or not running.

Security: who the agent acts as #

The agent has no privileges of its own. Every action is an API call made with a scoped session for one specific user:

  • Identity comes from the calling extension. There is no PIN. The agent identifies the directory user the call authenticated as, within that tenant's SIP domain.
  • It acts as the user the extension is assigned to (PBX → Extensions → Assign to User). If the extension is not assigned to anyone, it acts as the tenant admin. Assign every extension to its real user so that each caller gets only their own permissions.
  • It can only see and change what that user's role and tenant already allow. An End User cannot reach another tenant's data or admin-only settings.
  • Changes require spoken confirmation. Before any create, update or delete, the agent reads back what it is about to do and waits for you to say "yes".

How to use it #

  1. From your registered extension or softphone, dial *99.
  2. Wait for the greeting, then say what you want. Examples:
    • "What is my account balance?"
    • "How many extensions do I have?"
    • "Do I have any new voicemails?"
    • "Create a ring group called Support."
  3. For anything that changes configuration, confirm out loud: say "yes" to proceed or "no" to cancel.
  4. Hang up when you're done.

A Super Admin can manage the whole system this way, a Tenant Admin manages their tenant, and an End User can only work with their own extension, voicemail and account.

To check voicemail directly without the agent, dial *99<mailbox> (for example *991001).

Audit & privacy #

  • Every action the agent takes is logged in an audit trail.
  • Voice privacy: your spoken audio and the request text are sent to OpenAI for transcription and reasoning. Spoken replies are generated locally. If your organization has data-egress or compliance requirements, review this before enabling the agent.

Availability #

The agent is voice-only today: access is via *99. A browser chat surface over the same agent is planned.

What are your Feelings