Building a Real-Time Audio Pipeline with WebRTC

Real-Time Audio Pipeline with WebRTC : A FastAPI server using PyTorch

The demand for real-time audio processing has exploded with the rise of voice assistants, AI agents, and communication tools. Processing audio in real-time requires a combination of low-latency transport and efficient inference. This article explores how to build a production-ready audio processing pipeline using FastAPI, LiveKit's WebRTC infrastructure, and PyTorch for audio enhancement and understanding.

Why WebRTC for Real-Time Audio?

When it comes to real-time voice applications, the choice of transport protocol is critical. WebRTC is purpose-built for real-time media and consistently delivers sub-100ms latency. Here's how it compares to WebSocket:

WebRTC excels at handling the packet loss, jitter, and network variability that would cripple TCP-based WebSocket connections in voice applications.

System Architecture Overview

The architecture follows a microservices pattern with clearly separated responsibilities:

text:

┌─────────────────────────────────────────────────────────────┐
│                    Browser Client                           │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐ │
│  │ Microphone  │  │ WebRTC      │  │ Audio Output        │ │
│  │ Capture     │  │ Client      │  │ Playback            │ │
│  └─────────────┘  └─────────────┘  └─────────────────────┘ │
└────────────────────────┬────────────────────────────────────┘
                         │ WebRTC (UDP)
┌────────────────────────▼────────────────────────────────────┐
│                    LiveKit Server (RTC)                      │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐ │
│  │ Signaling   │  │ Audio Track │  │ SFU Forwarding      │ │
│  │ (Port 7880) │  │ Management  │  │ (Ports 50000-60000) │ │
│  └─────────────┘  └─────────────┘  └─────────────────────┘ │
└────────────────────────┬────────────────────────────────────┘
                         │
┌────────────────────────▼────────────────────────────────────┐
│              Agent Worker (Python + LiveKit SDK)            │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐ │
│  │ Room        │  │ Audio       │  │ PyTorch Models      │ │
│  │ Connection  │  │ Processor   │  │ (ONNX Optimized)    │ │
│  └─────────────┘  └─────────────┘  └─────────────────────┘ │
│  ┌─────────────────────────────────────────────────────────┐ │
│  │ DTLN Denoising │ Speech-to-Text │ Pitch Detection     │ │
│  └─────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘

The architecture consists of three main components:

1. Web UI: A React frontend that captures microphone audio and streams it via WebRTC
2. LiveKit Server: Handles WebRTC signaling, media routing, and room management
3. Agent Worker: Python application that connects to the room, processes audio, and streams it back

This pattern is demonstrated in projects like FireRedChat, where LiveKit RTC coordinates with AI processing services for real-time voice interactions.

Setting Up the FastAPI Token Server

Before clients can connect to LiveKit, they need a JWT token. A FastAPI server handles authentication and token issuance:

python:

from fastapi import FastAPI
from livekit import api
import os

app = FastAPI()

@app.post("/token")
async def get_token(room_name: str, identity: str):
    grant = api.VideoGrants(
        room_join=True,
        room=room_name,
        can_publish=True,
        can_subscribe=True,
    )
    
    token = api.AccessToken() \
        .with_identity(identity) \
        .with_grants(grant) \
        .with_ttl(3600)
    
    return {"token": token.to_jwt()}

The token server is the security gatekeeper, ensuring only authorized users can join rooms. This pattern is well-established in LiveKit-based voice applications.

The Agent Worker: Audio Processing Pipeline

The agent worker is where the audio magic happens. It connects to the LiveKit room, receives audio frames, processes them with PyTorch models, and streams the enhanced audio back.

Audio Denoising with DTLN

For real-time noise suppression, DTLN (Dual-Signal Transformation LSTM Network) offers a compelling self-hosted alternative to cloud-based services like Krisp:

Here's how to integrate DTLN noise suppression into the agent:

python:

from livekit.agents import room_io
from livekit.plugins import dtln

# In the agent worker
async def process_room():
    await session.start(
        audio_input=room_io.AudioInputOptions(
            noise_cancellation=dtln.noise_suppression(
                strength=0.5,  # 0.0 = bypass, 1.0 = full suppression
            ),
        ),
    )

DTLN runs entirely in-process using ONNX Runtime with pretrained models bundled in the PyPI wheel (~4 MB). Each instance maintains stateful LSTM hidden states scoped to a single session.

Speech-to-Text Integration

For real-time transcription, the modular audio pipeline pattern combines FFmpeg conversion, denoising, WebRTC VAD (Voice Activity Detection) for speech cutting, and OpenAI Whisper for transcription.

The VAD component is critical for real-time systems. Silero VAD, a lightweight PyTorch model, can process audio with near-zero latency (~1ms) and activates when a short silence window is detected (typically stop_secs = 0.2s). Once speech is detected, the system passes the audio buffer to a TurnAnalyser for deeper semantic evaluation.

VAD and Turn Detection

Turn-taking is essential for natural voice interactions. The system uses a two-stage approach:

1. VAD (Voice Activity Detection): High-speed binary classifier detecting speech vs silence
2. TurnAnalyser: Analyzes the trailing ~8-9 seconds of audio using linguistic patterns, pitch intonation, and syntax boundaries

The outcomes are:

• Complete Thought: Voice agent yields turn and begins speaking
• Incomplete Thought: System holds silence and waits (up to a 3-second safety window)

ONNX for Optimized Inference

For production deployments, ONNX Runtime significantly improves inference performance. Silero VAD performance data shows the efficiency gains:

ONNX consistently delivers higher throughput across all configurations, making it the preferred deployment format for production systems.

Browser Client Implementation

The browser uses LiveKit's client SDK to connect to the room and stream audio:

typescript:

import { Room, RoomEvent } from 'livekit-client';

const room = new Room();
room.on(RoomEvent.TrackSubscribed, (track) => {
    // Play received audio
});

// Get token from FastAPI server
const token = await fetch('/token', {
    method: 'POST',
    body: JSON.stringify({ roomName: 'my-room', identity: 'user-1' })
}).then(r => r.json());

await room.connect('wss://your-livekit-server.com', token.token);

// Publish microphone audio
const micTrack = await room.localParticipant.createAudioTrack();
await room.localParticipant.publishTrack(micTrack);

The frontend captures microphone audio, publishes it to the room, and plays back processed audio received from the agent.

Production Considerations

Scalability

The microservices architecture allows independent scaling:

• LiveKit Server: Handles the WebRTC data plane with UDP ports 50000-60000 for media and WebSocket on port 7880 for signaling
• Agent Workers: Can be scaled horizontally based on processing demand
• AI Services: GPU-accelerated services like FireRedASR and FireRedTTS can be deployed separately

Monitoring and Observability

Production systems benefit from comprehensive observability, as demonstrated in the FireRedChat project, which uses health check endpoints for service availability monitoring and structured logging.

Model Selection

The choice of models depends on use case requirements:

• DTLN: Self-hosted noise suppression with ~8ms latency, free, and privacy-preserving
• Krisp Viva: Commercial noise filter exposing a FrameProcessor interface with adjustable noise suppression levels
• Silero VAD: Lightweight voice activity detection with near-zero latency

Real-World Implementation Examples

Several production projects demonstrate this architecture:

Real-Time Voice Agent with RAG: Combines LiveKit for audio transport, Gemini Live API for speech-to-text and LLM reasoning, and a local RAG module for grounded responses. The system consists of a token server, voice agent worker, and React frontend.

FireRedChat: Distributed microservices architecture with LiveKit RTC handling real-time media, FireRedASR for speech recognition, FireRedTTS for speech synthesis, and an Agents service for AI orchestration.

FastRTC: A library that turns Python functions into real-time audio and video streams over WebRTC, with built-in voice detection and turn-taking.

Conclusion

Building a real-time audio pipeline with WebRTC requires careful integration of transport, processing, and inference components. LiveKit provides the WebRTC infrastructure, FastAPI handles authentication, and PyTorch with ONNX delivers efficient audio processing. The combination of DTLN for denoising, Silero VAD for speech detection, and modular processing pipelines creates a scalable architecture suitable for voice assistants, real-time transcription, and audio enhancement tools.

The template presented here demonstrates how modern web frameworks can be combined with cutting-edge audio ML to build practical, production-ready applications. Whether you're building a voice assistant, a real-time transcription service, or an audio enhancement tool, this architecture provides a solid foundation for low-latency, bidirectional audio processing at scale.