Audio Processing

Audio is the part of a call people notice first. A frozen video frame is tolerable. A garbled, echoing, or clipping voice is not. This page covers the full audio path: how capture constraints shape the signal, how the Opus codec carries it over the wire, and how the Web Audio API lets you measure, modify, and mix audio before it leaves the browser.

The data path stays peer-to-peer. Audio rides an RTCPeerConnection as SRTP, never through a relay. Signaling carries only SDP; no sample ever transits a server. Everything here runs on the client.

The three audio API layers

Audio in WebRTC spans three APIs that hand off to each other.

You do not need the middle layer for a basic call. getUserMedia gives you a track, you add it to the connection, and the browser handles encoding. The Web Audio layer is for when you want a level meter, a volume control, a mute that the far end cannot hear pop, or a single track mixed from several sources.

For the source track lifecycle and replaceTrack, see Video & Audio Tracks. For acquiring the device in the first place, see Capture. The reference page for the track object is MediaStreamTrack.

Capture constraints that shape the signal

Three audio constraints change the signal before your code ever sees it. The browser's audio processing module applies them at capture time. They default to true, which is correct for a voice call and wrong for music.

const stream = await navigator.mediaDevices.getUserMedia({
  audio: {
    echoCancellation: true,
    noiseSuppression: true,
    autoGainControl: true,
  },
});

echoCancellation

Acoustic echo cancellation (AEC) removes the far end's voice from your microphone signal. Without a headset, your speaker plays the remote person's voice, your microphone picks it up, and you send it back to them. They hear themselves a fraction of a second late. AEC subtracts the known speaker output from the captured input.

Keep echoCancellation: true for any call where audio plays through speakers. Disable it only when capturing a clean source that has no acoustic feedback path: line-in, a virtual audio device, or music you control. AEC assumes a voice-shaped echo and can damage sustained musical tones.

noiseSuppression

Noise suppression attenuates steady background sound: fan hum, traffic, keyboard clatter. It gates and filters anything that does not look like speech.

This helps speech and hurts music. A noise suppressor trained on voice treats a held cello note or a cymbal wash as noise and ducks it. For a music or instrument stream, set noiseSuppression: false.

autoGainControl

Automatic gain control (AGC) keeps perceived loudness steady. It raises the level when you lean back, lowers it when you lean in. For conversation this is what you want. Nobody has to ride a fader.

AGC fights you when input level carries meaning. Recording an instrument, AGC pumps quiet passages up and squashes loud ones, flattening dynamics. Disable it for music.

A summary of when to disable

// A music-grade capture: raw signal, no voice-tuned processing.
const stream = await navigator.mediaDevices.getUserMedia({
  audio: {
    echoCancellation: false,
    noiseSuppression: false,
    autoGainControl: false,
    sampleRate: 48000,
    channelCount: 2,
  },
});

Constraints are requests, not guarantees. Check what you actually got with getSettings().

const [track] = stream.getAudioTracks();
console.log(track.getSettings());
// { echoCancellation: false, channelCount: 2, sampleRate: 48000, ... }

A device may refuse stereo or ignore a sample-rate request. The settings object is the truth; the constraints object is the wish.

Opus: the codec that carries the audio

Once a track is added to an RTCPeerConnection, the browser encodes it with Opus. Opus is the mandatory-to-implement audio codec for WebRTC, so both peers always agree on it without negotiation tricks. You rarely configure it directly, but knowing its behavior explains audio quality on a call.

What Opus does well

Opus covers a wide range in one codec. It handles narrowband speech at low bitrates and full-band music at high ones, switching internally between a speech-tuned mode and a music-tuned mode. It runs at 48 kHz, frames at low latency (typically 20 ms), and degrades gracefully as bandwidth drops.

Bitrate

Bitrate is the main quality lever. The browser picks a target from bandwidth estimation, but the practical ranges are worth knowing.

For a voice call, more than ~40 kbps mono buys little. For a music stream, you want stereo and a higher ceiling.

Mono vs stereo

Voice is mono. Sending stereo voice doubles the bitrate for no benefit, since a single mic is one channel anyway. Music with real stereo content justifies two channels. Stereo is negotiated in SDP through the sprop-stereo and stereo parameters on the Opus payload.

DTX: discontinuous transmission

DTX stops sending packets during silence. When you are not talking, Opus sends occasional tiny comfort-noise updates instead of full frames. This cuts bandwidth in a conversation, where one side is usually quiet. The cost is that hard silence can sound slightly artificial; for music, where there is rarely true silence, DTX gives little and is often left off.

FEC: forward error correction

In-band FEC packs a low-bitrate copy of the previous frame into the current packet. If one packet is lost, the decoder reconstructs the missing audio from the redundant copy in the next packet. This trades a little extra bitrate for resilience to single-packet loss, which is the common case on a lossy link. For voice over an unreliable network, FEC noticeably reduces dropout artifacts.

Tuning Opus from SDP

The browser exposes no clean API for these knobs, so the practical route is munging the SDP fmtp line for the Opus payload before setLocalDescription. Do this carefully: malformed SDP breaks the connection.

// Bias Opus toward resilient stereo music on the offer.
function tuneOpus(sdp) {
  return sdp.replace(
    /a=fmtp:(\d+) ([^\r\n]*)/g,
    (line, pt, params) => {
      // Only touch the Opus payload (it advertises useinbandfec capability).
      if (!/useinbandfec/.test(params)) return line;
      return `a=fmtp:${pt} ${params};stereo=1;sprop-stereo=1;maxaveragebitrate=128000;useinbandfec=1`;
    },
  );
}

const offer = await pc.createOffer();
offer.sdp = tuneOpus(offer.sdp);
await pc.setLocalDescription(offer);

The modern alternative avoids SDP surgery: set encoding parameters on the sender.

const sender = pc.getSenders().find((s) => s.track?.kind === 'audio');
const params = sender.getParameters();
params.encodings[0].maxBitrate = 128_000;
await sender.setParameters(params);

setParameters is the supported path for bitrate. SDP munging remains the only route for stereo and FEC flags. Prefer setParameters where it covers your need.

Frame size and latency

Opus encodes in frames. The default WebRTC frame is 20 ms of audio, which balances latency against packetization overhead. Smaller frames cut latency but add per-packet header cost; larger frames are more efficient but feel less responsive in conversation. The browser picks a sensible frame size, and you rarely change it. What matters in practice: end-to-end mouth-to-ear latency is the sum of capture buffering, the encode frame, network transit, the jitter buffer at the receiver, and decode. The jitter buffer is usually the largest tunable contributor, and the browser sizes it adaptively from observed jitter. A laggy call is more often a swollen jitter buffer reacting to a bursty network than a codec problem.

Sample rate

Opus internally runs at 48 kHz and resamples anything else to it. Capturing at 48 kHz avoids a resample step. Requesting an exotic rate gains nothing, since the encoder normalizes regardless. The one rate that matters to your code is the AudioContext rate: if it differs from the capture rate, the graph resamples on the way in, which is fine but worth knowing when sample counts do not line up.

How Opus rides SRTP

Opus produces encoded frames. WebRTC packetizes those frames into RTP, attaching a sequence number and timestamp so the receiver can reorder and time playback. It then encrypts every packet with SRTP using keys derived from the DTLS handshake that secured the connection. The far end's RTCPeerConnection decrypts, depacketizes into a jitter buffer, and decodes back to samples for playback. None of this is something you call directly (adding the track is enough), but it means the audio is encrypted end to end between the two peers, and no intermediary can read it. Because the data path is peer-to-peer, the only machines that ever hold the decryption keys are the two endpoints.

The Web Audio API: measure and modify

The Web Audio API processes audio in a directed graph of nodes inside an AudioContext. A source node produces samples, processing nodes transform them, and a destination node consumes them. For calls, the useful nodes are a source built from your mic track, an analyser for metering, a gain node for volume, and a stream destination that gives you back a processed track to send.

AudioContext

The AudioContext is the graph's clock and engine. Create one per page; do not create one per call. It starts suspended until a user gesture resumes it, the same autoplay policy that blocks audio playback (covered below).

const ctx = new AudioContext();
// Resume inside a click handler if it starts suspended.
if (ctx.state === 'suspended') await ctx.resume();

MediaStreamAudioSourceNode

This node turns a captured MediaStream into a graph source. It reads the live mic track and feeds samples downstream.

const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
const source = ctx.createMediaStreamSource(stream);

A source node can fan out to several downstream nodes. That lets you tap the same mic into both a meter and the outgoing path without copying the track.

AnalyserNode: level meters and waveforms

AnalyserNode exposes the signal for visualization without altering it. It is a pass-through: connect it inline or as a side tap. It offers two views of the same data: a time-domain buffer for waveforms and level, and a frequency-domain buffer (FFT) for spectrum bars.

const analyser = ctx.createAnalyser();
analyser.fftSize = 2048;          // -> 1024 frequency bins
source.connect(analyser);          // side tap; does not consume the signal

A level meter reads the time-domain data and computes RMS, the root mean square of the samples, which tracks perceived loudness better than a single peak.

const buf = new Float32Array(analyser.fftSize);

function readLevel() {
  analyser.getFloatTimeDomainData(buf);
  let sum = 0;
  for (let i = 0; i < buf.length; i++) sum += buf[i] * buf[i];
  const rms = Math.sqrt(sum / buf.length); // 0 .. ~1
  return rms;
}

function loop() {
  const level = readLevel();
  meterEl.style.transform = `scaleX(${Math.min(1, level * 3)})`;
  requestAnimationFrame(loop);
}
loop();

For a frequency display, read the byte FFT data and draw bars.

const bins = new Uint8Array(analyser.frequencyBinCount); // fftSize / 2

function drawSpectrum() {
  analyser.getByteFrequencyData(bins); // 0 .. 255 per bin
  for (let i = 0; i < bins.length; i++) {
    const h = (bins[i] / 255) * canvas.height;
    canvasCtx.fillRect(i * 2, canvas.height - h, 1, h);
  }
  requestAnimationFrame(drawSpectrum);
}
drawSpectrum();

The live mic visualizer is the Audio Visualizer demo.

GainNode: volume

GainNode multiplies the signal by a scalar. Gain 1 is unchanged, 0 is silence, above 1 is amplification (with clipping risk). Unlike an HTML element's volume, a gain node sits in the graph, so it affects the track you send, not just local playback.

const gain = ctx.createGain();
gain.gain.value = 0.8;             // attenuate to 80%
source.connect(gain);

Set gain through its AudioParam to ramp smoothly and avoid clicks. A hard jump in gain produces an audible pop.

const now = ctx.currentTime;
gain.gain.cancelScheduledValues(now);
gain.gain.setValueAtTime(gain.gain.value, now);
gain.gain.linearRampToValueAtTime(0, now + 0.02); // 20 ms fade to silence

MediaStreamAudioDestinationNode: back to a track

A processing graph is useful for a call only if you can get a MediaStreamTrack back out of it. MediaStreamAudioDestinationNode is the bridge. Whatever you connect into it becomes a live track you can add to the peer connection.

const dest = ctx.createMediaStreamDestination();
gain.connect(dest);

const processedTrack = dest.stream.getAudioTracks()[0];

That processedTrack is the gain-staged signal. Send it instead of the raw mic track. If a sender already exists, swap it without renegotiating.

// Build the graph: mic -> gain -> destination
const source = ctx.createMediaStreamSource(micStream);
const gain = ctx.createGain();
gain.gain.value = 0.8;
const dest = ctx.createMediaStreamDestination();
source.connect(gain).connect(dest);

const processedTrack = dest.stream.getAudioTracks()[0];

// First time: add it to the connection.
pc.addTrack(processedTrack, dest.stream);

// Later, to change the source without renegotiation:
const sender = pc.getSenders().find((s) => s.track?.kind === 'audio');
await sender.replaceTrack(processedTrack);

Mixing multiple sources into one track

A call sends one outgoing audio track per sender. To combine sources (your mic plus a music bed, or two microphones), mix them in the graph and send the single mixed track. The Web Audio engine sums every node connected to the same downstream node.

Connect each source to a shared destination. The sum is the mix.

const ctx = new AudioContext();
const dest = ctx.createMediaStreamDestination();

// Source A: the microphone.
const micStream = await navigator.mediaDevices.getUserMedia({ audio: true });
const micSource = ctx.createMediaStreamSource(micStream);
const micGain = ctx.createGain();
micGain.gain.value = 1.0;
micSource.connect(micGain).connect(dest);

// Source B: a music file or another stream.
const music = new Audio('/track.mp3');
music.crossOrigin = 'anonymous';
const musicSource = ctx.createMediaElementSource(music);
const musicGain = ctx.createGain();
musicGain.gain.value = 0.4;          // duck the bed under the voice
musicSource.connect(musicGain).connect(dest);
music.play();

// The single mixed track to send P2P.
const mixedTrack = dest.stream.getAudioTracks()[0];
pc.addTrack(mixedTrack, dest.stream);

Each source keeps its own gain node, so you can balance the mix live: duck the music while talking, then bring it back up. The mixer in the Track Mixer demo combines two sources this way.

Watch the headroom when summing

Summing signals adds their amplitudes. Two sources near full scale can sum past 1.0 and clip. The gain nodes on each branch are your headroom control: pull each source down so the sum stays within range. A common starting point is to attenuate each of N sources by roughly 1/N, then bring the important one back up. A spectrum or peak meter tapped off the destination tells you whether the bus is clipping before the far end hears it.

// Tap a meter off the mix bus to watch the summed level.
const busAnalyser = ctx.createAnalyser();
dest.stream && micGain.connect(busAnalyser); // same node feeding dest
musicGain.connect(busAnalyser);

Automating a duck

Ducking (lowering the music when someone speaks) is a gain ramp driven by the voice level. Read the mic analyser, and when it crosses a threshold, ramp the music gain down; when it falls quiet, ramp it back. Use AudioParam ramps so the level moves smoothly instead of stepping.

function duck(voiceRms) {
  const now = ctx.currentTime;
  const target = voiceRms > 0.05 ? 0.15 : 0.4; // quiet vs full bed
  musicGain.gain.cancelScheduledValues(now);
  musicGain.gain.setValueAtTime(musicGain.gain.value, now);
  musicGain.gain.linearRampToValueAtTime(target, now + 0.15);
}

A note on createMediaElementSource: once an <audio> element is routed into the graph, the graph owns its output. The element no longer plays through the default speakers on its own; you hear it only where the graph sends it. To monitor the mix locally, connect the destination's stream to a muted-to-others element, or connect a branch to ctx.destination.

Muting correctly

There are three ways to silence outgoing audio. They are not equivalent. Picking the wrong one causes either wasted bandwidth, audible pops, or a renegotiation.

track.enabled = false: the right default

Setting enabled = false is the standard mute. The track stays in the connection, but the browser sends silence. Combined with DTX, bandwidth drops to near zero. Unmuting is instant and needs no renegotiation. This is the mute button you want.

const [mic] = micStream.getAudioTracks();
mic.enabled = false; // mute
mic.enabled = true;  // unmute, instant

Gain to zero: when you need a fade

A gain node ramped to zero gives a smooth fade with no click, useful for a soft mute or a push-to-talk that should not pop. The cost: the encoder still runs on a (silent) signal, so you pay full bandwidth. Use it when fade quality matters more than bandwidth, or layer it over enabled: fade with gain, then flip enabled once silent.

Removing the track: only for leaving

removeTrack() or replaceTrack(null) takes the track off the sender entirely. This frees the slot but triggers renegotiation, which is slow and visible to the far end as the track ending. Reserve it for actually leaving the call, not for muting.

Monitoring input level

Show the user their own input level so they can tell whether the mic works and whether they are clipping. Build a meter off an AnalyserNode tapped from the source, exactly as above. Two refinements make it trustworthy.

Convert RMS to decibels for a perceptually even meter, and flag clipping when samples approach full scale.

const buf = new Float32Array(analyser.fftSize);

function meter() {
  analyser.getFloatTimeDomainData(buf);
  let sum = 0, peak = 0;
  for (let i = 0; i < buf.length; i++) {
    const s = buf[i];
    sum += s * s;
    peak = Math.max(peak, Math.abs(s));
  }
  const rms = Math.sqrt(sum / buf.length);
  const db = 20 * Math.log10(rms || 1e-8); // -inf .. 0 dBFS
  const clipping = peak > 0.99;

  bar.style.transform = `scaleY(${(db + 60) / 60})`; // map -60..0 dB
  bar.classList.toggle('is-clipping', clipping);
  requestAnimationFrame(meter);
}
meter();

A meter pinned at the bottom means a dead mic, the wrong device, or a track muted by the OS. A meter slamming the top with the clip flag means the input is too hot: lower the gain or disable AGC if it is over-driving.

Reading the meter at animation rate

Drive the meter from requestAnimationFrame, not a timer. The analyser holds the most recent block of samples and you sample it once per frame, which matches the display refresh and keeps the meter smooth without burning cycles. Reading faster than the screen refreshes wastes work; reading on a fixed setInterval drifts against the frame clock and looks jittery. One read per frame is the right cadence.

Decide what the meter measures. RMS tracks loudness and is what a user reads as "am I being heard." Peak tracks the instantaneous maximum and is what you watch for clipping. A serious meter shows both: a filled bar for RMS and a thin held line for recent peak. The peak line should decay slowly so a brief transient stays visible long enough to notice.

let peakHold = 0;

function meterFrame() {
  analyser.getFloatTimeDomainData(buf);
  let sum = 0, peak = 0;
  for (const s of buf) { sum += s * s; peak = Math.max(peak, Math.abs(s)); }
  const rms = Math.sqrt(sum / buf.length);

  peakHold = Math.max(peak, peakHold * 0.95); // slow decay
  rmsBar.style.transform = `scaleY(${Math.min(1, rms * 3)})`;
  peakLine.style.bottom = `${Math.min(100, peakHold * 100)}%`;
  requestAnimationFrame(meterFrame);
}
meterFrame();

Local echo and feedback

The fastest way to ruin a call is to play your own outgoing audio back through your speakers, or to play the remote audio loudly enough that your mic re-captures it.

Never monitor your own mic through speakers

Do not connect your mic source to ctx.destination while speakers are on. You will hear yourself with latency, and the loop can build into feedback. If a user wants to monitor their own input, require headphones, and even then keep it optional.

// DON'T: this routes your own mic to your speakers.
// source.connect(ctx.destination);

// Monitor only to a level meter, which produces no sound:
source.connect(analyser);

Keep echoCancellation on for the remote audio loop

The remote person's voice comes out of your speakers, your mic catches it, and you send it back. AEC is exactly the defense. Leave echoCancellation: true on the capture whenever audio plays through speakers. Disabling it because you are "capturing clean" is correct only when there is no acoustic path from speaker to mic.

Headphones break every feedback path

A headset removes the speaker-to-mic path entirely. With headphones, echo and feedback stop being possible, which is why the safest "disable processing for music" setups assume a headset or a direct line-in.

Autoplay policy and remote audio playback

Receiving audio works only if the browser lets it play. Modern autoplay policies block audible playback that no user gesture authorized. A remote audio track attached to an element that the user never interacted with stays silent, and often gives no error.

Attach the remote track and play on a gesture

Take the remote track from ontrack, attach it to an <audio> element, and ensure playback is unlocked by a user action.

const remoteAudio = document.querySelector('#remote');

pc.ontrack = (event) => {
  remoteAudio.srcObject = event.streams[0];
};

// The "Join" / "Start call" button the user clicked is the gesture.
joinButton.addEventListener('click', async () => {
  try {
    await remoteAudio.play();
  } catch (err) {
    // Blocked: surface an explicit "Tap to enable audio" control.
    showUnmutePrompt();
  }
});

The patterns that survive the policy

If you route remote audio through Web Audio (for a remote level meter or per-peer volume), the same gesture must resume the AudioContext. A suspended context produces no sound regardless of the element state.

Putting it together

A full outgoing audio path with capture, metering, gain, and a sent track:

const ctx = new AudioContext();

// 1. Capture, tuned for voice.
const micStream = await navigator.mediaDevices.getUserMedia({
  audio: { echoCancellation: true, noiseSuppression: true, autoGainControl: true },
});

// 2. Graph: source fans out to a meter tap and a gain-staged send path.
const source = ctx.createMediaStreamSource(micStream);

const analyser = ctx.createAnalyser();
analyser.fftSize = 2048;
source.connect(analyser);                 // side tap, no sound

const gain = ctx.createGain();
gain.gain.value = 1.0;
const dest = ctx.createMediaStreamDestination();
source.connect(gain).connect(dest);       // send path

// 3. Send the processed track P2P.
const processedTrack = dest.stream.getAudioTracks()[0];
pc.addTrack(processedTrack, dest.stream);

// 4. Mute = silence the track, cheap and instant.
muteBtn.addEventListener('click', () => {
  processedTrack.enabled = !processedTrack.enabled;
});

// 5. Live input meter (RMS off the analyser tap).
const buf = new Float32Array(analyser.fftSize);
(function meter() {
  analyser.getFloatTimeDomainData(buf);
  let sum = 0;
  for (const s of buf) sum += s * s;
  meterEl.style.transform = `scaleX(${Math.min(1, Math.sqrt(sum / buf.length) * 3)})`;
  requestAnimationFrame(meter);
})();

Recap

• Three capture constraints shape audio before encoding: echoCancellation, noiseSuppression, autoGainControl. All default true: correct for speech, wrong for music.
• Opus is the codec. Bitrate is the main lever; mono for voice, stereo for music; DTX saves bandwidth in conversation; FEC adds loss resilience. Tune bitrate with sender.setParameters, stereo and FEC via SDP.
• Opus frames are packetized into RTP and encrypted with SRTP, end to end between peers. No sample transits a server.
• Web Audio measures and modifies: AudioContext hosts the graph, MediaStreamAudioSourceNode reads the mic, AnalyserNode meters it, GainNode controls volume, MediaStreamAudioDestinationNode hands back a track to send.
• Sum sources at a shared destination to mix several inputs into one outgoing track.
• Mute with track.enabled = false by default; ramp gain to zero for a fade; remove the track only when leaving.
• Never monitor your own mic through speakers; keep AEC on whenever audio plays through speakers; headphones break every feedback path.
• Remote audio plays only with a user gesture, and the AudioContext must be resumed on one too. Handle the blocked-play() case explicitly.

Going further

• Capture: acquiring the microphone and handling permissions.
• Video & Audio Tracks: track lifecycle, constraints, and replaceTrack.
• Audio Visualizer demo: a live level meter and spectrum from a captured mic.
• Track Mixer demo: combining two audio sources into one sent track.
• MediaStreamTrack reference: the track object, enabled, muted, and stop().

Troubleshooting