Troubleshooting Playbook

Start every investigation with the Diagnostics panel and relevant logs. This section catalogs common failure patterns, how to isolate them, and recommended tools.

1. Registration / signaling

2. Media & quality

1. One-way / no audio:
   • Inspect NAT/port mappings between server and peers.
   • Ensure rtp_start_port / rtp_end_port ranges are open in config.toml and firewalls.
   • Reproduce via Diagnostics → Web Dialer or a handset, then capture RTP with tcpdump/sngrep to verify return packets.
   • Check the CDR’s media evidence: proxy.leg_media_incomplete marks legs that never delivered media; media_loss_pct/media_jitter_ms/media_rtt_ms show post-answer quality per trunk leg.
   • Behind EIP/NAT where direct paths fail, set the dialplan-level relay_only switch to force media through the proxy.
2. Noise or jitter:
   • Switch to lower bitrate codecs.
   • Enable the denoise models from fixtures/ or turn on echo cancellation at the endpoint.
   • Check QoS policies and link bandwidth.
3. Media stalls mid-call (0.5+): the media bridge detects stalled flows and attempts recovery; if stalls correlate with a specific carrier, combine the CDR media counters with SipFlow capture and consider ice_lite = true for strict full-ICE peers.

3. Routing & billing

• Routing ineffective: confirm Reload ran and validate config/routes syntax via tomlcheck or CI.
• Wrong route selected: Diagnostics → Routing Evaluate shows the hit rule/trunk; adjust priority or match filters accordingly. The CDR records the matched route id/name, so you can confirm exactly which rule served a past call.
• Transfer failures: 0.5 hardens REFER handling with B2BUA fallback and application handoff preservation. If a transfer fails, check the call record’s leg timeline for the transfer marker and whether the target rejected (409 = unanswered consult); enable transfer-header passthrough on the involved trunk when carriers require context headers.
• Billing mismatch: export CDRs from Call Records, compare billing templates, and look for no_rate alerts caused by missing prefixes.

4. Console / API

• Cannot log in: inspect the [console] config and DB connection; make sure browser time is accurate to avoid expired tokens.
• API returns 500: read logs/console (or stdout) stack traces; most errors stem from missing config or unfinished DB migrations.
• Diagnostics blank page: typically SIP server is down or the user lacks permission; validate /health reports ok and grant diagnostics access.

5. Performance & stability

• High CPU: use top/bt to locate hot threads, lower concurrency or scale out, and check for excessive transcoding. RTP proxying with SRTP encryption is the primary CPU consumer — estimate 5–8 concurrent calls per vCPU.
• Growing memory: verify recording buffer cleanup in callrecord/storage.rs. Long-running calls with large recording buffers (particularly in SipFlow capture mode) can accumulate memory. Restart nodes during maintenance windows if leak patterns are detected.
• Call lifecycle leaks: ensure call-lifecycle resources are released properly. In versions prior to 0.4.10, certain call teardown paths could retain session state. Upgrade to the latest release if you observe memory growth proportional to call volume.
• Crashes / restarts: consult journalctl or container logs—configuration syntax errors or unreachable dependencies (DB/Redis) are common causes. The binary auto-retries with exponential backoff (up to 10 attempts).
• Concurrent call limits: per-trunk max_calls and per-tenant max_concurrency are enforced independently. Exceeded limits return SIP 503. Check Diagnostics → Trunks for capacity utilization.

6. Incident workflow

1. Gather evidence: screenshots from Diagnostics, log exports, precise timestamps.
2. Roll back quickly: if caused by configuration, revert config/ in Git and reload.
3. Validate fix: place test calls and confirm CDRs/alerts return to normal.
4. Document: record root cause, impact, and remediation steps in the internal wiki for future reference.