Multi-City VoIP Network Implementation

Background

In a previous project, I deployed a barebones PBX on Metaswitch Rhino TAS. While the Metaswitch platform itself was impressive, its bundled sample applications were rudimentary and intended only as source code examples. These sample applications lacked logs, metrics endpoints, and documentation. The key takeaway? Metaswitch Rhino TAS is a platform to build on, not anything flight-ready.

Overview

Using FreeSWITCH and Asterisk—both well-regarded platforms—I’ve designed a multi-city VoIP network connecting New York, Los Angeles, and Chicago. Each site combines an Asterisk PBX for local functionality with a dedicated FreeSWITCH-based SBC to interconnect the sites over SIP trunks.

My intention is to mimic a real-world setup, emphasizing modularity and control over call flow using Back-to-Back User Agent (B2BUA) principles. Although this system operates in a closed environment without PSTN integration, the SBC lays a solid foundation for external connectivity if needed. This setup demonstrates industry-relevant skills in telephony, systems administration, and network design.

Focus

This project isn’t about presenting a polished, Solutions Engineer-level blueprint, nor is it an attempt to teach industry professionals about their field. It’s about me devising and building an environment with enough complexity to showcase my hands-on approach to solving real-world systems administration challenges—and to prove that I’m someone you can trust to run your systems.

In my earlier project, I set up a barebones PBX on a single network segment—simple and straightforward. Now, I’m diving into something considerably more challenging. This project is designed to test and demonstrate my skills, showcasing my commitment to craft and determination to succeed.

Platform

Commercial SBC solutions are designed for enterprise environments, focusing on reliability, performance, and predictable features. Open-source projects, on the other hand, often aim to cover diverse telephony use cases rather than specializing exclusively in SBC functionality.

In my search for an open-source SBC, I found projects with SBC capabilities that varied in focus and maturity. After careful consideration, I chose FreeSWITCH for its powerful feature set, active community, and robust developer support. It exceeded my requirements, offering a dependable and comprehensive solution without compromise.

For PBX functionality, Asterisk was the natural choice. Its proven reliability allowed me to separate PBX and SBC roles effectively, ensuring a straightforward and dependable solution for local telephony.

Layout

Local Infrastructure

Location	Network	SBC IP	PBX IP
NY	192.168.254.0/24	192.168.254.221	192.168.254.222
CHI	192.168.253.0/24	192.168.253.221	192.168.253.222
LA	192.168.252.0/24	192.168.252.221	192.168.252.222

Dedicated Circuits

SIP Trunk	Trunk ID	Network	NY-SBC IP	LA-SBC IP	CHI-SBC IP
NY ↔ LA	Trunk 10	10.10.10.0/30	10.10.10.1	10.10.10.2
NY ↔ CHI	Trunk 20	10.10.20.0/30	10.10.20.1		10.10.20.2
LA ↔ CHI	Trunk 30	10.10.30.0/30		10.10.30.1	10.10.30.2

In case of a SIP trunk failure, calls are automatically rerouted through the alternate city.

Trunk Attachment

Determining whether to route the SIP trunks through the internal network via a firewall and NAT, or directly attach them to the SBC using a multi-homing strategy requires careful consideration of security, performance, network architecture, management complexity, and troubleshooting efficiency.

The Standard Approach: Routed Network with NAT

In the standard configuration, the SIP trunk is routed through the corporate network, typically via a firewall or router, with NAT applied. In this setup, the provider’s SIP traffic is directed to an external IP address on the firewall’s interface, which may be a private IP address if using a dedicated circuit (such as a point-to-point connection). The firewall then performs NAT to forward this traffic to the SBC’s internal IP address. Similarly, for outbound SIP traffic, the SBC uses the internal network to reach the provider, with the firewall translating the source IP to its external IP (again, typically a private IP in the case of a dedicated circuit), allowing the response to be correctly routed back.

This method is widely used because it acknowledges the firewall’s traditional role in perimeter security, effectively isolating the SBC from direct exposure to the external SIP trunk network. It also simplifies management by leveraging existing network infrastructure and firewall policies already in place for other services. The primary benefit is that the firewall handles both inbound and outbound traffic, ensuring security, address translation, and routing. The SBC does not need to sit on multiple networks and communicates to and from its peers via a single IP address.

However, deployment and troubleshooting can be delayed when different teams need to coordinate, especially when vendor support is required.

The Multi-Homing Approach: Direct Attachment to the SBC

Alternatively, the multi-homing strategy involves directly attaching the SIP trunk to the SBC, effectively treating the SBC as the gateway to the external network. This approach consolidates control within the SBC, positioning it as the central component for all SIP traffic, with fewer dependencies on external routers or firewalls for SIP management.

The appeal of multi-homing lies in the potential for simplified operations. By routing SIP traffic directly through the SBC, you centralize control, ensuring all troubleshooting and management can be done from a single system. This eliminates the need to coordinate with other network devices and reduces the complexity of dealing with multiple points of failure. Moreover, because the SBC handles the routing internally, it can provide a more streamlined and direct path for SIP communication, enhancing performance by reducing intermediary network hops.

However, this approach is not without its complexities. The SBC now takes on responsibilities traditionally handled by routers and firewalls. While this offers full control, it also increases the configuration complexity, requiring deeper engagement with FreeSWITCH to ensure proper setup. Additionally, this approach demands a higher level of attention to the SBC’s security and performance, as it becomes directly connected to the external network.

Decision: Streamlined Operations through Multi-Homing

While the multi-homing strategy requires more effort initially, it ultimately streamlines operations, providing a more efficient operation. This approach simplifies management by consolidating control into the SBC, where all key configurations can be handled from a single point.

With this setup, only one skill—managing the SBC—needs to be mastered. There’s no need to hand off tasks between teams or bounce between the SBC, firewall, router, and other devices. Deployments, moves, adds, and changes can be done directly within the SBC, eliminating delays and inefficiencies associated with multiple handoffs. Troubleshooting is also simplified, as issues can be diagnosed and resolved from a single system, reducing complexity and speeding up resolution times by minimizing the need for cross-team coordination.

Key considerations for adopting the multi-homing strategy include:

Consolidated Operations:
- Centralized control within the SBC.
- Streamlined management and troubleshooting into a single system.
- Simplified deployments, moves, adds, and changes within the SBC, reducing delays and inefficiencies.
Coordination Delays:
- Eliminated the need for handoffs between different teams or devices, minimizing potential deployment and troubleshooting delays.
- Reduced complexity by having a single point of control and responsibility.
Finger-Pointing:
- Reduced the likelihood of finger-pointing and evasive responses during troubleshooting.
- Simplified issue resolution with a clear, single point of focus, speeding up response times and minimizing miscommunication.

In summary, while the multi-homing strategy requires more effort initially, it brings significant long-term benefits by consolidating operations, minimizing coordination delays, and reducing the potential for finger-pointing. This results in a more efficient and streamlined operation, providing a cohesive VoIP infrastructure with minimal external dependencies.

Network bindings

root@NY-SBC:~# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether bc:24:11:e6:67:38 brd ff:ff:ff:ff:ff:ff
    altname enp0s18
    inet 192.168.254.221/24 brd 192.168.254.255 scope global ens18
       valid_lft forever preferred_lft forever
    inet6 fe80::be24:11ff:fee6:6738/64 scope link
       valid_lft forever preferred_lft forever
3: ens19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether bc:24:11:03:1f:36 brd ff:ff:ff:ff:ff:ff
    altname enp0s19
    inet 10.10.10.1/30 brd 10.10.10.3 scope global ens19
       valid_lft forever preferred_lft forever
    inet6 fe80::be24:11ff:fe03:1f36/64 scope link
       valid_lft forever preferred_lft forever
4: ens20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether bc:24:11:42:e4:ad brd ff:ff:ff:ff:ff:ff
    altname enp0s20
    inet 10.10.20.1/30 brd 10.10.20.3 scope global ens20
       valid_lft forever preferred_lft forever
    inet6 fe80::be24:11ff:fe42:e4ad/64 scope link
       valid_lft forever preferred_lft forever
root@NY-SBC:~#

SIP Profiles

freeswitch@NY-SBC> sofia status
                     Name          Type                                       Data      State
=================================================================================================
            external-ipv6       profile                   sip:mod_sofia@[::1]:5080      RUNNING (0)
                  trunk20       profile              sip:[email protected]:5060      RUNNING (0)
         trunk20::chi-sbc       gateway                      sip:[email protected]      REGED
                 external       profile           sip:[email protected]:5080      RUNNING (0)
    external::example.com       gateway                    sip:[email protected]      NOREG
       lab4.decoursey.com         alias                                    trunk10      ALIASED
            internal-ipv6       profile                   sip:mod_sofia@[::1]:5060      RUNNING (0)
                  trunk10       profile              sip:[email protected]:5060      RUNNING (0)
          trunk10::la-sbc       gateway                      sip:[email protected]      REGED
                 internal       profile         sip:[email protected]:5060      RUNNING (0)
=================================================================================================
6 profiles 1 alias

freeswitch@NY-SBC>

FreeSWITCH Multi-Homed Configuration

My initial approach to FreeSWITCH’s multi-homed setup involved configuring the NICs for the internal and SIP trunk networks, and then modifying the default FreeSWITCH configuration to define the required endpoints and parameters. However, this strategy didn’t work as expected.

There are several places in the default configuration where items like source IP and signaling IP addresses are hand-configured. In a multi-homed scenario, what’s correct in one context can be wrong in another. Therefore, simply editing the default configuration files (SIP profiles) isn’t sufficient.

The breakthrough came in defining distinct SIP profiles for each network segment and ensuring that remote gateways are affiliated with the correct network they are reachable through.

SIP Profiles

In FreeSWITCH, SIP profiles define how FreeSWITCH communicates with devices on specific network segments. A SIP profile is tied to a network interface (or IP address) and dictates how SIP traffic is handled on that segment. For example:

Internal Profile: Handles communication with internal devices or PBXs, usually bound to a private IP address on the local network.
SIP Trunk Profiles: Each SIP trunk network requires its own unique SIP profile, which directly binds to the IP assigned to that SIP trunk network. This profile is aligned with the network facing the specific SIP trunk and ensures that the traffic is routed correctly.

Gateway Definitions

Gateway Definitions in FreeSWITCH specify how the system connects to other SIP endpoints, such as remote PBXs, other SBCs, or SIP providers. Each gateway needs to be associated with a SIP profile, and this relationship ensures traffic is routed through the correct network interface. For example, a gateway might be defined to point to a remote SBC, specifying the SBC’s IP address or hostname, the necessary authentication credentials, and the associated SIP profile.

The Key Relationship: SIP Profiles and Gateways

The most important aspect of working with multiple network segments in FreeSWITCH is the relationship between SIP profiles and Gateway Definitions. Each gateway must be associated with the appropriate SIP profile that defines the network interface it should use. This is crucial because SIP traffic must be routed through the correct network interface—whether it’s the internal network or a dedicated SIP trunk network.

If you’re connecting to a remote SIP trunk over a secondary NIC, you would:

Create a SIP Trunk Profile bound to the IP of the secondary NIC.
Define a gateway within that profile, pointing to the remote endpoint (e.g., the remote SBC or SIP provider).
Ensure the gateway’s definition matches the SIP Trunk Profile’s network interface to correctly route the traffic.

Conclusion

In a multi-homed environment, especially when facing both internal and SIP trunk networks, FreeSWITCH must be explicitly configured to handle each network segment. This involves creating a unique SIP profile for each network interface and ensuring that each Gateway Definition points to the appropriate profile.

Dialplan

The FreeSWITCH dialplans are organized around where the traffic is presenting from.

Inbound arrivals

The below NY-SBC configuration is for handling calls arriving via Trunk 10, the circuit from LA. In general, it’s expected this should be calls destined for our NY-PBX extensions, and the plan is to hand these to NY-PBX. However, these might also be CHI-PBX destined calls that were failed over, and in that eventuality, we route those calls to CHI-SBC.

root@NY-SBC:/etc/freeswitch/dialplan# cat trunk10.xml
<include>
  <context name="trunk10">

    <!-- Safeguard against SIP loops -->
    <extension name="unloop">
      <condition field="${unroll_loops}" expression="^true$"/>
      <condition field="${sip_looped_call}" expression="^true$">
        <action application="deflect" data="${destination_number}"/>
      </condition>
    </extension>

    <!-- Route calls to the PBX -->
    <extension name="route-to-pbx">
      <condition field="destination_number" expression="^254\d{2}$">
        <action application="bridge" data="sofia/internal/${destination_number}@192.168.254.222"/>
      </condition>
    </extension>

    <!-- Failover routing in case of primary trunk failure. -->
    <extension name="route-to-chi">
      <condition field="destination_number" expression="^(253\d{2})$">
        <action application="bridge" data="sofia/gateway/chi-sbc/$1"/>
      </condition>
    </extension>

    <!-- Handle unmatched calls gracefully -->
    <extension name="unmatched-calls">
      <condition field="destination_number" expression=".*">
        <action application="hangup" data="UNALLOCATED_NUMBER"/>
      </condition>
    </extension>

  </context>
</include>
root@NY-SBC:/etc/freeswitch/dialplan#

Outbound dials

For outbound dials I ended up calling out to a Lua script.

root@NY-SBC:/etc/freeswitch/dialplan# cat public/01_route_to_la.xml
<extension name="route-to-la">
    <condition field="destination_number" expression="^252[0-9][0-9]$">
        <action application="lua" data="route_to_la.lua"/>
    </condition>
</extension>

root@NY-SBC:/etc/freeswitch/dialplan#

This is that script.

root@NY-SBC:/etc/freeswitch/dialplan# cat /usr/share/freeswitch/scripts/route_to_la.lua
-- Initialize the FreeSWITCH API object
local api = freeswitch.API()

-- Function to check gateway status
local function check_gateway_status(gateway_name)
    local status = api:execute("sofia", "status gateway " .. gateway_name)
    if string.match(status, "Status%s+UP") then
        return "UP"
    else
        return "DOWN"
    end
end

-- Retrieve the destination number from the session
local destination_number = session:getVariable("destination_number")

if not destination_number then
    freeswitch.consoleLog("ERROR", "Destination number is nil. Unable to proceed.\n")
    session:hangup("NORMAL_TEMPORARY_FAILURE")
    return
end

-- Define the primary and secondary gateways
local primary_gateway = { name = "la-sbc", dialstring = "sofia/gateway/la-sbc/" .. destination_number }
local secondary_gateway = { name = "chi-sbc", dialstring = "sofia/gateway/chi-sbc/" .. destination_number }

-- Check the status of the primary gateway
local primary_status = check_gateway_status(primary_gateway.name)
freeswitch.consoleLog("INFO", "Primary gateway '" .. primary_gateway.name .. "' status: " .. primary_status .. "\n")

if primary_status == "UP" then
    -- Route the call through the primary gateway
    freeswitch.consoleLog("INFO", "Routing through primary gateway: " .. primary_gateway.name .. "\n")
    session:execute("bridge", primary_gateway.dialstring)
else
    -- If primary is down, check the secondary gateway
    local secondary_status = check_gateway_status(secondary_gateway.name)
    freeswitch.consoleLog("INFO", "Secondary gateway '" .. secondary_gateway.name .. "' status: " .. secondary_status .. "\n")

    if secondary_status == "UP" then
        -- Route the call through the secondary gateway
        freeswitch.consoleLog("INFO", "Routing through secondary gateway: " .. secondary_gateway.name .. "\n")
        session:execute("bridge", secondary_gateway.dialstring)
    else
        -- Neither gateway is up; try the primary and be done with it
        freeswitch.consoleLog("WARNING", "Both gateways are down. Attempting primary as a last resort.\n")
        session:execute("bridge", primary_gateway.dialstring)
    end
end

-- If no gateways succeed, hang up
if session:getVariable("originate_disposition") ~= "SUCCESS" then
    freeswitch.consoleLog("ERROR", "All attempts failed. Hanging up.\n")
    session:hangup("NORMAL_TEMPORARY_FAILURE")
end
root@NY-SBC:/etc/freeswitch/dialplan#

Failover to Backup SIP Trunks

During the PoC, automatic backup failover was right at my fingertips when setting up the SIP trunks, so it got pulled in. While it was an easy win, any push to production will start with a discussion of likely failure points, and the actual first-pass redundancy design may focus elsewhere.

Each SBC is responsible for routing calls to a remote city via its SIP trunk. However, if the trunk is down, should we route the call through another city?

The challenge here is that we can’t know for certain whether this secondary routing will help. It ultimately depends on the root cause of the problem—whether it’s a network issue affecting the entire site or just a specific SIP trunk that’s down.

Guiding Principles and Analysis

Failover engineering has some guiding principles, with one of the most important being caution around blindly failing over to another site. That other site may not be in any better posture to handle the traffic, and routing requests without understanding the underlying issue can lead to further complications. The nuances and gotchas in failover design are varied, but the general takeaway is to approach such decisions thoughtfully and with an understanding of potential risks.

After analysis, SIP’s end-to-end setup design offers an advantage in this case. If the call can’t be completed all the way to the callee, regardless of how many hops are involved, the failure will propagate all the way back to the original bridging attempt. It’s not as if the call would be accepted at an intermediary site, only to be mishandled there.

FreeSWITCH Failover Implementation

FreeSWITCH’s documentation provides a straightforward approach for failover, using the | separator to attempt a call on a secondary trunk if the primary one fails:

<action application="bridge" data="sofia/gateway/primary/dialstring|sofia/gateway/secondary/dialstring"/>

During testing, this mechanism works well when the primary trunk is down. However, if the primary trunk is active and the callee rejects the call, FreeSWITCH’s failover mechanism attempts to bridge the call again, causing an unwanted re-ring for the party who just rejected it. I tried using scripting constructs to address this by evaluating the failure reason, but faced a race condition where I couldn’t get the failure reason in time to make an informed decision. While I haven’t ruled out a workable solution with developers or configuration experts, I have shifted focus for now.

Monitoring and Fail-Open Strategy

I was committed to implementing some form of failover, even if not perfect. Fortunately, my systems constantly monitor trunk status via SIP OPTIONS—I have the pings configured to occur every 15 seconds, providing real-time status feedback. Using this data, I can check via API call whether a trunk is up or down before attempting to route the call through it and then prioritize relay attempts likely to succeed. I even have some logic to try to “fail open” in the event of a status lookup failure, in which case the call is attempted down the primary route.

Conclusion: Effective Failover Mechanism

While this approach isn’t perfect, it’s reliable in all scenarios I’ve tested and provides an effective failover mechanism that avoids unnecessary retries or misrouting. This solution ensures that calls are handled efficiently, even in the event of a trunk outage, without introducing any significant delays or side effects.

SolarWinds Integration Progress Update

The integration of SNMP monitoring into the SBCs is an ongoing effort, and significant progress has been made. While this feature isn’t yet complete, the groundwork laid so far demonstrates a clear path forward. Here’s what has been accomplished:

Net-SNMP Integration

Successfully installed and configured the Linux Net-SNMP daemon on the SBCs.
Integrated FreeSWITCH’s built-in metrics using the AgentX protocol, enabling initial SNMP data collection.

SolarWinds Universal Device Pollers (UnDP)

Verified integration of FreeSWITCH metrics into SolarWinds through the Universal Device Poller feature.
Ensured stock metrics are now visible and trackable within the SolarWinds dashboard.

Dynamic SIP Trunk Discovery

Designed a strategy to dynamically discover SIP trunks defined on an SBC using table-based SNMP lookups.
This approach automates the addition of SIP trunk data into SolarWinds, eliminating the need for manual definitions.

Command Line and Shell Scripting

Extensively utilized the fs_cli command-line interface for FreeSWITCH to explore available metrics.
Developed and tested shell scripts using Net-SNMP’s “extend” and “pass” mechanisms to integrate these metrics into SolarWinds.

This progress marks a strong foundation for a fully functional monitoring solution. The project remains active, and I anticipate delivering a further update by this Friday. The next steps include finalizing the dynamic discovery mechanism and refining the data presentation in SolarWinds.

FreeSWITCH Logging

As a systems administrator, understanding and managing FreeSWITCH logging is crucial for maintaining system health and troubleshooting issues efficiently. This guide aims to provide a detailed overview of FreeSWITCH logging, including log file locations, rotation policies, retention periods, log details, and verbosity adjustments.

Out-of-the-Box Logging

FreeSWITCH logs a variety of information by default, including system events, errors, and call handling details. The primary log file is /var/log/freeswitch/freeswitch.log.

Configuring Log Storage, Rotation, Retention, and Verbosity

Log storage location, rotation, retention, and verbosity are all configured within the /etc/freeswitch/autoload_configs/logfile.conf.xml file. Here’s how you can adjust these settings to meet your needs:

<configuration name="logfile.conf.xml">
  <settings>
    <!-- Log file location -->
    <param name="logfile" value="/var/log/freeswitch/freeswitch.log"/>

    <!-- Log rotation settings -->
    <param name="logrotate-size" value="104857600"/> <!-- Default: 100MB -->
    <param name="logrotate-count" value="10"/> <!-- Default: 10 files -->

    <!-- Log verbosity level -->
    <param name="loglevel" value="INFO"/> <!-- Default: INFO -->
  </settings>
  <mappings>
    <!-- Default log level: info. Remove the comment from the next line to switch to debug -->
    <map name="all" value="console,info,notice,warning,err,crit,alert"/>
    <!-- Uncomment the following line if you want debug-level logging -->
    <!-- <map name="all" value="console,debug,info,notice,warning,err,crit,alert"/> -->
  </mappings>
  <!-- Prefix all log lines by the session's uuid  -->
  <param name="uuid" value="true" />
</configuration>

Rotation and Retention: Adjust the configuration settings to meet your retention requirements. For example, you might increase the file size limit or the number of retained logs if necessary.

Verbosity: The default log verbosity in FreeSWITCH was observed to be DEBUG, which may be excessive for regular operation. For my deployment, I have dialed this back to INFO, which provides information suitable for normal operation. Higher verbosity levels, such as DEBUG, are generally used when actively troubleshooting detailed issues. Adjust the verbosity level to your preference.

Log Details for Call Handling

The freeswitch.log file contains detailed information about call handling, including timestamps, log levels, source file names, line numbers, function names, and log data. Here is an example of a sample log line:

12812bf9-98c8-4307-8fca-ca542bad93e6 2025-01-05 04:02:32.021541 99.33% [NOTICE] switch_channel.c:1142 New Channel sofia/internal/[email protected] [12812bf9-98c8-4307-8fca-ca542bad93e6]
12812bf9-98c8-4307-8fca-ca542bad93e6 2025-01-05 04:02:32.021541 99.33% [INFO] sofia.c:10460 sofia/internal/[email protected] receiving invite from 192.168.253.222:5060 version: 1.10.12 -release-10222002881-a88d069d6fgit a88d069 2024-08-02 21:02:27Z 64bit call-id: f5ec2cfe-6b6c-43b3-b162-f22216eb5219
12812bf9-98c8-4307-8fca-ca542bad93e6 2025-01-05 04:02:32.041536 99.33% [INFO] sofia.c:10460 sofia/internal/[email protected] receiving invite from 192.168.253.222:5060 version: 1.10.12 -release-10222002881-a88d069d6fgit a88d069 2024-08-02 21:02:27Z 64bit call-id: f5ec2cfe-6b6c-43b3-b162-f22216eb5219
12812bf9-98c8-4307-8fca-ca542bad93e6 2025-01-05 04:02:32.041536 99.33% [INFO] mod_dialplan_xml.c:639 Processing 25301 <25301>->25401 in context public
12812bf9-98c8-4307-8fca-ca542bad93e6 EXECUTE [depth=0] sofia/internal/[email protected] set(outside_call=true)
12812bf9-98c8-4307-8fca-ca542bad93e6 EXECUTE [depth=0] sofia/internal/[email protected] export(RFC2822_DATE=Sun, 05 Jan 2025 04:02:32 -0500)
12812bf9-98c8-4307-8fca-ca542bad93e6 EXECUTE [depth=0] sofia/internal/[email protected] lua(route_to_ny.lua)
2025-01-05 04:02:32.041536 99.33% [INFO] switch_cpp.cpp:1466 Primary gateway 'ny-sbc' status: UP
2025-01-05 04:02:32.041536 99.33% [INFO] switch_cpp.cpp:1466 Routing through primary gateway: ny-sbc
12812bf9-98c8-4307-8fca-ca542bad93e6 EXECUTE [depth=0] sofia/internal/[email protected] bridge(sofia/gateway/ny-sbc/25401)
91777dc4-8d81-492b-bae1-af495eea7a97 2025-01-05 04:02:32.041536 99.33% [NOTICE] switch_channel.c:1142 New Channel sofia/trunk20/25401 [91777dc4-8d81-492b-bae1-af495eea7a97]
91777dc4-8d81-492b-bae1-af495eea7a97 2025-01-05 04:02:32.041536 99.33% [INFO] sofia_glue.c:1659 sofia/trunk20/25401 sending invite call-id: (null)
2025-01-05 04:02:32.641541 99.33% [INFO] sofia.c:1348 sofia/trunk20/25401 Update Callee ID to "Outbound Call" <sip:[email protected]>
91777dc4-8d81-492b-bae1-af495eea7a97 2025-01-05 04:02:32.641541 99.33% [NOTICE] sofia.c:7604 Ring-Ready sofia/trunk20/25401!
12812bf9-98c8-4307-8fca-ca542bad93e6 2025-01-05 04:02:32.641541 99.33% [NOTICE] mod_sofia.c:2514 Ring-Ready sofia/internal/[email protected]!
12812bf9-98c8-4307-8fca-ca542bad93e6 2025-01-05 04:02:32.641541 99.33% [NOTICE] switch_ivr_originate.c:572 Ring Ready sofia/internal/[email protected]!
2025-01-05 04:02:41.441459 99.23% [INFO] sofia.c:1348 sofia/trunk20/25401 Update Callee ID to "Outbound Call" <sip:[email protected]>
91777dc4-8d81-492b-bae1-af495eea7a97 2025-01-05 04:02:41.441459 99.23% [NOTICE] sofia.c:8681 Channel [sofia/trunk20/25401] has been answered
12812bf9-98c8-4307-8fca-ca542bad93e6 2025-01-05 04:02:41.461440 99.23% [NOTICE] sofia_media.c:90 Pre-Answer sofia/internal/[email protected]!
12812bf9-98c8-4307-8fca-ca542bad93e6 2025-01-05 04:02:41.461440 99.23% [NOTICE] switch_ivr_originate.c:3855 Channel [sofia/internal/[email protected]] has been answered

Remote Syslog

FreeSWITCH supports remote syslog logging through the mod_syslog module. This feature allows you to send log messages to a remote syslog server, which is useful for centralized logging and monitoring.

Load the mod_syslog Module: If the mod_syslog module is not already loaded, you need to load it by adding it to the FreeSWITCH modules configuration file. Open the /etc/freeswitch/autoload_configs/modules.conf.xml file and add the following line:
```
<load module="mod_syslog"/>
```

Edit /etc/freeswitch/autoload_configs/switch.conf.xml:

Add the remote syslog configuration:

<extension name="remote_syslog" priority="1">
  <action application="log" loglevel="debug"/>
  <action application="syslog" loglevel="debug" server="syslog.example.com" port="514"/>
</extension>

It looks to be possible to condition the send based on call criteria if desired via advanced configuration syntax.

Quick Tips and Tricks

Grep by UUID: Use grep <uuid> /var/log/freeswitch/freeswitch.log to filter log entries by the session’s UUID. This is useful for isolating log details related to a specific call session, even with interleaved calls.
Grep by Value then UUID: First, grep for a specific value, such as a phone number, to identify the UUID of a call attempt. For example: grep "25301" /var/log/freeswitch/freeswitch.log. Then, use the UUID to grep for all related log entries: grep <uuid> /var/log/freeswitch/freeswitch.log.
Tail the Log File: Use tail -f /var/log/freeswitch/freeswitch.log to continuously monitor the log file in real-time. This is helpful for observing live events and troubleshooting as they happen.
Count Errors: Use grep -c ERROR /var/log/freeswitch/freeswitch.log.* to count the number of error entries in the log files. This can give you a quick overview of the system’s health and highlight recurring issues.

Final Thoughts on Logging

Effective logging practices are essential for proactive system administration. Here are some final tips to help you get the most out of FreeSWITCH logging:

Practice Pulling Log Details:
- Regularly pull log details for test calls and interpret the entries, even if there are no issues. This practice helps develop muscle memory around log file locations and interpretation, ensuring the skills are readily available when needed urgently.
Routinely Examine Log Files:
- Examine log files regularly for benign or unnoticed errors. Try to clean up these errors if possible. If not, be aware of exactly what these benign errors are, so they do not become misleading distractions during emergency troubleshooting.
Keep Debug Logging in Mind:
- Always consider enabling debug logging if the standard logs are not providing enough information. Attempt to replicate the issue or request a fresh example to be performed while debug logging is enabled. This approach can provide deeper insights and help pinpoint the problem.

By incorporating these practices into your regular system maintenance routine, you can ensure that you are well-prepared to handle any issues that arise and maintain a healthy, well-functioning system.

Using `fs_cli` for Troubleshooting FreeSWITCH

When troubleshooting FreeSWITCH, fs_cli is your go-to tool for real-time management and debugging. It connects directly to the core FreeSWITCH process and provides a fast, interactive way to manage logs, monitor SIP traffic, and check profile status.

Getting Started

To connect interactively:

fs_cli

For one-off commands (good for bash scripting):

fs_cli -x "command"

Adjusting Logs

Set the logging level to debug for detailed troubleshooting:

log level debug

Note: Log verbosity changes here are not persistent.

Reduce verbosity when done:

log level info

SIP Tracing

Enable SIP signaling traces for live traffic monitoring:

sofia global siptrace on

Disable it once you’re finished:

sofia global siptrace off

Checking Profile Status

View an overview of all SIP profiles:

sofia status

Dive into the details of a specific profile:

sofia status profile <profile_name>

Beyond the Essentials

fs_cli is packed with commands for deeper troubleshooting:

Show active calls and channels
List registered endpoints
Inspect call variables by UUID

Stick with logs, traces, and profiles for most scenarios, but know the tool can handle much more when needed.

Closing Thoughts

This project highlights my ability to build a practical, multi-city VoIP network that balances complexity with real-world relevance. From SIP profile design to failover logic, it demonstrates a hands-on approach to solving problems and creating systems that work.

While the monitoring component and full validation details aren’t finished, the foundation is solid, and progress is on track for an update by Friday. This reflects my focus on delivering meaningful results rather than rushing to check boxes.

I look forward to discussing the project further and appreciate the opportunity to showcase how I approach systems administration challenges with thoughtfulness and follow-through.

Background

Overview

Focus

Platform

Layout

Local Infrastructure

Dedicated Circuits

Trunk Attachment

The Standard Approach: Routed Network with NAT

The Multi-Homing Approach: Direct Attachment to the SBC

Decision: Streamlined Operations through Multi-Homing

Network bindings

SIP Profiles

FreeSWITCH Multi-Homed Configuration

SIP Profiles

Gateway Definitions

The Key Relationship: SIP Profiles and Gateways

Conclusion

Dialplan

Inbound arrivals

Outbound dials

Failover to Backup SIP Trunks

Guiding Principles and Analysis

FreeSWITCH Failover Implementation

Monitoring and Fail-Open Strategy

Conclusion: Effective Failover Mechanism

SolarWinds Integration Progress Update

Net-SNMP Integration

SolarWinds Universal Device Pollers (UnDP)

Dynamic SIP Trunk Discovery

Command Line and Shell Scripting

FreeSWITCH Logging

Out-of-the-Box Logging

Configuring Log Storage, Rotation, Retention, and Verbosity

Log Details for Call Handling

Remote Syslog

Quick Tips and Tricks

Final Thoughts on Logging

Using fs_cli for Troubleshooting FreeSWITCH

Getting Started

Adjusting Logs

SIP Tracing

Checking Profile Status

Beyond the Essentials

Closing Thoughts

Using `fs_cli` for Troubleshooting FreeSWITCH