Nguyen Vu Minh Tuan

Responsible:

• Fail-over solution for high availability for FreeSWITCH
  • Using corosync and pacemaker to run FreeSWITCH on a Virtual IP, when failure on the active FreeSWITCH happened, corosync and pacemaker will switch Virtual IP to standby FreeSWITCH server to minimize downtime, in some cases, FreeSWITCH can recover calls on the active server which failed
• Load-balance solution for scaling concurrent calls on FreeSWITCH
  • Coding OpenSIPs with module mid_registrar, load_balancer, and dispatcher to load-balancing SIP traffic with round robin or weighted order on 3 FreeSWITCH servers to reach 5000 extensions and 1200 concurrent calls
  • Load test with SIPP makes calls to this OpenSIPs - 3 FreeSWITCH server for multiple days
  • Documented all information about how to system works, load test historic data, how to read a log file, how to deploy
• Use ISO Debian live image to create a custom ISO image containing VoIP System which is documented to install completely offline without a connection to the internet
• Write 2 months plan and personally train interns fresh out of college, or university on SIP, RTP, VoIP Systems, and Linux so they can support all current customers and handle any running project, 3 people has become full-time employee with my guidance

• R&D new VoIP solution
  • Successfully using FreeSWITCH to call video on 2 endpoints using H264 protocol, also enable recordings of both video and audio with mod_av
• Successfully using FreeSWITCH to detect both voice and keypress in the same calls with mod_umimrcp on the FreeSWITCH server to detect voice and keypress to detect medical records for a foreign medical company
  • Write multiple lua files to detect 2 languages English and Spanish, multiple string voice or single character voice, multiple press key or single press key
• Successfully using RTPengine to enable recordings on the SBC side instead of the PBX
• Write lua code to detect events from FreeSWITCH server and setup that lua code in mod_lua to detect specific header in SIP message and process it with custom dialplan, for example, when receiving SIP message 480 with Reason header cause is 600 and text is voicemail this is a voicemail calls
• Research, apply, documented, training DevOps tools for VoIP Engineer, System Admin, 2 people have become proficient in coding Jenkins pipeline and Ansible playbook to handle my tasks
• Code Jenkins for CI and Ansible for deployment, migrate data, backup, restore, load test, and stress test, some of them can be found on my GitHub repositories and some of them even have a video explain how it works
  • Deploy service API Golang as back-end and NodeJS as front-end developed by the developer team
  • Reduce deployment time VoIP System from 4 hours to 1 hour and reduce human error ▶ Video
  • Centralize the iptables file to manage on the Git server instead of remembering which iptables is the latest on which server
  • Reduce repetitive tasks such as installing zabbix-agent on a VoIP system, logging in to the Zabbix server to add a host, and adding action ▶ Video
  • Migrate one or multiple tenants from a FusionPBX server to another, by generating a bash shell script to dump data of that tenant of 1 server, transfer and restore it on another server
  • Stress test and load test a FusionPBX server using SIPP, Jenkins, and Ansible will install sipp on another server, generating scenario based on Jenkins params and Ansible extra-vars and send SIP message on sipp server to test FusionPBX server
  • Generating script for backup or restore based on what Jenkins parameters the user chooses on Jenkins, Ansible will generate a bash shell script for each service and add a crontab to run daily at a specific time, that script will backup that service to Google Drive via rclone
• All VoIP Engineer, System Admin responsible

Responsible:

• Proactively manage service performance. Provide in-depth analysis of usage trends and make recommendations for modifications to ensure services have sufficient capacity and increase/decrease capacity as needed.
• Operate Zabbix-server as a monitoring server
  • Install zabbix-agent on servers to monitor whether an open port is closed, CPU, RAM, and disk space left
  • Write a custom Zabbix template for monitoring PBX services
  • Write a bash shell script to detect concurrent calls of the FreeSWITCH, Asterisk, detect online extensions of the FreeSWITCH, OpenSIPs, and add that script to zabbix-agent UserParameters
• Operate Homer as a SIP capture server
  • Install captagent and config SIP port range on captagent and have it send SIP message log to Homer server to debug calls
  • Integrate Homer to Grafana and Prometheus to show the number of SIP messages sent to Homer by multiple PBX servers by id, and add notify on Grafana to send a notification when any server reaches a breaching point in a period of time, for example, 2 messages SIP 403 - Forbidden continuously for 1 minute will send a notification to email and telegram
• Operate Kamailio as a class 4 Softswitch server with carrier_route module for routing SIP numbers for PBX server
  • Write additional Kamailio code to handle different PSTN provider requirements for From To and Contact header

• Operate OpenSIPs as a SBC (Session Border Control) Server with mid_registrar module and RTPengine for re-write SDP for WebRTC SIP
  • Run between client and PBX server to reduce the rate of Re-REGISTER message
  • WebSocket on OpenSIPs for WebRTC client
  • Meeting with partner companies to help them integrate their system with WebRTC via JsSIP or SIP.js which is a library for web or apps that connect to OpenSIPs WebRTC to use PBX services
• Write a bash shell script for keeping a recording directory of a tenant on the FreeSWITCH server always below the given size or date but with the most recent date (tenant A keeping 5GB of the most recent recordings, tenant B keeping 10GB of the most recent recordings, tenant C keeping last 3 months of the most recent recordings, ...)
• Optimize and tunning database PostgreSQL and MariaDB or MySQL server to maximize concurrent calls and extensions can be run on a PBX server and Autocall server
• Write SIPP code, using SIPP to load test, and stress test with many scenarios to determine how many concurrent calls and extensions can be run on different specs, for example, 2 cores - 2 GB RAM - can run 800 extensions and 200 concurrent calls, 4 cores - 4 GB RAM can run 800 extensions and 200 concurrent calls, ...
• Planning backup and restore solutions when disaster happens
  • Write bash shell script for backup and restore services, dump or restore database PostgreSQL and MariaDB to or from a compressed file, compress or extract all directories for configuration file of asterisk, freeswitch, apache2, nginx, ... to or from a compressed file and upload all of them to a directory on Google Drive via rclone with a organize structure folder by name and date
  • Test restore each quarter to make sure backup files are ready when disaster hit
  • Documented step-by-step command on how to write backup and restore file
• Build, and deploy VoIP Systems such as FusionPBX, Autocall Server based on Asterisk, and OpenSIPs on the cloud and on-premise server using a list of step-by-step commands
• Understanding how VoIP Systems running behind NAT should handle NAT
  • Debug calls are disconnected after 30 seconds because of the setup NAT wrong or timeout RTP which is the default 30 second
• Understanding SIP and RTP
  • How a SIP client can "register" to a PBX server
  • Difference between the first REGISTER message and the second REGISTER message to authenticate a SIP client?
  • How expire time of a client works
  • How a SIP client can call each other, negotiate codec and transmit audio to each other
  • How to use sngrep, tcpdump to capture SIP message on PBX server side, run Wireshark on the client to capture SIP message on the client side
  • Differences between SIP dialog and SIP transaction
  • Remember by heart a basic SIP calls flow when client A calls client B via the PBX server. What is their SIP header i.e From To Contact header and what IP:port they are using to send the SIP message, what in SDP i.e what IP:port they are using to send RTP package
• Works with PSTN provider
  • How a CID or DID must be running to send calls to the mobile phone number
  • How outbound calls should send to the PSTN provider server i.e From & To & Contact headers of the SIP message must be
  • How inbound calls are sent to the PBX server i.e PBX server must not send a request for authentication on inbound calls
  • Debug cases of missing audio on 1 side or 2 sides, latency in audio, PSTN provider sending SIP request or response message to the wrong PBX server
• Operate Asterisk as an Autocall server
  • Setup trunk peer with PSTN provider or trunk account with another PBX
  • Write custom dialplan for fail-over calls for each trunk
  • Write custom dialplan by number regex to maximize cost on each mobile phone number
  • Write custom dialplan when calls are connected to the mobile phone number that calls with sent to another PBX server as inbound calls, to minimize the time the user on the PBX server has to call manually
  • Write custom dialplan to transfer autocall calls to another PBX on user response keypress when hearing auto call audio files
  • Meeting and training customers on how to use the Autocall server
• Operate FreeSWITCH as a PBX server
  • Setup multi-tenant with super-admin as the highest permission and admin for managing tenants and creating extensions on each tenant
  • Setup gateway or trunk peer with PSTN provider or trunk account with another PBX
  • Setup outbound calls by regex to maximize cost on each mobile phone number
  • Setup inbound call flow (ring group, call center, IVR, time condition) and consulting customers on how an inbound call flow should work
  • Setup calls cross-tenant for multi-department company
  • Meeting and training customer how to check calls history and calls recordings
• Operate VoIP peripheral
  • Config IP Phone or Softphone for end-users
  • Config and setup GSM Gateway (Simbox or Sim Gateway) to work with Asterisk and FreeSWITCH