From afd45a1778504786a86627dea9623f780577d8af Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 11:47:25 +0500 Subject: [PATCH 01/48] Create Homer Wiki --- Homer Wiki | 134 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 134 insertions(+) create mode 100644 Homer Wiki diff --git a/Homer Wiki b/Homer Wiki new file mode 100644 index 0000000..fd0d0c9 --- /dev/null +++ b/Homer Wiki @@ -0,0 +1,134 @@ +Quick Install + + +HOMER Setup +This document provides guidance, packages and details to get HOMER installed & running + +HOMER 10 (beta) +🐋 Docker Install (easiest) +HOMER 7 +🎁 Manual Install (easy) +📑 Quick Install (easier) +🐋 Docker Install (easiest) +Clients & Integrations +📞 HEP Capture Agents +📈 HEP Grafana Integration +Found an problem with this guide? Please open an issue and help us improve! + + +🎁 Manual Install: +Homer and its components can easily be installed manually in just minutes. + +Debian +Install the sipcapture repository for Debian/Ubuntu: + +curl -s https://packagecloud.io/install/repositories/qxip/sipcapture/script.deb.sh | sudo bash +Install the latest package + +HOMER/HEP Agent +apt install heplify +HOMER/HEP Server +apt install heplify-server +apt install homer-app +CentOS +Install the sipcapture repository for CentOS/EL 7: + +curl -s https://packagecloud.io/install/repositories/qxip/sipcapture/script.rpm.sh | sudo bash +HOMER/HEP Agent +yum install heplify +HOMER/HEP Server +yum install heplify-server +yum install homer-app +🔧 Configuration +Configure Homer Capture Server /opt/heplify-server/heplify-server.toml +Example Configuration +Configure DB connectors +Postgres +Loki (optional) +Prometheus Metrics Endpoint +Configure Homer Application /usr/local/homer/etc/webapp_config.json +Example Configuration +Configure DB connectors +Postgres +Prometheus API (optional) +InfluxDB API (optional) +Loki API (optional) +⏳ Initialization +Adjust with your database credentials, and execute the following commands to setup the backend components: + +Create Homer DBs +homer-app -create-config-db -database-root-user=postgres -database-host=localhost -database-root-password=postgres -database-homer-user=homer_user +homer-app -create-data-db -database-root-user=postgres -database-host=localhost -database-root-password=postgres -database-homer-user=homer_user +Create Tables, Populate defaults, Upgrade +homer-app -create-table-db-config +homer-app -populate-table-db-config +homer-app -upgrade-table-db-config +🔖 Start Services +systemctl start heplify-server +systemctl start homer-app +Done! +You should now be able to access your HOMER instance via HTTP on port 9080 and send HEPv3 traffic to port 9060/UDP or 9061/TCP (note ports can be modified by the configuration) + +image + + + +📑 Quick Install: +A script is provided to deploy the baseline elements with minimal interaction on a vanilla server: https://github.com/sipcapture/homer-installer + +Quick install currently supports modern debian and centos based operating systems. + +Instructions +Make sure the script is executed as root on a netinstall vanilla server: + +for Debian + +apt-get install libluajit-5.1-common libluajit-5.1-dev lsb-release wget curl git +For CentOS + +yum install redhat-lsb-core wget curl git +then Download Homer Installer script + +wget https://github.com/sipcapture/homer-installer/raw/master/homer_installer.sh +chmod +x homer_installer.sh +./homer_installer.sh + +🐋 Docker Install: +A set of docker-compose bundles is provided to bootstrap a full Homer 7.7 deployment including optional elements: https://github.com/sipcapture/homer7-docker + +Instructions +This procedure requires docker and docker-compose installed on the target system. + +Deployment +git clone https://github.com/sipcapture/homer7-docker +cd homer7-docker/heplify-server/hom7-prom-all +docker-compose up -d +Access & Usage +Homer:9080 (admin/sipcapture) +Grafana:9030 (admin/admin) +Prometheus:9090 (admin/admin) +Loki:3100 (admin/admin) +Alertmanager:9093 (admin/admin) +Done! +You should now be able to access your HOMER instance via HTTP on port 9080 and send HEPv3 traffic to port 9060/UDP or 9061/TCP (note ports can be modified by the docker-compose configuration) + +image + + +📞 Capture Agents +Ready to ship data to Homer? Install a HEP Capture Agent based on your needs and preferences: + +HEPlify: CA developed in go, portable, near zero configuration + +CaptAgent: CA developed in C, ideal for complex configurations + +Native: Native HEP Agents are available in OpenSIPS, Kamailio, Asterisk, Freeswitch, RTP:Engine and many more. Consult the Wiki to get specific examples for your platform. + +Event Agents: HOMER can collect, index and correlate non-packet events such as Logs, RTC stats, CDRs, and more using HEP supported by a variety of tools such as paStash and Telegraf + + +📈 Grafana Integration +Deploying HOMER alongside Grafana? + +Import the HOMER Preset Dashboards & Widgets to get started with standard KPIs in seconds. + From ac97afd67dde380bd475671771cdfba8f69ddca0 Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 11:49:34 +0500 Subject: [PATCH 02/48] Rename Homer Wiki to Homer-Wiki.md --- Homer Wiki => Homer-Wiki.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename Homer Wiki => Homer-Wiki.md (100%) diff --git a/Homer Wiki b/Homer-Wiki.md similarity index 100% rename from Homer Wiki rename to Homer-Wiki.md From 5bf0588ee0e3f964559831baaf2bf5bbce1d8a9a Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 11:50:20 +0500 Subject: [PATCH 03/48] Rename Homer-Wiki.md to How-to-installmd --- Homer-Wiki.md => How-to-installmd | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename Homer-Wiki.md => How-to-installmd (100%) diff --git a/Homer-Wiki.md b/How-to-installmd similarity index 100% rename from Homer-Wiki.md rename to How-to-installmd From 8df0a4be03dbe597679d19b7eee76b32a7fede35 Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 11:50:30 +0500 Subject: [PATCH 04/48] Rename How-to-installmd to How-to-install.md --- How-to-installmd => How-to-install.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename How-to-installmd => How-to-install.md (100%) diff --git a/How-to-installmd b/How-to-install.md similarity index 100% rename from How-to-installmd rename to How-to-install.md From 122513044d7580064cafba55b3e01c5cfa5d95c5 Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 11:52:40 +0500 Subject: [PATCH 05/48] Update How-to-install.md --- How-to-install.md | 217 +++++++++++++++++++++++++++++----------------- 1 file changed, 137 insertions(+), 80 deletions(-) diff --git a/How-to-install.md b/How-to-install.md index fd0d0c9..8542b16 100644 --- a/How-to-install.md +++ b/How-to-install.md @@ -1,134 +1,191 @@ -Quick Install - +
-HOMER Setup +# __HOMER__ Setup This document provides guidance, packages and details to get HOMER installed & running -HOMER 10 (beta) -🐋 Docker Install (easiest) -HOMER 7 -🎁 Manual Install (easy) -📑 Quick Install (easier) -🐋 Docker Install (easiest) -Clients & Integrations -📞 HEP Capture Agents -📈 HEP Grafana Integration -Found an problem with this guide? Please open an issue and help us improve! +### HOMER 10 _(beta)_ +* 🐋 [Docker Install](https://github.com/sipcapture/homer-docker) _(easiest)_ -🎁 Manual Install: -Homer and its components can easily be installed manually in just minutes. +### HOMER 7 +* 🎁 [Manual Install](#-manual-install) _(easy)_ +* 📑 [Quick Install](#-quick-install) _(easier)_ +* 🐋 [Docker Install](#-docker-install) _(easiest)_ + + +#### Clients & Integrations +* 📞 [HEP Capture Agents](#-capture-agents) +* 📈 [HEP Grafana Integration](#-grafana-integration) + + + +_Found an problem with this guide? Please open an [issue](https://github.com/sipcapture/homer/issues) and help us improve!_ -Debian -Install the sipcapture repository for Debian/Ubuntu: + +## 🎁 Manual Install: +Homer and its components can easily be installed manually in just minutes. + +### Debian +Install the `sipcapture `repository for Debian/Ubuntu: +``` curl -s https://packagecloud.io/install/repositories/qxip/sipcapture/script.deb.sh | sudo bash +``` Install the latest package +###### HOMER/HEP Agent + * `apt install heplify` +###### HOMER/HEP Server + * `apt install heplify-server` + * `apt install homer-app` -HOMER/HEP Agent -apt install heplify -HOMER/HEP Server -apt install heplify-server -apt install homer-app -CentOS -Install the sipcapture repository for CentOS/EL 7: +### CentOS +Install the `sipcapture `repository for CentOS/EL 7: +``` curl -s https://packagecloud.io/install/repositories/qxip/sipcapture/script.rpm.sh | sudo bash -HOMER/HEP Agent -yum install heplify -HOMER/HEP Server -yum install heplify-server -yum install homer-app -🔧 Configuration -Configure Homer Capture Server /opt/heplify-server/heplify-server.toml -Example Configuration -Configure DB connectors -Postgres -Loki (optional) -Prometheus Metrics Endpoint -Configure Homer Application /usr/local/homer/etc/webapp_config.json -Example Configuration -Configure DB connectors -Postgres -Prometheus API (optional) -InfluxDB API (optional) -Loki API (optional) -⏳ Initialization +``` + +###### HOMER/HEP Agent + * `yum install heplify` +###### HOMER/HEP Server + * `yum install heplify-server` + * `yum install homer-app` + + +#### 🔧 Configuration + * Configure Homer Capture Server + `/opt/heplify-server/heplify-server.toml` + * [Example Configuration](https://github.com/sipcapture/heplify-server/blob/master/example/homer7_config/heplify-server.toml) + * Configure DB connectors + * Postgres + * Loki _(optional)_ + * Prometheus Metrics Endpoint + * Configure Homer Application + `/usr/local/homer/etc/webapp_config.json` + * [Example Configuration](https://github.com/sipcapture/homer-app/blob/master/etc/webapp_config.json) + * Configure DB connectors + * Postgres + * Prometheus API _(optional)_ + * InfluxDB API _(optional)_ + * Loki API _(optional)_ + +#### ⏳ Initialization Adjust with your database credentials, and execute the following commands to setup the backend components: -Create Homer DBs +##### Create Homer DBs +``` homer-app -create-config-db -database-root-user=postgres -database-host=localhost -database-root-password=postgres -database-homer-user=homer_user homer-app -create-data-db -database-root-user=postgres -database-host=localhost -database-root-password=postgres -database-homer-user=homer_user -Create Tables, Populate defaults, Upgrade +``` + +##### Create Tables, Populate defaults, Upgrade +``` homer-app -create-table-db-config homer-app -populate-table-db-config homer-app -upgrade-table-db-config -🔖 Start Services +``` + +#### 🔖 Start Services +``` systemctl start heplify-server systemctl start homer-app -Done! -You should now be able to access your HOMER instance via HTTP on port 9080 and send HEPv3 traffic to port 9060/UDP or 9061/TCP (note ports can be modified by the configuration) +``` -image +#### Done! +You should now be able to access your HOMER instance via HTTP on port `9080` and send HEPv3 traffic to port `9060/UDP` or `9061/TCP` _(note ports can be modified by the configuration)_ +![image](https://user-images.githubusercontent.com/1423657/73536888-5513dd80-4427-11ea-82aa-b2ce53192a63.png) +
+ -📑 Quick Install: -A script is provided to deploy the baseline elements with minimal interaction on a vanilla server: https://github.com/sipcapture/homer-installer +## 📑 Quick Install: +A script is provided to deploy the baseline elements with minimal interaction on a vanilla server: +[https://github.com/sipcapture/homer-installer](https://github.com/sipcapture/homer-installer) -Quick install currently supports modern debian and centos based operating systems. +Quick install currently supports modern `debian` and `centos` based operating systems. -Instructions -Make sure the script is executed as root on a netinstall vanilla server: +### Instructions +Make sure the script is executed as `root` on a `netinstall` vanilla server: for Debian - +``` apt-get install libluajit-5.1-common libluajit-5.1-dev lsb-release wget curl git +``` For CentOS - +``` yum install redhat-lsb-core wget curl git +``` then Download Homer Installer script - +``` wget https://github.com/sipcapture/homer-installer/raw/master/homer_installer.sh chmod +x homer_installer.sh ./homer_installer.sh +``` + -🐋 Docker Install: -A set of docker-compose bundles is provided to bootstrap a full Homer 7.7 deployment including optional elements: https://github.com/sipcapture/homer7-docker + -Instructions -This procedure requires docker and docker-compose installed on the target system. +## 🐋 Docker Install: +A set of `docker-compose` bundles is provided to bootstrap a full Homer 7.7 deployment including optional elements: +[https://github.com/sipcapture/homer7-docker](https://github.com/sipcapture/homer7-docker/tree/7.7/heplify-server) -Deployment +### Instructions +This procedure requires `docker` and `docker-compose` installed on the target system. + +#### Deployment +``` git clone https://github.com/sipcapture/homer7-docker cd homer7-docker/heplify-server/hom7-prom-all docker-compose up -d -Access & Usage -Homer:9080 (admin/sipcapture) -Grafana:9030 (admin/admin) -Prometheus:9090 (admin/admin) -Loki:3100 (admin/admin) -Alertmanager:9093 (admin/admin) -Done! -You should now be able to access your HOMER instance via HTTP on port 9080 and send HEPv3 traffic to port 9060/UDP or 9061/TCP (note ports can be modified by the docker-compose configuration) +``` +#### Access & Usage + * Homer:9080 _(admin/sipcapture)_ + * Grafana:9030 _(admin/admin)_ + * Prometheus:9090 _(admin/admin)_ + * Loki:3100 _(admin/admin)_ + * Alertmanager:9093 _(admin/admin)_ + +#### Done! +You should now be able to access your HOMER instance via HTTP on port `9080` and send HEPv3 traffic to port `9060/UDP` or `9061/TCP` _(note ports can be modified by the docker-compose configuration)_ -image +![image](https://user-images.githubusercontent.com/1423657/73536888-5513dd80-4427-11ea-82aa-b2ce53192a63.png) + -📞 Capture Agents +## 📞 Capture Agents Ready to ship data to Homer? Install a HEP Capture Agent based on your needs and preferences: +* [HEPlify](https://github.com/sipcapture/heplify): + CA developed in go, portable, near zero configuration +* [CaptAgent](https://github.com/sipcapture/captagent): + CA developed in C, ideal for complex configurations + +* **Native**: + [Native HEP Agents](https://github.com/sipcapture/homer/wiki) are available in [OpenSIPS](https://github.com/sipcapture/homer/wiki/Examples%3A-OpenSIPS), [Kamailio](https://github.com/sipcapture/homer/wiki/Examples%3A-Kamailio), [Asterisk](https://github.com/sipcapture/homer/wiki/Examples%3A-Asterisk), [Freeswitch](https://github.com/sipcapture/homer/wiki/Examples%3A-FreeSwitch), [RTP:Engine](https://github.com/sipcapture/homer/wiki/Examples%3A-RTPEngine) and many more. Consult the Wiki to get specific examples for your platform. + +* **Event Agents**: +HOMER can collect, index and correlate non-packet events such as Logs, RTC stats, CDRs, and more using [HEP](https://github.com/sipcapture/hep) supported by a variety of tools such as [paStash](https://github.com/sipcapture/pastash/wiki) and [Telegraf](https://github.com/influxdata/telegraf/pull/6167) + + + + +## 📈 Grafana Integration +Deploying HOMER alongside Grafana? + +Import the [HOMER Preset Dashboards & Widgets](https://grafana.com/grafana/dashboards?direction=asc&orderBy=name&search=homer) to get started with standard KPIs in seconds. + + -HEPlify: CA developed in go, portable, near zero configuration -CaptAgent: CA developed in C, ideal for complex configurations + -Native: Native HEP Agents are available in OpenSIPS, Kamailio, Asterisk, Freeswitch, RTP:Engine and many more. Consult the Wiki to get specific examples for your platform. -Event Agents: HOMER can collect, index and correlate non-packet events such as Logs, RTC stats, CDRs, and more using HEP supported by a variety of tools such as paStash and Telegraf +Made with ❤️ by Humans +*(C) 2008-2021 [QXIP BV](http://qxip.net)* -📈 Grafana Integration -Deploying HOMER alongside Grafana? +---------- -Import the HOMER Preset Dashboards & Widgets to get started with standard KPIs in seconds. +##### If you use HOMER in production, please consider supporting the project with a [Donation](https://www.paypal.com/cgi-bin/webscr?cmd=_donations&business=donation%40sipcapture%2eorg&lc=US&item_name=SIPCAPTURE&no_note=0¤cy_code=EUR&bn=PP%2dDonationsBF%3abtn_donateCC_LG%2egif%3aNonHostedGuest) +[![Donate](https://www.paypalobjects.com/en_US/i/btn/btn_donateCC_LG.gif)](https://www.paypal.com/cgi-bin/webscr?cmd=_donations&business=donation%40sipcapture%2eorg&lc=US&item_name=SIPCAPTURE&no_note=0¤cy_code=EUR&bn=PP%2dDonationsBF%3abtn_donateCC_LG%2egif%3aNonHostedGuest) From 17e307f5eb7b15f254edcd130a0df469b271ba1e Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 12:04:48 +0500 Subject: [PATCH 06/48] Create How-to Install (Docker).md --- How-to Install (Docker).md | 160 +++++++++++++++++++++++++++++++++++++ 1 file changed, 160 insertions(+) create mode 100644 How-to Install (Docker).md diff --git a/How-to Install (Docker).md b/How-to Install (Docker).md new file mode 100644 index 0000000..6db74ee --- /dev/null +++ b/How-to Install (Docker).md @@ -0,0 +1,160 @@ +# HOMER 7.7 Docker +We provide a number of pre-configured `docker-compose` recipe to bootstrap `Homer` and its components in seconds: + +Repository: [homer7-docker](https://github.com/sipcapture/homer7-docker) + + From 8f1ecb676254cbb2b32952dbc0b0449fb4a34ccb Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 12:06:17 +0500 Subject: [PATCH 07/48] Create Homer 10.md --- Homer 10.md | 138 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 138 insertions(+) create mode 100644 Homer 10.md diff --git a/Homer 10.md b/Homer 10.md new file mode 100644 index 0000000..6cf8a9d --- /dev/null +++ b/Homer 10.md @@ -0,0 +1,138 @@ + + +# HOMER 10 +### 100% Open-Source VoIP & RTC Observability +> Real-Time Packet Capture, Troubleshooting & Monitoring + + + + + + + +![H5](https://img.shields.io/badge/HOMER-10-red.svg) +![HEP](https://img.shields.io/badge/proto-hep_eep-blue.svg) +![HEP](https://img.shields.io/badge/proto-sip-brightgreen.svg) +![HEP](https://img.shields.io/badge/proto-rtcp-brightgreen.svg) +![HEP](https://img.shields.io/badge/proto-rtcp_xr-brightgreen.svg) +![HEP](https://img.shields.io/badge/proto-rtp_stats-brightgreen.svg) +![HEP](https://img.shields.io/badge/text-QoS-green.svg) +![HEP](https://img.shields.io/badge/text-syslog-green.svg) +![HEP](https://img.shields.io/badge/text-CDRs-green.svg) + +## Introduction + +**HOMER** is a robust, carrier-grade, scalable Packet and Event Observability framework for VoiP/RTC Monitoring Applications based on the [HEP/EEP](http://github.com/sipcapture/hep) protocol and ready to ingest insane amounts of signaling, rtc events, logs and statistics with instant search, end-to-end correlation and drill-down capabilities. + +**HOMER** is already used by large enterprises, voice network operators, voip service providers and traffic carriers worldwide, has been implemented as a service in 3rd party voice platforms and is suitable for production. + +**HOMER** 10+ reimagines the platform as a headless observaility solution using standard protocols and visualization tools such as Grafana. + +
+ +##### Core Features +* Based on HEP Encapsulation, available everywhere +* Stand-Alone Capture Servers & Agents for any OS/Platform +* Supports advanced SIP, RTP/RTCP Reports, RTC Events and Custom protocols +* No SQL Databases. Powered by stock Observability APIs such as [qryn](https://qryn.dev) +* More than Packets and PCAPs. Ingest and process Logs, Metrics and Traces. +* Dynamic Mapping and Correlation for internal and external data sources +* Made by Humans, and Supported by the best community ever + + +Unlike its predecessors, HOMER 10 is designed to natively fit modern observability standards and to navigate VoIP and WebRTC troubleshooting into the present and future. Without loosing any feature and retaining full backwards compatibility with the HEPv3 encapsulation format HOMER 10 can capture and transform packets, sessions and reports into industry standard Logs, Metrics and Traces in realtime. + + +## [Installation](https://github.com/sipcapture/homer/wiki/Quick-Install) +Ready to Install HOMER? Choose your preferred method from our [Wiki](https://github.com/sipcapture/homer/wiki/Quick-Install) :thumbsup: + +### Support + +For community support, updates, user discussion and experience exchange please join our [Matrix channel](https://matrix.to/#/#sipcapture_homer:gitter.im) and [Mailing-List](https://groups.google.com/forum/#!forum/homer-discuss). If you'd like to help the project or donate resources, drop us an email at support@sipcapture.org + +For professional support, remote setups, customizations or commercial licensing please contact the QXIP Team at [http://qxip.net](http://qxip.net) + + + +---- + +### Presentations +If you'd like to get an idea about what HOMER is and what HOMER does, consider watching one of our presentations or workshops from ClueCon, OpenSIPS Summit, Kamailio World, CommCon, Fosdem and many more. + + +---------------- + + +### Developers +Contributors and Contributions to our project are always welcome! Developers and Users can coordinate with the existing team via our [Matrix channel](https://matrix.to/#/#sipcapture_homer:gitter.im). If you'd like to join our internal team and volunteer to help with the project's many needs, feel free to contact us anytime! + +### ⭐️ Project Assistance +If you want to say thank you or/and support active development: + +- Add a GitHub [Star to the project](https://github.com/sipcapture/homer/stargazers). +- Tweet about our project on Social Media `@qxip @sipcapture` +- Contribute guides and articles about our project on Dev.to, Medium, personal blog, etc. + +[![Stargazers over time](https://starchart.cc/sipcapture/homer.svg)](https://starchart.cc/sipcapture/homer) + + +### License & Copyright + +![H5](https://img.shields.io/badge/license-GNU_AGPL_v3-blue.svg) + +Homer components are released under the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. + +*(C) 2008-2022 [QXIP BV](http://qxip.net)* + +---------- + + + + + +#### Made by Humans + +This Open-Source project is made possible by actual Humans without corporate sponsors, angels or patreons.
+If you use this software in production, please consider supporting its development with contributions or [donations](https://www.paypal.com/cgi-bin/webscr?cmd=_donations&business=donation%40sipcapture%2eorg&lc=US&item_name=SIPCAPTURE&no_note=0¤cy_code=EUR&bn=PP%2dDonationsBF%3abtn_donateCC_LG%2egif%3aNonHostedGuest) + +[![Donate](https://www.paypalobjects.com/en_US/i/btn/btn_donateCC_LG.gif)](https://www.paypal.com/cgi-bin/webscr?cmd=_donations&business=donation%40sipcapture%2eorg&lc=US&item_name=SIPCAPTURE&no_note=0¤cy_code=EUR&bn=PP%2dDonationsBF%3abtn_donateCC_LG%2egif%3aNonHostedGuest) From 8215e8fb922d9760f3c793bc7efafcdd70bcbcbe Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 12:07:40 +0500 Subject: [PATCH 08/48] Homer10.md --- README.md | 136 +++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 134 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index f4a4814..6cf8a9d 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,138 @@ -#### 100% Open-Source VoIP & RTC Capture, Troubleshooting & Monitoring -## [Official Documentation](https://sipcapture.github.io/docs/) for [HOMER](https://github.com/sipcapture/homer) +# HOMER 10 +### 100% Open-Source VoIP & RTC Observability +> Real-Time Packet Capture, Troubleshooting & Monitoring + + + + + +![H5](https://img.shields.io/badge/HOMER-10-red.svg) +![HEP](https://img.shields.io/badge/proto-hep_eep-blue.svg) +![HEP](https://img.shields.io/badge/proto-sip-brightgreen.svg) +![HEP](https://img.shields.io/badge/proto-rtcp-brightgreen.svg) +![HEP](https://img.shields.io/badge/proto-rtcp_xr-brightgreen.svg) +![HEP](https://img.shields.io/badge/proto-rtp_stats-brightgreen.svg) +![HEP](https://img.shields.io/badge/text-QoS-green.svg) +![HEP](https://img.shields.io/badge/text-syslog-green.svg) +![HEP](https://img.shields.io/badge/text-CDRs-green.svg) + +## Introduction + +**HOMER** is a robust, carrier-grade, scalable Packet and Event Observability framework for VoiP/RTC Monitoring Applications based on the [HEP/EEP](http://github.com/sipcapture/hep) protocol and ready to ingest insane amounts of signaling, rtc events, logs and statistics with instant search, end-to-end correlation and drill-down capabilities. + +**HOMER** is already used by large enterprises, voice network operators, voip service providers and traffic carriers worldwide, has been implemented as a service in 3rd party voice platforms and is suitable for production. + +**HOMER** 10+ reimagines the platform as a headless observaility solution using standard protocols and visualization tools such as Grafana. + +
+ +##### Core Features +* Based on HEP Encapsulation, available everywhere +* Stand-Alone Capture Servers & Agents for any OS/Platform +* Supports advanced SIP, RTP/RTCP Reports, RTC Events and Custom protocols +* No SQL Databases. Powered by stock Observability APIs such as [qryn](https://qryn.dev) +* More than Packets and PCAPs. Ingest and process Logs, Metrics and Traces. +* Dynamic Mapping and Correlation for internal and external data sources +* Made by Humans, and Supported by the best community ever + + +Unlike its predecessors, HOMER 10 is designed to natively fit modern observability standards and to navigate VoIP and WebRTC troubleshooting into the present and future. Without loosing any feature and retaining full backwards compatibility with the HEPv3 encapsulation format HOMER 10 can capture and transform packets, sessions and reports into industry standard Logs, Metrics and Traces in realtime. + + +## [Installation](https://github.com/sipcapture/homer/wiki/Quick-Install) +Ready to Install HOMER? Choose your preferred method from our [Wiki](https://github.com/sipcapture/homer/wiki/Quick-Install) :thumbsup: + +### Support + +For community support, updates, user discussion and experience exchange please join our [Matrix channel](https://matrix.to/#/#sipcapture_homer:gitter.im) and [Mailing-List](https://groups.google.com/forum/#!forum/homer-discuss). If you'd like to help the project or donate resources, drop us an email at support@sipcapture.org + +For professional support, remote setups, customizations or commercial licensing please contact the QXIP Team at [http://qxip.net](http://qxip.net) + + + +---- + +### Presentations +If you'd like to get an idea about what HOMER is and what HOMER does, consider watching one of our presentations or workshops from ClueCon, OpenSIPS Summit, Kamailio World, CommCon, Fosdem and many more. + + +---------------- + + +### Developers +Contributors and Contributions to our project are always welcome! Developers and Users can coordinate with the existing team via our [Matrix channel](https://matrix.to/#/#sipcapture_homer:gitter.im). If you'd like to join our internal team and volunteer to help with the project's many needs, feel free to contact us anytime! + +### ⭐️ Project Assistance +If you want to say thank you or/and support active development: + +- Add a GitHub [Star to the project](https://github.com/sipcapture/homer/stargazers). +- Tweet about our project on Social Media `@qxip @sipcapture` +- Contribute guides and articles about our project on Dev.to, Medium, personal blog, etc. + +[![Stargazers over time](https://starchart.cc/sipcapture/homer.svg)](https://starchart.cc/sipcapture/homer) + + +### License & Copyright + +![H5](https://img.shields.io/badge/license-GNU_AGPL_v3-blue.svg) + +Homer components are released under the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. + +*(C) 2008-2022 [QXIP BV](http://qxip.net)* + +---------- + + + + + +#### Made by Humans + +This Open-Source project is made possible by actual Humans without corporate sponsors, angels or patreons.
+If you use this software in production, please consider supporting its development with contributions or [donations](https://www.paypal.com/cgi-bin/webscr?cmd=_donations&business=donation%40sipcapture%2eorg&lc=US&item_name=SIPCAPTURE&no_note=0¤cy_code=EUR&bn=PP%2dDonationsBF%3abtn_donateCC_LG%2egif%3aNonHostedGuest) + +[![Donate](https://www.paypalobjects.com/en_US/i/btn/btn_donateCC_LG.gif)](https://www.paypal.com/cgi-bin/webscr?cmd=_donations&business=donation%40sipcapture%2eorg&lc=US&item_name=SIPCAPTURE&no_note=0¤cy_code=EUR&bn=PP%2dDonationsBF%3abtn_donateCC_LG%2egif%3aNonHostedGuest) From e8847bc22510e263f4adea6e9c55cfb3c8386afa Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 12:11:35 +0500 Subject: [PATCH 09/48] Homer 10- Docker Setup.md --- Homer 10- Docker Setup.md | 82 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 82 insertions(+) create mode 100644 Homer 10- Docker Setup.md diff --git a/Homer 10- Docker Setup.md b/Homer 10- Docker Setup.md new file mode 100644 index 0000000..68e8161 --- /dev/null +++ b/Homer 10- Docker Setup.md @@ -0,0 +1,82 @@ + + + + +# homer 10 + qryn (all-in-one) +The ultimate **homer** + **qryn.js** docker demo w/ sample _hep traffic, logs, traces and metrics_ - _batteries included!_ :battery::battery: + +- HOMER 10 = _qryn.js + heplify-server + clickhouse-server + grafana-server_ + +> All the backend features for all supported protocols are provided by **qryn** and **clickhouse** - no other components required! + +
+ + + + + + + +
+ +### Usage + +#### Setup +Clone the repository and launch the **qryn** polyglot demo using _docker-compose_ + +```bash +docker-compose pull +docker-compose up -d +``` + +#### Send Data +This demo comes with auto-generated data, but you can still send your own streams +##### HEP +Send some HEP traffic to the HEP socket on port 9060/udp or 9061/tcp or use the demo `hepgen` generated traffic + +##### Others +Send data using a variety of protocols + + +  +  +  +  +  +  +  +
+
+ + +#### Login +Access the preconfigured Grafana instance as `admin/admin` +``` +http://localhost:3000 +``` +#### Explore +The demo generates correlated _logs, metrics and traces_ with autoconfigured datasources + + - ```Loki``` + - ```Tempo``` + - ```Prometheus``` + - ```Flux``` + +#### Data Ingestion +The demo supports all ingestion protocols supported by **homer** _(hep)_ and **qryn** _(logql, prometheus, influx, elastic, opentelemetry, etc)_ + + +#### Go Pro +Loving it? Learn how to send your own data using our [online documentation](https://qryn.metrico.in) + +
+ +![image](https://github.com/sipcapture/homer-docker/assets/1423657/a3ed8398-4930-4c00-a593-782ce1696756) + +![image](https://user-images.githubusercontent.com/1423657/183254312-b52811e5-f563-440e-84e4-8312714a4c9b.png) + +![image](https://user-images.githubusercontent.com/1423657/183254290-fac87747-51ce-4648-a7aa-073fdcdd6c10.png) + +![image](https://user-images.githubusercontent.com/1423657/186708038-685467ee-a135-4fa0-af31-eae487da2139.png) + +![image](https://user-images.githubusercontent.com/1423657/186280231-8fbcf1f1-69b7-43fe-91ad-7e6ee8389978.png) From f4d710df58214d201af0025d9e8eb57102fc3096 Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 12:12:30 +0500 Subject: [PATCH 10/48] Homer-10 Docker Setup.md --- README.md | 164 ++++++++++++++++++------------------------------------ 1 file changed, 54 insertions(+), 110 deletions(-) diff --git a/README.md b/README.md index 6cf8a9d..68e8161 100644 --- a/README.md +++ b/README.md @@ -1,138 +1,82 @@ - + -# HOMER 10 -### 100% Open-Source VoIP & RTC Observability -> Real-Time Packet Capture, Troubleshooting & Monitoring - +# homer 10 + qryn (all-in-one) +The ultimate **homer** + **qryn.js** docker demo w/ sample _hep traffic, logs, traces and metrics_ - _batteries included!_ :battery::battery: - +- HOMER 10 = _qryn.js + heplify-server + clickhouse-server + grafana-server_ +> All the backend features for all supported protocols are provided by **qryn** and **clickhouse** - no other components required! -![H5](https://img.shields.io/badge/HOMER-10-red.svg) -![HEP](https://img.shields.io/badge/proto-hep_eep-blue.svg) -![HEP](https://img.shields.io/badge/proto-sip-brightgreen.svg) -![HEP](https://img.shields.io/badge/proto-rtcp-brightgreen.svg) -![HEP](https://img.shields.io/badge/proto-rtcp_xr-brightgreen.svg) -![HEP](https://img.shields.io/badge/proto-rtp_stats-brightgreen.svg) -![HEP](https://img.shields.io/badge/text-QoS-green.svg) -![HEP](https://img.shields.io/badge/text-syslog-green.svg) -![HEP](https://img.shields.io/badge/text-CDRs-green.svg) +
-## Introduction + -**HOMER** is a robust, carrier-grade, scalable Packet and Event Observability framework for VoiP/RTC Monitoring Applications based on the [HEP/EEP](http://github.com/sipcapture/hep) protocol and ready to ingest insane amounts of signaling, rtc events, logs and statistics with instant search, end-to-end correlation and drill-down capabilities. + -**HOMER** is already used by large enterprises, voice network operators, voip service providers and traffic carriers worldwide, has been implemented as a service in 3rd party voice platforms and is suitable for production. + -**HOMER** 10+ reimagines the platform as a headless observaility solution using standard protocols and visualization tools such as Grafana. +
-
+### Usage -##### Core Features -* Based on HEP Encapsulation, available everywhere -* Stand-Alone Capture Servers & Agents for any OS/Platform -* Supports advanced SIP, RTP/RTCP Reports, RTC Events and Custom protocols -* No SQL Databases. Powered by stock Observability APIs such as [qryn](https://qryn.dev) -* More than Packets and PCAPs. Ingest and process Logs, Metrics and Traces. -* Dynamic Mapping and Correlation for internal and external data sources -* Made by Humans, and Supported by the best community ever +#### Setup +Clone the repository and launch the **qryn** polyglot demo using _docker-compose_ +```bash +docker-compose pull +docker-compose up -d +``` -Unlike its predecessors, HOMER 10 is designed to natively fit modern observability standards and to navigate VoIP and WebRTC troubleshooting into the present and future. Without loosing any feature and retaining full backwards compatibility with the HEPv3 encapsulation format HOMER 10 can capture and transform packets, sessions and reports into industry standard Logs, Metrics and Traces in realtime. +#### Send Data +This demo comes with auto-generated data, but you can still send your own streams +##### HEP +Send some HEP traffic to the HEP socket on port 9060/udp or 9061/tcp or use the demo `hepgen` generated traffic +##### Others +Send data using a variety of protocols -## [Installation](https://github.com/sipcapture/homer/wiki/Quick-Install) -Ready to Install HOMER? Choose your preferred method from our [Wiki](https://github.com/sipcapture/homer/wiki/Quick-Install) :thumbsup: - -### Support - -For community support, updates, user discussion and experience exchange please join our [Matrix channel](https://matrix.to/#/#sipcapture_homer:gitter.im) and [Mailing-List](https://groups.google.com/forum/#!forum/homer-discuss). If you'd like to help the project or donate resources, drop us an email at support@sipcapture.org - -For professional support, remote setups, customizations or commercial licensing please contact the QXIP Team at [http://qxip.net](http://qxip.net) - - - ----- - -### Presentations -If you'd like to get an idea about what HOMER is and what HOMER does, consider watching one of our presentations or workshops from ClueCon, OpenSIPS Summit, Kamailio World, CommCon, Fosdem and many more. - - ----------------- - - -### Developers -Contributors and Contributions to our project are always welcome! Developers and Users can coordinate with the existing team via our [Matrix channel](https://matrix.to/#/#sipcapture_homer:gitter.im). If you'd like to join our internal team and volunteer to help with the project's many needs, feel free to contact us anytime! - -### ⭐️ Project Assistance -If you want to say thank you or/and support active development: + +  +  +  +  +  +  +  +
+
-- Add a GitHub [Star to the project](https://github.com/sipcapture/homer/stargazers). -- Tweet about our project on Social Media `@qxip @sipcapture` -- Contribute guides and articles about our project on Dev.to, Medium, personal blog, etc. -[![Stargazers over time](https://starchart.cc/sipcapture/homer.svg)](https://starchart.cc/sipcapture/homer) +#### Login +Access the preconfigured Grafana instance as `admin/admin` +``` +http://localhost:3000 +``` +#### Explore +The demo generates correlated _logs, metrics and traces_ with autoconfigured datasources + - ```Loki``` + - ```Tempo``` + - ```Prometheus``` + - ```Flux``` -### License & Copyright +#### Data Ingestion +The demo supports all ingestion protocols supported by **homer** _(hep)_ and **qryn** _(logql, prometheus, influx, elastic, opentelemetry, etc)_ -![H5](https://img.shields.io/badge/license-GNU_AGPL_v3-blue.svg) -Homer components are released under the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. +#### Go Pro +Loving it? Learn how to send your own data using our [online documentation](https://qryn.metrico.in) -*(C) 2008-2022 [QXIP BV](http://qxip.net)* +
----------- +![image](https://github.com/sipcapture/homer-docker/assets/1423657/a3ed8398-4930-4c00-a593-782ce1696756) - - - +![image](https://user-images.githubusercontent.com/1423657/183254312-b52811e5-f563-440e-84e4-8312714a4c9b.png) -#### Made by Humans +![image](https://user-images.githubusercontent.com/1423657/183254290-fac87747-51ce-4648-a7aa-073fdcdd6c10.png) -This Open-Source project is made possible by actual Humans without corporate sponsors, angels or patreons.
-If you use this software in production, please consider supporting its development with contributions or [donations](https://www.paypal.com/cgi-bin/webscr?cmd=_donations&business=donation%40sipcapture%2eorg&lc=US&item_name=SIPCAPTURE&no_note=0¤cy_code=EUR&bn=PP%2dDonationsBF%3abtn_donateCC_LG%2egif%3aNonHostedGuest) +![image](https://user-images.githubusercontent.com/1423657/186708038-685467ee-a135-4fa0-af31-eae487da2139.png) -[![Donate](https://www.paypalobjects.com/en_US/i/btn/btn_donateCC_LG.gif)](https://www.paypal.com/cgi-bin/webscr?cmd=_donations&business=donation%40sipcapture%2eorg&lc=US&item_name=SIPCAPTURE&no_note=0¤cy_code=EUR&bn=PP%2dDonationsBF%3abtn_donateCC_LG%2egif%3aNonHostedGuest) +![image](https://user-images.githubusercontent.com/1423657/186280231-8fbcf1f1-69b7-43fe-91ad-7e6ee8389978.png) From f4b7c092d80a36fc105b2aace1726c8df1f70207 Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 12:13:49 +0500 Subject: [PATCH 11/48] Hep Validation.md --- Hep Validation.md | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) create mode 100644 Hep Validation.md diff --git a/Hep Validation.md b/Hep Validation.md new file mode 100644 index 0000000..dfdf5ce --- /dev/null +++ b/Hep Validation.md @@ -0,0 +1,39 @@ + + +# Testing HOMER with hepgen.js + +Not sure if your HOMER setup is operational? No problem! + +[hepgen.js](https://github.com/sipcapture/hepgen.js) simulates a HEP capture agent and allows you to quick validate your system. Think of it as a SIPP for HEP packets you can use to simulate various scenarios and features. + +### Requirements +* [NodeJS 14.x](https://nodejs.org/en/download/package-manager) or higher + +### Installation +Install `hepgen.js` globally using npm +``` +npm install -g hepgen.js +``` + +### Testing +Let's run `hepgen.js` against oue HOMER deployment! + +``` +hepgen.js -s HOMER_IP -p HOMER_PORT -c $(npm root -g)/hepgen.js/config/b2bcall_rtcp.js +``` + +Replace `HOMER_IP` _(127.0.0.1)_ and `HOMER_PORT` _(9060)_ to point at your setup. You want to simulate an agent! + + +### Validation +Login to your HOMER instance, set the time range to the last 10 minutes, and search. Your HEPgen.js call should appear. + +Our synthetic HEP session will contain randomly generated IPs and numbers from SIP user `hepgenjs` + +![image](https://user-images.githubusercontent.com/1423657/114322818-7f305780-9b22-11eb-922f-b03d5ee07024.png) + + + + + +For more information, please refer to the [hepgen.js](https://github.com/sipcapture/hepgen.js) repository. From b5bba5ff0a7dfdf3d1bf4c301f1c79d24038ccf6 Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 12:16:00 +0500 Subject: [PATCH 12/48] Create LuaJIT Scripting.md --- LuaJIT Scripting.md | 193 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 193 insertions(+) create mode 100644 LuaJIT Scripting.md diff --git a/LuaJIT Scripting.md b/LuaJIT Scripting.md new file mode 100644 index 0000000..fcb1885 --- /dev/null +++ b/LuaJIT Scripting.md @@ -0,0 +1,193 @@ + + +## HOMER + LuaJIT _(HEPjit)_ +One of hottest features in **HOMER 7** is packet manipulation using **LuaJIT** :boom: + +This feature delivers unprecedented power and flexibility in processing **HEP** messages to extract custom headers, anonymize or encrypt user data, tag traffic based on custom logic and beyond only limited by your imagination and skills - all performing at native speed! + +This idea has been incubating for a few years and @adubovikov now released it to heplify-server ❤️ + +-------- + +### *So how does this work?* + +### Requirements +* heplify-server w/ LuaJIT ([git](https://github.com/sipcapture/heplify-server/tree/script_lua)) +* liblua5.1-dev + +### Configuration +A few simple steps are required to enable and configure your first HEP Lua script. + +#### Heplify-Server +###### TOML +Enable scripting and configure your entrypoint LuaJIT script, by default inside the `lua` directory. +```toml +ScriptEnable = true +ScriptFolder = "lua/" +``` +### ScriptFile +The entrypoint LuaJIT script will be called for each matching HEP packet on its way to the core. The following default example provides a few usage examples for *HEP Type 1* (SIP) to extract a custom header using Regex, Add custom tags and Replace existing message fields, all commented -- inline. + +```lua +-- this function will be executed first +function checkRAW() + + local protoType = GetHEPProtoType() + + -- Check if we have SIP type + if protoType ~= 1 then + return + end + + -- original SIP message Payload + local raw = GetRawMessage() + -- Logp("DEBUG", "raw", raw) + + -- Create lua table + local headers = {} + headers["X-test"] = "Super TEST Header" + + -- local _, _, name, value = string.find(raw, "(Call-ID:)%s*:%s*(.+)") + local name, value = raw:match("(CSeq):%s+(.-)\n") + + if name == "CSeq" then + headers[name] = value + end + + SetCustomSIPHeader(headers) + + return + +end + +-- this function will be executed second +function checkSIP() + + -- get the parsed SIP struct + local sip = GetSIPStruct() + + -- a struct can be nil so better check it + if (sip == nil or sip == '') then + return + end + + if sip.FromHost == "127.0.0.1" then + -- Logp("ERROR", "found User-Agent:", sip.UserAgent) + end + + SetSIPHeader("FromHost", "1.1.1.1") + + -- Full SIP messsage can be changed + -- SetRawMessage("SIP 2/0") + + return + +end + +-- this function will be executed third +function changeNodeIDtoNameFast() + + -- get only nodeID + local nodeID = GetHEPNodeID() + if nodeID == 0 then + SetHEPField("NodeName","TestNode") + end + + return + +end + +-- this function will be executed fourth +function changeNodeIDtoNameSlow() + -- get the parsed HEP struct + local hep = GetHEPStruct() + + -- a struct can be nil so better check it + if (hep == nil or hep == '') then + return + end + + if hep.NodeID == 0 then + hep.NodeName="TestNode" + end + + return + +end +``` + +### Add Fields to Mapping +In order to display and search any added field, implement in Mapping for the desired transaction type: +``` +{ + "id": "CSeq", + "type": "string", + "index": "secondary", + "name": "CSeq Extraction", + "form_type": "input", + "position": 99, + "sid_type": true, + "hide": false + } +``` + +* Checkout the list of [standard preparsed headers](#standard-header-list) +* Thinking of complex logic and routing? The entrypoint script can also include and use sub-scripts. + +### Done +That's it, You're ready to go! + +If you find this powerful feature interesting and use it in your setup, please report any bugs and contribute your scripts or changes back to the community to grow a fantastic library of HEPxperiments! + +---- + +### Hot reload lua script +You can reload your lua script without interruption with following command: +``` +killall -HUP heplify-server +``` + +### Standard Function List +The following functions can be used.
+``` +GetHEPStruct() +GetSIPStruct() +GetHEPProtoType() +GetHEPSrcIP() +GetHEPSrcPort() +GetHEPDstIP() +GetHEPDstPort() +GetHEPTimeSeconds() +GetHEPTimeUseconds() +GetHEPNodeID() +GetRawMessage() +SetRawMessage(value string) +SetCustomSIPHeader(map luatable) +SetHEPField(field string, value string) +SetSIPHeader(header string, value string) +HashString(hashfunc string, text string) +Logp(level string, message string, data interface{}) +Print(text string) +``` + +### Standard Header List +The following headers can be directly manipulated.
+For custom extractions, use Regex against the RAW header. +``` +"ViaOne" +"FromUser" +"FromHost" +"FromTag" +"ToUser" +"ToHost" +"ToTag" +"CallID" +"XCallID" +"ContactUser" +"ContactHost" +"Authorization.Username" +"UserAgent" +"Server" +"PaiUser" +"PaiHost" +``` From eac91e6661bfe764ffefb9bf03b4a74bfc426f1f Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 12:17:19 +0500 Subject: [PATCH 13/48] Create Homer 7.md --- Homer 7.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 Homer 7.md diff --git a/Homer 7.md b/Homer 7.md new file mode 100644 index 0000000..92a67d4 --- /dev/null +++ b/Homer 7.md @@ -0,0 +1,15 @@ +# HOMER 7.7 Documentation + +This is a WORK IN PROGRESS! Please contribute sections and details - any contribution is welcome! + +### User Interface + + +### Query URL +`homer-app` provides a GET API to inject query parameters and return tailored results: +#### API Endpoint +`http://homer-app/call/result?` +#### URL Parameters +```json +{"timestamp":{"from":1579647311472,"to":1579648211472},"param":{"search":{"1_call":{"callid":["aaaasb@127.0.0.1","aaaasb@127.0.0.1_b2b-1"]}},"transaction":{"call":true,"registration":false,"rest":false},"timezone":{"value":-60,"name":"Local"}}}= +``` From b7890cb47055a3ab6b53cb901549a5cbbb2a4ef6 Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 12:19:31 +0500 Subject: [PATCH 14/48] Create Server Configuration.md --- Server Configuration.md | 204 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 204 insertions(+) create mode 100644 Server Configuration.md diff --git a/Server Configuration.md b/Server Configuration.md new file mode 100644 index 0000000..4ccd3b8 --- /dev/null +++ b/Server Configuration.md @@ -0,0 +1,204 @@ +##### HEP Socket for UDP Traffic +The address for the HEP server to listen on for HEP traffic. +``` +HEPAddr = "0.0.0.0:9060" +``` + +##### HEP Socket for TCP Traffic +The TCP address for the HEP server. (For now its empty, indicating it is not being used) +``` +HEPTCPAddr = "0.0.0.0:9062" +``` + +##### HEP Socket for TLS Traffic +The TLS address for the HEP server to listen on for secure HEP traffic. +``` +HEPTLSAddr = "0.0.0.0:9063" +``` + +##### Elasticsearch Endpoint Address +The address for Elasticsearch. +``` +ESAddr = "http://127.0.0.1:9200" +``` +##### Elasticsearch Endpoint discovery +A boolean value indicating whether Elasticsearch discovery is enabled or not. +``` +ESDiscovery = true +``` + +##### Loki Endpoint Address +The URL for Loki, a log aggregation system. +``` +LokiURL = "http://localhost:3100/api/prom/push" +``` +##### Loki Bulk Max Size +The number of log entries to send to Loki in bulk. +``` +LokiBulk = 200 +``` +##### Loki Bulk Timeout +The interval (in seconds) at which logs are sent to Loki. +``` +LokiTimer = 4 +``` +##### Loki Buffer size +The maximum number of log entries that can be buffered before sending to Loki. +``` +LokiBuffer = 100000 +``` +##### Loki HEP Filter +A list of HEP filter values to specify which types of HEP messages should be sent to Loki. +``` +LokiHEPFilter = [1,5,100] +``` +##### Loki HEP Payload +A list of HEP payload types to force their inclusion in the HEP messages. +``` +ForceHEPPayload = [] +``` +##### Prometheus Endpoint Address +The address for Prometheus. +``` +PromAddr = "0.0.0.0:8899" +``` +##### Prometheus Target IP +The IP address for the Prometheus target. +``` +PromTargetIP = "10.1.2.111,10.1.2.4,10.1.2.5,10.1.2.6,10.12.44.222" +``` +##### Prometheus Target Name +The name of the Prometheus target. +``` +PromTargetName = "sbc_access,sbc_core,kamailio,asterisk,pstn_gateway" +``` +##### Database Schema (homer7, homer5) +The database schema to use. +``` +DBShema = "homer7" +``` +##### Database Driver (postgres) +The database driver to use (in this case, "postgres"). +``` +DBDriver = "postgres" +``` +##### Database Address +The address of the database server. In this case, it is set to "localhost:5432", indicating that the database is located on the local machine and listening on port 5432. +``` +DBAddr = "localhost:5432" +``` +##### Database Auth User +The username used to authenticate with the database server. In this case, it is set to "postgres". +``` +DBUser = "postgres" +``` +##### Database Auth Password +The password used to authenticate with the database server. Currently empty, indicating that no password is provided. +``` +DBPass = "" +``` +##### Database Data Table name +The table name in the database where HEP data will be stored. In this case, it is set to "homer_data". +``` +DBDataTable = "homer_data" +``` +##### Database Config Table name +The table name in the database where HEP configuration will be stored. In this case, it is set to "homer_config". +``` +DBConfTable = "homer_config" +``` +##### Database Bulk max size +The number of database entries to insert in bulk. It is set to 200, meaning that database entries will be inserted in groups of 200. +``` +DBBulk = 200 +``` +##### Database Bulk Timeout +The interval (in seconds) at which database entries are inserted. +``` +DBTimer = 4 +``` +##### Database Buffer Size +The maximum number of database entries that can be buffered before inserting. +``` +DBBuffer = 400000 +``` +##### Database Worker Instances +The number of worker threads for handling database operations. +``` +DBWorker = 8 +``` +##### Rotation Enable +A boolean value indicating whether database table rotation is enabled or not. +``` +DBRotate = true +``` +##### Database Rotation Settings +``` +DBPartLog = "2h" +This param specifies the duration for which the log data should be stored in the database. + +DBPartSip = "1h" +This param determines the duration for which SIP data should be stored in the database. + +DBPartQos = "6h" +This param specifies the duration for which data related to Quality of Service (QoS) should be stored in the database. +``` +##### Database Drop Settings +``` +DBDropDays = 14 +This param specifies the number of days after which all data in the database (except logs) will be deleted automatically. + +DBDropDaysCall = 0 +This param sets the number of days after which call-related data will be deleted from the database. + +DBDropDaysRegister = 0 +This param indicate the number of days after which registration-related data will be dropped from the database. + +DBDropDaysDefault = 0 +This param specifies the number of days after which any other data that is not related to the above categories will be dropped from the database. + +DBDropOnStart = false +This param specifies the number of days after which all data in the database (except logs) will be deleted automatically. Users must use this action carefully as it is irreversible. +``` +##### Enable Packet De-Duplication +This param enables or disables deduplication of packets in the database. +``` +Dedup = false +``` +##### Discard Methods (array) +This param specifies the methods for discarding data that is not captured by the server. +``` +DiscardMethod = ["OPTIONS","NOTIFY"] +``` +##### Custom Headers +``` +AlegIDs = ["X-CID","P-Charging-Vector,icid-value=\"?(.*?)(?:\"|;|$)","X-BroadWorks-Correlation-Info"] +This param allows you to specify A-leg IDs to filter the captured data. Only data related to the specified A-leg IDs will be stored in the database. + +CustomHeader = ["X-CustomerIP","X-Billing"] + This param is used to define custom headers. +``` +##### Logger Settings +``` +LogDbg = "hep,sql,loki" +This param controls the debug log configuration. + +LogLvl = "info" +This param determines the log level for the server. + +LogStd = false +A param whether logs should be output to the standard output. + +LogSys = false +A param indicates whether logs should be sent to the system logger. +``` +##### Configuration Location +This param specifies the path to the configuration file for the server. +``` +Config = "./heplify-server.toml" +``` +##### Configuration Web Editor URL +This param determines the HTTP address for the configuration server. +``` +ConfigHTTPAddr = "0.0.0.0:9876" +``` From 39599999603ef6c4ab3bd18fd54582bd3c0a79ac Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 12:21:39 +0500 Subject: [PATCH 15/48] Create Local/OAuth2 Options.md --- Local/OAuth2 Options.md | 98 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 98 insertions(+) create mode 100644 Local/OAuth2 Options.md diff --git a/Local/OAuth2 Options.md b/Local/OAuth2 Options.md new file mode 100644 index 0000000..990e824 --- /dev/null +++ b/Local/OAuth2 Options.md @@ -0,0 +1,98 @@ +# HOMER Authentication + +## Local _(default)_ +By default, HOMER uses its internal database for users and permissions: +```javascript + "auth_settings": { + "_comment": "The type param can be internal, ldap, http_auth", + "token_expire": 1200, + "type": "internal" + } +``` + +### LDAP +The following example contains an LDAP integration template for generic services: +```javascript +"auth_settings": { + "token_expire": 1200, + "type": "ldap" +}, +"ldap_config": { + "admingroup": "admin", + "adminmode": true, + "anonymous": false, + "attributes": [ + "dn", + "givenName", + "sn", + "mail", + "uid" + ], + "base": "dc=example,dc=com", + "binddn": "uid=readonlysuer,ou=People,dc=example,dc=com", + "bindpassword": "readonlypassword", + "groupattribute": "cn", + "groupfilter": "(memberUid=%s)", + "host": "ldap.example.com", + "port": 389, + "skiptls": true, + "skipverify": true, + "userdn": "uid=%s,ou=People,dc=example,dc=com", + "userfilter": "(uid=%s)", + "usergroup": "HOMER_user", + "usermode": true, + "usessl": false + }, +``` + +### HTTP/S AUTH +Authentication can be delegated to an external API using the HTTP/S AUTH type and JSON User objects +``` +"auth_settings": { + "token_expire": 1200, + "type": "http_auth" +}, +"http_auth": { + "skipverify": true, + "url": "http://localhost:1323" +}, +``` + +##### Request +``` +{ + "username": "httpuser", + "password": sha1(password), +} +``` +##### Response +``` + "id": 122345 + "username": "httpuser", + "partid": 10, + "email": "homer@rocks", + "firstname": "Homer", + "lastname": "SIP", + "department": "Beer", + "usergroup": "family", + "guid": UUID +``` + + +## OAuth2 +The following example contains an OAuth2 integration with a generic service. This works in parallel to other methods. +```javascript +"oauth2": { + "enable": true, + "client_id": "XXXXXXXXXXXX-xxxxxxxxxxxxxxxxxxxxxxxxxxxxx.apps.googleusercontent.com", + "client_secret": "XXXXXX-XXXXXXXXXXXXXXXXXXXXXXXXXXXX", + "project_id": "Homer OAuth", + "auth_uri": "https://accounts.google.com/o/oauth2/auth", + "token_uri": "https://oauth2.googleapis.com/token", + "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", + "redirect_uri": "http://YOUR_HOMER_URL/api/v3/oauth2/auth", + "profile_uri": "https://www.googleapis.com/oauth2/v1/userinfo", + "provider_name": "google", + "provider_image": "" +} +``` From ca040c87fe6d207ba7d81aeb59f72021fc471838 Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 12:36:59 +0500 Subject: [PATCH 16/48] Delete Local/OAuth2 Options.md --- Local/OAuth2 Options.md | 98 ----------------------------------------- 1 file changed, 98 deletions(-) delete mode 100644 Local/OAuth2 Options.md diff --git a/Local/OAuth2 Options.md b/Local/OAuth2 Options.md deleted file mode 100644 index 990e824..0000000 --- a/Local/OAuth2 Options.md +++ /dev/null @@ -1,98 +0,0 @@ -# HOMER Authentication - -## Local _(default)_ -By default, HOMER uses its internal database for users and permissions: -```javascript - "auth_settings": { - "_comment": "The type param can be internal, ldap, http_auth", - "token_expire": 1200, - "type": "internal" - } -``` - -### LDAP -The following example contains an LDAP integration template for generic services: -```javascript -"auth_settings": { - "token_expire": 1200, - "type": "ldap" -}, -"ldap_config": { - "admingroup": "admin", - "adminmode": true, - "anonymous": false, - "attributes": [ - "dn", - "givenName", - "sn", - "mail", - "uid" - ], - "base": "dc=example,dc=com", - "binddn": "uid=readonlysuer,ou=People,dc=example,dc=com", - "bindpassword": "readonlypassword", - "groupattribute": "cn", - "groupfilter": "(memberUid=%s)", - "host": "ldap.example.com", - "port": 389, - "skiptls": true, - "skipverify": true, - "userdn": "uid=%s,ou=People,dc=example,dc=com", - "userfilter": "(uid=%s)", - "usergroup": "HOMER_user", - "usermode": true, - "usessl": false - }, -``` - -### HTTP/S AUTH -Authentication can be delegated to an external API using the HTTP/S AUTH type and JSON User objects -``` -"auth_settings": { - "token_expire": 1200, - "type": "http_auth" -}, -"http_auth": { - "skipverify": true, - "url": "http://localhost:1323" -}, -``` - -##### Request -``` -{ - "username": "httpuser", - "password": sha1(password), -} -``` -##### Response -``` - "id": 122345 - "username": "httpuser", - "partid": 10, - "email": "homer@rocks", - "firstname": "Homer", - "lastname": "SIP", - "department": "Beer", - "usergroup": "family", - "guid": UUID -``` - - -## OAuth2 -The following example contains an OAuth2 integration with a generic service. This works in parallel to other methods. -```javascript -"oauth2": { - "enable": true, - "client_id": "XXXXXXXXXXXX-xxxxxxxxxxxxxxxxxxxxxxxxxxxxx.apps.googleusercontent.com", - "client_secret": "XXXXXX-XXXXXXXXXXXXXXXXXXXXXXXXXXXX", - "project_id": "Homer OAuth", - "auth_uri": "https://accounts.google.com/o/oauth2/auth", - "token_uri": "https://oauth2.googleapis.com/token", - "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", - "redirect_uri": "http://YOUR_HOMER_URL/api/v3/oauth2/auth", - "profile_uri": "https://www.googleapis.com/oauth2/v1/userinfo", - "provider_name": "google", - "provider_image": "" -} -``` From 0b10928605f1e2613bfc242a41b1eb511a9456ba Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 12:38:50 +0500 Subject: [PATCH 17/48] Create Grafana Integration.md --- Grafana Integration.md | 11 +++++++++++ 1 file changed, 11 insertions(+) create mode 100644 Grafana Integration.md diff --git a/Grafana Integration.md b/Grafana Integration.md new file mode 100644 index 0000000..55cf156 --- /dev/null +++ b/Grafana Integration.md @@ -0,0 +1,11 @@ +When using Grafana on a different domain, HOMER offers an integrated proxy to bridge and authorize requests, embed widgets, etc: + +``` +"grafana_config": { + "help": "Settings for Grafana", + "host": "http://127.0.0.1:3000", + "path": "/grafana", + "proxy_control": false, + "token": "" +} +``` From 72c7985bb08e72da24d5e334574b7b2a6b18e147 Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 12:40:37 +0500 Subject: [PATCH 18/48] Rename Grafana Integration.md to docs/Grafana Integration.md --- Grafana Integration.md => docs/Grafana Integration.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename Grafana Integration.md => docs/Grafana Integration.md (100%) diff --git a/Grafana Integration.md b/docs/Grafana Integration.md similarity index 100% rename from Grafana Integration.md rename to docs/Grafana Integration.md From 624e7b129e01be256ee3d5e728493294561b7e94 Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 12:41:12 +0500 Subject: [PATCH 19/48] Rename Hep Validation.md to docs/Hep Validation.md --- Hep Validation.md => docs/Hep Validation.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename Hep Validation.md => docs/Hep Validation.md (100%) diff --git a/Hep Validation.md b/docs/Hep Validation.md similarity index 100% rename from Hep Validation.md rename to docs/Hep Validation.md From 696e7d3331b3d341070aba629c161a542a7ba59a Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 12:41:42 +0500 Subject: [PATCH 20/48] Rename Homer 10- Docker Setup.md to docs/Homer 10- Docker Setup.md --- Homer 10- Docker Setup.md => docs/Homer 10- Docker Setup.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename Homer 10- Docker Setup.md => docs/Homer 10- Docker Setup.md (100%) diff --git a/Homer 10- Docker Setup.md b/docs/Homer 10- Docker Setup.md similarity index 100% rename from Homer 10- Docker Setup.md rename to docs/Homer 10- Docker Setup.md From 531351c591cbaaae2e7e172518fd32cfbeae27a5 Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 12:42:07 +0500 Subject: [PATCH 21/48] Rename Homer 10.md to docs/Homer 10.md --- Homer 10.md => docs/Homer 10.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename Homer 10.md => docs/Homer 10.md (100%) diff --git a/Homer 10.md b/docs/Homer 10.md similarity index 100% rename from Homer 10.md rename to docs/Homer 10.md From e0da0de2d8511ad71031c93d8d21208a3ac354a2 Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 12:42:21 +0500 Subject: [PATCH 22/48] Rename Homer 7.md to docs/Homer 7.md --- Homer 7.md => docs/Homer 7.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename Homer 7.md => docs/Homer 7.md (100%) diff --git a/Homer 7.md b/docs/Homer 7.md similarity index 100% rename from Homer 7.md rename to docs/Homer 7.md From 2461cefdb341e02f2f092b033a356969b9a6b254 Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 12:42:37 +0500 Subject: [PATCH 23/48] Rename How-to Install (Docker).md to docs/How-to Install (Docker).md --- How-to Install (Docker).md => docs/How-to Install (Docker).md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename How-to Install (Docker).md => docs/How-to Install (Docker).md (100%) diff --git a/How-to Install (Docker).md b/docs/How-to Install (Docker).md similarity index 100% rename from How-to Install (Docker).md rename to docs/How-to Install (Docker).md From 81f26d985c54d1114dc2da5a6a6354d2a4fe8243 Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 12:42:55 +0500 Subject: [PATCH 24/48] Rename How-to-install.md to docs/How-to-install.md --- How-to-install.md => docs/How-to-install.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename How-to-install.md => docs/How-to-install.md (100%) diff --git a/How-to-install.md b/docs/How-to-install.md similarity index 100% rename from How-to-install.md rename to docs/How-to-install.md From 338f57378fc7ca97e4418ba17fdb2b7262a96b9a Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 12:43:11 +0500 Subject: [PATCH 25/48] Rename LuaJIT Scripting.md to docs/LuaJIT Scripting.md --- LuaJIT Scripting.md => docs/LuaJIT Scripting.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename LuaJIT Scripting.md => docs/LuaJIT Scripting.md (100%) diff --git a/LuaJIT Scripting.md b/docs/LuaJIT Scripting.md similarity index 100% rename from LuaJIT Scripting.md rename to docs/LuaJIT Scripting.md From ed965abf586b2931c51a6e684800672600cb0000 Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 12:43:26 +0500 Subject: [PATCH 26/48] Rename Server Configuration.md to docs/Server Configuration.md --- Server Configuration.md => docs/Server Configuration.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename Server Configuration.md => docs/Server Configuration.md (100%) diff --git a/Server Configuration.md b/docs/Server Configuration.md similarity index 100% rename from Server Configuration.md rename to docs/Server Configuration.md From 667283b60067f4643ade7643fb68e1587013b451 Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 12:44:20 +0500 Subject: [PATCH 27/48] Create Local-0Auth Options.md --- docs/Local-0Auth Options.md | 98 +++++++++++++++++++++++++++++++++++++ 1 file changed, 98 insertions(+) create mode 100644 docs/Local-0Auth Options.md diff --git a/docs/Local-0Auth Options.md b/docs/Local-0Auth Options.md new file mode 100644 index 0000000..990e824 --- /dev/null +++ b/docs/Local-0Auth Options.md @@ -0,0 +1,98 @@ +# HOMER Authentication + +## Local _(default)_ +By default, HOMER uses its internal database for users and permissions: +```javascript + "auth_settings": { + "_comment": "The type param can be internal, ldap, http_auth", + "token_expire": 1200, + "type": "internal" + } +``` + +### LDAP +The following example contains an LDAP integration template for generic services: +```javascript +"auth_settings": { + "token_expire": 1200, + "type": "ldap" +}, +"ldap_config": { + "admingroup": "admin", + "adminmode": true, + "anonymous": false, + "attributes": [ + "dn", + "givenName", + "sn", + "mail", + "uid" + ], + "base": "dc=example,dc=com", + "binddn": "uid=readonlysuer,ou=People,dc=example,dc=com", + "bindpassword": "readonlypassword", + "groupattribute": "cn", + "groupfilter": "(memberUid=%s)", + "host": "ldap.example.com", + "port": 389, + "skiptls": true, + "skipverify": true, + "userdn": "uid=%s,ou=People,dc=example,dc=com", + "userfilter": "(uid=%s)", + "usergroup": "HOMER_user", + "usermode": true, + "usessl": false + }, +``` + +### HTTP/S AUTH +Authentication can be delegated to an external API using the HTTP/S AUTH type and JSON User objects +``` +"auth_settings": { + "token_expire": 1200, + "type": "http_auth" +}, +"http_auth": { + "skipverify": true, + "url": "http://localhost:1323" +}, +``` + +##### Request +``` +{ + "username": "httpuser", + "password": sha1(password), +} +``` +##### Response +``` + "id": 122345 + "username": "httpuser", + "partid": 10, + "email": "homer@rocks", + "firstname": "Homer", + "lastname": "SIP", + "department": "Beer", + "usergroup": "family", + "guid": UUID +``` + + +## OAuth2 +The following example contains an OAuth2 integration with a generic service. This works in parallel to other methods. +```javascript +"oauth2": { + "enable": true, + "client_id": "XXXXXXXXXXXX-xxxxxxxxxxxxxxxxxxxxxxxxxxxxx.apps.googleusercontent.com", + "client_secret": "XXXXXX-XXXXXXXXXXXXXXXXXXXXXXXXXXXX", + "project_id": "Homer OAuth", + "auth_uri": "https://accounts.google.com/o/oauth2/auth", + "token_uri": "https://oauth2.googleapis.com/token", + "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", + "redirect_uri": "http://YOUR_HOMER_URL/api/v3/oauth2/auth", + "profile_uri": "https://www.googleapis.com/oauth2/v1/userinfo", + "provider_name": "google", + "provider_image": "" +} +``` From f8d7a0ebb966b607bb3220ef79459bf9670d3619 Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 12:44:56 +0500 Subject: [PATCH 28/48] Create Troubleshooting.md --- docs/Troubleshooting.md | 132 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 132 insertions(+) create mode 100644 docs/Troubleshooting.md diff --git a/docs/Troubleshooting.md b/docs/Troubleshooting.md new file mode 100644 index 0000000..27f4b6e --- /dev/null +++ b/docs/Troubleshooting.md @@ -0,0 +1,132 @@ +Guys, please if you have some issues, provide a little bit more information. We couldn't guess or read your mind remotely. + +## Latest versions: +First of all, please be sure that you have the latest version of homer-app / homer-ui / heplify-server: + +[Homer-APP](https://github.com/sipcapture/homer-app/releases/latest) + +[Homer-UI](https://github.com/sipcapture/homer-ui/releases/latest) + +[Heplify-Server](https://github.com/sipcapture/heplify-server/releases/latest) + +## version desync : homer-app vs homer-ui +The homer-app is an API server (backend). The homer-ui is an UI frontend (JS/HTML). The frontend communicates with the homer-API by using JSON/Rest protocol. The packages (rpm/deb) of homer-app contain the actual version of UI (minimized and gziped). If you have installed the homer-app (binary) manualy (download from a repo or compiled from the sources), this means you have to upgrade/install the homer-ui also manualy (check how it to do using sources, go to https://github.com/sipcapture/homer-ui). Once you have compiled it, copy the dist files to the dist directory of homer-app, usualy it's /usr/local/homer/dist, (to be sure, check your webapp_conf.json: https://github.com/sipcapture/homer-app/blob/master/etc/webapp_config.json#L48). If you are lazy or don't know how to use npm to compile javascript, you can download the latest compiled version of the frontend: https://github.com/sipcapture/homer-app/releases/latest . + + +1.1.32 - an example here. Please use the latest version! +``` +wget https://github.com/sipcapture/homer-app/releases/download/1.1.32/homer-ui-7.7.028.tgz +tar xzf homer-ui-7.7.028.tgz +cp -Rp dist/* /usr/local/homer/dist/ + +``` + + + +or use our packagecloud.io repository (https://github.com/sipcapture/homer/wiki/Quick-Install#-manual-install) + +So. You run the latest versions and you still have issues, than you go to next steps: + +**If you are upgrading Homer 7.0 to Homer 7.7. Please reinit your configuration database:** + +`./homer-app -populate-table-db-config -force-populate` + +or if you have some users and don't loose them, upgrade only these tables: + +`./homer-app -populate-table-db-config -force-populate -populate-table=mapping_schema -populate-table=global_settings -populate-table=user_settings -populate-table=hepsub_mapping_schema` + +*** +## Database inspection: + +**Be sure you have some SIP (or RTCP, LOG) data in the tables of your DB (homer_data)** + +Execute in the shell (terminal/ssh) of the DB host: + +``` +# su - postgres +$ psql homer_data +homer_data=# \dt +``` + +it will show you if you have any tables in your database: + +**I don't see any tables, the DB is empty or psql tells you: "psql: FATAL: database "homer_data" does not exist"** + +- you didn't install postgress or it's not up. +- you didn't create database (see the https://github.com/sipcapture/homer/wiki/Quick-Install#-initialization) +- you didn't run heplify-server or there is any issues with connectivity (wrong credentials, a wrong path to the config of heplify-server ( heplify-server.toml ) or any firewall/iptable, because your heplify-server will create the tables on first start and will do the partition rotation job. + +**Oh, I have tables:** + +Great, now: + +**Check if you have any data inside:** + +``` +select count(*) from hep_proto_1_call; + count +--------- + 3993340 +(1 row) + +``` +*** +## HEP inspection: + +if you see 0 rows = you didn't send any HEP data or again wrong configuration (heplify-server, captagent or firewall) Be sure that you send +- **SIP** +- encapsulation protocol **HEPv3** , not HEPv2 + +Again **non RTCP non LOG, you send SIP** + +(if you are not sure - capture packets with wireshark/ngrep/tcpdump (ports: 9060 or 9063) and do analyze with https://github.com/sipcapture/hep-wireshark . In worse case, check if you have HEPv3 (ASCII) at the begin of packet's payload) + +*** +## Debug in web console +**if you have records, but you don't see anything in your UI, but you can login and create widgets, you see configurations etc...** + +in the browser, press **F12** (Web console): + +and do a search again. Please be sure that you have selected the correct time range. +If you don't see any requests in the Network tab: + +**in Homer - UI do:** + +- Logout +- Login +- and press Search again + +and NOW: + +**Check the /api/v3/search/call/data +Don't execute this query, go to the Network tab in your Web console F12 (see the screenshoots) +and check the reply of this api request and analyze the data. Check if you have some records inside** + +*The screenshoots:** + +![Screenshot from 2020-02-13 09-06-48](https://user-images.githubusercontent.com/4513061/74415509-cbf7a000-4e43-11ea-9a45-18b0c8f6977e.png) + +![Screenshot from 2020-02-13 09-07-40](https://user-images.githubusercontent.com/4513061/74415533-d7e36200-4e43-11ea-8cea-8d2421c6e191.png) + +*** +## UI/APP misconfiguration + +**if you don't see any data in the result grid (table) of UI, but you have data in hep_proto_1_call (DB: homer_data, Table: hep_proto_1_call):** : + +- timerange is wrong - you don't have any data for this time range. Be sure you don't have any search parameters and make the range bigger. +- the database connection to homer_data in your webapp_conf.json is wrong or firewall issue.__ + +## Submit a bug report: + +**If you see a response with DATA inside, but nothing has been displayed:** + +Congratulation, probably you found a bug, and you can submit an issue +- Submit versions of your homer-app/UI/heplify/heplify-server (the version of UI you can find in the title.) +- How did you install the applications (package / installer script / self compiling) +- How do you send data to Homer (SIP/ RTCP/ RTP / LOG) +- Small network topology description +- any additonal information that can help us understand and solve your issue + + +I hope this small instruction helps you to localize your problem and avoid waste your and our time for a ping-pong communication. + From 7e48a109c569ea8025d19583f351566c6a361493 Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 12:46:00 +0500 Subject: [PATCH 29/48] Create Custom Headers & Correlation.md --- docs/Custom Headers & Correlation.md | 175 +++++++++++++++++++++++++++ 1 file changed, 175 insertions(+) create mode 100644 docs/Custom Headers & Correlation.md diff --git a/docs/Custom Headers & Correlation.md b/docs/Custom Headers & Correlation.md new file mode 100644 index 0000000..596ce59 --- /dev/null +++ b/docs/Custom Headers & Correlation.md @@ -0,0 +1,175 @@ +This article describes the process and steps required to extract custom SIP `X-headers` for search and correlation purposes. + +# 1. Add Custom Header in heplify-server + +`X-Headers` are extracted at the capture layer by `heplify-server` and converted to HEP header fields. + +- *Correlation Only* Headers which do not require search should be _"normalized"_ to `correlation_id` headers using `AlegIDs` +- *Search + Correlation* Headers which require full indexing should be extracted using `CustomHeader` + +Edit `/etc/heplify-server.toml` + +add custom header information +``` +AlegIDs = ["XCallID"] # Custom Header(s) will be used as Correlation ID (first match) +ForceALegID = true # Enable search for A Leg Headers. default is false +CustomHeader = ["X-CID","XCallID"] # Custom Headers will be Indexed for Header search +``` + +restart heplify-server +``` +systemctl restart heplify-server.service +``` +Now you will be able to see the custom `X-headers` in Packet details. Click on INVITE packet and then go to Details + +![](https://i.imgur.com/ajngs6K.jpg) + +# 2. Add Mappings in homer-app + +Each header in SIP is mapped to a name in the schema to view/search in Homer UI, which is referred to as `Mapping`. For each custom header, we have to add new mapping accordingly. + +Go to `Settings` in homer UI and click on `Mapping`, Edit **profile** 'call' with **HEP ID** '1'. + +![](https://i.imgur.com/w8DvAXQ.jpg) + +There will be two kinds of `Mapping` +1. **Correlation Mapping:** Defines any cross-protocol correlation mechanism +2. **Fields Mapping:** Defines schema and mapping for protocol fields + +Go to `Fields Mapping` + +![](https://i.imgur.com/9Xvtcts.jpg) + +Add the following at end of `Fields Mapping` + +```json + { + "id": "data_header.X-CID", + "name": "X-CID", + "type": "string", + "index": "none", + "form_type": "input", + "position": 25, + "skip": false, + "hide": true + }, + { + "id": "data_header.XCallID", + "name": "XCallID", + "type": "string", + "index": "none", + "form_type": "input", + "position": 26, + "skip": false, + "hide": true + } +``` + +# 3. Call SIP Search Widget Settings + +Go to `Home` screen of homer. Eidt settings of `Call SIP Search` widget. + +![](https://i.imgur.com/y1t6x4I.jpg) + +Move X-CID and XCallID to active list. + +![](https://i.imgur.com/YcbD3on.jpg) + +# 4. Search with Custom Header + +now you will be able to search with a custom Header field. + +![](https://i.imgur.com/6Vy6jYX.jpg) + +Note: Custom Header search will work with new calls only. + +---------- + +# Full Example for 2-way correlation + +#### Mapping +```json +[ + { + "source_field": "data_header.callid", + "lookup_id": 1, + "lookup_profile": "call", + "append_sid": true, + "lookup_field": "data_header->>'X-CID'", + "lookup_range": [ + -300, + 200 + ] + }, + { + "source_field": "data_header.X-CID", + "lookup_id": 1, + "lookup_profile": "call", + "append_sid": true, + "lookup_field": "data_header->>'X-CID'", + "lookup_range": [ + -300, + 200 + ] + }, + { + "source_field": "data_header.X-CID", + "lookup_id": 1, + "lookup_profile": "call", + "append_sid": true, + "lookup_field": "data_header->>'callid'", + "lookup_range": [ + -300, + 200 + ] + }, + { + "source_field": "data_header.callid", + "lookup_id": 1, + "lookup_profile": "call", + "lookup_field": "sid", + "lookup_range": [ + -300, + 200 + ] + } +] +``` + + +### Field Modification +Fields can be modified and extended using the `input_function_js` functionality. The following example extends an extracted callid with `_b2b-1` + +```json + { + "source_field": "data_header.callid", + "lookup_id": 1, + "lookup_profile": "call", + "lookup_field": "sid", + "input_function_js": "var returnData=[]; for (var i = 0; i < data.length; i++) { returnData.push(data[i]+'_b2b-1'); }; returnData;" + "lookup_range": [ + -300, + 200 + ] + } + + +``` + +Or multiple prefixes and suffixes `SBC-` , `_PBX-1` + +```json + { + "source_field": "data_header.callid", + "lookup_id": 1, + "lookup_profile": "call", + "lookup_field": "sid", + "input_function_js": "var returnData=[]; for (var i = 0; i < data.length; i++) { returnData.push('SBC-'+data[i]); returnData.push(data[i]+'_PBX-1'); }; returnData;" + "lookup_range": [ + -300, + 200 + ] + } + + +``` From b6af75eeb98d9e350fed822afdedc0ac1c5f5dde Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 12:46:56 +0500 Subject: [PATCH 30/48] Create Using Homer API.md --- docs/Using Homer API.md | 75 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+) create mode 100644 docs/Using Homer API.md diff --git a/docs/Using Homer API.md b/docs/Using Homer API.md new file mode 100644 index 0000000..c6e39f2 --- /dev/null +++ b/docs/Using Homer API.md @@ -0,0 +1,75 @@ +# HOMER 7 API + +The HOMER 7 API is documented using swagger and available directly within the ADMIN UI + +![image](https://user-images.githubusercontent.com/1423657/145407324-dc05160e-74a2-42b9-a79e-fe2b949074ef.png) + +------------- + +# HOMER 5 API (OUTDATED!) + +For full documentation, please install and consult the dedicated [APIDOC](https://github.com/sipcapture/homer-api/tree/master/apidoc) available on the [API repository](https://github.com/sipcapture/homer-api). A Few basic illustrative examples are available on the main [repository](https://github.com/sipcapture/homer/tree/master/utils/scripts) + +## Introduction +In order for requests to be accepted and processed by the API, a Client session ID needs to be authorized with the API Backend. Requests without an authorized or expired Cookie will always be rejected forcing Client re-authentication. + +## Auth + +### Auth - create a session +``` +http://localhost/api/v1/session +``` +Example usage: +``` +curl -v --cookie "HOMERSESSID=tcuass65ejl2lifoopuuurpmq7; path=/" -X POST -H "Content-Type: application/json" \ +-d '{"username":"admin","password":"test123"}' \ +http://localhost/api/v1/session +``` + +#### cookie + +| Field | Type | Description | +| ------------- |:-------------:| -----:| +| HOMERSESSID | String | cookie session id | + +Parameter + +| Field | Type | Description | +| ------------- |:-------------:| -----:| +| username | String | Login for session creation| +| password | String | Password for session creation| + + +#### Success-Response: + +The returned *session UUID* should be used in cookie request for subsequent API calls: +``` +HTTP/1.1 200 OK +{ + "status": 200, + "sid": "tcuass65ejl2lifoopuuurpmq7", + "auth": "true", + "message": "ok", + "data": { + "uid": "3", + "username": "admin", + "gid": "10", + "grp": "users,admins", + "firstname": "Alexandr", + "lastname": "Dubovikov", + "email": "admin@sipcapture.org", + "lastvisit": "2015-06-18 08:25:55" +} +``` +#### Error-Response: +``` +HTTP/1.1 200 OK +Set-Cookie: HOMERSESSID=tcuass65ejl2lifoopuuurpmq7; path=/ +Content-Type: application/json; charset=UTF-8 + +{ + "sid":"tcuass65ejl2lifoopuuurpmq7" + "auth":"false", + "status":"wrong-session" +} +``` From 85c3bce737ffa7e7c704d57401181b9d92bd0817 Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 12:47:53 +0500 Subject: [PATCH 31/48] Create Using DB Nodes.md --- docs/Using DB Nodes.md | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) create mode 100644 docs/Using DB Nodes.md diff --git a/docs/Using DB Nodes.md b/docs/Using DB Nodes.md new file mode 100644 index 0000000..b9abebd --- /dev/null +++ b/docs/Using DB Nodes.md @@ -0,0 +1,39 @@ +# HOMER 7 DB NODES +### Database Data +The data connections allow `homer-app` to communicate with one or more db nodes to fetch data results. +The `database_data` element can contain one or more objects reflecting each unique node: + +##### Single Node +``` + "database_data": { + "LocalNode": { + "help": "Settings for PGSQL Database (data)", + "node": "LocalNode", + "user": "homer_user", + "pass": "homer_password", + "name": "homer_data", + "host": "127.0.0.1" + } + } +``` +##### Multi Node +``` + "database_data": { + "LocalNode": { + "help": "Settings for PGSQL Database (data)", + "node": "LocalNode", + "user": "homer_user", + "pass": "homer_password", + "name": "homer_data", + "host": "127.0.0.1" + }, + "RemoteNode": { + "help": "Settings for PGSQL Database (data)", + "node": "RemoteNode", + "user": "homer_user", + "pass": "homer_password", + "name": "homer_data", + "host": "127.0.0.2" + } + } +``` From f1877293d884bdfd465fc6dd16baeb4bb10c41f7 Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 12:48:42 +0500 Subject: [PATCH 32/48] Create FAQ & Troubleshooting.md --- docs/FAQ & Troubleshooting.md | 366 ++++++++++++++++++++++++++++++++++ 1 file changed, 366 insertions(+) create mode 100644 docs/FAQ & Troubleshooting.md diff --git a/docs/FAQ & Troubleshooting.md b/docs/FAQ & Troubleshooting.md new file mode 100644 index 0000000..e4f0c2c --- /dev/null +++ b/docs/FAQ & Troubleshooting.md @@ -0,0 +1,366 @@ +[![HOMER](https://camo.githubusercontent.com/c287bf83f8d5969635b5bed047a3e70854bc1840/687474703a2f2f736970636170747572652e6f72672f646174612f696d616765732f736970636170747572655f6865616465722e706e67)](http://sipcapture.org) + +

+ +
+ + From 1d8f7b906a4852e9013a019d3ea53afa5cfa14a0 Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 12:49:42 +0500 Subject: [PATCH 33/48] Create Mailing-List.md --- docs/Mailing-List.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 docs/Mailing-List.md diff --git a/docs/Mailing-List.md b/docs/Mailing-List.md new file mode 100644 index 0000000..58ca38a --- /dev/null +++ b/docs/Mailing-List.md @@ -0,0 +1 @@ +https://groups.google.com/g/homer-discuss From 709de6d621be6bfa8fd4cec9067af26f9d1a1217 Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 12:52:06 +0500 Subject: [PATCH 34/48] Create CaptAgent.md --- docs/Hep-EEP Agent Examples/CaptAgent.md | 83 ++++++++++++++++++++++++ 1 file changed, 83 insertions(+) create mode 100644 docs/Hep-EEP Agent Examples/CaptAgent.md diff --git a/docs/Hep-EEP Agent Examples/CaptAgent.md b/docs/Hep-EEP Agent Examples/CaptAgent.md new file mode 100644 index 0000000..6b928b8 --- /dev/null +++ b/docs/Hep-EEP Agent Examples/CaptAgent.md @@ -0,0 +1,83 @@ +## CaptAgent: Installation + +This section provides guidance to download the latest code from our repository and compile it on your system. + + +### Requirements + +* Captagent 6.2+ requires ```libuv``` + + If your system does not provide ```libuv``` and ```libuv-dev```, please install from our +[repository](https://github.com/sipcapture/captagent/tree/master/dependency) or compile it from [source]( https://github.com/libuv/libuv/releases) + +##### Operating Systems +###### Debian 10 (buster): +``` +apt-get install libexpat-dev libpcap-dev libjson-c-dev libtool automake flex bison libgcrypt-dev libuv1-dev libpcre3-dev libfl-dev + +``` +###### Debian 9 (stretch): +``` +apt-get install libexpat1-dev libpcap-dev libjson-c-dev libtool automake flex bison libgcrypt11-dev libuv1-dev libpcre3-dev libfl-dev + +``` +###### Debian 8 (jessie): +``` +apt-get install libexpat-dev libpcap-dev libjson0-dev libtool automake flex bison libuv-dev libgcrypt11-dev libfl-dev +``` +###### Debian 7 (wheezy): +``` +wget https://github.com/sipcapture/captagent/raw/master/dependency/debian/wheezy/libuv_1.8.0-2_amd64.deb +dpkg -i libuv_1.8.0-2_amd64.deb + +apt-get install libexpat-dev libpcap-dev libjson0-dev libtool automake flex bison + +``` + +###### CentOS 7: +``` +yum -y install json-c-devel expat-devel libpcap-devel flex-devel automake libtool bison libuv-devel flex +``` + +###### CentOS 6: +``` +rpm -i https://github.com/sipcapture/captagent/raw/master/dependency/centos/6/libuv-1.8.0-1.el6.x86_64.rpm +rpm -i https://github.com/sipcapture/captagent/raw/master/dependency/centos/6/libuv-devel-1.8.0-1.el6.x86_64.rpm +yum -y install json-c-devel expat-devel libpcap-devel pcre-devel flex-devel automake libtool bison flex +``` + + + +### Clone & Compile +``` + cd /usr/src + git clone https://github.com/sipcapture/captagent.git captagent + cd captagent + ./build.sh + ./configure + make && make install +``` + +#### Build Options +| Name | Configure Flag | Libraries | +|--- |--- |--- | +| HEP Compression | --enable-compression | | +| IPv6 Support | --enable-ipv6 | | +| PCRE Support | --enable-pcre | libpcre | +| SSL Support | --enable-ssl | openssl | +| TLS Support | --enable-tls | libgcrypt20 openssl | +| MySQL Support | --enable-mysql | libmysqlclient | +| Redis Support | --enable-redis | libhiredis | + +-------------- + +#### TLS Support (experimental) + +To compile and enable TLS decryption features, please check the dedicated Wiki page. + +-------------- + + +Congratulations! You just installed your first basic instance of CaptAgent 6! + +#### Next: [Configure CaptAgent 6](https://github.com/sipcapture/captagent/wiki/Configuration) From 169cdbb206a29d1a34c09949b490532dd05c854a Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 12:52:50 +0500 Subject: [PATCH 35/48] Rename CaptAgent.md to CaptAgent.md --- .../CaptAgent.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename docs/{Hep-EEP Agent Examples => HEP-EEP Agent Examples}/CaptAgent.md (100%) diff --git a/docs/Hep-EEP Agent Examples/CaptAgent.md b/docs/HEP-EEP Agent Examples/CaptAgent.md similarity index 100% rename from docs/Hep-EEP Agent Examples/CaptAgent.md rename to docs/HEP-EEP Agent Examples/CaptAgent.md From 208ac64eaf1ffe3849b8b6e2f568d1d7bd88672d Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 12:53:47 +0500 Subject: [PATCH 36/48] Create Heplify.md --- docs/HEP-EEP Agent Examples/Heplify.md | 262 +++++++++++++++++++++++++ 1 file changed, 262 insertions(+) create mode 100644 docs/HEP-EEP Agent Examples/Heplify.md diff --git a/docs/HEP-EEP Agent Examples/Heplify.md b/docs/HEP-EEP Agent Examples/Heplify.md new file mode 100644 index 0000000..4b3d156 --- /dev/null +++ b/docs/HEP-EEP Agent Examples/Heplify.md @@ -0,0 +1,262 @@ + + + + + + +heplify is captagents little brother, optimized for speed and simplicity. It's a single binary which you can run +on Linux, ARM, MIPS, Windows to capture IPv4 or IPv6 packets and send them to Homer. Heplify is able to send +SIP, correlated RTCP, RTCPXR, DNS, Logs into homer. +It's able to handle fragmented and duplicate packets out of the box. + +## Requirements + +### Linux + +None if you use the binary from the releases + +### Windows + +[WinPcap](https://www.winpcap.org/install/default.htm) + +## Installation + +### Linux + +Download [heplify](https://github.com/sipcapture/heplify/releases) and execute 'chmod +x heplify' + +### Windows + +Download [heplify.exe](https://github.com/sipcapture/heplify/releases) + +### Development build + +If you have Go 1.18+ installed, build the latest heplify binary by running `make`. + +Now you should install LUA Jit: + +* Compile from sources: + + Install luajit dev libary + + `apt-get install libluajit-5.1-dev` + + or + + `yum install luajit-devel` + + or for macOS + + ```sh + # Assuming brew installs to /usr/local/ + brew install lua@5.1 luajit + ln -s /usr/local/lib/pkgconfig/luajit.pc /usr/local/lib/pkgconfig/luajit-5.1.pc + export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig/ + ``` + + [install](https://golang.org/doc/install) Go 1.11+ + + `go build cmd/heplify/heplify.go` + + + +### Docker + +You can also build a docker image: + +```bash +docker build --no-cache -t sipcapture/heplify:latest -f docker/heplify/Dockerfile . +``` + +You can use the image using docker compose: + +``` + heplify: + image: sipcapture/heplify:latest + user: 1000:1000 + cap_add: + - CAP_NET_ADMIN + - CAP_NET_RAW + command: + ./heplify -e -hs ${HOMER_DST}:9060 -m SIP -dd -zf -l info + network_mode: host + restart: unless-stopped +``` + +## Usage + +```bash + -assembly_debug_log + If true, the github.com/google/gopacket/tcpassembly library will log verbose debugging information (at least one line per packet) + -assembly_memuse_log + If true, the github.com/google/gopacket/tcpassembly library will log information regarding its memory use every once in a while. + -b int + Interface buffersize (MB) (default 32) + -bpf string + Custom BPF to capture packets + -collectonlysip + collect only sip + -d string + Enable certain debug selectors [defrag,layer,payload,rtp,rtcp,sdp] + -dd + Deduplicate packets + -di string + Discard uninteresting packets by any string + -didip string + Discard uninteresting SIP packets by Destination IP(s) + -diip string + Discard uninteresting SIP packets by Source or Destination IP(s) + -dim string + Discard uninteresting SIP packets by Method [OPTIONS,NOTIFY] + -disip string + Discard uninteresting SIP packets by Source IP(s) + -e + Log to stderr and disable syslog/file output + -eof-exit + Exit once done reading pcap file + -erspan + erspan + -fg uint + Fanout group ID for af_packet + -fi string + Filter interesting packets by any string + -fnum int + The total num of log files to keep (default 7) + -fsize uint + The rotate size per log file based on byte (default 10485760) + -fw int + Fanout worker count for af_packet (default 4) + -hep-buffer-activate + enable buffer messages if connection to HEP server broken + -hep-buffer-debug + enable debug buffer messages + -hep-buffer-file string + filename and location for hep-buffer file (default "HEP-Buffer.dump") + -hep-buffer-max-size string + max buffer size, can be B, KB, MB, GB, TB. By default - unlimited (default "0") + -hi uint + HEP node ID (default 2002) + -hin + HEP collector listening protocol, address and port (example: "tcp:10.10.99.10:9060") + -hn string + HEP node Name + -hp string + HEP node PW + -hs string + HEP server destination address and port (default "127.0.0.1:9060") + -i string + Listen on interface (default "any") + -keepalive uint + keep alive internal - 5 seconds by default. 0 - disable (default 5) + -l string + Log level [debug, info, warning, error] (default "info") + -lp int + Loop count over ReadFile. Use 0 to loop forever (default 1) + -m string + Capture modes [SIP, SIPDNS, SIPLOG, SIPRTCP] (default "SIPRTCP") + -n string + Log filename (default "heplify.log") + -nt string + Network types are [udp, tcp, tls] (default "udp") + -bpf string + Custom bpf filter (default "") + -o + Read packet for packet + -p string + Log filepath (default "./") + -pr string + Portrange to capture SIP (default "5060-5090") + -prometheus string + prometheus metrics - ip:port. By default all IPs (default ":8090") + -protobuf + Use Protobuf on wire + -rf string + Read pcap file + -rs + Use packet timestamps with maximum pcap read speed + -rt int + Pcap rotation time in minutes (default 60) + -s int + Snaplength (default 8192) + -script-file string + Script file to execute on each packet + -script-hep-filter string + HEP filter for script, comma separated list of HEP types (default "1") + -sipassembly + If true, sipassembly will be enabled + -skipverify + skip certifcate validation + -sl + Log to syslog + -t string + Capture types are [pcap, af_packet] (default "pcap") + -tcpassembly + If true, tcpassembly will be enabled + -tcpsendretries uint + Number of retries for sending before giving up and reconnecting (default 64) + -version + Show heplify version + -vlan + vlan + -wf string + Path to write pcap file + -zf + Enable pcap compression + -script-file string + LUA script file path to execute on each packet + -script-hep-filter string + HEP Type filter for LUA script, comma separated list (default "1") + +``` + +## Examples + +```bash +# Capture SIP and RTCP packets on any interface and send them to 127.0.0.1:9060 +./heplify + +# Capture SIP and RTCP packets on any interface and send them via TLS to 192.168.1.1:9060 +./heplify -hs 192.168.1.1:9060 -nt tls + +# Capture SIP and RTCP packets on any interface and send them to 192.168.1.1:9060. Use a someNodeName +./heplify -hs 192.168.1.1:9060 -hn someNodeName + +# Capture SIP and RTCP packets on any interface and send them to 192.168.1.1:9060. Print info to stdout +./heplify -hs 192.168.1.1:9060 -e + +# Capture SIP and RTCP packets on any interface and send them to 192.168.1.1:9060 and 192.168.2.2:9060 +./heplify -hs "192.168.1.1:9060,192.168.2.2:9060" + +# Capture SIP and RTCP packets on any interface and send them to 192.168.1.1:9060. Print debug selectors +./heplify -hs 192.168.1.1:9060 -e -d fragment,payload,rtcp + +# Capture SIP and RTCP packets with custom SIP port range on eth2 and send them to 192.168.1.1:9060 +./heplify -i eth2 -pr 6000-6010 -hs 192.168.1.1:9060 + +# Capture SIP and RTCP packets on eth2, send them to homer and compressed to /srv/pcapdumps/ +./heplify -i eth2 -hs 192.168.1.1:9060 -wf /srv/pcapdumps/ -zf + +# Read example/rtp_rtcp_sip.pcap and send SIP and correlated RTCP packets to 192.168.1.1:9060 +./heplify -rf example/rtp_rtcp_sip.pcap -hs 192.168.1.1:9060 + +# Capture and send packets except SIP OPTIONS and NOTIFY to 192.168.1.1:9060 +./heplify -hs 192.168.1.1:9060 -dim OPTIONS,NOTIFY + +# Capture SIP packet with HPERM encapsulation on port 7932 and interface eth2, send to 192.168.1.1:9060 and print debug info on stdout +./heplify -i eth2 -bpf "port 7932" -hs 192.168.1.1:9060 -l debug -e + +# Capture SIP packet with VXLAN encapsulation on port 4789 and interface eth0, send to 192.168.1.1:9060 and print debug info on stdout +./heplify -i eth0 -bpf "port 4789" -hs 192.168.1.1:9060 -l debug -e + +# Run heplify in "HEP Collector" mode in order to receive HEP input via TCP on port 9060 and fork (output) to two HEP servers listening on port 9063 +./heplify -e -hs HEPServer1:9063,HEPserver2:9063 -hin tcp:1.2.3.4:9060 + + +``` +### Made by Humans + +This Open-Source project is made possible by actual Humans without corporate sponsors, angels or patreons. + +If you use this software in production, please consider supporting its development with contributions or [donations](https://www.paypal.com/cgi-bin/webscr?cmd=_donations&business=donation%40sipcapture%2eorg&lc=US&item_name=SIPCAPTURE&no_note=0¤cy_code=EUR&bn=PP%2dDonationsBF%3abtn_donateCC_LG%2egif%3aNonHostedGuest) + +[![Donate](https://www.paypalobjects.com/en_US/i/btn/btn_donateCC_LG.gif)](https://www.paypal.com/cgi-bin/webscr?cmd=_donations&business=donation%40sipcapture%2eorg&lc=US&item_name=SIPCAPTURE&no_note=0¤cy_code=EUR&bn=PP%2dDonationsBF%3abtn_donateCC_LG%2egif%3aNonHostedGuest) From 1097d5d5fbae7c10a7562255db9242d3da845706 Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 12:54:30 +0500 Subject: [PATCH 37/48] Create Kamailio.md --- docs/HEP-EEP Agent Examples/Kamailio.md | 283 ++++++++++++++++++++++++ 1 file changed, 283 insertions(+) create mode 100644 docs/HEP-EEP Agent Examples/Kamailio.md diff --git a/docs/HEP-EEP Agent Examples/Kamailio.md b/docs/HEP-EEP Agent Examples/Kamailio.md new file mode 100644 index 0000000..dbd2959 --- /dev/null +++ b/docs/HEP-EEP Agent Examples/Kamailio.md @@ -0,0 +1,283 @@ + +![](http://kb.asipto.com/images/kamailio.jpg) + + + +### KAMAILIO 4.x Trace Node + +SIP capture functionalities are built into core kamailio. We have to only load required module, initialize it with the appropriate parameters and modify routing logic to use it. To keep the changes flexible and clean, this excample uses directives which allow us to simply switch on/off the additional functionality: + +##### kamailio.cfg +First, define WITH_HOMER directive at the head of your script: +``` +#!define WITH_HOMER +``` + +Next, move to the loadparam section of the script, and add a condition for siptrace: +``` +#!ifdef WITH_HOMER +loadmodule "siptrace.so" +#!endif +``` + +_NOTE: siptrace module MUST be loaded after loading mysql and tm modules!_ + +Next, configure the basic parameters for the module: +``` +#!ifdef WITH_HOMER +# check IP and port of your capture node +modparam("siptrace", "duplicate_uri", "sip:10.0.0.1:9060") +modparam("siptrace", "hep_mode_on", 1) +modparam("siptrace", "trace_to_database", 0) +modparam("siptrace", "trace_flag", 22) +modparam("siptrace", "trace_on", 1) +#!endif +``` + + +Finally, use SIPTRACE in your ```route {}``` logic where needed: + +``` +#!ifdef WITH_HOMER + #start duplicate the SIP message now + sip_trace(); + setflag(22); +#!endif +``` + +### KAMAILIO 5.x Trace Node + +##### kamailio.cfg +First, define WITH_HOMER directive at the head of your script: +``` +#!define WITH_HOMER +``` + +Next, move to the loadparam section of the script, and add a condition for siptrace: +``` +#!ifdef WITH_HOMER +loadmodule "siptrace.so" +#!endif +``` + +_NOTE: siptrace module MUST be loaded after loading mysql and tm modules!_ + +Next, configure the basic parameters for the module: +``` +#!ifdef WITH_HOMER +# check IP and port of your capture node +modparam("siptrace", "duplicate_uri", "sip:10.0.0.1:9060") +# Send from an IP +modparam("siptrace", "send_sock_addr", "sip:10.2.0.2:5000") +modparam("siptrace", "hep_mode_on", 1) +modparam("siptrace", "trace_to_database", 0) +modparam("siptrace", "trace_flag", 22) +modparam("siptrace", "trace_on", 1) +#!endif +``` + +Finally, use 'sip_trace' in your route {} logic where needed: + + +``` +#!ifdef WITH_HOMER + setflag(22); + + #start duplication mode: m or M for message; t or T for transaction; d or D for dialog + sip_trace_mode("t"); + + #start duplicate the SIP message now + sip_trace(); + + # Or you can use new syntax, if you want to have multipe copy of your data + sip_trace("sip:10.0.0.2:9060"); + + # send to 10.0.0.3 and assign a callid as correlation param + sip_trace("sip:10.0.0.3:9060", "$ci-abc"); + + # or by dialog: trace current dialog; needs to be done on initial INVITE and dialog has to be loaded + sip_trace("sip:10.0.0.4:9060", "$ci-abc", "d"); + + #Send HEPLog: + hlog("$hdr(P-MyID)", "Another one with a custom correlation ID"); + +#!endif +``` From a4aabf69ddd7cdb3d0754d4fc25b9c6545ccc774 Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 12:55:28 +0500 Subject: [PATCH 38/48] Create OpenSIPS.md --- docs/HEP-EEP Agent Examples/OpenSIPS.md | 184 ++++++++++++++++++++++++ 1 file changed, 184 insertions(+) create mode 100644 docs/HEP-EEP Agent Examples/OpenSIPS.md diff --git a/docs/HEP-EEP Agent Examples/OpenSIPS.md b/docs/HEP-EEP Agent Examples/OpenSIPS.md new file mode 100644 index 0000000..650a02a --- /dev/null +++ b/docs/HEP-EEP Agent Examples/OpenSIPS.md @@ -0,0 +1,184 @@ +![](http://i.imgur.com/MoOOI23.png) + +## OpenSIPS 2.2 as HEP/EEP Switch +Please see the [dedicated wiki page](https://github.com/sipcapture/homer/wiki/Examples%3A-opensips-hepswitch) +## OpenSIPS as Homer Capture server + +Homer's sipcapture module allows OpenSIPS to operate as a robust and scalable SIP sampling/capture server +with native support for HEPv1/v2, IPIP Encapsulation protocols and switch mirroring/monitoring port +traffic. OpenSIPS can be configured either as Capture Agent (siptrace module) sampling and forwarding packets, or as Capture Node (sipcapture module) collecting, indexing and storing SIP packets as received +from the available Capture Agents (HEP), SBCs (IPIP) or directly from the ethernet wire. + +Capture Agents can be distributed in a modular fashion, allowing support for any network topology. +In addition to the integrated sampling and capturing functions in OpenSIPS, a stand-alone capture +agent (captagent) is provided enabling HEP encapsulation for unsupported systems and soft-switches. +OpenSER/Kamailio and FreeSWITCH users already enjoy integrated HEP mirroring functionality. + +![](http://homer.googlecode.com/files/HOMER-OpenSIPS.png) + +### OpenSIPs - config for capture node. + +``` +####### Global Parameters ######### + +debug=3 +log_stderror=no +log_facility=LOG_LOCAL0 + +fork=yes +children=5 + +disable_tcp=yes +db_default_url="mysql://opensips:opensipsrw@localhost/opensips" +port=9060 + +/* uncomment and configure the following line if you want opensips to + bind on a specific interface/port/proto (default bind on all available) */ +listen=udp:10.0.0.1:9060 + +####### Modules Section ######## +#set module path +mpath="/usr/local/lib64/opensips/modules/" + +loadmodule "db_mysql.so" +loadmodule "sipcapture.so" + +####### Routing Logic ######## +modparam("sipcapture", "db_url", "mysql://homer:password@localhost/homer_db") +modparam("sipcapture", "capture_on", 1) +modparam("sipcapture", "table_name", "sip_capture") +/* activate HEP capturing */ +modparam("sipcapture", "hep_capture_on", 1) + +/* configuration for Mirroring PORT */ +modparam("sipcapture", "raw_socket_listen", "10.0.130.41:5060-6000") +modparam("sipcapture", "raw_interface", "eth1") +/* activate monitoring port capturing */ +modparam("sipcapture", "raw_moni_capture_on", 1) +modparam("sipcapture", "raw_sock_children", 4) +/* Promiscious mode */ +modparam("sipcapture", "promiscious_on", 1) + +####### Routing Logic ######## + + +# main request routing logic + +# Main SIP request routing logic +# - processing of any incoming SIP request starts with this route +route { + #For example, you can capture only needed methods... + #if (!(method =~ "^(OPTIONS|NOTIFY|SUBSCRIBE)$"))) { + sip_capture(); + #} + drop; +} + +onreply_route { + + #And only needed reply or needed requests method + #if(status =~ "^(1[0-9][0-9]|[3[0-9][0-9]|4[0-9]|[56][0-9][0-9])") { + #if(!($rm =~ "^(NOTIFY|SUBSCRIBE|OPTIONS|)$")) { + sip_capture(); + #} + drop; +} + + + +``` + +### OpenSIPs - config for trace node. + +``` + +####### Global Parameters ######### +debug=3 +log_stderror=no +log_facility=LOG_LOCAL0 +fork=yes +children=4 +disable_tcp=yes +db_default_url="mysql://opensips:opensipsrw@localhost/opensips" + +port=5060 + +listen=udp:192.168.0.1:5060 + + +####### Modules Section ######## + +#set module path +mpath="/usr/local/lib64/opensips/modules/" + +loadmodule "signaling.so" +loadmodule "sl.so" +loadmodule "tm.so" +loadmodule "rr.so" +loadmodule "maxfwd.so" +loadmodule "textops.so" +loadmodule "mi_fifo.so" +loadmodule "siptrace.so" + + +# ----------------- setting module-specific parameters --------------- + +# ----- mi_fifo params ----- +modparam("mi_fifo", "fifo_name", "/tmp/opensips_fifo") +modparam("rr", "append_fromtag", 0) +modparam("siptrace", "duplicate_uri", "sip:10.0.0.1:9060") +modparam("siptrace", "duplicate_with_hep", 1) +modparam("siptrace", "trace_to_database", 0) +modparam("siptrace", "trace_flag", 22) +modparam("siptrace", "trace_on", 1) +#HEPv2 == timestamp will be included to HEP header +modparam("siptrace", "hep_version", 2) + + Main SIP request routing logic +# - processing of any incoming SIP request starts with this route +route { + + ...... + + setflag(22); + + #Duplicate this sip message to Homer capture node + sip_trace(); + + ..... + route(1); +} + +route[1] { + + if (!t_relay()) { + sl_reply_error(); + }; + exit; +} + +``` + +---- + + + +### SIPTRACE on OpenSIPS 1.7.x: + +To enable HEP support in siptrace/opensips 1.7.x do the following: + +1. Download OpenSIPS SVN head + +2. Copy modules/siptrace and modules/sipcapture to the 1.7.1 source tree + +3. Modify modules/siptrace/siptrace.c as follows: +``` + - if (dlgb.create_dlg(msg,0)<1) { + + if (dlgb.create_dlg(msg)<1) { +``` +4. Recompile the siptrace module: +``` + # make modules=modules/siptrace modules +``` + +_NOTE: Module SIPCAPTURE is only used to compile module SIPTRACE and CANNOT be used on OpenSIPS 1.7.x (for Homer Capture always use TRUNK)_ From 497fee542ef0ade71256953aae3f2078db26c248 Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 12:56:15 +0500 Subject: [PATCH 39/48] Create FreeSwitch.md --- docs/HEP-EEP Agent Examples/FreeSwitch.md | 77 +++++++++++++++++++++++ 1 file changed, 77 insertions(+) create mode 100644 docs/HEP-EEP Agent Examples/FreeSwitch.md diff --git a/docs/HEP-EEP Agent Examples/FreeSwitch.md b/docs/HEP-EEP Agent Examples/FreeSwitch.md new file mode 100644 index 0000000..985f449 --- /dev/null +++ b/docs/HEP-EEP Agent Examples/FreeSwitch.md @@ -0,0 +1,77 @@ +![FreeSwitch](http://i.imgur.com/gylbBjz.png) +## FreeSWITCH Capture Agent + +Freeswitch ships with an integrated HEP Capture Agent designed to work with HOMER + +**FreeSwitch HEP3/EEP support is available in 1.6.8+ ** + +### Global Configuration +To enable HEP capturing, open sofia.conf.xml and set capture-server param + +``` + +``` + +Freeswitch >= 1.7 has support for HEPv2 and HEPv3. The new syntax is: + +``` + +``` +open internal.xml and external.xml and change sip-capture param to "yes". Also please do it on each profile in your sip_profiles/internal and sip_profiles/external (/etc/freeswitch/sip_profiles) + +``` + +``` +*note: the ip address and port must be same as the listen param in your kamailio.cfg or in heplify-server * + +---- + +To enable/disable the HEP agent on demand, you can use CLI commands: +``` +freeswitch@fsnode04> sofia global capture on + ++OK Global capture on +freeswitch@fsnode04> sofia global capture off + ++OK Global capture off +``` + +### Profile Configuration + +You can choose to activate HEP capturing only for a specific profile: +``` +freeswitch@fsnode04> sofia profile internal capture on + +Enabled sip capturing on internal + +freeswitch@fsnode04> sofia profile internal capture off + +Disabled sip capturing on internal +``` + +### B2BUA Correlation +To correlate B2BUA legs set the following before bridging the second leg: +``` + +``` + +Next, configure `heplify-server` to extract correlation from the newly created `X-cid` header _(TODO: link)_ + +### ESL Integration (beta) + +[hepipe.js](https://github.com/sipcapture/hepipe.js/blob/master/esl) provides **experimental** support for FreeSWITCH ESL integration for call quality reports feeding to HOMER 5, effectively providing external HEP3/EEP features with correlation support. + +#### Events + +| ESL Event | Hep Mode | HEP Type | +|:--|:--|:--| +| CHANNEL_CREATE | LOG | 100 | +| CHANNEL_ANSWER | LOG | 100 | +| CHANNEL_DESTROY | LOG | 100 | +| CUSTOM | LOG | 100 | +| RECV_RTCP_MESSAGE | RTCP | 5 | +| CHANNEL_DESTROY | CUSTOM QoS | 99 | + +For full instructions and details please checkout [hepipe.js](https://github.com/sipcapture/hepipe.js/blob/master/esl) + +If you test or extend this feature please share your feedback! From fe7903708d26df955a9f880d0ae65ceca8a451be Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 12:57:10 +0500 Subject: [PATCH 40/48] Create Asterisk.md --- docs/HEP-EEP Agent Examples/Asterisk.md | 74 +++++++++++++++++++++++++ 1 file changed, 74 insertions(+) create mode 100644 docs/HEP-EEP Agent Examples/Asterisk.md diff --git a/docs/HEP-EEP Agent Examples/Asterisk.md b/docs/HEP-EEP Agent Examples/Asterisk.md new file mode 100644 index 0000000..d90a5af --- /dev/null +++ b/docs/HEP-EEP Agent Examples/Asterisk.md @@ -0,0 +1,74 @@ +# ![](http://www.asterisk.org/sites/asterisk/themes/asterisk/logo.png) + + +Asterisk 12+ ships with HEP encapsulation support (res_hep) and is able to natively mirror its packets to a HEP/EEP Collector such as HOMER. When using ```chan_pjsip``` enabling HEP features is as simple as configuring ```/etc/asterisk/hep.conf``` + +### Requirements + +* loading ```res_hep_pjsip.so``` module + +### Example Configuration: +

+/etc/asterisk/hep.conf
+
+;
+; res_hep Module configuration for Asterisk
+;
+
+; All settings are currently set in the general section.
+[general]
+enabled = yes 
+; Enable/disable forwarding of packets to a
+; HEP server. Default is "yes".
+capture_address = 10.0.0.1:9060 
+; The address of the HEP capture server.
+capture_password = foo 
+; If specified, the authorization passsword
+; for the HEP server. If not specified, no
+; authorization password will be sent.
+capture_id = 1234 
+; A unique integer identifier for this
+; server. This ID will be embedded sent
+; with each packet from this server.
+
+ +Once configured and enabled, the module will begin forwarding all handled SIP packets to HOMER for handling. + + +---------- + +# RTCP statistics + +Asterisk 12+ ships with _res_hep_rtcp_. The module subscribes to Stasis and receives RTCP information back from the message bus, which it encodes into HEPv3 packets and sends to the res_hep module for transmission. Using this module, someone with a Homer server can get live call quality monitoring for all channels in their Asterisk 12+ systems. + +## Enable +To enable the functionality, just load the module alongside the [[res_hep|Examples:-Asterisk]] module + +------------ + +The module supports sender (SR) and receiver (RR) reports: + +* rtcp-sender: Audio is streamed from the first instance and echo'd back from the second to the first. This results in RTCP information from both instances being sent to a HEP server, where both sides are 'senders' and thus generate/receive SR packets. +* rtcp-receiver: Audio is streamed from the first instance and absorbed by the second. This results in RTCP information from both instances being sent to a HEP server, where the first transmits SR packets and received RR packets and the second transmits RR packets and received SR packets. + + +------------- + +# PJSIP X-CID Correlation for BLEG +``` +PJSIP_HEADER(add,X-CID)=$SIPCALLID +``` + +# CDR Correlation Example + +##### /etc/asterisk/extensions.conf +``` +exten => s,1,Set(CDR(callid)=${SIPCALLID}) +exten => s,2,Set(CDR(rtcpinfo)=${RTPAUDIOQOS}) +``` + +##### /etc/asterisk/cdr_custom.conf +``` +[mappings] +Master.csv => ${CSV_QUOTE(${CDR(clid)})},${CSV_QUOTE(${CDR(src)})},${CSV_QUOTE(${CDR(dst)})},${CSV_QUOTE(${CDR(dcontext)})},${CSV_QUOTE(${CDR(channel)})},${CSV_QUOTE(${CDR(dstchannel)})},${CSV_QUOTE(${CDR(lastapp)})},${CSV_QUOTE(${CDR(lastdata)})},${CSV_QUOTE(${CDR(start)})},${CSV_QUOTE(${CDR(answer)})},${CSV_QUOTE(${CDR(end)})},${CSV_QUOTE(${CDR(duration)})},${CSV_QUOTE(${CDR(billsec)})},${CSV_QUOTE(${CDR(disposition)})},${CSV_QUOTE(${CDR(amaflags)})},${CSV_QUOTE(${CDR(accountcode)})},${CSV_QUOTE(${CDR(uniqueid)})},${CSV_QUOTE(${CDR(userfield)})},${CDR(sequence)},${CDR(callid)},${CDR(rtcpinfo)} +``` From f7c13d07551aea185b364fb7ddf7f64bb5735436 Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 12:57:50 +0500 Subject: [PATCH 41/48] Create sipgrep.md --- docs/HEP-EEP Agent Examples/sipgrep.md | 40 ++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100644 docs/HEP-EEP Agent Examples/sipgrep.md diff --git a/docs/HEP-EEP Agent Examples/sipgrep.md b/docs/HEP-EEP Agent Examples/sipgrep.md new file mode 100644 index 0000000..a6da844 --- /dev/null +++ b/docs/HEP-EEP Agent Examples/sipgrep.md @@ -0,0 +1,40 @@ +#SIPGREP +### Using Sipgrep as disposable HEP3 Agent + +[Sipgrep](https://github.com/sipcapture/sipgrep) is a modern pcap-aware tool command line tool to capture, filter, display and help troubleshoot SIP signaling over IP networks, allowing the user to specify extended regular expressions matching against SIP headers and with nifty extra features. + +[Sipgrep](https://github.com/sipcapture/sipgrep) is able to act as a quick on-demand HEP3 capture agent and forward packets to a collector very easily when a simple terminal check does not suffice. In the following example, Sipgrep is used to display the traffic of interest as well as log it to a remote location, useful for instance when troubleshooting issues on hosted platforms or disposable instances on the cloud. + + + +#### HEP3 Example: +Display dialogs and duplicate all traffic to HOMER sipcapture in HEPv3: +``` +sipgrep -f 23333 -H udp:10.0.0.1:9061 +``` + +#### General Examples: +``` +# Find a dialog there From user contains '2323232' +sipgrep -f 2323232 +# Find a dialog there To user contains '1111' and print dialog report +sipgrep -f 1111 -G +# Display only 603 replies without dialog match +sipgrep '^SIP/2.0 603' -m +# Display only OPTIONS and NOTIFY requests +sipgrep '^(OPTIONS|NOTIFY)' +# Display only SUBSCRIBE dialog +sipgrep 'CSeq:\s?\d* (SUBSCRIBE|PUBLISH|NOTIFY)' -M +# Collect all messages while pcap_dump smaller than 20kb +sipgrep -q 'filesize:20' -O sipgrep.pcap +# Kill friendly-scanner automatically +sipgrep -J +# Kill friendly-scanner with custom UAC name +sipgrep -j sipvicious +# Collect all Calls/Registrations dialogs during 120 seconds, print reports and exit: +sipgrep -g -G -q 'duration:120' +# Split pcap_dump to 20 KB files in sipgrep_INDEX_YYYYMMDDHHMM pcap +sipgrep -Q 'filesize:20' -O sipgrep.pcap +# Split pcap_dump in sipgrep_INDEX_YYYYMMDDHHMM.pcap each 120 seconds +sipgrep -Q 'duration:120' -O sipgrep.pcap +``` From 0b228612c2596118fbc5115dfab9b923fac5f5a6 Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 12:58:21 +0500 Subject: [PATCH 42/48] Create sngrep.md --- docs/HEP-EEP Agent Examples/sngrep.md | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) create mode 100644 docs/HEP-EEP Agent Examples/sngrep.md diff --git a/docs/HEP-EEP Agent Examples/sngrep.md b/docs/HEP-EEP Agent Examples/sngrep.md new file mode 100644 index 0000000..8aad01f --- /dev/null +++ b/docs/HEP-EEP Agent Examples/sngrep.md @@ -0,0 +1,24 @@ +![](https://camo.githubusercontent.com/fbfe5cb64b1df2bf64f0d0616a24c36bfa17220b/687474703a2f2f69726f6e7465632e6769746875622e696f2f736e677265702f696d616765732f6c6f676f2e706e67) + +# sngrep +Irontec's awesome *sngrep* 1.x+ introduces command line option (-H) and settings (eep.send) to send capture data in HEP/EEP to Homer and to run headless as a capture agent: + +* ```-H or --eep-send```: Send captured data to other Homer (udp:10.10.10.10:9060) +* ```-N or --no-interface```: Don't display sngrep interface, just capture +* ```-q or --quiet```: Don't print captured dialogs in no interface mode + +### Example: SIP +Mirror all SIP packets from all devices with src||dst port 5060 to Homer +
+sngrep port 5060 -H udp:10.10.10.10:9060 --no-interface -q
+
+ +### Example: TLS +Mirror all SIP/TLS packets from all devices with src||dst portrange 5060-5061 to Homer +
+sngrep portrange 5060-5061 -k ./privkey.pem -H udp:10.10.10.10:9060 --no-interface -q
+
+ + + +For further information please visit the [sngrep](https://github.com/irontec/sngrep/wiki#how-to-use) wiki. From 092326cbbc3ddda67e272c8f740ac20c3e573c65 Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 12:59:06 +0500 Subject: [PATCH 43/48] Create RTPEngine.md --- docs/HEP-EEP Agent Examples/RTPEngine.md | 74 ++++++++++++++++++++++++ 1 file changed, 74 insertions(+) create mode 100644 docs/HEP-EEP Agent Examples/RTPEngine.md diff --git a/docs/HEP-EEP Agent Examples/RTPEngine.md b/docs/HEP-EEP Agent Examples/RTPEngine.md new file mode 100644 index 0000000..97b9c31 --- /dev/null +++ b/docs/HEP-EEP Agent Examples/RTPEngine.md @@ -0,0 +1,74 @@ +![](https://www.sipwise.com/wp-content/themes/sipwise/assets/images/logo.svg) + +RTPEngine +======================= + +The [Sipwise](http://www.sipwise.com/) NGCP rtpengine is a proxy for RTP traffic and other UDP based +media traffic. It's meant to be used with the [Kamailio SIP proxy](http://www.kamailio.org/) and [OpenSIPS SIP proxy](http://www.opensips.org/) and forms a drop-in replacement for any of the other available RTP and media proxies. + +RTPEngine mr4.4.1+ supports [HEP3](http://hep.sipcapture.org) Encapsulation and can mirror RTCP packets relayed between streams to **HOMER** complete with SIP correlation Call-IDs from the respective signaling session. + +#### NGCP Users +Users of the NGCP platform should enable HOMER support directly in their ```config.yml``` replacing the destination parameter with the correct HOMER HEP socket and applying with ```ngcpcfg apply``` +``` +... +homer_rtcp_stats: + destination: 10.20.30.40:9070 + enabled: yes + id: '2001' + protocol: udp +... +``` + + +##### Parameters +``` + --homer=IP46:PORT Address of Homer server for RTCP stats + --homer-protocol=udp|tcp Transport protocol for Homer (must be defined) + --homer-id=INT 'Capture ID' to use within the HEP protocol +``` + +### Setup +Configure RTPEngine to send RTCP reports to Homer using the same HEP settings you used in your Kamailio/OpenSIPS/Captagent configuration to achieve SIP session correlation: + +![](http://i.imgur.com/cWB7eWh.png) + +##### Example HEP Settings (command line) +``` + --homer=10.0.0.20:9060 + --homer-protocol=udp + --homer-id=999 +``` + +``` +/usr/sbin/rtpengine --interface=10.0.0.1 --listen-tcp=25060 \ +--listen-udp=12222 --listen-ng=22222 --listen-cli=9900 --timeout=60 \ +--silent-timeout=3600 --pidfile=/var/run/ngcp-rtpengine-daemon.pid \ +--table=0 --log-level=6 --log-facility=daemon --homer=10.0.0.20:9060 \ +--homer-id=999 --homer-protocol=udp +``` + +##### Example HEP Settings (rtpengine.default) +``` +HOMER=10.0.0.20:9060 +HOMER_PROTOCOL=udp +HOMER_ID=2099 +``` + +##### Example RTCP Report +``` +{"sender_information": +{"ntp_timestamp_sec":3667834977,"ntp_timestamp_usec":2355549043, +"octets":82240,"rtp_timestamp":84679,"packets":514}, +"ssrc":3724882677,"type":200,"report_count":1,"report_blocks": +[{"source_ssrc":2062957521,"highest_seq_no":4663,"fraction_lost":0, +"ia_jitter":13,"packets_lost":0,"lsr":3093100159,"dlsr":296222}], +"sdes_ssrc":3724882677,"sdes_report_count":1,"sdes_information": [] } +``` + +##### Example QoS Report + +![](https://camo.githubusercontent.com/6394f82d5a0511085f4f6980e2b31ca6fe334e38/687474703a2f2f692e696d6775722e636f6d2f38784b62513837672e706e67) + + +
From 0a37f192f342c53d604f71d78ea22a30e91c62b5 Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 12:59:46 +0500 Subject: [PATCH 44/48] Create ACME Packet.md --- docs/HEP-EEP Agent Examples/ACME Packet.md | 193 +++++++++++++++++++++ 1 file changed, 193 insertions(+) create mode 100644 docs/HEP-EEP Agent Examples/ACME Packet.md diff --git a/docs/HEP-EEP Agent Examples/ACME Packet.md b/docs/HEP-EEP Agent Examples/ACME Packet.md new file mode 100644 index 0000000..2df11cc --- /dev/null +++ b/docs/HEP-EEP Agent Examples/ACME Packet.md @@ -0,0 +1,193 @@ +## ORACLE/ACME Packet & Homer + + +### IPIP (obsolete) + +ORACLE / ACME ACLI provides a *"packet-trace"* command which can capture [RFC2003](https://tools.ietf.org/html/rfc2003) *(IP Encapsulation within IP)* SIP signaling on the Net-Net SBC and natively mirror it to your Homer SIP Capture server. + +To configure, we need to create a capture receiver: +``` +capture-receiver + state enabled + ip-address 192.168.0.1 + network-interface m00:0 + last-modified-by engineer@192.168.30.1 + last-modified-date 2011-10-10 00:00:01 +``` + +or alternatively: + +``` +packet-trace-config < state | address | network-interface | select| no | show | done | exit> +``` + +The *"packet-trace"* command on the Net-Net SBC can now duplicate all packets sent to and from the endpoint identified by the IP address on the specified Net-Net SBC network interface: +``` +#packet-trace start Access:0 192.168.30.5 5060 5060 +``` + +If you access and network side present different *Call-ID* header breaking call-flow correlation, you can use an *HMR* to inject a X-CID: header containing the Call-ID from A-leg into INVITE in your B-leg; + +##### *Fictional HMR Example:* + +``` +sip-manipulation + name xCID + description Create_XCID + split-headers + join-headers + header-rule + name XCIDstore + header-name Call-ID + action store + comparison-type pattern-rule + msg-type out-of-dialog + methods INVITE + match-value .* + new-value + header-rule + name DeleteXCID + header-name X-CID + action delete + comparison-type pattern-rule + msg-type request + methods INVITE + match-value .* + new-value + header-rule + name addXCID + header-name X-CID + action add + comparison-type boolean + msg-type out-of-dialog + methods INVITE + match-value !$XCIDstore + new-value $XCIDstore.$0 +``` + + +--------------------- + +## CDR QoS Correlation +The following table provides a general mapping convention for ACME/ORACLE SBC CDRs + +###Leg 1 Voice Quality Report +``` +| QoS Metric | CDR Parameter| +| ------------- |:------------:| +|"POST_DIAL_DELAY"|Acme-Post-Dial-Delay (PDD MAP)| +|"DISCONNECT_INIT"|Acme-Disconnect-Initiator (INIT MAP)| +|"DISCONNECT_CAUSE"|Acme-Disconnect-Cause (CAUSE MAP)| +|"LAST_STATUS"|Acme-Session-Disposition (STATUS MAP)| +|"FIRST_SWITCHED" | MAS-Setup-Time (Epoch conversion)| +|"IPV4_SRC_ADDR" | Acme-Flow-Out-Dst-Addr_FS1_R| +|"L4_SRC_PORT" | Acme-Flow-Out-Dst-Port_FS1_R| +|"IPV4_DST_ADDR" | Acme-Flow-Out-Src-Addr_FS1_R| +|"L4_DST_PORT" | Acme-Flow-Out-Src-Port_FS1_R| +|"RTP_IN_JITTER": | Acme-Calling-RTCP-MaxJitter_FS1| +|"RTP_OUT_JITTER" | Acme-Calling-RTP-MaxJitter_FS1| +|"RTP_IN_PKT_LOST" | Acme-Calling-RTCP-Packets-Lost_FS1| +|"RTP_OUT_PKT_LOST" | Acme-Calling-RTP-Packets-Lost_FS1| +|"OUT_BYTES" | Acme-Calling-Octets_FS1| +|"OUT_PKTS" | Acme-Calling-Packets_FS1| +|"IN_BYTES" | Acme-Called-Octets_FS1| +|"IN_PKTS" | Acme-Called-Packets_FS1| +|"RTP_IN_TRANSIT" | Acme-Calling-RTCP-MaxLatency_FS1| +|"RTP_IN_PAYLOAD_TYPE" | Acme-FlowType_FS1_R (CODEC MAP)| +|"RTP_OUT_PAYLOAD_TYPE" | Acme-FlowType_FS1_F (CODEC MAP)| +|"RTP_SIP_CALL_ID" | Acct-Session-Id| +|"RTP_MOS" | Acme-Calling-MOS| +|"RTP_R_FACTOR" | Acme-Calling-R-Factor| +``` + +###Leg 2 Voice Quality Report +``` +| QoS Metric | CDR Parameter| +| ------------- |:------------:| +|"FIRST_SWITCHED" | MAS-Setup-Time (Needs Epoch conversion)| +|"IPV4_SRC_ADDR" | Acme-Flow-Out-Src-Addr_FS1_F| +|"L4_SRC_PORT" | Acme-Flow-Out-Src-Port_FS1_F| +|"IPV4_DST_ADDR" | Acme-Flow-Out-Dst-Addr_FS1_F| +|"L4_DST_PORT" | Acme-Flow-Out-Dst-Port_FS1_F| +|"RTP_IN_JITTER": | Acme-Called-RTP-MaxJitter_FS1| +|"RTP_OUT_JITTER" | Acme-Called-RTCP-MaxJitter_FS1| +|"RTP_IN_PKT_LOST" | Acme-Called-RTP-Packets-Lost_FS1| +|"RTP_OUT_PKT_LOST" | Acme-Called-RTCP-Packets-Lost_FS1| +|"OUT_BYTES" | Acme-Calling-Octets_FS1| +|"OUT_PKTS" | Acme-Calling-Packets_FS1| +|"IN_BYTES" | Acme-Called-Octets_FS1| +|"IN_PKTS" | Acme-Called-Packets_FS1| +|"RTP_OUT_TRANSIT" | Acme-Called-RTCP-MaxLatency| +|"RTP_IN_PAYLOAD_TYPE" | Acme-FlowType_FS1_F (CODEC MAP)| +|"RTP_OUT_PAYLOAD_TYPE" | Acme-FlowType_FS1_R (CODEC MAP)| +|"RTP_SIP_CALL_ID" | Acct-Session-Id| +|"RTP_MOS" | Acme-Called-MOS| +|"RTP_R_FACTOR" | Acme-Called-R-Factor| +``` From 1d15074065e6a2aaf6b5da5856607d468b7b9425 Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 13:00:19 +0500 Subject: [PATCH 45/48] Create RTPPROXY.md --- docs/HEP-EEP Agent Examples/RTPPROXY.md | 27 +++++++++++++++++++++++++ 1 file changed, 27 insertions(+) create mode 100644 docs/HEP-EEP Agent Examples/RTPPROXY.md diff --git a/docs/HEP-EEP Agent Examples/RTPPROXY.md b/docs/HEP-EEP Agent Examples/RTPPROXY.md new file mode 100644 index 0000000..7492410 --- /dev/null +++ b/docs/HEP-EEP Agent Examples/RTPPROXY.md @@ -0,0 +1,27 @@ +# RTPPROXY + HEP + + + +### HEP Support +[RTPProxy](https://www.rtpproxy.org/) can natively send HEP RTCP Reports _(HEP type 5)_ for relayed RTP/RTCP streams + +### Module Configuration Parameters: +Create _(or extend)_ the optional configuration file containing the required settings: +``` +modules { + rtpp_acct_rtcp_hep { + load = ../modules/acct_rtcp_hep/.libs/rtpp_acct_rtcp_hep.so + capt_host = HEP_SERVER_IP + capt_port = 9060 + capt_ptype = udp + capt_id = 101 + } +} +``` + +Include the saved configuration in your RTPProxy arguments using the config parameter: +``` +RTPPROXY_ARGS="--config path/to/rtpp_acct_rtcp_hep.conf" +``` + +Further examples can be found in the [RTPProxy test coverage](https://github.com/sippy/rtpproxy/blob/master/tests/acct_rtcp_hep/basic.conf) From d42dd8a06e362964d1fc3c71103cb3c066c0ffea Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 13:00:56 +0500 Subject: [PATCH 46/48] Rename ACME Packet.md to Oracle ACME Packet.md --- .../{ACME Packet.md => Oracle ACME Packet.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename docs/HEP-EEP Agent Examples/{ACME Packet.md => Oracle ACME Packet.md} (100%) diff --git a/docs/HEP-EEP Agent Examples/ACME Packet.md b/docs/HEP-EEP Agent Examples/Oracle ACME Packet.md similarity index 100% rename from docs/HEP-EEP Agent Examples/ACME Packet.md rename to docs/HEP-EEP Agent Examples/Oracle ACME Packet.md From 54e105f169c93ecb06434d90e6532a926d2ab0db Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 13:02:09 +0500 Subject: [PATCH 47/48] Create Sansay SBC.md --- docs/HEP-EEP Agent Examples/Sansay SBC.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 docs/HEP-EEP Agent Examples/Sansay SBC.md diff --git a/docs/HEP-EEP Agent Examples/Sansay SBC.md b/docs/HEP-EEP Agent Examples/Sansay SBC.md new file mode 100644 index 0000000..6db1319 --- /dev/null +++ b/docs/HEP-EEP Agent Examples/Sansay SBC.md @@ -0,0 +1,15 @@ +# Sansay HEP Suppport +Recent (2021) versions of the Sansay SBC software support the HEP encapsulation protocol natively. + +## Configuration + + +### parameters +``` +TAP[1] +filterDnis=* +signalingIp=10.0.0.1 +signalingPort=9060 +protocol=HEP +expires=1800 +``` From 8bf943dd70c124bf4979e73afbeedd279594e180 Mon Sep 17 00:00:00 2001 From: Muhammad Naqash Saeed <110665986+naqashsaeed@users.noreply.github.com> Date: Mon, 25 Nov 2024 13:04:18 +0500 Subject: [PATCH 48/48] Create Sonus SBC.md --- docs/HEP-EEP Agent Examples/Sonus SBC.md | 85 ++++++++++++++++++++++++ 1 file changed, 85 insertions(+) create mode 100644 docs/HEP-EEP Agent Examples/Sonus SBC.md diff --git a/docs/HEP-EEP Agent Examples/Sonus SBC.md b/docs/HEP-EEP Agent Examples/Sonus SBC.md new file mode 100644 index 0000000..b9eb328 --- /dev/null +++ b/docs/HEP-EEP Agent Examples/Sonus SBC.md @@ -0,0 +1,85 @@ +# SONUS SBC + +The Sonus SBC does not (yet) provide dynamic means for always-on packet monitoring except switch-based port mirroring/spanning. The only available resource to fetch information from is ```weblog.log``` which can be to some extend parsed and recycled using [paStash](https://github.com/sipcapture/pastash) + +#### WARNING! This example setup is a workaround and might be unreliable! +## Requirements + +* [HOMER](http://sipcapture.org) or [HEPIC](http://hepic.tel) +* SONUS SBC w/ weblog enabled +* HOST w/ NodeJS, npm + +## Installation +The first step is to install paStash on our middleware HOST +``` +# git clone https://github.com/sipcapture.pastash +# cd pastash +# sudo npm install +``` + +## Configuration +Configuration involves three stages: ```input```, ```filter```, ```output``` + +Adjust the following parameters as needed: +* input > file > path +* output > hep > host, port + +Save the config file locally, ie: ```/opt/pastash/sonus.json``` + +``` +input { + file { + path => "/var/log/sonus/webui.log" +# When testing with a static file: +# start_index => 0 + } +} + +filter { + multiline { + start_line_regex => /^\[\d{4}-\d{2}-\d{2}/ + } + grok { + match => '(?m)\[%{TIMESTAMP_ISO8601:timestamp}\] %{WORD:pid} %{WORD:seq} \n%{GREEDYDATA:payload}' + } + regex { + regex => /From:.*\@(.*)\:(\d+)/ + fields => [srcIp,srcPort] + } + regex { + regex => /To:.*\@(.*)\:(\d+)/ + fields => [dstIp,dstPort] + } + regex { + regex => /Call-I.*: (.*)/ + fields => [correlation_id] + } +} + +output { + if [tags] != "_grokparsefailure" { + hep { + host => '127.0.0.1' + port => 9063 + hep_id => 2222 + hep_type => 1 + } + } +} +``` + +## HEP Time +It's time to run our recipe and check for results: +``` +./bin/pastash --config_file=/opt/pastash/sonus.json +``` + +If everything is configured correctly and wind is on our side, paStash should start convertnig relevant Sonus logs to HEP-SIP packets over UDP. NGREP to confirm this is true before searching for received data. + +-------- + +### Known Issues +* Fragmented messages are not yet handled +* IP/PORT extracted from signaling, unreliable source +* no caching from loglines preceding the SIP methods +* only tested by a few users, needs feedback & tuning