Benchmark of software architecture

Table of content

• General Information
• Used Technologies
• Aplications architecture overview
• Infrastructure Overview
• Benchmark Overview
• Start Guide
  • Prerequisite steps to prepare cloud environment
  • Steps to run benchmark in cloud environment
  • Steps to run benchmark in local environment
• About Authors

General Information

This is an implementation of a complete benchmarking application. It is adapted to compare the performance and scalability of applications depending on the used technology and software architecture.

Each application have been implemented in several technologies and integrated with tools that allow you to generate a load and collect metrics. More details are available in the Used Technologies section.

Each of the applications being the subject of research offers service for the traveling salesman algorithm implemented in two different versions. Each of them is based on a different computational and memory complexity. The input data for each application are cities in Poland correlated with their coordinates. Each of the technologies was implemented using two different software architectures - monolith and microservices. During the process of their implementation a big pressure was put to use the same logic for each application. More details about application architecture available in the section Aplications architecture overview.

Benchmark generates an even load on the application and collects data about resource consumption and response times. The application has been designed to conduct tests in the cloud environment using the AWS platform. Appropriate tools were used to automate the benchmark launch process and all processes were divided into separate stages. The details about the benchmark process are available in the sections Infrastructure Overview and Benchmark Overview.

Monitoring of application has been implemented in a way that allows to review dedicated metrics in the form of static charts after the benchmark ends. There is also a possibility to observe metrics in real time using the Graphana tool.

Used Technologies

Applications:

Programming Language	Framework
Java	Spring Boot
C#	.NET
Python	Flask

• Docker and docker-compose to containerize each applications
• Nginx as reverse proxy and load balancer for microservices

Infrastructure:

The AWS cloud was chosen as the environment for the application. The applications run on EC2 which is a virtual machine running the Linux operating system, with the size and performance depending on needs (to be selected during the launch of the workflow called Preparing the cloud environment)

Terraform and Ansible were used to create the entire infrastructure.

• Terraform is responsible for AWS configuration i.e. creating a virtual machine, creating a security group, selecting a key to connect via SSH, setting up a provider along with the region where machine is to be located, creating a VPC and using the Terraform cloud, it eliminates the need for unnecessary tooling and documentation for teams to use Terraform in production.
• Ansible is responsible for properly configuring the server. Installing all the dependencies, running all the necessary services and scripts, and granting the appropriate permissions.

Aplication load:

• To generate the load we use the JMeter tool that allows parameterization of load generation using multiple threads.
• The load generation process is embedded in a separate environment to do not disturb measurement process of resource consumption by the real application being the object under test.

Monitoring

• A set of technologies was used to monitor resource consumption by the application: CAdvisor, Prometheus and Grafana. Thanks to this, it is possible to view the collected metrics in real time via connecting to Grafana tool available on port 3000.

Postprocessing:

• R programming language - generating static charts

Aplications architecture overview

The microservice architecture consists of five components. One of them is Reverse Proxy and the other are applications written in a specific programming language. Microservices communicate with each other via the HTTP protocol. Each of service has a different responsibility:

• Place Service - Provide information about places and them coordinates
• Distance Service - Calculate spherical distance between passed points
• Route1 Service - Solve TSP problem using nearest neighbour algorithm
• Route2 Service - Solve TSP problem by checking all variants

The monolith architecture contains one component. All the necessary functionalities are implemented as part of this component. All communication between the implemented functionalities works with internal references in the code of a programming language.

Infrastructure overview

Benchmark overview

To run benchmark requires to prepare an environment using configuration scripts or use appropriate workflow. Environment should have all necessary dependendent software and properly configured monitoring environment. if the environment is properly configured, just run this script:

./script/run_benchmark.sh [-f <string> (required)] [-l <string> (required)] [-b <string> (required)]

The -f parameter determines name of docker-compose file on which the benchmark should be carried out. The -l parameter determines address of load generating host. The -b parameter determines address of benchmark host.

The benchmark process is based on several steps:

• Setup step runs the application on which the benchmark will be performed
• Benchmark step runs the load on the application
• Collecting data step collects and saves obtained data
• cleaning step closes the application and clears the obtained data at prometheus

Diagram below shows the complete infrastructure used to run the benchmark. Whole benchmark process is adapted to use it on AWS platform. At docker is running monitoring infrastructure (Grafana, Prometheus, cAdvisor) and application being under benchmark process. Application workload runs natively on the host operating system (JMeter).

The load is generated from an independent environment (Load Generating Environment) on the AWS EC2 platform, to don't disrupt resource use by the environment under test (Benchmark Environment).

Start Guide

Prerequisite steps to prepare cloud environment

1. Creating an AWS account
   • Create an IAM account named admin and add it to the group Administrators
   • Create an IAM account named terrafom and add it to the group Administrators
   • Generate a key for the admin account and save it as admin.pem
   Detailed instruction
2. Move the generated key to the system-benchmarking-monolith-vs-microservices/infrastructure/ssh_key folder
3. Encrypt the key with gpg

gpg --symmetric --cipher-algo AES256 admin.pem

Thanks to the fact that an entry admin.pem was added to gitignore, the key will not be added to the remote repository, it will only appear in the local one

4. Creating an Terraform cloud account
   • Name your organization Projekt-badawczy
   • Create workspace gh-actions
   Detailed instruction
5. Github secrets should be set as follows
   • TF_API_TOKEN - Terraform cloud API
   • EC2_SSH_KEY - Unencrypted content of admin.pem key
   • BENCHMARK_DNS - Public IPv4 DNS adress of a benchmark environment server
   • LOAD_GENERATING_DNS - Public IPv4 DNS adress of a load generating environment server
   • PASSWORD - password used to encrypt admin.pem key with gpg
6. Github actions

   Building the entire infrastructure is done automatically using workflows. They have been numbered from 1 to 4, respectively, according to the order in which they should be run.

   • Preparing cloud environment is responsible for running the terraform code, that is, building the entire infrastructure in AWS
   • Benchmark server configuration is responsible for running the ansible code, installing the appropriate dependencies and software i.e. cAdvisor and configuring the benchmark server
   • Load generating server configuration is responsible for running the ansible code, installing the appropriate dependencies and software i.e.JMeter, Grafana, Prometheus and configuring the load generating server
   • Run benchmarks simply runs benchmarks

Steps to run benchmark in cloud environment

To start benchmark process and prepare cloud environment dedicated github workflows have been prepared:

1. Prepare cloud environment prepare cloud environment on AWS platform using Terraform
2. Configure benchmark server prepare all dependencies on benchmark environment using Ansible
3. Configure load generating server prepare all dependencies on load generating environment using Ansible
4. Run benchmark run benchmark process and collect data

All related results with single run available at Run benchmark run history as workflow artifact.

Steps to run benchmark in local environment

To start benchmark in local environment run this script:

bash scripts/local-env/run_benchmark.sh -f <docker-compose file>

Results

The data was collected using a varying number of users, where each user was represented by a separate thread. The number of threads used varied depending on the scenario, and each workload performed requests for a duration of 30 seconds. The services were deployed on the AWS EC2 T2 Instances.

Vertical scaling - CPU Usage

Vertical scaling - Throughput

Horizontal scaling - Throughput

About Authors

This project is a part of the "Projekt Badawczy" program performed by students of Computer Science master degree studies at GUT.