puppet-prometheus

Table of Contents

• puppet-prometheus
  • Table of Contents
  • Compatibility
  • Background
    • What This Module Affects
    • Example Usage
    • Monitored Nodes
  • Example
  • How to Automatically Generate Scrape Configs
  • Known issues
  • Development
    • Component versions
  • Transfer Notice

---

Compatibility

This module supports below Prometheus architectures:

• x86_64/amd64
• i386
• armv71 (Tested on raspberry pi 3)

The prometheus::ipmi_exporter and prometheus::cgroup_exporter classes have a dependency on saz/sudo Puppet module.

Background

This module automates the install and configuration of Prometheus monitoring tool: Prometheus web site

What This Module Affects

• Installs the prometheus daemon, alertmanager or exporters(via url or package)
  • The package method was implemented, but currently there isn't any package for prometheus
• Optionally installs a user to run it under (per exporter)
• Installs a configuration file for prometheus daemon (/etc/prometheus/prometheus.yaml) or for alertmanager (/etc/prometheus/alert.rules)
• Manages the services via upstart, sysv, or systemd
• Optionally creates alert rules

Example Usage

class { 'prometheus::server':
  version        => '2.52.0',
  alerts         => {
    'groups' => [
      {
        'name'  => 'alert.rules',
        'rules' => [
          {
            'alert'       => 'InstanceDown',
            'expr'        => 'up == 0',
            'for'         => '5m',
            'labels'      => {
              'severity' => 'page',
            },
            'annotations' => {
              'summary'     => 'Instance {{ $labels.instance }} down',
              'description' => '{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes.'
            }
          },
        ],
      },
    ],
  },
  scrape_configs => [
    {
      'job_name'        => 'prometheus',
      'scrape_interval' => '10s',
      'scrape_timeout'  => '10s',
      'static_configs'  => [
        {
          'targets' => [ 'localhost:9090' ],
          'labels'  => {
            'alias' => 'Prometheus',
          }
        }
      ],
    },
  ],
}

Monitored Nodes

include prometheus::node_exporter

or:

class { 'prometheus::node_exporter':
  version            => '0.27.0',
  collectors_disable => ['loadavg', 'mdadm'],
}

Example

Real Prometheus >=2.0.0 setup example including alertmanager and slack_configs.

class { 'prometheus':
  manage_prometheus_server => true,
  version                  => '2.52.0',
  alerts                   => {
    'groups' => [
      {
        'name'  => 'alert.rules',
        'rules' => [
          {
            'alert'       => 'InstanceDown',
            'expr'        => 'up == 0',
            'for'         => '5m',
            'labels'      => {'severity' => 'page'},
            'annotations' => {
              'summary'     => 'Instance {{ $labels.instance }} down',
              'description' => '{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes.'
            },
          },
        ],
      },
    ],
  },
  scrape_configs           => [
    {
      'job_name'        => 'prometheus',
      'scrape_interval' => '10s',
      'scrape_timeout'  => '10s',
      'static_configs'  => [
        {
          'targets' => ['localhost:9090'],
          'labels'  => {'alias' => 'Prometheus'}
        }
      ],
    },
    {
      'job_name'        => 'node',
      'scrape_interval' => '5s',
      'scrape_timeout'  => '5s',
      'static_configs'  => [
        {
          'targets' => ['nodexporter.domain.com:9100'],
          'labels'  => {'alias' => 'Node'}
        },
      ],
    },
  ],
  alertmanagers_config     => [
    {
      'static_configs' => [{'targets' => ['localhost:9093']}],
    },
  ],
}

class { 'prometheus::alertmanager':
  version   => '0.27.0',
  route     => {
    'group_by'        => ['alertname', 'cluster', 'service'],
    'group_wait'      => '30s',
    'group_interval'  => '5m',
    'repeat_interval' => '3h',
    'receiver'        => 'slack',
  },
  receivers => [
    {
      'name'          => 'slack',
      'slack_configs' => [
        {
          'api_url'       => 'https://hooks.slack.com/services/ABCDEFG123456',
          'channel'       => '#channel',
          'send_resolved' => true,
          'username'      => 'username'
        },
      ],
    },
  ],
}

And if you want to use Hiera to declare the values instead, you can simply include the prometheus class and set your Hiera data as shown below:

Puppet Code

include prometheus

Hiera Data (in yaml)

---
prometheus::manage_prometheus_server: true

prometheus::version: '2.52.0'

prometheus::alerts:
  groups:
    - name: 'alert.rules'
      rules:
        - alert: 'InstanceDown'
          expr: 'up == 0'
          for: '5m'
          labels:
            severity: 'page'
          annotations:
            summary: 'Instance {{ $labels.instance }} down'
            description: '{{ $labels.instance }} of job {{ $labels.job }} has been
              down for more than 5 minutes.'

prometheus::scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: '10s'
    scrape_timeout: '10s'
    static_configs:
      - targets:
          - 'localhost:9090'
        labels:
          alias: 'Prometheus'
  - job_name: 'node'
    scrape_interval: '10s'
    scrape_timeout: '10s'
    static_configs:
      - targets:
          - 'nodexporter.domain.com:9100'
        labels:
          alias: 'Node'

prometheus::alertmanagers_config:
  - static_configs:
      - targets:
          - 'localhost:9093'

prometheus::alertmanager::version: '0.27.0'

prometheus::alertmanager::route:
  group_by:
    - 'alertname'
    - 'cluster'
    - 'service'
  group_wait: '30s'
  group_interval: '5m'
  repeat_interval: '3h'
  receiver: 'slack'

prometheus::alertmanager::receivers:
  - name: 'slack'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/ABCDEFG123456'
        channel: "#channel"
        send_resolved: true
        username: 'username'

How to Automatically Generate Scrape Configs

You can dynamically generate scrape configurations using exported resources. For any exporter managed by this module, you can enable this feature with a simple parameter. This exports the scrape job definition from the monitored node.

class {'prometheus::node_exporter':
  # set this to true to export the scrape job for any
  # prometheus exporter defined by this module
  export_scrape_job => true,
  # ... other parameters
}

For custom exporters not managed by this module, you can manually define and export a prometheus::scrape_job resource:

# on a node with a custom prometheus exporter
@@prometheus::scrape_job { "myjob-${facts['networking']['fqdn']}":
  job_name => 'myjob',
  targets  => [
    "${facts['networking']['fqdn']}:1234",
  ],
  labels   => {
    foo => 'bar',
  },
}

On your Prometheus server, you then collect these exported resources by defining prometheus::collect_scrape_jobs. You can also add or override any scrape configuration parameters.

# In your Prometheus server's Hiera data
prometheus::collect_scrape_jobs:
  # corresponds to the job_name from the node_exporter
  - job_name: 'node'
  - job_name: 'myjob'
    # add or override any scrape parameters for this job
    metrics_path: /custom
    scheme: https

This setup instructs the Prometheus server to collect all exported scrape_job resources that match the specified job_name values. The module then generates file-based service discovery configurations, resulting in a prometheus.yaml entry similar to this:

# /etc/prometheus/prometheus.yaml
scrape_configs:
- job_name: myjob
  metrics_path: /custom
  scheme: https
  file_sd_configs:
  - files:
    - "/etc/prometheus/file_sd_config.d/myjob_*.yaml"
# ... other jobs

Within the directory /etc/prometheus/file_sd_configs.d, you will find the configuration files for each target defined by the exported scrape_job resource. Example:

- targets:
  - <host.example.org>:1234
  labels:
    foo: bar

Known issues

Postfix is not supported on Archlinux because it relies on puppet-postfix, which does not support Archlinux.

Development

See https://voxpupuli.org/docs/how_to_run_tests/ for information on how to run test locally.

Component versions

For this repository a renovate github action is enabled. It will create PRs for updating the versions of the components. Each version defintion (in data/defaults.yaml or in the manifests directly) has a comment in the form of # renovate: depName=<github-repo-slug> which is used by renovate to identify the components to update. If new components (usually exporters) are added, please ensure to add the comment to the version definition. The PRs created by renovate have to be classified on a case-by-case basis by the reiviewer. Most of these PRs should be simple einhancements, but some might require more attention and be classiefied as backward-incompatible.

Transfer Notice

This plugin was originally authored by brutus333