Skip to content
This repository was archived by the owner on Feb 26, 2020. It is now read-only.

Automattic's Fork #3

Open
wants to merge 49 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
ce96297
Make it work for ES 1.0.1
May 27, 2014
e52e56d
Version bump and define encoding
xyu Aug 5, 2014
eff4ff2
Cleanup code and capitalization
xyu Aug 5, 2014
89a9c61
Add config to skip reporting of shard info
xyu Aug 5, 2014
77a384d
Fix Whitespace -- spaces to tabs
xyu Aug 6, 2014
d474c16
Add comments to args list, cleanup debug output
xyu Aug 6, 2014
0cdc884
Refactor reporters
xyu Aug 6, 2014
3e61e37
Fix whitespace
xyu Aug 6, 2014
10a749b
Cleanup Node reporting and standardize naming to match those used by ES
xyu Aug 6, 2014
e13cc58
Fix node indice stat name to match ES native ones and add some new stats
xyu Aug 6, 2014
d3df5a3
Extract out common index stats reporting functions
xyu Aug 6, 2014
0351865
Use common class to report shard stats
xyu Aug 7, 2014
1363d45
Report node stats on a per node basis with node name in the stats key
xyu Aug 7, 2014
58bb0e5
Audit stat types (counter vs guage) to match up ES and StatsD types f…
xyu Aug 7, 2014
980af49
Allow customization of node name used in StatsD metric key for node l…
xyu Aug 7, 2014
9fc5d9e
Spin up 2 nodes during tests
xyu Aug 7, 2014
84fffc4
Index more docs during tests to get more data
xyu Aug 7, 2014
7473971
Fix test assertions
xyu Aug 7, 2014
61feeaf
Add some error checking and fix names
xyu Aug 7, 2014
22097a4
Use for key prefix var is confusing
xyu Aug 7, 2014
185650d
Simplify error logging
xyu Aug 7, 2014
7b9b582
Simplify stat key for cluster index totals
xyu Aug 7, 2014
21fba7f
Sum and report index stats on a per index and per shard basis; make i…
xyu Aug 7, 2014
2f156ae
Document StatsD key format
xyu Aug 7, 2014
b144e88
Cleanup stat key name for shard level stats
xyu Aug 7, 2014
e4c43c3
Everything is a Guage
xyu Aug 7, 2014
ff6ab80
Update README
xyu Aug 7, 2014
a14f505
Remove debug use only code
xyu Aug 8, 2014
e8158f3
Check cluster state for elected master
xyu Aug 8, 2014
a5e1df0
Make node level index stats optional
xyu Aug 8, 2014
b365bac
Update README with new settings
xyu Aug 8, 2014
43aeb90
Indices Reporter Should Report Cluster Totals
xyu Aug 8, 2014
5231901
Fix var names, 'type' is a loaded word in ES so don't use it if it do…
xyu Aug 8, 2014
5a8bd2a
Fix name in credits
xyu Aug 8, 2014
6b9a0be
Version Bump to v0.3.1
xyu Aug 8, 2014
5c5cc05
Update README -- fix URL
xyu Aug 8, 2014
e2d2bb2
Upgrade Java StatsD Client\n\nNeeded to send longs not ints
xyu Aug 8, 2014
887836c
Make tests spin up 3 nodes to distribute docs better
xyu Aug 8, 2014
d92279e
Use IndicesStatsRequestBuilder / IndicesStatsResponse in ES to gather…
xyu Aug 8, 2014
a4143b7
Add Warmer and Percolate stats
xyu Aug 8, 2014
a5c3e2c
Version Bump
xyu Aug 8, 2014
97565dc
Upgrade Java StatsD Client\n\nNeeded to send longs not ints
xyu Aug 8, 2014
cd7ca20
Make tests spin up 3 nodes to distribute docs better
xyu Aug 8, 2014
ee59c95
Use IndicesStatsRequestBuilder / IndicesStatsResponse in ES to gather…
xyu Aug 8, 2014
512dbc2
Add Warmer and Percolate stats
xyu Aug 8, 2014
de7dbad
Version Bump
xyu Aug 8, 2014
7e8b67a
Merge branch 'master' into a8c-dev
xyu Aug 20, 2014
1d3ec42
Send fractional values for load average to StatsD
xyu Aug 20, 2014
1f8c8e6
Version bump to v0.3.3
xyu Aug 20, 2014
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 33 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,30 +1,42 @@
# Elasticsearch statsd plugin
# Elasticsearch StatsD Plugin

This plugin creates a little push service, which regularly updates a statsd host with indices stats and nodes stats. In case you are running a cluster, these datas are always only pushed from the master node.
This plugin creates a little push service, which regularly updates a StatsD host with indices stats and nodes stats.
Index stats that apply across the entire cluster is only pushed from the elected master which node level stats are pushed from every node.

The data sent to the statsd server tries to be roughly equivalent to [Indices Stats API](http://www.elasticsearch.org/guide/reference/api/admin-indices-stats.html) and [Nodes Stats Api](http://www.elasticsearch.org/guide/reference/api/admin-cluster-nodes-stats.html)
The data sent to the StatsD server tries to be roughly equivalent to the [Indices Stats API](http://www.elasticsearch.org/guide/reference/api/admin-indices-stats.html) and [Nodes Stats Api](http://www.elasticsearch.org/guide/reference/api/admin-cluster-nodes-stats.html).


## Installation

As plugins (except site plugins) cannot be automatically installed from github currently you need to build the plugin yourself (takes half a minute including an integrations test).
To install a prepackaged plugin use the following command:

```
git clone http://github.com/swoop-inc/elasticsearch-statsd-plugin.git
bin/plugin -install statsd -url https://github.com/Automattic/elasticsearch-statsd-plugin/releases/download/v0.3.3/elasticsearch-statsd-0.3.3.zip
```

You can also build your own by doing the following:

```
git clone http://github.com/Automattic/elasticsearch-statsd-plugin.git
cd elasticsearch-statsd-plugin
mvn package
/path/to/elasticsearch/bin/plugin -install statsd -url file:///absolute/path/to/current/dir/target/releases/elasticsearch-statsd-0.2-SNAPSHOT.zip
bin/plugin -install statsd -url file:///absolute/path/to/current/dir/target/releases/elasticsearch-statsd-0.3.3.zip
```


## Configuration

Configuration is possible via three parameters:
Configuration is possible via these parameters:

* `metrics.statsd.host`: The statsd host to connect to (default: none)
* `metrics.statsd.port`: The port to connect to (default: 8125)
* `metrics.statsd.every`: The interval to push data (default: 1m)
* `metrics.statsd.prefix`: The metric prefix that's sent with metric names (default: elasticsearch.your_cluster_name)
* `metrics.statsd.node_name`: Override the name for node used in the stat keys (default: the ES node name)
* `metrics.statsd.report.node_indices`: If per node index sums should be reported (default: false)
* `metrics.statsd.report.indices`: If index level sums should be reported (default: true)
* `metrics.statsd.report.shards`: If shard level stats should be reported (default: false)
* `metrics.statsd.report.fs_details`: If nodes should break down the FS by device instead of total disk (default: false)

Check your elasticsearch log file for a line like this after adding the configuration parameters below to the configuration file

Expand All @@ -33,19 +45,30 @@ Check your elasticsearch log file for a line like this after adding the configur
```


## Stats Key Formats

This plugin reports both node level and cluster level stats, the StatsD keys will be in the formats:

* `{PREFIX}.node.{NODE_NAME}.{STAT_KEY}`: Node level stats (CPU / JVM / etc.)
* `{PREFIX}.node.{NODE_NAME}.indices.{STAT_KEY}`: Index stats summed across the node (off by default)
* `{PREFIX}.indices.{STAT_KEY}`: Index stats summed across the entire cluster
* `{PREFIX}.index.{INDEX_NAME}.total.{STAT_KEY}`: Index stats summed per index across all shards
* `{PREFIX}.index.{INDEX_NAME}.{SHARD_ID}.{STAT_KEY}` -- Index stats per shard (off by default)


## Bugs/TODO

* No really nice cluster support yet (needed it for a single instance system)
* Not extensively tested
* In case of a master node failover, counts are starting from 0 again (in case you are wondering about spikes)


## Credits

Heavily inspired by the excellent [metrics library](http://metrics.codehale.com) by Code Hale and its [GraphiteReporter add-on](http://metrics.codahale.com/manual/graphite/).
This is a fork of the [Swoop plugin](https://github.com/swoop-inc/elasticsearch-statsd-plugin) for multi-node clusters on ES 1.x.

Heavily inspired by the excellent [metrics library](http://metrics.codahale.com) by Coda Hale and its [GraphiteReporter add-on](http://metrics.codahale.com/manual/graphite/).


## License

See LICENSE

11 changes: 6 additions & 5 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -6,21 +6,22 @@
<modelVersion>4.0.0</modelVersion>
<groupId>de.spinscale.elasticsearch</groupId>
<artifactId>elasticsearch-statsd</artifactId>
<version>0.2-SNAPSHOT</version>
<version>0.3.3</version>
<packaging>jar</packaging>
<description>Statsd monitoring plugin for Elasticsearch</description>
<url>https://github.com/swoop-inc/elasticsearch-statsd-plugin/</url>
<description>StatsD monitoring plugin for Elasticsearch</description>
<url>https://github.com/Automattic/elasticsearch-statsd-plugin/</url>

<properties>
<elasticsearch.version>0.90.3</elasticsearch.version>
<elasticsearch.version>1.0.1</elasticsearch.version>
<hamcrest.version>1.3</hamcrest.version>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>

<dependencies>
<dependency>
<groupId>com.timgroup</groupId>
<artifactId>java-statsd-client</artifactId>
<version>2.0.0</version>
<version>3.0.2</version>
</dependency>

<dependency>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ public String name()

public String description()
{
return "Statsd Monitoring Plugin";
return "StatsD Monitoring Plugin";
}

@SuppressWarnings("rawtypes")
Expand Down
Loading