Add new metric (histogram) to track number of reads using _bulk_get

[brkolla]

Add a new counter "bulk_request_docs" to track the number of documents being handled as part of the bulk request.

FogBugz 64560.

[kxepal]

Why? When you have a line above: couch_stats:update_histogram([couchdb, httpd, bulk_docs], length(DocsArray)) which is more informative than just constantly incremented counter.

[brkolla]

@kxepal ? I have asked the same question in 64560 and discussed it with Aaron. As per my understanding, counter gives a different perspective and probably easy to identify the spikes and trends? Can you add a comment to Aaron/Russel/Ops in 64560 and get his perspective on why this metric would help Operations?

[kxepal]

@brkolla Sorry, I can't since I'm not from Cloudant team. Can you ping them here or share the reason why? because I'm -1 on having two metrics that does the same thing while one of them (histogram) is actually what is really useful.

[brkolla]

I am actually asking Aaron about this in Slack. To be honest, I don't fully understand the difference between the histogram and the regular counter and if its possible to have one metric that does the both. Will ask around and find out more.

[kxepal]

@brkolla Histogram uses sliding time window and provides percentile-wise information about the recorded numbers. Just compare output of both these metrics to see the difference.

[chewbranca]

Yeah I agree with @kxepal on this, using a histogram is the correct choice here. It looks like there was also some confusion on this issue as @rnewson snuck in the actual fix for this issue back in cloudant@a49c474.

I do think it's worthwhile to also track the doc count in _bulk_get and that should be switched to a histogram like the above commit. The big question there is whether to use a different histogram to be able to distinguish between _bulk_get reads and _bulk_docs writes. I think having them separate would be reasonable.

Also, we should get similar stats for quantify of docs returned and provided in _all_docs and views and what not.

[kxepal]

Oh, it also counts _bulk_get. I miss that moment, sorry. But that should be a different metric and should have a correct name anyway.

[brkolla]

I think I have to revisit this and come up with a new metric to track the bulk gets and probably use one metric that tracks all the request that would result in bulk gets (_all_docs etc..) . We are still discussing on how we can do that.
I guess I will have to close this PR as we are not going to go ahead with this.

[chewbranca]

Yeah agreed, _bulk_get is sufficiently different we should track it separately.

[brkolla]

@chewbranca I have updated the code to add a new metric bulk_reads to track the number of doc reads done as part of the bulk_get. Can you review this code?

[kxepal]

I think the name is confusing. If we want to count only "/_bulk_get" requests here, better give such name to the metric instead of generic bulk_reads. Otherwise, if you want to use "bulk_reads", should we count requests with include_docs=true to views and all_docs as bulk reads?

[brkolla]

@kxepal I am not really sure, but this seems to be only used during the replication. I guess, we could change the name to bulk_get_docs? @chewbranca?

[kxepal]

@brkolla Not only. Replication is just a client for CouchDB. Other users may be this endpoint in own way (and for profit!).