Grafana-agent down in dashboard status

[rogmanster]

Description

Grafana Agent shows down in dashboard status - newly deployed.

Screenshots

Log Messages

Grafana-agent logs (all periodically with similar error):

ts=2024-09-27T02:09:44.980753674Z level=error msg="non-recoverable error" component=prometheus.remote_write.default subcomponent=rw remote_name=bd527e url=http://prometheus.coder-observability.svc/api/v1/write count=2000 exemplarCount=0 err="server returned HTTP status 400 Bad Request: out of order sample"
ts=2024-09-27T02:09:50.086696001Z level=error msg="non-recoverable error" component=prometheus.remote_write.default subcomponent=rw remote_name=bd527e url=http://prometheus.coder-observability.svc/api/v1/write count=1558 exemplarCount=0 err="server returned HTTP status 400 Bad Request: out of order sample"
ts=2024-09-27T02:10:05.040293177Z level=error msg="non-recoverable error" component=prometheus.remote_write.default subcomponent=rw remote_name=bd527e url=http://prometheus.coder-observability.svc/api/v1/write count=626 exemplarCount=0 err="server returned HTTP status 400 Bad Request: out of order sample"
ts=2024-09-27T02:10:44.998642978Z level=error msg="non-recoverable error" component=prometheus.remote_write.default subcomponent=rw remote_name=bd527e url=http://prometheus.coder-observability.svc/api/v1/write count=2000 exemplarCount=0 err="server returned HTTP status 400 Bad Request: out of order sample"
ts=2024-09-27T02:11:05.079034516Z level=error msg="non-recoverable error" component=prometheus.remote_write.default subcomponent=rw remote_name=bd527e url=http://prometheus.coder-observability.svc/api/v1/write count=626 exemplarCount=0 err="server returned HTTP status 400 Bad Request: out of order sample"

Grafana pod logs:

logger=context userId=0 orgId=0 uname= t=2024-09-27T02:41:46.069582049Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=10.0.22.6 time_ms=0 duration=888.896µs size=105 referer= handler=/api/live/ws status_source=server
logger=context userId=0 orgId=0 uname= t=2024-09-27T02:42:07.000216216Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=10.0.22.6 time_ms=0 duration=759.171µs size=105 referer= handler=/api/live/ws status_source=server
logger=provisioning.dashboard type=file name=sidecar t=2024-09-27T02:42:13.869036352Z level=error msg="Cannot read directory" error="stat /tmp/dashboards: no such file or directory"
logger=provisioning.dashboard type=file name=sidecar t=2024-09-27T02:42:13.869120292Z level=error msg="Failed to read content of symlinked path" path=/tmp/dashboards error="lstat /tmp/dashboards: no such file or directory"
logger=provisioning.dashboard type=file name=sidecar t=2024-09-27T02:42:13.869133508Z level=info msg="falling back to original path due to EvalSymlink/Abs failure"
logger=provisioning.dashboard type=file name=sidecar t=2024-09-27T02:42:13.869144257Z level=error msg="failed to search for dashboards" error="stat /tmp/dashboards: no such file or directory"
logger=context userId=0 orgId=0 uname= t=2024-09-27T02:42:27.003993831Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=10.0.22.6 time_ms=0 duration=667.764µs size=105 referer= handler=/api/live/ws status_source=server
logger=provisioning.dashboard type=file name=sidecar t=2024-09-27T02:42:43.868560334Z level=error msg="Cannot read directory" error="stat /tmp/dashboards: no such file or directory"
logger=provisioning.dashboard type=file name=sidecar t=2024-09-27T02:42:43.868611097Z level=error msg="Failed to read content of symlinked path" path=/tmp/dashboards error="lstat /tmp/dashboards: no such file or directory"
logger=provisioning.dashboard type=file name=sidecar t=2024-09-27T02:42:43.868620434Z level=info msg="falling back to original path due to EvalSymlink/Abs failure"
logger=provisioning.dashboard type=file name=sidecar t=2024-09-27T02:42:43.86862969Z level=error msg="failed to search for dashboards" error="stat /tmp/dashboards: no such file or directory"

To Reproduce

 helm install coder-observability coder-observability/coder-observability \
     --version 0.2.1 \
     --namespace coder-observability \
     --values values.yaml

• Note: Can share the values.yaml - latest and with updates only to postgres.