Improve our existing Grafana dashboard

[vnugent]

I'm looking for an experienced DevOps/infra engineer to help improve our Grafana cloud dashboard. We've already connected Granana with our K8s cluster.

1. Create a "single pane of glass" view containing essential metrics such as API requests, frontend request (not sure how to integerate with next.js)
2. Set up email or Discord notification alerts

[enapupe]

It seems we could push vercel logs to our grafana dash using the same loki endpoint we have configured already, however this requires Pro plan on vercel, I'm not sure we got it. Also, the log format vercel pushes doesn't seem to be accepted by loki, so we'd have to add a middleman to transform it, maybe a basic lambda function running on vercel itself?

https://logs-prod-006.grafana.net
    basicAuth:
      username: "<some user id>"
      password:  <some token here>

Regarding vercel TRACES, we must add an environment variable pointing open telemetry metrics to our grafana agent, however, our agent is not exposed publicly and I haven't considered the drawbacks of exposing it so I'm not 100% sure this is the way forward nor sure we even want to capture otel traces.

OTEL_EXPORTER_OTLP_ENDPOINT=http://grafana-k8s-monitoring-grafana-agent.default.svc.cluster.local:4318

also follow the instructions on https://vercel.com/docs/observability/otel-overview

pnpm i @opentelemetry/api @vercel/otel

nextConfig.experimental = { instrumentationHook: true };

Create an instrumentation.ts file in the root of your project and add the following code to initialize and configure OTEL using @vercel/otel.

import { registerOTel } from '@vercel/otel';
 
export function register() {
  registerOTel({ serviceName: 'your-project-name' });
}
// NOTE: You can replace `your-project-name` with the actual name of your project

[vnugent]

@enapupe we're on Vercel Pro plan