Secrets Management

[seydx]

Secrets Management

What's this about?

I've added a simple (but effective) secret management system to go2rtc. It lets you store credentials in a dedicated secrets section in the config file instead of embedding them directly in stream URLs. It works with existing template system and on top, every values used in secrets, will be auto redacted in log and stream info.

Why is this useful?

• Keeps sensitive stuff like tokens and passwords organized in one place
• Clients can update credentials on the fly (super handy for tokens that expire)
• Everything stays in the same config file - just better organized
• Works alongside existing config approaches, so nothing breaks
• Redact secrets in log/stream info

How it works

Config setup

1. Reference a secret by name (client would manually query the secret, see example code)

# go2rtc.yaml
secrets:
  ring_doorbell:
    camera_id: "12345678"
    device_id: "abc123def456"
    refresh_token: "eyJhbGciOi..."

streams:
  front_door:
    - ring:?secrets=ring_doorbell

2. Template system for direct value insertion:

# go2rtc.yaml
secrets:
  hikvision:
    username: admin
    password: passw0rd

streams:
  hikvision_4k:
    - rtsp://${hikvision_username}:${hikvision_password}@192.168.178.142/Streaming/channels/102

In the code

// Via query
query := u.Query()
secretsName := query.Get("secrets")

// or own secretsName
secretsName := "ring_doorbell"

var defaultConfig = struct {
  CameraID  string    `yaml:"camera_id"`
  DeviceID  string    `yaml:"device_id"`
  RefreshToken string `yaml:"refresh_token"`
} {
  CameraID:     query.Get("camera_id"),
  DeviceID:     query.Get("device_id"),
  RefreshToken: query.Get("refresh_token"),
}

// Get (or create) a secret with default values
secret := core.NewSecret(secretsName, defaultConfig)

// update the config with the secret values
secrets.Unmarshal(&config)

// Use it
cameraID := secret.Get("camera_id").(string)
refreshToken := secret.Get("refresh_token").(string)

// Update it when needed
secret.Set("refresh_token", newToken) 

// Writes back to go2rtc.yaml
secret.Save()

Implementation details

• Thread-safe with mutex locks
• Simple key-value storage with type conversion helpers
• Works with nested values
• Minimal changes to existing code

Note

I'd like to find a way to directly map stream names to secret names without explicitly specifying the secret in the URL, but haven't found a clean solution yet. I'm open to suggestions and ideas on how to best implement this or other improvements!

The code is pretty straightforward and doesn't change how anything else works, so existing stream configs will continue to work exactly as before.

[AlexxIT]

I don't like the idea of new template format. Go2rtc already has support environment variables.

I don't mind another section in the config file to hold secrets. But it is important that these secrets are in the same format as the environment variables.

[seydx]

I don't like the idea of new template format. Go2rtc already has support environment variables.

I don't mind another section in the config file to hold secrets. But it is important that these secrets are in the same format as the environment variables.

Thanks for the feedback! I understand your concerns about both the template format and adding a new section to the config file.

To clarify, the main purpose of this PR is to provide developers with a way to programmatically update credentials at runtime, particularly for integrations like Ring, Tuya, or Nest that require periodic token refreshes.

Environment variables are great for initial configuration, but they're static after the application starts. The proposed system allows:

1. Dynamic updating of credentials during runtime (crucial for OAuth tokens that expire)
2. Writing changes back to the config file
3. A clean API for source integrations to manage their own credentials

Regarding the config structure, I'm flexible on the implementation approach. Instead of a dedicated secrets section, we could:

• Integrate this functionality directly into the existing streams section
• Use a naming convention within the existing config structure
• Store these dynamic credentials in a separate file entirely

The template syntax was mainly introduced as a convenience, but it's not the core value of the PR and could be removed.

What approach would you prefer that maintains the existing config structure while still providing a way for integrations to handle dynamic credential updates?

[felipecrs]

Guys, if you ask me, I think it's a bad idea to store "programmatic" persistent data in go2rtc.yaml like dynamically retrieved credentials.

Instead, it should be stored in a separate file like go2rtc.json (a bit similar to how Home Assistant stores stuff in .storage, but it does not need to be that complicated).

Because go2rtc supports running with no configuration file at all. You can also pass everything through config flags, for example.

Or in case of Frigate, the go2rtc.yaml is dynamically generated on every container startup out of the frigate.yaml, and then stored in memory at /dev/shm/go2rtc.yml. This means changes made by go2rtc would not survive a container restart. (although Frigate could "fix" it by retrieving changes from /dev/shm/go2rtc.yml during the finish logic).

Again, that's just my opinion.

[AlexxIT]

I don't mind adding a new section to the config file.

For example, CAMERA_PASSWORD can be taken either from CREDENTIALS file, from environment variable or from the env section of the config file. Does this solve your problem?

env:
  CAMERA_PASSWORD: very-secret

streams:
  camera1: rtsp://rtsp:${CAMERA_PASSWORD}@192.168.1.123/av_stream/ch0

[AlexxIT]

In the first version, go2rtc stored dynamic data in the go2rtc.json file.
It seemed like an added complexity to me. So I spent a lot of effort to allow dynamic data (like homekit authorization) to be stored in the go2rtc.yaml file.
Considering that go2rtc is able to work with several config files at the same time - it is already possible to configure one of the config files to store dynamic data and the other to store static data.

[felipecrs]

Considering that go2rtc is able to work with several config files at the same time - it is already possible to configure one of the config files to store dynamic data and the other to store static data.

Is there some kind of documentation for that? I may be able to leverage this for Frigate.

[AlexxIT]

@felipecrs documentation above (go2rtc support multiple config files): https://github.com/AlexxIT/go2rtc/blob/master/internal/app/README.md

First file in the list will be selected as "dynamic":

go2rtc/internal/app/config.go

Lines 65 to 68 in ae8145f

	// config as file
	if ConfigPath == "" {
	ConfigPath = conf
	}

[felipecrs]

Thank you!!

[felipecrs]

I don't mind adding a new section to the config file.

For example, CAMERA_PASSWORD can be taken either from CREDENTIALS file, from environment variable or from the env section of the config file. Does this solve your problem?

env:
  CAMERA_PASSWORD: very-secret

streams:
  camera1: rtsp://rtsp:${CAMERA_PASSWORD}@192.168.1.123/av_stream/ch0

@seydx you can probably reconcile your approach with this suggestion. I.e. save the data back into env: rather than secret:.

[seydx]

I don't mind adding a new section to the config file.

For example, CAMERA_PASSWORD can be taken either from CREDENTIALS file, from environment variable or from the env section of the config file. Does this solve your problem?

env:
  CAMERA_PASSWORD: very-secret

streams:
  camera1: rtsp://rtsp:${CAMERA_PASSWORD}@192.168.1.123/av_stream/ch0

Are you sure this (checking config for env) is implemented? had to fix ReplaceEnvVars for this to work

// ReplaceEnvVars - support format ${CAMERA_PASSWORD} and ${RTSP_USER:admin}
func ReplaceEnvVars(text string) string {
	var yamlConfig struct {
		Env map[string]string `yaml:"env"`
	}

	yaml.Unmarshal([]byte(text), &yamlConfig)

	re := regexp.MustCompile(`\${([^}{]+)}`)
	return re.ReplaceAllStringFunc(text, func(match string) string {
		key := match[2 : len(match)-1]

		var def string
		var dok bool

		if i := strings.IndexByte(key, ':'); i > 0 {
			key, def = key[:i], key[i+1:]
			dok = true
		}

		if dir, vok := os.LookupEnv("CREDENTIALS_DIRECTORY"); vok {
			value, err := os.ReadFile(filepath.Join(dir, key))
			if err == nil {
				return strings.TrimSpace(string(value))
			}
		}

		if value, vok := os.LookupEnv(key); vok {
			return value
		}

		if yamlConfig.Env != nil {
			if value, vok := yamlConfig.Env[key]; vok {
				return value
			}
		}

		if dok {
			return def
		}

		return match
	})
}

[felipecrs]

Are you sure this is implemented?

It is not implemented. It was a suggestion from @AlexxIT on how you can implement it in a way that he would accept. I.e. you'd need to implement support for env:. :)

[seydx]

Are you sure this is implemented?

It is not implemented. It was a suggestion from @AlexxIT on how you can implement it in a way that he would accept. I.e. you'd need to implement env:. :)

but this means, we need to add a new section anyway? :) since there is no env section atm

[felipecrs]

And by the way, I would say the change should be done not only in ReplaceEnvVars, but also these env vars should be globally set in go2rtc so that invoking other processes like echo:, exec:, ffmpeg:, and whatnot can also inherit them.

[felipecrs]

but this means, we need to add a new section anyway? :) since there is no env section atm

Absolutely, that's what he meant. 😅

[seydx]

but this means, we need to add a new section anyway? :) since there is no env section atm

Absolutely, that's what he meant. 😅

Oh man, i got it now.... I should read everything to the end before i reply >_>

But it is important that these secrets are in the same format as the environment variables.

👍

[seydx]

Hey @AlexxIT , I updated the PR.

I've now added both env and secrets.

I kept these two separate for a specific reason. The env config is a key=value dict and basically serves the same purpose as replacing ${...} templates in the config.

The secrets config, however, is explicitly intended only for sensitive information, but it also replaces the ${...} templates in the config.

Completely optional, users can put their sensitive information in the secrets config so that clients can make use of the new secrets function (update secrets on runtime etc)

The next reason though is that there's also a new Redact function that makes use of all available secrets and removes all areas from the log that contain a secret. Stream info now also redacts sensitive information.

What do you think about it?

Example:

env:
  CAMERA_IP: 192.168.178.142

secrets:
  hikvision:
    username: admin2
    password: Admin!Admin
   
streams:
  hikvision:
    - rtsp://${hikvision_username}:${hikvision_password}@${CAMERA_IP}/Streaming/channels/101

Logs:

21:59:52.176 DBG [streams] start producer url=rtsp://*****:*****@192.168.178.142/Streaming/channels/101
22:00:04.194 DBG [streams] stop producer url=rtsp://*****:*****@192.168.178.142/Streaming/channels/101

Stream Info

{
  "producers": [
    {
      "id": 3,
      "format_name": "rtsp",
      "protocol": "rtsp+tcp",
      "remote_addr": "192.168.178.142:554",
      "url": "rtsp://*****:*****@192.168.178.142/Streaming/channels/101",
      ....
    }
  ],
  "consumers": [
    ...
  ]
}

[felipecrs]

Utilizing secrets to power the Redacting logic is very clever in my opinion, and a worthy feature.