Optionally allow the user to enable strict host key checking

[msimkunas]

Description

It would be useful if it was possible to have a template option which enables strict host key checking, and, optionally, the path to a known_hosts file.

This would introduce an additional layer of security when connecting to Lima hostnames, such as lima-default. The local port that they use may be reused by other services on the host when Lima VMs are powered off, and having strict host key checking would prevent accidental connections and/or data exposure (such as when syncing data).

[afbjorklund]

I enabled strict host key checking for external (not localhost) hosts, and re-enabled the default ssh known_hosts file:

• External driver, for cloud and physical #2109

  StrictHostKeyChecking no
  UserKnownHostsFile /dev/null

Also disabled BatchMode, because you could get an interactive prompt if not adding the host keys from cloud-init.

$ ssh -p 60022 127.0.0.1
The authenticity of host '[127.0.0.1]:60022 ([127.0.0.1]:60022)' can't be established.
ECDSA key fingerprint is SHA256:v8YAQ2idaF2vP5XpG+Lh1jYUOXXOIWtaQLy/Xlk1+tk.
Are you sure you want to continue connecting (yes/no/[fingerprint])? 

     BatchMode
             If set to yes, user interaction such as password prompts and host key confirmation requests will be disabled.

    PasswordAuthentication no

EDIT: Ideally, that prompt will not be shown.

The new host should be added to known_hosts...

[afbjorklund]

The ssh_known_hosts file should probably be in the instance directory, and not in the config directory (with the key).
That way we don't have to clean it up as much and have a long file shared by all instances, should there be many*?

* each new random port gets a new host entry, and many OS keep like three different key formats for each host

Extra paranoid users might even want to have separate identities for separate instances, but that's a different feature.
Currently the same lima key is used for all the instances for a user*, but different users would get different ssh keys.

* or alternatively (in addition) the regular ssh identities, if you enable the loadDotSSHPubKeys feature that is.

[afbjorklund]

This issue is a problem for the local instances, though:

• SSH host keys are regenerated each time a VM does a stop/start #678

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!

[msimkunas]

@afbjorklund My main goal is not to control which users have access to the VMs but to more strictly control where the SSH connections are tunneled.

Imagine that you spin up a VM. A random local port is then assigned and tunneled. You set up background syncing of files via that SSH tunnel to this VM.

Eventually you shut down the VM and the port becomes reused, this time for another SSH tunnel that points to an actual remote host. With bad luck and no strict host key checks, this could result in files being synced to the wrong remote.

[afbjorklund]

Oh, indeed. The chances of that happening increase, with the number of users on the host (and bad luck, I suppose)...

Just saying that the user account is a double check, with per-user keys (and ~/.ssh disabled) that is less likely to happen.

But why would you need ssh tunnels for remote hosts?

Wouldn't you just connect directly to the remote host (22)?

[msimkunas]

Wouldn't you just connect directly to the remote host (22)?

I think you would. If you assume that the user is conscious and human, a lot of things that make sense are given for granted. :)

I was thinking more in terms of how foolproof the entire solution is. If the user installs weird software that opens tunnels to remote hosts and somehow by accident the Lima local port becomes reused, I think there should be a way to bake in sane defaults for Lima, i.e. at the very least enabling strict host key checks so that the entire solution is a little more foolproof.

[msimkunas]

The way I do this now is I define my own ssh client config with my own custom host alias that points to 127.0.0.1 and a static port reserved that I reserved for this VM via the Lima template. I also enabled strict host key checks for this host alias.

[afbjorklund]

That sounds like a more sane approach, than inventing some new artificial config like podman system connection et al

The main feature of the ssh.config (beyond all other nice features) is otherwise that the port is automatically updated

[afbjorklund]

My workaround for getting the host keys was running ssh-keyscan right after creation and boot. It's not great, but.

Something like this code, for another project (vind):

• Allow using known hosts and host key checking brightzheng100/vind#5

I don't think there is any standardized way (or path) to get them, and I don't think that Lima can access it yet anyway?

[afbjorklund]

The problem is not exactly new, and people have been ignoring host keys for decades.

• Turn StrictHostKeyChecking on for SSH docker-archive-public/docker.machine#534

[msimkunas]

I don't use ssh-keyscan but instead fetch the public ssh host key by executing ssh-keygen inside the newly started VM with limactl shell. It's not much better (if at all) than your solution though, just different.

I think the only way to solve this properly is to automatically generate ssh host keys on the host that's creating the VM and to inject them into the VM via cloud-init or some such mechanism.

I'm aware that strict host key checks is a dying practice but we should always strive for a better security posture...

[afbjorklund]

The host keys are generated by cloud-init, and "normally" they are published to the cloud datasource.

  ssh_publish_hostkeys: (object)
    enabled: (boolean) If true, will read host keys from /etc/ssh/*.pub and publish them to the datasource (if supported). Default: true
    blacklist: (array of string) The SSH key types to ignore when publishing. Default: [dsa]

The key path varies by distribution. (cloud-init default is KEY_FILE_TPL = "/etc/ssh/ssh_host_%s_key")

/etc/ssh/ssh_host_ecdsa_key.pub
/etc/ssh/ssh_host_ed25519_key.pub
/etc/ssh/ssh_host_rsa_key.pub

https://cloudinit.readthedocs.io/en/latest/reference/modules.html#ssh

They can be injected with ssh_keys (and re-generated by ssh_delete_keys)

---

EDIT: Unfortunately, it is only implemented for GCE and not for other cloud providers (like our NoCloud):

    def publish_host_keys(self, hostkeys):
        """Publish the public SSH host keys (found in /etc/ssh/*.pub).

        @param hostkeys: List of host key tuples (key_type, key_value),
            where key_type is the first field in the public key file
            (e.g. 'ssh-rsa') and key_value is the key itself
            (e.g. 'AAAAB3NzaC1y...').
        """

https://github.com/canonical/cloud-init/tree/main/cloudinit/sources

[afbjorklund]

It (ssh) is run during the init-network phase: (says cloud-init analyze)

Starting stage: init-local
|`->no cache found @00.00500s +00.00000s
|`->found local data from DataSourceNoCloud @00.01100s +00.05800s
Finished stage: (init-local) 00.63400 seconds

Starting stage: init-network
|`->restored from cache with run check: DataSourceNoCloud [seed=/dev/sr0] @01.99500s +00.00100s
|`->setting up datasource @02.02700s +00.00000s
|`->reading and applying user-data @02.03500s +00.00500s
|`->reading and applying vendor-data @02.04000s +00.00000s
|`->reading and applying vendor-data2 @02.04000s +00.00100s
|`->activating datasource @02.06600s +00.00100s
|`->config-seed_random ran successfully and took 0.000 seconds @02.08300s +00.00100s
|`->config-write_files ran successfully and took 0.001 seconds @02.08400s +00.00100s
|`->config-growpart ran successfully and took 0.455 seconds @02.08500s +00.45500s
|`->config-resizefs ran successfully and took 0.196 seconds @02.54000s +00.19700s
|`->config-mounts ran successfully and took 0.176 seconds @02.73700s +00.17600s
|`->config-set_hostname ran successfully and took 0.001 seconds @02.91300s +00.00100s
|`->config-update_hostname ran successfully and took 0.001 seconds @02.91400s +00.00100s
|`->config-users_groups ran successfully and took 0.719 seconds @02.91500s +00.71900s
|`->config-ssh ran successfully and took 0.212 seconds @03.63500s +00.21200s
|`->config-set_passwords ran successfully and took 0.000 seconds @03.84700s +00.00100s
Finished stage: (init-network) 03.17400 seconds

So I guess it could be copied out (where?), with a /var/lib/cloud script

/var/lib/cloud/scripts/per-instance/99-lima.ssh.sh