[PVE-User] pve-cluster won't start with large ssh_known_hosts

[Derek W. Poon]

With pve-cluster 1.0-12 (Proxmox VE 2.0 beta3), /etc/init.d/pve-cluster fails to start if /etc/ssh/ssh_known_hosts is larger than 128 kiB. Could this limit be removed?

I have tracked this down to /usr/bin/pvecm,
which calls PVE::Cluster::ssh_merge_known_hosts(...),
which calls PVE::Tools::file_get_contents($sshglobalknownhosts, 128*1024),
which calls PVE::Tools::safe_read_from(...),
which dies because the maximum length is exceeded.

If you happen to install Proxmox VE on top of an existing Debian system, as I tried to do, and this error occurs, then pve-cluster and all of its dependent packages will fail to be configured.

Derek

[Dietmar Maurer]

I have tracked this down to /usr/bin/pvecm, which calls
PVE::Cluster::ssh_merge_known_hosts(...),
which calls PVE::Tools::file_get_contents($sshglobalknownhosts, 128*1024),
which calls PVE::Tools::safe_read_from(...), which dies because the maximum
length is exceeded.
If you happen to install Proxmox VE on top of an existing Debian system, as I
tried to do, and this error occurs, then pve-cluster and all of its dependent
packages will fail to be configured.

Why is that file that large?

- Dietmar

[Derek W. Poon]

Post by Dietmar Maurer

I have tracked this down to /usr/bin/pvecm, which calls
PVE::Cluster::ssh_merge_known_hosts(...),
which calls PVE::Tools::file_get_contents($sshglobalknownhosts, 128*1024),
which calls PVE::Tools::safe_read_from(...), which dies because the maximum
length is exceeded.

Why is that file that large?

Dietmar,

In our department, we use cfengine to distribute an ssh_known_hosts file to all Linux machines. Each host takes two lines -- one for the RSA key, one for DSA. Our file is currently 295172 bytes, or 566 lines, corresponding to roughly 283 hosts, which is actually not unreasonable, in my opinion. Our computers can process that amount of data in a negligible amount of time.

Is there a technical reason for the 128 kiB limit, or is it an arbitrary restriction? If it is the latter, then I suggest removing it, as there is no point in having Proxmox introduce a failure mode for no particular reason.

Derek

[Dietmar Maurer]

Post by Derek W. Poon
Is there a technical reason for the 128 kiB limit, or is it an arbitrary restriction? If
it is the latter, then I suggest removing it, as there is no point in having Proxmox
introduce a failure mode for no particular reason.

We store those file on the pmxcfs file system, which is RAM based, and is only
suited for small files.

Can't you use the GlobalKnownHosts file instead (/etc/ssh/ssh_known_hosts)?
Or should PVE use that file instead?

- Dietmar

[Derek W. Poon]

Post by Dietmar Maurer
We store those file on the pmxcfs file system, which is RAM based, and is only
suited for small files.
Can't you use the GlobalKnownHosts file instead (/etc/ssh/ssh_known_hosts)?
Or should PVE use that file instead?

Dietmar,

In what ways does Proxmox VE use SSH? Skimming the code, I see that it uses SSH (1) to establish VNC tunnels and (2) to add new nodes to the cluster ("pvecm add"). Anything else?

It would then follow that the known_hosts file in pmxcfs is just for deploying host keys for (1) hosts that are in the cluster and (2) hosts that are being added to the cluster. For both cases, the set of host keys is quite small and well known. It would be just as good to register specific host keys into pmxcfs when adding the hosts to the cluster.

Have I misunderstood anything?

Derek

[Dietmar Maurer]

Post by Derek W. Poon
In what ways does Proxmox VE use SSH? Skimming the code, I see that it uses
SSH (1) to establish VNC tunnels and (2) to add new nodes to the cluster ("pvecm
add"). Anything else?

Yes, we just need to be able to ssh to any cluster node.

Post by Derek W. Poon
It would then follow that the known_hosts file in pmxcfs is just for deploying
host keys for (1) hosts that are in the cluster and (2) hosts that are being added
to the cluster. For both cases, the set of host keys is quite small and well known.

yes.

Post by Derek W. Poon
It would be just as good to register specific host keys into pmxcfs when adding
the hosts to the cluster.

I thought that is what we are doing currently?

Or do you suggest that we should explicitly set the path to those files instead
of using the standard locations and symlinks?

- Dietmar

[Dietmar Maurer]

Post by Derek W. Poon
It would then follow that the known_hosts file in pmxcfs is just for deploying
host keys for (1) hosts that are in the cluster and (2) hosts that are being added
to the cluster. For both cases, the set of host keys is quite small and well known.
It would be just as good to register specific host keys into pmxcfs when adding
the hosts to the cluster.

OK, think I know what you mean now ;-) What you say is correct. We already do
it that way for the authorize_key file. There is 'ssh-copy-id' for authorized key, and
I guess we need to implement something similar for known_hosts.

So we can generate/update the known_hosts file on 'pvecm add' time.

- Dietmar

[Dietmar Maurer]

Post by Derek W. Poon

Post by Derek W. Poon
It would then follow that the known_hosts file in pmxcfs is just for
deploying host keys for (1) hosts that are in the cluster and (2)
hosts that are being added to the cluster. For both cases, the set of host keys

is quite small and well known.

Post by Derek W. Poon
It would be just as good to register specific host keys into pmxcfs
when adding the hosts to the cluster.

OK, think I know what you mean now ;-) What you say is correct. We already do
it that way for the authorize_key file. There is 'ssh-copy-id' for authorized key,
and I guess we need to implement something similar for known_hosts.

Seems I am a bit confused today - sorry. All those things does not really help to solve your problem.

To summarize: You use the global known_host file /etc/ssh/ssh_known_hosts, so we can't use that
to store the cluster hosts (because that file is too large to store on pmxcfs).

Using the user file /root/.ssh/known_hosts is also bad, because that can also grow quite large.

Using any other file is clumsy because it is not used by default, so we always need to pass extra parameter.

Any better ideas?

[f***@zipman.it]

Post by Dietmar Maurer
Using any other file is clumsy because it is not used by default, so we always need to pass extra parameter.
Any better ideas?

Yes. Create a dedicated ssh_config_pve configuration file in the
/etc/pve directory and pass it as the unique parameter to every call of ssh.

ssh -F /etc/pve/ssh_config_pve .....

I know it's a parameter to pass, but everything else is included in it
and it allows for really simple changes if you need them in the future.

For example you can lower the cryptographic security to lower cpu usage
or specify a dedicated ssh key for some hosts, or use specific users for
some others, etc.etc.

You create a default ssh_config_file specifying a pve specific
known_hosts and all other default parameters to use and if someone need
something different should only modify that file.