Hetzner VMs unavailable ("resource unavailable")

[jampy]

My situation is certainly slightly special because I regularily destroy my cluster and recreate it from scratch. I do this regularily because I'm in the middle of long-running process of learning/creating/adjusting the cluster for later production use. One of my goals is that the recreation of the cluster needs to be something very well tested and not something exceptional.

The problem I'm facing however also relates to other hetzner-k3s users using the autoscaler or simply setting up a new cluster.

What I'm talking about is that more often than not, Hetzner appears to not having enough VM ressources available to start a VM of a certain type. That's also noticable in the normal Hetzner Cloud UI, where some VMs are grayed out and can't be ordered, like right now for Nurenberg:

hetzner-k3s in such a case reports the problem like this:

[Instance my-test-cluster-master2] Creating instance my-test-cluster-master2 failed: {
  "error": {
    "message": "error during placement",
    "code": "resource_unavailable",
    "details": null
  }
}

Right after deleting my cluster I often can't re-order the same VM types again and need to wait until they become available again.

This issue seems to be common.

Do we have any other options apart from just being patient? Any ideas?

[vitobotta]

Yeah this is super annoying. I am working a lot on hetzner-k3s at the moment (we use it at the day job too) so I am creating and deleting clusters very often, and this past week I have been running into resource unavailability issues quite often unfortunately.

The only workaround I have for now is that at least for the worker nodes I specify an autoscaled node pool for each instance type so during my testing I let the autoscaler pick up whichever instances are available in any location among Falkenstein, Nuremberg, Helsinki and that works fine for the workers.

But for the master and static worker node pools there isn't much we can do other than wait for new capacity or just try another location unfortunately.

[jampy]

Good points.

The problem with using another location is that (persistent, using volumeHandle) Hetzner volumes can't be attached to servers in another location.

[vitobotta]

Ah true, I hadn't thought about that.

[jampy]

I don't know if it's just me but it's now at least a full day that no x86 VMs bigger than CPX21 are available in Nuremberg.

Very annoying. Seems like Falkenstein has more ressources available, but switching location is not that easy for me since Hetzner Volumes and LBs can't be moved to other locations AFAIK.

[vitobotta]

It's not just you. I am having this issue all the time lately.

[jampy]

Note: The problem is documented by Hetzner itself here: https://status.hetzner.com/incident/aa5ce33b-faa5-4fd0-9782-fde43cd270cf

[vitobotta]

I know, it's been there for many months now

[jampy]

Thought it would be interesting for others...

[jampy]

The problem has been documented at Hetzner for 10 months and it doesn't look like it will change anytime soon.

Do you see any possibility for a workaround? Perhaps by transferring existing (running) instances to hetzner-k3s so that the tool takes over these instances?

I built my clusters with the idea that I could completely rebuild them within minutes if something went very wrong. In reality, I am very hesitant to do so, as there is a risk that hetzner-k3s might not get enough instances for 1-2 hours, leaving the cluster offline for a longer period of time.

Ideally, there would be a way to avoid releasing instances to Hetzner when executing hetzner-k3s delete and to recycle these instances when executing hetzner-k3s create. This would allow us to completely bypass the problem with Hetzner, at least when rebuilding an equivalent cluster.

Instead of deleting the instances with the Hetzner API, they could simply be switched off by hetzner-k3s, for example. At the same time, I could imagine a hetzner-k3s option that allows switched-off (and perhaps labeled) instances of a specific Hetzner Cloud project to be reused.

I cannot estimate how complicated such a solution (or a similar one) would be to implement. I would be willing to pay for it.

[jampy]

You can uninstall k3s completely with a single script, see https://docs.k3s.io/installation/uninstall.

I could add a command to automate this in hetzner-k3s but I don't have time for this at the moment.

Thanks for pointing that out.

I don't think explicit uninstallation is necessary. Instead, the contents of the persistent storage can/should be completely destroyed/overwritten – just like a normal hetzner-k3s delete. According to my concept, a hetzner-k3s create would still reinstall an O/S image in such a situation and overwrite all previous data.

For us at work we are using autoscaling for everything basically, and a nice thing with the autoscaling is that it picks nodes from where they are available or changes the instance type. So if it tries to get instance type X from Falkenstein and it's not available there, it will pick it from another location where it's available, or just change instance type. So with this setup we haven't had any problems anymore really.

Switching between locations is an option until you reuse Hetzner volumes or load balancers. Unfortunately, these cannot be moved between locations.

A solution that does not require any modification of hetzner-k3s would certainly be preferable.

When you say “it will pick it from another location where it's available, or just change instance type,” do you mean that the autoscaler itself can choose different instance types (completely automatic)? If so, then I'm afraid I'm not sure how to define that.

I read the example to mean that exactly one instance_type can be configured, but not several...?

Apart from that, three servers must be free at Hetzner for the master servers (we use cpx21), and we have already experienced bottlenecks there, albeit not often. Am I missing something here?

[vitobotta]

Thanks for pointing that out.

I don't think explicit uninstallation is necessary. Instead, the contents of the persistent storage can/should be completely destroyed/overwritten – just like a normal hetzner-k3s delete. According to my concept, a hetzner-k3s create would still reinstall an O/S image in such a situation and overwrite all previous data.

I am not keen on the idea of having hetzner-k3s trigger OS reinstallation etc. It was designed with the idea in mind that instances can be easily and quickly created and deleted as needed, so I don't think I am going to implement this as it would be quite an effort that likely not many people would use. Like I mention there is a workaround. You can write a quick bash script that runs the uninstall script on each node and reboots it. It takes a few minutes to write such a script if you don't want to do it manually on each server so I really don't see the value in implementing what you are suggesting in hetzner-k3s.

Switching between locations is an option until you reuse Hetzner volumes or load balancers. Unfortunately, these cannot be moved between locations.

Good point, I had forgotten about that. I haven't come across this issue because like I mention I just use autoscaling with node pools with all instance types in all the EU locations.

When you say “it will pick it from another location where it's available, or just change instance type,” do you mean that the autoscaler itself can choose different instance types (completely automatic)? If so, then I'm afraid I'm not sure how to define that.

The autoscaler will pick available instances of the types configured in the node pools of course. We have a node pool for each instance type and let the autoscaler pick what's available and suitable for the workloads pending scheduling.

Apart from that, three servers must be free at Hetzner for the master servers (we use cpx21), and we have already experienced bottlenecks there, albeit not often. Am I missing something here?

hetzner-k3s automatically waits and retries creating a server if the desired instance type is not available, until the server can be created. From what I have seen, in the vast majority of cases when you try creating a server of a specific instance type it will eventually succeed that way, often in a matter of minutes. It would be great if this was not a problem at all, but we don't have control on this so not much we can do until Hetzner increases capacity.

[jampy]

I am not keen on the idea of having hetzner-k3s trigger OS reinstallation etc. It was designed with the idea in mind that instances can be easily and quickly created and deleted as needed, so I don't think I am going to implement this as it would be quite an effort that likely not many people would use. Like I mention there is a workaround. You can write a quick bash script that runs the uninstall script on each node and reboots it. It takes a few minutes to write such a script if you don't want to do it manually on each server so I really don't see the value in implementing what you are suggesting in hetzner-k3s.

Hello Vito.

Please do not misunderstand me; I am not trying to pressure you into anything, but I feel that my concern may have come across that way. I am simply trying to express myself more clearly and also gain a better understanding of the options available. As I said, I cannot estimate how easy or difficult it would be to implement such a solution. This is, after all, a discussion.

I'm not very keen on the solution involving reusing the Hetzner instances either, and I can understand why you don't want to integrate this into hetzner-k3s if it's complicated.

However, I don't understand what the uninstall script and reboot you suggested are supposed to do. This would give me unused instances without k3s, but I don't understand what I would do with these instances or how I could get hetzner-k3s to use them to reinstall and reconfigure k3s.

I can imagine that there is no simple solution of this kind, but your suggestion somehow implies that the uninstall script would be part of a pragmatic solution (which I don't understand).

The autoscaler will pick available instances of the types configured in the node pools of course. We have a node pool for each instance type and let the autoscaler pick what's available and suitable for the workloads pending scheduling.

Ah, that sounds good. I didn't realize it could be used that way. Brilliant! I'll give it a try!

hetzner-k3s automatically waits and retries creating a server if the desired instance type is not available, until the server can be created. From what I have seen, in the vast majority of cases when you try creating a server of a specific instance type it will eventually succeed that way, often in a matter of minutes. It would be great if this was not a problem at all, but we don't have control on this so not much we can do until Hetzner increases capacity.

Well, I've had situations where hetzner-k3s gave up 2-3 times in a row after a timeout of several minutes.

Since we can't influence the situation at Hetzner, I'm looking for workarounds. It's just annoying when you “delete” 5-6 instances at Hetzner and a minute later you can't order the same number of instances with the same specifications. Therefore, it makes sense not to delete the instances in the first place (especially since Hetzner does not offer reserved instances). I hope you understand my reasoning.

In any case, the autoscaling pools might improve the situation. I will test it out.

[jampy]

In any case, the autoscaling pools might improve the situation. I will test it out.

Sorry, but I don't seem to understand.

I tried the following configuration, assuming that without workloads, only one CPX41 instance would be started, but no others.

Immediately after hetzner-k3s create (i.e., without workloads), the following configuration starts a CPX31 first and then an additional CPX41.

Based on the previous discussion, I would have expected only one CPX41 to start (because min_instances: 1) and no other instances, since there is no need for them.

My ideal final goal would be to start CPX41 instances (since they are more efficient) and CPX31 only as a fallback.

What am I missing?

worker_node_pools:

  - name: worker-cpx41
    instance_type: cpx41
    location: nbg1
    autoscaling:
      enabled: true
      min_instances: 1
      max_instances: 3

  - name: worker-cpx31
    instance_type: cpx21
    location: nbg1
    autoscaling:
      enabled: true
      min_instances: 0
      max_instances: 3

  - name: worker-cpx21
    instance_type: cpx21
    location: nbg1
    autoscaling:
      enabled: true
      min_instances: 0
      max_instances: 4

NB: I tested this yesterday evening and today. According to the Hetzner Console, CPX41 instances were available at these times.

[vitobotta]

Autoscaling can be a bit confusing regarding this thing you noticed. For example, the minimum instance count for a node pool only kicks in after some scaling has already happened. So, for worker-cpx41, the minimum of 1 instance will apply only after a scaling event. If it scales up to 3 nodes first, then it will keep that one node even if it’s not needed, just to respect the minimum number.

In this case, it’s likely that the autoscaler preferred other pools when deciding which instances to create. This might seem a bit odd, but that’s how the Kubernetes autoscaler works. It’s not specific to Hetzner-k3s. In my opinion, autoscaling works best when you set up multiple node pools with different instance types and locations, setting the minimum to 0 and the maximum to what you need, and then let the autoscaler manage everything. If you want to always keep a minimum number of instances for a specific type from the start, it’s better to have a static node pool for that and let the autoscaler handle the rest.

At work, we have two static node pools for our database: 2 instances in Falkenstein and 1 in Nuremberg. These nodes are labeled and tainted so they’re only used for the database, with dedicated cores and all. These are the only static node pools we have. For everything else, we let the autoscaler choose the best instances and locations.