[Feature Request] Implement LBaaS in Yaook

[anjastrunk]

Yaook as a further implementation of SCS standards, does not support a standard conform load balancer, yet. We have to provide one. At this, the only requirement is to provide a OpenStack conform endpoint to the user. The behavior behind the sense does not matter.

Tasks:

• Evaluate options for LBaaS
• Extend Yaook to support standard conform LBaaS

This issue is related to #587, which standardizes mandatory and recommended IaaS Service and LBaaS should be part of it.

[markus-hentsch]

Evaluate options for LBaaS

FTR, here is one of the main problems that prevented integration of Octavia in Yaook so far: https://storyboard.openstack.org/#!/story/2007370#comment-153426

In Yaook, any database instances are running behind HAProxy instances. This does seem to lead to severe problems with Octavia in production, according to the linked issue.

We should at least consider having a look at improving Octavia and/or its integration as re-implementing the whole Octavia LBaaS v2 API using a different LB framework will be no easy feat either.

We should get in touch with @horazont and check if there were other issues observed with Octavia than the one mentioned above that would need to be addressed as well.

[markus-hentsch]

I had a discussion with @horazont about this:

• the upstream issue report¹ seems to suggest that the issue is a race condition between a) Octavia API instructing its workers via RPC and b) MariaDB syncing the database write of the Octavia API to the other replicas in conjunction with the workers attempting to read the entry while being scheduled to a different DB replica through HAProxy that has not yet received the sync
  • however, @horazont said he is not convinced that this actually is the problem since the HAProxy instances are configured to always schedule the DB queries to the first DB replica in Yaook
  • we should reproduce and analyze the issue
• there seems to be an OVN backend driver² for Octavia, we should have a look at that one too
• creating an Octavia alternative with full API compatibility is a huge task and we should try all other options of getting Octavia working correctly in Yaook first, I think

Footnotes

1. https://storyboard.openstack.org/#!/story/2007370#comment-153426 ↩

2. https://docs.openstack.org/ovn-octavia-provider/latest/admin/driver.html ↩

[markus-hentsch]

While discussing the topic in a small topic kickoff with @kgube, @josephineSei and @kitsudaiki we identified the following tasks:

• Get in touch with the relevant CSPs and check if the SCS reference implementation ever experienced issues like the one mentioned above¹.
• Research which subset of the Octavia API is actually used and strictly needed by the KaaS part of the SCS reference implementation.
• Identify all possible use cases that the Octavia API offers and how each can be tested.
• Implement an Octavia operator prototype for Yaook to integrate Octavia in Yaook.
• Test the Octavia integration in Yaook, try to reproduce the original issue¹ and find a fix for it.

Note that aside from the last point most of these tasks are independent and can be addressed in parallel.

Footnotes

1. https://storyboard.openstack.org/#!/story/2007370#comment-153426 ↩ ↩²

[berendt]

@markus-hentsch In Kolla (used by OSISM by default to deploy OpenStack) there are 2 ways to access the MariaDB Galera cluster: HAPRoxy + ProxySQL. With both ways all nodes in a cluster access the database through the same node. This is the node that holds the primary IP address managed by Keepalived. If Keepalived is not used and the database is accessed otherwise and possibly not via only one node, I think Galera ensures that the information is identical on all nodes because Galera implements a multi-master cluster.

[kitsudaiki]

Started with a first prototypical octavia-operator for YAOOK. Reactivated the old issue on gitlab regarding the octavia integration ( https://gitlab.com/yaook/operator/-/issues/186 ) and create a new brauch for an octavia-operator ( https://gitlab.com/yaook/operator/-/tree/feature/add-octavia-operator ) and octavia docker-image for YAOOK ( https://gitlab.com/yaook/images/octavia/-/tree/feature/initial-version ) for the implementation.

[berendt]

Started with a first prototypical octavia-operator for YAOOK. Reactivated the old issue on gitlab regarding the octavia integration ( https://gitlab.com/yaook/operator/-/issues/186 ) and create a new brauch for an octavia-operator ( https://gitlab.com/yaook/operator/-/tree/feature/add-octavia-operator ) and octavia docker-image for YAOOK ( https://gitlab.com/yaook/images/octavia/-/tree/feature/initial-version ) for the implementation.

For Amphora images you can use https://github.com/osism/openstack-octavia-amphora-image. I will add 2024.2 images later this day.

[josephineSei]

I was looking through the Octavia documentation today and tried to identify use cases and features.

Different drivers / possible other choices:

At first there it can be noted, that the VM or container with the load-balancer itself (amphora) can be substituted by other options(see: https://docs.openstack.org/octavia/latest/admin/providers/index.html):

1. Amphora: the reference driver from the Octavia project
2. A10 Networks OpenStack Octavia Driver: for Thunder, vThunder and AX Series Appliances
3. F5 Networks Provider Driver for OpenStack Octavia by SAP SE
4. OVN Octavia Provider Driver
5. Radware Provider Driver for OpenStack Octavia
6. VMware NSX

From all these drivers that could be used only OVN is compared to Amphora in several feature matrices here. Looking through this document, it can be seen that there is a huge gap that seems to mainly be around everything needed for Layer 7 loadbalancing, which OVN does not support.

Example Use Cases

Octavia guides for basic and Layer 7 load-balancing give many examples for use cases.

It could be very coarsely divided into:

• Load-balancers for each: UDP, TCP, HTTP, HTTPS
• Applying Health Monitors to you Load-Balancer
• Applying TLS-termination
• Applying Layer 7 Loadbalancing rules including authentication, redirecting requests with invalid certificates, using cookies

Problem: TLS

To make use of TLS termintation or re-encryption a deployment with a working Key-Manager is needed.
For the scs project this means, we cannot make use of these features, because we do not mandate having a key-manager within a deployment.

Further Features

There are other features that come with Octavia, that are useful for operators mostly:

• Amphora Log Offloading (via syslog over the lbaas-management network)
• API Auditing (using Oslo messaging notifier -> could be routed to e.g. a log file)
• API Health Monitoring
• Octavia Flavors (predefined sets of provider configuration options // defined per provider driver)
• Amphora Failover Circuit Breaker (threshold for failovers to prevent mass failovers)
• using SR-IOV ports for Amphorae (increasing performance)

[markus-hentsch]

This might be important concerning Octavia with OVN backend: osism/issues#959