Make IP Lease available to deployment through ENV variable

[anilmurty]

Problem Statement
Akash Network supports leasing IP addresses from providers that offer them but the leased IP is not easily accessible by the deployment. Having access to the IP lease from the deployment is important in cases where the deployment needs to use the IP address for communication once the application comes up. A good example of this is Kadena Nodes that need this info for communication.

Workaround
The current workaround (in case of Kadena) is to read the IP address manually and build the application (Kadena node) by manually inserting the IP address at build time.

Suggested Solution
The suggested solution is to improve user experience for tenants by exposing the IP lease to the deployment through an environment variable.

[anilmurty]

@boz @troian - please add context for where this code change should be made. From our discussion, I understand it is a relatively minor change that someone from the community (like @PJEstrada) could potentially make.

[PJEstrada]

Hey guys, I started to read through the ipOperator code and cluser code. Giving the context I currently see this place where I can potentially add the new env variable for the IP.

https://github.com/akash-network/provider/blob/e7aa0b5b81957a130f1dc584f335c6f9e41db6b1/cluster/kube/builder/workload.go#L99

If this is the correct place, then what I'll need to figure out is to find a way to get the IP operators data, specifically the IP address string in the context of this function, so I can add it as part of the params to the addEnvVarsForDeployment .

Does the workload object has any way to currently access this data? If you have any specific files that you suggest me to read more in detail I'll happily go though them.

Thanks a lot :)

[boz]

Hi @PJEstrada, thanks for working on this!

I would start with looking at the implementation of the lease status API endpoint.

That ultimately uses a couple "clients" defined within the codebase:

• cluster/operatorclients/ip_operator_client.go#GetIPAddressUsage
• cluster/kube/client.go#ForwardedPortStatus
• cluster/kube/client.go#LeaseStatus

I don't think that the workload object has direct access to these at the point in the code that you're looking at, It'll likely need to be added. Let me know if you have any issues finding a way to do that.

Cheers!

[PJEstrada]

Thanks a lot @boz . I'll get back to you if I have any further questions.🙏

[PJEstrada]

Some related information regarding updating env variables on an already running deployment. Thanks to @andy108369 for sharing

Working POC to change a configmap without triggering a redeploy:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cm1
data:
  VAR1: "value123"
---
apiVersion: v1
kind: Pod
metadata:
  name: pod1
spec:
  containers:
  - name: container1
    image: nginx
    volumeMounts:
    - name: cm1-vol
      mountPath: /etc/config
  volumes:
  - name: cm1-vol
    configMap:
      name: cm1

Shell excution:

root@node1:~# kubectl get cm cm1 -o json | jq -r '.data'
{
  "VAR1": "value123"
}

root@node1:~# kubectl exec -ti pod1 -- sh -c "cat /etc/config/VAR1" ; echo
value123

root@node1:~# kubectl edit cm cm1
configmap/cm1 edited

root@node1:~# kubectl get cm cm1 -o json | jq -r '.data'
{
  "VAR1": "value456"
}

root@node1:~# kubectl exec -ti pod1 -- sh -c "cat /etc/config/VAR1" ; echo
value123

root@node1:~# sleep 1m

root@node1:~# kubectl exec -ti pod1 -- sh -c "cat /etc/config/VAR1" ; echo
value456

[PJEstrada]

In terms of where to add the configmap creation, based on what I read so far, I believe we can add it on the handling of the events:

ctypes.ProviderResourceAdd, ctypes.ProviderResourceUpdate:

https://github.com/akash-network/provider/blob/e7aa0b5b81957a130f1dc584f335c6f9e41db6b1/operator/ipoperator/ip_operator.go#L237

I see we have access to the kube client from the ipoperator so probably the way we could implement this is potentially calling the kube client to create the configmap.

A couple questions I have:

1. The event has the service name available, however I believe we might need to attach the configmap to the deployments which are using the requested IP lease. Is this something we need to do beforehand, and is would it be OK to create a "default" configmap when creating deployments that require an IP lease?
2. If the configmap still does not exists, what would be the best place to create and attach this confimap as a volume? On the deployment creation?
3. Does the kube client offer any functionality as of now to update a configmap?

Thanks for the help :)

[boz]

Is it possible to not support updating the value for now?

[andy108369]

the schedulingGates that have been mentioned during today's call https://kubernetes.io/docs/concepts/scheduling-eviction/pod-scheduling-readiness/

and the readinessProbe https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/

my idea with the readinessProbe is that we could use something like this:

readinessProbe:
  exec:
    command:
    - <a shell command that would make a query against the akash-service to check whether the IP has been allocated for this particular deployment>
  initialDelaySeconds: 5
  periodSeconds: 5

[PJEstrada]

From discussions on call:

Possible Options for Solving this

• One option could be waiting for the IP address before starting the Pod.

• Implement and endpoint that returns "metadata" about the workload. Some concerns on permissions, and getting lease information from other leases that are not. The endpoint can talk directly to kube, probably a sidecar.

• Using a readinnes probe, you can specific a simple query on the endpoint.

• Using a configmap to update the value of the ip addres

• New kuberenetes version 1.27. Has a new feature called schedule gate. Similar to a readiness probe.

Notes and Discussion

• If we use schedule gate, still going to close the lease if IP address location takes longer that the healthchecks.

• To be able to wait for the IP we'll need to make the healthchecks time a bit longer since IP addresses can take several minutes to be available.

• Having a service that you could query would also solve other issues like monitoring. But having a query service will require dealing with authentication and authorization with K8s, and also opens vector of attacks for the providers' cluster + we are using extra resources from the provider

Proposed solution for now

1. Deployment manager needs to have an extra parameter called ip address for example.
2. The provider monitor then extends the time for the lease to get the IP addres and wail until the addres is ready if it sees these parameters describe on step 1. on the workload.
3. We also need to somehow show it in the status of the deployment, that the lease is waiting for an IP and not unhealthy.

[PJEstrada]

Hey everybody. I'm continuing the fix for the issue of the env variables adition. Based on previous conversations. I wanted to validate if this plan makes sense to you all of you as this is my first time touching the codebase and I want to make sure I'm understanding the code structure.

1. What we need to do is to wait until the IPOperator has the IP Lease ready for a given OrderID. For that we can use the GetIPAddressStatus located at ip_operator_client.go in the cluster folder.

func (ipoc *ipOperatorClient) GetIPAddressStatus(ctx context.Context, orderID mtypes.OrderID) ([]ipoptypes.LeaseIPStatus, error) {
	path := fmt.Sprintf("ip-lease-status/%s/%d/%d/%d", orderID.GetOwner(), orderID.GetDSeq(), orderID.GetGSeq(), orderID.GetOSeq())
	req, err := ipoc.newRequest(ctx, http.MethodGet, path, nil)
	if err != nil {
		return nil, err
	}

	ipoc.log.Debug("asking for IP address status", "method", req.Method, "url", req.URL)
	response, err := ipoc.client.DoRequest(req)
	if err != nil {
		return nil, err
	}
	ipoc.log.Debug("ip address status request result", "status", response.StatusCode)

	if response.StatusCode == http.StatusNoContent {
		return nil, nil // No data for this lease
	}

	if response.StatusCode != http.StatusOK {
		return nil, extractRemoteError(response)
	}

	var result []ipoptypes.LeaseIPStatus

	decoder := json.NewDecoder(response.Body)
	err = decoder.Decode(&result)
	if err != nil {
		return nil, err
	}

	return result, nil
}

I can use a waitgroup until the status for the IP is ready. (ie no errors on the above function right?)

2.Once that happens I can proceed with the deployment as normal. The waiting logic will be placed on the Deploy() function inside cluster/kube/client.go.

3. All of the above needs to be conditioned on an attribute in the SDL. I planning to name this attribute something like waitForIps.

4. One the wait logic is finished, I can add the logic to inject the Ip Addresses on the container() function in the kube/builder/workload.go file so that it can be addded when the addEnvVarsForDeployment() is called. At this point, I should have the guarantees that the IP addresses from the ipOperator are already available so I can use the same function GetIPAddressStatus to get the actual IP address string.

Some questions:

1. Is there anything wrong with the above plan? Are there any considerations I need to keep in mind when implementing this?
2. How can I access the attributes flag from inside the cluster manager in cluser/manager.go I see the deploymentManager struct has a mgroup *manifest.Group but I'm not quite sure on what section of the SDL would be more appropriate to add this new attribute.
3. The GetIPAddressStatus() function returns a LeaseIPStatus array. However looking at the LeaseIPStatus struct, I don't quite see how to identify if the IP is ready or not. Can you help me understand how to determine if the IP is ready?

Thanks again for the help.

[flikites]

@boz could you please review @PJEstrada questions above at your convenience.

[boz]

Hi @PJEstrada, great write-up! Sorry for the delay.

In general, we try to avoid blocking. An event-based approach would be preferable.

• Emit an event when the IP Operator has an IP ready.
• You can listen for these events in cluster/service.go and pass the event to the manager as needed from there.
• Don't start the deployment right away if it should wait for an IP.

It's not clear to me why you need to add a new attribute - shouldn't all leases that have an IP wait for the IP to deploy?

The GetIPAddressStatus() function returns a LeaseIPStatus array. However looking at the LeaseIPStatus struct, I don't quite see how to identify if the IP is ready or not. Can you help me understand how to determine if the IP is ready?

It looks to me like that function only returns LeaseIPStatuses that are ready to use.

Again, sorry for the delay. I'll try to watch this more closely.