Error when probing primary path after drop in multipath

[Adithya1435]

Hello, I'm currently developing an application in Windows utilizing the picoquic library for multipath. I have used picoquicdemo.c as a reference for configuring and enabling multipath support. This is my current implementation:

Check if there are available connection ids using picoquic_obtain_stashed_cnxid() -> Find all the interfaces and store its ip (Custom function) 
->  Iterate through this list and probe new paths for IPs which are not yet probed using picoquic_probe_new_path_ex()


if (picoquic_obtain_stashed_cnxid(cb_ctx->cnx, path_cnt) != NULL) {
    find_all_interfaces();
    prepare_multipath_config_();

    cb_ctx->nb_alt_paths = 0;

    struct sockaddr_storage path0_addr = { 0 };

    if (picoquic_get_path_addr(cb_ctx->cnx, 0, 1, &path0_addr) != 0) {
        printf("Cannot use multipath. Cannot get address of path 0");
    }

    char primary_ip[50];

    inet_ntop(AF_INET, &(((struct sockaddr_in *)&path0_addr)->sin_addr), primary_ip, 50);
    printf("Primary IP: %s\n", primary_ip);

    for (int i = 0; i < api_interface_count; i++) {

        if (path0_addr.ss_family != api_interface_list[i].alt_ip.ss_family) {
            continue;
        }

        memcpy(&cb_ctx->client_alt_address[cb_ctx->nb_alt_paths], &api_interface_list[i].alt_ip, sizeof(struct sockaddr_storage));
        cb_ctx->client_alt_if[cb_ctx->nb_alt_paths] = api_interface_list[i].index;
        cb_ctx->client_alt_state[cb_ctx->nb_alt_paths] = 0;
        cb_ctx->nb_alt_paths++;
    }
    
    ret = client_create_additional_path(quic, cb_ctx);
    
    printf("Probe done\n");
    cb_ctx->multipath_probe_done = 1;

}


int client_create_additional_path(picoquic_quic_t *quic, sample_client_ctx_t* cb_ctx){


	for (int i = 0; i < cb_ctx->nb_alt_paths; i++) {

		if (api_interface_existed(api_interface_list[i])) { 
			continue;
		}		
		if (strcmp(api_interface_list[i].ipaddr, primary_ip) == 0) {
			printf("Skipping primary probing \n");
			continue;	
		}	

		if (path0_addr.ss_family != 0 && api_interface_list[i].alt_ip.ss_family != path0_addr.ss_family) {
					printf("Cannot add new path, wrong address family, %d vs. %d\n", 
						api_interface_list[i].alt_ip.ss_family, path0_addr.ss_family);
						
					continue;
		}

		else {
			if ((ret = picoquic_probe_new_path_ex(cb_ctx->cnx, (struct sockaddr*)cb_ctx->server_address_copy,
						(struct sockaddr*)&api_interface_list[i].alt_ip, api_interface_list[i].index, picoquic_get_quic_time(quic), 0)) != 0) {
						printf("Probe new path failed with exit code %d\n", ret);
					}
					else {
						printf("New path added, total path available %d, new path added is %s\n", cb_ctx->cnx->nb_paths,  api_interface_list[i].ipaddr);	
						printf("\n");								
					}


		}
    }
}

While testing edge cases, I found one scenario that needs to be addressed. When multiple interfaces are available initially and the primary interface (path id 0) drops or gets removed mid-connection and later comes back online, picoquic_probe_new_path_ex() returns error code 1089 (PICOQUIC_ERROR_PATH_CID_BLOCKED) when attempting to probe this primary interface. However when any non-primary interface drops and comes back, it gets added without any issue and no such error is observed.

I also noticed something else worth mentioning. Suppose there are two interfaces initially with path IDs 0 and 1 respectively. If the second interface, with path ID 1, drops and later comes back, it gets assigned to path ID 2 instead of reclaiming its previous ID. This suggests that the path itself is abandoned. So even path ID 0 gets abandoned as well when the primary interface drops?

My main question is whether this is expected behavior, and if so, is there a recommended way to handle the primary interface dropping and reconnecting without hitting the PICOQUIC_ERROR_PATH_CID_BLOCKED error. I would appreciate any advice or suggestions on how to manage this situation. Thank you

[huitema]

Yes, the path 0 gets abandoned when the primary interface drops.

The path object is identified by a path id. If the interface drops and the path is abandoned, the path id cannot be used anymore. This is in conformity with the multipath draft.

[Adithya1435]

Thank you for the clarification. It is now clear that when the primary interface drops, path 0 gets abandoned and its path ID cannot be reused.

However, is there a way to assign a new path for primary interface when it reconnects without encountering the PICOQUIC_ERROR_PATH_CID_BLOCKED error? This would help in maintaining multipath support in cases where primary interfaces temporarily disconnect and come back. Kindly let me know your thoughts on this.

[huitema]

First, let's clarify that there is no concept of "primary path" in the QUIC multipath draft. There is a concept of "path status", which can be "Available" or "backup". Applications can use APIs to change this status:

int picoquic_set_path_status(picoquic_cnx_t* cnx, uint64_t unique_path_id, picoquic_path_status_enum status);

When the old interface comes back, you can create a new path on that interface using the API:

int picoquic_probe_new_path_ex(picoquic_cnx_t* cnx, const struct sockaddr* addr_peer,
    const struct sockaddr* addr_local, int if_index, uint64_t current_time, int to_preferred_address);

This new path will be created with a new path ID, not related to the path ID used by what you call the "primary interface". The call will succeed if there is an available path_id to use. It will fail if this is not possible, resulting in errors such as:

• PICOQUIC_ERROR_MIGRATION_DISABLED if the server is not supporting migrations
• PICOQUIC_ERROR_PATH_NOT_READY if the handshake is not finished yet
• PICOQUIC_ERROR_PATH_LIMIT_EXCEEDED if there are more paths in use than negotiated with the server
• PICOQUIC_ERROR_PATH_ID_BLOCKED if the next available unique_path_id is not yet authorized by the server
• PICOQUIC_ERROR_PATH_CID_BLOCKED if the server has not yet provided connection IDs for the next unqiue_path_id.

That said, we do have an API issue. The path related API identify the path using a unique_path_id, but the discovery of this unique path id is a bit convoluted. I can see a few issues:

• the parameter to_preferred_address is confusing. It is meant to be used if the server returns a "preferred address" parameter during the initial handshake. At that point, the stack should immediately try to migrate the 'default path` created during the handshake to that new address. The parameter should never be used after that. If the application sets it, the results are unpredictable, and probably wrong.

• in case of success, the application can only discover the unique_path_ID of the new path by using internal APIs -- looking at the unique_path_id parameter in the last created path.

• the concept of success is questionable. It means that the probe has been sent, but does not mean that the new path can be used. This only happens after the application receives the callback picoquic_callback_path_available, which itself only happens if the application enables path callbacks using picoquic_enable_path_callbacks or picoquic_enable_path_callbacks_default.

• there is no simple way to "wait until CIDs are available". Maybe we need a new callback.

I am open to suggestions on how to fix all that. Maybe you could open a specific issue to discuss API extensions.

[akh7177]

Hello @huitema,

Thanks for the detailed reply.

We indeed are using picoquic_probe_new_path_ex (with picoquicdemo.c as a reference) and have run into this specific error.

Here is the scenario:

• We have two interfaces, A (creating the initial path0) and B (creating path1).

• If interface B goes down, path1 is abandoned. When B comes back up, a new path (path2) is successfully created and multipath continues to work perfectly. This can even be repeated without issue.

• However, if interface A goes down, path0 is abandoned and traffic continues on path1 as expected. The problem occurs when interface A comes back up. Calling picoquic_probe_new_path_ex to re-add it fails with PICOQUIC_ERROR_PATH_CID_BLOCKED.

As you noted, this error suggests a CID isn't available and this leads us to the main question:

Why can interface (B) be re-added, while the initial path's interface (A) cannot be? It seems there are sufficient CIDs when B reconnects, so is it really a CID shortage that's causing this issue?

[huitema]

@akh7177 @Adithya1435 Thanks for raising this discussion and following on. The original design is indeed that if a path is closed, the application cannot reuse the same path ID, need to open another one. Problem there was a bug (issue #1918) in the way the stack tested whether opening that path should be possible. This is fixed in PR #1919.

The expected behavior is:

• application open a couple of paths (e.g., path 0 and path 1 on interface A and B)
• interface A becomes unavailable. The network stack receives an "address unreachable" error, which causes the code to abandon path number 0.
• some times later, interface A becomes available again. The application uses picoquic_probe_new_path to create a new path on that interface. The stack will automatically pick a new path ID.

We now have tests to verify that specific scenario, and after fixing the bug these tests are passing.