How to send packets (in datagram mode) immediately?

[a-andre]

Hi,
in my program all data that is received via an interface should be forwarded (in datagram mode) as payload of a QUIC packet from one end to the other. That works as expected, the only problem is, that the data received via the interface is send with a delay of up to 10s.
I found delay_max in sockloop.c which influences the behaviour.

picoquic/picoquic/sockloop.c

Line 723 in 27d9a8d

int64_t delay_max = 10000000;

Can I somehow force sending the data immediately (without modifying delay_max in sockloop.c)?

[hfstco]

As far as I know Picoquic is waiting in picoquic_packet_loop_select() until it receives data from the socket or delta_t time passed.

picoquic/picoquic/sockloop.c

Lines 615 to 628 in 27d9a8d

	if (delta_t <= 0) {
	tv.tv_sec = 0;
	tv.tv_usec = 0;
	} else {
	if (delta_t > 10000000) {
	tv.tv_sec = (long)10;
	tv.tv_usec = 0;
	} else {
	tv.tv_sec = (long)(delta_t / 1000000);
	tv.tv_usec = (long)(delta_t % 1000000);
	}
	}
	ret_select = select(sockmax + 1, &readfds, NULL, NULL, &tv);

You can wake up the network thread with picoquic_wake_up_network_thread() manually.

picoquic/picoquic/sockloop.c

Line 1262 in 27d9a8d

int picoquic_wake_up_network_thread(picoquic_network_thread_ctx_t* thread_ctx)

Another option is set a custom delta_t inside the picoquic_packet_loop_time_check callback.

picoquic/picoquic/sockloop.c

Line 828 in 27d9a8d

ret = loop_callback(quic, picoquic_packet_loop_time_check, loop_callback_ctx, &time_check_arg);

[huitema]

If your application wants real time behavior, it should use the "just in time" API, as explained in doc/send_receive_data.md, look for "sending datagrams just in time":

• when you receive a packet to be forwarded, add it to the internal queue and set the datagram readiness as active using the API picoquic_mark_datagram_ready.
• when you receive the callback picoquic_callback_prepare_datagram, prepare the buffer as explained in the doc. In the call to picoquic_provide_datagram_buffer_ex, set the is_still_active flag to 1 if there are more datagrams in the queue, to 0 if not.

If your implementation is multi-threaded (it probably should be), wake up the background thread using picoquic_wake_up_network_thread as explained in the Asynchronous operation section of doc/socket_loop.md. Be careful that the calls to picoquic_mark_datagram_ready should be done from within the picoquic thread.

[a-andre]

Thanks for the recommendations.

I'm already using the "just in time" API, sorry I forgot to mention that in the question.

To start the packet loop, I call picoquic_packet_loop_v2 and with the picoquic_callback_ready event, the thread that reads data from an interface is started. After adding that data to a queue, picoquic_mark_datagram_ready is called within that thread to trigger picoquic_callback_prepare_datagram.

Not sure yet, from where within the picoquic thread to call picoquic_mark_datagram_ready instead.

I'll look into using picoquic_start_network_thread and with it picoquic_wake_up_network_thread.

[a-andre]

I could fix the problem with switching from picoquic_packet_loop_v2 to picoquic_start_network_thread and calling picoquic_wake_up_network_thread directly after picoquic_mark_datagram_ready in the read-from-interface loop, though not sure if the latter is correct.

[huitema]

Yes, that's correct. Almost. The call to picoquic_mark_datagram_ready is not thread safe. You could have a race condition between marking the datagram ready in your read-from-interface loop and marking nothing more to send in the picoquic code after sending the previous datagram. It would be better to do four steps:

• in the read-from-interface loop, queue the newly read datagram in a thread safe way
• just after that, call picoquic_wake_up_network_thread
• the network thread will wake up and poll the application with a loop callback picoquic_packet_loop_wake_up. In that loop callback, you should test your queue of datagrams to be sent in a thread safe manner, and call picoquic_mark_datagram_ready if the queue is full.
• in the protocol callback picoquic_callback_prepare_datagram, dequeue the datagram from you internal queue in a thread safe manner.

There are many ways to manage the queue between your read-from-interface loop and the socket loop in a "thread safe" manner. Some involve locks and wait, but for an application like yours I would try to use a lock-free circular buffer.

[a-andre]

Thanks.

[huitema]

The root cause of your 10ms delay is because you are running single thread. In the single thread model, the network loop will call the application just before going to call select, and then after receiving or sending a packet. As @hfstco explains, the application can override the default timer by setting a shorter timer in response to the picoquic_packet_loop_time_check callback. You could make sure there that the wait time is set to a short value. Of course, doing that means engaging in "active polling", and that's not ideal for perf. That's why a multi-threaded model will work better.