docs: add vzNAT and lima:shared to benchmark results

[AkihiroSuda]

Preview: https://deploy-preview-4166--lima-vm.netlify.app/docs/config/port/

[jandubois]

These numbers look very suspicious to me: why is the reverse speed so much different; that doesn't make sense. And 130GB/s feels unrealistic for anything that goes through a real TCP stack.

Maybe the high memory bandwidth on Apple's Pro/Max/Ultra chips makes this possible when transferring large blocks, but I doubt this is a realistic benchmark. On the regular (non-Pro) M3 CPUs with 24GB of RAM the memory bandwidth is just 100GB/s, so this speed would be completely impossible.

[jandubois]

This looks almost like they are just pushing a pointer to the buffer from the host to the guest and back, so these would be just memory write speeds. Does iperf3 even read the buffers?

It would still be impressive if this is just passing shared memory around with no copies, but I wonder how this could work while putting data into TCP buffers. Wouldn't that require a copy of the memory?

[jandubois]

I also cannot reproduce the numbers at all:

❯ l start --network vzNAT
...
❯ lima
jan@lima-default:/Users/jan/suse/lima$ sudo apt install iperf3
...
jan@lima-default:/Users/jan/suse/lima$ iperf3 -s
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
Accepted connection from ::1, port 40018
[  5] local ::1 port 5201 connected to ::1 port 40028
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   309 MBytes  2.59 Gbits/sec
[  5]   1.00-2.00   sec   314 MBytes  2.64 Gbits/sec
[  5]   2.00-3.00   sec   318 MBytes  2.67 Gbits/sec
[  5]   3.00-4.00   sec   317 MBytes  2.66 Gbits/sec
[  5]   4.00-5.00   sec   314 MBytes  2.63 Gbits/sec
[  5]   5.00-6.00   sec   317 MBytes  2.66 Gbits/sec
[  5]   6.00-7.00   sec   318 MBytes  2.67 Gbits/sec
[  5]   7.00-8.00   sec   310 MBytes  2.60 Gbits/sec
[  5]   8.00-9.00   sec   318 MBytes  2.67 Gbits/sec
[  5]   9.00-10.00  sec   318 MBytes  2.67 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  3.08 GBytes  2.65 Gbits/sec                  receiver

And on the host:

❯ iperf3 -c 127.0.0.1
Connecting to host 127.0.0.1, port 5201
[  5] local 127.0.0.1 port 55241 connected to 127.0.0.1 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.01   sec   313 MBytes  2.61 Gbits/sec
[  5]   1.01-2.00   sec   313 MBytes  2.64 Gbits/sec
[  5]   2.00-3.00   sec   318 MBytes  2.67 Gbits/sec
[  5]   3.00-4.00   sec   317 MBytes  2.66 Gbits/sec
[  5]   4.00-5.00   sec   314 MBytes  2.63 Gbits/sec
[  5]   5.00-6.00   sec   317 MBytes  2.66 Gbits/sec
[  5]   6.00-7.00   sec   318 MBytes  2.67 Gbits/sec
[  5]   7.00-8.00   sec   310 MBytes  2.60 Gbits/sec
[  5]   8.00-9.01   sec   320 MBytes  2.67 Gbits/sec
[  5]   9.01-10.00  sec   316 MBytes  2.67 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  3.08 GBytes  2.65 Gbits/sec                  sender
[  5]   0.00-10.00  sec  3.08 GBytes  2.65 Gbits/sec                  receiver

iperf Done.

This is on an M1 Ultra with 128GB of memory, which has 800GB/s memory bandwith.

So what do you have to do to get these extreme transfer rates?

[jandubois]

I get 62GB/s regardless of direction when I run iperf3 -s on the host as well. Could that have happened during the benchmark?

Or is the benchmark using eth0 instead of lima0?

[AkihiroSuda]

This is on an M1 Ultra with 128GB of memory, which has 800GB/s memory bandwith.

What is your macOS version?

iperf3 -s on the host as well.

Nope

Or is the benchmark using eth0 instead of lima0?

Nope

$ lima ip a show dev lima0
3: lima0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:55:55:99:a0:bf brd ff:ff:ff:ff:ff:ff
    altname enx52555599a0bf
    inet 192.168.64.9/24 metric 100 brd 192.168.64.255 scope global dynamic lima0
       valid_lft 3521sec preferred_lft 3521sec
    inet6 fd21:a20d:ee48:a4d5:5055:55ff:fe99:a0bf/64 scope global dynamic mngtmpaddr noprefixroute 
       valid_lft 2591939sec preferred_lft 604739sec
    inet6 fe80::5055:55ff:fe99:a0bf/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
$ iperf3 -c 192.168.64.9 -R
Connecting to host 192.168.64.9, port 5201
Reverse mode, remote host 192.168.64.9 is sending
[  5] local 192.168.64.1 port 50100 connected to 192.168.64.9 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  14.8 GBytes   127 Gbits/sec                  
[  5]   1.00-2.00   sec  15.2 GBytes   131 Gbits/sec                  
[  5]   2.00-3.00   sec  15.1 GBytes   130 Gbits/sec                  
[  5]   3.00-4.00   sec  15.2 GBytes   131 Gbits/sec                  
[  5]   4.00-5.00   sec  15.3 GBytes   131 Gbits/sec                  
[  5]   5.00-6.00   sec  15.2 GBytes   130 Gbits/sec                  
[  5]   6.00-7.00   sec  15.0 GBytes   129 Gbits/sec                  
[  5]   7.00-8.00   sec  15.2 GBytes   131 Gbits/sec                  
[  5]   8.00-9.00   sec  10.7 GBytes  92.1 Gbits/sec                  
[  5]   9.00-10.00  sec  9.84 GBytes  84.6 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   142 GBytes   122 Gbits/sec    0            sender
[  5]   0.00-10.00  sec   142 GBytes   122 Gbits/sec                  receiver

iperf Done.

[AkihiroSuda]

The memory bandwidth is 410GB/s according to https://support.apple.com/en-us/121553

[jandubois]

I see my mistake:

❯ iperf3 -c 127.0.0.1

However, I can't connect on vzNAT, and don't have time to figure it out right now:

❯ lima ip a show dev lima0
3: lima0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:55:55:d3:43:0c brd ff:ff:ff:ff:ff:ff
    altname enx525555d3430c
    inet 192.168.205.8/24 metric 100 brd 192.168.205.255 scope global dynamic lima0
       valid_lft 2677sec preferred_lft 2677sec
    inet6 fde4:c21c:e1a1:61ad:5055:55ff:fed3:430c/64 scope global dynamic mngtmpaddr noprefixroute
       valid_lft 2591940sec preferred_lft 604740sec
    inet6 fe80::5055:55ff:fed3:430c/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever

❯ iperf3 -c 192.168.205.8
iperf3: error - unable to connect to server - server may have stopped running or use a different port, firewall issue, etc.: No route to host

Any ideas?

[jandubois]

The memory bandwidth is 410GB/s according to https://support.apple.com/en-us/121553

Yes, on the M4 Max, I was talking about the non-Pro version, which has 100GB/s (M3), or 120GB/s (M4).

The details of the benchmark setup were collapsed, so I didn't see them at first. I think it may be worth highlighting that this performance cannot be expected on the base models.