[syslinux] EFI & PXE-booting: very slow TFTP performance on a VMWare test setup

[Gene Cumm]

On Thu, Jun 18, 2015 at 1:18 PM, Gene Cumm <gene.cumm at gmail.com> wrote:
> On Wed, Jun 17, 2015 at 2:20 PM, Oscar Roozen via Syslinux
> <syslinux at zytor.com> wrote:
>>>> Are there any known problems with the performance of TFTP in (U)EFI
>>>> environments in general or maybe just on VMWare?
>>> I don't believe so.  I use Workstation 10.0.2 on Ubuntu 12.04 for the
>>> moment.
>>
>> <stupid question> and have EFI enabled? </stupid>
>>
>>> I hope this is a hex-core host with 6+GiB of RAM.
>>
>> Fedora runs on an AMD A8-6600K at 3Ghz and has 8G of RAM. Ubuntu runs
>> on an Intel i7-2600K at 3.4Ghz with 16G RAM. I think they perform at
>> least "good enough" to quikly boot and install in BIOS mode. The
>> trouble only starts when I toggle the EFI switch.
>>
>>
>>> Have you considered single vCPU client VMs?
>>
>> Yes. That's what I started with. But then I saw this piece of code in
>> efi/udp.c and thought it would be wise to enable an extra CPU to
>> receive the UDP packet. A single CPU has a 100% load here:
>>
>>     start = jiffies();
>>     while (cb_status == -1) {
>>         /* 15ms receive timeout... */
>>         if (jiffies() - start >= 30) {
>>             if (jiffies() - start >= 60)
>>                 printf("Failed to cancel UDP\n");
>>
>>             uefi_call_wrapper(udp->Cancel, 2, udp, &token);
>>             printf("core_udp_recv: timed out\n");
>>         }
>>         uefi_call_wrapper(udp->Poll, 1, udp);
>>     }
>>
>> Just to rule out some stupid missed interrupt or so, I gave it an extra CPU.

Boot loader environments rarely benefit from a second CPU.  The best
way to not miss an interrupt in a VM boot environment is to not
overcommit.

>>> I presume you disabled VMware's isc-dhcp on vmnet1?
>>
>> Yes. I should have mentioned that. Found out the hard way.. . ;-)
>>
>>
>>> Good idea to serve from a subtree.
>>
>> Thanks. We need to support both modes, so we now have /tftpboot/bios
>> and ../efi64.
>>
>>
>>>> # make spotless
>>>> # make bios
>>>> # make efi64
>>>
>>> 'make bios' should be unnecessary.
>>
>> I have to test both modes, but while debugging I mostly did just "make
>> efi64 && install-stuff-to-/tftpboot/efi64".
>>
>>>> The server does just fine. It typically reacts within 0.000100 seconds.
>>>> The client also starts out fine, reacting within about 0.000250 seconds,
>>>> but this delay slowly increases.
>>>
>>> This makes me wonder the source and there's at least 3 factors: VMware UEFI
>>> on multiple vCPU, memory or other leak, a strange bug in Syslinux.
>>
>> This also happens on a single vCPU.
>>
>> I did check efi/udp.c over and over again (especially the event
>> handling) but I don't see a place that could cause this behaviour.
>>
>> One thing I did just now is a bit - ehm - hackish. I symlinked
>> /tftpboot/efi64/syslinux.efi to /dev/zero and tried booting from that.
>> I wanted to see if the "built-in" tftp-client of VMWare's UEFI
>> firmware also showed this behaviour, but it didn't. Well, it got a bit
>> slower, but not as much as the client from syslinux.
>>
>>
>>> I recall finding the UEFI 2.4 spec really for free.
>>
>> I recall running into some NDA that had to be manually approved before
>> I could download anything. I also remember preferring to go on hacking
>> in more debug statements to find out where the delay comes from.
>> Unfortunately I still have no clue and no specs as a result. Even with
>> the specs, my guess is it will take some time understanding them. I'm
>> just becoming a bit more comfortable reading syslinux code.
>>
>>> By the way, what version/commit?
>>
>> A fresh syslinux-6.03.tar.gz from kernel.org.
>> I also tried 81ad566f155fac31089fde69c87059b217e7e9b6 with the same results.
>
> I see the nice exponential-like decay of IO rate :(

Consider one of the following workarounds:

1) HTTP/FTP.  Loading individual files over HTTP/FTP would be
suggested for a large file anyways.  For your situation, I'd suggest
trying to use DHCP option 210, either embedded in option 43 or
hard-coded into syslinux.efi as an after-DHCP option with
pxelinux-options.  Setting the plain option would require forcing your
dhcpd to always insert it (isc-dhcpd after v3 I think can do this) as
no PXE client will ever request it.  I have http://myip/tftp/ pointing
at my tftpd root just for this purpose.

2) Load multiple files.  If the initrd is a CPIO (compressed or not)
for Linux initramfs, split it to multiple files and only put a portion
of the file system into each.  I often load 3-10 GZipped CPIO archves,
each one an independent CPIO file that's extracted over the last.

-- 
-Gene