[syslinux] lpxelinux.0 issues with larger initrd.img files from RHEL >= 7.5 on UCS servers?

[Mathieu Chouquet-Stringer]

	Hello,

On Tue, Jun 18, 2019 at 05:31:17PM -0700, hpa at zytor.com wrote:
> Which servers, what threshold, what clients, what about pxelinux.0?

All affected servers so far are Cisco UCS B200 M3 blade servers.

The threshold seems to be around 50MB, I haven't tested precisely but
54525200 bytes is enough to trigger the reboot while 49812292 isn't
(what I did was to recompress the initrd with a higher compression
setting).

I tried today pxelinux.0 instead of lpxelinux.0 and it behaves the same
way: if the initrd is "too big", the server reboots.

What do you mean by "clients"?

Please find below what I initially wrote in my email, let me know if or
how I can help debug that.

Cheers,

>  	Hello,
>  
>  I am using lpxelinux.0 (latest stable version 6.03, using the official binaries
>  from kernel.org) to kickstart servers as http transfers really helps over links
>  with poor latencies... These servers are being booted in legacy mode,
>  not in UEFI.
>  
>  This has worked very well until recently. Starting with RHEL 7.5, on
>  some servers, we would see the machine rebooting while pxelinux is in the
>  middle of downloading the initrd.img file.
>  
>  I quick workaround was to tell people to boot/kickstart using a previous
>  minor (last working was 7.4): the install process taking care of updating to
>  the latest version.
>  
>  I experienced the same issue again yesterday and had time to think about
>  it again.
>  
>  On rhel 7.4, the last working version in my case, the file is that big:
>     -rw-r--r--   1 root     root     49763300 Dec  1  2017 initrd.img
>  
>  I saw on 7.5 and 7.6 they're slightly larger, respectively:
>     -r--r--r--   1 root     root     54525200 Mar 22  2018 initrd.img
>  and
>     -r--r--r--   1 root     root     54799220 Oct 10  2018 initrd.img
>  
>  Because I have no trace on the screen to explain the reboot (same thing
>  with a recorded session over a serial console), I was like: what if the
>  size is a factor? Given I had nothing else to try, I looked at the files
>  and saw they were compressed with xz.
>  
>  So I wondered, what if I compressed them more (I guessed they were compressed
>  with the default compression preset)? After uncompressing and compressing with
>  xz -9 -C crc32, here's what I get for 7.5 and 7.6 respectively:
>     -r--r--r--   1 root     root     48823576 May 21 11:45 initrd.img
>  and
>     -r--r--r--   1 root     root     49812292 May 21 11:27 initrd.img
>  
>  Pretty close to what I had in 7.4. And to my suprise, with these smaller
>  files, lpxelinux doesn't reboot while downloading the file over http. The
>  kernel boots up and the OS is installed properly....
>  
>  I haven't had the time to reproduce this issue over regular
>  pxelinux/tftp so I don't know if it's just tied to lpxelinux/http or
>  not.
>  
>  Also, so far this bug only seems to be triggered on some  Cisco UCS
>  servers such as UCSB-B200-M3 like the one described below. So maybe it
>  could be related to BIOS or memory maps, I am not sure!?
>  
>  I'd hate to go back to tftp because the switch to http was such a huge
>  step forward.
>  
>  Is there something I could do or provide to help debug this issue? I
>  read the "Hardware Compatibility" and "Common Problems" pages on the
>  wiki and found nothing close to what I'm seeing. I started reading the
>  "Development/Debugging" but while I could use "COM32 debug.c32" to get
>  more details, but I don't know which functions I should be tracing?
>  
>  Please let me know.
>  
>  Cheers,
>  Mathieu
>  
>  BIOS Information
>             Vendor: Cisco Systems, Inc.
>             Version: B200M3.2.2.6f.0.052120182033
>             Release Date: 05/21/2018
>             Address: 0xF0000
>             Runtime Size: 64 kB
>             ROM Size: 4096 kB
>             Characteristics:
>                     PCI is supported
>                     BIOS is upgradeable
>                     BIOS shadowing is allowed
>                     Boot from CD is supported
>                     Selectable boot is supported
>                     BIOS ROM is socketed
>                     EDD is supported
>                     5.25"/1.2 MB floppy services are supported (int 13h)
>                     3.5"/720 kB floppy services are supported (int 13h)
>                     3.5"/2.88 MB floppy services are supported (int 13h)
>                     Print screen service is supported (int 5h)
>                     8042 keyboard services are supported (int 9h)
>                     Serial services are supported (int 14h)
>                     Printer services are supported (int 17h)
>                     ACPI is supported
>                     USB legacy is supported
>                     BIOS boot specification is supported
>                     Targeted content distribution is supported
>                     UEFI is supported
>             BIOS Revision: 4.6
>  
>  System Information
>             Manufacturer: Cisco Systems Inc
>             Product Name: UCSB-B200-M3
>             Version: 1
>             Serial Number: MYSERIALNUMBER
>             UUID: SOMEUUID
>             Wake-up Type: Other
>             SKU Number:
>             Family:
-- 
Mathieu Chouquet-Stringer                               m+syslinux at thi.eu.com
            The sun itself sees not till heaven clears.
	             -- William Shakespeare --