[syslinux] PXELINUX 6.03 / vesamenu.c32 hang

[Ady]

> Hello,
> 
> I'm experiencing an issue where if I use vesamenu.c32 with PXELINUX 6.03 when network booting a virtual machine hosted on VMware ESXi 6.0, it hangs (the version message is printed and then nothing - the VM has to be powered off and back on again).
> 
> This happens no matter if I choose the E1000 network interface or the VMXNET3 network interface when creating the VM.
> 
> The same problem does not happen if I change the configuration to use menu.c32 instead of vesamenu.c32.

One possibility: double check that all your c32 modules are originated 
from the same package / version / build.

Mixing versions (sometimes inadvertently) of Syslinux-related files is 
one source of strange behaviors. Since vesamenu.c32 has more c32 
dependencies (lib*.c32) than menu.c32, I would suggest this double 
check as first step. Please check that you are using the c32 modules 
for the adequate platform (BIOS / EFI32 / EFI64).

It is also possible that ESXi is having some problem / conflict / 
incompatibility with the graphics mode of vesamenu.c32. This happens to 
be also the case with some UEFI implementations.

I seem to remember that VMWare has some customizations in their own c32 
modules. This is another type of "mix" that is unsupported, because the 
customizations are not public.
 
> 
> Both vesamenu.c32 and menu.c32 came from the same zip file as the PXELINUX binary. All PXELINUX related files came from the "bios" area of said zip file, and the VM is also set to use BIOS booting rather than EFI (this is because there are many older machines in our organisation which either do not support or are not set up for EFI booting).
> 
 
That relates to part of my above comments. Yet, I would suggest double 
checking the files against those from the zip under the bios 
subdirectory tree.
 
 
> I do not experience this problem when booting a physical machine.
> 
 
A compatibility issue with ESXi in particular, and/or any of my prior 
comments, can still be relevant.
 
 
> Our setup is as follows:
> 
> PXE server: Windows Server 2008 R2 running Windows Deployment Services
> DHCP server: not on the same server, WDS is authorised in DHCP
> PXELINUX version: 6.03 (but it was originally 4.04, I upgraded to see if it was some kind of bug which had been fixed in a later version)
 
So, is the same exact behavior seen when using Syslinux 4.04? What 
about 4.07?
 
 
> PXELINUX files in Boot/x64 and Boot/x86:
> 
> pxelinux.cfg\default
> chain.c32
> ldlinux.c32
> libcom32.c32
> libmenu.c32
> libutil.c32
> memdisk
> menu.c32
> pxelinux.com (renamed from pxelinux.0 as per instructions here http://www.syslinux.org/wiki/index.php?title=WDSLINUX)
> vesamenu.c32
> 
> Contents of pxelinux.cfg\default:
> 
> DEFAULT      vesamenu.c32
> #DEFAULT      menu.c32
> PROMPT       0
> NOESCAPE     1
> ALLOWOPTIONS 0
> # Timeout in units of 1/10 s
> TIMEOUT 300
> MENU MARGIN 10
> MENU ROWS 16
> MENU TABMSGROW 21
> MENU TIMEOUTROW 26
> #MENU COLOR BORDER 30;44                    #20ffffff #00000000 none
> #MENU COLOR SCROLLBAR 30;44                              #20ffffff #00000000 none
> #MENU COLOR TITLE 0                   #ffffffff #00000000 none
> #MENU COLOR SEL   30;47                            #40000000 #20ffffff
> #MENU BACKGROUND MyMenuBackgroundPicture640x480.jpg
> MENU COLOR SCREEN                   37;44     #ffffffff #ff00007f none
> MENU COLOR BORDER                  37;44     #ffffffff #ff00007f
> MENU COLOR PWDBORDER        37;41     #ffffffff #ff7f0000 none
> MENU COLOR PWDHEADER         37;41     #ffffffff #ff7f0000 none
> MENU COLOR PWDENTRY                            37;41     #ffffffff #ff7f0000 none
> MENU COLOR TITLE                         37;44     #ffffffff #ff00007f
> MENU COLOR TIMEOUT                37;44     #ffffffff #ff00007f none
> MENU COLOR TIMEOUT_MSG   37;44     #ffffffff #ff00007f none
> MENU COLOR UNSEL                      37;44     #ffffffff #ff00007f none
> MENU COLOR SCROLLBAR            37;44     #ffffffff #ff00007f
> MENU TITLE PXE Boot Menu
> #---
> LABEL wds
> MENU LABEL Windows Deployment Services
> MENU PASSWD [redacted]
> KERNEL pxeboot.0
> #---
> LABEL memtest
> MENU LABEL memtest86+ v4.20
> KERNEL memtest
> #---
> LABEL seatools
> MENU LABEL SeaTools for DOS v2.23
> LINUX memdisk
> INITRD seatools.ima
> #---
> LABEL wdcdlg
> MENU LABEL Western Digital Data LifeGuard Diagnostics v5.04f
> LINUX memdisk
> INITRD wdc_diag.img
> #---
> LABEL dban2
> MENU LABEL DBAN v2.2.6 Beta
> MENU PASSWD [redacted]
> KERNEL dban.bzi
> APPEND nuke="dwipe" silent
> #---
> LABEL pmagic
> MENU LABEL PartEd Magic 6.3
> MENU PASSWD [redacted]
> LINUX pmagic/bzImage
> INITRD pmagic/initramfs
> APPEND edd=off load_ramdisk=1 prompt_ramdisk=0 rw vga=normal loglevel=9 max_loop=256
> #---
> LABEL Abort
> MENU LABEL AbortPXE
> KERNEL abortpxe.0
> #---
> LABEL local
> MENU DEFAULT
> MENU LABEL Boot from Harddisk
> LOCALBOOT 0
> 
> Can anyone suggest what might be the problem?
> 
> Thanks,
> Dan Jackson (Lead ITServices Technician).
> Long Road Sixth Form College
> Cambridge, UK.

I would suggest trying a simplified pxelinux.cfg/default, with at least 
2 entries:

UI vesamenu.c32
TIMEOUT 300
LABEL memtest
KERNEL memtest
LABEL wds
KERNEL pxeboot.0
LABEL Abort
KERNEL abortpxe.0
LABEL local
LOCALBOOT 0


Please note that I used a non-pxe first entry (memtest), no passwords, 
and no non-essential directive at all (not even a background image). I 
also used the UI directive instead of the DEFAULT one.

We have also seen issues with some screen resolutions, and with using 
certain types of backgrounds (jpg vs. png).

Testing with 6.04-pre1 might be of interest (although, if the same 
behavior is seen with 4.04, I would doubt that using 6.04-pre1 would 
show a different behavior than 6.03 in this case).

Regards,
Ady.