[syslinux] PXELINUX: how to debug initrd corruption?

[Gene Cumm]

On Jan 8, 2012 5:53 PM, "Andy Walls" <awalls at md.metrocast.net> wrote:
> On Sat, 2012-01-07 at 22:27 -0500, Gene Cumm wrote:
> > On Jan 7, 2012 7:15 PM, "Andy Walls" <awalls at md.metrocast.net> wrote:
> > > Hi All,
> > >
> > > I have a system, that, ~80% of the time, fails to properly boot using
> > > pxelinux.0 as provided by RHEL 5.5 (syslinux 3.11, I think).  The
> > > gripe, before the console messages stop, is a message that comes from
> > > the linux kernel complaining the compressed ram drive image is bad:
> > > "invalid compressed format (err=[small number])".
> > >
> > > I have two questions:
> > >
> > > 1. How should I go about debugging the problem to find the root cause of
> > > the initrd image corruption?
> > >
> > > I've examined the TFTP transfers with Wireshark; they appear OK.
> > > I have not tried a more modern pxelinux.0 yet.
>
> Hi Gene,
>
> I won't have physical access to the system again in about a week or so.
> (I was asking the questions to ensure I had the right set of tools and
> tests when I get access to it again.)

Sounds like we may want to take the "shotgun" approach: Start with
more questions/possibilities than might be normal.

> However, I'll try to answer the best I can from memory:
>
> > How big are the kernel/initrd individually?  How much RAM does the system
> > have?  What tftpd?  Some tftpds don't handle rollover and will refuse to
> > transmit or truncate.
>
> The init ram drive is < 32MB when uncompressed IIRC, and the compressed
> kernel is smaller than that.  This is an NFS root setup, so I'm not
> trying to TFTP over the whole system.  Also, about 1 out of every 5
> times, the system does boot completely sucessfully.

Could you be a lot more precise on how large they are as transferred?

> I'm using the stock in.tftpd on the server.  I can switch to atfpd or
> tfpd-hpa as a troubleshooting step.  From the list archives, I see that
> atfpd doesn't handle rollover, but AFAIK, I'm not near the 1408*65535 =
> 87.999 MiB limit.

But you forgot that TFTP defaults to a smaller number, around 512B,
iirc.  I'd get the package name and version that provides the tftpd.
I'd also consider a packet capture to see if the tftpd negotiates the
length option.

> > > 2. Can anyone provide a description or simple diagram of what regions of
> > > system memory PXELINUX uses?
> > >
> > > From the linker file, I know that .earlybss starts at 0x800 (or 0x1000),
> > > and a number of sections directly follow it.
> > > I also realize some allocated regions are system specific, depending on
> > > what the BIOS returns in its E820 map.
> > > I am, not surprisingly, interested in the regions where PXELINUX stores
> > > the kernel and initramdisk images after TFTP-ing them over.
> >
> > PXELINUX itself stays in the first 1 MiB.  With regards to e820, I know the
> > current version does pay attention to these regions to fail loading.
> > linux.c32 will also take these into account to be more intelligent.  I
> > believe in at least some scenarios, kernel/initrd start at 1 MiB.  However,
> > 3.11 is old enough I can't speak for its internals.
> >
> > Any chance you could use meminfo.c32 (not in 3.11), screenshot or
> > transcibed?
>
> The next chance I get, I can run that on the problem system.  Though,
> since the Linux kernel actually boots, before failing to decompress the
> ram disk, I can capture the E820 map info the kernel emits.

Should be valid (hopefully).

> My hypothesis is that the EFI firmware (proving BIOS services), an
> option ROM, and/or the PXE boot agent is not updating the E820 map or
> providing accurate memory usage information, and PXELINUX ends up
> colliding with one of their data regions when storing the kernel and
> initrd image.

I'd consider collecting the make and model of the system and all of
its BIOS/firmware versions, along with knowledge of what the current
version is.  I'd also consider using sysdump.c32 and hdt.c32 to make
dumps of what they can see for system information.  If you have an
area for tftp upload, consider:

sysdump.c32 tftp <filename> [tftp_server]
(ie sysdump.c32 tftp dump1 10.1.1.1 )
hdt.c32 auto='dump;exit;' [tftp_ip=IP] [dump_path=PREFIX]

HDT's full output filename can't be adjusted at this time.  See
http://www.hdt-project.org/wiki/Dump for more information.  I'd regard
the output of both of these as private as it will likely contains
things like IP addresses, system serial numbers and MAC addresses.

> My fallback hypothesis is I'm running into an old PXELINUX bug.  Maybe
> something like this I found in the git log:
>
>        commit dd01ec19a62c769e37416878dbe63989d2526660
>        Author: H. Peter Anvin <hpa at zytor.com>
>        Date:   Sun Feb 14 15:09:04 2010 -0800
>
>            bootsect.inc: change 100000h -> free_high_memory
>
>            We can't load stuff at 100000h... that will overwrite the PM code.
>            That is what free_high_memory is for.
>
>
>
> > I'd also advise boot testing with a more current version.
>
> Agree.  I plan to go in with the latest git clone from:
>
>        git://git.kernel.org/pub/scm/boot/syslinux/syslinux.git

Only for testing.  If you want to move away from the distro-provided
version, I'd stick with using the precompiled binary/source archive
rather than utilizing a untested and possibly broken toolchain.  4.05
would be the most current full release.  4.10-pre17 and 4.06-pre1 are
the most current betas.  There may also be some other key versions if
these don't work.

> It sounds like many tools for examining the system are in the com32
> directory.  Are there any tools in there that will not work with a
> serial console?

Using sysdump.c32 to send the data over the serial port is the only
one that I can think of.  I'm not sure about hdt.c32 when a serial
console is used.  Is it a normal serial port that you utilize
Syslinux's serial output options or a console redirected over serial?

--
-Gene