[syslinux] Stumped by extlinux After OS Upgrade

[John.Florian at dart.biz]

Short version: It works!  Many thanks for the huge help.
Long version: inline below.

> From: Gene Cumm <gene.cumm at gmail.com>

> >>  If you're attempting to use an old
> >> version of EXTLINUX to patch the extlinux.sys of a new version, it
> >> probably won't work the way you want.
> > I was afraid I'd hear that.
> 
> You'll probably get the "new" file replaced with a copy from the old 
version.

Indeed, that is what I found.

> >> I think "extlinux --update"
> >> also refreshes the VBR (Volume boot record), which contains a pointer
> >> to the extlinux.sys file.
> >>
> >> I normally take the approach that if the core
> >> binary(extlinux.sys/ldlinux.sys) is moved (including the directory it
> >> sits in), renamed, or modified, it should be redone with --update for
> >> maximum reliability.  I can't say for certain if it's needed.
> >
> >
> > I wish it were not necessary, but given the behavior I suspect it 
must.
> > extlinux doesn't understand file systems like, say grub, does it? 
 Please
> > correct me if I'm wrong, but it's my guess that extlinux behaves more 
like
> > lilo than grub.
> 
> EXTLINUX understands file systems.  If I recall correctly, EXTLINUX's
> boot sector code does not.  I'm not sure how grub acts if you shuffle
> files around, especially if you're wanting to use a new version at the
> old name.  Until you run the installer binary again, I believe the
> boot sector code points to a static block on the partition.

I've never had to reinstall grub unless something happened to the MBR.  It 
just seems to be able to find its various stages.  I'm still not clear 
what extlinux was doing in my case.  I suspect the first stage was loading 
fine, but it couldn't find the next stage due to the shuffling my update 
script performs.  I also suspect the extlinux --update repairs that link. 
Maybe that's all wrong.  Maybe the stage transfer went fine, but the odd 
behavior was simply version incompatibility between the stages.

> > I really wish all boot
> > loaders would include their name and version in their prompt.  When 
trying
> > to unravel chain-loading "boot:" doesn't tell one much.  I don't think
> > there is any chain-loading going on here, but the Fedora Live images 
are
> > originally intended for a bootable CD that then gets transferred to 
the CF
> > card.  I know extlinux is here for sure; isolinux too?  I don't think 
so,
> > but a detailed prompt would help alleviate such concerns.
> 
> You won't see it in the "boot:" prompt but all versions output a
> banner before opening the config file.  You can also use the Ctrl-V
> keystroke (documented in doc/syslinux.txt) to display the version.

I found the Ctrl-V bit after rereading syslinux.txt for any other clues.

> >>  Also, what does your configuration
> >> look like?  Do your kernel, initrd, and squashfs sit in another
> >> directory?
> >
> > /mnt/live/syslinux
> > /mnt/live/syslinux/vmlinuz0
> > /mnt/live/syslinux/memtest
> > /mnt/live/syslinux/isolinux.bin
> > /mnt/live/syslinux/vesamenu.c32
> > /mnt/live/syslinux/extlinux.conf
> > /mnt/live/syslinux/splash.jpg
> > /mnt/live/syslinux/extlinux.sys
> > /mnt/live/syslinux/boot.cat
> > /mnt/live/syslinux/initrd0.img
> > /mnt/live/LiveOS
> > /mnt/live/LiveOS/osmin.img
> > /mnt/live/LiveOS/squashfs.img
> 
> Thank you.  So it looks like your kernel, vmlinuz0, and initrd,
> initrd0.img, sit in the same directory as EXTLINUX.  I'd assume
> osmin.img must be the image you use to perform the renames?

I actually have no idea what osmin.img is for.  I never see it get used. I 
actually do all the renames from what's provided in the initrd0.img.  I 
had to modify how it was built slightly to do so, but I had to modify it 
anyway to trigger the whole update process.  (On a tangent: Fedora's new 
dracut package that replaces mkinitrd makes such a hook so much cleaner.)
 
> > Here /mnt/live is the root file system of the CF card.  The 
initrd0.img
> > has a script to loop mount squashfs.img and find the regular kernel
> > within.
> 
> "find the regular kernel"?  I think it's unusual to use one kernel to
> load another.  It is common, however, to use a kernel/initrd to mount
> another file system (including the contents of the SquashFS).

You are, of course, correct.  My statement was poorly worded -- I meant to 
convey the kernel's regular modules.

> >> Which version loaded?  Which config did it load?  It might be the old
> >> version loading the old config still but I don't understand why it
> >> just reboots.
> >
> > I'm not sure I understand the Qs.  Are you asking what extlinux 
version
> > loads after I've manually made the repair?  I don't know how to ask 
the
> > "boot:" prompt to tell me anything useful here.  For example, if I 
respond
> > with "help" all I get back is "Could not find kernel image: help" and 
then
> > the perpetual two second timeout before it reboots unless I try typing
> > something else before the timeout expires.
> 
> I'm asking what version of EXTLINUX loads with which config file
> during the endless reboot cycle just before your manual repair.

Ctrl-V confirmed that indeed syslinux-3.61 -- the old one -- was being 
loaded here.

> Yep.  If you enter "help" at the "boot:" prompt, all Syslinux variants
> take that as you want to load a kernel/module/etc called help, which
> does not exist (hence "Could not find kernel image:")

Makes sense.  So extlinux and friends are much more like lilo in this 
sense.  For some reason, I was expecting something more like grub where 
there's a whole slew of commands to get yourself out of a pinch ... or 
make a bigger mess.  :-)
 
> >> If it didn't find a configuration file, it would have complained
> >> stating so.  The messages when there is no configuration file found
> >> have improved through the versions.
> >
> > That alone is good to know.  Do you know if 3.61 would be that
> > informative?
> 
> Correction: The missing config file I think was a change for 4.00
> (except for PXELINUX which has had that longer).  The closest
> indication would be a message of "No UI or DEFAULT directive found"
> (3.85+) or an error on not finding a kernel image named linux (the old
> "linux auto" default).

Thanks for digging up the facts.  Seems like a worthy enhancement.

> >> Also, --update always installs a fresh copy of the core binary.  This
> >> "growth" is either replacing the F10 version with F13 like you wanted
> >> initially or effectively just patching it with the needed data.
> >
> > That was my guess, but good to have confirmed.
> 
> To really confirm it, examine at least the file sizes, the contents or
> the output of Ctrl-V (not sure how long this has been around).

For extlinux, at least 3.61.  ;-)

> >> Shuffle the directories to their final destination then run extlinux
> >> from F13.  This probably can be done on the live system (assuming no
> >> library issues) or from your initrd as your live system shouldn't 
have
> >> any of those files open.
> >
> >
> > Well that seems like the sanest thing to try next, but it isn't going 
to
> > be easy.  I've gotten myself a shell from within the initrd and had
> > intended to loop mount the new squashfs image so that I could run 
extlinux
> > from there, but the mount fails from the F10-based initrd with 
"SQUASHFS
> > error: Major/Minor mismatch, trying to mount newer 4.0 filesystem".  I
> > seem to recall that Fedora did update SquashFS in this time frame.  I
> > guess this means I'm going to need to identify and extract the various
> > syslinux bits from an F13 system and provide them to the initrd 
outside of
> > the squashfs.img.
> 
> You should probably only need extlinux, the core installer binary.

I found this to not be true.  I'm a little foggy this morning so I don't 
recall what happened exactly.  IIRC, it got me the same old "boot:" 
prompt/reboot cycle.

> An alternative is to just use the official binaries rather than the
> ones provided by your distro.  The official package is a source/binary
> so you shouldn't have to worry about compiling.

I went this route.  I tried making a copy of everything from F13's 
syslinux package and having those available to the update script for use 
by the initrd image, but those failed with a cryptic "Could not execute" 
message.  I suspect linking problems to glibc or something like that. That 
approach wasn't very pretty anyway, so I just grabbed an official 
syslinux-3.86 tarball and exploded it in place on the CF card so that the 
update script could simply call "extlinux --update" from there.  That 
works just fine, but I cannot get away with just the executable file 
alone.

It might be nice to thin it down a bit, but it only added a few MB to the 
whole update package which is roughly 400MB already.


Thanks for all the help Gene.  I probably would have gotten here 
eventually, but it would have been a lot of work towards an uncertain end. 
 Further, I wouldn't have had much confidence in the stability of that any 
solution.  This way I feel much more comfortable in knowing that these 
update packages should just always bundle an official syslinux version 
that matches that of the new OS and that running the update is necessary 
and the right thing to do.  As this project scales from hundreds of 
devices using this update (and future updates) to thousands, that will 
bring a lot of peace of mind.

--
John Florian