[syslinux] Stumped by extlinux After OS Upgrade

[Gene Cumm]

On Mon, Nov 1, 2010 at 14:16,  <John.Florian at dart.biz> wrote:
>> From: Gene Cumm <gene.cumm at gmail.com>
>
>> My comments are inline but I'd suggest you follow my advice at the end
>> of my email as a first thing to try.
>
> First off, thanks for the reply!
>
> I agree, so that's where I'm heading first and have already ran into
> difficulty -- detailed at the end.  I will also respond inline to you
> various questions just to be thorough.

OK.

>> > I've developed a specialized spin of Fedora that runs off
> CompactFlash.
>> > The spin is actually produced by the livecd-tools package and is then
>> > transferred to the CF card -- which has been formatted with ext3 --
> using
>> > livecd-tools' livecd-iso-to-disk.  I have hundreds of embedded systems
>> > running this spin.  Every now and then we need to upgrade these
> systems
>> > and needless to say swapping CF cards is labor expensive.  Thus I
>> > developed an in-place upgrade which essentially downloads a new
> squashfs
>> > image and syslinux directory, both of which are obtained from a newer
>> > spin.  These downloads do not overwrite those of the running system;
> they
>> > get a name suffix of ".new".  Then a special initrd is booted into
> which
>> > looks for the new stuff and puts them in place after renaming the
>> > originals with an ".old" suffix.  Another reboot occurs and the new
> spin
>> > is then running.  Currently I'm trying to effect an upgrade that will
> take
>> > these systems from a Fedora 10 base to Fedora 13.
>> >
>> > Now the fun part and my problem.  Before shipping CF cards out with
> F10 on
>> > them, I went through rigorous functional testing and one step was to
>> > verify that future upgrades would be successful as best that I could.
>> > Since F10 was the latest Fedora at the time it was only a pseudo-test
> that
>> > tried to upgrade the F10 spin to an alternate, distinguishable F10
> spin. I
>> > made that work, but found it necessary to run "extlinux --update
>> > /sysroot/syslinux" after swapping in the replacement squashfs image
> and
>> > syslinux directory.  I don't recall exactly why it was necessary, but
>> > suffice it to say it solved the problem and I could reboot the system
> and
>> > be running the alternate F10 image.
>>
>> Which version(s) of EXTLINUX?
>
> Fedora 13 is using syslinux-3.84 whereas Fedora 10 has syslinux-3.61.

Still a little back but at least recent.

>>  If you're attempting to use an old
>> version of EXTLINUX to patch the extlinux.sys of a new version, it
>> probably won't work the way you want.
> I was afraid I'd hear that.

You'll probably get the "new" file replaced with a copy from the old version.

>> I think "extlinux --update"
>> also refreshes the VBR (Volume boot record), which contains a pointer
>> to the extlinux.sys file.
>>
>> I normally take the approach that if the core
>> binary(extlinux.sys/ldlinux.sys) is moved (including the directory it
>> sits in), renamed, or modified, it should be redone with --update for
>> maximum reliability.  I can't say for certain if it's needed.
>
>
> I wish it were not necessary, but given the behavior I suspect it must.
> extlinux doesn't understand file systems like, say grub, does it?  Please
> correct me if I'm wrong, but it's my guess that extlinux behaves more like
> lilo than grub.

EXTLINUX understands file systems.  If I recall correctly, EXTLINUX's
boot sector code does not.  I'm not sure how grub acts if you shuffle
files around, especially if you're wanting to use a new version at the
old name.  Until you run the installer binary again, I believe the
boot sector code points to a static block on the partition.

lilo, unlike grub and syslinux, last I checked, required a linux tool
to reconfigure/reinstall.  Reconfiguring grub and syslinux only
requires a text editor.  Reinstallation was a different matter.

>>  After
>> your initrd shuffles the directories around, it might be wise to
>> re-run your extlinux update (from the F13 installer, of course).
>>
>> > Now I'm doing it for real but instead of that alternate F10 spin, the
>> > upgrade would bring F13 into place.  Everything works smoothly -- the
>> > extlinux update runs without error and has the expected message
> indicating
>> > "/sysroot/syslinux is device /dev/sda1" -- right up until that final
>> > reboot which should take us into F13.  What happens?  Well, I get a
>> > "boot:" prompt for a second or two and then it disappears and then
> repeats
>> > endlessly.
>>
>> Strange to say the least.  Is this "second or two" possibly your
>> TIMEOUT in your configuration?
>
>
> Well, the timeout in the old (F10) extlinux.conf is 10 and the new (F13)
> is 30.  So if those are in 1/10s, I'd guess no.  I really wish all boot
> loaders would include their name and version in their prompt.  When trying
> to unravel chain-loading "boot:" doesn't tell one much.  I don't think
> there is any chain-loading going on here, but the Fedora Live images are
> originally intended for a bootable CD that then gets transferred to the CF
> card.  I know extlinux is here for sure; isolinux too?  I don't think so,
> but a detailed prompt would help alleviate such concerns.

You won't see it in the "boot:" prompt but all versions output a
banner before opening the config file.  You can also use the Ctrl-V
keystroke (documented in doc/syslinux.txt) to display the version.

>>  Also, what does your configuration
>> look like?  Do your kernel, initrd, and squashfs sit in another
>> directory?
>
> /mnt/live/syslinux
> /mnt/live/syslinux/vmlinuz0
> /mnt/live/syslinux/memtest
> /mnt/live/syslinux/isolinux.bin
> /mnt/live/syslinux/vesamenu.c32
> /mnt/live/syslinux/extlinux.conf
> /mnt/live/syslinux/splash.jpg
> /mnt/live/syslinux/extlinux.sys
> /mnt/live/syslinux/boot.cat
> /mnt/live/syslinux/initrd0.img
> /mnt/live/LiveOS
> /mnt/live/LiveOS/osmin.img
> /mnt/live/LiveOS/squashfs.img

Thank you.  So it looks like your kernel, vmlinuz0, and initrd,
initrd0.img, sit in the same directory as EXTLINUX.  I'd assume
osmin.img must be the image you use to perform the renames?

> Here /mnt/live is the root file system of the CF card.  The initrd0.img
> has a script to loop mount squashfs.img and find the regular kernel
> within.

"find the regular kernel"?  I think it's unusual to use one kernel to
load another.  It is common, however, to use a kernel/initrd to mount
another file system (including the contents of the SquashFS).

>> If you're running the F10 extlinux from your squashfs, it's probably
>> not accomplishing much.
>>
>> > I've tried a variety of things and have learned much, but not enough
> to
>> > understand the problem and have become rather perplexed.  Inspecting
> the
>> > CF card on my desktop system, I find what I'd expect.  I can make
> notes of
>> > kernel parameters as found in extlinux.conf, return the card to the
>> > device, power up and provide those same parameters at the "boot:"
> prompt
>> > and the device will boot into F13 just fine.  Once there I can
> manually
>> > run "extlinux --update /mnt/live/syslinux" and the system will reboot
> fine
>> > thereafter w/o intervention.  (The path changes, but is effectively
> the
>> > same directory, it's just mounted differently between the initrd and
> the
>> > regular running system.)  I have also manually run the "extlinux
> --update"
>> > from my desktop system (also F13) and found that also will "repair"
> the CF
>> > card to good working order.
>>
>> Which version loaded?  Which config did it load?  It might be the old
>> version loading the old config still but I don't understand why it
>> just reboots.
>
> I'm not sure I understand the Qs.  Are you asking what extlinux version
> loads after I've manually made the repair?  I don't know how to ask the
> "boot:" prompt to tell me anything useful here.  For example, if I respond
> with "help" all I get back is "Could not find kernel image: help" and then
> the perpetual two second timeout before it reboots unless I try typing
> something else before the timeout expires.

I'm asking what version of EXTLINUX loads with which config file
during the endless reboot cycle just before your manual repair.

Yep.  If you enter "help" at the "boot:" prompt, all Syslinux variants
take that as you want to load a kernel/module/etc called help, which
does not exist (hence "Could not find kernel image:")

>> > My hunch is that either extlinux cannot find it's conf file or maybe
> the
>> > path difference between initrd and regular somehow does matter.
>  Neither
>> > seems likely though since my F10-F10 test upgrade worked.  That makes
> me
>> > wonder about using "extlinux --update" from F10 on a syslinux
> directory
>> > that was already populated by a F13 system and some compatibility
> issue.
>> > I've noticed that my manually run "extlinux --update" makes
> extlinux.sys
>> > grow a bit too as if the run from the initrd may have left something
> out,
>> > but I really know nothing about that file (yet).  Lastly, there's been
> a
>> > change in the "root=" kernel parameter between F10 and F13 that maybe
> is
>> > at play here.  In F10, this read as "root=UUID=blablabla" but now
> reads as
>> > "root=live:UUID=blablabla".  When I provide the kernel parameters
> manually
>> > at the "boot:" prompt I must use the F13 form for a fully successful
> boot,
>> > but even if I use the F10 format it get's much further than on its own
> --
>> > it fails around the pivot_root point.  Maybe the "live" bit is
> confusing
>> > the syslinux package in the initrd of F10?
>>
>> If it didn't find a configuration file, it would have complained
>> stating so.  The messages when there is no configuration file found
>> have improved through the versions.
>
> That alone is good to know.  Do you know if 3.61 would be that
> informative?

Correction: The missing config file I think was a change for 4.00
(except for PXELINUX which has had that longer).  The closest
indication would be a message of "No UI or DEFAULT directive found"
(3.85+) or an error on not finding a kernel image named linux (the old
"linux auto" default).

>> Also, --update always installs a fresh copy of the core binary.  This
>> "growth" is either replacing the F10 version with F13 like you wanted
>> initially or effectively just patching it with the needed data.
>
> That was my guess, but good to have confirmed.

To really confirm it, examine at least the file sizes, the contents or
the output of Ctrl-V (not sure how long this has been around).

>> Shuffle the directories to their final destination then run extlinux
>> from F13.  This probably can be done on the live system (assuming no
>> library issues) or from your initrd as your live system shouldn't have
>> any of those files open.
>
>
> Well that seems like the sanest thing to try next, but it isn't going to
> be easy.  I've gotten myself a shell from within the initrd and had
> intended to loop mount the new squashfs image so that I could run extlinux
> from there, but the mount fails from the F10-based initrd with "SQUASHFS
> error: Major/Minor mismatch, trying to mount newer 4.0 filesystem".  I
> seem to recall that Fedora did update SquashFS in this time frame.  I
> guess this means I'm going to need to identify and extract the various
> syslinux bits from an F13 system and provide them to the initrd outside of
> the squashfs.img.

You should probably only need extlinux, the core installer binary.

An alternative is to just use the official binaries rather than the
ones provided by your distro.  The official package is a source/binary
so you shouldn't have to worry about compiling.

-- 
-Gene