[syslinux] bootloader installation improvements

[ian_bruce at mail.ru]

I have some concerns about the Syslinux boot process, which I'm now
investigating in connection with my work on booting Live-CD images from
USB flashdrives. Some of these are related to what seems to be
inadequate documentation, but I think that there are also aspects of
Syslinux operation which could be fairly easily improved.

quoting from the official documentation:

    Since version 5.00, when a SYSLINUX or EXTLINUX installer is used,
    the relevant boot sector will be modified and two files will be
    added to the "installation directory": the ldlinux.sys boot loader
    file, and an auxiliary ldlinux.c32 file. Note that these two files
    are not necessary so as to execute the installers; they are already
    embedded in the installers themselves.

http://www.syslinux.org/wiki/index.php?title=Library_modules#All_Syslinux_variants_need_an_additional_ldlinux_module

With regard to the "ldlinux.c32" file, this information seems to be
out-of-date. At least, the Debian <syslinux-common> package, which I'm
using, contains a file with that name, whose contents are identical to
the file which the EXTLINUX installer copies into the boot directory. As
far as I'm concerned, this is an improvement.

The semi-magical (as far as you would learn from the documentation)
"ldlinux.sys" file, however, is nowhere to be found, so presumably it
remains embedded inside {SYS,EXT}LINUX. Experiment shows that the two
installers create "ldlinux.sys" files which have the same length, but
differ in the values of two bytes. The difference increases, but remains
small, with the same file length, when one of the installers is applied
to a FAT32 filesystem with a different format. This suggest that the
content of "ldlinux.sys" is mostly static, with a few adjustable
parameters.

(If the only remaining difference between these utilities is that
SYSLINUX wants to work on an unmounted filesystem, while EXTLINUX wants
to work on a mounted filesystem, then why isn't one of them removed?)

It seems to me that having executable files that magically appear out of
nowhere, because they're embedded inside another executable file, is, in
general, a BAD IDEA, for reasons of flexibility and transparency, if
nothing else. It would be better to have this data stored in a separate
file, as "ldlinux.c32" now is. The small number of adjustable parameters
could be handled by having the first sector of the file dedicated to
environment strings of the form "VARIABLE=VALUE", or something like
that, which would be much more transparent than the current method.

Oddly, the best explanation of the Syslinux boot process that I've been
able to find is from the Arch Linux documentation:

    * Boot process overview *

    Stage 1 : Part 1 - Load MBR - At boot, the BIOS loads the 440 byte
    MBR boot code at the start of the disk
    (/usr/lib/syslinux/bios/mbr.bin or
    /usr/lib/syslinux/bios/gptmbr.bin).

    Stage 1 : Part 2 - Search active partition. The Stage 1 MBR boot
    code looks for the partition that is marked as active (boot flag in
    MBR disks). Let us assume this is the /boot partition, for example.

    Stage 2 : Part 1 - Execute volume boot record - The Stage 1 MBR boot
    code executes the Volume Boot Record (VBR) of the /boot partition.
    In the case of Syslinux, the VBR boot code is the starting sector of
    /boot/syslinux/ldlinux.sys which is created by the extlinux
    --install command. Note that ldlinux.sys is not the same as
    ldlinux.c32.

    Stage 2 : Part 2 - Execute /boot/syslinux/ldlinux.sys - The VBR will
    load the rest of /boot/syslinux/ldlinux.sys. The sector location of
    /boot/syslinux/ldlinux.sys should not change, otherwise syslinux
    will not boot.

    Note: In the case of Btrfs, the above method will not work since
    files move around resulting in changing of the sector location of
    ldlinux.sys. Therefore, in Btrfs the entire ldlinux.sys code is
    embedded in the space following the VBR and is not installed at
    /boot/syslinux/ldlinux.sys unlike the case of other filesystems.

    Stage 3 - Load /boot/syslinux/ldlinux.c32 - The
    /boot/syslinux/ldlinux.sys will load the /boot/syslinux/ldlinux.c32
    (core module) that contains the rest of the core part of syslinux
    that could not be fit into ldlinux.sys (due to file-size
    constraints). The ldlinux.c32 file should be present in every
    Syslinux installation and should match the version of ldlinux.sys
    installed in the partition. Otherwise Syslinux will fail to boot.

[what do they mean by "file-size constraints"? what is the hard limit on
the size of "ldlinux.sys"? is it an 8086 16-bit segment? oh, the
horror.]

    Stage 4 - Search and Load configuration file - Once Syslinux is
    fully loaded, it looks for /boot/syslinux/syslinux.cfg (or
    /boot/syslinux/extlinux.conf in some cases) and loads it if it is
    found. If no configuration file is found, you will be dropped to a
    Syslinux boot: prompt. This step and the rest of non-core parts of
    Syslinux (/boot/syslinux/*.c32 modules, excluding lib*.c32 and
    ldlinux.c32) require /boot/syslinux/lib*.c32 (library) modules to be
    present. The lib*.c32 library modules and non-core *.c32 modules
    should match the version of ldlinux.sys installed in the partition.

https://wiki.archlinux.org/index.php/syslinux#Boot_process_overview

For me, the most interesting part of that is this: "in Btrfs the entire
ldlinux.sys code is embedded in the space following the VBR and is not
installed at /boot/syslinux/ldlinux.sys unlike the case of other
filesystems." (Is this true? There doesn't seem to be any hint of it in
the official Syslinux documentation.)

That sounds like a much better arrangement. Data that will be accessed
by non-filesystem means, is better located outside of the filesystem
data area, exactly to avoid problems like files that cannot be relocated
(and must be contiguous?), or else everything will break.

Why is this not an option for FAT32, which must now be by far the most
common filesystem used with Syslinux? (Does anybody actually use floppy
disks anymore?) The standard "mkfs.fat" filesystem creation utility,
from the <dosfstools> project, offers the "-R" option, which will
provide as many "reserved sectors" as are required, before the
filesystem data area. "ldlinux.sys" could easily be placed there,
contiguously, safe from the OS filesystem driver, which does not
understand that it cannot be touched.

http://manpages.org/mkfsfat/8

https://en.wikipedia.org/wiki/Design_of_the_FAT_file_system#Layout

Alternatively, why is there not an option to install "ldlinux.sys" in a
separate partition, which is what GRUB does with its bootloader? A USB
flashdrive could easily be formatted with two partitions, a small one
for "ldlinux.sys", and a much bigger one for the FAT32 or other
filesystem.

To summarize, it seems to me that these changes would make the Syslinux
installation process more reliable, transparent, and understandable (I
didn't, until I read the Arch Linux wiki!):

- transfer of "ldlinux.sys" into a separate file, with a text-based
  parameter block at some fixed location. This could then be edited, not
  only by {SYS,EXT}LINUX, but if necessary, with a binary sector editor,
  or even "sed".

- reconciliation of SYSLINUX and EXTLINUX into a single utility, which
  operates only on the non-data parts of the filesystem. "ldlinux.c32"
  can be copied into the filesystem with the other Syslinux modules, by
  the normal operating system methods. So can "ldlinux.sys", with the
  requirement that {SYS,EXT}LINUX be run AFTER that, to inform the
  Volume Boot Record where to find it. (This is how LILO works.)

- an option to install "ldlinux.sys" in the reserved sectors of a FAT32
  filesystem, or the equivalent for other filesystems, as is apparently
  already done for BTRFS.

- an option to install "ldlinux.sys" into a separate partition, as GRUB
  does with its first-stage bootloader.

Thanks for your consideration of these suggestions for the improvement
of Syslinux. I volunteer to write the parameter block parser, if that
would be helpful.


-- Ian Bruce