[syslinux] bootloader installation improvements

[Gene Cumm]

On Fri, Aug 4, 2017 at 12:06 PM, ian_bruce--- via Syslinux
<syslinux at zytor.com> wrote:
> I have some concerns about the Syslinux boot process, which I'm now
> investigating in connection with my work on booting Live-CD images from
> USB flashdrives. Some of these are related to what seems to be
> inadequate documentation, but I think that there are also aspects of
> Syslinux operation which could be fairly easily improved.
>
> quoting from the official documentation:
>
>     Since version 5.00, when a SYSLINUX or EXTLINUX installer is used,
>     the relevant boot sector will be modified and two files will be
>     added to the "installation directory": the ldlinux.sys boot loader
>     file, and an auxiliary ldlinux.c32 file. Note that these two files
>     are not necessary so as to execute the installers; they are already
>     embedded in the installers themselves.
>
> http://www.syslinux.org/wiki/index.php?title=Library_modules#All_Syslinux_variants_need_an_additional_ldlinux_module

I thought the syslinux/extlinux nomenclature was adopted for the
installer but SYSLINUX and EXTLINUX for the bootloader itself.
</pedant>

> With regard to the "ldlinux.c32" file, this information seems to be
> out-of-date. At least, the Debian <syslinux-common> package, which I'm
> using, contains a file with that name, whose contents are identical to
> the file which the EXTLINUX installer copies into the boot directory. As
> far as I'm concerned, this is an improvement.

It is expected that a package like this will include this file since
you won't be able to build an ISO with ISOLINUX without it.  It is
both embedded in the installers and available as a plain file.  The
syslinux and extlinux installers contain ldlinux.c32 to ensure that a
system is fully bootable with a matching ldlinux.c32 upon successful
completion of the installer execution.

> The semi-magical (as far as you would learn from the documentation)
> "ldlinux.sys" file, however, is nowhere to be found, so presumably it
> remains embedded inside {SYS,EXT}LINUX. Experiment shows that the two
> installers create "ldlinux.sys" files which have the same length, but
> differ in the values of two bytes. The difference increases, but remains
> small, with the same file length, when one of the installers is applied
> to a FAT32 filesystem with a different format. This suggest that the
> content of "ldlinux.sys" is mostly static, with a few adjustable
> parameters.

I fully expect ldlinux.sys to not be in a package since it serves no
value.  You'll find it in the prebuilt binary/source archives mostly
to allow an installer-only rebuild.

I fully expect at least two bytes and up to something in the direction
of a full kilobyte to change.  It's a block map and rudimentary
checksum.  It's in the source.

> (If the only remaining difference between these utilities is that
> SYSLINUX wants to  work on an unmounted filesystem, while EXTLINUX wants
> to work on a mounted filesystem, then why isn't one of them removed?)

Yes, that is the precise purpose.  syslinux ONLY works on unmounted
FAT* file systems while extlinux works on any supported mounted file
system.

> It seems to me that having executable files that magically appear out of
> nowhere, because they're embedded inside another executable file, is, in
> general, a BAD IDEA, for reasons of flexibility and transparency, if
> nothing else. It would be better to have this data stored in a separate
> file, as "ldlinux.c32" now is. The small number of adjustable parameters
> could be handled by having the first sector of the file dedicated to
> environment strings of the form "VARIABLE=VALUE", or something like
> that, which would be much more transparent than the current method.

Think portability.  With just a single binary, it applies everything
that's necessary without the need for external include files akin to a
library.  At this time, syslinux and exlinux include ldlinux.sys and
ldlinux.c32.  Without having both, SYSLINUX would never do anything
useful aside from complain at a user.

See also above.  Perhaps reading the source is in order

> Oddly, the best explanation of the Syslinux boot process that I've been
> able to find is from the Arch Linux documentation:
>
>     * Boot process overview *
>
>     Stage 1 : Part 1 - Load MBR - At boot, the BIOS loads the 440 byte
>     MBR boot code at the start of the disk
>     (/usr/lib/syslinux/bios/mbr.bin or
>     /usr/lib/syslinux/bios/gptmbr.bin).
>
>     Stage 1 : Part 2 - Search active partition. The Stage 1 MBR boot
>     code looks for the partition that is marked as active (boot flag in
>     MBR disks). Let us assume this is the /boot partition, for example.
>
>     Stage 2 : Part 1 - Execute volume boot record - The Stage 1 MBR boot
>     code executes the Volume Boot Record (VBR) of the /boot partition.
>     In the case of Syslinux, the VBR boot code is the starting sector of
>     /boot/syslinux/ldlinux.sys which is created by the extlinux
>     --install command. Note that ldlinux.sys is not the same as
>     ldlinux.c32.
>
>     Stage 2 : Part 2 - Execute /boot/syslinux/ldlinux.sys - The VBR will
>     load the rest of /boot/syslinux/ldlinux.sys. The sector location of
>     /boot/syslinux/ldlinux.sys should not change, otherwise syslinux
>     will not boot.
>
>     Note: In the case of Btrfs, the above method will not work since
>     files move around resulting in changing of the sector location of
>     ldlinux.sys. Therefore, in Btrfs the entire ldlinux.sys code is
>     embedded in the space following the VBR and is not installed at
>     /boot/syslinux/ldlinux.sys unlike the case of other filesystems.
>
>     Stage 3 - Load /boot/syslinux/ldlinux.c32 - The
>     /boot/syslinux/ldlinux.sys will load the /boot/syslinux/ldlinux.c32
>     (core module) that contains the rest of the core part of syslinux
>     that could not be fit into ldlinux.sys (due to file-size
>     constraints). The ldlinux.c32 file should be present in every
>     Syslinux installation and should match the version of ldlinux.sys
>     installed in the partition. Otherwise Syslinux will fail to boot.
>
> [what do they mean by "file-size constraints"? what is the hard limit on
> the size of "ldlinux.sys"? is it an 8086 16-bit segment? oh, the
> horror.]

Several years back when btrfs support was first implemented, there was
only 64k available to install ldlinux.sys, since it needs to be
ensured that the blocks don't move even through the late-stage changes
to apply the block map.

>     Stage 4 - Search and Load configuration file - Once Syslinux is
>     fully loaded, it looks for /boot/syslinux/syslinux.cfg (or
>     /boot/syslinux/extlinux.conf in some cases) and loads it if it is
>     found. If no configuration file is found, you will be dropped to a
>     Syslinux boot: prompt. This step and the rest of non-core parts of
>     Syslinux (/boot/syslinux/*.c32 modules, excluding lib*.c32 and
>     ldlinux.c32) require /boot/syslinux/lib*.c32 (library) modules to be
>     present. The lib*.c32 library modules and non-core *.c32 modules
>     should match the version of ldlinux.sys installed in the partition.
>
> https://wiki.archlinux.org/index.php/syslinux#Boot_process_overview
>
> For me, the most interesting part of that is this: "in Btrfs the entire
> ldlinux.sys code is embedded in the space following the VBR and is not
> installed at /boot/syslinux/ldlinux.sys unlike the case of other
> filesystems." (Is this true? There doesn't seem to be any hint of it in
> the official Syslinux documentation.)

100% true.  It's in the source.

> That sounds like a much better arrangement. Data that will be accessed
> by non-filesystem means, is better located outside of the filesystem
> data area, exactly to avoid problems like files that cannot be relocated
> (and must be contiguous?), or else everything will break.

Except btrfs was designed for this.  Other file systems don't have
better provisions for this.

ldlinux.sys can be discontiguous but in most installs, one contiguous
block.  See also above.

> Why is this not an option for FAT32, which must now be by far the most
> common filesystem used with Syslinux? (Does anybody actually use floppy
> disks anymore?) The standard "mkfs.fat" filesystem creation utility,
> from the <dosfstools> project, offers the "-R" option, which will
> provide as many "reserved sectors" as are required, before the
> filesystem data area. "ldlinux.sys" could easily be placed there,
> contiguously, safe from the OS filesystem driver, which does not
> understand that it cannot be touched.

See above.  Also, it can not be expected that we install to a pristine
file system prepared in the perfect manner with a optional parameter.
Eating 64k on a 160k floppy is unfeasible.

> http://manpages.org/mkfsfat/8
>
> https://en.wikipedia.org/wiki/Design_of_the_FAT_file_system#Layout
>
> Alternatively, why is there not an option to install "ldlinux.sys" in a
> separate partition, which is what GRUB does with its bootloader? A USB
> flashdrive could easily be formatted with two partitions, a small one
> for "ldlinux.sys", and a much bigger one for the FAT32 or other
> filesystem.

multifs patch is pending and scheduled for 6.10.

> To summarize, it seems to me that these changes would make the Syslinux
> installation process more reliable, transparent, and understandable (I
> didn't, until I read the Arch Linux wiki!):
>
> - transfer of "ldlinux.sys" into a separate file, with a text-based
>   parameter block at some fixed location. This could then be edited, not
>   only by {SYS,EXT}LINUX, but if necessary, with a binary sector editor,
>   or even "sed".

I'm not keen on this idea.

> - reconciliation of SYSLINUX and EXTLINUX into a single utility, which
>   operates only on the non-data parts of the filesystem. "ldlinux.c32"
>   can be copied into the filesystem with the other Syslinux modules, by
>   the normal operating system methods. So can "ldlinux.sys", with the
>   requirement that {SYS,EXT}LINUX be run AFTER that, to inform the
>   Volume Boot Record where to find it. (This is how LILO works.)

Two utilities for two purposes.  EXTLINUX (loader) was already killed
and its functionality merged to SYSLINUX.  Further changes would
potentially break existing scripting.

> - an option to install "ldlinux.sys" in the reserved sectors of a FAT32
>   filesystem, or the equivalent for other filesystems, as is apparently
>   already done for BTRFS.

This MIGHT be worthwhile, if the reserved block is large enough.  Then
again, Windows might use it for things and using it might break
chainloading to an alternate boot from a stored VBR.

> - an option to install "ldlinux.sys" into a separate partition, as GRUB
>   does with its first-stage bootloader.
>
> Thanks for your consideration of these suggestions for the improvement
> of Syslinux. I volunteer to write the parameter block parser, if that
> would be helpful.
>
>
> -- Ian Bruce

Hope this helps clarify things.

-- 
-Gene