[syslinux] Improving TAILS, WAS: Module Versioning

[Pete Batard]

Hi Shao - looks like forgot to reply to this earlier post of yours, so 
here goes.

On 2016.03.08 04:59, Shao Miller via Syslinux wrote:
> 1. Customized Syslinux could be customized such that it refuses to boot
>     from USB.  No amount of version-matching nor downloading from the
>     same source can help with that scenario.  A different Syslinux would
>     be needed, by definition (of the customizer)

I assume that what you mean by the above is "the automated process you 
described could have a problem if Syslinux has been customized such that..."

Well, if that happens, that won't be due to ldlinux.sys, since that's 
what we download, and (since the people publishing an ISO that contains 
GPLv2 software are legally bound to publish their modified source) even 
if it's the ISO's ldlinux.c32, I don't think it's an insurmountable 
obstacle. If needed, I can probably create an ldlinux.sys that 
integrates a modified module, so that it bypasses the existing one from 
the ISO, and make that available for download. And of course, as soon as 
it's on my server, existing versions of Rufus will just work with the 
ISO. I also already maintain a version of ldlinux.c32 on my server, 
besides ldlinux.sys (as Rufus does provide the ability to install a 
"bare" Syslinux, in which case we download that one module), so I could 
very easily modify the process to download/replace all the ldlinuxes 
from the server.

Also, as I mentioned elsewhere, we're never actually evolving within a 
boundless context, but can estimate the likelihood of someone creating 
an ldlinux.c32 that prevents USB boot (because that's really the only 
piece I can remotely see with the capability to do that, and I'm not 
entirely sure that, if I were to look at the code that goes into that 
.c32, I wouldn't have a very hard time trying to devise a way to make 
HDD support from ldlinux.sys break), I see it so unlikely a scenario 
that I can choose to safely ignore it until proved otherwise (with a 
proof that of course comes from a real-life scenario, rather than one 
that is specifically designed to make Rufus fail, as it's not that 
difficult to do, if your goal has little to do with providing regular 
bootable content to users).

I should mention however that, should this arise, what is MUCH MORE 
likely to happen, again because we're not evolving in the absolute but 
working within the boundaries set by other processes, is that I'm going 
to contact the maintainers of that ISO, just as I did with Debian when I 
found that in a very specific cases their GRUB had an issue with USB 
conversion, and ask them to modify their content to make it compatible 
with Rufus in their next ISO release... while at the same time trying to 
determine, from the reports I receive: 1/ If I should also apply a 
fix/workaround in Rufus, and 2/ How urgently that fix/workaround needs 
to find its way to Rufus users.

In the case of Debian's GRUB incompatibility (which was actually due to 
Debian having forgotten to embed a GRUB module that provides GPT boot 
support - so much for human experts being able to devise flawless 
bootable solutions, as this was an issue that affected more than Rufus 
usage), while I could relatively easily have added a download for a 
replacement binary with the GPT module on my server, and issued a 
version of Rufus that handled such a download (and again, since we're 
not evolving in isolation, one should consider the ability that Rufus 
users have to seamlessly update to a new version as another part of the 
complete conversion process), I decided instead to sit on it and just 
wait for Debian's update, as I deemed it very unlikely that many people 
would be affected (which was confirmed) and also because there was an 
easy workaround along with what (I hope) is a relatively easy to find 
path of where affected users could obtain guidance.

In other words, my view on this is that:

Corner case scenarios will be dealt on a case by case basis, as they 
arise, because the process I have should be more than flexible enough to 
ensure that we have a few options at our disposal to address these 
unlikely corner cases.

> 2. Slightly less bad is where an ISO or CD/DVD is made with a key piece
>     of non-ISOLINUX Syslinux missing and completely unavailable.

Besides the GRUB+GPT+Debian example above, and to bring the topic back 
to Syslinux, I also dealt with something a bit similar in the past, when 
I found that older versions of menu.c32 and vesamenu.c32 were not 
compatible with Syslinux 4.x. That's actually when I added a file 
download process in Rufus, so that I could replace these 2 specific 
modules on the server. It was relatively trivial to handle then 
(especially as I didn't bother trying to harden that process for some 
corner cases I can envision... which I have yet to see happen for the 
past 4 years).

So here again, in the scenario we are discussing, I'm fairly confident 
that I can craft an ldlinux.sys or .c32 modules if/as needed and set 
them on the server for download, to provide whatever key piece is 
missing, should such an issue ever (which I consider exceedingly 
unlikely in the first place).

> 3. If you remove all binaries and leave only configuration-files, then
>     install your favourite version of Syslinux (which you keep packaged
>     with Rufus) to the USB, then fit the config-files into place, what
>     are the disadvantages?

1. I'd need to embed every possible .c32 module in Rufus, which will 
make it explode in size.

2. I'd also need to cater for custom/modified modules, which I see A LOT 
more likely than any custom/modified isolinux.bin + ldlinux.c32.

>      1. You'd mentioned "the application size would explode, and as more
>         version of Syslinux get releases", but perhaps there's a
>         misunderstanding...  If Rufus chooses one and only one golden
>         Syslinux, then Rufus doesn't need to keep more than one version
>         of modules and ldlinux.sys.  That is, forget about whichever
>         Syslinux (ISOLINUX) was on the original disc/ISO

One thing I did not mention is that I currently embed 2 versions of 
Syslinux: 4.07 and 6.03. Before that I used to embed 4.06 and 5.x, and 
when 6.x started to see the light of day, I actually considered whether 
I would embed all of 4.x, 5.x and 6.x. Ultimately, besides the "Well, I 
can reduce the app size" aspect, the reason I decided to get rid of 5.x 
is that few people were using it and the Syslinux project seemed to be 
very eager to increment the major version before people got a chance to 
get comfortable with 5.x.

However, I don't expect these considerations to necessarily hold true 
for the next major version bump, so, even if there is a download 
mechanism in Rufus, I can't be sure that there won't be a good reason I 
wouldn't want to embed 3 versions in the future... even more so as, for 
the user friendliness aspect, I am exceedingly keen on reducing the need 
for downloads and internet connectivity, and the more you embed, the 
less you need to download.

Also, even with a single version, the current sum of all .c32 modules is 
larger than the current size of the Rufus application. So, even if I 
were to embed just one version of Syslinux (and even as I do compress 
the content I embed) I'm not too thrilled about making the bandwidth of 
my download server close to double...

> 4. Assuming the last point leaves us wanting, if Rufus took the hash of
>     key files, it could consult a database:
>      1. If the Internet is available, it could check your web-site
>          1. If your web-site recognizes the hash, it could send
>             appropriate (exact or compatible) files
>          2. If your web-site doesn't recognize the hash, it could
>             request the user to report the miss
>      2. If the Internet is unavailable, it could check a
> Rufus-integrated DB
>          1. If the hash is recognized and an appropriate (exact or
>             compatible) fileset is shipped with Rufus, those files could
>             be used
>          2. If the hash is recognized but an appropriate fileset is
>             online, Rufus could report, "Sorry, but we need the Internet"
>          3. If the hash isn't recognized, Rufus could request the user
>             to report the miss

Great, now I have to force connectivity on my users (the nice thing 
about Syslinux 4.x is, unless you use a very old menu.c32/vesamenu.c32, 
Rufus should not need to connect to the internet to convert your ISO... 
which currently also holds true for the latest GRUB 2, as in the vast 
majority of cases the embedded will just work), and because some people 
like tails do recompile their own Syslinux stuff (which at least a 
custom version string), and tend to issue about 1 release every month, 
I'm going to have to embed sqlite or something as well as an ever 
growing DB which I also see likely to end up dwarfing everything else. 
Oh, and I also have to allocate time maintaining this DB...

If it comes to that, I think I might as well contact distro maintainers 
using Syslinux, and invite them to try to switch to GRUB 2, as it'd 
probably be a lot less trouble... ;)

This being said, I do appreciate that all you are trying to do here is 
look for possible alternate solutions/workarounds, which I am grateful 
for. However, I too did consider these (relatively) obvious 
possibilities and came to the conclusion that the best solution to all 
the problems exposed before is for isolinux.bin/ldlinux.bin/pxelinux.bin 
to become a single polymorphic beast, so that everything that is needed 
for USB-HDD boot is 99.99% likely to be available on the ISO, and so 
that the set of rules used by distro maintainers to create an ISO won't 
have to change.

Regards,

/Pete