[syslinux] fixing debian's hd-media image

[Ady Ady]

> On Thu, Nov 15, 2018 at 1:10 PM Ady Ady via Syslinux <syslinux at zytor.com> wrote:
> >
> > Here are some comments and questions for you.
> >
> 
> Thanks.
> 
> I've have my scripts to a better state, including installing/using
> qemu to uefi and legacy boot the usb stick for testing.
> 
> It mostly works for me, but the "wget a tarball of binaries" step
> isn't going to sit well with the debian folk, so I'm not done yet.  I
> may be close enough that they will know how to do it properly.
 
 
As I mentioned before, the current "Testing" repo in Debian has the 
most updated binaries of any Linux distro, and even newer (and better) 
than official upstream 6.04-pre1.

The current "syslinux.efi" files for Debian (both, for efi32 and for 
efi64) can be downloaded from the "syslinux-efi" package.

For the "Testing" repo, the mirrors (and links) can be seen at:
 https://packages.debian.org/testing/all/syslinux-efi/download 

The Syslinux-related files in Debian's packages are scattered through 
several binary packages. Unfortunately, the specific way in which these 
files are arranged makes not much sense. So, in order to get 
"ldlinux.e64" (and/or "ldlinux.e32", and/or "ldlinux.c32), you need to 
also get another ("big") binary package from Debian, "syslinux-common":

 https://packages.debian.org/testing/all/syslinux-common/download 


> 
> 
> > > I want to eufi boot this image:
> > > http://cdn-fastly.deb.debian.org/debian/dists/testing/main/installer-amd64/current/images/hd-media/boot.img.gz
> >
> >
> > I am going to assume that by "boot this image" you don't mean something
> > like memdisk or equivalent, but rather that you want to transfer that
> > image using "dd" to your (USB) device and boot the resulting device in
> > UEFI mode. Is this assumption correct?
> 
> correct.
> 
> 
> >
> >
> > >
> > > there is a bug against it:
> > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=913523
> >
> >
> > Although this is the Syslinux Mailing List and not Debian's in
> > particular...
> 
> syslinux list because I am trying to UEFI boot syslinux.
 
 
OK. So, at least for now, we are not talking about any kind of fix nor 
bug report about Syslinux itself. Now things are getting clearer.


> 
> > I would say that the above bug report is slightly
> > confusing. AFAIK, the files under "hd-media" are aimed at a specific
> > scenario (i.e. starting the _installation_ of the OS from a filesystem
> > volume that is already part of the HDD).
> 
> correct, kinda.  it boots a linux kernel, and runs the installer.
> Unless it is a UEFI only box, and then it doesn't boot, which is a
> problem.
 
 
Since the "hd-image" boot.img file was never prepared for UEFI booting 
(whereas the kernel and intird files within it might, perhaps, already 
support such case), then we are not talking about a traditional "bug" 
but rather a feature request, for Debian (D-I, not for Syslinux). Now 
we are rising the level of clarity. Good :).


> 
> > I could be wrong but, where
> > does it say that the boot.img(.gz) mentioned in that report is aimed at
> > (or that it supports) booting Debian (itself) from USB?
> 
> It's implied.  or desired, so we could call it a feature request.
 
 
Feature request for Debian (D-I). Agreed.


> 
> >
> > Moreover, where is the official info claiming that the boot.img(.gz)
> > file can support booting in UEFI mode? (FWIW, ATM it doesn't, unless
> > you add "some things").
> 
> correct - I was confused that a UEFI box booted it using compatibility mode.
 
 
CSM is like BIOS, so, sure, it should be bootable in such case (unless 
the "BIOS" is buggy, in which case YMMV).


> 
> >
> > Additionally, the boot.img(.gz) file is not supposed to have a
> > partition table. Not having a partition table is not a bug for that
> > image (the image is a (potentially bootable) filesystem volume). I
> > could be wrong but, AFAIU, the image is not aimed at being written to
> > the HDD as a complete, partitioned HDD. You could write a device with
> > it, but it would then be a (super)floppy-like device. Whether such
> > device effectively serves its purpose of launching the installer (for
> > Debian), that's another matter.
> 
> all correct.
 
 
The reason I mentioned that using a fs directly, for UEFI booting, is 
supposed to be supported, is because I have read the comments in the 
bug report (for Debian) you mentioned before. Whoever takes care (or 
plainly reads) that bug report might (wrongly) think that a GPT should 
be added to boot.img; it should _not_ be. Let me rephrase it: it shall 
not be added.

The boot.img needs a UEFI-compatible bootloader within it, using the 
"default" file names (e.g. "/EFI/BOOT/BOOTX64.EFI") for the feature 
request to be resolved.


> 
> >
> > Now, perhaps an important detail to be aware of: the UEFI specs indeed
> > support booting a filesystem volume in UEFI mode. Just think of it as a
> > (super)floppy (i.e. no partition table, no MBR, _no GPT_).
> 
> I was wondering about that.   I'll stop trying to figure out how to do
> that, so thank you,
> 
> >
> >
> >
> > >
> > > I'd like to fix the resulting usb stick for my use, and also fix it in
> > > a way that can be used to fix the upstream bug.
> >
> >
> > You are using the term "fix" twice here, and I have to admit that, to
> > me, it is not at all clear what exactly you mean.
> 
> I would like a quick workaround so I can boot the box I am in
> possession of (which I have now achieved)
> and also something that can be contributed to the debian project.  (I'm close)
> 
> >
> > What exactly is wrong in the resulting USB stick?
> 
> It is built by me floundering around, call it prof of concept.
> I'm a little bothered that syslinux.efi isn't in the debian package,
> and should I use the ldlinux.e64 from the tarbal or package?
 
 
See my reply above, in this same email, about both files and where to 
get them.

Also, as mentioned more than once in the wiki, Syslinux files should 
come from the exact same package / build / version; mixing files from 
different packages / builds / versions should be avoided.

In theory, you could use one set of Syslinux files (one version) for 
BIOS and another (version) for UEFI. I wouldn't recommend it.


> 
> >
> > What is "upstream" to you in this context? Debian? D-I? The Syslinux
> > Project? Other?
> 
> Debian.
> 
> I think add some lines around here:
> 
> https://salsa.debian.org/installer-team/debian-installer/blob/master/build/config/x86.cfg#L113-115
> 
> mcopy -i$(TEMP_BOOT) /usr/lib/syslinux/modules/bios/vesamenu.c32
> ::vesamenu.c32; \
> mcopy -i$(TEMP_BOOT) /usr/lib/syslinux/modules/bios/libcom32.c32
> ::libcom32.c32; \
> mcopy -i$(TEMP_BOOT) /usr/lib/syslinux/modules/bios/libutil.c32
> ::libutil.c32 ; \
 
 
I'm confused. Why exactly those (BIOS) files are supposed to be related 
to your feature request? Could you please explain (what exactly is 
triggering this thought)?


> 
> >
> >
> > >
> > > I'm trying to follow the guids
> > > https://www.syslinux.org/wiki/index.php?title=Install#UEFI
> > > and a ml post - here is what I have so far:
> > >
> > > https://salsa.debian.org/carlfk-guest/ansible/blob/uefiusb/usbinst/uefi/fix_hdmedia.sh
> > > (likely quite a bit of unneeded copies)
> > >
> > > it boots, (good)
> > > Initial menu has no LABEL enries!
> >
> >
> > I have to say that your cfg file(s) are not very clear to me (at the
> > time I was reading your script, which by now has been changed).
> 
> The scripts got a whole lot simpler, now basicly:
> 
> cp syslinux-6.04-pre1/efi64/efi/syslinux.efi $target/EFI/BOOT/BOOTX64.EFI
> cp syslinux-6.04-pre1/efi64/com32/elflink/ldlinux/ldlinux.e64 $target/EFI/BOOT/
> cp /usr/lib/syslinux/modules/efi64/* $target/EFI/syslinux
 
 
That last "cp" should, in theory, be optional (it could still be 
useful, but not essential). But, if you are adding all the c32 modules 
for efi64 (at this point, who knows why, and adding such amount of 
bytes, but OK, let's see where this goes), I am wondering why would you 
need to put them in another directory. Why not using 
"$target/EFI/BOOT/" as in your prior "cp" commands? Of course, we could 
think of eventually using a common set of cfg files (for BIOS and UEFI) 
in boot.img, and that would require the additional c32 files for 
efi(32/64) and some additional tweaks.

Also, ATM I would again suggest using the current Debian's "Testing" 
binaries.


> 
> > Could
> > you please post here the resulting directory tree, the list of file
> > names in each directory and the complete resulting content of all the
> > cfg files, without pointing us to a script?
> 
> carl at twist:/media/carl/Debian Inst$ tree -d
> └── EFI
>     ├── BOOT
>     └── syslinux
> 
> carl at twist:/media/carl/Debian Inst$ tree
> ├── adgtk.cfg
> ├── adspkgtk.cfg
> ├── adtxt.cfg
> ├── disk.lbl
> ├── EFI
> │   ├── BOOT
> │   │   ├── BOOTX64.EFI
> │   │   └── ldlinux.e64
> │   └── syslinux
> │       ├── cat.c32
> │       ├── chain.c32
> │       ├── cmd.c32
> │       ├── cmenu.c32
> │       ├── config.c32
> │       ├── cptime.c32
> │       ├── cpu.c32
> │       ├── cpuid.c32
> │       ├── cpuidtest.c32
> │       ├── debug.c32
> │       ├── dhcp.c32
> │       ├── disk.c32
> │       ├── dmi.c32
> │       ├── dmitest.c32
> │       ├── elf.c32
> │       ├── ethersel.c32
> │       ├── gfxboot.c32
> │       ├── gpxecmd.c32
> │       ├── hdt.c32
> │       ├── hexdump.c32
> │       ├── host.c32
> │       ├── ifcpu64.c32
> │       ├── ifcpu.c32
> │       ├── ifmemdsk.c32
> │       ├── ifplop.c32
> │       ├── kbdmap.c32
> │       ├── kontron_wdt.c32
> │       ├── ldlinux.e64
> │       ├── lfs.c32
> │       ├── libcom32.c32
> │       ├── libgpl.c32
> │       ├── liblua.c32
> │       ├── libmenu.c32
> │       ├── libutil.c32
> │       ├── linux.c32
> │       ├── ls.c32
> │       ├── lua.c32
> │       ├── mboot.c32
> │       ├── meminfo.c32
> │       ├── menu.c32
> │       ├── pci.c32
> │       ├── pcitest.c32
> │       ├── pmload.c32
> │       ├── poweroff.c32
> │       ├── prdhcp.c32
> │       ├── pwd.c32
> │       ├── pxechn.c32
> │       ├── reboot.c32
> │       ├── rosh.c32
> │       ├── sanboot.c32
> │       ├── sdi.c32
> │       ├── sysdump.c32
> │       ├── syslinux.c32
> │       ├── vesa.c32
> │       ├── vesainfo.c32
> │       ├── vesamenu.c32
> │       ├── vpdtest.c32
> │       ├── whichsys.c32
> │       └── zzjson.c32
> ├── exithelp.cfg
> ├── f10.txt
> ├── f1.txt
> ├── f2.txt
> ├── f3.txt
> ├── f4.txt
> ├── f5.txt
> ├── f6.txt
> ├── f7.txt
> ├── f8.txt
> ├── f9.txt
> ├── g2ldr
> ├── g2ldr.mbr
> ├── gtk.cfg
> ├── initrdg.gz
> ├── initrd.gz
> ├── ldlinux.c32
> ├── ldlinux.sys
> ├── libcom32.c32
> ├── libutil.c32
> ├── linux
> ├── menu.cfg
> ├── NvVars
> ├── prompt.cfg
> ├── rqgtk.cfg
> ├── rqspkgtk.cfg
> ├── rqtxt.cfg
> ├── setup.exe
> ├── spkgtk.cfg
> ├── splash.png
> ├── stdmenu.cfg
> ├── syslinux.cfg
> ├── txt.cfg
> ├── vesamenu.c32
> └── win32-loader.ini
> 
> carl at twist:/media/carl/Debian Inst$ cat syslinux.cfg
> # simple syslinux.cfg
> DEFAULT linux
> LABEL linux
> KERNEL linux
> INITRD initrd.gz
 
 
Thank you for posting the info.

I would suggest (at least for now):

DEFAULT linux
PROMPT 1
LABEL linux
LINUX /linux
INITRD /initrd.gz

The "PROMPT 1" is in preparation for potential troubleshooting.

In my suggestion, I also changed (for now) the paths to be absolute 
(note the initial "/" for the kernel and initrd files).

Please note the change from "KERNEL" to "LINUX".

IOW, if the boot fails, the user would receive a suggestion to change 
from "KERNEL" to "LINUX" and test again (because it _sometimes_ can 
help, when the size of kernel/initrd files is really "big"), so let's 
avoid such step and change it already.

Also, since you mentioned some kind of "menu" before, please note that 
my proposed "EFI/BOOT/syslinux.cfg" file would not present any boot 
menu (ATM).

Please also note that this is not "/syslinux.cfg" (already in 
boot.img).

Let's see whether there is any kind of improvement now (e.g. different 
messages, or to which boot step we can arrive with these changes, 
etc.).

Best Regards,
Ady.