[syslinux] fixing debian's hd-media image

[Carl Karsten]

On Sat, Nov 17, 2018 at 1:54 AM Ady Ady via Syslinux <syslinux at zytor.com> wrote:
>
>
> > On Thu, Nov 15, 2018 at 1:10 PM Ady Ady via Syslinux <syslinux at zytor.com> wrote:
> > >
> > > Here are some comments and questions for you.
> > >
> >
> > Thanks.
> >
> > I've have my scripts to a better state, including installing/using
> > qemu to uefi and legacy boot the usb stick for testing.
> >
> > It mostly works for me, but the "wget a tarball of binaries" step
> > isn't going to sit well with the debian folk, so I'm not done yet.  I
> > may be close enough that they will know how to do it properly.
>
>
> As I mentioned before, the current "Testing" repo in Debian has the
> most updated binaries of any Linux distro, and even newer (and better)
> than official upstream 6.04-pre1.
>
> The current "syslinux.efi" files for Debian (both, for efi32 and for
> efi64) can be downloaded from the "syslinux-efi" package.
>
> For the "Testing" repo, the mirrors (and links) can be seen at:
>  https://packages.debian.org/testing/all/syslinux-efi/download
>
> The Syslinux-related files in Debian's packages are scattered through
> several binary packages. Unfortunately, the specific way in which these
> files are arranged makes not much sense. So, in order to get
> "ldlinux.e64" (and/or "ldlinux.e32", and/or "ldlinux.c32), you need to
> also get another ("big") binary package from Debian, "syslinux-common":
>
>  https://packages.debian.org/testing/all/syslinux-common/download
>
>
> >
> >
> > > > I want to eufi boot this image:
> > > > http://cdn-fastly.deb.debian.org/debian/dists/testing/main/installer-amd64/current/images/hd-media/boot.img.gz
> > >
> > >
> > > I am going to assume that by "boot this image" you don't mean something
> > > like memdisk or equivalent, but rather that you want to transfer that
> > > image using "dd" to your (USB) device and boot the resulting device in
> > > UEFI mode. Is this assumption correct?
> >
> > correct.
> >
> >
> > >
> > >
> > > >
> > > > there is a bug against it:
> > > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=913523
> > >
> > >
> > > Although this is the Syslinux Mailing List and not Debian's in
> > > particular...
> >
> > syslinux list because I am trying to UEFI boot syslinux.
>
>
> OK. So, at least for now, we are not talking about any kind of fix nor
> bug report about Syslinux itself. Now things are getting clearer.
>
>
> >
> > > I would say that the above bug report is slightly
> > > confusing. AFAIK, the files under "hd-media" are aimed at a specific
> > > scenario (i.e. starting the _installation_ of the OS from a filesystem
> > > volume that is already part of the HDD).
> >
> > correct, kinda.  it boots a linux kernel, and runs the installer.
> > Unless it is a UEFI only box, and then it doesn't boot, which is a
> > problem.
>
>
> Since the "hd-image" boot.img file was never prepared for UEFI booting
> (whereas the kernel and intird files within it might, perhaps, already
> support such case), then we are not talking about a traditional "bug"
> but rather a feature request, for Debian (D-I, not for Syslinux). Now
> we are rising the level of clarity. Good :).
>
>
> >
> > > I could be wrong but, where
> > > does it say that the boot.img(.gz) mentioned in that report is aimed at
> > > (or that it supports) booting Debian (itself) from USB?
> >
> > It's implied.  or desired, so we could call it a feature request.
>
>
> Feature request for Debian (D-I). Agreed.
>
>
> >
> > >
> > > Moreover, where is the official info claiming that the boot.img(.gz)
> > > file can support booting in UEFI mode? (FWIW, ATM it doesn't, unless
> > > you add "some things").
> >
> > correct - I was confused that a UEFI box booted it using compatibility mode.
>
>
> CSM is like BIOS, so, sure, it should be bootable in such case (unless
> the "BIOS" is buggy, in which case YMMV).
>
>
> >
> > >
> > > Additionally, the boot.img(.gz) file is not supposed to have a
> > > partition table. Not having a partition table is not a bug for that
> > > image (the image is a (potentially bootable) filesystem volume). I
> > > could be wrong but, AFAIU, the image is not aimed at being written to
> > > the HDD as a complete, partitioned HDD. You could write a device with
> > > it, but it would then be a (super)floppy-like device. Whether such
> > > device effectively serves its purpose of launching the installer (for
> > > Debian), that's another matter.
> >
> > all correct.
>
>
> The reason I mentioned that using a fs directly, for UEFI booting, is
> supposed to be supported, is because I have read the comments in the
> bug report (for Debian) you mentioned before. Whoever takes care (or
> plainly reads) that bug report might (wrongly) think that a GPT should
> be added to boot.img; it should _not_ be. Let me rephrase it: it shall
> not be added.
>
> The boot.img needs a UEFI-compatible bootloader within it, using the
> "default" file names (e.g. "/EFI/BOOT/BOOTX64.EFI") for the feature
> request to be resolved.
>
>
> >
> > >
> > > Now, perhaps an important detail to be aware of: the UEFI specs indeed
> > > support booting a filesystem volume in UEFI mode. Just think of it as a
> > > (super)floppy (i.e. no partition table, no MBR, _no GPT_).
> >
> > I was wondering about that.   I'll stop trying to figure out how to do
> > that, so thank you,
> >
> > >
> > >
> > >
> > > >
> > > > I'd like to fix the resulting usb stick for my use, and also fix it in
> > > > a way that can be used to fix the upstream bug.
> > >
> > >
> > > You are using the term "fix" twice here, and I have to admit that, to
> > > me, it is not at all clear what exactly you mean.
> >
> > I would like a quick workaround so I can boot the box I am in
> > possession of (which I have now achieved)
> > and also something that can be contributed to the debian project.  (I'm close)
> >
> > >
> > > What exactly is wrong in the resulting USB stick?
> >
> > It is built by me floundering around, call it prof of concept.
> > I'm a little bothered that syslinux.efi isn't in the debian package,
> > and should I use the ldlinux.e64 from the tarbal or package?
>
>
> See my reply above, in this same email, about both files and where to
> get them.
>
> Also, as mentioned more than once in the wiki, Syslinux files should
> come from the exact same package / build / version; mixing files from
> different packages / builds / versions should be avoided.
>
> In theory, you could use one set of Syslinux files (one version) for
> BIOS and another (version) for UEFI. I wouldn't recommend it.
>
>
> >
> > >
> > > What is "upstream" to you in this context? Debian? D-I? The Syslinux
> > > Project? Other?
> >
> > Debian.
> >
> > I think add some lines around here:
> >
> > https://salsa.debian.org/installer-team/debian-installer/blob/master/build/config/x86.cfg#L113-115
> >
> > mcopy -i$(TEMP_BOOT) /usr/lib/syslinux/modules/bios/vesamenu.c32
> > ::vesamenu.c32; \
> > mcopy -i$(TEMP_BOOT) /usr/lib/syslinux/modules/bios/libcom32.c32
> > ::libcom32.c32; \
> > mcopy -i$(TEMP_BOOT) /usr/lib/syslinux/modules/bios/libutil.c32
> > ::libutil.c32 ; \
>
>
> I'm confused. Why exactly those (BIOS) files are supposed to be related
> to your feature request? Could you please explain (what exactly is
> triggering this thought)?

This is the existing code to build hd-media/boot.img,

It is where I think some lines need to be added to support uefi.


>
>
> >
> > >
> > >
> > > >
> > > > I'm trying to follow the guids
> > > > https://www.syslinux.org/wiki/index.php?title=Install#UEFI
> > > > and a ml post - here is what I have so far:
> > > >
> > > > https://salsa.debian.org/carlfk-guest/ansible/blob/uefiusb/usbinst/uefi/fix_hdmedia.sh
> > > > (likely quite a bit of unneeded copies)
> > > >
> > > > it boots, (good)
> > > > Initial menu has no LABEL enries!
> > >
> > >
> > > I have to say that your cfg file(s) are not very clear to me (at the
> > > time I was reading your script, which by now has been changed).
> >
> > The scripts got a whole lot simpler, now basicly:
> >
> > cp syslinux-6.04-pre1/efi64/efi/syslinux.efi $target/EFI/BOOT/BOOTX64.EFI
> > cp syslinux-6.04-pre1/efi64/com32/elflink/ldlinux/ldlinux.e64 $target/EFI/BOOT/
> > cp /usr/lib/syslinux/modules/efi64/* $target/EFI/syslinux
>
>
> That last "cp" should, in theory, be optional (it could still be
> useful, but not essential). But, if you are adding all the c32 modules
> for efi64 (at this point, who knows why, and adding such amount of
> bytes, but OK, let's see where this goes), I am wondering why would you
> need to put them in another directory. Why not using
> "$target/EFI/BOOT/" as in your prior "cp" commands? Of course, we could
> think of eventually using a common set of cfg files (for BIOS and UEFI)
> in boot.img, and that would require the additional c32 files for
> efi(32/64) and some additional tweaks.

I'm not sure where I got the idea to put the c32 modules under a syslinx/ dir.


>
> Also, ATM I would again suggest using the current Debian's "Testing"
> binaries.

now that I see the syslinux-efi package I can apt get everything
needed, which will make it easier to keep everything in sync.



>
>
> >
> > > Could
> > > you please post here the resulting directory tree, the list of file
> > > names in each directory and the complete resulting content of all the
> > > cfg files, without pointing us to a script?
> >
> > carl at twist:/media/carl/Debian Inst$ tree -d
> > └── EFI
> >     ├── BOOT
> >     └── syslinux
> >
> > carl at twist:/media/carl/Debian Inst$ tree
> > ├── adgtk.cfg
> > ├── adspkgtk.cfg
> > ├── adtxt.cfg
> > ├── disk.lbl
> > ├── EFI
> > │   ├── BOOT
> > │   │   ├── BOOTX64.EFI
> > │   │   └── ldlinux.e64
> > │   └── syslinux
> > │       ├── cat.c32
> > │       ├── chain.c32
> > │       ├── cmd.c32
> > │       ├── cmenu.c32
> > │       ├── config.c32
> > │       ├── cptime.c32
> > │       ├── cpu.c32
> > │       ├── cpuid.c32
> > │       ├── cpuidtest.c32
> > │       ├── debug.c32
> > │       ├── dhcp.c32
> > │       ├── disk.c32
> > │       ├── dmi.c32
> > │       ├── dmitest.c32
> > │       ├── elf.c32
> > │       ├── ethersel.c32
> > │       ├── gfxboot.c32
> > │       ├── gpxecmd.c32
> > │       ├── hdt.c32
> > │       ├── hexdump.c32
> > │       ├── host.c32
> > │       ├── ifcpu64.c32
> > │       ├── ifcpu.c32
> > │       ├── ifmemdsk.c32
> > │       ├── ifplop.c32
> > │       ├── kbdmap.c32
> > │       ├── kontron_wdt.c32
> > │       ├── ldlinux.e64
> > │       ├── lfs.c32
> > │       ├── libcom32.c32
> > │       ├── libgpl.c32
> > │       ├── liblua.c32
> > │       ├── libmenu.c32
> > │       ├── libutil.c32
> > │       ├── linux.c32
> > │       ├── ls.c32
> > │       ├── lua.c32
> > │       ├── mboot.c32
> > │       ├── meminfo.c32
> > │       ├── menu.c32
> > │       ├── pci.c32
> > │       ├── pcitest.c32
> > │       ├── pmload.c32
> > │       ├── poweroff.c32
> > │       ├── prdhcp.c32
> > │       ├── pwd.c32
> > │       ├── pxechn.c32
> > │       ├── reboot.c32
> > │       ├── rosh.c32
> > │       ├── sanboot.c32
> > │       ├── sdi.c32
> > │       ├── sysdump.c32
> > │       ├── syslinux.c32
> > │       ├── vesa.c32
> > │       ├── vesainfo.c32
> > │       ├── vesamenu.c32
> > │       ├── vpdtest.c32
> > │       ├── whichsys.c32
> > │       └── zzjson.c32
> > ├── exithelp.cfg
> > ├── f10.txt
> > ├── f1.txt
> > ├── f2.txt
> > ├── f3.txt
> > ├── f4.txt
> > ├── f5.txt
> > ├── f6.txt
> > ├── f7.txt
> > ├── f8.txt
> > ├── f9.txt
> > ├── g2ldr
> > ├── g2ldr.mbr
> > ├── gtk.cfg
> > ├── initrdg.gz
> > ├── initrd.gz
> > ├── ldlinux.c32
> > ├── ldlinux.sys
> > ├── libcom32.c32
> > ├── libutil.c32
> > ├── linux
> > ├── menu.cfg
> > ├── NvVars
> > ├── prompt.cfg
> > ├── rqgtk.cfg
> > ├── rqspkgtk.cfg
> > ├── rqtxt.cfg
> > ├── setup.exe
> > ├── spkgtk.cfg
> > ├── splash.png
> > ├── stdmenu.cfg
> > ├── syslinux.cfg
> > ├── txt.cfg
> > ├── vesamenu.c32
> > └── win32-loader.ini
> >
> > carl at twist:/media/carl/Debian Inst$ cat syslinux.cfg
> > # simple syslinux.cfg
> > DEFAULT linux
> > LABEL linux
> > KERNEL linux
> > INITRD initrd.gz
>
>
> Thank you for posting the info.
>
> I would suggest (at least for now):
>
> DEFAULT linux
> PROMPT 1
> LABEL linux
> LINUX /linux
> INITRD /initrd.gz
>
> The "PROMPT 1" is in preparation for potential troubleshooting.
>
> In my suggestion, I also changed (for now) the paths to be absolute
> (note the initial "/" for the kernel and initrd files).
>
> Please note the change from "KERNEL" to "LINUX".
>
> IOW, if the boot fails, the user would receive a suggestion to change
> from "KERNEL" to "LINUX" and test again (because it _sometimes_ can
> help, when the size of kernel/initrd files is really "big"), so let's
> avoid such step and change it already.
>
> Also, since you mentioned some kind of "menu" before, please note that
> my proposed "EFI/BOOT/syslinux.cfg" file would not present any boot
> menu (ATM).
>
> Please also note that this is not "/syslinux.cfg" (already in
> boot.img).
>
> Let's see whether there is any kind of improvement now (e.g. different
> messages, or to which boot step we can arrive with these changes,
> etc.).

This all works.  thank you again for the detailed reply.

But. the existing menu does not work.

2 problems I have identified:
If I don't put the above syslinux.cfg and go with what is on the d-i
setup, this seems to hang it:

default vesamenu.c32

#comment that out, and I get to another problem that I haven't taken
the time to track down yet.
Does any of this look like a problem?

carl at twist:~/src/tv/lca/ansible/usbinst/uefi$ cat /media/sdc/syslinux.cfg
# D-I config version 2.0
# search path for the c32 support libraries (libcom32, libutil etc.)
path
include menu.cfg
# default vesamenu.c32
prompt 0
timeout 0

carl at twist:~/src/tv/lca/ansible/usbinst/uefi$ cat /media/sdc/menu.cfg
menu hshift 7
menu width 61

menu title Debian GNU/Linux installer boot menu
include stdmenu.cfg
include gtk.cfg
include txt.cfg
menu begin advanced
    menu label ^Advanced options
    menu title Advanced options
    include stdmenu.cfg
    label mainmenu
        menu label ^Back..
        menu exit
    include adgtk.cfg
    include adtxt.cfg
    include adspkgtk.cfg
    include adspk.cfg
menu end
include x86menu.cfg
label help
    menu label ^Help
    text help
   Display help screens; type 'menu' at boot prompt to return to this menu
    endtext
    config prompt.cfg
include spkgtk.cfg
include spk.cfg

carl at twist:~/src/tv/lca/ansible/usbinst/uefi$ cat /media/sdc/stdmenu.cfg
menu background splash.png
menu color title    * #FFFFFFFF *
menu color border    * #00000000 #00000000 none
menu color sel        * #ffffffff #76a1d0ff *
menu color hotsel    1;7;37;40 #ffffffff #76a1d0ff *
menu color tabmsg    * #ffffffff #00000000 *
menu color help        37;40 #ffdddd00 #00000000 none
# XXX When adjusting vshift, take care that rows is set to a small
# enough value so any possible menu will fit on the screen,
# rather than falling off the bottom.
menu vshift 8
menu rows 12
# The help line must be at least one line from the bottom.
menu helpmsgrow 14
# The command line must be at least one line from the help line.
menu cmdlinerow 16
menu timeoutrow 16
menu tabmsgrow 18
menu tabmsg Press ENTER to boot or TAB to edit a menu entry

carl at twist:~/src/tv/lca/ansible/usbinst/uefi$ cat /media/sdc/txt.cfg
label install
    menu label ^Install
    kernel linux
    append vga=788 initrd=initrd.gz --- quiet
carl at twist:~/src/tv/lca/ansible/usbinst/uefi$ cat /media/sdc/gtk.cfg
default installgui
label installgui
    menu label ^Graphical install
    menu default
    kernel linux
    append vga=788 initrd=initrdg.gz --- quiet
carl at twist:~/src/tv/lca/ansible/usbinst/uefi$

>
> Best Regards,
> Ady.
>
> _______________________________________________
> Syslinux mailing list
> Submissions to Syslinux at zytor.com
> Unsubscribe or set options at:
> http://www.zytor.com/mailman/listinfo/syslinux



--
Carl K