[syslinux] linking external library into module

[jeff.williams at pideso.com]

I guess there were no dendrites to tickle. I did come up with a partial
answer, although why exactly these items are required is still somewhat
of a mystery to me. I'll probably investigate further at some point to
see about getting a full answer, but here's what I came up with so far.
So, in case anyone else is giving this a try:

1) It appears that external libraries will not link correctly into a
com32 module if they weren't built with the same flags as are used in
the syslinux com32 module build. There is no error output during the
link, you just end up with seemingly random run-time issues. This may
partially depend upon #2, below.

2) The final link command line order is surprisingly fussy. "$(LD)
$(LDFLAGS) -o $@ $^ $(C_LIBS)" builds a com32 module that runs pretty
reliably. "$(LD) -o $@ $^ $(LDFLAGS) $(C_LIBS)" builds a com32 module
that may or may not run depending upon how much and in what order your
.c source code may be. Order of flags on the tool's command line
matters? Looks like it. Weird.

Since this is now in the archive, maybe Google will have at least one
result next time.

Best regards,

Jeff Williams

> -------- Original Message --------
> Subject: linking external library into module
> From: <jeff.williams at pideso.com>
> Date: Tue, September 20, 2016 1:53 pm
> To: syslinux at zytor.com
> 
> 
> I'm hoping this tickles some dendrites as both google and the various
> documentation doesn't seem to provide any insight.
> 
> I'm writing a com32 module for use on a x86-32 machine with a BIOS (not
> EFI). The code that I write into the module executes correctly. However,
> I also have a static library that has several routines that I want to
> use. The static library doesn't rely on any other library for symbol
> resolution (e.g. no requirement for libc or anything else - it's just a
> collection of black-box algorithms).
> 
> When I link the library, but don't call any functions contained in the
> library, my com32 module code still works correctly, albeit not using
> the functionality found in the library. When I call a function or two in
> the library from my module code, things might or might not work
> correctly. If they go south, either I end up with the CPU running in
> circles or I get a seg-fault trying to execute an invalid address.
> Interestingly, if I *do* get a successful run when using one of the
> library functions, doing something as trivial as adding a single print
> to the module (*not* library) code can be enough to send me into the
> weeds again.
> 
> I've been debugging this using gdb 7.7.1 connected to a QEMU (x86-32)
> 2.0.0 target. My OS is Linux Mint 17.3 (Ubuntu 14.04 LTS) and gcc is
> 4.8.4. One of the myriad issues with that particular setup is the bug
> where, due to the compiler version, after I dig down a few levels into
> the call stack gdb loses context and I go from symbolic debugging to
> assembly language debugging. It should be noted that when I run the same
> binary on my actual target I get the same results, so this isn't an
> emulation issue.
> 
> Debugging this issue, I do see where, assembly-wise, the EIP gets
> assigned an invalid address, and it's not the same place from
> build-to-build, so I can't point to a particular source line that causes
> the problem. Mostly it looks like the final link is putting together a
> binary that steps on itself, but it's very unclear why.
> 
> I did try re-compiling the library with -Os rather than -O2 on the
> assumption that mixing the two different optimization levels might be
> causing a problem (yes, I know it shouldn't, but I've been fighting this
> for a couple of weeks) and didn't see any difference. I also spent some
> time with the linker script but was unable to find any magic that would
> make the external library and module code to play nicely together. I
> also tried a couple of other libraries with similar results.
> 
> Since all three libraries do work properly in other contexts, I'm pretty
> confident this has something to do with how the syslinux build works.
> Does anyone have any ideas on how to get a static library to link
> properly with a com32 module?
> 
> Best regards,
> 
> Jeff Williams