[syslinux] [PATCH 0/2] improve Lua API for files and initramfs objects

[Ferenc Wágner]

"H. Peter Anvin" <hpa at zytor.com> writes:

> On 11/20/16 00:13, Ferenc Wágner via Syslinux wrote:
> 
>> This indeed copies the data from the syslinux_file structure (which is
>> an opaque SYSLINUX_FILE object on the Lua side) into an (interned) Lua
>> string to make it available to the Lua string functions.
>> 
>> You probably ask why we copy the data when we create those SYSLINUX_FILE
>> userdata objects.  We don't, the data is read directly into the
>> syslinux_file structure by the loadfile() Syslinux function.  This was
>> already the case when I first saw the Lua binding in Syslinux.  Thinking
>> about it now, the SYSLINUX_FILE userdata objects and their entire
>> interface could be replaced by Lua tables with two string fields for
>> much the same effect, unless the native Lua string I/O is much less
>> efficient than loadfile().  If we can drop the name method, even simple
>> Lua strings would suffice.  The complication arises when you assemble
>> initramfs objects incrementally: either you'll have to copy the data, or
>> keep a reference to each constituent Lua string and delay creating the
>> actual initramfs structure until its point of usage (and destroy it if
>> you somehow fail back to Lua).  This might be worth exploring.
>> 
>
> This seems like a very sensible approach to me.  Keeping a reference and
> just pointing into the Lua memory representation makes a lot more sense.
>
> [f]loadfile() is "smart" in the sense that if, and only if, the size of
> the file is known ahead of time it allocates the buffer in one shot,
> rather than using realloc().  Do note that [f]loadfile() (as opposed to
> initramfs_load_file()) just loads data into a memory buffer.

The catch is that it's impossible to create a Lua string without copying
the memory buffer constituting it.  At least I couldn't find a way.  It
can be an issue even when Lua itself reads a file into a string: first
it reads the stream contents into a series of buffers of geometrically
growing sizes, then collects them all into the final string.  This
guarrantees at least twice the memory usage of the minimum needed (in
the best case).  We could replace the implementation of read_all() in
liolib.c with floadfile() to get this "best case" all the time, but it
we can't afford the final copy (the double memory footprint) anyway, it
doesn't gain us much.

> If there is a way to represent a memory buffer allocated by C as a Lua
> string that would pretty much take care of the same thing.

I'm afraid this is not possible, which pretty much ends this story.
An unfortunate turn.

> floadfile() even allows a chunk of data to be prepended to the buffer,
> which could be useful in the case of a Lua string needing something
> prepended to the data buffer.

That's clever and gets us halfway there.  But that other half...
certainly not supported by the official API.  And we don't really want
to use kernels and initramfs images as strings, so it probably isn't
worth mucking with the internals.

> Creating struct initramfs at the point of execution seems reasonable,
> but on the other hand, is there a reason we can't simply add a reference
> to the Lua string and let it be referenced by struct initramfs using
> initramfs_add_file()?  The main shortcoming I see with that is the
> object is opaque, and it could be nice to be allowed to examine it, but
> that seems like a second-order feature.

I'm not sure I follow you.  Based on the above, we can't exclusively use
Lua strings to build the initramfs.  We can represent the initramfs as a
Lua list (a special table), which can contain an arbitrary mix of
Syslinux file objects (opaque from Lua) and Lua strings, and build the
struct initramfs at the point of execution.
-- 
Regards,
Feri