[syslinux] Cluster Size discrepancy between FAT32 and NTFS

[Gene Cumm]

On Tue, Jan 21, 2014 at 2:41 PM, mike setzer <qualityana at sbcglobal.net> wrote:
> hello Gene,
>
> good to get your message
>
> I have not seen a link to reply to a particular thread,
> so I am replying to you right now
> hopefully the thread can be made to look right, also I double posted since my connection was lost
> right during the first posting.

Sorry, I thought you were subscribed but this sounds as if you are not.

> On NTFS I have tested cluster sizes from 4096 up through 64K.
> Did not try smaller sizes than 4096.

Good to note.  The most likely answer is that the code either assumes
4096, assumes no more than 4096 OR there's actually a bug in your file
system.  I'll peek to see if I can tell from your VBR captures.

> Windows is bootable from these NTFS volumes having larger-than-4096 clusters, see further info below*.
> As for terminology details, 4096 is the default size for NTFS volumes.
> Standard sizes include only 512b, 1kb, 2kb, 4kb, 8kb, 16kb, 32kb, & 64kb.
> So I am not technically using larger than "standard", only larger than default.

Good to know.  512/1024/2048 are also the default for smaller file systems.

> The experimentation is being done right now on the third partition of a multiboot system,
>
> On this HDD the third partition is the third & final primary, 16GB nominal partition size.
>
> NTFS formatting is then using either windows XP or the Windows 8.1 preview.
> This is when the cluster size is specified to begin with, so for NTFS experimentation, the same NTFS partition is
> maintained after zeroing, only reformatted with varying cluster sizes then reimaged from windows WIM files
> which handle the cluster differences well.
>
> Linux (distro which supports FAT and/or NTFS) is correctly placed on FAT32 or NTFS
> in the windows filesystem (whatever cluster size) by copying & pasting using windows.  The bootable linux fileset is copied directly
> from the folders on the live linux CD or DVD, and in this way is bootable not much differently than having the disk fileset on a
> bootable FAT or NTFS USB drive.
> Linux is not "installed", just the live disk behavior is obtained.
> Later, using grub to address the proper bootsector, by chainloading either to a windows NT bootsector or syslinux
> bootsector, either windows or linux are booted from the grub menu.
>
> in this simple application, syslinux is used as a direct substitute for isolinux, apparently as designed

It took reading your instructions to see that you prepared your boot
sector properly.

> The larger cluster sizes provide noticably faster boot times.

Faster for what OS?

> *Windows does have a bug about the cluster size itself.
> The windows setup routine does not support over 4096,
> Windows filesets are deployed to larger-than-4096 volumes by extracting from WIM images.
> Also the windows bootmanager is limited, where if the windows system files (WINDOWS folder, etc.) are on a NTFS volume having clusters larger than 4096, then the windows BOOT folder must be on a different volume having 4096 (or less?) if NTFS,
> or up to 64K if FAT.

I interpret this statement as "The Windows bootloader is incabable of
directly booting from NTFS with 8k+ clusters" as you need a jump point
first.  I'd say this now becomes an even more rare occurrence to
encounter since you state even Windows can't boot like this.

> The syslinux limitation of apparently booting only to the volume containing the target ldlinux.sys file I do consider ideal,
> but that feature precludes using syslinux as a direct substitute for NT bootmanagers which have always allowed
> separate boot & system volumes.
> Grub legacy does address the separate boot & system volumes fine so I see no need for that in syslinux.

multi-fs is coming.

> That is why it seems to me a good approach would be to correctly support larger NTFS cluster sizes in syslinux
> similar to how they are acceptable in FAT.
>
> also, in all cases it can be seen that syslinuxing an NTFS volume fails to replace the backup NT bootsector which
> is located at the final sector of the NTFS partition.  Syslinux only prepares the working bootsector (first sector of the partition) as a linux VBR,
> apparently leaving the backup (final sector of the partition) unchanged.

This is an overall unusual set of circumstances.

Why do you want to use NTFS with 8k+ clusters?  Why not consider the
same workaround that Windows requires?

-- 
-Gene

A: Because it messes up the order in which people normally read text,
especially the archives of mailing lists.
Q: Why is Top-posting such a bad thing?

> here is the preparation log for the attached bootsectors:
>
> this is a traditional HDD having 512byte sectors
> on a traditional BIOS/MBR system
> using LBA
>
> existing partition created by windows 8.1
> -this is primary partition3 of a multipartitioned HDD
> -which displays same behavior as partition1 of a single-volume HDD
> bootsector in place at desired sector 19595264
> recognized by windows as volume Z:
>
> TEST OF 4096 cluster size on NTFS
>
> wrote zeros to partition
> dd if=/dev/zero of=/dev/sda3
> still recognized by windows as unformatted volume Z:
> -partition table in sector0 which defines partition & location is unchanged by zeroing partition3 alone
>
> format using admin CMD prompt in W81 - "default" cluster size:
> FORMAT Z: /Q /FS:NTFS /A:4096 /V:NTFStest
> there is now a windows bootsector in place at 19595264 as a result of the formatting process.
> save this new NT bootsector as NT3_4K.bin
> now intentionally overwrite with what should be a duplicate windows bootsector:
> BOOTSECT /NT60 Z: /FORCE
> save this new NT bootsector as NT3B_4K.bin
> -if a bootable windows fileset with its BOOT folder were placed onto volume Z: at this point,
> -it would boot normally if this partiton3 was made active
>
> using windows,
> copied live fileset from puppy linux Lucid Puppy 5.28 live CD,
> pasted onto NTFS volume Z:
> -this is a typical modern puppy live CD naturally supporting FAT & NTFS.
> on volume Z:, rename isolinux.cfg to syslinux.cfg
>
> using windows and win32 syslinux version 4.07, overwrite the NT bootsector with a linux bootsector
> SYSLINUX -F Z:
> -ldlinux.sys is created on volume Z: at this point
> save the new syslinux bootsector as SL3_4K.bin
>
> set partition3 active and boot to it from MBR or chainload (hd0,2)+1 from grub legacy floppy
>
> PUPPY BOOTS as expected similar to a live CD, no differently than using a FAT volume having up to 64K clusters
>
> TEST OF 64K cluster size on NTFS
>
> wrote zeros to partition
> dd if=/dev/zero of=/dev/sda3
> still recognized by windows as unformatted volume Z:
> -partition table in sector0 which defines partition & location is unchanged by zeroing partition3 alone
>
> format using admin CMD prompt in W81 - largest cluster size
> FORMAT Z: /Q /FS:NTFS /A:64K /V:NTFStest
> there is now a windows bootsector in place at 19595264 as a result of the formatting process.
> save this new NT bootsector as NT3_64K.bin
> now intentionally overwrite with what should be a duplicate windows bootsector:
> BOOTSECT /NT60 Z: /FORCE
> save this new NT bootsector as NT3B_64K.bin
> -if a bootable windows fileset with its BOOT folder were placed onto volume Z: at this point,
> -it would not actually boot if this partiton3 was made active, windows bootmanager can not find all needed boot files
> -when the cluster size is greater than 4096 if formatted using NTFS.
> -instead it would require booting through a windows bootmanager (with BOOT folder) residing on a different active volume,
> -one having cluster size up to 64K if FAT or only up to 4096 if NTFS, or alternatively from a floppy
> -containing the windows bootmanager (with BOOT folder)

This in my mind is critical.  Linux can also have a separate /boot file system.

> using windows,
> copied live fileset from puppy linux Lucid Puppy 5.28 live CD,
> pasted onto NTFS volume Z:
> -this is a typical modern puppy live CD naturally supporting FAT & NTFS.
> on volume Z:, rename isolinux.cfg to syslinux.cfg
>
> using windows and win32 syslinux version 4.07, overwrite the NT bootsector with a linux bootsector
> SYSLINUX -F Z:
> -ldlinux.sys is created on volume Z: at this point
> save the new syslinux bootsector as SL3_64K.bin
>
> set partition3 active and boot to it from MBR or chainload (hd0,2)+1 from grub legacy floppy
>
> PUPPY FAILS TO BOOT, syslinux stops after displaying a single line followed by carriage return and blinking cursor:
> SYSLINUX 4.07 EDD 2013-07-25 Copyright (C) 1994-2013 H. Peter Anvin et al
> _
>
> at this point keyboard is now unresponsive
>
> repeating with progressively smaller NTFS cluster sizes continues to fail until 4096 is reached
>
> --------------------------------------------
> On Sun, 1/19/14, Gene Cumm <gene.cumm at gmail.com> wrote:
>
>  Subject: Re: [syslinux] Cluster Size discrepancy between FAT32 and NTFS
>  To: "mike setzer" <qualityana at sbcglobal.net>, "For discussion of Syslinux and tftp-hpa" <syslinux at zytor.com>
>  Date: Sunday, January 19, 2014, 10:57 AM
>
>  On Sun, Jan 19, 2014 at 9:44 AM, mike
>  setzer <qualityana at sbcglobal.net>
>  wrote:
>  > Hi,
>  >
>  > I am not an engineer or linux expert but I have been
>  using syslinux to boot
>  > live disk filesets (extracted from iso's) residing on
>  fat32 and NTFS volumes.
>  >
>  > In FAT32 there has been no problem going with larger
>  cluster sizes up to the nominal maximum of 64K
>  >
>  > however, with NTFS it has not been possible to exceed
>  the cluster size of 4096.
>  > With NTFS formatted using clusters of 8192 or larger,
>  > upon boot apparently the syslinux bootsector is
>  > read since the syslinux version is displayed, but
>  progress stops with no error message
>
>  Have you only tested 4096 and 8192?  Testing 16k+
>  probably won't be
>  useful if 8k fails but it may be good to know about
>  512/1024/2048
>  (eventually).
>
>  Are these NTFS volumes with non-standard cluster sizes
>  bootable by Windows?
>
>  How was the file system created?  Is there any way you
>  could export
>  the first sector of the partition to a file and either post
>  it
>  directly or a hex representation of it?  The first
>  sector should
>  specify things like sectors per cluster but it's possible
>  something is
>  being misinterpreted.
>
>  > this has been tested using real machines, not virtual
>  >
>  > ability to handle larger cluster sizes would seem to be
>  as useful for NTFS as in FAT32,
>  > if not more so.
>  >
>  > I stand by prepared for immediate testing of win32
>  syslinux versions as early as 4.06 which contain
>  > modifications to include this property
>
>  --
>  -Gene
>
>  search terms: 0.5 kiB, 1kiB, 2kiB, 4kiB 8kiB; half k, 1k,
>  2k, 4k, 8k;
>  0.5kB, 1kB, 2kB, 4kB, 8kB