How does Hddrive support drives >2TiB size?

[AdamK]

XHDI interface has 32bit sector addressing. So, with sector size of 512 bytes (does any software on Atari support other size?) this means that the device size limit is 2TiB. Hddriver claims support for devices bigger than 2TiB. How does it work?

[uweseimet]

Just as the HDDRIVER website (news page) says, the SCSI emulation for IDE/SATA drives of the SCSI Driver supports READ/WRITE/VERIFY (16) and READ CAPACITY (16) now. So you can use these SCSI commands to access sectors beyond the 2 TiB limit not just for real SCSI drives but also for IDE/SATA now. Any software using the SCSI Driver can access all sectors of drives > 2 TiB. Applications making use of this (i. e. support 64 bit sector numbers) currently are HDDRUTIL and DISKUS.
TOS and XHDI do not support 64 bit sector numbers, only the SCSI Driver does. For MiNT it would probably not be a big deal to also support 64 bit sector numbers. MiNT would have to (maybe optionally) use the SCSI Driver interface instead of XHDI.

As far as sector sizes are concerned HDDRIVER has supported drives with physical sectors bigger than 512 bytes for ages. Otherwise DVD-RAM, DVD+RW and BD-RE media, which have 2048 bytes per sector, would not work. There are also hard disk drives that support physical sectors bigger than 512 bytes. Upcoming big SATA drives will have 4096 bytes or even more per sector, see the latest standards. Currently these drives still map these big physical sectors to smaller logical ones for compatibility reasons, but this is only supposed to be a transitional phase. HDDRUTL 10 displays information about this mapping in the "Medium Details" dialog. READ CAPACITY (16) provides the mapping ratio.

TOS does not care for physical sector sizes, it just cannot boot from drives with physical sectors > 512 bytes. Applications also don't care (don't even know) whether a drive has 512 bytes per sector or more. Whether a sector physically or just logically has more than 512 bytes is transparent for them.

[AdamK]

Thanks for the explanation. Can you point me to SCSI Driver documentation please?

[uweseimet]

You can download it (together with tools related to XHDI and the SCSI Driver) on the HDDRIVER downloads page.

[AdamK]

Ok, I'v read it, but working with it is a bit SCSI specific ;)

Would it be possible to extend XHDI to handle 64bit sector addresses?

[uweseimet]

I don't think so, because none of the XHDI-compatible drivers except HDDRIVER seems to be maintained anymore. So this would become a proprietary extension, which only HDDRIVER would support, which I something I do not really like.

Regarding the SCSI Driver, despite its name it is not really SCSI-specific (see the list of available drives on the HDDRIVER website), and it is also non-proprietary. Besides HDDIVER CHBD supports it for at least some device types. The interface is more low-level compared with XHDI, but I don't think that MiNT needs much more than calling READ/WRITE.

[AdamK]

With Hddriver being the only driver in development, I think that after a discussion on relevant mailinglist (FreeMiNT, EmuDOS, Hatari, Aranym) extending XHDI might be a good idea. With that it is just a matter of small changes in anything that wants to use it. With SCSIDriver, you have to figure out device mappings, etc, which I did not find in the documentation.

[uweseimet]

If several parties are involved in extending XHDI to support 64 bit sector numbers, and if this is a transparent process, that's fine for me. One should probably increment the XHDI major version in this case.
So please just go ahead and start a discussion in the relevant mailing lists. A first step would probably be to collect the requirements for new XHDI calls of the operating systems or OS extensions that would make use of the new features.

[uweseimet]

After further investigating this I see a major problem caused by an XHDI-based approach: Extending XHDI to support 64 bit sector numbers also affects most of the driver code dealing with regular BIOS calls. This is not really surprising because XHDI is essentially a BIOS extension. It's not a completely new layer like the SCSI Driver, and explicitly (see the XHDI specification) was not supposed to be.
To migrate most of the BIOS-related driver code from 32 to 64 bit is a major effort, which will negatively affect performance and the binary size. Additionally, the compatibility with AHDI data structures (see AHDI specification) would suffer.
The SCSI Driver, on the other hand, supports 64 bit out of the box and does not negatively interfere at all with other existing interfaces. GCC also supports 64 bit (ULONGLONG) out of the box. So instead of extending XHDI (i. e. a hard disk driver) with something only MiNT would use it makes more sense for MiNT to use the SCSI Driver interface directly. MiNT will have to support 64 bit sector numbers anyway, regardless of the approach.

Note that (at least partial) support of GUID partition tables (GPT) is also required because MBR partition tables do not support 64 bit sector numbers. Getting partition boundaries from GPT tables is not a big issue, though, and in some respects more convenient than with MBR partition tables. I have already done this with an experimental version of HDDRIVER, but GPT support within HDDRIVER itself does not make much sense in practice because TOS can only deal with 32 bit sector numbers anway.