The arrow of time

Ivan Voras' blog

FreeBSD on 4K sector drives

All major FreeBSD filesystems support 4K sectors (UFS, ZFS, ext2), and so does the lower level - GEOM - but currently there's an issue of communicating this configuration between all the layers. A part of the problem is that the current drives (and the situation will probably not change during this new decade) advertise two sector sizes: both 512 byte and 4K, and the system needs to correctly interpret them. All this will be resolved when a consensus on the topic gets achieved, but until that happens (hopefully soon), there is a set of easy workarounds, which I'll describe here.

GEOM and the file systems do all IO access in byte offsets, so the underlying sector size isn't important as long as it's addressable. For instance, a command like "dd if=/dev/da0 bs=10 count=1 of=/tmp/foo" will fail since /dev/da0 is a raw device which cannot be read in increments of 10 bytes - only at "physical sector size", which is 512 (change bs=10 to bs=512). This means that basically, any file system that can do most of its IO as 4K-aligned will automagically work "good enough" in the configuration where the drive can do 512 byte access but optimally does 4K access. Luckily, UFS and ZFS can do all accesses aligned to 4K so they can work even on "pure" 4K drives. In fact, if the drive adverties 4K sector size as "one true sector size", both file systems will do fine by default, without any additional configuration. The rest of this text applies to the situation where the drives advertise both sizes.

1. Aligning partitions

The first step is to align the partitions to 4K offsets. Though again GEOM doesn't care, both fdisk and disklabel tools still behave like they are bound by old defunct CHS requirements and by default produce silly offsets like "63 sectors".

The default sysinstall configuration will also do this, so if installing on a 512/4K drive, manual intervention is required to make the partitions aligned to 4K (or in other words, 8 512 byte sectors). In this case, 64 sectors is a good alignment, 63 sectors is not.

Using the manual gpart command, both the "start" and the "size" arguments need to be 4K-aligned. Old fdisk and disklabel commands should not be used in FreeBSD 8+.

2. UFS

UFS formatting (newfs) reques only that the fragment size be set to 4096:

newfs -U -f 4096 /dev/da0

This will basically ensure most of the IO is done in 4K offsets. FreeBSD 9 will use 4K fragments by default.

3. ZFS

ZFS requires basically the same things like UFS, only the "zpool create" command cannot accept an argument telling it what the alignment should be (ashift). To make ZFS issue 4K-aligned IO, the "zpool create" phase needs to be tricked into thinking the drive has 4K physical sectors. This can be done by using gnop - a GEOM class which is used for testing and which normally does not produce permanent changes (i.e. it doesn't write its metadata).

For each device which will be made a part of the pool, a gnop "chained" device needs to be created, and then those devices need to be added to a pool:

gnop create -S 4096 /dev/da0
gnop create -S 4096 /dev/da1
zpool create data mirror /dev/da0.nop /dev/da1.nop

This will create the "data" pool with the *.nop devices, making ZFS think the drives have 4K physical sectors. Next, the pool needs to be exported and the gnop devices removed:

zpool export data
gnop destroy /dev/da0.nop /dev/da1.nop

Next, the pool can be imported from the "raw" devices:

zpool import data

You can check the configuration of the pool by using the "zdb" command on the pool:

zdb -C data | grep ashift

The ashift should be "12" for 4K alignment. This works because ZFS writes the ashift value in its metadata.

There could be problems with booting from such drives, but apparently there are patches to fix those.

4. Compatibility

For compatibility with older machines (which cannot boot from "pure" 4K drives) and operating systems (which cannot parse them), the 512/4K dual identification will probably remain with us for some time as one of the "legacy" technologies which are common in the PC world. It is because of this that the steps described above are necessary - if any drives appear which only advertise the 4K size and the BIOS (or EFI) and the controllers support it, FreeBSD will support it automagically, without the configuration dance described above.

The described methods are "future proof" even if the drive, the controller, BIOS or the OS decide one day to ignore the 512 byte advertisments and pronounce the drives as "pure 4K".

The changes are also, of course, irreversible. If for some reason it is discovered that the drives need to be used as 512 byte drives (e.g. booting problems, etc.), the only way to do it is to reformat them (i.e. destroy the ZFS pools, or use newfs again).

#1 Re: FreeBSD on 4K sector drives

Added on 2011-01-01T23:29 by Lukasz

Lots of talk about mixed 512 and 4K drives on FreeBSD forums lately. Thanks for this clean and simple guide.

#2 Re: FreeBSD on 4K sector drives

Added on 2011-01-03T21:44 by Frank Wall

Thank you!

#3 Re: FreeBSD on 4K sector drives

Added on 2011-01-06T13:58 by DES

Several errors in this post.

Firstly, the problem is not that drives advertise two sizes - quite the opposite! The problem is that current consumer-grade 4kB drives advertise 512B sectors and are indistinguishable from true 512B drives. I thought of adding a hint mechanism to the ata / ada stack to recognize the drives by name, but people may have pre-existing misaligned file systems on their 4k drives, so we can't just throw the switch...

Secondly, both fdisk and sysinstall force an incorrect partition alignment; you can not specify the correct alignment manually (unless you choose an alignment which is a multiple of both 8 and 63, which are coprime, so a multiple of 504 sectors).

Thirdly, disklabel (now called bsdlabel) does not know or care about CHS. It will offset the first partition by 16 sectors for historical reasons, but that is not a problem since 16 is a multiple of 8.

My recommendation is simply to use gpart instead of fdisk / bsdlabel, and do manual installs until we get a new sysinstall that understands GPT. Or just stay away from 4k drives, since the ones I've tested are extremely slow, even when the file system is correctly aligned.

See http://maycontaintracesofbolts.blogspot.com/search/label/freebsd for additional information.

#4 Re: FreeBSD on 4K sector drives

Added on 2011-01-06T14:41 by Ivan Voras

Not quite...

1) Drives do ship with 512/4K advertising (the WD drives seem to be an exception in that they lack 4K advertising) and FreeBSD currently has no way of propagating this information all around. From your "throw the switch" phrase it looks like you are thinking of simply pronouncing the drives as "pure" 4K drives (i.e. sectorsize=4096); I've also advocated this position a few months back but I've learned that it would do more harm than good. For compatibility, FreeBSD will just have to do what everyone else in the industry does and cope with 512/4096 advertising.

2) Neither fdisk nor sysinstall force 63-sector alignments (at least in 8-stable), though in sysinstall you have to use the atrocious and wholly undocumented "wizard mode". Try them, like I did.

(you're right about disklabel/bsdlabel)

#5 Re: FreeBSD on 4K sector drives

Added on 2011-01-06T18:37 by DES

fdisk does force cylinder alignment - believe me, I tried. I considered fixing it, but decided to just use gpart instead.

#6 Re: FreeBSD on 4K sector drives

Added on 2011-01-06T21:03 by Ivan Voras

Ok, please examine this script session: http://people.freebsd.org/~ivoras/stuff/fdisk.asc (for example with "less -r") and tell me what you think. As I see it, fdisk accepted a partition starting in sector 64, i.e. with parameters "beg: cyl 0/ head 1/ sector 2;". Am I wrong?

#7 Re: FreeBSD on 4K sector drives

Added on 2011-01-08T02:09 by DES

Which version is this?

The last time I tried (in September, I think) it silently adjusted the partition to cylinder boundaries, so I went and learned gpart instead.

#8 Re: FreeBSD on 4K sector drives

Added on 2011-01-08T02:37 by Ivan Voras

I've said it before: 8-stable. Maybe someone fixed it separately from you, but also as I said: both fdisk and sysinstall can be talked into desired alignments.

Of course, fdisk and disklabel really should be fixed or evicted, gpart is currently the way to go. If everything falls in its place for me, I'll probably rewrite both - there's a need for interactive partitioning tools even if it's only psychological.

#9 Re: FreeBSD on 4K sector drives

Added on 2011-08-15T13:22 by Martin Matuska

What I noticed: transferring ZFS snapshots from drives with ashift=9 to drives created with the above instructions (ashift=12) will result in larger space usage (= not good)

#10 Re: FreeBSD on 4K sector drives

Added on 2011-08-15T13:25 by Ivan Voras

Yeah but that's really expected - you DO have larger sector sizes (and more slack space) on 4K drives.

#11 Re: FreeBSD on 4K sector drives

Added on 2011-08-22T14:36 by tcb
Strangely I still only get about 9 to 15 MB/s writespeed on my single WD20EARS-60MVWB0.

#12 Re: FreeBSD on 4K sector drives

Added on 2011-11-07T02:53 by Floren

Interesting post. I recently opened a FreeNAS 8.0.2 ticket, hoping there is a way to correct the 4k misaligned disks:

https://support.freenas.org/ticket/984

If anyone knows a way to properly built the array, please let me know.

#13 Re: FreeBSD on 4K sector drives

Added on 2012-01-31T05:44 by Dan Pritts

Thanks for the post, it was very helpful.

Note that you do not need to create gnop devices for every device in your zpool. Doing it with just one of them forces the zpool to be created with the right ashift.

#14 Re: FreeBSD on 4K sector drives

Added on 2012-07-10T15:08 by Rick

Does this mean that installing FreeBSD 8.x with sysinstall and fdisk will result in slices not being laid out at head boundaries on 4K sector drives?

#15 Re: FreeBSD on 4K sector drives

Added on 2012-07-10T15:13 by Ivan Voras

Yes, FreeBSD 8 does nothing to align partitions and file systems.

#16 Re: FreeBSD on 4K sector drives

Added on 2012-07-27T12:33 by aleph

Hi. As it seems FreeBSD 9.0 should install on WD10EARS drive correct ? I recently installed and face following problem: extracting ports/src from tar, making ports take huge amount of time, not to mention system often became unresponsible showing load about 0. At the same time copy/extract huge files, or dd from /dev/random to /file work very well, at very good speed. Same procedures, like extract ports from tar, with regular drive, and on the same computer work flawlessly and very fast.

#17 Re: FreeBSD on 4K sector drives

Added on 2013-12-18T10:37 by Dean

Is it possible to change a drive that has been partitioned with 512 sector size to 4096 wiothout loosing the data?

On ZFS.

#18 Re: FreeBSD on 4K sector drives

Added on 2013-12-18T10:59 by Ivan Voras

No, there's no option to change the expected sector size except for backup + reformat + restore.

#19 Re: FreeBSD on 4K sector drives

Added on 2014-04-13T22:40 by martin

This article is a bit old (but still valid), google did show me this when I was dealing with my problem. This is something I didn't know when I was creating zvols on zpool and I had to resync all the data due to the utilization. Here's the thread at FreeBSD's forums:

http://forums.freebsd.org/viewtopic.php?f=48&t=45814

Basically you have to pay attention what blocksize you use when zvol is created on zpool with 4K disks.

Post your comment here!

Your name:
Comment title:
Text:
Type "xxx" here:

Comments are subject to moderation and will be deleted if deemed inappropriate. All content is © Ivan Voras. Comments are owned by their authors... who agree to basically surrender all rights by publishing them here :)