Warning! Technical stuff ahead! We use a lot of external drives for media work and storage. We’re constantly handing projects off to different people to work on different parts and that means these projects need to be mobile. It also means that they need to work on both PC’s and Mac’s. To do this, we reformat all our drives to exFAT which allows us to read and write to the drives using either a PC or a Mac. Very handy.
Now, the question I’ve always had was: 1) What allocation unit size should I pick for exFAT? and 2) Am I loosing read/write speed when I pick exFAT over NTFS?
For question #1, doing a Google search gives you some answers, but also leaves you with a lot of head scratching. First, a lot of people confuse the default NTFS file size of 4096 bytes with the exFAT option of 4096 kilobytes. These aren’t the same!!! 1 kilobyte = 1000 bytes (or 1024 bytes depending on context). So when I reformat to exFAT, I typically pick 256 kilobytes for the Allocation unit size. It’s the default the computer picks for me and I’ve read several places where it was at least alluded to being a good size. There is no right size, there are tradeoffs, but for us, it works just fine.
For questions #2, I’ve always been concerned that I may be limiting my speed when I pick the exFAT format vs. the NTFS. There are read/write tests on the web that break all this down, the problem is that it doesn’t really answer my question for me, it basically says it all depends on the type of files being used (not to mention the people who contradict each other). So, when in doubt, test it out. I grabbed a 11.2 GB folder full of various video files and project files and copied it to the drive when it was formatted as NTFS and exFAT. The NTFS took 130 seconds to move them over while the exFAT was only 107 seconds. I then exported an edited video file using both formats. The exFAT took 118 seconds to export the project, while the NTFS took…. 118 seconds. I’m doing this off a stop watch, manually clicking buttons, but from what I can tell, there wasn’t a difference between the two on typical projects we work on.
That answered it for me. exFAT gives us the option to work on either a PC or a Mac and it may be a bit faster for us and the type of work we do. As for the Allocation Unit Size, I’m comfortable with the 256 kilobytes size. Hope this helps.
Cheers,
Adam Kilbourn 🙂
Thanks for your article. Even though not scientific enough, but practically help a lot 🙂
Once again, thank you
Have you tried saving your files to cloud services? A lot of people use services like Dropbox, Google Cloud, Basecamp, etc and there are numerous companies that offer paid storage into the TB. A lot of my indie production designer and graphics clients like working with google cloud on projects. Multiple people can work on the files, and you won’t be encumbered carrying around fragile hard drives that can be physically lost or damaged. Plus it’s faster, you just send links when work is updated, etc.
We use Sugarsync to backup our important files, but for video files, it takes waaaaayyyyy too long to upload 250GB’s of footage for a single project. I like cloud storage, but I haven’t seen it successfully used yet for video production yet.
After using it for over a decade I moved away from SugarSync and use MEGA.nz ever since. Speed at SUgarSync was becoming terrible. Else it was a great tool. MEGA is also best in terms of encription. Speed and pricing is good.
Thanks alot. It was very helpful.
OSX Yosemite has cluster limitation/bug whatever that won’t recognize usb drives if anything over 1024 bytes is used when formatted for some weird reason. any solution?
Weird. I found this board discussing the same issue, but they have no idea why it is the case, https://discussions.apple.com/thread/2767345. If you find a solution, let me know. – Adam
Thanks for answering my question about unit size with exFAT. Regarding NTFS vs FAT speeds, years ago I had read that FAT was faster and I often format a partition for the swap/page file as FAT for the slight performance increase. I have since seen contradicting info that NTFS is faster. In my thinking, FAT should be faster. The drive hardware is going to perform the reading and writing of the data, but the NTFS file system has the added overhead of journaling any transaction that alters data. It makes sense that reading the files in both format is relatively the same speed, but that NTFS is slightly slower when writing due to this overhead. However, my knowledge of the inner-workings of both file systems is limited, and my thinking could be incorrect.
useful,thanks a lot
Thanks for your article.
I have 1TB USB drive and I prefer 64K unit allocation size.
Example for Admin Command Prompt:
format X: /FS:exFAT /A:64K /V:MYDRIVE /X /Q
It gets less space waste (I have a lot of big and small files) and only a little slower performance (relating to default 256K allocation unit).
it was the most proper answer i ever could find 🙂
Thank you very much for sharing this info, that´s exactly the answer that I was looking for!
I needed to format my 1TB external drive so I could be used in both my Macs and PCs, and after finding out about the exFAT compatibility, there came a lot of confusion regarding the bytes allocation size. You were clear and straight forward.
Thank you, straight to the point!
Thank you, only decent answer found on the net.
Yup, that’s helped me a lot, thanks.
I found this on internet. Credit due to original poster.
AUS – Allocation Unit size – It is the smallest data block on the disk. Your actual data will be separated to those units while saving to the disk. For example, if you have a file sized 512KB and you have 128KB allocation unit size, your file will be saved in 4 units in the disk (512KB/128KB).
If your file’s size is 500KB and you have 128KB AUS, your file still be saved in 4 units in the disk because as mentioned above 128KB is the smallest size of an allocation unit. 384KB will be allocated in 3 units and remaining 116KB will be allocated in another unit. You can observe this behaviour on file properties screen on Windows, what is your file size and how much space this file actually covers on the disk. And the operating system reads only that AUSd much data at a low level disk read operation.
Those being said, using large AUS significantly reduces the free space utilization due to not using the last allocation unit completely. And as a side effect, the number of files to store on the disk is reduced due to same problem, last AU not being used fully. But, here’s the trade-off, using large AUS, significantly again, improves the disk reading performance. The O.S can read more data at one read. Imagine, O.S makes couple of disk reads to completely read a GB sized file!..
Using small AUS improves the free space utilization but reduces the disk read performance. Think using large AUS in reverse, same category problems and improvements, but in reverse…
So, what is the conclusion here? If you will store large, I mean “large!”, files on the disk, higher AUS will give you appreciable read performance while reducing the file count and free space…
Which AUS you should use? This depends on how much your average file size is. Also you can compute the free space utilization according to your file sizes.
I am a software engineer, and have implemented the embedded code for multiple file systems including FAT32.
The above is true, however this are a few more complications to the equation. As stated, using a larger AUS causes small files to use large blocks which may not efficiently allow for a larger number of smaller files (that is a tonge twister). Using small AUS does fix this however it causes the need for larger allocation tables used to track each available AUS. So using very small AUS. The best selection for AUS is a balance of the typical file sizes you work with.
Another part of the equation is that most file systems have caching mechanisms which load one or more AUS (based on sectors or clusters if you want to google how it is done). Having larger AUS will likely allow the allocation of larger caches in RAM so data is transferred from the disk into the caches improving data throughput.
One fine interesting part about allocation units, especially with respect to solid state drives. SSD’s are implemented using FLASH devices. Typical flash devices are PAGE based meaning they are erased and written a page at a time. You might ask: “what is a page?” A page depends on the actual type of flash chip used, but the page size ranges anywhere between 128 bytes (for small flash devices) to 65K bytes or larger for the biggest of flash devices. Optimizing your allocation unit size to be the same as your flash page size will provide the absolute fastest performance when saving (writing files). This is because the erase operation is very slow in flash and a page erase must occur before the write. if your allocation unit does not match your page then more erase operations would be performed to write to the flash device. Finding the page size is tricky because you need to know the details of the SSD chips being used. I wouldn’t expect this to be available except to people who design SSD’s or who work with custom SSD storage devices, but I thought I would provide the information for completeness. It might be possible to find the page size by setting up different allocation unit sizes and performing various write throughput tests. — Paul
That was honestly one of the cleanest explanations of how to measure your needs.
Not just say, “I’m 110% too concerned about speed or capacity, when; generally not a significant change.
If swapping between Different OS, I would add the bit that FAT32 was based loosely off four aligned 1kb sectors, which now is one NTFS unit (4096 bytes) by default.
Simply to keep math easy & keep things aligned, I now see why many like 16Kb
1024 x 4 = 4096 (FAT32 to NTFS) & 4096 x 4 = 16Kb (NTFS default)
I know little of OSX, but feel unless you have gigantic files, I’ll stay with default or in rare cases alter exFAT only to continue alignment of sectors between file systems. Thanks!
to use on a windows 10 machine, are you formatting exFAT on a Mac or PC?
On a PC.
Thank you for posting this information. I’m sure it has helped a number of folks out there as well as it did for me..
By the way, exFAT is compatible with Android devices as well, so if you have an Android smartphone and need to quickly look at a file rather than pulling out he laptop, exFAT is the format to go with for PC, MAC and Android.
Thank you! You took the time to take the time of the operations. That was really noble of you. 🙂