Format a USB flash drive with UDF, and preserve Linux file ownership, permissions, and attributes

Bill Statler

Copyright © 2018 — Creative Commons Attribution-NonCommercial 4.0 International License

Scenario:

  • You're a Linux user, trying to prepare for disaster. So you want to copy all your files to a USB flash drive or other portable external drive.
  • After the disaster, you'll want to rebuild a Linux system. Which means you'll want all those files to have their original owner, group, permissions, and attributes. Which means you can't use FAT32/VFAT (the format of all commercial flash drives), because it's unable to store that data with the files.
  • But you also want your flash drive to be readable, and preferably writable, on Windows and Macintosh systems, since in your post-disaster world, you may not have access to a Linux system for a while. Which means you can't (easily) use a Linux format like EXT4, because this is unreadable on Windows/Mac unless you install additional software.

UDF (Universal Disk Format) provides a mostly-satisfactory solution. A UDF-formatted flash drive will handle all of the above. But I'd better warn you right now about the not-so-satisfactory bits:

  • To format the drive, you'll need to install some software on at least one Linux system. (Formatting from a Mac might also work, but I haven't tested it.) If your Linux version is fairly old, you may need to get some of the software from a newer repository.
  • To use the drive on a Linux system, you'll need to edit /etc/fstab so that it gets mounted correctly.
  • Writing to the drive from Linux is fast, but writing from a Mac is unbelievably slow. (I haven't tested writing speed from Windows yet.)

I think we're at Linux Nerd Level 7 here. If that doesn't scare you away, read on...

Installing the formatting software:

One does not simply format the drive in UDF. The problem is that this "universal" format hasn't been implemented in a truly universal way across Linux, Windows, and Mac. Luckily, some Level 10 nerds have already figured this out for us, and written a bash script to handle all the messy stuff:

https://github.com/JElchison/format-udf

Download this, copy the file format-udf.sh to somewhere convenient (I used /usr/local/sbin/ but you can run it from anywhere), and make it executable.

Also, please check the Environment and Prerequisites sections of the README. You will probably need to install three packages using your favorite package manager: udftools, coreutils, and vim-common.

Here's a problem: Older versions of udftools will not work. The drive will appear to format but won't be usable. And I don't know the minimum version required. By experimentation I found:

  • udftools 1.0.0b3-14.3 does not work.
  • udftools 2.0-2 does work.

If your default repository only has an old version, you may need to grab a version from a newer repository. In many Linux variants you can use apt-pinning to do this. Please don't try apt-pinning unless you've read up on how it works, because you can make a horrible mess of your system. I'm not going to write an apt-pinning tutorial here, but I'll give you the files I created. These are specifically for LMDE2 (Linux Mint Debian Edition version 2), and will likely need modification if you're using any other Linux.

/etc/apt/sources.list.d/testing.list

# Debian 'testing' repositories:

deb http://http.us.debian.org/debian/ testing main
deb-src http://http.us.debian.org/debian/ testing main

deb http://security.debian.org/debian-security testing/updates main
deb-src http://security.debian.org/debian-security testing/updates main

deb http://http.us.debian.org/debian/ testing-updates main
deb-src http://http.us.debian.org/debian/ testing-updates main

/etc/apt/preferences.d/testing.pref

Explanation: Debian 'testing' packages default to lower priority than currently-installed version (100)
Package: *
Pin: release a=testing
Pin-Priority: 50

Explanation: Get more-recent udftools from Debian 'testing'
Package: udftools
Pin: release a=testing
Pin-Priority: 800

Formatting the drive:

The first step is to find out where in the /dev directory your USB drive is. Use the mount command to show you what's currently mounted. Then insert the drive, wait a moment, run mount again, and see what's changed. Linux will (typically) automount the flash drive, and you'll see something like this:

/dev/sdb1 on /media/bill/7A53-16C6 type vfat
  (rw,nosuid,nodev,relatime,uid=1000,gid=1002,fmask=0022,dmask=0077,codepage=437,
  iocharset=utf8,shortname=mixed,showexec,utf8,flush,errors=remount-ro,uhelper=udisks2)

If your drive doesn't automount, try lsblk to get a list of block devices, or take a look at your /var/log/syslog for a clue. The only information you need is the block device of the drive (sdb in my example, but it may be different on your system).

Make sure you've copied anything you want to save from the flash drive! Formatting will delete everything.

Un-mount the drive. (The format-udf.sh script is supposed to handle un-mounting but it hasn't worked for me.) Use umount, or use the Eject or Safely remove drive functions from your desktop.

Ready? The formatting command looks something like this:

sudo format-udf.sh sdb "UDFdrive1"

Replace sdb with the block device of your drive. "UDFdrive1" is your drive label: call it anything you want, but do make sure to use some label, because you'll need it later. (UDF does not support UUID, so we'll use the drive label to distinguish your drive in /etc/fstab.)

After running some tests, the script will report what it found:

/dev/sdb: PTTYPE="dos"
Cruzer Glide 3.0
RO    RA   SSZ   BSZ   StartSec            Size   Device
rw   256   512  4096          0    249644974080   /dev/sdb
rw   256   512   512         64    249644941312   /dev/sdb1
The above-listed device (and partitions, if any) will be completely erased.
Type 'yes' if this is what you intend:

Take a really close look at this, and make certain you're about to format the correct drive, before you type yes. In a few seconds, you should see:

[+] Successfully formatted /dev/sdb: LABEL="UDFdrive1" TYPE="udf" PTTYPE="dos"
Please disconnect/reconnect your drive now.

Remove the drive.

Editing /etc/fstab:

If you plug the drive in at this point, it will most likely automount and be usable. But, you probably won't get any of the benefits which were the whole point of UDF formatting. The details depend on how your system automounts USB drives, but typically you'll see something like this if you run mount:

/dev/sdb1 on /media/bill/UDFdrive1 type udf
  (rw,nosuid,nodev,relatime,uid=1000,gid=1000,
  umask=77,iocharset=utf8,uhelper=udisks2)

Ownership is assigned to the current user, the permission bits for group and other are masked out, and basically you're emulating the FAT32/VFAT drive that you had when you started.

To avoid this, add a line to /etc/fstab to override the default mount options. This is where we use the drive label, so that these new mount options only apply to your UDF-formatted flash drive. Here's my version; modify this to your liking:

LABEL=UDFdrive1  /mnt/UDFdrive1  udf  noatime,user,noauto,iocharset=utf8,uhelper=udisks2  0  2

Remember to create a mount point that matches what you've set in /etc/fstab:

sudo mkdir /mnt/UDFdrive1

Mounting and using the drive:

Insert your UDF-formatted flash drive. It should not automount. The user mount parameter in /etc/fstab lets you manually mount this drive as a regular non-root user (without sudo):

mount /mnt/UDFdrive1

However, the drive is owned by root. And you will want to use sudo (or be root) when you back up your files to the drive. Also, remember to use the -a (archive) option (or similar) with cp, rsync, or whatever tool you use for file copying, in order to preserve ownerships, permissions, and attributes. So, for example, to back up your entire /home directory:

sudo cp -a /home /mnt/UDFdrive1

I like to use rsync for backups, since it avoids copying data that's already present on the destination drive:

sudo rsync -avv --delete-before --delete-excluded --exclude-from=MyExcludeFile /home /mnt/UDFdrive1

(Omit the sudo if you do want to ignore current ownership; all files copied to the destination will then be owned by you.)

When you're done, remember to un-mount the drive before removing it:

umount /mnt/UDFdrive1

Your system may prompt you for your password to authorize un-mounting. If you recently copied a lot of files to the drive, expect umount to hang for 1-2 minutes while writing completes. Don't unplug the drive until it's finished.

Using a UDF-formatted flash drive on Windows/Mac:

Just plug it in.  It should automount.

However, don't expect your Linux file ownerships, permissions, and attributes to be preserved (or even meaningful) on a non-Linux system. Also, as I noted earlier, Macintosh (at least) is excruciatingly slow when writing to a UDF-formatted drive. It works, it just takes ten times as long as it should. I wouldn't want to copy more than a gigabyte from a Mac to a UDF drive.

So, is this worth the effort?

Maybe.

In our household, we have a specific use scenario. Both my wife and I use Linux on our main desktop computers. If one computer dies, we could share a computer for a while. We've already set up both our machines with two users, just in case. So a UDF backup flash drive would give us a quick recovery.

But we're also planning for other disasters, like our house burning down, or a sudden need to visit a sick relative. In this situation, we'd be able to carry all our files with us. And if we only have access to non-Linux computers, we'll still be able to retrieve a copy of our household inventory for the insurance company, or whatever.

I hope these instructions make the process a bit easier for you.