20 April 2022

20 April 2022, 14:00 (Published)

Installing a new SSD on Ubuntu

BTRFS, LUKS2 and LVM

Knock, knock, is this thing still on? Let's dust things off here!

What

I recently needed to add a secondary NVMe SSD drive to not one, nor two, but altogether three physical boxes running either Ubuntu 22.04 or 22.10.

You'd think there isn't much to it, but Linux & filesystems, eh?! Looking into it, I ended up with the same layering as the Ubuntu installer does, when setting up a blank new system SSD:

  1. SSD (d-uh!)
  2. Partition (filling the drive)
  3. LUKS2 encryption
  4. LVM group
  5. LVM logical volume
  6. BTRFS filesystem

The main difference here to a system SSD, of course, is that the secondary drive is to be used for storage only, so there is no need to make it bootable. Going by past experience, that would just complicate & confuse things, but take that as just my opinion.

What else could I have done, then, you might ask?

Most obviously, you could leave out LVM altogether, and either stick to one partition for the entire disk forever, or try to come up with whatever everlasting partitioning scheme you might need.

You could also take out partitioning entirely, and just have LUKS2 encryption enclose the filesystem. Looking at what I've previously done, this is not unreasonable, and works. For example, for the 18TB mirrored ZFS pool's spinning disks that will never have a need for any other kind of a setup, it's been great for many years already.

Finally, in theory, you could skip encryption as well, but my tinhat trembles at the thought. Each to their own.

In any case, this layering scheme with LVM would be the most flexible one for any future tinkering. So, let's get on with it!

How

While the steps are straightforward as such, it does get somewhat involved. Here we go!

Device identification

First up, you must identify for certain the right device to perform this whole operation on.

While good old fdisk is quite verbose, its output does include the physical SSD make and model for each device, so fdisk -l it is:

root@server ~ {} fdisk -l
… 
Disk /dev/nvme1n1: 1.82 TiB, 2000398934016 bytes, 3907029168 sectors
Disk model: WD_BLACK SN850X 2000GB                  
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
…

In this case, /dev/nvme1n1 is the one.

⚠️ The rest of the steps in this post will use /dev/nvme1n1. Be very clear on what your particular device is! ⚠️

Partitioning

Let's do the partitioning also using fdisk. Assuming we are operating on an empty SSD device, create a partition table, and a partition within, as follows:

  • fdisk /dev/nvme1n1
  • g for a GUID partition table, instead of the MBR/DOS default
  • n followed by three Enter key presses to allocate the entire disk for a new partition
  • t to change partition type
  • lvm followed by Enter
  • w to write out changes to the device.

This is what it looked like for the 2TB drive in this particular case:

root@server ~ {} fdisk /dev/nvme1n1

Welcome to fdisk (util-linux 2.37.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): g
Created a new GPT disklabel (GUID: F766EBCA-36B8-43FC-BD19-396C3F831E83).

Command (m for help): n
Partition number (1-128, default 1): 
First sector (2048-3907029134, default 2048): 
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-3907029134, default 3907029134): 

Created a new partition 1 of type 'Linux filesystem' and of size 1.8 TiB.

Command (m for help): t
Selected partition 1
Partition type or alias (type L to list all): lvm
Changed type of partition 'Linux filesystem' to 'Linux LVM'.

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

Let's double-check what was actually done:

  • fdisk /dev/nvme1n1
  • p to print out the partition table
root@server ~ {} fdisk /dev/nvme1n1

Welcome to fdisk (util-linux 2.37.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): p
Disk /dev/nvme1n1: 1.82 TiB, 2000398934016 bytes, 3907029168 sectors
Disk model: WD_BLACK SN850X 2000GB                  
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: F766EBCA-36B8-43FC-BD19-396C3F831E83

Device         Start        End    Sectors  Size Type
/dev/nvme1n1p1  2048 3907029134 3907027087  1.8T Linux LVM

LUKS2 encryption

Next up, let's encrypt the partition with LUKS2:

  1. Prepare a disk encryption passphrase in your password manager of choice
  1. Format the partition as a LUKS2 container (⚠️ not the device; there is a p1 at the end now! ⚠️):
cryptsetup luksFormat --type luks2 /dev/nvme1n1p1`
  1. When prompted, type YES in capital letters, and provide the encryption passphrase, twice.
root@server ~ {} cryptsetup luksFormat --type luks2 /dev/nvme1n1p1

WARNING!
========
This will overwrite data on /dev/nvme1n1p1 irrevocably.

Are you sure? (Type 'yes' in capital letters): YES
Enter passphrase for /dev/nvme1n1p1: 
Verify passphrase:

Boot-time unlocking of encryption

Let's prepare and apply a key file for boot-time unlocking of the encrypted partition. My choice here is to use the /etc/luks directory as the location.

💡 Is it safe, to have the key on disk in clear text, that could be compromised any time in the future? Well, borderline. I have opted to do so, leaning on the system SSD that encloses /etc, and the key file, being encrypted, and key file permissions set to 0400 for root-only readability.

mkdir /etc/luks
touch /etc/luks/keyfile-storage
chmod 0400 /etc/luks/keyfile-storage
dd if=/dev/urandom of=/etc/luks/keyfile-storage bs=4096 count=1
cryptsetup luksAddKey /dev/nvme1n1p1 /etc/luks/keyfile-storage

You will need to provide the encryption passphrase here (but never again after this):

💡 The original encryption passphrase you specifically do not want to leak, because in that event all data on the disk would need to be re-encrypted. The key file at least can be replaced later with the steps described here.

root@server ~ {} cryptsetup luksAddKey /dev/nvme1n1p1 /etc/luks/keyfile-storage
Enter any existing passphrase:

Optionally, for manual unlocking & locking, you also want to add, say, the administrator password as another passphrase:

root@server ~ {} cryptsetup luksAddKey /dev/nvme1n1p1                          
Enter any existing passphrase: 
Enter new passphrase for key slot: 
Verify passphrase:

Finally, open the LUKS2 container using the key file, to enable the next steps:

cryptsetup luksOpen --key-file /etc/luks/keyfile-storage /dev/nvme1n1p1 storage

Set up LVM

Let's create an LVM volume group, and a logical volume within. I have opted to make use of the word storage here.

The -l 100%FREE argument is for using all available space for the logical volume.

pvcreate /dev/mapper/storage
vgcreate storage-vg /dev/mapper/storage
lvcreate -l 100%FREE -n storage-lv storage-vg

Initialise BTRFS filesystem

This final step is now, phew, as simple as mkfs.btrfs /dev/storage-vg/storage-lv:

root@server ~ {} mkfs.btrfs /dev/mapper/storage                                                 
btrfs-progs v5.16.2
See http://btrfs.wiki.kernel.org for more information.

NOTE: several default settings have changed in version 5.15, please make sure
      this does not affect your deployments:
      - DUP for metadata (-m dup)
      - enabled no-holes (-O no-holes)
      - enabled free-space-tree (-R free-space-tree)

Label:              (null)
UUID:               b55d1f89-8ac9-411a-abc7-db1177043df1
Node size:          16384
Sector size:        4096
Filesystem size:    1.82TiB
Block group profiles:
  Data:             single            8.00MiB
  Metadata:         DUP               1.00GiB
  System:           DUP               8.00MiB
SSD detected:       yes
Zoned device:       no
Incompat features:  extref, skinny-metadata, no-holes
Runtime features:   free-space-tree
Checksum:           crc32c
Number of devices:  1
Devices:
   ID        SIZE  PATH
    1     1.82TiB  /dev/mapper/storage

The filesystem is alive!

Automount filesystem

Finally, let's make the filesystem automount at boot time.

  1. Find out the LUKS2 container partition's UUID with blkid /dev/nvme1n1p1:
root@server ~ {} blkid /dev/nvme1n1p1
/dev/nvme1n1p1: UUID="25906b75-0ad7-4d29-9429-b39917b814c9" TYPE="crypto_LUKS" PARTUUID="3e135d09-0c80-4903-a576-b47686743777"
  1. Update /etc/crypttab to contain the UUID (not PARTUUID):
storage UUID=25906b75-0ad7-4d29-9429-b39917b814c9 /etc/luks/keyfile-storage luks,loud
  1. Create mount point. I like a plain, simple /storage:
mkdir /storage
  1. Add to /etc/fstab:
/dev/storage-vg/storage-lv /storage btrfs defaults 0 2

Here, the 0 and 2 are for the "fifth field" and "sixth field" in man fstab. The former will always be zero, and the latter will be two for non-root filesystems.

All done! To use the filesystem right away, mount it:

mount /storage

Testing

You may still want to test that all the things work, without a reboot.

To unmount / tear down all the involved storage layers:

umount /storage
lvchange --activate n /dev/storage-vg/storage-lv
vgchange --activate n /dev/storage-vg
cryptsetup close storage

Then, build it back up again with:

cryptsetup luksOpen --key-file /etc/luks/keyfile-storage /dev/nvme1n1p1 storage
vgchange --activate y /dev/storage-vg
lvchange --activate y /dev/storage-vg/storage-lv
mount /storage