My Virtual Private Server tutorial was fairly extensive, but it left out something important: information security!
I think it’s wise to revisit the topic now, since security and encryption are critical in any application of computer systems.
With the popularity of cloud computing and storage, it’s easy to take security for granted. Easy to think “Let someone else worry about it for me! That’s why I pay them.”
I disagree with that premise, for simple (and pessimistic) reasons:
- Nobody cares about your data except you
- Well, except for maybe the government
- Oh, and advertisers who want to sell it
- And hackers who want to steal it
Access Control vs. Encryption
Here’s the reality — a typical VPS cloud server is completely unencrypted. If it’s running Linux, you can be assured that the access control is robust (unless you use awful passwords).
Users expect reliable uptime, plus automatic backups and snapshotting, easy restores, and seamless reboots.
But to get these things, you must necessarily trade some security.
Here’s something you might not know… If you have a desktop PC that is not encrypted at the storage level, I can simply take the storage device out of your machine, plug it into mine, then copy your files off.
I can take files off of any sort of device if you give me enough time. I’ve done it to help friends before! Linux, MacOS, Windows, no problem. Flash cards from cameras and phones as well. Data is data, and if it’s not protected at the storage level, you don’t really control it.
If I do this for my friends and family, it’s simply being helpful. If we’re not friends, this is called an offline attack. It’s named as such because the system is taken offline and can be attacked at my discretion without you being able to intervene.
The risk with a VPS is that despite your server being protected with SSH access control and account passwords, the data is still totally unencrypted and sitting in some data center somewhere.
If someone wanted to harm you or take your data, they could perform an offline attack on your VPS and exfiltrate the data for analysis.
If you care about the security of your sensitive files, and especially if you’re using a VPS, you need to learn how to create and use encrypted storage.
A New Hope
Luckily for us, Linux is built by Chads for Chads. And Chad cares about his data.
Enter dm-crypt the block device encryption module built directly into the Linux kernel. Combine that module with the cryptsetup package and you have the power to encrypt files and whole storage devices as needed.
I host this website on a Linode VPS. It does not contain any particular sensitive data, but fuck ’em! Nobody gets to look through my data except me.
So here’s what I’m going to demonstrate: I will create a new block storage device, attach it to the VPS, encrypt it completely using cryptsetup, then move my Docker data over to that new location.
dm-crypt is a kernel module, and it operates by providing a virtual device that behaves like a real storage device. In one of my very first posts about the Linux file system structure, I wrote about the /dev directory —
/dev is a bit of an odd-ball. In Linux, devices are expressed as a special file on the filesystem. /dev contains all of these special device files, and access to this folder is restricted to Super User only.
We haven’t had to use /dev in any direct way, so I haven’t brought it up again. But now we do! Here’s the high level overview:
- The dm-crypt module creates a special directory called
/dev/mapper, populated by special virtual devices that are linked to physical devices. - The physical device is formatted with a special encryption key.
- The virtual device looks like physical storage to the system, but the kernel knows that it is a virtual device that must interact with the dm-crypt module.
- All data written to the virtual device will pass first through dm-crypt, where it is encrypted with your encryption key, then physically recorded to the actual device.
- All data read from the physical device will similarly pass through dm-crypt, where it is decrypted, then made available on the virtual device.
Since all data written to the physical storage device is encrypted at rest (an important term), it cannot be easily taken by simply copying it from storage. If you power the system down, you have no more kernel, and thus no more dm-crypt, so no more transparent decryption.
Storage encrypion will not protect you if your system has been compromised at the access control level (password compromise, remote vulnerability, or a rootkit). If they already have access to your running machine, you’re screwed. I will write about ways to secure your machine against these attacks soon (TO-DO REMINDER).
Attaching New Storage
In Linode, attaching new storage to a VPS is quite easy. Simply go to Volumes in the menu and choose Create Volume. From there, simply choose a volume size, give it a label, and choose the VPS you want to attach it to. After it’s created, it will appear in the Storage tab on that VPS.
From a shell session, verify that the volume has been “attached” by using dmesg. Mine displayed the following message:
[696931.055662] scsi 8:0:1:0: Direct-Access Linode Volume 2.5+ PQ: 0 ANSI: 5
[696931.056131] sd 8:0:1:0: Power-on or device reset occurred
[696931.056554] sd 8:0:1:0: [sde] 20971520 512-byte logical blocks: (10.7 GB/10.0 GiB)
[696931.056583] sd 8:0:1:0: [sde] Write Protect is off
[696931.056584] sd 8:0:1:0: [sde] Mode Sense: 63 00 00 08
[696931.056632] sd 8:0:1:0: [sde] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[696931.059194] sd 8:0:1:0: [sde] Attached SCSI disk
This tells me that the new volume has been attached, and its device label is sde. I can access it using the special character file /dev/sde. To display its size and other characteristics, I use fdisk:
linodevm:~# fdisk -l /dev/sde
Disk /dev/sde: 10 GB, 10737418240 bytes, 20971520 sectors
10240 cylinders, 64 heads, 32 sectors/track
Units: sectors of 1 * 512 = 512 bytes
Disk /dev/sde doesn't contain a valid partition table
Cool, looks good! You don’t need to know what much of that means, so just look at the storage capacity and the fact that it does not contain a partition table.
Encrypting the Device
I use Alpine Linux on my Linode, but you may be more familiar with Debian or Ubuntu variants. The commands are the same, but the package management tool is different. I installed the package using apk add cryptsetup, but you should use apt install cryptsetup. After that, the cryptsetup and mkfs commands are the same.
Now we can perform a low-level format using cryptsetup:
linodevm:~# cryptsetup luksFormat /dev/sde
WARNING!
========
This will overwrite data on /dev/sde irrevocably.
Are you sure? (Type 'yes' in capital letters): YES
Enter passphrase for /dev/sde: *****
Verify passphrase: *****
Now that the device is encrypted, we create the map to the virtual device:
linodevm:~# cryptsetup luksOpen /dev/sde encrypted_volume_1
Enter passphrase for /dev/sde: *****
Now we can verify that the virtual device exists by using fdisk again:
linodevm:~# fdisk -l /dev/mapper/encrypted_volume_1
Disk /dev/mapper/encrypted_volume_1: 10 GB, 10720641024 bytes, 20938752 sectors
1303 cylinders, 255 heads, 63 sectors/track
Units: sectors of 1 * 512 = 512 bytes
Disk /dev/mapper/encrypted_volume_1 doesn't contain a valid partition table
Looks good! It still does not have a partition table though, but that’s an easy last step using mkfs.ext4, which writes a new EXT4 partition to the device:
linodevm:~# mkfs.ext4 /dev/mapper/encrypted_volume_1
mke2fs 1.46.2 (28-Feb-2021)
Creating filesystem with 2617344 4k blocks and 655360 inodes
Filesystem UUID: 92a4ce89-b354-4845-b481-c6feacdbbcf7
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632
Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done
Now I can now mount this device to my filesystem and use it like it was physical storage:
linodevm:~# mkdir /mnt/encrypted/
linodevm:~# mount /dev/mapper/encrypted_volume_1 /mnt/encrypted/
linodevm:~# ls /mnt/encrypted/
lost+found
Note: EXT4 partitions always create a lost+found directory which it uses to store any file fragments or lost data that it finds during disk checks.
Now you have a completely encrypted storage device (/dev/sde) which is mapped to a virtual device (/dev/mapper/encrypted_volume_1) through the kernel dm-crypt module. It is manually opened using cryptsetup with the luksOpen option.
This setup will not automatically unlock your virtual device at boot time, which is a measure of additional security. You must unlock it yourself! This is excellent for servers, since they rarely reboot or power off.
Add the following line to /etc/fstab to ensure the proper options are applied on later mounts:
/dev/mapper/encrypted_volume_1 /mnt/encrypted ext4 defaults 0 0
Moving Docker Storage to Encrypted Volume
From here it’s a simple task of stopping Docker, changing a few options to point to the new encrypted location, then moving the data.
Here is the process if you’re using Alpine:
linodevm:~# /etc/init.d/docker stop
[or systemctl stop docker if you're using Debian/Ubuntu]
linodevm:~# mv /var/lib/docker/ /mnt/encrypted
lindoevm:~# nano /etc/conf.d/docker
[Add the following text]
DOCKER_OPTS="--data-root /mnt/encrypted/docker"
linodevm:~# /etc/init.d/docker start
And here is the process if you’re using Debian or Ubuntu:
linodevm:~# systemctl stop docker
linodevm:~# mv /var/lib/docker/ /mnt/encrypted
lindoevm:~# nano /etc/docker/daemon.json
[Add the following text]
{
"data-root": "/mnt/encrypted/docker"
}
linodevm:~# systemctl start docker
Wrapping Up
Now I have a totally encrypted block storage volume on a VPS that I cannot physically control. The extra steps at the end are only required because Docker expects its data directories to be located in a particular place.
I could have mounted the encrypted device directly into /var/lib/docker instead, but that may have confused the issue. Linux allows you to mount a storage device over an existing directory, effectively overlaying a complete file structure over existing files. To avoid weird behavior, I would move the old directory to a temporary location, mount the new location, then move the files back.
Final Turbo Autist Note: If you are looking to encrypt and secure data on a server in this way, be sure to create the encrypted volume BEFORE you create any data on it. All of the data stored before I created this volume should be considered compromised. Since this server only hosts this website and already-public IPFS, that’s OK.
If you want to maximize security, encrypt it from the start.