Recently, I started looking at using Amazon S3 as a shared storage service instead of Dropbox, which I have been using for quite some time. I found S3 appealing because:
- It's cheap. A free Dropbox account gives you
2GBof space, but if you want, say,
3GBof space, you'll need to sign up for a
1TBaccount which costs $8.25 per month. By comparison,
3GBof space would cost about 7 cents per month on S3. If you're looking for a reasonable amount of storage, S3 is very affordable.
- The ecosystem. There is a massive number of libraries, utilities, and applications that support S3 natively.
- Encryption at rest. While I'm sure you can pull this off with Dropbox, because of point 2 there are multiple ways to achieve encryption at rest (storing files on disk in an encrypted format), but I'm going to focus on the best one I've found, S3QL.
What is S3QL?
S3QL is a userspace file system that allows you to mount an S3 bucket (and the equivalent from various competing services) as a regular UNIX file system. While S3QL has probably the best performance of any S3 file system, what really appealed to me is that it handles encryption automatically. The old standby EncFS, by comparison, requires you to mount a special encrypted file system on top of a regular one (i.e., your S3 bucket). Furthermore, EncFS has a (depending on your viewpoint) major security flaw in that the number of files, permissions, sizes, etc. are still visible on disk.
The one disadvantage of S3QL is that it keeps file metadata in a local database and syncs it periodically. Because of this, you can't mount an S3QL file system more than once. We can get around that limitation by re-exporting the S3QL mount directory as a Samba or NFS share. What's nice about this approach is that it eliminates another Dropbox disadvantage: the need for a special, closed-source client application.
After installing S3QL, you have to initialize your S3 bucket
Note that you should replace
bucket with the actual name of your bucket. Follow the prompts. You'll be asked for:
- Your backend login. This is your AWS access key ID.
- Your backend passphrase. This is your AWS secret access key.
- Your encryption password. This is the password you're going to use to encrypt the file system.
Next, you'll want to create an authentication info file so S3QL can reference these values when mounting or performing
other operations. I usually save this file as
[s3] storage-url: s3://bucket backend-login: aws_access_key_id backend-password: aws_secret_access_key fs-passphrase: encryption_password
Be sure to use the values from when you ran
mkfs.s3ql. Now, we can mount the bucket to a pre-existing directory
sudo mount.s3ql --authfile /etc/s3ql.authinfo --allow-other s3://bucket /mnt/bucket
--allow-other parameter: this is absolutely required if you want users other than
root to acccess the file
We can umount the file system with
sudo umount.s3ql /mnt/bucket
Automatic Mounting and Unmounting
The S3QL documentation makes it clear that you shouldn't
put an entry in
/etc/fstab in order to automatically mount and unmount the file system but that
systemd should be used instead. Unfortunately, the documentation
systemd configuration as an exercise for the reader. Here's what I've managed to come up with after
consulting a few sources:
[Unit] Description=Mount S3 filesystem Wants=network-online.target After=network-online.target [Service] ExecStart=/usr/bin/mount.s3ql --fg --authfile /etc/s3ql.authinfo --allow-other --cachedir /var/cache/s3ql s3://bucket /mnt/bucket ExecStop=/usr/bin/umount.s3ql /mnt/bucket TimeoutStopSec=600 [Install] WantedBy=multi-user.target
A couple important notes about this configuration:
Afteritems having the same target is not a typo. What I found is that, without the
Afteritem, the unmounting process would fail because the network would go offline in the middle of uploading metadata. Adding an
Afteritem made this a sequential process rather than a parallel one (i.e., the network can't go offline until the unmounting finishes).
TimeoutStopSecis set to something “reasonable”, in my case, 10 minutes. In practice this is very quick but I don't want to take any chances. If the metadata upload fails, you're likely to lose data.
As mentioned earlier, you'll want to create a Samba or NFS share from your mount directory so you can access the file system from multiple systems simultaneously. Once that's done, you'll have a fairly good Dropbox substitute that is free or very cheap.