From a9fe75b40e7ea8f8fac0ad80cf005c7f6d6f48a1 Mon Sep 17 00:00:00 2001 From: Bill Niblock Date: Tue, 27 Sep 2022 18:14:18 -0400 Subject: [PATCH] New Post: Funkwhale on Linode with Object Storage --- _posts/2022-09-27-linode_funkwhale.md | 140 ++++++++++++++++++++++++++ 1 file changed, 140 insertions(+) create mode 100644 _posts/2022-09-27-linode_funkwhale.md diff --git a/_posts/2022-09-27-linode_funkwhale.md b/_posts/2022-09-27-linode_funkwhale.md new file mode 100644 index 0000000..4dfa6e2 --- /dev/null +++ b/_posts/2022-09-27-linode_funkwhale.md @@ -0,0 +1,140 @@ +--- +tabtitle: "Funkwhale On Linode with Object Storage" +title: "Funkwhale on Linode with Object Storage" +topics: [technology] +pub: "2022-09-27" +short_desc: "Funkwhale is a cool project, building a federated music platform. I +wanted to explore Funkwhale's ability to store music and associated files in +object storage, and since I already use Linode for my VPS, leveraging their +object storage offering makes a lot of sense." +--- + +# Funkwhale on Linode with Object Storage + +## Funkwhale Setup + +[Funkwhale](https://funkwhale.audio/) is a decentralized music service, +connecting to the [fediverse](https://en.wikipedia.org/wiki/Fediverse) using the +ActivityPub protocol. It is a web-based application, allowing users to upload, +listen, and share music and podcasts. I think it's a cool project, and I can +self-host it, so I did. For a while, Funkwhale offered an all-in-one Docker +container, but they shifted focus to a multi-container approach. I had delayed +my transition from all-in-one to multi-container, but finally this past weekend +I found myself with time and motivation to get it done. The installation of +Funkwhale using Docker is very straight forward. The community has developed a +series of templates that can be fetched, modified, and used to get started very +quickly and easily. Those instructions are +[here](https://docs.funkwhale.audio/installation/docker.html#multi-container-installation). +The only significant modification I made was using `/opt/funkwhale` as my +default data and media root. I keep all my Docker configuration in directories +in my home directory as well. Much of these changes can be established in the +`.env` file discussed in the installation instructions, but I also scrubbed the +template files created and used during installation to make sure the directories +were as I wanted them. I also proxy Funkwhale and many other services behind +nginx, and there were a few [additional +steps](https://docs.funkwhale.audio/installation/index.html#nginx) I had to +take. With all that complete, I had transitioned successfully. I already had SSL +certificates, but if that's also a requirement, they can easily be provisioned +using [Certbot](https://certbot.eff.org/). + +## Object Storage Setup + +In my old setup, I leveraged Funkwhale's ability to [in-place import +music](https://docs.funkwhale.audio/admin/importing-music.html). I transferred +about 70GB worth of music to my VPS (using Syncthing <3), filling the disk +almost entirely (_98%_). It was a temporary solution for a road trip, and I +knew I couldn't keep it that way for long. Funkwhale has the ability to leverage +S3-compatible object storage, and Linode, the provider I already use for my VPS, +offers object storage. Any of the other major cloud providers will also do the +trick; I just went with what was easiest. On the Linode side, there's not much +to it. I created a new bucket, labeled it accordingly, created an access key, +and that was it. The Funkwhale side proved to be a bit challenging, but not, it +turns out, due to configuration. Well, _technically_ it was. + +The relevant configuration options on the Funkwhale side, in the `.env` file: +``` +## External storages configuration +# Funkwhale can store uploaded files on Amazon S3 and S3-compatible storages (such as Minio) +# Uncomment and fill the variables below + +AWS_ACCESS_KEY_ID= +AWS_SECRET_ACCESS_KEY= +AWS_STORAGE_BUCKET_NAME= +# An optional bucket subdirectory were you want to store the files. This is especially useful +# if you plan to use share the bucket with other services +# AWS_LOCATION= + +# If you use a S3-compatible storage such as minio, set the following variable +# the full URL to the storage server. Example: +# AWS_S3_ENDPOINT_URL=https://minio.mydomain.com +AWS_S3_ENDPOINT_URL= + +# If you want to serve media directly from your S3 bucket rather than through a proxy, +# set this to false +# PROXY_MEDIA=false + +# If you are using Amazon S3 to serve media directly, you will need to specify your region +# name in order to access files. Example: +# AWS_S3_REGION_NAME=eu-west-2 +# AWS_S3_REGION_NAME= + +# If you are using Amazon S3, use this setting to configure how long generated URLs should stay +# valid. The default value is 3600 (60 minutes). The maximum accepted value is 604800 (7 days) + +# AWS_QUERYSTRING_EXPIRE= + +# If you are using an S3-compatible object storage provider, and need to provide a default +# ACL for object uploads that is different from the default applied by boto3, you may +# override it here. Example: +# AWS_DEFAULT_ACL=public-read +# Available options can be found here: https://docs.aws.amazon.com/AmazonS3/latest/userguide/acl-overview.html#canned-acl + +AWS_DEFAULT_ACL= +``` + +I've included the comments. The entire file is commented, and generally easy +enough to figure out. On the Linode side, when I generated the access key, it +provided me an _Access Key_ and a _Secret Key_. I had already created a bucket, +and so I had the _Bucket Name_. The challenge for me was what the _Endpoint URL_ +was, and if I needed to set a _Region Name_ and _ACL_. Linode's documentation on +their object storage offering is a bit anemic, and so I made use of their setup +instructions for [using s3cmd with Linode object +storage](https://www.linode.com/docs/products/storage/object-storage/guides/s3cmd/). +From this guide, I was able to both setup `s3cmd`, and also determine what the +_Endpoint URL_ would be. I also set the _Region Name_ and _ACL_ to match what +the UI was showing, but I'm still not certain their necessary. Here's the trick, +and the cause of a few hours-worth of confusion: restarting the Docker +containers wasn't re-reading the `.env` file; I had to completely stop and +re-create them. It wasn't until I ran `docker inspect funkwhale-docker_api_1` +and noticed the environment variables weren't set that I figured this out. Could +be this is common knowledge for Docker-gurus; now I know. With the +configuration in place, and the containers recreated, I was able to upload files +through Funkwhale, and watch them be stored in my Linode bucket. My final +configuration options were as follows: + +``` +AWS_ACCESS_KEY_ID= +AWS_SECRET_ACCESS_KEY= +AWS_STORAGE_BUCKET_NAME=funkwhale-music-bucket-name +AWS_S3_ENDPOINT_URL=https://us-southeast-1.linodeobjects.com +AWS_S3_REGION_NAME=us-southeast-1 +AWS_DEFAULT_ACL=public-read +``` + +## Next Steps + +The downside of Funkwhale's S3-compatible object storage integration lies in how +files are uploaded. One cannot simply ~~walk into~~ upload files to the bucket; +music must be uploaded through Funkwhale's API (via web or other means), and +then Funkwhale stores it accordingly (like for local uploads). I have a lot of +music, and I can't be asked to manually upload it all. That's, like, 2 hours of +half-hearted work. No, instead, the obvious solution is to build a script that +can automatically upload any new music from my local music directory to +Funkwhale automatically. What I'm considering now is how I want to do that. I +could leverage systemd to watch my local music directory, and run the upload +script whenever new music is uploaded. Could even expand it to remove music +whenever I delete it locally, though that seems a bit odd. I could instead setup +a cron or systemd-timer to run at a set interval, and check for any new files +since the last run, and upload them. Regardless of the trigger, the upload +functionality should ideally avoid duplicates, run in a non-blocking fashion, +maybe batch upload files, and be low impact on my desktop. So that's next.