2024-06-18 23:26:12 +00:00
<!DOCTYPE html>
< html >
< head >
< title > The Internet Vagabond :: Automatic Backups with RClone< / title >
< link type = "application/atom+xml" rel = "alternate" href = "https://www.theinternetvagabond.com/feed.xml" title = "The Internet Vagabond" / >
< meta name = "description"
content="Rants of a wandering techy, in search of truth, knowledge, and a decent ping." />
< meta name = "author" content = "Bill Niblock" / >
< meta name = "viewport" content = "width=device-width, initial-scale=1.0" / >
< link rel = "canonical" href = "https://www.theinternetvagabond.com/2023/05/07/rclone-backups.html" / >
< link rel = "stylesheet" type = "text/css"
href="https://www.theinternetvagabond.com/src/styles/corrupt_layout.css" />
< link rel = "stylesheet" type = "text/css"
href="https://www.theinternetvagabond.com/src/styles/corrupt_typog.css" />
< link rel = "icon" type = "image/x-icon"
href="https://www.theinternetvagabond.com/src/images/favicon.ico" />
< link rel = "stylesheet"
href="https://cdn.jsdelivr.net/npm/fork-awesome@1.2.0/css/fork-awesome.min.css"
integrity="sha256-XoaMnoYC5TH6/+ihMEnospgm0J1PM/nioxbOUdnM8HY="
crossorigin="anonymous">
< script data-goatcounter = "https://theinternetvagabond.goatcounter.com/count"
async src="https://www.theinternetvagabond.com/src/scripts/goatcounter.js">< / script >
< / head >
< body >
< div class = "cor_page" >
< header >
< a href = "/" >
< div >
< span class = "first" > T< / span > he
< span class = "first" > I< / span > nternet
< span class = "first" > V< / span > agabond
< / div >
< / a >
< / header >
< main >
< article >
< h1 id = "automatic-backups-with-rclone-systemd-and-backblaze" > Automatic Backups with RClone, systemd, and Backblaze< / h1 >
< h2 id = "quick-note" > Quick Note< / h2 >
< p > Backups are not complicated. They may seem like it, but in reality the
complications arise from restoration. If you’
your data now, then don’
methodology: 3 copies of (important) data, in 2 different locations, 1 of which
is off-site. Many others have written about this in better detail than I ever
can; Jeff Geerling has a great article and several videos about it < a href = "https://www.jeffgeerling.com/blog/2021/my-backup-plan" > on his
site< / a > . The time (and
often money) investment now can reduce worry, stress, and loss should the data
you care about ever become unusable.< / p >
< p > (There are no affiliate links in this post, nor was I paid to recommend any
product or service.)< / p >
< h1 id = "my-needs" > My Needs< / h1 >
< p > Backups are as important as the data you have. If all you’
full of meme GIFs that you don’
time and money. I have recently taken to buying as much of my music as possible
(especially through Bandcamp, and especially on Bandcamp Fridays!). While much
of the music I buy does exist on a remote server at a company somewhere, the
cost of having to re-download and re-organize all of it well outweighs the cost
of proper backups. Not to mention the music which I can’
anymore. Nor to further mention the other data which I have. All of this is to
say: backups are worth it to me.< / p >
< p > Recently I wanted to setup NFS on my home network. I was concerned about messing
something up, and erasing the directory I had intended to share, so I wanted to
backup the data. For a while I’
probably does), but it was never a priority. This project helped to prioritze
it. I had read about < a href = "https://rclone.org/" > RClone< / a > , a command-line utility for
interacting with an incredible number of cloud services. I messed around a bit
with it, found it to my liking, and started shopping around for a cloud storage
solution. Enter < a href = "https://www.backblaze.com/" > Backblaze< / a > . The folks that publish
all those hard-drive stats? Turns out they also run a business where they
provide cloud storage. It’
last step was to automate it with systemd timer units.< / p >
< h2 id = "backblaze-setup" > Backblaze Setup< / h2 >
< ul >
< li > < a href = "https://www.backblaze.com/" > Backblaze Site< / a > < / li >
< li > < a href = "https://www.backblaze.com/help.html" > Backblaze Docs< / a > < / li >
< / ul >
< p > First step is to setup Backblaze. Create an account, verify email address, all
that jazz. I’
< strong > Account< / strong > -> < strong > My Settings< / strong > page, under < strong > Security< / strong > . Next, click on the
< strong > Account< / strong > -> < strong > Application Keys< / strong > page, and generate a new key. Fill in the
blanks (I gave my key full access to all buckets), copy the important bits, and
store them somewhere safe (like your password vault).< / p >
< h2 id = "rclone-setup" > RClone Setup< / h2 >
< ul >
< li > < a href = "https://rclone.org/" > RClone Site< / a > < / li >
< li > < a href = "https://rclone.org/b2/" > RClone Backblaze B2 Page< / a > < / li >
< / ul >
< p > Download and install RClone. Next run < code class = "language-plaintext highlighter-rouge" > rclone config< / code > and walk through the
prompts. I’
Then I add the application key ID and application key secret (key) at the
relevant prompts. For all of this configuration, I named the remote “backblaze”,
though a shorter name can make commands easier. Regardless, verify the
configuration is setup properly by running < code class = "language-plaintext highlighter-rouge" > rclone lsd backblaze:< / code > , which will
list buckets. Unless a bucket was already configured, nothing will show up, and
also there won’ < / p >
< h2 id = "backup-configuration" > Backup Configuration< / h2 >
< p > Now, figure out how you want to backup your data. I have a < a href = "/2020/06/14/setting-up-btrfs.html" > BTRFS RAID setup
with multiple sub-volumes< / a > , each for a
different data type: one for Books, one for Music, and so on. Since creating a
bucket doesn’
the buckets I wanted, and did a “manual” RClone sync of the data.< / p >
< p > < code class = "language-plaintext highlighter-rouge" > rclone sync --fast-list --transfers 20 /path/to/Books
backblaze:bucket-for-Books-backups< / code > < / p >
< p > The “– – < a href = "https://rclone.org/b2/" > RClone
Backblaze B2 page< / a > , along with some others that may be
of interest.< / p >
< p > At this point, my data was “backed-up”, and I could muck about with it more
confidently. Also, at this point, configuring back-ups is done. Run those RClone
sync commands once a week, and all is set. I don’
things, though.< / p >
< h2 id = "automating-the-process" > Automating the Process< / h2 >
< p > The first thing to do is create a user-agnostic location for the configuration
file and some additional files. I chose < code class = "language-plaintext highlighter-rouge" > /etc/rclone< / code > , and copied the RClone
configuration file generated previously to this directory as < code class = "language-plaintext highlighter-rouge" > backblaze.conf< / code > .< / p >
< p > Next, I created a filter file. RClone has extensive < a href = "https://rclone.org/filtering/" > filtering
options< / a > . For my current needs, a single file
will suffice.< / p >
< h3 id = "defaultfilter" > default.filter< / h3 >
< div class = "language-plaintext highlighter-rouge" > < div class = "highlight" > < pre class = "highlight" > < code > # Exclude BTRFS snapshot directories
- .snapshots/**
# Exclude Syncthing configuration directories
- .stfolder/**
< / code > < / pre > < / div > < / div >
< p > systemd timer units ( [< a href = "https://wiki.archlinux.org/title/Systemd/Timers" > Arch
Wiki< / a > ]
[< a href = "https://man.archlinux.org/man/systemd.timer.5" > Manual< / a > ] ) are triggers that
activate on a schedule. That schedule can be dynamic (relative to a
previous/other trigger), or static (at 6:15 every day). A timer unit triggers a
service unit, which does the work. For my backups, I decided to run a sync every
hour, at sometime between the 15 and 45 minute mark of that hour. To simplify
having multiple timer units that all do the same thing, I setup a template unit
(see the < strong > Note< / strong > here: < a href = "https://wiki.archlinux.org/title/Systemd#Using_units" > Arch
Wiki< / a > ).< / p >
< h3 id = "rclone-backuptimer" > rclone-backup@.timer< / h3 >
< div class = "language-plaintext highlighter-rouge" > < div class = "highlight" > < pre class = "highlight" > < code > [Unit]
Description=RClone Backup Timer Template
[Timer]
# Run every hour, sometime between the 15 minute and 45 minute mark
OnCalendar=*-*-* *:15:00
AccuracySec=30min
RandomizedDelaySec=5min
# The %i is whatever value is after the "@" for the configured unit. For
# example, rclone-backup@Books.timer will run the rclone-backup@Books.service
Unit=rclone-backup@%i.service
[Install]
WantedBy=timers.target
< / code > < / pre > < / div > < / div >
< p > Then I can < code class = "language-plaintext highlighter-rouge" > enable< / code > and < code class = "language-plaintext highlighter-rouge" > start< / code > a timer for each directory to backup. To
minimize configuration, I also setup the service file to be a template. This
requires a bit of inflexible coordination: the directory name must match to a
part of the bucket name.< / p >
< h3 id = "rclone-backupservice" > rclone-backup@.service< / h3 >
< div class = "language-plaintext highlighter-rouge" > < div class = "highlight" > < pre class = "highlight" > < code > [Unit]
Description=RClone Backup of %I
[Service]
Type=simple
ExecStart=/usr/bin/rclone sync -v --config "/etc/rclone/backblaze.conf" --fast-list --transfers 20 --filter-from "/etc/rclone/default.filter" /path/to/%i/ backblaze:bucket-for-%i-backups
< / code > < / pre > < / div > < / div >
< p > The < code class = "language-plaintext highlighter-rouge" > --config< / code > option allows us to specify the configuration in the < code class = "language-plaintext highlighter-rouge" > /etc< / code >
directory. I include < code class = "language-plaintext highlighter-rouge" > -v< / code > to have some additional output in the journal.
Again, < code class = "language-plaintext highlighter-rouge" > --fast-list< / code > and < code class = "language-plaintext highlighter-rouge" > --transfers< / code > are used to speed up the process and keep
costs lower. Then I < code class = "language-plaintext highlighter-rouge" > --filter-from< / code > the “default.filter” file.< / p >
< p > Place each of these files (< code class = "language-plaintext highlighter-rouge" > rclone-backup@.timer< / code > and < code class = "language-plaintext highlighter-rouge" > rclone-backup@.service< / code > )
into < code class = "language-plaintext highlighter-rouge" > /etc/systemd/system< / code > . For each directory, enable and start the timer
unit; < code class = "language-plaintext highlighter-rouge" > systemctl enable rclone-backup@Example.timer< / code > and < code class = "language-plaintext highlighter-rouge" > systemctl start
rclone-backup@Example.timer< / code > will backup < code class = "language-plaintext highlighter-rouge" > /path/to/Example/< / code > to the
< code class = "language-plaintext highlighter-rouge" > bucket-for-Example-backups< / code > bucket.< / p >
< h1 id = "next-steps" > Next Steps< / h1 >
< p > I would like to get some sort of metrics and dashboards setup to track backup
status and statistics. It could be very useful to be notified if a backup ever
fails.< / p >
< p > Eventually, I’
backup. When I do, I’ < / p >
< div class = "author_info" >
Bill Niblock
< a href = "https://unlicense.org/"
aria-label="Code dedicated to the public domain under Unlicense">
< span class = "fa fa-cc-pd" aria-hidden = "true"
title="Code dedicated to the public domain under Unlicense"< / span >
< / a >
< a href = "https://creativecommons.org/publicdomain/zero/1.0/"
aria-label="Published to the public domain under CC0">
< span class = "fa fa-cc-zero" aria-hidden = "true"
title="Content dedicated to the public domain under CC0"< / span >
< / a >
2023-05-07
< br / >
[
< a href = "/topics/technology" > technology< / a >
]
< / div >
< / article >
< / main >
< footer >
< nav >
< div > < a href = "/" > home< / a > < / div >
< div > < a href = "/topics/all" > all< / a > < / div >
< div > < a href = "/topics/gaming" > gaming< / a > < / div >
< div > < a href = "/topics/other" > other< / a > < / div >
< div > < a href = "/topics/philosophy" > philosophy< / a > < / div >
< div > < a href = "/topics/technology" > technology< / a > < / div >
< div > < a href = "/topics/writing" > writing< / a > < / div >
< / nav >
< hr / >
< section class = "h-card" >
< section class = "footer_about" id = "about" >
< div > The Site< / div >
< div >
2024-06-20 19:38:41 +00:00
< a href = "https://www.theinternetvagabond.com/feed.xml"
2024-06-18 23:26:12 +00:00
aria-label="RSS feed for the site">
< span class = "fa fa-rss" aria-hidden = "true"
title="RSS Feed"< / span >
< / a > |
2024-06-20 19:38:41 +00:00
< a href = "https://theinternetvagabond.goatcounter.com/"
2024-06-18 23:26:12 +00:00
aria-label="GoatCounter statistics for the site">
< span class = "fa fa-bar-chart" aria-hidden = "true"
title="GoatCounter Statistics"< / span >
< / a > |
2024-06-20 19:38:41 +00:00
< a href = "https://codeberg.org/VagabondAzulien/the-internet-vagabond-dot-com"
2024-06-18 23:26:12 +00:00
aria-label="Source code repository for the site">
< span class = "fa fa-code" aria-hidden = "true"
title="Site Source Code"< / span >
< / a >
< / div >
< a class = "u-url u-uid" href = "https://theinternetvagabond.com" > < / a >
< p >
This site is a small slice of internet real-estate that I use for
occasional writing. Nothing I say is visionary or profound. I
focus on technology, gaming, and philosophy. All opinions my
own.
< / p >
< div > The Vagabond< / div >
< div >
2024-06-20 19:38:41 +00:00
< a rel = "me"
2024-06-18 23:26:12 +00:00
href="mailto:bill@theinternetvagabond.com"
aria-label="Email Bill at The Internet Vagabond dot com">
< span class = "fa fa-envelope-o" aria-hidden = "true"
title="Email bill at theinternetvagabond.com"< / span >
< / a > |
< a class = "u-url" rel = "me"
href="https://matrix.to/#/@vagabondazulien:matrix.org"
aria-label="Speak with me on Matrix">
< span class = "fa fa-matrix-org" aria-hidden = "true"
title="Speak with me on Matrix"< / span >
< / a > |
< a class = "u-url" rel = "me"
href="https://mastodon.social/@azulien"
aria-label="Find me on the Fediverse">
< span class = "fa fa-mastodon" aria-hidden = "true"
title="Find me on the Fediverse"< / span >
< / a > |
< a class = "u-url" rel = "me" href = "https://www.twitch.tv/vagabondazulien/profile"
aria-label="Link to my Twitch channel">
< span class = "fa fa-twitch " aria-hidden = "true"
title="My Twitch channel"< / span >
< / a >
< / div >
< p >
My name is < span class = "p-name" > Bill Niblock< / span > . < span
class="p-note">I'm a computer scientist by education, a technologist
by trade, a gamer by hobby, and a philosopher by accident. I
live in < span class = "p-locality" > Buffalo< / span > , < span class = "p-region" >
2024-06-20 19:38:41 +00:00
New York< / span > , < span class = "p-country-name" > USA< / span > .< br / >
< br / >
My PGP Key is < span class = "u-key" id = "key" > CCE7 3682 331B 5614 9FAB
7383 7359 80B2 6381 C91E< / span > .
2024-06-18 23:26:12 +00:00
< / p >
< / section >
< section style = "display: none;" >
< span class = "p-category" > Gaming< / span >
< span class = "p-category" > Technology< / span >
< span class = "p-category" > Philosophy< / span >
< span class = "p-category" > Open Source Software< / span >
< span class = "p-category" > Self-Hosting< / span >
< span class = "p-category" > Coffee< / span >
2024-06-20 19:38:41 +00:00
< span class = "u-email" > bill@theinternetvagabond.com< / span >
2024-06-18 23:26:12 +00:00
< / section >
< / section >
< / footer >
< / div >
< / body >
< / html >
