332 lines
16 KiB
HTML
332 lines
16 KiB
HTML
<!DOCTYPE html>
|
||
<html>
|
||
<head>
|
||
<title>The Internet Vagabond :: Automatic Backups with RClone</title>
|
||
<link type="application/atom+xml" rel="alternate" href="https://www.theinternetvagabond.com/feed.xml" title="The Internet Vagabond" />
|
||
<meta name="description"
|
||
content="Rants of a wandering techy, in search of truth, knowledge, and a decent ping." />
|
||
<meta name="author" content="Bill Niblock" />
|
||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||
<link rel="canonical" href="https://www.theinternetvagabond.com/2023/05/07/rclone-backups.html" />
|
||
<link rel="stylesheet" type="text/css"
|
||
href="https://www.theinternetvagabond.com/src/styles/corrupt_layout.css" />
|
||
<link rel="stylesheet" type="text/css"
|
||
href="https://www.theinternetvagabond.com/src/styles/corrupt_typog.css" />
|
||
<link rel="icon" type="image/x-icon"
|
||
href="https://www.theinternetvagabond.com/src/images/favicon.ico" />
|
||
<link rel="stylesheet"
|
||
href="https://cdn.jsdelivr.net/npm/fork-awesome@1.2.0/css/fork-awesome.min.css"
|
||
integrity="sha256-XoaMnoYC5TH6/+ihMEnospgm0J1PM/nioxbOUdnM8HY="
|
||
crossorigin="anonymous">
|
||
<script data-goatcounter="https://theinternetvagabond.goatcounter.com/count"
|
||
async src="https://www.theinternetvagabond.com/src/scripts/goatcounter.js"></script>
|
||
</head>
|
||
<body>
|
||
<div class="cor_page">
|
||
<header>
|
||
<a href="/">
|
||
<div>
|
||
<span class="first">T</span>he
|
||
<span class="first">I</span>nternet
|
||
<span class="first">V</span>agabond
|
||
</div>
|
||
</a>
|
||
</header>
|
||
<main>
|
||
<article>
|
||
<h1 id="automatic-backups-with-rclone-systemd-and-backblaze">Automatic Backups with RClone, systemd, and Backblaze</h1>
|
||
|
||
<h2 id="quick-note">Quick Note</h2>
|
||
|
||
<p>Backups are not complicated. They may seem like it, but in reality the
|
||
complications arise from restoration. If you’re not doing anything fancy with
|
||
your data now, then don’t do anything fancy with your backups. Follow the 3-2-1
|
||
methodology: 3 copies of (important) data, in 2 different locations, 1 of which
|
||
is off-site. Many others have written about this in better detail than I ever
|
||
can; Jeff Geerling has a great article and several videos about it <a href="https://www.jeffgeerling.com/blog/2021/my-backup-plan">on his
|
||
site</a>. The time (and
|
||
often money) investment now can reduce worry, stress, and loss should the data
|
||
you care about ever become unusable.</p>
|
||
|
||
<p>(There are no affiliate links in this post, nor was I paid to recommend any
|
||
product or service.)</p>
|
||
|
||
<h1 id="my-needs">My Needs</h1>
|
||
|
||
<p>Backups are as important as the data you have. If all you’ve got is a directory
|
||
full of meme GIFs that you don’t mind losing, then backups may be a waste of
|
||
time and money. I have recently taken to buying as much of my music as possible
|
||
(especially through Bandcamp, and especially on Bandcamp Fridays!). While much
|
||
of the music I buy does exist on a remote server at a company somewhere, the
|
||
cost of having to re-download and re-organize all of it well outweighs the cost
|
||
of proper backups. Not to mention the music which I can’t get anywhere else
|
||
anymore. Nor to further mention the other data which I have. All of this is to
|
||
say: backups are worth it to me.</p>
|
||
|
||
<p>Recently I wanted to setup NFS on my home network. I was concerned about messing
|
||
something up, and erasing the directory I had intended to share, so I wanted to
|
||
backup the data. For a while I’ve been intending to setup backups (as everyone
|
||
probably does), but it was never a priority. This project helped to prioritze
|
||
it. I had read about <a href="https://rclone.org/">RClone</a>, a command-line utility for
|
||
interacting with an incredible number of cloud services. I messed around a bit
|
||
with it, found it to my liking, and started shopping around for a cloud storage
|
||
solution. Enter <a href="https://www.backblaze.com/">Backblaze</a>. The folks that publish
|
||
all those hard-drive stats? Turns out they also run a business where they
|
||
provide cloud storage. It’s inexpensive, reliable, and straight-forward. The
|
||
last step was to automate it with systemd timer units.</p>
|
||
|
||
<h2 id="backblaze-setup">Backblaze Setup</h2>
|
||
|
||
<ul>
|
||
<li><a href="https://www.backblaze.com/">Backblaze Site</a></li>
|
||
<li><a href="https://www.backblaze.com/help.html">Backblaze Docs</a></li>
|
||
</ul>
|
||
|
||
<p>First step is to setup Backblaze. Create an account, verify email address, all
|
||
that jazz. I’d recommend enabling multi-factor authentication on the
|
||
<strong>Account</strong> -> <strong>My Settings</strong> page, under <strong>Security</strong>. Next, click on the
|
||
<strong>Account</strong> -> <strong>Application Keys</strong> page, and generate a new key. Fill in the
|
||
blanks (I gave my key full access to all buckets), copy the important bits, and
|
||
store them somewhere safe (like your password vault).</p>
|
||
|
||
<h2 id="rclone-setup">RClone Setup</h2>
|
||
|
||
<ul>
|
||
<li><a href="https://rclone.org/">RClone Site</a></li>
|
||
<li><a href="https://rclone.org/b2/">RClone Backblaze B2 Page</a></li>
|
||
</ul>
|
||
|
||
<p>Download and install RClone. Next run <code class="language-plaintext highlighter-rouge">rclone config</code> and walk through the
|
||
prompts. I’m using Backblaze, so I select “Backblaze B2” as my storage backend.
|
||
Then I add the application key ID and application key secret (key) at the
|
||
relevant prompts. For all of this configuration, I named the remote “backblaze”,
|
||
though a shorter name can make commands easier. Regardless, verify the
|
||
configuration is setup properly by running <code class="language-plaintext highlighter-rouge">rclone lsd backblaze:</code>, which will
|
||
list buckets. Unless a bucket was already configured, nothing will show up, and
|
||
also there won’t be any errors.</p>
|
||
|
||
<h2 id="backup-configuration">Backup Configuration</h2>
|
||
|
||
<p>Now, figure out how you want to backup your data. I have a <a href="/2020/06/14/setting-up-btrfs.html">BTRFS RAID setup
|
||
with multiple sub-volumes</a>, each for a
|
||
different data type: one for Books, one for Music, and so on. Since creating a
|
||
bucket doesn’t cost anything, I decided to split my backups similarly. I created
|
||
the buckets I wanted, and did a “manual” RClone sync of the data.</p>
|
||
|
||
<p><code class="language-plaintext highlighter-rouge">rclone sync --fast-list --transfers 20 /path/to/Books
|
||
backblaze:bucket-for-Books-backups</code></p>
|
||
|
||
<p>The “–fast-list” and “–transfers” options are specified on the <a href="https://rclone.org/b2/">RClone
|
||
Backblaze B2 page</a>, along with some others that may be
|
||
of interest.</p>
|
||
|
||
<p>At this point, my data was “backed-up”, and I could muck about with it more
|
||
confidently. Also, at this point, configuring back-ups is done. Run those RClone
|
||
sync commands once a week, and all is set. I don’t want to remember to do
|
||
things, though.</p>
|
||
|
||
<h2 id="automating-the-process">Automating the Process</h2>
|
||
|
||
<p>The first thing to do is create a user-agnostic location for the configuration
|
||
file and some additional files. I chose <code class="language-plaintext highlighter-rouge">/etc/rclone</code>, and copied the RClone
|
||
configuration file generated previously to this directory as <code class="language-plaintext highlighter-rouge">backblaze.conf</code>.</p>
|
||
|
||
<p>Next, I created a filter file. RClone has extensive <a href="https://rclone.org/filtering/">filtering
|
||
options</a>. For my current needs, a single file
|
||
will suffice.</p>
|
||
|
||
<h3 id="defaultfilter">default.filter</h3>
|
||
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># Exclude BTRFS snapshot directories
|
||
- .snapshots/**
|
||
# Exclude Syncthing configuration directories
|
||
- .stfolder/**
|
||
</code></pre></div></div>
|
||
|
||
<p>systemd timer units ( [<a href="https://wiki.archlinux.org/title/Systemd/Timers">Arch
|
||
Wiki</a>]
|
||
[<a href="https://man.archlinux.org/man/systemd.timer.5">Manual</a>] ) are triggers that
|
||
activate on a schedule. That schedule can be dynamic (relative to a
|
||
previous/other trigger), or static (at 6:15 every day). A timer unit triggers a
|
||
service unit, which does the work. For my backups, I decided to run a sync every
|
||
hour, at sometime between the 15 and 45 minute mark of that hour. To simplify
|
||
having multiple timer units that all do the same thing, I setup a template unit
|
||
(see the <strong>Note</strong> here: <a href="https://wiki.archlinux.org/title/Systemd#Using_units">Arch
|
||
Wiki</a>).</p>
|
||
|
||
<h3 id="rclone-backuptimer">rclone-backup@.timer</h3>
|
||
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[Unit]
|
||
Description=RClone Backup Timer Template
|
||
|
||
[Timer]
|
||
# Run every hour, sometime between the 15 minute and 45 minute mark
|
||
OnCalendar=*-*-* *:15:00
|
||
AccuracySec=30min
|
||
RandomizedDelaySec=5min
|
||
|
||
# The %i is whatever value is after the "@" for the configured unit. For
|
||
# example, rclone-backup@Books.timer will run the rclone-backup@Books.service
|
||
Unit=rclone-backup@%i.service
|
||
|
||
[Install]
|
||
WantedBy=timers.target
|
||
</code></pre></div></div>
|
||
|
||
<p>Then I can <code class="language-plaintext highlighter-rouge">enable</code> and <code class="language-plaintext highlighter-rouge">start</code> a timer for each directory to backup. To
|
||
minimize configuration, I also setup the service file to be a template. This
|
||
requires a bit of inflexible coordination: the directory name must match to a
|
||
part of the bucket name.</p>
|
||
|
||
<h3 id="rclone-backupservice">rclone-backup@.service</h3>
|
||
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[Unit]
|
||
Description=RClone Backup of %I
|
||
|
||
[Service]
|
||
Type=simple
|
||
ExecStart=/usr/bin/rclone sync -v --config "/etc/rclone/backblaze.conf" --fast-list --transfers 20 --filter-from "/etc/rclone/default.filter" /path/to/%i/ backblaze:bucket-for-%i-backups
|
||
</code></pre></div></div>
|
||
|
||
<p>The <code class="language-plaintext highlighter-rouge">--config</code> option allows us to specify the configuration in the <code class="language-plaintext highlighter-rouge">/etc</code>
|
||
directory. I include <code class="language-plaintext highlighter-rouge">-v</code> to have some additional output in the journal.
|
||
Again, <code class="language-plaintext highlighter-rouge">--fast-list</code> and <code class="language-plaintext highlighter-rouge">--transfers</code> are used to speed up the process and keep
|
||
costs lower. Then I <code class="language-plaintext highlighter-rouge">--filter-from</code> the “default.filter” file.</p>
|
||
|
||
<p>Place each of these files (<code class="language-plaintext highlighter-rouge">rclone-backup@.timer</code> and <code class="language-plaintext highlighter-rouge">rclone-backup@.service</code>)
|
||
into <code class="language-plaintext highlighter-rouge">/etc/systemd/system</code>. For each directory, enable and start the timer
|
||
unit; <code class="language-plaintext highlighter-rouge">systemctl enable rclone-backup@Example.timer</code> and <code class="language-plaintext highlighter-rouge">systemctl start
|
||
rclone-backup@Example.timer</code> will backup <code class="language-plaintext highlighter-rouge">/path/to/Example/</code> to the
|
||
<code class="language-plaintext highlighter-rouge">bucket-for-Example-backups</code> bucket.</p>
|
||
|
||
<h1 id="next-steps">Next Steps</h1>
|
||
|
||
<p>I would like to get some sort of metrics and dashboards setup to track backup
|
||
status and statistics. It could be very useful to be notified if a backup ever
|
||
fails.</p>
|
||
|
||
<p>Eventually, I’ll upload this to a repository somewhere for ease of access and
|
||
backup. When I do, I’ll update this post.</p>
|
||
|
||
<div class="author_info">
|
||
Bill Niblock
|
||
<a href="https://unlicense.org/"
|
||
aria-label="Code dedicated to the public domain under Unlicense">
|
||
<span class="fa fa-cc-pd" aria-hidden="true"
|
||
title="Code dedicated to the public domain under Unlicense"</span>
|
||
</a>
|
||
<a href="https://creativecommons.org/publicdomain/zero/1.0/"
|
||
aria-label="Published to the public domain under CC0">
|
||
<span class="fa fa-cc-zero" aria-hidden="true"
|
||
title="Content dedicated to the public domain under CC0"</span>
|
||
</a>
|
||
2023-05-07
|
||
<br />
|
||
[
|
||
|
||
<a href="/topics/technology">technology</a>
|
||
|
||
]
|
||
</div>
|
||
</article>
|
||
</main>
|
||
<footer>
|
||
<nav>
|
||
<div><a href="/">home</a></div>
|
||
|
||
<div><a href="/topics/all">all</a></div>
|
||
|
||
<div><a href="/topics/gaming">gaming</a></div>
|
||
|
||
<div><a href="/topics/life">life</a></div>
|
||
|
||
<div><a href="/topics/philosophy">philosophy</a></div>
|
||
|
||
<div><a href="/topics/technology">technology</a></div>
|
||
|
||
<div><a href="/topics/writing">writing</a></div>
|
||
|
||
</nav>
|
||
|
||
<hr />
|
||
|
||
<div><a href="https://www.theinternetvagabond.com/now">Life In Progress</a></div>
|
||
|
||
<hr />
|
||
|
||
<section class="h-card">
|
||
<section class="footer_about" id="about">
|
||
<div>The Site</div>
|
||
<div>
|
||
<a href="https://www.theinternetvagabond.com/feed.xml"
|
||
aria-label="RSS feed for the site">
|
||
<span class="fa fa-rss" aria-hidden="true"
|
||
title="RSS Feed"</span>
|
||
</a> |
|
||
<a href="https://theinternetvagabond.goatcounter.com/"
|
||
aria-label="GoatCounter statistics for the site">
|
||
<span class="fa fa-bar-chart" aria-hidden="true"
|
||
title="GoatCounter Statistics"</span>
|
||
</a> |
|
||
<a href="https://codeberg.org/VagabondAzulien/the-internet-vagabond-dot-com"
|
||
aria-label="Source code repository for the site">
|
||
<span class="fa fa-code" aria-hidden="true"
|
||
title="Site Source Code"</span>
|
||
</a>
|
||
</div>
|
||
<a class="u-url u-uid" href="https://theinternetvagabond.com"></a>
|
||
<p>
|
||
This site is a small slice of internet real-estate that I use for
|
||
occasional writing. Nothing I say is visionary or profound. I
|
||
focus on technology, gaming, and philosophy. All opinions my
|
||
own.
|
||
</p>
|
||
<div>The Vagabond</div>
|
||
<div>
|
||
<a rel="me"
|
||
href="mailto:bill@theinternetvagabond.com"
|
||
aria-label="Email Bill at The Internet Vagabond dot com">
|
||
<span class="fa fa-envelope-o" aria-hidden="true"
|
||
title="Email bill at theinternetvagabond.com"</span>
|
||
</a> |
|
||
<a class="u-url" rel="me"
|
||
href="https://matrix.to/#/@vagabondazulien:matrix.org"
|
||
aria-label="Speak with me on Matrix">
|
||
<span class="fa fa-matrix-org" aria-hidden="true"
|
||
title="Speak with me on Matrix"</span>
|
||
</a> |
|
||
<a class="u-url" rel="me"
|
||
href="https://mastodon.social/@azulien"
|
||
aria-label="Find me on the Fediverse">
|
||
<span class="fa fa-mastodon" aria-hidden="true"
|
||
title="Find me on the Fediverse"</span>
|
||
</a> |
|
||
<a class="u-url" rel="me" href="https://www.twitch.tv/vagabondazulien/profile"
|
||
aria-label="Link to my Twitch channel">
|
||
<span class="fa fa-twitch " aria-hidden="true"
|
||
title="My Twitch channel"</span>
|
||
</a>
|
||
</div>
|
||
<p>
|
||
My name is <span class="p-name">Bill Niblock</span>. <span
|
||
class="p-note">I'm a computer scientist by education, a technologist
|
||
by trade, a gamer by hobby, and a philosopher by accident. I
|
||
live in <span class="p-locality">Buffalo</span>, <span class="p-region">
|
||
New York</span>, <span class="p-country-name">USA</span>.<br />
|
||
<br />
|
||
My PGP Key is <span class="u-key" id="key">CCE7 3682 331B 5614 9FAB
|
||
7383 7359 80B2 6381 C91E</span>.
|
||
</p>
|
||
</section>
|
||
<section style="display: none;">
|
||
<span class="p-category">Gaming</span>
|
||
<span class="p-category">Technology</span>
|
||
<span class="p-category">Philosophy</span>
|
||
<span class="p-category">Open Source Software</span>
|
||
<span class="p-category">Self-Hosting</span>
|
||
<span class="p-category">Coffee</span>
|
||
<span class="u-email">bill@theinternetvagabond.com</span>
|
||
</section>
|
||
</section>
|
||
</footer>
|
||
|
||
</div>
|
||
</body>
|
||
</html>
|