<?xml version="1.0" encoding="utf-8" standalone="yes"?>

<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <title>Thomas Letan's Blog - self-hosting</title>
    <link>https://soap.coffee/~lthms/tags/self-hosting.html</link>
    <description>Posts tagged "self-hosting"</description>
    <atom:link href="https://soap.coffee/~lthms/tags/self-hosting.xml" rel="self"
               type="application/rss+xml" />
    
    
    <item>
      <title>I Cannot SSH Into My Server Anymore (And That’s Fine)</title>
      <link>https://soap.coffee/~lthms/posts/i-cannot-ssh-into-my-server-anymore.html</link>
      <guid>https://soap.coffee/~lthms/posts/i-cannot-ssh-into-my-server-anymore.html</guid>
      <pubDate>January 5, 2026</pubDate>
      <description>
        
        &lt;h1&gt;I Cannot SSH Into My Server Anymore (And That’s Fine)&lt;/h1&gt;&lt;div id=&quot;tags-list&quot;&gt;&lt;span class=&quot;icon&quot;&gt;&lt;svg&gt;&lt;use href=&quot;/~lthms/img/icons.svg#tag&quot;&gt;&lt;/use&gt;&lt;/svg&gt;&lt;/span&gt;&amp;nbsp;&lt;a href=&quot;/~lthms/tags/coreos.html&quot; class=&quot;tag hover-coral&quot; marked=&quot;&quot;&gt;coreos&lt;/a&gt; &lt;span class=&quot;icon&quot;&gt;&lt;svg&gt;&lt;use href=&quot;/~lthms/img/icons.svg#tag&quot;&gt;&lt;/use&gt;&lt;/svg&gt;&lt;/span&gt;&amp;nbsp;&lt;a href=&quot;/~lthms/tags/docker.html&quot; class=&quot;tag hover-lavender&quot; marked=&quot;&quot;&gt;docker&lt;/a&gt; &lt;span class=&quot;icon&quot;&gt;&lt;svg&gt;&lt;use href=&quot;/~lthms/img/icons.svg#tag&quot;&gt;&lt;/use&gt;&lt;/svg&gt;&lt;/span&gt;&amp;nbsp;&lt;a href=&quot;/~lthms/tags/meta.html&quot; class=&quot;tag hover-mint&quot; marked=&quot;&quot;&gt;meta&lt;/a&gt; &lt;span class=&quot;icon&quot;&gt;&lt;svg&gt;&lt;use href=&quot;/~lthms/img/icons.svg#tag&quot;&gt;&lt;/use&gt;&lt;/svg&gt;&lt;/span&gt;&amp;nbsp;&lt;a href=&quot;/~lthms/tags/self-hosting.html&quot; class=&quot;tag hover-lavender&quot; marked=&quot;&quot;&gt;self-hosting&lt;/a&gt; &lt;span class=&quot;icon&quot;&gt;&lt;svg&gt;&lt;use href=&quot;/~lthms/img/icons.svg#tag&quot;&gt;&lt;/use&gt;&lt;/svg&gt;&lt;/span&gt;&amp;nbsp;&lt;a href=&quot;/~lthms/tags/terraform.html&quot; class=&quot;tag hover-lemon&quot; marked=&quot;&quot;&gt;terraform&lt;/a&gt; &lt;span class=&quot;icon&quot;&gt;&lt;svg&gt;&lt;use href=&quot;/~lthms/img/icons.svg#tag&quot;&gt;&lt;/use&gt;&lt;/svg&gt;&lt;/span&gt;&amp;nbsp;&lt;a href=&quot;/~lthms/tags/vultr.html&quot; class=&quot;tag hover-sky&quot; marked=&quot;&quot;&gt;vultr&lt;/a&gt; &lt;/div&gt;
&lt;blockquote&gt;
&lt;p&gt;&lt;em&gt;I would like to thank Yann Régis-Gianas, Sylvain Ribstein and Paul Laforgue
for their feedback and careful review.&lt;/em&gt;&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;To kick off 2026, I had clear objectives in mind: decommissioning &lt;code class=&quot;hljs&quot;&gt;moana&lt;/code&gt;, my
trusty $100+/month VPS, and setting up &lt;code class=&quot;hljs&quot;&gt;tinkerbell&lt;/code&gt;, its far less costly
successor.&lt;/p&gt;
&lt;p&gt;On the one hand, I have been using &lt;code class=&quot;hljs&quot;&gt;moana&lt;/code&gt; to self-host a number of services,
and it was very handy to know that I had always a go-to place to experiment
with whatever caught my interest. On the other hand, $100/month is obviously a
lot of money, and looking back at how I used it in 2025, it was not
particularly well spent. It was time to downsize.&lt;/p&gt;
&lt;p&gt;Now that &lt;code class=&quot;hljs&quot;&gt;tinkerbell&lt;/code&gt; is up and running, I cannot even SSH into it. In fact,
&lt;em&gt;nothing&lt;/em&gt; can.&lt;/p&gt;
&lt;p&gt;There is no need. To update one of the services it hosts, I push a new container
image to the appropriate registry with the correct tag. &lt;code class=&quot;hljs&quot;&gt;tinkerbell&lt;/code&gt; will fetch
and deploy it. All on its own.&lt;/p&gt;
&lt;p&gt;In this article, I walk through the journey that led me to the smoke and
mirrors behind this magic trick: &lt;a href=&quot;https://fedoraproject.org/coreos/&quot; class=&quot;hover-peach&quot; marked=&quot;&quot;&gt;Fedora CoreOS&amp;nbsp;&lt;span class=&quot;icon&quot;&gt;&lt;svg&gt;&lt;use href=&quot;/~lthms/img/icons.svg#external-link&quot;&gt;&lt;/use&gt;&lt;/svg&gt;&lt;/span&gt;&lt;/a&gt;, &lt;a href=&quot;https://coreos.github.io/ignition/&quot; class=&quot;hover-mint&quot; marked=&quot;&quot;&gt;Ignition&amp;nbsp;&lt;span class=&quot;icon&quot;&gt;&lt;svg&gt;&lt;use href=&quot;/~lthms/img/icons.svg#github&quot;&gt;&lt;/use&gt;&lt;/svg&gt;&lt;/span&gt;&lt;/a&gt; and &lt;a href=&quot;https://docs.podman.io/en/latest/markdown/podman-quadlet.1.html&quot; class=&quot;hover-sky&quot; marked=&quot;&quot;&gt;Podman
Quadlets&amp;nbsp;&lt;span class=&quot;icon&quot;&gt;&lt;svg&gt;&lt;use href=&quot;/~lthms/img/icons.svg#external-link&quot;&gt;&lt;/use&gt;&lt;/svg&gt;&lt;/span&gt;&lt;/a&gt; in the main roles, with &lt;a href=&quot;https://developer.hashicorp.com/terraform&quot; class=&quot;hover-coral&quot; marked=&quot;&quot;&gt;Terraform&amp;nbsp;&lt;span class=&quot;icon&quot;&gt;&lt;svg&gt;&lt;use href=&quot;/~lthms/img/icons.svg#external-link&quot;&gt;&lt;/use&gt;&lt;/svg&gt;&lt;/span&gt;&lt;/a&gt; as an essential supporting
character. This stack checks all the boxes I care about.&lt;/p&gt;
&lt;div class=&quot;markdown-alert markdown-alert-note&quot;&gt;&lt;p class=&quot;markdown-alert-title&quot;&gt;&lt;svg class=&quot;octicon octicon-info mr-2&quot; viewbox=&quot;0 0 16 16&quot; version=&quot;1.1&quot; width=&quot;16&quot; height=&quot;16&quot; aria-hidden=&quot;true&quot;&gt;&lt;path d=&quot;M0 8a8 8 0 1 1 16 0A8 8 0 0 1 0 8Zm8-6.5a6.5 6.5 0 1 0 0 13 6.5 6.5 0 0 0 0-13ZM6.5 7.75A.75.75 0 0 1 7.25 7h1a.75.75 0 0 1 .75.75v2.75h.25a.75.75 0 0 1 0 1.5h-2a.75.75 0 0 1 0-1.5h.25v-2h-.25a.75.75 0 0 1-.75-.75ZM8 6a1 1 0 1 1 0-2 1 1 0 0 1 0 2Z&quot;&gt;&lt;/path&gt;&lt;/svg&gt;Note&lt;/p&gt;&lt;p&gt;For interested readers, I have published &lt;code class=&quot;hljs&quot;&gt;tinkerbell&lt;/code&gt;’s &lt;a href=&quot;https://github.com/lthms/tinkerbell&quot; class=&quot;hover-lavender&quot; marked=&quot;&quot;&gt;full setup&amp;nbsp;&lt;span class=&quot;icon&quot;&gt;&lt;svg&gt;&lt;use href=&quot;/~lthms/img/icons.svg#github&quot;&gt;&lt;/use&gt;&lt;/svg&gt;&lt;/span&gt;&lt;/a&gt; on
GitHub. This article reads as an experiment log, and if you are only
interested in the final result, you should definitely have a look.&lt;/p&gt;
&lt;/div&gt;
&lt;h2&gt;Container-Centric, Declarative, and Low-Maintenance&lt;/h2&gt;
&lt;p&gt;Going into this, I knew I didn’t want to reproduce &lt;code class=&quot;hljs&quot;&gt;moana&lt;/code&gt;’s setup—it was fully
manual&lt;label for=&quot;fn1&quot; class=&quot;sidenote-number margin-toggle&quot;&gt;&lt;/label&gt;&lt;input id=&quot;fn1&quot; type=&quot;checkbox&quot; class=&quot;margin-toggle&quot;&gt;&lt;span class=&quot;note-right sidenote note&quot;&gt;&lt;span class=&quot;footnote-p&quot;&gt;In the end, I never took the time to publish a write-up about it, so
in a nutshell: everything relied on &lt;a href=&quot;https://github.com/lthms/nspawn&quot; class=&quot;hover-periwinkle&quot; marked=&quot;&quot;&gt;a small script&amp;nbsp;&lt;span class=&quot;icon&quot;&gt;&lt;svg&gt;&lt;use href=&quot;/~lthms/img/icons.svg#github&quot;&gt;&lt;/use&gt;&lt;/svg&gt;&lt;/span&gt;&lt;/a&gt; that enabled
me to create interconnected &lt;a href=&quot;https://man7.org/linux/man-pages/man5/systemd.nspawn.5.html&quot; class=&quot;hover-coral&quot; marked=&quot;&quot;&gt;nspawn containers&amp;nbsp;&lt;span class=&quot;icon&quot;&gt;&lt;svg&gt;&lt;use href=&quot;/~lthms/img/icons.svg#external-link&quot;&gt;&lt;/use&gt;&lt;/svg&gt;&lt;/span&gt;&lt;/a&gt; on the spot. &lt;/span&gt;
&lt;/span&gt; and I no longer have the time or the motivation to fiddle with
the internals of a server. Instead, I wanted to embrace the principles my
DevOps colleagues had taught me over the past two years.&lt;/p&gt;
&lt;p&gt;My initial idea was to start with this very website, since it was the only
service deployed on &lt;code class=&quot;hljs&quot;&gt;moana&lt;/code&gt; that I really wanted to keep. Since &lt;a href=&quot;./DreamWebsite.html&quot; class=&quot;hover-lavender&quot; marked=&quot;&quot;&gt;I had written
a container image for this website&lt;/a&gt;, I just had to look
for the most straightforward and future-proof way to deploy it in production™—
something I could later extend to deploy more cool projects, if I ever wanted
to&lt;label for=&quot;fn2&quot; class=&quot;sidenote-number margin-toggle&quot;&gt;&lt;/label&gt;&lt;input id=&quot;fn2&quot; type=&quot;checkbox&quot; class=&quot;margin-toggle&quot;&gt;&lt;span class=&quot;note-left sidenote note&quot;&gt;&lt;span class=&quot;footnote-p&quot;&gt;If my goal was limited to host a static website, this whole setup
would arguably be both overengineered and counterproductive. &lt;code class=&quot;hljs&quot;&gt;tinkerbell&lt;/code&gt;
was to become my little foothold in the Internet, though. My “cloud
homelabs,” of sorts. &lt;/span&gt;
&lt;/span&gt;.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://docs.docker.com/compose/&quot; class=&quot;hover-peach&quot; marked=&quot;&quot;&gt;Docker Compose&amp;nbsp;&lt;span class=&quot;icon&quot;&gt;&lt;svg&gt;&lt;use href=&quot;/~lthms/img/icons.svg#external-link&quot;&gt;&lt;/use&gt;&lt;/svg&gt;&lt;/span&gt;&lt;/a&gt; alone wasn’t a good fit. I like compose files, but one needs
to provision and manage a VM to host them. Ansible can provision VMs, but that
road comes with its own set of struggles. Writing good playbooks has always
felt surprisingly difficult to me. In particular, a good playbook is supposed
to handle two very different scenarios—provisioning a brand new machine, and
updating a pre-existing deployment—and I have found it particularly challenging
to ensure that both paths reliably produce the same result.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://kubernetes.io/&quot; class=&quot;hover-lavender&quot; marked=&quot;&quot;&gt;Kubernetes&amp;nbsp;&lt;span class=&quot;icon&quot;&gt;&lt;svg&gt;&lt;use href=&quot;/~lthms/img/icons.svg#external-link&quot;&gt;&lt;/use&gt;&lt;/svg&gt;&lt;/span&gt;&lt;/a&gt; was &lt;em&gt;very&lt;/em&gt; appealing on paper. I have seen engineers turn compose
files into &lt;a href=&quot;https://helm.sh/docs/topics/charts/&quot; class=&quot;hover-rose&quot; marked=&quot;&quot;&gt;Helm charts&amp;nbsp;&lt;span class=&quot;icon&quot;&gt;&lt;svg&gt;&lt;use href=&quot;/~lthms/img/icons.svg#external-link&quot;&gt;&lt;/use&gt;&lt;/svg&gt;&lt;/span&gt;&lt;/a&gt; and be done with it. If I could do the same thing,
wouldn’t that be bliss? Unfortunately, Kubernetes is a notoriously complex
stack, resulting from compromises made to address challenges I simply don’t
face. Managed clusters could make things easier, but they aren’t cheap. That
would defeat the initial motivation behind retiring &lt;code class=&quot;hljs&quot;&gt;moana&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://fedoraproject.org/coreos/&quot; class=&quot;hover-peach&quot; marked=&quot;&quot;&gt;CoreOS&amp;nbsp;&lt;span class=&quot;icon&quot;&gt;&lt;svg&gt;&lt;use href=&quot;/~lthms/img/icons.svg#external-link&quot;&gt;&lt;/use&gt;&lt;/svg&gt;&lt;/span&gt;&lt;/a&gt;, being an operating system specifically built to &lt;em&gt;run containers&lt;/em&gt;,
obviously stood out. That said, I had very little intuition on how it could
work in practice. So I started digging. I learned about Ignition first. Its
purpose is to provision a VM exactly once, at first boot. If you need to change
something afterwards, you throw away your VM and create a new one. This may
seem counter-intuitive, but since it eliminates the main reason I was looking
for an alternative to Ansible, I was hooked&lt;label for=&quot;fn3&quot; class=&quot;sidenote-number margin-toggle&quot;&gt;&lt;/label&gt;&lt;input id=&quot;fn3&quot; type=&quot;checkbox&quot; class=&quot;margin-toggle&quot;&gt;&lt;span class=&quot;note-right sidenote note&quot;&gt;&lt;span class=&quot;footnote-p&quot;&gt;CoreOS and Ignition enable me to think about virtual machines the same
way OCaml or Haskell trained me to think about data: as immutable values. &lt;/span&gt;
&lt;/span&gt;.&lt;/p&gt;
&lt;p&gt;I found out how to use systemd unit files to start containers via &lt;code class=&quot;hljs&quot;&gt;podman&lt;/code&gt; CLI
commands. That was way too cumbersome, so I pushed on for a way to orchestrate
containers &lt;em&gt;à la&lt;/em&gt; Docker Compose. That’s when I discovered Podman Quadlets and
&lt;a href=&quot;https://docs.podman.io/en/stable/markdown/podman-auto-update.1.html&quot; class=&quot;hover-rose&quot; marked=&quot;&quot;&gt;auto-updates&amp;nbsp;&lt;span class=&quot;icon&quot;&gt;&lt;svg&gt;&lt;use href=&quot;/~lthms/img/icons.svg#external-link&quot;&gt;&lt;/use&gt;&lt;/svg&gt;&lt;/span&gt;&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;With that, everything clicked. I knew what I wanted to do, and I was very
excited about it.&lt;/p&gt;
&lt;h2&gt;Assembling &lt;code class=&quot;hljs&quot;&gt;tinkerbell&lt;/code&gt;&lt;/h2&gt;
&lt;p&gt;For more than a year now, my website has been &lt;a href=&quot;./DreamWebsite.html&quot; class=&quot;hover-lavender&quot; marked=&quot;&quot;&gt;served from RAM by a standalone,
static binary built in OCaml&lt;/a&gt;, with TLS termination handled by Nginx and
&lt;code class=&quot;hljs&quot;&gt;certbot&lt;/code&gt;’s certificates renewal performed by yours truly&lt;label for=&quot;fn4&quot; class=&quot;sidenote-number margin-toggle&quot;&gt;&lt;/label&gt;&lt;input id=&quot;fn4&quot; type=&quot;checkbox&quot; class=&quot;margin-toggle&quot;&gt;&lt;span class=&quot;note-left sidenote note&quot;&gt;&lt;span class=&quot;footnote-p&quot;&gt;I didn’t lie when I said &lt;code class=&quot;hljs&quot;&gt;moana&lt;/code&gt;’s setup was indeed &lt;em&gt;manual&lt;/em&gt;. &lt;/span&gt;
&lt;/span&gt;. I didn’t
have any reason to fundamentally change this architecture. I was simply looking
for a way to automate their deployment.&lt;/p&gt;
&lt;h3&gt;Container-Centric, …&lt;/h3&gt;
&lt;p&gt;The logical thing to do was to have &lt;code class=&quot;hljs&quot;&gt;tinkerbell&lt;/code&gt; run two containers:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;The reverse proxy:&lt;/strong&gt; I had been firmly on Team Nginx for years now, but
when I heard &lt;a href=&quot;https://caddyserver.com/&quot; class=&quot;hover-coral&quot; marked=&quot;&quot;&gt;Caddy&amp;nbsp;&lt;span class=&quot;icon&quot;&gt;&lt;svg&gt;&lt;use href=&quot;/~lthms/img/icons.svg#external-link&quot;&gt;&lt;/use&gt;&lt;/svg&gt;&lt;/span&gt;&lt;/a&gt; could “&lt;em&gt;automatically obtain&lt;/em&gt; and &lt;em&gt;renew&lt;/em&gt; TLS
certificates,” I was sold on giving it a try.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The website itself:&lt;/strong&gt; Static binaries can be wrapped inside a container
with close to zero overhead using the &lt;a href=&quot;https://hub.docker.com/_/scratch&quot; class=&quot;hover-lemon&quot; marked=&quot;&quot;&gt;&lt;code class=&quot;hljs&quot;&gt;scratch&lt;/code&gt;&amp;nbsp;&lt;span class=&quot;icon&quot;&gt;&lt;svg&gt;&lt;use href=&quot;/~lthms/img/icons.svg#external-link&quot;&gt;&lt;/use&gt;&lt;/svg&gt;&lt;/span&gt;&lt;/a&gt; base image, so I
did just that. I published it to a free-plan, public registry hosted on Vultr
that I created for the occasion&lt;label for=&quot;fn5&quot; class=&quot;sidenote-number margin-toggle&quot;&gt;&lt;/label&gt;&lt;input id=&quot;fn5&quot; type=&quot;checkbox&quot; class=&quot;margin-toggle&quot;&gt;&lt;span class=&quot;note-right sidenote note&quot;&gt;&lt;span class=&quot;footnote-p&quot;&gt;Which means getting an offline copy of this website is now as
simple as calling &lt;code class=&quot;hljs&quot;&gt;docker pull ams.vultrcr.com/lthms/www/soap.coffee:live&lt;/code&gt;. &lt;/span&gt;
&lt;/span&gt;.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;/p&gt;&lt;figure&gt;&lt;img src=&quot;https://mermaid.ink/img/pako:eNo9UMtOwzAQ_BVrTyCllZs4JPEBiYZjuQASEnUPTrxJLBK7ch1BafPvuC_2tDM7M7vaA9RWIXBoevtdd9J5snoVhoR6Wpe9RuM3ZDZ7PPKcEs5YciTLtYBSKrUnd--rt3sBm4t-N1atk9uOeG2-0FXY95fB8hZwJOX6A6ud9nj1oFEQQeu0Au7diBEM6AZ5gnA4SQT4DgcUwENrcPRO9gKEmYJtK82ntcPN6ezYdsAb2e8CGrdKenzWMtw0_LMuLERX2tF44AnLziHAD_ATIM3mMY3zlCZskRUPRQR74DGL58WCXrk8S6cIfs9b6TxlWZ4EcsFYkdOgR6W9dS-Xj9bWNLqF6Q97DWuS?type=png&quot;&gt;&lt;figcaption&gt;&lt;p&gt;Nothing beats a straightforward architecture&lt;/p&gt;&lt;/figcaption&gt;&lt;/figure&gt;&lt;p&gt;&lt;/p&gt;
&lt;p&gt;Nothing fancy or unexpected here, which made it a good target for a first
deployment. It was time to open Neovim to write some YAML.&lt;/p&gt;
&lt;h3&gt;Declarative, …&lt;/h3&gt;
&lt;p&gt;At this point, the architecture was clear. The next step was to turn it into
something a machine could execute. To that end, I needed two things: first an
Ignition configuration, then a CoreOS VM to run it.&lt;/p&gt;
&lt;h4&gt;The Proof of Concept&lt;/h4&gt;
&lt;p&gt;Ignition configurations (&lt;code class=&quot;hljs&quot;&gt;.ign&lt;/code&gt;) are JSON files primarily intended to be
consumed by machines. They are produced from YAML files using a tool called
&lt;a href=&quot;https://coreos.github.io/butane/&quot; class=&quot;hover-sky&quot; marked=&quot;&quot;&gt;Butane&amp;nbsp;&lt;span class=&quot;icon&quot;&gt;&lt;svg&gt;&lt;use href=&quot;/~lthms/img/icons.svg#github&quot;&gt;&lt;/use&gt;&lt;/svg&gt;&lt;/span&gt;&lt;/a&gt;. For instance, here is the first Butane configuration file I ended up
writing. It provisions a CoreOS VM by creating a new user (&lt;code class=&quot;hljs&quot;&gt;lthms&lt;/code&gt;), along with
a &lt;code class=&quot;hljs&quot;&gt;.ssh/authorized_keys&lt;/code&gt; file allowing me to SSH into the VM&lt;label for=&quot;fn6&quot; class=&quot;sidenote-number margin-toggle&quot;&gt;&lt;/label&gt;&lt;input id=&quot;fn6&quot; type=&quot;checkbox&quot; class=&quot;margin-toggle&quot;&gt;&lt;span class=&quot;note-left sidenote note&quot;&gt;&lt;span class=&quot;footnote-p&quot;&gt;I didn’t know at the time that I would &lt;em&gt;deliberately&lt;/em&gt; remove these
lines from the final Butane file. &lt;/span&gt;
&lt;/span&gt;.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;hljs language-yaml&quot;&gt;&lt;span class=&quot;hljs-attr&quot;&gt;variant:&lt;/span&gt; &lt;span class=&quot;hljs-string&quot;&gt;fcos&lt;/span&gt;
&lt;span class=&quot;hljs-attr&quot;&gt;version:&lt;/span&gt; &lt;span class=&quot;hljs-number&quot;&gt;1.5&lt;/span&gt;&lt;span class=&quot;hljs-number&quot;&gt;.0&lt;/span&gt;
&lt;span class=&quot;hljs-attr&quot;&gt;passwd:&lt;/span&gt;
  &lt;span class=&quot;hljs-attr&quot;&gt;users:&lt;/span&gt;
    &lt;span class=&quot;hljs-bullet&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;hljs-attr&quot;&gt;name:&lt;/span&gt; &lt;span class=&quot;hljs-string&quot;&gt;lthms&lt;/span&gt;
      &lt;span class=&quot;hljs-attr&quot;&gt;ssh_authorized_keys:&lt;/span&gt;
        &lt;span class=&quot;hljs-bullet&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;hljs-string&quot;&gt;ssh-ed25519&lt;/span&gt; &lt;span class=&quot;hljs-string&quot;&gt;AAAAC3NzaC1lZDI1NTE5AAAAIKajIx3VWRjhqIrza4ZnVnnI1g2q6NfMfMOcnSciP1Ws&lt;/span&gt; &lt;span class=&quot;hljs-string&quot;&gt;lthms@vanellope&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;What’s important to keep in mind is that Ignition runs exactly once, at first
boot. Then it is never used again. This single fact has far-reaching
consequences, and is the reason why any meaningful change implies replacing the
machine, not modifying it.&lt;/p&gt;
&lt;p&gt;Before going any further, I wanted to understand how the actual deployment was
going to work. I generated the Ignition configuration file.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;hljs language-bash&quot;&gt;butane main.bu &amp;gt; main.ign
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Then, I decided to investigate how to define the Vultr VM in Terraform. The
resulting configuration is twofold. First, we need to configure Terraform to be
able to interact with the Vultr API, using the &lt;a href=&quot;https://registry.terraform.io/providers/vultr/vultr/latest/docs&quot; class=&quot;hover-periwinkle&quot; marked=&quot;&quot;&gt;Vultr provider&amp;nbsp;&lt;span class=&quot;icon&quot;&gt;&lt;svg&gt;&lt;use href=&quot;/~lthms/img/icons.svg#external-link&quot;&gt;&lt;/use&gt;&lt;/svg&gt;&lt;/span&gt;&lt;/a&gt;. Second, I
needed to &lt;a href=&quot;https://registry.terraform.io/providers/vultr/vultr/latest/docs/resources/instance&quot; class=&quot;hover-peach&quot; marked=&quot;&quot;&gt;create the VM&amp;nbsp;&lt;span class=&quot;icon&quot;&gt;&lt;svg&gt;&lt;use href=&quot;/~lthms/img/icons.svg#external-link&quot;&gt;&lt;/use&gt;&lt;/svg&gt;&lt;/span&gt;&lt;/a&gt;&lt;label for=&quot;fn7&quot; class=&quot;sidenote-number margin-toggle&quot;&gt;&lt;/label&gt;&lt;input id=&quot;fn7&quot; type=&quot;checkbox&quot; class=&quot;margin-toggle&quot;&gt;&lt;span class=&quot;note-right sidenote note&quot;&gt;&lt;span class=&quot;footnote-p&quot;&gt;For discovering what values to put in most fields, &lt;code class=&quot;hljs&quot;&gt;vultr-cli&lt;/code&gt; is
pretty convenient. Kudos to the Vultr team for making it in the first
place. &lt;/span&gt;
&lt;/span&gt; and feed it the Ignition
configuration.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;hljs language-hcl&quot;&gt;resource &quot;vultr_instance&quot; &quot;tinkerbell&quot; {
  region = &quot;cdg&quot;
  plan = &quot;vc2-1c-1gb&quot;
  os_id = &quot;391&quot;

  label = &quot;tinkerbell&quot;
  hostname = &quot;tinkerbell&quot;

  user_data = file(&quot;main.ign&quot;)
}
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;And that was it. I invoked &lt;code class=&quot;hljs&quot;&gt;terraform apply&lt;/code&gt;, waited for a little while, then
SSHed into the newly created VM with my &lt;code class=&quot;hljs&quot;&gt;lthms&lt;/code&gt; user. Sure enough, the
&lt;code class=&quot;hljs&quot;&gt;tinkerbell&lt;/code&gt; VM was now listed in the Vultr web interface. I explored for a
little while, then called &lt;code class=&quot;hljs&quot;&gt;terraform destroy&lt;/code&gt; and rejoiced when everything
worked as expected.&lt;/p&gt;
&lt;h4&gt;The MVP&lt;/h4&gt;
&lt;p&gt;At this point, I was basically done with Terraform, and I just needed to write
the Butane configuration that would bring my containers to life. As I mentioned
earlier, the first approach I tried was to define a systemd service responsible
for invoking &lt;code class=&quot;hljs&quot;&gt;podman&lt;/code&gt;.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;hljs language-yaml&quot;&gt;&lt;span class=&quot;hljs-attr&quot;&gt;systemd:&lt;/span&gt;
  &lt;span class=&quot;hljs-attr&quot;&gt;units:&lt;/span&gt;
    &lt;span class=&quot;hljs-bullet&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;hljs-attr&quot;&gt;name:&lt;/span&gt; &lt;span class=&quot;hljs-string&quot;&gt;soap.coffee.service&lt;/span&gt;
      &lt;span class=&quot;hljs-attr&quot;&gt;enabled:&lt;/span&gt; &lt;span class=&quot;hljs-literal&quot;&gt;true&lt;/span&gt;
      &lt;span class=&quot;hljs-attr&quot;&gt;contents:&lt;/span&gt; &lt;span class=&quot;hljs-string&quot;&gt;|
        [Unit]
        Description=Web Service
        After=network-online.target
        Wants=network-online.target
&lt;/span&gt;
        [&lt;span class=&quot;hljs-string&quot;&gt;Service&lt;/span&gt;]
        &lt;span class=&quot;hljs-string&quot;&gt;ExecStart=/usr/bin/podman&lt;/span&gt; &lt;span class=&quot;hljs-string&quot;&gt;run&lt;/span&gt; &lt;span class=&quot;hljs-string&quot;&gt;\&lt;/span&gt;
          &lt;span class=&quot;hljs-string&quot;&gt;--name&lt;/span&gt; &lt;span class=&quot;hljs-string&quot;&gt;soap.coffee&lt;/span&gt; &lt;span class=&quot;hljs-string&quot;&gt;\&lt;/span&gt;
          &lt;span class=&quot;hljs-string&quot;&gt;-p&lt;/span&gt; &lt;span class=&quot;hljs-number&quot;&gt;8901&lt;/span&gt;&lt;span class=&quot;hljs-string&quot;&gt;:8901&lt;/span&gt; &lt;span class=&quot;hljs-string&quot;&gt;\&lt;/span&gt;
          &lt;span class=&quot;hljs-string&quot;&gt;--restart=always&lt;/span&gt; &lt;span class=&quot;hljs-string&quot;&gt;\&lt;/span&gt;
          &lt;span class=&quot;hljs-string&quot;&gt;ams.vultrcr.com/lthms/www/soap.coffee:latest&lt;/span&gt;
        &lt;span class=&quot;hljs-string&quot;&gt;ExecStop=/usr/bin/podman&lt;/span&gt; &lt;span class=&quot;hljs-string&quot;&gt;stop&lt;/span&gt; &lt;span class=&quot;hljs-string&quot;&gt;soap.coffee&lt;/span&gt;

        [&lt;span class=&quot;hljs-string&quot;&gt;Install&lt;/span&gt;]
        &lt;span class=&quot;hljs-string&quot;&gt;WantedBy=multi-user.target&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Adding this entry in my Butane configuration and redeploying &lt;code class=&quot;hljs&quot;&gt;tinkerbell&lt;/code&gt; got
me exactly what I wanted. My website was up and running. For the sake of
getting something working first, I added the necessary configuration for Caddy
(the container and the provisioning of its configuration file), redeployed
&lt;code class=&quot;hljs&quot;&gt;tinkerbell&lt;/code&gt; again, only to realize I also needed to create a network so that
the two containers could talk together. After half an hour or so, I got
everything working, but was left with a sour taste in my mouth.&lt;/p&gt;
&lt;p&gt;This would simply not do. I wasn’t defining anything, I was writing a shell
script in the most cumbersome way possible.&lt;/p&gt;
&lt;p&gt;Then, I remembered my initial train of thought and started to search for a way
to have Docker Compose work on CoreOS&lt;label for=&quot;fn8&quot; class=&quot;sidenote-number margin-toggle&quot;&gt;&lt;/label&gt;&lt;input id=&quot;fn8&quot; type=&quot;checkbox&quot; class=&quot;margin-toggle&quot;&gt;&lt;span class=&quot;note-left sidenote note&quot;&gt;&lt;span class=&quot;footnote-p&quot;&gt;Well, Podman Compose, I guess. &lt;/span&gt;
&lt;/span&gt;. That is when I discovered
Quadlet, whose &lt;a href=&quot;https://github.com/containers/quadlet&quot; class=&quot;hover-mint&quot; marked=&quot;&quot;&gt;initial repository does a good job justifying its
existence&amp;nbsp;&lt;span class=&quot;icon&quot;&gt;&lt;svg&gt;&lt;use href=&quot;/~lthms/img/icons.svg#github&quot;&gt;&lt;/use&gt;&lt;/svg&gt;&lt;/span&gt;&lt;/a&gt;&lt;label for=&quot;fn9&quot; class=&quot;sidenote-number margin-toggle&quot;&gt;&lt;/label&gt;&lt;input id=&quot;fn9&quot; type=&quot;checkbox&quot; class=&quot;margin-toggle&quot;&gt;&lt;span class=&quot;note-right sidenote note&quot;&gt;&lt;span class=&quot;footnote-p&quot;&gt;This repository is now archived, since Quadlet has got merged
upstream. &lt;/span&gt;
&lt;/span&gt;. In particular,&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;With quadlet, you describe how to run a container in a format that is very
similar to regular systemd config files. From these actual systemd
configurations are automatically generated (using &lt;a href=&quot;https://github.com/containers/quadlet&quot; class=&quot;hover-peach&quot; marked=&quot;&quot;&gt;systemd generators&amp;nbsp;&lt;span class=&quot;icon&quot;&gt;&lt;svg&gt;&lt;use href=&quot;/~lthms/img/icons.svg#github&quot;&gt;&lt;/use&gt;&lt;/svg&gt;&lt;/span&gt;&lt;/a&gt;).&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;To give a concrete example, here is the &lt;code class=&quot;hljs&quot;&gt;.container&lt;/code&gt; file I wrote for my
website server.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;hljs language-ini&quot;&gt;&lt;span class=&quot;hljs-section&quot;&gt;[Container]&lt;/span&gt;
&lt;span class=&quot;hljs-attr&quot;&gt;ContainerName&lt;/span&gt;=soap.c&lt;span class=&quot;hljs-literal&quot;&gt;off&lt;/span&gt;ee
&lt;span class=&quot;hljs-attr&quot;&gt;Image&lt;/span&gt;=ams.vultrcr.com/lthms/www/soap.c&lt;span class=&quot;hljs-literal&quot;&gt;off&lt;/span&gt;ee:live

&lt;span class=&quot;hljs-section&quot;&gt;[Service]&lt;/span&gt;
&lt;span class=&quot;hljs-attr&quot;&gt;Restart&lt;/span&gt;=always

&lt;span class=&quot;hljs-section&quot;&gt;[Install]&lt;/span&gt;
&lt;span class=&quot;hljs-attr&quot;&gt;WantedBy&lt;/span&gt;=multi-user.target
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;I wasn’t wasting my time teaching systemd how to start containers anymore. I
was now declaring what should exist, so that systemd—repurposed for the
occasion as a container orchestrator—could take care of the rest.&lt;/p&gt;
&lt;div class=&quot;markdown-alert markdown-alert-tip&quot;&gt;&lt;p class=&quot;markdown-alert-title&quot;&gt;&lt;svg class=&quot;octicon octicon-light-bulb mr-2&quot; viewbox=&quot;0 0 16 16&quot; version=&quot;1.1&quot; width=&quot;16&quot; height=&quot;16&quot; aria-hidden=&quot;true&quot;&gt;&lt;path d=&quot;M8 1.5c-2.363 0-4 1.69-4 3.75 0 .984.424 1.625.984 2.304l.214.253c.223.264.47.556.673.848.284.411.537.896.621 1.49a.75.75 0 0 1-1.484.211c-.04-.282-.163-.547-.37-.847a8.456 8.456 0 0 0-.542-.68c-.084-.1-.173-.205-.268-.32C3.201 7.75 2.5 6.766 2.5 5.25 2.5 2.31 4.863 0 8 0s5.5 2.31 5.5 5.25c0 1.516-.701 2.5-1.328 3.259-.095.115-.184.22-.268.319-.207.245-.383.453-.541.681-.208.3-.33.565-.37.847a.751.751 0 0 1-1.485-.212c.084-.593.337-1.078.621-1.489.203-.292.45-.584.673-.848.075-.088.147-.173.213-.253.561-.679.985-1.32.985-2.304 0-2.06-1.637-3.75-4-3.75ZM5.75 12h4.5a.75.75 0 0 1 0 1.5h-4.5a.75.75 0 0 1 0-1.5ZM6 15.25a.75.75 0 0 1 .75-.75h2.5a.75.75 0 0 1 0 1.5h-2.5a.75.75 0 0 1-.75-.75Z&quot;&gt;&lt;/path&gt;&lt;/svg&gt;Tip&lt;/p&gt;&lt;p&gt;If your containers are basically ignored by systemd, be smarter than me. Do
not try to blindly change your &lt;code class=&quot;hljs&quot;&gt;.container&lt;/code&gt; files and redeploy your VM in a
very painful and frustrating loop. Simply ask systemd for the generator logs.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;hljs&quot;&gt;sudo journalctl -b | grep -i quadlet 
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;
&lt;p&gt;I excitedly turned &lt;code class=&quot;hljs&quot;&gt;caddy.service&lt;/code&gt; into &lt;code class=&quot;hljs&quot;&gt;caddy.container&lt;/code&gt;, redeployed
&lt;code class=&quot;hljs&quot;&gt;tinkerbell&lt;/code&gt;, ran into the exact same issue I had encountered before and
discovered the easiest way for two Quadlet-defined containers to talk to each
other was to introduce a &lt;a href=&quot;https://docs.podman.io/en/latest/_static/api.html?version=latest#tag/pods&quot; class=&quot;hover-lemon&quot; marked=&quot;&quot;&gt;&lt;em&gt;pod&lt;/em&gt;&amp;nbsp;&lt;span class=&quot;icon&quot;&gt;&lt;svg&gt;&lt;use href=&quot;/~lthms/img/icons.svg#external-link&quot;&gt;&lt;/use&gt;&lt;/svg&gt;&lt;/span&gt;&lt;/a&gt;. Unlike Docker Compose which uses DNS
over a bridge network, a pod shares the network namespace, allowing containers
to communicate over &lt;em&gt;localhost&lt;/em&gt;.&lt;/p&gt;
&lt;p&gt;To define a pod, one needs to create a &lt;code class=&quot;hljs&quot;&gt;.pod&lt;/code&gt; file, and to reference it in
their &lt;code class=&quot;hljs&quot;&gt;.container&lt;/code&gt; files using the &lt;code class=&quot;hljs&quot;&gt;PodName=&lt;/code&gt; configuration option. A “few”
redeployments later, I got everything working again, and I was ready to call it
a day.&lt;/p&gt;
&lt;p&gt;And with that, &lt;code class=&quot;hljs&quot;&gt;tinkerbell&lt;/code&gt; was basically ready.&lt;/p&gt;
&lt;div class=&quot;markdown-alert markdown-alert-caution&quot;&gt;&lt;p class=&quot;markdown-alert-title&quot;&gt;&lt;svg class=&quot;octicon octicon-stop mr-2&quot; viewbox=&quot;0 0 16 16&quot; version=&quot;1.1&quot; width=&quot;16&quot; height=&quot;16&quot; aria-hidden=&quot;true&quot;&gt;&lt;path d=&quot;M4.47.22A.749.749 0 0 1 5 0h6c.199 0 .389.079.53.22l4.25 4.25c.141.14.22.331.22.53v6a.749.749 0 0 1-.22.53l-4.25 4.25A.749.749 0 0 1 11 16H5a.749.749 0 0 1-.53-.22L.22 11.53A.749.749 0 0 1 0 11V5c0-.199.079-.389.22-.53Zm.84 1.28L1.5 5.31v5.38l3.81 3.81h5.38l3.81-3.81V5.31L10.69 1.5ZM8 4a.75.75 0 0 1 .75.75v3.5a.75.75 0 0 1-1.5 0v-3.5A.75.75 0 0 1 8 4Zm0 8a1 1 0 1 1 0-2 1 1 0 0 1 0 2Z&quot;&gt;&lt;/path&gt;&lt;/svg&gt;Caution&lt;/p&gt;&lt;p&gt;I’ve later learned that restarting a container that is part of a pod will
have the (to me, unexpected) side-effect to restart all the other containers
of that pod.&lt;/p&gt;
&lt;/div&gt;
&lt;h3&gt;And Low-Maintenance&lt;/h3&gt;
&lt;p&gt;Now, the end of the previous section might have given you pause.&lt;/p&gt;
&lt;p&gt;Even a static website like this one isn’t completely “stateless.” Not only does
Caddy require a configuration file to do anything meaningful, but it is also a
stateful application as it manages TLS certificates over time. Besides, I &lt;em&gt;do&lt;/em&gt;
publish technical write-ups from time to time&lt;label for=&quot;fn10&quot; class=&quot;sidenote-number margin-toggle&quot;&gt;&lt;/label&gt;&lt;input id=&quot;fn10&quot; type=&quot;checkbox&quot; class=&quot;margin-toggle&quot;&gt;&lt;span class=&quot;note-left sidenote note&quot;&gt;&lt;span class=&quot;footnote-p&quot;&gt;Two in 2025, that’s true. But 2026 is only starting, you never know
what might come! &lt;/span&gt;
&lt;/span&gt;.&lt;/p&gt;
&lt;p&gt;Was I really at peace with having to destroy and redeploy &lt;code class=&quot;hljs&quot;&gt;tinkerbell&lt;/code&gt; every
time I need to change anything on my website?&lt;/p&gt;
&lt;p&gt;On the one hand, &lt;em&gt;yes&lt;/em&gt;. I believe I could live with that. I modify my website
only a handful of times even in good months, I think my audience could survive
with a minute of downtime before being allowed to read my latest pieces. It may
be an unpopular opinion, but considering my actual use case, it &lt;em&gt;was&lt;/em&gt; good
enough. Even the fact that I do not store the TLS certificates obtained by
Caddy anywhere persistent should not be an issue. I mean, Let’s Encrypt has
fairly generous weekly issuance limits per domain &lt;label for=&quot;fn11&quot; class=&quot;sidenote-number margin-toggle&quot;&gt;&lt;/label&gt;&lt;input id=&quot;fn11&quot; type=&quot;checkbox&quot; class=&quot;margin-toggle&quot;&gt;&lt;span class=&quot;note-right sidenote note&quot;&gt;&lt;span class=&quot;footnote-p&quot;&gt;How do I know that? Well... I might have hit the limit while hacking my
way to a working setup. &lt;/span&gt;
&lt;/span&gt;. I should be fine.&lt;/p&gt;
&lt;p&gt;On the other hand, the setup was starting to grow on me, and I have &lt;em&gt;other&lt;/em&gt; use
cases in mind that could be a good fit for it. So I started researching again,
this time to understand how a deployment philosophy so focused on immutability
was managing what seemed to be conflicting requirements.&lt;/p&gt;
&lt;p&gt;I went down other rabbit holes, looking for answers. The discovery that stood
out the most to me—to the point where it became the hook of this article—was
Podman auto-updates.&lt;/p&gt;
&lt;p&gt;To deploy a new version of a containerized application, you pull the new image
and restart the container. When you commit to this pattern, why should you be
the one performing this action? Instead, your VM can regularly check registries
for new images, and &lt;em&gt;update the required containers when necessary&lt;/em&gt;.&lt;/p&gt;
&lt;p&gt;In practice, Podman made this approach trivial to put in place. I just needed
to label my containers with &lt;code class=&quot;hljs&quot;&gt;io.containers.autoupdate&lt;/code&gt; set to &lt;code class=&quot;hljs&quot;&gt;registry&lt;/code&gt;,
enable the &lt;code class=&quot;hljs&quot;&gt;podman-auto-update&lt;/code&gt; timer&lt;label for=&quot;fn12&quot; class=&quot;sidenote-number margin-toggle&quot;&gt;&lt;/label&gt;&lt;input id=&quot;fn12&quot; type=&quot;checkbox&quot; class=&quot;margin-toggle&quot;&gt;&lt;span class=&quot;note-left sidenote note&quot;&gt;&lt;span class=&quot;footnote-p&quot;&gt;By default, the timer is triggered once a day, which felt
unnecessarily long, so I decided to make it hourly instead. &lt;/span&gt;
&lt;/span&gt;, and that was it. Now, every
time I update the tag &lt;code class=&quot;hljs&quot;&gt;www/soap.coffee:live&lt;/code&gt; to point to a newer version of my
image, my website is updated within the hour.&lt;/p&gt;
&lt;p&gt;And that is when the final piece clicked. At this point, publishing an image
becomes the only deployment step. I didn’t need SSH anymore.&lt;/p&gt;
&lt;h2&gt;The Road Ahead&lt;/h2&gt;
&lt;p&gt;&lt;code class=&quot;hljs&quot;&gt;tinkerbell&lt;/code&gt; has been running for a few days now, and I am quite pleased with
the system I have put in place. In retrospect, none of this is particularly
novel. It feels more like I am converging toward a set of practices the
industry has been gravitating toward for years.&lt;/p&gt;
&lt;p&gt;&lt;/p&gt;&lt;figure&gt;&lt;img src=&quot;/~lthms/img/iac-meme.jpg&quot;&gt;&lt;figcaption&gt;&lt;p&gt;A man looking at the “CoreOS &amp;amp; Quadlets” butterfly and wondering whether he’s looking at Infrastructure as Code. I’m not entirely sure of the answer.&lt;/p&gt;&lt;/figcaption&gt;&lt;/figure&gt;&lt;p&gt;&lt;/p&gt;
&lt;p&gt;The journey is far from being over, though. &lt;code class=&quot;hljs&quot;&gt;tinkerbell&lt;/code&gt; is up and running, and
it served you this HTML page just fine, but the moment I put SSH out of the
picture, it became a black box. Aside from some hardware metrics kindly
provided by the Vultr dashboard, I have no real visibility into what’s going on
inside. That is fine for now, but it is not a place I want to stay in forever.
I plan to spend a few more weekends building an observability stack&lt;label for=&quot;fn13&quot; class=&quot;sidenote-number margin-toggle&quot;&gt;&lt;/label&gt;&lt;input id=&quot;fn13&quot; type=&quot;checkbox&quot; class=&quot;margin-toggle&quot;&gt;&lt;span class=&quot;note-right sidenote note&quot;&gt;&lt;span class=&quot;footnote-p&quot;&gt;Oh, and maybe I will move these TLS certificates in a block storage or
something. That could be a good idea. &lt;/span&gt;
&lt;/span&gt;.
That will come in handy when things go wrong—as they inevitably do. I would
rather have the means to understand failures than guess my way around them.&lt;/p&gt;
&lt;p&gt;Did I ever mention I am an enthusiastic Opentelemetry convert?&lt;/p&gt;
        
      </description>
    </item>
    
    
    
    <item>
      <title>Installing a LUKS-Encrypted Arch Linux on a Vultr VPS</title>
      <link>https://soap.coffee/~lthms/posts/LUKSEncryptedVPS.html</link>
      <guid>https://soap.coffee/~lthms/posts/LUKSEncryptedVPS.html</guid>
      <pubDate>February 25, 2024</pubDate>
      <description>
        
        &lt;h1&gt;Installing a LUKS-Encrypted Arch Linux on a Vultr VPS&lt;/h1&gt;&lt;div id=&quot;tags-list&quot;&gt;&lt;span class=&quot;icon&quot;&gt;&lt;svg&gt;&lt;use href=&quot;/~lthms/img/icons.svg#tag&quot;&gt;&lt;/use&gt;&lt;/svg&gt;&lt;/span&gt;&amp;nbsp;&lt;a href=&quot;/~lthms/tags/self-hosting.html&quot; class=&quot;tag hover-peach&quot; marked=&quot;&quot;&gt;self-hosting&lt;/a&gt; &lt;span class=&quot;icon&quot;&gt;&lt;svg&gt;&lt;use href=&quot;/~lthms/img/icons.svg#tag&quot;&gt;&lt;/use&gt;&lt;/svg&gt;&lt;/span&gt;&amp;nbsp;&lt;a href=&quot;/~lthms/tags/vultr.html&quot; class=&quot;tag hover-coral&quot; marked=&quot;&quot;&gt;vultr&lt;/a&gt; &lt;/div&gt;
&lt;p&gt;I’ve been a happy customer of &lt;a href=&quot;https://vultr.com&quot; class=&quot;hover-periwinkle&quot; marked=&quot;&quot;&gt;Vultr&amp;nbsp;&lt;span class=&quot;icon&quot;&gt;&lt;svg&gt;&lt;use href=&quot;/~lthms/img/icons.svg#external-link&quot;&gt;&lt;/use&gt;&lt;/svg&gt;&lt;/span&gt;&lt;/a&gt; for three years now. For one, this little
corner of the Internet is hosted on one of their VPS, along with other services
I have self-hosted. Recently, I have decided to migrate to a new
VPS with more disk space and more powerful vCPUs.&lt;/p&gt;
&lt;p&gt;In this article, I describe how I have set up the host (an Arch Linux system on
a LUKS-encrypted partition) of &lt;code class=&quot;hljs&quot;&gt;jasmine&lt;/code&gt;&lt;label for=&quot;fn1&quot; class=&quot;sidenote-number margin-toggle&quot;&gt;&lt;/label&gt;&lt;input id=&quot;fn1&quot; type=&quot;checkbox&quot; class=&quot;margin-toggle&quot;&gt;&lt;span class=&quot;note-right sidenote note&quot;&gt;&lt;span class=&quot;footnote-p&quot;&gt;&lt;code class=&quot;hljs&quot;&gt;jasmine&lt;/code&gt; being my shiny new server—I have taken the habit to name
my servers and personal computers after Disney princesses. &lt;/span&gt;
&lt;/span&gt;. I had done a similar thing
for &lt;code class=&quot;hljs&quot;&gt;mulan&lt;/code&gt;, &lt;code class=&quot;hljs&quot;&gt;jasmine&lt;/code&gt;’s predecessor. Unfortunately, I didn’t take the time to
document &lt;code class=&quot;hljs&quot;&gt;mulan&lt;/code&gt; installation steps three years ago. Needless to say, I had
plenty of opportunities to regret this oversight while I painfully redicovered
how to make this work.&lt;/p&gt;
&lt;p&gt;Hopefully, most of what is in this write-up will stand the test of time and
remain true three years from now, when I migrate again 😅.&lt;/p&gt;
&lt;h2&gt;Booting on the Installation ISO&lt;/h2&gt;
&lt;p&gt;It is no surprise that Vultr does not propose an out-of-the-box LUKS-encrypted
Arch Linux installation in their OSes library. As a consequence, I had to boot
my brand new VPS on the &lt;a href=&quot;https://geo.mirror.pkgbuild.com/iso/&quot; class=&quot;hover-peach&quot; marked=&quot;&quot;&gt;Arch Linux ISO&amp;nbsp;&lt;span class=&quot;icon&quot;&gt;&lt;svg&gt;&lt;use href=&quot;/~lthms/img/icons.svg#external-link&quot;&gt;&lt;/use&gt;&lt;/svg&gt;&lt;/span&gt;&lt;/a&gt;.&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;In the Vultr dashboard, I visited the “ISOs” page (in the “Orchestrations”
menu) to add the latest Arch Linux installation ISO.&lt;/li&gt;
&lt;li&gt;Then, I went to the management panel of &lt;code class=&quot;hljs&quot;&gt;jasmine&lt;/code&gt;, more precisely in the
“Settings” tab. I visited the “Custom ISO” page, and attached the Arch Linux
ISO. This prompted a reboot.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;&lt;code class=&quot;hljs&quot;&gt;jasmine&lt;/code&gt; happily booted the image, but the resulting system was lacking a SSH
server to connect to. Instead, I used the “View Console” feature to get a shell
access to my VPS&lt;label for=&quot;fn2&quot; class=&quot;sidenote-number margin-toggle&quot;&gt;&lt;/label&gt;&lt;input id=&quot;fn2&quot; type=&quot;checkbox&quot; class=&quot;margin-toggle&quot;&gt;&lt;span class=&quot;note-left sidenote note&quot;&gt;&lt;span class=&quot;footnote-p&quot;&gt;It is noteworthy that I had to disable one of my Firefox extensions:
Vimium. Vimium provides me vim-like keystrokes in the browser, and the keys
I am using for motion (&lt;code class=&quot;hljs&quot;&gt;t&lt;/code&gt;, &lt;code class=&quot;hljs&quot;&gt;s&lt;/code&gt;, &lt;code class=&quot;hljs&quot;&gt;r&lt;/code&gt;, and &lt;code class=&quot;hljs&quot;&gt;n&lt;/code&gt;) were not forwarded to the
console. &lt;/span&gt;
&lt;/span&gt;.&lt;/p&gt;
&lt;p&gt;The Arch Linux installation ISO’s default keymap is qwerty, a keyboard layout I
am helpless with. The first command I always use in this situation is
&lt;code class=&quot;hljs&quot;&gt;loadkeys&lt;/code&gt;, which allows me to switch to the &lt;a href=&quot;https://bepo.fr&quot; class=&quot;hover-sky&quot; marked=&quot;&quot;&gt;&lt;code class=&quot;hljs&quot;&gt;fr-bepo&lt;/code&gt;&amp;nbsp;&lt;span class=&quot;icon&quot;&gt;&lt;svg&gt;&lt;use href=&quot;/~lthms/img/icons.svg#external-link&quot;&gt;&lt;/use&gt;&lt;/svg&gt;&lt;/span&gt;&lt;/a&gt; layout.&lt;/p&gt;
&lt;div class=&quot;markdown-alert markdown-alert-note&quot;&gt;&lt;p class=&quot;markdown-alert-title&quot;&gt;&lt;svg class=&quot;octicon octicon-info mr-2&quot; viewbox=&quot;0 0 16 16&quot; version=&quot;1.1&quot; width=&quot;16&quot; height=&quot;16&quot; aria-hidden=&quot;true&quot;&gt;&lt;path d=&quot;M0 8a8 8 0 1 1 16 0A8 8 0 0 1 0 8Zm8-6.5a6.5 6.5 0 1 0 0 13 6.5 6.5 0 0 0 0-13ZM6.5 7.75A.75.75 0 0 1 7.25 7h1a.75.75 0 0 1 .75.75v2.75h.25a.75.75 0 0 1 0 1.5h-2a.75.75 0 0 1 0-1.5h.25v-2h-.25a.75.75 0 0 1-.75-.75ZM8 6a1 1 0 1 1 0-2 1 1 0 0 1 0 2Z&quot;&gt;&lt;/path&gt;&lt;/svg&gt;Note&lt;/p&gt;&lt;p&gt;In this article, I am using &lt;code class=&quot;hljs&quot;&gt;;&lt;/code&gt; as the terminal prompt for &lt;a href=&quot;https://twitter.com/leostera/status/1740796853174596007&quot; class=&quot;hover-mint&quot; marked=&quot;&quot;&gt;ease of
copy/pasting&amp;nbsp;&lt;span class=&quot;icon&quot;&gt;&lt;svg&gt;&lt;use href=&quot;/~lthms/img/icons.svg#external-link&quot;&gt;&lt;/use&gt;&lt;/svg&gt;&lt;/span&gt;&lt;/a&gt;.&lt;/p&gt;
&lt;/div&gt;
&lt;pre&gt;&lt;code class=&quot;hljs language-bash&quot;&gt;; loadkeys fr-bepo
&lt;/code&gt;&lt;/pre&gt;
&lt;h2&gt;Remote Access to &lt;code class=&quot;hljs&quot;&gt;jasmine&lt;/code&gt;&lt;/h2&gt;
&lt;p&gt;The next step was to start a SSH server I could connect to using my terminal
emulator. A virtual console is nice enough, but when your muscle memory
commands you to hit &lt;code class=&quot;hljs&quot;&gt;Ctrl-W&lt;/code&gt; to delete the word before your cursor as soon as
you are connected to a shell, you don’t want to stay in a browser tab for
obvious reasons.&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;I set up a password for the &lt;code class=&quot;hljs&quot;&gt;root&lt;/code&gt; user using &lt;code class=&quot;hljs&quot;&gt;passwd&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;I modified the &lt;code class=&quot;hljs&quot;&gt;/etc/ssh/sshd_config&lt;/code&gt; file to let me connect with the &lt;code class=&quot;hljs&quot;&gt;root&lt;/code&gt;
user using a password (&lt;code class=&quot;hljs&quot;&gt;PermitRootLogin yes&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;I started the SSH server using &lt;code class=&quot;hljs&quot;&gt;systemctl start sshd&lt;/code&gt;.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;From there, I could connect &lt;code class=&quot;hljs&quot;&gt;jasmine&lt;/code&gt; from my terminal emulator.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;hljs language-bash&quot;&gt;&lt;span class=&quot;hljs-comment&quot;&gt;# after modifying /etc/hosts to add `jasmine` IP there&lt;/span&gt;
; ssh root@jasmine
&lt;/code&gt;&lt;/pre&gt;
&lt;h2&gt;Disk Partitions&lt;/h2&gt;
&lt;p&gt;Now that I was enjoying the comfort of my terminal emulator of choice
(&lt;a href=&quot;https://sw.kovidgoyal.net/kitty/&quot; class=&quot;hover-lemon&quot; marked=&quot;&quot;&gt;kitty&amp;nbsp;&lt;span class=&quot;icon&quot;&gt;&lt;svg&gt;&lt;use href=&quot;/~lthms/img/icons.svg#external-link&quot;&gt;&lt;/use&gt;&lt;/svg&gt;&lt;/span&gt;&lt;/a&gt;), the serious business could start. Moving forward, my next task was
to set up the disk layout and the filesystems.&lt;/p&gt;
&lt;div class=&quot;markdown-alert markdown-alert-warning&quot;&gt;&lt;p class=&quot;markdown-alert-title&quot;&gt;&lt;svg class=&quot;octicon octicon-alert mr-2&quot; viewbox=&quot;0 0 16 16&quot; version=&quot;1.1&quot; width=&quot;16&quot; height=&quot;16&quot; aria-hidden=&quot;true&quot;&gt;&lt;path d=&quot;M6.457 1.047c.659-1.234 2.427-1.234 3.086 0l6.082 11.378A1.75 1.75 0 0 1 14.082 15H1.918a1.75 1.75 0 0 1-1.543-2.575Zm1.763.707a.25.25 0 0 0-.44 0L1.698 13.132a.25.25 0 0 0 .22.368h12.164a.25.25 0 0 0 .22-.368Zm.53 3.996v2.5a.75.75 0 0 1-1.5 0v-2.5a.75.75 0 0 1 1.5 0ZM9 11a1 1 0 1 1-2 0 1 1 0 0 1 2 0Z&quot;&gt;&lt;/path&gt;&lt;/svg&gt;Warning&lt;/p&gt;&lt;p&gt;Did you know that Vultr VPS are using legacy boot and &lt;em&gt;not&lt;/em&gt; UEFI? I surely
did, three years ago, but sadly I had forgotten. So I did a first
installation setup with a EFI partition layout (GPT, &lt;code class=&quot;hljs&quot;&gt;/boot&lt;/code&gt; formatted with
VFAT, etc.), only to discover that my shiny system could not boot.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;To create an MBR partition system, &lt;code class=&quot;hljs&quot;&gt;fdisk&lt;/code&gt; is a nice enough tool. The disk is
exposed to the VPS under the name &lt;code class=&quot;hljs&quot;&gt;/dev/vda&lt;/code&gt;.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;hljs language-bash&quot;&gt;; fdisk /dev/vda
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Firstly, I created a new MBR.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;hljs&quot;&gt;Command (m for help): o
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Then, I created the &lt;code class=&quot;hljs&quot;&gt;/boot&lt;/code&gt; partition. To be sure I would not run out of space,
I allocated 3GBytes for it.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;hljs&quot;&gt;Command (m for help): n
Partition number (1-128, default 1): 
First sector (34-268435422, default = 2048) or {+-}size{KMGTP}: 
Last sector (2048-268435422, default = 268433407) or {+-}size{KMGTP}: +3G
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The root partition was to take the remaining space.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;hljs&quot;&gt;Command (m for help): n
Partition number (2-128, default 2): 
First sector (34-268435422, default = 6293504) or {+-}size{KMGTP}: 
Last sector (6293504-268435422, default = 268433407) or {+-}size{KMGTP}: 
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;I reviewed the resulting partition layout.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;hljs&quot;&gt;
Command (m for help): p

Disk /dev/vda: 128 GiB, 137438953472 bytes, 268435456 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xdbc3620d

Device     Boot   Start       End   Sectors  Size Id Type
/dev/vda1          2048   6293503   6291456    3G 83 Linux
/dev/vda2       6293504 268435455 262141952  125G 83 Linux
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;LGTM!&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;hljs&quot;&gt;Command (? for help): w
&lt;/code&gt;&lt;/pre&gt;
&lt;h2&gt;Filesystems&lt;/h2&gt;
&lt;h3&gt;&lt;code class=&quot;hljs&quot;&gt;/boot&lt;/code&gt; Partition&lt;/h3&gt;
&lt;p&gt;No need to be particularly fancy here. I formatted &lt;code class=&quot;hljs&quot;&gt;/boot&lt;/code&gt; with the &lt;code class=&quot;hljs&quot;&gt;ext3&lt;/code&gt;
filesystem. As far as I know, this is a reasonable default choice for this
partition.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;hljs language-bash&quot;&gt;; mkfs.ext3 /dev/vda1
&lt;/code&gt;&lt;/pre&gt;
&lt;h2&gt;LUKS-Encrypted Root Partition&lt;/h2&gt;
&lt;p&gt;I mostly followed the &lt;a href=&quot;https://wiki.archlinux.org/title/Dm-crypt/Encrypting_an_entire_system#LUKS_on_a_partition&quot; class=&quot;hover-peach&quot; marked=&quot;&quot;&gt;Arch Linux Wiki&amp;nbsp;&lt;span class=&quot;icon&quot;&gt;&lt;svg&gt;&lt;use href=&quot;/~lthms/img/icons.svg#external-link&quot;&gt;&lt;/use&gt;&lt;/svg&gt;&lt;/span&gt;&lt;/a&gt; for this. Generally speaking, the Arch
Linux Wiki is a wonderful source of information for anything related to Linux.&lt;/p&gt;
&lt;p&gt;I formatted &lt;code class=&quot;hljs&quot;&gt;/dev/vda2&lt;/code&gt; with &lt;code class=&quot;hljs&quot;&gt;cryptsetup&lt;/code&gt; to initialize the encryption scheme.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;hljs language-bash&quot;&gt;; cryptsetup -y -v luksFormat /dev/vda2
&lt;/code&gt;&lt;/pre&gt;
&lt;div class=&quot;markdown-alert markdown-alert-important&quot;&gt;&lt;p class=&quot;markdown-alert-title&quot;&gt;&lt;svg class=&quot;octicon octicon-report mr-2&quot; viewbox=&quot;0 0 16 16&quot; version=&quot;1.1&quot; width=&quot;16&quot; height=&quot;16&quot; aria-hidden=&quot;true&quot;&gt;&lt;path d=&quot;M0 1.75C0 .784.784 0 1.75 0h12.5C15.216 0 16 .784 16 1.75v9.5A1.75 1.75 0 0 1 14.25 13H8.06l-2.573 2.573A1.458 1.458 0 0 1 3 14.543V13H1.75A1.75 1.75 0 0 1 0 11.25Zm1.75-.25a.25.25 0 0 0-.25.25v9.5c0 .138.112.25.25.25h2a.75.75 0 0 1 .75.75v2.19l2.72-2.72a.749.749 0 0 1 .53-.22h6.5a.25.25 0 0 0 .25-.25v-9.5a.25.25 0 0 0-.25-.25Zm7 2.25v2.5a.75.75 0 0 1-1.5 0v-2.5a.75.75 0 0 1 1.5 0ZM9 9a1 1 0 1 1-2 0 1 1 0 0 1 2 0Z&quot;&gt;&lt;/path&gt;&lt;/svg&gt;Important&lt;/p&gt;&lt;p&gt;Be sure to save your LUKS passphrase somewhere, otherwise you are likely to
run in trouble at some point. I’ve personally added it to my Bitwarden.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;As-is, the partition cannot be used; it needs to be “open” with &lt;code class=&quot;hljs&quot;&gt;cryptsetup&lt;/code&gt;.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;hljs language-bash&quot;&gt;; cryptsetup open /dev/vda2 root
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;code class=&quot;hljs&quot;&gt;root&lt;/code&gt; here is the name given to the usable partition. More precisely,  after
this command the partition can be manipulated through the &lt;code class=&quot;hljs&quot;&gt;/dev/mapper/root&lt;/code&gt;
logical volume.&lt;/p&gt;
&lt;p&gt;Before, I had always used &lt;code class=&quot;hljs&quot;&gt;ext4&lt;/code&gt; for my root partition, but for a while I’ve
had an interest for alternative filesystems. And so, I decided out of the blue
to format &lt;code class=&quot;hljs&quot;&gt;/dev/mapper/root&lt;/code&gt; with &lt;code class=&quot;hljs&quot;&gt;btrfs&lt;/code&gt;&lt;label for=&quot;fn3&quot; class=&quot;sidenote-number margin-toggle&quot;&gt;&lt;/label&gt;&lt;input id=&quot;fn3&quot; type=&quot;checkbox&quot; class=&quot;margin-toggle&quot;&gt;&lt;span class=&quot;note-right sidenote note&quot;&gt;&lt;span class=&quot;footnote-p&quot;&gt;Time will tell if this was a mistake… The Internet seems to suggest
it might be one 😅. &lt;/span&gt;
&lt;/span&gt;. I like the idea of a
filesystem compressing its contents by default.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;hljs language-bash&quot;&gt;; mkfs.btrfs -d single -m single /dev/mapper/root
; mount -t btrfs -o defaults,noatime,compress=zstd /dev/mapper/root /mnt
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Finally, I could mount the two formatted partitions in &lt;code class=&quot;hljs&quot;&gt;/mnt&lt;/code&gt;.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;hljs language-bash&quot;&gt;; &lt;span class=&quot;hljs-built_in&quot;&gt;mkdir&lt;/span&gt; /mnt/boot
; mount /dev/vda1 /mnt/boot
&lt;/code&gt;&lt;/pre&gt;
&lt;h2&gt;Installing Arch Linux&lt;/h2&gt;
&lt;p&gt;I have installed the bare minimum with &lt;code class=&quot;hljs&quot;&gt;pacstrap&lt;/code&gt;. I personally prefer entering
in the chroot as soon as possible.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;hljs language-bash&quot;&gt;; pacstrap -K /mnt base linux
; genfstab -U /mnt &amp;gt;&amp;gt; /mnt/etc/fstab
; arch-chroot /mnt
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;In the rest of the article, I focus on the details that are directly related to
our main subject. That is, being able to boot a LUKS-encrypted Arch Linux VPS.
To that end, I have chosen to reproduce the exact setup I did three years ago
for &lt;code class=&quot;hljs&quot;&gt;mulan&lt;/code&gt;. It consists in configuring the initramfs to spawn an SSH server
(&lt;a href=&quot;https://tinyssh.org/&quot; class=&quot;hover-coral&quot; marked=&quot;&quot;&gt;tinyssh&amp;nbsp;&lt;span class=&quot;icon&quot;&gt;&lt;svg&gt;&lt;use href=&quot;/~lthms/img/icons.svg#external-link&quot;&gt;&lt;/use&gt;&lt;/svg&gt;&lt;/span&gt;&lt;/a&gt;) which let me enter the LUKS passphrase of &lt;code class=&quot;hljs&quot;&gt;/dev/vda2&lt;/code&gt; remotely.&lt;/p&gt;
&lt;h2&gt;&lt;code class=&quot;hljs&quot;&gt;sshd&lt;/code&gt;&lt;/h2&gt;
&lt;p&gt;Before configuring the boot process, it had to take a small detour and
configure the SSH server (&lt;code class=&quot;hljs&quot;&gt;sshd&lt;/code&gt; from OpenSSH) &lt;code class=&quot;hljs&quot;&gt;jasmine&lt;/code&gt; will start once it has
booted. The main reason is quite simple: I wanted tinyssh and &lt;code class=&quot;hljs&quot;&gt;sshd&lt;/code&gt; to use the
same host key.&lt;/p&gt;
&lt;p&gt;tinyssh is limited in terms of the key types it supports, but I personally
always default to Ed25519 so I’m fine with that.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;hljs language-bash&quot;&gt;ssh-keygen -f /etc/ssh/ssh_host_ed25519_key -N &lt;span class=&quot;hljs-string&quot;&gt;&apos;&apos;&lt;/span&gt; -t ed25519
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;I then modified &lt;code class=&quot;hljs&quot;&gt;/etc/ssh/sshd_config&lt;/code&gt; to ensure &lt;code class=&quot;hljs&quot;&gt;sshd&lt;/code&gt; would default to it.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;hljs language-bash&quot;&gt;&lt;span class=&quot;hljs-comment&quot;&gt;# in /etc/ssh/sshd_config, uncomment&lt;/span&gt;
HostKey /etc/ssh/ssh_host_ed25519_key
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;I used this opportunity to also (temporarily) allow login as root using a
password. Eventually, I will definitely remove this line from &lt;code class=&quot;hljs&quot;&gt;sshd&lt;/code&gt; config,
but while I’m configuring everything, it’s too convenient to bother with
another approach.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;hljs language-bash&quot;&gt;&lt;span class=&quot;hljs-comment&quot;&gt;# in /etc/ssh/sshd_config, add&lt;/span&gt;
PermitRootLogin &lt;span class=&quot;hljs-built_in&quot;&gt;yes&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;This line is only useful if &lt;code class=&quot;hljs&quot;&gt;root&lt;/code&gt; has a password, so I initialized one.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;hljs language-bash&quot;&gt;; passwd
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;I also enabled the &lt;code class=&quot;hljs&quot;&gt;sshd&lt;/code&gt; service, to let systemd start &lt;code class=&quot;hljs&quot;&gt;sshd&lt;/code&gt; automatically at
boot time.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;hljs language-bash&quot;&gt;; systemctl &lt;span class=&quot;hljs-built_in&quot;&gt;enable&lt;/span&gt; sshd
&lt;/code&gt;&lt;/pre&gt;
&lt;h2&gt;TinySSH&lt;/h2&gt;
&lt;p&gt;At this point, I knew I had only dealt with the easy bits. The hard part is
to be able the damn thing.&lt;/p&gt;
&lt;p&gt;I have always used Busybox-based initial ramdisk. While rereading the Arch
Linux Wiki, I discovered it was also possible to use systemd, which… well,
nobody is surprised by this, I guess. I am a happy adopter of systemd multiple
features in general, but for the particular setup I am targeting, using systemd
means &lt;a href=&quot;https://wiki.archlinux.org/title/Dm-crypt/Specialties#systemd_based_initramfs_(built_with_mkinitcpio)&quot; class=&quot;hover-peach&quot; marked=&quot;&quot;&gt;installing an AUR package&amp;nbsp;&lt;span class=&quot;icon&quot;&gt;&lt;svg&gt;&lt;use href=&quot;/~lthms/img/icons.svg#external-link&quot;&gt;&lt;/use&gt;&lt;/svg&gt;&lt;/span&gt;&lt;/a&gt;. So I decided to stick with Busybox.&lt;/p&gt;
&lt;p&gt;I gathered and installed the packages I needed for this to work.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;hljs&quot;&gt;; pacman -S tinyssh mkinitcpio-tinyssh mkinitcpio-netconf mkinitcpio-utils python3
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;And I modified &lt;code class=&quot;hljs&quot;&gt;/etc/mkinitcpio.conf&lt;/code&gt; to modify the &lt;code class=&quot;hljs&quot;&gt;HOOKS&lt;/code&gt; array.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;hljs language-diff&quot;&gt;&lt;span class=&quot;hljs-deletion&quot;&gt;-HOOKS=(base udev autodetect modconf kms keyboard keymap consolefont block filesystems fsck)&lt;/span&gt;
&lt;span class=&quot;hljs-addition&quot;&gt;+HOOKS=(base udev autodetect modconf block netconf tinyssh encryptssh filesystems keyboard keymap fsck)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;I tried to keep it minimal (hence the removal of &lt;code class=&quot;hljs&quot;&gt;kms&lt;/code&gt; for instance, which does
not seem very useful for a remote server considering its about setting display
resolution), and I borrowed from the &lt;code class=&quot;hljs&quot;&gt;/etc/mkinitcpio.conf&lt;/code&gt; file of &lt;code class=&quot;hljs&quot;&gt;mulan&lt;/code&gt; for
&lt;code class=&quot;hljs&quot;&gt;netconf&lt;/code&gt;, &lt;code class=&quot;hljs&quot;&gt;tinyssh&lt;/code&gt; and &lt;code class=&quot;hljs&quot;&gt;encryptssh&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;Turns out &lt;code class=&quot;hljs&quot;&gt;mkinitcpio-tinyssh&lt;/code&gt; is affected by &lt;a href=&quot;https://github.com/grazzolini/mkinitcpio-tinyssh/issues/10&quot; class=&quot;hover-sky&quot; marked=&quot;&quot;&gt;a long standing bug&amp;nbsp;&lt;span class=&quot;icon&quot;&gt;&lt;svg&gt;&lt;use href=&quot;/~lthms/img/icons.svg#github&quot;&gt;&lt;/use&gt;&lt;/svg&gt;&lt;/span&gt;&lt;/a&gt;, but
fortunately the patch to &lt;code class=&quot;hljs&quot;&gt;/usr/lib/initcpio/install/tinyssh&lt;/code&gt; worked like a
charm.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;hljs language-diff&quot;&gt;   local return_code=1
 
   if [ ! -d $destdir -a -x /usr/bin/tinyssh-convert ]; then
&lt;span class=&quot;hljs-deletion&quot;&gt;-      mkdir $destdir&lt;/span&gt;
&lt;span class=&quot;hljs-deletion&quot;&gt;-  fi&lt;/span&gt;
&lt;span class=&quot;hljs-deletion&quot;&gt;-  &lt;/span&gt;
&lt;span class=&quot;hljs-deletion&quot;&gt;-  if [ -s &quot;$osshed25519&quot; -a ! -s $destdir/.ed25519.sk -a ! -s $destdir/ed25519.pk -a -x /usr/bin/tinyssh-convert ]; then&lt;/span&gt;
&lt;span class=&quot;hljs-deletion&quot;&gt;-      tinyssh-convert -f $osshed25519 -d $destdir&lt;/span&gt;
&lt;span class=&quot;hljs-addition&quot;&gt;+      tinyssh-convert $destdir &amp;lt; $osshed25519&lt;/span&gt;
       if [ $? -eq 0 ]; then
           return_code=0
       fi
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;On my local machine, I generated a brand new SSH key using the Ed25519 key
type.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;hljs language-bash&quot;&gt;; ssh-keygen -t ed25519 -f jasmine
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;I copy/pasteed the content of &lt;code class=&quot;hljs&quot;&gt;jasmine.pub&lt;/code&gt; in &lt;code class=&quot;hljs&quot;&gt;/etc/tinyssh/root_key&lt;/code&gt; on
&lt;code class=&quot;hljs&quot;&gt;jasmine&lt;/code&gt;. It was also a good idea to delete the &lt;code class=&quot;hljs&quot;&gt;etc/tinyssh/sshkeydir&lt;/code&gt;
directory, before running the command to generate the initial ramkdisk.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;hljs language-bash&quot;&gt;; mkinitcpio -p linux
&lt;/code&gt;&lt;/pre&gt;
&lt;h2&gt;Grub&lt;/h2&gt;
&lt;p&gt;This is your periodic reminder that Vultr does not support UEFI and still
requires to the legacy MBR to boot. This means &lt;code class=&quot;hljs&quot;&gt;systemd-boot&lt;/code&gt; is out of the
picture. I always feel overwhelmed when i have to configure Grub. However,
three years ago I managed to make it work, so I decided to stick to this again.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;hljs language-bash&quot;&gt;; pacman -S grub
; grub-install --target=i386-pc /dev/vda
; grub-mkconfig -o /boot/grub/grub.cfg
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;code class=&quot;hljs&quot;&gt;/boot/grub/grub.cfg&lt;/code&gt; contains a lot of things. I looked for the line&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;hljs&quot;&gt;### BEGIN /etc/grub.d/10_linux ###
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;In the first &lt;code class=&quot;hljs&quot;&gt;menuentry&lt;/code&gt; after it, I modified one line&lt;label for=&quot;fn4&quot; class=&quot;sidenote-number margin-toggle&quot;&gt;&lt;/label&gt;&lt;input id=&quot;fn4&quot; type=&quot;checkbox&quot; class=&quot;margin-toggle&quot;&gt;&lt;span class=&quot;note-left sidenote note&quot;&gt;&lt;span class=&quot;footnote-p&quot;&gt;My initial thought was to install the &lt;code class=&quot;hljs&quot;&gt;intel-ucode&lt;/code&gt; package, and to
configure Grub to load Intel’s microcode upgrades at boot time. However,
since &lt;code class=&quot;hljs&quot;&gt;jasmine&lt;/code&gt; is a VPS, it does not directly access the physical CPU it
is running onto. As a consequence, this step is not necessary. &lt;/span&gt;
&lt;/span&gt;.&lt;/p&gt;
&lt;p&gt;First, the &lt;code class=&quot;hljs&quot;&gt;linux&lt;/code&gt; line.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;hljs&quot;&gt;        linux   /vmlinuz-linux root=/dev/mapper/root rw cryptdevice=UUID=7e58f868-529
a-4d70-bdfd-31317017c30b:root ip=&amp;lt;server ip&amp;gt;::&amp;lt;server gateway&amp;gt;:&amp;lt;server netmask&amp;gt;::eth0:none logle
vel=3 quiet
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The one is the most tricky.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code class=&quot;hljs&quot;&gt;root=/ev/mapper/root rw&lt;/code&gt; is the most straightforward. It means the root
filesystem in located in &lt;code class=&quot;hljs&quot;&gt;/dev/mapper/root&lt;/code&gt;, that is a logical volume.&lt;/li&gt;
&lt;li&gt;&lt;code class=&quot;hljs&quot;&gt;cryptdevice=UUID=7e58f868-529a-4d70-bdfd-31317017c30b:root&lt;/code&gt; tells the kernel
that the &lt;code class=&quot;hljs&quot;&gt;root&lt;/code&gt; logical volume is inside the encrypted (&lt;code class=&quot;hljs&quot;&gt;cryptdevice&lt;/code&gt;)
&lt;code class=&quot;hljs&quot;&gt;/dev/vda2&lt;/code&gt; partition. To get the &lt;code class=&quot;hljs&quot;&gt;UUID&lt;/code&gt; of &lt;code class=&quot;hljs&quot;&gt;/dev/vda2&lt;/code&gt;, I used &lt;code class=&quot;hljs&quot;&gt;blkid&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;
&lt;pre&gt;&lt;code class=&quot;hljs language-bash&quot;&gt;; blkid | grep UUID
/dev/mapper/root: UUID=&lt;span class=&quot;hljs-string&quot;&gt;&quot;0a44e4c7-10b7-4e85-8228-7663cf16a631&quot;&lt;/span&gt; UUID_SUB=&lt;span class=&quot;hljs-string&quot;&gt;&quot;0c9edc1c-31c0-417d-93e2-6f035c68b7cb&quot;&lt;/span&gt; BLOCK_SIZE=&lt;span class=&quot;hljs-string&quot;&gt;&quot;4096&quot;&lt;/span&gt; TYPE=&lt;span class=&quot;hljs-string&quot;&gt;&quot;btrfs&quot;&lt;/span&gt;
/dev/vda2: UUID=&lt;span class=&quot;hljs-string&quot;&gt;&quot;e9695de0-403c-4a6c-b8a2-9891dca2a60a&quot;&lt;/span&gt; TYPE=&lt;span class=&quot;hljs-string&quot;&gt;&quot;crypto_LUKS&quot;&lt;/span&gt; PARTUUID=&lt;span class=&quot;hljs-string&quot;&gt;&quot;dbc3620d-02&quot;&lt;/span&gt;
/dev/vda1: UUID=&lt;span class=&quot;hljs-string&quot;&gt;&quot;fa7a24d2-fc76-4af2-b26c-58548650253f&quot;&lt;/span&gt; BLOCK_SIZE=&lt;span class=&quot;hljs-string&quot;&gt;&quot;4096&quot;&lt;/span&gt; TYPE=&lt;span class=&quot;hljs-string&quot;&gt;&quot;ext3&quot;&lt;/span&gt; PARTUUID=&lt;span class=&quot;hljs-string&quot;&gt;&quot;dbc3620d-01&quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code class=&quot;hljs&quot;&gt;ip=&amp;lt;server ip&amp;gt;::&amp;lt;server gateway&amp;gt;:&amp;lt;server netmask&amp;gt;::eth0:none&lt;/code&gt; tells the
kernel enough information for tinyssh to be accessible from the Internet.
These three IPS can be found in the “Settings” tab of &lt;code class=&quot;hljs&quot;&gt;jasmine&lt;/code&gt; management
panel in the Vultr dashboard.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2&gt;Networking&lt;/h2&gt;
&lt;p&gt;There was only one step remaining before rebooting &lt;code class=&quot;hljs&quot;&gt;jasmine&lt;/code&gt; without the
installation ISO attached: configuring the system to ensure remote access. In
practice, it means enabling DHCP for the Ethernet interface (named &lt;code class=&quot;hljs&quot;&gt;ens3&lt;/code&gt; in
&lt;code class=&quot;hljs&quot;&gt;jasmine&lt;/code&gt;’s case).&lt;/p&gt;
&lt;p&gt;I did that by defining a &lt;code class=&quot;hljs&quot;&gt;.network&lt;/code&gt; unit in &lt;code class=&quot;hljs&quot;&gt;/etc/systemd/networkd&lt;/code&gt; to enable
DHCP so that &lt;code class=&quot;hljs&quot;&gt;jasmine&lt;/code&gt; can get its ip from Vultr DHCP server.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;hljs&quot;&gt;# /etc/systemd/network/50-ens3.network
[Match]
Name=ens3

[Network]
DHCP=yes
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;As a last touch, I enabled both &lt;code class=&quot;hljs&quot;&gt;systemd-networkd&lt;/code&gt; and &lt;code class=&quot;hljs&quot;&gt;systemd-resolved&lt;/code&gt;.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;hljs language-bash&quot;&gt;; systemctl &lt;span class=&quot;hljs-built_in&quot;&gt;enable&lt;/span&gt; systemd-networkd
; systemctl &lt;span class=&quot;hljs-built_in&quot;&gt;enable&lt;/span&gt; systemd-resolved
; &lt;span class=&quot;hljs-built_in&quot;&gt;ln&lt;/span&gt; -sf ../run/systemd/resolve/stub-resolv.conf /etc/resolv.conf
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;And I was good to go!&lt;/p&gt;
&lt;p&gt;I exited the chroot, came back to Vultr dashboard to detach the custom ISO from
&lt;code class=&quot;hljs&quot;&gt;jasmine&lt;/code&gt;, which prompted a reboot of the VPS. A look a the virtual console was
enough to confirm tinyssh had started and was waiting for incoming connection.
I edited my &lt;code class=&quot;hljs&quot;&gt;~/.ssh/known_hosts&lt;/code&gt; to remove &lt;code class=&quot;hljs&quot;&gt;jasmine&lt;/code&gt; now out-dated line, and
tried to connect to &lt;code class=&quot;hljs&quot;&gt;jasmine&lt;/code&gt;. tinyssh asked me kindly the passphrase for
&lt;code class=&quot;hljs&quot;&gt;/dev/vda1&lt;/code&gt;, before closing the connection.&lt;/p&gt;
&lt;p&gt;A new attempt at connection to &lt;code class=&quot;hljs&quot;&gt;jasmin&lt;/code&gt; using SSH led me to the expected
system. Hourray!&lt;/p&gt;
&lt;h2&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;Needless to say, while this article is meant to propose a linear story to its
readers, my personal journey was everything but. I had to reboot many times
into the installation ISO, to mount my root partition and tweak the
configuration until it finally worked.&lt;/p&gt;
&lt;p&gt;Hopefully, this article lists the various pitfalls I ran into, and how I fixed
them. It’ll be helpful for Future Me, and maybe to you before that.&lt;/p&gt;
        
      </description>
    </item>
    
    
  </channel>
</rss>
