The Problem

In that article, nearly at the very end, I was pointing out that a solution was needed for the XMPP traffic on non-Web ports (TCP and UDP other than 80 and 443) to be routed from a cloud VPS to my home server.

I tried some NGINX configuration using its module streams, but it didn’t work, so I will post here how I ended up solving that using Linux’ –the kernel itself– packet filter rules through iptables.

I learned and adapted my solution from the content of this article by Ryan Welch.

At the end of the post, though, I will also post how to configure NGINX to work with TCP and UDP general traffic, just in case we need it later, or maybe you want to try it and tell me what I did wrong.

Solution

As explained before, Linux (NOT GNU/Linux, but just “Linux”, the kernel) has network packet filtering capabilities built-in that can be configured to our needs.

To configure the VPS to do it, SSH into the machine and enable IP forwarding in the kernel:

sudo sysctl -w net.ipv4.ip_forward=1

This may or may not be persitent between boots, depending on your distribution. To be sure, uncomment the line net.ipv4.ip_forward=1 in /etc/sysctl.conf.

Then, you will be able to redirect ports and ranges with sentences like the following:

iptables -A PREROUTING -t nat -p tcp -i eth0 --dport 5222 -j DNAT --to-destination <Upstream VPN IP>:5222
iptables -A POSTROUTING -t nat -p tcp -d <Upstream VPN IP> --dport 5222 -j MASQUERADE

And, for ranges of ports:

iptables -A PREROUTING -t nat -p udp -i eth0 -m multiport --dports 49152:65535 -j DNAT --to-destination <Upstream VPN IP>:49152-65535
iptables -A POSTROUTING -t nat -p udp -d <Upstream VPN IP> -m multiport --dports 49152:65535 -j MASQUERADE

You’ll have to change the ports in this example, as well as putting your own remote IP address instead of the <Upstream VPN IP> placeholder.

Add all the forwarding rules corresponding to all the XMPP server ports, where <Upstream VPN IP> would be the Tailscale IP of my home server, the one we were calling potato in the previous article of the series):

iptables -A PREROUTING -t nat -p tcp -i eth0 --dport 5222 -j DNAT --to-destination <Upstream VPN IP>:5222
iptables -A POSTROUTING -t nat -p tcp -d <Upstream VPN IP> --dport 5222 -j MASQUERADE
iptables -A PREROUTING -t nat -p tcp -i eth0 --dport 5269 -j DNAT --to-destination <Upstream VPN IP>:5269
iptables -A POSTROUTING -t nat -p tcp -d <Upstream VPN IP> --dport 5269 -j MASQUERADE
iptables -A PREROUTING -t nat -p tcp -i eth0 --dport 5000 -j DNAT --to-destination <Upstream VPN IP>:5000
iptables -A POSTROUTING -t nat -p tcp -d <Upstream VPN IP> --dport 5000 -j MASQUERADE
iptables -A PREROUTING -t nat -p tcp -i eth0 --dport 3478 -j DNAT --to-destination <Upstream VPN IP>:3478
iptables -A POSTROUTING -t nat -p tcp -d <Upstream VPN IP> --dport 3478 -j MASQUERADE
iptables -A PREROUTING -t nat -p tcp -i eth0 --dport 3479 -j DNAT --to-destination <Upstream VPN IP>:3479
iptables -A POSTROUTING -t nat -p tcp -d <Upstream VPN IP> --dport 3479 -j MASQUERADE
iptables -A PREROUTING -t nat -p udp -i eth0 --dport 3478 -j DNAT --to-destination <Upstream VPN IP>:3478
iptables -A POSTROUTING -t nat -p udp -d <Upstream VPN IP> --dport 3478 -j MASQUERADE
iptables -A PREROUTING -t nat -p udp -i eth0 --dport 3479 -j DNAT --to-destination <Upstream VPN IP>:3479
iptables -A POSTROUTING -t nat -p udp -d <Upstream VPN IP> --dport 3479 -j MASQUERADE
iptables -A PREROUTING -t nat -p tcp -i eth0 --dport 5349 -j DNAT --to-destination <Upstream VPN IP>:5349
iptables -A POSTROUTING -t nat -p tcp -d <Upstream VPN IP> --dport 5349 -j MASQUERADE
iptables -A PREROUTING -t nat -p tcp -i eth0 --dport 5350 -j DNAT --to-destination <Upstream VPN IP>:5350
iptables -A POSTROUTING -t nat -p tcp -d <Upstream VPN IP> --dport 5350 -j MASQUERADE
iptables -A PREROUTING -t nat -p udp -i eth0 --dport 5349 -j DNAT --to-destination <Upstream VPN IP>:5349
iptables -A POSTROUTING -t nat -p udp -d <Upstream VPN IP> --dport 5349 -j MASQUERADE
iptables -A PREROUTING -t nat -p udp -i eth0 --dport 5350 -j DNAT --to-destination <Upstream VPN IP>:5350
iptables -A POSTROUTING -t nat -p udp -d <Upstream VPN IP> --dport 5350 -j MASQUERADE
iptables -A PREROUTING -t nat -p udp -i eth0 -m multiport --dports 49152:65535 -j DNAT --to-destination <Upstream VPN IP>:49152-65535
iptables -A POSTROUTING -t nat -p udp -d <Upstream VPN IP> -m multiport --dports 49152:65535 -j MASQUERADE

Test the set up, and when everything works, save the iptables rules by sudo iptables-save > /etc/iptables/rules.v4 so that these rules are applied at every boot of the VPS.

In his article, Ryan adds more considerations for things like congestion control and such. Check it out to know more and apply whatever suits your needs.

Bonus: NGINX With Generic TCP/UDP Traffic

Take this with a pinch of salt: as I stated at the beginning of the post, this didn’t work for me and my XMPP server of choice, Snikket. However, according to NGINX’ documentation, this is the way it is configured for handling non-Web traffic.

Install libnginx-mod-stream to support UDP and TCP reverse proxies.

Modify /etc/nginx/nginx.conf to add a stream block:

stream {
        include /etc/nginx/streams-enabled/*;
}

Then, create the directories following the existing schema NGINX has for webs: sudo mkdir -p /etc/nginx/streams-available /etc/nginx/streams-enabled

Create an /etc/nginx/streams-available/tcpudpservice.example.com.conf with rules like the following:

# TCP Port (non-HTTP traffic):
server {
        listen 5222;
        proxy_pass <Upstream VPN IP>:5222;
}

# ...

# UDP Port:
server {
        listen 3478 udp;
        proxy_pass <Upstream VPN IP>:3478;
}

# ...

# Ranges are accepted, too. In this case, note the
# use of $server_port in the proxy_pass sentence.

server {
        listen 49152-65535 udp;
        proxy_pass <Upstream VPN IP>:$server_port;
}

In the case of defining very large range of ports, such as the ranges that Snikket proposes for XMPP audio and video services in multi-users configurations, NGINX will throw an error due to having too many files open. Snikket expects to be able to establish 16384 UDP connections.

If you want to solve this instead of reducing the range of UDP ports, we must increase the LimitNOFILE parameter of NGINX in systemd by doing sudo systemctl edit nginx.service. This will open an editor for a drop-in extension to the service unit.

Write the following and save the file:

[Service]
LimitNOFILE=65535

Then, in the preamble of the /etc/nginx/nginx.conf (outside of any http, events or stream), add a line worker_rlimit_nofile with a value less than 65535 (30000 will work for the Snikket case), and increase the worker_connections inside events:

...
include /etc/nginx/modules-enabled/*.conf;

worker_rlimit_nofile 30000;

events {
        # worker_connections 768;
        # 3 times 8192 will suffice   
        worker_connections 24576;
        # multi_accept on;
}

...

As I said before, this configuration did not work with XMPP, but it can be useful for other use cases.

This text shows the steps to follow to connect an NGINX reverse proxy installed on a server A, and provide access through it to services hosted in a different machine, B. If the machines are in different locations or networks, we can solve the connectivity in several ways, one of them being a VPN.

In my personal case, this allows me to install NGINX on a VPS and use it to expose services hosted at my home server, “barcas”. To connect the two machines and allow NGINX to reach the upstream services, I have used Tailscale.

This post continues the series that started with “Changes to My Network, Sponsored by XMPP”. If you’re wondering why would we do that, or what value brings this configuration, please check read that one first.

Solution

You’ll need the following:

1. A VPS server rented from your provider of choice, to which you can access through SSH.
2. A server at your home, with at least one service accepting HTTP connections, let’s say, through the port 5000. (0.0.0.0:5000). I will call this host potato in the remainder of this post, just for the sake of brevity and readability.
3. A free Tailscale account. In general, we could do this with any VPN provider and even one self-hosted by us, preferably providing service based on WireGuard, as it is more performant than OpenVPN.
4. Optionally, or probably for this post to be meaningful, we should have a domain with their DNS records pointing to the VPS of point #1.

For the rest of the article I will assume that the hosts (the VPS and potato) have installed an Ubuntu LTS Server, such as 24.04.

If there were no issues you should have all this up and running in about 20 minutes.

Home Server

We’ll log into (the) potato, where our service is waiting for requests behind port 5000.

Install Tailscale and authenticate potato against our account. This will give potato an IP address within the VPN:

# Tailscale install commands are not reproduced, as they can 
# change at any point in time.

sudo tailscale up
# Follow the instructions

# Enable Tailscale as a system service, permanently running
sudo systemctl enable tailscaled --now

# Obtain and write down the IP address at the tailnet.
tailscale ip -4
100.85.67.22

At Tailscale’s dashboard we can see the host’s local name, and also the IP address returned by the last command.

We can also configure the host so that we don’t have to re-authenticate it into the VPN, disabling the expiry of the key used to authenticate the machine to Tailscale.

Optionally, we can also enable “MagicDNS” to be able to refer to the hosts by a domain name instead of by an IP address, with no local configuration to do in the machines.

Once the machine is already in Tailscale, we must open port 5000 in the potato’s firewall, if we hadn’t done so yet, so that it can receive requests from the VPS through that port.

sudo iptables -A INPUT -p tcp -m tcp --dport 5000 -j ACCEPT

Regarding this:

1. If your machine doesn’t have iptables-persistent installed, this is a good moment to install it so that we can save the firewall rules to be applied automatically at boot time.

sudo apt install iptables-persistent 
sudo iptables-save > /etc/iptables/rules.v4

2. I personally open the ports in all the network interfaces, not only in Tailscale’s.
   • As I won’t have that port open in the router, I am not doing anything that would leave me overly exposed.
   • This way I don’t need to always use Tailscale to perform tests on potato from home.
   • Each case is different: you should decide what to do with your firewall depending on how critical the service you’re configuring is, and how sensitive the data it manages is.

VPS

Connect to the VPS through SSH and install a reverse proxy. Even if we’re not going to have more than one web (80, 443) service listening, configuring, understanding and managing an HTTP reverse proxy is way easier than handle IP redirects and their relevant firewall DNAT rules.

I use NGINX.

sudo apt update && sudo apt install nginx

Install Tailscale and follow all the steps we did with potato. Write down the VPS’ IP address in “the tailnet”, for example 100.85.77.36.

Get a Let’s Encrypt certificate your your domain, with CertBot. For this:

• We must have already configured the DNS records of our domain, example.com, to point to the public IP of the VPS (not its IP at Tailscale) we’re working on.
• We may have to stop NGINX (sudo systemctl stop nginx) before, depending on our domain provider and their CertBot plugins available, if any.

sudo certbot certonly --standalone -d example.com
...
...
Successfully received certificate.
Certificate saved at: /etc/letsencrypt/live/example.com/fullchain.pem;
Key is saved at:      /etc/letsencrypt/live/example.com/privkey.pem;
...
...

Configure now the service in the reverse proxy, writing the IP of potato in Tailscale in the proxy_pass directive. For this, we’ll create a configuration file under /etc/nginx/sites-available/, for example example.com.conf for an https://example.com website.

upstream exampleweb {
        # Replace [IP or host name] for your 'potato' value
        # **in Tailscale**.
        # 100.85.67.22 in the post's example.
        server [IP or host name]:5000;
        keepalive 64;
}

server {
        server_name example.com;
        listen 80;

        location / {
                return 301 https://$host$request_uri;
        }
}

server {
        server_name example.com;
        listen 443 ssl http2;

        # Other SSL stuff goes here
        ssl_protocols TLSv1.2 TLSv1.3;
        ssl_ciphers HIGH:!MEDIUM:!LOW:!aNULL:!NULL:!SHA;
        ssl_prefer_server_ciphers on;
        ssl_session_cache shared:SSL:10m;
        ssl_session_tickets off;

        ssl_certificate     /etc/letsencrypt/live/example.com/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;

        location / {
                # The final `/` is important.
                proxy_pass http://exampleweb/;
                add_header X-Frame-Options SAMEORIGIN;
                add_header X-XSS-Protection "1; mode=block";
                proxy_redirect off;
                proxy_buffering off;
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_set_header X-Forwarded-Port $server_port;
                proxy_read_timeout 90;
        }
}

We’ll create now a symbolic link to this file under /etc/nginx/sites-enabled, test NGINX’ configuration and, if all goes well, reload it.

cd /etc/nginx/sites-enabled/
sudo ln -s ../sites-available/example.com.conf
sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
# If we had to stop NGINX to get the certificate, we start it up; 
# otherwise:
sudo systemctl reload nginx

At this point we should be able to interact with our service at potato through https://example.com.

Troubleshooting

• This thing doesn’t render anything, the service seems to be not replying. If there is activity in NGINX’ logs in the VPS (sudo tail -f /var/log/nginx/access.log), and the request is abandoned after a time out, there are a few possible causes for this in my limited experience. At potato:
  • The service upstream we want to access from NGINX is not working, and we have to start it up.
  • the service upstream is exposed through port 5000 of a specific network interface, instead of 0.0.0.0 (which means all network interfaces of potato), and other than Tailscale’s; for example, 127.0.0.1,
  • or we did something wrong in the firewall and the request is being rejected in that way.
• NGINX’ settings are OK (sudo nginx -t says 👍), but NGINX doesn’t start or dies when you reload the configuration. The most frequent cause is that NGINX can’t reach or find the upstream at potato, the one pointed by the proxy_pass directive.
  • We may have a typo in the proxy_pass,
  • NGINX may have tried to reach potato before Tailscale’s tunnel was up, for example if this happens upon a VPS reboot,
  • maybe Tailscale is down in any of the two hosts.
  • This cause can be checked by looking for a host not found in upstream in the output of sudo systemctl status nginx:

nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; preset: enabled)
    Drop-In: /etc/systemd/system/nginx.service.d
             └─override.conf
     Active: active (running) since Fri 2025-08-29 11:05:51 UTC; 3 days ago
       Docs: man:nginx(8)
    Process: 1154 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
    Process: 1156 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
    Process: 58676 ExecReload=/usr/sbin/nginx -g daemon on; master_process on; -s reload (code=exited, status=1/FAILURE)
   Main PID: 1157 (nginx)
      Tasks: 3 (limit: 1110)
     Memory: 46.9M (peak: 200.2M)
        CPU: 29min 57.349s
     CGroup: /system.slice/nginx.service
             ├─1157 "nginx: master process /usr/sbin/nginx -g daemon on; master_process on;"
             ├─1158 "nginx: worker process"
             └─1159 "nginx: cache manager process"

...
Sep 02 05:32:49 vps-hostingprovider nginx[58676]: 2025/09/02 05:32:49 [emerg] 58676#58676: host not found in upstream "100.85.77.22" in /etc/nginx/sites-enabled/example.com.conf>
Sep 02 05:32:49 vps-hostingprovider systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
Sep 02 05:32:49 vps-hostingprovider systemd[1]: Reload failed for nginx.service - A high performance web server and a reverse proxy server.
lines 1-28/28 (END)

• We get an HTTP Status Code 502. This means that, while NGINX is doing well, the upstream service we’re trying to reach at potato’s port 500 is throwing errors.
  • If you go and do a test from the potato and goes well, almost for sure this is a matter of trusted proxies. To describe the problem in depth and correctly is outside of the scope of this post, but in summary the point is that you must tell your application that the VPS that is hosting the reverse proxy, a machine other than potato, is trustworthy. This is done by configuring VPS’ IP address in Tailscale as a trusted proxy. Check your specific web application logs to confirm this hypothesis.
  • This is not a problem, but a rather interesting security feature.
  • For example, this happened to me with both my Mastodon instance and Nextcloud. In both cases, each application’s documentation had the solution for it: Mastodon – trusted proxies, Nextcloud – trusted proxies
  • Depending on how the service we’re trying to fix is written, you may have to set a trusted proxy using an IP or you may be able to set a domain name, or either. Don’t desist if it doesn’t work at first try.

In this one, I’m going to tell you what those problems are, why they arise, why I want to solve them, and how I’ve solved them, but from a design point of view for now. You won’t find a set of commands to solve all this, but rather some context about what I’m trying to solve; detailed instructions will come in the future.

Until now, all my services hosted “underneath my printer” were Web services, that is, based on communications through ports 80 and 443 (HTTP and HTTPs). These ports were enough to connect all my services with my devices or my browser, or with other servers in the case of my Mastodon instance. To serve so many services behind the same ports, I use NGINX as a reverse proxy:

Unlike my other services, XMPP uses more TCP and UDP ports than 80 and 443: 5000, 5222, 5269, etc. In the case of Snikket, as many or all other alternatives might as well, ports 80 and 443 are also used to provide a web control panel and to manage users and chat rooms.

In principle, the easiest configuration for installing Snikket would be as follows:

• Configure ports 80 and 443 in NGINX to be able to use Snikket’s user and conversation control panel.
• Configure direct access to all other Snikket ports: 5222, 5269, etc. Here is the complete list. As they are not shared amongst other services, reverse-proxying them is not needed.
• Configure the DNS for my XMPP service with my domain provider, which is Cloudflare.

This configuration, which would look good at first glance, looks like the following:

However, there is a problem that is not obvious, and the above configuration does not work out of the box. The Cloudflare security features that are available to free accounts in the form of “the Cloudflare proxy” only support ports 80 and 443. With the Cloudflare proxy enabled, all non-web traffic (80 and 443) was discarded, so none of the ports involved in XMPP conversations reached my server.

Disabling the Cloudflare proxy for the Snikket subdomain configuration makes everything work, and looks like this:

Disabling Cloudflare’s proxy is not ideal because it exposes the server’s IP address to the rest of the internet. My server is at home, so a DDoS attack that, by pure chance, included the IP address assigned to me by my internet service provider would bring down my entire home network. That would prevent us from working or studying from home and would probably cause problems with my service provider.

An alternative to this is to hire a cheap server from any cloud hosting provider (a VPS) and move all my NGINX configuration to it, connecting this cloud server to my home server via a VPN (the free Tailscale account works very well and is more than enough).

• The cloud hosting provider would protect my server because the exposed IP would be theirs, not my home’s, and these types of companies protect their infrastructure from attacks, at least at the network level. Behind the XMPP ports there is no content, just a messaging service that also encrypts traffic, so there is no problem with losing the security and anti-bot layer that Cloudflare provides for these ports.
• Connecting the VPS to my server via a VPN is perfect for this case because it hides the IP address of my home network, saving me from having to configure it in the DNS control panel and also from having to open ports on my router. In addition, for all traffic between the VPS and my server, a layer of encryption would be applied to protect it in case it was not already encrypted.

Why move only the NGINX configuration and not all services? For reasons of resources and price: to run NGINX and a series of firewall rules, the requirements we need are minimal. A server with 1 GB of RAM and a CPU is more than enough, and that costs between 3 and 6€ per month. If I moved everything², I would need between 8 and 16 GB of RAM, much more storage, CPU power, a backup solution, etc. All of this would cost me around 40€ or more per month. Since I have a server at home where I can run the services, moving everything to the VPS would be an unnecessary expense in my case.

So, it seems that hiring a simple VPS to have NGINX configured there and connecting it to the home server via Tailscale would solve my problems at a very low cost (3 to 6€ per month).

The only additional complication with this method is that we need to have some mechanism to redirect traffic from the XMPP ports (several ports such as 5222, 5269, etc. on TCP and UDP) from the server we hire to our server (at home or wherever). In the end, I solved it by configuring the VPS to do IP forwarding and then setting up a series of DNAT³ rules on its firewall.

We will look at the configuration of all the elements in more detail in future posts.

These are the podcasts where you can listen to my rambling:

• Sobre la marcha is a podcast about technology, various things that sparked my curiosity, and sometimes life in Australia.
  • It’s available wherever you listen to your podcasts.
  • You can also subscribe manually using its feed.
  • It’s got its own fediverse account at: @sobrelamarcha@podcast.gvisoc.com.

Devices, services, applications and operating systems that are useful and respectful to us. Those that don"t get in the way, instead of fighting to keep our attention when the work is already done. Things that are at our service and under our control, and not the other way around.

All this content is published under a Creative Commons CC BY-SA 4.0

This site uses no cookies. No tracking of you, whatsoever:

Find me on the fediverse at https://fedi.gvisoc.com/@gabriel.

This post is about the rationale of installing my own personal Mastodon instance at home, not in a VPS or in a cloud provider.

In it, you will find hints on how to do it, but not a detailed recipe to overcome all the tasks, as such is not the goal of the article.

I’m not going to relate all my trajectory on Mastodon because I want to make things short.

The idea is that Mastodon is an application with non-trivial machine requirements, and the Cloud is Expensive.

Mastodon is Heavy

Let’s see where Mastodon’s footprint comes from, from my very limited understanding of things and my limited experience:

• Mastodon is a Ruby on Rails application that has at least three independent processes, or services, as shown in systemd:

• The web application, that offers a user-facing applicaton. In it, we need to do things in order for the application to fetch information. It’s very passive.

• The streaming API, which does things in the background and streams things to the clients, both web and mobile. An example of this is sending the notifications to the web application, as they happen.

• The Sidekiq worker, which runs in the background and implements the federation, or so I believe, in the sense of replicating data in and out of our instance to the other known instances.[1]

• It depends also on a Postgresql database.

• It uses Redis, an in-memory cache backed into the disk, to support Sidekiq processes.

• It uses A Lot of disk space, as a result of two things:

• The activity and uploaded media of the instance’s users.

• The federated content coming from other instances.[2]

• You will need several other smaller services hanging around, like an NGINX to act as a web server and gateway.

In my experience, if we use a Digital Ocean Droplet of 2GB RAM and 2 CPU, the Mastodon services will have to be stopped for us to be able to run maintenance tasks without Digital Ocean killing them. 4 GB and 2 CPU will be the minimum suitable droplet, and a disk space of less than 60 GB will be tight.

And that’s just a minimal installation. Add HA (High Availablity) on those components, and you’ll have to put much more hardware and extra networking services.

That is to say, the beast is heavy.

The Cloud is Expensive

One of the aforementioned droplets would cost you US$24/month, which in Australian dollars is about A$40/month.

That’s a lot of money: A$480 a year. And the droplet is a dumb and isolated virtual machine that doesn’t do anything for you with regards to maintenance and administration.

• Do you want application backups? They are charged separately.
• Do you want machine snapshots? They are charged separately.
• Do you want alerting when the droplet is down or slow? Charged separately.
• Are you running things in HA? Do you need extra network things on top of the droplets you’re adding? Charged separately!

And, of course, consider your labour. You must keep not only Mastodon, but also the droplet’s operating system up to date and finely tuned. That’s a lot of work, on top of being a lot of money.

What it Takes

Here’s the thing:

• In little over 2 months, you would have spent more money than the cost of an ARM single board computer, that exceeds the above requirements (like the Raspberry Pi[3] 4B, 4GB) and the electricity cost of running it for that time.
• In another 2-3 months, you can add to the pack an SSD drive of hundreds of GB, and externalise all the disk usage to it.
• From there, you are effectively saving a lot of money that you could use to support the projects around the network, or the instances you sympathise with. Or ☕.

The only things that you need are:

• A small computer with enough resources.

• 4 GB of RAM, 60 GB or more of storage, 4 core is enough to get you up and running.

• In the medium term, if you’re using an SD card for the system, attach some proper storage for high I/O load for performance and reliability. Small SSD and USB 3.0 cases for them are very cheap.

• A domain managed by a domain provider that supports DDNS[4],

• Install ddclient (from its GitHub repository) on your machine [5],

• A router that supports NAT and that allow you to expose services at ports 80 and 443.[6]

• A decent internet connection.[7]

🌱 Preferably, choose a low power device. Don’t run Mastodon 24/7 on your forest-burning gaming PC 🌷

It won’t support more than a few friends and family members. But if your instance is personal, or if your crew is limited, you’ll be fine. The amount of extra work that you need to do by having the instance at home, compared to having it in a provider like Digital Ocean, is negligible.

And it’s a lot cheaper.

A federated application is not a distributed application. While in a distributed application all instances try to discover all other instances and be able to communicate with each other, a federated application is passive. The instances will communicate only if they know each other. How an instance can know other instances? In Activity Pub, only through the users’ actions. If I decide to follow you at https://mastodon.social from my https://fedi.gvisoc.com, then those two instances will begin talking to each other. From that moment, they will talk a lot, but the federation details are out of the scope of this article. Just let’s assume that there is a lot of media files and data going back and forth. ↩︎

Mastodon caches a lot of information, including, and especially, pictures and videos from other instances. It will cache media from all the accounts you follow across any instance, and from all the known, but not followed accounts, such as those whose posts appear in your timeline as someone you actually follow boosts them. From all those, it will cache avatars, headers, emojos, and all the attachments of the posts that are visible in your timelines. Visible doesn’t mean viewed: those resources will be cached in your instance’s disk even if you never see them. And I’m sure this doesn’t end there, but I am not 100% sure of all the implications of the federation and ActivityPub. ↩︎

Although the people around the Raspberry Pi is not very popular at the moment, the community around these machines is huge and I’m sure many of us will find some at the bottom of a drawer. ↩︎

I have all my domains at Cloudflare, who are ethically very slow and tone deaf as well (please suit yourself with a web search engine for this one claim), but very cheap. And in the free tier of Cloudflare, you can update the DNS record for your machine through their API, which is spoken by ddclient. ↩︎

Check my configuration (last reviewed at the time of this writing) in this public Gist at my GitHub account. ↩︎

I haven’t found any router that wouldn’t allow you to do so. Even the provided by the ISPs can be configured in such way (I am using one of those right now). ↩︎

I am getting very good results with an internet connection that offers 20 Mbps upload speed. This includes not only me accessing information on the go, but also the other instances to find my information quickly: there are posts that get reactions almost immediately after they are published… in those rare cases I publish anything interesting. ↩︎

• The integrity of the software we have just downloaded.
• The authenticity of the package.

Checking these two things will ensure that the download went well, and that the software is authentic (and not malware, for example).

For the impatient, here is the process assuming that we have downloaded:

• A software package we want to use, in the example software.pkg
• A file containing the reference hash; in the example, software.pkg.sha256
• A file containing the signature of the Software Author; in the example, software.sig
• The public keys of the Software Author; in the example, author.gpg

    $ sha256sum -c software.pkg.sha256          # 1. Check the download integrity
    $ gpg --import author.gpg                   # 2a) Import downloaded public keys
    $ gpg --keyserver hkp://<server> \
                   --search-keys <shortID>      # 2b) Import public keys from a server
    $ gpg --verify software.sig software.pkg    # 3) Verify signatures
    $ gpg --list-public-keys --with-sig-check   # 4) If needed, check web of trust

These commands, why we run them, how they work, and how we should understand them, are discussed next. All the commands are Linux-specific, but information on Windows equivalents can be found in plenty of sites and I will probably write other post in the next few days.

Checking the Integrity of a Download

We want to check the integrity of a downloaded package to ensure that there were no transmission errors, and that nobody between our computer and the server exchanged the package for something else. We are checking that the download went well and we obtained whatever there was in the server: no more, and no less.

To do this, we verify the checksums of the files. Checksums are representations of the content smaller than the content itself, calculated by algorithms that ensure a stupidly small probability of two different files producing the same result. Some of those algorithms are members of a family called “hashes”. Most useful checksums, and hashes particularly, always have the same length.

If there was a single bit of the file that got changed over the wire, the checksum calculated in our computer over the corrupted file would be different than the checksum of the server’s file: the checksum wouldn’t verify.

In order to calculate a checksum we only need the file itself, and the right program installed.

For example, if we go to the downloads page for VSCodium, a fully Open Source version of Visual Studio Code, we can see that for every package there is an extra file, ending in .sha256:

codium-1.55.2-1618361370.el8.x86_64.rpm
codium-1.55.2-1618361370.el8.x86_64.rpm.sha256

This extra file contains the reference checksum calculated over the server’s file, and that is the exact same value that we should obtain in our computer from the downloaded copy.

The idea is to download both files to the same directory, and run the following command sha256sum -c over the checksums file:

$ sha256sum -c codium-1.55.2-1618361370.el8.x86_64.rpm.sha256 
codium-1.55.2-1618361370.el8.x86_64.rpm: OK

What happened here is that the program sha256sum checked (-c) that the reference checksums present in the file codium-1.55.2-1618361370.el8.x86_64.rpm.sha256 could be calculated locally using the downloaded file. The .sha256 file content is the following:

75b9f844c22c98d4da33e64b6c7c49e8b4d0d94a438e4f8ce976e7e54b40a682  codium-1.55.2-1618361370.el8.x86_64.rpm

That line is a “properly formatted SHA256 checksum line”, which is built by a SHA256 hash, two (2) spaces, and the name of the file the hash was calculated from in the server. From there, the check sha256sum -c performs is just to find a file that has the same name as in that line, calculate its SHA256 hash, and compare it with the first part of the line.

In this case the verification was complete and correct.

A file like that can have plenty of lines, not just one, and sha256sum -c will run all the checks in a batch.

Another possibility can be to have only the file to download, and the SHA256 line in the website, for you to see. In that case, we’d download the file and get the SHA256 locally by executing the sha256sum over the downloaded file, without -c:

$ sha256sum codium-1.55.2-1618361370.el8.x86_64.rpm
75b9f844c22c98d4da33e64b6c7c49e8b4d0d94a438e4f8ce976e7e54b40a682  codium-1.55.2-1618361370.el8.x86_64.rpm

That would give us the previously mentioned line, and we should just visually compare the hashes.

When the checksums are correct, we are sure that the download went well, but nothing else. It’s up to us whether or not we trust that the publisher of the software is publishing authentic software, made by its true authors.

Checking the Authorship and Authenticity of the Software

If we don’t trust the Software Publisher, that is to say, the website or the server we downloaded the package from, then checking the integrity of the download is not enough. The Publisher could have offered anything else in that file, and calculate a perfect valid reference checksum for a counterfeit package.

In such case of distrust, we want to check the software authorship, and therefore its authenticity. We can do that by obtaining a cryptographic signature of the package, created by the Software Author, and verifying it cryptographically.

There are many cryptographic signature mechanisms and algorithms, but here we will focus on signatures produced using asymmetric cryptography because they are the ones used most often, if not the only ones used to sign software.

In most asymmetric cryptography frameworks like PGP and GPG, the signature works as follows:

• A Software Author would have two keys: Private or Secret key, and Public Key. They are generated in the same process and they are inextricably linked.

• The Public Key is generally available; the Secret Key is never shared with anyone.

• Using their Secret Key, the software author produce a cryptographic signature, which in the case of PGP (or GPG), it consists in:

• calculating a hash, similar that the ones we’ve just discussed,

• signing the hash using the Secret Key and a cryptographic singing algorithm.

The idea of signing a hash and not the file itself is that a file can have a size from a few bytes to Gigabytes or Terabytes. By signing the hash, which is much smaller and has a fixed length, the result is produced in a predictable amount of time.

As the Secret and Public Keys are an inextricable pair, only the Public Key of the software Author will verify correctly the signature produced with the corresponding Secret Key. As the Public Key is made available, everybody can verify the signatures using the right software.

Here’s what we need to do to check a signature with a real example, a Manjaro GNU/Linux ISO file.

First: Download the software from an, in principle, trusted server. In this case, we will download a file named manjaro-xfce-21.0.2-210419-linux510.iso from the Manjaro XFCE download page. This file has an associated GPG signature that we will also download, manjaro-xfce-21.0.2-210419-linux510.iso.sig, and a SHA1 hash printed in the web page, 7d9d5d886188ebb0a05c9ceeabdb068fb2544feb.

Second: Check the hash as we discussed earlier in this post, to make sure that the package is uncorrupted. In this case we need to compute the hash and compare it visually with the reference one, present in the download page.

$ sha1sum manjaro manjaro-xfce-21.0.2-210419-linux510.iso
7d9d5d886188ebb0a05c9ceeabdb068fb2544feb  manjaro-xfce-21.0.2-210419-linux510.iso

It verifies, so let’s go with the GPG signature.

Third: For us to be able to verify a GPG signature, we first need to obtain the public keys of the Software Author. In this case, Manjaro offers two sets of public keys, the general Manjaro ones, and Philip Müller’s, “just in case we didn’t trust the server where Manjaro made the general keys file available”. That statement is a bit of food for thought that we’ll discuss later, and a good example because it shows two ways for obtaining public keys.

First, we download and import the Manjaro general keys.

$ wget https://[url to the manjaro keys]/manjaro.gpg -O manjaro.gpg
$ gpg --import manjaro.gpg

On April 28th 2021 this imported 22 keys into the keyring, allowing us to verify packages signed by several people and machines at Manjaro.

To download Philip Müller’s keys from a public PGP Key Server, use the following:

$ gpg --keyserver hkp://pool.sks-keyservers.net --search-keys 11C7F07E
gpg: data source: http://194.95.66.28:11371
(1) Philip Müller (Called Little) <REDACTED@manjaro.org>
      2048 bit RSA key CAA6A59611C7F07E, created: 2012-05-05
Keys 1-1 of 1 for "11C7F07E".  Enter number(s), N)ext, or Q)uit > 1
gpg: key CAA6A59611C7F07E: "Philip Müller (Called Little) <REDACTED@manjaro.org>" not changed
gpg: Total number processed: 1
gpg:              unchanged: 1

Note that Enter number(s), N)ext, or Q)uit > is an actual prompt and we have to select the key, 1 in this case. Pay attention to the CAA6A59611C7F07E part: is an identifier for the key. Another thing to note is that the string we used to search the key (11C7F07E) is part of the key identifier. The lower part of a key’s ID is often referred to as “short identifier of the key” for obvious reasons.

The keys can also be searched by some other identificative fields, for example an email (real email omitted for privacy reasons!):

$ gpg --keyserver hkp://pool.sks-keyservers.net --search-keys "REDACTED@manjaro.org"

gpg: data source: http://194.95.66.28:11371
(1) Philip Müller (Called Little) <REDACTED@manjaro.org>
      2048 bit RSA key CAA6A59611C7F07E, created: 2012-05-05
...

The key ID identifies the pair of keys, not just the Public Key. It is important to know the ID of key pair, as it is the only way to really identify a key used to produce a signature and therefore the key needed to verify it. There can be many keys for a single email address, not all of them legit. The way to know the key needed to verify a signature is try to verify it without importing any key, as it will output the ID of the key pair that was used:

gpg: Signature made Mon 19 Apr 2021 18:47:28 AEST
gpg:                using RSA key 3B794DE6D4320FCE594F4171279E7CF5D8D56EC8
gpg: Can't check signature: No public key

Fourth: If the download of the iso verified the hash on previous steps, then check the signatures:

$ gpg --verify manjaro-xfce-21.0.2-210419-linux510.iso.sig  manjaro-xfce-21.0.2-210419-linux510.iso
gpg: Signature made Mon 19 Apr 2021 18:47:28 AEST
gpg:                using RSA key 3B794DE6D4320FCE594F4171279E7CF5D8D56EC8
gpg: Good signature from "Manjaro Build Server <build@manjaro.org>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 3B79 4DE6 D432 0FCE 594F  4171 279E 7CF5 D8D5 6EC8

In this case, the signing key was one of the keys present in the file manjaro.gpg, the one for the Manjaro Build Server with identifier 3B794DE6D4320FCE594F4171279E7CF5D8D56EC8. The verification process was correct, so we need that something called Manjaro Build Server is the creator of the ISO we’ve just downloaded.

Fifth: If we want to check the authenticity of such Manjaro Build Server dude, we can do so by validating its key using Philip Müller’s key (provided we trust him). That can be done by check the web of trust for Manjaro Build Server key, and see whether Philip is part of that web of trust. For this one, issue the following command:

$ gpg --list-public-keys --with-sig-check

Within the output of this command we can see all the public keys we have in our keyring, and who has signed them. Signing a Public Key with your own secret key is, in this context, the ultimate vouching act. In the case of the Manjaro Build Server, which we can identify by looking at its identifier, we can see that within its signers there is Philip Müller (the ID is the same previously imported CAA6A59611C7F07E):

pub   rsa3072 2020-10-28 [SC] [expires: 2022-10-28]
      3B794DE6D4320FCE594F4171279E7CF5D8D56EC8
uid           [ unknown] Manjaro Build Server <build@manjaro.org>
sig!         8238651DDF5E0594 2020-10-29  Matti Hyttinen <REDACTED@manjaro.org>
sig!         DAD3B211663CA268 2020-10-29  Bernhard Landauer <REDACTED@manjaro.org>
sig!         084A7FC0035B1D49 2020-10-29  Dan Johansen (Manjaro) <REDACTED@manjaro.org>
sig!         CAA6A59611C7F07E 2020-10-29  Philip Müller (Called Little) <REDACTED@manjaro.org>
sig!3        279E7CF5D8D56EC8 2020-10-28  Manjaro Build Server <build@manjaro.org>
sub   rsa3072 2020-10-28 [E] [expires: 2022-10-28]
sig!         279E7CF5D8D56EC8 2020-10-28  Manjaro Build Server <build@manjaro.org>

Therefore, if we trust Philip Müller, we can trust the downloaded ISO signature because Philip has endorsed the Manjaro Build Server by signing its key with his own.

Package Managers

The above process is what we need to do for the software we download manually, from the web.

Package Managers for GNU/Linux, as apt, pacman or yum among others, for Windows as Chocolatey, and for macOS as homebrew, should do all the avobe automatically, transparently and by default. Check the documentation of the system and package manager you use (and trust) to check specific details.

This is All About Who We Trust

If we pay attention to how this works, we’ll se that all this is about trust. There is no technique that would give us 100% certainty of the authenticity of a software package. We need to judge if we trust the Software Publisher, and if we trust the Software Author, and if we believe their public keys really belong to them –the actual persons or companies. Check after check, we are traversing a web of trust that is effective as long as we can find a point we can trust.

Further Reading

A good introduction of the web of trust, and links to further material, can be found in the Wikipedia.

More on the cryptographic suite used by most software publishers, authors and distribution channels can be found in the GNU Privacy Handbook.

I came across the cause of the problem and its solution; keep reading to know more.

The Symptoms

At some point you need to activate User Home service as per the requirements of some other package (for example, Synology Mail), or just because you want to work through SSH, or any other valid reason. Although you activate it, the dependent package keeps asking you to enable it. Or, if you just enabled it to use home folders via SSH, although you land on a home directory when you log in through SSH, you can’t see it through DS File or File Station:

In the above picture you should see a folder for each user under homes, plus a home directory that would host your files (it’s like a symbolic link dynamically created when you log in to the NAS through DSM or DS File).

If you try to disable and re-enable User Homes Service, DSM shows a sad and brief “Operation Failed” error pop up, and in Log Center you just see a useless message “User home service disable failed” like the one below:

The Problem

The problem is that the user homes’ hosting folder, named homes, should be created in one of the main volumes inside the disk array, for instance, /volume1/homes, and a symbolic link back should be created at /var/services/homes. Instead, in this case the homes folder was created in that location directly, as a regular folder (/var/services/homes). We can check it by doing SSH to the NAS:

user@hostname:~$ pwd
/var/services/homes/user
user@hostname:~$ cd ../..
user@hostname:/var/services$ ls -lda homes
drwxr-xr-x 6 root root 4096 Feb 5 08:00 homes 

The expected result of a successful User Home service activation is this one instead:

user@hostname:/var/services$ ls -lda homes
lrwxr-xr-x 6 root root 4096 Feb 5 08:10 homes -> /volume1/homes 

Not only you can see where it points, but also the l flag on the folder permissions, meaning link, instead of a d from directory.

It is important to note that, although the user still has its home folder that works, the user homes created this way are outside the disk array and in the operating system’s filesystem. This fact have several consequences, all of them dangerous for our files, for DSM’s stability, and therefore for our NAS reliability.

In the first place, it causes almost every Synology package that depends on the User Home service to fail. Synology packages usually operate with directories under the volumes created inside the disk array, not under /var/services/, and that’s why Synology Mail didn’t detect the user homes directories and kept asking for User Home service to be enabled. As with Synology Mail, if the NAS is kept in this situation, the files in those directories won’t be accessible by any backup or synchronisation service, and something as simple as updating DSM can have all the files under /var/services/homes/ wiped, and lost forever.

Another consequence is that all user home directories will be lost in the event of machine upgrades or replacement, meaning that in the event of moving the disks to a new NAS, those user home directories won’t be carried over.

A much worse scenario would be the one where we run out of space in the system’s partitions because we put too many files under our home directories, phisically located under /var, causing DSM to crash and, probably, to have problems to boot.

The Solution

Solving the problem is as easy as login in through SSH to back up our files and delete the /var/services/homes folder:

user@hostname:~$ cd /var/services
user@hostname:/var/services$ tar cfz /volume1/[dest. folder]/homes-backup.tgz homes
user@hostname:/var/services$ sudo rm -rf homes

After that, we create a symbolic link to /volume1/homes:

user@hostname:/var/services$ sudo ln -s /volume1/homes homes
user@hostname:/var/services$ ls -lda homes
lrwxr-xr-x 6 root root 4096 Feb 5 08:10 homes -> /volume1/homes

Finally, we log into the DSM web interface, to disable without error and re-enable the User Home service.

No services should fail on activation, and the DSM File Station should show all users’ home directories; time to restore the previous users’ home folders contents if needed.

Contrary to the mainstream trend of such a disablement and proceeding with an outdated bootstrap, security-wise, many distributions do support Secure Boot since a while ago.

In this post I will provide some links, references, and “glue text to make sense” in order to provide clarity about Secure Boot: what it is, what it provides, and what it takes from us –mostly in terms of user experience, or friction.

I do not want to reinvent the wheel or take credit for the existing state of the art, but to provide some insight that I hope it will be easy to understand to the Linux novel users. Or at least, to provide a lead that can be followed to read more and understand what’s happening.

Structure

You will find:

1. What I assume you know, or a list of things for you to search about.
2. A process to install a Linux system with Secure Boot enabled.
3. Post-Install situations affected by Secure Boot
4. What is Secure Boot, and what’s it based on.

What I assume you already know (a.k.a., what you’re supposed to search in the interwebs if not!)

I assume that you are familiar with the following concepts, which I try to summarise. But, please: use your search engine.

• Kernel: it’s the hert of the operating system. It provides the applications with access to the hardware.
• Modules: specialised code that the Kernel will load to tailor its capabilities to very specific hardware or configurations. Many times, when it allows the Kernel to understand a piece of hardware, it can be called “a driver” like in Windows lingo, but there are modules for many other things.
• Root password: it’s the password of the root user, which is the most powerful authority in a Unix or Unix-like system, as GNU/Linux for instance. Absolute privileges.
• BIOS: stands for Basic Input and Ouput System, and it was the way that older (early 2000’s and before) computers had to connect the Operating Systems to the hardware. This included a boot menu and a very few more options, and many specialised configurations for some hardware devices had to be done by plugging, unplugging or changing physical jumpers in the motherboard.
• (U)EFI: stands for (Unified) Extensible Firmware Interface. It’s the evolution of BIOS, looks after the same hardware to operating system bridge. It is much more advanced and comes with many more options, many of them enabled for modern system by default, like the GUID Partition Table for the Disks, and the Secure Boot.
• GUID Partition Table. A modern partition schema that identifies partitions on the disk by giving them Globally Unique IDentifiers, rather than names. Amongst other things, supports much larger disk sizes.
• Secure Boot deserves a whole section on its own. Go and check “What is Secure Boot” at the end of this post.

Installation Process

Check whether you actually have UEFI

First of all, have a look at what you have in your computer: whether BIOS or EFI. If you have BIOS you can skip the whole article, because Secure Boot was introduced in the UEFI.

If you’re not sure which system your motherboard uses, and if you still have Windows there, check the following:

• Look for an EFI folder within c:\windows\boot
• Open the Disk Management Utility and check if you have a partition labeled EFI in your system disk, like this one:

If you don’t have Windows or you are unsure,

• Reboot the computer and look for a message telling you to press a key to enter setup.
• Press that key, quick!
• No worries, it happens to everybody. Reboot and try again.
• Once inside, and without changing anything, look for indications of an EFI system: search for options like “Secure Boot”, “Legacy/UEFI” boot mode.

If your system shows any of the above symptoms, you have an UEFI.

Choosing a distribution that supports Secure Boot

There is a way to query distrowatch to get a list of the distributions that may support Secure Boot by searching included packages like shim or efibootmgr, but the best way is to check your favourite distribution’s documentation (kudos to firmwaresecurity.com for the queries hint).

The good news is that the support of Secure Boot is already in the Kernel and the base system for all of them, so it would be a matter of time and ideology for it to arrive to your computer. If you want to install a different distribution than those, you’ll have to disable Secure Boot in your EFI.

Download and verify an ISO, preferably a Live one with non-free firmware and modules included

It is advisable, always, to download an ISO installation image that offers you the possibility of trying Linux in your system without installing it first. This will give you an idea of whether or not your installation media will have the required modules for your hardware, most precisely (and critically neccessary) one module for the graphics adapter and another one for the network. Also, depending on your hardware and distribution, you may have to download specifically an ISO with non-free software included for it to work! This is almost always necessary (99%+ cases) in the case of Debian and most laptops.

Download your distribution in ISO format, as usual, and look for a way to verify the files you download. Most every website that offers free or open source software these days will have a side or footnote telling offering you “the hashes” or the “sums”: MD5 or SHA checksum strings.

Checksums are strings that represent the contents of a file, much smaller than the files they represent. They are generated by a mathematical algorithm that ensures that the probability of getting the same result with different files is ridiculously small. So ridiculous that it’s difficult to explain, but the chances are cosmically low.

These are techniques that allow you to check that the ISO you downloaded has not been corrupted by the download process, and more importantly, that they are authentic. It helps you know that the mirror or the server where the actual file is hosted, usually different than the server that publishes the checksums, has not been compromised: you’re downloading an authentic image, without any malware with it. What they do is run the same procedures that you’re about to run, in their side, and tell you to check that you get the same results.

If your distribution publisher offers you SHA verification sums, open Windows PowerShell and run Get-FileHash over the ISO you just downloaded:

If your distribution publisher if offering MD5 sums, then open the regular command prompt and use certutil -hashfile over the file.

Whichever method you use, your local result should match with the check sums provided by the software publisher (leaving uppercase or lowercase aside).

If they provided some other methods, check their documentation.

Creating a USB that boots under UEFI conditions

In order to create a bootable USB drive under UEFI’s newer boot schema, that USB has to be formatted using GPT (Guid Partition Table), as the MBR (Master Boot Record) drives are considered legacy!

To do that, you can use Rufus to create the USB. Rufus is an open source program that you can download from its official site https://rufus.ie/, and will offer you a simple, but powerful interface to create the bootable thumbdrive. Just make sure that you select GPT as Partition Scheme, and UEFI as Target System, leaving all other options by default (unless you wanted a persistent partition to use in a Live System):

Reboot in Linux

• Reboot the computer and look for a message telling you to press a key to show the boot menu.

• Press that key, quick!

• No worries, it happens to everybody. Reboot and try again.

• Search for your USB drive.

• They are usually by the EFI the word “USB” and the brand of the device, plus some other model numbers and maybe the capacity.

• Select it and press the Enter key.

Select the option to run Linux, or to try Linux without installing.

If everything works, specially the graphics and the network, then you’re good to go and install it.

Installing with Secure Boot: things that can happen.

In most of the cases, the installation process will be smooth and without any surprise. But, in some cases, either:

• The installation would remain stopped in a package for too long. If you expand the details of the installation process, you may see something like the below image.
• You will prompted by the system that some third party software needs to be installed and you will to do some funky stuff for it to work, something similar to this:

This is caused by Secure Boot, and nothing is wrong. What is happening is the following: the system found that it needs to install a package that will be loaded at boot time, but it is not signed. Secure Boot requires the software that runs at boot to be signed by a trusted key. The installation process can create a key for you, called “Machine Owner Key”, and sign the software with it, but in order to make the EFI trust the signature, it needs you to collaborate.

• Press the Enter key, or the tab key to focus OK and then Enter, or click OK if the prompt is run in a graphical environment.
• In the next screen, you will be prompted to set a password: do it and jot it down in a paper, as you will prompted to write it in the first reboot.

Here be dragons: if you have a non-english keyboard, refrain yourself from using symbols that may be in a different key than the one in a US layout keyboard, as they can cause problems when typing the password later.

Once done, the installation should continue.

When you finish installing the system, it will reboot.

If you were prompted with such an ugly message, you will see a succession of screens like the following (taken from this Gist, as they are the same in any case) in this first reboot:

Select Enroll and press Enter.

Press Continue. Yes, it is a weird screen. If you press View key 0 you’ll see some information about the key you are enrolling, but it’s quite technical and can be nonsense, specially if it was the installation program who created the MOK key.

Select Yes.

Type the password you’ve in your piece of paper and press Enter.

Press Enter

When the system reboot, and provided that the installation program meets its quality goals, things should work.

Post-Install Miscellanea

Manually signing binaries (e.g.: modules)

This is the case of a module compiled by hand, or when you install unsigned modules with Secure Boot installed, in more crude distributions like Debian.

The steps for Debian (credit to its official documentation and to my experience) are:

• Compile your software.
• modprobe will fail. Check that it is due to a Kernel Lockdown state imposed by the Secure Boot, to discard other errors. In dmesg, search for something with similar semantics to this:

Lockdown: modprobe: Loading of unsigned module is restricted; see man kernel_lockdown.7

• If you have already enrolled a Machine Owner Key (MOK) yourself, and it is available for you, put it in your home directory and jump to step “Open a terminal to sign the binary”. Otherwise, continue here.
• Install the packages mokutil and openssl.
• Open a terminal and create a MOK (Machine Owner Key) with the following process (# commented), documented here:

    # Go to your home folder
    cd ~
    # Create a Machine Owner Key pair valid for 100 years. 
    # In the following, replace "gabriel" for your user name
    openssl req -new -x509 -newkey rsa:2048 -keyout MOK.priv -outform DER -out MOK.der -days 36500 -subj "/CN=gabriel/" -nodes

• You will obtain two files, MOK.der and MOK.priv in your home folder. Enroll the keys in your EFI

    # If you don't have sudo installed, run `su` and type the root password
    # before the following commands without sudo
    sudo mokutil --import MOK.der
    # This will prompt for a one time password to set up.

• Reboot the computer. You will be promted with the blue screens of section “Installing with Secure Boot: things that can happen”. In this case, if you select the options to View Key 0, you will be able to see your name and the name of your computer. In my example, I saw “gabriel@rodas”, and rodas is my computer named after this beach very close to my hometown.
• Open a terminal to sign the binary. Follow the process changing the /path/to/file.ko with the path to the file you need to sign.

    # Assuming that you have kernel version X.Y.Z
    # Replace "gabriel" by your username.
    cd /lib/linux-kbuild-X-Y/
    sudo scripts/sign-file sha256 /home/gabriel/MOK.priv /home/gabriel/MOK.der /path/to/your/file.ko

• Try to load again the module

  sudo modprobe /path/to/your/file.ko

It should work now. This is the most common use case for this.

Here be dragons: hold your horses, one last thing before you go. Take these two files, MOK.der and MOK.priv, away from your user directory, hide them in a safe place. Personally, I keep them attached to an entry in my password manager, well encrypted. They are the door for someone to sign any malware so it can run at your computer’s boot!

What about signing a custom Kernel?

If you need to sign a Kernel, the /path/to/your/file.ko looks like /boot/vmlinuz-[KERNEL-VERSION], and probably you will have to follow more steps. Although I haven’t had the need for it, I found several forum and Stack Overflow questions that pointed to this document as a valid process to do it.

What is Secure Boot

Secure Boot is a mechanism designed to protect the process of booting your computer, that is to say, making your boot secure. It is not a very creative name.

It was introduced at some point in the UEFI specifications, and it made a weird entrance in the scene because some poor communications skills and shady decisions from Microsoft. They told at some point that, for any PC machine to be Windows-Certified, they should pre-ship a cryptographic key (similar to the Machine Owner Key we’ve used before in this post, but Microsoft’s) to digitally verify the Windows bootable parts and the Kernel. They didn’t specify at the point that they didn’t care whether the Secure Boot could be disabled, or that many other keys should be able to be shipped or installed alonside Microsoft’s! Thanks to that narrow statement, the community went crazy and, although they fixed the statement pretty much immediately, for many time Secure Boot was seen as a Microsoft attempt to close PC hardware to them. You can check this story here. It is true that the controversy continued for a while, specially related to ARM devices, but as always: technology isn’t bad by itself, it is us who make good or bad use of it.

Indeed, Secure Boot is (to me) pretty awesome as it is a simple but powerful way to protect the boot sequence of your system against malware.

The “Secure” word of Secure Boot comes from Information Security, as it is based on asymmetric criptography. It works in such a way that, for the EFI to load any code at boot time, being it a Kernel, modules, or the GRUB itself, such code has to be cryptoraphically signed by a trusted key. Let’s decypher such two sentences.

• Asymmetric cryptography refers to a compendium of techniques that are based in a pair of keys, public key and secret key. They public key is meant to be available to everybody, while the secret key has to be kept secret to its owner.

• That par of keys are often packed in a single software artefact called “certificate”, in such a way that the private part is protected by a passphrase. It can also be splitted in such a way that you can produce a certificate that will only contain the public key, to be able to share it. It provides two operations: encryption and signature.

• Asymmetric Encryption works the following way: if I want to encrypt a message for you, so that only you are able to understand it, I will use your public key and a lot of standard maths. Once done, only your private key, plus a lot of standard maths, can decrypt it!

• Asymmetric Signature works the opposite: if I want to prove you that it was me who wrote a message, I will sign it using my private key and a lot of standard maths. You will be able to check the signature using my public key and a lot of standard maths.

• If someone changes a single character of that message, the signature won’t be valid anymore.

• The last point is great: a signature identifies the author of a message, and protects the communication against tampering.

• We can cryptographically sign a message, or a binary file like a Kernel, or a module.

• According to the above, when we sign some code and install it to be loaded at boot time, like a Kernel or a module, the EFI need to have available the public key to check the signatures. This defines the need of enrolling public keys in the EFI.

• You can create a pair of keys, enroll the public key, and use them to sign software you create or compile, if you are the root user of a GNU/Linux system. When you create your own pair of keys, they are together called a Machine Owner Key, or MOK.

• Enrolling a public key in the EFI is a process not only means to make available the public key to the EFI: it also makes the EFI trust that signature. Practically speaking, it will ultimately trust the software that is signed with a trusted key, and load and execute it with all the consequences.

• Only a trusted software manudacturer, or an administrator of an operating system (root user in Linux) can enroll a key in the EFI and manually sign software.

• Enrolling a key in the EFI is a process that requires such an administrator to collaborate and actively accept and prove his or her identity after a reboot. It cannot be automated by a malware process.

A bit of recap. If you have the system bootstrap process checking signatures of the binary code that it loads, this means a lot of security.

Let’s understand the two main vectors of attack it mitigates.

1. Only trusted signers (meaning software manufaturers, or you) can install code that runs automatically in the bootstrap of the system. That means, that if some random guy comes to your desk with a USB while you’re away, he won’t be able to infect your computer’s bootstrap by putting some modules or writing code in the bootable partitions. He or she would have to enroll a key, too, and that requires (in Linux) to know the root password, which is meant to be secret and complex. They will have to be able not only to write code to the EFI, which may or may not be a challenge, but to hack the operating system they want to compromise.
2. If you accidentally allow a malware to run in your computer, that malware cannot infect the bootstrap chain even if it has the maximum privileges, as it will require active cooperation by an admin user to create and enroll its own MOK: check back the rest of the article at the point we set up a password, reboot, and type the password when enrolling the key. The only thing you have to do is to never leave your MOK private key in the computer to prevent the malware to sign itself with your own keys!

I hope you didn’t have to search many terms, and that this article was, at least, easy to follow. Feel free to send me some comments to comments (at) gvisoc (dot) com!