Recently, I was on a call with a friend who mentioned he was having an infrastructure problem with his home lab setup. He uses Docker Compose to manage his services, and wants to introduce health checks to all his images so he can use a tool like autoheal to restart unhealthy ones. However, some of the containers he uses in his setup are based off of distroless images, which prevented him from using existing tools in the container like curl and wget.

I’ve been working to improve my Rust skills, and a lot of what I do is related to infrastructure. This seemed like an interesting project to sink my teeth into, so I volunteered to build something to help.

A Word on Distroless

“Distroless” is a container setup in which the final image that you ship contains only the bare-minimum dependencies required to run your service. Some languages, like Go and Rust, are predominantly statically-linked- which means that it’s possible for them to have no external dependencies. For an application that is fully statically linked, none of the standard things most applications rely on (like libc) need to exist in the container, so you get a very tiny final image. That is exciting for several reasons that deserve their own blog post, but not great if you need to use tools that rely on those standard libraries.

First Attempts with Existing Tools

When we were talking it over, my friend mentioned he tried a few things to get these checks to work. Ultimately, he needed something that could make HTTP calls and return a status code for the health check, so at first he tried:

1. Mounting the curl binary in the container. This didn’t work because it’s dynamically linked and the libraries weren’t mounted
2. Mounting the wget binary in the container. Similarly, this is dynamically linked

Trying to mess around with mounting libraries was not desirable, and neither was building a new container based on the application used- those things are a little too finicky and it’s usually better to rely on the existing publishing infrastructure managed by the people actually writing these apps.

The Requirements

We chatted it over and came up (informally) with a small list of what was needed of the tool. Writing this up now, we can loosely split them up by priority as I understood them at the start:

P0 (main priority):

1. The tool needs to be able to make an HTTP GET request to a URL specified by the command line
2. On a successful request, it needs to return an exit code of 0
3. On any failure, it needs to return a non-zero exit code
4. The tool needs to be fully statically linked so it can run on any x86_64 container, including distroless

P1 (key, but less important):

1. The final binary needs to be reasonably small. The base container we were working with was ~4-5 mb in size, and I did not want to significantly bloat it
2. The tool needs to be reasonably efficient. It should not meaningfully increase the resource requirements to run the application.
3. The tool needs to be reasonably easy to use.

P2 (nice-to-haves):

1. The tool should be easy to consume. This could mean a pipeline, publishing it to a package registry, or even making the code available in an easy-to-build way.
2. Support for TLS as an optional feature

Benchmarking

As part of the P1-1 requirement, I wanted to also check what the size of the existing HTTP clients are, since they’re standard and contain quite a few features. If I’m building a minimal application, I should strive to not be substantially larger than existing tools that are more comprehensive. Ideally, my application should be even smaller than what is available.

I picked the alpine container to check the size of these binaries and their libraries since alpine tends to be very minimal. First, I checked the size of wget and its dependencies:

Then, I checked curl:

Now, I have my target to beat: 1.4 mb for wget and 6.1 mb for curl. Any binary that I build that has minimal functionality should be within that ballpark.

Implementing HttpGet

Preliminaries

From previous development experience (generally, and in Rust), there were a few things that I knew off-hand would be relatively simple:

1. I had existing Rust projects that had Github Actions workflow files I could reuse/adapt, so testing/releasing the finished binary was going to be fairly trivial
2. I knew the Rust Cookbook has an example on how to make web requests, which is the core functionality of this tool
3. Debug information and symbols could be stripped from the final binary to make it smaller
4. By using the x86_64-unknown-linux-musl target in Rust, the final binary is statically linked

Armed with this knowledge, I set out to create httpget.

First Pass Implementation

The first implementation of httpget focused around using the reqwest library as that was what the cookbook recommended. The whole implementation took about 40 minutes to create from start to finish, mostly because I was already familiar with reqwest and tokio. Since TLS support was a P2, I started first with limiting the features of these crates to try to have the most compact binary. After some fiddling, version httpget v0.1.2 was published on GitHub by the pipeline.

use reqwest::Client;
use std::env;

async fn run(endpoint: &str) {
    let client = Client::builder().build().unwrap();

    let res = client.get(endpoint).send().await;

    if res.is_err() {
        panic!("Can't reach route {}", endpoint);
    }
}

fn main() {
    let args: Vec<String> = env::args().collect();

    if args.len() > 2 {
        panic!("Too many arguments!")
    }

    let endpoint = args.last().unwrap();

    tokio::runtime::Builder::new_current_thread()
        .enable_time()
        .enable_io()
        .build()
        .unwrap()
        .block_on(run(endpoint))
}

#[cfg(test)]
mod tests {
    use super::*;

    #[tokio::test]
    async fn can_reach_google() {
        run("http://google.com").await
    }
}

Looking over our requirements, it mostly looked like it matched quite a few of them:

However, there’s some stuff I immediately didn’t like about this solution:

1. No way to have TLS support. This became a problem since httpget was emitting panics when trying to reach sites like http://github.com, presumably because of HTTPS redirects. This would not be a problem in my friend’s home lab since they only needed http, but was a limitation.
2. While 2.87 mb is relatively small (a little under half the curl benchmark), that seemed very excessive. I could not find a way to remove the tokio dependency from reqwest, which means that it was not possible to use this recommended crate without bundling an entire async runtime in the binary. It didn’t really matter too much since this isn’t being run on an embedded system, but it didn’t sit right with me that the tool would be this “big” with such limited functionality.

Fixing Incorrect Requirements

After presenting this version to my friend, we chatted a bit more and eventually I realized I had misunderstood one of the P0 requirements: For healthchecks, Docker requires very specific exit codes. For success, it does indeed use a 0 exit code (P0-2), but for failures it requires an exit code of 1. This means that P0-3 needed to be re-written:

P0-3: On any request failure, the tool should return an exit code of 1.

Unfortunately, since v0.1.2 relied on panic! to register failures, the exit code for httpget was 101 on failures. This could be useful for some things (like being unable to build the async runtime) that represent unrecoverable errors in the setup, but everything else should set the right code.

Luckily, Docker (like most platforms) seems to see a non-zero status code inherently as a failure, so the tool could be used until a fix was implemented. However, the behavior should not be relied on! It is possible that in the future, Docker could set exit codes that are neither 0 or 1 to mark health status as something else, like “unknown”.

After some more fiddling with pipelines and implementing some better dev processes, httpget v0.1.6 was released with TLS support baked in (with rustls)

use reqwest::Client;
use std::{env, process::ExitCode};

async fn run(endpoint: &str) -> Result<(), reqwest::Error> {
    let client = Client::builder().build().unwrap();

    client.get(endpoint).send().await.map(|_| ())
}

fn main() -> ExitCode {
    let args: Vec<String> = env::args().collect();

    if args.len() > 2 {
        panic!("Too many arguments!")
    }

    let endpoint = args.last().unwrap();

    let res = tokio::runtime::Builder::new_current_thread()
        .enable_time()
        .enable_io()
        .build()
        .expect("Could not build the runtime")
        .block_on(run(endpoint));

    if res.is_ok() {
        ExitCode::from(0)
    } else {
        println!("{}", res.unwrap_err());
        ExitCode::from(1)
    }
}

#[cfg(test)]
mod tests {
    use super::*;

    #[tokio::test]
    async fn can_reach_google() {
        let res = run("http://google.com").await;

        assert!(res.is_ok())
    }
}

This got a lot closer, but still had some key problems:

1. TLS support was now mandatory. My friend didn’t need this, and some of the use cases that I would need something like httpget for also didn’t require TLS, so I wouldn’t want either of us to incur the cost of additional unused dependencies
2. Since it now supported TLS, the binary was even larger: 4.07mb, or roughly the size of the service container we wanted to use.

For me, this meant that I should go back to the drawing board.

Reassessing Dependencies

A fundamental limit to the binary size came from the dependencies I was using. No matter what I did, I could only use reqwest if I also used tokio. This was far too excessive for my needs!

After a bit more searching, I finally managed to find a package that aligned with my goals: minreq. Although it looks a LOT less used than reqwest - 122 stars, used by 585 projects, about 290k downloads at time of writing compared to 7.1k stars, used by 142k projects, about 48M downloads - it still does what I need it to do. In this case, I considered the benefits of using the most popular package to not outweigh the drawbacks of its bulk: the usecase does not support anything more than a bare-minimum library.

I re-wrote the application to use the new client library instead, which supported minimal dependencies and no async. This means I can now forego having to bundle an async runtime, and making the final tool incredibly minimal.

use std::{env, process::ExitCode};

#[inline]
fn run(endpoint: &str) -> Result<minreq::Response, minreq::Error> {
    minreq::get(endpoint).send()
}

fn main() -> ExitCode {
    let args: Vec<String> = env::args().collect();

    if args.len() > 2 {
        panic!("Too many arguments!")
    }

    let endpoint = args.last().unwrap();

    let res = run(endpoint);

    if res.is_err() {
        println!("{}", res.unwrap_err());
        return ExitCode::from(1);
    }

    let code = res.unwrap().status_code;

    if code > 299 {
        println!("Received status code {}", code);
        return ExitCode::from(1);
    }

    ExitCode::from(0)
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn can_reach_google() {
        let res = run("http://google.com");

        assert!(res.is_ok())
    }

    #[test]
    fn cant_reach_nonsense() {
        let res = run("http://asdqeqweqweqweqwe.local/qweqweqweqwewqe");

        assert!(res.is_err())
    }
}

Lastly, I needed to resolve the TLS bundling problem. For that, I ended up learning that you can use features in your own crate to enable features in a dependent crate from reading the Cargo Book section on features. Finally, I updated the pipelines to produce multiple binaries so that users could opt-in to TLS support, or use the regular binary for most condensed but http-only. With that, I released httpget v0.1.7.

Final Review

Let’s look over the requirements list and see where we left off one more time:

With this latest version of httpget, the overall binary size is laughably small, making it a reasonably negligible footprint on even the incredibly small service image we were working from. It required only 8 dependencies total to compile with no TLS support, and 25 to include rustls TLS.

From the final binary, my friend was able to mount it to his container as a volume in Docker Compose, and then add a healthcheck as he originally intended. This meant he could still leverage the upstream image publishing, but also monitor container health.

As a last note on binary size, 610kb is very compact. For comparison, a blank “Hello, World!” application compiled with the same target and stripped in the same way is around 400kb. It would be incredibly difficult to get any smaller than that binary size without trying to move to using #[no_std].

What Did I Learn?

As part of this work, I think the most critical (technical) knowledge gain was in how to more effectively use Cargo features. My P2-2 requirement for TLS to be optional could not have been enabled in any other way (that I know of) and it was originally frustrating to work with.

I also learned quite a bit about properly gathering requirements, and double checking with the “client” (i.e. my friend). If I hadn’t been a bit more obsessive over the image size, I probably might not have noticed that I had a P0 that was incorrectly formed.

It was also interesting to see how different targets worked to produce different binary sizes, and what dependencies I could use (or alternatively, that I couldn’t) with my P0 requirement of static linking. For example, I tried to use OpenSSL as the TLS provider at first, but couldn’t figure out a way to get it to link properly, so I gave up in the end and called it a “later problem”.

Finally, I learned more about how binaries get linked, and what that means. To benchmark the existing web clients, I had to figure out how to find the libraries they were linked against so I could get the library sizes, which was new information.

Future Work / Opportunities for Improvement

1. Enabling additional features for different TLS providers (i.e. OpenSSL)
2. Additional documentation of the package itself
3. Allowing httpget to receive a sequence of URLs, and querying each one, returning 0 if all of the requests succeed and 1 otherwise. This could be useful if the health of a service cannot be determined from a single route
4. Using features and conditional compilation to let users choose the HTTP client at compile-time. This could allow someone a little more hesitant to use a less-popular library to still opt for using reqwest if they want to.

If you’re a service team, these concerns might not be front of mind. Investing in security tooling and processes takes away time from being able to develop new features and fix existing bugs and maybe, just maybe, pay back some of that tech debt. Dependency updates need to be tested to prevent additional bugs, taking away even more of the critical resource of time. With all of that in mind, is it so bad to let a few versions go by?

As it turns out, vulnerability patching is no joke: the Canadian Centre for Cyber Security ranks patching applications and operating systems as one of the top 10 security actions to take, placing it at #2. When you consider that 75% of attacks in 2020 used vulnerabilities known for at least 2 years, it becomes even more critical to keep software and dependencies up-to-date.

In this post I will go over some tools that you can use to scan dependencies and containers for vulnerabilities. We will also use Github Actions to automate the use of these tools to give us regular updates on the status of a service’s container image.

Preliminaries

If you are familiar with software development and containers as a whole, you can skip this section.

General Concepts

• A software vulnerability is a problem in a software that leaves it exposed to potential attacks.
• A Continuous Integration / Continuous Deployment (CI/CD) system is a system that lets you run automated testing, building, and deploying of applications
• Github Actions is a CI/CD platform integrated into GitHub
• A pipeline is a specific definition inside your CI/CD platform. It can be used to build, test, and release software, among other things. In Github Actions, a pipeline is synonymous to a workflow
• A branch is a concept related to the version control system called git. Usually, service deployments are done from the main branch, usually called (appropriately) main.
• A commit is a specific point in a branch. This effectively acts as a snapshot of your code, at a specific branch, at a specific moment in time.

Containers

I assume for this post some level of familiarity with container tools such as Docker. Below is a list of concepts that you need to know:

• A container is a way to package software and its dependencies to help it be more portable
• Docker is a tool to let you build and run containers
• An image is the immutable definition of the container. You can think of them as a template.
• An image has some base image, which usually has some tools and software installed on it. Unless you are building an image from scratch, the base image you use will have been published by someone. An example of this is alpine:latest, or rust:1.67.
• An image tag is some semantic name given to a specific version of the image. Tags are not immutable, meaning the image that is associated with a specific tag can change.
• A container registry allows you to upload container images, which can then be distributed. Github’s integrated container registry is called Github Packages.
• A container image is “uniquely” identified by its hash. This is done by a specific of a version called the Secure Hash Algorithm, or SHA

Required Software

In this post I will assume you have Docker installed. I also mention Rust tooling for vulnerability scanning, but no practical example is done.

You should be familiar with how to pull an image from a registry if you would like to follow along.

Dependency Scanning

A key benefit of having security be so front-of-mind in modern development is that there has been a lot of tooling developed specifically for improving security of applications. In particular, most languages have some sort of dependency scanning tool that compares your project’s dependencies against a list of known-vulnerable packages in your ecosystem.

For Rust, one such tool is cargo-audit, which uses the Rust Advisory Database to check for vulnerabilities. It integrates very nicely with cargo, the Rust package manager.

To install, run cargo install cargo-audit, and run a check with cargo audit. You can also check additional options for how to customize your scan a little more granularly.

Container Scanning with Trivy

You might assume that code dependencies are the only thing you have to worry about scanning, but sadly in the world of containers this isn’t really true.

Most base images available are based off of some operating system, and give you many of the same tools. For example, by using the ubuntu:latest image, you have access to a whole suite of tools. These are not dependencies that you explicitly opted-in to, but ones that came bundled in.

Beyond that, if you use a language that requires an interpreter (like Python or Javascript) or some additional runtime engine (like Java or C#), these will also have their own set of dependencies that they need to run. This introduces additional avenues for vulnerabilities that would not be caught by a dependency scanner.

Thankfully, container scanning tools are also fairly prevalent. One common one is a tool called trivy, which lets you (among other things) scan container images for vulnerable packages, tools, libraries, etc.

Manually Scanning Images

First, you should install Trivy however makes the most sense for your system and usecase. I will assume that however you’ve installed it, you can access it from your command-line with the trivy command.

Next, let’s pull an image with some known vulnerabilities:

docker pull python:3.7-slim-stretch

Next, let’s run a trivy scan:

trivy image python:3.7-slim-stretch

You should see a table output with lots of vulnerabilities listed (by code), the package that is vulnerable, it’s severity rating, what version you have installed, and the minimum version needed to fix the vulnerability.

Now, let’s run a scan on an image that has no vulnerabilities (at the time of writing). Note, there’s a chance that when you are reading this, the image might be vulnerable. I’ve picked Alpine for this purpose because their vulnerability patching tends to be very timely.

docker pull alpine:latest
trivy image alpine:latest

Running Regular Scans

Now that we’re able to scan our dependencies and containers for known vulnerabilities, it’s time to add some automation. While it’s lovely to be able to do the process manually, leaving this task for developers will add overhead that you just don’t need, or it will mean that it doesn’t get done.

Besides, I’m very lazy. If a computer can do something for me instead, that is preferable.

A few things to keep in mind as we start this process:

• Regular scanning is nice, but you also need to have a plan for when vulnerabilities are found.
  • It’s possible for a dependency to release a fix for a vulnerability alongside a breaking change. Good testing will allow you to catch this and address necessary changes.
  • If you prevent deployments from happening when the images fail a vulnerability scan, you could inadvertently prevent critical fixes from being deployed in your own service. Have a plan in mind for how to deploy hotfixes even if there is a dependency you can’t upgrade yet
• Not all vulnerabilities are created equal, but the security team won’t love you if you point that out
  • Especially as teams and companies grow, it’s harder and harder to ensure that whomever is in charge of security is aware of all the required context to accept the risk of a specific vulnerability.
  • The easiest way to ensure that your software isn’t vulnerable even by transitivity is to not let any part of it have an unpatched known vulnerability.
• Your release cycle needs to be considered when setting the frequency and type of scans

For web services I’ve written as personal projects, I like to do three types of scans as part of my nightly scans:

1. Scan the dependencies to see if anything needs to be updated
2. Scan the containers we most recently deployed (the latest tag, usually). These should be the containers that are running in production.
3. Build new release images at the latest commit of the main branch, and then scan those images.

This is by no means an exhaustive list (we could, for example, scan the files for secrets), but is a very good foundation.

A Note On Published vs Built Containers

It’s quite common for the most recent commit to the main branch to also be what was most recently deployed. You might be wondering, then: why do both scans?

This is when release cycle comes into play- Not only your own release cycle, but the release cycle of whatever container dependencies you use in your project!

It’s reasonably common to use version tags that get updated, such as rust:1.67 (which gets updated on patch versions), as well as the use of latest tags. If that is the case in your service, it is possible to have the most recently deployed version be flagged as vulnerable while the most recently built container is not.

If you use tags that don’t tend to change (like rust:1.67.1) or use SHA hashes for your containers, doing both scans won’t give you anything different. In those cases, you’ll need to upgrade the dependency manually.

Automating the Scans

Since I use Github as my primary code storage, I tend to use Github Actions for automation. Thankfully, it supports running on a schedule with a similar syntax to cron, which we can leverage.

Below is the workflow file definition used for my websvc-rs repository template, which covers all three of the discussed scans. I have annotated it with additional explanations of what everything each job does.

It’s important to keep in mind that, if your service gets released in multiple ways per version, you should scan all of them. In this example, websvc-rs releases 4 images for every version, broken down into:

• One version compiled with musl that is based off of scratch, and includes nothing except the service and a tool to check that the service is healthy, tagged latest
• One version compiled with musl that is based off of alpine:latest that includes additional tools on top of what I’ve built (i.e. a shell, a package manager, etc), tagged debug
• One version compiled with glibc that is based off of scratch, tagged latest-gnu
• One version compiled with glibc that is based off of alpine:latest, tagged debug-gnu

Each of these versions should be scanned.

# .github/workflows/nightly-scan.yml
name: nightly-scan

on:
  schedule:
    # 7 AM UTC every day. Roughly translates to 2 AM in the east coast
    - cron: "0 7 * * *"
  # Allows the workflow to be manually triggered
  workflow_dispatch:

env:
  # Github container registry's domain
  REGISTRY: ghcr.io
  # The name of the image. I always use the repository
  # name when only publishing one container image
  IMAGE_NAME: ${{ github.repository }}

jobs:
  # Job to run the audit scan for `cargo` dependencies
  cargo-audit:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v3

      - name: Install rust tool chain
        uses: dtolnay/rust-toolchain@master
        with:
          toolchain: stable
          components: rustfmt, clippy

      - name: Run the security scanner
        run: cargo audit

  # Job to build an image from the latest commit &
  # scan the resulting image. This is because the
  # service uses `alpine:latest` for debug containers.
  #
  # Prod targets are still scanned despite being based
  # off of `scratch` in case the basis changes. For example,
  # if instead of `scratch` I ended up needing to use one of
  # the distroless images, they will contain some dependencies
  sha-scan:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        target:
          - prod
          - debug
          - prod-gnu
          - debug-gnu
    steps:
      - name: Checkout repository
        uses: actions/checkout@v3

      - name: Build the docker container
        run: |
          docker build -t ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }} \
            --target ${{ matrix.target }} \
            .
      - name: Run Trivy vulnerability scanner
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: "${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }}"
          format: "table"
          exit-code: "1"
          ignore-unfixed: true

  # Pulls from Github Packages each of the 4 images that
  # are released, and scans them.
  registry-image-scan:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        tag:
          - latest
          - latest-gnu
          - debug
          - debug-gnu
    steps:
      - name: Pull the docker image to scan
        run: docker pull ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ matrix.tag }}

      - name: Run Trivy vulnerability scanner
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: "${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ matrix.tag }}"
          format: "table"
          exit-code: "1"
          ignore-unfixed: true

Now, any time this workflow fails, I will get a Github notification and be able to see what went wrong and, potentially, run a new deployment to fix it!

Why Scan scratch Images?

You might be wondering why I bother scanning the latest and latest-gnu images. After all, if they only have the service and the health check, won’t they always come back clean when our service doesn’t have a vulnerability itself?

Yes, that is exactly correct. However, this assumes that these images are currently based off of scratch and will always be based off of scratch. Similarly, it assumes that nothing else will ever get added to the image.

If any of these assumptions change down the line, we would have to update our nightly scanner, which means we will also have to remember to update our nightly scanner. However, if we scan them (even when we don’t need to), we are covered when our initial assumptions no longer hold.

Wrapping Up

In this post we talked about the importance of security scanning, and gave context to the different methods of scanning. We covered an example of scanning two real containers for vulnerabilities, and set up a Github Actions workflow to run regularly-scheduled scans.

Questions to Reflect On

1. Since we can use a workflow file to deploy our application, we could have this nightly scan deploy a new version from the newly-built containers if the existing ones are vulnerable and the new ones are not. What are some potential benefits to doing that? What are some potential drawbacks?
2. In this example, I configure Trivy to ignore unfixed issues. What are some benefits of doing this? What are the drawbacks?
3. In this post, we only covered one specific dependency scanning tool, and only for one programming language. If Rust is not your primary language, what dependency scanning tool is the most popular one for your language of choice?
4. If Trivy can scan so many targets, including filesystems and git repositories, why did we go over how to run a dependency scan using cargo-audit? Are there (security or otherwise) benefits to using it instead of Trivy?

In this post, I will be going over some house-keeping content. If we were to pretend that my blog is a class, you could consider this post to be its syllabus: this covers the general topics I try to cover, how I structure different material, what assumptions I make about the reader, etc.

Content

Ideally, I would like to do series of posts that cover related topics. There are a number that I am currently entertaining (some of which I have already decided to do), including:

• Cryptography primer
  • Cover math concepts used in cryptography
  • Cover how these concepts are actually used
  • Implementing the math / cryptography concepts discussed
• Creating and operating a k3s cluster
  • How to bootstrap a cluster
  • Setting up cluster storage
  • Monitoring with kube-prometheus-stack
  • Deploying applications
  • GitOps
• Containers and You
  • Creating containers to use for development
  • Production containers & a minimized setup
  • Security concepts in container technology
  • Emerging / existing container technology
• Bootstrapping a Rust-based web service

If you are interested in my writing any other series in specific, I highly encourage you to reach out! I would love to hear feedback (even if the feedback is negative), and would like to make content that is interesting to read. If you think I can cover anything in specific, or if you like my style of writing and want me to answer specific questions you may have, I would be glad to do so!

I will also try to do one-offs that I think are interesting or relevant to current events; for instance, if any big attacks are made public or if there is relevant policy I’ve heard about that touches on my topics of interest.

Finally, I will be trying to provide project writeups when I make new open-source projects that give a more narrative description of why the project was made, how it was made, what decisions went into it and why. I hope in doing so I can add a bit of clarity and context about my work.

Frequency & Structure

In general, I am hoping to make approximately 1-2 posts a week. Sometimes this might increase or decrease depending on additional circumstances: if an important new vulnerability is discovered, I might make an additional post even if I’ve already made 2, or if I am on vacation I might skip a week. I will try to post on the news if I know that will happen ahead of time.

For post series, I will be releasing a first post that goes over the preliminaries of the series, including what I expect a reader to know ahead of time, what format the posts will have, and other specifics that can change series-by-series.

For project writeups, I will be assuming a passing familiarity with the tools used, and will link to existing documentation for any dependencies and specifics that I use or encounter.

Finally, for one-offs and current events, I will try to have a preface section with an overview of everything you need to know and links to appropriate resources for more information.

Errata

If ever you find material in my posts that is incorrect, or there is a better way (i.e. a best practice I am not following), I highly encourage you to reach out by email to let me know.

In the last few guides, we have gone over how to set up a new headless Raspberry Pi with some sensible configurations, how to connect over SSH without using a password while ensuring only expected machines can connect, as well as how to use multi-factor authentication over SSH to add another layer of protection to our server. While these have all been good practices, we have not yet (at this point) actually used our Raspberry Pi for anything. Depending on where you've installed it your RasPi might be a fun conversational piece, but it would be a lot better if we could have it be working for us. Today, we'll be accomplishing that by adding PiHole to our Raspberry Pi as a DNS server, which will allow our entire network to benefit from their DNS-based ad-blocking.

First, let's go over some key concepts of the tech we're going to be using.

What Is a DHCP Server?

Very briefly, a DHCP server is a bit of software that is able to make all the devices on a network play nicely together by giving each of them a unique identifier (their IP Address), letting them know what DNS server to use, and trying to ensure that no two machines on the network have the same address. DHCP servers remove the need for each machine to be manually configured by someone who knows the network landscape, and instead offloads that task to a computer.

While most residential DHCP servers work entirely by dynamically allocating addresses to machines, they are also capable of providing reservations, which is when the DHCP server reserves an IP address for a specific machine. This is incredibly useful when you need to have a machine on your network that should always be given the same address because it needs to be predictably discovered by other machines. We see this often for network printers, servers, and other similar software.

What Is a DNS Server?

Whenever you click on a link or type a website's URL onto your browser's address bar there is a lot going on under the hood to bring you your content. While using the internet these days is pretty trivial, there are many systems in place that translate the experience from something that is easy for people to use (with words and text) to something that is easy for computers to operate on (broadly speaking, numbers). One of those systems is the Domain Name Service, which lets us convert URLs – such as natalia.dev – into IP addresses – such as 162.241.253.207.

Whenever you go to a website, your computer first makes a request to the DNS server that it is told to use by the DHCP server with the URL of the website. The DNS server then resolves the URL into an IP address, which your computer can then use as the destination on the packet that it sends.

One simple way of thinking about it is that the DNS server is similar to an address book that can tell you your friend's postal code. You consult it before you send a letter to your friend so that you can give the proper information to the post office, which will then send your letter.

What Is PiHole?

At its core, PiHole is akin to a DNS server. You configure it to receive DNS requests on your network, and it will take them in and resolve them. However, it has a crucial additional feature: It contains a comprehensive blocklist that allows it to determine if it should resolve it properly, or drop the request. This, in turn, allows it to drop all DNS calls to places that are known for serving ads, malicious content, trackers, and the like.

You might be thinking to yourself, "the internet is mostly free right now, isn't that powered by ads? Wouldn't it be a bad thing to block them?", and on some points you would be right. It is indeed true that much of the internet is able to sustain itself by ad-based revenue. However, this comes at a fairly substantial costs to users in that it exposes additional threats. There have been many recorded incidents of using ads to spread malware, and large companies that focus on generating ad revenue rely on using your data to generate comprehensive profiles on you to target you specifically for certain kinds of ads. There are several ethical and security implications to this that warrant a full discussion on another day.

For today, however, we will be going through a way that we can try to minimize this as much as possible through PiHole.

Setting Up Your Network

The first thing we should do is to make sure that we have a DHCP reservation for our Raspberry Pi up now. I usually like to get this step out of the way first, because I don't want to forget it.

Modern routers usually include a small HTTP server that allows you to manage it via a web interface, so we'll need to go on that to set up our DHCP reservation. My network's router is at address 192.168.0.1, so I can just punch that into my address bar to get the admin panel. Yours might be different, and you can consult this article by Lifewire on how to find yours.

Once you have that address, go to it and log in. If you have never logged in before, you might need to look up your router's default admin credentials are. I highly recommend changing the password if it is still the default one.

After logging in, search for the DHCP settings. This might be under a different name, such as my own router which has it under "LAN Setup". Find the setting for "DHCP Reservation", and make one for your Raspberry Pi to maintain the same IP address.

Once you have this reservation complete, save your configurations and close out of the admin panel. Now we're ready to start setting up our pihole.

Installing PiHole

It is broadly recommended that we always look at code we're downloading from the internet before we run it on our systems. So first, let's download the basic installer code with the following command:

wget -O basic-install.sh https://install.pi-hole.net

We can then review that code by opening it up in nano, or using less, and review it to make sure it doesn't seem to be doing anything wrong. Once we've finished with that process, we can run the script with the following command:

sudo bash basic-install.sh

This will start a series of checks and package installations. You will then have to go through some menus, and select which interface to use. Since I am using my Raspberry Pi connected to ethernet, I chose the eth0 interface instead of the wlan0. If your interface names don't quite match, a good rule of thumb is that if it starts with eth it is most likely ethernet. If you're unsure, press 'Cancel' and figure out which interface you need to use.

Eventually, you will be prompted with a screen notifying you that PiHole needs a static IP, and then finally a selection screen for what upstream DNS provider you would like to use.

Which server you pick ends up being a matter of personal preference, and there's quite a few available. Personally, I like to go with OpenDNS, which I believe is currently owned by CISCO, since they support DNSSEC and are broadly a security-minded company. You can choose whichever one you find most appropriate based on your own criteria.

Afterwards, press Enter to accept, and leave the remainder of the settings as default until you reach the menu shown below:

You can read a little bit more about what each privacy mode is on this link. I recommend you pick the one most appropriate for your home use. I won't touch too much on the ethics of logging information of the devices of your home, but I would like to take a minute to at least recommend that if you live with other people, you should check in with them before storing their data. It might be more appropriate to hide some or all of the information that is logged.

Once you have made your selection, the installer will continue setting up the config files and packages you need to run the software. Once you have completed the installation, it's time that we set up our DNS servers to point to our new PiHole!

Setting up DNS Server Locally

I'm a very big proponent of the idea that we should fail fast, fail often, and fail small. As such, before we set all of our devices to use our new PiHole, we should first make sure that it is actually working appropriately.

You'll need to change your DNS settings to point to the new PiHole. Once that is done, try navigating to the admin panel at http://pi.hole/admin. If you see the PiHole dashboard, your installation should be working! Try navigating to a few sites, such as your email or a local news site, to confirm that everything still works appropriately.

Setting up PiHole as Default DNS Server

Once we have confirmation that our PiHole works, it's time to make sure that our router sets the PiHole as the default DNS server for the entire network. Log back in to your admin panel, and search for the DNS settings of your router. Search for your DNS settings, and set the default server to be your PiHole's address

Once that is done, save your settings. If you are patient, now you can simply wait and whenever new devices connect (or old devices have their lease expire), you should have the PiHole be their DNS server. If you are like me and want to see the fruits of my labour kick in instantly, just restart your router. All devices will have to reacquire their lease, which will update the DNS setting for all devices immediately.

Using the PiHole as a DHCP Server

If you log in to the Admin Panel on the PiHole, you may have noticed that the clients are (at least for the most part) only shown by their IP address. This means that if an IP address changes, the logs cannot be properly attributed to a specific client, and the clients are not meaningfully described.

As well, some routers, especially the ones that are provided by Internet Service Providers (ISPs), don't actually allow you to change your default DNS server. If that is the case, you won't actually be able to use PiHole automatically without other changes.

In all of these cases, it might be wiser to use the PiHole itself as the DHCP server for your network.

To start, go on your PiHole admin panel, navigate to Settings and DHCP settings. Click the checkbox to enable using the Pi as a DHCP server, and set the IP address range to appropriate values. Then, press "Save", and go to your router's admin panel and disable the DHCP functionality on the router. Then, restart your router to force all devices to renew their leases and they should all start using the Raspberry Pi as a DHCP server instead of the router.

Wrapping Up

In this guide we have gone over the following concepts:

• What is a DHCP server
• What is a DNS server
• How does PiHole work
• How to install PiHole on a Raspberry Pi
• How to configure a DHCP reservation
• How to enable using the PiHole as a DNS server for all devices in the network

In my previous guide, we went over how to enable passwordless authentication for SSH to increase the security of our Raspberry Pi, as well as how to configure the SSH server to have improved security. If you've been following along, you should now have a system with the following properties:

• The only user that can connect over SSH is the pi user
• The Raspberry Pi only accepts incoming connections through public key authentication
• 5 minutes of inactivity causes your SSH session to be terminated by the server

These are all great practices to start with, and should we open up our Raspberry Pi so that we can connect over SSH from anywhere in the world, we should be protected from anyone guessing our password or exploiting any other potential accounts on our system to log in through SSH.

There are, however, a handful of things that our system would not be protected against. For example, if we forget the computer open on a table while at a coffee shop (if we ever truly get back to being able to hang out at a coffee shop) or if the computer gets stolen, it would then be possible for an attacker to access our Raspberry Pi. Adding a second factor will help abate more sophisticated attacks, which might be required due to your threat model.

How Does Multifactor Authentication Work With SSH?

SSH, and many other applications, can use something called PAM (which stands for Pluggable Authentication Module). This module allows various applications to use a common authentication layer, which can communicate with different modules. Different companies (or individuals) can then write these modules, which we can install through the apt package manager.

For this specific guide, we'll be using Google Authenticator to set up Time-based One Time Password (TOTP) through a phone application (such as Authy, Google Authenticator, or Microsoft Authenticator), as well as setting up a physical hardware key using YubiKey. With the way we will be configuring our PAM, you will be able to use either of those methods for two-factor authentication, but if you would like you are also able to set up triple-factor by requiring both of these methods instead of just one. Personally, this goes beyond what my threat model requires: I set up both methods so that one can act as a backup to the other.

Setting up Google Authenticator

While you might be familiar with Google Authenticator as a phone application that allows you to scan QR codes to add two-factor authentication to your accounts, there is also a Google Authenticator PAM library to allow you to create a QR code to scan on your phone to provide the TOTP codes. While these two applications share a name, they are not necessarily linked: you can use any application that supports TOTP to scan the resulting QR code.

To start, let's connect to our Raspberry Pi. If you followed along the last tutorial, you should be able to do so without writing in your password. Since my Raspberry Pi's hostname is steve, I can connect to it with the following command:

ssh pi@steve.local

Next, we need to install the PAM library, which we can do through our apt package manager:

sudo apt install libpam-google-authenticator -y

Once it has finished installing, we need to set up the TOTP application on a mobile phone. Personally, I use Google Authenticator for most of the 2FA I use for my accounts, so I will use it for this also, but any other authenticator would do.

When the download is completed, go back to your Pi terminal. Depending on how long it took your download to complete, you might have been timed out of your connection to the Raspberry Pi. This is good and expected behaviour: We do not want to allow idle sessions to live for very long. Once you are back in your RasPi shell, run the following command:

google-authenticator

This will create a new secret key and a QR code for you to scan. Open up your authenticator application, click on "Scan QR code", and point your phone camera at the terminal. You might have to increase your terminal size and/or scroll up.

Your terminal will also display "emergency scratch codes". These should be kept hidden and protected somewhere only you can reach, as they are codes you can use in lieu of the TOTP codes displayed on your application. Personally, I write mine as a secure note on my password manager, Lastpass.

You will also be prompted for a series of configuration choices. You can read through them and decide what is most suitable for your own setup and network. You can see my configuration choices in the screenshot below:

Now that you've got that set up, it's time to tell PAM to use the Google Authenticator as well as enabling SSH to properly allow PAM to run. Let's start by going back to our /etc/ssh/sshd_config file. As we did last time, make sure to create a backup of the current configurations by running sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.bak.

Open up the file now by using sudo nano /etc/ssh/sshd_config and make the following changes:

• Find the line ChallengeResponseAuthentication no and change it to ChallengeResponseAuthentication yes
• Find the line UsePAM yes, and add the line AuthenticationMethods publickey,keyboard-interactive below it.

Now press CTRL+S to save and CTRL+X to exit. Now it's time for us to tell PAM to use our Google Authenticator TOTP for login, and to prevent it from requesting the password.

First, make a backup of the configuration file by running sudo cp /etc/pam.d/sshd /etc/pam.d/sshd.bak. Then, open up the file by running sudo nano /etc/pam.d/sshd, and make the following changes:

• Find the line @include common-auth and change it to be #@include common-auth
  • This will "comment out" the line, which in turn will disable PAM from trying to use the password for authentication
• Immediately below the #@include common-auth line, add the line auth sufficient pam_google_authenticator.so
  • This will enable PAM to use the Google Authenticator module

Press CTRL+S to save the file and CTRL+X to close it. Then, run the following command to restart the SSH server:

sudo systemctl restart ssh

Now, before you close out of this session we should test to make sure our authentication is working appropriately.

Open up a new terminal window, and try to connect via SSH. You should be prompted to punch in your verification code (which is the TOTP that is now on your authenticator phone app). Put in the correct code, and it should let you in to your shell. If that is the case, congratulations! Your configurations work and you have now enabled TOTP multi factor authentication for connecting to your server.

If for some reason you cannot log in with your TOTP, you can go back to your original terminal session, and restore your original PAM and SSH configurations and try again. If the problem persists, I encourage you to reach out to me and I'd be happy to try to help you debug.

Setting up YubiKey for MFA

I'm a huge fan of hardware-based authentication. I wrote a little bit about how we used them when I interned at Google in my last blog post, and since working there I have purchased one of my own because I stand by their value. Additionally, while I am a security-conscious person, I am also a very lazy person by nature. I strongly believe in using multi factor authentication, but I also want it to be as simple as possible to use. Physical keys for MFA are perfect for this use: I can plug one in to my laptop when I log in, and whenever I need to prove my identity I just need to press a button. No need to pull out my phone, or copy over a code to my computer.

For those unfamiliar with what YubiKey is, they are a series of USB- & NFC-based security devices. The general premise is that you can plug one in to your computer, and by pressing a button you can generate a one-time passcode.

To use a Yubikey for MFA you will need your key identifier (which is part of your Yubico OTP that are generated by pressing the button on your Yubikey), and a Client ID from the Yubikey API.

Anatomy of a Yubico OTP

A Yubico OTP is comprised of two parts: The last 32 characters of the OTP are a randomly generated code, and the remaining characters at the beginning of the string are the identifier for the key. If you have a Yubikey, you can get the key identifier by cutting out the last 32 characters of a OTP that you generate, or you can run it through the python code below:

import sys

def extract_id(otp):
    return otp[:-32]

if __name__ == '__main__':
    print(extract_id(sys.argv[1]))

To run this code, copy and paste it into a file called otp.py then run the command python3 otp.py <OTP>, where <OTP> is a code you generated by pressing on your Yubikey.

Once you have your key ID, write it down somewhere as we will be using it soon.

Getting Your Yubico Client ID

Navigate to the Yubico API Key signup page and put in your email, then press on your Yubikey to generate a OTP on the "YubiKey OTP" input box. You will need to accept the terms and conditions to proceed, and I recommend at least glancing through them. Once you are done, you can press "Get API Key", and you will be given a Client ID and a secret. You can note down the secret somewhere safe, such as a password manager, but we will not be using it further. The thing that is most important for our use is the Client ID.

Now that we have the pre-requisites, we can start setting up the Yubico PAM module.

Configuring PAM to use your YubiKey

First, we need to install the PAM library that will be used to authenticate with the YubiKey. You can do so by running the following command:

sudo apt install libpam-yubico -y

Once that is finished, we need to write a file that will allow us to associate a specific key ID to a specific user on our system. Let's create a file that we can keep in the /etc/pam.d directory that has all the users and their associated keys. Run the following command:

sudo touch /etc/pam.d/yubiauth

Then open the file with nano. You should have a blank file now, and we need to populate it with our user ID, as well as the key ID, separated by the : character. For example, to associate the key with the ID abcdefg to our pi user, we would write the following:

pi:abcdefg

Then press CTRL+S to save, and CTRL+X to close the file.

We can also associate more than one key to a user, and associate keys with other users. To add a key to the same user, add :<KEY_ID> to the end of the line, and to add a new user you should place it on a new line and add in the username and key id (or key ids) you want to associate with that user. For example, to add the key xyzabc to our pi user and also associate it to a different user called test, we would have our final file look as such:

pi:abcdefg:xyzabc
test:xyzabc

This file will allow PAM to confirm that the OTP that you provide during login is actually connected to the account you are trying to use.

Next, open up the /etc/pam.d/sshd file, and add the following line above the Google Authenticator line.

auth sufficient pam_yubico.so id=<CLIENT_ID> authfile=/etc/pam.d/yubiauth

Replacing <CLIENT_ID> with the Client ID you got from the Yubico API key signup page.

Once that is complete, open up a new terminal window, and attempt to connect to your Pi through SSH. It should now prompt you to press your YubiKey to allow you to log in:

If you lose your YubiKey, or do not have it available, you will still be able to log in to your Raspberry Pi by using the Google Authenticator code that we set up earlier in the tutorial. You can see an example of that below:

If your want, or need, to have both of those set up simultaneously such that a user has to both have the YubiKey and the TOTP code you can simply edit the /etc/pam.d/sshd file, changing the word "sufficient" to "required" on the lines that we have added.

You might be wondering why we had to add the line above the one that was set up for the Google Authenticator. This is because we want the login to prompt for YubiKey first, and use the Google Authenticator more as a backup, as the YubiKey is more convenient. If this is not the case for you, and you want your SSH to prefer the Google Authenticator, you can swap the order of the lines. If you would like to disable either of those methods of 2FA, you can simply add a # to the beginning of the line. Note, if you want to disable both of the 2FA methods, you will also need to restore the old version of your /etc/ssh/sshd_config file, since your SSH will be expecting the multiple factors.

Wrapping Up

In this guide, we covered these concepts:

• How SSH can connect to the PAM modules to provide extra authentication
• How to enable multiple required factors for authentication
• How to install and set up Google Authenticator for using TOTP for our SSH sessions
• How to install and set up Yubico for hardware-based 2FA

At this point, our Raspberry Pi is fairly hardened against most common forms of attack: It does not use a password, it requires a private key that is encrypted in our system, and it requires the use of a secondary form of authentication.

For the next guide, we will be going over one incredibly cool software that we can run on our Raspberry Pi to improve our home networks by setting up a dedicated PiHole on our Raspberry Pi.

In the last blog post, we covered how to set up a headless Raspberry Pi, and we connected to our Raspberry Pi by using the Secure SHell (SSH) protocol. This is a really good protocol for connecting to servers, and is used pretty ubiquitously in the industry. However, with our current setup, our Raspberry Pi still suffers from one crucial issue when connecting through SSH: it relies on a password.

Passwords are something I have hated for quite some time. They are a simple (bad) solution to the complex problem of authentication. More often than not, they are easy to guess, regularly reused, and too easily we as human beings can be tricked into giving them away.

I believe it's important to understand why you should use SSH keys, and have at least some understanding as to how it works. I have included as part of this guide a quick primer on authentication and a very quick primer on public key cryptography. These are not important to setting up passwordless SSH access, so feel free to skip it if you already understand these concepts, or don't feel the need to read through them.

A Quick Primer on Authentication

There are three main ways you can be authenticated as a user:

• Something you know
• Something you have
• Something you are

Passwords would fall into the first category: something you know. Unfortunately, if your authentication relies only on something you know, it is often easy to steal or in some other way have it be compromised. One common way that this happens is through leaked passwords: If a service you once signed up for with the same password has their database be stolen, and they didn't take proper steps to secure that database, your password (regardless of how good it might be) could be compromised.

If you have an iPhone, or many of the more recent Android devices, you are likely familiar with the third category: with Face ID the iPhone is able to evaluate your face and determine if it is you, and with Touch ID (and the Android fingerprint sensors) your phone is able to use your fingerprint to determine identity.

You might also be familiar with the second category. It is becoming increasingly common for services to use multi-factor authentication, which uses more than one item to secure your account. Usually, this is something like a one-time password provided by applications such as Google Authenticator, Authy, or Microsoft Authenticator. A different method that uses the same basic principle is to use a hardware-based key, such as a YubiKey. When I was a Google intern, they provided hardware keys to secure our employee accounts, and it has been very successful in preventing account takeovers.

The goal of this guide is to move our SSH authentication from a password (something you know, which can be compromised) to something you have (your computer). The way we do this is through something called "Public Key Cryptography"

A Very Quick Primer on Public-Key Cryptography

I won't go over public-key cryptography in a lot of detail in this post, since it's a very complex topic that I could not hope to do justice in a single post, let alone a section of one. The most important thing to know at a high level is that public key cryptography uses a pair of very large numbers (which we call keys), one of which you must keep secret (the private key), and one of which you can give to others (the public key).

With your private key, you can sign some information to confirm that it came from you, and anyone that has a copy of your public key can verify the signature. This means that keeping the private key secret is very important, since anyone who has access to it can effectively impersonate you. However, unlike passwords, you never actually provide the private key to a service you are using, which means you can use the same pair of keys as much as you would like. Since your public key can be published anywhere safely (assuming your private key is secret), it can never render your account vulnerable if it is stolen from a service you use.

To authenticate with our Raspberry Pi, we will be using the ED25519 signing scheme, which is based on something called "Elliptic-curve Cryptography". If you're familiar with some amount of cryptography, you might be familiar with RSA, which is a commonly-used system, and might be wondering why we would use a different system. RSA is not considered to be secure for key sizes below 2048 bits, and compared to ED25519 it is substantially slower and has a substantially larger key size. ED25519 is still considered to be computationally secure, so it provides comparable protection (in that it cannot be broken, to our knowledge, with existing and expected near-future technology within the lifetime of the universe).

Why Should I Care?

That is a very good question. The first reason why you might care is that by switching over to using keys you will be able to log in without having to type a password each time. The convenience factor is the primary reason for most people. As an added perk, your system also becomes safer: it is easier to steal a password than it is to steal a specific file on your computer.

As a consequence, only the machines that you've previously uploaded SSH keys from will be able to connect to your RasPi.

Creating your SSH Key

The very first step in setting up public key based authentication is to actually create the key. If you're on Unix or Mac OS (this also includes using WSL/WSL 2 on Windows), you can run the following command to start the key creation process:

ssh-keygen -t ed25519

Select the file name or leave it blank to use the default filename (~/.ssh/id_ed25519). Then, set a passphrase for that key. This passphrase will be used to "lock" your private key, so that even if it gets stolen, the attacker would need to know that passphrase to access the key and subsequently access your Raspberry Pi.

This will generate two files: ~/.ssh/id_ed25519 and ~/.ssh/id_ed25519.pub. As you might guess, the .pub file is the public key, and the file with no extension is the private key.

Adding the SSH Key to the Agent

Since the goal is for passwordless authentication, we're going to be adding our SSH key to the SSH agent, as well as instructing it to use the Keychain (for Mac). If you are using Linux, you might have to look into a comparable solution, such as Seahorse for Ubuntu. For Windows users, you'll need to look up how to use PuTTY and PuTTYgen for this.

If you are on Linux, you will also have to make sure that the SSH agent is started up. In your ~/.bashrc file, add the line $(eval ssh-agent) > /dev/null to the end, which should start up the agent.

Now that the SSH agent is started and we have our key set up, add it to the agent with the following command:

ssh-add -K ~/.ssh/id_ed25519

This will prompt you for the passphrase to add it to the agent, and you will subsequently not need to type your password to use this identity file again until the next reboot.

Now, it is time for us to configure our SSH client. Create the file ~/.ssh/config with the following command and open it:

touch ~/.ssh/config && nano ~/.ssh/config

Now, copy the configuration below and paste it in your terminal:

Host *
    UseKeychain yes   # Add this line only if you're on MacOS
    AddKeysToAgent yes
    IdentityFile ~/.ssh/id_ed25519

This will enable you to use the ~/.ssh/id_ed25519 key for any machine you are attempting to connect with. If you want to create a more specific scope and only use that file for specifically connecting to your raspberry pi, you can set the "Host" field to whatever the hostname you configured in the previous guide. For example, since my Raspberry Pi has a hostname of "steve", I would write my configuration as such:

Host steve.local
    UseKeychain yes
    AddKeysToAgent yes
    IdentityFile ~/.ssh/id_ed25519

The important line for making sure you don't need to type the password almost ever again (for Mac users) is the "UseKeychain" line. This will allow the SSH agent to communicate with your native Keychain, which stores passwords securely on your system.

Adding the SSH Key to the Raspberry Pi

Transferring the SSH key to the host (the RasPi) is fairly simple. You can just run the following command, switching "steve.local" for the hostname of your Raspberry Pi.

ssh-copy-id pi@steve.local

You will be prompted for the password, and when you are done your key will be installed on the host system. It handles all of this for you.

Configuring the SSH Server

Once our key is installed, we just need to configure our server to require a known key, and disable password-based login. To start, make a backup copy of the current configurations with the command

sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.default

This will ensure that even if we mess up, we can always just return to the initial configuration that we know worked previously.

To start editing the file, run the command below, which will start up the text editor nano:

sudo nano /etc/ssh/sshd_config

nano is a very simple text editor that comes bundled with Raspberry Pi OS. There are some other popular ones, such as vim and my personal favourite, emacs. Ultimately, it's a matter of preference, but for very quick file edits we can use nano very quickly.

You may have noticed that we have had to run the last two commands with sudo, which you might remember from the previous guide that I mentioned is something you should be careful of doing. This is one of the cases where sudo (which stands for "super user do") is necessary: The SSH server is run as "root" (which is akin to an admin user), so we need to run these commands with elevated permissions (i.e. "running as an administrator") to make them work. The majority of the files that are in the /etc directory will be owned by the "root" user, so you'll need to run commands with sudo to change them.

Now that we can change our configurations, let's start making our SSH server a little more secure.

• Find the line #PermitRootLogin prohibit-password and change it to PermitRootLogin no
  • This will prevent the user "root" from ever being able to log in to SSH, which is generally desired since that is the user with highest permissions. Good practice is to use an account that has 'sudo' permissions, like our Pi account.
• Find the line #PasswordAuthentication yes and change it to PasswordAuthentication no
  • This will disable the use of passwords and require us to have the SSH key installed.
• Find the line #ClientAliveInterval 0 and change it to ClientAliveInterval 300
  • This will let the server check the client every 5 minutes to see if it is active
• Find the line #ClientAliveCountMax 3 and change it to ClientAliveCountMax 0
  • This will make it so that if the server determines the client is not active after 5 minutes, it will end the session
• At the top of the file, add the line AllowUsers pi
  • This will make it so that the only user that is allowed to login through SSH is the pi user. If you want to add additional users that can be used to remotely connect, make sure to add them to this line, separated by a space character.

When you are finished with these changes, press CTRL+S to save the file and CTRL+X to exit. Finally, we need to restart our SSH server to enable these configurations. You can do it with the following command:

sudo systemctl restart ssh

One last thing before we close out of our session! Open up a new terminal window (while leaving the current one open) and try to connect to the Raspberry Pi

ssh pi@steve.local

If you were able to connect without having to provide the password for the pi account, you are done! The Raspberry Pi will now allow you to log in without providing any password, while still allowing for protection against unauthorized access from any other machine.

To add more keys, you can do it in one of a few ways. If you would like to add more keys from the current machine, just create a new key and run ssh-copy-id again. Alternatively, if you want to add a key from a different machine, you can either:

• Revert back to the default configurations temporarily to add the new key:
  • Make a backup of your current configurations (sudo mv /etc/ssh/sshd_config /etc/ssh/sshd_config.bak)
  • Reload the default configurations (sudo cp /etc/ssh/sshd_config.default /etc/ssh/sshd_config)
  • Restart the SSH server (sudo systemctl restart ssh)
  • Add the new key the same way you did before (ssh-copy-id pi@steve.local)
  • Delete the old default configuration (sudo rm /etc/ssh/sshd_config)
  • Restore your proper settings (sudo cp /etc/ssh/sshd_config.bak /etc/ssh/sshd_config)
  • Restart the SSH server to restore your configs (sudo systemctl restart ssh)
  • This has the drawback that it requires you to lower the security of your pi, albeit just temporarily
• Or, you can send yourself the public key of the new machine to the one that can access the Raspberry Pi and copy it over:
  • Send yourself the .pub file somehow. Email is fine, as is using iCloud/Google Drive/OneDrive/etc
  • Save it to your home directory with the name id_ed25519_<machine_name> where <machine_name> is a name you choose for your computer
  • Run the following command:

cat ~/id_ed25519_<machine_name>.pub | ssh pi@steve.local "cat >> ~/.ssh/authorized_keys"

This will copy the public key to the Raspberry Pi's authorized keys without having to change your settings. Once the new key is installed on the system with the command above, the new machine will be able to log in!

Wrapping Up

In this guide we went over a number of key concepts. We covered:

• Basic concepts of authentication
• Basic concepts of cryptography/signing
• How to create an SSH key
• How to add the SSH key to the ssh-agent
• How to configure the SSH client to use Keychain
• How to install an SSH key to your Raspberry Pi server
• How to configure the SSH server
• How to install new keys to the SSH server
• How to restart the ssh server so it picks up our new configs

In the next guide, we'll go over how to enable two-factor authentication as part of connecting to our Raspberry Pi through SSH. While this is not necessarily required, it is a simple and quick way to add extra protection to our sessions.

Why headless?

While you can definitely use the Raspberry Pi as a desktop (which makes a fantastic first computer for a young tech enthusiast), I think one of the areas where the RasPi thrives is in being used as a very affordable, small home server. You can set it as a file share, an ad blocker, a backup server, or even a media streaming server! For all these purposes, it makes more sense to run the RasPi as a headless machine (which does not have a desktop environment installed in it). This will save you space, memory, and install time.

Getting Started

To follow along to this guide, you will need the following things:

1. A full-sized Raspberry Pi. I use the Raspberry Pi 3 Model B, but this should work for all models, other than the Zero, Zero W, and the Pico.
2. A suitable power supply for your Raspberry Pi
3. A MicroSD card. I’m using an 8 GB one I had laying around, but I’d recommend using a bigger one if you’re planning on installing lots of things on it
4. A MicroSD adapter. This can be a MicroSD to USB, or MicroSD to SD adapter, provided that your computer can read those
5. An ethernet cable to connect the RasPi to your router
6. A computer to flash the image onto the SD card

All of the above (except the computer to flash the image and the ethernet cable) should be available through one of the various kits that you can purchase. Some kits even include a pre-installed version of the Raspberry Pi OS in their SD card, though it most likely would include the desktop version.

Flashing the Image onto an SD Card

You can install the Raspberry Pi OS on your RasPi’s SD card in a couple of different ways. I’ve always previously used belenaEtcher to flash the images, since I’ve used it for more than just the Pi, but I was curious when I saw on the website that they now have a Raspberry Pi Imager that can take a lot of the heavy work out of the actual installation, so I decided to go with that.

Since we’re going to be running our Raspberry Pi as a headless machine, we want to make sure not to install a desktop environment. Typically, installing the “desktop” version of the OS will mean that you get an actual GUI, which can take up resources that we’re not going to be using. Instead, we want to pick the “lite”, or “server” versions of the OS since those typically only come with a command-line interface. When you open up the Imager, pick Raspberry Pi OS (Other), and select Raspberry Pi OS Lite (32-bit).

If you’re familiar with the specs of your board you might have noticed that if you have a Raspberry Pi 3 or 4, your device actually has a 64-bit processor. However, right now the Raspberry Pi OS only supports 32-bit images, with the 64-bit version out as a beta.

Once you have the OS picked out, just make sure you pick the correct disk to flash it on. If you have multiple SD cards, USB drives, etc. connected to your machine, you definitely want to avoid writing over the wrong one. Since I’ve only got my 8 GB MicroSD card connected, this wasn’t much of a problem for me. Once you’re ready to go, just click “Write” and select “Yes” on the dialog when it asks if you’re sure.

Once it is finished writing, the SD card will most likely be ejected from your system since the Imager thinks it’s ready to go. However, there’s one final step missing before we can pop the card into our RasPi: Enabling SSH so we can connect to it without having to connect to a monitor/keyboard.

Enabling SSH

I’m not going to go too in-depth with what SSH actually is. The important part is this: it stands for Secure SHell, and it is a way to (you guessed it) securely send commands to your shell.

To enable SSH on your RasPi on startup, you need to add a blank file called “ssh” to the Raspberry Pi’s boot image. If your SD card was ejected, just remove it and put it back in. I’m on a MacBook, so I can open my Terminal application (I use iTerm2) and type in the following command:

touch /Volumes/boot/ssh

This command will create the blank file on the “boot” volume. Confirm that it is actually in the SD card, and eject it when you’re done.

That’s it! Now, when we plug in our Raspberry Pi, it will start the SSH server which will let us connect to it remotely from the get-go. Put the MicroSD card in your RasPi, connect it to power, and plug in the ethernet on your RasPi and your router to connect it to your local network.

Connecting to the RasPi over SSH

The Raspberry Pi uses a service called Avahi for mDNS, which will try to make it easy for you to connect. If those words sound weird and scary for you, don’t worry about it: the general gist is that you should be able to connect without having to actually know the IP address of your RasPi.

Open up your Terminal application (if you’re on Linux or Mac OS), or your SSH client (such as PuTTY for Windows), and connect to pi@raspberrypi.local. On Mac and Linux, you can do this with the following command:

ssh pi@raspberrypi.local

Your terminal might say that it cannot determine the authenticity of the server. This is expected since this is the very first time you are connecting to this machine. Type “yes” and it will attempt to connect, and ask you for the password. The default password for the user pi is raspberry. Type it in, and it should show you the Raspberry Pi prompt, as well as a warning to change the default password.

You might assume that, since it is only in your local network, it is fine to keep the default password for the Raspberry Pi. While it is true that you should have a firewall from your router, it is always good practice to change all default passwords, since you never know what can happen. Maybe you share your network with other people in the household, or maybe someone can guess your wifi password, or through some other means they can gain access. Or, maybe, you want to be able to connect to your raspberry pi from anywhere in the world and want to open it up to connect from outside your home. In all these cases, we should add some security. Make sure you change the password immediately. You can do this by typing passwd on the Raspberry Pi prompt. Note, the password won’t be visible on the terminal while you type it.

Now it’s time to actually configure the settings on our Raspberry Pi!

Configuring our Raspberry Pi

The Raspberry Pi OS comes with a nice tool for setting up a lot of the basic settings. To start it up, run the following command on your RasPi shell:

sudo raspi-config

This command will run with elevated permissions, which we need to change system settings, but you should always be careful of when you are using “sudo”. You should see your terminal window change onto a menu, like the screenshot below. Since we’re running it headless, we can definitely ignore the Display Options menu and a handful of other configurations.

System Options

Some settings on this menu you might want to change:

1. Wireless LAN - If you’re hoping to use this Raspberry Pi through wifi, you can set the Wifi information by going through the wizard. You’ll need to know the actual name of your network and the passphrase for it.
   • Personally, I recommend leaving the Raspberry Pi on ethernet since it is a lot more stable than wifi, but if you need to connect from somewhere you can’t run a cable to, wifi makes sense.
2. Password - If you haven’t yet changed your password, this is the time. You can change your password from this menu.
3. Hostname - If you’re going to have more raspberry pis in your network, changing the hostname is a good idea. I like to give my RasPis a nickname, so I called mine “Steve”. If you’re using it for a specific use (such as pihole), giving it a meaningful name might also be a good idea.
4. Boot / Auto Login - If you installed the full graphical interface, that’s okay. If you mostly want to use it as a headless machine, but sometimes want to be able to boot up the GUI, you can set to boot into command line. I’d recommend making sure that you do not boot with autologin, for security reasons.
5. Network at Boot - If your RasPi is always being used for network-related tasks, it might interest you to make sure that your Raspberry Pi is only booted when there is a network connection present.

Interface Options

It is good practice to disable all interfaces you do not need. Since I’m not using my RasPi for messing with any prototyping or sensing, and I don’t want a graphical environment, I disable every interface other than SSH.

While it’s unlikely that anyone would break into my home and add an SPI device or a camera to my raspberry pi without me noticing, disabling these kernel modules from being loaded prevents any problems from occurring due to bugs or vulnerabilities that could eventually be introduced or discovered in these modules unexpectedly.

Localization Options

As far as localization goes, I personally only set the timezone. The Pi is usually pretty good at picking up the other things, and since English is my main language I don’t have to really change much else.

If you’re planning on using wifi, you can set your WLAN country in this menu, but you’ll also be asked that already when you configure wifi on the device so you should not need to set it independently.

Advanced Options

The most critical option to select here is “Expand Filesystem”. The reason for that is that flashing the image on often does not expand it to include the entire card size, and you’ll want to be able to use all of the space on that card (since presumably you won’t be using it for anything else at this point).

If you plan on using things that require you to know the network interface, the “Network Interface Names” option should help you, but your mileage may vary. In my case, when I set “predictable network interface names” to be enabled, it made the ethernet interface be a very long string, whereas disabling it set the interface name to “eth0”, which was what I had expected.

Wrapping Up

Once you’re finished with these settings, you can navigate to “Finish” and press Enter. It will prompt you for reboot, which is exactly what we need to do considering that we changed the partition size on our SD card.

After the Pi finishes rebooting, you’re all set! You should now be able to SSH into your raspberry pi whenever you’d like from your local network.

As a review, this is what we’ve gone over in this guide:

1. Using the Raspberry Pi Imager to install a Lite version of the Raspberry Pi OS on a micro SD card
2. Enabling the SSH server on the Raspberry Pi at boot time
3. Connecting to the Raspberry Pi over SSH through Ethernet
4. Changing the password on a user account on the Raspberry Pi
5. Configuring system settings on the Raspberry Pi with raspi-config

For the next guide, we will be going over how to set up passwordless authentication for your Raspberry Pi, which will allow us to connect through how to connect with SSH without having to type the password in the terminal every time.