Let’s boot some C code¹ with riscv64 using qemu: riscy²

After about half a year of os development abstinence. I fell into the riscv-os-dev-rabbit-hole. Those risc, arm, riscv buzzwords are in my news bubble for a while. Now it’s time to get my hands dirty.

Contents

• RISC-V vs. ARM vs. ARM64 vs. AArch64
• Getting started - barebone tutorial
  • Build
• Interact with the outside world
  • Devices
  • Drivers
  • Magic address
  • UART
  • RTC
• Summary

RISC-V vs. ARM vs. ARM64 vs. AArch64

ARM64 and Aarch64 are both the same³: an implementation of the proprietary RISC architecture family ARM. RISC-V on the other side is an open-source architecture.

Those RISC’s are fairly modern⁴ and faster than CISC due to their nature of being reduced instead of complex. I’m not sure what this “faster” actually means while a CISC operation may be slower than a single RISC operation, but with RISC more operations are needed.

Getting started - barebone tutorial

I initialized my riscy bootloader using the os.dev risc-v bare bones tutorial containing everything needed for a hello world:

• entry.S: Entry point and initial bootloader preparing everything before calling the main function in the C kernel code.
• kernel.c: C kernel printing hello world and reading from the keyboard by UART.
• linker.ld: Defines the memory layout of the resulting binary image that can be booted by qemu.

Build

Despite the different binaries names and the dropped -lgcc flag5, the commands look the same as in the Tutorial:

$ riscv64-linux-gnu-gcc -Wall -Wextra -c -mcmodel=medany kernel.c -o kernel.o -ffreestanding
$ riscv64-linux-gnu-gcc -c entry.S -o entry.o
$ riscv64-linux-gnu-ld -T linker.ld -nostdlib kernel.o entry.o -o kernel.elf

Let’s boot it via qemu-system-riscv64 -machine virt -bios none -kernel kernel.elf -serial mon:stdio.

\o/

Interact with the outside world

Devices

Qemu provides a list of supported devices for our machine type virt. We also can make out our UART device.

But how can we make use of them?

Drivers

Looking at the Tutorial, the driver consists of a magic address and… well actually nothing more than read and write from it:

unsigned char * uart = (unsigned char *)0x10000000; 
void putchar(char c) {
    *uart = c;
}

int main(){
  putchar(*uart);
}

How can this make any sense?

Magic address

The address is actually well-defined by Qemu’s machine type for the UART device. As far as I understood it, the address-space of the devices are mapped into our address space and directly usable. Qemu can dump all those devices and addresses which can be made readable using the device-tree-compiler, as I found out via this riscv-from-scratch blog post.

And again, we can can make out our UART device with some additional information:

chosen {
  bootargs = [00];
  stdout-path = "/soc/uart@10000000";
};
...
uart@10000000 {
  interrupts = <0x0a>;
  interrupt-parent = <0x03>;
  clock-frequency = "\08@";
  reg = <0x00 0x10000000 0x00 0x100>;
  compatible = "ns16550a";
};

• uart device emulates the ns16550A IC
• address spaces starts at magic address 0x10000000
• qemu uses this device for serial stdin/stdout

UART

The Universal Asynchronous Receiver-Transmitter is used for serial console. Its implementation is fairly simple.

RTC

rtc@101000 {
  interrupts = <0x0b>;
  interrupt-parent = <0x03>;
  reg = <0x00 0x101000 0x00 0x1000>;
  compatible = "google,goldfish-rtc";
};

This is the google goldfish rtc device mapped at 0x101000. Qemu has a documentation about the rtc device and the linux kernel has a pretty small implementation.

I came up with something like this:

uint64_t time() {
  volatile uint32_t *rtc = (uint32_t *) 0x101000;
  uint64_t time_low = rtc[0];
  uint64_t time_high = rtc[1];
  return (time_high << 32 | time_low);
}

Which lead to the first ever nanosecond unix timestamp printed to the riscy console: 1707410202187052000⁶.

Inline assembler

Incrementing a number should be a simple start, right? Well, unlike x86’s inc, there is no pedant here. But addi gotcha covered: addi result, a, number where result and a are variables and number is a so-called imediate (literal) value. The inline syntax is a bit odd since it breaks the usual format we know from C/printf. This SO Post helped me to find the right syntax.

int inc(int n) {
  volatile int result;
  asm volatile(
      "addi %0, %1, 1;"
      : "=r" (result)
      : "r" (n)
      );
  return result;
}

The keyword volatile is recommended to prevent compiler optimizations.

Summary

tl;dr:

1. add //usr/bin/gcc "$0" && exec ./a.out "$@" to the first line of your example.c
2. make executable: chmod +x example.c
3. run it: ./example.c

I’ve a folder called lab for those quick ‘n’ dirty programs to test some language features or other interesting behavior. While golang programs can be easily executed via go run <go file> and this is not the case for c programs needing a built step. There is AFAIK no such command like gcc run.

Shebang

Linux’s execve(2) syscall supports shebangs. When the first line starts with #! aka shebang, it will get parsed by the kernels binfmt_script module and used for executing the rest of the file. This enables to run various non-binary scripts (bash, python, ruby, ) like ./program without using the interpreter.

Can we use a shebang for c source files? tl;dr: nope, as far as I tried this is not possible.

I’ve ended up with something like:

#!/usr/bin/tail -n+2 main.c /tmp/main.c && /usr/bin/gcc /tmp/main.c -o main && ./main #
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char *argv[]) {
  for(int i = 0;i<argc; i++) {
    printf("%d: %s\n", i, argv[i]);
  }
}
/*
 * $ ./main.c 1 2 3
 * /usr/bin/tail: invalid number of lines: ‘+2 main.c /tmp/main.c && /usr/bin/gcc /tmp/main.c -o main && ./main #’
 */

This doesn’t work while everything after tail gets interpreted as one argument.

binfmt fallback

Just before I was about to give up, I’ve found an SO post suggesting:

//usr/bin/clang "$0" && exec ./a.out "$@"

I’ve adapted it to my use-case and voila:

//usr/bin/gcc "$0" && ./a.out "$@";exit
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char *argv[]) {
  for(int i = 0;i<argc; i++) {
    printf("%d: %s\n", i, argv[i]);
  }
}
/* 
 * $ ./main.c 1 2 3
 * 0: ./a.out
 * 1: 1
 * 2: 2
 * 3: 3
 */

• //: c single-line comment to avoid gcc to interpret this line
• //usr/bin/gcc "$0": compile file itself using the absolute path to the compiler, multiple slashes will automatically compacted to one, so // is no problem
• exec ./.aout "$@": execute the built binary (named a.out per default) with all arguments which were initially added to the program call ./program arg1 arg2..., exec replaces the current process which is important as the rest of the c source file otherwise gets interpreted

Well, I guess this works because after binfmt fails to detect any elf-binary or shebang, bash executes it in the current context, which is actually bash.

Today, I’ve checked out the BDD¹ testing framework ginkgo with the matcher library gomega for integration tests with go. The motivation was to find an alternative for my go-to testing framework rspec. Although in ruby, I anyway used it for go projects like blogctl.

• rspec - Were I came from
• Integration test frameworks in go
• ginkgo
  • Setup
  • Show version
  • Real tests
• Thoughts

rspec - Where I came from

The rspec DSL together with ruby itself makes it easy to write down tests pretty quickly. At least for me its quicker than doing the same in go (until now?):

require 'rspec'
require 'open3'

describe 'CLI' do
  context 'Trivial test cases' do
    it 'shows a version' do
      out, err, _ = Open3.capture3("blogctl --version")
      expect(err).to eq ""
      expect(out).to match(/BuildVersion:/)
      expect(out).to match(/BuildDate:/)
    end
  end
end

Integration test frameworks in go

There are a bunch of testing frameworks on the awesome-go list that look interesting for BDD integration testing.

• Active development: last contribution was within the last year
• Specify tests within go: The test can be specified within go.
• Various nestable containers: Analog to rspecs decribe and context container that can be nested.
• (Before|After)(All|Each) scripts: rspec alike setup and tear down functions for all and test-wise
• ?: I skipped evaluating this feature as it wasn’t obvious from the readme and another feature was unsupported anyway.

ginkgo

Ginkgo offers a similar Given-When-Then approach after looking at the example in the readme. Gomega is the matcher library, used by ginkgo. They’re both from the same author.

I’ve used my renaming tool r to check it out.

Setup

// test suite
// r_suite_test.go
package r_test

import (
  . "github.com/onsi/ginkgo/v2"
  . "github.com/onsi/gomega"
  "testing"
)

func TestBooks(t *testing.T) {
  RegisterFailHandler(Fail)
  RunSpecs(t, "R Suite")
}

// spec
// r_test.go
package r_test

import (
  . "github.com/onsi/ginkgo/v2"
  . "github.com/onsi/gomega"
)

var _ = Describe("R", func() {
})

Show version

package main_test

import (
	"os"
	"os/exec"
	"path/filepath"

	"github.com/bsm/gomega/gbytes"
	"github.com/bsm/gomega/gexec"
	. "github.com/onsi/ginkgo/v2"
	. "github.com/onsi/gomega"
)

var _ = Describe("R", func() {
	When("binary is used correctly", func() {
		It("shows version", func() {
			session, err := gexec.Start(
				exec.Command("ci-build/r", "--version"),
				GinkgoWriter,
				GinkgoWriter,
			)
			Expect(err).ShouldNot(HaveOccurred())
			Eventually(session).Should(gexec.Exit())
			Expect(session.Out).Should(gbytes.Say(`(?m)BuildVersion:.*\nBuildDate:`))
			Expect(session.Err).Should(gbytes.Say(""))
		})
	})
})

• provides containers like Describe, When and It similar to rspecs describe, context and it
• few more lines, words and brackets, as expected in go
• gomega provides so-called Asynchronous Assertions like Eventually which checks the expection periodically for some time, which is very handy
• helper libs like gexec and gbytes simplify the matcher that would otherwise be a bit more fiddly in go
• gbytes directly awaits the saying to be a regex, like a charm!

Real tests

The following tests use before and after scripts that set up the temporary environment for some real2 tests:

package main_test

import (
	"os"
	"os/exec"
	"path/filepath"

	"github.com/bsm/gomega/gbytes"
	"github.com/bsm/gomega/gexec"
	. "github.com/onsi/ginkgo/v2"
	. "github.com/onsi/gomega"
)

var _ = Describe("R", func() {
	When("binary is used correctly", Ordered, func() {
		var tempFile *os.File
		var tempDir string

		BeforeAll(func() {
			var err error
			tempDir, err = os.MkdirTemp(os.TempDir(), "")
			Expect(err).Should(BeNil())
		})

		AfterAll(func() {
			os.RemoveAll(tempDir)
		})

		BeforeEach(func(ctx SpecContext) {
			var err error
			tempFile, err = os.CreateTemp(tempDir, "")
			Expect(err).ShouldNot(HaveOccurred())
		})

		It("renames file", func() {
			renamed := filepath.Base(tempFile.Name()) + "2"
			session, err := gexec.Start(
				exec.Command("ci-build/r", tempFile.Name(), renamed),
				GinkgoWriter,
				GinkgoWriter,
			)
			Expect(err).ShouldNot(HaveOccurred())
			Eventually(session).Should(gexec.Exit())

			_, err = os.Stat(tempFile.Name())
			Expect(err).To(MatchError(os.IsNotExist, "It is not existing"))

			_, err = os.Stat(filepath.Join(tempDir, renamed))
			Expect(err).Should(BeNil())
		})

		It("shows version", func() {
			session, err := gexec.Start(
				exec.Command("ci-build/r", "--version"),
				GinkgoWriter,
				GinkgoWriter,
			)
			Expect(err).ShouldNot(HaveOccurred())
			Eventually(session).Should(gexec.Exit())
			Expect(session.Out).Should(gbytes.Say(`(?m)BuildVersion:.*\nBuildDate:`))
			Expect(session.Err).Should(gbytes.Say(""))
		})
	})

	When("binary is used incorrectly", func() {
		It("fails without any argument", func() {
			command := exec.Command("ci-build/r")
			session, err := gexec.Start(command, GinkgoWriter, GinkgoWriter)
			Expect(err).ShouldNot(HaveOccurred())
			Eventually(session).Should(gexec.Exit())
			Expect(session.Out).Should(gbytes.Say("usage: r <path> <new_filename>"))
			Expect(session.Err).Should(gbytes.Say(""))
		})
	})
})

• Ordered container caveat: When defining before scripts, the superior container (which is When in this case) needs to have the Ordered Decorator set. Looking at their example, they assume that resources are reused outside of one It, as some variables are defined in the superior container. But I’m not sure why it can’t just run all before’s in hierarchical order like rspec can do it…
• Despite the previous point, there are before and after scripts for suite, all and each. It just lacks in rspec’s around.

Thoughts

While falling into the routing engine rabbit hole planing my next hiking trip via nextcloud maps, I needed to upload a big file to my server. Upload bandwidth sucked as it mostly does in Germany¹. Well scp is my friend I guess:

$ scp jakobsweg.tar.gz root@my.server:~/
lost connection

Oops, after ten minutes with slow progress, the upload suddenly stopped due to a connection error.

Problem

Unfortunately scp can’t continue an interrupted upload. There are some StackOverflow solutions about using rsync but this solution using dd with its skip option got my attention.

Upload what’s missing

The solution suggest to just skip the already written bytes and append the rest to the destination file. Since this command is from the server perspective and my local machine is in a NAT’ed network which isn’t accessible from the www, I did the steps manually:

# checkout sizes of the source and aborted destination file
server $ ls jakobsweg.tar.gz
-rw-r--r--  1 root root 490766336 Jan 14 14:26 jakobsweg.tar.gz
local $ ls jakobsweg.tar.gz
-rw-r--r-- 1 raphael raphael 1102126303 Jan 14 14:18 jakobsweg.tar.gz
local $ UPLOADED=490766336
local $ REMAINING=$((1102126303 - $UPLOADED))
# copy source file locally but skip the already transmitted bytes, divide remaining by something to raise the blocksize
# while copying with a blocksize takes ages, better fill up the filesystem alignment
local $ dd if=jakobsweg.tar.gz of=jakobsweg.tar.gz.part2 skip=$UPLOADED count=$(($REMAINING/1024) bs=1024 status=progress
# upload the remaining data as usual
local $ scp jakobsweg.tar.gz.part2 root@my.server:~/
# append the remaining data to the aborted file
remote $ cat jakobsweg.tar.gz.part2 >> jakobsweg.tar.gz
# compare checksum -> tada
local $ shasum jakobsweg.tar.gz
a8b00b334319932b5794595f3eafc1c5bedfaa35  jakobsweg.tar.gz
remote $ shasum jakobsweg.tar.gz
a8b00b334319932b5794595f3eafc1c5bedfaa35  jakobsweg.tar.gz

I recently got a new motivation to update my vocabulary service verbose. Each word can have an example sentence together with its translation now. Verbose has also a new edit page.

Example support

When learning new words it could be useful to have a helpful example. This adds more context to the translation. Having an example is also useful for learning a whole new language, like spanish in my case. I barely know 100 words, where an example sentece would not only contain one unknown/hard to remember word but multiple.

To follow the KISS principle together with my I-only-add-a-word-per-month-and-view-it-once-a-day use-case, I still use my json file and added two fields containing the original example sentence such as its translation.

Edit support

In the past, if I made a typo, I ssh’ed to the server, opened the json file and corrected the word manually. However, having a new “column” which is initialy empty for each word, I felt the need of edit each entry via the service itself. I’ve added another site that just POST’s the changes. Unfortunately I can’t use PUT, which would be the right method to modify a resource, since this is not allowed.

I’m currently learning Spanish for my next vacation². While Duolingo does an impressive job of gamifying the learning process, I thought about combining this fairly unknown world with one that I’m familiar with.

I use Linux and its utils a lot. I like to read man pages and I know a bunch of them. Why not switch the system language to Spanish then?

Well, maybe because 100 words is not enough… but in a fire-and-forget environment like a virtual machine or Docker container, it should be safe.

Contents

• Dockerfile
  • Intermezzo - Minimized containers
• Groping in the dark, anticipating light
  • apt update
  • apropos .
  • man bash
• Today I learned

Dockerfile

After some fiddling with locale I came up with this Dockerfile:

FROM ubuntu:latest
RUN yes|unminimize
RUN apt update && \
    DEBIAN_FRONTEND=noninteractive apt install -y locales \
                                                  man-db \
                                                  manpages-es \
                                                  manpages-es-dev \
                                                  language-pack-es
RUN sed -i -e 's/# en_US.UTF-8 UTF-8/es_ES.UTF-8 UTF-8/' /etc/locale.gen && \
    dpkg-reconfigure --frontend=noninteractive locales && \
        update-locale LANG=es_ES.UTF-8
ENV LANG es_ES.UTF-8

• unminimize Ubuntu container so man pages works
• install man-db and their Spanish translations
• install the language pack containing the translations for the system
• replace the language where needed

Intermezzo - Minimized containers

The Ubuntu container […] is a minimal install of Ubuntu[…] and needs to get unminimized in order to be “human-friendly”.

Otherwise, man bash prints:

This system has been minimized by removing packages and content that are
not required on a system that users do not log into.

To restore this content, including manpages, you can run the 'unminimize'
command. You will still need to ensure the 'man-db' package is installed.

Actually /usr/bin/man is a bash script that only outputs this text.

What does the unminimize actually do? Tell me, cat $(which unminimize): I’ve put the output into a gist. It basically enabled documentation and man-pages again.

Groping in the dark, anticipating light

apt update

root@1e7e70970891:/# apt update
Des:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB]
Des:2 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [1.046 kB]
Obj:3 http://archive.ubuntu.com/ubuntu jammy InRelease
Des:4 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB]
Obj:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
Des:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1.599 kB]
Des:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1.305 kB]
Descargados 4.179 kB en 6s (755 kB/s)
Leyendo lista de paquetes... Hecho
Creando árbol de dependencias... Hecho
Leyendo la información de estado... Hecho
Todos los paquetes están actualizados.

apropos .

apropos . is a great way to display one tool per line, including a description3. Some favorites of mine:

echo (1)             - muestra una línea de texto
futex (7)            - bloqueos rápidos en espacio de usuario
hostname (7)         - descripción de la resolución del nombre de equipo
man (1)              - interfaz de los manuales de referencia del sistema
memccpy (3)          - copia un área de memoria
memcmp (3)           - compara areas de memoria
memcpy (3)           - copiar area de memoria
memmem (3)           - localiza una subcadena
memmove (3)          - copia un área de memoria
memset (3)           - rellena una zona de memoria con bytes repetidos
md5sum (1)           - genera y verifica sumas de comprobación MD5
stpcpy (3)           - copia una cadena devolviendo un apuntador a su final
strcasecmp (3)       - comparan dos cadenas ignorando si son mayúsculas o minúsculas
strstr (3)           - localiza una subcadena
strchr (3)           - localizan un carácter en una cadena de ellos
strdup (3)           - duplican una cadena de caracteres
strlen (3)           - calcula la longitud de una cadena de caracteres

Especially all those C mem* and str* functions were fairly easy to anticipate when listed together:

• caracteres: character, personality, both English and Spanish words seem to have the same origin, while both can be a “written symbol”, such as “personality”
• copia: copy
• compara: compare
• interfaz: interface
• repetidos: he repeats, male form of repetide
• texto: text
• sumas: you add, join

man bash

(lista)       lista se ejecuta en un subshell
{ lista; }    lista se ejecuta en el entorno actual de la shell
((expresión)) La expresión se evalúa de acuerdo a las reglas descritas abajo bajo la sección EVALUACIÓN ARITMÉTICA.

Today I learned

tl;dr

var hashSet map[string]struct{}

Introduction

The term HashSet is a data structure from the Java universe describing a value-less HashMap¹. It is perfect when uniqueness is needed but avoids the overhead of a value. Using golang they can be constructed using an ordinary map and an empty struct:

var hashSet map[key]struct{}

Use Cases

The key can be any supported type. Interestingly, next to the primitive ones (int, string,…), struct types are also supported. This comes quite handy when making custom cache keys.

Visited nodes

When searching through a grid or a graph, a HashSet can be used to avoid loops by caching already seen points:

type Point struct{
  x,y int
}

var visited map[Point]struct{}

Uniqueness

Another use-case would be to ensure uniqueness over a dataset. E.g. the bash command uniq reads a stream from stdin and only prints every line once to stdout:

package main

import (
    "bufio"
    "fmt"
    "os"
)

func main() {
    set := make(map[string]struct{})

    scanner := bufio.NewScanner(os.Stdin)
    for scanner.Scan() {
        if _, found := set[scanner.Text()]; !found {
            set[scanner.Text()] = struct{}{}
            fmt.Println(scanner.Text())
        }
    }
}

Turning…

 I've experiments to run.
 There is research to be done.
 On the people who are
 still alive.
 And believe me I am
 still alive.
 I'm doing science and I'm
 still alive.
 I feel fantastic and I'm
 still alive.
 While you're dying I'll be
 still alive.
 And when you're dead I will be
 still alive.

…via cat lyrics| go run unique.go to…

I've experiments to run.
There is research to be done.
On the people who are
still alive.
And believe me I am
I'm doing science and I'm
I feel fantastic and I'm
While you're dying I'll be
And when you're dead I will be

References

• example
• Hashicorp go-package for more sophisticated HashSet usages: go-set

How to avoid mv /this/is/a/long/path/to/my/guinea/pig/trfel.pig /this/is/a/long/path/to/my/guinea/pig/trueffel.pig and just do something like <cmd> /this/is/a/long/path/to/my/guinea/pig/trfel.file trueffel.pig?

Maybe I’m totally off today¹, or this naive question hasn’t a navie answer. Anyway, I couldn’t come up with a straight-forward solution within my bash².

I’d like to have an easy memorable solution without pattern matching or syntax.

DIY

I came up with my own renaming tool, just wrapping the os.Rename call with the right parameters:

package main

import (
	"fmt"
	"os"
	"path/filepath"
)

func main() {
	if len(os.Args) != 3 {
		fmt.Println("usage: r <path> <new_filename>")
		return
	}

	newPath := filepath.Join(filepath.Dir(os.Args[1]), os.Args[2])

	if err := os.Rename(os.Args[1], newPath); err != nil {
		fmt.Printf("error renaming %s to %s: %s\n", os.Args[1], newPath, err)
		return
	}

	fmt.Printf("%s -> %s\n", os.Args[1], newPath)
}

Which works like:

> r /home/jane/dev/r/main.c main.go
/home/jane/dev/r/main.c -> /home/jane/dev/r/main.go

SO solutions

I found this SO Post afterwards, but wasn’t satisfied. The acknowledged answer using mv and brace-expansion only works if the suffix need to be renamed. The next answer with rename is imo not intuitive. I just want to rename without any generic/pattern/expansion stuff.

How long does it take for the failover mechanism to fail-over? How long does it take for my server to be up again?

I recently had the challenge of measuring downtime durations. ping on the one hand “makes me see” when paket loss occurs and vanishes but lacks in aggregation. mtr on the other side aggregates the replies, but not for the metric I was looking for.

So I made up my own tool namely downtime. Aside the plain measurement logic, I tried a few new things like external lib interfaces and mocks which will shortly be addressed in the end.

• How it works
• Example
• Implementation design
  • External lib interface
  • Mocks
  • Extensibility

How it works

The icmp logic is based on my colleagues mtr go-rewrite providing me the most high-level icmp interface I could find. This enables me to use raw icmp pakets without caring about the low level stuff like encoding the PID into the request body to fish out the corresponding replies from the ocean of icmp traffic.

Example

$ downtime --target 23.88.32.x
start: 2023-09-22 12:27:31.751644064 +0200 CEST m=+11.151409405
end: 2023-09-22 12:27:49.278929522 +0200 CEST m=+28.678694794
duration: 17.527285389s

Implementation design

External lib interface

Define an interface for an external lib only containing needed functions. E.g. icmp.go for my dependency to tonobo’s SendICMP. While beeing a bit verbose in the first place, it has two advantages:

1. The dependency to the external lib is documented at one place. Usually you’d need to scan the whole code base for the external lib to check whcih functions are in use.
2. Testing of complex external functions gets a lot easier. E.g. my unit test don’t need to do real icmp stuff using mocks.

Mocks

Interfaces can easily be mocked using a lib like mockery. E.g. icmp mock for my icmp interface. As mentioned previously the mock can be used to test the lib without calling the external library. The goal of testing should be to test the own and not the external code.

Extensibility

In linux,“everything” is a file¹. Regardless if its a regular file, directory, socket or even a device. Know how to observe open files, attached to a process, can become quite handy for debugging a linux host.

• Files and file descriptors
• Observe
  • Wich process opened the file?
  • Which files do a process holding?²
  • Is my server already listening for incoming connections?
  • Does my process still hold fds to deleted files?

The first theoretical part will introduce file descriptors while the second part will give some practical examples.

Files and file descriptors

Whenever a program opens a file via a syscall from the open() family, the kernel returns a reference number that is unique for that process. This number is bettern known as file descriptor or fd for short. The program can act on that file (read, write, close,…) by using the fd from the open call.

Imagine a program like this:

// gcc example.c -o example
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <fcntl.h>

int main(int arc, char *argv[]){
    open("example.c", 0);

    while(1){sleep(1000);}
}

This program will open a file (itself for simplicity) and goes into an infinite loop in order to stay alive for observability.

On open the system creates a procfs entry under /proc/<pid>/fds/<fd> . Listing the proc dir for our process³ looks something like this:

$ ls -la /proc/1234/fd
total 0
dr-x------ 2 raphael raphael  0 Jun 29 21:11 .
dr-xr-xr-x 9 raphael raphael  0 Jun 29 21:11 ..
lrwx------ 1 raphael raphael 64 Jun 29 21:11 0 -> /dev/pts/0
lrwx------ 1 raphael raphael 64 Jun 29 21:11 1 -> /dev/pts/0
lrwx------ 1 raphael raphael 64 Jun 29 21:11 2 -> /dev/pts/0
lr-x------ 1 raphael raphael 64 Jun 29 21:11 3 -> /tmp/example.c

Observe

Wich process opened the file?

lsof got you covered:

$ lsof /tmp/example
COMMAND PID  USER  FD   TYPE DEVICE SIZE/OFF     NODE NAME
example 444 raphael txt    REG   0,40    23656 26759037 /tmp/example

Which files do a process holding?

It’s lsof again. The opened example.c can be seen at last with fd 3 linking to the original file. The other three fd’s 0 , 1 and 2 are the three pts devices for stdin, stdout stderr that get automatically created by the OS for any new process.

$ lsof -p 444
COMMAND PID    USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
example 444 raphael  cwd    DIR   8,32     4096 267422 /tmp/
example 444 raphael  rtd    DIR   8,32     4096      2 /
example 444 raphael  txt    REG   8,32    16088  46778 /tmp/example
example 444 raphael  mem    REG   8,32  2029592 142704 /usr/lib/...
example 444 raphael  mem    REG   8,32   191504 142576 /usr/lib/...
example 444 raphael    0u   CHR  136,0      0t0      3 /dev/pts/0
example 444 raphael    1u   CHR  136,0      0t0      3 /dev/pts/0
example 444 raphael    2u   CHR  136,0      0t0      3 /dev/pts/0
example 444 raphael    3r   REG   8,32      196  46772 /tmp/example.c

Is my server already listening for incoming connections?

Let’s spin up a tcp server waiting for incoming connections using ncat:

$ ncat -l 8080

Using lsof again, we can observe that ncat is even doing dual-stack waiting for incoming ipv4 and ipv6 connections, as one can see in the forelast entries.

$ lsof -p 532581
COMMAND    PID  USER   FD   TYPE             DEVICE SIZE/OFF     NODE NAME
ncat    532581 raphael  cwd    DIR               0,40      450 27369190 /tmp/example
ncat    532581 raphael  rtd    DIR               0,38      206      256 /
ncat    532581 raphael  txt    REG               0,38   435520  9123417 /usr/bin/ncat
ncat    532581 raphael  mem    REG               0,36           9123417 /usr/bin/ncat (path dev=0,38)
ncat    532581 raphael  mem    REG               0,36           9000771 /usr/lib64/...
ncat    532581 raphael    0u   CHR              136,1      0t0        4 /dev/pts/1
ncat    532581 raphael    1u   CHR              136,1      0t0        4 /dev/pts/1
ncat    532581 raphael    2u   CHR              136,1      0t0        4 /dev/pts/1
ncat    532581 raphael    3u  IPv6            9561146      0t0      TCP *:webcache (LISTEN)
ncat    532581 raphael    4u  IPv4            9561147      0t0      TCP *:webcache (LISTEN)
ncat    532581 raphael    6u  unix 0x0000000014cd383f      0t0    31426 type=STREAM (CONNECTED)

Let’s go one step further and establish a connection to ncat by browsing to localhost:8080. We can see that ncat closed the unused listener and from which address+port the connection is coming from:

$ lsof -p 532581
COMMAND    PID  USER   FD   TYPE             DEVICE SIZE/OFF     NODE NAME
ncat    532581 raphael  cwd    DIR               0,40      450 27369190 /tmp/example
ncat    532581 raphael  rtd    DIR               0,38      206      256 /
ncat    532581 raphael  txt    REG               0,38   435520  9123417 /usr/bin/ncat
ncat    532581 raphael  mem    REG               0,36           9123417 /usr/bin/ncat (path dev=0,38)
ncat    532581 raphael  mem    REG               0,36           9000771 /usr/lib64/...
ncat    532581 raphael    0u   CHR              136,1      0t0        4 /dev/pts/1
ncat    532581 raphael    1u   CHR              136,1      0t0        4 /dev/pts/1
ncat    532581 raphael    2u   CHR              136,1      0t0        4 /dev/pts/1
ncat    532581 raphael    5u  IPv4            9557641      0t0      TCP localhost:webcache->localhost:32886 (ESTABLISHED)
ncat    532581 raphael    6u  unix 0x0000000014cd383f      0t0    31426 type=STREAM (CONNECTED)

Does my process still hold fds to deleted files?

Let’s create a file and immediately unlink it but keep it open:

// gcc ulink.c -o ulink
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <fcntl.h>

int main(int argc, char* argv[]) {
    open("test.log", O_CREAT);
    unlink("test.log");

    while(1){sleep(1000);}
    return 0;
}

The file is not existent, but can be observed via lsof:

$ lsof -p 635
COMMAND PID    USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
ulink   635 raphael  cwd    DIR   8,32     4096 267422 /tmp/example
ulink   635 raphael  rtd    DIR   8,32     4096      2 /
ulink   635 raphael  txt    REG   8,32    16168  50490 /tmp/example
ulink   635 raphael  mem    REG   8,32  2029592 142704 /usr/lib/...
ulink   635 raphael    0u   CHR  136,0      0t0      3 /dev/pts/0
ulink   635 raphael    1u   CHR  136,0      0t0      3 /dev/pts/0
ulink   635 raphael    2u   CHR  136,0      0t0      3 /dev/pts/0
ulink   635 raphael    3r   REG   8,32        0  40397 /tmp/test.log (deleted)

Make your own customized blog using blogctl!

With blogctl v1.0.0, all personal hard coded information are now configurable. They can be specified using the introduced blog.json in the root of the blog directory:

{
    "version": 1,
    "author": "Raphael Pour",
    "title": "blog",
    "domain":"evilcookie.de",
    "description": "Hi, I'm Raphael. I write about my software developer journey...",
    "footer": "<a href='rss.xml'>RSS</a>|<a href='impressum.html'>Impressum</a>"
}

text/template vs. html/template

The index template gets now rendered using the text/template lib rather than html/template while the later escapes all data parameters due to its security model. Escaping e.g. the RSS or impressum link would result in unusable ones.

text/html doesn’t care while template and arguments are trusted by default.

man -K <term> got you covered¹:

 -K, --global-apropos
        Search for text in all manual pages.  This is a brute-
        force search, and is likely to take some time; if you can,
        you should specify a section to reduce the number of pages
        that need to be searched.  Search terms may be simple
        strings (the default), or regular expressions if the
        --regex option is used.

        Note that this searches the sources of the manual pages,
        not the rendered text, and so may include false positives
        due to things like comments in source files.  Searching
        the rendered text would be much slower.

some useful additions:

• --regex: search via regular expression
• -P less: use always a pager, otherwise short manpages will get printed without pager making it difficult to search within the page
• --html=firefox: show pages in the browser

some interesting searches via man -P less -K <term>:

• knuth, torvalds, ritchie: take some history lessons
• <swear>: look for your favorite swear
• 1337: reveal the elite
• coffee: shows that one shouldn’t use espdiff “while sleep-walking, or before your first cup of coffee”
• love: who spreads the most love

Let’s boot some x86 code with qemu.

I rediscovered osdev.org after 13 years of abstinence¹² and started off with the plain-assembly babystep. Then I tried the C based bare-bones kernel to have a bit more convenience.

• Minimal working example
• Babystep - plain assembler
• Bare-Bones - C
• Findings
  • Environment
  • Memory Layout
• Resources
• Vision

Minimal working example

[ORG 0x7c00]
hang:
    jmp hang

times 510-($-$$) db 0
db 0x55
db 0xAA

This nasm code is all you need to make an infinite blinking cursor:

What is going on?

• [ORG 0x7c00]: select the right sector, where the bios expects the bootloader
• hang: jmp hang: plain assembly instructing the CPU to infinitely loop by jumping to the jump instruction
• times 510-($-$$) db 0: Bootloader needs to be exactly 512 bytes large in size. This pads the bootloader up with zeroes.
  • pad 510 bytes instead of 512 to leave two bytes free for the magic number
  • $-$$ is a macro calculating <end of file> - <start of file> resulting in the size of the bootloader so far
  • db 0 is just a static byte set to zero
• 0x55aa is the magic number to tell the bios that this is a valid bootloader

Babystep - plain assembler

After the minimal working example, I approached the plain-assembler babystep bootloader, to get a better insight what’s going on.

Learnings: - basic memory alignment - enter protected mode: code tutorial - write to console without interrupt: code tutorial

Bare-Bones - C

It might be asked much operating at such a low level, but some convenience would be nice. C got you covered! At least a bit. The bare-bones bootloader contains has only as much assembler as necessary to load up C. Basically a stack must be allocated and here we go.

Learnings: - bridge between assembler and C - more sophisticated tty

Findings

Environment

Bootloaders aren’t just usual3 assembler codes in a hosted environment where an OS is providing all kinds of neat stuff like a libc. There is no such thing as ELF or PE, just the bare-metal CPU which we must satisfy. This not only includes valid assembler code but stuff like:

• width: 8, 16 or 32 bit instruction width?
• location: where needs the code to be placed to in order to get (not) executed. The cpu originally doesn’t care whats data and code, it just jumps to a defined address and interprets the corresponding bytes as instruction.

Memory Layout

Resources

Additionally to osdev.org, I also found a bunch of other interesting resources along the way.

General:

• Overview about the cpu registers
• BIOS interrupt table
• x86 instruction set
• Instructions

Tutorials:

• Babystep tutorial with pure assembler
• Example bootloader into c
• Bootloader tutorial
• NASM Tutorial
• German tutorial

Special:

• Linux inside handbook
• first linux
• seabios boot which qemu uses per default for booting

Vision

• practice assembly using exercism
• implement a basic shell as entry point for various sub programs

I did my first solo submission at the godot wild jam 55 about the theme dreams and reached rank 29! It’s called zzZ where the player catches sheep to catch some z’s¹.

Some more facts:

• jam submission
• game
• source code
• two wildcards implemented:
  • A fork in the road: four endings
  • Kabom!: missed sheep explode

Contents

• Idea
• Mechanics
  • Sheep spawner
  • Basket
  • Sunclock
  • Tacho (speedometer)
  • Game loop
  • Instantiated scenes
• Audio
  • Background music
  • Sheep
  • Music box - intro + dreaming-end sound
• Summary

Idea

Johannes had the idea of falling sheeps that must get catched but “gifted” it to me, since he couldn’t do it himself due to lack of time. The core mechanic was quite simple, but still fun to play for a few minutes. I was not hyped about it and that was actually my jam driver:

• simplicity: implement solid game mechanic with low innovation which makes it easy to kill my darlings.
• full stack: instead of delegating tasks/picking the ones I’m good at, everything must be concepted and implemented by myself
• own assets:
  • mspaint graphic: simple, sketchy, bare minimum that just works
  • ableton/dawless audio: game jams are a good opportunity to get passive feedback about my side quest ‘music production’
• focus within godot: Does it add value to the player?, avoid early optimization/polishing

Mechanics

Sheep spawner

Sheep are the essential entity of the game. They fall from the sky into the abyss unless the player catches them with the basket.

The spawner is a scene on its own deciding where and when to generate sheep such as registering the abyss event, that a sheep emits once it fell into the abyss. To know where the abyss is, the spawner awaits the position of it. Sheep get spawned once a second with random size, initial speed and position For choosing a random position, the spawner awaits another two points aka Marker2D:

@export var left_point:Marker2D
@export var right_point:Marker2D

func _on_timer_timeout():
	var sheep:Sheep = sheep_scene.instantiate()
	# ...some more initalization...
	sheep.global_position.x = randf_range(
		left_point.global_position.x, 
		right_point.global_position.x
		)
	sheep.global_position.y = left_point.position.y

Those points are visually placed above the game scene:

About every 10th sheep is special having one of the three complement colors cyan, magenta or yellow such as a smaller random size and a higher random initial speed. They’re worth double on collecting and triple on not collecting them.

Basket

The basket emits an event for every sheep it collides with. Godot 4 makes it easy to automate collision shape generation from a sprite with complex shape:

The basket’s x-axis can be controlled via mouse which is hidden (via game scene’s Input.set_mouse_mode(Input.MOUSE_MODE_HIDDEN)):

extends Area2D

signal item_collected(body:Node2D)

func _input(event) -> void:
	if event is InputEventMouseMotion:
		global_position.x = event.position.x

func _on_body_entered(body) -> void:
	emit_signal("item_collected", body)

Sunclock

The sunclock is the timer of the game, visually similar to the one in minecraft. It consists of two sprite layers. The bottom “clock”-layer gets rotated one revolution in a specified amount of time.

This can be implemented via tween:

extends Node2D

signal sunrise()

@export var duration:int = 30

func _ready():
	var tween = create_tween().set_parallel(true)
	
	tween.tween_property($Clock, "rotation_degrees", 180, duration)
	tween.tween_property($ClockSound, "volume_db", -10, duration)
	await tween.finished
	emit_signal("sunrise")

• set_parallel(true) is needed, so multiple properties can be controlled with the same tween simultaneously
• Additionally a ticking clock gets louder the more the clock approaches the end/next morning. Since the tween increases volume linearly but dB is logarithmic, the sound is barely hearable until the last few seconds, which is a nice side effect.
• await tween.finished is new in Godot 4 and replaces the yield
• the sunrise event gets emitted after the tween is finished/the time is over

Tacho (speedometer)

The tacho shows a half circle on which a needle indicates the state of sleepiness:

• left: fully awake, emit event
• middle: half asleep, game starts there
• right: fully asleep, emit event

There are three animated faces that should transport the meaning of the tacho:

The score is spread evenly across the 180° range. The code is quite messy but I don’t want to hide anything:

extends Node2D

signal fallen_asleep()
signal woken_up()

@export var total:int = 30
@onready var score:int = 0

var event_emitted:bool = false

func _ready():
	$Emoji.play("half_sleeping")

# Called every frame. 'delta' is the elapsed time since the previous frame.
func _process(delta):
	$Needle.global_rotation_degrees = clamp(180/total*score + 90.0, 0, 180)
	
	var animation = "half_sleeping"
	if score >= total/4:
		animation = "asleep"
	elif score <= -total/4:
		animation = "awake"
	
	if $Emoji.animation != animation:
		$Emoji.play(animation)

func inc(val=1)->void:
	score = min(score+val, total)
	if score == total and !event_emitted:
		emit_signal("fallen_asleep")
		event_emitted = true

func dec(val=1)->void:
	score = max(score-val, -total)
	if score == total and !event_emitted:
		emit_signal("woken_up")
		event_emitted = true

event_emitted debounces the event, so it gets only emitted once even if the player collectes more sheeps while the event wasn’t consumed quickly enough by the game scene.

Game loop

The game scene holds everything together while only waiting for three different events:

• sheep got collected: increment tacho, play sheep sound, free sheep node
• sheep felt into the byss: decrement tacho, play explosion animation+sound, free sheep node
• timer is over/ sunrise emitted by sunclock: change to game over scene

All three events check if an end has been reached and if so, set the globals correctly, make the mouse visible again and change to the game over scene. This is where the wildcard A fork in the road is implemented. Additionally the timeout event can decide between two ends depending if the player is more awake (score is negative, needle is on the left) or more sleepy (score is positive, needle is on the right or in the middle).

Retrospectively looking at the code: There is too much responsibility shared between game<->tacho and game<->game over. If I’d invest more time into the game, I’d refactor this inconsistency. The game itself shouldn’t need to provide any content to game over. Also the game shouldn’t need to know how to use the tacho’s score for deciding if and which end has been reached.

Intro

The intro is basically an animation player blending in all the logos synced with the music box. In order to make the synchronization more reliable, the three main parts of the intro sound are split and played after each other. In theory, if the CPU is too slow, the break between the notes are higher, but that is totally fine.

Godot - instantiated scenes

In some cases it might be handy to change to a scene with some context. E.g. tell the game over screen which end has been reached. Sadly I couldn’t find an idiomatic solution to create such instantiated scenes. If you instantiate a scene with parameters, you can’t change to it. It is only possible to add it as a child or use a hacky alternative.

I added a plain auto load script Globals.gd and stored the state there. I dislike this approach while everyone could read/write the state.

Audio

Since my alter ego loves to produce and play around with audio, I couldn’t resist to invest some time into music and sound effect production.

Background music

The first auditive association of dreamy game music for me was the Super Mario 64 Dire Dire Docks theme. Old consoles often use FM synthesis to generate complex instruments with a low memory footprint². I decided to just play around with some c major chords (because this scale is the easiest one to play), with my dawless³ setup consisting of a MIDI Keyboard connected to the Korg FM2 which gets recorded by my digital recorder. The FM8 comes close to the Nintendo64 music engine while it also implements fm synthesis. I pressed record and played around with some chords and presets. I layered two chord progression takes but unfortunately haven’t saved the project file… So the first layer is the first and loud chord progression and the second layer is the more quite progression that can be heard in between to given some kind of “response” if the first layer gave a “call”.

Sheep

I sliced a few different sheep sounds from milkotz’s recording and pre loaded them, so playing them back has a low latency:

var sheeps = [
	preload("res://milkotz_sheep1.mp3"),
	preload("res://milkotz_sheep2.mp3"),
	preload("res://milkotz_sheep3.mp3"),
	preload("res://milkotz_sheep4.mp3"),
]

Since there are small and big sheep, why not pitching the sample in relation to the size? Quickly a formula pitch = 1 - (size-0.9) was crafted with educated guessing such as trial-and-error. The pitch’s default is set to 1. Basically it maps the range of the sheep scale factor within the range [0.1,1]⁴ onto the pitch range [0.9, 1.8] in reverse, because smaller sheep have higher pitches and vice versa. This means the smallest sheep with a size of 0.1 has a pitch of 1-(0.1-0.9) = 1.8 where the biggest sheep has a pitch of 1-(1-0.9) = 0.9⁵.

To have limitless parallel sheep sounds, an audio player get initialized for each collected sheep:

func _on_basket_item_collected(body:Sheep):
	var sheep_player = AudioStreamPlayer.new()
	sheep_player.stream = sheeps[randi_range(0,len(sheeps)-1)]
	sheep_player.set_bus("SheepFX")
	sheep_player.pitch_scale -= (body.size-0.9)
	add_child(sheep_player)
	sheep_player.play()

Music box - intro + dreaming-end sound

• Ableton’s Collision with preset “Glockenspiel basic”: a typical music box sound but a bit to high
• Chorus: Add more vibrato/pitch modulation to sound more natural.
• Cabinet: Dampen the sound to make it more like it was recorded in a room.
• EQ:
  • Cut off the lows is always good if they’re not needed to avoid dirty mixes/rumbling sound.
  • Make the highs a bit more quiet additionally to the cabinet.
• Compressor: Nothing special, just make the sound more louder. This may be obsolete since I export all my samples normalized.

Notes

Although the whole-tone-scale is usually used for dream sequences, I wanted something more naive and chose the c major scale again. I tried to mimic the “ascending stairs” music box melodies often have and added a little offset to some notes so it sounds more analog.

Intro notes:

Dreaming-end notes:

Summary

I was looking for a useful service-oriented Rust project where I can test my project standards¹ and dive deeper into the language. I decided to replace the “caddy file server” that serves this blog. The result is called blogstage: Simple web server providing my static blog to the world.

It can be used via blogstage <ip:port> <path> or just blogstage 0.0.0.0:80 ./blog/.

Contents

• Requirements
• How to serve
  • Argument parser
  • File cache
    
  • TCP
  • HTTP
  • Parallel request handling
  • Exit
• Testing
  • Unit test
  • Integration test
• Benchmark
• Wrap it up
• Vision

Requirements

The requirements were fairly simple, because the blog is static consisting of a flat directory of rendered HTML files along with some images2 and my reverse-proxy is handling the TLS stuff:

• bind to given address+port
• serve files from given directory (and prevent serving other directories e.g. via relative paths)
• serve text and binary for HTML and images
• not too easy to dive deeper into rust

How to serve

Since the requirements weren’t hard to fulfill, there are tons of crates that can do the job. E.g. warp can do it in one line with something like warp::serve(warp::fs::dir("./blog")).run(([127, 0, 0, 1], 8080)). I decided against a ready-to-use crate, to make it a bit more difficult and educational.

web_server didn’t worked with dynamic strings, since routes need to be added with &'static str where I couldn’t figure out if and how this can be fixed³.

The solution was surprisingly easy to find. There is a tutorial about Building a Multithreaded Web Server in the official rust docs. What a coincidence!

Argument parser

/* parse arguments */
let uri = match std::env::args().nth(1) {
    Some(uri) => uri,
    None => {
        println!("usage: blogstage <URI> <PATH>");
        return;
    }
};

let path = match std::env::args().nth(2) {
    Some(path) => path,
    None => {
        println!("usage: blogstage <URI> <PATH>");
        return;
    }
};

File cache

The blog is pretty small, and its container4 gets rebuilt for any change. Therefore I decided to load all the files non-recursively into a hash map with with basename as key and content as value.

/* load files */
let raw_entries = match fs::read_dir(path.clone()) {
    Ok(entries) => entries,
    Err(e) => {
        println!("error reading files from {}: {}", path, e);
        return;
    }
};

let mut files = HashMap::new();

for entry in raw_entries {
    if !entry.as_ref().unwrap().path().is_file() {
        continue;
    }

    files.insert(
        entry
            .as_ref()
            .unwrap()
            .file_name()
            .into_string()
            .unwrap()
            .clone(),
        fs::read(entry.as_ref().unwrap().path().clone()).unwrap(),
    );
}

TCP

/* start server */
let listener = match TcpListener::bind(uri.clone()) {/* error handling */};

for stream in listener.incoming() {
    match stream {
        Ok(s) => on_request(stream, fileHashMap),
        Err(e) => /* error handling */
    }
}

HTTP

pub fn on_request(mut stream: TCPStream, files: HashMap<String, Vec<u8>>)

let reader = BufReader::new(&mut stream);
let request: Vec<_> = reader
    .lines()
    .map(|result| result.unwrap())
    .take_while(|line| !line.is_empty())
    .collect();

Parse the requested file quick ‘n’ dirty (see vision). Additionally index.html if the requested filename is empty for convenience:

let mut target: String = request[0].split(' ').collect::<Vec<&str>>()[1][1..].to_string();
if target.is_empty() {
  target = "index.html".into()
}

In order to serve more complex files like images (and to be future ready if I add openscad or mp3 stuff), guess the mime type via the crate…well…mime_guess:

let mime = mime_guess::from_path(target.clone()).first().unwrap();

If the requested file can be found in the hashmap, write its content along with a valid HTTP header to the stream. Otherwise just write a HTTP header with status code 404. On any error, the response handler panics caused by the massive use of unwrap, and the thread just vanishes into the void (see vision):

match files.get(&target) {
  Some(body) => {
    let length = body.len();

    println!("200 {}", target);
    stream.write_all(
      format!("HTTP/1.1 200 OK\r\nContent-Length: {length}\r\nContent-Type: {mime}\r\n\r\n")
      .as_bytes()
    ).unwrap();
    stream.write_all(body).unwrap();
  }
  None => {
    println!("404 {}", target);
    stream
      .write_all("HTTP/1.1 404 NOT FOUND\r\n\r\n".as_bytes())
      .unwrap();
  }
};

Parallel request handling

Since I want to implement parallel request handling as simple as possible and don’t need to limit the amount of requests5, thread::spawn will do the trick:

for stream in listener.incoming() {
    match stream {
        Ok(s) => {
            let f = files.clone();
            thread::spawn(move || on_request(s, f));
        }
        Err(e) => {
            println!("error accepting connection: {}", e);
            continue;
        }
    }
}

The file map must be cloned, so its ownership can be moved to the thread.

Exit

It doesn’t deserve the predicate graceful, but it provides some simple mechanism to simply abort the whole request handling via the crate ctrlc:

ctrlc::set_handler(move || {
   std::process::exit(0);
})
.unwrap();

Testing

Unit test

To test the on_request method, the stream must be mocked. Looking closely what the function actually does with the stream reveals that we actually only need to read from and write to it. As described in the tutorial, TCPStream can be replaced by allowing any type that implements the Read and Write trait using impl Read+Write:

pub fn on_request(mut stream: impl Read + Write, files: HashMap<String, Vec<u8>>)

The mock implements all methods needed for properly read and write to it. I modified the tutorial’s mock to be single-threaded as we don’t use async:

use assert_cmd::prelude::*;
use predicates::prelude::*;
use std::cmp::min;
use std::collections::HashMap;
use std::io;
use std::io::{Read, Write};
use std::process::Command;

struct MockTcpStream {
    read_data: Vec<u8>,
    write_data: Vec<u8>,
}

// https://doc.rust-lang.org/std/io/trait.Read.html
impl Read for MockTcpStream {
    fn read(self: &mut Self, buf: &mut [u8]) -> io::Result<usize> {
        let size: usize = min(self.read_data.len(), buf.len());
        buf[..size].copy_from_slice(&self.read_data[..size]);
        Ok(size)
    }
}

// https://doc.rust-lang.org/std/io/trait.Write.html
impl Write for MockTcpStream {
    fn write(self: &mut Self, buf: &[u8]) -> io::Result<usize> {
        self.write_data.extend(buf.iter().cloned());
        Ok(buf.len())
    }

    fn flush(self: &mut Self) -> io::Result<()> {
        Ok(())
    }
}

Then tests can simply be written as such:

#[test]
fn serve_not_found() {
    let input = b"GET /test.html HTTP/1.1\r\n\r\n";
    let mut contents = vec![0u8; 1024];

    contents[..input.len()].clone_from_slice(input);
    let mut stream = MockTcpStream {
        read_data: contents,
        write_data: Vec::new(),
    };

    let files = HashMap::new();
    blogstage::on_request(&mut stream, files);

    let expected_response = format!("HTTP/1.1 404 NOT FOUND\r\n\r\n");
    assert!(stream.write_data.starts_with(expected_response.as_bytes()));
}