articles

Until recently, I haven’t had many opportunities to use Go’s generics. I ran into a case where generics make sense. Best of all, this isn’t a contrived example. I’m working on a project and using openAPI to generate API contracts. One of the generated structs contains optional fields implemented as pointers. The only required field is Name. const ( Gzip PostPailsCompression = "gzip" None PostPailsCompression = "none" ) type PostPails struct { Compression *PostPailsCompression `json:"compression,omitempty"` // MaxArchiveSize Max size (bytes) before rotating to a new archive. MaxArchiveSize *int `json:"max_archive_size,omitempty"` // Name Name of the new pail Name string `json:"name"` } I need to populate the struct with values when writing unit tests. But dealing with pointers in Go test code usually results in using temporary variables. It’s not bad, but there’s some visual noise. ...

OpenAPI is a specification for documenting HTTP APIs for both humans and machines to consume. As OpenAPI is a specification, it is language agnostic. OpenAPI relies on generators for translating the specification. There’s more than just documentation that’s generated. Generators also create language-specific interfaces, tooling, and contracts. In some ways the OpenAPI pattern reminds me of either protobuf with gRPC or ORM schema-first design. As a result, a declarative API is created by the tooling. ...

With Docker, it’s not always obvious what storage options exist beyond the built-in local volume driver or a traditional bind mount. Exploring Docker volume drivers often turns up archived GitHub repositories or commercially backed plugins tied to specific cloud storage products. The volume ecosystem is especially limited for on-premise storage, and many plugins require more privileges than you’d expect. In this post, I’ll cover how Docker handles volume storage under the hood. I’ll also walkthrough how to create a volume plugin that interacts with remote storage without needing CAP_SYS_ADMIN privileges. ...

I’m looking at how object storage systems manage data on disk. Especially the idea of using append only archives with an index for fast retrieveal. While reading Facebook’s Haystack design, I noticed similarities to the tar file format and the potential to implement something similar at the local scale. Haystack Overview There are several components mentioned in the original Haystack paper, but at the core is the Haystack Store, where end user image files are physically kept. Instead of writing files directly to the filesystem, images are appended to a large file called a volume, which acts as an append-only archive. Each volume is typically capped at around 100 GB and is aligned to 8-byte offsets. Image files within this volume are referred to as needles. ...

UTM and Multipass are great apps for virtualization on macOS. But I wanted a lighter-weight approach by invoking QEMU directly. Which meant I needed to understand how QEMU’s networking options interact with the vmnet virtualization API on macOS. This becomes especially important when dealing with VM-to-VM connections, network isolation, and bridging on macOS. In this post, I’ll walk through creating a simple QEMU-based networking lab. Set up RouterOS and Alpine Linux VMs using QEMU on macOS Connect VMs with Apple’s Hypervisor vmnet networking APIs Use unified logging to troubleshoot QEMU network issues on macOS Lab Setup Overview The network diagram shows the network topology used in this lab. Both VMs run on on the same macOS host and connected to virtual network interfaces using QEMU’s support for Apple’s vmnet virtualization API. ...

QEMU uses files to emulate storage devices, and the features available depend on how those files are created. While QEMU can emulate disks from Parallels and VirtualBox, I’m going to focus on the formats most commonly used in automation and scripting, raw and qcow2. The default format is raw and raw offers the fewest features. It’s just plain storage. The other format qcow2 supports compression, snapshots, and copy-on-write in addition to storage. ...

I previously wrote an introduction to cloud-init. I’d like to now follow up with a discussion on troubleshooting. cloud-init failures on remote hosts can be challenging. Depending on the failure point, cloud-init may or may not provide clear error indicators. These are methods I use during provisioning issues related to cloud-init. Understanding cloud-init execution stages Before continuing, let’s cover some background. cloud-init follows five stages during boot which run sequentially. If a stage completes, output will contain a status that can be used to verify that stage was successful. ...

In a previous post, I discussed using cloud-init and Multipass as a method of provisioning virtual machines on a local computer with a cloud-like API. Today I am going to dive deeper with Ubuntu and how their autoinstall API can simplify on-premise host provisioning. autoinstall is a tool that allows for unattended installations of Ubuntu, ensuring consistency, reporducibility, and providing automation across a fleet of hosts. In this post I’ll walk through an example of using autoinstall to configure networking, local storage, and demonstrate shell command execution during provisioning. ...

Cloud compute companies like GCP, AWS, or Azure offer a management API for allocating resources. In the on-premise space, services such as Docker or Incus provide APIs for managing containers or virtual machines (VMs). But what about installing the operating system (OS) on bare-metal hosts? What API exists for this task? This is where cloud-init enters the picture, providing the ability to provision VMs or bare-metal hardware. cloud-init is a useful tool that doesn’t rely on network services like PXE as a dependency. Its simplicity saves time by removing the need to navigate OS installation menus, while ensuring user accounts and installed software packages are consistent across hosts. So why should one bother using cloud-init if they are managing a single host at home? In the event the OS needs to be reinstalled due to failure, cloud-init allows one to quickly restore the system to a known state. ...