Storage

Who Should Not Read This Post This post does not discuss: data support for multiple writers having zero downtime cloud-scale best practices Instead I discuss embracing single points of failure (SPOF) in exchange for simplicity within environments that can tolerate downtime. Permission to Not Over-Engineer In environments that tolerate downtime, having a SPOF is great because they are simple to deploy and it’s obvious what failed. Lets examine the typical home router. In addition to routing packets between the LAN and the Internet, the home router also provides DHCP and DNS services. Then someone learns of PiHole and runs it on a separate device like a Raspberry PI. Now the router queries PiHole for adblock entries during DNS resolution. Ad-free browsing works works great; until HTTP requests stop working. With this scenario, it’s not obvious if it’s an issue on the router or an issue with PiHole. ...

With Docker, it’s not always obvious what storage options exist beyond the built-in local volume driver or a traditional bind mount. Exploring Docker volume drivers often turns up archived GitHub repositories or commercially backed plugins tied to specific cloud storage products. The volume ecosystem is especially limited for on-premise storage, and many plugins require more privileges than you’d expect. In this post, I’ll cover how Docker handles volume storage under the hood. I’ll also walkthrough how to create a volume plugin that interacts with remote storage without needing CAP_SYS_ADMIN privileges. ...

I’m looking at how object storage systems manage data on disk. Especially the idea of using append only archives with an index for fast retrieveal. While reading Facebook’s Haystack design, I noticed similarities to the tar file format and the potential to implement something similar at the local scale. Haystack Overview There are several components mentioned in the original Haystack paper, but at the core is the Haystack Store, where end user image files are physically kept. Instead of writing files directly to the filesystem, images are appended to a large file called a volume, which acts as an append-only archive. Each volume is typically capped at around 100 GB and is aligned to 8-byte offsets. Image files within this volume are referred to as needles. ...

QEMU uses files to emulate storage devices, and the features available depend on how those files are created. While QEMU can emulate disks from Parallels and VirtualBox, I’m going to focus on the formats most commonly used in automation and scripting, raw and qcow2. The default format is raw and raw offers the fewest features. It’s just plain storage. The other format qcow2 supports compression, snapshots, and copy-on-write in addition to storage. ...