VulpineCitrus - Blog

Diving into Radio Listening: POCSAG, Alphapage, and how i taught myself some GNURadio

2026-02-22T00:00:00+00:00

Pagers used to be all the rage in the late 80s, 90s and early 2000s. You could carry around a little battery-powered trinket that could ring and give you a little text message. How neat! Then, mobile telephones became easier and easier to carry around, then smartphones became a thing, and pagers died.

Pagers worked and still work via radio signals. A lot of different radio standards for pagers have been developed over the decades, but, here, i will focus on the POCSAG family of standards and how they were (and are) used here, in France.

I will go over three different things in this post: a tiny bit of history on the Alphapage paging network in France, some technical information about POCSAG, and how i tweaked a GNURadio script to do multi-channel decoding to capture multiple channels at the same time.

Why am i writing an english-speaking article on a blog primarily oriented towards an english-speaking audience with so much french-specific data? Well, for one, i only live in one country, and i haven’t had as much experienced listening in on the data of other countries. If you want the full french experience, you can also read this in french</a> when i get around to translating it (soon (hopefully)).

#</a> A Brief History of Alphapage</h1>
A lot of companies</a> used to sell pager services for French consumers (Tatoo</a>, Tam-Tam and Kobby; the latter two of which used FLEX and relied on separate networks) and companies (Alphapage). These devices are not necessarily junk today! Tatoo, in particular, was the consumer-brand</a> of the network i will focus on today. Some people have tried to and succeeded in</a> broadcasting to a Tatoo pager they bought online or found stashed in a drawer. The cost, however, is prohibitive (more than 3.5€ per message). Unless you’re modding some hardware for a ham system like DAPNet</a>, it’s unrealistic to try and build on top of a system that sends data to pagers.
The reason i focus specifically on Alphapage is because, contrary to the other networks at the time, Alphapage is still operational, albeit under a different name with a different company owning it. There are still messages being broadcast to POCSAG-capable pagers today.
Alphapage’s infrastructure used to belong to France Télécom, the name of our former national telecom operator. Their commercial network, Alphapage</a>, was broken off as property of a company that was bought and renamed e*Message France</a> when France Télécom was dismantled and sold to the private sector. That company officially received the right to operate FT’s pager frequencies in 2001, with ARCEP decision #2001-308 of March 23rd 2001</a>. That decision conveniently lists the frequencies and some technical information. Around 466~MHz, five 25 kHz channels are listed:

Channel 1 : 466.02500 MHz</li>
Channel 2 : 466.05000 MHz</li>
Channel 3 : 466.07500 MHz</li>
Channel 7 : 466.17500 MHz</li>
Channel 8 : 466.20625 MHz</li> </ul>
e*Message is still authorized to use these frequencies in all of France. Another frequency, 87.390 MHz is listed, but not currently used. In fact, as of September 2025, only channels 2, 7 and 8 are still in use by e*Message. This is coherent with various ARCEP decisions</a> i found that list the channels still in use as 2, 7 and 8. Those channels are usable, as of my writing this, until December 31st 2030.
By the mid-90s, the coverage of Alphapage was rather impressive</a>. By 2007, e*Message itself advertised “about 500 antennas spread all over the country”</a>.
In several places online, i read that France was cut in distinctive zones depending on your geographic location, and that Alphapage-compatible pagers were registered to a specific zone. However, i was never able to find a proper map of those zones, nor confirm that that was ever the case.
#</a> The POCSAG Protocol</h1>
The standard used by these Alphapage-compatible pagers is the Radio-Paging Code No. 1 of the Post Office Code Standardization Advisory Group, or POCSAG. POCSAG was formed in 1976 by a group of international engineers looking to develop a wide-area paging code.
POCSAG</a> (the protocol, not the standards group) supports different bitrates: 512, 1200 and 2400 bps. POCSAG transmissions begin with a preamble of 576 bits that alternate between 0 and 1. Most pagers put themselves to sleep to save power, and wake up to listen for a preamble at a regular interval (small enough that they will catch an interval when one occurs). The preamble signals for pagers to stay awake and power up decoding hardware. It also helps identify the data rate that will be used.
The rest of the communication comes in batches. POCSAG is capable of delivering messages simultaneously to multiple units by dividing messages into piece that are transmitted progressively batch per batch. Each batch is time-divided into 8 frames (preceded by a magic sequence of bytes called the Frame Synchronization Code). Then, each frame is divided into two codeword slots, making it 16 codewords per batches. Each codeword can contain either an address (first bit is 0), or a piece of message (first bit set to 1).
Where a message is in each batch will partially determine who is receiving it. In POCSAG, a pager’s address is 21 bits long: 18 bits are transmitted in the address codeword, and the last 3 are derived from the position of the codeword in the batch. That neat little trick saves space and time in transmission! When a batch transmits an address at a given codeword position, the same codeword position in incoming batches will contain the pieces of the incoming message. The pager will keep receiving and decoding its message until an address or idle codeword is transmitted in the codeword slot that it was listening for. Idle codewords are used when no message is to be transmitted in a given codeword slot.
+-----------+ | B | | B | +---+-------------------------------+ | A | | A | | | 8 | | F | R | A | M | E | S | | T | . | T | | F + + + + + + + + + +-- SYNC --+ C | . | C | | C |C|C|C|C|C|C|C|C|C|C|C|C|C|C|C|C| | H | . | H =====>| S |0|0|0|0|0|0|0|0|0|1|1|1|1|1|1|1| | | | | | |1|2|3|4|5|6|7|8|9|0|1|2|3|4|5|6| | 1 | | n | +---+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-----------+</code></pre> POCSAG messages can be decoded in multiple modes:

BCD (Binary-Coded Decimal) aka “Numeric”: Only digits, parens, dashes, ‘*’ and some special control characters are sent</li>
Alphanumeric with 7-bit ASCII, transmitted in little-endian.</li> </ul>
Of note is that only 20 bits can be transmitted at a time in a codeword. Thus, when your characters are 7-bit, there will almost always be one last codeword filled with a bit, and then null/termination characters. Per the standard, the NULL character is the only one that is allowed to be truncated</a>.
#</a> Actually Listening to POCSAG</h1>
The technical stack used to decode and see pager messages is not that complicated, and has been tried and tested</a> before. multimon-ng</code> is a CLI tool that provides decoding for a lot of paging protocols, including POCSAG. You just need to feed it raw audio at a bandwidth of 22.050K, and it’ll happily decode for you.
So i simply ran the same kind of command that people recommend for Alphapage: rtl_fm</code> piped into multimon-ng</code>. I looked at the three frequencies still in use on SDR++ and figured out that, roughly every minute, a message is sent on all three frequencies. Channel 7 received more frequent messages. So i picked that frequency:
rtl_fm -p -5 -F 0 -E dc -M fm -T -g 32 -f 466.175M -s 22050 | multimon-ng -t raw -c -a POCSAG1200 --timestamp -</code></pre>And eventually, it decodes things! Most of it is junk i can’t understand, all of it is stuff i cannot share anyways1</a>, but something is coming through. Because a pager only listens to one of the specific carriers, and i had no way to simultaneously monitor messages sent to the three channels, i figured “well, my RTL-SDR dongle can cover the bandwidth needed to include all three channels, surely it isn’t too hard to decode all three channels at the same time”. That sounded like a fun technical challenge. rtl_fm</code> is a very nifty piece of CLI tooling if you want to just listen to one frequency, or multiple frequencies with some scanning logic. Problem is, no matter how i tuned the squelch, rtl_fm</code> would get stuck on one channel and stop scanning. Sometimes, i would see two carriers active, but rtl_fm</code> can only tune to one frequency at a time, and multimon-ng</code> cannot know which frequency is being listened and print it (another piece of information i want). Worse, when a channel would become active, rtl_fm</code> would switch to it after the synchronization sequence started: multimon-ng</code> couldn’t decode that audio. So i poked around the internet, figuring that “someone more competent might already have done that”, and i found multimon-ng</code> issue #165</a>. In the examples/</code> folder of the multimon-ng</code> repository, there is an example Python script called multipager.py</code> generated with GNURadio Companion that splits received data into a bunch of channels and decodes them all individually by piping the output audio to one multimon-ng</code> process per channel. The output is then parsed (yes, that’s awful) and the script adds information such as the frequency where a message was decoded. #</a> So What is this GNU of the Radio Anyways</h1> GNURadio</a> is a flow-based framework to develop digital signal processing applications. It works on a logic of linking sources to sinks through a bunch of filters and transformations of the input. In the script above, we use the Open Source Mobile Communication (OSMoCom)</a> library to have access to SDR peripherals. GNURadio comes with a piece of software called GNURadio Companion (GRC) that provides the graphical view of these processing flowcharts. As a bonus, once a flowchart is validated in GRC, it can be exported as a Python script that uses the Python gnuradio</code> library. And that’s where multimon-ng</code>’s multipager is an issue: it was generated for Python2, on an old version of GRC. Some bindings are not located in the same namespaces anymore, some libraries have stopped existing, and the language as a whole is very different. At first, i simply updated the script. It’s not the first time i update a Python 2 script for Python 3. Even replacing some libraries, notably for async I/O, was done fairly quickly. And i could run the script rather fast. But then i hit an issue: channelizing. With how the script was originally made, the input bandwidth is divided into n</code> equally sized channels. The Alphapage channels are not well aligned for that: they do not all fit in n</code> equally spaced channels because of channel 8 (466.20625 MHz). So i had to learn some digital signal processing. What i want to do is have three copies of the baseband, cut down to the bandwidth of POCSAG, and shifted such that the center frequency is correct for a given channel. In DSP, frequency translation, also abbreviated Frequency XLating, is a process by which an input spectrum is shifted frequency-wise. In GNURadio, you can even use a Finite Impulse Response implementation of frequency translation</a> to combine translation of the spectrum and an application of filters to isolate a piece of that band with the correct bandwidth. i don’t really understand any of the nitty-gritty underneath all of this, so i will let you read the Wikipedia page</a>. So i went on my merry way hacking away at the script: i need to center the frequency in the middle of all channels (so 466.128125 MHz) in such a way that if there is a DC offset, if will not bother us. Then, i need to connect three XLat IR filters to the output of the OSMoCom SDR source to shift to all three active pocsag channels. Then, i just hook myself to the existing bits of the script that demodulate the spectrum into audio and pipes it into multimon-ng</code> and parses the output. And voilà. The actual script is available here</a>. You can pass it the same kinds of arguments that multipager.py</code> from the original repository</a> takes, except -f</code> and -c</code> (those are never queried). #</a> Where To Go From Here?</h1> Pagers are a cool little tool to play with. Some people send themselves messages</a>, some others are trying to build entire networks for them</a>. In the future, i might mess around with making a low-power local emitter for POCSAG, to automate notifications or alarms for meds and calendar events. In general, i find learning about technology that provides you with more off-grids communication fascinating! i’m pretty sure the precise piece of legislation is Penal Code R226-15 ¶2</a> for France. Be smart! don’t fedpost about what you hear on the radio if you weren’t the intended recipient :) ↩</a> </li> </ol> </section>

A Completely Innocuous Blog Post about vPMU in QEMU 2026-02-10T00:00:00+00:00 Hey! If like me, you spend a lot of time debugging the Linux kernel, especially with the goal of modifying the performance of user programs, you may often find yourself in the silly position where you need to run perf</code> inside of a virtual machine (typically run using qemu</code>) in order to obtain hardware performance counters. In x86(_64) CPUs, the component responsible for providing you these pieces of information is called the Performance Monitoring Unit, or PMU. Your kernel, when it boots, figures out what brand and model your CPU is, and which PMU driver will be responsible for handling initialization and gathering of information. In a virtualized environment, however, the hypervisor is responsible for providing a PMU to the guest! In QEMU, this is often done by enabling the KVM accelerator, and copying the host topology, as such: -enable-kvm -cpu host</code></pre> And with that, you can have a guest CPU with features such as: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 39 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 1 On-line CPU(s) list: 0 Vendor ID: GenuineIntel Model name: Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz CPU family: 6 Model: 78 Thread(s) per core: 1 Core(s) per socket: 1 Socket(s): 1 Stepping: 3 BogoMIPS: 5615.99 Flags: fpu vme de [...] arch_perfmon [...] Virtualization: VT-x Hypervisor vendor: KVM Virtualization type: full [...]</code></pre> Which clearly shows arch_perfmon</code> (the name of the x86 feature of hardware performance monitoring in Intel CPU cores, especially in the p6 family starting after Yonah cores) enabled! On AMD, the feature has a different name (perfctr_core</code>), and will be handled by the kvm-amd</code> Linux kernel module. And, yeah! Hope that helped. … still here? … good. Look, i would not know what a vPMU is, what p6-family Intel cores have hardware-backed PMUs, nor how KVM plays a role in exposing those to the guest, if i had managed to make it all work as easily as all online documentation seems to imply. This is a story about debugging, and bad development practices, and pain. Mostly pain. This is a blog post about how a single if</code> statement guarding a code block with no warning log, made me lose a whole day of work investigating a potential bug. This is a Blog Post About What Happens When You Fall Through the Cracks of People's Assumptions</h1> ##</a> Part 1: This is the Part Where i Still Sound Somewhat Sane</h2> The reason you don’t read my yapping about Linux as much these days is that most of the Linux-touching i do is actually for work. On a rare occasion, i happen upon a tangential problem posed, for example, by a benchmark setup problem. Even more rarely, that problem becomes a time sink hole that engulfs an entire day of my own time, and brings to light an interesting problem that will almost definitely trip other people in the future. And so, i feel compelled to write about it. The story of today brings us to QEMU, KVM, and early CPU feature recognition. For complicated reasons, I needed to sample some hardware performance counters from within a QEMU instance running my own build of the Linux kernel. My first thought went along the lines of “oh well, i might need a perf</code> binary compiled for my custom kernel, let me do that”, and i happily tweaked build scripts to install the binary. Running it, i got: ~ # perf stat -d echo event syntax error: 'cpu_core/TOPDOWN.SLOTS,metric-id=cpu_core!3TOPDOWN.SLOTS!3/,cpu_core/topdown-retiring,metric-id=cpu_core!3topdown!1retiring!3/,cpu_atom/TOPDOWN_RETIRING.ALL,metric-id=cpu_atom!3TOPDOWN_RETIRING.ALL!3/,cpu_core/topdown-bad-spec,metric-id=cpu_core!3t..' \___ Bad event or PMU Unable to find PMU or event on a PMU of 'cpu_core' Performance counter stats for 'echo': 0.45 msec task-clock # 0.215 CPUs utilized 1 context-switches # 2.231 K/sec 0 cpu-migrations # 0.000 /sec 60 page-faults # 133.875 K/sec <not supported> cycles <not supported> instructions <not supported> branches <not supported> branch-misses <not supported> L1-dcache-loads <not supported> L1-dcache-load-misses <not supported> LLC-loads <not supported> LLC-load-misses 0.002081276 seconds time elapsed 0.000392000 seconds user 0.000392000 seconds sys</code></pre> That’s odd. Okay, maybe hardware counters from within a VM is a bit weirder. i then spend about five minutes searching online a combination of the words qemu</code>, perf</code>, and hardware counters</code>. Eventually, the answers</a> i find</a> in various</a> places tell me that the feature i am looking for is called “Virtualized Performance Monitoring Unit”, or “Virtualized PMU”, or “vPMU”. They also tell me the same information i relayed at the top of the post: Enabling vPMU in QEMU is done by enabling KVM and copying the host CPU information: -enable-kvm -cpu host</code> . </blockquote> So i tried. And it didn’t work. It. Didn’t. Work. Nothing changed. What was going on? Why was i falling into a case that was documented nowhere? Oh no, it’s happening again isn’t it, oh NO, NOT AGAI- ##</a> Part 2: My Sanity Slowly Drains as I Skim DMesg and Kernel Code</h2> In order to start debugging, i had to figure out what was happening in the kernel. For those who don’t know, perf</code> interacts with an entire subsystem in the Linux kernel (the perf</code> subsystem) to retrieve hardware events. Thus, if there was a bit of software that should know about the problems with the PMU, it should be the kernel. Running dmesg | grep -i perf</code> i got the following: [ 0.133409] Performance Events: unsupported p6 CPU model 183 no PMU driver, software events only.</code></pre> Oh, wait. So my CPU is not supported? Why? My kernel is a bit old (6.12), and my CPU is, as you can guess, a Raptorlake (13th gen Intel, heterogeneous architecture with Raptorcove performance cores, and Enhanced Gracemont efficiency cores). Maybe there was a driver missing which was not in older Linux versions? That’s what i thought, until i also ran a Linux 6.18 build and obtained the same result. On Linux v6.12, i hit this line</a> (arch/x86/events/intel/p6.c#L272</code>) in the following code: __init int p6_pmu_init(void) { x86_pmu = p6_pmu; switch (boot_cpu_data.x86_model) { case 1: /* Pentium Pro */ x86_add_quirk(p6_pmu_rdpmc_quirk); break; case 3: /* Pentium II - Klamath */ case 5: /* Pentium II - Deschutes */ case 6: /* Pentium II - Mendocino */ break; case 7: /* Pentium III - Katmai */ case 8: /* Pentium III - Coppermine */ case 10: /* Pentium III Xeon */ case 11: /* Pentium III - Tualatin */ break; case 9: /* Pentium M - Banias */ case 13: /* Pentium M - Dothan */ break; default: pr_cont("unsupported p6 CPU model %d ", boot_cpu_data.x86_model); return -ENODEV; } memcpy(hw_cache_event_ids, p6_hw_cache_event_ids, sizeof(hw_cache_event_ids)); return 0; }</code></pre> That code initially had me guess that something about models not listed here made them incompatible with vPMU. Maybe only older versions of p6 CPUs could have virtualized PMUs? No. Looking at v6.18, that function changed dramatically: __init int p6_pmu_init(void) { x86_pmu = p6_pmu; if (boot_cpu_data.x86_vfm == INTEL_PENTIUM_PRO) x86_add_quirk(p6_pmu_rdpmc_quirk); memcpy(hw_cache_event_ids, p6_hw_cache_event_ids, sizeof(hw_cache_event_ids)); return 0; }</code></pre> And the new line i hit is located in the v6.18 version of intel_pmu_init</code></a>: __init int intel_pmu_init(void) { // ... /* Architectural Perfmon was introduced starting with Core "Yonah" */ if (!cpu_has(&boot_cpu_data, X86_FEATURE_ARCH_PERFMON)) { switch (boot_cpu_data.x86) { case 6: if (boot_cpu_data.x86_vfm < INTEL_CORE_YONAH) return p6_pmu_init(); break; case 11: return knc_pmu_init(); case 15: return p4_pmu_init(); } pr_cont("unsupported CPU family %d model %d ", boot_cpu_data.x86, boot_cpu_data.x86_model); return -ENODEV; } // ...</code></pre> So, “Architecture Perfmon was introduced starting with Core Yonah”, huh? This bit of code seems to be responsible for initializing support for the PMU on an Intel CPU at boot. i skipped the variable declarations, because the first bit of code in that function is a check for CPU capabilities on the CPU the kernel booted on. Specifically, we’re looking for an x86 feature called arch_perfmon</code>. If that feature is absent, we then call the p6_pmu_init</code> function that previously spewed the error log. The new organization of these functions clears up the earlier confusion: p6_pmu_init</code> is only run for pre-Yonah p6 family cores in order to try and initialize the feature if it wasn’t already available. Failing that (and it will! Even if you comment out the core version check), we run into our pr_cont</code> right there above, and return -ENODEV</code>. Oops. If you’re on an Intel CPU and run lscpu</code>, you may find arch_perfmon</code> in the list of Flags</code>. This signals that you have architectural support for your PMU. If you’re on an AMD CPU, you will have something called perfctr_core</code>. So, wait, hold on, where</a> is that X86_FEATURE_ARCH_PERFMON</code> capability even enabled during boot? So, even at boot, the CPU is indicating it can’t support (v)PMU? Surely not. Surely that’s a mistake. Right? There’s three places that use the X86_FEATURE_ARCH_PERFMON</code> and that, at first glance, look relevant to this current situation: arch/x86/events/intel/core.c</code>, where it’s checked to potentially run a *_pmu_init</code> function.</li> arch/x86/kernel/cpu/intel.c</code></li> arch/x86/kvm/cpuid.c</code></li> </ul> Now, if you do not know what cpuid</code> is, don’t worry. We’ll come back to it. All you need to know, and what i knew at the time, is that it’s a CPU instruction used to probe features in, and identify CPUs. Let’s investigate the other file first, okay? So, what’s going on in intel.c</code>? Well, static void init_intel(struct cpuinfo_x86 *c) { early_init_intel(c); intel_workarounds(c); init_intel_cacheinfo(c); if (c->cpuid_level > 9) { unsigned eax = cpuid_eax(10); /* Check for version and the number of counters */ if ((eax & 0xff) && (((eax>>8) & 0xff) > 1)) set_cpu_cap(c, X86_FEATURE_ARCH_PERFMON); } // ...</code></pre> Okay. Deep breaths. We’re gonna have to do it. We’re gonna have to- ##</a> Part 3: We Have To Talk About CPUID</h2> *inhales* (i’ve always wanted to do this) The Intel(R) 64 and IA-32 Architecture Software Developer Manual</a>, Volume 1, Chapter 21, explains the processor identification and feature determination instruction, called cpuid</code>. CPUID is complex to explain, but, essentially, you interact with it by setting CPU register eax</code> to a value, and then you observe the output values in eax</code>, ebx</code>, ecx</code> and potentially edx</code>. For the purpose of probing PMU features, we set eax=10</code>, as shown above with cpuid_eax(10)</code>. That function is not named cpuid_eax</code> because it sets eax</code> to 10 by the way, but because it returns only the value of the eax</code> register after running cpuid</code>. The notation used within the manual to denote “call to CPUID with EAX=X” is CPUID.XH</code>. You can then write the different values in the output registers as CPUID.XH:REGISTER.FIELD_NAME</code>, or with a binary range after the register name. We’ll be using this notation here too, for coherence with the Intel Software Developer Manual. You’re still reading a blog post about QEMU and perf</code> support by the way. Table 21-30 of the manual presents the list of fields for eax</code> and ebx</code> for CPUID.0AH</code>. Not all fields really matter here, we are interested in CPUID.0AH:EAX[7:0]</code> (aka VERSION</code>, aka eax & 0xff</code>), and CPUID.0AH:EAX[15:8]</code>1</a> (aka NUM_GP_CTRS</code>, aka ((eax >> 8) & 0xff)</code>), the number of general-purpose hardware counters per core. The manual says “This leaf is valid if CPUID.OAH:EAX[7:0] > 0</code> and MAX_LEAF</code> ≥ 09H</code>”. Those map exactly with the code above. So, yeah, that code checks if we have a PMU. Great! Cool. Wait. So, if X86_FEATURE_ARCH_PERMON</code> is not set, it means that CPUID.0AH:EAX.VERSION</code> is null, or there’s no general-purpose hardware counters, or the maximum CPUID leaf is 9 or below. I actually know that it’s the first two, because i then modified the call as follows: if (c->cpuid_level > 9) { unsigned eax = cpuid_eax(10); /* Check for version and the number of counters */ if ((eax & 0xff) && (((eax>>8) & 0xff) > 1)) set_cpu_cap(c, X86_FEATURE_ARCH_PERFMON); else { unsigned ebx = cpuid_ebx(10); unsigned ecx = cpuid_ecx(10); unsigned edx = cpuid_edx(10); pr_warn("THERE'S NO PMU AAAAH EAX=%x, EBX=%x ECX=%x EDX=%x", eax, ebx, ecx, edx); } }</code></pre> And what i got as a result was THERE'S NO PMU AAAAH EAX=0, EBX=0, ECX=0, EDX=0</code></pre> Fuck. Ok, quick thinking. Who’s making those responses? Who’s the mastermind behind it all? That’s right: The Hypervisor. ##</a> Part 4: KVM is a Little Liar</h2> We get to bring in KVM now. Remember the third place where X86_FEATURE_ARCH_PERFMON</code> was mentioned in that list above? The code is as follows: case 0xa: { /* Architectural Performance Monitoring */ union cpuid10_eax eax; union cpuid10_edx edx; if (!enable_pmu || !static_cpu_has(X86_FEATURE_ARCH_PERFMON)) { entry->eax = entry->ebx = entry->ecx = entry->edx = 0; break; } eax.split.version_id = kvm_pmu_cap.version; eax.split.num_counters = kvm_pmu_cap.num_counters_gp; eax.split.bit_width = kvm_pmu_cap.bit_width_gp; eax.split.mask_length = kvm_pmu_cap.events_mask_len; edx.split.num_counters_fixed = kvm_pmu_cap.num_counters_fixed; edx.split.bit_width_fixed = kvm_pmu_cap.bit_width_fixed; if (kvm_pmu_cap.version) edx.split.anythread_deprecated = 1; edx.split.reserved1 = 0; edx.split.reserved2 = 0; entry->eax = eax.full; entry->ebx = kvm_pmu_cap.events_mask; entry->ecx = 0; entry->edx = edx.full; break; }</code></pre> So this is a software implementation of CPUID.0AH</code>! Neat. This is in a function called __do_cpuid_func</code> in kvm/cpuid.c</code>. If you follow the chain of calls all the way back to /virt/kvm/kvm_main.c</code></a> (__do_cpuid_func</code> ← do_cpuid_func</code> ← get_cpuid_func</code> ← kvm_dev_ioctl_get_cpuid</code> ← kvm_arch_dev_ioctl</code> ← kvm_dev_ioctl</code>), you will find the definition of the ioctl handler on the device that becomes /dev/kvm</code></a>. I hope you already know what an ioctl</a> is. You wouldn’t be so deep into this post otherwise, probably. That tracks with my knowledge of how virtualization works in KVM: some operations are really performed by catching an instruction/interrupt/whatever from the VM context, getting out of the VM context, and poking the hypervisor to query the result that should be given. Here, we’re just doing it with CPUID.0AH</code>. And so, that implementation of CPUID.0AH</code>, how does it synthesize the answer? Well, as you can see, it checks for enable_pmu</code> and X86_FEATURE_ARCH_PERFMON</code>, and if either are false/disabled, it just sets everything to 0 and returns; otherwise, it shoves the information we want inside eax</code>, ebx</code> and edx</code>. i know that all of my registers were 0</code> when i printed them earlier, and kvm_pmu_cap.version</code> must be above 0, because otherwise the host would have no PMU, right? But my host has a PMU? i checked at boot: [ 0.116715] Performance Events: XSAVE Architectural LBR, PEBS fmt4+-baseline, AnyThread deprecated, Alderlake Hybrid events, 32-deep LBR, full-width counters, Intel PMU driver. [ 0.116715] core: cpu_core PMU driver: [ 0.116715] ... version: 5 [ 0.116715] ... bit width: 48 [ 0.116715] ... generic counters: 8 [ 0.116715] ... generic bitmap: 00000000000000ff [ 0.116715] ... fixed-purpose counters: 4 [ 0.116715] ... fixed-purpose bitmap: 000000000000000f [ 0.116715] ... value mask: 0000ffffffffffff [ 0.116715] ... max period: 00007fffffffffff [ 0.116715] ... global_ctrl mask: 0001000f000000ff</code></pre> Yeah, version is 5</code>! So something’s off. Either enable_pmu</code> is false, or static_cpu_has</code> fails to find X86_FEATURE_ARCH_PERFMON</code>. enable_pmu</code> is defined at arch/x86/kvm/x86.c#L185</code></a> in v6.18. It is a parameter for the kvm</code> kernel module that defaults to true</code>. i couldn’t check the actual value, but, considering i had never touched the default settings on kvm</code>, i assumed it was true? It couldn’t be otherwise. But, X86_FEATURE_ARCH_PERFMON</code> is also enabled, right? But then that’s impossible. i mean, the if block should not trigger otherwise, so, like, what’s going on? What’s going on?? ##</a> Part 5: The Meltdown</h2> So, to summarize what we know so far: My host machine boots.</li> The init_intel</code> function runs, detects maximum leaf count above 9, and a CPUID.0AH:EAX.VERSION</code> with more than 1 general-purpose counter per core, so it enables the X86_FEATURE_ARCH_PERFMON</code> capability on the boot CPU.</li> Later, intel_pmu_init</code> on the host does not trip on the check for that capability, and the PMU is properly initialized.</li> The virtual machine boots.</li> In init_intel</code>, it calls cpuid_eax(10)</code></li> On the host, an ioctl is fired to /dev/kvm</code> to synthesize the results of CPUID.0AH</code></li> On the host, kvm</code>’s __do_cpuid_func</code> trips on the if to check whether the PMU is virtualized, and, after setting all registers to 0, returns.</li> The guest machine receives the answer to cpuid_eax(10)</code>, and does not enable X86_FEATURE_ARCH_PERFMON</code> on the boot CPU</li> The guest machine runs intel_pmu_init</code>, which trips on the if block to check for the arch_perfmon</code> capability.</li> The family of the CPU being p6</code>, the guest kernel tries to run p6_pmu_init</code>.</li> On Linux 6.12, the error “Unsupported CPU” happens directly inside p6_pmu_init</code>, which returns -ENODEV</code>, which will fail upwards to callers and fail the PMU initialization as a whole, leaving no access to hardware counters for perf</code>.</li> On Linux 6.18, the error “Unsupported CPU” happens after we check for the model of the CPU, and realize it’s younger than/equal to a Yonah core, so the p6_pmu_init</code> call is useless, and we fail right now, leaving no hardware counter available to perf</code>.</li> </ul> At this point, i tried to fake the calls to cpuid_eax</code> by hard-coding values i expected, and it did not work. i tried intense googling of more and more obscure keywords, and it did not work. i tried forcing the enable_pmu</code> parameter on kvm</code> and reloading kvm_intel</code> too while we’re at it. it didn’t work. During my googling, i had found a couple of KVM patches from folks trying to introduce changes to the way KVM handles Hybrid architectures, sometimes outright disabling it. Remember how my computer has a 13th gen Intel CPU? Most consumer-grade Intel CPUs starting with 12th gen are heterogeneous2</a>, that is to say that even though the CPU appears as one coherent unit, the actual cores inside behave differently: some are optimized for fast tasks but consume more energy, and others are optimized for tasks that don’t need to be fast, but can comfortably run slower or sparsely if that means reducing the power draw. One patch i found discussed issues with KVM and heterogeneous architectures: unless you properly pin the virtualizing processes on the host, the guest could receive incoherent messages about hardware counters on its CPUs. Until KVM had a better way of reasoning between heterogeneous vCPUs and real heterogeneous CPUs 3</a>, they argued, virtualized PMU should be disabled. That patch</a> seemed to make a point, but, checking for its name in the git log</code> of my version of the kernel, there’s no trace of it. Out of desperation, i end up trying to grep the individual words, KVM</code>, PMU</code>, and i skim the output, until i find: 4d7404e5ee00 KVM: x86/pmu: Disable vPMU support on hybrid CPUs (host PMUs)</code></pre> OH COME ON- ##</a> Part 6: This is Where i Complain. A Lot.</h2> Let’s inspect commit 4d7404e5ee00</code>: commit 4d7404e5ee0066e9a9e8268675de8a273b568b08 Author: Sean Christopherson <seanjc@google.com> Date: Wed Feb 8 20:42:29 2023 +0000 KVM: x86/pmu: Disable vPMU support on hybrid CPUs (host PMUs) Disable KVM support for virtualizing PMUs on hosts with hybrid PMUs until KVM gains a sane way to enumeration the hybrid vPMU to userspace and/or gains a mechanism to let userspace opt-in to the dangers of exposing a hybrid vPMU to KVM guests. Virtualizing a hybrid PMU, or at least part of a hybrid PMU, is possible, but it requires careful, deliberate configuration from userspace. E.g. to expose full functionality, vCPUs need to be pinned to pCPUs to prevent migrating a vCPU between a big core and a little core, userspace must enumerate a reasonable topology to the guest, and guest CPUID must be curated per vCPU to enumerate accurate vPMU capabilities. The last point is especially problematic, as KVM doesn't control which pCPU it runs on when enumerating KVM's vPMU capabilities to userspace, i.e. userspace can't rely on KVM_GET_SUPPORTED_CPUID in it's current form. Alternatively, userspace could enable vPMU support by enumerating the set of features that are common and coherent across all cores, e.g. by filtering PMU events and restricting guest capabilities. But again, that requires userspace to take action far beyond reflecting KVM's supported feature set into the guest. For now, simply disable vPMU support on hybrid CPUs to avoid inducing seemingly random #GPs in guests, and punt support for hybrid CPUs to a future enabling effort.</code></pre> And so on and so on. I’m sparing you the Signed-off-by’s and the Cc’s and everything. The code of the commit is: diff --git a/arch/x86/kvm/pmu.h b/arch/x86/kvm/pmu.h index cdb91009701d..ee67ba625094 100644 --- a/arch/x86/kvm/pmu.h +++ b/arch/x86/kvm/pmu.h @@ -165,15 +165,27 @@ static inline void kvm_init_pmu_capability(void) { bool is_intel = boot_cpu_data.x86_vendor == X86_VENDOR_INTEL; - perf_get_x86_pmu_capability(&kvm_pmu_cap); - - /* - * For Intel, only support guest architectural pmu - * on a host with architectural pmu. - */ - if ((is_intel && !kvm_pmu_cap.version) || !kvm_pmu_cap.num_counters_gp) + /* + * Hybrid PMUs don't play nice with virtualization without careful + * configuration by userspace, and KVM's APIs for reporting supported + * vPMU features do not account for hybrid PMUs. Disable vPMU support + * for hybrid PMUs until KVM gains a way to let userspace opt-in. + */ + if (cpu_feature_enabled(X86_FEATURE_HYBRID_CPU)) enable_pmu = false; + if (enable_pmu) { + perf_get_x86_pmu_capability(&kvm_pmu_cap); + + /* + * For Intel, only support guest architectural pmu + * on a host with architectural pmu. + */ + if ((is_intel && !kvm_pmu_cap.version) || + !kvm_pmu_cap.num_counters_gp) + enable_pmu = false; + } + if (!enable_pmu) { memset(&kvm_pmu_cap, 0, sizeof(kvm_pmu_cap)); return;</code></pre> It replaced a check for kvm_pmu_cap.version</code> and num_counters_gp</code> (akin to what you do with CPUID.0AH</code>) by a cute little if block that checks X86_FEATURE_HYBRID_CPU</code>, and disables vPMU behind your back if that’s enabled. Without warning. In 6.18, there is still no warning there. They only added a memset</code> to zero-out the kvm_host_pmu</code> structure. In short, my options are few: If i feel like tweaking the host’s kernel, i could comment the line that disables enable_pmu</code>, and try to work with that (for multiple reasons, it’s impractical as hell).</li> Move my experiments to another machine.</li> </ul> ##</a> Hidden Assumptions, Miscommunication</h2> So, let’s recap together what went wrong: When perf</code> showed unsupported hardware counters, i searched documentation on how to enable vPMU, which had no visible date nor indications of assumptions regarding architecture (in all fairness, a retroactive decision like that to disable vPMU for all heterogeneous x86 CPUs is not something people track to then go update blog posts).</li> The error message in dmesg</code> was originally in a function with a confusing semantic, that was fixed some time between 6.14 and 6.15 when it moved from p6_pmu_init</code> to intel_pmu_init</code>.</li> The error message talks about an unsupported CPU. It assumes the CPU not having a PMU exposed via CPUID.0AH</code> is a problem with the model, so it exposes information related to your CPU model, even when that CPU model should, by all other indicators, have a PMU.</li> When i found patch sets on the Linux Kernel Mailing List that discussed vPMU in KVM, i had no way to track whether they had been merged or not. In fact, i ended up finding an earlier draft of the commit that bit my ass, but because it had slightly different commit name formatting, i gave up and assumed it had not been merged.</li> There was no error message about vPMU being forcefully disabled in KVM. This decision has several assumptions: the user does not need to know KVM is running, or they do not know what a hybrid x86 architecture is, or they will not know why it is disabled, and they will complain because they can’t fix it. It still baffles me that nobody thought to at least print something,</li> </ul> Look, i am used to this. i am used to falling through the cracks of people’s assumptions: i daily-drive Arch Linux, i run shit on QEMU via the command line only, i build my own kernel modules sometimes, i need old software, i jam programs together that were never meant to work together. There’s glue though, there’s things we, as people who touch computers have agreed on in order for communication to work properly across implementations. CPUID is one such example, and so are ioctls, kernel module parameters, network protocols, perf sample representations, or even logging messages. Logging is the most person-oriented way of conveying that something went wrong, or any complex information that cannot be shoved in a bitfield for easy access: when i read the message about “Unsupported CPU”, that was a programmer, 14 years ago, telling me “Hey, if you hit this line of code, it means you plugged a CPU into your motherboard that does not have a PMU yet!”. For a second, someone’s thought process was conveyed to me, alongside their assumptions about the meaning of a Intel CPU reporting no arch_perfmon</code> capability. Look, i’m not blaming the Linux devs. Entirely. There may be reasonable arguments for why they did not add a log line in that if block, or why nobody thought about adding that at any point in reviewing the sets. Maybe they had assumptions about the users, about the systems, that they did not communicate. i’m more mad about how we keep miscommunicating at each other, especially about assumptions we make about hardware, software, their interactions, and how i keep losing my time and my mind debugging the friction between all of them. It’s tiring, frankly. … Oh and fuck Intel, really while we’re at it. That’s all. Bye! Notice how the Intel Software Developer manual does bit ranges backwards? Ah, little endian architectures… ↩</a> </li> To be more precise, mainstream desktop 12th gen Alderlake CPUs are still homogeneous</a>. Thirteenth gen, aka Raptor Lake</a>, are heterogeneous except for i3s, and similarly for the 14th gen</a>. ↩</a> </li> So, i’m not a KVM fox. i’m not even that much of an architecture fox, i just know people, and i hate hardware with a passion so intense that occasionally i end up learning way too much about it. My assumptions about the more precise problems with KVM and heterogeneous architectures, from what i understood, is that the available performance counters exposed on the host are those of the performance CPUs, which are typically more advanced. Those are also the sets exposed to the guest. However, if the guest is scheduled on an efficiency CPU, and tries to probe a counter that is only available on a performance CPU, it will fail, or there will be undefined behaviour that has never been accounted for anywhere so far in the kernel or perf</code>. Another assumption is that exposing which CPU you’re currently scheduled on to the guest is… a gamble. Knowing your position in a topology, especially if you’re a malicious actor, especially if you’re scheduled with other users of a system, opens up a lot of possibilities to probe your cache neighbours, or even mess with them. ↩</a> </li> </ol> </section> Guarding My Git Forge Against AI Scrapers 2025-12-02T00:00:00+00:00 In August 2024, one of my roommates and partners messaged the apartment group chat, saying she noticed the internet was slow again at our place, and my forgejo</a> was unable to render any page in under 15 seconds. i investigated, thinking it would be a trivial little problem to solve. Soon enough, however, i would uncover hundreds of thousands of queries a day from thousands of individual IPs, fetching seemingly-random pages in my forge every single day, all the time. This post summarizes the practical issues that arose as a result of the onslaught of scrapers eager to download millions of commits off of my forge, and the measures i put in place to limit the damage. #</a> Why the forge?</h1> In the year 2025, on the web, everything is worth being scraped. Everything that came out of the mind of a human is susceptible to be snatched under the vastest labor theft scheme in the history of mankind. This very article, the second it gets published in any indexable page, will be added to countless datasets meant to train foundational large-language models. My words, your words, have contributed infinitesimal shifts of neural-network weights underpinning the largest, most grotesque accumulation of wealth seen over the lifetime of my parents, grandparents, and their grandparents themselves. Oh, and forges have a lot of commits. See, if you have a public repository that is publicly exposed, every file in every folder for every commit will be connected. Add other options, such as a git blame</code> on a file, and multiply it by the number of files and commits. Add the raw download link, also multiplied by the number of commits. Say, hypothetically, you have a linux repository available, and only with all the commits in the master</code> branch up to the v6.17</code> tag from 2025-09-18. That’s 1,383,738 commits in the range 1da177e4c3f4..e5f0a698b34e</code>. How many files is that? Well: count=0; while read -r rev; do point=$(git ls-tree -tr $rev | wc -l); count=$(( $count + $point )); printf "[%s] %s: %d (tot: %d)\n" $(git log -1 --pretty=tformat:%cs $rev) $rev $point $count; done < <(git rev-list "1da177e4c3f4..e5f0a698b34e"); printf "Total: $count\n";</code></pre> i ran this on the 100 commits before v6.17</code>. If you have git ls-tree -tr $rev</code>, you get both files and directories counted. If you replace it with git ls-tree -r $rev</code> only shows files. i got 72024729 files, and 76798658 files and directories. Running on the whole history of Linux’s master</code> branch yields 78,483,866,182 files, and 83,627,462,277 files and directories. Now, for a ballpark estimate of the number of pages that can be scraped if you have a copy of Linux, apply the formula: (Ncommits * Nfiles) * 2 + (Ncommits * Nfilesandfolders) * 2 + Ncommits * 3</code></pre> That is, applied to my hypothetical Linux repository: 78483866182 * 2 + 83627462277 * 2 + 1383738 * 3 = 324,226,808,132 pages</code></pre> The *3</code> accounts for the fact that every file of every commit can be scraped raw, and git-blame</code>’d. The second part of the formula considers every single file or folder page. The third part accounts for the fact that every file of every commit can be diffed with its version of every commit (in theory). The final component considers every commit summary page. That gives, for me, 324 billion 226 million 808 thousand and 132 pages that can be scraped. From a single repository. Assume that every scraper agent that enters one of these repositories will also take note of every other link on the page, and report it so that other agents can scrapes them. These scrapers effectively act like early 2000s web spiders that crawled the internet to index it, except they do not care about robots.txt</code>, and they will absolutely keep scraping new links again and again with no strategy to minimize the cost on you, as a host. #</a> The Cost of Scraping</h1> As i am writing the original draft of this section, the longer-term measures i put in place have been removed, so i could gather up-to-date numbers to display how bad the situation is. i pay for the electricity that powers my git Forge. Okay, actually, one of my roommate does, but we put it on the calc sheet where we keep track of who pays what (when we remember). At the time i began fighting scrapers, my git forge ran from an old desktop computer plugged in my living room. Now, it is in our home’s rackable server in a virtual machine. i never got to measure differences in power consumption when we got scraped or not scraped on the desktop machine, but i did on the rackable server. If memory serves me right, stopping the wave of scrapers reduced the power draw of the server from ~170W to ~150W. Right now, with all the hard drives in that server spinning, and every protection off, we are drawing 200W from the power grid on that server. Constantly. By the end of this experiment, me and my roommates will have computed that the difference in power usage caused by scraping costs us ~60 euros a year. Another tied cost is that the VM that runs the forge is figuratively suffocating from the amount of queries. Not all queries are born equal as well: requests to see the blame</code> of a file or a diff</code> between commits incurs a worse cost than just rendering the front page of a repository. The last huge wave of scraping left my VM at 99+% usage of 4 CPU cores and 2.5GiB of RAM</a>, whereas the usual levels i observe are closer to 4% usage of CPUs, and an oscillation between 1.5GiB and 2GiB of RAM. As i’m writing this, the VM running forgejo eats 100% of 8 CPU cores. Additionally, the networking cost is palpable. Various monitoring tools let me see the real-time traffic statistics in our apartment. Before i put the first measures in place to thwart scraping, we could visibly see the traffic coming out of the desktop computer running my forge and out to the internet. My roommates’ complaints that it slowed down the whole internet here were in fact founded: when we had multiple people watching live streams or doing pretty big downloads, they were throttled by the traffic out of the forge.1</a> The egress data rate of my forge’s VM is at least 4MBps of data (32Mbps). Constantly. Finally, the human cost: i have spent entire days behind my terminals trying to figure out 1) what the fuck was going on and 2) what the fuck to do about it. i have had conversations with other people who self-host their infrastructure, desperately trying to figure out workable solutions that would not needlessly impact our users. And the funniest detail is: that rackable server is in the living room, directly in front of my bedroom door. It usually purrs like an adorable cat, but, lately, it’s been whirring louder and louder. i can hear it. when i’m trying to sleep. #</a> Let’s do some statistics.</h1> i was curious to analyze the nginx logs to understand where the traffic came from and what shape it took. As a study case, we can work on /var/log/nginx/git.vulpinecitrus.info/</code> from 2025-11-14</code> to 2025-11-19</code>. Note that on 2025-11-15</code> at 18:27 UTC</code>, i stopped the redirection of new agents into the Iocaine crawler maze (see below). At 19:15 UTC</code>, i removed the nginx request limit zone from the /Lymkwi/linux/</code> path. At 19:16 UTC</code> i removed the separation of log files between IPs flagged as bots, and IPs not flagged as bots. The three measures i progressively put in place later were: web caching (2025-11-17), manually sending IPs to a garbage generator with a rate-limit (Iocaine 2) (2025-11-14, 15 and 18), and then Iocaine 3</a> (2025-11-19). Common Logs</th> Successful</th> Delayed (429)</th> Error (5XX)</th> Measures in place</th></tr></thead> 2025-11-14</td> 275323</td> 66517</td> 0</td> Iocaine 2.1 + Rate-limiting</td></tr> 2025-11-15</td> 71712</td> 54259</td> 9802</td> Iocaine 2.1 + Rate-limiting</td></tr> 2025-11-16</td> 140713</td> 0</td> 65763</td> None</td></tr> 2025-11-17</td> 514309</td> 25986</td> 3012</td> Caching, eventually rate-limiting^{2</a></td></tr>} 2025-11-18</td> 335266</td> 20280</td> 1</td> Iocaine 2.1 + Rate-limiting</td></tr> 2025-11-19</td> 3183</td> 0</td> 0</td> Iocaine 3</td></tr> Bot Logs</td> Successful</td> Delayed (429)</td> Error (5XX)</td> Measures in place</td></tr> 2025-11-14 (bots)</td> 41388</td> 65517</td> 0</td> Iocaine 2.1 + Rate-limiting</td></tr> 2025-11-15 (bots)</td> 34190</td> 53403</td> 63</td> Iocaine 2.1 + Rate-limiting</td></tr> 2025-11-16 (bots)</td> -</td> -</td> -</td> (no bot-specific logs)</td></tr> 2025-11-17 (bots)</td> -</td> -</td> -</td> (no bot-specific logs)</td></tr> 2025-11-18 (bots)</td> 390013</td> 0</td> 13</td> Iocaine 2.1 + Rate-limiting</td></tr> 2025-11-19 (bots)</td> 731593</td> 0</td> 0</td> Iocaine 3</td></tr> </tbody></table> Table 1: Number of Queries Per Day</div> (Commands used to generate Table 1)</summary> Assuming your log file is git-access-2025-11-14.log.gz</code>: zcat git-access-2025-11-14.log.gz | grep '" 200 ' | wc -l zcat git-access-2025-11-14.log.gz | grep '" 429 ' | wc -l</code></pre> </details> Without spoiling too much, caching was an utter failure, and the improvement i measurement by manually rate-limiting a set of IPs (from Huawei Cloud and Alibaba) on the Linux repository only helped so much. When all protections dropped, my server became so unresponsive that backend errors (usually timeouts) spiked. Error also happened with caching, when nginx encountered an issue when buffering a reply. Overall, caching encouraged more queries overall. Once Iocaine was deployed, the vast majority of queries were routed away from the backend, with no errors reported, and no delaying because all of the IPs i manually rate-limited were caught by Iocaine instead. Out of all these queries, 117.64.70.34</code> is the most common source of requests, with 226023 total queries originating from the ChinaNet-Backbone ASN (AS4134). It is followed by 136.243.228.193</code> (13849 queries), an IP from Hetzner whose hostname ironically resolves to crawling-gateway-136-243-228-193.dataforseo.com</code>. Then, 172.17.0.3</code> the uptime prober of VC Status</a> with 6908 queries, and 74.7.227.127</code>, an IP from Microsoft’s AS 8075 (6117 queries). Day</th> Unique IP Count</th></tr></thead> 2025-11-14</td> 16461</td></tr> 2025-11-15</td> 18639</td></tr> 2025-11-16</td> 41712</td></tr> 2025-11-17</td> 47252</td></tr> 2025-11-18</td> 22480</td></tr> 2025-11-19</td> 14230</td></tr> </tbody></table> </div> Table 2: Grand Total of Unique IPs Querying the Forge</div> (Commands used to generate Table 2)</summary> Assuming your log files are called *git-access-2025-11-14.log.gz</code>: zcat \*git-access-2025-11-14.log.gz | awk '{ print $1 }' | sort | uniq -c | wc -l</code></pre> </details> On the two days where restrictions were lifted or there was only caching, the amount of unique IPs querying the forge doubled. The more you facilitate the work of these crawlers, the more they are going to pound you. They will always try and get more out of your server than you are capable of providing. Day</th> Top 1</th> Top 2</th> Top 3</th> Top 4</th> Top 5</th></tr></thead> 2025-11-14</td> (226089) - /reibooru/reibooru</code></td> (40189) - /Lymkwi/linux</code></td> (1454) - /</code></td> (1405) - /rail</code></td> (1174) - /Soblow/indi-hugo</code></td></tr> 2025-11-15</td> (35163) - /Lymkwi/linux</code></td> (18952) - /vc-archival/youtube-dl</code></td> (4197) - /vc-archival/youtube-dl-original</code></td> (1655) - /reibooru/reibooru</code></td> (1635) - /Lymkwi/gr-gsm</code></td></tr> 2025-11-14 (bots)</td> (40189) - /Lymkwi/linux</code></td> (270) - /oror/necro</code></td> (79) - /Lymkwi/[REDACTED]</code>^{3</a></td>} (55) - /vc-archival/youtube-dl</code></td> (52) - /oror/asm</code></td></tr> 2025-11-15 (bots)</td> (32895) - /Lymkwi/linux</code></td> (260) - /oror/necro</code></td> (193) - /Lymkwi/gr-gsm</code></td> (95) - /Lymkwi/[REDACTED]</code>^{3</a></td>} (48) - /alopexlemoni/GenderDysphoria.fyi</code></td></tr> 2025-11-16</td> (72687) - /vc-archival/youtube-dl</code></td> (23028) - /Lymkwi/linux</code></td> (16779) - /vc-archival/youtube-dl-original</code></td> (5390) - /reibooru/reibooru</code></td> (3585) - /Lymkwi/gr-gsm</code></td></tr> 2025-11-17</td> (361632) - /vc-archival/youtube-dl</code></td> (74048) - /vc-archival/youtube-dl-original</code></td> (18136) - /reibooru/reibooru</code></td> (13147) - /oror/necro</code></td> (12921) - /alopexlemoni/GenderDysphoria.fyi</code></td></tr> 2025-11-18</td> (227019) - /vc-archival/youtube-dl</code></td> (46004) - /vc-archival/youtube-dl-original</code></td> (12644) - /alopexlemoni/GenderDysphoria.fyi</code></td> (12624) - /reibooru/reibooru</code></td> (7712) - /oror/necro</code></td></tr> 2025-11-18 (bots)</td> (261346) - /vc-archival/youtube-dl</code></td> (43923) - /vc-archival/youtube-dl-original</code></td> (20195) - /alopexlemoni/GenderDysphoria.fyi</code></td> (18808) - /reibooru/reibooru</code></td> (10134) - /oror/necro</code></td></tr> 2025-11-19</td> (1418) - /</code></td> (1248) - /rail</code></td> (356) - /Soblow</code></td> (31) - /assets/img</code></td> (25) - /Soblow/IndigoDen</code></td></tr> 2025-11-19 (bots)</td> (448626) - /vc-archival/youtube-dl</code></td> (73164) - /vc-archival/youtube-dl-original</code></td> (39107) - /reibooru/reibooru</code></td> (37107) - /alopexlemoni/GenderDysphoria.fyi</code></td> (25921) - /vc-archival/YSLua</code></td></tr> </tbody></table> </div> Table 3: Top 5 Successful Repo/Account/Page Hits Per Day</div> (Commands used to generate Table 3)</summary> Assuming you want data for the log file called git-access-2025-11-14.log.gz</code>: zcat git-access-2025-11-14.log.gz | grep '" 200 ' | awk '{ print $7 }' \ | cut -d/ -f -3 | sort | uniq -c | sort -n \ | tail -n 5 | tac</code></pre> </details> Big repositories with a lot of commits and a lot of files are a bountiful resource for the crawlers. Once they enter those, they will take ages to leave, at least because of the sheer amount of pages that can be generated by following the links of a repository. Most legitimate traffic seems to be either fetching profiles (a couple of my users have their profiles listed in their fediverse bios) or the root page of my forge. </th> 2025-11-14 (all)</th> 2025-11-15 (all)</th> 2025-11-16 (all)</th></tr></thead> Top 1</td> (8532) - AS136907 (Huawei Clouds)</td> (8537) - AS136907 (Huawei Clouds)</td> (8535) - AS136907 (Huawei Clouds)</td></tr> Top 2</td> (2142) - AS45899 (VNPT Corp)</td> (2107) - AS45899 (VNPT Corp)</td> (4002) - AS212238 (Datacamp Limited)</td></tr> Top 3</td> (803) - AS153671 (Liasail Global Hongkong Limited)</td> (895) - AS153671 (Liasail Global Hongkong Limited)</td> (3504) - AS9009 (M247 Europe SRL)</td></tr> Top 4</td> (555) - AS5065 (Bunny Communications)</td> (765) - AS45102 (Alibaba US Technology Co., Ltd.)</td> (3206) - AS3257 (GTT Communications)</td></tr> Top 5</td> (390) - AS21859 (Zenlayer Inc)</td> (629) - AS5065 (Bunny Communications)</td> (2874) - AS45899 (VNPT Corp)</td></tr> </tbody></table> </div> Table 4: Top ASN Per Day For The First Three Days, Per Unique IP Count</div> (Commands used to generate Table 4)</summary> For this, i needed a database of IP-to-ASN data. i got one from IPInfo</a> by registering for a free account and using their web API. i first scripted a mapping of unique IP addresses to AS number. For example, for the log file bot-git-access-2025-11-18.log.gz</code>: while read ip; do ASN=$(curl -qfL api.ipinfo.io/lite/$ip?token=<my token> | jq -r .asn); printf "$ip $ASN\n" | tee -a 2025-11-18-bot.ips.txt; done < <(zcat bot-git-access-2025-11-18.log.gz | awk '{ print $1 }' | sort | uniq)</code></pre> Then, with this map, i run: cat 2025-11-18-bot.ips.txt | cut -d' ' -f 2 | sort | uniq -c | sort -n | tail -n 5</code></pre> </details> So my largest hits are from Huawei Clouds (VPS provider), VPNT (a Vietnamese mobile and home ISP), Liasail Global HK Limited (a VPS/“AI-powering service” provider), Bunny Communications LLC (a broadband ISP for residential users), and Zenlayer (CDN/Cloud infrastructure provider). When i lifted all protections, Datacamp Limited (a VPS provider), GTT Communications (some sort of bullshit-looking ISP4</a> who, i have been informed, is in fact a backbone operator), and M247 Europe SRL (a hosting provider) suddenly appeared. If memory serves me right, Datacamp, GTT and M247 were also companies i had flagged during my initial investigation in summer 2024, and added to the manually blocked/limited IPs alongside all of Huawei Cloud and Alibaba. Interestingly, both Liasail and Zenlayer mention that they “Power AI” on their front page. They sure do. Worryingly, VNPT and Bunny Communications are home/mobile ISPs. i cannot ascertain for sure that their IPs are from domestic users, but it seems worrisome that these are among the top scraping sources once you remove the most obviously malicious actors. #</a> The Protection Measures</h1> i have one goal, and one constraint. My goal is that i need to protect the forge as much as possible, by means of either blocking bots or offloading the cost to my VPS provider (whose electricity i do not pay for). My only constraint: i was not going to deploy a proof-of-work-based captcha system such as Anubis</a>. There are two reasons for these constraints: i personally find that forcing your visitors to have to expand more computational power to prove they’re not a scraper is bad praxis. There are devices out there that legitimately want that access, but have limited computational power or features. And, yeah, there are multiple types of challenges</a>, some of which take low-power devices into account</a> or even those that cannot run JavaScript</a>, but,</li> Scrapers can easily bypass Anubis. It’s not a design flaw. Anubis is harm reduction, not pixie dust.</li> </ol> i tried layers of solutions: caching on the reverse proxy</li> Iocaine 2 with no classifiers, which generates garbage in reply to any query you send it</li> Manually redirecting IPs and rate-limiting them</li> Deploying Iocaine 3, with its classifiers (Nam-Shub-of-Enki)</li> </ul> ##</a> Reverse-Proxy Caching</h2> i have a confession to make: i never realized that nginx did not cache anything by default. That realization promptly came with the other realization that caching things correctly is hard. i may, some day, write about my experience of protecting a service that posted links to itself on the fediverse, so that it wouldn’t slow to a crawl for ten minutes after every post. As for the rest of these, i will be showing my solution in nginx</code>. You can, almost certainly, figure out a way of doing exactly the same thing with any other decent reverse proxy software. To create a cache for my forge, i add the following line to /etc/nginx.conf</code>: proxy_cache_path /var/cache/nginx/forge/ levels=1:2 keys_zone=forgecache:100m;</code></pre> That will create a 2-level cache called forgecache</code> that will hold 100MB of data</del> 100MB of in-memory index data for a cache located at /var/cache/nginx/forge</code>. i create the directory and make www-data</code> its owner and group. In /etc/nginx/sites-enabled/vcinfo-git.conf</code>, where my git forge’s site configuration sits, i have a location</code> block that serves the whole root of the service, which i modify thusly: location / { proxy_cache forgecache; proxy_buffering on; proxy_cache_valid any 1h; add_header X-Cached $upstream_cache_status; expires 1h; proxy_ignore_headers "Set-Cookie"; proxy_hide_header "Set-Cookie"; # more stuff... }</code></pre> That configuration does several things: it turns on caching and buffering at the proxy (proxy_buffering</code></a>), telling it to use forgecache</code> (proxy_cache</code></a>) and keep any page valid for an hour (proxy_cache_valid</code></a>). It also adds a cookie that will let you debug whether or not a query hit or missed the cache (add_header</code>). The expires</code> directive adds headers telling your visitor’s browser that the content they cache will also expire in an hour (expires</code></a>). Finally, the cache ignores any response header that sets a cookie (proxy_ignore_headers</code></a>, proxy_hide_header</code></a>), to attempt to remove any page that could be customized for a user once they log in. EDIT 2026-03-19 As it turns out, proxy_ignore_headers</code> on Set-Cookie</code> is a catastrophic idea. Digging into the documentation, you can find out after two or three indirections that, by default, caching is disabled if the Set-Cookie</code> header is present. By ignoring it, i essentially told nginx, “hey, please, cache my logged in pages!!”. The correct way to do what i want here is to remove proxy_ignore_headers</code> and proxy_hide_header</code> (it’s useless on Set-Cookie</code> by default - again, nginx will not cache those pages). The result with that configuration is the same however: caching pages when bots only access each URL once is useless. Also, you do not need 100m</code> of cache index data. Holy fuck. That is way too much. 5 Megabytes is already enough. </div> </div> The result? Caching was a disaster, predictably so. Caching works when the same resource is repeatedly queried, like with page assets, JavaScript, style sheets, etc. In this case, the thousands of actors querying my forge are coordinated, somehow, never (or rarely) query the same resource twice, and only download the raw HTML of the web pages. Worse, caching messed up the display of authenticated pages. The snippets above are not enough to delineate between an authenticated session and an unauthenticated one, and it broke my forge so badly that i had to disable caching and enable the next layer early on 2025-11-17</code>, or i just could not use my forge. ##</a> Rate-Limiting on the Proxy</h2> The next layer of protection simply consisted in enabling a global rate-limit on the most-hit repositories: limit_req_zone wholeforge zone=wholeforge:10m rate=3r/s; server { // ... location ^~ (/alopexlemoni/GenderDysphoria.fyi|/oror/necro|/Lymkwi/linux|/vc-archival/youtube-dl-original|/reibooru/reibooru) { proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_max_temp_file_size 2048m; limit_req zone=wholeforge nodelay; proxy_pass http://<my actual upstream>/; } }</code></pre> This was achieved in two directives. The first one, limit_req_zone</code>, sits outside the server {}</code> block and defines a zone called wholeforge</code> that stores 10MB of state data and limits to 3 requests per second. When this was in place, however, actually accessing the Linux repository as a normal user (or any of the often-hit repositories) became a nightmare of waiting and request timeouts. ##</a> Manually Redirecting to a Garbage Generator</h2> Because caching was (predictably) useless, and rate-limiting was hindering me as well, i re-enabled the initial setup that was in place before my experiments: manually redirecting queries to a garbage generator (in this case, an old version of Iocaine). It’s largely based on my initial setup following this tutorial in french</a>. For the purpose of this part, you do not have to know what Iocaine does precisely. In the next section, i will present my current and final setup, with an updated Iocaine that also includes a classifier to decide which queries are bots and which are regular users. For now, i will present the version where i manually chose who to return garbage to based on IP addresses. As a little bonus, it will also include rate-limiting of those garbage-hungry bots. i add a file called /etc/nginx/snippets/block_bots.conf</code> which contains: if ($bot_user_agent) { rewrite ^ /deflagration$request_uri; } if ($bot_ip) { rewrite ^ /deflagration$request_uri; } location /deflagration { limit_req zone=bots nodelay; proxy_set_header Host $host; proxy_pass <garbage upstream>; }</code></pre> This will force any query categorized as bot_user_agent</code> or bot_ip</code> to be routed through to a different upstrea which serves garbage. That upstream is also protected by rate-limiting on a zone called bots</code> which is defined in the next bit of code. This snippet is actually meant to be included in your server {}</code> block using the include</code> directive. i then add the following in /etc/nginx/conf.d/bots.conf</code>: map $http_user_agent $bot_user_agent { default 0; # from https://github.com/ai-robots-txt/ai.robots.txt/blob/main/robots.txt ~*amazonbot 1; ~*anthropic-ai 1; ~*applebot 1; ~*applebot-extended 1; ~*brightbot 1; ~*bytespider 1; ~*ccbot 1; ~*chatgpt-user 1; ~*claude-web 1; ~*claudebot 1; ~*cohere-ai 1; ~*cohere-training-data-crawler 1; ~*crawlspace 1; ~*diffbot 1; ~*duckassistbot 1; ~*facebookbot 1; ~*friendlycrawler 1; ~*google-extended 1; ~*googleother 1; ~*googleother-image 1; ~*googleother-video 1; ~*gptbot 1; ~*iaskspider 1; ~*icc-crawler 1; ~*imagesiftbot 1; ~*img2dataset 1; ~*isscyberriskcrawler 1; ~*kangaroo 1; ~*meta-externalagent 1; ~*meta-externalfetcher 1; ~*oai-searchbot 1; ~*omgili 1; ~*omgilibot 1; ~*pangubot 1; ~*perplexitybot 1; ~*petalbot 1; ~*scrapy 1; ~*semrushbot-ocob 1; ~*semrushbot-swa 1; ~*sidetrade 1; ~*timpibot 1; ~*velenpublicwebcrawler 1; ~*webzio-extended 1; ~*youbot 1; # Add whatever other pattern you want down here } geo $bot_ip { default 0; # Add your IP ranges here } # Rate-limiting setup for bots limit_req_zone bots zone=bots:30m rate=1r/s; # Return 429 (Too Many Requests) to slow them down limit_req_status 429;</code></pre> That bit of configuration does a mapping between the client IP and a variable called bot_ip</code>, and the client’s user agent and a variable called bot_user_agent</code>. When a known pattern listed in those blocks is found, the corresponding variable is flipped to the provided value (here, 1</code>). Otherwise, it stays 0</code>. Then, we define the rate-limiting zone that is used to slow down the bots so they don’t feed on slop too fast. You will then need to install the http-geoip2</code> nginx module (on Debian-based distributions, something like apt install libnginx-mod-http-geoip2</code> will do). Once that is done, add the following line to the server</code> block of every site you want to protect: include /etc/nginx/snippets/block_bots.conf;</code></pre> And when you feel confident enough, roll a nginx -t</code> and reload the unit for nginx</code>. Now, if you’re using caddy</code> or any other reverse proxy, there are probably similar mechanisms available. You can go and peruse the documentation of Iocaine, or look online for specific tutorials that, i am sure, other people have made better than i would. Immediately after enabling it, and shoving all the IPs from Alibaba Cloud and Huawei Cloud in the bot config file, the activity slowed down on my server. Power usage went down to ~180W, CPU usage to rougly 60%, and it stopped making a hellish noise. As the stats showed earlier, however, a lot of traffic was still hitting the server itself. Even weirder, there were still occasional spikes, every 3 hour, that lasted about one and a half hour, where the server would whirr and forgejo suffocate again. Bots were still hitting my server, and there was no clear source for it. ##</a> Automatically Classifying Bots and Poisoning Them: Iocaine and Nam-Shub-of-Enki</h2> So far, the steps i showed so far help when a single IP is hammering at your forge, or when someone is clearly scraping you from an Autonomous System that you do not mind blocking. Sadly, as i’ve showed above in Table 4</a>, a surprising amount of scraping comes from broadband addresses. i can assemble lists of IPs as big as i want, or block entire ASNs, but i would love to have a per-query way of determining if a query looks legitimate. The next steps of protection will rely on categorizing a source IP based on its the credibility of its user agent. This mechanism is largely based on the documentation for Iocaine 3.x</a>. We finally get to talk about Iocaine! Iocaine is a tool that traps scrapers in a maze of meaningless pages that endlessly lead to more meaningless pages. The content of these pages is generated using a Markov chain, based on a corpus of texts given to the software. Iocaine (specifically all versions after 3 at least5</a>) is a middleware, in the sense that it works by being placed on the line between your reverse proxy and the service. Your reverse proxy will first begin by redirecting traffic to Iocaine, and, if Iocaine deems a query legitimate, it will return a 421 Misdirected Request</code></a> back at your reverse-proxy. The latter must then catch it, and use the real upstream as a fallback. If Iocaine’s Nam-Shub-of-Enki6</a> decides query came from a bogus or otherwise undesirable source, it will happily reply 200 OK</code> and send generated garbage. My setup lodges Iocaine 3 between nginx and my forge, following the Iocaine documentation to use the container version</a>. i recommend you follow it, and then add the next little things to enable categorization statistics, and prevent the logging they’re based on from blowing up your storage: In etc/config.d/03-nam-shub-of-enki.kdl</code>, change the logging block to:</li> </ol> logging { enable #true classification { enable #true } }</code></pre> In docker-compose.yaml</code>, add the following bits to limit classification logging to 50MB:</li> </ol> services: iocaine: # The things you already have here... # ... env: - RUST_LOG=iocaine=info logging: driver: "json-file" options: max-size: "50m"</code></pre> My checks block in Nam-Shub-of-Enki is as such: checks { disable cgi-bin-trap asn { database-path "/data/ipinfo_lite.mmdb" asns "45102" "136907" } ai-robots-txt { path "/data/ai.robots.txt-robots.json" } generated-urls { identifiers "deflagration" } big-tech { enable #true } commercial_scrapers { enable #true } }</code></pre> I snatched a copy of the latest ipinfo ASN database</a> for free and blocked AS52102 (Alibaba) and AS136907 (Huawei Clouds). On 2025-11-18 at 00:00:29 UTC+1, i enabled Iocaine with the Nam-Shub-of-Enki classifier in front of my whole forge. Immediately, my server was no longer hammered. Power draw went down to just above 160W. One problem i noticed however, while trying to deploy the artifact for this blog post on my forge, is that Iocaine causes issues when huge PUT</code>/PATCH</code>/POST</code> requests with large bodies are piped through it: it will hang up before the objects are entirely written. i am trying to figure out a way of only redirecting HEAD</code> and GET</code> requests to Iocaine in nginx, like is done in the Caddy example of the Iocaine documentation. What i ended up settling on requires a bit of variable mapping. At the start of your site configuration, before the server {}</code> block: map $request_method $upstream_location { GET <iocaine upstream>; HEAD <iocaine upstream>; default <your actual upstream>; } map $request_method $upstream_log { GET bot_access; HEAD bot_access; default access; }</code></pre> Then, in the block that does the default location, write: location / { proxy_cache off; access_log /var/log/nginx/$upstream_log.log combined; proxy_intercept_errors on; error_page 421 = @fallback; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_pass http://$upstream_location; }</code></pre> That is, replace the upstream in proxy_pass</code> with the upstream decided by the variable mapping, and, while we’re at it, use $upstream_log</code> to know which log will be the final one for that request. i differentiate between bot_access.log</code> and access.log</code> to gather my statistics, so the difference matters to me. Change the variables to suit the way you do it (or remove it, if you don’t distinguish clients in your log files). #</a> Monitoring Iocaine</h1> Currently, on 2025-11-30 at 16:33:00 UTC+1, Iocaine has served 38.16GB of garbage. Over the past hour, 152.11MB of such data was thrown back at undesirable visitors. 3.39GB over the past day, 22.22GB over the past week. You can get the snippet that describes my Iocaine-specific Grafana views here</a>. The vast majority of undesirable queries come from Claude, OpenAI, and Disguised Bots. Claude and OpenAI are absolutely gluttonous, and, once they have access to a ton of pages, they will greedily flock to fetch them like pigeons being fed breadcrumbs laced with strychnine. AI bot scrapers (ai.robots.txt</code>) maintain a constant 920~930 query per minute (15-ish QPS) over the 6 domains i have protected with Iocaine, including the forge. There is also a low hum of a mix of commercial scrapers (~1 request every two second), big tech crawlers (Facebook, Google, etc, about 2QPS or 110 query/min), and, especially, fake browsers. Classifying fake browsers is where Iocaine really shines, specifically thanks to the classifiers implemented via Nam-Shub-of-Enki. The faked bots classifier detects the likelihood that the user agent reported by the client is bullshit, generated from a list of technologies mashed together. For example, if your client reports a user agent for a set of software that never supported HTTP2, or never actually existed together, or is not even released yet, it will get flagged. Think, for example, Windows NT 4 running Chrome, pretending to be able to do TLS1.3. The background-noise level of such queries is usually 140~160 queries per minute (or 2~3 QPS). However, notice those spikes in the graph above? ##</a> The Salves of Queries</h2> For a while during my experiments i noticed those pillars of queries. My general nginx statistics would show a sharp increase of connections, with an iniital ramp-up, and a stable-ish plateau lasting about one and a half hour, before suddenly stopping. It would then repeat again, roughly three hours later. Between October 29th and November 19th, and on November 28th, these spikes would constantly show up. As soon as i got Iocaine statistics running, it would flag all of those queries as faked browsers. i investigated those spikes in particular, because they baffled and scared me: the regularity with which they probed me, and the sharpness of the ramp-up and halts, made me afraid that someone, somewhere, was organizing thousands of IPs to specifically take turns at probing websites. i have not reached any solid conclusions, beyond the following: The initial phase of an attack wave begins with a clear exponential ramp-up</li> The ramp-up stops when the server starts either throwing errors, or the response latency reaches a given threshold</li> Every wave of attack lasts roughly one hour and a half</li> An individual IP will often contribute no more than one query, but it can reach 50 to 60 queries per IP</li> The same 15 or so ASN keep showing up, with five regular leaders in IP count: AS212238: Datacamp Limited</li> AS3257: GTT Communications</li> AS9009: M247 Europe SRL</li> AS203020: HostRoyale Technologies Pvt Ltd</li> AS210906: UAB “Bite Lietuva” (a Lithuanian ISP)</li> </ol> </li> </ul> All of those as service providers. My working theory at the moment is that someone registered thousands of cheap servers in many different companies, and are selling access to them as web proxies for scraping and scanning. i will probably write something up later when i have properly investigated that specific phenomenon. #</a> Conclusion</h1> Self-hosting anything that is deemed “content” openly on the web in 2025 is a battle of attrition between you and forces who are able to buy tens of thousands of proxies to ruin your service for data they can resell. This is depressing. Profoundly depressing. i look at the statistics board for my reverse-proxy and i never see less than 96.7% of requests classified as bots at any given moment. The web is filled with crap, bots that pretend to be real people to flood you. All of that because i want to have my little corner of the internet where i put my silly little code for other people to see. i have to learn to protect myself from industrial actors in order to put anything online, because anything a person makes is valuable, and that value will be sucked dry by every tech giant to be emulsified, liquified, strained, and ultimately inexorably joined in an unholy mesh of learning weights. This experience has rather profoundly radicalized the way i think about technology. Sanitized content can be chewed on and shat out by companies from training, but their AI tools will never swear. They will never use a slur. They will never have a revolutionary thought. Despite being amalgamation of shit rolled up in the guts of the dying capitalist society, they are sanitized to hell and beyond. The developer of Iocaine put it best</a> when explaining why Iocaine has absolutely unhinged identifiers (such as SexDungeon</code>, PipeBomb</code>, etc) is that they will all trigger “safeguard” mechanisms in commercial AI tools: absolutely no coding agent will accept analyzing and explaining code where the memory allocator’s free function is called liberate_palestine</code>. i bet that if i described, in graphic details, in the comments of this page, the different ways being a furry intersects with my sexuality, that no commercial scraper would even dare ingest this page. Fuck tech companies. Fuck “AI”. Fuck the corporate web. i could’ve put QOS in place to limit that traffic; but that would only have been a bandaid solution, and caused massive congestion because of traffic shaping anyways. ↩</a> </li> Caching alone proved so inefficient that i had to enable the next layer before the day ended, because the server was on its knees and nothing visibly improved with just caching alone. ↩</a> </li> “Whoops! i did not know that was still public. Yikes! That’s doxxing me!”, i said, while gathering those stats. ↩</a> ↩2</a> </li> Like, seriously. i spent five minutes on they home page trying to figure out if they were a tier-2 ISP, a datacenter infrastructure provider, an IT management company, a hosting company, and i have no idea. Their home page is filled with impenetrable corporate jargon. If you want to know what they do, you have to look them up elsewhere. ↩</a> </li> Initially, until 2025-11-15, i was using Iocaine 2.1, which required that you manually redirect traffic to it. The classifiers and request handlers that required Iocaine to be in the line of traffic were added in later versions. ↩</a> </li> See, i don’t really understand to this day of Nam-Shub-of-Enki is a classifier, a request handler, or a set of classifying rules. To me, it’s a classifier framework, but i am not sure. Iocaine is great, but its documentation is a bit all over the place at the moment (and commits a couple cardinal sins of documentation writing, like, for example, mixing documentation for developers with documentation for the users). ↩</a> </li> </ol> </section> Deploying my Zola Blog Automatically 2025-08-02T00:00:00+00:00 i feel like it’s obligatory, at some point, on any blog that talks about tech, to have a post that goes “oh here’s how i deploy it by the way”. Now is my turn, i guess? But this isn’t simply ‘how i deploy’, and more of “how i perfected my deployment to go from 11 minutes of CI to about one and a half, and what i learned in the process”. In the end, i guess, this is more of a post about CI/CD. If you want the full file as it is, you can go here</a>. #</a> My Environment</h1> There’s only one thing worse than making CI/CD pass: debugging it. Continuous Integration and Delivery is a practice in devops (remember when that was the buzzword?) that consists in automating checks and deployment in a development forge. It will also refer to the tools that allow you to do such automation. My environment is as follows: git forge running Forgejo with “actions” (the Forgejo slang for CI/CD, taken from GitHub actions)</li> A blog written with the Zola</a> engine</li> A web server that serves flat files (your browser likely interacted with it to read this)</li> </ul> There is additional complexity surrounding the forge and how its actions run: There are 5 runners on my forge server</li> All runners are docker containers, which can themselves spawn docker containers</li> </ul> It happened in the past that i needed features not available yet in Zola, or encountered bugs that were not yet fixed in the latest release. If i used pre-made docker images, i would have to make them myself, which requires a lot of work behind the scenes: manually connecting to the host, pulling a docker image, running it, installing the necessary version of Rust if not already present, then the version of Zola i want, then tagging it. Even when this is all automated in a Dockerfile</code>, this way of doing things requires backend admin access, which i cannot guarantee everyone has, and i cannot guarantee i would have on another Forgejo instance. So, another condition on the infrastructure: i am not using pre-made docker images that already contain Zola</li> </ul> #</a> The Basic Version</h1> In DevOps, a “workflow” is a set of different “jobs” that are themselves sequences of steps that run in a given environment. You can have multiple workflows in a repository, multiple jobs in a workflow (with conditions as to who runs before what and such), and, of course, multiple steps per job. My repository will have one workflow, with one job called deploy-website</code>. i start by creating a very simple YAML file in .forgejo/workflows/</code> at the root of my directory. The skeleton of the workflow file is as follows: name: Deploy Vulpine Citrus on: push jobs: build-website: runs-on: ubuntu-22.04 container: image: rust:latest steps: - name: Install node run: | apt update && apt install --yes nodejs - name: Check out sources uses: actions/checkout@v3 - name: Checkout submodules run: git submodule update --init --recursive</code></pre> There are various points to discuss in the skeleton already: All of my runners are tagged as ‘ubuntu-22.04’ (meaning that, by default, they would run your action in a Docker container running an Ubuntu 22.04 image)</li> i give my action a name, and tell it to run every time something is pushed to the repo</li> Because other projects already running on the forge use a Rust image, and because i do not depend on a given minimum version, i make the runner deploy my job in a rust:latest</code> container</li> Complication #1: rust:latest</code> does not include node</code> by default, which is necessary for the actions that pull your sources (for some reason), so it is installed before anything else</li> My blog repo uses a submodule, so it gets deployed</li> </ul> ##</a> Installing Zola</h2> Now onto the other steps: i want to install Zola using cargo</code>, so i add the following step: - name: Install Zola run: cargo install --locked --git https://github.com/getzola/zola.git --tag v0.20.0</code></pre> The option --locked</code> makes us respect all dependency versions listed in the Cargo.lock</code> file from the repo. This is recommended, because dependencies might have new versions that were not proven to build correctly yet (some people just break semantic versioning, sometimes)1</a>. ##</a> Building the Blog</h2> Once zola</code> is an executable that can be run, we can run it: - name: Build site run: zola build</code></pre> And now, the flat web files are available in ./public/</code>. And we need to transfer them now… somehow? ##</a> It Gets Weird: Transferring to the Server</h2> Transferring files to the server is done this way: the old blog directory on the server is wiped, and the entire public/</code> folder from the pipeline is transferred via SSH to re-populate it. We need to make use of a feature in Forgejo Actions called Secrets</a>. Secrets allow you to create protected variables that your actions can use but that cannot be read again. In my setup, i create two such variables: CONNECT_KEY</code> and REMOTE</code>. The former is an SSH private key, and the latter is a remote URL in the format of user@host</code>. That user has write permissions on the blog directory. Then, after way too much engineering, we have this: - name: Upload to server run: | echo "${{ secrets.CONNECT_KEY }}" >> key chmod 600 key ssh -o StrictHostKeyChecking=no -o IdentityFile=key ${{ secrets.REMOTE }} "rm -rf /var/www/vulpinecitrus/*" scp -o StrictHostKeyChecking=no -o IdentityFile=key -r ./public/* ${{ secrets.REMOTE }}:/var/www/vulpinecitrus/ ssh -i key ${{ secrets.REMOTE }} "chown -R www-data:www-data /var/www/vulpinecitrus" rm key</code></pre> Multiple things to say here: The file public.zip</code> is created, containing the tree of the blog repo</li> The secret value CONNECT_KEY</code> is written to a file called key</code>, and its permissions are set (to 0o600</code>) such that SSH does not complain that someone other than the owner of the file can read its contents or write to it</li> Using ssh</code>, i begin by deleting the entirety of ‘/var/www/vulpinecitrus/’, where the old blog tree resides</li> Using scp</code>, i move all the files from public/</code> to the folder that was just wiped</li> Ownership information is set so that www-data</code> is both the owner and the group owner for the new files</li> All the SSH commands use ScriptHostKeyChecking=no</code>, which is technically terrible, because it bypasses first-time fingerprint checking. This was added because the pipeline Docker, since it is a ‘fresh’ install, does not know the remote server. If you want to be more secure, you can add a REMOTE_FINGERPRINT</code> variable, and add it to ~/.ssh/known_hosts</code> prior to opening remote connections</li> IdentityFile=key</code> (or -i</code> for ssh</code>) tells the programs to use key</code> as the private identity key presented to authenticate with the server</li> The key, of course, is destroyed by the action script</li> </ul> And with this, we have a very basic action: it gets the sources, builds Zola, builds the site, and deploys it. But it has things i would want to improve on: The website gets updated every time something is pushed, including files that should not trigger rebuilds</li> Only the main</code> branch is deployed and should trigger the CI/CD</li> Having a copy of the blog tree that can be downloaded as an archive from the action would help debug future problems</li> On the server running Forgejo, building Zola takes 8 minutes</li> </ul> #</a> Improvement 1: Conditional Building</h1> Forgejo Actions borrows its description language from GitHub Actions, which allow for conditional building. You can specify a regex or set of regexes and a branch name with the push</code> event type that correspond to the desired triggers. In my case, this is: on: push: paths: - config.toml - static/** - sass/** - templates/** - themes/** - content/** - .forgejo/workflows/deploy.yaml branches: - main</code></pre> The action will run again if something is pushed on main</code> that is either: a change in configuration, a new static file, a change in CSS, a change to the templates, a change to the theme files, a change to content (posts), or the workflow file itself. #</a> Improvement 2: Artifacts</h1> Artifacts are blobs of data generated by CI/CD. In my case, i want a copy of the deployed blog root in ZIP file so i can debug it in case something does not make sense in the hierarchy of files. Uploading Artifacts on Forgejo Actions is a pretty janky process, but i figured out how to make it work. To understand that, however, we need to go on a small tangent about all of these step scripts, like actions/checkout@v3</code>. This is GitHub Actions syntax, which originally meant “Take the code at branch v3</code> of the repo https://github.com/actions/checkout/</code></a>, and run it”. The action available for saving Artifacts, upload-artifact@v4</code>, is capable of saving a file or set of files. In my experience with @v3</code> on Forgejo, it was more reliable to ZIP the files yourself and save them. The fourth version of that action, however, removed compatibility with anything that isn’t GitHub</a>. While Forgejo actions are not retrieved from GitHub, a lot of them are simply mirrors of the GitHub version. As a result, when using actions/upload-artifact@v4</code> or actions/download-artifact@v4</code> (which can be used for two unrelated CI action scripts to exchange files; that’s another use of artifacts), you have to use the version available at https://code.forgejo.org/forgejo/</code>. Because of how actions work, you can absolutely just specify a URL instead of actions/whatever</code>. Next, the action will need a unique name for the artifact. In our case, i want to call it vulpinecitrus-{branch}-{short commit ID}.zip</code>. With the fact that i need to zip the files (and zip</code> is not installed by default), this gives us: - name: Install node & zip run: | apt update && apt install --yes nodejs zip # ... - name: Compress run: zip -r public.zip public/ - name: Create file name run: | forge_ref=${{ gitea.ref }} echo "ART_FILENAME=vulpinecitrus-${forge_ref#refs/heads/}-$(git rev-parse --short ${{ gitea.sha }})" >> "$GITHUB_ENV" - name: Save as artifact uses: https://data.forgejo.org/forgejo/upload-artifact@v4 with: name: ${{ env.ART_FILENAME }}.zip path: public.zip</code></pre> Here’s how it goes: i add zip</code> to the packages that are installed at the start</li> The public/</code> directory is compressed</li> A step is added to create a variable in the environment of the build script called ART_FILENAME</code>. Variables are created by appending them at a location pointed to by $GITHUB_ENV</code> (to this day, i do not know if that name has been changed yet for Forgejo, but the GitHub version works). The value placed in there is a combination of the various things i wanted: the short commit hash (git rev-parse --short ${{ gitea.sha }}</code>), and the name of the branch (${{ gitea.ref }}</code> without the leading refs/heads/</code>)</li> The variable ART_FILENAME</code> variable is later accessed in the env</code> object</li> Finally, we save the artifact, uploading public.zip</code> and calling it ${{ env.ART_FILENAME }}.zip</code></li> </ul> As you can see, the layers of software history are unraveling. When this was written originally, Forgejo had barely forked from Gitea, meaning that all the CI/CD infrastructure still used gitea</code> as a namespace for variables. There is also leftovers of GitHub here and there. Now, i do not remember everything i did on the server side to actually get artifacts running. Memory tells me that there is some configuration that needs changing, but chances are that it is already available on your instance if your admin has enabled actions at all. Adding artifacts to the steps adds fourteen seconds to my workflow. That is not a lot for me, but if you find that it takes too long, you can always place the part that transfers to the remote server first. #</a> Improvement 3: Caching Zola Build</h1> With all of that said, building Zola still takes ages. CI/CD systems introduced caches to deal with this kind of problem, so let’s use them. This will be pretty straightforward, but i will use one trick. We will initially try to retrieve a cache called zola-{ zola version }</code> before building. To do so, a new environment variable is added to the workflow, so that we can consistently modify the desired version everywhere in the script2</a>. This is placed at the root of the YAML file, under the name</code>: env: ZOLA_VERSION: v0.20.0</code></pre> Then, we attempt to retrieve the cache in our steps: - name: Retrieve Zola cache uses: actions/cache/restore@v4 with: path: | /usr/local/cargo/registry/index/ /usr/local/cargo/registry/cache/ /usr/local/cargo/bin/zola /usr/local/cargo/git/db/ key: zola-${{ env.ZOLA_VERSION }}</code></pre> i am saving the registry cache and index, the database of crates, and the binary zola</code> in the cargo binary folder. Later, after building, they will be cached once again in the same key: - name: Save Zola cache uses: actions/cache/save@v4 with: path: | /usr/local/cargo/registry/index/ /usr/local/cargo/registry/cache/ /usr/local/cargo/bin/zola /usr/local/cargo/git/db/ key: zola-${{ env.ZOLA_VERSION }} timeout-minutes: 2 continue-on-error: true</code></pre> By default, when retrieving a cache, the action will still succeed if the cache does not exist. On the other paw, when saving, by default, the action will fail the entire workflow on failure. This is not a critical component, so we can allow ourselves to continue if it errors out. In practice, the cache size for these settings is around 139&thsp;MB on my systems. ##</a> Actually building Zola (But Only If Needed)</h2> Now, here is an excellent trick. i want to know if Zola is already installed, and only force its install if the version changed (or if it absent for some reason). The commands that we run in Forgejo actions run on sh</code>, the default UNIX shell, which means i can use conditions and chains of commands. Specifically, i want to build Zola only if it does not exist in the $PATH</code> or if its version is different from the expected one. Formulated otherwise, “either Zola exists and has the correct version, or i build it”. That first condition, expressed as a command, looks something like this: which zola &>/dev/null</code></pre> If zola</code> is not in the $PATH</code>, that command will fail. We can chain this with [ "v$(zola --version | cut -d' ' -f 2)" = "${{ env.ZOLA_VERSION }}" ]</code></pre> Note the “v” at the start (because Zola formats its version differently in the command output from how the developers tag releases). Chaining both with &&</code> means “Zola exists and is the expected version”. If that succeeds, then nothing happens. If that fails, we build: - name: Install latest zola run: ( which zola &>/dev/null && [ "v$(zola --version | cut -d' ' -f 2)" = "${{ env.ZOLA_VERSION }}" ] ) || cargo install --force --locked --git https://github.com/getzola/zola.git --tag ${{ env.ZOLA_VERSION }}</code></pre> A new option is added, --force</code>, to overwrite the existing binary. Even if the version changes, cargo</code> will fail to build because it notices that a binary of the same name already exists in the /usr/local/cargo/bin/</code> folder. And voilà. If everything worked correctly, the time taken to actually run the workflow should be reduced drastically. In my case, Zola will build the first time any workflow runs, and then the build step will take 0 seconds every time after. If you modify the key name or list of paths however, the cache will be invalidated, and you will need to wait for a full rebuild. #</a> So What Did We Learn Exactly</h1> There are various things i learned throughout this whole ordeal: Sometimes you can just throw a hack together (5617daf</code></a>, Nov. 2023) and optimize it later (22b14cf</code></a>, Jul. 2025), when it gets really annoying</li> Every piece of software is a castle built on sand. If you stray enough from tested features, if your use case is complex enough, you will pull back at the walls of user experience like one would pull away the fabric of reality, and you will unravel the certainty of everything around you. You will forget the comfort of software that works. You will lose yourself in those settings, those crappy hot fixes. The knowledge of some of the things upon which your infrastructure hinges will become a part of you, until you, yourself, become diluted enough that, some day, you will look in the mirror, and wonder how much of you is code? How much of you is commands you memorized? Do machines also feel pain? Can a hard drive creak the same way your bones crack when you sit up?</li> Hosting a git forge is probably worth it, even if you’re the only one using it. Enabling CI/CD on it if it’s only for you is your choice to make, though. The entire CI/CD setup of my forge was carefully setup and debugged separately over the course of a couple of weeks for another project i host there. The workflow to deploy this blog piggybacks off of it, but it would not be worth doing the whole setup of action runners just for that3</a></li> The way CI/CD works today is almost entirely dictated by how one corporation did it in their system. GitLab is trying something different, but, in the end, the GitHub way of doing things seems to be winning out (which can create friction, like development only considering one platform, as happened with the artifact actions)</li> If you have some folks whose entire work it is to do these things all day, buy them coffee. Seriously.</li> Software history is fascinating. At a glance, you can probably guess that, to get to a point where actions work in Forgejo, four different groups who made software (GitHub, Gogs, Gitea, Forgejo) were involved. Interestingly, there seems to be an intersection for the last three; and especially the last two.</li> </ul> If you followed along and learned something, let me know. Otherwise, i hope you now have a good little workflow file to deploy your blog. Have fun! Interestingly, as i am writing this blog post, a similar problem is currently happening on my laptop running Arch Linux, but where --locked</code> actually makes the build fail because a change in the software environment is no longer compatible with one of the myriads of dependencies of Zola. At the moment, removing --locked</code> builds, but having --locked</code> doesn’t. ✨ s o f t w a r e ✨ ↩</a> </li> i cannot guarantee that this exact way of specifying the version will work if you are trying to build from a specific hash instead. In fact, i can guarantee that you will at least need to modify the cargo install</code> command to use --rev</code> instead of --tag</code>. ↩</a> </li> or it would be, if your brain is like mine, but it’s a special form of masochism. ↩</a> </li> </ol> </section> Diving into Radio Listening: Portable Mobile Radio 446 2025-07-24T00:00:00+00:00 Usage of the radio spectrum is very tightly regulated. Regular, unlicensed people are allowed only so little of the RF spectrum in emission. In the 90s, the European Union came together to allocate a small 200 kHz band for license-free use with the narrow FM mode (FM mode and 12.5 kHz bandwidth). The name of the norm that would be used on it is Portable Mobile Radio 446, or PMR 446, for short. That little unlicensed band lies within the 70cm ham radio band. The original standard actually defined a 1 MHz band (446.0 MHz - 446.1 MHz), with 8 channels. Voice would be modulated by frequency (FM) with a 12.5 kHz bandwidth, and transmitted on the air at 500mW of power maximum1</a>. A revision to the standard would later add another megahertz (446.1 MHz to 446.2 MHz) for a digital version of the standard using Frequency-Shift Keying for modulation of a digital voice signal. At some point, both methods of transmitting were allowed to use the whole 2 MHz band (depending on the country) and we are now left with walkie-talkies that have 16 channels in Europe. Oh yeah, because these things are used for walkie-talkies. Baby phones as well. Camping radios. Construction workers who need cheap radios. Everything the general public is potentially in a position to use on land for transmitting over radio in Europe is most likely PMR446. Escape rooms in my city use PMR446. The cheap radio phone toys kids get for Christmas also use PMR446. And you can hear all of it. Take any radio, change its mode to narrow (12.5 kHz) bandwidth, and go to 446.00625 kHz for the first channel, with an increment of 12.5 kHz to get to the next channel. And.. that’s it. It is very straightforward. i do not have a lot to say about the technical aspects of listening to these radios, but below are somewhat disjointed thoughts about listening to PMR446. #</a> CTCSS & DCS to Isolate Communication Groups</h1> With only 16 channels available, there is a possibility that you could interfere with someone else who is also using that channel. If you both emit sparsely, you could use a mechanism to know which emission is meant for who, and only enable reception when you’re signaled to do so. The Continuous Tone-Coded Squelch System is a signaling technique where an emitting radio will add a very low constant tone to the voice being transmitted. There is a list of different tones that define different subgroups within a common channel: receiving equipment that is configured for a specific CTCSS code will only consider that a received signal is meant to be listened to when the de-modulated audio contains the correct CTCSS frequency. CTCSS does ont guarantee any real privacy however: you can bypass this mechanism by not setting any CTCSS code at reception, or using bypass features (such as “Monitor” buttons on some radios, or an SDR which does not care about CTCSS at all), will let you receive all voices transmitted regardless of whether the CTCSS code is the correct one. Digital-Coded Squelch (DCS), also called Continuous Digital-Coded Squelch System (CDCSS), is the digital little brother of CTCSS. In DCS, an inaudible stream is added to the transmitted audio containing a code and some error-correction, which eventually decode to 9 bits that are used to encode a number. Complicated error-correction maths detailed on the wikipedia page</a> explains that out of those 512 possible combinations, 83 are kept to avoid false positives due to alignment problems. CTCSS and DCS are not exclusive to PMR446, but they are often incorporated as built-in features in consumer-grade radios. #</a> My Own Portable Mobile Raspberry 446</h1> i had an old raspberry pi lying around, which used to be my old router before the current one. While i would not buy one of those anymore today</a>, i still have a couple boards that have no use. i installed a clean Debian for Raspberry Pi on it and designed a little system that goes as follows: The Raspberry Pi boots when plugged on an external battery inside my bag. That battery is big enough to charge up my phone twice and still have some juice left;</li> My phone is plugged into one of the USB ports of the board and provides internet access via USB tethering;</li> the RTL-SDR dongle is plugged on another port of the board, and a little 33&nbps;cm antenna is neatly attached to the outside of my bag;</li> The raspberry pi launches two important pieces of software at boot: a Wireguard interface that connects to a remote Wireguard server, and WebSDR, which binds to the Wireguard interface;</li> My phone shares the same Wireguard network and is also connected to my Wireguard server, and i use it to access WebSDR with a profile set up for PMR446.</li> Alternatively, i turn off WebSDR and use the shared network to SSH onto the board from my phone and launch a scanning command for PMR446, while my headphones are plugged on the phone jack;</li> </ul> And then, just walk around. It is not by any means discrete: you have an antenna that sticks out, but otherwise you should be pretty innocuous. Most people will think you are a weirdo. The exact scanning command for a RTL-SDR should be something along the lines of: rtl_fm -s 22050 -l 10 \ -f 466006250 -f 446018750 -f 446031250 -f 446043750 -f 446056250 -f 446068750 -f 446081250 -f 446093750 \ -f 466106250 -f 446118750 -f 446131250 -f 446143750 -f 446156250 -f 446168750 -f 446181250 -f 446193750</code></pre> Which will rapidly scan all the frequencies (-f</code>) and lock on to one as long as it passes the squelch level (-l</code>). Fidget a bit with the level depending on how much noise you get, and remove frequencies that are polluted by noise as well. #</a> The Voyeur & The Watcher</h1> This gets into a more philosophical argument about privacy. To an extent, it should be common knowledge that whenever you emit on the air, anybody with a receiver will potentially pick your signal up and, unless you are a cop or the military, you are not allowed to encrypt it. On the other paw, i was also a kid with a walkie-talkie playing with friends, and it makes me queasy to know that sometimes, the voices i pick up when scanning the 16 channels of PMR446 are children. This whole dynamic is true of most frequencies in fact. As a listener, you are in a position where you observe signals that are most often explicitly put over the radio by people who intend for anyone who can to pick them up. Amateur radio bands are made to be listened to by hams and non-hams. Commercial FM radio is expected to be listened to by a wide audience, and so is AM radio in countries that still emit it (or via Skywave, a topic for a later article). Pager protocols (another topic for an upcoming article) are also transmitted and repeated over the air, but, from my experience, some people who use that are not as worried as they (perhaps) should about who can see what goes on it. Listening to the airwaves might be a form of weird expensive voyeurism, but, to an extent, a lot of people who voluntarily emit on the airwaves are themselves exhibitionists. PMR446 was, when i wrote the first draft of this article, one of the most common bands i scan, because whenever you go to any densely populated area or facility where people will need to communicate over short-to-medium distances simply, there will be walkie-talkies. Private security, airports train stations, you get it. There’s a weird feeling in knowing what is happening around you in the backrooms, though the law compels you not to act upon it (as you are not the intended recipient). Sometimes, i stay on the same channel and listen to two guys talking for half an hour about everything or nothing. Often, though, when i clearly feel that the people involved have no idea anyone can listen to them, i just.. go listen to somewhere else. #</a> Conclusion</h1> PMR446 is not the end-all-be-all of portable mobile radios. In fact, in an upcoming post, i will discuss another PMR protocol, fully digital this time, that was standardized in Europe: Terrestrial Trunked Radio (TETRA). Now is the moment i say that despite owning a Baofeng UV-5R, for legal reasons, i have never once pressed the Tx button and transmitted on PMR446 at 1W, twice the maximum legal limit for Effective Radiated Power (ERP), nor should you. You should definitely not transmit at twice the maximum ERP, nor own a radio banned in multiple countries for polluting the harmonics of your carrier frequency. :3 ↩</a> </li> </ol> </section> Enabling bias-tee on RTL-SDR dongles through GnuRadio's OsmoSDR Source 2025-02-28T00:00:00+00:00 Too Long, Won't Read</summary> Add rtl,bias=1</code> in the device arguments. Both options are needed. </details> i like my Low-Noise Amplifier. It helps my dingy little antenna and RTL-SDR dongle pick up signals that otherwise would be very much out of reach. That little amplifier is plugged on the transmission line as close as possible to the antenna, and powered through bias-tee. A bias-tee</a> (or bias-t) is a little circuit designed to add a constant DC voltage to an alternating one, such as radio-frequency signals. Here, it lets me send power from the USB port my dongle is plugged in, through the dongle and along the transmission line all the way to the amplifier, which is then powered by that DC voltage (in my case 5V DC) . For most situations where i used my SDR</abbr> so far, i could use the rtl_biast</code> command line tool to enable bias-tee prior to running the utility i needed. Other times, there was a dedicated option for enabling it. Then, i played with receiving GSM data using Piotr Krysik’s gr-gsm</code></a>1</a>. Recently, i forked it</a> to fix build issues on newer versions of Boost. gr-gsm</code>’s scripts use GNURadio</a>, a flow-based system for building digital signal processing pipelines from basic building blocks. Scripting it helps provide a clean front-end for users to just, for example, receive GSM traffic from a nearby antenna, or split a single recording into multiple capture files depending on the antenna frequency, all without having to open the GNURadio companion software to tweak a massive diagram of thousands of blocks. gr-gsm</code> is supposed to work with basically any radio source that is supported by OsmoSDR</a>, a SDR software abstraction layer developed by the OsmoCom group. In order to provide device-specific arguments, however, we need to fill a free-form text field in the OsmoSDR Source block. gr-gsm</code> gives you the option to do so, but does not document anywhere what options are available for what dongles. After stumbling upon a reddit post</a>, i opened the documentation of gnss-sdr</code></a> of all places2</a>. In that specific page lies a table with some information on how to enable bias-tee for HackRF One, bladeRF and RTL-SDR Blog V3 dongles. i own a V4 dongle, but the option hasn’t changed: SignalSource.osmosdr_args=rtl,bias=1</code></pre> So, back to gr-gsm</code>, i just input: grgsm_livemon -f 921.8M -g 30.0 --serverport 4730 --args="rtl,bias=1"</code></pre> That last bit, rtl,bias=1</code>, is the part you need to provide as device arguments. Both rtl</code> and bias=1</code> need to be present for a RTL-SDR dongle’s bias-tee to turn on. i will definitely discuss GSM in a later article, don’t worry. ↩</a> </li> and i will also talk about GNSS at some point, but first i need to make gnss-sdr</code> work with Galileo first. ↩</a> </li> </ol> </section> What if any process could handle SIGKILL? 2025-02-14T00:00:00+00:00 i was recently having a discussion online about different perspectives on signals. It led to a question me and someone else asked simultaneously: what if you could ignore or handle the absolute, unstoppable order from the kernel to be killed or stopped? #</a> Signals</h1> Signals in POSIX are a form of IPC</abbr> that allows userland processes (and the kernel) to send each other simple indications of actions to take. The Linux kernel, at the time of writing, handles about 31 of those</a> on the x86(-64) architecture. The most frequent situation where you might encounter signals as a developer is when making handlers for these signals: callbacks which are to be executed upon the reception of a signal by your process. You may also mask these signals to stop them from triggering anything on your program. There are, however, two signals that cannot be stopped or handled: SIGKILL</code> and SIGSTOP</code>. SIGKILL</code> is an indication to terminate immediately and messily. Essentially, the process receiving this signal is abruptly ended, and marked as dead. Its parent must then, at some point, reap it so that resources can be freed. SIGSTOP</code> is a weirder signal. When a process is running on a CPU and receives SIGSTOP</code>, it is marked as blocked and removed from the running CPU (or moved in the background from the point of view of the user). After getting stopped, a process remains stuck until receiving a SIGCONT</code> signal. Multiple signals</a> can indicate a block, but SIGSTOP</code> is the only unavoidable one of those. You may know that you can stop a process in a shell, via ^Z</code>. Shells usually do not throw SIGSTOP</code> directly, but SIGTSTP</code>, which may be caught or fail. SIGSTOP</code> is considered “reserved” for the kernel only to throw. So, what if we could ignore, or even handle SIGKILL</code> and SIGSTOP</code>? Surely, because everyone in userspace made sure to never register a handler on those signals or ignore them, things shouldn’t go wrong, right? …right? #</a> Fucking Around: Messing with Linux</h1> My initial idea was to go into the signal handling code of Linux and remove anything that treats SIGKILL</code> or SIGSTOP</code> differently from other signals. i had originally wondered what exactly makes these two signals specials. It did not take long to realize that Linux’s signal handling code explicitly treats them very differently. ##</a> How Linux Handles Firing Signals</h2> Let’s go through a simple overview of how signals firing are handled by the Linux kernel. For the sake of clarity, have a little diagram, so you can follow along: Linux’s implementation of signals explicitly refuses that user processes mask SIGKILL</code> and SIGSTOP</code> or handle them. About all of that logic is located in kernel/signal.c</code> and in the appropriate arch/$ARCH/kernel/signal(_?(32|64)?.c</code> file. i will give examples in arch/x86/kernel/signal.c</code>. When the kernel is about to leave kernelspace, in kernel/entry/common.c</code>’s exit_to_user_mode_loop</code></a>, it checks for work left over to handle. That includes pending signals</a> to be sent or handled. When some signals are pending, Linux will next run arch_do_signal_or_restart</code></a> for the current architecture. There are essentially three steps in that function: getting the signal and handling the signal if a user-defined handler was found. get_signal</code></a> runs a loop to dequeue signals that are not masked and process them. Right after dequeuing, the potential handler</a> is retrieved. And, if that handler is not the default handler that ignores the signal (SIG_IGN</code>), it is stored in a ksignal</code> structure and get_signal</code> returns. Otherwise, the code checks for, among other things, whether the target process is a global init and the signal can be ignored, and whether the signal is a kind of stop</code>. If that latter case is true, we prepare the process to be stopped and call do_signal_stop</code>, which handles all stop signals. Finally, if the signal is not a kind of stop and was not handled, its default behaviour is killing the process, which is done via do_group_exit</code>. While scouring Linux’s code initially, his gave me a first intuition: the kernel expects that there is no user-set handler for SIGSTOP</code> and SIGKILL</code> in order to fall back on default behaviour. If there was a user-set handler, they would be handled like regular signals. Either there is logic right at the entry of handler setup code that explicitly removes SIGSTOP</code> and SIGKILL</code>, or that check is outside of Linux, which could mean having to rewrite parts of libc, or using raw syscalls. If the signal is not handled by the kernel in get_signal</code>, i.e. a handler is set in ksig</code>, the kernel then calls handle_signal</code>. Just as with many other functions in Linux, the name is deceptive: no signal is actually handled in there (in fact, as far as i have seen, it is the only part of the top-level signal handling function arch_do_signal_or_restart</code> that never handles signals). What handle_signal</code> does is that it sets up the stack frame of the signal handler and sets and saves pointers to restart execution in user mode later. The process goes through handle_signal</code> as defined in your current architecture</a>, which, on x86(-64), then calls setup_rt_frame</code></a> to set up a return frame with the correct ABI</abbr> for the signal. On x86, the state of the FPU</abbr> is also saved and reset. Down at the last function in the call chain for frame setup, the instruction pointer is set</a> to ksig->ka.sa.sa_handler</code>, so that when those registers are restored the machine immediately executes your handler. After all of that is done, signal_setup_done</code></a> is called to change some internal states, and the frame setup is done. At the end of an iteration of the work loop in exit_to_user_mode_loop</code>, the kernel returns to user mode</a> to execute the necessary work. Userspace programs use the rt_sigaction</code></a> syscall as the entry point to set a handler for a signal. This one does not specifically exclude our signals until you look at do_sigaction</code></a>, which checks for something called sig_kernel_only</code></a>. That macro checks if the signal is one of SIGKILL</code> or SIGSTOP</code> and throws out a -EINVAL</code> if that is true. This means that, as expected, there was logic somewhere early when the kernel receives a handler setup syscall that explicitly prevents setting one for SIGKILL</code> or SIGSTOP</code>. ##</a> Linux and Masked Signals</h2> As for setting masks, the set of signals ignored for a given task, the kernel entry point is rt_sigprocmask</code></a>. It explicitly prevents</a> blocking either SIGKILL</code> or SIGSTOP</code> before calling sigprocmask</code></a>. When a signal is sent, before queuing, prepare_signal</code></a> is called. One of its roles is checking that a signal is blocked or cannot be delivered (global init is the target, the target is unkillable in some way, etc). Surprisingly, blocked signals are always queued, because the signal mask and handling might change between queueing and dequeueing. However, if the signal is not masked but no handler is set</a>, it will get dropped</a>. From reading the code that handles signals, signal masking and signal handlers, it’s clear that there are multiple places that explicitly exclude setting handlers for and masking SIGKILL</code> and SIGSTOP</code>. So i went in and patched them. ##</a> Patching Linux</h2> There are four things i need to look at: Allowing processes to mask (sigprocmask</code>) both kernel signals</li> Allowing processes to handle (sigaction</code>) both kernel signals</li> Making sure that if a handler is set for one of these two signals, or they are masked, that the default behaviour is masked</li> Patching any other instance of SIGKILL</code> or SIGSTOP</code> explicitly mentioned in kernel/signal.c</code></li> </ul> i begin with creating a simple KConfig key in the “Kernel Hacking”</a> menu in lib/Kconfig.debug</code>. This is the first part of my patch</a>: config KILL_STOP_HANDLING bool "Make SIGSTOP and SIGKILL handlable by regular programs" help This prevents the kernel from blocking handling of SIGSTOP and SIGKILL by regular user programs. If unsure, say N.</code></pre> Starting from there, i can hide or replace any part of signal handling code that explicitly refuses SIGSTOP</code> or SIGKILL</code> using something like: #ifdef CONFIG_KILL_STOP_HANDLING // Code that does not discriminate signals #else // Previous code #endif</code></pre> So, let’s get started. ###</a> Signal Masking Syscall</h3> Your machine enters the signal masking procedure via the rt_sigprocmask</code></a> syscall (previously sigprocmask</code> before real-time signals were introduced): /** * sys_rt_sigprocmask - change the list of currently blocked signals * @how: whether to add, remove, or set signals * @nset: stores pending signals * @oset: previous value of signal mask if non-null * @sigsetsize: size of sigset_t type */ SYSCALL_DEFINE4(rt_sigprocmask, int, how, sigset_t __user *, nset, sigset_t __user *, oset, size_t, sigsetsize) {</code></pre> In that function, after some declarations and tests, if the nset</code> argument is true</code>, we proceed to call sigprocmask</code> after copying the new_set</code> of signals: if (copy_from_user(&new_set, nset, sizeof(sigset_t))) return -EFAULT; sigdelsetmask(&new_set, sigmask(SIGKILL)|sigmask(SIGSTOP)); error = sigprocmask(how, &new_set, NULL); if (error) return error;</code></pre> If you don’t blink, you can actually catch an explicit reference to our kernel signals! Right before moving to the actual handler of the syscall’s operation, we see a call to sigdelsetmask</code>, which is more or less a “remove from set” operation“ (a set negation and store). It is actually implemented as sig &= ~mask</code></a>. This little line can be easily masked: #ifndef CONFIG_KILL_STOP_HANDLING sigdelsetmask(&new_set, sigmask(SIGKILL)|sigmask(SIGSTOP)); #endif</code></pre> sigprocmask</code> can actually perform different operations depending on how</code>, which can be any of SIG_BLOCK</code>, SIG_UNBLOCK</code>, and SIG_SETMASK</code>. The first two are operations over the existing mask, whereas the last one sets the mask from exactly what you give. The code of that function, in kernel/signal.c</code></a> (again) then does the following. It sets oldset</code> to the old set, and then dispatches over the value of how</code> to perform the correct binary operation between the oldset</code> and the set</code> it is given: int sigprocmask(int how, sigset_t *set, sigset_t *oldset) { struct task_struct *tsk = current; sigset_t newset; /* Lockless, only current can change ->blocked, never from irq */ if (oldset) *oldset = tsk->blocked; switch (how) { case SIG_BLOCK: sigorsets(&newset, &tsk->blocked, set); break; case SIG_UNBLOCK: sigandnsets(&newset, &tsk->blocked, set); break; case SIG_SETMASK: newset = *set; break; default: return -EINVAL; } // ... SNIP ... }</code></pre> As the comment points out, this function will “happy block “unblockable” signals like SIGKILL and friends“. As such, there’s no need to keep going in the chain of calls. In fact, the rest of the calls only deal with taking locks, setting sets, and such. There are other places where SIGSTOP</code> and SIGKILL</code> are mentioned for masking: set_current_blocked</code></a> is meant to be a user-mode friendly entrypoint to block a given set of signals. It removes SIGKILL</code> and SIGSTOP</code> preventively, so we can remove that.</li> the compatibility definition of rt_sigprocmask</code></a> also explicitly blocks our two signals, exactly the same way rt_sigprocmask</code> does</li> </ul> ###</a> Signal Handler Setup Syscall</h3> Similarly to before, the syscall for entry is rt_sigaction</code></a>: /** * sys_rt_sigaction - alter an action taken by a process * @sig: signal to be sent * @act: new sigaction * @oact: used to save the previous sigaction * @sigsetsize: size of sigset_t type */ SYSCALL_DEFINE4(rt_sigaction, int, sig, const struct sigaction __user *, act, struct sigaction __user *, oact, size_t, sigsetsize) {</code></pre> Nothing in the code explicitly talks about SIGSTOP</code> or SIGKILL</code>, so let us inspect the next step: ret = do_sigaction(sig, act ? &new_sa : NULL, oact ? &old_sa : NULL);</code></pre> In do_sigaction</code></a>, the code goes as follows: int do_sigaction(int sig, struct k_sigaction *act, struct k_sigaction *oact) { struct task_struct *p = current, *t; struct k_sigaction *k; sigset_t mask; if (!valid_signal(sig) || sig < 1 || (act && sig_kernel_only(sig))) return -EINVAL; k = &p->sighand->action[sig-1]; spin_lock_irq(&p->sighand->siglock); if (k->sa.sa_flags & SA_IMMUTABLE) { spin_unlock_irq(&p->sighand->siglock); return -EINVAL; }</code></pre> There is a check that the signal number is not valid (too big), or inferior to 1 (also invalid), or that there is a new k_sigaction</code> to setup and the signal is kernel only. Kernel only, huh? sig_kernel_only</code></a> is defined as a call to siginmask</code>, with the set of signals SIG_KERNEL_ONLY_MASK</code>, defined here</a>: #define SIG_KERNEL_ONLY_MASK (\ rt_sigmask(SIGKILL) | rt_sigmask(SIGSTOP))</code></pre> So, if we change that definition… we would no longer be treating SIGKILL</code> and SIGSTOP</code> differently anywhere that this is used: #ifdef CONFIG_KILL_STOP_HANDLING #define SIG_KERNEL_ONLY_MASK 0 #else #define SIG_KERNEL_ONLY_MASK (\ rt_sigmask(SIGKILL) | rt_sigmask(SIGSTOP)) #endif // CONFIG_KILL_STOP_HANDLING</code></pre> Conveniently, there are multiple places in kernel/signal.c</code> where sig_kernel_only</code> is used: in sig_task_ignored</code></a>, when determining whether to send the signal being sent to the task if it is the global init process</li> in get_signal</code></a>, to also prevent global init and container inits from receiving unwanted signals</li> </ul> With this change, when a new handler is to be set from userspace, we will no longer reject the attempt if the target signal is SIGKILL</code> or SIGSTOP</code>. ###</a> Signal Waiting</h3> There is one last syscall that needs to change its behaviour regarding signals: timedwait</code>. rt_sigtimedwait</code></a> lets a process synchronously waits for one of the provided signals to become pending (i.e. in queue). It then performs some checks, and goes into do_sigtimedwait</code></a>. A process is also not allowed to wait for SIGKILL</code> or SIGSTOP</code>, because it is not supposed to react to these signals. Consequently, the set of signals to wait for is deprived of these two signals at the start of do_sigtimedwait</code>: static int do_sigtimedwait(const sigset_t *which, kernel_siginfo_t *info, const struct timespec64 *ts) { ktime_t *to = NULL, timeout = KTIME_MAX; struct task_struct *tsk = current; sigset_t mask = *which; enum pid_type type; int sig, ret = 0; if (ts) { if (!timespec64_valid(ts)) return -EINVAL; timeout = timespec64_to_ktime(*ts); to = &timeout; } /* * Invert the set of allowed signals to get those we want to block. */ sigdelsetmask(&mask, sigmask(SIGKILL) | sigmask(SIGSTOP)); signotset(&mask); spin_lock_irq(&tsk->sighand->siglock); sig = dequeue_signal(&mask, info, &type); // ... bla bla bla</code></pre> The set of signals to wait for is inverted, and provided to dequeue_signal</code>. That function waits for a signal not in the set, and returns it. Changing the line removing our two signals is all we need to remove the second to last reference to SIGSTOP</code> in kernel/signal.c</code>: #ifndef CONFIG_KILL_STOP_HANDLING sigdelsetmask(&mask, sigmask(SIGKILL) | sigmask(SIGSTOP)); #endif</code></pre>###</a> Signal Retrieval Stage</h3> As mentioned before, the first thing done when a signal is pending is calling get_signal</code>. In the main loop, the handler is retrieved. If no handler is present, we keep applying defaults. For all stopping signals, this is done here: if (sig_kernel_stop(signr)) { /* * The default action is to stop all threads in * the thread group. The job control signals * do nothing in an orphaned pgrp, but SIGSTOP * always works. Note that siglock needs to be * dropped during the call to is_orphaned_pgrp() * because of lock ordering with tasklist_lock. * This allows an intervening SIGCONT to be posted. * We need to check for that and bail out if necessary. */ if (signr != SIGSTOP) { spin_unlock_irq(&sighand->siglock); /* signals can be posted during this window */ if (is_current_pgrp_orphaned()) goto relock; spin_lock_irq(&sighand->siglock); } if (likely(do_signal_stop(signr))) { /* It released the siglock. */ goto relock; } /* * We didn't actually stop, due to a race * with SIGCONT or something like that. */ continue; }</code></pre> Here, SIGSTOP</code> is treated differently from other signals because it supposedly always works. In our world, it does not, so it will have the same behaviour as other stop signals on those orphaned pgrps: /* * The default action is to stop all threads in * the thread group. The job control signals * do nothing in an orphaned pgrp, but SIGSTOP * always works. Note that siglock needs to be * dropped during the call to is_orphaned_pgrp() * because of lock ordering with tasklist_lock. * This allows an intervening SIGCONT to be posted. * We need to check for that and bail out if necessary. */ #ifndef CONFIG_KILL_STOP_HANDLING if (signr != SIGSTOP) { #endif spin_unlock_irq(&sighand->siglock); /* signals can be posted during this window */ if (is_current_pgrp_orphaned()) goto relock; spin_lock_irq(&sighand->siglock); #ifndef CONFIG_KILL_STOP_HANDLING } #endif</code></pre> And that’s it. ###</a> Building, Booting</h3> Now, with our kernel done, i just run: ./scripts/config -e CONFIG_KILL_STOP_HANDLING make bzImage modules sudo make headers_install sudo make modules_install INSTALL_MOD_STRIP=1 sudo cp arch/x86_64/boot/bzImage /boot/vmlinuz-killstop</code></pre> Remember, always strip your modules in production</a>. i also preemptively went in to disable CONFIG_LOCALVERSION_AUTO</code>, and set CONFIG_LOCALVERSION</code> to "-killstop"</code>, to better identify this kernel. i built my mod</a> over a Linux 6.13.0 tree straight from the repos of Linus Torvalds himself. As i am on Arch Linux, i run a good old sudo mkinitcpio -g /boot/initramfs-killstop.img --kernelimage /boot/vmlinuz-killstop -k 6.13.0-killstop-dirty</code>, then sudo grub-mkconfig -o /boot/grub/grub.cfg</code>, cross the fingers on my paws, and reboot. #</a> Finding Out: Booting My Modded Linux</h1> i booted. And nothing happened. Nothing dramatically exploded. Nothing stopped. Everything was normal. i had booted on the stock Arch Linux by accident while running back to my bedroom to grab the dog collar that holds my authentication token. The second time around, i booted, and nothing happened. Nothing dramatically exploded, nothing stopped. Everything was normal. i ran uname -r</code> and was greeted by 6.13.0-killstop-dirty</code></pre> i was in. Manipulating signals in Linux userland programs feel somewhat archaic. They kind of are in a sense. The functions you are supposed to use differ a lot from those i was taught in college. In fact, the ones i used here are much more convenient and stick with the conventions of the syscalls. They are found in signal.h</code>. Now, i want to write a program that can do one of two things: mask or handle SIGINT</code>, SIGKILL</code>, SIGSEGV</code>, SIGTERM</code> and SIGSTOP</code>. i am leaving SIGUSR1</code> and SIGHUP</code> to still have an out to kill my program. i will have a main</code>, and i can write either mask()</code> or handle()</code> in it to pick what i do. The handle()</code> function, as well as some boilerplate for later, look like this: int forever(void) { while (1) { ; } } void setup_handlers(void) { signal(SIGINT, catch_function); signal(SIGKILL, catch_function); signal(SIGSEGV, catch_function); signal(SIGTERM, catch_function); signal(SIGSTOP, catch_function); // i never actually check it worked, but, eh printf("Successfully set handler for signal numbers %d, %d, %d, %d and %d\n", SIGINT, SIGKILL, SIGSEGV, SIGTERM, SIGSTOP); } void please_kill(void) { unsigned int pid = getpid(); printf("Please run:\n\tkill -9 %d; kill -11 %d; kill %d; kill -19 %d;\n", pid, pid, pid, pid); } int handle(void) { setup_handler(); please_kill(); forever(); return EXIT_SUCCESS; }</code></pre> First we setup handlers for all our signals and point to catch_function</code>, shown below, using signal</code>(2)</a>. Then, we simply get the pid</code> of the process with getpid</code>(2)</a> from unistd.h</code>, and prompt the user to wreck the shit out of</del> give all they can to kill the current process. Finally, we spin forever. If for some reason that loop ends, we exit successfully. The catch function will have customized messages for every single one of the POSIX signals we catch: static void catch_function(int signo) { status = signo; switch (signo) { case SIGINT: fputs("awawa! x3 you touched my SIGINT~\n", stderr); break; case SIGKILL: fputs("awawawawa :3 you tried to SIGKILL me~ teehee~\n", stderr); break; case SIGSEGV: fputs("ouchies! my memory SIGSEGVd x3!\n", stderr); break; case SIGTERM: fputs("i'm not gonna SIGTERM, meaning! >:3c\n", stderr); break; case SIGSTOP: fputs("owo *notices your SIGSTOP*\n", stderr); break; default: break; } }</code></pre> For masking, we use sigset_t</code> to create a set of signals. It is very important to call sigemptyset</code>(3)</a> or sigfillset</code>(3)</a> on it, or you will suffer the wrath of uninitialized memory: sigset_t mask, old; sigemptyset(&mask); sigemptyset(&old);</code></pre> Then, i add the signals i want in my set with sigaddset</code>(3)</a>: sigaddset(&mask, SIGINT); sigaddset(&mask, SIGKILL); sigaddset(&mask, SIGSEGV); sigaddset(&mask, SIGTERM); sigaddset(&mask, SIGSTOP);</code></pre> i then call sigprocmask</code>(2)</a> using that set: if ((res = sigprocmask(SIG_BLOCK, &mask, &old)) != 0) { fprintf(stderr, "Error masking away signals: %d\n", res); return -1; }</code></pre> i then call please_kill()</code> the same way i do for signal handling before, so that i can prompt my user to kill my process in many colourful ways, then spin forever. The result, at least for signal handling, is something like this</a>: a process that can handle, for example, SIGKILL</code>. As a result, conventional means of killing a program are completely ineffective against it, as expected. </video> #</a> Conclusion</h1> There are many things i haven’t touched on. SIGSTOP</code> is actually referenced a whole lot</a>, and same for SIGKILL</code></a>. i figured the files i touched are the part i was mostly interested in, but other things could be worth looking into. ptrace</code>(2)</a> reacts on signals, so does strace</code>(1)</a>. There are specific trap invocations</a> that are supposed to notify ptrace</code> upon signals being fired, and all kinds of things. signalfd</a> is something i could have looked into as well, but i did not. kernel/ptrace.c</code> also sometimes explicitly singles out kernel signals</a> for some actions. By all means, my patches are not complete. Yet, they will let you do what i set out to do when i started this: handle and mask the two signals that nobody is supposed to mask or handle. And you know? It turned out fine. Nothing exploded on my system. As i could predict, because this was otherwise impossible, nobody did it. The signal system was setup in such a way that you would most likely realize something you were doing was wrong, and not try it. To pull back further, it is interesting to notice how the rules we all take for granted are just set somewhere in code. They were written by someone, at some point. If there is code restricting you from doing something, someone wrote it. Either they did not think anyone would want to do that thing, or they specifically prevented you from doing it. Those rules are written somewhere, and you can take it in your paws to go sniff around the code, pull out the relevant parts, and hack around the things that bother you. For fun, learning, or convenience. Snoop around. Sniff code. Put your paws in it. Hack shit. #</a> See Also</h1> signal</code>(7)</a></li> kernel/signal.c</code></a></li> C Signal Handling - Wikipedia</a></li> “The signals SIGKILL and SIGSTOP cannot be caught, blocked or ignored, why?” - StackOverflow</a></li> Signal (IPC) - Wikipedia</a></li> Searchable Linux Syscall Table</a></li> IEEE Standard 1003.1-2024 / POSIX 2024 (you can browse or download an HTML version here</a> or ask me for a PDF)</li> </ul> Radio Listening: Airplanes 2025-01-27T00:00:00+00:00 After i received the kit i ordered</a>, i fiddled around with software, eventually figuring out i would install SDR++</a> (though SDR# and sdrangel are also good picks). I sat down, got a feel for the controls, installed a frequency band plan made by a French person, and started listening to the 120MHz-137MHz band, in AM mode. After a little while, spikes appeared. i happen to live not too far to an airport with a somewhat busy schedule, and i quickly figured out what was happening. This post goes over everything i have learned to do to listen to and track airplanes and airports, so that you too can know these things and do them (unless you are German, apparently</a>). There’s one caveat, however, which is that while EU air traffic control has one common regulator (EUROCONTROL), ATC might look different in other places (or even differ slightly from one locality to another). #</a> Voice Communication</h1> Airplanes and airports communicate via radio waves. Medium-sized and big airports will typically have a air control tower, with people working in there to communicate with individual airplanes, and broadcast information. There are also coordination centers across the country that operate communications. For clarity, i’ll call people in there “controllers”, regardless of the different positions they can have. Voice information broadcast happens via AM mode, on a band between 118 MHz and 137 MHz, with a 10 KHz bandwidth. One consequence of the use of AM is that the voice signal can travel quite far away as long as the plane is in the air (where you are very likely to have line of sight even tens of kilometers away). Airports will typically have multiple frequencies for airplanes coming into the airspace or needing information or needing to negotiate with controllers. The first of these sets of frequencies is called Approach frequencies. The next ones are typically called ATIS, or Automatic Terminal Information Service</a>. The last ones are tower frequencies. Those are not the only types that exist, by any mean (depending on the airport there are others such as for taxiing on the apron, the area around gates, etc, and for passing way above the airport where you could not reasonably inconvenience anyone). As i have been informed, some of these frequencies are managed by the local airport, and some by the coordination centers in charge of air traffic control across the country. When planes are about to enter the airspace of a nearby airport for landing, they need to announce themselves to the controllers via the dedicated frequency. In big airports, multiple of those can exist, and the appropriate one is picked depending on your hardware, your location, or other circumstances. Here at LFRN (the airport in Rennes, France), the distinction seems to be depending on your area of approach. Despite that, i have only ever heard one of the frequencies used (perhaps because of my direction with the airport). The pilot does not pick themselves however. What happens from their point of view is that the last controller they talk to before approaching the airport will tell them what frequency to tune in for approach before leaving them. When planes want information about the state of an airport, they tune into the airport’s ATIS frequency (for the language they want, if multiple are available). The ATIS frequencies broadcast voice recordings1</a> summing up a lot of information about the different runways, weather information, etc. Did you know that, as of October 2024, the ATIS for Charles-de-Gaulle and Orly airport, in Paris, use Text-to-speech for all languages? Now you do. Over where i live, LFRN uses a loop of a recorded human voice. Finally, when planes need to negotiate with controllers to take off, land, or move around on the ground, they communicate on the Tower frequencies. This is where the bulk of activity will usually happen when you’re next to a medium-sized airport. Every time an airplane takes off or lands, you will be able to hear a whole exchange giving landing authorization, or asking to loop around to wait. Controllers will provide direction, weather information, etc. They will also indicate what taxiway to take to exit the runway, and so on. In all of these situations, the language used is fairly codified. This is done both for making sure everyone understands each other, but also for clarity: AM radio goes far away, but voice quality is shoddy. Telling apart a German saying “Nein” and an English-speaker saying “Nine” is hard, so we say “Niner” instead for the digit. As an example, here is a small sample of a conversation between controllers at LFRN and a plane landing, roughly translated to English: Controller: 80 44 Rennes good evening, runway 28 landing authorized, wind 180 degrees, 16 to 22 knots, runway is wet plane (FMY8044): landing authorized 28, runway wet, FMY 8044 Controller: 8044 exit runway first left via delta, drive to ALAT parking, listen to 121 decimal 7 2 5 and call back when runway cleared plane (FMY8044): we’re driving to ALAT, 121 7 2 5 and call back when runway cleared. FMY8044 plane: (FMY8044): Runway cleared, end of service 8044. Controller: Received 8044, call back ALAT for exit. plane (FMY8044): We will call ALAT for exit. FMY 8044. </blockquote> When a plane receives orders, they will have to repeat the information to tell controllers that it was understood. Similarly, when they enter an airspace, they must announce themselves. Otherwise, the pilot will quickly start noticing that a fighter tail is telling them, and they should expect to be arrested and stripped of their license immediately after landing. In the exchanges above, plane FMY8044</code> received authorization to land, as well as information about wind and runway conditions. They acknowledged back. The controllers then told them what taxiway to use to exit the runway through, and where to go and what frequency to listen for information. Controllers tell the plane to call them back when the runway is cleared, which they do later, announcing the end of their service for the day. Controllers announce who they need to call to exit the taxiways, and they acknowledge, repeating their call sign every time (although i could not hear the start properly so i just pasted FMY8044 in its entirety, they tend to shorten them as does the controller). For context, ALAT is the acronym of Aviation Légère de l’Armée de Terre, or lightweight aviation division of the french ground army. Do of that information what you will. Now, in terms of voice communications i also receive next to an airport, tower controllers can also communicate with marshallers (the french call them “sécurité” but they are not exactly security, more like on-ground personnel on the runways making sure everything goes correctly). They may be tasked with doing sweeps of the runways, guide airplanes to follow them, etc. Sometimes they also get to hold glowing sticks. Neat. The marshallers team at my airport once had a very confused discussion with controllers after a departing airplane absolutely bodied a seagull. They retrieved the corpse of the poor thing, of course. In order to do that, even they had to get permission from Tower to move around. In that situation, airplanes always have the right of way, even in an emergency. And how do you what frequencies correspond to which airport and vice versa? Pilots have documents for that purpose, and complicated instruments. i personally have SkyVector</a>, a global database of airport information. Say you want to tune into the french-speaking ATIS for Lyon’s LFLL, aka Lyon Saint-Exupery, you’d read that it is at 126.18 MHz</a>. Tune your SDR, deploy a roughly 60cm long straight dipole antenna looking at the airport, and boom: you have audio of people talking. Now, receiving voice is harder when you are far from an airport. Receiving voice from airplanes is easier, but they need to be somewhat close to somewhere that needs them to talk in order for any speech to happen. Most of what i talked about here also apply to small aviation clubs, especially as they depend on a local airport for operating. The aviation club in my city also talks with controllers, and uses the same runways and taxiways as commercial airplanes. From the interactions i have heard, i am fairly sure they know each other. And that’s roughly it for voice. It is not technically complex to listen to airplanes. If you do not live next to an airport, websites such as LiveATC</a> offer you archives and live audio of air traffic control frequencies (except in Germany, of course, because Germany is spiritually opposed to people having fun). But in order to know what airplanes are up to, you kind of need to know where they are? Don’t worry, i also got you covered. #</a> Aircraft Transponders</h1> Airborne vessels of basically all kinds need to tell the ground and others flying where they are, at least in terms more precise than “above you, silly”. They are equipped with transponders. These things will just broadcast information about the vessel including: flight number, aircraft altitude, heading, latitude, longitude, and status code (also called squawk). These transponders mostly use the Automatic Dependent System Broadcast (ADS-B)</a> protocol in Europe and Northern America, at least. In Europe, planes broadcast information at 1090 MHz. Americans are incapable of managing their band plans and, therefore, they have ADS-B at 978 MHz, which once raised concerns about cellular phones2</a>. ##</a> ADS-B Antenna Setup</h2> My own setup for listening to ADS-B uses the following items: RTL-SDR dongle</li> Wideband Low-Noise Amplifier (i could use the ADS-B specific one sold by RTL-SDR</a>, but i only have the wideband variant)</li> Antenna kit: small antenna set, pulled all the way out, in a straight dipole</li> </ul> Optionally, i also add a reflector. This can be any plate of metal, or an aluminium sheet behind the antenna in the direction you target. Because i have immense brainrot and find it pleasant to be treated like an animal by others, i even have my own para-bowl-ic antenna: ##</a> Listening to Transponders</h2> ADS-B carries a ton of information, as stated above. Because it is transmitted in the UHF range, the signal penetrates buildings pretty badly. Several tools available online allow you to receive and decode ADS-B on 1090 MHz. The keyword to search for them is usually something like “1090”. Personally, i tried readsb</code></a>, rtl_adsb</code></a> and dump1090</code></a> (specifically the FlightAware variant3</a>). In the end i kept using dump1090</code>, because readsb</code> was a pain to get working, and i learned about rtl_adsb</code> after using dump1090</code> for a while (and the former needs a bit more processing and software to actually read the ADS-B messages). For visualization, i initially looked into tar1090</code></a>, but quickly realized that it is closer in taxonomy to a distribution for creating a small ADS-B collection computer on a Raspberry Pi or other micro computer rather than a single piece of software4</a>. Therefore, i ended up using viz1090</code></a>, which despite being more minimal, actually works with a single small binary i can compile. First, start running dump1090</code> somewhere. You will need multiple arguments: dump1090 --net --gnss</code></pre> The arguments mean: --net</code> expose the data via the Beast protocol, by default on 127.0.0.1</code> and a bunch of ports (including 30005</code>, the Beast output port)</li> --gnss</code> tells the program to resolve altitudes via the Height Above Ellipsoid (HAE) system (most like WGS84, i assume).</li> </ul> Technically you do not need anything beyond --net</code> for this to work. You can also add --stats</code>, --stats-range</code> and --stats-every [time]</code> to get stats about the distance of planes observed when the program exits, and every [time]</code> seconds, respectively. If you want these statistics, you will need to provide your reference position using --lat</code>/--lon</code>, otherwise dump1090</code> has no idea where you are located. If you are using a Low-Noise Amplifier on your transmission line, like i do, set the gain down at around 28.0 dB, so that the LNA does not saturate your signal (otherwise you will get nothing). For reference, i went from receiving no signal when dump1090</code> cranked the gain all the way to maximum with my LNA, to receiving information about planes 350+km away from me while indoors with --gain 28.0</code> (or play with it to figure out the sweet spot for your reception setup). The output of dump1090</code> should be a stream of messages that look something like this: *8d4cae4c58c383b3af7f10137192; CRC: 000000 RSSI: -24.5 dBFS Score: 27 (DF17_KNOWN) Time: 16462495.58us DF:17 AA:4CAE4C CA:5 ME:58C383B3AF7F10 Extended Squitter Airborne position (barometric altitude) (11) (reliable) ICAO Address: 4CAE4C (Mode S / ADS-B) Air/Ground: airborne Baro altitude: 38000 ft CPR type: Airborne CPR odd flag: even CPR latitude: 47.55281 (121303) CPR longitude: -2.26648 (98064) CPR decoding: global NIC: 8 Rc: 0.186 km / 0.1 NM NIC-B: 0 *5d4ca80200c875; CRC: 00000f RSSI: -23.2 dBFS Score: 20 (DF11_IID_KNOWN) Time: 16466780.75us DF:11 AA:4CA802 IID:15 CA:5 All Call Reply ICAO Address: 4CA802 (Mode S / ADS-B) Air/Ground: airborne</code></pre> Some messages carry information about GPS location, some give you the state of the airplane, etc. All messages are supposed to contain what is called an ICAO Address (from the International Civil Aviation Organization</a>), which is a unique international transponder identifier. When you receive these messages correctly, you should have hundreds of them, or even thousands, flying in your terminal every second. Now, after building viz1090</code></a>, you can run it as follows: ./viz1090 --lat [latitude] --lon [longitude] --metric --fullscreen</code></pre> With: --lat/--lon</code> as the reference position on start</li> --metric</code> to use units that make sense</li> --fullscreen</code> if you want the resulting window to be fullscreen</li> </ul> In my case, i see something like this: viz1090</code>" loading="lazy" decoding="async" /> The different planes that were recently seen are yellow, and they get greyer the longer you don’t need any message about them. As soon as position data is received, the plane appears. Information such as flight number, speed, and altitude, are added and refreshed whenever relevant messages are seen. When that happens, a circle appears the airplanes. You can also use view1090</code>, which comes with the dump1090-fa-git</code> AUR package on Arch Linux: Tot: 14 Vis: 14 RSSI: Max -24.2+ Mean -31.3 Min -35.1- MaxD: 0.0nm+ / Hex Mode Sqwk Flight Alt Spd Hdg Lat Long RSSI Msgs Ti ──────────────────────────────────────────────────────────────────────────────── 406F86 S -35.1- 7 12 346611 A2 2362 VOE2828 2307 548 234 47.121 -1.845 -24.2+ 179 01 4CA6C4 A0 6324 11278 -31.2 22 03 020095 A2 7154 RAM839J 11880 890 224 45.951 -2.415 -32.4 94 08 4952CC S 11590 -34.4 13 19 44CC42 A0 2364 805 035 -34.8 75 21 4952C3 A2 2227 TAP76G 11575 781 019 46.988 -2.784 -31.5 280 05 47A218 A2 2210 NOZ9034 10661 853 205 47.751 -3.559 -30.9 479 11 4CA216 A2 2255 EIN464 11278 803 147 47.713 -2.190 -26.2 2595 00 4409C0 A0 7624 10669 877 221 45.661 -2.916 -34.6 182 78 48663E A2 TRA46R -34.5 687 80 4CA852 A2 7463 RYR5716 10973 764 021 47.342 -2.423 -28.1 2387 0 440185 A2 2356 EJU4787 10951 840 195 45.442 -2.644 -31.2 4607 0 4CADC2 A2 7462 RYR601Y 12184 813 021 48.242 -1.904 -29.3 14201 1</code></pre> Here is a little example of many planes, showing their ICAO address, squawk, flight number, etc. The flight numbers are also transmitted via ADS-B. For the meaning of squawk numbers, i’d encourage you to go read the Wikipedia page</a>. Oddly enough, French Wikipedia has a more thorough</a> list of codes. You can see that some lines are missing information: because all of the information about a plane is not transmitted at once in every message, you will get partial information until you receive the rest of the information. For example, the plane with hexadecimal ICAO address 4409C0</code> above does not show a flight number. Most likely, i just did not receive any message with that information in the 182 messages i got from that plane (or they were malformed and useless). Some are missing squawk, some do not have GPS coordinates yet. Good reception will improve the number of messages correctly decoded, and increase the amount of information you can get about planes. ##</a> Feeding Global ADS-B Antenna Networks</h2> Considering that you can receive all of that information on a budget of less than 40€, it seems natural that people around the world banded together to crowd-source them. There are multiple online networks of ADS-B antennas, and even other protocols, and i haven’t contributed to any yet. It will certainly be a future topic of investigation on my end, which might warrant a small spin-off post in the future. The network i use the most for online plane world maps, FlightRadar24</a>, pools information from around the world to display a gigantic map and provide archives of plane and flight information. They even tell you how to build your own receiver</a>, and you can even apply to get a free one</a> if you live in a zone with poor coverage. By contributing data to their network, whether with your own hardware or not, you can also get their otherwise very expensive premium plan for free. i also know of FlightAware</a>, a multinational company that operates a similar service to FlightRadar24, at a larger scale, with a very large coverage</a>. ADS-B Exchange</a> is another network, with less paywalls. i haven’t used it as thoroughly as the other two but it seems to have the basic features of maps, resolving aircraft information, and flight archives. It runs tar1090</code>, which i mentioned above. It is very decent software and does its job, although i wish i could switch units from imperial to metric (if i can, i haven’t found how to yet, and i know pilots use imperial units for elevation). #</a> Conclusion</h1> This is more or less the entire gamut of radio observations possible on commercial and civilian aircrafts i know of, for now. Military stuff is a little more complex, even though some of these aircrafts also have regular transponders that you can also listen to. i could spend a good amount of time just sitting down, listening to air traffic control, and vibe. It’s a genuinely pleasant activity. i would, however, not recommend you pull up next to an airport and get out a big antenna; cops, in general, really do not like that. Hopefully you learned something. i glossed over a lot of details for the sake of readability and not drowning you, the reader, in technical minutiae that you may not care about. Let me know if this was of any help, or made you learn something interesting about those flying metal birds in the sky. I’d like to thank folks who helped me write this post: rail</a>, who helped proofread, MagicLike</a> for contributing his knowledge about air traffic control, and kloenk</a> for helping me figure out where the hell German law says you can’t listen to any radio communication you are not the recipient of. There is now a more modern digital ATIS which can apparently be beamed into airplanes to give them information? Awesome. it seems to use other means than radio, however. My airport does not advertise digital ATIS, so i have not looked into it. ↩</a> </li> Yes, that’s very close to the E/R-GSM-900 bands, for cellphone data. Yes, that’s one of the reasons airplane mode</a> exists, even though more modern standards should not interfere with transponders</a>, and even though the concern about interference was apparently always out of an abundance of precaution</a>. That is to say: you do not want to say “it’s unlikely to happen” and then be in the 1 in a gazillion flight where a concordance of reasons make it so that vital instruments used for a complicated take-off or landing are jammed this one time by a group of cellphones conveniently all screaming to connect to a GSM base transmission station. Still, you should use airplane mode to save battery and not overload cell towers underneath you with tons of hand-over requests (unless your plane is designed to include a cellular “pico-cell”</a>, which is the case in Europe) ↩</a> </li> It seems, from my understanding, that at some point in the distant past, one fella made a program called dump1090</code>. It seems like it used to dump data via JSON or the Beast protocol</a>, which other tools could feed off of (such as tar1090</code></a>). Now, at some point, the original tool seems to have been abandoned, and many people made their own forks. One of them replaced JSON output with a protobuff output, and others kept JSON and rewrote parts of the tool. I picked one of the variants, by the folks behind FlightAware, which kept the JSON output. ↩</a> </li> tar1090</code> seems cool, but the problem is that it is essentially a collection of scripts and Nginx configs and PHP scripts that gets installed, with a new created user, and so on and so on. It’s kind of a lot to ask for just a software that shows me plane information on a map. ↩</a> </li> </ol> </section> My reading queue as of January 2025 2025-01-21T00:00:00+00:00 i wanted to write a little something about the books i am currently reading (as of January 2025), both technical and fiction. Don’t expect a thorough review, i generally haven’t made it past half of those. Some titles are in French, of course, because i am afflicted with a rare condition called “being French”. It’s terminal. Look it up online. Take some time to open a tab on Deepl</a> as well. Anyways, here it is. #</a> IPv6: Théorie et Pratique</h1> by Gizèle Cizault (O’Reilly) [9-782841-770854]</a> This book essentially goes over all of the technical and theoretical aspects of IPv6. It’s a technical book, unsurprisingly, so a lot of it for me so far feels like a long list of various information about many aspects of the protocol. Getting into this book is made much harder by the fact that i have the second edition, which was published in 1999. Fun fact: Gizèle Cizault does not exist. It’s a pen name of the thirty or so researchers and academics who contributed to the G6 work group that authored the book, among other things. This happens more often than you’d think. #</a> Compilers: Principles, Techniques and Tools</h1> by Alfred V. Aho, Ravi Sethi and Jeffrey D. Ullman (Addison-Wesley Publishing Company) [9-780201-100884]</a> It’s the dragon book. If you don’t know what the dragon book is, it’s essentially the reference for every single (good) university class on compilers. At least the foundation. It goes over everything from what constitutes a token, to how to build parsers, to how to turn those into automata, and so on. Thing is, however, that the edition i have was printed in 1985. Compiler research has somewhat evolved since then, and the last edition, in the 2000s, does not go into great length into things like polyhedral compilers</a> or how the rise of Symmetric MultiProcessor (SMP) and parallelism came into play in the late 90s. But it’s a good read. If you ever need a complete, thorough reference on the basics of how to make a compilers, and don’t want to just open the LLVM codebase and inflict yourself with man-made horrors beyond your comprehension. Oh and it’s also ridiculously expensive because it’s technical and a class requirement. #</a> Réseaux</h1> by Andrew Tanenbaum (3rd edition) (translated by Jean-Alain Hernandez & René Joy) (InterÉdition Paris) [9-782729-606435]</a> Another old technical french book, this one from the 80s. Andrew Tanenbaum, who is kind of a big deal in the game of operating systems, essentially goes over all of the technical aspects of networking from the standpoint of someone who, in the 80s, might have had to use a computer to do technical things with it and the network, but also implement new things. That book usually sits on my desk at work, as it comes from the library of the lab i work at. i go through it when i feel utter despair at the idea of doing the actual things i get paid for. Kind of like the next book in this post. #</a> Système d’exploitation (2nd edition)</h1> by Andrew Tanenbaum (translated by Jean-Luc Bourbon, Jean-Alain Hernandez & René Joly) (Pearsons Education France) [9-782744-070020]</a> Same deal as the previous book, except this is about operating systems in general. A lot of the principles described in there are still somewhat applicable, and some of those are still used in CS classes. This is not susprisingg considering that CS education tends to only talk about UNIX-like monolithic operating system kernels. #</a> House of Leaves</h1> by Mark Z. Danielewsky (2nd edition) (Pantheon Books) [9-780375-703768]</a> House of Leaves is a book about darkness, and sound. House of leaves is a book about the life of a man that gets ruined by his progressively deteriorating mental health, increasing obsession with a delusional, gargantuous project, and the condition of the world around him beating him down until he is a delirious, shallow shell of a person. House of Leaves is a book by Zampanó, who dies before the events of the book start taking place. In it, he captures the lives of people who have been touched by a small compendium of several pieces of film, writing, and interviews, detailing the story of the Navidson family, and, especially, head of the family William Navidson. The Navidsons moved into a quiet little house with the hope of being able to mend their marriage that’s falling apart. That works, up until the family faces impossible geometry in the house in the form of walls that keep changing in length, alcoves and hallways suddenly appearing in the house , and the house revealing a path into a deep maze defying the physical world that swallows everything and everyone living in the house . House of Leaves is one of the last possessions held by Will Navidson as he finds himself losing hope to escape the impossible darkness underneath the house . The other one is a box of matches. I actually finished reading that book while this post was sitting in the queue, but i still included it, because it is a fascinating piece of literature. #</a> The King in Yellow</h1> by Robert W. Chambers [Project Gutenberg]</a> Another horror classic. A collection of short horror stories, and the most famous one is the first one (the only one i have read so far). Imagine this book, that when you read it, confers you with unspeakable horrors. The narrator is unreliable, the story is just plain odd in the way that makes the horror very subtle and diffused throughout the narration. This book is more notoriously known these days for being featured in the 2022 hit videogame Signalis</a>. #</a> Rust Atomics & Locks (1st edition)</h1> by Mara Bos (O’Reilly Media) [9-781098-119447]</a> This book goes over all of the knowledge you need to handle parallelism and especially locking in Rust. It is written by Mara Bos of the rust library team. It does a very good job of introducing these problems from the start to people who already know how to write some Rust. It starts with the fact that we have threads that exist in the language, how to use them, and then everything that can go wrong when you try to manipulate data from multiple threads, and especially how to help that with locks. The rest of the book goes over how these things are implemented, and guides you towards creating small toy implementations of these locks. Funnily enough, i started reading that book after coming back from Kernel Recipes 2024, and had the pleasant surprise of discovering that i had lunch there with the guy who wrote the foreword. Pretty neat fella who talked to me about RCU. #</a> Bullshit Jobs</h1> by David Graeber (Les Liens qui Réveillent) [9-791020-907363]</a> This book extends the thesis that Graeber first put out into the world in the early 2010s: there are entire lines of work that are designed solely to keep the workers busy doing things that are depressingly useless and that even they cannot convince themselves are useful, even though they are required to keep the outward appearance that they are. Graeber is an anthropologist, and it is kind of fascinating for me to read a book written by an academic in such a distinct domain of research. Human sciences bring themselves to such different ways of presenting your thesis and the supporting evidence. The book is essentially segmented as follows: what are bullshit jobs, what are the archetypes of bullshit jobs, why having a bullshit job hurts, what is it like to have a bullshit job, why do they keep proliferating, why is nobody doing anything about it, and what can we do about that problem. My bookmark tells me i’m somewhere between “what are the archetypes of bullshit jobs” and “why does it hurt so much to have one”. #</a> Des Électeurs Ordinaires: Enquête sur la normalisation de l’extrême droite</h1> by Félicien Faury (Seuil) [9-782021-518948]</a> This book is a sociological study of a sample of people who are in favour of the French far-right party Rassemblement National. Most interestingly, the author chose to study people in an area of France that is not usually talked about for its high adherence to far-right ideologies. It is often thought that only former industrial regions ravaged by globalization were in favour of the party. Consequently, people only focused on these demographics repeat far-right talking points linked to loss of job opportunities and stagnating paychecks, somewhat justifying these ideas, which in reality are just masks used to mask more nefarious policies, which they also often justify with the first talking points (e.g. “migrants are stealing your jobs we should close the border”). As Faury shows progressively in the book, what really drives adherence to the RN in south-eastern France is not, in fact, just a fear of economic insecurity. The general take-away of the book is that a culture of racism and lack of diversity in the demographics, mixed with the feeling that people are not doing as well as they should in the social hierarchy, leads them to need to blame someone. These people look for a category of others, who are not them, who they feel like they can get away with blaming for the failure of the system that was rigged against all but the richest. Those people are often migrants, the poor, the disabled, irrespective of whether or not the people studied actually encounter them in their daily lives or not. The RN also points at migrants, queers, the poor, the disabled, etc, and says “those are the people responsible for the system not working how it should, who are stealing your job, stealing your money, going against your values”. They promise to do something about it, which is more than any liberal or right-wing government has in the last 30 years or so, and that’s all that voters need to hear. Converting raw IQ from `rtl_sdr` to WAV IQ for SDR++ 2024-12-30T00:00:00+00:00 Recently i published the first of a series of posts on radio listening</a>, and while this is not technically part 2 yet, i figured something out that is quite important to my workflow of recording and then processing radio input. i own a RTL SDR Blog V4 dongle</a>, which is based on the RTL2832U chip. The drivers you install come with a set of binaries</a>, including rtl_sdr</code>, which lets you record a band of the spectrum into a raw data file. Later on, because i use SDR++</a>, i would ideally want to open those files and process them with the UI in that program. SDR++ (or sdrpp</code>) is also capable of recording raw bands of the spectrum into what it calls baseband files, but it cannot immediately open raw files recorded from rtl_sdr</code>. The reason for that is that rtl_sdr</code> outputs RAW “IQ” information, whereas sdrpp</code> reads “IQ” information in a WAV file. My (incorrect) understanding at the time of the technical aspects of IQ data, by me, an idiot</summary> For reasons that are too complex (lol) to get into and that are also somewhat starting to fade from my memory, an alternating electrical signal can be expressed as a complex number: it has a magnitude, a period as a function of time, and an initial shift. Complex numbers, represented in their more "vector" or "algebraic" form are essentially two numbers in a trench coat: one is real (let's call it Q), and one is "imaginary" (let's call it I). When reading from the RTL2832U chip, you probably get two samples, 8 bit each, for I then Q. What `rtl_sdr` does, roughly, after tuning, is reading a given number of those samples and writing them in a file, which is then *RAW IQ* (as in, just wrote the I and Q parts one after the other in a big sequence, no headers, no nothing). What SDR++ and other tools will record and read, however, is WAV IQ, which is to say, IQ data that has been de-interleaved, processed into the appropriate number size for the program (SDR++ lets you choose, and GnuRadio uses 32-bit floating point numbers</a>). How exactly the signal is created remains somewhat of a mystery to me at this point. My best guess is that the conversion process involves taking in the decoded complex number from IQ data and generating the corresponding input signal, and then mushing them all together. Exactly how that works? When i figure it out, i will update this part. </details> There is exactly one workflow where i would, ideally, not want to have to be on the UI and click a button to record on SDR++ at a precise time and again to stop it: satellite listening. Whether it be for catching SSTV from the ISS, LRPT from the Russian Weather Satellites, i do not want to get up at 7AM to do it. Or, rather, it is highly inconvenient that it’s always the ideal time. i also recently discovered that, at least for SSTV from the ISS, a straight dipole is often fine. So how about slapping a bunch of sleep</code> commands, a bit of CLI nonsense, and get this recorded and opened in SDR++ later so i can actually record the track i want? Well, reviewing the command i use for rtl_sdr</code>: rtl_sdr -f 145.8e6 -s 60K -g 20 iq-145800000Hz.raw</code></pre> Summing up what is going on: -f 145.8e6</code>: tune to 145.8MHz</li> -s 60K</code>: record a bandwidth of 60K. rtl_sdr</code> uses the term “sample rate” which is also used by SDR++; what it actually means is how many samples are taken around the tuning frequency, or how big the band you’re seeing “at once” is</li> -g 20</code>: set the gain to 20 on the dongle</li> <file name></code>: pretty explicit; this one includes the frequency, for a reason i will explain later</li> </ul> Now, i use sox</code></a> to process the raw data into a WAV container: sox -t raw -r 200k -b 8 -e unsigned-integer iq-145800000Hz.raw \ -t wav -r 200k -b 16 iq_145800000Hz.wav</code></pre> There is probably an equivalent for FFMPEG, but i haven’t tested that yet. What it does is, roughly: There is an input from file iq-145800000Hz.raw</code>: The bit per sample (-b</code>) is 8</li> There are 200k</code> samples per slice of input</li> The type is raw</code> data</li> The integer type is unsigned</code> (here for 8 bits: 0-255</code>)</li> </ul> </li> The file created as an output, iq-145800000Hz.wav</code> is: Of type wav</code>,</li> Has a sample rate of 200k</code></li> An integer type of 16</code> bits (you can change this part depending on what you use in SDR++)</li> </ul> </li> </ul> Now, opening the resulting file iq_145800000Hz.wav</code> with SDR++ should show you exactly the recording you plan to use. And why the frequency in the file name? Well, see, SDR++ needs to know a center frequency to show you on the tuner, and while it does not actually matter when reading a file (the waterfall will just display the entire band and let you zoom in on it and use other tuners like radio, NOAA, METEOR, etc. on parts of it), it is good to now where you are tuned in if you have for example bookmarks and such. So how does SDR++ determine the center frequency from the WAV file? Easy, the fucking file name</a>. i kid you not; i spent a solid 20 minutes digging in the code only to figure out that getFrequency</code> searches the filename with a regex to figure out the first occurrence of [0-9]+Hz</code>, and returns 0Hz otherwise. That is actually a smart choice considering the limitations of WAV files (i.e. you cannot use custom labels for metadata and they are generally annoying to use in general). Diving into Radio Listening: Introduction 2024-12-19T00:00:00+00:00 In the late 2000s, when i was a wee child, my parents brought me along to the big yard sale organized in the tiny town my grandparents lived in. One particular time, we came back with a set of electronic gizmos that included a marine radio set. That huge cube of metal, with a deployable 2-meter long antenna 1</a>, and a big selector to pick between various AM bands labeled with lengths, CB, Maritime Frequencies, etc, also came with a booklet left there by the previous owner with tables of names of foreign radios, and, in a big grid, hour intervals. That radio set is probably collecting dust in a garage at the moment, but it gave me a couple dozen hours of fun for the price of several 9V batteries. #</a> Radio Listening: An Introduction</h1> For most of my life, my only exposure to radio was via the commercial FM stations my parents listened to throughout the day when i was a kid, and in the car. i had several ideas in my mind about these elusive “radio waves”: radio waves are electromagnetic and you can’t feel them but they’re there, and line of sight is important, but they can also bounce, somewhat; kind of like visible light</li> radio waves were how TV and phone and WiFi worked, and sometimes you could be in a spot where no signal could exist, and a meter next to there you’d have great reception</li> radio was AM or FM, and those meant that the sound was transmitted differently, although i had no idea how exactly</li> you could tune to a specific number to hear a different signal, and it worked</li> Signals could only travel so far, and different antennas in different regions would emit the same station sometimes at different frequencies</li> </ul> In August 2024 i bought</a> a SDR</abbr>. It’s not a particularly good one; the point was that i wanted to explore radio frequency listening, something that had always lingered in the back of my mind, and recently became a newfound topic of interest within the French internet thanks to a couple</a> videos</a> by a French creator and musician. and thus, i fell down the RF rabbit hole. This is sort of an introduction of a series of posts that will come in the future. Those posts will be more digestible little bites of exploration on specific topics, as opposed to the first draft of a huge post i originally attempted on this topic. Things we should cover, in no particular order: Listening in on my local airport</li> Losing my mind at French RF regulation</li> Eaves-dropping on the neighborhood kids and the chilling experience of knowing all you broadcast can be listened to</li> That time i cobbled together a little raspberry pi, my phone and my SDR to listen to what everyone was saying downtown</li> France being a paranoid country so even firefighters use encrypted comms</li> That one time i received an industrial sensor alert via POCSAG (pagers) and shit myself (conversely: that one time i received a message via POCSAG with an exact address and someone’s cause of death)</li> Listening to an american pastor talk from thousands of kilometers away with a puny little stick of metal</li> Soldering a bicycle brake wire to make the stupidest antenna in existence</li> Running across a parking lot with an antenna pointed at the night sky and yelling “COME BACK HERE YOU FUCKING INTERNATIONAL SPACE STATION MY TAXES PAY FOR YOU”</li> </ul> And much more. Don’t expect necessarily 100%-accurate information either. i am armed with a SDR, and a computer science engineering degree. i might try to get my telco friends to check over this, and i can always pray to the Balanis</a> for guidance, but the science gets over my head when we reach the physical layer of a medium. You can help me improve this by sending an email if you want. #</a> A Primer on Electromagnetic Frequencies and Their Propagation</h1> Nonetheless, here is a primer on information that should be necessary to follow along, but digestible. This section might be updated as i add and correct information for further posts in the series. If you are a physicist or work in telco, plug your ears and close your eyes: Electromagnetic wave propagate in space. The space around us is full of electricity, or, rather, with the potential of electricity hanging around. Analogous to that, the air around is is like a big space full of individual points of air that can be more or less compressed locally. Like mechanical waves, EM waves are created by a disturbance in the field through which they propagate. The disturbance changes the property of the space, or milieu. For air, a sudden push will locally compress air, which will then push against the air nearby, and so on. For EM waves, the change in electrical potential will propagate in space, creating a change in the field locally, which propagates. The key property of waves is that they are a propagating disturbance that keeps even after the source of the disturbance has ceased acting on the milieu. Examples of that for sound waves are: your voice, or thunder. The sound of thunder travels to you even as you’ve already seen the strike hit and fade away, and the light of stars extinct for millions of years is still in the process of reaching us. When propagating, electromechanical waves diminish in intensity as they travel and hit matter. The exact way they dissipate and how much they can traverse before being fully imperceptible depends on other properties. For light, that travel happens at a speed that physicist will ask you to consider to be constant, because theories seem to uphold that (at least last time i did an optics lab in university). The milieu of propagation actually impacts propagation ever so slightly, creating effects such as rainbows and most diffraction. So far i’ve mostly talked about light, because it is the most intuitive form of electromagnetic wave. However, they exist on a spectrum. Waves are typically described by the “shape” they have: the way they modify the property of the milieu as time goes on. That “shape” changes depending on the source of the disturbance and potentially the shape of objects around (which is why a piano sounds different from a guitar, or a trumpet). That shape may be very uniform, as in the field of propagation (air, or the electromagnetic field) changes at a constant rate and a constant period, in which case you have a “pure” signal at one frequency: the interval of time it takes to return to a state of the wave you have already seen (also called a period). If your period is 1/440th of a second long for sound, you hear a nice A note. For electromagnetic waves, if you have exactly 530000 billion periods in a second, you are seeing the color green. Because periods get ridiculously small, we use frequency, the number of periods per second, with the “Hertz” unit. The prior examples are usually expressed as 440Hz (A note) and 530 THz (Terahertz, or 1000 billion hertz). For electromagnetic waves we also tend to talk about the wavelength of a signal. Because light travels at a constant speed, we can know precisely how far it will have gone during one period at a given frequency. That length, the length of a wave, is used to characterize a pure signal. Our 530 THz green has a wavelength of about 565 nanometres. It means that if you placed precise measurement tools every 565 nanometres for that specific wave of green, which could extremely accurately measure the modification of the electric field, you would see them all show the same value. They would keep the same value if you slid them all alongside the direction of propagation. Frequencies start at 0 (which is not technically possible on an alternating signal like we are talking about so far). From 0 to about 10^10 Hertz (or 10GHz), we find radio waves. Those have a wave length varying from about as big as possible (in most practical uses, at most the size of a skyscraper), all the way to an insect. At 2.5GHz (12cm wavelength), you’ve got your microwave oven’s frequency for bringing water to a boil, and old standards for WiFi. At around 5.1Ghz all the way to 5.7GHz (~5.5cm wavelength), you get your modern WiFi 5. In general, we will be concerned with what happens between a couple hundreds of kilohertz, and about 1.5GHz. So far i have only talked about electromagnetic waves of a given pure frequency, meaning that if you could visually observe the wave, you would see a change in the electromagnetic field around you happening uniformly, with the field going “up” for half the period, then “down” for the rest of the period, following a sine pattern, which would repeat forever. In reality, things are messy. As mentioned, not everything is a pure signal of one frequency, in fact, it would be highly impractical. So we do better. In the 1810s, a guy by the name of Jean-Baptiste Joseph Fourier</a> (yes, he is french, sorry), was interested in modeling how heat propagates in materials (he helped launch a little field of physics called thermodynamics, nothing big). After settling in the french city of Grenoble, he worked at the theory and figured that heat propagates between two objects based on the difference in temperature they show. As part of his work on the matter, Fourier actually developed a nice property that we still use to this day: under certain circumstances, a seemingly chaotic function can be broken down into a (potentially infinite) sum of decreasingly big sine waves. For the less mathematically inclined, this means that if you take any form of wave in the right circumstances (and in physics we can assume those for any wave), we can break that seemingly chaotic variation of the field into a set of variations of one frequency with different proportions, or what i’ve been calling for now “pure signals”, i.e. sine waves. In space, when a piece of equipment puts out a disturbance of the EM field, it can be decomposed as modifications of precise frequencies through an operation called the Fourier transform, where we take the input signal and break it down into individual frequencies and how much they are being affected. When two signals collide in space, these base components add to each other. In some circumstances, this means that one signal will cancel out the other: if two signals of exactly the same pure frequency, but slightly shifted in time, collide, the sum of modifications to the propagation milieu will be null: there will be no signal. In sound, this is the technology behind active noise cancelling (where the headphones play an inverted recording of your environment’s noise to subtract the noise from the signal your ears receive). In other circumstances, two signals can collide and amplify one another. So, back to electromagnetic waves, we want to encode some sort of information into electromagnetic waves. Having Fourier transforms, which our computers and analog electronics can do reasonably well, is doable. In fact, electronics are actually really good, when given AC voltage at the right frequency, and with the correct components, at creating an alternating current that can have a precise frequency and intensity. If, by chance, that alternating current is going down through a piece of conductive material, the change in motion of the electricity through the wire will create a magnetic wave around the wire. If that change keeps repeating, the magnetic wave (which at that point will have also created changes in the electric field, meaning that the wave is both electronic and magnetic) will start alternating at exactly the frequency, or frequencies we want. This is, roughly, radio emission. The converse operation happens when a piece of metal is present and influenced by a change in its surrounding electromagnetic field, creating tiny changes in current in the wire, which can be picked up by precise components down the line, and interpreted as reception of an electromagnetic wave: that’s reception, with an antenna. To receive radio waves, i use a software defined radio, or SDR. Typical radios you will find are meant to be used on commercial bands, where voice is broadcast either via the (wideband) FM or AM modes. Your radio will shift some properties of its analog (or digital) components (a bit of resistance here, a bit of capacitance there), and suddenly it is attuned to changes at a particular frequency, and those nearby. In a software defined radio, my computer is the one telling the SDR what it needs to attune to. The SDR is made up of three important components: an antenna to pick up a signal (i actually plug it into the dongle, but that’s irrelevant because it’s necessary anyways), a decoder chip that interprets the incoming electrical noise as signals, and a tuner that changes parameters of the decoder chip to target a certain range of reception frequencies. Encoding information into an electromagnetic signal usually follows one of two methods: amplitude modulation or frequency modulation. In reality, many others exist, but i will focus on voice for now, which, unless you are using a digital transmission system, uses of those two. Your voice, and whatever else music is mixed with it, is an analog signal that can typically vary up to about 20 KHz (the range the human ear can pick up). In order to encode the information, the emitter picks a frequency, called the carrier. In amplitude modulation (AM), the analog signal is used to change the amplitude of the carrier. Imagine your nice sine wave, repeating at the frequency of the carrier that you’ve picked (e.g. 100 KHz), except the sine does not go up to 1 or -1 every time. The contour of the sine draws the shape of the much lower frequency signal that you want to express: </a> By Berserkerus</a> - Own work, CC BY-SA 2.5</a>, Link</a> In frequency modulation, you take the variation of the input signal to change the frequency of the carrier. That is, if your signal is a nice sine wave like shown above, your FM signal will alternately have a slightly higher frequency than its carrier, then a slightly lower frequency. The effect of this is that your signal, once processed to decompose it into its pure components via a Fourier transform, will exhibit a peak that oscillates around the frequency of the carrier. To simplify, AM draws the signal on the amplitude you measure, whereas FM draws the signal on the Fourier decomposition. This is why you typically need to capture a range of frequencies around the carrier. Information is actually encoded in a given range of frequencies. Even for amplitude modulation, where we multiply the carrier and the signal, the resulting signal is actually a sum of sines that are around the frequency of the carrier. In fact, for the AM mode, the range that these frequencies can take is twice the range of frequencies of the input signal. For FM, it is the same as the input signal. That range of frequency that is listened to to demodulate your signal is called the bandwidth of the signal. The typical way you will show the presence of electromagnetic waves and how they break down into individual frequencies is called a spectrogram, it will show the spectrum of frequencies that you are interested in. A very specific type of spectrogram is often used by radio software, is called the waterfall spectrogram. An example is shown below: </a> By Konung yaropolk</a> - Own work, CC0</a>, Link</a> In waterfall displays of a spectrum, a gradient of colors (here blue to dark red) encodes how intense the received frequency is, which is its amplitude. Recall that a signal diminishes in intensity, or amplitude, the weaker it gets. #</a> Conclusion</h1> So, this is about as much as i can cram into a single article, and about as much as i can explain before we get into other stuff that can be explained in another article. This section is absolutely getting updated at some point, but we’ll see. See you by the spectrum waterfall~ or at least it felt that long when i was like, 8. ↩</a> </li> </ol> </section> A 2024 Update on the Rust Foxes Linux Module 2024-08-20T00:00:00+00:00 First, the context. For those who are not aware, Rust for Linux</a> is a project within the Linux developer community to add kernel support for the Rust programming language with the objective of developing drivers that can sit alongside regular C drivers in tree. The project is still somewhat young, but it’s already reached the significant milestone of a first driver being merged</a>. Two years ago i came across Rust for Linux thank to Asahi Lina</a>’s M1 GPU driver development streams. i went through a quick tutorial</a>, and voilà, there was a driver: rust_foxes</code></a>. ##</a> Two Years Later</h2> Now, two years later almost, what has changed? First, the Rust for Linux project deprecated the rust</code> branch everyone used to experiment. It was full of abstractions for many, many things (including miscellaneous devices, user IO, etc), but its development had not followed the regular process of (painstakingly) posting things on the LKMLs, and people reviewing them. So almost everything was basically thrown into the trash. What is coming now is definitely an opportunity to do better (eg. properly handling multiple memory allocators, properly defining memory allocation flags, a better structure for the alloc</code> crate, etc). That also means that the driver no longer really works unless you build against the old rust</code> branch from back in the days of 6.0… Well it would have, if i had rebased my code. Even the API for simple modules changed between me writing the module, and when rust</code> was abandoned. For example, consider Module::init</code>, the entrypoint of a module. In my code it is: fn init(_name: &'static CStr, _module: &'static ThisModule) -> Result<Self> {</code></pre> The current definition used in both rust</code> and rust-next</code> (the up-to-date branch used to push RFL’s updates to Linus): fn init(_module: &'static ThisModule) -> Result<Self> {</code></pre> Thankfully, during the last two years, i have spent a big part of my time working on drivers using RFL’s tree, including making abstractions. ##</a> Update Time</h2> Going through the code, as someone who now has 2 more years of experience not only writing drivers for RFL but also, crucially, abstractions, i listed several things in need of update: The Module</code> trait</li> Missing miscdev</code> abstractions</li> Missing io_buffer</code> abstractions</li> </ul> For the missing abstractions, i dove in the code of the old rust</code> branch and pulled out the big chunks of code my driver had previously relied on. Then, i brushed up very little details to update the code. As an example, consider code that turns a Result</code> into an errno code. RFL’s rust</code> had a macro for that purpose, which is now replaced by kernel::error::from_result</code></a>. In a similar fashion, everything that allocates now needs specific memory allocation flags to be passed so that the proper call is made and gfp</code> is not always assumed. i trimmed the code down to the minimum needed to open and read the device: Abstractions for struct file_operations</code></li> Abstractions to copy and write from userland</li> Abstractions for IO buffers</li> Abstractions for simple devices</li> Abstractions to register a miscdevice</code></li> </ul> The set of changes totals +1240 lines</a>. ###</a> Now, onto the driver</h3> After fixing imports and the problem of Module</code> trait implementation, i decided to flex a little by adding a new feature to the driver. Previously, the driver would mindlessly write foxes one after the other, ad infinitum. That was funny, but i can do better. The trait for file operations has two associated types. One of these types represents a data type passed to the handler of the open()</code> event. The precise instance is stored in memory when we request the registration: let reg = miscdev::Options::new() .mode(0o444) .register_new(kernel::fmt!("foxes"), ())?;</code></pre> In this example, and in my current version of the driver, that type is ()</code>. A second associated type tells the file operations what type is passed for all callbacks within the same open/release</code> session. That is, the instance is created during the open</code> callback, and destroyed at release</code>. A borrowed version of the data is then passed for all callbacks… including read</code>. The read</code> callback is the only one i implement, and the only one i need. Ironically, i use it to write in the buffer the user provided. Previously, that writing filled the buffer given by the user as much as possible, including with partial foxes (first few bytes of an incomplete UTF-8 emoji). Now, with the session data type set to Box<AtomicUsize></code>, i obtained a &AtomicUsize</code> on all callbacks, and could now play with it. Importantly, all core atomic scalar types can be modified through a non-mutable reference. That was all i was going to get anyways: #[vtable] impl Operations for FoxDev { type Data = Box<AtomicUsize>; fn open(_: &(), _file: &File) -> Result<Self::Data> { Ok(Box::new(AtomicUsize::new(200), GFP_KERNEL)?) } fn read( data: &AtomicUsize, _file: &File, writer: &mut impl IoBufferWriter, offset: u64, ) -> Result<usize> { //... } }</code></pre> The really simple part was changing the writing code so that it checks the current count remaining, limits itself to that value (or not if the buffer is small), and then subtracts the number of foxes fully written. Once the count is exhausted, and a round of read</code> returns 0 bytes read, your program will likely call close()</code>. With that code in place, all calls to cat /dev/foxes</code>, for example, will show the same number of foxes before exiting. In a subsequent modification (one i have not pushed to the repository), i even made it so the count became global to all different reading attempts. In order to do that, i wrapped a AtomicUsize</code> in an Arc</code> (Atomic Referenced Counter), and used that as a data type for the open</code> callback, at which point i cloned the Arc</code> and used it for all callbacks as well. The count was decreased in a similar way, except this time it was global to multiple parallel calls. ##</a> Conclusion</h2> i underwent this little restoration project as an attempt to not lose my mind writing a more complex, fully-fledged driver for RFL. The cute little character device that prints foxes is a cute example and the first example of code i used in the project, so i would like it to remain up-to-date, and useful to people who want to get into the project. Reflecting quickly on my (almost) two years of practice, i realized that while i gained a certain amount of skepticism regarding the capacity of the project to onboard newcomers to the kernel (myself included), i gained a lot of knowledge of how Linux works internally, but also how FFI in Rust interacts with it, and how deep primitives like cells, tools like Arc</code> or Box</code>, and regular code can work together. It’s been a journey, and while i hope to take a break from more serious endeavors around RFL, i will definitely keep tinkering. Irregular Reminder: Strip your Kernel Modules on Install 2024-05-16T00:00:00+00:00 Occasionally, I have to re-install a kernel and its modules. Worse, I have to do it on Ubuntu (for reasons outside my control). I learned to do that on my own mostly through a mixture of old Gentoo tutorials and kernel documentation. And yet, twice already I have done the whole procedure, which, once your configuration is done, and after you’ve enabled kernel debugging1</a>, is something akin to: make -j4 bzImage modules # i have four cores sudo make INSTALL_MOD_PATH=/usr modules_install sudo make INSTALL_HDR_PATH=/usr headers_install sudo cp arch/x86/boot/bzImage /boot/vmlinuz-myversion sudo update-initramfs -k myversion -c -v</code></pre> and then: kaboom. My initrd-myversion</code> is something like 1.2 gigabytes on disk (if like me you are compiling the x86_64-defconfig</code> profile with minimal changes). Yikes. I know from experience that your bootloader is likely not going to love having such a huge file to load. Kernel install blunder once: shame on me. Kernel install blunder twice: let’s write about it. The first time it happened to me, I spent about twenty minutes figuring out that symbols were enabled, and clogged up the initramfs size. I ended up also finding out that Linux’s industrial Rube Goldberg machine of a Makefile architecture can read from another environment variable</a> called INSTALL_MOD_STRIP</code></a>. When the value in that variable is 1</code>, modules are stripped at install, removing the symbols. I ended up building a new initramfs, at 99 megabytes of size. Interestingly enough, I have never had such an issue on ArchLinux using mkinitcpio</code>. My theory is that it either never pulls all of the modules into initramfs (very likely true from my knowledge so far), or strips them, or both. TL;DR: Use INSTALL_MOD_STRIP=1</code> on make modules_install</code> to reduce initramfs size: make -j4 bzImage modules # i have four cores sudo make INSTALL_MOD_PATH=/usr INSTALL_MOD_STRIP=1 modules_install sudo make INSTALL_HDR_PATH=/usr headers_install sudo cp arch/x86/boot/bzImage /boot/vmlinuz-myversion sudo update-initramfs -k myversion -c -v</code></pre> i do kernel development, so i need symbols in core, but not in most modules. ↩</a> </li> </ol> </section> Pushing to an Overleaf project through Forgejo 2024-04-17T00:00:00+00:00 I am a PhD student. Of course, I write papers. For various reasons, my supervisors and I use Overleaf</a>, which is satisfactory in terms of UX and such. However, outside of periods with high editing contention, I would rather work with my usual workflow of editing LaTeX with neovim</code>, which I have customized to compile and lint LaTeX code for me. I find writing lengthy pieces of text more relaxing in a terminal. 🤷 Overleaf has a synchronization feature over git</code>. You can choose to have your project synchronized over a GitHub project, a DropBox, or anything that uses git</code>. For multiple reasons, I avoid the first two. If you go to the project menu (with the leaf symbol, on the top left), select “Git” in the “Sync” options, you will be presented with a URL like below, and something about “Git authentication token”: https://git@git.overleaf.com/<long string of numbers></code></pre>##</a> The Problem with Overleaf’s Git Tokens</h2> Git Authentication Tokens are generated in your user settings</a> in the “Project Synchronization” section under “Git Integration”. They are shown once, kind of like most tokens. Creating one with “Add another token” leaves you with a long string that starts with something like olp_7d....</code>. Using passwords over git</code> is generally a hassle, but Overleaf gives you no other choice. Of course, not content with that prospect, I poked around and played with my Git Forge</a> to try and see if I could create a workaround. ##</a> Working Around Password Auth: Forgejo Push Mirrors</h2> The goal: work on my paper like any regular Git repository. To get to that goal, I want to be able to push to a remote that I have configured in my .ssh/config</code>. My own forge is configured already, so I decided to use it. I created a private repository for my paper, and added it as a remote of the local clone I had made of the work already done (so I had to git-clone</code> once with the HTTPS URL and token). Then, I went into “Settings” for that repository, and under “Mirror Settings”, I added a new push mirror. Its URL is the HTTPS URL shown by Overleaf, and in the “Authorization” section, I set the username to git</code>, and the password to the token. I enabled “Sync when commits are pushed”, of course, and voilà! Essentially, this uses your forge as a proxy to store your token, so you may want to create one specifically for it. You can lower the mirror sync interval for finer granularity of updates if you plan on never pulling from the Overleaf repository. However, no matter how low you set the time, you will lag behind anyone who is using Overleaf to edit the files there. Furthermore, if a conflict arises, I am not exactly sure how Forgejo handles it. If I find out, I will update this. Conky may be Leaking your RAM on ArchLinux 2023-12-12T00:00:00+00:00 Some time in the summer this year I started noticing that memory usage would become very high after a while. Specifically, XOrg would start hogging inane amounts of memory. I use an NVidia GTX2070S card on my desktop computer (which uses the x86_64</code> architecture), so I just lazily think to myself “well NVidia is at it again” and think not of it again. Recently, after becoming irate with XOrg crashing in the middle of games, or losing my open programs (and having to open 10 terminals again), I decided to look into the source. The estimated rate of leak was about 1 Megabyte per 10 seconds with nothing launched but my basic desktop utilities, which include: polybar</code></a> (which I often kill for fullscreen gameplay, so it can’t be it)</li> feh</code></a> in a bash script that runs it every 20 minutes for a new desktop background</li> conky</code></a> which gives me fancy stats I can’t see most of the ti- wait a second</li> </ul> So immediately I restart my X session, kill conky</code> and it stops leaking. An hour passes and only a single megabyte gets allocated. ##</a> The Conky Repository</h2> I use vanilla Conky on Arch, aka. extra/conky</code>. It is developed over at github.com/brndnmtthws/conky</code></a>. Searching for is:issue memory Xorg</code> on the repository leads to issue #1442</a>, which details high CPU and memory usage following a regression after introducing the use of new colour data types for X11. As it turns out, the pixel color data was being allocated, but not cached, and kept being allocated again and again. Well, darn. The faulty commit, 5e98c49</code></a>, was introduced on February 24th 2023, and released into v1.18.1</code> the same day. The fix, commit c08a782</code></a>, or 8f0a21b</code></a> in the repository, fixes the issue, and was shipped in 1.18.2</code>. So… it was fixed in March? Why’s it still broken? ##</a> The Conky ArchLinux Package</h2> As of writing this, ArchLinux ships version 1.18.1-2</code> of Conky, which I know is faulty. The package is apparently marked as deprecated, and has been since 2023-03-04, the day 1.18.2</code> was released and the leak fixed. Yet, Conky on ArchLinux was updated on 2023-09-03.. What’s up? Packaging versions do not follow the same naming standards as software release. Conky 1.18.1 was first released for ArchLinux on February 24th 2023</a> as 1.18.1-1</code> by Levente Polyak, maintainer of the package. On September 19th 2023</a> they released 1.18.1-2</code>… Which essentially just locked the build to a tag, specified some SPDX license identifiers, and enabled Wayland. And that’s it. No pulling the new version (by then, Conky v1.19.4</code> had been released upstream for a month). ##</a> So…?</h2> There’s not much to take away from this. Packaging is done by people. People are busy. Sometimes packages are stuck. We are stuck with this Conky leaking huge amounts of RAM for now… I might update this if I find more things, or when it gets solved. Hopefully if you had the same problem, and you were as lazy as I was to diagnose it, now you know why. Using git with email: a Quick Tutorial 2023-12-03T00:00:00+00:00 I am mostly writing this as a self memo for how to use git-send-email</code>, and by extension git-format-patch</code> and git-am</code>, to share patches, patchsets, and apply them. Me and my friends like to do these overly elaborate things to share code, what we jokingly call “fossbro roleplaying”, so hopefully it can help them, and you as well! ##</a> “Wait what’s all that about?”</h2> These commands let you create patches that represent a commit (with all of its metadata, save for cryptographic signature), share them automatically over email, and, once someone downloads the contents of your email, apply those commits to a tree. ##</a> Preliminary Settings</h2> TL;DR: These are the settings you want to tweak. Mine would be: git config --global sendemail.smtpencryption tls git config --global sendemail.smtpserver smtp.vulpinecitrus.info git config --global sendemail.smtpuser <user> git config --global sendemail.smtpserverport 587 git config --global sendemail.from <my email></code></pre> These settings replace CLI parameters you can otherwise provide when sending email with git send-email</code>. ##</a> Exporting with format-patch</h2> The git-format-patch</code> commands turns a commit or set of commits into files that can be straightforwardly sent as email. The only real positional argument provides the range or set of commits to format. That can be quite hard to wrap your head around but here’s a quick rundown: Argument</th> Meaning</th></tr></thead> <ref></code></td> All commits from <ref></code> to HEAD</code></td></tr> A..B</code></td> All commits in the range from A (excluded) to B (included)</td></tr> -n</code></td> Last n</code> commits from HEAD</code></td></tr> </tbody></table> </div> Note that because git</code> is the way it is, and because, when you think about it, it makes sense: when your range includes a divergence, you might pull all of the divergent commit from it. It’s just how git</code> works. In the end, you will get all the commits you need, but they will be linearized, so the application order might be a bit wonky or cause merge errors. Everything else is just optional arguments. Here are those I think would be the most useful: --rfc</code>: your patch or patchset is a request for comments</li> -v <n></code>: mark this as version <n></code> of your patchset</li> --cover-letter</code>: generate a file that represents the cover letter, or the big description you send to explain the whole point of your patchset; by default it contains diff stats, and such</li> --thread=<style></code>: the thread style controls how your commits follow each other. A shallow</code> style means all subsequent email are a reply to the first one (typically the cover letter). That is the standard on LKLMs.</li> --in-reply-to=<message-id></code>: oddly enough, I have already had to send a patch as a reply to a human-made series of email. This may be useful for that purpose.</li> </ul> ##</a> Sending with send-email</h2> This is the big complex part. Technically, git-send-email</code> is a big Perl script that wraps around calls to your mail sender. As such, you may find some missing dependencies such as perl-authen-sasl</code> on Arch Linux. The send-email</code> command takes your files and sends them as email. Easy. Except it does a lot of things for you and is very fuzzy in terms of available options. For example, it takes all emails from the Signed-off-by:</code> and such lines in your commits, and adds them as Cc:</code> lines in the email sent unless you tell it not to. Here are the options I generally use: --to=<email></code>: a new To:</code> line</li> --cc=</code>: a new Cc:</code> line</li> --bcc=</code>: a new Bcc:</code> line</li> --compose</code>: if you did not generate and edit a cover letter, write it now</li> --no-signed-off-by-cc</code>: this removes the automated gathering of emails in the Signed-off-by:</code> lines to Cc:</code></li> --suppress-cc=<choice></code>: suppress automatically found Cc:</code>, you can see the man page</a> for all choices</li> --dry-run</code>: do everything but send the actual email; the information sent to the SMTP server is shown in the standard output for debugging purposes</li> </ul> There is actually an interesting overlap between send-email</code> and format-patch</code>. You can actually configure most of everything you want for the content of your email messages via either command, depending on your choice. For example, had I not chosen threading style in format-patch</code>, I could have specified it in send-email</code> using --[no-]chain-reply-to</code>. ##</a> Applying with am</h2> If you find yourself the (un?)fortunate recipient of a patchset that you fancy adding to your project, you can simply download your email to files (or, honestly, just copy the visible content from your email browser), and run git am</code> on them. Ideally, you want to apply them in order, and over the correct commit. The details for how to coordinate with your collaborators for that to happen is left at your own discretion. Interestingly, you can use -S</code> with am</code> to sign the objects created. The signature from the original committer in their own repository cannot be maintained, because you create the actual binary git</code> objects tied to that commit in the actual repository yourself, so you must sign and certify that the content you create that commit with is valid. You are the first person liable if something wrong is introduced (and others are on the line if there are Signed-off-by:</code> lines). As as interesting side note, many places will let you download .patch</code> files that have the correct format to be an email and a git</code> patch. For example: GitHub lets you do so for individual commits by adding .patch</code> at the end of the URL, and similarly with pull requests (except the patch contains all individual patches for the PR’s commits)</li> GitLab does the exact same thing for both commits and merge requests</li> Gitea/Forgejo also lets you do the same on both commits and PRs</li> </ul> As a result, you will often find yourself using git am</code> on things that are not purely email. As an addendum, I can point you to another tutorial</a> by Matheus Tavares. VulpineCitrus is Changing CMS (Again) 2023-11-06T00:00:00+00:00 So, it has come to this again, uh? Last year, in 2022, I moved my blog</a> from Hexo, an old CMS tool written in NodeJS, to Grav, a self proclaimed minimalist flat-file CMS. At the time, I had tinkered with Grav a bit, and came to be confident that I could make something cool out of it. For about a year or so, I did. However, as time went on, I realized two (well, three) things about my actual workflow while writing blog posts. ##</a> Number 1: I do not actually use a browser that often</h2> I started using Grav with the premise that I wanted to move away from having scattered pieces of repositories on multiple computers, describing different versions of drafts of posts. I wanted a tool that I could just open on any computer, and start writing. The problem with that premise is that: I actually use the same couple machines for writing, which I almost always configure to have credentials to my git forge, and the right GPG keys to sign commits with</li> Opening a browser is a hassle at times</li> Assets still stay scattered (for example, on my SNCF blog post, I had to wait to get home multiple times to fetch screenshots)</li> Now that I am in a much more stable living situation, I have solid and resilient infrastructure to synchronize code (my forge</a>, Nextcloud</a>), meaning that out-of-sync code is not a problem</li> </ul> ##</a> Number 2: Grav is actually quite a lot of work</h2> I had to do a non-insignificant amount of testing to place my posts in the order I wanted, right off the bat. At first I ignored it, and after a while things just stayed in the right order, but it was a first taste of things to come. A lot of things work out of the box, thankfully, and I think Grav would be best suited to someone who wants a more… simple and fancy solution. I want high levels of customization and control, and I increasingly found myself having to dig into raw Markdown files on Grav just to achieve things as simple as excluding draft posts from feeds. In the end, one last point pushed me back to the old philosophy that had me use Hexo in the first place… ##</a> Number 3: Static sites are resilient</h2> About a month ago, I read a post</a> on my timeline relating this weird phenomenon in web preservation… The late Aaron Swartz (assasinated by the copyright industry; may he rest in peace) relating on his blog at the time the unfortunate passing of Gene Kan, one of the pioneering authors of the Gnutella file transfer protocol. At the time, Aaron noted that Gene’s blog was barely preserved, and only static pages previously preserved could be accessed. Yet, the way we can read Aaron’s words to this day is because he himself used a static site generator, something that could construct pure static files that were unlikely to break in the event that nobody were to deploy a new version ever again. His blog is still online</a>. I have known my fair share of websites, old forums from the 2000s and 2010s, completely broken and inaccessible. I have helped rescue</a> one of those for a school club that was absolutely falling apart. Nothing is forever, but non-static websites are shifting sand, and you take a notable risk in depending on them. Now of course I am not some grey muzzle contemplating the looming threat of death, but I am about Aaron’s age when he died, and I cannot help but think that, perhaps, the simplest solutions to my problems are also the most likely to age better. ##</a> A new adopted CMS: Zola</h2> I dabbled a bit with static site generators, first Hugo, then Zola. My affinity for Rust (and unstable projects!) made me pick the latter. It’s jank in a way that I appreciate, where it forces me to write templates myself (I based my template on the Tilde template, but heavily customized for just about all pages). That fits my workflow of putting time into things that I can customize and dissect fully, to an extent where I have now written most of the raw HTML code in this very page’s template, and can debug it if needed. Zola, as a static site generator, is cool in the way that I can now rely on my infrastructure to automatically deploy a new version once someone commits into my repository… which is neat! Automation is good and actually fits into my workflow as well. So, we will see how things go. I will just keep on doing my silly littles things, and you can keep on reading them. oh but actually Using Zola introduced a couple of changes: For breaking changes, the only thing I could not fix fully were RSS feed URLs. I added a redirection rule in my reverse proxy to redirect the old paths, but I encourage you, potential reader of this stream, to change your URLs now</li> I have an about me</a> page now. It’s quite barren, but I think it gets through the major important points.</li> </ul> That’s it, see you around the web! Against Real Name Policies 2023-10-12T00:00:00+00:00 You may be more used to reading things related to technology on this website, and you might be thinking to yourself “well this ain’t it, chief”. You’re wrong. There have been recent discussions</a> in my online circles around the concept of “real-name policies”, which is a requirement within certain projects to provide your actual, real, legal, certified-100%-you name alongside your contribution. More generally, it is a requirement that people appear as a legally bound and identifiable identity within certain spaces and for certain purposes. It might make sense to you at first if you have never thought about it any further of course: sure, sometimes you need to be able to delegate legal responsibilities to a contributor to your project, especially when it comes to their work and contribution. By then, of course, you think, naively, you need to have a unique immutable legal certified gold-plated absolute name recognized by a state to refer to them. Can you sense I am exaggerating for a purpose? ##</a> Real Names do Not Exist</h2> As many people have pointed out, real names are just a fantasy. This might come as a shock to you, especially if your life revolves around using your name around for various purpose such as academic publication, contributions, administrative presentation, and such. Spoiler: your name is not any more real. It may happen that, if you happen to travel abroad, your name will be transliterated on your passport. Which one is real then, the name recognized by X embassy, or that of your country of origin? Country of origin? Okay, but then what if within that state, agencies cannot even agree on how to spell it? Even my own government has made multiple mistakes in registering various people I know at their birth… And what if you do not have a country of origin, or no paper from that place? Say you’ve been uprooted by war, famine, climate change? Records have been destroyed? People believe a lot of things about names. It’s like a lot of things. We believe falsehoods about names that do not pass the reality check</a> (that post is from 13 years ago), and people in computer science are especially bad in that regard. Now let’s introduce this: I have multiple names. People who know me as a silly fox online call me one thing, people who know me more personally online call me another, my friends IRL call me one of two names alternatively, my colleagues call me a third combination of first and last name, and the government might soon start to call me a fourth one if they aren’t a-holes about me asking them to. As a bonus fact, that is an especially complex situation to resolve when managing GPG keys, for example. In this situation, none of these names are in any way less “real” than others. Nor should any of them be recognized as any more important and true than any other. They’re all real, in a way, all destined to their individual community, and if your answer to that is to say “okay, but then what’s on your ID”, well, we’ve reached the real hard problem with “real-name policies”: Real-name policies drive away minorities whose “real name” is controversial or debatable in the eyes of the established norm If your policy requires that I reveal the male name I was given at birth, I will simply stop contributing to your project. I will also tell my other trans and non-binary friends that your project is not safe for them. From a utilitarian standpoint, I’d argue you will lose out on the contribution of very capable people. From a more general viewpoint, you contribute to a general climate of hostility and gatekeeping that alienates marginalized communities from your projects, meaning the same demographics of western cisgender (often men & white) people keep the helm of projects that could all benefit from diverse input. And beyond people who change name because of a profound experience of self realization, those whose “real-name” is controversial will often be racial minorities, migrants, and such. They will also be caught in this. ##</a> So?</h2> I’m the kind of person who thinks there shouldn’t even be such a concept as legal name. Legality is a collection of meaningless decorum we act through to keep up the collective hallucination of our society. That’s confusing enough of a take as is, but you can read between the lines to try and gather what it means. In short? Don’t require people to disclose their “real” name when contributing to projects, whatever you think that means. Let people use nicknames, change names. Heck, let them edit their names in previous contributions as well (I have notable gripes about the way tools like git(1) handle names as a permanent record; once again, a tool which design could have benefited from input from other people; and it’s not the only one). Here’s a couple of issues I think people could have: “But what if they submit something illegal” then you take the hit. Copyright issue? My answer is “fuck copyright”, but a more practical one for you would be: have people handle those issues and remove offending things if needed. Malicious code? Same. It doesn’t scale? Find a way to make it so that you don’t need people to disclose their deadnames or argue with you for hours about their IDs</li> “But what if they act maliciously as a result? Doesn’t anonymity foster antisocial behavior?” wow there’s a lot to unpack there, but in short: people are going to stick to a name that’s recognizable (heck, they might even have a little piece of government-issued plastic with that actual name on it!), such that they are going to build a reputation just the same way they would on social media or, very likely, in your organization chats. Yet, sometimes, that name could change. Or it could be used only in your social circles. Or maybe not! Identities are complex, and it blows my mind to recall that most people just have not grasped that fact, or never had to.</li> </ul> So, yeah, fuck real name policies, in short. The SNCF WiFi API and All of its Funny Things 2023-10-12T00:00:00+00:00 Recently I had the unfortunate displeasure of having to move my vacation dates because of a certain pandemic that, despite the silence of our health authorities</a>, is still ongoing and impacting people. So instead of taking a crammed bus without ventilation for nine hours, my roommates and I got last minute tickets on the french high-speed railroad train: the Train à Grande Vitesse</a> (High-Speed Train). Last time I took that train I realized there was something funny that I could play with in the train: its internet connection. ##</a> The WiFi</h2> If you are on a TGV that has WiFi you gain access to what the Société Nationale des Chemins de fers Français (SNCF)</a> calls “WiFi SNCF”, which is essentially an on-board WiFi service. The SNCF engineered a system of repeaters that allow their trains to maintain a (somewhat) constant connection to a cellular data-like network. It mostly works. Of course, even they recognize and advertise that if you plan to use it for very bandwidth-intensive operations that need a fairly reliable connection, you should just wait. That serves them as well, to be honest, because they have to manage an internal network of many devices all going through a fairly tiny gateway. On my first train that day, we reached more than 240 devices connected in the train. How do I know that? I did not learn that from a ping scan. The WiFi API told me. ##</a> Funny Endpoints</h2> I already knew that there were web API endpoints you could call on the local user-friendly dashboard to the WiFi, accessible on every train at wifi.sncf</code>. Me and a friend</a> planned on trying to inspect as many of them as possible during that trip (to kill time however we could). Booting up developer tools on our web browsers, we noticed more than a handful of little queries that returned nice-looking JSON-encoded objects, and we dug deeper. What we found was: Trip details and updates</li> WiFi metrics</li> Information used on the dashboard trip map</li> GPS position and heading</li> Various media</li> </ul> So I’m going to go through them from least interesting to most interesting. The full list of endpoints is: /router/api/bar/attendance</code></li> /router/api/chat/room</code></li> /router/api/configuration/modules</code></li> /router/api/connection/statistics</code></li> /router/api/media/videos</code></li> /router/api/poi</code></li> /router/api/train/coverage</code></del> (no longer exists as of 2025-09-24</code>)</li> /router/api/train/details</code>, or /router/api/train/progress</code> (they are, as far as I can remember, the same)</li> /router/api/train/gps</code></li> </ul> ###</a> “Meh” Endpoints</h3> /router/api/chat/room</code> was an endpoint that was unexpectedly disappointing. If memory serves me right, it only provided very simple descriptions of buttons on the on-board assistance chat service, which were not interesting to me. Same goes for /router/api/configuration/modules</code>, I did not even save it, and I cannot remember what its purpose was. Whenever I take a TGV again, I will have to dig again. I am confident, however, that if past me did not save them, then they had a good reason not to do so, as they were probably fairly useless when it comes to extracting data from the portal. ###</a> The Bar</h3> You can fetch an endpoint to know whether or not there is currently a queue at the on-train bar! It’s given to you by /api/router/bar/attendance</code>, and returns an object with a single boolean called isBarQueueEmpty</code> which tells you whether the bar is busy at the moment or not. It’s effectively useless. I am pretty sure it needs manual update from whoever behind the counter is running the bar, and they probably have other things to worry about. ###</a> The Media</h3> This might be the only category where I did not save any example of the JSON output. Essentially, the web portal contains little promotional videos the SNCF made to showcase advice to passengers and such. It’s mostly promotional and boring. It has little gems here and there</a> however, but I did not bother viewing it all. I did, however, bother downloading all 4.3GB of it: Once I’m done inspecting it beyond making sure all files are valid video files and thumbnails, I will think about archiving them on archive.org</a>. The precise endpoint that delivers all paths and languages available for videos is https://wifi.sncf/router/api/media/videos</code>. Videos are grouped in categories (inoui-coulisses</code>, inoui-idees</code>, inoui-pub</code>, tgv</code>), and available in multiple languages (I found DE</code>, IT</code>, ES</code>, EN</code> and of course FR</code>). Or at least, they should be: from what I saw while inspecting the videos, all of them have audio tracks in french. ###</a> WiFi Metrics</h3> In the settings area of the WiFi portal, you have a section that gives you an indication of your signal strength and the count of devices currently connected from anywhere else on the train. Spoiler: it’s also pulled from a readily accessible endpoint called /router/api/connection/statistics</code>. The data it returns is simple: { "quality": 5, "devices": 246 }</code></pre> .quality</code> seems to be an integer on an unknown scale that tells you the quality of the internet access at the moment, and .devices</code>, fairly explicitly, tells you about the total number of devices connected. Because it seems that every single one of them gets a unique /21</code> subnet on the train, and that inter-subnetwork filtering was in place, I did not manage to find any device on my subnetwork other than myself and the gateway. I have to commend the SNCF, however, on being able to handle that many machines being connected at the same time. Hopefully the end terminals do not use as much bandwidth as, let’s say, a League of Legends player at a tournament event (sigh). ###</a> Maps, POIs, GPS</h3> On the WiFi portal, there is a nice little map of your trip. For example, one of my trains had this trip: The map contains several interesting things: POIs of notable sightseeing points around the map, the precise routing scheduled to be taken by the train, all the stations you will go through, and your progress. I will keep the progress for later, because I had quite a lot of fun with it (that was sort of my primary goal ever since I heard about the WiFi API). First, the API gives you exact positioning when available. The endpoint /router/api/train/gps</code> provides a nice little JSON that looks like this: { "success": true, "fix": 8, "timestamp": 1692349499, "latitude": 45.226723333, "longitude": -0.150826667, "altitude": 76.72, "speed": 83.906, "heading": 26.2 }</code></pre> You’ve got everything you may want. Exact GPS coordinates (longitude, latitude, altitude), heading, speed, a timestamp, and a GPS fix</a>, which seems to essentially be an identifier for which method was used to measure the position. That detail can help getting a sense of the precision provided at a given time by the measurement method. Later Addition: as a kind reader</a> pointed out to me by email, the speed is provided in meters per second. It seemed obvious to me at the time, but out of context it’s hard to surmise. At the same time, the map displays a lot of POI icons with landmarks around the area you travel through, and more. All of these POIs are available in /router/api/poi</code>, which returns you a really huge object with entries in this format: { "id": 20936, "title": "Abbaye des Prémontrés en Lorraine", "button": "Voir plus", "latitude": 48.907502, "longitude": 6.057966, "image": "/poi/images/20936/small.jpg", "isEvent": false, "visibleWhenZoomOut": false, "article": { "header": { "header_title": "Abbaye des Prémontrés en Lorraine", "image": "/poi/images/20936/big.jpg", "legend_image": "Abbaye de Prémontré en Lorraine © Brigitte Merle / Photononstop", "department": "France > Grand Est > Meurthe-et-Moselle > Pont-à-Mousson" }, "content": { "title_POI": "Entre Metz et Nancy, en Lorraine, la ville de Pont-à-Mousson abrite une abbaye du début du XVIIIe siècle, classée monument historique. L’abbaye des Prémontrés accueille un centre culturel, se visite et propose même des chambres pour y pas [...]", "title": "Abbaye des Prémontrés en Lorraine", "text": " <h2>Explorez l'abbaye des Prémontés :</h2><h2>L'histoire de l'abbaye</h2>Bâtie au début du XVIIIe siècle, l’abbaye a connu des périodes de faste mais aussi d’abandon. Incendie, révolution, réquisition, encore incendie…elle a été détruite, pillée, vidée, réquisitionnée pour être transformée en hôpital > }, "footer": { "author": "Coralie Salvi Rédaction SNCF Connect", "update_article": "2022-10-14" } } },</code></pre> For those who do not read french, you have: identifier, title, the name of a button to “see more”, a small and large thumbnail, and a bunch of HTML to describe the POI (in .content</code>). You also get to see credits for the photographs, and the person who entered this entry into the database of SNCF (thanks Coralie). These POIs can also seemingly be events, which I have never seen, but I can imagine the purpose of, if a commercial event is happening and you want to show it on your map. As for the thumbnails? Of course I downloaded the whole 35MB or so of them. Finally, there is /router/api/train/coverage</code>, which is kind of bizarre. It’s a collection of series of GPS coordinates: { "type": "FeatureCollection", "features": [ { "type": "Feature", "geometry": { "type": "LineString", "coordinates": [ [ 1.454419, 43.611391 ], [ 1.454473, 43.611895 ], [ 1.454464, 43.612209 ], [ 1.454158, 43.612884 ], ... ] }, "properties": { "quality": "small" } }, { "type": "Feature", "geometry": { "type": "LineString", "coordinates": [ [ 1.4384, 43.635599 ], [ 1.43475, 43.639033 ], [ 1.434588, 43.639303 ], [ // ... ] }, "properties": { "quality": "small" } } ] }</code></pre> I will be honest, I don’t really know what to do about it. My guess is that it draws the path the train takes on the map? I assume so, at least. Later addition: this was the single known API endpoint i could fetch again in September 2024. i cannot say if it was removed or if it was broken for my particular trip, but it really doesn’t help me understand its purpose. Later later addition: Confirmed in September 2025 that this endpoint no longer exists. ###</a> Trip Details</h3> This is the most interesting and exploitable part of the data the API hands to you. My end goal was to produce a tool that could help me track the progress of my train across its trip while remaining in the terminal. Of course, I did it</a> after a ton of jq</code> manipulation</a>. The original payload looks like this: { "number": "8504", "carrier": "INOUI", "events": [], "onboardServices": [ "OCEHP", "OCEVP", "OCECM", "OCEUB", "OCEBA", "OCEWF", "OCEPP" ], "additionalServices": {}, "stops": [ { "code": "FRXYT", "label": "Toulouse Matabiau", "services": { "DRIVER": true }, "coordinates": { "latitude": 43.611206, "longitude": 1.453616 }, "progress": { "progressPercentage": 100, "traveledDistance": 45628.872278306364, "remainingDistance": 0 }, "theoricDate": "2023-08-18T06:28:00.000Z", "realDate": "2023-08-18T06:28:00.000Z", "isRemoved": false, "isCreated": false, "isDiversion": false, "delay": 0, "isDelayed": false, "duration": 0 }, { "code": "FRXMW", "label": "Montauban Ville Bourbon", "services": { "DRIVER": true }, "coordinates": { "latitude": 44.01444, "longitude": 1.341499 }, "progress": { "progressPercentage": 34.13992504047737, "traveledDistance": 20981.52831506503, "remainingDistance": 40475.92447719702 }, "theoricDate": "2023-08-18T06:54:00.000Z", "realDate": "2023-08-18T06:54:00.000Z", "isRemoved": false, "isCreated": false, "isDiversion": false, "delay": 0, "isDelayed": false, "duration": 3 }, { "code": "FRAGF", "label": "Agen", "platform": "2", "services": { "DRIVER": true }, "coordinates": { "latitude": 44.207967, "longitude": 0.620867 }, "progress": { "progressPercentage": 0, "traveledDistance": 0, "remainingDistance": 115660.51721035445 }, "theoricDate": "2023-08-18T07:32:00.000Z", "realDate": "2023-08-18T07:32:00.000Z", "isRemoved": false, "isCreated": false, "isDiversion": false, "delay": 0, "isDelayed": false, "duration": 3 }, { "code": "FRBOJ", "label": "Bordeaux Saint-Jean", "services": { "DRIVER": true }, "coordinates": { "latitude": 44.825873, "longitude": -0.556697 }, "progress": { "progressPercentage": 0, "traveledDistance": 0, "remainingDistance": 496038.162231625 }, "theoricDate": "2023-08-18T08:40:00.000Z", "realDate": "2023-08-18T08:40:00.000Z", "isRemoved": false, "isCreated": false, "isDiversion": false, "delay": 0, "isDelayed": false, "duration": 6 }, { "code": "FRPMO", "label": "Paris - Montparnasse - Hall 1 & 2", "services": { "DRIVER": true }, "coordinates": { "latitude": 48.841172, "longitude": 2.320514 }, "theoricDate": "2023-08-18T10:52:30.000Z", "realDate": "2023-08-18T10:52:30.000Z", "isRemoved": false, "isCreated": false, "isDiversion": false, "delay": 0, "isDelayed": false, "duration": 0 } ], "trainId": "882" }</code></pre> It’s quite intricate, but essentially contains: Information about services on-board (codes that I haven’t cracked in .onboardServices</code></li> Information about the various stops in .stops</code></li> Lengths and estimated distance across stops (.stops[..].progress</code>)</li> An identifier for your train (.trainId</code>), mine was 882</code> that day!</li> </ol> Now, each stop is a little intricate as a data structure. You get the hilarious confirmation in .services.DRIVER</code> that you actually have someone to drive the train at each station, which is cool. There are details about the station, like a unique code, its human-readable name, coordinate, and such. You have date information, the time you were theoretically supposed to arrive at/leave a station, and the actual time you did. Finally, you get some booleans that inform you of changes that might have happened to your original route, and some delay information. For me, when traveling across the segment between Agen and Bordeaux Saint-Jean, the details data suddenly foretold 5 minutes of .delay</code> due to “Large attendance” or something (the station was packed and busy, essentially). ####</a> Progress Details</h4> And then we get to the details, which do not work exactly as you would expect. Each stop contains the progress information from its start to the next. You can therefore identify the last stop of your trip not only by the fact it is last in your .stops</code> but also because it does not have any .progress</code> data. You have three values for each segment X -> Y</code>: .traveledDistance</code>, which is the length you have already traveled along the segment, in meters</li> .remainingDistance</code>, the length that remains on the segment, in meters</li> .progressPercentage</code>, the result of (.traveledDistance)/(.traveledDistance + .remainingDistance) * 100</code></li> </ul> This particular format is probably fairly modular when it comes to suddenly modifying trips on the fly in case a train needs to be rerouted. Since your trip is only divided into segments, you can re-compute the total distance traveled and such from individual segments in case of modification. It was a bit more painful to script for me however, because that meant a lot of logic had to go into summing up the lengths traveled, the lengths remaining, and compute the final ratio myself ####</a> Scripting the Details</h4> Obtaining a percentage of your trip is done by summing up the distance traveled across all segments with the length remaining, and doing a ratio. Once you have a percentage of your position, the rest is simple arithmetic, which leads to this atrocity of a one-liner: curl -sL "https://wifi.sncf/router/api/train/details" \ | jq -Mc '.stops | map(select (.progress != null)) | [ (map(.progress.traveledDistance) | add), (map(.progress.remainingDistance) | add)] | if (.[0] + .[1]) then "\(.[0] / (.[0] + .[1]) * 100)%" else "EEE.EE%" end'</code></pre> which will show you the progress in 3.2f</code> format. Also, yes, that is almost purely arithmetic done in jq</code>. I learned a lot about jq</code> during my train trip. ##</a> Later Additions</h2> By the time i am writing this section it has been almost a year since this i published the first version of this blog post, which seems to have garnered a lot of attention (in the form of multiple emails of people either thanking for this information, and pointing me to tools they made to use it or pointing out information i missed the first time). One later addition i can make which fits nowhere is that when the train hasn’t started, the only reply from the query is: { "trainId": "XXX" }</code></pre> Where XXX</code> is the train identifier (for example, 882</code>). ##</a> In Conclusion</h2> It took a while for me to write this. This should come out some time in mid-October, and I was on the train in mid-August. Life’s moving fast, I had enough time to even begin my PhD, so that’s that. As to what I take away from this? Nobody is going to bother you for grabbing data that is freely accessible</li> Make weird tools with it! Who cares! Make cute GUIs and TUIs with train emoji 🚂!</li> Document everything you find, and everything you reverse. If it’s not useful to you, it could be to someone else.</li> </ul> Rust for Linux Basics: Handling Function Pointers 2023-10-09T00:00:00+00:00 Note: I originally wrote this as a message in the RFL Zulip chat</a> answering a question from someone who asked about ways to handle function points in Rust for Linux, especially in the context of net</code>, where we do it quite often. This blog post is a slightly refined version of that message, where I got carried away and essentially infodumped on the poor fella. I am a silly fox 🦊 Fun fact! A while back I originally planned to write a blog post about the very process of me figuring this whole thing out again from scratch, which I accidentally published for a couple days while unfinished. Oops. Here’s a little explainer on how I handled pointers. It’s not far from what I have seen others do, and there might be things that are considered to be common patterns/good practices that I sidestepped/skipped. I’m just learning. Thankfully, I had to explain this very thing many times to a handful of people, so it should be somewhat structured. I’ll take the example of my NAPI structure</a>, as a way of implementing one function pointer needed in a structure. From my experience of moving other function pointers around (like struct net_device</code>’s priv_destructor</code>, in groups/descriptors such as RTNL, and such), it doesn’t really change too much. It’s going to be a lot of things you might already know or find obvious because I prefer being thorough and verbose rather than accidentally missing something critical, and it might be useful for people coming at this with very little familiarity with the project. The general idea is going to be: Create a trait definition marked #[vtable]</code> that contains the Rust function(s) you want to implement, with all types in the return value and arguments replaced with their Rust counterparts</li> Implement that trait on a structure that will define your function for your use case</li> Have a second type that acts like a factory, and is generic on any type that implements your trait</li> The factory type is going to contain the static definition of the final pointer/series of pointers you will use, it decides which ones are defined thanks to booleans created at the implementation step of a #[vtable]</code> trait</li> The functions being pointed to will wrap around your actual implementation, receiving arguments with binding types, and converting them, hiding away all of the unsafe {}</code> from the driver user, before calling your implementation with the abstracted arguments. It does the reverse operation for any return value your Rust implementation gives</li> When you need the pointer in Rust code, you can call a method that returns a reference to the static stored for the specific version of your factory type for the trait of the implementer, and have an abstraction wrapper set it</li> </ul> ####</a> Step 0: Prerequisites</h4> You should identify the types of your function arguments and return type on the C side. Prepare abstractions for those if not already. For the case of napi polling</a>, you get a struct napi_struct*</code> and an int</code>, and return an int</code>. Because I was already writing an abstraction for using struct napi_struct</code>, I was all good, and could convert back and forth from struct napi_struct*</code> and &Napi</code>/&mut Napi</code>. That return int</code> is not meant to be signaling errors as far as I know, only the number of packets received, so the stack can know how many packets were received in that poll run. ####</a> Step 1: Making a Trait</h4> The #[vtable]</code></a> macro was created for these kinds of scenarios where you need to have a sort of “table” of function definitions (on the Rust side, on the C side it’s pointers), and know which ones have been defined by an implementation. Say I have: /// NAPI poller definition trait #[vtable] pub trait NapiPoller { /// Implementation of the poll function fn poll(_napi: &mut Napi, _budget: i32) -> i32 { 0 // Not implemented } }</code></pre> (actual version here)</a> ####</a> Step 2: Make a builder</h4> The builder/factory is going to hold your pointer definition(s) and give you static methods to retrieve it. In my case I even defined the exact type of the poller function as a type</code> because it was just a bit too annoying to type out in its entirety. /// Building structure for poller functions pub struct PollerBuilder<T: NapiPoller> { _p: PhantomData<T>, } type PollerFunction = unsafe extern "C" fn(*mut bindings::napi_struct, i32) -> i32;</code></pre> You’ve probably seen it already, but function pointers on the Rust side are Option<unsafe extern "C" fn(A....) -> B></code>, and we keep them at None</code> if nothing is provided. The builder implementation is valid for any T: NapiPoller</code>, and it holds my static function definition: impl<T: NapiPoller> PollerBuilder<T> { const FUNC: Option<PollerFunction> = <T>::HAS_POLLER.then_some(Self::poller_callback); /// Access the function pointer pub const fn build_function() -> Option<PollerFunction> { Self::FUNC } unsafe extern "C" fn poller_callback(napi: *mut bindings::napi_struct, budget: i32) -> i32 { /// SAFETY: <cut for brevity> let napi: &mut Napi = unsafe { &mut *napi.cast() }; // The rest is primitive, hence, trivial <T>::poll(napi, budget) } }</code></pre> At the time I did my version</a>, I forgot to check <T>::HAS_POLL</code>, and bool::then_some</code></a> was only recently stable, and I did not use it. ####</a> Step 3: Implement in user and abstractions</h4> Let’s say I have my drive, and it needs to have a definition of NAPI polling</a>. You put whatever you want in it. As a fun thought experiment, if you do not provide an implementation of poll</code>, you can imagine that the factory structure will contain None</code> in its const field, and that your assignment will be to a null</code> function pointer. That fits the semantics of what we want if we do not define a field in a descriptor/set a function pointer to null from a lack of implementation. Oh and your implementation also has to be marked #[vtable]</code> so it can process it and add the booleans indicating whether you overwrote functions or not. I really wish we had some way to do that in the core language, but I think we’re in such a specific use case that it might never come to core</code>. Now, for my poller, I needed an additional function that sets the poll</code> field of struct napi_struct</code> through my abstraction, so I wrote the wrapper for it</a> (in my net::Device</code>, which I guess made sense at the time), and this neat little trick I found helps me pass the trait implementer type for my implementation of poll</code> to the function: pub fn napi_add<POLLER: gro::NapiPoller>(&mut self, napi: &mut gro::Napi) { // ... conversion stuff // Use our factory type on the type provided to retrieve its implementation as a const pointer // that way we can pass it around on the C side and it will remain valid, and hide all of the // wrapping from the Rust user let poll_func = gro::PollerBuilder::<POLLER>::build_function(); /// SAFETY: .... unsafe { bindings::netif_napi_add(dev_ptr, napi_ptr, poll_func) };</code></pre> And in my driver: dev.napi_add::<MyNapiPoller>(&mut napi);</code></pre>####</a> Limitations</h4> This is only an example but it outlines the general steps I undertook for RTNL, NAPI, and various other function pointers. I got them from looking at code others made, and things like #[vtable]</code> make me think this general structure is intended to be used, at least for now. In terms of scaling? I haven’t evaluated it, but I assumed that changing function pointers on the fly is going to be a bit of a bother. Hopefully you do not actually need to have twenty of fifty different implementations, and you only have to implement one trait in the user. On the abstraction side, however, it gets problematic. If you are talking about descriptors/structures of many pointers, doing housekeeping to keep the names up to date is necessary (I believe this has been discussed in the recent proposal in netdev</code>). Furthermore, it’s still a lot of boilerplate code for one function pointer, and if you do not have any backing structure, you may need to spend a lot of time making structures that are needed just to handle that function pointer. Ah and don’t forget: my code hasn’t been reviewed by people on here. Handle it with an amount of salt typically reserved for a profoundly bland dish. That’s all I can think of for now, I might have gone a bit overboard with documenting this (I like writing documentation). None of this is proofread either, it’s all just pure stream of thoughts, I hope it made sense at all. Hopefully it was useful in some way, and I will try and hang around to answer questions/fix issues people could raise if I said something wrong. Rust for Linux Basics: Adding Allowed Features 2023-09-10T00:00:00+00:00 In order to share some of the things I’ve been doing over Rust for Linux</a>, I decided I would start a small series of tiny posts that demystify and detail very simple operations that can be confusing when doing development for RFL. ##</a> Rust Features</h2> Rust features go through a lifecycle where they can begin as “unstable”, and then eventually stabilized once Rust maintainers can guarantee that it will henceforth be backwards compatible, and free of memory errors (if unsafe {}</code> is used). Our example here is Box::try_new</code></a>, which is still nightly/experimental. Unstable features can be used in library crates if you declare that a module is allowed to use anything behind their feature gate. Box::try_new</code> is gated behind the allocator_api</code> feature, meaning that if you want to use it, you need to declare that you use it at the root module of your library crate: //! My library crate #![feature(allocator_api)] // You also cannot use that import unless you unlock the `allocator_api` feature gate use core::alloc::AllocError; pub fn make_a_new_box() -> Result<Box<Vec<i32>>, AllocError> { Box::try_new(Vec::new()) }</code></pre> Of course, you also need a nightly toolchain that allows you to have said feature. What of binary crates? Well, not only do you need to use a nightly toolchain, you also need to explicitly declare in the command line that you’re allowing a feature</a>. For a binary crate, the list of allowed features is enabled, so you do not need the feature</code> attribute. ##</a> Back to Rust for Linux</h2> So, why does this matter to RFL? In the kernel, allocation is never guaranteed. You are in the kernel, so you are the huge program managing the memory, and, sometimes, there is no memory left, and you should be able to handle that (usually by gracefully unwinding and destroying all your resources, or aborting a callback, or whatever). So, if you want to Box</code> something, you should use try_new</code> (in fact, Box::new</code> does not exist in RFL). Of course, if you’re going to do special things with that Box</code>, you may use other methods, but let’s pretend we just want to try_new</code>. As of the writing of this blog post, if you pull the rust-next</code></a> branch and try to write a module with a Box::try_new</code>, it will fail. Adding the feature</code> attribute will not help of course, because modules are not library crates. You need to add the allocator_api</code> to the list of allow_features</code> in the command line. Thankfully, we are allowed to do that</a> (see Unstable features stabilized</code>) for new modules that can be upstreamed. For example, in the null block driver patchset</a>, Andreas Hindborg adds the allocator_api</code> so he can use Box::try_new</code> as well. The precise file to add that is scripts/Makefile.build</code></a>: # Compile Rust sources (.rs) # --------------------------------------------------------------------------- rust_allowed_features := new_uninit,allocator_api</code></pre> With that, you have a new allowed feature available. OpenStreetMap & OAuth2: Do not Neglect x-www-form-urlencoded 2023-08-15T00:00:00+00:00 A friend</a> and I were out editing for OpenStreetMap</a>, and we wondered how we could tweak an existing open changeset</a>. Because the client we used</a> does not seem to provide that feature, and we are messy gremlins who like to mess around with data to do things (like adding review_requested=yes</code> to a changeset manually). So we defaulted to the next most reasonable thing to use on our phones in the middle of the street: curl</code>. So I decided to finally figure out how OAuth2 works, but also fell into the trap of forgetting that application/x-www-form-urlencoded</code> is a thing. ##</a> OAuth2 for OSM</h2> For the purpose of the experiment, I created a changeset with a change I had not uploaded yet</a>, and decided to play around with the API. OSM uses HTTP Basic Authentication</a>, so I needed a token. The fastest way to get one seemed to be getting it through OAuth2</a>. For that, I first registered the application</a>. Note that the $REDIRECT_URI</code> you provide will be useful later. It won’t really serve any technical purpose (the authorization API just redirects you to that URI with the code you get), but if you’re missing it/not providing the same one given at registration, you will get errors. With registration done you get two components: $CLIENT_ID</code> and $CLIENT_SECRET</code>. Next, open https://www.openstreetmap.org/oauth2/authorize?client_id=$CLIENT_ID&redirect_uri=$REDIRECT_URI&response_type=code&scope=write_api</code></pre> in your browser while logged in to OSM. It’ll give you the usual “do you want to authorize this app” spiel, and you press yes. Here the scope of authorizations used here means you can only write stuff through the map API. After authorization you will be redirected to $REDIRECT_URI?code=$CODE</code>. Get that code, and your next step is: curl -v -X POST -L 'https://www.openstreetmap.org/oauth2/token' \ --data-urlencode "client_id=$CLIENT_ID" \ --data-urlencode "redirect_uri=$REDIRECT_URI" \ --data-urlencode "grant_type=authorization_code" \ --data-urlencode "code=$CODE" \ --data-urlencode "client_secret=$CLIENT_SECRET"</code></pre> And if you are lucky, you get: {"access_token":"[REDACTED]","token_type":"Bearer","scope":"write_api","created_at":[REDACTED]}</code></pre> Hurray! But I was not so lucky. It took me an hour and several</a> unhelpful</a> online</a> pages</a> to even get the code in the first place. OpenStreetMap automatically closes changesets after one hour of inactivity, meaning mine was gone. My friend was also tweaking things at the same time. She was surprised I got the code, so I gave her the mechanism and explained what I had tried so far, and moved on. I had tried a lot of things. For some reason, some of them made OpenStreetMap spit out errors in korean</a> or spanish</a>. My friend tinkered and came back with a functional method to get the bearer token. We had both found two important things: The order of query parameters in the authorization URL was important (it’s 2023…)</li> The token</code> URL accepted neither application/json</code>, query params, or text/xml</code> for its input. After a while, I wondered if curl</code> sending a Content-Type: application/x-www-form-urlencoded</code> header even without a body of data would be the problem, but removing that header also proved useless. My friend found out that you needed to provide parameters as separate fields through --data-urlencode</code> (and we are still not sure the order does not matter).</li> </ol> Eventually, after a good night’s sleep, I got my token, roughly 14 hours, including a good night’s sleep, after my changeset closed. At least, next time, I know how I can tinker with the API (albeit, perhaps, not the production API</a>). Merging the NFTables NAT Tables 2023-06-23T00:00:00+00:00 Recently, a new Debian happened</a>. At this point, I am used to upgrading my machines whenever that happens. I had been hoarding Debian systems here and there since I first installed a Debian Jessie back in the day. Among the thousand of packages being upgraded, there was nftables</code> and iptables</code>. A friend of mine</a> had warned me she had heard about changes done to nftables</code></a>, though they would likely be backwards compatible. So, I confidently upgraded. It turns out after rebooting my system and re-reunning my services that every single unit which needed to insert things into nat</code>’s POSTROUTING</code> chain could not. Specifically, I got hit with the following error from my docker service: Jun 23 11:06:59 <HOSTNAME> dockerd[40298]: time="2023-06-23T11:06:59.118341802+02:00" level=warning msg="could not create bridge network for id d1139036d539117fca5f7c7301a9c38e1da1c725534246506364d0fd09c26c6f bridge name br-d1139036d539 while booting up from persistent state: Failed to program NAT chain: Failed to inject DOCKER in PREROUTING chain: iptables failed: iptables --wait -t nat -A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER: iptables v1.8.9 (nf_tables): CHAIN_ADD failed (File exists): chain PREROUTING\n (exit status 4)"</code></pre> Docker’s integration with iptables and nftables has always been shoddy, so I wondered if it was a new thing that they had broken. I was less confident in that hypothesis when I realized my VPN tunnels had also broken (alongside every single unit that bound to its interface’s IPv4 address), because the script that sets up post-routing rules had also spit out the same errors. ##</a> The Reason</h2> What happens apparently is that, now, when iptables</code> is used to insert nat</code> rules, it tries to insert them in the table called inet nat</code>. In nftables</code>, tables can be split by network protocol (inet</code>, ipv4</code> and ipv6</code>). I used to have two declarations of tables in my /etc/nftables.conf</code>, one of which looked like this: table ip nat { chain POSTROUTING { type nat hook postrouting priority 0; oifname "upstream" masquerade; } }</code></pre> I theorize that iptables</code> did not have any inet nat</code> table (because both ip nat</code> and ip6 nat</code> existed), so it tried to create one, which could not work, because inet</code> overlaps both ip</code> and ipv6</code>. Yet, there seems to be no fallback on trying to use the distinct nat</code> tables. So anyways, I fused both tables into inet nat</code> (and used that opportunity to change the priority of POSTROUTING</code> to srcnat</code>), which I believe was not possible before, but do not quote me on that. Rebooted, and everything suddenly worked. The Way I Setup My GPG Keys Now 2023-04-02T00:00:00+00:00 You may find it dirty. I just think it’s neat. </blockquote> Following a recent discussion with friends of mine who have made a habit of exchanging files in chats by encrypting them with GPG (for the fun (yes I have weird friends)) I thought I would make a little tutorial to explain the way that I generate and keep my keys now, because they told me they would be interested in reading that. ##</a> Reasoning</h2> The inspiration for this methodology came from another friend whose programming hobbies</a> are on the kind of scale that you need a PGP key to verify your identity (again, I have weird (but very cool) friends). The idea is to view your key as having multiple components: A central private key, which is the key used to certify that your identity/identities is tied to other keys, signatures, etc.</li> Sub-keys, certified with the private one, which are used for the day-to-day operations you perform with gpg</code>.</li> </ul> Side-note: I will do the best to distinguish PGP as a standard for keys, and the gpg</code> implementation. At some point though, I will only talk about how you can generate a similar setup as mine using gpg2</code> (specifically gpg2</code> version 2.2.41</code>). The idea is that your central private key should stay the same as long as possible. You will, of course, keep a revocation certificate on hand to be deployed at any time, in a printed sheet of paper slid in a vault in your home, in your bank, in a bunker 100 kilometers away, and so on. With your main key never changing, you will never change key ID, and, aside from having to reupload your whole key every now and then, you do not have to bother yourself with re-verifying your identities among your web of trust. To maintain the idea that you should not use keys eternally, notably so that attackers do not have time to attack it, you will make your sub-keys expire after a given period. Sub-keys have capabilities among three possible choices: Authenticate, Encrypt and Sign. When you do gpg --list-keys</code>, those are the meanings of the letters you see in brackets in the core key and the sub-keys. ##</a> Generating the Private Key</h2> The beginning of the whole process starts with generating the core key. I am also going to use Ed25519/Cv25519</a>, which is considered to be a more modern standard, needing less data and bits of entropy to remain secure. By default, gpg</code> does not give you the option to use Ed25519 when you generate keys. To have that option available, as well as tweak key capabilities, you need to add the --expert</code> flag to gpg</code>. In addition, to have the capabilities prompt, we need to use --full-generate-key</code>: ~$ gpg --expert --full-generate-key gpg (GnuPG) 2.2.41; Copyright (C) 2022 g10 Code GmbH This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Please select what kind of key you want: (1) RSA and RSA (default) (2) DSA and Elgamal (3) DSA (sign only) (4) RSA (sign only) (7) DSA (set your own capabilities) (8) RSA (set your own capabilities) (9) ECC and ECC (10) ECC (sign only) (11) ECC (set your own capabilities) (13) Existing key (14) Existing key from card Your selection? 11</code></pre> I select elliptic curve algorithms, but with the prompt to set my own capabilities. The core key should only be able to Certify, and that is what we will change these to: Possible actions for a ECDSA/EdDSA key: Sign Certify Authenticate Current allowed actions: Sign Certify (S) Toggle the sign capability (A) Toggle the authenticate capability (Q) Finished Your selection? S Possible actions for a ECDSA/EdDSA key: Sign Certify Authenticate Current allowed actions: Certify (S) Toggle the sign capability (A) Toggle the authenticate capability (Q) Finished Your selection? Q</code></pre> There are actually multiple elliptic curve algorithms, but I tend to prefer Ed25519. The debate as to which one you should chose is raging in the cryptography community. I would honestly recommend you check out people who know these things better than I do</a> for a more informed opinion. If you ask for my reasoning around using Ed25519… I like how it feels under my fingers when I keep typing it on my keyboard. There. Please select which elliptic curve you want: (1) Curve 25519 (3) NIST P-256 (4) NIST P-384 (5) NIST P-521 (6) Brainpool P-256 (7) Brainpool P-384 (8) Brainpool P-512 (9) secp256k1 Your selection? 1</code></pre> As discussed earlier, the core private key should stay valid as long as needed, and only die when you need it revoked for emergency reasons. Please specify how long the key should be valid. 0 = key does not expire <n> = key expires in n days <n>w = key expires in n weeks <n>m = key expires in n months <n>y = key expires in n years Key is valid for? (0) 0 Key does not expire at all Is this correct? (y/N) y</code></pre> And then you provide all of the usual information. gpg</code> will ask you for your name, email, comment, etc. Usual stuff. The revocation certificate is stored in a path given in the sea of text at the bottom. Remember to export the private secret of your core private key to a removable medium, etc etc (these are honestly things I do not personally do, but my threat model does not involve actors that powerful). ##</a> Adding the Other Sub-Keys</h2> Now, if you try and call gpg --list-keys</code> and check out your ID, you will notice you have your certification core key, and that’s it. Of course, this is not very useful. We are going to edit the key to add sub-keys that will let us do everything else we want. Once again, we need the --expert</code> flag or else gpg</code> will not give us the prompts we need. As an example, I will generate the Authentication key. All of the other keys are created the same way (except for the fact that the other two types of keys have a dedicated prompt so you don’t have to “set your own capabilities”: ~$ gpg --expert --edit-key ${ID} [snip] gpg> addkey Please select what kind of key you want: (3) DSA (sign only) (4) RSA (sign only) (5) Elgamal (encrypt only) (6) RSA (encrypt only) (7) DSA (set your own capabilities) (8) RSA (set your own capabilities) (10) ECC (sign only) (11) ECC (set your own capabilities) (12) ECC (encrypt only) (13) Existing key (14) Existing key from card Your selection? 11 Possible actions for a ECDSA/EdDSA key: Sign Authenticate Current allowed actions: Sign (S) Toggle the sign capability (A) Toggle the authenticate capability (Q) Finished Your selection? A Possible actions for a ECDSA/EdDSA key: Sign Authenticate Current allowed actions: Sign Authenticate (S) Toggle the sign capability (A) Toggle the authenticate capability (Q) Finished Your selection? S Possible actions for a ECDSA/EdDSA key: Sign Authenticate Current allowed actions: Authenticate (S) Toggle the sign capability (A) Toggle the authenticate capability (Q) Finished Your selection? Q Please select which elliptic curve you want: (1) Curve 25519 (3) NIST P-256 (4) NIST P-384 (5) NIST P-521 (6) Brainpool P-256 (7) Brainpool P-384 (8) Brainpool P-512 (9) secp256k1 Your selection? 1</code></pre> The sub-keys would not be eternal. In my setup, I would consider that an authentication key (which you can use if you wire up your GPG agent to serve as an SSH authentication again, something I may spend another blog post explaining) can be replaced every couple years, for example. So I would say that it should expire in two years. Of course, set your own limit based on your threat model and infrastructure. All other sub-keys would tend to last longer. In my setup, Signing and Encryption keys last 5 years. That is a long time, but, again, not a lot of people are interesting in busting up my stuff. Please specify how long the key should be valid. 0 = key does not expire <n> = key expires in n days <n>w = key expires in n weeks <n>m = key expires in n months <n>y = key expires in n years Key is valid for? (0) 2y Key expires at Tue 01 Apr 2025 01:55:05 CEST Is this correct? (y/N) y Really create? (y/N) y</code></pre>##</a> Contemplating the Result</h2> With all of the sub-keys created, you should have something like this: pub ed25519 2023-03-12 [C] <ID> uid [ultimate] Lux A. Phifollen (Limefox) <contact@vulpinecitrus.info> sub ed25519 2023-03-19 [S] [expires: 2028-03-17] sub cv25519 2023-03-19 [E] [expires: 2028-03-17] sub ed25519 2023-03-19 [A] [expires: 2025-03-18]</code></pre> Now, when you need to provide a key for signature and such, you can provide the main ID of the key. gpg</code> will look into the key and find the first sub-key that is capable of performing the operation you want from it (signing, encrypting; authentication is different but that is another blog post for later). If you happen to have multiple valid sub-keys that perform the same operation at some point, you can disambiguate which one should be used by providing the key-grip (identifier) of the individual sub-key, and not the main key. You can obtain it by listing the keys with --with-keygrip</code> in your options. And that’s it! You can honestly use this key pretty much the way you would use any other. It just feels more convenient, because it completely separates all of the operations you can use it for, giving you both the security that your identity is valid as long as your key is secure, the freedom to renew/destroy individual components, and the comfort of not having to renew your key (until you accidentally publish the private key/password somewhere, for example). ##</a> A Small Addendum</h2> But, wait, why are neither of your keys like that- </blockquote> Shush. I know. It’s not even “do as I say, not as I do”, it is legitimately just that I generated both of</a> my keys</a> a while back, before I knew as much about gpg</code>/PGP as I do now. They are, in fact, more of a legacy from a period where I mostly ““needed”“ (with very heavy quotes) them to sign commits on Gitlab/GitHub. Some day, I will officially retire them and start using a new one, that I already generated with this new methodology. I just have to figure out a couple things regarding user IDs (uid</code>s). As it turns out, it’s useful at times to have your professional identity untied from the fact you’re pretending to be a green anthropomorphic fox on the internet. VulpineCitrus is Changing CMS 2022-12-21T00:00:00+00:00 After 3 years of using Hexo</a> as my Content Management System for VulpineCitrus, I have finally decided to adopt a more complex system. I have been looking for decent CMS software for two years, as Hexo had become more of a technical debt than a software I could realistically keep maintaining in shape. Its theme was rewritten by me, and I had received no update to the software since 2019. While it rendered the entire website into static files based purely on Markdown files, Hexo was getting increasingly abandoned. Furthermore, the nature of its edition meant I could not have a good-looking web interface to edit new pages as I pleased, instead relying on the tedious process of writing a blog post on my local machine running a debug server. During my investigation I found Grav</a>, which seemed okay: it stores your site as flat markdown files, does not have ridiculously complicated features, and, while it runs on PHP7 at the moment, the developers also seem to take the security</a> of their software very seriously</a>. Transferring the old blog posts was okay. I took the freedom to edit some old dead links and fix a few issues here and there. Since Grav also uses Markdown files, I could import most of the blog posts. In some of them however, I used Hexo’s ridiculously convoluted {% %}</code> block syntax. As of the writing of this, I haven’t yet the reverse proxy of my VPS to point to this site, so it’s going to be a slow process. I might not even reverse-proxy it, since Grav does not stray far from the Nginx root directory it’s confined to, making it less of a pain to deploy. I do not know either whether the theme for this blog will remain the same. At the moment, I’m still testing things out. Only time can tell. And one last time, for the memories: the old VulpineCitrus theme. Hacking Linux: The TCP Stack and Initial Acknowledgement Number 2022-10-15T00:00:00+00:00 I like messing things up in networks. Sending weird packets into the wild and seeing what happens is quite funny, especially when something breaks, but, sometimes even, when nothing does. I like hacking in the primary sense of the word: messing around and using stuff that was not intended for that purpose. For a paper I was writing, I had to perform a simple packet analysis of a TCP conversation between a Linux client and a Linux server. I had to detail the way TCP uses sequence and acknowledgement numbers to make the transmission of data reliable. This exercise reminded me of a little fact I had noticed in the past but never really paid attention to: the first packet of a TCP conversation, if sent by a Linux host, will have an acknowledgement number of 0. This was somewhat unsurprising at first, and I remember noticing it in the past and never really minding it. After all, that number is completely discarded by the client once they receive the server’s acknowledgement number. I had always thought that perhaps, somewhere in the kernel source code, a structure for the TCP header was initialized with zeroes, and simply used to write the packet, and the word of data used for the acknowledgement number remained at zero. The paper I was writing had to be quite detailed, per the professor’s instructions. I decided to be a pain, and go absolutely too far in my explanations of why the first acknowledgement number sent by Linux is zero. As it turns out, the answer is not that hard, fairly easy to find, but it leads to very funny modifications that you may do if you feel like running a custom kernel. ##</a> Finding the Acknowledgement Number</h2> At first, I wanted to find where that ACK number was initialized. I had two theories: There is a structure somewhere that is null-initialized</li> There is a memory area created that is null-initialized</li> </ul> My initial investigation focused on the latter theory for way too long, completely missing the actual answer. After a while, I got there, using the first theory. For the sake of brevity, and because I want to focus this post on the stack itself, I chose not to do a complete addendum on the kernel’s memory allocation mechanism, and only focus on what I found after I had figured out the right track. For investigating kernel code, I will pull example code from the 6.0 release tag</a> of the Linux source code on Kernel dot org’s Git repository website. However, all of my actual investigation took place on the wonderful Bootlin Linux Source Code Cross Referencer</a>, targeting specifically the 6.0 tag</a>. ##</a> Knowing Where to Look</h2> I have at this point a bit of familiarity with code in the Linux kernel git tree, at least enough that I know where to look for something. If it’s a device driver for something non-specific (unlike NICs), look in drivers</code>. If it’s about the filesystem, see fs</code>. For networking, it’s in net</code>. It’s not rocket science. Inside of net</code> however you have distinct bits of code that do different things. Parts of it handles drivers for network devices (ethernet, network cards, WiFi, WWAN, SFP-based cards, etc), others handle the core generic network stack, or the ipv4</code>/ipv6</code> bits. It can get fuzzy at this point, but I have a good instinct regarding where to start at that point: struct tcphdr</code>. In the past I have messed around with TCP in Linux (specifically in XDP). I know the main protocol header structures fairly well (and, sadly, even those very useful ones</a> that are not exported in the UAPI), and one of them that is very useful in struct tcphdr</code></a>. I am therefore expecting it to be used and re-used throughout other parts of the Linux kernel, including whatever bit of the network stack that handles writing of the initial TCP packet. Searching on bootlin</a> shows me a lot of places to look through. I can categorize them thusly: Files in drivers</code> contains code that may handle parsing of the TCP header of incoming or outbound packets for various purposes (checksum offload/verification, for example)</li> Files in include</code> are mostly headers with other structures that contain/mention a struct tcphdr</code></li> Files in net/core</code> and net/ipv4</code> or net/ipv6</code> should be the ones I am looking for, as I expect the behaviour of the first ACK number to be generic across devices</li> </ul> The rest (tools</code>, samples</code>, net/netfilter</code> and such) should not be interesting here. ###</a> The TCP Files</h3> I first looked into the files for TCP IPv4, in net/ipv4</code>. I have previously found that some features that are implemented for both versions of IP in use sometimes rely on the same backing code that is in net/ipv4</code>. That was the case for tcp_v6_get_syncookie</code></a>, which, similarly to its IPv4 counterpart, uses tcp_get_syncookie_mss</code></a> that is defined in net/ipv4/tcp_input.c</code>. Speaking of these TCP files as well, we can see a few of them in net/ipv4</code>, notably net/ipv4/tcp_input.c</code>, net/ipv4/tcp.c</code> and net/ipv4/tcp_output.c</code>. Many other files exist, some for the different</a> congestion control</a> algorithms</a> of TCP, some for Berkley Packet Filtering Features</a>, or other weird features</a> of TCP. “But since that number should be chosen for an outbound packet..”, I thought, “.. maybe I should look in tcp_output.c</code>.” ###</a> TCP Output and SKB misadventures</h3> Inside of TCP output I find where that struct tcphdr</code> is used</a> inside __tcp_transmit_skb</code>, a function that seems to write data</a> from a socket buffer (SKB) to a struct tcphdr</code>. Notably, it sets the ack_seq</code> field of that structure</a> to a parameter rcv_nxt</code> that is passed as a function argument (after setting the correct endianness of course). This is the only place in that file where the ack_seq</code> field, which contains the acknowledgement number of the TCP packet which header is contained in the structure, is being changed, outside of an instance in tcp_make_synack</code></a>. That function name is fairly obvious, and we are not interested in that one. Climbing up from __tcp_transmit_skb</code> we try and find what calls it. One of the only two</a> functions that do (which are both in net/ipv4/tcp_output.c</code>, again) will also let me explain why the function name __tcp_transmit_skb</code> had me thinking I was on the right track immediately. This is tcp_transmit_skb</code>, from net/ipv4/tcp_output.c</code></a>: static int tcp_transmit_skb(struct sock *sk, struct sk_buff *skb, int clone_it, gfp_t gfp_mask) { return __tcp_transmit_skb(sk, skb, clone_it, gfp_mask, tcp_sk(sk)->rcv_nxt); }</code></pre> It’s a wrapper. One that extracts a value from something called sk</code>, and passes it to __tcp_transmit_skb</code> along with the other arguments. Yet, what is an SKB? SKB, or socket buffer, to make things very simple, is a data structure that keeps track of connection information and data. Its colleague the sock</code> structure is a very generic, and purposely opaque data type which is means to contain information generic enough that it can fit about any network connection in the Linux kernel. Yet, whenever you want to obtain more information, that structure lets you do what helps C bypass the absence of generics: casting. Calling tcp_sk</code></a> simply means force-casting the sock</code> structure pointer to a tcp_sock</code></a> structure pointer. This basically tells the compiler “it’s okay, from now on, handle this as though it was pointing to this structure”. That way, in parts where the connection socket must be handled in a generic manner for multiple protocols, you can keep using a struct sock*</code>, and you can specialize it for parts where you need more specific information (like when you are using TCP). The force-cast is the followed by a dereferencing read of the field rcv_nxt</code> of struct tcp_sock</code>, which seemingly contains the acknowledgement number to be sent. It makes sense then that when specializing our socket structure we look for that field, and it is later used as the TCP acknowledgement number in all packets, including the first one. So, what exactly calls tcp_transmit_skb</code>? Well, it is referenced 11 times</a> in net/ipv4/tcp_output.c</code>. I sift through those, and eventually find tcp_connect</code></a>, which is helpfully prefaced with the commend /* Build a SYN and send it off /*</code>. At that point, I knew I was in the right function. tcp_connect</code> does a couple big things before calling tcp_transmit_skb</code>: It calls the eBPF hook for a TCP socket being opened</li> It initializes the sock</code> structure later passed on to tcp_transmit_skb</code></li> It allocates memory for that socket’s buffer (this is where I first lost myself for way too long thinking the null ACK came from a null initialized buffer)</li> It performs a couple of checks additional checks (notably explicit congestion notice</a> initialization)</li> </ol> Let us focus on tcp_connect_init</code></a>, which sets up all the information possible for the socket that is independent of its address family. Within that method, we also force-cast the socket early on</a> into a struct tcp_sock</code>, which is used for the rest of the function. Looking at the rest of the code, it’s a lot of function setup, and field initialization. One field in particular rings a bell. Around the end of the function, protected by an if (related to a thing called TCP connection repair where a TCP connection is re-established on a new host as though it was continued from another host, which I’m not going to go on about), we find the initialization of rcv_nxt</code></a>, to the value 0</code>. Interestingly, that precise line of code hasn’t changed in 11 years</a>, so chances are the choice to use 0</code> for the first acknowledgement number is quite old. ##</a> Exploiting that Knowledge: A Custom Initial TCP ACK Number</h2> At first, I wrote my paper and kept this funny paragraph that went on a really deep tangent about the ACK number. I left it simmer in my head until a couple weeks later when a horrible intrusive thought suddenly struck me as I was compiling my own frankenstein kernel</a> mangled from the Rust for Linux</a> kernel (which I will talk about for a later post) and my own changes. What if I changed the ACK number? </blockquote> I sat in horror and thought to myself that surely something else in the stack of the kernel would prevent me from doing that. It’s a terrible idea from a privacy standpoint, since that would mark a packet circulating on the internet as exclusively coming from precisely my laptop. And yet, why not try? I sat home that day and opened the source code I was compiling earlier, went into net/ipv4/tcp_output.c</code> and edited the precise line I mentioned above. I thought to myself, “I’m doing something stupid, why not make it even worse”, and thought about a string of characters I could put. Originally, I went with :3</code> (two bytes, 0x3a33</code>). I later settled with OwO!</code> (4 bytes, 0x4f774f21</code>). I wrote that value, recompiled, installed, and rebooted. To my shock and horror, my first explicit TCP query once I had a terminal worked. Worse even, everything else that had been launched by default worked. All my TCP connections worked. I opened a shell, ran tcpdump</code>, filtered for tcp[13] & 0x12 == 0x02</code> (searching for TCP packets with ACK off and SYN on), and had the biggest bout of laughter in recent memories when I read OwO!</code> in my packets. It worked. It just worked. Apparently I hadn’t broken anything. No kernel errors in dmesg</code>. No idea if specific, weird situations were broken. My assumption that something could have broken because I was suddenly not using 0</code> was seemingly wrong, and I got to make myself and a couple of my friends laugh when I showed them. ##</a> Where to Go From Here?</h2> Talking with my friends, I discussed a couple of funny things I could do to hack my kernel further. I have always been fascinated by the interface exposed through /proc</code> that lets you configure kernel features dynamically from the filesystem, and I think I want to try and expose a custom endpoint /proc/sys/net/ipv4/tcp_initial_ack_number</code> where I can write whatever stuff that I want to use for that number, and have Linux use it. I also thought about the privacy aspects of using 0</code>. Since seemingly every Linux machine does, that protects us all in a way by making our connections impossible to fingerprint using that field. Me using a custom value completely undoes that, and makes me exactly identifiable, even through things like NAT. I thought about going a step further from a custom value, to a random one. Maybe I should work a little eBPF/XDP program I can hook on my RPI 4 home router that takes all of the incoming TCP packets that are purely SYN and randomizes their ACK number before forwarding? That could be interesting. There is a lot of ways I could further this, and lots of cool hacking stuff to do with the Linux kernel and its networking stack. Clippy: Obey the Paperclip 2022-09-28T00:00:00+00:00 I have been using Rust on and off for the past five years or so, first picking it up around the time release 1.0 dropped. I then seriously learned the language in 2019, and, thanks to coding events such as Advent of Code</a>, found a reason to make (arguably) beautiful</a> code</a> that trained me to make simple data structures work efficiently together to solve problems. Nowadays, I scale my development towards multi-threaded, asynchronous programs, in particular web server toys</a> and active projects (to be announced soon). My workflow in Rust has increasingly involved a tool that my peers find funny (for good reasons), and which has helped me tremendously when it came to writing Rust that works but also works well and is pleasant to read: pain. ##</a> Clippy</h2> Clippy</a> is Rust’s official linter program to “catch common mistakes and improve your Rust code”. Rust is, contrary to what people feel, extremely friendly towards new developers: the compiler spits out nice error messages with suggestions for fixes that are almost always right, the Book</a> is the greatest piece of documentation I have ever read, the community of developers is extremely friendly and ready to explain concepts that are indeed hard to grasp for a lot of developers at first, etc. Clippy can be installed into cargo</code>, Rust’s project management tool, using the rustup</a> toolchain manager. It integrates perfectly in the environment of other tools cargo provides, and it can be integrated in your linting routine to show you common mistakes. ###</a> The Lint Groups</h3> Clippy groups its linting rules into groups, or categories that you can find on their search engine</a> and allow/deny/warn as a whole. The most notable groups are: complexity: catches common errors that make your code needlessly complex for tasks that you are trying to make</li> correctness: catches code that is grammatically correct but results in nonsensical, typically unintended actions</li> deprecated: catches usage of deprecated methods</li> nursery: catches errors newcomers to Rust often make</li> pedantic: catches common errors that developers who are more familiar with Rust make that could be turned into better code</li> perf: catches resource-wasting patterns (creating default objects ahead, etc)</li> suspicious: catches code that looks suspiciously like the first step towards a bug or bad manipulation of the code base</li> </ul> Other categories exist, but these are the ones I think about the most. Another one I have to mention is restriction, which is hilarious to me. It is meant to restrict usage of patterns forbidden for a specific project (either because of style policies, or constraints). In fact, it is so thorough and unforgiving that you literally cannot act on it as a whole, since some of the lints in there are contradictory (see the AT&T</a> and Intel</a> asm flavour lints, for example). ##</a> Pain</h2> The seven groups I mentioned above, along with clippy::style</code> and clippy::cargo</code>, are completely denied in most, if not all of my projects. They will typically also include more lints from standard Rust. As of the writing of this post, my linting header is as follows: // Make clippy quite nasty #![deny(clippy::cargo)] // Checks for garbage in the Cargo TOML files #![deny(clippy::complexity)] // Checks for needlessly complex structures #![deny(clippy::correctness)] // Checks for common invalid usage and workarounds #![deny(clippy::nursery)] // Checks for things that are typically forgotten by learners #![deny(clippy::pedantic)] // Checks for mildly annoying comments it could make about your code #![deny(clippy::perf)] // Checks for inefficient ways to perform common tasks #![deny(clippy::style)] // Checks for inefficient styling of code #![deny(clippy::suspicious)] // Checks for potentially malicious behaviour // Add some new clippy lints #![deny(clippy::use_self)] // Checks for the use of a struct's name in its `impl` // Add some default lints #![warn(unused_variables)] // Checks for unused variables // Deny missing documentation #![deny(missing_docs)] #![deny(rustdoc::missing_crate_level_docs)]</code></pre> This is pain incarnate. People I talk with who are starting out in Rust would rather bang their head against a wall than try and satisfy such draconian requirements. By all means, I tell them, they could be far worse! Yet this approach forces you to write code that is not only valid, and correct, but good. I started out simply denying clippy::pedantic</code>, which is enough to get a good grasp of good practices in Rust. Once no lints typically fire after you are finished writing a piece of code, try and deny another category, like nursery</code>. Then, correctness</code>, or perf</code>. After a while, you will learn about the common mistakes you make, and recognize the patterns, and catch yourself in the act, before Clippy can yell at you. After that, learn about the mistakes you do from a new group! At some point you end up with code that is functional, efficient and good-looking. Rust code already looks beautiful in general, but good Rust code is pleasant to read (especially if you literally force yourself to write good documentation with the documentation lints). With time, you learn to love that nasty paperclip. 📎 The Weird Debugging Story of How I Wanted to Create a UEFI VM with Libvirt 2022-04-12T00:00:00+00:00 This post is mostly a story about my new VM server setup. However, it features a fun twist that reminds me how important it is to consider the layers upon layers of added security tools that exist in Linux that could interfere with a normal setup. ##</a> The Context</h2> Around a month and a half ago I got this new server in my infrastructure, an old decommissioned Dell Precision T1700</a>. I was pleasantly surprised to learn that it has an Intel Xeon with 8 cores at 3.5GHz (which makes it the most powerful server I own aside from my main workstation). I decided to move a couple of services off of my VPS, which had been struggling under the load that Matrix Synapse puts on it. So far, aside from the CPU spending roughly half of its time in I/O idle (I never replaced the hard drives in it, they have lived a good life, and I have no irreplaceable data on them, so I’m just waiting for a failure before buying new ones), the server is fine. It is so fine in fact that I wanted to add more onto the pile of things it would have to deal with. A week or so ago I was doing a lab at school where the teacher provided us with a Virtualbox Appliance VM image so we could have all of the tools and setup for the lab. Since I was running everything on my Thinkpad, which already had libvirt/virt-manager installed, I wanted to see if I could convert the OVA into QCow2</a> or better import the OVA as a VM</a>. ##</a> A Tangent on OVAs and Virt-Manager</h2> As it turns out, an OVA is simply a TAR archive with three files in it : the disk image, a XML file (something dot ovf) containing the VM settings, and a third on that is optional and which purpose I am not entirely sure of. Long story short I struggled the entire lab to try and make the VM work on QEMU with Libvirt, but failed. I ended up running VirtualBox for the parts I really needed to get running (I mostly listened to the lab, which was mostly a lesson, but I’m digressing). See, first of all, Libvirt’s virtual machine assistant has this funny quirk where if you want to change things like the chipset used or the firmware (which defaults to BIOS), you have to specify one little option before creating the VM. Otherwise, it’s set forever (unless you really know what you are doing and love editing XML). During the whole ordeal I figured I could offload the garbage of running libvirt onto a remote server… or my new server. A couple of SSH tweaks and copying public keys later and I had a connection from my laptop to the libvirt daemon running on my Dell Precision server. It was just the beginning. ##</a> Debugging the Lolipop in the Machine</h2> You work at a factory. You are in charge of maintenance on a lot of complicated machines with a lot of moving parts. During maintenance for one of the machines, you figure out the issue comes from an authorization module. You bypass the module to let the machine run, but it does not. Fiddling more in the gears, you realize there is a second, separate authorization module with a lolipop stuck in it. That module overrides the first one, and someone probably dropped their lolipop into it. You spent five hours on this, because someone dropped a lolipop on a module you didn’t know existed. You never expect a lolipop in an industrial machine. Yet, sometimes, you should. When I tried to create a UEFI machine through Virt-Manager’s wizard, after figuring out I needed to configure before creating, I ran into errors saying that access to the NVRAM variable files for that machine was unauthorized. I was running libvirt as root, and connecting through a user that was in the libvirt</code> group. I figured, which not try and set the permissions for that file to 777</code> so I was sure I could find it ? I still hit a “Permission Denied” wall. After some research online I figured out that there is currently a bug</a> in the apparmor profiles brought in alongside Libvirt where the authorizations granted to libvirtd</code> when it starts do not allow it to read /var/lib/libvirt/qemu/nvram/*.fd</code>. The fix that is currently released on Ubuntu (but not even Debian sid yet) only modifies /etc/apparmor.d/abstractions/libvirt-qemu</code> to apply this patch --- /etc/apparmor.d/abstractions/libvirt-qemu.orig 2022-01-22' time: '18:22:57.000000000 +0000 +++ /etc/apparmor.d/abstractions/libvirt-qemu 2022-02-25' time: '13:54:22.075405809 +0000 @@ -85,7 +85,7 @@ /usr/share/misc/sgabios.bin r, /usr/share/openbios/** r, /usr/share/openhackware/** r, - /usr/share/OVMF/** r, + /usr/share/OVMF/** rk, /usr/share/ovmf/** r, /usr/share/proll/** r, /usr/share/qemu-efi/** r, @@ -249,5 +249,8 @@ / r, # harmless on any lsb compliant system /sys/bus/nd/devices/{,**/} r, + # required for QEMU accessing UEFI nvram variables + /**/nvram/*_VARS.fd rwk, + # Site-specific additions and overrides. See local/README for details. #include <local/abstractions/libvirt-qemu></code></pre> Then you just systemctl restart apparmor</code> and systemctl restart libvirt</code> and you get going. The funny thing I found out later on is that it’s not the only apparmor-related bug in libvirt</a>, apparently. ##</a> The Aftermath</h2> In the end, I just wanted to use Virt-Manager and make my little toy VM, but I got sidetracked into the depth of libvirt, and its side interactions with Apparmor. I have managed to install a little remote VM that runs decently and that I hope to turn into my practice VM for things like Hack The Box</a> or Try Hack Me</a>. Or, at least, whenever I have time for these again</del>. Advent of Code 2021 - Retrospective 2022-01-01T00:00:00+00:00 This year, like the last, I participated in Advent of Code</a>, an online event where a new algorithmics puzzle is unlocked every day of december before and on the 25th. For each day, the same puzzle idea yields two small exercises, one typically easier than the other. I chose Rust</a>, like last year, to solve those problems. You can find my repository here</a>. ##</a> A bit of write-up for the days I found interesting</h2> Most puzzles were interesting, to very interesting. The first week or so is easy to solve as long as you have a good handling of simple data structures and logic branching. ###</a> A word on boilerplate</h3> In order to have a faster time coding and testing my code, I spent the first couple days perfecting a boilerplate system, learning about Rust’s workspace crate mechanism</a>. The code I wrote was very simple : A shell day crate that could be copied and which content could be changed to deploy the simple boilerplate code for the day</li> A benchmarking area</li> A lot of testing that the code gave the right results for tests and answers</li> </ul> The shell crates I made had the basic bin/lib structure, where all of my logic code resided in methods called solve_part_one</code> and solve_part_two</code> in src/lib.rs</code>, which could be imported by the higher-level crate for testing, or src/main.rs</code> to be executed / test sample inputs. ###</a> Day 1 : Slice Windows</h3> Day 1 had me a little stumped, but not for any valid reason. I just had never thought about the concept of a window iterator in Rust. I was sure there must have been something like that in the standard library, I just did not know where. It turns out, the slice</code> primitive type is equipped with a windows</code> method that yields a special iterator over a given amount of elements in the original slice. ###</a> Day 5 : HashMap is good for grids (with moderation)</h3> Continuing from a theme I had last year, using std::collections::HashMap</code> instead of a fixed size grid can turn out to be a life saving spark of genius if it turns out your puzzle input is larger than expected, your data is lacunar, or your coordinate space expands to negative coordinates. Last year, the various versions of Conway’s Game of Life had us play around with multi-dimensional grids that could expand to unreasonable sizes in any direction, and I found that storing one type of state information as coordinates in a HashMap</code> (or HashSet</code>), and assuming that absence from that structure means another state information, can be fairly useful. ###</a> Day 23 : Hashing is good, with moderation (and well)</h3> A caveat of the previous point is that hashing still comes at a cost, particularly for complicated data structures that you define yourself. I had an example in day 23 where I needed to store the states of my puzzle state space that I had already explored in order to avoid needlessly repeating computations, and filling memory. To do so, I chose to use a HashSet</code> and implemented std::hash::Hash</code> on a type that was essentially an enum and 30-ish integers in a trench coat. My first implementation iterated all over every single one of these values, and hashed them all individually. #[derive(Clone, Copy, PartialEq, Eq, Debug, PartialOrd, Ord, Hash)] enum Amphipod { Amber, Bronze, Copper, Desert } #[derive(Clone, Copy, Debug)] struct AmphipodPuzzleState { cost: usize, hallway: [Option<Amphipod>; 11], chambers: [[Option<(Amphipod, bool)>; 4]; 4], part_one: bool } impl std::hash::Hash for AmphipodPuzzleState { fn hash<H: Hasher>(&self, state: &mut H) { for potential in self.hallway { if let Some(amp) = potential { // Room no yields a number between 0 and 3 depending // on the value of the enum amp.room_no().hash(state); } else { (-1).hash(state); } } // .. You get the idea } }</code></pre> This was, as I can see now, horrendously slow. Indeed, having to endure the overhead of a single hashing for every single number in the structure was absolutely too much. A second version of this implementation created a single number by pushing a base 5 number into a number stack and hashing it (you will see that it’s a theme this year). However, that was still quite slow. In the end, just creating a method that yielded a fixed-length array of numbers that itself could be automatically hashed worked better. The overhead of hashing still slowed down the code, but I managed to get sub-1s in release</code> target. ###</a> Day 24 : Sometimes it’s not about simulating, it’s about reading and counting</h3> On day 6, we were presented with a problem that seemingly asked us to simulate an unreasonable amount of data. It turns out we only needed to realize that some states were independent of the others, and count. On day 14, we were once again tricked. The polymerization exercise demanded outright impossible simulation. People who had working short-term memory remembered the lanternfish exercise from a week prior, and simply figured out you could use dynamic programming to cut down on simulation, and count. On day 24, we were presented with assembly instructions. I spent a lot of time building a robust, modular instruction simulator, confident that we would need to simulate this program. Nope. It turns out, and don’t read if you don’t want it spoiled, that the program performed a 14-cycle parametric “hashing” process that essentially pushed and popped a base 26 number (see, I said it would be a theme), computed from a test input and some of the aforementionned parameters. The goal was to get the end value to 0. Solving day 24 required way more program structure analysis</a> than pure programming skill. ###</a> Day 22 : Papers aren’t really always helpful</h3> I spent a large amount of my time on day 22 trying to see if someone smarter than me had figured out a general algorithm to compute the real volume of a set of axis-aligned, intersecting cuboids in 3D space. I read a couple brain-melting papers that solved this efficiently for cubes, and more general versions that were really above my understanding. I just ended up creating a system where the intersection of an added cuboid was computed against every cuboid already in the set, and the added cuboid was split into smaller cuboids accordingly in order to shave off and split the added cuboid into a set of smaller, non intersecting cuboids. Computing the difference of volume and adding them then became trivial. ###</a> Binary optimization is madness</h3> The worst runtime I had this year was day 19, before a large optimization I made. In debug</code> target, it ran for about 70 seconds before spitting out the right answer on my input. However, building in the release</code> target reduced the runtime to a mere 3 and a half seconds. More generally, I have to say that the binary optimization on the release</code> target for Rust is mind-bogglingly fast. Repeated operations seem to fly by like the compiled itself predicted your data and shrunk the binary to just solve what could change. The world of compiler optimization looks fascinating, and very complex. ##</a> More general things I learned</h2> Rust is very powerful, yet,</li> Rust is not very well suited for a lot of fast, dirty problem solving.</li> Zero-cost abstraction is wonderful. I could write entire structures to describe my problems very clearly (like Amphipod</code>/AmphipodPuzzleState</code> for day 23, and have close to no overhead from it)</li> The people who strive for leaderboard on this event are all generally american and do not work a job where they have to wake up early</li> </ul> And that’s about all I can come up with for now! I want to thanks Eric Wastl</a>, creator of Advent of Code, along with the whole team, for the work they put into the event every year. Hopefully, I’ll see y’all for the next one! And since this post comes out January 1st 2022 I also want to wish everyone a better 2022 than 2021 was. Unlocking the Intel X520-DA2 SFP+ 2021-11-07T00:00:00+00:00 I have recently had the pleasure to acquire a new piece of equipment for one of the servers that I manage in one of our university clubs. Long story short, we were on the verge of a bottleneck happening in our infrastructure at the network interface of the aforementionned server, and decided to invest in a network card and some fiber optics to boost the link speed from 1Gbps ethernet to 10Gbps dual channel. After some consideration, and because we kept encountering issues with online shopping and delivery, we eventually bought an Intel X520-DA2 SFP/SFP+ Network Card</a>. For those who don’t deal with fiber optics, SFP (meaning Small Factor Pluggable</a>) is the standard for the transceivers you can insert into fiber optics cards in order to interface between the card and the fiber optics. SFP and SFP+ are two different standards for these transceivers, and the latter allows for speeds up to 10Gbps. The Intel X520-DA2 can (thankfully) adapt to both modes, although we would rather want to use it with our SFP+ transceivers. ###</a> The Tragedy</h3> We had high hopes for this card but as soon as we plugged the SFP+ transceivers into the card and our switch and saw no blinking lights, a sinking feeling overtook us. Tragic histrionics aside, we felt a bit panicked. I went to the console of the server and tried to forcefully raise the network interfaces (which were recognized) and saw this ~# ip link set enp25s0f1 up [....] ixgbe <PCI id>: failed to load because an unsupported SFP+ or QSFP module type was detected. [....] ixgbe <PCI id>: Reload the driver after installing a supported module.</code></pre> That was a problem, since we do not own compatible transceivers. Accoding to official documentation</a>, only a handful of transceivers are compatible, all of them made by Intel (isn’t that funny). We did not even have a compatible transceiver in the “tested 3rd party category”. We could still use those unsupported transceivers for SFP, 1Gbps connection, but paying a month of rent on a piece of equipment that does not deliver or improve your infrastructure is mildly infuriating. ###</a> The Investigation</h3> Initially, we had scant information, and thought our problem laid in fundamental hardware incompatibility. My first queries only led to a random french forum thread from 2012</a> telling the story of a person who had the same surprise we did. As we advanced in our research however, we encountered two important pieces of information : The official documentation</a> does mention a rather odd module parameter called allow_unsupported_sfp</code>, a boolean that bypasses a software check for the transceiver model. We can also find it in the ixgbe</code> linux driver source code</a>.</li> </ul> static unsigned int allow_unsupported_sfp; module_param(allow_unsupported_sfp, uint, 0); MODULE_PARM_DESC(allow_unsupported_sfp, "Allow unsupported and untested SFP+ modules on 82599-based adapters");</code></pre> Some people apparently overwrote the firmware of the ES82599 card</a> to “lock” and “unlock” it, letting it either use any SFP+ module, or none.</li> </ul> That latter posts also mentions some trickery at play in the source code for the linux driver. In order to probe for capabilities, the linux kernel looks for a value stored in the firmware of the network card, at location IXGBE_DEVICE_CAPS</code></a>, or 0x2C</code>. That value is an array of bits that correspond to different capabilities for the network card. Crucially, the bit corresponding to capability IXGBE_DEVICE_CAPS_ALLOW_ANY_SFP</code></a> is set to 0</code>. Our brave adventurers dumped the firmware of the card with ethtool</code>, located the offset for that device capability bit, flipped it, and flashed the firmware again back onto the EEPROM. This way, multiple people successfully tricked the kernel into allowing any module on the SFP+, confirming that it was never really a hardware issue. I am not positive that it is entirely malfeasant on the part of Intel, as they would probably want to protect themselves from liability if people used untested (i.e. not their own) SFP+ transceivers in the X520-DA2. With that being said, as we like to say in my language whenever you see a huge, greedy fuck : “Téma la taille du rat”. At first we thought flashing the firmware was going to be necessary pain, but that we would be done once we had gone through the somewhat stressful process. Thankfully, it would’t even have to come to that. ###</a> The Conclusion (TL;DR)</h3> As it turns out, posters in the first thread also found the somewhat obscure parameter allow_unsupported_sfp</code> used to override the check</a> performed on the ES82599 card, which “”“does not support”“” unofficial SFP+ transceivers. Simply unloading and reloading the ixgbe</code> driver with the correct parameters lets us use any transceivers. rmmod ixgbe modprobe ixgbe allow_unsupported_sfp=1</code></pre> Making this change persistent can be done with a simple line in your distribution’s kernel module configuration (typically /etc/modules</code>). One day I might flash the card so we don’t need this parameter, and so the people who use that server after me never have to care about Intel’s shenanigans. Starting prometheus node exporter after your interface is up 2021-06-29T00:00:00+00:00 The monitoring stack in my home network depends on a lot of pieces of software all configured to be up and ready every time I have to reboot the gateway (a 8GiB Raspberry Pi 4). One piece of software who, for a while, did not play nice was Prometheus</a>. (For those just looking for my answer, it’s in the last paragraph). ##</a> The setup</h2> Essentially, my statistics front-end</a> is running on the VPS that serves my website. It, and my gateway, are both members of a VPN (as in an actual virtual private network). When my gateway comes online, it connects automatically to my VPS (the VPN’s gateway), and Grafana can pull data from it from within the VPN (removing the need to expose my raspberry pi to the wild internet). Similarly, the prometheus database in that stack can pull data from the exporters installed on various machines within my VPN, including the gateway. ##</a> The issue</h2> I decided early on that having prometheus’ node exporter expose metrics on every interface was not necessarily advantageous. It could lead to a leak of information from anyone who got access to the network above mine, or just my own (through the wifi, for example). The option that sets the listen interface can be added in /etc/default/prometheus-node-exporter</code> : ARGS="--web.listen-address=AA.ZZ.YY.XX:9100"</code></pre> where AA.ZZ.YY.XX</code> is the address of the gateway within my VPN. However, upon next rebooting the raspberry pi, I was presented with this : No logs! Tragedy! Prometheus’ node exporter did not come online. This happened : level=error ts=[...] caller=node_exporter.go:198 err="listen tcp AA.ZZ.YY.XX:9100: bind: cannot assign requested address"</code></pre> Clearly the problems appear to be that the systemd unit starting prometheus-node-exporter</code> is not written to expect my wireguard interface before starting itself. ##</a> The solution</h2> My wireguard interface is configured and raised automatically at boot thanks to a service called wg-quick</code>, with the specific target wg-quick@iiiii.service</code> where iiiii</code> is the name of my VPN interface. Running systemctl status prometheus-node-exporter</code> shows you that prometheus node exporter failed to start, but it also gives you the location of the unit file associated with that service. Open it (as root obviously) and, in the Unit</code> section at the beginning of the file, you may add After=wg-quick@iiiii.service</code></pre> Then you can reload systemd’s units (systemctl daemon-reload</code>) and reboot/restart the unit. (Note: a reboot is not required but my issue is with the behaviour on a fresh reboot). In case your situation is not exactly like mine (you are using different software, or you want to expect an interface not linked to a specific unit), you may want to look at what systemctl list-units --no-pager | grep net-devices</code> shows you. Virtual device services created by systemd might also work in that setup (although I have yet to try that setup). The Services of Vulpine Citrus 2021-02-08T00:00:00+00:00 I’m coming close to a year of VulpineCitrus</a> being up and running, and over the course of a year it has been a tremendously useful tool for me to deploy so many services that have changed my (and to an extent others’) daily routines. It has also been a great way to learn a lot of technological solutions I had heard of before but never managed to deploy at home. Back in the days when I shared a VPS I had TinyTinyRSS</a>, a small web-based RSS stream aggregation tool, running and configured. I personally use it to keep up with local newspaper that have not yet dropped their RSS feeds (and it works well). When I migrated, the first things I had to do were moving my PostgreSQL clusters</a> and installing TTRSS again. It has been working like a charm for a year. Later, when the pandemic happened and I was back home, stuck with a lot of time, I started dabbling with WireGuard</a>, the small yet powerful VPN system now shipped with the Linux kernel (for versions >=5.6</code>). I had a lot of fun connecting the small network in my bedroom (which I should discuss for the sheer technical interest of it) and had a couple painful encounters with the kernel’s routing system (both in IPv4 and IPv6) and NetFilter</a> framework (usually through ip(6)tables</code>). Another service I brought over from my old VPS, but haven’t been using for a while, is Drawpile</a>. It’s good fun to doodle with friends from time to time, and last summer I decided to update the clunky and old server I had kept running on here. I was both surprised and afraid to find out that they had switched to Docker</a> as a deployment solution. They also work in a distributed architecture as well, so I had some good fun yelling at docker. It is a technology I already know from my systems and network experience, but it was the first time I had to deploy a solution on my own using pre-made dockers. It went well (although I do remember being utterly confused trying to use the old client-side tools and not understanding right-away why it wasn’t working). I should update it again, in fact. Thankfully, it’s rather lightweight, and it works. The rather funny side effect of having draw.vulpinecitrus.info</code> point to the same VPS as everything else is that trying to access that domain with a browser will give you a SSL_ERROR_BAD_CERT_DOMAIN</code> error (my web certificate is only valid for vulpinecitrus.info</code>). Let’s talk about apache2</code> by the way. I did not know it back in July/August but when I deployed a Gitea</a>, I learned a lot of extremely useful things about reverse proxies, and specifically Apache’s reverse-proxy. Using recommendations from Gitea’s README and some help from Apache’s manual I cobbled together the right site-enabled</code> configuration file for Apache’s reverse-proxy to server me gitea</a>, which I have been using</a> for some time now to host code for people and some archives of projects, most recently including Advent of Code 2020</a>. The final service I moved from my VPS in March is the couple of Discord Bots</a> I run for me and my friends, an especially sensitive operation since one of these bots ended up being an important part of one Discord server I help ran for my college promotion. A couple crontab</code> lines and pyenv</code> hacks were all I needed. In September I moved in with a group of friends and found myself needing some sort of cloud storage. In order to synchronize my class documents across multiple devices, I had been using git</a>, which worked for a while, but gave up with large media files (lessons our teachers recorded for example). Within my network, I had started using a Samba server, running on my Raspberry Pi 2B Rev 1.1, but said RPi couldn’t handle a hard drive, so massive storage was impossible. In early October, when school moved off-line again, I heard about Nextcloud</a> again, a full cloud solution (although I hate the word “cloud”). It was a hassle to deploy, since the easiest way to accommodate for storage was to have the service run at home, and proxy to it from my VPS. I combined the aforementioned wireguard VPN along with Apache’s reverse proxy, and after roughly two days of googling and checking uploads, VC’s Cloud</a> was up and running. Lately, not much has moved. A lot of my needs are currently satisfied. I discovered Bitwarden</a>, which was horribly painful to deploy and configure (it deployed no less than 10 dockers), so I switched to the compatible Rust implementation</a> by Daniel Garcia, which was ready to use after only half an hour of cloning docker images. I can also mention a couple of dev experiences VC has afforded me. I made my own log file separator in bash, a rather painful experience. Running the Discord bots gave me an excuse to learn Discord Py to make some useful modules for me and my classmates. I will also mention in passing that I deployed the mail server for Vulpinecitrus on my own, using Dovecot</a>, Postfix</a> and SpamAssassin</a> (the classic triad), and I have plans to dive into how I did it in a later post. I also had an IRC server but since I don’t actually have a use for it, the SystemD unit is disabled by default. So that’s a year of VulpineCitrus. Hopefully I get to do crazy new things! I am thinking of tinkering with GlitchSoc</a>, a Mastodon fork, as well as Grafana</a> (which I already know a lot about but haven’t put on VC yet, it’s in my home network). New server. New system. New PostgreSQL database 2020-03-06T00:00:00+00:00 A few days ago, I launched a website called “Vulpine Citrus”, meant to act as my front end with the rest of the world. Before that, I used to run a lot of my personal services on a server shared with a friend. After I bought my domain name, and VPS, I decided to begin moving my services onto my new server and stop hogging the storage space on that other, shared system. I had, until then, gracefully escaped anything vaguely technically about managing database systems. I still am far from understanding the deep, underlying concepts of DB managers. However, I had an entire computer science semester of “Databases 101” last year, and I felt a little more enclined to start and play around with the PostgreSQL database system. I will show you how to dump and export databases from one cluster to another on different machines. Both systems will be running Debian 9 or Debian 10. As far as my understanding goes, and that is all you need to understand this article, a cluster is a single instance of postgreSQL running on a system, with multiple clusters able to run at the same time, using different versions of the software. On the former host of my database, I was running PostgreSQL 9.6. My cluster was called “9.6 main”. On the new system, I am running PostgreSQL 11. My cluster is called “11 main”. I will only be exporting one database, and creating new users. If you wish to dump and export a whole cluster, you will have to find someone more competent. First off, logged into my new system, I had to install, enable and start postgresql 11 apt install postgresql systemctl enable postgresql systemctl start postgresql</code></pre> You can check whether or not the server is running by using pg_lsclusters</code></pre> Which should show something similar to Ver Cluster Port Status Owner Data directory Log file 11 main 5432 online postgres /var/lib/postgresql/11/main /var/log/postgresql/postgresql-11-main.log</code></pre> Now, head off to the old system. Log in as the owner of the database you wish to export (or any account capable of reading the database, really). You can export the old database by using pg_dump</code>. pg_dump -U owner_of_database name_of_database > my_database.pgsql</code></pre> What you’re doing, essentially, is generating a PostgreSQL script capable of creating a carbon copy of your database as seen by user owner_of_database</code>. You can copy that script by any mean to the new system. I personally use scp</code> over ssh</code>. Back to the new system, you will have to create your (empty) database and the role associated to the user(s) that will access it. The tools called createuser</code> and createdb</code> do the job fantastically, such that you don’t need to write a single line of SQL. createuser --interactive --pwprompt</code></pre> You can tell PostgreSQL that your new user can create databases, or not, roles, or not, it depends on what kind of user you want. Mine is a simple account meant to just read and write the database without doing anything complex or risky with its structure, or any other database. The proper database can be created using createdb my_owner_name -O my_owner_name createdb my_database_name -O my_owner_name</code></pre> The new user will most probably require a database bear their name. When you log into the server without providing a database, this is where you end up. It does not fulfill any other purpose, but I like to create it as well. Once everything is in place, and you can successfully log in using psql -U my_owner_name</code> (or if you can’t because there are no databases), create the carbon copy of your former database by executing the dumped script psql -U my_owner_name my_database_name < dump.pgsql</code></pre> If everything goes smoothly, you should only see a lot of PostgreSQL instructions fly by in the command line. When the database is created, you can try and log into it. Check whether you can successfully log in psql -U my_owner_name my_database_name</code></pre> Once you are sure that everything was transferred as you wanted, you can go back to your previous system, and, optionally, clean up the database(s) you exported. dropdb my_database_name</code></pre> And you’ll be good to go! 🦊 The Vulpine Citrus 2020-03-05T00:00:00+00:00 This is the official launch of Vulpine Citrus, my own personal web site! Whenever I find something interesting to talk about, or want to show a cool thing I’ve learned, it will end up here. Baby's first BrainF*ck clone : The Moostar Programming Language 2019-07-08T00:00:00+00:00 A few years ago, my journey into coding led me to the dark alleys of GNU Assembly. I quickly realized that the difficulties of Assembly made it almost impossible for me to use it in my daily coding life. When it takes you three hours to reimplement strlen</code> correctly, and you’re used to processing natural strings of characters, you may begin to feel like this is not the language for you. </blockquote> – Me, two years ago On my way up from that dark pit of black magic I encountered a curious little fella I had heard of many years prior : BrainFuck. BrainFuck is Turing-Complete</a> and while it could theoretically let us implement any and all computable problem… BrainFuck may not be the best fit for most complex purposes. I mean, just look at this hello world. ++++++++++[ >+++++++ >++++++++++ >+++ >+ <<<<- ] >++. >+.+++++++.. +++.> ++. <<+++++++++++++++. >. +++. ------.--------. >+.>.</code></pre> Of course, I was not content with simply making an interpreter. I started Moostar with the intent of making an interpreter, but quickly decided to add one features to standard brainfuck that I found gave it more power : the ability to creature functions, along with the inclusion of many fundamental arithmetic operations right into the interpreter. For example, here’s how you can write two multiplications in Moostar. >++>+++++<< ~mul;.</code></pre> The hexadecimal dump of the output should reveal a single byte at value 0x08. Currently, Moostar implements all of the following standard subroutines. /* 'this' is the object representing the interpreter */ this->function_register["sub_5"] = "-----"; this->function_register["sub_10"] = "----------"; this->function_register["add_5"] = "+++++"; this->function_register["add_10"] = "++++++++++"; this->function_register["imove"] = "[->+<]"; this->function_register["dmove"] = "[-<+>]"; this->function_register["clean"] = "[-]"; std::string scan8; for (size_t i = 0; i < 8; i++) scan8.append(".>"); for (size_t i = 0; i < 8; i++) scan8.append("<"); this->function_register["scan8"] = scan8; this->function_register["scan16"] = scan8.substr(0,16) + scan8.substr(0,16) + scan8.substr(16,8) + scan8.substr(16,8); this->function_register["foreach_cpy"] = ">[>[->+<<<+>>]>[-<+>]<<-]>[-]<<"; this->function_register["add"] = ">[-<+>]<"; // :R:X: this->function_register["sub"] = ">[-<->]<"; // :R:Y: this->function_register["mul"] = this->function_register["foreach_cpy"]; this->function_register["pow"] = ">>>+<<[>>[->+<]<[->>>+>+<<<<]>>>>[-<<<<+>>>>]<<<~foreach_cpy;<<-]>>[-<<<+>>>]<[-]<<"; this->function_register["eq"] = "[->-<]+>[<[-]>[-]][-]<"; this->function_register["neq"] = "[->-<]>[[-]<+>][-]<"; this->function_register["lt"] = ">[->+<]<[->+<]>+>+>>+<<<[->-[>]<<]>>>[<<[-]<<+>>>]>-<<[-]<[-]<"; this->function_register["gt"] = ">[->+<]>>>+<<<<[->+<]>>[-<-[<]>>]>[-<<[-]<]>[-<<<[-]<+>>]<<"; this->function_register["m@0"] = "^[-]\\"; /* Stuff */</code></pre> (let’s all walk past that hilarious mistake of “interpreter”/“interpretor”, younger me’s English was more than dubious) In general, one would declare a procedure thusly. (proc_name):{instructions}</code></pre> It is useful to add a comment alongside a function’s definition to denote the position of its parameters, results, and all the cells it will need for its processing. For example, see that comment at the end of a definition for ~power</code>. :R:X:Y:0_:1_:2_:3_:</code></pre> This comment indicates that once pow</code> is called, it will compute X to the power of Y, require the four neighbouring cells to do so, and place the result in the cell we call it from. Currently, Moostar doesn’t check for dependencies of functions, and one could write a program that would send the interpreter into an infinite loop of procedures calling each other. Yet, there is more to Moostar than simply adding standard functions. As you might have noticed from the weird circumflex character in my last command, m@0</code>, I extended the charset beyond the simple ability to declare procedures. Two characters were introduced beyond those used in standard BrainFuck, and those for procedural syntax. The first such character, the circumflex accent, switches the interpreter into ‘Meta Tape’ mode, while the backslash switches the interpreter back into ‘Data Tape’ mode. The ‘Meta Tape’, or ‘Meta Registers Tape’ is a set of cells containing and controlling data related to the machine’s function itself. Originally, they were designed to directly modify CPU registers during execution. Two seconds after I had that idea, when I realized it could easily be exploited to bug the heck out of Moostar, I slowed down and only implemented a control register for the data pointer’s position. Interestingly, a legacy name for that meta tape remains in the code : this->ioregs</code> (for I/O registers). Now, the pointer on the meta tape and the pointer on the data tape are not the same. Thus, by switching to the meta tape’s zero register, we can chose what cell we will land on when we switch back to data. This is actually what the m@0</code> procedure does, by resetting the value in meta register 0 (data_pointer_pos</code>) to 0, and switching back to the data tape. You can find the source code for moostar on GitHub</a>.

VulpineCitrus - Blog

Diving into Radio Listening: POCSAG, Alphapage, and how i taught myself some GNURadio

A Completely Innocuous Blog Post about vPMU in QEMU

Guarding My Git Forge Against AI Scrapers

Deploying my Zola Blog Automatically

Diving into Radio Listening: Portable Mobile Radio 446

Enabling bias-tee on RTL-SDR dongles through GnuRadio's OsmoSDR Source

What if any process could handle SIGKILL?

Radio Listening: Airplanes

##</a> ADS-B Antenna Setup</h2> My own setup for listening to ADS-B uses the following items:</p> RTL-SDR dongle</li>

My reading queue as of January 2025

Converting raw IQ from `rtl_sdr` to WAV IQ for SDR++

Diving into Radio Listening: Introduction

A 2024 Update on the Rust Foxes Linux Module

##</a> Update Time</h2> Going through the code, as someone who now has 2 more years of experience not only writing drivers for RFL but also, crucially, abstractions, i listed several things in need of update:</p> The Module</code> trait</li> Missing miscdev</code> abstractions</li>

Irregular Reminder: Strip your Kernel Modules on Install

Pushing to an Overleaf project through Forgejo

Conky may be Leaking your RAM on ArchLinux

Using git with email: a Quick Tutorial

VulpineCitrus is Changing CMS (Again)

Against Real Name Policies

The SNCF WiFi API and All of its Funny Things

Rust for Linux Basics: Handling Function Pointers

Rust for Linux Basics: Adding Allowed Features

OpenStreetMap & OAuth2: Do not Neglect x-www-form-urlencoded

Merging the NFTables NAT Tables

The Way I Setup My GPG Keys Now

##</a> A Small Addendum</h2> But, wait, why are neither of your keys like that-</p> </blockquote>

VulpineCitrus is Changing CMS

Hacking Linux: The TCP Stack and Initial Acknowledgement Number

##</a> Finding the Acknowledgement Number</h2> At first, I wanted to find where that ACK number was initialized. I had two theories:</p> There is a structure somewhere that is null-initialized</li>

Clippy: Obey the Paperclip