The Serpent

Our take on systemd changes

Sun, 22 Mar 2026 00:00:00 +0000

Unless you’ve been living under a rock in the Linux community this year, you’ve probably heard about the fuss that this pull request has made three weeks ago.

The change is a simple one; adding a birthDate field to the JSON provided by the userdb service within systemd. It’s small and relatively low risk, so why all the noise?

Simple. It’s to do with the proposed change of law in California:

Stores the user's birth date for age verification, as required by recent laws
in California (AB-1043), Colorado (SB26-051), Brazil (Lei 15.211/2025), etc.

Notice how the change itself isn’t providing any kind of validation for the Date of Birth, and is in fact just a field like any other. Hell, Debian still asks for a Room Number if you’re doing things the old fashioned way.

It’s simply the devs getting a head-start on allowing corporate Linux distributions to implement their own age-verification systems if they want. However I don’t see the open-source community going much further, nor to I anticipate Linux ever trying to figure out if I entered the right date. If you really want to run Linux in California, there will be plenty of proprietary closed-source solutions available to help you stay legal.

So we’re not worried. We don’t need any of the knee-jerk forks of systemd that this change has spurred (and will almost certainly be abandoned repos in a year), and we’re not likely to see every implementation of Linux adopt an approach that quite frankly isn’t even available on Windows or Apple yet.

Opinion - Taking the User out of Cybersecurity

Wed, 08 Jan 2025 00:00:00 +0000

What are the chances that just one of your users will compromise the security of your network? Many would argue that it’s 100%. I agree, the user has routinely demonstrated themselves to be the weak point in a long line of network defence. After all, it only takes one.

So if you accept this this, I argue that organisations should simply side-step people when testing defences and leave that to the HR\people teams. It can detract from the technical measures that you’re paying a lot of money for, and ultimately if we’re testing the human condition, the results won’t change anytime soon.

In both defence, offence and the purple world in between, we’re quick to blame the user for clicking a link, downloading a suspicious file, or blindly following instructions from an unknown caller - but when performing our own cyber defence tests, we’re still suprised when this happens. The test is considered a success; everyone pats each others backs and employees get lots more ‘education’. The cycle repeats.

Finished at Reconnaissance

But is the test complete? Using the MITRE Attack* framework as an example, it’s suprisingly common for an organisation to stop once the phishing link has been clicked. “Why?” they ask, and put everything on hold until an “investigation” has been done (little do they know, they are investigating the Human condition, good luck with that one!). What about the rest of the framework? Execution? Forget about it! Persistence? Who cares! The technical controls barely get a chance to save the day.

Why did the email get through the filters? Why did it land in the inbox? Focusing on the technical “why?” is important, and requires attention. But it can be simulated without a user, which allows for all the remaining technical controls to be tested too. It’s surprising how many of the layered aproaches to security rarely get peeled back.

Hello, this is the CEO

Some methods are just plain cruel. There’s stories of employees breaking down in tears after a particuly nasty phishing call from their own organisation attempting to extract an MFA code or open an attachment. Of course it is a real threat and does happen often - but that doesn’t mean you need to do it. For the same reason we don’t break into peoples homes and steal their badges, we shouldn’t be bullying them into breaking security best practices. We should be educating them to prevent it.

Starting the tests when you have the MFA code of an employee? Now that’s where it gets interesting. The employee doesn’t even have to be real, a dedicated ‘dummy’ user that is managed properly doesn’t have to introduce anymore risk than a real one, and it leaves your employees free to get on with their jobs.

Having technical teams test users is an exercise in futility. Not only does it create a “them vs us” mentality within the organisation, it leads to a game of entrapment where users have no chance of success. This is especially true if you have no defined success criteria where the pressure against a human can increase unchecked. This is made worse when attackers\victims are both within the same organsiation. Privileged knowledge and access make it almost impossible for the user to know the best course of action.

So how can we break from this cycle? Here’s a thought.

Split the Technical and the Human

So what’s a better way? Some of the most successful teams I’ve worked with in 2024 have split out the technical aspects from the human. Not in all tests, and they still converge on occasion, but for the most part all engagements consider the user compromised; bad; uneducated; malicious. All the terms we scream when they click our intentionally placed malicious link from a source we know is allow-listed and crafted to bypass the very controls we manage.

It also means we can get right into the details of our technical controls. Let’s start the engagement at the point where the link has been clicked, and let HR worry about the why. This allows the tests of network security, EDR and Incident Response to run more regularly, and puts the real defence strategy to the test. We can still measure the success of a phishing link to an account we control, and remove the random nature of determining if it will ever be clicked of not.

It’s too easy to go after the user, and a lot of defence\offence testing naturally focuses there. Is it because they are the highest risk? Again, 100%. But if we ignore the user for now, we can discover a lot of other weak points we may have yet to discover (like “why does that exception even exist!?”)

Keep Measuring Success

Cybersecurity can’t be judged on completed goals, but rather how it functions. We still need to test the user base to determine if our expensive online video training is getting through or not, and these tests open up useful insights into an organisation (for example, that one location or office that always fails the test!) But it shouldn’t be used as the first step every time. My argument is that it should take a back seat role and let the technical controls that if properly implemented, demonstrate their effectiveness at protecting an organisation from both internal, external, santioned-exercises and malicious threat actors alike.

* Yes, it’s actually stylised ‘ATT&CK’ but lets keep it simple, shall we?

The Beelink Mini S12 Pro PC

Thu, 02 Jan 2025 00:00:00 +0000

For the last six months, I’ve been running Plex Media Server on a Beelink Mini S12 Pro. It’s got the grunt to perform encoding on the fly, and it’s certainly more stable than a Raspberri Pi, while still running completely slient.

This job used to be handled by a small Rasperrry Pi, but after that decided to no longer boot, I wanted to see what else was out there in the small PC market, and it’sa suprisingly big market. There’s many different vendors out there targeting the gap between ssmall microprocessor’s and full size PC’s.

Specifications

The Beelink is powered by the Intel N100 processor, a 4 core chip running at 3.40 GHz, which targets the mobile industry. The Mini S12 Pro ships with 16GB RAM, 500GB SSD for storage and as a nice touch - a choice of five different international power adapters.

If also comes with Windows pre-installed (though this was removed in favour of good ol’ Debian!) and all of this for around $200.

There’s two USB3 ports on the front, and dual HDMI for 4K output if you want to run it direct into your TV as well. It’s the perfect little addition to the entertainment shelf. It also sports a gigabit eithernet port, something most small form-factor computers have dropped to save on space, but for a media server, it’s something I wouldn’t want to be without, espectially since the NAS is stored right next to it. But there is WiFi6 of course, if needed.

Overall Beelink have made a very robust, powerful and small computer that’s a step up from microcontrollers that tend to die after heavy, prolonged use - but small enough to remain quiet and stable.

Check out the full range over at https://www.bee-link.com.

Merry Christmas, Happy new Year

Mon, 16 Dec 2024 00:00:00 +0000

Well that was a fast year! We’ve been working pretty hard on other ‘day-job’ projects this year, which means side projects like the good ol’ Serpent tend to fall by the wayside as other priorities come and go. But I always like to end the year with an update to remind you all we’re still kicking.

We’ve been hosting a fair amount of infrastructure for friends of ours this year, ranging from Matrix, Minecraft and Nginx - our preferred web server of choice. As a result, we’ve learnt a lot about how the big three (Amazon, Microsoft and Google) charge for resources. It also helped us understand the competition in cloud computing, who are offering some of the best deals right now to entice customers in; Check out IBM, Oracle and DigitalOcen for better deals in 2025.

The world of Cybersecurity has taken up pretty much most of the year, attending shows and helping organisations navigate the complicated world of SASE\SSE\SWG\DLP\RBI\SaaS\IaaS\FWaaS\CASB and probably many other technologies I already had forgotton by May!

There’s also some interesting things happening in the Linux world. Check out The GNU Shepherd if you’re fed up of systemd taking over the entire machine.

We’re also dropping the X sharing link because we’re done with that dump!

So I wish you all happy holidays, and a great new year. There’s plenty of look forward to in 2025!

Goodbye to Some Legends

Tue, 28 May 2024 00:00:00 +0100

I try to embrace change; modernisation, new tech and just getting old in general. But sometimes it hurts a little, when once great technology gets retired into oblivion, never to be seen again.

This year, we’re going to say goodbye to two greats: ICQ and VBScript.

I Seek You

First on the chopping board is ICQ, an instant chat messaging platform from the mid 90’s. You’d be excused if you thought it had died long ago, but it held on after being sold to a Russian firm back in 2010 and became the western version of VK. ICQ allowed the masses to enjoy instant chat without having to learn commands, or care about servers and networks required to access the original chat system – IRC. It made the complexities transparent and brought about easy to use features such as group chats, actions and even dedicated server support for business use (though I don’t know anyone who ever used it in such a way). By about 2000, there were over 100 million users. No doubt anyone who was using it back in those days will brag about their ICQ number, which effectively issued numbers in sequence – so the lower the number, the earlier an adopter you were.

ICQ started in 1996 and will finally stop seeking on June 26th, 2024. The only thing we won’t miss is that giant foghorn start-up sound introduced sometime around version 2000b.

VBScript – The Malware writers’ choice

Next on the chopping block is VBScript, Microsoft’s answer to automation in the 1990’s, but realistically every script kiddie’s dream-come-true which resulted in countless worms destroying Exchange servers throughout the turn of the millennium.

VBScript has a longer send off than ICQ – with Microsoft doing a phased deprecation. In fact, the notice was somewhat lowkey, featuring in the Deprecated features for Windows client (Windows 10\11) and not much else. I’m not sure how many users still rely on it, but it’s still a big player on some older ASP based websites. It’s going to stick around as an “on demand” feature for a while, before eventually being removed altogether. PowerShell is the official recommendation for those looking to alternatives.

And let’s not forget…

Lastly, we should probably say goodbye to WordPad. The lightweight text editor that wasn’t quite Word but not quite Notepad either has finally be removed as of this year. So long old friend, you were good for opening a hosts file and not much else, but we’ll miss you.

We've moved, again!

Fri, 24 May 2024 00:00:00 +0100

Welcome to the housewarming party, because we’re now hosted on Digital Ocean! Pull up an ethernet cable, pour some packets into a glass and enjoy that sweet new IaaS feeling without having to lift a washing machine.

We’re on our forth cyber-home here at The Serpent, thanks to AWS increasing costs for reserving static IP’s. After a little shopping to compare compute, storage and network costs - Digital Ocean came out the cheapest (for now). The move went super smootly, thanks to how we package our Serpent application, and years of doing it wrong.

We shopped around of course, comparing AWS with Azure, GCP, Oracle and even IBM. While Oracle\IBM are throwing decent amounts of free tier goodies to new customers, they still charge quite a preium for static IPv4 addresses. Something Digital Ocean seems to have avoided.

We’re also not quite ready to go fully IPv6 yet. Give it another 10 years.

What the move did teach us however is that having a ready to go jekyl based package, and a repository for our source code, management scripts, backend and all the other crap we host - made the process extremely easy with only about 15 minutes of downtime. Even our Matrix friends only complained six times (you get what you pay for @ampe!). It also highlighted the value in not being locked into specific cloud services, by using generic opensource tools and ensuring that any services we do use, aren’t proprietary to that vendor. While many make these services exportable, there’s a complex process involved, and we wanted to avoid all that.

So from Tripod, to 1&1, to AWS and now to Digital Ocean - we hope our new home serves us well for the future.

Until of course they jack up the prices.

ESP32's and the Real-time Clock

Sun, 10 Mar 2024 00:00:00 +0000

I’ve been doing a lot of amateur projects with microcontroller’s lately. It’s a fun hobby which I incorrectly assumed would be quite straightforward yet rewarding. I was half right.

It’s rewarding all right, yet the world of tiny computers is mildly frustrating at times, and I wanted to share a brief article on a topic I could find no answer to online. If you’ve arrived here in frustration as to why your ESP is always able to remember the time, read on.

The Problem

I’m using Thonny with Micropython for most of my projects, thanks to it being very quick and easy to set up. I’ working on a project where the ESP32 needs to keep accurate track of time. I know the ESP32 has a real-time clock onboard, but does it have persistent storage\battery to keep the time while switched off?

The answer is no. It does not. The real-time Clock (RTC) is present, but it has no persistent storage or other power supply to keep track of time. In fact it defaults to an arbitrary Epoch of 1st January 2000 (check with utime.gmtime(0)[0]). Ideally, you’d plug the ESP32 into an external RTC which keeps time, as explained in the Espressif manual:

ESP32 uses two hardware timers for the purpose of keeping system time. System time can be kept by using either one or both of the hardware timers depending on the application’s purpose and accuracy requirements for system time. The two hardware timers are:

RTC timer: This timer allows time keeping in various sleep modes, and can also persist time keeping across any resets (with the exception of power-on resets which reset the RTC timer). The frequency deviation depends on the RTC Timer Clock Sources and affects the accuracy only in sleep modes, in which case the time will be measured at 6.6667 μs resolution.

High-resolution timer: This timer is not available in sleep modes and will not persist over a reset, but has greater accuracy. The timer uses the APB_CLK clock source (typically 80 MHz), which has a frequency deviation of less than ±10 ppm. Time will be measured at 1 μs resolution.

I don’t need high accuracy, so the RTC timer will do nicely, but since it doesn’t persist across power offs, I’ll need to set it myself on boot. No problem, just a bit of extra code.

But when I ran my test code, shortly after boot - the ESP32 always had the correct time!

import machine
import utime

rtc = machine.RTC()
print(rtc.datetime())

now = utime.time()
print("Epoch: " + str(now))

MPY: soft reboot
(2000, 1, 1, 5, 0, 0, 0, 745174)
Epoch: 0
WARNING: Could not determine epoch year (can't import name localtime), assuming 2000

MPY: soft reboot
(2024, 3, 10, 6, 19, 0, 45, 930672)
Epoch: 763412445
>>>

WiFi\Bluetooth were disabled, and I didn’t even have any code to query NTP. So how was the ESP32 gaining correct time?

After lots of trial and error, I finally discovered that the source was the IDE itself, and the following setting was triggering it:

Disabling this setting allowed the ESP32 to work as expected, clueless of the time and needing NTP to come to the rescue. Mystery solved.

A New Year Project - Debian Networking

Fri, 22 Dec 2023 00:00:00 +0000

Christmas is a time of year for family, friends, presents, Turkey and yes, Linux networking.

It’s a strange holiday tradition to have. But I like to brush up on various Linux skills over Christmas when things calm down. Work starts to take it easy and the kid is old enough to lap up hours of Minecraft and YouTube. It’s a perfect time to get stuck into a simple and useful project.

This year I’m going to take a look at systemd-networkd because quite frankly, it’s not going anywhere and will probably become the default Network Manager in the future. But there are others that dominate the Linux world, so I wanted to brush up on the basics and see what, if any, would be a good enough reason to switch.

Debian History - ifupdown

Long before we ever needed a Network Manager, things were managed with a simple interfaces file and a handful of common network tools. In fact, this all still works perfectly today in Debian 12 and is still arguably the quickest way to get a static IP configured when you’re using a system without a GUI.

Things took a turn for the worse when Wi-Fi came along. ifupdown had to rely on a second package, wpasupplicant which enabled the use of pre-shared keys in the interfaces config. This worked, but caused a few headaches for administrators who forgot to either hash the key, or lock down the permissions so that users couldn’t read it.

Still, it worked quite well.

Enter NetworkManager

Things got even better when NetworkManager came onto the scene in the mid 200’s. No need to edit configuration files! Instead you could create profiles for each interface and have NetworkManager handle all the details. It’s main purpose was to fix issues with roaming between networks, something Linux was terrible at doing in the early 2000’s when users were just starting to switch between Wi-Fi and Wired connections. While it was possible to use NetworkManager without the GUI elements, it wasn’t ideal. GUI-less servers still had to do things the old fashioned way.

Killing ifconfig

After years of faithful service, ifconfig was deprecated in Debian and many other distributions in favour of iproute2. While a sad moment for us old hacks, it made sense to bring multiple network tools together under one package.

This action kicked off a whole modernization era within Linux networking, and we saw the rise of systemd wanting to expand its already overbearing reach into the network stack.

OK, but it’s really easy

systemd-networkd brings a lot of sense to network configuration (including the frustrating yet true to form case-sensitivity!). For example, to configure a basic wired ethernet adapter, we simply create the following:

[Match]
Name=eth0

[Network]
DHCP=ipv4

Saved as /etc/systemd/network/10-eth0.network, this sets our Ethernet adapter to use DHCP. Easy eh?

The file name needs to end in .network, and should have a priority number. Otherwise it doesn't matter what it's called.

It’s important to make sure you move any existing interfaces out of the way, otherwise they may override the systemd-networkd settings during boot. This can be easily done by moving the file to interfaces.old or something similar.

Cities Skylines II Out Now

Tue, 24 Oct 2023 00:00:00 +0100

For those with an eternal love of city builder simulation games, Cities Skylines II is for you. It comes just over eight years since the first game released, and promises to deliver all the realism that mod’s covered since.

There’s one small catch - the game is huge and not quite optimized yet.

My rig struggled on high settings, and it’s no budget PC. But after finding this post, I was able to modify the settings to get the game in a decent playable state.

If you’re planning an all-nighter and want to get the game running smoothly, at the cost of a few clouds looking silly, then check it out for some decent performance tips. With any luck, we’ll see a stream of patches in the immediate future which will allow those settings to return to full on room-cooking speeds.

Choose your Clouds Wisely

Mon, 25 Sep 2023 00:00:00 +0100

Ask any IT engineer about their home lab and you’ll probably get into a conversation that rivals Cat lovers uniting or new parents talking about their babies. It can be long, passionate and you may end up never speaking to the person ever again. Well, ok maybe not that extreme.

But it’s a subject many techies take to heart. We spend many years honing our labs, updating them, reinventing them and tearing them down to start afresh. Just like that DIY project - they’re never finished.

What follows is a consumers thoughts on Enterprise cloud solutions, when needing something simple for home use, and definitely not a detailed analysis of all their capabilities and differences!

I’ve been quite fortunate and have been able to work from home since about 2016, and I’ve gotten myself into a good routine, keeping in touch with the team, taking regular breaks and setting up my home to give me access to all the tech I need. But working from home can be lonely work, which is why the best advice I can give to new WFH’ers is to get out of the house once in a while.

Thus the problem arises, my home lab doesn’t move with me. As I type this now, I’m working in a cafe nearby the water. the coffee is great, as is the Wi-Fi (the music however is terrible). I still need access to those ESXi servers, pfSense firewall, Windows Desktops, NAS and other things all running at home consuming about the same amount of electricity as a medium sized old peoples home. So how do you keep in touch with the lab?

For a while Tailscale was my preferred choice, and in fact I still use it on occasion to access the really meaty stuff that hasn’t moved elsewhere. it’s simplicity and stability are a welcome change from the usual OpenVPN\Wireguard setup we’ve all tried. Yet it doesn’t really solve the problems of cost and performance when accessing resources via home broadband connections. Even fibre-to-the-home (FTTH) doesn’t quite cut it sometimes when throwing GB’s of data in both directions (and doing about the same amount of throughput for YouTube and Disney Plus during those school holidays).

Just virtualize it…

One key use case is to run a Windows 10\11 client. For this, I want to spin it up, run some tests and shut it down. Simple right? “Spin up a virtual machine” I hear you all say! Ideally yes, though here’s some things to consider:

Traveling remotely means I don’t always have power, and virtual machines are a pain to run on battery.
The laptop I’ve opted for is a compromise between speed\size. Carrying an engineers laptop around the world results in eventual back\neck\shoulder pain, so I’ve gone for something a little less powerful but hardly weighs anything.
Free hypervisors such as VirtualBox and VMware Player lack some cool features such as snapshots.
They take up lots of space.

I started to look at cloud compute services to host basic things like a Windows\Linux client, and compare the cost to see if it was easier to just maintain a machine in the cloud.

Microsoft Azure

For my first use case, there’s only one option. Azure. Not because it’s great, but because Microsoft simply make it impossible to use anything else. If you want to spin up a Windows Client without having to roll your own image or bring your own license - you’ll need Azure.

You can run Windows Server on others, but for Windows Desktop, Azure is the only one that will let you spin up a machine using a pre-built Azure image. The costs are pretty low, but as always, the devil is in the details - and they soon can rack up.

Windows averages just under 30p a day when not in use.

Using the Microsoft Price Calculator, you can see that Windows never hits more than £1 for daily use, and this is a Standard B2 image with 4GB RAM, pretty reasonable! Yet why is the bill £11 a month?

Storage. You gotta pay for that stuff. What many of these calculators fail to mention is the ongoing charge for disk images. As you can see, the VM compute cost pails in comparison to the storage cost, and is responsible for anywhere between £10 and £25 a month of usage for a single Windows client.

The Good

Easily spin up Windows Desktop clients
Also gives you free Azure AD (sorry, Entra ID)
Uses PowerShell as its CLI of choice, easy if you already know how
Public IP costs are low

The Bad

Storage costs aren’t advertised very well, soon builds up
Networking costs for IPSEC are high
Documentation can still be confusing and polluted with outdated articles

Amazon AWS

The other 800 pound cloud in the sky is Amazon AWS, a service that now makes more for Bezos than all the crap we buy on Amazon put together.

I started with AWS because it was simpler to use, and still is. The Serpent is still hosted on it. The GUI is straightforward, the documentation is excellent. It was also very cheap (see below as to why I say was). Additionally, Amazon are always developing AWS, it changes every month. There’s more services in it than I can shake a stick at, usually with names that do nothing to explain its purpose (Amazon Lex, Amazon Kendra, Amazon Polly to name a few!)

AWS also consolidated services early on and even spun out lite versions of compute, which was very attractive to small outfits like us. But how does it stack up for cost?

From zero to something, the new price of public IP's in AWS.

Well we couldn’t compare Windows Desktop because it’s easier to build a fully functioning Boeing 747 out of Lego than it is to import a Windows desktop image into AWS, and we do actually host a lot on it.

The Good

It’s easier to use, documentation is good and GUI explains most things
Lots of image support, including Windows Server (as a free tier!) and MacOS Ventura (Charged daily, and charged a lot)
Provides LightSail, a cut down version of their EC2 compute platform
Provides Route53, fully integrated domain registrar and DNS manager

The Bad

From February 2024, AWS will start charging for public IP’s.
While billing is all done in GBP, Cost Explorer always defaults to US Dollars

Oracle Cloud

Oracle you say? Yes! They’ve been running Oracle Cloud for some time now, but only recently decided to actually get some customers excited about it. Sign up today and you can get $250 free credit, as well as a very impressive always-free tier of ARM based machines available to all.

Oracle was also one of the easiest to use, and thanks to the Networking wizard, I was able to get an IPSEC tunnel up in not time.

The Good

Cheaper than the competition
Easy to use interface
Supports ARM based machines for easy and customization hardware specifications

The Bad

Admin portal doesn’t support Opera!
No native Debian image, they prefer you to use Oracle Linux

Google Cloud Platform

I’ve not got round to trying Google out in anger yet, if you have thoughts of your own about how this service stacks up against the others, please let us know.