Amrou Bellalouna

How one single byte lead to silent Celery task corruption

Fri, 04 Sep 2026 15:25:56 +0100

During our RabbitMQ v4.x migration preparation, we started seeing exceptions we couldn’t make sense of — the kind that knock your services over with no obvious reason why.

One of the steps I mentioned in our RabbitMQ v4.x migration was to transfer messages between queues in different vhosts, and that meant reading from queue1 in vhost1 and copying those messages to queue2 in vhost2.

We rolled this out environment by environment, and it went smoothly for the first few — until one day Sentry lit up with:

TypeError: unsupported operand type(s) for +: 'float' and 'str'

Digging into the stack trace, the crash was happening deep inside Billiard, one of Celery’s core libraries, at this line:

if monotonic() >= start + timeout:
 ...

start was a perfectly valid float. timeout, on the other hand, was ,. Just a comma, when it was supposed to be a float/int as well.

Sentry Snippet

Where did a comma come from? The answer is buried in how RabbitMQ encodes values on the wire — and specifically in a disagreement between two Python libraries about what a single byte means.

Strap in.

AMQP frames

RabbitMQ speaks AMQP — it’s the protocol that handles everything from publishing messages to consuming them.

AMQP runs over TCP but doesn’t just pour raw bytes down the pipe. It wraps everything in frames — structured envelopes that each carry a type, a channel number, and a payload. The type tells the receiver what kind of data it’s looking at.

If you want the full picture, the official protocol spec has it all. Brian’s article on AMQP’s frame structure is a much quicker read if you’d rather not wade through a technical PDF.

The `headers` property

When you publish a message, one of the frames contains what AMQP calls “properties” — metadata about the message like content-type, content-encoding, and so on. Think of it as HTTP headers but for your queue.

One of those properties is headers, which lets you attach arbitrary key-value pairs to a message. In AMQP terms, that structure is called a Field Table — which is really just a binary encoding of a dictionary.

Field table encoding

This is the part that actually matters, so it’s worth slowing down here.

Each entry in a field table is packed as four consecutive pieces:

Key length — 1 byte telling you how many bytes the key name occupies
Key name — N bytes (where N came from the previous byte)
Type tag — 1 byte identifying what kind of value follows
Value — M bytes, where M is determined entirely by the type tag

That type tag is the crux of everything. AMQP defines a fixed set of value types — short int, long string, boolean, decimal, etc. — and each one has a single-byte identifier. The decoder reads the tag first, then knows exactly how many bytes to consume for the value.

Section 4.2.1 of the spec lists all of them.

If you want more details of how the decoding happens, you can dive into the following section:

Field table entry Example

The best way to make sense out of this is via an example.

Let’s suppose we want to send {"amount": 300} as the headers property.

In AMQP, the dictionary itself is called a field-table, and each key-value pair inside it is a field-value pair.

So for the amount=300 field-value pair, the byte sequence in the frame is:

06 61 6D 6F 75 6E 74 55 01 2C

Let’s go over how this gets interpreted, step-by-step:

First, it reads 06, which tells it that the key of this entry has a length of 6 bytes.

This leads it to reading 6 bytes to get the key name, those 6 bytes being: 61 6D 6F 75 6E 74.

These are the hexadecimal ASCII characters that decode to amount.

Next, it reads the type tag to determine the value that comes with the amount key.

Type tags are always 1 byte. Here, that byte is 55.

55 in hexadecimal corresponds to the ASCII character U, which means the value is a Short Int.

Short Ints are always 2 bytes, so AMQP reads the next 2: 01 2C, which evaluates to 300.

Why a Short Int and not something smaller? A Short Short Int (signed, 1 byte) has a max value of 127. 300 exceeds that, so the next type up is a Short Int, which uses 2 bytes.

Here’s an animation of how the decoding happens:

The discrepancy in codec

To transfer the messages, we used aio-pika — read from the source queue, tweak the headers, republish to the destination.

aio-pika delegates wire encoding to pamqp, and pamqp uses s as the type tag for 16-bit signed integers.

Celery, on the other hand, uses Kombu at the transport layer, which uses py-amqp for encoding and decoding.

Here’s the problem: pamqp and py-amqp disagree on what the s type tag means. That single-byte disagreement is where everything falls apart.

Suppose we want to send the following headers over the wire {"some-key": 300}, so by the time we’d want to encode the 300 value, the following happens:

pamqp sends the following 73 01 2C byte sequence over the wire:
- The type tag byte is 73 (hex), which is the ASCII character s — meaning signed 16-bit integer in pamqp’s convention
- Since it’s a 16-bit integer, 300 is encoded over the next 2 bytes: 01 2C
py-amqp receives the same 73 01 2C byte sequence:
- The type tag byte is 73, ASCII character s — but in py-amqp’s convention, s means short string
- For a short string, the next byte is the string length: 01 = 1 character
- It then reads 1 byte for the content: 2C, which is the ASCII character ,

That’s exactly what happened to us. The headers on our messages included a timelimit key with the value [None, 300] — None for the soft timeout, 300 for the hard timeout in seconds.

pamqp encoded 300 with the s tag. When py-amqp decoded it, it read s as “short string”, consumed \x01 as the length (1 character), then read \x2c as the content — which is ASCII for ,.

So timeout became ,, and Billiard blew up trying to do float + str.

Why does this discrepancy exist?

The obvious question is: who’s doing it wrong?

The answer is frustratingly: it depends on which spec you follow, and the specs contradict each other.

AMQP 0-9-1 introduced a set of field table type tags, but RabbitMQ (and Qpid before it) had already shipped their own extensions using the same byte values with different meanings. The two conventions conflict.

The RabbitMQ errata page documents this conflict explicitly. Here’s the relevant part of the type tag table:

Tag	0-9-1 spec	Qpid / RabbitMQ
`s`	short string	signed 16-bit
`U`	signed 16-bit	—
`S`	long string	long string

Worth noting: the original 0-9 spec didn’t define s or U at all — it only had S, I, D, T, F, V. Both tags were introduced in 0-9-1, but Qpid and RabbitMQ were already shipping their own extensions that reused the same byte values differently.

The errata puts it plainly:

In Qpid and Rabbit, s means a signed 16-bit integer; in 0-9-1, it means a short string.

And then:

RabbitMQ continues to use the tags in the third column.

So RabbitMQ’s broker uses s for signed 16-bit. pamqp deliberately follows this — its v1.5.0 changelog explicitly says it aligned field table type indicators to the RabbitMQ protocol errata. py-amqp went the other way and follows the 0-9-1 spec — s = short string, U = signed 16-bit — with no mention of the conflict anywhere in its codebase.

Neither library is being reckless. They each picked a side of a genuine spec conflict. The problem is they disagree on the one tag that affected our timelimit value, and the disagreement produces no error — just silently wrong data.

How we solved this

We swapped aio-pika out for pika to do the message transfer. pika has its own wire encoder and uses the U type tag for 16-bit integers — the same convention py-amqp expects on the receiving end, so the two sides finally agreed. Problem solved for new messages.

The already-corrupt messages in the queues needed a different approach:

Set a delivery limit policy in RabbitMQ
When messages hit that delivery limit, route them to a dead letter queue
Wrote a script to read from those DLQs, check the timelimit header, and fix the value from [None, ','] to [None, 300]
Redrive those messages back to the original destination queue

I guess the real lesson here isn’t “don’t use aio-pika” — it’s that AMQP’s fragmented spec history means two fully compliant-looking libraries can silently disagree at the byte level. If you’re ever mixing libraries across the publish/consume boundary, verify they share the same type tag conventions before you find out the hard way.

Making it easy to track my diet via a web app

Sun, 16 Aug 2026 12:25:56 +0100

I used an AI agent to build meal and workout plans that would help me lose weight and deployed it to Cloudflare pages

The problem

Back in my university days, I used to be super lean, workout everyday, and was able to remain motivated and stay fit.

As I was getting older, my metabolism started slowing down, and I had gotten injured a lot of times, and most importantly, got caught up in life.

All of these factors made me gain weight, and lose my motivation to get back in shape.

The difference was that back when I was younger, it was much easier to just go from slim to fit, as the process didn’t involve losing weight at all. It was just about training day after day, and the body responded quite nicely.

However now, I had to work my way backwards as I had gained over 20 Kgs from my university days, and trying to just raw-dogging it by trying to remember what I ate, ballpark estimate the calories and refrain from certain stuff was just not working out for me. I was seeing no results, and just kept on gaining weight.

I think we all know that diets are no fun at all, and I personally think that it’s super difficult to stay consistent, but to also remember what you should and should not eat, how much you should/must eat, but also include enough things to not get bored, not starve yourself out if you have a big appetite like I do, but also not feel like you’re in a military boot camp and you have a sergeant yelling constantly at you and treating you like his bitch.

Doing it the “conventional” way

Of course, this wasn’t the first time I had tried to do this, and I think it’s a “classic” problem that people try to solve.

It usually goes by making a google search on how to lose some weight, find some diet program, and try to follow that.

That stuff just doesn’t really work well with me, and I can’t really explain how, it felt like I needed some sort of process that just grouped it all in once place:

The list of stuff I need to buy
The meals I can prepare, with the portions
The workout plan, because the purpose was to keep the muscle mass, and lose fat

I tried doing it this way twice or three times before, but it had never worked out.

So this made me wonder: What if I built a custom app that would:

Give me daily meal plans, that change every day
Gives me a list of ingredients to buy once and for all, enough stock for a week.
Adapt it to my taste, so that it makes meals I’d be willing to eat.

I’m sure that there are apps that exist and do this already, but I figured this could be a fun project to do, and I write code for a living so why not ?

Moreover, we live in an era where agents can do this for us super easily, so why the heck not ?

Making a diet tracking app

The requirements were the following:

I wanted to maximize fat loss, and minimize muscle loss, without being too aggressive on myself
I wanted a diverse, but no so boring diet, meals that changed weekly
I wanted the meals to be easy to make from both a time/prep and ingredient perspective
I wanted the diet to be volume heavy, because I like to eat a lot
I wanted to avoid going on an ingredient hunt at all costs, so only stuff I can almost find everywhere
I wanted it to be high on fibers as well, for good transit
Include sweet treats from time to time, because who doesn’t like that

Now that I had the requirements in place, it was time to lay out how I imagined this would work:

I wanted a web app that only I could access, deployed on some domain I own
I wanted the app to list all the ingredients I need to buy for the entire week, as checkboxes I could tick to keep track
I wanted the list of meals to shuffle every day, and have a button to shuffle either the entire day meal plan, or individual meals should I not feel like it.

The above resulted in the following initial prompt:

Show/hide code

I want to build a web app that allows me to track my diet.

Here are the diet's requirements:

* I wanted to maximize fat loss, and minimize muscle loss, without being too aggressive on myself, so make it protein heavy
* I wanted a diverse, but no so boring diet, meals that changed weekly, preferably from a list of cuisines that I can chose: asian/italian/mexicain
* I wanted the meals to be easy to make from both a time/prep and ingredient perspective
* I wanted the diet to be volume heavy, but not high on calories, because I like to eat a lot
* I want the ingredients list to be super easy to find in most supermarkets
* Include sweet treats from time to time
* I wanted it to be high on fibers as well, for good transit. I have crackers that have these nutritional values per 100g: 46g carbs, 5g fat, 26g fiber. Include it in breakfasts and snacks

As for the app, i want it to do the following:

* I want a static site, deployed on cloudflare pages
* I want it so that only I could access it
* The app needs to allow me to change the weekly cuisine should I change my mind
* The app will list the weekly ingredients, with exact portions to buy. These might be persisted somwehre to be able to track what I bought and what I haven't yet, and resets every Sunday
* Randomize the meals per day, so that I don't eat the same thing everyday. And make it so that I can shuffle each meal individually should I feel like it.

For deployment, I had Cloudflare as my domain registrar, and I knew that they had a solution called Pages, so I thought this could be more than enough for what I wanted to do, and it fits nicely since it’s all configured in one place. As for persisting state, I used Cloudflare Worker KV, which is a simple key-value store.

For the curious ones, the application code can be found at the following repository.

This of course took a heuristic approach where I’d serve the content locally, test out the difference features before deploying and potentially fix issues.

Some of the issues were:

Clicking a checkbox would immediately uncheck itself: The click event was fired twice - once from the row and once from the input inside it. Fixed by checking the origin of the event: e.target.type === "checkbox" ? e.target.checked: undefined
The state was being saved, but not retrieved properly: This was a dumb AI moment where key construction was wrong, so just made it so that they to use to fetch data is as simple as the {section}/{date}, where section is either the ingredients list, workouts, other misc actions such as measuring my daily weight, etc.
Sometimes the app would deploy in preview mode: This was because the wrangler CLI would check the branch you’re in, and if it’s not the main one, it won’t deploy to production. I fixed this by having a deploy.sh script that would explicitly prompt me to which environment I’d like to deploy, regardless of the branch.
The deploy version was not being rendered properly, and that’s because it built a function that fetches the deployed version, but wasn’t rendering it.

Results

I’ve been doing this for 3 weeks now, and I’m losing the projected weight of 1Kg/Week.

The agent added a checkbox to check weight daily, but I felt that doing this on a weekly basis would yield greater satisfaction, so that’s a part I wasn’t using of the application.

There’s still a long way to go, because the overall weight I’m trying to lose is 10Kgs, so that’s 7 more weeks in total, but the road ahead seems solid as long as I keep my spirits high and do my best to follow the plan.

This was a fun project that I don’t regret at all, even though I hardly spent 2 hours in total on it, mostly giving feedback to the agent about what’s not working.

Whether this is worth it for others to do is a completely subjective topic, as things that don’t work for me can work pretty well for others, and vice versa.

And finally, for the curious, here’s what the application looks like.

Full application preview

Writing easily accessible text from your terminal

Fri, 17 Jul 2026 12:25:56 +0100

A shell function I wrote to stop reaching for Sublime Text every time I need to write a Slack message.

The Itch

The further I go in my career, the more obsessed I get with shaving steps off things that mildly annoy me. It’s a sickness, and I’ve made peace with it.

The Problem

At work, I juggle three things all day: my terminal, my IDE, and Slack.

Slack is the weak link. As a Vim user, editing messages is a pain — navigating text with arrow keys and no modal editing makes me feel like I’ve forgotten how to type. So I’d do this:

Open Sublime Text (vim mode, no save required)
Open a new tab
Write and polish the text
Select all, copy
Close the tab (maybe)
Send the message

Cumbersome. Four tools open just to write a Slack message.

So I wrote this:

Show/hide code

### in your ~/.zshrc

s() {
 local f
 f=$(mktemp /tmp/scratch.XXXXXX)
 vi "$f"
 cat "$f" | pbcopy
 rm -f "$f"
}

Now it’s:

Type s in the terminal
Write and edit the text in Vim
Close the file — it’s already in your clipboard
Send the message

“That’s it?”

Yes. Two steps fewer, one tool fewer, zero context switching.

I know it sounds absurdly small. But small friction compounds. Every time you reach for a separate tool, you break flow, lose a few seconds, and add just enough overhead that you sometimes just… don’t bother refining what you’re writing.

This little function removed that resistance entirely.

If you have a similar itch, here’s a challenge: what other tools in your workflow could be replaced by a scratch buffer and a copy to clipboard?

Testing Ansible Playbooks Locally

Sat, 03 Jan 2026 17:25:56 +0100

At Python Discord, we use Ansible to set up a part of our infrastructure.

A “bad” pattern we had is that we always tested these playbooks in production. I say “bad” because I think this can be a pretty controversial topic, and I’ve worked in a lot of companies who do this, or cheated their way around it, but that’s off-topic for now.

This led us to setting up a way to easily test our playbooks, but on a similar environment, without running the risk of unintentionally performing an undesired/destructive action or whatever.

Disclaimer

This post assumes familiarity with:

Ansible fundamentals (playbooks, roles, inventory)
Vagrant basics (managing VMs via Vagrantfiles)

I’ll explain concepts briefly as they come up, but having baseline knowledge will help you follow along more easily.

Prerequisites

At the time of writing this, I was on macOS Tahoe 26.1, using HomeBrew version 5.0.8-43-gfe0a384.

To do this, you’ll have to install:

virtualbox version 7.2.4,170995
vagrant version 2.4.9

Project overview

The project we’ll be working on in this tutorial is a made up version for the sake of simplicity.

The one we use at Python Discord is open source, and can be found in our GitHub repository.

The project root’s will be at the infra directory

└── infra/
 ├── Vagrantfile
 ├── playbook.yml
 ├── roles/
 │ └── hello_world/
 │ └── tasks/
 │ └── main.yml
 └── inventory/
 ├── hosts.yaml
 └── vagrant_hosts.yaml

Vagrantfile: The Vagrant configuration that we’ll use to configure our VMs to simulate our servers.
playbook.yml: The Ansible playbook that orchestrates all the roles/tasks that need to run on our infra
roles/hello_world/tasks/main.yml: The main task of the hello_world Ansible Role
inventory/hosts.yaml: The production hosts file
inventory/vagrant_hosts.yaml: The hosts to use when testing the playbooks locally with Vagrant

You can find the content of all of these files in the File content section

Setting Up Vagrant

Very quickly, Vagrant is a tool that allows you to configure/provision/manage virtual machines on your host in complete isolation.

This configuration and provisioning is done via a VagrantFile which allows you to write some sort of manifest in Ruby that’ll set up the virtual machines for you.

General box configuration

Vagrant allows you to have either “global” or per VM configuration. We actually need and will be using both, but the one in this section explains the global one.

Vagrant.configure("2") do |config|
 # This is the base image that will be used for all VMs
 config.vm.box = "bento/debian-13"

 # This is provisioning that will run on all VMs, but custom provisioning per specific VM is also possible. 
 config.vm.provision "shell", inline: <<-SHELL
 sudo apt-get update
 apt-get install -y python3 python3-pip sshpass
 SHELL
end

Configuring the control node/VM

The playbook/roles, etc. need to run from what Ansible calls the control node.

The following is the Vagrant configuration to create that node.

Vagrant.configure("2") do |config|
 # ...
 # Global provisioning seen previously has been omitted to focus solely on the control node's config

 # Define `control` as the primary VM

 config.vm.define "control", primary: true do |control|

 # Copy the content of the folder running `Vagrant` into control's `/infra` folder
 # This allows us to have the playbooks/roles that will run from the control node
 config.vm.synced_folder ".", "/infra", type: "rsync"

 # We assign a static IP for all VMs to easily identify them
 control.vm.network "private_network", ip: "192.168.56.2", virtualbox__intnet: true
 control.vm.hostname = "control"

 # Install Ansible on the control node, required to run the playbooks 
 control.vm.provision "Install Ansible", type: "shell", inline: <<-SHELL
 apt-get update
 apt-get install -y ansible
 SHELL

 # SSH configuration to allow the control node to connect to all managed nodes effortlessly
 control.vm.provision "setup_ssh", type: "shell", privileged: false, run: "never", inline: <<-SHELL
 ssh-keygen -t rsa -N '' -f ~/.ssh/id_rsa <<< y
 ssh-keyscan 192.168.56.3 >> ~/.ssh/known_hosts
 sshpass -p vagrant ssh-copy-id 192.168.56.3
 SHELL

 # Specify the provider to use for creating the VM 
 control.vm.provider "virtualbox" do |v|
 v.name = "control_node"
 end
 end
end

SSH Provisioning

You might have noticed that setup_ssh that is configured to never execute upon running vagrant up.

The reason we have it in the first place is to allow the control node to easily connect over SSH to hopper.

We do this by:

Including hopper in the list of control’s known hosts
Copying control’s public key over to hopper

However, it’s set to never because we want to run the provisioning when both control and hopper are up and running, since it requires fetching hopper’s host key, and adding control‘ls public key to hopper

Managed node’s configuration

The actual machines being configured by Ansible are called the managed nodes.

In our case, we will be configuring one VM only to keep things simple.

If you want to manage and configure multiple VMs, the principle is the same, you just need to replicate the configuration for this one, and alter the assigned static ip.

Vagrant.configure("2") do |config|
 # ...
 # Global provisioning seen previously has been omitted to focus solely on the control node's config

 config.vm.define "hopper" do |hopper|

 # We assign a static IP for the managed node to easily identify it
 hopper.vm.network "private_network", ip: "192.168.56.3", virtualbox__intnet: true
 hopper.vm.hostname = "hopper"
 # Disable any folder syncing, as the machines are supposedly fresh/empty
 hopper.vm.synced_folder '.', '/vagrant', disabled: true

 # Chose the provider to create the VM. 
 hopper.vm.provider "virtualbox" do |v|
 v.name = "hopper"
 end
 end
end

This one’s quite simple, we’re basically just creating a fresh VM, and assigning a static ip of 192.168.56.3 to it so that we can easily configure and target it in Ansible’s host inventory.

Complete VagrantFile content

This is included here just to see what the Vagrantfile will finally look like once we’ve combined all the previously explained sections.

Show/hide code

Vagrant.configure("2") do |config|
 config.vm.box = "bento/debian-13"
 config.vm.provision "shell", inline: <<-SHELL
 sudo apt-get update
 apt-get install -y python3 python3-pip sshpass
 SHELL

 config.vm.define "control", primary: true do |control|
 config.vm.synced_folder ".", "/infra", type: "rsync"
 control.vm.network "private_network", ip: "192.168.56.2", virtualbox__intnet: true
 control.vm.hostname = "control"
 control.vm.provision "Install Ansible", type: "shell", inline: <<-SHELL
 apt-get update
 apt-get install -y ansible
 SHELL

 control.vm.provision "setup_ssh", type: "shell", privileged: false, run: "never", inline: <<-SHELL
 ssh-keygen -t rsa -N '' -f ~/.ssh/id_rsa <<< y
 ssh-keyscan 192.168.56.3 >> ~/.ssh/known_hosts
 sshpass -p vagrant ssh-copy-id 192.168.56.3
 SHELL

 control.vm.provider "virtualbox" do |v|
 v.name = "control_node"
 end
 end

 config.vm.define "hopper" do |hopper|
 hopper.vm.network "private_network", ip: "192.168.56.3", virtualbox__intnet: true
 hopper.vm.hostname = "hopper"
 hopper.vm.synced_folder '.', '/vagrant', disabled: true
 hopper.vm.provider "virtualbox" do |v|
 v.name = "hopper"
 end
 end
end

Setting Up The Virtual Machines

Enough with the boring configuration stuff. It’s now time to bring those VMs up and running in order to be able to use them.

This is actually quite simple, and requires running the following two commands from within the infra directory

vagrant up
vagrant provision control --provision-with setup_ssh

Running The Playbooks

Once the VMs are provisioned, we can now move on to testing our Ansible playbooks.

First, we ssh into control

vagrant ssh control

Then, move into the infra folder.

cd /infra

Finally, we run the playbooks

ansible-playbook playbook.yml --inventory inventory/vagrant_hosts.yml

You can verify that the hello_world role ran by running the following commands

vagrant ssh hopper -c "cat ~/hello.txt"

This will display “Hello World” in the console, proving the role ran successfully

It’s important to note that whenever you change something, e.g. the roles, the playbook, etc. you’re gonna have to sync those into the VM by running vagrant rsync.

Cleanup

When you’re done testing or want to start fresh, simply destroy all VMs by running:

vagrant destroy -f

Final thoughts

While I was writing this, it came to my knowledge that there is an Ansible provisioner that can be used to do all of this but differently.

However, I still wanted to demonstrate how we actually did it back then.

I must say that this is obviously not the only way to test playbooks locally, but it is the one we decided to use back then.

I might make updates to this post at some point to illustrate how we could the Ansible provisioner instead, or maybe even test the playbooks on a docker container instead of using Vagrant.

File contents

This section exists to display the contents of all the different files that take part of this tutorial, in case someone wants to test this end to end.

Playbook.yml

Show/hide code

- name: Setup infra
 hosts: all
 roles:
 - hello_world

roles/hello_world/tasks/main.yml

Show/hide code

- name: Create hello world file in user's home directory
 ansible.builtin.copy:
 content: "Hello world"
 dest: "{{ ansible_env.HOME }}/hello.txt"
 mode: 'a=r'

vagrant_hosts.yml

Show/hide code

all:
 hosts:
 hopper:
 ansible_host: 192.168.56.3
 ip: 192.168.56.3
 access_ip: 192.168.56.3

Upgrading RabbitMQ To v4.x Without Breaking Celery ETA Tasks

Sun, 21 Dec 2025 13:25:56 +0100

Upgrading to RabbitMQ v4 threatened to break our entire usage of Celery, more specifically tasks with ETAs.

At 8M messages/day with zero downtime tolerance, we needed a migration strategy that preserves delayed task execution while switching from classic to quorum queues.

Prerequisites

This post assumes familiarity with:

Celery task queues (workers, tasks, brokers)
RabbitMQ fundamentals (exchanges, queues, routing)
Message queue concepts (producers, consumers, brokers)
Virtual hosts (multi-tenancy in RabbitMQ)

If you’re comfortable with distributed task processing in Python, you’re good to go.

Context

At Kraken, we use Celery to offload long-running tasks to workers via RabbitMQ (v3.13.7.1). Our platform team needed to upgrade to RabbitMQ v4.2.2, which introduces breaking changes:

Global QoS removal - ETA/countdown tasks now block workers until execution time
Classic Queue Mirroring removal - forced migration to Quorum Queues

Challenges

Can’t switch queue types on the fly: RabbitMQ doesn’t let you change a queue’s type after it’s created. Normally you’d just delete the queue and recreate it with the new type, but that wasn’t an option for us. We’re pushing 8M messages/day across many environments, and our SLAs don’t allow for any downtime.

Workers would block indefinitely: Without global QoS, any task with an ETA would block a worker until that ETA arrived, completely defeating the purpose of async processing. Fortunately, Celery has a feature called Native Delayed Delivery that solves this—but it requires quorum queues bound to topic exchanges. Lucky for us, that’s exactly what we needed to migrate to anyway.

Solution

Our migration strategy:

Create a new vhost (qhost), which will host quorum queues bound to topic exchanges
Configure application code to support both queue types via a feature flag
Transfer messages from old vhost (chost) to new vhost without losing ETA information
Decommission chost
Rolling upgrade to RabbitMQ v4

Phase 1: New Virtual Host

The first step was straightforward: create a new virtual host (qhost) to run alongside our existing one (chost). For us, this meant updating some infrastructure manifests to provision the new vhost. Not the most exciting part, but essential for what comes next—we needed both vhosts running simultaneously to avoid any downtime.

Phase 2: Feature-Flagged Queue Configuration

Next, we needed to make our application code flexible enough to handle both the old and new queue configurations. We introduced a USE_QUORUM_QUEUES environment variable (a common feature flag pattern) to control which type of queues to create:

from kombu import Queue, Exchange
from django.conf import settings

def build_queue(queue_name: str) -> Queue:
 queue_type = "quorum" if settings.USE_QUORUM_QUEUES else "classic"
 exchange_type = "topic" if settings.USE_QUORUM_QUEUES else "direct"

 return Queue(
 name=queue_name,
 exchange=Exchange(queue_type, type=exchange_type),
 queue_arguments={"x-queue-type": queue_type}
 )

task_queues = [build_queue("first_queue"), ..., build_queue("last_queue")]

When we deployed this change, Kubernetes did its usual rolling update, and pods gradually switched from chost to qhost. This meant we temporarily had some pods on the old vhost and some on the new one, which was totally fine. Any stragglers would get caught in Phase 3.

Phase 3: Message Transfer with ETA Transformation

Here’s where things got interesting. We needed to transfer potentially millions of messages from chost to qhost without losing any data or causing downtime.

At first glance, this sounds like a perfect job for RabbitMQ’s Shovel plugin, right? Just copy messages from one vhost to another. Unfortunately, a shovel copies messages as-is, including the eta header. That’s exactly what we’re trying to avoid—those headers would cause workers to block, bringing us right back to square one.

Instead, we had to transform messages during transfer. The idea was to replicate what Celery does internally when Native Delayed Delivery is enabled: extract the ETA header, calculate the appropriate delay-based routing key, and route to the right exchange. If you want to understand how this works under the hood, this guide explains the mechanics well.

Here’s the core logic (simplified for clarity):

Show/hide code

import pika # RabbitMQ Python client: <https://pika.readthedocs.io/>
from kombu.transport import native_delayed_delivery as kombu_utils

def get_routing_details(method, properties, queue_name):
 target_exchange_name = method.exchange or queue_name
 target_routing_key = method.routing_key or queue_name

 eta_str = str(properties.headers.pop("eta", ""))
 countdown_in_seconds = compute_countdown(eta_str)

 if countdown_in_seconds and countdown_in_seconds > 0:
 target_routing_key = kombu_utils.calculate_routing_key(int(countdown_in_seconds), target_routing_key)
 target_exchange_name = "celery_delayed_27"

 return target_routing_key, target_exchange_name

chost_connection_string = read_from_env("CHOST_CONNECTION_STRING")
qhost_connection_string = read_from_env("QHOST_CONNECTION_STRING")

source_channel = pika.BlockingConnection(pika.URLParameters(chost_connection_string)).channel()
dest_channel = pika.BlockingConnection(pika.URLParameters("amqps://user:pwd@host:port/qhost")).channel()

for method, properties, body in source_channel.consume("target_queue"):
 try:
 routing_key, exchange = get_routing_details(method, properties, "target_queue")
 dest_channel.basic_publish(exchange=exchange, routing_key=routing_key,
 body=body, properties=properties)
 source_conn.channel().basic_ack(delivery_tag=method.delivery_tag)
 except Exception:
 source_conn.channel().basic_nack(delivery_tag=method.delivery_tag, requeue=True)

Some notes on the shoveling part

The above is pseudocode to illustrate the concept. Our production version uses aio_pika for async I/O (to avoid blocking), multiprocessing to handle high throughput, message backups for disaster recovery, and extensive logging to track everything.
The script was deployed to run as a daemon, and would only work if USE_QUORUM_QUEUES was set to True.
More mechanics were in place to determine if there were any messages to transfer in the first place, do transfers in batches, etc.

Phase 4 & 5: Cleanup and Upgrade

Once all messages were safely transferred to qhost, our platform team took over for the final steps. They deleted the chost virtual host (which removed all the classic queues that are incompatible with v4), and then performed a rolling upgrade to RabbitMQ v4.2.2.

And that was it! We successfully migrated from RabbitMQ v3 to v4 with zero downtime, while preserving ETA task behavior and handling 8M messages/day across all our environments.

About

Mon, 01 Jan 0001 00:00:00 +0000

My name is Amrou Bellalouna, and I’m a Software Engineer at Kraken Technologies.

Community Work

Beyond my professional work, I’m deeply involved in the Python Discord community:

Core Developer - contributing to the development and maintenance of the platform that serves the largest Python community on Discord
DevOps Team Member - helping maintain infrastructure and ensure smooth operations for thousands of community members

Python Discord is a community where developers of all skill levels come together to learn, share knowledge, and collaborate.

If you’re interested in Python, I highly encourage you to join us!

What I Write About

I will try to use this space to share some of the technical lessons I ran into on my path, or some thoughts, whenever I have any.

Get In Touch

I’m always happy to connect further beyond this blog. Feel free to reach out if you:

Have questions about something I’ve written
Want to discuss Python, DevOps, or software architecture
Are interested in contributing to open source
Just want to chat about technology!

You can find me on:

GitHub: @shtlrs
LinkedIn: amrou.bellalouna
Python Discord: Look for me in the community!

Amrou Bellalouna

How one single byte lead to silent Celery task corruption

AMQP frames

The headers property

Field table encoding

The discrepancy in codec

Why does this discrepancy exist?

How we solved this

Making it easy to track my diet via a web app

The problem

Doing it the “conventional” way

Making a diet tracking app

Results

Writing easily accessible text from your terminal

The Itch

The Problem

“That’s it?”

Testing Ansible Playbooks Locally

Disclaimer

Prerequisites

Project overview

Setting Up Vagrant

General box configuration

Configuring the control node/VM

SSH Provisioning

Managed node’s configuration

Complete VagrantFile content

Setting Up The Virtual Machines

Running The Playbooks

Cleanup

Final thoughts

File contents

Playbook.yml

roles/hello_world/tasks/main.yml

vagrant_hosts.yml

Upgrading RabbitMQ To v4.x Without Breaking Celery ETA Tasks

Prerequisites

Context

Challenges

Solution

Phase 1: New Virtual Host

Phase 2: Feature-Flagged Queue Configuration

Phase 3: Message Transfer with ETA Transformation

Some notes on the shoveling part

Phase 4 & 5: Cleanup and Upgrade

About

Community Work

What I Write About

Get In Touch

The `headers` property