Our batch scheduler is really a load balancer

Most of what Norman does, you can see. You open the app, a transaction is categorized, a receipt is matched, a VAT return is ready. What you do not see is the night shift. While Germany sleeps, a fleet of background workers balances two very different kinds of tenant on the same hardware: patient, network-bound bank syncs across thousands of connections, and hungry, compute-bound AI jobs that read receipts, categorize a constant stream of new transactions, and replay yesterday's agent evals. None of that is glamorous, and none of it is where I expected the hard problems to be.

The hard problem is that these two tenants want the same machine at the same time, and they stress it in completely different ways. A background scheduler looks like a list of "run this overnight" entries. In practice it is a load balancer wearing a cron hat: its real job is to keep any one workload, and it is almost always the AI one, from starving the others. The boring decisions about what runs next to what are most of what keeps the mornings quiet.

The window is a shared resource

Here is the failure mode we walked into. Our heaviest recurring jobs had all drifted into the same narrow window in the small hours, because that is the obvious place to put "nightly" work. Inside that one window the schedule looked like this:

• update_all_transactions, which fans out an open-banking sync across every connected account, thousands of them, each a slow call to an external aggregator.
• update_accounts, a balance refresh, more external calls.
• categorize_uncategorized_transactions, the heavy AI and OCR post-processing sweep, sitting right on top of it.
• sync_all_active_integrations closing the window.

The two peaked against each other, and they are opposites. The open-banking fan-out is patient and slow, spending its time waiting on someone else's API. The AI sweep is impatient and hungry: it wants workers and model capacity now. In the same window they do not add up, they interfere. The sync jobs hold workers hostage waiting on I/O while the AI jobs queue behind them.

The fix was almost embarrassing in its simplicity, and I will come back to why. But you cannot make it safely until the work is separated by kind, so that is where the real design is.

First, split the work by what it waits on

The single most useful thing we did was stop treating "a background job" as one category. A job that waits on a bank API and a job that waits on a language model fail differently, retry differently, and starve each other if they share a worker pool. So they do not share one.

Work is routed onto dedicated queues by what it waits on, each with its own worker pool:

# Route each task to a queue by what it waits on (illustrative):
app.conf.task_routes = {
    # user-facing AI / OCR / document work
    "categorize_transaction": {"queue": "transaction_postprocess"},
    "extract_data_from_attachment": {"queue": "transaction_postprocess"},

    # open-banking API calls: network-bound and slow
    "update_transactions_by_bank_account": {"queue": "transaction_sync"},

    # background agent prep vs. user-visible agent actions
    "run_evals": {"queue": "agent_batch"},
    "submit_approved_run": {"queue": "agent_realtime"},
    # ...
}

The distinctions that matter are the traffic profiles, not the names. transaction_sync is full of jobs that are cheap for us and slow because of someone else, so its pool holds many jobs mostly-idle on a socket. transaction_postprocess is the opposite: expensive per job, and you do not want a flood landing at once. The two agent queues encode a priority split rather than a resource one: agent_batch is preparation nobody is waiting for, agent_realtime is the moment a human clicked "submit" and is watching a spinner. Routing them apart means a nightly eval replay can never sit in front of a filing a person is trying to send.

Once the lanes exist, scheduling becomes a question of which lanes you light up together.

Then, do not schedule two heavy things at once

With the work separated, the actual scheduler change was a few lines of cron. We moved the AI and agent trio out of the banking window entirely:

"categorize_uncategorized_transactions": {
    # Moved out of the open-banking window so the heavy AI/OCR post-process
    # sweep doesn't peak at the same time as the sync fan-out; it now runs in
    # a quieter slot later in the night.
    "schedule": crontab(hour=..., minute=...),
    "options": {"expires": ...},  # a generous window, an hour or two
},
"promote_findings": {
    "schedule": crontab(hour=..., minute=...),  # just before run_evals
    "options": {"expires": ...},
},
"run_evals": {
    "schedule": crontab(hour=..., minute=...),  # nightly eval replay
    "options": {"expires": ...},
},

The AI sweep now runs in a quieter slot later in the night, after the overnight housekeeping audit and comfortably before the morning invoice send. The banking window keeps its early slot to itself. Nothing about the jobs changed; they just stopped colliding.

I will be honest that this is not a clever change. It is a schedule edit. It is only available as a schedule edit because the less visible work below, the queue split and the retry semantics, made every one of these jobs safe to move by a couple of hours. When jobs are entangled, moving one is a risk assessment; when they are isolated and idempotent, it is a one-line diff. Most of the value is in earning the right to make boring changes.

expires: a late job is a job you do not want

Look again at those entries and you will see every one carries an expires window, measured in hours. That is not decoration. It is the answer to a specific question: what should happen if a scheduled job is dispatched but the workers are too busy to pick it up in time?

The wrong answer is "run it whenever a worker frees up." A nightly categorization sweep is useful when it fires, in the small hours. If the pool was saturated and the same sweep finally gets picked up in the middle of the day, it is now competing with live user traffic to do work that a later run will redo anyway. A stale batch job replayed into the middle of the day is worse than a batch job skipped.

So beat entries expire. If the scheduled job is not started within its window, the broker drops it and the next scheduled run handles the backlog. expires turns "eventually" into "now, or not at all," which is exactly the contract you want for periodic work. The windows are deliberately generous, so a normal shift fits and only a genuinely stuck pool triggers the drop.

Checking the load, continuously

A load balancer that cannot see load is just a static router. So the scheduler runs a probe on a short cycle that reads the actual depth of every workload queue, not a single "is the box busy" number:

QUEUE_SIZE_THRESHOLD = ...  # a few hundred pending messages
MONITORED_QUEUES = [
    "transaction_sync",        # bank-sync fan-out
    "transaction_postprocess", # AI / OCR
    "agent_batch", "agent_realtime",
]

@app.task(name="monitoring.check_queue_health")
def check_queue_health() -> None:
    """Alert when any workload queue backs up."""
    for name in MONITORED_QUEUES:
        if queue_depth(name) > QUEUE_SIZE_THRESHOLD:
            logger.critical("Queue %s is backing up", name)
    # ...and a separate watch on the dead-letter queue

Two things matter here. First, it reads each workload separately, so we never just see "the system is busy," we see which tenant is under pressure: the bank-sync lane stalled on a slow aggregator, or the AI and OCR lane grinding through a backlog. You cannot tell those apart from a single CPU graph, and they call for opposite responses. Second, it watches a dead-letter queue, where tasks land after failing one too many times, because a growing dead-letter count means something is not merely slow but broken.

Today the probe's job is to page a human. But it is also the sensor a smarter scheduler needs. You cannot balance a load you cannot measure, and this is the part that measures it, per workload, continuously.

At-least-once means idempotent or bust

The other thing that makes jobs safe to reschedule is what happens when one dies mid-flight. We run the workers with two settings that trade one problem for a better one:

task_acks_late=True,
task_reject_on_worker_lost=True,

acks_late means a job is only acknowledged after it finishes, not when it is picked up, so a worker that crashes halfway does not silently lose the job. reject_on_worker_lost puts that job back on the queue. Together they buy you at-least-once delivery: nothing gets dropped on a crash. The price is that a job can occasionally run twice, once on the worker that died and once on its replacement.

Which means every job has to be safe to run twice, and that invariant quietly shapes the whole system. The AI agent layer stamps each run with an idempotency_key, so a replayed model call resolves to the existing run instead of a fresh duplicate, across the tens of thousands of agent runs we have recorded. The bank-sync path dedupes at the account level, with a per-account lock on each dispatch, so aggressive scheduling never turns into aggressive double-work. The point of at-least-once delivery is that you stop preventing duplicates at the queue and instead make them harmless at the job.

Assume it will get stuck, and build the broom

The last category of scheduled job is the one that exists purely because the others will misbehave. Distributed batch work does not fail cleanly; it gets stuck. A worker dies between two commits, an external call hangs, a lock is held by a process that no longer exists. So a good chunk of the schedule is janitorial:

• recover_stuck_syncs runs on a regular sweep and un-wedges bank accounts that got stranded mid-sync.
• requeue_stuck_runs recovers agent runs whose worker task was lost.
• update_all_transactions runs several times a day, not because banks change that often but as a safety net that re-drives accounts a previous pass left behind.

None of these do anything on a good night. They are the broom in the closet. But the assumption behind them, that any single run can and will fail, is what lets the happy path stay simple: no job has to be bulletproof, because a sweeper will come along and re-drive whatever it left in a bad state.

Toward a scheduler that balances itself

Everything so far is deliberately static. The queues are fixed, the cron minutes are hand-picked, and when the AI sweep collided with the bank-sync window, a human noticed and moved it. That works, and boring-and-correct beats clever-and-fragile every time. But you can see the shape of the next step from here.

We already have the two halves an adaptive balancer needs. We can sense load: the queue-depth probe reads live pressure per workload and watches the dead-letter queue for breakage. And we can place load: the routing table decides where a job runs, the schedule decides when. The missing piece is the controller in between, the part that reads the signal and moves the work without waiting for a person to notice.

The direction we are exploring is a scheduler that treats the AI and OCR jobs as elastic tenants. Instead of a fixed "run the categorization sweep at this minute," it would ask, at dispatch time, which window has headroom: hold the heavy compute-bound work back while the bank-sync fan-out is peaking, and release it into the next quiet slot. That is the decision we made by hand this month, made continuously from live load. The hard part of an adaptive load balancer was never the placement policy; it is being able to move work around without breaking anything, and that groundwork is done. Every job is idempotent, so the controller can reshuffle freely; every job has an expires window, so a misplaced one is dropped rather than run stale; every workload is isolated, so shifting one never starves another. The smart part is small. The boring part underneath it is the whole product.

The boring parts are the product

The whole scheduler is a couple hundred lines of Python: a dictionary of cron entries, a routing table, and a probe that reads queue depth. The change that prompted this post moved a handful of those entries into a quieter window. If you were looking for the AI in an AI accounting product, the model that reads your receipts is only the loudest tenant here, not the hard part.

The hard part is the layer around it: whether that model actually runs before your morning, whether a filing you are trying to submit waits behind a nightly eval, and whether a worker crash at the wrong moment quietly re-files something it should not. The interesting engineering in a system like ours is rarely the model call. It is the unglamorous balancer that decides when the model runs, what it runs next to, and what happens when it doesn't. We spend a lot of time on that balancer, precisely because you should never have to think about it.