Six hours a day to thirty minutes

The task as it existed

Someone opened 32 different portals every morning, searched each one, read the results, and typed the relevant entries into a spreadsheet. Six hours, every working day. Tedious, and — as with all tedious work — error-prone in ways nobody could measure, because there was no second pass to compare against.

The brief was to automate it. The unstated requirement, which turned out to be the real one, was that the output had to be good enough that the person stopped double-checking it. Automation that still needs full manual verification saves nobody any time.

Why 32 portals is not 32× one portal

Each portal was built by a different team at a different time with different assumptions. Across the set I dealt with:

• Server-rendered tables, JavaScript-rendered tables, and one that rendered results into a PDF
• Dates written five different ways, including two ambiguous ones where 03/04 could be March or April
• Names in inconsistent case and order, sometimes with titles attached, sometimes not
• Pagination by page number, by offset, by cursor, and one with no pagination that just returned everything
• Session handling ranging from none to aggressive, with timeouts as short as a few minutes

Writing 32 bespoke scripts would have worked for about three weeks, until portals started changing and I had 32 things to maintain.

The structure that made it maintainable

I split the system into three layers, so that portal-specific mess stayed contained:

# 1. adapters — the only place portal weirdness lives
class PortalAdapter:
    def search(query) -> raw_rows
    def paginate() -> more_rows

# 2. normaliser — one schema, one set of rules
raw_rows → clean_records   # dates, names, IDs, whitespace

# 3. sink — storage + change detection
clean_records → MySQL       # dedup, diff against yesterday

Each portal only ever had to implement the adapter. Everything downstream — normalisation, deduplication, storage, reporting — was written once and shared. When a portal changed its layout, I fixed one adapter and nothing else moved.

Normalisation is where the value is

This was the part I underestimated. Getting the data out is a solved problem. Getting 32 sources to agree on what a date is, what a name looks like, and when two records are the same record — that's the actual work.

Ambiguous dates were the sharpest edge. 03/04/2025 means different things on different sites, and there is no way to resolve it from the string alone. I handled it per-portal, with the format declared explicitly in the adapter rather than guessed at parse time. Guessing would have been faster to write and would have silently corrupted months of data.

Resilience over speed

I optimised almost nothing for throughput. Thirty minutes was already a twelve-fold improvement, and nobody needed it in five. What they needed was for it to finish every day.

So the design assumed failure everywhere:

• Retry with backoff on every network call, because portals go down and come back
• Per-portal isolation — one portal failing never aborted the run, it just got flagged in the report
• Proxy rotation and human-paced request timing, because hammering a government portal from one IP is both rude and self-defeating
• Checkpointing, so a crash at portal 28 didn't mean redoing portals 1 through 27
• An explicit run report listing what succeeded, what failed and what looked unusual

Earning the handover

For the first stretch, the system ran alongside the manual process and I compared the two outputs daily. Every discrepancy was either a bug on my side or — more often than I expected — a manual miss that the automation had caught.

That parallel period was what made the switch possible. Not a demo, not a metric in a deck; a couple of weeks of the automated output being visibly correct against the thing it was replacing. The thirty minutes that remained were spot-checking the flagged rows, which is the right amount of human attention for this kind of task.

Metric	Before	After
Daily effort	~6 hours	~30 minutes
Sources covered	32+ manually	32+ automated
Reduction in manual work	—	~90%
Failure visibility	none	per-run report

What I'd do differently

I'd build the run report first, not last. It was the feature that created trust, and I added it well after the scrapers. Building it early would have made my own debugging faster too.

I'd write adapter contract tests from day one. A tiny saved fixture per portal plus a test asserting the adapter still extracts the right fields would have caught layout changes immediately, instead of the following morning's run.

I'd separate "portal changed" from "no results today" in alerting. Early on both looked the same from the outside, and telling them apart cost me more time than either problem deserved.