The client needed to know how their catalogue compared with two much larger competitors — product for product, on price. Done by hand it is not a project, it is a permanent job, and it is never finished because both catalogues keep moving.
Collection
The first problem is simply getting the data. Large retail catalogues are not designed to be read in bulk, so collection runs as a set of managed crawlers that work through categories methodically, in parallel where it is safe to and slowly where it is not. This is scheduled and repeatable, because a one-off scrape answers the question once and then rots.
Normalisation is the actual problem
Two companies rarely describe the same product the same way. One writes 24" x 36", the other 24 x 36 in. One puts the brand first, the other last. One calls it a carton, the other a box.
So the matching does not run on raw product names. A dedicated normalisation service — a separate Python component alongside the main application — parses names, pulls out attributes and dimensions, and converts everything into a comparable form first. Splitting it out kept the messy, iterative text work away from the application and let it be tuned without redeploying everything else.
Scoring, not guessing
Candidate pairs are then scored against a configurable weight profile: how much a title similarity counts, how much an exact attribute match counts, how much a dimension mismatch costs. The weights live in the database, not the code, so the people who understand the products can adjust the balance and immediately see what changes.
That matters more than it sounds. A matching system nobody can tune is a black box, and a black box that is confidently wrong about pricing is worse than a spreadsheet.
A person still decides
High-confidence pairs go through. Everything ambiguous lands in a review workbench where a human confirms or rejects it. The goal was never full automation — it was moving the human effort from comparing everything to arbitrating the few percent that are genuinely unclear.
Why this one matters
This is the shape most "automation" work actually takes: a collection pipeline, a normalisation layer, a scoring model somebody can adjust, and an interface that respects the fact a person still has to be accountable for the answer. It runs across two languages and two datastores because the job needed it, not because it was interesting.