Integration and syncing
Connectors at scale
Syncing many sources into a single destination, with queues, retries and observability, so you always know what came in, what failed and why.
Who it's for Marketplaces, branch networks and platforms integrating dozens of suppliers or locations at once.
The problem
You can eyeball a single point-to-point integration. You can't eyeball twenty. When every supplier has its own ERP, its own data quality and its own upload schedule, errors stop being incidents and become background noise.
Without a queue per source and a board watching it, answering "why didn't this catalog publish?" takes hours, and the answer usually arrives after the customer has already complained.
How I solve it
01One queue per source
Each supplier writes to its own queue. A worker consumes it in batches every few seconds and publishes to the destination. If a source goes down or sends garbage, the problem stays contained in its queue instead of dragging the others down.
02Explicit, batched operations
Create, update and delete are distinct, declared operations processed in batches. That means reprocessing only what failed instead of pushing the whole catalog again, and it leaves a trail of what happened to each record.
03Data quality diagnostics
Most rejections aren't connector bugs: they're malformed data from the source. The system classifies and reports them by name — duplicate codes, mis-declared encoding, empty required fields — so the supplier can actually fix them.
04A board watching the queues
A monitoring dashboard shows volume, error rate and latency per source. "What's going on with this supplier?" is answered by looking at a screen, not by reading logs.
05Your own staging environment
New sources are tested against a staging database before touching production. The supplier validates its integration without any risk of polluting the real catalog.
What I've already built
- 40,367 operations processed with zero errors for one of the bookstores integrated into a book marketplace.
- 5,020 offers from a wholesale supplier published and kept in sync by cron against the master catalog.
- A diagnostic that identified 57% of rejections at another source caused by misuse of create vs. update, and 1,226 codes affected by a mis-declared character encoding.