Integration and syncing
Scraping & data
The data someone copies from someone else's website today — or that you pay for per query — extracted automatically, normalized and available in your own system.
Who it's for Operations that depend on information living in third-party portals with no API: suppliers, carriers, agencies, competitors.
The problem
A good share of what your operation needs lives on someone else's website: a shipment status, a supplier price, a filing at a government agency. There's rarely an API, and when there is, it's usually billed per query.
The usual alternative is someone logging into ten different portals several times a day and copying by hand. It's expensive, it's slow, and the information arrives late exactly when being late costs money.
How I solve it
01One adapter per source
Every portal has its own logic, its own login and its own quirks. I build an adapter per source and normalize everything into a single schema, so whoever consumes the data doesn't need to know where it came from.
02Anti-bot protections, solved case by case
There's no universal recipe. Some sources block datacenter IPs, others detect automated browsers. Each is solved with the technique it calls for, including driving a real browser when there's no other way. It's craft work, which is why it holds.
03Cross-checking sources to catch inconsistencies
When the same fact appears in more than one place, comparing them reveals what a single source hides. A date that doesn't match between two systems is usually the early warning of a problem you can still avoid.
04Scheduled refresh, with a watchdog
Data refreshes every few hours and a watchdog checks that sources keep responding. When a portal changes and breaks an adapter, you find out from an alert, not from a customer complaint.
05Data where you need it
The output isn't a loose file: it lands in a queryable database, with its own dashboard or exposed via API to the system you already use.
What I've already built
- 8+ shipping lines scraped 24/7 for a freight forwarder, with anti-bot protections solved one by one.
- Tracking coverage raised from 83% to 95% by solving datacenter IP blocking with a hybrid flow.
- Cross-checks against 3 port terminals to catch estimated-date inconsistencies before they turn into delays.
- Replaced a pay-per-query service, over a volume of ~100 containers a month.