Use Case: Price Monitoring with Proxies (Amazon, eBay, Retail Sites)

Price monitoring needs stability more than speed. This guide shows a safe architecture using rotation + retries + throttling to avoid bans and keep data fresh.

Recommended defaults

Concurrency: 5–20 per domain
Rotate: every 5–20 requests
Retry: 2–4 with exponential backoff
User-agent: realistic list
Cache: short TTL (5–30 minutes)

Simple retry logic idea

Retry only on:
- 429
- 403 (sometimes)
- timeouts
Use backoff: 1s → 3s → 7s

• If you see 403 quickly: rotate more often and reduce concurrency
• If 429: you are too fast; add delays and caching
• If HTML changes: use robust selectors and fallback extraction