Webhooks at Scale: What We Learned Sending 10M/Day
We send about 10 million webhooks per day now. Started at 100K/day a year ago. Thought it would scale fine. It didn’t. Here’s everything that broke and how we fixed it. The naive implementation def send_webhook(event_type, payload, url): try: requests.post( url, json={"event": event_type, "data": payload}, timeout=5 ) except Exception: # ¯\_(ツ)_/¯ pass Call this function whenever something happens. Fire and forget. What could go wrong? Problem 1: Blocking the main thread Webhook delivery was part of the request cycle. User creates an order, we save it, then send webhooks to their configured endpoints. If their endpoint is down or slow, the user’s request times out. ...