Getting Rate Limited for Fun and Profit

When we set out to add a first-class email client to Macro last year, we started with Gmail support.

To make email a first-class entity in Macro — linked with docs and agents, in unified search, with native mentions and powerful filtering and sorting — we needed the user's email in our own database. Which means that on sign-up, we needed to backfill the user's inbox as quickly as possible.

Hitting the quota limit

Gmail's API has a robust usage limit system, as detailed in their docs, and when trying to pull down a user's inbox at light speed, we quickly ran into the 6,000 quota units per minute per user limit. As soon as you hit that limit, the API starts returning 429 Too Many Requests until it resets.

Our backfill process is a queue-driven bulk-sync job. Each operation in the backfill is represented by a single message on the queue, which makes the retry mechanism for usage limits pleasantly simple: when Gmail returns a 429, we just don't ack the message responsible for the call. The message is automatically retried after its 60-second visibility timeout, by which point Gmail's usage limit has reset and the data comes back successfully.

Email backfill was the first thing I built in the backend, and it worked swimmingly. With this strategy, we were confident we were backfilling inboxes as fast as Gmail would let us.

No headroom left for anything else

You may have already guessed the problem we ran into as soon as we started implementing other Gmail functionality: there were no quota units left for anything else during a backfill. Want to send an email? 429. Want to mark a thread as read? 429. We could read emails as they were backfilled, and that was about the extent of it.

It became clear we would have to preserve some headroom before the usage limiter kicked in. And how do you do that? By building your own usage limiter, of course.

Building our own usage limiter

As seen in rate_limit.rs, the email backfill job uses a cost-based sliding window. Redis was the perfect fit, because it lets the entire usage limit check — expiring old entries, summing current usage, evaluating it, and storing the result — happen as a single atomic operation. Any number of pub/sub workers can run checks concurrently without racing past the limit.

The usage limiter is fully configurable, so we can dynamically set the percentage of available quota units that backfill is allowed to spend, leaving the rest for normal inbox operations in the Macro UI. Problem solved.

Round two: bulk operations

That is, until we added bulk email operations to Macro. Up until this point, every operation acted on a single email message at a time. Once we added the ability to act on many emails at once, we ran into the dreaded 429s all over again.

Note that this isn't a third-party-only problem. Gmail's own UI acknowledges the usage limit when you try to archive or mark a large number of emails at once:

The only way to confidently process large numbers of operations that are guaranteed to hit the usage limit is either to bottleneck requests or to have a solid retry mechanism in place. We already had the latter from the backfill job, so we took a page out of that book: all Gmail inbox operations are now done asynchronously. After updating our database, we send a message to a queue for the Gmail operations worker pool, which makes the request to the Gmail API. If that request is usage-limited, the message is auto-retried after the 60-second visibility timeout, exactly like a backfill message.

In summary

After some trial and error, the best strategy for working around Gmail's usage limit — for both backfilling and everyday inbox management — turned out to be queue-based systems with auto-retry through visibility timeouts, sitting behind a limiter that reserves headroom for the things a user is actually doing right now.

Now to do it all again with Outlook.