PDF cache builder function causing rate limits to kick in from Intuit
This is a major issue with the new PDF cache system. If there are a lot of invoices refreshed or synced, a background job kicks off to fetch new refreshed copies of the PDF invoice. This happens asynchronously by several worker pools on the app servers. Intuit quickly enforces rate limiting which causes a ton of 429 status codes. This also results in our storing of the error in the cache field.
When this happens, there is little choice but to completely purge the PDF cache on that account and allow it to rebuild as invoices are requested. This defeats the purpose of having the cache in the first place.
There are two solutions to this problem, and likely both need to be implemented.
-
Comply with rate limits and throttle back our requests. This would involve a special flag we would set on invoices once the rate limit is hit. We would then reschedule the pdf cache build to a later time so it can try it again.
-
We will limit how far back PDF cache will go - unless specifically requested by a user or process. Hence, we would check the invoice aging. If it does not meet the criteria, we will purge any existing cache copy. If it meets the criteria, we will fetch a new copy.
For the time being, PDF cache has been disabled for background processing (sync). It will work though when invoices are specifically requested by a user.
Once the above items are developed and tested, we will re-enable caching on background jobs as well.
Issue metadata
- Issue type: Bug
- Priority: Medium
- Environment: Production