So, the 2nd of November has been and gone, and Docker are now enforcing Docker Hub pull rate limits ... have you been caught out yet ? I know many have ...
One aspect of Docker Hub that is not well understood is that their "pulls" counter/metric is actually a representation of the number of times that a manifest for a particular image has been retrieved; It is not an indicator on the number of times an image has actually been downloaded.
If, for example, we take our own repository portainer/portainer-ce, we have over 77M pulls (we have 2Bn pulls of portainer/portainer), but i can tell you categorically, we don't have anywhere near that number of deployments (we have more like 70k deployments) ... so what causes the massive difference? Simple ... auto-updaters.
Auto-Updaters, such as watchtower or Ouroboros, are relatively rudimentary tools that facilitate a way to ensure Docker Containers running in your environment are always the latest version as per an image published on Docker Hub. These tools work by asking Docker Hub for the manifest of a container image, and then comparing the running container's image with the image available via the manifest.. and they do this (by default), every 5 minutes (think... are we there yet, are we there yet, and you get the gist). These tools get the job done and they allow organizations to implement a really simple CI/CD pipeline that triggers whenever a new container image is published. However, as mentioned above, this manifest download is counted as a "pull".
Historically, there was no real need to care about the number of pulls you (as a user or organization) were generating as they caused no harm, and served no real purpose (other than being a vanity number for organizations to quote, us included!).
However.. now that Docker have implemented these rate limits, every single pull you make counts towards your pull quota... so you need to make pulls count!
For a home user, this change is unlikely to really cause any impact, but for a company that uses auto-updaters in their environments, its likely to have a very negative effect.
Why? Two reasons... first, because the rate limits that Docker are enforcing for anonymous requests are per SOURCE IP, and second, even for authenticated users, the pull rate limit can quickly be exhausted through the use of auto-updaters.
Most organizations have a single internet POP for their access to the internet, and so anonymous pulls from that organization will appear (to docker) to come from the one IP... which means the rate limit that is applied is actually for the entire organization!!! so now the 100 anonymous pulls per 6 hours is actually 100 pulls per 6 hours for the entire organization. Not ideal.
I would hazard a guess that most deployments of auto-updaters are using these against public repositories, and likely do not authenticate their connection to Docker Hub, further aggravating the issue. That said, even if you are using an auto-updater with Docker Hub authentication you are now using up your pull quota for no purpose other than checking "is the current image i'm using the latest"..
The only solution here is to ensure that all use of Docker Hub (interactive or using auto-update tools) is as an authenticated user, and that your users are members of a Docker Hub organization that is on a paid plan, as these have no pull limits. However, if you don't want to do this (due to cost), you really do need to rethink your use of auto-updaters so as to remove unnecessary pulls from your quota.
Our view, its time for these auto-updating tools to be sunset, and everyone switch to using WebHooks to trigger the update of a container image.
Hope this helps you in some way.
Neil@Portainer
COMMENTS