Pull-Through Caching
Angos can act as a caching proxy for upstream container registries, reducing bandwidth, improving performance, and protecting against rate limits.
How It Works
Cache Behavior
Cache Miss
When content isn't cached:
- Registry checks first upstream
- If unavailable, tries next upstream (fallback)
- For manifests: fetches, stores, then returns
- For blobs: streams to client while caching in background
- Subsequent requests serve from cache
Cache Hit
When content is cached:
Immutable tags (e.g., nginx:1.29.0):
- Served directly from cache
- No upstream check
- Maximum performance
Mutable tags (e.g., nginx:latest):
- Registry checks upstream for updates
- If same digest, serves from cache
- If different, refreshes cache
- If no upstream can answer, serves from cache
Upstream Unavailable
An upstream is unavailable when it cannot be reached, answers a server error (5xx) or
429 Too Many Requests (the registry or its token service), or sends a response Angos cannot
read. With several upstreams, a request is unavailable unless one of them serves it or every one
of them answers.
- Cached content keeps being served. A mutable tag no upstream can check is served as
cached. A tag the upstream reports as moved but then fails to serve, such as a throttled
GETbehind an unthrottledHEAD, is served from the copy it would have replaced. Digests, immutable tags and blobs are served from the cache without asking the upstream, so an outage does not reach them. - Uncached content answers
503 UPSTREAM_UNAVAILABLE, not404, so clients retry instead of reporting a missing image. - An upstream that answers is believed. A
404,401or403is an answer: a mutable tag the upstream reports as gone, or no longer lets Angos read, is not served from the cache. Docker Hub answers401for a repository that does not exist.
Each pull served this way logs a warning and counts as outcome="stale" in
angos_pull_through_total.
Configuration
Basic Setup
[[repository."library".upstream]]
url = "https://registry-1.docker.io"
username = "dockerhub-user"
password = "dockerhub-token"
Multiple Upstreams (Fallback)
[[repository."library".upstream]]
url = "https://registry-1.docker.io"
username = "user"
password = "pass"
[[repository."library".upstream]]
url = "https://mirror.example.com"
# Fallback if primary is unavailable
Immutable Tag Optimization
[repository."library"]
immutable_tags = true
immutable_tags_exclusions = ["^latest$", "^nightly.*$"]
[[repository."library".upstream]]
url = "https://registry-1.docker.io"
Request Flow
Blob Request
Note: On a blob cache miss, Angos fetches the blob twice from the upstream registry: once to stream directly to the client, and once to store in the cache. This doubles upstream bandwidth usage for uncached blobs. For rate-limited upstreams (e.g., Docker Hub), consider pre-warming your cache during off-peak hours.
Streaming Architecture
For blob cache misses, Angos streams data to the client while caching in the background:
This approach:
- Minimizes client latency (immediate streaming)
- Caches content asynchronously
- Handles large blobs efficiently
Rate Limit Protection
Problem
Upstream registries impose rate limits. Docker Hub limits pulls per account or IP over a time window, with higher limits for authenticated and paid accounts; see Docker Hub usage limits for current figures. Other registries vary.
Solution
Pull-through cache reduces upstream requests:
- First pull: Fetches from upstream (counts against limit)
- Subsequent pulls: Served from cache (no limit impact)
- Immutable tags: Never re-check upstream
- Mutable tags: Only lightweight HEAD requests
- Throttled upstream: Cached content keeps being served (see Upstream Unavailable)
Optimization Tips
- Use authenticated upstream access (higher limits)
- Enable immutable tags for versioned content
- Pre-warm cache for frequently used images
Authentication Methods
Anonymous
[[repository."public".upstream]]
url = "https://public.registry.io"
Basic Auth
[[repository."private".upstream]]
url = "https://private.registry.io"
username = "user"
password = "pass"
mTLS
[[repository."secure".upstream]]
url = "https://secure.registry.io"
client_certificate = "/certs/client.crt"
client_private_key = "/certs/client.key"
server_ca_bundle = "/certs/ca.crt"
Write Behavior
When pull-through cache is enabled:
- Push operations are disabled
- Clients receive
401 Unauthorized - The namespace is read-only
This prevents confusion between cached and local content.
Multi-Registry Setup
Mirror multiple registries:
# Docker Hub official images
[repository."library"]
immutable_tags = true
immutable_tags_exclusions = ["^latest$"]
[[repository."library".upstream]]
url = "https://registry-1.docker.io"
username = "dockerhub-user"
password = "dockerhub-token"
# GitHub Container Registry
[repository."ghcr.io"]
immutable_tags = true
[[repository."ghcr.io".upstream]]
url = "https://ghcr.io"
# Quay.io
[repository."quay.io"]
immutable_tags = true
[[repository."quay.io".upstream]]
url = "https://quay.io"
# Private registry
[repository."internal"]
[[repository."internal".upstream]]
url = "https://registry.internal.example.com"
client_certificate = "/certs/client.crt"
client_private_key = "/certs/client.key"
Upstream Selection and the ns Parameter
Which upstream serves a request is decided by configuration. The leading segments of a namespace name
the [repository] entry that owns it: docker-hub/library/nginx is served by
[repository."docker-hub"], that entry's upstream list is what gets consulted, and the prefix is
stripped before the request is forwarded (library/nginx upstream). A namespace no entry matches has
no upstream and is served from local content alone.
A mirroring client such as containerd does not prefix its paths. It requests the upstream's own path
and names the registry it believes it is addressing in the ns query parameter, which the
distribution spec defines for exactly this in Registry
Proxying:
optional on pull operations, naming the host component of the repository name the client used.
Declare that namespace on the repository mirroring it:
[repository."docker-hub"]
namespace = "docker.io"
[[repository."docker-hub".upstream]]
url = "https://registry-1.docker.io"
GET /v2/library/nginx/manifests/latest?ns=docker.io is then served from docker-hub, exactly as
GET /v2/docker-hub/library/nginx/manifests/latest is, and the response carries
OCI-Namespace: docker.io. Content cached either way lands under docker-hub/library/nginx, so the
two spellings share one cache, and an access policy sees that namespace whichever the client used.
The parameter scopes pulls, which is what the spec defines it for: a manifest or blob GET/HEAD, a
tag listing, a referrers listing. A write naming it is left addressing the namespace it spelled out,
so ns cannot put client-pushed content into a mirror's cache.
The parameter selects among configured repositories and nothing else. An ns no repository declares
as its namespace is ignored: the request is served as it arrived and no OCI-Namespace is echoed. The spec permits
that ("a registry MAY choose to ignore the ns query parameter") and pairs the header with use, so
its absence is what tells the client the namespace it named had no effect. A client cannot make Angos fetch from a
registry the configuration does not already name, and each upstream's content stays under its own
prefix rather than colliding with another's.
Two repositories declaring the same namespace is refused at startup: the parameter must resolve to one.
Token Caching
Upstream authentication tokens are cached:
[cache.redis]
url = "redis://redis:6379"
key_prefix = "token-cache"
Without Redis, tokens are cached in-memory per-instance.
Performance Tuning
Concurrency
[global]
max_concurrent_cache_jobs = 8 # Parallel upstream fetches
For multi-replica deployments, see Enable Durable Cache Jobs
to route cache-fill work through a shared queue drained by angos worker.
Timeouts
[[repository."library".upstream]]
url = "https://registry-1.docker.io"
connect_timeout_secs = 30 # TCP + TLS handshake (default: 30)
read_timeout_secs = 300 # Inactivity between reads (default: 300)
read_timeout_secs bounds the stall between two reads, not the whole transfer: a slow but progressing blob download is never cut off by a total deadline.
Redirects
max_redirect (default 5) caps how many upstream redirects a request follows:
[[repository."library".upstream]]
url = "https://registry-1.docker.io"
max_redirect = 5
Troubleshooting
Slow First Pull
- Check upstream connectivity
- Verify credentials are valid
- Consider pre-warming cache
Content Not Updating
- Check if tag is marked immutable
- Verify upstream connectivity: a tag no upstream can check is served as cached, counted as
angos_pull_through_total{outcome="stale"} - Check exclusion patterns
503 UPSTREAM_UNAVAILABLE
No upstream answered for content Angos has not cached. The warning log names each upstream's failure.
- Verify upstream connectivity and the token service it redirects to
- Check for upstream rate limiting (
429) - Add fallback upstreams
Cached Tag Answers 404
The upstream answered that the tag is gone, or refused Angos access (401/403). Verify the
upstream credentials have not expired, or mark the tag immutable to serve it without asking.
Rate Limit Errors
- Add authentication
- Enable immutable tags
- Add fallback upstreams