AngusRepo cache architecture, in practice

A download from an artifact repository looks like a static file fetch. What actually decides latency and correctness is the chain of checks before the file is opened. Before a client receives a JAR, an image layer, or a deb, the repository must know that the target exists, that the repository is still online, that this identity may read it, and that the coordinate has not been quarantined. Only then does it return an index document or the file itself. Querying the database for every one of those checks drags uploads and downloads behind metadata. Holding the answers in memory for too long means a switch to private, or an overwrite, keeps taking effect on later requests. AngusRepo treats the cache as a consistency mechanism with a fixed boundary. The shared layer only speeds up facts that are small, invalidatable, and worth warming after a restart. Correctness comes from invalidation after the transaction commits. Time-to-live only cuts repeated lookups. The sections below start with the conclusion and the vocabulary, then follow placement, the read path, epochs, invalidation, and what each repository type does with files. Guide Problem. A read depends on a chain of checks. A cache that only speeds up files lets permissions, overwrites, and per-process memory drift apart. Approach. The shared cache holds only repository snapshots and group member lists. Authorization results, quarantine decisions, derived documents, and proxy misses stay in the process. Four epochs on the repository row say which kind of fact changed. A single node deletes its process cache after commit. A cluster does the same and also drops one note into every other live instance's mailbox, with an epoch probe as the fallback. The shared layer uses the platform's in-process cache and cache rows in the database. Invalidation uses the durable work queue. Little data needs to be shared, and the message only says "delete the related entries in this process," so Redis is not added. Promise. After commit, the process that handled the write sees the new state on its next request. A rollback leaves both the cache and the epochs where they were. An overwrite shows a reader either the complete old file or the complete new file. Multiple instances bring their memories into line. That does not turn a single node into a highly available deployment. This is written for engineers who design or review artifact repositories, cache consistency, and multi-process deployments. Hosted, proxy, and group repositories show up later. Default numbers are collected under "Defaults." A "public to private" timeline is in the visibility section. Words used below Repositories come in three kinds. The file sections follow that split. Term Meaning here Hosted repository The repository stores the file. That copy is authoritative Proxy repository The repository fetches from upstream on demand and keeps a local copy Group repository Members are tried in order. The group does not copy their files Cache and invalidation use the following words. Term Meaning here Repository snapshot An immutable description used for the online check and the public flag. Upstream credentials are not included Quarantine gate Whether an artifact at a path exists, whether it is quarantined, plus the reason and trust Derived document A protocol response built from repository content, such as Maven metadata, an npm packument, an index page, or a small image manifest Freshness record A proxy repository's description of its local copy: checksum, upstream identifier, next revalidation time, and whether the coordinate is mutable Epoch One of four integers on the repository row that only increase: metadata, authorization, content, and members Work queue A message is taken and acknowledged by one consumer. It is not copied to every process Mailbox In a cluster, the queue that belongs to one process and is consumed only by that process Fallback probe Beside the mailbox, a process periodically reads epochs for repositories it has cached and drops stale memory when it is behind Member-epoch digest A fixed-length digest of member content epochs, placed in a group index key so the key stays short 1. What enters the cache Shared-cache values are strings. A key is at most 256 characters, and the process holds about 10,000 entries by default. JARs, image layers, Helm charts, and packages do not belong in that space. Forcing them in would evict the repository facts that need to stay hot. Files already live in object storage, and hot files can sit in the operating system's page cache. A file in a hosted repository is the artifact. A file in a proxy repository is a local copy of upstream, and the freshness record decides whether it can be returned as-is. The process does not keep a second copy of the bytes. What belongs in the shared cache is small, few in number, and worth warming after a restart: the repository snapshot, and the member list of a group repository. A snapshot carries the identifier, tenant, name, format, type, online status, public flag, upstream address, whether upstream caching is allowed, and the four epochs. Upstream account, password, and token stay out of the snapshot. When a fetch needs credentials, they are read for that repository id and dropped immediately, so the process does not hold a secret and later requests cannot rewrite the snapshot in place. Entries that are numerous, or that depend on the current user, stay in process memory. Their capacities do not compete. Region Why it stays local How correctness is kept Authorization result Expanded by user and permission, far more numerous than repositories Kept for 30 seconds by default. The repository's own public flag and access rules have a separate invalidation path Quarantine gate Expanded by path. Only existence, quarantine, reason, and trust 60 seconds by default. After the content epoch moves, the old key is no longer read Derived document Protocol metadata and indexes 256KB per entry, about 50,000 entries, about 64MB in total. Larger responses are generated for that request Proxy miss Remember a path upstream has confirmed missing, so the proxy does not fetch it again About 100,000 entries. Lifetime follows the repository setting Snapshots go in the shared cache because a restart should be able to warm them from cache rows. Authorization results and index bodies are numerous and short-lived. Putting them in the same limited space only evicts snapshots. The two layers exist because their sharing scope and eviction cost differ. 2. How a read crosses these layers A download or an index read follows a fixed order: snapshot, then whether the repository is online, then public read or access rules, then the quarantine gate, and only then a derived document or the file. A hit on the document or the file happens after those checks. A private repository may keep its files on disk. Permission is computed on each request. The path does not carry authorization, and the system does not issue a long-lived download URL. Figure 2. Checks finish before a document or a file is returned. A public repository allows READ for anonymous and signed-in users alike. That conclusion comes from the snapshot and is not written into the authorization cache. Other requests first separate anonymous from signed-in, then consult the token, administrator, repository creator, deploy token, and access rules. Only an allow or deny produced by access rules enters the local authorization region. Whether the creator, an administrator, or a deploy token is allowed is decided with the identity on this request. It is not frozen into the snapshot, so a change of identity is not stuck behind an old snapshot. Whether a group repository is readable depends on the group's own snapshot. After a member file is chosen, that member artifact still passes the quarantine gate, so a quarantined artifact cannot leave through the group entrance. The member repository's own sign-in rules are not evaluated again for the end user. The user faces one entrance, the group. Re-checking every member layer would reject the same pull more than once, and the group's permission model would become hard to explain. Names of repositories that do not exist are not cached. If a random probe could occupy a "missing" entry, the limited snapshot capacity would fill with meaningless names. A snapshot is kept for 10 minutes by default, only to avoid repeated resolution. Whether going offline, turning private, or an access-rule change is visible immediately depends on invalidation, not on that lifetime. 3. Four epochs describe the scope of invalidation The repository row keeps four integers that only increase, and they advance in the same transaction as the business update. A change advances only the numbers it actually affects. The read path uses them to know which cache entries are still usable. Epoch When it advances Metadata Rename, status, upstream address, format or type, and other settings that are not about permission Authorization Public flag, access-rule changes, repository deletion Content Publish, overwrite, delete, quarantine, or trust on this repository; a proxy fetch that writes or refreshes Members Adding, removing, or reordering members of a group repository When the public flag changes, the metadata epoch and the authorization epoch advance together: the flag inside the snapshot changed, so conclusions computed from the old rules must expire. Publishing an artifact advances only the content epoch. An access-rule change advances only the authorization epoch. During a steady stream of uploads, the database gains one integer update, and the repository settings do not have to be reloaded every time. The split separates "settings changed" from "content changed," so one publish does not throw away the whole snapshot. Each process remembers the four numbers currently in effect for a repository. Keys for the gate, derived documents, and the member list are built from those numbers. The rule is a single comparison: take the larger of the number stored in the snapshot and the number written by the latest invalidation. A content epoch shows how that rule works. Repository R has content epoch 12 in the database, the process view is also 12, and the gate key carries 12. After a publish commits, the database value becomes 13 and the writing process sets its view to 13. Later reads use the key that carries 13. A conclusion still sitting in memory under 12 is never read again. If another process is still on 12, the mailbox or the fallback probe lifts its view to 13 when it observes 13. Even if a snapshot that has not been refreshed still says 12, the key is built with 13. A publish only raises the content epoch. It does not delete and reread the whole snapshot for that upload. Figure 3. The content epoch used to build keys, before and after one publish. The shared cache cannot delete by prefix, so the design changes the key instead of scanning old entries by path. The member list puts the member epoch in the key, and the old key remains until its own lifetime ends. Snapshot keys do not contain these epochs, so a metadata or authorization change deletes the snapshot explicitly. Gates, derived documents, and member lists expire by changing keys. Snapshots expire by explicit deletion. Together the two cover every region. Index responses follow the repository content epoch. Any successful publish in that repository makes the old Maven metadata, or a similar index, miss. If rebuilding later proves too expensive, the key can be narrowed to a coordinate. That is an optimization on the same epoch model. The invalidation method stays the same. A group repository's merged index also has to reflect its members. The key contains the group's member epoch and the member-epoch digest. A publish on any member, or a change in member order, changes the digest, and the old merged result is no longer read. The group merges an index view. Member files stay in their own object storage. 4. A single node deletes directly. A cluster propagates by mailbox Single node The deployment mode is explicit, and the default is a single node. The cache then exists only in that one process. The write transaction updates business data and advances epochs. After the transaction commits, it deletes the affected cache in this process. If the commit fails, the epochs stay put and memory stays as it was, so the next request still sees the pre-commit state. A single node does not read the epochs again just to confirm invalidation, and it does not start queue consumption. One deletion has to cover every region the change affects. Change What is removed after commit Rename Snapshots under both the old name and the new name Public flag or access rules The snapshot, and authorization results for that repository Publish, overwrite, delete, or quarantine Raise the local content epoch so the gate for that path and the repository's derived documents expire Group membership The member list and the group snapshot A proxy path is stored again, or upstream confirms deletion The miss record for that path. Whether a mutable path can still be returned directly still follows the freshness record Deleting a snapshot in the shared cache removes both the memory entry and the cache row. Clearing only memory would reload the old snapshot from the cache row on restart, and a switch to private would reappear. If the local deletion fails, that repository's snapshot is voided immediately. The next read goes to the database and records the error. Correctness is not left to the 10-minute lifetime. Mailbox Each process has its own memory, so invalidation has to reach every copy. The platform queue is a work queue: after one consumer takes a message and acknowledges it, other processes never see it. Partitioning by a key only makes one consumer handle that key in order. It does not copy the message to every process. A shared topic for all instances, or pinning a repository id to one machine, therefore cannot reach the cache on the other processes. A cluster gives each live instance its own mailbox, a topic consumed only by that instance. The instance creates the mailbox from its own id at startup. The id is unique in the cluster. In containers the Pod name is used, because a node hostname can repeat. Figure 4. How one commit reaches memory on the other processes. The online set is maintained by heartbeats, refreshed every 10 seconds by default. An instance with no heartbeat for 30 seconds is treated as offline. A send addresses only instances that have a recent heartbeat and are not the sender. The message and the business data commit in the same transaction, so a rollback drops the message as well. This process still deletes its own memory after the commit succeeds, so a concurrent read before commit cannot put the old data back into a cache that was just cleared. Within one transaction, each target instance receives one message per repository. The scope of the change and the related paths are merged into it. A batch publish does not fill the message table with one row per file per instance. The body only describes what to delete: tenant, repository, name change, which epoch classes are affected, and optional paths. It carries neither file bytes nor upstream credentials. The receiving instance performs the same local deletion as a single node and writes the epochs from the message into its local view. Other instances normally observe the change after about one poll plus handling time. The poll interval is 200 milliseconds. A new process starts with empty memory. It loads current facts from the database and from shared cache rows. It does not replay old mail. An instance whose heartbeat has been absent for more than 30 seconds receives no new messages. Unconsumed rows in its mailbox are deleted by a repository cleanup task. The queue's own cleanup removes completed messages and does not remove rows that nobody consumes, so that sweep belongs to the repository. A failed consume is rejected and retried briefly, then dead-lettered. A dead letter is not replayed and does not repair the cache. The fallback probe pulls state back within its interval. Running the same message more than once is safe, because deletion and raising an epoch can be repeated. Heartbeat and probe While the queue is healthy, invalidation is not discovered by polling the database. There is a second layer: for repositories this process has actually cached, the four epochs are read every 5 seconds by default. If the database value is larger, the process performs the same local deletion it would perform on a message. That read covers a few integers on hot repositories. It does not parse settings and it does not query access rules. When a message is lost, the instance was missing from the heartbeat set, or consumption stalls, stale memory stays no longer than that probe interval. A single node does not do this read. The four epochs are still maintained, so switching to a cluster later does not require backfilling history. Whether the heartbeat succeeds decides if this process follows the mailbox or only the probe. State How this process learns about other writes Heartbeat healthy Its own mailbox is primary. The 5-second probe is only a fallback Heartbeat cannot be written Stop claiming to be online, and read epochs on the probe interval No heartbeat for 30 seconds Other instances stop sending it new messages. Unconsumed rows are deleted by the repository cleanup task Message is dead-lettered The cache is not repaired by replay. The probe pulls it back within 5 seconds Process just started Memory is empty. Current facts are loaded. Old mail is not replayed Cluster mode without a unique id Refuse to start, so two processes cannot share one mailbox User groups and departments belong to the identity system. They are not part of this repository transaction. Both deployments cap an allow or deny computed from access rules at 30 seconds by default. The repository's own public flag and access rules follow deletion after commit, or the mailbox. They do not depend on those 30 seconds. If group membership later has to take effect immediately, the identity system can tell the repository to raise the authorization epoch, and the same invalidation path applies. 5. How the three repository types treat files Files of all three types stay out of the cache. They differ in where the copy lives, when it is rechecked, and which epoch the change advances. Repository Where the file lives When it is rechecked Which epoch expires it Hosted The authoritative copy in object storage After the atomic replace, a newly opened request reads the new file Content Proxy A local copy of upstream An immutable coordinate is returned when a local file exists. A mutable coordinate is conditionally revalidated after 60 seconds by default Content Group No extra copy. Members are chosen in order Every request follows the current order and does not remember which member owned the path Members. Merged indexes use the member-epoch digest Hosted repository An overwrite writes the new content to a temporary file in the same directory, checks that it is complete, and then renames it atomically. The temporary file must be on the same file system as the target. Otherwise the rename becomes a copy, and a concurrent reader can still see a half-written file. A download that already opened the old file may finish the old content. A request opened after the rename reads the new file. At any moment a reader sees either the complete old file or the complete new file. After a successful commit the content epoch advances. The writing process raises its local content epoch and drops the gate for that path and the repository's derived documents. A cluster also places this content change in the other instances' mailboxes. If the transaction fails, the temporary file is deleted, the epoch is unchanged, no message is sent, and memory is unchanged. Proxy repository The freshness record describes the local copy: checksum, size, time stored, upstream validator and modification time, the next time to revalidate, and whether the coordinate is immutable or mutable. A release coordinate that is immutable is returned directly when a local file exists. Snapshot versions, Maven metadata, latest, image tags, and indexes are conditionally revalidated against upstream after 60 seconds by default. An upstream "not modified" only pushes the next check later. New upstream content replaces the local file with the same atomic replace used by a hosted repository, and advances that proxy repository's content epoch. An upstream confirmation that the path is gone deletes the local copy and writes a bounded miss record. When the repository disables caching, nothing is stored. If upstream returns a server error or times out, and a local copy still exists, that copy is returned. The response and the log say the local copy is being reused, and the next check is not pushed later. The next request after upstream recovers still revalidates, instead of turning one failure into a new period that skips checks. Group repository A group repository does not keep a second copy of member files. Each request walks the current member order and chooses the first member that can return the path. It does not remember "this path belongs to that member" for long. If a higher-priority member publishes the artifact later, and the group still returns the lower-priority member chosen earlier, the client keeps receiving the stale copy. The member list expires through the member epoch. The merged index expires through the member-epoch digest. 6. When permissions and content become visible When the next request sees a change depends on which epoch moved, and on whether the request lands on the writing process or on another process. Change Process that handled the write Other processes (cluster) Public flag or access rules The next request after commit About one poll interval. If the message is lost, no longer than the probe interval Offline or maintenance Rejected before the file is opened About one poll interval. If the message is lost, no longer than the probe interval Overwrite or release from quarantine The new content epoch is used in this process's keys immediately The content epoch is raised the same way once the mailbox arrives Group membership in the identity system No longer than the 30-second authorization cache No longer than the 30-second authorization cache Turning a repository from public to private walks that table as a timeline. Let the commit be T0, and let the authorization epoch move from 7 to 8. Moment What happens T0 The same transaction updates the public flag, advances the metadata and authorization epochs together, and writes one mailbox message for each other live instance The writing process's next read after T0 This process has deleted the snapshot and the repository's authorization results. Anonymous requests are no longer allowed. A signed-in user is evaluated against the access rules again About T0 + 200 ms Other instances take their mailbox message and perform the same deletion The mailbox does not arrive Other instances notice the larger authorization epoch on a probe within about 5 seconds, and delete the old snapshot and the old authorization results Only a user-group change at the same time That change is outside this transaction. If the repository rules did not change, a previous allow lasts at most another 30 seconds Turning private into public, taking the repository offline, or entering maintenance follows the same path as T0, and the rejection happens before the file is opened. Revoking a user's read also advances the authorization epoch. The writing process drops the snapshot and authorization results after commit. Other processes catch up by mailbox or by probe. 7. Promises that still hold when something fails Visibility is bounded by a committed transaction. Before that, the cache and the epochs stay at their pre-commit values. After that, the writing process's deletion and the other processes' mailboxes take effect. Situation Result The business transaction rolls back Epochs did not increase, cluster messages were not committed, and memory is unchanged Commit succeeded, but the process exits before deleting its local cache After a single-node restart, memory is empty and the next load comes from the database. Cluster messages were committed with the business data, so other instances still delete their own memory Shared cache rows cannot be written for a while Snapshots fall back to this process's memory only. Local deletion can still correct them. Warming after restart is weaker A derived document exceeds the per-entry size limit It is not cached. It is generated for that request and returned Clearing every cache row The operation stays closed. That table may hold other users' data. Invalidation stays scoped to a repository and a region Observations are recorded per region: hits, fetches, mailbox delay, how often the fallback probe corrected state, and how often a proxy revalidation was not modified, updated, or served from the existing copy. The shared cache's own aggregate statistics mix every key together. They cannot by themselves say whether the repository cache is correct. 8. Defaults These numbers are consistency bounds, not a tuning starting point. Shortening a lifetime only increases reads. Lengthening it does not make a public-flag change or an overwrite more correct. Those two follow deletion after commit, and the epochs. Setting Default What it bounds Deployment mode Single node After commit, only this process's cache is deleted. The queue is not started Snapshot lifetime 10 minutes Cuts repeated resolution. It does not carry correctness for permissions or going offline Authorization result 30 seconds, about 100,000 entries Allow or deny computed from access rules Quarantine gate 60 seconds, about 100,000 entries After the content epoch changes, the old key is not read Derived document 256KB each, about 50,000 entries, about 64MB total Anything larger is generated for the request and not cached Proxy miss About 100,000 entries, lifetime from the repository setting Remember a path upstream confirmed missing Mutable coordinate revalidation 60 seconds After it is due, revalidate upstream with a conditional request Shared cache capacity Keys at most 256 characters, about 10,000 entries in memory Snapshots and member lists only Mailbox poll 200 milliseconds Cluster only. Normal visibility delay for other processes Heartbeat 10 seconds Refresh the online set Treated as offline 30 seconds without a heartbeat No new messages are sent to that instance Fallback probe 5 seconds Cluster only, and only for repositories this process has cached 9. What a configuration drift looks like The timeline above holds when the deployment mode matches the real process count, cluster ids are unique, heartbeats can be written, and a snapshot deletion removes both memory and the cache row. When those conditions drift, the system still runs as designed. What an operator expects to see, and what is actually visible, come apart. Drift What you see Several processes still configured as a single node Other processes never receive this deletion. They wait out their own lifetimes. That is a configuration error, not an error in the invalidation algorithm Cluster mode without a unique instance id The process refuses to start. If two processes shared one mailbox, only one of them would take each message Heartbeat writes fail, but the instance is still treated as online Other instances may stop sending it messages, and its old permissions remain. The design then stops claiming to be online and reads epochs every 5 seconds A snapshot deletion clears memory only After restart the old snapshot loads from the cache row, and a public-flag change reappears. The deletion has to remove the cache row as well A mutable proxy path is not revalidated from its freshness record The local copy stays stale. Once due, it should be conditionally revalidated. During an upstream failure the copy may be reused, but the next check must not be pushed later 10. Boundaries The shared layer stops at the platform's in-process cache, cache rows in the database, and the durable queue. What a cluster has to solve is that every machine deletes its own memory. A mailbox per instance, plus the epoch probe, closes that. The queue keeps the take-and-acknowledge semantics of a work queue. File bytes, authorization results, miss records, and large protocol responses stay in the process or in object storage. They are not written into the shared cache or into messages. A single node does not read epochs on every request. Names of missing repositories do not occupy snapshot capacity. User-group changes in the identity system keep a 30-second cap. This design does not add a separate synchronization path for them. Content deduplication answers how many times the same bytes occupy disk. An overwrite answers which complete content a path points at now. The two stay separate: deduplicated storage stays as it is, and the path is replaced by an atomic rename. For someone running the repository, the default single node means the next request after a successful commit sees the new repository state, the new public flag, and the newly uploaded bytes. A rollback leaves both unchanged. When several processes must share one set of data, turn on cluster mode and give each process a unique id. Other processes catch up within about one poll. If consumption stops, the epoch probe corrects them within a few seconds. The file layer shows a reader a complete old package or a complete new package. The proxy layer rechecks mutable coordinates with upstream, and keeps serving immutable coordinates from the local copy. Originally published at anguskit.com.