Project Logs

Collapse

• Brother mirrors

  SUNsung • 07/16/2026 at 18:20 • 0 comments

  This log is about brother mirrors in yggvault: one node can read another yggvault as an upstream source. This mode is for small independent mirrors, where several operators want to keep the same releases without shared storage, a quorum, or a consensus layer.

  A brother mirror solves a practical availability problem. If the original forge is temporarily down, blocked, has deleted a release, or is poorly reachable from a specific region, the follower node can update from a pre-selected brother. After publication, the version becomes a local part of the follower's storage and is served like an ordinary imported release.

  For this log, the left-hand part of the diagram matters: yggvault first works with the original upstream, then uses a trusted brother as a fallback source. The right-hand part with web/Yggdrasil access was already covered in Yggdrasil mode.

  What counts as a brother mirror

  A brother is an ordinary yggvault node whose package URL is listed in another node's release_mirrors:

  release_mirrors:
    errors: "https://seed.example.org/errors"

  or via a Yggdrasil address:

  release_mirrors:
    errors: "http://0123456789abcdef.pk.ygg/errors"

  The follower keeps its own local key errors and derives the remote key from the path of the remote URL. This works in nested mode too: if the seed lives under /pkg, a URL like https://seed.example.org/pkg/errors is parsed with the route_prefix that the seed advertises through /info.

  The model is pull-based: the follower runs the rescan itself, chooses the transport itself, downloads the index/tree/blobs or public archives itself, and goes through the publication pipeline itself.

  Seed and follower

  A seed usually points at a real upstream forge:

  release_mirrors:
    errors: "https://github.com/go-faster/errors"
  
  brother:
    rpc:
      web_enabled: true
      ygg_enabled: true
      max_parallel: 4
      max_parallel_per_peer: 2

  A follower points at the seed:

  release_mirrors:
    errors: "http://0123456789abcdef.pk.ygg/errors"
  
  brother:
    prefer: "first_source"
    transport_order: "web_ygg"
    first_source_timeout: "5s"
    rpc:
      web_enabled: false
      ygg_enabled: true

  prefer: "first_source" is the default mode. In this mode, the brother helps find and store the origin, but the original source remains the preferred path for updates. prefer: "brother" enables the opposite strategy: read the brother first, then use the origin as a fallback.

  Discovery

  The source layer identifies a brother through several small probes:

  1. GET /health must return the service yggvault.
  2. GET /info returns the public node card: web domain, ygg_host, route prefix.
  3. CONNECT /rpc opens a net/rpc session.
  4. Brother.Hello returns the protocol version and fetch limits.

  The current wire protocol is called mesh1. A protocol mismatch turns the source unavailable for the current cycle.

  /info is used as a guard against address drift. If the operator configured a Yggdrasil host, the remote /info must advertise the same ygg_host. If a web host is configured, the remote /info must advertise the same domain. This catches the situation where a URL leads to a similar service or to a node with a different public identity.

  If /health confirms yggvault but RPC is closed or unavailable, the source is still classified as a brother. In that case, the rescan uses the public fallback path.

  Why the original source stays the priority

  The brother index may advertise a source URL for a key. On the seed, this value comes from its release_mirrors[key]. The follower stores it as OriginURL and uses it as the first source on subsequent cycles.

  In the default first_source mode, the cycle looks like this:

  1. the follower opens the brother and reads the index;
  2. gets the advertised origin from the index;
  3. checks the origin's reachability within brother.first_source_timeout;
  4. if the origin is reachable, runs the ordinary git ingest path;
  5. if the origin is unreachable, reads versions from the brother.

  This order is for provenance. The brother improves availability, but...

  Read more »

• Yggdrasil mode

  SUNsung • 07/16/2026 at 18:16 • 0 comments

  This log is about Yggdrasil mode in yggvault: an additional mesh ingress for the same release mirror. The same node can serve already-imported releases through the ordinary web listener, through a Yggdrasil address, or through both access paths at once.

  The idea is practical: the mirror should stay useful where the public domain, DNS, reverse proxy, or regional access to the ordinary internet behave unreliably. The Yggdrasil layer adds an address bound to the node's key and lets mesh participants read the same package routes without public IPv4/IPv6 ingress.

  In this diagram, the right-hand part matters for this log: clients read the same vault through an HTTPS domain or through a Yggdrasil address. The brother fallback in the diagram will be needed in the next log; here it serves as an example that the mesh transport is used both for serving and for sync between trusted mirrors.

  Where this is useful

  yggvault can be run in different modes:

  • a web-only mirror on a VPS, home server, or local machine;
  • a Yggdrasil-only mirror with no public ingress;
  • an ordinary local/web mirror with temporarily enabled Yggdrasil access, when you need to share a release inside the mesh;
  • a public node with an HTTPS domain and a permanent Yggdrasil address;
  • a follower node that pulls releases from a brother mirror over web or Yggdrasil transport.

  In all variants, the storage model stays the same. The node first imports the release archive, validates it, builds artifacts, and commits the version to local storage. Yggdrasil changes the way published data is accessed, without creating a separate copy of the package registry.

  A simple scenario for the operator: the mirror lives locally on a PC, periodically updates selected projects, and, when needed, opens mesh access for another machine or a small team. Already-imported versions keep being served from local storage even when there are problems with the upstream.

  How it's enabled

  Yggdrasil mode is controlled by a single key config field:

  ygg:
    pem_key: "./yggvault.pem"
    peers:
      initial:
        - "tls://example-peer-1:443"
        - "tcp://example-peer-2:62486"
      max_per_proto: 1
      probe_timeout: "10s"
      refresh_interval: "1h"
      batch_size: 8

  An empty ygg.pem_key fully disables mesh ingress. If a key path is set, yggvault brings up an embedded Yggdrasil node via Ratatoskr and opens an HTTP service on the fixed mesh port 80.

  The key is created with the command:

  ./yggvault --make-ygg-key

  The command creates a PEM key and prints a host of the form <PublicKey>.pk.ygg. This host is derived from the public key. If the same key is kept, the node keeps the same Yggdrasil identity. If the key is replaced, the address changes.

  For a first run, 2–3 nearby public peers are usually enough. It's better to pick the nearest stable routes with reasonable latency. A mesh link should be treated as a constrained link: requests should have deadlines, retries should have limits, and large responses should have size caps.

  What actually appears at runtime

  With ygg.pem_key enabled, a Ratatoskr-backed userspace network stack appears inside the process. This mode works without a system Yggdrasil daemon, a TUN device, or root privileges.

  The mesh object gives the runtime three things:

  • Host() — a Yggdrasil host of the form <hex>.pk.ygg;
  • Address() — an ordinary Yggdrasil IPv6 address from the 200::/7 range;
  • DialContext() and a listener for HTTP traffic inside the mesh.

  The server layer adds a separate listener context with listenerID = ygg, scheme = http, and entryHost = <hex>.pk.ygg. The web listener gets its own context with a domain and scheme. From there, both listeners go into one router and serve the same routes:

  /health
  /info
  /catalog.json
  /{key}
  /{key}/latest
  /{key}/{version}.zip
  /{key}/{version}.tar.gz
  /{key}/@v/list
  /{key}/@v/{version}.info
  /{key}/@v/{version}.mod
  /{key}/@v/{version}.zip
  /feed.xml

  /info returns the node's public card: name, version, web domain, ygg_host, and route prefix. Brother discovery...

  Read more »

• Web/API routes and nested mode

  SUNsung • 07/16/2026 at 18:09 • 0 comments

  This log is about the public surface of yggvault: which URLs the browser, go, Composer, curl, another yggvault node, monitoring, and a static frontend receive.

  The main idea is simple: a single binary can be a release mirror, a package endpoint, a JSON API, a Web UI, an Atom feed provider, a Yggdrasil HTTP node, and a small static web server. All these roles live on one route map. Nested mode is needed when the operator wants to keep / for their own site and gather the mirror routes under a single prefix.

  One public surface

  After publish, a version appears in several interfaces at once. Handlers read a single storage/state layer and build different representations of the same version: HTML page, JSON, package-manager metadata, feeds, and archives.

  Root-mode routes look like this:

  Group	Routes
  Service	/health, /info, /openapi.json
  Metrics	/metrics, /metrics/core, /metrics/cache, /metrics/errors, /metrics/rescan, /metrics/ygg, /metrics/internal
  Browser/media	/, /{key}, /{key}/{version}, /favicon.ico, /logo/{size}, /og.png,/og/{key}, /og/{key}/{version}, /sitemap.xml
  Release API	/catalog.json, /{key}/releases.json, /{key}/{version}.json, /{key}/latest, /{key}/list, /{key}/list/full
  Archives	/{key}/{version}.zip, /{key}/{version}.tar.gz
  Go proxy	/{key}/@latest, /{key}/@v/list, /{key}/@v/{version}.info, /{key}/@v/{version}.mod, /{key}/@v/{version}.zip
  Composer	/packages.json, /packages/list.json, /p2/{vendor}/{package}.json, /p2/{vendor}/{package}~dev.json
  Atom	/feed.xml, /{key}/releases.xml
  Brother	CONNECT /rpc

  Go routes support major module paths:

  /{key}/vN/@v/list
  /{key}/vN/@latest
  /{key}/vN/@v/{version}.info
  /{key}/vN/@v/{version}.mod
  /{key}/vN/@v/{version}.zip

  The same route map works on the ordinary web listener and on the Yggdrasil listener. The entry host changes: a normal domain or a Yggdrasil address. The storage content and route semantics stay the same.

  Root mode

  Root mode is used when web.static.dir is empty. The built-in catalog, Web UI, JSON APIs, feeds, Composer repository, and Go proxy routes live from the host root.

  Examples:

  https://modules.example.org/
  https://modules.example.org/catalog.json
  https://modules.example.org/errors
  https://modules.example.org/errors/latest
  https://modules.example.org/errors/v0.7.1.zip
  https://modules.example.org/errors/@v/list
  https://modules.example.org/packages.json
  https://modules.example.org/feed.xml

  The Go module path matches the public route:

  GOPROXY=https://modules.example.org GOSUMDB=off \
    go get modules.example.org/errors@v0.7.1

  Composer gets the repository root at the host:

  {
    "repositories": [
      {
        "type": "composer",
        "url": "https://modules.example.org"
      }
    ]
  }

  Nested mode

  Nested mode is enabled through web.static.dir. In this mode, / belongs to the user's static frontend, and the mirror routes move under web.routing.prefix.

  A minimal config fragment:

  web:
    static:
      dir: "./site"
      index_file: "index.html"
      max_size: "256mb"
      cache_max_age: "24h"
      deny:
        - ".env"
        - ".key"
        - ".pem"
        - "/.git/"
        - "/config.yml"
    routing:
      prefix: "pkg"

  An example of ownership after that:

  URL	Owner	Meaning
  /	static frontend	web.static.index_file, usually index.html
  /assets/app.css	static frontend	an ordinary file from web.static.dir
  /pkg/	yggvault	built-in mirror catalog
  /pkg/catalog.json	yggvault	JSON catalog
  /pkg/{key}	yggvault	key page
  /pkg/{key}/latest	yggvault	latest version endpoint
  /pkg/{key}/{version}.zip	yggvault	universal archive
  /pkg/{key}/@v/list	yggvault	Go proxy list
  /health, /info, /metrics, /openapi.json	yggvault	stable service routes at root
  CONNECT /rpc	yggvault	brother RPC gate

  The prefix is set in the config. pkg is a common example, not a hard-coded path. The route model is intentionally flat: the API base matches the mirror base; /api/ is not a mandatory layer. The prefix is one segment matching ^[a-z0-9][a-z0-9._-]{0,31}$; reserved names like health, info, metrics, openapi.json...

  Read more »

• Storage internals

  SUNsung • 07/16/2026 at 18:06 • 0 comments

  This log is about the storage layer inside yggvault: what stays the durable source of truth, what is a regenerable cache, where the version-visibility boundary runs, and why a mirror can serve releases for a long time without an external database or object storage.

  The import/refresh pipeline is already responsible for source discovery, archive validation, and ecosystem overlays. Storage begins after that: the node has a canonical tree, a set of blobs, metadata for routes and artifacts, and a decision to publish or reject.

  Why this is a separate part of the project

  A temporary solution for a few Go modules could live as a set of files on disk. A full release mirror quickly runs into other problems:

  • different versions often contain identical files;
  • one version must be served as a universal archive, a Go module zip, Composer metadata, a JSON page, a feed entry, and a web page;
  • the upstream may delete a version or replace a tag with a different tree;
  • public serving requires ready files, range-friendly responses, stable ETags, and checksums;
  • the operator must be able to see how much the durable state takes and how much the serving cache takes.

  That's why yggvault separates durable content, durable metadata, and hot serving artifacts.

  Three layers under storage.dir

  storage.dir contains several layers that differ in meaning.

  Layer	What it stores	Why it matters
  SQLite index	versions, lifecycle flags, source hashes, tree hashes, detection results, artifact metadata, blob_refs, history events, key/source bindings	An SQLite transaction makes a version visible. This is where the relations and history live.
  Pebble object store	blob objects and tree objects by BLAKE3-24 hash	Identical bytes get the same object key and are not written again.
  Hot cache	ready .zip, .tar.gz, Go zip, and other generated artifacts	The files can be deleted and rebuilt from durable storage.
  RAM metadata cache	small JSON/metadata responses	Bounded by cache.metadata_max_size; this is a performance layer.

  Practical result: the durable state is responsible for truth, the hot cache is responsible for serving speed.

  Blob object

  A blob object is the bytes of a single file from the release tree. Its identity is a BLAKE3 digest truncated to 24 bytes / 192 bits. In the code this is core.HashObj: a fixed-size array, comparable, convenient as a map key and as part of a Pebble key. The 24-byte size is a deliberate step to optimize index handling while keeping sufficient collision resistance.

  Before a write, storage verifies that the bytes really produce the expected hash. Before a verified read, storage can recompute the hash again, depending on storage.pebble.verify_on_read.

  An identical file from two versions gets the same blob hash. If a README.md, license file, or generated sources repeat across several releases, Pebble stores the body once, while SQLite counts references through blob_refs.

  Tree object

  Each version in SQLite has a tree_hash. It's what links the version metadata to the canonical file tree.

  A tree object contains a canonical list of entries:

  • path;
  • mode (file or symlink);
  • size;
  • blob hash.

  The tree encoding is intentionally strict. The code sorts paths, rejects duplicates, rejects file/child conflicts, checks the canonical path form, checks supported modes, and encodes the object into a binary format with the magic YRV_TREE\x00. The tree object hash is computed from these canonical bytes.

  This gives a simple model:

  version -> tree_hash -> tree entries -> blob hashes -> blob bytes

  If the upstream publishes the same release content again, the tree hash matches. If the tag name is the same but the tree is different, storage sees the mutation as a change in tree_hash.

  Publish boundary

  PublishStaged is the boundary where staged data becomes a published version.

  Storage runs several checks before visibility:

  1. a staged blob must lie inside the temporary spool;
  2. a staged blob must be a direct child of the spool...

  Read more »

• Import and refresh pipeline

  SUNsung • 07/16/2026 at 18:00 • 0 comments

  This log is about the journey of a single version inside yggvault: from a line in the config to the routes for the browser, Go tooling, Composer, JSON API, and universal archives.

  The main idea: strict architecture is one of the project's core features. A version is better left out of the publish path than placed into storage in an unsafe or ambiguous form. This applies to archive parsing, symlinks, path rules, size limits, Go module zip constraints, and brother verification.

  A config entry as identity

  A minimal mirror starts with release_mirrors:

  release_mirrors:
    errors: "https://github.com/go-faster/errors"

  errors becomes the local key. This key is part of the public path, the Go module path, archive URLs, JSON endpoints, and feeds.

  The source URL is stored as the first binding for the key. If the operator later changes the URL for an already-published key, yggvault brings up the last-known-good data and suspends update checks for that key. This is a guard against accidental source substitution: a name must continue to mean the same upstream project.

  A single upstream URL under several keys is also treated as a suspicious config. The affected keys are suppressed until it's fixed.

  Source discovery

  Before downloading versions, the node classifies the configured URL.

  The source layer tries service endpoints like /health and /info. If the endpoint responds like a yggvault node, brother discovery kicks in: Brother.Hello, remote key detection, RPC capability, public API fallback.

  If the URL looks like an ordinary git forge, the source layer selects a provider parser. GitHub, GitLab, Bitbucket, and Gitea/Gogs-family servers are supported. A Yggdrasil address goes through the same source layer, only the outbound request leaves through the embedded mesh.

  The outbound layer already has guard rails: request deadlines, redirect limits, SSRF checks for clearnet, provider credentials only for matching hosts, bounded metadata reads, retry with backoff and jitter.

  Releases, tags, and sticky listing mode

  For a git source, the pipeline first looks at releases. If storable releases are found, the key is pinned to releases mode. If releases are empty, the pipeline tries tags and may pin the key to tags mode.

  The listing mode is sticky: the decision is saved to storage and survives restart. This is needed so that a single key doesn't change its package history just because the upstream later started publishing a different listing type.

  In tags mode, yggvault does a light releases probe. When the upstream starts serving releases for a key pinned to tags, the node raises a diagnostic conflict. The operator resolves the migration by hand.

  The listing includes versions that can be stored and published. Release prereleases are dropped. Semver prerelease tags are dropped too. Raw names are allowed only after route-safe validation.

  First import

  The initial import for a git source goes like this:

  1. get the version listing, honoring rescan.initial_depth;
  2. get the archive URL for each selected version;
  3. download the source archive into a temporary spool;
  4. compute the source hash;
  5. unpack the archive through mod/archive;
  6. check the archive boundaries;
  7. build the canonical source tree;
  8. determine Go/Composer viability and the artifact plan;
  9. build universal archives, Go artifacts, Composer metadata, and JSON-visible metadata;
  10. make a durable commit through PublishStaged.

  yggvault works with source release archives. Binary assets, installers, and other upstream attachments stay outside the pipeline.

  The download layer can resume via HTTP Range if the server provides a validator (ETag or Last-Modified). Downloading is bounded by request timeout, idle timeout, throughput floor, archive size cap, and source.download_max_parallel.

  Strict archive boundary

  Archive extraction is an abuse boundary, so the code treats the archive as untrusted input.

  Default limits from the config schema:

  • compressed archive: 100mb;
  • total unpacked bytes:...

  Read more »

• Why release mirroring matters

  SUNsung • 07/16/2026 at 17:56 • 0 comments

  yggvault started with a narrow breakage. The Go module was known, the required version was known, the dependent project had built before. Then the repository path stopped working, and the build failed while downloading the module archive.

  The first version was a personal utility for that scenario. It took GitHub releases, stored the source archives locally, and served them through an endpoint that Go tooling understood. Its scope was narrow: selected Go modules, my own mirror, less dependence on someone else's repository URL.

  I came back to the project because of Yggdrasil. I wanted to fetch my own work over a Yggdrasil address, without being tied to DNS, a public domain, and ordinary web ingress. Since the temporary server had to be touched seriously anyway, it made sense to carve out a proper model once: source discovery, local storage, archive validation, derived artifacts, routes, CLI, metrics, and UI.

  Once that model existed, extending the project became easier. Universal .zip / .tar.gz, Composer metadata, JSON API, Web UI, Atom feeds, support for GitLab, Bitbucket, Gitea/Gogs-family servers, and brother mirrors all use the same stored source tree. The project grew from a temporary Go workaround into a small tool for release availability.

  And after four months of "refactoring", I realized I'd gotten too carried away, so I set myself a release deadline. As a wrap-up report I decided to write an article about the project, because while writing a report you often notice things you missed while working on the code. And that's how this article came to be. :)

  The initial pain: selected Go modules

  Originally everything was built for Go modules only, and that was simple. The module path is written in go.mod. The tooling expects to receive a version list, .info, .mod, and .zip.

  But even in such a simple scheme, the subtlety with imports turned out to run deeper. At first I rewrote the import address across all text files with a regular expression. I knew there could be problems, but over all the time I used it for my own tasks, I never hit any. When I got down to rewriting it properly, though, I also took up thorough testing across every possible case for gathering information. And it turned out that this approach really is dangerous — especially for repositories that hold modules for very old versions.

  The problem was solved, but that was only the beginning.

  Why the release archive matters

  A source release archive often looks like a secondary detail: the repository exists somewhere, the source can be obtained. In practice, build systems are tied to a specific way of getting a specific version.

  When that breaks, a local checkout doesn't always help. A CI runner can be ephemeral. A consumer may have access to a package route while git access is closed. A package manager may expect its own protocol. And there's also team development and deployment from scratch.

  Initially I wasn't even thinking about anything beyond Go modules, but while working on efficient storage and serving, I realized I was already storing all the data needed to serve a "clean release" anyway — so the functionality could be extended. That's how yggvault came to be in its current form.

  The yggvault model

  In a minimal config, the mirror starts with release_mirrors:

  release_mirrors:
    errors: "https://github.com/go-faster/errors"

  release_mirrors binds a local key to an upstream URL. After a rescan, the node finds versions, downloads the source tree, validates the archive, stores it in local durable storage, and builds derived artifacts.

  The same release then becomes available through several interfaces:

  http://127.0.0.1:8080/errors/latest
  http://127.0.0.1:8080/errors/v0.7.1.tar.gz
  http://127.0.0.1:8080/errors/@v/list
  http://127.0.0.1:8080/errors/releases.json

  For a human, it's a web page and archive links. For Go tooling, it's a GOPROXY endpoint. For automation, it's JSON routes and feeds.

  After a successful sync, already-stored versions keep being...

  Read more »

View all 6 project logs