How yesod publishes `yesod-mirror`

Copybara on a nightly Forgejo Actions job, run through Bazel. A write-up for someone setting up the same pattern on their own forge and CI.

Space for context from the operator: why the mirror exists, who reads it, anything that isn't in the code.

1. The pieces at a glance

yesod is a private monorepo on a self-hosted Forgejo. yesod-mirror is a public repo on the same Forgejo that holds a filtered snapshot of yesod's main. Once a night a CI job runs Copybara. Copybara reads yesod, drops the paths that must not leave the private repo, renames one directory, and pushes the result as a single new commit to the mirror. A Sourcebot instance, source-mirror, indexes the mirror, so the public subset can be browsed and searched. Every link in this document goes through it.

Data flows one way. Nothing is ever written back to yesod, and nobody commits to the mirror by hand.

Piece	File	Job
Trigger and runner	`.forgejo/workflows/publish-yesod-mirror.yml`	Nightly cron plus a manual button. Sets up git credentials and runs Copybara.
Copybara binary	`MODULE.bazel` (`copybara_jar`) and `tools/copybara/BUILD.bazel`	Pins the upstream release jar by SHA-256 and wraps it as a runnable Bazel target.
What gets copied	`tools/copybara/yesod-mirror/copy.bara.sky`	Origin, destination, include and exclude globs, path moves, authorship.
Credentials	Repository secret `YESOD_MIRROR_TOKEN`	One Forgejo token that can read yesod and push to yesod-mirror.
Consumer	Sourcebot config in `k8s/configs/environments/sourcebot/main.jsonnet`	A second Sourcebot instance that indexes only `acmcarther/yesod-mirror`, with anonymous access.

Dependencies, in order: the job needs a CI runner with git and Bazel (via bazelisk). Bazel fetches a JDK and the Copybara jar. Copybara needs network access and credentials for both repos. Sourcebot only needs to be able to read the mirror.

Notes on the overview.

2. The CI workflow

Source: publish-yesod-mirror.yml. In yesod it lives under .forgejo/workflows/. In the mirror it appears under .forgejo-disabled/workflows/ (see the move). The file is short, so here are its steps with the internal details generalised.

Trigger

on:
  schedule:
    - cron: '0 1 * * *'   # 01:00 nightly
  workflow_dispatch:      # manual "Run workflow" button

It does not run on push. The mirror is up to a day behind main, which is fine for a browse-and-search copy. Use workflow_dispatch when you want an update now.

The workflow also declares permissions: packages: write. That line was copied from the Homebrew workflow, which does publish packages. The mirror job doesn't need it.

Container

The job runs on a docker-labelled Forgejo runner, inside yesod's own CI base image. That image already contains git and a bazelisk binary installed as bazel. A Copybara job needs only those two things. Java comes from Bazel (next section), so the image needs no JDK.

Bazelisk and the binary cache

- name: Link baked-in bazelisk
  run: |
    mkdir -p "${GITHUB_WORKSPACE}/bin/"
    ln -sf "$(command -v bazel)" "${GITHUB_WORKSPACE}/bin/bazel"
    echo "${GITHUB_WORKSPACE}/bin" >> $GITHUB_PATH

Older versions of this workflow downloaded bazelisk with curl on every run. Now the image ships it, and this step only puts it on PATH. When the Copybara step runs, it sets BAZELISK_BASE_URL to an internal binary cache. Bazelisk then downloads the Bazel release named in .bazelversion from that cache instead of from GitHub. This is a speed and reliability choice only. On a hosted runner you can leave BAZELISK_BASE_URL unset and bazelisk will use GitHub releases.

Credentials: the git `insteadOf` rewrite

Copybara runs plain git underneath, against the clean URLs in copy.bara.sky. The workflow puts the credential in git's global config, not in the Copybara config:

git config --global user.name  "Copybara"
git config --global user.email "copybara@<your-domain>"
git config --global \
  url."https://<user>:${{ secrets.YESOD_MIRROR_TOKEN }}@<forge>/<owner>/yesod-mirror.git".insteadOf \
      "https://<forge>/<owner>/yesod-mirror.git"
git config --global \
  url."https://<user>:${{ secrets.YESOD_MIRROR_TOKEN }}@<forge>/<owner>/yesod.git".insteadOf \
      "https://<forge>/<owner>/yesod.git"

Whenever git sees the plain URL, it uses the URL with the token in it. Copybara never knows a credential exists, and copy.bara.sky holds only public URLs, so the config can itself be mirrored.
Both URLs are rewritten. yesod is private, so Copybara's fresh clone of the origin needs the token too, not just the push to the mirror. The actions/checkout clone only supplies the .sky file. Copybara clones the origin again on its own.
One token, YESOD_MIRROR_TOKEN, covers both: read on the source, write on the destination. The git history suggests the job first used a general token and later moved to a dedicated one. A dedicated, narrowly scoped token is the better pattern.
The token sits in the runner container's ~/.gitconfig while the job runs. That is acceptable for an ephemeral CI container. Don't copy this step onto a long-lived shared machine.

Running Copybara, and why exit code 4 means success

bazel run //tools/copybara:copybara --config=remote --java_runtime_version=remotejdk_21 -- \
  migrate \
  "${GITHUB_WORKSPACE}/tools/copybara/yesod-mirror/copy.bara.sky" \
  --force \
  || { EC=$?; [ "$EC" -eq 4 ] || exit "$EC"; }

Copybara exits with code 4 (NO_OP) when there is nothing to migrate: the origin revision has already been migrated, or the filtered tree is the same as what the destination has. If yesod's main didn't change any mirrored file since the last run, the nightly job hits this case. Without the wrapper the job would go red on every quiet day. The || clause swallows only exit code 4 and passes any other non-zero code through, so real failures (auth errors, a bad config, push rejected) still fail the job.

--config=remote turns on yesod's shared Bazel remote cache. It is optional. The build is just one java_binary around a downloaded jar, and it builds fine without a cache.

Notes on the workflow.

3. Running Copybara under Bazel

Copybara publishes a self-contained copybara_deploy.jar on its GitHub releases. yesod pins one of them in MODULE.bazel:

http_file(
    name = "copybara_jar",
    downloaded_file_path = "copybara_deploy.jar",
    sha256 = "a1feedd74d85d55f5e2c965aa81d42b6bba2f8d99674453e6926c4bb0dbd8feb",
    urls = ["https://github.com/google/copybara/releases/download/v20260112/copybara_deploy.jar"],
)

It wraps the jar in tools/copybara/BUILD.bazel:

load("@rules_java//java:defs.bzl", "java_binary", "java_import")

java_import(
    name = "copybara_import",
    jars = ["@copybara_jar//file"],
)

java_binary(
    name = "copybara",
    main_class = "com.google.copybara.Main",
    runtime_deps = [":copybara_import"],
    visibility = ["//visibility:public"],
)

java_import turns the downloaded file into a Java library target. java_binary adds a launcher with the right main_class, so bazel run //tools/copybara:copybara -- <args> works like the copybara CLI.
--java_runtime_version=remotejdk_21 makes Bazel download a hermetic JDK 21 and run Copybara on it. Copybara ships class files at version 65, so it needs Java 21 or newer. Its own README recommends this exact flag. With this flag the runner image needs no Java at all. yesod's .bazelrc already sets it for every build. The workflow repeats it on the command line, so the job doesn't depend on that.
To upgrade, change the release tag in the URL and the sha256. Nothing else moves.

If you don't use Bazel, you need the same three things by other means: a JDK 21, the pinned jar, and java -jar copybara_deploy.jar migrate copy.bara.sky --force. Bazel only gives yesod reproducible pins for the jar and the JDK.

Notes on the Bazel wrapping.

4. The Copybara config

Source: tools/copybara/yesod-mirror/copy.bara.sky. Here is the whole file, with the URLs generalised:

sourceUrl      = "https://<forge>/<owner>/yesod.git"
destinationUrl = "https://<forge>/<owner>/yesod-mirror.git"

core.workflow(
    name = "default",
    origin = git.origin(
        url = sourceUrl,
        ref = "main",
    ),
    destination = git.destination(
        url = destinationUrl,
        fetch = "main",
        push = "main",
    ),
    origin_files = glob(["*", "**/*"], exclude = [
        "notes/**",
        "**/*.sops.yaml",
        "experimental/users/acmcarther/k8s/**",
    ]),
    authoring = authoring.pass_thru("Copybara <copybara@<your-domain>>"),
    transformations = [
        core.move(".forgejo", ".forgejo-disabled"),
    ],
)

(The file's header comment says tools/homebrew/copy.bara.sky. That is a leftover from copying the Homebrew config and doesn't matter.)

Origin and destination

git.origin reads main of the private repo. git.destination fetches the mirror's main, so Copybara can find the last migrated revision, and pushes the new commit to the same branch. No mode is set, so the workflow uses Copybara's default, SQUASH: each run makes one destination commit containing the filtered tree at the origin's head (see §5).

What is excluded, and why

Exclude	Why
`notes/**`	Personal notes. Not secret, but not meant to be public.
`*/.sops.yaml`	SOPS-encrypted secret files. The values are encrypted, but the mirror has no use for them, and encrypted secrets are better kept off a public repo: key names, file layout and ciphertext all leak something, and an encryption key could be compromised later.
`experimental/users/acmcarther/k8s/**`	The operator's personal Kubernetes environments. These are excluded as a whole subtree, not file by file.

The include glob is ["*", "**/*"]. * matches top-level files and **/* matches everything below. Copybara applies origin_files to the whole tree, and excluded files simply don't exist in the destination. The mirror's API confirms this: notes/, experimental/users/acmcarther/k8s/ and, for example, k8s/configs/environments/monitoring/secrets.sops.yaml all return 404 there.

Glob gotcha. **/*.sops.yaml does not match a top-level file named exactly .sops.yaml, and that file is in the mirror. In yesod it is harmless: it is the SOPS config (creation rules that list public age recipients) and holds no secrets. The lesson still applies. Test your excludes against the real tree (see the checklist) instead of trusting how a glob reads.

Authoring: `authoring.pass_thru`

pass_thru(default) keeps the origin commit's author. Copybara's reference says the default author "is used in squash mode workflows or if author cannot be determined". This workflow runs in SQUASH mode, so every mirror commit is authored and committed by Copybara. pass_thru would only start passing real authors through if you switched to ITERATIVE. The message is always:

Project import generated by Copybara.

GitOrigin-RevId: <full SHA of the yesod commit>

The other modes are authoring.overwrite(...), which always uses one fixed author, and authoring.allowed(...), which passes authors through only from an allow-list. For a public mirror, think about whether you want contributors' names and emails published. SQUASH mostly answers that question for you.

The `.forgejo` → `.forgejo-disabled` move

Forgejo Actions runs any workflow it finds under .forgejo/workflows/ (and GitHub-style .github/workflows/). Mirrored as-is, yesod's CI definitions would try to run in the mirror every time Copybara pushed. They would fail for lack of secrets and runners, or worse, run on a runner that happens to be shared. core.move(".forgejo", ".forgejo-disabled") keeps the workflows readable in the mirror, which is how you are reading this one, while Forgejo no longer treats them as workflows. Excluding .forgejo/** would also work, but then the CI setup would be unpublished.

The move is also the reason source-mirror links to workflows go through .forgejo-disabled/.

Notes on the config: other excludes considered, plans to widen or narrow the subset.

5. History, squash and `--force`

SQUASH mode: each nightly run makes one mirror commit with the filtered tree at yesod's head, tagged GitOrigin-RevId. yesod's own history (authors, messages, intermediate states) is never published.

To work out what is new, Copybara reads the destination's history and looks for the most recent GitOrigin-RevId: label. That commit is the last one migrated, and anything after it in the origin is pending.

First migration. The destination has no label yet: it is empty, or it holds only the initial commit Forgejo makes when a repo is created with a README. Copybara refuses to migrate because it can't find "a change that is an ancestor of the one(s) being migrated", so it can't rule out losing work. --force overrides that check, and you need it (or the first-run-only --init-history) on the first run. Copybara then writes the whole filtered tree as one commit on top of whatever the destination already had.
Later migrations. The label exists, so --force is not strictly needed. Copybara finds the last migrated SHA, sees new origin commits, and squashes them into one new commit. With nothing new it exits 4 (see exit code 4).
Why yesod keeps --force on every run. It keeps the nightly job working through odd states: the destination label missing or unreadable, a rewritten origin history where the last migrated SHA is no longer an ancestor of main, a destination someone reset by hand. The cost is that Copybara's safety check is off for good. If someone commits to the mirror directly, the next run silently writes over their tree. For a mirror that is only ever written by the job, that is the right trade-off.
The history is not reproducible from the mirror. SQUASH publishes snapshots, not commits. If you want the origin's commit-by-commit history in the mirror, use mode = "ITERATIVE". Then every origin commit is published, so its message, author and every intermediate state of a file you later excluded, if it was ever outside the exclude list, go public too. Audit the entire history before choosing that.

The mirror currently has about 70 Copybara commits, starting in January 2026. yesod's own history has thousands.

Tightening the exclude list doesn't take anything back. Adding a path to exclude removes it from the next snapshot. Every earlier mirror commit still contains it. If something sensitive was ever mirrored, fixing the config is not enough: you have to rewrite or recreate the destination repo, rotate whatever leaked, and re-run with --force.

Notes on history, for example whether the mirror has ever been recreated.

6. A second example: the Homebrew tap

The same pattern publishes yesod's internal CLI tools as a public Homebrew tap, yesod-homebrew-tools. It differs from the mirror in three useful ways.

Sources: publish-homebrew-packages.yml, tools/copybara/homebrew/copy.bara.sky, homebrew/README.md.

	yesod-mirror	Homebrew tap
Selection	Everything minus excludes	An allow-list: `origin_files = glob(["homebrew/*.rb"])`
Path move	`.forgejo` → `.forgejo-disabled`	`core.move("homebrew", "")`, so formulas land at the tap's root, where Homebrew looks for them
Origin	The remote yesod repo	The CI workspace: the job generates fresh `.rb` formulas, commits them locally, and runs `sed` to point `sourceUrl` at `file://${GITHUB_WORKSPACE}`. That commit is never pushed to yesod. Copybara reads it from the local checkout.
Trigger	Nightly and manual	Push to `main` touching `tools/multitool/`, `homebrew/` or `tools/copybara/homebrew/**`; also nightly and manual
Credential	`YESOD_MIRROR_TOKEN`	`HOMEBREW_REPO_TOKEN`, push to the tap only. The local `file://` origin needs no credential.

Before Copybara runs, the tap job builds binaries for several platforms and uploads them to Forgejo's generic package registry. It treats HTTP 409 ("version exists") as success, the same idea as exit code 4. It then generates formulas that point at those uploads, and it skips Copybara entirely when git diff --cached --quiet shows the formulas didn't change. Exit-4 handling and --force are the same as in the mirror job.

Takeaway for porting: an allow-list (glob(["dir/**"])) is much safer than a deny-list when the public part is small. The mirror uses a deny-list only because nearly the whole repo is meant to be public.

Notes on the tap.

7. Downstream: code search over the mirror

yesod runs two Sourcebot (Zoekt-based code search) instances. The private one indexes yesod and a few other repos. The second, source-mirror, has a single connection to Forgejo that lists one repo, acmcarther/yesod-mirror, and it has anonymous access turned on. Sourcebot polls the forge, so new mirror commits show up in search shortly after the nightly push. In practice the searchable copy is about a day behind main. The configuration is in k8s/configs/environments/sourcebot/main.jsonnet.

Because the index covers only the mirror, a public search can never surface excluded files: they were never pushed. If you want to share search without sharing the whole private repo, point the public index at the mirror, not at the source with a path filter.

yesod's own tooling links to the mirror too. Its internal cluster dashboard can link to source files through source-mirror, and its code-search CLI docs use repo:yesod-mirror as the example filter.

Notes on source-mirror.

8. Porting checklist

What you need

A forge with two repos: the source (private) and an empty destination (public). Copybara's git.origin and git.destination work with any git host: Forgejo, Gitea, GitHub, GitLab.
A token that can read the source and push to the destination's target branch, stored as a CI secret. Prefer a dedicated bot account with only those two permissions. If the source is public, the token only needs to push.
If the destination branch is protected, allow that bot account to push to it, including over its own earlier commits.
A CI runner with git, network access to the forge, and either Bazel/bazelisk or a JDK 21.
Copybara: the upstream copybara_deploy.jar from GitHub releases, pinned by SHA-256. Wrap it in Bazel as in §3, or run java -jar.

What you decide

Allow-list or deny-list. Use an allow-list if the public part is small. Use a deny-list (like yesod) only if nearly everything is public, and then review it whenever a new top-level directory appears.
Excludes: personal notes, encrypted or plaintext secret files, environment and inventory configs, anything with internal hostnames or addresses you don't want listed, vendored code with incompatible licenses.
Path moves: at minimum, disable the source's CI definitions (.forgejo, .github, .gitlab-ci.yml) so they don't run in the mirror. Exclude them, or move them as yesod does if you want them readable.
Mode: SQUASH (snapshots, the default, no source history published) or ITERATIVE (every commit published).
Authoring: pass_thru, overwrite or allowed. With SQUASH this mostly comes out as "Copybara".
Cadence: a nightly cron is enough for a reading copy. Add a push trigger if readers need it fresher.

Pitfalls

Test your excludes before the first push. Run Copybara with a local folder destination and look at what would be published. Copybara can write to a directory instead of a git remote (folder.destination(), or the --folder-dir flag), or you can push to a private scratch repo first. grep the output for secret-looking file names and for strings like BEGIN, password and token. Remember the .sops.yaml case: **/*.x may not match a top-level .x.
The first run needs --force, because the destination has no GitOrigin-RevId yet. If you keep --force on later runs, Copybara won't stop a run that overwrites manual commits in the destination. Treat the destination as write-only by the bot.
Treat exit code 4 as success, and only 4, or the job fails every day nothing changes.
Credentials go in git config (insteadOf), not in copy.bara.sky. Rewrite both URLs if the source is private. Never echo the rewritten URL into logs.
Leaks are permanent in history. A file mirrored once stays in every earlier destination commit. Check before the first push, not after.
The source's CI firing in the mirror is the most visible failure, and the move or exclude above prevents it.
Java version: recent Copybara releases need JDK 21. An older JDK on the runner fails with a class-version error. Bazel's remotejdk_21 avoids the problem.

Notes for the peer's own setup.

9. Source files

All links go to the public mirror through source-mirror. CI files are under .forgejo-disabled/ there. Paths that are excluded from the mirror are deliberately not linked.

.forgejo-disabled/workflows/publish-yesod-mirror.yml: the mirror job
tools/copybara/yesod-mirror/copy.bara.sky: the mirror config
tools/copybara/BUILD.bazel: the Bazel wrapper
MODULE.bazel: the copybara_jar pin (search for copybara)
.bazelrc: the repo-wide remotejdk_21 setting
.forgejo-disabled/workflows/publish-homebrew-packages.yml: the tap job
tools/copybara/homebrew/copy.bara.sky: the tap config
homebrew/README.md: the tap from the user's side
k8s/configs/environments/sourcebot/main.jsonnet: the Sourcebot instances, including source-mirror
Upstream: github.com/google/copybara and its config reference

Further links.

Written from yesod main on 2026-11-01. If the code and this page disagree, trust the files linked above. This page lives at tools/copybara/README-mirror.html.

How yesod publishes yesod-mirror

1. The pieces at a glance

2. The CI workflow

Trigger

Container

Bazelisk and the binary cache

Credentials: the git insteadOf rewrite

Running Copybara, and why exit code 4 means success

3. Running Copybara under Bazel

4. The Copybara config

Origin and destination

What is excluded, and why

Authoring: authoring.pass_thru

The .forgejo → .forgejo-disabled move

5. History, squash and --force

6. A second example: the Homebrew tap

7. Downstream: code search over the mirror

8. Porting checklist

What you need

What you decide

Pitfalls

9. Source files

How yesod publishes `yesod-mirror`

Credentials: the git `insteadOf` rewrite

Authoring: `authoring.pass_thru`

The `.forgejo` → `.forgejo-disabled` move

5. History, squash and `--force`