Why every team has a pipeline.

Two jobs hide behind the letters in the title. One is remembering every version of a project, so several people can change it at once and anyone can go back in time. The other is handing every change to a fresh, borrowed computer that checks it within minutes and, if it passes, puts it in front of users without anyone lifting a finger.

The reason to care is the gap between making a mistake and finding out. Push a change to your ranker’s training code and learn two minutes later that it broke something, and the fix is obvious. Learn it three weeks later, on merge day, with 140 changes to suspect, and you lose an afternoon or a week. Serious teams ship models this way, behind checks no one can skip, and interviewers expect you to know why.

It builds on Containers vs VMs, because the fresh computer is the same “clean and identical every time” idea. By the end, the site you’re reading will be checked and published by an automatic assembly line you wrote. Next comes Logs, metrics and traces, which answers the question shipping leaves open: did the change actually work once it was live?

Today

• Say what git actually keeps on disk: not a list of edits, but full snapshots of the project, each filed under a fingerprint of its own contents, with a branch being a tiny file that points at one snapshot. You’ll prove it by opening git’s storage folder by hand.
• Explain, with arithmetic, why ten small merges hurt less than one big one, and why a bug found in two minutes is cheaper than the same bug found in two weeks.
• Tell apart the three things people mean by “CI/CD” (checking every change automatically, keeping every checked change ready to ship, and shipping it automatically), plus the handful of words around them: the packaged build you ship, the places it runs, going back to the last good version, switches that hide unfinished code, everyone working on one main line, and rules that stop unchecked code from merging.
• Build a real automatic assembly line for this site: on every push, a fresh borrowed computer installs, builds, checks every link, checks every lesson page exists, and publishes to GitHub Pages. Then break the live site on purpose and time how fast you can undo it.
• Map all of it onto ML: what “all the checks passed” should mean for a model, and where your validation harness fits.

You only need to have used git and GitHub Actions before. Budget one evening for the reading and for poking around inside git’s storage, and one for the assembly line and the undo drill.

The Problem

You’ve been on a branch for three weeks. It adds hard-negative mining to the cross-encoder’s training data: a new sampler, a change to the feature pipeline, a tweak to the loss. Locally everything passes and NDCG@10 is up two points. Meanwhile a teammate has refactored the feature pipeline you touched, renamed the function your sampler calls, and moved the tokenizer config.

Merge day. Git reports 14 conflicting files. You resolve them over an afternoon, mostly by guessing what the other person meant. The code now merges, but doesn’t run: your sampler calls a function that no longer exists under that name. You fix that. Now it runs, but NDCG is down a point and nobody knows whose change did it, because there are 140 commits between the last known-good state and now. And when your teammate tries it on their machine, it fails outright: your code reads a cached tokenizer file that exists only on your laptop.

None of these problems are about git commands. You know the commands. They’re about time: how long work stays apart before it’s combined, and how long a mistake lives before anyone sees it. Source control and CI/CD are, at heart, tools for shrinking both.

The Idea

Source control is a history of snapshots

Strip away the commands and source control answers three questions: what does the project look like right now, what did it look like at any point in the past, and how do several people change it at once without trampling each other?

The oldest tools (SCCS at Bell Labs in 1972, then RCS and CVS) tracked one file at a time, storing each file’s changes as a chain of edits. Subversion (2000) tracked whole-project versions but kept one central server that everyone had to talk to. Git took a different and much simpler idea, and that idea is the thing worth understanding.

The core idea is content addressingcontent addressingNaming a piece of data by a fingerprint computed from the data itself, instead of by a location or a made-up ID. Same content, same name, always.. Git takes your file’s bytes, computes a hashhashA function that turns any amount of data into a fixed-size fingerprint (here 40 hex characters, SHA-1). Change one bit of input and the fingerprint changes completely. Two different inputs having the same hash is practically impossible. of them, and stores the bytes under that hash. The hash is the name. Git keeps exactly three kinds of objects (plus a fourth, tags, which we’ll skip):

• A blobblobGit’s name for the raw contents of one file. No filename, no permissions, just bytes. is one file’s contents. Just bytes. Not even the filename.
• A treetreeA git object listing one directory: for each entry, its mode, its name, and the hash of the blob (file) or tree (subfolder) it points to. is one directory: a list of (name, hash) pairs, pointing at blobs and other trees.
• A commitcommitA git object that points at one root tree (the full snapshot of the project), the commit(s) before it, an author, a time, and a message. points at one root tree, which is a full snapshot of the project, plus the commit (or commits) that came before it, and who, when, why.

That’s the whole data model. A commit doesn’t store “what changed”. It stores the entire project, by pointing at a tree. It looks expensive, but content addressing makes it nearly free: if a file didn’t change, its blob has the same hash, so the new tree just points at the old blob. Only the things that changed get new objects.

And now the part that makes git feel like magic once you see it: a branchbranchIn git, a file containing one commit hash. That’s all. “Being on a branch” means new commits will move that pointer forward. is a 41-byte text file. .git/refs/heads/main holds a commit hash and a newline. Making a branch writes one more such file. HEAD is a file that says which branch you’re on (ref: refs/heads/main). Committing creates objects and then moves the pointer. Nothing else.

Three consequences you already rely on without knowing why:

1. History can’t be quietly edited. A commit’s hash covers its tree and its parent’s hash, which covers its parent, all the way down. Change one byte in an old file and every hash above it changes. This is the same chain-of-hashes trick as a blockchain, used fifteen years earlier for something useful. It’s why git rebase gives your commits new hashes: they are genuinely new commits.
2. Branches are free. Creating one writes 41 bytes. That’s why git encouraged a branching style that would have been unthinkable in CVS, where a branch meant copying history around on a server.
3. Every clone is a full backup. The object store is the whole history. GitHub holds one more copy; it’s not special to git, only to your team.

A note on “git doesn’t store diffs”: as an optimisation, git gc later packs objects into packfilepackfileA single compressed file holding many git objects, where similar objects are stored as deltas against each other to save space. A storage trick only; the model is still snapshots.s and delta-compresses similar ones. But that’s a storage detail underneath. The model is snapshots, and git diff computes differences on demand by comparing two trees.

Merging, and why small merges win

Now the part that bit you on merge day. When you merge branch B into main, git finds the merge basemerge baseThe most recent commit that both branches have in their history: the point where they split., the last commit both sides share, and does a three-way mergethree-way mergeCombining two versions of a file by comparing each to their common ancestor. A change made on only one side is taken automatically. Different changes to the same lines on both sides is a conflict.: for each file, compare base to main and base to B. If only one side changed a region, take that change. If both sides changed the same region differently, that’s a merge conflictmerge conflictWhen both branches changed the same part of the same file since they split, and git can’t know which version (or which combination) is right. A human has to decide., and a human must decide. The result is a commit with two parents.

Conflicts aren’t random bad luck. They have arithmetic.

Say the repo has 1,000 files and each developer touches about 10 different files a week, spread at random. Two developers on week-long branches: what’s the chance their branches touch at least one common file? Each of my 10 files has a 1% chance of being in your 10, so the chance of no overlap is about 0.99^10 ≈ 0.90, so roughly a 10% chance of a conflict.

So integration pain scales like (people)² × (branch lifetime)² per merge. Merging four times as often divides the pain of each merge by about sixteen and multiplies the number of merges by four, so the total still drops by about four. And each conflict you do hit is smaller and fresher: you wrote the code this morning, so you remember what it was for.

There’s a worse kind of conflict that git can’t even see. A semantic conflictsemantic conflictTwo changes that merge cleanly line-by-line but break when combined, e.g. one branch renames a function while another adds a new call to the old name. merges with no complaint and is broken anyway. Your sampler called build_features(); your teammate renamed it featurize(). Different files, no textual overlap, a clean merge, and a crash. Only building and running the combined code catches this. Hold that thought; it’s the whole argument for CI.

Play with it. Same edits, same bugs, two schedules: merge every 20 days, or every day.

The model is a toy (600 files, a few hot ones that everybody touches, 2% of edits carry a bug, tests catch a bug at the next merge), but the shape is honest. Three things to try:

• One developer: no conflicts at any schedule. Integration hell is a team problem. Bugs still hide until a merge runs the tests, which is the part that’s about feedback, not people.
• Drag developers from 2 to 12 with big-bang on. Conflict work grows much faster than linearly, the N² pairs.
• Drag the big-bang interval down. Somewhere around every 2 to 5 days most of the pain is gone. The win isn’t “daily” as a magic number. It’s “short enough that branches can’t drift far”.

The feedback loop: the cost of a bug is mostly time

The orange lines in that figure are the second half of the story. A bug is cheap at the moment you write it: you’re looking at the code, it’s in your head, and the only suspect is the last thing you typed. Every hour it survives makes it more expensive, for mechanical reasons, not mystical ones:

• More suspects. Found right after one commit, there’s one place to look. Found after 140 commits, you’re bisecting.
• Lost context. In two weeks you’ve forgotten why you wrote it, and the person who wrote the code next to it may be on vacation.
• Things built on top. Other code, other people’s branches, a trained checkpoint, a report to your manager, all assumed the bug wasn’t there.
• Blast radius. In your editor it hurts you. In production it hurts users, and maybe their data.

You will often see a chart claiming a bug costs 100x more to fix in production than in design, attributed to Barry Boehm’s 1981 book. Be careful with that number: the original data came from big 1970s projects, and later people (Laurent Bossavit’s The Leprechauns of Software Engineering is the fun takedown) showed how thin and mangled the evidence behind the famous curve is. The direction is not in dispute; the exact multiplier is. Argue from the mechanism above, not from the chart.

Continuous integration, precisely

Now the definition will sound obvious, which is the point.

Continuous integrationContinuous integrationThe practice of everyone merging their work into the shared main line at least daily, with every merge automatically built and tested on a clean machine, and a broken build fixed before anything else. Often shortened to CI. is a practice, not a tool: everyone merges into the shared main line at least once a day, and every merge is automatically built and tested, and if it breaks, fixing it is the team’s top priority. Fowler’s article lists the habits that make it work. The core ones:

• One shared main line, and everybody merges into it often. If people keep branches for weeks and “use CI” on them, they’ve bought the tool and skipped the practice.
• The build is automated and the build is self-testing: one command gets you from source to a checked, runnable thing.
• Every change builds on a separate, clean machine, not just on the author’s laptop.
• Keep it fast. Fowler’s rule of thumb is about ten minutes. A slow build gets skipped, batched, or ignored, and then you’re back to big merges.
• A red build is fixed immediately, usually by reverting the change, not by debugging on top of it.

The tool side has a handful of words. A pipelinepipelineThe automated sequence of steps a change goes through: install, build, test, package, deploy. Each step must pass for the next to run. is the sequence of steps. In GitHub Actions a workflowworkflowA YAML file in .github/workflows/ that says when to run (on push, on pull request, on a schedule, by hand) and what jobs to run. defines one, made of jobs, each made of steps, each running on a runnerrunnerThe machine (usually a fresh virtual machine) that executes a CI job, then gets thrown away.. The result gets reported back to the pull request as a status checkstatus checkA pass/fail mark that CI attaches to a commit. GitHub can be told to block merging until certain checks pass..

Why insist on a clean machine? Because your laptop lies. It has the cached tokenizer from the opening scene. It has a package you pip installed by hand six months ago. It has a case-insensitive filesystem (macOS does by default), so import "./Figure.astro" finds a file called figure.astro, and the Linux server won’t. “Works on my machine” isn’t a joke about careless people; it’s the default state of every machine that’s been used for a while. A fresh runner starts from nothing but what’s written down in the repo: the code, the lockfilelockfileA file (package-lock.json, uv.lock, poetry.lock) recording the exact version of every dependency, including dependencies of dependencies, so every install gets the same bytes., and the workflow. If the build passes there, it’ll pass anywhere that starts from the same things. That’s also why CI uses npm ci rather than npm install: ci installs exactly what the lockfile says, and fails if package.json and the lockfile disagree, instead of quietly updating the lockfile.

Continuous delivery, and the things that make it safe

CI answers “does the combined code work?”. The next question is “how does it reach users?”, and here the two “CD”s split:

• Continuous deliveryContinuous deliveryEvery change that passes the pipeline is releasable at the push of a button, and the release is automated. A human still decides when.: every green build could go to production, with one button and no heroics. A human chooses when.
• Continuous deploymentContinuous deploymentEvery change that passes the pipeline goes to production automatically, with no human step.: every green build does go out, automatically. Your site will work this way.

Either way, a few ideas carry the weight.

Build once, promote the artifact. An artifactartifactThe packaged output of a build: a folder of static files, a container image, a wheel, a model checkpoint. Identified by a version or hash, stored, and deployed as-is. is the thing the build produced: for this site, the dist/ folder; for a service, a container image; for you, often a model checkpoint. The rule is to build it once, test it, and deploy those exact bytes, rather than rebuilding for each stage and hoping the rebuild matches. It’s the same idea as git’s content addressing: if you can name exactly what you tested, you can deploy exactly what you tested.

Environments. An environmentenvironmentA place the software runs, with its own configuration and data: e.g. preview, staging, production. Same artifact, different settings. is somewhere the artifact runs. Typically a preview per pull request, a staging copy that mimics production, and production itself. The artifact moves through them unchanged; only configuration differs.

Rollbacks. A rollbackrollbackReturning production to a previous known-good version, ideally by redeploying an artifact that already exists rather than building a fix under pressure. is going back to the last version that worked. Notice how much this looks like a git branch: if production is “a pointer to artifact v42”, rolling back is moving the pointer to v41. That takes seconds and needs no new code, which is exactly what you want at 3am. The alternative, “roll forward” (write a fix, push it through the pipeline), is fine when the fix is obvious and the pipeline is fast, and a disaster when neither is true. One honest caveat: a rollback only undoes what the pipeline controls. If version 42 changed a database schema or wrote bad data, putting v41’s code back doesn’t un-write it. That’s why teams change databases in backward-compatible steps (add the new column, deploy code that writes both, then later drop the old one).

Feature flags. A feature flagfeature flagA switch, read at run time from config, that turns a piece of code on or off without deploying again. Lets code ship “dark” and be released later, or to a few users first. separates deploying code from releasing it. Your hard-negative sampler can be merged to main on day two, behind if flags.hard_negatives:, defaulting to off. It’s integrated (it builds, it’s tested, nobody’s branch drifts from it), but nothing changes for users until you flip it, and flipping it back is instant. That’s the trick that makes daily merging possible for work that takes three weeks. The Knight Capital story is the warning label: flags are code too. They reused an old flag’s name for new behaviour, a server still had the old code that read it, and nobody removed dead flags. Delete flags when they’ve done their job.

Trunk-based development. Put those together and you get trunk-based developmenttrunk-based developmentA way of working where everyone merges small changes into one main branch (the trunk) at least daily, using short-lived branches (hours to a couple of days) and feature flags for unfinished work, rather than long-lived feature or release branches.: one main branch, short-lived branches that live hours or a day or two, flags for unfinished work. Google famously runs most of its code in one huge repository on a single trunk with tens of thousands of engineers committing to it, and the CACM paper about it (Potvin and Levenberg, 2016) is worth reading for the scale alone. The Accelerate research (Forsgren, Humble and Kim, from the State of DevOps surveys) found trunk-based habits and frequent deploys go together with both faster delivery and fewer failures, which kills the intuition that moving fast and being careful are a trade-off.

Branch protection. The last piece makes the rules enforceable. branch protectionbranch protectionRepository settings (on GitHub, “rulesets” or “branch protection rules”) that stop anyone pushing to main directly, and require a pull request with passing status checks before merging. tells GitHub: nobody pushes to main directly; changes come through a pull request; the PR can’t merge until the CI check is green. Without it, CI is advice. With it, a red build physically can’t reach production.

What changes for ML

In ordinary software, the behaviour lives in the code, so testing the code tests the behaviour. In ML, the behaviour lives in three things: the code, the data, and the weights. A pipeline that only tests code will happily ship a model trained on a broken dataset. Google’s “Hidden Technical Debt in Machine Learning Systems” (Sculley et al., 2015) made this point memorably: the model code is a small box in the middle of a big diagram of data collection, feature extraction, configuration and monitoring, and most of the failures live in the big diagram.

So “the build is green” needs more checks, and each catches a different class of bug:

• Unit tests on the code: the tokenizer pads correctly, the loss is zero for a perfect prediction, the sampler never returns the positive as a negative. Fast, run on every push, exactly like normal CI.
• Data validation: the new training set has the expected columns and types, null rates and label balance haven’t jumped, no query in the eval set appears in the training set (leakage, the classic way to get a beautiful number that means nothing). Run when data changes, not just code.
• An eval gate: train or fine-tune (or for a big model, load the candidate checkpoint), run it on a frozen eval set, and fail the pipeline if the metric drops past a threshold versus the current production model. For your cross-encoder that’s NDCG@10 or MRR on a fixed query set. This is where your validation harness lives. A harness you run by hand is a test suite you run sometimes; wired into the pipeline as a required check, it becomes a gate nothing can skip.
• The model as an artifact: the checkpoint is stored with its hash, the commit that produced it, and the dataset version it saw. Deploying means pointing the server at a checkpoint ID; rollback means pointing it back. Same pointer trick, again.
• Gradual release: shadow traffic (the new model scores real queries, results logged but not shown) or a canary (1% of traffic), watched with metrics before going to 100%. A feature flag for models.

One ML-specific trap: an eval metric is noisy. If NDCG@10 on 500 queries wobbles by ±0.01 between two runs of the same model with different seeds, a gate at “fail if it drops 0.005” is a coin flip, and a flaky gate trains the team to click “re-run” until it’s green. Measure the noise first, then set the threshold above it. Is that improvement real? is the lesson on doing that properly.

Build It

Two parts. First you take git apart by hand, so “a commit is a pointer to a snapshot” is something you’ve seen rather than been told. Then you give this site the pipeline it deserves: you’re reading a lesson that, by the end, will have been tested and deployed by the pipeline it describes.

mkdir -p ~/scratch/gitguts && cd ~/scratch/gitguts && git init -b main
find .git/objects -type f            # empty
echo hello > a.txt && git add a.txt
find .git/objects -type f            # one file appeared. Just from `add`, before any commit.
git cat-file -t ce013625030ba8dba906f756967f9e9ca394464a
git commit -m first
find .git/objects -type f | sort     # how many now? which is which?
git cat-file -p HEAD                 # the commit: a tree, an author, a message
git cat-file -p 'HEAD^{tree}'        # the tree: mode, type, hash, name
cat .git/HEAD .git/refs/heads/main   # HEAD names a branch; the branch names a commit

import sys, zlib, hashlib, pathlib

def read_object(repo: pathlib.Path, sha: str) -> tuple[str, bytes]:
    """Return (type, body) for an object, e.g. ('blob', b'hello\n')."""
    # objects live at .git/objects/<first 2 hex chars>/<remaining 38>
    # the file is zlib-compressed:  b"<type> <size>\x00<body>"
    ...

def hash_blob(data: bytes) -> str:
    """What git would name a file with these bytes."""
    # sha1 over the header + body, NOT over the body alone
    ...

// astro.config.mjs
export default defineConfig({
  site: 'https://<you>.github.io',
  base: '/workbench',
  integrations: [mdx()],
  // ...markdown config unchanged
});

# usage: python3 scripts/check_site.py dist /workbench
import sys, re, pathlib
from html.parser import HTMLParser
from urllib.parse import urlsplit, unquote

class LinkCollector(HTMLParser):
    """Collect every href/src on a page, and every id (for #fragment links)."""
    def handle_starttag(self, tag, attrs): ...

def resolve(page: pathlib.Path, url: str, dist: pathlib.Path, base: str) -> pathlib.Path | None:
    """Map a link to the file it should hit in dist/, or None for external links."""
    # skip http(s)://, mailto:, and pure '#...' links
    # '/workbench/lessons/x/' -> dist/lessons/x/index.html
    # a root-relative link that does NOT start with base is broken by definition
    # relative links ('../x/') resolve against the page's own folder
    ...

def main():
    # 1. every slug in src/data/curriculum.ts has dist/lessons/<slug>/index.html
    # 2. every internal link and #fragment on every page points at something that exists
    # print each failure as: BROKEN <page> <link>, then a one-line summary, exit 1 if any
    ...

name: site

on:
  push:
    branches: [main]
  pull_request:          # build and test PRs, but never deploy them
  workflow_dispatch:     # a "Run workflow" button, used for rollbacks
    inputs:
      ref:
        description: 'Commit SHA to deploy (blank = latest main)'
        required: false
        default: ''

permissions:
  contents: read
  pages: write           # allowed to publish to Pages
  id-token: write        # proves to Pages that this run is who it says it is

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          ref: ${{ inputs.ref }}          # empty on push/PR, so it uses the pushed commit
      - uses: actions/setup-node@v4
        with:
          node-version: 22                # astro 7 needs >= 22.22; match your local major if you can
          cache: npm
      - run: npm ci
      - run: npx astro build
      - run: python3 scripts/check_site.py dist /workbench
      - uses: actions/upload-pages-artifact@v3
        with:
          path: dist

  deploy:
    if: github.event_name != 'pull_request'
    needs: build                          # no green build, no deploy
    runs-on: ubuntu-latest
    concurrency:                          # one deploy at a time; let a running one finish
      group: pages
      cancel-in-progress: false
    environment:
      name: github-pages
      url: ${{ steps.deployment.outputs.page_url }}
    steps:
      - id: deployment
        uses: actions/deploy-pages@v4

  deploy-home:
    if: github.event_name != 'pull_request'
    needs: build
    runs-on: ubuntu-latest
    steps:
      - uses: actions/download-artifact@v4   # or rebuild; better: upload dist/ as a normal artifact in build
      - uses: tailscale/github-action@v3     # check its README: it uses an OAuth client + a tag like tag:ci
        with:
          oauth-client-id: ${{ secrets.TS_OAUTH_CLIENT_ID }}
          oauth-secret: ${{ secrets.TS_OAUTH_SECRET }}
          tags: tag:ci
      - run: |
          # copy into a new folder named by commit, then flip a symlink
          # releases/<sha>/  and  current -> releases/<sha>
          ...

Measure

Fill this in from the Actions UI and your stopwatch.

The CI rows are rough expectations from GitHub-hosted runners, not measurements; they vary by day and queue. What the numbers should tell you:

Most of your pipeline’s time isn’t your build. Starting a runner, checking out, installing Node and dependencies will dwarf a one-second Astro build. That’s normal, and it’s why caching dependencies matters far more than optimising the build itself for small projects. For a big service the build and tests dominate, and Fowler’s ten-minute rule starts to bite.

Feedback in about a minute is the whole point. Compare the broken-link row to the opening scene: three weeks to find out a link-shaped mistake (a renamed function, a missing file) versus one minute, on a machine that isn’t yours, before anyone else sees it. That ratio is the case for CI in one number.

The two rollbacks are close here, and that’s misleading. On this site both take a minute or two, because the build is trivial. Multiply the build by twenty (a service with real tests, or a model with an eval gate that takes 20 minutes) and redeploying an existing artifact becomes the only rollback fast enough to use under pressure. Notice what was fast in both cases: no human had to build anything by hand. That’s what Knight Capital didn’t have.

The 55 broken links should be the most interesting row. They were invisible in dev, invisible in astro build (which passed), and only visible to a test that checked the site the way it will actually be served. “The build passes” and “the thing works” are different claims; good pipelines test the second.

Break It

1. Make it work on your machine only. In a lesson, import a figure with the wrong case: figures/pipelineintegration.astro instead of PipelineIntegration.astro. Build locally. On this Mac it built without a warning (I checked: macOS’s default filesystem ignores case). Push it. The Linux runner’s filesystem is case-sensitive, so the build should fail there (check it yourself). This is the cleanest demonstration of why the clean machine exists: your laptop isn’t wrong, it’s just not the machine your users get.

2. Delete the lockfile. Remove package-lock.json and push. npm ci refuses to run without one. Now imagine CI used npm install instead: it would happily resolve the newest versions that match ^7.4.0, and one day a new minor release of a dependency changes the build output, with no commit in your repo to blame. Lockfiles turn “same code” into “same bytes”. Restore it.

3. Add a flaky test. Add a check to check_site.py that fails at random 20% of the time (random.random() < 0.2). Push a few times. Notice how quickly you stop reading red builds and start clicking “Re-run”. That’s the real cost of flakiness: a check that is sometimes wrong for no reason trains everyone to ignore it, and then it misses the real failure. In ML this is the eval gate with a threshold inside the noise. Remove it, and promise yourself to quarantine flaky tests rather than live with them.

4. Roll back something the pipeline doesn’t own. Change a Pages setting by hand (or, in a real service, run a database migration that drops a column) and then “roll back” by redeploying the previous commit. Nothing changes, because that state was never in the repo. Rollback is only as complete as the part of your system that’s described in version control, which is the whole argument for config-as-code and backward-compatible migrations.

Explain It Back

Answer out loud, in plain words, as if in an interview. Then check.

Go Deeper

• Martin Fowler, “Continuous Integration” (2000, revised 2006 and again since). The definition everyone means, with the reasoning behind each practice. Read it once now and once after a bad merge.
• Jez Humble and David Farley, Continuous Delivery (Addison-Wesley, 2010). The book that named the deployment pipeline. Chapters on the pipeline, artifact promotion and environments are the ones to read.
• Scott Chacon and Ben Straub, Pro Git, chapter 10 “Git Internals” (free at git-scm.com/book). Builds blobs, trees and commits by hand with plumbing commands. The perfect companion to Task 1.
• Nicole Forsgren, Jez Humble and Gene Kim, Accelerate (IT Revolution, 2018). The research behind “deploying more often goes with fewer failures”, from years of State of DevOps survey data. Where the four DORA metrics come from.
• Rachel Potvin and Josh Levenberg, “Why Google Stores Billions of Lines of Code in a Single Repository”, Communications of the ACM, 2016. Trunk-based development at a scale that should be impossible, and what tooling it took.
• D. Sculley et al., “Hidden Technical Debt in Machine Learning Systems”, NeurIPS 2015, and Eric Breck et al., “The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction” (Google, 2017). The first explains why ML systems rot in ways code tests miss; the second is a concrete checklist of tests for data, models and infrastructure.
• Steve McConnell, “Daily Build and Smoke Test”, IEEE Software, 1996. Short, and a time capsule: CI before it had a name.
• Astro docs, “Deploy your Astro Site to GitHub Pages”, and GitHub docs on Pages with custom workflows. Check here for the current action versions and the site/base details.

Connections

This builds on Containers vs VMs: a CI runner is a clean machine, and a container image is the way to make “clean and identical” cheap, both for CI and for the artifact you deploy. The rollback drill and the “what changed?” question lead straight into Logs, metrics and traces, because a pipeline tells you a change is safe to ship and observability tells you whether it actually was.

On the ML side, the eval gate is only as good as its threshold, which is Is that improvement real?, and Why is the eval separate? explains why the frozen eval set has to stay frozen. The tunnel for the optional home-server deploy is in Your home server, explained.