The problem with manually managed infrastructure

My Docker Compose file and Nginx virtual hosts originally lived only on the server. That worked, but it left several operational problems:

The active configuration had no useful review history.
A typo could reach production before any syntax check.
Old backup files accumulated beside the live configuration.
It was difficult to distinguish configuration from application state.
Reproducing the server depended on notes and memory.

I moved the non-secret configuration into a private Gitea repository and made Git the source of truth. A pull request now validates a proposed change, while a successful merge into protected main deploys that exact commit.

This pipeline manages service definitions and Nginx configuration. The separate Hugo deployment pipeline still owns the blog build and publication process.

Deciding what belongs in Git

The infrastructure repository contains only reproducible configuration:

compose.yaml
nginx/
  conf.d/
scripts/
  validate.sh
  deploy.sh
  health-check.sh
.gitea/
  workflows/

It deliberately excludes:

Database files and Docker volumes
Gitea repositories, attachments, and application state
TLS certificates and ACME account data
Passwords, tokens, private keys, and runner credentials
Generated site output and backups

The Compose file refers to secret files stored outside the checkout:

services:
  database:
    image: mariadb:10.10
    environment:
      MYSQL_ROOT_PASSWORD_FILE: /run/secrets/database_root_password
      MYSQL_PASSWORD_FILE: /run/secrets/application_database_password
    secrets:
      - database_root_password
      - application_database_password

secrets:
  database_root_password:
    file: /opt/example-secrets/database-root-password
  application_database_password:
    file: /opt/example-secrets/application-database-password

The files are readable only by the deployment account. The repository contains their paths, not their values.

Preserving the existing database volume

Moving a Compose file does not move its data. My MariaDB data already lived in a Docker volume created by the old project. I inspected the running container's mounts and declared that existing volume explicitly:

services:
  database:
    volumes:
      - database_data:/var/lib/mysql

volumes:
  database_data:
    external: true
    name: <existing-docker-volume-name>

Marking it external prevents Compose from silently creating a new empty volume under the new checkout. Before recreating MariaDB, I made a native database dump, verified that it was non-empty, and confirmed that the secret files could authenticate against the running database.

After recreation, I checked the exact /var/lib/mysql mount and tested both the administrative and application accounts. I did not use docker compose down -v or docker volume prune; either command can destroy state that the deployment is meant to preserve.

Separating validation from deployment

The Gitea workflow has two jobs with different trust boundaries:

Pull request
  -> containerized validation
  -> deployment skipped

Push to protected main
  -> containerized validation
  -> repository-scoped host deployment
  -> production health checks

The validation job uses an isolated runner label:

validate:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4
    - name: Validate Compose
      run: ./scripts/validate.sh

The validation script fails on an invalid Compose model or whitespace errors:

#!/usr/bin/env bash

set -euo pipefail

repository_root="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$repository_root"

docker compose -f compose.yaml config --quiet
git diff --check

Pull requests never receive the production runner label. They can prove that configuration parses, but they cannot change the host.

Using a dedicated host-runner label

Deployment needs controlled access to the existing checkout, Docker daemon, secret files, and Nginx mounts. I registered a second runner specifically to the private infrastructure repository:

runner:
  capacity: 1
  labels:
    - "ubuntu-latest:docker://docker.gitea.com/runner-images:ubuntu-latest"
    - "infra-deploy:host"

The container label handles validation. infra-deploy:host runs the deployment job directly as a dedicated operating-system user.

This is a powerful permission boundary. A user who can merge arbitrary workflow code can execute commands with the runner account's access, and Docker access is effectively host-level access. I therefore use:

A private repository
A repository-scoped runner
Disabled public registration
Protected main
Pull-request validation
A deployment label not shared with unrelated repositories
Runner credentials stored outside Git with restrictive permissions

Deploying only the validated commit

The deployment job runs only for a push to main:

deploy:
  if: gitea.event_name == 'push' && gitea.ref == 'refs/heads/main'
  needs:
    - validate
  runs-on: infra-deploy
  concurrency:
    group: production-infrastructure
    cancel-in-progress: false

The host checks that its production checkout is clean, fetches main, and compares origin/main with the commit validated by the workflow. It refuses to continue if the branch moved in the meantime:

set -euo pipefail

DEPLOY_DIR=/opt/example-infrastructure
EXPECTED_COMMIT="<validated-commit-sha>"

test -z "$(git -C "$DEPLOY_DIR" status --porcelain)"
git -C "$DEPLOY_DIR" fetch --prune origin main

REMOTE_COMMIT="$(git -C "$DEPLOY_DIR" rev-parse origin/main)"
test "$REMOTE_COMMIT" = "$EXPECTED_COMMIT"

git -C "$DEPLOY_DIR" merge --ff-only "$EXPECTED_COMMIT"
"$DEPLOY_DIR/scripts/deploy.sh" "$EXPECTED_COMMIT"

The real workflow obtains the expected SHA from the Gitea event rather than hard-coding it.

The deployment script also takes an advisory lock, repeats the clean-checkout and commit checks, and validates Nginx using the production mounts before reconciling services:

docker compose -f compose.yaml \
  run --rm --no-deps nginx nginx -t

docker compose -f compose.yaml \
  up -d --no-deps database application nginx

It intentionally omits one-shot or separately managed services. It also avoids down, --remove-orphans, and volume deletion.

Reloading bind-mounted Nginx configuration

Changing a bind-mounted Nginx file does not necessarily change the Compose service definition, so docker compose up may leave the existing container running without reloading its configuration.

After reconciliation, the pipeline validates the running container and asks Nginx to reload gracefully:

docker compose -f compose.yaml exec nginx nginx -t
docker compose -f compose.yaml exec nginx nginx -s reload

The first command prevents a bad configuration from replacing the active one. The second applies the new virtual hosts without an unnecessary hard stop. The role of the Nginx document root and read-only blog mount is covered in Building and serving a production Hugo site with Docker and Nginx.

The host checkout failure

My first deployment job used:

- uses: actions/checkout@v4

That worked in the containerized validation job but failed on the host runner:

Cannot find: node in PATH

The checkout action is implemented with JavaScript and requires a Node.js runtime. Installing Node on the server was possible, but unnecessary. The host already had a production checkout and Git, so I removed the checkout action from the deployment job and used guarded git fetch and git merge --ff-only commands instead.

The failure happened before any Compose command ran, which also demonstrated the value of keeping checkout, validation, and deployment as distinct steps.

Health checks and completion criteria

Starting containers is not the same as completing a deployment. The final script waits for authenticated database readiness and checks the public HTTPS endpoints:

docker exec example-database sh -ec '
  mariadb-admin ping \
    --host=127.0.0.1 \
    --user=root \
    --password="$(cat /run/secrets/database_root_password)" \
    --silent
'

curl --fail --location --silent --show-error \
  --output /dev/null \
  https://git.example.com/

curl --fail --location --silent --show-error \
  --output /dev/null \
  https://example.com/

The password is read inside the container and is not printed in workflow logs. Each check retries for a bounded period so a normal service startup does not look like an immediate deployment failure.

A successful run now means:

The proposed configuration passed pull-request validation.
Protected main contains the reviewed commit.
The server deployed that exact commit with a fast-forward update.
Compose and Nginx syntax checks passed.
MariaDB accepted an authenticated health check.
Gitea and the public site returned successful HTTPS responses.

The working practice

For every infrastructure change, I update local main, create a short-lived branch, make one logical change, and open a pull request. A one-commit branch is merged with fast-forward only; several temporary commits representing one change can be squashed. The feature branch is deleted afterward.

I do not edit the production checkout manually. If emergency intervention is unavoidable, I stop automated deployment, document the change, reproduce it in Git, and reconcile the checkout before allowing the pipeline to run again.

The important outcome is not merely automatic deployment. It is a narrow, auditable path from reviewed configuration to a verified production state.

The problem with manually managed infrastructure#

Deciding what belongs in Git#

Preserving the existing database volume#

Separating validation from deployment#

Using a dedicated host-runner label#

Deploying only the validated commit#

Reloading bind-mounted Nginx configuration#

The host checkout failure#

Health checks and completion criteria#

The working practice#