The symptom was not the cause

My self-hosted Gitea instance started returning 502 Bad Gateway. Nginx was still answering requests, so the error only told me that its upstream service was unavailable. Before changing the reverse proxy, I checked the host:

df -h

The root filesystem was completely full:

Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        50G   48G     0 100% /

A full filesystem can stop a database from writing, prevent applications from creating temporary files, and even make configuration files impossible to save. The 502 was therefore a downstream symptom of a storage incident.

Narrowing the search

I worked from the broadest directory down instead of guessing which Docker object to delete:

sudo du -xhd1 / 2>/dev/null | sort -h
sudo du -xhd1 /home 2>/dev/null | sort -h
sudo du -xhd1 /home/minhdb 2>/dev/null | sort -h
sudo du -xhd1 /home/minhdb/docker 2>/dev/null | sort -h

The largest branches were /var and /home. Under /home, almost all of the space led to Gitea:

24G  /home/minhdb/docker/gitea

The Gitea data then split into roughly 5 GB of Git repositories and 19 GB under its application data:

5.0G  .../gitea/git
19G   .../gitea/gitea

One final pass revealed the actual hotspot:

19G  .../gitea/gitea/repo-archive

Finding the files

The archive directory contained many files named like:

repo-archive/tmp/upload-4078008142
repo-archive/tmp/upload-4014387447

Several individual files were between hundreds of megabytes and 2.5 GB. They had been left behind by abusive repository upload activity.

This distinction mattered. The files were under an archive upload temporary directory, not under Gitea’s bare repository storage and not inside MariaDB. Deleting arbitrary directories from either of those locations could leave Gitea’s database and filesystem out of sync.

Before cleanup, I:

  1. Confirmed the exact directory consuming the space.
  2. Listed file sizes and modification times.
  3. Stopped or stabilized the application where appropriate.
  4. Preserved database and configuration backups on a separate attached volume.
  5. Limited deletion to abandoned temporary upload files.

After cleanup, I checked both blocks and inodes:

df -h /
df -i /

I then restarted the affected services and inspected their logs:

docker compose ps
docker compose logs --tail 100 mariadb
docker compose logs --tail 100 server

Why the investigation worked

The useful pattern was:

external symptom
  -> host resource
  -> largest top-level directory
  -> largest application directory
  -> individual files

Starting with docker system prune would have been tempting, but it would not have explained the incident and might not have touched bind-mounted Gitea data at all. Measuring first made the cleanup both safer and more effective.

Preventing a repeat

The long-term fixes are more important than freeing space once:

  • Disable public Gitea registration unless it is intentionally needed.
  • Set upload and repository size limits.
  • Monitor disk usage and alert well before 100%.
  • Keep backups on a different filesystem.
  • Review temporary application directories periodically.
  • Pin and maintain application versions.
  • Investigate unexpected accounts and repositories as a security event.

The main lesson was simple: a 502 does not necessarily mean the proxy is broken. When several services behave strangely at once, verify the host’s basic resources before debugging individual applications.