The symptom was not the cause

My self-hosted Gitea instance started returning 502 Bad Gateway. Nginx was still answering requests, so the error only told me that its upstream service was unavailable. Before changing the reverse proxy, I checked the host:

df -h

The root filesystem was completely full:

Filesystem      Size  Used Avail Use% Mounted on
/dev/root         50G   48G     0 100% /

A full filesystem can stop a database from writing, prevent applications from creating temporary files, and even make configuration files impossible to save. The 502 was therefore a downstream symptom of a storage incident.

Narrowing the search

I worked from the broadest directory down instead of guessing which Docker object to delete:

sudo du -xhd1 / 2>/dev/null | sort -h
sudo du -xhd1 /home 2>/dev/null | sort -h
sudo du -xhd1 /home/deploy 2>/dev/null | sort -h
sudo du -xhd1 /home/deploy/docker 2>/dev/null | sort -h

-x keeps du on one filesystem, which prevented the attached backup volume from being counted as part of the root disk. The paths shown here are generic; I followed the largest real directory reported at each level.

The largest branches were /var and /home. Under /home, almost all of the space led to Gitea:

24G  /home/deploy/docker/gitea

The Gitea data then split into roughly 5 GB of Git repositories and 19 GB under its application data:

5.0G  .../gitea/git
19G   .../gitea/gitea

One final pass revealed the actual hotspot:

19G  .../gitea/gitea/repo-archive

Finding the files

The archive directory contained many files named like:

repo-archive/tmp/upload-<random-id>

Several individual files were between hundreds of megabytes and 2.5 GB. They had been left behind by abusive repository upload activity.

This distinction mattered. The files were under an archive upload temporary directory, not under Gitea's bare repository storage and not inside MariaDB. Deleting arbitrary directories from either of those locations could leave Gitea's database and filesystem out of sync.

I used a read-only inventory to inspect the largest regular files:

sudo find /home/deploy/docker/gitea \
  -xdev -type f \
  -printf '%s %TY-%Tm-%Td %TH:%TM %p\n' |
sort -nr |
head -50

I also checked for deleted files still held open by a process:

sudo lsof +L1

Deleting a pathname does not release its blocks until the process closes the open file descriptor. If df and du disagree substantially, this check can explain why.

Before cleanup, I:

  1. Confirmed the exact directory consuming the space.
  2. Listed file sizes and modification times.
  3. Stopped or stabilized the application where appropriate.
  4. Preserved database and configuration backups on a separate attached volume.
  5. Limited deletion to abandoned temporary upload files.

I intentionally do not publish a wildcard deletion command. A path copied from another installation may point at live data, and a variable or glob mistake is especially dangerous while operating as root. Cleanup should use an explicit, reviewed file list and application-specific guidance.

After cleanup, I checked both blocks and inodes:

df -h /
df -i /

I then restarted the affected services and inspected their logs:

docker compose ps
docker compose logs --tail 100 mariadb
docker compose logs --tail 100 server

Why the investigation worked

The useful pattern was:

external symptom
  -> host resource
  -> largest top-level directory
  -> largest application directory
  -> individual files

Starting with docker system prune would have been tempting, but it would not have explained the incident and might not have touched bind-mounted Gitea data at all. Measuring first made the cleanup both safer and more effective.

docker system df is still a useful read-only measurement for images, layers, containers, and Docker-managed volumes:

docker system df

I use it to compare Docker-managed storage with bind-mounted application data, not as justification to run a broad prune command.

Preventing a repeat

The long-term fixes are more important than freeing space once:

  • Disable public Gitea registration unless it is intentionally needed.
  • Set upload and repository size limits.
  • Monitor disk usage and alert well before 100%.
  • Keep backups on a different filesystem.
  • Review temporary application directories periodically.
  • Pin and maintain application versions.
  • Investigate unexpected accounts and repositories as a security event.

The main lesson was simple: a 502 does not necessarily mean the proxy is broken. When several services behave strangely at once, verify the host's basic resources before debugging individual applications.

The follow-up account and repository cleanup is covered in Cleaning and securing a spam-abused Gitea instance, rather than repeated here.