Add borg backup scope and database dump workflow

ops/borg-ui/BACKUP_SCOPE.md

@@ -0,0 +1,98 @@
+# Borg Backup Scope for KalliLabcore
+
+This file defines the target state for replacing Backrest with Borg in this homelab.
+
+## Goal
+
+Use Borg as the single backup system for:
+
+- critical file-backed application data
+- secrets, keys, and reverse-proxy state
+- database dumps generated before each Borg backup
+
+Do not back up raw live database storage directories as the primary recovery artifact.
+
+## Strategy
+
+1. A pre-backup dump script runs on the host and writes fresh dumps to `/mnt/user/appdata/borg-ui/dumps/latest`.
+2. Borg backs up `/local/borg-dumps` plus the critical mounted paths below.
+3. Borg retention handles history; the dump directory itself keeps only the latest artifacts.
+
+## Service Inventory
+
+| Service | Recovery Method | What Borg Should Capture |
+| --- | --- | --- |
+| Vaultwarden | file data | `/local/appdata/vaultwarden` |
+| Paperless | DB dump + file data | `/local/borg-dumps`, `/local/appdata/paperless-ngx/data`, `/local/paperless/media`, `/local/paperless/export`, `/local/paperless/consume` |
+| Immich | DB dump + file data | `/local/borg-dumps`, `/local/immich/upload`, `/local/immich/external` |
+| Gitea | file data (SQLite inside `/data`) | `/local/gitea/data` |
+| Mealie | DB dump + file data | `/local/borg-dumps`, `/local/appdata/mealie/data` |
+| Firefly | MariaDB dump + uploads | `/local/borg-dumps`, `/local/appdata/firefly/upload` |
+| Mail-archiver | shared Postgres dump + data protection keys | `/local/borg-dumps`, `/local/appdata/mailarchiver/data-protection-keys` |
+| Authelia | shared Postgres dump + config + secrets | `/local/borg-dumps`, `/local/appdata/authelia/config`, `/local/secrets` |
+| Traefik | file data | `/local/appdata/traefik` |
+| Homepage | file data | `/local/appdata/homepage` |
+| ntfy | file data | `/local/appdata/ntfy` |
+| Paperless-GPT | file data | `/local/appdata/paperless-gpt` |
+| Tailscale | file data | `/local/appdata/tailscale` |
+| AdGuard | config only | `/local/appdata/adguard/conf` |
+| Borg UI | self-backup | `/local/appdata/borg-ui/data` |
+| Komodo | config/cache only, optional | `/local/appdata/komodo/periphery`, `/local/appdata/komodo/core` |
+
+## Database Dumps Required
+
+### Shared PostgreSQL (`postgresql17`)
+
+- `mailarchiver`
+- `paperless`
+- `semaphore`
+- `authelia`
+
+### Dedicated PostgreSQL
+
+- `mealie`
+- `immich`
+
+### Other Databases
+
+- Firefly MariaDB
+- Komodo MongoDB
+
+## Explicitly Not Backed Up as Raw Live DB Files
+
+- `/mnt/user/appdata/postgresql17`
+- `/mnt/user/appdata/mealie/postgres`
+- `/mnt/user/appdata/immich_postgres`
+- `/mnt/user/appdata/firefly/db`
+- `/mnt/user/appdata/komodo/mongo`
+- `/mnt/user/appdata/redis`
+- `/mnt/user/appdata/scrutiny/influxdb`
+
+## Low-Priority / Rebuildable
+
+These are not part of the first-class Borg scope:
+
+- Plex metadata and cache
+- AdGuard query log
+- code-server extensions cache
+- uptime-kuma
+- scrutiny metrics history
+- dozzle, glances, speedtest
+- filebrowser app state
+- portainer
+
+## Special Case
+
+`ops/Semaphore` currently stores app state in named Docker volumes, not host bind mounts. That state is not covered by the current Borg mounts and should be migrated to bind mounts before claiming full Borg coverage for Semaphore.
+
+## Suggested Retention
+
+- daily: 7
+- weekly: 4
+- monthly: 6
+
+## Repository Recommendation
+
+Recommended primary Borg repository: `critical-infra`
+
+Primary sources are listed in `all-important-sources.txt`.

ops/borg-ui/all-important-sources.txt

@@ -0,0 +1,23 @@
+/local/borg-dumps
+/local/appdata/vaultwarden
+/local/appdata/paperless-ngx/data
+/local/paperless/media
+/local/paperless/export
+/local/paperless/consume
+/local/immich/upload
+/local/immich/external
+/local/gitea/data
+/local/appdata/mealie/data
+/local/appdata/firefly/upload
+/local/appdata/mailarchiver/data-protection-keys
+/local/secrets
+/local/appdata/authelia/config
+/local/appdata/traefik
+/local/appdata/homepage
+/local/appdata/ntfy
+/local/appdata/paperless-gpt
+/local/appdata/tailscale
+/local/appdata/adguard/conf
+/local/appdata/borg-ui/data
+/local/appdata/komodo/periphery
+/local/appdata/komodo/core

ops/borg-ui/docker-compose.yml

@@ -16,6 +16,7 @@ services:
       - /mnt/user/appdata/borg-ui/cache:/home/borg/.cache/borg
       - /mnt/user/appdata:/local/appdata:ro
       - /mnt/user/appdata/secrets:/local/secrets:ro
+      - /mnt/user/appdata/borg-ui/dumps:/local/borg-dumps:ro
       - /mnt/user/documents/scans_inbox:/local/paperless/consume:ro
       - /mnt/user/documents/paperless:/local/paperless/media:ro
       - /mnt/user/documents/paperless/export:/local/paperless/export:ro

ops/borg-ui/scripts/README.md

@@ -0,0 +1,21 @@
+# Borg dump scripts
+
+These scripts are intended to run on the Unraid host before a Borg backup starts.
+
+## Current script
+
+- `pre-backup-dumps.sh`
+
+## Output
+
+Fresh dump artifacts are written to:
+
+- `/mnt/user/appdata/borg-ui/dumps/latest`
+
+Borg UI should include `/local/borg-dumps` as a backup source.
+
+## Notes
+
+- The script is written for host execution where `docker` is available.
+- It does not assume Backrest.
+- It keeps only the latest dump set because Borg itself provides history.

ops/borg-ui/scripts/pre-backup-dumps.sh

@@ -0,0 +1,170 @@
+#!/bin/sh
+set -eu
+
+# Run this on the Unraid host before Borg starts.
+# It refreshes the latest database dumps in a stable directory so Borg can
+# version the dump artifacts instead of raw live database files.
+
+DUMP_ROOT="${DUMP_ROOT:-/mnt/user/appdata/borg-ui/dumps}"
+LATEST_DIR="$DUMP_ROOT/latest"
+TMP_DIR="$DUMP_ROOT/.tmp"
+
+log() {
+  printf '%s %s\n' "[borg-dumps]" "$*"
+}
+
+warn() {
+  printf '%s %s\n' "[borg-dumps][warn]" "$*" >&2
+}
+
+need_cmd() {
+  if ! command -v "$1" >/dev/null 2>&1; then
+    warn "Required command missing: $1"
+    exit 1
+  fi
+}
+
+need_container() {
+  docker inspect "$1" >/dev/null 2>&1
+}
+
+ensure_dirs() {
+  mkdir -p "$LATEST_DIR" "$TMP_DIR"
+}
+
+atomic_write() {
+  target="$1"
+  tmp="$2"
+  mkdir -p "$(dirname "$target")"
+  mv "$tmp" "$target"
+}
+
+dump_pg_db() {
+  container="$1"
+  password="$2"
+  user="$3"
+  db="$4"
+  output="$5"
+
+  tmp="$TMP_DIR/$(basename "$output").tmp"
+  log "Dumping PostgreSQL database '$db' from $container"
+  docker exec -e "PGPASSWORD=$password" "$container" \
+    pg_dump -U "$user" -d "$db" -Fc > "$tmp"
+  atomic_write "$output" "$tmp"
+}
+
+dump_pg_globals() {
+  container="$1"
+  password="$2"
+  user="$3"
+  output="$4"
+
+  tmp="$TMP_DIR/$(basename "$output").tmp"
+  log "Dumping PostgreSQL globals from $container"
+  docker exec -e "PGPASSWORD=$password" "$container" \
+    pg_dumpall -U "$user" --globals-only > "$tmp"
+  atomic_write "$output" "$tmp"
+}
+
+dump_optional_pg_db() {
+  container="$1"
+  password="$2"
+  user="$3"
+  db="$4"
+  output="$5"
+
+  if docker exec -e "PGPASSWORD=$password" "$container" \
+    psql -U "$user" -d postgres -tAc "SELECT 1 FROM pg_database WHERE datname = '$db'" \
+    | grep -q 1; then
+    dump_pg_db "$container" "$password" "$user" "$db" "$output"
+  else
+    warn "Skipping missing PostgreSQL database '$db' in $container"
+  fi
+}
+
+dump_mysql_container() {
+  container="$1"
+  output="$2"
+
+  if ! need_container "$container"; then
+    warn "Skipping missing container: $container"
+    return 0
+  fi
+
+  info="$(docker exec "$container" sh -lc 'printf "%s|%s|%s" "${MARIADB_DATABASE:-${MYSQL_DATABASE:-}}" "${MARIADB_USER:-${MYSQL_USER:-root}}" "${MARIADB_PASSWORD:-${MYSQL_PASSWORD:-}}"' || true)"
+  db="$(printf '%s' "$info" | cut -d'|' -f1)"
+  user="$(printf '%s' "$info" | cut -d'|' -f2)"
+  password="$(printf '%s' "$info" | cut -d'|' -f3)"
+
+  if [ -z "$db" ] || [ -z "$password" ]; then
+    warn "Skipping MySQL/MariaDB dump for $container because DB credentials were not discoverable"
+    return 0
+  fi
+
+  tmp="$TMP_DIR/$(basename "$output").tmp"
+  log "Dumping MariaDB/MySQL database '$db' from $container"
+  docker exec "$container" sh -lc "mysqldump --single-transaction --quick -u\"$user\" -p\"$password\" \"$db\"" > "$tmp"
+  atomic_write "$output" "$tmp"
+}
+
+dump_mongo_container() {
+  container="$1"
+  output="$2"
+
+  if ! need_container "$container"; then
+    warn "Skipping missing container: $container"
+    return 0
+  fi
+
+  if ! docker exec "$container" sh -lc 'command -v mongodump >/dev/null 2>&1'; then
+    warn "Skipping Mongo dump for $container because mongodump is not available in the container image"
+    return 0
+  fi
+
+  tmp="$TMP_DIR/$(basename "$output").tmp"
+  log "Dumping MongoDB archive from $container"
+  docker exec "$container" sh -lc 'mongodump --archive --gzip --username "$MONGO_INITDB_ROOT_USERNAME" --password "$(cat /run/secrets/mongo_password)" --authenticationDatabase admin' > "$tmp"
+  atomic_write "$output" "$tmp"
+}
+
+main() {
+  need_cmd docker
+  ensure_dirs
+
+  # Shared PostgreSQL 17
+  if need_container "postgresql17"; then
+    shared_pg_password="$(cat /mnt/user/appdata/secrets/postgres_password.txt)"
+    dump_pg_globals "postgresql17" "$shared_pg_password" "mailarchiver" "$LATEST_DIR/postgresql17-globals.sql"
+    dump_pg_db "postgresql17" "$shared_pg_password" "mailarchiver" "mailarchiver" "$LATEST_DIR/postgresql17-mailarchiver.dump"
+    dump_pg_db "postgresql17" "$shared_pg_password" "mailarchiver" "paperless" "$LATEST_DIR/postgresql17-paperless.dump"
+    dump_optional_pg_db "postgresql17" "$shared_pg_password" "mailarchiver" "semaphore" "$LATEST_DIR/postgresql17-semaphore.dump"
+    dump_optional_pg_db "postgresql17" "$shared_pg_password" "mailarchiver" "authelia" "$LATEST_DIR/postgresql17-authelia.dump"
+  else
+    warn "Skipping shared PostgreSQL dumps because container 'postgresql17' is missing"
+  fi
+
+  # Dedicated PostgreSQL databases
+  if need_container "mealie-postgres"; then
+    mealie_password="$(cat /mnt/user/appdata/secrets/mealie_postgres_password.txt)"
+    dump_pg_db "mealie-postgres" "$mealie_password" "mealie" "mealie" "$LATEST_DIR/mealie.dump"
+  else
+    warn "Skipping missing container: mealie-postgres"
+  fi
+
+  if need_container "immich_postgres"; then
+    immich_password="$(cat /mnt/user/appdata/secrets/immich_postgres_password.txt)"
+    dump_pg_db "immich_postgres" "$immich_password" "immich" "immich" "$LATEST_DIR/immich.dump"
+  else
+    warn "Skipping missing container: immich_postgres"
+  fi
+
+  # MariaDB / MySQL
+  dump_mysql_container "firefly-db" "$LATEST_DIR/firefly.sql"
+
+  # MongoDB
+  dump_mongo_container "komodo-mongo" "$LATEST_DIR/komodo-mongo.archive.gz"
+
+  log "Finished refreshing dump set in $LATEST_DIR"
+}
+
+main "$@"