Add host-ready restore automation scripts
This commit is contained in:
+12
-12
@@ -104,16 +104,16 @@ Alle validierten Restore-Tests folgen demselben Muster:
|
||||
|
||||
### V1
|
||||
|
||||
- manuell validierte Restore-Pfade
|
||||
- validierte Bash-Host-Jobs
|
||||
- Host-Job-Definitionen liegen im Repo
|
||||
- Scheduler kann bereits Plan- und Frische-Checks fahren
|
||||
- volle Automatik je Dienst wird danach gezielt nachgezogen
|
||||
- Scheduler kann bereits echte Frische- und Restore-Checks fahren
|
||||
- `ntfy` und Hermes-Auswertung folgen danach
|
||||
|
||||
### V2
|
||||
|
||||
- echte Vollautomatik fuer die drei validierten Dienste
|
||||
- `ntfy` bei Erfolg/Fehler
|
||||
- Hermes liest spaeter Reports und baut Uebersichten
|
||||
- Hermes liest Reports und baut Uebersichten
|
||||
- zusaetzliche Rotation, Sammelreports und weitere Dienste
|
||||
|
||||
---
|
||||
|
||||
@@ -169,25 +169,25 @@ Nur `Container laeuft` reicht nicht.
|
||||
Auf dem Unraid-Host:
|
||||
|
||||
```bash
|
||||
pwsh -File /mnt/user/services/homelab/ops/restore-tests/run-restore-checks.ps1 -Mode freshness
|
||||
bash /mnt/user/services/homelab/ops/restore-tests/run-restore-checks.sh freshness
|
||||
```
|
||||
|
||||
### Vaultwarden Planlauf
|
||||
### Vaultwarden Restore-Check
|
||||
|
||||
```bash
|
||||
pwsh -File /mnt/user/services/homelab/ops/restore-tests/run-restore-checks.ps1 -Mode vaultwarden -WhatIf
|
||||
bash /mnt/user/services/homelab/ops/restore-tests/run-restore-checks.sh vaultwarden
|
||||
```
|
||||
|
||||
### Gitea Planlauf
|
||||
### Gitea Restore-Check
|
||||
|
||||
```bash
|
||||
pwsh -File /mnt/user/services/homelab/ops/restore-tests/run-restore-checks.ps1 -Mode gitea -WhatIf
|
||||
bash /mnt/user/services/homelab/ops/restore-tests/run-restore-checks.sh gitea
|
||||
```
|
||||
|
||||
### Paperless Planlauf
|
||||
### Paperless Restore-Check
|
||||
|
||||
```bash
|
||||
pwsh -File /mnt/user/services/homelab/ops/restore-tests/run-restore-checks.ps1 -Mode paperless -WhatIf
|
||||
bash /mnt/user/services/homelab/ops/restore-tests/run-restore-checks.sh paperless
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
@@ -21,16 +21,22 @@ Ziel:
|
||||
|
||||
- `schedule.md`: Intervalle und Verantwortlichkeiten
|
||||
- `vaultwarden-restore-test.ps1`: erster Mini-Restore-Ablauf
|
||||
- `vaultwarden-restore-test.sh`: hosttauglicher Vaultwarden-Restore-Job
|
||||
- `vaultwarden-plan.md`: konkreter Vaultwarden-Testplan
|
||||
- `vaultwarden-compose.test.yml`: isolierte Testinstanz fuer Vaultwarden
|
||||
- `gitea-restore-test.ps1`: Gitea-Mini-Restore-Ablauf
|
||||
- `gitea-restore-test.sh`: hosttauglicher Gitea-Restore-Job
|
||||
- `gitea-plan.md`: konkreter Gitea-Testplan
|
||||
- `gitea-compose.test.yml`: isolierte Testinstanz fuer Gitea
|
||||
- `paperless-restore-test.ps1`: Paperless-Mini-Restore-Ablauf
|
||||
- `paperless-restore-test.sh`: hosttauglicher Paperless-Restore-Job
|
||||
- `paperless-plan.md`: konkreter Paperless-Testplan
|
||||
- `paperless-compose.test.yml`: isolierte Testinstanz fuer Paperless inkl. Test-Postgres und Test-Redis
|
||||
- `check-restore-freshness.ps1`: woechentlicher Frische-Check fuer Dumps und Reports
|
||||
- `run-restore-checks.ps1`: einfacher Dispatcher fuer Restore-Jobs
|
||||
- `check-restore-freshness.sh`: hosttauglicher Frische-Check
|
||||
- `run-restore-checks.sh`: hosttauglicher Dispatcher
|
||||
- `common.sh`: gemeinsame Host-Helferfunktionen
|
||||
- `automation-plan.md`: Host-Job- und Automatisierungsmodell
|
||||
|
||||
## Automatisierungsmodell
|
||||
@@ -43,7 +49,8 @@ Ziel:
|
||||
|
||||
Wichtig:
|
||||
|
||||
- `check-restore-freshness.ps1` und spaetere automatische Restore-Jobs sind fuer den Unraid-Host gedacht
|
||||
- die Bash-Skripte `*.sh` sind die produktive Host-Variante
|
||||
- `check-restore-freshness.ps1` und die `*.ps1`-Dateien bleiben als lokale Plan-/Hilfsvariante nutzbar
|
||||
- im Windows-Clone fehlen die `/mnt/user/...`-Pfade naturgemaess
|
||||
|
||||
## Validiertes Grundmuster
|
||||
@@ -69,6 +76,7 @@ Aktuell ist das erste validierte Muster vorhanden.
|
||||
- echter Vaultwarden-Restore am 2026-05-07 erfolgreich verifiziert
|
||||
- echter Gitea-Restore am 2026-05-07 erfolgreich verifiziert
|
||||
- echter Paperless-Restore am 2026-05-07 erfolgreich verifiziert
|
||||
- Bash-Dispatcher und Bash-Restore-Jobs am 2026-05-07 erfolgreich hostseitig verifiziert
|
||||
- Restore-Lab und Report-Pfade auf dem Host angelegt
|
||||
- V1-Ablauf weiter ohne `ntfy`, mit Bereinigung nach Erfolg
|
||||
- naechster grosser Kandidat ist ein weiterer datenbankgestuetzter Dienst oder die Automatisierung
|
||||
|
||||
@@ -16,7 +16,7 @@ Die bereits validierten Restore-Tests fuer `vaultwarden`, `gitea` und `paperless
|
||||
|
||||
### Woechentlicher Frische-Check
|
||||
|
||||
- Script: `check-restore-freshness.ps1`
|
||||
- Script: `check-restore-freshness.sh`
|
||||
- Ziel:
|
||||
- Dump-Dateien vorhanden
|
||||
- Dump-Dateien nicht zu alt
|
||||
@@ -26,19 +26,20 @@ Die bereits validierten Restore-Tests fuer `vaultwarden`, `gitea` und `paperless
|
||||
|
||||
### Monatliche / zweimonatliche Restore-Jobs
|
||||
|
||||
- Script-Dispatcher: `run-restore-checks.ps1`
|
||||
- Script-Dispatcher: `run-restore-checks.sh`
|
||||
- Modi:
|
||||
- `freshness`
|
||||
- `vaultwarden`
|
||||
- `gitea`
|
||||
- `paperless`
|
||||
- V1 ruft die existierenden dienstspezifischen Scripts zunaechst im `WhatIf`- oder Plan-Modus auf, bis die Vollautomatisierung je Dienst gezielt nachgezogen wird.
|
||||
- diese Bash-Jobs sind jetzt hostseitig praktisch verifiziert
|
||||
- die `*.ps1`-Dateien bleiben als Plan-/Hilfsvariante fuer die Windows-Arbeitskopie erhalten
|
||||
|
||||
## V2
|
||||
|
||||
- echte Vollautomatisierung pro Dienst
|
||||
- `ntfy` Erfolg/Fehler
|
||||
- optional Hermes-Zusammenfassung ueber vorhandene Reports
|
||||
- spaeter Job-Metadaten, Rotation und Sammel-Reports weiter ausbauen
|
||||
|
||||
## Host-Integration
|
||||
|
||||
|
||||
@@ -0,0 +1,81 @@
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
|
||||
DUMP_ROOT="${DUMP_ROOT:-/mnt/user/backups/borg/dumps/latest}"
|
||||
REPORT_ROOT="${REPORT_ROOT:-/mnt/user/backups/restore-reports}"
|
||||
MAX_DUMP_AGE_HOURS="${MAX_DUMP_AGE_HOURS:-36}"
|
||||
MAX_REPORT_AGE_DAYS="${MAX_REPORT_AGE_DAYS:-45}"
|
||||
|
||||
now_epoch="$(date +%s)"
|
||||
critical=()
|
||||
warnings=()
|
||||
info=()
|
||||
|
||||
check_file_age_hours() {
|
||||
local path="$1"
|
||||
local mtime
|
||||
mtime="$(stat -c %Y "$path")"
|
||||
echo $(( (now_epoch - mtime) / 3600 ))
|
||||
}
|
||||
|
||||
check_file_age_days() {
|
||||
local path="$1"
|
||||
local mtime
|
||||
mtime="$(stat -c %Y "$path")"
|
||||
echo $(( (now_epoch - mtime) / 86400 ))
|
||||
}
|
||||
|
||||
for dump in postgresql17-paperless.dump postgresql17-mailarchiver.dump mealie.dump immich.dump; do
|
||||
path="$DUMP_ROOT/$dump"
|
||||
if [ ! -f "$path" ]; then
|
||||
critical+=("DUMP_MISSING $dump")
|
||||
continue
|
||||
fi
|
||||
age="$(check_file_age_hours "$path")"
|
||||
if [ "$age" -gt "$MAX_DUMP_AGE_HOURS" ]; then
|
||||
warnings+=("DUMP_STALE $dump age=${age}h")
|
||||
else
|
||||
info+=("DUMP_OK $dump age=${age}h")
|
||||
fi
|
||||
done
|
||||
|
||||
for service in vaultwarden gitea paperless; do
|
||||
latest="$(find "$REPORT_ROOT" -maxdepth 1 -type f -name "$service-*.md" | sort | tail -n 1 || true)"
|
||||
if [ -z "$latest" ]; then
|
||||
warnings+=("REPORT_MISSING $service")
|
||||
continue
|
||||
fi
|
||||
age="$(check_file_age_days "$latest")"
|
||||
if [ "$age" -gt "$MAX_REPORT_AGE_DAYS" ]; then
|
||||
warnings+=("REPORT_STALE $service age=${age}d file=$(basename "$latest")")
|
||||
else
|
||||
info+=("REPORT_OK $service age=${age}d file=$(basename "$latest")")
|
||||
fi
|
||||
done
|
||||
|
||||
echo "# Restore Freshness Check"
|
||||
echo
|
||||
echo "Timestamp: $(date '+%F %T')"
|
||||
echo "Critical: ${#critical[@]}"
|
||||
echo "Warnings: ${#warnings[@]}"
|
||||
echo "Info: ${#info[@]}"
|
||||
echo
|
||||
|
||||
if [ "${#critical[@]}" -gt 0 ]; then
|
||||
echo "## Critical"
|
||||
printf -- '- %s\n' "${critical[@]}"
|
||||
echo
|
||||
fi
|
||||
|
||||
if [ "${#warnings[@]}" -gt 0 ]; then
|
||||
echo "## Warnings"
|
||||
printf -- '- %s\n' "${warnings[@]}"
|
||||
echo
|
||||
fi
|
||||
|
||||
if [ "${#info[@]}" -gt 0 ]; then
|
||||
echo "## Info"
|
||||
printf -- '- %s\n' "${info[@]}"
|
||||
fi
|
||||
|
||||
[ "${#critical[@]}" -eq 0 ]
|
||||
@@ -0,0 +1,84 @@
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
|
||||
RESTORE_TESTS_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
BORG_CONTAINER="${BORG_CONTAINER:-borg-ui}"
|
||||
BORG_RESTORE_HOST_ROOT="${BORG_RESTORE_HOST_ROOT:-/mnt/user/appdata/borg-ui/restore}"
|
||||
BORG_PASSPHRASE_FILE_DEFAULT="${BORG_PASSPHRASE_FILE_DEFAULT:-/mnt/user/appdata/secrets/borg_repo_passphrase.txt}"
|
||||
|
||||
require_cmd() {
|
||||
command -v "$1" >/dev/null 2>&1 || {
|
||||
echo "Missing command: $1" >&2
|
||||
exit 1
|
||||
}
|
||||
}
|
||||
|
||||
require_path() {
|
||||
[ -e "$1" ] || {
|
||||
echo "Missing path: $1" >&2
|
||||
exit 1
|
||||
}
|
||||
}
|
||||
|
||||
latest_archive_name() {
|
||||
docker exec "$BORG_CONTAINER" python3 - <<'PY'
|
||||
import sqlite3
|
||||
conn = sqlite3.connect('/data/borg.db')
|
||||
cur = conn.cursor()
|
||||
cur.execute("select archive_name from backup_jobs where status='completed' order by created_at desc limit 1")
|
||||
row = cur.fetchone()
|
||||
if not row:
|
||||
raise SystemExit("No completed borg archive found")
|
||||
print(row[0])
|
||||
PY
|
||||
}
|
||||
|
||||
borg_repo_url() {
|
||||
docker exec "$BORG_CONTAINER" python3 - <<'PY'
|
||||
import sqlite3
|
||||
conn = sqlite3.connect('/data/borg.db')
|
||||
cur = conn.cursor()
|
||||
cur.execute("select path from repositories where path is not null and path != '' order by id asc limit 1")
|
||||
row = cur.fetchone()
|
||||
if not row:
|
||||
raise SystemExit("No borg repository configured")
|
||||
print(row[0])
|
||||
PY
|
||||
}
|
||||
|
||||
borg_extract() {
|
||||
local extract_dir="$1"
|
||||
shift
|
||||
local paths=("$@")
|
||||
docker exec -i "$BORG_CONTAINER" python3 - "$extract_dir" "${paths[@]}" <<'PY'
|
||||
import os, sys, subprocess
|
||||
extract_dir = sys.argv[1]
|
||||
paths = sys.argv[2:]
|
||||
import sqlite3
|
||||
conn = sqlite3.connect('/data/borg.db')
|
||||
cur = conn.cursor()
|
||||
cur.execute("select path from repositories where path is not null and path != '' order by id asc limit 1")
|
||||
repo = cur.fetchone()[0]
|
||||
cur.execute("select archive_name from backup_jobs where status='completed' order by created_at desc limit 1")
|
||||
archive = cur.fetchone()[0]
|
||||
with open('/local/secrets/borg_repo_passphrase.txt', 'r', encoding='utf-8') as f:
|
||||
os.environ['BORG_PASSPHRASE'] = f.read().strip()
|
||||
os.makedirs(extract_dir, exist_ok=True)
|
||||
os.chdir(extract_dir)
|
||||
subprocess.run(['borg', 'extract', f'{repo}::{archive}', *paths], check=True)
|
||||
PY
|
||||
}
|
||||
|
||||
write_report() {
|
||||
local report_file="$1"
|
||||
shift
|
||||
mkdir -p "$(dirname "$report_file")"
|
||||
cat > "$report_file"
|
||||
}
|
||||
|
||||
cleanup_compose() {
|
||||
local compose_file="$1"
|
||||
if [ -f "$compose_file" ]; then
|
||||
docker compose -f "$compose_file" down >/dev/null 2>&1 || true
|
||||
fi
|
||||
}
|
||||
@@ -0,0 +1,97 @@
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
. "$SCRIPT_DIR/common.sh"
|
||||
|
||||
WHATIF=0
|
||||
KEEP_DATA=0
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--what-if) WHATIF=1 ;;
|
||||
--keep-data) KEEP_DATA=1 ;;
|
||||
*) echo "Unknown argument: $arg" >&2; exit 1 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
RESTORE_ROOT="/mnt/user/backups/restore-lab/gitea"
|
||||
REPORT_ROOT="/mnt/user/backups/restore-reports"
|
||||
DATA_DIR="$RESTORE_ROOT/data"
|
||||
EXTRACT_DIR="$BORG_RESTORE_HOST_ROOT/gitea-extract"
|
||||
COMPOSE_FILE="$SCRIPT_DIR/gitea-compose.test.yml"
|
||||
REPORT_FILE="$REPORT_ROOT/gitea-$(date +%F).md"
|
||||
|
||||
if [ "$WHATIF" -eq 1 ]; then
|
||||
cat <<EOF
|
||||
Gitea restore test
|
||||
Mode: WhatIf
|
||||
RestoreRoot: $RESTORE_ROOT
|
||||
ReportRoot: $REPORT_ROOT
|
||||
Expected Borg source path: local/gitea/data
|
||||
EOF
|
||||
exit 0
|
||||
fi
|
||||
|
||||
require_cmd docker
|
||||
require_cmd curl
|
||||
require_path "$BORG_PASSPHRASE_FILE_DEFAULT"
|
||||
require_path "$COMPOSE_FILE"
|
||||
|
||||
cleanup() {
|
||||
cleanup_compose "$COMPOSE_FILE"
|
||||
if [ "$KEEP_DATA" -ne 1 ]; then
|
||||
rm -rf "$DATA_DIR"
|
||||
fi
|
||||
rm -rf "$EXTRACT_DIR"
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
rm -rf "$EXTRACT_DIR" "$RESTORE_ROOT"
|
||||
mkdir -p "$RESTORE_ROOT"
|
||||
|
||||
archive="$(latest_archive_name)"
|
||||
repo="$(borg_repo_url)"
|
||||
borg_extract "/restore/gitea-extract" "local/gitea/data"
|
||||
mv "$EXTRACT_DIR/local/gitea/data" "$DATA_DIR"
|
||||
|
||||
repo_sample="$(find "$DATA_DIR/git/repositories" -maxdepth 3 -type d | sed -n '2p')"
|
||||
|
||||
docker compose -f "$COMPOSE_FILE" up -d >/dev/null
|
||||
sleep 8
|
||||
status="$(curl -s -o /tmp/gitea-body.html -w '%{http_code}' http://127.0.0.1:13000)"
|
||||
grep -qi "Gitea" /tmp/gitea-body.html
|
||||
if timeout 5 bash -lc '</dev/tcp/127.0.0.1/12222' >/dev/null 2>&1; then
|
||||
ssh_state="open"
|
||||
else
|
||||
echo "Gitea SSH port not reachable" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
write_report "$REPORT_FILE" <<EOF
|
||||
# Gitea Restore Test Report - $(date +%F)
|
||||
|
||||
- Service: \`gitea\`
|
||||
- Source repo: \`$repo\`
|
||||
- Archive: \`$archive\`
|
||||
- Restore target: \`$DATA_DIR\`
|
||||
- Test container: \`restoretest-gitea\`
|
||||
- Test endpoints:
|
||||
- Web: \`http://127.0.0.1:13000\`
|
||||
- SSH: \`127.0.0.1:12222\`
|
||||
- Result: \`SUCCESS\`
|
||||
|
||||
## Checks
|
||||
|
||||
- Borg extract into isolated restore-lab: \`ok\`
|
||||
- HTTP status: \`$status\`
|
||||
- HTML content: \`Gitea\`
|
||||
- SSH port: \`$ssh_state\`
|
||||
- Repository sample: \`$repo_sample\`
|
||||
|
||||
## Notes
|
||||
|
||||
- Test ran without Traefik and without the productive domain.
|
||||
- Test data was cleaned after success: \`$([ "$KEEP_DATA" -eq 1 ] && echo no || echo yes)\`
|
||||
EOF
|
||||
|
||||
echo "Gitea restore test ok -> $REPORT_FILE"
|
||||
@@ -0,0 +1,113 @@
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
. "$SCRIPT_DIR/common.sh"
|
||||
|
||||
WHATIF=0
|
||||
KEEP_DATA=0
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--what-if) WHATIF=1 ;;
|
||||
--keep-data) KEEP_DATA=1 ;;
|
||||
*) echo "Unknown argument: $arg" >&2; exit 1 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
RESTORE_ROOT="/mnt/user/backups/restore-lab/paperless"
|
||||
REPORT_ROOT="/mnt/user/backups/restore-reports"
|
||||
EXTRACT_DIR="$BORG_RESTORE_HOST_ROOT/paperless-extract"
|
||||
COMPOSE_FILE="$SCRIPT_DIR/paperless-compose.test.yml"
|
||||
REPORT_FILE="$REPORT_ROOT/paperless-$(date +%F).md"
|
||||
|
||||
if [ "$WHATIF" -eq 1 ]; then
|
||||
cat <<EOF
|
||||
Paperless restore test
|
||||
Mode: WhatIf
|
||||
RestoreRoot: $RESTORE_ROOT
|
||||
ReportRoot: $REPORT_ROOT
|
||||
Expected Borg source paths:
|
||||
- local/appdata/paperless-ngx/data
|
||||
- local/paperless/media
|
||||
- local/paperless/export
|
||||
- local/paperless/consume
|
||||
- local/borg-dumps/latest/postgresql17-paperless.dump
|
||||
EOF
|
||||
exit 0
|
||||
fi
|
||||
|
||||
require_cmd docker
|
||||
require_cmd curl
|
||||
require_path "$BORG_PASSPHRASE_FILE_DEFAULT"
|
||||
require_path "$COMPOSE_FILE"
|
||||
|
||||
cleanup() {
|
||||
cleanup_compose "$COMPOSE_FILE"
|
||||
if [ "$KEEP_DATA" -ne 1 ]; then
|
||||
rm -rf "$RESTORE_ROOT"
|
||||
fi
|
||||
rm -rf "$EXTRACT_DIR"
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
rm -rf "$EXTRACT_DIR" "$RESTORE_ROOT"
|
||||
mkdir -p "$RESTORE_ROOT/postgres" "$RESTORE_ROOT/dumps/latest"
|
||||
|
||||
archive="$(latest_archive_name)"
|
||||
repo="$(borg_repo_url)"
|
||||
borg_extract "/restore/paperless-extract" \
|
||||
"local/appdata/paperless-ngx/data" \
|
||||
"local/paperless/media" \
|
||||
"local/paperless/export" \
|
||||
"local/paperless/consume" \
|
||||
"local/borg-dumps/latest/postgresql17-paperless.dump"
|
||||
|
||||
mv "$EXTRACT_DIR/local/appdata/paperless-ngx/data" "$RESTORE_ROOT/data"
|
||||
mv "$EXTRACT_DIR/local/paperless/media" "$RESTORE_ROOT/media"
|
||||
mv "$EXTRACT_DIR/local/paperless/export" "$RESTORE_ROOT/export"
|
||||
mv "$EXTRACT_DIR/local/paperless/consume" "$RESTORE_ROOT/consume"
|
||||
mv "$EXTRACT_DIR/local/borg-dumps/latest/postgresql17-paperless.dump" "$RESTORE_ROOT/dumps/latest/postgresql17-paperless.dump"
|
||||
|
||||
docker compose -f "$COMPOSE_FILE" up -d restoretest-paperless-postgres restoretest-paperless-redis >/dev/null
|
||||
until docker exec restoretest-paperless-postgres pg_isready -U paperless -d paperless >/dev/null 2>&1; do sleep 2; done
|
||||
cat "$RESTORE_ROOT/dumps/latest/postgresql17-paperless.dump" | docker exec -i restoretest-paperless-postgres pg_restore -U paperless -d paperless --clean --if-exists --no-owner --no-privileges
|
||||
|
||||
docker compose -f "$COMPOSE_FILE" up -d restoretest-paperless >/dev/null
|
||||
sleep 12
|
||||
status="$(curl -s -o /tmp/paperless-body.html -w '%{http_code}' -L http://127.0.0.1:18120)"
|
||||
grep -qi "Paperless-ngx sign in" /tmp/paperless-body.html
|
||||
doc_count="$(docker exec restoretest-paperless-postgres psql -U paperless -d paperless -tAc "select count(*) from documents_document;" | tr -d '[:space:]')"
|
||||
doc_sample="$(find "$RESTORE_ROOT/media/documents/originals" -type f | sed -n '1p')"
|
||||
|
||||
write_report "$REPORT_FILE" <<EOF
|
||||
# Paperless Restore Test Report - $(date +%F)
|
||||
|
||||
- Service: \`paperless-ngx\`
|
||||
- Source repo: \`$repo\`
|
||||
- Archive: \`$archive\`
|
||||
- Restore root: \`$RESTORE_ROOT\`
|
||||
- Test containers:
|
||||
- \`restoretest-paperless\`
|
||||
- \`restoretest-paperless-postgres\`
|
||||
- \`restoretest-paperless-redis\`
|
||||
- Test endpoint: \`http://127.0.0.1:18120\`
|
||||
- Result: \`SUCCESS\`
|
||||
|
||||
## Checks
|
||||
|
||||
- Borg extract of file data: \`ok\`
|
||||
- Borg extract of dump: \`ok\`
|
||||
- Dump import into isolated Postgres: \`ok\`
|
||||
- HTTP status after redirect: \`$status\`
|
||||
- Login page content: \`Paperless-ngx sign in\`
|
||||
- Document count in test DB: \`$doc_count\`
|
||||
- Document sample in media path: \`$doc_sample\`
|
||||
|
||||
## Notes
|
||||
|
||||
- Test ran without Traefik and without the productive domain.
|
||||
- Test used isolated Postgres and Redis containers.
|
||||
- Test data was cleaned after success: \`$([ "$KEEP_DATA" -eq 1 ] && echo no || echo yes)\`
|
||||
EOF
|
||||
|
||||
echo "Paperless restore test ok -> $REPORT_FILE"
|
||||
@@ -0,0 +1,35 @@
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
|
||||
MODE="${1:-}"
|
||||
WHATIF="${2:-}"
|
||||
|
||||
case "$MODE" in
|
||||
freshness)
|
||||
exec "$SCRIPT_DIR/check-restore-freshness.sh"
|
||||
;;
|
||||
vaultwarden)
|
||||
if [ "$WHATIF" = "--what-if" ]; then
|
||||
exec "$SCRIPT_DIR/vaultwarden-restore-test.sh" --what-if
|
||||
fi
|
||||
exec "$SCRIPT_DIR/vaultwarden-restore-test.sh"
|
||||
;;
|
||||
gitea)
|
||||
if [ "$WHATIF" = "--what-if" ]; then
|
||||
exec "$SCRIPT_DIR/gitea-restore-test.sh" --what-if
|
||||
fi
|
||||
exec "$SCRIPT_DIR/gitea-restore-test.sh"
|
||||
;;
|
||||
paperless)
|
||||
if [ "$WHATIF" = "--what-if" ]; then
|
||||
exec "$SCRIPT_DIR/paperless-restore-test.sh" --what-if
|
||||
fi
|
||||
exec "$SCRIPT_DIR/paperless-restore-test.sh"
|
||||
;;
|
||||
*)
|
||||
echo "Usage: $0 {freshness|vaultwarden|gitea|paperless} [--what-if]" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
@@ -40,7 +40,7 @@ Spaeter:
|
||||
## Konkreter Kalender
|
||||
|
||||
- Jeden Montag, 06:30:
|
||||
- `check-restore-freshness.ps1`
|
||||
- `check-restore-freshness.sh`
|
||||
- Jeden 1. Samstag im Monat, 07:00:
|
||||
- `vaultwarden`
|
||||
- Jeden 3. Samstag im Monat, 07:00:
|
||||
@@ -53,7 +53,7 @@ Spaeter:
|
||||
## Betriebsmodus
|
||||
|
||||
- V1:
|
||||
- Jobs laufen hostseitig manuell oder per User Script
|
||||
- Bash-Jobs laufen hostseitig manuell oder per User Script
|
||||
- `ntfy` ist optional und folgt nach stabiler Basis
|
||||
- Hermes wertet spaeter nur Reports aus
|
||||
- V2:
|
||||
|
||||
@@ -20,7 +20,7 @@ Inhalt:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
pwsh -File /mnt/user/services/homelab/ops/restore-tests/run-restore-checks.ps1 -Mode freshness \
|
||||
bash /mnt/user/services/homelab/ops/restore-tests/run-restore-checks.sh freshness \
|
||||
> /mnt/user/backups/restore-reports/freshness-$(date +%F).md
|
||||
```
|
||||
|
||||
@@ -40,8 +40,8 @@ V1-Inhalt:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
pwsh -File /mnt/user/services/homelab/ops/restore-tests/run-restore-checks.ps1 -Mode vaultwarden -WhatIf \
|
||||
> /mnt/user/backups/restore-reports/vaultwarden-plan-$(date +%F).md
|
||||
bash /mnt/user/services/homelab/ops/restore-tests/run-restore-checks.sh vaultwarden \
|
||||
> /mnt/user/backups/restore-reports/vaultwarden-$(date +%F).md
|
||||
```
|
||||
|
||||
## Script 3 - `restore-gitea-monthly`
|
||||
@@ -54,8 +54,8 @@ V1-Inhalt:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
pwsh -File /mnt/user/services/homelab/ops/restore-tests/run-restore-checks.ps1 -Mode gitea -WhatIf \
|
||||
> /mnt/user/backups/restore-reports/gitea-plan-$(date +%F).md
|
||||
bash /mnt/user/services/homelab/ops/restore-tests/run-restore-checks.sh gitea \
|
||||
> /mnt/user/backups/restore-reports/gitea-$(date +%F).md
|
||||
```
|
||||
|
||||
## Script 4 - `restore-paperless-bimonthly`
|
||||
@@ -68,19 +68,19 @@ V1-Inhalt:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
pwsh -File /mnt/user/services/homelab/ops/restore-tests/run-restore-checks.ps1 -Mode paperless -WhatIf \
|
||||
> /mnt/user/backups/restore-reports/paperless-plan-$(date +%F).md
|
||||
bash /mnt/user/services/homelab/ops/restore-tests/run-restore-checks.sh paperless \
|
||||
> /mnt/user/backups/restore-reports/paperless-$(date +%F).md
|
||||
```
|
||||
|
||||
## Warum V1 mit `-WhatIf`
|
||||
## Stand
|
||||
|
||||
- keine unkontrollierten Restore-Laeufe im Cron
|
||||
- erst Host-Scheduler sauber verdrahten
|
||||
- spaeter gezielt auf echte Vollautomatik umstellen
|
||||
- die Bash-Jobs wurden am 2026-05-07 hostseitig erfolgreich verifiziert
|
||||
- `freshness`, `vaultwarden`, `gitea` und `paperless` laufen damit prinzipiell automatisch
|
||||
- `ntfy` kommt erst als naechster Ausbau
|
||||
|
||||
## V2 Zielbild
|
||||
|
||||
Spaeter werden die drei Restore-Scripts von Plan-/Scaffold-Modus auf echte Host-Ausfuehrung umgestellt:
|
||||
Als naechster Ausbau kommen dazu:
|
||||
|
||||
1. Restore aus Borg
|
||||
2. Testcontainer starten
|
||||
@@ -98,7 +98,7 @@ Beispiel:
|
||||
```bash
|
||||
#!/bin/bash
|
||||
REPORT="/mnt/user/backups/restore-reports/freshness-$(date +%F).md"
|
||||
if pwsh -File /mnt/user/services/homelab/ops/restore-tests/run-restore-checks.ps1 -Mode freshness > "$REPORT"; then
|
||||
if bash /mnt/user/services/homelab/ops/restore-tests/run-restore-checks.sh freshness > "$REPORT"; then
|
||||
echo "Restore freshness check ok: $REPORT"
|
||||
else
|
||||
echo "Restore freshness check failed: $REPORT"
|
||||
|
||||
@@ -0,0 +1,85 @@
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
. "$SCRIPT_DIR/common.sh"
|
||||
|
||||
WHATIF=0
|
||||
KEEP_DATA=0
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--what-if) WHATIF=1 ;;
|
||||
--keep-data) KEEP_DATA=1 ;;
|
||||
*) echo "Unknown argument: $arg" >&2; exit 1 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
RESTORE_ROOT="/mnt/user/backups/restore-lab/vaultwarden"
|
||||
REPORT_ROOT="/mnt/user/backups/restore-reports"
|
||||
DATA_DIR="$RESTORE_ROOT/data"
|
||||
EXTRACT_DIR="$BORG_RESTORE_HOST_ROOT/vaultwarden-extract"
|
||||
COMPOSE_FILE="$SCRIPT_DIR/vaultwarden-compose.test.yml"
|
||||
REPORT_FILE="$REPORT_ROOT/vaultwarden-$(date +%F).md"
|
||||
|
||||
if [ "$WHATIF" -eq 1 ]; then
|
||||
cat <<EOF
|
||||
Vaultwarden restore test
|
||||
Mode: WhatIf
|
||||
RestoreRoot: $RESTORE_ROOT
|
||||
ReportRoot: $REPORT_ROOT
|
||||
Expected Borg source path: local/appdata/vaultwarden
|
||||
EOF
|
||||
exit 0
|
||||
fi
|
||||
|
||||
require_cmd docker
|
||||
require_cmd curl
|
||||
require_path "$BORG_PASSPHRASE_FILE_DEFAULT"
|
||||
require_path "$COMPOSE_FILE"
|
||||
|
||||
cleanup() {
|
||||
cleanup_compose "$COMPOSE_FILE"
|
||||
if [ "$KEEP_DATA" -ne 1 ]; then
|
||||
rm -rf "$DATA_DIR"
|
||||
fi
|
||||
rm -rf "$EXTRACT_DIR"
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
rm -rf "$EXTRACT_DIR" "$RESTORE_ROOT"
|
||||
mkdir -p "$RESTORE_ROOT"
|
||||
|
||||
archive="$(latest_archive_name)"
|
||||
repo="$(borg_repo_url)"
|
||||
borg_extract "/restore/vaultwarden-extract" "local/appdata/vaultwarden"
|
||||
mv "$EXTRACT_DIR/local/appdata/vaultwarden" "$DATA_DIR"
|
||||
|
||||
docker compose -f "$COMPOSE_FILE" up -d >/dev/null
|
||||
sleep 8
|
||||
status="$(curl -s -o /tmp/vaultwarden-body.html -w '%{http_code}' http://127.0.0.1:18080)"
|
||||
grep -qi "vaultwarden" /tmp/vaultwarden-body.html
|
||||
|
||||
write_report "$REPORT_FILE" <<EOF
|
||||
# Vaultwarden Restore Test Report - $(date +%F)
|
||||
|
||||
- Service: \`vaultwarden\`
|
||||
- Source repo: \`$repo\`
|
||||
- Archive: \`$archive\`
|
||||
- Restore target: \`$DATA_DIR\`
|
||||
- Test container: \`restoretest-vaultwarden\`
|
||||
- Test endpoint: \`http://127.0.0.1:18080\`
|
||||
- Result: \`SUCCESS\`
|
||||
|
||||
## Checks
|
||||
|
||||
- Borg extract into isolated restore-lab: \`ok\`
|
||||
- HTTP status: \`$status\`
|
||||
- Login page content: \`ok\`
|
||||
|
||||
## Notes
|
||||
|
||||
- Test ran without Traefik and without the productive domain.
|
||||
- Test data was cleaned after success: \`$([ "$KEEP_DATA" -eq 1 ] && echo no || echo yes)\`
|
||||
EOF
|
||||
|
||||
echo "Vaultwarden restore test ok -> $REPORT_FILE"
|
||||
Reference in New Issue
Block a user