Add host-ready restore automation scripts
This commit is contained in:
+12
-12
@@ -104,16 +104,16 @@ Alle validierten Restore-Tests folgen demselben Muster:
|
|||||||
|
|
||||||
### V1
|
### V1
|
||||||
|
|
||||||
- manuell validierte Restore-Pfade
|
- validierte Bash-Host-Jobs
|
||||||
- Host-Job-Definitionen liegen im Repo
|
- Host-Job-Definitionen liegen im Repo
|
||||||
- Scheduler kann bereits Plan- und Frische-Checks fahren
|
- Scheduler kann bereits echte Frische- und Restore-Checks fahren
|
||||||
- volle Automatik je Dienst wird danach gezielt nachgezogen
|
- `ntfy` und Hermes-Auswertung folgen danach
|
||||||
|
|
||||||
### V2
|
### V2
|
||||||
|
|
||||||
- echte Vollautomatik fuer die drei validierten Dienste
|
|
||||||
- `ntfy` bei Erfolg/Fehler
|
- `ntfy` bei Erfolg/Fehler
|
||||||
- Hermes liest spaeter Reports und baut Uebersichten
|
- Hermes liest Reports und baut Uebersichten
|
||||||
|
- zusaetzliche Rotation, Sammelreports und weitere Dienste
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -169,25 +169,25 @@ Nur `Container laeuft` reicht nicht.
|
|||||||
Auf dem Unraid-Host:
|
Auf dem Unraid-Host:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
pwsh -File /mnt/user/services/homelab/ops/restore-tests/run-restore-checks.ps1 -Mode freshness
|
bash /mnt/user/services/homelab/ops/restore-tests/run-restore-checks.sh freshness
|
||||||
```
|
```
|
||||||
|
|
||||||
### Vaultwarden Planlauf
|
### Vaultwarden Restore-Check
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
pwsh -File /mnt/user/services/homelab/ops/restore-tests/run-restore-checks.ps1 -Mode vaultwarden -WhatIf
|
bash /mnt/user/services/homelab/ops/restore-tests/run-restore-checks.sh vaultwarden
|
||||||
```
|
```
|
||||||
|
|
||||||
### Gitea Planlauf
|
### Gitea Restore-Check
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
pwsh -File /mnt/user/services/homelab/ops/restore-tests/run-restore-checks.ps1 -Mode gitea -WhatIf
|
bash /mnt/user/services/homelab/ops/restore-tests/run-restore-checks.sh gitea
|
||||||
```
|
```
|
||||||
|
|
||||||
### Paperless Planlauf
|
### Paperless Restore-Check
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
pwsh -File /mnt/user/services/homelab/ops/restore-tests/run-restore-checks.ps1 -Mode paperless -WhatIf
|
bash /mnt/user/services/homelab/ops/restore-tests/run-restore-checks.sh paperless
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|||||||
@@ -21,16 +21,22 @@ Ziel:
|
|||||||
|
|
||||||
- `schedule.md`: Intervalle und Verantwortlichkeiten
|
- `schedule.md`: Intervalle und Verantwortlichkeiten
|
||||||
- `vaultwarden-restore-test.ps1`: erster Mini-Restore-Ablauf
|
- `vaultwarden-restore-test.ps1`: erster Mini-Restore-Ablauf
|
||||||
|
- `vaultwarden-restore-test.sh`: hosttauglicher Vaultwarden-Restore-Job
|
||||||
- `vaultwarden-plan.md`: konkreter Vaultwarden-Testplan
|
- `vaultwarden-plan.md`: konkreter Vaultwarden-Testplan
|
||||||
- `vaultwarden-compose.test.yml`: isolierte Testinstanz fuer Vaultwarden
|
- `vaultwarden-compose.test.yml`: isolierte Testinstanz fuer Vaultwarden
|
||||||
- `gitea-restore-test.ps1`: Gitea-Mini-Restore-Ablauf
|
- `gitea-restore-test.ps1`: Gitea-Mini-Restore-Ablauf
|
||||||
|
- `gitea-restore-test.sh`: hosttauglicher Gitea-Restore-Job
|
||||||
- `gitea-plan.md`: konkreter Gitea-Testplan
|
- `gitea-plan.md`: konkreter Gitea-Testplan
|
||||||
- `gitea-compose.test.yml`: isolierte Testinstanz fuer Gitea
|
- `gitea-compose.test.yml`: isolierte Testinstanz fuer Gitea
|
||||||
- `paperless-restore-test.ps1`: Paperless-Mini-Restore-Ablauf
|
- `paperless-restore-test.ps1`: Paperless-Mini-Restore-Ablauf
|
||||||
|
- `paperless-restore-test.sh`: hosttauglicher Paperless-Restore-Job
|
||||||
- `paperless-plan.md`: konkreter Paperless-Testplan
|
- `paperless-plan.md`: konkreter Paperless-Testplan
|
||||||
- `paperless-compose.test.yml`: isolierte Testinstanz fuer Paperless inkl. Test-Postgres und Test-Redis
|
- `paperless-compose.test.yml`: isolierte Testinstanz fuer Paperless inkl. Test-Postgres und Test-Redis
|
||||||
- `check-restore-freshness.ps1`: woechentlicher Frische-Check fuer Dumps und Reports
|
- `check-restore-freshness.ps1`: woechentlicher Frische-Check fuer Dumps und Reports
|
||||||
- `run-restore-checks.ps1`: einfacher Dispatcher fuer Restore-Jobs
|
- `run-restore-checks.ps1`: einfacher Dispatcher fuer Restore-Jobs
|
||||||
|
- `check-restore-freshness.sh`: hosttauglicher Frische-Check
|
||||||
|
- `run-restore-checks.sh`: hosttauglicher Dispatcher
|
||||||
|
- `common.sh`: gemeinsame Host-Helferfunktionen
|
||||||
- `automation-plan.md`: Host-Job- und Automatisierungsmodell
|
- `automation-plan.md`: Host-Job- und Automatisierungsmodell
|
||||||
|
|
||||||
## Automatisierungsmodell
|
## Automatisierungsmodell
|
||||||
@@ -43,7 +49,8 @@ Ziel:
|
|||||||
|
|
||||||
Wichtig:
|
Wichtig:
|
||||||
|
|
||||||
- `check-restore-freshness.ps1` und spaetere automatische Restore-Jobs sind fuer den Unraid-Host gedacht
|
- die Bash-Skripte `*.sh` sind die produktive Host-Variante
|
||||||
|
- `check-restore-freshness.ps1` und die `*.ps1`-Dateien bleiben als lokale Plan-/Hilfsvariante nutzbar
|
||||||
- im Windows-Clone fehlen die `/mnt/user/...`-Pfade naturgemaess
|
- im Windows-Clone fehlen die `/mnt/user/...`-Pfade naturgemaess
|
||||||
|
|
||||||
## Validiertes Grundmuster
|
## Validiertes Grundmuster
|
||||||
@@ -69,6 +76,7 @@ Aktuell ist das erste validierte Muster vorhanden.
|
|||||||
- echter Vaultwarden-Restore am 2026-05-07 erfolgreich verifiziert
|
- echter Vaultwarden-Restore am 2026-05-07 erfolgreich verifiziert
|
||||||
- echter Gitea-Restore am 2026-05-07 erfolgreich verifiziert
|
- echter Gitea-Restore am 2026-05-07 erfolgreich verifiziert
|
||||||
- echter Paperless-Restore am 2026-05-07 erfolgreich verifiziert
|
- echter Paperless-Restore am 2026-05-07 erfolgreich verifiziert
|
||||||
|
- Bash-Dispatcher und Bash-Restore-Jobs am 2026-05-07 erfolgreich hostseitig verifiziert
|
||||||
- Restore-Lab und Report-Pfade auf dem Host angelegt
|
- Restore-Lab und Report-Pfade auf dem Host angelegt
|
||||||
- V1-Ablauf weiter ohne `ntfy`, mit Bereinigung nach Erfolg
|
- V1-Ablauf weiter ohne `ntfy`, mit Bereinigung nach Erfolg
|
||||||
- naechster grosser Kandidat ist ein weiterer datenbankgestuetzter Dienst oder die Automatisierung
|
- naechster grosser Kandidat ist ein weiterer datenbankgestuetzter Dienst oder die Automatisierung
|
||||||
|
|||||||
@@ -16,7 +16,7 @@ Die bereits validierten Restore-Tests fuer `vaultwarden`, `gitea` und `paperless
|
|||||||
|
|
||||||
### Woechentlicher Frische-Check
|
### Woechentlicher Frische-Check
|
||||||
|
|
||||||
- Script: `check-restore-freshness.ps1`
|
- Script: `check-restore-freshness.sh`
|
||||||
- Ziel:
|
- Ziel:
|
||||||
- Dump-Dateien vorhanden
|
- Dump-Dateien vorhanden
|
||||||
- Dump-Dateien nicht zu alt
|
- Dump-Dateien nicht zu alt
|
||||||
@@ -26,19 +26,20 @@ Die bereits validierten Restore-Tests fuer `vaultwarden`, `gitea` und `paperless
|
|||||||
|
|
||||||
### Monatliche / zweimonatliche Restore-Jobs
|
### Monatliche / zweimonatliche Restore-Jobs
|
||||||
|
|
||||||
- Script-Dispatcher: `run-restore-checks.ps1`
|
- Script-Dispatcher: `run-restore-checks.sh`
|
||||||
- Modi:
|
- Modi:
|
||||||
- `freshness`
|
- `freshness`
|
||||||
- `vaultwarden`
|
- `vaultwarden`
|
||||||
- `gitea`
|
- `gitea`
|
||||||
- `paperless`
|
- `paperless`
|
||||||
- V1 ruft die existierenden dienstspezifischen Scripts zunaechst im `WhatIf`- oder Plan-Modus auf, bis die Vollautomatisierung je Dienst gezielt nachgezogen wird.
|
- diese Bash-Jobs sind jetzt hostseitig praktisch verifiziert
|
||||||
|
- die `*.ps1`-Dateien bleiben als Plan-/Hilfsvariante fuer die Windows-Arbeitskopie erhalten
|
||||||
|
|
||||||
## V2
|
## V2
|
||||||
|
|
||||||
- echte Vollautomatisierung pro Dienst
|
|
||||||
- `ntfy` Erfolg/Fehler
|
- `ntfy` Erfolg/Fehler
|
||||||
- optional Hermes-Zusammenfassung ueber vorhandene Reports
|
- optional Hermes-Zusammenfassung ueber vorhandene Reports
|
||||||
|
- spaeter Job-Metadaten, Rotation und Sammel-Reports weiter ausbauen
|
||||||
|
|
||||||
## Host-Integration
|
## Host-Integration
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,81 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
DUMP_ROOT="${DUMP_ROOT:-/mnt/user/backups/borg/dumps/latest}"
|
||||||
|
REPORT_ROOT="${REPORT_ROOT:-/mnt/user/backups/restore-reports}"
|
||||||
|
MAX_DUMP_AGE_HOURS="${MAX_DUMP_AGE_HOURS:-36}"
|
||||||
|
MAX_REPORT_AGE_DAYS="${MAX_REPORT_AGE_DAYS:-45}"
|
||||||
|
|
||||||
|
now_epoch="$(date +%s)"
|
||||||
|
critical=()
|
||||||
|
warnings=()
|
||||||
|
info=()
|
||||||
|
|
||||||
|
check_file_age_hours() {
|
||||||
|
local path="$1"
|
||||||
|
local mtime
|
||||||
|
mtime="$(stat -c %Y "$path")"
|
||||||
|
echo $(( (now_epoch - mtime) / 3600 ))
|
||||||
|
}
|
||||||
|
|
||||||
|
check_file_age_days() {
|
||||||
|
local path="$1"
|
||||||
|
local mtime
|
||||||
|
mtime="$(stat -c %Y "$path")"
|
||||||
|
echo $(( (now_epoch - mtime) / 86400 ))
|
||||||
|
}
|
||||||
|
|
||||||
|
for dump in postgresql17-paperless.dump postgresql17-mailarchiver.dump mealie.dump immich.dump; do
|
||||||
|
path="$DUMP_ROOT/$dump"
|
||||||
|
if [ ! -f "$path" ]; then
|
||||||
|
critical+=("DUMP_MISSING $dump")
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
age="$(check_file_age_hours "$path")"
|
||||||
|
if [ "$age" -gt "$MAX_DUMP_AGE_HOURS" ]; then
|
||||||
|
warnings+=("DUMP_STALE $dump age=${age}h")
|
||||||
|
else
|
||||||
|
info+=("DUMP_OK $dump age=${age}h")
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
for service in vaultwarden gitea paperless; do
|
||||||
|
latest="$(find "$REPORT_ROOT" -maxdepth 1 -type f -name "$service-*.md" | sort | tail -n 1 || true)"
|
||||||
|
if [ -z "$latest" ]; then
|
||||||
|
warnings+=("REPORT_MISSING $service")
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
age="$(check_file_age_days "$latest")"
|
||||||
|
if [ "$age" -gt "$MAX_REPORT_AGE_DAYS" ]; then
|
||||||
|
warnings+=("REPORT_STALE $service age=${age}d file=$(basename "$latest")")
|
||||||
|
else
|
||||||
|
info+=("REPORT_OK $service age=${age}d file=$(basename "$latest")")
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
echo "# Restore Freshness Check"
|
||||||
|
echo
|
||||||
|
echo "Timestamp: $(date '+%F %T')"
|
||||||
|
echo "Critical: ${#critical[@]}"
|
||||||
|
echo "Warnings: ${#warnings[@]}"
|
||||||
|
echo "Info: ${#info[@]}"
|
||||||
|
echo
|
||||||
|
|
||||||
|
if [ "${#critical[@]}" -gt 0 ]; then
|
||||||
|
echo "## Critical"
|
||||||
|
printf -- '- %s\n' "${critical[@]}"
|
||||||
|
echo
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ "${#warnings[@]}" -gt 0 ]; then
|
||||||
|
echo "## Warnings"
|
||||||
|
printf -- '- %s\n' "${warnings[@]}"
|
||||||
|
echo
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ "${#info[@]}" -gt 0 ]; then
|
||||||
|
echo "## Info"
|
||||||
|
printf -- '- %s\n' "${info[@]}"
|
||||||
|
fi
|
||||||
|
|
||||||
|
[ "${#critical[@]}" -eq 0 ]
|
||||||
@@ -0,0 +1,84 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
RESTORE_TESTS_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
BORG_CONTAINER="${BORG_CONTAINER:-borg-ui}"
|
||||||
|
BORG_RESTORE_HOST_ROOT="${BORG_RESTORE_HOST_ROOT:-/mnt/user/appdata/borg-ui/restore}"
|
||||||
|
BORG_PASSPHRASE_FILE_DEFAULT="${BORG_PASSPHRASE_FILE_DEFAULT:-/mnt/user/appdata/secrets/borg_repo_passphrase.txt}"
|
||||||
|
|
||||||
|
require_cmd() {
|
||||||
|
command -v "$1" >/dev/null 2>&1 || {
|
||||||
|
echo "Missing command: $1" >&2
|
||||||
|
exit 1
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
require_path() {
|
||||||
|
[ -e "$1" ] || {
|
||||||
|
echo "Missing path: $1" >&2
|
||||||
|
exit 1
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
latest_archive_name() {
|
||||||
|
docker exec "$BORG_CONTAINER" python3 - <<'PY'
|
||||||
|
import sqlite3
|
||||||
|
conn = sqlite3.connect('/data/borg.db')
|
||||||
|
cur = conn.cursor()
|
||||||
|
cur.execute("select archive_name from backup_jobs where status='completed' order by created_at desc limit 1")
|
||||||
|
row = cur.fetchone()
|
||||||
|
if not row:
|
||||||
|
raise SystemExit("No completed borg archive found")
|
||||||
|
print(row[0])
|
||||||
|
PY
|
||||||
|
}
|
||||||
|
|
||||||
|
borg_repo_url() {
|
||||||
|
docker exec "$BORG_CONTAINER" python3 - <<'PY'
|
||||||
|
import sqlite3
|
||||||
|
conn = sqlite3.connect('/data/borg.db')
|
||||||
|
cur = conn.cursor()
|
||||||
|
cur.execute("select path from repositories where path is not null and path != '' order by id asc limit 1")
|
||||||
|
row = cur.fetchone()
|
||||||
|
if not row:
|
||||||
|
raise SystemExit("No borg repository configured")
|
||||||
|
print(row[0])
|
||||||
|
PY
|
||||||
|
}
|
||||||
|
|
||||||
|
borg_extract() {
|
||||||
|
local extract_dir="$1"
|
||||||
|
shift
|
||||||
|
local paths=("$@")
|
||||||
|
docker exec -i "$BORG_CONTAINER" python3 - "$extract_dir" "${paths[@]}" <<'PY'
|
||||||
|
import os, sys, subprocess
|
||||||
|
extract_dir = sys.argv[1]
|
||||||
|
paths = sys.argv[2:]
|
||||||
|
import sqlite3
|
||||||
|
conn = sqlite3.connect('/data/borg.db')
|
||||||
|
cur = conn.cursor()
|
||||||
|
cur.execute("select path from repositories where path is not null and path != '' order by id asc limit 1")
|
||||||
|
repo = cur.fetchone()[0]
|
||||||
|
cur.execute("select archive_name from backup_jobs where status='completed' order by created_at desc limit 1")
|
||||||
|
archive = cur.fetchone()[0]
|
||||||
|
with open('/local/secrets/borg_repo_passphrase.txt', 'r', encoding='utf-8') as f:
|
||||||
|
os.environ['BORG_PASSPHRASE'] = f.read().strip()
|
||||||
|
os.makedirs(extract_dir, exist_ok=True)
|
||||||
|
os.chdir(extract_dir)
|
||||||
|
subprocess.run(['borg', 'extract', f'{repo}::{archive}', *paths], check=True)
|
||||||
|
PY
|
||||||
|
}
|
||||||
|
|
||||||
|
write_report() {
|
||||||
|
local report_file="$1"
|
||||||
|
shift
|
||||||
|
mkdir -p "$(dirname "$report_file")"
|
||||||
|
cat > "$report_file"
|
||||||
|
}
|
||||||
|
|
||||||
|
cleanup_compose() {
|
||||||
|
local compose_file="$1"
|
||||||
|
if [ -f "$compose_file" ]; then
|
||||||
|
docker compose -f "$compose_file" down >/dev/null 2>&1 || true
|
||||||
|
fi
|
||||||
|
}
|
||||||
@@ -0,0 +1,97 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||||
|
. "$SCRIPT_DIR/common.sh"
|
||||||
|
|
||||||
|
WHATIF=0
|
||||||
|
KEEP_DATA=0
|
||||||
|
for arg in "$@"; do
|
||||||
|
case "$arg" in
|
||||||
|
--what-if) WHATIF=1 ;;
|
||||||
|
--keep-data) KEEP_DATA=1 ;;
|
||||||
|
*) echo "Unknown argument: $arg" >&2; exit 1 ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
RESTORE_ROOT="/mnt/user/backups/restore-lab/gitea"
|
||||||
|
REPORT_ROOT="/mnt/user/backups/restore-reports"
|
||||||
|
DATA_DIR="$RESTORE_ROOT/data"
|
||||||
|
EXTRACT_DIR="$BORG_RESTORE_HOST_ROOT/gitea-extract"
|
||||||
|
COMPOSE_FILE="$SCRIPT_DIR/gitea-compose.test.yml"
|
||||||
|
REPORT_FILE="$REPORT_ROOT/gitea-$(date +%F).md"
|
||||||
|
|
||||||
|
if [ "$WHATIF" -eq 1 ]; then
|
||||||
|
cat <<EOF
|
||||||
|
Gitea restore test
|
||||||
|
Mode: WhatIf
|
||||||
|
RestoreRoot: $RESTORE_ROOT
|
||||||
|
ReportRoot: $REPORT_ROOT
|
||||||
|
Expected Borg source path: local/gitea/data
|
||||||
|
EOF
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
require_cmd docker
|
||||||
|
require_cmd curl
|
||||||
|
require_path "$BORG_PASSPHRASE_FILE_DEFAULT"
|
||||||
|
require_path "$COMPOSE_FILE"
|
||||||
|
|
||||||
|
cleanup() {
|
||||||
|
cleanup_compose "$COMPOSE_FILE"
|
||||||
|
if [ "$KEEP_DATA" -ne 1 ]; then
|
||||||
|
rm -rf "$DATA_DIR"
|
||||||
|
fi
|
||||||
|
rm -rf "$EXTRACT_DIR"
|
||||||
|
}
|
||||||
|
trap cleanup EXIT
|
||||||
|
|
||||||
|
rm -rf "$EXTRACT_DIR" "$RESTORE_ROOT"
|
||||||
|
mkdir -p "$RESTORE_ROOT"
|
||||||
|
|
||||||
|
archive="$(latest_archive_name)"
|
||||||
|
repo="$(borg_repo_url)"
|
||||||
|
borg_extract "/restore/gitea-extract" "local/gitea/data"
|
||||||
|
mv "$EXTRACT_DIR/local/gitea/data" "$DATA_DIR"
|
||||||
|
|
||||||
|
repo_sample="$(find "$DATA_DIR/git/repositories" -maxdepth 3 -type d | sed -n '2p')"
|
||||||
|
|
||||||
|
docker compose -f "$COMPOSE_FILE" up -d >/dev/null
|
||||||
|
sleep 8
|
||||||
|
status="$(curl -s -o /tmp/gitea-body.html -w '%{http_code}' http://127.0.0.1:13000)"
|
||||||
|
grep -qi "Gitea" /tmp/gitea-body.html
|
||||||
|
if timeout 5 bash -lc '</dev/tcp/127.0.0.1/12222' >/dev/null 2>&1; then
|
||||||
|
ssh_state="open"
|
||||||
|
else
|
||||||
|
echo "Gitea SSH port not reachable" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
write_report "$REPORT_FILE" <<EOF
|
||||||
|
# Gitea Restore Test Report - $(date +%F)
|
||||||
|
|
||||||
|
- Service: \`gitea\`
|
||||||
|
- Source repo: \`$repo\`
|
||||||
|
- Archive: \`$archive\`
|
||||||
|
- Restore target: \`$DATA_DIR\`
|
||||||
|
- Test container: \`restoretest-gitea\`
|
||||||
|
- Test endpoints:
|
||||||
|
- Web: \`http://127.0.0.1:13000\`
|
||||||
|
- SSH: \`127.0.0.1:12222\`
|
||||||
|
- Result: \`SUCCESS\`
|
||||||
|
|
||||||
|
## Checks
|
||||||
|
|
||||||
|
- Borg extract into isolated restore-lab: \`ok\`
|
||||||
|
- HTTP status: \`$status\`
|
||||||
|
- HTML content: \`Gitea\`
|
||||||
|
- SSH port: \`$ssh_state\`
|
||||||
|
- Repository sample: \`$repo_sample\`
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
- Test ran without Traefik and without the productive domain.
|
||||||
|
- Test data was cleaned after success: \`$([ "$KEEP_DATA" -eq 1 ] && echo no || echo yes)\`
|
||||||
|
EOF
|
||||||
|
|
||||||
|
echo "Gitea restore test ok -> $REPORT_FILE"
|
||||||
@@ -0,0 +1,113 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||||
|
. "$SCRIPT_DIR/common.sh"
|
||||||
|
|
||||||
|
WHATIF=0
|
||||||
|
KEEP_DATA=0
|
||||||
|
for arg in "$@"; do
|
||||||
|
case "$arg" in
|
||||||
|
--what-if) WHATIF=1 ;;
|
||||||
|
--keep-data) KEEP_DATA=1 ;;
|
||||||
|
*) echo "Unknown argument: $arg" >&2; exit 1 ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
RESTORE_ROOT="/mnt/user/backups/restore-lab/paperless"
|
||||||
|
REPORT_ROOT="/mnt/user/backups/restore-reports"
|
||||||
|
EXTRACT_DIR="$BORG_RESTORE_HOST_ROOT/paperless-extract"
|
||||||
|
COMPOSE_FILE="$SCRIPT_DIR/paperless-compose.test.yml"
|
||||||
|
REPORT_FILE="$REPORT_ROOT/paperless-$(date +%F).md"
|
||||||
|
|
||||||
|
if [ "$WHATIF" -eq 1 ]; then
|
||||||
|
cat <<EOF
|
||||||
|
Paperless restore test
|
||||||
|
Mode: WhatIf
|
||||||
|
RestoreRoot: $RESTORE_ROOT
|
||||||
|
ReportRoot: $REPORT_ROOT
|
||||||
|
Expected Borg source paths:
|
||||||
|
- local/appdata/paperless-ngx/data
|
||||||
|
- local/paperless/media
|
||||||
|
- local/paperless/export
|
||||||
|
- local/paperless/consume
|
||||||
|
- local/borg-dumps/latest/postgresql17-paperless.dump
|
||||||
|
EOF
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
require_cmd docker
|
||||||
|
require_cmd curl
|
||||||
|
require_path "$BORG_PASSPHRASE_FILE_DEFAULT"
|
||||||
|
require_path "$COMPOSE_FILE"
|
||||||
|
|
||||||
|
cleanup() {
|
||||||
|
cleanup_compose "$COMPOSE_FILE"
|
||||||
|
if [ "$KEEP_DATA" -ne 1 ]; then
|
||||||
|
rm -rf "$RESTORE_ROOT"
|
||||||
|
fi
|
||||||
|
rm -rf "$EXTRACT_DIR"
|
||||||
|
}
|
||||||
|
trap cleanup EXIT
|
||||||
|
|
||||||
|
rm -rf "$EXTRACT_DIR" "$RESTORE_ROOT"
|
||||||
|
mkdir -p "$RESTORE_ROOT/postgres" "$RESTORE_ROOT/dumps/latest"
|
||||||
|
|
||||||
|
archive="$(latest_archive_name)"
|
||||||
|
repo="$(borg_repo_url)"
|
||||||
|
borg_extract "/restore/paperless-extract" \
|
||||||
|
"local/appdata/paperless-ngx/data" \
|
||||||
|
"local/paperless/media" \
|
||||||
|
"local/paperless/export" \
|
||||||
|
"local/paperless/consume" \
|
||||||
|
"local/borg-dumps/latest/postgresql17-paperless.dump"
|
||||||
|
|
||||||
|
mv "$EXTRACT_DIR/local/appdata/paperless-ngx/data" "$RESTORE_ROOT/data"
|
||||||
|
mv "$EXTRACT_DIR/local/paperless/media" "$RESTORE_ROOT/media"
|
||||||
|
mv "$EXTRACT_DIR/local/paperless/export" "$RESTORE_ROOT/export"
|
||||||
|
mv "$EXTRACT_DIR/local/paperless/consume" "$RESTORE_ROOT/consume"
|
||||||
|
mv "$EXTRACT_DIR/local/borg-dumps/latest/postgresql17-paperless.dump" "$RESTORE_ROOT/dumps/latest/postgresql17-paperless.dump"
|
||||||
|
|
||||||
|
docker compose -f "$COMPOSE_FILE" up -d restoretest-paperless-postgres restoretest-paperless-redis >/dev/null
|
||||||
|
until docker exec restoretest-paperless-postgres pg_isready -U paperless -d paperless >/dev/null 2>&1; do sleep 2; done
|
||||||
|
cat "$RESTORE_ROOT/dumps/latest/postgresql17-paperless.dump" | docker exec -i restoretest-paperless-postgres pg_restore -U paperless -d paperless --clean --if-exists --no-owner --no-privileges
|
||||||
|
|
||||||
|
docker compose -f "$COMPOSE_FILE" up -d restoretest-paperless >/dev/null
|
||||||
|
sleep 12
|
||||||
|
status="$(curl -s -o /tmp/paperless-body.html -w '%{http_code}' -L http://127.0.0.1:18120)"
|
||||||
|
grep -qi "Paperless-ngx sign in" /tmp/paperless-body.html
|
||||||
|
doc_count="$(docker exec restoretest-paperless-postgres psql -U paperless -d paperless -tAc "select count(*) from documents_document;" | tr -d '[:space:]')"
|
||||||
|
doc_sample="$(find "$RESTORE_ROOT/media/documents/originals" -type f | sed -n '1p')"
|
||||||
|
|
||||||
|
write_report "$REPORT_FILE" <<EOF
|
||||||
|
# Paperless Restore Test Report - $(date +%F)
|
||||||
|
|
||||||
|
- Service: \`paperless-ngx\`
|
||||||
|
- Source repo: \`$repo\`
|
||||||
|
- Archive: \`$archive\`
|
||||||
|
- Restore root: \`$RESTORE_ROOT\`
|
||||||
|
- Test containers:
|
||||||
|
- \`restoretest-paperless\`
|
||||||
|
- \`restoretest-paperless-postgres\`
|
||||||
|
- \`restoretest-paperless-redis\`
|
||||||
|
- Test endpoint: \`http://127.0.0.1:18120\`
|
||||||
|
- Result: \`SUCCESS\`
|
||||||
|
|
||||||
|
## Checks
|
||||||
|
|
||||||
|
- Borg extract of file data: \`ok\`
|
||||||
|
- Borg extract of dump: \`ok\`
|
||||||
|
- Dump import into isolated Postgres: \`ok\`
|
||||||
|
- HTTP status after redirect: \`$status\`
|
||||||
|
- Login page content: \`Paperless-ngx sign in\`
|
||||||
|
- Document count in test DB: \`$doc_count\`
|
||||||
|
- Document sample in media path: \`$doc_sample\`
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
- Test ran without Traefik and without the productive domain.
|
||||||
|
- Test used isolated Postgres and Redis containers.
|
||||||
|
- Test data was cleaned after success: \`$([ "$KEEP_DATA" -eq 1 ] && echo no || echo yes)\`
|
||||||
|
EOF
|
||||||
|
|
||||||
|
echo "Paperless restore test ok -> $REPORT_FILE"
|
||||||
@@ -0,0 +1,35 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||||
|
|
||||||
|
MODE="${1:-}"
|
||||||
|
WHATIF="${2:-}"
|
||||||
|
|
||||||
|
case "$MODE" in
|
||||||
|
freshness)
|
||||||
|
exec "$SCRIPT_DIR/check-restore-freshness.sh"
|
||||||
|
;;
|
||||||
|
vaultwarden)
|
||||||
|
if [ "$WHATIF" = "--what-if" ]; then
|
||||||
|
exec "$SCRIPT_DIR/vaultwarden-restore-test.sh" --what-if
|
||||||
|
fi
|
||||||
|
exec "$SCRIPT_DIR/vaultwarden-restore-test.sh"
|
||||||
|
;;
|
||||||
|
gitea)
|
||||||
|
if [ "$WHATIF" = "--what-if" ]; then
|
||||||
|
exec "$SCRIPT_DIR/gitea-restore-test.sh" --what-if
|
||||||
|
fi
|
||||||
|
exec "$SCRIPT_DIR/gitea-restore-test.sh"
|
||||||
|
;;
|
||||||
|
paperless)
|
||||||
|
if [ "$WHATIF" = "--what-if" ]; then
|
||||||
|
exec "$SCRIPT_DIR/paperless-restore-test.sh" --what-if
|
||||||
|
fi
|
||||||
|
exec "$SCRIPT_DIR/paperless-restore-test.sh"
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
echo "Usage: $0 {freshness|vaultwarden|gitea|paperless} [--what-if]" >&2
|
||||||
|
exit 1
|
||||||
|
;;
|
||||||
|
esac
|
||||||
@@ -40,7 +40,7 @@ Spaeter:
|
|||||||
## Konkreter Kalender
|
## Konkreter Kalender
|
||||||
|
|
||||||
- Jeden Montag, 06:30:
|
- Jeden Montag, 06:30:
|
||||||
- `check-restore-freshness.ps1`
|
- `check-restore-freshness.sh`
|
||||||
- Jeden 1. Samstag im Monat, 07:00:
|
- Jeden 1. Samstag im Monat, 07:00:
|
||||||
- `vaultwarden`
|
- `vaultwarden`
|
||||||
- Jeden 3. Samstag im Monat, 07:00:
|
- Jeden 3. Samstag im Monat, 07:00:
|
||||||
@@ -53,7 +53,7 @@ Spaeter:
|
|||||||
## Betriebsmodus
|
## Betriebsmodus
|
||||||
|
|
||||||
- V1:
|
- V1:
|
||||||
- Jobs laufen hostseitig manuell oder per User Script
|
- Bash-Jobs laufen hostseitig manuell oder per User Script
|
||||||
- `ntfy` ist optional und folgt nach stabiler Basis
|
- `ntfy` ist optional und folgt nach stabiler Basis
|
||||||
- Hermes wertet spaeter nur Reports aus
|
- Hermes wertet spaeter nur Reports aus
|
||||||
- V2:
|
- V2:
|
||||||
|
|||||||
@@ -20,7 +20,7 @@ Inhalt:
|
|||||||
|
|
||||||
```bash
|
```bash
|
||||||
#!/bin/bash
|
#!/bin/bash
|
||||||
pwsh -File /mnt/user/services/homelab/ops/restore-tests/run-restore-checks.ps1 -Mode freshness \
|
bash /mnt/user/services/homelab/ops/restore-tests/run-restore-checks.sh freshness \
|
||||||
> /mnt/user/backups/restore-reports/freshness-$(date +%F).md
|
> /mnt/user/backups/restore-reports/freshness-$(date +%F).md
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -40,8 +40,8 @@ V1-Inhalt:
|
|||||||
|
|
||||||
```bash
|
```bash
|
||||||
#!/bin/bash
|
#!/bin/bash
|
||||||
pwsh -File /mnt/user/services/homelab/ops/restore-tests/run-restore-checks.ps1 -Mode vaultwarden -WhatIf \
|
bash /mnt/user/services/homelab/ops/restore-tests/run-restore-checks.sh vaultwarden \
|
||||||
> /mnt/user/backups/restore-reports/vaultwarden-plan-$(date +%F).md
|
> /mnt/user/backups/restore-reports/vaultwarden-$(date +%F).md
|
||||||
```
|
```
|
||||||
|
|
||||||
## Script 3 - `restore-gitea-monthly`
|
## Script 3 - `restore-gitea-monthly`
|
||||||
@@ -54,8 +54,8 @@ V1-Inhalt:
|
|||||||
|
|
||||||
```bash
|
```bash
|
||||||
#!/bin/bash
|
#!/bin/bash
|
||||||
pwsh -File /mnt/user/services/homelab/ops/restore-tests/run-restore-checks.ps1 -Mode gitea -WhatIf \
|
bash /mnt/user/services/homelab/ops/restore-tests/run-restore-checks.sh gitea \
|
||||||
> /mnt/user/backups/restore-reports/gitea-plan-$(date +%F).md
|
> /mnt/user/backups/restore-reports/gitea-$(date +%F).md
|
||||||
```
|
```
|
||||||
|
|
||||||
## Script 4 - `restore-paperless-bimonthly`
|
## Script 4 - `restore-paperless-bimonthly`
|
||||||
@@ -68,19 +68,19 @@ V1-Inhalt:
|
|||||||
|
|
||||||
```bash
|
```bash
|
||||||
#!/bin/bash
|
#!/bin/bash
|
||||||
pwsh -File /mnt/user/services/homelab/ops/restore-tests/run-restore-checks.ps1 -Mode paperless -WhatIf \
|
bash /mnt/user/services/homelab/ops/restore-tests/run-restore-checks.sh paperless \
|
||||||
> /mnt/user/backups/restore-reports/paperless-plan-$(date +%F).md
|
> /mnt/user/backups/restore-reports/paperless-$(date +%F).md
|
||||||
```
|
```
|
||||||
|
|
||||||
## Warum V1 mit `-WhatIf`
|
## Stand
|
||||||
|
|
||||||
- keine unkontrollierten Restore-Laeufe im Cron
|
- die Bash-Jobs wurden am 2026-05-07 hostseitig erfolgreich verifiziert
|
||||||
- erst Host-Scheduler sauber verdrahten
|
- `freshness`, `vaultwarden`, `gitea` und `paperless` laufen damit prinzipiell automatisch
|
||||||
- spaeter gezielt auf echte Vollautomatik umstellen
|
- `ntfy` kommt erst als naechster Ausbau
|
||||||
|
|
||||||
## V2 Zielbild
|
## V2 Zielbild
|
||||||
|
|
||||||
Spaeter werden die drei Restore-Scripts von Plan-/Scaffold-Modus auf echte Host-Ausfuehrung umgestellt:
|
Als naechster Ausbau kommen dazu:
|
||||||
|
|
||||||
1. Restore aus Borg
|
1. Restore aus Borg
|
||||||
2. Testcontainer starten
|
2. Testcontainer starten
|
||||||
@@ -98,7 +98,7 @@ Beispiel:
|
|||||||
```bash
|
```bash
|
||||||
#!/bin/bash
|
#!/bin/bash
|
||||||
REPORT="/mnt/user/backups/restore-reports/freshness-$(date +%F).md"
|
REPORT="/mnt/user/backups/restore-reports/freshness-$(date +%F).md"
|
||||||
if pwsh -File /mnt/user/services/homelab/ops/restore-tests/run-restore-checks.ps1 -Mode freshness > "$REPORT"; then
|
if bash /mnt/user/services/homelab/ops/restore-tests/run-restore-checks.sh freshness > "$REPORT"; then
|
||||||
echo "Restore freshness check ok: $REPORT"
|
echo "Restore freshness check ok: $REPORT"
|
||||||
else
|
else
|
||||||
echo "Restore freshness check failed: $REPORT"
|
echo "Restore freshness check failed: $REPORT"
|
||||||
|
|||||||
@@ -0,0 +1,85 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||||
|
. "$SCRIPT_DIR/common.sh"
|
||||||
|
|
||||||
|
WHATIF=0
|
||||||
|
KEEP_DATA=0
|
||||||
|
for arg in "$@"; do
|
||||||
|
case "$arg" in
|
||||||
|
--what-if) WHATIF=1 ;;
|
||||||
|
--keep-data) KEEP_DATA=1 ;;
|
||||||
|
*) echo "Unknown argument: $arg" >&2; exit 1 ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
RESTORE_ROOT="/mnt/user/backups/restore-lab/vaultwarden"
|
||||||
|
REPORT_ROOT="/mnt/user/backups/restore-reports"
|
||||||
|
DATA_DIR="$RESTORE_ROOT/data"
|
||||||
|
EXTRACT_DIR="$BORG_RESTORE_HOST_ROOT/vaultwarden-extract"
|
||||||
|
COMPOSE_FILE="$SCRIPT_DIR/vaultwarden-compose.test.yml"
|
||||||
|
REPORT_FILE="$REPORT_ROOT/vaultwarden-$(date +%F).md"
|
||||||
|
|
||||||
|
if [ "$WHATIF" -eq 1 ]; then
|
||||||
|
cat <<EOF
|
||||||
|
Vaultwarden restore test
|
||||||
|
Mode: WhatIf
|
||||||
|
RestoreRoot: $RESTORE_ROOT
|
||||||
|
ReportRoot: $REPORT_ROOT
|
||||||
|
Expected Borg source path: local/appdata/vaultwarden
|
||||||
|
EOF
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
require_cmd docker
|
||||||
|
require_cmd curl
|
||||||
|
require_path "$BORG_PASSPHRASE_FILE_DEFAULT"
|
||||||
|
require_path "$COMPOSE_FILE"
|
||||||
|
|
||||||
|
cleanup() {
|
||||||
|
cleanup_compose "$COMPOSE_FILE"
|
||||||
|
if [ "$KEEP_DATA" -ne 1 ]; then
|
||||||
|
rm -rf "$DATA_DIR"
|
||||||
|
fi
|
||||||
|
rm -rf "$EXTRACT_DIR"
|
||||||
|
}
|
||||||
|
trap cleanup EXIT
|
||||||
|
|
||||||
|
rm -rf "$EXTRACT_DIR" "$RESTORE_ROOT"
|
||||||
|
mkdir -p "$RESTORE_ROOT"
|
||||||
|
|
||||||
|
archive="$(latest_archive_name)"
|
||||||
|
repo="$(borg_repo_url)"
|
||||||
|
borg_extract "/restore/vaultwarden-extract" "local/appdata/vaultwarden"
|
||||||
|
mv "$EXTRACT_DIR/local/appdata/vaultwarden" "$DATA_DIR"
|
||||||
|
|
||||||
|
docker compose -f "$COMPOSE_FILE" up -d >/dev/null
|
||||||
|
sleep 8
|
||||||
|
status="$(curl -s -o /tmp/vaultwarden-body.html -w '%{http_code}' http://127.0.0.1:18080)"
|
||||||
|
grep -qi "vaultwarden" /tmp/vaultwarden-body.html
|
||||||
|
|
||||||
|
write_report "$REPORT_FILE" <<EOF
|
||||||
|
# Vaultwarden Restore Test Report - $(date +%F)
|
||||||
|
|
||||||
|
- Service: \`vaultwarden\`
|
||||||
|
- Source repo: \`$repo\`
|
||||||
|
- Archive: \`$archive\`
|
||||||
|
- Restore target: \`$DATA_DIR\`
|
||||||
|
- Test container: \`restoretest-vaultwarden\`
|
||||||
|
- Test endpoint: \`http://127.0.0.1:18080\`
|
||||||
|
- Result: \`SUCCESS\`
|
||||||
|
|
||||||
|
## Checks
|
||||||
|
|
||||||
|
- Borg extract into isolated restore-lab: \`ok\`
|
||||||
|
- HTTP status: \`$status\`
|
||||||
|
- Login page content: \`ok\`
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
- Test ran without Traefik and without the productive domain.
|
||||||
|
- Test data was cleaned after success: \`$([ "$KEEP_DATA" -eq 1 ] && echo no || echo yes)\`
|
||||||
|
EOF
|
||||||
|
|
||||||
|
echo "Vaultwarden restore test ok -> $REPORT_FILE"
|
||||||
Reference in New Issue
Block a user