fix(restore): harden restore checks and add authelia smoke scaffold
This commit is contained in:
@@ -10,18 +10,22 @@ Host-Repo-Pfad:
|
||||
/mnt/user/services/homelab-infra
|
||||
```
|
||||
|
||||
**Wichtig - Cron-Semantik**: Vixie-Cron verknuepft `day-of-month` und `day-of-week` mit **OR**, sobald beide gesetzt sind. Wir triggern daher an jedem Samstag/Sonntag und filtern den Monatstag per Shell-Guard im User-Script. Siehe `ops/restore-tests/schedule.md`.
|
||||
|
||||
**Wichtig - keine doppelten Schreiber**: die Restore-Skripte schreiben ihren Markdown-Report **selbst** nach `/mnt/user/backups/restore-reports/<service>-YYYY-MM-DD.md`. User-Scripts duerfen den Job-Output **nicht** in dieselbe Datei umleiten, sonst gewinnt der letzte Writer. Wrapper-Output landet stattdessen in `/mnt/user/backups/restore-reports/_wrapper-<mode>-YYYY-MM-DD.log`.
|
||||
|
||||
## Script 1 - `restore-freshness-weekly`
|
||||
|
||||
Zeit:
|
||||
Cron:
|
||||
|
||||
- Montag, 06:30
|
||||
- `30 6 * * 1` (Montag 06:30)
|
||||
|
||||
Inhalt:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
bash /mnt/user/services/homelab-infra/ops/restore-tests/run-restore-checks.sh freshness \
|
||||
> /mnt/user/backups/restore-reports/freshness-$(date +%F).md
|
||||
exec /mnt/user/services/homelab-infra/ops/restore-tests/run-restore-job-with-ntfy.sh \
|
||||
freshness homelab-info
|
||||
```
|
||||
|
||||
Erwartung:
|
||||
@@ -32,77 +36,110 @@ Erwartung:
|
||||
|
||||
## Script 2 - `restore-vaultwarden-monthly`
|
||||
|
||||
Zeit:
|
||||
Cron:
|
||||
|
||||
- 1. Samstag im Monat, 07:00
|
||||
- `0 7 * * 6` (jeden Samstag 07:00)
|
||||
|
||||
V1-Inhalt:
|
||||
Guard: nur am ersten Samstag im Monat ausfuehren.
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
bash /mnt/user/services/homelab-infra/ops/restore-tests/run-restore-checks.sh vaultwarden \
|
||||
> /mnt/user/backups/restore-reports/vaultwarden-$(date +%F).md
|
||||
# Guard: nur 1.-7. Tag im Monat, damit "1. Samstag" eindeutig getroffen wird.
|
||||
day=$(date +%-d)
|
||||
if [ "$day" -lt 1 ] || [ "$day" -gt 7 ]; then
|
||||
exit 0
|
||||
fi
|
||||
exec /mnt/user/services/homelab-infra/ops/restore-tests/run-restore-job-with-ntfy.sh \
|
||||
vaultwarden homelab-info
|
||||
```
|
||||
|
||||
## Script 3 - `restore-gitea-monthly`
|
||||
|
||||
Zeit:
|
||||
Cron:
|
||||
|
||||
- 3. Samstag im Monat, 07:00
|
||||
- `15 7 * * 6` (jeden Samstag 07:15)
|
||||
|
||||
V1-Inhalt:
|
||||
Guard: nur am dritten Samstag im Monat ausfuehren.
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
bash /mnt/user/services/homelab-infra/ops/restore-tests/run-restore-checks.sh gitea \
|
||||
> /mnt/user/backups/restore-reports/gitea-$(date +%F).md
|
||||
day=$(date +%-d)
|
||||
if [ "$day" -lt 15 ] || [ "$day" -gt 21 ]; then
|
||||
exit 0
|
||||
fi
|
||||
exec /mnt/user/services/homelab-infra/ops/restore-tests/run-restore-job-with-ntfy.sh \
|
||||
gitea homelab-info
|
||||
```
|
||||
|
||||
## Script 4 - `restore-paperless-bimonthly`
|
||||
|
||||
Zeit:
|
||||
Cron:
|
||||
|
||||
- jeder 2. Monat, 2. Samstag, 08:00
|
||||
- `0 8 * * 6` (jeden Samstag 08:00)
|
||||
|
||||
V1-Inhalt:
|
||||
Guard: nur am zweiten Samstag in ungeraden Monaten ausfuehren.
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
bash /mnt/user/services/homelab-infra/ops/restore-tests/run-restore-checks.sh paperless \
|
||||
> /mnt/user/backups/restore-reports/paperless-$(date +%F).md
|
||||
month=$(date +%-m)
|
||||
day=$(date +%-d)
|
||||
case "$month" in
|
||||
1|3|5|7|9|11) ;;
|
||||
*) exit 0 ;;
|
||||
esac
|
||||
if [ "$day" -lt 8 ] || [ "$day" -gt 14 ]; then
|
||||
exit 0
|
||||
fi
|
||||
exec /mnt/user/services/homelab-infra/ops/restore-tests/run-restore-job-with-ntfy.sh \
|
||||
paperless homelab-info
|
||||
```
|
||||
|
||||
## Script 5 - `restore-immich-quarterly`
|
||||
|
||||
Cron:
|
||||
|
||||
- `30 8 * * 0` (jeden Sonntag 08:30)
|
||||
|
||||
Guard: nur am zweiten Sonntag in Feb/Mai/Aug/Nov ausfuehren.
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
month=$(date +%-m)
|
||||
day=$(date +%-d)
|
||||
case "$month" in
|
||||
2|5|8|11) ;;
|
||||
*) exit 0 ;;
|
||||
esac
|
||||
if [ "$day" -lt 8 ] || [ "$day" -gt 14 ]; then
|
||||
exit 0
|
||||
fi
|
||||
exec /mnt/user/services/homelab-infra/ops/restore-tests/run-restore-job-with-ntfy.sh \
|
||||
immich homelab-info
|
||||
```
|
||||
|
||||
## Script 6 - `monthly-random-restore`
|
||||
|
||||
Cron:
|
||||
|
||||
- `0 9 1 * *` (erster Kalendertag im Monat 09:00) - kein Guard noetig.
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
exec /mnt/user/services/homelab-infra/ops/restore-tests/monthly-random-restore.sh
|
||||
```
|
||||
|
||||
## Stand
|
||||
|
||||
- die Bash-Jobs wurden am 2026-05-07 hostseitig erfolgreich verifiziert
|
||||
- `freshness`, `vaultwarden`, `gitea` und `paperless` laufen damit prinzipiell automatisch
|
||||
- `ntfy` kann jetzt optional per Wrapper-Skript ergaenzt werden
|
||||
- ntfy-Wrapper schreibt Erfolg/Fehler-Meldungen an die definierten Topics
|
||||
|
||||
## V2 Zielbild
|
||||
## Fehler-Topic
|
||||
|
||||
Als naechster Ausbau kommen dazu:
|
||||
Fehler gehen unabhaengig vom Erfolgstopic nach `homelab-alerts` (siehe `RESTORE_FAILURE_TOPIC` im Wrapper), damit Restore-Probleme auf demselben Handy-Topic landen wie Prometheus-, Docker-, Borg- und Posture-Alarme.
|
||||
|
||||
1. Restore aus Borg
|
||||
2. Testcontainer starten
|
||||
3. Smoke-Test
|
||||
4. Report schreiben
|
||||
5. optional `ntfy`
|
||||
6. Bereinigung
|
||||
|
||||
## Optionales `ntfy` Wrapper-Muster
|
||||
|
||||
Wenn `ntfy` genutzt wird, soll der Host-Job nur Erfolg/Fehler referenzieren, nicht den ganzen Report in die Nachricht kippen.
|
||||
|
||||
Beispiel:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
bash /mnt/user/services/homelab-infra/ops/restore-tests/run-restore-job-with-ntfy.sh freshness homelab-info
|
||||
```
|
||||
|
||||
Fehler gehen unabhaengig vom Erfolgstopic nach `homelab-alerts`, damit Restore-Probleme auf dem gleichen Handy-Topic landen wie Prometheus-, Docker-, Borg- und Posture-Alarme.
|
||||
|
||||
Verwendete Hilfsskripte:
|
||||
## Verwendete Hilfsskripte
|
||||
|
||||
- `ops/restore-tests/send-ntfy.sh`
|
||||
- `ops/restore-tests/run-restore-job-with-ntfy.sh`
|
||||
- `ops/restore-tests/run-restore-checks.sh`
|
||||
|
||||
Reference in New Issue
Block a user