Add daily operations report with hardened log-noise filtering

Brings the previously untracked daily-status-report.sh and
send-operations-report-mail.sh into the repo, plus a refactor of the
log-noise pipeline:

- New helper services/posture-check/lib/normalize-noise-patterns.sh
  strips comments, empty lines and trailing whitespace from
  log-noise.patterns before grep -f sees it. A stray empty line in
  the pattern file would otherwise have made grep -Eaif match every
  hit and silently wipe the log highlights.
- log-noise.patterns is now documented per-pattern (Why / Re-check).
  The Vaultwarden pattern is split: token/session noise stays as
  noise; DNS/Connect/Resolve/reqwest/hyper errors are removed from
  the noise set so real network signals stay visible.
- collect_log_highlights now reports a per-container and per-pattern
  noise breakdown (Top N) and an escalation flag when any pattern
  exceeds NOISE_ESCALATION_THRESHOLD (default 500). The flag is fed
  into derive_report_status and the management summary.
- New shell tests under services/posture-check/tests/ verify the
  normalize helper handles comments, empty lines, whitespace-only
  lines, and that unknown error lines remain in the attention set.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
2026-05-23 10:41:33 +02:00
parent b7cbbe51de
commit 9e7bebbd3c
5 changed files with 2026 additions and 0 deletions
@@ -0,0 +1,27 @@
#!/usr/bin/env bash
# normalize-noise-patterns.sh
#
# Read a log-noise.patterns file and emit a normalized stream of patterns
# that is safe to feed into `grep -Eaif`.
#
# Behaviour:
# - Lines starting with `#` (after optional leading whitespace) are dropped.
# - Empty / whitespace-only lines are dropped.
# - Leading and trailing whitespace is trimmed from each pattern.
# - Patterns that become empty after trimming are dropped.
#
# Why this exists:
# A single empty / whitespace-only line in the input file would make
# `grep -Eaif` match every input line, silently wiping the entire log
# highlights signal. Always pipe patterns through this normalizer first.
#
# Usage:
# normalize-noise-patterns.sh <file>
# cat patterns | normalize-noise-patterns.sh
set -euo pipefail
src="${1:-/dev/stdin}"
grep -Ev '^[[:space:]]*(#|$)' "$src" \
| sed -E 's/^[[:space:]]+//; s/[[:space:]]+$//' \
| grep -v '^$' || true