Skip to content

Database Failover Runbook

Severity: P1 — Critical

Triggers

  • CloudSQL instance unreachable
  • Connection pool exhaustion
  • Replication lag > 60s

Steps

1. Check CloudSQL Status

1
gcloud sql instances describe zashita-postgres-production --project=viktor-integration

2. Check Connection via Proxy

1
2
3
4
# On dev-server
psql -h 127.0.0.1 -p 5432 -U app_user -d zcity -c "SELECT 1;"
# On citizen-107
psql -h 127.0.0.1 -p 5432 -U app_user -d zcity -c "SELECT 1;"

3. Restart CloudSQL Proxy

1
2
3
4
5
# Docker proxy on dev-server
ssh dev-server "docker restart monorepo-cloudsql-proxy"

# Systemd proxy on citizen-107
sudo systemctl restart cloud-sql-proxy

4. If Instance Down

1
2
3
# Point-in-time recovery (last 7 days)
gcloud sql instances clone zashita-postgres-production zashita-postgres-recovery \
  --point-in-time="2026-02-26T00:00:00.000Z"

Escalation

  • Contact GCP support for CloudSQL infrastructure issues