Skip to content

GitLab Restore Runbook

Severity: P1 — Critical

Triggers

  • GitLab container crash loop
  • Database corruption
  • Accidental data deletion

Prerequisites

  • SSH access to dev-server (10.128.0.14)
  • CloudSQL admin access

Steps

1. Check GitLab Health

1
2
ssh dev-server "docker exec gitlab gitlab-ctl status"
ssh dev-server "curl -s https://git.zashita.io/-/health"

2. Check Recent Backups

1
ssh dev-server "ls -la /var/opt/gitlab/backups/ | tail -5"

3. Restore from Backup

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# Stop GitLab processes (keep Postgres/Redis)
ssh dev-server "docker exec gitlab gitlab-ctl stop puma sidekiq"

# Restore from latest backup
ssh dev-server "docker exec gitlab gitlab-backup restore BACKUP=<timestamp>"

# Restart
ssh dev-server "docker restart gitlab"

# Verify
ssh dev-server "curl -s https://git.zashita.io/-/health"

4. Verify Data Integrity

1
ssh dev-server "docker exec gitlab gitlab-rake gitlab:check SANITIZE=true"

Escalation

  • P1: @commander via Telegram immediately
  • If restore fails: check CloudSQL backups (4-hour window)