# Alert System Setup Instructions

## Quick Start (5 minutes)

### Step 1: Execute Database Migrations

```bash
# Make script executable
chmod +x scripts/execute-alert-migrations.sh

# Run migration script
./scripts/execute-alert-migrations.sh
```

Or manually:
```bash
mysql -h $DB_HOST -u $DB_USER -p$DB_PASSWORD $DB_NAME < drizzle/migrations/alert_system.sql
```

### Step 2: Configure Credentials

Navigate to **Settings → Secrets** and add:

#### Slack Integration
1. Go to https://api.slack.com/apps
2. Create New App → From scratch
3. Name: "CoinKrazy Alerts"
4. Go to "Incoming Webhooks" → Add New Webhook to Workspace
5. Select channel (e.g., #alerts)
6. Copy webhook URL
7. Add to Settings → Secrets:
   ```
   SLACK_WEBHOOK_URL=https://hooks.slack.com/services/YOUR/WEBHOOK/URL
   ```

#### PagerDuty Integration
1. Go to https://pagerduty.com
2. Create Service → Integration
3. Select "Events API v2"
4. Copy Integration Key
5. Add to Settings → Secrets:
   ```
   PAGERDUTY_INTEGRATION_KEY=your-integration-key
   ```

#### Email Configuration
1. Go to https://app.brevo.com
2. Settings → API Keys → Create API Key
3. Copy API Key
4. Add to Settings → Secrets:
   ```
   ALERT_EMAIL_RECIPIENTS=admin@example.com,ops@example.com
   BREVO_API_KEY=your-brevo-api-key
   ```

### Step 3: Test Alert Delivery

1. Navigate to **Admin → Monitoring Dashboard**
2. Click **"Check Metrics Now"** button
3. Verify alerts appear in:
   - Slack channel
   - PagerDuty incidents
   - Email inbox

## Detailed Configuration

### Database Tables Created

```sql
-- Alert Logs (stores all alerts)
CREATE TABLE alert_logs (
  id TEXT PRIMARY KEY,
  alert_type TEXT NOT NULL,
  severity TEXT NOT NULL,
  message TEXT NOT NULL,
  status TEXT NOT NULL,
  team TEXT,
  escalation_count INTEGER DEFAULT 0,
  acknowledged_at INTEGER,
  resolved_at INTEGER,
  created_at INTEGER NOT NULL,
  updated_at INTEGER NOT NULL
);

-- Alert Templates (message templates for each channel)
CREATE TABLE alert_templates (
  id TEXT PRIMARY KEY,
  alert_type TEXT NOT NULL UNIQUE,
  title TEXT NOT NULL,
  slack_template TEXT NOT NULL,
  email_template TEXT NOT NULL,
  pagerduty_template TEXT NOT NULL,
  severity TEXT NOT NULL,
  enabled INTEGER DEFAULT 1,
  created_at INTEGER NOT NULL,
  updated_at INTEGER NOT NULL
);

-- Alert Delivery Logs (tracks delivery to each channel)
CREATE TABLE alert_delivery_logs (
  id TEXT PRIMARY KEY,
  alert_id TEXT NOT NULL,
  channel TEXT NOT NULL,
  status TEXT NOT NULL,
  recipient TEXT,
  error TEXT,
  retry_count INTEGER DEFAULT 0,
  sent_at INTEGER,
  created_at INTEGER NOT NULL,
  FOREIGN KEY (alert_id) REFERENCES alert_logs(id)
);

-- Alert Template Usage (tracks template effectiveness)
CREATE TABLE alert_template_usage (
  id TEXT PRIMARY KEY,
  template_id TEXT NOT NULL,
  alert_id TEXT NOT NULL,
  used_at INTEGER NOT NULL,
  delivery_success INTEGER,
  acknowledged_within INTEGER,
  resolved_within INTEGER,
  FOREIGN KEY (template_id) REFERENCES alert_templates(id),
  FOREIGN KEY (alert_id) REFERENCES alert_logs(id)
);
```

### Default Alert Templates

Four default templates are automatically created:

1. **Critical Latency Alert**
   - Triggers: WebSocket latency > 500ms
   - Severity: Critical
   - Channels: Slack, Email, PagerDuty

2. **Delivery Failure Alert**
   - Triggers: Email/SMS success rate < 90%
   - Severity: Warning
   - Channels: Slack, Email, PagerDuty

3. **Forecast Accuracy Alert**
   - Triggers: Revenue forecast accuracy < 85%
   - Severity: Warning
   - Channels: Slack, Email, PagerDuty

4. **Connection Drop Alert**
   - Triggers: Active WebSocket connections < 100
   - Severity: Info
   - Channels: Slack, Email, PagerDuty

## Monitoring Dashboard

### Real-time Metrics
- **Active Alerts**: Current number of unresolved alerts
- **Critical**: Count of critical severity alerts
- **Warnings**: Count of warning severity alerts
- **Monitoring**: System monitoring status (ACTIVE/INACTIVE)

### Alert Management
- View all active alerts with details
- Acknowledge alerts (mark as seen)
- Resolve alerts (mark as fixed)
- Filter by severity, type, or status

### Threshold Configuration
- View current alert thresholds
- Update thresholds via Admin → Alert Thresholds
- Adjust severity levels
- Enable/disable specific alert types

## Testing Checklist

- [ ] Database tables created successfully
- [ ] Slack webhook URL configured and tested
- [ ] PagerDuty integration key configured and tested
- [ ] Email recipients and Brevo API key configured
- [ ] "Check Metrics Now" button triggers alerts
- [ ] Alerts appear in Slack channel
- [ ] Alerts appear in PagerDuty
- [ ] Alerts appear in email inbox
- [ ] Monitoring Dashboard displays active alerts
- [ ] Alert acknowledgment works
- [ ] Alert resolution works

## Troubleshooting

### Alerts Not Appearing

**Check 1: Database Tables**
```sql
SHOW TABLES LIKE 'alert%';
SELECT COUNT(*) FROM alert_templates;
```

**Check 2: Monitoring Status**
- Navigate to Admin → Monitoring Dashboard
- Verify "Monitoring" status shows "ACTIVE"

**Check 3: Credentials**
- Verify all environment variables are set
- Test each channel individually via Admin → Webhook Configuration

### Delivery Failures

**Slack Issues**
- Verify webhook URL is correct
- Check Slack workspace permissions
- Test webhook: Admin → Webhook Configuration → Test Slack

**PagerDuty Issues**
- Verify integration key is correct
- Check PagerDuty service is active
- Test integration: Admin → Webhook Configuration → Test PagerDuty

**Email Issues**
- Verify Brevo API key is correct
- Check email recipients list
- Verify Brevo account has email credits
- Test email: Admin → Webhook Configuration → Test Email

### Performance Issues

If alert processing is slow:
1. Check database indexes: `SHOW INDEX FROM alert_logs;`
2. Archive old alerts: `DELETE FROM alert_logs WHERE created_at < DATE_SUB(NOW(), INTERVAL 90 DAY);`
3. Increase monitoring interval: `ALERT_CHECK_INTERVAL=120000`

## Next Steps

1. **Customize Thresholds**
   - Adjust alert thresholds based on your metrics
   - Start conservative, then lower gradually

2. **Set Up Escalations**
   - Configure escalation policies
   - Define runbooks for each alert type

3. **Monitor Effectiveness**
   - Review alert history weekly
   - Track false positive rate
   - Adjust thresholds as needed

4. **Integrate with Incident Management**
   - Connect to your incident response process
   - Set up on-call rotations
   - Define SLA targets

## Support Resources

- Alert System Config: `ALERT_SYSTEM_CONFIG.md`
- API Reference: `server/routers/alertTriggers.ts`
- Database Schema: `drizzle/migrations/alert_system.sql`
- Monitoring Dashboard: `client/src/pages/admin/MonitoringDashboardIntegrated.tsx`

For issues, check:
1. `.manus-logs/devserver.log` for server errors
2. `.manus-logs/browserConsole.log` for frontend errors
3. Admin → Alert History for delivery logs