# Alert System Testing Guide

## Pre-Testing Checklist

- [ ] Database migrations executed successfully
- [ ] All environment variables configured (SLACK_WEBHOOK_URL, PAGERDUTY_INTEGRATION_KEY, etc.)
- [ ] Dev server running (`pnpm dev`)
- [ ] Admin account logged in
- [ ] Monitoring Dashboard accessible at `/admin/monitoring`

## Test 1: Database Verification

### Verify Tables Exist
```bash
mysql -h $DB_HOST -u $DB_USER -p$DB_PASSWORD $DB_NAME << EOF
SHOW TABLES LIKE 'alert%';
SELECT COUNT(*) as template_count FROM alert_templates;
SELECT COUNT(*) as alert_count FROM alert_logs;
SELECT COUNT(*) as delivery_count FROM alert_delivery_logs;
EOF
```

**Expected Output:**
- 4 tables: alert_logs, alert_templates, alert_delivery_logs, alert_template_usage
- template_count: 4 (default templates)
- alert_count: 0 (no alerts yet)
- delivery_count: 0 (no deliveries yet)

## Test 2: Webhook Configuration

### Test Slack Webhook

1. Navigate to **Admin → Webhook Configuration**
2. Paste Slack webhook URL in "Slack Webhook URL" field
3. Click **"Test Slack Webhook"** button
4. **Expected Result**: 
   - Success message appears
   - Test message appears in Slack channel with format:
     ```
     🧪 Test Alert from CoinKrazy
     This is a test message to verify Slack integration
     ```

### Test PagerDuty Integration

1. Navigate to **Admin → Webhook Configuration**
2. Paste PagerDuty integration key in "PagerDuty Integration Key" field
3. Click **"Test PagerDuty Integration"** button
4. **Expected Result**:
   - Success message appears
   - Test incident appears in PagerDuty with title "Test Alert from CoinKrazy"

### Test Email Delivery

1. Navigate to **Admin → Webhook Configuration**
2. Verify email recipients list
3. Click **"Test Email Delivery"** button
4. **Expected Result**:
   - Success message appears
   - Test email arrives in configured inboxes with subject "Test Alert from CoinKrazy"

## Test 3: Metric Monitoring

### Trigger Alert Manually

1. Navigate to **Admin → Monitoring Dashboard**
2. Click **"Check Metrics Now"** button
3. **Expected Result**:
   - Dashboard shows "Checking metrics..." briefly
   - Active alerts count may increase
   - New alerts appear in the "Active Alerts" section

### Verify Alert Delivery

1. Check **Slack channel** for alert message
2. Check **PagerDuty** for incident
3. Check **Email inbox** for alert notification
4. **Expected Result**: Alert appears in all configured channels within 5 seconds

## Test 4: Alert Lifecycle

### Acknowledge Alert

1. In Monitoring Dashboard, find an active alert
2. Click **"Acknowledge"** button
3. **Expected Result**:
   - Alert status changes to "acknowledged"
   - Button changes to show acknowledgment time

### Resolve Alert

1. In Monitoring Dashboard, find an acknowledged alert
2. Click **"Resolve"** button
3. **Expected Result**:
   - Alert is removed from active list
   - Alert appears in Alert History with status "resolved"
   - Slack/PagerDuty shows resolution

## Test 5: Threshold Management

### View Current Thresholds

1. Navigate to **Admin → Monitoring Dashboard**
2. Scroll to "Alert Thresholds" section
3. **Expected Result**: See all configured thresholds:
   - latency_critical: websocket_latency > 500
   - latency_warning: websocket_latency > 300
   - delivery_failure: delivery_success_rate < 0.9
   - forecast_accuracy_degrading: forecast_accuracy < 0.85
   - connection_drop: active_connections < 100

### Update Threshold

1. Navigate to **Admin → Alert Thresholds** (if available)
2. Update latency_critical threshold to 1000
3. Click **"Save"**
4. **Expected Result**:
   - Threshold updated in database
   - Monitoring Dashboard reflects new value
   - Alerts only trigger when new threshold is exceeded

## Test 6: Real-time Updates

### Verify Socket.io Connection

1. Open browser DevTools (F12)
2. Go to Network tab
3. Filter for "WebSocket"
4. Refresh Monitoring Dashboard
5. **Expected Result**: WebSocket connection established to activity-feed namespace

### Test Real-time Alert

1. Keep Monitoring Dashboard open
2. Trigger alert via "Check Metrics Now"
3. **Expected Result**: Alert appears in real-time without page refresh

## Test 7: Alert History

### Search Alerts

1. Navigate to **Admin → Alert History**
2. Enter search term (e.g., "latency")
3. Click **"Search"**
4. **Expected Result**: Only matching alerts appear

### Filter by Severity

1. In Alert History, click severity filter
2. Select "Critical"
3. **Expected Result**: Only critical alerts displayed

### Filter by Status

1. In Alert History, click status filter
2. Select "Resolved"
3. **Expected Result**: Only resolved alerts displayed

### Export Alerts

1. In Alert History, click **"Export as CSV"**
2. **Expected Result**: CSV file downloads with all alert data

## Test 8: Performance

### Load Testing

1. Trigger 10+ alerts rapidly
2. Monitor Monitoring Dashboard performance
3. **Expected Result**:
   - Dashboard remains responsive
   - All alerts display correctly
   - No performance degradation

### Database Performance

```bash
# Check alert table size
mysql -h $DB_HOST -u $DB_USER -p$DB_PASSWORD $DB_NAME << EOF
SELECT 
  TABLE_NAME,
  ROUND(((DATA_LENGTH + INDEX_LENGTH) / 1024 / 1024), 2) AS SIZE_MB
FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME LIKE 'alert%';
EOF
```

**Expected Result**: Tables remain small (< 10MB) with proper indexing

## Test 9: Error Handling

### Test Invalid Webhook URL

1. Enter invalid Slack webhook URL
2. Click "Test Slack Webhook"
3. **Expected Result**: Error message displayed, no crash

### Test Missing Credentials

1. Remove BREVO_API_KEY from environment
2. Trigger alert
3. **Expected Result**: Email delivery fails gracefully, alert still created

### Test Database Connection Loss

1. Stop database server
2. Try to trigger alert
3. **Expected Result**: Error logged, alert not created, system recovers when DB comes back

## Test 10: Integration

### End-to-End Flow

1. **Setup**: Verify all credentials configured
2. **Trigger**: Click "Check Metrics Now"
3. **Deliver**: Verify alert in Slack, PagerDuty, Email
4. **Acknowledge**: Click acknowledge in Dashboard
5. **Resolve**: Click resolve in Dashboard
6. **Verify**: Alert appears in History with correct status
7. **Export**: Export alerts as CSV

**Expected Result**: Complete flow works without errors

## Performance Benchmarks

| Operation | Expected Time | Threshold |
|-----------|---------------|-----------|
| Metric check | < 1s | 5s |
| Alert creation | < 100ms | 500ms |
| Slack delivery | < 2s | 10s |
| PagerDuty delivery | < 2s | 10s |
| Email delivery | < 3s | 15s |
| Dashboard load | < 500ms | 2s |
| Alert acknowledge | < 100ms | 500ms |
| Alert resolve | < 100ms | 500ms |

## Troubleshooting Common Issues

### Alerts Not Triggering
1. Check metrics are being collected
2. Verify thresholds are configured
3. Check monitoring status is ACTIVE
4. Review logs: `.manus-logs/devserver.log`

### Delivery Failures
1. Test webhook directly: Admin → Webhook Configuration
2. Verify credentials in Settings → Secrets
3. Check network connectivity
4. Review delivery logs in Alert History

### Dashboard Not Updating
1. Verify Socket.io connection in DevTools
2. Check browser console for errors
3. Refresh page
4. Clear browser cache

### Performance Issues
1. Check database indexes
2. Archive old alerts (> 90 days)
3. Increase monitoring interval
4. Check server resources

## Sign-off Checklist

- [ ] All 10 tests completed successfully
- [ ] No errors in browser console
- [ ] No errors in server logs
- [ ] All performance benchmarks met
- [ ] Alerts deliver to all channels
- [ ] Dashboard updates in real-time
- [ ] Alert history searchable and filterable
- [ ] Export functionality works
- [ ] Error handling works gracefully
- [ ] System recovers from failures

## Next Steps After Testing

1. **Deploy to Production**
   - Execute migrations on production database
   - Configure production credentials
   - Set up monitoring and alerting

2. **Monitor Effectiveness**
   - Track alert accuracy
   - Monitor delivery success rates
   - Adjust thresholds based on data

3. **Optimize Performance**
   - Archive old alerts regularly
   - Monitor database growth
   - Adjust monitoring intervals

4. **Continuous Improvement**
   - Gather team feedback
   - Refine alert templates
   - Update escalation policies