EPCC Implementation Plan: GAP-PerfTest¶

┌─────────────────────────────────────────────────────────────────┐
│                    LOAD TESTING ARCHITECTURE                     │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐       │
│  │   Load Gen   │    │   Guardrail  │    │   Metrics    │       │
│  │   (Locust)   │───▶│   Service    │───▶│  Collector   │       │
│  │              │    │              │    │ (CloudWatch) │       │
│  └──────────────┘    └──────────────┘    └──────┬───────┘       │
│         │                                        │               │
│         │            ┌───────────────────────────┘               │
│         ▼            ▼                                           │
│  ┌─────────────────────────────────────────────────────────┐    │
│  │                    METRICS DASHBOARD                     │    │
│  │  ┌─────────┐  ┌─────────┐  ┌─────────┐  ┌─────────┐     │    │
│  │  │ p50 lat │  │ p95 lat │  │ p99 lat │  │   TPS   │     │    │
│  │  │  <100ms │  │  <500ms │  │ <1000ms │  │  >100/s │     │    │
│  │  └─────────┘  └─────────┘  └─────────┘  └─────────┘     │    │
│  └─────────────────────────────────────────────────────────┘    │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

"""
Guardrail Service Load Test
Issue #2: GAP-PerfTest
"""

from locust import HttpUser, task, between
import json
import random

# Sample payloads representing realistic proposals
SAMPLE_PAYLOADS = [
    {
        "proposal_id": "test-001",
        "risk_base": 0.3,
        "profit_base": 100000,
        "risk_prop": 0.35,
        "profit_prop": 120000,
        "novelty_score": 0.6,
        "complexity_score": 0.7,
        "quality_score": 0.8
    },
    # Add more representative payloads...
]


class GuardrailUser(HttpUser):
    """Simulates user submitting proposals for guardrail evaluation."""

    wait_time = between(0.1, 0.5)  # 2-10 requests per second per user

    @task(10)
    def evaluate_proposal(self):
        """Primary task: evaluate a proposal."""
        payload = random.choice(SAMPLE_PAYLOADS)
        payload["proposal_id"] = f"load-test-{random.randint(1, 1000000)}"

        with self.client.post(
            "/evaluate",
            json=payload,
            catch_response=True
        ) as response:
            if response.status_code == 200:
                result = response.json()
                if "guardrail_decision" in result:
                    response.success()
                else:
                    response.failure("Missing guardrail_decision in response")
            else:
                response.failure(f"Status code: {response.status_code}")

    @task(1)
    def health_check(self):
        """Occasional health check."""
        self.client.get("/health")


class StressUser(HttpUser):
    """High-frequency user for stress testing."""

    wait_time = between(0.01, 0.05)  # 20-100 requests per second per user

    @task
    def rapid_evaluate(self):
        """Rapid-fire evaluations."""
        payload = random.choice(SAMPLE_PAYLOADS)
        self.client.post("/evaluate", json=payload)

{
  "widgets": [
    {
      "type": "metric",
      "properties": {
        "title": "Response Time Percentiles",
        "metrics": [
          ["Guardrail", "Latency", "Percentile", "p50"],
          ["Guardrail", "Latency", "Percentile", "p95"],
          ["Guardrail", "Latency", "Percentile", "p99"]
        ],
        "period": 60,
        "stat": "Average",
        "annotations": {
          "horizontal": [
            { "value": 500, "label": "p95 Target", "color": "#ff0000" }
          ]
        }
      }
    },
    {
      "type": "metric",
      "properties": {
        "title": "Throughput (TPS)",
        "metrics": [
          ["Guardrail", "RequestCount", "Service", "Evaluator"]
        ],
        "period": 60,
        "stat": "Sum"
      }
    },
    {
      "type": "metric",
      "properties": {
        "title": "Error Rate",
        "metrics": [
          ["Guardrail", "ErrorCount", "Service", "Evaluator"]
        ],
        "period": 60,
        "stat": "Sum"
      }
    }
  ]
}

# Guardrail Service Performance Report

**Test Date**: 2026-01-15 (example — replace with actual test date)
**Environment**: [Staging/Production]
**Service Version**: [version]

## Summary

| Metric | Target | Achieved | Status |
|--------|--------|----------|--------|
| p95 Latency | < 500 ms | ___ ms | ✅/❌ |
| p99 Latency | < 1000 ms | ___ ms | ✅/❌ |
| Max Throughput | > 100 RPS | ___ RPS | ✅/❌ |
| Error Rate | < 0.1% | ___% | ✅/❌ |

## Test Scenarios

### Baseline (10 RPS)
- Duration: 10 minutes
- Results: [table of metrics]

### Scale (100 RPS)
- Duration: 30 minutes
- Results: [table of metrics]

### Stress (200 RPS)
- Duration: 30 minutes
- Results: [table of metrics]

### Soak (100 RPS, 4 hours)
- Duration: 4 hours
- Results: [table of metrics]

## Bottlenecks Identified

1. [Description of bottleneck]
   - Impact: [latency impact]
   - Resolution: [fix applied]

## Recommendations

1. [Optimization recommendation]
2. [Scaling recommendation]

## Appendix: Raw Data

[Link to detailed metrics export]

# Start Locust with web UI
locust -f load_test.py --host=https://aegis-staging.acme-corp.test

# Headless baseline test
locust -f load_test.py \
  --host=https://aegis-staging.acme-corp.test \
  --headless \
  --users 10 \
  --spawn-rate 1 \
  --run-time 10m \
  --csv=baseline

# Headless scale test
locust -f load_test.py \
  --host=https://aegis-staging.acme-corp.test \
  --headless \
  --users 100 \
  --spawn-rate 10 \
  --run-time 30m \
  --csv=scale

# GitHub Actions workflow for performance regression
name: Performance Regression Test

on:
  push:
    branches: [main]
  schedule:
    - cron: '0 2 * * 1'  # Weekly Monday 2 AM

jobs:
  perf-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Run Performance Test
        run: |
          pip install locust
          locust -f tests/perf/load_test.py \
            --host=${{ secrets.STAGING_URL }} \
            --headless \
            --users 50 \
            --spawn-rate 5 \
            --run-time 5m \
            --csv=results

      - name: Check p95 Latency
        run: |
          P95=$(tail -1 results_stats.csv | cut -d',' -f8)
          if [ "$P95" -gt 500 ]; then
            echo "p95 latency $P95 ms exceeds 500 ms target"
            exit 1
          fi