┌──────────────────────────────────────────────────────┐
│       Loco Deployment Engine (Rust Axum)            │
├──────────────────────────────────────────────────────┤
│  Deployment Orchestrator                            │
│  ├─ Pre-Deployment Validator                        │
│  ├─ Stage Manager (dev → staging → prod)            │
│  ├─ Script Executor (shell, Go, Python)             │
│  ├─ Health Check Manager                            │
│  └─ Rollback Coordinator                            │
├──────────────────────────────────────────────────────┤
│  Cloud Platform Adapters                            │
│  ├─ Railway Adapter                                 │
│  ├─ Render Adapter                                  │
│  ├─ Fly.io Adapter                                  │
│  └─ Vercel Adapter                                  │
├──────────────────────────────────────────────────────┤
│  Observability & Reliability                        │
│  ├─ Event Logging (PostgreSQL)                      │
│  ├─ Metrics Exporter (Prometheus)                   │
│  ├─ Deployment Tracing (Jaeger)                     │
│  └─ Alert Manager Integration                       │
├──────────────────────────────────────────────────────┤
│  Async Runtime (Tokio)                              │
│  └─ High concurrency, sub-millisecond latencies     │
└──────────────────────────────────────────────────────┘

Railway:
  - Native project API integration
  - Environment variable management
  - Custom domain support
  - PostgreSQL/Redis provisioning

Render:
  - Blueprint-based deployment
  - Environment variable inheritance
  - Custom domains via CNAME
  - Service discovery

Fly.io:
  - Apps platform deployment
  - Multi-region support
  - Autoscaling configuration
  - Health checks via fly.io native

Vercel:
  - Serverless functions
  - Edge function deployment
  - Environment variable management
  - Preview deployments for PRs

Blue-Green:
  - Deploy to inactive environment
  - Run health checks
  - Switch traffic instantly
  - Zero downtime but requires 2x resources

Canary:
  - Deploy to 5-10% of traffic initially
  - Monitor metrics for anomalies
  - Gradually increase traffic (e.g., 10%, 50%, 100%)
  - Automatic rollback on error threshold

Rolling:
  - Deploy instance by instance
  - Maintain service availability
  - Slower than blue-green
  - Resource efficient

1. SAST (Static Application Security Testing)
   - Code scanning for common vulnerabilities
   - Framework-specific vulnerability detection
   - Configuration security analysis

2. Dependency Scanning
   - Vulnerability database checks
   - License compliance verification
   - Outdated dependency detection

3. Secrets Detection
   - AWS keys, GitHub tokens, API keys
   - Database credentials
   - Certificate detection

4. Compliance Checks
   - GDPR data handling compliance
   - PCI DSS for payment systems
   - HIPAA for healthcare data

Build Artifact
     ↓
[Pre-Deployment Validation]
  ✓ Artifact integrity
  ✓ Security scanning
  ✓ Configuration validation
     ↓
Development Stage
  • Auto-deploy
  • Run smoke tests
  • Monitor for 5 minutes
     ↓
Staging Stage
  ✓ Requires approval (auto-approved if dev stable)
  • Run full integration tests
  • Monitor for 15 minutes
  • Load testing (optional)
     ↓
Production Stage
  ✓ Requires explicit approval
  • Deploy with selected strategy
  • Health checks (5 min)
  • Monitor (15 min)
  • Automatic rollback if issues detected

Pre-Deployment Scripts (Pre-Deploy):
  • Database migrations
  • Configuration transforms
  • Data cleanup
  • Dependency installation

Post-Deployment Scripts (Post-Deploy):
  • Health check verification
  • Smoke tests
  • Cache warming
  • Monitoring setup

Rollback Scripts (On-Failure):
  • Database rollback
  • Cache invalidation
  • State restoration

package main

import (
    "log"
    "os"
)

func main() {
    // Get environment variables injected by Loco
    deploymentID := os.Getenv("DEPLOYMENT_ID")
    stage := os.Getenv("DEPLOYMENT_STAGE")
    appURL := os.Getenv("APP_URL")

    log.Printf("Running post-deployment for %s in %s", deploymentID, stage)

    // Custom deployment logic
    // - Run migrations
    // - Warm caches
    // - Verify health

    os.Exit(0) // Exit with 0 for success, non-zero for failure
}

HTTP Checks:
  GET /health → HTTP 200
  GET /api/status → {"status": "ok"}
  Timeout: 10 seconds
  Retries: 5 with exponential backoff

Database Checks:
  Connection test
  Query execution test (SELECT 1)
  Replication lag <1 second

Metrics Checks:
  CPU usage <80%
  Memory usage <85%
  Disk usage <90%
  Response time <500ms

Custom Checks:
  User-defined health check scripts
  External monitoring integration

Application Metrics:
  • Request latency (p50, p95, p99)
  • Error rate (4xx, 5xx, exceptions)
  • Request throughput
  • Database query duration

Resource Metrics:
  • CPU utilization
  • Memory usage
  • Disk I/O
  • Network I/O

Business Metrics:
  • Transaction success rate
  • Payment processing failures
  • User session failures

1. Receive rollback request
   ↓
2. Verify deployment history
   ↓
3. Select previous deployment (or specified version)
   ↓
4. Revert cloud platform to previous deployment
   ↓
5. Execute post-deployment validation
   ↓
6. Run health checks (strict mode)
   ↓
7. Monitor for 5 minutes (abbreviated window)
   ↓
8. Declare rollback complete or failed
   ↓
9. Send notifications to team
   ↓
10. Log incident with details

Critical (Immediate Rollback):
  - Health checks fail (3 consecutive failures)
  - Error rate >50%
  - Latency spike >5x
  - Out of memory errors
  - Database connection failures

High (Rollback with Approval):
  - Error rate >10% increase
  - Latency spike >2x increase
  - CPU spike >90%
  - Memory usage >85%

Configuration:
  - Per-stage auto-rollback thresholds
  - Team notification on auto-rollback
  - Manual override to prevent rollback

{
    "deployment_id": "dep_123456",
    "build_id": "build_789",
    "stage": "production",
    "status": "success",
    "strategy": "blue-green",
    "started_at": "2025-12-03T10:30:00Z",
    "completed_at": "2025-12-03T10:31:45Z",
    "duration_seconds": 105,
    "deployed_by": "alice@example.com",
    "approved_by": "bob@example.com",
    "metadata": {
        "instances_deployed": 3,
        "validation_passed": true,
        "health_checks_passed": true,
        "rollback_triggered": false
    }
}

{
    "timestamp": "2025-12-03T10:30:45.123Z",
    "level": "INFO",
    "component": "loco-deployment-engine",
    "correlation_id": "dep_req_123456",
    "deployment_id": "dep_123456",
    "event": "deployment_started",
    "stage": "production",
    "strategy": "blue-green",
    "user": "alice@example.com",
    "metadata": {
        "targets": 3,
        "platform": "railway",
        "estimated_duration": 120
    }
}

sparki_deployment_total (counter)
  Labels: stage, platform, strategy, status

sparki_deployment_duration_seconds (histogram)
  Labels: stage, platform, strategy
  Buckets: 10s, 30s, 60s, 120s, 300s, 600s, 1800s

sparki_deployment_health_check_duration_seconds (histogram)
  Labels: stage, check_type

sparki_deployment_validation_errors_total (counter)
  Labels: stage, error_type

sparki_deployment_rollback_total (counter)
  Labels: trigger_reason, automatic

// Adapter interface for cloud platforms
type CloudAdapter interface {
    // Deploy artifacts to platform
    Deploy(ctx context.Context, config DeployConfig) (*DeployResult, error)

    // Verify deployment health
    HealthCheck(ctx context.Context, deployment Deployment) (*HealthCheckResult, error)

    // Rollback to previous deployment
    Rollback(ctx context.Context, deployment Deployment) error

    // Get deployment status
    Status(ctx context.Context, deployment Deployment) (*DeploymentStatus, error)

    // Stream deployment logs
    StreamLogs(ctx context.Context, deployment Deployment) (<-chan string, error)
}

# Platform information
LOCO_PLATFORM=railway
LOCO_DEPLOYMENT_ID=dep_123456
LOCO_STAGE=production
LOCO_STRATEGY=blue-green

# Deployment information
LOCO_BUILD_ID=build_789
LOCO_COMMIT_SHA=abc1234567890def
LOCO_BRANCH=main
LOCO_BUILD_ARTIFACTS=/tmp/artifacts

# Application information
LOCO_APP_URL=https://api.example.com
LOCO_APP_NAME=api-server

# Team information
LOCO_DEPLOYED_BY=alice@example.com
LOCO_WORKSPACE_ID=ws_12345