Sparki Technical Architecture Document
:::info readme This document provides the comprehensive technical architecture for the Sparki platform, detailing the design decisions, technology choices, deployment patterns, and scalability strategies that enable Sparki to deliver zero-configuration CI/CD at massive scale. The architecture is designed around core principles of performance, reliability, scalability, and developer experience. ::: Version: 1.0Date: December 3, 2025
Architecture Owner: Chief Technology Officer
Last Updated: December 3, 2025
1. Architecture Overview
1.1 System Architecture Diagram
1.2 Layered Architecture
2. Core Components
2.1 Sparki API (Go Fiber)
Purpose: Central REST API providing project management, pipeline orchestration, and CI/CD operations. Technology Stack:- Framework: Go Fiber v2 (ultra-high-performance HTTP framework)
- Runtime: Go 1.22+ with goroutine-based concurrency
- Database Driver: pgx for PostgreSQL
- Caching: go-redis/redis
- JSON Parsing: encoding/json (stdlib optimized)
- Error Handling: Custom error types with structured context
- Zero-allocation JSON parsing
- Minimal garbage collection pressure
- Sub-50ms p95 response times at scale
- 100K+ concurrent connections per instance
- Horizontal scaling without session affinity
2.2 Loco Deployment Engine (Rust Axum/Tokio)
Purpose: Intelligent deployment orchestration with validation, health checks, and rollback capabilities. Technology Stack:- Framework: Rust Axum web framework
- Async Runtime: Tokio (industry-leading async runtime)
- HTTP Client: reqwest for cloud platform APIs
- Database: sqlx for async PostgreSQL queries
- Caching: redis crate for Redis integration
- Serialization: serde/serde_json for JSON
- Error Handling: anyhow/thiserror for result chains
- Lock-free async with zero-copy operations
- Memory-safe concurrency without garbage collection
- Sub-millisecond latencies for deployment operations
- Predictable performance under load
- Automatic resource cleanup via RAII pattern
2.3 Terminal UI (Go Bubbletea)
Purpose: Delightful, keyboard-centric terminal interface for CI/CD management. Technology Stack:- Framework: Charmbracelet Bubbletea
- Terminal Rendering: tcell for cross-platform terminal support
- Colors: charmbracelet/lipgloss for styling
- Input Handling: Custom keybind mapping
- WebSocket Client: gorilla/websocket for real-time updates
- Graphics: ASCII art and Unicode support
2.4 Shield Authentication Service (Django)
Purpose: Identity and access management for users, workspaces, and RBAC. Technology Stack:- Framework: Django 4.2+
- Database: PostgreSQL via psycopg2
- Caching: Redis via django-redis
- Password Hashing: bcrypt
- JWT: djangorestframework-simplejwt
- OAuth/SAML: django-allauth, python3-saml
- Async Tasks: Celery for async operations
3. Data Architecture
3.1 PostgreSQL Schema
Core Tables:3.2 Redis Cache Strategy
Cache Keys Pattern:- User profiles: 1 hour
- Permissions: 15 minutes
- Pipelines: 5 minutes
- Build logs: Streaming (no TTL)
- Deployments: 10 minutes
- Sessions: 24 hours
3.3 Object Storage (S3/MinIO)
Bucket Structure:4. Storage Integration Strategy
4.1 Fiber Storage Adapter Selection
Primary Choices for Sparki:| Adapter | Use Case | Why Selected |
|---|---|---|
| PostgreSQL | Primary database | ACID compliance, scalability, proven reliability |
| Redis | Caching layer | Sub-millisecond performance, native data structures |
| S3 | Build artifacts | Scalability, durability, cost efficiency |
| Badger | Embedded state (optional) | Fast embedded key-value for local caching |
| SurrealDB | Future real-time DB | Multi-platform, time-series for metrics |
4.2 Storage Adapter Implementation
5. Observability Architecture
5.1 Logging Stack
Three-Tier Logging Architecture:5.2 Metrics Stack
Prometheus Metrics:- Grafana dashboards for real-time monitoring
- Custom dashboards for pipeline/deployment metrics
- Alert rules for performance/reliability thresholds
5.3 Distributed Tracing
Jaeger Integration:- Correlation IDs across all requests
- Span tracking through service calls
- Performance analysis per service
6. Deployment Architecture
6.1 Kubernetes Deployment
Service Topology:| Service | CPU | Memory | Replicas | Notes |
|---|---|---|---|---|
| Sparki API | 2000m | 2Gi | 3+ | Auto-scaling enabled |
| Loco Engine | 1000m | 1Gi | 2+ | Job queue backed |
| Shield Auth | 1000m | 1Gi | 2+ | Session stateful |
| PostgreSQL | 2000m | 4Gi | 1 master | HA via replication |
| Redis | 1000m | 2Gi | 3 nodes | Cluster mode |
6.2 High Availability Strategy
Service-Level Agreements:- API: 99.95% uptime SLA
- Loco: 99.95% uptime SLA
- Auth: 99.95% uptime SLA
- Multi-replica deployments with load balancing
- Database replication and automatic failover
- Redis cluster with sentinel monitoring
- Cross-region deployment option
7. Security Architecture
7.1 Authentication Flow
7.2 Permission Validation Flow
7.3 Encryption Strategy
- In Transit: TLS 1.3 for all external traffic
- At Rest: AES-256 for sensitive data in database
- Credentials: Bcrypt for password hashing, encrypted vault for API keys
8. Scalability Patterns
8.1 Horizontal Scaling
Stateless Services:- Sparki API: Scale horizontally without session affinity
- Loco Engine: Scale via job queue (no local state)
- Shield Auth: Scale via shared token store (Redis)
- PostgreSQL: Vertical scaling + read replicas
- Redis: Cluster mode for horizontal scaling
- Object Storage: Inherently scalable (S3)
8.2 Performance Optimization
Caching Strategy:- Framework detection results: 5 minutes
- Pipeline configurations: 5 minutes
- User permissions: 15 minutes
- Project metadata: 1 hour
- Indexed queries for frequent operations
- Connection pooling to reduce overhead
- Query optimization and monitoring
- Read replicas for scaling read-heavy workloads
9. Disaster Recovery
9.1 Backup Strategy
Database Backups:- Continuous replication to standby
- Daily snapshots to S3
- Weekly full backups for archives
- 30-day retention policy
- Version control for all pipeline configurations
- Immutable audit logs
- Configuration snapshots before changes
9.2 Recovery Procedures
RTO (Recovery Time Objective): <15 minutes RPO (Recovery Point Objective): <5 minutes Recovery Steps:- Automated failover to replica
- Restore from latest snapshot if needed
- Verify data integrity
- Resume operations
10. Compliance & Security
10.1 Data Privacy
- GDPR Compliance: Data subject rights, right to be forgotten
- HIPAA Compliance: Encryption, audit logging, access controls
- SOC 2 Type II: Annual audit, continuous controls
10.2 Audit Logging
All authentication, authorization, and data access events logged:- User: Who performed action
- Resource: What was affected
- Action: What happened
- Timestamp: When it occurred
- Result: Success/failure
- IP Address: Where from
Conclusion
Sparki’s technical architecture is designed around core principles of performance, reliability, and developer experience. The combination of Go Fiber for ultra-fast HTTP performance and Rust Axum for deterministic deployment orchestration positions Sparki as the highest-performance CI/CD platform available. The architecture supports millions of concurrent developers, delivers sub-50ms API response times, and provides 99.95% availability required for mission-critical production workloads.Document History:
| Version | Date | Author | Changes |
|---|---|---|---|
| 1.0 | 2025-12-03 | Sparki Engineering | Initial technical architecture |