Sparki Fiber Storage Adapters Integration Matrix

:::info overview This document provides a comprehensive matrix of all supported Fiber storage adapters, their characteristics, use cases within Sparki, and implementation guidance. It serves as a reference for architecture decisions and adapter selection. ::: Version: 1.0
Date: December 3, 2025
Document Owner: Infrastructure Team

Storage Adapter Selection Matrix

Primary Adapters (In Production)

Adapter	Type	Latency	Throughput	Durability	Use Case in Sparki	Priority
PostgreSQL	Relational DB	5-20ms	10K+ qps	99.99%	Primary data store (users, projects, pipelines, builds, deployments)	P0
Redis	In-Memory Cache	<1ms	100K+ ops/sec	99.9%	Session caching, permission cache, build logs, rate limiting	P0
S3 (AWS)	Object Storage	50-200ms	Unlimited	99.999999%	Build artifacts, deployment logs, long-term storage	P0
Badger	Embedded KV	1-5ms	100K+ ops/sec	99%	Local build state, temporary caching (edge deployments)	P1

Secondary Adapters (Extended Support)

Adapter	Type	Latency	Throughput	Use Case	Priority
MongoDB	Document DB	5-15ms	10K+ qps	Alternative: build metadata, pipeline configs	P2
SurrealDB	Multi-model	10-30ms	5K+ qps	Time-series metrics, real-time collaboration	P2
Memcache	Cache	<1ms	10K+ ops/sec	High-performance distributed caching	P2
Cassandra	Distributed DB	10-50ms	100K+ qps	Audit logs at massive scale	P3
ArangoDB	Graph DB	10-30ms	5K+ qps	Workflow dependency graphs, relationship queries	P3
MinIO	S3-Compatible	50-100ms	Unlimited	Self-hosted S3 alternative	P1

Supported But Not Primary

Adapter	Type	Rationale
MySQL	Relational	PostgreSQL preferred for performance
MSSQL	Relational	Enterprise option if required
DynamoDB	NoSQL	AWS-only, higher costs
Firestore	NoSQL	Google Cloud only
Couchbase	Document DB	Enterprise option
Neo4j	Graph DB	Specialized use case
Etcd	Key-Value	Kubernetes configuration only
LevelDB	Embedded KV	Single-instance limitation
Pebble	Embedded KV	Go-native alternative to Badger
Nats	Message Queue	Event streaming (future)
Clickhouse	Analytics DB	Time-series/analytics (future)
SurrealDB	Multi-model	Future real-time collaboration
Azure Blob	Object Storage	Multi-cloud option
Cloudflare KV	Distributed Cache	Edge caching (future)

Detailed Adapter Specifications

PostgreSQL

Classification: P0 Primary - Relational Database Fiber Adapter: storage/postgres Characteristics:

Latency: 5-20ms per query
Throughput: 10,000+ queries per second (per instance)
Concurrency: Connection pooling (20-100 connections)
Durability: ACID transactions, replication support
HA: Master-replica replication, automatic failover

Sparki Use Cases:

User Accounts: users table with authentication data
Workspaces & Teams: workspace/team management
Projects: Git repository metadata
Pipelines: Pipeline definitions and configurations
Builds: Build execution records
Deployments: Deployment history and status
Audit Logs: Immutable event logs
RBAC: Roles, permissions, memberships

Configuration:

// Fiber PostgreSQL adapter
func setupPostgres(ctx context.Context) fiber.Storage {
    dsn := fmt.Sprintf(
        "postgres://%s:%s@%s:%d/%s",
        os.Getenv("PG_USER"),
        os.Getenv("PG_PASSWORD"),
        os.Getenv("PG_HOST"),
        os.Getenv("PG_PORT"),
        os.Getenv("PG_DATABASE"),
    )

    return postgres.New(postgres.Config{
        DSN:            dsn,
        MaxConnections: 100,
        PreparedStatementsCacheSize: 512,
    })
}

Data Schema Highlights:

-- Projects table
CREATE TABLE projects (
    id UUID PRIMARY KEY,
    workspace_id UUID REFERENCES workspaces(id),
    name VARCHAR(255) NOT NULL,
    git_url VARCHAR(255) NOT NULL,
    framework VARCHAR(50),
    created_at TIMESTAMP DEFAULT NOW()
);

-- Pipelines table
CREATE TABLE pipelines (
    id UUID PRIMARY KEY,
    project_id UUID REFERENCES projects(id),
    configuration JSONB,
    version INT,
    created_at TIMESTAMP DEFAULT NOW()
);

-- Audit logs table
CREATE TABLE audit_logs (
    id UUID PRIMARY KEY,
    user_id UUID REFERENCES users(id),
    event_type VARCHAR(50),
    resource_type VARCHAR(50),
    details JSONB,
    created_at TIMESTAMP DEFAULT NOW()
);

-- Indexes
CREATE INDEX idx_projects_workspace ON projects(workspace_id);
CREATE INDEX idx_pipelines_project ON pipelines(project_id);
CREATE INDEX idx_audit_logs_timestamp ON audit_logs(created_at);

Performance Considerations:

Use connection pooling to maintain max 100 connections
Index frequently queried columns
Vacuum tables regularly to prevent bloat
Monitor query performance with pg_stat_statements
Consider read replicas for read-heavy workloads

Redis / Valkey

Classification: P0 Primary - In-Memory Cache Fiber Adapter: storage/redis or storage/valkey Characteristics:

Latency: <1ms (sub-millisecond)
Throughput: 100,000+ operations per second
Data Types: Strings, Lists, Sets, Hashes, Sorted Sets, Streams
Persistence: Optional (snapshots, AOF)
HA: Sentinel or Cluster mode

Sparki Use Cases:

Session Storage: User session tokens
Permission Cache: Role-based permissions (<1ms validation)
Build Logs: Real-time log streaming
Pipeline Status: Real-time pipeline state
Rate Limiting: Request throttling per user/IP
WebSocket Sessions: Active connections
Deployment Status: Real-time deployment progress
Temporary State: In-flight build/deployment data

Configuration:

// Fiber Redis adapter
func setupRedis(ctx context.Context) fiber.Storage {
    return redis.New(redis.Config{
        Host:        os.Getenv("REDIS_HOST"),
        Port:        6379,
        Password:    os.Getenv("REDIS_PASSWORD"),
        DB:          0,
        Reset:       false,
        MaxRetries:  3,
        PoolSize:    10 * runtime.NumCPU(),
    })
}

// Or Redis Cluster for scaling
func setupRedisCluster(ctx context.Context) fiber.Storage {
    return redis.New(redis.Config{
        Host: os.Getenv("REDIS_CLUSTER_NODES"), // comma-separated
        Cluster: true,
    })
}

Cache Key Design:

// Permission cache key pattern
const permissionCacheKey = "perm:%s:%s" // user_id:workspace_id

// Build log cache key pattern
const buildLogsCacheKey = "build:%s:logs" // build_id

// Rate limit key pattern
const rateLimitKey = "ratelimit:%s:%s" // user_id:endpoint

// Session key pattern
const sessionKey = "session:%s" // session_id

TTL Configuration:

User sessions:           24 hours
Permissions:             15 minutes
Build logs:              Streaming (no TTL)
Pipeline status:         5 minutes
Project metadata:        1 hour
Deployment status:       10 minutes
Rate limit counters:     1 minute
Temporary build state:   3 hours

Performance Considerations:

Use pipeline commands for batch operations
Implement proper TTL to prevent memory bloat
Use Cluster mode for horizontal scaling
Monitor memory usage and eviction policies
Consider Valkey for open-source alternative

S3 / Object Storage

Classification: P0 Primary - Object Storage Fiber Adapter: storage/minio (S3-compatible) Characteristics:

Latency: 50-200ms per request
Throughput: Unlimited
Durability: 99.999999% (AWS S3)
Scalability: Infinite
Cost: Pay-per-GB

Sparki Use Cases:

Build Artifacts: Compiled binaries, packages
Deployment Logs: Long-term log storage
Test Reports: JUnit XML, Coverage reports
Configuration Backups: Pipeline/deployment snapshots
Audit Log Archives: Long-term compliance storage
Build Cache: Container image layers, dependency caches

Configuration:

// AWS S3 adapter
func setupS3(ctx context.Context) *s3.Client {
    cfg, err := config.LoadDefaultConfig(ctx)
    if err != nil {
        log.Fatalf("Failed to load config: %v", err)
    }

    return s3.NewFromConfig(cfg)
}

// MinIO (S3-compatible) adapter
func setupMinIO(ctx context.Context) *minio.Client {
    return minio.New(os.Getenv("MINIO_ENDPOINT"), &minio.Options{
        Creds:  credentials.NewStaticV4(
            os.Getenv("MINIO_ACCESS_KEY"),
            os.Getenv("MINIO_SECRET_KEY"),
            "",
        ),
        Secure: true,
    })
}

Bucket Structure:

sparki-platform/
├── artifacts/
│   └── {project_id}/{build_id}/
│       ├── build.tar.gz
│       ├── docker-image.tar
│       └── test-report.json
├── logs/
│   ├── builds/{build_id}.log
│   └── deployments/{deployment_id}.log
├── backups/
│   └── {date}/
│       ├── database-dump.sql
│       └── config-snapshot.json
└── cache/
    └── {project_id}/
        ├── node_modules/
        └── go.mod.cache/

Performance Considerations:

Use multipart uploads for large files (>100MB)
Implement retry logic for transient failures
Consider S3 transfer acceleration
Use CloudFront CDN for frequent downloads
Archive old logs to Glacier for cost optimization

Badger (Embedded)

Classification: P1 Secondary - Embedded Key-Value Store Fiber Adapter: storage/badger Characteristics:

Latency: 1-5ms
Throughput: 100,000+ operations/second
Data Structure: Key-value pairs
Persistence: LSM tree based
Limitations: Single-instance only

Sparki Use Cases:

Edge Deployments: Local build state on deployment agents
Build Cache: Temporary build artifacts during pipeline
Fallback Storage: When Redis unavailable
Development Mode: Local testing without external storage

Configuration:

// Fiber Badger adapter
func setupBadger(ctx context.Context) fiber.Storage {
    return badger.New(badger.Config{
        Directory: "/data/badger",
        ValueDir:  "/data/badger",
        // Optional compression
        Compression: true,
        // Cleanup interval
        CleanupInterval: 10 * time.Minute,
    })
}

Use Case Example:

// During build execution on deployment agent
type BuildCache struct {
    db fiber.Storage
}

func (bc *BuildCache) CacheBuildOutput(ctx context.Context, buildID string, output []byte) error {
    key := fmt.Sprintf("build:%s:output", buildID)
    return bc.db.Set(key, output, 3*time.Hour)
}

func (bc *BuildCache) RetrieveBuildOutput(ctx context.Context, buildID string) ([]byte, error) {
    key := fmt.Sprintf("build:%s:output", buildID)
    return bc.db.Get(key)
}

Performance Considerations:

Not suitable for distributed systems
Use for single-instance edge scenarios only
Regular compaction needed to manage disk space
Monitor LSM tree growth

MongoDB (Optional)

Classification: P2 Secondary - Document Database Sparki Use Cases (if selected):

Pipeline configuration storage (schemaless)
Build metadata (flexible structure)
Analytics data (document model fits well)

Configuration:

func setupMongo(ctx context.Context) (*mongo.Client, error) {
    return mongo.Connect(ctx, options.Client().ApplyURI(
        os.Getenv("MONGODB_URI"),
    ))
}

Not Recommended For:

Critical transactional data (use PostgreSQL)
Real-time permission checks (use Redis)
Audit logs (PostgreSQL better)

SurrealDB (Future)

Classification: P2 Secondary - Multi-Model Database Potential Sparki Use Cases:

Time-series pipeline metrics
Real-time collaboration data
Workflow dependency graphs
Schema-flexible configuration

Status: Planned for future versions

Adapter Selection Decision Matrix

Choosing Storage for New Feature

Flow Chart:

Is data relationship-heavy?
├─ YES → Use PostgreSQL
├─ NO  → Does it need &lt;1ms latency?
        ├─ YES → Use Redis
        ├─ NO  → Will data exceed 1GB?
                ├─ YES → Use S3
                ├─ NO  → Is it temporary?
                        ├─ YES → Use Redis
                        ├─ NO  → Use PostgreSQL

Technology Selection Examples

Build Artifacts:

Characteristic:     Large binary files, long-term storage
Decision:           Use S3
Reasoning:          Object storage optimized for files,
                    unlimited scalability, cost-effective

Permission Cache:

Characteristic:     Frequent reads, &lt;1ms requirement
Decision:           Use Redis
Reasoning:          Sub-millisecond performance,
                    perfect for authorization checks

Audit Logs:

Characteristic:     Immutable, queried infrequently,
                    compliance requirement
Decision:           PostgreSQL + S3 archive
Reasoning:          ACID guarantees for compliance,
                    archive to S3 for long-term storage

Real-Time Build Logs:

Characteristic:     Streaming data, temporary
Decision:           Redis (Streams) + PostgreSQL archive
Reasoning:          Redis Streams for real-time,
                    archive to PostgreSQL for retention

Integration Patterns

Pattern 1: Multi-Tier Storage

type StorageLayer struct {
    cache       fiber.Storage  // Redis - fast access
    database    fiber.Storage  // PostgreSQL - persistent
    archive     fiber.Storage  // S3 - long-term
}

// Check cache first, fall back to database
func (sl *StorageLayer) GetPipeline(ctx context.Context, id string) (*Pipeline, error) {
    // Try cache
    if cached, err := sl.cache.Get(id); err == nil {
        return unmarshal(cached), nil
    }

    // Fall back to database
    if db, err := sl.database.Get(id); err == nil {
        // Repopulate cache
        sl.cache.Set(id, db, 5*time.Minute)
        return unmarshal(db), nil
    }

    return nil, errors.New("not found")
}

Pattern 2: Event Sourcing with Archive

type EventStore struct {
    primary   fiber.Storage  // PostgreSQL - recent events
    archive   fiber.Storage  // S3 - old events
}

// Archive old events to S3 after 30 days
func (es *EventStore) ArchiveOldEvents(ctx context.Context) error {
    cutoff := time.Now().AddDate(0, -1, 0) // 30 days ago

    events := es.primary.List(...)
    for _, event := range events {
        if event.timestamp < cutoff {
            // Move to S3 archive
            es.archive.Set(event.id, event.data, 10*365*24*time.Hour)
            es.primary.Delete(event.id)
        }
    }

    return nil
}

Pattern 3: Cache Invalidation

// When permissions change, invalidate cache
func (auth *AuthService) UpdateUserRole(ctx context.Context, userID, role string) error {
    // Update in database
    if err := auth.db.Update(...); err != nil {
        return err
    }

    // Invalidate all caches for this user
    workspaces := auth.getWorkspacesForUser(userID)
    for _, wsID := range workspaces {
        key := fmt.Sprintf("perm:%s:%s", userID, wsID)
        auth.cache.Delete(key)
    }

    return nil
}

Monitoring & Observability

Key Metrics per Storage Type

PostgreSQL:

- Query latency (p50, p95, p99)
- Connection pool utilization
- Transaction rate
- Cache hit ratio
- Replication lag (if replicated)

Redis:

- Operation latency
- Memory usage
- Eviction rate
- Cache hit ratio
- Command throughput

S3:

- Upload/download latency
- Request count
- Data transfer volume
- Cost per GB
- Failure rate

Alerting Thresholds

Metric	Adapter	Warning	Critical
Latency p95	PostgreSQL	50ms	100ms
Latency p95	Redis	5ms	10ms
Memory Usage	Redis	80%	95%
Connection Pool	PostgreSQL	80%	95%

Conclusion

Sparki’s storage architecture leverages a polyglot persistence approach:

PostgreSQL for relational, transactional data
Redis for ultra-fast caching and real-time data
S3 for scalable object storage
Badger for edge/embedded deployments

This combination provides optimal performance, scalability, and cost-efficiency for the Sparki CI/CD platform.

Document History:

Version	Date	Author	Changes
1.0	2025-12-03	Sparki Engineering	Initial integration matrix

Getting Started

User Guides

Deployment & Operations

DevOps

API Reference

Architecture

Specifications

Forgery Advanced Specs

System Requirements Specs

UI Specifications

Monetization

Integration

Testing

Reports & Status

Sparki Fiber Storage Adapters Integration Matrix

Sparki Fiber Storage Adapters Integration Matrix

Storage Adapter Selection Matrix

Primary Adapters (In Production)

Secondary Adapters (Extended Support)

Supported But Not Primary

Detailed Adapter Specifications

PostgreSQL

Redis / Valkey

S3 / Object Storage

Badger (Embedded)

MongoDB (Optional)

SurrealDB (Future)

Adapter Selection Decision Matrix

Choosing Storage for New Feature

Technology Selection Examples

Integration Patterns

Pattern 1: Multi-Tier Storage

Pattern 2: Event Sourcing with Archive

Pattern 3: Cache Invalidation

Monitoring & Observability

Key Metrics per Storage Type

Alerting Thresholds

Conclusion

Getting Started

User Guides

Deployment & Operations

DevOps

API Reference

Architecture

Specifications

Forgery Advanced Specs

System Requirements Specs

UI Specifications

Monetization

Integration

Testing

Reports & Status

​Sparki Fiber Storage Adapters Integration Matrix

​Storage Adapter Selection Matrix

​Primary Adapters (In Production)

​Secondary Adapters (Extended Support)

​Supported But Not Primary

​Detailed Adapter Specifications

​PostgreSQL

​Redis / Valkey

​S3 / Object Storage

​Badger (Embedded)

​MongoDB (Optional)

​SurrealDB (Future)

​Adapter Selection Decision Matrix

​Choosing Storage for New Feature

​Technology Selection Examples

​Integration Patterns

​Pattern 1: Multi-Tier Storage

​Pattern 2: Event Sourcing with Archive

​Pattern 3: Cache Invalidation

​Monitoring & Observability

​Key Metrics per Storage Type

​Alerting Thresholds

​Conclusion

Sparki Fiber Storage Adapters Integration Matrix

Storage Adapter Selection Matrix

Primary Adapters (In Production)

Secondary Adapters (Extended Support)

Supported But Not Primary

Detailed Adapter Specifications

PostgreSQL

Redis / Valkey

S3 / Object Storage

Badger (Embedded)

MongoDB (Optional)

SurrealDB (Future)

Adapter Selection Decision Matrix

Choosing Storage for New Feature

Technology Selection Examples

Integration Patterns

Pattern 1: Multi-Tier Storage

Pattern 2: Event Sourcing with Archive

Pattern 3: Cache Invalidation

Monitoring & Observability

Key Metrics per Storage Type

Alerting Thresholds

Conclusion