Sparki.tools DevOps Configuration Guide

Last Updated: 2025-12-16 Status: ✅ Complete and Ready for Use

Overview

This guide documents the complete DevOps infrastructure for Sparki.tools, including containerization, development environments, and staging configurations.

What’s Included

✅ Development Container (.devcontainer/Dockerfile) - Full polyglot dev environment
✅ Dev Container Config (.devcontainer/devcontainer.json) - VS Code/Codespaces integration
✅ Extended Dev Environment (docker-compose.dev.extended.yml) - Advanced services
✅ Staging Environment (docker-compose.staging.yml) - Pre-production replica
✅ Dockerfiles (web, storm) - Containerized service definitions
✅ Ignore Files (.gitignore, .dockerignore) - Version control optimization
✅ Helper Scripts (setup.sh, start.sh, update.sh) - Container automation

File Structure

/Users/alexarno/materi/lab/sparki.tools/
├── .devcontainer/
│   ├── Dockerfile                  # Development container image
│   ├── devcontainer.json           # Dev container configuration
│   ├── setup.sh                    # Post-create setup script
│   ├── start.sh                    # Post-start setup script
│   └── update.sh                   # Update dependencies script
├── docker-compose.dev.yml          # Core dev environment (existing)
├── docker-compose.dev.extended.yml # Enhanced dev services
├── docker-compose.staging.yml      # Staging environment
├── docker-compose.test.yml         # Test environment (existing)
├── .gitignore                      # Git ignore rules
├── .dockerignore                   # Docker build ignore rules
├── engine/
│   └── Dockerfile                  # API service (existing)
├── web/
│   └── Dockerfile                  # Web application
├── storm/
│   └── Dockerfile                  # Storm service
└── DEVOPS_CONFIGURATION.md         # This file

1. Development Container (.devcontainer)

Purpose

Provides a complete, reproducible development environment with all necessary tools pre-installed.

Features

Installed Tools:

Go 1.21+
Rust 1.78+ with clippy, rustfmt, rust-analyzer
Python 3.11+ with Poetry, pytest, mypy, black, flake8
Node.js 18+ with pnpm, yarn, Angular CLI, TypeScript
Database clients (PostgreSQL, MySQL, Redis, SQLite)
Container tools (Docker, Docker Compose)
CLI tools (AWS CLI, jq, yq)
Network tools (curl, wget, ping, traceroute, dnsutils)
Development utilities (vim, nano, htop, tmux, tree)
Oh-My-Zsh with Agnoster theme

Configuration Files

`.devcontainer/Dockerfile`

Base Image: Ubuntu 22.04 LTS
Size: ~1.2GB (includes all tooling)
User: Non-root sparki user with Docker access
Ports Exposed: 8080, 8081, 8000, 3000, 5432, 6379, 9090, 16686, 14268

`.devcontainer/devcontainer.json`

Container Image: Built from .devcontainer/Dockerfile
Mount Points:
- SSH config (read-only)
- Git config (read-only)
- Docker config (read-only)
- Workspace volume
Port Forwarding: All dev service ports
VS Code Extensions: 35+ development extensions installed
Feature Flags: Git, GitHub CLI, AWS CLI, Docker-in-Docker

Setup Scripts

`setup.sh` - Post-create setup

Runs after container creation to:

Download Go dependencies
Verify Rust project
Install Node.js dependencies (pnpm/npm)
Install Python dependencies
Configure pre-commit hooks
Create .env.local with development configuration
Configure SSH and Git

`start.sh` - Post-start setup

Runs every container start to:

Verify Docker services are running
Start Docker Compose services if needed
Wait for service health checks
Load environment variables
Display service status

`update.sh` - Dependency updates

Updates dependencies when reopening container:

Update Go modules
Update Rust dependencies
Update Node.js packages
Update Python dependencies
Auto-update pre-commit hooks

Usage

Opening in VS Code with Dev Containers

# Install "Dev Containers" extension in VS Code
# Clone repository
git clone https://github.com/sparki-tools/sparki.git
cd sparki

# Open in container
# VS Code: Cmd+Shift+P → "Dev Containers: Reopen in Container"

Using GitHub Codespaces

# Open repository in GitHub Codespaces
# Automatic setup will run devcontainer.json
# All dependencies installed automatically

Manual Docker Build

docker build -t sparki-dev:latest -f .devcontainer/Dockerfile .
docker run -it -v $(pwd):/workspace sparki-dev:latest /bin/zsh

2. Docker Compose Environments

docker-compose.dev.yml (Core)

Purpose: Standard development environment Services:

PostgreSQL 16 (data + test)
Redis 7 (cache + test)
Docker-in-Docker (executor testing)
Prometheus (metrics)
Grafana (visualization)
Jaeger (tracing)

Usage:

docker-compose -f docker-compose.dev.yml up -d

Service Endpoints:

PostgreSQL: localhost:5432 (user: sparki)
Redis: localhost:6379
Prometheus: http://localhost:9090
Grafana: http://localhost:3000 (admin/admin)
Jaeger: http://localhost:16686

docker-compose.dev.extended.yml (Enhanced)

Purpose: Advanced development with additional services Additional Services:

Elasticsearch + Kibana (log visualization)
Logstash (log processing)
MinIO (S3-compatible storage)
OpenTelemetry Collector
Envoy (API gateway testing)
pgAdmin (PostgreSQL management)
Redis Commander (Redis GUI)
Adminer (database administration)
MailHog (email testing)
LocalStack (AWS services emulation)
Vault (secrets management)
k6 (load testing)
API dev service (hot reload)
Web dev service (hot reload)

Usage:

docker-compose -f docker-compose.dev.yml -f docker-compose.dev.extended.yml up -d

Additional Service Endpoints:

Elasticsearch: localhost:9200
Kibana: http://localhost:5601
MinIO: http://localhost:9000 (admin/admin)
pgAdmin: http://localhost:5050
Redis Commander: http://localhost:8081
Adminer: http://localhost:8082
MailHog: http://localhost:8025
LocalStack: localhost:4566
Vault: http://localhost:8200
API (dev): http://localhost:8080 (with hot reload)
Web (dev): http://localhost:3000 (with hot reload)

docker-compose.staging.yml (Pre-production)

Purpose: Mirror production environment for testing Services:

PostgreSQL 16 (production-like)
Redis 7 (production-like)
Sparki API (compiled)
Sparki Web (optimized)
Prometheus with retention
Grafana with provisioning
Jaeger with persistence
Loki (log aggregation)
Promtail (log shipping)
AlertManager (alert management)
Nginx (reverse proxy)

Configuration:

Environment variables via .env file
Persistent volumes
Health checks
Restart policies
Logging configuration

Usage:

docker-compose -f docker-compose.staging.yml up -d

# View logs
docker-compose -f docker-compose.staging.yml logs -f

# Stop services
docker-compose -f docker-compose.staging.yml down

Staging Service Endpoints:

API: http://localhost:8080
Web: http://localhost:3000
PostgreSQL: localhost:5434 (staging database)
Redis: localhost:6380 (staging cache)
Prometheus: http://localhost:9091
Grafana: http://localhost:3001
Jaeger: http://localhost:16687
Loki: http://localhost:3100
AlertManager: http://localhost:9093
Nginx: http://localhost:80, https://localhost:443

3. Dockerfile Specifications

engine/Dockerfile (API)

Purpose: Build Sparki API service Base Image: Alpine 3.18 Stages: 2 (builder + runtime) Key Features:

Multi-stage build for minimal size
Go compilation with version info
Non-root user (sparki:1000)
Health checks enabled
Security hardened

Build:

docker build -t sparki/api:latest engine/

web/Dockerfile (Web Application)

Purpose: Build Sparki Web application Base Image: Node 18-Alpine Stages: 3 (dependencies + builder + runtime) Key Features:

Multi-stage build with dependency caching
Support for pnpm/yarn/npm
Next.js build optimization
Non-root user
Health checks
Proper signal handling (dumb-init)

Build:

docker build \
  --build-arg NEXT_PUBLIC_API_URL=http://localhost:8080 \
  --build-arg NEXT_PUBLIC_WS_URL=ws://localhost:8081 \
  -t sparki/web:latest web/

storm/Dockerfile (Storm Service)

Purpose: Build Storm observability service Base Image: Alpine 3.18 Key Features:

Template for storm service build
Flexible for multiple language implementations
Configuration file management
Non-root user
Health checks

Build:

docker build -t sparki/storm:latest storm/

4. .gitignore Configuration

Includes:

Environment files with secrets
IDE & editor configurations
Build artifacts (bin, dist, build, etc.)
Language-specific artifacts:
- Go: vendor, *.prof, coverage.out
- Rust: target/, Cargo.lock
- Node: node_modules, pnpm-lock.yaml, yarn.lock
- Python: pycache, *.pyc, venv/
Docker configuration
Logs and temporary files
Database files
IDE cache

Excludes (force tracked):

.gitignore itself
.dockerignore
.env.example (template for environment)
.devcontainer/ directory

5. .dockerignore Configuration

Includes:

Source control files (.git, .github)
CI/CD configuration
IDE & editor config
Documentation files
Test files and coverage
Build artifacts
Dependencies (let Docker handle)
Development configuration
Package manager lock files (let Docker handle fresh installs)

Keeps:

Dockerfile and docker-entrypoint scripts
Key application files

6. Common Development Workflows

Scenario 1: Initial Project Setup

# 1. Clone repository
git clone https://github.com/sparki-tools/sparki.git
cd sparki

# 2. Open in dev container
# VS Code: Cmd+Shift+P → "Dev Containers: Reopen in Container"

# 3. Container auto-setup runs setup.sh
# - Dependencies installed
# - .env.local created
# - Pre-commit hooks configured

# 4. Start services
make dev

# 5. Verify health
curl http://localhost:8080/health
curl http://localhost:3000/health

Scenario 2: Full Stack Development (Extended)

# 1. Start extended development environment
docker-compose -f docker-compose.dev.yml \
  -f docker-compose.dev.extended.yml up -d

# 2. Wait for services
sleep 10

# 3. Verify services
docker-compose ps

# 4. View logs
docker-compose logs -f

# 5. Access services
# - API: http://localhost:8080
# - Web: http://localhost:3000 (hot reload)
# - API: http://localhost:8080 (hot reload)
# - Grafana: http://localhost:3000
# - MinIO: http://localhost:9000
# - Vault: http://localhost:8200

Scenario 3: Staging Deployment Test

# 1. Build production images
docker build -t sparki/api:v1.0.0 engine/
docker build -t sparki/web:v1.0.0 web/

# 2. Start staging environment
docker-compose -f docker-compose.staging.yml up -d

# 3. Monitor logs
docker-compose -f docker-compose.staging.yml logs -f

# 4. Run tests
make test-integration

# 5. Check metrics
open http://localhost:9091/targets  # Prometheus
open http://localhost:3001          # Grafana

# 6. Verify health
docker-compose -f docker-compose.staging.yml ps

Scenario 4: Load Testing

# 1. Start extended environment with k6
docker-compose -f docker-compose.dev.yml \
  -f docker-compose.dev.extended.yml up -d k6

# 2. Run load tests
k6 run infrastructure/k6/tests/smoke.js

# 3. Monitor during test
watch 'curl -s http://localhost:9090/api/v1/query?query=http_requests_total | jq'

7. Environment Variables

Development (.env.local)

# API Configuration
API_HOST=0.0.0.0
API_PORT=8080
API_DEBUG=true

# Database
DATABASE_HOST=postgres
DATABASE_PORT=5432
DATABASE_USER=sparki
DATABASE_PASSWORD=sparki
DATABASE_NAME=sparki_dev

# Cache
REDIS_HOST=redis
REDIS_PORT=6379

# Logging
LOG_LEVEL=debug
ENABLE_METRICS=true
ENABLE_TRACING=true

Staging (.env.staging)

# Use secure passwords in production
DATABASE_PASSWORD=<secure-password>
REDIS_PASSWORD=<secure-password>
GRAFANA_PASSWORD=<secure-password>

# Service configuration
ENVIRONMENT=staging
LOG_LEVEL=info
ENABLE_METRICS=true
ENABLE_TRACING=true

8. Health Checks

All services include health checks configured in docker-compose files:

healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
  interval: 30s
  timeout: 3s
  retries: 3
  start_period: 10s

Checking Health:

# Check single service
docker ps --format "table {{.Names}}\t{{.Status}}"

# Check in docker-compose
docker-compose ps

# Manual check
curl http://localhost:8080/health

9. Volume Management

Development Volumes

# List volumes
docker volume ls

# Inspect volume
docker volume inspect sparki-postgres_data

# Clean up volumes
docker volume prune

# Remove specific volume
docker volume rm sparki-postgres_data

Backup/Restore

# Backup PostgreSQL volume
docker run --rm -v sparki-postgres_data:/data -v $(pwd):/backup \
  alpine tar czf /backup/postgres-backup.tar.gz -C /data .

# Restore PostgreSQL volume
docker run --rm -v sparki-postgres_data:/data -v $(pwd):/backup \
  alpine tar xzf /backup/postgres-backup.tar.gz -C /data

10. Networking

Container Networks

All services use sparki-network bridge network for communication:

# List networks
docker network ls | grep sparki

# Inspect network
docker network inspect sparki-network

# Service DNS resolution within network
# Example: API can connect to PostgreSQL via: postgres:5432

Port Forwarding

Development Ports (from containers to host):

5432: PostgreSQL
6379: Redis
8080: API
8081: WebSocket/Relay
3000: Web/Grafana
9090: Prometheus
16686: Jaeger
14268: Jaeger Collector

To expose different port on host:

ports:
  - "5432:5432"  # host:container
  - "3333:3000"  # Expose on different port

11. Security Considerations

Non-Root Users

All images run with non-root users:

API: sparki:1000
Web: node:1000
Storm: storm:1000

Secret Management

Never commit secrets - Use .env.local (in .gitignore)
Use environment variables - Pass via docker-compose
Vault integration - For production secrets
Secret rotation - Automated via CI/CD

Network Security

Internal network - Services communicate via bridge network
Port exposure - Only necessary ports exposed to host
HTTPS - Nginx reverse proxy for TLS termination
Firewalls - Configure host firewall for production

Image Security

Alpine base - Minimal attack surface
Multi-stage builds - Reduced final image size
Security scanning - Use Trivy for vulnerability scanning
Minimal dependencies - Only required packages

12. Troubleshooting

Common Issues

Port already in use:

# Find process using port
lsof -i :8080

# Kill process
kill -9 <PID>

# Or change port in docker-compose

Services unhealthy:

# Check health status
docker-compose ps

# View logs
docker-compose logs <service-name>

# Inspect health check
docker inspect <container-id> | grep -A 5 Healthcheck

Volume issues:

# Reset volumes
docker-compose down -v

# Recreate from scratch
docker-compose up -d

Network connectivity:

# Test from container
docker-compose exec api ping postgres

# Check DNS
docker-compose exec api nslookup postgres

13. Performance Optimization

Memory Management

# Limit service memory
services:
  api:
    deploy:
      resources:
        limits:
          memory: 512M
        reservations:
          memory: 256M

CPU Management

# Limit service CPU
deploy:
  resources:
    limits:
      cpus: '1'
    reservations:
      cpus: '0.5'

Disk I/O

# Monitor disk usage
docker stats

# Cleanup
docker system prune -a  # Remove unused images
docker volume prune     # Remove unused volumes

14. CI/CD Integration

GitHub Actions Example

name: Docker Build and Test

on: [push, pull_request]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2

      - name: Build images
        run: |
          docker build -t sparki/api:latest engine/
          docker build -t sparki/web:latest web/

      - name: Start services
        run: docker-compose -f docker-compose.dev.yml up -d

      - name: Run tests
        run: make test

      - name: Push to registry
        run: |
          docker push sparki/api:latest
          docker push sparki/web:latest

15. Maintenance

Regular Tasks

# Daily: Check health
docker-compose ps
docker-compose logs --tail 20

# Weekly: Update images
docker pull postgres:16-alpine
docker pull redis:7-alpine
docker pull grafana/grafana:latest

# Monthly: Clean up
docker image prune -a
docker volume prune
docker network prune

# Quarterly: Security updates
docker pull alpine:3.18  # Update base images
docker build --no-cache  # Rebuild to get latest patches

Quick Reference

Essential Commands

# Development
make dev              # Start development environment
make dev-down         # Stop development
make dev-logs         # View logs
make test             # Run all tests

# Docker Compose
docker-compose up -d  # Start services
docker-compose down   # Stop services
docker-compose ps     # Service status
docker-compose logs   # View logs

# Dev Container (VS Code)
Cmd+Shift+P → "Dev Containers: Reopen in Container"
Cmd+Shift+P → "Dev Containers: Rebuild Container"

# Staging
docker-compose -f docker-compose.staging.yml up -d
docker-compose -f docker-compose.staging.yml down

Support

For Issues:

Check logs: docker-compose logs <service>
Review .env configuration
Verify port availability
Check disk/memory availability
Reset volumes: docker-compose down -v && docker-compose up -d

For Questions:

See /lab/sparki.tools/docs/ for additional documentation
Check existing issues on GitHub
Review infrastructure configuration files

Generated: 2025-12-16 Status: ✅ Complete Maintainer: Platform Engineering

Getting Started

User Guides

Deployment & Operations

DevOps

API Reference

Architecture

Specifications

Monetization

Reports & Status

​Sparki.tools DevOps Configuration Guide

​Overview

​What’s Included

​File Structure

​1. Development Container (.devcontainer)

​Purpose

​Features

​Configuration Files

​.devcontainer/Dockerfile

​.devcontainer/devcontainer.json

​Setup Scripts

​setup.sh - Post-create setup

​start.sh - Post-start setup

​update.sh - Dependency updates

​Usage

​Opening in VS Code with Dev Containers

​Using GitHub Codespaces

​Manual Docker Build

​2. Docker Compose Environments

​docker-compose.dev.yml (Core)

​docker-compose.dev.extended.yml (Enhanced)

​docker-compose.staging.yml (Pre-production)

​3. Dockerfile Specifications

​engine/Dockerfile (API)

​web/Dockerfile (Web Application)

​storm/Dockerfile (Storm Service)

​4. .gitignore Configuration

​5. .dockerignore Configuration

​6. Common Development Workflows

​Scenario 1: Initial Project Setup

​Scenario 2: Full Stack Development (Extended)

​Scenario 3: Staging Deployment Test

​Scenario 4: Load Testing

​7. Environment Variables

​Development (.env.local)

​Staging (.env.staging)

​8. Health Checks

​9. Volume Management

​Development Volumes

​Backup/Restore

​10. Networking

​Container Networks

​Port Forwarding

​11. Security Considerations

​Non-Root Users

​Secret Management

​Network Security

​Image Security

​12. Troubleshooting

​Common Issues

​13. Performance Optimization

​Memory Management

​CPU Management

​Disk I/O

​14. CI/CD Integration

​GitHub Actions Example

​15. Maintenance

​Regular Tasks

​Quick Reference

​Essential Commands

​Support

Sparki.tools DevOps Configuration Guide

Overview

What’s Included

File Structure

1. Development Container (.devcontainer)

Purpose

Features

Configuration Files

`.devcontainer/Dockerfile`

`.devcontainer/devcontainer.json`

Setup Scripts

`setup.sh` - Post-create setup

`start.sh` - Post-start setup

`update.sh` - Dependency updates

Usage

Opening in VS Code with Dev Containers

Using GitHub Codespaces

Manual Docker Build

2. Docker Compose Environments

docker-compose.dev.yml (Core)

docker-compose.dev.extended.yml (Enhanced)

docker-compose.staging.yml (Pre-production)

3. Dockerfile Specifications

engine/Dockerfile (API)

web/Dockerfile (Web Application)

storm/Dockerfile (Storm Service)

4. .gitignore Configuration

5. .dockerignore Configuration

6. Common Development Workflows

Scenario 1: Initial Project Setup

Scenario 2: Full Stack Development (Extended)

Scenario 3: Staging Deployment Test

Scenario 4: Load Testing

7. Environment Variables

Development (.env.local)

Staging (.env.staging)

8. Health Checks

9. Volume Management

Development Volumes

Backup/Restore

10. Networking

Container Networks

Port Forwarding

11. Security Considerations

Non-Root Users

Secret Management

Network Security

Image Security

12. Troubleshooting

Common Issues

13. Performance Optimization

Memory Management

CPU Management

Disk I/O

14. CI/CD Integration

GitHub Actions Example

15. Maintenance

Regular Tasks

Quick Reference

Essential Commands

Support