Skip to main content

Sparki Practical Examples - Real-World Workflows

Purpose: Show exactly what to type and what to expect with real examples.

Example 1: Complete Setup from Scratch (15 minutes)

Scenario

You just cloned the repository and want to get everything running.

Step-by-Step Walkthrough

# Step 1: Navigate to domain directory
$ cd /Users/alexarno/materi/domain
$ pwd
/Users/alexarno/materi/domain

# Step 2: Create workspace structure
$ make setup
Setting up domain workspace structure...
mkdir -p .workspace/logs/{api,relay,shield}
mkdir -p .workspace/config
mkdir -p .workspace/data
mkdir -p scripts
mkdir -p tests/{unit,integration,e2e}
 Workspace structure created
total 0
drwxr-xr-x  3 user  staff  96 Dec 16 10:30 .workspace
drwxr-xr-x  2 user  staff  64 Dec 16 10:30 scripts
drwxr-xr-x  4 user  staff 128 Dec 16 10:30 tests

# Step 3: Start database infrastructure
$ docker-compose -f docker-compose.unified.yml up -d postgres redis
Creating postgres ... done
Creating redis   ... done

# Verify they're running
$ docker ps
CONTAINER ID   IMAGE           COMMAND                  NAMES
abc123def456   postgres:15    "docker-entrypoint..."   postgres
xyz789uvw012   redis:7-alpine "redis-server"           redis

# Step 4: Assemble all services (this takes ~3 minutes)
$ make assemble ENV=development
Assembling domain nest...
[✓] Cloning manuscripts service...
[✓] Installing manuscripts dependencies...
[✓] Cloning api service...
[✓] Installing api dependencies (go mod download)...
[✓] Cloning relay service...
[✓] Installing relay dependencies (cargo fetch)...
[✓] Cloning shield service...
[✓] Installing shield dependencies (pip install)...
[✓] Domain nest assembly complete
Workspace summary:
  .workspace/manuscripts/    (Go)
  .workspace/api/           (Go, 32MB)
  .workspace/relay/         (Rust, 450MB)
  .workspace/shield/        (Python)

# Step 5: Build all services (this takes ~5 minutes)
$ make build
Building manuscript service...
 Manuscript built (1.2s)
Building core services...
  Building api...
 API built (45s)
  Building relay...
 Relay built (120s)
  Building shield...
 Shield built (30s)
 Core services built

# Step 6: Start all services
$ make start
Starting domain services...
Checking infrastructure...
  Starting api...
 api started (PID: 12345)
  Starting relay...
 relay started (PID: 12346)
  Starting shield...
 shield started (PID: 12347)

 All services started

Service Endpoints:
  - API:     http://localhost:8080/health
  - Relay:   http://localhost:8081/health
  - Shield:  http://localhost:8000/health

View logs:
  - make logs-api
  - make logs-relay
  - make logs-shield

# Step 7: Verify everything is running
$ make health
Service Health Status:
├─ API (8080):     ✅ Healthy (response time: 2ms)
├─ Relay (8081):   ✅ Healthy (response time: 1ms)
├─ Shield (8000):  ✅ Healthy (response time: 3ms)
├─ PostgreSQL: Ready
├─ Redis: Ready
└─ Overall: All services operational

# SUCCESS! You now have a complete, running Domain Nest

Example 2: Run Tests and See Results

Scenario

You want to verify that everything is working correctly by running the complete test suite.

Step-by-Step Walkthrough

# Step 1: Run the full test orchestrator
$ cd /Users/alexarno/materi/domain
$ ./tests/sparki-test-orchestrator.sh all development
Sparki Test Orchestrator - Starting full test suite
Environment: development
Timestamp: 2025-12-16T14:30:00Z

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[PHASE 1/6] Unit Tests
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Running API unit tests (Go)...
 test_document_creation ... PASS (45ms)
 test_document_update ... PASS (38ms)
 test_document_delete ... PASS (32ms)
 test_permission_check ... PASS (52ms)
 test_error_handling ... PASS (28ms)
[✓] API: 5/5 tests passed (4.2s total)

Running Relay unit tests (Rust)...
 test_websocket_connect ... PASS (120ms)
 test_message_validation ... PASS (95ms)
 test_ot_transform ... PASS (145ms)
 test_presence_update ... PASS (88ms)
[✓] Relay: 4/4 tests passed (5.1s total)

Running Shield unit tests (Python)...
 test_user_creation ... PASS (156ms)
 test_jwt_validation ... PASS (142ms)
 test_permission_cache ... PASS (168ms)
[✓] Shield: 3/3 tests passed (6.2s total)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[PHASE 2/6] Integration Tests
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Testing cross-service communication...
 test_api_calls_shield_auth ... PASS (234ms)
 test_document_event_sync ... PASS (567ms)
 test_websocket_collaboration ... PASS (892ms)
 test_redis_stream_events ... PASS (445ms)
[✓] Integration: 4/4 tests passed (8.6s total)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[PHASE 3/6] Linting & Code Quality
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Running golangci-lint (API)...
 No issues found

Running cargo fmt + clippy (Relay)...
 Code formatting OK
 No clippy warnings

Running black + flake8 (Shield)...
 Code formatting OK
 No flake8 issues

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[PHASE 4/6] Security Scanning
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Running gosec (API)...
 No security issues found

Running cargo audit (Relay)...
 No vulnerable dependencies

Running bandit (Shield)...
 No security issues found

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[PHASE 5/6] Protocol Validation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Validating HTTP/REST protocol...
 Status codes valid
 Rate limiting configured
 Timeouts configured

Validating JWT authentication...
 Algorithm: HS256
 Required claims present
 Token expiry configured

Validating WebSocket protocol...
 Connection rules valid
 Message types defined
 Resource limits set

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[PHASE 6/6] E2E Tests
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Running end-to-end workflows...
 Complete document workflow (create→edit→share→publish) ✅ PASS (1.2s)
 Real-time collaboration (2 users) ✅ PASS (2.1s)
 Permission verification (3 scenarios) ✅ PASS (1.8s)
[✓] E2E: 3/3 tests passed (5.1s total)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
TEST SUMMARY
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Total Tests Run:      77
Passed:              77
Failed:               0
Skipped:              0
Success Rate:      100%

Coverage Report:
  Lines Covered:     590/590
  Coverage:          100%

Execution Time:
  Total:           42.3 seconds
  Average per test: 0.55 seconds

Report Location:
  JSON: test-reports/test-report-20251216_143000.json
  Log:  test-reports/orchestrator.log

 ALL TESTS PASSED - SYSTEM READY FOR DEPLOYMENT

# Step 2: View detailed JSON report (machine-readable)
$ cat test-reports/test-report-*.json | jq .
{
  "timestamp": "2025-12-16T14:30:00Z",
  "environment": "development",
  "summary": {
    "total_tests": 77,
    "passed": 77,
    "failed": 0,
    "skipped": 0,
    "success_rate": 1.0,
    "duration_seconds": 42.3
  },
  "phases": {
    "unit_tests": {
      "api": 5,
      "relay": 4,
      "shield": 3
    },
    "integration_tests": 4,
    "security_scan": 0,
    "protocols": {
      "http": "✓",
      "jwt": "✓",
      "websocket": "✓"
    }
  }
}

# Step 3: View human-readable log
$ tail -30 test-reports/orchestrator.log
[14:30:42.100] Running end-to-end workflows...
[14:30:42.234] ✓ Complete document workflow passed (1234ms)
[14:30:44.456] ✓ Real-time collaboration passed (2123ms)
[14:30:46.234] ✓ Permission verification passed (1789ms)
[14:30:48.100]
[14:30:48.100] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[14:30:48.100] ✅ TEST SUITE COMPLETE
[14:30:48.100] Total: 77 tests, All passed (100%)
[14:30:48.100] Duration: 42.3 seconds
[14:30:48.100] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Example 3: Debug a Service Issue

Scenario

Shield service is showing as unhealthy. You need to figure out why and fix it.

Step-by-Step Walkthrough

# Step 1: Run health check to see which service is down
$ make health
Service Health Status:
├─ API (8080):     ✅ Healthy (response time: 2ms)
├─ Relay (8081):   ✅ Healthy (response time: 1ms)
├─ Shield (8000):  ❌ UNHEALTHY (connection refused)
├─ PostgreSQL: Ready
├─ Redis: Ready
└─ Overall: System degraded - Shield unavailable

# Step 2: Check Shield's logs to see what's wrong
$ make logs-shield | tail -50
2025-12-16 14:35:22 | ERROR | [django.security] Forbidden (403): /health
2025-12-16 14:35:23 | ERROR | [django.security] PermissionDenied at /health
2025-12-16 14:35:23 | ERROR | Authentication middleware raised exception

# Step 3: Check the Shield service's recent changes
$ cat /Users/alexarno/materi/domain/sparki/shield.sparki.yml | grep -A 5 "health_check"
health_check:
  endpoint: /health
  timeout: 5000
  expected_status: 200
  interval: 10000

# Step 4: Connect to PostgreSQL to check database state
$ make db-shell
psql (15.2)
materi=> SELECT * FROM django_migrations WHERE app='auth' ORDER BY name DESC LIMIT 5;
 id |        app         |                          name                           |  applied
----+--------------------+---------------------------------------------------------+------------------------
 10 | auth               | 0010_add_permission_groups                               | 2025-12-16 10:30:00
  9 | auth               | 0009_alter_user_groups                                  | 2025-12-16 10:30:00
  8 | auth               | 0008_alter_user_email                                   | 2025-12-16 10:30:00
  7 | auth               | 0007_add_user_fields                                    | 2025-12-16 10:30:00
  6 | auth               | 0006_initial                                            | 2025-12-16 10:30:00

# All migrations present, database looks OK
materi=> \q

# Step 5: Check if Shield process is actually running
$ ps aux | grep shield
user  12347  0.5  2.3  543210  234567  S  14:30  0:15 python manage.py runserver

# Process is running, so let's restart it
$ make restart

# Step 6: Verify it's back
$ make health
Service Health Status:
├─ API (8080):     ✅ Healthy
├─ Relay (8081):   ✅ Healthy
├─ Shield (8000):  ✅ Healthy (response time: 2ms)
├─ PostgreSQL: Ready
├─ Redis: Ready
└─ Overall: All services operational

# SUCCESS! Shield is back online.

Example 4: Monitor Metrics in Real-Time

Scenario

You want to watch metrics as requests come in to see how your system is performing.

Step-by-Step Walkthrough

# Step 1: Open multiple terminal windows or use tmux

# Terminal 1: Watch overall health
$ watch -n 2 'make health'

# Output updates every 2 seconds:
Service Health Status:
├─ API (8080):     ✅ Healthy (response time: 2ms)
├─ Relay (8081):   ✅ Healthy (response time: 1ms)
├─ Shield (8000):  ✅ Healthy (response time: 3ms)
├─ PostgreSQL: Ready
├─ Redis: Ready
└─ Overall: All services operational

# Terminal 2: Watch API metrics
$ watch -n 1 'curl -s http://localhost:9090/metrics | grep -E "(http_requests|http_request_duration|database_connections)" | tail -10'

# Output updates every second:
http_requests_total{method="GET",path="/health",status="200"} 1234
http_requests_total{method="POST",path="/api/v1/documents",status="200"} 567
http_request_duration_seconds{method="GET",path="/health"} 0.002
http_request_duration_seconds{method="POST",path="/api/v1/documents"} 0.156
database_connections{state="active"} 12
database_connections{state="idle"} 8

# Terminal 3: Watch Relay WebSocket metrics
$ watch -n 1 'curl -s http://localhost:9092/metrics | grep websocket'

# Output:
websocket_connections_total 145
websocket_connections_active 32
websocket_message_latency_seconds{p50="0.045",p95="0.234",p99="0.567"}
operation_transform_conflicts_total 3

# Terminal 4: Watch Shield authentication metrics
$ watch -n 1 'curl -s http://localhost:9091/metrics | grep jwt'

# Output:
jwt_validations_total 892
jwt_validation_errors_total 2
permission_cache_hits 234
permission_cache_misses 12

Example 5: Make a Request Through the Complete System

Scenario

You want to trace a request from API through Shield authentication and to Relay for real-time updates.

Step-by-Step Walkthrough

# Step 1: Generate an authentication token (from Shield)
$ TOKEN=$(curl -X POST http://localhost:8000/api/v1/auth/token \
  -H "Content-Type: application/json" \
  -d '{"username":"testuser","password":"testpass"}' \
  | jq -r .access_token)

$ echo $TOKEN
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ1c2VyLTE...

# Step 2: Create a document via API with that token
$ curl -X POST http://localhost:8080/api/v1/documents \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "title": "My Document",
    "content": "Hello World",
    "workspace_id": "workspace-123"
  }' | jq .

{
  "id": "doc-456",
  "title": "My Document",
  "content": "Hello World",
  "created_at": "2025-12-16T14:40:00Z",
  "owner": "user-123"
}

# Step 3: Check that the event was published to Redis
$ make redis-shell
redis> XREAD COUNT 5 STREAMS materi:events:documents 0
1) "materi:events:documents"
   1) 1) "1702738800000-0"
      2) 1) "type"
         2) "document_created"
         3) "document_id"
         4) "doc-456"
         5) "user_id"
         6) "user-123"
         7) "workspace_id"
         8) "workspace-123"
redis> QUIT

# Step 4: Connect to Relay via WebSocket to receive real-time updates
$ wscat -c "ws://localhost:8081/ws?token=$TOKEN" 2>/dev/null

Connected (press CTRL+C to quit)
>

# In another terminal, update the document
$ curl -X PUT http://localhost:8080/api/v1/documents/doc-456 \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"content": "Hello World - Updated!"}' | jq .

{
  "id": "doc-456",
  "title": "My Document",
  "content": "Hello World - Updated!",
  "updated_at": "2025-12-16T14:41:00Z"
}

# Back in WebSocket terminal, you see the update in real-time:
< {"type":"document_updated","document_id":"doc-456","content":"Hello World - Updated!","timestamp":1702738860000}

# Step 5: Check metrics to see the requests
$ curl -s http://localhost:9090/metrics | grep -E 'method="POST"|method="PUT"'
http_requests_total{method="POST",path="/api/v1/documents",status="200"} 1
http_requests_total{method="PUT",path="/api/v1/documents",status="200"} 1
http_request_duration_seconds{method="POST",path="/api/v1/documents"} 0.234
http_request_duration_seconds{method="PUT",path="/api/v1/documents"} 0.189

$ curl -s http://localhost:9092/metrics | grep message_latency
websocket_message_latency_seconds{p50="0.012",p95="0.045",p99="0.089"}

# Step 6: Check Shield validation metrics
$ curl -s http://localhost:9091/metrics | grep jwt_validations
jwt_validations_total 3
jwt_validation_cache_hits 2
jwt_validation_cache_misses 1

# SUMMARY: You've traced a complete request:
# 1. Authenticated with Shield (JWT token generated)
# 2. Created document via API (HTTP POST)
# 3. Event published to Redis Streams
# 4. Real-time update delivered via Relay WebSocket
# 5. All metrics recorded and available for monitoring

Example 6: Add a New Environment (Staging)

Scenario

You want to create a staging environment separate from development.

Step-by-Step Walkthrough

# Step 1: Create staging environment configuration
$ cat > /Users/alexarno/materi/domain/.env.staging << 'EOF'
# Staging environment configuration
ENVIRONMENT=staging
LOG_LEVEL=info
DATABASE_URL=postgresql://user:pass@staging-db:5432/materi_staging
REDIS_URL=redis://staging-redis:6379/0
JWT_SECRET=your-staging-jwt-secret-here
DEBUG=false
ENABLE_METRICS=true
ENABLE_TRACING=false
EOF

# Step 2: Load staging environment
$ export $(cat /Users/alexarno/materi/domain/.env.staging | xargs)

# Step 3: Setup workspace for staging
$ make setup
$ make assemble ENV=staging

# Step 4: Build with staging environment
$ make build

# Step 5: Start services pointing to staging infrastructure
$ docker-compose -f docker-compose.staging.yml up -d postgres redis
$ make start

# Step 6: Verify staging services are running
$ make health
Service Health Status:
├─ API (8080):     ✅ Healthy (environment: staging)
├─ Relay (8081):   ✅ Healthy (environment: staging)
├─ Shield (8000):  ✅ Healthy (environment: staging)
└─ Overall: All services operational (staging)

# Step 7: Run staging tests
$ ./tests/sparki-test-orchestrator.sh all staging
[✓] All tests passed in staging environment (100%)

# SUCCESS! Staging environment is ready

Example 7: Deploy to Production (Manual Approval)

Scenario

After staging tests pass, you need to manually approve and deploy to production.

Step-by-Step Walkthrough

# Step 1: Ensure all tests pass in staging
$ ./tests/sparki-test-orchestrator.sh all staging
 ALL TESTS PASSED (77/77)

# Step 2: Smoke tests in staging
$ make validate
[✓] Configuration validation passed
[✓] Security validation passed
[✓] Dependency check passed

# Step 3: Tag the release
$ git tag -a v1.2.3 -m "Release: Production deployment"
$ git push origin v1.2.3

# Step 4: This triggers GitHub Actions workflow (if set up)
# Check status: Navigate to GitHub Actions in your repo
# Or via CLI:
$ gh workflow view sparki-domain-deploy.yml

# Step 5: Wait for pre-deployment checks
# The workflow will:
# - Run all tests
# - Security scans
# - Build Docker images
# - Deploy to staging
# - Wait for manual approval

# Step 6: When prompted, approve production deployment
# Via GitHub UI: Actions → sparki-domain-deploy → Review deployments
# Or via CLI:
$ gh run list --workflow=sparki-domain-deploy.yml --status=waiting
$ gh run view <RUN_ID> --log

# Step 7: Approve the deployment
$ gh run approve <RUN_ID> --comment "Approved for production"

# Workflow continues:
# - Deploys to production
# - Runs smoke tests
# - Creates release notes
# - Tags release

# Step 8: Monitor production deployment
$ watch -n 5 'make health'
Service Health Status:
├─ API (8080):     ✅ Healthy (version: v1.2.3)
├─ Relay (8081):   ✅ Healthy (version: v1.2.3)
├─ Shield (8000):  ✅ Healthy (version: v1.2.3)
└─ Overall: All services operational (PRODUCTION)

# Step 9: Verify production metrics
$ curl -s http://localhost:9090/metrics | grep version
build_info{version="v1.2.3",commit="abc123def456"}

# SUCCESS! Production deployment complete

Example 8: Rollback from Production

Scenario

Something went wrong in production and you need to quickly rollback.

Step-by-Step Walkthrough

# Step 1: Identify the issue
$ make health
Service Health Status:
├─ API (8080):     ❌ ERROR 500 (version: v1.2.3)
├─ Relay (8081):   ✅ Healthy (version: v1.2.3)
├─ Shield (8000):  ✅ Healthy (version: v1.2.3)
└─ Overall: Degraded

# Step 2: Check recent API logs
$ make logs-api | tail -100
2025-12-16 14:50:22 | ERROR | [api.service] Database connection pool exhausted
2025-12-16 14:50:23 | ERROR | [api.handler] Internal Server Error

# Step 3: Trigger rollback to previous version
$ git log --oneline | head -5
v1.2.3 Release: Production deployment
v1.2.2 Minor bug fix
v1.2.1 Database optimization
v1.2.0 Feature: New UI components

# Rollback to v1.2.2 (previous stable version)
$ git checkout v1.2.2
$ make build
$ make restart

# Step 4: Verify rollback
$ make health
Service Health Status:
├─ API (8080):     ✅ Healthy (version: v1.2.2)
├─ Relay (8081):   ✅ Healthy (version: v1.2.2)
├─ Shield (8000):  ✅ Healthy (version: v1.2.2)
└─ Overall: All services operational

# Step 5: Run tests to confirm rollback is stable
$ ./tests/sparki-test-orchestrator.sh all production
 ALL TESTS PASSED (77/77)

# Step 6: Notify team
$ echo "⚠️  ROLLBACK COMPLETED: v1.2.3 → v1.2.2 at $(date)"

# Step 7: Later, investigate the issue and fix
# ... debugging and fixes happen here ...
# Then tag a new fixed version (v1.2.4)

# SUCCESS! Rollback successful, system stable

Quick Reference: Common Commands by Use Case

Use CaseCommandTime
Full setupmake setup && make assemble && make build && make start15 min
Quick test./tests/sparki-test-orchestrator.sh unit5 min
All tests./tests/sparki-test-orchestrator.sh all45 min
Check healthmake health5 sec
View logsmake logs or make logs-service SERVICE=api1 sec
Restartmake restart10 sec
Deploygit push && approve in GitHub15 min
Rollbackgit checkout <previous-tag> && make restart5 min
Debugmake health && make logs-<service> && make db-shellvaries

Last Updated: 2025-12-16 Status: Ready to use Created For: Practical hands-on learning with real examples