Sparki Practical Examples - Real-World Workflows
Purpose: Show exactly what to type and what to expect with real examples.Example 1: Complete Setup from Scratch (15 minutes)
Scenario
You just cloned the repository and want to get everything running.Step-by-Step Walkthrough
Copy
# Step 1: Navigate to domain directory
$ cd /Users/alexarno/materi/domain
$ pwd
/Users/alexarno/materi/domain
# Step 2: Create workspace structure
$ make setup
Setting up domain workspace structure...
mkdir -p .workspace/logs/{api,relay,shield}
mkdir -p .workspace/config
mkdir -p .workspace/data
mkdir -p scripts
mkdir -p tests/{unit,integration,e2e}
✓ Workspace structure created
total 0
drwxr-xr-x 3 user staff 96 Dec 16 10:30 .workspace
drwxr-xr-x 2 user staff 64 Dec 16 10:30 scripts
drwxr-xr-x 4 user staff 128 Dec 16 10:30 tests
# Step 3: Start database infrastructure
$ docker-compose -f docker-compose.unified.yml up -d postgres redis
Creating postgres ... done
Creating redis ... done
# Verify they're running
$ docker ps
CONTAINER ID IMAGE COMMAND NAMES
abc123def456 postgres:15 "docker-entrypoint..." postgres
xyz789uvw012 redis:7-alpine "redis-server" redis
# Step 4: Assemble all services (this takes ~3 minutes)
$ make assemble ENV=development
Assembling domain nest...
[✓] Cloning manuscripts service...
[✓] Installing manuscripts dependencies...
[✓] Cloning api service...
[✓] Installing api dependencies (go mod download)...
[✓] Cloning relay service...
[✓] Installing relay dependencies (cargo fetch)...
[✓] Cloning shield service...
[✓] Installing shield dependencies (pip install)...
[✓] Domain nest assembly complete
Workspace summary:
.workspace/manuscripts/ (Go)
.workspace/api/ (Go, 32MB)
.workspace/relay/ (Rust, 450MB)
.workspace/shield/ (Python)
# Step 5: Build all services (this takes ~5 minutes)
$ make build
Building manuscript service...
✓ Manuscript built (1.2s)
Building core services...
Building api...
✓ API built (45s)
Building relay...
✓ Relay built (120s)
Building shield...
✓ Shield built (30s)
✓ Core services built
# Step 6: Start all services
$ make start
Starting domain services...
Checking infrastructure...
Starting api...
✓ api started (PID: 12345)
Starting relay...
✓ relay started (PID: 12346)
Starting shield...
✓ shield started (PID: 12347)
✓ All services started
Service Endpoints:
- API: http://localhost:8080/health
- Relay: http://localhost:8081/health
- Shield: http://localhost:8000/health
View logs:
- make logs-api
- make logs-relay
- make logs-shield
# Step 7: Verify everything is running
$ make health
Service Health Status:
├─ API (8080): ✅ Healthy (response time: 2ms)
├─ Relay (8081): ✅ Healthy (response time: 1ms)
├─ Shield (8000): ✅ Healthy (response time: 3ms)
├─ PostgreSQL: ✅ Ready
├─ Redis: ✅ Ready
└─ Overall: ✅ All services operational
# SUCCESS! You now have a complete, running Domain Nest
Example 2: Run Tests and See Results
Scenario
You want to verify that everything is working correctly by running the complete test suite.Step-by-Step Walkthrough
Copy
# Step 1: Run the full test orchestrator
$ cd /Users/alexarno/materi/domain
$ ./tests/sparki-test-orchestrator.sh all development
Sparki Test Orchestrator - Starting full test suite
Environment: development
Timestamp: 2025-12-16T14:30:00Z
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[PHASE 1/6] Unit Tests
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Running API unit tests (Go)...
• test_document_creation ... ✅ PASS (45ms)
• test_document_update ... ✅ PASS (38ms)
• test_document_delete ... ✅ PASS (32ms)
• test_permission_check ... ✅ PASS (52ms)
• test_error_handling ... ✅ PASS (28ms)
[✓] API: 5/5 tests passed (4.2s total)
Running Relay unit tests (Rust)...
• test_websocket_connect ... ✅ PASS (120ms)
• test_message_validation ... ✅ PASS (95ms)
• test_ot_transform ... ✅ PASS (145ms)
• test_presence_update ... ✅ PASS (88ms)
[✓] Relay: 4/4 tests passed (5.1s total)
Running Shield unit tests (Python)...
• test_user_creation ... ✅ PASS (156ms)
• test_jwt_validation ... ✅ PASS (142ms)
• test_permission_cache ... ✅ PASS (168ms)
[✓] Shield: 3/3 tests passed (6.2s total)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[PHASE 2/6] Integration Tests
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Testing cross-service communication...
• test_api_calls_shield_auth ... ✅ PASS (234ms)
• test_document_event_sync ... ✅ PASS (567ms)
• test_websocket_collaboration ... ✅ PASS (892ms)
• test_redis_stream_events ... ✅ PASS (445ms)
[✓] Integration: 4/4 tests passed (8.6s total)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[PHASE 3/6] Linting & Code Quality
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Running golangci-lint (API)...
✅ No issues found
Running cargo fmt + clippy (Relay)...
✅ Code formatting OK
✅ No clippy warnings
Running black + flake8 (Shield)...
✅ Code formatting OK
✅ No flake8 issues
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[PHASE 4/6] Security Scanning
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Running gosec (API)...
✅ No security issues found
Running cargo audit (Relay)...
✅ No vulnerable dependencies
Running bandit (Shield)...
✅ No security issues found
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[PHASE 5/6] Protocol Validation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Validating HTTP/REST protocol...
✅ Status codes valid
✅ Rate limiting configured
✅ Timeouts configured
Validating JWT authentication...
✅ Algorithm: HS256 ✓
✅ Required claims present ✓
✅ Token expiry configured ✓
Validating WebSocket protocol...
✅ Connection rules valid ✓
✅ Message types defined ✓
✅ Resource limits set ✓
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[PHASE 6/6] E2E Tests
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Running end-to-end workflows...
• Complete document workflow (create→edit→share→publish) ✅ PASS (1.2s)
• Real-time collaboration (2 users) ✅ PASS (2.1s)
• Permission verification (3 scenarios) ✅ PASS (1.8s)
[✓] E2E: 3/3 tests passed (5.1s total)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
TEST SUMMARY
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Total Tests Run: 77
Passed: 77 ✅
Failed: 0
Skipped: 0
Success Rate: 100%
Coverage Report:
Lines Covered: 590/590
Coverage: 100%
Execution Time:
Total: 42.3 seconds
Average per test: 0.55 seconds
Report Location:
JSON: test-reports/test-report-20251216_143000.json
Log: test-reports/orchestrator.log
✅ ALL TESTS PASSED - SYSTEM READY FOR DEPLOYMENT
# Step 2: View detailed JSON report (machine-readable)
$ cat test-reports/test-report-*.json | jq .
{
"timestamp": "2025-12-16T14:30:00Z",
"environment": "development",
"summary": {
"total_tests": 77,
"passed": 77,
"failed": 0,
"skipped": 0,
"success_rate": 1.0,
"duration_seconds": 42.3
},
"phases": {
"unit_tests": {
"api": 5,
"relay": 4,
"shield": 3
},
"integration_tests": 4,
"security_scan": 0,
"protocols": {
"http": "✓",
"jwt": "✓",
"websocket": "✓"
}
}
}
# Step 3: View human-readable log
$ tail -30 test-reports/orchestrator.log
[14:30:42.100] Running end-to-end workflows...
[14:30:42.234] ✓ Complete document workflow passed (1234ms)
[14:30:44.456] ✓ Real-time collaboration passed (2123ms)
[14:30:46.234] ✓ Permission verification passed (1789ms)
[14:30:48.100]
[14:30:48.100] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[14:30:48.100] ✅ TEST SUITE COMPLETE
[14:30:48.100] Total: 77 tests, All passed (100%)
[14:30:48.100] Duration: 42.3 seconds
[14:30:48.100] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Example 3: Debug a Service Issue
Scenario
Shield service is showing as unhealthy. You need to figure out why and fix it.Step-by-Step Walkthrough
Copy
# Step 1: Run health check to see which service is down
$ make health
Service Health Status:
├─ API (8080): ✅ Healthy (response time: 2ms)
├─ Relay (8081): ✅ Healthy (response time: 1ms)
├─ Shield (8000): ❌ UNHEALTHY (connection refused)
├─ PostgreSQL: ✅ Ready
├─ Redis: ✅ Ready
└─ Overall: ❌ System degraded - Shield unavailable
# Step 2: Check Shield's logs to see what's wrong
$ make logs-shield | tail -50
2025-12-16 14:35:22 | ERROR | [django.security] Forbidden (403): /health
2025-12-16 14:35:23 | ERROR | [django.security] PermissionDenied at /health
2025-12-16 14:35:23 | ERROR | Authentication middleware raised exception
# Step 3: Check the Shield service's recent changes
$ cat /Users/alexarno/materi/domain/sparki/shield.sparki.yml | grep -A 5 "health_check"
health_check:
endpoint: /health
timeout: 5000
expected_status: 200
interval: 10000
# Step 4: Connect to PostgreSQL to check database state
$ make db-shell
psql (15.2)
materi=> SELECT * FROM django_migrations WHERE app='auth' ORDER BY name DESC LIMIT 5;
id | app | name | applied
----+--------------------+---------------------------------------------------------+------------------------
10 | auth | 0010_add_permission_groups | 2025-12-16 10:30:00
9 | auth | 0009_alter_user_groups | 2025-12-16 10:30:00
8 | auth | 0008_alter_user_email | 2025-12-16 10:30:00
7 | auth | 0007_add_user_fields | 2025-12-16 10:30:00
6 | auth | 0006_initial | 2025-12-16 10:30:00
# All migrations present, database looks OK
materi=> \q
# Step 5: Check if Shield process is actually running
$ ps aux | grep shield
user 12347 0.5 2.3 543210 234567 S 14:30 0:15 python manage.py runserver
# Process is running, so let's restart it
$ make restart
# Step 6: Verify it's back
$ make health
Service Health Status:
├─ API (8080): ✅ Healthy
├─ Relay (8081): ✅ Healthy
├─ Shield (8000): ✅ Healthy (response time: 2ms)
├─ PostgreSQL: ✅ Ready
├─ Redis: ✅ Ready
└─ Overall: ✅ All services operational
# SUCCESS! Shield is back online.
Example 4: Monitor Metrics in Real-Time
Scenario
You want to watch metrics as requests come in to see how your system is performing.Step-by-Step Walkthrough
Copy
# Step 1: Open multiple terminal windows or use tmux
# Terminal 1: Watch overall health
$ watch -n 2 'make health'
# Output updates every 2 seconds:
Service Health Status:
├─ API (8080): ✅ Healthy (response time: 2ms)
├─ Relay (8081): ✅ Healthy (response time: 1ms)
├─ Shield (8000): ✅ Healthy (response time: 3ms)
├─ PostgreSQL: ✅ Ready
├─ Redis: ✅ Ready
└─ Overall: ✅ All services operational
# Terminal 2: Watch API metrics
$ watch -n 1 'curl -s http://localhost:9090/metrics | grep -E "(http_requests|http_request_duration|database_connections)" | tail -10'
# Output updates every second:
http_requests_total{method="GET",path="/health",status="200"} 1234
http_requests_total{method="POST",path="/api/v1/documents",status="200"} 567
http_request_duration_seconds{method="GET",path="/health"} 0.002
http_request_duration_seconds{method="POST",path="/api/v1/documents"} 0.156
database_connections{state="active"} 12
database_connections{state="idle"} 8
# Terminal 3: Watch Relay WebSocket metrics
$ watch -n 1 'curl -s http://localhost:9092/metrics | grep websocket'
# Output:
websocket_connections_total 145
websocket_connections_active 32
websocket_message_latency_seconds{p50="0.045",p95="0.234",p99="0.567"}
operation_transform_conflicts_total 3
# Terminal 4: Watch Shield authentication metrics
$ watch -n 1 'curl -s http://localhost:9091/metrics | grep jwt'
# Output:
jwt_validations_total 892
jwt_validation_errors_total 2
permission_cache_hits 234
permission_cache_misses 12
Example 5: Make a Request Through the Complete System
Scenario
You want to trace a request from API through Shield authentication and to Relay for real-time updates.Step-by-Step Walkthrough
Copy
# Step 1: Generate an authentication token (from Shield)
$ TOKEN=$(curl -X POST http://localhost:8000/api/v1/auth/token \
-H "Content-Type: application/json" \
-d '{"username":"testuser","password":"testpass"}' \
| jq -r .access_token)
$ echo $TOKEN
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ1c2VyLTE...
# Step 2: Create a document via API with that token
$ curl -X POST http://localhost:8080/api/v1/documents \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{
"title": "My Document",
"content": "Hello World",
"workspace_id": "workspace-123"
}' | jq .
{
"id": "doc-456",
"title": "My Document",
"content": "Hello World",
"created_at": "2025-12-16T14:40:00Z",
"owner": "user-123"
}
# Step 3: Check that the event was published to Redis
$ make redis-shell
redis> XREAD COUNT 5 STREAMS materi:events:documents 0
1) "materi:events:documents"
1) 1) "1702738800000-0"
2) 1) "type"
2) "document_created"
3) "document_id"
4) "doc-456"
5) "user_id"
6) "user-123"
7) "workspace_id"
8) "workspace-123"
redis> QUIT
# Step 4: Connect to Relay via WebSocket to receive real-time updates
$ wscat -c "ws://localhost:8081/ws?token=$TOKEN" 2>/dev/null
Connected (press CTRL+C to quit)
>
# In another terminal, update the document
$ curl -X PUT http://localhost:8080/api/v1/documents/doc-456 \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"content": "Hello World - Updated!"}' | jq .
{
"id": "doc-456",
"title": "My Document",
"content": "Hello World - Updated!",
"updated_at": "2025-12-16T14:41:00Z"
}
# Back in WebSocket terminal, you see the update in real-time:
< {"type":"document_updated","document_id":"doc-456","content":"Hello World - Updated!","timestamp":1702738860000}
# Step 5: Check metrics to see the requests
$ curl -s http://localhost:9090/metrics | grep -E 'method="POST"|method="PUT"'
http_requests_total{method="POST",path="/api/v1/documents",status="200"} 1
http_requests_total{method="PUT",path="/api/v1/documents",status="200"} 1
http_request_duration_seconds{method="POST",path="/api/v1/documents"} 0.234
http_request_duration_seconds{method="PUT",path="/api/v1/documents"} 0.189
$ curl -s http://localhost:9092/metrics | grep message_latency
websocket_message_latency_seconds{p50="0.012",p95="0.045",p99="0.089"}
# Step 6: Check Shield validation metrics
$ curl -s http://localhost:9091/metrics | grep jwt_validations
jwt_validations_total 3
jwt_validation_cache_hits 2
jwt_validation_cache_misses 1
# SUMMARY: You've traced a complete request:
# 1. Authenticated with Shield (JWT token generated)
# 2. Created document via API (HTTP POST)
# 3. Event published to Redis Streams
# 4. Real-time update delivered via Relay WebSocket
# 5. All metrics recorded and available for monitoring
Example 6: Add a New Environment (Staging)
Scenario
You want to create a staging environment separate from development.Step-by-Step Walkthrough
Copy
# Step 1: Create staging environment configuration
$ cat > /Users/alexarno/materi/domain/.env.staging << 'EOF'
# Staging environment configuration
ENVIRONMENT=staging
LOG_LEVEL=info
DATABASE_URL=postgresql://user:pass@staging-db:5432/materi_staging
REDIS_URL=redis://staging-redis:6379/0
JWT_SECRET=your-staging-jwt-secret-here
DEBUG=false
ENABLE_METRICS=true
ENABLE_TRACING=false
EOF
# Step 2: Load staging environment
$ export $(cat /Users/alexarno/materi/domain/.env.staging | xargs)
# Step 3: Setup workspace for staging
$ make setup
$ make assemble ENV=staging
# Step 4: Build with staging environment
$ make build
# Step 5: Start services pointing to staging infrastructure
$ docker-compose -f docker-compose.staging.yml up -d postgres redis
$ make start
# Step 6: Verify staging services are running
$ make health
Service Health Status:
├─ API (8080): ✅ Healthy (environment: staging)
├─ Relay (8081): ✅ Healthy (environment: staging)
├─ Shield (8000): ✅ Healthy (environment: staging)
└─ Overall: ✅ All services operational (staging)
# Step 7: Run staging tests
$ ./tests/sparki-test-orchestrator.sh all staging
[✓] All tests passed in staging environment (100%)
# SUCCESS! Staging environment is ready
Example 7: Deploy to Production (Manual Approval)
Scenario
After staging tests pass, you need to manually approve and deploy to production.Step-by-Step Walkthrough
Copy
# Step 1: Ensure all tests pass in staging
$ ./tests/sparki-test-orchestrator.sh all staging
✅ ALL TESTS PASSED (77/77)
# Step 2: Smoke tests in staging
$ make validate
[✓] Configuration validation passed
[✓] Security validation passed
[✓] Dependency check passed
# Step 3: Tag the release
$ git tag -a v1.2.3 -m "Release: Production deployment"
$ git push origin v1.2.3
# Step 4: This triggers GitHub Actions workflow (if set up)
# Check status: Navigate to GitHub Actions in your repo
# Or via CLI:
$ gh workflow view sparki-domain-deploy.yml
# Step 5: Wait for pre-deployment checks
# The workflow will:
# - Run all tests
# - Security scans
# - Build Docker images
# - Deploy to staging
# - Wait for manual approval
# Step 6: When prompted, approve production deployment
# Via GitHub UI: Actions → sparki-domain-deploy → Review deployments
# Or via CLI:
$ gh run list --workflow=sparki-domain-deploy.yml --status=waiting
$ gh run view <RUN_ID> --log
# Step 7: Approve the deployment
$ gh run approve <RUN_ID> --comment "Approved for production"
# Workflow continues:
# - Deploys to production
# - Runs smoke tests
# - Creates release notes
# - Tags release
# Step 8: Monitor production deployment
$ watch -n 5 'make health'
Service Health Status:
├─ API (8080): ✅ Healthy (version: v1.2.3)
├─ Relay (8081): ✅ Healthy (version: v1.2.3)
├─ Shield (8000): ✅ Healthy (version: v1.2.3)
└─ Overall: ✅ All services operational (PRODUCTION)
# Step 9: Verify production metrics
$ curl -s http://localhost:9090/metrics | grep version
build_info{version="v1.2.3",commit="abc123def456"}
# SUCCESS! Production deployment complete
Example 8: Rollback from Production
Scenario
Something went wrong in production and you need to quickly rollback.Step-by-Step Walkthrough
Copy
# Step 1: Identify the issue
$ make health
Service Health Status:
├─ API (8080): ❌ ERROR 500 (version: v1.2.3)
├─ Relay (8081): ✅ Healthy (version: v1.2.3)
├─ Shield (8000): ✅ Healthy (version: v1.2.3)
└─ Overall: ❌ Degraded
# Step 2: Check recent API logs
$ make logs-api | tail -100
2025-12-16 14:50:22 | ERROR | [api.service] Database connection pool exhausted
2025-12-16 14:50:23 | ERROR | [api.handler] Internal Server Error
# Step 3: Trigger rollback to previous version
$ git log --oneline | head -5
v1.2.3 Release: Production deployment
v1.2.2 Minor bug fix
v1.2.1 Database optimization
v1.2.0 Feature: New UI components
# Rollback to v1.2.2 (previous stable version)
$ git checkout v1.2.2
$ make build
$ make restart
# Step 4: Verify rollback
$ make health
Service Health Status:
├─ API (8080): ✅ Healthy (version: v1.2.2)
├─ Relay (8081): ✅ Healthy (version: v1.2.2)
├─ Shield (8000): ✅ Healthy (version: v1.2.2)
└─ Overall: ✅ All services operational
# Step 5: Run tests to confirm rollback is stable
$ ./tests/sparki-test-orchestrator.sh all production
✅ ALL TESTS PASSED (77/77)
# Step 6: Notify team
$ echo "⚠️ ROLLBACK COMPLETED: v1.2.3 → v1.2.2 at $(date)"
# Step 7: Later, investigate the issue and fix
# ... debugging and fixes happen here ...
# Then tag a new fixed version (v1.2.4)
# SUCCESS! Rollback successful, system stable
Quick Reference: Common Commands by Use Case
| Use Case | Command | Time |
|---|---|---|
| Full setup | make setup && make assemble && make build && make start | 15 min |
| Quick test | ./tests/sparki-test-orchestrator.sh unit | 5 min |
| All tests | ./tests/sparki-test-orchestrator.sh all | 45 min |
| Check health | make health | 5 sec |
| View logs | make logs or make logs-service SERVICE=api | 1 sec |
| Restart | make restart | 10 sec |
| Deploy | git push && approve in GitHub | 15 min |
| Rollback | git checkout <previous-tag> && make restart | 5 min |
| Debug | make health && make logs-<service> && make db-shell | varies |
Last Updated: 2025-12-16 Status: Ready to use Created For: Practical hands-on learning with real examples