Skip to main content

Overview

This runbook covers operational procedures for managing the Veritas prompt library using SO1 prompt management agents. Veritas is the centralized prompt repository that powers all SO1 agents, ensuring consistency, quality, and versioning of AI prompts across the platform. Purpose: Provide step-by-step instructions for creating, refining, and managing prompts in Veritas Scope: Prompt engineering, chain architecture, fragment management, version control, quality assurance Target Audience: Prompt engineers, AI engineers, agent developers

Prerequisites

  • Veritas prompt library access (GitHub: so1-io/veritas)
  • Control Plane API access (CONTROL_PLANE_API_KEY)
  • OpenAI API access (for prompt testing)
  • Agent execution permissions
  • OpenCode with prompt management agents installed
  • Git CLI (for Veritas repository management)
  • curl or API client
  • JSON/YAML editor
  • Prompt testing framework
  • Understanding of prompt engineering principles
  • Familiarity with chain-of-thought prompting
  • Knowledge of SO1 agent architecture
  • Basic understanding of LLM capabilities and limitations

Procedure 1: Refine Agent Prompt

Step 1: Identify Prompt for Refinement

Common triggers for prompt refinement:
  • Agent producing inconsistent outputs
  • Agent missing edge cases
  • Agent not following instructions
  • Agent verbosity issues (too long/short responses)
  • New requirements or capabilities added
# Get current agent prompt
curl -s https://veritas.so1.io/api/v1/prompts/agents/workflow-architect \
  -H "Authorization: Bearer ${CONTROL_PLANE_API_KEY}" | jq '.'
Example Current Prompt:
{
  "prompt_id": "prompt_workflow_architect_v1.2",
  "agent": "workflow-architect",
  "version": "1.2.0",
  "content": "You are an expert n8n workflow architect. Design efficient automation workflows...",
  "fragments": ["system_instructions", "workflow_best_practices", "output_format"],
  "performance": {
    "success_rate": 0.87,
    "avg_response_time_ms": 3200,
    "user_satisfaction": 4.1
  }
}

Step 2: Invoke Prompt Refiner Agent

curl -X POST https://control-plane.so1.io/api/v1/agents/prompt-refiner/refine \
  -H "Authorization: Bearer ${CONTROL_PLANE_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "prompt_id": "prompt_workflow_architect_v1.2",
    "refinement_goals": [
      "improve_consistency",
      "reduce_verbosity",
      "better_error_handling"
    ],
    "test_cases": [
      {
        "input": "Design a webhook processing workflow",
        "expected_output_characteristics": [
          "includes error handling nodes",
          "response under 2000 characters",
          "follows n8n best practices"
        ]
      },
      {
        "input": "Create a scheduled data sync workflow",
        "expected_output_characteristics": [
          "includes retry logic",
          "optimized for performance",
          "proper cron expression"
        ]
      }
    ],
    "constraints": {
      "max_tokens": 4000,
      "temperature": 0.7,
      "preserve_tone": "professional_technical"
    }
  }' | jq '.'
Expected Response:
{
  "refinement_id": "ref_5Xy9mPqRs",
  "status": "completed",
  "original_prompt_id": "prompt_workflow_architect_v1.2",
  "refined_prompt_id": "prompt_workflow_architect_v1.3",
  "changes": [
    {
      "type": "addition",
      "section": "instructions",
      "content": "Always include error handling nodes with retry logic."
    },
    {
      "type": "modification",
      "section": "output_format",
      "before": "Provide detailed explanations for each node.",
      "after": "Provide concise explanations (1-2 sentences per node)."
    },
    {
      "type": "addition",
      "section": "constraints",
      "content": "Keep total response under 2000 characters."
    }
  ],
  "test_results": {
    "test_cases_passed": 2,
    "test_cases_failed": 0,
    "improvements": {
      "consistency_score": "+0.15",
      "avg_response_length": "-35%",
      "error_handling_coverage": "+100%"
    }
  },
  "recommendation": "deploy_to_staging"
}

Step 3: Review Refined Prompt

# Get refined prompt details
curl -s https://control-plane.so1.io/api/v1/agents/prompt-refiner/refinements/ref_5Xy9mPqRs \
  -H "Authorization: Bearer ${CONTROL_PLANE_API_KEY}" | jq '.refined_prompt'

# Compare with original
curl -X POST https://control-plane.so1.io/api/v1/agents/prompt-refiner/compare \
  -H "Authorization: Bearer ${CONTROL_PLANE_API_KEY}" \
  -d '{
    "prompt_a": "prompt_workflow_architect_v1.2",
    "prompt_b": "prompt_workflow_architect_v1.3",
    "output_format": "diff"
  }' | jq -r '.diff'

Step 4: Test Refined Prompt

# Run A/B test with 100 sample inputs
curl -X POST https://control-plane.so1.io/api/v1/agents/prompt-refiner/ab-test \
  -H "Authorization: Bearer ${CONTROL_PLANE_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "prompt_a": "prompt_workflow_architect_v1.2",
    "prompt_b": "prompt_workflow_architect_v1.3",
    "sample_size": 100,
    "metrics": [
      "response_quality",
      "consistency",
      "execution_time",
      "token_usage"
    ]
  }' | jq '.'
A/B Test Results:
{
  "test_id": "ab_7TnVw9Kj2m",
  "results": {
    "prompt_a": {
      "avg_quality_score": 4.1,
      "consistency": 0.87,
      "avg_exec_time_ms": 3200,
      "avg_tokens": 1850
    },
    "prompt_b": {
      "avg_quality_score": 4.4,
      "consistency": 0.94,
      "avg_exec_time_ms": 2800,
      "avg_tokens": 1200
    }
  },
  "winner": "prompt_b",
  "confidence": 0.95,
  "recommendation": "deploy_to_production"
}

Step 5: Deploy Refined Prompt

# Promote to production in Veritas
curl -X POST https://veritas.so1.io/api/v1/prompts/promote \
  -H "Authorization: Bearer ${CONTROL_PLANE_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "prompt_id": "prompt_workflow_architect_v1.3",
    "environment": "production",
    "rollout_strategy": "gradual",
    "rollout_percentage": 10
  }'

# Monitor performance
watch -n 60 'curl -s https://veritas.so1.io/api/v1/prompts/prompt_workflow_architect_v1.3/metrics \
  -H "Authorization: Bearer ${CONTROL_PLANE_API_KEY}" | jq ".success_rate, .avg_response_time_ms"'

# Increase rollout if metrics are good
curl -X PATCH https://veritas.so1.io/api/v1/prompts/prompt_workflow_architect_v1.3/rollout \
  -H "Authorization: Bearer ${CONTROL_PLANE_API_KEY}" \
  -d '{"percentage": 50}'

# Full rollout after 24 hours of successful operation
curl -X PATCH https://veritas.so1.io/api/v1/prompts/prompt_workflow_architect_v1.3/rollout \
  -H "Authorization: Bearer ${CONTROL_PLANE_API_KEY}" \
  -d '{"percentage": 100}'

Procedure 2: Design Prompt Chain

Step 1: Define Chain Requirements

Identify when a chain is needed:
  • Complex task requiring multiple reasoning steps
  • Need for specialized sub-tasks
  • Output from one prompt feeds into another
  • Quality gating (verification/validation steps)
interface ChainSpec {
  name: string;
  purpose: string;
  steps: {
    name: string;
    prompt_template: string;
    inputs: string[];
    outputs: string[];
  }[];
  flow_control: 'sequential' | 'parallel' | 'conditional';
}

Step 2: Invoke Chain Architect Agent

curl -X POST https://control-plane.so1.io/api/v1/agents/chain-architect/design \
  -H "Authorization: Bearer ${CONTROL_PLANE_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "chain_name": "api_endpoint_generation",
    "purpose": "Generate complete API endpoint with validation, implementation, and tests",
    "steps": [
      {
        "name": "design_api_spec",
        "description": "Design API specification with request/response schemas",
        "inputs": ["user_requirements"],
        "outputs": ["api_specification"]
      },
      {
        "name": "generate_implementation",
        "description": "Generate Hono route handler with validation",
        "inputs": ["api_specification"],
        "outputs": ["route_code", "validation_schemas"]
      },
      {
        "name": "generate_tests",
        "description": "Generate test suite for endpoint",
        "inputs": ["api_specification", "route_code"],
        "outputs": ["test_code"]
      },
      {
        "name": "review_quality",
        "description": "Review generated code for quality and best practices",
        "inputs": ["route_code", "validation_schemas", "test_code"],
        "outputs": ["quality_report", "recommendations"]
      }
    ],
    "flow_control": "sequential",
    "error_handling": "retry_with_feedback"
  }' | jq '.'
Expected Response:
{
  "chain_id": "chain_api_endpoint_gen_v1",
  "structure": {
    "steps": [
      {
        "step_id": "step_1_design",
        "prompt_id": "prompt_api_spec_designer_v1.0",
        "type": "generative",
        "retry_policy": {
          "max_retries": 3,
          "backoff": "exponential"
        }
      },
      {
        "step_id": "step_2_implement",
        "prompt_id": "prompt_hono_code_gen_v2.1",
        "type": "generative",
        "depends_on": ["step_1_design"]
      },
      {
        "step_id": "step_3_test",
        "prompt_id": "prompt_test_generator_v1.5",
        "type": "generative",
        "depends_on": ["step_1_design", "step_2_implement"]
      },
      {
        "step_id": "step_4_review",
        "prompt_id": "prompt_code_reviewer_v1.0",
        "type": "validative",
        "depends_on": ["step_2_implement", "step_3_test"],
        "quality_gate": {
          "min_score": 0.85,
          "retry_on_fail": true
        }
      }
    ],
    "execution_strategy": "sequential_with_gates"
  },
  "estimated_tokens": 8500,
  "estimated_time_ms": 12000
}

Step 3: Test Chain Execution

# Execute chain with sample input
curl -X POST https://control-plane.so1.io/api/v1/chains/chain_api_endpoint_gen_v1/execute \
  -H "Authorization: Bearer ${CONTROL_PLANE_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "inputs": {
      "user_requirements": "Create GET /api/v1/users/:id endpoint that returns user details with authentication"
    },
    "config": {
      "environment": "staging",
      "collect_metrics": true
    }
  }' | jq '.'
Chain Execution Result:
{
  "execution_id": "exec_2mP7nQzXy",
  "status": "completed",
  "steps_completed": 4,
  "steps_failed": 0,
  "total_tokens": 8234,
  "total_time_ms": 11750,
  "outputs": {
    "api_specification": {
      "path": "/api/v1/users/:id",
      "method": "GET",
      "authentication": "bearer_token",
      "response_schema": "{ id, name, email, created_at }"
    },
    "route_code": "// Hono route implementation...",
    "validation_schemas": "// Zod schemas...",
    "test_code": "// Vitest test suite...",
    "quality_report": {
      "score": 0.92,
      "passed_checks": 15,
      "failed_checks": 0,
      "recommendations": ["Consider adding rate limiting"]
    }
  }
}

Step 4: Deploy Chain to Veritas

# Register chain in Veritas
curl -X POST https://veritas.so1.io/api/v1/chains \
  -H "Authorization: Bearer ${CONTROL_PLANE_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "chain_id": "chain_api_endpoint_gen_v1",
    "name": "API Endpoint Generation Chain",
    "description": "Complete API endpoint generation with validation and tests",
    "steps": [...],
    "version": "1.0.0",
    "tags": ["api", "code_generation", "hono"]
  }'

# Make available to agents
curl -X POST https://control-plane.so1.io/api/v1/agents/hono-backend/config \
  -H "Authorization: Bearer ${CONTROL_PLANE_API_KEY}" \
  -d '{
    "add_chains": ["chain_api_endpoint_gen_v1"]
  }'

Procedure 3: Manage Prompt Fragments

Step 1: Identify Reusable Prompt Components

Common fragment types:
  • System instructions: Role definitions, general behavior
  • Best practices: Domain-specific guidelines
  • Output formats: JSON schemas, markdown templates
  • Error handling: How to handle edge cases
  • Constraints: Token limits, formatting rules
# List existing fragments
curl -s https://veritas.so1.io/api/v1/fragments \
  -H "Authorization: Bearer ${CONTROL_PLANE_API_KEY}" \
  | jq '.fragments[] | {id, name, usage_count}'

Step 2: Create New Fragment with Fragment Curator

curl -X POST https://control-plane.so1.io/api/v1/agents/fragment-curator/create \
  -H "Authorization: Bearer ${CONTROL_PLANE_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "fragment_name": "error_handling_best_practices",
    "category": "best_practices",
    "content": "When generating code:\n1. Always include try-catch blocks for async operations\n2. Use proper error types (never throw strings)\n3. Include descriptive error messages\n4. Log errors with appropriate severity levels\n5. Return user-friendly error responses",
    "target_agents": ["hono-backend", "nextjs-frontend", "typescript-shared"],
    "version": "1.0.0",
    "metadata": {
      "author": "prompt-engineering-team",
      "created_date": "2026-03-10",
      "review_status": "approved"
    }
  }' | jq '.'
Expected Response:
{
  "fragment_id": "fragment_error_handling_bp_v1",
  "status": "created",
  "validation": {
    "grammar_check": "passed",
    "clarity_score": 0.94,
    "specificity_score": 0.89
  },
  "recommendations": [
    {
      "type": "enhancement",
      "suggestion": "Consider adding language-specific examples for TypeScript"
    }
  ]
}

Step 3: Test Fragment Impact

# A/B test prompts with and without fragment
curl -X POST https://control-plane.so1.io/api/v1/agents/fragment-curator/test-impact \
  -H "Authorization: Bearer ${CONTROL_PLANE_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "fragment_id": "fragment_error_handling_bp_v1",
    "test_agents": ["hono-backend"],
    "sample_size": 50,
    "metrics": ["error_handling_coverage", "code_quality", "user_satisfaction"]
  }' | jq '.'

Step 4: Deploy Fragment

# Add fragment to target agent prompts
curl -X POST https://veritas.so1.io/api/v1/fragments/fragment_error_handling_bp_v1/deploy \
  -H "Authorization: Bearer ${CONTROL_PLANE_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "agents": ["hono-backend", "nextjs-frontend", "typescript-shared"],
    "position": "before_output_format",
    "required": true
  }'

# Verify fragment is active
curl -s https://veritas.so1.io/api/v1/prompts/agents/hono-backend \
  -H "Authorization: Bearer ${CONTROL_PLANE_API_KEY}" \
  | jq '.fragments'

Step 5: Monitor Fragment Performance

# Get fragment usage analytics
curl -s https://veritas.so1.io/api/v1/fragments/fragment_error_handling_bp_v1/analytics \
  -H "Authorization: Bearer ${CONTROL_PLANE_API_KEY}" \
  | jq '{
    usage_count: .usage_count,
    avg_impact_on_quality: .metrics.avg_quality_improvement,
    agents_using: .agents_using
  }'

Procedure 4: Version Control and Rollback

Step 1: Tag Prompt Version

# Create version tag for prompt
curl -X POST https://veritas.so1.io/api/v1/prompts/prompt_workflow_architect_v1.3/tag \
  -H "Authorization: Bearer ${CONTROL_PLANE_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "tag": "stable",
    "notes": "Improved consistency and reduced verbosity. Tested with 100 samples.",
    "changelog": [
      "Added error handling requirements",
      "Reduced verbosity in output format",
      "Added response length constraints"
    ]
  }'

Step 2: Monitor Production Performance

# Set up alerts for prompt performance degradation
curl -X POST https://veritas.so1.io/api/v1/alerts \
  -H "Authorization: Bearer ${CONTROL_PLANE_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "prompt_id": "prompt_workflow_architect_v1.3",
    "metrics": [
      {
        "metric": "success_rate",
        "threshold": 0.90,
        "condition": "below"
      },
      {
        "metric": "avg_response_time_ms",
        "threshold": 5000,
        "condition": "above"
      }
    ],
    "notification_channels": ["slack_engineering", "email_oncall"]
  }'

# Check real-time metrics
curl -s https://veritas.so1.io/api/v1/prompts/prompt_workflow_architect_v1.3/metrics/realtime \
  -H "Authorization: Bearer ${CONTROL_PLANE_API_KEY}" \
  | jq '{success_rate, avg_response_time_ms, error_count}'

Step 3: Rollback if Needed

When prompt performance degrades:
# Immediate rollback to previous version
curl -X POST https://veritas.so1.io/api/v1/prompts/rollback \
  -H "Authorization: Bearer ${CONTROL_PLANE_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "current_prompt_id": "prompt_workflow_architect_v1.3",
    "target_version": "v1.2",
    "reason": "Success rate dropped to 0.82, below threshold",
    "immediate": true
  }'

# Verify rollback
curl -s https://veritas.so1.io/api/v1/prompts/agents/workflow-architect \
  -H "Authorization: Bearer ${CONTROL_PLANE_API_KEY}" \
  | jq '.version'

Verification Checklist

After completing prompt management operations, verify:

Troubleshooting

IssueSymptomsRoot CauseResolution
Inconsistent OutputsSame input produces different resultsTemperature too high, ambiguous instructionsLower temperature (0.3-0.5), add specific constraints
Prompt Too LongContext window errors, high costsExcessive verbosity, redundant instructionsUse fragments, remove redundant sections, increase conciseness
Chain Step FailuresChain execution stops mid-wayStep dependency failure, quality gate not metAdd retry logic, review quality gate thresholds
Fragment ConflictsContradictory instructionsMultiple fragments with conflicting guidanceReview fragment order, merge conflicting fragments
Low Success RateAgent outputs don’t meet requirementsUnclear instructions, missing edge casesAdd examples, specify edge case handling
High Token UsageCosts increasing rapidlyVerbose prompts, unnecessary chain stepsOptimize prompt length, remove redundant steps
Slow Response TimesAgent execution takes >10sLarge context, complex reasoningSimplify prompt, use chain for multi-step reasoning
Rollback FailedPrevious version not availableVersion not tagged, deletedAlways tag stable versions, implement retention policy

Detailed Troubleshooting: Inconsistent Outputs

# Analyze output variance
curl -X POST https://control-plane.so1.io/api/v1/agents/prompt-refiner/analyze-variance \
  -H "Authorization: Bearer ${CONTROL_PLANE_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "prompt_id": "prompt_workflow_architect_v1.3",
    "test_input": "Design a webhook processing workflow",
    "runs": 20
  }' | jq '.variance_metrics'

# Common fixes:

# 1. Lower temperature
curl -X PATCH https://veritas.so1.io/api/v1/prompts/prompt_workflow_architect_v1.3 \
  -H "Authorization: Bearer ${CONTROL_PLANE_API_KEY}" \
  -d '{"temperature": 0.3}'

# 2. Add output format constraints
curl -X POST https://control-plane.so1.io/api/v1/agents/prompt-refiner/add-constraint \
  -H "Authorization: Bearer ${CONTROL_PLANE_API_KEY}" \
  -d '{
    "prompt_id": "prompt_workflow_architect_v1.3",
    "constraint": "Always output valid JSON with keys: name, nodes, connections"
  }'

# 3. Use seed for reproducibility (if supported by LLM)
curl -X PATCH https://veritas.so1.io/api/v1/prompts/prompt_workflow_architect_v1.3 \
  -H "Authorization: Bearer ${CONTROL_PLANE_API_KEY}" \
  -d '{"seed": 42}'


Best Practices

Prompt Engineering

  1. Be specific and clear: Avoid ambiguous language, provide concrete examples
  2. Use structured output formats: JSON schemas, markdown templates
  3. Include constraints: Token limits, response length, formatting rules
  4. Test with edge cases: Empty inputs, very long inputs, malformed data
  5. Version everything: Tag stable versions, maintain changelog

Chain Design

  1. Keep chains focused: Each step should have a single, clear purpose
  2. Add quality gates: Validate outputs before proceeding to next step
  3. Implement error handling: Retry with feedback, fallback strategies
  4. Minimize token usage: Only pass necessary context between steps
  5. Monitor chain performance: Track success rates, execution times per step

Fragment Management

  1. Keep fragments atomic: Single responsibility per fragment
  2. Use descriptive names: Clear, searchable fragment IDs
  3. Document usage: Specify which agents should use each fragment
  4. Test impact: A/B test before deploying to all agents
  5. Regular review: Audit fragments quarterly, remove unused ones

Version Control

  1. Tag all stable versions: Use semantic versioning (major.minor.patch)
  2. Maintain rollback capability: Keep previous 3 versions accessible
  3. Document changes: Clear changelogs for every version
  4. Gradual rollout: Use canary deployments (10% → 50% → 100%)
  5. Monitor after deployment: Alert on performance degradation