Automation Operations

Overview

This runbook covers operational procedures for managing SO1 automation agents, including n8n workflow deployment, webhook engineering, and schedule optimization. These procedures ensure reliable automation infrastructure across all stages of the FORGE pipeline. Purpose: Provide step-by-step instructions for deploying and managing automation workflows, webhooks, and scheduled tasks Scope: n8n workflow operations, webhook configuration, schedule management, automation debugging Target Audience: Automation engineers, DevOps operators, workflow architects

Prerequisites

Required Access

n8n instance access (https://n8n.so1.io)
Control Plane API access (CONTROL_PLANE_API_KEY)
Railway project access (n8n service)
GitHub repository access (so1-io/n8n-workflows)
Webhook endpoints access (various integrated services)

Required Tools

curl or API client (Postman, Insomnia)
n8n CLI (n8n command)
Railway CLI (railway command)
jq for JSON parsing
OpenCode with automation agents installed

Required Knowledge

Understanding of n8n workflow structure
Familiarity with webhook protocols (REST, GraphQL)
Basic knowledge of cron expressions
Understanding of SO1 automation architecture

Procedure 1: Deploy n8n Workflow

Step 1: Prepare Workflow Definition

Create or update workflow JSON:

# Example workflow: Webhook processing pipeline
cat > webhook-pipeline.json <<'EOF'
{
  "name": "Webhook Processing Pipeline",
  "nodes": [
    {
      "name": "Webhook Trigger",
      "type": "n8n-nodes-base.webhook",
      "position": [250, 300],
      "parameters": {
        "path": "inbound/events",
        "httpMethod": "POST",
        "responseMode": "onReceived"
      }
    },
    {
      "name": "Validate Payload",
      "type": "n8n-nodes-base.function",
      "position": [450, 300],
      "parameters": {
        "functionCode": "// Validation logic\nreturn items;"
      }
    },
    {
      "name": "Process Event",
      "type": "n8n-nodes-base.httpRequest",
      "position": [650, 300],
      "parameters": {
        "url": "https://control-plane.so1.io/api/v1/events",
        "method": "POST"
      }
    }
  ],
  "connections": {
    "Webhook Trigger": {
      "main": [[{"node": "Validate Payload", "type": "main", "index": 0}]]
    },
    "Validate Payload": {
      "main": [[{"node": "Process Event", "type": "main", "index": 0}]]
    }
  }
}
EOF

Step 2: Validate Workflow with Workflow Architect

curl -X POST https://control-plane.so1.io/api/v1/agents/workflow-architect/validate \
  -H "Authorization: Bearer ${CONTROL_PLANE_API_KEY}" \
  -H "Content-Type: application/json" \
  -d @webhook-pipeline.json | jq '.'

Expected Response:

{
  "validation_status": "passed",
  "workflow_id": "workflow_5Xy9mPqRs",
  "issues": [],
  "recommendations": [
    {
      "severity": "info",
      "message": "Consider adding error handling node",
      "node": "Process Event"
    }
  ],
  "estimated_execution_time_ms": 250
}

Step 3: Deploy to n8n

# Import workflow to n8n
curl -X POST https://n8n.so1.io/api/v1/workflows \
  -H "X-N8N-API-KEY: ${N8N_API_KEY}" \
  -H "Content-Type: application/json" \
  -d @webhook-pipeline.json | jq '.id'

# Activate workflow
WORKFLOW_ID=$(jq -r '.id' < response.json)
curl -X PATCH https://n8n.so1.io/api/v1/workflows/${WORKFLOW_ID} \
  -H "X-N8N-API-KEY: ${N8N_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"active": true}'

Step 4: Verify Deployment

# Test webhook endpoint
curl -X POST https://n8n.so1.io/webhook/inbound/events \
  -H "Content-Type: application/json" \
  -d '{
    "event_type": "test",
    "payload": {"test": true},
    "timestamp": "2026-03-10T14:30:00Z"
  }' -v

# Check execution history
curl -s https://n8n.so1.io/api/v1/executions?workflowId=${WORKFLOW_ID} \
  -H "X-N8N-API-KEY: ${N8N_API_KEY}" | jq '.data[0] | {
    id: .id,
    status: .finished,
    startedAt: .startedAt,
    stoppedAt: .stoppedAt
  }'

Procedure 2: Configure Webhook Engineering

Step 1: Design Webhook Endpoint

Use Webhook Engineer agent to generate endpoint specification:

curl -X POST https://control-plane.so1.io/api/v1/agents/webhook-engineer/design \
  -H "Authorization: Bearer ${CONTROL_PLANE_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "endpoint_name": "github_push_handler",
    "protocol": "REST",
    "method": "POST",
    "authentication": "hmac_sha256",
    "payload_schema": {
      "repository": "string",
      "branch": "string",
      "commits": "array"
    },
    "response_format": "json",
    "rate_limit": "100/minute"
  }' | jq '.'

Expected Response:

{
  "endpoint_id": "endpoint_3PqRs7TnVw",
  "specification": {
    "path": "/webhooks/github/push",
    "method": "POST",
    "headers": {
      "X-GitHub-Event": "push",
      "X-Hub-Signature-256": "sha256=..."
    },
    "payload_validation": {
      "type": "object",
      "required": ["repository", "branch", "commits"]
    },
    "authentication": {
      "type": "hmac_sha256",
      "secret_env": "GITHUB_WEBHOOK_SECRET"
    },
    "rate_limiting": {
      "requests": 100,
      "window": "1m",
      "strategy": "sliding_window"
    }
  },
  "implementation_code": "// Generated TypeScript code..."
}

Step 2: Implement Webhook Handler

// Generated by webhook-engineer agent
import { Hono } from 'hono';
import { createHmac } from 'crypto';

const app = new Hono();

app.post('/webhooks/github/push', async (c) => {
  // Verify HMAC signature
  const signature = c.req.header('X-Hub-Signature-256');
  const payload = await c.req.text();
  const secret = process.env.GITHUB_WEBHOOK_SECRET!;
  
  const hmac = createHmac('sha256', secret);
  hmac.update(payload);
  const expectedSignature = `sha256=${hmac.digest('hex')}`;
  
  if (signature !== expectedSignature) {
    return c.json({ error: 'Invalid signature' }, 401);
  }
  
  // Parse and validate payload
  const data = JSON.parse(payload);
  
  // Trigger n8n workflow
  await fetch('https://n8n.so1.io/webhook/github/push-handler', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify(data)
  });
  
  return c.json({ status: 'accepted' }, 202);
});

export default app;

Step 3: Deploy Webhook Handler

# Deploy to Railway using hono-backend agent
curl -X POST https://control-plane.so1.io/api/v1/agents/hono-backend/deploy \
  -H "Authorization: Bearer ${CONTROL_PLANE_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "service_name": "webhook-handler",
    "code": "...",
    "environment": "production",
    "railway_project": "so1-webhooks"
  }'

Step 4: Register Webhook with External Service

# Example: Register with GitHub
curl -X POST https://api.github.com/repos/so1-io/so1-agents/hooks \
  -H "Authorization: token ${GITHUB_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "config": {
      "url": "https://webhooks.so1.io/webhooks/github/push",
      "content_type": "json",
      "secret": "'"${GITHUB_WEBHOOK_SECRET}"'"
    },
    "events": ["push"],
    "active": true
  }'

Procedure 3: Optimize Workflow Schedules

Step 1: Analyze Current Schedule Load

# Get all scheduled workflows
curl -s https://n8n.so1.io/api/v1/workflows?filter=active:true \
  -H "X-N8N-API-KEY: ${N8N_API_KEY}" \
  | jq '[.data[] | select(.nodes[].type | contains("schedule"))] | length'

# Check execution patterns
curl -s https://control-plane.so1.io/api/v1/analytics/workflow-executions \
  -H "Authorization: Bearer ${CONTROL_PLANE_API_KEY}" \
  -d '{"timeframe": "24h", "group_by": "hour"}' \
  | jq '.executions_per_hour'

Step 2: Invoke Schedule Optimizer Agent

curl -X POST https://control-plane.so1.io/api/v1/agents/schedule-optimizer/optimize \
  -H "Authorization: Bearer ${CONTROL_PLANE_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "workflows": [
      {
        "id": "workflow_backup_daily",
        "current_schedule": "0 2 * * *",
        "priority": "high",
        "estimated_duration_min": 30
      },
      {
        "id": "workflow_metrics_sync",
        "current_schedule": "*/15 * * * *",
        "priority": "medium",
        "estimated_duration_min": 5
      },
      {
        "id": "workflow_report_generation",
        "current_schedule": "0 8 * * 1",
        "priority": "low",
        "estimated_duration_min": 45
      }
    ],
    "constraints": {
      "peak_hours": ["09:00-17:00"],
      "max_concurrent": 5,
      "avoid_overlap": true
    }
  }' | jq '.'

Expected Response:

{
  "optimization_id": "opt_9Kj2mP7nQz",
  "recommendations": [
    {
      "workflow_id": "workflow_backup_daily",
      "current_schedule": "0 2 * * *",
      "recommended_schedule": "0 3 * * *",
      "reason": "Avoid overlap with metrics sync peak load",
      "impact": {
        "resource_utilization": "-15%",
        "execution_reliability": "+10%"
      }
    },
    {
      "workflow_id": "workflow_metrics_sync",
      "current_schedule": "*/15 * * * *",
      "recommended_schedule": "*/20 * * * *",
      "reason": "Reduce execution frequency during low-change periods",
      "impact": {
        "cost_savings": "$45/month",
        "data_freshness": "-5min average"
      }
    }
  ],
  "overall_impact": {
    "resource_utilization": "-12%",
    "cost_savings": "$67/month",
    "reliability_improvement": "+8%"
  }
}

Step 3: Apply Schedule Changes

# Update workflow schedules in n8n
for workflow in $(jq -r '.recommendations[].workflow_id' < optimization.json); do
  NEW_SCHEDULE=$(jq -r ".recommendations[] | select(.workflow_id == \"$workflow\") | .recommended_schedule" < optimization.json)
  
  curl -X PATCH https://n8n.so1.io/api/v1/workflows/${workflow} \
    -H "X-N8N-API-KEY: ${N8N_API_KEY}" \
    -H "Content-Type: application/json" \
    -d "{
      \"nodes\": [{
        \"type\": \"n8n-nodes-base.scheduleTrigger\",
        \"parameters\": {
          \"cronExpression\": \"${NEW_SCHEDULE}\"
        }
      }]
    }"
  
  echo "Updated ${workflow} to schedule: ${NEW_SCHEDULE}"
done

Step 4: Monitor Post-Optimization Performance

# Monitor for 24 hours after changes
curl -s https://control-plane.so1.io/api/v1/analytics/workflow-performance \
  -H "Authorization: Bearer ${CONTROL_PLANE_API_KEY}" \
  -d '{
    "optimization_id": "opt_9Kj2mP7nQz",
    "timeframe": "24h",
    "metrics": ["execution_count", "avg_duration", "failure_rate", "resource_usage"]
  }' | jq '.'

Procedure 4: Debug Workflow Failures

Step 1: Identify Failed Executions

# Get recent failed executions
curl -s https://n8n.so1.io/api/v1/executions?status=error&limit=10 \
  -H "X-N8N-API-KEY: ${N8N_API_KEY}" \
  | jq '.data[] | {
    id: .id,
    workflowId: .workflowId,
    startedAt: .startedAt,
    error: .data.resultData.error
  }'

Step 2: Retrieve Execution Details

# Get detailed execution data
EXECUTION_ID="exec_7PqRs4TnVw"
curl -s https://n8n.so1.io/api/v1/executions/${EXECUTION_ID} \
  -H "X-N8N-API-KEY: ${N8N_API_KEY}" \
  | jq '.data.resultData'

Example Error Output:

{
  "error": {
    "message": "Request failed with status code 429",
    "node": "Process Event",
    "timestamp": "2026-03-10T14:45:23Z",
    "stack": "Error: Request failed...",
    "context": {
      "url": "https://control-plane.so1.io/api/v1/events",
      "method": "POST",
      "statusCode": 429
    }
  }
}

Step 3: Common Failure Patterns

Error Type	Symptom	Root Cause	Resolution
Rate Limit (429)	Requests rejected	Too many API calls	Add rate limiting node, implement backoff
Timeout	Execution hangs	Long-running operation	Increase timeout, split into smaller workflows
Authentication (401)	Unauthorized	Expired credentials	Rotate API keys, check secret management
Payload Validation	400 Bad Request	Invalid data format	Add validation node, update payload schema
Missing Node Data	Workflow incomplete	Node dependency failure	Add error handling, check node connections

Step 4: Apply Fixes and Retry

# Update workflow with fixes
curl -X PATCH https://n8n.so1.io/api/v1/workflows/${WORKFLOW_ID} \
  -H "X-N8N-API-KEY: ${N8N_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "nodes": [
      {
        "name": "Rate Limit Handler",
        "type": "n8n-nodes-base.wait",
        "parameters": {
          "amount": 1,
          "unit": "seconds"
        }
      }
    ]
  }'

# Retry failed execution
curl -X POST https://n8n.so1.io/api/v1/executions/${EXECUTION_ID}/retry \
  -H "X-N8N-API-KEY: ${N8N_API_KEY}"

Verification Checklist

After completing automation operations, verify:

Troubleshooting

Issue: Webhook Not Receiving Requests

Symptoms: External service reports success, but n8n shows no executions Possible Causes:

Webhook URL incorrect
Workflow not activated
Firewall blocking requests

Resolution:

# Verify webhook URL
curl -s https://n8n.so1.io/api/v1/workflows/${WORKFLOW_ID} \
  -H "X-N8N-API-KEY: ${N8N_API_KEY}" \
  | jq '.nodes[] | select(.type == "n8n-nodes-base.webhook") | .parameters'

# Check workflow active status
curl -s https://n8n.so1.io/api/v1/workflows/${WORKFLOW_ID} \
  -H "X-N8N-API-KEY: ${N8N_API_KEY}" \
  | jq '.active'

# Test webhook directly
curl -X POST https://n8n.so1.io/webhook/YOUR_WEBHOOK_PATH \
  -H "Content-Type: application/json" \
  -d '{"test": true}' -v

Issue: Scheduled Workflow Not Executing

Symptoms: Schedule trigger configured, but no executions at expected times Possible Causes:

Invalid cron expression
n8n service restarted during execution window
Schedule trigger node disabled

Resolution:

# Validate cron expression
curl -X POST https://control-plane.so1.io/api/v1/utils/validate-cron \
  -H "Authorization: Bearer ${CONTROL_PLANE_API_KEY}" \
  -d '{"expression": "*/15 * * * *"}' | jq '.next_executions'

# Check n8n uptime and restarts
railway logs --project so1-n8n --service n8n | grep -i restart

# Manually trigger workflow to test
curl -X POST https://n8n.so1.io/api/v1/workflows/${WORKFLOW_ID}/execute \
  -H "X-N8N-API-KEY: ${N8N_API_KEY}"

Issue: Workflow Execution Extremely Slow

Symptoms: Executions taking 10x longer than expected Possible Causes:

Large payload processing
External API latency
Inefficient node configuration

Resolution:

# Analyze execution performance
curl -s https://n8n.so1.io/api/v1/executions/${EXECUTION_ID} \
  -H "X-N8N-API-KEY: ${N8N_API_KEY}" \
  | jq '.data.resultData.runData | to_entries[] | {
    node: .key,
    duration_ms: (.value[0].executionTime // 0)
  } | select(.duration_ms > 1000)'

# Use workflow-architect to optimize
curl -X POST https://control-plane.so1.io/api/v1/agents/workflow-architect/optimize \
  -H "Authorization: Bearer ${CONTROL_PLANE_API_KEY}" \
  -d '{"workflow_id": "'"${WORKFLOW_ID}"'", "optimization_target": "performance"}'

Workflow Architect Agent

n8n workflow design and optimization

Webhook Engineer Agent

Webhook endpoint design and implementation

Schedule Optimizer Agent

Cron schedule optimization

Orchestration Runbook

Factory Orchestrator workflows

Emergency Procedures

Emergency Workflow Shutdown

When a workflow must be stopped immediately (e.g., causing system overload):

# Deactivate workflow immediately
curl -X PATCH https://n8n.so1.io/api/v1/workflows/${WORKFLOW_ID} \
  -H "X-N8N-API-KEY: ${N8N_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"active": false}'

# Stop all running executions
curl -X POST https://n8n.so1.io/api/v1/executions/stop-all \
  -H "X-N8N-API-KEY: ${N8N_API_KEY}" \
  -d '{"workflowId": "'"${WORKFLOW_ID}"'"}'

Emergency Webhook Disable

When a webhook is receiving malicious traffic:

# Disable webhook at n8n level
curl -X PATCH https://n8n.so1.io/api/v1/workflows/${WORKFLOW_ID} \
  -H "X-N8N-API-KEY: ${N8N_API_KEY}" \
  -d '{"active": false}'

# Block at Railway ingress (if applicable)
railway run --project so1-webhooks \
  'railway plugin:add @railway/firewall && railway firewall:block-ip <ATTACKER_IP>'

# Revoke webhook at source (e.g., GitHub)
curl -X DELETE https://api.github.com/repos/so1-io/so1-agents/hooks/${HOOK_ID} \
  -H "Authorization: token ${GITHUB_TOKEN}"

Runbooks

Domain Runbooks

Overview

Prerequisites

Procedure 1: Deploy n8n Workflow

Step 1: Prepare Workflow Definition

Step 2: Validate Workflow with Workflow Architect

Step 3: Deploy to n8n

Step 4: Verify Deployment

Procedure 2: Configure Webhook Engineering

Step 1: Design Webhook Endpoint

Step 2: Implement Webhook Handler

Step 3: Deploy Webhook Handler

Step 4: Register Webhook with External Service

Procedure 3: Optimize Workflow Schedules

Step 1: Analyze Current Schedule Load

Step 2: Invoke Schedule Optimizer Agent

Step 3: Apply Schedule Changes

Step 4: Monitor Post-Optimization Performance

Procedure 4: Debug Workflow Failures

Step 1: Identify Failed Executions

Step 2: Retrieve Execution Details

Step 3: Common Failure Patterns

Step 4: Apply Fixes and Retry

Verification Checklist

Troubleshooting

Issue: Webhook Not Receiving Requests

Issue: Scheduled Workflow Not Executing

Issue: Workflow Execution Extremely Slow

Workflow Architect Agent

Webhook Engineer Agent

Schedule Optimizer Agent

Orchestration Runbook

Emergency Procedures

Emergency Workflow Shutdown

Emergency Webhook Disable

Runbooks

Domain Runbooks

​Overview

​Prerequisites

​Procedure 1: Deploy n8n Workflow

​Step 1: Prepare Workflow Definition

​Step 2: Validate Workflow with Workflow Architect

​Step 3: Deploy to n8n

​Step 4: Verify Deployment

​Procedure 2: Configure Webhook Engineering

​Step 1: Design Webhook Endpoint

​Step 2: Implement Webhook Handler

​Step 3: Deploy Webhook Handler

​Step 4: Register Webhook with External Service

​Procedure 3: Optimize Workflow Schedules

​Step 1: Analyze Current Schedule Load

​Step 2: Invoke Schedule Optimizer Agent

​Step 3: Apply Schedule Changes

​Step 4: Monitor Post-Optimization Performance

​Procedure 4: Debug Workflow Failures

​Step 1: Identify Failed Executions

​Step 2: Retrieve Execution Details

​Step 3: Common Failure Patterns

​Step 4: Apply Fixes and Retry

​Verification Checklist

​Troubleshooting

​Issue: Webhook Not Receiving Requests

​Issue: Scheduled Workflow Not Executing

​Issue: Workflow Execution Extremely Slow

​Related Resources

Workflow Architect Agent

Webhook Engineer Agent

Schedule Optimizer Agent

Orchestration Runbook

​Emergency Procedures

​Emergency Workflow Shutdown

​Emergency Webhook Disable

Overview

Prerequisites

Procedure 1: Deploy n8n Workflow

Step 1: Prepare Workflow Definition

Step 2: Validate Workflow with Workflow Architect

Step 3: Deploy to n8n

Step 4: Verify Deployment

Procedure 2: Configure Webhook Engineering

Step 1: Design Webhook Endpoint

Step 2: Implement Webhook Handler

Step 3: Deploy Webhook Handler

Step 4: Register Webhook with External Service

Procedure 3: Optimize Workflow Schedules

Step 1: Analyze Current Schedule Load

Step 2: Invoke Schedule Optimizer Agent

Step 3: Apply Schedule Changes

Step 4: Monitor Post-Optimization Performance

Procedure 4: Debug Workflow Failures

Step 1: Identify Failed Executions

Step 2: Retrieve Execution Details

Step 3: Common Failure Patterns

Step 4: Apply Fixes and Retry

Verification Checklist

Troubleshooting

Issue: Webhook Not Receiving Requests

Issue: Scheduled Workflow Not Executing

Issue: Workflow Execution Extremely Slow

Related Resources

Emergency Procedures

Emergency Workflow Shutdown

Emergency Webhook Disable