Skip to main content

Telemetry

The Evolution System relies on comprehensive telemetry to learn and improve. This document defines the data capture schema and storage design.

Execution Record Schema

Every workflow execution captures a complete execution record:
execution_record:
  # Identity
  execution_id: uuid
  request_fingerprint: hash(raw_input + audience + type)
  
  # Inputs
  raw_input: string
  input_complexity_score: float
  detected_intent_signals: string[]
  
  # Process - Conceptual Design
  conceptual_design:
    sections_planned: int
    concepts_identified: int
    comparisons_required: int
    estimated_word_count: int
    generation_latency_ms: int
    prompt_tokens_used: int
  
  # Process - Drafts
  drafts:
    - version: int
      word_count: int
      section_count: int
      generation_latency_ms: int
      quality_scores:
        technical_accuracy: float
        clarity_structure: float
        completeness: float
        audience_fit: float
        actionability: float
        overall: float
      feedback_items_count: int
      critical_issues: string[]
  
  # Outcomes
  final_quality_score: float
  iterations_required: int
  total_duration_seconds: int
  total_tokens_consumed: int
  
  # Human Signals (post-publication)
  pr_approved: boolean
  pr_approval_latency_hours: float
  pr_changes_requested: string[]
  post_publish_edits: int
  post_publish_edit_diff_size: int
  page_views_30d: int
  time_on_page_avg_seconds: float
  feedback_signals:
    thumbs_up: int
    thumbs_down: int
    comments: string[]

Corpus Database Schema

-- Document corpus for pattern analysis
CREATE TABLE generated_documents (
  id UUID PRIMARY KEY,
  execution_id UUID REFERENCES executions(id),
  
  -- Content
  raw_input TEXT,
  final_output TEXT,
  document_type VARCHAR(50),
  audience VARCHAR(50),
  
  -- Metrics
  input_complexity_score FLOAT,
  final_quality_score FLOAT,
  iterations_required INT,
  total_duration_seconds INT,
  total_tokens_consumed INT,
  
  -- Timestamps
  created_at TIMESTAMP,
  published_at TIMESTAMP,
  
  -- Human Signals
  pr_approved BOOLEAN,
  post_publish_edits INT,
  page_views_30d INT
);

-- Quality scores per draft
CREATE TABLE draft_quality_scores (
  id UUID PRIMARY KEY,
  document_id UUID REFERENCES generated_documents(id),
  draft_version INT,
  
  technical_accuracy FLOAT,
  clarity_structure FLOAT,
  completeness FLOAT,
  audience_fit FLOAT,
  actionability FLOAT,
  overall FLOAT
);

Query Patterns

Find High-Quality Examples

SELECT * FROM generated_documents 
WHERE final_quality_score >= 0.9 
  AND pr_approved = true
  AND post_publish_edits < 3
ORDER BY page_views_30d DESC;

Identify Problem Patterns

SELECT document_type, audience,
       AVG(iterations_required) as avg_iterations,
       AVG(final_quality_score) as avg_quality
FROM generated_documents
GROUP BY document_type, audience
HAVING AVG(iterations_required) > 3;

Data Retention

Data TypeRetention Period
Full documents1 year
Execution metrics2 years
Aggregated statisticsIndefinite
Human feedback2 years