​

Django Multi-Platform Scraper Management CLI (dj.sh)

​

Table of Contents

• Quick Start
• Basic Django Commands
• Development Environment Management
• Platform & Application Management
• Testing & Validation
• Database Management
• Development Workflows
• API Testing
• Common Use Cases
• Troubleshooting

​

Quick Start

# First time setup
./dj.sh mig              # Apply database migrations
./dj.sh devstart         # Set up development environment
./dj.sh run              # Start development server

# Check everything is working
./dj.sh test_auth        # Test authentication
./dj.sh devstatus        # Check environment status

​

Basic Django Commands

​

run - Start Development Server

./dj.sh run [options]

• Daily development server startup
• Testing with different ports
• Making server accessible to network

​

mkmig - Create Migrations

./dj.sh mkmig [app_name]

• After modifying Django models
• Creating initial migrations for new apps
• Preparing database schema changes

​

mig - Apply Migrations

./dj.sh mig [app_name] [migration_name]

• Initial database setup
• Applying schema changes
• Updating database structure

​

sudo - Create Superuser

./dj.sh sudo

• Username
• Email address
• Password (entered twice)

• Initial admin account creation
• Creating additional admin users
• Accessing Django admin interface

​

shell - Django Shell

./dj.sh shell

• Interactive model testing
• Data manipulation
• Debugging database queries
• Testing Django functionality

​

check - System Check

./dj.sh check

• Pre-deployment validation
• Identifying configuration issues
• Checking for deprecated code

​

test - Run Tests

./dj.sh test [app_name] [test_class] [test_method]

• Running full test suite
• Testing specific functionality
• Continuous integration validation

​

Development Environment Management

​

devstart / setup - Setup Development Environment

./dj.sh devstart [options]

• NOTION_TOKEN - Notion API token
• FEATURE_DB_ID - Notion features database ID
• TIER_DB_ID - Notion tiers database ID
• IO_DB_ID - Notion I/O database ID
• TAG_DB_ID - Notion tags database ID
• KEYWORD_DB_ID - Notion keywords database ID

• User account with specified tier subscription
• Platform (Notion workspace) configuration
• Two sample applications (Skyflow, Nexus)
• Data source connections
• Platform access permissions

• Initial development setup
• Creating test environments
• Onboarding new developers

​

devstart_full - Full Development Setup

./dj.sh devstart_full [additional_options]

./dj.sh devstart --tier pro --username devuser --clear-cache

• Complete feature testing
• Demo environments
• Full integration testing

​

devstart_minimal - Minimal Development Setup

./dj.sh devstart_minimal [additional_options]

./dj.sh devstart --tier free --username testuser

• Testing tier limitations
• Basic functionality verification
• Minimal resource usage

​

devclear / clear - Clear Development Data

./dj.sh devclear [options]

• Applications and data sources
• Platform access permissions
• User subscriptions
• Cache entries (if specified)
• API logs (if specified)

• Clean slate development
• Removing test data
• Preparing for fresh setup

​

dryclear / preview - Preview Cleanup

./dj.sh dryclear [options]

• Users and their applications
• Platform connections
• Subscription details
• Data source counts

• Verifying cleanup scope
• Understanding data relationships
• Safe exploration of cleanup impact

​

devclear_user - Clear Specific User

./dj.sh devclear_user <username> [options]

• Removing specific test users
• Targeted data cleanup
• User account management

​

devstatus / status - Development Status

./dj.sh devstatus [options]

• User accounts and permissions
• Subscription tiers and usage
• Applications and data sources
• Platform connections
• Global system statistics

• Environment health checks
• Understanding current setup
• Debugging configuration issues

​

devstatus_quick - Quick Status Check

./dj.sh devstatus_quick

• Quick health checks
• Daily status verification
• Rapid troubleshooting

​

Platform & Application Management

​

apps - List Applications

./dj.sh apps

Applications by User:
========================================
devuser: 2 apps
  - Skyflow (skyflow): 2 sources
  - Nexus (nexus): 1 sources
testuser: No applications

• Understanding application structure
• Verifying data source connections
• Application inventory

​

platforms - List Platforms

./dj.sh platforms

Platforms and Datasets:
========================================
Dev Notion Workspace (notion): 2 datasets
  - Features Database: 2 connected apps
  - Analytics Database: 1 connected apps

• Platform inventory
• Dataset usage analysis
• Connection relationship mapping

​

users - List Users with Details

./dj.sh users

Users and Subscriptions:
========================================
devuser | pro | 0/50000 | 2 apps | staff=True | active=True
testuser | free | 0/1000 | 0 apps | staff=False | active=True

• Username
• Subscription tier
• Request usage (used/limit)
• Application count
• Staff status
• Active status

• User management
• Subscription monitoring
• Access level verification

​

Testing & Validation

​

test_integration - Comprehensive Integration Test

./dj.sh test_integration [options]

• Data structure validation
• Registry integration
• Application data access
• Filtering system functionality
• Data source connections
• API endpoint testing
• Backward compatibility
• Error handling

• Pre-deployment validation
• Feature development verification
• System health assessment

​

test_quick - Quick Integration Test

./dj.sh test_quick

• Rapid functionality check
• Daily development validation
• Quick smoke testing

​

validate - Validate Setup

./dj.sh validate

• User accounts and subscriptions
• Platform configurations
• Application and data source connections
• Registry functionality
• Database consistency

• Post-setup verification
• Troubleshooting configuration issues
• Environment health assessment

​

test_auth - Test Authentication

./dj.sh test_auth

• User login capability
• Session management
• API request authentication
• Permission verification

Status: 200
Success: True
Apps: 2

• Authentication troubleshooting
• Permission verification
• API access testing

​

api_test - Test API Endpoints

./dj.sh api_test

• /scrape/apps/ - Applications list
• /scrape/platforms/ - Platforms list
• /scrape/features/ - Features data
• /scrape/tiers/ - Tiers data

/scrape/apps/: OK (200)
/scrape/platforms/: OK (200)
/scrape/features/: OK (200)
/scrape/tiers/: FAIL (404)

• API functionality verification
• Endpoint availability checking
• Integration testing

​

Database Management

​

reset_db - Reset Database

./dj.sh reset_db

• Interactive confirmation required
• Complete data loss warning
• Suggests follow-up commands

1. Prompts for confirmation
2. Deletes database file
3. Runs migrations to recreate schema
4. Suggests running devstart for test data

• Complete environment reset
• Resolving migration conflicts
• Starting with clean database

​

backup - Backup Database

./dj.sh backup

• Location: /tmp/db_backup_YYYYMMDD_HHMMSS.json
• Format: Django fixtures JSON
• Includes all data and relationships

Backing up database to: /tmp/db_backup_20250909_143052.json
Backup complete: /tmp/db_backup_20250909_143052.json

• Pre-deployment backups
• Development snapshots
• Data preservation before major changes

​

restore - Restore Database

./dj.sh restore <backup_file>

• Lists available backups if no file specified
• Interactive confirmation for data overwrite
• Validates backup file exists

• Disaster recovery
• Rolling back changes
• Environment restoration

​

Development Workflows

​

dev_reset - Complete Development Reset

./dj.sh dev_reset

1. Clear existing data (all users, platforms, applications)
2. Set up fresh PRO environment with devuser
3. Validate setup completeness
4. Run integration tests
5. Report completion status

• Major environment corruption
• Starting fresh development cycle
• Resolving complex configuration issues

​

dev_check - Development Environment Check

./dj.sh dev_check

1. Detailed status report
2. Setup validation
3. Authentication test

• Daily environment verification
• Quick troubleshooting
• Development readiness check

​

Tier-Specific Setup Commands

​

setup_free - FREE Tier Setup

./dj.sh setup_free

• User: freeuser
• Tier: FREE (1,000 requests/month)
• Limitations: 1 platform, 1 dataset, no analytics/export

​

setup_basic - BASIC Tier Setup

./dj.sh setup_basic

• User: basicuser
• Tier: BASIC (10,000 requests/month)
• Features: 2 platforms, 3 datasets, analytics enabled

​

setup_pro - PRO Tier Setup

./dj.sh setup_pro

• User: prouser
• Tier: PRO (50,000 requests/month)
• Features: 5 platforms, 10 datasets, all features enabled

​

setup_enterprise - ENTERPRISE Tier Setup

./dj.sh setup_enterprise

• User: enterpriseuser
• Tier: ENTERPRISE (200,000 requests/month)
• Features: Unlimited platforms/datasets, all features

• Feature limitation verification
• Subscription upgrade testing
• User experience validation across tiers

​

API Testing

​

api_test - Comprehensive API Testing

./dj.sh api_test

• Authentication verification
• Endpoint availability
• Response format validation
• Error handling

• Applications management
• Platform information
• Legacy compatibility endpoints
• Data retrieval endpoints

• API functionality verification
• Integration testing
• Deployment validation

​

Common Use Cases

​

First Time Project Setup

# 1. Apply database migrations
./dj.sh mig

# 2. Set up development environment
./dj.sh devstart

# 3. Create admin user (optional)
./dj.sh sudo

# 4. Verify setup
./dj.sh validate

# 5. Test authentication
./dj.sh test_auth

# 6. Start development server
./dj.sh run

​

Daily Development Workflow

# Quick environment check
./dj.sh devstatus_quick

# Test authentication still works
./dj.sh test_auth

# Start development server
./dj.sh run

​

Testing Different Subscription Tiers

# Set up users for each tier
./dj.sh setup_free
./dj.sh setup_basic
./dj.sh setup_pro
./dj.sh setup_enterprise

# View all test users
./dj.sh users

# Test API with different users
./dj.sh api_test

​

Preparing for Deployment

# Create backup
./dj.sh backup

# Run comprehensive tests
./dj.sh test_integration

# Validate environment
./dj.sh validate

# Check system health
./dj.sh check

​

Development Environment Maintenance

# Weekly cleanup and refresh
./dj.sh devclear
./dj.sh devstart

# Or complete reset if needed
./dj.sh dev_reset

​

Troubleshooting

​

Common Issues and Solutions

​

No Applications Showing

# Check if development data exists
./dj.sh devstatus

# If no data, set up development environment
./dj.sh devstart

# If data exists, check authentication
./dj.sh test_auth

​

Authentication Failures

# Verify user exists
./dj.sh users

# Create development user if missing
./dj.sh devstart

# Test authentication specifically
./dj.sh test_auth

​

Database Migration Issues

# Check for issues
./dj.sh check

# Reset database completely
./dj.sh reset_db

# Reapply migrations
./dj.sh mig

# Set up fresh data
./dj.sh devstart

​

Environment Corruption

# Nuclear option: complete reset
./dj.sh dev_reset

# Or step by step
./dj.sh backup  # Save current state first
./dj.sh devclear
./dj.sh devstart
./dj.sh validate

​

Diagnostic Commands

# Comprehensive environment check
./dj.sh dev_check

# Detailed validation
./dj.sh validate

# Integration testing
./dj.sh test_integration --verbose

# Preview cleanup impact
./dj.sh dryclear

​

Getting Help

# Show all available commands
./dj.sh help

# Django system check
./dj.sh check

# Run with verbose output for debugging
./dj.sh test_integration --verbose

​

Best Practices

1. Always backup before major changes: Use ./dj.sh backup
2. Use dry-run commands: Preview changes with ./dj.sh dryclear
3. Regular validation: Run ./dj.sh validate after setup changes
4. Test different tiers: Use tier-specific setup commands for comprehensive testing
5. Clean environment: Use ./dj.sh dev_reset for fresh starts
6. Monitor status: Regular ./dj.sh devstatus checks for environment health