Skip to main content

​
Summary

The IRIS MERIDIAN adapter uses an in-memory store (MappingStore) to cache sprite-to-capability mappings. The implementation demonstrates two scalable patterns: dual indexing (primary by ID, secondary by composite key) for O(1) filtering instead of O(n) iteration, and Base64-encoded pagination tokens for safe cursor-based pagination without off-by-one errors.

​
The Problem

Naive in-memory storage with a single dict keyed by ID requires O(n) iteration for filtering by composite keys (e.g., “all sprites for iris_id=X and council_id=Y”). For IRIS MERIDIAN, council role taxonomies require rapid filtering—thousands of sprites, millions of lookups. Linear iteration is too slow. Offset-based pagination (?offset=50&limit=10) breaks when items are deleted between requests. Cursor-based pagination is safer but requires safe encoding of position.

​
The Solution: Dual Indexing + Base64 Pagination

Maintain two independent indexes:
  • Primary: {id → item} for direct lookups (O(1))
  • Secondary: {(iris_id, council_id) → [items]} for fast filtering (O(1))
Composite key lookups use tuple keys. When adding an item, insert into both indexes. Filtering by composite key now queries the secondary index in constant time. Base64 pagination tokens: Encode the offset position as a Base64 string. Clients send the token with the next request. Decode on the server. This is opaque to clients (prevents guessing) and extensible (future tokens can encode filter metadata or version info). 
# Pseudocode
def add(self, item):
    self._items_by_id[item.id] = item
    key = (item.iris_id, item.council_id)
    self._items_by_composite[key].append(item)

def filter_by_composite(self, iris_id, council_id):
    # O(1) lookup—no iteration!
    return self._items_by_composite.get((iris_id, council_id), [])

def list_paginated(self, offset_token, limit=10):
    offset = decode_offset(offset_token)  # Base64 to int
    items = self._items_by_id.values()
    page = items[offset:offset+limit]
    next_token = encode_offset(offset+limit) if offset+limit < len(items) else None
    return page, next_token

​
Scaling Consideration

The dual-index pattern scales to external databases. Define a StorageBackend interface; implement InMemoryBackend for tests, PostgresBackend for production. RPC handlers don’t care which backend is active. When scale requires persistence, swap backends—no RPC interface changes needed.

​
Why This Matters

  • Performance: O(1) composite filtering supports sub-100ms RPC latency for council role resolution.
  • Pagination safety: Base64 tokens prevent pagination bugs (duplicate rows, skipped rows) that cause data loss.
  • Testability: In-memory backend is fast; no database setup needed for unit tests.
  • Production ready: Pluggable design enables migration from in-memory (dev) to persistent storage (production) without code rewrites.