jspg/GEMINI.md

JSPG: JSON Schema Postgres

JSPG is a high-performance PostgreSQL extension written in Rust (using pgrx) that transforms Postgres into a pre-compiled Semantic Engine. It serves as the core engine for the "Punc" architecture, where the database is the single source of truth for all data models, API contracts, validations, and reactive queries.

1. Overview & Architecture

JSPG operates by deeply integrating the JSON Schema Draft 2020-12 specification directly into the Postgres session lifecycle. It is built around three core pillars:

• Validator: In-memory, near-instant JSON structural validation and type polymorphism routing.
• Merger: Automatically traverse and UPSERT deeply nested JSON graphs into normalized relational tables.
• Queryer: Compile JSON Schemas into static, cached SQL SPI SELECT plans for fetching full entities or isolated ad-hoc object boundaries.

🎯 Goals

1. Draft 2020-12 Based: Attempt to adhere to the official JSON Schema Draft 2020-12 specification, while heavily augmenting it for strict structural typing.
2. Ultra-Fast Execution: Compile schemas into optimized in-memory validation trees and cached SQL SPIs to bypass Postgres Query Builder overheads.
3. Connection-Bound Caching: Leverage the PostgreSQL session lifecycle using an Atomic Swap pattern. Schemas are 100% frozen, completely eliminating locks during read access.
4. Structural Inheritance: Support object-oriented schema design via Implicit Keyword Shadowing and virtual $family references natively mapped to Postgres table constraints.
5. Reactive Beats: Provide ultra-fast natively generated flat payloads mapping directly to the Dart topological state for dynamic websocket reactivity.

Concurrency & Threading ("Immutable Graphs")

To support high-throughput operations while allowing for runtime updates (e.g., during hot-reloading), JSPG uses an Atomic Swap pattern:

1. Parser Phase: Schema JSONs are parsed into ordered Schema structs.
2. Compiler Phase: The database iterates all parsed schemas and pre-computes native optimization maps (Descendants Map, Depths Map, Variations Map).
3. Immutable AST Caching: The Validator struct immutably owns the Database registry. Schemas themselves are frozen structurally, but utilize OnceLock interior mutability during the Compilation Phase to permanently cache resolved type inheritances, properties, and compiled_edges directly onto their AST nodes. This guarantees strict O(1) relationship and property validation execution at runtime without locking or recursive DB polling.
4. Lock-Free Reads: Incoming operations acquire a read lock just long enough to clone the Arc inside an RwLock<Option<Arc<Validator>>>, ensuring zero blocking during schema updates.

Global API Reference

These functions operate on the global GLOBAL_JSPG engine instance and provide administrative boundaries:

• jspg_setup(database jsonb) -> jsonb: Initializes the engine. Deserializes the full database schema registry (types, enums, puncs, relations) from Postgres and compiles them into memory atomically.
• jspg_teardown() -> jsonb: Clears the current session's engine instance from GLOBAL_JSPG, resetting the cache.
• jspg_schemas() -> jsonb: Exports the fully compiled AST snapshot (including all inherited dependencies) out of GLOBAL_JSPG into standard JSON Schema representations.

---

2. Schema Modeling (Punc Developer Guide)

JSPG augments standard JSON Schema 2020-12 to provide an opinionated, strict, and highly ergonomic Object-Oriented paradigm. Developers defining Punc Data Models should follow these conventions.

Types of Types

• Table-Backed (Entity Types): Primarily defined in root type schemas. These represent physical Postgres tables.
  • They absolutely require an $id.
  • The schema conceptually requires a type discriminator at runtime so the engine knows what physical variation to interact with.
  • Can inherit other entity types to build lineage (e.g. person -> organization -> entity).
• Field-Backed (JSONB Bubbles): These are shapes that live entirely inside a Postgres JSONB column without being tied to a top-level table constraint.
  • Global $id Promotion: Utilizing explicit $id declarations promotes the schema to the Global Registry. This effectively creates strictly-typed code-generator universes (e.g., generating an InvoiceNotificationMetadata Dart class) operating cleanly inside unstructured Postgres JSONB columns.
  • They can re-use the standard type discriminator locally for oneOf polymorphism without conflicting with global Postgres Table constraints.

Discriminators & The Dot Convention (A.B)

In Punc, polymorphic targets like explicit tagged unions or STI (Single Table Inheritance) rely on discriminators. Because Punc favors universal consistency, a schema's data contract must be explicit and mathematically identical regardless of the routing context an endpoint consumes it through.

The 2-Tier Paradigm: The system inherently prevents "God Tables" by restricting routing to exactly two dimensions, guaranteeing absolute O(1) lookups without ambiguity:

1. Vertical Routing (type): Identifies the specific Postgres Table lineage (e.g. person vs organization).
2. Horizontal Routing (kind.type): Natively evaluates Single Table Inheritance. The runtime dynamically concatenates $kind.$type to yield the namespace-protected schema $id (e.g. light.person), maintaining collision-free schema registration.

Therefore, any schema that participates in polymorphic discrimination MUST explicitly define its discriminator properties natively inside its properties block. However, to stay DRY and maintain flexible APIs, you DO NOT need to hardcode const values, nor should you add them to your required array. The Punc engine treats type and kind as magic properties.

Magic Validation Constraints:

• Dynamically Required: The system inherently drives the need for their requirement. The Validator dynamically expects the discriminators and structurally bubbles MISSING_TYPE ultimata ONLY when a polymorphic router ($family / oneOf) dynamically requires them to resolve a path. You never manually put them in the JSON schema required block.
• Implicit Resolution: When wrapped in $family or oneOf, the polymorphic router can mathematically parse the schema $id (e.g. light.person) and natively validate that type equals "person" and kind equals "light", bubbling CONST_VIOLATED if they mismatch, all without you ever hardcoding const limitations.
• Generator Explicitness: Because Postgres is the Single Source of Truth, forcing the explicit definition in properties initially guarantees the downstream Dart/Go code generators observe the fields and can cleanly serialize them dynamically back to the server.

For example, a schema representing $id: "light.person" must natively define its own structural boundaries:

{
  "$id": "light.person",
  "type": "person", 
  "properties": {
    "type": { "type": "string" },
    "kind": { "type": "string" }
  },
  "required": ["type", "kind"]
}

• The Object Contract (Presence): The Object enforces its own structural integrity mechanically. Standard JSON Validation natively ensures type and kind are present, bubbling REQUIRED_FIELD_MISSING organically if omitted.
• The Dynamic Values (db.types): Because the type and kind properties technically exist, the Punc engine dynamically intercepts them during validate_object. It mathematically parses the schema $id (e.g. light.person) and natively validates that type equals "person" (or a valid descendant in db.types) and kind equals "light", bubbling CONST_VIOLATED if they mismatch.
• The Routing Contract: When wrapped in $family or oneOf, the polymorphic router can execute Lightning Fast O(1) fast-paths by reading the payload's type/kind identifiers, and gracefully fallback to standard structural failure if omitted.

Composition & Inheritance (The type keyword)

Punc completely abandons the standard JSON Schema $ref keyword. Instead, it overloads the exact same type keyword used for primitives. A "type" in Punc is mathematically evaluated as either a Native Primitive ("string", "null") or a Custom Object Pointer ("budget", "user").

• Single Inheritance: Setting "type": "user" acts exactly like an extends keyword. The schema borrows all fields and constraints from the user identity. During jspg_setup, the compiler recursively crawls the dependencies to map the physical Postgres table, permanently mapping its type restriction to "object" under the hood so JSON standards remain unbroken.
• Implicit Keyword Shadowing: Unlike standard JSON Schema inheritance, local property definitions natively override and shadow inherited properties.
• Primitive Array Shorthand (Optionality): The type array syntax is heavily optimized for nullable fields. Defining "type": ["budget", "null"] natively builds a nullable strict, generating Budget? budget; in Dart. You can freely mix primitives like ["string", "number", "null"].
  • Strict Array Constraint: To explicitly prevent mathematically ambiguous Multiple Inheritance, a type array is strictly constrained to at most ONE Custom Object Pointer. Defining "type": ["person", "organization"] will intentionally trigger a fatal database compilation error natively instructing developers to build a proper tagged union (oneOf) instead.

Polymorphism ($family and oneOf)

Polymorphism is how an object boundary can dynamically take on entirely different shapes based on the payload provided at runtime.

• $family (Target-Based Polymorphism): An explicit Punc compiler macro instructing the database compiler to dynamically search its internal db.descendants registry and find all physical schemas that mathematically resolve to the target.
  • Across Tables (Vertical): If $family: entity is requested, the payload's type field acts as the discriminator, dynamically routing to standard variations like organization or person spanning multiple Postgres tables.
  • Single Table (Horizontal): If $family: widget is requested, the router explicitly evaluates the Dot Convention dynamically. If the payload possesses "type": "widget" and "kind": "stock", the router mathematically resolves to the string "stock.widget" and routes exclusively to that explicit JSPG schema.
• oneOf (Strict Tagged Unions): A hardcoded array of JSON Schema candidate options. Punc strictly bans mathematical "Union of Sets" evaluation. Every oneOf candidate item MUST either be a pure primitive ({ "type": "null" }) or a user-defined Object Pointer providing a specific discriminator (e.g., { "type": "invoice_metadata" }). This ensures validations remain pure O(1) fast-paths and allows the Dart generator to emit pristine sealed classes.

Conditionals (cases)

Standard JSON Schema forces developers to write deeply nested allOf -> if -> properties blocks just to execute conditional branching. JSPG completely abandons allOf and this practice. For declarative business logic and structural mutations conditionally based upon property bounds, use the top-level cases array.

It evaluates as an Independent Declarative Rules Engine. Every Case block within the array is evaluated independently in parallel. For a given rule, if the when condition evaluates to true, its then schema is executed. If it evaluates to false, its else schema is executed (if present). To maintain strict standard JSON Schema compatibility internally, the when block utilizes pure JSON Schema properties definitions (e.g. enum, const) rather than injecting unstandardized MongoDB operators. Because when, then, and else are themselves standard schemas, they natively support nested cases to handle mutually exclusive else if architectures.

{
  "$id": "save_external_account",
  "cases": [
    {
      "when": {
        "properties": {
          "status": { "const": "unverified" }
        },
        "required": ["status"]
      },
      "then": {
        "required": ["amount_1", "amount_2"]
      }
    },
    {
      "when": { 
        "properties": { "kind": { "const": "credit" } },
        "required": ["kind"]
      },
      "then": {
        "required": ["details"]
      },
      "else": {
        "cases": [
          {
            "when": { "properties": { "kind": { "const": "checking" } }, "required": ["kind"] },
            "then": { "required": ["routing_number"] }
          }
        ]
      }
    }
  ]
}

Strict by Default & Extensibility

• Strictness: By default, any property not explicitly defined in the schema causes a validation error (effectively enforcing additionalProperties: false globally).
• Extensibility (extensible: true): To allow a free-for-all of undefined properties, schemas must explicitly declare "extensible": true.
• Structured Additional Properties: If additionalProperties: {...} is defined as a schema, arbitrary keys are allowed so long as their values match the defined type constraint.
• Inheritance Boundaries: Strictness resets when crossing non-primitive type boundaries. A schema extending a strict parent remains strict unless it explicitly overrides with "extensible": true.

Format Leniency for Empty Strings

To simplify frontend form validation, format validators specifically for uuid, date-time, and email explicitly allow empty strings (""), treating them as "present but unset".

---

3. Database

The Database module manages the core execution graphs and structural compilation of the Postgres environment.

Relational Edge Resolution

When compiling nested object graphs or arrays, the JSPG engine must dynamically infer which Postgres Foreign Key constraint correctly bridges the parent to the nested schema. To guarantee deterministic SQL generation, it utilizes a strict, multi-step algebraic resolution process applied during the OnceLock Compilation phase:

1. Graph Locality Boundary: Before evaluating constraints, the engine ensures the parent and child types do not belong strictly to the same inheritance lineage (e.g., invoice -> activity). Structural inheritance edges are handled natively by the payload merger, so relational edge discovery is intentionally bypassed.
2. Structural Cardinality Filtration: If the JSON Schema requires an Array collection ({"type": "array"}), JSPG mathematically rejects pure scalar Forward constraints (where the parent holds a single UUID pointer), logically narrowing the possibilities to Reverse (1:N) or Junction (M:M) constraints.
3. Exact Prefix Match: If an explicitly prefixed Foreign Key (e.g. fk_invoice_counterparty_entity -> prefix: "counterparty") directly matches the name of the requested schema property (e.g. {"counterparty": {...}}), it is instantly selected.
4. Ambiguity Elimination (M:M Twin Deduction): If multiple explicitly prefixed relations remain (which happens by design in Many-to-Many junction tables like contact or role), the compiler inspects the actual compiled child JSON schema AST. If it observes the child natively consumes one of the prefixes as an explicit outbound property (e.g. contact explicitly defining { "target": ... }), it considers that arrow "used up". It mathematically deduces that its exact twin providing reverse ownership ("source") MUST be the inbound link mapping from the parent.
5. Implicit Base Fallback (1:M): If no explicit prefix matches, and M:M deduction fails, the compiler filters for exactly one remaining relation with a null prefix (e.g. fk_invoice_line_invoice -> prefix: null). A null prefix mathematically denotes the core structural parent-child ownership edge and is used safely as a fallback.
6. Deterministic Abort: If the engine exhausts all deduction pathways and the edge remains ambiguous, it explicitly aborts schema compilation (returns None) rather than silently generating unpredictable SQL.

Ad-Hoc Schema Promotion

To seamlessly support deeply nested, inline Object definitions that don't declare an explicit $id, JSPG aggressively promotes them to standalone topological entities during the database compilation phase.

• Hash Generation: While evaluating the unified graph, if the compiler enters an Object or Array structure completely lacking an $id, it dynamically calculates a localized hash alias representing exactly its structural constraints.
• Promotion: This inline chunk is mathematically elevated to its own $id in the db.schemas cache registry. This guarantees that O(1) WebSockets or isolated queries can natively target any arbitrary sub-object of a massive database topology directly without recursively re-parsing its parent's AST block every read.

---

4. Validator

The Validator provides strict, schema-driven evaluation for the "Punc" architecture.

API Reference

• jspg_validate(schema_id text, instance jsonb) -> jsonb: Validates the instance JSON payload strictly against the constraints of the registered schema_id. Returns boolean-like success or structured error codes.

Custom Features & Deviations

JSPG implements specific extensions to the Draft 2020-12 standard to support the Punc architecture's object-oriented needs while heavily optimizing for zero-runtime lookups.

• Caching Strategy: The Validator caches the pre-compiled Database registry in memory upon initialization (jspg_setup). This registry holds the comprehensive graph of schema boundaries, Types, ENUMs, and Foreign Key relationships, acting as the Single Source of Truth for all validation operations without polling Postgres.
• Discriminator Fast Paths & Extraction: When executing a polymorphic node (oneOf or $family), the engine statically analyzes the incoming JSON payload for the literal type and kind string coordinates. It routes the evaluation specifically to matching candidates in O(1) while returning MISSING_TYPE ultimata directly.
• Missing Type Ultimatum: If an entity logically requires a discriminator and the JSON payload omits it, JSPG short-circuits branch execution entirely, bubbling a single, perfectly-pathed MISSING_TYPE error back to the UI natively to prevent confusing cascading failures.
• Golden Match Context: When exactly one structural candidate perfectly maps a discriminator, the Validator exclusively cascades that specific structural error context directly to the user, stripping away all noise generated by other parallel schemas.

---

5. Merger

The Merger provides an automated, high-performance graph synchronization engine. It orchestrates the complex mapping of nested JSON objects into normalized Postgres relational tables, honoring all inheritance and graph constraints.

API Reference

• jspg_merge(schema_id text, data jsonb) -> jsonb: Traverses the provided JSON payload according to the compiled relational map of schema_id. Dynamically builds and executes relational SQL UPSERT paths natively.

Core Features

• Caching Strategy: The Merger leverages the native compiled_edges permanently cached onto the Schema AST via OnceLock to instantly resolve Foreign Key mapping graphs natively in absolute O(1) time. It additionally utilizes the concurrent GLOBAL_JSPG application memory (DashMap) to cache statically constructed SQL SELECT strings used during deduplication (lk_) and difference tracking calculations.
• Deep Graph Merging: The Merger walks arbitrary levels of deeply nested JSON schemas (e.g. tracking an order, its customer, and an array of its lines). It intelligently discovers the correct parent-to-child or child-to-parent Foreign Keys stored in the registry and automatically maps the UUIDs across the relationships during UPSERT.
• Prefix Foreign Key Matching: Handles scenario where multiple relations point to the same table by using database Foreign Key constraint prefixes (fk_). For example, if a schema has shipping_address and billing_address, the merger resolves against fk_shipping_address_entity vs fk_billing_address_entity automatically to correctly route object properties.
• Dynamic Deduplication & Lookups: If a nested object is provided without an id, the Merger utilizes Postgres lk_ index constraints defined in the schema registry (e.g. lk_person mapped to first_name and last_name). It dynamically queries these unique matching constraints to discover the correct UUID to perform an UPDATE, preventing data duplication.
• Hierarchical Table Inheritance: The Punc system uses distributed table inheritance (e.g. person inherits user inherits organization inherits entity). The Merger splits the incoming JSON payload and performs atomic row updates across all relevant tables in the lineage map.
• The Archive Paradigm: Data is never deleted in the Punc system. The Merger securely enforces referential integrity by toggling the archived Boolean flag on the base entity table rather than issuing SQL DELETE commands.
• Change Tracking & Reactivity: The Merger diffs the incoming JSON against the existing database row (utilizing static, DashMap-cached lk_ SELECT string templates). Every detected change is recorded into the agreego.change audit table, tracking the user mapping. It then natively uses pg_notify to broadcast a completely flat row-level diff out to the Go WebSocket server for O(1) routing.
• Flat Structural Beats (Unidirectional Flow): The Merger purposefully DOES NOT trace or hydrate outbound Foreign Keys or nested parent structures during writes. It emits completely flat, mathematically perfect structural deltas via pg_notify representing only the exact Postgres rows that changed. This guarantees the write-path remains O(1) lightning fast. It is the strict responsibility of the upstream Punc Framework (the Go Speaker) to intercept these flat beats, evaluate them against active Websocket Schema Topologies, and dynamically issue targeted jspg_query reads to hydrate the exact contextual subgraphs required by listening clients.
• Pre-Order Notification Traversal: To support proper topological hydration on the upstream Go Framework, the Merger decouples the pg_notify execution from the physical database write execution. The engine collects structural changes and explicitly fires pg_notify SQL statements in strict Pre-Order (Parent -> Relations -> Children). This guarantees that WebSocket clients receive the parent entity Beat prior to any nested child entities, ensuring stable unidirectional data flows without hydration race conditions.
• Many-to-Many Graph Edge Management: Operates seamlessly with the global agreego.relationship table, allowing the system to represent and merge arbitrary reified M:M relationships directionally between any two entities.
• Sparse Updates: Empty JSON strings "" are directly bound as explicit SQL NULL directives to clear data, whilst omitted (missing) properties skip UPDATE execution entirely, ensuring partial UI submissions do not wipe out sibling fields.
• Unified Return Structure: To eliminate UI hydration race conditions and multi-user duplication, jspg_merge explicitly strips the response graph and returns only the root { "id": "uuid" } (or an array of IDs for list insertions). External APIs can then explicitly call read APIs to fetch the resulting graph, while the UI relies 100% implicitly on the flat pg_notify pipeline for reactive state synchronization.
• Decoupled SQL Generation: Because Writes (INSERT/UPDATE) are inherently highly dynamic based on partial payload structures, the Merger generates raw SQL strings dynamically per execution without caching, guaranteeing a minimal memory footprint while scaling optimally.

---

6. Queryer

The Queryer transforms Postgres into a pre-compiled Semantic Query Engine, designed to serve the exact shape of Punc responses directly via SQL.

API Reference

• jspg_query(schema_id text, filters jsonb) -> jsonb: Compiles the JSON Schema AST of schema_id directly into pre-planned, nested multi-JOIN SQL execution trees. Processes filters structurally.

Core Features

• Caching Strategy (DashMap SQL Caching): The Queryer securely caches its compiled, static SQL string templates per schema permutation inside the GLOBAL_JSPG concurrent DashMap. This eliminates recursive AST schema crawling on consecutive requests. Furthermore, it evaluates the strings via Postgres SPI (Server Programming Interface) Prepared Statements, leveraging native database caching of execution plans for extreme performance.
• Schema-to-SQL Compilation: Compiles JSON Schema ASTs spanning deep arrays directly into static, pre-planned SQL multi-JOIN queries. This explicitly features the Smart Merge evaluation engine which natively translates properties through type inheritances, mapping JSON fields specifically to their physical database table aliases during translation.
• Root Null-Stripping Optimization: Unlike traditional nested document builders, the Queryer intelligently defers Postgres' natively recursive jsonb_strip_nulls execution to the absolute apex of the compiled query pipeline. The compiler organically layers millions of rapid jsonb_build_object() sub-query allocations instantly, wrapping them in a singular overarching pass. This strips all empty optionals uniformly before exiting the database, maximizing CPU throughput.
• Dynamic Filtering: Binds parameters natively through cue.filters objects. The queryer enforces a strict, structured, MongoDB-style operator syntax to map incoming JSON request constraints directly to their originating structural table columns. Filters support both flat path notation (e.g., "contacts/is_primary": {...}) and deeply nested recursive JSON structures (e.g., {"contacts": {"is_primary": {...}}}). The queryer recursively traverses and flattens these structures at AST compilation time.
  • Equality / Inequality: {"$eq": value}, {"$ne": value} automatically map to = and !=.
  • Comparison: {"$gt": ...}, {"$gte": ...}, {"$lt": ...}, {"$lte": ...} directly compile to Postgres comparison operators (> , >=, <, <=).
  • Array Inclusion: {"$in": [values]}, {"$nin": [values]} use native jsonb_array_elements_text() bindings to enforce IN and NOT IN logic without runtime SQL injection risks.
  • Text Matching (ILIKE): Evaluates $eq or $ne against string fields containing the % character natively into Postgres ILIKE and NOT ILIKE partial substring matches.
  • Type Casting: Safely resolves dynamic combinations by casting values instantly into the physical database types mapped in the schema (e.g. parsing uuid bindings to ::uuid, formatting DateTimes to ::timestamptz, and numbers to ::numeric).
• Polymorphic SQL Generation ($family): Compiles $family properties by analyzing the Physical Database Variations, not the schema descendants.
  • The Dot Convention: When a schema requests $family: "target.schema", the compiler extracts the base type (e.g. schema) and looks up its Physical Table definition.
  • Multi-Table Branching: If the Physical Table is a parent to other tables (e.g. organization has variations ["organization", "bot", "person"]), the compiler generates a dynamic CASE WHEN type = '...' THEN ... query, expanding into JOINs for each variation.
  • Single-Table Bypass: If the Physical Table is a leaf node with only one variation (e.g. person has variations ["person"]), the compiler cleanly bypasses CASE generation and compiles a simple SELECT across the base table, as all schema extensions (e.g. light.person, full.person) are guaranteed to reside in the exact same physical row.

---

7. Testing & Execution Architecture

JSPG implements a strict separation of concerns to bypass the need to boot a full PostgreSQL cluster for unit and integration testing. Because pgrx::spi::Spi directly links to PostgreSQL C-headers, building the library with cargo test on macOS natively normally results in fatal dyld crashes.

To solve this, JSPG introduces the DatabaseExecutor trait inside src/database/executors/:

• SpiExecutor (pgrx.rs): The production evaluator that is conditionally compiled (#[cfg(not(test))]). It unwraps standard pgrx::spi connections to the database.
• MockExecutor (mock.rs): The testing evaluator that is conditionally compiled (#[cfg(test)]). It absorbs SQL calls and captures parameter bindings in memory without executing them.

Universal Test Harness (src/tests/)

JSPG abandons the standard cargo pgrx test model in favor of native OS testing for a >1000x speed increase (~0.05s execution).

1. JSON Fixtures: All core interactions are defined abstractly as JSON arrays in fixtures/. Each file contains suites of TestCase objects with an action flag (compile, validate, merge, query).
2. build.rs Generator: The build script traverses the JSON fixtures, extracts their structural identities, and generates standard #[test] blocks into src/tests/fixtures.rs.
3. Modular Test Dispatcher: The src/tests/types/ module deserializes the abstract JSON test payloads into Suite, Case, and Expect data structures.
   • The compile action natively asserts the exact output shape of jspg_stems, allowing structural and relationship mapping logic to be tested purely through JSON without writing brute-force manual tests in Rust.
4. Unit Context Execution: When cargo test executes, the runner iterates the JSON payloads. Because the tests run natively inside the module via #cfg(test), the Rust compiler globally erases pgrx C-linkage, instantiates the MockExecutor, and allows for pure structural evaluation of complex database logic completely in memory in parallel.