query test progress
This commit is contained in:
229
GEMINI.md
229
GEMINI.md
@ -1,159 +1,94 @@
|
||||
# JSPG: JSON Schema Postgres
|
||||
|
||||
**JSPG** is a high-performance PostgreSQL extension for in-memory JSON Schema validation, specifically targeting **Draft 2020-12**.
|
||||
**JSPG** is a high-performance PostgreSQL extension written in Rust (using `pgrx`) that transforms Postgres into a pre-compiled Semantic Engine. It serves as the core engine for the "Punc" architecture, where the database is the single source of truth for all data models, API contracts, validations, and reactive queries.
|
||||
|
||||
It is designed to serve as the validation engine for the "Punc" architecture, where the database is the single source of truth for all data models and API contracts.
|
||||
## 1. Overview & Architecture
|
||||
|
||||
## 🎯 Goals
|
||||
JSPG operates by deeply integrating the JSON Schema Draft 2020-12 specification directly into the Postgres session lifecycle. It is built around three core pillars:
|
||||
* **Validator**: In-memory, near-instant JSON structural validation and type polymorphism routing.
|
||||
* **Merger**: Automatically traverse and UPSERT deeply nested JSON graphs into normalized relational tables.
|
||||
* **Queryer**: Compile JSON Schemas into static, cached SQL SPI `SELECT` plans for fetching full entities or isolated "Stems".
|
||||
|
||||
### 🎯 Goals
|
||||
1. **Draft 2020-12 Compliance**: Attempt to adhere to the official JSON Schema Draft 2020-12 specification.
|
||||
2. **Ultra-Fast Validation**: Compile schemas into an optimized in-memory representation for near-instant validation during high-throughput workloads.
|
||||
3. **Connection-Bound Caching**: Leverage the PostgreSQL session lifecycle to maintain a per-connection schema cache, eliminating the need for repetitive parsing.
|
||||
4. **Structural Inheritance**: Support object-oriented schema design via Implicit Keyword Shadowing and virtual `$family` references.
|
||||
5. **Punc Integration**: validation is aware of the "Punc" context (request/response) and can validate `cue` objects efficiently.
|
||||
|
||||
## 🔌 API Reference
|
||||
|
||||
The extension exposes the following functions to PostgreSQL:
|
||||
|
||||
### `cache_json_schemas(enums jsonb, types jsonb, puncs jsonb) -> jsonb`
|
||||
|
||||
Loads and compiles the entire schema registry into the session's memory, atomically replacing the previous validator.
|
||||
|
||||
* **Inputs**:
|
||||
* `enums`: Array of enum definitions.
|
||||
* `types`: Array of type definitions (core entities).
|
||||
* `puncs`: Array of punc (function) definitions with request/response schemas.
|
||||
* **Behavior**:
|
||||
* Parses all inputs into an internal schema graph.
|
||||
* Resolves all internal references (`$ref`).
|
||||
* Generates virtual union schemas for type hierarchies referenced via `$family`.
|
||||
* Compiles schemas into validators.
|
||||
* **Returns**: `{"response": "success"}` or an error object.
|
||||
|
||||
### `mask_json_schema(schema_id text, instance jsonb) -> jsonb`
|
||||
|
||||
Validates a JSON instance and returns a new JSON object with unknown properties removed (pruned) based on the schema.
|
||||
|
||||
* **Inputs**:
|
||||
* `schema_id`: The `$id` of the schema to mask against.
|
||||
* `instance`: The JSON data to mask.
|
||||
* **Returns**:
|
||||
* On success: A `Drop` containing the **masked data**.
|
||||
* On failure: A `Drop` containing validation errors.
|
||||
|
||||
### `validate_json_schema(schema_id text, instance jsonb) -> jsonb`
|
||||
|
||||
Validates a JSON instance against a pre-compiled schema.
|
||||
|
||||
* **Inputs**:
|
||||
* `schema_id`: The `$id` of the schema to validate against (e.g., `person`, `save_person.request`).
|
||||
* `instance`: The JSON data to validate.
|
||||
* **Returns**:
|
||||
* On success: `{"response": "success"}`
|
||||
* On failure: A JSON object containing structured errors (e.g., `{"errors": [...]}`).
|
||||
|
||||
### `json_schema_cached(schema_id text) -> bool`
|
||||
|
||||
Checks if a specific schema ID is currently present in the cache.
|
||||
|
||||
### `clear_json_schemas() -> jsonb`
|
||||
|
||||
Clears the current session's schema cache, freeing memory.
|
||||
|
||||
### `show_json_schemas() -> jsonb`
|
||||
|
||||
Returns a debug dump of the currently cached schemas (for development/debugging).
|
||||
|
||||
## ✨ Custom Features & Deviations
|
||||
|
||||
JSPG implements specific extensions to the Draft 2020-12 standard to support the Punc architecture's object-oriented needs while heavily optimizing for zero-runtime lookups.
|
||||
|
||||
### 1. Polymorphism & Referencing (`$ref`, `$family`, and Native Types)
|
||||
|
||||
JSPG replaces the complex, dynamic reference resolution logic of standard JSON Schema (e.g., `$defs`, relative URIs, `$dynamicRef`, `$dynamicAnchor`, `anyOf`) with a strict, explicitly structured global `$id` system. This powers predictable code generation and blazing-fast runtime validation.
|
||||
|
||||
#### A. Global `$id` Conventions & Schema Buckets
|
||||
Every schema is part of a flat, globally addressable namespace. However, where a schema is defined in the database determines its physical boundaries:
|
||||
* **Types (Entities)**: Schemas defined within a Postgres `type` represent entities. The `$id` must be exactly the type name (`person`) or suffixed (`full.person`). All schemas in this bucket receive strict Native Type Discrimination based on the physical table hierarchy.
|
||||
* **Puncs (APIs)**: Schemas defined within a `punc` are ad-hoc containers. The `$id` must be exactly `[punc_name].request` or `[punc_name].response`. They are never entities themselves.
|
||||
* **Enums (Domains)**: Schemas defined within an `enum` represent enum definitions. The `$id` must be exactly the enum name (`job_status`) or suffixed (`short.job_status`).
|
||||
|
||||
#### B. Native Type Discrimination (The `variations` Property)
|
||||
Because `jspg` knows which schemas are Entities based on their origin bucket (Types), it securely and implicitly manages the `"type"` property by attaching `compiled_variations`.
|
||||
If a schema originates in the `user` bucket, the validator does *not* rigidly require `{"type": "user"}`. Instead, it queries the physical Postgres type inheritance graph (e.g. `[entity, organization, user]`) and allows the JSON to be `{"type": "person"}` or `{"type": "bot"}` automatically, enabling seamless API polymorphism.
|
||||
|
||||
#### C. Structural Inheritance & Viral Infection (`$ref`)
|
||||
`$ref` is used exclusively for structural inheritance.
|
||||
* **Viral Infection**: If an anonymous schema or an ad-hoc schema (like a Punc Request) `$ref`s a strict Entity schema (like `person.light`), it *virally inherits* the `compiled_variations` of that target. This means a Punc request instantly gains the exact polymorphic security boundaries of the Entity it points to.
|
||||
* **`$ref` never creates a Union.** When you use `$ref`, you are asking for a single, concrete struct/shape.
|
||||
|
||||
#### D. Shape Polymorphism & Virtual Unions (`$family`)
|
||||
To support polymorphic API contracts (e.g., heterogeneous arrays of generic widgets) without manually writing massive `oneOf` blocks, JSPG provides the `$family` macro.
|
||||
While `$ref` defines rigid structure, `$family` relies on an abstract **Descendants Graph**.
|
||||
|
||||
During compilation, `jspg` temporarily tracks every `$ref` pointer globally to build a reverse-lookup graph of "Descendants". It also calculates the **Inheritance Depth** of every schema (how far removed it is from the root entity).
|
||||
When `{"$family": "widget"}` is encountered, JSPG:
|
||||
1. Locates the `widget` schema in the Descendants graph.
|
||||
2. Expands the macro by finding *every* schema in the entire database that structurally `$ref`s `widget`, directly or indirectly (e.g., `stock.widget`, an anonymous object, etc.).
|
||||
3. Evaluates the incoming JSON against **every** descendant schema in that family *strictly*.
|
||||
|
||||
If you request `{"$family": "light.widget"}`, it simply evaluates all schemas that `$ref` the generic abstract `light.widget` interface.
|
||||
|
||||
#### E. Strict Matches & The Depth Heuristic
|
||||
JSPG strictly enforces that polymorphic structures (`oneOf`, or a `$family` expansion) match **exactly one** valid schema permutation. It does not support fuzzy matching (`anyOf`).
|
||||
If a JSON payload matches more than one schema in a union (which happens frequently due to implicit inheritance where an object might technically satisfy the requirements of both `entity` and `user`), JSPG automatically applies the **Depth Heuristic Tie-Breaker**:
|
||||
* It looks up the pre-calculated Inheritance Depth for all valid passing candidates.
|
||||
* It selects the candidate that is **deepest** in the inheritance tree (the most explicitly defined descendant).
|
||||
* If multiple passing candidates tie at the exact same depth level, an `AMBIGUOUS` error is thrown, forcing the developer to supply a more precise type discriminator or payload.
|
||||
|
||||
This cleanly separates **Database Physics** (derived from the Postgres `Types` bucket and viral `$ref` inheritance) from **Structural Polymorphism** (derived purely from the abstract `$ref` tree).
|
||||
|
||||
### 2. Strict by Default & Extensibility
|
||||
JSPG enforces a "Secure by Default" philosophy. All schemas are treated as if `unevaluatedProperties: false` (and `unevaluatedItems: false`) is set, unless explicitly overridden.
|
||||
|
||||
* **Strictness**: By default, any property or array item in the instance data that is not explicitly defined in the schema causes a validation error. This prevents clients from sending undeclared fields or extra array elements.
|
||||
* **Extensibility (`extensible: true`)**: To allow a free-for-all of additional, undefined properties or extra array items, you must add `"extensible": true` to the schema. This globally disables the strictness check for that object or array, useful for types designed to be completely open.
|
||||
* **Structured Additional Properties (`additionalProperties: {...}`)**: Instead of a boolean free-for-all, you can define `additionalProperties` as a schema object (e.g., `{"type": "string"}`). This maintains strictness (no arbitrary keys) but allows any extra keys as long as their values match the defined structure.
|
||||
* **Ref Boundaries**: Strictness is reset when crossing `$ref` boundaries. The referenced schema's strictness is determined by its own definition (strict by default unless `extensible: true`), ignoring the caller's state.
|
||||
* **Inheritance**: Strictness is inherited. A schema extending a strict parent will also be strict unless it declares itself `extensible: true`. Conversely, a schema extending a loose parent will also be loose unless it declares itself `extensible: false`.
|
||||
|
||||
### 3. Implicit Keyword Shadowing
|
||||
Standard JSON Schema composition (`allOf`) is additive (Intersection), meaning constraints can only be tightened, not replaced. However, JSPG treats `$ref` differently when it appears alongside other properties to support object-oriented inheritance.
|
||||
|
||||
* **Inheritance (`$ref` + properties)**: When a schema uses `$ref` and defines its own properties, JSPG implements Smart Merge (or Shadowing). If a property is defined in the current schema, its constraints take precedence over the inherited constraints for that specific keyword.
|
||||
* **Example**: If Entity defines `type: { const: "entity" }` and Person (which refs Entity) defines `type: { const: "person" }`, validation passes for "person". The local const shadows the inherited const.
|
||||
* **Granularity**: Shadowing is per-keyword. If Entity defined `type: { const: "entity", minLength: 5 }`, Person would shadow `const` but still inherit `minLength: 5`.
|
||||
* **Composition (`allOf`)**: When using `allOf`, standard intersection rules apply. No shadowing occurs; all constraints from all branches must pass. This is used for mixins or interfaces.
|
||||
|
||||
### 4. Format Leniency for Empty Strings
|
||||
To simplify frontend form logic, the format validators for `uuid`, `date-time`, and `email` explicitly allow empty strings (`""`). This treats an empty string as "present but unset" rather than "invalid format".
|
||||
|
||||
## 🏗️ Architecture
|
||||
|
||||
The extension is written in Rust using `pgrx` and structures its schema parser to mirror the Punc Generator's design:
|
||||
|
||||
* **Single `Schema` Struct**: A unified struct representing the exact layout of a JSON Schema object, including standard keywords and custom vocabularies (`form`, `display`, etc.).
|
||||
* **Compiler Phase**: schema JSONs are parsed into this struct, linked (references resolved), and then compiled into an efficient validation tree.
|
||||
* **Validation Phase**: The compiled validators traverse the JSON instance using `serde_json::Value`.
|
||||
2. **Ultra-Fast Execution**: Compile schemas into optimized in-memory validation trees and cached SQL SPIs to bypass Postgres Query Builder overheads.
|
||||
3. **Connection-Bound Caching**: Leverage the PostgreSQL session lifecycle using an **Atomic Swap** pattern. Schemas are 100% frozen, completely eliminating locks during read access.
|
||||
4. **Structural Inheritance**: Support object-oriented schema design via Implicit Keyword Shadowing and virtual `$family` references natively mapped to Postgres table constraints.
|
||||
5. **Reactive Beats**: Provide natively generated "Stems" (isolated payload fragments) for dynamic websocket reactivity.
|
||||
|
||||
### Concurrency & Threading ("Immutable Graphs")
|
||||
|
||||
To support high-throughput validation while allowing for runtime schema updates (e.g., during development or hot-reloading), JSPG uses an **Atomic Swap** pattern based on 100% immutable schemas.
|
||||
|
||||
To support high-throughput operations while allowing for runtime updates (e.g., during hot-reloading), JSPG uses an **Atomic Swap** pattern:
|
||||
1. **Parser Phase**: Schema JSONs are parsed into ordered `Schema` structs.
|
||||
2. **Compiler Phase**: The database iterates all parsed schemas and pre-computes native optimization maps:
|
||||
* **Descendants Map**: A reverse `$ref` lookup graph for instant `$family` resolution.
|
||||
* **Depths Map**: The `$ref` lineage distance of every schema for heuristic tie-breaking.
|
||||
* **Variations Map**: The Native Type inheritance hierarchy.
|
||||
3. **Immutable Validator**: The `Validator` struct immutably owns the `Database` registry and all its global maps. Once created, a validator instance (and its registry) never changes. Schemas themselves are completely frozen; `$ref` strings are resolved dynamically at runtime using the pre-computed O(1) maps, eliminating the need to physically mutate or link pointers across structures.
|
||||
4. **Global Pointer**: A global `RwLock<Option<Arc<Validator>>>` holds the current active validator.
|
||||
5. **Lock-Free Reads**: Validation requests acquire a read lock just long enough to clone the `Arc` (incrementing a reference count), then release the lock immediately. Validation proceeds on the snapshot, ensuring no blocking during schema updates.
|
||||
6. **Atomic Updates**: When schemas are reloaded (`cache_json_schemas`), a new `Registry` and `Validator` are built entirely on the stack. The global pointer is then atomically swapped to the new instance under a write lock.
|
||||
2. **Compiler Phase**: The database iterates all parsed schemas and pre-computes native optimization maps (Descendants Map, Depths Map, Variations Map).
|
||||
3. **Immutable Validator**: The `Validator` struct immutably owns the `Database` registry and all its global maps. Schemas themselves are completely frozen; `$ref` strings are resolved dynamically at runtime using pre-computed O(1) maps.
|
||||
4. **Lock-Free Reads**: Incoming operations acquire a read lock just long enough to clone the `Arc` inside an `RwLock<Option<Arc<Validator>>>`, ensuring zero blocking during schema updates.
|
||||
|
||||
## 🧪 Testing
|
||||
---
|
||||
|
||||
Testing is driven by standard Rust unit tests that load JSON fixtures.
|
||||
## 2. Validator
|
||||
|
||||
* **Isolation**: Each test file runs with its own isolated `Registry` and `Validator` instance, created on the stack. This eliminates global state interference and allows tests to run in parallel.
|
||||
* **Fixtures**: The tests are located in `tests/fixtures/*.json` and are executed via `cargo test`.
|
||||
The Validator provides strict, schema-driven evaluation for the "Punc" architecture.
|
||||
|
||||
### API Reference
|
||||
* `jspg_setup(database jsonb) -> jsonb`: Loads and compiles the entire registry (types, enums, puncs, relations) atomically.
|
||||
* `mask_json_schema(schema_id text, instance jsonb) -> jsonb`: Validates and prunes unknown properties dynamically, returning masked data.
|
||||
* `jspg_validate(schema_id text, instance jsonb) -> jsonb`: Returns boolean-like success or structured errors.
|
||||
* `jspg_teardown() -> jsonb`: Clears the current session's schema cache.
|
||||
|
||||
### Custom Features & Deviations
|
||||
JSPG implements specific extensions to the Draft 2020-12 standard to support the Punc architecture's object-oriented needs while heavily optimizing for zero-runtime lookups.
|
||||
|
||||
#### A. Polymorphism & Referencing (`$ref`, `$family`, and Native Types)
|
||||
* **Native Type Discrimination (`variations`)**: Schemas defined inside a Postgres `type` are Entities. The validator securely and implicitly manages their `"type"` property. If an entity inherits from `user`, incoming JSON can safely define `{"type": "person"}` without errors, thanks to `compiled_variations` inheritance.
|
||||
* **Structural Inheritance & Viral Infection (`$ref`)**: `$ref` is used exclusively for structural inheritance, *never* for union creation. A Punc request schema that `$ref`s an Entity virally inherits all physical database polymorphism rules for that target.
|
||||
* **Shape Polymorphism (`$family`)**: Auto-expands polymorphic API lists based on an abstract Descendants Graph. If `{"$family": "widget"}` is used, JSPG evaluates the JSON against every schema that `$ref`s widget.
|
||||
* **Strict Matches & Depth Heuristic**: Polymorphic structures MUST match exactly **one** schema permutation. If multiple inherited struct permutations pass, JSPG applies the **Depth Heuristic Tie-Breaker**, selecting the candidate deepest in the inheritance tree.
|
||||
|
||||
#### B. Strict by Default & Extensibility
|
||||
* **Strictness**: By default, any property not explicitly defined in the schema causes a validation error (effectively enforcing `additionalProperties: false` globally).
|
||||
* **Extensibility (`extensible: true`)**: To allow a free-for-all of undefined properties, schemas must explicitly declare `"extensible": true`.
|
||||
* **Structured Additional Properties**: If `additionalProperties: {...}` is defined as a schema, arbitrary keys are allowed so long as their values match the defined type constraint.
|
||||
* **Inheritance Boundaries**: Strictness resets when crossing `$ref` boundaries. A schema extending a strict parent remains strict unless it explicitly overrides with `"extensible": true`.
|
||||
|
||||
#### C. Implicit Keyword Shadowing
|
||||
* **Inheritance (`$ref` + properties)**: Unlike standard JSON Schema, when a schema uses `$ref` alongside local properties, JSPG implements **Smart Merge**. Local constraints natively take precedence over (shadow) inherited constraints for the same keyword.
|
||||
* *Example*: If `entity` has `type: {const: "entity"}`, but `person` defines `type: {const: "person"}`, the local `person` const cleanly overrides the inherited one.
|
||||
* **Composition (`allOf`)**: When evaluating `allOf`, standard intersection rules apply seamlessly. No shadowing occurs, meaning all constraints from all branches must pass.
|
||||
|
||||
#### D. Format Leniency for Empty Strings
|
||||
To simplify frontend form validation, format validators specifically for `uuid`, `date-time`, and `email` explicitly allow empty strings (`""`), treating them as "present but unset".
|
||||
|
||||
---
|
||||
|
||||
|
||||
## 3. Merger
|
||||
|
||||
The Merger provides an automated, high-performance graph synchronization engine via the `jspg_merge(cue JSONB)` API. It orchestrates the complex mapping of nested JSON objects into normalized Postgres relational tables, honoring all inheritance and graph constraints.
|
||||
|
||||
### Core Features
|
||||
|
||||
* **Deep Graph Merging**: The Merger walks arbitrary levels of deeply nested JSON schemas (e.g. tracking an `order`, its `customer`, and an array of its `lines`). It intelligently discovers the correct parent-to-child or child-to-parent Foreign Keys stored in the registry and automatically maps the UUIDs across the relationships during UPSERT.
|
||||
* **Prefix Foreign Key Matching**: Handles scenario where multiple relations point to the same table by using database Foreign Key constraint prefixes (`fk_`). For example, if a schema has `shipping_address` and `billing_address`, the merger resolves against `fk_shipping_address_entity` vs `fk_billing_address_entity` automatically to correctly route object properties.
|
||||
* **Dynamic Deduplication & Lookups**: If a nested object is provided without an `id`, the Merger utilizes Postgres `lk_` index constraints defined in the schema registry (e.g. `lk_person` mapped to `first_name` and `last_name`). It dynamically queries these unique matching constraints to discover the correct UUID to perform an UPDATE, preventing data duplication.
|
||||
* **Hierarchical Table Inheritance**: The Punc system uses distributed table inheritance (e.g. `person` inherits `user` inherits `organization` inherits `entity`). The Merger splits the incoming JSON payload and performs atomic row updates across *all* relevant tables in the lineage map.
|
||||
* **The Archive Paradigm**: Data is never deleted in the Punc system. The Merger securely enforces referential integrity by toggling the `archived` Boolean flag on the base `entity` table rather than issuing SQL `DELETE` commands.
|
||||
* **Change Tracking & Reactivity**: The Merger diffs the incoming JSON against the existing database row (utilizing static, `DashMap`-cached `lk_` SELECT string templates). Every detected change is recorded into the `agreego.change` audit table, tracking the user mapping. It then natively uses `pg_notify` to broadcast a completely flat row-level diff out to the Go WebSocket server for O(1) routing.
|
||||
* **Many-to-Many Graph Edge Management**: Operates seamlessly with the global `agreego.relationship` table, allowing the system to represent and merge arbitrary reified M:M relationships directionally between any two entities.
|
||||
* **Sparse Updates**: Empty JSON strings `""` are directly bound as explicit SQL `NULL` directives to clear data, whilst omitted (missing) properties skip UPDATE execution entirely, ensuring partial UI submissions do not wipe out sibling fields.
|
||||
* **Unified Return Structure**: To eliminate UI hydration race conditions and multi-user duplication, `jspg_merge` explicitly strips the response graph and returns only the root `{ "id": "uuid" }` (or an array of IDs for list insertions). External APIs can then explicitly call read APIs to fetch the resulting graph, while the UI relies 100% implicitly on the flat `pg_notify` pipeline for reactive state synchronization.
|
||||
* **Decoupled SQL Generation**: Because Writes (INSERT/UPDATE) are inherently highly dynamic based on partial payload structures, the Merger generates raw SQL strings dynamically per execution without caching, guaranteeing a minimal memory footprint while scaling optimally.
|
||||
|
||||
---
|
||||
|
||||
## 4. Queryer
|
||||
|
||||
The Queryer transforms Postgres into a pre-compiled Semantic Query Engine via the `jspg_query(schema_id text, cue jsonb)` API, designed to serve the exact shape of Punc responses directly via SQL.
|
||||
|
||||
### Core Features
|
||||
* **Schema-to-SQL Compilation**: Compiles JSON Schema ASTs spanning deep arrays directly into static, pre-planned SQL multi-JOIN queries.
|
||||
* **DashMap SQL Caching**: Executes compiled SQL via Postgres SPI execution, securely caching the static string compilation templates per schema permutation inside the `GLOBAL_JSPG` application memory, drastically reducing repetitive schema crawling.
|
||||
* **Dynamic Filtering**: Binds parameters natively through `cue.filters` objects. Dynamically handles string formatting (e.g. parsing `uuid` or formatting date-times) and safely escapes complex combinations utilizing `ILIKE` operations correctly mapped to the originating structural table.
|
||||
* **The Stem Engine**: Rather than over-fetching heavy Entity payloads and trimming them, Punc Framework Websockets depend on isolated subgraphs defined as **Stems**.
|
||||
* During initialization, the generator auto-discovers graph boundaries (Stems) inside the schema tree.
|
||||
* The Queryer prepares dedicated SQL execution templates tailored precisely for that exact `Stem` path (e.g. executing `get_dashboard` queried specifically for the `/owner` stem).
|
||||
* These Stem outputs instantly hydrate targeted Go Bitsets, providing `O(1)` real-time routing for fractional data payloads without any application-layer overhead.
|
||||
Reference in New Issue
Block a user