Compare commits

...

22 Commits
1.0.75 ... main

Author SHA1 Message Date
14707330a7 subschema id queryer test added 2026-03-22 05:54:31 -04:00
77bc92533c version: 1.0.84 2026-03-22 03:35:54 -04:00
4060119b01 schema ids can now contain a subschema 2026-03-22 03:35:47 -04:00
95546fe10c version: 1.0.83 2026-03-21 20:33:48 -04:00
882bdc6271 merger now requires a schema id, queryer and merger now use pre-compiled edges for O(1) relations 2026-03-21 20:33:28 -04:00
9bdb767685 version: 1.0.82 2026-03-20 18:05:43 -04:00
bdd89fe695 cleanup 2026-03-20 18:05:37 -04:00
8135d80045 cleanup 2026-03-20 18:05:18 -04:00
9255439d53 added support for root schema compiled properties for the mixer 2026-03-20 18:04:49 -04:00
9038607729 version: 1.0.81 2026-03-20 15:53:59 -04:00
9f6c27c3b8 support ad-hoc refing without entity types 2026-03-20 15:53:48 -04:00
75aac41362 version: 1.0.80 2026-03-20 06:48:19 -04:00
dbcef42401 merger fixes 2026-03-20 06:48:08 -04:00
b6c5561d2f version: 1.0.79 2026-03-20 05:58:53 -04:00
e01b778d68 jsob and test array handling improved in merger 2026-03-20 05:58:43 -04:00
6eb134c0d6 test checkpoint 2026-03-20 05:17:28 -04:00
7ccc4b7cce version: 1.0.78 2026-03-20 04:41:46 -04:00
77bfa4cd18 historical and notify respected 2026-03-20 04:41:35 -04:00
b47a5abd26 version: 1.0.77 2026-03-20 01:59:56 -04:00
fcd8310ed8 added new and old to changes and pg notify 2026-03-20 01:59:48 -04:00
31519e8447 version: 1.0.76 2026-03-18 22:04:51 -04:00
847e921b1c stems removed from queryer 2026-03-18 22:04:29 -04:00
25 changed files with 2357 additions and 1394 deletions

11
.test/tests.md Normal file
View File

@ -0,0 +1,11 @@
# 🗒️ Test Report (punc/framework)
_Generated at Wed Mar 18 05:21:40 EDT 2026_
## Summary
| Lang | Status | Tests | Passed | Failed | Duration |
| :--- | :---: | :---: | :---: | :---: | ---: |
## Results

View File

@ -20,9 +20,16 @@ JSPG operates by deeply integrating the JSON Schema Draft 2020-12 specification
To support high-throughput operations while allowing for runtime updates (e.g., during hot-reloading), JSPG uses an **Atomic Swap** pattern: To support high-throughput operations while allowing for runtime updates (e.g., during hot-reloading), JSPG uses an **Atomic Swap** pattern:
1. **Parser Phase**: Schema JSONs are parsed into ordered `Schema` structs. 1. **Parser Phase**: Schema JSONs are parsed into ordered `Schema` structs.
2. **Compiler Phase**: The database iterates all parsed schemas and pre-computes native optimization maps (Descendants Map, Depths Map, Variations Map). 2. **Compiler Phase**: The database iterates all parsed schemas and pre-computes native optimization maps (Descendants Map, Depths Map, Variations Map).
3. **Immutable Validator**: The `Validator` struct immutably owns the `Database` registry and all its global maps. Schemas themselves are completely frozen; `$ref` strings are resolved dynamically at runtime using pre-computed O(1) maps. 3. **Immutable AST Caching**: The `Validator` struct immutably owns the `Database` registry. Schemas themselves are frozen structurally, but utilize `OnceLock` interior mutability during the Compilation Phase to permanently cache resolved `$ref` inheritances, properties, and `compiled_edges` directly onto their AST nodes. This guarantees strict `O(1)` relationship and property validation execution at runtime without locking or recursive DB polling.
4. **Lock-Free Reads**: Incoming operations acquire a read lock just long enough to clone the `Arc` inside an `RwLock<Option<Arc<Validator>>>`, ensuring zero blocking during schema updates. 4. **Lock-Free Reads**: Incoming operations acquire a read lock just long enough to clone the `Arc` inside an `RwLock<Option<Arc<Validator>>>`, ensuring zero blocking during schema updates.
### Global API Reference
These functions operate on the global `GLOBAL_JSPG` engine instance and provide administrative boundaries:
* `jspg_setup(database jsonb) -> jsonb`: Initializes the engine. Deserializes the full database schema registry (types, enums, puncs, relations) from Postgres and compiles them into memory atomically.
* `jspg_teardown() -> jsonb`: Clears the current session's engine instance from `GLOBAL_JSPG`, resetting the cache.
* `jspg_schemas() -> jsonb`: Exports the fully compiled AST snapshot (including all inherited dependencies) out of `GLOBAL_JSPG` into standard JSON Schema representations.
--- ---
## 2. Validator ## 2. Validator
@ -30,10 +37,7 @@ To support high-throughput operations while allowing for runtime updates (e.g.,
The Validator provides strict, schema-driven evaluation for the "Punc" architecture. The Validator provides strict, schema-driven evaluation for the "Punc" architecture.
### API Reference ### API Reference
* `jspg_setup(database jsonb) -> jsonb`: Loads and compiles the entire registry (types, enums, puncs, relations) atomically. * `jspg_validate(schema_id text, instance jsonb) -> jsonb`: Validates the `instance` JSON payload strictly against the constraints of the registered `schema_id`. Returns boolean-like success or structured error codes.
* `mask_json_schema(schema_id text, instance jsonb) -> jsonb`: Validates and prunes unknown properties dynamically, returning masked data.
* `jspg_validate(schema_id text, instance jsonb) -> jsonb`: Returns boolean-like success or structured errors.
* `jspg_teardown() -> jsonb`: Clears the current session's schema cache.
### Custom Features & Deviations ### Custom Features & Deviations
JSPG implements specific extensions to the Draft 2020-12 standard to support the Punc architecture's object-oriented needs while heavily optimizing for zero-runtime lookups. JSPG implements specific extensions to the Draft 2020-12 standard to support the Punc architecture's object-oriented needs while heavily optimizing for zero-runtime lookups.
@ -69,11 +73,14 @@ To simplify frontend form validation, format validators specifically for `uuid`,
## 3. Merger ## 3. Merger
The Merger provides an automated, high-performance graph synchronization engine via the `jspg_merge(cue JSONB)` API. It orchestrates the complex mapping of nested JSON objects into normalized Postgres relational tables, honoring all inheritance and graph constraints. The Merger provides an automated, high-performance graph synchronization engine. It orchestrates the complex mapping of nested JSON objects into normalized Postgres relational tables, honoring all inheritance and graph constraints.
### API Reference
* `jspg_merge(schema_id text, data jsonb) -> jsonb`: Traverses the provided JSON payload according to the compiled relational map of `schema_id`. Dynamically builds and executes relational SQL UPSERT paths natively.
### Core Features ### Core Features
* **Caching Strategy**: The Merger leverages the `Validator`'s in-memory `Database` registry to instantly resolve Foreign Key mapping graphs. It additionally utilizes the concurrent `GLOBAL_JSPG` application memory (`DashMap`) to cache statically constructed SQL `SELECT` strings used during deduplication (`lk_`) and difference tracking calculations. * **Caching Strategy**: The Merger leverages the native `compiled_edges` permanently cached onto the Schema AST via `OnceLock` to instantly resolve Foreign Key mapping graphs natively in absolute `O(1)` time. It additionally utilizes the concurrent `GLOBAL_JSPG` application memory (`DashMap`) to cache statically constructed SQL `SELECT` strings used during deduplication (`lk_`) and difference tracking calculations.
* **Deep Graph Merging**: The Merger walks arbitrary levels of deeply nested JSON schemas (e.g. tracking an `order`, its `customer`, and an array of its `lines`). It intelligently discovers the correct parent-to-child or child-to-parent Foreign Keys stored in the registry and automatically maps the UUIDs across the relationships during UPSERT. * **Deep Graph Merging**: The Merger walks arbitrary levels of deeply nested JSON schemas (e.g. tracking an `order`, its `customer`, and an array of its `lines`). It intelligently discovers the correct parent-to-child or child-to-parent Foreign Keys stored in the registry and automatically maps the UUIDs across the relationships during UPSERT.
* **Prefix Foreign Key Matching**: Handles scenario where multiple relations point to the same table by using database Foreign Key constraint prefixes (`fk_`). For example, if a schema has `shipping_address` and `billing_address`, the merger resolves against `fk_shipping_address_entity` vs `fk_billing_address_entity` automatically to correctly route object properties. * **Prefix Foreign Key Matching**: Handles scenario where multiple relations point to the same table by using database Foreign Key constraint prefixes (`fk_`). For example, if a schema has `shipping_address` and `billing_address`, the merger resolves against `fk_shipping_address_entity` vs `fk_billing_address_entity` automatically to correctly route object properties.
* **Dynamic Deduplication & Lookups**: If a nested object is provided without an `id`, the Merger utilizes Postgres `lk_` index constraints defined in the schema registry (e.g. `lk_person` mapped to `first_name` and `last_name`). It dynamically queries these unique matching constraints to discover the correct UUID to perform an UPDATE, preventing data duplication. * **Dynamic Deduplication & Lookups**: If a nested object is provided without an `id`, the Merger utilizes Postgres `lk_` index constraints defined in the schema registry (e.g. `lk_person` mapped to `first_name` and `last_name`). It dynamically queries these unique matching constraints to discover the correct UUID to perform an UPDATE, preventing data duplication.
@ -91,7 +98,10 @@ The Merger provides an automated, high-performance graph synchronization engine
## 4. Queryer ## 4. Queryer
The Queryer transforms Postgres into a pre-compiled Semantic Query Engine via the `jspg_query(schema_id text, cue jsonb)` API, designed to serve the exact shape of Punc responses directly via SQL. The Queryer transforms Postgres into a pre-compiled Semantic Query Engine, designed to serve the exact shape of Punc responses directly via SQL.
### API Reference
* `jspg_query(schema_id text, filters jsonb) -> jsonb`: Compiles the JSON Schema AST of `schema_id` directly into pre-planned, nested multi-JOIN SQL execution trees. Processes `filters` structurally.
### Core Features ### Core Features

View File

819
d1.json Normal file
View File

@ -0,0 +1,819 @@
{
"database": {
"puncs": [],
"enums": [
{
"id": "11111111-1111-1111-1111-111111111111",
"type": "relation_type",
"enum": "relation_type",
"values": [
"foreign_key",
"polymorphic",
"graph"
]
}
],
"relations": [
{
"id": "22222222-2222-2222-2222-222222222222",
"type": "relation",
"constraint": "fk_order_customer",
"source_type": "order",
"source_columns": [
"customer_id"
],
"destination_type": "person",
"destination_columns": [
"id"
],
"prefix": "customer"
},
{
"id": "33333333-3333-3333-3333-333333333333",
"type": "relation",
"constraint": "fk_order_line_order",
"source_type": "order_line",
"source_columns": [
"order_id"
],
"destination_type": "order",
"destination_columns": [
"id"
],
"prefix": "lines"
},
{
"id": "44444444-4444-4444-4444-444444444444",
"type": "relation",
"constraint": "fk_relationship_source_entity",
"source_type": "relationship",
"source_columns": [
"source_id",
"source_type"
],
"destination_type": "entity",
"destination_columns": [
"id",
"type"
],
"prefix": "source"
},
{
"id": "55555555-5555-5555-5555-555555555555",
"type": "relation",
"constraint": "fk_relationship_target_entity",
"source_type": "relationship",
"source_columns": [
"target_id",
"target_type"
],
"destination_type": "entity",
"destination_columns": [
"id",
"type"
],
"prefix": "target"
}
],
"types": [
{
"name": "entity",
"schemas": [
{
"$id": "entity",
"type": "object",
"properties": {
"id": {
"type": "string"
},
"type": {
"type": "string"
},
"archived": {
"type": "boolean"
},
"created_by": {
"type": "string"
},
"modified_by": {
"type": "string"
},
"created_at": {
"type": "string",
"format": "date-time"
},
"modified_at": {
"type": "string",
"format": "date-time"
}
},
"required": [
"id",
"type",
"created_by",
"created_at",
"modified_by",
"modified_at"
]
}
],
"hierarchy": [
"entity"
],
"fields": [
"id",
"type",
"created_at",
"created_by",
"modified_at",
"modified_by",
"archived"
],
"grouped_fields": {
"entity": [
"id",
"type",
"created_at",
"created_by",
"modified_at",
"modified_by",
"archived"
]
},
"lookup_fields": [],
"historical": true,
"notify": true,
"relationship": false
},
{
"name": "organization",
"schemas": [
{
"$id": "organization",
"$ref": "entity",
"properties": {
"name": {
"type": "string"
}
}
}
],
"hierarchy": [
"organization",
"entity"
],
"fields": [
"id",
"type",
"name",
"created_at",
"created_by",
"modified_at",
"modified_by",
"archived"
],
"grouped_fields": {
"organization": [
"id",
"type",
"name"
],
"entity": [
"id",
"type",
"created_at",
"created_by",
"modified_at",
"modified_by",
"archived"
]
},
"lookup_fields": [],
"historical": true,
"notify": true,
"relationship": false
},
{
"name": "user",
"schemas": [
{
"$id": "user",
"$ref": "organization",
"properties": {}
}
],
"hierarchy": [
"user",
"organization",
"entity"
],
"fields": [
"id",
"type",
"name",
"created_at",
"created_by",
"modified_at",
"modified_by",
"archived"
],
"grouped_fields": {
"user": [
"id",
"type"
],
"organization": [
"id",
"type",
"name"
],
"entity": [
"id",
"type",
"created_at",
"created_by",
"modified_at",
"modified_by",
"archived"
]
},
"lookup_fields": [],
"historical": true,
"notify": true,
"relationship": false
},
{
"name": "person",
"schemas": [
{
"$id": "person",
"$ref": "user",
"properties": {
"first_name": {
"type": "string"
},
"last_name": {
"type": "string"
},
"date_of_birth": {
"type": "string"
},
"pronouns": {
"type": "string"
},
"contact_id": {
"type": "string"
},
"contacts": {
"type": "array",
"items": {
"$ref": "contact",
"properties": {
"target": {
"oneOf": [
{
"$ref": "phone_number"
},
{
"$ref": "email_address"
}
]
}
}
}
}
}
}
],
"hierarchy": [
"person",
"user",
"organization",
"entity"
],
"fields": [
"id",
"type",
"first_name",
"last_name",
"date_of_birth",
"pronouns",
"contact_id",
"name",
"created_at",
"created_by",
"modified_at",
"modified_by",
"archived"
],
"grouped_fields": {
"person": [
"id",
"type",
"first_name",
"last_name",
"date_of_birth",
"pronouns",
"contact_id"
],
"user": [
"id",
"type"
],
"organization": [
"id",
"type",
"name"
],
"entity": [
"id",
"type",
"created_at",
"created_by",
"modified_at",
"modified_by",
"archived"
]
},
"lookup_fields": [
"first_name",
"last_name",
"date_of_birth",
"pronouns"
],
"historical": true,
"notify": true,
"relationship": false
},
{
"name": "order",
"schemas": [
{
"$id": "order",
"$ref": "entity",
"properties": {
"total": {
"type": "number"
},
"customer_id": {
"type": "string"
},
"customer": {
"$ref": "person"
},
"lines": {
"type": "array",
"items": {
"$ref": "order_line"
}
}
}
}
],
"hierarchy": [
"order",
"entity"
],
"fields": [
"id",
"type",
"total",
"customer_id",
"created_at",
"created_by",
"modified_at",
"modified_by",
"archived"
],
"grouped_fields": {
"order": [
"id",
"type",
"total",
"customer_id"
],
"entity": [
"id",
"type",
"created_at",
"created_by",
"modified_at",
"modified_by",
"archived"
]
},
"lookup_fields": [
"id"
],
"historical": true,
"notify": true,
"relationship": false
},
{
"name": "order_line",
"schemas": [
{
"$id": "order_line",
"$ref": "entity",
"properties": {
"order_id": {
"type": "string"
},
"product": {
"type": "string"
},
"price": {
"type": "number"
}
}
}
],
"hierarchy": [
"order_line",
"entity"
],
"fields": [
"id",
"type",
"order_id",
"product",
"price",
"created_at",
"created_by",
"modified_at",
"modified_by",
"archived"
],
"grouped_fields": {
"order_line": [
"id",
"type",
"order_id",
"product",
"price"
],
"entity": [
"id",
"type",
"created_at",
"created_by",
"modified_at",
"modified_by",
"archived"
]
},
"lookup_fields": [],
"historical": true,
"notify": true,
"relationship": false
},
{
"name": "relationship",
"relationship": true,
"hierarchy": [
"relationship",
"entity"
],
"fields": [
"source_id",
"source_type",
"target_id",
"target_type",
"id",
"type",
"name",
"archived",
"created_at",
"created_by",
"modified_at",
"modified_by"
],
"grouped_fields": {
"entity": [
"id",
"type",
"name",
"archived",
"created_at",
"created_by",
"modified_at",
"modified_by"
],
"relationship": [
"source_id",
"source_type",
"target_id",
"target_type"
]
},
"field_types": {
"id": "uuid",
"type": "text",
"archived": "boolean",
"source_id": "uuid",
"source_type": "text",
"target_id": "uuid",
"target_type": "text",
"name": "text",
"created_at": "timestamptz",
"created_by": "uuid",
"modified_at": "timestamptz",
"modified_by": "uuid"
},
"schemas": [
{
"$id": "relationship",
"$ref": "entity",
"properties": {}
}
],
"lookup_fields": [],
"historical": true,
"notify": true
},
{
"name": "contact",
"relationship": true,
"hierarchy": [
"contact",
"relationship",
"entity"
],
"fields": [
"is_primary",
"source_id",
"source_type",
"target_id",
"target_type",
"id",
"type",
"name",
"archived",
"created_at",
"created_by",
"modified_at",
"modified_by"
],
"grouped_fields": {
"entity": [
"id",
"type",
"name",
"archived",
"created_at",
"created_by",
"modified_at",
"modified_by"
],
"relationship": [
"source_id",
"source_type",
"target_id",
"target_type"
],
"contact": [
"is_primary"
]
},
"field_types": {
"id": "uuid",
"type": "text",
"archived": "boolean",
"source_id": "uuid",
"source_type": "text",
"target_id": "uuid",
"target_type": "text",
"is_primary": "boolean",
"name": "text",
"created_at": "timestamptz",
"created_by": "uuid",
"modified_at": "timestamptz",
"modified_by": "uuid"
},
"schemas": [
{
"$id": "contact",
"$ref": "relationship",
"properties": {
"is_primary": {
"type": "boolean"
}
}
}
],
"lookup_fields": [],
"historical": true,
"notify": true
},
{
"name": "phone_number",
"hierarchy": [
"phone_number",
"entity"
],
"fields": [
"number",
"id",
"type",
"name",
"archived",
"created_at",
"created_by",
"modified_at",
"modified_by"
],
"grouped_fields": {
"entity": [
"id",
"type",
"name",
"archived",
"created_at",
"created_by",
"modified_at",
"modified_by"
],
"phone_number": [
"number"
]
},
"field_types": {
"id": "uuid",
"type": "text",
"archived": "boolean",
"number": "text",
"name": "text",
"created_at": "timestamptz",
"created_by": "uuid",
"modified_at": "timestamptz",
"modified_by": "uuid"
},
"schemas": [
{
"$id": "phone_number",
"$ref": "entity",
"properties": {
"number": {
"type": "string"
}
}
}
],
"lookup_fields": [],
"historical": true,
"notify": true,
"relationship": false
},
{
"name": "email_address",
"hierarchy": [
"email_address",
"entity"
],
"fields": [
"address",
"id",
"type",
"name",
"archived",
"created_at",
"created_by",
"modified_at",
"modified_by"
],
"grouped_fields": {
"entity": [
"id",
"type",
"name",
"archived",
"created_at",
"created_by",
"modified_at",
"modified_by"
],
"email_address": [
"address"
]
},
"field_types": {
"id": "uuid",
"type": "text",
"archived": "boolean",
"address": "text",
"name": "text",
"created_at": "timestamptz",
"created_by": "uuid",
"modified_at": "timestamptz",
"modified_by": "uuid"
},
"schemas": [
{
"$id": "email_address",
"$ref": "entity",
"properties": {
"address": {
"type": "string"
}
}
}
],
"lookup_fields": [],
"historical": true,
"notify": true,
"relationship": false
},
{
"name": "attachment",
"schemas": [
{
"$id": "type_metadata",
"type": "object",
"properties": {
"type": {
"type": "string"
}
}
},
{
"$id": "other_metadata",
"type": "object",
"properties": {
"other": {
"type": "string"
}
}
},
{
"$id": "attachment",
"$ref": "entity",
"properties": {
"flags": {
"type": "array",
"items": {
"type": "string"
}
},
"type_metadata": {
"$ref": "type_metadata"
},
"other_metadata": {
"$ref": "other_metadata"
}
}
}
],
"hierarchy": [
"attachment",
"entity"
],
"fields": [
"id",
"type",
"flags",
"type_metadata",
"other_metadata",
"created_at",
"created_by",
"modified_at",
"modified_by",
"archived"
],
"grouped_fields": {
"attachment": [
"id",
"type",
"flags",
"type_metadata",
"other_metadata"
],
"entity": [
"id",
"type",
"created_at",
"created_by",
"modified_at",
"modified_by",
"archived"
]
},
"field_types": {
"id": "uuid",
"type": "text",
"flags": "_text",
"type_metadata": "jsonb",
"other_metadata": "jsonb",
"created_at": "timestamptz",
"created_by": "uuid",
"modified_at": "timestamptz",
"modified_by": "uuid",
"archived": "boolean"
},
"lookup_fields": [],
"historical": true,
"notify": true,
"relationship": false
}
]
}
}

View File

@ -143,7 +143,8 @@
] ]
}, },
"lookup_fields": [], "lookup_fields": [],
"historical": false, "historical": true,
"notify": true,
"relationship": false "relationship": false
}, },
{ {
@ -191,6 +192,7 @@
}, },
"lookup_fields": [], "lookup_fields": [],
"historical": true, "historical": true,
"notify": true,
"relationship": false "relationship": false
}, },
{ {
@ -239,6 +241,7 @@
}, },
"lookup_fields": [], "lookup_fields": [],
"historical": true, "historical": true,
"notify": true,
"relationship": false "relationship": false
}, },
{ {
@ -341,6 +344,7 @@
"pronouns" "pronouns"
], ],
"historical": true, "historical": true,
"notify": true,
"relationship": false "relationship": false
}, },
{ {
@ -355,6 +359,15 @@
}, },
"customer_id": { "customer_id": {
"type": "string" "type": "string"
},
"customer": {
"$ref": "person"
},
"lines": {
"type": "array",
"items": {
"$ref": "order_line"
}
} }
} }
} }
@ -395,6 +408,7 @@
"id" "id"
], ],
"historical": true, "historical": true,
"notify": true,
"relationship": false "relationship": false
}, },
{ {
@ -452,6 +466,7 @@
}, },
"lookup_fields": [], "lookup_fields": [],
"historical": true, "historical": true,
"notify": true,
"relationship": false "relationship": false
}, },
{ {
@ -515,7 +530,8 @@
} }
], ],
"lookup_fields": [], "lookup_fields": [],
"historical": true "historical": true,
"notify": true
}, },
{ {
"name": "contact", "name": "contact",
@ -588,7 +604,8 @@
} }
], ],
"lookup_fields": [], "lookup_fields": [],
"historical": true "historical": true,
"notify": true
}, },
{ {
"name": "phone_number", "name": "phone_number",
@ -646,6 +663,7 @@
], ],
"lookup_fields": [], "lookup_fields": [],
"historical": true, "historical": true,
"notify": true,
"relationship": false "relationship": false
}, },
{ {
@ -704,6 +722,98 @@
], ],
"lookup_fields": [], "lookup_fields": [],
"historical": true, "historical": true,
"notify": true,
"relationship": false
},
{
"name": "attachment",
"schemas": [
{
"$id": "type_metadata",
"type": "object",
"properties": {
"type": {
"type": "string"
}
}
},
{
"$id": "other_metadata",
"type": "object",
"properties": {
"other": {
"type": "string"
}
}
},
{
"$id": "attachment",
"$ref": "entity",
"properties": {
"flags": {
"type": "array",
"items": {
"type": "string"
}
},
"type_metadata": {
"$ref": "type_metadata"
},
"other_metadata": {
"$ref": "other_metadata"
}
}
}
],
"hierarchy": [
"attachment",
"entity"
],
"fields": [
"id",
"type",
"flags",
"type_metadata",
"other_metadata",
"created_at",
"created_by",
"modified_at",
"modified_by",
"archived"
],
"grouped_fields": {
"attachment": [
"id",
"type",
"flags",
"type_metadata",
"other_metadata"
],
"entity": [
"id",
"type",
"created_at",
"created_by",
"modified_at",
"modified_by",
"archived"
]
},
"field_types": {
"id": "uuid",
"type": "text",
"flags": "_text",
"type_metadata": "jsonb",
"other_metadata": "jsonb",
"created_at": "timestamptz",
"created_by": "uuid",
"modified_at": "timestamptz",
"modified_by": "uuid",
"archived": "boolean"
},
"lookup_fields": [],
"historical": true,
"notify": true,
"relationship": false "relationship": false
} }
] ]
@ -728,6 +838,7 @@
"contact_id": "old-contact" "contact_id": "old-contact"
} }
], ],
"schema_id": "person",
"expect": { "expect": {
"success": true, "success": true,
"sql": [ "sql": [
@ -785,7 +896,8 @@
], ],
[ [
"INSERT INTO agreego.change (", "INSERT INTO agreego.change (",
" changes,", " \"old\",",
" \"new\",",
" entity_id,", " entity_id,",
" id,", " id,",
" kind,", " kind,",
@ -793,6 +905,7 @@
" modified_by", " modified_by",
")", ")",
"VALUES (", "VALUES (",
" NULL,",
" '{", " '{",
" \"first_name\":\"IncompleteFirst\",", " \"first_name\":\"IncompleteFirst\",",
" \"last_name\":\"IncompleteLast\",", " \"last_name\":\"IncompleteLast\",",
@ -816,6 +929,11 @@
" \"modified_at\":\"{{timestamp}}\",", " \"modified_at\":\"{{timestamp}}\",",
" \"modified_by\":\"00000000-0000-0000-0000-000000000000\",", " \"modified_by\":\"00000000-0000-0000-0000-000000000000\",",
" \"type\":\"person\"", " \"type\":\"person\"",
" },",
" \"new\":{",
" \"first_name\":\"IncompleteFirst\",",
" \"last_name\":\"IncompleteLast\",",
" \"type\":\"person\"",
" }", " }",
" }')" " }')"
] ]
@ -844,6 +962,7 @@
"contact_id": "old-contact" "contact_id": "old-contact"
} }
], ],
"schema_id": "person",
"expect": { "expect": {
"success": true, "success": true,
"sql": [ "sql": [
@ -872,7 +991,8 @@
], ],
[ [
"INSERT INTO agreego.change (", "INSERT INTO agreego.change (",
" changes,", " \"old\",",
" \"new\",",
" entity_id,", " entity_id,",
" id,", " id,",
" kind,", " kind,",
@ -881,6 +1001,9 @@
")", ")",
"VALUES (", "VALUES (",
" '{", " '{",
" \"contact_id\":\"old-contact\"",
" }',",
" '{",
" \"contact_id\":\"abc-contact\",", " \"contact_id\":\"abc-contact\",",
" \"type\":\"person\"", " \"type\":\"person\"",
" }',", " }',",
@ -893,10 +1016,6 @@
], ],
[ [
"SELECT pg_notify('entity', '{", "SELECT pg_notify('entity', '{",
" \"changes\":{",
" \"contact_id\":\"abc-contact\",",
" \"type\":\"person\"",
" },",
" \"complete\":{", " \"complete\":{",
" \"contact_id\":\"abc-contact\",", " \"contact_id\":\"abc-contact\",",
" \"date_of_birth\":\"1990-01-01T00:00:00Z\",", " \"date_of_birth\":\"1990-01-01T00:00:00Z\",",
@ -907,6 +1026,13 @@
" \"modified_by\":\"00000000-0000-0000-0000-000000000000\",", " \"modified_by\":\"00000000-0000-0000-0000-000000000000\",",
" \"pronouns\":\"they/them\",", " \"pronouns\":\"they/them\",",
" \"type\":\"person\"", " \"type\":\"person\"",
" },",
" \"new\":{",
" \"contact_id\":\"abc-contact\",",
" \"type\":\"person\"",
" },",
" \"old\":{",
" \"contact_id\":\"old-contact\"",
" }", " }",
" }')" " }')"
] ]
@ -930,6 +1056,7 @@
"last_name": "OldLast" "last_name": "OldLast"
} }
], ],
"schema_id": "person",
"expect": { "expect": {
"success": true, "success": true,
"sql": [ "sql": [
@ -959,7 +1086,8 @@
], ],
[ [
"INSERT INTO agreego.change (", "INSERT INTO agreego.change (",
" changes,", " \"old\",",
" \"new\",",
" entity_id,", " entity_id,",
" id,", " id,",
" kind,", " kind,",
@ -968,6 +1096,10 @@
")", ")",
"VALUES (", "VALUES (",
" '{", " '{",
" \"first_name\":\"OldFirst\",",
" \"last_name\":\"OldLast\"",
" }',",
" '{",
" \"first_name\":\"NewFirst\",", " \"first_name\":\"NewFirst\",",
" \"last_name\":\"NewLast\",", " \"last_name\":\"NewLast\",",
" \"type\":\"person\"", " \"type\":\"person\"",
@ -981,11 +1113,6 @@
], ],
[ [
"SELECT pg_notify('entity', '{", "SELECT pg_notify('entity', '{",
" \"changes\":{",
" \"first_name\":\"NewFirst\",",
" \"last_name\":\"NewLast\",",
" \"type\":\"person\"",
" },",
" \"complete\":{", " \"complete\":{",
" \"first_name\":\"NewFirst\",", " \"first_name\":\"NewFirst\",",
" \"id\":\"11111111-1111-1111-1111-111111111111\",", " \"id\":\"11111111-1111-1111-1111-111111111111\",",
@ -993,6 +1120,15 @@
" \"modified_at\":\"2026-03-10T00:00:00Z\",", " \"modified_at\":\"2026-03-10T00:00:00Z\",",
" \"modified_by\":\"00000000-0000-0000-0000-000000000000\",", " \"modified_by\":\"00000000-0000-0000-0000-000000000000\",",
" \"type\":\"person\"", " \"type\":\"person\"",
" },",
" \"new\":{",
" \"first_name\":\"NewFirst\",",
" \"last_name\":\"NewLast\",",
" \"type\":\"person\"",
" },",
" \"old\":{",
" \"first_name\":\"OldFirst\",",
" \"last_name\":\"OldLast\"",
" }", " }",
" }')" " }')"
] ]
@ -1010,6 +1146,7 @@
"date_of_birth": "1990-01-01T00:00:00Z", "date_of_birth": "1990-01-01T00:00:00Z",
"pronouns": "" "pronouns": ""
}, },
"schema_id": "person",
"expect": { "expect": {
"success": true, "success": true,
"sql": [ "sql": [
@ -1079,7 +1216,8 @@
], ],
[ [
"INSERT INTO agreego.change (", "INSERT INTO agreego.change (",
" changes,", " \"old\",",
" \"new\",",
" entity_id,", " entity_id,",
" id,", " id,",
" kind,", " kind,",
@ -1087,6 +1225,7 @@
" modified_by", " modified_by",
")", ")",
"VALUES (", "VALUES (",
" NULL,",
" '{", " '{",
" \"date_of_birth\":\"1990-01-01T00:00:00Z\",", " \"date_of_birth\":\"1990-01-01T00:00:00Z\",",
" \"first_name\":\"John\",", " \"first_name\":\"John\",",
@ -1114,6 +1253,13 @@
" \"modified_by\":\"00000000-0000-0000-0000-000000000000\",", " \"modified_by\":\"00000000-0000-0000-0000-000000000000\",",
" \"pronouns\":\"\",", " \"pronouns\":\"\",",
" \"type\":\"person\"", " \"type\":\"person\"",
" },",
" \"new\":{",
" \"date_of_birth\":\"1990-01-01T00:00:00Z\",",
" \"first_name\":\"John\",",
" \"last_name\":\"Doe\",",
" \"pronouns\":\"\",",
" \"type\":\"person\"",
" }", " }",
" }')" " }')"
] ]
@ -1133,6 +1279,7 @@
"date_of_birth": "2000-01-01" "date_of_birth": "2000-01-01"
} }
}, },
"schema_id": "order",
"expect": { "expect": {
"success": true, "success": true,
"sql": [ "sql": [
@ -1192,7 +1339,8 @@
], ],
[ [
"INSERT INTO agreego.change (", "INSERT INTO agreego.change (",
" changes,", " \"old\",",
" \"new\",",
" entity_id,", " entity_id,",
" id,", " id,",
" kind,", " kind,",
@ -1200,6 +1348,7 @@
" modified_by", " modified_by",
")", ")",
"VALUES (", "VALUES (",
" NULL,",
" '{", " '{",
" \"date_of_birth\":\"2000-01-01\",", " \"date_of_birth\":\"2000-01-01\",",
" \"first_name\":\"Bob\",", " \"first_name\":\"Bob\",",
@ -1247,7 +1396,8 @@
], ],
[ [
"INSERT INTO agreego.change (", "INSERT INTO agreego.change (",
" changes,", " \"old\",",
" \"new\",",
" entity_id,", " entity_id,",
" id,", " id,",
" kind,", " kind,",
@ -1255,6 +1405,7 @@
" modified_by", " modified_by",
")", ")",
"VALUES (", "VALUES (",
" NULL,",
" '{", " '{",
" \"customer_id\":\"{{uuid:customer_id}}\",", " \"customer_id\":\"{{uuid:customer_id}}\",",
" \"total\":100.0,", " \"total\":100.0,",
@ -1278,6 +1429,11 @@
" \"modified_by\":\"00000000-0000-0000-0000-000000000000\",", " \"modified_by\":\"00000000-0000-0000-0000-000000000000\",",
" \"total\":100.0,", " \"total\":100.0,",
" \"type\":\"order\"", " \"type\":\"order\"",
" },",
" \"new\":{",
" \"customer_id\":\"{{uuid:customer_id}}\",",
" \"total\":100.0,",
" \"type\":\"order\"",
" }", " }",
" }')" " }')"
], ],
@ -1293,6 +1449,12 @@
" \"modified_at\":\"{{timestamp}}\",", " \"modified_at\":\"{{timestamp}}\",",
" \"modified_by\":\"00000000-0000-0000-0000-000000000000\",", " \"modified_by\":\"00000000-0000-0000-0000-000000000000\",",
" \"type\":\"person\"", " \"type\":\"person\"",
" },",
" \"new\":{",
" \"date_of_birth\":\"2000-01-01\",",
" \"first_name\":\"Bob\",",
" \"last_name\":\"Smith\",",
" \"type\":\"person\"",
" }", " }",
" }')" " }')"
] ]
@ -1314,6 +1476,7 @@
} }
] ]
}, },
"schema_id": "order",
"expect": { "expect": {
"success": true, "success": true,
"sql": [ "sql": [
@ -1388,7 +1551,8 @@
], ],
[ [
"INSERT INTO agreego.change (", "INSERT INTO agreego.change (",
" changes,", " \"old\",",
" \"new\",",
" entity_id,", " entity_id,",
" id,", " id,",
" kind,", " kind,",
@ -1396,6 +1560,7 @@
" modified_by", " modified_by",
")", ")",
"VALUES (", "VALUES (",
" NULL,",
" '{", " '{",
" \"order_id\":\"abc\",", " \"order_id\":\"abc\",",
" \"price\":99.0,", " \"price\":99.0,",
@ -1411,7 +1576,8 @@
], ],
[ [
"INSERT INTO agreego.change (", "INSERT INTO agreego.change (",
" changes,", " \"old\",",
" \"new\",",
" entity_id,", " entity_id,",
" id,", " id,",
" kind,", " kind,",
@ -1419,6 +1585,7 @@
" modified_by", " modified_by",
")", ")",
"VALUES (", "VALUES (",
" NULL,",
" '{", " '{",
" \"total\":99.0,", " \"total\":99.0,",
" \"type\":\"order\"", " \"type\":\"order\"",
@ -1440,6 +1607,10 @@
" \"modified_by\":\"00000000-0000-0000-0000-000000000000\",", " \"modified_by\":\"00000000-0000-0000-0000-000000000000\",",
" \"total\":99.0,", " \"total\":99.0,",
" \"type\":\"order\"", " \"type\":\"order\"",
" },",
" \"new\":{",
" \"total\":99.0,",
" \"type\":\"order\"",
" }", " }",
" }')" " }')"
], ],
@ -1455,6 +1626,12 @@
" \"price\":99.0,", " \"price\":99.0,",
" \"product\":\"Widget\",", " \"product\":\"Widget\",",
" \"type\":\"order_line\"", " \"type\":\"order_line\"",
" },",
" \"new\":{",
" \"order_id\":\"abc\",",
" \"price\":99.0,",
" \"product\":\"Widget\",",
" \"type\":\"order_line\"",
" }", " }",
" }')" " }')"
] ]
@ -1495,6 +1672,7 @@
} }
] ]
}, },
"schema_id": "person",
"expect": { "expect": {
"success": true, "success": true,
"sql": [ "sql": [
@ -1572,13 +1750,15 @@
], ],
[ [
"INSERT INTO agreego.change (", "INSERT INTO agreego.change (",
" changes,", " \"old\",",
" \"new\",",
" entity_id,", " entity_id,",
" id,", " id,",
" kind,", " kind,",
" modified_at,", " modified_at,",
" modified_by", " modified_by",
") VALUES (", ") VALUES (",
" NULL,",
" '{\"number\":\"555-0001\",\"type\":\"phone_number\"}',", " '{\"number\":\"555-0001\",\"type\":\"phone_number\"}',",
" '{{uuid:phone1_id}}',", " '{{uuid:phone1_id}}',",
" '{{uuid}}',", " '{{uuid}}',",
@ -1626,13 +1806,15 @@
], ],
[ [
"INSERT INTO agreego.change (", "INSERT INTO agreego.change (",
" changes,", " \"old\",",
" \"new\",",
" entity_id,", " entity_id,",
" id,", " id,",
" kind,", " kind,",
" modified_at,", " modified_at,",
" modified_by", " modified_by",
") VALUES (", ") VALUES (",
" NULL,",
" '{", " '{",
" \"is_primary\":true,", " \"is_primary\":true,",
" \"source_id\":\"{{uuid:person_id}}\",", " \"source_id\":\"{{uuid:person_id}}\",",
@ -1674,13 +1856,15 @@
], ],
[ [
"INSERT INTO agreego.change (", "INSERT INTO agreego.change (",
" changes,", " \"old\",",
" \"new\",",
" entity_id,", " entity_id,",
" id,", " id,",
" kind,", " kind,",
" modified_at,", " modified_at,",
" modified_by", " modified_by",
") VALUES (", ") VALUES (",
" NULL,",
" '{", " '{",
" \"number\":\"555-0002\",", " \"number\":\"555-0002\",",
" \"type\":\"phone_number\"", " \"type\":\"phone_number\"",
@ -1731,13 +1915,15 @@
], ],
[ [
"INSERT INTO agreego.change (", "INSERT INTO agreego.change (",
" changes,", " \"old\",",
" \"new\",",
" entity_id,", " entity_id,",
" id,", " id,",
" kind,", " kind,",
" modified_at,", " modified_at,",
" modified_by", " modified_by",
") VALUES (", ") VALUES (",
" NULL,",
" '{", " '{",
" \"is_primary\":false,", " \"is_primary\":false,",
" \"source_id\":\"{{uuid:person_id}}\",", " \"source_id\":\"{{uuid:person_id}}\",",
@ -1779,13 +1965,15 @@
], ],
[ [
"INSERT INTO agreego.change (", "INSERT INTO agreego.change (",
" changes,", " \"old\",",
" \"new\",",
" entity_id,", " entity_id,",
" id,", " id,",
" kind,", " kind,",
" modified_at,", " modified_at,",
" modified_by", " modified_by",
") VALUES (", ") VALUES (",
" NULL,",
" '{", " '{",
" \"address\":\"test@example.com\",", " \"address\":\"test@example.com\",",
" \"type\":\"email_address\"", " \"type\":\"email_address\"",
@ -1836,13 +2024,15 @@
], ],
[ [
"INSERT INTO agreego.change (", "INSERT INTO agreego.change (",
" changes,", " \"old\",",
" \"new\",",
" entity_id,", " entity_id,",
" id,", " id,",
" kind,", " kind,",
" modified_at,", " modified_at,",
" modified_by", " modified_by",
") VALUES (", ") VALUES (",
" NULL,",
" '{", " '{",
" \"is_primary\":false,", " \"is_primary\":false,",
" \"source_id\":\"{{uuid:person_id}}\",", " \"source_id\":\"{{uuid:person_id}}\",",
@ -1860,13 +2050,15 @@
], ],
[ [
"INSERT INTO agreego.change (", "INSERT INTO agreego.change (",
" changes,", " \"old\",",
" \"new\",",
" entity_id,", " entity_id,",
" id,", " id,",
" kind,", " kind,",
" modified_at,", " modified_at,",
" modified_by", " modified_by",
") VALUES (", ") VALUES (",
" NULL,",
" '{", " '{",
" \"first_name\":\"Relation\",", " \"first_name\":\"Relation\",",
" \"last_name\":\"Test\",", " \"last_name\":\"Test\",",
@ -1890,6 +2082,11 @@
" \"modified_at\":\"{{timestamp}}\",", " \"modified_at\":\"{{timestamp}}\",",
" \"modified_by\":\"00000000-0000-0000-0000-000000000000\",", " \"modified_by\":\"00000000-0000-0000-0000-000000000000\",",
" \"type\":\"person\"", " \"type\":\"person\"",
" },",
" \"new\":{",
" \"first_name\":\"Relation\",",
" \"last_name\":\"Test\",",
" \"type\":\"person\"",
" }", " }",
" }')" " }')"
], ],
@ -1907,6 +2104,14 @@
" \"target_id\":\"{{uuid:phone1_id}}\",", " \"target_id\":\"{{uuid:phone1_id}}\",",
" \"target_type\":\"phone_number\",", " \"target_type\":\"phone_number\",",
" \"type\":\"contact\"", " \"type\":\"contact\"",
" },",
" \"new\":{",
" \"is_primary\":true,",
" \"source_id\":\"{{uuid:person_id}}\",",
" \"source_type\":\"person\",",
" \"target_id\":\"{{uuid:phone1_id}}\",",
" \"target_type\":\"phone_number\",",
" \"type\":\"contact\"",
" }", " }",
" }')" " }')"
], ],
@ -1920,6 +2125,10 @@
" \"modified_by\":\"00000000-0000-0000-0000-000000000000\",", " \"modified_by\":\"00000000-0000-0000-0000-000000000000\",",
" \"number\":\"555-0001\",", " \"number\":\"555-0001\",",
" \"type\":\"phone_number\"", " \"type\":\"phone_number\"",
" },",
" \"new\":{",
" \"number\":\"555-0001\",",
" \"type\":\"phone_number\"",
" }", " }",
" }')" " }')"
], ],
@ -1937,6 +2146,14 @@
" \"target_id\":\"{{uuid:phone2_id}}\",", " \"target_id\":\"{{uuid:phone2_id}}\",",
" \"target_type\":\"phone_number\",", " \"target_type\":\"phone_number\",",
" \"type\":\"contact\"", " \"type\":\"contact\"",
" },",
" \"new\":{",
" \"is_primary\":false,",
" \"source_id\":\"{{uuid:person_id}}\",",
" \"source_type\":\"person\",",
" \"target_id\":\"{{uuid:phone2_id}}\",",
" \"target_type\":\"phone_number\",",
" \"type\":\"contact\"",
" }", " }",
" }')" " }')"
], ],
@ -1950,6 +2167,10 @@
" \"modified_by\":\"00000000-0000-0000-0000-000000000000\",", " \"modified_by\":\"00000000-0000-0000-0000-000000000000\",",
" \"number\":\"555-0002\",", " \"number\":\"555-0002\",",
" \"type\":\"phone_number\"", " \"type\":\"phone_number\"",
" },",
" \"new\":{",
" \"number\":\"555-0002\",",
" \"type\":\"phone_number\"",
" }", " }",
" }')" " }')"
], ],
@ -1967,6 +2188,14 @@
" \"target_id\":\"{{uuid:email1_id}}\",", " \"target_id\":\"{{uuid:email1_id}}\",",
" \"target_type\":\"email_address\",", " \"target_type\":\"email_address\",",
" \"type\":\"contact\"", " \"type\":\"contact\"",
" },",
" \"new\":{",
" \"is_primary\":false,",
" \"source_id\":\"{{uuid:person_id}}\",",
" \"source_type\":\"person\",",
" \"target_id\":\"{{uuid:email1_id}}\",",
" \"target_type\":\"email_address\",",
" \"type\":\"contact\"",
" }", " }",
" }')" " }')"
], ],
@ -1980,6 +2209,10 @@
" \"modified_at\":\"{{timestamp}}\",", " \"modified_at\":\"{{timestamp}}\",",
" \"modified_by\":\"00000000-0000-0000-0000-000000000000\",", " \"modified_by\":\"00000000-0000-0000-0000-000000000000\",",
" \"type\":\"email_address\"", " \"type\":\"email_address\"",
" },",
" \"new\":{",
" \"address\":\"test@example.com\",",
" \"type\":\"email_address\"",
" }", " }",
" }')" " }')"
] ]
@ -2003,6 +2236,7 @@
"archived": false "archived": false
} }
], ],
"schema_id": "person",
"expect": { "expect": {
"success": true, "success": true,
"sql": [ "sql": [
@ -2023,13 +2257,17 @@
], ],
[ [
"INSERT INTO agreego.change (", "INSERT INTO agreego.change (",
" changes,", " \"old\",",
" \"new\",",
" entity_id,", " entity_id,",
" id,", " id,",
" kind,", " kind,",
" modified_at,", " modified_at,",
" modified_by", " modified_by",
") VALUES (", ") VALUES (",
" '{",
" \"archived\":false",
" }',",
" '{\"archived\":true,\"type\":\"person\"}',", " '{\"archived\":true,\"type\":\"person\"}',",
" 'abc-archived',", " 'abc-archived',",
" '{{uuid}}',", " '{{uuid}}',",
@ -2040,8 +2278,6 @@
], ],
[ [
"SELECT pg_notify('entity', '{", "SELECT pg_notify('entity', '{",
" \"changes\":{",
" \"archived\":true,\"type\":\"person\"},",
" \"complete\":{", " \"complete\":{",
" \"archived\":true,", " \"archived\":true,",
" \"first_name\":\"ArchivedFirst\",", " \"first_name\":\"ArchivedFirst\",",
@ -2050,6 +2286,116 @@
" \"modified_at\":\"{{timestamp}}\",", " \"modified_at\":\"{{timestamp}}\",",
" \"modified_by\":\"00000000-0000-0000-0000-000000000000\",", " \"modified_by\":\"00000000-0000-0000-0000-000000000000\",",
" \"type\":\"person\"", " \"type\":\"person\"",
" },",
" \"new\":{",
" \"archived\":true,",
" \"type\":\"person\"",
" },",
" \"old\":{",
" \"archived\":false",
" }",
" }')"
]
]
}
},
{
"description": "Attachment with text[] and jsonb metadata structures",
"action": "merge",
"data": {
"type": "attachment",
"flags": [
"urgent",
"reviewed"
],
"other_metadata": {
"other": "hello"
},
"type_metadata": {
"type": "type_metadata"
}
},
"schema_id": "attachment",
"expect": {
"success": true,
"sql": [
[
"INSERT INTO agreego.\"entity\" (",
" \"created_at\",",
" \"created_by\",",
" \"id\",",
" \"modified_at\",",
" \"modified_by\",",
" \"type\"",
")",
"VALUES (",
" '{{timestamp}}',",
" '00000000-0000-0000-0000-000000000000',",
" '{{uuid:attachment_id}}',",
" '{{timestamp}}',",
" '00000000-0000-0000-0000-000000000000',",
" 'attachment'",
")"
],
[
"INSERT INTO agreego.\"attachment\" (",
" \"flags\",",
" \"id\",",
" \"other_metadata\",",
" \"type\",",
" \"type_metadata\"",
")",
"VALUES (",
" '{\"urgent\",\"reviewed\"}',",
" '{{uuid:attachment_id}}',",
" '{\"other\":\"hello\"}',",
" 'attachment',",
" '{\"type\":\"type_metadata\"}'",
")"
],
[
"INSERT INTO agreego.change (",
" \"old\",",
" \"new\",",
" entity_id,",
" id,",
" kind,",
" modified_at,",
" modified_by",
")",
"VALUES (",
" NULL,",
" '{",
" \"flags\":[\"urgent\",\"reviewed\"],",
" \"other_metadata\":{\"other\":\"hello\"},",
" \"type\":\"attachment\",",
" \"type_metadata\":{\"type\":\"type_metadata\"}",
" }',",
" '{{uuid:attachment_id}}',",
" '{{uuid}}',",
" 'create',",
" '{{timestamp}}',",
" '00000000-0000-0000-0000-000000000000'",
")"
],
[
"SELECT pg_notify('entity', '{",
" \"complete\":{",
" \"created_at\":\"{{timestamp}}\",",
" \"created_by\":\"00000000-0000-0000-0000-000000000000\",",
" \"flags\":[\"urgent\",\"reviewed\"],",
" \"id\":\"{{uuid:attachment_id}}\",",
" \"modified_at\":\"{{timestamp}}\",",
" \"modified_by\":\"00000000-0000-0000-0000-000000000000\",",
" \"other_metadata\":{\"other\":\"hello\"},",
" \"type\":\"attachment\",",
" \"type_metadata\":{\"type\":\"type_metadata\"}",
" },",
" \"new\":{",
" \"flags\":[\"urgent\",\"reviewed\"],",
" \"other_metadata\":{\"other\":\"hello\"},",
" \"type\":\"attachment\",",
" \"type_metadata\":{\"type\":\"type_metadata\"}",
" }", " }",
" }')" " }')"
] ]

View File

@ -858,27 +858,6 @@
] ]
} }
}, },
{
"description": "Simple entity select on root stem",
"action": "query",
"schema_id": "entity",
"stem": "",
"expect": {
"success": true,
"sql": [
[
"(SELECT jsonb_build_object(",
" 'archived', entity_1.archived,",
" 'created_at', entity_1.created_at,",
" 'id', entity_1.id,",
" 'name', entity_1.name,",
" 'type', entity_1.type)",
"FROM agreego.entity entity_1",
"WHERE NOT entity_1.archived)"
]
]
}
},
{ {
"description": "Simple entity select with multiple filters", "description": "Simple entity select with multiple filters",
"action": "query", "action": "query",
@ -1430,10 +1409,9 @@
} }
}, },
{ {
"description": "Full person stem query on phone number contact", "description": "Person ad-hoc email addresses select",
"action": "query", "action": "query",
"schema_id": "full.person", "schema_id": "full.person/email_addresses",
"stem": "phone_numbers.#",
"expect": { "expect": {
"success": true, "success": true,
"sql": [ "sql": [
@ -1446,19 +1424,18 @@
" 'name', entity_3.name,", " 'name', entity_3.name,",
" 'target',", " 'target',",
" (SELECT jsonb_build_object(", " (SELECT jsonb_build_object(",
" 'address', email_address_4.address,",
" 'archived', entity_5.archived,", " 'archived', entity_5.archived,",
" 'created_at', entity_5.created_at,", " 'created_at', entity_5.created_at,",
" 'id', entity_5.id,", " 'id', entity_5.id,",
" 'name', entity_5.name,", " 'name', entity_5.name,",
" 'number', phone_number_4.number,",
" 'type', entity_5.type", " 'type', entity_5.type",
" )", " )",
" FROM agreego.phone_number phone_number_4", " FROM agreego.email_address email_address_4",
" JOIN agreego.entity entity_5 ON entity_5.id = phone_number_4.id", " JOIN agreego.entity entity_5 ON entity_5.id = email_address_4.id",
" WHERE", " WHERE",
" NOT entity_5.archived", " NOT entity_5.archived",
" AND relationship_2.target_id = entity_5.id", " AND relationship_2.target_id = entity_5.id),",
" ),",
" 'type', entity_3.type", " 'type', entity_3.type",
")", ")",
"FROM agreego.contact contact_1", "FROM agreego.contact contact_1",
@ -1469,54 +1446,6 @@
] ]
} }
}, },
{
"description": "Full person stem query on phone number contact on phone number",
"action": "query",
"schema_id": "full.person",
"stem": "phone_numbers.#.target",
"expect": {
"success": true,
"sql": [
[
"(SELECT jsonb_build_object(",
" 'archived', entity_2.archived,",
" 'created_at', entity_2.created_at,",
" 'id', entity_2.id,",
" 'name', entity_2.name,",
" 'number', phone_number_1.number,",
" 'type', entity_2.type",
")",
"FROM agreego.phone_number phone_number_1",
"JOIN agreego.entity entity_2 ON entity_2.id = phone_number_1.id",
"WHERE NOT entity_2.archived)"
]
]
}
},
{
"description": "Full person stem query on contact email address",
"action": "query",
"schema_id": "full.person",
"stem": "contacts.#.target#(type==\"email_address\")",
"expect": {
"success": true,
"sql": [
[
"(SELECT jsonb_build_object(",
" 'address', email_address_1.address,",
" 'archived', entity_2.archived,",
" 'created_at', entity_2.created_at,",
" 'id', entity_2.id,",
" 'name', entity_2.name,",
" 'type', entity_2.type",
")",
"FROM agreego.email_address email_address_1",
"JOIN agreego.entity entity_2 ON entity_2.id = email_address_1.id",
"WHERE NOT entity_2.archived)"
]
]
}
},
{ {
"description": "Order select with customer and lines", "description": "Order select with customer and lines",
"action": "query", "action": "query",

View File

@ -1,312 +0,0 @@
[
{
"description": "Stem Engine Unit Tests",
"database": {
"puncs": [],
"enums": [],
"relations": [
{
"id": "rel1",
"type": "relation",
"constraint": "fk_contact_entity",
"source_type": "contact",
"source_columns": [
"entity_id"
],
"destination_type": "person",
"destination_columns": [
"id"
],
"prefix": null
},
{
"id": "rel2",
"type": "relation",
"constraint": "fk_relationship_target",
"source_type": "relationship",
"source_columns": [
"target_id",
"target_type"
],
"destination_type": "entity",
"destination_columns": [
"id",
"type"
],
"prefix": "target"
}
],
"types": [
{
"name": "entity",
"hierarchy": [
"entity"
],
"schemas": [
{
"$id": "entity",
"type": "object",
"properties": {}
}
]
},
{
"name": "person",
"hierarchy": [
"person",
"entity"
],
"schemas": [
{
"$id": "person",
"$ref": "entity",
"properties": {}
}
]
},
{
"name": "email_address",
"hierarchy": [
"email_address",
"entity"
],
"schemas": [
{
"$id": "email_address",
"$ref": "entity",
"properties": {}
}
]
},
{
"name": "phone_number",
"hierarchy": [
"phone_number",
"entity"
],
"schemas": [
{
"$id": "phone_number",
"$ref": "entity",
"properties": {}
}
]
},
{
"name": "relationship",
"relationship": true,
"hierarchy": [
"relationship",
"entity"
],
"schemas": [
{
"$id": "relationship",
"$ref": "entity",
"properties": {}
}
]
},
{
"name": "contact",
"relationship": true,
"hierarchy": [
"contact",
"relationship",
"entity"
],
"schemas": [
{
"$id": "contact",
"$ref": "relationship",
"properties": {
"target": {
"oneOf": [
{
"$ref": "phone_number"
},
{
"$ref": "email_address"
}
]
}
}
}
]
},
{
"name": "save_person",
"schemas": [
{
"$id": "save_person.response",
"$ref": "person",
"properties": {
"contacts": {
"type": "array",
"items": {
"$ref": "contact"
}
}
}
}
]
}
]
},
"tests": [
{
"description": "correctly squashes deep oneOf refs through array paths",
"action": "compile",
"expect": {
"success": true,
"stems": {
"contact": {
"": {
"schema": {
"$id": "contact",
"$ref": "relationship",
"properties": {
"target": {
"oneOf": [
{
"$ref": "phone_number"
},
{
"$ref": "email_address"
}
]
}
}
},
"type": "contact"
},
"target#(type==\"email_address\")": {
"relation": "target_id",
"schema": {
"$id": "email_address",
"$ref": "entity",
"properties": {}
},
"type": "email_address"
},
"target#(type==\"phone_number\")": {
"relation": "target_id",
"schema": {
"$id": "phone_number",
"$ref": "entity",
"properties": {}
},
"type": "phone_number"
}
},
"email_address": {
"": {
"schema": {
"$id": "email_address",
"$ref": "entity",
"properties": {}
},
"type": "email_address"
}
},
"entity": {
"": {
"schema": {
"$id": "entity",
"properties": {},
"type": "object"
},
"type": "entity"
}
},
"person": {
"": {
"schema": {
"$id": "person",
"$ref": "entity",
"properties": {}
},
"type": "person"
}
},
"phone_number": {
"": {
"schema": {
"$id": "phone_number",
"$ref": "entity",
"properties": {}
},
"type": "phone_number"
}
},
"relationship": {
"": {
"schema": {
"$id": "relationship",
"$ref": "entity",
"properties": {}
},
"type": "relationship"
}
},
"save_person.response": {
"": {
"schema": {
"$id": "save_person.response",
"$ref": "person",
"properties": {
"contacts": {
"items": {
"$ref": "contact"
},
"type": "array"
}
}
},
"type": "person"
},
"contacts.#": {
"relation": "contacts_id",
"schema": {
"$id": "contact",
"$ref": "relationship",
"properties": {
"target": {
"oneOf": [
{
"$ref": "phone_number"
},
{
"$ref": "email_address"
}
]
}
}
},
"type": "contact"
},
"contacts.#.target#(type==\"email_address\")": {
"relation": "target_id",
"schema": {
"$id": "email_address",
"$ref": "entity",
"properties": {}
},
"type": "email_address"
},
"contacts.#.target#(type==\"phone_number\")": {
"relation": "target_id",
"schema": {
"$id": "phone_number",
"$ref": "entity",
"properties": {}
},
"type": "phone_number"
}
}
}
}
}
]
}
]

38
fixtures/t1.json Normal file
View File

@ -0,0 +1,38 @@
{
"description": "Person ad-hoc email addresses select",
"action": "query",
"schema_id": "full.person/email_addresses",
"expect": {
"success": true,
"sql": [
[
"(SELECT jsonb_build_object(",
" 'archived', entity_3.archived,",
" 'created_at', entity_3.created_at,",
" 'id', entity_3.id,",
" 'is_primary', contact_1.is_primary,",
" 'name', entity_3.name,",
" 'target',",
" (SELECT jsonb_build_object(",
" 'address', email_address_4.address,",
" 'archived', entity_5.archived,",
" 'created_at', entity_5.created_at,",
" 'id', entity_5.id,",
" 'name', entity_5.name,",
" 'type', entity_5.type",
" )",
" FROM agreego.email_address email_address_4",
" JOIN agreego.entity entity_5 ON entity_5.id = email_address_4.id",
" WHERE",
" NOT entity_5.archived",
" AND relationship_2.target_id = entity_5.id),",
" 'type', entity_3.type",
")",
"FROM agreego.contact contact_1",
"JOIN agreego.relationship relationship_2 ON relationship_2.id = contact_1.id",
"JOIN agreego.entity entity_3 ON entity_3.id = relationship_2.id",
"WHERE NOT entity_3.archived)"
]
]
}
}

7
src/database/edge.rs Normal file
View File

@ -0,0 +1,7 @@
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
pub struct Edge {
pub constraint: String,
pub forward: bool,
}

View File

@ -1,3 +1,4 @@
pub mod edge;
pub mod r#enum; pub mod r#enum;
pub mod executors; pub mod executors;
pub mod formats; pub mod formats;
@ -18,24 +19,19 @@ use executors::pgrx::SpiExecutor;
#[cfg(test)] #[cfg(test)]
use executors::mock::MockExecutor; use executors::mock::MockExecutor;
pub mod stem;
use punc::Punc; use punc::Punc;
use relation::Relation; use relation::Relation;
use schema::Schema; use schema::Schema;
use serde_json::Value; use serde_json::Value;
use std::collections::{HashMap, HashSet}; use std::collections::{HashMap, HashSet};
use std::sync::Arc;
use stem::Stem;
use r#type::Type; use r#type::Type;
pub struct Database { pub struct Database {
pub enums: HashMap<String, Enum>, pub enums: HashMap<String, Enum>,
pub types: HashMap<String, Type>, pub types: HashMap<String, Type>,
pub puncs: HashMap<String, Punc>, pub puncs: HashMap<String, Punc>,
pub relations: Vec<Relation>, pub relations: HashMap<String, Relation>,
pub schemas: HashMap<String, Schema>, pub schemas: HashMap<String, Schema>,
// Map of Schema ID -> { Entity Type -> Target Subschema Arc }
pub stems: HashMap<String, HashMap<String, Arc<Stem>>>,
pub descendants: HashMap<String, Vec<String>>, pub descendants: HashMap<String, Vec<String>>,
pub depths: HashMap<String, usize>, pub depths: HashMap<String, usize>,
pub executor: Box<dyn DatabaseExecutor + Send + Sync>, pub executor: Box<dyn DatabaseExecutor + Send + Sync>,
@ -46,10 +42,9 @@ impl Database {
let mut db = Self { let mut db = Self {
enums: HashMap::new(), enums: HashMap::new(),
types: HashMap::new(), types: HashMap::new(),
relations: Vec::new(), relations: HashMap::new(),
puncs: HashMap::new(), puncs: HashMap::new(),
schemas: HashMap::new(), schemas: HashMap::new(),
stems: HashMap::new(),
descendants: HashMap::new(), descendants: HashMap::new(),
depths: HashMap::new(), depths: HashMap::new(),
#[cfg(not(test))] #[cfg(not(test))]
@ -81,10 +76,21 @@ impl Database {
if db.types.contains_key(&def.source_type) if db.types.contains_key(&def.source_type)
&& db.types.contains_key(&def.destination_type) && db.types.contains_key(&def.destination_type)
{ {
db.relations.push(def); db.relations.insert(def.constraint.clone(), def);
} }
} }
Err(e) => println!("DATABASE RELATION PARSE FAILED: {:?}", e), Err(e) => {
return Err(crate::drop::Drop::with_errors(vec![crate::drop::Error {
code: "DATABASE_RELATION_PARSE_FAILED".to_string(),
message: format!("Failed to parse database relation: {}", e),
details: crate::drop::ErrorDetails {
path: "".to_string(),
cause: None,
context: None,
schema: None,
},
}]));
}
} }
} }
} }
@ -142,38 +148,67 @@ impl Database {
} }
pub fn compile(&mut self) -> Result<(), crate::drop::Drop> { pub fn compile(&mut self) -> Result<(), crate::drop::Drop> {
self.collect_schemas(); let mut harvested = Vec::new();
for schema in self.schemas.values_mut() {
if let Err(msg) = schema.collect_schemas(None, &mut harvested) {
return Err(crate::drop::Drop::with_errors(vec![crate::drop::Error {
code: "SCHEMA_VALIDATION_FAILED".to_string(),
message: msg,
details: crate::drop::ErrorDetails { path: "".to_string(), cause: None, context: None, schema: None },
}]));
}
}
self.schemas.extend(harvested);
if let Err(msg) = self.collect_schemas() {
return Err(crate::drop::Drop::with_errors(vec![crate::drop::Error {
code: "SCHEMA_VALIDATION_FAILED".to_string(),
message: msg,
details: crate::drop::ErrorDetails {
path: "".to_string(),
cause: None,
context: None,
schema: None,
},
}]));
}
self.collect_depths(); self.collect_depths();
self.collect_descendants(); self.collect_descendants();
self.compile_schemas();
self.collect_stems()?; // Mathematically evaluate all property inheritances, formats, schemas, and foreign key edges topographically over OnceLocks
let mut visited = std::collections::HashSet::new();
for schema in self.schemas.values() {
schema.compile(self, &mut visited);
}
Ok(()) Ok(())
} }
fn collect_schemas(&mut self) { fn collect_schemas(&mut self) -> Result<(), String> {
let mut to_insert = Vec::new(); let mut to_insert = Vec::new();
// Pass 1: Extract all Schemas structurally off top level definitions into the master registry. // Pass 1: Extract all Schemas structurally off top level definitions into the master registry.
// Validate every node recursively via string filters natively!
for type_def in self.types.values() { for type_def in self.types.values() {
for mut schema in type_def.schemas.clone() { for mut schema in type_def.schemas.clone() {
schema.harvest(&mut to_insert); schema.collect_schemas(None, &mut to_insert)?;
} }
} }
for punc_def in self.puncs.values() { for punc_def in self.puncs.values() {
for mut schema in punc_def.schemas.clone() { for mut schema in punc_def.schemas.clone() {
schema.harvest(&mut to_insert); schema.collect_schemas(None, &mut to_insert)?;
} }
} }
for enum_def in self.enums.values() { for enum_def in self.enums.values() {
for mut schema in enum_def.schemas.clone() { for mut schema in enum_def.schemas.clone() {
schema.harvest(&mut to_insert); schema.collect_schemas(None, &mut to_insert)?;
} }
} }
for (id, schema) in to_insert { for (id, schema) in to_insert {
self.schemas.insert(id, schema); self.schemas.insert(id, schema);
} }
Ok(())
} }
fn collect_depths(&mut self) { fn collect_depths(&mut self) {
@ -229,83 +264,10 @@ impl Database {
self.descendants = descendants; self.descendants = descendants;
} }
pub fn get_relation(
&self,
parent_type: &str,
child_type: &str,
prop_name: &str,
relative_keys: Option<&Vec<String>>,
) -> Option<(&Relation, bool)> {
if parent_type == "entity" && child_type == "entity" {
return None; // Ignore entity <-> entity generic fallbacks, they aren't useful edges
}
let p_def = self.types.get(parent_type)?;
let c_def = self.types.get(child_type)?;
let mut matching_rels = Vec::new();
let mut directions = Vec::new();
for rel in &self.relations {
let is_forward = p_def.hierarchy.contains(&rel.source_type)
&& c_def.hierarchy.contains(&rel.destination_type);
let is_reverse = p_def.hierarchy.contains(&rel.destination_type)
&& c_def.hierarchy.contains(&rel.source_type);
if is_forward {
matching_rels.push(rel);
directions.push(true);
} else if is_reverse {
matching_rels.push(rel);
directions.push(false);
}
}
if matching_rels.is_empty() {
return None;
}
if matching_rels.len() == 1 {
return Some((matching_rels[0], directions[0]));
}
let mut chosen_idx = 0;
let mut resolved = false;
// Reduce ambiguity with prefix
for (i, rel) in matching_rels.iter().enumerate() {
if let Some(prefix) = &rel.prefix {
if prefix == prop_name {
chosen_idx = i;
resolved = true;
break;
}
}
}
// Reduce ambiguity by checking if relative payload OMITS the prefix (M:M heuristic)
if !resolved && relative_keys.is_some() {
let keys = relative_keys.unwrap();
let mut missing_prefix_ids = Vec::new();
for (i, rel) in matching_rels.iter().enumerate() {
if let Some(prefix) = &rel.prefix {
if !keys.contains(prefix) {
missing_prefix_ids.push(i);
}
}
}
if missing_prefix_ids.len() == 1 {
chosen_idx = missing_prefix_ids[0];
}
}
Some((matching_rels[chosen_idx], directions[chosen_idx]))
}
fn collect_descendants_recursively( fn collect_descendants_recursively(
target: &str, target: &str,
direct_refs: &HashMap<String, Vec<String>>, direct_refs: &std::collections::HashMap<String, Vec<String>>,
descendants: &mut HashSet<String>, descendants: &mut std::collections::HashSet<String>,
) { ) {
if let Some(children) = direct_refs.get(target) { if let Some(children) = direct_refs.get(target) {
for child in children { for child in children {
@ -315,246 +277,4 @@ impl Database {
} }
} }
} }
fn compile_schemas(&mut self) {
// Pass 3: compile_internals across pure structure
let schema_ids: Vec<String> = self.schemas.keys().cloned().collect();
for id in schema_ids {
if let Some(schema) = self.schemas.get_mut(&id) {
schema.compile_internals();
}
}
}
fn collect_stems(&mut self) -> Result<(), crate::drop::Drop> {
let mut db_stems: HashMap<String, HashMap<String, Arc<Stem>>> = HashMap::new();
let mut errors: Vec<crate::drop::Error> = Vec::new();
let schema_ids: Vec<String> = self.schemas.keys().cloned().collect();
for schema_id in schema_ids {
if let Some(schema) = self.schemas.get(&schema_id) {
let mut inner_map = HashMap::new();
Self::discover_stems(
self,
&schema_id,
schema,
String::from(""),
None,
None,
false,
&mut inner_map,
Vec::new(),
&mut errors,
);
if !inner_map.is_empty() {
db_stems.insert(schema_id, inner_map);
}
}
}
self.stems = db_stems;
if !errors.is_empty() {
return Err(crate::drop::Drop::with_errors(errors));
}
Ok(())
}
fn discover_stems(
db: &Database,
root_schema_id: &str,
schema: &Schema,
current_path: String,
parent_type: Option<String>,
property_name: Option<String>,
is_polymorphic: bool,
inner_map: &mut HashMap<String, Arc<Stem>>,
seen_entities: Vec<String>,
errors: &mut Vec<crate::drop::Error>,
) {
let mut is_entity = false;
let mut entity_type = String::new();
// First check if the Schema's $id is a native Database Type
if let Some(ref id) = schema.obj.id {
let parts: Vec<&str> = id.split('.').collect();
if let Some(last_seg) = parts.last() {
if db.types.contains_key(*last_seg) {
is_entity = true;
entity_type = last_seg.to_string();
}
}
}
// If not found via $id, check the $ref pointer
// This allows ad-hoc schemas (like `save_person.response`) to successfully adopt the Type of what they $ref
if !is_entity {
if let Some(ref r) = schema.obj.r#ref {
let parts: Vec<&str> = r.split('.').collect();
if let Some(last_seg) = parts.last() {
if db.types.contains_key(*last_seg) {
is_entity = true;
entity_type = last_seg.to_string();
}
}
}
}
if is_entity {
if seen_entities.contains(&entity_type) {
return; // Break cyclical schemas!
}
}
let mut relation_col = None;
if is_entity {
if let (Some(pt), Some(prop)) = (&parent_type, &property_name) {
let expected_col = format!("{}_id", prop);
let mut found = false;
if let Some((rel, _)) = db.get_relation(pt, &entity_type, prop, None) {
if rel.source_columns.contains(&expected_col) {
relation_col = Some(expected_col.clone());
found = true;
}
}
if !found {
relation_col = Some(expected_col);
}
}
let mut final_path = current_path.clone();
if is_polymorphic && !final_path.is_empty() && !final_path.ends_with(&entity_type) {
if final_path.ends_with(".#") {
final_path = format!("{}(type==\"{}\")", final_path, entity_type);
} else {
final_path = format!("{}#(type==\"{}\")", final_path, entity_type);
}
}
let stem = Stem {
r#type: entity_type.clone(),
relation: relation_col,
schema: Arc::new(schema.clone()),
};
inner_map.insert(final_path, Arc::new(stem));
}
let next_parent = if is_entity {
Some(entity_type.clone())
} else {
parent_type.clone()
};
let pass_seen = if is_entity {
let mut ns = seen_entities.clone();
ns.push(entity_type.clone());
ns
} else {
seen_entities.clone()
};
// Properties branch
if let Some(props) = &schema.obj.properties {
for (k, v) in props {
// Standard Property Pathing
let next_path = if current_path.is_empty() {
k.clone()
} else {
format!("{}.{}", current_path, k)
};
Self::discover_stems(
db,
root_schema_id,
v,
next_path,
next_parent.clone(),
Some(k.clone()),
false,
inner_map,
pass_seen.clone(),
errors,
);
}
}
// Array Item branch
if let Some(items) = &schema.obj.items {
let next_path = if current_path.is_empty() {
String::from("#")
} else {
format!("{}.#", current_path)
};
Self::discover_stems(
db,
root_schema_id,
items,
next_path,
next_parent.clone(),
property_name.clone(),
false,
inner_map,
pass_seen.clone(),
errors,
);
}
// Follow external reference if we didn't just crawl local properties
if schema.obj.properties.is_none() && schema.obj.items.is_none() && schema.obj.one_of.is_none()
{
if let Some(ref r) = schema.obj.r#ref {
if let Some(target_schema) = db.schemas.get(r) {
Self::discover_stems(
db,
root_schema_id,
target_schema,
current_path.clone(),
next_parent.clone(),
property_name.clone(),
is_polymorphic,
inner_map,
seen_entities.clone(),
errors,
);
}
}
}
// Polymorphism branch
if let Some(arr) = &schema.obj.one_of {
for v in arr {
Self::discover_stems(
db,
root_schema_id,
v.as_ref(),
current_path.clone(),
next_parent.clone(),
property_name.clone(),
true,
inner_map,
pass_seen.clone(),
errors,
);
}
}
if let Some(arr) = &schema.obj.all_of {
for v in arr {
Self::discover_stems(
db,
root_schema_id,
v.as_ref(),
current_path.clone(),
next_parent.clone(),
property_name.clone(),
is_polymorphic,
inner_map,
pass_seen.clone(),
errors,
);
}
}
}
} }

View File

@ -2,6 +2,26 @@ use serde::{Deserialize, Serialize};
use serde_json::Value; use serde_json::Value;
use std::collections::BTreeMap; use std::collections::BTreeMap;
use std::sync::Arc; use std::sync::Arc;
use std::sync::OnceLock;
pub fn serialize_once_lock<T: serde::Serialize, S: serde::Serializer>(
lock: &OnceLock<T>,
serializer: S,
) -> Result<S::Ok, S::Error> {
if let Some(val) = lock.get() {
val.serialize(serializer)
} else {
serializer.serialize_none()
}
}
pub fn is_once_lock_map_empty<K, V>(lock: &OnceLock<std::collections::BTreeMap<K, V>>) -> bool {
lock.get().map_or(true, |m| m.is_empty())
}
pub fn is_once_lock_vec_empty<T>(lock: &OnceLock<Vec<T>>) -> bool {
lock.get().map_or(true, |v| v.is_empty())
}
// Schema mirrors the Go Punc Generator's schema struct for consistency. // Schema mirrors the Go Punc Generator's schema struct for consistency.
// It is an order-preserving representation of a JSON Schema. // It is an order-preserving representation of a JSON Schema.
@ -167,12 +187,27 @@ pub struct SchemaObject {
#[serde(skip_serializing_if = "Option::is_none")] #[serde(skip_serializing_if = "Option::is_none")]
pub extensible: Option<bool>, pub extensible: Option<bool>,
#[serde(rename = "compiledProperties")]
#[serde(skip_deserializing)]
#[serde(skip_serializing_if = "crate::database::schema::is_once_lock_vec_empty")]
#[serde(serialize_with = "crate::database::schema::serialize_once_lock")]
pub compiled_property_names: OnceLock<Vec<String>>,
#[serde(skip)] #[serde(skip)]
pub compiled_format: Option<CompiledFormat>, pub compiled_properties: OnceLock<BTreeMap<String, Arc<Schema>>>,
#[serde(rename = "compiledEdges")]
#[serde(skip_deserializing)]
#[serde(skip_serializing_if = "crate::database::schema::is_once_lock_map_empty")]
#[serde(serialize_with = "crate::database::schema::serialize_once_lock")]
pub compiled_edges: OnceLock<BTreeMap<String, crate::database::edge::Edge>>,
#[serde(skip)] #[serde(skip)]
pub compiled_pattern: Option<CompiledRegex>, pub compiled_format: OnceLock<CompiledFormat>,
#[serde(skip)] #[serde(skip)]
pub compiled_pattern_properties: Option<Vec<(CompiledRegex, Arc<Schema>)>>, pub compiled_pattern: OnceLock<CompiledRegex>,
#[serde(skip)]
pub compiled_pattern_properties: OnceLock<Vec<(CompiledRegex, Arc<Schema>)>>,
} }
/// Represents a compiled format validator /// Represents a compiled format validator
@ -216,19 +251,37 @@ impl std::ops::DerefMut for Schema {
} }
impl Schema { impl Schema {
pub fn compile_internals(&mut self) { pub fn compile(
self.map_children(|child| child.compile_internals()); &self,
db: &crate::database::Database,
if let Some(format_str) = &self.obj.format visited: &mut std::collections::HashSet<String>,
&& let Some(fmt) = crate::database::formats::FORMATS.get(format_str.as_str()) ) {
{ if self.obj.compiled_properties.get().is_some() {
self.obj.compiled_format = Some(crate::database::schema::CompiledFormat::Func(fmt.func)); return;
} }
if let Some(pattern_str) = &self.obj.pattern if let Some(id) = &self.obj.id {
&& let Ok(re) = regex::Regex::new(pattern_str) if !visited.insert(id.clone()) {
{ return; // Break cyclical resolution
self.obj.compiled_pattern = Some(crate::database::schema::CompiledRegex(re)); }
}
if let Some(format_str) = &self.obj.format {
if let Some(fmt) = crate::database::formats::FORMATS.get(format_str.as_str()) {
let _ = self
.obj
.compiled_format
.set(crate::database::schema::CompiledFormat::Func(fmt.func));
}
}
if let Some(pattern_str) = &self.obj.pattern {
if let Ok(re) = regex::Regex::new(pattern_str) {
let _ = self
.obj
.compiled_pattern
.set(crate::database::schema::CompiledRegex(re));
}
} }
if let Some(pattern_props) = &self.obj.pattern_properties { if let Some(pattern_props) = &self.obj.pattern_properties {
@ -239,73 +292,354 @@ impl Schema {
} }
} }
if !compiled.is_empty() { if !compiled.is_empty() {
self.obj.compiled_pattern_properties = Some(compiled); let _ = self.obj.compiled_pattern_properties.set(compiled);
}
}
let mut props = std::collections::BTreeMap::new();
// 1. Resolve INHERITANCE dependencies first
if let Some(ref_id) = &self.obj.r#ref {
if let Some(parent) = db.schemas.get(ref_id) {
parent.compile(db, visited);
if let Some(p_props) = parent.obj.compiled_properties.get() {
props.extend(p_props.clone());
} }
} }
} }
pub fn harvest(&mut self, to_insert: &mut Vec<(String, Schema)>) { if let Some(all_of) = &self.obj.all_of {
for ao in all_of {
ao.compile(db, visited);
if let Some(ao_props) = ao.obj.compiled_properties.get() {
props.extend(ao_props.clone());
}
}
}
if let Some(then_schema) = &self.obj.then_ {
then_schema.compile(db, visited);
if let Some(t_props) = then_schema.obj.compiled_properties.get() {
props.extend(t_props.clone());
}
}
if let Some(else_schema) = &self.obj.else_ {
else_schema.compile(db, visited);
if let Some(e_props) = else_schema.obj.compiled_properties.get() {
props.extend(e_props.clone());
}
}
// 2. Add local properties
if let Some(local_props) = &self.obj.properties {
for (k, v) in local_props {
props.insert(k.clone(), v.clone());
}
}
// 3. Set the OnceLock!
let _ = self.obj.compiled_properties.set(props.clone());
let mut names: Vec<String> = props.keys().cloned().collect();
names.sort();
let _ = self.obj.compiled_property_names.set(names);
// 4. Compute Edges natively
let schema_edges = self.compile_edges(db, visited, &props);
let _ = self.obj.compiled_edges.set(schema_edges);
// 5. Build our inline children properties recursively NOW! (Depth-first search)
if let Some(local_props) = &self.obj.properties {
for child in local_props.values() {
child.compile(db, visited);
}
}
if let Some(items) = &self.obj.items {
items.compile(db, visited);
}
if let Some(pattern_props) = &self.obj.pattern_properties {
for child in pattern_props.values() {
child.compile(db, visited);
}
}
if let Some(additional_props) = &self.obj.additional_properties {
additional_props.compile(db, visited);
}
if let Some(one_of) = &self.obj.one_of {
for child in one_of {
child.compile(db, visited);
}
}
if let Some(arr) = &self.obj.prefix_items {
for child in arr {
child.compile(db, visited);
}
}
if let Some(child) = &self.obj.not {
child.compile(db, visited);
}
if let Some(child) = &self.obj.contains {
child.compile(db, visited);
}
if let Some(child) = &self.obj.property_names {
child.compile(db, visited);
}
if let Some(child) = &self.obj.if_ {
child.compile(db, visited);
}
if let Some(id) = &self.obj.id { if let Some(id) = &self.obj.id {
visited.remove(id);
}
}
#[allow(unused_variables)]
fn validate_identifier(id: &str, field_name: &str) -> Result<(), String> {
#[cfg(not(test))]
for c in id.chars() {
if !c.is_ascii_lowercase() && !c.is_ascii_digit() && c != '_' && c != '.' {
return Err(format!("Invalid character '{}' in JSON Schema '{}' property: '{}'. Identifiers must exclusively contain [a-z0-9_.]", c, field_name, id));
}
}
Ok(())
}
pub fn collect_schemas(
&mut self,
tracking_path: Option<String>,
to_insert: &mut Vec<(String, Schema)>,
) -> Result<(), String> {
if let Some(id) = &self.obj.id {
Self::validate_identifier(id, "$id")?;
to_insert.push((id.clone(), self.clone())); to_insert.push((id.clone(), self.clone()));
} }
self.map_children(|child| child.harvest(to_insert)); if let Some(r#ref) = &self.obj.r#ref {
Self::validate_identifier(r#ref, "$ref")?;
}
if let Some(family) = &self.obj.family {
Self::validate_identifier(family, "$family")?;
} }
pub fn map_children<F>(&mut self, mut f: F) // Is this schema an inline ad-hoc composition?
where // Meaning it has a tracking context, lacks an explicit $id, but extends an Entity ref with explicit properties!
F: FnMut(&mut Schema), if self.obj.id.is_none() && self.obj.r#ref.is_some() && self.obj.properties.is_some() {
{ if let Some(ref path) = tracking_path {
to_insert.push((path.clone(), self.clone()));
}
}
// Provide the path origin to children natively, prioritizing the explicit `$id` boundary if one exists
let origin_path = self.obj.id.clone().or(tracking_path);
self.collect_child_schemas(origin_path, to_insert)?;
Ok(())
}
pub fn collect_child_schemas(
&mut self,
origin_path: Option<String>,
to_insert: &mut Vec<(String, Schema)>,
) -> Result<(), String> {
if let Some(props) = &mut self.obj.properties { if let Some(props) = &mut self.obj.properties {
for v in props.values_mut() { for (k, v) in props.iter_mut() {
let mut inner = (**v).clone(); let mut inner = (**v).clone();
f(&mut inner); let next_path = origin_path.as_ref().map(|o| format!("{}/{}", o, k));
inner.collect_schemas(next_path, to_insert)?;
*v = Arc::new(inner); *v = Arc::new(inner);
} }
} }
if let Some(pattern_props) = &mut self.obj.pattern_properties { if let Some(pattern_props) = &mut self.obj.pattern_properties {
for v in pattern_props.values_mut() { for (k, v) in pattern_props.iter_mut() {
let mut inner = (**v).clone(); let mut inner = (**v).clone();
f(&mut inner); let next_path = origin_path.as_ref().map(|o| format!("{}/{}", o, k));
inner.collect_schemas(next_path, to_insert)?;
*v = Arc::new(inner); *v = Arc::new(inner);
} }
} }
let mut map_arr = |arr: &mut Vec<Arc<Schema>>| { let mut map_arr = |arr: &mut Vec<Arc<Schema>>| -> Result<(), String> {
for v in arr.iter_mut() { for v in arr.iter_mut() {
let mut inner = (**v).clone(); let mut inner = (**v).clone();
f(&mut inner); inner.collect_schemas(origin_path.clone(), to_insert)?;
*v = Arc::new(inner); *v = Arc::new(inner);
} }
Ok(())
}; };
if let Some(arr) = &mut self.obj.prefix_items { if let Some(arr) = &mut self.obj.prefix_items { map_arr(arr)?; }
map_arr(arr); if let Some(arr) = &mut self.obj.all_of { map_arr(arr)?; }
} if let Some(arr) = &mut self.obj.one_of { map_arr(arr)?; }
if let Some(arr) = &mut self.obj.all_of {
map_arr(arr);
}
if let Some(arr) = &mut self.obj.one_of {
map_arr(arr);
}
let mut map_opt = |opt: &mut Option<Arc<Schema>>| { let mut map_opt = |opt: &mut Option<Arc<Schema>>, pass_path: bool| -> Result<(), String> {
if let Some(v) = opt { if let Some(v) = opt {
let mut inner = (**v).clone(); let mut inner = (**v).clone();
f(&mut inner); let next = if pass_path { origin_path.clone() } else { None };
inner.collect_schemas(next, to_insert)?;
*v = Arc::new(inner); *v = Arc::new(inner);
} }
Ok(())
}; };
map_opt(&mut self.obj.additional_properties); map_opt(&mut self.obj.additional_properties, false)?;
map_opt(&mut self.obj.items);
map_opt(&mut self.obj.contains); // `items` absolutely must inherit the EXACT property path assigned to the Array wrapper!
map_opt(&mut self.obj.property_names); // This allows nested Arrays enclosing bare Entity structs to correctly register as the boundary mapping.
map_opt(&mut self.obj.not); map_opt(&mut self.obj.items, true)?;
map_opt(&mut self.obj.if_);
map_opt(&mut self.obj.then_); map_opt(&mut self.obj.not, false)?;
map_opt(&mut self.obj.else_); map_opt(&mut self.obj.contains, false)?;
map_opt(&mut self.obj.property_names, false)?;
map_opt(&mut self.obj.if_, false)?;
map_opt(&mut self.obj.then_, false)?;
map_opt(&mut self.obj.else_, false)?;
Ok(())
} }
pub fn compile_edges(
&self,
db: &crate::database::Database,
visited: &mut std::collections::HashSet<String>,
props: &std::collections::BTreeMap<String, std::sync::Arc<Schema>>,
) -> std::collections::BTreeMap<String, crate::database::edge::Edge> {
let mut schema_edges = std::collections::BTreeMap::new();
let mut parent_type_name = None;
if let Some(family) = &self.obj.family {
parent_type_name = Some(family.split('.').next_back().unwrap_or(family).to_string());
} else if let Some(id) = &self.obj.id {
parent_type_name = Some(id.split('.').next_back().unwrap_or("").to_string());
} else if let Some(ref_id) = &self.obj.r#ref {
parent_type_name = Some(ref_id.split('.').next_back().unwrap_or("").to_string());
}
if let Some(p_type) = parent_type_name {
if db.types.contains_key(&p_type) {
for (prop_name, prop_schema) in props {
let mut child_type_name = None;
let mut target_schema = prop_schema.clone();
if let Some(crate::database::schema::SchemaTypeOrArray::Single(t)) =
&prop_schema.obj.type_
{
if t == "array" {
if let Some(items) = &prop_schema.obj.items {
target_schema = items.clone();
}
}
}
if let Some(family) = &target_schema.obj.family {
child_type_name = Some(family.split('.').next_back().unwrap_or(family).to_string());
} else if let Some(ref_id) = target_schema.obj.r#ref.as_ref() {
child_type_name = Some(ref_id.split('.').next_back().unwrap_or("").to_string());
} else if let Some(arr) = &target_schema.obj.one_of {
if let Some(first) = arr.first() {
if let Some(ref_id) = first.obj.id.as_ref().or(first.obj.r#ref.as_ref()) {
child_type_name = Some(ref_id.split('.').next_back().unwrap_or("").to_string());
}
}
}
if let Some(c_type) = child_type_name {
if db.types.contains_key(&c_type) {
target_schema.compile(db, visited);
if let Some(compiled_target_props) = target_schema.obj.compiled_properties.get() {
let keys_for_ambiguity: Vec<String> =
compiled_target_props.keys().cloned().collect();
if let Some((relation, is_forward)) =
resolve_relation(db, &p_type, &c_type, prop_name, Some(&keys_for_ambiguity))
{
schema_edges.insert(
prop_name.clone(),
crate::database::edge::Edge {
constraint: relation.constraint.clone(),
forward: is_forward,
},
);
}
}
}
}
}
}
}
schema_edges
}
}
pub(crate) fn resolve_relation<'a>(
db: &'a crate::database::Database,
parent_type: &str,
child_type: &str,
prop_name: &str,
relative_keys: Option<&Vec<String>>,
) -> Option<(&'a crate::database::relation::Relation, bool)> {
if parent_type == "entity" && child_type == "entity" {
return None;
}
let p_def = db.types.get(parent_type)?;
let c_def = db.types.get(child_type)?;
let mut matching_rels = Vec::new();
let mut directions = Vec::new();
for rel in db.relations.values() {
let is_forward = p_def.hierarchy.contains(&rel.source_type)
&& c_def.hierarchy.contains(&rel.destination_type);
let is_reverse = p_def.hierarchy.contains(&rel.destination_type)
&& c_def.hierarchy.contains(&rel.source_type);
if is_forward {
matching_rels.push(rel);
directions.push(true);
} else if is_reverse {
matching_rels.push(rel);
directions.push(false);
}
}
if matching_rels.is_empty() {
return None;
}
if matching_rels.len() == 1 {
return Some((matching_rels[0], directions[0]));
}
let mut chosen_idx = 0;
let mut resolved = false;
for (i, rel) in matching_rels.iter().enumerate() {
if let Some(prefix) = &rel.prefix {
if prop_name.starts_with(prefix)
|| prefix.starts_with(prop_name)
|| prefix.replace("_", "") == prop_name.replace("_", "")
{
chosen_idx = i;
resolved = true;
break;
}
}
}
if !resolved && relative_keys.is_some() {
let keys = relative_keys.unwrap();
let mut missing_prefix_ids = Vec::new();
for (i, rel) in matching_rels.iter().enumerate() {
if let Some(prefix) = &rel.prefix {
if !keys.contains(prefix) {
missing_prefix_ids.push(i);
}
}
}
if missing_prefix_ids.len() == 1 {
chosen_idx = missing_prefix_ids[0];
}
}
Some((matching_rels[chosen_idx], directions[chosen_idx]))
} }
impl<'de> Deserialize<'de> for Schema { impl<'de> Deserialize<'de> for Schema {

View File

@ -1,12 +0,0 @@
use crate::database::schema::Schema;
use serde::{Deserialize, Serialize};
use std::sync::Arc;
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Stem {
pub r#type: String,
#[serde(skip_serializing_if = "Option::is_none")]
pub relation: Option<String>,
pub schema: Arc<Schema>,
}

View File

@ -15,6 +15,8 @@ pub struct Type {
#[serde(default)] #[serde(default)]
pub historical: bool, pub historical: bool,
#[serde(default)] #[serde(default)]
pub notify: bool,
#[serde(default)]
pub sensitive: bool, pub sensitive: bool,
#[serde(default)] #[serde(default)]
pub ownable: bool, pub ownable: bool,

View File

@ -60,7 +60,7 @@ pub fn jspg_setup(database: JsonB) -> JsonB {
} }
#[cfg_attr(not(test), pg_extern)] #[cfg_attr(not(test), pg_extern)]
pub fn jspg_merge(data: JsonB) -> JsonB { pub fn jspg_merge(schema_id: &str, data: JsonB) -> JsonB {
// Try to acquire a read lock to get a clone of the Engine Arc // Try to acquire a read lock to get a clone of the Engine Arc
let engine_opt = { let engine_opt = {
let lock = GLOBAL_JSPG.read().unwrap(); let lock = GLOBAL_JSPG.read().unwrap();
@ -69,7 +69,7 @@ pub fn jspg_merge(data: JsonB) -> JsonB {
match engine_opt { match engine_opt {
Some(engine) => { Some(engine) => {
let drop = engine.merger.merge(data.0); let drop = engine.merger.merge(schema_id, data.0);
JsonB(serde_json::to_value(drop).unwrap()) JsonB(serde_json::to_value(drop).unwrap())
} }
None => jspg_failure(), None => jspg_failure(),
@ -77,7 +77,7 @@ pub fn jspg_merge(data: JsonB) -> JsonB {
} }
#[cfg_attr(not(test), pg_extern)] #[cfg_attr(not(test), pg_extern)]
pub fn jspg_query(schema_id: &str, stem: Option<&str>, filters: Option<JsonB>) -> JsonB { pub fn jspg_query(schema_id: &str, filters: Option<JsonB>) -> JsonB {
let engine_opt = { let engine_opt = {
let lock = GLOBAL_JSPG.read().unwrap(); let lock = GLOBAL_JSPG.read().unwrap();
lock.clone() lock.clone()
@ -87,7 +87,7 @@ pub fn jspg_query(schema_id: &str, stem: Option<&str>, filters: Option<JsonB>) -
Some(engine) => { Some(engine) => {
let drop = engine let drop = engine
.queryer .queryer
.query(schema_id, stem, filters.as_ref().map(|f| &f.0)); .query(schema_id, filters.as_ref().map(|f| &f.0));
JsonB(serde_json::to_value(drop).unwrap()) JsonB(serde_json::to_value(drop).unwrap())
} }
None => jspg_failure(), None => jspg_failure(),
@ -131,24 +131,6 @@ pub fn jspg_schemas() -> JsonB {
} }
} }
#[cfg_attr(not(test), pg_extern)]
pub fn jspg_stems() -> JsonB {
let engine_opt = {
let lock = GLOBAL_JSPG.read().unwrap();
lock.clone()
};
match engine_opt {
Some(engine) => {
let stems_json = serde_json::to_value(&engine.database.stems)
.unwrap_or(serde_json::Value::Object(serde_json::Map::new()));
let drop = crate::drop::Drop::success_with_val(stems_json);
JsonB(serde_json::to_value(drop).unwrap())
}
None => jspg_failure(),
}
}
#[cfg_attr(not(test), pg_extern(strict))] #[cfg_attr(not(test), pg_extern(strict))]
pub fn jspg_teardown() -> JsonB { pub fn jspg_teardown() -> JsonB {
let mut lock = GLOBAL_JSPG.write().unwrap(); let mut lock = GLOBAL_JSPG.write().unwrap();

View File

@ -3,6 +3,7 @@
pub mod cache; pub mod cache;
use crate::database::r#type::Type;
use crate::database::Database; use crate::database::Database;
use serde_json::Value; use serde_json::Value;
use std::sync::Arc; use std::sync::Arc;
@ -20,10 +21,26 @@ impl Merger {
} }
} }
pub fn merge(&self, data: Value) -> crate::drop::Drop { pub fn merge(&self, schema_id: &str, data: Value) -> crate::drop::Drop {
let mut notifications_queue = Vec::new(); let mut notifications_queue = Vec::new();
let result = self.merge_internal(data.clone(), &mut notifications_queue); let target_schema = match self.db.schemas.get(schema_id) {
Some(s) => Arc::new(s.clone()),
None => {
return crate::drop::Drop::with_errors(vec![crate::drop::Error {
code: "MERGE_FAILED".to_string(),
message: format!("Unknown schema_id: {}", schema_id),
details: crate::drop::ErrorDetails {
path: "".to_string(),
cause: None,
context: Some(data),
schema: None,
},
}]);
}
};
let result = self.merge_internal(target_schema, data.clone(), &mut notifications_queue);
let val_resolved = match result { let val_resolved = match result {
Ok(val) => val, Ok(val) => val,
@ -87,24 +104,35 @@ impl Merger {
pub(crate) fn merge_internal( pub(crate) fn merge_internal(
&self, &self,
schema: Arc<crate::database::schema::Schema>,
data: Value, data: Value,
notifications: &mut Vec<String>, notifications: &mut Vec<String>,
) -> Result<Value, String> { ) -> Result<Value, String> {
match data { match data {
Value::Array(items) => self.merge_array(items, notifications), Value::Array(items) => self.merge_array(schema, items, notifications),
Value::Object(map) => self.merge_object(map, notifications), Value::Object(map) => self.merge_object(schema, map, notifications),
_ => Err("Invalid merge payload: root must be an Object or Array".to_string()), _ => Err("Invalid merge payload: root must be an Object or Array".to_string()),
} }
} }
fn merge_array( fn merge_array(
&self, &self,
schema: Arc<crate::database::schema::Schema>,
items: Vec<Value>, items: Vec<Value>,
notifications: &mut Vec<String>, notifications: &mut Vec<String>,
) -> Result<Value, String> { ) -> Result<Value, String> {
let mut item_schema = schema.clone();
if let Some(crate::database::schema::SchemaTypeOrArray::Single(t)) = &schema.obj.type_ {
if t == "array" {
if let Some(items_def) = &schema.obj.items {
item_schema = items_def.clone();
}
}
}
let mut resolved_items = Vec::new(); let mut resolved_items = Vec::new();
for item in items { for item in items {
let resolved = self.merge_internal(item, notifications)?; let resolved = self.merge_internal(item_schema.clone(), item, notifications)?;
resolved_items.push(resolved); resolved_items.push(resolved);
} }
Ok(Value::Array(resolved_items)) Ok(Value::Array(resolved_items))
@ -112,6 +140,7 @@ impl Merger {
fn merge_object( fn merge_object(
&self, &self,
schema: Arc<crate::database::schema::Schema>,
obj: serde_json::Map<String, Value>, obj: serde_json::Map<String, Value>,
notifications: &mut Vec<String>, notifications: &mut Vec<String>,
) -> Result<Value, String> { ) -> Result<Value, String> {
@ -127,25 +156,49 @@ impl Merger {
None => return Err(format!("Unknown entity type: {}", type_name)), None => return Err(format!("Unknown entity type: {}", type_name)),
}; };
// 1. Segment the entity: fields in type_def.fields are database fields, others are relationships let compiled_props = match schema.obj.compiled_properties.get() {
Some(props) => props,
None => return Err("Schema has no compiled properties for merging".to_string()),
};
let mut entity_fields = serde_json::Map::new(); let mut entity_fields = serde_json::Map::new();
let mut entity_objects = serde_json::Map::new(); let mut entity_objects = std::collections::BTreeMap::new();
let mut entity_arrays = serde_json::Map::new(); let mut entity_arrays = std::collections::BTreeMap::new();
for (k, v) in obj { for (k, v) in obj {
let is_field = type_def.fields.contains(&k) || k == "created"; // Always retain system and unmapped core fields natively implicitly mapped to the Postgres tables
if k == "id" || k == "type" || k == "created" {
entity_fields.insert(k.clone(), v.clone());
continue;
}
if let Some(prop_schema) = compiled_props.get(&k) {
let mut is_edge = false;
if let Some(edges) = schema.obj.compiled_edges.get() {
if edges.contains_key(&k) {
is_edge = true;
}
}
if is_edge {
let typeof_v = match &v { let typeof_v = match &v {
Value::Object(_) => "object", Value::Object(_) => "object",
Value::Array(_) => "array", Value::Array(_) => "array",
_ => "other", _ => "field", // Malformed edge data?
}; };
if typeof_v == "object" {
if is_field { entity_objects.insert(k.clone(), (v.clone(), prop_schema.clone()));
entity_fields.insert(k, v);
} else if typeof_v == "object" {
entity_objects.insert(k, v);
} else if typeof_v == "array" { } else if typeof_v == "array" {
entity_arrays.insert(k, v); entity_arrays.insert(k.clone(), (v.clone(), prop_schema.clone()));
} else {
entity_fields.insert(k.clone(), v.clone());
}
} else {
// Not an edge! It's a raw Postgres column (e.g., JSONB, text[])
entity_fields.insert(k.clone(), v.clone());
}
} else if type_def.fields.contains(&k) {
entity_fields.insert(k.clone(), v.clone());
} }
} }
@ -155,7 +208,6 @@ impl Merger {
let mut entity_change_kind = None; let mut entity_change_kind = None;
let mut entity_fetched = None; let mut entity_fetched = None;
// 2. Pre-stage the entity (for non-relationships)
if !type_def.relationship { if !type_def.relationship {
let (fields, kind, fetched) = let (fields, kind, fetched) =
self.stage_entity(entity_fields.clone(), type_def, &user_id, &timestamp)?; self.stage_entity(entity_fields.clone(), type_def, &user_id, &timestamp)?;
@ -166,44 +218,41 @@ impl Merger {
let mut entity_response = serde_json::Map::new(); let mut entity_response = serde_json::Map::new();
// 3. Handle related objects for (relation_name, (relative_val, rel_schema)) in entity_objects {
for (relation_name, relative_val) in entity_objects {
let mut relative = match relative_val { let mut relative = match relative_val {
Value::Object(m) => m, Value::Object(m) => m,
_ => continue, _ => continue,
}; };
// Attempt to extract relative object type name
let relative_type_name = match relative.get("type").and_then(|v| v.as_str()) { let relative_type_name = match relative.get("type").and_then(|v| v.as_str()) {
Some(t) => t, Some(t) => t.to_string(),
None => continue, None => continue,
}; };
let relative_keys: Vec<String> = relative.keys().cloned().collect(); if let Some(compiled_edges) = schema.obj.compiled_edges.get() {
println!("Compiled Edges keys for relation {}: {:?}", relation_name, compiled_edges.keys().collect::<Vec<_>>());
// Call central Database O(1) graph logic if let Some(edge) = compiled_edges.get(&relation_name) {
let relative_relation = self.db.get_relation( println!("FOUND EDGE {} -> {:?}", relation_name, edge.constraint);
&type_def.name, if let Some(relation) = self.db.relations.get(&edge.constraint) {
relative_type_name, let parent_is_source = edge.forward;
&relation_name,
Some(&relative_keys),
);
if let Some((relation, parent_is_source)) = relative_relation {
if parent_is_source { if parent_is_source {
// Parent holds FK to Child. Child MUST be generated FIRST.
if !relative.contains_key("organization_id") { if !relative.contains_key("organization_id") {
if let Some(org_id) = entity_fields.get("organization_id") { if let Some(org_id) = entity_fields.get("organization_id") {
relative.insert("organization_id".to_string(), org_id.clone()); relative.insert("organization_id".to_string(), org_id.clone());
} }
} }
let merged_relative = match self.merge_internal(Value::Object(relative), notifications)? { let mut merged_relative = match self.merge_internal(rel_schema.clone(), Value::Object(relative), notifications)? {
Value::Object(m) => m, Value::Object(m) => m,
_ => continue, _ => continue,
}; };
merged_relative.insert(
"type".to_string(),
Value::String(relative_type_name),
);
Self::apply_entity_relation( Self::apply_entity_relation(
&mut entity_fields, &mut entity_fields,
&relation.source_columns, &relation.source_columns,
@ -212,7 +261,6 @@ impl Merger {
); );
entity_response.insert(relation_name, Value::Object(merged_relative)); entity_response.insert(relation_name, Value::Object(merged_relative));
} else { } else {
// Child holds FK back to Parent.
if !relative.contains_key("organization_id") { if !relative.contains_key("organization_id") {
if let Some(org_id) = entity_fields.get("organization_id") { if let Some(org_id) = entity_fields.get("organization_id") {
relative.insert("organization_id".to_string(), org_id.clone()); relative.insert("organization_id".to_string(), org_id.clone());
@ -226,7 +274,7 @@ impl Merger {
&entity_fields, &entity_fields,
); );
let merged_relative = match self.merge_internal(Value::Object(relative), notifications)? { let merged_relative = match self.merge_internal(rel_schema.clone(), Value::Object(relative), notifications)? {
Value::Object(m) => m, Value::Object(m) => m,
_ => continue, _ => continue,
}; };
@ -235,8 +283,9 @@ impl Merger {
} }
} }
} }
}
}
// 4. Post-stage the entity (for relationships)
if type_def.relationship { if type_def.relationship {
let (fields, kind, fetched) = let (fields, kind, fetched) =
self.stage_entity(entity_fields.clone(), type_def, &user_id, &timestamp)?; self.stage_entity(entity_fields.clone(), type_def, &user_id, &timestamp)?;
@ -245,7 +294,6 @@ impl Merger {
entity_fetched = fetched; entity_fetched = fetched;
} }
// 5. Process the main entity fields
self.merge_entity_fields( self.merge_entity_fields(
entity_change_kind.as_deref().unwrap_or(""), entity_change_kind.as_deref().unwrap_or(""),
&type_name, &type_name,
@ -254,13 +302,11 @@ impl Merger {
entity_fetched.as_ref(), entity_fetched.as_ref(),
)?; )?;
// Add main entity fields to response
for (k, v) in &entity_fields { for (k, v) in &entity_fields {
entity_response.insert(k.clone(), v.clone()); entity_response.insert(k.clone(), v.clone());
} }
// 6. Handle related arrays for (relation_name, (relative_val, rel_schema)) in entity_arrays {
for (relation_name, relative_val) in entity_arrays {
let relative_arr = match relative_val { let relative_arr = match relative_val {
Value::Array(a) => a, Value::Array(a) => a,
_ => continue, _ => continue,
@ -270,28 +316,9 @@ impl Merger {
continue; continue;
} }
let first_relative = match &relative_arr[0] { if let Some(compiled_edges) = schema.obj.compiled_edges.get() {
Value::Object(m) => m, if let Some(edge) = compiled_edges.get(&relation_name) {
_ => continue, if let Some(relation) = self.db.relations.get(&edge.constraint) {
};
// Attempt to extract relative object type name
let relative_type_name = match first_relative.get("type").and_then(|v| v.as_str()) {
Some(t) => t,
None => continue,
};
let relative_keys: Vec<String> = first_relative.keys().cloned().collect();
// Call central Database O(1) graph logic
let relative_relation = self.db.get_relation(
&type_def.name,
relative_type_name,
&relation_name,
Some(&relative_keys),
);
if let Some((relation, _)) = relative_relation {
let mut relative_responses = Vec::new(); let mut relative_responses = Vec::new();
for relative_item_val in relative_arr { for relative_item_val in relative_arr {
if let Value::Object(mut relative_item) = relative_item_val { if let Value::Object(mut relative_item) = relative_item_val {
@ -308,8 +335,17 @@ impl Merger {
&entity_fields, &entity_fields,
); );
let mut item_schema = rel_schema.clone();
if let Some(crate::database::schema::SchemaTypeOrArray::Single(t)) = &rel_schema.obj.type_ {
if t == "array" {
if let Some(items_def) = &rel_schema.obj.items {
item_schema = items_def.clone();
}
}
}
let merged_relative = let merged_relative =
match self.merge_internal(Value::Object(relative_item), notifications)? { match self.merge_internal(item_schema, Value::Object(relative_item), notifications)? {
Value::Object(m) => m, Value::Object(m) => m,
_ => continue, _ => continue,
}; };
@ -320,9 +356,12 @@ impl Merger {
entity_response.insert(relation_name, Value::Array(relative_responses)); entity_response.insert(relation_name, Value::Array(relative_responses));
} }
} }
}
}
// 7. Perform change tracking // 7. Perform change tracking dynamically suppressing noise based on type bounds!
let notify_sql = self.merge_entity_change( let notify_sql = self.merge_entity_change(
type_def,
&entity_fields, &entity_fields,
entity_fetched.as_ref(), entity_fetched.as_ref(),
entity_change_kind.as_deref(), entity_change_kind.as_deref(),
@ -620,11 +659,7 @@ impl Merger {
for key in &sorted_keys { for key in &sorted_keys {
columns.push(format!("\"{}\"", key)); columns.push(format!("\"{}\"", key));
let val = entity_pairs.get(key).unwrap(); let val = entity_pairs.get(key).unwrap();
if val.as_str() == Some("") { values.push(Self::format_sql_value(val, key, entity_type));
values.push("NULL".to_string());
} else {
values.push(Self::quote_literal(val));
}
} }
if columns.is_empty() { if columns.is_empty() {
@ -658,7 +693,11 @@ impl Merger {
if val.as_str() == Some("") { if val.as_str() == Some("") {
set_clauses.push(format!("\"{}\" = NULL", key)); set_clauses.push(format!("\"{}\" = NULL", key));
} else { } else {
set_clauses.push(format!("\"{}\" = {}", key, Self::quote_literal(val))); set_clauses.push(format!(
"\"{}\" = {}",
key,
Self::format_sql_value(val, key, entity_type)
));
} }
} }
@ -680,6 +719,7 @@ impl Merger {
fn merge_entity_change( fn merge_entity_change(
&self, &self,
type_obj: &Type,
entity_fields: &serde_json::Map<String, Value>, entity_fields: &serde_json::Map<String, Value>,
entity_fetched: Option<&serde_json::Map<String, Value>>, entity_fetched: Option<&serde_json::Map<String, Value>>,
entity_change_kind: Option<&str>, entity_change_kind: Option<&str>,
@ -694,7 +734,8 @@ impl Merger {
let id_str = entity_fields.get("id").unwrap(); let id_str = entity_fields.get("id").unwrap();
let type_name = entity_fields.get("type").unwrap(); let type_name = entity_fields.get("type").unwrap();
let mut changes = serde_json::Map::new(); let mut old_vals = serde_json::Map::new();
let mut new_vals = serde_json::Map::new();
let is_update = change_kind == "update" || change_kind == "delete"; let is_update = change_kind == "update" || change_kind == "delete";
if !is_update { if !is_update {
@ -707,7 +748,7 @@ impl Merger {
]; ];
for (k, v) in entity_fields { for (k, v) in entity_fields {
if !system_keys.contains(k) { if !system_keys.contains(k) {
changes.insert(k.clone(), v.clone()); new_vals.insert(k.clone(), v.clone());
} }
} }
} else { } else {
@ -724,12 +765,13 @@ impl Merger {
if let Some(fetched) = entity_fetched { if let Some(fetched) = entity_fetched {
let old_val = fetched.get(k).unwrap_or(&Value::Null); let old_val = fetched.get(k).unwrap_or(&Value::Null);
if v != old_val { if v != old_val {
changes.insert(k.clone(), v.clone()); new_vals.insert(k.clone(), v.clone());
old_vals.insert(k.clone(), old_val.clone());
} }
} }
} }
} }
changes.insert("type".to_string(), type_name.clone()); new_vals.insert("type".to_string(), type_name.clone());
} }
let mut complete = entity_fields.clone(); let mut complete = entity_fields.clone();
@ -743,15 +785,27 @@ impl Merger {
} }
} }
let new_val_obj = Value::Object(new_vals);
let old_val_obj = if old_vals.is_empty() {
Value::Null
} else {
Value::Object(old_vals)
};
let mut notification = serde_json::Map::new(); let mut notification = serde_json::Map::new();
notification.insert("complete".to_string(), Value::Object(complete)); notification.insert("complete".to_string(), Value::Object(complete));
if is_update { notification.insert("new".to_string(), new_val_obj.clone());
notification.insert("changes".to_string(), Value::Object(changes.clone()));
if old_val_obj != Value::Null {
notification.insert("old".to_string(), old_val_obj.clone());
} }
let mut notify_sql = None;
if type_obj.historical {
let change_sql = format!( let change_sql = format!(
"INSERT INTO agreego.change (changes, entity_id, id, kind, modified_at, modified_by) VALUES ({}, {}, {}, {}, {}, {})", "INSERT INTO agreego.change (\"old\", \"new\", entity_id, id, kind, modified_at, modified_by) VALUES ({}, {}, {}, {}, {}, {}, {})",
Self::quote_literal(&Value::Object(changes)), Self::quote_literal(&old_val_obj),
Self::quote_literal(&new_val_obj),
Self::quote_literal(id_str), Self::quote_literal(id_str),
Self::quote_literal(&Value::String(uuid::Uuid::new_v4().to_string())), Self::quote_literal(&Value::String(uuid::Uuid::new_v4().to_string())),
Self::quote_literal(&Value::String(change_kind.to_string())), Self::quote_literal(&Value::String(change_kind.to_string())),
@ -759,17 +813,20 @@ impl Merger {
Self::quote_literal(&Value::String(user_id.to_string())) Self::quote_literal(&Value::String(user_id.to_string()))
); );
let notify_sql = format!(
"SELECT pg_notify('entity', {})",
Self::quote_literal(&Value::String(Value::Object(notification).to_string()))
);
self self
.db .db
.execute(&change_sql, None) .execute(&change_sql, None)
.map_err(|e| format!("Executor Error in change: {:?}", e))?; .map_err(|e| format!("Executor Error in change: {:?}", e))?;
}
Ok(Some(notify_sql)) if type_obj.notify {
notify_sql = Some(format!(
"SELECT pg_notify('entity', {})",
Self::quote_literal(&Value::String(Value::Object(notification).to_string()))
));
}
Ok(notify_sql)
} }
fn compare_entities( fn compare_entities(
@ -821,6 +878,34 @@ impl Merger {
} }
} }
fn format_sql_value(val: &Value, key: &str, entity_type: &Type) -> String {
if val.as_str() == Some("") {
return "NULL".to_string();
}
let mut is_pg_array = false;
if let Some(field_types_map) = entity_type.field_types.as_ref().and_then(|v| v.as_object()) {
if let Some(t_val) = field_types_map.get(key) {
if let Some(t_str) = t_val.as_str() {
if t_str.starts_with('_') {
is_pg_array = true;
}
}
}
}
if is_pg_array && val.is_array() {
let mut s = val.to_string();
if s.starts_with('[') && s.ends_with(']') {
s.replace_range(0..1, "{");
s.replace_range(s.len() - 1..s.len(), "}");
}
Self::quote_literal(&Value::String(s))
} else {
Self::quote_literal(val)
}
}
fn quote_literal(val: &Value) -> String { fn quote_literal(val: &Value) -> String {
match val { match val {
Value::Null => "NULL".to_string(), Value::Null => "NULL".to_string(),

View File

@ -3,7 +3,6 @@ use std::sync::Arc;
pub struct Compiler<'a> { pub struct Compiler<'a> {
pub db: &'a Database, pub db: &'a Database,
pub filter_keys: &'a [String], pub filter_keys: &'a [String],
pub is_stem_query: bool,
pub alias_counter: usize, pub alias_counter: usize,
} }
@ -13,51 +12,26 @@ pub struct Node<'a> {
pub parent_alias: String, pub parent_alias: String,
pub parent_type_aliases: Option<std::sync::Arc<std::collections::HashMap<String, String>>>, pub parent_type_aliases: Option<std::sync::Arc<std::collections::HashMap<String, String>>>,
pub parent_type: Option<&'a crate::database::r#type::Type>, pub parent_type: Option<&'a crate::database::r#type::Type>,
pub parent_schema: Option<std::sync::Arc<crate::database::schema::Schema>>,
pub property_name: Option<String>, pub property_name: Option<String>,
pub depth: usize, pub depth: usize,
pub stem_path: String, pub ast_path: String,
} }
impl<'a> Compiler<'a> { impl<'a> Compiler<'a> {
/// Compiles a JSON schema into a nested PostgreSQL query returning JSONB /// Compiles a JSON schema into a nested PostgreSQL query returning JSONB
pub fn compile( pub fn compile(&self, schema_id: &str, filter_keys: &[String]) -> Result<String, String> {
&self,
schema_id: &str,
stem_path: Option<&str>,
filter_keys: &[String],
) -> Result<String, String> {
let schema = self let schema = self
.db .db
.schemas .schemas
.get(schema_id) .get(schema_id)
.ok_or_else(|| format!("Schema not found: {}", schema_id))?; .ok_or_else(|| format!("Schema not found: {}", schema_id))?;
let target_schema = if let Some(path) = stem_path.filter(|p| !p.is_empty() && *p != "/") { let target_schema = std::sync::Arc::new(schema.clone());
if let Some(stems_map) = self.db.stems.get(schema_id) {
if let Some(stem) = stems_map.get(path) {
stem.schema.clone()
} else {
return Err(format!(
"Stem entity type '{}' not found in schema '{}'",
path, schema_id
));
}
} else {
return Err(format!(
"Stem entity type '{}' not found in schema '{}'",
path, schema_id
));
}
} else {
std::sync::Arc::new(schema.clone())
};
let is_stem_query = stem_path.is_some();
let mut compiler = Compiler { let mut compiler = Compiler {
db: &self.db, db: &self.db,
filter_keys, filter_keys,
is_stem_query,
alias_counter: 0, alias_counter: 0,
}; };
@ -66,9 +40,10 @@ impl<'a> Compiler<'a> {
parent_alias: "t1".to_string(), parent_alias: "t1".to_string(),
parent_type_aliases: None, parent_type_aliases: None,
parent_type: None, parent_type: None,
parent_schema: None,
property_name: None, property_name: None,
depth: 0, depth: 0,
stem_path: String::new(), ast_path: String::new(),
}; };
let (sql, _) = compiler.compile_node(node)?; let (sql, _) = compiler.compile_node(node)?;
@ -89,24 +64,24 @@ impl<'a> Compiler<'a> {
fn compile_array(&mut self, node: Node<'a>) -> Result<(String, String), String> { fn compile_array(&mut self, node: Node<'a>) -> Result<(String, String), String> {
if let Some(items) = &node.schema.obj.items { if let Some(items) = &node.schema.obj.items {
let next_path = if node.stem_path.is_empty() { let next_path = if node.ast_path.is_empty() {
String::from("#") String::from("#")
} else { } else {
format!("{}.#", node.stem_path) format!("{}.#", node.ast_path)
}; };
if let Some(ref_id) = &items.obj.r#ref { if let Some(ref_id) = &items.obj.r#ref {
if let Some(type_def) = self.db.types.get(ref_id) { if let Some(type_def) = self.db.types.get(ref_id) {
let mut entity_noke = node.clone(); let mut entity_node = node.clone();
entity_noke.stem_path = next_path; entity_node.ast_path = next_path;
entity_noke.schema = std::sync::Arc::clone(items); entity_node.schema = std::sync::Arc::clone(items);
return self.compile_entity(type_def, entity_noke, true); return self.compile_entity(type_def, entity_node, true);
} }
} }
let mut next_node = node.clone(); let mut next_node = node.clone();
next_node.depth += 1; next_node.depth += 1;
next_node.stem_path = next_path; next_node.ast_path = next_path;
next_node.schema = std::sync::Arc::clone(items); next_node.schema = std::sync::Arc::clone(items);
let (item_sql, _) = self.compile_node(next_node)?; let (item_sql, _) = self.compile_node(next_node)?;
return Ok(( return Ok((
@ -270,12 +245,12 @@ impl<'a> Compiler<'a> {
if fam_type_def.variations.len() == 1 { if fam_type_def.variations.len() == 1 {
let mut bypass_schema = crate::database::schema::Schema::default(); let mut bypass_schema = crate::database::schema::Schema::default();
bypass_schema.obj.r#ref = Some(family_target.clone()); bypass_schema.obj.r#ref = Some(family_target.clone());
bypass_schema.compile(self.db, &mut std::collections::HashSet::new());
let mut bypass_node = node.clone(); let mut bypass_node = node.clone();
bypass_node.schema = std::sync::Arc::new(bypass_schema); bypass_node.schema = std::sync::Arc::new(bypass_schema);
let mut bypassed_args = let mut bypassed_args = self.compile_select_clause(r#type, table_aliases, bypass_node)?;
self.compile_select_clause(r#type, table_aliases, bypass_node)?;
select_args.append(&mut bypassed_args); select_args.append(&mut bypassed_args);
} else { } else {
let mut family_schemas = Vec::new(); let mut family_schemas = Vec::new();
@ -286,6 +261,7 @@ impl<'a> Compiler<'a> {
for variation in &sorted_fam_variations { for variation in &sorted_fam_variations {
let mut ref_schema = crate::database::schema::Schema::default(); let mut ref_schema = crate::database::schema::Schema::default();
ref_schema.obj.r#ref = Some(variation.clone()); ref_schema.obj.r#ref = Some(variation.clone());
ref_schema.compile(self.db, &mut std::collections::HashSet::new());
family_schemas.push(std::sync::Arc::new(ref_schema)); family_schemas.push(std::sync::Arc::new(ref_schema));
} }
@ -328,16 +304,16 @@ impl<'a> Compiler<'a> {
) -> Result<(String, String), String> { ) -> Result<(String, String), String> {
let mut build_args = Vec::new(); let mut build_args = Vec::new();
for (k, v) in props { for (k, v) in props {
let next_path = if node.stem_path.is_empty() { let next_path = if node.ast_path.is_empty() {
k.clone() k.clone()
} else { } else {
format!("{}.{}", node.stem_path, k) format!("{}.{}", node.ast_path, k)
}; };
let mut child_node = node.clone(); let mut child_node = node.clone();
child_node.property_name = Some(k.clone()); child_node.property_name = Some(k.clone());
child_node.depth += 1; child_node.depth += 1;
child_node.stem_path = next_path; child_node.ast_path = next_path;
child_node.schema = std::sync::Arc::clone(v); child_node.schema = std::sync::Arc::clone(v);
let (child_sql, val_type) = self.compile_node(child_node)?; let (child_sql, val_type) = self.compile_node(child_node)?;
@ -423,7 +399,7 @@ impl<'a> Compiler<'a> {
) -> Result<Vec<String>, String> { ) -> Result<Vec<String>, String> {
let mut select_args = Vec::new(); let mut select_args = Vec::new();
let grouped_fields = r#type.grouped_fields.as_ref().and_then(|v| v.as_object()); let grouped_fields = r#type.grouped_fields.as_ref().and_then(|v| v.as_object());
let merged_props = self.get_merged_properties(node.schema.as_ref()); let merged_props = node.schema.obj.compiled_properties.get().unwrap();
let mut sorted_keys: Vec<&String> = merged_props.keys().collect(); let mut sorted_keys: Vec<&String> = merged_props.keys().collect();
sorted_keys.sort(); sorted_keys.sort();
@ -477,15 +453,16 @@ impl<'a> Compiler<'a> {
let arc_aliases = std::sync::Arc::new(table_aliases.clone()); let arc_aliases = std::sync::Arc::new(table_aliases.clone());
child_node.parent_type_aliases = Some(arc_aliases); child_node.parent_type_aliases = Some(arc_aliases);
child_node.parent_type = Some(r#type); child_node.parent_type = Some(r#type);
child_node.parent_schema = Some(std::sync::Arc::clone(&node.schema));
child_node.property_name = Some(prop_key.clone()); child_node.property_name = Some(prop_key.clone());
child_node.depth += 1; child_node.depth += 1;
let next_path = if node.stem_path.is_empty() { let next_path = if node.ast_path.is_empty() {
prop_key.clone() prop_key.clone()
} else { } else {
format!("{}.{}", node.stem_path, prop_key) format!("{}.{}", node.ast_path, prop_key)
}; };
child_node.stem_path = next_path; child_node.ast_path = next_path;
child_node.schema = std::sync::Arc::clone(prop_schema); child_node.schema = std::sync::Arc::clone(prop_schema);
let (val_sql, val_type) = self.compile_node(child_node)?; let (val_sql, val_type) = self.compile_node(child_node)?;
@ -517,7 +494,13 @@ impl<'a> Compiler<'a> {
where_clauses.push(format!("NOT {}.archived", entity_alias)); where_clauses.push(format!("NOT {}.archived", entity_alias));
self.compile_filter_conditions(r#type, type_aliases, &node, &base_alias, &mut where_clauses); self.compile_filter_conditions(r#type, type_aliases, &node, &base_alias, &mut where_clauses);
self.compile_relation_conditions(r#type, type_aliases, &node, &base_alias, &mut where_clauses)?; self.compile_relation_conditions(
r#type,
type_aliases,
&node,
&base_alias,
&mut where_clauses,
)?;
Ok(where_clauses) Ok(where_clauses)
} }
@ -532,7 +515,10 @@ impl<'a> Compiler<'a> {
for (t_name, fields_val) in gf { for (t_name, fields_val) in gf {
if let Some(fields_arr) = fields_val.as_array() { if let Some(fields_arr) = fields_val.as_array() {
if fields_arr.iter().any(|v| v.as_str() == Some(field_name)) { if fields_arr.iter().any(|v| v.as_str() == Some(field_name)) {
return type_aliases.get(t_name).cloned().unwrap_or_else(|| base_alias.to_string()); return type_aliases
.get(t_name)
.cloned()
.unwrap_or_else(|| base_alias.to_string());
} }
} }
} }
@ -593,13 +579,13 @@ impl<'a> Compiler<'a> {
let full_field_path = parts.next().unwrap_or(filter_key); let full_field_path = parts.next().unwrap_or(filter_key);
let op = parts.next().unwrap_or("$eq"); let op = parts.next().unwrap_or("$eq");
let field_name = if node.stem_path.is_empty() { let field_name = if node.ast_path.is_empty() {
if full_field_path.contains('.') || full_field_path.contains('#') { if full_field_path.contains('.') || full_field_path.contains('#') {
continue; continue;
} }
full_field_path full_field_path
} else { } else {
let prefix = format!("{}.", node.stem_path); let prefix = format!("{}.", node.ast_path);
if full_field_path.starts_with(&prefix) { if full_field_path.starts_with(&prefix) {
let remainder = &full_field_path[prefix.len()..]; let remainder = &full_field_path[prefix.len()..];
if remainder.contains('.') || remainder.contains('#') { if remainder.contains('.') || remainder.contains('#') {
@ -629,13 +615,31 @@ impl<'a> Compiler<'a> {
)); ));
} else { } else {
let sql_op = match op { let sql_op = match op {
"$eq" => if is_ilike { "ILIKE" } else { "=" }, "$eq" => {
"$ne" => if is_ilike { "NOT ILIKE" } else { "!=" }, if is_ilike {
"ILIKE"
} else {
"="
}
}
"$ne" => {
if is_ilike {
"NOT ILIKE"
} else {
"!="
}
}
"$gt" => ">", "$gt" => ">",
"$gte" => ">=", "$gte" => ">=",
"$lt" => "<", "$lt" => "<",
"$lte" => "<=", "$lte" => "<=",
_ => if is_ilike { "ILIKE" } else { "=" }, _ => {
if is_ilike {
"ILIKE"
} else {
"="
}
}
}; };
let param_sql = if is_ilike && (op == "$eq" || op == "$ne") { let param_sql = if is_ilike && (op == "$eq" || op == "$ne") {
@ -662,20 +666,20 @@ impl<'a> Compiler<'a> {
) -> Result<(), String> { ) -> Result<(), String> {
if let Some(prop_ref) = &node.property_name { if let Some(prop_ref) = &node.property_name {
let prop = prop_ref.as_str(); let prop = prop_ref.as_str();
println!("DEBUG: Eval prop: {}", prop);
let mut parent_relation_alias = node.parent_alias.clone(); let mut parent_relation_alias = node.parent_alias.clone();
let mut child_relation_alias = base_alias.to_string(); let mut child_relation_alias = base_alias.to_string();
if let Some(parent_type) = node.parent_type { if let Some(parent_type) = node.parent_type {
let merged_props = self.get_merged_properties(node.schema.as_ref()); if let Some(parent_schema) = &node.parent_schema {
let relative_keys: Vec<String> = merged_props.keys().cloned().collect(); if let Some(compiled_edges) = parent_schema.obj.compiled_edges.get() {
if let Some(edge) = compiled_edges.get(prop) {
let (relation, is_parent_source) = self let is_parent_source = edge.forward;
.db let relation = self.db.relations.get(&edge.constraint).ok_or_else(|| {
.get_relation(&parent_type.name, &r#type.name, prop, Some(&relative_keys))
.ok_or_else(|| {
format!( format!(
"Could not dynamically resolve database relation mapping for {} -> {} on property {}", "Could not find exact relation constraint {} statically mapped from {} -> {} property {}",
parent_type.name, r#type.name, prop edge.constraint, parent_type.name, r#type.name, prop
) )
})?; })?;
@ -716,27 +720,9 @@ impl<'a> Compiler<'a> {
where_clauses.push(sql_string); where_clauses.push(sql_string);
} }
} }
}
}
}
Ok(()) Ok(())
} }
fn get_merged_properties(
&self,
schema: &crate::database::schema::Schema,
) -> std::collections::BTreeMap<String, Arc<crate::database::schema::Schema>> {
let mut props = std::collections::BTreeMap::new();
if let Some(ref_id) = &schema.obj.r#ref {
if let Some(parent_schema) = self.db.schemas.get(ref_id) {
props.extend(self.get_merged_properties(parent_schema));
}
}
if let Some(local_props) = &schema.obj.properties {
for (k, v) in local_props {
props.insert(k.clone(), v.clone());
}
}
props
}
} }

View File

@ -21,7 +21,6 @@ impl Queryer {
pub fn query( pub fn query(
&self, &self,
schema_id: &str, schema_id: &str,
stem_opt: Option<&str>,
filters: Option<&serde_json::Value>, filters: Option<&serde_json::Value>,
) -> crate::drop::Drop { ) -> crate::drop::Drop {
let filters_map = filters.and_then(|f| f.as_object()); let filters_map = filters.and_then(|f| f.as_object());
@ -43,11 +42,10 @@ impl Queryer {
} }
}; };
let stem_key = stem_opt.unwrap_or("/"); let cache_key = format!("{}:{}", schema_id, filter_keys.join(","));
let cache_key = format!("{}(Stem:{}):{}", schema_id, stem_key, filter_keys.join(","));
// 2. Fetch from cache or compile // 2. Fetch from cache or compile
let sql = match self.get_or_compile_sql(&cache_key, schema_id, stem_opt, &filter_keys) { let sql = match self.get_or_compile_sql(&cache_key, schema_id, &filter_keys) {
Ok(sql) => sql, Ok(sql) => sql,
Err(drop) => return drop, Err(drop) => return drop,
}; };
@ -90,7 +88,6 @@ impl Queryer {
&self, &self,
cache_key: &str, cache_key: &str,
schema_id: &str, schema_id: &str,
stem_opt: Option<&str>,
filter_keys: &[String], filter_keys: &[String],
) -> Result<String, crate::drop::Drop> { ) -> Result<String, crate::drop::Drop> {
if let Some(cached_sql) = self.cache.get(cache_key) { if let Some(cached_sql) = self.cache.get(cache_key) {
@ -100,11 +97,10 @@ impl Queryer {
let compiler = compiler::Compiler { let compiler = compiler::Compiler {
db: &self.db, db: &self.db,
filter_keys: filter_keys, filter_keys: filter_keys,
is_stem_query: stem_opt.is_some(),
alias_counter: 0, alias_counter: 0,
}; };
match compiler.compile(schema_id, stem_opt, filter_keys) { match compiler.compile(schema_id, filter_keys) {
Ok(compiled_sql) => { Ok(compiled_sql) => {
self self
.cache .cache

View File

@ -1457,24 +1457,6 @@ fn test_queryer_0_7() {
crate::tests::runner::run_test_case(&path, 0, 7).unwrap(); crate::tests::runner::run_test_case(&path, 0, 7).unwrap();
} }
#[test]
fn test_queryer_0_8() {
let path = format!("{}/fixtures/queryer.json", env!("CARGO_MANIFEST_DIR"));
crate::tests::runner::run_test_case(&path, 0, 8).unwrap();
}
#[test]
fn test_queryer_0_9() {
let path = format!("{}/fixtures/queryer.json", env!("CARGO_MANIFEST_DIR"));
crate::tests::runner::run_test_case(&path, 0, 9).unwrap();
}
#[test]
fn test_queryer_0_10() {
let path = format!("{}/fixtures/queryer.json", env!("CARGO_MANIFEST_DIR"));
crate::tests::runner::run_test_case(&path, 0, 10).unwrap();
}
#[test] #[test]
fn test_not_0_0() { fn test_not_0_0() {
let path = format!("{}/fixtures/not.json", env!("CARGO_MANIFEST_DIR")); let path = format!("{}/fixtures/not.json", env!("CARGO_MANIFEST_DIR"));
@ -3449,12 +3431,6 @@ fn test_if_then_else_13_1() {
crate::tests::runner::run_test_case(&path, 13, 1).unwrap(); crate::tests::runner::run_test_case(&path, 13, 1).unwrap();
} }
#[test]
fn test_stems_0_0() {
let path = format!("{}/fixtures/stems.json", env!("CARGO_MANIFEST_DIR"));
crate::tests::runner::run_test_case(&path, 0, 0).unwrap();
}
#[test] #[test]
fn test_empty_string_0_0() { fn test_empty_string_0_0() {
let path = format!("{}/fixtures/emptyString.json", env!("CARGO_MANIFEST_DIR")); let path = format!("{}/fixtures/emptyString.json", env!("CARGO_MANIFEST_DIR"));
@ -8566,3 +8542,9 @@ fn test_merger_0_7() {
let path = format!("{}/fixtures/merger.json", env!("CARGO_MANIFEST_DIR")); let path = format!("{}/fixtures/merger.json", env!("CARGO_MANIFEST_DIR"));
crate::tests::runner::run_test_case(&path, 0, 7).unwrap(); crate::tests::runner::run_test_case(&path, 0, 7).unwrap();
} }
#[test]
fn test_merger_0_8() {
let path = format!("{}/fixtures/merger.json", env!("CARGO_MANIFEST_DIR"));
crate::tests::runner::run_test_case(&path, 0, 8).unwrap();
}

View File

@ -10,7 +10,7 @@ fn test_library_api() {
// 1. Initially, schemas are not cached. // 1. Initially, schemas are not cached.
// Expected uninitialized drop format: errors + null response // Expected uninitialized drop format: errors + null response
let uninitialized_drop = jspg_validate("test_schema", JsonB(json!({}))); let uninitialized_drop = jspg_validate("source_schema", JsonB(json!({})));
assert_eq!( assert_eq!(
uninitialized_drop.0, uninitialized_drop.0,
json!({ json!({
@ -27,17 +27,44 @@ fn test_library_api() {
let db_json = json!({ let db_json = json!({
"puncs": [], "puncs": [],
"enums": [], "enums": [],
"relations": [], "relations": [
"types": [{ {
"id": "11111111-1111-1111-1111-111111111111",
"type": "relation",
"constraint": "fk_test_target",
"source_type": "source_schema",
"source_columns": ["target_id"],
"destination_type": "target_schema",
"destination_columns": ["id"],
"prefix": "target"
}
],
"types": [
{
"name": "source_schema",
"hierarchy": ["source_schema", "entity"],
"schemas": [{ "schemas": [{
"$id": "test_schema", "$id": "source_schema",
"type": "object", "type": "object",
"properties": { "properties": {
"name": { "type": "string" } "name": { "type": "string" },
"target": { "$ref": "target_schema" }
}, },
"required": ["name"] "required": ["name"]
}] }]
},
{
"name": "target_schema",
"hierarchy": ["target_schema", "entity"],
"schemas": [{
"$id": "target_schema",
"type": "object",
"properties": {
"value": { "type": "number" }
}
}] }]
}
]
}); });
let cache_drop = jspg_setup(JsonB(db_json)); let cache_drop = jspg_setup(JsonB(db_json));
@ -56,20 +83,39 @@ fn test_library_api() {
json!({ json!({
"type": "drop", "type": "drop",
"response": { "response": {
"test_schema": { "source_schema": {
"$id": "test_schema", "$id": "source_schema",
"type": "object", "type": "object",
"properties": { "properties": {
"name": { "type": "string" } "name": { "type": "string" },
"target": {
"$ref": "target_schema",
"compiledProperties": ["value"]
}
}, },
"required": ["name"] "required": ["name"],
"compiledProperties": ["name", "target"],
"compiledEdges": {
"target": {
"constraint": "fk_test_target",
"forward": true
}
}
},
"target_schema": {
"$id": "target_schema",
"type": "object",
"properties": {
"value": { "type": "number" }
},
"compiledProperties": ["value"]
} }
} }
}) })
); );
// 4. Validate Happy Path // 4. Validate Happy Path
let happy_drop = jspg_validate("test_schema", JsonB(json!({"name": "Neo"}))); let happy_drop = jspg_validate("source_schema", JsonB(json!({"name": "Neo"})));
assert_eq!( assert_eq!(
happy_drop.0, happy_drop.0,
json!({ json!({
@ -79,7 +125,7 @@ fn test_library_api() {
); );
// 5. Validate Unhappy Path // 5. Validate Unhappy Path
let unhappy_drop = jspg_validate("test_schema", JsonB(json!({"wrong": "data"}))); let unhappy_drop = jspg_validate("source_schema", JsonB(json!({"wrong": "data"})));
assert_eq!( assert_eq!(
unhappy_drop.0, unhappy_drop.0,
json!({ json!({

View File

@ -16,9 +16,6 @@ pub struct Case {
pub schema_id: String, pub schema_id: String,
// For Query // For Query
#[serde(default)]
pub stem: Option<String>,
#[serde(default)] #[serde(default)]
pub filters: Option<serde_json::Value>, pub filters: Option<serde_json::Value>,
@ -38,7 +35,7 @@ fn default_action() -> String {
} }
impl Case { impl Case {
pub fn run_compile(&self, db: Arc<Database>) -> Result<(), String> { pub fn run_compile(&self, _db: Arc<Database>) -> Result<(), String> {
let expected_success = self.expect.as_ref().map(|e| e.success).unwrap_or(false); let expected_success = self.expect.as_ref().map(|e| e.success).unwrap_or(false);
// We assume db has already been setup and compiled successfully by runner.rs's `jspg_setup` // We assume db has already been setup and compiled successfully by runner.rs's `jspg_setup`
@ -52,24 +49,6 @@ impl Case {
)); ));
} }
// Assert stems
if let Some(expect) = &self.expect {
if let Some(expected_stems) = &expect.stems {
// Convert the Db stems (HashMap<String, HashMap<String, Arc<Stem>>>) to matching JSON shape
let db_stems_json = serde_json::to_value(&db.stems).unwrap();
let expect_stems_json = serde_json::to_value(expected_stems).unwrap();
if db_stems_json != expect_stems_json {
let expected_pretty = serde_json::to_string_pretty(&expect_stems_json).unwrap();
let got_pretty = serde_json::to_string_pretty(&db_stems_json).unwrap();
return Err(format!(
"Stem validation failed.\nExpected:\n{}\n\nGot:\n{}",
expected_pretty, got_pretty
));
}
}
}
Ok(()) Ok(())
} }
@ -120,7 +99,7 @@ impl Case {
let merger = Merger::new(db.clone()); let merger = Merger::new(db.clone());
let test_data = self.data.clone().unwrap_or(Value::Null); let test_data = self.data.clone().unwrap_or(Value::Null);
let result = merger.merge(test_data); let result = merger.merge(&self.schema_id, test_data);
let expected_success = self.expect.as_ref().map(|e| e.success).unwrap_or(false); let expected_success = self.expect.as_ref().map(|e| e.success).unwrap_or(false);
let got_success = result.errors.is_empty(); let got_success = result.errors.is_empty();
@ -158,8 +137,7 @@ impl Case {
use crate::queryer::Queryer; use crate::queryer::Queryer;
let queryer = Queryer::new(db.clone()); let queryer = Queryer::new(db.clone());
let stem_opt = self.stem.as_deref(); let result = queryer.query(&self.schema_id, self.filters.as_ref());
let result = queryer.query(&self.schema_id, stem_opt, self.filters.as_ref());
let expected_success = self.expect.as_ref().map(|e| e.success).unwrap_or(false); let expected_success = self.expect.as_ref().map(|e| e.success).unwrap_or(false);
let got_success = result.errors.is_empty(); let got_success = result.errors.is_empty();

View File

@ -2,7 +2,6 @@ pub mod pattern;
pub mod sql; pub mod sql;
use serde::Deserialize; use serde::Deserialize;
use std::collections::HashMap;
#[derive(Debug, Deserialize)] #[derive(Debug, Deserialize)]
#[serde(untagged)] #[serde(untagged)]
@ -16,7 +15,6 @@ pub struct Expect {
pub success: bool, pub success: bool,
pub result: Option<serde_json::Value>, pub result: Option<serde_json::Value>,
pub errors: Option<Vec<serde_json::Value>>, pub errors: Option<Vec<serde_json::Value>>,
pub stems: Option<HashMap<String, HashMap<String, serde_json::Value>>>,
#[serde(default)] #[serde(default)]
pub sql: Option<Vec<SqlExpectation>>, pub sql: Option<Vec<SqlExpectation>>,
} }

View File

@ -8,7 +8,7 @@ impl<'a> ValidationContext<'a> {
result: &mut ValidationResult, result: &mut ValidationResult,
) -> Result<bool, ValidationError> { ) -> Result<bool, ValidationError> {
let current = self.instance; let current = self.instance;
if let Some(ref compiled_fmt) = self.schema.compiled_format { if let Some(compiled_fmt) = self.schema.compiled_format.get() {
match compiled_fmt { match compiled_fmt {
crate::database::schema::CompiledFormat::Func(f) => { crate::database::schema::CompiledFormat::Func(f) => {
let should = if let Some(s) = current.as_str() { let should = if let Some(s) = current.as_str() {

View File

@ -13,13 +13,18 @@ impl<'a> ValidationContext<'a> {
) -> Result<bool, ValidationError> { ) -> Result<bool, ValidationError> {
let current = self.instance; let current = self.instance;
if let Some(obj) = current.as_object() { if let Some(obj) = current.as_object() {
// Entity Bound Implicit Type Validation // Entity implicit type validation
if let Some(lookup_key) = self.schema.id.as_ref().or(self.schema.r#ref.as_ref()) { // Use the specific schema id or ref as a fallback
let base_type_name = lookup_key.split('.').next_back().unwrap_or("").to_string(); if let Some(identifier) = self.schema.id.as_ref().or(self.schema.r#ref.as_ref()) {
if let Some(type_def) = self.db.types.get(&base_type_name) // Kick in if the data object has a type field
&& let Some(type_val) = obj.get("type") if let Some(type_val) = obj.get("type")
&& let Some(type_str) = type_val.as_str() && let Some(type_str) = type_val.as_str()
{ {
// Get the string or the final segment as the base
let base = identifier.split('.').next_back().unwrap_or("").to_string();
// Check if the base is a global type name
if let Some(type_def) = self.db.types.get(&base) {
// Ensure the instance type is a variation of the global type
if type_def.variations.contains(type_str) { if type_def.variations.contains(type_str) {
// Ensure it passes strict mode // Ensure it passes strict mode
result.evaluated_keys.insert("type".to_string()); result.evaluated_keys.insert("type".to_string());
@ -33,8 +38,15 @@ impl<'a> ValidationContext<'a> {
path: format!("{}/type", self.path), path: format!("{}/type", self.path),
}); });
} }
} else {
// Ad-Hoc schemas natively use strict schema discriminator strings instead of variation inheritance
if type_str == identifier {
result.evaluated_keys.insert("type".to_string());
} }
} }
}
}
if let Some(min) = self.schema.min_properties if let Some(min) = self.schema.min_properties
&& (obj.len() as f64) < min && (obj.len() as f64) < min
{ {
@ -44,6 +56,7 @@ impl<'a> ValidationContext<'a> {
path: self.path.to_string(), path: self.path.to_string(),
}); });
} }
if let Some(max) = self.schema.max_properties if let Some(max) = self.schema.max_properties
&& (obj.len() as f64) > max && (obj.len() as f64) > max
{ {
@ -53,6 +66,7 @@ impl<'a> ValidationContext<'a> {
path: self.path.to_string(), path: self.path.to_string(),
}); });
} }
if let Some(ref req) = self.schema.required { if let Some(ref req) = self.schema.required {
for field in req { for field in req {
if !obj.contains_key(field) { if !obj.contains_key(field) {
@ -114,10 +128,14 @@ impl<'a> ValidationContext<'a> {
// Entity Bound Implicit Type Interception // Entity Bound Implicit Type Interception
if key == "type" if key == "type"
&& let Some(lookup_key) = sub_schema.id.as_ref().or(sub_schema.r#ref.as_ref()) && let Some(schema_bound) = sub_schema.id.as_ref().or(sub_schema.r#ref.as_ref())
{ {
let base_type_name = lookup_key.split('.').next_back().unwrap_or("").to_string(); let physical_type_name = schema_bound
if let Some(type_def) = self.db.types.get(&base_type_name) .split('.')
.next_back()
.unwrap_or("")
.to_string();
if let Some(type_def) = self.db.types.get(&physical_type_name)
&& let Some(instance_type) = child_instance.as_str() && let Some(instance_type) = child_instance.as_str()
&& type_def.variations.contains(instance_type) && type_def.variations.contains(instance_type)
{ {
@ -133,7 +151,7 @@ impl<'a> ValidationContext<'a> {
} }
} }
if let Some(ref compiled_pp) = self.schema.compiled_pattern_properties { if let Some(compiled_pp) = self.schema.compiled_pattern_properties.get() {
for (compiled_re, sub_schema) in compiled_pp { for (compiled_re, sub_schema) in compiled_pp {
for (key, child_instance) in obj { for (key, child_instance) in obj {
if compiled_re.0.is_match(key) { if compiled_re.0.is_match(key) {
@ -165,7 +183,7 @@ impl<'a> ValidationContext<'a> {
{ {
locally_matched = true; locally_matched = true;
} }
if !locally_matched && let Some(ref compiled_pp) = self.schema.compiled_pattern_properties if !locally_matched && let Some(compiled_pp) = self.schema.compiled_pattern_properties.get()
{ {
for (compiled_re, _) in compiled_pp { for (compiled_re, _) in compiled_pp {
if compiled_re.0.is_match(key) { if compiled_re.0.is_match(key) {

View File

@ -28,7 +28,7 @@ impl<'a> ValidationContext<'a> {
path: self.path.to_string(), path: self.path.to_string(),
}); });
} }
if let Some(ref compiled_re) = self.schema.compiled_pattern { if let Some(compiled_re) = self.schema.compiled_pattern.get() {
if !compiled_re.0.is_match(s) { if !compiled_re.0.is_match(s) {
result.errors.push(ValidationError { result.errors.push(ValidationError {
code: "PATTERN_VIOLATED".to_string(), code: "PATTERN_VIOLATED".to_string(),

View File

@ -1 +1 @@
1.0.75 1.0.84