Structured Native Language (SNL)

Structured Native Language (SNL) is the native query language of shadeDB. It was designed specifically for the shadeDB engine, providing a deterministic, structured, and lightweight syntax for interacting with database partitions. Rather than adopting a general-purpose relational query language, SNL is tightly integrated with the engine's internal architecture, allowing queries to map directly onto shadeDB's execution pipeline.

Every SNL query is composed of explicit execution stages separated by semicolons (;). Each statement has a clearly defined purpose, eliminating ambiguous execution paths and allowing the engine to process queries in a predictable top-down order.

Design Goals

SNL was created around a simple philosophy: make the query language match the database engine rather than forcing the engine to adapt to a general-purpose language.

Because SNL is purpose-built for shadeDB, its syntax reflects the engine's native execution model. Operations such as fetching, updating, recycling related records, freezing data, and projecting fields are expressed directly as engine instructions rather than being translated through multiple abstraction layers.

Why SNL?

• Engine-Native Design
  SNL was designed alongside shadeDB itself, allowing the query language and execution engine to evolve together.
• Deterministic Execution
  Queries execute in the order they are written. The engine does not rewrite queries or perform hidden execution plan transformations.
• Explicit Syntax
  Every operation is declared directly using dedicated clauses and execution modes, making query behaviour predictable and easy to reason about.
• Relationship Traversal
  Features such as RECYCLE, TARGET(), and RETRIEVE() allow related partitions to be traversed without requiring multiple client-side requests.
• Minimal Query Grammar
  A compact grammar reduces parser complexity while keeping queries readable and consistent across every supported operation.

Core Principles

Query Philosophy

SNL follows an explicit execution model. Each clause contributes a specific step to the query pipeline, and the engine processes those steps sequentially from top to bottom. There are no implicit joins, hidden mutations, or automatic query restructuring.

This approach makes query behaviour predictable, simplifies debugging, and allows developers to understand exactly how their queries are executed.

Query Lifecycle

1. 1. Receive Query
   The engine receives the SNL request from an official client or API endpoint.
2. 2. Parse & Validate
   The parser validates the query grammar, clauses, delimiters, and type wrappers.
3. 3. Authorize
   The caller's permissions are verified before any partition is accessed.
4. 4. Execute
   The requested operation is performed using the supplied clauses and execution modes.
5. 5. Shape Response
   Projection, filtering, ordering, and pagination clauses are applied before constructing the final response.
6. 6. Return Result
   The engine returns a standardized response containing the operation status, the query result (or error identifier), and the database execution latency.

Formal Language Grammar

SNL follows a deterministic, non-backtracking grammar designed for sequential parsing. Every query is processed from top to bottom using explicit delimiters and well-defined clauses, allowing the engine to validate syntax before execution begins.

The language specification below is presented using Extended Backus–Naur Form (EBNF). It describes the general structure of valid SNL queries. Individual execution modes may support additional clauses or impose further constraints, as documented throughout this reference.

Query            ::= Mode ";" Statement*

Mode             ::= "FETCH"
                  |  "INSERT"
                  |  "UPDATE"
                  |  "OVERWRITE"
                  |  "DELETE"
                  |  "FREEZE"
                  |  "UNFREEZE"
                  |  "RECYCLE"

Statement        ::= Condition ";"
                  |  JsonDocument ";"
                  |  GetClause ";"
                  |  ExcludeClause ";"
                  |  UpdateClause ";"
                  |  StrictClause ";"
                  |  VerifyClause ";"
                  |  TargetClause ";"
                  |  RetrieveClause ";"
                  |  UseClause ";"
                  |  OrderClause ";"
                  |  Pagination ";"

Condition        ::= Field "::" Type "(" Literal ")"

GetClause        ::= "GET(" FieldList ")"

ExcludeClause    ::= "EXCLUDE(" FieldList ")"

UpdateClause     ::= "UPDATE(" AssignmentList ")"

StrictClause     ::= "STRICT("
                        AssignmentList
                     ")"

VerifyClause     ::= "VERIFY("
                        AssignmentList
                     ")"

TargetClause     ::= "TARGET("
                        Field ","
                        Field
                     ")"

RetrieveClause   ::= "RETRIEVE("
                        Field ","
                        Field
                     ")"

UseClause        ::= "USE("
                        ClauseList
                     ")"

OrderClause      ::= "ORDER("
                        "ASCEND"
                     |  "DESCEND"
                     |  "RANDOM"
                     ")"

Pagination       ::= "START" Integer
                  |  "LIMIT" Integer
                  |  "START" Integer "," "LIMIT" Integer

Type             ::= "int"
                  |  "float"
                  |  "bool"
                  |  "bytes"
                  |  "str"

Lexical Rules

Every SNL query must conform to the language's lexical rules before execution. Queries that violate these rules are rejected during parsing and do not reach the execution pipeline.

Keyword Case

SNL keywords are case-insensitive. The following statements are therefore equivalent:

FETCH;

fetch;

Fetch;

Field names, however, are matched exactly as they exist within the database schema. Applications should therefore preserve the original field casing defined by the stored documents.

Query Anatomy

Every SNL query follows a predictable top-down structure. A query begins with an Execution Mode, followed by one or more clauses that refine how the operation should execute. Each statement is terminated with a semicolon (;), allowing the parser to process the query sequentially without ambiguity.

Unlike languages that rely on complex nested syntax, SNL uses independent execution clauses. Each clause has a single responsibility, making queries easy to read, validate, and execute.

General Query Structure

FETCH;
username::str(john);

VERIFY(password=str(secret));

GET(username,email,age);

ORDER(ASCEND);

START 0, LIMIT 25;

Reading the query from top to bottom:

1. Select an execution mode.
2. Specify the record matching condition.
3. Apply optional behavioural clauses.
4. Project or filter returned fields.
5. Sort the result set if required.
6. Apply pagination.
7. Return the final response.

1. Execution Mode

Every SNL query begins with an execution mode. The execution mode determines the primary action that the database should perform.

2. Record Selection

Most execution modes require a record selection statement immediately after the execution mode. This statement identifies which record or records the operation should target.

username::str(john);

Record selection uses explicit type wrappers, allowing the engine to compare values without implicit type conversions.

3. Behaviour Clauses

Behaviour clauses modify how the selected execution mode operates. Their availability depends on the chosen execution mode.

4. Projection Clauses

Projection clauses determine which fields appear in the final response without modifying the stored record.

5. Result Processing

After records have been located and processed, SNL can optionally sort and paginate the result set before returning it to the client.

Typical Execution Order

Regardless of execution mode, SNL evaluates queries in a deterministic sequence. The parser does not reorder clauses or generate hidden execution plans.

1. Validate query syntax.
2. Identify the execution mode.
3. Locate matching records.
4. Execute behavioural clauses.
5. Apply projection clauses.
6. Sort the result set.
7. Apply pagination.
8. Construct the response.

Connecting to Your Partition Cluster

Every shadeDB partition exposes a dedicated connection endpoint that applications use to communicate with the cluster. To establish a connection, three values are required:

• Partition Endpoint — The HTTPS endpoint assigned to your partition.
• Connection Token — Authenticates the client making requests.
• Cluster Token — Identifies the target cluster and authorizes access.

These credentials are available from the shadeDB dashboard after creating a partition. Treat both tokens as confidential credentials and never expose them in public repositories or client-side applications.

Install the Official Python Client

The official shadeDB Python client is distributed through PyPI. Install it before connecting to your partition:

pip install shadedb-api

Official package: https://pypi.org/project/shadedb-api/

Step 1 — Obtain Your Connection Details

From the shadeDB dashboard, open the target partition and copy the following values:

• Partition Endpoint
• Connection Token
• Cluster Token

One-Time Connection

To connect without saving the configuration locally, execute:

shadedb-api CONNECTION_ENDPOINT CONNECTION_TOKEN CLUSTER_TOKEN

Example:

shadedb-api http://host.com/connect/sdb7f585246acf1480b9caac05f811c3 
8da846a452f13d1d4cf4ddfac656a115 
8c2ff6b8c408f68

This establishes a temporary connection for the current session only. The credentials must be supplied again the next time the client starts.

Persistent Initialization

To save the connection information locally, initialize the client once using:

shadedb-api-init CONNECTION_ENDPOINT CONNECTION_TOKEN CLUSTER_TOKEN

Example:

shadedb-api-init http://host.com/connect/sdb7f585246acf1480b9caac05f811c3 
8da846a452f13d1d4cf4ddfac656a115 
8c2ff6b8c408f68

The initialization command stores the connection configuration securely for subsequent sessions.

Future Sessions

Once initialization has completed successfully, reconnecting is as simple as executing:

shadedb-api

The client automatically loads the previously saved endpoint and authentication tokens, allowing immediate access to the configured partition cluster.

The Wildcard (*) Clause

The *; clause performs a wildcard search across an entire database partition. Unlike FETCH;, which evaluates records against a specific field-value condition, the wildcard ignores record schemas and matching conditions, treating every record in the partition as a potential result.

Because wildcard searches are schema-independent, records are not required to share the same fields or document structure. Every visible record is considered during execution regardless of its individual schema.

*;

The query above performs a full-partition scan and returns every visible record stored within the database partition.

Supported Clauses

Wildcard queries support the same result-shaping clauses used by FETCH;.

Example

*;

GET(username,email);

ORDER(DESCEND);

START 0, LIMIT 50;

This query searches every record in the partition, projects only the username and email fields where available, sorts the results in descending order, and returns the first fifty records.

FETCH Mode

FETCH; is the primary retrieval mode in SNL. It searches the database for records matching the supplied condition and establishes the working dataset for the remainder of the query. By default, FETCH returns the matching records without modifying them.

FETCH also serves as the execution context for several optional clauses. After the requested records have been located, additional clauses may update, validate, verify, or shape the returned result before execution completes.

Basic Syntax

FETCH;
username::str(john);

The above query searches for every record whose username field equals "john".

Execution Flow

1. Parse the FETCH statement.
2. Locate all matching records.
3. Execute any supplied FETCH clause.
4. Return the final result.

Supported FETCH Clauses

FETCH supports several optional clauses that extend its behavior. Each clause executes after matching records have been found.

Examples

Basic Retrieval

FETCH;
username::str(john);

Returns every record whose username is john.

Updating a Retrieved Record

FETCH;
username::str(john);
UPDATE(age=int(19));

Finds the matching record and updates its age field to 19.

Verifying Credentials

FETCH;
username::str(john);
VERIFY(password=input_password);

Compares the supplied password with the stored password and returns only true or false.

Returning Selected Fields

FETCH;
username::str(john);
GET(username,bio,profile_pic);

Returns only the specified fields from the matching record.

Excluding Sensitive Fields

FETCH;
username::str(john);
EXCLUDE(password,auth_token);

Returns the matching record while omitting the specified fields.

INSERT & STATE Modes

shadeDB provides two distinct pathways for inserting documents based on performance vs durability tradeoffs:

Standard Persistent Ingestion (WAL Logged)

INSERT;
{
    "username": "john",
    "email": "john@example.com"
};

Direct Disk Write (STATE Mode)

INSERT;
STATE;
{
    "username": "john",
    "email": "john@example.com"
};

Ingestion Mechanism Comparison

UPDATE Mode

UPDATE; modifies existing records by updating only the fields provided in the update payload. Unlike OVERWRITE;, UPDATE performs a partial modification, preserving every field that is not explicitly changed.

The engine searches for all records matching the supplied query and applies the specified changes to each matching record. If multiple records satisfy the query, each record is updated independently.

UPDATE;
username::str(john);

{
    "email": "new@example.com"
};

The query above searches for every record whose username is "john" and updates only the email field. Every other field remains unchanged.

Execution Flow

1. Parse the UPDATE statement.
2. Locate all matching records.
3. Apply the supplied update payload.
4. Write the modified records back to storage.
5. Return the updated records.

Modes

Default Mode

In its default mode, UPDATE applies every field contained in the supplied payload. If a field does not already exist in the record, it is added to the schema.

STRICT Mode

STRICT; protects the existing schema by preventing unknown fields from being introduced. During execution, any field that does not already exist in the target record is silently discarded, while recognised fields are updated normally.

Behaviour Summary

• Updates only the specified fields.
• Leaves all unspecified fields unchanged.
• Can update one or multiple matching records.
• Supports STRICT; mode for schema protection.
• Use OVERWRITE; when replacing an entire record instead of modifying individual fields.

OVERWRITE Mode

OVERWRITE; replaces the entire contents of every record matching the supplied query with a new document. Unlike UPDATE;, which modifies individual fields, OVERWRITE discards the existing document and stores only the fields present in the supplied payload.

Any field omitted from the new document is permanently removed from the record. For this reason, OVERWRITE should be used when the complete record structure is known and intended to replace the existing one.

OVERWRITE;
username::str(john);

{
    "username": "john",
    "email": "new@example.com"
};

The query above searches for every record whose username is "john" and replaces the existing record with the supplied document. Any previous fields not included in the new document are removed.

Execution Flow

1. Parse the OVERWRITE statement.
2. Locate every matching record.
3. Discard the existing document.
4. Store the supplied document as the new record.
5. Return the overwritten record(s).

Modes

Default Mode

In its default mode, OVERWRITE replaces the existing record with the supplied document exactly as provided. Fields omitted from the payload are removed, while new fields are introduced into the record.

STRICT Mode

STRICT; preserves the existing schema during an overwrite operation. Any field in the supplied document that does not already exist within the target record is silently discarded before the overwrite is completed. Existing fields continue to be replaced normally.

OVERWRITE;

STRICT;

username::str(john);

{
    "username":"john",
    "email":"new@example.com",
    "role":"admin"
};

If the existing record does not contain the role field, it is discarded before the overwrite is applied. The resulting record contains only recognised fields, preventing unintended schema expansion.

Behaviour Summary

• Replaces the entire contents of every matching record.
• Fields omitted from the payload are removed.
• Can overwrite one or multiple matching records.
• Supports STRICT; mode to prevent schema expansion.
• Use UPDATE; when only specific fields need to be modified.

DELETE Mode

DELETE; permanently removes a record from the database using a unique field. To prevent accidental data loss, DELETE only accepts queries against fields declared as unique. Queries targeting non-unique fields are rejected and no records are removed.

This design guarantees that every DELETE operation targets at most one record, eliminating the risk of unintentionally deleting multiple records that share the same value.

DELETE;
username::str(john);

Assuming username is defined as a unique field, the query above permanently removes the matching record from the database.

Execution Flow

1. Parse the DELETE statement.
2. Verify that the queried field is unique.
3. Locate the matching record.
4. Remove the record from storage.
5. Update indexes and metadata.
6. Return the operation result.

Behaviour

• Only unique fields may be used as DELETE conditions.
• At most one record can be deleted per operation.
• Queries against non-unique fields are rejected.
• If no matching record exists, nothing is deleted.
• Deleted records cannot be recovered through SNL.

Example

DELETE;

id::int(42);

If id is a unique field, the matching record is permanently removed.

FREEZE & UNFREEZE Modes

FREEZE; and UNFREEZE; provide a reversible way to hide records without permanently deleting them. A frozen record remains physically stored in the database but becomes inaccessible to normal database operations until it is explicitly unfrozen.

Once frozen, a record is excluded from standard retrieval operations and cannot be modified or deleted. This allows records to be temporarily archived, quarantined, or disabled while preserving their original contents.

FREEZE Syntax

FREEZE;
id::int(56);

The query above freezes the record whose unique id is 56. The record remains stored but is hidden from subsequent queries and protected against modification or deletion.

UNFREEZE Syntax

UNFREEZE;
id::int(56);

This restores the previously frozen record, making it visible to queries and allowing normal database operations to resume.

Execution Flow

FREEZE

1. Validate the supplied unique key.
2. Locate the matching record.
3. Mark the record as frozen.
4. Hide the record from normal queries.
5. Prevent future updates and deletions until unfrozen.

UNFREEZE

1. Validate the supplied unique key.
2. Locate the frozen record.
3. Verify the original freeze identifier.
4. Remove the frozen state.
5. Restore normal database access.

Behaviour

• Frozen records remain physically stored.
• Frozen records are hidden from normal FETCH operations.
• Frozen records cannot be updated.
• Frozen records cannot be overwritten.
• Frozen records cannot be deleted.
• UNFREEZE completely restores the original record.

RECYCLE Mode

RECYCLE; is SNL's multi-stage execution engine. Rather than requiring applications to perform multiple database requests and manually combine their results, RECYCLE executes an initial query, extracts one or more values from its result, and immediately reuses those values to perform a secondary query against another database partition.

RECYCLE is designed for traversing relationships between partitions, making it ideal for retrieving related data such as users and their messages, customers and their orders, blog posts and their comments, or any other linked datasets—all within a single SNL request.

RECYCLE;

FETCH;
id::int(my_id);

TARGET(message_composed_by_userid, user_id);
USE(FETCH/WHERE);

GET(message, created_at);

ORDER(ASCEND);
START 0, LIMIT 100;

In this example, the engine first retrieves the user identified by my_id. The value stored in the user's user_id field is then recycled into another partition, where it is matched against the message_composed_by_userid field. Only the message and created_at fields are returned from each recycled record.

Execution Flow

1. Execute the initial query.
2. Retrieve the matching record.
3. Extract the required field values.
4. Map those values through TARGET() or RETRIEVE().
5. Execute the secondary query specified by USE().
6. Apply response projection using GET() or EXCLUDE(), if present.
7. Apply ordering and pagination.
8. Return the final result.

Execution Modes

RECYCLE supports two execution modes depending on how the final response should be structured.

Conjoined Mode

Conjoined Mode is enabled through TARGET(). During execution, the parent record obtained from the initial query is merged into every matching secondary record, producing a unified dataset containing information from both partitions.

TARGET(message_composed_by_userid, user_id);

This mode is particularly useful when every related record should include information about its parent without requiring another database request.

Retrieve-Only Mode

Retrieve-Only Mode is enabled through RETRIEVE(). The extracted values are used solely for filtering the secondary partition. The original parent record is omitted from the final response, returning only the related records.

RETRIEVE(message_composed_by_userid, user_id);

This mode is useful when only the destination records are required and embedding the parent record would unnecessarily increase the response size.

Supported Clauses

Advantages

• Performs relationship traversal in a single SNL request.
• Eliminates multiple client-side database queries.
• Reduces network latency by executing multi-stage operations inside the engine.
• Supports both conjoined and retrieve-only execution modes.
• Supports response projection using GET() and EXCLUDE().
• Supports ordering and efficient pagination.

The GET Clause

GET(...) is a projection clause that limits the fields returned by a query. Instead of returning the complete record, GET extracts only the specified attributes, reducing response size and preventing unnecessary data from being transmitted.

GET is particularly useful when applications require only a small subset of a record, such as displaying usernames, profile pictures, or email addresses, without retrieving the remaining fields.

FETCH;
username::str(john);

GET(
    username,
    email
);

The query above searches for the record whose username is "john" and returns only the username and email fields.

How GET Works

1. Execute the parent query.
2. Locate the matching record or records.
3. Extract only the fields listed inside GET().
4. Discard every other field from the response.
5. Return the projected result.

Behaviour

• Returns only the requested fields.
• Preserves the original record stored in the database.
• Reduces response payload size.
• Can return one or multiple fields.
• Can be combined with any FETCH operation that returns records.

Example

FETCH;

id::int(12);

GET(
    username,
    profile_pic,
    bio,
    verified
);

Only the four requested fields are included in the response. All remaining attributes, such as passwords, authentication tokens, timestamps, and other stored values, are omitted.

STRICT Mode

STRICT; enables strict schema validation during query execution. Rather than allowing operations to proceed with missing or unknown fields, STRICT enforces schema integrity according to the operation being performed. This helps prevent accidental updates, schema expansion, and invalid query execution.

STRICT behaves differently depending on the parent SNL mode. When used with FETCH, it validates that the referenced field exists before continuing. When used with UPDATE or OVERWRITE, it silently discards any fields that are not already part of the existing record.

FETCH;

username::str(john);

STRICT(
    age=int(18)
);

The query above retrieves the matching record and verifies that the age field exists before continuing. If the field is missing, execution stops and a strict validation error is returned.

STRICT Behaviour

FETCH

During a FETCH operation, STRICT validates that every referenced field exists within the retrieved record. If any field is missing, the operation fails instead of silently continuing.

• Validates field existence.
• Returns a strict validation error if a field is missing.
• Does not modify the retrieved record.

UPDATE

During an UPDATE operation, STRICT protects the existing schema. Any field present in the update payload that does not already exist within the record is ignored, while recognised fields continue to update normally.

• Unknown fields are discarded.
• Existing fields continue to update.
• No strict error is returned.

OVERWRITE

During an OVERWRITE operation, STRICT prevents the replacement document from introducing new fields. Unrecognised fields are removed before the overwrite is applied, preserving the existing schema.

• Unknown fields are discarded.
• Existing fields are overwritten normally.
• No strict error is returned.

Why Use STRICT?

• Protects the database schema from unintended expansion.
• Prevents typographical errors from creating unexpected fields.
• Ensures required fields exist before validation operations.
• Provides predictable behaviour across different SNL modes.

The TARGET Clause

TARGET(...) is a RECYCLE clause that establishes the relationship between the primary query and a secondary database partition. It maps a field from the secondary partition to a value extracted from the initial record, allowing the engine to locate related records automatically.

Unlike RETRIEVE(), TARGET performs a conjoined execution. After the related records have been found, the fields from the initial record are merged into every matching secondary record, producing a unified result.

RECYCLE;

FETCH;
id::int(my_id);

TARGET(message_composed_by_userid, user_id);
USE(FETCH/WHERE);

START 0, LIMIT 100;

In the example above, the initial query retrieves a user using my_id. The value stored in the user's user_id field is then mapped to the message_composed_by_userid field in the secondary partition, allowing the engine to retrieve every message composed by that user.

How TARGET Works

1. Execute the initial FETCH query.
2. Extract the mapped value from the returned record.
3. Match that value against the target field specified in TARGET().
4. Retrieve every matching record from the secondary partition.
5. Merge the original parent record into each matching result.
6. Return the conjoined dataset.

Syntax

TARGET accepts two arguments:

• First argument — The field in the secondary partition that should be searched.
• Second argument — The field extracted from the initial record whose value will be recycled.

For example:

TARGET(message_composed_by_userid, user_id);

Here, user_id is obtained from the first query, while message_composed_by_userid belongs to the secondary partition.

Behaviour

• Available only within RECYCLE;.
• Automatically maps values between related partitions.
• Merges the parent record into every matching secondary record.
• Supports one-to-one and one-to-many relationships.
• Can be combined with USE(), ORDER(), and START x, LIMIT y;.

Example Use Cases

• Retrieve every message written by a user.
• Retrieve all comments belonging to a post.
• Retrieve every order placed by a customer.
• Retrieve every session belonging to an account.
• Retrieve child records while embedding their parent information.

The RETRIEVE Clause

RETRIEVE(...) is a RECYCLE clause used to establish relationships between database partitions without merging the parent record into the final response. It extracts a value from the initial query, maps it to a field in the secondary partition, and returns only the matching secondary records.

Unlike TARGET(), RETRIEVE performs a filter-only operation. The initial record is used solely as a source of values for the secondary query and is omitted entirely from the returned payload.

RECYCLE;

FETCH;
id::int(my_id);

RETRIEVE(follower_id, id);
USE(FETCH/WHERE);

START 0, LIMIT 100;

In the example above, the engine first retrieves the record identified by my_id. The value of its id field is then recycled into the secondary partition, where it is matched against the follower_id field. Only the matching follower records are returned.

How RETRIEVE Works

1. Execute the initial FETCH query.
2. Extract the mapped value from the returned record.
3. Match that value against the destination field specified in RETRIEVE().
4. Retrieve every matching record from the secondary partition.
5. Return only the secondary records.

Syntax

RETRIEVE accepts two arguments:

• First argument — The field in the secondary partition that will be searched.
• Second argument — The field from the initial record whose value will be recycled.

For example:

RETRIEVE(follower_id, id);

Here, the value of id obtained from the first query is matched against the follower_id field in the secondary partition.

Behaviour

• Available only within RECYCLE;.
• Maps values between related database partitions.
• Returns only the secondary records.
• Does not merge the parent record into the response.
• Supports one-to-one and one-to-many relationships.
• Can be combined with USE(), ORDER(), and START x, LIMIT y;.

Example Use Cases

• Retrieve a user's followers.
• Retrieve all comments belonging to a post.
• Retrieve every order placed by a customer.
• Retrieve every file owned by a workspace.
• Return related records without duplicating parent information.

The USE Clause

USE(...) controls which clauses from the initial FETCH; statement are inherited and executed during the secondary phase of a RECYCLE; operation. Rather than automatically applying every compatible clause, USE allows developers to explicitly specify which query behaviour should continue into the recycled execution.

This provides fine-grained control over the secondary query, ensuring that only the desired filtering and projection logic is carried forward while ignoring unnecessary clauses.

USE(
    FETCH,
    WHERE
);

The example above instructs the RECYCLE engine to inherit only the FETCH and WHERE clauses during the secondary execution. Any compatible clauses not listed inside USE() are ignored for that recycle stage.

How USE Works

1. Execute the initial FETCH; query.
2. Recycle the required values using TARGET() or RETRIEVE().
3. Read the clauses listed inside USE().
4. Apply only those clauses to the secondary query.
5. Continue execution and return the recycled result.

Behaviour

• Available only within RECYCLE;.
• Controls which compatible clauses are inherited by the secondary query.
• Allows multiple clauses to be specified.
• Improves control over recycled query execution.
• Prevents unnecessary clauses from affecting the secondary query.

Example

RECYCLE;

FETCH;
id::int(my_id);

TARGET(message_composed_by_userid, user_id);

USE(
    FETCH,
    WHERE
);

ORDER(ASCEND);
START 0, LIMIT 100;

In this example, only the FETCH and WHERE clauses are inherited during the recycled execution. The engine ignores any other compatible clauses that are not explicitly listed inside USE().

The EXCLUDE Clause

EXCLUDE(...) is a response projection clause that removes specified fields from the returned result while leaving every other field intact. Unlike GET(), which explicitly selects the fields to return, EXCLUDE() returns the complete record except for the fields listed inside the clause.

EXCLUDE is commonly used to omit sensitive or unnecessary information such as passwords, authentication tokens, API keys, internal identifiers, or other metadata that should not be exposed to the client.

FETCH;
username::str(john);

EXCLUDE(
    password,
    api_key
);

The query above retrieves the record whose username is "john" and returns every field except password and api_key.

How EXCLUDE Works

1. Execute the parent query.
2. Retrieve the matching record or records.
3. Remove every field listed inside EXCLUDE() from the response.
4. Return the remaining fields.

Behaviour

• Returns all fields except those explicitly excluded.
• Supports excluding one or multiple fields.
• Does not modify the stored record.
• Reduces response size by omitting unnecessary data.
• Can be used with any FETCH or RECYCLE operation that returns records.

Example

FETCH;

id::int(15);

EXCLUDE(
    password,
    auth_token,
    recovery_key,
    internal_id
);

Every field belonging to the matched record is returned except password, auth_token, recovery_key, and internal_id.

GET() vs EXCLUDE()

The ORDER Clause

ORDER(...) controls the sequence in which matching records are returned. Rather than sorting by a specific field, ORDER determines whether the result set should be returned in ascending, descending, or randomized order.

ORDER is commonly used alongside START and LIMIT to control pagination and the presentation of query results.

Ascending Order

ORDER(ASCEND); returns matching records in ascending order.

ORDER(ASCEND);

Descending Order

ORDER(DESCEND); returns matching records in descending order.

ORDER(DESCEND);

Random Order

ORDER(RANDOM); randomizes the returned result set. Each execution may produce a different ordering, making it useful for recommendations, featured content, random sampling, or shuffled feeds.

ORDER(RANDOM);

Behaviour

• Supports ASCEND, DESCEND, and RANDOM.
• Can be used with FETCH and RECYCLE queries.
• Works seamlessly with START and LIMIT for pagination.
• RANDOM produces a different ordering on successive executions.
• Affects only the order in which records are returned.

Examples

Ascending

WHERE;

ORDER(ASCEND);

START 0, LIMIT 20;

Descending

WHERE;

ORDER(DESCEND);

START 0, LIMIT 20;

Random

WHERE;

ORDER(RANDOM);

LIMIT 10;

The START & LIMIT Clauses

START and LIMIT control pagination within SNL queries. They determine where the returned result set begins and how many records are returned. The clauses may be used independently or combined into a single statement.

These clauses are commonly used with FETCH and RECYCLE queries to efficiently browse large collections of records.

Combined Syntax

START 20, LIMIT 50;

The query above skips the first 20 matching records and returns at most the next 50.

START Only

START may be used on its own to specify the zero-based offset from which matching records should begin to be returned.

START 100;

LIMIT Only

LIMIT may also be used independently to restrict the maximum number of records returned, beginning from the default starting offset of zero.

LIMIT 25;

Behaviour

• START specifies the first matching record to return.
• LIMIT specifies the maximum number of returned records.
• The clauses may be used individually or together.
• When combined, the syntax is START x, LIMIT y;.
• Supported by both FETCH and RECYCLE.
• Can be combined with ORDER() for predictable pagination.
• Neither clause modifies the underlying database records.

Examples

Type System Wrappers

SNL is a strongly typed query language. Every value supplied to the engine is explicitly wrapped with its data type, eliminating the need for runtime type guessing or implicit conversions. This allows the engine to parse queries predictably while preserving the original data type throughout execution.

Unlike traditional query languages that infer types from raw literals, SNL requires each value to declare its intended type using a wrapper. This removes ambiguity and ensures comparisons, updates, and validations are performed against values of the correct type.

Supported Wrappers

Examples

age::int(25);

price::float(19.99);

verified::bool(true);

username::str(john);

avatar::bytes(binary_data);

Why Wrappers?

• Eliminates implicit type conversion.
• Prevents ambiguous comparisons.
• Ensures consistent query execution.
• Preserves the original data type throughout the query lifecycle.
• Reduces parser complexity by making every value's type explicit.

Behaviour

• Every non-string primitive value should be wrapped using its corresponding type.
• The wrapper becomes part of the SNL syntax and is interpreted by the query engine.
• Typed values are supported across FETCH, UPDATE, OVERWRITE, DELETE, FREEZE, UNFREEZE, and other compatible SNL operations.
• Field declarations follow the syntax field::type(value).

Execution Pipeline

Every SNL query passes through a deterministic execution pipeline before a response is returned. Each stage performs a specific responsibility, ensuring queries are executed consistently while maintaining security, integrity, and predictable performance.

Depending on the operation being performed, certain stages may be skipped. For example, read-only queries do not require write locks or persistence, whereas write operations continue through the full commit process.

Execution Stages

1. 1. Parsing
   The SNL parser reads the query sequentially, separating statements using semicolon (;) delimiters. Clauses, values, type wrappers, and execution modes are tokenized and validated before execution begins.
2. 2. Authorization Validation
   The engine verifies that the current security context has permission to access the requested database partition and perform the requested operation. Queries that fail authorization terminate immediately.
3. 3. Query Planning
   Based on the parsed clauses, the engine determines the execution strategy, including the query mode, compatible sub-clauses, pagination, ordering, and any recycle operations that must be performed.
4. 4. Index Resolution
   Indexed fields are consulted to locate candidate records efficiently. Whenever possible, the engine avoids scanning unnecessary records by using available indexes to narrow the search space.
5. 5. Record Evaluation
   Candidate records are evaluated against the supplied conditions. Operations such as STRICT(), VERIFY(), and filtering clauses are applied before records are accepted or rejected.
6. 6. Transaction Isolation
   Write operations acquire the necessary locks to ensure transactional consistency while preventing concurrent modifications. Read operations bypass write locking, allowing high-throughput concurrent reads.
7. 7. Operation Execution
   The requested SNL operation is performed. Depending on the query, this may involve fetching records, updating existing values, overwriting documents, deleting records, freezing entries, or executing multi-stage RECYCLE queries.
8. 8. Response Projection
   Clauses such as GET(), EXCLUDE(), ORDER(), START, and LIMIT are applied to shape the final response returned to the client.
9. 9. Persistence & Commit
   Standard write operations are first appended to the Write-Ahead Log (WAL) to guarantee recoverability before being synchronized with persistent storage. Queries executed using the STATE insert mode bypass the WAL and are written directly to disk before becoming available in memory.
10. 10. Response Generation
    The engine serializes the final result and returns it to the client. Depending on the executed operation, the response may contain records, status values, Boolean results, or confirmation that the operation completed successfully.

Pipeline Characteristics

• Deterministic execution from parsing to response.
• Permission validation before any data access occurs.
• Index-assisted record discovery for efficient lookups.
• Transaction isolation for write operations.
• Response shaping through projection, ordering, and pagination clauses.
• Crash recovery through Write-Ahead Logging for standard write operations.
• Direct-to-disk persistence available through STATE inserts.

Performance Optimization

SNL is designed to execute efficiently under high concurrency, but query structure still plays an important role in achieving optimal throughput and low latency. Choosing the appropriate clauses and execution modes can significantly reduce CPU usage, memory allocation, disk activity, and network transfer sizes.

The following recommendations help maximize performance while minimizing resource consumption across both small and large datasets.

Optimization Best Practices

• Use RETRIEVE instead of TARGET whenever parent data is unnecessary.
  TARGET() merges the parent record into every matching recycled result, requiring additional memory allocations and larger response payloads. If only the related records are required, prefer RETRIEVE().
• Project only required fields with GET().
  Returning fewer fields reduces serialization time, memory usage, and network bandwidth. Whenever possible, request only the fields your application actually needs.
• Hide unnecessary fields using EXCLUDE().
  Large attributes such as binary objects, lengthy descriptions, or internal metadata can unnecessarily increase response sizes. Excluding them results in smaller payloads and faster transfers.
• Paginate large result sets.
  Rather than retrieving every matching record in a single request, combine START and LIMIT to return manageable batches of data.
• Combine ORDER() with pagination.
  Using ORDER() together with START and LIMIT produces stable and predictable pagination across multiple requests.
• Prefer VERIFY() for credential validation.
  When checking passwords, API keys, or authentication tokens, use VERIFY() instead of fetching the value and comparing it inside application code. This reduces response payloads and simplifies validation.
• Use STRICT() when schema validation is required.
  Allowing the engine to validate fields during execution helps avoid invalid updates and prevents unnecessary retry operations.
• Use RECYCLE for related data retrieval.
  Instead of issuing multiple client-side queries and manually joining the results, use RECYCLE to perform relationship traversal within a single engine execution.
• Use unique fields whenever possible.
  Queries against unique identifiers generally require less work than searches across non-unique values, making them ideal for record lookups and mutation operations.

General Recommendations

Security Matrix

Security is integrated into every stage of the SNL execution pipeline. Before a query is allowed to access any database partition, shadeDB validates the client's authorization context, ensuring that only permitted operations can proceed. Unauthorized requests are rejected before any records, indexes, or storage blocks are examined.

Beyond access control, SNL's grammar is intentionally restrictive. Every query follows a deterministic syntax, allowing the parser to validate structure before execution and reject malformed or invalid statements.

Security Features

• Partition-Level Authorization — Every query is validated against the caller's permissions before any data is accessed.
• Deterministic Parsing — Queries are parsed using a fixed grammar with explicit delimiters, reducing ambiguity during execution.
• Strong Type Validation — Explicit type wrappers such as int(), float(), bool(), and str() ensure values are interpreted as their intended types.
• STRICT() Validation — Schema validation can be enforced during supported operations to reject invalid or unexpected fields.
• VERIFY() — Allows credential verification without returning the stored value, reducing unnecessary exposure of sensitive fields.
• Projection Controls — GET() and EXCLUDE() allow applications to expose only the data required by the client.
• Controlled Mutations — Operations such as DELETE, FREEZE, and UNFREEZE are constrained by their execution rules, helping prevent unintended modifications.

Injection Defense

Security Workflow

1. The query is parsed and validated.
2. User permissions are verified for the target partition.
3. The query structure and compatible clauses are validated.
4. Optional schema validation is performed when STRICT() is used.
5. The authorized operation is executed.
6. The response is filtered through projection clauses such as GET() or EXCLUDE() before being returned.

Error Catalog

Every SNL query returns a consistent response structure, regardless of whether the operation succeeds or fails. When an error occurs, the engine returns an error identifier through the message field while preserving the normal response format. This allows applications to handle successful queries and failed operations using the same response parser.

Errors can occur during parsing, authorization, schema validation, query execution, transaction processing, or storage operations. Each error identifier is stable and intended for programmatic handling, making it preferable to compare error codes rather than relying on human-readable text.

Successful Response

{
    "status": "success",
    "message": [
        {
            "username": "john",
            "age": 19
        }
    ],
    "db_latency": "0.41ms"
}

For successful operations, message contains the query result. The returned value depends on the executed operation and may contain records, Boolean values, status confirmations, or other valid SNL responses.

Error Response

{
    "status": "failed",
    "message": "ERR_SCHEMA_MISMATCH",
    "db_latency": "0.17ms"
}

When execution fails, message contains the error identifier instead of a query result. Applications should first inspect status. If the status indicates failure, the value contained in message should be interpreted as an SNL error code.

Core Error Reference

Error Categories

Debugging Checklist

1. Verify the status field.
2. If the status indicates failure, inspect the value of message.
3. Compare the returned error identifier with the Error Reference table.
4. Review the query for syntax, clause compatibility, and correct type wrappers.
5. If STRICT() is used, ensure every referenced field exists within the record schema.
6. Confirm that the executing client has permission to access the target partition.

Production Examples

This section demonstrates complete SNL queries commonly used in production environments. Each example combines multiple clauses to illustrate how different parts of the language work together during execution.

The examples below assume that the executing client already has permission to access the target database partition.

Example 1 — Fetch Active Users

WHERE;
status::str(active);

GET(
    username,
    email,
    profile_pic
);

EXCLUDE(
    auth_token
);

ORDER(DESCEND);

START 0, LIMIT 25;

This query retrieves active users, returns only the requested public fields, removes the authentication token from the response, sorts the result in descending order, and returns the first twenty-five matching records.

Typical Response

{
    "status": "success",
    "message": [
        {
            "username": "john",
            "email": "john@example.com",
            "profile_pic": "/images/john.png"
        }
    ],
    "db_latency": "0.34ms"
}

Example 2 — Verify Credentials

FETCH;
username::str(john);

VERIFY(
    password=str(my_password)
);

Rather than returning the stored password, this query compares the supplied password against the stored value and returns a Boolean result indicating whether the verification succeeded.

Example 3 — Atomic Update

FETCH;
username::str(john);

UPDATE(
    bio=str(Software Engineer)
);

Updates only the specified fields while preserving every other attribute within the record.

Example 4 — Strict Update

FETCH;
username::str(john);

STRICT(
    bio=str(Software Engineer)
);

Performs the update only if every supplied field already exists within the record schema. Otherwise, the operation returns a schema validation error.

Example 5 — RECYCLE Relationship Query

RECYCLE;

FETCH;
id::int(my_id);

RETRIEVE(
    message_composed_by_userid,
    user_id
);

USE(WHERE);

GET(
    message,
    created_at
);

ORDER(DESCEND);

START 0, LIMIT 50;

Retrieves a user's messages using a single SNL request. Only message data is returned because RETRIEVE() performs filtering without merging the parent record.

Example 6 — Freeze a Record

FREEZE;

id::int(42);

Prevents the matching record from participating in normal queries and protects it from modification until it is explicitly unfrozen.

Example 7 — Delete a Record

DELETE;

id::int(42);

Deletes the matching record using its unique field. Since DELETE only operates on unique fields, accidental removal of multiple records is prevented by design.