TabDat Language Semantics

This document records stable cross-command behavior. It covers identifier spelling, quoting, explicit missing values, expression domains, arithmetic results, write-target, and failure semantics; ordering, randomness, estimation samples, and exit behavior remain separate contracts.

Identifier spelling and quoting

• Bare identifiers preserve their exact spelling and are case-sensitive. age and Age refer to different variables; command names remain case-insensitive.
• A bare identifier starts with a Unicode letter or _, followed by Unicode letters, digits, or _.
• Backtick-quoted identifiers can contain whitespace and punctuation. They are accepted for variable targets, variable lists, and expression references, and their contents are preserved exactly.
• Inside a backtick-quoted identifier, two consecutive backticks represent one literal backtick. Empty quoted identifiers are invalid.
• SQL text keeps SQL's own identifier rules; this command-language contract does not rewrite SQL.

For example:

generate `age group` = age + 1
replace `weight kg` = `weight kg` * 2 if `status flag` == "active"

Quoted identifiers are not command names or option names. An exact spelling mismatch remains an unknown-variable error and follows the write-validation atomicity policy below.

Explicit missing values and predicates

• The unquoted token null is a case-insensitive missing-value literal. A quoted `null` is still an ordinary identifier with exact spelling.
• value == null and null == value are true only for missing values; value != null and null != value are true only for nonmissing values.
• null == null is true and null != null is false.
• A direct null expression can generate or replace an all-missing value and can be used as a row predicate. It follows the existing keep/drop policy below.
• Null used with another comparison operator, unary arithmetic, or a function is rejected with a stable execution error. This slice does not add is null, missing(), or nonmissing() syntax.

Missing values and predicates

• Missing values are represented as null values (None at the Python boundary); this slice does not introduce a special numeric missing sentinel.
• keep if expression retains only rows where the expression is true. False and missing predicate results are not kept.
• drop if expression removes only rows where the expression is true. False and missing predicate results remain in the active dataset.
• replace name = expression if condition updates only true-condition rows. False and missing conditions preserve the existing value.
• summarize counts nonmissing numeric values; its means, minima, and maxima ignore missing values. codebook reports nonmissing and missing counts explicitly.
• missing [varlist] reports total, missing, nonmissing, and missing-percentage counts for each requested column. It uses explicit nulls only; empty strings and user-defined sentinel codes remain nonmissing. With no varlist, schema order is used; requested order is preserved.
• missing reports 0.0 percent for columns in an empty dataset and performs a bounded aggregate scan without converting a Polars-lazy session to eager mode.
• assert <boolean-expression> checks every active row without replacing the active relation. True predicates pass; false and missing predicates fail. Empty datasets pass, and failures report deterministic checked/failed counts while preserving the session data state. Aggregate checks keep Polars-lazy sessions lazy.
• duplicates [report] [varlist] groups rows by all public columns or the requested key variables. Null key values compare equal for this report. It reports total rows, unique groups, duplicate groups, rows in duplicate groups, surplus rows, and maximum group size without changing data. Empty datasets return zero counts; aggregate reports preserve Polars-lazy sessions. Listing, tagging, and dropping duplicate rows are intentionally not part of this command contract.
• isid varlist [, missok] checks that each key combination occurs at most once without changing data. Null key values compare equal for duplicate grouping; without missok, any null key row fails, while missok permits such rows only when their complete key combination is unique. Empty datasets pass and aggregate checks preserve Polars-lazy sessions.
• datasignature computes a versioned SHA-256 fingerprint of public schema, active row order, and cell values with explicit null/non-finite encodings. It excludes source path, backend, execution mode, and session-local labels. The scan is read-only, preserves Polars-lazy plans, and returns a valid schema-dependent signature for an empty dataset.
• tabulate and bar omit missing categories by default. Their missing option includes missing categories where the command supports it; bar charts display that category as <missing>.

Expression domains and coercion

• Numeric columns and numeric literals share one numeric domain. Numeric widening within that domain is allowed; values are not parsed from or stringified into text implicitly. Integral arithmetic has the exact-width rule below.
• Unsafe combinations of unsigned numeric columns and negative numeric literals are rejected consistently rather than relying on backend-specific signed/unsigned coercion.
• Text values and string literals share the string domain. Boolean values and comparison results use the boolean domain. Other backend scalar types are not coerced into either domain.
• Equality/inequality and ordering require matching domains, except for numeric pairs and the existing null-aware equality/inequality rules.
• Arithmetic and unary minus require numeric operands. String concatenation is not defined here.
• Numeric functions (abs, ceil, floor, ln, log, round, sqrt) require numeric arguments; lower and upper require string arguments.
• keep, drop, replace if, and tabulate if require boolean or missing conditions. Numeric or string truthiness is rejected.
• replace permits null or an expression in the target's domain; cross-domain assignment is rejected before the active relation changes.
• Mixed-domain failures use deterministic type-mismatch diagnostics, and Polars-lazy validation occurs before fallback materialization.

Arithmetic results and non-finite values

• Arithmetic operators +, -, *, and /, plus unary minus, continue to require numeric operands. Integral +, -, *, and unary minus subtrees use exact DECIMAL(38,0) results for signed and unsigned integer operands and integer literals.
• An integral result outside DECIMAL(38,0) becomes missing for that row rather than wrapping. Other rows remain eligible in generate, replace, and row predicates under their existing policies.
• Missing operands propagate to missing results. The explicit null literal remains rejected by arithmetic and functions; this rule covers missing values supplied by data columns.
• Division is real division. A zero denominator, including 0 / 0, produces a missing result for that row rather than infinity or NaN.
• Floating operands and decimal-scale arithmetic retain their existing backend numeric behavior; this slice does not promise their storage widths or scale/precision propagation.
• sqrt(x) produces missing when x < 0; ln(x) and log(x) produce missing when x <= 0.
• Any computed NaN or infinity from supported arithmetic or numeric functions is normalized to missing. A direct identifier does not rewrite a non-finite value already present in the source.
• Subtraction involving an unsigned numeric column and unary minus of an unsigned numeric expression are rejected before execution; wraparound and implicit signed widening are not inferred.
• The policy is row-level: valid rows remain usable in generate, replace, and arithmetic predicates while affected rows become missing and follow the existing predicate rules.
• Successful generate, replace, keep, and drop results append overflow rows: N when exact integral overflow affected one or more command rows. Missing operands, false/missing predicates, zero division, non-finite values, and scale-bearing decimal/floating arithmetic are not counted.
• A zero overflow count does not change the existing transform-result text; diagnostics are informational and do not turn a successful command into an error.

Grouped-result ordering

• by summarize, by count, collapse, and long-form tabulate sort grouping dimensions in native scalar order with missing values last. Numeric values sort numerically; strings sort lexicographically; booleans sort false before true.
• Wide-form tabulate uses the same native order for row keys and column headers. Numeric labels are not ordered by their rendered text, so 2 precedes 10.
• bar sorts nonmissing categories by descending count, then native category order for ties; the missing category is always last.
• SQL without explicit order by remains a separate contract.

Categorical ordering

• Category labels use native scalar order: numeric values sort numerically, text values lexicographically, and booleans false before true. Numeric labels are not compared by rendered text.
• tabulate excludes missing categories by default. With missing, missing categories appear after all nonmissing row keys and column headers.
• bar orders nonmissing categories by descending count, then native category order for ties. With missing, the missing category remains last and displays as <missing>.
• Rendered labels are collision-safe: a missing label and a literal reserved-looking text label remain distinct, and multi-key tabulate labels remain distinct even when their separator text would collide.
• Source arrival order and user-defined category levels are not ordering contracts; no category metadata is persisted by this slice.
• Eager, DuckDB-lazy, and Polars-lazy tabulate/bar outputs agree; formatting does not alter order.

Active row order

• The active dataset has one current row sequence; source order is the initial sequence.
• head n returns the first n rows and tail n returns the last n rows in sequence order, restoring the tail rows to their original relative order. A zero limit returns no rows.
• keep if and drop if preserve the relative order of retained rows. False and missing predicate results follow the existing keep/drop policy and never reorder survivors.
• sort <varlist> orders ascending by native scalar values, places nulls last, and preserves prior row order for ties. gsort [+|-]varlist adds explicit per-key directions with the same native comparison, nulls-last, stable-tie, metadata, and eager/lazy rules; it does not support expression keys or random ordering.
• Column projection and row-preserving value transformations preserve the current row sequence.
• Grouped or relation-changing commands such as collapse establish separate result-sequence contracts; this slice does not redefine their later preview order.

SQL and named-table row order

• A direct SQL result follows the row sequence produced by its query. An explicit order by defines the listed-key order; a reproducible total sequence requires tie-breaker keys that distinguish tied rows. SQL without order by has no guarantee here.
• sql ... into name stores and activates the query result without reordering it. use name restores that stored sequence, and head/tail consume it using the active row-order rules.
• SQL remains an eager boundary for named-table creation; a Polars-lazy input uses the existing fallback path before the query executes, and successful named-table activation resets the prior materialization reason.
• Reshape row order remains a separate contract.

Join row order

• The active dataset is the left input. For each active row, join name on keys emits matching rows from named table name in its stored sequence.
• Output is grouped by active-row sequence. Duplicate right-side matches remain present, and a later active row never appears before an earlier active row's matches.
• An inner join omits active rows with no match. A left join emits one row with missing right-side values for each active row with no match.
• Existing key equality, suffixing, output-column, and missing-key behavior remain unchanged.
• Join validates named-table existence and key columns before Polars fallback; a validation failure preserves the active rows, execution mode, and materialization metadata.
• Reshape row order and categorical order remain separate contracts.

Reshape row order

• reshape long preserves the active source-row sequence. For each source row, generated rows follow the established wide-column j-value sequence: scan requested stubs in command varlist order, scan each stub's matching source columns in schema order, and keep each suffix's first appearance once.
• reshape wide emits one row per identifier group in the order of the first active row belonging to that group. Existing generated-column order and duplicate-cell aggregation remain unchanged.
• head/tail consume each reshape result using the active row-order rules.
• Reshape crosses the existing eager boundary where required; eager, DuckDB-lazy, and Polars-lazy inputs produce the same result sequence.
• Append/join order and categorical order remain separate contracts.

Append row order

• append name emits every row in the active dataset first, followed by every row in named table name in its stored sequence.
• Append does not sort, deduplicate, or interleave the two inputs. head/tail consume the combined sequence using the active row-order rules.
• Reshape row order remains a separate contract.

Data-dictionary persistence

• label save <path> [, replace] writes a deterministic, versioned UTF-8 JSON dictionary for the active dataset's variable labels, value-label sets, and attachments. It does not alter the active data or force a lazy scan.
• label use <path> replaces the active dataset's label metadata only after the file, schema, value mappings, and variable references validate. A malformed or incompatible dictionary leaves the active dataset and existing labels unchanged.
• TabDat's dictionary format uses schema_version: 1 and is intentionally TabDat-native; it is not a claim of direct Stata .dta, SAS catalog, or SPSS .sav metadata compatibility.
• Label metadata operations preserve a DuckDB- or Polars-lazy execution mode because they operate on schema and session state rather than data values.

Write targets

Command family	Target rule	Failure behavior
generate name = expression	name must not already exist	Reject the collision; do not replace the active relation
rename old new	old must exist and new must not exist	Reject either invalid source/destination before replacement
replace name = expression	name must already exist	Reject the missing target; do not create it implicitly
recode ..., generate(names)	one output per input; outputs must be unique and new	Validate all outputs before changing the active relation

The diagnostics identify the target/source problem. Existing wording is covered by focused tests and can only change through a later language-error contract.

Atomic validation failures

Write commands validate target existence, source existence, output cardinality, and expression requirements before replacing the active relation. A validation failure preserves:

• active columns and their order;
• active rows and values;
• execution mode, lazy engine, and active table metadata;
• last-successful-operation and materialization-reason metadata.

This is an active-dataset boundary guarantee. It does not promise rollback of unrelated external side effects, such as an already-created artifact file from another command.

Deliberate limits

Machine-readable overflow envelopes, arbitrary-precision arithmetic, decimal-scale/precision propagation, floating storage widths, SQL without explicit ordering, randomness, estimation samples, and exit codes are not defined here yet.