Skip to content

Reviewer 1

This is a well-conceived and well-executed project. Here are my thoughts, organized from high-level impressions down to specific observations.

Overall Impression

The concept hits a genuinely underserved niche. Stata users switching to modern data stacks lose a great deal of command ergonomics — summarize, codebook, tabulate, by: forms — and nothing has cleanly replaced that feel. TabDat-Explore fills exactly that gap. The tech choices (DuckDB, Parquet, prompt-toolkit, Altair) are all well-suited and contemporary. The phased dev approach is also disciplined — you clearly shipped end-to-end slices rather than building a horizontal slab first.

Architecture Strengths

Clean separation of concerns. Parser → Executor → Backend → Formatter is a solid pipeline. Keeping format_result completely decoupled from backend.py is the right call — backends should not own display.

Immutable models. Using pydantic frozen dataclasses for both commands (UseCommand, SummarizeCommand, etc.) and results (SummarizeResult, TableResult, etc.) is FP-disciplined and makes the pipeline easy to reason about. Pattern matching on the Command union type in executor.py is clean.

pymonad.Either in the parser. Visible in the imports — this is a nice touch for railway-oriented parse error handling. It's rare to see this in Python project and it fits your FP preferences well.

Lazy loading design. The execution_mode: Literal["eager", "lazy"] field on DatasetInfo and the DuckDB read_parquet scan view approach for lazy mode is elegant. You get pushdown without materializing the full frame.

Things to Think About

1. Single active dataset as a ceiling

The single active dataset is great for MVP mental clarity, but it will become the primary friction point before scripting (Phase 8) even arrives. Users will want to join or compare two frames without reaching for sql. A lightweight named table registry — not replacing the active model, just augmenting it — might be worth designing early even if not exposed until Phase 8. The sql ... into <table> path already creates a logical "named table", but it immediately replaces the active one, losing the original. That's a one-way door that users will quickly bump into.

2. pymonad dependency is risky

pymonad is essentially unmaintained (last real update ~2021) and the Either type it provides is something you could trivially replace with your own two-liner, or use result (the returns library) instead. It's also a non-obvious dep for contributors. Worth considering:

# trivial replacement
type ParseResult[T] = T | ParseError

or returns.result.Result from the returns library if you want the full railway monad.

3. executor.py is growing into a dispatch table

The execute() method is already ~130 lines of isinstance checks. This is fine now, but it'll be painful at Phase 8 when scripting adds looping, conditionals, and macro-like constructs. It might be worth registering handlers in a dict[type[Command], Callable] or using functools.singledispatch before the method grows further.

4. CountResult reads session state instead of querying

if isinstance(command, CountCommand):
    dataset = self._require_active_dataset("count")
    return CountResult(row_count=dataset.row_count)

dataset.row_count is captured at use time. After keep if ... or drop if ..., the DatasetInfo is replaced via self.state.active_dataset = next_dataset, so it should usually be correct. But the row_count on DatasetInfo is whatever the backend last reported — if there's ever a code path where the active dataset info goes stale (e.g., future in-place mutation), this becomes a silent bug. Making count always issue a live COUNT(*) would be safer and trivially cheap.

5. CollapseCommand replaces the active dataset silently

collapse is destructive — it shrinks the frame to aggregated groups. This is Stata behavior, so it's intentional. But worth adding a confirmatory message that makes the irreversibility obvious, maybe something like:

Collapsed dataset (3 rows, 2 columns) — original data replaced

The current TransformResult("Collapsed dataset", ...) is easy to miss.

6. Plot path naming doesn't include timestamp

The default_plot_path("histogram", ("age",)) presumably produces something like artifacts/plots/histogram-age.svg. Without a timestamp, repeated histogram age calls silently overwrite the prior artifact. The project proposal sketched histogram_age_20260502_143012.png — was this intentionally dropped?

7. Error hierarchy is thin

errors.py has just TabDatError, ParseError, and ExecutionError. As Phase 8+ adds scripting, you'll likely want finer distinctions: NoActiveDatasetError, ColumnNotFoundError, TypeMismatchError, etc. These can all subclass ExecutionError but letting callers catch them specifically (in a future REPL that wants to give context-sensitive hints) will be easier than string-matching error messages.

Minor Observations

  • _is_numeric_type in executor.py does prefix-based string matching on DuckDB type names. This is fragile — DuckDB can return DECIMAL(10, 2) or DOUBLE PRECISION depending on context. Consider normalizing via DuckDB's own typeof() or checking against a set after stripping parameters.

  • formatter.py column alignment assumes monospace terminal output, which is correct. But it materializes all rows before computing widths (materialized = list(rows)). For a head 5 that's fine; for a large tabulate result it could be unexpectedly slow. Streaming output with progressive column widths is complex, but at least a row-count cap would prevent surprising pauses.

  • The pyproject.toml specifies requires-python = ">=3.13" which is forward-looking and bold. Worth documenting explicitly since many CI environments and user machines will be on 3.11/3.12.

Phase 8 (Scripting) Thoughts — Looking Ahead

The biggest architectural question before Phase 8: where do script-level constructs live? A .do-like script is essentially a sequence of Command objects, which your parser already produces. But scripting adds:

  • foreach, forvalues loops
  • macro define, macro substitution (\var'`)
  • if/else flow at the script level (not row-level filter if)
  • capture / error suppression

These are meta-commands that don't fit cleanly in the current Command union without expanding the AST significantly. Designing a ScriptBlock node type sooner rather than later — even as ParsedCommand placeholder — would keep Phase 8 from requiring a parser rewrite.

Overall: this is a thoughtful, well-structured project with good taste in both tooling and design philosophy. The phased discipline has clearly paid off — you've shipped seven phases worth of usable behavior without overbuilding. The main things to watch are the growing execute() dispatch table, the single-dataset ceiling, and getting ahead of the scripting AST needs before Phase 8.

Reviewer 2

At a Glance: TabDat-Explore is an exceptionally well-structured personal project that effectively bridges the gap between legacy statistical ergonomics and modern data engineering. By prioritizing CLI-native UX (completions, syntax highlighting) and leveraging a robust backend (DuckDB/Polars), the project addresses a genuine friction point in the current data science workflow—the "middle ground" between heavy notebooks and low-level SQL.

Strategic Observations

  • Vertical Slice Discipline: The transition from Phase 1 to Phase 7 demonstrates a mature approach to software development. By building "vertically"—ensuring a single command flows through the parser, executor, and backend before expanding the command list—you have avoided the common pitfall of "horizontal bloat" (having many commands that are all partially broken).
  • Ergonomic Arbitrage: The decision to be "Stata-inspired but not compatible" is a strategic masterstroke. It allows you to leverage decades of proven command-line ergonomics (like the concise by: and if syntax) without being shackled to legacy data formats or archaic memory management.
  • Infrastructure Synergy: Utilizing DuckDB as the primary execution engine provides immediate support for Parquet pushdown and lazy execution, which are critical for the "modern" part of your value proposition.

Architectural Strengths

Component Strategic Value
Parser The use of pymonad for functional error handling in the parser foundation ensures high reliability during complex expression evaluation.
CLI UX Treating autocomplete and syntax highlighting as "core value" rather than "polish" significantly lowers the barrier to entry for terminal-native users.
Visualization The artifact-based model (saving to artifacts/plots/) is superior for reproducibility compared to volatile pop-up windows.

Potential Growth Areas & Challenges

  • Polars vs. DuckDB Duality: As you move deeper into Phase 7 and 8, maintaining dual-engine support (DuckDB and Polars) may introduce "leaky abstractions" where a command works in one but not the other due to different SQL or expression support.
  • The "Save" Boundary: Currently, transformations are session-local. Introducing a save or export command will be a significant milestone that moves the tool from "exploration" to "data preparation."
  • Scripting Reproducibility: The upcoming Phase 8 (.do files) will be the ultimate test of the tool's consistency. Ensuring that a script run today produces identical results in six months will require careful versioning of the execution environment.

Actionable Recommendations

  1. Dogfooding Milestone: Before moving to Phase 8 (Scripting), try to perform a complete EDA on a public dataset (e.g., NYC Taxi or Titanic) using only tabdat. This will likely reveal if the current collapse and generate operations are expressive enough for real-world "feature engineering".
  2. Type Safety: Continue leveraging pydantic and mypy for strict data integrity, especially as you add more complex group operations.
  3. Dependency Management: Stay aligned with Python 3.13 features as planned; the uv integration is already providing a very modern developer experience.

Would you like me to help you draft the command contract for a specific future command, such as a save or export function for Phase 8?

Reviewer 3

I like it. The project has a clear product thesis: Stata-like command ergonomics for modern Parquet/DuckDB workflows, and the repo is unusually well-structured for a personal project.

Strongest parts:

  • Clear phased roadmap, spec, changelog, and architecture docs.
  • Good separation: parser → executor → backend → formatter/visualization.
  • Tests appear broad across parser, executor, CLI, and shell.
  • The command surface is coherent and already useful for first-pass EDA.
  • SQL escape hatch is a smart way to avoid overbuilding early commands.

Main concerns:

  • “Lazy” is currently more of a load-time entrypoint than true lazy execution. Transformations seem to materialize into temp tables, and use ..., lazy still counts rows immediately, which weakens the large-data story.
  • engine=polars is accepted but not truly executed through Polars. I would either hide it for now or label it experimental very visibly.
  • Parser complexity is growing. It is still manageable, but options, SQL, expressions, and Stata-like syntax will become harder to evolve without a tighter grammar strategy.
  • No script mode yet, but the product promise strongly implies reproducible scripted EDA. I would prioritize this before adding many more commands.

My recommended next priorities:

  1. Add script execution: tabdat script.td or tabdat -f script.td.
  2. Tighten lazy semantics: avoid load-time full counts unless requested.
  3. Decide whether Polars is real now or future-only.
  4. Add CSV support only after script mode, not before.
  5. Add golden-output tests for full mini sessions.

Overall: this is a strong foundation. The biggest strategic advice is do not add many commands yet. Make the existing language reproducible, reliable, and honest about eager/lazy behavior first.