Standard diff-based code review shows what changed syntactically, not what broke behaviorally. A function can be refactored in a way that looks clean in the diff, passes all tests, and still violate the invariant it was supposed to maintain. NoDiff addresses this by combining Tree-sitter AST diffing (to identify structurally significant changes) with two-stage LLM reasoning: first extract the intent of the change, then check whether that intent is preserved. The key insight: asking 'what changed?' before 'what broke?' gives the model better grounding for regression analysis.
Building AI-powered code intelligence for semantic regression detection
›Research on a class of bugs invisible to tests: intent-layer regressions
›Multi-language AST parsing with Tree-sitter; LLM intent analysis for code review
›Prototype case study of 10 real-world regressions catches the two-stage approach works
Read more
Hiring Summary
Hiring Summary
NoDiff is a research-stage AI code review tool I'm building that catches behavioral regressions invisible to standard diff review. It combines Tree-sitter AST parsing (four languages) with two-stage LLM reasoning: intent extraction followed by regression check. The key insight: most code review tools show what changed, not what broke. NoDiff bridges that gap. Currently in prototype/research stage with a 10-example case study supporting the core approach.
Evidence & Proof
Each claim is paired with evidence that a recruiter can request or verify.
“Two-stage LLM reasoning (intent extraction before regression check) catches behavioral regressions that are invisible to test suites and line-level diff review — supported by a prototype case study of 10 real-world examples.”
writeupRegression Detection Case Study● coming soon
engineerai
“AST-level diffing eliminates formatting noise from line-level diffs, producing a semantics-only ChangeSet that provides higher-signal context for LLM intent analysis.”
Planned evaluation: formal recall/precision measurement on larger corpus pending
Context & Problem
Code review at scale systematically misses non-obvious behavioral regressions. Reviewers read diffs, not behavior. Tests don't cover all cases — they cover the cases the original author thought to test. The gap between 'looks fine in the diff' and 'breaks in production' is the intent layer: what was this function supposed to guarantee, and does it still?
Challenges Addressed
Must work with existing GitHub PR workflows without requiring changes to the review process
Must not generate excessive false positives — regression alerts that are wrong destroy trust faster than missing regressions
Must support multiple languages without a separate implementation per language
Per-PR latency must stay under 30 seconds to be practical in a review workflow
Architecture
NoDiff has three layers. Structural layer: Tree-sitter parses both the before and after versions into ASTs. A structural differ walks the two trees and produces a ChangeSet of (old_node, new_node, change_type) tuples, tagged with semantic labels (function_body_change, parameter_removed, return_type_changed, etc.). Only changed function subtrees are passed to the next layers. Intent layer: Claude receives the changed subtree with surrounding context (function signature, docstring, call sites) and produces an IntentSummary: what did this function guarantee before, and what does it guarantee now? Regression layer: Claude receives the IntentSummary plus the ChangeSet and runs a chain-of-thought check: are any pre-change guarantees weakened? Are new edge cases exposed? Are invariants violated?
Stack
TypeScriptPythonTree-sitterClaude APIGitHub API
Technical Decisions
Decision
AST-level diffing over line-level diffs
Line diffs conflate formatting with logic changes. A rename-and-reformat can produce a large, noisy diff with no semantic change. AST diffs are semantics-only — a ChangeSet contains only structurally significant changes, providing higher-signal context to the LLM.
Decision
Two-stage reasoning: intent extraction before regression check
Asking 'what broke?' directly is underspecified — the model has no anchor for what should have been preserved. Extracting intent first ('what did this function guarantee?') provides explicit grounding for the regression check. This mirrors how experienced reviewers actually think.
Decision
Function-scoped analysis, not file-level
Full-file analysis is too slow for PR workflows (>60s for large files) and produces too much context for the LLM to reason carefully about. Scoping to changed functions keeps latency under 30s and focuses the model's attention on the highest-signal part of the diff.
Tradeoff
Analysis depth vs. latency
Chose: Function-scoped analysis with chain-of-thought
Full-file analysis with detailed chain-of-thought reasoning exceeds practical PR latency budgets. Scoping to changed functions makes the latency tractable while preserving the reasoning quality. Cross-file impact analysis (shared state, interface changes) is a known gap and a future work item.
Tradeoff
Recall vs. precision
Chose: Tuned toward false negatives over false positives
A regression detector that cries wolf on every PR will be ignored or disabled. In the prototype phase, tuning toward higher precision (fewer false positives) preserves trust even at the cost of missing some regressions. Planned formal evaluation will establish the actual recall/precision curve.
Future Work
01Formal recall/precision evaluation on a larger corpus of known regressions (planned evaluation)
02Cross-file impact analysis: trace shared state and interface changes across the full diff, not just changed functions
03CI/CD integration: block PRs when high-confidence regressions are detected; surface as GitHub check
04Feedback loop: learn from false positives via thumbs-down annotations to improve precision over time