Overview

Chapter 05: Agentic Coding, Context Gathering & Tree-sitter AST Mechanics

Playbook Track: 04 – AI Coding & Software Engineering (AI-DLC & Autonomous Developer Workflows)
Target Audience: Year 1 Computer Science & Software Engineering Students Core Tooling Stack: Gemini 2.5 Flash, Tree-sitter / Python AST, Unified Diff (diff -u), Model Context Protocol (MCP), Python 3.11+
Delivery Status: 🔍 Ready for Review (Tier 1 Markdown)


1. The Big Picture & Real-World Analogy

The Building Blueprint Index vs. Reading the Whole Library

Imagine you are hired as a plumbing technician to fix a leaky faucet in Apartment 4B of a 50-story skyscraper:

  • The Naive Disaster Approach: You carry 50 thick binders containing 10,000 pages of architectural blueprints for the entire building—electrical, elevator wiring, foundation concrete, roofing. You spend 6 hours reading every page, get completely overwhelmed, and when you finally try to fix the sink, you accidentally smash the wall and replace the entire kitchen counter!
  • The Surgical Engineering Approach: You open the Building Index. In 5 seconds, you locate "Building A -> Floor 4 -> Apartment 4B Faucet Schematic". You look at the 1-page drawing, grab a 2-inch rubber washer, swap the damaged washer, and test the faucet. The repair takes 5 minutes and leaves the rest of the apartment untouched.

In AI coding, this is the difference between amateur prompt dumping and professional AST & Diff Engineering:

  1. If you dump 50 full Python files into an AI's prompt, the attention heads get overwhelmed (Lost-in-the-Middle phenomenon), the AI forgets edge cases, and your API bill explodes!
  2. If you ask an AI to rewrite a full 600-line file to change 3 lines of code, it will often accidentally erase working helper functions or delete comments!

Instead, we use:

  • AST (Abstract Syntax Tree) Repo Maps: A compact table of contents showing only class and function signatures without implementation bodies (cutting token usage by 85%).
  • Unified Diff Patches (diff -u): The AI emits surgical line-by-line patches (--- / +++) that modify only the exact lines being changed.

2. Engineering Jargon Demystifier Table

Industry Term What It Actually Means Freshman Student Analogy
AST (Abstract Syntax Tree) A tree representation of source code structure created by a compiler (nodes for functions, loops, if-statements). A sentence diagram in English class showing subject, verb, and direct objects.
Tree-sitter An ultra-fast, incremental parser library that generates ASTs for almost any programming language in milliseconds. An instant X-ray machine for source code.
Repo-Map (Repository Map) A condensed text outline of a codebase showing filenames, class names, and function signatures without the code bodies. The table of contents at the beginning of a textbook.
Unified Diff (diff -u) The standard format used by Git to represent changes: lines with - are deleted, lines with + are added. A professor's red-pen markup on your essay showing exact words to delete or insert.
Blast Radius The scope of code affected by a change. A small blast radius touches 1 function; a large blast radius touches 20 files. Replacing a single lightbulb (small blast radius) vs turning off the main circuit breaker (large blast radius).
In-Memory AST Guard Parsing modified code in memory with ast.parse() before saving it to disk to prevent syntax errors. Checking for spelling errors before clicking "Send Email".
Lost-in-the-Middle The tendency of LLMs to pay attention to the beginning and end of long prompts while ignoring details in the middle. Skimming a 50-page reading and only remembering the first and last pages.

3. The 5-Minute Micro-Lab: The AST Symbol Extractor

Python has a built-in ast module! Run this script to see how easy it is to inspect code structure without executing it:

"""
Micro-Lab: Python AST Symbol Extractor
PB-04 Chapter 5 Micro-Lab (Zero External Dependencies)
"""
import ast

sample_code = """
class PaymentGateway:
    def __init__(self, api_key: str):
        self.api_key = api_key

    def process_refund(self, charge_id: str, amount_cents: int) -> bool:
        # Implementation hidden
        return True

def calculate_tax(subtotal: float, rate: float = 0.08) -> float:
    return subtotal * rate
"""

def extract_symbols(source: str) -> list:
    tree = ast.parse(source)
    symbols = []
    for node in tree.body:
        if isinstance(node, ast.ClassDef):
            methods = [n.name for n in node.body if isinstance(n, ast.FunctionDef)]
            symbols.append(f"CLASS {node.name} -> Methods: {methods}")
        elif isinstance(node, ast.FunctionDef):
            args = [a.arg for a in node.args.args]
            symbols.append(f"FUNCTION {node.name}({', '.join(args)})")
    return symbols

if __name__ == "__main__":
    print("=== Extracting Symbols via Python AST ===")
    outlines = extract_symbols(sample_code)
    for sym in outlines:
        print(f"  [SYMBOL]: {sym}")

4. System Architecture & AST Repo-Map Pipeline

In early conversational AI coding tools, context management was treated as a brute-force text dumping problem: developers or IDE plugins dumped dozens of full source files into the prompt, hoping the LLM's multi-hundred-thousand-token context window would magically locate the right variables.

In practice, this approach fails due to fundamental transformer attention dynamics: The Lost-in-the-Middle Phenomenon and Attention Dilution. When an LLM is flooded with 80,000 tokens of irrelevant boilerplate code, its ability to reason accurately about precise variable mutations and edge cases drops precipitously. Furthermore, asking an LLM to output a full 500-line file when only a 3-line bugfix was required leads to catastrophic latency, high API costs, and silent regressions where unprompted helper functions are erased.

Production-grade AI-DLC engineering treats codebase interaction as a compiler problem:

  1. Precision Context Gathering: Rather than ingesting full files, agents query Tree-sitter AST Symbol Graphs (Repo Maps), which extract symbol hierarchies (classes, methods, arguments, return types) without implementation bodies.
  2. Unified Diff Patching: Agents never write entire files. They emit strictly bounded Unified Diff Patches (diff -u).
  3. In-Memory AST Verification: Patches are applied to in-memory AST buffers and parsed with ast.parse() to certify grammatical validity before anything touches the filesystem.
+---------------------------------------------------------------------------------------------------+
|                              AST REPO MAP & DIFF PATCHING PIPELINE                                |
+---------------------------------------------------------------------------------------------------+
|                                                                                                   |
|   +--------------------------+         +--------------------------+                               |
|   |  SOURCE CODE REPOSITORY  | ------> |  TREE-SITTER AST PARSER  |                               |
|   |  - 50,000+ Lines of Code |         |  - Symbol Call Graph     |                               |
|   |  - Multiple Packages     |         |  - Class/Method Outlines |                               |
|   +--------------------------+         +--------------------------+                               |
|                                                      |                                            |
|                                                      v                                            |
|   +-------------------------------------------------------------------------------------------+   |
|   |               COMPACT REPO MAP (85% Token Reduction, Signatures Only)                     |   |
|   +-------------------------------------------------------------------------------------------+   |
|                                                      |                                            |
|                                                      v                                            |
|   +--------------------------+         +--------------------------+                               |
|   |  LEAD DEVELOPER AGENT    | ------> |    UNIFIED DIFF PATCH    |                               |
|   |  - Gemini 2.5 Flash      |         |    - Exact Line Hunks    |                               |
|   |  - Target Symbol Focus   |         |    - diff -u Format      |                               |
|   +--------------------------+         +--------------------------+                               |
|                                                      |                                            |
|                                                      v                                            |
|   +-------------------------------------------------------------------------------------------+   |
|   |                        IN-MEMORY TWO-PHASE PATCH & AST GUARD                              |   |
|   |  Phase 1: Apply unified diff hunk in memory buffer                                        |   |
|   |  Phase 2: Validate syntax with ast.parse() -> ABORT if SyntaxError                        |   |
|   |  Phase 3: Commit atomic write to disk only upon 100% clean AST parse                      |   |
|   +-------------------------------------------------------------------------------------------+   |
|                                                                                                   |
+---------------------------------------------------------------------------------------------------+

The Autonomous TDD Red-Green-Refactor Loop

sequenceDiagram
    autonumber
    participant RM as AST Repo Mapper
    participant Dev as Lead Developer Agent
    participant Patcher as Unified Diff Patcher
    participant AST as AST Syntax Guard
    participant Disk as Repository Disk
    participant Test as Pytest Harness

    RM->>Dev: Inject compact Repo Map (signatures only)
    Dev->>Dev: Formulate Test-Driven implementation plan
    Dev->>Patcher: Emit Unified Diff patch for test suite (RED)
    Patcher->>AST: Validate test file AST integrity
    AST->>Disk: Commit test file
    Disk->>Test: Run pytest -> Verify failure (RED confirmed)
    Dev->>Patcher: Emit Unified Diff patch for source code (GREEN)
    Patcher->>AST: Validate source file AST integrity
    alt Syntax Error Detected
        AST-->>Dev: Rejection (ASTSyntaxCorruptionError at line X)
        Dev->>Patcher: Emit corrected diff hunk
    end
    AST->>Disk: Commit atomic source write
    Disk->>Test: Run pytest -> Verify pass (GREEN confirmed)
    Dev->>Dev: Refactor & optimize AST complexity


5. Freshman Survival Guide: 3 Traps to Avoid

Trap 1: Full-File Rewriting Anti-Pattern

  • The Mistake: Asking an AI: "Here is my 500-line server.py. Please add input validation to the login route and return the full file."
  • Why it fails: The model will almost certainly truncate long files midway, hallucinate different variable names in other routes, or delete auxiliary helper functions.
  • Fix: Instruct the model to return a Unified Diff Patch (--- server.py / +++ server.py) or use targeted SEARCH/REPLACE blocks.

Trap 2: Context Window Flooding

  • The Mistake: Concatenating all 30 files in your repository into one giant prompt.
  • Why it fails: When prompt context exceeds 30,000 tokens, LLMs suffer from the Lost-in-the-Middle effect: the model overlooks subtle variable types and hallucinated imports slip through.
  • Fix: Feed the model a condensed AST Repo-Map (signatures only) plus ONLY the 1 or 2 specific files directly relevant to the task.

Trap 3: Direct-to-Disk Overwriting Without AST Pre-Flight

  • The Mistake: Taking raw AI generated code and writing it directly to disk (open("app.py", "w").write(ai_text)) without validation.
  • Why it fails: If the AI omitted a closing parenthesis or messed up indentation, your application will crash immediately on startup.
  • Fix: Always parse code in memory with ast.parse() first. If it raises SyntaxError, reject the patch and ask the AI to self-correct before saving.

6. Naive vs. Production Contrasts

The table below contrasts naive conversational code editing with production AST-verified Unified Diff patching:

Dimension Naive Conversational Code Editing (Anti-Pattern) Production AST-Verified Diff Patching (Production Standard)
Context Strategy Stuff entire files into prompt until context window overflows. High-density Tree-sitter Repo Map containing symbol signatures and PageRank weights.
Code Modification Rewrites entire 500-line file to change 3 lines of logic. Bounded Unified Diff (@@ -12,4 +12,5 @@) modifying only the relevant AST hunk.
Token Cost per Edit 25,000 - 60,000 tokens per modification. 2,500 - 5,000 tokens (85% - 90% reduction).
Accidental Erasure Risk High; agent frequently omits unprompted helper methods or comments. Zero; unified diffs cannot erase code outside their explicit hunk boundaries.
Syntax Error Defense None; invalid code is written directly to disk, breaking the build. Two-Phase AST Guard (ast.parse()); bad syntax is rejected before disk write.
Git Commit Hygiene Massive noisy 500-line diffs where 99% of lines are identical formatting. Clean, microscopic 5-10 line semantic diffs easily audited by human reviewers.
Edit Latency 30 - 60 seconds generating repetitive tokens. 2 - 5 seconds generating only diff lines via Gemini 2.5 Flash.

7. Frontier Model Configurations & MCP Tool Schemas

Agentic coding demands high speed and strict adherence to diff formats. Gemini 2.5 Flash with ultra-low temperature (0.05) is calibrated specifically for code diff generation.

Lead Developer Agent Calibration

LEAD_DEVELOPER_AGENT_CONFIG = {
    "model": "gemini-2.5-flash",
    "temperature": 0.05,
    "top_p": 0.85,
    "max_output_tokens": 16384,
    "system_instruction": """You are the Lead Developer Agent in an autonomous AI-DLC team.
Your mission:
1. Review the provided AST Repo Map and contract schemas.
2. Formulate a minimal, surgically precise implementation plan.
3. Emit strictly formatted Unified Diffs (diff -u format) with exact hunk headers.
4. Never emit whole-file rewrites. Never output stubs, empty bodies, or // TODO comments.
5. Preserve existing function signatures, comments, and docstrings outside your hunk."""
}

Model Context Protocol (MCP) Code Editing Schemas

{
  "tools": [
    {
      "name": "get_repository_symbol_map",
      "description": "Extracts compact Tree-sitter AST symbol signatures and class outlines for the codebase.",
      "parameters": {
        "type": "object",
        "properties": {
          "directory": {"type": "string", "description": "Target source directory path."},
          "max_depth": {"type": "integer", "default": 4}
        },
        "required": ["directory"]
      }
    },
    {
      "name": "apply_unified_diff",
      "description": "Applies a unified diff patch to a target file with pre-commit AST syntax validation.",
      "parameters": {
        "type": "object",
        "properties": {
          "file_path": {"type": "string", "description": "Relative target file path."},
          "diff_content": {"type": "string", "description": "Unified diff chunk (@@ -x,y +x,y @@)."},
          "verify_ast": {"type": "boolean", "default": true}
        },
        "required": ["file_path", "diff_content"]
      }
    }
  ]
}

4. Quantitative Trade-Off Matrix: Code Editing Strategies

Code Editing Strategy Token Overhead Patch Failure Rate Multi-File Editing Speed AST Syntax Safety Human Auditability
Full File Overwrite Disastrous (50K+ tokens) 0% (Blind write) Slow (45s per file) None (Blind disk write) Unusable (Huge noisy diffs)
Search-and-Replace Blocks Moderate (15K tokens) 12% (Mismatched whitespace) Moderate (15s) Low Moderate
Unified Diff (diff -u) Minimal (< 4K tokens) < 2% (Offset tolerance) Fast (< 4s) High (With AST Guard) Exceptional (Standard Git)
AST Tree Transformation (LibCST) Moderate (8K tokens) 5% (Grammar edge cases) Fast (< 5s) Absolute (Enforced AST) High

5. The 10 Operational Failure Modes in Agentic Coding & Patching

1. Line Offset Drift

  • Mechanism: The agent calculates unified diff headers based on an outdated view of the file (@@ -45,6 +45,8 @@), causing patch rejections when previous edits shifted line counts.
  • Defense Mechanism: Reverse Hunk Application. Apply diff hunks in reverse line order (bottom-to-top), so line additions/deletions at the bottom of the file do not alter the line offsets of hunks above them.

2. Whitespace & Indentation Mismatch

  • Mechanism: Python requires strict indentation. The agent emits a replacement block with 2 spaces instead of 4, causing IndentationError when patched.
  • Defense Mechanism: Pre-Commit AST Syntax Validation (ast.parse()). Any indentation or syntax failure raises ASTSyntaxCorruptionError, preventing disk writes and giving the agent immediate line-numbered feedback.

3. Accidental Sibling Function Deletion

  • Mechanism: Agent is asked to modify method calculate_tax(); it emits code that replaces both calculate_tax() and the adjacent unmentioned calculate_discount().
  • Defense Mechanism: Bounded Diff Enforcement. Reject any diff that contains deletion markers (-) for function signatures not explicitly targeted in the task prompt.

4. Undeclared Symbol Import Hallucination

  • Mechanism: Agent calls datetime.utcnow() or uuid.uuid4() without adding the corresponding import datetime or import uuid at the top of the file.
  • Defense Mechanism: AST Undefined Symbol Linter. Walk the AST with ast.walk(), collect all ast.Name(ctx=ast.Load()) nodes, and verify they exist in the module's local or imported symbol table.

5. Mutating Read-Only Ports / Domain Interfaces

  • Mechanism: When experiencing difficulty implementing an adapter, the agent edits the abstract interface port to fit its broken adapter, violating hexagonal separation.
  • Defense Mechanism: Directory Permission Mutex. Core domain and interface directories (src/domain/ports/) are marked read-only during adapter implementation tasks.

6. The Lazy Stub Hallucination (// TODO: Implement later)

  • Mechanism: Agent outputs method signatures with empty bodies or placeholder comments.
  • Defense Mechanism: AST Stub Detector. Reject any function body composed solely of ast.Pass, ast.Constant(value=...) (docstrings only), or ast.Raise(exc=NotImplementedError).

7. Infinite Patch Retry Thrashing

  • Mechanism: Agent repeatedly attempts to patch a file, failing each time with the same offset error and consuming thousands of tokens.
  • Defense Mechanism: Max Retry Budget (3 attempts) combined with exponential backoff. If 3 consecutive diff attempts fail, fall back to search-and-replace block matching.

8. Corrupted Multi-Line Docstrings

  • Mechanism: Agent terminates a docstring with unmatched quotes or broken indentation.
  • Defense Mechanism: AST Docstring Linter verifying ast.get_docstring(node) parses successfully.

9. Typo-Squatted Internal Module Imports

  • Mechanism: Agent imports from services.refund import RefundEngine instead of from services.refunds import RefundEngine.
  • Defense Mechanism: Import Path Resolution Checker. Verify that every ast.ImportFrom path resolves to a physical file on disk before accepting the patch.

10. Commenting Out Existing Code Rather Than Updating

  • Mechanism: Agent leaves 50 lines of commented-out legacy code above its new 5-line implementation.
  • Defense Mechanism: Dead Code Linter. Flag diffs that introduce blocks of commented-out code larger than 3 lines.

10. Mandatory Hands-On Lab: AST Repo Mapper & Unified Diff Patcher

Lab Objective

In this hands-on lab, you will:

  1. Parse a Python source file using ASTSymbolRepoMapper to extract classes, methods, and functions into a structured symbol graph.
  2. Generate a compact Repo Map and quantify token savings compared to raw source text.
  3. Apply a valid Unified Diff Patch using UnifiedDiffPatcher, verifying that modifications are applied in memory and certified by ast.parse().
  4. Submit an intentionally corrupt diff that breaks Python syntax, and observe ASTSyntaxCorruptionError blocking the patch before disk write.

Lab Step-by-Step Instructions

Step 1: Initialize the AST Symbol Repo Mapper

Pass a sample PaymentProcessor class and calculate_tax function into ASTSymbolRepoMapper.parse_symbols(). Verify that classes, methods, and functions are categorized into SymbolNode objects.

Step 2: Generate the Compact Repo Map

Invoke generate_repo_map(). Verify that the output contains clean signatures (def process_refund(...) -> ...) without method bodies.

Step 3: Apply a Valid Unified Diff Patch

Pass a unified diff hunk that updates process_refund to return status: 'succeeded'. Run UnifiedDiffPatcher.apply_patch() with verify_ast=True and assert that the returned string contains the new dictionary keys.

Step 4: Test Syntax Corruption Rejection

Construct a diff with a missing closing parenthesis. Attempt to apply it; verify that the patcher raises ASTSyntaxCorruptionError and preserves original code integrity.

Step 5: Execute Self-Test Verification

Run the built-in unit test suite to certify 100% compliance with zero external dependencies.


12. Summary & Next Steps

This chapter established agentic coding mechanics for Playbook 04:

  • Solved context window bloat using Tree-sitter AST Symbol Graphs (Repo Maps).
  • Banished whole-file rewrites in favor of surgical Unified Diff Patching.
  • Instituted the Two-Phase AST Guard to prevent syntax corruption before disk commit.
  • Delivered and verified the zero-dependency Python 3.11+ ASTSymbolRepoMapper & UnifiedDiffPatcher.

Upcoming Chapters in Playbook 04:

  • Chapter 06: Autonomous Verification, Testing & Self-Healing Code Loops.
  • Chapter 07: Automated CI/CD, GitOps & Agentic Review Workflows.
  • Chapter 08: End-to-End Autonomous Software Engineering Suite.