Crawler🤖🕵️

Posted on by

Using Search to Solve Real Business Problems Through Intelligent Code Discovery

The Challenge: Code Knowledge Scattered Across Repositories

We had a critical business problem: our code knowledge was scattered across multiple repositories and files. When developers needed to understand a feature, troubleshoot an issue, or find where an API was used, they had to manually search through repositories, trace imports, and read through multiple files. This was time-consuming, error-prone, and made it difficult to understand code relationships and dependencies.

We needed a way to crawl code repositories and make everything searchable in one place. But more than that—we needed search to be intelligent enough to understand code structure, extract relationships between files, and surface insights that weren’t obvious when code lived in separate files.

We needed a crawler that didn’t just index—it understood code.

The Solution: Intelligent Code Repository Search Crawler

We built an intelligent code search crawler using Azure AI Search that crawls Azure DevOps repositories, understands code structure and relationships, and provides unified semantic search across all code files. It doesn’t just find code—it connects the dots, surfaces relationships, and solves real business problems.

Here’s how we did it.

The Architecture: Crawling Code Repositories, One Intelligent Index

Our crawler architecture:

Azure DevOps Git → Code Crawler → Azure AI Search → Intelligent Search Interface
     ↓                  ↓              ↓                      ↓
* Git Repos      • Code Parser    • Unified Index      • Semantic Search
* Code Files     • Relationship   • Vector Search      • Code Context
* Branches       • Extractor      • Metadata           • Dependencies
                 • Chunking       • Embeddings         • Answers

Each component works together to create an intelligent code search experience that solves real business problems.

Step 1: Code Repository Crawler

Our code crawler systematically indexes code repositories from Azure DevOps, extracting not just the code content but also relationships between files, imports, and dependencies.

The crawler clones the repository, parses each code file, extracts relationships (imports, API endpoints, database references), and indexes everything into Azure AI Search with rich metadata including file paths, programming languages, and code structure.

Key capabilities:

  • Crawls Azure DevOps Git repositories systematically
  • Parses multiple code file types (Python, TypeScript, JavaScript, C#, Java, etc.)
  • Extracts code structure and relationships
  • Indexes with metadata (file path, language, category)
  • Understands code structure to provide context

Step 2: Intelligent Code Relationship Extraction

The crawler doesn’t just index—it extracts relationships within code. When a code file is indexed, it automatically identifies imports, dependencies, and references to other files.

The parser detects the programming language and extracts language-specific relationships: Python imports, TypeScript/JavaScript imports, C# using statements, API endpoint definitions, and database references. Each relationship is mapped and stored, creating connections between code files.

Relationship types discovered:

  • Code files → Files they import
  • Code files → API endpoints they define
  • Code files → Database tables/queries they reference
  • Functions → Where they’re called
  • Classes → Where they’re instantiated

This creates a code knowledge graph connecting files and components, enabling powerful search capabilities that go beyond simple keyword matching.

Step 3: Business Problem Solving with Intelligent Code Search

We use the crawler to solve real business problems:

Problem 1: “Where is this function used?”

When developers need to understand where a function is called, the crawler searches across all code files using semantic search. It finds all occurrences, groups them by file, and generates insights about usage patterns.

Result: Find all usages in seconds instead of manually searching through files.

Problem 2: “What does this API endpoint do?”

The crawler searches for an API endpoint across all code files, finding both the implementation and where it’s called. It maps the data flow and generates insights about usage patterns and potential issues.

Result: Complete API understanding in seconds instead of hours of manual code reading.

Problem 3: “What’s the impact of changing this file?”

Before making changes, the crawler assesses impact by finding all files that import this file and all usages of exported functions and classes. It calculates a risk level based on the number of dependencies and provides warnings when changes could affect multiple systems.

Result: Understand change impact before making modifications, reducing breaking changes.

Step 4: Dual Assistant Modes – Same Question, Different Perspectives

We built two distinct assistant modes that serve different audiences, both powered by the same intelligent code crawler. The innovation here is that the same question can be answered from two completely different perspectives, each tailored to the user’s needs.

The interface allows users to switch between Developer Assistant and Business User Assistant modes

Developer Assistant Mode

The Developer Assistant helps developers understand code implementation details. When asked “How does the cart functionality work?”, it provides:

  • Specific file paths and line numbers (CartContext.tsx:45)
  • Code structure and implementation details (React Context, TypeScript interfaces)
  • Technical debugging information (useEffect hooks, localStorage persistence)
  • Code-focused answers with technical precision

Developer Assistant provides code-focused answers with file paths, interfaces, and implementation details

Example Developer Answer:

“The cart functionality is implemented using a React context called CartContext. It manages the cart state as an array of CartItem objects (which extend Product with a quantity field) and provides functions to add, remove, and update item quantities. The cart state is initialized from localStorage on component mount using useEffect to load saved data…”

Citation: CartContext.tsx

Business User Assistant Mode

The Business User Assistant helps non-technical users understand the application from a business perspective. When asked the same question “How does the cart functionality work?”, it provides:

  • Business logic explanations in plain language
  • User-facing feature descriptions (what users can do)
  • Workflow explanations (WHAT and WHY, not HOW)
  • Bug reporting assistance with Azure DevOps integration

Business User Assistant provides user-focused answers in plain language with a “Create Bug in DevOps” button

Example Business User Answer:

“From a user perspective, the shopping cart feature works as follows:

  • Users can add items to their cart while browsing the product catalog
  • They can remove items from the cart
  • They can update item quantities
  • A ‘save for later’ option is available
  • The cart automatically calculates prices, including applicable taxes
  • Cart management is accessible via a dedicated app section
  • The overall experience is mobile-friendly and responsive
  • Users proceed to secure checkout with Stripe integration

This creates an intuitive and seamless shopping experience where users can easily manage their selected products before completing their purchase.”

Citation: README.md

The Power of Dual Perspectives

Same Code, Different Explanations:

  • Developer asks: “How does the cart functionality work?”
  • Gets: Code structure, React Context implementation, TypeScript interfaces, localStorage persistence
  • Business User asks: “How does the cart functionality work?”
  • Gets: User workflows, feature descriptions, business value, plain language explanations

Both answers come from the same indexed code, but the AI agent adapts its explanation based on the selected mode. The Developer Assistant focuses on HOW (implementation), while the Business User Assistant focuses on WHAT and WHY (business logic and user experience).

When a business user reports an issue, the assistant:

  1. Explains WHAT is happening from a user perspective
  2. Explains WHY the issue is occurring (derived from code but explained in business terms)
  3. Identifies the root cause in business-friendly language
  4. Offers to create a work item in Azure DevOps to track the bug (via the “Create Bug in DevOps” button)

This dual-mode approach makes code knowledge accessible to both technical and non-technical users, bridging the gap between code and business understanding. The same crawler, the same indexed code, but two completely different ways of explaining it.

Step 5: Unified Code Search Interface

We built a unified search interface that searches across all code files intelligently. The interface supports semantic search, language filtering, relationship mapping, and provides code context with surrounding lines.

Search capabilities:

  • Semantic search: Finds related code, not just keywords
  • Language filtering: Filter by programming language
  • Relationship mapping: Shows how code files connect
  • Context-aware: Provides code context and surrounding lines
  • Dual modes: Developer-focused or business-friendly explanations

Step 6: Business Value Through Code Search Insights

The crawler provides business value through intelligent insights. It automatically analyzes search results to identify duplicate code patterns, unused functions, complex files, and other code quality issues, providing actionable recommendations.

The crawler automatically generates insights like:

  • Duplication: “Found similar code patterns. Consider refactoring.”
  • Unused code: “Found potentially unused functions. Review and remove.”
  • Complexity: “Found complex files. Consider breaking them down.”

Real-World Business Problems Solved

Problem: Developer Onboarding

Before: New developers spent weeks finding relevant code, understanding structure, and tracing dependencies manually.

After: Search crawler provides instant access to all code with semantic search. Developers can ask “How does authentication work?” and get relevant code files, relationships, and context.

Business Impact: Faster time-to-productivity, reduced onboarding costs.

Problem: Code Understanding and Maintenance

Before: Developers manually traced dependencies, often missing connections. Understanding a feature required reading through multiple files.

After: Search crawler automatically maps code relationships. Developers can search for a feature and see all related code, dependencies, and usage patterns.

Business Impact: Faster feature understanding, better code maintenance.

Problem: Change Impact Analysis

Before: Developers manually traced dependencies before making changes, often missing connections. Changes broke unexpected systems.

After: Search crawler automatically maps all dependencies and relationships. Impact analysis takes minutes instead of hours.

Business Impact: Reduced breaking changes, faster release cycles.

Problem: Business User Support and Bug Reporting

Before: Business users struggled to understand application behavior and report bugs effectively. They had to describe issues without understanding root causes, leading to miscommunication with development teams.

After: Business User Assistant mode helps users understand features from a business perspective, explains WHY issues occur in plain language, and seamlessly creates bug reports in Azure DevOps with proper context.

Business Impact: Faster bug resolution, better communication between business and development teams, reduced support tickets.

The Technical Architecture

┌─────────────────────────────────────────────────────────────┐
│                    DATA SOURCE                                 │
│  ┌──────────────────────────────────────────────────────┐    │
│  │           Azure DevOps Git Repository                 │    │
│  │  • Code Files    • Branches    • Commits             │    │
│  └──────────────────────┬───────────────────────────────┘    │
│                         │                                      │
│  ┌──────────────────────┴───────────────────────────────┐    │
│  │              CODE CRAWLER                              │    │
│  │  • Git Clone      • File Parser    • Code Parser     │    │
│  │  • Relationship   • Metadata       • Chunking        │    │
│  │    Extractor      Extractor        • Embeddings       │    │
│  └──────────────────────┬───────────────────────────────┘    │
└──────────────────────────┼───────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────┐
│              AZURE AI SEARCH                                │
│  • Unified Index    • Vector Search   • Semantic Search    │
│  • Code Metadata   • Relationships    • File Structure     │
└───────────────────────┬───────────────────────────────────┘
                         │
                         ▼
┌─────────────────────────────────────────────────────────────┐
│           INTELLIGENT CODE SEARCH INTERFACE                  │
│  • Semantic Search     • Relationship Mapping               │
│  • Code Context       • Dependency Tracking                 │
│  • Insight Generation • Business Problem Solving           │
└─────────────────────────────────────────────────────────────┘

What Makes This Innovative

1. Code Relationship Discovery

We don’t just index—we discover and map relationships between code files. Imports connect to files, functions link to callers, APIs connect to implementations. This creates a code knowledge graph that enables powerful search capabilities.

2. Semantic Code Understanding

Search results include semantic understanding, not just keyword matching. When you search for “authentication”, you find login functions, auth middleware, token validation—all semantically related code.

3. Business Problem Solving

The crawler doesn’t just search—it solves business problems. It finds function usages, assesses change impact, understands API endpoints, and generates insights automatically.

4. Intelligent Context

Search results include code context. When you search for a function, you see the function definition, where it’s called, related imports, and surrounding code—all in one place.

5. Dual-Mode Accessibility

The crawler serves both technical and non-technical users. Developers get code-focused answers with file paths and implementation details. Business users get business-friendly explanations that translate code into understandable workflows and help create actionable bug reports.

Real-World Use Cases

Use Case 1: Finding Function Usage

Scenario: A developer needs to refactor a function but needs to know where it’s used.

How crawler helps:

  1. Search for function name → Finds all occurrences
  2. Filter by usage vs definition → Shows where it’s called
  3. Map relationships → Shows call chain
  4. Generate insights → Identifies refactoring impact

Result: Complete usage understanding in seconds.

Use Case 2: Understanding API Endpoints

Scenario: A developer needs to understand what an API endpoint does and how it’s used.

How crawler helps:

  1. Search for endpoint path → Finds implementation
  2. Find callers → Shows where it’s used
  3. Map relationships → Shows data flow
  4. Generate insights → Provides usage patterns

Result: Complete API understanding without reading multiple files.

Use Case 3: Change Impact Analysis

Scenario: Before changing a file, a developer needs to understand the impact.

How crawler helps:

  1. Search for file → Finds all references
  2. Map dependencies → Shows what depends on it
  3. Assess risk → Calculates impact level
  4. Generate insights → Provides warnings and recommendations

Result: Understand impact before making changes, reducing risk.

Use Case 4: Business User Bug Reporting

Scenario: A business user encounters an issue when adding items to cart and wants to report it.

How crawler helps:

  1. User asks “Why might a user see an error when adding items to cart?” → Business User Assistant searches code
  2. Assistant explains WHAT is happening (user perspective) and WHY (business logic derived from code)
  3. Assistant identifies root cause in business-friendly terms
  4. User clicks “Create Bug in DevOps” → Assistant creates work item in Azure DevOps with proper context
  5. Development team receives bug report with business context and technical details

Result: Faster bug resolution, better communication, reduced back-and-forth between teams.

The Technical Innovation

Vector Search for Semantic Code Understanding

We use vector embeddings to understand code meaning, not just keywords. This allows searches like “authentication logic” to find:

  • Login functions
  • Token validation code
  • Auth middleware
  • All semantically related, not just keyword matches

Code Relationship Graph

We build a relationship graph connecting code files. This graph enables:

  • Impact analysis
  • Dependency tracking
  • Code discovery
  • System understanding

Intelligent Chunking

Code files are intelligently chunked to preserve context:

  • Functions are kept together
  • Classes include their methods
  • Imports are preserved
  • Comments provide context

The Business Impact

Before Crawler

  • Code scattered: Code across multiple repositories and files
  • Slow discovery: 10-30 minutes to find relevant code
  • Missed relationships: Dependencies between files unknown
  • Manual work: Hours spent tracing dependencies
  • Knowledge gaps: Code exists but can’t be found

After Crawler

  • Unified search: One interface searches all code
  • Fast discovery: Seconds to find relevant code
  • Intelligent connections: Relationships automatically mapped
  • Automated insights: Code insights generated automatically
  • Knowledge accessible: All code searchable and connected

What’s Next

The crawler opens new possibilities:

  • Real-time indexing: Index code changes as they happen
  • Code quality insights: Identify code smells and anti-patterns
  • Automated documentation: Generate docs from code and relationships
  • Intelligent refactoring: Suggest improvements based on patterns
  • Visual code graphs: Visual representation of code relationships

But the foundation is set: we have an intelligent code crawler that solves real business problems through innovative search.

This crawler demonstrates the power of using search innovatively. By crawling code repositories, extracting relationships, and generating insights, we’ve created a solution that doesn’t just find code—it solves business problems. One search. All code. Infinite possibilities.