AI GitHub Knowledge Base: Codebase Intelligence with RAG and Graph

TL;DR — AI GitHub knowledge base transforms code repositories into queryable knowledge graphs using RAG and AST analysis. CodeRAG: "Creates a semantic vector database (RAG) from your source code, then exposes it as MCP tools that give AI agents deep understanding of your entire codebase." CogniRepo: "On Python repos >= 15K LOC, cuts AI coding agent token usage by 50-80% compared to raw file reads. Works with Claude Code, Cursor, and Gemini CLI. Fully offline." VortexRAG: "Integrates vector storage (Qdrant) with structural AST call-graphs (Neo4j) — understands not only what code exists, but how classes, methods, and decorators communicate." Central Code Knowledge Graph: "One Neo4j graph spans all registered repos — Functions, classes, files, imports, calls stored as labeled nodes + typed relationships." Key patterns: AST chunking (tree-sitter, 15-26 languages), GraphRAG (Neo4j call graphs), Hybrid search (vector + BM25 with Reciprocal Rank Fusion), MCP tools (coderag_search, context_pack, lookup_symbol), Incremental sync (Git diffs, 1.8s delta, webhooks). Integrate with Google Drive, company knowledge, Nango, and RAG pipeline.

CogniRepo states the problem: "AI coding tools re-read your entire codebase on every task. CogniRepo fixes that. One server indexes every repo in your org with Tree-sitter across 26 languages, stores the structural map as a Neo4j property graph, keeps it fresh via incremental ingest + webhooks, and serves precise context to your AI assistant via MCP so it reads only what matters."

GitHub Smart Agent describes the capability: "Turn any repository into an interactive AI assistant for code understanding, debugging, and optimization. Instead of reading files one-by-one, you can ask architecture, bug, and optimization questions and get context-aware answers grounded in your code."

Codebase Intelligence Architecture

AST Chunking vs Line Chunking

GraphRAG Query Types

Implementation

import tree_sitter
from tree_sitter import Parser
import hashlib

class GitHubKnowledgeBase:
    """AI knowledge base for GitHub repositories with AST + GraphRAG."""

    SUPPORTED_LANGUAGES = {
        "python": "tree_sitter_python",
        "javascript": "tree_sitter_javascript",
        "typescript": "tree_sitter_typescript",
        "go": "tree_sitter_go",
        "rust": "tree_sitter_rust",
        "java": "tree_sitter_java",
        "c": "tree_sitter_c",
        "cpp": "tree_sitter_cpp",
    }

    FUNCTION_NODE_TYPES = {
        "python": ["function_definition", "decorated_definition"],
        "javascript": ["function_declaration", "method_definition", "arrow_function"],
        "typescript": ["function_declaration", "method_definition", "arrow_function"],
        "go": ["function_declaration", "method_declaration"],
    }

    def __init__(self, vector_db, neo4j, embedding_model, llm, audit):
        self.vector_db = vector_db
        self.neo4j = neo4j
        self.embeddings = embedding_model
        self.llm = llm
        self.audit = audit
        self.parsers = {}

    async def ingest_repository(self, tenant_id: str, repo_url: str,
                                branch: str = "main") -> dict:
        """Ingest a GitHub repository into the knowledge base."""
        # 1. Clone repository
        repo_path = await self._clone_repo(repo_url, branch)

        # 2. Get last indexed commit (for incremental sync)
        last_commit = await self._get_last_commit(tenant_id, repo_url)
        current_commit = await self._get_current_commit(repo_path)

        if last_commit == current_commit:
            return {"status": "up_to_date", "commit": current_commit}

        # 3. Get changed files (Git diff)
        if last_commit:
            changed_files = await self._get_changed_files(repo_path, last_commit)
            # Delete old vectors and graph nodes for changed files
            for file_path in changed_files:
                await self._delete_file_vectors(tenant_id, repo_url, file_path)
                await self._delete_file_graph_nodes(tenant_id, repo_url, file_path)
        else:
            changed_files = await self._list_all_files(repo_path)

        # 4. Parse and chunk each changed file
        chunks_created = 0
        for file_path in changed_files:
            language = self._detect_language(file_path)
            if language not in self.SUPPORTED_LANGUAGES:
                continue

            chunks = await self._parse_file(repo_path, file_path, language)
            for chunk in chunks:
                await self._store_chunk(tenant_id, repo_url, file_path, chunk)
                chunks_created += 1

            # Build graph nodes and edges
            await self._build_graph(tenant_id, repo_url, file_path, chunks)

        # 5. Update commit hash
        await self._update_commit(tenant_id, repo_url, current_commit)

        # 6. Audit log
        await self.audit.log(
            tenant_id=tenant_id,
            action="github_repo_ingested",
            entity_type="repository",
            entity_id=repo_url,
            after={
                "commit": current_commit,
                "files_processed": len(changed_files),
                "chunks_created": chunks_created,
            },
        )

        return {
            "status": "ingested",
            "commit": current_commit,
            "files_processed": len(changed_files),
            "chunks_created": chunks_created,
        }

    async def _parse_file(self, repo_path: str, file_path: str,
                          language: str) -> list:
        """Parse a source file using tree-sitter AST."""
        parser = self._get_parser(language)

        with open(f"{repo_path}/{file_path}", "rb") as f:
            source_code = f.read()

        tree = parser.parse(source_code)

        # Extract semantic chunks (functions, classes, methods)
        chunks = []
        function_types = self.FUNCTION_NODE_TYPES.get(language, [])

        def traverse(node, class_name=None):
            if node.type in function_types:
                # Extract function/method chunk
                name_node = node.child_by_field_name("name")
                func_name = source_code[name_node.start_byte:name_node.end_byte].decode() if name_node else "anonymous"

                chunk_text = source_code[node.start_byte:node.end_byte].decode()

                # Generate natural language summary
                summary = await self._generate_summary(chunk_text, func_name, language)

                chunks.append({
                    "type": "function",
                    "name": func_name,
                    "class": class_name,
                    "file": file_path,
                    "start_line": node.start_point[0] + 1,
                    "end_line": node.end_point[0] + 1,
                    "text": chunk_text,
                    "summary": summary,
                    "language": language,
                })

            elif node.type == "class_definition" or node.type == "class_declaration":
                name_node = node.child_by_field_name("name")
                cls_name = source_code[name_node.start_byte:name_node.end_byte].decode() if name_node else "anonymous"

                chunks.append({
                    "type": "class",
                    "name": cls_name,
                    "file": file_path,
                    "start_line": node.start_point[0] + 1,
                    "end_line": node.end_point[0] + 1,
                    "text": source_code[node.start_byte:node.end_byte].decode(),
                    "language": language,
                })

                # Traverse children with class context
                for child in node.children:
                    traverse(child, class_name=cls_name)
            else:
                for child in node.children:
                    traverse(child, class_name=class_name)

        traverse(tree.root_node)
        return chunks

    async def _store_chunk(self, tenant_id: str, repo_url: str,
                           file_path: str, chunk: dict):
        """Store a code chunk in vector database with embedding."""
        # Combine code + summary for embedding
        embed_text = f"{chunk['summary']}\n\n{chunk['text']}"
        embedding = await self.embeddings.embed(embed_text)

        content_hash = hashlib.sha256(chunk["text"].encode()).hexdigest()

        await self.vector_db.insert({
            "tenant_id": tenant_id,
            "repo_url": repo_url,
            "file_path": file_path,
            "chunk_type": chunk["type"],
            "chunk_name": chunk["name"],
            "class_name": chunk.get("class"),
            "start_line": chunk["start_line"],
            "end_line": chunk["end_line"],
            "text": chunk["text"],
            "summary": chunk["summary"],
            "language": chunk["language"],
            "embedding": embedding,
            "content_hash": content_hash,
        })

    async def _build_graph(self, tenant_id: str, repo_url: str,
                           file_path: str, chunks: list):
        """Build Neo4j knowledge graph nodes and edges."""
        for chunk in chunks:
            if chunk["type"] == "class":
                await self.neo4j.execute(
                    "CREATE (c:Class {tenant_id: $tenant, repo: $repo, "
                    "file: $file, name: $name, line: $line})",
                    tenant=tenant_id, repo=repo_url, file=file_path,
                    name=chunk["name"], line=chunk["start_line"],
                )
            elif chunk["type"] == "function":
                await self.neo4j.execute(
                    "CREATE (f:Function {tenant_id: $tenant, repo: $repo, "
                    "file: $file, name: $name, class: $class, line: $line})",
                    tenant=tenant_id, repo=repo_url, file=file_path,
                    name=chunk["name"], class=chunk.get("class"),
                    line=chunk["start_line"],
                )

        # Create CONTAINS edges (file → functions/classes)
        await self.neo4j.execute(
            "MATCH (f:File {tenant_id: $tenant, path: $path}), "
            "(s {tenant_id: $tenant, file: $path}) "
            "CREATE (f)-[:CONTAINS]->(s)",
            tenant=tenant_id, path=file_path,
        )

    async def hybrid_search(self, tenant_id: str, query: str,
                            top_k: int = 10) -> list:
        """Hybrid search: vector similarity + BM25 with Reciprocal Rank Fusion."""
        # Vector search
        query_embedding = await self.embeddings.embed(query)
        vector_results = await self.vector_db.search(
            embedding=query_embedding,
            filter={"tenant_id": tenant_id},
            top_k=top_k * 3,
        )

        # BM25 keyword search
        keyword_results = await self.vector_db.bm25_search(
            query=query,
            filter={"tenant_id": tenant_id},
            top_k=top_k * 3,
        )

        # Reciprocal Rank Fusion
        fused = self._reciprocal_rank_fusion(vector_results, keyword_results)

        return fused[:top_k]

    def _reciprocal_rank_fusion(self, vector_results: list,
                                keyword_results: list, k: int = 60) -> list:
        """Fuse ranked lists using Reciprocal Rank Fusion."""
        scores = {}

        for rank, result in enumerate(vector_results):
            chunk_id = result["id"]
            scores[chunk_id] = scores.get(chunk_id, 0) + 1 / (k + rank + 1)

        for rank, result in enumerate(keyword_results):
            chunk_id = result["id"]
            scores[chunk_id] = scores.get(chunk_id, 0) + 1 / (k + rank + 1)

        # Sort by fused score
        all_results = {r["id"]: r for r in vector_results + keyword_results}
        sorted_ids = sorted(scores, key=scores.get, reverse=True)

        return [
            {**all_results[chunk_id], "fused_score": scores[chunk_id]}
            for chunk_id in sorted_ids
        ]

MCP Tools for AI Agents

AI GitHub Knowledge Base Checklist

• [ ] Set up GitHub App or OAuth for repository access
• [ ] Configure webhook for push events (trigger incremental sync)
• [ ] Clone/fetch repositories via git
• [ ] Track last indexed commit hash per repository
• [ ] Implement incremental sync using git diff (only changed files)
• [ ] Delete old vectors and graph nodes for changed files before re-indexing
• [ ] Install tree-sitter with grammars for 15-26 programming languages
• [ ] Parse source files using tree-sitter AST (not line-based chunking)
• [ ] Chunk by semantic units: functions, classes, methods
• [ ] Preserve metadata: name, class, file path, line range, parameters
• [ ] Generate natural language summaries for each chunk
• [ ] Combine code + summary for embedding (better semantic matching)
• [ ] Store embeddings in vector database (Qdrant, pgvector, Pinecone)
• [ ] Build BM25 sparse index for keyword search
• [ ] Implement Reciprocal Rank Fusion for hybrid search
• [ ] Build Neo4j knowledge graph: File, Class, Function nodes
• [ ] Create graph edges: calls, imports, contains
• [ ] Implement graph queries: callers_of, callees_of, blast_radius
• [ ] Implement entry_points query (most-connected functions)
• [ ] Expose MCP tools: coderag_search, context_pack, lookup_symbol
• [ ] Support MCP stdio transport for local agents (Claude Code, Cursor)
• [ ] Support MCP SSE transport for remote agents
• [ ] Implement context_pack with token budget management
• [ ] Implement lookup_symbol for O(1) symbol → file + line lookup
• [ ] Implement who_calls for caller tracing and impact analysis
• [ ] Support multi-repo indexing (one graph spans all repos)
• [ ] Index documentation (Markdown, Confluence, SharePoint)
• [ ] Integrate with project backlog (Jira, ClickUp, Azure DevOps)
• [ ] Connect to company knowledge RAG pipeline
• [ ] Integrate with RAG to actuation pipeline
• [ ] Use Nango for GitHub OAuth management
• [ ] Log all ingestion and query events in audit logs
• [ ] Apply RBAC to repository access
• [ ] Apply zero-trust architecture to code data
• [ ] Apply OWASP LLM Top 10 security controls
• [ ] Set up platform monitoring for ingestion
• [ ] Monitor: files indexed, sync latency, query latency, graph size
• [ ] Handle large repositories (15K+ LOC) with efficient parsing
• [ ] Implement retry logic for git operations and API failures
• [ ] Consider multi-tenant repo isolation
• [ ] Support private repositories (GitHub App permissions)
• [ ] Consider semantic answer caching for queries
• [ ] Test: AST parsing across all supported languages
• [ ] Test: incremental sync detects changes correctly
• [ ] Test: hybrid search returns relevant results
• [ ] Test: graph queries return correct callers/callees
• [ ] Test: MCP tools work with Claude Code, Cursor, Gemini CLI
• [ ] Document supported languages and MCP tool usage
• [ ] Consider AI incident response for ingestion failures

FAQ

What is an AI GitHub knowledge base?

An AI GitHub knowledge base is a system that ingests code repositories and transforms them into a queryable knowledge graph using RAG and AST analysis. CodeRAG defines it: "CodeRAG creates a semantic vector database (RAG) from your source code, documentation, and project backlog, then exposes it as MCP tools that give AI agents deep understanding of your entire codebase." CogniRepo reports: "On Python repos >= 15K LOC, CogniRepo cuts AI coding agent token usage by 50-80% compared to raw file reads." Key capabilities: (1) AST-based code parsing with tree-sitter — chunks by functions, classes, methods, not arbitrary line splits. (2) GraphRAG — Neo4j call graphs showing how classes, methods, and decorators communicate. (3) Hybrid search — vector similarity (semantic) + BM25 (keyword) with Reciprocal Rank Fusion. (4) MCP tools — coderag_search, coderag_context, coderag_explain. (5) Incremental sync — Git diffs to re-index only changed files.

How does AST-based code chunking work?

AST-based code chunking uses tree-sitter to parse source code into an Abstract Syntax Tree and split it into semantically meaningful units — functions, classes, methods — instead of arbitrary line splits. CodeRAG: "Tree-sitter parses source code into semantically meaningful chunks (functions, classes, methods), not arbitrary line splits." VortexRAG: "AST-based semantic chunking using tree-sitter." Benefits: (1) Each chunk is a complete semantic unit — no half-functions. (2) Chunks include metadata: function name, class name, file path, line range, parameters. (3) Natural language summaries are generated for each chunk, enriching the embedding. (4) Supports 15-26 programming languages via tree-sitter grammars. (5) Chunks can be enriched with import context and decorator information. This produces 10x better retrieval quality compared to arbitrary line-based chunking.

What is GraphRAG for code repositories?

GraphRAG for code repositories combines vector search with structural code analysis using a knowledge graph (Neo4j). VortexRAG: "By integrating vector storage (Qdrant) with structural AST call-graphs (Neo4j), the application understands not only what code exists, but how classes, methods, and decorators communicate with each other." Central Code Knowledge Graph: "One Neo4j graph spans all registered repos — Functions, classes, files, imports, calls all stored as labeled nodes + typed relationships." Graph queries enable: (1) callers_of — who calls this function? (2) callees_of — what does this function call? (3) blast_radius — what files are affected if this file changes? (4) downstream_dependencies — what does this file depend on? (5) entry_points — most-connected functions. Vector search finds semantically relevant code; graph traversal adds structural context (sibling functions, call paths, containing classes).

How do you keep a code knowledge base current?

Keep a code knowledge base current with incremental sync using Git diffs. VortexRAG: "Smart Delta Ingestion uses Git diffs to delete, chunk, and upsert only modified files, completing ingestion in seconds. Full ingestion (150 large files): ~45 seconds. Smart Delta Sync (2 modified files): 1.8 seconds." CodeRAG: "Subsequent coderag index runs are incremental — only changed files are re-processed." Central Code Knowledge Graph: "keeps it fresh via incremental ingest + webhooks." Implementation: (1) Track last indexed commit hash. (2) On sync, run git diff to find changed files. (3) Delete vectors and graph nodes for changed files. (4) Re-parse, re-chunk, re-embed, and re-insert only changed files. (5) Update commit hash. (6) Use GitHub webhooks to trigger sync on push events.

How do MCP tools enable AI agents to use a code knowledge base?

MCP (Model Context Protocol) tools expose the code knowledge base to AI agents like Claude Code, Cursor, and Gemini CLI. CodeRAG provides 6 MCP tools: (1) coderag_search — semantic + keyword hybrid search across the codebase. (2) coderag_context — assemble relevant context within a token budget. (3) coderag_explain — explain a code symbol with full surrounding context. (4) coderag_status — check index health and statistics. (5) coderag_docs — search indexed documentation (Markdown, Confluence, SharePoint). (6) coderag_backlog — query project backlog items (Jira, ClickUp, Azure DevOps). CogniRepo provides 34 MCP tools including lookup_symbol (O(1) symbol lookup), who_calls (trace callers), and context_pack (token-budget code context). MCP tools work with stdio and SSE transport, enabling both local and remote agent connections.

Want a self-hosted AI company brain that does all of this out of the box?
Book a demo →