CodeDoc - AI-Powered Documentation & Test Generator

A lightweight tool that leverages open-source LLMs to automatically generate documentation and unit tests for Python code, complete with evaluation metrics.

Project Overview

This tool demonstrates key competencies in:

• LLM Application Development: Integration with open-source models
• Prompt Engineering: Optimized prompts for code analysis
• Evaluation Metrics: Quantitative assessment of LLM outputs
• Workflow Automation: End-to-end code analysis pipeline

Features

1. Automatic Documentation Generation

   • Docstrings for functions and classes
   • Module-level documentation
   • Inline comments for complex logic

2. Unit Test Generation

   • Pytest-compatible test cases
   • Edge case detection
   • Coverage-focused test generation

3. Evaluation Metrics

   • Token efficiency analysis
   • Response time tracking
   • Quality scoring (completeness, correctness)
   • Comparative model performance

Prerequisites

• Python 3.8+
• Ollama installed locally (for open-source LLMs)

Installing Ollama

# macOS/Linux
curl -fsSL https://ollama.com/install.sh | sh

# Or visit: https://ollama.com/download

Pull a Model

ollama pull llama3.2        

Installation

# Clone or navigate to project directory
cd CodeDoc

# Install dependencies
pip install -r requirements.txt

Usage

Basic Usage

# Generate documentation for a Python file
python codedoc.py --file examples/sample.py --mode doc

# Generate unit tests
python codedoc.py --file examples/sample.py --mode test

# Generate both documentation and tests
python codedoc.py --file examples/sample.py --mode all

# Run with evaluation metrics
python codedoc.py --file examples/sample.py --mode all --evaluate

Advanced Options

# Specify a different model
python codedoc.py --file mycode.py --mode all --model codellama

# Compare multiple models
python codedoc.py --file mycode.py --mode all --compare-models llama3.2,mistral

# Output to specific directory
python codedoc.py --file mycode.py --mode all --output ./generated

Evaluation Metrics

The tool tracks:

1. Performance Metrics

   • Response time per request
   • Tokens per second
   • Total tokens used

2. Quality Metrics

   • Documentation completeness score
   • Test coverage estimation
   • Syntax validity

3. Comparative Analysis

   • Side-by-side model comparison
   • Cost-benefit analysis
   • Speed vs. quality trade-offs

Project Structure

CodeDoc/
├── codedoc.py              # Main application
├── llm_handler.py          # LLM integration layer
├── doc_generator.py        # Documentation generation
├── test_generator.py       # Unit test generation
├── evaluator.py            # Evaluation metrics
├── config.py               # Configuration settings
├── requirements.txt        # Dependencies
├── examples/               # Sample Python files
│   ├── sample.py
│   └── calculator.py
└── output/                 # Generated files (created at runtime)

Key Technical Concepts

1. LLM Optimization

   • Iterative prompt refinement
   • Context window management
   • Temperature and parameter tuning

2. Evaluation Framework

   • Quantitative metrics design
   • Benchmarking methodology
   • Performance tracking

3. Workflow Automation

   • Code parsing and analysis
   • Batch processing
   • Error handling and retry logic

4. Scalability Considerations

   • Rate limiting
   • Caching strategies
   • Parallel processing opportunities

Example Output

See examples/ directory for sample inputs and output/ for generated documentation and tests.

Contributing

This is a portfolio project designed to demonstrate LLM application development capabilities.

License

MIT License - Free to use and modify