diff --git a/docs/README.md b/docs/README.md index df486afbaa..3b81f8b034 100644 --- a/docs/README.md +++ b/docs/README.md @@ -72,4 +72,14 @@ This guide is for contributors looking to make changes to the documentation in t 2. **Push your new changes on a new branch**: Feel free to add or edit existing documentation and open a PR for your changes. Once your PR is reviewed and approved, the changes will be ready to merge into main. -3. **Updating the website**: Once your changes are merged to main, the changes would be reflected on live websites usually in 5-15 mins. \ No newline at end of file +3. **Updating the website**: Once your changes are merged to main, the changes would be reflected on live websites usually in 5-15 mins. + +## LLMs.txt Files + +DSPy documentation automatically generates [llmstxt.org](https://llmstxt.org/) compliant files to help LLMs understand and work with DSPy: + +- **`/llms.txt`** - A structured overview with links to key documentation sections +- **`/llms-full.txt`** - A comprehensive version with full content for deep understanding + +These txt files are automatically generated during the docs build process using the [mkdocs-llmstxt](https://github.com/pawamoy/mkdocs-llmstxt) plugin. When modifying documentation, consider updating the `llmstxt` plugin section within `mkdocs.yml` to help LLMs better understand DSPy's capabilities. + diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index 9d9c90f6eb..f6ba7154d5 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -236,6 +236,39 @@ plugins: "docs/quick-start/getting-started-02.md": "tutorials/rag/index.ipynb" "quick-start/getting-started-01.md": "tutorials/rag/index.ipynb" "quick-start/getting-started-02.md": "tutorials/rag/index.ipynb" + - llmstxt: + markdown_description: > + DSPy is the framework for programming—rather than prompting—language models. + DSPy unifies techniques for prompting, fine-tuning, reasoning, tool use, and evaluation of LMs. + It provides a systematic approach to building AI applications through composable modules, + optimization techniques, and evaluation frameworks. + sections: + Getting Started: + - index.md: DSPy overview and quick start guide + - cheatsheet.md: DSPy cheatsheet for quick reference + Core Concepts: + - learn/programming/overview.md: Programming paradigm and philosophy + - learn/programming/signatures.md: Signatures - declarative input/output specifications + - learn/programming/modules.md: Modules - composable AI components + - learn/programming/language_models.md: Language model interfaces and configuration + Essential Tutorials: + - tutorials/rag/index.ipynb: Retrieval-Augmented Generation (RAG) tutorial + - tutorials/classification/index.md: Classification with DSPy + - tutorials/agents/index.ipynb: Building AI agents with DSPy + Optimization: + - learn/optimization/overview.md: Optimization techniques overview + - tutorials/optimize_ai_program/index.md: Guide to optimizing AI programs + - api/optimizers/BootstrapFewShot.md: Bootstrap few-shot optimizer + Key Modules API: + - api/modules/Predict.md: Basic prediction module + - api/modules/ChainOfThought.md: Chain of thought reasoning + - api/modules/ReAct.md: ReAct agent module + Core API Reference: + - api/signatures/Signature.md: Signature system documentation + - api/primitives/Example.md: Example primitive for training data + Production: + - tutorials/deployment/index.md: Production deployment guide + - tutorials/observability/index.md: Debugging and observability extra: social: diff --git a/docs/requirements.txt b/docs/requirements.txt index 8083e88fe6..36badd2323 100644 --- a/docs/requirements.txt +++ b/docs/requirements.txt @@ -5,5 +5,6 @@ mkdocs-material[imaging] mkdocs-redirects mkdocstrings mkdocstrings-python +mkdocs-llmstxt>=0.3.0 urllib3==1.26.6 mistune==3.0.2 diff --git a/docs/vercel.json b/docs/vercel.json index 1e9c7bc398..9e4754e934 100644 --- a/docs/vercel.json +++ b/docs/vercel.json @@ -1,3 +1,14 @@ { - "trailingSlash": true + "trailingSlash": true, + "headers": [ + { + "source": "/(.*).md", + "headers": [ + { + "key": "Content-Type", + "value": "text/markdown; charset=utf-8" + } + ] + } + ] }