[Feature] Add automatic llms.txt generation via mkdocs-llmstxt plugin

docs/README.md

@@ -72,4 +72,14 @@ This guide is for contributors looking to make changes to the documentation in t
 
 2. **Push your new changes on a new branch**: Feel free to add or edit existing documentation and open a PR for your changes. Once your PR is reviewed and approved, the changes will be ready to merge into main. 
 
-3. **Updating the website**: Once your changes are merged to main, the changes would be reflected on live websites usually in 5-15 mins.
+3. **Updating the website**: Once your changes are merged to main, the changes would be reflected on live websites usually in 5-15 mins.
+
+## LLMs.txt Files
+
+DSPy documentation automatically generates [llmstxt.org](https://llmstxt.org/) compliant files to help LLMs understand and work with DSPy:
+
+- **`/llms.txt`** - A structured overview with links to key documentation sections
+- **`/llms-full.txt`** - A comprehensive version with full content for deep understanding
+
+These txt files are automatically generated during the docs build process using the [mkdocs-llmstxt](https://github.com/pawamoy/mkdocs-llmstxt) plugin. When modifying documentation, consider updating the `llmstxt` plugin section within `mkdocs.yml` to help LLMs better understand DSPy's capabilities.
+

docs/mkdocs.yml

@@ -236,6 +236,39 @@ plugins:
             "docs/quick-start/getting-started-02.md": "tutorials/rag/index.ipynb"
             "quick-start/getting-started-01.md": "tutorials/rag/index.ipynb"
             "quick-start/getting-started-02.md": "tutorials/rag/index.ipynb"
+    - llmstxt:
+        markdown_description: >
+          DSPy is the framework for programming—rather than prompting—language models.
+          DSPy unifies techniques for prompting, fine-tuning, reasoning, tool use, and evaluation of LMs.
+          It provides a systematic approach to building AI applications through composable modules, 
+          optimization techniques, and evaluation frameworks.
+        sections:
+          Getting Started:
+            - index.md: DSPy overview and quick start guide
+            - cheatsheet.md: DSPy cheatsheet for quick reference
+          Core Concepts:
+            - learn/programming/overview.md: Programming paradigm and philosophy
+            - learn/programming/signatures.md: Signatures - declarative input/output specifications
+            - learn/programming/modules.md: Modules - composable AI components
+            - learn/programming/language_models.md: Language model interfaces and configuration
+          Essential Tutorials:
+            - tutorials/rag/index.ipynb: Retrieval-Augmented Generation (RAG) tutorial
+            - tutorials/classification/index.md: Classification with DSPy
+            - tutorials/agents/index.ipynb: Building AI agents with DSPy
+          Optimization:
+            - learn/optimization/overview.md: Optimization techniques overview
+            - tutorials/optimize_ai_program/index.md: Guide to optimizing AI programs
+            - api/optimizers/BootstrapFewShot.md: Bootstrap few-shot optimizer
+          Key Modules API:
+            - api/modules/Predict.md: Basic prediction module
+            - api/modules/ChainOfThought.md: Chain of thought reasoning
+            - api/modules/ReAct.md: ReAct agent module
+          Core API Reference:
+            - api/signatures/Signature.md: Signature system documentation
+            - api/primitives/Example.md: Example primitive for training data
+          Production:
+            - tutorials/deployment/index.md: Production deployment guide
+            - tutorials/observability/index.md: Debugging and observability
 
 extra:
     social:

docs/requirements.txt

@@ -5,5 +5,6 @@ mkdocs-material[imaging]
 mkdocs-redirects
 mkdocstrings
 mkdocstrings-python
+mkdocs-llmstxt>=0.3.0
 urllib3==1.26.6
 mistune==3.0.2

docs/vercel.json

@@ -1,3 +1,14 @@
 {
-  "trailingSlash": true
+  "trailingSlash": true,
+  "headers": [
+    {
+      "source": "/(.*).md",
+      "headers": [
+        {
+          "key": "Content-Type",
+          "value": "text/markdown; charset=utf-8"
+        }
+      ]
+    }
+  ]
 }