Skip to content
@llm-d

llm-d

llm-d is a Kubernetes-native high-performance distributed LLM inference framework
README.md

Welcome to llm-d: a Kubernetes-native high-performance distributed LLM inference framework

GitHub Org's stars Documentation License

Join Slack X (formerly Twitter) Follow LinkedIn Reddit

llm-d is a well-lit path for serving large language models at scale with the fastest time-to-value and competitive performance per dollar. Built on vLLM, Kubernetes, and Inference Gateway, llm-d provides modular solutions for distributed inference with features like KV-cache aware routing and disaggregated serving.

Key Resources

🤝 How to Contribute

Join the Community

  1. 💬 Slack: Join our development discussions at llm-d.slack.com
  2. 📧 Google Group: Subscribe to llm-d-contributors for architecture docs and meeting invites
  3. 🗓️ Weekly Standup: Wednesdays at 1230 ET - Public Calendar

Contributing Code

  1. Read Guidelines: Review our Code of Conduct and contribution process
  2. Sign Commits: All commits require DCO sign-off (git commit -s)

Ways to Contribute

  • 🐛 Bug fixes and small features - Submit PRs directly to component repos
  • 🚀 New features with APIs - Require project proposals
  • 📚 Documentation - Help improve guides and examples
  • 🧪 Testing & Benchmarking - Contribute to our test coverage
  • 💡 Experimental features - Start in llm-d-incubation org

License: Apache 2.0

Pinned Loading

  1. llm-d llm-d Public

    llm-d is a Kubernetes-native high-performance distributed LLM inference framework

    Makefile 1.1k 65

  2. llm-d-inference-scheduler llm-d-inference-scheduler Public

    Inference scheduler for llm-d

    Go 50 20

  3. llm-d-deployer llm-d-deployer Public

    Helm charts for llm-d

    Shell 37 22

  4. llm-d-kv-cache-manager llm-d-kv-cache-manager Public

    Distributed KV cache coordinator

    Go 31 5

  5. llm-d-model-service llm-d-model-service Public

    Simplified model deployment on llm-d

    Go 21 8

  6. llm-d-benchmark llm-d-benchmark Public

    llm-d benchmark scripts and tooling

    Shell 13 6

Repositories

Showing 10 of 12 repositories
  • llm-d-benchmark Public

    llm-d benchmark scripts and tooling

    llm-d/llm-d-benchmark’s past year of commit activity
    Shell 13 Apache-2.0 6 1 1 Updated Jun 5, 2025
  • llm-d.github.io Public

    Website for llm-d: This repository builds the website seen at llm-d.ai

    llm-d/llm-d.github.io’s past year of commit activity
    JavaScript 9 13 2 0 Updated Jun 5, 2025
  • llm-d-inference-scheduler Public

    Inference scheduler for llm-d

    llm-d/llm-d-inference-scheduler’s past year of commit activity
    Go 50 Apache-2.0 20 51 (3 issues need help) 5 Updated Jun 4, 2025
  • llm-d-inference-sim Public

    A light weight vLLM simulator, for mocking out replicas.

    llm-d/llm-d-inference-sim’s past year of commit activity
    Go 22 11 9 4 Updated Jun 4, 2025
  • .github Public
    llm-d/.github’s past year of commit activity
    0 0 0 0 Updated Jun 4, 2025
  • llm-d Public

    llm-d is a Kubernetes-native high-performance distributed LLM inference framework

    llm-d/llm-d’s past year of commit activity
    Makefile 1,068 Apache-2.0 65 18 16 Updated Jun 3, 2025
  • llm-d-kv-cache-manager Public

    Distributed KV cache coordinator

    llm-d/llm-d-kv-cache-manager’s past year of commit activity
    Go 31 4 9 (2 issues need help) 0 Updated Jun 3, 2025
  • llm-d-model-service Public

    Simplified model deployment on llm-d

    llm-d/llm-d-model-service’s past year of commit activity
    Go 21 Apache-2.0 7 27 3 Updated Jun 2, 2025
  • llm-d-deployer Public

    Helm charts for llm-d

    llm-d/llm-d-deployer’s past year of commit activity
    Shell 37 Apache-2.0 21 19 5 Updated May 30, 2025
  • llm-d-pd-utils Public
    llm-d/llm-d-pd-utils’s past year of commit activity
    Makefile 3 4 0 0 Updated May 21, 2025
View all repositories

Most used topics

incubating ai