OpenAPI Generator Python stub for MCP server

.gitignore

@@ -1,2 +1,2 @@
 .DS_STORE
-*.gguf
+*.gguf

06-service-recommender/.env-template

@@ -0,0 +1,14 @@
+# Copy this to a .env file and populate with secrets
+# .env is used by Docker compose to set environment variables
+
+OPENAI_API_KEY='...'
+
+AWS_DEFAULT_REGION=us-east-1
+AWS_ACCESS_KEY_ID='...'
+AWS_SECRET_ACCESS_KEY='...'
+
+# Optional: Enable authentication for Phoenix.
+# A long string value that is used to sign JWTs for your deployment.
+# It should be a good mix of characters and numbers and should be kept secret.
+# https://arize.com/docs/phoenix/self-hosting/features/authentication
+# PHOENIX_SECRET=20250613navalabs0000SomeLongAlphanumericString

06-service-recommender/.gitignore

@@ -0,0 +1 @@
+.env

06-service-recommender/README.md

@@ -0,0 +1,210 @@
+
+## Architecture
+
+* When a user submits a question in the Streamlit UI (`frontend` subfolder), the frontend calls the API (`backend` subfolder) provided by the Hayhooks service.
+* Haystack pipelines are registered with the Hayhooks service. Each pipeline is associated with an API endpoint by name, e.g., the pipeine defined in `backend/pipelines/first` can by triggered via a `POST` request to `http://localhost:1416/first/run`.
+* During Haystack pipeline execution, traces are sent to the Phoenix observability tool (running as `phoenix` Docker compose service).
+* Phoenix persists those traces to a DB (running as `db` Docker compose service).
+
+### Details
+
+* The Streamlit UI is at http://localhost:8501
+    * This can be replaced with a different frontend implemnentation.
+* The Phoenix UI is at http://localhost:6006
+    * and receives tracing data (i.e., OTLP HTTP data) via `http://localhost:6006/v1/traces` (and OTLP gRPC data via `http://localhost:4317`)
+* A Haystack pipeline can query Phoenix for prompt templates.
+    * Non-engineers can conduct prompt engineering in the Phoenix UI and have the changes be immediately reflected in pipelines that use the prompt template.
+* The Hayhooks service listens on port 1416 -- API docs are at http://localhost:1416/docs
+    * New pipelines under a folder can be registered dynamically using `hayhooks pipeline deploy-files -n my_pipeline SOME_PIPELINES_FOLDER`
+    * Pipelines can be tested using `curl -X 'POST' 'http://localhost:1416/first/run' -H 'accept: application/json' -H 'Content-Type: application/json' -d '{  "question": "Who lives in Paris?" }'` or using the `hayhooks` commandline.
+    * The Hayhooks service can be replaced or supplemented with a separate API implementation that [calls the Hayhooks service](https://docs.haystack.deepset.ai/docs/hayhooks#running-programmatically) or runs Haystack pipelines directly.
+
+## Running
+
+### Create `.env` with API keys
+
+Copy `.env-template` to a `.env` file and populate with your secrets. These secrets (e.g., `OPENAI_API_KEY`) are referenced in the `compose.yaml` file and are used by Haystack pipelines to call LLMs.
+
+### Docker Compose
+
+```
+cd backend
+
+# Add a prompt template to Phoenix for Haystack pipeline to use
+uv run src/bootstrap.py
+
+cd ..
+```
+
+Build and start all necessary Docker containers: `docker compose up --build`
+
+### Running outside of Docker containers
+
+The frontend and backend can be run outside of Docker containers if desired.
+
+Install Python and `uv`.
+
+Note the `requirements.txt` files are only for building Docker images. It is created by running `uv pip compile pyproject.toml -o requirements.txt` using dependencies declared in `pyproject.toml`.
+
+Tip: During development, open the `frontend` and `backend` subfolders as different VSCode projects. For each project, have VSCode use the Python interpreter in the respective `.venv` subfolder.
+
+#### Run Streamlit frontend
+```
+cd frontend
+uv sync
+uv run streamlit run src/main.py
+```
+
+Optionally, run `source .venv/bin/activate` to avoid having to type `uv run` or `uvx`.
+
+#### Run API backend
+```
+# Export all variables in .env
+set -o allexport
+source .env
+
+cd backend
+
+# Download Python dependencies
+uv sync
+
+# Add a prompt template to Phoenix for Haystack pipeline to use
+uv run src/bootstrap.py
+
+# Start Hayhooks service
+uvx hayhooks run --additional-python-path .
+```
+
+During pipeline development, test a Haystack pipeline before deploying it to Hayhooks.
+For example: `uv run src/haystack_rag.py`
+
+## To enable Phoenix authentication
+
+Based on [documentation](https://arize.com/docs/phoenix/self-hosting/features/authentication), set `PHOENIX_SECRET` in `.env` and modify `compose.yaml` as follows.
+* Add these environment variables to the `phoenix` service:
+```
+      - PHOENIX_ENABLE_AUTH=True
+      - PHOENIX_SECRET=${PHOENIX_SECRET}
+```
+* Log into the Phoenix UI at http://localhost:6006 and create an API key
+* Add these environment variables to the `backend` service:
+```
+      - PHOENIX_API_KEY=${PHOENIX_API_KEY}
+      - OTEL_EXPORTER_OTLP_HEADERS=${OTEL_EXPORTER_OTLP_HEADERS}
+```
+
+## MCP Services
+
+* https://haystack.deepset.ai/integrations/mcp
+    - https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/mcp/examples
+* https://docs.haystack.deepset.ai/docs/mcptool#with-the-agent-component
+
+### MCP Python SDK
+
+* https://github.com/modelcontextprotocol/python-sdk
+* referring to [Haystack MCP examples](https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/mcp/examples)
+
+https://github.com/modelcontextprotocol/python-sdk?tab=readme-ov-file#streamable-http-transport:
+> Note: Streamable HTTP transport is superseding SSE transport for production deployments.
+
+and https://brightdata.com/blog/ai/sse-vs-streamable-http#:~:text=As%20explained%20here%2C%20third%2Dparty,specs%20must%20implement%20Streamable%20HTTP
+
+```sh
+uv run python src/mcp_server.py
+```
+
+Useful for MCP inspecting and debugging:
+```sh
+npx @modelcontextprotocol/inspector
+```
+
+```sh
+uv run python src/first_mcp.py
+```
+
+### Simpler Grants.gov API
+
+- Get OpenAPI from https://api.simpler.grants.gov/openapi.json
+- Docs at https://api.simpler.grants.gov/docs
+
+### generate_tools_from_openapi
+
+This create regular REST endpoints, not Streamable HTTP or SSE, so the MCP inspector isn't relevant.
+
+From https://earthkhan.medium.com/turn-your-openapi-in-mcp-server-in-5-minutes-e2859383d0dc
+and https://github.com/nafiul-earth/openapi-2-mcpserver/blob/main/ibmcloud-cos-mcp-server/main.py,
+add `mcp_server_simpler_grants.py`,
+
+```sh
+uv run python src/mcp_server_simpler_grants.py
+```
+
+Quick test: `curl http://localhost:8000/tools`
+
+MCP client test:
+```sh
+uv run python src/mcp_simpler_grants_client.py
+```
+
+### Convert to SSE-based MCP service
+
+Use https://github.com/tadata-org/fastapi_mcp to quickly create MCP service via SSE. Unfortunately, Streamable HTTP is [not available](https://github.com/tadata-org/fastapi_mcp/discussions/142).
+
+```sh
+uv run python src/mcp_server_simpler_grants_sse.py
+```
+
+Run MCP inspector
+```sh
+npx @modelcontextprotocol/inspector
+```
+and connect to http://localhost:8000/mcp.
+
+However, only 2 tools are listed:
+- `list_tools_tools_get`
+- `invoke_tool_invoke_post`
+since the `generate_tools_from_openapi` approach was used.
+
+### Convert to Streamable HTTP transport protocol
+
+> MCP (Model Context Protocol) can use non-streamable HTTP, but it's not the preferred or recommended approach. While older versions of MCP relied heavily on Server-Sent Events (SSE) for streaming data, the current specification favors Streamable HTTP. Streamable HTTP allows for a more efficient and stateless way to communicate with MCP servers, making it the preferred method for new implementations. 
+
+#### Test https://mcp.liblab.com/
+* Had to fix OpenAPI spec using https://editor.swagger.io/#/
+* The 4th version of the yaml file worked, though with some errors
+* Saved in `mcp_simpler_grants_gov` folder -- see `mcp_simpler_grants_gov/PyPI_README.md`
+
+```
+cd mcp_simpler_grants_gov
+uv venv
+uv sync
+```
+
+Wait this is the SDK!
+
+#### jlowin's FastMCP
+
+(Not to be confused with MCP Python SDK which uses the `mcp.server.fastmcp` package.)
+
+Use https://github.com/jlowin/fastmcp, specifically https://gofastmcp.com/servers/openapi#openapi-integration.
+
+> FastMCP can automatically generate an MCP server from an OpenAPI specification or FastAPI app. Instead of manually creating tools and resources, you provide an OpenAPI spec and FastMCP intelligently converts your API endpoints into the appropriate MCP components.
+
+```sh
+uv run python src/mcp_server_simpler_grants_streamable.py
+```
+
+- Had to remove `  "servers": "."` from json to address `OpenAPI schema validation failed`
+- Ran into `RecursionError: maximum recursion depth exceeded` (recent issues [931](https://github.com/jlowin/fastmcp/issues/931) and [1016](https://github.com/jlowin/fastmcp/issues/1016)). Downgrading to v2.8.1 works
+
+Run MCP inspector
+```sh
+npx @modelcontextprotocol/inspector
+```
+and connect to http://localhost:8000/mcp using Streamable HTTP transport protocol and list tools.
+
+- Select `Health` tool and "Run Tool"
+- Select `Opportunity_Search` tool and 
+   - Ran into `PointerToNowhere: '/components/schemas/SortOrderOpportunityPaginationV1' does not exist within ...`
+   - Found [PR](https://github.com/jlowin/fastmcp/pull/995) merged 3 day ago
+   - Replacing `SortOrderOpportunityPaginationV1` in the json with its definitions works!

06-service-recommender/backend/.dockerignore

@@ -0,0 +1 @@
+**/__pycache__

06-service-recommender/backend/.env_template

@@ -0,0 +1,4 @@
+
+# For MCP client
+SIMPLER_GRANTS_API_KEY="PasteFrom1Password"
+MCP_URL="http://localhost:8000/invoke"