Using Corporate Proxy with Langchain, Langsmith, OpenAI etc...

[thibaudbrg]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

.

Description

Hi everyone,
First of all, thank you for your awesome work at Langchain.

I'm building a rather complex python project pimped with AI using Lanchain, Langsmith and any LLMs (I mostly use OpenAI, Claude and Ollama+llama3-70B). The core logic of the app is built with the help of the LangGraph framework. This clearly make the code conceptually more readable and conveniant for everyone.

I'm trying to move the code to my corporate desktop but I'm facing the issue with the corporate proxy.
I'd like to be able to forward any requests that the code makes to any endpoint (OpenAI or Claude or Langsmith) through my corporate proxy.

E           httpx.ProxyError: 407 Proxy Authentication Required
E           openai.APIConnectionError: Connection error.

LangSmithConnectionError('Connection error caused failure to POST https://api.smith.langchain.com/runs/batch  in LangSmith API. Please confirm your internet connection.. ProxyError(MaxRetryError("HTTPSConnectionPool(host=\'api.smith.langchain.com\', port=443): Max retries exceed
ed with url: /runs/batch (Caused by ProxyError(\'Unable to connect to proxy\', OSError(\'Tunnel connection failed: 407 Proxy Authentication Required\')))"))

Even if I saw several solution tentatives with OpenAI, it does not really work for me and doesn't solve the solution in general. I'm looking for an easy way to pass my http_proxy and my https_proxy urls to Langchain so that it automatically uses them when requesting any endpoint. (say OpenAI or Claude or Langsmith APIs)

In fact, for OpenAI solely, I tried setting up OPENAI_PROXY or even http_proxy and https_proxy in the .env file but without success. (#1423). I also tried to implement the variables directly into the ChatOpenAI() instance class (#8896).

For Langsmith, I have no idea.

To let you have a better understanding of my code, here are some highlights:

• I use LangGraph:

from langgraph.graph import END, StateGraph
.....
workflow = StateGraph(GraphState)

workflow.add_node("process_directory", process_directory)
.....
workflow.add_node("handle_logs", handle_logs)

workflow.set_entry_point("process_directory")
workflow.add_edge("process_directory", "analyze_next_file")
.....
workflow.add_conditional_edges("critic_generation", evaluate_quality_decider, {
    "re_gen": "generate",
    "human_check": "human_review",
})
.....
app = workflow.compile()

• The usual calls to OpenAI are:

model = ChatOpenAI(temperature=0, model=MODEL_NAME, streaming=True, openai_proxy=OPENAI_PROXY)

• My .env looks like this:

export LANGCHAIN_TRACING_V2=true
export LANGCHAIN_API_KEY=lsv2_pt_######################
export LANGCHAIN_PROJECT=######
export MODEL_NAME =gpt-4-turbo-preview
export OPENAI_API_KEY=sk-##################
export OPENAI_PROXY=http://######:8080

• I run the code from this python test file:

from dotenv import load_dotenv
load_dotenv()

.......
from langchain_openai import ChatOpenAI
.......

def test_workflow():
    inputs = {}

    # Run the application with the initialized inputs
    try:
        for output in app.stream(inputs):
            for key, value in output.items():
                print("\n---\n")
    except WorkflowExit:
        print("Workflow exited early as expected.")

Thanks for your help!

System Info

Python 3.11.9

Windows 10

• langchain-cli 0.0.21
• langchain-community 0.0.33
• langchain-core 0.1.44
• langchain-openai 0.1.3
• langchainhub 0.1.15
• langgraph 0.0.38
• langserve 0.1.0

requests 2.31.0

[dosubot[bot]]

Hello, @thibaudbrg! I'm here to help you with your Langchain issue. I'm a bot that can assist you with bugs, questions, and becoming a contributor. Let's tackle this challenge together!

To configure your Python project using Langchain, Langsmith, and various LLMs to forward requests through your corporate proxy, you need to set up the proxy settings for each component. Here are the steps to achieve this:

1. Configure ChatOpenAI to use a proxy:

   The ChatOpenAI class handles proxy settings through the openai_proxy parameter. You can configure it to use a proxy directly by passing the proxy URL to this parameter.

   from langchain_openai import ChatOpenAI
   
   model = ChatOpenAI(
       temperature=0,
       model=MODEL_NAME,
       streaming=True,
       openai_proxy="http://username:password@proxyserver:port"
   )

   Replace username, password, proxyserver, and port with your actual proxy credentials and server details.

2. Configure LLMRequestsChain to use a proxy:

   Update the TextRequestsWrapper to accept proxy settings and pass the proxy settings when initializing the LLMRequestsChain.

   from langchain.chains import LLMChain
   from langchain_community.utilities.requests import TextRequestsWrapper
   
   # Define your proxy settings
   proxy_settings = {
       "http": "http://username:password@proxyserver:port",
       "https": "http://username:password@proxyserver:port"
   }
   
   # Update the TextRequestsWrapper to use the proxy settings
   requests_wrapper = TextRequestsWrapper(headers=DEFAULT_HEADERS, proxies=proxy_settings)
   
   # Initialize the LLMChain (assuming you have the necessary LLMChain setup)
   llm_chain = LLMChain(...)
   
   # Create the LLMRequestsChain with the updated requests_wrapper
   llm_requests_chain = LLMRequestsChain(llm_chain=llm_chain, requests_wrapper=requests_wrapper)
   
   # Now you can use llm_requests_chain to make requests through your corporate proxy

3. Environment Variables:

   Ensure your .env file is correctly set up with the proxy settings:

   export LANGCHAIN_TRACING_V2=true
   export LANGCHAIN_API_KEY=lsv2_pt_######################
   export LANGCHAIN_PROJECT=######
   export MODEL_NAME=gpt-4-turbo-preview
   export OPENAI_API_KEY=sk-##################
   export OPENAI_PROXY=http://username:password@proxyserver:port

4. Langsmith Proxy Configuration:

   Unfortunately, specific details on configuring Langsmith to use a proxy are not available. However, you can try setting the proxy environment variables (http_proxy and https_proxy) and see if Langsmith respects these settings.

By following these steps, you should be able to forward requests through your corporate proxy for Langchain, Langsmith, and various LLMs.

For more details on configuring proxies for specific components, you can refer to the relevant sections in the Langchain documentation and source code:

• ChatOpenAI proxy configuration ^[1]
• LLMRequestsChain proxy configuration ^[2]
• TextRequestsWrapper proxy configuration ^[3]

To continue talking to Dosu, mention @dosu.

[thibaudbrg]

So I indeed managed to make OpenAI work with the proxy by simply setting up OPENAI_PROXY (specifying openai_proxy in the ChatOpenAI() constructor is equivalent.

But my question remains for Langsmith.
And a general solution configurable with some kind of env variable such as LANGCHAIN_PROXY_URL for any requests would be really appreciated! See this Langsmith SDK question (202)

[dosubot[bot]]

To address your need for configuring Langsmith to use a proxy and a general solution for LangChain, here are some steps and suggestions:

1. Langsmith Proxy Configuration:

Currently, there is no built-in support for configuring proxy settings directly in Langsmith. However, you can try setting the http_proxy and https_proxy environment variables, which many libraries respect by default. Here’s how you can set these environment variables in your .env file:

export http_proxy=http://username:password@proxyserver:port
export https_proxy=http://username:password@proxyserver:port

2. General Proxy Configuration for LangChain:

To create a general solution for LangChain that uses a proxy for all HTTP requests, you can modify the Requests class to include proxy settings. Here’s an example of how you can do this:

from typing import Any, Dict, Optional
import requests
from langchain_core.pydantic_v1 import BaseModel, Extra

class Requests(BaseModel):
    """Wrapper around requests to handle auth and async."""

    headers: Optional[Dict[str, str]] = None
    aiosession: Optional[aiohttp.ClientSession] = None
    auth: Optional[Any] = None
    proxies: Optional[Dict[str, str]] = None  # Add proxies attribute

    class Config:
        """Configuration for this pydantic object."""
        extra = Extra.forbid
        arbitrary_types_allowed = True

    def get(self, url: str, **kwargs: Any) -> requests.Response:
        """GET the URL and return the text."""
        return requests.get(url, headers=self.headers, auth=self.auth, proxies=self.proxies, **kwargs)

    def post(self, url: str, data: Dict[str, Any], **kwargs: Any) -> requests.Response:
        """POST to the URL and return the text."""
        return requests.post(
            url, json=data, headers=self.headers, auth=self.auth, proxies=self.proxies, **kwargs
        )

    def patch(self, url: str, data: Dict[str, Any], **kwargs: Any) -> requests.Response:
        """PATCH the URL and return the text."""
        return requests.patch(
            url, json=data, headers=self.headers, auth=self.auth, proxies=self.proxies, **kwargs
        )

    def put(self, url: str, data: Dict[str, Any], **kwargs: Any) -> requests.Response:
        """PUT the URL and return the text."""
        return requests.put(
            url, json=data, headers=self.headers, auth=self.auth, proxies=self.proxies, **kwargs
        )

    def delete(self, url: str, **kwargs: Any) -> requests.Response:
        """DELETE the URL and return the text."""
        return requests.delete(url, headers=self.headers, auth=self.auth, proxies=self.proxies, **kwargs)

You can then instantiate the Requests class with the proxy settings:

proxies = {
    "http": "http://username:password@proxyserver:port",
    "https": "http://username:password@proxyserver:port",
}

requests_instance = Requests(proxies=proxies)

3. Environment Variable for General Proxy Configuration:

To make this more convenient, you can set up an environment variable like LANGCHAIN_PROXY_URL and use it in your code:

import os

proxies = {
    "http": os.getenv("LANGCHAIN_PROXY_URL"),
    "https": os.getenv("LANGCHAIN_PROXY_URL"),
}

requests_instance = Requests(proxies=proxies)

Ensure your .env file includes the LANGCHAIN_PROXY_URL:

export LANGCHAIN_PROXY_URL=http://username:password@proxyserver:port

By following these steps, you should be able to configure Langsmith and LangChain to use a proxy for all HTTP requests. This approach leverages environment variables for easy configuration and ensures that all requests are routed through your corporate proxy.

To continue talking to Dosu, mention @dosu.

[thibaudbrg]

This does not really solves it because I would still need to have Langsmith components to use my new requests_instance wrapper.

The problem needs a more general solution.

Another example, while trying to use ChatOllama, I need to pass the proxy and the corporate certificate. This cannot be achieved as there does not seem to exist any parameters in ChatOllama() see doc that pass cert= and proxies= up to requests.

[dosubot[bot]]

To address your need for a more general solution that can be applied across various components like Langsmith and ChatOllama, you can leverage environment variables and utility functions to configure proxy and certificate settings globally. Here’s a comprehensive approach:

1. Environment Variables for Proxy and Certificate Configuration:

Set up your environment variables in the .env file:

export LANGCHAIN_PROXY_URL=http://username:password@proxyserver:port
export LANGCHAIN_CERT_PATH=/path/to/your/corporate/certificate

2. Utility Functions to Retrieve Environment Variables:

Use the utility functions from libs/core/langchain_core/utils/env.py to retrieve these environment variables in your code:

from langchain_core.utils.env import get_from_env

proxy_url = get_from_env("proxy_url", "LANGCHAIN_PROXY_URL")
cert_path = get_from_env("cert_path", "LANGCHAIN_CERT_PATH")

3. Global Configuration for HTTP Requests:

Modify the Requests class or create a global configuration function to apply these settings to all HTTP requests:

from typing import Any, Dict, Optional
import requests
from langchain_core.pydantic_v1 import BaseModel, Extra

class Requests(BaseModel):
    """Wrapper around requests to handle auth and async."""

    headers: Optional[Dict[str, str]] = None
    aiosession: Optional[aiohttp.ClientSession] = None
    auth: Optional[Any] = None
    proxies: Optional[Dict[str, str]] = None
    cert: Optional[str] = None  # Add cert attribute

    class Config:
        """Configuration for this pydantic object."""
        extra = Extra.forbid
        arbitrary_types_allowed = True

    def get(self, url: str, **kwargs: Any) -> requests.Response:
        """GET the URL and return the text."""
        return requests.get(url, headers=self.headers, auth=self.auth, proxies=self.proxies, cert=self.cert, **kwargs)

    def post(self, url: str, data: Dict[str, Any], **kwargs: Any) -> requests.Response:
        """POST to the URL and return the text."""
        return requests.post(
            url, json=data, headers=self.headers, auth=self.auth, proxies=self.proxies, cert=self.cert, **kwargs
        )

    def patch(self, url: str, data: Dict[str, Any], **kwargs: Any) -> requests.Response:
        """PATCH the URL and return the text."""
        return requests.patch(
            url, json=data, headers=self.headers, auth=self.auth, proxies=self.proxies, cert=self.cert, **kwargs
        )

    def put(self, url: str, data: Dict[str, Any], **kwargs: Any) -> requests.Response:
        """PUT the URL and return the text."""
        return requests.put(
            url, json=data, headers=self.headers, auth=self.auth, proxies=self.proxies, cert=self.cert, **kwargs
        )

    def delete(self, url: str, **kwargs: Any) -> requests.Response:
        """DELETE the URL and return the text."""
        return requests.delete(url, headers=self.headers, auth=self.auth, proxies=self.proxies, cert=self.cert, **kwargs)

4. Instantiate the Requests Class with Environment Variables:

Instantiate the Requests class with the proxy and certificate settings retrieved from the environment variables:

proxies = {
    "http": proxy_url,
    "https": proxy_url,
}

requests_instance = Requests(proxies=proxies, cert=cert_path)

5. Integrate with Langsmith and Other Components:

Ensure that all components, including Langsmith and ChatOllama, use this requests_instance for their HTTP requests. This might require modifying the components to accept a requests parameter or globally patching the request handling mechanism.

Example for ChatOllama:

If ChatOllama does not support direct proxy and certificate configuration, you can wrap its HTTP requests using the requests_instance:

from langchain_community.chat_models.ollama import ChatOllama

class CustomChatOllama(ChatOllama):
    def __init__(self, *args, **kwargs):
        self.requests_instance = kwargs.pop('requests_instance', None)
        super().__init__(*args, **kwargs)

    def _make_request(self, url, **kwargs):
        if self.requests_instance:
            return self.requests_instance.get(url, **kwargs)
        return super()._make_request(url, **kwargs)

# Usage
chat_ollama = CustomChatOllama(requests_instance=requests_instance)

By following these steps, you can create a more general solution that configures proxy and certificate settings for all components in your LangChain-based project ^[1][2].

To continue talking to Dosu, mention @dosu.

[pascuC]

Hello,

Any update on this? I also have the proxy problem with LangSmith. When I try to use LangSmith with my LangChain project from within the organization I got the following error:

Failed to get info from https://api.smith.langchain.com: ConnectionError(MaxRetryError('HTTPSConnectionPool(host='api.smith.langchain.com', port=443): Max retries exceeded with url: /info (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x000001D....>: Failed to resolve 'api.smith.langchain.com' ([Errno 11001] getaddrinfo failed)"))'))

Thank you!

[JulianHandrup]

Not to forget: "no_proxy".
Imagine trying to support internally hosted models and APIs as well as external ones.
Some requests mustn't use the proxy.