langchain-ai · samnoyes · Jun 6, 2025 · tanushree-sharma · Jun 7, 2025 · tanushree-sharma
diff --git a/docs/evaluation/how_to_guides/bind_evaluator_to_dataset.mdx b/docs/evaluation/how_to_guides/bind_evaluator_to_dataset.mdx
@@ -2,7 +2,7 @@
 sidebar_position: 2
 ---
 
-# How to bind an evaluator to a dataset in the UI
+# Automatically run evaluators on experiments
 
 While you can specify evaluators to grade the results of your experiments programmatically (see [this guide](./evaluate_llm_application) for more information), you can also bind evaluators to a dataset in the UI.
-While you can specify evaluators to grade the results of your experiments programmatically (see [this guide](./evaluate_llm_application) for more information), you can also bind evaluators to a dataset in the UI.
+You can grade your experiment results in two ways: **programmatically**, by specifying evaluators in your code (see [this guide](./evaluate_llm_application) for details) or you can **automatically run evaluators** defined in the UI. By binding evaluators to a dataset in the UI, they'll automatically run on your experiments. These UI-configured evaluators complement any you've set up via the SDK. We support both LLM-based and custom Python code evaluators.
-While you can specify evaluators to grade the results of your experiments programmatically (see [this guide](./evaluate_llm_application) for more information), you can also bind evaluators to a dataset in the UI.
+You can grade your experiment results in two ways: **programmatically**, by specifying evaluators in your code (see [this guide](./evaluate_llm_application) for details) or you can **automatically run evaluators** defined in the UI. By binding evaluators to a dataset in the UI, they'll automatically run on your experiments. These UI-configured evaluators complement any you've set up via the SDK. We support both LLM-based and custom Python code evaluators.
 This allows you to configure automatic evaluators that grade your experiment results. We have support for both LLM-based evaluators, and custom python code evaluators.
@@ -13,40 +13,18 @@ The process for configuring this is very similar to the process for configuring
 When you configure an evaluator for a dataset, it will only affect the experiment runs that are created after the evaluator is configured. It will not affect the evaluation of experiment runs that were created before the evaluator was configured.
 :::
 
-1. **Navigate to the dataset details page** by clicking **Datasets and Testing** in the sidebar and selecting the dataset you want to configure the evaluator for.
-2. **Click on the `Add Auto-Evaluator` button** to add an evaluator to the dataset. This will open a modal you can use to configure the evaluator.
+1. **Navigate to the dataset details page** by clicking **Datasets and Experiments** in the sidebar and selecting the dataset you want to configure the evaluator for.
+2. **Click on the `+ Evaluator` button** to add an evaluator to the dataset. This will open a pane you can use to configure the evaluator.
 
 The next steps vary based on the evaluator type.
 
-## LLM as judge evaluators
+## LLM as a judge evaluators
 
-1. **Select the LLM as judge type evaluator**
-2. **Give your evaluator a name** and **set an inline prompt or load a prompt from the prompt hub** that will be used to evaluate the results of the runs in the experiment.
+[See this page for instructions on setting up an LLM-as-a-judge evaluator](./llm_as_judge?mode=ui#customize-your-llm-as-a-judge-evaluator)
 
-![Add evaluator name and prompt](./static/create_evaluator.png)
+Once you have saved your new evaluator, **subsequent** experiment run from the dataset will now be evaluated by the evaluator you configured. Note that in the below image, each run in the experiment has a "correctness" score.
-Once you have saved your new evaluator, **subsequent** experiment run from the dataset will now be evaluated by the evaluator you configured. Note that in the below image, each run in the experiment has a "correctness" score.
+Once you have created an evaluator, **subsequent** experiments on that dataset will be automatically graded by the evaluator(s) you configured.
-Once you have saved your new evaluator, **subsequent** experiment run from the dataset will now be evaluated by the evaluator you configured. Note that in the below image, each run in the experiment has a "correctness" score.
+Once you have created an evaluator, **subsequent** experiments on that dataset will be automatically graded by the evaluator(s) you configured.
 
-Importantly, evaluator prompts can only contain the following input variables:
-
-- `input` (required): the input to the target you are evaluating
-- `output` (required): the output of the target you are evaluating
-- `reference`: the reference output, taken from the dataset
-
-:::note
-
-Automatic evaluators you configure in the application will only work if the `inputs` to your evaluation target, `outputs` from your evaluation target, and `examples` in your dataset are all single-key dictionaries.
-LangSmith will automatically extract the values from the dictionaries and pass them to the evaluator.
-
-LangSmith currently doesn't support setting up evaluators in the application that act on multiple keys in the `inputs` or `outputs` or `examples` dictionaries.
-
-:::
-
-You can specify the scoring criteria in the "schema" field. In this example, we are asking the LLM to grade on "correctness" of the output with respect to the reference, with a boolean output of 0 or 1. The name of the field in the schema will be interpreted as the feedback key and the type will be the type of the score.
-
-![Evaluator prompt](./static/evaluator_prompt.png)
-
-3. **Save the evaluator** and navigate back to the dataset details page. Each **subsequent** experiment run from the dataset will now be evaluated by the evaluator you configured. Note that in the below image, each run in the experiment has a "correctness" score.
-
-![Playground evaluator results](./static/playground_evaluator_results.png)
+![Playground evaluator results](./static/experiment_view_results.png)
 
 ## Custom code evaluators
 

diff --git a/docs/evaluation/how_to_guides/index.md b/docs/evaluation/how_to_guides/index.md
@@ -59,7 +59,7 @@ Evaluate and improve your application before deploying it.
 
 Set up evaluators that automatically run for all experiments against a dataset.
 
-- [Set up an auto-evaluator](./how_to_guides/bind_evaluator_to_dataset)
+- [Automatically run evaluators on experiments](./how_to_guides/bind_evaluator_to_dataset)
 - [Create a few-shot evaluator](./how_to_guides/create_few_shot_evaluators)
 
 ## Testing integrations

diff --git a/docs/evaluation/how_to_guides/llm_as_judge.mdx b/docs/evaluation/how_to_guides/llm_as_judge.mdx
@@ -100,7 +100,7 @@ See [here](./custom_evaluator) for more on how to write a custom evaluator.
 
   You can configure these evaluators::
   - When running an evaluation using the [playground](/prompt_engineering/concepts#prompt-playground)
-  - As part of a dataset to [automatically run experiments over a dataset](/evaluation/how_to_guides/bind_evaluator_to_dataset)
+  - As part of a dataset to [automatically run evaluations on experiments](/evaluation/how_to_guides/bind_evaluator_to_dataset)
   - When running an [online evaluation](/observability/how_to_guides/online_evaluations#configure-llm-as-judge-evaluators)
 
   ## Customize your LLM-as-a-judge evaluator
@@ -113,7 +113,7 @@ See [here](./custom_evaluator) for more on how to write a custom evaluator.
   - In the playground or from a dataset: Select the **+Evaluator** button
   - From a tracing project: Select **Add rules**, configure your rule and select **Apply evaluator**
 
-  Select the **Create your own evaluator option**. Alternativley, you may start by selecting a pre-built evaluator and editing it. 
+  Select the **Create your own evaluator option**. Alternatively, you may start by selecting a pre-built evaluator and editing it. 
 
 ### Configure the evaluator
 

diff --git a/docs/evaluation/how_to_guides/static/create_evaluator.png b/docs/evaluation/how_to_guides/static/create_evaluator.png
diff --git a/docs/evaluation/how_to_guides/static/experiment_view_results.png b/docs/evaluation/how_to_guides/static/experiment_view_results.png
diff --git a/docs/evaluation/how_to_guides/static/feedback_configuration.png b/docs/evaluation/how_to_guides/static/feedback_configuration.png