More improvements to studio2 publishing #2333

aaronvg · 2025-08-18T04:38:21Z

Wire through client details on stream data

Important

Enhance client details handling in HTTP requests, add new metrics and cost structures, and update protobuf message handling.

Behavior:
- Add client_details to HTTPRequest in events.rs and IntermediateData in trace_event.rs.
- Update ApiKeyEnvironment::environment() in api_key.rs to handle longer environment strings.
- Add new metrics and cost statistics structures in ui_dashboard.rs and ui_dashboard_cost.rs.
Protobuf:
- Modify cffi.pb.go to include protoimpl.MessageState in various structs for better message handling.
Misc:
- Add GetFunctionCallHttpCalls endpoint in ui_function_call_http_calls.rs.
- Add llm_only filter to ListTracesRequest in ui_traces.rs.
- Add cost estimates to UiUsageEstimate in ui_types.rs.

^{This description was created by}^{for df0ed86. You can customize this summary. It will automatically update as commits are pushed.}

vercel · 2025-08-18T04:38:27Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Preview	Comments	Updated (UTC)
promptfiddle	Ready	Preview	Comment	Sep 8, 2025 3:10am

entelligence-ai-pr-reviews · 2025-08-18T04:39:13Z

Review Summary

entelligence-ai-pr-reviews · 2025-08-18T04:39:14Z

LGTM 👍

ellipsis-dev

Caution

Changes requested ❌

Reviewed everything up to 4a84bd2 in 1 minute and 19 seconds. Click for details.

Reviewed 158 lines of code in 4 files
Skipped 0 files when reviewing.
Skipped posting 4 draft comments. View those below.
Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.

1. engine/baml-rpc/src/ui/mod.rs:5

Draft comment:
New module 'ui_dashboard_cost' added. Consider keeping module ordering consistent with similar modules for clarity.
Reason this comment was not posted:
Confidence changes required: 33% <= threshold 50% None

2. engine/baml-rpc/src/ui/ui_dashboard.rs:188

Draft comment:
GetDashboardDataResponse now includes new metric fields. If these values might sometimes be missing, consider making them optional or ensuring backend always provides them.
Reason this comment was not posted:
Confidence changes required: 33% <= threshold 50% None

3. engine/baml-rpc/src/ui/ui_dashboard_cost.rs:9

Draft comment:
The cost stats endpoint defines its own TimeSeriesFloatPoint and MetricSeriesFloat types. Consider reusing shared types to avoid duplication across dashboard endpoints.
Reason this comment was not posted:
Decided after close inspection that this draft comment was likely wrong and/or not actionable: usefulness confidence = 20% vs. threshold = 50% While code deduplication is generally good, we don't have strong evidence that duplicate types exist elsewhere. The comment requires knowledge of other files to verify. Following our rules, we should err on the side of deletion when we need more context. Additionally, these types might have subtle differences from other similar types that justify their separate existence. The comment could be valid if there really are identical types defined elsewhere. Type duplication is a legitimate concern for maintainability. Without seeing the other files to confirm the duplication, we can't be certain this comment is correct or actionable. We need strong evidence to keep comments. Delete the comment since we lack strong evidence of type duplication and would need to look at other files to verify the claim.

4. engine/baml-rpc/src/ui/ui_types.rs:157

Draft comment:
Cost estimate fields (input_cost, output_cost, total_cost) have been added to UiUsageEstimate. Ensure these are computed and populated consistently across providers.
Reason this comment was not posted:
Confidence changes required: 33% <= threshold 50% None

Workflow ID: wflow_P9sAxFivRUEl1DE2

^{You can customize}^{by changing your verbosity settings, reacting with 👍 or 👎, replying to comments, or adding code review rules.}

ellipsis-dev · 2025-08-18T04:39:44Z

engine/baml-rpc/src/ui/ui_dashboard.rs

@@ -151,13 +151,48 @@ pub struct StatusCountOverTime {
    pub error_count: u64,
 }

+#[derive(Debug, Serialize, Deserialize, TS)]


New time series and metric types (TimeSeriesFloatPoint, TimeSeriesIntPoint, MetricSeriesFloat, MetricSeriesInt) are defined here. These are duplicated in ui_dashboard_cost. Consider extracting shared types to a common module to reduce code duplication.

github-actions · 2025-08-18T04:40:10Z

🌿 Preview your docs: https://boundary-preview-6383cdca-7d06-41d5-9682-b4a7e97f216c.docs.buildwithfern.com

github-actions · 2025-08-18T04:40:58Z

https://sage-backend-6kor8fhva-baml.vercel.app

codecov · 2025-08-18T04:44:40Z

Codecov Report

❌ Patch coverage is 43.33333% with 17 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...ntime/src/internal/llm_client/primitive/request.rs	0.00%	5 Missing ⚠️
engine/baml-runtime/src/lib.rs	0.00%	5 Missing ⚠️
...c/tracingv2/publisher/rpc_converters/trace_data.rs	0.00%	5 Missing ⚠️
engine/baml-rpc/src/auth/api_key.rs	0.00%	1 Missing ⚠️
...rc/internal/llm_client/primitive/aws/aws_client.rs	0.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

ellipsis-dev

Important

Looks good to me! 👍

Reviewed 864ef02 in 1 minute and 15 seconds. Click for details.

Reviewed 121 lines of code in 6 files
Skipped 0 files when reviewing.
Skipped posting 9 draft comments. View those below.
Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.

1. engine/baml-runtime/src/tracingv2/publisher/rpc_converters/trace_data.rs:34

Draft comment:
Ensure the recursive call to self.to_rpc_event(lookup) here correctly resolves to the impl returning Option (for baml_function_content) without ambiguity.
Reason this comment was not posted:
Confidence changes required: 33% <= threshold 50% None

2. engine/baml-runtime/src/tracingv2/publisher/rpc_converters/trace_data.rs:60

Draft comment:
Verify that using lookup.function_lookup(&self.name) with a fallback to 'unknown_hash' is the desired behavior when function_type is BamlLlm.
Reason this comment was not posted:
Confidence changes required: 33% <= threshold 50% None

3. engine/baml-runtime/src/tracingv2/publisher/rpc_converters/trace_data.rs:124

Draft comment:
In the LoggedLLMRequest conversion, params are mapped using Cow::Borrowed. Ensure that the lifetimes of these borrowed values remain valid for the RPC event.
Reason this comment was not posted:
Confidence changes required: 0% <= threshold 50% None

4. engine/baml-runtime/src/tracingv2/publisher/rpc_converters/trace_data.rs:156

Draft comment:
The conversion for LLMChatMessagePart handles Text, Media, and WithMeta variants; double-check that boxing and cloning are optimal and that no unnecessary allocations occur.
Reason this comment was not posted:
Confidence changes required: 0% <= threshold 50% None

5. engine/baml-runtime/src/tracingv2/publisher/rpc_converters/trace_data.rs:195

Draft comment:
Using Cow::Borrowed in the HTTPBody conversion is efficient; ensure that self.raw() remains valid for the RPC lifetime.
Reason this comment was not posted:
Confidence changes required: 0% <= threshold 50% None

6. engine/baml-runtime/src/tracingv2/publisher/rpc_converters/trace_data.rs:211

Draft comment:
In the HTTPRequest conversion, headers are passed through redact_headers after cloning. Confirm that this redaction meets security requirements.
Reason this comment was not posted:
Confidence changes required: 0% <= threshold 50% None

7. engine/baml-runtime/src/tracingv2/publisher/rpc_converters/trace_data.rs:237

Draft comment:
For HTTPResponse conversion, ensure that optional headers are handled correctly when cloning and applying redact_headers.
Reason this comment was not posted:
Confidence changes required: 0% <= threshold 50% None

8. engine/baml-runtime/src/tracingv2/publisher/rpc_converters/trace_data.rs:259

Draft comment:
In the HTTPResponseStream conversion, using Cow::Borrowed(&self.event.data) should be reviewed to ensure the event data remains valid during RPC processing.
Reason this comment was not posted:
Confidence changes required: 0% <= threshold 50% None

9. engine/baml-runtime/src/tracingv2/publisher/rpc_converters/trace_data.rs:267

Draft comment:
The LoggedLLMResponse conversion maps optional raw_text_output to a Cow; verify that this conversion correctly represents absence or presence of text without data loss.
Reason this comment was not posted:
Confidence changes required: 0% <= threshold 50% None

Workflow ID: wflow_9PwQJy5vd7yGrkDP

^{You can customize}^{by changing your verbosity settings, reacting with 👍 or 👎, replying to comments, or adding code review rules.}

github-actions · 2025-09-03T02:16:27Z

🌿 Preview your docs: https://boundary-preview-5dacf81b-482e-46af-adbd-07c79163c90e.docs.buildwithfern.com

github-actions · 2025-09-03T02:16:52Z

https://sage-backend-amnql2od8-baml.vercel.app

github-actions · 2025-09-03T21:18:11Z

🌿 Preview your docs: https://boundary-preview-822eaa11-481d-496d-8b18-125a2d69779f.docs.buildwithfern.com

ellipsis-dev

Important

Looks good to me! 👍

Reviewed ba1a717 in 2 minutes and 4 seconds. Click for details.

Reviewed 37 lines of code in 2 files
Skipped 0 files when reviewing.
Skipped posting 2 draft comments. View those below.
Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.

1. engine/baml-rpc/src/auth/api_key.rs:42

Draft comment:
Fallback slice length changed from 3 to 4. Confirm this is intentional and that it maintains consistency with the predefined codes ('dev', 'stg', 'prod'). Also, consider Unicode safety.
Reason this comment was not posted:
Decided after close inspection that this draft comment was likely wrong and/or not actionable: usefulness confidence = 10% vs. threshold = 50% The comment has multiple issues: 1) It asks for confirmation of intention which violates our rules 2) The consistency concern is not valid since the predefined codes are hardcoded separately 3) While Unicode safety could be a real issue, the comment doesn't provide evidence that it's actually problematic in this context 4) The change from 3 to 4 chars seems like a minor change that doesn't warrant discussion. The Unicode safety concern could be legitimate since string slicing in Rust can panic on invalid UTF-8 boundaries. While Unicode safety is a valid concern in general, this code is working with a String type which guarantees valid UTF-8, and len().min(4) ensures we won't slice beyond string bounds. Delete this comment as it primarily asks for confirmation of intention and raises concerns without strong evidence of actual problems.

2. engine/baml-runtime/src/tracingv2/publisher/rpc_converters/mod.rs:618

Draft comment:
Added 'client_details' in the RawLLMRequest test. Ensure this test data reflects the production API requirements and remains updated if the RPC schema changes.
Reason this comment was not posted:
Comment did not seem useful. Confidence is useful = 0% <= threshold 50% The comment is asking the PR author to ensure that the test data reflects production API requirements and remains updated if the RPC schema changes. This falls under asking the author to ensure something is tested or verified, which is against the rules.

Workflow ID: wflow_YtlHc4jHPn0iFqoh

^{You can customize}^{by changing your verbosity settings, reacting with 👍 or 👎, replying to comments, or adding code review rules.}

github-actions · 2025-09-03T21:46:24Z

https://sage-backend-co4fcy998-baml.vercel.app

github-actions · 2025-09-06T17:45:03Z

🌿 Preview your docs: https://boundary-preview-d1c7be64-073e-4ff2-bc99-abcb885eed86.docs.buildwithfern.com

github-actions · 2025-09-06T17:45:37Z

https://sage-backend-q9rty0rl3-baml.vercel.app

ellipsis-dev

Important

Looks good to me! 👍

Reviewed 869e45e in 1 minute and 19 seconds. Click for details.

Reviewed 36 lines of code in 1 files
Skipped 0 files when reviewing.
Skipped posting 2 draft comments. View those below.
Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.

1. engine/baml-rpc/src/ui/ui_function_call_http_calls.rs:29

Draft comment:
Add a newline at end of file for POSIX compliance.
Reason this comment was not posted:
Comment did not seem useful. Confidence is useful = 0% <= threshold 50% This comment is purely informative and does not provide a specific code suggestion or ask for a specific test. It simply states a general best practice without context.

2. engine/baml-rpc/src/ui/ui_function_call_http_calls.rs:7

Draft comment:
Consider adding Rust doc comments (///) for public structs (e.g., GetFunctionCallHttpCallsRequest and GetFunctionCallHttpCallsResponse) to improve maintainability.
Reason this comment was not posted:
Confidence changes required: 50% <= threshold 50% None

Workflow ID: wflow_oTtcQbAdA3dqXuC4

^{You can customize}^{by changing your verbosity settings, reacting with 👍 or 👎, replying to comments, or adding code review rules.}

github-actions · 2025-09-08T01:03:26Z

🌿 Preview your docs: https://boundary-preview-a804f65d-d5f0-416b-8cd8-6b29f26401c7.docs.buildwithfern.com

github-actions · 2025-09-08T01:03:50Z

https://sage-backend-n0yauvk72-baml.vercel.app

github-actions · 2025-09-08T02:57:08Z

🌿 Preview your docs: https://boundary-preview-cb2bb46a-f9b7-44d8-b692-618dab1df67a.docs.buildwithfern.com

ellipsis-dev

Important

Looks good to me! 👍

Reviewed df0ed86 in 1 minute and 34 seconds. Click for details.

Reviewed 70 lines of code in 4 files
Skipped 0 files when reviewing.
Skipped posting 4 draft comments. View those below.
Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.

1. engine/baml-rpc/src/lib.rs:30

Draft comment:
Reordering of public exports: The ui_function_call_http_calls export has been moved before ui_function_calls. This reordering is non-functional but check that it aligns with the project's conventions to avoid any confusion in consumers.
Reason this comment was not posted:
Confidence changes required: 33% <= threshold 50% None

2. engine/baml-rpc/src/ui/mod.rs:6

Draft comment:
Module declaration reordering: The ui_function_calls module is now declared after ui_function_call_http_calls. Although this does not change behavior, ensuring a consistent ordering (e.g. alphabetical or logical grouping) helps maintain clarity.
Reason this comment was not posted:
Confidence changes required: 33% <= threshold 50% None

3. engine/baml-rpc/src/ui/ui_dashboard_cost.rs:12

Draft comment:
Formatting update for struct fields: Multi-line formatting for fields in GetCostStatsRequest improves readability. Please ensure this style is consistent with other similar definitions in the project.
Reason this comment was not posted:
Confidence changes required: 0% <= threshold 50% None

4. engine/baml-rpc/src/ui/ui_function_call_http_calls.rs:29

Draft comment:
Trailing newline added at EOF: This small formatting fix ensures consistency with standard style guidelines.
Reason this comment was not posted:
Confidence changes required: 0% <= threshold 50% None

Workflow ID: wflow_rOte96drImEHgJ9p

^{You can customize}^{by changing your verbosity settings, reacting with 👍 or 👎, replying to comments, or adding code review rules.}

github-actions · 2025-09-08T03:02:31Z

🌿 Preview your docs: https://boundary-preview-e410932e-99bb-4d36-a10d-548a36d37633.docs.buildwithfern.com

github-actions · 2025-09-08T03:12:24Z

https://sage-backend-kqrl45vo3-baml.vercel.app

seawatts and others added 3 commits August 14, 2025 11:47

feat: start to add more dashboard queries

03ed195

types

e053a41

dashboard types

4a84bd2

aaronvg temporarily deployed to boundary-tools-dev August 18, 2025 04:38 — with GitHub Actions Inactive

ellipsis-dev bot reviewed Aug 18, 2025

View reviewed changes

Fix client details not appearing for streams

864ef02

vercel bot temporarily deployed to Preview September 3, 2025 02:14 Inactive

aaronvg temporarily deployed to boundary-tools-dev September 3, 2025 02:14 — with GitHub Actions Inactive

ellipsis-dev bot reviewed Sep 3, 2025

View reviewed changes

aaronvg added 2 commits September 3, 2025 13:39

Merge remote-tracking branch 'origin/canary' into aug-14-studio

f090ef8

Fix client details test, and fix api key name

ba1a717

aaronvg temporarily deployed to boundary-tools-dev September 3, 2025 21:16 — with GitHub Actions Inactive

ellipsis-dev bot reviewed Sep 3, 2025

View reviewed changes

vercel bot had a problem deploying to Preview September 3, 2025 21:45 Failure

aaronvg added 2 commits September 7, 2025 18:01

add new api to get http data

2fdda5d

actually add api

869e45e

vercel bot temporarily deployed to Preview September 8, 2025 01:01 Inactive

aaronvg temporarily deployed to boundary-tools-dev September 8, 2025 01:01 — with GitHub Actions Inactive

ellipsis-dev bot reviewed Sep 8, 2025

View reviewed changes

Merge branch 'canary' into aug-14-studio

b51bafb

aaronvg temporarily deployed to boundary-tools-dev September 8, 2025 02:55 — with GitHub Actions Inactive

aaronvg had a problem deploying to boundary-tools-dev September 8, 2025 02:55 — with GitHub Actions Error

aaronvg temporarily deployed to boundary-tools-dev September 8, 2025 02:55 — with GitHub Actions Inactive

aaronvg changed the title ~~aug 14 studio~~ More improvements to studio2 publishing Sep 8, 2025

cargo fmt

df0ed86

vercel bot temporarily deployed to Preview September 8, 2025 03:00 Inactive

aaronvg temporarily deployed to boundary-tools-dev September 8, 2025 03:00 — with GitHub Actions Inactive

ellipsis-dev bot reviewed Sep 8, 2025

View reviewed changes

aaronvg enabled auto-merge September 8, 2025 03:09

aaronvg added this pull request to the merge queue Sep 8, 2025

Merged via the queue into canary with commit 39e731a Sep 8, 2025
24 of 25 checks passed

aaronvg deleted the aug-14-studio branch September 8, 2025 03:20

More improvements to studio2 publishing #2333

More improvements to studio2 publishing #2333

Uh oh!

Conversation

aaronvg commented Aug 18, 2025 • edited by ellipsis-dev bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vercel bot commented Aug 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

entelligence-ai-pr-reviews bot commented Aug 18, 2025

Review Summary

Uh oh!

entelligence-ai-pr-reviews bot commented Aug 18, 2025

Uh oh!

ellipsis-dev bot left a comment

Choose a reason for hiding this comment

Uh oh!

ellipsis-dev bot Aug 18, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Aug 18, 2025

Uh oh!

github-actions bot commented Aug 18, 2025

Uh oh!

codecov bot commented Aug 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ellipsis-dev bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Sep 3, 2025

Uh oh!

github-actions bot commented Sep 3, 2025

Uh oh!

github-actions bot commented Sep 3, 2025

Uh oh!

ellipsis-dev bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Sep 3, 2025

Uh oh!

github-actions bot commented Sep 6, 2025

Uh oh!

github-actions bot commented Sep 6, 2025

Uh oh!

ellipsis-dev bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Sep 8, 2025

Uh oh!

github-actions bot commented Sep 8, 2025

Uh oh!

github-actions bot commented Sep 8, 2025

Uh oh!

ellipsis-dev bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Sep 8, 2025

Uh oh!

github-actions bot commented Sep 8, 2025

Uh oh!

Uh oh!

Uh oh!

aaronvg commented Aug 18, 2025 •

edited by ellipsis-dev bot

Loading

vercel bot commented Aug 18, 2025 •

edited

Loading

codecov bot commented Aug 18, 2025 •

edited

Loading