Skip to content

Modifying Replay Deserializer To Parse Tap Message #7439

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Oct 9, 2025
Merged

Modifying Replay Deserializer To Parse Tap Message #7439

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
d8c117e
modify deserializer
cliffordxing Oct 2, 2025
842d42c
updating snap
cliffordxing Oct 2, 2025
cf0af80
snapshot updates
cliffordxing Oct 3, 2025
2bc04c6
snap
cliffordxing Oct 3, 2025
e639d6d
more snapshot updates!
cliffordxing Oct 3, 2025
1357479
adding unit test for tap event
cliffordxing Oct 6, 2025
548d8c8
default string changes
cliffordxing Oct 9, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
131 changes: 129 additions & 2 deletions rust_snuba/src/processors/replays.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,12 +49,11 @@ pub fn deserialize_message(
click_testid: click.testid,
click_text: click.text,
click_title: click.title,
environment: event.environment.clone().unwrap_or("".to_string()),
environment: event.environment.clone().unwrap_or_default(),
error_sample_rate: -1.0,
event_hash: click.event_hash,
offset,
partition,
platform: "".to_string(),
project_id: replay_message.project_id,
replay_id: replay_message.replay_id,
retention_days: replay_message.retention_days,
Expand All @@ -63,6 +62,26 @@ pub fn deserialize_message(
..Default::default()
})
.collect(),
ReplayPayload::TapEvent(event) => event
.taps
.into_iter()
.map(|tap| ReplayRow {
tap_message: tap.message,
tap_view_class: tap.view_class,
tap_view_id: tap.view_id,
environment: event.environment.clone().unwrap_or_default(),
error_sample_rate: -1.0,
event_hash: tap.event_hash,
offset,
partition,
project_id: replay_message.project_id,
replay_id: replay_message.replay_id,
retention_days: replay_message.retention_days,
session_sample_rate: -1.0,
timestamp: tap.timestamp as u32,
..Default::default()
})
.collect(),
ReplayPayload::Event(event) => {
let event_hash = match (event.event_hash, event.segment_id) {
(None, None) => Uuid::new_v4(),
Expand Down Expand Up @@ -260,6 +279,8 @@ struct ReplayMessage {
enum ReplayPayload {
#[serde(rename = "replay_actions")]
ClickEvent(ReplayClickEvent),
#[serde(rename = "replay_tap")]
TapEvent(ReplayTapEvent),
#[serde(rename = "replay_event")]
Event(Box<ReplayEvent>),
#[serde(rename = "event_link")]
Expand All @@ -277,6 +298,15 @@ struct ReplayClickEvent {
clicks: Vec<ReplayClickEventClick>,
}

// Replay Tap Event

#[derive(Debug, Deserialize)]
struct ReplayTapEvent {
#[serde(default)]
environment: Option<String>,
taps: Vec<ReplayTapEventTap>,
}

#[derive(Debug, Deserialize)]
struct ReplayClickEventClick {
alt: String,
Expand All @@ -297,6 +327,15 @@ struct ReplayClickEventClick {
title: String,
}

#[derive(Debug, Deserialize)]
struct ReplayTapEventTap {
message: String,
view_class: String,
view_id: String,
event_hash: Uuid,
timestamp: f64,
}

// Replay Event

#[derive(Debug, Serialize, Deserialize)]
Expand Down Expand Up @@ -505,6 +544,9 @@ pub struct ReplayRow {
tags_key: Vec<String>,
#[serde(rename = "tags.value")]
tags_value: Vec<String>,
tap_message: String,
tap_view_class: String,
tap_view_id: String,
Comment on lines +547 to +549
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Potential bug: The new fields tap_message, tap_view_class, and tap_view_id added to ReplayRow are missing corresponding columns in the ClickHouse database schema, which will cause insertion failures.

  • Description: The ReplayRow struct has been updated with new fields tap_message, tap_view_class, and tap_view_id to support tap events. However, the corresponding columns have not been added to the ClickHouse database schema. The schema definition in snuba/datasets/configuration/replays/storages/replays.yaml and the lack of a new database migration file confirm this omission. When the Rust processor serializes a tap event, the resulting JSON will contain these new fields. The subsequent attempt to insert this data into the replays table will fail with a "Missing columns" error from ClickHouse, preventing tap event data from being saved.

  • Suggested fix: Create a new database migration to add the tap_message, tap_view_class, and tap_view_id columns to the replays_local and replays_dist tables in ClickHouse. Also, update the YAML schema configuration at snuba/datasets/configuration/replays/storages/replays.yaml to include these new columns.
    severity: 0.85, confidence: 0.98

Did we get this right? 👍 / 👎 to inform future reviews.

timestamp: u32,
title: Option<String>,
trace_ids: Vec<Uuid>,
Expand Down Expand Up @@ -1005,6 +1047,91 @@ mod tests {
assert_eq!(replay_row.warning_id, Uuid::nil());
}

#[test]
fn test_parse_replay_tap_event() {
let payload = r#"{
"type": "replay_tap",
"replay_id": "048aa04be40243948eb3b57089c519ee",
"environment": "prod",
"taps": [{
"message": "add_attachment",
"view_class": "androidx.appcompat.widget.AppCompatButton",
"view_id": "add_attachment",
"event_hash": "b4370ef8d1994e96b5bc719b72afbf49",
"timestamp": 1702659275
}]
}"#;

let payload_value = payload.as_bytes();

let data = format!(
r#"{{
"payload": {payload_value:?},
"project_id": 1,
"replay_id": "048aa04be40243948eb3b57089c519ee",
"retention_days": 30,
"segment_id": null,
"start_time": 100,
"type": "replay_event"
}}"#
);

let (rows, _) = deserialize_message(data.as_bytes(), 0, 0).unwrap();
let replay_row = rows.first().unwrap();

// Columns in the critical path.
assert_eq!(&replay_row.tap_message, "add_attachment");
assert_eq!(
&replay_row.tap_view_class,
"androidx.appcompat.widget.AppCompatButton"
);
assert_eq!(&replay_row.tap_view_id, "add_attachment");

assert_eq!(
&replay_row.replay_id,
&Uuid::parse_str("048aa04be40243948eb3b57089c519ee").unwrap()
);
assert_eq!(replay_row.retention_days, 30);
assert_eq!(replay_row.segment_id, None);
assert_eq!(&replay_row.environment, "prod");

// Default columns - not providable on this event.
assert_eq!(&replay_row.browser_name, "");
assert_eq!(&replay_row.browser_version, "");
assert_eq!(&replay_row.device_brand, "");
assert_eq!(&replay_row.device_family, "");
assert_eq!(&replay_row.device_model, "");
assert_eq!(&replay_row.device_name, "");
assert_eq!(&replay_row.dist, "");
assert_eq!(&replay_row.os_name, "");
assert_eq!(&replay_row.os_version, "");
assert_eq!(&replay_row.release, "");
assert_eq!(&replay_row.replay_type, "");
assert_eq!(&replay_row.sdk_name, "");
assert_eq!(&replay_row.sdk_version, "");
assert_eq!(&replay_row.user_email, "");
assert_eq!(&replay_row.user_id, "");
assert_eq!(&replay_row.user_name, "");
assert_eq!(&replay_row.user, "");
assert_eq!(replay_row.debug_id, Uuid::nil());
assert_eq!(replay_row.error_id, Uuid::nil());
assert_eq!(replay_row.error_ids, vec![]);
assert_eq!(replay_row.error_sample_rate, -1.0);
assert_eq!(replay_row.fatal_id, Uuid::nil());
assert_eq!(replay_row.info_id, Uuid::nil());
assert_eq!(replay_row.ip_address_v4, None);
assert_eq!(replay_row.ip_address_v6, None);
assert_eq!(replay_row.is_archived, 0);
assert_eq!(replay_row.platform, "".to_string());
assert_eq!(replay_row.replay_start_timestamp, None);
assert_eq!(replay_row.session_sample_rate, -1.0);
assert_eq!(replay_row.title, None);
assert_eq!(replay_row.trace_ids, vec![]);
assert_eq!(replay_row.urls, Vec::<String>::new());
assert_eq!(replay_row.viewed_by_id, 0);
assert_eq!(replay_row.warning_id, Uuid::nil());
}

#[test]
fn test_parse_replay_event_link_event() {
let payload = r#"{
Expand Down
3 changes: 3 additions & 0 deletions ..._schemas@ingest-replay-events-ReplaysProcessor-ingest-replay-events__1__archive.json.snap
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,9 @@ expression: snapshot_payload
"session_sample_rate": -1.0,
"tags.key": [],
"tags.value": [],
"tap_message": "",
"tap_view_class": "",
"tap_view_id": "",
"timestamp": 1702848658,
"title": null,
"trace_ids": [],
Expand Down
3 changes: 3 additions & 0 deletions ...ingest-replay-events-ReplaysProcessor-ingest-replay-events__1__click-serialized.json.snap
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,9 @@ expression: snapshot_payload
"session_sample_rate": -1.0,
"tags.key": [],
"tags.value": [],
"tap_message": "",
"tap_view_class": "",
"tap_view_id": "",
"timestamp": 1702848658,
"title": null,
"trace_ids": [],
Expand Down
3 changes: 3 additions & 0 deletions ...s__schemas@ingest-replay-events-ReplaysProcessor-ingest-replay-events__1__click.json.snap
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,9 @@ expression: snapshot_payload
"session_sample_rate": -1.0,
"tags.key": [],
"tags.value": [],
"tap_message": "",
"tap_view_class": "",
"tap_view_id": "",
"timestamp": 1702848658,
"title": null,
"trace_ids": [],
Expand Down
3 changes: 3 additions & 0 deletions ...hemas@ingest-replay-events-ReplaysProcessor-ingest-replay-events__1__event-link.json.snap
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,9 @@ expression: snapshot_payload
"session_sample_rate": -1.0,
"tags.key": [],
"tags.value": [],
"tap_message": "",
"tap_view_class": "",
"tap_view_id": "",
"timestamp": 1702848658,
"title": null,
"trace_ids": [],
Expand Down
3 changes: 3 additions & 0 deletions ..._schemas@ingest-replay-events-ReplaysProcessor-ingest-replay-events__1__segment.json.snap
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,9 @@ expression: snapshot_payload
"tags.value": [
"aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"
],
"tap_message": "",
"tap_view_class": "",
"tap_view_id": "",
"timestamp": 1687904093,
"title": null,
"trace_ids": [],
Expand Down
3 changes: 3 additions & 0 deletions ...__schemas@ingest-replay-events-ReplaysProcessor-ingest-replay-events__1__viewed.json.snap
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,9 @@ expression: snapshot_payload
"session_sample_rate": -1.0,
"tags.key": [],
"tags.value": [],
"tap_message": "",
"tap_view_class": "",
"tap_view_id": "",
"timestamp": 1712009295,
"title": null,
"trace_ids": [],
Expand Down
Loading