Skip to content

Add Sparse Vector Index Options for Token Pruning to Specification #4761

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jul 3, 2025

Conversation

markjhoy
Copy link
Contributor

@markjhoy markjhoy commented Jul 2, 2025

Index options for token pruning configurations were introduced for sparse vector field types in 8.19/9.1 via this PR. This PR adds the corresponding specification definitions for this.

# Conflicts:
#	output/schema/schema.json
@markjhoy markjhoy force-pushed the markjhoy/sparse_vector_index_options branch from 47f8758 to c3b758e Compare July 2, 2025 21:04
Copy link
Member

@kderusso kderusso left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall LGTM but one question about pruning config

@@ -6022,9 +6022,15 @@ export interface MappingSourceField {

export type MappingSourceFieldMode = 'disabled' | 'stored' | 'synthetic'

export interface MappingSparseVectorIndexOptions {
prune?: boolean
pruning_config?: QueryDslTokenPruningConfig
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we have both QueryDslTokenPruningConfig and TokenPruningConfig? Can we consolidate?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know - this file is one of the ones automatically generated when running make contrib

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmm... I'm pretty sure it added the QueryDsl* prefix as it imports the TokenPruningConfig from there ('@_types/query_dsl/TokenPruningConfig')... @kderusso - WDYT about moving the TokenPruningConfig to a more common place? I don't think it would break backwards compatibility (as the final output structure would be the same)...?

@markjhoy markjhoy requested review from kderusso and a team July 3, 2025 16:58
Copy link
Member

@kderusso kderusso left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@markjhoy markjhoy merged commit 0c77ae9 into main Jul 3, 2025
8 checks passed
@markjhoy markjhoy deleted the markjhoy/sparse_vector_index_options branch July 3, 2025 18:23
Copy link
Contributor

github-actions bot commented Jul 3, 2025

The backport to 8.19 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-8.19 8.19
# Navigate to the new working tree
cd .worktrees/backport-8.19
# Create a new branch
git switch --create backport-4761-to-8.19
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 0c77ae9561b54c103dd102de6cde777841504596
# Push it to GitHub
git push --set-upstream origin backport-4761-to-8.19
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-8.19

Then, create a pull request where the base branch is 8.19 and the compare/head branch is backport-4761-to-8.19.

Copy link
Contributor

github-actions bot commented Jul 3, 2025

The backport to 9.1 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-9.1 9.1
# Navigate to the new working tree
cd .worktrees/backport-9.1
# Create a new branch
git switch --create backport-4761-to-9.1
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 0c77ae9561b54c103dd102de6cde777841504596
# Push it to GitHub
git push --set-upstream origin backport-4761-to-9.1
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-9.1

Then, create a pull request where the base branch is 9.1 and the compare/head branch is backport-4761-to-9.1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants