solr search backend #3683

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Open

thoniTUB wants to merge 60 commits into develop from feature/solr-search

Collaborator

thoniTUB commented Mar 3, 2025

No description provided.


          wip: refactor search related classes

e5a989a

thoniTUB added backend refactoring labels

thoniTUB self-assigned this


          wip first solr tests (test-external solr needed)

04a4a5b

thoniTUB requested a review from awildturtok as a code owner

March 4, 2025 08:26

thoniTUB added 24 commits

March 5, 2025 10:44


          adds solr health check

bf437c8


          clean up IndexServiceTest

b2307f9


          wip solr search

dcad9cd


          one solr collection/core per dataset

50d17b4


          rework exact search

117e747


          improve non-exact search


          use solr's pagination

0ab9598


          use regex wild card instead of fuzzy

8aebd9b


          commit after finalization of search

40dc66c


          remove "duplicates" from query

11dc7a8


          revert checking for duplicates

3ae2f3a


          use commitWithin to ease update pressure

67acc38


          adds handling of null and empty strings

a95985d


          Merge branch 'develop' into feature/solr-search

5e7556d


          calculate size for SolrSearch

2b93eb7


          gather docs to send larger chunks

8068ad7


          improve LabelMap toString

2c442bb


          clear openDocs after submission

b57844d


          revert mapping csv reference in IndexServiceTest

70a456a


          clean up chunking

bd76e50


          rework source priority sorting

0f18dc1


          use progress reporter

987ce23


          make chunk size configurable

212298b


          avoid sending values multiple times

44ae11c


          Merge branch 'develop' into feature/solr-search

b213288

awildturtok requested changes

View reviewed changes

backend/src/main/java/com/bakdata/conquery/models/config/search/SolrConfig.java Outdated Show resolved Hide resolved

backend/src/main/java/com/bakdata/conquery/models/config/search/SolrConfig.java Outdated Show resolved Hide resolved

backend/src/main/java/com/bakdata/conquery/models/config/search/InternalSearchConfig.java Show resolved Hide resolved

backend/src/main/java/com/bakdata/conquery/models/config/search/SolrConfig.java Outdated Show resolved Hide resolved

backend/src/main/java/com/bakdata/conquery/models/config/search/SolrConfig.java Outdated Show resolved Hide resolved

backend/src/main/java/com/bakdata/conquery/util/search/solr/CombinedSolrSearch.java Outdated Show resolved Hide resolved

backend/src/main/java/com/bakdata/conquery/util/search/solr/SolrProcessor.java Outdated Show resolved Hide resolved

backend/src/main/java/com/bakdata/conquery/util/search/solr/SolrSearch.java Outdated Show resolved Hide resolved

backend/src/main/java/com/bakdata/conquery/util/search/solr/SolrSearch.java Outdated Show resolved Hide resolved

backend/src/test/java/com/bakdata/conquery/integration/tests/FilterAutocompleteTest.java Outdated Show resolved Hide resolved

thoniTUB added 10 commits

June 13, 2025 11:08


          special "no value" entry added

772e9a8


          Merge branch 'develop' into feature/solr-search

99e7cfa


          fix build error

879dd6b


          sort initial search by source prio and label

a9b0c5d


          improve log line

849302a


          cleanup

4eba87a


          cleanup and fix test

1c77a27


          group columns based on secondary id, use concurrent import client, un…

9f22e28

…couple cluster chunk size from solr chunk size


          fix skipping of column values

9ab9f4f


          fixes healthcheck, makes clients refreshable

ae6ab2d

thoniTUB requested a review from awildturtok

June 23, 2025 06:48


          submitts CollectColumnValuesJob to jobmanager

e810ee9

awildturtok requested changes

View reviewed changes

backend/src/main/java/com/bakdata/conquery/apiv1/FilterTemplate.java Outdated Show resolved Hide resolved

backend/src/main/java/com/bakdata/conquery/models/config/search/InternalSearchConfig.java Show resolved Hide resolved

backend/src/main/java/com/bakdata/conquery/models/config/search/InternalSearchConfig.java Outdated

Comment on lines 54 to 66

+              	public TrieSearch<FrontendValue> createSearch(Searchable searchable) {
+              		if (searchable instanceof FilterTemplate temp) {
+              			return getFilterTemplateSearch(temp);
+              		}
+              		if (searchable instanceof LabelMap labelMap) {
+              			return getLabelMapSearch(labelMap);
+              		}
+              		return new TrieSearch<>(searchable.isGenerateSuffixes() ? getNgramLength() : Integer.MAX_VALUE, getSearchSplitChars());
+              	}

Collaborator

awildturtok Jun 23, 2025

kannst du Searchable sealen und das dann über ein switch lösen? Das würde hier klarer machen, was das else ist.

Collaborator Author

thoniTUB Jun 24, 2025

Dann müssen alle Searchables in einem package liegen, das ist auch unschön 🤔

Collaborator

awildturtok Jun 24, 2025

Also ich glaube nicht, dass du es sealen musst um switch zu bekommen. Im gleichen Package wäre tatsächlich sehr schlecht für uns.

backend/src/main/java/com/bakdata/conquery/models/config/search/solr/FilterValueConfig.java Outdated Show resolved Hide resolved

backend/src/main/java/com/bakdata/conquery/models/config/search/solr/FilterValueConfig.java Outdated Show resolved Hide resolved

backend/src/main/java/com/bakdata/conquery/util/search/solr/FilterValueSearch.java Outdated Show resolved Hide resolved

backend/src/main/java/com/bakdata/conquery/util/search/solr/FilterValueSearch.java Outdated Show resolved Hide resolved

backend/src/main/java/com/bakdata/conquery/util/search/solr/SolrProcessor.java Outdated Show resolved Hide resolved

backend/src/main/java/com/bakdata/conquery/util/search/solr/SolrProcessor.java Show resolved Hide resolved

backend/src/test/java/com/bakdata/conquery/integration/tests/FilterAutocompleteTest.java Show resolved Hide resolved

thoniTUB added 9 commits

June 24, 2025 08:36


          move SearchShrinkerJob to InternalFilterSearch

5a749b2


          deprecate InternalFilterSearch


          use StringSubstitutor for query template

65e44c3


          smaller review changes

b656fc6


          Merge remote-tracking branch 'origin/develop' into feature/solr-search

867dee0


          improve default query template

a4afc52


          wip: performant filter value resolve

4e29009


          cleanup

c5c520d


          merge develop

a4df147

thoniTUB requested a review from Kadrian as a code owner

July 2, 2025 19:28


          reset frontend/src/js/ui-components/InputMultiSelect/InputMultiSelect…

62acadf

….tsx

awildturtok self-requested a review

July 7, 2025 06:55

awildturtok reviewed

View reviewed changes

Collaborator

awildturtok left a comment

Habe noch ein paar große Klassen vor mir

backend/src/main/java/com/bakdata/conquery/models/config/search/solr/FilterValueConfig.java

+                   * If you have a complex expression, consider to surround your template with parentheses, so the concatenating "AND"s apply to the whole template.
+                   */
+                  @NotEmpty
+                  private String queryTemplate = "( ${term} value_s:\"${term}\"^100 )";

Collaborator

awildturtok Jul 7, 2025

danke, deutlich lesbarer!

backend/src/main/java/com/bakdata/conquery/models/config/search/solr/SolrConfig.java

+              			builder.withBasicAuthCredentials(username, password);
+              		}
+                      return builder.build();

Collaborator

awildturtok Jul 7, 2025

hier hast du keinen ConcurrentClient verwendet, ist das absicht?

backend/src/main/java/com/bakdata/conquery/models/execution/ManagedExecution.java

               	@Getter
               	@JsonIgnore
               	@EqualsAndHashCode.Exclude
+              	@ToString.Exclude

Collaborator

awildturtok Jul 7, 2025

=> onlyExplicitlyIncluded?

backend/src/main/java/com/bakdata/conquery/util/search/solr/FilterValueIndexer.java

+              	/**
+              	 * Buffer docs to send larger update chunks (size {@link FilterValueIndexer#updateChunkSize}).
+              	 */
+              	private final LinkedList<SolrFrontendValue> openDocs = new LinkedList<>();

Collaborator

awildturtok Jul 7, 2025

Suggested change

      
            	private final LinkedList<SolrFrontendValue> openDocs = new LinkedList<>();
          
            	private final Queue<SolrFrontendValue> openDocs = new LinkedList<>();

backend/src/main/java/com/bakdata/conquery/util/search/solr/FilterValueIndexer.java

+              	@Override
+              	public List<FrontendValue> findExact(String searchTerm, int maxValue) {
+              		throw new UnsupportedOperationException("Must not be used. Use " + FilterValueSearch.class);

Collaborator

awildturtok Jul 7, 2025

Warum keine delegation?

backend/src/main/java/com/bakdata/conquery/util/search/solr/FilterValueIndexer.java

+              	@Override
+              	public Iterator<FrontendValue> iterator() {
+              		throw new UnsupportedOperationException("Must not be used. Use " + FilterValueSearch.class);

Collaborator

awildturtok Jul 7, 2025

Wo wird der verwendet?

backend/src/main/java/com/bakdata/conquery/util/search/solr/FilterValueIndexer.java

Comment on lines +129 to +132

+              		while (openDocs.size() >= updateChunkSize) {
+              			log.trace("Adding {} documents for {}", openDocs.size(), searchable);
+              			registerValues(openDocs);
+              			openDocs.clear();

Collaborator

awildturtok Jul 7, 2025

Ich sehe nicht wo du hier chunking machst

backend/src/main/java/com/bakdata/conquery/util/search/solr/FilterValueIndexer.java Show resolved Hide resolved

backend/src/main/java/com/bakdata/conquery/util/search/solr/FilterValueSearch.java

+                                  .filter(Predicate.not(String::isBlank))
+                                  // Escape
+                                  .map(ClientUtils::escapeQueryChars)
+                                  //

Collaborator

awildturtok Jul 7, 2025

// ?

:D


          don't use commit within, to reduce the number of commits

073a208

thoniTUB force-pushed the feature/solr-search branch from 64bf39f to 073a208 Compare

July 11, 2025 13:47

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend refactoring