Scrolling with Qdrant connector

[lilhoser]

I am trying to use the SK qdrant connector to pull all points in a collection, but I've not had any success. I understand that Qdrant is not well suited for keyword searches, and that it's built for vector searches. But for the sake of argument, let's assume "get all the records" is a necessary thing to do. I see only 3 ways to do it:

1. Call Collection.GetAsync with a filter that always returns true - I simply cannot get this to work. I'm using a very simple linq expression "_ => true" and keep getting GetAsync: System.NotSupportedException: Invalid equality/comparison
2. Bypass the connector entirely and use the qdrant client to call ScrollAsync - well that's fine and works, but then I have no convenient way to map returned qdrant point data to my record (all the SK mappers are marked internal/private).
3. Use SK connector's Search or HybridSearch, but both of these require either a vector or a search value that is translated to a vector via embeddings. There is no way to pass a "neutral vector", as neither prototype accepts a vector directly.

My understanding is qdrant added scrolling to support this corner case of "get all the records" as it was a legitimate feature, but I'm not seeing how to accomplish it in SK.

[westey-m]

Hi @lilhoser, thanks for sharing this issue. This is definitely a gap in our interface and I've logged an issue to consider adding this to the abstraction: #12875

In the mean time there are two possible workarounds, which granted may be a little bit hacky.

Workaround 1

You could define a nullable dummy string field on your schema that you always leave as null:

        [VectorStoreData]
        public string? DummyString { get; set; }

You can then filter on this dummy field to any records that have this field null (all of them):

collection.GetAsync(x => x.DummyString == null, 5)

Workaround 2

Use SK connector's Search or HybridSearch, but both of these require either a vector or a search value that is translated to a vector via embeddings. There is no way to pass a "neutral vector", as neither prototype accepts a vector directly.

You can actually make up a vector. For vector search, records are not filtered by the vector, they are just ordered by the vector similarity. So unless ordering is really important for you, can you make up a vector that is all 1s (or really any number), and just search with that. If ordering is important I'd say Workaround 1 makes more sense.