DOC-12434: DDL for GSI Indexes with a Vector field

diagrams.sh

@@ -3,10 +3,10 @@
 [[ -d ./build ]] || mkdir ./build
 
 # install-rr
-[[ -f ./build/rr-2.0-java11.zip ]] || curl -L -o ./build/rr-2.0-java11.zip https://github.com/GuntherRademacher/rr/releases/download/v2.0/rr-2.0-java11.zip
+[[ -f ./build/rr-2.1-java11.zip ]] || curl -L -o ./build/rr-2.1-java11.zip https://github.com/GuntherRademacher/rr/releases/download/v2.1/rr-2.1-java11.zip
 
 # extract-rr
-[[ -f ./build/rr/rr.war ]] || unzip ./build/rr-2.0-java11.zip -d ./build/rr
+[[ -f ./build/rr/rr.war ]] || unzip ./build/rr-2.1-java11.zip -d ./build/rr
 
 for file in n1ql dcl ddl dml dql tcl hints utility
 do

modules/indexes/pages/index-replication.adoc

@@ -40,7 +40,7 @@ This ensures:
 * _Availability_: If one Index-Service node is lost, the other continues to provide access to replicated indexes.
 * _High Performance_: If original and replica copies are available, incoming queries are load-balanced across them.
 
-Index-replicas can be created with the {sqlpp} `CREATE INDEX` statement.
+Index-replicas can be created with the {sqlpp} `CREATE INDEX` or `CREATE VECTOR INDEX` statement.
 Note that whenever a given number of index-replicas is specified for creation, the number must be less than the number of cluster-nodes currently running the {index-service}[Index Service].
 If it is not, the index creation fails.
 Note also that if, following creation of the maximum number of copies, the number of nodes running the Index Service decreases, Couchbase Server progressively assigns replacement index-replicas to any and all Index-Service nodes subsequently be added to the cluster, until the required number of index-replicas again exists for each replicated index.
@@ -69,10 +69,10 @@ include::example$replication-num.n1ql[]
 +
 Note that if `nodes` and `num_replica` are both specified in the `WITH` clause, the specified number of nodes must be _one greater_ than `num_replica`.
 
-* Specifying a number of index-replicas to be created by the Index Service whenever `CREATE INDEX` is invoked.
+* Specifying a number of index-replicas to be created by the Index Service whenever `CREATE INDEX` or `CREATE VECTOR INDEX` is invoked.
 The default is `0`.
 If the default is changed to, say, `2`, creation of a single index is henceforth accompanied by the creation of two replicas, which are automatically distributed across the nodes of the cluster running the Index Service.
-No explicit specification within the `CREATE INDEX` statement is required.
+No explicit specification within the `CREATE INDEX` or `CREATE VECTOR INDEX` statement is required.
 +
 With credentials that provide appropriate authorization, this default can be changed by means of the `curl` command, as follows:
 +
@@ -83,10 +83,10 @@ curl -X POST -u 'Administrator:password' \
 -d 'numReplica=2'
 ----
 +
-Here, `numReplica` is an integer that establishes the default number of replicas that must be created whenever `CREATE INDEX` is invoked.
+Here, `numReplica` is an integer that establishes the default number of replicas that must be created whenever `CREATE INDEX` or `CREATE VECTOR INDEX` is invoked.
 Note that this call only succeeds if the cluster contains enough Index Service nodes to host each new index and its replicas: for example, if `2` is specified as the default number of replicas, the Index Service must have been established on at least 3 nodes.
 +
-Note also that whenever explicit specification of replica-numbers is made within the `CREATE INDEX` statement, this explicit specification takes precedence over any established default.
+Note also that whenever explicit specification of replica-numbers is made within the `CREATE INDEX` or `CREATE VECTOR INDEX` statement, this explicit specification takes precedence over any established default.
 
 You can change index replication settings via the {index-storage-mode}[UI] or the {index-settings-via-rest}[REST API].
 For further information on using {sqlpp}, refer to {query}[Query Fundamentals].

modules/indexes/pages/indexing-and-query-perf.adoc

@@ -12,6 +12,8 @@
 :bucket-analyzer: xref:tools:query-workbench.adoc#bucket-analyzer
 :storage-modes: xref:indexes:storage-modes.adoc
 :additional-storage-use: xref:learn:data/transactions.adoc#additional-storage-use
+:hyperscale-vector-index: xref:vector-index:hyperscale-vector-index.adoc
+:composite-vector-index: xref:vector-index:composite-vector-index.adoc
 
 Creating the right index -- with the right keys, in the right order, and using the right expressions -- is critical to query performance in any database system.
 This is true for Couchbase as well.
@@ -20,7 +22,7 @@ This is true for Couchbase as well.
 .Examples on this Page
 ****
 The examples in this topic use the travel-sample dataset which is shipped with Couchbase Server.
-For instructions on how to install the sample bucket, see {install-sample-buckets}[Sample Buckets].
+For instructions on how to install the sample bucket, see {install-sample-buckets}[].
 
 include::ROOT:partial$query-context.adoc[tag=statement]
 ****
@@ -79,7 +81,7 @@ Use the primary index for full keyspace scans (primary scans) when the query doe
 
 NOTE: A primary index does not index any {additional-storage-use}[transaction records] that may be stored in a keyspace.
 This means that if you are counting the number of documents in a keyspace, you may see slightly different results, depending on whether you are using a primary index or not.
-Refer to {aggregatefun}[Aggregate Functions] and {bucket-analyzer}[Viewing the Data Insights].
+See {aggregatefun}[] and {bucket-analyzer}[Viewing the Data Insights].
 
 .Metadata for Primary Index
 ====
@@ -479,7 +481,7 @@ CREATE INDEX travel_sched ON route
 ----
 ====
 
-For further details and examples, refer to {indexing-arrays}[Array Indexing].
+For further details and examples, see {indexing-arrays}[].
 
 [#partial-index]
 == Partial Index
@@ -581,4 +583,21 @@ After the index selection is made, the query engine analyzes the query to see if
 If it does, the query engine skips retrieving the whole document from the data nodes.
 This is a performance optimization to keep in mind when designing your indexes.
 
-For further details and examples, refer to {covering-indexes}[Covering Indexes].
+For further details and examples, see {covering-indexes}[].
+
+[#hyperscale-vector-index]
+== Hyperscale Vector Index
+
+Hyperscale vector indexes are a type of <<secondary-index>> which contain a single vector field.
+They excel at indexing huge datasets that can scale into the billions of documents.
+They're optimized for pure vector searches, offering the highest performance of any index for your AI applications.
+
+For further details and examples, see {hyperscale-vector-index}[].
+
+[#composite-vector-index]
+== Composite Vector Index
+
+Composite vector indexes are a type of <<composite-secondary-index>> which contain a single vector field and one or more scalar fields.
+Your AI applications can use the index's scalar fields to filter the dataset before performing a vector similarity search.
+
+For further details and examples, see {composite-vector-index}[].

modules/indexes/pages/indexing-overview.adoc

@@ -35,10 +35,13 @@ endif::[]
 * xref:n1ql:n1ql-language-reference/advise.adoc[ADVISE]
 * xref:n1ql:n1ql-language-reference/createindex.adoc[CREATE INDEX]
 * xref:n1ql:n1ql-language-reference/createprimaryindex.adoc[CREATE PRIMARY INDEX]
+* xref:n1ql:n1ql-language-reference/createvectorindex.adoc[CREATE VECTOR INDEX]
 * xref:n1ql:n1ql-language-reference/build-index.adoc[BUILD INDEX]
 * xref:n1ql:n1ql-language-reference/alterindex.adoc[ALTER INDEX]
+* xref:n1ql:n1ql-language-reference/altervectorindex.adoc[ALTER VECTOR INDEX]
 * xref:n1ql:n1ql-language-reference/dropindex.adoc[DROP INDEX]
 * xref:n1ql:n1ql-language-reference/dropprimaryindex.adoc[DROP PRIMARY INDEX]
+* xref:n1ql:n1ql-language-reference/dropvectorindex.adoc[DROP VECTOR INDEX]
 
 == Storage and Availability
 
@@ -52,3 +55,4 @@ endif::[]
 * xref:learn:services-and-indexes/services/index-service.adoc[Index Service Architecture]
 * xref:manage:manage-indexes/manage-indexes.adoc[Manage Indexes]
 * xref:manage:monitor/monitoring-indexes.adoc[Monitor Indexes]
+* xref:vector-index:vectors-and-indexes-overview.adoc[]

modules/n1ql/pages/n1ql-language-reference/alterindex.adoc

@@ -1,6 +1,7 @@
 = ALTER INDEX
 :description: The ALTER INDEX statement moves the placement of an existing index or replica among different GSI nodes.
 :page-edition: Enterprise Edition
+:page-partial:
 :imagesdir: ../../assets/images
 
 :rebalancing-the-index-service: xref:learn:clusters-and-availability/rebalance.adoc#rebalancing-the-index-service
@@ -9,16 +10,21 @@
 :identifiers: xref:n1ql-language-reference/identifiers.adoc
 :logical-hierarchy: xref:n1ql:n1ql-intro/queriesandresults.adoc#logical-hierarchy
 :querying-indexes: xref:n1ql-intro/sysinfo.adoc#querying-indexes
+:alter-vector-index: xref:n1ql-language-reference/altervectorindex.adoc
 
 // TEMP
 include::partial$n1ql-language-reference/horizontal-style.adoc[]
 
 [abstract]
 {description}
 
+The {alter-vector-index}[ALTER VECTOR INDEX] statement is a synonym for the ALTER INDEX statement.
+The two statements have the same functionality.
+
+//tag::purpose[]
 == Purpose
 
-You can use the `ALTER INDEX` statement to change the placement of an existing index or replica among different GSI nodes, to increase or decrease the number of replicas, or to drop a specified index replica temporarily.
+You can use the `{doctitle}` statement to change the placement of an existing index or replica among different GSI nodes, to increase or decrease the number of replicas, or to drop a specified index replica temporarily.
 You can also use it to perform the same alterations to a partitioned index and any replica partitions.
 You may use this statement when you encounter any of the following situations:
 
@@ -28,27 +34,31 @@ You may use this statement when you encounter any of the following situations:
 * The automated process of rebalancing does not give the expected results.
 * Other types of scaling up or scaling down are needed.
 
-For example, if a node fails, you can use the `ALTER INDEX` statement to move an index to another node.
+For example, if a node fails, you can use the `{doctitle}` statement to move an index to another node.
 See <<examples>> below.
 
-NOTE: The ALTER INDEX move operation is asynchronous.
+NOTE: The {doctitle} move operation is asynchronous.
 As soon as the move alter index command is executed, the command returns.
 If there is no error in the input, the move operation can be tracked through the console UI and any error can be found in the Console logs and Indexer logs.
 
-If a node goes down while an ALTER INDEX operation is in progress, then the index would rollback to its original node (not affecting queries) and a notification would appear.
+If a node goes down while an {doctitle} operation is in progress, then the index would rollback to its original node (not affecting queries) and a notification would appear.
 
 IMPORTANT: It is not possible to move an index or index replica and change the number of index replicas at the same time.
 
+//end::purpose[]
+
+//tag::prerequisites[]
 == Prerequisites
 
-Only users with the RBAC role of `Administrator` are allowed to run the `ALTER INDEX` directive.
+Only users with the RBAC role of `Administrator` are allowed to run the `{doctitle}` statement.
 
-This directive is applicable only for Standard GSI (Plasma) and MOI Indexes; and hence supported only for Enterprise Edition nodes.
+This statement is applicable only for Standard GSI (Plasma) and MOI Indexes; and hence supported only for Enterprise Edition nodes.
 (Not applicable for Forest DB.)
 
-This statement is only supported only in Couchbase Server 5.5 and later; if the cluster is a mix of Couchbase Server 5.5 and previous versions, then this statement will not work.
+Altering indexes is only supported in Couchbase Server 5.5 and later; if any nodes in the cluster are running a version of Couchbase Server earlier than 5.5, then this statement will not work.
 
-In addition, altering the number of replicas or deleting an index replica is only supported in Couchbase Server 6.5 and later; if the cluster is a mix of Couchbase Server 6.5 and previous versions, then these operations will not work.
+In addition, altering the number of replicas or deleting an index replica is only supported in Couchbase Server 6.5 and later; if any nodes in the cluster are running a version of Couchbase Server earlier than 6.5, then these operations will not work.
+//end::prerequisites[]
 
 == Syntax
 
@@ -57,27 +67,31 @@ In addition, altering the number of replicas or deleting an index replica is onl
 include::partial$grammar/ddl.ebnf[tag=alter-index]
 ----
 
+//tag::syntax-arguments[]
 image::n1ql-language-reference/alter-index.png["Syntax diagram: refer to source code listing", align=left]
 
-The ALTER INDEX statement provides two possible syntaxes for specifying the index and the keyspace where the index is located.
+The {doctitle} statement provides two possible syntaxes for specifying the index and the keyspace where the index is located.
 
 // TODO: Automatic links in EBNF.
+// FIXME: Maybe try to make consistent with DROP INDEX
 
 [horizontal]
 index-name:: (Required) A unique name that identifies the index.
 
 index-path:: (Optional) One possible syntax for specifying the keyspace.
-Refer to <<index-path>> below.
+See <<index-path>>.
 
 keyspace-ref:: (Optional) The other possible syntax for specifying the keyspace.
-Refer to <<keyspace-ref>> below.
+See <<keyspace-ref>>.
 
 index-using:: (Optional) Specifies the index type.
-Refer to <<index-using>> below.
+See <<index-using>>.
 
 index-with:: (Required) Specifies options for the index.
-Refer to <<index-with>> below.
+See <<index-with>>.
+//end::syntax-arguments[]
 
+//tag::index-path[]
 [[index-path]]
 === Index Path
 
@@ -93,7 +107,7 @@ This syntax provides compatibility with legacy versions of Couchbase Server.
 The index path may be a <<keyspace-full-index>>, a <<keyspace-prefix-index>>, or a <<keyspace-partial-index>>.
 
 NOTE: If there is a hyphen (-) inside the index name or any part of the index path, you must wrap the index name or that part of the index path in backticks ({backtick}{nbsp}{backtick}).
-Refer to the examples below.
+See the examples on this page.
 
 [[keyspace-full-index,full keyspace path]]
 ==== Index Path: Full Keyspace
@@ -172,7 +186,9 @@ collection::
 ====
 For example, `airline.{backtick}idx-name{backtick}` indicates the `idx-name` index on the `airline` collection, assuming that the query context is set.
 ====
+//end::index-path[]
 
+//tag::keyspace-ref[]
 [[keyspace-ref]]
 === Index Name ON Keyspace Reference
 
@@ -187,7 +203,7 @@ You can use the index name with the `ON` keyword and a keyspace reference to spe
 The keyspace reference may be a <<keyspace-path>> or a <<keyspace-partial>>.
 
 NOTE: If there is a hyphen (-) inside the index name or any part of the keyspace reference, you must wrap the index name or that part of the keyspace reference in backticks ({backtick}{nbsp}{backtick}).
-Refer to the examples below.
+See the examples on this page.
 
 [[keyspace-path,keyspace path]]
 ==== Keyspace Reference: Keyspace Path
@@ -245,7 +261,9 @@ collection::
 ====
 For example, `{backtick}idx-name{backtick} ON airline` indicates the `idx-name` index on the `airline` collection, assuming the query context is set.
 ====
+//end::keyspace-ref[]
 
+//tag::index-using[]
 [[index-using]]
 === USING Clause
 
@@ -258,7 +276,9 @@ image::n1ql-language-reference/index-using.png["Syntax diagram: refer to source
 
 The index type for a secondary index must be Global Secondary Index (GSI).
 The `USING GSI` keywords are optional and may be omitted.
+//end::index-using[]
 
+//tag::index-with[]
 [[index-with]]
 === WITH Clause
 
@@ -330,12 +350,14 @@ __optional__
 An integer, specifying a replica ID.
 | Integer
 |===
+//end::index-with[]
 
+//tag::usage[]
 == Usage
 
 If you attempt to alter an index which is still scheduled for background creation, the request fails.
 
-`ALTER INDEX` will not work while the cluster is undergoing a rebalance.
+The statement will not work while the cluster is undergoing a rebalance.
 
 === Moving an Index or Index Replicas
 
@@ -372,16 +394,24 @@ To find the ID of an index replica and see which node it is placed on, you can u
 When dropping a replica, it is possible to leave a server group with no replica.
 For a partitioned index, run a rebalance to move a replica into the vacant server group.
 
+=== Index Redistribution
+
+Using this statement to move one index at a time may be cumbersome if there are a lot of indexes to be moved.
+The _index redistribution_ setting enables you to specify how Couchbase Server redistributes indexes automatically on rebalance.
+Refer to {rebalancing-the-index-service}[Rebalance] for more information.
+//end::usage[]
+
+//tag::return-value[]
 == Return Value
 
-If the `ALTER INDEX` succeeds, then:
+If the statement succeeds, then:
 
 * The query returns an empty array.
 * The index progress is visible in the Query Workbench.
 * After the movement is complete, the new indexes begin to service query scans.
 * The command line displays the new index nodes.
 
-If the `ALTER INDEX` fails, then:
+If the statement fails, then:
 
 * The original indexes continue to service query scans.
 * The UI Log and Query Workbench has the appropriate error message.
@@ -417,6 +447,7 @@ a|
 a|
 * Mistyped the `"action"`
 |===
+//end::return-value[]
 
 [[examples]]
 == Examples
@@ -549,6 +580,10 @@ WITH {"action": "drop_replica", "replicaId": 2};
 
 == Related Links
 
-Using the ALTER INDEX command to move one index at a time may be cumbersome if there are a lot of indexes to be moved.
-The _index redistribution_ setting enables you to specify how Couchbase Server redistributes indexes automatically on rebalance.
-Refer to {rebalancing-the-index-service}[Rebalance] for more information.
+* xref:indexes:indexing-overview.adoc[]
+* xref:vector-index:composite-vector-index.adoc[]
+* xref:n1ql:n1ql-language-reference/createindex.adoc[]
+* xref:n1ql:n1ql-language-reference/createprimaryindex.adoc[]
+* xref:n1ql:n1ql-language-reference/build-index.adoc[]
+* xref:n1ql:n1ql-language-reference/dropindex.adoc[]
+* xref:n1ql:n1ql-language-reference/dropprimaryindex.adoc[]