feat: Shell API autocomplete type definitions MONGOSH-2031 MONGOSH-2032 MONGOSH-2173 #2434

lerouxb · 2025-04-17T14:58:03Z

This is a continuation of Anna's work here.

Usage:

npm run api-generate

Implements:

Changes:

merged in main, resolving the conflicts
added generics to the rs, sh and sp objects
added a separate api.ts entrypoint, generating api-processed.d.ts (although this needs work - trying to figure out how to reference it in package.json)
bundled more deps so that we don't import those types in the api-processed.d.ts file. The node.js packages are still being imported, though. Will have to test if that works with the language server
fixed the tests (at least the ones I've noticed were broken so far)
added an api-globals.d.ts which declares the ShellAPI methods and the ShellBson functions as globals. This then gets appended to the end of api-processed.d.ts.
now exporting @mongosh/shell-api/api which is just api-processed exported as one giant string.

Notes:

We'll have to do https://jira.mongodb.org/browse/MONGOSH-2170 as a follow-up once feat(autocomplete): add type generator from the php yaml definitions devtools-shared#521 is in

```sh cd packages/shell-api && \ npm run compile && \ npx api-extractor run ; \ npx ts-node bin/api-postprocess.ts ; \ cat lib/api-processed.d.ts ```

packages/autocomplete/src/index.spec.ts

lerouxb · 2025-04-17T15:08:12Z

packages/shell-api/package.json

+      "types": "./lib/index.d.ts"
+    },
+    "./api": {
+      "types": "./lib/api-processed.d.ts"


I'm not sure what would be the best way to export/publish this and then reference it from the outside.

I think import type ShellApi from '@monogdb-js/shell-api/api' works with this and I tried it manually from a file, but I'll see shortly when I plug this into mongodb-ts-autocomplete

In the end I have to fs.readFile() to get the file as text so we can bundle it as a giant string, so this is kinda moot.

In the end I have to fs.readFile() to get the file as text so we can bundle it as a giant string, so this is kinda moot.

Yeah, exactly. Do we do this as part of this PR? Put it in a .ts file that only exports that giant string?

Right now I generate that giant string in mongodb-ts-autocomplete (see npm run extract-types in mongodb-js/devtools-shared#520), but I suppose we can also export it here. I don't know which way is best. I don't want to accidentally bundle a giant string..

So ... the two options seem to be:

Make shell-api a dev dependency of mongodb-ts-autocomplete, run the extract-types script there as a compile/prepublish step and then include the generated "big string" file in the mongodb-ts-autocomplete exports

Make shell-api a peer dependency of mongodb-ts-autocomplete, move the "big string" generation script here a compile/prepublish and export it from shell-api, then consume it from mongodb-ts-autocomplete

Obviously both of these work, but I don't really see disadvantages to the latter; the former does have at least one notable disadvantage (having to update mongodb-ts-autocomplete to incorporate shell-api changes).

I don't want to accidentally bundle a giant string..

I think we'll end up bundling this string in all our products anyway. But for now, this should actually just work fine, right? webpack wouldn't include it if it isn't being used.

Here's what I do now: 0bd794f

Then in devtools-shared I can:

import ShellApiText from '@mongosh/shell-api/api';

Without having to require.resolve() and read the file and all that. Which is.. marginally better, I guess? I still read mql lib for now (until we do the same thing in the mql types package) and I still do it for reading the bson package.

We can only really get rid of the bson package if we revert 9b3779d and therefore bundle the bson types we use inside the extracted API. But then we'd have to do the same in mql types too.

packages/shell-api/src/api.ts

packages/shell-api/src/collection.spec.ts

packages/shell-api/src/api.ts

…tside

lerouxb · 2025-04-22T10:47:25Z

packages/build/src/packaging/package/helpers.ts

-    'Running "' + joinedCommand + '" in ' + args[2]?.cwd ?? process.cwd()
+    // eslint-disable-next-line @typescript-eslint/ban-ts-comment
+    // @ts-ignore TS2869 Right operand of ?? is unreachable because the left operand is never nullish.
+    `Running "${joinedCommand}" in ${args[2]?.cwd ?? process.cwd()}`


drive-by. For some reason I got a compile error on this line in CI.

lerouxb · 2025-04-23T15:01:08Z

packages/shell-api/api-extractor.json

+    "@mongosh/service-provider-core",
+    "@mongosh/types",
+    "mongodb",
+    "bson"


We have bson set up in mongodb-ts-autocomplete, so I might remove it from here. We'd then just have to find/replace the line where we import from 'bson' to import from '/bson.ts' so that it will be loaded from the typescript service at runtime. (I already got that working for my stubbed mql and shell api implementations, so we can just reuse it.)

I removed it from here and then extract-types.ts in mongodb-ts-autocomplete in devtools-shared replaces the bson import to use its one. That way we can use the same bson shared everywhere (shell-api and mql) and we don't end up extracting all the types again and again. This might premature optimisation, I don't know. But it isn't hard to undo either.

lerouxb · 2025-04-24T17:06:21Z

packages/shell-api/src/api-globals.d.ts

@@ -0,0 +1,44 @@
+declare global {


This file is basically a snippet that gets appended at the end of api-globals.d.ts after we api-extract that. See bin/api-postprocess.ts.

I first tried to write a babel plugin to walk the ShellAPI class's methods and the ShellBson interface's methods, but that turned out to be really complicated.

There is one benefit to having them all mapped here explicitly: We don't end up accidentally leaking everything on ShellAPI into the global scope whether we want to or not. And it is arguably handy seeing them all enumerated here.

The only other variables added as globals would be db, sh, sp and rs which all get specified dynamically by mongodb-ts-autocomplete so they reference the server/database/collection schema generics.

packages/shell-api/bin/api-postprocess.ts

addaleax · 2025-05-08T11:35:38Z

packages/shell-api/package.json

+      "types": "./lib/index.d.ts"
+    },
+    "./api": {
+      "types": "./lib/api-processed.d.ts"


In the end I have to fs.readFile() to get the file as text so we can bundle it as a giant string, so this is kinda moot.

Yeah, exactly. Do we do this as part of this PR? Put it in a .ts file that only exports that giant string?

packages/shell-api/src/api.ts

addaleax

Commented on the giant-string location question but either way, LGTM

addaleax · 2025-05-19T09:52:29Z

packages/shell-api/package.json

+      "types": "./lib/index.d.ts"
+    },
+    "./api": {
+      "types": "./lib/api-processed.d.ts"


So ... the two options seem to be:

Make shell-api a dev dependency of mongodb-ts-autocomplete, run the extract-types script there as a compile/prepublish step and then include the generated "big string" file in the mongodb-ts-autocomplete exports

Make shell-api a peer dependency of mongodb-ts-autocomplete, move the "big string" generation script here a compile/prepublish and export it from shell-api, then consume it from mongodb-ts-autocomplete

Obviously both of these work, but I don't really see disadvantages to the latter; the former does have at least one notable disadvantage (having to update mongodb-ts-autocomplete to incorporate shell-api changes).

I don't want to accidentally bundle a giant string..

I think we'll end up bundling this string in all our products anyway. But for now, this should actually just work fine, right? webpack wouldn't include it if it isn't being used.

nirinchev

Looks good - I mostly have naming/nitty suggestions, but the core logic makes sense to me.

nirinchev · 2025-05-19T15:46:16Z

packages/shell-api/src/database.ts

+  M extends GenericServerSideSchema = GenericServerSideSchema,
+  D extends GenericDatabaseSchema = GenericDatabaseSchema


Maybe I'm too old or perhaps due to not being a ts developer, but these are overly vague for me. Fine leaving them as they are if that's idiomatic for ts, but any reason not to use more descriptive generic names here? Reading those further down in the file, it's very much unclear what M or D is.

I admittedly just left them as I found them and haven't given it much thought. Any suggestions?

I felt like the default values make it clear enough what these are respectively, and it doesn't take a ton of getting used to, but if there are better names for these, then I don't think there's any reason not to adopt them

nirinchev · 2025-05-19T15:52:18Z

packages/shell-api/src/bulk.spec.ts

@@ -57,7 +57,7 @@ describe('Bulk API', function () {
    });
    describe('Metadata', function () {
      describe('toShellResult', function () {
-        const collection = stubInterface<Collection>();
+        const collection = stubInterface<CollectionImpl>();


Any reason to be stubbing CollectionImpl here (and elsewhere instead of Collection)?

Collection (like Database) is now just a type, not a class. This is what I called out in a review comment here. We can keep Collection and Database as classes and then make types named something like CollectionWithSchema and DatabaseWithSchema.

I'm in two minds about which way around is better. I just left it this way because it is what @addaleax did in the PoC.

I suppose there is probably also a third way where we just extend the class and end up with another Class rather than building a type 🤔

CollectionImpl is now gone and unless something is a unit test that just checks Collection, CollectionWithSchema is the thing that's used.

packages/shell-api/src/bulk.spec.ts

packages/shell-api/src/database.ts

packages/shell-api/src/collection.ts

nirinchev · 2025-05-19T16:08:11Z

packages/shell-api/src/helpers.spec.ts

@@ -131,7 +131,7 @@ describe('getPrintableShardStatus', function () {
  const testServer = startSharedTestServer();

  let mongo: Mongo;
-  let configDatabase: Database;
+  let configDatabase: DatabaseImpl;


Why not Database?

These are all fallout from the Database vs DatabaseImpl and Collection vs CollectionImpl thing above. But also as it stands Database & Collection are now generic which means every unit test in the codebase that uses those have to specify the generic types which.. would be a very large PR. And unit tests generally don't care about the schema provided by the generics - they just test the base Collection and Database classes which are just called DatabaseImpl and CollectionImpl now.

I think I can probably "solve" it by renaming CollectionImpl back to Collection and DatabaseImpl back to Database then making either derived classes or types called CollectionWithSchema and DatabaseWithSchema.

DatabaseImpl is gone now. I also removed the default exports from collection.ts and database.ts to force you to pick something explicitly.

DatabaseWithSchema and CollectionWithSchema are the ones you want if you want types, Database and Collection are only things you need when constructing objects, but that's rarely done in the main code and also rare in unit tests where it gets done when unit testing the database or collection and I think that's fair. Can nitpick over whether some existing tests should be making their own database or collection objects in the first place, but I'd prefer leaving them to do what they are already doing in this PR to keep the change size down.

packages/shell-api/src/index.ts

packages/shell-api/src/mongo-errors.spec.ts

packages/shell-api/src/mongo.spec.ts

addaleax

LGTM – only real blocker is the fact that we need to make sure to return DatabaseWithSchema and CollectionWithSchema in public APIs now

addaleax · 2025-05-20T15:13:26Z

packages/cli-repl/src/cli-repl.spec.ts

-        expect(output).not.to.include('Tab completion error');
-        expect(output).not.to.include(
+        expect(output, output).not.to.include('Tab completion error');
+        expect(output, output).not.to.include(


Is this because the output was otherwise too short?

Can't remember the exact way in which it was useless :)

This is kinda a hacky but very easy way to make sure it print the full output if it ever fails which is the very first thing anyone debugging it would do. And it should keep quiet when the expect succeeds so is kinda zero cost.

packages/shell-api/src/mongo.spec.ts

addaleax · 2025-05-20T16:08:17Z

packages/shell-api/src/database.ts

+  M extends GenericServerSideSchema = GenericServerSideSchema,
+  D extends GenericDatabaseSchema = GenericDatabaseSchema


I felt like the default values make it clear enough what these are respectively, and it doesn't take a ton of getting used to, but if there are better names for these, then I don't think there's any reason not to adopt them

packages/shell-api/src/database.ts

packages/shell-api/package.json

packages/shell-api/bin/api-postprocess.ts

Co-authored-by: Anna Henningsen <[email protected]>

…om/mongodb-js/mongosh into shell-api-ac-type-definitions-fork

addaleax and others added 9 commits January 24, 2024 17:59

[Do Not Merge] PoC for shell-api autocomplete type definitions

cfbafad

```sh cd packages/shell-api && \ npm run compile && \ npx api-extractor run ; \ npx ts-node bin/api-postprocess.ts ; \ cat lib/api-processed.d.ts ```

fixup: add index signature support

0e17bc2

fixup

4d72278

more bundled packages

256fc60

Merge branch 'main' into shell-api-ac-type-definitions

d81ff54

add rs, sh, sp generics

198fb06

fix tests

59c04f3

hopefully fix the api export

3545fd8

something went wrong with the package-lock.json file

f34e213

lerouxb commented Apr 17, 2025

View reviewed changes

packages/autocomplete/src/index.spec.ts Outdated Show resolved Hide resolved

lerouxb commented Apr 17, 2025

View reviewed changes

packages/shell-api/src/api.ts Outdated Show resolved Hide resolved

lerouxb commented Apr 17, 2025

View reviewed changes

packages/shell-api/src/collection.spec.ts Outdated Show resolved Hide resolved

lerouxb commented Apr 17, 2025

View reviewed changes

packages/shell-api/src/api.ts Outdated Show resolved Hide resolved

lerouxb added 2 commits April 22, 2025 11:36

don't include the REPLACEME boilerplate which will be added on the ou…

51946ea

…tside

workaround possible false positive error

41a4560

lerouxb commented Apr 22, 2025

View reviewed changes

lerouxb added 2 commits April 22, 2025 11:49

no idea why it added that

33ee0b5

we should probably run api-generate automatically

ae48313

This was referenced Apr 22, 2025

feat: add mongodb-ts-autocomplete package MONGOSH-2034 mongodb-js/devtools-shared#520

Merged

feat: initial feature-flagged, new autocompleter MONGOSH-2036 #2424

Merged

add missing deps

880135f

lerouxb marked this pull request as ready for review April 22, 2025 12:54

lerouxb commented Apr 23, 2025

View reviewed changes

add the ShellApi and ShellBson globals to the shell api

25240df

lerouxb changed the title ~~feat: Shell API autocomplete type definitions MONGOSH-2032~~ feat: Shell API autocomplete type definitions MONGOSH-2032 MONGOSH-2173 Apr 24, 2025

lerouxb changed the title ~~feat: Shell API autocomplete type definitions MONGOSH-2032 MONGOSH-2173~~ feat: Shell API autocomplete type definitions MONGOSH-2031 MONGOSH-2032 MONGOSH-2173 Apr 24, 2025

remove comment

5511582

lerouxb commented Apr 24, 2025

View reviewed changes

linebreak

c6d8219

lerouxb commented Apr 28, 2025

View reviewed changes

packages/shell-api/bin/api-postprocess.ts Show resolved Hide resolved

add help comments to the extracted/generated shell-api

8f11992

lerouxb commented May 1, 2025

View reviewed changes

packages/shell-api/bin/api-postprocess.ts Outdated Show resolved Hide resolved

lerouxb mentioned this pull request May 2, 2025

[WIP DO NOT MERGE] Autocomplete2 demo #2445

Closed

addaleax reviewed May 8, 2025

View reviewed changes

lerouxb added 3 commits May 8, 2025 14:04

strip down api.ts

4ad7978

remove commented code

adaf032

no idea

109ce54

addaleax approved these changes May 19, 2025

View reviewed changes

lerouxb added 4 commits May 19, 2025 16:10

remove TODO

d88317b

keep bson separate so we can ship it once

9b3779d

generate and export a file that just exports the api as a string

0bd794f

Merge branch 'main' into shell-api-ac-type-definitions-fork

a982b05

nirinchev approved these changes May 19, 2025

View reviewed changes

lerouxb added 5 commits May 20, 2025 14:10

CollectionWithSchema, DatabaseWithSchema

26451e2

remove _typeLaunder()

115281b

more generics

3ace495

api extractor warnings cause errors

619c7e6

export the new types

359d155

addaleax approved these changes May 20, 2025

View reviewed changes

lerouxb and others added 6 commits May 20, 2025 19:30

Update packages/shell-api/src/mongo.spec.ts

5443bf6

Co-authored-by: Anna Henningsen <[email protected]>

Fix type assertions to be to collection and database with schema

a80c347

api-generate as part of compile

3b29328

Update packages/shell-api/bin/api-postprocess.ts

637500e

Co-authored-by: Anna Henningsen <[email protected]>

Merge branch 'shell-api-ac-type-definitions-fork' of https://github.c…

37fa446

…om/mongodb-js/mongosh into shell-api-ac-type-definitions-fork

add some logging as documentation

22ac9bd

lerouxb merged commit 86ac326 into main May 21, 2025
118 of 124 checks passed

lerouxb deleted the shell-api-ac-type-definitions-fork branch May 21, 2025 13:23

		M extends GenericServerSideSchema = GenericServerSideSchema,
		D extends GenericDatabaseSchema = GenericDatabaseSchema

feat: Shell API autocomplete type definitions MONGOSH-2031 MONGOSH-2032 MONGOSH-2173 #2434

feat: Shell API autocomplete type definitions MONGOSH-2031 MONGOSH-2032 MONGOSH-2173 #2434

Uh oh!

Conversation

lerouxb commented Apr 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lerouxb May 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lerouxb May 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lerouxb Apr 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lerouxb May 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

addaleax left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nirinchev left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lerouxb May 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

lerouxb commented Apr 17, 2025 •

edited

Loading

lerouxb May 8, 2025 •

edited

Loading

lerouxb May 19, 2025 •

edited

Loading

lerouxb Apr 23, 2025 •

edited

Loading

lerouxb May 19, 2025 •

edited

Loading

lerouxb May 20, 2025 •

edited

Loading