feat: Setup Kotest runner with output #70

NickHackman · 2025-07-22T04:51:17Z

⚠️ This is a work in progress ⚠️

Relates to #65

Goals

Have an executable / gradle plugin
Execute Kotest tests
Standardize output for neotest-kotlin usage
integrate with neotest-kotlin
remove Lua output parsing

Unsure on the name here, but just using this as of right now and we can change it later

Simple kotest test runner that uses the TestEngineListener to listen to all tests output in order to construct a JSON report. Currently, the specs that it's finding are hardcoded.

This is missing some features and lacks error handling when Kotest fails to run the tests in the first place.

NickHackman · 2025-07-22T04:55:42Z

neotest-kotlin/app/src/main/kotlin/io/github/codymikol/neotestkotlin/App.kt

+        listOf(
+            Class.forName("org.example.KotestFunSpec").kotlin,
+        )


TODO here, need to actually allow for passing in the classes to run

NickHackman · 2025-07-22T04:56:07Z

...otlin/app/src/main/kotlin/io/github/codymikol/neotestkotlin/framework/TestFrameworkRunner.kt

+                val result = runner.run(runnableClasses)
+                when (result) {
+                    is TestRunResult.Success -> result.report
+                    is TestRunResult.Failure -> TODO()


No idea what we should do when this fails, I imagine we want to spit out some JSON that explains what went wrong here?

I think that makes sense, on the lua side we can probably handle that by printing that into the overall output and failing all test cases with a generic error that basically says to look at the output for more information.

It doesn't look like you'd normally get this output and that it would either be a mistake on our end, or an internal kotest error.

NickHackman · 2025-07-22T04:57:41Z

neotest-kotlin/app/src/main/kotlin/io/github/codymikol/neotestkotlin/framework/TestNode.kt

+@JsonClassDiscriminator("type")
+@Serializable


I'm using Kotlinx serialization here, which is nice and all, but have some caveats from good ole Jackson. Namely, the fields in TestNode.Container for status and duration aren't evaluated because they're not backed fields. I'll probably pivot away from this if there's no way to handle this.

docs

NickHackman · 2025-07-22T04:58:45Z

neotest-kotlin/app/src/main/kotlin/io/github/codymikol/neotestkotlin/framework/TestNode.kt

+                                Failure.Error(
+                                    message = it.message,
+                                    lineNumber = traceOrigin?.lineNumber,
+                                    filename = traceOrigin?.fileName,
+                                )


This mostly works, outside of soft assertions which seem to reference a lineNumber that is higher than the file. Maybe just need to specially handle that?

.../app/src/test/kotlin/io/github/codymikol/neotestkotlin/framework/kotest/KotestExampleSpec.kt

NickHackman · 2025-07-22T05:02:27Z

.../kotlin/io/github/codymikol/neotestkotlin/framework/kotest/KotestTestRunnerFunctionalSpec.kt

+                    [
+                      {
+                        "name": "io.github.codymikol.neotestkotlin.framework.kotest.KotestExampleSpec",
+                        "nodes": [


Don't love the "nodes" naming here, but we effectively have a tree. I also don't know a good word for either a Container or a Test

.../kotlin/io/github/codymikol/neotestkotlin/framework/kotest/KotestTestRunnerFunctionalSpec.kt

Better name for the project

This will be the base for the gradle plugin and keep the core logic of running and discovering based on Frameworks opaque and following a contract.

This makes it easier to use in the context of the gradle plugin

this is just a simple maven plugin that can eventually hook into the core lib Fixes N/A

for consistency Fixes N/A

Takes in classes via -Pclasses=org.example.KotestFunSpec and writes to the output file in the build

NickHackman · 2025-08-01T23:05:36Z

...in/src/main/kotlin/io/github/codymikol/kotlintestlauncher/plugin/KotlinTestLauncherPlugin.kt

+
+            classes.set(project.properties["classes"]?.toString())
+
+            outputFile.convention(project.layout.buildDirectory.file("$name/output-${UUID.randomUUID()}.json"))


Currently just dumping to a randomly named output JSON file, but we should be good to provide a custom file via -PoutputFile=/path/to/output/file/here.json

This now leverages the kotlin-test-launcher plugin

Feels more clear than having two nested statuses

Prevent execution of multiple of the same class

NickHackman added 13 commits July 17, 2025 11:50

feat: gradle init neotest-kotlin

ce999dd

Unsure on the name here, but just using this as of right now and we can change it later

build(deps): remove guava

9871597

rebase onto setup

7d9998f

feat: basic Kotest runner

467f8bd

Simple kotest test runner that uses the TestEngineListener to listen to all tests output in order to construct a JSON report. Currently, the specs that it's finding are hardcoded.

build: show events and stdout

e70f086

build: remove JUnit

8f0b6c2

build: add kotlin reflect

d7fad59

build: add all Kotest dependencies

61815c9

refactor: TestNode represents a Container and Test

83fada2

feat: Kotest test running

d4bf763

This is missing some features and lacks error handling when Kotest fails to run the tests in the first place.

test: TestNodeSpec

466db73

test: KotestTestRunner functional test

75e30d8

test: KotestTestRunner unit tests

d3d289b

NickHackman commented Jul 22, 2025

View reviewed changes

.../app/src/test/kotlin/io/github/codymikol/neotestkotlin/framework/kotest/KotestExampleSpec.kt Outdated Show resolved Hide resolved

NickHackman commented Jul 22, 2025

View reviewed changes

.../kotlin/io/github/codymikol/neotestkotlin/framework/kotest/KotestTestRunnerFunctionalSpec.kt Outdated Show resolved Hide resolved

NickHackman and others added 10 commits July 22, 2025 01:07

test: assert duration and stackTrace exist on tests

45988d0

refactor: use AbstractTestEngineListener

790ed60

test: don't run Example specs

424fc5a

refactor: neotest-kotlin -> kotlin-test-launcher

05b298d

Better name for the project

refactor: extract out core subproject

2698c36

This will be the base for the gradle plugin and keep the core logic of running and discovering based on Frameworks opaque and following a contract.

refactor: set up basic gradle plugin

9a211c9

refactor(core): runBlocking rather than suspend

64eecb1

This makes it easier to use in the context of the gradle plugin

feat: add the maven scaffolding

11cbd09

this is just a simple maven plugin that can eventually hook into the core lib Fixes N/A

chore: moved maven -> maven-plugin

8ec07df

for consistency Fixes N/A

build: publish to mavenLocal + jackson

f3e51bc

NickHackman added 2 commits August 1, 2025 19:03

refactor: don't use kotlinx serialization instead use Jackson

b31ef9a

feat: gradle-plugin

1208332

Takes in classes via -Pclasses=org.example.KotestFunSpec and writes to the output file in the build

NickHackman commented Aug 1, 2025

View reviewed changes

NickHackman added 22 commits August 1, 2025 19:11

fix(test-logging.init.gradle.kts): remove test changes for custom plugin

05605b4

feat(gradle-plugin): provide custom outputFile to gradle-plugin

57d48df

refactor: local functions

3ed41ac

chore(gitignore): .idea

390c394

feat: log out outputFile

0317712

feat: use kotlin-test-launcher plugin output instead of gradle

f839015

test: TestNode tests

871270b

test: update

e94e5e1

refactor: remove output_parser entirely

def67c4

This now leverages the kotlin-test-launcher plugin

test: output/init.lua

f55320d

fix: remove output_parser import

5a2ee10

refactor: TestNode.status.status -> TestNode.status.type

d12df8b

Feels more clear than having two nested statuses

fix: use Jackson over kotlinx serialization

9e69cb9

ci: kotlin-test and lua-test workflows

784dd37

fix: exclude abstract Kotest specs

ef4f0dc

fix: never allow plugin to be up to date

f726e5c

build: use same version of kotlin

ac7c854

build(deps): add asm

5dfb409

feat: support executing entire packages

a9c40a8

style: ktlint format

f018821

refactor: use Set rather than List

664180a

Prevent execution of multiple of the same class

feat: kotlin-test-launcher -> kotlin-test and run from root project

098d42d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: Setup Kotest runner with output #70

feat: Setup Kotest runner with output #70

Uh oh!

NickHackman commented Jul 22, 2025

Uh oh!

NickHackman Jul 22, 2025

Uh oh!

NickHackman Jul 22, 2025

Uh oh!

codymikol Jul 22, 2025

Uh oh!

NickHackman Jul 22, 2025

Uh oh!

NickHackman Jul 22, 2025

Uh oh!

Uh oh!

NickHackman Jul 22, 2025

Uh oh!

Uh oh!

NickHackman Aug 1, 2025

Uh oh!

Uh oh!


		classes.set(project.properties["classes"]?.toString())

		outputFile.convention(project.layout.buildDirectory.file("$name/output-${UUID.randomUUID()}.json"))

		@JsonClassDiscriminator("type")
		@Serializable

feat: Setup Kotest runner with output #70

Are you sure you want to change the base?

feat: Setup Kotest runner with output #70

Uh oh!

Conversation

NickHackman commented Jul 22, 2025

Goals

Uh oh!

NickHackman Jul 22, 2025

Choose a reason for hiding this comment

Uh oh!

NickHackman Jul 22, 2025

Choose a reason for hiding this comment

Uh oh!

codymikol Jul 22, 2025

Choose a reason for hiding this comment

Uh oh!

NickHackman Jul 22, 2025

Choose a reason for hiding this comment

Uh oh!

NickHackman Jul 22, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

NickHackman Jul 22, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

NickHackman Aug 1, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!