[FEATURE] Replace depstar with tools.build in template

[kwojcik-blockether]

Depstar is deprecated and should be replaced with the tools.build script.

[lowecg]

I was wondering if there has been any progress on this.

The uberjar creation using depstar was fine until I tried to bump tick from 0.5.0-RC5 to 0.5.0, upon which my builds started to fail.

This builds:
tick/tick {:mvn/version "0.5.0-RC5"}

This fails uberjar creation:
tick/tick {:mvn/version "0.5.0"}

{:error "unable to copy file", :name "data_readers.cljc", :exception java.lang.RuntimeException, :message "No dispatch macro for: ?"}
Execution error (ExceptionInfo) at hf.depstar.uberjar/build-jar-as-exec (uberjar.clj:554).
Copying files into the JAR failed

Full report at:
/var/folders/cy/fqw_6j3d4xv22j1nd3nk74z40000gn/T/clojure-9201595171044751625.edn

Content of the report:

 "Execution error (ExceptionInfo) at hf.depstar.uberjar/build-jar-as-exec (uberjar.clj:554).\nCopying files into the JAR failed\n",
 :clojure.main/triage
 {:clojure.error/class clojure.lang.ExceptionInfo,
  :clojure.error/line 554,
  :clojure.error/cause "Copying files into the JAR failed",
  :clojure.error/symbol hf.depstar.uberjar/build-jar-as-exec,
  :clojure.error/source "uberjar.clj",
  :clojure.error/phase :execution},
 :clojure.main/trace
 {:via
  [{:type clojure.lang.ExceptionInfo,
    :message "Copying files into the JAR failed",
    :data {:success false, :reason :copy-failure},
    :at
    [hf.depstar.uberjar$build_jar_as_exec
     invokeStatic
     "uberjar.clj"
     554]}],
  :trace
  [[hf.depstar.uberjar$build_jar_as_exec
    invokeStatic
    "uberjar.clj"
    554]
   [hf.depstar.uberjar$build_jar_as_exec invoke "uberjar.clj" 541]
   [hf.depstar$uberjar invokeStatic "depstar.clj" 173]
   [hf.depstar$uberjar invoke "depstar.clj" 144]
   [clojure.lang.AFn applyToHelper "AFn.java" 154]
   [clojure.lang.AFn applyTo "AFn.java" 144]
   [clojure.lang.Var applyTo "Var.java" 705]
   [clojure.core$apply invokeStatic "core.clj" 667]
   [clojure.core$apply invoke "core.clj" 662]
   [clojure.run.exec$exec invokeStatic "exec.clj" 48]
   [clojure.run.exec$exec doInvoke "exec.clj" 39]
   [clojure.lang.RestFn invoke "RestFn.java" 423]
   [clojure.run.exec$_main$fn__205 invoke "exec.clj" 180]
   [clojure.run.exec$_main invokeStatic "exec.clj" 176]
   [clojure.run.exec$_main doInvoke "exec.clj" 139]
   [clojure.lang.RestFn invoke "RestFn.java" 397]
   [clojure.lang.AFn applyToHelper "AFn.java" 152]
   [clojure.lang.RestFn applyTo "RestFn.java" 132]
   [clojure.lang.Var applyTo "Var.java" 705]
   [clojure.core$apply invokeStatic "core.clj" 667]
   [clojure.main$main_opt invokeStatic "main.clj" 514]
   [clojure.main$main_opt invoke "main.clj" 510]
   [clojure.main$main invokeStatic "main.clj" 664]
   [clojure.main$main doInvoke "main.clj" 616]
   [clojure.lang.RestFn applyTo "RestFn.java" 137]
   [clojure.lang.Var applyTo "Var.java" 705]
   [clojure.main main "main.java" 40]],
  :cause "Copying files into the JAR failed",
  :data {:success false, :reason :copy-failure}}}

[kwojcik-blockether]

No progress so far. However, you are lucky since I finally have some energy to donate my time to OSS. Expect a fix today/tomorrow.

All the best to you Chris!

[lowecg]

Thank you, Karol - glad to hear the energy levels have improved!

Let me know if I can help with any testing.

[kwojcik-blockether]

@lowecg Here is the potential support for tools.build: origin/master 1275359

How to test:

1. Clone the repository.
2. Go to modules/holy-lambda-template
3. lein install
4. Generate a project with lein new holy-lambda <a-project>.

Happy testing.

[lowecg]

@FieryCod I've tested this locally.

The generated project using the template, compiles fine. And when upgrading the tick dependency in my project from 0.5.0-RC5 to 0.5.0, the build succeeds.

It wasn't a straight upgrade, however.

When transferring to my project, the basic process was simple enough but I had to make changes to the GraalVM native build config to ultimately get this to work.

Before changing any of the deps, I tried the new build with my project, but it failed with the following:

Fatal error: org.graalvm.compiler.debug.GraalError: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: No instances of clj_tuple$hash_map are allowed in the image heap as this class should be initialized at image runtime. To see how this object got instantiated use --trace-object-instantiation=clj_tuple$hash_map.

Adding "--initialize-at-build-time='clj_tuple$hash_map,clj_tuple$vector'" (note single quotes) to the native config addressed this.

The Lambda executes successfully.

However, the Lamdba package size has increased by 0.5 MiB: Was 24.5 MiB, now 25 MiB.

The GraalVM compiled output has increased by almost 3MiB, and the GraalVM builds have slowed by 10s.

Minor observation - I updated the location of the generated target directory to put it under .holy-lambda.

[lowecg]

Comparing GraalVM output between depstar and tools.build, the number of reachable fields has increased significantly between the two compile runs, and clojure.core has grown with the latter.

GraalVM depstar:

GraalVM Native Image: Generating 'output' (executable)...
========================================================================================================================
[clj-easy/graal-build-time] WARN: Single segment package found for class: clj_tuple__init.class. This class has no package and it will not be added to the result packages.
[clj-easy/graal-build-time] WARN: Single segment package found for class: potemkin__init.class. This class has no package and it will not be added to the result packages.
[clj-easy/graal-build-time] Registering packages for build time initialization: clojure, clj_easy, clj_http, cljc.java_time, cognitect, com.rpl, connect.aws_dynamodb_utils, connect.aws_ssm_utils, connect.aws_utils, connect.coll_utils, connect.date_utils, connect.key_utils, connect.lambda.lambda_context, fierycod.holy_lambda, io.pedestal, jsonista, potemkin, riddley, scalably.lambda.export_order_status_event_to_byl, slingshot, tick, time_literals
[1/7] Initializing...                                                                                    (3.8s @ 0.17GB)
 Version info: 'GraalVM 22.2.0 Java 17 CE'
 Java version info: '17.0.4+8-jvmci-22.2-b06'
 C compiler: gcc (redhat, aarch64, 7.3.1)
 Garbage collector: Epsilon GC
 1 user-specific feature(s)
 - InitAtBuildTimeFeature
WARNING: abs already refers to: #'clojure.core/abs in namespace: cljc.java-time.duration, being replaced by: #'cljc.java-time.duration/abs
[2/7] Performing analysis...  [*********]                                                               (28.8s @ 3.39GB)
  18,964 (93.76%) of 20,227 classes reachable
  24,174 (47.57%) of 50,823 fields reachable
  75,617 (61.92%) of 122,124 methods reachable
     485 classes,   109 fields, and 1,732 methods registered for reflection
      71 classes,    76 fields, and    62 methods registered for JNI access
       6 native libraries: dl, m, pthread, rt, stdc++, z
[3/7] Building universe...                                                                               (3.2s @ 2.64GB)
[4/7] Parsing methods...      [**]                                                                       (4.4s @ 4.19GB)
[5/7] Inlining methods...     [***]                                                                      (1.0s @ 3.24GB)
[6/7] Compiling methods...    [****]                                                                    (14.9s @ 3.47GB)
[7/7] Creating image...                                                                                  (5.3s @ 3.51GB)
  30.22MB (36.96%) for code area:    52,531 compilation units
  51.27MB (62.71%) for image heap:  659,859 objects and 79 resources
 269.20KB ( 0.32%) for other data
  81.76MB in total
------------------------------------------------------------------------------------------------------------------------
Top 10 packages in code area:                               Top 10 object types in image heap:
   2.57MB clojure                                              9.80MB java.lang.Class
   1.44MB sun.security.ssl                                     7.23MB byte[] for code metadata
   1.30MB clojure.lang                                         6.21MB java.lang.Object[]
 824.43KB java.util                                            3.63MB clojure.lang.PersistentHashMap$BitmapIndexedNode
 725.57KB clojure.core                                         3.17MB java.lang.String
 576.77KB com.sun.crypto.provider                              2.71MB byte[] for general heap data
 503.46KB clojure.spec                                         2.70MB byte[] for java.lang.String
 472.78KB java.lang.invoke                                     1.59MB com.oracle.svm.core.hub.DynamicHubCompanion
 419.80KB java.lang                                            1.50MB byte[] for embedded resources
 414.03KB com.sun.org.apache.xerces.internal.impl              1.32MB clojure.lang.Symbol
  20.77MB for 463 more packages                               11.41MB for 8726 more object types
------------------------------------------------------------------------------------------------------------------------
                        4.3s (6.5% of total time) in 42 GCs | Peak RSS: 6.34GB | CPU load: 5.44
------------------------------------------------------------------------------------------------------------------------
Produced artifacts:
 /project/.holy-lambda/build/output (executable)
 /project/.holy-lambda/build/output.build_artifacts.txt (txt)
========================================================================================================================
Finished generating 'output' in 1m 5s.

GraalVM tools.build:

GraalVM Native Image: Generating 'output' (executable)...
========================================================================================================================
[clj-easy/graal-build-time] WARN: Single segment package found for class: clj_tuple__init.class. This class has no package and it will not be added to the result packages.
[clj-easy/graal-build-time] WARN: Single segment package found for class: potemkin__init.class. This class has no package and it will not be added to the result packages.
[clj-easy/graal-build-time] Registering packages for build time initialization: clojure, clj_easy, clj_http, cljc.java_time, cognitect, com.rpl, connect.aws_dynamodb_utils, connect.aws_ssm_utils, connect.aws_utils, connect.coll_utils, connect.date_utils, connect.key_utils, connect.lambda.lambda_context, fierycod.holy_lambda, io.pedestal, jsonista, potemkin, riddley, scalably.lambda.export_order_status_event_to_byl, slingshot, tick, time_literals
[1/7] Initializing...                                                                                    (3.7s @ 0.18GB)
 Version info: 'GraalVM 22.2.0 Java 17 CE'
 Java version info: '17.0.4+8-jvmci-22.2-b06'
 C compiler: gcc (redhat, aarch64, 7.3.1)
 Garbage collector: Epsilon GC
 1 user-specific feature(s)
 - InitAtBuildTimeFeature
WARNING: abs already refers to: #'clojure.core/abs in namespace: cljc.java-time.duration, being replaced by: #'cljc.java-time.duration/abs
[2/7] Performing analysis...  [*********]                                                               (29.0s @ 1.28GB)
  18,963 (93.76%) of 20,226 classes reachable
  24,172 (38.48%) of 62,816 fields reachable
  75,815 (62.08%) of 122,121 methods reachable
     485 classes,   109 fields, and 1,732 methods registered for reflection
      71 classes,    76 fields, and    62 methods registered for JNI access
       6 native libraries: dl, m, pthread, rt, stdc++, z
[3/7] Building universe...                                                                               (3.7s @ 4.08GB)
[4/7] Parsing methods...      [****]                                                                    (13.6s @ 4.55GB)
[5/7] Inlining methods...     [***]                                                                      (1.0s @ 3.52GB)
[6/7] Compiling methods...    [****]                                                                    (14.8s @ 4.05GB)
[7/7] Creating image...                                                                                  (5.4s @ 4.14GB)
  32.31MB (38.23%) for code area:    53,170 compilation units
  51.93MB (61.45%) for image heap:  659,805 objects and 79 resources
 278.78KB ( 0.32%) for other data
  84.51MB in total
------------------------------------------------------------------------------------------------------------------------
Top 10 packages in code area:                               Top 10 object types in image heap:
   2.57MB clojure                                              9.95MB java.lang.Class
   1.44MB sun.security.ssl                                     7.72MB byte[] for code metadata
   1.39MB clojure.core                                         6.21MB java.lang.Object[]
   1.30MB clojure.lang                                         3.62MB clojure.lang.PersistentHashMap$BitmapIndexedNode
 827.04KB java.util                                            3.17MB java.lang.String
 576.74KB com.sun.crypto.provider                              2.71MB byte[] for general heap data
 503.46KB clojure.spec                                         2.70MB byte[] for java.lang.String
 472.78KB java.lang.invoke                                     1.59MB com.oracle.svm.core.hub.DynamicHubCompanion
 439.00KB clj_http                                             1.52MB byte[] for embedded resources
 420.26KB java.lang                                            1.32MB clojure.lang.Symbol
  22.15MB for 463 more packages                               11.41MB for 8725 more object types
------------------------------------------------------------------------------------------------------------------------
                        4.4s (5.9% of total time) in 50 GCs | Peak RSS: 6.31GB | CPU load: 5.67
------------------------------------------------------------------------------------------------------------------------
Produced artifacts:
 /project/.holy-lambda/build/output (executable)
 /project/.holy-lambda/build/output.build_artifacts.txt (txt)
========================================================================================================================
Finished generating 'output' in 1m 15s.

[lowecg]

As a very coarse comparison between the content of the respective uber jars reveals some differences:

jar -ft .holy-lambda.depstar/build/output.jar | sed -E 's/([0-9]+)//g' | sort > x
jar -ft .holy-lambda/build/output.jar | sed -E 's/([0-9]+)//g' | sort > y

diff x y
8762d8761
< cognitect/aws/protocols$json_parse_error.class
8764c8763,8764
< cognitect/aws/protocols$parse_error_STAR_.class
---
> cognitect/aws/protocols$parse_encoded_string_STAR_.class
> cognitect/aws/protocols$parse_http_error_response.class
8766d8765
< cognitect/aws/protocols$xml_parse_error.class
11623d11621
< native-configuration/agent-extracted-predefined-classes/

[lowecg]

I'm going to roll this out to a few other lambda projects.

I suspect the increase in code size is from tools.build being more thorough.

[lowecg]

I've applied the new build to another 6 Lambda functions and the package size inflation is negligible: 0 - 0.1MiB

[kwojcik-blockether]

Could you share the repro? I would love to take the look into the generated bytecode and compare it side by side (tools.build vs depstar). Maybe it would be something worth reporting upstream to tools.build maintainers.

The single-segment warnings are something you should address rather than add the init options as you did.

[lowecg]

The single-segment warnings are something you should address rather than add the init options as you did.

Not sure how I'd do that. clj-tuple and potemkin are transitive dependencies for clj-http.

[clj-easy/graal-build-time] WARN: Single segment package found for class: clj_tuple__init.class. This class has no package and it will not be added to the result packages.
[clj-easy/graal-build-time] WARN: Single segment package found for class: potemkin__init.class. This class has no package and it will not be added to the result packages.

[lowecg]

The depstar version had the following, which I can't see an alternative for in the new version. Could there be anything in that?

 :jvm-opts   ["-Dclojure.compiler.direct-linking=true"
                      "-Dclojure.spec.skip-macros=true"]}

[lowecg]

tools.build is more aggressive at pulling in Clojure code.

In another Lambda, I had a namespace amongst my lambda code called repl-helper.clj. It just had some utils that I could interact with the Lamdba code using local resources.

The code is not referenced from the Lambda code.

With depstar, it was always ignored. With tools.build, it was causing the build to fail.