Feedback on build + build_runner 2.5.0

[davidmorgan]

Place to put general feedback on the release c014314 :)

[davidmorgan]

Test fixes for generators:

Tests that are using build_test will have to update to 3.0.0, there are a pile of breaking changes--but hopefully it's now nicer to use overall, particularly TestReaderWriter.

Here are the fixes that were needed to built_value tests: google/built_value.dart#1367

I noticed that built_value build errors now show stack traces, which is not wanted, this should be fixed in build_runner #4037

[simolus3]

Thank you for all the hard work that went into this release!

I see that thebuild_test changes generally make the API much easier to use. However, I'd like to raise some points that have become much harder (perhaps deliberately) now, that I think are still relevant:

1. I have a use case where I want to test a chain of multiple builders on multiple sources at once (running them like build_runner would run them). I see that build_test now shares much more logic with build_runner_core which is great, but there's also been a loss in flexibility. For instance, I want to capture not just which assets have been read, but also by which builder, and on which primary input id. This is information not exposed with the new testing API. Previously, I'd just write my own asset reader for each phase to record that information, but it has become much harder to do that now (it requires implementation imports because we also need to implement AssetReaderState - a beaking change that aparently isn't mentioned anywhere!).
2. Having to explicitly spell out all sources that are part of a test can be helpful, but I think also a little restrictive. Especially when using the analyzer/resolver in tests, you only really get accurate tests when all transitive input dependencies are readable. Previously, we were able to write a delegating reader with in-memory sources that falls back to the PackageAssetReader. Now, everything needs to be copied into memory up-front. I know that restricting custom readers has been done for efficiency, but I don't understand why everything has to be in-memory. For instance, would it be possible to design something similar to the "overlay" system in the analyzer for tests? That can still give us synchronous reads and a list of assets that have been written / deleted, except that we have an immutable file system layer below all that for external packages that need to be read without really being part of the build.

[davidmorgan]

Thanks Simon--yes, the build_test changes are quite big, I was expecting some feedback here :)

They're more about simplicity than performance: I wanted to get test and non-test setups as much as possible on the same codepath in order to have confidence in the refactor.

But I also want generator authors to have what they need, of course, so let's see what we can do :)

Re: 1, one way to get more information is to run with builder A, then again with builder A+B, then again with A+B+C ... that's what I ended up doing in the build_web_compilers tests, for example this one. But there isn't a way to get to which build step did the read. Possibly we can just record that and have it part of the testing API.

Re: 2, because caching and checking for changes are so central to what build_runner does, I wanted the codepaths around files to be as close as possible. Now they are almost identical, but indeed it does cost some flexibility. What if PackageAssetReader had a way to load all transitive deps, so you can easily add them to a TestReaderWriter?

Happy to dig more into these this week, I've filed #4041. Thanks!

[simolus3]

Possibly we can just record that and have it part of the testing API.

I haven't checked, but I wonder if there's a way to expose the generated asset graph when testing builders? That should have the dependency information at a very low level encoded into it, right? I think that may be good enough for me, this is quite a specialized use case so I'm not sure if it deserves to compliate the ReaderWriterTesting API.

What if PackageAssetReader had a way to load all transitive deps, so you can easily add them to a TestReaderWriter?

I think that would help a lot 👍

[davidmorgan]

Follow-up re: exposing more information to tests on #4041

Filed #4043 for the PackageAssetReader feature.

[Colton127]

I hope this can be of assistance:

2.4.14:

Running `flutter pub run build_runner build --delete-conflicting-outputs`
Current working folder: `/Users/admin/Development/flutter_project`

[INFO] Generating build script...
[INFO] Generating build script completed, took 233ms

[INFO] Precompiling build script......
[INFO] Precompiling build script... completed, took 2.5s

[INFO] Initializing inputs
[INFO] Building new asset graph...
[INFO] Building new asset graph completed, took 1.1s

[INFO] Checking for unexpected pre-existing outputs....
[INFO] Deleting 77 declared outputs which already existed on disk.
[INFO] Checking for unexpected pre-existing outputs. completed, took 1ms

[INFO] Running build...
[INFO] Generating SDK summary...
[INFO] 1.4s elapsed, 0/16 actions completed.
[INFO] 2.5s elapsed, 0/16 actions completed.
[INFO] Generating SDK summary completed, took 3.1s

[INFO] 4.2s elapsed, 65/151 actions completed.
[INFO] 5.3s elapsed, 118/170 actions completed.
[INFO] 6.3s elapsed, 375/415 actions completed.
[INFO] 17.0s elapsed, 531/547 actions completed.
[INFO] 18.6s elapsed, 834/850 actions completed.
[INFO] 19.6s elapsed, 902/917 actions completed.
[INFO] 20.7s elapsed, 998/1014 actions completed.
[INFO] 21.9s elapsed, 1046/1061 actions completed.
[INFO] 22.9s elapsed, 1138/1153 actions completed.
[INFO] 24.0s elapsed, 1221/1235 actions completed.
[INFO] 25.0s elapsed, 1329/1345 actions completed.
[INFO] 26.1s elapsed, 1434/1449 actions completed.
[INFO] 27.1s elapsed, 1626/1633 actions completed.
[INFO] Running build completed, took 28.3s

[INFO] Caching finalized dependency graph...
[INFO] Caching finalized dependency graph completed, took 410ms

[INFO] Succeeded after 28.8s with 276 outputs (2468 actions)

2.5.0 (run 1):

Running `flutter pub run build_runner build --delete-conflicting-outputs`
Current working folder: `/Users/admin/Development/flutter_project`

  Generating the build script.
  Compiling the build script.
  Reading the asset graph.
  Creating the asset graph.
  Doing initial build cleanup.
  Deleting 77 declared outputs which already existed on disk.
  Updating the asset graph.
  Building, full build because builders changed.
  0s freezed on 672 inputs; lib/...dart
  3s freezed on 672 inputs: 1 no-op; spent 3s sdk; lib/...dart
  17s freezed on 672 inputs: 1 same, 14 no-op; spent 11s analyzing, 3s sdk, 2s resolving; lib/...dart
  18s freezed on 672 inputs: 44 same, 628 no-op; spent 11s analyzing, 3s sdk, 2s resolving
  0s json_serializable on 1344 inputs; lib/...dart
  1s json_serializable on 1344 inputs: 1 no-op; lib/...dart
  2s json_serializable on 1344 inputs: 95 skipped, 3 output, 97 no-op; spent 2s analyzing; lib/...dart
  3s json_serializable on 1344 inputs: 183 skipped, 12 output, 198 no-op; spent 2s analyzing; lib/...dart
  4s json_serializable on 1344 inputs: 263 skipped, 26 output, 302 no-op; spent 3s analyzing; lib/...dart
  5s json_serializable on 1344 inputs: 442 skipped, 32 output, 487 no-op; spent 4s analyzing; lib/...dart
  6s json_serializable on 1344 inputs: 619 skipped, 34 output, 674 no-op; spent 4s analyzing; lib/...dart
  6s json_serializable on 1344 inputs: 628 skipped, 34 output, 682 no-op; spent 5s analyzing
  0s source_gen:combining_builder on 1344 inputs; lib/...dart
  1s source_gen:combining_builder on 1344 inputs: 441 skipped, 31 output, 485 no-op; lib/...dart
  1s source_gen:combining_builder on 1344 inputs: 628 skipped, 33 output, 683 no-op; spent 1s building
  Running the post build.
  Writing the asset graph.
  Built with build_runner in 27s; wrote 111 outputs.

Run 2:

  Running `flutter pub run build_runner build --delete-conflicting-outputs`
Current working folder: `/Users/admin/Development/flutter_project`

  Generating the build script.
  Compiling the build script.
  Reading the asset graph.
  Creating the asset graph.
  Doing initial build cleanup.
  Deleting 77 declared outputs which already existed on disk.
  Updating the asset graph.
  Building, full build because builders changed.
  0s freezed on 672 inputs; lib/...dart
  3s freezed on 672 inputs: 1 no-op; spent 3s sdk; lib/...dart
  17s freezed on 672 inputs: 1 same, 14 no-op; spent 11s analyzing, 3s sdk, 2s resolving; lib/...dart
  18s freezed on 672 inputs: 44 same, 628 no-op; spent 12s analyzing, 3s sdk, 2s resolving
  0s json_serializable on 1344 inputs; lib/...dart
  1s json_serializable on 1344 inputs: 1 no-op; lib/...dart
  2s json_serializable on 1344 inputs: 95 skipped, 3 output, 99 no-op; spent 2s analyzing; lib/...dart
  3s json_serializable on 1344 inputs: 186 skipped, 13 output, 200 no-op; spent 2s analyzing; lib/...dart
  4s json_serializable on 1344 inputs: 263 skipped, 26 output, 302 no-op; spent 3s analyzing; lib/...dart
  5s json_serializable on 1344 inputs: 441 skipped, 32 output, 486 no-op; spent 4s analyzing; lib/...dart
  6s json_serializable on 1344 inputs: 612 skipped, 34 output, 667 no-op; spent 4s analyzing; lib/...dart
  6s json_serializable on 1344 inputs: 628 skipped, 34 output, 682 no-op; spent 5s analyzing
  0s source_gen:combining_builder on 1344 inputs; lib/...dart
  1s source_gen:combining_builder on 1344 inputs: 471 skipped, 32 output, 516 no-op; lib/...dart
  1s source_gen:combining_builder on 1344 inputs: 628 skipped, 33 output, 683 no-op; spent 1s building
  Running the post build.
  Writing the asset graph.
  Built with build_runner in 27s; wrote 111 outputs.

PC Specifications:
macOS 15, i9-13900k, 64GB DDR5 6000, nVME SSD

My build time has been ~27 seconds for a while now, so no changes in the new release, at least when building fresh.

Some of the files listed in the output do not exist, even upon searching my file system. In fact, every time I execute build_runner build it outputs new random "freezed.dart" files that do not exist. The parent file does exist, but nowhere is a freezed.dart sub file referenced. Most of the files are not data classes at all.

[davidmorgan]

Thanks @Colton127, good to see it working with flutter pub run :)

The freezed files appear because they might exist: build_runner doesn't know if they will be output until it runs freezed for each input. I do want to check and see if there is any related performance improvement we should be getting that we're not getting today, particularly for source_gen:combining_builder. #3803

[lume-code]

I tried v2.5.1 and the logging is great. Performance is improved too (Around ~%24 cold-build in my case). Here are my results:

v2.4.15:

[INFO] Checking for unexpected pre-existing outputs. completed, took 3.1s
[INFO] Generating SDK summary completed, took 2.4s
[INFO] 2m 34s elapsed, 11821/11821 actions completed.[FlutterGen] Generated: lib/gen/assets.gen.dart
[FlutterGen] Finished generating.
[INFO] Running build completed, took 2m 36s
[INFO] Caching finalized dependency graph completed, took 4.6s
[INFO] Succeeded after 2m 41s with 1249 outputs (25632 actions)

v2.5.1:

7s graphql_codegen on 236 inputs: 236 same; spent 7s building                                                                                                                                             
34s freezed on 2143 inputs: 212 same, 1931 no-op; spent 21s analyzing, 7s resolving, 3s building, 3s sdk                                                                                                  
7s graphql_codegen on 236 inputs: 236 same; spent 7s building                                                                                                                                             
34s freezed on 2143 inputs: 212 same, 1931 no-op; spent 21s analyzing, 7s resolving, 3s building, 3s sdk                                                                                                  
17s json_serializable on 4286 inputs: 1931 skipped, 4 output, 2351 no-op; spent 14s analyzing, 2s building                                                                                                
27s source_gen:combining_builder on 4286 inputs: 1931 skipped, 4 same, 2351 no-op; spent 27s building                                                                                                     
10s injectable_generator:injectable_builder on 8572 inputs: 6213 skipped, 329 output, 2030 no-op; spent 6s analyzing, 4s building                                                                         
4s injectable_generator:injectable_config_builder on 8572 inputs: 6213 skipped, 1 same, 2358 no-op; spent 3s building                                                                                     
0s flutter_gen_runner on 1 input: 1 same                                                                                                                                                                  
2s auto_route_generator on 211 inputs: 166 output, 45 no-op; spent 2s analyzing                                                                                                                           
0s auto_route_generator:auto_router_generator on 1 input: 1 output

The most problematic issue I could notice and that has been there in the previous versions too is the incremental builds long runs. For instance, a change in a file that is not a generated file itself and not that much imported throughout the codebase results in this incremental build time:

0s graphql_codegen on 236 inputs: 236 skipped                                                                                                                                                             
6s freezed on 2143 inputs: 1799 skipped, 187 same, 157 no-op; spent 2s resolving, 2s analyzing, 1s building                                                                                               
2s json_serializable on 4286 inputs: 3508 skipped, 2 same, 776 no-op; spent 1s building                                                                                                                   
26s source_gen:combining_builder on 4286 inputs: 4284 skipped, 2 same; spent 26s tracking                                                                                                                 
1s injectable_generator:injectable_builder on 8572 inputs: 7794 skipped, 10 same, 768 no-op                                                                                                               
3s injectable_generator:injectable_config_builder on 8572 inputs: 8228 skipped, 1 same, 343 no-op; spent 3s building                                                                                      
0s flutter_gen_runner on 1 input: 1 skipped                                                                                                                                                               
0s auto_route_generator on 211 inputs: 95 skipped, 116 same                                                                                                                                               
0s auto_route_generator:auto_router_generator on 1 input: 1 same                                                                                                                                          
                                                                                                                                                                                                          
Built with build_runner in 45s; wrote 319 outputs

The 45-second build time between each minor change and the Analyzer not working during this is a bit suboptimal.

[davidmorgan]

Oh, that's really interesting, thanks

26s source_gen:combining_builder on 4286 inputs: 4284 skipped, 2 same; spent 26s tracking

that looks like a specific case that can be improved #3803

[davidmorgan]

The 45-second build time between each minor change and the Analyzer not working during this is a bit suboptimal.

Hmm, what do you mean about the analyzer not working?

[lume-code]

I see well, thanks. Overall, great. Regarding the analyzer, don't mind it. I am referring to temporary errors during the build times, which should be less of a problem once the performance is further improved.

[davidmorgan]

I see well, thanks. Overall, great. Regarding the analyzer, don't mind it. I am referring to temporary errors during the build times, which should be less of a problem once the performance is further improved.

Ah, I see, yes, planning on working on that as well :) #3909

[ykmnkmi]

Thanks for the work. Each time I run the build, this happens:

Resolving dependencies in `C:\...\workspace_root\package_in_workspace`... 
Downloading packages... 
Got dependencies in `C:\...\workspace_root\package_in_workspace`.
Resolving dependencies in `C:\...\workspace_root\package_in_workspace`... 
Downloading packages...

... build log ...

[ykmnkmi]

build. No path or git deps., but I have deps. within the workspace.

[davidmorgan]

What deps do you have onto build packages? For example in built_value/end_to_end_test/pubspec.yaml I have

dev_dependencies:
  build: '>=1.0.0 <3.0.0'
  build_runner: '>=1.0.0 <3.0.0'

and this works without any extra logging.

[ykmnkmi]

I forgot that one of the packages in workplace is a builder I'm using in the target package.

[davidmorgan]

Ah, that explains it, then :)

[davidmorgan]

Note to self: for fast builds, the overhead of this extra check if you depend on a builder in the same workspace is actually not that small; #4019.