How can CDK pipeline asset publishing steps be made faster?

[isker]

Hi.

We have a pretty large CDK app deployed using a CDK pipeline. There are many stacks resulting in a few dozen asset "publish" CodeBuild jobs. Most of these are just "file" jobs but a few are "docker" jobs.

We run this pipeline with some frequency, and the latency overhead added by CDK, especially in these publish jobs, is a meaningful amount on top of the latency of CloudFormation actually deploying stacks. It would be ideal to reduce this added latency.

The typical publish job takes 20-40s to start running on CodeBuild (using on-demand EC2 instances, which is the default), then spends 20-30s in the INSTALL phase running npm install cdk-assets@2 (the transitive dependency graph of which expands to almost 300 npm packages!), then spends less than a second in the BUILD phase invoking cdk-assets to upload the assembly to S3. Because we have dozens of such jobs, the overall pipeline ends up getting a (I use this term vaguely) "worst-case" sampling of the variable latencies here.

We could eliminate a lot of the startup latency by running the publish jobs on Lambda instead of EC2, but docker is not supported in Lambda, so it's not actually an option.

We could eliminate the "worst-case" sampling problem by setting publishAssetsInParallel to false, which might be a worthwhile tradeoff, if the serialized invocations of cdk-assets take less time than the difference between median and "worst-case" CodeBuild EC2 startup + npm install latency. But in cases where we are modifying docker images such that docker build needs to be invoked, the cost of serializing such docker builds probably is too high for this to be a good strategy.

Maybe CodeBuild S3 caching (or local caching, with reserved capacity hosts?) could be used to elide the cost of npm install cdk-assets@2? I have not experimented with this.

Maybe the newly added CodeBuild docker server feature could help reduce latency of "docker" jobs? But it requires docker buildx, which cdk-assets does not use. I've heard that you can kind of hackily convince cdk-assets to invoke docker buildx with clever abuse of the CDK_DOCKER environment variable, though I've not actually experimented with this myself.

Some combination of publishAssetsInParallel: false + caching + a single reserved capacity CodeBuild instance to be used throughout the CDK pipeline might be the right approach to meaningfully and cost effectively reduce latency, but I think it'd be ideal if things were lower latency out of the box, so that all users of CDK pipelines could benefit.

A point of confusion for me, which maybe you can help clarify: why are there separate publish jobs in the first place? Why does the single synth job have to upload a single massive zip file as an artifact for all the publish jobs to download and then collectively break apart and push to S3/ECR? Why can't the synth job do all that itself?

Relatedly, why is cdk-assets consumed as an unpinned dependency outside of my project (that is, I have aws-cdk and aws-cdk-lib both pinned in my yarn.lock) that has to be installed from npm at the start of every job? Naively, I'd expect the functionality of cdk-assets to be baked into the CDK CLI, and for publish jobs to use the version of the CLI that I'm using elsewhere in my project.

Sorry for such an expansive post, but this seems to be an expansive problem! I'd love to hear anybody's thoughts on how to tackle this kind of problem, but maybe some "inside baseball" from CDK maintainers would be especially helpful 🌞. Thanks!

[mrgrain]

We are working on bundling all of cdk-assets deps into the package. Beyond that we could probably look into caching the npm package locally with https://docs.aws.amazon.com/codeartifact/latest/ug/dependency-caching.html.

Re buildx please open a feature request. that's probably a useful feature to have anyway.

[isker]

Since filing this I've been playing around with CodeBuild Fleets with specific EC2 instance types. I've got an open CDK PR that adds support for that feature. I've found that, when using a Fleet with t3.medium (the smallest/cheapest x64 option available), the npm install cdk-assets time goes way down, just to a second or two. It's surprising to me that the on-demand instances struggle so much with this operation in comparison, but it really does seem to be true.

So, we're using a Fleet with two such instance types and publishAssetsInParallel: false and local Docker layer caching to get the best perf without spending a lot of money. It works for our situation because everything we do in synth is single threaded, and our synth and docker image builds do not need much memory. (I learned that the S3 cache is basically useless but this does not really matter because the npm install goes so fast on this Fleet.)

We are working on bundling all of cdk-assets deps into the package.

That sounds ideal for the typical user that's not using a reserved instance Fleet. I think this would fix many of the perf problems on its own, but a CodeArtifact cache would probably still be ideal for times when npm goes down.

Re buildx please open a feature request. that's probably a useful feature to have anyway.

There's already aws/aws-cdk#28517, would you like another issue on this repo?

[mrgrain]

Thank you. aws/aws-cdk#28517 will be enough for us to track, thanks!