CI/CD Performance Optimization and Modernization #1721
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Updates the checkout and upload-artifact GitHub actions to their latest major versions in all workflow files. This ensures that the workflows benefit from the latest features, improvements, and security patches provided by these actions.
Speeds up build times by: - Implementing ccache for faster rebuilds - Adding date-based keys for cache invalidation - Installing clang from default repos for Ubuntu 24.04 - Parallelizing test execution and preparation - Improving artifact upload efficiency with batching - Refactors release creation to improve reliability - Adds static code analysis workflow
Refactors CI workflows for better reliability and performance. - Adapts Linux builds to automatically use the distribution-provided clang on Ubuntu 24.04. - Enhances macOS builds with explicit tool installation and improved caching strategies using timestamp-based keys and save operations. - Reduces test timeout and enables parallel test execution on macOS. - Fixes an issue preventing creation of necessary directories. - Uses `npm install` for func-js due to potentially missing `package-lock.json`. - Updates MacOS versions, target architectures, and artifacts names to reflect build changes.
Streamlines macOS build workflows by consolidating dependency installations and removing the 3pp cache. This change centralizes the installation of build dependencies within the setup environment step, ensuring consistency across different macOS versions and architectures. It also removes the 3pp cache, simplifying the workflow and potentially reducing build times.
Adds `libssl-dev` to the system libraries installation step in the build workflows to resolve potential build issues related to SSL. Removes caching of 3pp and OpenSSL in the GitHub Actions workflows. These caches were not effectively speeding up the builds and were causing issues. The build scripts now handle OpenSSL dependencies directly. Enables parallel test execution with `ctest` to speed up the test suite execution in the shared build workflow. Adds `-j$(sysctl -n hw.ncpu)` to `ninja` commands in macOS build scripts to improve build times by utilizing all available cores. The build scripts for Ubuntu no longer attempt to download and compile OpenSSL. Instead the system-wide OpenSSL is used. The Windows build script is updated to pass the `/m` flag to `msbuild` to enable parallel builds.
Increases the test timeout to prevent failures due to resource contention or slow environments.
Uses `brew bundle` to install dependencies for macOS builds, ensuring consistent and reproducible environments. Excludes `test-tolk` from the Windows test suite to improve stability.
Addresses several issues to enhance the stability of Windows builds: - Increases the timeout for specific tests to prevent failures on slower machines. - Disables specific compiler warnings to resolve build errors. - Fixes compatibility issues with logging macros on MSVC. - Corrects an issue with spinlock implementation, ensuring proper memory ordering and resolves potential thread-safety analysis warnings.
Removes the custom logging macros. This simplifies the codebase by relying on standard logging practices.
Fixes the SpinLock unlock implementation on MSVC. Ensures correct memory ordering for the flag clear operation. Includes common header to resolve Windows.h dependency.
|
Awesome! Also, might be good:
|
Adds a compiler flag to suppress warnings during the build process across multiple platforms. This improves build log clarity and reduces noise, without altering the compiled output. Also statically links libgcc, libstdc++, and libatomic when building portable executables on Linux to reduce external dependencies.
Adds a compiler flag to suppress warnings when building RocksDB with MSVC. This resolves a build issue caused by excessive warnings being treated as errors.
Hi @tvorogme, Thanks for the great suggestions—I’ve implemented both of them: • The changes are in these commits: 6b4c3b4 and 5d580d9. |
|
@awesome-doge thanks for the contribution! It looks outstanding indeed. It will take time for me to review it carefully, since multiple external products can be impacted. |
|
ឈប់ជាវ |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR significantly enhances CI build workflows for improved stability and performance across all platforms.
Key Improvements
🚀 Performance Optimizations
🔧 Infrastructure Updates
🐛 Bug Fixes & Stability
Performance Comparison
*Static code analysis now runs conditionally and was skipped in this build.
Summary
This optimization reduces CI resource usage while providing faster feedback to developers, significantly improving the development workflow efficiency.