1.2.8

1.2.8 (2025-10-10)

Release 1.2.8 is a maintenance release focused on quality assurance improvements, with enhanced testing infrastructure, better configuration validation, and useful configuration enhancements.

Key Features

• Configurable Labels and Annotations: All Kubernetes resources now support customizable labels and annotations through Helm values, providing better integration with organizational policies and tooling.
• Enhanced Build System: Specialized build configurations now support environment-specific customization (e.g., Replicated builds), simplifying multi-environment deployments.

Configuration Improvements

• Centralized Validation: Moved apiKey/existingSecretName validation from Helm templates to JSON Schema, centralizing all configuration validation in a single location for improved maintainability.
• Service Port Protocol: Added configurable protocol field for webhook server service ports, improving compatibility with service mesh configurations.

Quality Assurance

• CI/CD Infrastructure Overhaul: Restructured the entire CI testing infrastructure to support more comprehensive testing during development, including expanded Kubernetes version coverage (now testing against 1.33 and 1.34) and improved test isolation.
• Unified Testing Framework: Introduced consolidated testing infrastructure with new test-all target covering unit tests, integration tests, Helm tests, and KUTTL end-to-end tests.
• Workflow Validation: Added actionlint for GitHub Actions workflow validation and markdownlint-cli2 for documentation quality checks.
• Documentation Expansion: Significantly expanded project documentation with comprehensive guides for development, testing, architecture, and troubleshooting.

Upgrade Steps

To upgrade to version 1.2.8, run the following command:

helm upgrade --install <RELEASE_NAME> cloudzero/cloudzero-agent -n <NAMESPACE> --create-namespace -f configuration.example.yaml --version 1.2.8

1.2.7

1.2.7 (2025-08-22)

Release 1.2.7 removes a dependency on the Bitnami kubectl container image, replacing it instead with a new cloudzero-certifik8s executable. This is critical as Broadcom (who controls Bitnami through VMware) is doing away with the old bitnami images.

Key Features

• Go-Based Certificate Management: Complete transformation from bash scripts to modern Go-based certificate management with the new cloudzero-certifik8s tool, providing enhanced security, testability, and maintainability.
• Comprehensive Security Context: Added security context to all Kubernetes resources (pods, containers, jobs, deployments, daemonsets) with secure defaults and component-specific overrides.
• Enhanced Shipper Reliability: Improved shipper logging and fixed a replay file processing bug that could cause successful uploads to be incorrectly abandoned.

Security Enhancements

• Certificate Management Security: Replaced bash scripts with secure Go-based certificate generation, eliminating dependency on deprecated bitnami/kubectl Docker image and implementing proper RBAC with reduced permissions.
• Security Context Implementation: Added comprehensive security context to all Helm templates with secure defaults (runAsUser: 65534, runAsNonRoot: true) and proper property filtering for pod vs container contexts.
• Checkov Security Compliance: Enabled security context rules (CKV_K8S_29, CKV_K8S_30, CKV_K8S_23) after implementing proper security contexts across all resources.
• RBAC Improvements: Enhanced cluster-scoped permissions for certificate management with resource-specific restrictions and proper Kubernetes client integration.

Shipper Reliability Improvements

Replay File Processing Fix:

• Fixed critical bug where successfully uploaded files were incorrectly abandoned
• Corrected replay request loop to iterate over reference IDs instead of URLs
• Enhanced abandon operation logging with file-specific details (reference_id and reason)
• Added comprehensive debug logging for replay request processing

Enhanced Logging:

• Improved abandon operation logging to include file-specific details
• Added debug logging for replay request processing
• Fixed smoke test failures related to replay request processing

Configuration Enhancements

CloudAccountId Validation:

• Enhanced JSON schema to allow quoted values for better user experience
• Added support for quoted numeric and UUID values (e.g., '1234567890', '123e4567-e89b-12d3-a456-426614174000')
• Implemented comprehensive test coverage for all quote scenarios
• Added warning notes discouraging manual configuration of auto-detectable properties

Upgrade Steps

To upgrade to version 1.2.7, run the following command:

helm upgrade --install <RELEASE_NAME> cloudzero/cloudzero-agent -n <NAMESPACE> --create-namespace -f configuration.example.yaml --version 1.2.7

1.2.6

1.2.6 (2025-08-05)

Release 1.2.6 introduces a new CronJob-based backfill system, comprehensive resource management improvements, and enhanced security and reliability features.

Key Features

• Enhanced Shutdown Coordination: Implemented robust file-based shutdown coordination between collector and shipper containers with intelligent waiting mechanisms and timeout protection, ensuring graceful shutdown sequences.
• Dual Backfill System: Implemented both a CronJob for scheduled runs (default: every 12 hours) and an immediate Job for instant execution on install, providing both immediate execution and configurable recurring runs for ongoing data collection.
• Comprehensive Resource Management: Systematic refactoring of all components with centralized resource generation, providing consistent resource request/limit configurations across all containers.

Security Enhancements

• Checkov Security Integration: Added comprehensive security analysis with Checkov to build system and CI, fixing multiple Kubernetes security violations including missing liveness and readiness probes.
• Fail-Open Webhook Validation: Implemented true fail-open behavior for webhook validation with always-allow behavior, ensuring webhook validation never blocks Kubernetes resource operations.
• Enhanced Health Monitoring: Added proper liveness and readiness probes to prometheus-config-reloader container, improving container health monitoring and automatic restart capabilities.
• Comprehensive Security Context Implementation: Added configurable security context support to all Kubernetes pods and containers.

Additional Enhancements

• Observability Improvements: Removed observability files on upload to prevent storage bloat and improve performance.
• Scout Configuration Enhancement: Updated scout to return Google project number instead of project ID for improved metadata accuracy.
• Cloud Account Validation: Added JSON Schema validation for cloudAccountId contents to ensure proper configuration.
• Image Pull Secrets: Added image pull secrets support for config loader and helmless jobs for enhanced security.

Technical Improvements

• Centralized Resource Generation: Created reusable helper functions for consistent resource configuration patterns across all templates.
• Backward Compatibility: Maintained full backward compatibility through legacy precedence logic, ensuring existing deployments continue to work without changes.
• Comprehensive Testing: Added 20 test suites with 87 total tests covering all fallback scenarios, security context functionality, and edge cases.

Resource Configuration Details

New Component Structure:

• Core Components: components.agent.resources, components.aggregator.collector.resources, components.aggregator.shipper.resources, components.webhookServer.resources
• Job Components: components.miscellaneous.configLoader.resources, components.webhookServer.backfill.resources, components.agent.federatedNode.resources
• New Components: components.helmless.resources, components.initCertJob.resources
• Specialized Components: components.agent.configmapReloader.resources, components.validator.resources, components.kubeStateMetrics.resources

Upgrade Steps

To upgrade to version 1.2.6, run the following command:

helm upgrade --install <RELEASE_NAME> cloudzero/cloudzero-agent -n <NAMESPACE> --create-namespace -f configuration.example.yaml --version 1.2.6

1.2.5

1.2.5 (2025-07-25)

Release 1.2.5 is a critical maintenance release that fixes a webhook configuration issue affecting resource metadata collection. Due to a single-character difference in resource names (using singular instead of plural), the webhook server was not collecting the necessary information for labels and annotations. Customers on versions 1.2.3 and 1.2.4 should upgrade immediately.

Critical Fix

• Webhook Configuration Fix: Fixed a critical bug where the webhook server was not collecting resource metadata due to incorrect resource name configuration. This affected label and annotation collection for all resources processed by the webhook.

Key Features

• Enhanced Webhook Configuration: Fixed webhook misconfiguration issues and improved integration testing infrastructure with comprehensive validation and debugging capabilities.
• AWS IMDSv1 Fallback Support: The CloudZero Agent's AWS scout implementation now gracefully falls back from IMDSv2 to IMDSv1 when the token endpoint is unavailable, ensuring compatibility with clusters that don't have IMDSv2 enabled. This maintains security preference for IMDSv2 while providing compatibility with IMDSv1-only environments.
• Comprehensive Troubleshooting Guide: Added a troubleshooting guide covering quick diagnosis, component-specific troubleshooting, network policies, certificate issues, and scaling problems with clear escalation paths.

Additional Enhancements

• Security Documentation: Significantly expanded SECURITY.md with detailed security considerations, vulnerability reporting procedures, and best practices for secure deployment.
• Scout Error Messages: Enhanced scout configuration error messages with specific Helm chart parameter guidance, making troubleshooting more actionable.
• Cloud Provider Detection: Added cloud provider information to cluster configuration for improved metadata collection and environment awareness.
• Test Infrastructure: Improved webhook integration testing with centralized Kind cluster configuration, enhanced test maintainability, and comprehensive validation.
• Dependency Updates: All third-party dependencies have been update to the latest versions.

Technical Improvements

• Webhook Reliability: Fixed service name resolution and improved webhook test validation with comprehensive debugging capabilities
• Documentation Quality: Added systematic troubleshooting approach with label selector commands and component-specific diagnostic procedures
• Build System: Enhanced test infrastructure with better organization and maintainability
• AWS Metadata Service Compatibility: Implemented robust fallback mechanism for AWS metadata retrieval with clear error distinction between IMDSv2 and IMDSv1 failures

Upgrade Steps

⚠️ CRITICAL: Customers on versions 1.2.3 and 1.2.4 should upgrade immediately due to the webhook configuration fix.

To upgrade to version 1.2.5, run the following command:

helm upgrade --install <RELEASE_NAME> cloudzero/cloudzero-agent -n <NAMESPACE> --create-namespace -f configuration.example.yaml --version 1.2.5

1.2.4

1.2.4 (2025-07-17)

⚠️ CRITICAL: This version contains a webhook configuration bug that prevents proper resource metadata collection. Please upgrade to version 1.2.5 immediately.

Release 1.2.4 is a maintenance release including Improved Metrics Filtering, and Collector Interval Adjustments for better performance. This release focuses on operational improvements, build efficiency, and enhanced visibility into metric processing.

Key Features

• Optimized Collection Intervals: Increased cost metrics collection interval from 10 minutes to 30 minutes for better performance in smaller clusters, while reducing observability metrics timeout to 10 minutes to maintain cluster connectivity visibility.
• Enhanced Scout Auto-Detection: The confload job now leverages the Scout system to automatically detect cloud environment metadata (region, account ID, cluster name) when these values are not explicitly provided, significantly simplifying deployment configuration.
• Dramatic Docker Build Performance: Build times reduced from 2:30-3:00 minutes to ~12 seconds through multi-stage builds with platform-specific caching, selective file copying, and conditional dependency generation.
• Dropped Metrics Tracking: The metric filter now provides visibility into filtered-out metrics through debug logging, making it easier to debug filter configurations and understand metric processing behavior.

Additional Enhancements

• Backfiller Reliability: Fixed GroupVersionKind issues and race conditions in namespace and node processing, with comprehensive integration testing.
• Test Infrastructure: Improved test reliability by fixing flaky tests related to file monitoring, file locking, and SQL timestamp formatting.
• Development Tooling: Added semantic diff targets (*.{yaml,json}-semdiff) for better visibility into Helm template changes during development.
• Dependency Management: Updated Dependabot to run on Wednesdays instead of Fridays for better alignment with patch release cycles.

Upgrade Steps

To upgrade to version 1.2.4, run the following command:

helm upgrade --install <RELEASE_NAME> cloudzero/cloudzero-agent -n <NAMESPACE> --create-namespace -f configuration.example.yaml --version 1.2.4

1.2.3

1.2.3 (2025-07-02)

⚠️ CRITICAL: This version contains a webhook configuration bug that prevents proper resource metadata collection. Please upgrade to version 1.2.5 immediately.

Release 1.2.3 introduces Cloud Service Provider Auto-Detection, significant Performance Optimizations for the admission controller, enhanced Istio Integration, and numerous reliability improvements. This release dramatically simplifies deployment configuration while improving performance and compatibility with service mesh environments.

Key Features

• Cloud Service Provider Auto-Detection: The CloudZero agent now includes a comprehensive "scout" system that automatically detects cloud environment metadata including provider, region, account ID, and cluster name. This eliminates the need to manually configure these values in many deployments.
  • AWS Support: Automatically detects region, account ID from EC2 instance metadata
  • Google Cloud Support: Automatically detects region, project ID, and cluster name from GCE metadata
  • Azure Support: Automatically detects region and subscription ID from Azure IMDS
• Webhook Server Optimization: The webhook server now explicitly requests only the Kubernetes resource types it needs instead of receiving all resources, significantly reducing network traffic and improving performance.
• Enhanced Istio Integration: The webhook server now automatically includes sidecar.istio.io/inject: "false" annotation by default, providing seamless out-of-the-box compatibility with Istio service mesh environments without requiring manual configuration.

Additional Enhancements

• Improved Load Balancing: Enhanced webhook server connection handling with periodic connection rotation to ensure proper load distribution across service replicas in multi-replica deployments.
• Configurable Webhook Timeout: Added ability to configure webhook admission controller timeout values, and changed the default from 15 seconds to 1 second.
• Enhanced Pod Disruption Budget (PDB) Configuration: Completely reworked PDB validation and override behavior to prevent common configuration errors and provide more intuitive component-level overrides.

Upgrade Steps

To upgrade to version 1.2.3, run the following command:

helm upgrade --install <RELEASE_NAME> cloudzero/cloudzero-agent -n <NAMESPACE> --create-namespace -f configuration.example.yaml --version 1.2.3

1.2.2

1.2.2 (2025-06-24)

This is a maintenance release that includes important bug fixes and dependency updates to improve reliability and stability.

Bug Fixes

• Configuration Management: Fixed an issue where component-specific configuration merging was incorrectly modifying default values, potentially causing unexpected behavior.
• ConfigMap References: Updated ConfigMap name references in the loader job to use the correct naming convention, preventing resource lookup failures.
• JSON Schema Validation: Added support for properties which were previously not present in values.yaml, but were used in the template.
• Invalid Template Fixes: Fixed template generation for options were causing invalid Kubernetes resources to be generated.
• Allow resource_type Labels: The Aggregator no longer filters out "resource_type" and "workload" labels.

Enhancements

• Helmless Tool: Improved the helmless implementation by splitting it out from the CLI with enhanced testing coverage and removal of unnecessary functionality.
• Testing Infrastructure: Added checks to verify that all Kubernetes resources are created successfully during deployment validation.
• Testing Template Generation: Added kubeconform tests to validate generated templates.

Upgrade Steps

To upgrade to version 1.2.2, run the following command:

helm upgrade --install <RELEASE_NAME> cloudzero/cloudzero-agent -n <NAMESPACE> --create-namespace -f configuration.example.yaml --version 1.2.2

1.2.1

1.2.1 (2025-06-17)

This is primarily a bugfix release that resolves JSON Schema validation issues when the cloudzero-agent Helm chart is used as a subchart.

Bug Fixes

• Subchart Schema Validation: Fixed JSON Schema validation error that occurred when the cloudzero-agent chart was used as a subchart. Helm automatically adds a top-level 'global' property for subcharts, which was not previously allowed by the schema, causing validation failures.

Additional Enhancements

• Helmless Job: Added a Helm job that runs the helmless tool, providing an easy way to determine minimal configuration overrides by checking the job logs.
• Improved Logging: Both the collector and shipper now emit regular info-level log messages, providing positive confirmation that the agent is working correctly.

Testing Improvements

• Subchart Testing: Added comprehensive test coverage for subchart scenarios to prevent regression of schema validation issues.

Upgrade Steps

To upgrade to version 1.2.1, run the following command:

helm upgrade --install <RELEASE_NAME> cloudzero/cloudzero-agent -n <NAMESPACE> --create-namespace -f configuration.example.yaml --version 1.2.1

1.2.0

1.2.0 (2025-06-05)

Release 1.2.0 introduces Federated Mode support, comprehensive Helm Schema Validation, enhanced Configuration Management, and numerous improvements to observability, reliability, and maintainability. This release significantly expands deployment flexibility while strengthening data quality and operational efficiency.

Key Features

• Federated Mode Support: The CloudZero agent now supports federated mode deployment to help support large clusters. In this mode, the Agent runs on each node in the cluster instead of a single Agent for all nodes. To enable, set defaults.federation.enabled to true.
• Comprehensive Helm Schema Validation: We have extended the JSON Schema validation to cover the entire configuration, providing much earlier feedback of any issues.
• New "helmless" tool: We have added a new "cloudzero-helmless" tool, which can be used to show the minimal difference between the default configuration and the configuration as used in the chart installation. This effectively recreates a minimized overrides overrides file.

Additional Enhancements

• Improved Load Balancing: Enhanced HTTP connection handling with periodic connection rotation to ensure proper load distribution across service replicas in multi-replica deployments.
• Observability and Debugging: Enhanced logging with configurable Prometheus log levels, and reduced log noise by moving health checks to trace level.
• Security and Reliability: Added default Pod Disruption Budgets for improved availability.
• Configuration ConfigMap: We now create a ConfigMap with the entire Helm chart configuration, to make debugging much easier.
• Reduced storage usage: By default, we now store metric files on the aggregator for 7 days instead of 90, significantly reducing storage requirements.

Notable Bug Fixes

• Eliminate unnecessary replays: Fixed an issue in the shipper which could cause the shipper to replay files repeatedly, instead of once, after receiving a request to replay a file.
• Allow out-of-order metrics: In some (relatively rare) cases, we were seeing metrics being dropped as they were arriving out of order. There is now a configuration option for setting a window where they will still be accepted, which defaults to 5 minutes.

Upgrade Steps

To upgrade to version 1.2.0, run the following command:

helm upgrade --install <RELEASE_NAME> cloudzero/cloudzero-agent -n <NAMESPACE> --create-namespace -f configuration.example.yaml --version 1.2.0

1.1.2

1.1.2 (2025-05-08)

This is a bugfix release.

Key Features

• Fix domain name for webhook service: Under certain situations the domain passed to the webhook configuration would not match the service name of the agent, causing a communication failure.