-
Notifications
You must be signed in to change notification settings - Fork 34
Vault: Revamp audit logging docs #631
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
Rearrange the order of sections in the main Audit Logging documentation to keep related content closer together.
Remove the paragraph on blocking audit device failures. This content predates the rearchitecture of Vault's audit logging system and is no longer relevant. Vault's audit logging pipeline currently enforces a 10-second timeout for all audit devices.
Fix links to related concepts and API documentation. Add a note to recommend that users test the performance impact of their filtering rules in a non-production environment before deploying to production.
Vercel Previews Deployed
|
Broken Link CheckerNo broken links found! 🎉 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Worked through the best practices doc. Will pick up the remaining docs tomorrow.
@@ -0,0 +1,88 @@ | |||
--- | |||
layout: docs | |||
page_title: Audit Logging Best Practices |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
page_title: Audit Logging Best Practices | |
page_title: Audit logging best practices |
Style correction: use sentence case
description: Recommendations for setting up audit logging in HashiCorp Vault. | ||
--- | ||
|
||
# Best practices |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
# Best practices | |
# Best practices for audit logging |
<Highlight title="Mileage may vary"> | ||
This page contains recommendations that are <strong>generally applicable</strong> to Vault customers. You should independently evaluate whether each recommendation on this page is advisable in the context of your organization's environment and use cases for Vault. | ||
</Highlight> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
<Highlight title="Mileage may vary"> | |
This page contains recommendations that are <strong>generally applicable</strong> to Vault customers. You should independently evaluate whether each recommendation on this page is advisable in the context of your organization's environment and use cases for Vault. | |
</Highlight> | |
The following recommendations apply generally to most Vault deployments. But you | |
should always independently evaluate whether a given recommendation makes sense | |
in the context of your deployment and use of Vault. |
Style correction: write in active voice, limit use of asides
|
||
## Test configuration changes | ||
|
||
You should test every configuration change to Vault's audit logging in a non-production environment. The non-production environment should closely mirror the production environment in all aspects relevant to the audit logging configuration and should include performance benchmarking under production-like loads. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You should test every configuration change to Vault's audit logging in a non-production environment. The non-production environment should closely mirror the production environment in all aspects relevant to the audit logging configuration and should include performance benchmarking under production-like loads. | |
Always test your audit logging changes in a non-production environment that | |
closely mirrors your production environment. Your test environment should | |
include performance benchmarking under production-like loads and match in all | |
aspects relevant to the audit logging configuration you want to test. |
Style correction: write in active voice, use directive language
|
||
## Enable audit devices at cluster initialization | ||
|
||
You should ensure that at least one audit device is enabled immediately after a new Vault cluster is initialized. The audit device's configuration must be [valid for all Vault cluster nodes](/vault/docs/audit#enabling-and-disabling-audit-devices). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You should ensure that at least one audit device is enabled immediately after a new Vault cluster is initialized. The audit device's configuration must be [valid for all Vault cluster nodes](/vault/docs/audit#enabling-and-disabling-audit-devices). | |
Ensure your configuration enables at least one audit device immediately after a | |
new Vault cluster initializes. The configuration of your initialization audit | |
device must be | |
[valid for all Vault cluster nodes](/vault/docs/audit#enabling-and-disabling-audit-devices). |
Style correction: write in active voice, avoid possessives
## Configure audit devices | ||
|
||
You should review all configuration options that are [common](/vault/api-docs/system/audit#parameters) to all audit device types *and* those specific to a given audit device type (`file`, `syslog`, `socket`) and ensure that all audit devices are configured in a way that is appropriate for your organization and deployment. | ||
|
||
In general, we recommend you set the following options: | ||
|
||
- `elide_list_responses = true`. This reduces the volume of log data produced by API list requests (see [Eliding list response bodies](/vault/docs/audit#eliding-list-response-bodies)). | ||
- `hmac_accessor = false`. A token accessor is generally not considered sensitive information: it does not grant any access to Vault, but rather serves as a unique identifier for a single Vault token. A token accessor enables you to [revoke](/vault/docs/commands/token/revoke) its corresponding Vault token, even when you do not know the token itself. By disabling token accessor hashing, if you identify unusual access patterns in your audit logs, you can quickly revoke the corresponding Vault token through its accessor. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
## Configure audit devices | |
You should review all configuration options that are [common](/vault/api-docs/system/audit#parameters) to all audit device types *and* those specific to a given audit device type (`file`, `syslog`, `socket`) and ensure that all audit devices are configured in a way that is appropriate for your organization and deployment. | |
In general, we recommend you set the following options: | |
- `elide_list_responses = true`. This reduces the volume of log data produced by API list requests (see [Eliding list response bodies](/vault/docs/audit#eliding-list-response-bodies)). | |
- `hmac_accessor = false`. A token accessor is generally not considered sensitive information: it does not grant any access to Vault, but rather serves as a unique identifier for a single Vault token. A token accessor enables you to [revoke](/vault/docs/commands/token/revoke) its corresponding Vault token, even when you do not know the token itself. By disabling token accessor hashing, if you identify unusual access patterns in your audit logs, you can quickly revoke the corresponding Vault token through its accessor. | |
## Fully configure your audit devices | |
In addition to the following deployment recommendations, we suggest reviewing | |
the type-specific configuration options for each audit device type in addition | |
to the [common](/vault/api-docs/system/audit#parameters) configuration options. | |
Reviewing type-specific options helps ensure you configure your audit devices | |
appropriately for your organization. | |
<Tabs> | |
<Tab heading="Recommended options"> | |
We recommend setting the following options for all audit devices: | |
@include 'cli/audit/options/elide_list_responses.mdx' | |
<hr /> | |
@include 'cli/audit/options/hmac_accessor.mdx' | |
</Tab> | |
<Tab heading="File"> | |
When you configure a `file` audit device, we recommend using a dedicated volume | |
or partition for audit logs to protect against other workloads on the system | |
taking up the disk space intended for your log files. | |
Vault does not rotate log files automatically and unrotated logs can grow to | |
consume all available disk space which eventually makes the audit device | |
unavailable. We recommend establishing a log rotation process appropriate for | |
your organization, deployment, and applicable policies with purpose-built | |
systems like `logrotate`. | |
You can configure your log rotation system to send Vault an `HUP` signal, which | |
tells Vault to start writing audit log entries to the new log file. | |
<Tip title="Logging with Kubernetes"> | |
When running Vault on a Kubernetes cluster, a common approach is to configure | |
a `file` audit device with `file_path` set to `stdout`. Using `stdout` for the | |
file path writes the audit logs to standard output for the container, which | |
lets you process the logs with a Kubernetes cluster-level log collector. | |
If you choose to use cluster-level log collection, we recommend configuring | |
the audit device with a `prefix` so your log collector can separate audit logs | |
from server logs. | |
</Tip> | |
</Tab> | |
<Tab heading="Syslog"> | |
To configure a `syslog` audit device, you must deploy Vault on Unix and we | |
recommend using a TCP listener so you can successfuly record large audit log | |
entries. Vault can only write audit logs to the default Unix syslog service | |
running on the same host as the Vault server. | |
Vault logs API requests and responses in detail, so some audit log entries may | |
exceed the message size allowed for UDP transmission of syslog messages. | |
</Tab> | |
<Tab heading="Socket"> | |
When you configure a `socket` audit device, we recommend configuring a local | |
service to bind a Unix socket. Socket devices using TCP and UDP listeners can | |
lead to lost audit log data if something interrupts the TCP connection or the | |
receiving UDP endpoint becomes unavailable. | |
</Tab> | |
</Tabs> |
You should configure monitoring and alerting on the health of your audit devices. | ||
|
||
Vault produces several [telemetry metrics](/vault/docs/internals/telemetry/metrics/audit) related to audit logging. You should especially monitor `vault.audit.log_request_failure`, `vault.audit.log_response_failure`, `vault.audit.{DEVICE}.log_request`, and `vault.audit.{DEVICE}.log_response` for spikes, which can indicate one or more audit devices failing. | ||
|
||
Additionally, you should configure monitoring and alerting specific to each type of audit device, such as available disk space, disk IOPS, and log rotation status for `file` audit devices. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You should configure monitoring and alerting on the health of your audit devices. | |
Vault produces several [telemetry metrics](/vault/docs/internals/telemetry/metrics/audit) related to audit logging. You should especially monitor `vault.audit.log_request_failure`, `vault.audit.log_response_failure`, `vault.audit.{DEVICE}.log_request`, and `vault.audit.{DEVICE}.log_response` for spikes, which can indicate one or more audit devices failing. | |
Additionally, you should configure monitoring and alerting specific to each type of audit device, such as available disk space, disk IOPS, and log rotation status for `file` audit devices. | |
We strongly recommend monitoring and alerting on the health of your audit | |
devices, with specific alerts for each type of audit device. For example, | |
alerting on available disk space, disk IOPS, and log rotation status for `file` | |
audit devices. | |
Vault produces several audit logging | |
[telemetry metrics](/vault/docs/internals/telemetry/metrics/audit) and we | |
recommend monitoring the following metrics for spikes, which can indicate one or | |
more audit devices failing: | |
- [`vault.audit.log_request_failure`](/vault/docs/internals/telemetry/metrics/audit#vault-audit-log_request_failure) | |
- [`vault.audit.log_response_failure`](/vault/docs/internals/telemetry/metrics/audit#vault-audit-log_response_failure) | |
- [`vault.audit.{DEVICE}.log_request`](/vault/docs/internals/telemetry/metrics/audit#vault-audit-device-log_request) | |
- [`vault.audit.{DEVICE}.log_response`](/vault/docs/internals/telemetry/metrics/audit#vault-audit-device-log_response) |
For each authentication backend and secrets engine mount you configure in Vault, you should evaluate which request and response attributes need *not* be [hashed](/vault/docs/audit#hashing-of-sensitive-values) in the audit logs. | ||
|
||
<Tip> | ||
Hashing should be left on for attributes that are sensitive (e.g., passwords, private keys) or that may be excessively large for your audit log. | ||
</Tip> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For each authentication backend and secrets engine mount you configure in Vault, you should evaluate which request and response attributes need *not* be [hashed](/vault/docs/audit#hashing-of-sensitive-values) in the audit logs. | |
<Tip> | |
Hashing should be left on for attributes that are sensitive (e.g., passwords, private keys) or that may be excessively large for your audit log. | |
</Tip> | |
We recommend [hashing](/vault/docs/audit#hashing-of-sensitive-values) for all | |
sensitive attributes (for example, passwords and private keys) and any attribute | |
may be excessively large for your audit log. | |
For all other attributes, we recommend reviewing each authentication backend and | |
secrets engine mount you configure in Vault to evaluate which request and | |
response attributes do not need hashing in the audit logs. |
Hashing should be left on for attributes that are sensitive (e.g., passwords, private keys) or that may be excessively large for your audit log. | ||
</Tip> | ||
|
||
You can disable hashing for individual request and response attributes through the `audit_non_hmac_request_keys` and `audit_non_hmac_response_keys` attributes in the "tune" API endpoints of each [authentication backend](https://developer.hashicorp.com/vault/api-docs/system/auth#tune-auth-method) and [secret engine](https://developer.hashicorp.com/vault/api-docs/system/mounts#tune-mount-configuration) mount. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can disable hashing for individual request and response attributes through the `audit_non_hmac_request_keys` and `audit_non_hmac_response_keys` attributes in the "tune" API endpoints of each [authentication backend](https://developer.hashicorp.com/vault/api-docs/system/auth#tune-auth-method) and [secret engine](https://developer.hashicorp.com/vault/api-docs/system/mounts#tune-mount-configuration) mount. | |
You can disable hashing for individual request and response attributes with the | |
`audit_non_hmac_request_keys` and `audit_non_hmac_response_keys` attributes for | |
the following tuning endpoints: | |
- [`/sys/auth/{path}/tune`](/vault/api-docs/system/auth#tune-auth-method) | |
- [`/sys/mounts/{path}/tune`](/vault/api-docs/system/mounts#tune-mount-configuration) |
You should configure monitoring and alerting of Vault's audit logs for unusual or suspicious activity. | ||
|
||
Examples of events to monitor include root token creation, root token usage, modifications to audit loggig configuration, spikes in authentication failures, and spikes in permission denied failures. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You should configure monitoring and alerting of Vault's audit logs for unusual or suspicious activity. | |
Examples of events to monitor include root token creation, root token usage, modifications to audit loggig configuration, spikes in authentication failures, and spikes in permission denied failures. | |
We strongly recommend monitoring and alerting for unusual or suspicious activity | |
in your audit logs related to: | |
- root token creation | |
- root token usage | |
- modifications to audit logging configuration | |
- spikes in authentication failures | |
- spikes in permission denied failures |
requests to Vault, and their responses. Because every operation with Vault is an API | ||
request/response, when using a single audit device, the audit log contains _every_ interaction with | ||
the Vault API, including errors - except for a few paths which do not go via the audit system. | ||
Audit devices enable the detailed recording of all Vault API requests and responses, except for a small set of [API endpoints](#excepted-api-endpoints). An audit device can be a [file](/vault/docs/audit/file), a [syslog server](/vault/docs/audit/syslog), or a [socket](/vault/docs/audit/socket). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Audit devices enable the detailed recording of all Vault API requests and responses, except for a small set of [API endpoints](#excepted-api-endpoints). An audit device can be a [file](/vault/docs/audit/file), a [syslog server](/vault/docs/audit/syslog), or a [socket](/vault/docs/audit/socket). | |
With a a small set of [exceptions](#excepted-api-endpoints), Vault audit devices enable the detailed recording of all API requests and responses. Audit devices can be [files](/vault/docs/audit/file), a [syslog server](/vault/docs/audit/syslog), or a [socket](/vault/docs/audit/socket). |
Better to highlight the exceptions first, instead of claiming you can audit "everything" and then contradicting ourselves.
sys/storage/raft/bootstrap | ||
sys/storage/raft/join | ||
sys/internal/ui/feature-flags | ||
<Tip> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
<Tip> |
This isn't a tip, it's just more information about audit devices
sys/internal/ui/feature-flags | ||
<Tip> | ||
Audit logs are distinct from server logs (also known as operational logs). Server logs emit information about the Vault server process, such as process errors and configuration warnings. Audit logs record the details of every request and response received and sent by the Vault API. Refer to the <a href="/vault/tutorials/monitoring/troubleshooting-vault#vault-server-logs">Troubleshoot Vault</a> guide for more information about server logs. | ||
</Tip> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
</Tip> |
sys/storage/raft/join | ||
sys/internal/ui/feature-flags | ||
<Tip> | ||
Audit logs are distinct from server logs (also known as operational logs). Server logs emit information about the Vault server process, such as process errors and configuration warnings. Audit logs record the details of every request and response received and sent by the Vault API. Refer to the <a href="/vault/tutorials/monitoring/troubleshooting-vault#vault-server-logs">Troubleshoot Vault</a> guide for more information about server logs. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Audit logs are distinct from server logs (also known as operational logs). Server logs emit information about the Vault server process, such as process errors and configuration warnings. Audit logs record the details of every request and response received and sent by the Vault API. Refer to the <a href="/vault/tutorials/monitoring/troubleshooting-vault#vault-server-logs">Troubleshoot Vault</a> guide for more information about server logs. | |
Audit logs are distinct from server logs (also known as operational logs). | |
Server logs emit information about the Vault server process, such as process | |
errors and configuration warnings. Audit logs record the details of every | |
request received and response and sent by the Vault API. | |
Refer to the | |
[Troubleshoot Vault](/vault/tutorials/monitoring/troubleshooting-vault#vault-server-logs) | |
tutorial for more information about server logs. |
|
||
The non-audited paths are: | ||
Refer to the [best practices](/vault/docs/audit/best-practices) page for important elements to consider when configuring Vault's audit devices. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Refer to the [best practices](/vault/docs/audit/best-practices) page for important elements to consider when configuring Vault's audit devices. | |
Refer to the [best practices](/vault/docs/audit/best-practices) page for important considerations before configuring your audit devices. |
sys/metrics | ||
sys/pprof/* | ||
sys/in-flight-req | ||
When a new Vault cluster is initialized, no auditing is enabled. Audit devices must be enabled via the [API](/vault/api-docs/system/audit#enable-audit-device) or related tooling (such as the Vault CLI or the Terraform Vault provider). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When a new Vault cluster is initialized, no auditing is enabled. Audit devices must be enabled via the [API](/vault/api-docs/system/audit#enable-audit-device) or related tooling (such as the Vault CLI or the Terraform Vault provider). | |
When a new Vault cluster initializes, auditing is disabled. You must explicitly | |
enable audit devices with the | |
[Vault CLI](/vault/docs/commands/audit/enable), | |
[API](/vault/api-docs/system/audit#enable-audit-device), or the | |
[Terraform Vault provider](/vault/docs/configuration/programmatic-management). |
Style correction: write in active voice
|
||
## Enabling multiple devices | ||
When enabling an audit device, options can be passed to it to configure it. There is a set of [common configuration options](/vault/api-docs/system/audit#common-configuration-options) that apply to all types of audit devices. In addition, each type of audit device has its own set of options; refer to the documentation on the left for each type. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When enabling an audit device, options can be passed to it to configure it. There is a set of [common configuration options](/vault/api-docs/system/audit#common-configuration-options) that apply to all types of audit devices. In addition, each type of audit device has its own set of options; refer to the documentation on the left for each type. | |
Each audit device has unique configuration options in addition to the | |
[common configuration options](/vault/api-docs/system/audit#common-configuration-options). |
Style correction: write simply
When multiple audit devices are enabled, Vault will attempt to send the audit | ||
logs to all of them. This allows you to not only have redundant copies, but also | ||
a way to check for data tampering in the logs themselves. | ||
For example, the command below enables the file audit device and configures it to write to the file at the `file_path` location, to disable [hashing](#hashing-of-sensitive-values) of token accessors, and to [elide list responses](#eliding-list-response-bodies): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For example, the command below enables the file audit device and configures it to write to the file at the `file_path` location, to disable [hashing](#hashing-of-sensitive-values) of token accessors, and to [elide list responses](#eliding-list-response-bodies): | |
For example, the command below enables a `file` type audit device that writes to | |
the file at `/var/log/vault_audit.log`. The additional options disable | |
[hashing](#hashing-of-sensitive-values) of token accessors, and enables | |
[elide list responses](#eliding-list-response-bodies). |
Style correction: write in active voice
Revamp the documentation for Vault's audit logging. Changes include:
These changes cover most of VSM-226 and were reviewed internally by Engineering and Resident Architects/Engineers from the Professional Services team (see Google Docs history).
The changes also include minor fixes for the Audit Filtering docs page, which were identified in the course of Engineering review.