TBS: Document discard_on_write_failure + expose it to the APM Integration

[lucabelluccini]

Potential follow up of #15159

• Docs: document discard_on_write_failure
• Integration (APM): expose the setting via
  • https://github.com/elastic/integrations/blob/a55e9eecb844d1d9192eb8584b81c687a213ef1b/packages/apm/agent/input/template.yml.hbs#L74
  • https://github.com/elastic/integrations/blob/main/packages/apm/manifest.yml

Changes required

This task will be similar to the sampling.tail.ttl task in #13525 in terms of the changes required

[carsonip]

It was intentional to have sampling.tail.discard_on_write_failure undocumented, as it was supposed to be an escape hatch for users who are facing TBS storage limit issues while we look for a long term solution.

I understand the pain to configure this config (and many more configs not exposed in apm integration package) and I agree we should make it easier.

Regarding documentation, I'm not sure if the team is committed to support this config in the long run. cc @simitt

[isaacaflores2]

• APM Integrations PR: apm: Add config for tail-based sampling discard on write integrations#13950
  • v8.19.0 Backport: apm: Backport TBS ttl and discard_on_write_failure to 8.19 integrations#14029
  • Validate agent policy for 9.1.0-SNAPSHOT using local stack with elastic-package stack up -d --version 9.1.0-SNAPSHOT
  • Validate agent policy for 8.19.0
• Docs:
  • 9.1 docs PR: apm: Document sampling.tail.discard_on_write_failure config docs-content#1453
  • 8.x docs PR: [apm]: Document sampling.tail.discard_on_write_failure config observability-docs#4908
• APM Integration policy UI Issue: [APM] Support sampling discard_on_write_failure configuration in apm integration policy kibana#221441
  • Perform E2E validation to confirm the config is correctly applied from the UI
    • 9.1.0 complete as part of release plan
    • 8.19.0
• Update any APM Server docs

[isaacaflores2]

I documented all the steps here to validate the APM integration changes per @rubvs recommendation.
apm-integrations-validation-steps.md

[simitt]

@isaacaflores2 IMO https://github.com/elastic/apm-server/tree/main/dev_docs would be a good place for documenting how to test integration package changes for APM. You can reduce the discard_on_write_failure specifics, or keep them in as an example.

[isaacaflores2]

UI PR: elastic/kibana#224479

edit: UI PR merged and e2e testing with the UI in progress (9.1.0 has been completed as part of the release test plan). 8.19.0 still pending.

We are still waiting for the release to complete so we can merge the doc PR