Feedback on "Data tiers" docs

[kilfoyle]

This is feedback that we received here from @stefnestor (Thanks Stef!) on the data tiers documentation.

Many of these things seem to be questions that we'd need help from either Development or Support to answer, so we may need to expand this beyond being a simple "docs" issue.

For now, I'll park the issue here so that we don't lose this valuable feedback.

---

• data tiers
  • could/should header#4 show on the right-side TOC?
    • 
  • Per this searchable snapshots can happen on hot+cold+frozen for quotes but only guaranteed frozen. there's various wrong logic from not understanding SS can happen on hot in the steps after these quotes

    The hot and warm tiers store regular indices, while the frozen tier stores searchable snapshots. However, the cold tier can store either regular indices or searchable snapshots.

    When data reaches the cold or frozen phases, it is automatically converted to a searchable snapshot by ILM.

  • "Move shards off the nodes to be removed from the cluster."
    • they may want to push farther down not up (e.g. warm>cold not warm>hot, maybe noting frozen would be invalid option)
    • needs to ensure to call out they need to drain all shards (confirm with CAT Shards) not just update allocation (not sure how you want to handle node attrs saying "we'll do that during plan") but calling out because frequent support volume for total_shards_per_node and/or watermark on destination to block migrations off
  • "If you do not intend to delete this data, you should manually restore each of the searchable snapshot indices to a regular index before disabling the data tier, by following these steps" fully mounted could just be ported across hot+cold, no? it doesn't have to be rehydrated.
  • "Capture a comprehensive list of index and searchable snapshot names." I believe this assumes repo is found-snapshots but users can have custom repos so this isn't sufficiently valid
  • "Remove the associated ILM policy (set it to null). If you want to apply a different ILM policy, follow the steps to Switch lifecycle policies." if you're modifying it you always set to null and then remove and then add the new ilm, you do not set <new-policy-name>
  • "Optionally, specify the desired number of replica shards." > "index.lifecycle.rollover_alias": "<alias-for-rollover>": we usually don't re-add a rollover alias because we're not ingesting more into the index, instead we usually set "index.lifecycle.indexing_complete": true to bypass the potential new ilm policy's rollover action (which noop if NA)
  • FWIW
    • the searchable snapshot rehydration section comes up if you need to fix data not just remove a data tier so it may potentially be worth a different section in the docs
    • see also KB > Resolve > 4) Clean-up > Searchable Snapshots
  • these are not equal nor even generally related sub-sections. IMO maybe "remove a data tier" should be a par-header since it applies across the board saying something like "to remove a data tier we recommend draining off shards first to avoid data loss, ECE+ECK+ECH will enforce this"
    • 
    • if you disagree then FWIW "Elastic Cloud Hosted and Elastic Cloud Enterprise try to move all data from the nodes that are removed during plan changes." also applies to ECK software-side.