Add support for zone-aware replication

[timonegk]

What this PR does:
Add support for zone-aware replication to ingesters. Similar to #632, but with less duplication.
I have documented the migration process and tested it in a cluster with six ingesters.

Which issue(s) this PR fixes:
Fixes #203.

Checklist

• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX], [DEPENDENCY]

[kd7lxl]

Why would you modify the chart when the chart already supports another technique for zone awareness that doesn't require further modification? Is there something I'm missing?

[nschad]

another technique for zone awareness

That requires the admission controller, otherwise you won't get the env-vars. If that is something you are ok with running, then yes you are correct. Otherwise there is currently no true "stand-a-lone" way of deploying zone-aware ingesters

right?

[kd7lxl]

another technique for zone awareness

That requires the admission controller, otherwise you won't get the env-vars. If that is something you are ok with running, then yes you are correct. Otherwise there is currently no true "stand-a-lone" way of deploying zone-aware ingesters

right?

Yes, I would run the admission controller. There is no need for the admission controller to be deployed in the same chart as cortex (nor would I couple them), so this is possible today. A guide doc may be the only contribution needed.

A strong benefit of the admission controller is that is does not require prior knowledge of the available zones. In contrast, the configuration method requires first collecting information about the target cluster and what zones are available, then configuring cortex for those zones. This is significant increased deployment complexity.

[nschad]

another technique for zone awareness

That requires the admission controller, otherwise you won't get the env-vars. If that is something you are ok with running, then yes you are correct. Otherwise there is currently no true "stand-a-lone" way of deploying zone-aware ingesters
right?

Yes, I would run the admission controller. There is no need for the admission controller to be deployed in the same chart as cortex (nor would I couple them), so this is possible today. A guide doc may be the only contribution needed.

A strong benefit of the admission controller is that is does not require prior knowledge of the available zones. In contrast, the configuration method requires first collecting information about the target cluster and what zones are available, then configuring cortex for those zones. This is significant increased deployment complexity.

on the other hand, this PR allows you have to one Deployment/StatefulSet per zone which can be beneficial if you want to do per-zone rollouts. For example facialited by the grafana rollout operator

I think there is a case to be made for both. Also the admission controller (even though the code is not complicated) is not maintained.

[timonegk]

@kd7lxl we are mostly benefitting from this change because of faster rollouts. With zone-aware ingesters, you can roll out one zone at a time, so a rollout out is O(zones) instead of O(ingesters). For us, that brings rollout times down from several hours to about 15 minutes.
If I understand your proposal correctly, this is not something we could achieve with the admission controller.

[kd7lxl]

@kd7lxl we are mostly benefitting from this change because of faster rollouts. With zone-aware ingesters, you can roll out one zone at a time, so a rollout out is O(zones) instead of O(ingesters). For us, that brings rollout times down from several hours to about 15 minutes. If I understand your proposal correctly, this is not something we could achieve with the admission controller.

That is a valuable benefit!

[nschad]

@kd7lxl can you have a look when you have time?

I'm biased :D

PR should be ready.

[kd7lxl]

While I have you here, what is the scaling strategy with zone-aware ingesters?

[timonegk]

I would recommend against autoscaling ingesters (zone-aware or not) and prefer static setup or manual scaling. The ingester ring is not very forgiving to changes.
I can think of the following issues:
When scaling down, you must ensure that unregister-on-shutdown is set to avoid the need to manually remove ingesters from the ring. However, this should not always be set lest a rolling restart will result in many changes in the ring and data will potentially be stored at different ingesters than expected while the rollout is running. To avoid new data missing from the ingester ring, you should also set the ingesters to READONLY for at least query-store-after before removing the ingester, or set flush-blocks-on-shutdown and wait for query-store-after between scaling down floor(RF/2) replicas.
When scaling up, new ingesters join the ring, so you should disable shuffle sharding and use shard-by-all-labels to avoid that only a subring of ingesters is queried (which could only contain the new ingesters), or wait for query-store-after between scaling up RF-1 replicas.
The ingester HPA default values currently scales one replica per 3h, so even for the default settings, users would need to adapt it.
In a zone-aware setup, most of these concerns remain. Scaling within a zone is easier, so adding/removing multiple replicas in the same zone could be done in parallel, but the same constraints as in the single-zone setup apply between the zones (e.g. wait query-store-after before starting the scaledown in the second zone). I am not aware that this coordination is currently possible in Kubernetes.