-
Notifications
You must be signed in to change notification settings - Fork 173
Add support for zone-aware replication #668
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
15 commits
Select commit
Hold shift + click to select a range
f5c4ae4
ingester: support zone-awareness
timonegk 5c16a9d
remove accidental "end"
timonegk 1dfbefc
add rollout guide and adapt templates where necessary
timonegk fdf275e
remove accidentally added extraLabels
timonegk 1ecc473
move shuffle shard disable to beginning
timonegk 610f981
update migration guide
timonegk ea3d6a7
don't let new ingesters leave the ring on restart
timonegk 5aab646
update documentation
timonegk b235278
add explicit fails for ingester HPA and deployment
timonegk 231e352
fix zone name in excluded-zones
timonegk fa91ac5
update read path migration
timonegk 2b6ee31
Update migration documentation title and link in changelog
timonegk 469bc74
regenerate Chart.lock
timonegk 7c9c4b3
add grafana helm repo
timonegk d565d13
Regenerate README
timonegk File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Binary file not shown.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,87 @@ | ||
| --- | ||
| layout: page | ||
| title: Migrate to zone-aware replication | ||
| parent: Guides | ||
| has_children: false | ||
| has_toc: false | ||
| --- | ||
|
|
||
| # Migrate to zone-aware replication | ||
| {: .no_toc } | ||
|
|
||
| ## Table of contents | ||
| {: .no_toc .text-delta } | ||
|
|
||
| 1. TOC | ||
| {:toc} | ||
|
|
||
| ## Overview | ||
|
|
||
| This migration guide shows how to migrate to zone-aware replication without downtime or data loss. | ||
| With zone-aware replication, each replica of incoming samples is distributed across ingesters in different zones. This means that loss of a full zone is possible without downtime. | ||
|
|
||
| The general migration process is the following: New stateful sets are created, the write traffic is routed to them, the read traffic is routed to them, the old stateful set is disabled. | ||
| During the migration, it is ensured that at most one ingester is unavailable at the time, and that an ingester's data is always written to persistent storage before it is shut down. | ||
|
|
||
| The chart makes use of the [rollout-operator](https://github.com/grafana/rollout-operator) to coordinate rollouts of the stateful sets. This will automatically set the stateful set's update strategy to `OnDelete`. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| Make sure to set the following settings before starting the migration: | ||
| - Ingesters are deployed as a stateful set (deployment is currently not supported) | ||
| - Autoscaling is disabled (autoscaling is not supported for zone-aware ingesters) | ||
| - `podManagementPolicy` is "OrderedReady" (default), not "Parallel" (OrderedReady creates pods consecutively when scaling up or down) | ||
| - `frontend_address` is set in the ruler config (make the ruler read from the queriers, not directly from the ingesters. Otherwise, recording and alerting rules may not be evaluated correctly during migration) | ||
| - The replication factor and the number of zones are at least 3 and the replication factor is not larger than the number of zones | ||
|
|
||
| ## Migration steps | ||
|
|
||
| **Important**: Always continue the next step only when all pods are in the ready state. | ||
|
|
||
| 1. Before starting the migration, you should ensure that the querier uses all ingesters during the migration. This means that shuffle sharding should be disabled and sharding by all labels should be enabled. | ||
| It is sufficient to set these settings on the querier using `querier.extraArgs`. Set `distributor.sharding-strategy` to `default` and `distributor.shard-by-all-labels` to `"true"` there. | ||
| Warning: This may increase resource usage of the queriers. | ||
|
|
||
| 1. Set `ingester.zoneAwareReplication.enabled=true`, `ingester.zoneAwareReplication.migration.enabled=true`, `ingester.zoneAwareReplication.zones` to the desired zones but with `replicas=0`. Set `rollout_operator.enabled=true`. Upgrade the chart. | ||
| ```yaml | ||
| ingester: | ||
| zoneAwareReplication: | ||
| enabled: true | ||
| migration: | ||
| enabled: true | ||
| zones: | ||
| - name: zone-a | ||
| replicas: 0 | ||
| nodeSelector: | ||
| topology.kubernetes.io/zone: zone-a | ||
| - name: zone-b | ||
| replicas: 0 | ||
| nodeSelector: | ||
| topology.kubernetes.io/zone: zone-b | ||
| - name: zone-c | ||
| replicas: 0 | ||
| nodeSelector: | ||
| topology.kubernetes.io/zone: zone-c | ||
| ``` | ||
| The stateful sets will be scaled up in the next steps and not created at once to ensure that at most one ingester is unavailable at a time. | ||
|
|
||
| 1. In `ingester.zoneAwareReplication.zones`, set `replicas` to the desired replicas for **the first** zone, the install the Helm chart. | ||
|
|
||
| 1. Repeat the process for the other zones. | ||
|
|
||
| 1. Enable zone-awareness on the write path by setting `ingester.zoneAwareReplication.migration.writePath=true` and install the Helm chart. This makes the distributors ship data to the new ingesters while the queriers still use all ingesters. Wait for `querier.query_store_after` so that the data that is still on the old ingesters can be queried from the object storage. If `query_store_after` is unset, wait at least `3 x bucket_store.sync_interval` (default 3x15m). | ||
| This also disables the distributors from writing to the old ingesters. | ||
|
|
||
| 1. Enable zone-awareness on the read path by setting `ingester.zoneAwareReplication.migration.readPath=true` and install the Helm chart. This makes the queriers use the new ingesters. | ||
| This also disables the queriers from reading from the old ingesters. | ||
|
|
||
| 1. Set `ingester.replicas` to 0. This will scale down the stateful set, one replica at a time, so that the ring remains healthy. | ||
|
|
||
| 1. Remove all values below `ingester.zoneAwareReplication.migration`. This will delete the old stateful set. | ||
|
|
||
| 1. If you have set any querier arguments in the first step, wait `-querier.shuffle-sharding-ingesters-lookback-period` before removing `querier.extraArgs`. | ||
|
|
||
| ## Faster rollouts | ||
|
|
||
| With zone-awareness enabled, it is possible to roll all ingesters in a zone simultaneously. | ||
| If you want to benefit from these faster rollouts, set `ingester.zoneAwareReplication.maxUnavailable` to the number of replicas per zone and set `ingester.statefulSet.podManagementPolicy` to "Parallel". This will require recreating the stateful sets. Use `kubectl delete sts <...> --cascade=orphan` to delete only the stateful set, not the pods. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
While I have you here, what is the scaling strategy with zone-aware ingesters?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would recommend against autoscaling ingesters (zone-aware or not) and prefer static setup or manual scaling. The ingester ring is not very forgiving to changes.
I can think of the following issues:
When scaling down, you must ensure that
unregister-on-shutdownis set to avoid the need to manually remove ingesters from the ring. However, this should not always be set lest a rolling restart will result in many changes in the ring and data will potentially be stored at different ingesters than expected while the rollout is running. To avoid new data missing from the ingester ring, you should also set the ingesters toREADONLYfor at leastquery-store-afterbefore removing the ingester, or setflush-blocks-on-shutdownand wait forquery-store-afterbetween scaling downfloor(RF/2)replicas.When scaling up, new ingesters join the ring, so you should disable shuffle sharding and use
shard-by-all-labelsto avoid that only a subring of ingesters is queried (which could only contain the new ingesters), or wait forquery-store-afterbetween scaling upRF-1replicas.The ingester HPA default values currently scales one replica per 3h, so even for the default settings, users would need to adapt it.
In a zone-aware setup, most of these concerns remain. Scaling within a zone is easier, so adding/removing multiple replicas in the same zone could be done in parallel, but the same constraints as in the single-zone setup apply between the zones (e.g. wait
query-store-afterbefore starting the scaledown in the second zone). I am not aware that this coordination is currently possible in Kubernetes.