Conversation
zvonand
added
antalya-26.6
port-antalya
zvonand
force-pushed
the
feature/antalya-26.6/aouth-forward-to-datalake
branch
from
September 9, 2026 06:48
5d6ced1 to
33ba90f
Compare
zvonand
force-pushed
the
feature/antalya-26.6/aouth-forward-to-datalake
branch
from
September 9, 2026 20:59
4acc349 to
b5f1b30
Compare
|
cool |
Member
Author
|
Add support for amazon SSO |
zvonand
force-pushed
the
feature/antalya-26.6/aouth-forward-to-datalake
branch
from
September 21, 2026 12:15
dadc31f to
37a108b
Compare
A `DataLakeCatalog` database contacted its catalog as one shared service identity, so the catalog could not see, authorize or audit the human behind a query, and every ClickHouse user got identical catalog and storage access. With `oauth_forward_user_token = 1` the catalog is contacted as the user running the query. It requires the server-level `enable_token_forwarding` setting -- off by default, and re-read on every forwarded request, so turning it off stops already-authenticated sessions from forwarding -- and users who authenticated with a token (`Authorization: Bearer`, or `--jwt` on the native protocol). The token is carried on the session and copied into the query context. There is no service session to fall back to: a query without a token fails with `CATALOG_USER_TOKEN_NOT_AVAILABLE` instead of reaching the catalog as the service principal. How the token is used depends on the catalog type; the setting is rejected for every other type rather than silently ignored: - An Iceberg REST catalog takes bearer tokens directly, optionally after an RFC 8693 exchange at `oauth_token_exchange_uri`, with `oauth_subject_token_type` / `oauth_requested_token_type` describing the token and `oauth_forward_actor_token` sending the service principal's own token as the `actor_token` for delegation. Exchanged session tokens and the credentials the catalog vends are cached per user, bounded by `oauth_user_token_cache_ttl` and `vended_credentials_cache_ttl`. - Glue authenticates with AWS SigV4 and takes no bearer token, so the token goes to AWS STS instead: `AssumeRoleWithWebIdentity` against `aws_role_arn` returns temporary credentials that sign every Glue request and the S3 reads that follow, with the ClickHouse user name as the `RoleSessionName`. The Glue client and its credentials provider are built per user and cached on the token fingerprint, so one user is never handed another's session. `CREATE DATABASE` rejects the combinations that could not be honoured: forwarding without `aws_role_arn`, with static `aws_access_key_id` / `aws_secret_access_key`, or with `oauth_token_exchange_uri`. Rotating credentials with `ALTER DATABASE ... MODIFY SETTING catalog_credential` is carried out as the calling user when forwarding is on, and an `auth_generation` counter scopes every cached token and vended credential to the credentials it was minted under, so a request that authenticated before the ALTER cannot put a rotated-away artifact back into the caches. A lazy `/v1/config` load racing with the same ALTER is dropped rather than republishing the credentials it started with. An asynchronous insert keeps the token verified when it was pushed and flushes under it; the token's fingerprint takes part in the queue key, so inserts authenticated with different tokens never share a batch. Tests: the `test_datalake_token_forwarding`, `test_datalake_glue_token_forwarding` and `test_datalake_sso_lakekeeper` (Keycloak with Lakekeeper) integration suites; `gtest_rest_catalog_token_forwarding` and `gtest_sts_assume_role_with_web_identity` unit tests; the `05027_datalake_token_forwarding_masking`, `05028_datalake_token_forwarding_fail_closed` and `05053_glue_token_forwarding_validation` stateless tests.
zvonand
force-pushed
the
feature/antalya-26.6/aouth-forward-to-datalake
branch
from
September 21, 2026 12:21
37a108b to
1072cc3
Compare
Remove commentary that restates the code and keep concise explanations of credential lifetime, publication ordering, and test synchronization. Consolidate exchange and STS assertions, remove duplicate integration cases, and drop checks that do not exercise token expiry or rotation. Retain user isolation, credential-rotation races, async token lifetime, and fail-closed coverage. Related: #2329
Drop the three added stateless tests and their references. Runtime refusal and log-safety checks overlap retained unit and integration coverage, while masking uses existing behavior. The remaining cases restate straightforward settings-validation branches rather than cover distinct regressions. Related: #2329
Reduce the token-forwarding unit suite to five cases covering cache invalidation, failed preparation, and in-flight credential/config races. Remove unused forwarding helpers and simplify shared request recording. Related: #2329
Shorten documentation and comments, remove redundant code, and replace unnecessary integration setup with direct forwarding checks. Pass the query token to `updateMetadata` during Iceberg compaction. Validation: full unit-test build; 19 affected gtests and all 7 remaining integration cases passed. Related: #2329
Simplify request serialization, settings validation, and uncached token exchange. Remove unused state, instrumentation, fixture settings, and the STS rejection gtest already covered by integration testing. Trim repeated documentation and restore unrelated client OAuth encoding to the release-branch implementation. Validation: rebuilt ClickHouse and unit tests; affected gtests and all seven integration cases passed. After removing the redundant gtest, both remaining STS mock tests and both Glue integration tests passed. Related: #2329
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Forward auth tokens to data lake catalogs
CI/CD Options
Exclude tests:
Regression jobs to run: