CLDSRV-992: Wait for the file data daemon before the S3 API - #6277
Merged
bert-e merged 1 commit intoSep 10, 2026
Merged
Conversation
Contributor
Hello anurag4dsb,My role is to assist you with the merge of this Available options
Available commands
Status report is not available. |
Contributor
Incorrect fix versionThe
Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:
Please check the |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files
@@ Coverage Diff @@
## development/9.3 #6277 +/- ##
===================================================
+ Coverage 85.28% 85.32% +0.03%
===================================================
Files 206 206
Lines 13435 13435
===================================================
+ Hits 11458 11463 +5
+ Misses 1977 1972 -5
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
anurag4DSB
force-pushed
the
improvement/CLDSRV-992-wait-for-file-backend-ports
branch
from
September 8, 2026 06:13
c853cdb to
7d31356
Compare
anurag4DSB
force-pushed
the
improvement/CLDSRV-992-wait-for-file-backend-ports
branch
2 times, most recently
from
September 8, 2026 06:32
9572552 to
cec915a
Compare
This was referenced Sep 8, 2026
anurag4DSB
force-pushed
the
improvement/CLDSRV-992-wait-for-file-backend-ports
branch
from
September 8, 2026 10:57
cec915a to
b8e6df3
Compare
anurag4DSB
changed the base branch from
development/9.3
to
improvement/CLDSRV-992-server-access-log-match
September 8, 2026 10:58
anurag4DSB
force-pushed
the
improvement/CLDSRV-992-server-access-log-match
branch
from
September 8, 2026 11:02
a4c6f76 to
5036d12
Compare
anurag4DSB
force-pushed
the
improvement/CLDSRV-992-wait-for-file-backend-ports
branch
from
September 8, 2026 11:06
b8e6df3 to
365405b
Compare
anurag4DSB
force-pushed
the
improvement/CLDSRV-992-server-access-log-match
branch
from
September 8, 2026 11:07
5036d12 to
65cb3c4
Compare
anurag4DSB
force-pushed
the
improvement/CLDSRV-992-wait-for-file-backend-ports
branch
from
September 8, 2026 11:07
365405b to
86af73d
Compare
anurag4DSB
marked this pull request as ready for review
September 8, 2026 11:17
anurag4DSB
force-pushed
the
improvement/CLDSRV-992-wait-for-file-backend-ports
branch
3 times, most recently
from
September 8, 2026 11:31
efe3fd8 to
11b1f52
Compare
anurag4DSB
changed the title
anurag4DSB
force-pushed
the
improvement/CLDSRV-992-server-access-log-match
branch
2 times, most recently
from
September 8, 2026 11:47
278feb5 to
c79db25
Compare
anurag4DSB
force-pushed
the
improvement/CLDSRV-992-wait-for-file-backend-ports
branch
from
September 8, 2026 11:47
11b1f52 to
d979972
Compare
anurag4DSB
force-pushed
the
improvement/CLDSRV-992-server-access-log-match
branch
from
September 8, 2026 13:18
c79db25 to
d000474
Compare
`yarn start` launches the S3 API and the file daemons as separate parallel
processes (npm-run-all --parallel start_dmd start_s3server), so they race. On
run 33734792029 the API bound :8000 at t+10s and served writes from t+15s while
the dataserver only bound :9991 at t+84s, and the ten HTTP 500s in between are
all ECONNREFUSED 0.0.0.0:9991. The same subdir pre-creation step took 2.7s on a
healthy runner, which is why this is intermittent.
Context on why cloudserver does not catch this itself. metadata.setup() opens
the metadata client and throws if 9990 is unreachable, so the API cannot bind
without working metadata, which matches the run: CreateBucket and every metadata
read succeeded and only the data path failed. Nothing guards the data daemon,
because clientCheck returns a hardcoded { code: 200, message: 'OK' } for clients
that implement no probe and DataFileInterface implements none. So the deep
healthcheck only really checks metadata, and only for the mongo and bucketd
clients. Adding the file backend to the deep check would let the product gate
itself and make this change unnecessary, but that belongs in Arsenal.
Gate on 9991 in the four jobs that use the file data backend: file-ft-tests,
kmip-ft-tests, kmip-cluster-ft-tests and sse-kms-migration-tests, the last of
which starts a fresh cloudserver container in a second phase and needs the gate
there too. 120s because 84s was observed; a healthy run pays nothing since the
wait returns as soon as the port answers.
Clears CLDSRV-992 row F1 (ten of its twelve failures) and is the leading
candidate for rows F2 and F3, whose jobs share this boot path.
Issue: CLDSRV-992
anurag4DSB
force-pushed
the
improvement/CLDSRV-992-wait-for-file-backend-ports
branch
from
September 8, 2026 13:19
d979972 to
7904b09
Compare
anurag4DSB
added a commit
that referenced
this pull request
Sep 8, 2026
`yarn start` launches the S3 API and the file daemons as separate parallel
processes (npm-run-all --parallel start_dmd start_s3server), so they race. On
run 33734792029 the API bound :8000 at t+10s and served writes from t+15s while
the dataserver only bound :9991 at t+84s, and the ten HTTP 500s in between are
all ECONNREFUSED 0.0.0.0:9991. The same subdir pre-creation step took 2.7s on a
healthy runner, which is why this is intermittent.
Cloudserver's own startup partly guards this already: metadata.setup() opens the
metadata client and throws if 9990 is unreachable, so the API cannot bind
without working metadata. This branch's Arsenal pin was checked and behaves the
same way, so only the data daemon needs a gate. Nothing guards that, because
clientCheck returns a hardcoded { code: 200, message: 'OK' } for clients that
implement no probe and DataFileInterface implements none, so the deep
healthcheck only really checks metadata and only for the mongo and bucketd
clients.
Gate on 9991 in the four jobs that use the file data backend: file-ft-tests,
kmip-ft-tests, kmip-cluster-ft-tests and sse-kms-migration-tests, the last of
which starts a fresh cloudserver container in a second phase and needs the gate
there too. 120s because 84s was observed; a healthy run pays nothing since the
wait returns as soon as the port answers.
Backport of #6277
Issue: CLDSRV-992
anurag4DSB
added a commit
that referenced
this pull request
Sep 8, 2026
`yarn start` launches the S3 API and the file daemons as separate parallel
processes (npm-run-all --parallel start_dmd start_s3server), so they race. On
run 33734792029 the API bound :8000 at t+10s and served writes from t+15s while
the dataserver only bound :9991 at t+84s, and the ten HTTP 500s in between are
all ECONNREFUSED 0.0.0.0:9991. The same subdir pre-creation step took 2.7s on a
healthy runner, which is why this is intermittent.
Cloudserver's own startup partly guards this already: metadata.setup() opens the
metadata client and throws if 9990 is unreachable, so the API cannot bind
without working metadata. This branch's Arsenal pin was checked and behaves the
same way, so only the data daemon needs a gate. Nothing guards that, because
clientCheck returns a hardcoded { code: 200, message: 'OK' } for clients that
implement no probe and DataFileInterface implements none, so the deep
healthcheck only really checks metadata and only for the mongo and bucketd
clients.
Gate on 9991 in the four jobs that use the file data backend: file-ft-tests,
kmip-ft-tests, kmip-cluster-ft-tests and sse-kms-migration-tests, the last of
which starts a fresh cloudserver container in a second phase and needs the gate
there too. 120s because 84s was observed; a healthy run pays nothing since the
wait returns as soon as the port answers.
Backport of #6277
Issue: CLDSRV-992
anurag4DSB
added a commit
that referenced
this pull request
Sep 8, 2026
`yarn start` launches the S3 API and the file daemons as separate parallel
processes (npm-run-all --parallel start_dmd start_s3server), so they race. On
run 33734792029 the API bound :8000 at t+10s and served writes from t+15s while
the dataserver only bound :9991 at t+84s, and the ten HTTP 500s in between are
all ECONNREFUSED 0.0.0.0:9991. The same subdir pre-creation step took 2.7s on a
healthy runner, which is why this is intermittent.
Cloudserver's own startup partly guards this already: metadata.setup() opens the
metadata client and throws if 9990 is unreachable, so the API cannot bind
without working metadata. This branch's Arsenal pin was checked and behaves the
same way, so only the data daemon needs a gate. Nothing guards that, because
clientCheck returns a hardcoded { code: 200, message: 'OK' } for clients that
implement no probe and DataFileInterface implements none, so the deep
healthcheck only really checks metadata and only for the mongo and bucketd
clients.
Gate on 9991 in the four jobs that use the file data backend: file-ft-tests,
kmip-ft-tests, kmip-cluster-ft-tests and sse-kms-migration-tests, the last of
which starts a fresh cloudserver container in a second phase and needs the gate
there too. 120s because 84s was observed; a healthy run pays nothing since the
wait returns as soon as the port answers.
Backport of #6277
Issue: CLDSRV-992
This was referenced Sep 8, 2026
leif-scality
approved these changes
Sep 8, 2026
tcarmet
approved these changes
Sep 8, 2026
Base automatically changed from
improvement/CLDSRV-992-server-access-log-match
to
development/9.3
September 10, 2026 05:25
Contributor
Request integration branchesWaiting for integration branch creation to be requested by the user. To request integration branches, please comment on this pull request with the following command: Alternatively, the |
Contributor
Author
|
/create_integration_branches |
Contributor
Author
|
/approve |
Contributor
Integration data createdI have created the integration data for the additional destination branches.
The following branches will NOT be impacted:
You can set option The following options are set: create_integration_branches |
Contributor
Waiting for approvalThe following approvals are needed before I can proceed with the merge:
The following options are set: create_integration_branches |
Contributor
Author
|
/approve |
Contributor
In the queueThe changeset has received all authorizations and has been added to the The changeset will be merged in:
The following branches will NOT be impacted:
This pull request does not target the following hotfix branch(es) so they
There is no action required on your side. You will be notified here once IMPORTANT Please do not attempt to modify this pull request.
If you need this pull request to be removed from the queue, please contact a The following options are set: approve, create_integration_branches |
Contributor
|
I have successfully merged the changeset of this pull request
The following branches have NOT changed:
Please check the status of the associated issue CLDSRV-992. Goodbye anurag4dsb. |
bert-e
deleted the
improvement/CLDSRV-992-wait-for-file-backend-ports
branch
September 10, 2026 06:03
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
yarn startlaunches the S3 API and the file daemons as separate parallel processes, so they race. On various CI runs, the API bound:8000at t+10s and served writes from t+15s while the dataserver only bound:9991at t+84s, and the ten HTTP 500s in between are allECONNREFUSED 0.0.0.0:9991; the same subdir pre-creation step took 2.7s on a healthy runner, which is why it is intermittent.Context on why cloudserver does not catch this itself:
metadata.setup()opens the metadata client and throws if9990is unreachable, so the API cannot bind without working metadata, which matches the run where CreateBucket and every metadata read succeeded and only the data path failed. Nothing guards the data daemon, becauseclientCheckreturns a hardcoded{ code: 200, message: 'OK' }for clients that implement no probe andDataFileInterfaceimplements none. So the deep healthcheck only really checks metadata, and only for the mongo and bucketd clients. We could add the file backend to the deep check and let the product gate itself, which would make this change unnecessary, but that is an Arsenal change.So the four jobs on the file data backend now wait on :9991 before :8000, including the second startup phase of sse-kms-migration-tests where a fresh cloudserver container is started. 120s because 84s was observed; a healthy run pays nothing since the wait returns as soon as the port answers.