Skip to content

Host reconnector that stops for good is never cleared, permanently losing the host #1026

Description

@dkropachev

Problem

_HostReconnectionHandler stops retrying after an AuthenticationFailed (cassandra/pool.py:368), and _ReconnectionHandler.run() stops rescheduling once the retry schedule is exhausted. In neither case does the handler remove itself from host._reconnection_handler, so Host.is_currently_reconnecting() (cassandra/pool.py:220) reports a reconnection in progress forever.

Cluster.on_down() consults exactly that to avoid starting a duplicate reconnector (cassandra/cluster.py:2048), and _start_reconnector() (cassandra/cluster.py:1989) -- which would cancel and replace a dead handler -- only runs once on_down() gets past the check. The stale handler therefore gates its own replacement.

A host whose reconnector hit an AuthenticationFailed, which rotated or expired credentials reach, is never reconnected again for the life of the Cluster, even after the credentials are fixed.

Expected behavior

A reconnection handler that will not run again releases the host's reconnector slot, so a later DOWN starts a fresh reconnection as it would for any other host.

ControlConnection's handler has the same shape; that half is fixed in #1024.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingrelease-blockerMust be resolved before the next release.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions