feat(testing): overhaul test harness with docker compose and restructure kazoo.testing package

[ceache]

Important

Review Scope Notice:
This PR is stacked on top of PR #808 (fix/protocol-defer-connected-until-auth). Because PR #808 has not yet been merged into master, this PR includes its commit history.

Please only review the last 2 commits:

1. 6913472 test(core): split offline unit tests into kazoo/tests/unit
2. 8533a04 feat(testing): overhaul test harness with docker compose and restructure kazoo.testing package

Fixes #821

Why is this needed?

The legacy test harness (kazoo.testing.harness) relied on local JVM/ZooKeeper binary installations, custom environment setup scripts (ensure-zookeeper-env.sh), and hardcoded port configurations. This created multiple maintenance and developer experience challenges:

1. Environment friction & matrix testing: Running tests across different ZooKeeper versions (3.7 through 3.9) and authentication schemes (Plain, Digest, SASL Digest-MD5, SASL GSSAPI/Kerberos, mutual TLS) required manual daemon management and external scripts that were fragile across Linux, macOS, and Windows.
2. Slow feedback loops: Offline unit tests were mixed with integration tests, requiring a live cluster setup even for simple logic and parsing tests.
3. Flakiness and protocol edge cases: Long-standing race conditions and unhandled protocol scenarios (such as SASL disconnection retry handling and TLS SNI formatting in four-letter word commands) hindered reliable CI runs.

Proposed Changes

This PR introduces a modular, containerized test harness powered by Docker Compose and pytest fixtures, separates offline unit tests, and hardens client/protocol reliability:

• Docker Compose Test Harness (kazoo.testing):

  • Replaced legacy harness.py with modular packages: kazoo.testing.common (axis resolution, mount handling, ensemble state) and kazoo.testing.fixtures (pytest fixtures).
  • Composable Docker Compose overlays:
    • Base 3-node ZooKeeper cluster (zoo1, zoo2, zoo3).
    • Authentication overlays: Digest (superDigest), SASL Digest (DIGEST-MD5), SASL GSSAPI (with embedded MIT Kerberos KDC sidecar), and mutual TLS (with certgen and tls-secrets-agent sidecars).
    • Capture overlay: On-demand packet capture (tshark) generating .pcapng traces and TLS keylog material.
  • Native ephemeral host port publishing (2181 / 2281), eliminating static port math and port collisions.
  • Per-test chroot isolation (zkchroot) shared cleanly across zkclient and zksuperadmin_client.
  • Process environment isolation via docker_env snapshot and restoration on session teardown.
  • Standardized KAZOO_TESTING_ZK_* environment variables (with fallback to ZK_* in CLI option parsing).
  • Registered collection-time pytest markers (zk_version, zk_auth, zk_features) and CLI options (--zk-version, --zk-auth, --zk-features).

• Offline Unit Test Separation (kazoo/tests/unit):

  • Separated offline tests from integration tests; pytest kazoo/tests/unit runs 192 tests in ~12 seconds with zero network or Docker dependencies.
  • Added unit test suite kazoo/tests/unit/test_testing.py providing comprehensive coverage for harness logic and fixtures.

• Style & Python 3.12+ AST Compatibility:

  • Codebase reformatted with black==24.10.0 and commit registered in .git-blame-ignore-revs.

• Client & Protocol Bug Fixes:

  • fix(client): Pass peer hostname instead of port as TLS SNI server name in KazooClient.command().
  • fix(protocol): Propagate ConnectionDropped during SASL handshake to enable cluster failover retry instead of stopping connection attempts.
  • fix(protocol): Defer CONNECTED state notification until SASL and initial auth data dispatch complete, and handle SASLException cleanly.
  • test(client): Eliminate flakiness in request queuing and reconnect tests by awaiting explicit server state and synchronization events.
  • test(integ): Flush replication via client1.sync("/1") in test_connect_auth to prevent replication lag assertions.
  • test(unit): Use socket.socketpair() in test_select_mutated to eliminate macOS pipe deadlock.

Does this PR introduce any breaking change?

• The legacy kazoo.testing.harness.KazooTestCase test base class is replaced by standard pytest fixtures (zkensemble, zkclient, zksuperadmin_client, zkchroot).
• Harness configuration environment variables now use the KAZOO_TESTING_ZK_* prefix (legacy ZK_* variable names remain supported as fallbacks in CLI resolution).

[codecov]

Codecov Report

❌ Patch coverage is 95.27721% with 23 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.76%. Comparing base (f54c4fd) to head (8533a04).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
kazoo/testing/common.py	97.26%	9 Missing ⚠️
kazoo/testing/fixtures.py	93.69%	7 Missing ⚠️
kazoo/client.py	85.71%	5 Missing ⚠️
kazoo/protocol/connection.py	77.77%	2 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (f54c4fd) and HEAD (8533a04). Click for more details.

HEAD has 7 uploads less than BASE

Flag	BASE (f54c4fd)	HEAD (8533a04)
	28	21

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #802      +/-   ##
==========================================
- Coverage   94.90%   88.76%   -6.15%     
==========================================
  Files          27       30       +3     
  Lines        3811     4289     +478     
==========================================
+ Hits         3617     3807     +190     
- Misses        194      482     +288     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ceache]

I will extract the black, python38 and all the bugfixes in separate PRs to try to minimize the diff in this one. But it is inherently a large rewrite of our test harness