[Test] Measure benefits of StringIndex replacement for Lettuce

[amarziali]

What Does This Do

Evaluates StringIndex as replacement for HashSet on Lettuce helper to make command classification faster. Three structures compared:

• hashSet — what ships today
• stringIndexInstance — static final StringIndex via StringIndex.of(...)
• stringIndexEmbedded — StringIndex.EmbeddingSupport.indexOf over static final int[]/String[]

Two new measurement classes in lettuce-5.0: a JMH benchmark (speed + allocation) and a JOL test (footprint). Each benchmark arm does both lookups, matching one Redis command. _miss arms use ordinary commands (GET, SET, HGETALL, INCR), which are in neither set; _hit arms use the administrative commands the sets contain.

Results

Speed — ns per command, JDK 17.0.18 / Apple M4 Max, @Fork(3), @Threads(1):

Benchmark	Score ±(99.9%)	(min, avg, max)	stdev	CI (99.9%)
hashSet_perCommand_miss	1.715 ± 0.071	(1.679, 1.715, 1.788)	0.042	[1.643, 1.786]
stringIndexEmbedded_perCommand_miss	1.881 ± 0.295	(1.622, 1.881, 2.056)	0.176	[1.586, 2.177]
stringIndexInstance_perCommand_miss	2.687 ± 0.151	(2.609, 2.687, 2.817)	0.090	[2.536, 2.838]
hashSet_perCommand_hit	1.938 ± 0.158	(1.803, 1.938, 2.030)	0.094	[1.780, 2.096]
stringIndexEmbedded_perCommand_hit	1.890 ± 0.117	(1.808, 1.890, 1.989)	0.070	[1.772, 2.007]
stringIndexInstance_perCommand_hit	2.286 ± 0.100	(2.202, 2.286, 2.380)	0.060	[2.186, 2.386]

All values ns/op, avgt mode, Cnt 9 (3 forks × 3 iterations). Lower is better.

Miss-path means over six runs: hashSet 1.719, stringIndexEmbedded 1.737, stringIndexInstance 2.685.

Footprint — retained bytes, both sets measured as one graph: hashSet 1056, stringIndexInstance 736, stringIndexEmbedded 688.

Allocation — gc.alloc.rate.norm at the 10⁻⁶ B/op floor for every arm.

Conclusions

1. EmbeddingSupport is a tie, not a win. Means within 1% over six runs, marginally slower on balance, error bars overlapping every time. Not distinguishable from zero.
2. The instance form is slower on the common path (+0.97 ns/command, ~56%), consistent across six runs. It is the form the proposal specified.
3. Footprint saving is ~368 bytes, paid once per classloader, not per command.
4. Recommendation: don't make the change. The instance form is a regression, and the embedded form shows no measurable saving on the common path.

Motivation

Additional Notes

Contributor Checklist

• Format the title according to the contribution guidelines
• Assign the type: and (comp: or inst:) labels in addition to any other useful labels
• Avoid using close, fix, or any linking keywords when referencing an issue
  Use solves instead, and assign the PR milestone to the issue
• Update the CODEOWNERS file on source file addition, migration, or deletion
• Update public documentation with any new configuration flags or behaviors
• Once approved, use merge queue to merge the PR

Jira ticket: [PROJ-IDENT]

[datadog-prod-us1-5]

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 69.74% (+10.96%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 0c574de | Docs | View more details | Give us feedback!}

[dd-octo-sts]

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.03 s	13.87 s	[+0.3%; +2.0%] (maybe worse)
startup:insecure-bank:tracing:Agent	12.94 s	12.98 s	[-1.2%; +0.6%] (no difference)
startup:petclinic:appsec:Agent	17.40 s	17.15 s	[+0.5%; +2.4%] (maybe worse)
startup:petclinic:iast:Agent	16.87 s	17.43 s	[-7.6%; +1.2%] (no difference)
startup:petclinic:profiling:Agent	17.18 s	16.85 s	[-2.8%; +6.8%] (no difference)
startup:petclinic:sca:Agent	16.76 s	17.18 s	[-6.9%; +2.0%] (no difference)
startup:petclinic:tracing:Agent	16.46 s	16.29 s	[-3.2%; +5.3%] (no difference)

Commit: 0c574def · CI Pipeline · Benchmarking Platform UI

---

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

[dougqh]

Typically, we should use a larger number of threads. That will simulate a server workload more realistic and show the throughput impact of allocation.

Admittedly, allocation isn't a big concern here, but it is still a good habit for dd-trace-java benchmarks.

[dougqh]

To realistically benchmark HashSet / HashMap, we need to first pre-populate the type profile of HashSet. Right now, HashSet will use monomorphic caller specialization in way that isn't realistic for a production load.

I thought I'd done that in the ImmutableSetBenchmark, but apparently not. I think we should actually just update the ImmutableSetBenchmark, since it is doing the same thing as the benchmark but for more set-like set-ups.

[dougqh]

I think it is okay, but typically, we want to be careful about using string literals for lookups in a benchmark. If we aren't careful, the JIT can perform a branch speculation on a reference equals check which skips compiling the equals check entirely.

In this case, I suspect that using string-literals / string-constants is the norm, so it is fine, but just something to be aware of.

[amarziali]

closing as superseeded by #12288