Skip to content

SOLR-18407: Update to newer library to fix duplicate promethesus metric emitting - #4907

Open
epugh wants to merge 8 commits into
apache:mainfrom
epugh:SOLR-18407
Open

epugh wants to merge 8 commits into
apache:mainfrom
epugh:SOLR-18407

Conversation

@epugh

@epugh epugh commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

https://issues.apache.org/jira/browse/SOLR-18407

Description

Doing what @mlbiscoc suggested!

Solution

Upgrade is dragging along some other upgrades and code updates, but does fix the bug!

Tests

existing and ai

@mlbiscoc

Copy link
Copy Markdown
Contributor

Sorry the way I wrote my comment on the last PR was confusing. What I mean to say lets fix up #4851 and merge that. Then we fix this PR after.

…ivate code .

PrometheusMetricReader's constructor became package-private in favor of a builder, so using FilterablePrometheusMetricReader.
@epugh

epugh commented Sep 11, 2026

Copy link
Copy Markdown
Contributor Author

Sorry the way I wrote my comment on the last PR was confusing. What I mean to say lets fix up #4851 and merge that. Then we fix this PR after.

Oh, that makes sense... I lost track of #4851....

@mlbiscoc mlbiscoc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we have a test proving that the duplicate jvm metrics are gone?

Comment on lines +80 to +88
// The runtime-telemetry-java8/java17 split was unified into this single module (see
// https://github.com/open-telemetry/opentelemetry-java-instrumentation/pull/16087). Used this
// way (as a library, not via the OTel javaagent), its extra metric sets are only reachable
// through this internal-but-public "Experimental" bridge, not system properties -- those are
// only read by the javaagent's own auto-configuration. The two calls below reproduce the old
// library's .enableAllFeatures() breadth (buffers, file descriptors, GC cause, network I/O,
// etc.) and activate JFR, which -- unlike the old java8/java17 split this module replaced --
// suppresses each metric's JMX series whenever its JFR series can serve as a full replacement,
// instead of emitting both.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need all this here. Keep the TODO as well.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks...

Comment thread changelog/unreleased/SOLR-18407.yml Outdated
@@ -0,0 +1,7 @@
title: Fix Solr emitting duplicate/conflicting JVM metrics from both runtime-telemetry-java8 and runtime-telemetry-java17 OTel scopes on Java 17, by migrating to the unified opentelemetry-runtime-telemetry module (upgraded to 2.31.1-alpha) that replaced the java8/java17 split.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are also upgrading OTEL versions here. Worth flagging.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@epugh

epugh commented Sep 15, 2026

Copy link
Copy Markdown
Contributor Author

Can we have a test proving that the duplicate jvm metrics are gone?

Done!

@epugh
epugh requested a review from mlbiscoc September 15, 2026 17:26
@epugh

epugh commented Sep 15, 2026

Copy link
Copy Markdown
Contributor Author

@mlbiscoc I think I've addressed your comments, would love a +1.

@epugh
epugh requested a review from sigram September 16, 2026 17:12
@epugh

epugh commented Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

I've got an itchy merge finger....

Comment on lines +147 to +149
// Guards against the java8/java17 RuntimeMetrics split emitting the same series twice (each
// scope reporting e.g. jvm.memory.committed with identical labels), which is what motivated
// migrating to the unified opentelemetry-runtime-telemetry module.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment looks unnecessary. We are just testing no duplicates. No need to reference past dependency.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

totally!

Comment on lines +158 to +167
for (MetricSnapshot snapshot : snapshots) {
Set<Labels> seen = new HashSet<>();
for (DataPointSnapshot dataPoint : snapshot.getDataPoints()) {
assertTrue(
"Duplicate series for metric "
+ snapshot.getMetadata().getPrometheusName()
+ " with labels "
+ dataPoint.getLabels(),
seen.add(dataPoint.getLabels()));
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this test is actually testing to catch the bug. I recommend doing TDD for this then make the change to confirm we actually fix it. This is a sample what the duplicate looks like:

jvm_memory_used_bytes{...,otel_scope_name="io.opentelemetry.runtime-telemetry-java8"}  
jvm_memory_used_bytes{...,otel_scope_name="io.opentelemetry.runtime-telemetry-java17"}

The reason is that otel_scope_name is part of the label set that makes it a "duplicate".

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will dig in

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants