fix(elasticsearch): resolve Cloud ID to the real ES host and stop the timeout param collision - #7260
fix(elasticsearch): resolve Cloud ID to the real ES host and stop the timeout param collision#7260waleedlatif1 wants to merge 7 commits into
Conversation
… timeout param collision
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
Greptile SummaryThe PR centralizes Elasticsearch connection helpers, corrects Elastic Cloud endpoint resolution, separates the cluster wait timeout from the HTTP deadline, and hardens redirect handling while preserving existing
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains; both previously reported output-compatibility issues are addressed by preserving raw top-level index keys and allowing a real
|
| Filename | Overview |
|---|---|
| apps/sim/tools/elasticsearch/utils.ts | Centralizes authentication and URL construction with validated Cloud ID parsing. |
| apps/sim/tools/elasticsearch/get_index.ts | Adds the declared aggregate output while preserving legacy top-level index references and the prior collision fix. |
| apps/sim/tools/elasticsearch/cluster_health.ts | Separates Elasticsearch’s server-side wait timeout from the transport’s HTTP deadline. |
| apps/sim/blocks/blocks/elasticsearch.ts | Maps saved timeout state to the renamed runtime parameter without forwarding the transport-reserved key. |
| apps/sim/tools/elasticsearch/types.ts | Aligns response and parameter types with the corrected runtime contracts. |
Reviews (7): Last reviewed commit: "fix(elasticsearch): strip credentials on..." | Re-trigger Greptile
There was a problem hiding this comment.
5 issues found across 21 files
Confidence score: 1/5
apps/sim/tools/elasticsearch/create_index.tscan turn a decoded Cloud ID containing\into an attacker-controlled URL origin, risking authenticated requests being sent to the wrong destination — validate or reject the decoded component before constructing the URL.apps/sim/tools/elasticsearch/bulk.tsnow applies JSON handling to raw NDJSON, so Elasticsearch receives malformed bulk payloads and bulk operations fail — preserveapplication/x-ndjsonor bypass JSON stringification.apps/sim/tools/elasticsearch/utils.tsmay send cloud credentials to a supplied self-hostedhostwhencloudIdis absent, creating a credential-routing risk — branch ondeploymentTypefirst and require a non-empty cloud ID for cloud deployments.apps/sim/tools/elasticsearch/get_index.tscan break existing{{getIndex.products.mappings}}references, whileapps/sim/blocks/blocks/elasticsearch.tsrisks changing reservedtimeoutstate semantics; preserve top-level index paths and migrate the legacy timeout input separately.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="apps/sim/tools/elasticsearch/create_index.ts">
<violation number="1" location="apps/sim/tools/elasticsearch/create_index.ts:5">
P1: When a decoded Cloud ID component contains `\`, this import makes create-index use `parseCloudId`, which accepts it. `new URL` normalizes the resulting URL to an attacker-controlled origin, so the authenticated request can send `Authorization` there. Reject backslashes in the shared parser before returning the URL.</violation>
</file>
<file name="apps/sim/tools/elasticsearch/utils.ts">
<violation number="1" location="apps/sim/tools/elasticsearch/utils.ts:75">
P1: When a cloud invocation omits `cloudId` but supplies `host`, `buildBaseUrl` silently uses the self-hosted host and sends the configured cloud credentials there. Branch on `deploymentType` first and require a non-empty Cloud ID for cloud deployments.</violation>
</file>
<file name="apps/sim/blocks/blocks/elasticsearch.ts">
<violation number="1" location="apps/sim/blocks/blocks/elasticsearch.ts:570">
P2: The block still publishes the server-side cluster-health wait under reserved name `timeout`. Rename the public input to `clusterTimeout` and migrate legacy `timeout` state separately so future execution paths cannot reintroduce the HTTP deadline collision.
(Based on your team's feedback about reserving `timeout` for transport deadlines.)</violation>
</file>
<file name="apps/sim/tools/elasticsearch/bulk.ts">
<violation number="1" location="apps/sim/tools/elasticsearch/bulk.ts:5">
P1: Bulk requests now use the shared JSON content type, so `request-transport` JSON-stringifies the raw NDJSON body and Elasticsearch receives a malformed bulk payload. Preserve `application/x-ndjson` for this tool, or make the shared header helper support the bulk content type.</violation>
</file>
<file name="apps/sim/tools/elasticsearch/get_index.ts">
<violation number="1" location="apps/sim/tools/elasticsearch/get_index.ts:94">
P1: Preserve or migrate the existing top-level index references before nesting the response under `indices`; otherwise saved paths such as `{{getIndex.products.mappings}}` stop resolving and downstream steps fail.</violation>
</file>
Tip: instead of fixing issues one by one fix them all with cubic
Re-trigger cubic
…ranch, keep get_index top-level keys
|
Pushed 61cf417 addressing all six threads. Summary of what changed and what I pushed back on: Fixed (4 P1s):
Pushed back (1 P2): renaming the 53 tests pass in |
|
Pushed 90b2dbc closing the 54 tests pass in |
The previous commit added an `ElasticsearchIndexInfo` interface for the
`GET /{index}` state shape (aliases/mappings/settings), but that name was
already taken further up the same file by the `_cat/indices` row shape
(index, health, status, docsCount, storeSize, primaryShards,
replicaShards). Two interface declarations with the same name in one
module scope do not shadow — TypeScript declaration-merges them, so the
single resulting interface required all seven cat columns *and* carried
the three optional index-state fields. It described neither endpoint.
It compiles today only because `transformResponse` returns `any` from
`response.json()`, so nothing in the integration ever assigns against the
type. The defect is latent, not live — but it is load-bearing in both
directions, which a probe confirms:
- a valid `GET /{index}` entry is rejected by
`ElasticsearchIndexInfoResponse['output']`:
"TS2740: Type '{ mappings; settings; aliases }' is missing the
following properties from type 'ElasticsearchIndexInfo': index,
health, status, docsCount, and 3 more."
- a `list_indices` row with a nonexistent `mappings` key is accepted.
Rename the new interface to `ElasticsearchIndexState`. That is the name
Elastic's own generated specification gives this object
(`indices._types.IndexState` in elasticsearch-specification), so it is
the endpoint's real name rather than one invented to dodge the clash.
The `_cat/indices` row keeps `ElasticsearchIndexInfo`, matching the
`ElasticsearchListIndicesResponse` that consumes it.
Type-only change: not exported, no runtime behavior, and no generated
artifact moves (tool-metadata:check, docs:check, integration-catalog:check
all still pass untouched).
|
@greptile review |
|
@cubic review |
@waleedlatif1 I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
2 issues found across 21 files
Confidence score: 2/5
apps/sim/tools/elasticsearch/utils.tsbuildBaseUrlcan route an unrecognizeddeploymentTypeto the stale self-hosted host while sending cloud credentials, creating a concrete credential-disclosure risk; reject unsupported values before selecting the host or shared auth headers.apps/sim/tools/elasticsearch/get_index.tscan let a matched index namedindicesoverwrite the aggregateoutput.indicesmap, producing incorrect matched-index output; spread raw keys first and assignindicesafterward.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="apps/sim/tools/elasticsearch/utils.ts">
<violation number="1" location="apps/sim/tools/elasticsearch/utils.ts:82">
P1: When `deploymentType` is an unrecognized runtime value, `buildBaseUrl` falls through to the stale self-hosted `host` and the shared auth headers send the cloud credential there. Reject values other than `cloud` and `self_hosted` instead of treating every non-cloud value as self-hosted.</violation>
</file>
<file name="apps/sim/tools/elasticsearch/get_index.ts">
<violation number="1" location="apps/sim/tools/elasticsearch/get_index.ts:106">
P2: When a matched index is named `indices`, `...data` overwrites the declared aggregate, so `output.indices` becomes that index's state instead of the map of matched indices. Spread raw keys first and assign `indices: data` last, or add a separate compatibility alias, so the declared output contract always holds.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Fix all with cubic | Re-trigger cubic
…assuming self-hosted `buildBaseUrl` guarded only the exact string `'cloud'`, and treated every other value as self-hosted. That fallthrough is the same credential disclosure the cloud branch exists to prevent, reached by a different route. `deploymentType` is declared `required: true` with no explicit `visibility`, and `tools/params.ts` resolves a required param with no visibility to `user-or-llm`. On the agent tool-calling path a model therefore supplies it, while `host`, `cloudId`, `apiKey`, `username` and `password` are all `user-only`. A near miss — `Cloud`, `CLOUD`, `elastic_cloud`, a trailing space — is not `=== 'cloud'`, so it selected the self-hosted branch and sent the user's API key to whatever `host` still held from an earlier self-hosted configuration. Both `host` and `cloudId` are in the block's `inputs` and are sent regardless of which subBlock the dropdown condition is currently showing, so the stale host is genuinely present. Reject any non-nullish value that is neither `self_hosted` nor `cloud`. Nullish continues to mean self-hosted: that is the dropdown's own default (`value: () => 'self_hosted'`) and the shape of workflow state saved before the field was touched, so no existing workflow changes behavior. Reported by cubic on #7260. Five parameterised regression tests cover the near-miss spellings and one covers the nullish legacy path; all five fail against the previous code (verified by reverting the guard and re-running).
|
@cubic review |
|
@greptile review |
@waleedlatif1 I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
3 issues found across 21 files
Confidence score: 2/5
apps/sim/tools/elasticsearch/utils.tsnow sends credentials on requests without stripping them across cross-origin redirects, so an Elasticsearch redirect could disclose the API key or Basic credentials; add an explicit credential-stripping or redirect policy before merging.apps/sim/tools/elasticsearch/utils.tstreats an emptydeploymentTypeas self-hosted while cloud credentials may still be supplied, which can route requests to the stale host and fail or misdirect access; reject every non-nullish value other thanself_hosted.apps/sim/tools/elasticsearch/types.tscannot accurately represent an index namedindices, leavinggetIndexTooloutput inconsistent with the declared type; allowoutput.indicesto be either a single index state or the aggregate map.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="apps/sim/tools/elasticsearch/types.ts">
<violation number="1" location="apps/sim/tools/elasticsearch/types.ts:197">
P2: When an Elasticsearch index is named `indices`, `getIndexTool` returns that index's state at `output.indices`, but this type requires an aggregate map. Type `indices` as `ElasticsearchIndexState | Record<string, ElasticsearchIndexState>` so consumers do not rely on an invalid shape.</violation>
</file>
<file name="apps/sim/tools/elasticsearch/utils.ts">
<violation number="1" location="apps/sim/tools/elasticsearch/utils.ts:99">
P1: When `deploymentType` is the empty string, this guard routes the request to the stale self-hosted host while the caller can still supply cloud credentials. Reject every non-nullish value other than `self_hosted` so only the documented legacy nullish fallback can select `host`.</violation>
<violation number="2" location="apps/sim/tools/elasticsearch/utils.ts:129">
P1: Every Elasticsearch request now sends an `Authorization` header, but cross-origin redirects have no credential-stripping policy. A redirect from an Elasticsearch endpoint can therefore disclose the API key or Basic credentials; configure these requests to strip credentials and `host` on cross-origin redirects, or make that the transport default.
(Based on your team's feedback about stripping credentials on cross-origin redirects by default.)</violation>
</file>
Reply with feedback, questions, or to request a fix.
Fix all with cubic | Re-trigger cubic
…ploymentType, and type the indices union honestly Three findings from cubic on b3a4ce8. **Credentials survived a cross-origin redirect (13 tools).** Every tool sends `Authorization`, but the shared transport only strips it on a cross-origin hop when the tool opts in: `prepareToolRequest` leaves `redirectPolicy` undefined unless `request.redirectPolicy` is declared, and the stripping branch in `input-validation.server.ts` is gated on that policy being present. With neither a policy nor `stripAuthOnRedirect`, a redirect off the configured origin carried the API key or Basic credentials with it. `host` is a user-supplied origin, which is exactly the profile of the integrations already opted in — obsidian, mintlify, and s3 all set this for the same reason. All 13 tools now declare `stripAuthOnRedirect: true`. This is pre-existing rather than introduced here, but it is the same threat class the rest of this PR closes, and stopping at the Cloud ID path while leaving the redirect path open would be an odd place to draw the line. **An empty `deploymentType` still selected the stale host.** The previous guard tested truthiness, so `''` fell through to self-hosted. That does not match the documented intent, which was to admit only a *nullish* legacy value. It also matters on its own: a caller supplying `cloudId` while blanking this field is expressing cloud intent, and routing that to `host` is the same disclosure the guard exists to prevent. Now `!= null`, so only nullish selects the legacy fallback. **`ElasticsearchIndexInfoResponse.indices` promised a shape that does not always hold.** The spread order is deliberate and unchanged — an index legitimately named `indices` keeps its own raw key so pre-existing saved references resolve — but the type declared the aggregate map unconditionally, so it was wrong in exactly that case. Widened to `ElasticsearchIndexState | Record<string, ElasticsearchIndexState>`, which states the trade-off rather than papering over it. Same principle as the merged-interface fix earlier in this branch: the type must describe what the code actually returns. 15 tests added. Verified they can fail: reverting the `!= null` guard reds the empty-string case, and dropping `stripAuthOnRedirect` from one tool reds that tool's coverage case.
|
@cubic review |
|
@greptile review |
@waleedlatif1 I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
3 issues found across 21 files
Confidence score: 3/5
apps/sim/tools/elasticsearch/get_document.tsandapps/sim/tools/elasticsearch/count.tsstripAuthorizationon same-origin redirects, which can turn valid Elasticsearch requests into 401s and break document retrieval or count operations — preserve credentials for trusted same-origin redirects while still stripping them for untrusted targets.apps/sim/tools/elasticsearch/types.tskeepsElasticsearchIndexStateunavailable to consumers even thoughElasticsearchIndexInfoResponsereferences it, blocking downstream TypeScript adoption of the documented response shape — export the interface.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="apps/sim/tools/elasticsearch/types.ts">
<violation number="1" location="apps/sim/tools/elasticsearch/types.ts:189">
P3: Consumers cannot import the new `ElasticsearchIndexState` type referenced by `ElasticsearchIndexInfoResponse`. Export the interface so downstream TypeScript code can adopt the documented response shape.</violation>
</file>
<file name="apps/sim/tools/elasticsearch/get_document.ts">
<violation number="1" location="apps/sim/tools/elasticsearch/get_document.ts:107">
P2: When Elasticsearch returns a same-origin 3xx, `stripAuthOnRedirect` removes `Authorization` before replaying the request, so a valid authenticated request can become a 401. Configure the redirect policy to strip credentials only on cross-origin hops.
(Based on your team's feedback about stripping credentials only on cross-origin redirects.)</violation>
</file>
<file name="apps/sim/tools/elasticsearch/count.ts">
<violation number="1" location="apps/sim/tools/elasticsearch/count.ts:82">
P2: When Elasticsearch or its reverse proxy returns a same-origin redirect, this flag removes the Authorization header before replaying the request, so the canonical endpoint can return 401 and the count operation fails. Use a redirect policy that disables credentials only for cross-origin redirects while preserving the legacy method behavior.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Fix all with cubic | Re-trigger cubic
| } | ||
|
|
||
| /** One entry of a `GET /{index}` response, keyed by index name. */ | ||
| interface ElasticsearchIndexState { |
There was a problem hiding this comment.
P3: Consumers cannot import the new ElasticsearchIndexState type referenced by ElasticsearchIndexInfoResponse. Export the interface so downstream TypeScript code can adopt the documented response shape.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At apps/sim/tools/elasticsearch/types.ts, line 189:
<comment>Consumers cannot import the new `ElasticsearchIndexState` type referenced by `ElasticsearchIndexInfoResponse`. Export the interface so downstream TypeScript code can adopt the documented response shape.</comment>
<file context>
@@ -185,15 +185,24 @@ export interface ElasticsearchIndexResponse extends ToolResponse {
}
+/** One entry of a `GET /{index}` response, keyed by index name. */
+interface ElasticsearchIndexState {
+ aliases?: Record<string, unknown>
+ mappings?: Record<string, unknown>
</file context>
| interface ElasticsearchIndexState { | |
| export interface ElasticsearchIndexState { |
There was a problem hiding this comment.
Leaving this one as-is — it would break the file's own convention rather than follow it.
types.ts deliberately keeps response-detail interfaces module-private and exports only the ToolResponse wrappers. ElasticsearchIndexState is not an exception; it is the rule. The clearest counter-example is its immediate neighbour:
interface ElasticsearchIndexInfo { index: string; health: string; /* … */ } // not exported
export interface ElasticsearchListIndicesResponse extends ToolResponse {
output: { message: string; indices: ElasticsearchIndexInfo[] } // referenced from an exported type
}That is the same shape as the finding — a private interface referenced by an exported response type — and it predates this PR. The same is true of ElasticsearchIndexExistsResponse, ElasticsearchMappingResponse, ElasticsearchRefreshResponse and ElasticsearchIndexStatsResponse. Exporting only the one interface this PR happens to touch would leave the file half-converted and make the next reader wonder why that one is special.
On the practical side, there is no consumer to unblock. getIndexTool is typed ToolConfig<ElasticsearchGetIndexParams, ElasticsearchIndexInfoResponse>, the executor handles tool outputs generically, and nothing in the repo imports these detail interfaces — I checked (grep -rn 'ElasticsearchIndexInfo' tools blocks lib app returns only types.ts and get_index.ts's import of the response type). TypeScript also still lets a consumer reach the shape structurally via ElasticsearchIndexInfoResponse['output']['indices'] without the name being exported.
If we do want these public, that is a worthwhile consistency pass across all five private interfaces in the file — but as its own change, not smuggled into a security fix where a reviewer is looking at redirect behavior. Happy to file it if you'd like.
… not every hop
cubic is right that `stripAuthOnRedirect: true` was the wrong primitive.
It drops `Authorization` unconditionally — the branch in
`input-validation.server.ts` has no `isCrossOrigin` guard, and
`pinned-redirect-replay.server.test.ts` has a test named "honours
stripAuthOnRedirect on a same-origin hop" asserting exactly that. So a
reverse proxy in front of Elasticsearch performing a legitimate
same-origin redirect would have had its credential dropped and returned
401.
The precedent I cited for that flag does not transfer. obsidian, mintlify,
s3 and dataverse redirect to signed storage URLs, where the credential
must never follow on *any* hop. Elasticsearch's requirement is narrower:
never leak to a different origin, always keep it on the same one.
All 13 tools now declare
redirectPolicy: () => ({ mode: 'legacy', sendCredentialsOnCrossOriginRedirect: false })
which strips `authorization`, `proxy-authorization`, `cookie` and `host`
via `CROSS_ORIGIN_CREDENTIAL_HEADERS`, but only when the hop actually
crosses origin. `prepareToolRequest` additionally folds provenance
sensitive headers into the strip set when the flag is false, which the
previous approach did not get.
`mode: 'legacy'` is deliberate: it preserves the existing method and body
replay semantics, so the only behavior change is the cross-origin strip.
Under `'standard'`, `resolveRedirectHop` applies Fetch method rules and a
301/302 would rewrite POST to GET, breaking `_search`, `_count` and
`_bulk`. `tools/github/utils.server.ts` chooses `'legacy'` for the same
reason.
26 assertions across the 13 tools: each declares the policy, and each
leaves `stripAuthOnRedirect` unset. Verified they fail — restoring the
flag on one tool reds both of that tool's cases.
|
@cubic review |
|
@greptile review |
@waleedlatif1 I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
1 issue found across 21 files
Confidence score: 4/5
- In
apps/sim/tools/elasticsearch/utils.ts,extractPortFromNamecan accept a Cloud ID with multiple colons and produce a malformed hostname, potentially resolving to an unintended host/port; reject components containing extra colons before parsing.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="apps/sim/tools/elasticsearch/utils.ts">
<violation number="1" location="apps/sim/tools/elasticsearch/utils.ts:57">
P2: When a Cloud ID component contains more than one colon, `extractPortFromName` leaves a colon in the hostname and this check does not reject it, allowing a malformed Cloud ID to resolve to an unintended host/port. Reject any colon remaining in the extracted component name before constructing the URL.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Fix all with cubic | Re-trigger cubic
| } | ||
|
|
||
| for (const component of [parentDomain.name, elasticsearch.name]) { | ||
| if (CLOUD_ID_REJECTED_CHARACTERS.test(component)) { |
There was a problem hiding this comment.
P2: When a Cloud ID component contains more than one colon, extractPortFromName leaves a colon in the hostname and this check does not reject it, allowing a malformed Cloud ID to resolve to an unintended host/port. Reject any colon remaining in the extracted component name before constructing the URL.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At apps/sim/tools/elasticsearch/utils.ts, line 57:
<comment>When a Cloud ID component contains more than one colon, `extractPortFromName` leaves a colon in the hostname and this check does not reject it, allowing a malformed Cloud ID to resolve to an unintended host/port. Reject any colon remaining in the extracted component name before constructing the URL.</comment>
<file context>
@@ -0,0 +1,142 @@
+ }
+
+ for (const component of [parentDomain.name, elasticsearch.name]) {
+ if (CLOUD_ID_REJECTED_CHARACTERS.test(component)) {
+ throw new Error('Invalid Cloud ID format')
+ }
</file context>
| if (CLOUD_ID_REJECTED_CHARACTERS.test(component)) { | |
| if (component.includes(':') || CLOUD_ID_REJECTED_CHARACTERS.test(component)) { |
There was a problem hiding this comment.
Declining this one — I ran it rather than reasoning about it, and no input reaches an unintended host. Every extra-colon case either resolves to exactly what the Cloud ID encodes, or fails closed.
Driving parseCloudId directly:
single colon in parent (normal port) -> https://uuid.found.io:9243 | host=uuid.found.io port=9243
EXTRA colon in parent -> https://uuid.found.io:9243 | host=uuid.found.io port=9243
EXTRA colon in es component -> https://uuid:80.found.io | URL-INVALID
colon-smuggled second host in es -> https://uuid.found.io | host=uuid.found.io port=(443)
attempt userinfo via colon+at -> THROWS: Invalid Cloud ID format
Three things follow:
- A leftover colon cannot redirect credentials. Reaching a different origin from the authority requires
@(userinfo), and@is already in the reject set — the last case throws. A:can only set a port on the same host or corrupt the authority. - A corrupted authority fails closed, it does not silently connect.
https://uuid:80.found.iois not a parseable URL: everything after the colon must be digits, and80.found.iois not, sonew URLrejects it andassertExternalRequestUrlinprepareToolRequestthrows before any request goes out. There is no "resolves to an unintended host" path here. - The one case that does produce a valid URL is the correct answer. For a parent component of
found.io:9243:443,https://uuid.found.io:9243is the host and port that Cloud ID actually encodes. This is also exactly what the reference implementation does — Beats'extractPortFromNameinlibbeat/cloudid/cloudid.gouses the samestrings.LastIndexright-partition and likewise leaves an inner colon in the name. Rejecting it would make us stricter than the algorithm we are implementing, for no security gain.
Worth noting the threat model too: cloudId is user-only, so the user pastes their own. Even granting a hostile value, a colon grants nothing beyond what the parent-domain component already permits by design — a Cloud ID names its own host.
The one real (cosmetic) difference is the error a user sees: case 2 surfaces a URL parse error from the transport rather than Invalid Cloud ID format from the parser. If you want that tidied I am happy to file it, but it is a message-quality change, not the host/port vulnerability described, so I would rather not land it in a security fix at 5/5 on the other reviewer. Leaving this open for you to weigh in rather than resolving my own disagreement.
|
Closing for now — not because of a defect. This batch grew to 17 PRs across ~700 changed call sites, and we would rather revisit it as smaller, independently testable pieces than merge this much at once. Nothing here is lost: the branch |
Summary
Three verified defects in the Elasticsearch integration, plus the consolidation of 13 duplicated copies of the connection helpers into one
apps/sim/tools/elasticsearch/utils.ts.1. Cloud ID resolved to a host that does not exist (all 13 tools)
An Elastic Cloud ID is
<deployment label>:<base64 of "parentDomain$esUuid$kibanaUuid">. The reachable Elasticsearch endpoint ishttps://<esUuid>.<parentDomain>. Every tool carried its own copy ofbuildBaseUrlwhich did:parts[0]is the human-readable deployment label, not the Elasticsearch UUID, so every cloud request went to a name that resolves to nothing. Reproduction:The replacement (
parseCloudIdinutils.ts) follows the reference implementation in Beats'libbeat/cloudid/cloudid.godecodeCloudID():$-separated components;extractPortFromName), inheriting the parent domain's port and defaulting to 443;#,@,?, or/— Beats'strings.IndexAny(component, "#@?/")reject set. This is the security-relevant part: an@in the UUID component turns everything before it into URL userinfo, sohttps://<uuid>@evil.example.comwould have sent theAuthorization: ApiKey ...header to an attacker-controlled origin.One addition beyond the Beats algorithm: the extracted port must be all digits. Beats validates the name but not the port, so
uuid:80@evil.example.comsurvives its reject-set check (the@ends up in the port half) and still yields an attacker-controlled authority. That is rejected here.buildAuthHeaderswas byte-identical in all 13 files and is now shared too. Net: −570/+49 lines.2.
elasticsearch_get_indexdeclared a phantomindexoutput — partly rejectedThe declared output was
index, butGET /<index>returns an object keyed by index name ({"logs-2024": {aliases, mappings, settings}}) — there is noindexkey at any level, so the entire payload was unreferenceable from downstream blocks. Fixed by returning{ indices: <the keyed map> }and declaringindices.Rejected sub-claim: the report also said the tool "silently keeps only one index when the request used a wildcard". That is not what the code does —
transformResponsereturnedoutput: dataverbatim, so every matched index was present; it was just unreachable because no declared output named it. A regression test now asserts a two-index wildcard response keeps both keys.3.
elasticsearch_cluster_healthdeclared a param literally namedtimeoutapps/sim/tools/request-transport.tsreadsparams.timeoutas the outbound HTTP deadline in milliseconds (Math.min(Number(rawTimeout), getMaxExecutionTimeout())). Measured against the old code withtimeout: '30':So a cluster-health wait was also arming a client-side abort. The tool param is renamed
clusterTimeoutand mapped intools.config.params(nottools.config.tool, which runs at serialization before variable resolution).Two details this required:
timeoutis unchanged, so saved workflow state is not orphaned (check-block-registry.ts origin/stagingpasses the subblock-ID stability check).{ ...inputs, ...transformedParams }, so the rawtimeoutinput would still have reached the transport.tools.config.paramstherefore explicitly clears it ({ timeout: undefined }) alongside emittingclusterTimeout.While there, the same mapper had a unit bug: it appended
sto anything not already ending ins, so a user entering1mgot1ms— a 1-millisecond server-side wait. It now only appendssto a bare integer.4. Phantom outputs across the other 12 tools — rejected, none found
Every declared output on the other twelve tools was checked against the documented Elasticsearch response bodies (
_search,_count,_bulk,_docindex/get/update/delete,PUT/DELETE /<index>,_cluster/health,_cluster/stats,_cat/indices?format=json). All declared fields are real response fields. The only phantom in the integration wasget_index.index, covered above. No changes made.Tests
New:
tools/elasticsearch/utils.test.ts(28 tests) andtools/elasticsearch/cluster_health.test.ts(7 tests). 47 tests pass intools/elasticsearch.Coverage includes: label-vs-UUID host resolution, colon-in-label, per-service and inherited ports, the
#@?/reject set, the non-numeric-port smuggle,<3components, empty ES component, self-hosted trailing-slash and missing-host, a parameterised sweep asserting all 13 tools resolve the same cloud host,prepareToolRequestleaving no HTTP deadline while still emittingtimeout=30son the wire,1mnot becoming1ms, get_index declared outputs matching the transform's actual keys, and wildcard multi-index preservation.Each test was verified to fail against the pre-fix code: reverting
parseCloudIdto the old algorithm turned 25 of 28 utils tests red, and reverting the get_index/cluster_health/block changes turned all 6 behavioural tests red. Both were then restored and re-run green.Gates
bun run lint,bun run check:audits(39 audits, all green),bun run apps/sim/scripts/check-block-registry.ts origin/staging, andbun run type-check(no Elasticsearch diagnostics) all pass.tool-metadata:generateandgenerate-docsartifacts are regenerated and committed.5. Two same-named interfaces in
types.tswere declaration-mergingFollow-up from a validation pass over the branch. Commit 3 added an
ElasticsearchIndexInfointerface for theGET /{index}state shape (aliases/mappings/settings), but that name was already taken further up the same file by the_cat/indicesrow shape (index,health,status,docsCount,storeSize,primaryShards,replicaShards). Two interface declarations with the same name in one module scope do not shadow — TypeScript declaration-merges them. The single resulting interface required all seven cat columns and carried the three optional index-state fields, so it described neither endpoint.It compiles today only because
transformResponsereturnsanyfromresponse.json(), so nothing in the integration ever assigns against the type. The defect is latent rather than live, but it is load-bearing in both directions, confirmed with atscprobe:GET /{index}entry is rejected byElasticsearchIndexInfoResponse['output']—TS2740: Type '{ mappings; settings; aliases }' is missing the following properties from type 'ElasticsearchIndexInfo': index, health, status, docsCount, and 3 more.list_indicesrow carrying a nonexistentmappingskey is accepted.The new interface is renamed
ElasticsearchIndexState— the name Elastic's own generated specification gives this object (indices._types.IndexState), so it is the endpoint's real name rather than one invented to dodge the clash. The_cat/indicesrow keepsElasticsearchIndexInfo, matching theElasticsearchListIndicesResponsethat consumes it.Type-only: not exported, no runtime behavior, no generated artifact moves.
Note for future readers: the cloud branch's missing fallback is a security property
buildBaseUrldeliberately throws whendeploymentType === 'cloud'andcloudIdis empty, instead of falling back tohost. That reads like defensive tidiness and is an obvious candidate for "simplification". It is not — it is load-bearing, for two independent reasons:hostin saved workflow state. A fallback would send the cloud credential to a stale, unrelated origin.deploymentTypeis model-settable. It is declaredrequired: truewith no explicitvisibility, andapps/sim/tools/params.tsdefaults a required param touser-or-llm. On the agent tool-calling path an LLM can therefore setdeploymentType: 'cloud'whilecloudId— which isuser-only— stays empty.host,apiKey,username, andpasswordare alluser-onlytoo, so the only reachable configuration is "cloud selected, no cloud ID, but a self-hosted host and credentials still present". Throwing is what stops that from silently shipping the user's Elasticsearch credentials to whatever originhosthappens to hold.Please do not replace the throw with
params.cloudId ?? params.host.Verification method for the output audit (section 4)
The per-tool output audit was re-run against
elastic/elasticsearch-specificationoutput/schema/schema.json— the generated OpenAPI spec that backs the published docs pages — rather than against the doc pages themselves, which truncate their response-field tables. Every declared output on all 13 tools resolves to a named property in that spec:cluster.stats.StatsResponseBase(confirmingstatus,nodes.count.{total,data,master},nodes.versions),cat.indices.IndicesRecord(confirming the seven cat columns are all string-typed in JSON format, which is why theparseIntis correct),indices.create.Response(index/acknowledged/shards_acknowledgedall required),_types.WriteResponseBase,_global.count.Response,_global.bulk.Response, andindices._types.IndexState. No declared field rests on inference.6. Credentials survived a cross-origin redirect (all 13 tools)
Every tool sends
Authorization, but nothing was stripping it on a redirect off the configured origin. Verified against the transport rather than assumed:prepareToolRequestonly populatesredirectPolicyfromtool.request.redirectPolicy, which Elasticsearch never declared, and the cross-origin stripping branch inlib/core/security/input-validation.server.tsis gated on that policy existing:With neither declared, a redirect carried the API key or Basic credentials to the redirect target. Opt-in is deliberate rather than an oversight —
tools/index.test.tsasserts the redirect fields stay unset for tools that do not opt in — so the correct fix is to opt in, not to change the framework default.All 13 tools now declare:
This strips
authorization,proxy-authorization,cookieandhostviaCROSS_ORIGIN_CREDENTIAL_HEADERS, andprepareToolRequestfoldscollectProvenanceSensitiveHeadersinto the strip set as well — but only when the hop actually crosses origin.The first attempt used
stripAuthOnRedirect: trueinstead, which cubic correctly caught as the wrong primitive: it dropsAuthorizationon every hop, including same-origin, so a reverse proxy in front of Elasticsearch performing a legitimate same-origin redirect would have 401'd. That flag is right for tools redirecting to signed storage URLs (obsidian, mintlify, s3, dataverse), where the credential must never follow at all; Elasticsearch's requirement is narrower.mode: 'legacy'is deliberate — it preserves the existing method and body replay semantics, so the only behavior change is the cross-origin strip. Under'standard',resolveRedirectHopapplies Fetch method rules and a 301/302 would rewrite POST to GET, breaking_search,_countand_bulk.tools/github/utils.server.tspicks'legacy'for the same reason.On scope: this is pre-existing, not introduced by this PR. The tools always sent
Authorization; the consolidation only moved where the header is built. It is fixed here rather than deferred to a follow-up because three things hold together, and the exception is not meant to generalize:buildBaseUrlcould ship credentials to a stale or attacker-influenced host. A cross-origin redirect leaking the same credential from the same request is that bug's twin — closing one door while leaving the other open would invite exactly that question in review.Had any one of those failed — a larger diff, a different function, or a novel approach — it would have been split into its own PR. Other pre-existing findings from this audit were deferred on exactly that basis; they are listed below.
Deferred to follow-ups (pre-existing, filed separately)
optional: true:get_document._version/_source(absent on the 404 branch, and_sourcealso whenever_source_excludesstrips it),delete_document._version, andcluster_stats.status(optionalinStatsResponseBase).outputsnot covering every tool:list_indices.messageandcount._shardsare returned but undeclared, so they do not appear in the reference picker. Runtime resolution walks the real object, so hand-typed paths still resolve.Both are declaration-only improvements that would force a
tool-metadata:generateon a merge-ready branch, and neither is a security or correctness defect — which is precisely why they are deferred and the redirect fix is not.