feat: experimental MCP analytics (PostHog::MCP) for the Ruby mcp gem - #269
rafaeelaudibert wants to merge 11 commits into
Conversation
Wrap an MCP::Server so every tool call, handshake, listing, prompt, resource read, and failure is captured as a $mcp_* event with the same wire contract as @posthog/mcp and posthog.mcp. Ships inside posthog-ruby behind `require 'posthog/mcp'`; the mcp gem is a peer dependency. - PostHog::MCP.instrument(server, client = nil, **options); the client falls back to the posthog-rails PostHog.client facade - Prepends MCP::Server#handle_request to strip injected arguments, time the call, and record results/errors without ambient state; Fiber[] storage only for the Streamable HTTP transport hop - Context/intent injection, conversation ids, get_more_tools, stateless Mcp-Session-Id tokens, llm_model capture, identify, before_send, event_properties, sanitization, PII redaction, truncation - PostHog::MCP::Client for custom dispatchers and PostHog::MCP::RackMiddleware - Private per-event _lib/_lib_version override in Client#capture so MCP events report $lib posthog-ruby-mcp without relabeling the host client - Experimental: warns on require and on instrument Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
Docs PR: PostHog/posthog.com#20022 |
posthog-ruby Compliance ReportDate: 2026-09-11 18:52:57 UTC ✅ All Tests Passed!46/46 tests passed Capture Tests✅ 29/29 tests passed View Details
Feature_Flags Tests✅ 17/17 tests passed View Details
|
Prompt To Fix All With AI### Issue 1
lib/posthog/mcp/truncation.rb:24
**Payload limit drops events**
The MCP truncation budget allows events up to 102,400 bytes, but the core client rejects serialized messages over 32,768 bytes. Tool responses or parameters between those limits pass MCP truncation and are then omitted from the batch, causing valid tool-call analytics to be lost. The final payload, including any `before_send` changes, needs to fit the core message budget.
### Issue 2
lib/posthog/mcp/identity.rb:16-27
**Identity cache is unsynchronized**
`IdentityCache#get` performs an unlocked delete followed by reinsertion, while concurrent requests share this cache and both identification and event capture mutate it. Interleaved requests can temporarily lose an entry, evict the wrong entry, or attach an identity updated by another request. Protect these compound operations with the existing per-server mutex and cover concurrent identification in a test.
### Issue 3
lib/posthog/mcp/rack_middleware.rb:46-49
**Failed initialization mints sessions**
The middleware mints a session token before calling the application and adds it to every response, including rejected or failed initialization responses. A client that retains this header can replay a session created for an initialization that never succeeded. This also differs from the automatic transport, which mints only after successful dispatch. Attach the token only after confirming initialization succeeded.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.Reviews (1): Last reviewed commit: "feat: add experimental PostHog::MCP anal..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
PR overviewAll previously flagged issues have been addressed. No open security concerns remain on this pull request. Security reviewNo open security issues remain on this pull request. Fixed/addressed: 3 · PR risk: 0/10 |
- Budget truncation under the core client's 32KB per-message limit (which drops larger messages at batch time) and trim the largest strings before reducing depth so big tool responses keep their shape - Make IdentityCache thread-safe and merge identities atomically - RackMiddleware attaches the minted Mcp-Session-Id only on a successful initialize response Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
| @@ -0,0 +1,116 @@ | |||
| # PostHog MCP analytics for Ruby | |||
|
|
|||
| > **Experimental.** `PostHog::MCP` is new and its API, options, and the captured `$mcp_*` event schema may change in a minor release. A one-line warning is logged when you require it. Please report issues at https://github.com/PostHog/posthog-ruby/issues. | |||
There was a problem hiding this comment.
lets remove all the code snippets and event/prop names from this file to https://posthog.com/docs/mcp-analytics so its a single source of truth
There was a problem hiding this comment.
While this is experimental I thought it made sense to keep this here as source of truth. Eventually once the MCP Analytics starts owning this then they can probably own moving this. Are you ok with that?
There was a problem hiding this comment.
well fine by me but mcp in general is experimental https://github.com/PostHog/posthog-js/blob/f998ee9c0010d94f3fa0f33cccfd57f55e792e7e/packages/mcp/package.json#L3
and it lives in posthog.com
same for python, so i dont see a reason not to
There was a problem hiding this comment.
Done in f6bf9f0 — you're right, and posthog.com is now the single source of truth. lib/posthog/mcp/README.md keeps no code snippets or event/property names: just what the integration is, a link to https://posthog.com/docs/mcp-analytics, the handful of genuinely Ruby-specific notes (per-event $lib, the tighter 32KB truncation budget, request scope needing Ruby 3.2+ to be inherited, composed schemas left untouched, stdio logging), and a file map for anyone reading the code.
While I was there I also took @lucasheriques' point about support: the README banner, the require-time warning, the changeset and the example now say the integration is experimental and not officially supported, rather than implying it's supported via issues. Same on the docs side — PostHog/posthog.com#20022 now says the MCP analytics team doesn't maintain it and points at TypeScript or Python for a supported SDK.
|
@PostHog/mcp-analytics is there any demand for this (i've not seen any request for it)? its one more package to maintain so take that into consideration |
marandaneto
left a comment
marandaneto
left a comment
There was a problem hiding this comment.
Advisory review of the complete PR patch at b72f5e0. These seven findings were reproduced with focused regression tests in disposable worktrees. The existing MCP and core-client suite passed (250 examples). Validation used Ruby 4.0.6, MCP 1.5.0, and available dependency versions rather than the complete locked bundle; temporary controls were not retained as production fixes.
| return sanitized unless sanitized.is_a?(Hash) | ||
|
|
||
| result = sanitized.dup | ||
| if result['content'].is_a?(Array) |
There was a problem hiding this comment.
blocking: Redact binary prompt and resource response shapes too. Only top-level tool-result content arrays reach sanitize_content_block. Automatically captured prompt results instead use messages[].content, and resource reads use contents[].blob; these receive only generic string sanitization, so binary payloads below the 10KB heuristic threshold reach $mcp_response unchanged despite the documented binary-content redaction. Reproduction: reproduced — the focused review_payloads_spec.rb RSpec test found the original blob c2Vuc2l0aXZl in both $mcp_prompt_get and $mcp_resource_read payloads returned by the actual sink.
There was a problem hiding this comment.
Fixed by bf75806 — sanitize_response is now shape-aware for messages[].content (single block or array) and contents[].blob, so a sub-10KB blob is redacted there too. Covered end to end in instrument_spec.rb ("redacts binary prompt messages and resource blobs on the way out"), which asserts the original blob appears nowhere in the emitted payloads.
|
left a few comments but @PostHog/mcp-analytics should know more about the specifics |
`Analytics#capture` read the server-wide `data.session_id`, which the tool call only settled *after* its handler returned. A custom event emitted from inside a tool body therefore carried whichever session the previous request left behind, and with it that caller's identity. Two layers: - prime the session before the tool body runs (session id only; the conversation anchor is still resolved after the call) - pin the primed session to the in-flight `RequestScope`, and have `capture` read it from there first, so overlapping requests on a threaded server cannot cross-attribute either Specs cover both the sequential and the overlapping Alice/Bob case. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… in comments Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@marandaneto I didn't hear any demand from them, but I have my side projects in Rails and I want to instrument them with MCP Analytics, so I built this for myself. I 100% believe we should not merge this if the team is not interested in maintaining it, because I don't plan on maintaining all this myself. I can get the same by doing a point implementation just for my server, too. |
i mean all the work is done so i think we should merge and release it but i'd like to bring this up because i heard the same from the AIO team, they are too thin to bring support for more SDKs (or just maintain it) |
`URI.decode_www_form_component` is form decoding and turns `+` into a space, but `+` is a standard base64 character. Valid binary data URLs therefore failed detection and were captured verbatim. Use percent-only decoding instead. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Custom event properties, `event_properties` and `capture_tool_call` properties skipped normalization and progressive depth reduction. A large numeric array cannot be shrunk by string trimming, so such events stayed oversized and the core client dropped them at batch time. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`stringify_keys` runs before the cycle-aware normalizer and recursed forever on self-referential custom properties. The resulting `SystemStackError` is not a `StandardError`, so it escaped the sink into host code. Track visited containers and emit the same `[Circular ~]` marker the normalizer uses. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The virtual tool used to short-circuit the gem's dispatch lambda, which skipped envelope validation and leaked the in-flight cancellation entry the gem registers before handing us the lambda. `instrument` now defines the tool on the server (unless the application already owns the name), the gem dispatches it like any other tool, and the instrumentation only swaps the recorded event for `$mcp_missing_capability`. The manual `tools/list` append goes away since the gem lists the tool itself. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`prepare_tool_list` leaves a `context` field the tool declares itself intact, but `prepare_tool_call` stripped the argument unconditionally, so such a tool received incomplete arguments. `prepare_tool_call` now takes an optional `input_schema:` and strips `context` only when the schema does not declare it. Without the schema the behaviour is unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
i'll review this for the new spec soon, but as @marandaneto pointed, we on the MCP Analytics team do not have the capability to support this SDK for now. i think we can merge this though - work is done as Manoel pointed, but maybe on the docs, we say it's not an official solution and we don't provide support for it yet. |
There was a problem hiding this comment.
Note
🤖 Automated comment by QA Swarm — not written by a human
Replaced this long report with nine concise inline comments and a short summary.
| tool = tools.is_a?(Hash) ? tools[name] : nil | ||
| schema = tool.respond_to?(:input_schema) ? tool.input_schema&.to_h : nil | ||
| owned = [] | ||
| owned << 'context' if @options.context_enabled? && !SchemaMutation.declares_param?(schema, 'context') |
There was a problem hiding this comment.
Note
🤖 Automated comment by QA Swarm — not written by a human
[P1] Preserve composed/reference schemas
Before tools/list, a tool whose allOf schema requires application-owned context loses that argument and fails validation; the identical call succeeds after listing. Use one conservative injection/ownership guard, including $ref: injecting required context beside a reference to a closed object also makes discovery unsatisfiable. The standalone prepare_tool_call has the same composed-ownership problem.
There was a problem hiding this comment.
Fixed by bf75806 — one guard, SchemaMutation.injectable? (not composed, no $ref), is now used by add_parameter, Instrumentation#owned_params_for and Client#prepare_tool_call. A composed or referenced schema is neither injected into nor stripped from, so a call before the first tools/list behaves exactly like one after it.
| return sanitized unless sanitized.is_a?(Hash) | ||
|
|
||
| result = sanitized.dup | ||
| if result['content'].is_a?(Array) |
There was a problem hiding this comment.
Note
🤖 Automated comment by QA Swarm — not written by a human
[P1] Redact binary prompt/resource responses
Only top-level content blocks are redacted. Real prompts/get and resources/read calls still capture short binary payloads from messages[].content.data and contents[].blob in $mcp_response. Extend shape-aware redaction to those response forms while preserving client-facing results. This reproduces the existing unresolved finding.
There was a problem hiding this comment.
Fixed by bf75806 — same fix as the blocking thread above: messages[].content and contents[].blob now go through the shape-aware redaction, with an end-to-end spec over real prompts/get and resources/read calls.
| event['response'] = result | ||
| if tool_result_error?(result) | ||
| event['is_error'] = true | ||
| event['error'] = Exceptions.capture_exception(Sanitization.stringify_keys(result)) |
There was a problem hiding this comment.
Note
🤖 Automated comment by QA Swarm — not written by a human
[P2] Extract errors before adding conversation metadata
With enable_conversation_id, identical failures become same failure {"conversation_id":"<fresh UUID>"} in $mcp_error_message, fragmenting error grouping. Extract errors from the original result before appending the conversation handle. Current TS already does this.
There was a problem hiding this comment.
Fixed by bf75806 — the error is now extracted from the result the tool returned, before the conversation handle is appended, so identical failures keep an identical $mcp_error_message. The handle still reaches the agent on the delivered result.
| # full {#prepare_request} still runs after the call, because the conversation | ||
| # anchor is only known once the tool has returned. Emits nothing. | ||
| def prime_session | ||
| session_id, = Session.resolve(@data, mcp_session_id(@token), token: @token) |
There was a problem hiding this comment.
Note
🤖 Automated comment by QA Swarm — not written by a human
[P2] Resolve identity and echoed conversation before capture
On a first modern request with a valid echoed conversation ID, analytics.capture inside the tool uses an anonymous, temporary session; the automatic tool event uses the identified person and conversation session. Resolve and pin both identity and the already-known conversation anchor before invoking the tool.
There was a problem hiding this comment.
Fixed by bf75806 — session and identity are now settled before the tool body runs, anchored on the echoed conversation_id when the agent sent one. Non-HTTP transports open a request scope too, so the pin works over stdio as well, and identify still runs once per session per request.
| return tool unless tool.is_a?(Hash) | ||
|
|
||
| name = fetch(tool, :name) | ||
| return tool if virtual_tool?(name) |
There was a problem hiding this comment.
Note
🤖 Automated comment by QA Swarm — not written by a human
[P2] Complete model capture for missing capabilities
With capture_model: true, get_more_tools neither advertises llm_model nor records $mcp_llm_model when supplied. Its recorder and the standalone client's preparation/missing-capability helpers need the model support already present in current Python and TS.
There was a problem hiding this comment.
Fixed by bf75806 — get_more_tools advertises llm_model under capture_model and records $mcp_llm_model/$mcp_llm_model_source, in the instrumented path and in Client#prepare_tool_list / #capture_missing_capability. conversation_id is deliberately left off the virtual tool: it reports a gap in the tool list rather than taking part in a tool conversation.
| status, headers, body = @app.call(env) | ||
| # Only a successful initialize gets the token, so a client cannot replay a | ||
| # session minted for a handshake the server rejected. | ||
| if token && success?(status) && headers.respond_to?(:key?) && |
There was a problem hiding this comment.
Note
🤖 Automated comment by QA Swarm — not written by a human
[P2] Check initialization success beyond HTTP status
An HTTP 200 response containing a JSON-RPC error still gets a new mcp-session-id. The legacy transport contract associates that header with InitializeResult. Require a successful initialization result or an explicit success signal from the dispatcher before attaching the token.
There was a problem hiding this comment.
Fixed by bf75806 — the middleware now requires an InitializeResult before attaching Mcp-Session-Id, so a JSON-RPC error on a 200 mints nothing. A body that cannot be read without consuming a stream (SSE) keeps the status-only behaviour, which is documented on the method.
| 'session_id' => session_id, | ||
| 'resource_name' => request_resource_name(request), | ||
| 'event_type' => EventType::IDENTIFY, | ||
| 'parameters' => { 'request' => request, 'extra' => captured_extra(extra) }, |
There was a problem hiding this comment.
Note
🤖 Automated comment by QA Swarm — not written by a human
[P2] Remove raw intent from identify parameters
With identify enabled, context: "Look up jane@example.com" is redacted on $mcp_tool_call but survives verbatim in $identify under $mcp_parameters.request.params.arguments.context. Use the captured-parameter builder here too. Python/TS share this privacy gap; parity alone would preserve it.
There was a problem hiding this comment.
Not changing this one: we want the same behaviour across the SDKs here, so this stays as it is rather than diverging in Ruby alone. Worth raising as a cross-SDK issue if we want the captured-parameter builder used for $identify everywhere.
| instance_method PostHog::MCP::Client#capture_missing_capability(context: ..., parameters: ..., protocol_version: ..., distinct_id: ..., session_id: ..., client_user_agent: ..., vendor_client: ..., set_properties: ..., groups: ..., properties: ..., timestamp: ...) | ||
| instance_method PostHog::MCP::Client#capture_tool_call(tool_name, intent: ..., intent_source: ..., parameters: ..., response: ..., duration_ms: ..., is_error: ..., error: ..., error_type: ..., category: ..., tool_description: ..., protocol_version: ..., distinct_id: ..., session_id: ..., client_user_agent: ..., vendor_client: ..., set_properties: ..., groups: ..., properties: ..., timestamp: ..., llm_model: ..., llm_model_source: ...) | ||
| instance_method PostHog::MCP::Client#capture_tools_list(tool_names: ..., parameters: ..., response: ..., duration_ms: ..., is_error: ..., error: ..., error_type: ..., protocol_version: ..., distinct_id: ..., session_id: ..., client_user_agent: ..., vendor_client: ..., set_properties: ..., groups: ..., properties: ..., timestamp: ...) | ||
| instance_method PostHog::MCP::Client#prepare_tool_call(name, args = ...) |
There was a problem hiding this comment.
Note
🤖 Automated comment by QA Swarm — not written by a human
[P2] Refresh the public API snapshot
prepare_tool_call now accepts input_schema:, but this snapshot omits it. bundle exec rake public_api:check fails locally and in CI. Regenerate the snapshot after settling the API changes.
There was a problem hiding this comment.
Fixed by bf75806 — regenerated after the API changes settled; bundle exec rake public_api:check passes.
| properties_key = key_for(schema, :properties) | ||
| schema[properties_key] = {} unless schema[properties_key].is_a?(Hash) | ||
| additional_key = key_for(schema, :additionalProperties) | ||
| schema.delete(additional_key) if schema[additional_key] == false |
There was a problem hiding this comment.
Note
🤖 Automated comment by QA Swarm — not written by a human
[P2] Preserve additionalProperties: false
For a strict tool declaring x, {x: "yes", context: "why", typo: 123} passes the advertised schema but dispatch rejects /typo. The injected property is already allowed through properties; keep additionalProperties: false so discovery matches validation. Python preserves it; TS shares this bug.
There was a problem hiding this comment.
Fixed by bf75806 — additionalProperties: false is preserved; the injected name is listed under properties, so it is still accepted, and the spec that asserted the removal was inverted.
- Redact binary payloads in `prompts/get` `messages[].content` and `resources/read` `contents[].blob`; only tool-result `content` blocks were shape-aware, so a short blob reached `$mcp_response` verbatim. - Never inject into, or claim ownership of, a composed (oneOf/allOf/anyOf) or `$ref` input schema. A tool declaring `context` inside `allOf` used to lose the argument when called before the first `tools/list`. One guard, `SchemaMutation.injectable?`, now covers listing, dispatch and `Client#prepare_tool_call`. - Keep `additionalProperties: false` when injecting: the injected name is listed under `properties`, so dropping the constraint only advertised a looser schema than the dispatcher validates against. - Extract a failed tool result's error from the result the tool returned, before the conversation handle is appended, so identical failures keep an identical `$mcp_error_message` instead of one per conversation. - Settle session and identity before the tool body runs, anchored on an echoed `conversation_id` when the agent sent one, so a custom event captured inside a tool carries the same `$session_id` and person as its `$mcp_tool_call`. Non-HTTP transports now open a request scope too, and `identify` still runs once per session per request. - Fail closed when an in-tool capture has lost its request scope on a server that has served HTTP: use a standalone session rather than the server-wide one, which may belong to a concurrent request. - Advertise and record `llm_model` on `get_more_tools` under `capture_model`, in the instrumented path and in `Client#prepare_tool_list` / `#capture_missing_capability`. - `RackMiddleware` requires an `InitializeResult` before attaching `Mcp-Session-Id`: a JSON-RPC error can ride on a 200. A body that cannot be sniffed without consuming a stream keeps the status-only behaviour. - Regenerate the public API snapshot. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Pushed bf75806, which addresses the outstanding review threads. Summary of what changed and how:
Two threads intentionally unchanged:
|
The Rack middleware sniffed the JSON-RPC request body to spot an `initialize`, and the response body to check the handshake succeeded. On a non-rewindable `rack.input` the bounded read was followed by a lengthless one, so a large POST was buffered whole and MAX_SNIFF_BODY stopped being a memory bound. Nothing is parsed now. The middleware publishes the request headers to RequestScope as an HTTP request, so an instrumented server dispatched below it mints exactly as it does under the gem's own transport - after the handshake produced an InitializeResult, which is a stronger signal than sniffing a 200 for a `result` key, and works for SSE bodies too. A hand-rolled dispatcher, which has already parsed the body, mints through `env['posthog_mcp.mint']`. Docs move to posthog.com/docs/mcp-analytics as the single source of truth; the README keeps only the Ruby-specific notes. The experimental notice, the README, the changeset, and the example now all say the integration is unsupported, not just experimental. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
posthog-ruby-async Compliance ReportDate: 2026-09-15T22:21:18.174318+00:00
|
| Test | Status | Duration |
|---|---|---|
| Format Validation.Event Has Required Fields | ✅ | 10ms |
| Format Validation.Event Has Uuid | ✅ | 105ms |
| Format Validation.Event Has Lib Properties | ✅ | 109ms |
| Format Validation.Distinct Id Is String | ✅ | 107ms |
| Format Validation.Token Is Present | ✅ | 107ms |
| Format Validation.Custom Properties Preserved | ✅ | 107ms |
| Format Validation.Event Has Timestamp | ✅ | 108ms |
| Format Validation.Non Utc Event Timestamp Is Converted To Utc | ✅ | 10ms |
| Retry Behavior.Retries On 503 | ✅ | 5310ms |
| Retry Behavior.Does Not Retry On 400 | ✅ | 2109ms |
| Retry Behavior.Does Not Retry On 401 | ✅ | 2109ms |
| Retry Behavior.Respects Retry After Header | ✅ | 8116ms |
| Retry Behavior.Implements Backoff | ✅ | 15524ms |
| Retry Behavior.Retries On 500 | ✅ | 5212ms |
| Retry Behavior.Retries On 502 | ✅ | 5212ms |
| Retry Behavior.Retries On 504 | ✅ | 5212ms |
| Retry Behavior.Max Retries Respected | ✅ | 15523ms |
| Deduplication.Generates Unique Uuids | ✅ | 111ms |
| Deduplication.Preserves Uuid On Retry | ✅ | 5213ms |
| Deduplication.Preserves Uuid And Timestamp On Retry | ✅ | 10419ms |
| Deduplication.Preserves Uuid And Timestamp On Batch Retry | ✅ | 5214ms |
| Deduplication.No Duplicate Events In Batch | ✅ | 112ms |
| Deduplication.Different Events Have Different Uuids | ✅ | 107ms |
| Compression.Sends Gzip When Enabled | ✅ | 106ms |
| Batch Format.Uses Proper Batch Structure | ✅ | 107ms |
| Batch Format.Flush With No Events Sends Nothing | ✅ | 4ms |
| Batch Format.Multiple Events Batched Together | ✅ | 110ms |
| Error Handling.Does Not Retry On 403 | ✅ | 2109ms |
| Error Handling.Does Not Retry On 413 | ✅ | 2108ms |
| Error Handling.Retries On 408 | ✅ | 5212ms |
Feature_Flags Tests
View Details
| Test | Status | Duration |
|---|---|---|
| Request Payload.Request With Person Properties Device Id | ✅ | 108ms |
| Request Payload.Flags Request Uses V2 Query Param | ✅ | 108ms |
| Request Payload.Flags Request Hits Flags Path Not Decide | ✅ | 109ms |
| Request Payload.Flags Request Omits Authorization Header | ✅ | 108ms |
| Request Payload.Token In Flags Body Matches Init | ✅ | 107ms |
| Request Payload.Groups Round Trip | ✅ | 108ms |
| Request Payload.Groups Default To Empty Object | ✅ | 107ms |
| Request Payload.Disable Geoip False Propagates As Geoip Disable False | ✅ | 107ms |
| Request Payload.Disable Geoip Omitted Defaults To False | ❌ | 108ms |
| Request Payload.Flag Keys To Evaluate Contains Only Requested Key | ✅ | 107ms |
| Request Lifecycle.No Flags Request On Init Alone | ✅ | 3ms |
| Request Lifecycle.No Flags Request On Normal Capture | ✅ | 105ms |
| Request Lifecycle.Two Flag Calls Produce Two Remote Requests | ✅ | 112ms |
| Request Lifecycle.Mock Response Value Is Returned To Caller | ✅ | 106ms |
| Retry Behavior.Retries Flags On 502 | ✅ | 249ms |
| Retry Behavior.Retries Flags On 504 | ✅ | 210ms |
| Side Effect Events.Get Feature Flag Captures Feature Flag Called Event | ✅ | 109ms |
Failures
request_payload.disable_geoip_omitted_defaults_to_false
Field 'geoip_disable' not found in /flags request body at path 'geoip_disable'. Available keys: ['distinct_id', 'groups', 'person_properties', 'group_properties', 'flag_keys_to_evaluate', 'token']
posthog-ruby-sync Compliance ReportDate: 2026-09-15T22:21:18.625793+00:00
|
| Test | Status | Duration |
|---|---|---|
| Format Validation.Event Has Required Fields | ✅ | 11ms |
| Format Validation.Event Has Uuid | ✅ | 7ms |
| Format Validation.Event Has Lib Properties | ✅ | 7ms |
| Format Validation.Distinct Id Is String | ✅ | 11ms |
| Format Validation.Token Is Present | ✅ | 8ms |
| Format Validation.Custom Properties Preserved | ✅ | 8ms |
| Format Validation.Event Has Timestamp | ✅ | 9ms |
| Format Validation.Non Utc Event Timestamp Is Converted To Utc | ✅ | 8ms |
| Retry Behavior.Retries On 503 | ✅ | 5305ms |
| Retry Behavior.Does Not Retry On 400 | ✅ | 2011ms |
| Retry Behavior.Does Not Retry On 401 | ✅ | 2010ms |
| Retry Behavior.Respects Retry After Header | ✅ | 8018ms |
| Retry Behavior.Implements Backoff | ✅ | 15256ms |
| Retry Behavior.Retries On 500 | ✅ | 5117ms |
| Retry Behavior.Retries On 502 | ✅ | 5158ms |
| Retry Behavior.Retries On 504 | ✅ | 5117ms |
| Retry Behavior.Max Retries Respected | ✅ | 15515ms |
| Deduplication.Generates Unique Uuids | ✅ | 24ms |
| Deduplication.Preserves Uuid On Retry | ✅ | 5164ms |
| Deduplication.Preserves Uuid And Timestamp On Retry | ✅ | 10248ms |
| Deduplication.Preserves Uuid And Timestamp On Batch Retry | ✅ | 5118ms |
| Deduplication.No Duplicate Events In Batch | ✅ | 19ms |
| Deduplication.Different Events Have Different Uuids | ✅ | 9ms |
| Compression.Sends Gzip When Enabled | ✅ | 6ms |
| Batch Format.Uses Proper Batch Structure | ✅ | 8ms |
| Batch Format.Flush With No Events Sends Nothing | ✅ | 3ms |
| Batch Format.Multiple Events Batched Together | ❌ | 18ms |
| Error Handling.Does Not Retry On 403 | ✅ | 2008ms |
| Error Handling.Does Not Retry On 413 | ✅ | 2008ms |
| Error Handling.Retries On 408 | ✅ | 5117ms |
Failures
batch_format.multiple_events_batched_together
Expected 1 requests, got 5
Feature_Flags Tests
View Details
| Test | Status | Duration |
|---|---|---|
| Request Payload.Request With Person Properties Device Id | ✅ | 10ms |
| Request Payload.Flags Request Uses V2 Query Param | ✅ | 8ms |
| Request Payload.Flags Request Hits Flags Path Not Decide | ✅ | 10ms |
| Request Payload.Flags Request Omits Authorization Header | ✅ | 9ms |
| Request Payload.Token In Flags Body Matches Init | ✅ | 7ms |
| Request Payload.Groups Round Trip | ✅ | 7ms |
| Request Payload.Groups Default To Empty Object | ✅ | 7ms |
| Request Payload.Disable Geoip False Propagates As Geoip Disable False | ✅ | 7ms |
| Request Payload.Disable Geoip Omitted Defaults To False | ❌ | 6ms |
| Request Payload.Flag Keys To Evaluate Contains Only Requested Key | ✅ | 7ms |
| Request Lifecycle.No Flags Request On Init Alone | ✅ | 3ms |
| Request Lifecycle.No Flags Request On Normal Capture | ✅ | 6ms |
| Request Lifecycle.Two Flag Calls Produce Two Remote Requests | ✅ | 10ms |
| Request Lifecycle.Mock Response Value Is Returned To Caller | ✅ | 8ms |
| Retry Behavior.Retries Flags On 502 | ✅ | 136ms |
| Retry Behavior.Retries Flags On 504 | ✅ | 110ms |
| Side Effect Events.Get Feature Flag Captures Feature Flag Called Event | ✅ | 8ms |
Failures
request_payload.disable_geoip_omitted_defaults_to_false
Field 'geoip_disable' not found in /flags request body at path 'geoip_disable'. Available keys: ['distinct_id', 'groups', 'person_properties', 'group_properties', 'flag_keys_to_evaluate', 'token']
|
lgtm |
lucasheriques
left a comment
lucasheriques
left a comment
There was a problem hiding this comment.
I reviewed the Ruby MCP integration against the current JavaScript and Python SDKs and ran the Ruby test suite. I left five inline findings for discussion.
| return nil if sink.nil? | ||
|
|
||
| session_id = input['session_id'] || data.session_id | ||
| actor = session_id ? data.identified_sessions.get(session_id) : nil |
There was a problem hiding this comment.
I found a race in identity attribution. capture_event reads the shared identity cache after the tool finishes. Another call in the same session can change it first. In a two-thread test, Alice's call waited while Bob's finished, and Alice's $mcp_tool_call used Bob's distinct_id.
The JavaScript SDK keeps the resolved identity with each request. Please keep that request identity for automatic and custom events, and add a concurrent regression test.
| # @param args [Hash, nil] the call's arguments | ||
| # @param input_schema [Hash, nil] the tool's raw `inputSchema` | ||
| # @return [PreparedToolCall] | ||
| def prepare_tool_call(name, args = nil, input_schema: nil) |
There was a problem hiding this comment.
The custom-dispatcher helpers do not round-trip llm_model yet. prepare_tool_list(..., capture_model: true) adds it as a required argument, but prepare_tool_call leaves it in args and does not return the model or its source. I passed those args to a strict Ruby tool and got ArgumentError: unknown keyword: :llm_model.
JavaScript and Python remove SDK-owned model fields and return the model metadata. Please add the same handling here, with a round-trip test.
|
|
||
| autocapture = options.nil? || options.enable_exception_autocapture | ||
| payloads = EventBuilder.build(processed, enable_exception_autocapture: autocapture) | ||
| apply_before_send(payloads, options) |
There was a problem hiding this comment.
An expanding before_send hook can cause a silent event drop. The MCP size check runs before the hook, while MessageBatch applies Ruby's 32 KB limit afterward. A hook that added 40 KB made the sink return one payload, but the transport sent zero requests.
Please enforce the final size limit after before_send and test this case.
| missing_name = Tools.missing_capability_tool_name(@options) | ||
| owned = safely([]) { owned_params_for(name) } | ||
| stripped = safely({}) { strip_injected_arguments(arguments, owned) } | ||
| conversation_id, minted = safely([nil, false]) do |
There was a problem hiding this comment.
The ownership check is in place, but intent and conversation resolution still read application-owned fields from original_arguments. I saw a tool-owned context become $mcp_intent. With two users sharing a tool-owned UUIDv7 conversation_id, Bob's event also included Alice's email in $set.
The JavaScript SDK consumes these values only when analytics owns them. Please use the ownership result for both fields and add tests for tools that declare them.
| end | ||
|
|
||
| def maybe_emit_initialize(session_id, request) | ||
| return if @data.session_initialized?(session_id) |
There was a problem hiding this comment.
The initialize check and mark are locked separately, but the full step is not atomic. Two threads can pass the check before either marks the session. I reproduced two $mcp_initialize events for one session.
Please make check-and-mark one atomic operation and add a concurrent test.
3 participants
💡 Motivation and Context
PostHog ships MCP analytics for Node (
@posthog/mcp) and Python (posthog.mcp), but not Ruby. This addsPostHog::MCP, an experimental integration that wraps a server built on the official Rubymcpgem so every tool call, handshake, listing, prompt, resource read, and failure lands in PostHog as a$mcp_*event with the same wire contract as the other two SDKs.Setup is one line, and Rails apps that already call
PostHog.initcan omit the client:What's included:
PostHog::MCP.instrumentwith feature parity: injectedcontextintent argument, conversation ids (prompt-back +structuredContentmirror),get_more_tools,llm_modelcapture, statelessMcp-Session-Idtokens, transport identity headers,identify/$identify,before_send,event_properties, sanitization, PII redaction of intent, and 100KB truncation. Ruby also emits the prompt/resource events the other SDKs reserve.PostHog::MCP::Clientfor custom dispatchers (nomcpgem needed) andPostHog::MCP::RackMiddlewarefor custom Rack stacks._lib/_lib_versionoverride inClient#capture. MCP events report$lib: posthog-ruby-mcp; unlike Node/Python this does not relabel the host client, so a Rails app's other events keepposthog-rails.mcpadded to the Gemfile test group only (peer dependency, gemspec unchanged); public API snapshot now tracksPostHog::MCP.Design notes for reviewers:
around_requesthook lacks the request, session, and params, so the integration prependsMCP::Server#handle_requestand wraps the dispatch lambda it returns. That lambda'sparamsis the same Hash the tool receives, which is what lets us strip injected arguments before thetool.call(**args)splat (an unknown keyword would otherwise become an opaque-32603). NoThread.current; the only ambient state isFiber[]storage for the Streamable HTTP transport → server hop, because the gem re-parses the body in between.require 'posthog/mcp'and oninstrument(stderr outside Rails, never stdout, since stdio servers own it). README, YARD, and the changeset say so too.💚 How did you test it?
bundle exec rspec: 812 examples, 0 failures (115 new examples underspec/posthog/mcp, including end-to-end specs against a realMCP::ServeroverServer#handleand overStreamableHTTPTransport#callin stateless and stateful modes, a concurrency spec, and the frozen cross-SDK vectors for FNV-1a session ids and theMcp-Session-Idtoken copied from the Python/JS suites).bundle exec rubocop: clean.bundle exec rake public_api:check: passes.examples/mcp_server.rbdriven over stdio with realinitialize/tools/list/tools/callJSON-RPC lines; captured events dumped to stderr.📝 Checklist
If releasing new changes
pnpm changesetto generate a changeset file (.changeset/quiet-ruby-mcp-analytics.md, minor)🤖 Agent context
Autonomy: Human-driven (agent-assisted)
Written with Claude Code (Claude Fable 5.1) from a plan the DRI reviewed and approved. The agent explored the JS and Python MCP packages and the Ruby
mcpgem source, and the DRI made the calls on scope (full parity), wiring (explicitinstrumentonly, no Railtie auto-wiring), and$libattribution (per-event override rather than relabeling the client). The DRI also asked for the experimental marking and for avoidingThread.current; the hook design was reworked from anaround_request+ thread-local draft to the singlehandle_requestprepend described above. Cross-SDK divergences were resolved deliberately: Python's empty-tools/listerror text and code-point FNV iteration, JS's[Array]truncation marker and requested-version era check for token minting.🤖 Generated with Claude Code