Conversation
4861e77 to
25b4c26
Compare
9d07722 to
c5895fc
Compare
c5895fc to
028e4b8
Compare
## Why `ElasticGraph::Indexer` accepts decoded events and should stay independent of their wire format. Applications may need separate JSON and protobuf indexers, while `elasticgraph-indexer_lambda` still needs ElasticGraph to decode its JSON Lines payloads from SQS. ## What - Add `ElasticGraph::JSONIngestion::Indexer`, which wraps the base indexer with JSON Lines support. - Make `elasticgraph-indexer_lambda` use that JSON-aware wrapper while leaving its SQS payload format unchanged. - Keep payload decoder configuration and extension points out of the base indexer. ## How The wrapper can build its own `ElasticGraph::Indexer` from YAML or wrap an existing instance. Its `process` methods decode JSON Lines and pass the events to the base processor. `SqsProcessor` uses the wrapper to decode each SQS body, adds transport metadata, and sends the combined batch to the base processor. This lets a future `ElasticGraph::ProtoIngestion::Indexer` own protobuf decoding without adding another format-specific hook to `elasticgraph-indexer`. ## Risk Low. The Lambda still accepts JSON Lines and follows the same batching and failure-handling paths. The new wrapper changes internal construction but does not add a configuration migration. ## Testing No manual testing. ## Bigger picture This keeps the Ruby indexer format-neutral while giving applications an ElasticGraph-owned decoder for each supported format. ## Stack Current PR is marked with `->`. - [#1301 Extract an ingestion adapter seam inside elasticgraph-indexer](#1301) (merged) - [#1302 Move JSON ingestion into elasticgraph-json_ingestion via an indexer extension](#1302) (merged) - -> [#1220 Add a JSON-aware indexer wrapper](#1220) - [#1351 Extract indexing field metadata behind a format-neutral value object](#1351) - [#1284 Keep ingestion schema versions adapter-owned](#1284) - [#1376 Pass transport metadata to event decoders](#1376) - [#1377 Model decoded indexing events as typed values](#1377) - [#1384 Reject indexed schemas without an indexer extension](#1384)
028e4b8 to
6cd8b8b
Compare
| # @param indexer [ElasticGraph::Indexer] the format-neutral indexer to wrap | ||
| def initialize(indexer) | ||
| unless indexer.ingestion_adapters_by_format.key?("json") && | ||
| indexer.schema_artifacts.respond_to?(:available_json_schema_versions) && |
There was a problem hiding this comment.
indexer.schema_artifacts.respond_to?(:available_json_schema_versions)
This reminds me--I had designed ElasticGraph::SchemaDefinition::Results to have the same interface as ElasticGraph::SchemaArtifacts::FromDisk so that the in-memory results object could stand in for a FromDisk instance of schema_artifacts in our tests so that tests can operate entirely in memory w/o disk in most cases.
But currently they diverge:
FromDiskstill has JSON schema APIs on it:
- ...whereas
Resultsdoes not have them:
That's a mismatch that leads to .respond_to? checks which I'd like to avoid, and also compromises the clean "FromDisk and Results can are interchangeable story" we had before.
We should figure out a solution--maybe as part of this PR (or in another). (And we should make sure to follow the same pattern for proto ingestion).
f83012f to
ac0236c
Compare
a419d34 to
add7f7b
Compare
add7f7b to
ab59025
Compare
Why
Missing ingestion support should fail before events reach the indexer. Artifact generation currently accepts schemas with no ingestion extension, and the JSON-aware indexer accepts configurations without JSON artifacts or an adapter.
What
Require indexed schemas to register an indexer extension that defines
ingestion_adapters_by_format. Reject construction ofJSONIngestion::Indexerunless its base indexer has JSON schema artifacts and ajsoningestion adapter.How
Runtime metadata generation checks the registered extension methods without constructing runtime services. The indexer still checks for an empty adapter registry at runtime.
The JSON wrapper checks for versioned JSON schema artifacts and reports an actionable configuration error when JSON support is absent. Tests use a shared non-JSON adapter whose registration is verified through generated indexing operations.
Risk
Schemas that register only unrelated indexer extensions now fail artifact generation. The schema check verifies the extension interface; it does not instantiate adapters or validate their runtime configuration.
Testing
No manual testing. Local datastore tests were blocked because Docker Desktop requires sign-in to the squareup organization.
Bigger picture
This follows the ingestion adapter extraction in #1302 and the JSON-aware indexer in #1220. It addresses the review feedback deferred from #1220.