Skip to content

Validate ingestion support for indexed schemas - #1384

Open
jwils wants to merge 2 commits into
mainfrom
joshuaw/validate-ingestion-adapters
Open

jwils wants to merge 2 commits into
mainfrom
joshuaw/validate-ingestion-adapters

Conversation

@jwils

@jwils jwils commented Sep 12, 2026

Copy link
Copy Markdown
Collaborator

Why

Missing ingestion support should fail before events reach the indexer. Artifact generation currently accepts schemas with no ingestion extension, and the JSON-aware indexer accepts configurations without JSON artifacts or an adapter.

What

Require indexed schemas to register an indexer extension that defines ingestion_adapters_by_format. Reject construction of JSONIngestion::Indexer unless its base indexer has JSON schema artifacts and a json ingestion adapter.

How

Runtime metadata generation checks the registered extension methods without constructing runtime services. The indexer still checks for an empty adapter registry at runtime.

The JSON wrapper checks for versioned JSON schema artifacts and reports an actionable configuration error when JSON support is absent. Tests use a shared non-JSON adapter whose registration is verified through generated indexing operations.

Risk

Schemas that register only unrelated indexer extensions now fail artifact generation. The schema check verifies the extension interface; it does not instantiate adapters or validate their runtime configuration.

Testing

No manual testing. Local datastore tests were blocked because Docker Desktop requires sign-in to the squareup organization.

Bigger picture

This follows the ingestion adapter extraction in #1302 and the JSON-aware indexer in #1220. It addresses the review feedback deferred from #1220.

@jwils
jwils added this pull request to stack #1336 September 12, 2026 23:32
@jwils jwils changed the title joshuaw/validate ingestion adapters Reject indexed schemas without an indexer extension Sep 12, 2026
@jwils
jwils force-pushed the joshuaw/validate-ingestion-adapters branch 2 times, most recently from 4861e77 to 25b4c26 Compare September 13, 2026 14:26
@jwils
jwils force-pushed the joshuaw/validate-ingestion-adapters branch 2 times, most recently from 9d07722 to c5895fc Compare September 13, 2026 14:54
@jwils
jwils force-pushed the joshuaw/validate-ingestion-adapters branch from c5895fc to 028e4b8 Compare September 13, 2026 15:28
@jwils
jwils removed this pull request from stack #1336 September 13, 2026 15:29
@jwils
jwils changed the base branch from joshuaw/typed-indexing-events to main September 13, 2026 15:29
@jwils
jwils marked this pull request as ready for review September 13, 2026 15:52
jwils added a commit that referenced this pull request Sep 13, 2026
## Why

`ElasticGraph::Indexer` accepts decoded events and should stay
independent of their wire format. Applications may need separate JSON
and protobuf indexers, while `elasticgraph-indexer_lambda` still needs
ElasticGraph to decode its JSON Lines payloads from SQS.

## What

- Add `ElasticGraph::JSONIngestion::Indexer`, which wraps the base
indexer with JSON Lines support.
- Make `elasticgraph-indexer_lambda` use that JSON-aware wrapper while
leaving its SQS payload format unchanged.
- Keep payload decoder configuration and extension points out of the
base indexer.

## How

The wrapper can build its own `ElasticGraph::Indexer` from YAML or wrap
an existing instance. Its `process` methods decode JSON Lines and pass
the events to the base processor. `SqsProcessor` uses the wrapper to
decode each SQS body, adds transport metadata, and sends the combined
batch to the base processor.

This lets a future `ElasticGraph::ProtoIngestion::Indexer` own protobuf
decoding without adding another format-specific hook to
`elasticgraph-indexer`.

## Risk

Low. The Lambda still accepts JSON Lines and follows the same batching
and failure-handling paths. The new wrapper changes internal
construction but does not add a configuration migration.

## Testing

No manual testing.

## Bigger picture

This keeps the Ruby indexer format-neutral while giving applications an
ElasticGraph-owned decoder for each supported format.

## Stack

Current PR is marked with `->`.

- [#1301 Extract an ingestion adapter seam inside
elasticgraph-indexer](#1301)
(merged)
- [#1302 Move JSON ingestion into elasticgraph-json_ingestion via an
indexer extension](#1302)
(merged)
- -> [#1220 Add a JSON-aware indexer
wrapper](#1220)
- [#1351 Extract indexing field metadata behind a format-neutral value
object](#1351)
- [#1284 Keep ingestion schema versions
adapter-owned](#1284)
- [#1376 Pass transport metadata to event
decoders](#1376)
- [#1377 Model decoded indexing events as typed
values](#1377)
- [#1384 Reject indexed schemas without an indexer
extension](#1384)
@jwils
jwils force-pushed the joshuaw/validate-ingestion-adapters branch from 028e4b8 to 6cd8b8b Compare September 13, 2026 17:56
@jwils jwils changed the title Reject indexed schemas without an indexer extension Require complete ingestion support for indexed schemas Sep 13, 2026
@jwils jwils changed the title Require complete ingestion support for indexed schemas Validate ingestion support for indexed schemas Sep 13, 2026
Comment thread elasticgraph-json_ingestion/lib/elastic_graph/json_ingestion/indexer.rb Outdated
Comment thread elasticgraph-json_ingestion/lib/elastic_graph/json_ingestion/indexer.rb Outdated
Comment thread elasticgraph-schema_definition/lib/elastic_graph/schema_definition/rake_tasks.rb Outdated
# @param indexer [ElasticGraph::Indexer] the format-neutral indexer to wrap
def initialize(indexer)
unless indexer.ingestion_adapters_by_format.key?("json") &&
indexer.schema_artifacts.respond_to?(:available_json_schema_versions) &&

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

indexer.schema_artifacts.respond_to?(:available_json_schema_versions)

This reminds me--I had designed ElasticGraph::SchemaDefinition::Results to have the same interface as ElasticGraph::SchemaArtifacts::FromDisk so that the in-memory results object could stand in for a FromDisk instance of schema_artifacts in our tests so that tests can operate entirely in memory w/o disk in most cases.

But currently they diverge:

  • FromDisk still has JSON schema APIs on it:

def json_schemas_for(version)
unless available_json_schema_versions.include?(version)
raise Errors::MissingSchemaArtifactError, "The requested json schema version (#{version}) is not available. " \
"Available versions: #{available_json_schema_versions.sort.join(", ")}."
end
json_schemas_by_version[version] # : ::Hash[::String, untyped]
end
# Provides the set of available JSON schema versions.
#
# @return [Set<Integer>]
# @see #json_schemas_for
# @see #latest_json_schema_version
#
# @example Print the list of available JSON schema versions
# artifacts = ElasticGraph::SchemaArtifacts::FromDisk.new(schema_artifacts_dir)
# puts artifacts.available_json_schema_versions.sort.join(", ")
def available_json_schema_versions
@available_json_schema_versions ||= begin
versioned_json_schemas_dir = ::File.join(artifacts_dir, JSON_SCHEMAS_BY_VERSION_DIRECTORY)
if ::Dir.exist?(versioned_json_schemas_dir)
::Dir.entries(versioned_json_schemas_dir).filter_map { |filename| filename[/v(\d+)\.yaml/, 1]&.to_i }.to_set
else
::Set.new
end
end
end
# Provides the latest JSON schema version.
#
# @return [Integer]
# @raise [Errors::MissingSchemaArtifactError] when no JSON schemas files exist within the `artifacts_dir`.
# @see #available_json_schema_versions
# @see #json_schemas_for
#
# @example Print the latest JSON schema version
# artifacts = ElasticGraph::SchemaArtifacts::FromDisk.new(schema_artifacts_dir)
# puts artifacts.latest_json_schema_version
def latest_json_schema_version
@latest_json_schema_version ||= available_json_schema_versions.max || raise(
Errors::MissingSchemaArtifactError,
"The directory for versioned JSON schemas (#{::File.join(artifacts_dir, JSON_SCHEMAS_BY_VERSION_DIRECTORY)}) could not be found. " \
"Either the schema artifacts haven't been dumped yet or the schema artifacts directory (#{artifacts_dir}) is misconfigured."
)
end

  • ...whereas Results does not have them:

def graphql_schema_string
@graphql_schema_string ||= generate_sdl
end
# @return [Hash<String, Object>] the Elasticsearch/OpenSearch configuration dumped as `datastore_config.yaml`
def datastore_config
@datastore_config ||= generate_datastore_config
end
# @return [Hash<String, Object>] runtime metadata used by other parts of ElasticGraph and dumped as `runtime_metadata.yaml`
def runtime_metadata
@runtime_metadata ||= build_runtime_metadata
end
# @private
STATIC_SCRIPT_REPO = Scripting::FileSystemRepository.new(::File.join(__dir__.to_s, "scripting", "scripts"))
# @private
def derived_indexing_type_names
@derived_indexing_type_names ||= state
.object_types_by_name
.values
.flat_map { |type| type.derived_indexed_types.map { |dit| dit.destination_type_ref.name } }
.to_set
end
# @private
def sourced_update_targets_by_source_type_name
@sourced_update_targets_by_source_type_name ||= Indexing::SourcedFromUpdateTargetsResolver.new(state).resolve
end

That's a mismatch that leads to .respond_to? checks which I'd like to avoid, and also compromises the clean "FromDisk and Results can are interchangeable story" we had before.

We should figure out a solution--maybe as part of this PR (or in another). (And we should make sure to follow the same pattern for proto ingestion).

@jwils
jwils force-pushed the joshuaw/validate-ingestion-adapters branch 3 times, most recently from f83012f to ac0236c Compare September 13, 2026 23:57
@jwils
jwils force-pushed the joshuaw/validate-ingestion-adapters branch 2 times, most recently from a419d34 to add7f7b Compare September 14, 2026 02:40
@jwils
jwils force-pushed the joshuaw/validate-ingestion-adapters branch from add7f7b to ab59025 Compare September 14, 2026 13:40
@jwils
jwils added this pull request to stack #1390 September 14, 2026 13:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants