Conversation
|
Before you submit for review:
If you did not complete any of these, then please explain below. |
f11246a to
da703e8
Compare
da703e8 to
b7dff72
Compare
b7dff72 to
6a304ba
Compare
6a304ba to
a54013f
Compare
Build on patches introduced earlier in the series to allow BenchYAML to process that don't fit in physical memory, by memory-mapping the index vectors instead of fully loading them upfront. This behaviour is opt-in and requires you to set `-Djvector.bench.dataset.mmap.enable=true` to enable. Additionally, fixes a concurrency issue with RAVV handling in Grid, and makes the output directories configurable Summary of changes: - Add a `RavvDataSet` wherein the index vectors are backed by a RAVV. - Add a function to `DataSetInfo` to return a `DataSet` backed by memory-mapped files, using `RavvDataSet`. - Consume the new `DataSet` from `BenchYAML` - Fix a concurrency issue in Grid where `RAVV#getVector` was being called from multiple threads at once - Make Grid's work directory and index cache directory configurable via environment variables. (Useful if you have an SSD on a separate mount)
Fix conflicts generated from 0e8c7d6e (Make DataSetInfo an interface, 2026-09-15)
a54013f to
f4cb252
Compare
| @@ -24,4 +24,7 @@ public interface DataSetInfo extends DataSetProperties { | |||
| /// | |||
| /// @return the ready-to-use {@link InMemoryDataSet} | |||
| public InMemoryDataSet getDataSet(); | |||
There was a problem hiding this comment.
Why not just make getDataSet() return a DataSet, and have the implementation of that which is provided support RAVV? That was always the intent here.
A DataSet may implement its own loading methods by design. Some of our current implementation have historic limitations which should be removed. The way to do this would be to have a DataSet impl type which does it properly, and to remove any other surface but the RAVV style accessor.
To keep that supported, the getDataSet on DataSetInfo should remain a contract type, not any concrete type.
jshook
left a comment
There was a problem hiding this comment.
Let's align on the contract vs concrete boundaries.
Build on patches introduced earlier in the series to allow BenchYAML to process that don't fit in physical memory, by memory-mapping the index vectors instead of fully loading them upfront.
This behaviour is opt-in and requires you to set
-Djvector.bench.dataset.mmap.enable=trueto enable.Additionally, fixes a concurrency issue with RAVV handling in Grid, and makes the output directories configurable
Summary of changes:
RavvDataSetwherein the index vectors are backed by a RAVV.DataSetInfoto return aDataSetbacked by memory-mapped files, usingRavvDataSet.DataSetfromBenchYAMLRAVV#getVectorwas being called from multiple threads at once