Add concept map Postgres load scripts for MIMIC-IV and NW.

[danamouk]

This pull request adds PostgreSQL scripts for loading concept map data into both MIMIC-IV and NW databases.

For each database (mimic-iv/concepts_postgres/concept_map/ and nw/concepts_postgres/concept_map/), it adds 5 SQL scripts and a README:

• create.sql — Creates the mimiciv_concept_map / nw_concept_map schema with 8 tables:
  labevents_to_loinc, labevents_to_omop,
  prescriptions_to_rxnorm, prescriptions_to_omop,
  chartevents_to_loinc, chartevents_to_omop,
  procedureevents_to_snomed, procedureevents_to_omop

• load.sql — Loads mapping CSVs from concepts/concept_map/hosp/ and icu/ into the tables

• constraint.sql — Adds primary keys on subject_id to enforce uniqueness and ensure data integrity.
  Prescriptions tables are excluded because different drug names share the same NDC code, so subject_id is not unique.

• index.sql — Creates indexes on subject_id and object_id to speed up lookups and joins against the mapping tables

• validate.sql — Checks row counts against expected values

• README.md — Quickstart and step-by-step setup guide

These concept maps provide mappings from local hospital codes to standard terminologies (LOINC, OMOP, RxNorm, SNOMED) using the SSSOM format.

[Chessing234]

concept map load scripts help. please note whether they assume a specific postgres version / search_path.

[Chessing234]

i asked about postgres version / search_path assumptions earlier, but having diffed this against your other two open prs i think the packaging needs sorting out first.

this branch isn't just the loader scripts — it carries the full contents of #1930 and #1935 as well. concretely, all seven files under nw/buildnw/postgres/ in this pr are byte-identical to the same seven files in #1935, and the twelve mimic-iv/concepts/concept_map/*.csv plus eight nw/concepts/concept_map/*.csv files are byte-identical to those in #1930 (i checksummed create.sql, load_gz.sql, labevents_to_loinc.csv, chartevents_to_omop.csv and procedureevents_to_omop.csv to confirm). so whichever of the three merges first leaves the other two with large conflicts in files their authors didn't intend to change, and a reviewer looking at this pr can't tell which of the 39 files are actually new here.

the genuinely new content looks like the six files under mimic-iv/concepts_postgres/concept_map/ and the six under nw/concepts_postgres/concept_map/ — create/load/index/constraint/validate plus the readme. could you rebase this onto #1930 and #1935 so it contains only those? that also makes the "one concern per pr" story clean: #1935 is the nw build, #1930 is the mapping data, this one is the postgres loader for it.

on the loaders themselves, once they're isolated: load.sql reading the csvs needs its path convention stated (server-side COPY vs \copy behave very differently for anyone not running psql on the db host), and validate.sql should say what it asserts and what a failure means. i'd also want to know whether create.sql assumes the mimiciv_concept* schema already exists or creates it, since that determines whether this can run standalone.

happy to go through the loader sql properly once the diff only contains it.

[Chessing234]

went through the twelve loader files on their own terms this time rather than waiting for the rebase. the shape is right — it mirrors buildmimic/postgres closely and the note in constraint.sql explaining why the two prescriptions tables can't take a PK on subject_id is exactly the kind of thing i wish more build scripts had. i checked that claim and it's correct: 416 NDCs repeat, all with one distinct object_id each, so it's label variants and nothing else. the PKs you do declare all hold — subject_id is unique in all six of the other files.

the thing i'd most want resolved before this lands is how anyone is supposed to join these tables to the data.

subject_id is stored as the SSSOM CURIE, so labevents_to_loinc.subject_id is 'mimic-itemid:50912' while mimiciv_hosp.labevents.itemid is INTEGER. every real query then has to write

JOIN mimiciv_concept_map.labevents_to_loinc m
  ON m.subject_id = 'mimic-itemid:' || le.itemid::text

or split_part(m.subject_id, ':', 2)::int, and neither uses labevents_to_loinc_idx01. on a 158M-row labevents that's the difference between an index join and a seq scan per lookup. loading the map into postgres is specifically about making it joinable, so it seems worth carrying a bare key alongside the CURIE — either a plain itemid INTEGER column populated after load, or a generated column with its own index:

ALTER TABLE mimiciv_concept_map.labevents_to_loinc
  ADD COLUMN itemid INTEGER GENERATED ALWAYS AS (split_part(subject_id, ':', 2)::int) STORED;

(subject_id, object_id) as text is still the SSSOM record; the extra column is just the join handle.

the prescriptions maps have two different NDC formats in them, and only one of them can be the join key. in prescriptions_to_omop.csv, 1763 subject NDCs are 11 characters and 1340 are 9:

mimic-ndc:58160082152    (11)
mimic-ndc:023916330      (9)

hosp.prescriptions.ndc is a fixed 11-digit zero-padded string — the drug mapping file in #1753 is uniformly 11 digits, which is the convention i'd expect. if that's right, roughly 43% of the rows in both prescriptions tables silently match nothing, and the loader is where you'd catch it (lpad on the way in, or reject anything not 11 digits). could you check a handful of the 9-character ones against hosp.prescriptions and say which form is correct?

validate.sql bakes in two numbers that i think are data bugs rather than facts. prescriptions_to_rxnorm is expected at 3107 and prescriptions_to_omop at 3103. the four extra rows are the ones i flagged on #1930 — their object_id is the subject NDC copied across rather than an RxCUI (mimic-ndc:89141045602 -> rxnorm:89141045602). same story in the nw copy, procedureevents_to_snomed 331 vs procedureevents_to_omop 330, the odd one out being northwestern-itemid:772050. as written, validate PASSES on exactly the state that's wrong, which is the opposite of what you want from a validate script. if those get fixed in #1930 these constants need to move with them, so they're worth a comment saying where they came from.

smaller ones:

• index.sql creates *_idx01 ON (subject_id) for all eight tables, but six of them already have a primary key on subject_id and therefore a unique index on it. those six are pure duplicates — write cost and disk for no read benefit. keep idx01 only on the two prescriptions tables.
• the comment at the top of load.sql says the paths are "relative to this script's location". \COPY resolves a relative filename against psql's working directory, not the script's, so it's really "relative to wherever you ran psql". the readme gets this right by telling people to cd first, but the two disagree, and every other build script in this repo takes the data directory as -v instead. a \cd at the top would make it work from the repo root like buildmimic/postgres/load_gz.sql does.
• the quickstart omits ON_ERROR_STOP=1 on create.sql while setting it everywhere else. minor, but it's the step that drops the schema.

on the 416 duplicate rows: since they differ only in subject_label and the mapping itself is identical, DISTINCT ON (subject_id, object_id) in the load — or moving the label variants out to a synonyms table — would let both prescriptions tables carry the same PK as the other six, and would stop hosp.prescriptions joins fanning out 2-3x for those NDCs. that's a change to #1930's data rather than to these scripts, but this is where it shows up.

and the packaging ask from my earlier note still stands — this branch still carries all of #1930 and #1935, so the twelve files above are the only ones i've reviewed here.