Updates concept mappings for MIMIC-IV and NW datasets.

[danamouk]

This pull request updates the existing concept mappings for the chartevents and procedureevents tables, and adds new mapping files for clinical concepts from the labevents and prescriptions tables in MIMIC-IV v2.0, as well as from the Northwestern ICU (NWICU) database.

These mappings align local concept identifiers (itemid, ndc) with standard clinical vocabularies, including LOINC, RxNorm, SNOMED, and OMOP, to support interoperability across ICU datasets.

All mapping files are formatted according to the Simple Standard for Sharing Ontology Mappings (SSSOM) specification @sssom.

[briangow]

@danamouk , thanks for this contribution.

Does this mix Northwestern and MIMIC terms in these concept mappings? If so, would it be possible to separate them, keeping only the MIMIC terms under mimic-code and putting the Northwestern terms in a separate repo?

[danamouk]

thanks @briangow for the suggestion, this is an iteration on the previous concept maps in the repository which already contains both mimic and nw under mimic-code/mimic-iv/concepts/concept_map as discussed with @tompollard , this is just an extension with additional updates for chartevents/procedureevents along with the other mappings for labevents and prescriptions. For reference, the subject-id distinguishes the source either as mimic or nw to comply with sssom format. These mappings will be useful to harmonize mimic and nw datasets.

[tompollard]

@danamouk I have mixed feelings about mixing northwestern and mimic concepts in a single file. I wonder whether it is cleaner to separate them?

[danamouk]

Sounds good, we can go ahead and separate mimic and nw to keep things clean, currently the mappings reside in the following directory mimic-code/mimic-iv/concepts/concept_map, which would seems like the right place for MIMIC mappings, @tompollard is there a specific directory in the mimic-code repo you’d suggest for adding the NWICU concept maps, possibly mimic-code/nwicu/concepts/concept_map ?

[danamouk]

@tompollard, MIMIC mappings are now under mimic-code/mimic-iv/concepts/concept_map, and NWICU concept maps are in mimic-code/nw/concepts/concept_map to keep things consistent.

[danamouk]

hello @tompollard, @briangow — the concept mappings have been updated in this pull request. Is it all set to merge, or are there any remaining changes needed?

[briangow]

Thanks for the update @danamouk , I'm fine with merging this.

[danamouk]

Thanks @briangow for the review! It looks like 3 checks (duckdb, sqlite, psql) were cancelled after timing out waiting
for a runner. I rebased to pick up @alistairewj's SQLFluff update (#1947), which fixed the earlier linting failure. Would
you be able to re-run these remaining checks to make sure everything passes, or confirm it's okay to merge as-is?

[Chessing234]

mapping updates look broad. any breaking renames worth calling out for downstream users?

[Chessing234]

this is two datasets in one PR — 12 files under mimic-iv/ and 8 under nw/. they have different provenance, different reviewers who can actually check them, and different risk: changing an existing MIMIC-IV mapping affects everyone's published results, whereas adding NW mappings for a dataset that has no build scripts on main yet (they're in #1935) affects nobody. splitting would let the MIMIC-IV half move at the speed of whoever can verify it, instead of both halves waiting on the slower one.

on the MIMIC-IV half specifically: the description says it updates existing mappings for chartevents and procedureevents. that's the part that needs the most care, because anything already computed against the old mapping silently changes. it'd help a lot to state, for each updated file, how many rows changed and why — a corrected LOINC code, a newly available concept, or a source-vocabulary version bump are very different things to sign off on.

i can't verify the mappings themselves. i don't have MIMIC-IV or NWICU access, so i can't check any code against the data, and i'd rather say that than imply otherwise. the split and the change summary are what i can usefully ask for.

[Chessing234]

coming back to this one with something more useful than "i can't verify the mappings". i still can't check a mapping against the data, but the files can be checked against each other and against what's on main, and that turns up several things worth fixing before this lands.

four rows in prescriptions_to_rxnorm.csv have an NDC in the object_id, not an RxCUI. these are exactly the four NDCs that appear in the rxnorm file but not in prescriptions_to_omop.csv (3107 vs 3103 rows), which is what led me to them:

mimic-ndc:019481604    -> rxnorm:19481604        Gastroview (Diatrizoate Meglumine & Sodium)
mimic-ndc:89141045602  -> rxnorm:89141045602     Gelclair
mimic-ndc:08290309653  -> rxnorm:8290309653      Syringe (subQ)
mimic-ndc:60258000601  -> rxnorm:36709, 8588     Neutra-Phos

the first three are the subject NDC copied into the object with the leading zero dropped — RxCUIs are at most 7 digits, and 89141045602 is 11. the fourth packs two RxCUIs into one field, which is presumably also why it has no omop counterpart. they are the only four rows in the file whose object_id isn't RxCUI-shaped (^\d{1,7}$), and the nw rxnorm file has none, so this looks like four bad rows rather than a convention i'm misreading.

the prescriptions files contain a lot of duplicate rows on the join key. hosp.prescriptions.ndc joins on subject_id. in prescriptions_to_omop.csv, 416 (subject_id, object_id) pairs occur more than once, and 19 rows are byte-identical duplicates of another row. no NDC in the file maps to more than one distinct object_id, so the repeats carry no mapping information — they're subject_label variants:

mimic-ndc:023916330,CYCLOSPORINE 0.05% OPHTH EMULSION,skos:exactMatch,omop_id:702619,...
mimic-ndc:023916330,Cyclosporine 0.05% Ophth Emulsion,skos:exactMatch,omop_id:702619,...
mimic-ndc:023916330,RESTASIS,skos:exactMatch,omop_id:702619,...

anyone joining prescriptions to this file gets 2-3x the rows for those NDCs. same counts in the rxnorm file, and 13 duplicated pairs in the nw copy. if the label variants are worth keeping, they'd be better as a separate synonyms file or collapsed with the variants in the comment column.

the CURIE prefixes change on files that are already on main. the two mimic-iv files that exist today use mimic:220045; this branch renames them to mimic-itemid:220045, and adds mimic-ndc:, northwestern-itemid:, northwestern-ndc:. i think the new prefixes are better — the old mimic: was ambiguous once NDCs arrived — but it silently breaks anything joining on the old identifier, and there's no curie_map or SSSOM metadata anywhere in the repo declaring what any of these expand to. worth calling out in the body, and worth a prefix declaration given the SSSOM framing.

the column order changes too. both existing files are ...,author_id,reviewer_id,confidence,comment; every file here is ...,author_id,confidence,comment,reviewer_id. header-driven readers are fine, positional ones aren't — and the loader scripts in #2000 are the immediate consumer, so it's worth confirming they read by name.

chartevents_to_omop.csv loses two rows in the move (54 -> 40 mimic rows; 42 of the 12 dropped are the nw rows correctly relocated). the two that aren't accounted for are:

mimic:223763,Bladder Pressure,,,,,,,,No suitable LOINC code
mimic:224166,Doppler BP,,,,,,,,No suitable LOINC code

both survive in chartevents_to_loinc.csv. dropping them from the omop file may well be deliberate — the comment is a LOINC comment sitting in the omop file, so it was wrong there anyway — but as it stands the omop map no longer distinguishes "reviewed, nothing suitable" from "never looked at" for those two itemids. a row with an empty object_id and a no suitable OMOP concept comment would preserve that.

one off-by-one in nw: procedureevents_to_snomed.csv has 331 rows to procedureevents_to_omop.csv's 330. the extra is northwestern-itemid:772050.

separately, the split-into-two-PRs ask from my earlier note still stands, and it applies more strongly now — the four bad rxnorm rows and the 416 duplicates are all on the mimic-iv side, and none of them should hold up the nw files.

(structural checks only: row counts, key uniqueness, identifier shape, cross-file consistency. i have no MIMIC-IV or NWICU access so nothing here is a check of whether a mapping is clinically right, and someone who does have access should still do that pass.)