Every delivered file points back to the same note, run, and release

One output does not replace all of the others

Start with four different views of the same run. SQLite is one queryable database used as the structured source of record. CSV means two flat, spreadsheet-friendly tables with the exact requested columns. XMI is the native XML form of cTAKES' complete analysis object, including details the flat tables do not carry. The clinician viewer is one interactive HTML file that opens a retained row beside its source-note evidence.

Replay figure

Open the clinical glossary for CAS, XMI, UIMA, offsets, and the run identities before reading the field-level contract below.

SQLite commits one note and its rows as one transaction

The Java writer prepares statements once, begins a transaction for the note, writes the document/source record, concept rows, medication rows, and CUI counts, then commits. An exception rolls that note back, so the database does not keep a partially written document.

PRAGMA user_version=6 identifies the schema. The sanity checker reads the database in run scope, so append mode cannot mix earlier rows into a new delivery view.

Read the transaction and schema: tools/reporting/uima/AnnotationSqliteWriter.java.

The workbook field decisions are visible in the two CSV headers

The Concepts sheet requests 26 fields. The Drugs sheet lists 50 possible fields, but eight legacy Medication* value/text pairs are marked zero. Removing those 16 fields leaves 34. The workbook does not define an Allergy/AllergyText pair, so that older extra pair is absent.

core:Document, HubID, NoteID, core:Begin, core:End, core:Text, core:Section, core:CUI, core:RxCUI, core:PreferredText, core:TUI, core:SemanticGroup, core:SemanticTypeLabel, assertion:Polarity, assertion:Uncertainty, assertion:Conditional, assertion:Generic, assertion:Subject, syntax:POS, relations:HasDegree, relations:DegreeIndicator, relations:LocationText, relations:LocationCUI, wsd:Disambiguated, wsd:Score, cTAKES_version

Document, HubID, NoteID, Begin, End, Text, Section, CUI, RxCUI, RxNormName, TUI, SemanticGroup, SemanticTypeLabel, Dose, DoseText, TotalDose, TotalDoseText, DosageCount, DosageCountText, Route, RouteText, Frequency, FrequencyText, Form, FormText, Duration, DurationText, StatusChange, StatusChangeText, StartDate, StartDateText, EndDate, EndDateText, cTAKES_version

Blank values still carry meaning

For a blank WSD pair, read run_metadata.wsd_enabled first. A value of 0 means the run used the explicit WSD-off comparison. A value of 1 means WSD ran; a same-span group with more than one distinct CUI and blank WSD cells is an abstention, while a singleton had nothing to disambiguate. The run metadata prevents those three cases from being collapsed into one undocumented blank.

Read the exact export views and cell comparison: scripts/export_results.py and scripts/check_run.py.

XMI is the native cTAKES analysis, not another CSV representation

bash scripts/run_pipeline.sh \
  --input /approved/usdhub/batch \
  --output /approved/usdhub/results-standard \
  --writers standard

Each XMI file serializes the full UIMA CAS, including source text, tokens, syntax, concepts, assertions, relations, medication features, and native metadata. It can contain features that the agreed workbook does not request.

xmi_provenance.json records one expected XMI per document, parse success, bytes, SHA-256, release/analysis/run identity, dictionary hashes, pipeline hash, and WSD provenance. The wrapper does not claim that every native XMI feature has a one-to-one CSV field.

Read the XMI register: scripts/write_xmi_provenance.py.

Three identities keep human versioning separate from exact reproducibility

UMLS 2026AA is recorded separately in the manifest and SQLite. It is also bound to the dictionary through the strict manifest and database hash. Runtime path, thread count, heap, and elapsed time do not change the semantic analysis ID.

The batch reaches complete only after its outputs are read back

The sanity pass checks the document set, source hashes/UTF-16 spans, nonblank concept and atomic medication identities, literal medication evidence, duplicate rows, exact CSV headers/cells, WSD invariants, XMI receipts, and version agreement.

python3 -m json.tool /approved/usdhub/results/run_manifest.json
python3 -m json.tool /approved/usdhub/results/sanity_report.json

The viewer opens the row inside the full source note

python3 viewer/build_viewer.py \
  --sqlite /approved/usdhub/results/annotations.sqlite \
  --output /approved/usdhub/results/clinician_viewer.html

The builder requires schema 6, manifest complete, sanity pass, matching release/analysis/run IDs, and valid source hashes and spans. It compresses the completed payload, embeds it in one HTML file, and records an integrity hash so changed payload bytes can be detected when the viewer opens. The file works without a server or network connection.

The embedded example has no network permission. A clinical viewer contains full source notes and remains under the same controls as SQLite and XMI.

Read the viewer build and validation: viewer/build_viewer.py.

The output format does not make the clinical data less sensitive