Deliver the two CSVs; keep the audit and review artifacts local

Two routine deliverables, with local evidence behind them

The two CSVs are flat, spreadsheet-friendly projections carrying exactly the requested columns. The SQLite database is the canonical structured result, contains the full source notes, and stays local so the CSV values can be reviewed and checked. XMI is the XML serialization of the CAS, the Common Analysis Structure, cTAKES' complete in-memory analysis object; it is optional and also stays local. The built-in TUI viewer reads the checked SQLite file directly and creates no second copy. JSON receipts, logs, and manifests record what was checked and which bytes ran. These supporting artifacts matter for audit and recovery, but they are not routine delivery files.

Replay figure

Every term below also gets a short gloss at its first use; the clinical glossary collects them all in one lookup table.

SQLite commits one note and its rows as one transaction

The Java writer prepares its statements once, begins one transaction per note, writes the document record, every concept row, every medication row, and the concept counts, then commits. A transaction either commits completely or rolls back completely, so an exception rolls that note back and the database never keeps a partially written document.

On a fresh run, those transactions use insert-only writes and the ten secondary indexes are deferred until ingestion finishes. After Java closes the database, the wrapper verifies the exact run identity and row scope, builds and validates the canonical ten-index set in one transaction, and runs SQLite quick_check before CSV export and final status. Append mode keeps replacement writes and live indexes because it may intentionally update an existing run database.

PRAGMA user_version=8 identifies the schema, the eighth revision of the table layout, recorded inside the database itself. The sanity checker reads the database in run scope, so append mode cannot mix earlier rows into a new delivery view.

Read the transaction and schema: tools/reporting/uima/AnnotationSqliteWriter.java.

The workbook field decisions are visible in the two CSV headers

Column B in the study workbook, the field-selection spreadsheet agreed for delivery, is the outbound selection. A value of 1 includes the field, a blank excludes it, and column C value 0 excludes the legacy Medication* field even when column B contains 1. Applying those decisions leaves 13 concept fields and 16 medication fields. Contract ctakes-fields-v02-selected-r2 also uses CUI/RxCUI consistently and replaces the remaining concept-name colons with underscores so database columns do not need special quoting.

core_Document,HubID,NoteID,CUI,RxCUI,assertion_Polarity,assertion_Uncertainty,assertion_Conditional,assertion_Generic,assertion_Subject,wsd_Disambiguated,wsd_Score,cTAKES_version

Document,HubID,NoteID,CUI,RxCUI,Dose,TotalDose,DosageCount,Route,Frequency,Form,Duration,StatusChange,StartDate,EndDate,cTAKES_version

The standard folder-named handoff is automatic

During the passing sanity check, the controller compares both CSV streams with the accepted SQLite views and records each CSV hash. It then creates one owner-only folder-named copy for each logical CSV and hashes the copy while writing it. Nothing is uploaded or sent.

Check output is the single place that records both mappings. Technical files keeps the fixed run artifacts available for local review and reports AUTOMATIC CSV COPIES READY without repeating the folder-named files. It shows Retry automatic CSV copies only when the deterministic copy needs attention, or Prepare automatic CSV copies for a verified older run without that record. An exceptional extra labeled copy is CLI-only: ./ctakes handoff <job-id> --name <safe-label>. Never put PHI in the label.

The TUI walkthrough shows this step in the complete run sequence.

Blank values still carry meaning

For a blank WSD pair, read run_metadata.wsd_enabled first. A value of 0 means the run used the explicit WSD-off comparison. A value of 1 means WSD ran; a same-span group with more than one distinct CUI and blank WSD cells is an abstention, while a singleton, a span with only one candidate meaning, had nothing to disambiguate. The run metadata prevents those three cases from being collapsed into one undocumented blank.

Read the exact export views and cell comparison: scripts/export_results.py and scripts/check_run.py.

XMI is the native cTAKES analysis, not another CSV representation

XMI is written only when the run is started with --writers standard; the full command sits on the run step of Getting started. Each XMI file serializes the full UIMA CAS (UIMA is the Apache framework cTAKES runs on), including source text, tokens, syntax, concepts, assertions, relations, medication features, and native metadata. It can contain features that the agreed workbook does not request.

xmi_provenance.json records one expected XMI per document, parse success, bytes, SHA-256, release/analysis/run identity, dictionary hashes, pipeline hash, and WSD provenance. The wrapper does not claim that every native XMI feature has a one-to-one CSV field.

Read the XMI register: scripts/write_xmi_provenance.py.

Three identities keep human versioning separate from exact reproducibility

The completed run records three identifiers with different jobs, and each answers a different review question: which software-and-terminology version ran, which exact extraction definition ran, and which single execution produced these files.

UMLS 2026AA is recorded separately in the manifest, the strict inventory naming the expected hashes, and in SQLite. It is also bound to the dictionary through the strict manifest and database hash. Runtime path, thread count, heap, and elapsed time do not change the semantic analysis ID.

The batch reaches complete only after its outputs are read back

Eight receipts cover the batch from the filename gate to the final storage measurement, so a completed run can be audited without rerunning it.

The sanity pass checks the document set, source hashes/UTF-16 spans, nonblank concept and atomic medication identities, literal medication evidence, duplicate rows, exact CSV headers/cells, WSD invariants, XMI receipts, and version agreement.

The handoff condition is literal: manifest status=complete and sanity status=pass. The two read-back commands that check it sit on the acceptance step of Getting started.

Review the SQLite result inside the TUI

From Runs & cTAKES logs, select a completed run and choose Open built-in viewer. It verifies the accepted run receipts before reading annotations.sqlite, then pages through documents, concepts, and medications. Extracted annotations stay underlined in the full note, and selecting a row gives its exact source phrase the stronger focus. Technical files gives each fixed artifact a short plain-language purpose, colors JSON keys and values by type, and opens the fixed CSV and safe SQLite views without an arbitrary SQL prompt.

Optional browser export

python3 viewer/build_viewer.py \
  --sqlite /approved/usdhub/results/annotations.sqlite \
  --output /approved/usdhub/results/clinician_viewer.html

Use this only when a self-contained browser handoff is useful. The builder requires schema 8, manifest complete, sanity pass, matching release/analysis/run IDs, and valid source hashes and spans. It compresses the completed payload, embeds it in one HTML file, and records an integrity hash so changed payload bytes can be detected when the export opens. The file works without a server or network connection.

The embedded example demonstrates the same evidence-review pattern with fictional data. A clinical HTML export contains full source notes and remains under the same controls as SQLite and XMI.

Read the optional browser-export build and validation: viewer/build_viewer.py.

The output format does not make the clinical data less sensitive