What the wrapper checks around cTAKES
USDHUB00000000000000000001.SYNTHETIC-DEMO.txt enters as a text file. Apache cTAKES, the clinical text-processing system, does the analysis; the USDHUB wrapper does the checking. Before any Java process starts, while cTAKES runs, and after it exits, the wrapper verifies that the filename identity, the source text, the dictionary and model bindings, the committed rows, and the receipts still agree with each other.
Download the finalized 2.1.19 operator MOP as a Word document.
How it works explains the analysis itself; this page keeps only the checks around it. cTAKES marks spans, exact character stretches, over the unchanged text. The local dictionary, the terminology database built from one release of UMLS, the Unified Medical Language System, attaches candidate coded meanings to those spans; its build and activation are on Dictionary and recipes. The recipe, the ordered engine list stored as a small .piper file, decides which engines run and in what order. Word-sense disambiguation, shortened to WSD, scores one span that carries several candidate meanings, and its method and evidence have their own page. Each note then commits as one transaction in SQLite, the single-file database that holds the canonical result, and every delivered file and field is on Outputs and review.
The filename is the note's identity
The name carries two identifiers. The HubID is the literal prefix USDHUB followed by exactly 20 digits, and the NoteID names this note inside that hub. The wrapper parses both before any Java process starts, then writes them beside the note text and every extracted row, so any row can be traced back to one file.
The whole batch is validated at once, and one bad name stops it. If two files normalized to the same HubID and NoteID, two source notes would be written under one identity, and no later check could untangle them, so the gate refuses the batch instead of skipping the file. The gate writes a receipt, a small JSON file recording what was checked, before any note enters cTAKES.
The interactive filename anatomy and the receipt fields are on How it works; the validator command sits in the getting-started run path.
The wrapper checks the batch before, during, and after cTAKES
The wrapper surrounds the cTAKES process because a Java process exiting successfully does not prove that every input identity, source span, output row, and receipt still agrees.
The table below uses the run's own objects. A CUI is a UMLS concept identifier, and an RxCUI is the RxNorm identifier for a normalized drug; every medication row carries the pair. A note hash is its SHA-256 fingerprint, computed from the exact bytes. UTF-16 spans are character offsets counted the way Java counts them. The JDBC asset is the Java driver jar that opens SQLite. CSV is a plain comma-separated table, and XMI is the XML serialization of cTAKES' complete in-memory analysis, written only when explicitly requested. The manifest is the JSON summary whose status must read complete before a batch is handed off.
| When | Checks tied to the note | What happens on failure |
|---|---|---|
| Before Java | Every filename matches USDHUB####################.NoteID.txt; normalized identities are unique; input, output, work, and SQLite paths do not overlap; the recipe, dictionary, WSD model, JDBC asset, capacity plan, and expected output space are available. | The batch stops before a note enters cTAKES. |
| During cTAKES | The effective recipe controls stage order; dictionary and WSD assets remain bound to the run; concept and medication rows retain their CUI/RxCUI pair, source text span, and run identity. | The note transaction rolls back or the run exits incomplete. |
| After Java | SQLite opens cleanly; note hashes and UTF-16 spans match the stored text; candidate identities are atomic; CSV cells equal their SQLite views; XMI hashes are registered when XMI is selected; manifest and database identities agree. | The manifest is marked failed, and the batch is not presented as complete. |
Release 2.1.19 publishes the cTAKES/dictionary runtime and WSD runtime as separate checksum-pinned release assets. The normal bash scripts/setup.sh path downloads and verifies both, checks the WSD model and receipt against the active dictionary, and stages the pinned SQLite JDBC jar before it runs the fictional prerequisite check.
The run starts only when the job fits the available machine
The same server can expose different capacity to different jobs. A container or scheduler may allow four CPUs on a 32-core host, or a WSL environment may have less usable memory and disk than the physical machine. The wrapper plans against the capacity visible to the cTAKES job, not the hardware label. A cgroup is the Linux mechanism that limits the CPUs and memory one job can use, and the Java heap is the memory ceiling the Java process starts with; both terms appear in the probes below.
| Capacity input | How the wrapper uses it |
|---|---|
| CPU affinity and cgroup quota | Caps analysis threads and total async workers at the CPUs the job can actually use. |
| Available memory and cgroup headroom | Sizes the Java heap from memory still available inside the job boundary. |
| Linux commit headroom | Checks CommitLimit - Committed_AS so a heap is not admitted only because free-RAM math looked safe. |
| Input size and selected writers | Estimates persistent and temporary output for SQLite, CSV, XMI, optional browser export, logs, receipts, and async shards. |
| Free space on each filesystem | Checks output, work, and custom SQLite locations separately and leaves the configured reserve. |
A plan with missing probes, insufficient memory, or insufficient disk stops before Java. If an operator supplies a heap size manually, the wrapper compares that value with the same live job boundary before accepting it.
Startup and capacity explains the admission math, cold start, first-note timing, and async-shard plan. An async shard is one parallel child run that owns a partition of the larger batch.
A reviewer can move from a row back to the source phrase
The release, analysis, and run identities that name a completed run are on Outputs and review; every check above binds the batch to those three identifiers.
Each concept and medication row stores a document identity plus begin and end offsets. The built-in TUI viewer is the normal review path. It reads the completed SQLite database, translates the Java/UIMA UTF-16 offsets into the displayed character positions, underlines extracted annotations in the note, and gives the selected row a stronger source-text focus. It also verifies the stored note hash before showing the evidence, so a row cannot quietly point into different text. A self-contained browser HTML export is optional. An assertion is the engine's reading of the language around a span, such as a negation; a relation is a typed connection between two annotations, such as an anatomical location.
The reviewer can check
- whether the selected phrase is the intended clinical mention;
- whether the CUI, RxCUI, assertion, relation, and medication fields fit the sentence;
- whether WSD decided or abstained for a same-span candidate group.
The wrapper records
- the source note and SHA-256 hash;
- the exact UTF-16 offsets and evidence text;
- the release, analysis, run, dictionary, recipe, schema, and WSD identities.
The result stays reviewable because the coded row never loses its link to the original phrase. That same link also lets a site correct local labels and build a training set without sending the source note to another site.