How one clinical note moves through the wrapper
USDHUB00000000000000000001.SYNTHETIC-DEMO.txt enters as a text file. cTAKES marks phrases in that text and attaches coded candidates. The wrapper keeps the filename, source text, character offsets, dictionary, recipe, output rows, and run receipts connected until a reviewer can open the same phrase again.
The filename stays attached to the note
The name USDHUB00000000000000000001.SYNTHETIC-DEMO.txt carries two identifiers. USDHUB00000000000000000001 is the literal USDHUB prefix plus a 20-digit hub identifier. SYNTHETIC-DEMO is the note identifier inside that hub. The wrapper validates both before cTAKES starts, then writes them beside the note text and every extracted row.
If two files normalize to the same hub and note identifiers, the full batch stops. That prevents two source notes from being written under one identity.
cTAKES marks text and attaches clinical information
The words in the note remain available after extraction. cTAKES reads them into a Common Analysis Structure, usually called the CAS, then attaches structured information to exact character positions. One annotation can mark characters 85 through 90 as the fever phrase, another can attach a candidate concept to that span, and an assertion can record that the preceding “No” negates it.
Source note
USDHUB00000000000000000001.SYNTHETIC-DEMO.txtSYNTHETIC NOTE - no patient data.
Assessment
Right flank pain started yesterday. No fever.
Plan
Start DemoMed 10 mg by mouth twice daily for 5 days.
Imaging context
A mass near the right kidney is described.
The concept span points back to the literal text fever. Its character offsets identify the same phrase after cTAKES attaches a CUI, while the assertion fields record that the note negates it.
| Object in the CAS | What it carries |
|---|---|
| Source text | The complete note exactly as cTAKES received it. |
| Span | The begin and end character offsets for a phrase in that source text. |
| Concept candidate | A possible coded meaning such as a UMLS CUI, semantic type, preferred term, or ontology code. |
| Medication mention | The drug phrase plus cTAKES fields such as dose, route, frequency, form, duration, and RxNorm identity when available. |
| Assertion | Context such as negation, uncertainty, conditional use, generic use, and subject. |
| Relation | A typed connection between annotations, including degree and anatomical location in this recipe. |
The terminology supplies possible coded meanings
One note says “renal calculus” and another says “kidney stone.” A selected terminology can map both phrases to the same candidate identity. An ontology also defines relationships among concepts. UMLS integrates concepts from many source vocabularies under CUIs while preserving their source codes, and RxNorm is one source used for medication identity. A returned mapping remains a candidate, not a guaranteed reading of the sentence.
The runtime dictionary is the searchable part cTAKES actually loads. It contains selected terms, codes, semantic types, and source rows from one UMLS release. When cTAKES sees a phrase in the note, it queries that dictionary and adds the supported candidate concepts to the same text span.
The dictionary can do
- match a source phrase to one or more coded candidates;
- retain the vocabulary source and semantic type for each candidate;
- make the same selected terminology available at each participating site.
The dictionary cannot do alone
- decide which same-span meaning fits the surrounding sentence;
- decide whether the note denies or questions the concept;
- prove that the extracted candidate is clinically correct.
The wrapper activates one versioned dictionary release and records its manifest and database hashes. A run cannot quietly load a different set of terminology rows under the same analysis identity.
The recipe controls the order of analysis
The note does not pass through every component available in cTAKES. The effective recipe lists the components used for this run and their order. That order changes the result because later components use annotations created by earlier components.
- Structure stages divide the note into sections, sentences, tokens, parts of speech, and chunks.
- The dictionary stage adds coded concept candidates to matching spans.
- WSD examines spans that have more than one distinct CUI candidate.
- Drug NER adds medication mentions and their structured attributes.
- Five assertion models add negation, uncertainty, conditional, generic, and subject context.
- Relation stages connect supported degree and anatomical-location annotations.
- The selected writers serialize the completed CAS.
The wrapper builds an effective recipe for the run and saves it as run_pipeline.effective.piper. That file records the stage order cTAKES actually received.
Open the full recipe with the input and output expected at each stage.
The wrapper checks the batch before, during, and after cTAKES
The wrapper surrounds the cTAKES process because a Java process exiting successfully does not prove that every input identity, source span, output row, and receipt still agrees.
| When | Checks tied to the note | What happens on failure |
|---|---|---|
| Before Java | Every filename matches USDHUB####################.NoteID.txt; normalized identities are unique; input, output, work, and SQLite paths do not overlap; the recipe, dictionary, WSD model, JDBC asset, capacity plan, and expected output space are available. | The batch stops before a note enters cTAKES. |
| During cTAKES | The effective recipe controls stage order; dictionary and WSD assets remain bound to the run; concept and medication rows retain their CUI/RxCUI pair, source text span, and run identity. | The note transaction rolls back or the run exits incomplete. |
| After Java | SQLite opens cleanly; note hashes and UTF-16 spans match the stored text; candidate identities are atomic; CSV cells equal their SQLite views; XMI hashes are registered when XMI is selected; manifest and database identities agree. | The manifest is marked failed, and the batch is not presented as complete. |
WSD keeps every candidate while it ranks the same-span meanings
Word-sense disambiguation runs when one text span has more than one distinct CUI candidate. It compares the words around that span with the candidate names, definitions, semantic types, and dictionary relations stored in the versioned WSD model. A decision requires complete candidate profiles, direct lexical, graph, or preferred-term evidence, a sufficient raw score, and a sufficient normalized margin.
| Candidate group | Stored result |
|---|---|
| One distinct CUI | The row stays in SQLite and both WSD fields stay blank because there was nothing to disambiguate. |
| Several CUIs with enough evidence and separation | Every row stays. One receives Y, the others receive N, and the scores sum to one inside that span group. |
| Several CUIs with weak or closely tied evidence | Every row stays and all WSD fields stay blank. That is an abstention, so the uncertainty remains visible. |
The WSD model is generated from the same validated dictionary release that cTAKES queries. The wrapper checks the model hash, build receipt, profile, and dictionary binding before Java starts. WSD is on for the production recipe, and a missing or mismatched model stops the run.
WSD shows the candidate scores, decision gates, abstention path, model receipt, and review plan.
The source note and its structured rows commit together
The SQLite writer records run metadata when it initializes. It then opens one transaction for each note and writes the document identity, full source text, source hash, concept candidates, medication rows, and CUI counts together. If that note write fails, the note and its partial rows roll back together.
| File | Where it comes from | What it is for |
|---|---|---|
annotations.sqlite | Written from the completed CAS, one note transaction at a time. | The canonical structured result, source text, run metadata, and row-level evidence. |
concepts.csv and drugs.csv | Streamed from run-scoped SQLite views after the database closes cleanly. | The reviewed 26-field concept and 34-field drug delivery tables. |
| XMI files | Written directly from the completed cTAKES CAS when standard output is selected. | The native serialized analysis structure for tools that consume CAS/XMI. |
| Clinician viewer HTML | Built from a completed and checked SQLite database. | Offline review of each row beside the exact phrase and full source note. |
Outputs and review lists every delivered file, field, receipt, and read-back check.
The run starts only when the job fits the available machine
The same server can expose different capacity to different jobs. A container or scheduler may allow four CPUs on a 32-core host, or a WSL environment may have less usable memory and disk than the physical machine. The wrapper plans against the capacity visible to the cTAKES job.
| Capacity input | How the wrapper uses it |
|---|---|
| CPU affinity and cgroup quota | Caps analysis threads and total async workers at the CPUs the job can actually use. |
| Available memory and cgroup headroom | Sizes the Java heap from memory still available inside the job boundary. |
| Linux commit headroom | Checks CommitLimit - Committed_AS so a heap is not admitted only because free-RAM math looked safe. |
| Input size and selected writers | Estimates persistent and temporary output for SQLite, CSV, XMI, viewer, logs, receipts, and async shards. |
| Free space on each filesystem | Checks output, work, and custom SQLite locations separately and leaves the configured reserve. |
A plan with missing probes, insufficient memory, or insufficient disk stops before Java. If an operator supplies a heap size manually, the wrapper compares that value with the same live job boundary before accepting it.
Three identifiers answer three different questions
The CSV row needs a readable software release, two hospitals need to know whether they ran the same extraction definition, and an operator needs to find one execution in the logs. Those are three different identities.
| Identifier | What it binds | Question it answers |
|---|---|---|
release_id | Wrapper version + dictionary release + cTAKES version | Which named software and terminology release produced this row? |
analysis_id | Release + exact wrapper source + normalized recipe + dictionary database + SQLite schema + WSD model or off state | Did these runs use the same extraction definition? |
run_id | One UTC start time and one unique execution nonce | Which logs, files, rows, and receipts belong to this execution? |
Two sites can have the same analysis_id and different run_id values. That means they used the same extraction definition in two separate executions. A recipe, dictionary, WSD model, schema, or wrapper-source change produces a new analysis_id.
The filenames, full source note, output rows, effective recipe, dictionary manifest, WSD receipt, timing report, storage report, sanity report, and viewer all carry or connect back to these identifiers.
A reviewer can move from a row back to the source phrase
Each concept and medication row stores a document identity plus begin and end offsets. The viewer uses those offsets to select the phrase in the full note. It also verifies the stored note hash before showing the evidence, so a row cannot quietly point into different text.
The reviewer can check
- whether the selected phrase is the intended clinical mention;
- whether the CUI, RxCUI, assertion, relation, and medication fields fit the sentence;
- whether WSD decided or abstained for a same-span candidate group.
The wrapper records
- the source note and SHA-256 hash;
- the exact UTF-16 offsets and evidence text;
- the release, analysis, run, dictionary, recipe, schema, and WSD identities.
The result stays reviewable because the coded row never loses its link to the original phrase. That same link also lets a site correct local labels and build a training set without sending the source note to another site.