How one clinical note moves through cTAKES
One unchanged clinical note enters as UTF-8 text. Apache cTAKES marks exact character spans, asks the local terminology dictionary for possible coded meanings, adds model attributes, and returns candidate annotations. The USDHUB wrapper commits the note and rows together, projects the two agreed CSVs, and records the recipe, dictionary, WSD model, and run identities needed to trace a delivered cell back to its source characters.
The note we will follow
One hand-authored note runs through this entire article. It contains a symptom, a symptom the patient does not have, a medication plan, and one deliberately ambiguous word. The note, rows, codes, and scores are synthetic fixtures that match the real output schema. The codes are fake on purpose, so the mechanism can be shown without patient data or licensed terminology content. None of it is a cTAKES accuracy or WSD evaluation result.
The file begins as ordinary text. There is no diagnosis column, medication table, or standardized code inside it. Those structures appear only after the software finds a phrase, records exactly where it occurred, and asks a terminology dictionary which clinical meanings could match it.
Synthetic note
USDHUB00000000000000000001.SYNTHETIC-DEMO.txtSYNTHETIC NOTE - no patient data.
Assessment
started yesterday. No .
Plan
Start 10 mg by mouth twice daily for 5 days.
Imaging context
A near the right kidney is described.
Select a highlighted source span. The matching row later in the article selects the same exact characters.
SYNTHETIC-DEMO.txtit crosses seven stations on the way to two CSV filesThe note crosses seven stations
Every station below is a real gate or engine in the wrapper's default run. Green means the station passed the note through, and blue means dictionary candidates were attached and kept. Amber marks the deliberate abstention path. Coral marks the failure that stops a batch. Violet marks the artifacts that carry the note text and stay inside clinical-data controls.
- The file USDHUB00000000000000000001.SYNTHETIC-DEMO.txt enters the batch as plain text; no row, code, or span exists yet.
- The name splits into HubID USDHUB plus exactly 20 digits and NoteID SYNTHETIC-DEMO before any Java process starts.
- A neighbor file named USDHUB123.BAD.txt fails the 20-digit rule, so the wrapper stops the whole batch here; no note gets halfway through on a bad identity.
- Section detection marks Assessment, Plan, and Imaging context, then sentence and token boundaries are added over the unchanged characters.
- Dictionary lookup attaches DEMO-C001 to pain at 46–62, DEMO-C002 to fever at 85–90, DEMO-C003 with DEMO-RX001 to DemoMed at 104–111, and two candidates to mass at 170–174.
- Both mass candidates stay on span 170–174: one scores 0.682 and records Y, the other scores 0.318 and records N; neither row is removed.
- When a score or margin gate fails instead, the WSD fields stay blank on both rows rather than forcing a winner, so the ambiguity stays visible for review.
- "No" writes negative polarity on the fever row, "Right flank" links to the pain row as a location relation, and DemoMed keeps 10 mg, by mouth, twice daily, 5 days, and Start.
- The note text, document identity, concept rows, and medication row commit as one SQLite transaction; a failed write rolls back instead of leaving a partial note that looks complete.
- concepts.csv takes the 13 contracted concept columns from a run-scoped SQLite view; the database keeps the full rows and the stored note, and those stay local.
- drugs.csv takes its 16 contracted medication columns the same way, and both exports are checked back against the views before the batch is marked complete.
- One note in, one checked transaction, two projected CSVs out; each row still names the span characters it came from in the stored note.
The rest of this article walks the same seven stations slowly, one object at a time.
The filename is the note's identity
USDHUB is this workflow's name, and it is also the literal first six characters of every accepted filename. The name carries two identifiers: the HubID, which is USDHUB plus exactly 20 digits, and the NoteID, which names this note inside that hub. The wrapper parses both before any Java process starts, then writes them beside the note text and every extracted row.
The HubID is the literal prefix USDHUB followed by exactly 20 digits.
The whole batch is validated at once, and one bad name stops it. That is deliberate: if two files normalized to the same HubID and NoteID, two source notes would be written under one identity, and no later check could untangle them. The gate writes a receipt, a small JSON file named filename_validation.json, recording the document count, the exact rule any name failed, a hash of the batch listing, and the strict policy that was active.
USDHUB00000000000000000001 / SYNTHETIC-DEMOcTAKES starts marking spans inside the textcTAKES marks spans over the unchanged text
Apache cTAKES does not rewrite the clinician's sentence. It reads the note into memory and adds annotations over the original characters. An annotation is a marked stretch of text, called a span, plus structured fields attached to it. For "Right flank pain," the span is characters 46 through 62; those two offsets let a reviewer return from any later row to the exact words that produced it.
Before any clinical meaning is attached, structure comes first: section detection marks that "Right flank pain" sits under Assessment while "Start DemoMed" sits under Plan, then sentence splitting, tokenization, and part-of-speech tagging add the language boundaries later engines need. The in-memory container holding the note and all of its annotations is the CAS, the Common Analysis Structure. Everything downstream reads and writes this one object.
A marked span is not a clinical conclusion
"No fever" produces a fever span even though the patient has no fever. The span records where the phrase occurred; whether the note affirms, denies, or merely discusses it is separate information that later engines attach. That separation is what lets the row for "fever" carry both the phrase and the evidence that it was negated.
46–62 inside the CASthe dictionary attaches candidate meanings to itThe dictionary attaches candidate coded meanings
"Fever," "febrile," and a supported abbreviation are different letters for the same clinical idea. A terminology is a maintained set of coded clinical concepts and their names; an ontology also records how those concepts relate to each other. UMLS, the Unified Medical Language System, integrates many source terminologies and gives aligned concepts one shared identifier called a CUI, a Concept Unique Identifier. A real CUI is the letter C plus seven digits: the feces concept, for example, is C0015733. The example note uses fabricated codes shaped like DEMO-C002 instead. The fixture rows and codes are made up, and the site does not embed a dictionary extract. The separately labeled reviewed failure on the WSD page names real CUIs.
The dictionary is the local database built from one UMLS release that cTAKES actually searches while it processes the note. When lookup reaches "fever" at span 85–90, it returns the matching candidates: possible coded meanings, not verdicts. For the medication span, the candidate carries an RxCUI, the RxNorm identifier for a normalized drug concept, and every candidate carries a TUI, a semantic-type code saying what kind of meaning it is, such as sign-or-symptom versus body structure.
Dictionary selection changes what cTAKES can find, so the wrapper records exactly which dictionary build was loaded. The active release here is usd-star-2026AA-r1, built from selected UMLS 2026AA sources, audited, and given an immutable identity that the wrapper checks before the first note enters cTAKES. If a concept is absent from the dictionary, nothing downstream can bring it back.
The exact source selection, build, audit, and activation process is on Dictionary and recipes.
DEMO-C002 and DEMO-C003/DEMO-RX001the recipe decides which engines touch them, in what orderThe recipe orders the engines that touch the note
cTAKES is a collection of engines, not one fixed extraction. The recipe is a small file, called a piper, that lists which engines run and in what order. Order changes the result because a later engine can only use annotations that already exist. The default recipe's internal key is star, and the wrapper saves the exact recipe the Java process received as run_pipeline.effective.piper, so the run's definition is a file you can read back.
The USDHUB recipe keeps one explicit path: language structure, dictionary candidates, default WSD, medication recognition, the five selected assertion engines, degree and location relations, and the requested writers. The filename and runtime checks sit outside cTAKES, so they surround this path without becoming clinical annotators themselves.
Medication recognition adds the details around the drug
The medication engine treats "DemoMed 10 mg by mouth twice daily for 5 days" as more than a drug name. It attaches the dose, route, frequency, duration, and the status phrase "Start" to the medication mention, and it keeps the literal evidence text beside each normalized value so a reviewer can see how the field was obtained.
Assertions record how the note talks about a span
An assertion is an engine-assigned reading of the language around a span: polarity, uncertainty, conditional, generic, and subject. "No" flips the fever row's polarity to negative. A blank assertion value means the engine did not assign that attribute; it is never silently filled in as a positive clinical fact.
Relations connect one annotation to another
"Right" and "flank" locate the pain. A location relation carries that connection onto the symptom row, so the reviewer sees which anatomy modified it. The recipe also supports degree relations, where an indicator such as "severe" modifies a clinical mention.
WSD handles two meanings on the same words
Word-sense disambiguation, shortened to WSD, only has work to do when one source span carries multiple candidate meanings. "Mass" can be a body structure in one sentence and a quantitative concept in another, and dictionary lookup attached both candidates to span 170–174. WSD reads the nearby sentence, surrounding concepts, semantic types, preferred terms, and dictionary relationships, then scores the candidates against each other.
It records a decision only when every candidate has a model profile, the leading candidate has direct evidence, the leading raw score reaches its threshold, and the margin over the runner-up is wide enough. Otherwise it abstains and writes nothing. Either way, both candidate rows remain in the result; WSD never deletes a candidate.
Same phrase, same two candidates
Change only the strength of the context evidence
Both rows remain. Complete profiles, direct evidence, score, and margin allow Y and N to be written.
A Y means that candidate won within that one ambiguous span. An N means another retained candidate on the same span won. Blank WSD fields mean there was no multi-candidate decision or the evidence was too weak; they never mean the concept was absent. The abstention is deliberate: forcing a winner when two candidates are close hides uncertainty inside a clean-looking row. In the wrapper 2.1.1 controlled 858-note benchmark, the wrapper decided 3 of 2,437 ambiguous spans and abstained on the rest. Release 2.1.19 keeps the same STAR clinical stage order, so those measurements remain historical benchmark evidence rather than relabeled 2.1.19 results.
Y and N on span 170–174every row commits in one transactionThe note and its rows commit as one transaction
SQLite is a database that lives in a single file, here annotations.sqlite. The writer opens one transaction per note and writes the document identity, the full source text with its SHA-256 fingerprint, every concept row, every medication row, and the concept counts together. A transaction either commits completely or rolls back completely, so a failed write cannot leave a partial note that looks complete. This is what the synthetic note's committed unit actually contains:
| Table | Row for this note | What the fields carry |
|---|---|---|
documents | 1 row | USDHUB00000000000000000001, SYNTHETIC-DEMO, the complete source text, its SHA-256 fingerprint, its UTF-16 length, and the run identity. |
concepts | 4 rows | Pain DEMO-C001 at 46–62 with polarity POS and the right-flank location; fever DEMO-C002 at 85–90 with polarity NEG; mass DEMO-C004A at 170–174 with WSD Y 0.682 and DEMO-C004B with WSD N 0.318. |
rxnorm | 1 row | DemoMed at 104–111: the DEMO-C003/DEMO-RX001 pair plus the literal evidence "10 mg", "by mouth", "twice daily", "5 days", and "Start". |
run_metadata | 1 set | Release, analysis, and run identity, dictionary and WSD hashes, schema version 8. |
POS here is an extractor assertion code meaning not-negated, not proof of a diagnosis. After the batch closes, the wrapper reads the database back and checks the document set, the source hashes, the spans against the stored text, and the row identities. Native XMI, the XML serialization of the complete CAS, is written in parallel only when explicitly requested; it is the largest output and carries the full note too.
annotations.sqliteonly the contracted columns leave as CSVThe CSV projects thirteen columns out of the database
CSV is a plain comma-separated table, and the two delivery files are projections: they take only the workbook-selected columns from run-scoped SQLite views, 13 for concepts and 16 for medications. This is the fever row as it leaves in concepts.csv, under its exact header:
core_Document,HubID,NoteID,CUI,RxCUI,assertion_Polarity,assertion_Uncertainty,assertion_Conditional,assertion_Generic,assertion_Subject,wsd_Disambiguated,wsd_Score,cTAKES_versionUSDHUB00000000000000000001.SYNTHETIC-DEMO,USDHUB00000000000000000001,SYNTHETIC-DEMO,DEMO-C002,,NEG,,,,,,,2.1.19_synthetic-demo_6.0.0core_Document carries the document identity, which is the filename stem, and the cTAKES_version column carries the run's complete composite version: this synthetic fixture records 2.1.19_synthetic-demo_6.0.0, and a real 2.1.19 run records 2.1.19_2026AA.r1_6.0.0, not the bare cTAKES number.
The blanks carry meaning. RxCUI is blank because fever is not a medication. The four assertion fields after polarity are blank because the engines assigned nothing, and nothing was invented to fill them. The two WSD fields are blank because fever had a single candidate, so there was no ambiguity group to decide. On the two mass rows, and only there, wsd_Disambiguated carries Y and N with their scores.
After export, every cell is compared back to the SQLite views it came from, so the delivered files cannot drift from the canonical database. The note text itself is in no CSV column; it stays in the database and the viewer, which is why those two artifacts stay under clinical-data controls while the projected CSVs carry only identifiers and selected values. Outputs and review lists every field of both files.
concepts.csv linethree identities say exactly what produced itThree identities name what produced the row
The completed run records three identifiers with different jobs. The release ID, 2.1.19_2026AA.r1_6.0.0, is the readable software-and-terminology version. The analysis ID, an analysis- prefix with 24 hex digits, is derived from the exact wrapper source, normalized recipe, dictionary database, output schema, writer definition, and WSD state; two runs with the same analysis ID used the same extraction definition. The run ID names one execution with its UTC start time and a unique suffix, so logs and receipts can be tied to exactly one run.
Keeping them separate prevents a timestamp from pretending to be a method version, and prevents a readable version label from hiding a changed recipe or dictionary. Outputs and review carries the full identity table.
The reviewer moves from a row back to the note
The normal review path is Open built-in viewer in the TUI. It reads the accepted SQLite result directly, translates cTAKES' Java/UIMA UTF-16 offsets into displayed character positions, and underlines the extracted annotations in the note. Select a concept or medication row and that source phrase receives the stronger focus beside the structured fields. The embedded fictional browser export below demonstrates the same review loop: the row points to the source characters, the note shows the nearby language, and the field panel shows the assertion or medication attribute attached to that span.
The built-in viewer displays the full note text stored in SQLite. An optional browser export embeds that text in another file. Both stay under the same access controls as the database.
Synthetic browser export
Hand-authored note, fake concept codes, and no patient data.
Matching local extraction comes before federated learning
Site A has its own notes, identifiers, security controls, and computing environment; Site B has different ones. Federated learning is the arrangement where the notes never leave those environments: each site computes locally and only protocol-approved results cross the boundary.
Each site runs the same wrapper analysis locally, so "Right flank pain," negated fever, medication attributes, and any study-specific features are created under the same recorded extraction definition. The source-linked rows remain local by default. A separate study protocol decides whether approved derived features, aggregates, or model updates may leave either site.
run_id Arun_id BOnly the exact payload named by the protocol leaves a site.
A matching analysis_id means the extraction definition matched; a different run_id at each site means each execution keeps its own audit trail. Federated learning still adds machinery this wrapper does not supply: a coordinating service, site authentication, secure exchange or aggregation, failure handling, privacy accounting, and rules for what leaves each site.
The complete study path, the artifacts that stay local, and the parts a federated system still needs are on Multi-site use.
Each stage has its own page
Run one setup command that downloads and verifies the cTAKES/dictionary asset and its separate release-matched WSD asset, then validate filenames, run a batch, and read the receipts.
Look a term upClinical basicsThe same note example in more depth, the assertion and relation fields, and the site-wide glossary.
Inspect terminology choicesDictionary and recipesWhich UMLS sources and semantic types enter the dictionary, how the database is built, and why recipe order changes the result.
Inspect the controlsWrapper detailsFilename validation, the before/during/after checks, capacity admission, and source-linked review.
Plan a real batchStartup and capacityJava startup, models, CPU, memory, commit, storage, and async shards, with the measured timings.
Review a completed resultOutputs and reviewThe SQLite schema, every CSV field, the XMI role, version identities, sanity receipts, and handling rules.