What cTAKES can find depends on the dictionary it actually loads
The raw UMLS release supplies terminology source tables. A controlled build filters those tables into the local dictionary cTAKES actually searches. A separate WSD model describes the candidates in that exact dictionary. The run binds all three identities, because a release label by itself cannot prove which terms, database, or model bytes produced a row.
UMLS release, local dictionary, and WSD model answer different questions
Apache cTAKES, the clinical text-processing system, does not search the raw UMLS release while it reads a note. UMLS, the Unified Medical Language System, integrates many source terminologies; a controlled build filters one UMLS release into the local dictionary that lookup actually queries. Lookup returns candidates as coded identifiers: a CUI, the UMLS Concept Unique Identifier for one clinical concept; an RxCUI, the RxNorm identifier for a normalized drug; and a TUI, a semantic-type code saying what kind of meaning a candidate is. The WSD model is a third artifact, a SQLite file (a database that lives in a single file) built from that exact dictionary.
| Object | What it contains | What the run uses it for |
|---|---|---|
| UMLS 2026AA META files | The source Metathesaurus tables such as MRCONSO, MRSAB, MRSTY, MRDEF, MRRANK, and MRREL. | Input to controlled dictionary and WSD builds. The raw release is not queried for every note. |
| cTAKES dictionary database | The selected terms, CUIs, semantic types, preferred terms, and RxNorm source-code table in cTAKES HSQL shape. | Fast phrase lookup while cTAKES processes the note. |
| Graph/context WSD SQLite model | Profiles derived from the exact dictionary candidates plus ranked UMLS terms, definitions, types, and bounded relations. | Compare same-span candidates after lookup has created them. |
If a concept is absent from the cTAKES dictionary, WSD cannot bring it back later. WSD only compares the candidates that dictionary lookup already attached to the span.
Release 2.1.19 packages these runtime objects separately. The large cTAKES archive carries cTAKES 6.0.0 and the approved dictionary. CtakesWSD-runtime_usd-star-2026AA-r1.tgz carries the exact WSD model, term-free build receipt, distribution review, and third-party notices. Normal setup downloads and verifies both assets and checks their shared dictionary identity before staging them.
The same dictionary database hash is bound into the run and into the WSD model receipt, a small JSON file recording what was checked and which bytes ran. How it works draws the phrase-to-candidate flow on the synthetic note.
The source files used to build the dictionary
| Build term | Plain-language meaning |
|---|---|
MRCONSO.RRF | The UMLS table of concept names and source terms. A row connects wording, a CUI, a source vocabulary, a language, and a term type. |
MRSAB.RRF | The UMLS source-vocabulary inventory. The build reads it to prove which SNOMED CT, RxNorm, and LOINC releases are current and included. |
SAB | Source abbreviation on a terminology row, such as SNOMEDCT_US, RXNORM, or LNC. |
TTY | Term type: the source vocabulary's label for the kind of name, such as preferred term, synonym, ingredient, or official LOINC name. |
HSQL | HSQLDB, the Java relational-database format used by this cTAKES dictionary. |
MEMORY | An HSQL table type whose contents are materialized in Java heap when the database opens. |
CACHED | An HSQL table type that keeps persisted rows on disk and uses a bounded cache plus indexes instead of loading the complete table into heap. |
Source selection happens before cTAKES DictionaryBuilder runs
The preparation step filters MRCONSO by language, source, source-specific term type, suppression, and allowed semantic types first. The builder’s code-table argument is a separate choice, so the workflow passes RXNORM as the source-code table cTAKES should create.
| Selection | Meaning | Reason |
|---|---|---|
| English rows | MRCONSO language is English. | The source notes and downstream models are English. |
| Selected SAB | SNOMEDCT_US, RXNORM, and LNC. | Clinical concepts, medication identities, and LOINC concepts requested for this release. |
| Source-specific TTY | Term types are chosen separately for each source rather than applying one generic list blindly. | Each vocabulary uses its own term-type conventions. |
| Suppression filter | Suppressed terms are excluded according to the release profile. | Prevents known-retired or suppressed strings from entering routine lookup. |
| Semantic-type filter | Allowed TUIs constrain the clinical domain included in the dictionary. | Reduces unsupported or irrelevant candidate classes before runtime. |
Code target RXNORM | Only the RxNorm source-code table is requested. | The delivery needs CUI/RxCUI. Unused SNOMED and LOINC source-code tables would add size without a consuming output field. |
Exact 2026AA source profile used for this candidate release
UMLS source (SAB) | Included term types (TTY) | What those labels mean |
|---|---|---|
SNOMEDCT_US | PT,SY | Designated preferred names and designated synonyms. |
RXNORM | BN,IN,MIN,PIN | Brand name, ingredient, multi-ingredient name, and precise ingredient name. |
LNC | LC,LN,OSN | Long common name, LOINC official fully specified name, and official short name. |
| Suppression | SUPPRESS=N | Only rows marked not suppressed are retained. |
| Code-table target | RXNORM | The builder persists the RxNorm source-code lookup needed for RxCUI output; this is not another term-type list. |
Allowed semantic types (TUI). T017, T018, T019, T020, T021, T022, T023, T024, T025, T026, T029, T030, T031, T033, T034, T037, T046, T047, T048, T049, T059, T060, T061, T074, T079, T080, T081, T082, T109, T110, T111, T114, T115, T116, T118, T119, T120, T121, T122, T123, T124, T125, T126, T127, T130, T184, T190, T191.
This is the literal configured profile, including configured TUIs that happened to have no selected rows in this release. The source inventory records that distinction instead of silently rewriting the profile after the build.
Prove the source release, not only the folder name
The builder reads MRSAB and accepts one current row per selected source where both CURVER=Y and SABIN=Y. A missing row or more than one accepted current row fails the build. That prevents a historical source row from being recorded as current.
Read the mechanism: scripts/prepare_dictionary_source.py applies the source filters, and the content-addressed release manifest (the JSON record naming the release's expected files, hashes, filters, and counts) is dictionaries/usd-star-2026AA-r1/manifest.json.
The build creates a candidate release without changing the active runtime
bash scripts/build_dictionary_full.sh \
--umls /path/to/umls-2026AA/2026AA/META \
--ctakes-home /approved/apache-ctakes-6.0.0- Validate the required META files and record their hashes without copying licensed terminology into the receipt.
- Prepare a filtered dictionary source tree using the reviewed source, term-type, suppression, and semantic-type rules.
- Run cTAKES DictionaryBuilder in an isolated candidate directory.
- Open the completed HSQL database and audit table presence, counts, identities, and indexes.
- Convert startup-loaded MEMORY tables to indexed CACHED storage.
- Export a deterministic WSD seed from that exact validated database.
- Write build, validation, inventory, storage-optimization, and strict manifest receipts.
- Write
activate_command.txtonly when every required phase passes.
Read the mechanism: scripts/build_dictionary_full.sh, tools/dictionary/DictionaryDatabaseAudit.java, and tools/dictionary/DictionaryStorageOptimizer.java.
The accepted database has measured contents and storage behavior
The audit phase reopens the finished database and records what is inside it. SHA-256, a fingerprint of the exact bytes, names one specific database file, so the shortened hash below can be matched against the manifest and the activation receipt.
- CUI_TERMS rows
- 469,083
- TUI rows
- 319,935
- PREFTERM rows
- 271,611
- RXNORM rows
- 26,015
- Database SHA-256
5a5e7c8d…e581- Rare lookup p50 / p95
- 54 µs / 1,577 µs
The lookup values are database microbenchmarks, not end-to-end note speed. They measure selected queries against the built database. JVM startup, model initialization, note length, annotation count, writer profile, and host load remain separate.
CACHED tables keep the dictionary out of Java heap
Large HSQL MEMORY tables are materialized into heap when the database opens. Indexed CACHED tables keep persisted data on disk with cached access and explicit indexes. The optimizer converts only the tables recorded as safe for this dictionary layout, reopens the database, reruns the audit, and records the resulting database hash.
The WSD seed is exported after this validation from the exact database that will be activated. That closes the gap where a WSD model could be built from an earlier dictionary script while cTAKES later loaded different bytes.
Activation is a separate, atomic operation
Activation requires cTAKES to be stopped. The script verifies the candidate manifest and database again, stages the new descriptor/database, records the prior active selection for rollback, and moves the active pointer only after the candidate is in place.
| Activation receipt | Why it is recorded |
|---|---|
| Candidate manifest hash | Proves which reviewed candidate was selected. |
| Dictionary database hash | Proves the runtime bytes match the candidate manifest. |
| Descriptor paths/hashes | Proves which XML points cTAKES to those bytes. |
| Prior active release | Provides a concrete rollback target. |
| Activation time and status | Separates build completion from the moment the runtime selection changed. |
The current literal dictionary release is usd-star-2026AA-r1. It remains in the hashed, content-addressed manifest and composite release ID. The manifest is not digitally signed. USDHUB is the workflow name; renaming this dictionary artifact requires a new manifest, activation receipt, WSD binding, analysis identity, and acceptance run.
Read the activation mechanism: scripts/activate_dictionary.sh and dictionaries/active_release.json.
The recipe decides which engines consume that dictionary output
A dictionary and a recipe are not interchangeable. The dictionary says which phrase-to-concept candidates exist. The recipe is the ordered list of cTAKES components for the run, stored as a small .piper file, and it says when lookup runs, which later models add attributes or relations, and which writers serialize the result. The reviewed stage order is stored at pipelines/production/star.piper; the internal key remains star.
| Recipe choice | Current default |
|---|---|
| Text structure | Stock section, sentence, token, POS, and chunk components; fallback segment only when no section exists. |
| Lookup | Selected 2026AA dictionary release. |
| WSD | Candidate-preserving graph/context scorer with validated active model; --no-wsd remains the explicit comparison override. |
| Medication | Stock Drug NER through a synchronized wrapper, before assertion models. |
| Assertions | Only the five requested stock ClearTK models: polarity, uncertainty, conditional, generic, subject. |
| Relations | Stock degree and location models; self-links removed at serialization. |
| CAS writers | Canonical SQLite by default; native XMI is added when --writers standard is selected. |
| Post-run export | The checked concept and drug CSV files are streamed from completed SQLite views. |
How it works walks this order on the synthetic note, and the WSD page follows the dictionary candidates into the default WSD step. The recipe source is pipelines/production/star.piper.
The manifest and review receipt bind the distribution decision to exact bytes
The manifest records the selected sources, versions, file and database hashes, filters, counts, build phases, and required notice entries. The project owner's distribution review names the same dictionary release, UMLS release, database hash, and package scope. Its basis records direct approval for these files and the prior ctakes-usd-2026AA-product-scope-v145 runtime release as the established HSQL dictionary packaging precedent.
The current wrapper release ID is 2.1.19_2026AA.r1_6.0.0. Its exact Codeberg source tag, Codeberg release, main runtime archive, multipart manifest, WSD archive, and checksums are published. The dictionary archive remains CtakesBun-bundle_ctakes-6.0.0_dict-usd-star-2026AA-r1.tgz; wrapper 2.1.19 does not change its dictionary bytes or STAR clinical stage order.
The repository now carries its Apache-2.0 license, NOTICE, and approved public-release review. Release 2.1.19 passed the complete-repository publication gate; clinical production approval remains a separate decision.