Release 2.1.19Split trust boundaryControlled internal test

Install cTAKES USDHUB

The installation path has three separate proofs. Setup verifies the cTAKES 6 runtime, 2026AA dictionary, WSD model, and SQLite writer. The included 100-note run proves that the complete default pipeline produces the expected fictional result. A clinical batch starts only after both proofs pass, and ctakes tui keeps the preflight, extraction, finalization, and handoff states visible.

Clone, verify, then open the TUI

1. Clone without running repository code

git clone --branch main --depth 1 https://codeberg.org/editnori/Ctakes_USD.git Ctakes_USD

2. Verify the local checkout

This local Git check requires the exact remote, the main branch, and a clean working tree. A pass prints Verified clean main checkout: and the commit ID. If it prints nothing or exits with an error, stop.

test "$(git -C Ctakes_USD remote get-url origin)" = "https://codeberg.org/editnori/Ctakes_USD.git" && test "$(git -C Ctakes_USD branch --show-current)" = "main" && test -z "$(git -C Ctakes_USD status --porcelain)" && printf 'Verified clean main checkout: ' && git -C Ctakes_USD rev-parse --short=12 HEAD

3. Run setup from the verified checkout

cd Ctakes_USD && bash scripts/setup.sh --tui && ./ctakes verify && ./ctakes tui

LaterRun ctakes tui from any directory. If a newly installed shell has not refreshed its PATH yet, run ./ctakes tui from the repository folder.

Install without the TUIUse the shell commands only

Run the same clone and local verification steps above. Then use this command when the machine does not need the terminal app. It installs and verifies the same runtime, dictionary, WSD, and output writers.

cd Ctakes_USD && bash scripts/setup.sh --no-tui && ./ctakes verify

This boundary prevents the official site from substituting clipboard contents and prevents a single browser-provided command from cloning and immediately running repository code. It does not make a compromised source repository trustworthy.

Host runtime: Java 17+; Python 3.9+ for TUI
cTAKES: 6.0.0
Dictionary: UMLS 2026AA release
Default WSD: On, receipt-validated model
Default writer: SQLite + two CSVs
Network during run: Not required

What setup prepares before notes arrive

Use an approved encrypted clinical-data volume and a restricted service account. The setup command above stages cTAKES, SQLite JDBC, the active dictionary, and its release-matched WSD model while the input folder is still empty. SQLite is the single-file database the wrapper writes the canonical result into; the JDBC jar is the Java driver that lets the writer open it.

Item	Why the wrapper needs it	What proves the artifact matches
Java 17	Runs cTAKES 6 and the wrapper Java helpers.	Java version check plus the archive SHA-256 (a fingerprint of the exact bytes) when portable Java is used.
cTAKES 6.0.0 bundle	Supplies Apache engines, models, UIMA, and type systems.	Verified bundle receipt and archive SHA-256.
Active dictionary	Turns note text into candidate concept codes: UMLS CUIs and RxNorm RxCUIs.	Strict manifest naming the expected descriptor hash, database hash, UMLS release, and source inventory.
WSD model and receipt	Scores one text span that carries several possible dictionary meanings, while preserving all candidates.	Model SHA-256, build receipt, exact dictionary binding, SQLite quick check, and validation receipt.
SQLite JDBC 3.45.3.0	Lets the Java writer commit the canonical database.	Pinned jar SHA-256 in the offline kit or local runtime setup.

Setup, run checking, extraction, and output checking are different states

State	What is happening	What the state permits
Setup	Downloads or stages the approved runtime assets and verifies their hashes and receipts. It does not process clinical notes.	Run the fictional known-answer test after setup passes.
Sample testing	Processes the included 100 fictional notes through the same default recipe, writers, WSD path, and final read-back used by a clinical run.	Use a clinical note folder only after the known-answer result passes.
Check run	Inventories the selected batch once, validates every filename, and calculates CPU, memory, heap, shard, and storage admission. It does not create clinical extraction output.	Start remains disabled until the displayed plan passes every required gate.
Extracting	The background controller runs cTAKES workers and commits each completed note and its rows into private shard databases.	Monitor or leave after the background-start handshake. Do not hand off partial files.
Finalizing	The controller merges validated shards, builds the final SQLite indexes, exports both CSVs, compares them back to SQLite, and writes the final receipts.	A full note count is still not an accepted delivery.
Passed	The manifest is complete, the sanity report passes, and the two CSV byte streams match the accepted SQLite views.	Use Check output to read the exact two folder-named files to send.

A receipt is a small JSON file that records what was checked and which bytes ran. Every gate below writes one, so a completed batch can be audited without rerunning it.

Understand what setup checks

Latest wrapper and exact runtime. The clone command follows Codeberg's default main branch so wrapper and TUI fixes do not require a new install line. Setup still resolves and verifies the exact Codeberg release files published under 2.1.19_2026AA.r1_6.0.0. Use that versioned source tag when a checkout must remain pinned.

setup.sh resolves one release ID, then downloads the matching WSD asset and the cTAKES 6.0.0 plus 2026AA dictionary bundle and verifies both. It stages the pinned SQLite JDBC driver, writes repository-relative runtime paths, and runs a fictional no-write flight check.

For the TUI, setup uses the newest supported Python already installed on the machine, creates a private environment, and adds the ctakes launcher to the user PATH. It does not replace the system Python.

The runtime archive is CtakesBun-bundle_ctakes-6.0.0_dict-usd-star-2026AA-r1.tgz. Codeberg carries it as numbered parts with a parts manifest and the full-archive checksum. Setup validates every part and the rebuilt archive before extraction.

Stage Java 17 when the host does not provide it

JAVA_ARCHIVE=/approved/usdhub/runtime/temurin-17-jdk.tar.gz
JAVA_SHA256="$(awk '{print $1}' "${JAVA_ARCHIVE}.sha256")"
bash scripts/get_java.sh --from-file "$JAVA_ARCHIVE" --sha256 "$JAVA_SHA256"
export CTAKES_JAVA_HOME="$PWD/.ctakes_deps/java/current"
bash scripts/flight_check.sh

WSD is automatic and still separately verified

Release 2 publishes the WSD model and term-free receipt as their own checksum-pinned asset. Setup downloads that asset and validates its model hash, receipt, SQLite integrity, dictionary binding, and distribution review before installing owner-only copies under .ctakes_deps/wsd.

Included in Release 2 does not mean clinically approved. WSD stays candidate-preserving and on by default, but its clinical status remains not-evaluated. Do not use its winner flag as a clinical decision.

flight_check.sh always creates its own synthetic, filename-valid no-write input. A pass reaches the real dictionary manifest and database, WSD model and receipt, SQLite JDBC, release identity, CPU, memory, Linux commit, and storage gates without starting the cTAKES pipeline or processing a clinical note.

Run the complete known-answer test

./ctakes verify

This runs the actual default path on 100 included original fictional notes: strict USDHUB filenames, the STAR recipe, receipt-validated WSD, SQLite, concepts.csv, and drugs.csv. It then compares the result with the release's exact known-answer manifest.

Continue only after this exact line appears. PASS: cTAKES USDHUB is ready for a full batch.

./ctakes verify and Sample testing in the TUI automatically use samples/synthetic_usdhub_100/ and release/validation/synthetic100_star_semantic_manifest.json.gz. There is no sample folder or manifest to select. The generator, every note, and the answer manifest are checksum-bound in synthetic100_fixture_inventory.json. This exercises the real software path without distributing patient data. An authorized site can separately run its established local MIMIC-IV-Note fixture and reviewed answer file inside its credentialed environment. Those site-held files are not copied into this release.

Read the result literally

Flight-check result	Action
Pass	The local prerequisites are internally consistent. Continue to the synthetic or approved batch.
Dictionary manifest/hash failure	Do not run. Activate the controlled-test dictionary candidate or correct the selected descriptor.
WSD model/receipt failure	Do not bypass validation. Restore the model built for that exact dictionary, or run explicitly with `--no-wsd` only for an approved comparison.
Capacity probe failure	Fix the CPU, memory, commit, or disk probe. Do not substitute total host RAM for available job memory.

Prepare one text file per note

The filename carries the site HubID and NoteID used throughout SQLite, CSV, XMI receipts, and the viewer.

USDHUB12345678901234567890.987654321.txt

The HubID is the literal prefix USDHUB followed by exactly 20 digits.

Encode every note as UTF-8.
Keep the source line endings. The offsets and evidence strings are checked against those exact decoded characters.
NoteID may use letters, digits, periods, underscores, or hyphens and must begin with a letter or digit.
Nested folders are allowed. Duplicate HubID/NoteID pairs anywhere in the batch are not.

Keep the Phase I metadata CSV separate

USDHubID,NoteID,NoteDate,NoteType,NoteTitle,VisitOccurrenceID,ProviderID

The wrapper checks the note filename contract. It does not join this site metadata CSV into concept or medication extraction.

Optional: validate names without starting Java

python3 scripts/validate_filenames.py \
  --input /approved/usdhub/batch \
  --policy strict \
  --json-out /approved/usdhub/filename_validation.json

Use this when a site wants an early filename receipt before the processing window. The real run repeats the same strict check, so this is not an extra required step. One malformed name, empty NoteID, non-portable character, case-normalized collision, or duplicate HubID/NoteID pair fails the batch before cTAKES starts.

Receipt field	Why it is kept
Document count	Proves how many note files passed the gate.
Error summary	Names the exact filename rule that failed.
Relative-path hash	Detects a changed batch listing between validation and handoff.
Policy	Shows that strict USDHUB validation was active.

Run the controlled-test recipe

A recipe is the ordered list of cTAKES components for the run, stored as a small .piper file; the default recipe's internal key is star, retained from wrapper 2.1.0 engineering acceptance. The default run loads the active dictionary's receipt-validated WSD model, preserves every terminology candidate, and writes schema-8 SQLite (the eighth revision of the table layout, recorded inside the database) plus both delivery CSVs. Release 2.1.19 keeps the clinical stage order. Medication attributes without concrete source evidence stay out of the delivery CSV.

Default

bash scripts/run_pipeline.sh \
  --input /approved/usdhub/batch \
  --output /approved/usdhub/results

Writes annotations.sqlite, concepts.csv, drugs.csv, and the validation, version, timing, and storage receipts.

bash scripts/run_async.sh \
  --input /approved/usdhub/batch \
  --output /approved/usdhub/results \
  --flatten-output

Validates once, plans shards from CPU, memory, commit, and disk, runs child SQLite databases, then merges and rechecks one aggregate result.

Run the explicit WSD-off comparison

bash scripts/run_pipeline.sh \
  --input /approved/usdhub/batch \
  --output /approved/usdhub/results-wsd-off \
  --no-wsd

Every distinct same-span CUI still remains. Only the WSD marker and score columns are blank. This produces a different analysis_id, the recorded identity of the exact extraction definition.

Advanced comparison: add native XMI

bash scripts/run_pipeline.sh \
  --input /approved/usdhub/batch \
  --output /approved/usdhub/results-standard \
  --writers standard

This adds one native XMI file per note plus xmi_provenance.json. XMI includes the full source note and normally uses the most disk. It is not part of the default delivery.

What the wrapper does before the first note

Rechecks the filename batch and input/output path boundaries.
Verifies wrapper, cTAKES, UMLS, dictionary, and SQLite JDBC identities.
Validates the active dictionary database and WSD model/receipt binding.
Calculates safe threads, heap, shard count, and storage admission.
Builds the exact effective piper and temporary runtime dictionary descriptor.
Starts Java and records startup separately from note processing.

The startup page explains each phase, the measured timings, and the repeated WSD database-check correction.

Accept the batch only after the read-back passes

python3 -m json.tool /approved/usdhub/results/run_manifest.json
python3 -m json.tool /approved/usdhub/results/sanity_report.json

The handoff condition is literal: manifest status complete and sanity status pass.

Check	What is read back
SQLite integrity and schema	`quick_check` (SQLite's own corruption scan), schema 8, run metadata, exact document set, and committed rows.
Source evidence	UTF-8 source text, Java/UIMA UTF-16 offsets, text spans, source length, and SHA-256.
Concept identity	Nonblank CUI and one distinct row per supported concept identity.
Medication identity	Atomic nonblank CUI/RxCUI pair, no duplicate pair, and deterministic matching RxNorm metadata.
Medication evidence text	Every nonblank dose/route/frequency/form/duration/date/status text field occurs literally in the bounded source clause.
CSV parity	Exact 13/16 workbook-selected headers and every exported cell compared back to run-scoped SQLite views.
WSD	Model/receipt/validation hashes, candidate preservation, one winner per decision group, and blank scores on abstention.
XMI when selected	One parseable native CAS per note plus path, byte count, hash, and run identity sidecar.

A structural pass is not a clinical accuracy result. It proves that the files agree with each other and with the source note. WSD clinical status remains not-evaluated until representative gold review measures it. Precision, recall, relation accuracy, medication interpretation, and WSD usefulness still need that review.

Review the completed SQLite result inside the TUI

Open Runs & cTAKES logs, select a completed run, then choose Open built-in viewer. The native viewer checks the accepted receipts before reading annotations.sqlite, then pages through documents, concepts, and medications without creating another note-bearing file. Technical files opens the fixed JSON receipts, exact-contract CSV files, and five safe SQLite views without an arbitrary SQL prompt.

Optional browser export

python3 viewer/build_viewer.py \
  --sqlite /approved/usdhub/results/annotations.sqlite \
  --output /approved/usdhub/results/clinician_viewer.html

Build this self-contained HTML file only when an approved browser handoff is useful. The builder repeats the schema, complete/pass, identity, source-hash, and span checks, then embeds the full source notes with a payload integrity hash. Offline does not mean encrypted. The HTML stays under the same controls as SQLite, and no runtime Java HTML writer is involved.

Open the synthetic browser export and see the same evidence-review pattern.

python3 scripts/analyze_storage.py \
  --input /approved/usdhub/batch \
  --output /approved/usdhub/results \
  --sqlite /approved/usdhub/results/annotations.sqlite

Refresh the storage receipt after building the optional HTML export so its bytes are included in the handoff measurement.

Open the TUI again

Setup adds the launcher to your user PATH. Open the TUI from any directory with:

ctakes tui

If the current shell has not refreshed its PATH yet, use ./ctakes tui from the Ctakes_USD folder.

Screen	What it does
System diagnostics	Runs `scripts/flight_check.sh` and lists every gate as ok, warn, or fail with its fix.
Sample testing	Automatically uses the included 100 fictional notes and their release-bound known-answer manifest, then checks cTAKES, WSD, SQLite, both CSV files, filenames, receipts, and expected results.
Setup & updates	Repairs release-pinned files or checks Codeberg for the latest compatible immutable Release 2 tag. It shows the tag, refuses tracked local edits, and changes the checkout only after confirmation. Setup verifies matching WSD.
New run	Searches nested note folders, shows the actual leaf filenames, and suggests a sibling results folder that remains editable. Its no-write check reaches strict filename, dictionary, WSD, JDBC, CPU, memory, and storage gates before Start is enabled.
Runs & cTAKES logs	Shows every job and the included sample test as a real row, with the receipt-derived note count, elapsed time, rate, estimate, and phase. A sample at 100/100 stays CHECKING until final checks finish, then shows PASSED, FAILED, or ATTENTION. Open built-in viewer and Technical files are described in step 07. Failed or stopped files stay labeled partial and cannot look sendable. Local logs and result values each need one confirmation per console session.
Comparison settings	Turns on the two recorded deviations: XMI writers or WSD off. Either one requires typing `NOT CLINICAL` and permanently marks the job `ADVANCED`.

Every screen carries the run contract strip: star recipe, strict USDHubID.NoteID.txt filenames, SQLite plus concepts.csv and drugs.csv, WSD on and fail-closed, autoscale on, sanity on, no XMI. Routine monitoring never reads log text. If you confirm the local-log warning, the console shows all controller and cTAKES sources together and labels the run as running, passed, failed, or stopped from process identity and receipts, never by matching words inside the log.

The console does not move the WSD boundary. It runs the same checksum-pinned download and validation scripts as setup, never stores a UMLS key, and does not change the model's not-evaluated clinical status. A default run remains fail-closed until the release-matched WSD pair validates.

Keep every note-derived artifact under clinical-data controls

Artifact	Sensitive content
`concepts.csv`, `drugs.csv`	Identifiers and the selected coded, categorical, and medication values; no full source-note column.
`annotations.sqlite`	Full source notes, identifiers, all canonical rows, and run metadata.
XMI	Full source note plus the complete native CAS and intermediate annotations.
Built-in TUI viewer	Displays the full source notes stored in SQLite.
`clinician_viewer.html` (optional)	Full source notes and review data embedded in one browser file.
Logs and receipts	HubID, NoteID, local paths, counts, errors, hashes, and operational metadata.

The runner uses owner-only POSIX creation modes where supported, but the real boundary is the encrypted volume, service account, access control, backup, transfer, retention, and deletion process. Text normalization is not de-identification.

Continue to outputs, versioning, viewer behavior, and security.

Production boundary

Engineering acceptance, dictionary distribution, and clinical production are recorded separately. The first two are complete for the exact release below. Clinical production is still open.

Use	Current state	Boundary
Controlled internal test	Engineering pass	The wrapper tests, runtime checks, output read-back, and controlled synthetic runs pass. Each site still stages the protected runtime and follows its approved clinical-data controls.
Clinical production	Not approved by this repository	A representative gold set must measure concept, medication, assertion, relation, and WSD performance. The site also needs its clinical, security, change, operations, monitoring, backup, and recovery approvals.
Release asset distribution	Approved	The runtime bundle covers cTAKES and the reviewed 2026AA dictionary. A separate reviewed release asset carries the WSD model and term-free receipt. The runtime and source archives exclude raw UMLS files, MIMIC clinical notes, and note-level derived answer files.

Wrapper 2.1.19 uses release ID 2.1.19_2026AA.r1_6.0.0. Verify that tag and its checksum assets before installing it. The unchanged approved runtime archive remains bound to cTAKES 6.0.0 and dictionary usd-star-2026AA-r1; the separate WSD asset is bound to that exact dictionary. Dictionary and recipes records the exact distribution boundary.

Nextthe same synthetic note, explained step by stepHow one clinical note moves through cTAKES