Getting startedCopyable commandsControlled internal test

Install, run, and check one USDHUB batch

The safe path finishes Java, cTAKES, dictionary, WSD, and SQLite setup before clinical notes arrive. The first command that sees a note is the filename check.

Host runtime: Java 17 + Python 3
cTAKES: 6.0.0
Dictionary: UMLS 2026AA release
Default WSD: Validated model on
Default writer: SQLite + two CSVs
Network during run: Not required

Before notes arrive on the machine

Use an approved encrypted clinical-data volume and a restricted service account. Stage the cTAKES bundle, Java archive when needed, SQLite JDBC jar, active dictionary, and its WSD model while the input folder is still empty.

Item	Why the wrapper needs it	What proves it is the reviewed item
Java 17	Runs cTAKES 6 and the wrapper Java helpers.	Java version check plus the archive SHA-256 when portable Java is used.
cTAKES 6.0.0 bundle	Supplies Apache engines, models, UIMA, and type systems.	Verified bundle receipt and archive SHA-256.
Active dictionary	Turns note text into CUI/RxCUI candidates.	Strict manifest, descriptor hash, database hash, UMLS release, and source inventory.
WSD model and receipt	Scores same-span ambiguity groups while preserving all candidates.	Model SHA-256, build receipt, exact dictionary binding, SQLite quick check, and validation receipt.
SQLite JDBC 3.45.3.0	Lets the Java writer commit the canonical database.	Pinned jar SHA-256 in the offline kit or local runtime setup.

Install the wrapper and run the flight check

git clone https://github.com/editnori/Ctakes_USD.git
cd Ctakes_USD
bash scripts/setup.sh

setup.sh prepares the public wrapper and runtime dependencies. It cannot supply licensed terminology. A clean source checkout is not ready for the production recipe until the protected dictionary database and matching WSD files are staged.

Install a checksum-reviewed cTAKES archive

BUNDLE=/approved/runtime/CtakesBun-bundle.tgz
export CTAKES_BUNDLE_SHA256="$(awk '{print $1}' "${BUNDLE}.sha256")"
bash scripts/get_bundle.sh --file "$BUNDLE"
bash scripts/flight_check.sh

The downloader stops when it cannot verify the archive. A mutable latest URL is not a version pin by itself.

Stage Java 17 when the host does not provide it

JAVA_ARCHIVE=/approved/runtime/temurin-17-jdk.tar.gz
JAVA_SHA256="$(awk '{print $1}' "${JAVA_ARCHIVE}.sha256")"
bash scripts/get_java.sh --from-file "$JAVA_ARCHIVE" --sha256 "$JAVA_SHA256"
export CTAKES_JAVA_HOME="$PWD/.ctakes_deps/java/current"
bash scripts/flight_check.sh

Stage the protected dictionary and WSD files

The site terminology owner supplies one accepted dictionary candidate plus the WSD model and term-free receipt built from that exact dictionary. Keep those files on the approved clinical runtime volume; do not fetch them during a note-processing run.

export CTAKES_HOME=/approved/usdhub/runtime/apache-ctakes-6.0.0

bash scripts/activate_dictionary.sh \
  --candidate /approved/usdhub/runtime/dictionary-candidate \
  --ctakes-home "$CTAKES_HOME"

mkdir -p .ctakes_deps/wsd
install -m 600 \
  /approved/usdhub/runtime/usd-star-2026AA-r1-wsd.sqlite \
  .ctakes_deps/wsd/usd-star-2026AA-r1-wsd.sqlite
install -m 600 \
  /approved/usdhub/runtime/usd-star-2026AA-r1-wsd.receipt.json \
  .ctakes_deps/wsd/usd-star-2026AA-r1-wsd.receipt.json

bash scripts/flight_check.sh

flight_check.sh always creates its own synthetic, filename-valid no-write input. A pass reaches the real dictionary manifest/database, WSD model and receipt, SQLite JDBC, release identity, CPU, memory, Linux commit, and storage gates without starting Java or processing a clinical note.

Read the result literally

Flight-check result	Action
Pass	The local prerequisites are internally consistent. Continue to the synthetic or approved batch.
Dictionary manifest/hash failure	Do not run. Activate the reviewed dictionary candidate or correct the selected descriptor.
WSD model/receipt failure	Do not bypass validation. Restore the model built for that exact dictionary or run explicitly with `--no-wsd` only for an approved comparison.
Capacity probe failure	Fix the CPU, memory, commit, or disk probe. Do not substitute total host RAM for available job memory.

Prepare one text file per note

The filename carries the site HubID and NoteID used throughout SQLite, CSV, XMI receipts, and the viewer.

USDHUB12345678901234567890.987654321.txt

The HubID is the literal prefix USDHUB followed by exactly 20 digits.

Encode every note as UTF-8.
Keep the source line endings. The offsets and evidence strings are checked against those exact decoded characters.
NoteID may use letters, digits, periods, underscores, or hyphens and must begin with a letter or digit.
Nested folders are allowed. Duplicate HubID/NoteID pairs anywhere in the batch are not.

Keep the Phase I metadata CSV separate

USDHubID,NoteID,NoteDate,NoteType,NoteTitle,VisitOccurrenceID,ProviderID

The wrapper checks the note filename contract. It does not join this site metadata CSV into concept or medication extraction.

Validate the whole batch before Java starts

python3 scripts/validate_filenames.py \
  --input /approved/usdhub/batch \
  --policy strict \
  --json-out /approved/usdhub/filename_validation.json

The validator walks the complete input tree. One malformed name, empty NoteID, non-portable character, case-normalized collision, or duplicate HubID/NoteID pair fails the batch. cTAKES has not started at this point.

Receipt field	Why it is kept
Document count	Proves how many note files passed the gate.
Error summary	Names the exact filename rule that failed.
Relative-path hash	Detects a changed batch listing between validation and handoff.
Policy	Shows that strict USDHUB validation was active.

Run the reviewed extraction

The default run loads the active dictionary’s validated WSD model, preserves every terminology candidate, and writes SQLite plus both delivery CSVs. The technical recipe key remains star for compatibility.

Default

bash scripts/run_pipeline.sh \
  --input /approved/usdhub/batch \
  --output /approved/usdhub/results

Writes annotations.sqlite, concepts.csv, drugs.csv, and the validation/version/timing/storage receipts.

Native CAS

bash scripts/run_pipeline.sh \
  --input /approved/usdhub/batch \
  --output /approved/usdhub/results-standard \
  --writers standard

Adds native per-note XMI and xmi_provenance.json. XMI normally uses the most disk.

Resource planned

bash scripts/run_async.sh \
  --input /approved/usdhub/batch \
  --output /approved/usdhub/results \
  --flatten-output

Validates once, plans shards from CPU/memory/commit/disk, runs child SQLite databases, then merges and rechecks one aggregate result.

Run the explicit WSD-off comparison

bash scripts/run_pipeline.sh \
  --input /approved/usdhub/batch \
  --output /approved/usdhub/results-wsd-off \
  --no-wsd

Every distinct same-span CUI still remains. Only the WSD marker and score columns are blank. This produces a different analysis_id.

What the wrapper does before the first note

Rechecks the filename batch and input/output path boundaries.
Verifies wrapper, cTAKES, UMLS, dictionary, and SQLite JDBC identities.
Validates the active dictionary database and WSD model/receipt binding.
Calculates safe threads, heap, shard count, and storage admission.
Builds the exact effective piper and temporary runtime dictionary descriptor.
Starts Java and records startup separately from note processing.

The startup page explains each phase, the measured timings, and the repeated WSD database-check correction.

Accept the batch only after the read-back passes

python3 -m json.tool /approved/usdhub/results/run_manifest.json
python3 -m json.tool /approved/usdhub/results/sanity_report.json

The handoff condition is literal: manifest status complete and sanity status pass.

Check	What is read back
SQLite integrity and schema	`quick_check`, schema 6, run metadata, exact document set, and committed rows.
Source evidence	UTF-8 source text, Java/UIMA UTF-16 offsets, text spans, source length, and SHA-256.
Concept identity	Nonblank CUI and one distinct row per supported concept identity.
Medication identity	Atomic nonblank CUI/RxCUI pair, no duplicate pair, and deterministic matching RxNorm metadata.
Medication evidence text	Every nonblank dose/route/frequency/form/duration/date/status text field occurs literally in the bounded source clause.
CSV parity	Exact 26/34 headers and every exported cell compared back to run-scoped SQLite views.
WSD	Model/receipt/validation hashes, candidate preservation, one winner per decision group, and blank scores on abstention.
XMI when selected	One parseable native CAS per note plus path, byte count, hash, and run identity sidecar.

Build the clinician viewer from the completed SQLite file

python3 viewer/build_viewer.py \
  --sqlite /approved/usdhub/results/annotations.sqlite \
  --output /approved/usdhub/results/clinician_viewer.html

The viewer builder checks schema 6, complete/pass status, run identities, embedded source hashes, and spans. It then embeds a compressed payload and integrity hash in one offline HTML file. The file is not encrypted and is not safe to move outside clinical-data controls. No runtime Java HTML writer is involved.

Open the synthetic embedded viewer and see the exact interaction.

python3 scripts/analyze_storage.py \
  --input /approved/usdhub/batch \
  --output /approved/usdhub/results \
  --sqlite /approved/usdhub/results/annotations.sqlite

Refresh the storage receipt after building the viewer so its bytes are included in the handoff measurement.

Keep every note-derived artifact under clinical-data controls

Artifact	Sensitive content
`concepts.csv`, `drugs.csv`	Identifiers and original note substrings.
`annotations.sqlite`	Full source notes, identifiers, all canonical rows, and run metadata.
XMI	Full source note plus the complete native CAS and intermediate annotations.
`clinician_viewer.html`	Full source notes and review data embedded in one file.
Logs and receipts	HubID, NoteID, local paths, counts, errors, hashes, and operational metadata.

The runner uses owner-only POSIX creation modes where supported, but the real boundary is the encrypted volume, service account, ACL, backup, transfer, retention, and deletion process. Text normalization is not de-identification.

Continue to outputs, versioning, viewer behavior, and security.