Why cTAKES can look idle before the first note

The run has visible phases before, during, and after note processing

A default run passes through six phases between the launch command and the final receipt, and only one phase is steady note processing. Checking means the wrapper is proving the input and resource plan without writing extraction output. Preparing means the controller is partitioning a large batch or starting workers. Processing means completed-note transactions are accumulating. Finalizing means the note count may already be full while the controller merges shards, builds indexes, exports CSVs, and reads the outputs back. Only Passed means the final delivery checks completed.

Before the first note completes, the wrapper checks the batch and runtime, plans capacity from CPU, memory, Linux commit, and disk, then validates the dictionary together with its word-sense disambiguation model. Each phase leaves a bounded non-PHI event or receipt, so the TUI can explain the current work without scraping clinical log text and a completed run can be audited without watching it live.

Replay figure

The TUI reads these phases from process identity, progress files, and receipts. Runs & cTAKES logs and the cTAKES log show the same completed and planned counts, elapsed time, recent processing rate, estimated time remaining, and current phase. The rate and estimate appear only after enough completed notes make the estimate meaningful. A count of 100/100 means note processing is done, not that the batch is ready: the sample test stays CHECKING while final validation and receipts finish. Missing information stays unknown instead of being guessed, and routine monitoring does not parse clinical log text.

Preflight finishes non-mutating checks before known outputs are cleaned

1. Resolve the input, output, work, and optional SQLite paths to canonical locations. SQLite is the single-file database the canonical result is written into.
2. Reject any input/output/work nesting or a SQLite target inside the input tree.
3. Walk the complete input batch and enforce strict USDHUB filenames and unique normalized document identities.
4. Locate Java 17, cTAKES 6.0.0, the compiled wrapper helpers, and the pinned SQLite JDBC jar, the Java driver that opens SQLite.
5. Resolve the selected dictionary descriptor and its strict manifest, the JSON file naming the expected descriptor hash, database hash, and UMLS release. UMLS, the Unified Medical Language System, is the terminology source the dictionary was built from.
6. Hash the descriptor and the dictionary database, then compare the release, UMLS, and hash fields with the manifest.
7. Resolve and validate the default WSD model and its build receipt against that exact dictionary.
8. Read CPU, memory, commit, and disk availability and calculate the thread, heap, and storage plan.
9. Only after those checks pass, replace known prior wrapper outputs or create the new run workspace.

Normal Release 2.1.19 setup downloads the checksum-pinned cTAKES/dictionary runtime and its separate release-matched WSD asset, verifies both, stages SQLite JDBC, and runs this fictional prerequisite path automatically. A note-processing run does not fetch missing runtime files.

Read the runner gate in scripts/run_pipeline.sh and the reusable capacity probes in scripts/system_capacity.sh.

The JVM has to assemble the exact extraction system before it processes text

A recipe is the ordered list of cTAKES components for the run, stored as a small .piper file. The wrapper creates run_pipeline.effective.piper by taking the current controlled-test recipe and inserting the selected writers and WSD profile, and it creates a temporary dictionary descriptor whose database path points to the protected runtime copy. Those two files are audit copies of what the JVM was asked to use.

Adding more threads does not make every one of these steps parallel. It can raise heap and CAS pressure, where the CAS is the in-memory container holding one note and its annotations, while the synchronized Drug NER step still runs one note at a time.

The timing receipt separates work that earlier logs blended together

The receipt uses monotonic clocks for elapsed intervals, so a wall-clock adjustment cannot make a phase negative or move an event backward.

The measured numbers on this page come from one 858-note matched run of wrapper 2.1.1; its row counts, arm-by-arm completion, and candidate-parity hash are recorded once on the WSD comparison. Release 2.1.19 keeps the same STAR clinical stage order, SQLite schema 8, and 13/16 CSV contract, but these measurements remain labeled as wrapper 2.1.1 evidence. The full-arm wall times are retained in the run receipts. They are not used for the startup comparison because cache state and host activity differed between those executions; the comparison below used three matched warm trials per path. A site should still measure representative notes with its intended writer and WSD profile, and plan from the slower end of its observed trials.

Read the timing derivation in scripts/write_timing_report.py.

WSD startup reuses one completed validation without skipping model checks

The wrapper hashes the model, verifies its build receipt and dictionary binding, opens the SQLite file, and runs quick_check, SQLite's own corruption scan, before Java starts. It then passes three values into Java: the fact that wrapper validation completed, the validated real model path, and the complete model SHA-256, the fingerprint of the exact bytes.

The annotator inside Java reuses that completed validation only when its resolved path and expected hash match. It still checks the schema, the build-complete marker, and the non-empty profile table. Standalone or mismatched use performs the full validation path.

This avoids repeating the same full database scan during one startup. It is a startup mechanism, not a claim that WSD makes note processing faster.

Three warm trials per path

The median startup difference was 7.82%. These six trials used the same one-note, two-byte fixture, one thread, a 4 GiB heap limit, and warm local storage. They measure duplicate validation work and do not establish a clinical-note processing rate.

The scoring method, thresholds, and abstention behavior are on the WSD page; the validation reuse itself is described here only.

CPU, memory, and Linux commit are all separate limits

Several different ceilings can stop the same run, so the wrapper probes each one instead of reading total host RAM. A cgroup is the Linux container resource boundary a job runs inside; it can hand this job a two-core quota or a slice of memory while the host itself looks idle. Commit is read as CommitLimit - Committed_AS: the virtual memory the kernel can still promise after its existing promises, which a heap request can exceed even when free RAM appears to exist.

Automatic heap selection uses the smaller safe memory/commit budget, keeps a 512 MiB planning gap (MiB and GiB are binary megabytes and gigabytes), applies the configured percentage, and caps the result. The initial heap is one quarter of maximum with a 512 MiB floor and 1 GiB cap, so Java does not commit the full maximum at launch.

bash scripts/run_pipeline.sh \
  --input /approved/usdhub/batch \
  --output /approved/usdhub/results \
  --xmx 4096

An explicit heap still has to pass the current RAM and commit hard guards. --no-autoscale disables automatic selection, not those safety checks.

Storage admission changes with the selected writer

The wrapper 2.1.1 858-note matched run measured the retained output after all CSV and receipt work completed. Every arm produced the same candidates, so the difference below comes from adding native XMI, not from extracting fewer rows.

The XMI directory was 154,369,746 bytes and the canonical SQLite database was 34,185,216 bytes. XMI alone was 4.52 times the database size. Omitting XMI removed 78.34% of the complete standard-profile retained bytes on this corpus.

Linear projection using the same note mix

These decimal-GB projections multiply the measured per-note values. A different note length, annotation density, filesystem, compression policy, or retained failure workspace will move the real number.

The selected CSV fields also reduce outbound data

The selected shape removed 38.81% of the CSV bytes. It also keeps unselected note-text and evidence-text fields out of the outbound CSV. The local SQLite database still contains the source notes and remains PHI (protected health information).

Admission factors remain conservative guards

The factor is an admission estimate, not an output guarantee. A 256 MiB output floor, separate SQLite/work checks, 2 GiB per concurrent async JVM, and a 10 GiB free-space reserve are applied where relevant.

storage_report.json records the persistent bytes and observed work/shard footprint at measurement time. It does not replace infrastructure monitoring for true peak disk, peak RSS (resident set size, the RAM a process actually holds), I/O latency, or cgroup throttling.

Read the admission formulas in scripts/run_pipeline.sh and scripts/run_async.sh; read observed post-run categories in scripts/analyze_storage.py.

Async helps only when the notes can amortize another JVM startup

The planner starts from at least 25 notes per JVM, then caps the shard count by available documents, CPU, aggregate heap, and storage. A shard is one slice of the batch processed by its own child JVM, so two shards mean two dictionaries and two sets of models in memory.

1. Validate the whole batch once and calculate one accepted identity set.
2. Partition unique notes across shards.
3. Give each child a unique run_id, the identity of one execution, while requiring matching release and analysis inputs.
4. Run each child into a separate SQLite database and retained log tree.
5. Merge child databases into a new aggregate database inside a transaction.
6. Regenerate aggregate CSV, XMI provenance, manifest, sanity, storage, and timing receipts.
7. Delete temporary shards only after all aggregate checks pass. Preserve them on any child, merge, export, provenance, sanity, or finalization failure.

bash scripts/run_async.sh \
  --input /approved/usdhub/batch \
  --output /approved/usdhub/results \
  --flatten-output

Use the retained events to tell slow startup from a stopped run

tail -100 /approved/usdhub/results/run_pipeline.log
python3 -m json.tool /approved/usdhub/results/timing_report.json
python3 -m json.tool /approved/usdhub/results/run_manifest.json
python3 -m json.tool /approved/usdhub/results/sanity_report.json

Resume is restricted because a partial canonical SQLite run cannot satisfy a full-batch run scope. A clean rerun uses a new or intentionally cleared output folder. Any resume must keep input, writer, dictionary, recipe, WSD profile, and release identity unchanged.