Why cTAKES can look idle before the first note

The run has visible phases before, during, and after note processing

Replay figure

Preflight finishes non-mutating checks before known outputs are cleaned

1. Resolve the input, output, work, and optional SQLite paths to canonical locations.
2. Reject any input/output/work nesting or SQLite target inside the input tree.
3. Walk the complete input batch and enforce strict USDHUB filenames and unique normalized document identities.
4. Locate Java 17, cTAKES 6.0.0, compiled wrapper helpers, and the pinned SQLite JDBC jar.
5. Resolve the selected dictionary descriptor and strict manifest.
6. Hash the descriptor and dictionary database, then compare release/UMLS/hash fields with the manifest.
7. Resolve and validate the default WSD model and build receipt against that dictionary.
8. Read CPU, memory, commit, and disk availability and calculate the write/heap/thread plan.
9. Only after those checks pass, replace known prior wrapper outputs or create the new run workspace.

Read the runner gate: scripts/run_pipeline.sh and the reusable capacity probes in scripts/system_capacity.sh.

The JVM has to assemble the exact extraction system before it processes text

The wrapper creates run_pipeline.effective.piper by taking the reviewed recipe and inserting the selected writers and WSD profile. It also creates a temporary dictionary descriptor whose database path points to the protected runtime copy. Those two files are audit copies of what the JVM was asked to use.

Adding more threads does not make every one of these steps parallel. It can increase heap and CAS pressure while synchronized Drug NER remains one-at-a-time.

The timing receipt separates work that earlier logs blended together

The receipt uses monotonic clocks for elapsed intervals, so a wall-clock adjustment cannot make a phase negative or move an event backward.

Current wrapper 2.1 correctness receipts

These are literal receipts from completed wrapper 2.1 correctness runs on a busy host. They are not a benchmark and they do not prove that WSD improves speed: cache state, host load, helper compilation, and filesystem activity differed between executions. A site should measure 25–100 representative notes with the intended writer and WSD profile, alternate cold and warm starts, and size from the slower end of the observed trial distribution.

Read the timing derivation: scripts/write_timing_report.py.

WSD startup reuses one completed validation without skipping model checks

The wrapper hashes the model, verifies its build receipt and dictionary binding, opens SQLite, and runs quick_check before Java starts. It then passes three values into Java: wrapper validation is complete, the validated real model path, and the complete model SHA-256.

The annotator reuses that completed validation only when its resolved path and expected hash match. It still checks the schema, build-complete marker, and non-empty profile table. Standalone or mismatched use performs the full validation path.

This avoids repeating the same full database scan during one startup. It is a startup mechanism, not a claim that WSD makes note processing faster.

Read the WSD scoring and startup boundary, including why this shortcut cannot bypass path/hash matching.

CPU, memory, and Linux commit are all separate limits

Automatic heap selection uses the smaller safe memory/commit budget, keeps a 512 MiB planning gap, applies the configured percentage, and caps the result. The initial heap is one quarter of maximum with a 512 MiB floor and 1 GiB cap, so Java does not commit the full maximum at launch.

bash scripts/run_pipeline.sh \
  --input /approved/usdhub/batch \
  --output /approved/usdhub/results \
  --xmx 4096

An explicit heap still has to pass current RAM/commit hard guards. --no-autoscale disables automatic selection, not those safety checks.

Storage admission changes with the selected writer

The factor is an admission estimate, not an output guarantee. A 256 MiB output floor, separate SQLite/work checks, 2 GiB per concurrent async JVM, and a 10 GiB free-space reserve are applied where relevant.

storage_report.json records the persistent bytes and observed work/shard footprint at measurement time. It does not replace infrastructure monitoring for true peak disk, peak RSS, I/O latency, or cgroup throttling.

Read the admission formulas in scripts/run_pipeline.sh and scripts/run_async.sh; read observed post-run categories in scripts/analyze_storage.py.

Async helps only when the notes can amortize another JVM startup

The planner starts from at least 25 notes per JVM, then caps shard count by available documents, CPU, aggregate heap, and storage. Two shards mean two dictionaries and two sets of models in memory.

1. Validate the whole batch once and calculate one accepted identity set.
2. Partition unique notes across shards.
3. Give each child a unique run_id while requiring matching release and analysis inputs.
4. Run each child into a separate SQLite database and retained log tree.
5. Merge child databases into a new aggregate database inside a transaction.
6. Regenerate aggregate CSV, XMI provenance, manifest, sanity, storage, and timing receipts.
7. Delete temporary shards only after all aggregate checks pass. Preserve them on any child, merge, export, provenance, sanity, or finalization failure.

bash scripts/run_async.sh \
  --input /approved/usdhub/batch \
  --output /approved/usdhub/results \
  --flatten-output

Use the retained events to tell slow startup from a stopped run

tail -100 /approved/usdhub/results/run_pipeline.log
python3 -m json.tool /approved/usdhub/results/timing_report.json
python3 -m json.tool /approved/usdhub/results/run_manifest.json
python3 -m json.tool /approved/usdhub/results/sanity_report.json

Resume is restricted because a partial canonical SQLite run cannot satisfy a full-batch run scope. A clean rerun uses a new or intentionally cleared output folder. Any resume must keep input, writer, dictionary, recipe, WSD profile, and release identity unchanged.