What survives beyond a random split?

Can a useful benchmark avoid easy shortcuts?

The task is binary classification of disease presence in the historical UCI Heart Disease dataset. The target is positive when the source's num field is greater than zero. The goal is to compare practical pipelines and understand their errors, especially missed positive labels and false alarms.

The accompanying Streamlit form lets someone enter features and inspect a saved model's output. That output is not a diagnosis or a validated personal risk estimate. Several inputs require clinical tests, and the training records come from historical hospital cohorts rather than people using a public website.

More rows do not always mean more patients.

The sole training source has 920 rows from Cleveland, Hungary, Switzerland, and VA Long Beach. Removing two duplicate profiles apart from ID leaves 918 records: 508 positive and 410 negative. Two other supplied datasets overlap these records, so appending them would duplicate patients rather than add independent observations.

Data source: UCI Heart Disease, credited to Andras Janosi, William Steinbrunn, Matthias Pfisterer, and Robert Detrano under CC BY 4.0.

Missingness is substantial and differs by hospital. In the raw source, exercise ST slope is missing in 309 rows, major-vessel count in 611, and thallium scan category in 486. Zero cholesterol in 172 rows and one zero resting blood pressure are treated as missing. The cleaned data keep these gaps until each training fold learns its own imputation values.

The experiment compares a core set of 10 features, an 11-feature set adding ST slope, and all 13 features adding vessel count and thallium scan. Each is tested with median/mode imputation and with missingness indicators or a separate categorical Missing level. Numeric inputs are standardized for logistic regression; tree models use their original scales. Hospital and record ID are not predictors.

Make the choice before reading the holdout.

The cleaned data are split once with seed 42, stratified jointly by hospital and label: 734 training and 184 development holdout records. Logistic regression provides the baseline; random forest and XGBoost are the alternatives. Each family searches 18 predeclared combinations of three feature sets, two missing-data treatments, and three parameter settings.

Within training, three outer stratified folds each run a three-fold inner search. Mean outer F2 chooses the model family, giving recall extra weight at a fixed 0.50 threshold. A five-fold search on all training records chooses each family's final settings. Imputation, scaling, and encoding are fitted inside the training folds. The holdout describes the fitted models; it does not choose their settings or family.

A separate leave-one-hospital-out check on training records asks whether the selected configuration works when the source hospital changes. This is useful because a random split can put patients from every hospital on both sides of the evaluation.

Similar holdout recall, different selection scores.

The selected random forest uses all 13 features, missingness indicators, 300 trees, maximum depth 6, and at least five samples per leaf. Its holdout confusion matrix is [[64, 18], [10, 92]], with actual negative/positive labels as rows and predicted negative/positive labels as columns.

Logistic regression had one fewer false positive on this holdout, while the forest led the predeclared outer-CV F2 comparison. The small differences do not establish a universal winner; they describe this dataset and evaluation design.

A simpler feature set deserves attention.

With the selected forest settings, held-out-hospital recall ranges from 0.772 in Switzerland to 0.941 in Hungary. Keeping the forest parameters fixed while changing the feature and missing-data treatment gives this comparison across the four hospital folds:

The 13-feature choice finds four more positive labels overall but produces 28 more false positives and needs two sparsely recorded tests. These are repeated evaluations of the same 734 training records, not new patients. The app retains the predeclared nested-CV winner while exposing this transfer tradeoff.

A research result, not patient advice.

The 184-row holdout was inspected during baseline development, so it is a development holdout, not a fresh final test after tuning. Source prevalence and missingness vary sharply across hospitals. An independent cohort representative of intended users is needed before making a final performance claim.

The app's percentage is an uncalibrated model score, not a validated probability of disease. The fixed 0.50 threshold has not been chosen for a defined clinical use. The full repository README includes data lineage, search settings, artifacts, and further limitations.