Case 06 / Machine learning
Can a card's design reveal its mana value?
Four models estimate Magic: The Gathering mana value from card attributes and rules text while the printed mana cost stays hidden. A name-grouped test split makes the result more meaningful than a random split of reprints.
- CatBoost
- XGBoost
- Compact MLP
- Expanded MLP
- PyTorch
- Grouped Holdout
- 01 / QUESTIONHide the answerExclude mana cost, CMC, card name, and set from predictors
- 02 / SPLITGroup by nameKeep reprints together across train, validation, and test
- 03 / MODELCompare four pipelinesTwo boosted trees and two multilayer perceptrons
- 04 / EVALUATEScore the holdoutWithin-one, class, and expected-value errors
- Eligible card rows
- 93,452CMC classes 0–10
- Held-out test rows
- 9,4912,901 distinct card names
- Within-one accuracy
- 82.86%CatBoost on this test split
Estimate a number without its defining input.
A card's mana value comes from its printed mana cost. This project deliberately withholds that cost and asks whether the rest of the card—type, rarity, color identity, combat statistics, and rules text—contains enough signal to estimate its rounded mana value from 0 through 10.
The problem is hard because these properties are clues, not a formula. Similar abilities can cost different amounts; the same type or color appears at many costs. Rules text can contain mana symbols for abilities, but those are not the card's casting cost. The value is therefore not always recoverable from the permitted inputs.
A fairer test for repeated cards.
The data contain multiple printings of some cards. If rows were split at random, a model could train on one printing and be tested on another. Instead, card names are stripped and case-folded before grouping: no name can cross the training, validation, and test partitions.
The resulting split contains 75,229 training, 8,732 validation, and 9,491 test rows. Training-only fitting supplies normalization and categorical vocabularies. Validation chooses stopping iterations or neural-network checkpoints; the held-out test partition supplies the table below. The same scoring code evaluates all four models.
Four routes from card text to a class.
XGBoost and CatBoost
The two boosted-tree models share 59 engineered features: five categorical descriptors, type and color indicators, power/toughness/loyalty values and missingness, plus counts of rules-text patterns. XGBoost uses categorical columns with a training-derived vocabulary; CatBoost handles those columns natively. Both predict probabilities over the eleven CMC classes and stop when validation multiclass loss no longer improves.
Compact neural network
A 512-dimensional stable hash representation encodes categorical and type cues, joined to 45 numeric features. The MLP has hidden layers of 512, 256, and 128 units with dropout. It uses class-weighted cross-entropy and selects its checkpoint using validation classification performance.
Expanded neural network
The second MLP expands the hashed representation to 1,024 dimensions and adds a broader set of rules-text indicators for 80 numeric features. Its hidden layers are 768, 512, 256, and 128 units. It uses the same broad training objective, but selects the checkpoint with the lowest validation expected-value error.
These are complete pipelines: the neural networks do not share the tree models' exact feature representation, and the two neural networks do not share a checkpoint criterion. The results compare practical approaches, not architecture alone.
Different measures, different leaders.
Each figure below comes from the same 9,491 test rows. Within-one accuracy allows an error of one class. Class MAE measures the average distance between the chosen integer class and the target; expected-value MAE instead evaluates the probability-weighted average of classes.
| Model | Within ±1 ↑ | Class MAE ↓ | Expected MAE ↓ |
|---|---|---|---|
| XGBoost | 80.56% | 0.822 | 0.768 |
| CatBoost | 82.86% | 0.765 | 0.766 |
| Compact MLP | 80.23% | 0.870 | 0.779 |
| Expanded MLP | 81.07% | 0.803 | 0.753 |
CatBoost leads on within-one accuracy and integer-class MAE. The expanded MLP has the lowest expected-value MAE. For a chosen whole-number cost, CatBoost is strongest on the displayed measures; for a probability-weighted estimate, the expanded MLP has a small advantage.
Useful signal, with a real ceiling.
More than four in five CatBoost test predictions fall within one mana-value class. The hidden printed cost cannot always be recovered from card attributes, so the remaining errors matter. Row-level scores can also overemphasize frequently reprinted cards; giving each distinct test card name equal weight yields a within-one rate of 80.67% for CatBoost.
High-cost classes are sparse in this holdout—only 25 test rows have CMC 9 and three have CMC 10—so the aggregate numbers should not be read as a guarantee for rare classes. The benchmark uses one dataset and one name-grouped split; it does not establish future-release performance or statistical certainty across many splits.