The chart below shows the decision tree with four levels. What's striking, though, is what the optimization revealed:
the best tree has just a single branch. Deeper trees increasingly memorize the training data but perform worse on new data
— a textbook example of overfitting. Because credit history is so dominant in this dataset, a single split is enough to capture
almost all the predictable information. The shallow tree is not only more accurate but also easier to explain — a real advantage in
lending decisions.
What the Confusion Matrix Reveals
The model is considerably better at identifying good customers than at rejecting bad ones: of 30 actual rejection cases,
it correctly identifies 20 — while 10 are wrongly approved. On the approval side, it performs noticeably more reliably,
catching 54 out of 60. This is a typical pattern with slightly imbalanced data. And this is where the interesting boundary lies:
this shortfall cannot be fixed through hyperparameter tuning.
A systematic search across 48 parameter combinations yielded no improvement — the limitation isn't in the settings,
but in the available features.
More data wouldn't have changed much either. The data generator deliberately includes a random component:
even with a good credit history, the approval probability is 85%, not 100%. This residual uncertainty is what's known
as the irreducible error — no model can predict genuine randomness, regardless of how much data it sees.
What would have helped are
better features a debt-to-income ratio, length of employment, income stability, or a continuous
credit score. A single such feature would have been worth more than ten times the number of rows. This reflects a well-known hierarchy:
feature quality beats data volume beats model choice. The model isn't poor — it is as good as it can be given the available features.
And that is precisely where the line runs between model work and data work.
