Ah, hyperparameter tuning.  Time & compute-intensive.  Frequently containing weird non-linearities in how changing a parameter changes the score and/or the time it takes to train the model.

RandomizedSearchCV goes noticeably faster than a full GridSearchCV but it still takes a while - which can be rough, because in my experience you do still need to be iterative with it and experiment with different distributions.  Plus, then you've got hyper-hyperparameters to tune - how many iterations SHOULD you run it for, anyway?

I've been experimenting with using the trusty old Binary Search to tune hyperparameters.  I'm finding it has two advantages.

1. It's blazing fast
2. The performance is competitive with a Randomized Search
3. It gives you a rough sketch of "the lay of the land".  An initial binary search can then provide parameters for future searches, including with Grid or Randomized Searches.

Code is here

Notebook summary is here

Let's see it in action!

from sklearn.ensemble import RandomForestClassifier

We'll be using a Random Forest classifier, because, as with all my code posts, it's what I've been using recently.

from sklearn.datasets import load_breast_cancer


data = load_breast_cancer()
X, y = data.data, data.target

We'll be using scikit-learn's breast cancer dataset, because I remembered that these packages I'm posting about have built-in demo datasets that I should be using for posts.

rfArgs = {
    "random_state": 0,
    "n_jobs": -1,
    "class_weight": "balanced",
    "oob_score": True
}

Let's set our random_state for better reproducibility.
We'll set n_jobs=-1 because obviously we want to use all our cores, we are not patient people.

We'll have class_weight="balanced" because that'll compensate for the fact that the breast cancer dataset (like most medical datasets) has unbalanced classes.
We'll use oob_score because we like being lazy, part of the appeal of Random Forests is the opportunity to be extra lazy (no need to normalize features!), and oob lets us be even lazier by giving some built-in cross-validation.

Now let's define a function that'll take all this, and spit out a score.  I wrote the binary search function to take a function like this as an argument - scikit-learn is usually pretty consistent when it comes to the interface it provides you, but sometimes different algorithms need to work a little differently.  For instance, since we'll be using Area Under precision_recall_curve as our metric (a good choice for classifiers with unbalanced classes!), it takes a teensy bit of extra fiddling to get it to play nicely with our oob_decision_function_.

from sklearn.metrics import precision_recall_curve
from sklearn.metrics import auc


def getForestAccuracy(X, y, kwargs):
    clf = RandomForestClassifier(**kwargs)
    clf.fit(X, y)
    y_pred = clf.oob_decision_function_[:, 1]
    precision, recall, _ = precision_recall_curve(y, y_pred)
    return auc(recall, precision)

We'll try to optimize the n_estimators parameter first.  For two reasons:

1. Finding a good mix between speed and accuracy here will make it easier to tune subsequent parameters.
2. It's the most straightforward to decide upper and lower bounds for.  Other ones (like, say, max_depth) require a little work to figure the potential range to search in.

Okay!  So, let's put our lower limit as 32 and our upper limit as 128, because I read in a StackOverflow post that there's a paper that says to search within that range.

n_estimators = bgs.compareValsBaseCase(
    X, 
    y, 
    getForestAccuracy,        
    rfArgs, 
    "n_estimators", 
    0, 
    18, 
    128
)

bgs.showTimeScoreChartAndGraph(n_estimators)

Plotting score, time, and the ratio between them - we're not just optimizing for the best score right now, we're looking for tipping points that give us good tradeoffs.  Scores and times are normalized for a more-meaningful ratio between them.

Hrm, looks like the score starts getting somewhere interesting around 68, and time starts shooting up at about 80.  Let's do another with those as our bounds!

n_estimators = bgs.compareValsBaseCase(
    X, 
    y, 
    getForestAccuracy,        
    rfArgs, 
    "n_estimators", 
    0, 
    68, 
    80
)

bgs.showTimeScoreChartAndGraph(n_estimators)

71 looks like our winner!  Or close enough for our purposes while we then go optimize other things.  And we only had to train our model 13 times - as opposed to the 96 we would have with a brute-force grid search.

Hopefully this will become a series on using this to tune other RF hyperparameters - other ones have some interesting quirks that I'd like to expound upon.  Or you could just look at the GitHub repo for spoilers.  Or both!