• Part 1 (n_estimators) here
• Part 2 (max_depth) here
• Notebook here

Another parameter, another set of quirks!

min_samples_leaf is sort of similar to max_depth.  It helps us avoid overfitting.  It's also non-obvious what you should use as your upper and lower limits to search between.  Let's do what we did last week - build a forest with no parameters, see what it does, and use the upper and lower limits!

import pandas as pd

from sklearn.metrics import precision_recall_curve
from sklearn.metrics import auc
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_breast_cancer


data = load_breast_cancer()
X, y = data.data, data.target

rfArgs = {"random_state": 0,
          "n_jobs": -1,
          "class_weight": "balanced",
         "n_estimators": 18,
         "oob_score": True}

clf = RandomForestClassifier(**rfArgs)
clf.fit(X, y)

Let's use the handy function from here to crawl the number of samples in a tree's leaf nodes:

def leaf_samples(tree, node_id = 0):
    left_child = tree.children_left[node_id]
    right_child = tree.children_right[node_id]
    
    if left_child == _tree.TREE_LEAF:
        samples = np.array([tree.n_node_samples[node_id]])
        
    else:
        
        left_samples = leaf_samples(tree, left_child)
        right_samples = leaf_samples(tree, right_child)
        
        samples = np.append(left_samples, right_samples)
        
    return samples

Last week we made a function to grab them for a whole forest - since this is the second time we're doing this, and we may do it again, let's make a modular little function that takes a crawler function as an argument!

def getForestParams(X, y, param, kwargs):
    clf = RandomForestClassifier(**kwargs)
    clf.fit(X, y)
    params = np.hstack([param(estimator.tree_) 
                 for estimator in clf.estimators_])
    return {"min": params.min(),
           "max": params.max()}

def getForestParams(X, y, param, kwargs):
    clf = RandomForestClassifier(**kwargs)
    clf.fit(X, y)
    params = np.hstack([param(estimator.tree_) 
                 for estimator in clf.estimators_])
    return {"min": params.min(),
           "max": params.max()}

Let's see it in action!

data = load_breast_cancer()
X, y = data.data, data.target

rfArgs = {"random_state": 0,
          "n_jobs": -1,
          "class_weight": "balanced",
         "n_estimators": 18,
         "oob_score": True}

getForestParams(X, y, leaf_samples, rfArgs)
#> {'max': 199, 'min': 1}

Almost ready to start optimizing!  Since part of what we get out of optimizing min_samples_leaf is regularization (and because it's just good practice!), let's make a metric with some cross-validation.  Luckily, Scikit has a builtin cross_val_score function.  We'll just need to do a teensy bit of tweaking to make it use the area under a precision_recall_curve.

from sklearn.model_selection import cross_val_score

def auc_prc(estimator, X, y):
    estimator.fit(X, y)
    y_pred = estimator.oob_decision_function_[:, 1]
    precision, recall, _ = precision_recall_curve(y, y_pred)
    return auc(recall, precision)

def getForestAccuracyCV(X, y, kwargs):
    clf = RandomForestClassifier(**kwargs)
    return np.mean(cross_val_score(clf, X, y, scoring=auc_prc, cv=5))

Awesome, now we have a metric that can be fed into our binary search.

min_samples_leaf = bgs.compareValsBaseCase(X, 
                    y, 
                    getForestAccuracyCV,        
                    rfArgs, 
                    "min_samples_leaf", 
                    0, 
                    1, 
                    199)
bgs.showTimeScoreChartAndGraph(min_samples_leaf)

Looks like the action's between 1 and 51.  More than that, and the score goes while simultaneously increasing the runtime - the opposite of what we want!

min_samples_leaf = bgs.compareValsBaseCase(X, 
                    y, 
                    getForestAccuracyCV,        
                    rfArgs, 
                    "min_samples_leaf", 
                    0, 
                    1, 
                    14)
bgs.showTimeScoreChartAndGraph(min_samples_leaf)

Big drop-off after 26, it seems!

min_samples_leaf = bgs.compareValsBaseCase(X, 
                    y, 
                    getForestAccuracyCV,        
                    rfArgs, 
                    "min_samples_leaf", 
                    0, 
                    1, 
                    26)
bgs.showTimeScoreChartAndGraph(min_samples_leaf)

One more with 14 as our upper limit!

min_samples_leaf = bgs.compareValsBaseCase(X, 
                    y, 
                    getForestAccuracyCV,        
                    rfArgs, 
                    "min_samples_leaf", 
                    0, 
                    1, 
                    14)
bgs.showTimeScoreChartAndGraph(min_samples_leaf)

I suppose when it gets this small we could use a regular Grid Search, but...maybe next week!  Or maybe another variable!  Or maybe benchmarks vs GridSearchCV and/or RandomizedSearchCV.  Who knows what the future holds?