## Implementing a Weighted Majority Rule Ensemble Classifier in scikit-learn

- 11/01/2015
- 148
- 0 Like

**Published In**

- Big Data
- Analytics
- Artificial Intelligence

Republished with Author’s Permission – Originally published on >Sebastian Raschka Blog, dated >11 Jan 2015.

Here, I want to present a simple and conservative approach of implementing a weighted majority rule ensemble classifier in scikit-learn that yielded remarkably good results when I tried it in a kaggle competition. For me personally, kaggle competitions are just a nice way to try out and compare different approaches and ideas – basically an opportunity to learn in a controlled environment with nice datasets.

Of course, there are other implementations of more sophisticated ensemble methods in scikit-learn, such as bagging classifiers, random forests, or the famous AdaBoost algorithm. However, as far as I am concerned, they all require the usage of a common “base classifier.”

In contrast, my motivation for the following approach was to combine conceptually different machine learning classifiers and use a majority vote rule. The reason for this was that I had trained a set of equally well performing models, and I wanted to balance out their individual weaknesses.

**Sections**

~ Sections

~ Classifying Iris Flowers Using Different Classification Models

~ Implementing the Majority Voting Rule Ensemble Classifier

~ Additional Note About the EnsembleClassifier Implementation: Class Labels vs. Probabilities

1) Prediction based on majority class labels:

2) Prediction based on predicted probabilities (equal weights, weights=[1,1,1])

~ EnsembleClassifier - Tuning Weights

~ EnsembleClassifier - Pipelines

~ Some Final Words

**Classifying Iris Flowers Using Different Classification Models**

For a simple example, let us use three different classification models to classify the samples in the Iris dataset: Logistic regression, a naive Bayes classifier with a Gaussian kernel, and a random forest classifier – an ensemble method itself. At this point, let’s not worry about preprocessing the data and training and test sets. Also, we will only use 2 feature columns (sepal width and petal height) to make the classification problem harder.

from sklearn import datasets

iris = datasets.load_iris()

X, y = iris.data[:, 1:3], iris.target

from sklearn import cross_validation

from sklearn.linear_model import LogisticRegression

from sklearn.naive_bayes import GaussianNB

from sklearn.ensemble import RandomForestClassifier

import numpy as np

np.random.seed(123)

clf1 = LogisticRegression()

clf2 = RandomForestClassifier()

clf3 = GaussianNB()

print('5-fold cross validation:\n')

for clf, label in zip([clf1, clf2, clf3], ['Logistic Regression', 'Random Forest', 'naive Bayes']):

scores = cross_validation.cross_val_score(clf, X, y, cv=5, scoring='accuracy')

print("Accuracy: %0.2f (+/- %0.2f) [%s]" % (scores.mean(), scores.std(), label))

5-fold cross validation:

Accuracy: 0.90 (+/- 0.05) [Logistic Regression]

Accuracy: 0.92 (+/- 0.05) [Random Forest]

Accuracy: 0.91 (+/- 0.04) [naive Bayes]

As we can see from the cross-validation results above, the performance of the three models is almost equal.

**Implementing the Majority Voting Rule Ensemble Classifier**

Now, we will implement a simple EnsembleClassifier class that allows us to combine the three different classifiers. We define a predict method that let’s us simply take the majority rule of the predictions by the classifiers. E.g., if the prediction for a sample is

- classifier 1 -> class 1

- classifier 2 -> class 1

- classifier 3 -> class 2

we would classify the sample as “class 1.”

Furthermore, we add a weights parameter, which let’s us assign a specific weight to each classifier. In order to work with the weights, we collect the predicted class probabilities for each classifier, multiply it by the classifier weight, and take the average. Based on these weighted average probabilties, we can then assign the class label.

To illustrate this with a simple example, let’s assume we have 3 classifiers and a 3-class classification problems where we assign equal weights to all classifiers (the default): w1=1, w2=1, w3=1.

The weighted average probabilities for a sample would then be calculated as follows:

We can see in the table above that class 2 has the highest weighted average probability, thus we classify the sample as class 2.

Now, let’s put it into code and apply it to our Iris classification.

from sklearn.base import BaseEstimator

from sklearn.base import ClassifierMixin

import numpy as np

import operator

class EnsembleClassifier(BaseEstimator, ClassifierMixin):

"""

Ensemble classifier for scikit-learn estimators.

Parameters

----------

clf : `iterable`

A list of scikit-learn classifier objects.

weights : `list` (default: `None`)

If `None`, the majority rule voting will be applied to the predicted class labels.

If a list of weights (`float` or `int`) is provided, the averaged raw probabilities (via `predict_proba`)

will be used to determine the most confident class label.

"""

def __init__(self, clfs, weights=None):

self.clfs = clfs

self.weights = weights

def fit(self, X, y):

"""

Fit the scikit-learn estimators.

Parameters

----------

X : numpy array, shape = [n_samples, n_features]

Training data

y : list or numpy array, shape = [n_samples]

Class labels

"""

for clf in self.clfs:

clf.fit(X, y)

def predict(self, X):

"""

Parameters

----------

X : numpy array, shape = [n_samples, n_features]

Returns

----------

maj : list or numpy array, shape = [n_samples]

Predicted class labels by majority rule

"""

self.classes_ = np.asarray([clf.predict(X) for clf in self.clfs])

if self.weights:

avg = self.predict_proba(X)

maj = np.apply_along_axis(lambda x: max(enumerate(x), key=operator.itemgetter(1))[0], axis=1, arr=avg)

else:

maj = np.asarray([np.argmax(np.bincount(self.classes_[:,c])) for c in range(self.classes_.shape[1])])

return maj

def predict_proba(self, X):

"""

Parameters

----------

X : numpy array, shape = [n_samples, n_features]

Returns

----------

avg : list or numpy array, shape = [n_samples, n_probabilities]

Weighted average probability for each class per sample.

"""

self.probas_ = [clf.predict_proba(X) for clf in self.clfs]

avg = np.average(self.probas_, axis=0, weights=self.weights)

return avg

np.random.seed(123)

eclf = EnsembleClassifier(clfs=[clf1, clf2, clf3], weights=[1,1,1])

for clf, label in zip([clf1, clf2, clf3, eclf], ['Logistic Regression', 'Random Forest', 'naive Bayes', 'Ensemble']):

scores = cross_validation.cross_val_score(clf, X, y, cv=5, scoring='accuracy')

print("Accuracy: %0.2f (+/- %0.2f) [%s]" % (scores.mean(), scores.std(), label))

Accuracy: 0.90 (+/- 0.05) [Logistic Regression]

Accuracy: 0.92 (+/- 0.05) [Random Forest]

Accuracy: 0.91 (+/- 0.04) [naive Bayes]

Accuracy: 0.95 (+/- 0.03) [Ensemble]

**Additional Note About the EnsembleClassifier Implementation: Class Labels vs. Probabilities**

You might be wondering why I implemented the EnsembleClassifier class so that it applies the majority voting purely on the class labels if no weights are provided and is the predicted probability values otherwise.

Let’s consider the following scenario:

**1) Prediction based on majority class labels:**

To achieve this behavior, initialize the EnsembleClassifier like this:

eclf = EnsembleClassifier(clfs=[clf1, clf2, clf3])

**2) Prediction based on predicted probabilities (equal weights, weights=[1,1,1])**

To achieve this behavior, initialize the EnsembleClassifier like this:

eclf = EnsembleClassifier(clfs=[clf1, clf2, clf3], weights=[1,1,1])

As we can see, the results are different depending on whether we apply a majority vote based on the class labels or take the average of the predicted probabilities. In general, I think it makes more sense to use the predicted probabilities (scenario 2). Here, the “very confident” classifier 1 overules the very unconfident classifiers 2 and 3.

The reason for the different behaviors is that not all classifiers in scikit-learn support the predict_proba method. In this case, the EnsembleClassifier can still be used just based on the class labels if no weights are provided as parameter.

**EnsembleClassifier - Tuning Weights**

Let’s get back to our weights parameter. Here, we will use a naive brute-force approach to find the optimal weights for each classifier to increase the prediction accuracy.

import pandas as pd

np.random.seed(123)

df = pd.DataFrame(columns=('w1', 'w2', 'w3', 'mean', 'std'))

i = 0

for w1 in range(1,4):

for w2 in range(1,4):

for w3 in range(1,4):

if len(set((w1,w2,w3))) == 1: # skip if all weights are equal

continue

eclf = EnsembleClassifier(clfs=[clf1, clf2, clf3], weights=[w1,w2,w3])

scores = cross_validation.cross_val_score(

estimator=eclf,

X=X,

y=y,

cv=5,

scoring='accuracy',

n_jobs=1)

df.loc[i] = [w1, w2, w3, scores.mean(), scores.std()]

i += 1

df.sort(columns=['mean', 'std'], ascending=False)

**EnsembleClassifier - Pipelines**

Of course, we can also use the EnsembleClassifier in Pipelines. This is especially useful if a certain classifier does a pretty good job on a certain feature subset or requires different preprocessing steps. For demonstration purposes, let us implement a simple ColumnSelector class.

class ColumnSelector(object):

"""

A feature selector for scikit-learn's Pipeline class that returns

specified columns from a numpy array.

"""

def __init__(self, cols):

self.cols = cols

def transform(self, X, y=None):

return X[:, self.cols]

def fit(self, X, y=None):

return self

from sklearn.pipeline import Pipeline

from sklearn.lda import LDA

pipe1 = Pipeline([

('sel', ColumnSelector([1])), # use only the 1st feature

('clf', GaussianNB())])

pipe2 = Pipeline([

('sel', ColumnSelector([0, 1])), # use the 1st and 2nd feature

('dim', LDA(n_components=1)), # Dimensionality reduction via LDA

('clf', LogisticRegression())])

eclf = EnsembleClassifier([pipe1, pipe2])

scores = cross_validation.cross_val_score(eclf, X, y, cv=5, scoring='accuracy')

print("Accuracy: %0.2f (+/- %0.2f) [%s]" % (scores.mean(), scores.std(), label))

Accuracy: 0.95 (+/- 0.03) [Ensemble]

pipe1 = Pipeline([

('sel', ColumnSelector([1])), # use only the 1st feature

('clf', RandomForestClassifier())])

pipe2 = Pipeline([

('sel', ColumnSelector([0, 1])), # use the 1st and 2nd feature

('dim', LDA(n_components=1)), # Dimensionality reduction via LDA

('clf', LogisticRegression())])

pipe3 = Pipeline([

('eclf', EnsembleClassifier([pipe1, pipe2])),

])

parameters = {

'eclf__clfs__dim__n_components':(1,1),

}

grid_search = GridSearchCV(pipe3, parameters, n_jobs=-1, cv=5, verbose=5, refit=True, scoring=None)

grid_search.fit(X, y)

**Some Final Words**

When we applied the EnsembleClassifier to the iris example above, the results surely looked nice. But we have to keep in mind that this is just a toy example. The majority rule voting approach might not always work so well in practice, especially if the ensemble consists of more “weak” than “strong” classification models. Also, although we used a cross-validation approach to overcome the overfitting challenge, please always keep a spare validation dataset to evaluate the results.

Anyway, if you are interested in those approaches, I added them to my mlxtend Python module; in mlxtend (short for “machine learning library extensions”), I collect certain things that I personally find useful but are not available in other packages yet.

- 11/01/2015
- 148
- 0 Like

## Implementing a Weighted Majority Rule Ensemble Classifier in scikit-learn

- 11/01/2015
- 148
- 0 Like

#### Sebastian Raschka

Ph.D. Candidate at Michigan State University

Opinions expressed by Gladwin Analytics members are their own.

#### Top Authors

Republished with Author’s Permission – Originally published on >Sebastian Raschka Blog, dated >11 Jan 2015.

Here, I want to present a simple and conservative approach of implementing a weighted majority rule ensemble classifier in scikit-learn that yielded remarkably good results when I tried it in a kaggle competition. For me personally, kaggle competitions are just a nice way to try out and compare different approaches and ideas – basically an opportunity to learn in a controlled environment with nice datasets.

Of course, there are other implementations of more sophisticated ensemble methods in scikit-learn, such as bagging classifiers, random forests, or the famous AdaBoost algorithm. However, as far as I am concerned, they all require the usage of a common “base classifier.”

In contrast, my motivation for the following approach was to combine conceptually different machine learning classifiers and use a majority vote rule. The reason for this was that I had trained a set of equally well performing models, and I wanted to balance out their individual weaknesses.

**Sections**

~ Sections

~ Classifying Iris Flowers Using Different Classification Models

~ Implementing the Majority Voting Rule Ensemble Classifier

~ Additional Note About the EnsembleClassifier Implementation: Class Labels vs. Probabilities

1) Prediction based on majority class labels:

2) Prediction based on predicted probabilities (equal weights, weights=[1,1,1])

~ EnsembleClassifier - Tuning Weights

~ EnsembleClassifier - Pipelines

~ Some Final Words

**Classifying Iris Flowers Using Different Classification Models**

For a simple example, let us use three different classification models to classify the samples in the Iris dataset: Logistic regression, a naive Bayes classifier with a Gaussian kernel, and a random forest classifier – an ensemble method itself. At this point, let’s not worry about preprocessing the data and training and test sets. Also, we will only use 2 feature columns (sepal width and petal height) to make the classification problem harder.

from sklearn import datasets

iris = datasets.load_iris()

X, y = iris.data[:, 1:3], iris.target

from sklearn import cross_validation

from sklearn.linear_model import LogisticRegression

from sklearn.naive_bayes import GaussianNB

from sklearn.ensemble import RandomForestClassifier

import numpy as np

np.random.seed(123)

clf1 = LogisticRegression()

clf2 = RandomForestClassifier()

clf3 = GaussianNB()

print('5-fold cross validation:\n')

for clf, label in zip([clf1, clf2, clf3], ['Logistic Regression', 'Random Forest', 'naive Bayes']):

scores = cross_validation.cross_val_score(clf, X, y, cv=5, scoring='accuracy')

print("Accuracy: %0.2f (+/- %0.2f) [%s]" % (scores.mean(), scores.std(), label))

5-fold cross validation:

Accuracy: 0.90 (+/- 0.05) [Logistic Regression]

Accuracy: 0.92 (+/- 0.05) [Random Forest]

Accuracy: 0.91 (+/- 0.04) [naive Bayes]

As we can see from the cross-validation results above, the performance of the three models is almost equal.

**Implementing the Majority Voting Rule Ensemble Classifier**

Now, we will implement a simple EnsembleClassifier class that allows us to combine the three different classifiers. We define a predict method that let’s us simply take the majority rule of the predictions by the classifiers. E.g., if the prediction for a sample is

- classifier 1 -> class 1

- classifier 2 -> class 1

- classifier 3 -> class 2

we would classify the sample as “class 1.”

Furthermore, we add a weights parameter, which let’s us assign a specific weight to each classifier. In order to work with the weights, we collect the predicted class probabilities for each classifier, multiply it by the classifier weight, and take the average. Based on these weighted average probabilties, we can then assign the class label.

To illustrate this with a simple example, let’s assume we have 3 classifiers and a 3-class classification problems where we assign equal weights to all classifiers (the default): w1=1, w2=1, w3=1.

The weighted average probabilities for a sample would then be calculated as follows:

We can see in the table above that class 2 has the highest weighted average probability, thus we classify the sample as class 2.

Now, let’s put it into code and apply it to our Iris classification.

from sklearn.base import BaseEstimator

from sklearn.base import ClassifierMixin

import numpy as np

import operator

class EnsembleClassifier(BaseEstimator, ClassifierMixin):

"""

Ensemble classifier for scikit-learn estimators.

Parameters

----------

clf : `iterable`

A list of scikit-learn classifier objects.

weights : `list` (default: `None`)

If `None`, the majority rule voting will be applied to the predicted class labels.

If a list of weights (`float` or `int`) is provided, the averaged raw probabilities (via `predict_proba`)

will be used to determine the most confident class label.

"""

def __init__(self, clfs, weights=None):

self.clfs = clfs

self.weights = weights

def fit(self, X, y):

"""

Fit the scikit-learn estimators.

Parameters

----------

X : numpy array, shape = [n_samples, n_features]

Training data

y : list or numpy array, shape = [n_samples]

Class labels

"""

for clf in self.clfs:

clf.fit(X, y)

def predict(self, X):

"""

Parameters

----------

X : numpy array, shape = [n_samples, n_features]

Returns

----------

maj : list or numpy array, shape = [n_samples]

Predicted class labels by majority rule

"""

self.classes_ = np.asarray([clf.predict(X) for clf in self.clfs])

if self.weights:

avg = self.predict_proba(X)

maj = np.apply_along_axis(lambda x: max(enumerate(x), key=operator.itemgetter(1))[0], axis=1, arr=avg)

else:

maj = np.asarray([np.argmax(np.bincount(self.classes_[:,c])) for c in range(self.classes_.shape[1])])

return maj

def predict_proba(self, X):

"""

Parameters

----------

X : numpy array, shape = [n_samples, n_features]

Returns

----------

avg : list or numpy array, shape = [n_samples, n_probabilities]

Weighted average probability for each class per sample.

"""

self.probas_ = [clf.predict_proba(X) for clf in self.clfs]

avg = np.average(self.probas_, axis=0, weights=self.weights)

return avg

np.random.seed(123)

eclf = EnsembleClassifier(clfs=[clf1, clf2, clf3], weights=[1,1,1])

for clf, label in zip([clf1, clf2, clf3, eclf], ['Logistic Regression', 'Random Forest', 'naive Bayes', 'Ensemble']):

scores = cross_validation.cross_val_score(clf, X, y, cv=5, scoring='accuracy')

print("Accuracy: %0.2f (+/- %0.2f) [%s]" % (scores.mean(), scores.std(), label))

Accuracy: 0.90 (+/- 0.05) [Logistic Regression]

Accuracy: 0.92 (+/- 0.05) [Random Forest]

Accuracy: 0.91 (+/- 0.04) [naive Bayes]

Accuracy: 0.95 (+/- 0.03) [Ensemble]

**Additional Note About the EnsembleClassifier Implementation: Class Labels vs. Probabilities**

You might be wondering why I implemented the EnsembleClassifier class so that it applies the majority voting purely on the class labels if no weights are provided and is the predicted probability values otherwise.

Let’s consider the following scenario:

**1) Prediction based on majority class labels:**

To achieve this behavior, initialize the EnsembleClassifier like this:

eclf = EnsembleClassifier(clfs=[clf1, clf2, clf3])

**2) Prediction based on predicted probabilities (equal weights, weights=[1,1,1])**

To achieve this behavior, initialize the EnsembleClassifier like this:

eclf = EnsembleClassifier(clfs=[clf1, clf2, clf3], weights=[1,1,1])

As we can see, the results are different depending on whether we apply a majority vote based on the class labels or take the average of the predicted probabilities. In general, I think it makes more sense to use the predicted probabilities (scenario 2). Here, the “very confident” classifier 1 overules the very unconfident classifiers 2 and 3.

The reason for the different behaviors is that not all classifiers in scikit-learn support the predict_proba method. In this case, the EnsembleClassifier can still be used just based on the class labels if no weights are provided as parameter.

**EnsembleClassifier - Tuning Weights**

Let’s get back to our weights parameter. Here, we will use a naive brute-force approach to find the optimal weights for each classifier to increase the prediction accuracy.

import pandas as pd

np.random.seed(123)

df = pd.DataFrame(columns=('w1', 'w2', 'w3', 'mean', 'std'))

i = 0

for w1 in range(1,4):

for w2 in range(1,4):

for w3 in range(1,4):

if len(set((w1,w2,w3))) == 1: # skip if all weights are equal

continue

eclf = EnsembleClassifier(clfs=[clf1, clf2, clf3], weights=[w1,w2,w3])

scores = cross_validation.cross_val_score(

estimator=eclf,

X=X,

y=y,

cv=5,

scoring='accuracy',

n_jobs=1)

df.loc[i] = [w1, w2, w3, scores.mean(), scores.std()]

i += 1

df.sort(columns=['mean', 'std'], ascending=False)

**EnsembleClassifier - Pipelines**

Of course, we can also use the EnsembleClassifier in Pipelines. This is especially useful if a certain classifier does a pretty good job on a certain feature subset or requires different preprocessing steps. For demonstration purposes, let us implement a simple ColumnSelector class.

class ColumnSelector(object):

"""

A feature selector for scikit-learn's Pipeline class that returns

specified columns from a numpy array.

"""

def __init__(self, cols):

self.cols = cols

def transform(self, X, y=None):

return X[:, self.cols]

def fit(self, X, y=None):

return self

from sklearn.pipeline import Pipeline

from sklearn.lda import LDA

pipe1 = Pipeline([

('sel', ColumnSelector([1])), # use only the 1st feature

('clf', GaussianNB())])

pipe2 = Pipeline([

('sel', ColumnSelector([0, 1])), # use the 1st and 2nd feature

('dim', LDA(n_components=1)), # Dimensionality reduction via LDA

('clf', LogisticRegression())])

eclf = EnsembleClassifier([pipe1, pipe2])

scores = cross_validation.cross_val_score(eclf, X, y, cv=5, scoring='accuracy')

print("Accuracy: %0.2f (+/- %0.2f) [%s]" % (scores.mean(), scores.std(), label))

Accuracy: 0.95 (+/- 0.03) [Ensemble]

pipe1 = Pipeline([

('sel', ColumnSelector([1])), # use only the 1st feature

('clf', RandomForestClassifier())])

pipe2 = Pipeline([

('sel', ColumnSelector([0, 1])), # use the 1st and 2nd feature

('dim', LDA(n_components=1)), # Dimensionality reduction via LDA

('clf', LogisticRegression())])

pipe3 = Pipeline([

('eclf', EnsembleClassifier([pipe1, pipe2])),

])

parameters = {

'eclf__clfs__dim__n_components':(1,1),

}

grid_search = GridSearchCV(pipe3, parameters, n_jobs=-1, cv=5, verbose=5, refit=True, scoring=None)

grid_search.fit(X, y)

**Some Final Words**

When we applied the EnsembleClassifier to the iris example above, the results surely looked nice. But we have to keep in mind that this is just a toy example. The majority rule voting approach might not always work so well in practice, especially if the ensemble consists of more “weak” than “strong” classification models. Also, although we used a cross-validation approach to overcome the overfitting challenge, please always keep a spare validation dataset to evaluate the results.

Anyway, if you are interested in those approaches, I added them to my mlxtend Python module; in mlxtend (short for “machine learning library extensions”), I collect certain things that I personally find useful but are not available in other packages yet.

- 11/01/2015
- 148
- 0 Like

## Sebastian Raschka

Ph.D. Candidate at Michigan State University

Opinions expressed by Gladwin Analytics members are their own.