We will provide a walk-through example of how you can choose the most important features. For this example, we will work with a classification problem but can be extended to regression cases too by adjusting the parameters of the function.

We will work with the breast-cancer dataset. Let’s start:

import pandas as pd import numpy as np from scipy import stats from sklearn.ensemble import RandomForestClassifier from sklearn.feature_selection import SelectFpr, chi2, SelectKBest, SelectFwe, f_classif, SelectFdr import matplotlib.pyplot as plt %matplotlib inline # https://www.kaggle.com/uciml/breast-cancer-wisconsin-data?select=data.csv df = pd.read_csv("data.csv") # replace M with 1 and B with 0 my_map = { 'M':1, 'B' :0 } df['diagnosis'] = df['diagnosis'].map(my_map) # remove the id column df.drop(['id'], axis=1, inplace=True) df

## Statistical Significant features with t-test

Since our target is binary, we can compare the values of the independent variables for each group (0,1) by applying `t-test`

.

my_important = [] for c in df.columns[1:]: pvalue = stats.ttest_ind(df.loc[df.diagnosis==1], df.loc[df.diagnosis==0])[1] if pvalue<0.05: my_important.append(c) print(f'The variable {c} is statistically significant with a pvalue = {pvalue:.2}') else: print(f'The variable {c} is NOT statistically significant')

And we get:

The variable radius_mean is statistically significant with a pvalue = 8.5e-96 The variable texture_mean is statistically significant with a pvalue = 4.1e-25 The variable perimeter_mean is statistically significant with a pvalue = 8.4e-101 The variable area_mean is statistically significant with a pvalue = 4.7e-88 The variable smoothness_mean is statistically significant with a pvalue = 1.1e-18 The variable compactness_mean is statistically significant with a pvalue = 3.9e-56 The variable concavity_mean is statistically significant with a pvalue = 1e-83 The variable concave points_mean is statistically significant with a pvalue = 7.1e-116 The variable symmetry_mean is statistically significant with a pvalue = 5.7e-16 The variable fractal_dimension_mean is NOT statistically significant The variable radius_se is statistically significant with a pvalue = 9.7e-50 The variable texture_se is NOT statistically significant The variable perimeter_se is statistically significant with a pvalue = 1.7e-47 The variable area_se is statistically significant with a pvalue = 5.9e-46 The variable smoothness_se is NOT statistically significant The variable compactness_se is statistically significant with a pvalue = 1e-12 The variable concavity_se is statistically significant with a pvalue = 8.3e-10 The variable concave points_se is statistically significant with a pvalue = 3.1e-24 The variable symmetry_se is NOT statistically significant The variable fractal_dimension_se is NOT statistically significant The variable radius_worst is statistically significant with a pvalue = 8.5e-116 The variable texture_worst is statistically significant with a pvalue = 1.1e-30 The variable perimeter_worst is statistically significant with a pvalue = 5.8e-119 The variable area_worst is statistically significant with a pvalue = 2.8e-97 The variable smoothness_worst is statistically significant with a pvalue = 6.6e-26 The variable compactness_worst is statistically significant with a pvalue = 7.1e-55 The variable concavity_worst is statistically significant with a pvalue = 2.5e-72 The variable concave points_worst is statistically significant with a pvalue = 2e-124 The variable symmetry_worst is statistically significant with a pvalue = 3e-25 The variable fractal_dimension_worst is statistically significant with a pvalue = 2.3e-15

So, the statistically significant variables are the:

my_important

['radius_mean', 'texture_mean', 'perimeter_mean', 'area_mean', 'smoothness_mean', 'compactness_mean', 'concavity_mean', 'concave points_mean', 'symmetry_mean', 'radius_se', 'perimeter_se', 'area_se', 'compactness_se', 'concavity_se', 'concave points_se', 'radius_worst', 'texture_worst', 'perimeter_worst', 'area_worst', 'smoothness_worst', 'compactness_worst', 'concavity_worst', 'concave points_worst', 'symmetry_worst', 'fractal_dimension_worst']

## Random Forest Feature Importance

We can also run a model, like Random Forest and see which are the most important features.

clf = RandomForestClassifier( n_estimators=50) X = df.drop(['diagnosis'], axis=1) y = df.diagnosis model = clf.fit(X,y) feat_importances = pd.DataFrame(model.feature_importances_, index=X.columns, columns=["Importance"]) feat_importances.sort_values(by='Importance', ascending=False, inplace=True)

Let’s see the 10 most important features according to Random Forest:

feat_importances.index[0:10]

Index(['concave points_worst', 'perimeter_worst', 'radius_worst', 'concave points_mean', 'area_worst', 'perimeter_mean', 'area_mean', 'concavity_worst', 'area_se', 'perimeter_se'], dtype='object')

## Feature Selection with Scikit-Learn

We can work with the `scikit-learn`

. You can find more details at the documentation. We will provide some examples:

**k-best**

It selects the k most important features. In our case, we will work with the chi-square test. Keep in mind that the **new_data **are the final data after we removed the non-significant variables.

selector = SelectKBest(score_func=chi2, k=5) new_data = selector.fit_transform(X, y) mask = selector.get_support() new_features = X.columns[mask] new_features

Index(['perimeter_mean', 'area_mean', 'area_se', 'perimeter_worst', 'area_worst'], dtype='object')

**Fwe**

This is similar to what we did at the beginning with the t-test. It can be done with a `chi-square test`

or with `ANOVA`

(wherein binary case is the same as the t-test)

# chi-ssquare selector = SelectFwe(score_func=chi2, alpha=0.05) new_data = selector.fit_transform(X, y) mask = selector.get_support() new_features = X.columns[mask] new_features

Index(['radius_mean', 'texture_mean', 'perimeter_mean', 'area_mean', 'concavity_mean', 'concave points_mean', 'radius_se', 'perimeter_se', 'area_se', 'radius_worst', 'texture_worst', 'perimeter_worst', 'area_worst', 'compactness_worst', 'concavity_worst', 'concave points_worst'], dtype='object')

# ANOVA selector = SelectFwe(score_func=f_classif, alpha=0.05) new_data = selector.fit_transform(X, y) mask = selector.get_support() new_features = X.columns[mask] new_features

Index(['radius_mean', 'texture_mean', 'perimeter_mean', 'area_mean', 'smoothness_mean', 'compactness_mean', 'concavity_mean', 'concave points_mean', 'symmetry_mean', 'radius_se', 'perimeter_se', 'area_se', 'compactness_se', 'concavity_se', 'concave points_se', 'radius_worst', 'texture_worst', 'perimeter_worst', 'area_worst', 'smoothness_worst', 'compactness_worst', 'concavity_worst', 'concave points_worst', 'symmetry_worst', 'fractal_dimension_worst'], dtype='object')

**FDR**

The false discovery rate (FDR) takes into account the multiple comparisons.

selector = SelectFdr(chi2, alpha=0.05) new_data = selector.fit_transform(X, y) mask = selector.get_support() new_features = X.columns[mask] new_features

Index(['radius_mean', 'texture_mean', 'perimeter_mean', 'area_mean', 'compactness_mean', 'concavity_mean', 'concave points_mean', 'radius_se', 'perimeter_se', 'area_se', 'radius_worst', 'texture_worst', 'perimeter_worst', 'area_worst', 'compactness_worst', 'concavity_worst', 'concave points_worst'], dtype='object')

**From the Model**

We can keep the most important features derived from the model. Let’s consider again the random forest:

from sklearn.feature_selection import SelectFromModel selector = SelectFromModel(estimator=RandomForestClassifier(n_estimators=50)).fit(X, y) mask = selector.get_support() new_features = X.columns[mask] new_features

ndex(['perimeter_mean', 'area_mean', 'concavity_mean', 'concave points_mean', 'radius_worst', 'perimeter_worst', 'area_worst', 'concave points_worst'], dtype='object')

## Discussion

This was an example of how you can get a good idea of which variables are the most important for your model. Keep in mind that you should always run exploratory data analysis. For binary cases, a box plot is always an appropriate plot. Let’s see the box plot of the `concave points_worst`

for both benign (0) and malignant tumors (1). As we can see, the difference between the groups seems to be significant and this what we confirmed by running the tests above.

Finally, you can find a detailed explanation of the Chi-Square test.