KNN for classification (of glass types)
We will make use of KNN algorithms to classify the type of glass.
What is covered?
- Explained KNN algorithm
- Exploring dataset using visualization - scatterplot,pairplot, heatmap (correlation matrix).
- Feature scaling
- Applying KNN to classify
- Distance metrics
- Finding the best K value
- It is an instance-based algorithm.
- As opposed to model-based algorithms which pre trains on the data, and discards the data. Instance-based algorithms retain the data to classify when a new data point is given.
- The distance metric is used to calculate its nearest neighbors (Euclidean, manhattan)
- Can solve classification(by determining the majority class of nearest neighbors) and regression problems (by determining the means of nearest neighbors).
- If the majority of the nearest neighbors of the new data point belong to a certain class, the model classifies the new data point to that class.
Fig.4:Classifying new data point using KNN
For example, in the above plot, Assuming k=5,
the black point (new data) can be classified as class 1(Blue), because 3 out 5 of its nearest neighbors belong to class 1.
Glass classification dataset . Download to follow along.
This is a Glass Identification Data Set from UCI. It contains 10 attributes including id. The response is glass type(discrete 7 values)
- Id number: 1 to 214 (removed from CSV file)
- RI: refractive index
- Na: Sodium (unit measurement: weight percent in corresponding oxide, as are attributes 4-10)
- Mg: Magnesium
- Al: Aluminum
- Si: Silicon
- K: Potassium
- Ca: Calcium
- Ba: Barium
- Fe: Iron
- Type of glass: (class attribute)
- 1 buildingwindowsfloatprocessed
- 2 buildingwindowsnonfloatprocessed
- 3 vehiclewindowsfloatprocessed
- 4 vehiclewindowsnonfloatprocessed (none in this database)
- 5 containers
- 6 tableware
- 7 headlamps
About Type 2,4 -> Float processed glass means they are made on a floating molten glass on a bed of molten metal, this gives the sheet uniform thickness and flat surfaces.
Load dependencies and data
#import dependencies import pandas as pd import numpy as np import matplotlib.pyplot as plt from sklearn.neighbors import KNeighborsClassifier from sklearn.model_selection import train_test_split from sklearn.preprocessing import StandardScaler import seaborn as sns from sklearn.metrics import classification_report, accuracy_score from sklearn.model_selection import cross_val_score
#load data df = pd.read_csv('./data/glass.csv') df.head()
# value count for glass types df.Type.value_counts()
2 76 1 70 7 29 3 17 5 13 6 9 Name: Type, dtype: int64
Data exploration and visualizaion
correlation matrix -
cor = df.corr() sns.heatmap(cor)
<matplotlib.axes._subplots.AxesSubplot at 0x1b1c6b4c248>
We can notice that Ca and K values don't affect Type that much.
Also Ca and RI are highly correlated, this means using only RI is enough.
So we can go ahead and drop Ca, and also K.(performed later)
Scatter plot of two features, and pairwise plot
<matplotlib.axes._subplots.AxesSubplot at 0x1b1c6c3cd48>
Suppose we consider only RI, and Na values for classification for glass type.
- From the above plot, We first calculate the nearest neighbors from the new data point to be calculated.
- If the majority of nearest neighbors belong to a particular class, say type 4, then we classify the data point as type 4.
But there are a lot more than two features based on which we can classify. So let us take a look at pairwise plot to capture all the features.
#pairwise plot of all the features sns.pairplot(df,hue='Type') plt.show()
The pairplot shows that the data is not linear and KNN can be applied to get nearest neighbors and classify the glass types
Scaling is necessary for distance-based algorithms such as KNN. This is to avoid higher weightage being assigned to data with a higher magnitude.
Using standard scaler we can scale down to unit variance.
z = (x - u) / s
where x -> value, u -> mean, s -> standard deviation
scaler = StandardScaler()
StandardScaler(copy=True, with_mean=True, with_std=True)
#perform transformation scaled_features = scaler.transform(df.drop('Type',axis=1)) scaled_features
array([[ 0.87286765, 0.28495326, 1.25463857, ..., -0.14576634, -0.35287683, -0.5864509 ], [-0.24933347, 0.59181718, 0.63616803, ..., -0.79373376, -0.35287683, -0.5864509 ], [-0.72131806, 0.14993314, 0.60142249, ..., -0.82894938, -0.35287683, -0.5864509 ], ..., [ 0.75404635, 1.16872135, -1.86551055, ..., -0.36410319, 2.95320036, -0.5864509 ], [-0.61239854, 1.19327046, -1.86551055, ..., -0.33593069, 2.81208731, -0.5864509 ], [-0.41436305, 1.00915211, -1.86551055, ..., -0.23732695, 3.01367739, -0.5864509 ]])
df_feat = pd.DataFrame(scaled_features,columns=df.columns[:-1]) df_feat.head()
- Drop features that are not required
- Use random state while splitting the data to ensure reproducibility and consistency
- Experiment with distance metrics - Euclidean, manhattan
dff = df_feat.drop(['Ca','K'],axis=1) #Removing features - Ca and K X_train,X_test,y_train,y_test = train_test_split(dff,df['Type'],test_size=0.3,random_state=45) #setting random state ensures split is same eveytime, so that the results are comparable
knn = KNeighborsClassifier(n_neighbors=4,metric='manhattan')
KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='manhattan', metric_params=None, n_jobs=None, n_neighbors=4, p=2, weights='uniform')
y_pred = knn.predict(X_test)
precision recall f1-score support 1 0.69 0.90 0.78 20 2 0.85 0.65 0.74 26 3 0.00 0.00 0.00 3 5 0.25 1.00 0.40 1 6 0.50 0.50 0.50 2 7 1.00 0.85 0.92 13 accuracy 0.74 65 macro avg 0.55 0.65 0.56 65 weighted avg 0.77 0.74 0.74 65
With this setup, We found the accuracy to be 73.84%
Finding the best K value
We can do this either -
- by plotting Accuracy
- or by plotting the error rate
Note that plotting both is not required, however both are plottted here to show as an example.
k_range = range(1,25) k_scores =  error_rate = for k in k_range: knn = KNeighborsClassifier(n_neighbors=k) #kscores - accuracy scores = cross_val_score(knn,dff,df['Type'],cv=5,scoring='accuracy') k_scores.append(scores.mean()) #error rate knn.fit(X_train,y_train) y_pred = knn.predict(X_test) error_rate.append(np.mean(y_pred!=y_test)) #plot k vs accuracy plt.plot(k_range,k_scores) plt.xlabel('value of k - knn algorithm') plt.ylabel('Cross validated accuracy score') plt.show() #plot k vs error rate plt.plot(k_range,error_rate) plt.xlabel('value of k - knn algorithm') plt.ylabel('Error rate') plt.show()
we can see that k=4 produces the most accurate results
- Manhattan distance produced better results (improved accuracy - more than 5%)
- Applying feature scaling improved accuracy by almost 5%.
- The best k value was found to be 4.
- Dropping ‘Ca’ produced better results by a bit, ‘K’ feature did not affect results in any way.
- Also, we noticed that RI and Ca are highly correlated, this makes sense as it was found that the Refractive index of glass was found to increase with the increase in Cao. (https://link.springer.com/article/10.1134/S1087659614030249)
Further improvements -
We can see that the model can be improved further so we get better accuracy. Some suggestions -
- Using KFold Cross-validation
- Try different algorithms to find the best one for this problem - (SVM, Random forest, etc)
Other Useful resources -
- K Nearest Neighbour Easily Explained with Implementation by Krish Naik - video
- KNN by sentdex -video
- KNN sklearn - docs
- Complete guide to K nearest neighbours - python and R - blog
- Why scaling is required in KNN and K-Means - blog
This is part of my #100daysofcode challenge, checkout my github to see the progress.