4 mins read
## Implementing Logistic Regression from Scratch

### Step-1: Understanding the Sigmoid function

### Step 2: The Loss Function

### Step 3: Gradient descent

## Putting it all together

### Now, it’s time to test our implementation.

## Conclusion

The objective of this tutorial is to implement our own Logistic Regression from scratch. This is going to be different from our previous tutorial on the same topic where we used built-in methods to create the function.

Logistic regression is a classic method mainly used for Binary Classification problems. even though it can be used for multi-class classification problems with some modification, in this article we will perform binary classification.

Step by step we will break down the algorithm to understand its inner working and finally will create our own class.

The sigmoid function in logistic regression returns a probability value that can then be mapped to two or more discrete classes. Given the set of input variables, our goal is to assign that data point to a category (either 1 or 0). The sigmoid function outputs the probability of the input points belonging to one of the classes.

```
#Defining a sigmoid function
def sigmoid(z):
op = 1/(1 + np.exp(-z))
return op
```

The loss function consists of parameters/weights, when we say we want to optimize a loss function by this we simply refer to finding the best values of the parameters/weights.

The loss function for Logistic Regression is defined as:

```
#Loss Function
def loss(h, y):
return (-y * np.log(h) - (1 - y) * np.log(1 - h)).mean()
```

The Gradient descent is just the derivative of the loss function with respect to its weights.

We get this after we find the derivative of the loss function:

```
#Gradient_descent
def gradient_descent(X, h, y):
return np.dot(X.T, (h - y)) / y.shape[0]
```

The weights are updated by subtracting the derivative (gradient descent) times the learning rate. Updating the weights:

Here – **alpha** is the learning rate.

Let’s create a class to compile the steps mentioned above. Here’s the complete code for implementing Logistic Regression from scratch. We have worked with the Python numpy module for this implementation.

```
#import required modules
import numpy as np
class LogisticRegression:
def __init__(self,x,y):
self.intercept = np.ones((x.shape[0], 1))
self.x = np.concatenate((self.intercept, x), axis=1)
self.weight = np.zeros(self.x.shape[1])
self.y = y
#Sigmoid method
def sigmoid(self, x, weight):
z = np.dot(x, weight)
return 1 / (1 + np.exp(-z))
#method to calculate the Loss
def loss(self, h, y):
return (-y * np.log(h) - (1 - y) * np.log(1 - h)).mean()
#Method for calculating the gradients
def gradient_descent(self, X, h, y):
return np.dot(X.T, (h - y)) / y.shape[0]
def fit(self, lr , iterations):
for i in range(iterations):
sigma = self.sigmoid(self.x, self.weight)
loss = self.loss(sigma,self.y)
dW = self.gradient_descent(self.x , sigma, self.y)
#Updating the weights
self.weight -= lr * dW
return print('fitted successfully to data')
#Method to predict the class label.
def predict(self, x_new , treshold):
x_new = np.concatenate((self.intercept, x_new), axis=1)
result = self.sigmoid(x_new, self.weight)
result = result >= treshold
y_pred = np.zeros(result.shape[0])
for i in range(len(y_pred)):
if result[i] == True:
y_pred[i] = 1
else:
continue
return y_pred
```

To implement the Algorithm we defined a `fit`

method that requires the learning rate and the number of iterations as the input arguments.

The above class can be initialized by providing the input data and the target values.

```
from sklearn.datasets import load_breast_cancer
#Loading the data
data = load_breast_cancer()
#Preparing the data
x = data.data
y = data.target
#creating the class Object
regressor = LogisticRegression(x,y)
regressor.fit(0.1 , 5000)
y_pred = regressor.predict(x,0.5)
print('accuracy -> {}'.format(sum(y_pred == y) / y.shape[0]))
```

**Output:**

```
fitted successfully to data
accuracy -> 0.9209138840070299
```

Our implemented model achieved an accuracy of 92%, not bad.

You can find the notebook for this tutorial here.

This article was all about implementing a Logistic Regression Model from scratch to perform a binary classification task. We also unfold the inner working of the regression algorithm by coding it from 0.

Source: