Building Classifiers from Scratch: A Hands-On Guide to Logistic Regression and MLP for Text Classification

Introduction: Why Build Classifiers from Scratch?

In the era of AI-driven apps and large language models, understanding the fundamentals of classification algorithms is more relevant than ever. Whether you're analyzing tweets about the latest gaming release or classifying customer feedback for a fintech app, the core techniques remain the same. This tutorial walks you through building logistic regression and a multilayer perceptron (MLP) from scratch to classify text paragraphs from classic authors: Fyodor Dostoyevsky, Arthur Conan Doyle, and Jane Austen. By the end, you'll have a solid grasp of gradient descent, backpropagation, and model evaluation—skills essential for any data science role.

1. Data Preprocessing and Example Construction

Before feeding text into any model, we must clean and structure it. Start by removing punctuation, irrelevant symbols, URLs, and numbers. For the provided Project Gutenberg texts, also strip the boilerplate at the beginning of each file. Then, split each document into paragraphs. Each paragraph becomes one training example. Discard any non-paragraph text. Report the total number of examples per category—this ensures balanced datasets. For instance, you might get 500 paragraphs from Dostoyevsky, 600 from Doyle, and 450 from Austen. Use Python's re module for efficient cleaning.

import re
def clean_text(text):
    text = re.sub(r'http\S+', '', text)  # remove URLs
    text = re.sub(r'[^\w\s]', '', text) # remove punctuation
    text = re.sub(r'\d+', '', text)      # remove numbers
    return text.lower()

2. Data Split and Feature Extraction with TF-IDF

Split your paragraphs into training (80%) and testing (20%) sets. Use only training data to build a vocabulary and compute TF-IDF features. TF-IDF (Term Frequency-Inverse Document Frequency) highlights words that are important to a document but not too common across all documents. This is crucial for distinguishing between authors' writing styles. For example, words like 'elementary' might be frequent in Doyle's Sherlock Holmes stories, while 'soul' appears more in Dostoyevsky.

from sklearn.feature_extraction.text import TfidfVectorizer
vectorizer = TfidfVectorizer(max_features=5000)
X_train = vectorizer.fit_transform(train_paragraphs)
X_test = vectorizer.transform(test_paragraphs)

3. Logistic Regression with L2 Regularization

3.1 Deriving the Gradient

Given the cost function J with softmax and L2 regularization, we need to compute the gradient with respect to weight matrix w_k. The cost for one sample is:

J = - (1/N) * sum_i sum_k y_ik * log(softmax(f_k)) + lambda * sum_j w_kj^2

Where f_k is the score for class k. The softmax function is p_k = exp(f_k) / sum_c exp(f_c). The gradient for the weight vector of class k (for a single sample) is:

dJ/dw_k = (p_k - y_k) * x + 2 * lambda * w_k

This result is obtained by applying the chain rule and noting that the derivative of softmax is p_k * (1 - p_k) for the correct class and -p_k * p_c for others. The regularization term adds 2 * lambda * w_k.

3.2 Mini-Batch Gradient Descent Implementation

Mini-batch gradient descent updates weights using a subset of training data. Implement a function that iterates over batches, computes the average gradient, and updates w with a learning rate. Plot training loss over epochs to monitor convergence.

def mini_batch_gd(X, y, lr, lambda_reg, batch_size, epochs):
    n_features = X.shape[1]
    n_classes = y.shape[1]
    w = np.zeros((n_features, n_classes))
    for epoch in range(epochs):
        indices = np.random.permutation(len(X))
        X_shuffled = X[indices]
        y_shuffled = y[indices]
        for i in range(0, len(X), batch_size):
            X_batch = X_shuffled[i:i+batch_size]
            y_batch = y_shuffled[i:i+batch_size]
            scores = X_batch @ w
            probs = softmax(scores)
            grad = (X_batch.T @ (probs - y_batch)) / batch_size + 2 * lambda_reg * w
            w -= lr * grad
    return w

3.3 Stochastic Gradient Descent Implementation

Stochastic gradient descent (SGD) updates weights after each single example. This can lead to faster convergence but noisier gradients. Implement a function that loops over each example and updates w. Compare convergence plots with mini-batch: SGD typically shows more fluctuation but may escape local minima.

def sgd(X, y, lr, lambda_reg, epochs):
    n_samples = X.shape[0]
    n_features = X.shape[1]
    n_classes = y.shape[1]
    w = np.zeros((n_features, n_classes))
    for epoch in range(epochs):
        for i in range(n_samples):
            xi = X[i].reshape(1, -1)
            yi = y[i].reshape(1, -1)
            scores = xi @ w
            probs = softmax(scores)
            grad = xi.T @ (probs - yi) + 2 * lambda_reg * w
            w -= lr * grad
    return w

4. Multilayer Perceptron (MLP) with Backpropagation

4.1 Model Architecture

Build a two-layer neural network: input layer (TF-IDF features), one hidden layer with ReLU activation, and output layer with softmax. Choose the number of hidden neurons via cross-validation (e.g., 128). Use cross-entropy loss and L2 regularization. The optimizer can be SGD with a learning rate of 0.01. Implement backpropagation manually: compute gradients for W1, b1, W2, b2 using chain rule.

4.2 Training and Validation Curves

Plot training and validation loss every 100 epochs. Observe if the model overfits; if so, increase regularization or reduce hidden neurons. Use early stopping based on validation loss.

for epoch in range(epochs):
    # forward pass
    z1 = X @ W1 + b1
    a1 = np.maximum(0, z1)  # ReLU
    z2 = a1 @ W2 + b2
    probs = softmax(z2)
    loss = cross_entropy(probs, y) + lambda_reg * (np.sum(W1**2) + np.sum(W2**2))
    # backward pass
    dz2 = probs - y
    dW2 = a1.T @ dz2 + 2 * lambda_reg * W2
    db2 = np.sum(dz2, axis=0)
    da1 = dz2 @ W2.T
    dz1 = da1 * (z1 > 0)  # ReLU derivative
    dW1 = X.T @ dz1 + 2 * lambda_reg * W1
    db1 = np.sum(dz1, axis=0)
    W1 -= lr * dW1
    b1 -= lr * db1
    W2 -= lr * dW2
    b2 -= lr * db2

5. Model Comparison and Analysis

Use cross-validation to select the best λ for logistic regression and the optimal number of hidden neurons for MLP. Report precision and recall for each author on test and validation sets. Typically, logistic regression performs well with high regularization (e.g., λ=0.1), while MLP may achieve higher accuracy but requires careful tuning. Analyze why: logistic regression is linear, so it captures simple patterns; MLP can model non-linearities but risks overfitting on small data. For example, MLP might better distinguish Dostoyevsky's complex sentences from Austen's structured prose.

6. Advanced Topics: Word2Vec and Softmax Properties

While not the focus of this tutorial, understanding related concepts like word embeddings enhances your NLP toolkit. For instance, the softmax function is invariant to constant offsets: softmax(x) = softmax(x + c). This property is used for numerical stability by subtracting the maximum score. Similarly, the sigmoid gradient can be expressed as σ'(x) = σ(x)(1 - σ(x)). These fundamentals appear in modern models like transformers and are tested in assignments like Cs584.

Conclusion

Building classifiers from scratch demystifies machine learning and prepares you for more advanced topics. Whether you're analyzing classic literature or building a recommendation system for a gaming app, these skills are invaluable. Experiment with different hyperparameters, compare algorithms, and always validate your results. Happy coding!