Machine & Deep Learning Reference

Regression, classification, clustering, evaluation, and the neural network architectures — condensed from the Machine Learning A-Z and Deep Learning A-Z course material.

1. Foundations

• Supervised learning — learns from labeled examples (input → known output): regression and classification.
• Unsupervised learning — finds structure in unlabeled data: clustering, dimensionality reduction.
• Train / validation / test split — fit on train, tune hyperparameters on validation, report final performance on test — never let the model see test data during training.
• Feature scaling — standardization (zero mean, unit variance) or normalization (0–1 range); required for distance-based and gradient-based models (KNN, SVM, neural nets), not for tree-based models.

2. Regression

from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
model = LinearRegression().fit(X_train, y_train)
predictions = model.predict(X_test)

3. Classification

• Logistic Regression — linear decision boundary, outputs class probabilities via sigmoid.
• K-Nearest Neighbors — classifies by majority vote of the k closest training points; sensitive to feature scale.
• Support Vector Machine — finds the maximum-margin boundary; kernels (RBF, polynomial) handle non-linear separation.
• Naive Bayes — probabilistic, assumes feature independence; fast and strong on text classification.
• Decision Tree / Random Forest — same as regression variants but for discrete classes; forests reduce a single tree's overfitting.
• Gradient Boosting (XGBoost/LightGBM) — builds trees sequentially, each correcting the previous ensemble's errors — typically the strongest tabular-data performer.

from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report

clf = RandomForestClassifier(n_estimators=200, random_state=42)
clf.fit(X_train, y_train)
print(classification_report(y_test, clf.predict(X_test)))

4. Clustering

• K-Means — partitions data into k clusters by minimizing distance to centroids; use the elbow method or silhouette score to pick k.
• Hierarchical Clustering — builds a dendrogram of nested clusters; no need to fix k upfront.
• DBSCAN — density-based, finds arbitrarily shaped clusters and flags outliers as noise.

5. Model Evaluation

Watch for the bias–variance trade-off: an underfit model has high bias (too simple), an overfit model has high variance (memorizes training noise).

6. Neural Networks (ANN)

A network of layered neurons: each neuron computes a weighted sum of inputs plus a bias, then applies a non-linear activation (ReLU, sigmoid, tanh). Training uses forward propagation to predict, a loss function to score error, and backpropagation with gradient descent (or Adam) to update weights.

from tensorflow import keras

model = keras.Sequential([
    keras.layers.Dense(64, activation="relu", input_shape=(n_features,)),
    keras.layers.Dense(32, activation="relu"),
    keras.layers.Dense(1, activation="sigmoid")
])
model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"])
model.fit(X_train, y_train, epochs=50, batch_size=32, validation_split=0.2)

7. CNNs & RNNs

• Convolutional Neural Networks — convolution + pooling layers extract spatial features from images; used for image classification, object detection.
• Recurrent Neural Networks / LSTM / GRU — maintain hidden state across a sequence; used for time series and text, where order matters. LSTM/GRU gates address vanishing-gradient issues in plain RNNs.
• Overfitting controls — dropout, early stopping, batch normalization, data augmentation (especially for CNNs on small image sets).

8. Feature Engineering

• Encoding categoricals — one-hot encoding for low-cardinality nominal fields; ordinal/label encoding when order is meaningful; target/frequency encoding for high-cardinality fields where one-hot would explode dimensionality.
• Handling missing values — drop rows/columns above a missingness threshold, or impute (mean/median for numeric, mode for categorical, or a model-based imputer) — and consider adding an "was_missing" flag when missingness itself is informative.
• Feature scaling — standardization/normalization, required for distance- and gradient-based models (see §1).
• Interaction & derived features — ratios, differences, date parts (day-of-week, is_holiday), and domain-specific combinations often beat raw inputs alone.
• Dimensionality reduction — PCA projects correlated numeric features onto uncorrelated components that explain the most variance; useful before distance-based models or for visualization.

from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline

preprocess = ColumnTransformer([
    ("num", StandardScaler(), ["age", "income"]),
    ("cat", OneHotEncoder(handle_unknown="ignore"), ["region"])
])
pipe = Pipeline([("prep", preprocess), ("model", RandomForestClassifier())])
pipe.fit(X_train, y_train)

9. Hyperparameter Tuning

from sklearn.model_selection import GridSearchCV

param_grid = {"n_estimators": [100, 200, 400], "max_depth": [None, 10, 20]}
search = GridSearchCV(RandomForestClassifier(), param_grid, cv=5, scoring="f1")
search.fit(X_train, y_train)
print(search.best_params_, search.best_score_)

10. Imbalanced Data

When one class dominates (fraud detection, rare disease), accuracy stops being meaningful — a model predicting the majority class every time can still score 99%.

• Resampling — oversample the minority class (e.g. SMOTE, which synthesizes new minority examples) or undersample the majority class.
• Class weights — penalize misclassifying the minority class more heavily during training, without touching the dataset itself.
• Better metrics — precision, recall, F1, and ROC-AUC/PR-AUC tell the real story where accuracy hides it; precision-recall curves are more informative than ROC when the positive class is rare.

11. Model Deployment

• Batch inference — score a large dataset on a schedule (e.g. nightly) and write predictions to a table; simplest, fits most business-intelligence use cases.
• Real-time inference — serve predictions behind an API for a live request (fraud check at checkout); needs low-latency serving infrastructure.
• Model registry — versions trained models with their metrics and lineage, so a deployed model can be traced back to the data and code that produced it.
• Drift monitoring — track input feature distributions and prediction accuracy over time; real-world data shifts away from training data, silently degrading a model that was never retrained.

import joblib

joblib.dump(pipe, "model.joblib")
loaded = joblib.load("model.joblib")
loaded.predict(new_data)