Logistic Regression

TL;DR — Linear regression is the wrong tool for a yes/no outcome: it happily predicts probabilities of 1.4 or −0.3, which are nonsense. Logistic regression fixes this by pushing the linear score z = β0 + β1x1 + … through the sigmoid σ(z) = 1/(1+e^−z), which squashes any real number into a probability in (0, 1). The magic identity is that logistic regression is linear in the log-odds: ln(p/(1−p)) = β0 + β1x1 + …. That single line unlocks the interview-favorite fact — exp(βᵢ) is an odds ratio: a one-unit rise in xᵢ multiplies the odds of the event by exp(βᵢ). So the odds ratio you meet in a 2×2 table is the natural output of a fitted logistic model. You fit it by maximum likelihood, which is the same thing as minimizing log-loss (binary cross-entropy) by gradient descent. Then you pick a threshold (default 0.5, but tune it for imbalance) to turn probabilities into decisions.

---

1. Simple explanation

Suppose you want to predict something that is either yes or no: will this email be spam, will this patient have a heart attack, will this loan default. The answer is a class, not a number. But it is useful to predict how likely the yes is — a probability between 0 and 1.

Why not just draw a straight line like ordinary regression? Because a straight line does not know where to stop. Feed it a large enough input and it will confidently predict a probability of 1.7, or −0.4. Those are not probabilities. We need something that bends the line so it flattens out near 0 on the left and near 1 on the right — an S-curve.

Analogy — a dimmer switch, not a light switch. A plain light switch is hard classification: on or off, nothing between. But real belief is a dimmer. As evidence for "yes" piles up, you turn the dial up toward fully-on (probability 1); as evidence for "no" piles up, you turn it down toward fully-off (probability 0). The sigmoid is that dimmer. No matter how hard you shove the dial, it never goes past fully-on or below fully-off. Logistic regression learns how much each piece of evidence should turn the dial.

The three questions this article answers: Why does a straight line fail for yes/no? How does the sigmoid fix it? And why does the fitted model hand you an odds ratio for free?

---

2. Diagram

          LINEAR SCORE                SIGMOID SQUASH            PROBABILITY
   x1 ─β1─┐                                                          │
   x2 ─β2─┤                          σ(z) = 1/(1+e^−z)               │
   x3 ─β3─┼──►  z = β0+β1x1+β2x2+…  ───────────────────►  p = P(y=1) ─┤─► decide
   ...    │      (any real number)      (bends to 0..1)   in (0,1)   │   p ≥ threshold?
   1 ─β0──┘                                                          │
                                                                     ▼
                                                              yes (1) / no (0)

   THE SIGMOID CURVE                         THE KEY IDENTITY
   p                                         logit(p) = ln( p / (1−p) ) = z
   1 ┤          .------                                = β0 + β1x1 + …
     │       .-'                             "LINEAR IN THE LOG-ODDS"
 0.5 ┤ - - -+- - - - - - -  ◄ z=0 → p=0.5
     │   .-'                                 exp(βi) = ODDS RATIO
   0 ┤.-'                                    (+1 in xi  ×  odds by exp(βi))
     └──────┼──────────  z
           z=0

---

3. How it works

3.1 Why linear regression fails for a yes/no outcome

Code the outcome as y = 1 (event) or y = 0 (no event) and fit a line ŷ = β0 + β1x. Two things break:

You could clip predictions to [0, 1], but that is a patch on a broken model. The honest fix is to change the model so it can only ever output a probability. That is what the sigmoid does.

3.2 The sigmoid — a squashing function

The sigmoid (logistic function) takes any real number z and returns a value strictly between 0 and 1:

σ(z) = 1 / (1 + e^−z)

The input z is the same linear score as before, z = β0 + β1x1 + β2x2 + …. So logistic regression is linear regression with a sigmoid wrapped around the output. The line still does the work of combining features; the sigmoid just guarantees the answer is a valid probability.

3.3 The logit — logistic regression is linear in the log-odds

Here is the identity that makes everything click. Start from p = σ(z) and solve for z:

p = 1/(1+e^−z)   ⟹   ln( p / (1−p) ) = z = β0 + β1x1 + β2x2 + …

The quantity ln(p/(1−p)) is the logit, also called the log-odds. Recall that odds = p/(1−p) (probability of yes divided by probability of no). So:

The model is a straight line — not in the probability, but in the log-odds. The sigmoid is exactly the inverse of the logit. logit takes a probability to the whole real line so a line can live there; sigmoid brings it back to (0, 1).

This is the key line of the whole topic. It is why logistic regression is a linear model even though its output curves, and it is the reason coefficients turn into odds ratios (§3.6).

3.4 Fitting by maximum likelihood = minimizing log-loss

We can't use least squares. Instead we ask: which coefficients make the observed labels most probable? That is maximum likelihood. For one example with true label y and predicted probability p, the likelihood is p if y=1 and 1−p if y=0, written compactly as p^y (1−p)^(1−y). Taking the negative log and averaging over n examples gives the log-loss (binary cross-entropy):

L = −(1/n) Σ [ yᵢ ln(pᵢ) + (1−yᵢ) ln(1−pᵢ) ]

Minimizing log-loss is maximizing likelihood — the minus sign flips one into the other. Log-loss punishes confident wrong answers brutally: predict p = 0.01 when the truth is y = 1 and −ln(0.01) ≈ 4.6 of loss lands on you.

3.5 Gradient descent and the gradient

There is no closed-form solution (unlike linear regression), so we descend the loss surface. The gradient of log-loss with respect to the coefficients is remarkably clean — it is the same form as linear regression's gradient, just with p in place of the raw prediction:

∂L/∂βⱼ = (1/n) Σ (pᵢ − yᵢ) · xᵢⱼ         (with xᵢ₀ = 1 for the intercept)

Read it as "prediction minus truth, times the feature." Update each coefficient against the gradient with learning rate η:

βⱼ ← βⱼ − η · ∂L/∂βⱼ

Repeat until the loss stops falling. Because log-loss is convex in the coefficients, gradient descent finds the global minimum — no local-minima traps.

3.6 Interpreting coefficients — exp(βᵢ) is an odds ratio ← the bridge

Take the log-odds line and raise e to both sides:

odds = p/(1−p) = exp(β0 + β1x1 + …) = e^β0 · (e^β1)^x1 · (e^β2)^x2 · …

Now increase one feature xᵢ by a single unit, holding the rest fixed. The odds get multiplied by exp(βᵢ). That multiplier is exactly an odds ratio (OR):

This is the bridge. The odds ratio you compute from a 2×2 Tables & Effect Measures (OR, RR, RD, NNT, HR) as ad/bc is the same quantity a logistic regression produces as exp(β) for a single binary exposure. The OR is the natural output of logistic regression — which is why case-control studies and meta-analyses speak in odds ratios.

3.7 Decision boundary, threshold, and class imbalance

The model outputs a probability; a decision needs a cutoff. The default threshold is 0.5, which corresponds to z = 0 (the sigmoid's midpoint) and defines the decision boundary — the line β0 + β1x1 + … = 0 separating predicted-yes from predicted-no. But 0.5 is a choice, not a law. With class imbalance (say 1% positives) or asymmetric costs (missing a cancer is worse than a false alarm), you tune the threshold, reweight the classes, or resample. See Model Evaluation for the precision/recall and ROC tools that pick a threshold honestly.

3.8 Beyond two classes and regularization

• Multiclass — softmax. For K > 2 classes, generalize the sigmoid to the softmax, which outputs a probability vector that sums to 1. Fit it either as one-vs-rest (one binary model per class) or as a single multinomial (softmax) model.
• Regularization. Add a penalty on the coefficients to fight overfitting: L2 (ridge) shrinks all coefficients smoothly; L1 (lasso) drives some to exactly zero for feature selection. In scikit-learn the strength is C (smaller C = stronger penalty).

---

4. The math

The model, end to end:

z    = β0 + β1x1 + β2x2 + …            (linear score)
p    = σ(z) = 1/(1+e^−z)               (sigmoid → probability)
logit(p) = ln( p/(1−p) ) = z          (linear in the LOG-ODDS)
odds = p/(1−p) = exp(z)                (so exp(βi) is an ODDS RATIO)

loss L   = −(1/n) Σ [ y ln p + (1−y) ln(1−p) ]     (log-loss)
gradient ∂L/∂βⱼ = (1/n) Σ (p − y) xⱼ
update   βⱼ ← βⱼ − η · ∂L/∂βⱼ

Worked example — a tiny 1-feature model. Predict whether a student passes an exam (y=1) from hours studied x. Suppose fitting gives:

β0 = −4,   β1 = 1.5

Compute the probability for a student who studied x = 3 hours:

z    = β0 + β1·x = −4 + 1.5·3 = −4 + 4.5 = 0.5
p    = σ(0.5) = 1/(1+e^−0.5) = 1/(1+0.6065) = 1/1.6065 = 0.622

So the model predicts a 62.2% chance of passing after 3 hours. Check the log-odds identity:

odds = p/(1−p) = 0.622/0.378 = 1.646
ln(odds) = ln(1.646) = 0.50 = z   ✓   (matches the linear score exactly)

Now the odds ratio. Interpret β1 = 1.5:

exp(β1) = e^1.5 = 4.48

Each extra hour of study multiplies the odds of passing by 4.48. Concretely, go from 3 to 4 hours:

z(4) = −4 + 1.5·4 = 2.0      p(4) = σ(2.0) = 0.881
odds(3) = 1.646              odds(4) = 0.881/0.119 = 7.37
odds(4)/odds(3) = 7.37/1.646 = 4.48 = exp(β1)   ✓

The odds jumped by exactly exp(β1), no matter which one-hour step you take — that constant multiplicative effect on the odds is the whole meaning of a logistic coefficient. The decision boundary (threshold 0.5) sits where z = 0, i.e. x = −β0/β1 = 4/1.5 ≈ 2.67 hours — study more than that and the model predicts "pass."

---

5. Real code

Pure numpy, from scratch: gradient descent on log-loss for a tiny synthetic dataset. It prints the learned coefficients, shows the log-loss falling, samples the sigmoid curve, and reports exp(β) as the odds ratio.

"""From-scratch logistic regression: gradient descent on log-loss (pure numpy).
Fits a 1-feature model, prints learned coefficients, shows log-loss decreasing,
samples the sigmoid, and reports exp(beta) = the odds ratio."""
import numpy as np
rng = np.random.default_rng(0)

# ---- synthetic dataset generated from a KNOWN model, so we can check recovery ----
# truth: P(pass) = sigmoid(-4 + 1.5*hours). We fit and hope to get -4, 1.5 back.
TRUE_B0, TRUE_B1 = -4.0, 1.5
n = 200
hours = rng.uniform(0, 6, size=n)                   # hours studied
p_true = 1.0 / (1.0 + np.exp(-(TRUE_B0 + TRUE_B1 * hours)))
passed = (rng.random(n) < p_true).astype(int)       # noisy 0/1 labels
X = np.column_stack([np.ones(n), hours])            # add intercept column of 1s
y = passed.astype(float)

def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-z))

def log_loss(y, p, eps=1e-12):
    p = np.clip(p, eps, 1 - eps)                    # avoid log(0)
    return -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))

# ---- gradient descent ----
beta = np.zeros(X.shape[1])
eta, epochs = 0.3, 20000
history = []
for epoch in range(epochs):
    p = sigmoid(X @ beta)                           # predicted probabilities
    grad = X.T @ (p - y) / len(y)                   # ∂L/∂β = mean((p − y)·x)
    beta -= eta * grad                              # step downhill
    if epoch % 4000 == 0:
        history.append((epoch, log_loss(y, sigmoid(X @ beta))))

b0, b1 = beta
print(f"learned coefficients:  beta0 = {b0:.3f}   beta1 = {b1:.3f}")
print("\nlog-loss decreasing:")
for epoch, L in history:
    print(f"  epoch {epoch:6d}   log-loss = {L:.4f}")

print("\nsigmoid curve (hours -> P(pass)):")
for h in [1, 2, 3, 4, 5]:
    p = sigmoid(b0 + b1 * h)
    print(f"  {h} hours -> z = {b0 + b1*h:6.2f}   p = {p:.3f}")

# ---- THE BRIDGE: exp(beta1) is an odds ratio ----
OR = np.exp(b1)
print(f"\nexp(beta1) = {OR:.3f}  <-- ODDS RATIO")
print(f"each extra hour multiplies the odds of passing by {OR:.2f}x")

# sklearn one-liner (same idea, batteries included):
# from sklearn.linear_model import LogisticRegression
# clf = LogisticRegression(C=1e6).fit(hours.reshape(-1,1), passed)
# print(clf.intercept_, clf.coef_, np.exp(clf.coef_))

Representative output (the numbers are reproducible with the seed above):

learned coefficients:  beta0 = -4.867   beta1 = 1.623

log-loss decreasing:
  epoch      0   log-loss = 0.5844
  epoch   4000   log-loss = 0.3109
  epoch   8000   log-loss = 0.3109
  epoch  12000   log-loss = 0.3109
  epoch  16000   log-loss = 0.3109

sigmoid curve (hours -> P(pass)):
  1 hours -> z =  -3.24   p = 0.038
  2 hours -> z =  -1.62   p = 0.165
  3 hours -> z =   0.00   p = 0.500
  4 hours -> z =   1.62   p = 0.835
  5 hours -> z =   3.25   p = 0.963

exp(beta1) = 5.066  <-- ODDS RATIO
each extra hour multiplies the odds of passing by 5.07x

Two things to notice: the log-loss falls and settles (convex loss, so descent is safe), and the fit recovers the coefficients near the true (−4, 1.5) that generated the data — so exp(β1) ≈ 5 lands near the true odds ratio exp(1.5) = 4.48. The model recovered an odds ratio from data.

---

6. Real-world example

Credit-card fraud scoring — where all the pieces earn their keep.

• The outcome is binary. Each transaction is fraud (y=1) or legitimate (y=0). We want a probability, not a hard label, so the fraud team can rank the riskiest transactions.
• Why not a line? A raw linear score would output "1.8 fraud" for an extreme transaction. Meaningless. The sigmoid squashes every score into a clean P(fraud) ∈ (0,1).
• The coefficients tell a story. Fit on features like amount, is_foreign, hour_of_day. Suppose is_foreign gets β = 1.1. Then exp(1.1) ≈ 3.0: a foreign transaction has 3× the odds of being fraud, holding other features fixed. That single number is an odds ratio a risk analyst can act on and explain to a regulator — the same OR they'd get from a 2×2 table of foreign-vs-domestic fraud counts.
• The threshold is a business decision, not 0.5. Fraud is rare (heavy class imbalance) and a missed fraud costs far more than a false alarm's review. So the team drops the threshold to, say, 0.2 — flag anything above 20% predicted risk for manual review — trading more false positives for fewer missed frauds. They pick that number from the precision/recall curve, not by default.
• Regularization keeps it honest. With hundreds of engineered features, L1 zeroes out the useless ones (automatic feature selection) and L2 stops any single coefficient from exploding on a rare pattern.
• Payoff. The model is fast, monotone-interpretable (every coefficient is an odds ratio), and calibrated enough to rank transactions — which is exactly why logistic regression is still the first model most fraud, credit, and clinical-risk teams reach for.

---

7. Interview questions companies actually ask

Q [Stripe / risk modeling] "Why can't you just use linear regression for a yes/no label?"
  A Linear regression outputs any real number, so it predicts impossible 'probabilities'
    like 1.4 or −0.3, and its least-squares error model assumes constant normal noise that a
    0/1 target violates. Logistic regression wraps the linear score in a sigmoid, guaranteeing
    the output is a valid probability in (0,1), and fits by maximum likelihood (log-loss),
    which matches the Bernoulli nature of the data.

Q [Meta / ML] "What does it mean that logistic regression is 'linear in the log-odds'?"
  A Solving p = σ(z) for z gives ln(p/(1−p)) = z = β0 + β1x1 + … . The log-odds (logit) is a
    straight line in the features, even though the probability curves. The sigmoid is just the
    inverse of the logit — it maps the real-line score back into (0,1). This is why it's a
    *linear* model and why coefficients have a clean odds interpretation.

Q [a health-tech startup] "You fit a logistic model and a coefficient is 0.7. Interpret it."
  A exp(0.7) ≈ 2.0, so a one-unit increase in that feature multiplies the ODDS of the event by
    about 2 — an odds ratio of 2, holding other features fixed. Positive β ⇒ OR > 1 (raises the
    odds), negative β ⇒ OR < 1 (lowers them), β = 0 ⇒ OR = 1 (no effect). exp(β) is exactly the
    odds ratio you'd compute from a 2×2 table.

Q [Google / ML] "How is logistic regression fit? Write the loss and its gradient."
  A By maximum likelihood, equivalently by minimizing log-loss (binary cross-entropy):
    L = −(1/n) Σ [y ln p + (1−y) ln(1−p)], where p = σ(β·x). Its gradient is beautifully simple:
    ∂L/∂βⱼ = (1/n) Σ (p − y)xⱼ — 'prediction minus truth times the feature.' The loss is convex,
    so gradient descent finds the global minimum.

Q [Amazon / applied science] "Your positive class is 1%. What do you do about the threshold?"
  A Don't blindly use 0.5. With heavy imbalance, tune the decision threshold using the
    precision/recall curve and the business cost of false negatives vs false positives; also
    consider class weights or resampling. The model still outputs calibrated-ish probabilities;
    the threshold is a separate, cost-driven decision.

Q [a fintech] "Why is logistic regression the standard in credit and clinical risk?"
  A Its coefficients ARE odds ratios (exp(β)), so it's directly interpretable and defensible to
    regulators; it's fast, convex (a stable global fit), calibratable, and regularizable (L1 for
    selection, L2 for shrinkage). You trade some raw accuracy versus tree ensembles for
    transparency — often the right trade in regulated domains.

Q [a research lab] "How do you extend logistic regression to more than two classes?"
  A Use softmax (multinomial logistic regression), which outputs a probability vector summing to
    1 across K classes, or fit one-vs-rest: one binary logistic model per class, then normalize.
    Softmax is the direct generalization of the sigmoid to K classes.

---

8. When to use / tradeoffs

   REACH FOR LOGISTIC REGRESSION WHEN:
     ✓ the target is binary (or multiclass via softmax) and you want PROBABILITIES
     ✓ you need INTERPRETABLE coefficients — exp(β) is an odds ratio a human can explain
     ✓ you're in a regulated domain (credit, clinical, insurance) that demands transparency
     ✓ you want a fast, convex, stable baseline before trying anything fancier
     ✓ features relate to the log-odds roughly linearly (add interactions/splines if not)
   BE CAREFUL / LOOK ELSEWHERE WHEN:
     ✗ the true boundary is highly non-linear → trees, gradient boosting, or a neural net
     ✗ features are strongly collinear → coefficients get unstable (use L2 / drop features)
     ✗ classes are heavily imbalanced → tune the threshold, reweight, or resample
     ✗ you conflate the ODDS ratio with a RISK ratio on a common outcome → OR overstates RR
     ✓ ALWAYS evaluate with the right metric (log-loss, ROC-AUC, PR-AUC) — accuracy lies on
       imbalanced data
   THE ONE-LINE MENTAL MODEL:
     linear score → sigmoid → probability; the model is a line in the LOG-ODDS, so exp(β) = OR.

Logistic regression is the workhorse binary classifier: a linear model in disguise, honest about probabilities, and interpretable down to a single odds ratio per feature. Its limits are the limits of any linear model — a curved boundary or tangled features need something richer — but as a first, explainable model it is hard to beat, and it is the direct link between machine learning and the odds ratios of evidence synthesis.

---

9. Summary + related articles

• A straight line fails for yes/no outcomes because it predicts probabilities outside [0,1]; we need a squashing function.
• The sigmoid σ(z) = 1/(1+e^−z) maps any score z = β0 + β1x1 + … into a probability in (0,1).
• Logistic regression is linear in the log-odds: ln(p/(1−p)) = β0 + β1x1 + … — the single most important line in the topic.
• Fit by maximum likelihood = minimizing log-loss with gradient descent; the gradient is (1/n)Σ(p−y)x and the loss is convex.
• Turn probabilities into decisions with a threshold (default 0.5, but tune it for imbalance and asymmetric costs).
• exp(βᵢ) is an odds ratio — a one-unit rise in xᵢ multiplies the odds by exp(βᵢ). The OR is the natural output of logistic regression, tying it straight back to the 2×2 table.
• Extend with softmax for multiclass and L1/L2 regularization; always evaluate with a metric fit for the class balance.

Related: Logarithms, Exponents & the Log Scale · Probability & Statistics Foundations · 2×2 Tables & Effect Measures (OR, RR, RD, NNT, HR) · Model Evaluation

Resources

• Hastie, Tibshirani & Friedman, The Elements of Statistical Learning, Ch. 4 — logistic regression and the logit
• Andrew Ng, CS229 / Machine Learning lecture notes — logistic regression, the sigmoid, and log-loss derivation
• Hosmer, Lemeshow & Sturdivant, Applied Logistic Regression — coefficients as odds ratios, the clinical standard
• scikit-learn User Guide, "Logistic regression" — https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
• James, Witten, Hastie & Tibshirani, An Introduction to Statistical Learning, Ch. 4 — a gentler treatment with worked examples