Statistical Arbitrage with Machine Learning: A Practical Guide

Statistical arbitrage (StatArb) is a class of short-term financial trading strategies that employ mean reversion models involving broadly diversified portfolios of securities.

While traditional StatArb relies on cointegration tests and linear models, modern quantitative funds are applying machine learning to uncover complex, non-linear relationships.

The Foundation: Pairs Trading

The simplest form of StatArb is pairs trading. You find two highly correlated assets (e.g., Coca-Cola and Pepsi). When their price ratio diverges significantly from its historical mean, you short the outperforming asset and long the underperforming one, betting that the spread will revert to the mean.

Detecting Cointegration

Before using ML, it's crucial to understand cointegration. Two price series might be non-stationary (random walks), but a linear combination of them could be stationary.

python
import numpy as np
import pandas as pd
import statsmodels.tsa.stattools as ts

# Assuming df contains price series for two assets: 'Asset_A' and 'Asset_B'
def check_cointegration(df, asset_a, asset_b):
    score, p_value, _ = ts.coint(df[asset_a], df[asset_b])
    print(f"Cointegration test p-value: {p_value}")
    return p_value < 0.05

Enter Machine Learning: Non-Linear Spreads

The problem with linear cointegration is that financial markets are inherently non-linear and non-stationary. A relationship that existed in 2020 might break in 2024.

We can use ML algorithms (like Random Forests or Gradient Boosting) to model the spread dynamically based on a wide array of features.

Feature Engineering for StatArb

The success of your ML model depends almost entirely on your features. Common features include:

Feature Category	Examples
Price Action	Moving averages (SMA, EMA), Bollinger Bands, RSI, MACD
Microstructure	Bid-ask spread, order book imbalance, tick volume
Macro/Alternative	Interest rate spreads, sector ETF flows, sentiment scores

Modeling the Spread

Instead of a simple ratio, we train a model to predict the fair value of Asset A given Asset B and our feature set.

python
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.model_selection import TimeSeriesSplit

# Features (X) and Target (y - e.g., next period return of Asset A)
X = df[['Asset_B_Return', 'RSI_A', 'Volume_Imbalance_A', 'Sector_Momentum']]
y = df['Asset_A_Next_Return']

# Use TimeSeriesSplit to avoid look-ahead bias!
tscv = TimeSeriesSplit(n_splits=5)
model = HistGradientBoostingRegressor(max_iter=100)

for train_index, test_index in tscv.split(X):
    X_train, X_test = X.iloc[train_index], X.iloc[test_index]
    y_train, y_test = y.iloc[train_index], y.iloc[test_index]
    
    model.fit(X_train, y_train)
    predictions = model.predict(X_test)
    # Evaluate Strategy Profitability...

The Golden Rule: Beware of Overfitting

Machine learning models are incredibly good at finding patterns in noise. In finance, the signal-to-noise ratio is notoriously low.

1. Always use TimeSeriesSplit: Never use standard cross-validation. You cannot use data from 2025 to predict 2024.
2. Transaction Costs: A model might look profitable on paper but lose money after accounting for bid-ask spreads, slippage, and exchange fees. Always include conservative cost estimates in your backtest.
3. Regime Shifts: Train models to recognize market regimes (high volatility vs. low volatility) and adjust their aggressiveness accordingly.