Chapter 12
Crypto Perps — Exploratory Data Analysis
NotebookPython 3 (ipykernel)25 cells
Crypto Perps — Exploratory Data Analysis
Docker image: ml4t
Purpose
Profile the Binance Futures hourly OHLCV dataset for 19 perpetual contracts alongside the 8-hourly Premium Index that captures the perpetual–spot basis. The notebook anchors data shape, units, coverage, and OHLC integrity before the strategy work begins in Chapter 6.
Learning Objectives
- Load hourly OHLCV and 8-hourly premium index data via the canonical loaders.
- Document premium-index units (decimal, multiply by 100 for percent).
- Quantify cross-frequency join coverage between OHLCV and premium data.
- Run OHLC invariant checks and inspect for time-stamp gaps.
Book Reference
Chapter 2 §2.2 (asset-class market data landscape — digital assets).
Prerequisites
- Familiarity with daily OHLCV equity data (
01_us_equities_eda). - The Binance Futures parquet at
$ML4T_DATA_PATH/crypto/(OHLCV + premium). - Methodology continues in
11_crypto_premium_analysis; the case-study pipeline lives undercase_studies/crypto_perps_funding/.
In [1]python · cell 2
python
"""Crypto Perps EDA — hourly OHLCV and premium index exploration."""
import plotly.graph_objects as go
import polars as pl
from data import load_crypto_perps, load_crypto_premium
from utils.data_quality import check_ohlc_invariants, per_asset_stats
from utils.style import COLORSIn [2]python · cell 3
python
MAX_SYMBOLS = 0 # 0 = all symbols1. Load and Inspect OHLCV
Hourly OHLCV data from Binance Futures for 19 cryptocurrencies. Trading is 24/7 (8,760 hours/year vs 252 days for equities).
In [3]python · cell 5
python
ohlcv = load_crypto_perps(frequency="1h")
print("=== OHLCV Dataset ===")
print(f"Shape: {ohlcv.shape}")
print(f"Columns: {ohlcv.columns}")Output
=== OHLCV Dataset === Shape: (866484, 7) Columns: ['timestamp', 'open', 'high', 'low', 'close', 'volume', 'symbol']
In [4]python · cell 6
python
# Schema overview
print("\nSchema:")
for col, dtype in ohlcv.schema.items():
print(f" {col}: {dtype}")Output
Schema: timestamp: Datetime(time_unit='ms', time_zone='UTC') open: Float64 high: Float64 low: Float64 close: Float64 volume: Float64 symbol: String
2. Coverage Summary
In [5]python · cell 8
python
# Symbols and date range
symbols = ohlcv["symbol"].unique().sort().to_list()
print("=== Coverage ===")
print(f"Number of symbols: {len(symbols)}")
print(f"\nSymbols: {', '.join(symbols)}")Output
=== Coverage === Number of symbols: 19 Symbols: AAVEUSDT, ADAUSDT, APTUSDT, ATOMUSDT, AVAXUSDT, BNBUSDT, BTCUSDT, COMPUSDT, DOGEUSDT, DOTUSDT, ETHUSDT, INJUSDT, LINKUSDT, MKRUSDT, NEARUSDT, SOLUSDT, SUIUSDT, UNIUSDT, XRPUSDT
In [6]python · cell 9
python
# Overall date range
date_range = ohlcv.select(
[
pl.col("timestamp").min().alias("start"),
pl.col("timestamp").max().alias("end"),
pl.col("timestamp").n_unique().alias("unique_hours"),
]
)
print(f"\nDate range: {date_range['start'][0]} to {date_range['end'][0]}")
print(f"Unique hours: {date_range['unique_hours'][0]:,}")Output
Date range: 2020-01-01 00:00:00+00:00 to 2025-12-31 23:00:00+00:00 Unique hours: 52,608
In [7]python · cell 10
python
# Per-symbol statistics
symbol_stats = per_asset_stats(
ohlcv,
time_col="timestamp",
asset_col="symbol",
price_col="close",
volume_col="volume",
)
print("\nSymbol Statistics (top 5 by volume):")
symbol_stats.sort("avg_volume", descending=True).head(5)Output
Symbol Statistics (top 5 by volume):
shape: (5, 6) ┌──────────┬───────┬─────────────────────────┬─────────────────────────┬───────────┬────────────┐ │ symbol ┆ rows ┆ start ┆ end ┆ avg_price ┆ avg_volume │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ u32 ┆ datetime[ms, UTC] ┆ datetime[ms, UTC] ┆ f64 ┆ f64 │ ╞══════════╪═══════╪═════════════════════════╪═════════════════════════╪═══════════╪════════════╡ │ DOGEUSDT ┆ 48015 ┆ 2020-07-10 09:00:00 UTC ┆ 2025-12-31 23:00:00 UTC ┆ 0.137195 ┆ 2.7592e8 │ │ XRPUSDT ┆ 52360 ┆ 2020-01-06 08:00:00 UTC ┆ 2025-12-31 23:00:00 UTC ┆ 0.89747 ┆ 5.2917e7 │ │ ADAUSDT ┆ 51880 ┆ 2020-01-31 08:00:00 UTC ┆ 2025-12-31 23:00:00 UTC ┆ 0.644055 ┆ 3.0205e7 │ │ SUIUSDT ┆ 23360 ┆ 2023-05-03 16:00:00 UTC ┆ 2025-12-31 23:00:00 UTC ┆ 1.917873 ┆ 1.2587e7 │ │ NEARUSDT ┆ 45568 ┆ 2020-10-15 08:00:00 UTC ┆ 2025-12-31 23:00:00 UTC ┆ 4.297627 ┆ 2.1940e6 │ └──────────┴───────┴─────────────────────────┴─────────────────────────┴───────────┴────────────┘
.dataframe > thead > tr,
.dataframe > tbody > tr {
text-align: right;
white-space: pre-wrap;
}
shape: (5, 6)
| symbol | rows | start | end | avg_price | avg_volume |
|---|---|---|---|---|---|
| str | u32 | datetime[ms, UTC] | datetime[ms, UTC] | f64 | f64 |
| "DOGEUSDT" | 48015 | 2020-07-10 09:00:00 UTC | 2025-12-31 23:00:00 UTC | 0.137195 | 2.7592e8 |
| "XRPUSDT" | 52360 | 2020-01-06 08:00:00 UTC | 2025-12-31 23:00:00 UTC | 0.89747 | 5.2917e7 |
| "ADAUSDT" | 51880 | 2020-01-31 08:00:00 UTC | 2025-12-31 23:00:00 UTC | 0.644055 | 3.0205e7 |
| "SUIUSDT" | 23360 | 2023-05-03 16:00:00 UTC | 2025-12-31 23:00:00 UTC | 1.917873 | 1.2587e7 |
| "NEARUSDT" | 45568 | 2020-10-15 08:00:00 UTC | 2025-12-31 23:00:00 UTC | 4.297627 | 2.1940e6 |
Contracts list at different dates
The universe is not fixed: BTC and a handful of majors are present from 2020, and newer contracts (SUI lists in 2023) switch on later. Counting distinct symbols per month makes the staggered onboarding explicit — coverage that any cross-sectional signal has to account for.
In [8]python · cell 12
python
active_by_month = (
ohlcv.with_columns(pl.col("timestamp").dt.truncate("1mo").alias("month"))
.group_by("month")
.agg(pl.col("symbol").n_unique().alias("contracts"))
.sort("month")
)
fig = go.Figure()
fig.add_trace(
go.Scatter(
x=active_by_month["month"].to_list(),
y=active_by_month["contracts"].to_list(),
mode="lines",
line=dict(color=COLORS["blue"], width=2, shape="hv"),
name="Contracts with data",
)
)
fig.update_layout(
title="Perpetual contracts with data, by month (staggered listings)",
xaxis_title="Month",
yaxis_title="Contracts",
yaxis_range=[0, 20],
height=420,
)
fig.show()Output
3. Premium Index Data
The premium index measures the spread between perpetual futures and spot prices:
Premium = (Perpetual Price - Spot Price) / Spot Price
- Positive premium: Futures above spot (bullish sentiment)
- Negative premium: Futures below spot (bearish sentiment)
Units
Premium values are stored as decimals (0.001 = 0.1%). When displaying, multiply by 100 for percentage representation.
In [9]python · cell 14
python
premium = load_crypto_premium(frequency="8h")
print("=== Premium Dataset ===")
print(f"Shape: {premium.shape}")
print(f"Columns: {premium.columns}")Output
=== Premium Dataset === Shape: (107839, 6) Columns: ['timestamp', 'symbol', 'premium_index_open', 'premium_index_high', 'premium_index_low', 'premium_index_close']
In [10]python · cell 15
python
# Premium range — values are decimals, not percentages
premium_range = premium.select(
[
pl.col("premium_index_close").min().alias("min"),
pl.col("premium_index_close").max().alias("max"),
pl.col("premium_index_close").mean().alias("mean"),
pl.col("premium_index_close").std().alias("std"),
]
)
print("\nPremium Range (decimals):")
print(f" Mean: {premium_range['mean'][0]:.6f} ({premium_range['mean'][0] * 100:.4f}%)")
print(f" Std: {premium_range['std'][0]:.6f} ({premium_range['std'][0] * 100:.4f}%)")
print(f" Min: {premium_range['min'][0]:.6f} ({premium_range['min'][0] * 100:.4f}%)")
print(f" Max: {premium_range['max'][0]:.6f} ({premium_range['max'][0] * 100:.4f}%)")Output
Premium Range (decimals): Mean: -0.000111 (-0.0111%) Std: 0.001137 (0.1137%) Min: -0.191547 (-19.1547%) Max: 0.012701 (1.2701%)
The basis is small — until it isn't
The premium sits within a fraction of a percent almost all the time, but the distribution has a fat negative tail: the minimum reaches roughly −19% during a dislocation. That asymmetry is the whole reason the funding strategy exists, so it is worth seeing, not just tabulating. (Axis clipped to ±2% so the central mass is legible; the tail runs far past the left edge.)
In [11]python · cell 17
python
premium_pct = (premium["premium_index_close"] * 100).to_list()
fig = go.Figure()
fig.add_trace(
go.Histogram(
x=premium_pct,
xbins=dict(start=-2, end=2, size=0.05),
marker_color=COLORS["slate"],
name="Premium (%)",
)
)
fig.add_vline(x=0, line_color=COLORS["amber"], line_width=1)
fig.add_annotation(
x=-2,
y=1,
xref="x",
yref="paper",
text=f"tail reaches {min(premium_pct):.1f}%",
showarrow=False,
xanchor="left",
yanchor="top",
font=dict(color=COLORS["copper"]),
)
fig.update_layout(
title="Premium-index distribution (8-hourly, % — axis clipped to ±2%)",
xaxis_title="Premium (%)",
yaxis_title="8-hour observations",
height=420,
)
fig.show()Output
4. Data Quality
In [12]python · cell 19
python
# OHLC invariants
invariants = check_ohlc_invariants(ohlcv)
print("=== OHLC Invariants ===")
for row in invariants.iter_rows(named=True):
status = "[OK]" if row["valid_pct"] >= 99.99 else "[WARN]"
print(f" {status} {row['check']}: {row['valid_pct']:.2f}%")Output
=== OHLC Invariants === [OK] high_gte_low: 100.00% [OK] high_gte_open: 100.00% [OK] high_gte_close: 100.00% [OK] low_lte_open: 100.00% [OK] low_lte_close: 100.00% [OK] volume_non_negative: 100.00%
In [13]python · cell 20
python
# Check for nulls
ohlcv_nulls = ohlcv.null_count().sum_horizontal()[0]
premium_nulls = premium.null_count().sum_horizontal()[0]
print(f"\nNull values: OHLCV={ohlcv_nulls}, Premium={premium_nulls}")Output
Null values: OHLCV=0, Premium=0
In [14]python · cell 21
python
# Check for gaps > 1 hour (use BTC as reference)
btc = ohlcv.filter(pl.col("symbol") == "BTCUSDT").sort("timestamp")
btc_gaps = btc.with_columns(pl.col("timestamp").diff().dt.total_hours().alias("hours_diff")).filter(
pl.col("hours_diff") > 1
)
print(f"\nGaps > 1 hour in BTC: {len(btc_gaps)}")
if len(btc_gaps) > 0:
print("(Small gaps expected during exchange maintenance)")Output
Gaps > 1 hour in BTC: 0
5. Joining OHLCV and Premium
Use left join to preserve all OHLCV rows and identify missing premium coverage.
In [15]python · cell 23
python
# Left join to identify missing premium data
combined = ohlcv.join(premium, on=["timestamp", "symbol"], how="left")
# Coverage analysis
total_rows = len(combined)
missing_premium = combined.filter(pl.col("premium_index_close").is_null()).height
coverage_pct = (total_rows - missing_premium) / total_rows * 100
print("=== Join Coverage ===")
print(f"OHLCV rows: {len(ohlcv):,}")
print(f"Premium rows: {len(premium):,}")
print(f"Combined rows: {total_rows:,}")
print(f"Missing premium: {missing_premium:,} ({100 - coverage_pct:.2f}%)")
print(f"Coverage: {coverage_pct:.2f}%")Output
=== Join Coverage === OHLCV rows: 866,484 Premium rows: 107,839 Combined rows: 866,484 Missing premium: 758,716 (87.56%) Coverage: 12.44%
In [16]python · cell 24
python
# Where does the missing premium concentrate?
missing_by_symbol = (
combined.filter(pl.col("premium_index_close").is_null())
.group_by("symbol")
.len()
.sort("len", descending=True)
)
print("Missing premium by symbol (top 5):")
missing_by_symbol.head(5)Output
Missing premium by symbol (top 5):
shape: (5, 2) ┌──────────┬───────┐ │ symbol ┆ len │ │ --- ┆ --- │ │ str ┆ u32 │ ╞══════════╪═══════╡ │ ETHUSDT ┆ 46053 │ │ BTCUSDT ┆ 46053 │ │ XRPUSDT ┆ 45833 │ │ LINKUSDT ┆ 45710 │ │ ADAUSDT ┆ 45416 │ └──────────┴───────┘
.dataframe > thead > tr,
.dataframe > tbody > tr {
text-align: right;
white-space: pre-wrap;
}
shape: (5, 2)
| symbol | len |
|---|---|
| str | u32 |
| "ETHUSDT" | 46053 |
| "BTCUSDT" | 46053 |
| "XRPUSDT" | 45833 |
| "LINKUSDT" | 45710 |
| "ADAUSDT" | 45416 |
Key Takeaways
- 24/7 trading: Crypto runs continuously — 52,608 unique hours across six full years (2020-01-01 to 2025-12-31), ~8,760 hours/year for the longest-history symbols.
- Universe: 19 perpetual contracts span 866,484 OHLCV bars; symbol coverage is non-uniform because contracts list at different dates.
- Premium units: Stored as decimals (0.001 = 0.1%); always multiply by 100 for percentage display in figures or text.
- Frequency mismatch: OHLCV is hourly, premium is 8-hourly, so a left join lands ~12.4% premium coverage on the hourly grid by design.
- Clean data: OHLC invariants hold for 100% of records on this snapshot; BTC has zero gaps > 1 hour.
Next Steps
11_crypto_premium_analysis: Premium dynamics, basis seasonality, and alignment to the 8-hourly funding cadence.- Chapter 8: Feature engineering for premium signals
(
case_studies/crypto_perps_funding/03_financial_features.py). - Chapter 16: Backtests for the funding-arbitrage case study.