Add design spec: beta computed fallback + short interest FMP fallback

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

1 files changed, 105 insertions, 0 deletions

diff --git a/docs/superpowers/specs/2026-05-17-beta-short-interest-fallbacks-design.md b/docs/superpowers/specs/2026-05-17-beta-short-interest-fallbacks-design.md
new file mode 100644
index 0000000..5366032
--- /dev/null
+++ b/docs/superpowers/specs/2026-05-17-beta-short-interest-fallbacks-design.md
@@ -0,0 +1,105 @@
+# Design: Beta Computed Fallback + Short Interest FMP Fallback
+
+**Date:** 2026-05-17  
+**Scope:** `backend/app/services/data_service.py` only — no schema changes, no frontend changes.
+
+---
+
+## Problem
+
+Two fields in the Overview payload silently return `null` for certain tickers (e.g. NFLX):
+
+- `stats.beta` — read exclusively from `yfinance.Ticker.info["beta"]`, which is sparse.
+- `short_interest.*` — all four fields read exclusively from `info`, which returns nothing for some tickers.
+
+The frontend already handles null gracefully (hides rows / hides card), but the data should be present whenever it can be computed or fetched.
+
+---
+
+## Beta: Computed Fallback from Price History
+
+### Approach
+
+When `info["beta"]` is `None`, compute beta from 2 years of weekly price returns against SPY.
+
+**Formula:** `β = cov(r_ticker, r_spy) / var(r_spy)`
+
+### Implementation
+
+1. Add `"2y"` to `PERIODS` and `YF_PERIOD_MAP` (`"2y": "2y"`).
+2. Add `compute_beta(symbol: str) -> float | None` cached in `RATIO_CACHE` (TTL=3600):
+   - Calls `get_price_history(symbol, "2y")` and `get_price_history("SPY", "2y")` — reuses existing cache.
+   - Converts close prices to weekly returns (resample to week-end, pct_change).
+   - Aligns on date index, drops NaN rows. Requires ≥ 52 observations; returns `None` if sparse.
+   - Caps result to `[-3.0, 3.0]` — extreme values are data artifacts.
+3. In `_build_quote_and_stats`, change line 714:
+   ```python
+   # before
+   beta = _safe_float(_field(source_map, field_sources, "stats.beta", ("info", "beta")))
+   # after
+   beta = _safe_float(_field(source_map, field_sources, "stats.beta", ("info", "beta")))
+   if beta is None:
+       beta = compute_beta(sym)
+       if beta is not None:
+           field_sources["stats.beta"] = "computed"
+   ```
+   `sym` is already threaded through `get_ticker_overview` → `_build_quote_and_stats`.
+
+### Trade-offs
+
+- No new API keys required.
+- SPY history fetch is cached; cold path adds one extra `HISTORY_CACHE` lookup.
+- Computed beta is a trailing market beta, not an adjusted/predicted beta. Acceptable for an overview display.
+
+---
+
+## Short Interest: FMP Fallback
+
+### Approach
+
+When all four `info`-sourced short interest fields are `None`, call FMP `/v4/short-of-float-symbol`.
+
+### Implementation
+
+1. Add `SHORT_CACHE = TTLCache(maxsize=256, ttl=3600)` alongside the other caches at the top of the file.
+2. Add `get_fmp_short_interest(symbol: str) -> dict[str, Any]` cached in `SHORT_CACHE`:
+   - Reads `FMP_API_KEY` from env; returns `{}` immediately if absent.
+   - `GET https://financialmodelingprep.com/api/v4/short-of-float-symbol?symbol=SYMBOL&apikey=KEY`
+   - Extracts from first row of response array: `shortPercent` → `short_percent_of_float`, `shortRatio` → `short_ratio`, `shortsVolume` → `shares_short`.
+   - Returns `{}` on any exception or unexpected response shape.
+3. In `get_ticker_overview`, after building `short_interest` (lines 812–823):
+   ```python
+   if all(v is None for v in short_interest.values()):
+       fmp_short = get_fmp_short_interest(sym)
+       if fmp_short:
+           short_interest.update(fmp_short)
+   ```
+4. `shares_short_prior_month` and `shares_short_delta_pct` are not available from this FMP endpoint and stay `None`.
+
+### Trade-offs
+
+- Costs FMP quota (250 req/day free tier) only when yfinance returns nothing — not on every request.
+- `shares_short_delta_pct` unavailable from FMP; partial data is acceptable (frontend handles it).
+- Guard on "all fields None" avoids unnecessary FMP calls when yfinance returns partial data.
+
+---
+
+## What Does Not Change
+
+- Response schema (`schemas.py`) — `ShortInterest` and `TickerStats` shapes unchanged.
+- Frontend — no changes required.
+- All existing tests — monkeypatching `data_service` functions continues to work.
+- No new backend routes.
+
+---
+
+## Testing
+
+Backend tests should:
+- Verify `compute_beta` returns a float when given valid history for both ticker and SPY.
+- Verify `compute_beta` returns `None` when history is too sparse (< 52 observations).
+- Verify `compute_beta` caps extreme values at ±3.
+- Verify `get_fmp_short_interest` returns mapped fields when FMP responds correctly.
+- Verify `get_fmp_short_interest` returns `{}` when FMP key is absent or request fails.
+- Verify `get_ticker_overview` uses computed beta when `info["beta"]` is None (monkeypatch).
+- Verify `get_ticker_overview` populates short interest from FMP when all info fields are None (monkeypatch).