Skip to content

QuantStats metrics show inconsistent results between benchmark-included and return-only scenarios #480

@none2003

Description

@none2003

Hi,

I recently noticed a potential issue with qs.reports.metrics/html when a value is provided for the benchmark parameter. Specifically, some metrics in the results seem to show inconsistencies.

To better understand this, I conducted a comparison by passing both benchmark and return together, and then using only return to call qs.reports.metrics. The "Same?" column in my analysis highlights the comparison between the "Return with Benchmark" and "Only Return" outputs.

It is worth noting that metrics such as "R^2" and "Information Ratio" are missing when using only return. Since these metrics rely on both return and benchmark return for their calculations, I would like to suggest displaying "-" for benchmark, similar to the method used for Beta and Alpha.

Additionally, metrics like "Kelly Criterion," "Expected Shortfall (cVaR)," "Payoff Ratio," "Outlier Win Ratio," "Outlier Loss Ratio," "Serenity Index," "Avg. Up Month," and "Avg. Down Month" yield differing results when a benchmark is included compared to when they are calculated individually. This discrepancy could potentially relate to the issue I previously mentioned in "The calculation result of CVAR in the HTML report appears to be questionable #467." It seems there may be variations in the logic applied during calculations, depending on whether a DataFrame or Series is used.

  Benchmark Return with Benchmark Only Return Same?
Start Period 2020/2/27 2020/2/27 2020/2/27 Y
End Period 2025/11/7 2025/11/7 2025/11/7 Y
Risk-Free Rate 1.81 1.81 1.81 Y
Time in Market 99 99 99 Y
Cumulative Return 30.57 210.24 210.24 Y
CAGR﹪ 4.98 22.91 22.91 Y
Sharpe 0.26 1.89 1.89 Y
Prob. Sharpe Ratio 56.36 100 100 Y
Smart Sharpe 0.24 1.75 1.75 Y
Sortino 0.38 2.9 2.9 Y
Smart Sortino 0.35 2.69 2.69 Y
Sortino/√2 0.27 2.05 2.05 Y
Smart Sortino/√2 0.25 1.9 1.9 Y
Omega 1.4 1.4 1.4 Y
Max Drawdown -45.6 -6.49 -6.49 Y
Max DD Date 2024/9/13 2022/7/15 2022/7/15 Y
Max DD Period Start 2021/2/18 2022/6/13 2022/6/13 Y
Max DD Period End 2025/11/7 2022/9/9 2022/9/9 Y
Longest DD Days 1724 168 168 Y
Volatility (ann.) 18.38 10.26 10.26 Y
R^2 0.23 0.23   N
Information Ratio 0.06 0.06   N
Calmar 0.11 3.53 3.53 Y
Skew 0.12 -0.05 -0.05 Y
Kurtosis 5.8 4.82 4.82 Y
Expected Daily 0.02 0.08 0.08 Y
Expected Monthly 0.38 1.63 1.63 Y
Expected Yearly 4.55 20.77 20.77 Y
Kelly Criterion -1.83 17.09 17.5 N
Risk of Ruin 0 0 0 Y
Daily Value-at-Risk -1.88 -0.98 -0.98 Y
Expected Shortfall (cVaR) -2.8 -2.8 -1.54 N
Max Consecutive Wins 10 9 9 Y
Max Consecutive Losses 7 9 9 Y
Gain/Pain Ratio 0.07 0.44 0.44 Y
Gain/Pain (1M) 0.32 6.78 6.78 Y
Payoff Ratio 0.94 1.06 1.07 N
Profit Factor 1.07 1.44 1.44 Y
Common Sense Ratio 1.22 1.73 1.73 Y
CPC Index 0.51 0.88 0.88 Y
Tail Ratio 1.14 1.2 1.2 Y
Outlier Win Ratio 2.96 5.1 3.86 N
Outlier Loss Ratio 2.92 5.27 3.71 N
MTD 0 0.35 0.35 Y
3M 15.42 7.53 7.53 Y
6M 24.66 12.18 12.18 Y
YTD 20.66 16.42 16.42 Y
1Y 17.98 17.86 17.86 Y
3Y (ann.) 6.6 19.36 19.36 Y
5Y (ann.) -1.36 18.05 18.05 Y
10Y (ann.) 4.98 22.91 22.91 Y
All-time (ann.) 4.98 22.91 22.91 Y
Best Day 8.48 3.49 3.49 Y
Worst Day -7.05 -4.05 -4.05 Y
Best Month 20.97 14.69 14.69 Y
Worst Month -7.9 -4.71 -4.71 Y
Best Year 43.31 44.05 44.05 Y
Worst Year -21.63 13.45 13.45 Y
Avg. Drawdown -4.02 -1.24 -1.24 Y
Avg. Drawdown Days 90 13 13 Y
Recovery Factor 0.79 17.89 17.89 Y
Ulcer Index 0.28 0.02 0.02 Y
Serenity Index 0.03 12.94 9.9 N
Avg. Up Month 4.53 3.56 2.49 N
Avg. Down Month -3.32 -1.38 -1.16 N
Win Days 50.7 57.31 57.31 Y
Win Month 51.47 78.26 78.26 Y
Win Quarter 58.33 91.67 91.67 Y
Win Year 50 100 100 Y
Beta - 0.27  
Alpha - 0.19  
Correlation - 47.82%  
Treynor Ratio - 780.77%  

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions