Braking Ranks
Ron Surz PPCA, Inc.
949/488-8339 www.PPCA-Inc.com
Introduction
Because they put the degree of success or failure into
perspective, ranks are central to investment performance evaluation. Investment
return and rank go hand in glove: “Our performance for 2002 is 10%, which ranks
in the top decile”. But there are significant problems with investment
performance ranks. Peer groups, from which ranks are derived, suffer from
significant biases. Also, because it takes many weeks to construct peer groups,
ranks are generally not available until long after the performance results have
been calculated. In the following we describe a new ranking approach that puts
a stop to these problems. That is why we call this article “braking ranks.”
Those of you who are familiar with Portfolio Opportunity Distributions (PODs)
will recognize the following as an enhanced adaptation that we call “PIPODs”,
for Popular Index PODs.
Peer groups are used to compare a portfolio's performance to that of other managed portfolios with the same style. The idea is to give the manager a report card based on his ranking among competitors with the same style of management by assembling a universe of similar managers. But traditional peer groups can be skewed by a number of biases.
Critics of peer groups have documented various biases that undermine evaluations based on such universes. Three of these biases are classification, composition, and survivorship.
o
Classification bias results
from the practice of forcing every manager into some pre-specified pigeonhole,
such as growth or value. It is now
commonly understood that most managers employ a blend of styles, so that pigeonhole
classifications misrepresent the manager's actual style as well as that of
peers.
o
Composition biases result
both from concentrations of certain fund types in databases, such as bank
commingled funds, and from small sample size.
International managers and socially responsible managers cannot be
properly evaluated because there are no databases of adequate size.
o
Survivorship bias causes
performance results to be overstated because accounts that have been
terminated, which may have underperformed, are no longer in the database. This is the most documented and best
understood source of peer group bias.
For example, an unsuccessful management product that was terminated in
the past is excluded from current peer groups.
This screening out of losers results in an overstatement of past
performance. A good illustration of how
survivor bias can skew things is the “marathon analogy”, which asks: If only
100 runners out of a 1,000-contestant marathon actually finish, is the 100th
the last? Or in the top ten percent?
All these biases cause performance yardsticks based on traditional peer groups to be unpredictably too long or too short. A solution to the problems with traditional peer groups is benchmarking. Benchmarks solve the bias problems but introduce a whole new problem. It takes decades to develop confident inferences of success or failure with benchmarks. In most cases, management changes over the course of time invalidate these inferences. Among the unfortunate consequences of the deficiencies of peer groups and benchmarks is that managers are frequently fired or retained for the wrong reasons. This is where Portfolio Opportunity Distributions (PODs) come in. PODs combine the better properties of peer groups with those of benchmarks to create a superior performance ranking system.
Portfolio Opportunity Distributions (PODs)
Portfolio Opportunity Distributions (PODs) eliminate the biases of peer groups by harnessing today's computing power with classical statistics to generate all the possible portfolios a manager could conceivably hold in line with the manager's mandate, or benchmark. The answer to the question “What portfolios are in a POD universe?” is “All of them.” In this way, the manager’s success, rather than the peer group of choice, is properly evaluated. Traditional peer groups cannot be tailored to the manager’s benchmark, so success cannot be properly differentiated from failure.
The basis for PODs is that, in common practice, the statistician constantly compares his results with those expected purely by chance. By applying this concept to performance evaluation, POD generates thousands of cyber portfolios at random, drawn from the universe of stocks that comprise the manager’s benchmark. This assures that the resulting opportunity distribution fairly reflects the manager's opportunities to add value versus the benchmark. PODs are available for many standard benchmarks, as shown in the Appendix to this article. The idea is to find the benchmark that is most appropriate for the manager, and to then locate the manager’s rank within the opportunities available for that benchmark.
The resulting distribution provides a grading system that shows the full range of results (or opportunities) that could have been achieved by the manager while eliminating the biases inherent to traditional peer group universes. PODs also solve the waiting-time problem associated with benchmarks. These popular index PODs (PIPODs) have a standard benchmark as their median, with fractiles around the median representing degrees of success or failure. A ranking in the top decile of a POD universe gives the statistician 90% confidence that the return was not merely random, but a significant indication of success. Similarly, a ranking in the bottom decile is a significant indicator of failure. The statistician commonly defines significance as an event that can be interpreted with 90% or greater confidence. A ranking outside a POD universe is a good indicator that the selected benchmark is inappropriate, since the manager had to hold a portfolio that could not have been constructed from securities in the benchmark.
Evaluation against a POD universe tells the evaluator whether the observed performance result was good or bad relative to the unique opportunities available. As discussed earlier, no index, benchmark, or traditional peer group universe can provide this insight. Further investigations into the reasons for success or failure -- such as attribution analyses and manager interviews -- can reveal the manager's level of skill. In a poetic sense, the POD approach actually allows the manager to be hoisted with his or her own petard.
Portfolio Opportunity Distributions offer a new standard appropriate to the ongoing technological revolution that characterizes our entrance into the twenty-first century. This innovation is only recently possible due to the fact that the computing power necessary to run POD simply did not exist as little as 10 years ago. The introduction of POD coincides with growing recognition of the inadequacies of the old approaches we’ve used for the last three decades. Importantly, POD ranks are available within days after the end of a performance measurement period, versus weeks with traditional approaches.
Traditional peer groups will continue to be used for manager searches because the intent is to hire the best of the survivors. PODs will be used for performance evaluations, where the intent is to determine manager value-added above a benchmark.
Popular Index PODs for periods ending December
31, 2002
Quarter
5th Top Bottom 95th
Prcntl Qrtile
Median Qrtile Prcntl
---- ----
---- ---- ----
S&P 500 14.9 10.8
8.4 6.2 2.1
S&P 400 MidCap 15.0 9.3
5.8 2.7 -1.3
S&P 600 SmallCap 15.0 9.1
4.9 1.5 -3.3
Russell 3000 14.7 10.6
8.0 5.5 1.7
Russell 200 14.6 10.6
8.2 5.8 1.1
Russell 1000 14.9 10.7
8.2 5.6 1.5
Russell Mid Cap 16.3 10.8
7.9 5.1 1.5
Russell 2000 16.5 10.0
6.2 2.6 -2.8
Russell 2500 16.5 10.1
6.6 3.2 -1.8
S&P/Barra Value 15.1 11.8
9.9 7.6 3.2
Russell 3000 Value 13.7
10.7 8.9 6.8
2.6
Russell 200 Value 15.2
11.9 10.1 7.9
3.2
Russell 1000 Value 14.0
11.0 9.2 7.0
2.7
Russell Mid Value 12.8
9.3 7.1 5.2
2.1
Russell 2000 Value 12.6
8.1 4.9 2.5
-1.7
Russell 2500 Value 12.6
8.3 5.5 3.1
-.5
S&P/Barra Growth 15.0 9.8
7.1 4.8 .4
Russell 3000 Growth 16.5
10.4 7.2 3.9
-1.1
Russell 200 Growth 14.6
9.7 6.7 3.8
-1.2
Russell 1000 Growth 16.3
10.3 7.2 4.0
-1.2
Russell Mid Growth 22.1
13.0 9.2 4.8
-1.2
Russell 2000 Growth 21.3
12.2 7.5 2.6
-4.5
Russell 2500 Growth 21.9
12.7 8.3 3.5
-3.3
Year
5th Top Bottom 95th
Prcntl Qrtile
Median Qrtile Prcntl
---- ----
---- ---- ----
S&P 500 -10.0 -17.7
-22.1 -25.7 -31.5
S&P 400 MidCap -.7 -8.7
-14.5 -20.5 -26.9
S&P 600 SmallCap .6
-8.1 -14.6 -22.4
-30.0
Russell 3000 -11.2 -17.4
-21.5 -25.5 -30.5
Russell 200 -11.7 -18.8
-23.4 -26.8 -32.4
Russell 1000 -10.4 -17.2
-21.6 -25.5 -30.8
Russell Mid Cap -2.1
-10.6 -16.2 -21.0
-27.2
Russell 2000 .4 -13.2
-20.5 -26.9 -35.0
Russell 2500 1.1 -10.8
-17.8 -23.6 -31.2
S&P/Barra Value -8.9
-15.6 -20.9 -24.5
-30.0
Russell 3000 Value -2.1
-9.7 -15.2 -19.0
-25.0
Russell 200 Value -5.9
-12.9 -18.0 -21.6
-27.2
Russell 1000 Value -3.4
-10.2 -15.5 -19.2
-24.9
Russell Mid Value 6.4
-3.9 -9.6 -13.9
-21.5
Russell 2000 Value 8.9
-5.1 -11.4 -16.6
-24.9
Russell 2500 Value 7.6
-3.8 -9.9 -15.1
-23.1
S&P/Barra Growth -9.5
-19.2 -23.6 -27.3
-34.0
Russell 3000 Growth -14.8
-23.0 -28.0 -32.2
-38.1
Russell 200 Growth -16.0
-23.8 -28.0 -31.6
-37.4
Russell 1000 Growth -14.1
-22.9 -27.9 -31.8
-38.0
Russell Mid Growth -8.3
-19.4 -27.4 -32.9
-40.0
Russell 2000 Growth -7.3
-21.4 -30.3 -38.5
-47.0
Russell 2500 Growth -7.2
-20.4 -29.1 -36.4
-44.3
3 Years
5th Top Bottom 95th
Prcntl Qrtile
Median Qrtile Prcntl
---- ----
---- ---- ----
S&P 500 -5.7 -11.7
-14.6 -17.4 -22.0
S&P 400 MidCap 10.6 4.7
-.1 -4.1 -13.0
S&P 600 SmallCap 12.7 7.2
.6 -3.7 -15.0
Russell 3000 -4.8 -10.5
-13.7 -16.7 -22.6
Russell 200 -8.0 -14.0
-16.8 -19.7 -24.2
Russell 1000 -5.1 -11.1
-14.2 -17.2 -22.7
Russell Mid Cap 5.2 -1.2
-5.0 -9.1 -16.5
Russell 2000 5.6 -1.8
-7.5 -12.7 -22.7
Russell 2500 7.6 .4
-4.6 -9.6 -18.2
S&P/Barra Value -.8 -6.3
-9.5 -12.8 -18.2
Russell 3000 Value 4.8
-.9 -4.3 -7.7
-13.4
Russell 200 Value -.5
-5.8 -8.5 -11.4
-16.0
Russell 1000 Value 3.6
-1.9 -5.1 -8.4
-13.8
Russell Mid Value 14.4
7.3 3.3 -.8
-7.0
Russell 2000 Value 20.1
12.8 7.4 2.8
-6.7
Russell 2500 Value 18.0
11.2 6.1 1.8
-6.2
S&P/Barra Growth -10.1
-17.0 -19.6 -22.2
-26.1
Russell 3000 Growth -12.8
-20.0 -23.4 -27.2
-33.2
Russell 200 Growth -14.1
-21.3 -24.4 -27.5
-32.2
Russell 1000 Growth -12.9
-20.4 -23.6 -27.2
-32.5
Russell Mid Growth -7.8
-15.7 -20.0 -25.9
-34.1
Russell 2000 Growth -6.5
-14.4 -21.1 -27.1
-38.5
Russell 2500 Growth -5.2
-13.2 -19.0 -25.0
-35.3
5 Years
5th Top Bottom 95th
Prcntl Qrtile
Median Qrtile Prcntl
---- ----
---- ---- ----
S&P 500 6.6 2.4
-.6 -3.0 -7.8
S&P 400 MidCap 15.4 10.2
6.4 2.9 -4.8
S&P 600 SmallCap 12.8 6.8
2.4 -1.7 -9.7
Russell 3000 7.3 2.6
-.7 -3.5 -8.9
Russell 200 6.3 1.8
-1.3 -3.8 -8.2
Russell 1000 7.3 2.6
-.6 -3.3 -8.6
Russell Mid Cap 10.1 5.6
2.2 -.9 -8.4
Russell 2000 9.3 3.0
-1.4 -5.6 -12.9
Russell 2500 10.9 5.5
1.6 -2.2 -10.0
S&P/Barra Value 5.4 1.9
-.8 -3.6 -8.2
Russell 3000 Value 7.1
3.9 1.2 -1.6
-6.3
Russell 200 Value 6.5
3.1 .6 -2.1
-5.7
Russell 1000 Value 7.1
3.8 1.2 -1.6
-6.1
Russell Mid Value 8.9
5.8 3.0 .3
-6.8
Russell 2000 Value 12.3
6.8 2.7 -1.0
-7.3
Russell 2500 Value 11.1
6.9 3.5 .4
-6.8
S&P/Barra Growth 6.8 2.1
-1.1 -3.3 -8.2
Russell 3000 Growth 5.9
-.2 -4.1 -7.2
-13.9
Russell 200 Growth 5.8
-.3 -4.1 -6.8
-12.2
Russell 1000 Growth 6.1
.0 -3.8 -6.7
-13.1
Russell Mid Growth 8.7
2.4 -1.8 -5.8
-15.4
Russell 2000 Growth 5.8
-1.6 -6.6 -11.7
-20.7
Russell 2500 Growth 8.6
1.5 -3.2 -7.9
-17.2