More on Miller & Sanjurjo

Edit: Jonathan Miller was nice enough to explain my error in interpreting their claim. See below. 

I wrote last week about Miller & Sanjurjo (2015), a working paper which shows how taking unweighted averages of ratios of conditional proportions of success (conditional on previous success) can lead to a biased estimate of the true conditional probability. I then claimed that this result does not extend meaningfuly to the context that they're trying to extend it to: the "hot hand" in basketball, particularly Gilovich, et al. (1985).

Various people smarter than me, notably Andrew Gelman, disagree. They think that the Sanjurjo & Miller critique matters even for the sample sizes considered by Gilovich et al.





This question is easily answerable with some Monte Carlo simulations. In particular, I'll set my sample size to 248, which is the minimum number of shots recorded by any player in the Gilovich study (see their Table 1). For the sake of brevity, I set the true probability equal to 1/2. Basically, I repeat the 248-shot trial a large number (10000) times and take the unweighted mean of the empirical conditional successes across trials.

The Stata code is pretty simple:

clear
set more off
set seed 12304
global obsmax = 248

tempfile tofill

forval j = 1/10000 {
    clear
    qui set obs $obsmax
    gen heads = runiform()>0.5
    gen success = heads==1 & heads[_n-1]==1 // the numerator
    gen elig = heads==1 & _n!=$obsmax // the denominator
   
    collapse (sum) success elig
    gen trial = `j'
    gen ratio = success/elig
    if `j'>1 {
        append using `tofill'
    }
    qui save `tofill', replace
 }

mean ratio

 We're interested in the mean of the ratio, taken across trials. I get 0.4974, with a standard error of 0.0005 (which accounts for the fact that 10,000 is less than infinity) relative to the true conditional probability of 0.5. So, the bias is of the order of 0.0025 (or, 0.0035 at the bottom of the 95% confidence interval), not 0.02 as Gelman hypothesized. In other words, the Miller & Sanjurjo effect is at most a second-order concern for Gilovich et al.

Or am I missing something?

P.S., as a sanity check, when I run this code with $obsmax = 4, I get a mean ratio of about 0.4, as predicted by Miller & Sanjurjo.

Edit: Yes, I was missing something! Miller & Sanjurjo's claim with respect to Gilovich et al was not about "Study 2" of Gilovich (reported in Table 1), but "Study 4" (reported in Table 4). Study 2 involved actual shots by 76ers players during a large number of games, but there are reasons to doubt those results even without the Miller & Sanjurjo sampling bias. "Study 4" is a much purer study, involving Cornell basketball players as part of a controlled experiment. In this study, the typical sample size was 100, not 250.

Furthermore, Gilovich et al were examining P(hit | 3 misses) relative to P(hit | 3 hits). Conditioning on three hits (or misses), rather than just one, exacerbates the Miller & Sanjurjo bias. When I edit the code above to use a sample size of 100, and condition on three hits instead of one, I get an average probability of 0.461, so the bias is on the order of 0.04 (or 0.08, if we're comparing it to P(hit | 3 misses)), which is certainly nontrivial.

Many thanks to Jonathan Miller for pointing this out to me.