Hi Isaac,

Thanks for this interesting research that extends the work on trying to determine objective predictors of performance.  The FAE is a huge issue in every domain - even graduate school admissions - and underlies some of the justifications for racist policing and social policies in this country.  Making sure people understand the impact of the FAE is incredibly important, so that ultimately we can address the underlying systemic and situational factors that actually impact people's "performance" in all domains. 

I'm actually surprised by your data that show that scouts pay as much attention as they DO to situational factors (or at least claim to pay attention - which is a big difference.)  So given how much variance in performance situational factors actually predict, how do YOU think scouts should make decisions about who to draft?

Looking forward to hearing more detail on Monday!

Cheers,

Dr. Hunt

Fun! Two questions: Do you think some scouts thought individual and situational factors were hard (or impossible) to separate? Second, what would you do if you wanted to teach scouts about this error? What would be most convincing? Study 1 data? Study 2 data? Or something else?
Thanks,

Barb Mellers

Nice job, Isaac.

Interesting to identify some situational factors that contribute to quarterback performance in the NFL and that scouts may need to re-evaluate some of their weighting factors. I have to imagine that some machine learning or other big data approaches have been thrown at this question to understand how individual attributes and prior experience (e.g. quality of receivers in college, offensive scheme) interact with offensive scheme, receiver quality, left tackle experience, etc. etc. to ultimately shape performance in the NFL. Indeed, you could think of a myriad of external features as well, such as the quality of the defenses within the conference etc. that would also contribute to success. In the face of all these complex interactions, the "eye" test often works, and seasoned scouts like Mel Kiper have done a pretty good job predicting NFL outcomes for QBs coming out of college, with a couple of notable exceptions (e.g. Ryan Leaf, Tom Brady): https://www.saturdaydownsouth.com/tennessee-football/mel-kiper-jr-ranks…

We'll see how Trevor Lawrence does in the next couple of years. How would your scouts predict his performance given Jacksonville's receivers?

Dr Platt