Sunday, November 19, 2006

Statistics Without Context

I'm nothing if not fair- or at the very least, I hold everyone to the same standards, no matter how insane those standards may be. One of the themes of this blog has always been the intense scrutiny of statistics- particularly statistics that supposedly tell us important truths about the world that we live in. Generally, I have no complaints about numbers, only about what people purport certain numbers to mean.

In that grand tradition I give you FamilyFacts.org, a website devoted to providing conservative sociological research about the family.

Just to give you a taste, there is this finding: Children raised in intact families have, on average, higher academic achievement, better emotional health, and fewer behavioral problems. Again, I don't question the results- I just questions whether they have any real meaning. What studies such as these don't take into account are what factors into why families remain intact in the first place. For instance, if poverty is a factor in not having intact families in the first place, maybe poverty is the social problem in need of addressing. In other words, are family structure issues a cause of problems, or are they merely an effect of other social problems?

This is the same problem that I often point out with drug abuse statistics. Correlation does not equal causation, and correlation alone doesn't tell you very much. (For instance there is a correlation between drunk driving accidents and late night driving- that is, drunk driving accidents are much more likely to occur at night. But that doesn't mean that the time of day is the cause of drunk driving accidents. As far as dealing with the problem, we may want to provide a greater police presence of the roads at night because we know that is when they are likely to happen, but the fact that such accidents happen at night doesn't provide much help in stopping drunk driving in the first place.)

My point is- as always- to question those statistics that couch an agenda. Numbers can be manipulated, and this Family Facts site is no different. Beware.

3 Comments:

Blogger A Fan For All Seasons said...

Warning: My post will have nothing to do with politics.

I find it very interesting that you posted in this manner. I know I'm going to compare two completely different subjects, but I don't care. What you wrote is exactly what I always argue about statistics in sports. Numbers can't tell the whole story of a player, even for a sport like baseball. I hear it all the time from you and the racoon-fearer, "the baseball season is so long that numbers are the tell-all tale of a player". To an extent, there is truth to that statement. There are so many different statistical categories and so many breakdowns, that it may seem you can disect a player head-to-toe, but like any other sport, you have to watch a player to really know what's up and here's why. They're all under different context. If each player saw the exact same pitch, in the exact same location, at the exact same speed, then numbers would tell you everything you need to know. However, each batter gets pitched to based on their own strengths and weaknesses. Take David Ortiz, who batted .287 last year. Some will say he's an average hitter because of his batting average, but that average doesn't take into account the shift that Papi faces everygame. He may have been robbed of dozens of hits because of that. Batting average doesn't figure in how well you're swinging the bat either. You could have 5 hard hit line drives caught, and then you could bunt a ball down the 3rd base line and get a hit. My favorite example of faulty numbers - Alex Rodriguez. I probably watched 2/3 of the games this year, and I will tell you this with complete honesty, A-Rod's numbers are not a reflection of his play. Yes he had over 100 RBI, but how much of that was the opportunites he had? With Damon and Jeter ahead of him, and Abreu for half of the season too, A-Rod had plenty of guys on base during his at bats. He struck out or popped up in almost every big situation and I can only remember 1 big hit he had. He didn't play well this year, but his numbers might make you think otherwise. Numbers are nice sometimes, but as I always say, seeing is believing.

2:50 PM  
Blogger QU 3L said...

wGood point- Let me continue with the sports examples. Last year, I thought the NFL's MVP should have been Tiki Barber, not Shaun Alexander. I'm a numbers geek, so let's take a look at the numbers.

On the year, Barber finished with 2390 total yards and 12 TD's. Alexander finished with 1958 yards and 28 TD's. But remember, we're talking about the MVP (Most Valuable Player) not the BSP (Best Statistical Performer.

Alexander's numbers were padded by the Seahawks easy schedule- they twice played the 49ers (4-12), Cardinals (5-11) and Rams (6-10), and had additional games against the Texans (2-14), Titans (4-12), Packers (4-12), and Eagles (6-10). Additionally, the Seahawks caught the Colts the week after the Colts on a week when the Colts rested everyone. This is not to say Alexander's performances weren't impressive- just that the Seahawks would likely have won the vast majority of these games without him. In 5 games against teams .500 or better (excluding the Colts game), Alexander finished with 405 total yards and 3 TD's. Not bad, but nothing spectacular. Over the course of a 16 game season that would work out to 1296 total yards and 10 TD's.

Now let's look at Barber- Unlike the Seahawks, the Giants played a very difficult schedule. In 9 games against teams finishing over .500, Tiki amassed 1461 total yards and 6 TD's. Over the course of a 16 game season this would work out to 2597 total yards and 11 TD's.

The point here, is that statistics can be useful if they're used properly. Shaun Alexander may have put up 621 yards and 11 TD's in wins over the Cardinals and Rams, but how much does that matter in determining who was the MVP.

Although even my numbers have their weaknesses. Not included in my stats for Tiki is his 263 yard, 1 TD performance at Oakland in week 17 when the Giants needed a win, and the Raiders put points on the board and played them tough.

Ultimately, even numbers can lie. Last year, I can recall several games where Tiki carried the Giants- I can't say the same for Alexander.

3:11 PM  
Anonymous familyfacts.org said...

The familyfacts.org Web site is designed to enhance public discourse about family, religion, and society by making social science research easily accessible to the non-specialist.

An empirical social science research tool, familyfacts.org is a searchable database of the most pertinent findings on family and religion research. The database presents findings gleaned primarily from studies published in respected peer-reviewed journals, ranging from the Journal of Marriage and Family to Pediatrics.

When familyfacts.org presents a finding, it does so with clear reference to its context, providing information about the source, sample size, and variables studied. Moreover, familyfacts.org users can also see findings from one source in the context of other research by searching by category, article, journal, author or keyword.

Simply summarizing the research findings as published by the researchers, familyfacts.org does not interpret the findings, take policy positions, or make recommendations.

3:53 PM  

Post a Comment

<< Home