[翻译]90%的统计数据是错的，以及其他垃圾调查事实

这篇文章很好地演示了怎么反驳人家的调查报告，写得很不错，大家可以参考。
我花了些时间把它翻译了一下，翻译水平不高，见谅。

[译文]
90%的统计数据是错的，以及其他垃圾调查事实

在四月份的时候，我描述过这样的现象：正规的研究报告会意外地被歪曲并传播到网络上，甚至最终得出了和研究报告实际显示的结论180度相反的结论。垃圾研究将这种研究提高到了一个新的层次。

聪明的公共关系部门的人员和精明的产业玩家很清楚地知道网络上的许多信息是被误解的。你可以想象他们中的一些人会想：为什么不确保一些"误解"是我所期望的呢？这个游戏当然比互联网的历史更久，但互联网提供了更强大的扩散能力。所以有时候好的研究被排挤成了边缘研究，而坏的研究获得了远远超过应有的关注。

一个案例指出：由在线测试交流公司(Online Testing Exchange)主持并由美国电影协会(Motion Picture Association of America)发布的一项近期的调查。在其他令人难以置信的断言外，MPAA针对这项调查的总结声称，“大约1/4的互联网用户下载过一部电影”(这个陈述后来被证明并非事实，但这个陈述在其后的一周内传遍了全世界)

不难了解到这些日子里MPAA关心的是什么。只需要访问他们的网站，你会看到无数的链接提到“尊重”版权的无比重要性以及“盗版”的问题。这会让你联想到由他们发布的调查——支持他们观点的——无疑会在结果发布前至少被要求接受一下检查，对不？

可惜没有。如果你想阅读一些关于该调查的典型的新闻，可以参阅ClickZ Stats，IDG新闻服务(在PC World杂志上发表)，The Guardian，或者那些配着典型的耸人听闻的标题例如“韩国猖獗的图像盗版：超过一半的网络用户曾下载过图片”的娱乐工业各种各样的小报(这个站点要求注册)。

令人失望的是，看到媒体轻信这些研究——在很多情况下毫不怀疑地报道调查结论。如果要为他们辩解，我只能猜想当这些东西出现在许多过度操劳的日报记者收件箱的时候，他们没有时间或者经验来质疑这些问题。对努力填满自己栏目空白的记者来说有争议的调查报告是从天而降的礼物。

但这仍然是不可宽恕的。被简单地刊载了的这份调查报告甚至通不过“嗅觉测试”。

有太多的警告标志了。这份研究是由OTX——一家以电影工业为客户，前身是iFilm子公司的调查公司主持的。他们声称这项调查是“提供的一项服务”，这是一条可疑的声明，说明这项调查结构明显地标示着“MPAA和OTX磋商过的调查”。

调查结果很快地（而且不确切地）刊载在了MPAA的网站上并被广泛地发送给媒体，这正是垃圾调查的另一个标志。
至于方法论……好，我引用批评家的话来评论它。我最欣赏的辩驳来自The Register，它的一个记者，Ashlee Vance，花时间和OTX的人交谈并询问了一些巧妙的问题。借助Vance精巧设计的一些问题和观察，这项研究结论很快地崩溃了。例如，Vance询问调研问卷是否明确提及完整长度的有版权的长片。调研问卷显示没有明确指明——因此即使你合法地下载了一分钟的免费电影，MPAA也把你算成了“盗版”。

和Ashlee Vance一起，Yankee集团的Mike Goodman也反驳了这项调查，指出MPAA没有提及报导的41%的“电影下载者”比以前去电影院次数更多了。

还有许多明显的方法学瑕疵，包括在OTX没有得到100人最小样本数的国家，调查人数被放大了。这意味着结论或者是从一个小的样本中外推出来的，或者是混合了其他来源的数据。任何一种做法，都会引入了极大的误差。

聪明人能从这项调查报告中看到实质：新闻传播的运作。虽然如此，MPAA还是获得了它所期望的：许多出版社和至少一些人确信下载电影的“盗版”的假想的25%的人造成了“几十亿美元”的损失。

不要被愚弄了。这些资料不是分析；它们甚至没有好的原始数据。正当的调查不会掩饰它的方法学或者赞助人，并且总是承认潜在的其他观点和解释。像MPAA调研报告这样的资料是垃圾调查，它所具有的信息价值也一样。

[原文]

Back in April, I wrote about the phenomenon of legitimate research being accidentally twisted and distributed through the net, until it seems to support conclusions that are sometimes 180 degrees opposite of what the research actually showed. Junk research takes this phenomenon to a new level.

Clever PR folks and canny industry players know perfectly well that a lot of information on the net is misconstrued. You can imagine a few of them thinking, "Why not be sure that some of it is 'misconstrued' in my favor?" This game is of course much older than the internet, but the internet gives it a greater level of distribution. So occasionally good research gets spun into marginal research, and bad research gets much more play than it deserves.

A case in point: a recent survey conducted by Online Testing Exchange (OTX) and distributed by the Motion Picture Association of America (MPAA). The MPAA's summary of the survey claims, among other hard-to-believe assertions, that "about one in four Internet users have downloaded a movie." (It turns out this isn't true, but this is the factoid that was heard around the world the following week.)

It's not hard to figure out what the MPAA's hot button is these days. Just visit their website and you'll see numerous links referring to the critical importance of "respect" for copyrights, and also the issue of "piracy." So you would think that a survey released by them—supporting their position—inherently demands at least a token amount of examination before the results are published, right?

Unfortunately not. If you want to read a few typical news articles that resulted from this survey, see ClickZ Stats, IDG News Service (as printed in PC World), The Guardian, or the entertainment industry's tabloid Variety, with its classic screaming headline: "Pic piracy rampant in South Korea: More than half of Net users have downloaded pix." (Registration is required for this site.)

It is disappointing to see how the media gullibly picked up this study—in many cases unquestioningly reporting the conclusions of the survey. In their defense, I can only guess that many overworked daily reporters don't have the time or experience to question these things when they show up in the inbox. Controversial surveys are a gift from heaven for a reporter who is struggling to fill his quota of column inches.

But it is still inexcusable. Simply put, this survey didn't even pass the "sniff test."

There were plenty of warning signs. The study was conducted by OTX—a market research company with clients in the movie industry, and a former subsidiary of iFilm. They claim to have done it "as a service to the industry," which is a suspicious statement, given that the study results are clearly marked as "A Motion Picture Association of America survey in consultation with OTX."

The results were rapidly (and inaccurately) summarized on the MPAA's website and widely distributed to the media, which is another hallmark of junk research.

And as for the methodology...well, I'll let the critics speak to that. My favorite rebuttal is from The Register, whose reporter, Ashlee Vance, took the time to actually speak to someone from OTX and ask a few intelligent questions. With just a few well-placed inquiries and observations from Vance, the study quickly collapsed. For example, Vance asked if the survey questions referred to full-length, copyrighted, feature films. Turns out the survey didn't specify—so if you've ever legally downloaded a free one-minute movie, the MPAA would count you as a "pirate."

Along with Ashlee Vance, Mike Goodman of Yankee Group also rebutted the survey, pointing out that the MPAA failed to mention that 41 percent of "movie downloaders" reported going to the movies more than they had before.

And there were numerous other glaring methodological flaws, including the fact that the survey numbers were "augmented" in certain countries where OTX didn't get its minimum sample of 100. That means they either extrapolated from a smaller sample, or mixed in data from another source. Either way, it introduces a much greater margin of error.

The smart guys saw through this survey for what it was—a vehicle for press releases. But nonetheless, the MPAA got what it wanted: a lot of press, and at least a few people convinced that "billions of dollars" are being lost to the supposed 25% of us who are movie-downloading "pirates."

Don't be fooled. This stuff isn't analysis; it's not even good raw data. Valid research doesn't hide its methodology or its sponsorship, and always admits the potential for alternative viewpoints and interpretations. Stuff like the MPAA survey is junk research, and it has about as much informational value as a twinkie.

原文链接：http://www.alwayson-network.com/comments.php?id=5459_0_9_0_C

其后一些评论也可以看看