Sunday, 26 January 2014

A critique of the [adjective][species] survey methodology

The [adjective][species] (AJ) surveys and their results and various analyses have been getting quite a bit of attention lately, and I wanted to draw attention to the fundamental flaws in the methodology of the analysis of the data they collect.

I'll start with a slight disclaimer about this critique, I'm not a social scientist, but I am a Zoologist, or more specifically an Ethologist, and, as such, a core part of my work involves collecting data on the behaviour of animals and then analysing it statistically.

For this critique I'll be specifically focusing on their article on the furry fandom and re-evaluating one's sexual orientation as an example of the site's methodological failings that are so fundamental that I'd be surprised if any of them have any training in basic statistics or the scientific method.

So, where to begin...

A pretty visualisation does not analysis make


One of the major problems with this article is that it conflates making a pretty graph with analysing data. The writer uses the graph to assert a trend which indicates the hypothesis to be true; that the furry fandom does lead people to re-assess their sexual orientation. But there are no analytical statistics to back up this assertion, and there is no test (such as a chi square) to prove that this distribution did not occur due to random variation.

Some data points on the distribution make it obvious that you can't take anything for granted with this graph without a significance test, as I've highlighted in the image below, and these are just the worst offenders:


The number of heterosexuals can increase by about half in 3 years across the distribution (years 8-11) but apparently this isn't worthy of discussion. The number of pansexuals can more than double in a 1 year span and then drop off but this isn't worthy of discussion. There has been no critical analysis of the reliability of the dataset anywhere in the article. There has been no actual analytical test to check the statistical significance of the dataset anywhere in the article. This is a very basic principle of using any dataset like this.

Without testing that the data distribution did not occur by chance, why would I accept your hypothesis? For a scientist to accept your hypothesis, you need a 0.05 probability or less (less than 5%) that the data was acquired by chance alone. Simply looking at the distribution, I'd posit that the data would likely fail this test.

Even if you proved that the distribution is not random, you would then still have to establish a correlation coefficient, etc. with other tests.

Look, ma! No hands control!


For any scientific test, you really need a control sample. Re-evaluation of one's sexuality occurs all the time in the general population; running something like a Mann-Whitney test against a data sample from the general population is basically essential to establish that it differs at all from other populations.

Lack of alternative hypotheses


They present no alternative hypotheses that don't involve the fandom. There are a couple of obvious ones:

  • Furries who have been in the fandom for a shorter period of time are by definition going to be younger on average than those who have been in the fandom for longer; it's established that many furries first join the fandom in their teenage years. The author's own hypothesis asserts that the fandom doesn't "turn" people gay or bi, just makes them re-evaluate their sexuality. If these younger furries are in their teens, it stands to reason that they may re-evaulate their sexuality, anyway.
  • Since this is not a longitudinal study but based on a question asking someone to report how long they've been in the fandom for, it is impossible to determine whether they have re-evaluated their sexuality during their time in the fandom based on this data, or whether the growth and greater attention the fandom has received recently has attracted a greater proportion of heterosexuals to the fandom among younger cohorts.

Spurious claims


"The trend is almost certainly starker than the chart shows."
Says what statistical test?

"It’s safe to conclude that more than half of the heterosexual furries coming into the community will change their sexual preference."
Even if you were to prove that this is not a random distribution and establish a correlation coefficient, correlation does not imply causality, as I've demonstrated with my above alternative hypotheses.

Conclusion

This analysis is based entirely on a single data visualisation with no statistical testing whatsoever. The conclusions reached by the author of the article are completely pseudo-scientific, and even assuming, for the sake of argument, the data is non-random, and has a significant correlation coefficient, the author makes a spurious conclusion that correlation implies causality.

2 comments:

  1. i Xolani, I'm the author of the original article (@jmhorse on twitter).

    Thanks for the intelligent and informed criticism. You are, in general, spot on. It's nice to read something written by someone who knows what they are talking about.

    For the record, I have an engineering degree and a science degree, chemical engineering and chemistry respectively. My qualifications include statistics, and I use them from time to time in my daily work life. I am also familiar with the scientific method and have had several articles published in peer-reviewed journals.

    The [a][s] data is self-selecting - it's from people who visit furrypoll.com and fill in a survey. We get 5000 to 10000 responses each year. We have no way of testing how closely this data represents the real furry community, because we have no way of taking a statistically valid furry census. (Any ideas on how we can improve or test our data would be appreciated.)

    This key limitation in our data makes any statistical analysis moot. But then [a][s] is sociology, not a hard science. It is very, very common for sociological studies to be based on self-selecting samples, with no control group and no formal statistical analysis. Data is used to support ideas, and to test theories, as I have done in this particular [a][s] article. It's closer to philosophy than the hypothesis-data-proof model that you and I use in our work lives.

    ReplyDelete
  2. One specific comment: I ignored some features of the visualization (such as your example of changes in numbers of heterosexuals & pansexuals in years 8, 11, & 14), because the data becomes junk towards the right-hand end. You can see the actual responses here - http://vis.adjectivespecies.com/yearsorientation/ (bottom plot) - you'll note that we have few datapoints for people who have been around for more than 5 years (or so). The big changes shown in the visualization beyond this point are just magnified random variations. The "doubling" of pansexuals you cite, for example, is an increase from 5 people to 11 people.

    In any event, my analysis is not a mathematical one. The data visualization is evidence that supports the idea that it's common for furries to re-evaluate their sexual preference, which is why I've used it. (I also present other evidence, such as the A.L.F. question.) You'll note that I'm careful with my language, and never suggest that furry is "causing" anything; just that there is evidence for a correlation.

    The survey data is a great resource, but it can't prove anything because of the way it is collected. It can, however, provide a reference point for discussion and philosophical thought. That's what we're trying to do on [a][s], and that's what I've tried to do with this article.

    By the way, the International Anthropomorphic Research Project, started a longitudinal study just last year. Their datapoints are also self-selecting and so suffer from many of the limitations you point out in our data. But it should be very interesting to see, after 5 years or so, whether their results support our conclusions.

    Anyway, thanks for taking the time to write this article. I appreciate the thoughtful criticism and intelligent skepticism. And we're currently working through the 2013 Furrypoll dataset - I'm curious to see what interesting results we can tease out.

    ReplyDelete