OK Cupid has been doing some interesting things recently with data mined from their users which, on the one hand, is fascinating and, on the other hand, is extremely creepy.
It all boils down to some pretty interesting data points (what do white/black/asian/Hispanic people like or have in common, what is the focus of men and women of different races, etc). The one thing I was a little shocked by was this graphic at the end.
I was a little surprised that the Protestant reading level was so low. And when I am surprised by data, I try to see if I can replicate it. (This strikes me as an eminently scientific thing to do and yet I am bemused when people think I’m “attacking” their data. Whatever.)
Note: I should mention at the outset that it does kind of irk me that OKCupid displays this data and then kind of assumes that it holds for all people across the board when it is obvious to anyone who devotes more than 5 seconds to thinking about it that the data really only holds for OKCupid users who (I’m going out on a limb here) are probably disproportionately young, tech savy and single.
Moving along. To determine reading levels they ran the Coleman-Liau Index on the profiles so I went and typed up two sample religious profile summaries, one Christian and one atheist. They’re only a couple sentences long which I figure is fine since OKCupid profile summaries aren’t exactly known for their complex narrative arcs.
Here are the profiles that I typed up, attempting to mimic what I thought would be a fair religious summary from a similar reading level.
Atheist: I am an atheist. I believe that there is no God and that most people only believe religion because they are taught to do so by society and possibly also their parents.
Christian: I am a Christian. I believe that Jesus died on the cross for my sins and that he was raised again on the third day. I think that the Bible teaches us the truth and that God loves us very much.
Running them through this Coleman Liau tester to see what popped up.
Christian: 5.47 grade level
Atheist: 10.15 grade level
I laughed.
Really? Those little blurbs are so radically different that the Christian one is 4.68 grades stupider based on nothing more than a readability analysis? Sounds like BS to me.
Let’s try adding some evolution in there:
Atheist: Same as before + “When it comes to the world around us, evolution is the most likely explanation for everything.”
Score: 12.23
Christian: Same as before + “When it comes to the world around us, I think there are probably gaps in evolutionary theory and that evolution can’t explain everything.”
Score: 8.90
Well, that closes the gap by 1.35 points, but we’re still looking at a 3.33 grade gap between two positions that are transparently written with an identical textual style.
I progressively tried to add more and more to the Christian profile to counter-act the low score I got from starting with the basics of Christian belief.
Finally I ended up with:
I am a Christian. I believe that Jesus died on the cross for my sins and that he was raised again on the third day. I think that the Bible is true and that God loves us very much. I think there are probably gaps in evolutionary theory and that evolution can’t explain everything. Additionally, the philosophical underpinnings for views that argue against Christianity frequently neglect to apply the same standard of ideological rigor to their own faith based assumptions. Consequently, they hold Christianity to a double standard assuming that their position is the default one and that there is no need to defend it.
Score: 12.33
This “Christian profile view” scores about as well as an absurdly simple statement of atheism with a supporting line about evolutionary theory. Basically, the algorithm they used translates “6th grade atheism” at the same level of textual complexity as “basic Christian beliefs + a philosophy degree”. (I flatter myself somewhat, but the final two lines are clearly a college level writing style.)
Here is not what I’m saying: I don’t think there is any level of conspiracy theory behind any of this. No one designed the algorithm so that Christians would look stupid.
However!
It seems likely that a simple statement of Christian belief like the one entered above anchors the score at the low end. The more someone communicates their Christian belief in the language that has been familiar in churches for centuries, the less likely they are to score well regardless of the remaining textual analysis of their profile. This anchoring effect might get lost if the profile was a three page essay. But profiles tend toward being short, simple statements meant to clearly indicate basic beliefs, inclinations, or personality traits.
Note: I promise I’ll pull back on religious topics. I just get irritated when people pull “evidence” of religious people being inferior in some way shape or form. It usually strikes me as hackery that the creators or purveyors of whatever data set are perfectly happy to accept and so they neglect to do any sort of skeptical follow-up.
You can actually see the same thing with a lot of war-based data. Half the time, the people pointing to the data didn’t even get the data right and a good chunk of the remaining examples strip context out of the data. Bugs the hell out of me.
There is another problem with the algorithm. If I say, “I’m buddhist because Buddha is cool,” it cannot be stated conclusively that I am stupid, or intelligent. It can merely mean that I don’t care that much about religion, or that I just don’t care that much about stupid surveys. This survey seems to equate effort with intelligence. Clearly the more effort you put into the Christian answer, the higher the score. Conversely, a cursory or off-the-cuff answer wouldn’t necessarily prove a shallow mind.
little confused as to your surprise. some of the results re: religion have to be with ethnic confounds. the overwhelming majority of blacks are protestants, and the majority of latinos are catholics. these two groups don’t do well on the other tests. hindus, buddhists, are disproportionate indian and asian (50% of american buddhists are white, the other half asian). additionally, there’s a moderately robust trend whereby liberal protestants, hindus, secularists, etc., do better on psychometric tests and attainment on various academic measures than fundamentalists protestants, who are going to be disproportionately represent in this protestant set (since blacks are more fundamentalist than not).
you can confirm what said above in the GSS and NLSY. if you need pointers to the references, or have already put up critiques, i’m game.
This was interesting, I ran the following through the tester and got a 5.73 (despite the use of “ain’t no” and the use of contractions)
“I ain’t no Christian. I don’t believe that Jesus died on the cross for my sins and that he was raised again on the third day. I don’t think that the Bible teaches us the truth or that God loves us very much.”
On a side note the figures you were using were from the Gunning Fog index, not the Coleman Liau index (which is generally lower than the Gunning Fog and is located in the box below it).
I think the reason for the difference is clear if you look at the formula. The formula ranks longer words withing longer sentences higher. When you stop to think about the content of the phrases itself, it makes more sense. “God” has three letters, “Jesus” is a five. If you’re writing about a Christian faith, odds are you’re going to have those words pretty often which will bring down the average pretty quickly.
With that in mind, look at your original phrases:
* The Christian phrase has 152 non-space characters making up 41 words (3.7 letters/word) and three sentences (14 words/sentence).
* The atheist phrase has 136 non-space characters making up 32 words (4.25 letters/word) and two sentences (16 words/sentence).
In the Christian phrase, the average word is 12% shorter and requires 50% more sentences but only adds 16 characters. The additional phrases – despite their similar complexity – served to bring up their averages to 4.1 and 4.4 letters/word while putting both at 16 words/sentence. The deficit for the Christian phrase was just to great.
Though Christians could make a few tweaks to skew higher:
* “I am Christian” -> “I practice Christianity” – 4 char/word -> 7 char/word – 3 words/sentence
* “I believe in Jesus” -> “I follow the Messiah” – 3.75 char/word -> 4.25 char/word – 4 words/sentence
* Or simply: “I practice Christianity and follow the Messiah” – 5.7 char/word – 7 words/sentence
Regardless, an interesting thought exercise.
OW!
KEITH CASEY EVERYONE! Thank god, or Buddha, or nobody, because somebody has posted a reasoned, and thought out response to an article posted on the internet. Let’s everyone thank Keith for being here, on the internet, saving articles like some sort of statistical bat-man.
Pingback: Fundamentalists are not as “intelligent” | Gene Expression | Discover Magazine
Are you sure that OKCupid only did the sentence analysis on the writing that the Christians or Atheists did solely about religion? I can’t see them doing that, even though it’s not stated. What’s more likely is that they processed their entire profile blurb to get the writing score, and that is NOT limited to religious statements.
I can appreciate the fact checking, but make sure you know all the assumptions you’re starting from as well. Perhaps it would have been more informative if you had contacted OKCupid and asked them what they based their writing ability score on before writing this post?
I believe you begin to hint at the real problem as you extend your writing samples. I don’t know much about this metric, but from what I’ve read, it seems to be applied to very large writing samples (entire text books). You are basically trying to get an average from a set of data that is too small. I imagine the OKCupid profiles are significantly shorter than a text book as well.
Also, it is worth noting that they truncated the bar graph to show only the top 30% of the bars which greatly exaggerates the differences visually.
Thank you for such an insightful and clear blog. If the Coleman-Liau Index has taught me anything, it’s that it doesn’t matter what you say as long as it’s with very big words in the fewest sentences possible 😉 Simplicity may be the rule in Math, Engineering, Art, Music, and just about anything else in this world worth it’s salt. But when it comes to language describing religion, lets change all the rules now, shall we? GK Chesterton said it better, “The simplification of anything is always sensational.” – Varied Types
http://pewforum.org/Other-Beliefs-and-Practices/U-S-Religious-Knowledge-Survey.aspx I was wondering what your take on this was, such as where they got their data and how they played with it? It seems wrong somehow…
Pingback: Everybody loves charts, right? « About…
So I decided to check your math (or rather, that of the Coleman-Liau tester you used in your calculations) and I discovered some interesting things. First, the numbers you used from the tester were not the Coleman-Liau index numbers, they were the Gunning-Fog index numbers. No, really: the ones you’re looking for are the first listed under “Approximate representation,” and they’re considerably lower than the GFI values you wrote about.
I also crunched the CLI numbers myself and came up with slightly different values from the tester app. The values I got skewed lower than the tester app’s, and were within 0.05 of the app’s values with a constant percent error of about 0.25%. In light of the following, however, it’s completely insignificant.
Looking at the algorithm listed on the wikipedia page you linked to, I noticed that the CLI index performs calculations per 100 words, and this led me to question the accuracy of the index on very small blocks of text. I found that changing the word “me” to “our” in your first Christian blurb, by my application of the wikipedia algorithm, moves it up from 3.403 to 3.546, a jump of over 4% with the change of a single letter. I applied the same logic to the longest version and came up with 10.956 for “my” and 11.012 for “our,” – a difference of slightly more than 0.5%.
I then changed the original to “I am a Christian. I believe that Jesus died on the cross for my sins, that he was raised again on the third day, that the Bible teaches us the truth, and that God loves us very much.” This time I got a value of 4.305. That’s a 26.5% increase over the version with an extra period. But who cares – we’re comparing this to the atheist numbers.
The original atheist blurb came out at 6.973 for me, which is still a huge difference over the Christian one. Clearly we’ve got a problem with our test samples and we need to try again with more normalized input. Let’s take the two sentences you added to your originals, even them out a little bit, and compare again.
Atheist: “When it comes to the world around us, evolution is the most likely explanation for everything.”
Christian: “When it comes to the world around us, I think that the theory of evolution can’t explain everything.”
There – that’s pretty even, let’s check again:
Atheist: 77 chars, 16 words, 1 sentence, L=481.25, S=6.25, CLI=10.648
Christian: 83 chars, 18 words, 1 sentence, L=461.111, S=5.556, CLI=9.668
Less than one grade difference, which is considerably nicer than the OKCupid data shows and is simply the result of having an extra couple of words. What can I say, it takes more words to argue against something than to agree to it. Unfortunately, what this shows us is that once again, the length of the text blocks we’re feeding in are far too short, and as a result this kind of analysis is completely pointless. The algorthim simply isn’t meant to be used on blocks of text this short.
As David Kopp pointed out, in all likelihood the entire profile was polled. Take a look at the profiles pictured at the top of the OKCupid page all this is taken from: they’re easily 100+ words each, and only one of the default subheadings (“My Self Summary”) would really accommodate any kind of religious self-identification. Indeed, by the data listed on that same page, the average profile essay is a little over 530 words long, and less than 20% of all profiles even mention any religion or deity at all. The data about religion vs. CLI is taken from the information people enter about themselves, not by data mining the descriptions. Hence there can’t be any kind of bias against specific religions for at least 80% of all profiles.
But let’s assume that the 20% of profiles that do mention religion bring the rest of them down. Your basic argument is that “God” and “Jesus” are shorter words than “Allah” and “Muhammed,” and therefore the Christians skew lower because the names they use are shorter. You’re absolutely right about this: even if everything else in a profile essay is 100% identical, a dozen or so such replacements per hundred words could raise the CLI by more than one grade point. Even if we assume that this is true, and the average overtly protestant user’s profile includes more than 60 uses of “God” or “Jesus”, weighting it down to 20% means that those users would have to be scoring around 3 grades lower, on average, than other protestants to get them to ~0.6 grades lower than the comparable Muslims (which by your logic also skew lower for the same reason).
One last gripe: you mention that the OKCupid people are dishonest for displaying this data because they don’t mention that their users skew “young, tech savvy and single.” You’re right that they don’t give a direct disclaimer in that blog post, but they did say “I remind you that OkCupid’s user base is almost all in large cities,” and they have made these kinds of disclaimers time and again in other blog posts. I would assume they’re comfortable in the knowledge that their actual users – the people who read their stats blog for cool help using their site and not for profound racial and religious commentary – have read their old stuff and know what’s going on. I think it’s also safe to assume they don’t give a crap about the people who followed a link from graphjam and didn’t read any further than the first few paragraphs.
There are still two problems with your argument for chicanery, though. First, they never said that their data holds true for the wider population, they are a dating site and their purpose is to inform their users, not to be a platform for social commentary. The closest they’ve ever come to such a declaration, in any of the blog posts, is saying that their data is likely to be better than that of any other dating site (and they’re right, it’s much better). Second, the same bias applies across the board, to all the religions evenly. Unless you believe that the data miner just happened to randomly choose a disproportionately high number of old Christians and young Jewish, Buddhist, atheist and agnostic college students, it just doesn’t make any sense.
So the upshot is this: your objections to the data are valid, but not for the reasons you give. The problem is not that there’s any kind of bias in the data set, it’s that Christian users have shorter words to describe their beliefs and the profile essays aren’t long enough for a readability index to be applicable. Try not to take this so seriously, though. The graph we’re arguing about isn’t even the main focus of the blog post; it’s the silly crap they stuck in at the end to joke about the data. There’s not even any commentary on it, it’s just presented as-is with a description.
If you really want to talk about OKCupid and experimental bias, the post “The Democrats are Doomed” outright ignores factors in their data leading to bias, and is overtly written from the perspective of the democratic side. That one actually tries to make arguments about the political climate of the entire country in the context of a biased data set, and uses false assumptions about that data.
Two points:
1) Readability indices as a measure of reading proficiency.
Readability indices do not measure the reading proficiency of the writer. Rather, they measure appropriateness of material for a TARGET audience. For the sake of argument, let’s assume that you can come up with two tests that generate scores for reading proficiency and readability on an equivalent scale. At best you can argue that the readability score of an individual’s writing is unlikely to exceed that individual’s reading proficiency score. You cannot safely argue the reverse.
2) Readability indices as a measure of writing proficiency. As a technical writer, I take pride in writing clearly, so that anyone can understand my work. I know from personal experience that it takes a lot of effort to transform engineering gobblety-gook into simple language appropriate for any audience. So, from my perspective, rather than indicating relative “stupidity,” a lower score is indicative of a more inclusive communication style.
Way to use two data points (that you yourself created) to refute an entire study’s worth of data.
As others pointed out, OKCupid in all likelihood pulled the ENTIRE profile from probably THOUSANDS of profiles, not just their responses for the religious item or from only two profiles.
Please stop polluting the internet with horrible statistical abuse. Thanks!
Besides the disturbingly divisive nature of the correlation they are trying to draw (regardless of the outcome), one factor that they probably didn’t take into account, based on aggregate numbers, is that most likely serious Christians are not on OK Cupid, which lowers the overall pool of analysis. In fact, I’d wager that a majority are on specialized sites targeted to their demographic as these demographics tend towards serious relationships with long-term prospects and very similar belief systems. I would think that if you did this study based on samples from those specialized dating sites targeted to specific groups and compared the results, that the marginalized Christian cohort would attain a much higher number than this study suggests (even if there are shorter words used by some groups). Also, a site like eHarmony, which isn’t necessarily biased towards a religious component, would probably produce better results because it requires you to actually take time to setup a profile with substance. Their site is so daunting that it filters out less intelligent individuals (which is really what this graph is trying to convey) because anyone without patience quickly abandons the sign up process.
What’s most disturbing about this is that people will cite this type of thing as ‘gospel’ to prove that other groups are less intelligent than they are. Sadly, this will only help, even in error, to justify Atheistic attacks on Christianity.
Of course, statistics always takes samples, but why? If you have a whole user base to draw from, why not just process the whole thing? In the past it made sense to sample a population because you couldn’t possibly sample the whole; but now, in this golden information age, we can use the whole set.
Anyway, thanks for such a great site, keep up the good work 🙂
oil expeller is a leading manufacture
I have recently started a web site, the information you offer on this website has helped me tremendously. Thanks for all of your time & work.
I know this is an old blog post, but I just had to reply.
Some tests:
“The evolutionary theories of emminent scientists have been remarkably quiescent with respect to historical trends.” Score: 19.33
“Aaa bbbbbbbbbbbb cccccccc dd eeeeeeee ffffffffff gggg hhhh iiiiiiiiii jjjjjjjjj kkkk lllllll mm nnnnnnnnnn oooooo.” Score: 14.00
“Aaa babababababa cacacaca dd ebebebeb fafafafafa gagg hahh ibibibibib jajajajaj kakk lalalal ma nanananana obobob.” Score: 30.00
The index metric relies very strongly on the number of syllables, but not at all on spelling, grammar, and certainly not word definitions. Assuming that it is scoring legitimate text and not the nonsense I inserted, it will very roughly correspond to the reading level one needs to understand it. If one were trying to score high on such a metric, one would be encouraged to use the passive voice (e.g., “has been” instead of “is”), long polysyllabic words, and avoid excessive use of articles and conjunctions.
Those who write clearly and plainly will score low. Those who write obscurely will score high. Even the test that used real English was just pretentious nonsense, using lots of words to say essentially nothing.
I also note that the purpose of the metric appears to be to improve readability, not measure the education level of the writer. One’s goal should be to have a low score. OKCupid is clearly misusing the test by giving the impression that high scorers are somehow smarter or more educated. A correct reading of the results would be that Protestants and Catholics speak plainly, while atheists hem and haw and speak pretentiously.
I’m with JamesD and some of the others. In the first example, for the Christian description, if you change the first period to a semicolon the score jumps by about 3 grade levels.
The larger point is that it is a readability score: that is the grade level you need to understand what is being written, not the grade level of the writer. In technical writing, to convey complex ideas, it will usually be necessary to resort to big words and complex sentence structures. In everyday writing that will almost never be necessary.
One could better conclude that Protestants are straight talkers while atheists are bloviating blowhards.
Hello, i believe that i noticed you visited my web site thus i got here to go back the prefer?.I am trying to find things to enhance my web site!I assume its good enough to use a few of your concepts!!