Research Article

Questions You Should Never Ask an Atheist: Towards Better Measures of Nonreligion and Secularity

Authors: Ryan T. Cragun

Abstract

As interest in the nonreligious has grown, attention has turned to how to measure both the growth of and variation within the nonreligious. This interest has also revealed that prior measures of religiosity are often problematic. In this research note, I detail some of these problems. For instance, some measures fail to contrast nonreligiosity with religiosity. Other measures are double-barreled or one-and-a-half barreled, making them impossible for nonreligious individuals to answer. Finally, I note that how questions are worded can result in very different estimates of how many nonreligious people and atheists there are in a population.

Keywords: Article has no keywords
DOI: http://doi.org/10.5334/snr.122
 Accepted on 05 Jul 2019            Submitted on 05 Jun 2019

Introduction

Social scientists have known that religiosity is multidimensional since the 1960s (Glock and Stark, 1966). Since then, there have been numerous measures developed to try to capture individual religiosity, some with just a few dimensions, others with as many as eight dimensions (Hill and Hood, 1999). Various measures of religiosity continue to be used, depending on the context and group of interest (Cragun, Hammer and Nielsen, 2015). While those measures are varied, there seems to be growing agreement among scholars that there are least three core dimensions of religiosity that should be captured when trying to get a comprehensive measure of how religious someone is: belief, behavior, and belonging (Keysar, 2014).

We now know that measures of religiosity are typically not also measures of nonreligiosity or secularity (Hall, Meador and Koenig, 2008; Hall, Koenig and Meador, 2009; Hwang, Hammer and Cragun, 2011). Many measures of religiosity capture only how religious people are, ranging from very high levels of religiosity (presumably along the three dimensions just noted) to very low levels of religiosity. This is particularly problematic when researchers try to make comparisons between religious and nonreligious individuals as they may not actually be capturing whether or not someone is nonreligious or if they are affirmatively secular (Galen, 2012; Baker and Smith, 2015). To what extent the problems in measuring nonreligion are intentional or unintentional biases of pro-religious researchers is unclear (Cragun and Hammer, 2011). Additionally, while there have now been some efforts to measure the dimensions of secularity (Schnell, 2015), it’s not entirely clear whether the same dimensions of religiosity (i.e., belief, behavior, and belonging) apply to the nonreligious (Lee, 2014). The answer to the question, “How are people secular or nonreligious?” is still an ongoing concern.

While we have some ways to measure how religious or nonreligious someone, the use of lengthy scale measures in large, nationally-representative surveys is rare, likely because of the cost. Instead, most large-scale surveys include at most a few questions about religiosity (and, to date, virtually no explicit questions about how people are secular). In this research note, I will examine some better and worse ways to measure religiosity in light of the idea that such measures should also indicate something about an individual’s nonreligiosity or secularity as well.

Examples of Problematic Questions

An instructive way to think about better questions about nonreligious identity is to examine worse questions. There is a widely-used measure of religiosity that is extremely problematic. This measure has been used by WIN/Gallup International (2012) for decades and is also the primary measure of religiosity used in the World Values Survey (“WVS”; 2014). Here is how the question is worded:

Irrespective of whether you attend a place of worship or not would you say you are: (a) a religious person, (b) not a religious person, (c) a convinced atheist, (d) do not know/no response.1

There are numerous problems with this question. First, the researchers fail to realize that there is a difference between ‘believing’ and ‘belonging’. Religious affiliation (i.e., ‘belonging’) has to do with an individual’s sense of connectedness or alignment with a religious tradition (Day, 2011). Belief in a god or higher power can be and often is correlated with religious affiliation but it is not a religious affiliation (Cragun, 2016). By asking about both belonging and belief in this question, the question conflates two dimensions of religiosity. As prior research has shown, some atheists report religious affiliations (Sherkat, 2008) and not all nonreligious people are atheists (Kosmin et al., 2009; Lee, 2014).

Related to this, the question also falls afoul of warnings against double-barreled questions, or questions that ask about two separate and distinct issues simultaneously; these are strongly discouraged by principles of good survey design (Converse, 1986), as respondents may not be able to accurately address both elements of the question with a single response. The problems of double-barreled questions can be illustrated with the following example, “How strongly do you identify as being a Christian and a terrorist?” The respondent may very well be both a Christian and a terrorist, but, in all likelihood, they are probably only one or the other. The question asks about two issues simultaneously. This is really two separate questions, but people are forced to respond as though it is one. Likewise, WIN/Gallup’s question is double-barreled as it asks about both believing by asking whether people are atheists and also about belonging by asking about religious affiliation or nonaffiliation. Notably, the question is only double-barreled for atheists, because belief or, presumably, ‘convinced theism’ is not an option.

The question also violates the principle of exclusivity, that is, that each respondent should only fit into one category (Converse, 1986; Ritchey, 2008); respondents cannot choose ‘not a religious person’ and ‘a convinced atheist,’ even though most atheists are also not religious. The practical consequence of this question design is that it will reduce the number of people in both of those categories by forcing them to pick one of the two. The question also violates the principle of inclusivity, namely, that there should be a category for everyone. Which of the three response options would work for an agnostic who visits random religious organizations, but only on holidays? They are not “convinced atheists” but may also not consider themselves to be either religious or not religious.

Finally, the question also uses a modifier on ‘atheist’ that reduces the odds of people choosing that option, but there is not a comparable modifier for being a religious person (e.g., ‘a convinced religious person’) or a ‘convinced theist.’ Without imputing motives to the individual(s) who created this question, the likely consequence of including a modifier is that it will reduce the number of people who choose this option. In short, this question is a model of questions you should not ask atheists. I have actually raised this issue in conversations with atheists on several occasions, and most such individuals have quickly pointed out the various flaws with the questions, even if they were not well-versed in scale development (DeVellis, 2012) and questionnaire design (Converse, 1986).

WIN/Gallup and the WVS are not the only groups that have fielded extremely flawed survey questions (Bruce, 2002). Another highly problematic set of questions comes from Christian Smith’s National Study of Youth and Religion (Smith and Denton, 2005). In that survey, religious youth were asked, “In the last year, how much, if at all, have you had doubts about whether your religious beliefs are true?” In contrast, nonreligious youth were asked, “In the last year, how much, if at all, have you had doubts about being non-religious?” A cursory glance at these questions might leave some people thinking they are comparable. However, more careful scrutiny reveals that they are not asking the same questions. The first question asked of religious youth examines whether they have doubted their religious beliefs only. The second question, asked of nonreligious youth does not ask about nonreligious beliefs, which would make this a one-to-one comparison. Instead, it asks about identity or belonging. In other words, the researchers asked religious youth if they ever questioned some of their beliefs while they asked nonreligious youth if they ever questioned their identity.

Other problematic questions come from the work of Canadian sociologist Reginald Bibby (2012). In his surveys, he regularly asks participants, “Would you consider the possibility of being more involved in a religious group if you found it to be worthwhile for yourself or your family?” This question is double-barreled as it asks about self or family, either of which could elicit very different responses. That concern aside, there is an implicit bias in the question as it assumes that involvement in a religious group is worthwhile. As a result, the question puts the onus for finding religious group involvement not worthwhile on the individual, since it assumes involvement in religious groups is worthwhile. Finally, the question is leading, potentially contributing to social desirability bias (Stöber, 2001) as who would not want to be involved in something worthwhile?

The aim of the above questions could be as seen as trying to minimize nonreligion or secularity and maximize religiosity. As previous research has shown biases among social scientists who study religion against nonreligion (Cragun and Hammer, 2011), it is possible that these biases are intentional. However, even if the biases are unintentional, it is important for scholars studying the nonreligious or scholars studying religion who want to accurately portray the nonreligious to recognize that these questions are problematic at best and biased at worse. Question wording matters, as survey design experts have long known (Converse, 1986) and as others have shown with regard to nonreligion, specifically (Voas and Day, 2007; Day, 2011; Day and Lee, 2014). In the section that follows, I detail how even slight differences in question wording can result in very different estimates of the nonreligious population in a country.

“Unbiased” Measures Can Still Bias Estimates

An important lesson from the WIN/Gallup and WVS question is that scholars can build specific arguments about trends in religiosity by choosing carefully how they word the questions they ask. This can be illustrated by looking at some of the most widely cited surveys in the US, the General Social Survey (“GSS”; Smith, Marsden, and Kim 2012), the Religious Landscape Survey of The Pew Research Center (“Pew”; 2015), and Gallup’s regular surveys on religion (“Gallup”; 2017). Each of the surveys includes questions about religious affiliation and belief in a higher power or god in different ways, and each gets slightly different results.

Since 1972, the GSS has asked participants the following question to capture their religious affiliation:

What is your religious preference? Is it Protestant, Catholic, Jewish, some other religion, or no religion?

Pew asks the question differently:

What is your present religion, if any? Are you protestant, Roman Catholic, Mormon, Orthodox such as Greek or Russian Orthodox, Jewish, Muslim, Buddhist, Hindu, atheist, agnostic, something else, or nothing in particular?’2

Gallup’s question is closer to the GSS version:

What is your religious preference – Protestant, Roman Catholic, Jewish, another religion, or no religion?3

The resulting portraits of religion affiliation in the US for 2014 are shown in Figure 1.

Figure 1 

US Religious Affiliations in 2014 from GSS, Pew, and Gallup.

While there are differences, most are not that large. Gallup appears to be under-representing the nonreligious, estimating that group at just 16%, while the other two surveys estimate they are above 20% of the population, a difference of potentially tens of millions of Americans. The other sizable difference is the varied percentage of Protestants in the three samples, though most of this difference is accounted for in the percentage of Christians in Gallup. Data from years prior to 2014 (and since) suggest that either the way Gallup asks about religious affiliation or Gallup’s sampling methodology is resulting in lower estimates of the percentage of nonreligious Americans than is the case in the other two surveys. Of note, all of these surveys now include the option of ‘nonreligious’ or the less preferable ‘nothing in particular’. This was not always the case. As late as 2005, Gallup did not include this option and in surveys prior to the change, their estimate of the percentage of the US population that was nonreligious was even lower compared to the GSS and other national surveys than it is now. More accurate estimates of the nonreligious appear to require the inclusion of this option in the question.

There are bigger differences when it comes to belief in a god or higher power. In the GSS, participants are presented with a variety of response options after being shown a card, as described below:

  • Please look at this card and tell me which statement comes closest to expressing what you believe about God.
    • I don’t believe in God,
    • I don’t know whether there is a God and I don’t believe there is any way to find out
    • I don’t believe in a personal God, but I do believe in a Higher Power of some kind
    • I find myself believing in God some of the time, but not at others
    • While I have doubts, I feel that I do believe in God
    • I know God really exists and I have no doubts about it

Gallup uses a yes/no question with a slight variant for half of the respondents:

Do you believe in God? [or] Do you believe in God or a universal spirit?

Pew uses the same yes/no question as Gallup, initially:

Do you believe in God or a universal spirit?

Pew then follows up the question with a clarification:

How certain are you about this belief? Are you absolutely certain, fairly certain, not too certain, or not at all certain?

Figure 2 presents estimates from the three surveys of belief toward a god or higher power. Given that the questions are substantially different, I did my best to overlay Pew’s and Gallup’s response options with those of the GSS.

Figure 2 

Belief in a god or higher power from GSS, Pew, and Gallup in 2014.

As Figure 2 indicates, how you ask the question matters a great deal. If you ask only a yes/no question, like Gallup does, it appears as though 86% of Americans believe in a god or higher power. That is a full 20% higher than what Pew and the GSS estimate. Even Pew, with its two part approach, reduces the possible response options as it groups belief in a ‘God’ with belief in ‘a universal spirit’— not a small oversight, as the GSS estimates that 12% of adults in the US believe not in a personal God but in a Higher Power of some kind. The dichotomous response options in Pew and Gallup also hide a considerable amount of nuance that is revealed in the GSS around agnosticism. The GSS estimates that 6% of Americans are agnostics; technically, Gallup and Pew do not capture agnostics, though I grouped Pew respondents who indicated their certainty of belief in a god or universal spirit as ‘not at all certain’. It’s not a perfect match but illustrates the point that Gallup and Pew are not actually capturing agnosticism in a clear way.

Similar discrepancies in surveys can be seen in Europe. The European Social Survey (“ESS”) asks participants a yes/no question about their religious affiliation,

Do you consider yourself as belonging to any particular religion or denomination?

In 2012, the WVS, using the very problematic question discussed above, and the ESS question just noted were fielded in The Netherlands, providing an opportunity to compare the results of the two questions. Figure 3 presents the results from the two surveys. The WVS finds that about 11% of Dutch people identify as “convinced atheists,” an option that is not reflected in the response options of the ESS. In the ESS, 37% of Dutch people identify as having a religious affiliation, while 46% of Dutch people do in the WVS, a difference of 9%. In a population of 16.75 million people in 2012, that is a difference of about 1.5 million people. Given the problems with the WVS question, I’m inclined to believe that the question underestimates the percentage of nonreligious people (55% in the WVS vs. 63% in the ESS). As the differences in the US illustrate, how you ask questions about religious affiliation can influence the resulting estimates.

Figure 3 

Religious Affiliation in The Netherlands in 2012 from ESS and WVS.

Limitations

While the varied question wording is likely to have contributed to the differences noted above, it is also possible that other factors played a role. The surveys used different methods, had different response rates, and had different question orders, all of which are known to influence survey results (Kalton 1983; Converse 1986). It may also be the case that, despite the very poor question design in some of these surveys, respondents understand the intent behind the questions and are still able to respond to them in an accurate way. Without data collected directly from survey participants on how they understood the questions it is impossible to ascertain how they understood the questions they were asked.

Conclusion

Comparing questions from various surveys shows that how questions are asked about the nonreligious can influence what is found. Some of the problem is poor survey design (whether intentional or not). Another issue is that the lack of options can result in people being forced into categories that may not accurately reflect their views. For instance, Gallup’s simple ‘yes/no’ question about belief in God inflates the number of people who believe in a god or higher power by forcing those who doubt to opt into belief. As others have suggested (Lee, 2014), providing respondents with more options and, ideally, options that are not mutually exclusive may be the best path forward for gaining a better understanding of how people are nonreligious or how atheists identity themselves.

Notes

1The wording in the World Values Survey is almost identical, it reads, “Independently of whether you attend religious services or not, would you say you are: (1) a religious person, (2) not a religious person, (3) an atheist.” 

2Note that Pew conflates belonging (religious affiliation) with belief as well, as they include atheist and agnostic among the options, even though those are not religious affiliations. Admittedly, some atheists and agnostics report “atheist” or “agnostic” as their religious affiliation when asked (Kosmin et al., 2009), but those are both positions on the existence of a deity and not religious affiliations (Cragun, 2016). Also, including “nothing in particular” at the end of the question, while it may be intended to recognize the nonreligious, gives the impression that people who are not religious may be so for no meaningful reason, which is certainly not the case for many people who extricate themselves from exclusive, strict religions (Bromley, 1998; Cottee, 2015). 

3Note that Gallup’s question has changed; they added ‘no religion’ as an option in the question starting in 2006. 

Competing Interests

The author has no competing interests to declare.

References

  1. Baker, JO and Smith, BG. 2015. American Secularism: Cultural Contours of Nonreligious Belief Systems. New York: NYU Press. 

  2. Bibby, RW. 2012. ‘Why Bother with Organized Religion? A Response to Joel Thiessen’. Canadian Review of Sociology/Revue canadienne de sociologie, 49(1): 91–101. DOI: https://doi.org/10.1111/j.1755-618X.2011.01282.x 

  3. Bromley, DG. 1998. ‘The Social Construction of Contested Exit Roles: Defectors, Whistleblowers, and Apostates’. In: Bromley, DG (ed.), The Politics of Religious Apostasy: The Role of Apostates in the Transformation of Religious Movements, 19–48. Westport, CT: Praeger Publishers. 

  4. Bruce, S. 2002. God Is Dead: Secularization in the West. London: Blackwell Publishers. 

  5. Converse, JM. 1986. Survey Questions: Handcrafting the Standardized Questionnaire. Beverly Hills: Sage Publications. 

  6. Cottee, S. 2015. The Apostates: When Muslims Leave Islam. London: Hurst. 

  7. Cragun, RT. 2016. ‘Defining That Which is “Other to” Religion: Secularism, Humanism, Atheism, Freethought, etc.’ In: Zuckerman, P (ed.), Religion: Beyond Religion, 1–16. New York City: MacMillan Publishing Company (Macmillan Reference USA). 

  8. Cragun, RT and Hammer, JH. 2011. ‘“One Person’s Apostate is Another Person’s Convert”: Reflections on Pro-Religion Hegemony in the Sociology of Religion’. Humanity & Society, 35(February/May): 149–175. DOI: https://doi.org/10.1177/016059761103500107 

  9. Cragun, RT, Hammer, JH and Nielsen, M. 2015. ‘The Nonreligious-Nonspiritual Scale (NRNSS): Measuring Everyone from Atheists to Zionists’. Science, Religion, and Culture, 2(3): 36–53. DOI: https://doi.org/10.17582/journal.src/2015/2.3.36.53 

  10. Day, A. 2011. Believing in Belonging: Belief and Social Identity in the Modern World. Oxford; New York: Oxford University Press. 

  11. Day, A and Lee, L. 2014. ‘Making sense of surveys and censuses: Issues in religious self-identification’. Religion, 44(3): 345–356. DOI: https://doi.org/10.1080/0048721X.2014.929833 

  12. DeVellis, RF. 2012. Scale development: theory and applications. Thousand Oaks, Calif. [u.a.: Sage Publ]. 

  13. Galen, LW. 2012. ‘Does religious belief promote prosociality? A critical examination’. Psychological bulletin, 138(5): 876–906. DOI: https://doi.org/10.1037/a0028251 

  14. Gallup, Inc. 2017. Religion – Historical Trends, Gallup.com. Available at: http://news.gallup.com/poll/1690/Religion.aspx (Accessed: 12 December 2017). 

  15. Glock, C and Stark, R. 1966. Christian beliefs and anti-Semitism. New york: Harper & Row. 

  16. Hall, DE, Koenig, HG and Meador, KG. 2009. ‘Hitting the target: why existing measures of “religiousness” are really reverse-scored measures of “secularism”’. Explore (New York, N.Y.), 4(6): 368–73. DOI: https://doi.org/10.1016/j.explore.2008.08.002 

  17. Hall, DE, Meador, KG and Koenig, HG. 2008. ‘Measuring religiousness in health research: review and critique’. Journal of Religion and Health, 47(2): 134–163. DOI: https://doi.org/10.1007/s10943-008-9165-2 

  18. Hill, PC and Hood, RWJ. (eds.) 1999. Measures of Religiosity. Birmingham, Alabama: Religious Education Press. 

  19. Hwang, K, Hammer, JH and Cragun, RT. 2011. ‘Extending Religion-Health Research to Nontheistic Minorities: Issues and Concerns’. Journal of Religion and Health, 50(3): 608–22. DOI: https://doi.org/10.1007/s10943-009-9296-0 

  20. Kalton, G. 1983. Introduction to Survey Sampling. Sage university papers series. Beverly Hills: Sage Publications. DOI: https://doi.org/10.4135/9781412984683 

  21. Keysar, A. 2014. ‘Shifts Along the American Religious-Secular Spectrum’. Secularism and Nonreligion, 3(1): 1–16. DOI: https://doi.org/10.5334/snr.am 

  22. Kosmin, BA, et al. 2009. American Nones: The Profile of the No Religion Population. Hartford, CT: Institute for the Study of Secularism in Society and Culture, 29. 

  23. Lee, L. 2014. ‘Secular or nonreligious? Investigating and interpreting generic “not religious” categories and populations’. Religion, 44(3): 466–482. DOI: https://doi.org/10.1080/0048721X.2014.904035 

  24. Pew Research Center. 2015. America’s Changing Religious Landscape. Washington, D.C.: Pew Research Center, 201. 

  25. Ritchey, FJ. 2008. The Statistical Imagination: Elementary Statistics for the Social Sciences. 2nd ed. Boston: McGraw Hill. 

  26. Schnell, T. 2015. ‘Dimensions of Secularity (DoS): An Open Inventory to Measure Facets of Secular Identities’. The International Journal for the Psychology of Religion, 25(4): 272–292. DOI: https://doi.org/10.1080/10508619.2014.967541 

  27. Sherkat, DE. 2008. ‘Beyond Belief: Atheism, Agnosticism, and Theistic Certainty in the United States’. Sociological Spectrum, 28(5): 438–459. DOI: https://doi.org/10.1080/02732170802205932 

  28. Smith, C and Denton, ML. 2005. Soul Searching: The Religious and Spiritual Lives of American Teenagers. USA: Oxford University Press. 

  29. Smith, TW, Marsden, P and Kim, J. 2012. General Social Survey. Chicago, IL: National Opinion Research Center. 

  30. Stöber, J. 2001. ‘The Social Desirability Scale-17 (SDS-17)’. European Journal of Psychological Assessment, 17(3): 222–232. DOI: https://doi.org/10.1027//1015-5759.17.3.222 

  31. Voas, D and Day, A. 2007. ‘Secularity in Great Britain’. In: Keysar, A and Kosmin, BA (eds.), Secularism and Secularity: Contemporary International Perspectives, 95–112. Institute for the Study of Secularism in Society and Culture. 

  32. WIN-Gallup International. 2012. Global Index of Religion and Atheism. Pakistan: Worldwide Independent Network of Market Research and Gallup International, 25. Available at: http://redcresearch.ie/wp-content/uploads/2012/08/RED-C-press-release-Religion-and-Atheism-25-7-12.pdf. 

  33. World Values Survey. 2014. World Values Survey Wave 6 2010–2014. Madrid, Spain: Asep/JDS. Available at: http://www.worldvaluessurvey.org/ (Accessed: 13 February 2007).