Principal An introduction to survey research

An introduction to survey research

, , ,
Survey research is a widely used data collection method that involves asking people questions and using their answers as information. Such data can then be used to understand individuals views in a variety of areas such as political issues, community quality of life, and satisfaction with services and products, to name but a few. Decision makers in both the public and private sectors use survey results to examine past efforts and guide future direction. Yet, we are often surprised by the misperceptions of students, professionals and even high-level managers regarding what is required to conduct a good survey. Our purpose in developing this book is to provide an introduction and overview of survey research. We begin our exploration at the foundation of gathering information about the worldobservation and questioning. We examine the processes of each, and talk about identifying the information needed and the best approach to getting that information. We then discuss the processes commonly involved in conducting a survey including both obtaining a representative sample and dealing with the types of errors that can distort results. From here we focus on the components of constructing and carrying out a survey including the elements to consider when developing a survey, types of surveys, writing good questions, conducting the survey, and presenting the results. . The purpose of this book is to provide an introduction to survey research for those who want an overview of the survey process. It is intended to describe fundamental survey components to help both students and managers understand and use surveys effectively and avoid the pitfalls stemming from bad survey construction and inappropriate methods
First edition
Business Expert Press
ISBN 10:
ISBN 13:
Quantitative approaches to decision making collection
PDF, 6.00 MB
Descarga (pdf, 6.00 MB)

You may be interested in


Most frequently terms

You can write a book review and share your experiences. Other readers will always be interested in your opinion of the books you've read. Whether you've loved the book or not, if you give your honest and detailed thoughts then people will find new books that are right for them.
An Introduction to Survey Research


overview of the survey process. It is intended to describe

Ernest L. Cowles • Edward Nelson
An Introduction to Survey Research is for those who want an
­fundamental survey components to help both students and

Curriculum-oriented, borndigital books for advanced
business students, written
by academic thought
leaders who translate realworld business experience
into course readings and
reference materials for
students expecting to tackle
management and leadership
challenges during their
professional careers.

managers understand and use surveys effectively and avoid


Ernest L. Cowles is professor emeritus of sociology and past

The Digital Libraries are a
comprehensive, cost-effective
way to deliver practical
treatments of important
business issues to every
student and faculty member.

inappropriate methods. The authors discuss how best to
­identify the information needed and the best approach to get
that information. They also highlight the processes c
­ ommonly
involved in conducting a survey including the value of both
­obtaining a representative sample and dealing with the types
of errors that can distort results. Each chapter focuses on one
of the key components of constructing and carrying out a
­survey, including the elements to consider when developing a
survey, the modes of survey delivery, writing good questions,

director of the Institute for Social Research at ­
State University, Sacramento (CSUS). He received his PhD
in c
­riminology from Florida State University. P
­rior to his
­appointment at CSUS, he served as the director of the ­Institute
for Legal and Policy Studies at the University of I­llinois for
10 years and interim director of the Institute for Public ­Affairs,
Abraham Lincoln Center for Policy Studies for two years.
He has received more than $25 million in research ­funding and
has authored numerous publications and r­ esearch reports.
Edward Nelson is professor emeritus of ; sociology at ­California
State University, Fresno. He received his PhD in sociology
from the University of California, Los Angeles, specializing in
­research methods. He was the director of the Social Research
Laboratory at CSU, Fresno from 1980 to 2013 and directed more
than 150 surveys. He taught research methods, quantitative
methods, critical thinking, and computer applications. He has
published books on observation in sociological research and
using SPSS, a statistical computing package widely used in the
social sciences.

For further information, a
free trial, or to order, contact:

Quantitative Approaches
to Decision Making Collection
Donald N. Stengel, Editor

An Introduction
to Survey

conducting the survey, and presenting the results.

Quantitative Approaches
to Decision Making Collection
Donald N. Stengel, Editor
ISBN: 978-1-60649-818-7


• Unlimited simultaneous
• Unrestricted downloading
and printing
• Perpetual access for a
one-time fee
• No platform or
maintenance fees
• Free MARC records
• No license to execute

the pitfalls stemming from bad survey construction and



Ernest L. Cowles
Edward Nelson

An Introduction to Survey

An Introduction to Survey
Ernest L. Cowles and Edward Nelson

An Introduction to Survey Research
Copyright © Business Expert Press, LLC, 2015.
All rights reserved. No part of this publication may be reproduced,
stored in a retrieval system, or transmitted in any form or by any
means—electronic, mechanical, photocopy, recording, or any other
except for brief quotations, not to exceed 400 words, without the prior
permission of the publisher.
First published in 2015 by
Business Expert Press, LLC
222 East 46th Street, New York, NY 10017
ISBN-13: 978-1-60649-818-7 (paperback)
ISBN-13: 978-1-60649-819-4 (e-book)
Business Expert Press Quantitative Approaches to Decision Making
Collection ISSN: 2163-9515 (print)
Collection ISSN: 2163-9582 (electronic)
Cover and interior design by Exeter Premedia Services Private Ltd.,
Chennai, India
First edition: 2015
10 9 8 7 6 5 4 3 2 1
Printed in the United States of America.

Ernest Cowles. First, I would like to thank my lovely wife, Ellison, ­without
whose insights, patience, and support this book would have remained on my
to do list. I would also like to take a moment to thank my friends, colleagues,
and family for their wisdom and guidance ­professionally and personally
across the years. Finally, I am deeply indebted to my coauthor, Ed Nelson, for
his perseverance, thoroughness, and hard work during the writing process.
Without his effort, this work would have still likely existed only as electrons
dancing around inside my computer.
Edward Nelson. I want to dedicate this book to my wife, Elizabeth ­Nelson,
and my children, Lisa and David, for all their support over many years.
Elizabeth and I were both in the Sociology Department at C
­ alifornia State
University, Fresno for many years and shared so much both at work and
at home with our family. It has been a pleasure to work with my coauthor,
Ernest Cowles, on this book. Both of us were directors of survey research
centers until our retirements and we have combined our years of
experience in this project.

This book is an introduction to survey research for those who want an
overview of the survey process. It is intended to describe fundamental
survey components to help both students and managers understand and
use surveys effectively and avoid the pitfalls stemming from bad survey
construction and inappropriate methods. We begin by talking about
how best to identify the information needed and the best approach
to get that information. We then discuss the processes commonly
involved in c­ onducting a survey including the value of both obtaining a
­representative sample and dealing with the types of errors that can distort
results. Next, each chapter focuses on one of the key components of constructing and ­carrying out a survey, including the elements to consider
when ­developing a survey, the modes of survey delivery, writing good
questions, ­conducting the survey, and presenting the results. Each chapter
concludes with a summary of important points contained in the chapter
and an annotated set of references indicating where readers can go for
more information on chapter topics.

ethical issues, internet surveys, interviewer training, mailed surveys,
mixed-mode surveys, sampling, survey, survey construction, telephone
surveys, web surveys

Chapter 1


Chapter 2

Probability Sampling������������������������������������������������������13

Chapter 3

Total Survey Error����������������������������������������������������������35

Chapter 4

Factors to Consider When Thinking About Surveys �������65

Chapter 5

Modes of Survey Delivery�����������������������������������������������81

Chapter 6

Writing Good Questions����������������������������������������������101

Chapter 7

Carrying Out the Survey����������������������������������������������127

Chapter 8

Presenting Survey Results����������������������������������������������145


Survey research is a widely used data collection method that involves
getting information from people typically by asking them questions
and collecting and analyzing the answers. Such data can then be used
to understand individuals’ views, attitudes, and behaviors in a variety of
areas such as political issues, quality of life at both the community and
individual levels, and satisfaction with services and products, to name but
a few. Decision makers in both the public and private sectors use survey
results to understand past efforts and guide future direction. Yet there
are many misperceptions regarding what is required to conduct a good
survey. Poorly conceived, designed, and executed surveys often ­produce
results that are meaningless, at best, and misleading or ­inaccurate, at worst.
The resultant costs in both economic and human terms are ­enormous.
Our purpose of writing this book is to provide an introduction and
overview of survey research. We begin our exploration at the foundation
of gathering information about the world—observation and questioning.
We talk about identifying the information needed and the best approach
to get that information. We discuss the processes commonly involved in
conducting a survey including both obtaining a representative sample and
dealing with the types of errors that can distort results. Next, we focus
on the components of constructing and carrying out a survey, including
the elements to consider when developing a survey, the modes of survey
delivery, writing good questions, conducting the survey, and presenting
the results.
Making use of what people tell us in surveys depends on a number
of factors. The mechanics of putting a survey together and proper s­ urvey
administration determine whether useful information is obtained or
whether the results are meaningless or even misleading. The way q­ uestions
are worded and the order in which questions are asked affect what respondents tell us. The way the researcher interacts with survey participants can
influence not only what people tell us but even whether they will respond.



Factors such as economic status, gender, race and ethnicity, and age can
also influence how people respond to questions.
This book describes fundamental survey components to help both
students and managers understand and use surveys effectively and avoid
the pitfalls stemming from bad survey construction and inappropriate
methods. Each chapter focuses on a key component and has an annotated
set of references indicating where readers can go for more information.

We want to acknowledge the work and contributions of Exeter team,
whose suggestions have made this a better work. We also want to
acknowledge the help and support of Scott Isenberg, the Executive
­Acquisitions Editor of Business Expert Press and Donald Stengel, the
­Collection Editor for a group of their books under a collection called
Quantitative Approaches for Decision Making. Don also read our
­manuscript and gave useful and valuable suggestions for the book.


Research starts with a question. Sometimes these are why questions.
Why do some people vote Democrat and others vote Republican? Why
do some people purchase health insurance and others do not? Why do
some people buy a particular product and others buy different products?
Why do some people favor same-sex marriage and others oppose it? Why
do some people go to college and others do not? Other times they are
how questions. If you are a campaign manager, how can you get people
to vote for your candidate? How could we get more people to purchase
health insurance? How could you get customers to buy your product?
How could we convince more people to go to college? But regardless,
research starts with a question.
Think about how we go about answering questions in everyday life?
Sometimes we rely on what people in authority tell us. Other times we
rely on tradition. Sometimes we use what we think is our common sense.
And still other times we rely on what our gut tells us. But another way we
try to answer questions is to use the scientific approach.
Duane Monette and his associates suggest that one of the characteristics of the scientific approach is that science relies on systematic
­observations.1 We often call these observations data and say that science
is empirical. That means it is data based. However, the scientific approach
doesn’t help you answer every question. For example, you might ask
whether there is a God or you might ask whether the death penalty is
right or wrong. These types of questions can’t be answered empirically.
But if you want to know why some people vote Democrat and others vote
Republican, the scientific method is clearly the best approach. R
­ elying on
what people in authority tell you or what tradition tells you or your gut
won’t work.



Research Design
Your research design is your plan of action. It’s how you plan to answer
your research questions. The research design consists of four main parts—
measurement, sampling, data collection, and data analysis. Measurement
is how you will measure each of the variables in your study. Sampling
refers to how you will select the cases for your study. Data collection is
how you plan to collect the information that you will need to answer
the research questions. And data analysis is how you plan to analyze the
data. You need to be careful to decide on your research design before you
collect your data.
In this book, we’re going to focus on data collection and specifically on
surveys. We’ll talk about sampling, survey error, factors to consider when
planning a survey, the different types of surveys you might use, writing
good questions, the actual carrying out of surveys, and survey reporting.

Observation and Questioning
Irwin Deutscher succinctly summarizes the different ways we collect
data—“(1) we can observe it in process; (2) we can view the records men
[and women] leave behind…; and (3) we can ask questions and listen to
answers.”2 In this chapter, we’re interested in two of these approaches—
observation and questioning.
Matilda White Riley makes the following comment about observation and questioning noting that one method isn’t inherently superior to
the other but that observation and questioning focus on different aspects
of the social setting we are studying.3
Researchers sometimes feel—mistakenly, we believe—that they
can obtain a true picture of a social phenomenon only if they
observe it with their own eyes. To be sure observation and questioning often give different results; but this occurs, not because
one method is more valid than the other, but because the two
focus … on different sets of social system properties.
Observation and questioning give us different information about
what is going on in the world. Observation gives us information about



what people do. Questioning gives us information about what people say
and the context to help interpret their observations.* This suggests that
we often need both observation and questioning to give us a complete
picture of what is happening and why it happens.
Elliot Liebow in his book, Tally’s Corner, provides a clear example of
these two different approaches to data collection.4 Liebow studied a group
of men who hung out on street corners in Washington, DC. He notes that
“men and women talk of themselves and others as cynical, self-­serving
marauders, ceaselessly exploiting one another as use objects or objects of
income.”5 The men in Liebow’s study “are eager to present themselves as
exploiters to women as well as to men.”6 In other words, this is what they
say. He goes on to say that “in practice, in their real relationships with
real women, the men frequently gave the lie to their own words.”7 This is
what the men do. So how does Liebow explain this apparent contradiction
between what men say and what they do? He suggests that there are two
opposing impulses at work. “The impulse to use women as objects of economic or sexual exploitation is deflected by countervailing impulses and
goals, especially the desire to build personal, intimate relationships based
on mutual liking and love.”8 The apparent contradiction between what the
men say and what they do is explained by the “interplay of these opposing
Let’s consider another example. You’re doing a market research survey
for a company that manufactures condoms. You want to know whether
people purchase condoms and the particular brands they buy. It’s easy to
imagine a discrepancy between what people say and what they do. Some
people might be embarrassed to give you this information and others
might feel that it’s none of your business. What people say might not
accurately reflect what they do.
Even though we see that observation and questioning give us different information about the world, we are still surprised when there is a
lack of consistency between what we learn from observation and from

* Someone might point out that we can ask people questions about their behavior. For example,
we might ask what products they bought or whether they voted in the last presidential election.
Certainly we are getting information about how people report their behavior. But we know that
people overreport how often they vote and underreport other things such as illegal or deviant
behavior. We’re still getting their subjective reports of their own behavior.



questioning. Deutscher in his book, What We Say/What We Do, describes
an early study by Richard LaPiere.10 In the early 1930s, LaPiere travelled across the United States with a Chinese couple. They ate together in
restaurants and stayed at hotels and auto camps and were refused service
only once, and this was during a time in the United States when there was
considerable prejudice toward Chinese. Six months later, LaPiere sent a
questionnaire to these same hotels and restaurants asking the following
question—“Will you accept members of the Chinese race as guests in
your establishment?”11 He describes the results of his survey as follows:
With persistence, completed replies were obtained from 128 of
the establishments we had visited; 81 restaurants and cafes and
47 hotels, auto-camps, and “Tourist Homes.” In response to the
relevant question, 92 percent of the former and 91 percent of the
latter replied “No.” The remainder replied “Uncertain, depends
upon circumstances.”12
So what are we to make of this? Is this an example of the ­inconsistency
between what people say and what they do? Or does it simply reflect
that observation and questioning are telling us different things about
the world? LaPiere’s classic study sparked a great deal of interest and
­follow-up studies. Howard Schuman in his book, Method and Meaning
in Polls and Surveys, describes a study that he and Robert Brannon carried
out in 1969.13 He refers to this as “an attitude-behavior field study.”14
In a survey, respondents were asked their opinion of open-housing laws.
Here’s the question they were asked and the percent of respondents giving
each answer.15 (DK stands for don’t know and NA for no answer.)
Suppose there is a community-wide vote on the general housing
issue. There are two possible laws to vote on.…
Which law would you vote for?

One law says that a homeowner can decide for himself who to sell his
house to or, even if he prefers not to sell to blacks.



The second law says that a homeowner cannot refuse to sell to someone because of their race or color.


DK, Neither, NA








Three months after the survey the same respondents were asked to
sign a petition. One of the petitions supported the first law and the second petition supported the other law. Those who said they would sign
the petition were then asked if they would be willing to have their name
appear in the newspaper as a supporter of that petition. Schuman summarizes the overall consistency between what people said and what they
were willing to do for those opposed to the open-housing law—“…
85 percent were consistent in signing the Owner’s Rights petition, and
78 percent were consistent in refusing to sign the Open Housing petition
which gives an overall average of 82 percent consistency.”16 The same type
of consistency was also found for those who supported open housing.
Schuman concludes that in this study “attitudes can predict behavior to a
reasonable extent, though of course not perfectly.”17
In a more recent study, Eleanor Singer and her associates studied the
“impact of privacy and confidentiality concerns on participation in the
2000 Census.”18 This is another example of what Schuman referred to
as the attitude–behavior question. Their analysis found that attitudes
toward confidentiality and privacy were significantly related to behavior
(i.e., returning the census form). It’s interesting that they also report that
other researchers found that “many more people … say they would not
provide their SSN [Social Security number] to the Census Bureau than
actually fail to provide it when it is asked for on their census form.”19
There are many more examples of the attitude–behavior issue, but
these are sufficient to show that sometimes people behave in a way that
is consistent with what they say and other times what they say is different from what they do. As Liebow pointed out, there are various factors
affecting both what people say and what they do, and it is the interplay
of these factors that eventually determines the outcome. For our purposes, it is important to keep in mind that observation and questioning
provide us with different information. Questioning tells us how people
feel and observation provides us with information about how people
behave. It’s not s­ urprising that sometimes these two types of information
are consistent with each other and other times they are not. The focus in
this book is on questioning and how we carry out surveys. We’ll cover
these topics in C
­ hapters 4 (Factors to Consider When Thinking About
­Surveys), 5 (Modes of Survey Delivery), 6 (Writing Good Questions), and
7 ­(Carrying Out the Survey). But we should never lose sight of the fact



that this is not the same thing as observing how people behave and interact
with each other.

Triangulation refers to the use of data from different sources and methods
of data collection. All data suffer from different types of error and error is
inevitable. We’ll have more to say about that in Chapter 3 on Total Survey
Error. It follows then that using data from different sources and methods
of data collection is a powerful research strategy. Eugene Webb and his
associates in their book on Nonreactive Measures in the Social Sciences put
it this way—“If a proposition can survive the onslaught of a series of
imperfect measures, with all their irrelevant error, confidence should be
placed in it.”20 In other words, if data from different sources and methods
of data collection lead to the same conclusion, then we can have more
confidence in that finding.
An informative example of triangulation is William Foote Whyte’s
work on community conflict and cooperation in Peru.21 Whyte studied
12 communities using both surveys and observation in the 1960s. In 1964
and 1969, the questions used to measure conflict were slightly different.
In 1964 the survey asked “is there much conflict or division among the
people of this village?” In 1969 the question was “is there much conflict
in this village between the people who want to change things and the
people who want to keep to the old ways?” The question used to measure
cooperation was the same in both years: “When it comes to cooperating
on some project for the community, how well do the people cooperate?”22
Whyte’s analysis of the survey data focused on change over the ­five-year
period. He found that four communities shifted from high ­conflict and
low cooperation to low conflict and low cooperation. Only one community, Huayopampa, shifted from low conflict and high cooperation
to high conflict and low cooperation. This single piece of data would
probably have been dismissed were it not for the fact that Whyte also
had observational data from graduate students who observed behavior in
these communities. The data from the observers corroborated the survey
findings. This led Whyte to reject the commonly held belief that conflict
and cooperation were different ends of a single continuum and to assert
that it was possible for communities to be high or low in both conflict



and cooperation, an important theoretical breakthrough. Whyte concludes that this “theoretical breakthrough… would have been impossible
without the combination of research methods used in our program.”23
This example of triangulation focuses on conflict and cooperation at
the community level. But it is easy to see how this could be relevant
for other studies of conflict and cooperation at the organizational level.
Imagine that you were studying hospitals or multinational business
­corporations and you wanted to study conflict and cooperation among
staff. You might employ the same strategy of using both survey and observational data to achieve triangulation.

Questioning (Interviewing) as a Social Process*
Interviewing is in some ways similar to the types of conversations we
engage in daily but in other ways it’s very different. For example, the
interviewer takes the lead in asking the questions and the respondent has
little opportunity to ask the interviewer questions. Once the respondent
has consented to be interviewed, the interviewer has more control over
the process than does the respondent. However, it is the respondent who
has control over the consent process and it is the respondent who determines if and when to terminate the interview. We’ll discuss nonresponse
in ­Chapter 3 on Total Survey Error and in Chapter 7 on Carrying Out
the Survey.
Raymond Gorden has provided a useful framework for viewing the
interview as a social process involving communication. Gordon says that
this communication process depends on three factors—“the interviewer,
the respondent, and the questions asked.”24 For example, the race and gender of the interviewer relative to that of the respondent can influence
what people tell us and we know that the wording and order of questions
can also influence what people tell us. We’ll discuss these considerations
in Chapter 3 on Total Survey Error.
Gorden goes on to suggest that the interaction of interviewer,
respondent, and questions exists within the context of the interview situation.25 For example, are we interviewing people one-on-one or in a
group ­setting? Many job interviews occur in a one-on-one setting, but
* From now on we will refer to questioning as interviewing.



one of the authors recalls a time when he was one of several job applicants
who were ­interviewed in a group setting involving other applicants. Rest
assured that this affected him and the other applicants. Are we interviewing people in their homes or in another setting? Think of what happens in
court when witnesses are questioned in a courtroom setting. That clearly
affects their comfort level and what they say.
Gorden notes that the interview and the interview situation exist
within the context of the culture, the society, and the community.26 There
may be certain topics such as religion and sexual behavior that are difficult to talk about in certain cultures. Norms of reciprocity may vary from
culture to culture. Occupational subcultures, for example, the subcultures
of computer programmers and lawyers, often have their own language.
It’s helpful to keep in mind that the interview can be viewed as a social
setting that is affected by other factors just as any social setting is influenced by many factors. In this book, we will be looking at many of the
factors that affect the interview. We’ll look at the research that has been
done and how we can use this research to better conduct our interviews.

Brief History of Surveys
Don Dillman and his associates have an excellent and brief history of surveys
in their book Internet, Mail and Mixed-Mode Surveys – The Tailored Design
Method. Dillman says that “during the first two thirds of the 20th century,
there existed only one generally accepted mode for conducting surveys:
the in-person interview.”27 Edith DeLeeuw reports that the “first scientific
face-to-face survey” was a “study of working-class conditions in five B
­ ritish
cities” in 1912 and “the first documented mail survey” was in 1788
sent to “ministers of all parishes of the Church of Scotland.”29 By 1980,
­Dillman says that the mail survey was commonly used, and by the early
1980s, the phone survey became a dominant mode. By the ­mid-1990s,
another form of surveying had emerged—the web survey.
One of the factors that influenced surveys was technological change.30
By 1970 almost 90 percent of households had a landline phone.31 By
the early 2000s, the cell phone was commonly used, particularly by
young males. Caller ID made it easier for people to screen their calls and
not take unwanted calls. E-mail and the web created a new medium of



communication. By the mid to late 2000s web surveys were common.
Today there has clearly been a shift to what are called mixed-mode surveys,
which rely on a combination of face-to-face, mail, phone, and web-based
surveys along with new technologies that have appeared such as the Interactive Voice Response survey where respondents use their t­ouch-tone
phone to record their answers and Audio-Computer-Assisted Self-­
Interviews, which are respondent-administered surveys on a computer.
During the last 50 years, there were shifts in the way people used
these technologies. Response rates to phone surveys began to drop
because of the difficulty in contacting respondents and survey refusal.
Roger ­Tourangeau and Thomas Plewes conducted an extensive review
that looked at ­nonresponse in a number of large surveys. They conclude
that “the ­experience of this illustrative set of surveys provide evidence
that ­nonresponse rates continue to increase in all types of cross-sectional
­surveys, with little to suggest that the trend has plateaued.”32 They go on
to point out that the increase in nonresponse rates for phone surveys has
been particularly large.
Two events are worth noting in the history of surveys. In 1936,
The ­Literary Digest conducted a mailed survey of potential voters in the
presidential election which pitted the Democrat incumbent President
Franklin Delano Roosevelt against the Republican Governor Alf Landon.33
They sampled phone numbers in telephone directories and names in state
automobile registration lists. Their sample was extremely large totaling
over two million potential voters. However, it did not include those without phones and automobiles and clearly did not adequately represent the
population. The Literary Digest survey predicted a victory by Landon but
Roosevelt won in a landslide. This clearly demonstrated the danger of
using lists that were biased in terms of variables such as education and
Another significant event occurred in the 1948 presidential c­ ontest
between Democrat incumbent President Harry Truman and the
­Republican Governor Tom Dewey.34 The major polls such as Gallup and
Roper predicted that Dewey would defeat Truman but Truman prevailed
in the election. This brought about changes in surveys such as the decline
of quota sampling and the practice of continuing polling right up until
­Election Day.



The history of surveys demonstrates the influence of societal changes
on surveying. The dramatic increase in landlines in households by 1970
and the rapid increase in the 2000s in the use of cell phones along with
the development of a significant segment of individuals who use only
cell phones have changed surveys drastically.35 It is likely that this will
­continue in the future. Address-based sampling is another development
that is becoming common where residential addresses are sampled from
the U.S. Postal Service’s Computerized Data Sequence File and made
available through third-party providers.

The Rest of the Book
Here’s a brief summary of what we will cover in the rest of the book.
• Chapter 2—Sampling—What are samples and why are they
used? In this chapter, we discuss why we use sampling in survey research, and why probability sampling is so important.
Common types of samples are discussed along with information on choosing the correct sample size.
• Chapter 3—Total Survey Error—Error is inevitable in every
scientific study. We discuss the four types of survey error—
sampling, coverage, nonresponse, and measurement error
focusing on how we can best minimize it.
• Chapter 4—Factors to Consider When Thinking About Surveys—In this chapter some of the fundamental considerations
about surveys are presented: the stakeholders and their roles in
the survey process, ethical issues that impact surveys, factors
that determine the scope of the survey, and how the scope, in
turn, impacts the time, effort, and cost of doing a survey.
• Chapter 5—Modes of Survey Delivery—There are four basic
modes of survey delivery—face-to-face, mailed, telephone,
and web delivery. We focus on the critical differences among
these different modes of delivery and the relative advantages
and disadvantages of each. We also discuss mixed-mode surveys, which combine two or more of these delivery modes.


• Chapter 6—Writing Good Questions—Here we look at
survey questions from the perspective of the researchers and
the survey participants. We focus on the fundamentals of the
design, formatting, and wording of open- and closed-ended
questions, and discuss some of the most commonly used
formats used in survey instruments.
• Chapter 7—Carrying Out the Survey—Every survey goes
through different stages including developing the survey,
pretesting the survey, administering the survey, processing and
analyzing the data, and reporting the results. Surveys administered by an interviewer must also pay particular attention to
interviewer training.
• Chapter 8—Presenting Survey Results—This final chapter
talks about the last step in the survey process—presenting
the survey findings. Three major areas, the audience, content,
and expression (how we present the survey), which shape the
style and format of the presentation, are each discussed along
with their importance in the creation of the presentation.
The chapter concludes with a discussion on how to structure
different types of presentations such as reports, executive
summaries, and PowerPoints, and how to effectively present
survey data and results.

Annotated Bibliography
Research Design
• Matilda White Riley, Sociological Research I: A Case Approach,
is an early but excellent discussion of research design.36 Her
paradigm of the 12 decisions that must be made in constructing a research design includes the alternative methods of
collecting data—observation, questioning, and the combined
use of observation and questioning.
• Earl Babbie, The Practice of Social Research, is a more recent
introduction to the process of constructing a research




• Delbert Miller and Neil Salkind, Handbook of Research Design
& Social Measurement, provides many examples of the components of the research design.38
Attitudes and Behavior
• Irwin Deutscher, What We Say/What We Do, is an excellent
discussion of the age-old question of the consistency between
what people say and what people do.39
• Richard LaPiere, Attitudes vs. Actions, is the classic example of
the inconsistency between attitudes and behavior.40
• William Foote Whyte is the author of Street Corner Society,
which is one of the classic examples of participant observation
that relies not only on observation but also on questioning.41
His study of 12 rural communities in Peru is an excellent
example of how observation and questioning can be combined in the research design.42
Questioning (Interviewing) as a Social Process
• Raymond Gorden, Interviewing: Strategy, Techniques, and
­Tactics, is one of the clearest discussions of the communication process and the factors that affect this process.43
Archival or Available Data
• There are different sources of data—observation, questioning,
and archival or available data. Although archival or available
data is not something that we’re going to discuss in this book,
if you want to learn more about it, the classic introduction is found in Eugene Webb and his associates’ book on
­Nonreactive Measures in the Social Sciences.44


Probability Sampling
Did you ever go into a coffee shop and discover they were giving away
samples of a new blend they were serving? Or how about going to a grocery store and finding a person in one of the aisles distributing hot samples
of a new pizza bread snack in the frozen food section? Beyond the obvious purpose of enticing you to buy the drink or food being offered, you
probably never gave much thought to the small portion of the drink or
food you were offered. Whether you would consider giving the merchant
your hard-earned money for a full cup or a large package of pizza bread
depends not only on your food preferences and whether you liked the
taste, but also on an underlying assumption that you make regarding the
sample you were given—namely that it is representative of the gallons of
that particular coffee blend that is brewed, or that the large boxes of pizza
bread snacks found in the freezer case are the same as the bite size piece
you were offered. Imagine the disaster for both merchants and customers
if it were not. This little example illustrates the important elements that
pertain to the kind of sampling done in conjunction with surveys.
If done properly, a sample can adequately replace the need to examine
each item or individual in a population (sometimes referred to as the population or universe) in order to determine if a particular characteristic is
present. For example, suppose you want to find out if students who attend
colleges that are affiliated with particular religious faiths have different
attitudes toward the legalization of marijuana than students who attend
colleges that have no religious affiliation.1 It’s obvious that if you planned
to answer your question by interviewing or providing a written survey to
all the students in the country who attended both religiously affiliated and
nonaffiliated colleges, it would be impossible due to the cost and time
required (along with a host of other practical considerations). Instead, by
carefully selecting a manageable sample that actually r­epresents students
from both types of educational institutions, it is possible to answer the



question. The key is in getting a sample of individuals that accurately represents all of the individuals in the larger population that you’re interested
in studying. Now here’s a little secret: the only way we can be absolutely
certain that we have determined the true picture of the attribute that we
are interested in studying, in this case attitudes toward marijuana legalization, is to ask each individual in the population about her or his attitudes
toward legalization, then count up, and report the responses that are given;
a process called enumeration. If we could actually pull this off, we might
find, for example, that 86 percent of students at nonreligiously affiliated
colleges favored legalization, but only 63 percent of students at religiously
affiliated schools were in favor of legalization. We could be very certain
that our findings were actually correct, and it really wouldn’t take any
more complicated math than addition and division to get our percentages.
But that creates a dilemma—we’ve already said that it is not possible to
get information from each person for a variety of reasons. As a result, we
are forced to draw a sample and be content knowing that there is some
probability that our sample will miss the mark in terms of accurately representing the population. This fact has been the force driving two major
fields of study regarding research, including survey research. One of these
has concerned itself with methodology of sampling and problems due to
the way the sample was selected or drawn—which we term sampling error.
The second area, also part of the larger field of statistics, has focused on
determining mathematical probabilities of error and developing mathematical formulas to estimate true values with a variety of data.
In the next chapter we discuss sampling error and some ways to deal
with it, but for now, we’ll simply look at different kinds of sampling
approaches and see that how a sample is selected can make a big difference
in how much confidence we can have that it actually represents the larger
population we want to study.
A few years ago, a large state mental health agency approached one
of us about improving the quality of a survey that it administered annually to individuals who were receiving community mental health services
throughout the state. The survey was designed to capture the kinds of
services used at community mental health facilities and clients’ satisfaction with those services. After reviewing the data and the process used to
administer the survey, a number of problems came to light. One of the



most serious of these was the manner of selecting clients to participate in
the survey. The selection of participants and administration of the survey
involved staff members handing the survey to every individual who came
into a community mental health facility during one specified two-week
period during the year. Can you see any problems with selecting a sample
in this way? If your answer included the idea that samples selected this
way might not be truly representative of the population using community
mental health services, you are right on target. This sample is essentially
what is known as a convenience sample, and it is a nonprobability sample because it relies on individuals who happen to be available to take
the survey. Essentially, individuals who did not pay a visit during the
two-week sample period had no chance of being included in the sample.
Further, while the agency knew the total number of individuals receiving
services over the course of a year, they had no way of determining how
many would be in the sample beforehand. This is important because as
we’ll discuss later in the chapter, the size of the sample affects its ability to
accurately represent the population from which it is drawn.

What Is a Probability Sample?
The basic idea of survey sampling emerged in Europe at the beginning
of the last century,2 but a major theoretical underpinning for probability
sampling seen today is commonly attributed to a Polish born mathematician and statistician who presented a paper entitled “On the Two Different Aspects of the Representative Method: The Method of Stratified
Sampling and the Method of Purposive Selection,” at the Royal Statistical
Society in June 1934.3 Neyman’s influential work would help create an
acceptance of probability sampling and shape the basic construction of
probability samples that has occurred since. During the ensuing eight
decades since his paper, the basic structure of probability sampling has
been further developed and refined, but at its core are three assumptions:
(1) that a frame of all units in the population can be created (this is called
the sampling frame), (2) that a person or thing in the frame has a (positive)
likelihood of being selected into the sample, and (3) that the likelihood
or probability of being selected can be computed for each sampled unit.
The advantage of probability sampling over nonprobability methods is not



that it eliminates the possibility of picking a sample of individuals or units
that doesn’t accurately reflect the population, but that it does allow us to
estimate the amount of error that may be present in our findings. In other
words, we can determine the probability or odds that we have accurately
represented the population about which we’re trying to draw inferences.
In more technical language, we can gauge the precision of our sample estimates. Thus, when you read that a survey found 46 percent of the public
approve of the job the president of the United States is doing and the
survey had ±3 point margin of error, you can assume (all else being equal)
that the actual percentage of public approval has a known probability of
being within 43 and 49 percent. We cannot make this type of statement
with nonprobability sampling methods because we lack the fundamental
ability to calculate the probability that each population element will be
chosen, and in fact, we cannot be sure that each individual or thing has
a possibility of being selected. Stated in more technical language, with
probability sampling we say that each element of the population element
has a nonzero probability of being chosen. Thus, while a nonprobability
sample may accurately mirror the population from which it is drawn, we
don’t know whether it does or not.
Returning to the example of the mental health clients’ survey, clients
who had no visits during the two-week sample period had no chance
(a zero probability) of being in the sample. On the other hand, because
the survey’s selection criterion was a time interval, individuals who visited
community mental health centers many times during the year would have
a greater likelihood or greater probability of being selected to take the survey. Making this selection approach even more troublesome was the fact
that clients were asked to complete a survey each time they came to the
community mental health center—thus a single client could have multiple
surveys included in the sample. In fact, one individual had completed the
survey 16 times during the two-week sample period! The possibility that
more frequent visits to the mental health centers might be associated with
more chronic or severe mental health issues highlights the possible problems of this sampling approach—the greater likelihood these individuals
would be completing surveys might present a different picture of services
than the larger population utilizing community mental health services.
The samples that have the greatest likelihood of being representative
are those in which the units (in this case people) have an equal probability



of being selected. We term these, equal probability of selection methods
Simple Random Samples
There are many ways to obtain samples in which each person in a population has an equal chance of being selected for a sample; the most
straightforward of these is called the simple random sample. Simple random samples work when every individual or thing in the population that
is to be sampled can be identified. For example, while possible, can you
imagine the work you would have to do to compile a complete list with
contact information of everyone who had purchased a new Toyota in the
past five years? However, if you had such a listing you would be able to use
simple random sampling in the same way that you would use it to select
a sample of students from three introductory business classes. Once you
have a listing of every element (people or things) in the population, you
simply need a way to select the number of them you need for your sample
in a truly random fashion. The key here is ensuring the randomness of
the selection, which gives each element an equal chance or probability of
being selected.
A simple random sample method with which most people in this
country are familiar (although it is doubtful they think about it in that
way) is the ubiquitous state or multistate lottery where five or six ping
pong balls are (hypothetically) randomly drawn from a universe of around
50 numbers that are represented by numbered ping pong balls, which
are randomly mixed in some type of rotating cage or swirling air container. Each numbered ball should have an equal chance of being selected
from the hopper on a given draw except, of course, those already selected,
which would entail a different kind of sampling.* The objective is to correctly choose the five or six numbers that are represented on the ping
pong balls before they are drawn, which results in winning an enormous
jackpot usually worth millions of dollars. Individuals paying to play the
* The typical sampling done in lotteries is termed sampling without replacement because once
a numbered ball is selected it is taken from the pool so it cannot be selected again. If it were
returned to the pool of balls, the sampling approach would be termed with replacement. The odds
of correctly choosing the entire sequence of five or six balls needed to win the lottery changes
dramatically under the two methods.



lotto hope to preselect the numbers drawn and frequently invent mysterious and unfathomable methods to choose the right numbers. In reality,
because the drawing represents a simple random sample of all the ping
pong balls in the lotto universe, they could simply pick any sequence of
numbers, say one through five, and they would have just as good a chance
of winning as using some supposed system of getting their lucky numbers.
Unfortunately, not all probability sampling methods are as easily done
as drawing a simple random sample of ping pong balls in a lottery, or
pulling random numbers on strips of paper from a hat. After the introduction and widespread acceptance of probability sampling, but before
researchers had the power of today’s computers and statistical software at
their fingertips, statisticians and researchers devoted considerable energy
to finding consistently reliable ways to randomly select samples. One of
the centerpieces of most of the methods was the random number table,
which dates back to the early 1900s.4 When used in conjunction with
beginning and ending points, a table of random numbers allows researchers to select enough random numbers to draw a desired sample. In a book
chapter published in the early 1980s, Seymour Sudman reviewed the
development of random numbers in sampling. In his discussion, he talks
about the lengths undertaken to develop massive random number tables.
The most convenient and most accurate procedure for obtaining
a random process is through the use of tables of random numbers.
The largest printed table is A Million Random Digits by the Rand
Corporation (1955)…. The Rand random digits were generated
by an electronic roulette wheel. A random-frequency pulse source
passed through a five-binary counter and was then converted to
a decimal number. The process continued for 2 months and even
with careful tuning of the equipment the numbers produced at
the end of the period began to show signs of nonrandomness
­indicating that the machine was running down.5
With the introduction of random number tables, the idea of randomly selecting numbers became much easier, however, particularly
when needing larger samples, the process could become cumbersome and
tedious. A variant of the simple random sample is the systematic sample,



which generally maintains the randomness of the selection process, but
is easier to use when a very large sample is needed and is often easier to
execute without mistakes when manually completed.
Systematic Sampling
Like simple random sampling, systematic sampling starts with a listing that
is an identification of the units making up the population. The systematic
approach usually starts by first determining the size of the sample needed.
For illustration, let’s say a researcher is conducting a survey and will require
200 individuals in the sample selected from a population of 2,000 individuals. A calculation is then done by dividing the sample size into the population to determine the interval between individuals to be drawn from the
population to complete the sample. Every nth case is then selected, which, in
the present example, would be every 10th individual. To inject randomness
into the process, the key to the systematic selection is to randomly select a
starting point number. Again, random number tables become a handy tool
to find this starting point. So if we selected 7 as our starting point using the
random number tables, we would draw the 7th individual on the list, the
17th, 27th, 37th, and so forth. In some cases, the systematic selection even
allows us to select the cases without numbering them. For example, suppose a medical facility wanted a survey of patients who had been diagnosed
with cancer in the past two years.6 The medical facility is interested in this
because it had installed a new imaging scanner a year earlier and it wanted
to determine patients’ perceptions of the diagnostic process before and after
the installation of the new equipment. If the medical researchers used a
systematic sample with the same parameters previously mentioned, that is,
2,000 total cases and a sample of 200 patients, they could simply identify
the starting point, say the 7th patient in the case records, then select every
10th patient for the survey. Using the case records, the researcher could pull
up the case file of every 10th patient after the 7th, and contact that individual about participating in the survey. However, a note of caution should
be voiced here about the systematic sample selection process. If the dataset for the population is already sorted by some characteristics (sometimes
unknown to the researcher), it can seriously bias the sample and wreak
havoc on the study. Suppose in our example that the medical case records



had not been arranged alphabetically by the patients’ last names but by the
date of their diagnosis (this would be highly correlated with whether they
were diagnosed with the old equipment or the new imaging scanner). Can
you see a selection problem here? The influence of the present filing system
on type of cancer would bias the sample and have serious implications for
a survey dealing with diagnostic procedures.
Before moving on to more complex ways of selecting samples, a couple of points are worth noting. First, while some statistics and survey
books still contain random number tables, today’s students and researchers have a much easier time using random numbers to select samples
because many of today’s statistical analysis programs incorporate random
number generators (subroutines) capable of generating random numbers
for simple random and systematic sampling and even more sophisticated
sampling designs. However, random number tables are still useful tools
and a number of statistics and methodology texts continue to include
them and provide instructions on their use.7
Second, despite the power of the computer to process samples, a number of problems may still make it impractical or impossible to carry out the
process needed for the simple random or systematic selection of individuals
from a population. Fortunately, statisticians and survey methodologists have
developed a number of sampling designs that allow us to overcome many
of these problems and still maintain the integrity of the probability sample.
In the following section, we briefly review some of these approaches.
Stratified Sampling
Let’s suppose that a large software company based in California with an
extensive internship program is interested in how interns of various racial
and ethnic backgrounds perceive the value of their internship with the
company, such as whether interns believe they have been treated well, and
whether they would be interested in seeking full-time employment with
the company based on their internship experience. There are 3,060 interns
currently with the company, and the company’s human resources department estimates that it will have time and resources to conduct interviews
with about 306 of them. The breakdown of the race and e­ thnicity of the
interns is as follows:




1,200 Caucasians (39.2 percent)
660 Chinese/Southeast Asian (21.6 percent)
540 East Indian (17.6 percent)
240 Latino/Latina or other Hispanic (7.8 percent)
180 African/African American (5.9 percent)
120 Middle Eastern (3.9 percent)
120 Other ethnic groups (3.9 percent)

If we assume that the individuals of various racial and ethnic
­backgrounds would appear in a random sample in the same p
­ roportions as
they appear in the population, then we would expect to see the f­ ollowing
racial and ethnic distribution in our sample of 306 interns:

120 Caucasians (39.2 percent)
66 Chinese (21.6 percent)
54 East Indian (17.6 percent)
24 Latino/Latina or other Hispanic (7.8 percent)
18 African/African American (5.9 percent)
12 Middle Eastern (3.9 percent)
12 Other ethnic groups (3.9 percent)

However, because of the small proportion of certain racial and ethnic backgrounds, it is quite possible that some of the groups might have
very few or no interns selected in a simple random sample due to sampling error. This is particularly problematic in our illustration because
the researchers are primarily concerned with perceptions by the different racial and ethnic groups. To overcome this problem, we can use
a technique termed stratified sampling, which works particularly well
when we have subgroups within the population that are of very different
sizes or small proportions of the population. By dividing the population
into homogenous groups or layers called strata, then sampling within
those strata, we reduce sampling error. In this example we would have
seven strata or groups. Once we have the stratum identified, we can then
use simple random sampling to select individuals within each stratum.
There are actually two types of stratified samples, proportional and
­disproportional. In proportional stratified random sampling, the size



of each stratum is proportionate to the population size of the strata.
This means that each stratum has the same sampling fraction. In our
illustration, there are 180 African American interns and 120 M
­ iddle
­Eastern interns, which are 6 and 4 percent of the total number of
interns, ­respectively, so if our stratified sample is proportional, we would
­randomly select 18 interns from the 180 African American intern group
and 12 interns from the Middle Eastern intern group. On the other hand,
if we use a disproportionate stratified sampling method, the number of
individuals from each stratum is not proportional to their representation
in the total population. Population elements are not given an equal chance
to be included in the sample (recall the previous EPSEM ­discussion).
Therefore, while this allows us to build up or oversample the individual
numbers in each stratum, which otherwise would have low numbers of
individuals, it creates a problem if we’re trying to generalize back to a population. Suppose in our example we sample disproportionately so that we
have approximately 44 interns in each sample. In that case, the responses
given by Latino/Latina/Hispanic, African American, Middle Eastern, and
our Other category of interns would be overrepresented in the responses,
while the responses of ­Caucasians and Chinese/Southeast Asian and East
Indian interns would be underrepresented. To compensate for this, we
would need to weight our stratum responses back to the proportions of
the strata seen in the populations.
In our example here, we would likely be interested in comparing
our strata or conducting what is termed a between-stratum analysis.*
This would permit us to compare responses on the survey interviews from
each of our seven strata against one another.
Cluster Sampling
Another form of sampling that also uses grouping of individuals in the
process is called cluster sampling. Because both stratified sampling and
cluster sampling use groups in their process, they are frequently confused.
* To do so, we would use a balanced allocation (also termed as factorial allocation), so we would
select strata with an equal number of interns in each. Since we have limited our study to 306
individuals and we have seven strata, we would disproportionately sample so we had 44 interns
in each stratum.



Recall that a stratified sample begins by placing individuals into groups
based on some characteristic such as race and ethnicity, marital status,
religious preference, and so forth. In cluster sampling, we begin by first
randomly selecting a sample of some naturally occurring or known grouping. For example, we might create a cluster sample by randomly selecting
a group of outlet malls. We then take all units from each of our randomly
selected clusters for our sample. Thus, we might select all the stores from
our randomly selected group of outlet malls. This approach is particularly
useful when there is no satisfactory list of individuals or things in a larger
population that we want to study, and no way to get at the population
directly making it impossible to draw a simple random sample. To illustrate the process, let’s consider using this approach to solve the problem
of an inability to get a listing of individuals in a population. Suppose the
National Collegiate Athletic Associate (NCAA), responding to growing
concern with the rising problem of its athletes getting concussions while
playing, decides to survey NCAA school athletes. The NCAA thinks that
a survey of players would be good to determine how aware players were
of the problem of concussions, if they had ever suffered a concussion, and
if they had suffered any longer-term effect from a competition-related
head injury. Because of the large number of college sports and players,
the NCAA decides to start by first conducting surveys of athletes in two
sports with higher probabilities of concussions: football and soccer. It
contracts with a university to design and conduct two surveys, one for
each sport. The league tells the university that it is very difficult to get a
single listing of all current players across NCAA football and soccer players from which to pull a simple random sample. This is not an uncommon problem even with well-defined populations such as college sports
teams; so can you imagine then the struggle to identify all the members
of a less well-defined group such as aerospace workers or the residents of
a particular country! Because of this problem, the researchers decide to
use cluster sampling. Just as with stratified sampling, every member of the
population can be a member of one, and only one, group or cluster—in
this case one NCAA college or university. The first step is to identify
known or accessible clusters, so in our example, the researchers will start
by listing NCAA schools (because they are identifiable) across the country, then using a random selection method, they will choose a certain



number of schools that are the clusters from which individual athletes will
be drawn. Basically, the researchers would ask for the team rosters from
each of the selected schools for each of the two sports in order to produce
its final two samples of athletes who will receive a survey.
We can extend the basic ideas of clustering, stratification, and r­ andom
selection to create much more complex designs to deal with specific
issues that might present themselves in sampling. Such designs are commonly referred to as multistage sampling. With multistage sampling, we
select a sample by using combinations of different sampling methods.
For ­example, because of the large number of student athletes on NCAA
­college football and soccer teams, the researchers may decide that it’s too
costly to send a survey to every athlete at schools in the cluster samples.
They might then propose a two-stage sampling process. In Stage 1, for
example, they might use cluster sampling to choose clusters from the
NCAA college and university population. Then, in Stage 2, they might
use simple random sampling to select a subset of students from the rosters
of each of the chosen cluster schools for the final sample.
As long as every individual (in this case players) can be attached to one
of the groups, this sampling approach works very well.8 As you can see, it
is quite easy to include additional clustering criteria in the design.

How Do We Select the Right Sample Size?
As Arlene Fink points out, “The ideal sample is a miniature version of
the population.”9 For those pursuing this ideal, there is always a tradeoff
in considering sample sizes—what is optimal and what is practical? In
sampling methodology terms, the researcher must decide how much sampling error he or she is willing to tolerate, balanced against budget, time,
and effort constraints.
When talking about sample sizes, it is important to keep in mind that
sample size is based on the number of individuals who respond to the survey, not the number of individuals who initially receive the survey. Thus,
the response rate needs to be taken into consideration when the sample
is being selected. For example, if you have a one in 15 response rate, it
means that for a sample of 300 individuals you need to start with an initial sample group of 4,500. If your response rate is one in 20, your initial



sample would need to be 6,000 individuals, so you can see how changing
the response rate by only a small amount can have a fairly large impact on
the number of individuals you need in your initial sample.
Before continuing to explore sample size determination, let’s revisit
some fundamental ideas about drawing inferences about a population
from a sample. Recall that the only way we can be absolutely certain we
have accurately captured a population attribute is to determine the value
of the characteristic from every individual or thing in the population.
So if we want to determine the average age of an alumni group and be
100 percent certain we are correct, we would need to ask each person’s
age, then sum up the responses, and divide by the number of individuals,
which would give us an arithmetic average (mean). For sake of illustration, let’s say that we have 15,000 alumni who range in age from 19 to
73 years and the average age (mean) of the population is 34 years. But,
as is the case with most surveys in the real world, we have neither time
nor resources to get the information from 15,000 alumni, so we’ll draw a
sample and use the average age of the sample to estimate the average age
of the population. For a little drama, suppose we only ask one person’s
age, and that person’s age is 26 years. This would be a point estimate of
our population average age. Simply put, if we use that one person sample
to estimate the average population of the entire alumni group we will be
wrong by quite a bit! In fact, if we repeated our rather absurd sampling
of one individual many times over, the only time our sample estimate of
the population parameter would be correct is when our sample picked an
individual who was exactly 34 years old. This would be a fairly rare event
given there is likely to be a very small percent of the alumni population
exactly 34 years old. So our sampling error will be very high. Now, suppose that we selected five alumni and determined their average age. You
would still have a high likelihood of being off (i.e., have a lot of sampling
error) from the population’s true mean of 34, but the chances of getting
an accurate estimate would slightly improve over the approach where you
only asked one person. This is due to the number of different combinations of ages among five individuals. In other words, there is less sampling
error with large samples. Stated another way, our sample of five individuals more closely resembles the distribution of age in the population than a
sample of one. By contrast, if any of the following three different samples



of five individuals were pulled from the population, each would produce
an accurate estimate of the true average age of the alumni population:
Group I: 20, 30, 30, 40, and 50
Group 2: 25, 28, 28, 32, and 57
Group 3: 30, 32, 34, 36, and 38
You may have also noticed that if you had pulled the same 15 individuals as repeated single person samples, you would have seen an accurate
estimation of the population’s average age only once while 14 of your
samples would have yielded an incorrect estimate. You can see how as
you increase your sample size, you increase the probability of accurately
mirroring or representing the larger population.
Confidence Intervals and Confidence Levels
If you went to a statistician to ask for help in determining the best
sample size for a study you had planned, chances are the statistician
would ask you several questions relative to sampling error and the precision of the sampling estimates you would like to have. The two primary
questions in this regard would be “What confidence interval would you
like to use?” and “What confidence level are you comfortable with?”
A confidence ­interval is the range of values around the true population
value within which the estimates from our samples are expected to fall
a specific percentage of the time. It is also commonly called the margin
of error. That is why when survey results are reported, they are usually accompanied by a disclaimer about the margin of error (recall the
example of the president’s approval earlier in the chapter). The statistician would also ask about the confidence level to describe uncertainty
associated with the interval estimate. For example, the confidence level
might be the 95 percent confidence level. This means that if we used the
same sampling method we could potentially create an infinite number
of different samples and compute an interval estimate for each sample. We would expect the true population parameter to fall within the
interval estimates 95 percent of the time. In survey research, confidence
levels are commonly set at the 90th, 95th, or 99th percentile. In the first
case, this would mean that you can expect your sample to contain the



true mean value, such as average age, within your confidence interval
90 ­percent of time. Conversely, 10 ­percent of the time your sample
interval estimates would not contain the true mean. With the 95th percentile, you would reduce the probability of being wrong to 5 percent,
and at the 99th you would reduce it even further to 1 percent.* The
effect of increasing the confidence level is that the c­ onfidence interval
becomes wider if the sample size stays constant. One way to counteract
that is to increase the size of your sample. While the actual calculations for establishing confidence intervals are outside the scope of the
discussion here, many good introductory statistics books will take you
through the process. There are also a number of sampling calculators
available online, which, once you provide the size of the population,
confidence level, and confidence interval parameters, will give the sample size you will need for your survey.† These sample size calculators also
will let you see how adjusting the confidence interval and confidence
level affects your sample size.
One common mistake that people who are not familiar with survey
sampling make is assuming that the size of samples must be proportional
to the size of the populations. That is, while small samples work well
for small populations, very large samples must be obtained for very large
populations. However, the following illustrates how as the population
grows, we can still obtain reliable estimates with samples that are modestly bigger. Let’s say we are conducting a survey to determine whether
voters approve of the president’s performance. For sake of simplicity, let’s
say we start with the assumption that half will approve and half will not
(which would result in the largest sample size because this is the maximum variation possible). If we use a standard 5 percent margin of error
and a confidence level of 95 percent (both of which are commonly used),
our sample sizes would vary across different population sizes as follows:
* You may have heard the term significance level when we are reporting results of statistical
tests for differences between individuals or groups. The significant level is set by the confidence
level, so when a researcher reports that there were significant differences between the average
ages of the two groups at the p ≤ 0.05 level, what the person is really saying is that the apparent
differences in average age between the groups could be due to chance 5 percent of the time or
See for example, online sample size calculators provided by Creative Research Systems at, or the one offered by Raosoft Inc. at http://www.



Size of population

Size of sample needed













With a confidence interval of ±5 percent at the 95 percent confidence level.

As you can see, initially our sample size will need to increase fairly significantly as the population gets bigger, however, after a point, the increase
in the sample size essentially flattens out, as the population increases by a
factor of 10 and finally by a factor of 100.
This is the beauty of probability sampling; if done correctly, we can
use rather small representative groups or samples to accurately estimate
characteristics of large populations.
Before turning our attention to survey error in the next chapter, a final
note on probability sampling is in order. As you may have noted, we have
not explored nonprobability sampling methods in this chapter. We chose
to omit this because of space limitations, the unevenness of some of the
emerging approaches, and the fact that probability sampling has been a
tested framework since its inception in the 1930s, chiefly because of its
power in generalizing back to a larger population. Recently, however, there
has been increased concern about some of the problems facing probability-based sampling designs as rapidly evolving technology alters the landscape of communication essential for conducting surveys. Landline phones
are disappearing as cell-phone-only households become commonplace and
electronic communication such as e-mail and text messaging is substituted
for hard copy. New messaging platforms that integrate rapid transmission
of written material, video and still images, and audio are emerging on an
almost daily basis. Recent development of address-based sampling frames
has allowed researchers the ability to use probability sampling of addresses
from a database with near universal coverage of residential homes. The web
has become an easily accessible and inexpensive tool for survey delivery,
even though a large number of web applications use nonprobability sampling methods, such as survey panels, and therefore are suspect in terms



of generalizing back to a larger population of interest. With these new
technologies come sampling problems that affect the reliability and validity of probability sampling methods when they are simply layered over
designs created around different data collection methods. The creation
of new platforms for survey delivery requires an examination of alternative approaches.10 Recently the American Association of Public Opinion
Research (AAPOR), one of the most respected professional organizations
in this area, commissioned a task force to “examine the conditions under
which various survey designs that do not use probability samples might
still be useful for making inferences to a larger population.”11
The massive amount of data collected through Internet enterprise has
even offered what some see as the elimination of the need to do surveys
at all. The ability to collect, store, and analyze so called Big Data clearly
offers opportunities to look at the relationships between variables (topics
of interest) on a scale of populations rather than samples. In so doing,
many of the concerns about sampling and sampling error presumably fall
away. The use of Big Data also shifts much of the focus in analytic process
away from concentrated statistical efforts after data collection is complete to approaches centered around collecting, organizing, and mining
of information, “… the fundamental challenge in every Big Data analysis
project: collecting the data and setting it up for analysis. The analysis step
itself is easy; pre-analysis is the tricky part.”12 However, critics such as
Tim Hartford point out that in the rush to use, and sometimes sell, big
data approaches, proponents sometimes present “optimistic oversimplifications.”13 Hartford is particularly critical of the notion that theory-free
data correlations obtained from Big Data can tell us what we need to
know without the need to examine causality further and that those large
data sets somehow remove all the analytic pitfalls that are seen in smaller
data sets.
Clearly the changing landscape of electronic data collection will impact
sampling methodologies used in surveys and create new approaches to get
information about a population. Online surveys, opt-in panels, and the
use of Big Data techniques serve as examples of the drive to capitalize
on electronic data collection and storage capabilities offered through the
Internet. However, the technology alone does not automatically improve
our ability to make valid and reliable inferences about a larger population.



Note: In this summary we use the term individuals broadly to refer to
the units or elements, which can be people or other individual entities,
making up the population from which we draw samples.
• To collect information from a target population, well-done
sampling can replace collecting information from each
individual in that population (a process called enumeration).
Sampling is more practical because it reduces time, effort, and
costs needed to gather information.
• There are two basic types of sampling—probability and nonprobability.
°° Nonprobability may be used in some cases, but it has a
major limitation; it doesn’t allow us to judge how well our
sample reflects the larger population about which we are
trying to draw inferences. Essentially, we cannot determine
sampling error with nonprobability samples.
°° While probability sampling doesn’t eliminate the possibility
of picking a sample of individuals that doesn’t accurately
reflect the larger population, it does allow us to calculate
how much error might be present.
• Probability samples have three fundamental elements:
°° A group or frame of all individuals in the population can be
created. This is termed the sampling frame.
°° Each individual in the population has a positive chance of
being selected into the sample.
°° The probability of an individual being selected can be computed for each individual in the population.
• Simple random sampling is a basic type of probability sampling, which uses techniques to randomly choose individuals
from the population for the sample. Each individual in the
larger population has an equal chance of being selected for the
sample. We call the methods that have this characteristic as
Equal Probability of Selection Methods (EPSEM).


• Systematic samples are also EPSEM samples and are very similar to simple random samples, but differ in their approach of
selecting the sample. Systematic samples use a process where
the sample is formed by dividing the number of individuals
needed for the sample into the number of individuals in the
population to determine an interval between individuals for
selection purposes. A random number located within the first
interval is selected as a starting point, and then every subsequent nth individual is selected until the number of individuals needed for the sample is reached.
• Stratified sampling is a more complex sampling strategy that
works well when there are subgroups within the population
that are of very different sizes or are very small proportions
of the larger population. Strata are formed by dividing the
population into homogenous groups or layers, and then sampling is done within those strata. This reduces sampling error.
We might, for example, create strata based on the racial and
ethnic backgrounds of interns in a large company in order
to ensure that certain racial/ethnic subgroups are not missed
because they make up such a very small part of the intern
• Cluster sampling is another form of more complex sampling,
which also uses a grouping technique like stratified sampling.
With cluster sampling, however, the groups are formed by
using some population (known or naturally occurring) characteristic like high schools or organizations such as businesses.
• Multistage sampling extends the basic ideas of stratification, clustering, and random selection to create much more
complex designs to deal with specific issues that might
present themselves in sampling. When engaging in multistage
sampling, we simply break our sampling design down into
separate stages, sequencing one method after another.
• Selecting the right size for a sample is a bit of an art and a bit
of a science. There is always a tradeoff in considering sample
sizes—what is optimal and what is practical.




°° It is important to keep in mind that the size of the sample
refers to the number of individuals that respond to the
survey, not the number who initially receive the survey.
°° Response rate refers to the proportion of individuals who
respond out of the number of individuals that are initially
selected and asked to participate. For example, a one in 15
response rate means that one out of every 15 individuals
asked to participate actually completed a survey.
°° Confidence intervals, also called the margin of error, refer
to the range of values around the true population value
within which our samples are expected to fall a specific
percentage of the time.
◊ Confidence levels reflect the amount of time that we can
expect the values (estimates) derived from our samples
to be in error. In social science research, confidence levels are typically set at the 90th (10 percent error), 95th
(5 percent error), or 99th (1 percent error) percentiles.
◊ Increasing the confidence level without increasing the
sample size widens the confidence interval.
• One of the common mistakes that people not familiar with
surveys make is assuming that the size of samples must be
proportional to the size of the population. In reality, after
population sizes reach a certain level, the sample size only
needs to increase a small amount even if the population
increases by magnitudes of 10 or 100 or more.
• This is the beauty of probability sampling; if done correctly,
we can use rather small representative groups to accurately
estimate characteristics of large populations.

Annotated Bibliography
• There are a number of good books and articles on research
sampling, from very technical presentations to general introductory discussions.
• A good general introduction to sampling can be found in Earl
Babbie’s classic The Practice of Social Research.14


• The Handbook of Survey Research edited by Peter H. Rossi,
James D. Wright, and Andy B. Anderson provides a comprehensive look at survey methodology and data. Chapter 2 on
Sampling Theory by Martin Frankel15 provides the underlying
foundation for survey sampling and Chapter 5 by Seymour
Sudman16 covers the theory of sampling and different sampling approaches.
• If you haven’t had statistics or are new to the idea of sampling,
some of the self-help websites such as Stat Trek provide overviews to survey methodology on topics such as sampling.17
• For insight into more advanced sampling techniques such
as proportionate stratified sampling and more complex disproportionate stratification methods such as disproportionate
optimum allocation, see Daniel’s Essentials of Sampling.18
• Our Chapter 2 focuses on probability sampling. Other nonprobability types of sampling are discussed in most statistics
or survey research methodology texts. Advances in technology
are resulting in rapid change in survey sampling and data
collection, such as the use of online survey panels. These
methods have come under increasing scrutiny because of
questions surrounding their ability to produce representative
samples. Information on a recent examination of these by the
­American Association for Public Opinion Research is available.19 A nice review of recent advances and how methodology
can be improved is provided by Engel and his associates.20



Total Survey Error
Error is inevitable and occurs in any survey. If you need perfection, don’t
bother doing research. What is important is to identify the possible
sources of error and then try to minimize these errors.
Herbert Weisberg defines error as the “difference between an obtained
value and the true value.”1 Typically we don’t know what the true value
is but that doesn’t change our definition of error. When we discussed
sampling in the previous chapter, it was the population value. When we
focus on measurement, it is the true or actual value of whatever is being
measured. Error is the difference between that true value and whatever
the obtained or observed value turns out to be. It’s important to keep in
mind that error can occur at any point in the process of doing a survey
from the initial design of the survey through the writing of the report.
Weisberg points out that error can be random or systematic.2 For
­example, when we select a sample from a population, there will be sampling error. No sample is a perfect representation of the population.
Assuming we are using probability sampling, this error will be random.
However, sometimes some elements in the population are systematically left out of the sample. For example, if we are doing a phone survey
and rely exclusively on landlines that could produce a systematic error
because we have left out cell-phone-only households. Systematic error
is often referred to as bias. We need to be aware of both random and
systematic error.
There are many types of error that can occur. In the past, the focus was
on sampling error and nonresponse error, which occurred as a result of
refusals or the inability to contact respondents. Instead of focusing on just
a couple of types of error, we should focus on all possible types of survey
error. This is often referred to as total survey error.3 Paul Biemer defines
total survey error as the “accumulation of all errors that may arise in the
design, collection, processing and analysis of survey data.”4



There are various ways of categorizing the different types of survey
error. Typically we consider the following types of error:5

Sampling error
Coverage error
Nonresponse error
Measurement error

Weisberg also discusses survey administration issues such as the
• Mode effects, which refers to the fact that different modes of
survey delivery such as telephone, face-to-face, mailed, and
web surveys sometimes produce different results; and
• Postsurvey error, which occurs during the processing and analysis of data.6
To this we would add error that occurs in the reporting of survey data.
We’re going to look at each of these types of error, discuss some of
the research findings about each type, and talk about how you can try to
minimize error.

Sampling Error
As discussed in Chapter 2, sampling error is one of the issues in sample
design and occurs whenever you select a sample from a population. No
sample is ever a perfect picture of the population. Let’s say that your population is all households in the city in which you live. You select a sample
of 500 households from this population.* You’re interested in the proportion of households that recycle such things as cans, bottles, and other
recyclable materials. It turns out that 45 percent of the sample recycles.
* We discussed sampling in Chapter 2 so we’re not going to revisit the details of sampling here.
You might want to look back at Chapter 2.



That doesn’t mean that 45 percent of the population recycles. Why? Sampling always carries with it some amount of sampling error. It’s inevitable.
Here’s another way to understand sampling error. We can use sample
data to estimate population values. If you were to select repeated random samples of the same size from the same population, your sample
estimates would vary from sample to sample. If you think about it, this
makes sense. Each sample will contain a different set of households. So
why would you expect all the samples to give you the same estimate of the
households that recycle?
One of the advantages of probability sampling is that you can estimate the amount of sampling error there will be from sample to sample.
Assuming that you used probability sampling to get your sample and that
you properly selected your sample, the resulting sampling error will be
random. And to make things even better, there are things you can do to
reduce sampling error.
Minimizing Sampling Error
Here are two ways you can reduce sampling error.
• Increase the size of your sample. Of course, there are practical
limits to how big a sample you choose. You’re limited by the
cost and time it will take to collect the data. If you can decide
how much sampling error you’re willing to tolerate, you can
determine the size of the sample that you will need.
• You can also stratify your sample to reduce sampling error.
With stratified sampling you start by dividing your population into homogenous groups such as males and females.
Then you sample from each of these groups. Often you
choose your sample such that the sample has the same
proportion of males and females as does your population.
If you stratify your sample by some variable that is related
to what you want to estimate, then this will reduce sampling error.7



Coverage Error
Earl Babbie distinguishes between the population and the study population. The population is the “theoretically specified aggregation of the
elements in a study” while the study population is the “aggregation of
elements from which a sample is actually selected.”8 In other words, the
population you want to make statements about can be different from the
study population from which you draw your sample. The sampling frame
is the actual list from which the sample is selected.9
Coverage error occurs when the sampling frame does not match the
population. In other words, sometimes the list from which the sample is
selected does not match the population and this produces coverage error.
For example, some elements in the population may have been left out of
the list from which the sample is selected.* Let’s look at some examples.
• The university wants to know how students feel about raising
student fees to partially fund a new student center. The
population is all students registered at the university during
the current semester (or quarter). The list from which the
sample is drawn is the registrar’s official student roster. In this
case, the list from which the sample is drawn almost perfectly
matches the population. The only coverage error would be a
result of errors in the registrar’s list.
• Our research group has a contract to do a consumer attitudes survey in your county. We want to know how people
feel about consumer spending, investments, borrowing, and
savings. The population is all adults (18 years and over) living
in your county at the time of the survey. We decide to do a
telephone survey but we’re not sure how to select our sample.
Here are some possibilities that have been suggested.
°° One member of the team suggests that we draw our sample
from all individuals listed in the phone directory published
by the telephone company. However, this is quickly rejected
* Another problem occurs when elements that are not part of the population are included in
the sampling frame. Sometimes this can be dealt with by screening. For example, in a phone
survey some phone numbers that are outside the geographical area you want to cover might
be included in your sampling frame. If you are aware of this possibility, you could include a
screening question in which you ask if the household is in the desired geographical area.


when another member points out that this would systematically omit all people with unlisted numbers and those with
only cell phones. That would create coverage error since we
would be systematically omitting a large proportion of our
population and people with listed numbers are systematically different from those with unlisted numbers and with
only cell phones.
°° Another team member suggests using a random-digit dialing approach in which residential prefixes of landlines in
your county are sampled and then random digits are added
to these prefixes to produce the list of phone numbers
which we would call.10 This is also rejected when someone
points out that while we would be including those with
unlisted landlines we would be omitting households which
have only cell phones or no landline.
°° Then someone tells us that the U.S. Postal Service has
a list of residential addresses which is available through
commercial providers and which might work for us.
This is referred to as address-based sampling. Shelley Roth
and associates suggest that this provides “nearly complete
coverage of residential addresses in the United States.”11
David McNabb notes that there are some coverage issues
that you might encounter when using this approach.
The list tends to undercover rural areas and groups such as
college students living in dorms. New homes are constantly being built and some homes may be destroyed by
fire and natural disasters.12 So there will still be coverage
error but it would be considerably less than in the first
two options.13
• The General Social Survey (GSS) is a large national probability survey that began in 1972 and is now conducted
biannually by the National Opinion Research Center at the
University of Chicago.14 The population is all adults (18 years
and over) residing in households in the United States as of
a particular date. The sample was originally drawn from all
adults who speak English and live in noninstitutionalized
settings. In 2006, Spanish-speaking individuals were added




to the sample. That means that individuals living in institutionalized settings are systematically excluded and prior to
2006 non-English speaking individuals were excluded. From
2006 onwards those who didn’t speak English or Spanish
were excluded. If those who are excluded are a small part of
the population, this will probably introduce a small amount
of coverage error into the design. Cost considerations may
compel the researcher to live with this small amount of bias in
order to reduce costs.
• Let’s say you want to do a survey of churches in your county.
You want to find out why some church members are more
active in their church than other members. First, you have to
compile a list of all churches in your county. You’re surprised
to find out that such a list is not immediately available but
with some work you assemble the list. Now you select a sample of churches to which you plan to send your survey. You
contact the churches in your sample and ask them to send you
their membership lists so you can select a sample of members
from each church in the sample.* Some churches are not willing to send you their membership list but most offer to send
the list on the condition that you do not share it with anyone
other than the project staff. However, many of the churches
tell you that their membership list is out of date. After more
discussion, you find out that there are several problems.
°° Some members have moved away but are still on the membership list.
°° Not all new members have been added to the list.
°° It’s possible that some members appear twice on the list.
You realize this is going to produce coverage error. The best
solution is to work with each church in your sample to delete
members who are no longer there, add in the new members,
and take out the duplicates. It takes some work but it’s worth
it because it reduces coverage error.

* This is often referred to as a multistage cluster sample.



Minimizing Coverage Error
So how do we try to reduce coverage error? First we have to ask the right
questions. Don Dillman and his associates suggest that there are five questions that we ought to always consider.15
• “Does the list contain everyone in the sample population?”
• “Does the list include names of people who are not in the
study population?”
• “Are the same sample units included in the list more than
• “Does the list contain other information that can be used to
improve the survey?” This could include information such as
phone numbers and e-mail addresses, which could be used to
follow-up those who don’t respond to our survey.
• “What to do when no list is available?” Dillman and his associates use the example of visitors to a national park. In cases like
this, we might sample people as they enter or leave the park.16
So the general strategy for dealing with coverage error is to first identify the sources of error. Once we know what the problems are, then we
can try to reduce them keeping in mind that eliminating all coverage
error is probably not possible. This can be done in several ways.
• We can try to make the list from which we draw our sample
more complete by taking out the elements that shouldn’t
be in the list, adding in the missing elements, and deleting
duplicates. Using the example of sampling church members
discussed earlier in this chapter, we could work with the staff
of the churches to bring their membership lists up to date.
• We can look to see if there are other lists that we can use to
improve coverage. For example, churches might have lists of new
members or members who have transferred out of the church
even though they haven’t updated their membership lists.
• Even if we aren’t able to completely eliminate all coverage
error, we can at least be aware of the error that exists and take



this into account when we write our report. We need to be
careful to limit the degree to which we generalize our results.
For example, with the GSS discussed previously, we should be
careful to only generalize to adults living in noninstitutionalized settings who speak English or Spanish.

Nonresponse Error
Ideally, we want everyone in our sample to complete the survey but we
know that probably isn’t possible for two reasons.
• We probably won’t be able to contact every person in our
sample. For example, in a phone survey, some people are
seldom at home or use caller ID to screen their calls and only
answer when they know who is calling.
• Some of the people who we do contact will refuse to do the
survey. Refusals can occur in two ways.
°° People might completely refuse our request to do the survey. In other words, they don’t answer any of our questions.
This is sometimes referred to as unit nonresponse. We’ll
discuss this next.
°° Other people consent to being interviewed but refuse to
answer certain questions such as family income or race.
This is often referred to as item nonresponse since they
are refusing to answer particular questions or items in our
Theories of Survey Participation
It helps to think about the different reasons that people might agree
or refuse to be interviewed. These are often referred to as theories of
• Social exchange theory. This approach looks at interviewing
as a type of social exchange. Dillman suggests that “people
engage in a social exchange with others when the perceived



rewards outweigh the perceived costs.”17 Costs include such
things as the time it takes to complete the survey or the
amount of energy required to do the survey. Individuals
also receive rewards from doing the survey such as monetary
incentives they might be given or the satisfaction of helping.
Another important factor affecting participation is the trust
respondents have that completing the survey will “provide a
valued reward in the future.”18 From the perspective of social
exchange theory, participation can be encouraged by reducing
the costs associated with the survey, increasing the rewards
from survey participation, and ensuring the trust of the
respondent that rewards will be forthcoming.
• Leverage-salience theory. Robert Groves and his associates
developed the leverage-salience theory of survey participation.
Groves outlines this theory.
Under the theory, different persons place different importance on
features of the survey request (e.g., the topic of the survey, how
long the interview might take, the sponsor of the survey, what the
data will be used for). Some persons may positively value some
attributes, others negatively. Of course, these differences are generally unknown to the survey researcher. When the survey approach
is made to the sample person, one or more of these attributes are
made salient in the interaction with the interviewer or the survey
materials provided to the sample person. Depending on what is
made salient and how much the person negatively or positively
values the attribute, the result could be a refusal or an acceptance.19
In other words, different things are important to different people.
Some place importance on the length of the survey while others focus
more on the topic or incenti