Bonus
Appendix: The Qualitative-Quantitative Debate
©
2000-2005 Mark L. Mitchell & Janina M. Jolley
All
rights reserved.
In
this appendix, we bravely and foolishly go where few textbook authors have gone
before— into the jaws of one of the most controversial and incendiary
topics in all of psychology: the qualitative-quantitative debate. We would like
to rise above the war of words between the quantitative and qualitative camps.
However, to do this, we must first address some of the rhetoric used by both
sides.
To
begin to understand the controversy, you must realize that some scientific
researchers find even the terms “quantitative research” and
“qualitative research” offensive. For most scientific
psychologists, for most of the past 50 years, the only legitimate research
methods have been the traditional, objective, scientific methods discussed in Research
design explained. As advocates of
the qualitative approach sadly concede, it is virtually impossible to do a
dissertation in psychology that does not use these traditional methods (Tolman
& Brydon-Miller, 1997). The
people pushing for different methods, however, did not want to call their
approaches "different methods," “unscientific methods,”
“non-traditional methods,” or “alternative methods”
because such terms would suggest that their methods were inferior. Therefore,
they called traditional scientific methods “quantitative,” and
their methods “qualitative.” Using these terms was a clever way of
implying that (1) the two approaches were of equal value, and that (2)
exclusion of qualitative methods would be narrow and unfair.
As qualitative advocates admit, there were two
drawbacks to this rhetorical strategy. First, the term “qualitative
research methods” is misleading because (a) qualitative methods have very
little in common, except that they are not traditional methods and (b) qualitative research is sometimes
analyzed quantitatively. Second, because of the fundamental differences between
“qualitative” and “quantitative” approaches, most
advocates of qualitative research view the approaches as incompatible (Burman,
1997; Rabinowitz & Weseen, 1997). Rather than having qualitative research
incorporated into the traditional research methods, qualitative advocates want
to fundamentally transform the way psychologists think about psychology and
research (Brydon-Miller & Tolman, 1997; Burman, 1997).
To begin to understand what qualitative research is
and why qualitative research is so incompatible with traditional scientific
research, let us look at one strategy that qualitative advocates have suggested
for promoting qualitative research. This strategy is to reclaim the word
"empirical" from the quantitative researchers (Brydon-Miller &
Tolman, 1997). That is, psychologists often say that psychology is an empirical
field of study. However, by "empirical" most psychologists mean
objective experience. That is, quantitative researchers have defined
"empirical" as the kind of data that their research produces.
However, as qualitative researchers point out, the literal definition of the
term "empirical" means "from experience." Thus, your
feelings toward a participant, as well as your own reflections on your
experiences during the research, although not objective, are “empirical.”
In other words, under the literal definition of "empirical," personal
experiences and subjective impressions would be "data" —if
psychology were defined as "the empirical study of the mind and behavior
of individuals."
Now that you have a hint as to what qualitative research
is, let us define it. We will tentatively define qualitative research
methods as a wide range of
non-objective tools (including interviews and participant observation) designed
to provide rich, in-depth, meaningful descriptions.
The key is that these methods, unlike all the other
methods we have discussed in this text, avoid objectivity. Their focus is more
on interpreting events and finding themes in experiences, rather than in
focusing on the objective facts (Mason, 1996). Qualitative researchers usually
avoid objectivity for at least one of the following four reasons:
1.
They think that there
are certain phenomena (such as
guilt about lying or the experience of being powerless) that objective methods
cannot address.
2.
They think that
objective methods, by ignoring the researcher’s and participant’s
insights, lose too much information.
3.
They do not believe
that “objective” methods are truly objective.
4. They think that the objective approach does not allow them to tell their story. They want to allow their participants' voices and, in some cases, their own voice to be heard. Objectivity, by trying to minimize the role of the observer and by treating the participants as "objects" of study, muffles both the voices of the participant and the observer.
As
we have seen, qualitative advocates do not pride themselves on being objective
scientists. Some openly reject the scientific approach. Others believe what
they are doing is scientific, but argue that objectivity is not a defining
characteristic of science. In contrast, the field of psychology prides itself
on being an objective science (Porter, 1996; Tolman & Brydon-Miller, 1997).
Thus, qualitative methods are infrequently used in psychology. However, in
social sciences that are less wed to objectivity, qualitative methods are more
frequently used. For example, anthropologists use participant observation;
sociologists conduct in-depth interviews; teachers and journalists intensively
observe and interview a single person for days; marketing consultants use focus
groups; and philosophers may write papers using their own reactions—or
even how they think someone else would think—as “data.”
To get a better idea of (a) what qualitative methods
are and (b) how qualitative methods differ from traditional quantitative
methods, we will look at two examples of qualitative research from the point of
view of a traditional, quantitative researcher. Note that we are looking at
these examples from the viewpoint of a quantitative researcher merely to
highlight the differences between the two perspectives. We are quite aware that
some qualitative researchers would say that looking at qualitative research
from the perspective of a quantitative research is unfair (just as many
quantitative researchers would object to looking at quantitative research from
a qualitative perspective). The first tool we will look at is a tool that
businesses often use—the focus group. Then, we will look at a qualitative
method that some people use for getting insights into human nature—the
in-depth interview.
In
the focus group, six to twelve paid volunteers are brought into a room to
discuss an issue. Typically, the moderator has about five set questions he or
she asks during the 90-minute group interview. Beyond asking these questions,
the moderator will ask questions to (a) follow-up on what volunteers say and to
(b) involve all members of the group. Often, the client and another researcher
will be in an adjoining room watching the session through a one-way mirror.
Soon after the session is over, the researcher will
give the client a tape of the session and a short summary of the session. The
summary will focus on the themes that the researcher “saw” in the
session.
Although many businesses and most qualitative
researchers would accept the focus group summary as valid, the quantitative
researcher would not. As you will see, the psychological scientist would have
at least three questions about the conclusions contained in the
moderator’s summary.
First, did these themes come from the participants
or from the moderator? Rather than representing the reality of the situation,
the moderator’s “findings” may only represent the
moderator’s biases. Unlike the traditional scientist, the moderator does
not have to meet some objective standard, like showing that the results are
statistically significant, before declaring to have discovered a pattern in the
data. Instead, the focus group moderator may have to do little more than claim
that she sensed a certain feeling in the group. Admittedly, she may back up
this feeling with a few selected quotes from the focus group members. However,
she might have backed up the opposite conclusion by selecting some different
quotes.
In rebuttal, the focus group moderator could make
six points:
1.
The client has access
to a videotape of the entire session, thus the client can decide whether the
interpretation is biased. At the very least, unlike in structured surveys, the
client is hearing the participants’ own freely chosen words.
2.
The moderator is aware
of the possibility of bias and has been trained to deal with it.
3.
The moderator’s
“lack of objectivity” could also be called “insight.”
4.
There is probably more
than one subjective reality. Some participants may hate the product; others may
love it. One moderator may pick up on one feeling that group members are
having; another moderator might pick up on a different, possibly conflicting
feeling.
5.
The moderator may ask participants
whether the moderator’s interpretation “rings true.”
6.
With such rich,
complex data and with such few participants, statistical analysis is not
feasible.
The second question a quantitative researcher might
have is, "Even if the moderator correctly captured what participants said,
can we believe what the participants said?" There are numerous problems
with self-report data. Participants may lie, try to please the moderator,
exaggerate, forget, and very often, just not really know. As we mention
throughout Research design explained (e.g., pages 92-95 and pages 184-186), self-reports can be inaccurate,
even for such simple things as reporting how one uses a new product.
The qualitative researcher might reply that only
self-report data can do justice to understanding what people are thinking. Any
other method would be too incomplete and too indirect.
The third question a quantitative researcher might
have is, "Even if the moderator is (a) correct about the participants and
(b) the participant’s verbal statements accurately reflect their thought
processes, how can we make any generalizations from the focus group?" The
12 participants are a small, non-random sample of the population of interest.
Furthermore, the participants are not responding as independent individuals. Instead, the participants are
responding as a group. Consequently, the more dominant participants’
voices will be heard more, and there will be a tendency for members to conform
to those leaders.
Because of concerns about the external validity of
small, non-random, non-independent samples, the popularity of focus groups has
declined. For example, some ad agencies have stopped using focus groups because
of cases of television ads that were approved by focus groups, but were later
labeled by a large percentage of television viewers as offensive. In some cases,
companies had to remove those ads after only one airing and apologize.
In another notable case, two-term President Bill
Clinton may have won re-election because he put less faith in focus groups than
Bob Dole, his republican challenger. Bob Dole asked voters to vote for the
presidential candidate they trusted more. Then, Dole, in an attempt to convince
voters that they trusted him more than Clinton, asked voters to think about
whom they would rather have baby-sit their kids. He was sure people would choose
him because his focus groups had told him that was the case. However, in a
telephone survey of a random sample of voters, over 75% of the respondents said
they would prefer Clinton to baby sit their kids.
Qualitative researchers acknowledge that errors can
be made when generalizing from samples, but point out that such errors occur in
both qualitative and quantitative research. Quantitative pollsters, for
example, have made large mistakes in predicting election results. Furthermore,
the problem with the focus groups in the cases mentioned above could have been
due to the investigator improperly selecting the sample of participants—a
problem that can also ruin external validity in quantitative research.
Not
all qualitative research is designed to address an applied problem. Often, as
in quantitative research, the goal is to either describe or understand some
aspect of human nature or experience, such as love. However, rather than use
the objective methods we described in this text, qualitative investigators
adopt more subjective, personal approach. For example, consider a study in
which an investigator follows two classmates over the period of a year. She
extensively interviews her classmates about their relationships and intensively
analyzes the transcripts of those interviews. She concludes that people often
do not want to commit to a relationship because they have a “fear of
losing themselves, of having themselves devoured.”
As with focus groups, some people consider this type
of investigation quite acceptable. Many philosophers believe in this approach.
Many qualitative researchers believe in this approach. The traditional
psychological scientist, however, would have at least three questions about
this approach.
First, how does the investigator know that the
classmates have this extreme fear of losing themselves? A psychologist would be
skeptical about whether the construct “fear of losing oneself”
exists. If it does exist, the psychologist would still need objective evidence
that the measure of this construct was reliable and valid.
The qualitative investigator would argue that she
has done a thorough job of establishing that the construct exists. The
difference is that whereas the quantitative researcher would attack the problem
from the outside in (from observable behavior to internal thoughts), the
qualitative researcher attacks the problem from the inside—by empathizing
with the clients. To rule out bias, she came into the project with an
open-mind. She was not, like some “objective” scientists, trying to
confirm a theory or hypothesis[1].
To determine what participants thought, she has extensively analyzed
transcripts of what they said, empathized with them, and asked them if her
interpretation was accurate.
The second question a quantitative researcher might
have is, "How does the investigator know what caused the break up of
relationships?" Traditional psychologists are very skeptical about
cause-effect conclusions. They realize how difficult it is to isolate the cause
of an effect. The qualitative investigator would say that (a) as a result of
the in-depth interviews, she was aware of any other relevant, potential causal
variables and was able to rule them out, and (b) the participants agreed with
her analysis, and they ought to know.
The third question a quantitative researcher might
have is, "How can anyone make generalizations about most people based on
studying only two participants?" This is a question that many qualitative
researchers would also ask. The qualitative investigator would probably concede
that the study has only documented the inner lives of the two participants and
that replication would be a good idea. However, the qualitative researcher
might argue that there are universal truths that apply to everyone.
As
our discussion of the focus group and the in-depth interview suggest,
quantitative researchers do not readily accept qualitative research. Indeed,
some traditional psychologists see these alternative methods as being
frighteningly similar to those methods used by infomercials, journalists,
novelists, and some talk show hosts. Some traditional
psychologists—although accepting that these methods are serious,
systematic, legitimate methods of inquiry—still believe that these
methods have no place in psychology.
The conflict between quantitative and qualitative researchers is not
one-sided. Some qualitative researchers, for example, find scientific research
quite appalling.
In short, qualitative and quantitative researchers have quite different
perspectives. In the next sections, we will highlight seven of the main
differences:
1.
Disagreements about
the need for an alternative to traditional scientific methods,
2.
The value of
self-report,
3.
The value of objectivity,
4.
The value of using
numbers to describe data,
5.
The value of training
and standardization,
6.
The best way to deal
with the fact that human thought and behavior is complex, and
7.
The way to evaluate
the internal, construct, and external validity of research.
Psychology is usually defined as the science of human and animal behavior. As we discussed in
Chapter 1, the scientific approach is what separates psychology from many other
fields, including philosophy. As you also saw in Chapter 1, the scientific
method has been incredibly productive for psychology. Indeed, psychological
science is progressing so fast that it is difficult for even professors to keep
up with the developments in their own specialty areas, much less with all the
developments in the entire field.
The qualitative researcher, however, claims that,
despite the apparent progress of scientific psychology, the human world is too
different from the natural world to use the objective scientific method. Human
traits such as goals, plans, traditions, and the ability to reflect require
that humans be studied through non-scientific means. Thus, qualitative
researchers think the objective scientific approach is oversimplified and
leaves too much out. According to some qualitative researchers, if we want a
rich picture of how people think and feel, objective scientific psychology has
little to offer.
Qualitative
researchers believe that one reason quantitative research is limited is that
quantitative researchers do not take full advantage of what participants have
to say. Unlike quantitative researchers, qualitative researchers tend to place
a great deal of faith in self-report. This belief in self-report allows
qualitative researchers to use participants as collaborators.
According to quantitative researchers, this faith is
unjustified. If people were aware of their own minds, human behavior would not
be so mysterious. Indeed, there would not have been a need for psychologists.
Long before the beginning of scientific psychology, people would have known
everything there was to know about the human mind. However, as scientific
research has clearly established, self-report data is highly questionable
(Sechrest & Sidani, 1995). Research has clearly established that people
often
1.
Do not remember what
they did;
2.
Do not know why they
do what they do; and
3.
Cannot accurately
predict what they would do in a certain situation.
Furthermore, if one really believed
self-reports, one would have to believe in all sorts of things, such as men
giving birth, Elvis sightings, and UFO abductions (thousands of normal people
have reported them).
The
history of psychology testifies to two facts: (a) introspective reports cannot
be trusted and (b) people would like to believe in introspective, self-reports.
By 1867, 12 years before Wilhelm Wundt established the first laboratory in
psychology, the problems with self-report had been clearly articulated.
Nevertheless, in this lab, Wundt and his followers relied heavily on
introspective self-reports.
Admittedly, they went to great lengths to try to make such self-report
valid. His students went through a year of training and more than 10,000
practice self-reports before they were considered qualified to make self-reports.
Despite these efforts, there was little evidence that these self-reports were
accurate. Furthermore, Wundt was never able to successfully address the
problems with self-report that had been clearly articulated back in 1867. A
little later, other psychologists tried a more natural, common sense approach
to introspection. Once again, there was little evidence that these self-reports
were valid. Once again, the problems with self-report identified in 1867 were
still not successfully addressed. Today, after more than 100 years of trying to
rescue introspective self-reports, the main outcome is that we have discovered
that there are even more problems with self-report than those identified in
1867.
To
most quantitative researchers, a careful analysis of the history of both
science and scientific psychology shows not only that we should question
self-report data, but also that we should try to be as objective as possible.
Objectivity, as we pointed out in Chapter 1, is central to the scientific
approach. Scientists try to eliminate bias and subjectivity to find knowable
truths. Without objectivity, we may end up just seeing what we expect to see
rather than testing reality. Without objectivity, you selectively notice
information that fits your biases and you selectively interpret ambiguous
information to fit your biases. According to quantitative researchers,
objective methods help you find the truth, whereas as non-objective methods
allow you to find what you want to find. Thus, quantitative researchers would
argue that people using non-objective methods are being seduced. In other
words, the person who has strong beliefs may not be pleased when the cold,
objective truth, unearthed by quantitative methods, does not support all these
passionately held beliefs. That same person may be pleased when, by using
qualitative methods, the person discovers strong support for those same
beliefs. Thus, the person may come to believe in qualitative research.
As
you will see, qualitative researchers do not accept the quantitative
researchers’ arguments about the value of objectivity. Some argue that
the price of objectivity is too high. Others claim that the objectivity of
quantitative methods is just an illusion. Still others argue that objectivity
is not a valuable goal.
Some
qualitative researchers claim that, even if objectivity removes the researcher
from bias, the cost of objectivity is too high. To be more specific,
qualitative researchers see two main drawbacks to objectivity. First,
objectivity costs the researcher access to rich sources of information. Second,
by treating the participant as objects, participants are not treated in a moral
and ethical way.
The
objective approach means that you have to exclude many sources of information.
The researcher cannot use any personal insights he or she may have about the
participant. The researcher is similarly cut off from valuable information that
the participant might provide. Consequently, many questions about people cannot
be answered because there is no objective way to study those questions. For
those questions that researchers can study objectively, the cost of finding out
is that they must restrict how the participant responds. Thus, the participant
may be forced to respond in some artificially simple way (pressing a key on a
computer or circling a number on a rating scale) rather than in a complex and
natural way. As a result of objective researchers isolating themselves from
participants, objective researchers lose access to the in-depth knowledge of
the human condition that art, literature, philosophy, and other less objective
fields can provide.
As
we have suggested, being objective may involve treating the participant as the object of study. As we mentioned earlier, some qualitative
researchers believe that the cost of this impersonal, “objective” treatment is that the researcher
loses out on the valuable information the participant has to share with the
researcher.
Part of this loss of information is due to the
one-sided relationship between the researcher and the “object” of
study. If the researcher would treat the participant as another thinking,
intelligent life form—rather than as an object—then the researcher
could benefit from the participant’s insights. The participant may have
much to teach the researcher, but the researcher does not allow the participant
to collaborate with the researcher. Indeed, from the objective
researcher’s point of view, collaboration between the researcher and the
participant would bias the results.
To
prevent collaboration—and to keep the participant in the role of an
object of study—the participant in a quantitative study is often less
than completely informed about the study—and may even be deceived about
the study. Furthermore, the participant expected to do what the researcher
instructs the participant to do. The conventional researcher views this
momentary failure to equally share power and knowledge as the cost of getting
objectively valid information.
Some
qualitative researchers view this impersonal, “object-ive”
treatment as highly unethical. Treating participants like objects is demeaning,
and it gives the researcher the excuse to exert power over the
“objects” of the research.
Rather than “oppressing” the participant, the participant
should be treated as an equal. In short, another cost of objectivity is that
the participant is treated as an object, instead of in a moral and ethical way.
Some
qualitative researchers charge that "objective methods" are not
really objective because objectivity is impossible. They may attack the
scientists’ claim of objectivity by making such mocking comments as,
“scientists believe in ‘the immaculate perception’ of
behavior” and that “scientists believe they have a
‘God’s eye view’ of reality.” Qualitative advocates
point out that objective researchers ignore research showing that human
observers are not completely objective (Rabinowitz & Weseen, 1997).
Furthermore, qualitative proponents argue that no procedure can guarantee
objectivity. Ultimately, the objectivity of the research lies in the honesty,
integrity, and objectivity of the individual researcher rather than in the
particular method used.
Even when the observation itself is objective,
scientists still make several subjective decisions and interpretations. The
researcher decides what to look for, which, in turn, affects what the
researcher will find. In questionnaire research, for example, the researcher
chooses to ask certain questions and not others. Furthermore, the researcher
interprets participants’ responses. In questionnaire research, the
researcher assumes the participant understands the questions and interprets the
number the participant circles on the rating scale as reflecting how the
participant really feels. That is, there is a leap from the participant’s
actual behavior (e.g., circling a few circles on a multiple-choice form) and
the researcher’s description of that behavior (e.g., “the participant
is prejudiced”).
To the qualitative researcher, the fact that
quantitative researchers make these subjective decisions is especially
unsettling because quantitative researchers do not come into the research
process with what qualitative researchers consider an open-minded, objective
attitude. Instead, the quantitative researcher is viewed as trying to find
evidence to support a favored hypothesis or theory.
A few qualitative researchers would go further,
saying that scientists are trying to support traditional political views.
Indeed, it is not uncommon to hear qualitative researchers consider traditional
psychological research sexist, racist, and “heterosexualist”
(Rabinowitz & Weseen, 1997). Some qualitative researchers argue that
"objective" science, while outwardly appearing to be objective and
fair, is really a tool for oppressing and marginalizing less powerful groups.
They do not think that qualitative research oppresses or marginalizes other
groups. Indeed, it is not unusual to hear a Marxist or feminist qualitative
researcher say that they were drawn to qualitative research because it allowed
their voice to be heard.
In short, qualitative researchers believe that
quantitative researchers do not go into a research project with an open-minded,
objective attitude. Instead, quantitative researchers have some hypothesis they
want to confirm or some political position they want to justify. This
preconceived idea drives what the how quantitative researchers conduct their
research. Given
that—according to qualitative researchers—quantitative researchers
tend to find what they look for, the quantitative researchers’
“objective” system fails. The key to objectivity, according to
qualitative investigators, is to adopt an objective attitude by going into the
research without any preconceptions.
Most quantitative researchers strongly disagree with
the qualitative position on how to achieve objectivity. Indeed, they think the
qualitative researchers have it backwards.
According to quantitative researchers, the solution
of “trying to be open-minded” is really the problem. People think
they can be open-minded when they cannot. As science has shown, people process
information automatically and unconsciously according to their preconceptions.
Because this processing does not occur at the conscious level, people fool
themselves into thinking that they are being fair and open-minded.
The way to avoid fooling oneself is to use the
scientific approach. Over the centuries, scientists have developed a system for
overcoming bias. As evidence that the system works, scientists do not find
exactly what they expect to find. Indeed, when investigating a new area,
scientists are almost as likely to find just the opposite of what they expect
to find than to find exactly what they expect to find.
In summary, qualitative researchers believe that we can trust the
individual researcher to be fair, objective, and trustworthy; whereas
quantitative researchers believe that the only way we can get trustworthy data
is to trust the safeguards that scientists have devised to prevent people from
fooling themselves. Thus, qualitative researchers may trust the
individual’s conscious intentions, whereas quantitative researchers trust
the scientific method. Put another way, quantitative researchers are concerned
about the hidden, unspoken biases of the individual researcher; whereas
qualitative researchers are concerned about the hidden, unspoken biases of the
scientific community.
Rather
than arguing about whether quantitative methods provide objective information,
a few qualitative researchers argue that objectivity is not a worthwhile goal.
These qualitative researchers do not believe that there is one objective
reality to be found. Instead, the social constructionists believe that reality
is an opinion, or as they call it, “a social construction.” Reality
is different for everyone. A heterosexual’s reality is different from a
homosexual’s; a chauvinist’s reality is different from a
feminist’s, and so on. Given that there is no one reality,
everyone’s reality is equally valid. Therefore, the job of psychologists
should be to make sure that everyone’s voice is heard. We should not let
the voice of the dominant majority drown out other equally valid views of
reality.
Because
of this view that there is no objective reality, social constructionists do not
accept the idea of objectively analyzing data. Some other qualitative
researchers also reject quantitative analysis.
Most traditional scientific researchers, on the
other hand, want to see what the numbers say. Indeed, to many quantitative
researchers, the name “qualitative research” suggests that its
advocates are people who know very little about statistics. Almost any data can
be quantitatively analyzed. If you want to find out whether men or women use
different words or whether men are more likely to use a certain phrase, you can
use statistics. In fact, you should use statistics. Without statistics, you may
either (a) see relationships that do not exist or (b) fail to find
relationships that do exist.
To reiterate, if there really is a relationship
between variables, we have the mathematical tools to uncover that relationship.
If, on the other hand, numerical analysis cannot find the relationship, the
relationship is not some “glimmer” in the data— it is in the
eye of the beholder.
Although the quantitative researchers’
arguments may initially seem compelling, qualitative researchers have strong
rebuttals to these arguments. We will now look at the qualitative
researchers’ rebuttal for each of four main points that quantitative
researchers make.
1.
“Numbers speak
for themselves.” Numbers do not speak for themselves. Numbers alone can
be quite meaningless. Prematurely reducing experience to numbers may prevent,
rather than facilitate, understanding. For example, the rush to measure
intelligence in terms of numbers occurred before intelligence was properly
defined and understood (Sternberg, 1996). Similarly, numbers may not adequately
shed light on how it feels to be a person who is discriminated against.
Finally, we all know of meaningless and irrelevant quantitative studies that
produced little more than numbers. Because of the limitations of numbers,
non-numerical analyses may be necessary to find out meaningful information and
to powerfully communicate findings to others.
2.
“Qualitative
researchers just do not understand statistics.” Rather than being
statistically illiterate, some of the qualitative researchers, in their
arguments for not using statistics, show a deep understanding of the
assumptions underlying statistical tests. Indeed, experts in quantitative
research share some of these concerns about the way statistical tests are used.
For example, some quantitative researchers often act, when doing statistical
analyses, like their survey was a random sample of participants—even when
more than half the people in the original sample refused to participate in the
survey. Similarly, when conducting hypothesis tests, some quantitative
researchers act as though they are testing the null hypothesis that there is
absolutely no relationship between their variables. In some cases, the
assumption that a treatment, for example, would have absolutely no effect on
any participant (or that the positive effects and negative effects would
balance out perfectly) is hard to defend (Dillon, 1999). If the hypothesis that
quantitative researchers are trying to disprove is almost certainly false, one
could argue that the quantitative researcher is just using statistics to find what
they want to find. Finally, in many qualitative studies, samples are small and
observations that are not independent. In such cases, it may not be reasonable
to do statistical tests.
3.
“Qualitative
researchers simply ‘eye ball’ the data.” Qualitative researchers
use a variety of techniques to analyze their data. They may make tables,
charts, matrixes, concept maps, graphs, and diagrams to summarize the data
(Cresswell, 1998; Miles & Huberman, 1994). They may use computer programs
to help sort, index, and organize their data. They may even use statistical
techniques.
4.
"Qualitative
researchers ask the reader to trust that their impression of what the data
‘feel like’ is correct.” This is usually not true. The
qualitative researcher knows that the article will be reviewed and read by
skeptical peers. Qualitative researchers have to support their claims. One
common way that they support their claims is by making all or most of their raw
data available to the reader.
As
we suggested in the previous section, some quantitative researchers think that
qualitative researchers are too soft, casual, and informal in analyzing data.
Quantitative researchers also think that qualitative researchers are too lax in
terms of training researchers and in terms of following standardized
procedures.
Part of this perceived lack of rigor in training
comes from the fact that training in qualitative research does not require
training in statistics. Part of this perceived lack of rigor comes from the
view that the training of qualitative techniques is often very brief.
Allegedly, one result of this inadequate training is that qualitative
researchers do not know how to judge whether a qualitative study is poorly or
well executed, much less to know how to do “good” qualitative
research.
Some qualitative researchers share the concern that
it is hard to know whether qualitative research is good or bad (Rabinowitz
& Weseen, 1997). They also express some concern that it is unclear about
when the analysis of a qualitative study, such as the intense study of an
individual or a participant observation study, is complete. Some point out that
the standards of training and quality are better for some qualitative methods
than for others (Cresswell, 1998). Many qualitative researchers hope that as
qualitative methods become more accepted, more formal rules— and more
training in those rules— will follow (Cresswell, 1998; Tolman &
Brydon-Miller, 1997).
Other qualitative researchers respond that training
in qualitative training is different, but no less rigorous, than training in
quantitative research methods.
Still others claim that the “rigor” in quantitative training
simply reflects the desire of a power elite to impose initiation rites on
would-be scientists. This rigorous initiation rite serves to (a) restrict
access to acceptable ways of generating knowledge to members acceptable to the
elite, thereby providing a way to exclude people who are oppressed or who have
different viewpoints; and to (b) make scientists more committed to both the
scientific fraternity and to these traditional methods.
Although some quantitative researchers strongly
believe that training in qualitative research lacks rigor, most are more
concerned about the failure of qualitative researchers to follow rigorously
standardized procedures. The quantitative researcher sees two problems with the
lack of standardization. First, without standardizing your procedures, others
cannot replicate your study. Second, because the researcher influences the
participant, we can only make sense of the participant’s behavior if we
know what the researcher did.
Qualitative researchers respond that it is
impossible to have such hard and fast rules in the complex situations that
qualitative researchers study. Even if you could start out with some hard and
fast rules, since your relationship with the participants is evolving, your
research question is changing, and you are often interacting with participants
for a long period of time, you could not follow those rules for the entire
study. The best that one could do is to (a) come into the situation with no
preconceived ideas and (b) to see if others, who also adopt an open-minded
attitude, obtain similar results.
Not surprisingly, quantitative researchers do not
completely agree with either of these points. Quantitative researchers
vigorously object to the idea that investigators can go into a situation
without any preconceptions. They point to research showing that people
automatically and unconsciously process information according to their
preconceived notions. If you say you are unbiased, you are either lying or
fooling yourself. If you admit that you have biases and claim that you will try
to counter them, you will still fail because much of the bias happens at a
nonconscious level.
Quantitative researchers are in partial agreement
with qualitative researchers about the value and meaning of replication. Both
qualitative and quantitative advocates agree that if a person with a different
set of biases finds the same results, confidence in the original study's
results are increased. However, quantitative and qualitative people disagree
about the meaning of just about any other outcome of a replication. For
example, suppose that a researcher with a similar set of biases obtains the
same results. In that case, the quantitative researcher would argue that little
has been proved. Bias, perhaps due
to lack of standardized procedures, could account for the similarity of
findings.
If the replication obtains a different set of
findings, the quantitative researcher would believe that the credibility of one
or both studies was seriously damaged.
The qualitative researcher, on the other hand, might not be so
concerned. The qualitative researcher might explain the failure to replicate by
saying either that (a) there are multiple realities or that (b) each situation
is unique and the two investigators are not studying the same exact same
situation.
Note
that some of the conflict between quantitative and qualitative researchers
about standardization might be avoided if qualitative researchers would study
phenomena in simpler, more controlled situations. If qualitative researchers
were to study simpler processes and situations, standardization would be
easier. However, the qualitative researcher thinks that the way to study
complex phenomena (such as aggression, forgetting, etc.) is to study these
phenomena in natural settings using qualitative means.
The quantitative researcher’s response to the complexity of
psychological phenomena, on the other hand, is that “the more complex a
phenomenon, the greater the need to study it under controlled conditions and
the less it ought to be studied in its natural complexity” (Banaji &
Crowder, 1989, p. 1192). As quantitative researchers point out (Banaji &
Crowder, 1989), it has been the ability to study complex phenomena under
controlled conditions that has led to our understanding of phenomena such as
obedience, eye-witness memory, helping behavior, and so forth.
The
argument about simplifying the situation highlights a difference between
quantitative and qualitative researchers on the three types of validity. In
general, the quantitative researcher thinks the qualitative researcher is not
skeptical enough when evaluating the validity of research. The qualitative
researcher, on the other hand, does not point to these three validities as the
standard for which validity should be judged (Tolman & Brydon-Miller,
1997). Indeed, some qualitative researchers argue that thinking about proposed
research in these scientific terms should be avoided because it will steer the
research toward being scientific and away from being humane (Denzin, 1989).
The
quantitative researcher sees the task of identifying the cause of some event as
very difficult, requiring the isolation of potential causes. The qualitative
researcher thinks the cause can be determined, even in a deeply complex,
naturalistic setting. A few qualitative researchers think that the cause can be
determined simply by asking participants why they are doing the behavior.
Similarly,
some qualitative researchers are willing to accept the construct validity of
introspective self-reports. The quantitative researcher, on the other hand,
would be skeptical about the validity of any measure, especially self-report
measures. For example, the quantitative researcher would like objective
evidence of the reliability and validity of the self-report measure.
Although
the qualitative researcher likes to study participants in complex, naturalistic
settings, quantitative researchers still see qualitative researchers as
naïve when it comes to external validity (Banaji & Crowder, 1989). For
example, they question how a qualitative researcher can generalize from one
complex naturalistic, real-world context to another, especially since, as
qualitative researchers acknowledge, each naturalistic, real world situation is
unique?
Quantitative researchers particularly object to
those few qualitative researchers who do try to make generalizations based on
studying only one individual or after extensively interviewing only a few
participants. They find such qualitative researchers incredibly naïve.
We
have highlighted the significant differences between people on either side of
the quantitative-qualitative debate. The differences are so great that many
people on either side of the debate would not accept the other’s
research. Many quantitative researchers sneer at focus groups, unstructured
surveys, and in-depth interviews of a single person. Some quantitative
researchers view these techniques like inkblot tests: They tell you more about
the investigator making the interpretation than they tell you about what
participants are really thinking or doing. Others view these techniques as
telling cute stories, but not providing real information. Conversely, some
qualitative researchers sneer at surveys and experiments as being artificial
and shallow. However, as you will see in the next sections, (a) people on both
sides of the debate share some common concerns; and (b) not everyone is
completely one side or the other of the quantitative-qualitative debate.
Despite
their differences, even the most adamant quantitative researcher and the most
adamant qualitative researcher would probably agree on three problems with
quantitative research. First, some quantitative research is unimaginative or
poorly done. Second, some quantitative studies would benefit from obtaining
richer data. Third, some quantitative analyses are so sophisticated that they
are hard to communicate.
Both
quantitative and qualitative researchers would agree that not all quantitative
research is well planned and well executed. For example, as we discussed in
Chapter 7,
1.
Many questionnaires
are hastily and poorly written;
2.
Some surveys have a 2%
response rate; and
3.
Some researchers use
convenience rather than random samples.
One approach to the problem of poor quantitative research is to use
qualitative research. For example, consider the following scenario proposed by
a qualitative researcher:
Quantitative researchers
discover a correlation between children with behavior problems and those with
parents with marital problems. However, those with behavior problems may not
have actually been exposed to their parent’s conflict, but instead were
hostile before the conflict arose.
A qualitative researcher would propose a qualitative solution. For
example, a qualitative researcher might propose intensively interviewing some
of the children to find out what was happening.
Solving the problem, however, does not require a qualitative approach.
Indeed, intensive interviews may be misleading because children may not
accurately remember whether they themselves were hostile before they learned of
their parent’s conflict.
The quantitative approach to problem would be to administer
questionnaires several times over a period of years. In this way, we could
determine whether parental conflict came before or after the children’s
hostility.
Both
quantitative and qualitative researchers would agree that some quantitative
research should obtain richer, more detailed descriptions of behavior and
experience. The qualitative researcher’s solution is to use qualitative
methods, such as in-depth interviews of individual’s or even reporting
the researcher’s own conscious experience.
The quantitative researcher, on the other hand,
would solve the problem by using quantitative methods more imaginatively. For
example, Baumeister (1995)
collected stories from college students about their experiences. In one study,
he asked them to write about an event in which they had made someone angry and
another event in which someone had made them angry. Although many of these
stories were fascinating, complex, and rich in detail, the stories could be
coded objectively. That is, coders could agree on whether each story had certain
features. For example, coders could agree on whether the story contained an
apology from the perpetrator. The findings of this study yielded interesting,
quantitative data showing, among other things, that “victims describe
perpetrators as inconsistent and immoral and rarely mention an apology, whereas
perpetrators often say they apologize and made amends and often state they
could not help what happened.”
Finally,
both quantitative and qualitative researchers admit that communicating the
results of some quantitative studies is quite difficult. Some qualitative
researchers see the solution as avoiding statistics and using qualitative
studies in which the raw data are the participants’ own words. These
qualitative analyses can be quite powerful and moving. Quantitative researchers think that the
problem can be solved without abandoning quantitative methods. Specifically,
they envision a two-part solution. First, researchers should use more graphs
and more descriptive statistics to help others understand the pattern in their
data. Second, schools need to do a better job of teaching quantitative
literacy.
Thus
far, we have talked about cases in which quantitative and qualitative
researchers saw the same problem, but found different solutions. Qualitative
researchers found qualitative solutions, quantitative researchers found
quantitative solutions. However, as you will see in the next sections,
researchers are not always such purists: Some qualitative researchers use
quantitative methods, some quantitative researchers use qualitative methods.
Some
qualitative researchers, for example, have decided to use quantitative methods
to analyze their data. They see that there are problems with relying on memory
and intuitive statistics to see relationships among variables. Therefore, they
use computer programs such as NVivo® to link data from such sources as
in-depth interviews or participant observation with statistical software. Some
of these qualitative researchers see the main difference between their research
and that of quantitative researchers is that their data are richer, more
detailed, more naturalistic, and more unstructured (“messier”) than
that of quantitative researchers.
Just
as some qualitative researchers use quantitative methods, some quantitative
researchers use qualitative methods. Most quantitative researchers use
qualitative methods as exploratory tools. For example, when trying to get a hypothesis
for a study, the quantitative researcher probably will search the literature.
In addition, he or she may do some qualitative research to get some starting
point. She may reflect on her own experiences in similar situations, interview
some friends, or informally observe some people who are similar to the group
she wants to study. In short, the quantitative researcher may use qualitative
research to determine what to study.
Before
conducting a full-blown study, many researchers conduct a pilot study. This
pilot study may involve having one’s friends participate in the study and
asking them about their impressions of the study. The pilot study may even
involve having the researcher take the role of a participant.
When
devising a questionnaire, one of the first steps that many researchers take is
to do some qualitative research. They may ask people who are like those they
want to survey questions, such as
1.
What does _____ (the
construct the researcher plans to measure, e.g., stress) mean to you?
2.
How does someone who
is _____ (high in the construct) behave differently than someone who is _____
(low in the construct)?
In
addition, the researcher would test out the questions on a sample of people to
see if they understood those questions. Without such qualitative research, the
questionnaire is likely to be meaningless to participants and, consequently,
invalid.
When
giving an oral report of their research, psychologists are likely to report
some qualitative findings that support the quantitative results. For example,
they may make their findings more accessible to the audience by describing how
a participant acted, repeating an interesting quote from a participant, or even
showing video clips of participants' reactions. Researchers may even talk about
their own experiences about being in similar situations. In so doing, they are
applying a well-known finding from quantitative research— that
qualitative evidence is more convincing to most people than statistical
evidence.
Finally, we should mention that some well-known
research has a qualitative component. From your introductory psychology course,
you may recall an interesting study in which the researchers checked themselves
into mental hospitals to see what the experience was like (Rosenhan, 1973). One
reason that study is so influential is that the participants’ reports of
feeling powerless are so moving. Similarly, some of the remarkable findings of
Zimbardo’s prison study (Zimbardo, Haney, Banks, & Jaffe, 1975) were
based on the qualitative data about guards abusing prisoners and prisoners
siding with unfairly abusive guards against a prisoner who wanted to be treated
fairly. In short, many landmark studies, including those by Darwin, Freud, and
Piaget, have included a qualitative component.
In
conclusion, both qualitative and quantitative research involve serious,
systematic scholarship and both can yield insights. The fact that there are
psychologists who advocate qualitative methods highlights some problems with
how traditional research is conducted and communicated. It is quite possible
that qualitative methods will be more frequently used in psychology. In fact,
some methods that were once considered to be “qualitative,” such as
some forms of content analysis, are now considered traditional research
methods.
However, as we have discussed and as you can see from Table 1 (below), there are substantial differences between the
two methods. The differences stem primarily from the fact that
qualitative methods are not objective. Consequently, the non-objective evidence
that qualitative researchers consider "data" (themes that the
investigator senses, the investigator's subjective impressions, etc.) is not
what quantitative researchers consider "data." Given the strong
tradition of objective methods in psychology (Burman, 1997; Porter, 1998) and the success of quantitative
methods in psychology, it is unlikely that qualitative methods will replace
quantitative methods. The methods you have learned in this course should serve
you and psychology well into the future.
Table
1
Qualitative
Methods in the Context of Principles Taught in Selected Chapters of Research
Design Explained
|
Chapter |
Key
Principle Challenged |
Key
Principle Agreed With |
Comments |
|
1 |
Objective
methods are productive. |
Publicly
sharing findings is important. Replication is very important. Ethical
considerations are all important. |
Qualitative
researchers tend to view themselves as more moral and ethical because they
tend not to manipulate variables, they never deceive participants, and they
treat participants as equals rather than as objects. |
|
2 |
Research
should be designed to test a specific hypothesis |
Literature
review and relating one’s investigation to theory are vital elements in
many types of qualitative research. |
To
the qualitative researcher, being open-minded, rather than wed to a
hypothesis, is very important. The hypothesis may actually change in the
middle of the study. |
|
4 |
Try
to eliminate the role of the human observer and beware of participant biases,
especially in self-report measures. |
Trying
to reduce bias and get converging evidence that impressions are correct. |
The
human observer is the most valuable measuring instrument the qualitative
researcher has. |
|
8 |
Establishing
cause and effect is difficult. However, control of variables may make it
easier. |
|
In
qualitative research,
establishing cause-effect is often not a goal. |
|
9 |
Statistical
considerations should affect almost every aspect of the design process. |
|
In
qualitative research, (a) small samples are legitimate, and (b) statistics
may or many not be used. |
References
Brydon-Miller, M. & Tolman,
D. L. (1997). Engaging the process of transformation. Journal of Social Issues, 53, 803-810.
Burman, E. Minding the gap: Positivism, psychology,
and the politics of qualitative methods. Journal of Social Issues, 53, 785-802.
Creswell, J. W. (1998). Qualitative inquiry and
research design: Choosing among five traditions. Thousand Oaks, CA: Sage.
Denzin, N. K. (1989). Interpretive interactionism. Newbury Park, CA: Sage.
Dillon, K. M. (1999). I am 95% confident that the
earth is round: An interview about statistics with Chris Spatz. Teaching of
Psychology, 26, 232-234.
Mason, J. (1996). Qualitative researching. Thousand Oaks, CA: Sage.
Miles, M. B. & Huberman, A. M. (1994). Qualitative
data analysis: An expanded sourcebook. Thousand Oaks, CA: Sage.
Rabinowitz, V. C. & Weseen, S. (1997).
Elucidating epistemological impasses: Re-Viewing the qualitative-quantitative
debates in psychology. Journal of Social Issues, 53, 605-630.
Seechrest, L. & Sidani, S. (1995). Quantitative
and qualitative methods: Is there an alternative? Evaluation and Program
Planning, 18, 77-87.
Tolman, D. L. & Brydon-Miller, M. (1997).
Transforming psychology: Interpretative and participatory research
methods. Journal of Social
Issues, 53, 597-603.
[1] Quantitative researchers would argue that, rather than trying to confirm hypotheses, scientists test and attempt to disprove hypotheses.