7400.685-080 - Research Methods in FCS
School of Family and Consumer Sciences
Spring Semesters - Tuesday Evenings 5:20-7:55pm in 209 Schrank Hall South
Instructor: David D. Witt, Ph.D.
Chapter Experimental Designs
Experiments that are done well provide results that are difficult to challenge, provided the researcher maintains adherence to all tenets of the scientific method and paying close attention to issues of sampling, randomization of subjects, reliability and validity.  Usually, when we think of experiments, we think of the scientist in a laboratory, working with various bubbling liquids and physical elements. The reason for that image is because experimental science has its beginnings in the laboratory, where most elements in the experiement are under the tight control of the scientist.  Control is fundamental to experimental research.

Basically, the researcher sets up a laboratory situation in which measures are taken of the variables of interest.  Then a change agent is introduced into the environment and given time to have its suspected effect.  Finally measures are retaken of the variables of interest. Logically, Any change in the variable of interest must be due to the change agent, as long as care is taken to control all the other extraneous elements that may have caused the change.  In these classical experiments, measurement times are referred to as Time 1, Time 2 ... Time nth. The change agent is often referred to as a Treatment or Exposure agent.

Experiments in social science, as opposed to physical science, are more difficult and often less reliable because human beings are difficult to control. Using an example from social science, if I hypothesize that violence witnessed on television increases antisocial behavior among children, I can set up an experiment to test that hypothesis.   I can define an Experimental group of children based on age and gender criteria. I would then assign observers to watch for the number of antisocial behaviors exhibited in a 30 minute play session. 

Lets suppose that at Time 1 there were 14 instances of name calling, hitting, grabbing, and refusals to share (all these could be defined generally as "antisocial").    Then I could have the children watch 15 minutes of violent children's programming before allowing children to play together. Immediately after the t.v. viewing, I would set the group free to play for another 30 minute session and observe the number of antisocial behaviors for at Time 2.  The difference in the number of antisocial behaviors at time 2 minus the number at time 1 would be attributable to the effects of violent t.v. viewing.

Time 1 measurement                Treatment               Time 2 measurement
antisocial behaviors at time 1     violent t.v. episode    antisocial behaviors at time 2

The change in value of the variable of interest between Time 2 and Time 1 would be attributable to the Treatment.

To guard against other variables creeping into, and mitigating, my findings, I would probably want to have an Experimental group as illustrated above, and a Control group, which would consist of children who are similar in age and gender composition to the Experimental group, are measured twice at the times the Experimental group is measured, but are not exposed to violent t.v. programming.  This added set of data points allows the research to further show the effects of any unknown, or hidden variables not realized.  Here's how that works:

A change in antisocial behaviors (time 2 - time 1) for the experimental group is expected.  A change in antisocial behaviors (time 2 - time 1) for the control group is NOT expected.
Thus, to guage the effects of hidden factors, any change in the control group from t1 to t2 should be subtracted from the changes found in the experimental group differences - leaving only the effect of the treatment.  In equation terms: Effects of  violent t.v. = (exp. group time2 - exp. group time1) - (cont. group time2 - cont. group time1)

Experimental designs that include an experimental and a control group at the outset of the study are good for assessing the impact of events, the effectiveness of policy or program implementation because they allow the researcher to begin to evaluate the answer to the question, "Does this policy work?" - "Is this program effective?"     The experimental group outcomes show that something is changing the variable of interest.  The control group outcomes show that any change change in the variable of interest has not taken place because of the treatment factor.   For example, if a treatment or program is demonstrated to have the desired change on a specific behavior, while simultaneously showing that no change in behavior occurs without the treatment or program, the researcher is well on the way to making a strong case for continued funding of a program.  The research can say, with data to support the assertion, that this program is good for its subjects.

It is even possible, within the limits of logic, to extrapolate findings into the future. With good experimental data one can assert "savings" (i.e., we've saved 100 lives with this program in two years). Absent our program, 100 lives would have been lost between 1999 and 2001.  That would be 200 lives by 2003, 300 lives by 2005, 400 lives by 2007, and so on.  See  a Consumers Union Report on Increasing Speed Limits and Highway Fatalities at http://www.consumersunion.org/other/speedlimits/speed031500a6.htm

What we've been discussing here is the pretest-posttest experimental design with control group.  It is a true experimental design, and one that requires detailed planning before its implementation.  It is best used in relatively short duration experiments when human subjects are used, and not at all good for long term study.  Several threats to validity are always present in experimental designs.
These are:
  • history - events take place during the experiment that cloud findings
  • maturation - subjects develop (i.e., cognitively) during the experiment
  • testing - subjects become testwise - learning how to give appropriate responses
  • instrumentation - measurement failure or inaccuracy
  • statistical regression - the effects of subjects with extreme scores
  • selection bias - failure of random assignment of subjects to groups
  • experimental mortality - attrition of subjects from groups
Additionally, there are interaction effects between two or more of these threats.   Designs that work well will attend to all of these threats to insure valid conclusions.

Advantages & Disadvantages of using Experimental Designs

If applied carefully and cautiously, experimental designs afford the researcher control sufficient to theorize about cause and effect. With logic and timing well in hand, and by controlling for outside influences in the laboratory, experiments may provide insight into answering the question "what is the cause of change in variable X?"

It is important to keep in mind that the laboratory setup, while convenient, is artificial.  In nature, or the real world, there is rarely found the opportunity to see developing phenomena without an endless array of "influences" - and these influences are  likely - almost always - important aspects of the phenomena under study.  Thus, isolating a likely "cause" is only the first step in understanding. 

Another disadvantage for experiments is that only a limited share of possible research questions are amenable to the method.  Usually, Stanley Milgram's http://www.stanleymilgram.com/milgram.php now famous experimental study of obedience to authority is mentioned.  In this landmark study subjects were asked to apply electric shock to a protesting victim.  No one was actually harmed in these studies, excepting possibly the subjects themselves who were not made aware that the victims were actors.   Other ethical lapses can be found in social and medical science.  The Tuskegee Syphilis study is one such experiment in which a selection of infected black males were observed without medical treatment for their disease until their ultimate demise.  This study lasted from 1932 until 1972 http://www.cdc.gov/nchstp/od/tuskegee/time.htm.  Obviously, there are many problem areas of human social life deserving of study. Fortunately, there are other research methods to be used that would not violate basic human rights in the process of analysis. 

It is important to note that the simplest, most straightforward experimental design has been discussed here. Life is not always straightforward nor simple, and so it is with experiments in reality.  Before embarking on an experiment, the wise student will seek out further study, begining with Campbell and Stanley's monograph "Experimental and Quasi-experimental Designs for Research"  Rand McNally 1966.
Return to Syllabus