Designing an At-Risk Prevention Study using Supplemental Instruction
Dr. Steph is worried about the many students who are at-risk of failing her introductory psychology course at City College. She knows that almost 60% of students drop out of City College and most never obtain a degree. And even though 1 in 6 CCNY graduates is a psychology major, her course has a very high dropout rate (second only to calculus).
Thus, she wants to implement some intervention that will help at-risk students pass her course. She decides on a system of supplemental instruction, or SI, in which trained college graduates hold special weekly sessions to review course material and focus on problem areas. Dr. Steph wants to conduct a research study to evaluate the effectiveness for the at-risk students of SI in her course.
You have been hired as a consultant to submit a research proposal to Dr. Steph, taking all her concerns into consideration. She wants to compare students who utilize SI with those who don't, but she's worried that if she intentionally withholds SI from some students (say, in the control group) it may cause them to fail the course.
Also, she's not sure how to operationally define at-risk students or course success. She wants to ensure that at-risk students take advantage of SI, but many of them leave campus after class for outside work and so can't attend the SI sessions. Also, some attend SI sessions less consistently than others.
In fact, she's found in the past that it's actually the better students, and not the at-risk students, who tend to use SI, perhaps because poor students are reluctant to admit they have a learning problem. Finally, Dr. Steph thinks that if SI is made unavailable to some students (control group), they may seek help in other ways, such as paying more visits to their teaching assistants.
She wants you to design the study, operationally defining the variables, and controlling for effects of (1) outside work, (2) TA visits, (3) reluctance of at-risk students to attend SI session, and (4) intentional withholding of SI in the control participants.
Your job is to write a research proposal with 4 headings: Introduction, Methods, Results, and Conclusions. Within each heading, answer each of the following questions.
For the Introduction, you will need some background material found in this newspaper article: "Who Gets to Graduate", written by Paul Tough, and published in the New York Times on May 15, 2014.
1. Introduction.
a. Certain groups of students in college today are at much higher risk of course failure and drop out than others. Using data from "Who Gets to Graduate," describe two big risk factors in college retention and graduation.
b. Do SAT scores predict college graduation? How likely is a poor student with high SAT scores to graduate compared with a rich student with low SAT scores?
c. Describe one intervention from the article used at universities to address college failure and dropout of at-risk students.
d. What is the hypothesis being addressed in the study you are designing?
2. Method.
This section should outline how you plan to go about doing your research.
a. (i) Describe the participants. Define the target population? How are participants selected and recruited for the study? Would you sample the target population randomly? Why or why not?
(ii) How do you plan to address Dr. Steph's concerns about the reluctance of at-risk students to participate? How many groups will you test? How many individuals will you test in each group? Would you use random assignment of participants to groups? Why or why not? (6 pts)
b. (i) Describe the details of the SI intervention used in the experimental group. What intervention (if any) would you give to the control participants? How would you address Dr. Steph's concern about intentionally withholding treatment from the participants in the control group?
(ii) How do you plan to control for the following three variables: (1) TA visits, (2) outside work, and (3) inconsistency of attendance in SI sessions. Name two other variables that you would need to control in this study and describe how you would do that. (6 pts.)
c. How would you operationally define the dependent variable? Dr. Steph believes that attendance at SI sessions will have several beneficial effects. Describe three possible outcome measures.
d. What is the independent variable? Name all of the control variables.
3. Results
Below is a reproduction from an earlier version of the textbook (Myers 2006, p. 460). Here, a researcher wishes to compare how well daisies grow in two types of soils: poor soil and fertile soil. She hypothesizes that the average daisy grown in the fertile soil will be taller than the average daisy grown in the poor soil. The difference in average height between the daisies grown in poor soil and those grown in fertile soil is called the "difference between groups." But even if she finds a big difference between groups it may not be statistically significant.
That's because to be statistically significant the difference between groups needs to be about twice as large (actually 1.96 times as large) as the "variation within group." If a group of daisies are all raised in the same kind of soil (e.g., poor soil), some of the daisies in that group will grow a bit higher than others. This is the "variation within group." If the variation within group is as big as the difference between groups then that difference is probably just due to chance.
A statistical test helps us decide whether the observed difference between two groups is real or simply due to chance.
a. Describe how you would perform a statistical test on the experiment you designed for Dr. Steph. How would you calculate the difference between groups?
b. How would you calculate the variation within groups?
c. Define statistical significance. What would lead you to conclude that the difference between groups is statistically significant? (see pp. 39-40 of the textbook to understand the concept of statistical significance).
4. Conclusions
Dr. Steph is worried that making a large investment in an SI program will waste money if it fails to show a difference between the experimental and control groups.
So, before launching a full-scale study, she decides to conduct a small pilot study in which she asks two at-risk students who visit her after class each day if they would volunteer for intensive SI help (experimental group) and compares their course performance to two other students selected at random (control group). Dr. Steph is overjoyed when she sees the results depicted in the figure below.
a. Would Dr. Steph be correct in concluding that after undergoing SI any at-risk student is at least as capable of passing her course as any student not at risk? Support your position.
b. How is it possible that the at-risk pilot participants actually have much better course performance than the comparison pilot participants?
c. Consider the counterclaim that SI had nothing to do with the difference between groups in the pilot study. Name three extraneous variables that may have been the real source of this difference?
d. Give two recommendations for improving Dr. Steph's pilot study.
