RCT Shows “Substantial Positive Effect” on Bottom Line-Served Students

By Kendall Cook, Graduate Data and Research Intern

In the spring of 2014, NCAN member Bottom Line launched a randomized control trial (RCT) to follow the classes of 2015 and 2016 for at least six years after high school graduation to document Bottom Line’s impact on students’ postsecondary outcomes. Preliminary results in the first of a series of evaluation reports show that students receiving Bottom Line services (“treatment group” students) were 7 percent more likely to enroll in college and 14 percent more likely to enroll at four-year institutions than students not served by Bottom Line (“control group” students). Treatment group students also attended colleges with significantly higher average graduation rates and lower cohort default rates than their control group peers.

Andrew Barr, an assistant professor of economics at Texas A&M University, and Ben Castleman, an assistant professor of education and public policy at the University of Virginia, are conducting Bottom Line’s program evaluation. Their methodology is both important for and rare within the college access field. Although many college access programs tout higher application, enrollment, persistence, and enrollment rates in the students they serve, these outcomes are often difficult to separate from the characteristics – motivation, college aspirations, and academic ability – that may have made them candidates for a college access program and more likely to apply to and attend college in the first place.

RCTs, which are considered the gold standard in program evaluations, are rigorous and make it possible to isolate a program’s effect on students’ outcomes from those students’ inherent characteristics. RCTs do this by randomly selecting which students are in the treatment and control groups. In this case, the evaluators worked with Bottom Line staff to determine which applicants from the class of 2015 were eligible to be served by Bottom Line. Then a computer randomly flipped a coin to put students in one of two groups. The evaluators later confirmed randomization by comparing the characteristics of students in the two groups. Because the two groups are very similar, except for the receipt of Bottom Line services, the outcomes of the treatment group show the effect of those services and the outcomes of the control group are an approximation of what students’ outcomes would have been without those services.

Bottom Line, an intensive college advising program that operates in Boston, Worcester, New York City, and Chicago, offers an access program that helps students enroll in college and a success program that helps students persist in college. Bottom Line advisors meet several times per month with students and provide guidance on college applications, financial aid, and college choice. Once students are on campus, “advisors help with course and major selection, making use of campus-based resources, and engaging in campus social activities,” describes the report.

In initial surveys, administered  in the spring of the senior year of high school and the following fall, Bottom Line students reported applying to more colleges than their peers and were nearly 30% more likely to report having met with someone to review financial aid award letters while make their college decision. Additionally, treatment group students reported both higher rates of participation in on-campus students groups and feeling more comfortable on campus than students in the control group.

In addition to the surveys, the evaluation report draws on Bottom Line applications and National Student Clearinghouse (NSC) enrollment data.

Moving forward, the evaluators will repeat the above process with the high school class of 2016 and also continue to monitor the 2015 cohort’s persistence rate and other outcomes. Beyond that, the evaluators “hope to obtain additional sources of administrative data to evaluate important outcomes like workforce participation and earnings” along with other outcomes like “career trajectory and aspirations, family formation, residential patterns, and overall satisfaction with the course of their lives.”