This Week in Student Success

Was this forwarded to you by a friend? Sign up for the On Student Success newsletter and get your own copy of the news that matters sent to your inbox every week.

It has been a minute since I wrote one of these posts. So what’s been happening in student success? Quite a lot, apparently: institutions are trying to figure out what to do with AI, predictive analytics may be changing the outcomes they are supposed to predict, a free-college program made graduation rates worse by some measures while producing more graduates, and we have 14 years of evidence on ASAP.

The more things change

EAB has a new report on student success in an era of constraint, based on a survey of 459 student success leaders. The title is well chosen. Institutions are under growing pressure to improve retention and completion while simultaneously dealing with budget constraints, staffing shortages and difficulties coordinating student support across the institution.

The data show a field that is fairly clear about what it wants to achieve, has been trying to do much of it for years, and still struggles with the organizational capacity to do it consistently. AI adoption is slow, but to the extent it is being adopted, it is being layered onto that existing operating model rather than prompting much rethinking of the model itself.

Take institutional priorities. Retention remains overwhelmingly at the top, followed by familiar goals around persistence, academic progress and student engagement.

The fact that AI support for student success is last is notable. These aren't entirely comparable categories—retention is an objective while AI is a means of achieving one—but the fact that it is so low a priority is still striking given the enormous amount of attention AI is receiving elsewhere in higher education.

And when institutions are using AI for student success, they aren't generally doing anything particularly revolutionary with it.

The most common uses include things like drafting student communications, chatbots and automating repetitive tasks. The field has discovered generative AI and apparently decided: “Excellent. Now we can write that advising email faster and with fewer typos!”

There is nothing inherently wrong with that. Saving advisers time is useful, and some of these applications may ultimately prove quite valuable. But it also tells us something about how institutions are approaching AI. So far, they appear much more interested in using it to make the existing student-success operating model more efficient than in asking whether AI makes a different student success system possible.

That matters because the existing model isn't exactly firing on all cylinders.

Despite the enormous amount of attention institutions have devoted to retention and student success, a substantial share still report falling short of the goals they set for themselves. And this is where I think the survey gets particularly interesting. Asked what is preventing greater progress, respondents point to the budget, but surprisingly they don’t seem to be that worried about inadequate technology.

Instead, they point to problems such as coordination across departments, resistance to change, leadership alignment, staffing capacity and difficulty engaging students.

These are the same sorts of organizational problems that have bedeviled student-success efforts for years. Better technology and better data may help institutions address some of them, but they don't make them disappear.

And that seems particularly important as we start talking about agentic AI in student success. There is an understandable tendency to focus on what the agents will be capable of doing: identifying students who need help, answering questions, coordinating services, making recommendations and eventually taking actions on students' behalf.

But many of the hardest problems identified in this survey aren't really problems of technological capability. They are problems of institutional operating capacity.

If an institution struggles to get advising, financial aid, faculty, student affairs and other units to coordinate around a student today, adding an AI agent doesn't automatically solve that problem. In fact, giving an agent the ability to act across those same organizational boundaries may make the unresolved problems even worse. Who has authority to act? Which unit owns the workflow? What happens when policies conflict? Who is accountable when the agent gets it wrong?

The interesting question may therefore be less “What can AI do for student success?” and more “Is the institution organized to make useful use of what AI can do?”

The problem with predictive analytics

I have plenty of complaints about predictive analytics in student success. They are one of those “shiny” things that people seem irrationally to love, causing them to sometimes treat prediction as the goal rather than asking what happened and why, as well as jumping over the descriptive and diagnostic work that ought to precede it. The familiar Gartner analytics hierarchy is old but still useful on this point.

But a LinkedIn post this week illustrated an additional problem with predictive analytics.

I think the image makes two important points about how we use predictive analytics in student success.

The first is relatively simple: a prediction is not evidence that the predicted outcome will occur. An at-risk flag, for example, is an estimate based on patterns in existing data. However accurate the model may be, the prediction itself is not proof of anything about an individual student.

The second point is more complicated. Once we act on a prediction, we create what the image describes as a loop: prediction → expectation → decision → outcome. The initial prediction can shape how we think about the student, which can shape the decisions we make, which in turn can shape the student's eventual outcome, which may then appear to confirm the original prediction.

The danger comes when we then treat that outcome as confirmation of the original prediction or of the intervention that followed it. As the image puts it, “the system may seem right because people acted like it was.” The outcome has not occurred independently of the prediction; it has emerged from a process partly set in motion and shaped by that prediction.

That doesn't make the outcome meaningless. But it means we need to question every step in the loop rather than treating the eventual outcome as validation of the whole process. Was the prediction accurate? What expectations did it create? What decisions followed? How did those decisions affect the outcome?

Prediction is not proof, and an outcome is not necessarily validation.

When worse graduation rates mean better student success

A new NBER working paper looks at the long-term effects of Tennessee Promise, the state's free community college program that fills in any gap between full-time tuition and fees and a student’s other sources of grant aid. Promise is often discussed as a college affordability policy, but the researchers find that it also substantially changed where—and whether—students enrolled in college.

The program increased postsecondary enrollment at two-year institutions by about 5.4 percentage points. Importantly, the authors estimate that roughly three-quarters of the increase came from what they call “New Access Enrollees”—students who otherwise would not have enrolled in college immediately after high school.

That is an important finding in itself. But the finding I find much more interesting is what happened to one of the standard ways we measure whether community colleges are successful. Their graduation rates got worse.

That sounds like a fairly damning student-success result. Except that at the same time, the colleges were producing more graduates. Associate-degree production increased by roughly 15%. The number of students transferring out increased by approximately 24%.

And when the researchers follow individual students rather than looking at institutional graduation rates, they find that Promise increased associate-degree attainment. By ages 23–24, students exposed to Promise were 4 percentage points more likely to have earned an associate degree or higher. There is also little evidence that this came at the expense of bachelor's-degree attainment.

So we have the wonderfully counterintuitive result that community colleges had lower graduation rates while producing more community-college graduates. The explanation makes sense once you look at who Promise brought into the system.

Many of the additional students were students who otherwise wouldn't have attended college at all. Those marginal students were less likely to complete than the students community colleges had previously enrolled, putting downward pressure on the institutional graduation rate. Other students transferred to four-year institutions without first earning an associate degree—another potentially successful student outcome that can hurt the originating college's graduation metric. The authors explicitly identify both mechanisms as explanations for the apparent contradiction.

And this is why I think this paper matters beyond Tennessee Promise.

We use graduation rates constantly as measures of institutional student success. But a graduation rate is a ratio, and changes in whom an institution serves can change that ratio in ways that have surprisingly little to do with whether the institution is producing more successful outcomes.

Expanding access changes the denominator. Successful transfer can hurt the numerator. An institution can therefore produce more graduates, more transfers and greater individual educational attainment while simultaneously reporting a worse graduation rate. Neither measure is necessarily wrong. They are answering different questions.

• The graduation rate asks something like: What proportion of a particular group of students entering this institution completed a credential here within a specified period?

• Degree production asks: How many students actually earned degrees?

• And the student-level analysis asks the question I ultimately care about more: Did the policy increase the likelihood that an individual student attained a credential?

In this case, those questions produce very different answers. That is a useful warning whenever we use institutional performance metrics to judge student success.

Quick, adopt the model ASAP

SUNY recently announced another expansion of its student success programs Advancing Success in Associate Pathways (ASAP) and its four-year counterpart, Advancing Completion through Engagement (ACE). The programs will now operate at forty-four SUNY institutions, with a goal of serving 10,000 students this year.

ASAP originated at CUNY and has since been replicated at institutions in Ohio, Colorado, Arizona and California. The basic model combines a fairly intensive package of supports: dedicated advising and career guidance, financial assistance for expenses such as textbooks and transportation, priority registration and access to high-demand courses, and sometimes block scheduling. Students, in turn, generally commit to full-time enrollment and other program requirements.

And there really isn't much reason left to argue about whether ASAP improves degree completion, because it does.

The Ohio replication, for example, found that after six years 44% of ASAP students had earned a degree compared with 29% of control-group students. Fourteen percent had earned a bachelor's degree compared with 9% of the control group.

But we now have something even more valuable: Fourteen years of follow-up data from the original randomized CUNY evaluation.

The chart is fascinating because it shows both the power and the limits of looking at short-term graduation rates. At the end of the three-year ASAP program, 39.5% of the program group had graduated compared with 22.4% of the control group—a huge 17.1 percentage-point effect. The gap then narrowed as some students in the control group eventually graduated. But it never fully disappeared. After 14 years, 57.7% of students offered ASAP had earned a degree compared with 50% of the control group.

The researchers interpret that pattern as evidence that ASAP did two things. It helped some students graduate faster than they otherwise would have, but it also appears to have caused some students to graduate who would never have earned a degree without the program. That is an impressive result.

But there is another finding in the 14-year study that complicates the story. In the final two years for which earnings data were available, ASAP students earned about $36,000 annually compared with about $38,000 for the control group. That difference was not statistically significant, so we shouldn't interpret it as evidence that ASAP reduced earnings. The finding is that researchers could detect no earnings benefit despite a persistent increase in degree attainment. That is particularly interesting because the six-year Ohio study had found something different. By Year 6, Ohio ASAP participants were earning $1,948 more than the control group, an increase of about 11%. The authors attribute this in part to the kinds of majors students chose.

I don't think the new 14-year CUNY finding means ASAP failed. Degree completion is an important outcome in its own right, and ASAP produced a large and remarkably durable improvement in it. But it does complicate the tendency to move casually from more graduates to better economic outcomes. Those are two different claims and require two different pieces of evidence.

It also makes me think again about my longstanding concerns about ASAP.

I have previously criticized the model for its cost, its focus on full-time students, and the difficulty of determining which parts of a large bundle of interventions actually produce the results. Those concerns remain. But increasingly I think the more important issue is scale.

SUNY's latest announcement is a useful illustration. Its goal is to serve 10,000 students across 44 institutions. That sounds like a lot—and for the 10,000 students involved, it certainly is. But it works out to an average of about 227 students per institution in a system enrolling well over 300,000 students.

A targeted intervention can be highly effective without being a scalable operating model for student success. And there is a risk in allowing the success of boutique interventions to distract us from the much harder problem of improving the experience of the entire student population.

There will always be students who benefit from intensive programs and specialized support, and we should absolutely provide those programs when the evidence supports them. But the larger student-success challenge is figuring out how to remove barriers and improve the educational experience for all students, including those who cannot enroll full time or participate in a highly structured program.

This is also why I get nervous when a particularly successful intervention becomes a “model” that institutions are encouraged to replicate. ASAP was developed for particular students, with particular participation requirements, backed by a relatively intensive and expensive bundle of services. Evidence that it works is extremely valuable. But that doesn't mean the answer to student success everywhere is ASAP, any more than it is Guided Pathways, the National Institute for Student Success (NISS), or whatever model comes next.

Evidence that something works is the beginning of the strategy question, not the end of it. We still need to ask: for whom, under what conditions, at what cost, and at what scale?

Musical coda

Since writing the Tennessee Promise section I’ve had this song playing in my head. A very young, spiky-haired Rosanne Cash. And if you ever get a chance to see her live, “slip away and put your jewellery in hock” and do whatever it takes to go see her.

And yes, her version is better than her dad's.

Evidence that sharing this newsletter improves student success remains limited. But why wait for the RCT? Forward it to a colleague.

Thanks for being a subscriber.