This Week in Student Success

Was this forwarded to you by a friend? Sign up for the On Student Success newsletter and get your own copy of the news that matters sent to your inbox every week.

What happened and what did I read this week on student success?

This week offered four reminders that choosing the wrong measure, or measuring the wrong thing entirely, can easily lead us to the wrong conclusions.

The part-time comeback

Buried deep in the latest National Student Clearinghouse report is a surprisingly hopeful trend. Persistence and retention for part-time students have improved year-over year after almost a decade of decline.

But across all institution types, persistence and retention are noticeably up for part-time students for all institution types but especially for 4-year institutions of all types.

4-Year public institutions saw gains this year of 3.2% and 2.9% for second fall persistence and retention.

4-Year private institution part-time persistence and enrollment rates were up a similar amount. Part-time persistence and retention rates even improved year-over-year at 4-year for-profit institutions, albeit from a lower base level.

By contrast the uptick in persistence and retention for students at public 2-year institutions was more modest.

In all institution types these upticks in persistence and retention rates for part-time students reverse a long trend of declining rates since 2015. For years the story of part-time students has been one of slow decline. This is the first evidence that story may finally be changing.

Accidental pedagogy

Most OER debates ask a simple question: do open textbooks impact student finances? A new AAC&U study asks a much more interesting one: under what circumstances do they improve student success?

It is a really good study, based on a huge dataset and using mixed methods. The authors conclude OER should qualify as a new High Impact Practice. I came away convinced of something different: the real intervention isn't open textbooks, it's better teaching.

OERs and student success

Initially they seem to prove their point. They find that OERs are correlated with student success measured in terms of withdrawal rates which fall from 8.6% to 7.1% with the use of OERs. They examine these in terms of institution type but also learner type, using the concept they call Composite Learner Complexity (CLC)—a measure incorporating part-time status, Pell eligibility, first-generation status, age over 25, and race/ethnicity.

There is no single OER effect. It varies substantially by institution type.

Implementation beats adoption

The heart of the report is about implementation. To measure this, the researchers separated implementation into three basic modes: no OER, simple adoption and more complex adoption and creation where OERs were revised, remixed, or created by the instructor.

Higher-level implementation generally produces better outcomes than merely swapping textbooks, particularly in community colleges and regional public universities.

Unfortunately the report does not always fully explore the interesting issues that the chart raises.

Doctoral universities. There is a perverse result where higher-level implementation of OERs leads to higher withdrawal rates than more simple implementations. They explain this as a consequence of a poor sample and an especially tough class. Highlights mine

Once the authors realized a single gateway mathematics course dominated the higher-level implementation category, I would have expected them to rerun the analysis without it. Without that, the doctoral findings are difficult to interpret.

Regional public universities. Adopting an OER textbook increased withdrawal rates at least for low complexity learners. The researchers dug into the data and found that the courses where withdrawal rates went up were predominantly non-STEM courses. They speculate that this is because of cost and because STEM textbooks are generally higher cost than non-STEM textbooks.

This is pure speculation and suggests to me that the authors have not recently for example been in the business section of a college bookstore.

The wrong intervention

These issues and frustrations aside, the findings that in general more complex implementation leads to lower withdrawal rates at least in community colleges and regional public universities suggests to me that something else is going on rather than OERs. Different pedagogy appears to lead to lower withdrawal rates.

The authors describe this process themselves. Faculty with more complex implementations of OER made many more changes to their courses than faculty who simply adopted an OER textbook.

Better pedagogy, better outcomes. You could knock me over with a feather. What if OER isn't the intervention at all? What if OERs are a form of accidental pedagogy, simply prompting instructors to redesign their courses? The real high-impact practice may not be open textbooks. The real HIP may be course redesign.

Wirecutter headlines from the revolutionary war

On Saturday it is of course American Independence Day. To celebrate the occasion, McSweeneys wrote a list of Wirecutter headlines from the American Revolutionary War. For the uninitiated, Wirecutter is a New York Times service that has an often cult-like following. They rate and review consumer items using a very distinctive vocabulary. In my household, we aren’t allowed to buy anything without consulting it.

Here are my favorites from their list.

Beyond the player with the ball

While I must confess that I haven’t actually watched any of the World Cup football (all my sports time has been taken up by cricket), I have been fascinated to read about the use of data and analytics by football teams in the World Cup. Football as a sport has for a long time relied on data to improve team tactics and play (and there are some great books about this, far more interesting than Moneyball IMHO).

Footballers regularly wear trackers, there is a tracker in the ball and of course cameras tracking players, leading to all kinds of insights. But use of data has reached some new heights during this World Cup in part due to FIFA making a meaningful amount of that data public through the Enhanced Football Intelligence (EFI) platform. This public data layer gives all teams (and others) access to the data as well as the ability to query it directly rather than relying on experts.

The data goes beyond traditional stats and instead uses the numbers to create visualizations of how teams play. The system combines event data with tracking data from all 22 players, allowing analysts to understand the shape of the game rather than simply what happened to the player with the ball.

There are many lessons this use of data has for student success, but for now I want to focus on three.

Lesson 1: Context matters more than individuals.

Student success data still focuses overwhelmingly on the student. Football increasingly focuses on the interactions surrounding the player. A broader set of data about what is happening outside of the learner seems a fruitful avenue to explore.

Lesson 2: Flow matters more than snapshots.

Football analytics is shifting from counting events to understanding systems. Again this seems like a great evolution for student success data to make.

Lesson 3: Democratization has hidden labor.

Many of the accounts of World Cup data use make the argument that the public availability of the data has enabled a democratization of analytics in soccer, making it available and easily understandable. It turns out that a lot depends on your definition of democracy.

While the sensors produce millions of data points, real people are often required to annotate that data before it can be useful. This work often happens in developing countries.

We hear a lot of claims about data democratization in student success. It too is often more complex than it seems.

What gets measured gets exaggerated

Every few weeks another article appears claiming that students should skip university because some non-degree jobs can pay six figures. But the arguments about salaries in non-degree fields often obscure the fact that there is a wide variance in salaries, and most non-degree job earnings cluster at the lower end of the spectrum.

A post in Chalkbeat analyzed salaries for jobs where workers typically don’t have a degree (or low degree attainment jobs), those where roughly half of workers hold degrees (medium degree attainment) and where a majority of workers are credentialed (high degree attainment).

This is a useful corrective to the kind of overblown claims we sometimes see such as the one they quote from the New York Times “Job demand in fields like construction, along with the allure of potential six-figure salaries, have some high schools investing in hands-on classes that are redefining what success looks like for the Class of 2026.”

These discussions also tend to gloss over how physically demanding and sometimes dangerous many of these occupations are.

The analysis also lumps together occupations that probably shouldn't be treated as a single category. The post mentions some of this passing but never digs into the fact that child care workers and home health aides (which are usually the lowest paid non-degree workers) are different from skilled trades. The latter require extensive training and often a period of apprenticeship while the former (unfortunately) do not. Salaries reflect these differences. A good bit of this stems from the fact that they are looking only at whether a degree is required or not. Looking at actual occupations would be much more useful as these dictate salaries considerably more than the simple binary of whether a degree is required or not.

Whether a job requires a degree turns out to be a much weaker predictor of earnings than the occupation itself.

Musical coda

I saw the Australian band the Audreys in a very small venue once in 2007. The lead singer noticed a song had made me sad, reached out and squeezed my knee with her hand and reassured me it was all going to be OK. True story. They are also amazing.

The population of people who might enjoy this newsletter is almost certainly immeasurable. Rather than attempting a full census, I'll settle for a small random sample. Please forward it to a colleague who likes thinking about higher education.

Thanks for being a subscriber.