Presented online by Jennifer Lawrence, Program Director - Academic Success, at the University of New England. The sixth session in the Students First 2020 Studiosity Symposium, chaired by Prof Judyth Sachs.
Jennifer Lawrence: I'm going to talk about a study that started from my personal curiosity.
My role at UNE involves management of both our internal academic skills support area and the Studiosity external support and a number of other student academic support services. And I started getting questions from various elements of the university around things like a business case for renewed contracts. What was the efficacy of various programmes?
And so I started looking into data, as I'm want to.
Looking at the particular nature of student support at uni, there were a couple of areas that I was particularly interested in. We have external support services such as Studiosity. We have our internal academic skills support, which is one that I oversee the Academic Skills Office. We do have some peer tutoring programmes. These are quite limited at the moment. But we're developing and we have some support programmes we call Restart and Prep, which are about reaching out to students either in the early stages of their studies or to get them reconnected with their studies later on.
And so in the context of a study like this, I'm looking particularly at generalised academic support, particularly the first two, and really wanting to examine from an internal perspective what the impact of those services are:
Who's using them?
What impact it has on this success?
What is their attention across multiple trimesters?
Are we making an impact?
Are these services useful to our students?
Are they just reaching the students who would succeed anyway?
Are we actually making a difference?
So that's kind of where I started out.
So one of the first things I noticed when I was looking into this and the first thing that really caused me to go down this road was when I looked at the data, I noticed that there was only a very small group of students who accessed both services. So generally, students either accessed Studiosity help or they accessed academic skills. They didn't tend to use both.
"Students either access Studiosity help or they access academic skills. They didn't tend to use both."
Generally students picked one and stuck with it. And I actually went further into this and looked at whether or not students were changing from trimester to trimester - and they tended to stick with one in the first trimester of study. They might shop around a little bit. And then after that, they tended to stick with one particular service.
And that was that really the first point of curiosity to me. I wanted to know why that was what what was it that was influencing their patterns of seeking help from support service? Was it pragmatic reasons? Were they feeling like one was more accessible to them?
Firstly, we found that international students were much more likely to seek help from the academic skills office than from their Studiosity service. They tended to want that in-person contact. They tended not to book phone appointments or online appointments with the academic skills office. They tended to go into the office and book an appointment that way. But domestic students were most likely to use the Studiosity written feedback service.
Online students were much more likely than most to use the Studiosity written feedback. The on-campus students were much more likely than the online students to be seeking help from the Academic Skills Office. So I think that's about proximity and awareness because the academic skills of office are physically located on our campus.
The school leaver age group were more likely to seek help than mature students overall - the mature students tended not to to engage with support services so much.
So we see these subtle differences in the cohorts emerging.
The commencing students, as again, were much more likely to seek help than continuing students. So students are much more likely to seek help in the first year and then less likely after that as they get more confident.
And again, only a very small proportion of these students were switching services between trimesters or within trimesters. So they tend to pick one and then stick with it. They tend not to shop around.
So I wanted to know the reason students give for these choices. So I got responses from around 780 students at uni - or around five percent of our active cohort at that time.
And what really surprised me, is that students are mostly pragmatic.
That is, a lot of the findings at other institutions had been that students would act on friend or peer recommendation. Whereas with our mature online cohort, friend recommendations were not really a factor for them.
However, their previous experiences with a particular support service was important. Their awareness, the timing. So they're making choices based on very pragmatic factors, whether they thought the support was easy to use and navigate.
They didn't really care too much whether it was an internal service provided by uni staff or an external service. I thought that would be a bigger factor than it was.
And they didn't really care too much whether it was something that was focussed on subject material or more generalised academic support that tended to vary a little bit depending on what particular help they were seeking.
"So students are quite pragmatic in their choices, influenced by practicalities of time and accessibility."
And we know that the peak usage times of the Studiosity service and the times when they email into the academic skills office asking for written feedback tend to be about nine o'clock at night. And that seems to be when the kids are in bed after dinner, when everyone's gone to sleep. I'm going to sit at home and do my study at that point in time, and that's particular to that mature cohort.
Makes sense. Different types of students tend to seek help from different services, different sources. So again, we see the on-campus students behaving quite differently to the online students, the mature students behaving differently to the school leaders. International students behaving differently to domestic students.
Expectations setting for both staff and students is really key to their satisfaction with those services. So making it clear that this is the type of support you will get from this support service. This is the type of support you'll get from that support service becomes really critical.
So first I started to look at whether or not students access support - any support. Didn't matter what kind. So I break them into two groups, the group that access support, the group that didn't and looked at their marks.
And so this was using what's called a one way ANOVA test and analysis of variance. For those of you who may not be as statistically aware and this test essentially looks at the main approach across different groups of data and says, is it significant? Is the other differences between these two sets of numbers statistically significant? And you can see from the graph here already. "Yes, they did seek support" are in red. The "No, they didn't seek support" are in blue. And you can say just by looking at it that the red ones look higher.
So we can kind of eyeball it and say this looks pretty good, but eyeballing it isn't quite enough for me. So I ran the ANOVA test and in this case I was looking at over 100000 students. One hundred twelve thousand four hundred sixty five set that in n-number of sample size. And ultimately, the bigger sample size that we're looking at, the more confident we can be with the results.
"So we can kind of eyeball it and say this looks pretty good, but eyeballing it isn't quite enough for me."
In statistics, we look for a P value of less than point O five. So that normally indicates that there's a 95 percent chance or greater that the difference in grades is because of help seeking rather than just coincidence. And in this case, we got a P value of 0.007. So it's quite significant. Essentially, that means that we can be very sure that students who seek help do better.
"And in this case, we got a P value of 0.007. So it's quite significant. Essentially, that means that we can be very sure that students who seek help do better."
We know that students who seek support are more likely to be successful. It's positively correlated, but particular kinds of support aren't particularly making the difference there. It's just if they seek help, they do better. It doesn't really matter where they seek help from. And that, I think, for me, was really impactful.
There's a lot of conversations that take place at institutions around different types of support, particularly internal, external support. And what we're seeing here is that they don't tend to matter, that the impact on student grades particularly will be consistent across those if they're seeking help, regardless of where from, then they will do better.
Looking at 80,000 samples, I did a binomial test. Looking at persistence into one or two trimesters into the future was 0.03. So that's significant, but it's not quite as significant as what we saw previously. So we're reasonably sure that students who seek help are more likely to persist with their studies. Whereas with the previous tests, we were very, very sure that students who seek help were going to get higher grades.
"So we're reasonably sure that students who seek help are more likely to persist with their studies."
I'll often hear, 'is this particular type of support service worth it?' or 'is this particular type of support service worth it?' And those questions are often asked separately from each other. And what the data was showing me is essentially it doesn't matter the type of support service so long as the students do seek it.
And because we have quite a diverse cohort, they're seeking help from diverse support services. They're seeking help from a social service that is particularly useful to them for whatever reason.
A conventional support model, or the way people were were discussing it before, was to look at the student and say, "well, they could access this support service or they could access that support service. So we only really need one." And it was coming from a very one size fits all kind of mindset. And that didn't really fit with the data that I was seeing.
So I started proposing a slightly different model, what I'm calling an ecosystem of support services, to say - we have lots of different students of different types with different needs, and we need support services that are just as diverse as our student cohorts.
"...we have lots of different students of different types with different needs, and we need support services that are just as diverse as our student cohorts."
You can watch Jennifer's presentation or download the slides, here.
Or catch up on this year's Symposium sessions.
