Looking forward and looking back – highlights of a year of Graduate Outcomes

It is hard to believe that it’s been just over a year since I joined Jisc as the new Head of Surveys. As a self-confessed survey geek with 20 years of experience in HE, I am truly thrilled to be here.

Prior to arriving at Jisc, I spent 12 years at the University of Oxford, where among many various duties I analysed both DLHE and Graduate Outcomes data. I believe the provider experience gives me a unique insight into challenges that providers face with the survey but also the opportunities awaiting.

Still, being on the other side now, I realised I was a bit naïve and did not fully appreciate how big and complicated the Graduate Outcomes survey is. It has been a steep learning curve but the support I received from everyone at Jisc was second to none. In 2015, HESA embarked on a review of DLHE and Graduate Outcomes was born a few years later after a lot of hard work and many challenges. Change is never comfortable even when necessary. 

However, the hard work has paid off - Graduate Outcomes data has recently been awarded the kitemark of ‘Accredited Official Statistics’ from the Office for Statistics Regulation (OSR). The glowing report is a true testimony to the teams involved, and the scrutiny, meticulous attention to detail and quality control that goes into the survey every day. 

Our response rate is our strength

It is fair to say, that probably the biggest concern the providers have had with the new survey is the decrease in response rate compared to DLHE. This is why I am especially pleased with the OSR assessment stating that the response rate is one of the “strengths” of the survey, something we have known all along. The Home graduate response rate - 51% in year five -  is on par with other major social surveys in the UK and globally:

• Graduate Outcomes Survey Australia – 39% 
• National Graduate Survey Canada – 46%   
• Labour Force Survey – 15% 

It is important to note, that response rate is only one of many measures of data quality. Our repeated analyses strongly and consistently suggest that the Graduate Outcomes survey data are highly representative of the UK graduate population at the sector level. Saying that, I am also aware that this may not be the case at every subject level for certain providers.

Likewise, the international online response rate remains a challenge. However, despite the lower response rate, there is no other dataset available that can even remotely match the volume and quality offered by the Graduate Outcomes survey.

These are the reasons why we are constantly looking for solutions that will boost our already high response rate, while keeping the costs down. This means less reliance on the calling and more emphasis on the online mode of the survey. 

The National Student Survey (NSS) achieved a 71% response rate in 2023 and almost 80% of responses were submitted online (40% of our responses are submitted online and 60% from calls). Admittedly it is much easier to survey students who are still physically at their provider  and have a valid provider-supplied email address, than graduates 15 months after they left. If you are a final year student it is almost impossible not to be aware of the NSS.

I believe that raising awareness should be a priority. In short, we need to make graduates more aware that the survey is coming their way and communicate its purpose. The survey invitation email is delivered to over 99% of provided email addresses but only 27% of Home students click on the survey link. Moreover, almost 30% of graduates do not engage with the survey at all, regardless of the channel. 

Plenty in the pipeline

We launched the new eye-catching and purpose-driven Graduate Outcomes survey brand precisely to increase awareness and engagement. We have increased the number of our social media channels to four (as this is where graduates will do their research), we are running paid social media adverts to boost awareness and working with strategic partners to increase legitimacy and relevance. The graduate facing website is also getting an overhaul in the coming months.

We are constantly tweaking our engagement strategy for the maximum impact and are currently exploring the use of RCS (Rich Communication Services) to make our survey text messages look more trustworthy and enticing. We have joined forces with the Student Loans Company to exchange contact details, hoping this can increase our engagement. We await with anticipation the legislation change coming later this year, which will allow us to gather more diagnostic information on our email targeting. We are also conducting an experiment to see if delaying calling at the beginning of the cohort has any positive effect on the online response rate. This summer, work will commence on potential use of incentives, with an incentives pilot in year 7. 

There is also a separate strand of work addressing the decline of the international response rate, particularly from Chinese graduates. We are exploring using alternative channels to deliver the survey link, i.e. WeChat and we are looking at developing some optional questions for the international cohort that would make the data more useful to providers with a particularly large international population. For this work we have partnered with the British Council, UKCISA and the AGCAS International task group. 

The ideas and drive coming from the team at HESA is fantastic and I look forward to updating you on these developments.

Collaboration is key

The bottom line is, however, that we are not going to succeed on our own. Providers have the strongest call to action when it comes to promoting the survey, not to mention direct access through trusted channels of communication. As someone who spent many years at a provider, I understand that these efforts can sometimes be disjointed. We would like to support providers in creating their holistic approaches to the survey. But first and foremost, there are two things that every provider should concentrate their efforts on: the quality of their contact details and promoting the survey to their students and graduates. 

In the months and hopefully years to come, I look forward to supporting providers in their journey. In the meantime, we look forward to this time next week (Thursday 13 June) when HESA releases the accredited official statistics from the class of 2021/22.