Assurance of the employment and self-employment sections

Introduction

The quality of survey data is a high priority for Graduate Outcomes and the ongoing data quality assurance plan continued for the 21/22 (Year 5) collection, with the aim of assessing quality across all survey questions. This analysis was designed to present some of the preliminary information on the quality of general survey questions and to continue from investigations last year, where we highlighted that research could benefit from further work type splits. It’s also aimed at identifying further lines of investigation we may want to take to assess data quality across the Graduate Outcomes survey, for example, research last year opened up some more in depth projects for other survey questions.

In the following section, we will present some of the research covering the two employment sections of the survey. This includes ‘Section B’, which is also known as the employment section, and is answered by respondents in paid work for an employer or voluntary/unpaid work for an employer. It also covers ‘Section C’, which is referred to as the self-employment section and is answered by those in self-employment, running their own business or developing a creative, artistic or professional portfolio. The research from both of these sections will be presented in parallel, as the majority of the questions are identical in both sections, although due to routing the order can differ.

Employment Intensity

This review begins with the employment intensity question, which determines if a graduate was working full-time or part-time. This is the first question in Section C and for some graduates in Section  B. It’s worth noting that dependent on routing, the first question in Section B for some respondents will be the multiple jobs question instead. However, it is not included in the current review and has previously been assessed, with actions being taken separately to further analysis of the question.

Initially, item response rates to the employment intensity question are presented in Table 7.

Table 7: Item response rates to employment intensity in each survey section across years, split by completion mode

	CATI		ONLINE
	Year 4	Year 5	Year 4	Year 5
Section B	99.56%	99.51%	99.01%	98.91%
Section C	98.13%	97.70%	93.99%	93.47%

As can be seen in Table 7, response rates were high in both sections. Looking at completion modes, CATI showed higher item response rates than the online completion mode in both Section B and C.  When comparing Year 4 to Year 5, rates look very similar, with a very slight decrease in item response rates in both sections and in both completion modes. The lowest item response rate can be seen in Section C in the online completion mode across both survey years. This highlights a difference in response which is worth investigating, in order to determine if there is a particular group impacted, particularly for the online mode. As a result, the following analysis looks at the questions split by the type of work being selected. This was also highlighted as a useful line of investigation in last years analysis of Section B only.

Table 8: Item response rates to employment intensity in each survey section across years, split by the type of work and completion mode

	CATI		ONLINE
	Year 4	Year 5	Year 4	Year 5
Section B
Paid work	99.59%	99.54%	99.11%	99.00%
Volunteering	98.60%	98.96%	93.96%	93.74%
Both	99.52%	99.33%	97.61%	97.51%
Section C
Self-employed	98.01%	97.55%	94.51%	93.83%
Own business	98.32%	97.17%	94.35%	94.65%
Portfolio	97.98%	97.32%	91.57%	90.90%
Multiple selections	98.32%	98.26%	94.86%	94.20%

When looking at item response rates by the type of work being selected, it can be seen that in Section B the highest item response rate is for graduates who were in paid work and the lowest is for those in volunteering (Table 8). When both work types were selected, the item response rates were higher than volunteering but lower than paid work. In Section C, the two work types with the highest item response rates were for own business and those with multiple selections, and the lowest for portfolio. These findings are potentially as expected, as paid work for an employer is the role that is likely to relate most to a question about employment intensity and this has far higher rates of response to the question. In comparison, the volunteering rate is much more similar to that of the Section C rates, and all of these are activities where some respondents may feel it is not as relevant to them. Indeed, even within Section C, it appears that those in portfolio work may feel the question is less applicable. This may relate to the variety of portfolio activities that respondents could be referring to when selecting this option, that may not fit typical employment questions. However, interestingly, we see much less impact and variation in the CATI completion mode. This may be a result of intervention or reassurance from the interviewers. Further completion mode comparisons show that the lowest item response rates were in Section C, for those who selected portfolio online. In Section B, the lowest item response rate was for volunteering. Year comparisons showed a slight decrease of many of the item response rates, although changes are low and item response rates are continually tracked. The largest decrease was for Section C own business in the CATI completion mode.

 

Table 9: Proportion of employment intensity selections across years, split by survey section

	Section B		Section C
	Year 4	Year 5	Year 4	Year 5
Full-time	84.83%	84.79%	34.18%	33.39%
Part-time	15.17%	15.21%	65.82%	66.61%

These show that majority  of graduates in Section B were in full-time work (>84%) whereas the majority of graduates in Section C were in part-time work (>65%) (Figure 4, Table 9). This seems to align with expectations, but may also further support the idea that item response may be lower in this section as respondents may feel that the questions do not align with their view of the role. Year comparisons show a slight overall decrease in full-time proportions.

To further investigate this, proportions in each section are further split down by the type of work being completed.

Table 10: Proportions of employment intensity selections across years in Section B, split by type of work

	CATI		ONLINE
	Year 4	Year 5	Year 4	Year 5
Full-time
Paid work	85.25%	85.36%	88.69%	88.91%
Volunteering	19.40%	18.79%	27.40%	27.91%
Both	71.30%	70.84%	56.59%	58.94%
Part-time
Paid work	14.75%	14.64%	11.31%	11.09%
Volunteering	80.60%	81.21%	72.60%	72.09%
Both	28.70%	29.16%	43.41%	41.06%

Proportions in Section B show that the type of work with the most graduates in full-time work was paid work for an employer. Comparatively, volunteering had the highest proportions of respondents working part-time. When both work types were selected, there were more graduates in full-time employment, however this proportion was more prominent in CATI than the online completion mode. This could be that online respondents may be more likely to only select their main role, as we have identified in previous years, but could also be related to other factors such as social desirability bias and should be considered further.

The same analysis was performed on Section C, to determine if there were any differences.

Table 11: Proportions of employment intensity selections across years in Section C, split by type of work

	CATI		ONLINE
	Year 4	Year 5	Year 4	Year 5
Full-time
Self-employed	33.30%	31.85%	40.55%	41.11%
Own business	44.25%	40.96%	58.13%	56.59%
Portfolio	20.87%	21.14%	22.38%	23.30%
Multiple selections	36.50%	37.31%	38.85%	36.28%
Part-time
Self-employed	66.70%	68.15%	59.45%	58.89%
Own business	55.75%	59.04%	41.87%	43.41%
Portfolio	79.13%	78.86%	77.62%	76.70%
Multiple selections	63.50%	62.69%	61.15%	63.72%

 

Proportions in Section C show that the type of work with the most graduates in full-time work were running their own business, although these proportions are much lower when compared with Section B. Comparatively, the portfolio selection had the highest proportions of respondents working part-time. Overall, the work type proportions in Section C are more equally distributed than in Section B. Completion mode comparisons show that there were more graduates in the online completion mode that were in full-time employment when self-employed or running their own business. The main difference in between year comparisons was the reduction in full-time and increase in part-time own business proportions.

The differences seen between the different types of work and completion modes need to be considered in reviews of the survey to ensure ensuring respondent feel more confident in their selections. Item response rates will continue to be tracked.

Job title

Job title and job duties have the potential to be asked in both sections B and C, depending on the activities selected and the route taken by the graduate. Job title is a mandatory question that requires graduates to provide their job title during the relevant census week for their cohort. Responses are entered in a free-text field, so it is possible for graduates to provide a response that is not accurate. For example, they may edit their job title or enter random characters to bypass the question, due to influences such as privacy concerns or social desirability bias. 

Job title has previously been identified as a question that some respondents may view as sensitive, and as a result hover text was added which generally seemed to have a positive impact, although further research was needed into variations between completion modes. As a result, analysis this year across both years begins with an assessment of item response rates and expands upon previous analysis, with the aim of identifying any potential concerns.   

Table 12: Item response rates to the job title question across years, split by section.

	CATI		ONLINE
	Year 4	Year 5	Year 4	Year 5
Section B	99.26%	99.11%	93.47%	92.67%
Section C	99.04%	98.75%	88.53%	87.98%

As can be seen in Table 12, Section B item response rates were higher than in Section C. Overall, CATI showed higher item response rates than the online completion mode in both Section B and C. This could be due to the sensitivity of this question and the effect of an interviewer in CATI (Conrad & Schober, 2000; Schober & Conrad, 1997) making graduates more likely to provide information. It may also be helpful that they can ask for clarifications or that an interviewer may feel more confident entering a response that a graduate is unsure of. When comparing Year 4 to Year 5, there was a slight decrease in item response rates in both sections and in both completion modes and whilst the change is generally small, it will continue to be tracked closely. The lowest item response rate can be seen in the Section C online completion mode, which would potentially suggest that respondents are most unsure of responses for this section, or that it is more sensitive. In order to consider this further, splits by the type of work selected in each section are presented below. 

Table 13: Item response rates to job title in each survey section across years, split by the type of work and completion mode

	CATI		ONLINE
	Year 4	Year 5	Year 4	Year 5
Section B
Paid work	99.26%	99.09%	93.57%	92.73%
Volunteering	99.28%	99.02%	86.93%	86.90%
Both	99.35%	99.36%	93.98%	93.69%
Section C
Self-employed	99.09%	98.74%	89.00%	88.32%
Own business	98.93%	98.80%	86.65%	87.07%
Portfolio	99.07%	98.62%	87.39%	86.53%
Multiple selections	99.03%	98.92%	90.58%	89.65%

When looking at item response rates by the type of work being selected, it can be seen that in Section B the highest item response rate is for graduates who selected both work types (Table 13) and the lowest is for those in volunteering (particularly evident online). In Section C, the item response rates showed less variance, but also indicated higher rates for multiple selections and the lowest rate this year in the online portfolio group. For both portfolio and volunteering, this seems to align with expectations. Both are less likely to have traditional job titles applied than some of the other types of work. Own business is also a group which has lower rates (although slightly improved compared to Year 4), and may be particularly sensitive for business owners, or may not feel as relevant for those in the early stages of creating a business. When looking at online completion mode across both sections, the item response rates were lower, likely due to the sensitivity of the question and this being one of the first indications of more personal questions. Year comparisons showed a slight decrease of many of the item response rates, although changes are low and item response rates are continually tracked.

Job duties

Job duties is asked in the same section as job title and we would expect to see similar levels of response to both.

Table 14: Item response rates to the job duties question across years, split by section.

	CATI		ONLINE
	Year 4	Year 5	Year 4	Year 5
Section B	99.25%	99.10%	93.47%	92.66%
Section C	99.03%	98.73%	88.53%	87.98%

As can be seen in Table 14, item response rates were higher in Section B, as was the case with job title. Looking at completion modes, CATI showed higher item response rates than the online completion mode in both Section B and C.  When comparing Year 4 to Year 5, there was a slight decrease in item response rates in both sections and in both completion modes. The lowest item response rate can be seen in Section C in the online completion mode across both survey years. Rates are very similar to job title, as we would expect to see, although slight variation is more likely in the CATI mode as interviewers have the ability to remove or edit text at the end of the survey. As with job title, it is worth considering splits by the type of work selected.

Table 15: Item response rates to job duties in each survey section across years, split by the type of work and completion mode

	CATI		ONLINE
	Year 4	Year 5	Year 4	Year 5
Section B
Paid work	99.24%	99.09%	93.56%	92.73%
Volunteering	99.28%	99.02%	86.93%	86.90%
Both	99.35%	99.36%	93.98%	93.69%
Section C
Self-employed	99.08%	98.74%	89.00%	88.32%
Own business	98.90%	98.76%	86.65%	87.07%
Portfolio	99.06%	98.62%	87.39%	86.53%
Multiple selections	99.03%	98.90%	90.58%	89.65%

When looking at item response rates by the type of work being selected, it can be seen that the same trends exist as in job title, with only some slight variations in rates. This is reassuring and is as expected, and is likely due to the same reasons as discussed in the previous section. Once again, multiple selections across both sections had the highest item response rates and volunteering and portfolio had the lowest. Rates will continue to be tracked across years.

Employment basis

The employment basis question is only asked in Section B of the survey and records the basis of employment of the respondent.

Table 16: Item response rates to employment basis in Section B across years, split by the type of work and completion mode

	CATI		ONLINE
	Year 4	Year 5	Year 4	Year 5
Paid work	99.76%	99.73%	99.73%	99.69%
Volunteering	99.70%	99.65%	99.64%	99.78%
Both	99.76%	99.73%	99.58%	99.85%

Item response rates are high and show very little variation between the work types being selected, with only slightly higher rates in the online completion mode. Year comparisons show slight variations but no concerning changes.  To assess the question further, proportion of responses selected are presented below based on the work type.

Table 17:Proportion of employment basis selections for each work type, split by year

	Paid work		Volunteering		Both
	Year 4	Year 5	Year 4	Year 5	Year 4	Year 5
On a permanent/open ended contract	72.05%	73.67%	3.32%	4.22%	57.42%	59.01%
On a fixed-term contract lasting 12 months or longer	15.29%	14.75%	1.27%	1.23%	13.21%	13.47%
On a fixed-term contract lasting less than 12 months	4.49%	3.70%	1.62%	2.00%	6.10%	4.69%
Temping (including supply teaching)	1.59%	1.34%	1.55%	1.28%	2.35%	1.90%
On a zero hours contract	3.87%	4.06%	2.87%	3.57%	6.55%	6.75%
Volunteering	0.04%	0.03%	77.73%	76.77%	10.86%	10.69%
On an internship	0.76%	0.68%	3.66%	2.99%	0.98%	0.72%
Other	1.13%	1.07%	5.97%	5.71%	1.65%	1.75%
Not known	0.78%	0.70%	2.01%	2.22%	0.88%	1.02%

Proportion analysis show that the majority of graduates in paid work or both work types were “On a permanent/open ended contract”, followed by “On a fixed-term contract lasting 12 months or longer”. For volunteering, it’s important to note that since the fourth year, auto population is in place where the main activity is volunteering or unpaid work, at the request of stakeholders. This is clear when considering that such a large percentage are in this group, and the next largest % being “Other” indicates that these roles are less likely to fit into traditional contracts. There could be some graduates who would consider their contract in a different way or who have multiple activity selections, and we can see this in the responses of those who selected both work types. It may be worth considering whether auto population suits all needs. Year analyses show that overall there was an slight increase in proportions of graduates  “On a permanent/open ended contract” and “On a zero hours contract” with decrease in all other selections. In order to consider this question more carefully, additional proportion analysis was completed in relation to work intensity.

 

Table 18: Proportion of employment basis selections for each employment intensity split by year

	Full-time		Part-time
	Year 4	Year 5	Year 4	Year 5
On a permanent/open ended contract	74.24%	75.73%	46.93%	48.87%
On a fixed-term contract lasting 12 months or longer	16.08%	15.49%	8.49%	8.51%
On a fixed-term contract lasting less than 12 months	4.02%	3.35%	7.22%	5.70%
Temping (including supply teaching)	1.22%	1.02%	3.86%	3.25%
On a zero hours contract	1.68%	1.82%	16.71%	17.27%
Volunteering	0.59%	0.60%	9.84%	9.98%
On an internship	0.77%	0.67%	1.13%	1.03%
Other	0.85%	0.83%	3.49%	3.24%
Not known	0.54%	0.49%	2.31%	2.16%

The proportions show that majority of graduates were “On a permanent/open ended contract” both in full-time and part-time employment. The second most common selection was “On a fixed-term contract lasting 12 months or longer” for full-time employment but “On a zero hours contract” for part-time employment. These seem to align with expectations and may mean that graduates are understanding the relevant options to select at employment intensity, for example.

Year analysis shows an increase in proportions for “On a permanent/open ended contract” for both work intensities and “On a zero hours contract” for part-time employment, with most of the other proportions showing a slight decrease.

Employment name

Employer name and duties are asked in both sections, but are not asked of those only completing a portfolio in Section C. They are sensitive questions which appear on the same page, in a similar way to job title and duties. As with these questions, they have had reassurances added to them in previous survey years with the aim of reducing drop-out. There has also been confusion in the past around the guidance provided that asks graduates working through an agency to supply the name of the placement employer, rather than the name of the agency they are working for. This has since been resolved through trialing of different text.  As these are free-text fields, respondents may answer the question in a way that avoids providing a sensitive answer. This cannot be picked up in item response rates, but these are useful in identifying drop out.

Table 19: Item response rates to employment name in each section across years, split by completion mode

	CATI		ONLINE
	Year 4	Year 5	Year 4	Year 5
Section B	97.29%	96.82%	94.54%	94.17%
Section C	99.24%	98.93%	94.20%	93.77%

As can be seen in Table 19, item response rates on the CATI completion mode were higher in Section C than in Section B. Item response rates were more equal between the sections in the online completion mode, with only slightly higher rates in Section B.  When comparing Year 4 to Year 5, there was a slight decrease in item response rates in both sections and in both completion modes which, although not concerning, will continue to be tracked. It’s clear to see that this is a more sensitive survey question, although rates don’t tend to be as low as the questions that are presented earlier in the survey. The lowest item response rate can be seen in Section C in the online completion mode across both survey years. Respondents may feel unsure of what to put for certain types of work in this section. As a result, the following analysis considers item response split by the type of work selected.

Table 20: Item response rates to employment name in each section  across years, split by the type of work and completion mode

	CATI		ONLINE
	Year 4	Year 5	Year 4	Year 5
Section B
Paid work	97.61%	97.13%	94.65%	94.30%
Volunteering	87.84%	88.24%	87.93%	86.20%
Both	96.51%	96.05%	93.95%	92.72%
Section C
Self-employed	99.54%	98.84%	93.99%	93.05%
Own business	98.81%	98.74%	93.58%	94.42%
Multiple selections	99.16%	99.04%	95.31%	94.71%

When looking at item response rates in Table 20 by the type of work being selected, it can be seen that in Section B the highest item response rate was for graduates in paid work with the lowest being for graduates who were volunteering. Indeed, volunteering item response rates were lowest across both sections and all work types. This may explain the lower Section B rates, with largely different results between the two work types. Perhaps this indicates that volunteers feel less comfortable providing this information or do not feel that it is relevant to them. Interventions for improvement in this group could be considered further. Completion mode comparisons show that that all item response rates were lower in the online mode. Year comparisons showed a slight decrease in many of the item response rates, whilst CATI Section B volunteering and online Section C own business show a slight increase.

Employment duties

Employment duties is asked in the same section as employment name and similarly to this, is not asked of portfolio only respondents in Section C. Both should be similar in terms of item response rates.

Table 21: Item response rates to employment duties in each section across years, split by completion mode

	CATI		ONLINE
	Year 4	Year 5	Year 4	Year 5
Section B	96.59%	96.04%	94.47%	94.11%
Section C	99.22%	98.92%	94.20%	93.78%

As can be seen in Table 21, response rates in the CATI completion mode were again higher in Section C than in Section B. Item response rates were more equal between the sections in the online completion mode, with only slightly higher response in Section B. Between employment name (Table 19) and employment duties (Table 21) there is a bigger variation in responses on CATI, likely as a result of interviewers ability to remove answers.

Splits by the type of work in each section are presented below for employment duties.

Table 22: Item response rates to employment duties in each section  across years, split by the type of work and completion mode

	CATI		ONLINE
	Year 4	Year 5	Year 4	Year 5
Section B
Paid work	96.97%	96.43%	94.60%	94.25%
Volunteering	86.14%	86.00%	86.84%	85.04%
Both	95.53%	94.98%	93.68%	92.31%
Section C
Self-employed	99.54%	98.84%	93.99%	93.07%
Own business	98.81%	98.74%	93.58%	94.42%
Multiple selections	99.13%	99.04%	95.27%	94.71%

When looking at item response rates by the type of work being selected, as we saw with employment name, it can be seen that the lowest item response rate was graduates who were volunteering (Table 22). Indeed, volunteering item response rates were lowest across both sections and all work types. In Section C, on the CATI completion mode, the highest item response rate was for self-employed graduates compared to multiple selections when in the online mode. Completion mode comparisons show that that all item response rates were lower in the online mode in general, as was the case for employment name. It will be useful to look into the differences between item response rates in employment name and duties further and continue tracking these over the year. 

NHS organisation

The NHS organisation question identifies whether a graduate’s employer is part of the NHS, and is only asked of graduates in Section B. It does offer a response option of “Don’t know” but is a mandatory question.

Table 23: Item response rates to the NHS organisation question in Section B  across years, split by type of work and completion mode

	CATI		ONLINE
	Year 4	Year 5	Year 4	Year 5
Paid work	99.86%	99.83%	99.89%	99.89%
Volunteering	99.80%	99.88%	99.94%	99.60%
Both	99.93%	99.85%	100.00%	99.92%

Item response rates to this question were high across all the work types and completion modes. Year comparisons show only a slight decrease in some of the work types, with volunteering showing a slight increase. Due to the presence of a “Don’t know” response option, selections are presented below to understand the trends in each type of work better.

 

Table 24: Proportion of NHS organisation selections for each work type, split by year and completion mode

	CATI		ONLINE
	Year 4	Year 5	Year 4	Year 5
Yes
Paid work	16.86%	18.29%	13.01%	14.15%
Volunteering	5.36%	5.53%	3.89%	5.80%
Both	11.88%	12.85%	9.26%	11.55%
No
Paid work	80.72%	79.36%	84.36%	83.18%
Volunteering	90.43%	90.37%	90.28%	88.55%
Both	85.17%	84.44%	87.56%	86.37%
Don't know
Paid work	2.42%	2.35%	2.62%	2.67%
Volunteering	4.21%	4.10%	5.83%	5.64%
Both	2.94%	2.71%	3.19%	2.09%

Proportion analysis shows that the majority of graduates selected that their employment was not part of NHS, which is as expected. Graduates who were in paid work had the biggest proportion who selected that their employment was part of NHS then followed by both work types. Volunteering had the biggest proportion of graduates who indicated “Don’t know”, perhaps as a result of the types of roles people volunteer in, or due to respondents having less knowledge of the role due to it being a volunteering position. Completion mode analysis shows that graduates in the online completion mode were more likely to indicate that their employment was not part of NHS, or that they did not know, although generally rates are similar. This could be due to various factors, but may also link to the reluctance of online respondents to provide information about employer. It could be useful to complete some analysis with comparisons to other free-text job and employment information, or with the coding assigned during the SIC and SOC coding process. Year analysis shows an increase in “Yes” proportions and a decrease in “No” and “Don’t know”, with the biggest decrease in “No” selections for the volunteering work type. 

Employment location

The employment location questions are mandatory for those in Section B and for those who select self-employment or own business options for Section C. It asks the location of the business and presents 6 options which will route to further questions depending on the answer provided. In some cases, graduates will be presented with a postcode and town/city list question and analysis of these can be found in the location data section of the quality report. Other respondents who are not in the UK will be routed to a more specific country question instead. Initially, item response rates are presented for the employment location question.

Table 25: Item response rates to the employment location question in Section B and C  across years, split by completion mode

	CATI		ONLINE
	Year 4	Year 5	Year 4	Year 5
Section B	99.78%	99.74%	99.81%	99.79%
Section C	99.77%	99.80%	99.88%	99.71%

Item response rates to the question were high overall in both sections, completion modes and years. Year comparisons show a slight increase in item response rates in Section C CATI, but a slight decrease in all other groups. The table below splits these by work type for further information.

Table 26: Item response rates to the employment location question in Section B and C  across years, split by type of work and completion mode

	CATI		ONLINE
	Year 4	Year 5	Year 4	Year 5
Section B
Paid work	99.78%	99.74%	99.81%	99.79%
Volunteering	99.67%	99.86%	99.61%	99.52%
Both	99.84%	99.73%	99.93%	99.84%
Section C
Self-employed	99.74%	99.80%	99.90%	99.72%
Own business	99.74%	99.80%	99.74%	99.69%
Multiple selections	99.81%	99.80%	99.96%	99.74%

When looking at the work types selected by graduates, reassurance is offered that item response rates are high across all. There were no major differences between work types in both sections, completion modes or years and it appears that generally the item response rates do not cause major concerns.

Further analysis below considers the proportion of selections in the employment location questions.

 

Table 27: Proportion of employment locations across each section and year

	Section B		Section C
	Year 4	Year 5	Year 4	Year 5
England	77.43%	79.01%	75.26%	78.07%
Wales	3.80%	3.89%	3.31%	3.57%
Scotland	7.25%	7.84%	5.96%	6.16%
Northern Ireland	2.96%	2.69%	2.26%	2.07%
Outside the United Kingdom	8.41%	6.43%	13.11%	9.98%
Channel Islands or Isle of Man	0.13%	0.15%	0.10%	0.15%

Proportion analysis between Section B and Section C shows that the most selected option was England in both sections, followed by Outside the United Kingdom. However, graduates in Section B were more likely to select England than in Section C. There is some consideration to be made regarding the order of the list, as England does appear first. Year comparisons show an increase in proportions for all selections apart from Northern Ireland and Outside the United Kingdom, which show a decrease.

To further this analysis, selections for employment location in Section B are split by the type of work and completion mode.

Table 28: Proportion of employment locations for each work type in Section B, split by completion mode and year

	CATI		ONLINE
	Year 4	Year 5	Year 4	Year 5
England
Paid work	81.49%	82.00%	72.38%	74.76%
Volunteering	73.81%	75.54%	61.44%	64.92%
Both	77.95%	79.11%	69.53%	73.76%
Scotland
Paid work	7.23%	7.66%	7.27%	8.10%
Volunteering	5.77%	6.39%	6.63%	7.72%
Both	7.74%	8.28%	9.71%	10.43%
Wales
Paid work	4.12%	4.18%	3.34%	3.41%
Volunteering	4.22%	4.10%	3.12%	3.90%
Both	4.04%	3.92%	4.00%	3.93%
Northern Ireland
Paid work	3.43%	3.04%	2.31%	2.10%
Volunteering	2.48%	1.83%	1.43%	1.99%
Both	3.87%	3.50%	2.00%	1.61%
Outside the United Kingdom
Paid work	3.60%	2.98%	14.55%	11.47%
Volunteering	13.62%	11.90%	27.18%	21.16%
Both	6.32%	5.02%	14.60%	9.71%
Channel Islands or Isle of Man
Paid work	0.13%	0.13%	0.15%	0.16%
Volunteering	0.10%	0.23%	0.20%	0.32%
Both	0.08%	0.19%	0.15%	0.56%

Proportions of answers given to the question in Section B show that the majority of graduates in all types of work selected England as their location, although this proportion was largest for the paid work group. Volunteering has the highest proportion selecting Outside the United Kingdom, with this as the volunteering groups second highest selection overall after England. This is interesting, and whilst it may relate to the type of work being completed and the location of volunteering opportunities, further analysis will be useful. Completion mode comparisons show higher proportions in CATI selecting England, with some of this also seen at a small level across other selections such as Northern Ireland. The biggest increase in online proportions (vs. CATI) can be seen for the Outside the United Kingdom selection in all work types, with the largest difference seen for paid work. Whilst this may look concerning, this is due to the ceasing of non-EU international calling, meaning these graduates can only answer the survey in the online mode.

In order to compare results across sections, selections for employment location in Section C are split by the type of work below.

 

Table 29: Proportion of employment locations for each work type in Section C, split by completion mode and year

	CATI		ONLINE
	Year 4	Year 5	Year 4	Year 5
England
Self-employed	81.98%	81.41%	68.90%	73.23%
Own business	78.78%	80.91%	59.18%	63.09%
Multiple selections	79.42%	81.23%	67.32%	73.07%
Scotland
Self-employed	5.57%	6.45%	5.84%	6.16%
Own business	6.28%	5.90%	4.89%	5.53%
Multiple selections	6.11%	5.97%	7.43%	7.23%
Wales
Self-employed	3.49%	3.51%	2.64%	3.66%
Own business	4.27%	4.11%	2.33%	2.87%
Multiple selections	3.53%	3.65%	3.42%	3.21%
Northern Ireland
Self-employed	2.46%	2.21%	1.80%	1.60%
Own business	2.30%	1.83%	1.81%	1.19%
Multiple selections	2.63%	2.59%	1.65%	0.90%
Outside the United Kingdom
Self-employed	6.44%	6.31%	20.68%	15.23%
Own business	8.33%	7.13%	31.61%	27.06%
Multiple selections	8.24%	6.38%	20.00%	15.43%
Channel Islands or Isle of Man
Self-employed	0.06%	0.10%	0.14%	0.12%
Own business	0.04%	0.12%	0.19%	0.27%
Multiple selections	0.07%	0.17%	0.18%	0.17%

Proportions in Section C show similar trends as in Section B. The self-employed group had the highest proportion of graduates who selected their employment location as England, followed by graduates who selected multiple work types. The second most selected option was Outside the United Kingdom, and this was most common for those who were running their own business.  Completion mode comparisons show smaller online proportions for England for all work types, with the smallest proportion for graduates who are running their own business. Online selections of Outside the United Kingdom are considerably higher than CATI, which aligns with Section B and is once again related to the ceasing of non-EU international calling.

Employment country

Graduates who select at the employment location questions that they are outside the United Kingdom or in the Channel Islands or Isle of Man will be routed to the employment country section. This is mandatory for certain routes, but is not asked of those in Section C who are only doing a portfolio. Item response rates are presented below.

Table 30: Item response rates to employment country for each section and year, split by completion mode

	CATI		ONLINE
	Year 4	Year 5	Year 4	Year 5
Section B	96.71%	96.66%	99.03%	98.94%
Section C	99.68%	99.93%	99.58%	99.26%

Item response rates to this question are generally high, however, it appears that the Section B CATI completion mode is lower than expected. In order to identify potential causes, work type splits are below.

Table 31: Item response rates to employment country for each section and year, split by completion mode and type of work

	CATI		ONLINE
	Year 4	Year 5	Year 4	Year 5
Section B
Paid work	97.84%	97.29%	99.10%	99.08%
Volunteering	87.73%	90.72%	96.54%	94.39%
Both	95.28%	97.07%	99.00%	96.24%
Section C
Self-employed	99.78%	99.78%	99.51%	99.10%
Own business	100.00%	100.00%	99.53%	99.35%
Multiple selections	99.55%	100.00%	99.82%	99.45%

When considering these splits, it appears that Section C has very high item response rates. In contrast, Section B has some areas which appear to be lower. Whilst the online mode and the paid work group are not overly concerning, the CATI completion mode appears to have lower levels of selection in general. Equally, the volunteering group within CATI and online is lower. This may be a result of survey fatigue, confusion if people have taken the wrong route, or various other factors. We will continue looking into this to determine if there are any actions that need to be taken in order to improve this.

In order to further look into the question, the tables and figures below compare whether the country of domicile of the graduate and the country provided in the survey match. This may help to identify any patterns or areas of further research. Initially, proportion of matches in country in each section are presented below.

 

Table 32: Proportions of country of domicile and country selected being the same or different in each survey section across years

	Different country		Same country
	Year 4	Year 5	Year 4	Year 5
Section B	30.54%	37.46%	69.46%	62.54%
Section C	32.75%	38.38%	67.25%	61.62%

In general, it appears that across both sections the levels of matching countries are similar, although there are slightly higher levels of mismatch in Section C. Between years, it appears that there are higher levels of mismatch this year for Year 5 (21/22). We would not expect this difference with no changes in question, however, response levels between these survey years varied for the international and EU groups. In Year 5 (21/22) the response levels dropped, which may impact response to the question as these are the groups that are more likely to have matching countries. We have previously researched this topic and found large variations depending on the group (home, EU or non-EU international). Those in the home domiciled groups will generally always mismatch due to these options appearing in the employment location, rather than country, question. Comparatively, the EU and international graduates can have either a match or mismatch depending on selections. As a result, this does not seem overly concerning. However, in order to delve into this further Figure 14, Table 33, Figure 15 and Table 34  further present results split by completion mode and work types in order to look into the question in more depth.

 

Table 33: Comparison proportions of country of domicile and country selected being the same or different in Section B, split by type of work and completion mode

	CATI		ONLINE
	Year 4	Year 5	Year 4	Year 5
Different country
Paid work	43.76%	50.65%	24.60%	30.56%
Volunteering	55.41%	58.88%	40.33%	44.98%
Both	39.93%	44.58%	29.15%	33.59%
Same country
Paid work	56.24%	49.35%	75.40%	69.44%
Volunteering	44.59%	41.12%	59.67%	55.02%
Both	60.07%	55.42%	70.85%	66.41%

 

Table 34: Comparison proportions of country of domicile and country selected being the same or different in Section C, split by type of work and completion mode

	CATI		ONLINE
	Year 4	Year 5	Year 4	Year 5
Different country
Self employed	44.61%	50.11%	32.48%	35.32%
Own business	41.30%	47.80%	19.65%	25.24%
Multiple selections	43.28%	45.53%	23.18%	31.68%
Same country
Self employed	55.39%	49.89%	67.52%	64.68%
Own business	58.70%	52.20%	80.35%	74.76%
Multiple selections	56.72%	54.47%	76.82%	68.32%

The increase in mismatches has occurred for both completion modes and for each type of work across both of the employment sections. Reassuringly, patterns are the same across both years, with the volunteering work type for Section B having the most mismatches across the groups, perhaps as volunteers are more likely to be completing their activity in different countries. In Section C, differences are not generally as large but the self-employed work type consistently sees higher levels of mismatching locations. In terms of the mode of completion, the online mode consistently has higher match levels. This further supports the likelihood that the make up of the responses may be impacting these changes, as those on CATI cannot be non-EU international as these graduates are no longer called. Comparatively, the online mode will have more international responses and hence is more likely to see a match. It would be useful to explore this question further and to look into mismatches in more depth to aid in identifying the potential presence of survey effects such as primacy or recency effects. Although not presented in this report, we have also previously considered splits by home, EU and non-EU international groups. It may be useful to repeat this on Year 5 data. 

Qualification required

Qualification required aims to ascertain whether the qualification that the graduate received approximately 15 months prior was necessary in getting the main job they are talking about in Section B. This question is only asked in this section and asks about whether the qualification, level or subject was required. It does have a ‘don’t know’ option present. Initially, item response levels are presented for the question, split by the Section B work types.

Table 35: Item response rates for qualification required in Section B, split by completion mode and type of work being undertaken

	CATI		ONLINE
	Year 4	Year 5	Year 4	Year 5
Paid work	97.92%	97.82%	98.82%	98.68%
Volunteering	88.99%	89.65%	93.14%	92.44%
Both	97.10%	96.85%	97.79%	97.92%

The highest response levels across both completion modes were for paid work. Levels were lower for volunteering or unpaid work, which was slightly higher in the online mode. In some cases, it may be that graduates are uncertain that this question is as relevant to volunteering. Equally, survey fatigue may be a larger factor at this stage. As this is not a mandatory question graduates may also realise that they can skip if they are feeling uncertain about the relevance, and interviewers of those volunteering may skip if the graduate expresses reluctance to answer. Indeed, understanding more about the options selected may aid in understanding the response to this question a little more clearly and these are presented below.

Table 36: Selections made in response to the qualification required question in Section B across years, split by the type of work

	Paid work		Volunteering		Both
	Year 4	Year 5	Year 4	Year 5	Year 4	Year 5
Yes: both the level and subject of qualification was a formal requirement	33.85%	33.15%	14.52%	13.35%	24.47%	23.55%
Yes: the level of qualification was a formal requirement	11.17%	10.47%	4.26%	4.13%	7.86%	8.06%
Yes: the subject of the qualification was a formal requirement	4.44%	4.45%	3.42%	3.15%	3.97%	3.74%
Yes: while the qualification was not a formal requirement it did give me an advantage	23.84%	24.96%	26.65%	26.87%	29.02%	29.52%
No: the qualification was not required	25.39%	25.71%	48.14%	50.06%	33.94%	34.23%
Don't know	1.30%	1.26%	3.01%	2.43%	0.74%	0.88%

Levels looks steady across years, but it is clear to see that requirements are different depending on the type of work being completed. This aligns with expectations and aids in our understanding of the item response rates for volunteers, as we can see that there are larger selections of ‘don’t know’ for this group along with ‘No: the qualification was not required’. It may be that those in this work type feel more uncertain and as though this is less relevant and are therefore reluctant to answer. We will continue to track item response levels to this question.

Main reason for taking the job

In each section, respondents are asked to select from a list of options the main reason that they are in their employment. It is an optional question. Item response rates are in Table 37.

Table 37: Item response rates to the main reason for employment question in each section across years, split by completion mode

	CATI		ONLINE
	Year 4	Year 5	Year 4	Year 5
Section B	96.94%	96.73%	97.93%	97.83%
Section C	98.27%	98.14%	98.07%	97.55%

Item response rates are generally fairly steady although slightly lower than the previous survey year. Between the survey modes, rates are also similar with slightly higher online in Section B and slightly higher on CATI in Section C. Generally, Section C appears to have slightly higher item response rates than Section B. In order to identify any particular issues, the type of work is also split in Table 38.

Table 38: Item response rates to the main reason for employment question in each section across years, split by completion mode and type of work

	CATI		ONLINE
	Year 4	Year 5	Year 4	Year 5
Section B
Paid work	97.34%	97.12%	98.07%	98.00%
Volunteering	85.03%	85.99%	89.18%	86.86%
Both	96.22%	95.89%	97.27%	96.33%
Section C
Self employed	98.40%	98.25%	98.40%	97.69%
Own business	98.02%	98.12%	97.98%	97.40%
Portfolio	98.15%	98.08%	96.98%	96.53%
Multiple selections	98.38%	98.15%	98.55%	98.49%

When splitting by type of work, it’s clearer to see the cause of lower item response rates in Section B. Indeed, the volunteering work type has lower levels of item response, as seen in some of the other questions that have been assessed in this section. Once again, it may be that these options do not seem as relevant to the graduates in volunteering or unpaid work. Indeed, assessing these options it’s clear that they may be more applicable to other forms of work. Whilst Section C has less difference between the types of work, online in Section C we can also see an area where this may be the case, with the lower rates in the portfolio group.

Whilst these options suit the purposes laid out by the sector, there may be more work to be done to clarify the requirements, and more actions that could be taken to reassure respondents of certain groups. To assess the question further, the options selected by graduates in different work types are presented below for Section B (Figure 17) and Section C (Figure 18).

 

Whilst generally the option “It fitted into my career plan/ it was exactly the type of work I wanted” is the most commonly selected for the work types, this is not the case for the volunteering group, which sees higher levels of selection for the option of “To gain and broaden my experience in order to get the type of job I really want”. This option was also popular (although not the most common) for the portfolio work type in Section C. This seems to support the suggestion that lower item response rates in these work type groups may result from respondents feeling as though the options are not as applicable. This will be exacerbated by the ordering, which is designed to be in an order to make selection easier. However, this option ordering is more likely to suit the majority of graduates in other forms of work. There is a concern that primacy or recency effects could be impacting this question. Whilst the first option is more commonly selected, the second option and the last option are not. Generally, patterns are as expected, for example with the family business options more selected for own business and self-employment, as well as the volunteering work types. There is scope to explore this further alongside other survey answers, or to create randomisation experiments to test changes in response. However, at present this ordering aids in presenting relevant options to the graduate first to reduce drop out and aids interviewers in ensuring they cover all options.

 

Next steps

As highlighted in various stages of the report, work is continuing to develop this work in more depth through the data quality plan. Tracking of item response rates will continue and more in depth work is planned for the next year. 

Next: Location data

---

Schober, M. F., & Conrad, F. G. (1997). Does conversational interviewing reduce survey measurement error?. Public opinion quarterly, 576-602. 

Conrad, F. G., & Schober, M. F. (2000). Clarifying question meaning in a household telephone survey. Public opinion quarterly, 64(1), 1-28.