Reliability of sensitive data

Introduction and context

Within the Graduate Outcomes survey there are some questions that could be perceived by a respondent as sensitive in nature. Sensitive questions in particular can be at risk of reduced data quality, for example through increased item non-response or misreporting. Many factors can influence the responses provided to potentially sensitive questions including the mode of completion, question wording, presence of third parties whilst completing a survey and assurances about privacy, confidentiality, or use of the data (Tourangeau and Yan, 2007; Ong and Weiss, 2000). For mode of completion, self-administration modes are generally found to increase respondents reporting potentially undesirable behaviours (Tourangeau and Smith, 1996; DeLeeuw, 2018). Confidentiality and privacy assurances have also been found to improve responses to sensitive questions. However, in some cases, these assurances can have the opposite effect, potentially as they bring data usage or privacy concerns to the forefront of a respondent’s mind who previously may not have considered it (Acquisti, Brandimarte and Lowenstein, 2015).

There are some questions in the Graduate Outcomes survey that could be viewed as more sensitive by respondents. Income is one such question, and the following section will provide some insight into research that has taken place around this question this year. In previous editions of this report, we have also completed investigations into other areas, such as the subjective wellbeing data (2^nd edition of the Graduate Outcomes Survey Quality Report) and employer name and job title (3^rd edition of the Graduate Outcomes Survey Quality Report).

Methods and results

 
The salary and currency questions

Income is commonly considered to be a sensitive topic for a survey question, and it often has higher levels of item non-response associated with it due to the intrusive nature of the question and concerns of disclosure (Tourangeau and Yan, 2007). Since Year 3 of the survey, salary has only been asked of respondents who selected that they are paid in a currency of ‘United Kingdom, Pounds, £’. This helped reduce survey burden and reduced the collection of unnecessary data, but also meant that the order of the questions was switched. In previous years different strategies were used to attempt to increase question coverage, as having an optional currency question after salary was resulting in the collection of unusable data. It was therefore made compulsory to answer currency when salary was populated during the second year of the survey, which heightened the risk of survey drop-out, but ensured currency was provided. The new question order removes the need for a compulsory response hoping this would reduce drop-out rates and ensure that respondents who provided a salary always have the corresponding currency information available.

As well as these changes, additional hover text was also added for a few questions, including salary, for cohort D of Year 3, in order to reassure graduates about the use of their data. Assessments in Year 3 indicated that the change in order and the addition of hover text had increased the response levels to either of the questions. It was determined that there would be value in investigating item non-response again in Year 4 to assess the additional changes that were made to improve response to the question. The changes in the fourth year included creating a new system for calculating typical salary ranges for full-time graduates which is recalculated and applied each year, and the removal of warning limits for part-time work to reduce the number of validation warnings and ensure they are as relevant as possible.  

In order to improve response rates on the Computer Assisted Telephone Interviewing (CATI) mode following the findings in the 3rd edition of the quality report, an action plan was developed to improve response rates and was subsequently implemented from cohort C. Equally, a pop-up was added for desktop completion from cohort D which informs graduates who are exiting the survey that the question is optional, in order to try and reduce overall drop-out from the survey as a whole. Last year’s findings (further detail of which can be viewed in the 4th edition of the quality report) indicated that the changes had generally positive impact on both the response levels to salary and quality of the salaries being provided. However, the report highlighted that further tracking would be beneficial, particularly to determine if the desktop pop-up was having a positive impact. It will also be useful to assess the question further following the collection of more data that may aid in identifying the impact of various other changes that occurred last year. 

Table 6: Table indicating item response rates for salary, split by completion mode, and including a base description of restrictions

Response to salary question (when paid in UK £) has increased from Year 4 to Year 5 overall, and indeed, rates have also risen for all completion modes, as shown in the chart. This may indicate that the various steps that were put in place have continued to have a positive impact on the response to the question. Equally, it offers reassurance that  response on CATI has seen good improvements, indicating that the action plan we worked on with the contact centre has been successful. Rates are still lower than online, which is as expected.Self-administration modes are known to increase the likelihood of a graduate disclosing sensitive information (Brown et al., 2008), and we do see that more respondents provide this information on the online completion mode in Graduate Outcomes. The response levels for the desktop element of online completion appear to have increased.. This is a positive outcome as in the previous report the lower levels of response led to the introduction of a pop-up in the final cohort of the year . and it seems that the pop-up is having a beneficial effect and is helping to reduce overall drop-out from the survey.  

Generally, questions about income often see increasedlevels of item non-response (Tourangeau and Yan, 2007), and in Graduate Outcomes, item non-response is higher for this question comparing to others in the survey. However, when compared with other surveys and the rates of response you may expect for such a question, levels of response are good. Regardless, this question remains high priority for tracking and for ensuring the information collected is as useful as possible. 

Distribution of responses received to salary

Whilst item non-response is one useful way of understanding quality, it is important to assess data quality in other ways. Another indication of data quality in relation to salary may be reductions in salaries outside the ‘expected’ range. Previous changes to the question aimed to reduce confusion that may have been causing some graduates to provide one-digit or two-digit salaries, but whilst this seemed effective some particularly low or high salaries remained. Though this cannot be avoided fully, and some may be genuine responses, it is likely that some of these responses are a result of graduates feeling reluctant to provide a genuine response to this question due to its sensitive nature, perhaps leading to measurement error (Tourangeau and Yan, 2007). To help us tracking the online salary provision following the various alterations, distributions of salaries provided in United Kingdom Pounds are split into broad salary groupings for quality analysis purposes and are shown in Figure 1. 

As a broad overview, an increase in provision of salaries of more than £15,000 to £100,000 may be useful in determining a potential improvement in quality for the full-time salary provision in particular. This increase can be seen graduates in full-time paid work, part-time paid work and self-employed. There was a slight decrease for part-time self-employed graduates. Provision of salaries of £0 have slightly increased across the work types with only full-time self-employment showing a decrease, however the changes seen are small, particularly for those in paid work. This may be a sign that the question is settling but could also perhaps show that more graduates didn’t feel comfortable providing a salary in year five, particularly for self-employed part-time graduates. However, coupled with the changes in item non-response it may be that more graduates are responding who previously would have dropped out of the survey entirely, or skipped the question. Alternatively, these may be genuine responses in some cases (particularly for those in self-employed part-time work which saw the biggest increase) Equally, many of the groups appear to have seen a reduction in graduates providing salaries for both the under £100 and £100-£15,000 groups. Whilst it must be noted that there are a number of factors that will influence these salary groupings, including fluctuations in the economy, it does seem that provision of salaries may be improving. 

Conclusions

The various interventions that have been put in place to improve response to the salary question seem to have had generally positive impact on both the response levels to salary and the salaries being provided. The pop-up appears to have had a positive impact on the desktop mode and it seems beneficial to maintain this in the survey. Equally, interventions on the CATI completion mode appear to have continued having positive effects on response levels to salary, which is particularly important given the difficulty in achieving responses to sensitive questions on this mode in comparison to the online self-completion mode.  

Next: Assurance of the employment and self-employment sections

References 

Acquisti, A., Brandimarte, L. and Loewenstein, G., 2015. Privacy and human behavior in the age of information. Science, 347(6221), pp.509-514.

Brown, J. L., Vanable, P. A., & Eriksen, M. D. (2008). Computer-assisted self-interviews: a cost effectiveness analysis. Behavior research methods, 40(1), 1–7. https://doi.org/10.3758/brm.40.1.1

DeLeeuw, E.D., 2018, August. Mixed-mode: Past, present, and future. In Survey Research Methods (Vol. 12, No. 2, pp. 75-89).

Ong, A.D. and Weiss, D.J., 2000. The impact of anonymity on responses to sensitive questions 1. Journal of Applied Social Psychology, 30(8), pp.1691-1708.

Tourangeau, R., & Smith, T. W., 1996, Asking Sensitive Questions: The Impact of Data Collection Mode, Question Format, and Question Context. The Public Opinion Quarterly, 60(2), 275–304. http://www.jstor.org/stable/2749691

Tourangeau, R. and Yan, T., 2007. Sensitive questions in surveys. Psychological bulletin, 133(5), p.859.