Workshop B - Coding Qualitative Verbal Protocol Data for Test Validation
Increasingly, analysing and coding qualitative data generated through introspective methods (e.g. think-alouds, stimulated recall) constitutes a key component of test validation projects, reflecting the growing recognition of the benefits of mixed-methods approaches to investigating test-takers’ cognitive processes. The aim of this workshop is to make participants familiar with the major steps involved in coding qualitative verbal protocol data, that is, organizing and classifying raw data into categories for the purpose of further analysis and interpretation. We will primarily focus on theory-driven coding methods, but we will also consider qualitative coding that emerges bottom-up from the data.
We will begin the workshop with discussing the concepts of validity and reliability in relation to coding qualitative verbal protocol data. Then, we will review the various steps involved in coding and consider strategies that can help increase the validity and reliability of the coding process at each stage.
We will focus on the following specific steps:
- Selecting verbal protocol data for coding, that is, how many and which part of the data to code.
- Preparing data for coding, for example, planning the level of detail to include in the transcription process.
- Deciding whether to adopt or adapt an existing coding scheme or develop one’s own.
- Selecting and training coders to ensure accurate and consistent coding.
- Checking the reliability of coding, calculating coder reliability, and dealing with disagreements between coders.
- Reporting coding procedures.
When considering each of these steps, workshop participants will have the opportunity to apply the concepts covered to sample coding situations and datasets, taken from a variety of L2 assessment projects. Participants will also gain hands-on experience with coding verbal protocol data collected in studies investigating L2 users’ cognitive processes during test performance. In particular, workshop activities will include coding test-takers’ think-aloud/stimulated recall comments describing/recalling their thoughts during listening, speaking, reading, and writing assessments as well as integrated testing tasks involving a combination of skills and/or modalities. We will also look at ways of triangulating various data sources at the coding stage to obtain a fuller and more complete picture of the cognitive activities in which L2 users are involved during test performance. The workshop is intended for graduate students and researchers who are interested in validation research but have little experience coding qualitative verbal protocol data.
Speaker: Andrea Révész, UCL - University College London
Andrea Révész is a Professor of Second Language Acquisition at the IoE, UCL’s Faculty of Education and Society. Her main research interests lie at the interfaces of second language acquisition, instruction, and assessment, with particular emphases on the roles of task, input, interaction, and individual differences in SLA. Currently, she is also working on projects investigating the neurocognitive processes underlying second language speaking and writing performance. She is co-winner of the 2017 TBLT Best Research Article Award and co-recipient of the 2018 TESOL Award for Distinguished Research. Currently, she serves as associate editor of Studies in Second Language Acquisition, co-editor of the John Benjamins Task-based Language Teaching series, and past president of the International Association for Task-based Language Teaching (TBLT).