5.4: Coding Procedures
- Page ID
- 98096
Manifest and latent coding methods were used for analytic reduction and a systematic interpretation of underlying patterns in the student focus group data logs. Transcripts of the sessions, recorded using Otter.ai software, were also used once participants’ names were removed from the files and machine transcription errors were fixed. Focus groups, not individual student responses, were used as the unit for our coding. Eight coding properties were used to analyze comments in all 16 focus groups. These properties were intended to capture what kinds of experiences students, in their own words, had with computer algorithms. In cases where students in a single group mentioned a concern, e.g. “the next generation,” more than once in a session, we only counted it once in our final coding results.
Krippendorff’s alpha (KALPHA), considered the most rigorous means of testing intercoder reliability, was run on two pilot test round of focus group logs and coded by two PIL researchers. KALPHA takes into account chance agreement among content analysis coders. While there is no universally accepted standard for intercoder reliability using Krippendorff’s alpha, communications researchers have suggested that a coefficient between 0.81 and 0.99 is “almost perfect,” between 0.61 and 0.80 is “substantial,” and 0.41 to 0.60 is “moderate.” Two pilot coding rounds of three interview logs each were used. During the second pilot round, the coding practices reached the acceptable reliability level of 0.84. Thereafter, we coded the focus group logs using eight individual properties for “concerns.”