codemotion: expanding the design space of interactions with programming tutorial videos

Videos are popular for computer programming tutorials since they can convey expert narration alongside the dynamic effects of editing and running code. However, these videos simply consist of raw pixels, so there is no way to interact with the code embedded inside of them. Can we expand the design space of learner interactions with programming videos beyond conventional video players? To do so, we1 first developed Codemotion, a computer vision algorithm that automatically extracts source code and dynamic edits from existing videos. Codemotion segments a video into regions that likely contain code, performs OCR on those segments, recognizes source code, and merges together related code edits into contiguous intervals. We then used Codemotion to build a novel video player and elicit interaction design ideas from potential users by running an elicitation study on 10 students followed by four participatory design workshops with 12 additional students. Participants collectively generated ideas for 28 kinds of interactions such as inline code editing, code-based skimming, pop-up video search, and in-video coding exercises.

(paper under submission to Learning at Scale 2018, shared on request)

  1. Kandarp Khandwala, Philip Guo.