Date Thesis Awarded
11-2021
Access Type
Honors Thesis -- Open Access
Degree Name
Bachelors of Science (BS)
Department
Computer Science
Advisor
Denys Poshyvanyk
Committee Members
Oscar Chaparro
Stephen Trefethen
Peter Kemperer
Dana Willner
Abstract
Screen recordings of mobile applications are easy to obtain and capture a wealth of information pertinent to software developers (e.g., bugs or feature requests), making them a popular mechanism for crowdsourced app feedback. Thus, these videos are becoming a common artifact that developers must manage. In light of unique mobile development constraints, including swift release cycles and rapidly evolving platforms, automated techniques for analyzing all types of rich software artifacts provide benefit to mobile developers. Unfortunately, automatically analyzing screen recordings presents serious challenges, due to their graphical nature, compared to other types of (textual) artifacts. To address these challenges, this thesis introduces V2S+, an automated approach for translating video recordings of Android app usages into replayable scenarios. V2S+ is based primarily on computer vision techniques and adapts recent solutions for object detection and image classification to detect and classify user gestures captured in a video, and convert these into a replayable test scenario. Given that V2S+ takes a computer vision-based approach, it is applicable to both hybrid and native Android applications. We performed an extensive evaluation of V2S+ involving 240 videos depicting 4,271 GUI-based actions collected from users exercising features and reproducing bugs from a collection of over 90 popular native and hybrid Android apps. Our results illustrate that V2S+ can accurately replay scenarios from screen recordings, and it is capable of reproducing 90% of sequential actions recorded in native application scenarios on physical devices and 83% of sequential actions recorded in hybrid application scenarios on emulators, both with low overhead. A case study with three industrial partners illustrates the potential usefulness of V2S+ from the viewpoint of developers.
Recommended Citation
Havranek, Madeleine, "Translating Video Recordings of Mobile App UI Gestures into Replayable Scenarios for Native and Hybrid Apps" (2021). Undergraduate Honors Theses. William & Mary. Paper 1734.
https://scholarworks.wm.edu/honorstheses/1734
