An Investigation on Automatically Assessing an Application Tutorial’s Difficulty

Download Publication File

Abstract

Online step-by-step tutorials play an integral role in how users learn feature-rich software applications (e.g., Photoshop, AutoCAD, Fusion360). However, when searching for a tutorial, users can find it difficult to assess whether a given tutorial is designed for their level of software expertise. Novice users can struggle when a tutorial is out of their reach, whereas more advanced users can end up wasting time with overly simple, first-principles instruction. To assist users in selecting tutorials based on expertise, I investigate the feasibility of using machine learning techniques to automatically assess and label a tutorial’s difficulty level. Using Photoshop as a testbed, I develop a set of distinguishable tutorial features and use these features to train a classifier that can label a tutorial as either Beginner or Advanced with 85% accuracy. To illustrate a potential application of my classifier, I developed a tutorial selection interface called TutVis. TutVis annotates each tutorial with its difficulty level, along with visual representations of other tutorial features that contribute to this difficulty assessment. An initial evaluation comparing TutVis to two other interfaces (which varied in the number of different tutorial features displayed) showed a strong preference for and use of TutVis’s novel features.

Slideshow

Download the An Investigation on Automatically Assessing an Application Tutorial’s Difficulty slideshow.

Citation

Shahed Anzarus Sabab. An Investigation on Automatically Assessing an Application Tutorial’s Difficulty. M.Sc. Thesis (2019). University of Manitoba, Canada.