Not Ready for the Runway

a woman on a runway wearing a long lacy spaghetti strap dress with the legs of the audience in the background. below the main image is a label that says “a woman in a dress holding a surfboard”
Original image by Duane Mendes on Unsplash; label from RunwayML Caption Images/im2txt machine learning model

Assumptions

Assumptions about the users of RunwayML that I’m writing about:

  • they are not data scientists or machine learning experts, and thus have no in-depth knowledge of machine learning systems or pipelines
  • have a familiarity with common types of software and web apps
  • tend to use RunwayML for creative uses

Heuristic Evaluation Summary

For this evaluation, I used Nielsen’s 10 Usability Heuristics for User Interface Design. What follows below is only a summary of a larger evaluation, and the following are the most important in terms of what’s done well and what’s done poorly on RunwayML’s site.

The Good: Effective Use of Language

Maps to Nielsen’s Usability Heuristic #2: Match between system and the real world

a screenshot of the RunwayML app with some models available such as Video Post Processing, Stylization, Animation, Image Analytics, Latent Videos, and Text Generalization
Some available models in RunwayML

The Bad: What’s Going on With the Model?

Maps to Nielsen’s Usability Heuristics #1: Visibility of System Status and #5: Error Prevention.

a set of screenshots of buttons in RunwayML with labels: the button labeled one says “Run Model”, the button labeled two says “Starting (Click to stop)”, the button labeled three says “this might take a while…” and the button labeled four says “stop”
Buttons in a model training sequence in RunwayML

Cognitive Walkthrough

I looked at three tasks in my cognitive walkthrough of the RunwayML site:

Action: Get New Data

two screenshots. one shows the RunwayML app showing a process still “in progress…” while the other shows an email stating that the process has completed and the output is available for download
Screenshot of the “In Progress…” status on the left; at the same time, the notification on the right appeared in my email

Action: Get Results

three screenshots from RunwayML. One shows the model still running, but the other two show the assets page with the output ready to be downloaded.
On the left, the model appears to still be running, but on the assets page, the output is ready to be downloaded

Summary

For the really basic function of uploading data, RunwayML is fine. It is nothing revolutionary, but it doesn’t need to be. If the web app couldn’t handle that, it wouldn’t even be a functional app.

Conclusions and Recommendations

The summary of RunwayML’s usability:

Recommendations

A few recommendations for RunwayML’s UX team:

  • Create a Usage Maturity Matrix to transition users from ML novices to “expert enough” for the no code ML platform. Even non-expert users, for instance, should understand the concept of benchmarking in some form, even for images or words to be used in artistic endeavors.
  • Create a common design language to fix readability and consistency issues
  • Design models beyond the machine learning pipeline paradigm for non-developer users of machine learning products.
a brick wall with a rectangle painted a different color
Photo by Patrick Tomasso on Unsplash

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store