Going Beyond A/B Testing: Insights from Waitrose's UI Design Process
Watch the full talk here.
Waitrose effectively utilizes user experience (UX) research to inform their user interface (UI) design. Chloe, the UX research manager, and Sebastian, the senior web designer, address the limitations often associated with A/B testing, and how a deeper understanding of user behavior can drive better design decisions.
A/B testing displays two design variations to users, but it often fails to uncover the reasons behind user behavior or the overall impact on their experience. Waitrose focused on alert messages within their grocery delivery site to explore this limitation. They aimed to improve user engagement with these critical notifications.
The team’s approach starts with clearly defined component purposes and problem statements, which guide the design process. They emphasize aligning on the intent before any new UI designs. Engaging users through testing informs assumptions, but testing must extend beyond just measuring which design performs better.
A recent remote usability test revealed a significant engagement gap. Users frequently did not notice alert messages deemed important—like those addressing delivery minimums or availability changes. The results highlighted that simply making alerts more visually distinctive does not guarantee they will be read.
In redesigning the alerts, Waitrose focused on making them visually prominent through color and icon changes, creating an informative status alongside warning messages to enhance user attention. They introduced a highlighted line for visual separation and revamped icons to resonate with user associations, which reflects best practices in iconography.
Through testing with old and new UI designs, Waitrose discovered that users who interacted with the revamped alerts reported higher recall of both information notices and warning messages. Specifically, 97% of users connected the success status with positive outcomes, while 93% associated the informative status correctly.
However, despite achieving greater visibility and recall rates, the challenge persisted: users tended to overlook the informative messages in favor of more urgent warnings, indicating a semblance of hierarchy where the importance the users attached to alerts affected their engagement level.
The findings led to further exploration of user sentiment towards alert types. When presented with color-coded alerts without messages, users prepared themselves emotionally for either good or bad news, based solely on the type of alert shown. This signals the importance of perception in user experience, particularly when offering sensitive information like delivery availability.
Ultimately, testing demonstrated that alerts deemed neutral were viewed more negatively than urgent warnings. The underlying message was significant; users place a high value on how alerts contribute to their actions, weighing the effort to read against the importance of the information provided.
Further tests revealed that the context in which an alert is presented significantly influences user sentiment. When users found alerts helpful for completing tasks, their responses skewed positively, including descriptors like “essential” and “helpful.” This supported the notion that alerts need a dual focus on functionality and user sentiment.
The iterative cycle of user feedback resulted in an expanded vision for their overall UI approach. The team realized that not every communication must be delivered as an alert. They explored informing users through design elements in the booking interface rather than relying solely on traditional alert messaging strategies.
By altering visual cues in the slot selection interface, Waitrose planned to highlight available offerings without irritating the user with excessive alerts. This evolved approach streamlines user interactions and promotes critical information without overly disruptive messages.
The key takeaway for designers is that enhancing UI requires a solid understanding of how users interpret, engage with, and feel about messages. This shift from merely functional UI adjustments to a deeper exploration of user sentiment results in a holistic user experience that enhances brand interaction. The findings show that understanding why users behave a certain way yields insights that extend beyond initial testing objectives, ultimately guiding improvements across the broader web experience.
Testing assumptions and gaining insight into user behavior elevate the design process, as evidenced by the journey Waitrose undertook to refine their UI. Such efforts don’t just elevate component significance; they reframe the overall communication landscape for users interacting with the brand online.