E-commerce Experimentation Shatters Records in Early 2025
John Lloyd
Situation: Ensuring Experience Work
Didn’t Get Left Behind
With increasing pressure to deliver features quickly, there was a real risk that user experience improvements would be overlooked. I needed to create space for design-led experimentation by aligning cross-functional stakeholders, keeping the team focused on long-term growth levers like engagement and bookings while still meeting delivery commitments.
Task: Create Environment for Increased Testing
I needed to collaborate with our portfolio delivery managers to ensure that 20% of the product backlog was reserved for experience-focused work. This allocation created the space for my team to run structured experiments aimed at improving the shop and booking journey, ultimately driving higher guest engagement and increased bookings.
Action: Facilitated Quartely Experiment Workshops
We brought designers, product managers, and engineers together to generate hypotheses grounded in both guest pain points and business priorities. This ensured our testing roadmap was collaborative, feasible, and impactful.
Action: Prioritized the Roadmap with Stakeholders
Partnered with marketing, revenue, and product leadership to prioritize high-leverage tests (e.g., shop form redesign, branded search logic) while deprioritizing lower-impact ideas.
Action: Q1 Experiment Highlights
A/B: Sticky Shop Form on Brand Pages
Hypothesis
Creating a sticky shop form experience on the brand and portfolio pages will increase engagement with the shop form.
Results
Increased engagement with “Find a Hotel” by 1.7%.
Overall increase in clickthrough to Rooms by 0.3%.
Increase in clickthrough to Payment on desktop +0.5%
Default
Winner
A/B: Shop by Price Checkbox
Hypothesis
Users who click into the date selector and see a Shop by Price checkbox will enter Flex Dates calendar at an increased rate.
Results
Flex Dates page entry increased by 12.3%
Explore Rates CTA clicks increased by 1.9%
No impact to clickthrough to Book or conversion.
Default
Winner
Action: Q1 Experiment Highlights
A/B: Month Tabs in Shop Calendar Modal
Hypothesis
Adding months tabs to the calendar modal in Shop will increase clickthrough/ conversion.
Results
Decreased overall RPV (-2.6%, -$1.98 change).
Decreased clickthrough to Payment (-0.5%, -0.23 pt change).
Estimated -$16.42M in quarterly attributable savings.
Default
New Variant
A/B: “Find a Hotel” Opens in a New Tab
Hypothesis
Sending users clicking on "Find a Hotel" to Shop pages to a new tab rather than the existing tab will increase conversion .
Results
Increased conversion (+3.6%, +0.73 pt change).
Increased RPV (+3.6%, $3.04 change).
Estimated +$4.37M in quarterly attributable revenue.
Default
Winner
Action: Q1 Experiment Highlights
A/B: Key Amenity Info on Hotel Card
Hypothesis
Adding key amenities of a property to the hotel card will increase clickthrough.
Results
Increase in RPV (+0.7%, +$0.37 change).
Increase in overall conversion (+0.5%, +0.08 pt change).
Estimated +$7.89M in quarterly attributable revenue.
Default
Winner
A/B: Recently Viewed Chip
Hypothesis
Highlighting properties that the user has recently viewed will increase clickthrough to Book.
Results
Increase in mobile conversion (+0.7%, +0.12 pt change).
Increase in clickthrough to Payment on desktop (+0.2%, +0.12 pt change).
Estimated +$1.22M in quarterly attributable revenue.
Default
Winner
Action: Q2 Experiment Highlights
A/B: Selected Dates Indicator on Form
Hypothesis
Showing date selection indicators on the Property shop form will increase clickthrough as check in + check out dates are now clearly displayed for the user to confirm.
Results
Positive impact to revenue per visitor: +1.0%
Increased mobile conversion: +1.0%, 0.08 pt change
Estimated +$2.47M in quarterly attributable revenue.
Default
Winner
A/B: Curated Shop Form Pilot
Hypothesis
Users who see the shop form component on Curated will engage with shop form and enter booking flow at a higher rate.
Results
Increase in Check Rates" CTA clicks: +30.7%
Increased conversion: +13.0%, 0.31 pt change.
Estimated +3.34M in quarterly attributable revenue.
Default
Winner
Action: Q2 Experiment Highlights
A/B: App - Map/Filter Sticky Button
Hypothesis
Presenting users with a sticky and more prominent (color) map/filter button on search results page will increase conversion and/or revenue per visitor.
Results
Amenities Filter Clicks increased: 18.5%, 0.27 pt change.
Increase in RPV: 1.4%, $1.09 change.
Estimated $13.39M in quarterly attributable revenue.
Default
Winner
A/B: Reassurance Messaging: Stay Dates
Hypothesis
Users who are opting to edit stay dates in the booking funnel will note an increase in clickthrough as the reassurance is helpful to guests in other areas of the site.
Results
Increase in clickthrough to the flexible dates calendar (2.9%, 0.27 pt change).
Increase clickthrough to Payment (0.6%, 0.17 pt change).
Default
Winner
Action: Q2 Experiment Highlights
A/B: Desktop Shop Form Redesign
Hypothesis
Updating the shop form to a new look and feel will do no harm to clickthrough or conversion as this test is the first step to bringing the new shop form to OHW.
Results
Increased conversion and RPV (+1.8%, +0.23 pt change and +1.8%, +$0.90 respectively).
Increased engagement with the shop form.
Default
Winner
A/B: Mobile Shop Form Redesign
Hypothesis
By redesigning the search results screen including reduced font sizes and readjust content hierarchy, clickthrough will increase compared to the previous M3 screen version.
Results
Search to rooms click (9.4%, 5.25 pt change)
Dateless to dates click (28.1%, 0.52 pt change)
Rooms page load (2.7%, 1.86 pt change)
Default
Winner