Cruise Competitive Benchmark Report

Cruise Competitive Benchmarking

Stay Ahead of Your Competition
REQUEST A DEMOLEARN MORE
Cruise mobile ux report

Cruise Competitive Benchmarking

(Published Annually in March)

In these annual benchmark industry reports, Key Lime Interactive (KLI) uses a multi-faceted research approach combining: quantitative, qualitative, and behavioral data to provide meaningful consumer insights for a variety of industries.

Cruise Competitive Index Overview

In these annual benchmark industry reports, KLI combines quantitative, qualitative, and behavioral data from hundreds of participants who complete a series of common tasks on each cruise line website. Statistical analysis of this data is employed to rank sites and uncover the top drivers of loyalty and site satisfaction.

Here are the seven Contemporary & Premium Cruise Line websites and mobile sites KLI took a closer look at:

  • Carnival
  • Celebrity Cruises
  • Disney Cruise Line
  • Holland America
  • Norwegian Cruise Line
  • Princess Cruises
  • Royal Caribbean International
  • Virgin Voyages

In our 2020 Competitive Benchmark Report on the Cruise industry, we performed a survey of 2,800 people on Mobile and Desktop.

 

Read the Overview: Cruise Competitive Index Overview PDF

Mobile UX Competitive Intelligence

Senior leadership and online marketers alike can make at-a-glance decisions about key areas of the user experience to satisfy their existing and prospective customers. What does your customer want and how can you better serve them?

Our competitive analysis can help address some key strategic marketing questions like:

  • How do you rank compared to your top competitors?
  • What are the user’s top pain points using your website?
  • What are the best practices for cruise research and booking online?
  • How do you measure up in key segments (such as first-time cruisers)?

Cruise Competitive Analysis Details

Each year the benchmark will examine sites’ top problem areas from the previous year and any site redesigns/changes made subsequently, to answer the question: Have the sites made changes that have improved or worsened the user experience?

This Cruise UX Analysis will include:

  • Statistical analysis of quantitative data
  • Multiple regression analysis to yield top predictors of desired outcomes (Loyalty and Site Satisfaction)
  • Behavioral data
  • First click analysis
  • Average time on task
  • Uncovers problems with the sites’ navigation, nomenclature, layout, clarity or prominence of key features/ functionality/ content
  • Direct user feedback
  • Insights from consumers in their own words about what they like, dislike, want, need, and their expectations and frustration
  • In-depth analysis of report buyers’ website
  • What does the site do well relative to the competition?
  • Where are the gaps between the site and its competitors?
  • Specifically, what is responsible for the gaps?  And what specific issues with the online experience drive negative user perceptions of the site?
  • Success Factors and Best-in-class examples
  • Best-in-class sites highlighted to illustrate why more users say it’s easy to use makes the best first impression, has the most appealing  activities and excursions, why more users are likely to return/recommend/ book on the site or with cruise company in future
  • Longitudinal tracking and analysis

Additional information about KLI’s methodology is detailed in our detailed report. Request Detailed Report.

View More Mobile UX Reports:

Home & Renters Insurance

This report goes far beyond a basic heuristic evaluation or expert review, KLI’s methodology is unique because it incorporates consumer preferences.

View Details

Credit Card

These insights summarize new, successful, and well-received features across the finance industry.

View Now

Auto Insurance

The evaluation includes a general discussion about what these companies are doing for their existing and prospective customers via mobile-optimized websites and apps (iOS and Android). Data is captured from actual customers of each issuer and includes screenshots of features from behind-the-login screens of each provider.

Learn More

Mobile Banking Insights

In this detailed report, KLI profiles the competitive mobile landscape of eight (8) of the top US retail banks, and ultimately reveals which bank offers its users the best mobile solutions.

Learn More

Request a Competitive Insights Subscription

Best-in-class Mobile CX Comparisons

Gain insights into typical behaviors and context for the use of your product with year-over-year brand performance ratings across mobile web + mobile app across iOS and Android.

Play-by-Play Screenshot

Get full access to a library of screenshots of your competitor’s latest mobile designs indexed by feature, brand, and release date.

Dynamic Change Tracking Tables

See when top brands in your industry update, remove or add new mobile features.

Features Ranked By Consumers

View results from over 500 active consumers on the mobile features they value most in your industry.

Chart your Course for a Better Customer Experience Today

Meet with a KLI Team Member

 

{

"These rankings acknowledge our continued focus on providing both current and prospective State Farm customers with the same great experience, regardless of how they come to us"

Patty Gaumond / State Farm

"Our C-suite asked us to find an A-Team… Having worked with Key Lime at a previous employer, I knew that there was no better team than them for actionable insights and strategic design recommendations.”

VP of Digital Strategy & Innovation / Miami Heat

“…This methodology provides the details we need in order to focus our development plans in the coming year....”

VP of eCommerce / Norwegian Cruise Line

Locations

Subscribe to Our CX Newsletter

Home & Renters Insurance Competitive Analysis

Home & Renters Insurance UX Competitive Analysis

See How You Rank
REQUEST A DEMOLEARN MORE
mobile ux home report

Home & Renters Insurance Mobile UX Competitive Analysis

(Published Bi-Annual in March and September)

This Mobile UX Competitive benchmark report goes far beyond a basic heuristic evaluation or expert review. Additionally, thanks to our innovation arm Key Lime Labs, these insights are now available in a comprehensive platform designed for business intelligence leaders, market analysts, UX/CX strategists, and product managers who want on-demand access to the competition’s mobile customer experience.

Home & Renters Insurance Competitive Index Overview

Key Lime Interactive’s (KLI) Home/Renter’s Insurance Mobile Competitive Index Report reviews eight (8) of the largest home and renter’s insurance companies in the U.S. including:

  • Allstate
  • Farmers Insurance
  • Liberty Mutual
  • Nationwide
  • State Farm/span>
  • The Hartford
  • Travelers
  • USAA

KLI’s methodology is unique because it incorporates consumer preferences and goes far beyond a basic heuristic evaluation or expert review. An important part of our analysis is a 500-person survey of smartphone owners in which they are asked to weigh in and identify the features and capabilities that they consider critical for a successful experience with the mobile site or app offered by their home/renter insurance company.

KLI’s intent in creating this third-party syndicated report is to:

    1. Provide consumer-driven data to help guide insurance companies as they prioritize features to implement. This guidance will be based on the results of a current consumer survey.
    2. Summarize how the insurance companies differentiate themselves from their competitors through the capabilities and features that they offer.

View Competitive Benchmark Overview: Home/Renters Insurance Competitive Overview

Mobile UX Methodology

To create an overall score, we combine a capabilities assessment with user feature importance ratings. The capabilities assessment is created by examining the feature coverage of insurance sites and applications. User ratings are determined by a consumer survey and card sort.

Our review of the primary mobile properties includes a full verification of the insurance companies’ capabilities. The unmodified score represents feature coverage or the company’s offerings by category. This is a binary evaluation reflecting the possession of certain criteria.

Additional information about KLI’s methodology is detailed in the detailed report. Request Detailed Report.

Incorporating User Feedback

KLI conducted a consumer survey and card sort (n=500) to gather feedback about how customers prioritized features when using a home/renter’s insurance company’s mobile property. Individual feature scores were then weighted by their value to customers. The goal is to provide a metric of relative importance so that the highest-scoring company is also the one providing customers’ desired features.

 

Summary of Capabilities & Features within UX Analysis

  • Secure Log In Process
  • Access Policy Info
  • Account Settings/Management
  • Bill Payment
  • Get Quote
  • Claims
  • Alerts
  • Help/Self Service
  • Locate an Agent
  • App Interface
  • Customer Support
  • Social Media

View More Mobile UX Reports:

Mobile Banking Benchmark Report

The report features key areas that drive user satisfaction and quickly identifies areas of improvement to satisfy existing and prospective customers. KLI’s research experts highlight opportunities based on a rigorous independent evaluation and that ultimately identifies trends and opportunities that move the entire industry forward.

View Details

Credit Card

These insights summarize new, successful, and well-received features across the finance industry.

View Now

Auto Insurance

The evaluation includes a general discussion about what these companies are doing for their existing and prospective customers via mobile-optimized websites and apps (iOS and Android). Data is captured from actual customers of each issuer and includes screenshots of features from behind-the-login screens of each provider.

Learn More

Cruise Competitive Index

Each year the benchmark will examine sites’ top problem areas from the previous year and any site redesigns/changes made subsequently, to answer the question: Have the sites made changes that have improved or worsened the user experience?

View Insights

Brand Ratings

Curious who the top-ranking brands in your vertical are? Here’s a sneak peek of the top 3, but click below to request a full UX competitive analysis:

1

State Farm

Mobile Web Score: 82%

2

USAA

Mobile Web Score: 77%

3

Travelers

Mobile Web Score: 70%

Unbiased Third-party Mobile UX Research

Get the Competitive Intelligence you need to stay ahead in your industry. Competitive Insights is an intuitive competitive intelligence software, incorporates consumer preferences of features and capabilities that they consider critical for a successful experience including:

 

  • A complete listing of overall rankings
  • A detailed scoring breakdown of all capabilities and features reviewed
  • Screenshots from behind the login screens of all competitors sites & apps
  • A comparative list of current mobile capabilities available on each mobile app
  • Detailed descriptions of how users can complete tasks on the mobile device
  • User feedback collected regarding feature preferences
  • Identifies key success factors, best-in-class features, areas for improvement, and why some companies are lagging behind
KLI UX Consultants

Mobile UX Competitive Insights Subscription

Best-in-class Mobile CX Comparisons

Gain insights into typical behaviors and context for the use of your product with year-over-year brand performance ratings across mobile web + mobile app across iOS and Android.

Play-by-Play Screenshot

Get full access to a library of screenshots of your competitor’s latest mobile designs indexed by feature, brand, and release date.

Dynamic Change Tracking Tables

See when top brands in your industry update, remove or add new mobile features.

Features Ranked By Consumers

View results from over 500 active consumers on the mobile features they value most in your industry.

Chart your Course for a Better Customer Experience Today

Meet with a KLI Team Member

 

{

"These rankings acknowledge our continued focus on providing both current and prospective State Farm customers with the same great experience, regardless of how they come to us"

Patty Gaumond / State Farm

"Our C-suite asked us to find an A-Team… Having worked with Key Lime at a previous employer, I knew that there was no better team than them for actionable insights and strategic design recommendations.”

VP of Digital Strategy & Innovation / Miami Heat

“…This methodology provides the details we need in order to focus our development plans in the coming year....”

VP of eCommerce / Norwegian Cruise Line

Locations

Subscribe to Our CX Newsletter

Mobile UX Credit Card Competitive Benchmarking

Credit Card Competitive Benchmarking

Gain UX Mobile Competitive Insights
REQUEST A DEMOLEARN MORE
mobile ux credit card report

Credit Card Competitive Benchmarking

(Published Bi-Annual in February and August)

The Mobile Credit Card Competitive Analysis by our innovation arm, Key Lime Labs, summarizes these new, successful, and well-received features across the finance industry so that both the industry leaders and institutions, large and small, can make at-a-glance decisions. This analysis features key areas that drive user satisfaction and quickly identify areas of improvement to satisfy existing and prospective customers. KLI’s research experts highlight opportunities based on a rigorous independent evaluation and that ultimately identifies trends and opportunities that move the entire industry forward.

Mobile Credit Card Competitive Index Report


In this detailed report, Key Lime Interactive (KLI) reviews the competitive mobile landscape of nine (9) of the top US credit card issuers, and ultimately reveals which bank offers its users the best mobile solutions:

  • American Express
  • Bank of America
  • Chase (JP Morgan Chase)
  • Citibank (Citigroup)
  • Capital One
  • Discover
  • USAA
  • Wells Fargo & Company
  • U.S. Bank

This report goes far beyond a basic heuristic evaluation or expert review; KLI’s methodology is unique because it incorporates consumer preferences. An important part of our analysis is a 500-person survey of smartphone owners in which they are asked to weigh in and identify the features and capabilities that they consider critical for a successful experience with the mobile site or app offered by their credit card company. KLI’s intent in creating this third-party syndicated report is to:

    1. Provide consumer-driven data to help guide credit card issuers as they prioritize features to implement. This guidance will be based on the results of a current consumer survey.
    2. To summarize how credit card companies differentiate themselves from their competitors through the capabilities and features that they offer.

View Credit Card Competitive Index Overview: Credit Card Competitive Index Overview

Mobile UX Credit Card Methodology

To create an overall score, we combine a capabilities assessment with user feature importance ratings. The capabilities assessment is created by examining the feature coverage of credit card sites and applications. User ratings are determined by a consumer survey and card sort.

Our review of the primary mobile properties includes a full verification of the financial institutions’ capabilities. The unmodified score represents feature coverage or the bank’s offerings by category. This is a binary evaluation reflecting the possession of certain criteria.

Request a Detailed Analysis Today

Capabilities and Features to be Assessed

  • Secure Log In Process
  • Fraud Protection & Security
  • Account Activity/History
  • Account Settings/Management
  • Bill Payment
  • Rewards
  • Customer Support
  • Social Media

View More Mobile UX Reports:

Home & Renters Insurance

This report goes far beyond a basic heuristic evaluation or expert review, KLI’s methodology is unique because it incorporates consumer preferences.

View Details

Mobile Banking Analysis

In this detailed report, KLI profiles the competitive mobile landscape of eight (8) of the top US retail banks, and ultimately reveals which bank offers its users the best mobile solutions.

Learn More

Mobile Auto Competitive Index

The evaluation includes a general discussion about what 8 of the largest companies are doing for their existing and prospective customers via mobile-optimized websites and apps (iOS and Android).

Access UX Mobile Auto Report

Cruise Competitive Index

Each year the benchmark will examine sites’ top problem areas from the previous year and any site redesigns/changes made subsequently, to answer the question: Have the sites made changes that have improved or worsened the user experience?

View Details

Brand Ratings

Curious who the top-ranking brands in your industry are? Here’s a sneak peek, but click below to request a full mobile UX competitive analysis:

1

Chase (JP Morgan Chase)

Mobile Web Score: XX%

2

Wells Fargo & Company

Mobile Web Score: XX%

3

U.S. Bank

Mobile Web Score: XX%

Competitive Insights

The unbiased third-party mobile UX research you need to stay ahead of your competitors. Competitive Insights, an intuitive competitive intelligence software, incorporates consumer preferences of features and capabilities that they consider critical for a successful experience including:

 

  • A complete listing of overall rankings
  • A detailed scoring breakdown of all capabilities and features reviewed
  • Screenshots from behind the login screens of all competitors sites & apps
  • A comparative list of current mobile capabilities available on each mobile app
  • Detailed descriptions of how users can complete tasks on the mobile device
  • User feedback collected regarding feature preferences
  • Identifies key success factors, best-in-class features, areas for improvement, and why some companies are lagging behind
Ux highlights

Mobile UX Competitive Insights Platform

Best-in-class Mobile CX Comparisons

Gain insights into typical behaviors and context for the use of your product with year-over-year brand performance ratings across mobile web + mobile app across iOS and Android.

Play-by-Play Screenshot

Get full access to a library of screenshots of your competitor’s latest mobile designs indexed by feature, brand, and release date.

Dynamic Change Tracking Tables

See when top brands in your industry update, remove or add new mobile features.

Features Ranked By Consumers

View results from over 500 active consumers on the mobile features they value most in your industry.

See How your Mobile UX Compares to the Top Brands in your Vertical

Insights

Locations

Subscribe to Our CX Newsletter

Mobile UX Banking Competitive Index Report

Mobile Banking Competitive Analysis

Gain UX Insights Into Mobile Banking
REQUEST A DEMOLEARN MORE
mobile ux banking

Banking Mobile UX Competitive Analysis

(Published Bi-Annual in April and October)

The Mobile Banking Competitive Index provided summarizes these new, successful, and well-received features across the banking industry so that both industry leaders and institutions, large and small, can make at-a-glance strategic decisions. The report features key areas that drive user satisfaction and quickly identify areas of improvement to satisfy existing and prospective customers. KLI’s research experts highlight opportunities based on a rigorous independent evaluation and that ultimately identifies trends and opportunities that move the entire industry forward.

Mobile Banking Competitive Index Overview

In this detailed report, KLI profiles the competitive mobile landscape of eight (8) of the top US retail banks, and ultimately reveals which bank offers its users the best mobile solutions:

  • State Farm
  • Bank of America
  • Chase (JP Morgan Chase)
  • Citibank (Citigroup)
  • Pen Fed
  • U.S. Bank
  • USAA
  • Wells Fargo & Company

The evaluation includes a general discussion about what these banks are doing to provide for their existing and prospective retail banking users via mobile-optimized sites and apps (iOS and Android). Data is captured from actual users of each bank and includes screenshots of features from behind-the-login screens of each provider.

 

Additionally, Key Lime Interactive (KLI) shares the methods employed to rank the banks based on consumers’ self-reported preferences and insights; provides a best-in-class feature list, as well as a comparative list of the available features of the six; and identify trends and opportunities that bridge the gap between consumer expectations and the current banking solutions. This consumer-based solution differs from other similar reports as KLI uses these consumers’ preferences to weigh the importance of features.

View Competitive Benchmark Report: Mobile Banking Competitive Index Overview PDF

Competitive Index UX Methodology

An expert capabilities assessment, comparative feature ranking, and user ratings were combined to determine an overall score and ranking.

User feedback is incorporated by gathering responses (n=500), to understand which features and tasks were must-haves when it comes to interacting with the mobile property of a retail bank. From those results, KLI prioritized user’s needs and preferences, and then placed the features into four tiers relative to the importance and these tiers played a part in the weighted score we awarded each feature when tabulating a final score for each mobile property.

Additional information about KLI’s methodology is detailed in the detailed report. Request Detailed Report.

View More Mobile UX Reports:

Home & Renters Insurance

This report goes far beyond a basic heuristic evaluation or expert review, KLI’s methodology is unique because it incorporates consumer preferences.

View Details

Credit Card

These insights summarize new, successful, and well-received features across the finance industry.

View Now

Auto Insurance

The evaluation includes a general discussion about what these companies are doing for their existing and prospective customers via mobile-optimized websites and apps (iOS and Android). Data is captured from actual customers of each issuer and includes screenshots of features from behind-the-login screens of each provider.

Learn More

Cruise Competitive Index

Each year the benchmark will examine sites’ top problem areas from the previous year and any site redesigns/changes made subsequently, to answer the question: Have the sites made changes that have improved or worsened the user experience?

Access Report

Brand Ratings

Here’s a sneak peek of the top-ranking brands in your industry. Contact us to request a full UX competitive analysis:

1

Chase (JP Morgan Chase)

Mobile Web Score: 83%

2

Wells Fargo & Company

Mobile Web Score: 82%

3

U.S. Bank

Mobile Web Score: 78%

Unbiased Third-party Mobile UX Research

Get the Competitive Intelligence you need to stay ahead of your competitors. Competitive Insights, an intuitive competitive intelligence software, incorporates consumer preferences of features and capabilities that they consider critical for a successful experience including:

  • A complete listing of overall rankings
  • A detailed scoring breakdown of all capabilities and features reviewed
  • Screenshots from behind the login screens of all competitors sites & apps
  • A comparative list of current mobile capabilities available on each mobile app
  • Detailed descriptions of how users can complete tasks on the mobile device (e.g. pay a bill, look at account history, etc.)
  • User feedback collected regarding feature preferences
  • Identifies key success factors, best-in-class features, areas for improvement, and why some companies are lagging behind
UX success

Request a Competitive Insights Subscription

Best-in-class Mobile CX Comparisons

Gain insights into typical behaviors and context for the use of your product with year-over-year brand performance ratings across mobile web + mobile app across iOS and Android.

Play-by-Play Screenshot

Get full access to a library of screenshots of your competitor’s latest mobile designs indexed by feature, brand, and release date.

Dynamic Change Tracking Tables

See when top brands in your industry update, remove or add new mobile features.

Features Ranked By Consumers

View results from over 500 active consumers on the mobile features they value most in your industry.

See How your Mobile UX Compares to the Top Brands in your Vertical

Insights

Locations

Subscribe to Our CX Newsletter

UX Mobile Auto Insurance Index

Mobile Auto Competitive Analysis

Gain Insights Into your Customers & Competitors
REQUEST A DEMOLEARN MORE
mobile auto report

Mobile UX Competitive Analysis

(Published Bi-Annual in May and December)
The Auto Insurance Mobile UX Competitive benchmark report summarizes key features across the industry with the click of a button so that both industry leaders and small institutions can make at-a-glance strategic decisions. This detailed analysis features key areas that drive user satisfaction and quickly identify areas of improvement to satisfy existing and prospective customers. KLI’s research experts highlight opportunities based on a rigorous independent evaluation and that ultimately identifies trends and opportunities that move the entire industry forward.

Auto Insurance Competitive Index Overview

In this detailed report, KLI profiles the competitive mobile landscape of the eight largest US auto insurance providers:

  • State Farm
  • Allstate
  • GEICO
  • Progressive
  • Nationwide
  • Esurance
  • USAA
  • Liberty Mutual

The evaluation includes a general discussion about what these companies are doing for their existing and prospective customers via mobile-optimized websites and apps (iOS and Android). Data is captured from actual customers of each issuer and includes screenshots of features from behind-the-login screens of each provider.

KLI shares the methods employed to rank the providers based on consumers’ self-reported preferences and insights; provides a best-in-class feature list, as well as a comparative list of the available features; and identify trends and opportunities that bridge the gap between user expectations and the current mobile auto insurance solutions.

Read the overview here: Auto Insurance Competitive Index Overview PDF

Mobile UX Methodology

To create an overall score, we combine a capabilities assessment with user feature importance ratings. The capabilities assessment is created by examining the feature coverage of the auto insurance sites and applications. User ratings are determined by a consumer survey and card sort (n=500), to understand which features and capabilities are must-haves when interacting with the mobile property of an auto insurance provider. From those results, we were able to understand what users believed to be most important to them. We use this consumer data to determine the weighted score we give each feature when tabulating a final score for each mobile property.

Additional information about KLI’s methodology is detailed in the full report.

What’s in this Analysis?

In summary, this Competitive Insights report includes:

  • A complete listing of overall rankings
  • A detailed scoring breakdown of all capabilities and features reviewed
  • A complete list of best-in-class features
  • A comparative list of current mobile capabilities available on each mobile app
  • Detailed descriptions of how users can complete tasks on the mobile device (e.g. pay a bill, file a claim, etc.)
  • User feedback collected regarding feature preferences
  • A comprehensive list of all mobile apps currently available by these eight companies

View More Mobile UX Reports:

Home & Renters Insurance

This report goes far beyond a basic heuristic evaluation or expert review, KLI’s methodology is unique because it incorporates consumer preferences.

Learn More

Mobile UX Credit Card

These insights summarize new, successful, and well-received features across the finance industry.

View Mobile Credit Card Insights

Mobile Banking

The evaluation includes a general discussion about what these banks are doing to provide for their existing and prospective retail banking users via mobile-optimized sites and apps (iOS and Android). Data is captured from actual users of each bank and includes screenshots of features from behind-the-login screens of each provider.

Access Mobile Banking Report

Cruise Competitive Index

Each year the benchmark will examine sites’ top problem areas from the previous year and any site redesigns/changes made subsequently, to answer the question: Have the sites made changes that have improved or worsened the user experience?

View Details

Brand Ratings

Curious who the top-ranking brands in your industry are? Here's a sneak peek, but click below to request a full UX competitive analysis:

1

State Farm

Mobile Web Score: 90%

2

GEICO

Mobile Web Score: 87%

3

USAA

Mobile Web Score: 84%

Competitive Insights

The unbiased third-party mobile UX research you need to stay ahead of your competitors. Competitive Insights, an intuitive competitive intelligence software, incorporates consumer preferences of features and capabilities that they consider critical for a successful experience including:

 

  • A complete listing of overall rankings
  • A complete list of best-in-class features
  • A detailed scoring breakdown of all capabilities available on each mobile app
  • Detailed descriptions of how users can complete tasks on the mobile device (e.g. pay a bill, look at account history, etc.)
  • Identifies key success factors, best-in-class features, areas for improvement, and why some companies are lagging behind.
KLI UX Consultants

Inside your Competitive Insights Subscription

Best-in-class Mobile CX Comparisons

Gain insights into typical behaviors and context for the use of your product with year-over-year brand performance ratings across mobile web + mobile app across iOS and Android.

Play-by-Play Screenshot

Get full access to a library of screenshots of your competitor’s latest mobile designs indexed by feature, brand, and release date.

Dynamic Change Tracking Tables

See when top brands in your industry update, remove or add new mobile features.

Features Ranked By Consumers

View results from over 500 active consumers on the mobile features they value most in your industry.

Chart your Course for a Better Customer Experience Today

Meet with a KLI Team Member

 

{

"These rankings acknowledge our continued focus on providing both current and prospective State Farm customers with the same great experience, regardless of how they come to us"

Patty Gaumond / State Farm

"Our C-suite asked us to find an A-Team… Having worked with Key Lime at a previous employer, I knew that there was no better team than them for actionable insights and strategic design recommendations.”

VP of Digital Strategy & Innovation / Miami Heat

“…This methodology provides the details we need in order to focus our development plans in the coming year....”

VP of eCommerce / Norwegian Cruise Line

Locations

Subscribe to Our CX Newsletter

Survey Design Sins to Avoid

by Phil McGuinness

 

checklist

Are you a User Experience professional who uses online survey tools to deliver insights? If so, you’re in luck! For the last four years, I’ve been working extensively with various online survey tools to deliver everything from simple one-off consumer surveys, to large scale multinational Competitive Benchmarking tests. Throughout that time, I’ve had endless opportunities to experiment with different design methods and survey tools – and to make mistakes, and learn from them – so that you don’t have to. In this article, I would like to share with you some of the potential pitfalls to designing and programming these studies that you can avoid in your next survey. Proper survey design can save you countless hours of frustration when it comes time to analyze the data and deliver your report.

  1. Not Scripting to Your Reporting Tool

Sometimes when you script a survey, you want branching pathways for users who are successful or not, so you create multiple paths. For instance, in UserZoom, you can have a “Success Questionnaire” and an “Error Questionnaire” depending on a particular question. If you only want to look at the success group and the failure group individually, that’s a perfectly sound approach. However, if you want to look at any of those answers cumulatively in the results, you’ll now force yourself to manually compile answers to the same question from those two questionnaires. If you do this across multiple tasks and multiple studies, suddenly you’ll find yourself doing busywork that could have been avoided, had you taken some time to assess how these results would look in the reporting tool. If you’re unsure, run a preview or soft launch with yourself as the participant, and see how the data looks. This could save you hours of time when you get to the analysis phase, trust me!

 

  1. Not Making the Most of a Tool’s Capabilities

Knowledge of the survey tool you’re using is extremely valuable when scripting. For example, many survey tools allow you to tag your questions with an alphanumeric code, allowing easier identification when you export the results. Taking a moment to label your questions with clear, concise tags will make your analysis phase easier and less prone to errors.

 

  1. Script Once, QA Twice (or more!)

Okay, the old adage is measure twice, cut once, but you get the picture. It’s important to lock everything down before you go into field. If you make sure that you have everything you need before gathering results, you avoid common pitfalls like leaving out a key question, or any number of logic issues that could tank a survey. Survey software typically makes it difficult to make script changes once the study has launched, so you could end up throwing away data from days of fielding. That’s why I recommend at least two QA’s, one from the panelist or participant perspective, and one from the client perspective. Ideally this QA will be done by another member of your team, not the person who wrote the script. Experienced survey designers know that it’s easy to develop blind spots for your own script. A proper QA should first take in the participant point of view, making sure the instructions make sense to someone with no knowledge of what is being tested. The second QA should both verify logic and setup, but more importantly, map back the study design to the goals of yourself or your client. This added verification can prevent costly mistakes and time lost when the study goes live.

 

Kitchensink

  1. The Kitchen Sink

Finally, the kitchen sink. It’s tempting to shove everything you can into a survey – especially when pressure mounts from your client and stakeholders – but remind them that the most elegant surveys avoid unnecessary questions and focus on what is most important. It’s of paramount importance to minimize survey fatigue, a real problem that lowers response rate and quality. A good rule of thumb for the longest surveys is a length of 20-25 minutes maximum, and that’s stretching it. Even at 20 minutes there is a large drop off on quality comments near the end of the survey. You may end up throwing out results that would have been valid in a 15-20 minute survey. Ask yourself, or your client, “Do we want 50 questions of middling to poor data or 35 questions with high quality responses?”

 

That’s all for now. I hope you’ve found this educational, or at the very least, entertaining! Subscribe to our newsletter for monthly articles and updates.