Month: April 2025

Evaluation Methods – Which One Do You Need?

Choosing the right evaluation method depends on your specific needs and constraints. Heuristic evaluations offer quick, cost-effective insights, while usability studies provide in-depth user feedback. Task analysis enhances both methods by focusing on user tasks and uncovering challenges. By integrating these approaches, you can achieve a comprehensive and user-centric evaluation of your product.

When Do You Need a Usability Study vs. a Heuristic Evaluation? Where Does Task Analysis Fit In?

Understanding when to use a usability study versus a heuristic evaluation can significantly impact the effectiveness of your product development process. Here’s a breakdown of each method, how task analysis fits in, and examples of how these methods have been applied in our day-to-day work.

HEURISTIC EVALUATION

What is it? A heuristic evaluation involves usability experts assessing a design based on established “rules of thumb.” Historically used for human-computer interaction systems, this method evaluates software interfaces for usability. Building on well-established guidelines, experts have tailored 14 Usability Heuristics specifically for medical devices. We apply these 14 heuristics to conduct thorough heuristic evaluations, ensuring usability excellence in client projects.

Why Use It? Heuristic evaluations are quick and cost-effective, making them ideal when time and resources are limited. They provide valuable insights early in the product development process, even before a fully-fledged prototype is available. Our approach includes:

  • Expert Reviews: Conducted by seasoned usability experts.
  • Tailored Heuristics: Using our custom heuristics for medical devices.
  • Comprehensive Analysis: Identifying potential areas for improvement and their impacts.

FORMATIVE USABILITY STUDY

What is it? At a high level, a usability study typically involves simulated use testing of a product to observe how users interact with it. Usability studies inform product design, identify use-related risks, and can help engineers and designers discover the root cause of use errors to include risk mitigations in the product’s design. This evaluation method uses representative users in a representative environment to gather user feedback on specific product components, or the product as a whole.

Why Use It? For medical devices and combination products, FDA requirements necessitate usability studies to ensure device safety and effectiveness. These studies help iterate through device design and thoroughly evaluate components. Compared to heuristic evaluations, usability studies show how end users interact with the device and reveal more opportunities for design improvements.

Our approach includes:

  • Simulated Use Testing: Observing real users in a controlled environment.
  • FDA Compliance: Ensuring all regulatory requirements are met.
  • Comprehensive Study Design: Simulating a representative use environment, recruiting representative users, and applying study findings to design recommendations.

TASK ANALYSIS


What is it? Task analysis goes hand in hand with both heuristic evaluations and usability studies. It involves breaking down the tasks users will perform with the device to understand their needs and challenges.

Why Use It? It provides a deeper understanding of user interactions, enhancing the effectiveness of heuristic evaluations by ensuring that the heuristics are applied in a context that reflects real user behavior. It also enriches usability studies by identifying specific tasks that need to be tested, ensuring comprehensive coverage of user interactions. Our task analysis process includes:

  • User-Centric Focus: Understanding user needs and challenges.
  • Ethnographic Methods: Observing users in their natural environment.
  • Detailed Task Breakdown: Analyzing each step users take with the product.

 

How These Methods Work Together
By combining heuristic evaluations, usability studies, and task analysis, you can ensure a thorough and user-centric design of your product. These evaluation methods complement each other, providing a comprehensive understanding of both potential usability issues and real-world user interactions.

For more detailed examples and insights, check out our case studies on heuristic evaluations and usability studies. These case studies illustrate how Kaleidoscope Innovation has successfully applied these methods to improve product design and usability.

 

For over 7 years, Kaleidoscope Innovation has been a trusted partner to industry leaders like Eli Lilly, Pfizer, and Baxter, helping bring safer, smarter medical products to market. Our integrated Human Factors expertise ensures that the right evaluation methods—whether heuristic, usability-focused, or task-based—are applied at the right time. Whether you're developing a new device or improving an existing one, we’re here to guide your team with insights that reduce risk, streamline development, and enhance user experience. Let’s talk about how we can help you choose and apply the best evaluation methods for your product.

Back to Insights + News

Author

  • Taylor Morgan

    Human Factors Engineer | [email protected]

    Taylor is a Human Factors Engineer at Kaleidoscope Innovation. She brings experience from roles in Human Factors, Research and Design, and Clinical Research. Her background in Human Factors Engineering, combined with her collaborative approach, ensures that user-centered design is seamlessly integrated into every project.

Formative Human Factors Studies Elevate the Design Process and Ultimately Drive Market Success

A formative human factors study plays a pivotal role in the design process, particularly in industries where usability directly impacts safety, efficiency, and regulatory compliance. This research method, typically conducted during the early to mid-stages of the design process, provides actionable insights to align the product with real-world user needs and workflows. Beyond improving usability, formative studies deliver significant business value by reducing costly redesigns and enhancing a product's competitive edge.

When to Conduct a Formative Human Factors Study

Formative studies are most effective during the early stages of the design process, when prototypes or workflows remain flexible. By iterating on the design based on user feedback, teams can identify and resolve usability issues before they escalate into expensive fixes later in development. Repeating formative studies at key points throughout the design process ensures that the product evolves in step with user expectations and emerging requirements.

Why Conduct a Formative Study?

The primary goal of formative studies is to proactively identify potential user challenges, especially those that could lead to errors, frustration, or inefficiency. In high-stakes industries like healthcare, addressing these challenges early helps mitigate risks, enhance safety, and ensure compliance with standards such as the FDA’s human factors guidance for medical devices.

From a business perspective, formative studies create products that resonate more effectively with users, driving customer satisfaction and market adoption. They also streamline regulatory approval by ensuring the product design adheres to usability and safety standards, reducing delays and associated costs.

Ethnographic Methods in Formative Studies

Ethnographic research is a cornerstone of formative human factors studies, offering deep insights into how users interact with products in real-world settings. Methods such as observational studies, contextual inquiries, and think-aloud protocols help uncover hidden pain points and inefficiencies that might not emerge in controlled environments. For example, observing medical professionals using a device in a clinical setting can reveal critical workflow issues that may be overlooked in a simulated lab.

These methods allow designers and engineers to empathize with users, enabling them to create products that better accommodate user environments, limitations, and preferences. This user-centered approach enhances product usability and builds a reputation for intuitive, high-quality solutions—key drivers of customer loyalty.

FDA Guidance and Compliance

In regulated industries, following FDA human factors guidelines is essential. The FDA emphasizes that human factors studies, including formative testing, must demonstrate that users can safely and effectively operate the product under real-world conditions. Formative studies aligned with these guidelines provide evidence of due diligence and risk mitigation, paving the way for smoother regulatory submissions.

The Business Value of Formative Studies

Beyond usability and safety improvements, formative studies deliver significant business benefits. By identifying usability issues early in the design process, these studies help minimize development costs and prevent expensive late-stage redesigns. Products created with a strong focus on user needs are more likely to succeed in the marketplace, driving customer satisfaction and loyalty. Additionally, addressing regulatory requirements early reduces the risk of delays or setbacks in bringing products to market.

Conclusion

Formative human factors studies are more than a tool for improving usability—they are a strategic investment in product success. By integrating ethnographic methods and adhering to FDA guidelines, these studies enhance safety, usability, and user satisfaction while providing measurable business advantages. Incorporating formative studies at the right points in the design process enables companies to create products that are safer, more effective, and better aligned with user needs, all while optimizing costs and time to market.

For over 7 years, Kaleidoscope Innovation has been a trusted partner for industry leaders like Eli Lilly, Pfizer, and Baxter to bring safer, smarter medical products to life. Our integrated Human Factors expertise ensures that usability is built in, helping you reduce risk, accelerate development, and deliver intuitive, high-performing solutions. Whether you're launching a new product or refining an existing one, we’re here to support every step of the process. Let’s talk about how we can help elevate your design through formative research.

Back to Insights + News

Author

Human Factors Engineering Process for Medical Devices

In the medical product industry, ensuring the safety, efficacy, and usability of products is paramount. One of the key methods for achieving this goal is through the proper implementation of a robust Human Factors engineering (also known as usability engineering or ergonomics) process. Human Factors engineering (HFE) focuses on optimizing the interaction between people and technology to prevent errors, enhance performance, and improve the overall user experience.

Because of the risks involved, medical device development demands an especially thorough Human Factors process. They often require precise operation under conditions where an error can result in reduced clinical effectiveness, harm, or even death. Whether these products are used by health care providers, caregivers, or even the patients themselves, medical products that are not designed with the capabilities of the end user in mind can increase the chance of errors. To mitigate these risks, regulatory bodies such as the U.S. Food and Drug Administration (FDA) mandate that Human Factors principles be integrated into the design and development of medical products.

The HFE process for medical products is a structured, iterative approach that aims to identify potential use-related risks early in the design process and systematically reduce those risks throughout product development. This process incorporates both user needs and environmental considerations to create intuitive, safe, and effective products.

Human Factors process

User Research and Task Analysis

The first step in the HFE process is to develop a deep understanding of the intended users and the tasks they will perform with the product. It is important to recognize that there may be distinct types of users (user groups) that have varying tasks, capabilities, and responsibilities. This step involves the following activities:

  • User needs: Identify the specific requirements and expectations of users that the product needs to fulfill to be used safely and effectively. These needs generate testable design requirements.
  • User characteristics: Understand the characteristics of the intended users that could impact product use. Common characteristics include age range, experience, education, physical abilities, cognitive capabilities, and any disabilities. These characteristics may drive segmentation of the intended users into separate user groups for usability testing.
  • Use environment: Identify factors such as lighting, noise, and other products in the intended use environment that may impact use of the product, keeping in mind there may be multiple environments in which the product is used, such as a clinic or a home setting.
  • Task Analysis: Based on the intended use of the product, identify and break down the tasks that users must accomplish to safely and effectively use the product. Read about Task Analysis here.

Identifying and Analyzing Risks

Once the user characteristics, environments, and tasks are understood, the next step is to conduct a use-related risk analysis (URRA). This process involves:

  • identification of what and how use errors may occur
  • assessment of the potential consequences and severity of those errors
  • identification of potential mitigations
  • validation plans for those mitigations

By systematically analyzing use-related risks, the design team can prioritize which potential issues are most critical to address based on severity, likelihood, and impact on patient safety. Additionally, this is the step in which critical tasks are identified. According to the FDA guidance document “Applying Human Factors and Usability Engineering to Medical Devices,” critical tasks are those which, if performed incorrectly or not performed at all, would or could cause serious harm to the patient or user, where harm is defined to include compromised medical care. Identification of critical tasks drives creation of the usability validation protocol, to ensure that any critical use error mitigations are properly validated.

The URRA can be considered a “living document,” since it should be continuously updated throughout the product development process when new hazards are identified through usability testing and existing hazards are mitigated through design.

Human Factors processDesign and Prototyping

Human Factors specialists collaborate closely with designers and engineers—and and at Kaleidoscope, that means working under the same roof—to create prototypes that align with user needs and risk mitigation strategies. This integrated approach streamlines communication, accelerates iteration, and results in more effective, user-centered designs. Multiple prototypes may be developed that can be evaluated with representative users in the next phase.

Human Factors processFormative Usability Testing and Evaluation

Once prototypes are developed, formative evaluation is conducted to preliminarily assess the use-related risk mitigations as well as the degree to which the product is effective and easy to use. This is an iterative process, where feedback may be used to update user needs, user requirements, and the use-related risk analysis. This stage will often result in updates to the product user interface and labeling, as well as a determination whether and to what degree user training will be required.

There are two broad categories of formative evaluation, based on the degree to which representative users and use environment is involved. Read about Heuristic vs. Usability Studies here.

  • Heuristic or Expert Reviews: “Quick and dirty” evaluations involving the assessment of a design by usability experts based on established Human Factors heuristics (rules of thumb).
  • Formative Usability Testing: Usability testing with representative users by simulating product use in realistic scenarios. Formative testing can be performed either with a subset of tasks or the entire system, and with everything from early-stage prototypes to near validation-ready products. Read about Formative Studies here.

Summative Usability Validation

Once formative evaluations have demonstrated that the use-related risk mitigations are adequate and the product is sufficiently usable, usability validation testing (also referred to as summative evaluation) can be conducted.

The usability validation process involves formally validating that the intended users can safely and effectively use the product to perform all critical tasks, that all identified risks are adequately mitigated, and that the product performs as expected under (usually simulated) real-world conditions. Testing should simulate actual use conditions as closely as possible, with a sufficient sample size of representative users of all intended user groups.

For example, when Tandem Diabetes Care developed the world’s smallest insulin pump—Tandem Mobi—they partnered with Kaleidoscope to support formative and summative studies for FDA submission (learn more here).

Once usability validation is complete, a comprehensive Human Factors report is created as part of the regulatory submission for product approval.

Conclusion

The Human Factors engineering process for a medical product is complex and requires the expertise of experienced professionals to fully realize the potential of the product concept. But the result of a properly scoped and applied Human Factors process is a product that is safe, effective, and easy to use, making it more likely to result in a successful regulatory submission and ultimate success in the market.

For over 7 years, Kaleidoscope Innovation has been a trusted partner in Human Factors engineering, supporting industry leaders like Eli Lilly, Pfizer, and Baxter. Whether you're bringing a new device to market or refining an existing product, our team brings deep expertise and real-world insight to every stage of development. Let’s talk about how we can help you design safer, smarter, and more user-friendly medical products.

Back to Insights + News

Author