Tag: Logistics

Choosing the right evaluation method depends on your specific needs and constraints. Heuristic evaluations offer quick, cost-effective insights, while usability studies provide in-depth user feedback. Task analysis enhances both methods by focusing on user tasks and uncovering challenges. By integrating these approaches, you can achieve a comprehensive and user-centric evaluation of your product.

When Do You Need a Usability Study vs. a Heuristic Evaluation? Where Does Task Analysis Fit In?

Understanding when to use a usability study versus a heuristic evaluation can significantly impact the effectiveness of your product development process. Here’s a breakdown of each method, how task analysis fits in, and examples of how these methods have been applied in our day-to-day work.

HEURISTIC EVALUATION

What is it? A heuristic evaluation involves usability experts assessing a design based on established “rules of thumb.” Historically used for human-computer interaction systems, this method evaluates software interfaces for usability. Building on well-established guidelines, experts have tailored 14 Usability Heuristics specifically for medical devices. We apply these 14 heuristics to conduct thorough heuristic evaluations, ensuring usability excellence in client projects.

Why Use It? Heuristic evaluations are quick and cost-effective, making them ideal when time and resources are limited. They provide valuable insights early in the product development process, even before a fully-fledged prototype is available. Our approach includes:

• Expert Reviews: Conducted by seasoned usability experts.
• Tailored Heuristics: Using our custom heuristics for medical devices.
• Comprehensive Analysis: Identifying potential areas for improvement and their impacts.

FORMATIVE USABILITY STUDY

What is it? At a high level, a usability study typically involves simulated use testing of a product to observe how users interact with it. Usability studies inform product design, identify use-related risks, and can help engineers and designers discover the root cause of use errors to include risk mitigations in the product’s design. This evaluation method uses representative users in a representative environment to gather user feedback on specific product components, or the product as a whole.

Why Use It? For medical devices and combination products, FDA requirements necessitate usability studies to ensure device safety and effectiveness. These studies help iterate through device design and thoroughly evaluate components. Compared to heuristic evaluations, usability studies show how end users interact with the device and reveal more opportunities for design improvements.

Our approach includes:

• Simulated Use Testing: Observing real users in a controlled environment.
• FDA Compliance: Ensuring all regulatory requirements are met.
• Comprehensive Study Design: Simulating a representative use environment, recruiting representative users, and applying study findings to design recommendations.

TASK ANALYSIS

What is it? Task analysis goes hand in hand with both heuristic evaluations and usability studies. It involves breaking down the tasks users will perform with the device to understand their needs and challenges.

Why Use It? It provides a deeper understanding of user interactions, enhancing the effectiveness of heuristic evaluations by ensuring that the heuristics are applied in a context that reflects real user behavior. It also enriches usability studies by identifying specific tasks that need to be tested, ensuring comprehensive coverage of user interactions. Our task analysis process includes:

• User-Centric Focus: Understanding user needs and challenges.
• Ethnographic Methods: Observing users in their natural environment.
• Detailed Task Breakdown: Analyzing each step users take with the product.

How These Methods Work Together
By combining heuristic evaluations, usability studies, and task analysis, you can ensure a thorough and user-centric design of your product. These evaluation methods complement each other, providing a comprehensive understanding of both potential usability issues and real-world user interactions.

For more detailed examples and insights, check out our case studies on heuristic evaluations and usability studies. These case studies illustrate how Kaleidoscope Innovation has successfully applied these methods to improve product design and usability.

For over 7 years, Kaleidoscope Innovation has been a trusted partner to industry leaders like Eli Lilly, Pfizer, and Baxter, helping bring safer, smarter medical products to market. Our integrated Human Factors expertise ensures that the right evaluation methods—whether heuristic, usability-focused, or task-based—are applied at the right time. Whether you're developing a new device or improving an existing one, we’re here to guide your team with insights that reduce risk, streamline development, and enhance user experience. Let’s talk about how we can help you choose and apply the best evaluation methods for your product.

A Time & Motion (T&M) study can be a valuable addition to a user-centered design process. Time & Motion studies are usually conducted to identify potential bottlenecks in productivity but can also identify physiological risks associated with working in a warehouse environment, factory, health care environment or lab setting. Significant enhancements in productivity have been linked to ergonomically designed workspaces, leading to better worker morale and increased revenue due to reduced cycle times and fewer repetitive stress injuries.

At Kaleidoscope, we perform several time & motion studies for our clients every year. One of the advantages of this research technique are the insights gained by observing actual users performing the workflow in context and in real time. Our process for conducting a T&M study usually follows this sequence:

1. Meet with stakeholders to define the target users and workflow, and to determine if user experience and motion data will be in scope. If motion data is required, collaborate with ergonomic engineers to coordinate efforts. 
2. Schedule onsite data collection. Send enough researchers to collect observational data, operate recording equipment, and conduct contextual interviews with participants.
3. Extract data from video footage through frame-by-frame video manipulation. Analyze data, conduct descriptive statistical analysis and inferential analysis when appropriate. Identify insights and themes relevant to the research question(s). 
4. Synthesize and present results to stakeholders. When possible, compare current results with historical data to view changes in time requirements that could be related to workspace/workstation design improvements. Incorporate user experience findings to give research participants a voice in future workstation design changes and continue Kaleidoscope’s commitment to human-centered design and research. 

WHAT IS NOT MEASURED IN A TIME & MOTION STUDY

Time and motion studies are ideal for identifying productivity obstructions and potentially unsafe body movements. Not all the important factors related to employees’ work experiences are measured, however. As technology and infrastructure become more robust and complex, we have responded by expanding our capabilities to provide value for our clients. We share one challenge here to illustrate the complexity of studying modern warehouse environments. 

Challenge

• Time studies often target one piece of a complex system.
  • When only one component of a facility/system is studied, external factors that influence time requirements may not be observable or included in data collection.
  • Changes made to the target component may create unanticipated changes elsewhere in a facility or system.
  • Factors out of employees’ control may be misinterpreted as inefficiency.

Potential Solutions

• It may be helpful to apply systems thinking near the beginning of a time & motion project.
  • Consider the larger system structure within which the T&M study is taking place.
  • Document who may be influenced by the design or redesign of the target component.
  • Discuss what stakeholders prioritize that could be influenced by the redesign of the target component.
• Consider whether there are opportunities to change the system structure.
  • At what points can we intervene?
  • If possible, design interventions that benefit the entire system.
• Map the system, even parts outside the area of focus. Create a visualization that allows stakeholders to envision how the system might react to changes in its structure (e.g., process map, schematic illustration, storyboard, animation). 

Potential Add-Ons

To maximize the human potential embedded within workplaces, other research techniques may be added to time & motion studies for an even greater degree of comprehension.

• Visual Aids: Diagrams of the facility the study is based on may help the audience better understand research findings.
• Surveys: Surveys provide an inexpensive method of gathering large amounts of data quickly. Often, responses are provided in numeric format which allows for historical comparisons.
• Interviews: 1:1 or group interviews may be added to a time & motion study to gain an understanding of time requirement results. Contextual information known to participants but not researchers may be shared in an interview to provide a deeper understanding of the “why” behind observed time requirements.

THE HUMAN COST OF EFFICIENCY IMPROVEMENTS 

Any improvements to efficiency should be weighed against the human cost to the workers employed in the facilities we study. If efficiency improvements create a stressor where none was present, carefully consider whether the cost is worth the price. Constant time pressure and feeling hurried will take a toll on even the hardiest employees. Consider workarounds that value the worker, and which place them at the center of decision-making. The payoff in retention and increased job satisfaction will likely outweigh any efficiency improvements under consideration.