Formative Human Factors Studies Elevate the Design Process and Ultimately Drive Market Success
A formative human factors study plays a pivotal role in the design process, particularly in industries where usability directly impacts safety, efficiency, and regulatory compliance. This research method, typically conducted during the early to mid-stages of the design process, provides actionable insights to align the product with real-world user needs and workflows. Beyond improving usability, formative studies deliver significant business value by reducing costly redesigns and enhancing a product's competitive edge.
When to Conduct a Formative Human Factors Study
Formative studies are most effective during the early stages of the design process, when prototypes or workflows remain flexible. By iterating on the design based on user feedback, teams can identify and resolve usability issues before they escalate into expensive fixes later in development. Repeating formative studies at key points throughout the design process ensures that the product evolves in step with user expectations and emerging requirements.
Why Conduct a Formative Study?
The primary goal of formative studies is to proactively identify potential user challenges, especially those that could lead to errors, frustration, or inefficiency. In high-stakes industries like healthcare, addressing these challenges early helps mitigate risks, enhance safety, and ensure compliance with standards such as the FDA’s human factors guidance for medical devices.
From a business perspective, formative studies create products that resonate more effectively with users, driving customer satisfaction and market adoption. They also streamline regulatory approval by ensuring the product design adheres to usability and safety standards, reducing delays and associated costs.
Ethnographic Methods in Formative Studies
Ethnographic research is a cornerstone of formative human factors studies, offering deep insights into how users interact with products in real-world settings. Methods such as observational studies, contextual inquiries, and think-aloud protocols help uncover hidden pain points and inefficiencies that might not emerge in controlled environments. For example, observing medical professionals using a device in a clinical setting can reveal critical workflow issues that may be overlooked in a simulated lab.
These methods allow designers and engineers to empathize with users, enabling them to create products that better accommodate user environments, limitations, and preferences. This user-centered approach enhances product usability and builds a reputation for intuitive, high-quality solutions—key drivers of customer loyalty.
FDA Guidance and Compliance
In regulated industries, following FDA human factors guidelines is essential. The FDA emphasizes that human factors studies, including formative testing, must demonstrate that users can safely and effectively operate the product under real-world conditions. Formative studies aligned with these guidelines provide evidence of due diligence and risk mitigation, paving the way for smoother regulatory submissions.
The Business Value of Formative Studies
Beyond usability and safety improvements, formative studies deliver significant business benefits. By identifying usability issues early in the design process, these studies help minimize development costs and prevent expensive late-stage redesigns. Products created with a strong focus on user needs are more likely to succeed in the marketplace, driving customer satisfaction and loyalty. Additionally, addressing regulatory requirements early reduces the risk of delays or setbacks in bringing products to market.
Conclusion
Formative human factors studies are more than a tool for improving usability—they are a strategic investment in product success. By integrating ethnographic methods and adhering to FDA guidelines, these studies enhance safety, usability, and user satisfaction while providing measurable business advantages. Incorporating formative studies at the right points in the design process enables companies to create products that are safer, more effective, and better aligned with user needs, all while optimizing costs and time to market.
For over 7 years, Kaleidoscope Innovation has been a trusted partner for industry leaders like Eli Lilly, Pfizer, and Baxter to bring safer, smarter medical products to life. Our integrated Human Factors expertise ensures that usability is built in, helping you reduce risk, accelerate development, and deliver intuitive, high-performing solutions.Whether you're launching a new product or refining an existing one, we’re here to support every step of the process. Let’s talk about how we can help elevate your design through formative research.
Human Factors Engineering Process for Medical Devices
In the medical product industry, ensuring the safety, efficacy, and usability of products is paramount. One of the key methods for achieving this goal is through the proper implementation of a robust Human Factors engineering (also known as usability engineering or ergonomics) process. Human Factors engineering (HFE) focuses on optimizing the interaction between people and technology to prevent errors, enhance performance, and improve the overall user experience.
Because of the risks involved, medical device development demands an especially thorough Human Factors process. They often require precise operation under conditions where an error can result in reduced clinical effectiveness, harm, or even death. Whether these products are used by health care providers, caregivers, or even the patients themselves, medical products that are not designed with the capabilities of the end user in mind can increase the chance of errors. To mitigate these risks, regulatory bodies such as the U.S. Food and Drug Administration (FDA) mandate that Human Factors principles be integrated into the design and development of medical products.
The HFE process for medical products is a structured, iterative approach that aims to identify potential use-related risks early in the design process and systematically reduce those risks throughout product development. This process incorporates both user needs and environmental considerations to create intuitive, safe, and effective products.
User Research and Task Analysis
The first step in the HFE process is to develop a deep understanding of the intended users and the tasks they will perform with the product. It is important to recognize that there may be distinct types of users (user groups) that have varying tasks, capabilities, and responsibilities. This step involves the following activities:
User needs: Identify the specific requirements and expectations of users that the product needs to fulfill to be used safely and effectively. These needs generate testable design requirements.
User characteristics: Understand the characteristics of the intended users that could impact product use. Common characteristics include age range, experience, education, physical abilities, cognitive capabilities, and any disabilities. These characteristics may drive segmentation of the intended users into separate user groups for usability testing.
Use environment: Identify factors such as lighting, noise, and other products in the intended use environment that may impact use of the product, keeping in mind there may be multiple environments in which the product is used, such as a clinic or a home setting.
Task Analysis: Based on the intended use of the product, identify and break down the tasks that users must accomplish to safely and effectively use the product. Read about Task Analysis here.
Identifying and Analyzing Risks
Once the user characteristics, environments, and tasks are understood, the next step is to conduct a use-related risk analysis (URRA). This process involves:
identification of what and how use errors may occur
assessment of the potential consequences and severity of those errors
identification of potential mitigations
validation plans for those mitigations
By systematically analyzing use-related risks, the design team can prioritize which potential issues are most critical to address based on severity, likelihood, and impact on patient safety. Additionally, this is the step in which critical tasks are identified. According to the FDA guidance document “Applying Human Factors and Usability Engineering to Medical Devices,” critical tasks are those which, if performed incorrectly or not performed at all, would or could cause serious harm to the patient or user, where harm is defined to include compromised medical care. Identification of critical tasks drives creation of the usability validation protocol, to ensure that any critical use error mitigations are properly validated.
The URRA can be considered a “living document,” since it should be continuously updated throughout the product development process when new hazards are identified through usability testing and existing hazards are mitigated through design.
Design and Prototyping
Human Factors specialists collaborate closely with designers and engineers—and and at Kaleidoscope, that means working under the same roof—to create prototypes that align with user needs and risk mitigation strategies. This integrated approach streamlines communication, accelerates iteration, and results in more effective, user-centered designs. Multiple prototypes may be developed that can be evaluated with representative users in the next phase.
Formative Usability Testing and Evaluation
Once prototypes are developed, formative evaluation is conducted to preliminarily assess the use-related risk mitigations as well as the degree to which the product is effective and easy to use. This is an iterative process, where feedback may be used to update user needs, user requirements, and the use-related risk analysis. This stage will often result in updates to the product user interface and labeling, as well as a determination whether and to what degree user training will be required.
Heuristic or Expert Reviews: “Quick and dirty” evaluations involving the assessment of a design by usability experts based on established Human Factors heuristics (rules of thumb).
Formative Usability Testing: Usability testing with representative users by simulating product use in realistic scenarios. Formative testing can be performed either with a subset of tasks or the entire system, and with everything from early-stage prototypes to near validation-ready products. Read about Formative Studies here.
Summative Usability Validation
Once formative evaluations have demonstrated that the use-related risk mitigations are adequate and the product is sufficiently usable, usability validation testing (also referred to as summative evaluation) can be conducted.
The usability validation process involves formally validating that the intended users can safely and effectively use the product to perform all critical tasks, that all identified risks are adequately mitigated, and that the product performs as expected under (usually simulated) real-world conditions. Testing should simulate actual use conditions as closely as possible, with a sufficient sample size of representative users of all intended user groups.
For example, when Tandem Diabetes Care developed the world’s smallest insulin pump—Tandem Mobi—they partnered with Kaleidoscope to support formative and summative studies for FDA submission (learn more here).
Once usability validation is complete, a comprehensive Human Factors report is created as part of the regulatory submission for product approval.
Conclusion
The Human Factors engineering process for a medical product is complex and requires the expertise of experienced professionals to fully realize the potential of the product concept. But the result of a properly scoped and applied Human Factors process is a product that is safe, effective, and easy to use, making it more likely to result in a successful regulatory submission and ultimate success in the market.
For over 7 years, Kaleidoscope Innovation has been a trusted partner in Human Factors engineering, supporting industry leaders like Eli Lilly, Pfizer, and Baxter. Whether you're bringing a new device to market or refining an existing product, our team brings deep expertise and real-world insight to every stage of development. Let’s talk about how we can help you design safer, smarter, and more user-friendly medical products.
Unlock Hidden Productivity with Time & Motion Studies
A Time & Motion (T&M) study can be a valuable addition to a user-centered design process. Time & Motion studies are usually conducted to identify potential bottlenecks in productivity but can also identify physiological risks associated with working in a warehouse environment, factory, health care environment or lab setting. Significant enhancements in productivity have been linked to ergonomically designed workspaces, leading to better worker morale and increased revenue due to reduced cycle times and fewer repetitive stress injuries.
At Kaleidoscope, we perform several time & motion studies for our clients every year. One of the advantages of this research technique are the insights gained by observing actual users performing the workflow in context and in real time. Our process for conducting a T&M study usually follows this sequence:
Meet with stakeholders to define the targetusers and workflow, and to determine if user experience and motion data will be in scope. If motion data is required, collaborate with ergonomic engineers to coordinate efforts.
Schedule onsite data collection. Send enough researchers to collect observational data, operate recording equipment, and conduct contextual interviews with participants.
Extract data from video footage throughframe-by-frame video manipulation. Analyze data, conduct descriptive statistical analysis and inferential analysis when appropriate. Identify insights and themes relevant to the research question(s).
Synthesize and present results to stakeholders. When possible, compare current results with historical data to view changes in time requirements that could be related to workspace/workstation design improvements. Incorporate user experience findings to give research participants a voice in future workstation design changes and continue Kaleidoscope’s commitment to human-centered design and research.
WHAT IS NOT MEASURED IN A TIME & MOTION STUDY
Time and motion studies are ideal for identifyingproductivityobstructions and potentially unsafe body movements. Not all the important factors related to employees’ work experiences are measured, however. As technology and infrastructure become more robust and complex, we have responded by expanding our capabilities to provide value for our clients. We share one challenge here to illustrate the complexity of studying modern warehouse environments.
Challenge
Time studies often target one piece of a complex system.
When only one component of a facility/system is studied, external factors that influence time requirements may not be observable or included in data collection.
Changes made to the target component may create unanticipated changes elsewhere in a facility or system.
Factors out of employees’ control may be misinterpreted as inefficiency.
Potential Solutions
It may be helpful to apply systems thinking near the beginning of a time & motion project.
Consider the larger system structure within which the T&M study is taking place.
Document who may be influenced by the design or redesign of the target component.
Discuss what stakeholders prioritize that could be influenced by the redesign of the target component.
Consider whether there are opportunities to change the system structure.
At what points can we intervene?
If possible, design interventions that benefit the entire system.
Map the system, even parts outside the area of focus. Create a visualization that allows stakeholders to envision how the system might react to changes in its structure (e.g., process map, schematic illustration, storyboard, animation).
Potential Add-Ons
To maximize the human potential embedded within workplaces, other research techniques may be added to time & motion studies for an even greater degree of comprehension.
Visual Aids: Diagrams of the facility the study is based on may help the audience better understand research findings.
Surveys: Surveys provide an inexpensive method of gathering large amounts of data quickly. Often, responses are provided in numeric format which allows for historical comparisons.
Interviews: 1:1 or group interviews may be added to a time & motion study to gain an understanding of time requirement results. Contextual information known to participants but not researchers may be shared in an interview to provide a deeper understanding of the “why” behind observed time requirements.
THE HUMAN COST OF EFFICIENCY IMPROVEMENTS
Any improvements to efficiency should be weighed against the human cost to the workers employed in the facilities we study. If efficiency improvements create a stressor where none was present, carefully consider whether the cost is worth the price. Constant time pressure and feeling hurried will take a toll on even the hardiest employees. Consider workarounds that value the worker, and which place them at the center of decision-making. The payoff in retention and increased job satisfaction will likely outweigh any efficiency improvements under consideration.
Rachael brings over 10 years of research experience to her role at Kaleidoscope Innovation. She has advanced training in clinical psychology and mixed methods research methodology. Guided by the principles of positive psychology, Rachael uses a human-centered lens for deeply understanding the user experience. Her work at Kaleidoscope focuses on human-machine interaction and identifying design changes capable of positively impacting well-being at the individual and institutional levels.
Breaking Down the FDA's New URRA Guidance: July 2024
The FDA has recently released a draft guidance on Use-Related Risk Analyses (URRAs) for drugs, biological products, and combination products. This new guidance consolidates previously available information into one comprehensive document, making it easier for industry stakeholders to access and apply the guidelines effectively.
KEY HIGHLIGHTS OF THE GUIDANCE:
Purpose: The guidance aims to explain identifying use-related hazards and implement measures to reduce associated risks. This is crucial for ensuring that medical products are safe and effective for their intended users.
Scope: The guidance is applicable to drugs, biological products, and combination products. This broad scope ensures that a wide range of products are covered, promoting consistency in risk management practices across different types of medical products.
Process: URRAs should be initiated early in product development and updated throughout the product lifecycle. This proactive approach helps in identifying potential risks early and allows for timely mitigation strategies.
Consolidation: This guidance combines previously available information into one comprehensive document, streamlining the process for stakeholders. This consolidation helps in reducing confusion and ensures that all relevant information is easily accessible.
WHY THIS MATTERS:
The new guidance ensures the safety and effectiveness of medical products by helping to determine the need for Human Factors (HF) study results in new marketing applications. By identifying and mitigating use-related risks early in the product development process, companies can enhance the overall safety and usability of their products. This not only protects end-users but also helps companies avoid costly recalls and redesigns.
KALEIDOSCOPE'S EXPERTISE:
At Kaleidoscope Innovation, we specialize in Human Factors for medical products, and have already integrated FDA guidance's into our process. Our team of experts interprets the guidance and applies it directly to your FDA submissions, ensuring you meet the latest requirements. Our HF team works closely with our regulatory team to help you understand the complexities of the regulatory landscape and provide tailored solutions to meet your specific needs.
HOW WE CAN HELP:
Interpretation: Our experts can help you understand the nuances of the new guidance. We provide detailed explanations and practical insights to ensure you fully grasp the requirements.
Application: We assist in applying the guidance to your product development and FDA submissions. Our team can help you develop comprehensive Use-Related Risk Analyses (URRAs) that meet FDA standards.
Compliance: Ensure your products meet the latest FDA requirements with our comprehensive consulting services. We offer ongoing support to help you stay compliant throughout the product lifecycle.
Taylor is a Human Factors Engineer at Kaleidoscope Innovation. She brings experience from roles in Human Factors, Research and Design, and Clinical Research. Her background in Human Factors Engineering, combined with her collaborative approach, ensures that user-centered design is seamlessly integrated into every project.
Eradicating Eye Disease
PulseSight (Formally Eyevensys) approached Kaleidoscope’s ID team with the following need:
“Help us develop and train ophthalmologists on how to perform our innovative electrotransfection gene therapy procedure.”
Unlike most pharmaceuticals that treat conditions of the eye, PulseSight’s non-viral gene therapy teaches the eye to heal itself. Very cool stuff!
So, Kaleidoscope’s designers got to work, collaborating with surgeons, practicing the procedure in situ, not only illustrating but also developing techniques for the procedure itself. The final procedure guide and accompanying trainer helped ensure PulseSight’s clinical trial progressed even more smoothly.
“Sunny” is the nickname for the anatomical model Kaleidoscope’s design and engineering team developed to train retinal surgeons on the innovative new Electrotransfection Gene Therapy pioneered by PulseSight.
To properly prepare surgeons for the novel obstacles experienced in the procedure, Kaleidoscope’s team of designers created this model to accurately simulate limited orbital access and the rotational effects of the rectus muscles of the human eye.
While Sunny doesn’t talk much, he definitely has seen a lot!
As experts in the field of Ophthalmic drug delivery devices, Kaleidoscope's team helped redesign the product from the ground up, improving the overall usability of the device and the efficacy and accuracy of the drug delivery.
Jake Schubert is a Senior Lead Industrial Designer at Kaleidoscope Innovation. His technical expertise and collaboration with our engineering team are instrumental in ensuring that user needs, design, and engineering intent converge seamlessly in our market-ready products.
Virtual Tools for Innovative Product Design
Design influences a product’s lifecycle performance and cost, starting from its development. Product development costs rise significantly if a defect is identified at a later stage. Using virtual tools for new product introduction simulates possible scenarios upfront for comprehensive testing. It gets products to the market quickly and saves money for a successful launch.
Insights
Design influences a product’s lifecycle performance and cost, starting from its development.
Conceptualization and design stages determine more than 70% of a product’s lifecycle decisions and cost.
Virtual tools are an effective way to design new products that serve specific customer needs.
Virtual models of new products accelerate their evaluations to shrink the development cycle time.
Organizations should create virtual replicas of workplaces for human-machine interactions studies from multiple perspectives.
Lifecycle cost is the total cost (direct and indirect) a product incurs in its life span. Conceptualization and design stages determine more than 70% of a product’s lifecycle decisions and cost.1 The earlier an issue is identified, specifically in the design stage, the easier it is to fix and avoid costly rework. Virtual replicas (or digital twins) of products, processes, and environments streamline design and new product development to reduce costs and time to market.
A common assertion is between 80% and 90% of new products fail. However, realistic failure rates vary by industry, from 36% in healthcare to 45% in consumer goods.2 Professor Clayton Christensen, best known for his theory of disruptive innovation, believes the success mantra is to design products that serve its intended customers. Manufacturers should focus on the function that a customer who buys a product would want it to do.3
To enable that, virtual representations of the product under development, in orchestration with humans and other entities in the ecosystem, is an effective approach. The approach encourages innovation. Designers visualize the product’s operating condition, create digital prototypes for trial runs, and carry out tests on a global scale. Virtual tools like 3D computer models and digital twins support informed decisions in early product design stages. This mitigates the risk of a wrong product release or a poor customer experience.
→ Virtual products are an effective way to design new products that serve specific customer needs
When end users receive virtual training of a complicated product’s operation (like an aircraft engine), memory retention happens in the background. Any number of such instances can be created at a negligible marginal cost for repetitive usage. A central digital setup saves the cost of setting up multiple physical arrangements at different locations.
PARAMETERS OF SUCCESSFUL NEW PRODUCTS
Product failures are more from a commercial perspective than technical. More than 25% of revenue and profits across industries come from new products, according to a study by McKinsey. Successful products relate to a set of core capabilities, with the top-most as follows:4
Collaboration to execute tasks as a team.
Investment to mine market insights and their inclusion in the product.
Plans for new product launches, comprising target customer segments, key messages to communicate, and objectives to achieve.
Talent development for new product launches with defined career paths and incentives.
At the same time, the primary reasons for product failures and mitigants are the following:5
Gap in meeting product expectations; delay launch until product completion.
Inability to support rapid growth if a product is successful; set ramp-up plans to avoid this.
Low demand for a new product; perform due diligence for customer requirement before planning a product. Launch products in suitable markets.
Difficulty in new product usage; provide proper customer orientation and training.
Virtual tools for product design address the above reasons for failure and increase the chances of successful product launches.
DESIGN THINKING WITH VIRTUAL TOOLS
Design thinking is a popular, technology-agnostic approach for new systems design and problem solving. It balances the technical feasibility of products, financial viability, and desirability from a customer’s perspective (see Figure 1). It is even more impactful when implemented along with virtual product design tools.
Figure 1. Design thinking at the sweet spot of desirability, viability, and feasibility
Source: Infosys
The design thinking cycle starts from empathy to understand a customer’s needs from their perspective, followed by defining, ideating, prototyping, and validating, in iterative loops. New product development and customer participation encourage collaboration in a virtual environment to practice design thinking. Immersive environments using mixed reality (combinations of augmented reality or AR and virtual reality or VR) create a working environment close to the real world, to identify and correct issues much ahead (see Figure 2).
Figure 2. Virtual tools used across design thinking stages
Source: Infosys
→ Virtual models of new products accelerate their evaluations to shrink the development cycle time
Design firm IDEO, for example, wanted to perform ethnographic research to capture customer requirements for new products. However, it was difficult to identify key observations from many data points and recreate them later, even with expensive videos or photos. It addressed the challenge through a VR camera.6
Kaleidoscope Innovation, a design and development unit within Infosys, designed a large freezer project using virtual tools. Such projects usually undergo several time-consuming team reviews. The team created a 3D model in a VR environment that helped designers walk around the product early in the design phase, evaluate its usability from multiple perspectives, and tackle proposed changes to design.
This virtual model did not change the overall project plan, but accelerated evaluation and decisions around it, shrinking the product development cycle time. The team selected the best design without spending time and money on physical prototypes.
CUSTOMER PRODUCT INTERACTION
Mathematical models of an individual product’s performance are important and popular. However, the product should also be ergonomic for safe and productive human-machine interaction. The operations and other associated systems (such as material handling) should be easy to perform. Its ease of making is important for the manufacturer to ensure quality and cost control. The carbon footprint across the lifecycle and its end-of-life disposal should be sustainable.
For example, a hospital has large equipment such as scanners, working in orchestration with other smaller tools, and humans. Room layout is an important aspect of such an infrastructure. Any change post implementation is expensive.
Infosys has created a three-dimensional room planning VR tool. Non-technical sales team members create, change, import layouts, view them from multiple perspectives along with the human and machine elements, and finalize the layout. That final layout is exported as two-dimensional drawings for implementation. This streamlined the sales process and significantly reduced iterations and sales cycle time.
AUTOMATION IN WAREHOUSES
Humans work with machines in warehouses. Material handlers carry out order fulfillment along with pick-and-place robots. Workers’ safety in all situations is important.
A leading e-commerce player wanted to validate design decisions for robots working in its order fulfillment warehouses to gain insights into their safe working alongside humans. Kaleidoscope Innovation created a virtual environment where employees interacted with robots in different situations. The team created a digital twin to simulate several configurations of robots and their working environment. The company recorded the results and interviewed employees about pros and cons of each situation.
The VR-based solution provided a cost-effective and safe way for the e-commerce firm to test new concepts in human-robot interaction and capture data and feedback before implementation. It helped the managers zoom out and look at the big picture, in contrast to one robot or equipment at a time.
TRAINING FOR PRODUCT USAGE
Operators need training to work on machines with complex functionality and procedures, to stay safe and productive. VR-based training prepares humans before hands-on operation on a machine. For instance, Rolls-Royce has rolled out a VR-based training kit for its airline customers to manage aircraft engine maintenance and repair.
Infosys’s VR-based program provides step-by-step instructions to train employees in a hospital environment. The program uses physical gestures to simulate actual tasks involved in a job. Gamification with scores and points keeps employees engaged and motivated. Scores reflect an individual’s strengths and weaknesses. Training data is integrated with the central learning management system for records.
A multinational industrial and consumer goods manufacturer wanted to create an e-training platform for its new operators. It had a few integrated assembly lines for its finished items. The Kaleidoscope Innovation team created a virtual training module along the assembly line, one workstation at a time. The team used front-end user interface elements to guide users for equipment operations. It tracked performance metrics in the backend to provide feedback for correction. Best practices of creating a virtual replica of one workstation are used at later stations.
FUTURISTIC WORKPLACES
While collaborative, remote and hybrid working has surged since the pandemic, the future is in three-dimensional virtual and mixed reality workspaces. Organizations benefit from a virtual 3D replica of its workspaces, equipment, products, avatars, or personas. Employee collaborations lead to faster new product development with effective interactions. Teams share ideas, explore, and invent new concepts. Early collaboration of team members in multiple locations enables them to make more informed decisions in the product development process.
→ Organizations should create virtual replicas of workplaces for human-machine interactions studies from multiple perspectives
The future of work in healthcare, retail, engineering, and manufacturing is where humans and human-like machines work together. Organizations should proactively create such workspaces virtually and study human-machine interaction from safety, productivity, and employee morale perspectives before any physical implementation.
Myths About New Product Failure Rates, George Castellion, Stephen K. Markham, 2013, published in the Journal of Product Innovation & Management 30 pp. 976-979.
Karl is passionate about creating worthwhile and valuable product design with an amazing team, focused on building beautiful and intuitive experiences prioritizing the user’s needs and expectations. His team brings these visions to reality by collaborating closely with Kaleidoscope’s Insights, Human Factors and Engineering teams. Karl’s Industrial Design background includes working with brands across a wide variety of industries, ranging from healthcare and medical, to consumer and housewares, to industrial products and toys. He has earned more than 40 design and utility patents over the past 20+ years.
Ramachandran S
Principal Consultant at Infosys
Qualitative Research: AI's Role in Analysis Advancement
BACKGROUND
Qualitative research plays a pivotal role in enriching our comprehension of individual narratives and experiences. It is a cornerstone methodology for design researchers seeking to forge a deep connection with user perspectives, particularly during the initial phases of the design process. This approach is instrumental in guiding iterative design developments, ensuring that end-user needs are comprehensively addressed. Qualitative data encompasses a diverse array of formats, including textual content, photographs, and videos. Typically, these studies involve a more focused sample size, often with 10 or fewer participants, to facilitate an intensive, detail-oriented analysis that quantitative methods may not capture.
Although qualitative research is the methodology of choice for design researchers, the approach requires a considerable time commitment. Qualitative data is known for being unwieldy at times, and words and images require more hours of analysis than numeric data. Often, our clients are eager to obtain research findings as quickly as possible to move a product or system into production. Therefore, large scale qualitative studies are not feasible formost design research projects.With the recent surge in the availability of AIlanguage model tools, we speculated thatChatGPT could be used to analyze extremely large sets of qualitative data more efficiently.To that end, we conducted a 6-month project testing ChatGPT as a potential tool for qualitative data analysis.
THE CURRENT PROJECT
Our aim in conducting this project was to determine if AI could produce insights from a large dataset that would otherwise be unmanageable and time prohibitive for a human researcher. We used data from 25,000 open response questions to explore the capacity and capability of ChatGPT as computer-assisted qualitative data analysis software (CAQDAS). The dataset we used was provided by the VIA Institute on Character, a local non-profit organization with which we are affiliated.We decided to experiment withChatGPT to determine if it could reliably and accurately analyze text data. Our expectationwas that if ChatGPT could analyze qualitative datasets with tens or hundreds of thousands of respondents, new pathwaysfor qualitative researchers may develop.Using AI for data analysiscouldchange the trajectory of a research design and lead to large scale qualitative studies that were not possible before now.
PROCEDURE
To test the limits of ChatGPT 4.0 (the only version with the means to upload files), we tried two different approaches to determine the capability of the tool.
METHOD 1: QUICK AND EASY
We started with avague set of user queriestoplace the data preparation loadon the CAQDAS and to determine if it would complete the same tasks a human researcher would.
User Query:Analyze the data in column AQ, identify themes, and provide 3-5 insightsbased on participant responses.
Result: Not useful.
ChatGPTdid not automatically clean the data without instruction whichcaused an error. The output from ChatGPTindicated the data file was either too long or too complex and it was unable to proceed with analysis.The raw data included responses such as “N/A” or random strings of letters, which a researcher would have deleted or ignored before analysis.
Lacking more specific instruction, ChatGPT defaulted to a quantitative approach to data analysis, even though the data were text responses. One of the first outputs ChatGPTproduced was a count of the most common phrases in the dataset.
We concluded that this approach to creating user queries was not useful.ChatGPT attempted to analyze the data but quickly became overwhelmed and either produced an error message or continued to attempt analysis, getting caught in the AI version of theMac’s “spinning wheel of death.”
METHOD 2: THE GUIDED ANALYST
Wethen providedChatGPT with more specific instructions. We instructed ittoclean, review and code the data, then create insights using a theoretical framework as a guide for analysis.
User Query: I'd like to analyze some text data using Peace Psychology and Positive Psychology as theoretical frameworks. Include content from the VIA Institute on Character as an additional framework. Focus on data in the 'Open Responses_Political Differences' column.
First, ignore text that indicates a respondent did not want to answer such as 'N/A' or random strings of letters. Leave those cells blank. Next, use descriptive codessuch as a phrase that describes the content of the targeted data.
Create a new document and filter the data from columnAD. Group the data according to the codes created in columnAD and list each data point that corresponds to the code.
Create 3-5 insights using the coded data in ColumnAD using positive psychology and peace psychology as theoretical frameworks.
Create a Word document and place the insights you created in it. Make the file available for download.
Result: Success
ChatGPT produced a list of five insights using the VIA classification of character strengths as a framework, supported by references to positive psychology and peace psychology.
Our last step was tothink about how to test the accuracy and reliability of the findings. Rachael has a strong background in qualitative methodology, so she extracted a sample of 20 data points and used a traditional qualitative approach to code the data. We also queried ChatGPT using the same data set with similar user queries to determine the degree of similarity between both sets of outputs.
Accuracy
Within the first 20 rows of participant responses, we identified at least three codes related to the five insights ChatGPT produced. We were pleased with the accuracy of the codes as they related to the insights from our sample data.
Reliability
Both researchers requested insight generation fromChatGPT using similar but not identical queries. Chat GPT’s output for both sets of queries was closely aligned. Three of the five generated insights were nearly identical between the two researchers and two were unique per researcher. It is important to note that even within research teams, differences arise between researchers’ interpretation of qualitative data. Disagreements are resolved through discussion and are typical of a qualitative research project.
Finally, we tested some of the limits of ChatGPT with several ad hoc user queries. Ourqueries produced a great deal of information in a very short period.
AD HOC REQUESTS:
Code Participant Country of Origin: We queried the data set to determine if each participants’ country of origin represented an individualistic or collectivist nation(coded as “country designation”). We asked ChatGPT to code each participant’s country of origin using this designation. The codes were placed in a new column created by ChatGPT.
Group Data and Analyze Frequency:We asked ChatGPT to examine the most common keywords and themes by country of origin. It provided a summary for all countries represented in this data set with the most frequently used codes according to participant’s country of origin.
Character Strength Frequency by Country Designation: We prompted ChatGPT to organize the data according to country designation and create a bar chart to illustrate the frequency of character strengths chosen as most beneficial by participants.
All the ad hoc requests were completed successfully. ChatGPT confirmed its approach with users before completing each step.For example, after requesting the data be coded according to country designation, ChatGPT responded:
Sure! In order to label countries as "collectivist" or "individualist," I would need a reference or criteria to determine which category each country falls into. There are various sources and studies that have categorized countries based on their cultural dimensions, including the work by Geert Hofstede.
Do you have a specific reference or criteria in mind that you'd like me to use? Alternatively, I can use Hofstede's cultural dimensions as a general guideline.
ChatGPTalso encouraged us to use caution when interpreting the findings from the between-country analysis, because significantly more participants were from individualistic nations.
LESSONS LEARNED
We learned through this project thatChatGPTexhibitsseveral esoteric preferences for working with Excel files.We only used Excel to upload data sets, so our suggestions are restricted to this software.
1. ChatGPTcannot analyze data if it has been tagged with a data type. The output will state that it completed the user query, but new files will not show any changes.
SOLUTION: Remove any Data Types tags before uploading Excel files toChatGPT.
2. ChatGPT prefers references to column names instead of the letters Excel uses to identify columns.
SOLUTION: If a user query contains a letter identifier instead of a column name, remove the space between the word “Column” and the letter.
CORRECT:“Provide a mean for the data in columnAI.”
INCORRECT:“Provide a mean for the data in Column AI.”
3. Unless instructed, ChatGPT will not automatically clean uploaded data. If a user attempts to request analysis before cleaning, it will respond with an error message.
SOLUTION: Provide explicit instructions for data cleaning before analysis.
HUMAN RESEARCHER VALUE
We shared just a fraction of the user queries we submitted over a 6-month period to test ChatGPT as a qualitative analysis tool. We presented the successes and failuresaslinear, concise processes for readability. However, early in the project, ChatGPT was often overwhelmed with requests and our queries resultedin error messages. Queries usually required several back-and-forth inputs between researchers and the AI to clarify instructions. With little or no guidance, ChatGPT was unable to produce results. We found that the AI required specificinstructions to function as computer-assisted qualitative data analysis software.Our bottom-line recommendation is that well trained researchers test the tool using a data set for which they already possess human produced findings. Compare those findings with ChatGPT's output and evaluate its reliability and accuracy.
Based on our brief examination of ChatGPT’scapability, we advise only well-trained researchers with extensive qualitative research experiencetouse AI as a computer-assisted data analysis tool.As in any other profession, expertise and training are the best predictors of quality work.As the saying goes, garbage in garbage out.Users with no idea how to design a rigorous research study will not provide the needed input for AI to perform adequately.
Our early work indicatesthe potential for AI to assistin qualitative data analysis. Like other CAQDAS products such as MAXQDA and NVivo, the software serves as a management and organizational tool. We envision ChatGPT as a marginallyhigher-leveltool with the capacity for categorizing and summarizing qualitative data, with the proper guidance and instruction.
Rachael brings over 10 years of research experience to her role at Kaleidoscope Innovation. She has advanced training in clinical psychology and mixed methods research methodology. Guided by the principles of positive psychology, Rachael uses a human-centered lens for deeply understanding the user experience. Her work at Kaleidoscope focuses on human-machine interaction and identifying design changes capable of positively impacting well-being at the individual and institutional levels.
Grant is a Senior Design Engineer who enjoys being faced with new challenges, and recognizes that well-designed products lead to better experiences and outcomes for users. He loves the design process, and has operated primarily in medical device and industrial applications. His passion to understand, innovate, and simplify has been supported and strengthened by the Kaleidoscope team and their talented partners. At home, he always has a fun project in the works (think: wooden bicycle, handheld Theremin, one-string electric guitar)!
Kaleidoscope Earns Gold
We are thrilled to share the exciting news that Kaleidoscope Innovation has achieved a Gold EcoVadis rating for 2023. This marks a major milestone in our journey to being a leader in environmental, social, and governance (ESG) excellence.
EcoVadis, a globally recognized platform, evaluates companies based on their policies, training initiatives, and reporting mechanisms related to ESG practices. This comprehensive assessment provides a holistic view of our organization's sustainability performance, positioning us as a standout in our commitment to environmental stewardship, social responsibility, and corporate governance.
Our latest EcoVadis assessment resulted in a remarkable increase of 15 points, propelling us to the 97th percentile among companies of similar size. This accomplishment not only highlights our dedication to sustainability but also underscores our team’s ability to outperform industry benchmarks.
This achievement is a testament to the collective efforts of every member of the Kaleidoscope Innovation family, and it reflects our shared values and determination to make a positive impact on the world. Our success in this area wouldn't be possible without the dedication and hard work of each team member who has contributed to our sustainability initiatives.
"We are grateful to every one of our dedicated team members for their contributions to this achievement," said Pam Ridgley, Associate Director of Business Planning. "Together, we are shaping a brighter and more sustainable future for Kaleidoscope Innovation and the communities we serve."
Here is just a sample of how we achieved our goals in each of the three ESG categories.
ENVIONMENTAL
We created a Design for Environment (DfE) Guide to help our team members ensure we use sustainable practices not just in our everyday work environment but also in the products we design and develop.
We reduced facility energy consumption by nearly 18% by resealing windows and creating an energy conservation culture
Our action team tested and implemented recommendations for sustainable 3D printing as they related to energy consumption, filament usage, and more eco-friendly materials
Our K-Cup recycling program diverted over 13,000 pods from the landfill and converted 479 lbsof material to compost and 160 lbs of material to recycled products.
SOCIAL
Community outreach including May We Help, Adopt-a-Class, IDSA Cincinnati, Ronald McDonald House, Habitat for Humanity, and more.
Expanded employee engagement & wellness teams and programs including the Employee Engagement Group, DEI Committee, Sustainability Committee, Employee Lunch & Learns, Women in Product Development Monthly Lunch, and more.
GOVERNANCE
Added numerous policies regarding diversity, equity, and inclusion, particularly with a focus on labor and human rights to our employee handbook
Created our own Institutional Review Board to ensure the ethical and safe development of healthcare products.
Our journey doesn't end here. This accomplishment serves as a stepping-stone toward even higher standards that continuously improve our ESG practices. The world is evolving, and so are the expectations placed upon businesses to operate sustainably. Kaleidoscope will remain agile and proactive in our approach to ensure we remain at the forefront of responsible corporate citizenship.
“We are incredibly proud of our EcoVadis achievement,” said Kaleidoscope CEO Ben Ko. “And we’re not stopping there. Our DfE program sets a new standard for ensuring that all products Kaleidoscope creates are more sustainable than they have been in the past, thereby amplifying the impact we can have on the world through good design principles.”
Precision in Practice: Navigating Usability Studies
Conducting a successful usability study, particularly for summative validation of medical products, can require a great deal of preparation and coordination of many moving parts. Whether it’s successfully simulating your target use environment or nailing down the exact scenarios and tasks to be presented, everything must come together. After all, nobody wants to waste the time and money it requires to complete a study if the result doesn’t align with what was intended.
One of the best ways to ensure success is to perform a pilot study with your protocol before starting the actual study. A pilot study is like a miniature version of the actual study conducted with far fewer participants. This approach helps confirm the study design will work as expected, the desired data can be obtained, the participants understand the task prompts, and more. For best results, the pilot study participants should be as close to actual participants as feasible; the same applies to the use environment. You’ll also want to conduct the pilot study early enough before the actual study to ensure there is sufficient time to update the protocol according to the findings.
Many people use the terms “dry run” and “pilot study” interchangeably, but there is technically a difference. A dry run is done to practice the protocol both before the pilot study and after the finalization of the protocol, ensuring the moderator and notetaker/analyst are comfortable with the product and script. This activity can be done with proxy participants or with no participants at all. What’s notable about a dry run is that it can be done in place of a pilot study if the participant population is expensive or difficult to recruit, if the protocol is very simple or essentially a repeat of a previous study, or if the timing does not permit a full pilot study.
While there can never be guarantees, these are proven ways to exponentially increase the odds of a study’s success. Taking the time upfront will ultimately save time and effort when it’s time to conduct the actual study. At Kaleidoscope, we've conducted hundreds of studies, so our process is as rigorous and dialed in as possible. If you're looking for a rock-solid research partner, we're ready to roll up our sleeves.
For a product to be successful, it’s critical for designers to understand the environment in which the product will be used. For a medical device, this environment is often inside the human body. While some anthropometric data such as height, weight, and arm reach, are well documented, there are many critical anatomical measures that remain unknown, particularly in the realm of women’s health.
While collaborating with a medical partner to address postpartum hemorrhage (PPH,) the leading cause of maternal death worldwide, Kaleidoscope encountered this common product design challenge. During preliminary research, the team found that there was little to no readily available data on vaginal dimensions immediately following childbirth. The scarcity of this particular data is not surprising, as the anatomy changes rapidly postpartum. Understandably, collecting this data isn’t a priority for mothers or caregivers, who are focused on the wellbeing of the newborn. Nevertheless, this lack of data created a significant challenge for the Kaleidoscope PPH design team.
TOOLS FOR BRIDGING THE GAP
Whether we are creating a medical device, a smart pet collar, or an industrial freezer, the team at Kaleidoscope utilizes a number of different methods when designing for the unknown. One way we obtain the data we need is simply to collect it ourselves! Armed with calipers and tape measures, we might venture into the field or bring samples into our studio to take direct measurements. Direct observation, whether in person or through videos and photos, is another way we round out our understanding of a unique user experience.
Sometimes—like trying to determine dimensions of internal anatomy—this just isn’t feasible. In those cases, we turn to subject matter experts. Surgeons, with their deep experiential knowledge of anatomy, are able to describe what they have encountered in situ, providing additional insights into the nuanced aspects of human anatomy, such as texture, firmness and what it feels like to manipulate different anatomical structures. These insights proved to be a vital element in overcoming the data gaps encountered by the PPH design team.
OUT-OF-THE-BOX INSPIRATION
When the Kaleidoscope team explores new product categories, we find that drawing inspiration from successful analogous products is another valuable strategy. If we’re creating a handheld device, referencing power tools, hair dryers, or hot glue guns as adjacent products can help guide the design in the correct direction. The key here is relevance—referencing products familiar to end-users ensures that the design resonates with their expectations. If we are developing a surgical device for ophthalmologists, (who are used to small, delicate instruments that they control with their fingertips,) it would be more appropriate to reference delicate tools such as those used by sculptors than it would be to reference tools used by auto mechanics.
While designing for a post-partum hemorrhage solution, analogous products included menstrual cups and discs, which share similar placement within the vaginal canal. These adjacent products provided the Kaleidoscope team with a good starting point for shape and dimensions of the device, as well as inspiration for materials and durometers to explore. These analogous references were part of the constellation of information used by the PPH team while exploring potential solutions to our data gap.
EMBRACING FLEXIBLE SOLUTIONS
At the end of the day, secondary research can only get us so far. In the absence of precise anatomical dimensions, adaptability can be a powerful tool in the designer’s toolbox. Whether the solution is fully adjustable (like an office chair) or offers different size options (like audio earbuds with multiple size tips,) a thoughtfully designed adjustable or flexible product ensures that one size does NOT need to fit all—rather, we can design a solution that easily adapts to meet the needs of all users.
Being on the cutting edge of new product development often means navigating uncharted territory. At Kaleidoscope, we've mastered the art of designing for the unknown with a combination of creative data collection, analogous product inspiration, and thoughtful adaptability. By transforming uncertainty into opportunity for our partners, we create products and experiences that improve outcomes for everyone.
Tom Gernetzke is a senior lead industrial designer at Kaleidoscope Innovation and has spent the last 12 years creatively bringing new product ideas to life.
|
Design and Prototyping
Formative Usability Testing and Evaluation
