Qualitative Research: AI’s Role in Analysis Advancement


Qualitative research plays a pivotal role in enriching our comprehension of individual narratives and experiences. It is a cornerstone methodology for design researchers seeking to forge a deep connection with user perspectives, particularly during the initial phases of the design process. This approach is instrumental in guiding iterative design developments, ensuring that end-user needs are comprehensively addressed. Qualitative data encompasses a diverse array of formats, including textual content, photographs, and videos. Typically, these studies involve a more focused sample size, often with 10 or fewer participants, to facilitate an intensive, detail-oriented analysis that quantitative methods may not capture.

Although qualitative research is the methodology of choice for design researchers, the approach requires a considerable time commitment. Qualitative data is known for being unwieldy at times, and words and images require more hours of analysis than numeric data. Often, our clients are eager to obtain research findings as quickly as possible to move a product or system into production. Therefore, large scale qualitative studies are not feasible for most design research projects. With the recent surge in the availability of AI language model tools, we speculated that ChatGPT could be used to analyze extremely large sets of qualitative data more efficiently. To that end, we conducted a 6-month project testing ChatGPT as a potential tool for qualitative data analysis.


Our aim in conducting this project was to determine if AI could produce insights from a large dataset that would otherwise be unmanageable and time prohibitive for a human researcher. We used data from 25,000 open response questions to explore the capacity and capability of ChatGPT as computer-assisted qualitative data analysis software (CAQDAS). The dataset we used was provided by the VIA Institute on Character, a local non-profit organization with which we are affiliated. We decided to experiment with ChatGPT to determine if it could reliably and accurately analyze text data. Our expectation was that if ChatGPT could analyze qualitative datasets with tens or hundredsof thousands of respondents, new pathways for qualitative researchers may develop.Using AI for data analysis could change the trajectory of a research design and lead to large scale qualitative studies that were not possible before now.


To test the limits of ChatGPT 4.0 (the only version with the means to upload files), we tried two different approaches to determine the capability of the tool.


We started with a vague set of user queries to place the data preparation load on the CAQDAS and to determine if it would complete the same tasks a human researcher would.

User Query: Analyze the data in column AQ, identify themes, and provide 3-5 insights based on participant responses.

Result: Not useful.

  1. ChatGPT did not automatically clean the data without instruction which caused an error. The output from ChatGPT indicated the data file was either too long or too complex and it was unable to proceed with analysis. The raw data included responses such as “N/A” or random strings of letters, which a researcher would have deleted or ignored before analysis.
  2. Lacking more specific instruction, ChatGPT defaulted to a quantitative approach to data analysis, even though the data were text responses. One of the first outputs ChatGPT produced was a count of the most common phrases in the dataset.
We concluded that this approach to creating user queries was not useful. ChatGPT attempted to analyze the data but quickly became overwhelmed and either produced an error message or continued to attempt analysis, getting caught in the AI version of theMac’s “spinning wheel of death.



We then provided ChatGPT with more specific instructions. We instructed it toclean, review and code the data, then create insights using a theoretical framework as a guide for analysis.

User Query: I'd like to analyze some text data using Peace Psychology and Positive Psychology as theoretical frameworks. Include content from the VIA Institute on Character as an additional framework. Focus on data in the 'Open Responses_Political Differences' column.

First, ignore text that indicates a respondent did not want to answer such as 'N/A' or random strings of letters. Leave those cells blank. Next, use descriptive codessuch as a phrase that describes the content of the targeted data.

Create a new document and filter the data from columnAD. Group the data according to the codes created in columnAD and list each data point that corresponds to the code.

Create 3-5 insights using the coded data in ColumnAD using positive psychology and peace psychology as theoretical frameworks.

Create a Word document and place the insights you created in it. Make the file available for download.

Result: Success

ChatGPT produced a list of five insights using the VIA classification of character strengths as a framework, supported by references to positive psychology and peace psychology.

Our last step was to think about how to test the accuracy and reliability of the findings. Rachael has a strong background in qualitative methodology, so she extracted a sample of 20 data points and used a traditional qualitative approach to code the data. We also queried ChatGPT using the same data set with similar user queries to determine the degree of similarity between both sets of outputs.


Within the first 20 rows of participant responses, we identified at least three codes related to the five insights ChatGPT produced. We were pleased with the accuracy of the codes as they related to the insights from our sample data.


Both researchers requested insight generation from ChatGPT using similar but not identical queries. Chat GPT’s output for both sets of queries was closely aligned. Three of the five generated insights were nearly identical between the two researchers and two were unique per researcher. It is important to note that even within research teams, differences arise between researchers’ interpretation of qualitative data. Disagreements are resolved through discussion and are typical of a qualitative research project.

Finally, we tested some of the limits of ChatGPT with several ad hoc user queries. Ourqueries produced a great deal of information in a very short period.


  1. Code Participant Country of Origin: We queried the data set to determine if each participants country of origin represented an individualistic or collectivist nation(coded as “country designation”). We asked ChatGPT to code each participants country of origin using this designation. The codes were placed in a new column created by ChatGPT.
  2. Group Data and Analyze Frequency: We asked ChatGPT to examine the most common keywords and themes by country of origin. It provided a summary for all countries represented in this data set with the most frequently used codes according to participant’s country of origin.
  3. Character Strength Frequency by Country Designation: We prompted ChatGPT to organize the data according to country designation and create a bar chart to illustrate the frequency of character strengths chosen as most beneficial by participants.

All the ad hoc requests were completed successfully. ChatGPT confirmed its approach with users before completing each step. For example, after requesting the data be coded according to country designation, ChatGPT responded:

Sure! In order to label countries as "collectivist" or "individualist," I would need a reference or criteria to determine which category each country falls into. There are various sources and studies that have categorized countries based on their cultural dimensions, including the work by Geert Hofstede.

Do you have a specific reference or criteria in mind that you'd like me to use? Alternatively, I can use Hofstede's cultural dimensions as a general guideline.

ChatGPT also encouraged us to use caution when interpreting the findings from the between-country analysis, because significantly more participants were from individualistic nations.


We learned through this project that ChatGPT exhibits several esoteric preferences for working with Excel files. We only used Excel to upload data sets, so our suggestions are restricted to this software.

1. ChatGPT cannot analyze data if it has been tagged with a data type. The output will state that it completed the user query, but new files will not show any changes.

SOLUTION: Remove any Data Types tags before uploading Excel files to​​​ChatGPT.

qualitative research data types

2. ChatGPT prefers references to column names instead of the letters Excel uses to identify columns.

SOLUTION: If a user query contains a letter identifier instead of a column name, remove the space between the word “Column” and the letter.

CORRECT: “Provide a mean for the data in columnAI.”

INCORRECT: “Provide a mean for the data in Column AI.”

3. Unless instructed, ChatGPT will not automatically clean uploaded data. If a user attempts to request analysis before cleaning, it will respond with an error message.

SOLUTION: Provide explicit instructions for data cleaning before analysis.


We shared just a fraction of the user queries we submitted over a 6-month period to test ChatGPT as a qualitative analysis tool. We presented the successes and failuresas linear, concise processes for readability. However, early in the project, ChatGPT was often overwhelmed with requests and our queries resulted in error messages. Queries usually required several back-and-forth inputs between researchers and the AI to clarify instructions. With little or no guidance, ChatGPT was unable to produce results. We found that the AI required specific instructions to function as computer-assisted qualitative data analysis software. Our bottom-line recommendation is that well trained researchers test the tool using a data set for which they already possess human produced findings. Compare those findings with ChatGPT's output and evaluate its reliability and accuracy.

Based on our brief examination of ChatGPT’s capability, we advise only well-trained researchers with extensive qualitative research experience to use AI as a computer-assisted data analysis tool. As in any other profession, expertise and training are the best predictors of quality work. As the saying goes, garbage in garbage out.Users with no idea how to design a rigorous research study will not provide the needed input for AI to perform adequately.

Our early work indicates the potential for AI to assist in qualitative data analysis. Like other CAQDAS products such as MAXQDA and NVivo, the software serves as a management and organizational tool. We envision ChatGPT as a marginally higher-leveltool with the capacity for categorizing and summarizing qualitative data, with the proper guidance and instruction.

Back to Insights + News


  • Rachael Clark

    Rachael Clark

    Senior Design Researcher | [email protected]

    Rachael brings over 10 years of research experience to her role at Kaleidoscope Innovation. She has advanced training in clinical psychology and mixed methods research methodology. Guided by the principles of positive psychology, Rachael uses a human-centered lens for deeply understanding the user experience. Her work at Kaleidoscope focuses on human-machine interaction and identifying design changes capable of positively impacting well-being at the individual and institutional levels.

  • Grant Cothrel

    Grant Cothrel

    Senior Design Engineer | [email protected]

    Grant is a Senior Design Engineer who enjoys being faced with new challenges, and recognizes that well-designed products lead to better experiences and outcomes for users. He loves the design process, and has operated primarily in medical device and industrial applications. His passion to understand, innovate, and simplify has been supported and strengthened by the Kaleidoscope team and their talented partners. At home, he always has a fun project in the works (think: wooden bicycle, handheld Theremin, one-string electric guitar)!

A Research Outline for Industrial Designers: Unleashing the Power of Data-driven Creativity

In today's fast-paced and competitive world, industrial designers face the exciting challenge of creating innovative and user-centric products that capture the market's attention. While their expertise lies in design aesthetics and functionality, the role of research in the design process cannot be underestimated. Research is the key that unlocks valuable insights, fuels inspiration, and ensures that designs are grounded in real-world needs and preferences. However, for industrial designers and other professionals who are not trained in research methods, navigating the realm of research can feel daunting. In this article, we will define research methodology and provide suggestions for selecting the right one for your project. 


Once a client settles on a research question, it is up to the design researcher to select the methodology that facilitates a rigorous approach. Think of methodology as a framework for conducting a research study. The chosen methodology will guide a researcher in methods and procedures that ensure the results or findings are valid and reliable. 

QUANTITATIVE: Quantitative methodology is used to determine if relationships between variables exist, to test a hypothesis, or to measure a phenomenon. Quantitative data is used to make group comparisons or identify patterns. Data are numbers and reported in a standard reporting structure. Descriptive and inferential statistics require quantitative data. The output of quantitative analysis is referred to as results. 

QUALITATIVE: Qualitative methodologies are used to understand a phenomenon more deeply, to obtain a detailed description of an experience, or to understand how or why an event occurs. Qualitative data may be text or images and uses a flexible reporting structure. Interview transcripts and video recordings represent qualitative data types. The output of qualitative analyses is called findings. 

MIXED METHODS: Mixed methods research includes aspects of quantitative and qualitative methodologies in the same study or series of studies. Mixed methods approaches can be used sequentially or concurrently. Often, results or findings from one phase will be used to design a subsequent phase of a project. A time and motion study consisting of quantitative measurement of a motion in a workflow followed by a one-on-one interview is an example of a mixed methods study. The qualitative findings could be used to understand the results of the quantitative phase more deeply, to provide context for interpreting the results, or to triangulate the results and findings. 


Choice of research methodology should be determined using several factors: 

  1. Research Purpose: If the purpose is to understand or explore, a qualitative methodology is likely the best approach. If a client wants to know how much or to determine if a new workflow is more productive than the old one, a quantitative approach will likely be appropriate. If a client wants both, a mixed methods approach will be best. 
  2. Budget: Generally, qualitative studies are more time-intensive than quantitative studies. If a client’s budget is limited, a quantitative approach may be best. 
  3. Decisions: If a client wants to use the findings of a study to generate ideas or inform iterative design requirements, a qualitative approach may be best. If a client wants to evaluate changes to a process or product, a quantitative approach is required. 

The next step in planning a research study is to decide what methods will be used to collect data. Methods specific to each methodology exist but are beyond the scope of this article. If you are interested in learning more, check out some of the popular methods from a reliable source: narrative inquiry, survey, and ethnography are a few examples of methods you may encounter in the field. By embracing research methodology as an integral part of the design process, industrial designers can confidently embark on their creative journey, armed with insights that empower them to craft extraordinary products that not only meet user needs but also set new standards of innovation in their industry. 

Back to Insights + News


  • Rachael Clark

    Rachael Clark

    Senior Design Researcher | [email protected]

    Rachael brings over 10 years of research experience to her role at Kaleidoscope Innovation. She has advanced training in clinical psychology and mixed methods research methodology. Guided by the principles of positive psychology, Rachael uses a human-centered lens for deeply understanding the user experience. Her work at Kaleidoscope focuses on human-machine interaction and identifying design changes capable of positively impacting well-being at the individual and institutional levels.

Mastering Combination Product Development: From Immersion to Validation


Our journey kicks off with immersion, a creative problem-solving phase. Here, we ensure that solutions are at the ready for any potential roadblocks. We dive into the waters to test the concept's feasibility and identify potential challenges. We also map out short and long-term objectives, charting the course for product development.


With a clear vision in mind, we start breathing life into it through meticulous planning and execution. Crafting a combination product resembles assembling an intrurate puzzle, where every detail carries significance. This stage revolves around rigorous testing and evaluation to pinpoint the best and most efficient design solutions.


This phase is undeniably exhilarating. Building the product is where the concept takes tangible form. Transitioning from design to reality, prototyping takes center stage. It grants us the opportunity to scrutinize every element, ensuring the product's integrity and functionality.


Validation stands out as perhaps the most pivotal step in the entire process. During this phase, the product undergoes comprehensive reviews and testing to unveil any last-minute imperfections or errors. This thorough examination ensures the product is primed for its grand debut in the market. Validation acts as the ultimate litmus test, determining the readiness of the combination product for integration into various healthcare services.

Back to Insights + News


  • Taylor Schmitt

    Taylor Schmitt

    Marketing Co-op | [email protected]

    Taylor Schmitt is currently a student at The Ohio State University, where she studies marketing. She loves exploring new opportunities and facing new challenges. While working at Kaleidoscope she has been able to work closely with the sales team to support business growth and brand visibility

  • Matt Suits

    Matt Suits

    Head of Sales | [email protected]

    Matt has always loved interacting with clients to find solutions for their challenges. He was drawn to business development at Kaleidoscope Innovation because of the great potential he saw. After graduating from the Lindner College of Business at the University of Cincinnati, he worked with two startups, a marketing consultancy, a financial services company and the non-profit 3CDC. He believes that listening is the most important part of sales. In his free time, Matt enjoys movies, trying new foods, traveling and the great outdoors.

AI as Intelligent Design? Not Yet, But It’s Coming.

From art generators to chatbots, AI seems to be having its zeitgeist moment in popular culture. But for those of us who work in design, the near-term and future applications of AI have been lively discussion points in strategic planning meetings for quite some time. There is no doubt that AI will be an instrumental part of our world’s future. It will allow us to rapidly synthesize all the data being collected via our phones, cameras, computers, smart devices, and much more, giving us the ability to decipher and understand that data in illuminating, meaningful, and likely, world-changing ways.  

What does this mean for the design industry? Though it may be a long time before AI is able to design a product from the ground up, the potential is clearly there. In fact, we believe AI is a tool that designers should be adding to their arsenal sooner rather than later. 


Putting AI to Work 

To put our money where our industry-informed opinions are, the Kaleidoscope Innovation team recently embarked on a studio project to design a high-end lighting fixture that could mimic lighting patterns found in nature. The project would enable our team to flex our aesthetic skills while using the full range of our design toolbox. One of those tools is Midjourney, a proprietary artificial intelligence program produced by an independent research lab by the same name. Though still in the open beta phase, Midjourney proved to be a useful partner in our mission. The collaboration between AI and the guiding hand of our expert design team delivered intriguing results. 

One important distinction about the AI portion of the project: We were not setting out to produce real-world functionality, and in fact, we had no expectation or need for the AI to produce fleshed-out ideas or even design sketches. This experiment was about exploring new territories in aesthetics and applying them to materials and manufacturability considerations. 

Our first step was to gather a team to collaborate on the search terms that would help visually articulate the aesthetic aspirations for our new fixture. Midjourney works by inputting text-based prompts, which the AI algorithm uses to generate new images using vast databases of existing images. The terms we fed the algorithm included chandelier, lighting, brilliant, elegant light, airy, crystalline patterns of light, dancing, photorealistic detailed plants, greenery, daytime, bright, modern, beautiful, natural colors, garden, and greenery. The team also used technical inputs alongside these qualitative descriptors to determine the aspect ratio and resolution while also guiding the algorithm to reference certain lighting styles and rendering approaches.  

Digesting these descriptive words, Midjourney searched vast amounts of data across the internet to create original—albeit amalgamated—artwork. The images it produced reflected the algorithm’s interpretation of the inputs the team provided. From there, we tweaked specific inputs to alter the color, lighting, tone, and subject matter, continuing to iterate until we had collected a series of AI-generated lighting fixtures that could inspire the team.

How Did AI Do?  

Based on the text inputs the team provided, Midjourney was able to identify design elements that could produce the effect of light shining through leaves. The images it produced looked organic, almost surreal in the way they were able to capture the kind of nature-made glow and transparency that is elusive in real-world lighting solutions. The various iterations of artwork then became mood boards that set up our team to brainstorm ways in which the effect could conceivably be produced.  

The algorithm’s interesting use of materials, colors, lighting effects, and overall mood inspired us to apply those attributes to a holistic design. In other words, instead of our team scratching their heads visualizing how the light should transmit, AI provided us with ideas that enabled us to focus on materials, manufacturability, technical requirements, and more. Rather than spending hours scouring the internet for inspirational imagery, the team was able to craft that inspiration imagery ourselves through AI in a fraction of the time—imagery that exactly aligned with our design vision. 

concept board

Without question, Midjourney served as a highly effective springboard that sparked ideas our team would probably not have come up with starting from a blank sheet of paper and pen. In this sense, AI provides an upfront efficiency that can move a project farther down the road faster than it might otherwise have gone. Perhaps more than that, a significant strength of AI in this application is that it can cast a wide net in terms of inspiration and exploration. It’s an open mind, and designers should be willing—and eager—to go down the rabbit holes, teasing out new possibilities. Once an intriguing direction is established, the designer can take over to turn the AI-generated inspiration into an actual product.  

The key to a successful AI collaboration is plugging in the right words or phrases to best draw out the AI. And so, crafting prompts could be viewed more as art than science. Further, with a program like Midjourney, there is an element of unpredictability: You don’t have much control over what you’re going to get out of it. There is a lot of trial and error and shooting in the dark. Therefore, if you already have a set idea in mind, using AI to design it will probably be more frustrating than productive.  

The inherent aspect of exploration and discovery is a factor to consider as well. Our team felt excited about experimenting with this technology specifically because the lighting fixture was an internal project. Had we been designing for a client, we would have been more hesitant to use AI while balancing product requirements, timeline, budget, and resources.  

Lastly, because this was a purely aesthetic exercise, we weren’t trying to solve any mechanical problems through AI—that’s skill is not in its wheelhouse at this point. This limitation provides a real barrier to the widespread adoption of AI, but as the algorithms improve over time, AI may be able to help us solve even our stickiest mechanical problems. 

Beyond leveraging AI for creative exploration, Kaleidoscope has also put it to use in some of our research work. As part of our insights and user experience programs, we often do ethnography or time-and-motion studies in which we observe individuals interacting with a tool or experience. Typically, one of our team members is responsible for reviewing videos to log data, tracking everything from how often someone does something to the amount of time it takes them to do it. It’s a time-consuming process that has led us to start dabbling with programming AI to analyze video recordings for certain elements and then export the data quickly and effectively. Using AI to track the frequency and duration of actions for time-and-motion studies shows tremendous potential to save time and reduce costs while freeing our team members to focus on more creative assignments. 

The Verdict 

The Kaleidoscope team came away with an appreciation for where AI can support our design efforts today, particularly as a powerful aid in producing aesthetic inspiration and as a tool to sort and output raw data. Both help the design process in productive ways and serve as a small window to what may someday be an AI-driven design future.

This was written for IDSA, if you'd like to see the INNOVATION Magazine article, please check out idsa.org/news-publications/innovation-magazine/spring-2023/

Back to Insights + News


  • Tony Siebel

    Tony Siebel

    Director of Design | [email protected]

    Tony Siebel is director of design at Kaleidoscope Innovation, delivering a user-centered mindset to products and experiences.

  • Tom Gernetzke

    Tom Gernetzke

    Senior Industrial Designer | [email protected]

    Tom Gernetzke is a senior lead industrial designer at Kaleidoscope Innovation and has spent the last 12 years creatively bringing new product ideas to life.

  • Caterina Rizzoni

    Caterina Rizzoni

    Lead Industrial Designer | [email protected]

    Caterina Rizzoni is a lead industrial designer at Kaleidoscope Innovation and is the Director-at-Large of Conferences for IDSA.

Infosys Medical Devices and Engineering Services x Kaleidoscope Innovation

The Healthcare and Medical devices industry is undergoing a revolutionary transformation in the way solutions and devices are being formulated and developed. Medical devices are becoming more connected than ever and remote patient monitoring with data analytics is becoming a norm.

It is imperative for the medical device companies to adopt a strategic approach to stay ahead of the innovation curve by leveraging technology advancements in multiple areas such as mobility, wireless, cloud, and analytics to drive innovation that addresses market needs and challenges of longer device development cycles, optimization of development processes, and high production costs.

At Infosys, we help our clients in designing customized devices, end-to-end product development, maintenance, manufacturing support, regulatory documentation, and product compliance and certifications. We also help optimize R&D cost and improve supply chain efficiencies by leveraging new technologies and partner ecosystems. This is to bring innovative medical devices and Software as a Medical Device applications into the market with the objective of improving patient care while reducing the cost of care.

Our ISO 13485 certified processes and Quality Management System ensures high-quality product development which enables our client to meet their regulatory needs and objectives. With our recent acquisition of product design and development firm, Kaleidoscope Innovation, we plan to redefine patient treatment and consumer health across the globe.

Full article can be found on Infosys.com

Back to Insights + News
The Design of Everyday Things

Inspired by Don Norman’s classic work, the Design of Everyday Things, we’ve been thinking about mundane, everyday items that can have annoying usability flaws. While we have a particular focus on the human factors of healthcare and medical products here at Kaleidoscope, we can apply that same rigorous, analytical human factors approach to these everyday things.

So, here we have the seemingly benign 2.5 gallon jug of drinking water, a household staple used by a variety of brands across the country.

Problem 1: As water is dispensed from the jug, additional air is required to replace the dispensed water to ensure consistent water flow and prevent the jug from collapsing due to the pressure of the surrounding air. To add air flow into the jug, a small hole must be punctured into top with a sharp knife. The use of a sharp knife poses a potential safety hazard when considering the orientation and motion in which the knife must be used and the force necessary for the knife to puncture the slick plastic material of the jug. In addition, the most obvious place to puncture this hole is the top side facing the front of the jug, which has a slight slant toward the user. The angle of the stabbing motion must be just right; if the angle is too shallow, the knife blade can skid across the surface of the plastic, with the blade pointing in toward the user’s body.

A potential mitigation for this problem is to provide an adhesive pull tab that can be removed to reveal a pre-punctured vent hole.

Problem 2: The spigot contains a small strip of plastic that extends from the spigot base to the dispenser handle. The plastic strip is intended to prevent the dispenser handle from being pulled open until the user intentionally breaks the strip, pulls the dispenser handle, and begins dispensing the water. However, the plastic strip can be easily broken unintentionally, and the dispenser handle then opens with very little resistance. This can lead to the dispenser handle opening inadvertently when force is applied to the spigot during loading, or the spigot catches on a surface while unloading, potentially emptying water into a shopping cart or the trunk of a car.

A potential mitigation for this problem is to provide a screw cap over the spigot, similar to the caps on water bottles.

What’s an aspect of an everyday item that you would change to improve the user experience?

Back to Insights + News


Virtual Tools for Innovative Product Design

Co-authored article with Infosys

Design influences a product’s lifecycle performance and cost, starting from its development. Product development costs rise significantly if a defect is identified at a later stage. Using virtual tools for new product introduction simulates possible scenarios upfront for comprehensive testing. It gets products to the market quickly and saves money for a successful launch.


  • Design influences a product’s lifecycle performance and cost, starting from its development.
  • Conceptualization and design stages determine more than 70% of a product’s lifecycle decisions
    and cost.
  • Virtual tools are an effective way to design new products that serve specific customer needs.
  • Virtual models of new products accelerate their evaluations to shrink the development cycle time.
  • Organizations should create virtual replicas of workplaces for human-machine interactions studies from multiple perspectives.

Lifecycle cost is the total cost (direct and indirect) a product incurs in its life span. Conceptualization and design stages determine more than 70% of a product’s lifecycle decisions and cost.1 The earlier an issue is identified, specifically in the design stage, the easier it is to fix and avoid costly rework. Virtual replicas (or digital twins) of products, processes, and environments streamline design and new product development to reduce costs and time to market.

A common assertion is between 80% and 90% of new products fail. However, realistic failure rates vary by industry, from 36% in healthcare to 45% in consumer goods.2 Professor Clayton Christensen, best known for his theory of disruptive innovation, believes the success mantra is to design products that serve its intended customers. Manufacturers should focus on the function that a customer who buys a product would want it to do.3

To enable that, virtual representations of the product under development, in orchestration with humans and other entities in the ecosystem, is an effective approach. The approach encourages innovation. Designers visualize the product’s operating condition, create digital prototypes for trial runs, and carry out tests on a global scale. Virtual tools like 3D computer models and digital twins support informed decisions in early product design stages. This mitigates the risk of a wrong product release or a poor customer experience.

Virtual products are an effective way to design new products that serve specific customer needs.

When end users receive virtual training of a complicated product’s operation (like an aircraft engine), memory retention happens in the background. Any number of such instances can be created at a negligible marginal cost for repetitive usage. A central digital setup saves the cost of setting up multiple physical arrangements at different locations.

Parameters of Successful New Products

Product failures are more from a commercial perspective than technical. More than 25% of revenue and profits across industries come from new products, according to a study by McKinsey. Successful products relate to a set of core capabilities, with the top-most as follows:4

  • Collaboration to execute tasks as a team.
  • Investment to mine market insights and their inclusion in the product.
  • Plans for new product launches, comprising target customer segments, key messages to communicate, and objectives to achieve.
  • Talent development for new product launches with defined career paths and incentives.

At the same time, the primary reasons for product failures and mitigants are the following:5

  • Gap in meeting product expectations; delay launch until product completion.
  • Inability to support rapid growth if a product is successful; set ramp-up plans to avoid this.
  • Low demand for a new product; perform due diligence for customer requirement before planning a product. Launch products in suitable markets.
  • Difficulty in new product usage; provide proper customer orientation and training.

Virtual tools for product design address the above reasons for failure and increase the chances of successful product launches.

Design Thinking with Virtual Tools

Design thinking is a popular, technology-agnostic approach for new systems design and problem solving. It balances the technical feasibility of products, financial viability, and desirability from a customer’s perspective (see Figure 1). It is even more impactful when implemented along with virtual product design tools.

Figure 1. Design thinking at the sweet spot of desirability, viability, and feasibility


Source: Infosys

The design thinking cycle starts from empathy to understand a customer’s needs from their perspective, followed by defining, ideating, prototyping, and validating, in iterative loops. New product development and customer participation encourage collaboration in a virtual environment to practice design thinking. Immersive environments using mixed reality (combinations of augmented reality or AR and virtual reality or VR) create a working environment close to the real world, to identify and correct issues much ahead (see Figure 2).

Figure 2. Virtual tools used across design thinking stages


Source: Infosys

Virtual models of new products accelerate their evaluations to shrink the development cycle time.


Design firm IDEO, for example, wanted to perform ethnographic research to capture customer requirements for new products. However, it was difficult to identify key observations from many data points and recreate them later, even with expensive videos or photos. It addressed the challenge through a VR camera.6

Kaleidoscope Innovation, a design and development unit within Infosys, designed a large freezer project using virtual tools. Such projects usually undergo several time-consuming team reviews. The team created a 3D model in a VR environment that helped designers walk around the product early in the design phase, evaluate its usability from multiple perspectives, and tackle proposed changes to design.

This virtual model did not change the overall project plan, but accelerated evaluation and decisions around it, shrinking the product development cycle time. The team selected the best design without spending time and money on physical prototypes.

Automation in WareHouses

Humans work with machines in warehouses. Material handlers carry out order fulfillment along with pick-and-place robots. Workers’ safety in all situations is important.

A leading e-commerce player wanted to validate design decisions for robots working in its order fulfillment warehouses to gain insights into their safe working alongside humans. Kaleidoscope Innovation created a virtual environment where employees interacted with robots in different situations. The team created a digital twin to simulate several configurations of robots and their working environment. The company recorded the results and interviewed employees about pros and cons of each situation.

The VR-based solution provided a cost-effective and safe way for the e-commerce firm to test new concepts in human-robot interaction and capture data and feedback before implementation. It helped the managers zoom out and look at the big picture, in contrast to one robot or equipment at a time.

Training for Product Usage

Operators need training to work on machines with complex functionality and procedures, to stay safe and productive. VR-based training prepares humans before hands-on operation on a machine. For instance, Rolls-Royce has rolled out a VR-based training kit for its airline customers to manage aircraft engine maintenance and repair.

Infosys’s VR-based program provides step-by-step instructions to train employees in a hospital environment. The program uses physical gestures to simulate actual tasks involved in a job. Gamification with scores and points keeps employees engaged and motivated. Scores reflect an individual’s strengths and weaknesses. Training data is integrated with the central learning management system for records.

A multinational industrial and consumer goods manufacturer wanted to create an e-training platform for its new operators. It had a few integrated assembly lines for its finished items. The Kaleidoscope Innovation team created a virtual training module along the assembly line, one workstation at a time. The team used front-end user interface elements to guide users for equipment operations. It tracked performance metrics in the backend to provide feedback for correction. Best practices of creating a virtual replica of one workstation are used at later stations.

Futuristic Workplaces

While collaborative, remote and hybrid working has surged since the pandemic, the future is in three-dimensional virtual and mixed reality workspaces. Organizations benefit from a virtual 3D replica of its workspaces, equipment, products, avatars, or personas. Employee collaborations lead to faster new product development with effective interactions. Teams share ideas, explore, and invent new concepts. Early collaboration of team members in multiple locations enables them to make more informed decisions in the product development process.

Organizations should create virtual replicas of workplaces for human-machine interactions studies from multiple perspectives.

The future of work in healthcare, retail, engineering, and manufacturing is where humans and human-like machines work together. Organizations should proactively create such workspaces virtually and study human-machine interaction from safety, productivity, and employee morale perspectives before any physical implementation.


  1. Product life cycle cost analysis: State of the art review, Y. Asiedu &P. Gu, 2010, International Journal of Production Research.
  2. Myths About New Product Failure Rates, George Castellion, Stephen K. Markham, 2013, published in the Journal of Product Innovation & Management 30 pp. 976-979.
  3. What Customers Want from Your Products, Clayton M. Christensen, Scott Cook and Taddy Hall, January 16, 2006, Harvard Business School.
  4. How to make sure your next product or service launch drives growth, Alessandro Buffoni, Alice de Angelis, Volker Grüntges, and Alex Krieg, October 13, 2017, McKinsey.
  5. Why Most Product Launches Fail, Joan Schneider and Julie Hall, April 2011, Harvard Business Review.
  6. IDEO: Getting closer to the customer through virtual reality, Lauren, April 27, 2017, Harvard Business School.


Back to Insights + News
Designer Centered Design: Humane Design

While “User Experience Design” is often used interchangeably with “User Interface Design,” UX goes far beyond mere interface design to encompasses a user’s complete experience of a product, system or service. For Don Norman, the usability engineer and researcher who coined the term “User Experience,” all aspects of the product experience, “from initial intention to final reflections,” ought to support the user’s needs and desires. Years before Norman came onto the scene, this same concept inspired Jef Raskin, a human-computer interface expert, to define the ideal computing system. Though his vision of a computer, which was nothing more than a glorified word processor, was uninspired even in its own time, Raskin developed a set of UX Design principles, including UI consistency and encouraging users to develop productive habits, that are still relevant today.

“The Canon Cat and the Mac that Steve Jobs Killed,” an article by Matthew Guay, describes Raskin’s desire to create a computer with a humane interface. “An interface (i.e. ‘The way that you accomplish tasks with a product’) is humane if it is responsive to human needs and considerate of human frailties,” wrote Raskin. His goal was to liberate computer users through increased productivity—getting more done in less time. Inspired by Isaac Asimov’s laws of robots, Raskin defined his own laws of computing to achieve this goal:

“A computer shall not harm your work or, through inaction, allow your work to come to harm.

“A computer shall not waste your time or require you to do more work than is strictly necessary.”

Raskin’s second law is applicable far beyond word processing and seems to emphasize a common struggle faced by UX and UI designers alike. Powerpoint is a notable example of a poorly designed interface that results in decreased productivity. Its predictive toolbar feature that attempts to anticipate the user’s needs based on what has been selected. While this feature can be helpful when it correctly predicts the user’s needs, it can be very inconvenient when it guesses incorrectly, adding multiple mouse clicks to the user’s workflow.

Another violation of Raskin’s second law is inconsistency between user interface elements. Consider Apple’s latest iOS update. Previously, incoming text messages appeared at the top of the lock screen. Following the 16.1.1 update, incoming text messages now appear at the bottom of the lock screen. Neither location is objectively right or wrong, except for the user’s previous experiences of seeing new messages at the top. Now users must unlearn a previous habit to relearn a new interaction. Does the new feature add sufficient value to be worth the friction it introduces into the user’s experience?

The quintessential mnemonic “righty tighty lefty loosey” illustrates the socially ingrained understanding of how to lock or unlock a rotating mechanism. This convention becomes apparent when a user encounters an experience that is counter to what they expect. Because a user intuitively expects to turn the mechanism a certain way, requiring the opposite is a source of confusion and frustration.

When designing products, consistency is one of many usability principles, known as heuristics, that act as general guidelines for creating intuitive user interactions. Usability expert Jakob Neilsen, who cofounded the Nielsen-Norman Group with our good friend Don Norman, created the most well-known and widely used set of usability heuristics. These heuristics are used by product designers across the globe to design more intuitive and user-friendly products and experiences.

Another key heuristic that Nielsen defined is the user’s ability to match the design of the system to their understanding of the real world. Imagine a stove top with 4 burners arranged in a square and knobs that are arranged in a line. This creates confusion and tension because the user does not know which knob controls which burner. However, if the knobs are arranged in the same square pattern as the burners, and each knob activates its corresponding burner, users quickly understand which knob needs to be turned to ignite the intended burner.

The ultimate goal of user-centered design is to increase productivity and create an experience that is “responsive to human needs and considerate of human frailties.”  No product is experienced in a vacuum—each user encounters that product within the context of a lifetime of other experiences. Understanding the needs and frailties of the end user empowers designers to create more intuitive, efficient, and enjoyable experiences for users. While Jef Raskin’s Canon Cat was a commercial failure, in a world inundated with widgets, tools and systems—both physical and digital—his concept of a humane interface is perhaps more relevant now than ever.

Headquartered in Cincinnati, Ohio, Kaleidoscope Innovation provides medical, consumer, and industrial clients with full-service insights, design, human factors, and product development. For more than 30 years we have been helping our clients grow their capabilities, gain usable knowledge, and get worthwhile results.

As a full-spectrum product design and development firm, we are an expert extension of your product vision. Our teams collaborate across disciplines, providing specialized input to produce the ideal intersection between function and form. To ensure the soundness of our work, Kaleidoscope houses a full range of test labs, and we employ an award-winning team that embraces every challenge, applying their experience, ingenuity, and passion.

Back to Insights + News


  • Tom Gernetzke

    Tom Gernetzke

    Senior Industrial Designer | [email protected]

    Tom Gernetzke is a senior lead industrial designer at Kaleidoscope Innovation and has spent the last 12 years creatively bringing new product ideas to life.

Designer Centered Design: Using VR for User Research and Testing

What do you do when you are in the early concept development of the design process and want to get user feedback to inform future development? Maybe you would 3d print or hand prototype your design. Putting an early mockup of your design in the hands of the user for them to assess is an important part of any user centered design process. But what if your design concept involves autonomy, a UI or a complex series of physical interactions with the user? Without additional functionality, a physical low fidelity mockup in this context loses its effectiveness in garnering insight.

“Product design” as a whole has shifted. Increasingly, the objects that we design and use in our daily lives have a component of digital interaction and/or are part of a larger virtual ecosystem. These challenges of gaining early insights from low fidelity mockups is epitomized when designing something like an autonomous robot. This design process sometimes involves years of hardware and software development for even basic functionality. So how can designers run ahead of this development to put a concept in front of users early enough to inform how such a complex product should be designed to work in these interactions to instill trust, engagement, and even enjoyment?

Let’s say that we are designing a new autonomous robot to deliver room service orders to guests at a hotel. The first issue to address is how people react to an autonomous device sharing their space. How close is too close? Is there a violation of a social contract by placing this robot in what was otherwise a dedicated space for people? How do they expect the robot to behave? Most importantly, how do you begin to probe those expectations of the customer when hardware and software development are not mature enough to represent the final design concept? You cannot put an engineering prototype in close proximity with the user without creating a potential safety risk. If you were to make a remote-controlled mockup of the robot, how can you truly test user comfort with autonomy when the test subject knows that there is a human in control? And how do those reactions to autonomy change with multiple robots? This is where VR stands out as a remarkably effective tool for gaining insight.

Utilizing VR in complex product interactions allows designers to not only save on the resource cost of hardware prototyping and manufacturing, but also allows them to iterate much more rapidly and push boundaries of comfort with users without ever putting the user at risk. By conducting user testing in VR, not only can you present a complex and interactive product experience in front of the user, but you can also transport them to specific environments and scenarios with the push of a button. This enables the development of not only a guiding model for the design, but also a guiding model for software development as VR interactions can inform what does and does not work in interactions between humans and autonomous systems. However, VR still has its shortcomings and is not the definitive means of user testing in product development.

Virtual reality for user testing and concept evaluation is simply another tool in our toolbox as designers and design researchers. While it offers new capabilities for testing and evaluation, there is a major tradeoff between a VR mockup and a physical one… namely the nature of “virtual” reality itself. There is no physical feedback, and while there is a strong sense of depth perception, it is not the same as an actual physical interaction. While augmented reality may better incorporate both the physical and virtual, the virtual assets can stand out as even more artificial than a full virtual immersive experience because of the difference in fidelity of virtual vs real world objects. Does this eliminate the need for physical prototyping and low fidelity physical mockups? No. But VR enables designers and developers to test more complex products earlier in the design process with users where alternative approaches are less feasible due to complexity and cost.

While the role of a designer can be reductively described as “stylist” I think the true value we bring to a team are as story tellers both outwardly to the customer/target user and internally. VR enables us to share virtual models without “CAD scale blindness” and to collaborate more seamlessly even while remote. Having a VR headset brings even remote collaborators together and immerses them in a 3d virtual experience. Meaning there is less misunderstanding and room for interpretation than just a concept sketch, 2d render, or even a 3d CAD model on a screen.

As this technology continues to mature and becomes more accessible, I see the use of VR as an increasingly valuable tool for designers. Where paper and markers gave way to Cintiqs and iPads, I could see CAD modeling and user testing making room for VR modeling, collaboration, and design evaluation. We are entering a new frontier for design and media with VR that will undoubtedly influence how we live and work. Pick up a headset and explore the possibilities for yourself. There is plenty of undiscovered opportunity and impact to be harnessed with this new technology!

Back to Insights + News


  • Nikko Van Stolk

    Nikko Van Stolk

    Lead Industrial Designer

    Interested in evolving processes with new capabilities and new technology. A proven track record of experience working with surgical and industrial robotics. A strong storyteller and team leader that takes the initiative constantly seeking to exceed client expectations. A good people person and collaborator able to wade into the thickest CAD assemblies with large engineering teams, deep experience working with human factors in developing and executing human factors research, and facilitating and engaging in creative problem solving and brainstorms with fellow designers. A full designers toolkit of sketching, rendering, CAD modeling/surfacing, and DFM.

Innovation Leadership & How to Build the Ideal Innovation Team

In this article about innovation leadership and the ideal innovation team, Kaleidoscope VP of R&D, Medical, Michael Clem DVM, MS examines the functional and cross-functional expertise needed to drive the best innovation resources to turn creative ideas into quality products that benefit consumers.

People often think of Thomas Edison when they think of innovation. This focus on the single inventor can sometimes give the wrong impression of how successful innovations take place. In reality, Edison surrounded himself with teams of creative individuals. It has been said that innovation is a team sport, requiring teamwork.

But how do you approach staffing your ideal team to drive innovation in your company?

Perhaps the most important aspect of innovation leadership and building the ideal innovation team is to foster cross-functionality. Really focus on getting people who serve specific functions in the team, but who also have a shared vision and shared incentives. This ideal innovation team is not just assigned to a common project, not just sitting together. The ideal innovation team is really working together and pushing boundaries to take on additional roles outside of their areas of specialization.

From a functional expertise perspective, here are the essential functions or team members:


Engineers bring technical expertise in product design and development to the team. Over time, I came to broadly characterize two types of engineers as critical to successful development. Both types have their own inherent strengths and are equally valuable.

"Idea" Engineers
These are the creative “inventive engineers” who are always coming up with new ideas. These individuals are extremely important to have on the team, but often hard to keep focused. They like to move on to the next challenge or exciting problem to solve. These out-of-the-box thinkers come up with new solutions to break new ground.

"Closer" Engineers
These are the heavy lifters who are needed to follow through to make the big, creative ideas become a reality. They work out the problems, build the prototypes and run the tests. They are essential to getting to final designs that can be manufactured. This is certainly not to say that these individuals are not creative or that they do not also come up with great ideas. They just tend to excel in dotting the “i’s” and crossing the “t’s” required to advance a radical idea.

These team members capture opportunities as defined by Thomas Edison when he said, “Opportunity is missed by most people, because it shows up dressed in overalls and looks like work.”

Occasionally, someone will find an individual engineer who embodies both characteristics, but in my experience, most people excel in one direction or the other.

CAD Design

Fifteen years ago, it was not the norm for engineers to do their own CAD. However, the digital design world has evolved and this is no longer the case. With that said, having a dedicated CAD designer on the team can free engineers to concentrate on other tasks. Otherwise, engineers would be devoting “screen time” to refining concept design for rapid prototyping. In many instances, a seasoned CAD designer can fulfill the role of the “closer” or heavy-lifting engineer. They will build models, test and refine designs for manufacturing.

Industrial Design

Industrial Design is an extremely valuable skill to have at all stages of concept development. Industrial designers bring the Design Thinking process to life. Beginning with understanding customer needs, translating insights into concept design, and integrating human factors into usability, their work is essential. Good industrial design work isn’t possible when the designers are brought in at the end to “make it look good.” It must be incorporated from the beginning of the process.


Early in the process, the team needs to be thinking about the market and whether their ideas would fit in the current market environment. Marketers and business people know what sells and how to make the case for the product. In many companies, they often drive a project and should always be included in discussions and planning from the outset.

“Hard work is still wasted on features that don’t make the marketing headlines,” says GV Design Partner Jake Knapp in an article on product design and marketing. “Instead of the icing on the cake, I like to think of marketing as the sugar in the batter. You’ve got to get it in before the cake gets baked.”


In medical device design and development, the team should include an expert with in-depth clinical understanding. This is often the end-user physician. In addition to physicians though, it is crucial that the team also consider inputs from the entire healthcare ecosystem. This includes personnel who may be involved in the purchase, such as the value-analysis committee. (Learn more about the essential role of a value analysis committee.) Also the project needs to consider those involved in the preparation or use of the final product, such as technicians and nursing staff. This clinical knowledge may come from an individual clinician or a physician advisory panel, augmented by formal usability and concept research with users.

Support Functions

In an innovation setting, there will need to be additional support functions. Some of these important roles might be contracted from the outside, depending on the size of the organization. These roles include legal advisors, HR professionals, finance professionals, IT professionals, regulatory affairs, quality and operations management professionals.


Ideally, the team leader should come from one of the functional roles on the team, engineering, marketing or design. The team leader serves as the main point of contact with management and other entities that need to be engaged to keep the project moving forward, and they must be able to recruit.

With the team leader also playing a functional role on the team. They are more like a “Player Coach,” providing direction while making meaningful contributions to the advancement of the project. At the same time, every member of a high performance innovation team needs to be a leader in his or her own right.

Transitional Innovation Leadership
In this model, leadership may be transitional. Marketing and Industrial Design may lead the early stages of the project. For example, in the phases of understanding the customer needs, conducting market research and developing insights that shape the work. Later, leadership may transition to Engineering and Design leading during the ideation and concept development phases. At that time, Marketing then focuses on developing the business case for moving forward. With concepts in hand, leadership may transition back to Marketing for final validation research, pricing and launch planning.

Regardless of which function is taking the lead for a project phase, the best innovation comes when other functions are included throughout. This helps everyone to better understand what needs to be done from a big picture perspective. This also helps teams feel more invested in the outcome as well. Cross functional teams mean getting rid of information silos and opening communication.

Hot Teams

These ideal innovation teams can also be thought of as “hot teams.” These embody the idea of a cohesive group, working well together. Here the whole is greater than the sum of the parts.

The ideal innovation team does not need people who can only work in their specific areas of expertise, but who exhibit cross-functionality. People who don’t fear trying new roles. These individuals possess certain characteristics, such as their abilities to work together as a team. They also use their leadership skills to advance the work before them, take directives from management and embrace a fluctuating team structure.

They should also possess the tact and ability to navigate corporate processes to accomplish their team’s goals. But, be highly focused on reaching these relevant milestones in line with the end objective(s).

All things considered, ideal innovation teams need to have the right combination of skill sets, and must be willing to work collaboratively.

For those interested in learning more about how to form the ideal innovation team, I have written an e-book with input from my innovation-minded colleagues at Kaleidoscope that is free and available for download here. In the Ships and Castles Model I describe details on how to navigate front-end innovation efforts while fortifying an existing product line.

Ideal Innovation Team Sources

Clinical, Technical, Commercial and Organizational Considerations

Over the course of my 25 years working on and leading teams engaged in medical device development, I have experienced a variety of approaches to staffing the ideal innovation team.

In Part 1 of this series, I described a “functional approach” based on key technical skills team members should possess. Alternatively, in this section I describe a leaner approach based on critical categories of thinking required for medical device development.

In a lean startup environment, you can’t always access or afford all of the specific skills you might desire. At the same time, you do need to ensure your team is prepared to address the clinical, technical and commercial considerations inherent in developing medical product innovations. Depending on your organization’s size, the team may also need to be prepared to address organizational variables.

Building a team to address the clinical, technical, commercial and organizational considerations of product development requires a different way of looking at the individuals you choose. Rather than focusing on a person’s primary technical skill (i.e. engineering, design, marketing), identify team members who have the breadth of experiences necessary to successfully navigate the requirements in each category of thinking. From my experience, these individuals can come from various technical backgrounds.

Let’s look at the role each category plays in medical product innovation.

Clinical Considerations

In medical device development, a deep understanding of the users and clinical problem is critical to developing successful solutions. For instance, the concept development team must understand the problem, anatomy, physiology, pathology, users, use environment and so on.

Someone on the team needs to develop this multilayered understanding. This allows the team to represent patients, physicians, other healthcare professionals and key stakeholders who will benefit from the solution.

Depending on their training and backgrounds, this in-depth clinical knowledge might be a stretch for some. But with diligent observational research, relationships with consulting subject matter experts and secondary research, this knowledge can (and must) be integrated into the team. A good scientific or clinical advisory group, composed of relevant subject matter experts, can be invaluable.

Although this clinical understanding speaks specifically to medical device development, it has an equally critical corollary in any field of innovative product development. Simply foster a deep understanding of the end users and the job(s) they are trying to accomplish.

Some methods and tools that can help develop this knowledge include:

  • Ethnography and customer observation
  • Regulatory assessments
  • Procedure maps
  • Clinical stakeholder assessment

Commercial Considerations

Much like fostering an understanding of clinical considerations in your team members, integrating commercial considerations is highly important. Even if your innovation team is staffed exclusively with engineers or individuals with technical backgrounds, someone on the team needs to be ready and able to put on a business thinking hat. Ideally, this individual would come from a business or marketing background or have additional experience in these fields.

Examples of commercial considerations the team should address include:

  • Customer value proposition
  • Claims exploration
  • Competitive assessment
  • Business plan development

Innovation teams that fail to incorporate these commercial considerations in developing their solutions run the risk of creating wonderful technical solutions that the market will not embrace for any number of reasons.

  • Tools to help make sure the innovation team addresses these considerations include:
  • Concept selection criteria
  • Concept exploratory research
  • Customer segmentation and persona development
  • Financial modeling
Back to Insights + News


  • Dr. Mike Clem, DVM, MS

    Dr. Mike Clem, DVM, MS

    Vice President of R&D

    Vice President of Research and Development, Medical, Mike Clem, DVM, MS, thrives at the juncture of medical technology and clinical understanding. From his training as a veterinary surgeon through more than 20 years with Johnson & Johnson, Ethicon Endo-Surgery, Ethicon and Cordis, he has developed innovative solutions to complex customer problems. He covers medical device design for Kaleidoscope.