Project Lead: Ashley Solloway
UX Researcher: Timothy Nichols
Research Analyst: Mariah Coolridge
Testing Moderator: Ellis Skillman
NOVEMBER 19, 2022
Table of Contents
1. Executive Summary
2. Goals
3. Methodology
- Outreach & Screening
- Participant Profiles
- Testing Setup, Moderation & Post-Test Questions
4. Results
- Overall Concept
- Category 1
- Category 2
5. Bugs & Issues
6. Recommendations & Action Items
7. Disclaimer
8. Appendix
- Documentation
- Media
- Notable Quotes
K8 For Kids wants to embrace future technology and connect on a more personable level with the students involved in the K8 program. We help them to bring their star personality to life through a combination of technology and imagination. The result was a robot with as much sparkle and life as the artificial intelligence housed within it. Along the way, we learned a great deal about what makes us individuals and how to fuse personality with body language. The study shows K8's functionality and effects among students.
- Comprehensive study to get feedback on how K8 faired in comparison to other AI programs.
- Surveys focused on parents
- Collaboration with Menlow Park Elementary School Science Fair
Focus group:
- All of the participants were interested in the product show some or high interest.
- 40% of the participants lost interest within the first 10 minutes of testing the product
- 45% of the participants had difficulty with K8.
This research report includes the details of how we conducted our research, the results we found, and a brief list of recommended changes on K8.
72%
of participants were able to successfully complete all tasks.
54%
of participants enjoyed this specific experience.
68%
of participants had difficulty using this specific aspect of the product.
28%
of participants were unable to complete all tasks.
33%
of participants complained about this specific experience.
53%
of participants would continue using this product.
Executive Summary
Goals
Test product concept with target audience.
It is important to give K8 the related functions aiming the right target audience. Main goals were to:
- Improve K8's communication skills
- Understand K8's weaknesses
Reveal friction points and confusing experiences.
Although K8 is very interesting for prospective users, it is crucial to identify the difficulties that users potentially face with. Product team's goal is to find confusing experiences in real-time.
Identify bugs and issues with the product.
During the first test before K8 was completely ready, multiple bugs were found and these bugs & issues helped technical team to develop the product according to users' needs. Second test's goal was to make it perfect in every aspect.
Methodology
Outreach & Screening
Users tested K8 in various ways. Test groups were divided into age groups and Robotics software was adapted depending on users age. For the group who are between 8-12, separate method was followed.
Channels Used:
- Through Robotics software
- Robotics live experiment
Screening Criteria:
- Participants are from different age groups between 8-25
- Participants trained separately to use Robotics software depending on their age
Participant Profiles
Age | Gender | Income | Income | |
|---|---|---|---|---|
Alicia Mace | 10 | F | Elementary school | No income |
Chloe Johnson | 8 | F | Elementary school | No income |
Jack Nichols | 20 | M | College | $15,000 |
Ashton Grace | 25 | F | Masters | $50,000 |
Brad May | 15 | M | High School | $10,000 |
Taylor Bobson | 9 | M | Elementary school | No income |
Patricia Joy | 12 | F | Middle school | No income |
Maria Soledad | 22 | F | College | $14,000 |
Tim Nuyang | 16 | M | High school | $9,000 |
Matthew Albertson | 19 | M | High school | $9,000 |
Pie Chart
K8 User
Experiment
30% 10 and Under
30% 20-26
40% 18 and Under
Testing Setup, Moderation & Post-Test Questions
1. Introduction
We did a comprehensive study and interviewed over 3500 students to get feedback on how K8 faired in comparison to other AI programs. We quickly discovered that there was a serious communication problem between K8 and the kids in the program. Only 7% of human communication is done through written language, so with only typed commands to guide her, K8 was at a serious disadvantage.
2. Tasks
The tasks participants were ask to complete:
Create an account by screening process
- Edit your profile by answering questions
- Get your photo taken
- Take a timer and a name card
- Find your guide
- Bring back materials after the steps are completed
3. Post-Test Questions
After the test was completed; following questions were ask to the participants:
How would you describe this product to someone?
What was your favorite aspect of the product?
What was the most confusing part of the test?
Would you continue using this product?
Would you recommend this product to a friend or colleague?
Results
The response to K8 has been phenomenal. By the end of 2017, K8 For Kids had made the Gizmodo's list of 100 best companies to watch. They have been written up in Gizmodo and received the coveted "Best Educational Startup" from Tech Investors Weekly and Educational Award from the PNW Teachers Association.
Overall Concepts
83%
of participants understood the core premise and overall concept of this product.
54%
of participants successfully completed all tasks related to the overall concept of the product.
72%
of participants enjoyed the concept and the overall experience of using this product.
24%
of participants experienced difficulty using this product.
THE GOOD
"K8 is so smart. It was a lot of fun to speak
with her."
Alicia, 10 years old
THE BAD
"K8 couldn't follow my orders. I was a little disappointed."
Taylor, 9 years old
Category 1
83%
of participants understood the core premise of the first category for this product.
54%
of participants were unable to successfully complete all tasks related to the first category.
72%
of participants enjoyed the overall experience related to the first category using the product.
24%
of participants had difficulty using certain functions related to the first category for this product.
THE GOOD
"I observe Robotics very closely and K8 is the greatest product they have so far."
Ashton, 25 years old
THE BAD
"I don't think K8 is going to be helpful since it can only be useful for a few simple tasks. I was expecting more skills."
Maria, 22 years old
Category 2
83%
of participants understood the core premise of the second category for this product.
54%
of participants were unable to successfully complete all tasks related to the second category.
72%
of participants enjoyed the overall experience related to the second category using the product.
24%
of participants had difficulty using certain functions related to the second category for this product.
THE GOOD
"The best part of K8 is that it is so much fun!"
Chloe, 8 years old
THE BAD
"I was expecting K8 to function and communicate so much better."
Tim, 16 years old
Bugs & Issues
Some issues and bugs were identified during the testing process.
GENERAL
K8 receptors were not functioning as it supposed to.
An error occurred when the user tried to give commands to K8.
CATEGORY 1
K8 had hard time to manage multiple orders at once.
An error occurred during the second-phase of function test.
CATEGORY 2
Unexpected functions appeared although they weren't commanded.
The process was slowed down after multiple usage.
Recommendations & Action Items
Task List
0 out of 3 items completed
The overall concept should be improved and clarified.
K8 needs to be tested again after certain improvements.
Receptors should be renewed and tested, as a result of testing.
Task List
0 out of 1 items completed
K8 needs to be programmed again, considering multiple order functionality. As requested by users, it is significant for K8 to be able to receive multiple commands at once.
Task List
0 out of 2 items completed
Testing showed that over-usage can affect K8's performance. It needs to be improved and speed should be maximized.
Unexpected functions should be tested again clearly and reasons for them to occur should be identified.
Disclaimer
Mostly related to technical issues, several factors may have affected the results of the usability test.
The sample size may not be representative of all people that may use the product.
While moderation began with a set agenda, questions and tasks administered varied from participant to participant.
Human error and misinterpretations of qualitative feedback affected the results to slight but varying degrees.
A series of common product bugs and errors occurred (listed in section above).
Appendix
Documentation
Menlow Park Elemantary School - www.menlowk12.edu
Tech Stars Association - www.starsoftech.info
Robotics Association Annual Report, 2018
Menlow Community Fair Research Group, 2018
Media
Tech Investors Weekly- www.techinvestorsweekly.com
- Robotics media - www.roboticsmedianews.com
- Menlow News Group - www.menlownews.com
Notable Quotes
"I observe Robotics very closely and K8 is the greatest product they have so far."
"K8 is so smart. It was a lot of fun to speak with her."
"The best part of K8 is that it is so much fun!"