The project review below was built in August 2017, three years ago.
Here are some of my recent reflections on this project.
Please keep scrolling down to see the original full version project reivew.
FastPass
A personalized learning tool for adult learners to learn how to strategically solve their learning problems
Immediate Value:
-
Be able to do the independent mistake analysis;
-
Be able to better prepared for the assessment;
-
Knows how to effectively seek help in the learning process.
Long-term Value:
-
Improve their meta-learning skills to better plan, monitor and evaluate their learning process, so that they become the effective leaners of any circumstances.
Immediate Value:
-
Better understanding of WGU students' learning behavior patterns;
-
Improve the efficiency and effectiveness of learning service ( e.g. the course mentors and the student mentors ) based on the student data collected through FastPass;
-
Gain insights for the curriculum design and content purchase.
Long-term Value:
-
Stand out from competitors through not only giving students personalized learning experience, but also helping students become the effective leaners.
Project Overview
In my capstone project in CMU, my team works with Western Governors University (WGU), U.S., to help WGU students improve their career development fluency and learning fluency. We built an instructional online course module to teach career management skills. We prototyped a personalized online tool -- FastPass -- to help WGU students strategically solve their learning problems.
This portfolio will only present my work of FastPass since I am responsible for the whole design work of FastPass over this summer. We prototyped the tool based on the context of the algebra course and provided WGU with the design roadmap for other domains.
My role in this four-people capstone project team:
-
Research lead of the whole capstone project (assigned) : career development fluency and learning fluency;
-
Responsible for "FastPass" and created all artifacts: design research, product design, wireframing, user testing, UX and UI design, design roadmap, demo video production.
About Our Client
www.wgu.edu
Western Governors University (WGU) is a private, nonprofit, online American university based in Salt Lake City, Utah.
The university was founded by 19 U.S. governors in 1997 after the idea was formulated at a 1995 meeting of the Western Governors Association. WGU is ranked by US News and World Report's 2017 edition of best colleges.
The university uses a competency-based learning model, with students working online. Competency-based programs allow students to demonstrate through assessments that they have acquired the set of competencies (levels of knowledge, skill, or ability) required for a particular degree or certificate. WGU is the first accredited school to use the competency-based approach, driving a growing trend toward competency-based higher education.
-- Wikipedia
With our client and faculties
With my team
Research
-
Literature Review
-
Learning Data Analytics
-
User Interviews
-
Cognitive Task Analysis
Ideation
-
Affinity Diagraming
-
Storyboards
-
User Journey Maps
-
Persona
-
Sketching
-
Wireframing
Design
-
High-fidelity UI Design
-
Mockups
-
Learning Science Validation
-
User Testing
-
User Flow
Vision
-
Iterative Design
-
Feedback Integration
-
Design roadmap
-
Service Blueprint
Solution Scalability
Research
-
Define the Design Problem
Design Problem
Literature
Review
Data
Analysis
User
Interviews
Definition of the “Learning Fluency” suggested to WGU based on our literature review
-
Understanding and having the motivation to learn;
-
Analyzing the learning context, setting and managing meaningful learning goals;
-
Determining which learning strategies to use;
-
Assessing whether the strategies are effective in meeting the learning goals;
-
Evaluating emerging understanding of the topic, and determining whether there are aspects of the learning context which could be used to facilitate learning.
Literature
Review
Literature
Review
Data
Analysis
User
Interviews
WGU has been using DAACS assessment(http://srl.daacs.net/), a diagnostic assessment designed to help students transit to college, to assess students’ level of learning skills. The data we got from WGU contains assessment score of 1711 students, which covers 15 skills related to students’ grit, strategies, motivation and metacognition in terms of learning. From the data analysis, we found that:
-
In comparison to other skills, WGU students have lower performance when it comes to grit and motivation.
-
Evaluation, managing time, monitoring and planning are four skills that especially need improvement.
-
Managing time, motivation, help seeking, planning are significant predictors of WGU students’ grit.
The liner regression (stepwise) showed that managing time, motivation, help seeking, planning, self-efficacy, and monitoring are significant predictors for grit (R=0.714).
We compared the liner models include different numbers of predictors, and it showed that a simpler model including managing time, motivation, help-seeking, planning performs only slightly lower than the full model (R=0.710).
Grit = 0.022+0.342* managing time + 0.290* motivation+ 0.112* help seeking+0.101* planning
Literature
Review
Data
Analysis
User
Interviews
We conducted 30 user interviews face-to-face and remotely with WGU students. We grouped the user interview insights based on the guiding questions.
What are WGU students' learning goals?
WGU students are very assessment-oriented in their learning. They want to pass the assessment as soon as possible.
"I have a full-time job and two kids. I have very limited time and energy to spend on study. So I just wanna quickly pass the assessments and get my degree."
- Emily WGU student
Most of the users we talked to wanted to pass the assessment as quickly as possible. This insight helped us narrow our scope to focus on learning activities that will directly help students pass the assessment to better align with students' needs and motivations.
What are WGU students' learning behavior patterns and mindsets?
Many students have a negative self-judgment if they fail a pre-assessment and don't know how to effectively correct their mistakes.
"I never take the pre-assessment first, even if I have some experience on the subject. I'm scared of getting a low score. If I don't pass the 'cut score', I would feel frustrated and anxious."
- Jennifer, WGU student
From our literature review, we found that the appropriate use of the pre-assessment can accelerate students' learning. However, students currently don't fully utilize the pre-assessment. We want to motivate students to take the assessment.
While many students are self-confident about their learning strategies, they have difficulty accurately detecting their knowledge deficits. Students rarely reflect on how they can effectively identify and improve the knowledge gaps.
"I don't go back to the textbooks to review the chapters. I don't think that is useful because I already read that but still got low score. I usually go to the recorded webinars because I find the questions mentioned in the webinars to be very similar to the real pre-assessment questions. But not every course has the useful webinars."
- Jack, WGU student
While many students are self-confident about their learning strategies, we want to help them become aware of the effectiveness of different types of learning strategies and always use effective learning strategies in their learning process.
What are WGU students' learning needs?
While many students enjoy learning at their own pace, they often struggle with planning and monitoring their learning due to lack of strong meta-learning skills.
"I decided to go to WGU because I could go at my own pace. Sometimes I really don't know how I'm doing, and I often get off track since my study routine isn't consistent. I don't know how to assess my learning process."
- Emily, WGU student
WGU students have lots of control over when and what they study and the resources they use. In our interviews, we found that not all students have sufficient metacognitive skills to learn by themselves. If they cannot effectively manage their learning, students can lose interest and motivation in learning. Thus, our design should help students improve their metacognitive skills.
I was grouping the interview quotes and creating the affinity diagram
Based on the research, we defined our design problem as:
WGU students don’t strategically solve their learning problems while they expect to accelerate their learning to get the degree as soon as possible.
-
Define the Learning Goals
We conducted cognitive task analysis with experts and novices. The experts are the professors whose research areas are metacognitive learning skills in CMU. The novices are the WGU students who failed to pass the "cut score" of their pre-assessment. The tasks both the experts and the novices are given are how they plan, monitor and evaluate their learning. The main differences between the experts and the novices are:
Planning
Experts
Novices
-
Ask themselves questions before learning:
"What do I want to learn? How do these topics relate to what I already know?"
"How much time I will spend on this? What learning resources do I have? "
V.S
-
Don't brainstorm the prior knowledge on purpose before learning something new
-
May think about the study plan but don't verbally state it or write it down
-
Write down the study plan
Monitoring
Experts
Novices
-
Ask themselves questions after learning:
" Did I learn what I set out to?" " What did I do that worked well?" "What should I change?"
V.S
-
Pay more attention to the assessment scores and the correct solutions rather than why they got wrong in the assessment
-
Rarely reflect on what learning strategies worked well or not, and how they could do differently
-
Recognize concepts they still need to master and find the resources to clarify confusions
Evaluation
Experts
Novices
-
Generating questions to quiz themselves
-
Reorganizing the information in a different way (e.g., list similarities between concepts in a Venn diagram)
V.S
-
Don't intentionally use the methods to check their comprehension and heavily rely on the pre-assessments to monitor their learning.
-
Ask them selves:
“What have I learned? Does what I learned match my learning goals? If not, where did I diverge from my goals and how can I get back on track?”
-
Don't realize they need to check if they are meeting their learning goals
Based on the design problem and the cognitive task analysis, we created the learning goals specifications:
Planning
Monitoring
Evaluation
Ideation
User Journey Map of WGU Students' Current Learning Experience
User Persona
Mapping the learning problems, learning needs and the learning strategies showed effective
Sketching ideas
I was laying out the sketches according to the students learning process (based on the persona I created).
Wireframing
Learning Goals
1
Landing page
WGU Coaching Report
FastPass is located right next to the coaching report to get the user's attention. The marketing content of FastPass can be added around the button. The key information conveyed to the user will be: this is the tool can help you quickly and effectively learn from the pre-assessment and the coaching report so that they can quickly pass the assessment and get the desirable score
2
FastPass Home
User gets big idea of the learning process
Explicitly let the user know the mapping among the learning problem, learning needs, recommended learning strategies, the expected outcome
Clickable questions
are practice entrance
3
Practice of "Break it down"
Emphasizes the purpose of this practice and the estimated time to spend on this
Highlights the concepts needed in each step
if the user put the wrong answer, FastPass can automatically diagnose which concept the user doesn't understand, and will prompt the user to review the concept. The detail explanation and the worked example of the concept will pop out if the user click it
The user chooses the question that was got wrong in the pre-assessment to learn as the worked example
4
Fading of the scaffolding
Guide the user to try another similar question to practice. and check if the user knows how to identify their knowledge deficits through articulating the solution step by step
All the concepts that are tested in this question are listed here for the user to decide which one should be used in which step
5
Back to Home
The questions have been practiced will be differentiated with the questions haven't by using different colors, so that the user can decide the next step
6
Practice of "Create your own final sprint"
The user chooses the question that was got wrong in the pre-assessment as the practice question
All the concepts that are tested in this question and the learning resources (e.g. webinars, textbook chapters) are listed here for the user to choose and archive to the final sprint
Questions and learning resources the user wants to archive to the final sprint
7
The date of the final assessment of the course
Questions, concepts and learning resources are categorized
Personalized final sprint
The calendar visually helps the user to plan and monitor how much time left to prepare for the assessment
Colorful labeled dots helps the user quickly make the notes in the final sprint by pinning the dots next to the questions, concepts and learning resources. The user can customize the label of the dot, including the color and the meaning, according to own preference and needs
The user can set the reminder frequency and the time
Visualize the review timeline and the review task
Personalized review plan of the final sprint
The user has two options to set the daily review tasks:
-
Manually drag and drop the questions got wrong, the concepts, and the learning resources to the task review plan for each day;
-
Allow FastPass to automatically set the personalized task review plan based on the user's learning behavior log data.
8
Note: The above wireframes are for the practice of "break it down" and "create your own final sprint". we will only prototype these two learning strategies. The design for remainders will be presented as the design roadmap.
1
The design solution that is assumed to solve the design problem we defined:
Build personalized learning tool to help WGU students solve the learning problems associated with assessments through mastering effective learning strategies.
Why we chose to create a tool not build a course to teach learning skills?
-
Emerging pedagogy: Learn by using
Learning science research shows that, especially for the higher education, students demonstrate more behavioral changes if they use or practice the learning skills through technology-based tools, which improve both learning outcome and students’ motivation to learn (Dragon, T., et al. 2013);
-
Students' characteristics
WGU students are very sensitive to the study load because of their busy schedule. Compared with building a traditional online course, we believe a personalized interactive learning tool can better motivate and engage students to learn and practice the learning skills.
Why we focus on the learning problems associated with assessments?
-
Tap into intrinsic motivation
From user interviews, we understood that WGU students want to pass the assessment as quickly as possible. We want to help students achieve this goal while teaching them learning skills, which is the best way to make students realize the practical value of the learning skills.
-
Solve real learning problems
Our user research shows that the typical learning problems that WGU students have are related to the assessment: for example, students don't fully utilize the pre-assessment to diagnose the knowledge deficits and they don't have effective strategies to prepare for the assessments. We want to help students solve their real learning problems by using effective learning strategies.
Design
-
User Testing
3 Rounds
10 WGU Student Participants
Round 1
Speed Dating: Understood More about WGU students' learning problems and learning needs
We presented three storyboards to the participants. Each storyboard shows one assumption about WGU students' learning needs. We use the lead questions to guide the discussion with the participants.
Storyboard 1
Storyboard 2
Storyboard 3
We were observing one WGU student using WGU online learning recourses
One WGU student was doing card sorting during the user testing
Key findings
Solving problems associated with the pre-assessment is the key incentive for WGU students to learn and improve their learning strategies.
"That final sprint idea is more appealing to me. Honestly, I only want to spend time on the learning activities that can help me pass the assessment."
- Emily, WGU student
The design should be integrated into the pre-assessment results. It is important to make students recognize the benefit of using FastPass is directly relative to the pre-assessment.
The chatbot is telling students what this learning strategy is, the value of this learning strategy and what to expect from the learning activities.
Many WGU students don't strategically solve their learning problems.
"I haven't really thought about using any tools described in the scenarios to solve my problems. I mean I do have some of those learning problems, but I just always go back to the textbook or the webinars."
- Jennifer, WGU student
The design should make students understand that the effective learning strategy can help them overcome learning challenges instead of continuing to run into the same obstacles.
The chatbot is encouraging students to try similar question to master the learning strategy so that students are able to solve certain type of questions not just one specific question.
Think Aloud: Tested out the tool features
Round 2
We shared our design with the participants through the video chat tool and asked the participants to think out loud as they interacted with the prototype. Our goals are to further locate the pain points of the WGU students' learning experience associated with the pre-assessment, determine students' expectations of the learning tool, and identify what aspects of the design are confusing.
I was doing the remote user testing with one WGU student
One WGU student was doing "think aloud" when looking at the screen we shared through the video chat tool
Key findings
WGU students need more scaffolding and training related to the coaching report.
"The current WGU coaching report doesn't really help me in terms of study competencies. It is static and only directs you to the textbook."
- Jack, WGU student
We embedded the coaching report to FastPass and made all the learning activities to be closely related to the coaching report.
The coaching report is on the FastPass interface. The practice questions and the learning problems are all based on the coaching report.
Students expect more user-friendly learning scaffolding from WGU.
"I really like the personalized final sprint! WGU gives us a study plan, but I can't move around the items based on my own schedule. I don't follow the study plan. It is static. If I could customize the study plan, then it would be more practical."
- Jack, WGU student
Using this insight we added more personalized features to FastPass to better engage students to plan, monitor and evaluate their own learning.
Students can customize the concepts and the learning materials they want to review for the final assessment by "drag and drop". The color labelled dots allows students to pin the concepts and the learning materials to make the quick review notes.
Students get off track not only because of their busy schedule but because they lack the tools to help them better plan and monitor their learning.
"I sometimes forget to study because I have so many thing to-do in a day. But I know I actually have at least 20 minutes to read the textbook. I don't know how feasible it would be, but I hope WGU can somehow remind me of my study plan everyday. I need this to stay on track."
- Emily, WGU student
FastPass should work as the tool that not only solves students' learning problems but also provide students with the personalized, interactive services. Small features can have big impact on students' learning behavior.
FastPass allow students to set the review time and the reminders. The daily review agenda visualizes the review plan and helps students to monitor their review tasks.
Round 3
Think Aloud and Direct Observation: Usability Testing
In this round, the participants were asked to interact with the whole prototype of FastPass end-to-end, while they were being observed, to see where they encountered the problems and experienced confusion.
Key findings
Students would be more motivated to continue the use of FastPass if they could anticipate how much time and efforts they would spend on the learning activities.
"I would be more inclined to click the next button if I know where I am now and what to expect next."
- Emily, WGU student
The design should take into account the students' needs as the users not just as the learners. We want to decrease the cognitive load which are not beneficial for the learning engagement.
Although the learning experience within FastPass is not necessarily linear, the students can get a high level of understanding about where they are now and what to expect next through the progress bar located at the top of the interface.
The learning scaffolding can leverage the learning experience only when it is concise and interactive.
"I like the prompts from the chatbot but sometimes I found the prompts to be annoying , especially when there is no substantial information there. I just want to skip and skip.""
- Jack, WGU student
The prompts for the metacognitive level thinking should be engaging in terms of both the content and the format.
Instead of just asking the students to think about the concepts tested in each step through the pop-out window, FastPass prompts them to think deeper while providing interactive tasks and allows the students to access the relevant information without going outside of FastPass.
-
Learning Design Experts Consulting
In CMU, we have first-tier learning design experts in the world. We asked for feedback and suggestions from our faculties who are the experts both in learning science and learning experience design.
Marti Louw, the Director of Learning Media Design Center in CMU was giving us feedback to our initial versions of design from the learning experience design perspective.
Main takeaway:
For any learning design, the pursue of learning outcome should not sacrifice the user experience.
What did differently based on the experts suggestions:
-
Change the learning science jargons into easy-to-understand terms;
For example: we changed one of the learning strategy's name from "Decompose the knowledge components" into "Break it down"
-
Consider the user flow to make the user experience of FastPass to be seamless;
For example: we spread out the entrances of different learning strategies across the interaction with FastPass to allow the students can easily switch between the different learning activities based on their own learning needs.
Based on the user testing and learning design consulting, we finalized the design solution as:
Build a personalized learning tool
to help WGU students solve the typical learning problems associated with the coaching report through scaffolding students to practice and use effective learning strategies.
Final Design
The first-time landing page
-
Learn the learning strategy based on the learning problems
" Break it down"
The student is guided to solve the question step by step
Recaps the purpose of the learning activity
The student should think about the concepts needs to be used in each step first, and then compare the answer with the correct one. The student can directly review the concepts and worked examples with a single click without going out of FastPass
FastPass chatbot is prompting the student to strategically think about the steps at the metacognitive level
The student can still get access to the detailed explanation of the concepts and the worked examples if he or she put the correct answers.
After solving the question, the student can see how the key concepts are mapped to the steps. FastPass prompts the student to archive the question to the error deck and the final print. The student can get access to the error deck and the final print here.
" Formulate the question"
Recap on the purpose of the learning activity
The key features and their outcomes. Students can get access to the error deck, question sheet and final sprint directly either from HOME page menu or through the entrance spread through the learning activities.
The progress of the activities helps students monitor where they are and understand what they are supposed to do next.
Students' pre-assessment performance of each competency. Different colors indicate the different performance levels.
The learning path and outcomes are explicitly explained to let students understand the value of the learning activities.
The learning content is course-based while the layout of the interface will be the same.
The question the student got wrong in the pre-assessment. The questions are mapped with competencies and grouped under the correspondent competency.
The learning problems are generally categorized and the student is asked to choose the learning problem she or he has about this question.
Map the question, the learning problem and the effective learning strategy.
Explain in detail about what is this learning strategy, the value of this learning strategy and what to expect from the learning activities.
Explains why setting the context of the question is important and how to ask specific question while the student is doing these activities.
FastPass explains the value of doing each step
Prompts the student to reflect on what efforts he or she has made before turning to the course mentor, and help the student articulate the expectation of the feedback.
Encourage the student to share the question and the feedback with peers.
-
Learn from the errors
The students can archive all the questions they got wrong in the pre-assessment to the Error Deck, which allows them to review their errors and the relevant learning materials.
The student can review their errors by competency (learning topic), by Concepts, and by date
Prompts the student to strategically learn from the errors by using the learning strategy to figure out why he or she got wrong. This is the another touchpoint of the learning strategy recommendation
The relevant learning resources have been mapped with the question. The student can review the learning resources with a single click
The student can take notes when they are reviewing the errors.
Prompts the student to archive the question to the final sprint so that it can be reviewed for the final assessment
-
Get prepared for the assessment
The students can create their own final sprint to prepare for the assessment. The template of the final sprint allows the students to customize the pace and content to review based on their own needs. The process of creating the final sprint does not only help the students plan and monitor their review, but also help them self-reflect on their learning progress.
Prompts the student to drag and drop the concepts and the learning resources to the daily review tasks.
Visualizes the time to prepare for the final assessment
Visualizes how the review tasks are scheduled and if it is implemented as planned. The student can get the reminder from FastPass
The questions, concepts and the learning resources are mapped together, and are grouped based on the competency, which is the basic learning unit in WGU's learning system
The student is guided step by step to frame the question
How it Works
The students self-diagnose the learning problems of each question of the pre-assessment. Then FastPass will recommend the most effective learning strategy and guide the students to use that learning strategy to solve their learning problems.
Key screens
User Flow Chart ( to be updated)
Service Blueprint
Final Prototype Demo Video
Video scripts
Version 1
What I learned
1
The learning experience design and the learning content design should go hand in hand.
It is true that learning experience design can leverage the learning motivation and the learning effectiveness. However, if the meatiest part of the learning: the learning content doesn't carefully consider the learners' characteristics and not structured to address their needs, the learning experience would not be improved as expected. As most of the WGU students have full-time or part-time jobs, their study time is actually fragmented and they get access to the learning materials more often through mobile devices. To motivate the students to access the learning materials more often, the video, the textbook, and the quiz/tests should be re-structured. For example, the study time for each unit should cost no more than 3 ~ 5 minutes so that students can absorb formal learning in smaller chunks; the interactive activities should include more "click box", "drag and drop" and "touch to highlight" features, which are more convenient to proceed through the mobile devices; More "recap" session should be attached to each unit to help activate the prior knowledge since students don't study in a linear way. The learning designer should consider the learning experience and the learning content as a whole, and should include the re-design of the content into the design solution.
2
Fill the gap between the teaching objectives and the learning needs.
The educator's teaching objectives and the learner's learning needs are not always matched. It is easy to think just as the educator or the learner, but it is hard to always think from both angles. For example, while WGU wants to improve their students' learning skills, the students just want to pass the courses and they didn't see the value of spending the extra efforts on learning to learn. We cannot just deny or ignore the students needs because we believe the teaching objectives are beneficial to the students in the long run. It is the learning designer's job to explicitly connect the teaching objectives and the learning needs through design. For example, we make the students to realize the underlying reason that prevent them from passing the course quickly is lacking of effective learning strategies. We embedded the teaching of learning strategies into the students' learning process to help them solve their practical learning problems. So that the students understand the value of the learning activities.
3
What should the learning design aim for? Knowledge delivery? Engaging? Or effective learners?
My simple answer to this question is we should target the adult learners meta-level learning to learn skills. It is especially true for the online learning. From the user research of this project, I found the big difference between the effective learners and the ineffective learners lies on how they plan, monitor and evaluate their learning process, which require strong metacognitive learning skills. But many online learning content are still focus on delivering the knowledge in a engaging way not the learning skills training. We can't blame the students that only focusing on the knowledge itself or the assessments if the learning experience we created neglected to teach them how to better navigate, digest and utilize the knowledge to solve problems. I think the curriculum design should be re-imagined for the adult learning. The fundamental goal for the curriculum design or any learning design should be teaching students to become the effective learners. The domain knowledge is the material we use to teach the learning skills, which will be beneficial for the students' learning in any stage of their life.