Frick and Boling - Effective Web Instruction (Chapters 3 & 4)

Summary:

This reading provides instruction for creating a paper-based prototype, conducting a pilot session, and analyzing results in order to provide necessary changes.

According to Frick and Boling, the main goal of prototype testing is to address three key issues with any instructional site:

1. Is the instruction effective?
2. Are students satisfied with the instruction?
3. Is the product usable by students? (p.70)

Designers want to make these determinations without spending unnecessary time and money. This is one reason for testing a paper-based prototype instead of a computer-based prototype. In addition, subjects are more forthcoming with feedback when what they are testing is clearly in draft form.

Critique:

The sections on prototype testing are excellent. I thoroughly enjoyed the reading and learned a great deal. At one point, the authors suggest that the best way to learn how to test prototypes is to watch someone who knows what they're doing. I couldn't agree more because this is how I learned how to facilitate focus groups. I observed a more experienced consultant conducting focus groups until I felt comfortable taking the lead. Unfortunately, I didn't have this opportunity with prototype testing. Now, I wish I had read these sections before my first prototype test. I can see where I went wrong in many areas and how important some of Frick and Boling's points are. Key points that I will take away from the reading include:

• "Spend the least amount of time, resources and effort in order to learn as much as we can about how to improve the instructional product we hope to eventually develop." (p. 19) - This is always important to keep in mind, especially since many clients have limited resources for formative evaluations. 
• "Observe subjects using the materials and ask them to think aloud." (p. 20) - I've certainly failed to do this in the past. Most of my clients insist on group pilot sessions, which makes it nearly impossible to observe all participants. Also, it forces us to draw our own conclusions because there are too many subjects for everyone to think aloud. 
• "Your learners will be more frank about their problems if they feel that the instruction is still being worked on--they usually don't want to bring up issues that might disappoint you." (p. 27) - This is very true and the best argument for testing a paper prototype initially. 
• "Don't bog down the process by testing a prototype with technical problems. Users will stumble on those problems and you won't get the best data that you could." (p. 38) - This is another mistake I've made in the past. People will jump to point out the obvious problems and many won't get past these issues. 
• "The cause of observed problems is always in the design, so diagnoses (or findings) are always stated in terms of design and not the subjects." (p. 72) - This is a great reminder. It's too easy and too tempting to blame subjects (e.g., they weren't computer savvy enough for e-learning) when test results aren't what you anticipated.

 The reading also left me with a few questions. First, are the authors advising that you conduct one-on-one sessions, so one subject per observer? It's been my experience that when clients ask for formative evaluation like this (and many don't), they set up group sessions, where it's hard to observe and listen to individual subjects. The way suggested makes sense, but I just want clarification that I understood correctly.
My other question is how do you conduct a pilot for a mult-module course with many different tasks? If each module is 40 minutes long and there are 8 modules, that's a lot of testing.

A final question is whether the paper prototype can take the place of storyboarding or if the design phase would include both.

Overall, I found the reading very informative and wish that the authors covered other sections such as learner and context analysis in such depth.