Frick and Boling - Effective Web Instruction (Chapters 5 & 6)

Summary:

The final reading by Frick and Boling is divided into three sections: building a Web prototype, assessing and maintaining the site, and conducting the analysis. The first section provides an overview of hosting servers and the programming languages you might use for your site. It offers different approaches for building the site as well as precautions.

The second section explains in detail how to test your site for quality assurance. It provides a sample QA matrix, explains how to use this matrix, and discusses the QA process.

The final section explains how to conduct a thorough needs assessment and analyze the results. It presents advantages and disadvantages of one-on-one interviews and focus groups. It also offers many tips for reviewing the data collected and eventually drawing conclusions. 

Critique:

This last reading started off slow but really provided some good information in the final two sections. I didn’t find the first section extremely useful because it either covered basic information such as its overview of the Web or information that I wasn’t interested in, such as the descriptions of different programming languages. However, sections 2 and 3 were well worth reading. I liked the QA matrix that was presented in section 2 as well as the systematic approach the authors take to QA. I also liked how they explained prioritizing bugs and their distinctions between priorities one through four. My favorite part of section 3 explained how analyze and synthesize data. I find this to be the hardest part of conducting interviews and focus groups. Collecting data can be fun and interesting but analyzing the data is often challenging and overwhelming. The authors describe an approach that I believe is very effective. I appreciated the practical tips that were provided throughout this section. For example, to identify trends in data, the authors recommend:

• Really look at what’s there
• Look for items that are similar
• Look for items that may be linked
• Look for items that are missing
• Pay attention to the questions that come to your mind
• Don’t be too quick to summarize the data
• Be sure to summarize the data eventually (p. 131-134)

Merrill's What Makes Effective, Efficient, Engaging Instruction?

Summary:

In this reading, Merrill starts off by identifying three recent approaches for instructional design and development including “problem-based learning, communities of learners, and distributed learning via the Internet” (p. 1). Merrill states that many times people consider these approaches to be distinct and separate when really they should be integrated in order to produce instruction that is effective, efficient, and engaging. Merrill asserts that his First Principles can be used to integrate the approaches and bring on e³ instruction.

While explaining his argument, Merrill briefly discusses two memory processes: associative and mental models. He explains that much of the instruction in the world relies on an associative model, which is not as effective as a mental model. By being problem-centered, Merrill’s First Principles rely more on mental models. 

Critique:

I’ve enjoyed exploring Merrill’s thoughts in more depth this semester and this reading is no exception. However, one of my pet peeves with Merrill is that his diagrams aren’t very intuitive to me and he doesn’t provide as many examples as I would like. Also, the examples he does provide could be more concrete and visual. For example in the second paragraph on page 4, he outlines how peer interaction might take place in the classroom. Rather than keep the scenario in general terms, I would have loved a specific example that I could actually visualize.

Snyder's Making a Paper Prototype

Summary:

The reading explains paper prototyping in great detail including what it is, how to do it, and why it's useful. It begins by outlining the materials needed to create a prototype and then explains why you might choose to include a background (e.g., poster board) and how to do it. The majority of the reading discussed three closely related topics: representing interface widgets, representing user's choices, and simulating interactions.

Near the end, the reading moves away from software prototyping and provides recommendations for hardware prototyping.

Critique:

In a way, this reading complemented the week 4 reading by Frick and Boling. Whereas that reading focused more on the prototyping testing process as a whole, this reading concentrated on the paper-based prototype itself. While I found some of the tips/techniques interesting, the Frick and Boling reading was more helpful. My biggest problem with the reading is that the prototype they created sounded very confusing, complicated, and cumbersome to me. It's a process I would have to observe in order to understand. I found myself thinking wouldn't it be easier to create a low-fidelity computer mock-up at this point, rather than keeping track of all these note cards, transparencies, and invisible tape? 

Frick and Boling - Effective Web Instruction (Chapters 3 & 4)

Summary:

This reading provides instruction for creating a paper-based prototype, conducting a pilot session, and analyzing results in order to provide necessary changes.

According to Frick and Boling, the main goal of prototype testing is to address three key issues with any instructional site:

1. Is the instruction effective?
2. Are students satisfied with the instruction?
3. Is the product usable by students? (p.70)

Designers want to make these determinations without spending unnecessary time and money. This is one reason for testing a paper-based prototype instead of a computer-based prototype. In addition, subjects are more forthcoming with feedback when what they are testing is clearly in draft form.

Critique:

The sections on prototype testing are excellent. I thoroughly enjoyed the reading and learned a great deal. At one point, the authors suggest that the best way to learn how to test prototypes is to watch someone who knows what they're doing. I couldn't agree more because this is how I learned how to facilitate focus groups. I observed a more experienced consultant conducting focus groups until I felt comfortable taking the lead. Unfortunately, I didn't have this opportunity with prototype testing. Now, I wish I had read these sections before my first prototype test. I can see where I went wrong in many areas and how important some of Frick and Boling's points are. Key points that I will take away from the reading include:

• "Spend the least amount of time, resources and effort in order to learn as much as we can about how to improve the instructional product we hope to eventually develop." (p. 19) - This is always important to keep in mind, especially since many clients have limited resources for formative evaluations. 
• "Observe subjects using the materials and ask them to think aloud." (p. 20) - I've certainly failed to do this in the past. Most of my clients insist on group pilot sessions, which makes it nearly impossible to observe all participants. Also, it forces us to draw our own conclusions because there are too many subjects for everyone to think aloud. 
• "Your learners will be more frank about their problems if they feel that the instruction is still being worked on--they usually don't want to bring up issues that might disappoint you." (p. 27) - This is very true and the best argument for testing a paper prototype initially. 
• "Don't bog down the process by testing a prototype with technical problems. Users will stumble on those problems and you won't get the best data that you could." (p. 38) - This is another mistake I've made in the past. People will jump to point out the obvious problems and many won't get past these issues. 
• "The cause of observed problems is always in the design, so diagnoses (or findings) are always stated in terms of design and not the subjects." (p. 72) - This is a great reminder. It's too easy and too tempting to blame subjects (e.g., they weren't computer savvy enough for e-learning) when test results aren't what you anticipated.

 The reading also left me with a few questions. First, are the authors advising that you conduct one-on-one sessions, so one subject per observer? It's been my experience that when clients ask for formative evaluation like this (and many don't), they set up group sessions, where it's hard to observe and listen to individual subjects. The way suggested makes sense, but I just want clarification that I understood correctly.
My other question is how do you conduct a pilot for a mult-module course with many different tasks? If each module is 40 minutes long and there are 8 modules, that's a lot of testing.

A final question is whether the paper prototype can take the place of storyboarding or if the design phase would include both.

Overall, I found the reading very informative and wish that the authors covered other sections such as learner and context analysis in such depth.

Frick and Boling - Effective Web Instruction (Chapters 1 & 2)

In this blog post, I summarize and critique the first two chapters of Effective Web Instruction: Handbook for an Inquiry-Based Process.

Summary:

Frick and Boling (2002) present an inquiry-based, iterative instructional design and development process in order to avoid common instructional pitfalls such as:

• No user input
• Little or no testing
• “No record of decision-making”
• No justification for design decisions (p.2-3)

In their process, objectives are created upfront and the assessments are built before work on the content begins. The process includes iterative reviews first of a paper prototype, then a computer prototype, and finally the site itself. The results of each iteration are analyzed and the site is improved based on this analysis. (p.4)

The instructional goals are developed with all stakeholders, recognizing that some perspectives are more valuable than others. The reading recommends thinking about how you’ll assess the instructional goals while you’re developing the instruction, which is in line with Mager’s guidelines for writing instructional objectives. (p.12)

The learner analysis section discusses the importance of knowing your learners and contends that the best way to do this is to try teaching the content to them, at least once. The final section, context analysis advises that you shouldn’t pursue a Web solution for no reason. You should ask yourself: “What can we do with information technology that could not be done without it to help students learn?”

Critique:

I didn’t find the learner analysis section particularly helpful or practical. Yes, it’s great to teach the subject matter at least once in order to better understand your learner needs, but I’ve never had this opportunity as an e-learning developer. So if you can’t teach the subject matter beforehand, how does one uncover learner needs?

I found the paper prototype recommendation interesting and am looking forward to that section of the reading and better understanding such an approach. My clients tend to request computer prototypes, so I’m not sure how the paper prototype would actually be implemented.

Mager's Tips on Instructional Objectives

This blog post summarizes and critiques "Mager's Tips on Instructional Objectives."

Summary:

The reading provides a good summary of Mager’s book—Preparing Instructional Objectives. The key points from Mager’s book are summarized.

The three main reasons for stating objectives:

1. They lay out a road map for you to follow when creating instruction.
2. If you don’t state the objective upfront, you won’t know if it’s being followed.
3. They provide an overview for the learner, which allows students to create a personal strategy for accomplishing the instructional goals (p.1).

Useful objectives are those that clearly define the audience, behavior, condition, and degree (p.1). Behaviors can be overt (observed directly) or covert (not observed directly). Covert behaviors require an indicator, so the performance can be demonstrated (p.2).

The reading concludes with some common pitfalls for writing objectives including: false performance, false givens, teaching points, gibberish, instructor performance, and false criteria. 

Critique:

While Mager’s book is very informative and helpful for people who have little experience with creating objectives, it is also long and spends a great deal of time explaining specific points. This reading summarizes Mager’s key points in eight pages. It’s an ideal reference for people who have already read the book but need a refresher or instructional designers who already have some familiarity with instructional objectives. 

The reading does a good job of reminding you to always state the main intent, not just an observable behavior. Mager recommends that you state the main intent and then an observable behavior in parenthesis if the objective is covert. I think this is a helpful practice for an ID who is planning their courses but it hasn’t been my experience as a student to see objectives stated this way. What do others think?

I appreciated the pitfall section, but I would have liked to have seen revised statements. Examples are provided of what not to do but it would have been helpful to see a revised example of the same statement, specifically for the teaching points section, which I found confusing. 

 

Kim and Frick - Changes in Student Motivation During Online Learning

In this blog post, I summarize and critique Kim and Frick's (2011) "Changes in Student Motivation during Online Learning." 

Summary:

Taking into account the high attrition rate in online courses and the fact that attrition is often caused by a lack of motivation, Kim and Frick (2011) investigate learner motivation in online learning. The first section of this reading is a literature review and the second section describes the actual study.

The literature review discusses internal, external, and personal factors that influence motivation in Web-based instructions. One internal factor is that instruction is more motivating if it applies the ARCS model (attention, relevance, confidence, satisfaction) or Merril’s first principles. On the other hand, “cognitive overload” can decrease motivation to learn (p.3). External factors such as technical difficulties with the learning environment or lack of support from an employer also influence motivation. Finally, personal factors such as preference for a particular learning style can impact motivation.

The rest of the reading describes the current study in great detail including participants, research instrument, and data collection/analysis methods. Table 2 on page 18 outlines eight instructional design principles based on the findings:

1. Provide learners with content that is relevant and useful to them.
2. Incorporate multimedia presentations that stimulate learner interest.
3. Include learning activities that simulate real-world situations.
4. Provide content at a difficulty level which is in a learner's zone of proximal development.
5. Provide learners with hands-on activities that engage them in learning.
6. Provide learners with feedback on their performance.
7. Design the website so that it is easy for learners to navigate.
8. If possible, incorporate some social interaction in the learning process (e.g., with an instructor, technical support staff, or an animated pedagogical agent). (p.18). 

Critique:

I appreciated table 2, which included practical guidelines for increasing learner motivation based on the study results. These are things I can actually apply to my Web-based courses.

In addition, I found the notion of disruptive innovations very interesting. I think we’re at an exciting point for Web-based instruction where quality is improving and it’s being distributed more widely. Still, the following statistic blew me away: "50% of high school courses will be offered online by 2019" (p.2). That's in eight years from now. 50% seems high to me. What do others think?

Kim, K.-J. & Frick, T. W. (2011). "Changes in Student Motivation During Online Learning." Journal of Educational Computing Research, 44(1), 1-24.