How to Evaluate a Multimedia Project

by Prof. Jim Lengel, College of Communication, Boston University

A Tale of Two Teachers

It was the worst of times...

Molly's sixth grade teacher assigned the class a research project: Greek and Roman Art. This was in the early days of computing in schools, and Molly was among the few whose home computer contained all the latest multimedia programs. She quickly found on the British Museum CD-ROM enough pictures of classical sculpture to fill several slide shows, along with explanatory text. She was good at copying and pasting. It didn't take her long to produce a slide show with dramatic views of statues and facades, which she delivered to the class on a big projection screen in the computer lab. Nobody, including her teacher, had seen anything like this multimedia report. By a 12-year old, no less. Her peers, who worked much harder (and probably learned much more) were relegated to standing in front of the class and reading from their note cards. Molly got an A, no questions asked.

It was the best of times...

The assignment: Choose a battle from the American Revolution, research it, and present it. Next week, turn in a complete outline of your research plan. Week after, bring in at least five original sources.Week three, make your presentation to the class. Week four, respond in writing to fellow students' questions on your battle. Molly, as usual, found quickly on the internet enough original sources to overwhelm even the rustiest Redcoat. But not her teacher. What does this cartoon of the Boston Massacre, he asked, have to do with the Battle of Lexington and Concord? Molly went back to the drawing board. In the end, her project had more multimedia than the others - including a live rendition of an historical fife tune -- but earned only a B, not because it wasn't the flashiest in the class, but because not all the examples related directly to the battle she had chosen.

If it's multimedia, it must be good

In the early days of computers is school, it was enough to assemble a series of seemingly related images and sounds, with funky fonts and dithering dissolves, to make a report that impressed the teacher. But now we have learned to see beyond the pretty pixels and can apply the same rigorous rubrics to multimedia projects as we do to their traditional precursors. What do those rubrics look like? This article proposes a method for grading a multimedia project that focuses on content as well as creativity, on ideas as well as images. The rubric consists of four areas for evaluation: the ideas behind the project; the extent of the evidence for each idea; the quality of the presentation of each piece of evidence; the logic of the narrative; and the student's ability to defend the ideas and evidence through questioning.

Ideas (30 points)

The first step in grading a multimedia project is to identify the ideas that the project presents. Not the pictures, not the words, but the ideas that are behind them. In a good report, an idea or three will come through clearly. For the revolutionary war project mentioned above, the ideas might be

• the British were outnumbered at Concord.
• the Americans used guerilla tactics in the battle.
• the British lost more men on the march back to Boston than on the battlefield.

If you can't easily identify the main ideas that the project is getting at, it probably does not deserve a good grade. Make a note of the main ideas of the project. Then evaluate these ideas in terms of

• how many ideas are presented.(10 possible points)
• how well they match the assigned curriculum.(10 points)
• how deeply they delve into the subject.(10 points)

A grading sheet may help you here. In a simple table, jot down the main ideas in the first column, and in the second award the points. At the end of this article, you will find a sample grading sheet for your information. Because the strength of the ideas is the most important aspect of a multimedia project, ideas are worth 30 points on our 100-point scale.

Evidence (20 points)

For each idea, the project should present evidence that supports or explains the idea. Evidence can come in may forms, from images to text to voice to music to diagrams, and each idea must be accompanied by adequate evidence. Points should be awarded on the strength of the evidence at this point:

• How well does the evidence support the particular idea? (5 points)
• How much evidence is provided for this idea? (5 points)
• How interesting is this evidence? (5 points)
• How much of it is in the form of original or unexpected source material? (5 points)

You are looking here not for how well the evidence is presented - that comes in the next section. Here you are evaluating the quality and extent of the evidence presented for each main idea.

Presentation (20 points)

For each piece of evidence, you should evaluate how well it is presented, judging its format, its technical quality, and its visibility (or audibility) to the audience. Here you can also evaluate the design of the project in terms of its readability and consistency. A good way to evaluate is to examine the work with these questions in mind:

• Format: How well does the format of the evidence serve to support the main idea? Is it the best medium to get the message across? Might another format have worked better? (5 points)
• Technical Quality: How well did the evidence survive the process of digitization and presentation? Was its original quality preserved? Is it free of rough edges and unclear aspects? Are the words spelled correctly? (5 points)
• Readability: How well did the evidence come across to the target audience? Was it the right size or the right volume? Did the audience immediately perceive its meaning? (5 points)
• Design: Did the color scheme, font, and placement of objects facilitate the audience's understanding of the ideas? Did the evidence and the idea maintain center stage in spite of the design and decoration? (5 points)

You are not evaluating here how flashy the project appears, but how well the evidence is presented so as to achieve its objectives. In fact, unnecessary decorative elements that distract the audience from the main ideas should be discounted as you compute this score.

Logic (20 points)

A good presentation is more than a collection of ideas and evidence .A good presentation tells a story through a series of related facts that build upon each other, and takes the audience through a thought process that increases their understanding. Sometimes this logic is embedded into the presentation itself, but often it's provided by the presenter orally as the project is shown to the audience. Sometimes referred to as the narrative or flow of the work, this logic serves as the manifestation of the student's ability to think through the ideas and to understand their meaning in a larger context. Some questions to help evaluate this aspect of the project might be:

• How well does the presentation place the issue in a larger context? (5 points)
• How strong is the link or transition form one piece of evidence to the next? (5 points)
• How well are the individual pieces of evidence tied to the main idea? (5 points)
• How well did the audience follow the flow of the argument? (5 points)

This is a bit harder to evaluate than the other aspects of the project - there's more subjective judgment involved. But it's important, and often distinguishes the thoughtful multimedia project from the slapdash collection of pictures.

Defense (10 points)

A student may assemble a great collection of evidence around a set of key ideas, present them well, and tell a nice story, but not understand the topic in any depth. Sometimes the only way to find out if they truly understand their ideas is to ask them questions and challenge their pronouncements. Often both the teacher and fellow students participate in this process - after the presentation, they ask questions that test the student's knowledge of the subject. Some points to consider here include:

• Can they explain the relationship of each piece of evidence to the main idea? (5 points)
• Can they defend their conclusions in the face of contradictory evidence? (5 points)

If they can do this well, and use the items in their presentation to explain their answers, the chances are good that they have a deep understanding of the subject.

Multimedia Grading Table

Many teachers develop a system for evaluating multimedia projects that centers on a grade sheet on which she records her evaluation. A sample sheet that follows the method describe above might look like this:

This project scored 76 on a 100-point scale.

Communicating with Students

All of these standards and rubrics serve little purpose unless the students are aware of them. Explain in advance to your students how you will evaluate their multimedia projects. Develop a list of questions that you will use as you evaluate their projects - feel free to copy those listed above - and distribute it to students. Encourage them to apply these questions to their own projects before they turn them in. Provide a time for them to apply the rubric to each others' works. In this way, you can raise the expectations for the work they do on their computers.