Assessment in Education 2.0

A better title for this chapter might be Technology and Testing. It's about how we measure the performance of our schools, the skills we need for the 21st century, and the role of technology in all of the above. It begins with a school district which fired its technology integration teacher and hired a remedial reading teacher in her place. "If it's not on the State Mastery Test, we don't consider it important," was the explanation. Technology is not on the test; neither for that matter is history or literature. Or science or citizenship. Or music, or the arts.

And yet this same school district spouts on its web site that it's "Leading the way toward 21st-century skills." Seems like a contradiction to me. So let's take a look at those 21st-century skills, and at the state mastery test, and see how well they match. And then think about the role of technology in moving students forward.

21st-century skills

Probably the most useful list of 21st-century skills is the one published by the Partnership for 21st-Century Skills, a consortium of business and education groups that includes Apple, Cisco, Ford, the NEA, Pearson, and ASCD, among many others. Here is what they say is important for every student to learn:

Many of these skills were also important in the 20th, and maybe even the 19th centuries -- this is really a list of what you ought to learn in school these days. Now, let's use this list as a scorecard to analyze the items on the state mastery test.

Test Items

We'll start with the 8th-grade math test from a northeastern state with a large and diverse population of city-dwellers, rural folk, and suburbanites. Here's the first item:

Which of the 21st-century skills does this measure? You might say it relates to several of them: reason effectively; solve problems; mastery of mathematics. But most of us would agree that the chief purpose of this question is to measure the students mastery of math. So put a check next to that skill.

Now let's look at the next item on this test:

This one seems to require some relational thinking -- more than just remembering a formula. You need to know some algebra, and be able to relate algebra to a real-life problem. But talk about a trick question! The answers are designed to confuse you, all containing the right values but with a slightly different syntax. So maybe it's measuring reading ability more than math. Or it's measuring test-taking ability. Do you ever confront questions like this outside of multiple-choice tests? Which (if any) of the 21st-century skills does this question measure? Is it a good measure of that skill?

On to question 3:

I'll bet you are trying to figure out if this is a trick question. Answer B, 72 degrees, seems to easy for it to be true. Even if you didn't know a thing about geometry, you might get this one right -- none of the other answers seem even close. So which skill from the list above is this measuring? It seems to be measuring my test-taking skill, but that's not on the list. Math? Critical thinking?

For some variety, let's look at the reading test. First item:

Seems familiar, no? We all grew up with these kinds of fill-in-the-bubble, multiple choice test, and number 2 pencils. The style and norms for these tests were developed in the 40's and 50's, and have been used ever since, over and over again. Which of our 21st-century skills is this question measuring? Could we have used this same question in 1910? 1810?

Which century?

What's your response so far as to whether or not these tests measure 21st-century skills? From what you've seen so far, would passing this test indicate a student's readiness to survive and thrive in the 21st century?

Do modern information technologies play any role in doing better on these tests? Do the questions take into account in any way the advances in digital information that have taken place in the last 50 years?

If you continue through these tests, checking off the boxes in the table above you'll find that 90% of your marks are in two of the 38 boxes: mastery of mathematics, and mastery of English language arts. With a few ticks in critical thinking or problem solving. But no ticks for the other 34 skills. Try it your self; these tests can be found online.

The results of these tests determine whether a student moves to the next grade; whether the school receives federal and state funds; and soon, whether its teachers are considered effective or defective. Because of the high stakes wagered on these numbers, educators have no choice but to take them seriously. And to spend less time developing the skills that the tests don't measure.

Measuring the other 36 skills

We are pretty good at measuring a certain subset of math and reading skills -- those that are easy to measure with bubble-tests. And we spend inordinate amounts of time and money teaching and testing them. But they account for less than 5% of the things students ought to learn in school.

If we wanted to evaluate a restaurant, we would send a sensitive gourmet to eat a full meal and write up the entire experience; if we wanted to be more thorough, we would send a second critic with different tastes. The critics would describe the atmosphere, the smells, the presentation of the food, its taste, texture, and color; the service, the harmonization of the wines; the quality of the silverware, napkins, salt and pepper. Readers would have a good holistic sense of the quality of the place.

If we applied the current educational assessment logic to this problem, we wouldn't do it this way. Instead, we'd look for a subset of the whole that's easy and quick to measure, such as the quality of the salt. We'd develop a rubric to describe the various degrees of salt quality, and perhaps even develop a special magnifying glass so we can see the individual grains, and a picture-card to match the grains to a 5-point standard range. We'd send a minimum-wage inspector out to the restaurant, and get results for dozens of of them each day. Very inexpensive, yet reliable, accurate results, that no one could argue with. Significant to the .05 level if you want.

My grandmother used to tell me, concerning my grandfather, who was wont to exaggerate, "Take everything he says with a grain of salt." When I was very young, I wondered what she meant -- should I grab the salt shaker whenever he started telling one of his stories? It took me a while, but I figured out her figure of speech. (My grandmother didn't now it, but this phrase has its origins in the Latin cum grano salis.)

A 21st-century feast

There's more to this meal than meets the eye. If we allow a few grains of salt to determine the quality, we'll soon be disappointed in our eating. Restaurants will learn to game the system, focusing on the condiments at the expense of the main dish; getting good reviews but serving mediocre food.

Let's stop looking only at the salt, and consider the entire panoply of skills as we assess our students and our schools. We need to pay more attention to the other 36 essential skills that aren't on the tests. And teach them seriously. Are be held accountable for our students learning them. We won't move from Education 2.0 to 3.0 by peering at the salt.

So next time someone touts their "high-performing school," take their claim with a grain of salt.

If we moved beyond the salt in our assessment, how would we do it? And if we spent more time with the other 36 skills, what would be the role of digital technology in teaching them and in assessing them? Those are questions we'll explore in the next chapter.