3.2 - How We Measure

Hi. We're going to continue talking about teaching like it's 1992. And I want to talk a little bit about measurement and assessment. As we've seen already, 1914 is when Frederick Kelly invented the multiple choice test. And by 1926, this test that he had intended to be for lower order thinking, and a way of getting students massively though the educational system, was adopted by the Scholastic Aptitude Organization as the way of testing who would or would not get into college. That's very important, because it also signals different ways that higher education in the United States, but also internationally, becomes prepared for as early as-- now, in the United States, it's even-- preschool. I heard from Duke admissions officers that they've heard from parents of preschool kids, wanting to know what's the best preschool for them to go to in order for their child to get into Duke eventually. There's different systems around the world, and I hope that we can use the forums in this Coursera course, so that people from different countries can talk about the educational funnel and filter system that leads from early child education all the way to higher education, and professional school. In the U.S., there's more of an emphasis on liberal education. In many other countries, the system starts funneling by subject matter; by aptitude; by ability much, much earlier. At the same time, in the inverse advantage, the United States has really eroded its vocational training systems, and all education is implicitly college education. I think that actually, personally, I believe that's one of the reasons for our very high dropout rate: that we don't offer alternative versions of education to our people. At the same time, I'm not so crazy about a filtering system that makes you choose your field or your discipline or your aptitude very, very early on. What we know more-- and this is, again, how to teach like internet exists-- what we've come to see more and more is that in fact people are having to merge aptitudes across the qualitative and the quantitative; across the interpretive and the moot, or mechanical ways of learning. That said, not only in the U.S. but throughout the world, how we measure is more and more reduced to productive, summative testing; standardized testing on on multiple choice tests. Interestingly, the United States tests earlier and more often, and uses standardized testing at an earlier age and more often than any other country in the world-- even though we don't do very well on standardized tests. One reason for that is the increasing wage gap and income disparity between our richest and poorest citizens. In fact, in the United States, if you take out income as a metric, our test scores are extremely high and competitive with scores around the world. It's only when you look at the aggregate of the poorest students in the poorest school systems and the richest students at the fanciest private schools where they get lots of individual attention, that you have these disparate test scores. There's some very interesting things about standardized tests that it's important to think about. And two words summarize one of the most interesting things that happened, and that's Finnish lessons. In the early 2000s, the O.E.C.D. invited the people of Finland to have their students take standardized tests. This was interesting and important, because the Finns had abolished standardized testing 20 years before. So the Department of Education said, "Sure, we're happy to give our students the same test that the O.E.C.D. uses the test students around the world to find out what they know in different fields like mathematics; and writing; and history; and the basic three R's-- reading, writing, and arithmetic." What was shocking to the Finns-- and to everybody-- was these kids in Finland who had never seen a multiple-choice test before, ended up scoring in the top five in just about every field. When the test scores came back, at first, the Department of Education in Finland though there had to be something wrong; they simply couldn't believe that kids who had never even seen the form of a test would score in the top five internationally on those standardized tests. Because, as we know, throughout the world, in other countries-- and that's true in Asia, it's true in Europe, it's certainly true in the United States-- affluent kids take, it's called juku in Japan. When I lived in Japan, all my students were in had gone to juku, which is after-school cram school, which is a kind of testing you do specifically to prepare you for the test; it's what we call teaching to the test. We know from the Kaplan industry and other kinds of proctoring systems, that teaching to the test works. You can teach students, not how to get the knowledge and the content, but actually how to actually psych out the form of the test so that you do better on the test: when to guess, when not to guess, how to look at the form of the grammar of the question and figure out the right answer. I have a friend who insists he got a passing score on Greek for graduate school simply by learning how the Greek tests work, without learning a word of ancient Greek at all. Clearly, that's the "reductio ad absurdum": the reudction to absurdity of any kind of knowledge. If what you're doing is having a test that you can fake; or scam; or beat simply by knowing the method of the test, what you're then doing is teaching-- not knowledge-- but teaching how to take a test that you don't need in any other place in your life. In fact, we know that multiple choice tests only cover about 20 percent of the content of the actual grade they measure; and that leaves out the whole thing of the range of different abilities. The simplicity of thinking that you're really learning something when you're choosing one item out of 4 or 5 possible choices, it's often called "one best answer" or "item response testing". So, back to Finland. The Finns, it took a while for them to absorb the fact that, "My gosh, we did so well on tests that our students haven't studied for." And they found it kind of amusing. And certainly, they were proud and they were gratified. So now, we're all-- the whole world of educators and those that are sort of passionate about education-- are really looking to figure out what lessons we can learn from the Finnish. Pasi Sahlberg, who was one of the people who was behind this idea of letting the students take these tests, has written a book called Finnish Lessons-- and the full bibliography will be on our course website-- which details some of the principles of Finnish education. Here they are: 1) Everybody from first grade teachers and beyond who teaches in Finland has to have at least a Master's degree. 2) One day a week, teachers in Finland are given a chance to work with other teachers to come up with the best lesson plans they possibly can for their students. So professionalism of the teaching staff-- professional exchange, professional development, taking professional charge of the creative ways you teach-- are all built into the Finnish system. 3) A third very important thing is nobody fails relative to anybody else in failing. The bell curve is just gone from the Finnish education system. The basic tenant of Finnish education-- and this is hard for us to grasp and if we're used to grading on a curve-- the basic tenant of the Finnish system is that professional teachers will get together to decide what constitutes the best platform, the best plateau of knowledge for that grade and that subject. And then they will work together to have every student attain that plateau. Not who gets there first-- that isn't the highest grade-- but how to get everybody there. In fact, those students who attain that level of knowledge first then have the privilege of teaching other students what they've learned. Needless to say, that is a method we're trying to do in this Coursera course, too. And we're trying to develop assignments that are based on that idea of "See one, do one, teach one, share one": the idea that if you learn something and you get an idea, you should talk about it with somebody else. Tell them that idea, get their feedback, and that will help you assimilate that idea into your own practices; into your own behaviors; into your own conceptual framework; into your own paradigm-- far better than just hearing it coming from me or from anybody else, and certainly better than standardized testing. The difference between formative and summative testing: in formative testing, there's real-time feedback. That's what Finnish lessons is about: how you give real time feedback so everybody can succeed. What summative testing is is how you come up with a metric at the end that simply allows you to standardize the grading; that's 1) to make for machine readable grading; that's one part of it. And 2) how to compare how one individual does with another within a single institution; and from institution to institution; country to country to compare grades. Interestingly, I have had a fascinating other sub-field of going to Asian countries to talk about Finnish education. That's pretty funny: to be an American going to Asia to talk about Finnish education. But I've been a very ardent critic of summative testing. Asian cultures tend to be in the top ten with their summative tests. And I've been invited to Japan, Hong Kong, I worked with the Singapore Department of Education, an Thailand to talk about what would be a better way. The reason is those countries do brilliantly on summative tests. But: 1) educators feel it's a disincentive to learning, and a lot of students just turn off to education entirely because of the tests. 2) there's so much stress with the tests that everybody's unhappy. Parents, teachers, students are all unhappy about having to study for the test. 3) Nobody is sure that doing well on tests under that extreme pressure of test-taking actually prepares you for a responsible, creative, innovative adult life. In fact, I learned from visitors from Singapore last year, from both business and the Department of Education-- the Ministry of Education there-- that, although the Singaporeans do brilliantly on tests, they have a very low level of productivity. So what does that mean? It means suddenly these people who are doing brilliantly on tests in the countries I've mentioned in Asia are looking at Finland and saying well they're doing even better than we are, or at least as well as we are on these tests, and Finnish teachers; parents; and students love their education system. So how can we get the same excellent test results; the excellent learning objectives; the excellent concrete, innovative, creative learning that we have in Finland without putting ourselves; our students; our parents; our teachers through this awful grind of summative testing? That's the challenge, and that's the challenge that that we're all facing right now. We'll be talking about this in the very last part of the class, when we talk about values, but basically the ideal here is that we should think about what we value, and make sure that what we value is what we count. The way we test has to be geared to our value system. In this particular course, what you'll be finding is that we actually are going to be doing research on 4 different kinds of educational testing tools that we're going to be using in this class. As I've said, there's always somebody behind the camera in our educational systems. That's the metaphor for this class. It's not just me here in this makeshift studio teaching, but there's Casey behind the camera, lots of other people that are helping us. We also have a team of TA's-- teaching assistants and research assistants, who are going to be developing a range of different metrics and methods and studying to try to figure out what is the best way that we all can learn online? The 4 methods we're doing are: 1) The method I've been talking about: multiple choice, machine-graded testing that's basically not much different than Frederick Kelly developed in 1914. It's a version of what you see on Scholastic Aptitude Tests, SAT tests, GREs, Graduate Record Exams, and other kinds of testing. 2) We're going, we're using a tool developed by a doctoral student at North Carolina State University named Lakshmi Ramachandran. She calls it the Automated Metareviewing System. And it will be looking at the narrative parts of the test-- not the multiple choice, but the narrative parts of the assignments that you're doing in this class-- to see how her automated metareviewing tool can actually help to automate feedback, and give students the most complex, but also machine-read feedback. We'll be humans looking at how the machine is grading humans writing essays in an online course. I love that combination. It's very post-1992. It's not teaching like it's 1992; it's thinking like it's 2014 to be thinking about how the machine and the human can work together. 3) We'll be doing peer-to-peer editing in this class. You're already doing this in the forums and editing each other. 4) We'll be using peer badging systems, where you can give credit to students. Not failure-- wwe're not having anybody grade each other for failure-- but praising students who do a great job on the different kinds of forums in this class. And what we'll be doing after all of that is over, after the course is over, is we'll be working with a team of researchers to see what worked best. So maybe the next time we do this class, and certainly on blogs and follow-ups that you'll have access to in this class, we'll be able to report on the different ways that these systems work together. But none of that is really teaching like it's 2014. The real way to teaching like it's 2014 is not only peer learning, but peer teaching. It's one of the things we've talked about in our rights and principles for online learning: that everyone has a right to be a teacher. By that, we mean we really would love you to take the ideas; the concepts; the information from this class. Talk to people. Talk to your parents. Talk to your kids. Talk to teachers. Talk to others in your society. Report back, come to the forums, and tell us what your own ideas are. This is a process we're in together. Learning never stops, learning is real time, learning is communal. Learning isn't just about how you do well on the test. It's on the biggest test of all, which is how we can be productive, engaged citizens of the world. Thank you, see you next time. Bye.