Effective Classroom Data Use with Dr. Matthew Courtney
Teachers are drowning in data and much of this data has little value to inform and improve instruction. Scott and Matthew Courtney discuss how to solve this problem- and do so simply and without wasting valuable time.
Matthew Courtney is author of the book Exploratory Data Analysis in the Classroom and he shares how to effectively choose useful data (that teachers are probably already collecting anyway) and manipulate that data to help make real, meaningful instructional decisions. And this can be done without buying any software or taking a class in multivariate statistics: these are accessible processes that any teacher and any team can use and immediately begin developing a culture of instruction that is richly data informed.
Episode topics
Why useful data is important-and it usually is not state test data that is useful
How to keep it simple
Creating a culture for data use in the classroom and school
What policymakers should know (but don’t always get right)
Links
Matthew’s book Exploratory Data Analysis in the Classroom
You may also follow Matthew on facebook, LinkedIn and Twitter @mbcourtneyedd
Transcript
Scott Lee 0:03
Greetings friends and colleagues. Welcome to The Thoughtful Teacher Podcast, the professional educator's thought partner, a service of Oncourse Education Solutions. I am Scott Lee. During several episodes this year, I have commented on problems with testing and assessment. But I have not offered very much in the way of practical help for classroom teachers. In this episode, we change all that. My guest today is Dr. Matthew Courtney, author of the book Exploratory Data Analysis in the Classroom. And we will discuss just that how teachers can meaningfully use classroom data. In addition to his work helping teachers improve data use, Matthew is a policy advisor at the Kentucky Department of Education. He has also served as the executive director of the Bluegrass Center for Teacher Quality, and he has been a classroom music teacher. He has earned several honors, including a Strategic Excellence and Achievement Award from the Kentucky Education Commissioner. He is also a past fellow of Lead, Kentucky, had a fellowship on the Future of the Teaching Trofession, and has been a member of the National Cadre on LGBTQ Safety in Schools. We begin our discussion, talking about his book, and how teachers can effectively use data in the classroom. Thank you so much, and welcome Matthew to the Thoughtful Teacher Podcast.
Matthew Courtney 1:43
Hi, Scott. Thanks for having me today.
Scott Lee 1:45
I want to start right away talking about your book. When I'm in schools, sometimes teachers tell me that they're kind of already drowning in data and have enough data. So why write a book on classroom data analysis in the first place?
Matthew Courtney 2:02
Yeah Scott, I've had that same experience time after time. Teachers certainly are drowning in data. The one thing they're not drowning in is time. And I have spent several years teaching these exploratory data analysis techniques to educators all over the country. And it's really a great technique for processing a large volume of data really quickly. And for mining out useful insights from your data in a replicable, repeatable process without having to learn a new software and add a new visualization tool and a package for this test or a package for this curriculum. It's basic skills that we can apply over and over and over again,
Scott Lee 2:47
Since you mentioned not buying a piece of software, I just want to want to focus one quick thing in your book, basically, everything you teach is set up on a spreadsheet. So if you're using Microsoft Excel, or Numbers, if you're in an Apple environment, or even some of the free- freeware options, it all works and it'll be the same right?
Matthew Courtney 3:11
Yeah, that's exactly right. I took extra care to make sure that every process that I teach in the book works across all of those platforms. So like you said, numbers, Excel, Google Sheets, or any of the other free wares, these techniques should should be replicable no matter what kind of package you're already using. Okay.
Scott Lee 3:29
So can you share a little bit with us about useful data that teachers often do not use very well?
Matthew Courtney 3:38
I think that's a great question. I think, within the field of education, we have over the last decade or so really developed a narrow view of what data is and what kinds of data are valuable. And usually, when I talk with teachers about data, their immediate response is to start talking about standardized testing. And the data pulled from standardized testing. But really, that teacher created data, those classroom level data, data points that teachers are collecting organically all day every day are incredibly rich data sources, that teachers don't often think about is data, because we've sort of conditioned educators at all levels of the profession to focus on those standardized data points in our data meetings and data conversation. So a big part of my work is helping teachers to just expand their mind about what data is and what kind of data they have and how they could use it.
Scott Lee 4:31
I mean, even looking to see if a student appears to be on task or paying attention can be useful, right?
Matthew Courtney 4:33
Oh, yeah, for sure. Another one that I like to talk about is phone numbers. We all have a little Rolodex or a spreadsheet or whatever, where we keep track of all our students phone numbers, and we know students who have phone number changes 2, 3, 4, 5 times during the school year. That's a data point five phone number changes during the school year is a great data point to point towards social instability, financial instability in the home, and that is going to tell you something about that kid that you might not have known before.
Scott Lee 5:07
Yeah, that's, that's interesting. I would never have thought about that. I'm going to use that one. If you don't mind. Please do you specifically say in your book, exploratory data analysis, and I'm quoting you "lacks statistical rigor." So why is it useful?
Matthew Courtney 5:25
This is one of the first pieces of pushback that I often get when I go in and start working with schools or educators in a variety of settings. They always say, This is too easy. These techniques are too low level. And I think most educators, during their preparation have had some kind, of course, in statistics, for most of us has probably called something like educational tests and measurements or something like that. So like a statistics course, sort of hidden behind a curriculum or a testing framework. And the thing that I think gets lost in translation a lot of times is that these mathematical processes exist on a spectrum, from sort of low level things. Many of the techniques that I teach are like fourth and fifth grade level standards and a lot of states, things like mean, median, and mode, we learn those and then we think, Oh, well, that's not real statistics, we need to be looking at t tests. And they know of us and those sorts of more informally called inferential statistics, statistics to help us look at causality. But for our classroom teacher who's looking at the students in front of them and the data in front of them, those high rigor tests are really not appropriate for guiding they're in the moment day to day decision making. And these low level tests, if we can learn to use them and harness them in a really meaningful way. They can help us to spot important anomalies that we didn't know where their kids who we were missing, because maybe we weren't looking for them, and then respond to those anomalies in real time.
Scott Lee 7:00
And I'm sure this happened to you, too. And a dissertation, my dissertation, I had to use multivariate statistics, but I still had to report mean, median, and mode as well, that was still part of the part of the data set that you include. So
Matthew Courtney 7:14
Yeah, exactly. And they all build off of each other. So I like to think of exploratory data analysis as the first foundational step and longer term school and classroom improvement, we can find those anomalies, and then we can turn them into Plan, Do Study Act cycles, we can feed them into action research cycles. And so we can continue to build the rigor over time if that's what we need to do.
Scott Lee 7:35
And you don't always need to do that even
Matthew Courtney 7:37
Exactly.
Scott Lee 7:38
Do you find any other pushback from teachers and doing this? And I'm, I'm thinking that if I'm a teacher, and you're coming in to do a presentation on data analysis, the first thing that may pop into my head is I don't have time for this, or just somebody else from the State Department coming in to make me do something else. So and maybe you get different kinds of pushback, I don't know, what kind of pushback do you sometimes get when you want to start talking about data? And and what is what's the workaround? What's the mindset change that you really want a teacher to have when you get that pushback?
Matthew Courtney 8:21
Yeah, so a few things to that, I certainly do get a lot of those same kinds of push backs that you mentioned, that I think really come from a toxic professional learning culture that, that we've really built, where professional learning is something that is done to you, not with you. And I think you know that that happens to us at all levels of the system, even our superintendents and board members go to professional development that is done to them, not with them. And so that's, you know, as a trainer, you have to kind of work through that in its own right. But when I think about the data analysis technique, I often hear, I just don't have time for this. I think that that is a really common feeling among our educators, because they are so overworked and overburdened. The beauty of this technique is that once you learn it, once you learn these 10 or 12 essential skills, you can deploy them over and over and over and over again. And on any kind of data. If you can get it in a spreadsheet, you can replicate this process. And so what I find when I work with schools and districts is that, you know, they have this software that goes with this test, and it produces this kind of report and another software with another test and a different kind of report. And they spent a lot of time learning how to use those different platforms and different software's, but what we're teaching here are fundamental skills. So all I have to really learn is how to pull a spreadsheet out of each of those different platforms, and then I can replicate this process. And so it's a lot less work in the long run. The other thing I do, frankly, as a consultant that I think more consultants should do is I refuse to do after school PD that isn't paid for it. We do too much to teachers and we ask them to do too much for free. If this work is important enough for you to hire me to come talk to your school or your district or whatever, it's important enough for you to provide protected time and pay time for your people to learn it.
Scott Lee 10:16
And I want to want to say this about the book as well. When I finished reading it, I turned to my wife, who's a retired school psychologist, and I showed it to her. And of course, nobody can see this. But if I was holding it up, look at how thin this book is, you know, 125 pages on useful statistics and useful data analysis. It's not like any other book I've ever seen. You know, usually it's, yeah,
Matthew Courtney 10:47
500 pages.
Scott Lee 10:49
Yeah. And so, you know, I want to point that out to people who are listening and can't actually see what we're talking about. And point out that, that this is a book that is easy to read. And the exercises are easy to do. But it's also it's a book, you can read in a quick amount of time and start using it right away. And everything in it is useful. So
Matthew Courtney 11:14
Yeah, thanks for that observation, too. I mean, I'm looking at my bookshelf here. And I probably have 20,000 pages worth of books that are deep dives into how I use data. But now I don't actually tell you how to do anything. And this book breaks it down step by step, you click in the cell, you type this formula, you highlight this column. And so it really is designed to strip all the fluff out in the introduction of the book, I say my goal was to not create another theoretical tome about data analysis. But to create a really a guide book, here's how you actually do the thing.
Scott Lee 11:49
If you go into a classroom, how can you tell? Or what are some of the clues that a teacher is using good data or using good data analysis to make instructional decisions? What are you going to see,
Matthew Courtney 12:04
One of my favorite things is when I have the opportunity to get to a school a little early and do just that, walk through the school get to meet the teachers get to see the kids in action. And one of the things that I I like to do is ask teachers, why are you doing this? Why are you doing this activity? Or why is that student on a computer and this students got a book out? And what you can find really quick, it's when you when teachers are really engaged in data analysis, they're really engaged in research, use that application, which is another thing I train and talk about a lot. They can always tell you why everything in their classroom is happening for a reason. And they've really dug deep to think about, what does this data tell me about each kid in my classroom? Or about third period versus fourth period? And how can I teach those groups of kids differently in order to meet them where they're at and get them where I need to be?
Scott Lee 12:58
In the book. Also, you offer several vignettes that illustrate data use in real and real life situations. I'd like for you to just tell us about why you did that. And then also want to talk about about how they're structured in a second.
Matthew Courtney 13:16
Sure. So I thought it was really important to really show how easy the work can be. Because data analysis often feels very hard. I work with a lot of what I call data hesitant educators who are afraid they're gonna get in and mess the spreadsheet up. Well, that's a simple trick, do a "Save As" always have a backup copy, then you can't mess anything up. And I wanted to really show the the thinking. So as I wrote those vignettes, I really tried to narrate as what would be happening in my mind as I'm doing this activity. The other great thing about them is you can go to my website and download the data set that the teachers in the vignette are working on. And you can actually follow along. And so you can, you will read about some skills, you will read a vignette where a teacher applies those skills. And then you can take that same data set, apply those skills and check your work against the work of the teacher in the vignette. And I thought that was just a really powerful way to help really send home the learning behind the book. So you're not just reading it, you can actually do it follow along, you can step into another teacher's classroom.
Scott Lee 14:25
Sometimes because of the way some of us at least were taught math, we forget that learning how to do the math is not really what this is about. I want to show you the answers. It sounds like you're saying, here's here's the exact process. Ya know the question is how do I interpret the data? And once once I get the answer, or once I get my solution. The other thing that I noticed about the vignette is in almost all of them you have some sort of conversation, or there's more than one person in the vignette and They're having collegial conversations about how to use the data and that sort of thing. Why did you do that? Although I think I know the answer.
Matthew Courtney 15:09
I just think it's so important that we view ourselves as professionals in the context of a profession where we will work together and collaborate together. It is so often that we, as educators close the door, we're in our four walls with 30 kids in front of us, and we get so consumed in that space. But through this data analysis process, we can really come together as a team and really learn together. There's an interesting vignette, I was a music teacher, so I had to include a music teacher and one of the vignettes. And so there's a vignette where a social studies teacher and a music teacher come together, they look at their data together, then they co design a unit, and they do an action research project together to see if their unit is leading to greater success for their kids. And that's such a great collegial example of how we're all in this together, across grade spans across content areas. Another thing I love to do, when I'm working with a school or a district, it's I love to do this exercise where we swap data. So Scott, you're teaching your class, I'm teaching my class, we have our spreadsheets, we swap, you analyze my spreadsheet, I analyze yours. And then when we do that, we can have a really deep professional conversation about the kids in our classrooms. And I'm gonna see things that you're gonna miss on your kids just like we swap when we need to, you know, I wrote this newsletter, can you proofread it? Right? That's a common activity. Why don't we swap data and we can proofread and have those exchanges there, too.
Scott Lee 16:44
Part of the reason I wanted to make sure to bring that out when I read that in the book, and I love hearing, hearing you say what you said, because I've been advocating in a lot of episodes of this podcast about the problems that we face as a profession, written in regards to isolation. And this should not be teaching should not be isolating way too often in practice it is. So I was really happy to see that want to change gears a little bit and kind of talk about more of the data that everybody is associated with everybody knows about a lot of teachers are kind of worried about. And you mentioned, again, using your words, the sordid history of the standardized testing industry. What's good about the standardized testing industry, what's not so good about it? How is whatever we're getting from them. And by we, I mean educators getting from them useful or not useful? Now, that's a big, big thing.
Matthew Courtney 17:53
[both laughing] But yeah, how long is this podcast?
Scott Lee 17:54
We could we could do an entire season on that. There's a lot there.
Matthew Courtney 17:59
Yeah, well, I'll try to be brief on something I'm extremely passionate about. Yeah. So I do say in the book that standardized testing has kind of a sordid history, and there certainly are legitimate criticisms of standardized tests and standardized testing, both as a concept and as a industry, I don't get into the history is very deeply in the blood really just want to acknowledge that it's there. Because we feel that and teachers certainly feel that I am someone who believes that standardized testing can be useful. And it's important when it is done correctly, and in a healthy atmosphere with a healthy approach to data. I'm also someone who believes that for the most part, we do not have a healthy atmosphere and a healthy approach to standardized testing data. So my concern with this with this work is less about the standardized testing, and more about standardized testing policy procedures and beliefs. One one really common frustration that I hear from teachers is that they're being asked to use standardized tests wrong. We use them for the wrong in the wrong ways. So think about, if we think about testing for just a moment, tests are created for a reason. They're created with an end goal in mind and a use that is pre designed and built into the test. And if you're using the test incorrectly, you're getting bad information, you're wasting a lot of time. So think about, you know, federally mandated annual standardized testing, every state does that because the Every Student Succeeds Act requires some standardized testing. I often work with teachers who get those those we get those tests in April or May teachers get those results in September or October, and then they're brought into the library or the cafeteria. bring their laptops and then someone says okay, let's use this to play at our instruction. Well, that's ridiculous. You should not be using that pestis. Plan your instruction because that's is not what that test is meant to do. That test isn't for teachers or kids. That test is for system level leaders, state level leaders, federal leaders, researchers, politicians, advocacy groups, that test is a snapshot of what's happening designed to inform all of that other work that goes around the school system. Some standardized testing is great for teachers, I'm a big fan of sort of those quarterly benchmark assessments, or some schools doing three times a year, where we say, you know, let's, let's take a glimpse at where all of our kids are in a standardized way. So we're able to step back and remove some of that teacher bias that we have in the moment, we can remove some of the relationship oriented bias where you know, we've all done it, where we interpret something a different way, because we just really love that kid, or we know their situation. And so we give them a different kind of opportunity. That's totally appropriate. I'm not saying we shouldn't do that. But occasionally, it's good to just put all the kids on the same level, use the same standardized test and take a look at them. That is really useful for schools and teachers when it's done. Right. I also think we have a really unhealthy culture around data use, we have used standardized testing data to punish teachers to punish kids to embarrass families and communities. That's not healthy. That's not productive, and it hasn't proven to be productive yet. I said I'd be brief. I don't know if I
Scott Lee 21:23
wait, no, no, this is fine. I know. You're great. Keep going. Yeah,
Matthew Courtney 21:26
I mean, we can talk we could, like you said a whole season on this one. But yeah, I really am here to advocate for a healthy, beneficial testing use culture. And one of the things I always say this will be the last thing I say about it, I always tell teachers work with a data that works for you. And so if you're working with a set of data, that just doesn't jive with you, it doesn't feel genuine, it doesn't feel meaningful, you are really wasting your time. And so Move on, move to a data set that does work for you. If you're a teacher, who is very philosophically opposed to standardized testing, I understand and respect that position completely. Don't start your data analysis journey with your standardized testing. Look at some of those other teacher oriented data's don't throw the baby out with the bathwater, so to speak, because you don't like one source of data, just move on from that source and build your skills using other sources of data down the road.
Scott Lee 22:23
We talked about our school culture, and individual teachers could really benefit from using the state testing in the right way. What message should we be sending to policymakers and I'm thinking state school boards or local school boards or even elected officials and state legislatures.
Matthew Courtney 22:45
So I think we really need to shift our mindset from a place where accountability systems are about pointing fingers, which is very much where I feel like we are now into a space where accountability systems are about supporting ongoing improvement. I talk a lot about school improvement, and through my blog and some other resources that I have. And I believe that school improvement is for every school. School Improvement is how we make good schools great and great schools even better. But we have to shift our mindset to where almost have a feeling that nothing is ever good enough that we could have well, we got 98% proficiency, but we could have 100. And let's drive to that goal, always thinking about how are we better this year than last year. To me, that's the shift that we need to make in terms of policy. And really, in terms of messaging, because a lot of a lot of this is really a messaging, it's like a PR problem. We don't actually have to change a lot of policy in order to do that. But we have to change the way we talk about it, we have to change the way we message accountability systems, we have to do more critical thinking thinking and less sort of hand wringing on CNN,
Scott Lee 23:59
you're mentioning that and I'm thinking about one of our local schools here, got a little bit of bad press, when their proficiency rates on one of literacy tests went from 98% of students proficient and 95% of students proficient, and, you know, kind of just the opposite. It's the same problem that just the opposite way. 95% Should we should be, you know, cheering loudly for for the students in that school and the teachers in that school. It cuts across even people and even schools that are doing very well.
Matthew Courtney 24:42
Yeah, and I think a lot you know, we have a lot of systems in the country that include things like gret student growth or student changing from one category to the other in various different ways. And so that's a metric that for a traditionally really high performing school can be a challenging metric because If you're already at 95%, proficiency, 98% proficiency, it's doesn't look like a lot in the numbers. But that's a huge achievement. It's a huge accomplishment. It takes a lot of work on that high end to move just a little bit. And so that's where the messaging comes in. Right? That's not a bad thing. But if that's the thing that the newspaper editor picks and says, we're going to talk about not about how they're at 95% proficiency, but about how they only went up one point, you know, that they were just really sensationalizing the data. And that's really unhealthy and not helpful to anyone.
Scott Lee 25:36
No, yeah, it's really not. I appreciate this conversation, Matthew, it's been, it's been a lot of fun. The book is exploratory data analysis in the classroom. And we will have a link to purchase the book and also to Matthews blog on our website. Once again, thank you for joining us today, Matthew.
Matthew Courtney 26:00
Thanks for having me, Scott. It's been a lot of fun.
Scott Lee 26:03
The Thoughtful Teacher Podcast is brought to you as a service of Oncourse Education Solutions. If you would like to learn more about how we help schools and youth organizations, embed social emotional learning within their cultures and implement strength based restorative interventions, please visit our website, www.oncoursesolutions.net.
This has been episode 12 of the fall 2022 season. If you enjoy this podcast, please tell your friends and colleagues about it either in person or using social media. We also greatly appreciate positive reviews on the podcast app you use. The Thoughtful Teacher Podcast is hosted and produced by R. Scott Lee who retains copyright. We encourage diverse opinions, however, opinions expressed do not necessarily reflect the views of producer, partners, or underwriters. Guests are never compensated for appearance, nor do guests pay to appear. Transcripts are available following podcast publication at our website, thoughtfulteacherpodcast.com. Sponsorship opportunities or other inquiries may be made on the "Contact Us" page at our website thoughtfulteacherpodcast.com. You may follow me on social media using Mastodon @drrscottlee@universeodon.com both Instagram and Twitter @drrscottlee and Facebook at Thoughtful Teacher Podcast
