Value Added Good for Some Districts; When Will Teachers Be Analyzed?

A great post sent to me by the Columbus Education Association at their blog shows why some folks are enthusiastic about the new value-added metric that the State of Ohio is implementing to demonstrate growth.

It could finally provide some concrete evidence for traditionally underperforming districts that their students really are making progress, even if they weren't considered and "excellent" district under the old system.

Columbus is a SOAR district, which means that it was part of the pilot program operated by Battelle for Kids. So the Columbus schools have been tracking their own value added scores for a couple of years. And now they can show the their parents and voters that students are, indeed, making progress.

I have one quibble with CEA President Rhonda Johnson, when she says this:

This closer look at student achievement could be misinterpreted by some as an effort to gather more data on the effectiveness of individual teachers. The state will not issue teacher-level value added reports to the public, but reports on individual subject, school and grade level value added areas will be available. “There is no way value-added could be used against teachers.” says CEA President Rhonda Johnson. “There is an article in the contract that prevents administrators from using standardized test scores to evaluate Association members.

Dream on, lady. It's coming. It is entirely possible to track an individual teacher's performance this way--go look at what Tennessee has been doing in this area--and there's no reason to expect that it won't happen here.

The piece from CEA blog is worth reading in its entirety, in part because they provide another overview of what value-added analysis means, but an excerpt from the post is found after the jump:

The Ohio Department of Education recently released the first annual report on students’ progress using value-added data for Ohio’s 610 school districts. The students of Columbus City Schools earned the district the highest rating possible from the state, achieving over a year’s worth of academic progress in a single school year. CCS joins other local suburban school districts, such as Bexley, Dublin, Hilliard, New Albany-Plain and Upper Arlington that are ranked in the top category. This rating puts CCS in the top half of all school districts across the state, surpassing Grandview Heights and Worthington City Schools, the only district in Franklin County given the “A” rating by the state of Ohio.

This data shows the enormous gains that Columbus Public School teachers make with their students. “Our students are growing more academically than anywhere in Central Ohio,” responded Columbus Education Association President Rhonda Johnson. “With the implementation of value-added, the high performance of our students and teachers have truly been documented in a way that has not yet accurately been shown on state report cards.”

Three color-coded ratings are used to determine the progress of school districts as well as individual schools within each district. Earning a green rating indicates that district’s students have exceeded a year’s worth of academic growth in one school year; yellow indicates students have achieved a year’s worth of growth in one school year, and a red rating indicates students have not made a full year’s worth of academic growth in one school year.

Value-added measurement is the method used to analyze student scores on standardized test over two or more different points in time. By looking at how a student’s test scores change, the growth of individual students or groups can be shown over time. A recent three-year study conducted by Ohio University showed that there is a clear connection between school districts that use value-added information to make strategic decisions and improved student achievement.

Read More...

Summary only...

Value-Added: A Primer

Value added results are out, at least sort of. The state has identified whether schools are plus or minus in terms of a year's growth, or even. Yellow and green are good in this model, red is bad.

The Enquirer has a run-down of Cincinnati area schools here, and they highlight one of the problems:

Oak Hills School District has enjoyed an "excellent" rating on its report card for six straight years, but appeared in the lowest category - "red" - for failing to make a year's worth of progress in the new data.

Meanwhile, the "green" category, which means students exceeded a year's worth of progress, included Rockdale Academy, an Avondale school in "academic emergency" since 2004.

"Some of the data were scaled improperly when it was done," said Jay Kemen, assistant superintendent of curriculum at Oak Hills. "The state is aware there are some glitches."

Kemen added: "This year's red, yellow, purple or blue designations - it doesn't have the integrity it should."

I'm not sure I understand how Oak Hills can be so sure about the data, and one of the problems with value-added as I see it is that there's really no way to know if what Oak Hills is saying is true. The statistical models being used to develop value added designations are so complex that the average voter or citizen has no chance of understanding it... or the average educator, for that matter. (Then again, the same could be said for school finance...)

BY THE WAY: If you look at the Enquirer's chart of local districts, you'll see that nearly every district is in the red on reading--only one district is "green". Now, gentle readers, why is that? Think about it, and maybe later this weekend when I have more time I'll tell you why I think so many districts appear to be behind in reading.

For those wondering what "value-added" means, I wrote a series on it last year, and you can find it here:

Part I: An Overview

Part II: News and Implications

Part III: More Thoughts and Implications

Jill has more here on the other corner of the state.

Read More...

Summary only...

Merit pay bonuses go to affluent schools; other merit pay news

In Orlando, it appears that teachers in white schools were more likely to earn a bonus under a merit pay system that went into effect there. One of the obvious dangers with merit pay is that it rewards teachers of students who were already doing well. The Orlando (Orange County) system was supposed to account for that, by used a system that rewarded improvement, much like our own value-added system.

It also appears that teachers of electives were at a disadvantage... but that seems to be a quirk of the system.

Interesting analysis. Here's a quote from the Sentinel article.

The Sentinel's review showed that teachers at predominantly white and affluent schools were twice as likely to get a bonus as teachers from schools that are predominantly black and poor.

It wasn't supposed to work that way.

Florida education officials promised that imbalances along racial or income lines would not happen under the state's beleaguered and now-defunct merit-pay program known as Special Teachers Are Rewarded, or STAR. Officials said the best teachers could win a bonus no matter where they worked or what they taught.

"It certainly doesn't inspire much confidence in the system," said Mark Pudlow, a spokesman with the Florida Education Association, a teachers union.

Makes you wonder a bit, doesn't it, what will happen when Ohio goes value-added? Will the value-added formula simply reward districts that are already doing well? Experts say no, but...the proof will have to come later.

A second article describes the problems teachers of electives (art, music, p. e., language) had in earning bonuses--another of my arguments against merit pay:

The Sentinel's review of the STAR plan showed only 7 percent of high-school teachers in elective classes -- such as Spanish, art and geography -- won a bonus though they made up 35 percent of the teachers who qualified for the money.

That would be pretty discouraging to teachers in elective areas, although to be fair, the article suggests that the discrepancy had to do with tests used for the pay system that were poorly designed.

That said, does anyone else see a problem with designing a test simply to determine whether teachers get bonuses?

Meanwhile, Houston's program also highlight some of the problems associated with merit pay programs. They have gone through multiple formulas to try to find one that's fair to the teachers affected by it:

HISD officials said they've gone to great lengths to find formulas that are fair for all teachers. One major element of this plan, for example, looks at performance gains over three years.

"Just because it's complicated doesn't mean it's smoke and mirrors," board member Dianne Johnson said.

Gayle Fallon, president of the Houston Federation of Teachers, still isn't sold.

"Anything would be an improvement. It's one of the few programs they had that couldn't get worse," she said. "But it's still got the same flaws. It's still very complicated and still, as a teacher, I can't sit there with this plan and tell you how much I'm supposed to improve."

The plan still puts too much emphasis on test scores and doesn't fully reward those who teach non-core subjects, she said. But Fallon said she doubts many teachers will opt out.

For a merit pay program to truly affect teacher behaviors, teachers have to believe that their efforts will be rewarded... and if they don't think the formula is fair, they won't believe that they'll be rewarded if they improve student performance. And if their previous efforts to improve student performance aren't validated by a merit pay system... well, that's pretty disheartening. Does more harm than good.

Read More...

Summary only...

New York City Schools: The Crucible

If you've been following the NYC Schools soap opera, it's been fascinating. If you haven't, you should: like an art house movie, it'll be playing soon at a theater near you.

I've been arguing here that the system of "accountability" created by NCLB is a mechanism for planned failure designed to justify the corporate takeover of the schools. And New York is the juiciest plum, one of the largest school districts with the largest budgets, rip for the picking by the likes of Edison and White Hat. Just look at the dollars at stake:

This year, for the first time, the schools are offering cash incentives to students and administrators. Some low-income students will be able to earn $25 a test for doing well on periodic exams. Principals can earn $25,000 bonuses if their schools shows considerable improvement.

All this comes at a time when the city’s Education Department has more financial resources than ever. After a long court battle forced the state to provide New York City schools with more money, the system will see a $1 billion infusion this year — $700 million from the state and $300 million from the city.

But for weeks now, city and state education officials have wrangled over whether the spending plan for $258 million of that money, subject to state approval, is being drawn up properly by the Bloomberg administration. And there are potentially far more significant battles with Albany ahead as the system of mayoral control comes up for reauthorization in 2009.

To be fair, I'm not on the ground in NYC, so I don't know the merits of the myriad reform proposals being implemented in the district. The small schools movement is promising to me, for various reasons I'll deal with another time.

But the testing regime and report card scores being planned for schools looks like another mechanism to attack schools, if the reports of NYC Educator are any indication.

You really need to read NYC Educator's full post (it's long) and the Times reports to get a handle on the shape of things, but here's a taste of NYC Ed's analysis:

Some critics say you cannot run a school system like a corporation, of course. Treating adult employees as cogs in the corporate machinery is one thing, treating kids in pre-K to 12th grade like that is something else. Children are people, not investments, and cannot be educated solely by paying attention to data.

Other critics (myself included) say that treating kids like cogs in the corporate machinery is EXACTLY the purpose of the "reforms." The education reformers - Bloomberg, Klein, Bill Gates, Steve Jobs - are all CEO's and corporate officers with decidedly corporate agendas. Socializing kids into the corporate agenda (longer work hours, longer work years, less pay, salary by incentive, data accountability) is REALLY the idea behind all of the "reforms." You'll note that Bloomberg/Gates/Jobs have created an education reform movement for the 2008 election season known as Ed in '08 and their agenda is: 1) longer school days/longer school year; 2) more standardized tests; 3) national standards for these additional standardized tests; and 4) merit pay for teachers. This agenda is EXACTLY the one Bloomberg is slowly instituting here in New York City, along with a corporate structure where each school becomes an island to itself.

Critics argue that paying kids to do well on tests sets up a very scary reward system for the future (will kids learn only if they're being paid to learn?). Critics also argue that merit pay based solely on standardized tests privileges teachers with AP and honors students over teachers with lower performing students and special education students. How will merit pay and incentives be doled out fairly when the level of student achievement is so different through no fault of the teachers or the schools themselves? Critics also note how all these reforms have been done without any input from parents (it goes without saying that CEO's Klein and Bloomberg would do them without any input from teachers or administrators.) Parent groups are very angry at the chancellor for a variety of reasons, but the biggest one is his refusal to consult with them over anything. (But again, why would a CEO consult with his customer base? If they don't like what he's selling, they can go elsewhere...)

If you read the Times report on the upcoming "grades" schools will be receiving, it seems pretty clear that the district is using a variant of value-added score reporting.

Officials say the assessment system is intended to measure actual student progress and not to reward those selective schools that already start out with a talented, high-performing student pool or to penalize those with poor disadvantaged students.

Thirty percent of the grade is based on student performance. But most of the grade — 55 percent — is based on how much students improve over time, as measured by English and math test scores, credits earned and gains on Regents exams. And 15 percent is based on an evaluation of each school’s overall environment, including attendance and the results from surveys of parents and teachers asking about their satisfaction.

In other words, the score reporting is based on the progress of individual students. I discussed the meaning and merits of Value-added in a series of posts last year, here, here, and here. Why should we care in Ohio or anywhere else? Because Value-added is coming soon--real soon--to a school district near you.

This is all tied to Merit Pay, by the way. Principals in NYC could receive up to 25,000 smackers bonus based on test results. Are you kidding me?

That could buy a lot of books.

Read More...

Summary only...

Value Added Education, part III

In my first post I gave an overview of what the term "value-addedd" means. In my second post I talked about various issues/concerns raised by value-added.

In the comments to my first post there were a couple of comments that I want to respond to here as a way of trying to explain more fully how the value added metrics works, how the numbers are arrived at. Then I will share some criticism of value added models, including an excerpt from a report by ETS.

But first I want to remind my out of state readers that this matters to you also. The formula that Battelle for Kids uses is created in Tennessee, and is called EVAAS. And this commodity has spread from Tennessee to other parts of the union... if it's not in your state yet, just wait. It may be on its way.

First of all, I want to try to explain again how a student's predicted score is created. I will be using some notes I have from a presentation by Battelle, who created Ohio's value-added model and pilot program. (More value-added resources available at their site.)

Remember, value-added education is all about each student making one predicted year's growth in a school year. So how does the model predict a year's worth of growth for a student?

First of all, a student is grouped with similar students from similar schools. (The similar schools concept should be familiar to Ohio readers: Ohio groups schools for accountability purposes based on size of school, relative income levels, and other socioeconomic factors.) Similar students are essentially students with a similar testing profile. Scores from previous years are used to identify students with similar patterns. According to Battelle (no link, because, again, these are my personal notes from a previous value-added conference):

Using the test data for students with similar prior performance on common tests and the tests' relationship to each other allows for the creation of statistically reliable predicted scores for each student in each subject.

So then, using this similar group, a student's predicted score is created. Then, the average predicted score for students within a particular school is calculated. When the average predicted score is subtracted from the average actual score, a school's effect is determined.

So if the average predicted scaled score in School Q is 410, and the average actual score is 405, then the school has a negative effect of -5. Not good.

Similar comparisons can be made between classrooms as well. In one class, the predicted score might be 420. But the actual score might be 410. A negative effect of 10. Would that teacher be in trouble? It looks bad.

But ETS has a report out that says, it's not such a great idea to judge teachers by value added scores. Here's an excerpt:

The application of most VAM's (value added models) involves both intricate statistical methodology and knotty questions of interpretations...

At the conclusion of value added analysis, a number is associated with each teacher. That number, expressed in scale score points, may take on both positive and negative values. It describes how different that teachres's performance is from the performance of the typical teacher, with respect to the average growth realized by the students in their classes. It is often referred to as a measure of "teacher effectiveness". A problem arises because the word effectiveness denotes a causal interpretation... if a teacher with an effectiveness of +6 were replaced by a teacher with an effectiveness of only +2, we should expect that the test scores in a typical class would be lower by an average of four points...

Obviously dusch a change can never be directly observed because the same class cannot be simultaneously taught by two different teachers. so we must somehow infer, from the data we do have, what the relative contributions of different teachers would be...

According to statistical theory, the ideal setting for obtaining proper estimates of teacher effectiveness (as defined above) is a school system in which, for each grade, students are randomly grouped into classes, and teachers in that grade are randomly allocated to those classes. Roughly speaking, randomization levels the playing field for all teachers in that each teacher has an equal chance of being assigned to any class.11 The data generated in such a setting would allow us to obtain a reasonable estimate of each teacher’s effectiveness, as well as a measure of the precision to be attached to the estimate. A finding that the average student growth associated with a particular teacher is significantly greater than the district average would be credible evidence for that teacher’s relative effectiveness.

Unfortunately, school systems do not operate by randomization. Many parents have strong opinions about which districts (and even which schools within districts) they want their children to attend, and make corresponding decisions about housing. Within a school, parents often exert influence on the class or teacher to which their child is assigned. Similarly, teachers can sometimes select which district to work in and, by dint of seniority, have some choice in the classes they teach, or even the schools in which they are placed.

.

Go read more of the report (pdf), which goes on to warn against making too much of comparisons between teachers based on value-added models. By extension, we might say the same thing about schools. And yet, very important decisions about schools will be made by policy makers and parents based on the results of this complicated statistical analysis that most laypeople are in no position to assess critically.

As more news and information about value-added becomes available, I will post it here on my site. It's about to become a huge part of how we view schools in Ohio, and if you have children it will affect you. My hope is that voters and parents will take the time to make intelligent judgements about value added measurements.

Read More...

Summary only...

Value Added Education, part II: News and Implications

This is the second of three parts on value-added education. The first part is here.

I had originally planned this as the final part, but the final part now will be a more thorough discussion of how the value-added scoring is done, and a response to questions raised in comments.

In this piece I want to talk about some implications for value added, and what it will mean to the students and teachers. I also want to provide some links to interesting news on value added in other states. (So far only Pennsylvania, Ohio, and Tennessee have value added education on a state wide system as part of their accountability systems.)

I have some thoughts about the implications of value-added measurements, which I will simply enumerate:

1. Will value added analysis be used to evaluate teachers? I think it's inevitable that it will. Right now the state and many administrators are saying publicly that it won't be used for that. But why not?

Just as the data from value added will allow the state to show whether a school is helping students make greater or less than expected progress, the same can be done with individual teachers. A district or school administrator could easily look at progress in one teacher's class of fourth graders and compare to another class of fourth graders. Could raises or tenure decisions be based on this? Absolutely. The state is claiming right now that this won't happen, because they don't want teachers to freak out on them. But it can and probably will happen at some point, if I had to guess.

It's already been done in Tennessee, where value added began. I think parents and legislators will demand it at some point.

2. What about high school? Right now value added only goes through the eighth grade, because NCLB only demands annual testing in reading and math up to that point. But in Ohio, with our 10th grade test, we could easily extend value added to that high school by adding a 9th grade achievement test.

Frankly I wish the state would do this anyway, since most schools give a practice test in the ninth grade for diagnostic purposes.

But now, there is even more reason to do it, since the state could give a value added measure if there were a continuous series of annual tests. In order to get a value added measure for 10th graders, there has to be a test in 9th grade.

It won't be long.

3. How do you think urban schools will do?

Value added was sold to urban schools as a way of showing the progress being made at their schools. Even if they aren't proficient, they could show that they were making progress. Think it'll work out that way?

A little too early to say. One of the commenters to my first post seemed to think that in Texas it didn't work that way. We'll see.

4. What is the impact on high achieving schools? Consider: students who are high achieving have to maintain that level of progress every year. If they don't the school takes a hit.

Is it harder to take a school with high achieving kids and make a year's growth, or to take a low achieving school and outpace it? Experts I've heard talk about value-added seem to confirm that the gifted classes and high achieving schools have the hardest time showing progress. Again, we'll see.

Some news, and links to discover more about value-added.

Great resource from Pennsylvania, explaining the ins and outs of value added.

In Tennessee, students making progress are now going to be counted as proficient under NCLB. This is a pilot and it is the first state to be given permission to do this.

What if they tried to tie student growth/progress to teacher education programs, and found out that students taught by graduates of certain universities showed more growth than students taught by graduates of others? You can do that. Louisiana did. It's a brave new world. Ohio, by the way, is doing this also. They just aren't making it public. Yet. (Seriously.)

I'll try to come up with some more links in part III, and I'll try to answer any questions addressed in the comments.

Read More...

Summary only...

Value-Added Education: Part I

I asked last week if anyone would be interested in a discussion of value-added education. I'm going to do it in sections, I don't know how many right now. I'll cover it from as many different angles as I can until I run out of gas. So keep checking back.

This post will be a little introduction to what the phrase "value-added" means and why it's important. Expand the post by clicking below, and we'll get started.

Think about the "district report card" that the state publishes for each district. It's a series of data points that determine whether a district is ranked Excellent, Effective and so on. The report card consists of points given for a percentage of the students in the district passing parts of the OGT, the OATs, student attendance and so on. I'm sure everyone is familiar with those report cards and what they look like.

Starting next year, the report card will include a value-added measure for grades 4-8 based on scores on the Ohio Achievement Tests. This score is a measure of whether students have made a predicted one-year growth on the test.

What happens is this. Say a student, we'll call him Dave, is in the sixth grade. He has taken the OAT for the past two years at Blogger School (BS). The state puts his previous test scores into a computer. It throws his scores in with lots of other students who are similar demographically. They attend similar schools, with similar income ratios and several demographics included.

I don't pretend to be a statistician, but essentially a prediction is made on how a student should perform. A student is expected to make a predicted year's worth of growth. Then, the students in the school are aggregated to determine whether the school as a whole has made a year's worth of growth. And that is reported on the school report card, just as a passage rate on a test would be reported.

It's a measure of progress. Students, and schools, could fall into one of three categories: within the predicted range (good thing), above the predicted range (even better), or below the predicted range. You don't want to be in that last category.

It's important to keep a few things in mind. First, the math behind all of this is more or less hidden from view. The average person could not sit down and determine mathematically what their child's predicted score would be. Parent reports will be sent out which show what Dave's predicted score was and what Dave's actual score was. But Dave's parents will not be able to look at that report and say, "Oh, I see where they got that number." It's a complicated statistical model that is multi-variate, so the average parent or educator will simply have to take the state's word for it.

Another key thing to keep in mind is that a school or district could have failing rates all the way across their report card but appear as a school or district that is making progress. That's because even if only 50% of the students are achieving the passing level, it could be that the students are statistically doing better than predicted. Maybe the scores are improving even if they haven't improved enough to get 75% passage. In other words, that school could be doing better than expected even if the majority of students still aren't passing rate. At least they are closer than they were last year. The value-added metric recognizes that.

On the other hand, maybe Dave's school has traditionally been excellent with passage rates in the 90's. But maybe the students' progress is less than what it should be. They aren't quite making a year's predicted growth. The value-added metric shows that also, and a school that has traditionally met all standards might lose the value-added, or "progress", standard.

I'll explain some of the implications for this in future parts of this series, and I'll look at some other states where this sort of thing has been tried.

If you have questions about what I've said so far, leave them in the comments and I'll try to address them next time around.

Read More...

Summary only...