Standardized Testing and the Analytics Paradox

Two new articles caught my eye over the weekend. The first was Alan Borsuk’s latest column touching on the state of testing in Wisconsin, which he appropriately describes as being in a “state of limbo.” Borsuk states that Wisconsin students will, in 2015-2016, be taking their third different test in three years (and what test will be used is still undecided). In addition, the school report card will be taking a year off. I guess that means my son’s school will continue to “meet few expectations” for a while.

The other article is an academic piece entitled Value-Added Measures: Undermined Intentions and Exacerbated Inequities by Kimberly Kappler Hewitt. In the article Dr. Hewitt reports the results of a survey on value adding testing she administered to teachers in a decent-sized North Carolina school district. Generally, she found the use of value-added testing in teacher effectiveness models to increase perceived inequities and undermine teacher morale and methods.

Reading these two pieces on the same day got me thinking about what I call the analytics paradox. It is impossible to perfectly and objectively measure public performance, yet the public sector nonetheless needs to use analytics to improve public performance. As I tell my students in my research methods course, analytics are nothing more than attempts to measure a concept. So, in K-12 education a standardized test score is an attempt to translate student performance (or teacher performance, or school performance) into a number (or grade, or percentage, etc.). Obviously, it is impossible to perfectly translate a concept as complex as student performance into such a simple variable. Yet there is huge pressure to try. And I think that it good.

Why? As I wrote a few years back in a report I co-authored for the Wisconsin Policy Research Institute, data, when in the hands of public employees, can be actionable intelligence used to improve the performance of public organizations. Now, if that data is viewed as an attempt to undermine public employees, as the Hewitt piece illustrates, the chances of said data having a positive impact on public performance goes way down.

Going back to my first paragraph, I mentioned my son’s school is deemed to be meeting few expectations under the state report card system. What exactly does that mean? A closer look shows the low score is mostly driven by a failure of the school to close reading and math achievement gaps. On its face, that is good to know. But the story is more complex. The school is only K-5, meaning the school has very little time (grade 3 – 5) to close the gaps. In addition the school has a fairly large ESL population (in addition to being majority low-income). The way I take this, as a parent and taxpayer, is it is important to know where the school can improve, but also not surprising that the gaps exist given the population served and limited time of attendance.

On the heels of the push to turn over control of low-performing schools in Milwaukee to outside providers, I could see how the school district may view this report card as a threat. As a researcher, I’d like to view this as a baseline by which to determine if the school is showing progress. But, as Borsuk articulates, this is impossible because Wisconsin keeps changing the tests.

The lessons for policy makers and public managers here is actually fairly simple. First, policy makers need to make performance data widely available to public managers in a format they can use on the ground. Simplicity, like letter grades for schools, sounds nice, but doesn’t mean all that much to the practitioner. Second, policy makers also need to be consistent with how they measure public performance. Three years of different tests means three years of very limited-use data. There is no perfect analytic, and the search for it undermines the usefulness of what is available. Last, practitioners need to continue to embrace the use of analytics (the reality is public employees constantly make good use of data), and demonstrate (celebrate?) exactly how they are using data to make evidence-based decisions.

It is disheartening to watch how politicized and divisive the boring world of K-12 performance data has become. Creating consistent, useful systems with high face validity need not be this complicated.

Advertisement

Brief Thoughts on Last Night’s Debate

Everyone else is doing it, so why not chime in? Yes I watched the debate and yes I was entertained. It is really odd to see 10 successful people used to having power and respect bicker like little kids. Not sure who won, but here are my thoughts on each candidate.

Jeb Bush: I kind of liked him in this debate. He looked uncomfortable and boring, but he at least came across as serious and competent at governing.

Ben Carson: Carson’s play was obviously his intelligence. We get it that you are the smartest guy on the stage, but why do you want to be president? Why do you know how to govern? And why the heck are you putting yourself through this?

Chris Christie: I liked his smack down of Rand Paul, it sticks with my preference for candidates that recognize the seriousness of governing.   I thought he came across fine.

Ted Cruz: Cruz makes me feel uncomfortable when he talks. He obviously has a constituency, but I don’t get the appeal.

Mike Huckabee. See Cruz.

John Kasich: Wow, federalism is ok? Healthcare is something people should be entitled to? You are tolerant of gay marriage even if you are against it? I thought he did great.

Rand Paul: He is wrong on most things, in my opinion. He held true to form from my perspective.

Marco Rubio: Still kind of looks uncomfortable, but I thought he came across as a serious candidate….which is perhaps the bar for success in these debates.

Donald Trump: Is his platform that he is a jerk? I don’t get the appeal aside from the entertainment value. I thought he was a train wreck last night with no substantive answers to any questions.

Scott Walker: He knows his talking points, but I am not sure I know who he is. Which is odd for a guy who seemed to know exactly who he was when he became Governor. I find his response to the Black Lives Matter question a missed opportunity. In my opinion if a group of Americans are losing trust in a public institution it is a huge problem whether you agree with their position or not. The answer that a few bad apples just need to be fixed through better training ignores the larger government legitimacy issue.

I guess I left the debate feeling like I do after most debates. Government needs to meet the world as it is, not advocate policies and positions based on a hoped-for but fantasy reality. But that is why I like the administrative side of things.

Embracing Complexity: Why is Studying Public Management Important?

What exactly is a MPA?  What do you do with it? These two questions are often asked of me by prospective students, friends and family, and even the occasional academic. Sometimes I give the canned answer: It’s like a MBA but for the public sector. Or the snarky answer: It’s a MBA with a conscience. Or the official answer: It’s a professional degree that will prepare you for public sector management. All are true to some degree. But there is a more important question we in the field should be answering: Why is a MPA important?

The PA Times recently published an interesting column on this very subject that is worth checking out. The piece got me thinking, especially where the author argues that MPA graduates are valuable because they are able to see the big picture. I agree. I’d argue that the complexity of the big picture is the defining difference between management in the private and public sectors.

This is not to say there is not a need to understand the big picture in private sector management. There is. However, the private sector has the distinct advantages of having an obvious monetary definition of profit, and a defined shareholder base. In the public sector the nature of profit is not so easy to define. It is value-laden and defined by the needs and wants of an incredibly diverse group of shareholders, i.e. the public. One of the reasons I enjoy teaching about and researching policy issues in Wisconsin is the state’s sharp political divide. Just about every significant public reform will be opposed by a large chunk of the electorate. This may be ok for politicians who need 50% + 1 of the vote to keep their job, but it is incredibly challenging for public sector managers who must implement programs in equitable ways despite a divided public. How do you maximize public good when there is widespread disagreement about what constitutes the public good!?!

I’d say you give it your best shot be utilizing the skills taught in a MPA program. One of these skills is asking what H. George Frederickson called “the second question.” The first question is “whether an existing public program or proposed program is effective or good.” The second question is “For whom is the program effective or good.” Simply, the public sector manager does not have the right to refuse service, and thus must consider things like equity that may reduce performance (as narrowly defined), but are still crucial considerations in government programming.

An illustrative example is public education. If I wanted to improve the performance of the American K-12 education system I could decide that certain students are not worth educating. I could do a cost/benefit analysis of the likelihood of a successful outcome, and direct tax dollars and capital to the students most likely to succeed. Test scores would skyrocket. It would also be unconscionable. Now if I was running a business, I would of course locate it and dedicate resources to the locations and clientele who are most likely to make my business successful. And there would be nothing at all wrong with doing that.

The reality is that public sector management is unique, and requires a different skill set. There is much that overlaps, but the value-laden complexities of managing public resources equitably in a society that embraces a diversity of values, goals, and needs, requires an ability to make sense of a big-picture that is always vague and ever-changing. It’s not easy to obtain and maintain that skill set, but it sure is important (and fun) to try. So there you go, that is why I think a MPA is important.

The Importance of Voucher Transparency

Looking back, much of my professional involvement with the school voucher issue dealt with a single topic: Accountability. I discuss that issue in an upcoming article (not yet published) where I found that voucher school administrators generally have a positive view of the Department of Public Instruction, and a wide variety of takes on the issue of accountability. The concept of accountability fascinates me, mainly because different audiences define it in very different ways.

One of those ways is basic transparency. Here I too have done some work. Last year I published an article in Education and Urban Society entitled “Changes in School Enrollment Patterns After the First-Time Release of School Level Test Scores in Milwaukee’s School Voucher Program: A First Look.” This article is part of a larger agenda of mine in which I explore the organizational characteristics of actors in the Milwaukee Parental Choice Program (MPCP) rather than the issue of whether or not vouchers on aggregate help or hinder education. Why? First, the issue of voucher impacts on test scores is pretty settled…there is little to no aggregate impact …and second I believe the proper unit of analysis is the school, not the program as a whole.

Anyways, in this article I take advantage of the 2010-2011 policy change requiring voucher schools to publicly release the WKCE scores of their voucher students. Under basic voucher theory, you would expect enrollment to increase in higher performing schools, and decrease in low-performing schools due to the transparency intervention. Indeed, I found schools with enrollment growth had significantly higher test scores than those with enrollment declines. I also found that above average voucher schools (as measured by test scores) gained students at a significantly higher rate than their average growth prior to the release of test scores. I also found that schools with lower reading scores were comparatively more likely to close. Interestingly, when I control for school level variables like religious affiliation, these relationships lose their significance.

Hence, my conclusions are a bit preliminary (it is a topic I will revisit). However, there is evidence that enrollment patterns were changed by the release of test scores. And importantly, changed in a good way as parents gravitated toward higher performing schools. Now, it does appear that other factors, many of which go unmeasured in my study, also drive enrollment patterns (see Paul Teske and company’s piece on how parents choose schools for more on this). So what to make of this? My conclusion is straightforward and simple, voucher programs should be required to publicly release test scores in a manner that allows parents and policy makers to make comparisons between schools across sectors. There is no evidence that doing so his harmful, and some evidence that doing so increases enrollment in higher-performing schools. From a practical standpoint, my findings suggest Wisconsin policy makers need to move to a stable universal accountability framework for all publicly funded schools. Most probably don’t need an academic paper to reach that conclusion, and policy makers have indeed articulated the need for such an approach…we just never seem to get there.

Will the Milwaukee Opportunity School District Work?

Probably not. For those unfamiliar, the Milwaukee Opportunity District will transfer the governing authority of certain Milwaukee schools over to “a Commissioner appointed by the County Executive”(read all about it starting on page 557 of the linked document).   Eventually, up to five Milwaukee Public Schools (MPS) deemed failing under the Wisconsin accountability report card will annually be transferred over to the Milwaukee opportunity district. That district will hand operating authority of these failing schools over to charter school, voucher school, or new school operators via a contracting process. The caveat is that the schools under contract must have a demonstrated track record of operating a school with test scores that are higher than the schools they are charged with turning around. Simple, right?

While I have no doubt this plan comes from a sincere desire to improve the lot of some of Wisconsin’s neediest students, it is premised on many questionable assumptions. First, it assumes that the MPS governance structure is the reason these schools are failing to meet expectations. Where is the evidence of this? Consider, if you accept that the MPS governance structure is the cause of the lowest-performing schools in the district than you must accept that the MPS governance structure is also the cause of the highest performing district schools. Further, does this mean that the lowest performing voucher and charter schools are low-performing simply because they are voucher and charter schools? This discussion is getting a bit muddled, and that is the point. If 25+ years of macro-governance reform in Milwaukee have proved anything, it is that there is nothing inherently superior or inferior about a voucher, charter, or MPS school. Yet, the Milwaukee plan seems to ignore this fact.

Second, the unit of analysis is wrong. While it is smart that policy makers want to ensure only high-quality operators are given authority over schools in this new district, the spotty track record of replication and expansion efforts is telling. Every school is a unique organization with a unique culture. A high-performing school has built and cultivated that culture over time and cannot quickly transfer that into a different organization. Doing so will take time and effort, and it will not necessarily work. As a wise person in Milwaukee education once told me, everyone seems to underestimate just how difficult it is to create a quality school. It takes more than will.

Third, this proposal further fragments Milwaukee’s disjointed public education system. I have written before and still believe the maintenance and strengthening of Milwaukee’s publicly funded education institutions is a moral imperative. While I doubt anyone would go back 25 years ago and create the system we have today, especially given the results, it is the reality on the ground. The public policy priority should be, in my opinion, recognizing the value of all sectors of Milwaukee’s education system by working towards building a coherent governance model with consistent goals, funding, and accountability expectations. Most importantly, any coherent governance model must be representative of the diversity of views held by Milwaukee citizens. I do not see how this is possible without a democratically elected board.

I hope I am wrong and that the Milwaukee Opportunity School District turns around the lowest performing schools in Milwaukee. Unfortunately, so much of this plan tells me it cannot work.

Who Cares Who Controls Scott Walker

I was recently asked my opinion on a Milwaukee Journal Sentinel blog post, “What is driving Scott Walker’s war on Wisconsin Universities.” As the person asking my opinion obviously knew, budgetary changes in higher education interest me on many levels. One, I teach public budgeting, two, I have considerable professional experience in the field, and three, UW cuts personally impact my ability to serve my customers…i.e. the students paying for an education.

Personally I am opposed to the UW-System cuts, decreasing investment in higher education creates short-term management challenges for Wisconsin universities, and is impossible to justify as a long-term strategy for strengthening the UW-System…especially when the corresponding flexibilities were removed from the budget. Of course, my opinions on this are hardly surprising given my place of employment. What really caught my eye in this post was the attempt to paint Scott Walker as a puppet for special interests. Simply, I do not think is the case, and more importantly, the attempt to make this case detracts from the legitimate arguments of Walker opponents.

The blog post in question makes a big deal about the Bradley Foundation and the organizations it has funded over the years. (Full disclosure, I am the former Research Director of the Wisconsin Policy Research Institute, one of the organizations discussed in the article). I get why the author makes a big deal about the Bradley Foundation, the foundation is a big deal. It funds organizations and causes it believes in, some controversial, some not. This is pretty much what a foundation does regardless of their core philosophies, so it should not be surprising. The author specifically cites a 1994 magazine article written by the now deceased former head of the Bradley Foundation as evidence that the foundation is driving Walker’s current position on the UW-System. I find it hard to believe that Walker’s most recent budget is at all informed by a 20-year old magazine article. The author goes on:

Walker is executing the Bradley Foundation’s ideological crusade to undermine institutions that serve the public instead of exploit them for profit. In Walker’s war, economic development is no longer a tool to create jobs, grow business, and support the middle class, but is instead just a giveaway for his wealthy donors to profit from taxpayer dollars. Neighborhood schools and the students who attend them are sacrificed for the profits of the unaccountable voucher school industry. Rules regarding public health and safety are cast aside, replaced with the demands of special interests and corporations. According to the vision of the Bradley Foundation, public policy and public services are sacrificed for profit and exploitation.

This is over the top on many levels. I do not claim to be in the head of Scott Walker, but isn’t it possible that he takes positions on issues because he supports them, whether or not the Bradley Foundation or anyone else supports them? Isn’t it possible that the majority of voters in Wisconsin, who keep electing Walker, do so because they support these positions too? Not because they are stupid, or because they are being duped, but because they made a rationale decision based on their core beliefs and outlook on life? I am burying the main point, but here is where I am going with this, it is unproductive and divisive to assume the views of politicians or citizens with whom you disagree are illegitimate. Doing so makes any serious discussion or progress on policy issues impossible. It also takes attention away from Scott Walker’s very real flaws.

A compelling case against Scott Walker as Governor (or president) is fairly straightforward to make. The debacle of the Wisconsin Economic Development Corporation displays a serious management deficiency. His scattershot K-12 education policies suggest he has no clear vision for K-12 education; thereby ensuring progress will not be made. His post Act-10 approach to budgeting is jeopardizing what could have been a public management revolution. His policy positions on hot-button issues are becoming embarrassingly reactive. I could go on. The point is, focusing on theories of who controls Scott Walker means we are not focusing on issues that have a real impact on the future of Wisconsin. All that guarantees is more division, and less progress.

Improving Achievement Through School Board Governance: Evidence From Wisconsin

One of the more frustrating parts of academia is the struggle to get your work in front of people that can actually use it. To paraphrase the fictional Bunny Colvin, what is the point of doing research (especially in public affairs and nonprofit management) if the only audience is other academics? In my pre-academic career it was always easy to get policy reports out there, which makes me wonder if critics of academia are on to something when they argue our research is too esoteric to be useful. Personally, I do think academic research can and should be of practical use. With that in mind, one thing I hope to do here is summarize some of my own research in ways that can be of use to practitioners.

Earlier this summer, Douglas Ihrke and I published a piece entitled: Do School Board Governance Best Practices Improve District Performance? Testing the Key Work of School Boards in Wisconsin? This article is part of a larger project in which we attempt to understand if and how school board governance impacts student achievement. The general impetus for this project is the continuing debate over democratic governance in public education. My thinking is that before getting rid of school boards, we ought to know a little something about them. Specifically, we surveyed school board members in the state of Wisconsin on a wide variety of issues, including the extent to which they focus on the ideas embedded in the Key Work of School Boards. What is the Key Work of School Boards? More or less, it is a guide of best practices created by the National School Boards Association.   The basic idea is that high-functioning school boards should be placing their time and efforts on specific areas in order to boost achievement. Fair enough, but is there evidence that boards placing more focus in these areas are actually obtaining better results?

We asked board members the extent to which they agreed (on a 1 to 5 scale) with the statements below. Each statement operationalizes one of the key work focus areas, which are also listed below in bold.

• Vision: We engage in continuous strategic planning.
• Standards: We set and tweak district academic standards in response to student needs.
• Assessment: We set and tweak district assessment policies in response to student needs.
• Accountability: Members freely admit when they are wrong.
• Alignment: We view the Superintendent as a full partner in the governing process.
• Climate: Members can take each other at their word.
• Collaboration and engagement: We regularly listen to the ideas of community members and act on their input when we 157 deem it appropriate.
• Continuous improvement: We frequently and consistently engage in board development activities.

Using their responses, we created an index that measures the overall extent to which board members adhere to the key work concepts. We then performed an analysis to determine if board members adhering to the concepts oversee comparatively higher-performing districts. Once we accounted for the impact of district size and student demographics, we found no clear link between adherence to the key work concepts and reading, math, or overall accountability scores. However, there is more to the story.

When we restricted our analysis to board members who have served for five or more years…the reasoning being it takes time for board governance to impact performance…we found a statistically significant link between adherence to the key work concepts and reading proficiency levels, and district accountability scores. In English, there is evidence that adherence to the concepts embedded in the Key Work of School Boards does positively impact academic achievement.

The practical takeaway of this research is that school board governance matters. In particular, board members seeking to improve the performance of their district through improved governance should focus on the Key Work areas, and should do so over a period of time. Notably, the link we found between school board governance practices and academic achievement is much smaller than the link between student demographics and academic achievement. However, a school district cannot control the make-up of their student body, but school board members can control their areas of focus. Which is why I think our finding is substantively significant. In the future, we are working to see of this relationship holds true in other states. But for now, it suggests the National School Boards Association was onto something when they developed the Key Work of School Boards.

Explaining Scott Walker’s Appeal to Walker Opponents

Scott Walker’s announcement that he was running for president kicked of a fury of activity on my Facebook account. The posts were fairly consistent, conservative friends praised him, and liberal friends denounced him. However, one post from a particularly intelligent, likely left-of-center friend of mine caught my eye. My friend asked a simple and honest question, what is the appeal of Scott Walker? To my friend, and so many others in our divided state, the three-time election of Scott Walker as governor is confounding, but the potential of a Walker presidency is unfathomable.

This got me thinking, what exactly is it about Scott Walker that draws support, particular in a state that has voted blue in every presidential election since Ronald Reagan?

Reason one, while not very exciting is nonetheless true: The low quality of his electoral opponents. Every election is its own animal, and in a closely divided state the winner is often decided by voters who do not reflexively vote for one party’s nominee. Tom Barrett was hurt by the Doyle legacy in 2010, and was a retread in the 2012 recall. Mary Burke seemed over her head from the start, and like Barrett, offered no compelling positive message. Now, this could be dismissed as luck, but remember Walker withdrew from the 2006 Republican primary, surely knowing the difficulty it beating the incumbent Jim Doyle.

Reason two is Walker’s early public attitudes on social issues. Many conservatives cringe when Republican politicians begin talking about abortion, gay-marriage, and other hot-button cultural issues. The idea of an elected official who views his or her job as running government in the most efficient way possible, rather than spouting off about social issues beyond his or her control, is appealing to many. Walker cultivated this image as Milwaukee County Executive, and in his first term in the Governor’s mansion. More recently, and I think unfortunately, Walker has been vocal in his conservative stance on social issues. These stances may play well to presidential primary voters, but will likely alienate some of his early Wisconsin supporters (even his sons disagree with some of his positions).

Reason three is Walker’s regular-guy image. Several years ago I was waiting in the concession line with my son at a Marquette-Wisconsin game only to look up and see the Governor standing next to me. He was by himself, wearing a Marquette sweatshirt, and not at all intimidating to talk with. He came off as comfortable, friendly, and genuine. In my career I have had the chance to meet several notable politicians, and none came across as relatable as Walker… and it is not close.  Why does this matter? Well, there is a sizable chunk of the electorate that wants leadership with lived experience on middle- and working-class issues.

Reason four is Walker’s first budget. In 2011, Wisconsin’s bi-partisan legacy of using short-term gimmicks to have our cake and eat it too, finally stopped (if ever so briefly). This is not to say Walker’s budget was perfect, or that alternatives to spending cuts did not exist. However, Walker was plain about his aversion to revenue increases in an honest approach to addressing the state’s structural deficit. Such an approach was highly appealing on the heels of the excesses of the Thompson and Doyle administrations.

Reason five is his survival of the Act 10 protests, and the recall. No matter one’s views on Act 10, the Governor’s ability to weather the massive protests and win the recall election demonstrated a level of steadfastness and toughness that many find compelling in a leader. The fact that Walker weathered these storms calmly, without resorting to the abrasiveness of say a Chris Christie, demonstrated a calm temperament that is attractive in a leader. Remember, President Obama emphasized this very quality during his first successful presidential run.

Reason six is Act 10 itself. One Act 10 takeaway is that a sizable percentage of Wisconsinites felt that public sector employees were not fairly sharing in the economic pain following the 2008 great recession. Walker was able to translate the unfairness felt by this population into a piece of policy he supported despite tremendous pressure to back down, helping to build a coalition of support that may propel him to the White House.

Scott Walker has plenty of flaws. Take a look, for example, at his shrinking approval ratings here in Wisconsin. However, Governor Walker is a top-tier presidential candidate for a reason. To answer my original question, what is the appeal of Scott Walker? He is a Governor who, in a very short time period, has successfully navigated various political landmines, demonstrated steadfastness, portrayed a calm everyman image, and significantly altered an array of Wisconsin’s public policies. Love him or hate him, it is a record that holds a rational appeal to many voters here, and around the country.