My budgeting students just finished pouring through municipal budgets, CAFRS, and 990s as part of their final projects. Basically, I ask them to evaluate the health of a municipality of nonprofit organization using publicly available data. For some the project is frustrating. Why? First, financial health is an ambiguous concept, there are plenty of indicators that exist, but none directly measure financial health. Second, the reporting of information in budgets and annual reports is often inconsistent over time. Staff turnover, new GASB or state reporting requirements, and new elected officials or EDs often means new ways of reporting financial information. My advice to students is to use their judgment to make the best analysis they can. I do have a method behind my madness, I think a key lesson for students of public and nonprofit management is that ambiguity is imbedded in systems in which there are two sometimes-competing bottom lines of profit and mission.
Alan Borsuk’s column in Sunday’s Milwaukee Journal Sentinel, which looks at the legacy of No Child Left Behind (as well as the 20 budgeting projects I just graded), has me thinking about the difficulties of performance measurement in the public sector. Arguably, K-12 education has an advantage over many public services because test scores are an externally accepted measure of organizational performance. The problem is, just like with some of the budgets my students examined, Wisconsin changes its approach to measuring student achievement far too frequently. Borsuk notes that Wisconsin is “bringing another new test to students this year (with hopes that things will stabilize).”
History suggests that the testing situation will likely not stabilize, though I hope it does. The Wisconsin Department of Public Instruction (DPI) has a nice history of state testing policy that is worth reading: http://dpi.wi.gov/assessment/historical/wkce/1975-2014. The bottom line is that for almost four decades Wisconsin policy makers have tinkered with or outright changed the way in which student performance is measured, making longitudinal comparisons over more than a few years impossible. Simply, the revolving door of tests has made using student achievement data to inform decision making extremely difficult.
Practitioners of public administration and nonprofit management are well aware of the pressure to make performance measurable. Donors and taxpayers want to see what they are getting for their investments. I actually think this is good from both an accountability perspective, and a management perspective. Quality, consistent long-term data can show managers what is and is not working. No measure will ever be perfect because concepts such as student achievement, financial health, and more generally public performance, cannot be perfectly measured. There will always be some measurement error. However, the inconsistent use of measures makes a difficult task that much harder. So I hope Borsuk is right and the testing situation does stabilize in Wisconsin. More broadly, I hope the next generation of policy makers and public and nonprofit managers understand that analytics are a much more powerful management tool when measures are consistent over time.