Value-Added Accountability and Public Confidence in Education

J. E. Stone, Ed. D.

House of Representatives
Testimony to the Select Committee on House Resolution 495
Harrisburg, Pennsylvania
Tuesday, September 19, 2000

Introduction

Good morning.

My name is John Stone. I am a licensed educational psychologist and licensed school psychologist, and I have taught prospective teachers for the past 30 or so years. I am a graduate of the University of Florida and currently a professor in the College of Education at East Tennessee State University.

As I am sure you will notice, unlike Vice President Gore, who is also from Tennessee, I was not raised in the Washington DC area.

My purpose today is to talk about accountability and value-added assessment. Since the early nineties, Tennessee has used a system of educational accountability that serves the needs of both educators and the consuming public.1 It is called the Tennessee Value Added Assessment System (TVAAS).

Instead of looking at school system averages as does the Pennsylvania System of School Assessment (PSSA), value-added assessment compares each student to his or her own track record of achievement and summarizes that information by teacher, school, and school district. School and school district reports are made public. They enable parents to see not just which schools have high test scores, but which schools are helping students the most. Parents want schooling that optimizes their child’s talents and value-added assessment encourages that kind of schooling.

A critically important feature of Tennessee’s system is that expert help is not required to interpret differences between schools and districts.2 The scores are analyzed and reported in such a way that the influence of all preexisting differences is removed. I mean that everything from socioeconomic factors, to differences in talents, to differences in initial skill levels are removed so that everyone including the lay public can accurately understand the outcomes. Just as the newest generations of computer programs use very sophisticated programming to make software user-friendly, Tennessee’s system of accountability uses some very sophisticated statistics to create user-friendly educational data.

Teacher reports are sent to teachers and principals. They are enormously helpful because the parties on the business end of schooling need to see how well classroom practices and policies are working, and without the uncertainties created by student differences. Tennessee’s system, in effect, creates a level playing field for teachers.

Before I continue, I should make clear that my point of view may a bit different than that of other witnesses. I head an online service called the Education Consumers ClearingHouse (www.education-consumers.com). The ClearingHouse is a kind of Consumers Union for the consumers of public education.

It is a paid subscription service that serves parents, employers, school board members, lawmakers, taxpayers, and all who are on the consumer side of the education marketplace. We afford our subscribers access to consumer-friendly, information and expert advice. The ClearingHouse also supports the development of local Education Consumer Associations (they are like PTAs) and a Consultants Network. I am appearing here today as a representative of our Consultants Network.

All of this is to say that my remarks will not be from the perspective of the education community or an education interest group or from that of a vendor who is marketing something to the education community. Rather I am here to talk about accountability from the standpoint of what parents and taxpayers would like to see in their public schools.

Two Key Issues

I want to begin by saying that I am very much in sympathy with your effort to strengthen your state’s accountability system. There is a rising tide of public concern about whether students are getting what they should get from their school experiences. I believe that an effective accountability system can address that problem but a flawed system will produce change too, and maybe the wrong kind.

No system of summarizing the results of an enterprise as large and expensive as a school will reflect everything that the institution has produced and do so to everyone’s satisfaction. In truth, there is no perfect system of auditing or accounting whether we are talking about industrial outputs or financial management. Yet independent auditing is as necessary to public confidence in the schools as it is to consumer confidence in banks and corporations.

I am going to take the next few minutes to talk about two issues I believe are crucial to an accountability system that will build confidence in public education. They are:

  • How well does it respect the public’s educational aims and priorities, i.e., does it make them accountable for what the public wants?
  • Does it produce results that can be used to improve events at the classroom level? How well does it help teachers and administrators to see the results of their efforts?

These are by no means the only issues that you will want to consider but they are ones that I believe are essential to the kind of accountability desired by parents and taxpayers. I will close my remarks by presenting a brief description of Tennessee’s value-added system in the hope that it will suggest some refinements that you will want to consider.

Priorities

Let me begin by talking about the public’s aims and priorities. Parents and taxpayers value education primarily because of what it does for a child’s future. Responsible adults understand that kids grow up and that the knowledge and skills they acquire in school are critical to their becoming independent, self-sustaining members of society.3 Adults also understand that the opportunity to fully participate in and benefit from school does not last forever. They recognize that the competing responsibilities of adult life have undermined many well-intentioned efforts to return to school.

Educators believe that knowledge and skills are important to adult success but they do not necessarily agree that teaching focused primarily on knowledge and skills provides the most desirable sort of learning experience.4 Rather, virtually all educators are taught that the ideal learning experience is, first and foremost, pleasant, engaging, interesting, and enjoyable; and they are given to believe that such experiences automatically produce optimal learning outcomes. These outcomes may not be outcomes that will contribute to the child’s long-term welfare but educators are given to believe they are what is best for the child. Many parents have their doubts but they are supportive and cooperative in the hope that the desired outcomes will eventually emerge.

In my judgment, this misalignment between educational ideals focused on the immediate experience and parents’ desires for their child to acquire recognized knowledge and skills is the root cause of the public’s dissatisfaction with public schooling. Establishing clear accountability is needed not so much as a guide to adding resources or a means of identifying slackers; it is needed to redirect the very considerable effort that is already underway.

Let me digress a moment to clarify my meaning. Several recent studies have identified inner city public schools whose learning outcomes are equal to or better than most suburban schools. Kipp Academy in the Bronx, P. S. 161 in Brooklyn, and Wesley Elementary in Houston are three that have recently gotten attention in the national media. One would suppose that schooling methods used by the highly successful schools would be widely imitated but they are not. Why? Because their more structured and orderly approaches to teaching sharply disagree with the mainstream pedagogical wisdom, i.e., the educational ideal I am describing.

The ideal to which I refer is the foundation of a wide variety of educational practices, present and past.5 Diane Ravitch’s recent Left Back: A Century of Failed School Reforms offers an excellent description. They include fads such as the “open classroom” concept of some years ago and the current approach to reading instruction called “whole language.” They include “discovery learning,” “developmentally appropriate instruction,” and a number of other popular approaches to teaching that have been around since the sixties and seventies. Collectively they are called student-centered or learner-centered schooling.6 Student-centered is the term applied by Jeanne Chall in her posthumous The Academic Achievement Challenge.

The learner-centered ideal is an appealing but unsubstantiated concept that requires teachers to approach the task of teaching as though the student’s immediate satisfaction is more important than the long-term outcome. By contrast, the more traditional teaching methods used by Kipp Academy and other successful schools focuses, first and foremost, on the demonstration of knowledge and skills, not the student’s immediate reaction. They seek to be stimulating interesting, and engaging but not at the expense of the outcome.

As teachers at Kipp Academy and other successful educators have demonstrated, learning requires students to behave themselves, pay attention, and make an effort. With students who already fit that description, learner-centered approaches can work well. With the other 95%, however, a more structured and orderly approach is usually needed.

Learner-centered thinking is at odds with the public’s educational priorities in a subtle but critical way. Instead of teaching in a way that is hoped to be entertaining, teachers are encouraged to entertain in a way that they hope will be educational. Learner-centered approaches typically result in kids loving school and parents being favorably impressed when they visit the classroom. It’s only later that the substantive results are noticed.

My reason for addressing this issue is to make it clear that an accountability system that will build public confidence in education must be one that makes the public’s desire for improved achievement its top priority. Yet it must also be sold to an education community that is not convinced of the public’s aims. The task will require extraordinary leadership and, indeed, salesmanship.

The learner-centered pedagogical orthodoxy is widely held and many educators view the public’s insistence on objectively measured achievement as an infringement on professional prerogatives and a requirement that is contrary to good educational practice. Many have been trained exclusively in learner-centered thinking. Many have been given to believe that it is the only valid if not the only ethically defensible approach to teaching. In truth, many teachers do not know of, much less believe in, result-oriented approaches to teaching.

I will give you one quick example. The largest educational experiment in history was Project Follow Through of the sixties and seventies.7 It was part of the Johnson administration’s War on Poverty and it cost over a half billion dollars. Follow Through compared 11 of the major approaches to teaching by field-testing them at sites all over the U. S. Nine models competed the study and seven of the nine were learner-centered. Five of the seven learner-centered models produced less achievement than the comparison schools–schools using traditional teaching methods. The two non-learner centered models greatly outperformed the field beating both the other models and the comparison schools. Yet as of the present, the more effective of the two successful models-Engelmann and Becker’s Direct Instruction-is included in the curriculum of only one teacher training program, that of the University of Oregon.

For all of these reasons, I would urge you to be patient and to recognize that change will take time and significant rethinking by many in the education community. Change will have to be addressed on many levels. Professional development, for example, will be a key factor. Open-minded teachers-especially the newly minted ones-will need to be introduced to resulted-oriented teaching methods.8 The learner-centered view has been around since the early nineteen hundreds and it is ardently held by many fine, intelligent educators. Many feel morally obligated to oppose more result-oriented approaches because they have been told that they are a threat to student self-esteem or student creativity.

Teacher accreditation and licensure standards must also be reviewed and aligned. The new standards promoted by the National Commission on Teaching and America’s Future, for example, continue to favor learner-centered teaching. The Commission’s proposals agree with accountability, in principle, but urge fuzzy assessments of learning, in practice.9

All of this is to say that implementation an accountability system that makes objectively measured learning a clear and unrivaled educational priority will be a long-term undertaking and one that will have to address more than the mechanisms of accountability.

Helpful results

Individual classrooms and individual schools are where the rubber hits the road in education. There are many teachers and principals-especially experienced ones–who recognize that learner-centered teaching doesn’t really result in the kind of learning wanted by the public or by educators but they have to keep their skepticism under wraps. They know that education’s experts do not support their views and they are reminded of that fact every time they attend an inservice training session or continuing education course or teacher conference. Moreover, teachers and principals know they are expected to cooperate with various learner-centered reform initiatives that are implemented in schools. These research projects and educational innovations are almost all externally funded and administratively supported.

In such an environment, deviation from learner-centered orthodoxy can be career threatening. Teachers who want to implement result-oriented instruction frequently have to do it behind closed doors. In California, for example, the legislature mandated a return to more traditional methods of reading instruction after it was found that bottom had fallen out of reading scores. The essence of the problem was the learner-centered “whole language” reading instruction that had been promoted by the State Department of Education in the nineteen eighties. Despite the legislative mandate, teachers who were trying to improve results found it prudent to hide their phonics-based instructional materials. They knew that many of their peers and supervisors did not agree no matter what has been mandated.

Teachers and principals hear educational leaders talk about aims and policies but they see what is rewarded. Good teaching is theoretically that teaching which produces learning. In truth, however, actual measured learning plays little or no role in decisions about which teachers are given awards for excellent teaching. Popularity with students, parents, and other teachers is far more important. The same holds true with principals who are given promotions and pay increases. Measured learning plays only a small, if any, role in these decisions.

Two factors are primarily responsible for this inattention to results. One is the learner-centered orthodoxy. It practically obliges educators to be dismissive of results as indicators of true educational quality. The other factor is the difficulty of accessing, summarizing, and interpreting achievement data. Objective test score differences among teachers and schools have to be interpreted in light of all kinds of mitigating factors such as differences in entry level skills, socioeconomic differences, and myriad other factors. Differences on fuzzy outcome measures such as student portfolios are, as a practical matter, not interpretable. Kentucky, for example, tried to use portfolio assessment as part of its high stakes accountability system. After several years of effort and millions expended, it was discovered that the assessments lacked reliability.

The point I am getting to is that an accountability system that supports effective teachers and principals must permit educators to see not only differences among schools but differences among teachers. Some teachers are doing an effective job of getting students to learn. Others are not and they need to know what is working and what is not. Rarely is the case that they are insufficiently dedicated to serving students or not working hard.

The Educational Empowerment Act applies various consequences to schools and school districts, thus principals and superintendents will have to know which teachers are producing results and which are in need of help. Given the vagaries of test score interpretation, both administrators and teachers would be better served by an accountability system that provides fair, accurate, and objective data at the teacher level in addition to that which available for the larger units. Teacher data would not need to be made public (it isn’t in Tennessee) and its role in performance evaluations could be subject to negotiation. For example, it could be used in peer evaluations. In any case, improvement will require both administrative decisions and classroom practice to be tightly coupled with achievement data.

A study by Sanders and Rivers in Tennessee found that a student who had the good fortune to have effective teachers three years in a row averaged 50% more learning than one having poor teachers over a three-year span.10 As presently configured, the PSSA reports data at the school and system level yet learning is mainly governed by events at the classroom level. Ideally, parents want assurance that their school is providing an effective teacher, not just a popular one.

Tennessee’s Value-Added Assessment System

Now I will outline the workings of the Tennessee Value Added Assessment System (TVAAS) and try to show something of how it addresses these two issues.11 I will minimize details about how the scores are derived, but I will furnish you several articles that contain a fuller description and reference sources.12 Incidentally, Tennessee’s system has been subject to review by some of the foremost authorities in educational measurement and evaluation and it has performed as expected.

TVAAS measures teacher, school, and school system performance by looking at the year-to-year increases in achievement test scores earned by individual students. Every regular education student in every grade (2-8) takes an annual achievement test-currently a custom version of McGraw-Hill’s Terra Nova. Fresh items are written annually and the scores are reported on a 1000-point scale. The basic datum on which the value-added analysis is performed is the annual gain in individual achievement.

The annual test scores are analyzed using a statistical procedure called “mixed model analysis.” It is a type of analysis previously known mostly to geneticists and it has now been applied to educational achievement by Dr. William Sanders-developer of the Tennessee Value Added Assessment System. The Sanders analysis estimates the degree to which students have improved in a given year over their previous years of schooling.

In other words, Sanders’ value-added analysis looks at each student relative to his or her track record and estimates the individual’s gain over his or her history of academic achievement. Such an analysis is a very close approximation of the informal appraisal of student’s test scores that might be made by a teacher or parent. Although the actual computations do not produce individual student gain scores, teacher, school, and school district scores are the average of the gains earned by the students who were served by a given teacher, school, or school system.

Unlike assessment systems that compare scores to a norm or fixed criterion, value-added assessment encourages schools to maximize the talents of each student, not just those that are below a certain level. Such an approach is very much in agreement with the aspirations of most parents.

There is one other equally important advantage to Dr. Sanders’ “mixed model” methodology: Through a process statisticians call “blocking,” Sanders analysis excludes the influence of all preexisting differences among students in the estimates of gain. These include any differences that might be due to race, socioeconomic status, previous learning, intelligence, and any other biasing factors, known or unknown. These are the same kind of biasing factors that PSSA sought to eliminate from its school reports in the early nineties through a procedure called multiple regression analysis. The important difference is that Sanders’ analysis removes all such biasing factors and without having to individually measure them.

The attached charts illustrate the ability of TVAAS to level the playing field. One shows the relationship between school district gains in reading and the percentage of students receiving a free or reduced lunch. The other shows the relationship between school district gains in reading and percent minority students.

The importance of a level playing field

I want to take a minute to elaborate on the importance of a level playing field. I have taught college freshmen for thirty years and my wife is in preschool special education. We have both seen instance after instance of something that frustrates most teachers (whether they say so or not): The teachers with the most talented students are almost always thought of as the best teachers. Obviously, the most talented and advantaged students tend to learn the most and be most satisfied, so it is not surprising that their parents and most everyone else think they are doing a great job.

When school administrators say things like “everyone knows who are the best teachers,” these are the teachers they have in mind. In truth, the teachers who work with the weaker students may be working much harder and, in fact, may be contributing more to their students’ educational development. They rarely are identified as the best teachers, however.

Value-added assessment of teacher performance changes the accepted image because it looks achievement objectively. Even modest gains can be very impressive with a student that has not done well previously. Conversely modest gains with talented and advantaged students may not be satisfactory. In Tennessee, one of the most disquieting findings in the six years that TVAAS data that has been collected is that students with high achievement scores are in many cases making less than average gains. In other words, Tennessee has many bright students who have been permitted to coast toward the average, i.e., they have been “dumbed down.” When parents who have sacrificed to live in the “good” school districts learn the truth, they are very unhappy.

Other aspects of Tennessee’s value added assessment system

Testing all students annually is more expensive and time consuming than the selective sampling used by the PSSA, but the added cost is minor relative to state educational budgets. Additional time is used but it too is a small amount relative to the days lost on account of weather and extracurricular activities.

School and school system performance data is publicly reported in the form of a 3-year rolling average for each of the five subjects measured by the annual test-officially known as the Tennessee Comprehensive Assessment Program or TCAP. I have a copy of a TVAAS Report Card Supplement available for your perusal.

Teacher performance scores are based on all students taught by the individual teacher for at least 150 days in the course of the school year. Teacher performance is also reported in the form of a 3-year rolling average in each of the 5 subjects but unlike the school and system reports, it is not made public. Rather teacher reports, by law, are sent only to the teacher and the school system.

Although Tennessee has not yet done so, the teacher reports can be used as an indicator of teacher training effectiveness. The performance of novice teachers during their first 3 years can serve as an objective indicator of how well a given program is preparing its graduates.

TVAAS report cards are freely available to the public although many parents and taxpayers are only beginning to recognize their importance. Schools typically try to control their impact by spinning the annual results to the media. Unfortunately, education reporters often file a rewritten press release. However, Education Consumers Associations and real estate agents are beginning to read and use TVAAS data to advise parents and prospective homebuyers about the best schools and the best neighborhoods.

Summary and Conclusion

Renewed public confidence in education will require accountability that is fair and useful to educators and convincing to parents and taxpayers. Value-added assessment can meet all of these requirements, but the education community will have to be sold on the idea of teaching dedicated to achievement. If they accept that concept, classroom practices that work can be developed, identified, or adopted on the basis of their value added outcomes.

Endnotes

  1. For more complete information about value-added assessment, see https://www.sas.com/en_us/software/evaas.html and www.shearonforschools.com. Dr. William Sanders-the creator of Tennessee’s value-added assessment system may be contacted at the SAS Corporation <https://www.sas.com/en_us/software/evaas.html>.
  2. J. Stone, “Value-Added Assessment: An Accountability Revolution” in M. Kanstoroom and C. Finn, Jr., eds., Better Teachers, Better Schools (Washington, DC: Thomas B. Fordham Foundation) 1999.
  3. J. Johnson and J. Immerwahr, First Things First: What Americans Expect from the Public Schools (New York: Public Agenda) 1994.
  4. J. Stone, “Aligning Teacher Training with Public Policy,” The State Education Standard, 1(1), pp.34-38, (2000).
  5. D. Ravitch, Left Behind: A Century of Failed School Reforms, (New York: Simon & Schuster, 2000).
  6. J. Chall, The Academic Achievement Challenge, (New York: The Guilford Press, 2000).
  7. C. L. Watkins, “Project Follow Through: A Story of the Identification and Neglect of Effective Instruction,” Youth Policy, (July 1988), 7-11.
  8. D. Carnine, Why Education Experts Resist Effective Practices (April 2000). Available at the Fordham Foundation website: www.edexcellence.net
  9. J. Stone, “The National Council for Accreditation of Teacher Education: Whose Standards?” in M. Kanstoroom and C. Finn, Better Teachers, Better Schools (Washington, DC: Thomas Fordham Foundation) 1999.
  10. W. Sanders & J. Rivers, Cumulative and Residual Effects of Teachers on Future Student Academic Achievement, 1996 (Available from UTVARC, 225 Morgan Hall, P. O. Box 1071, Knoxville, TN 37901-1071). See also, J. C. Rivers Sanders, The Impact of Teacher Effect on Student Math Competency Achievement, unpublished doctoral dissertation (Knoxville, TN: University of Tennessee, 1999).
  11. Ibid. J. Stone, “Value-Added Assessment:”
  12. D. Hill, “He’s Got Your Number,” Teacher Magazine, Vol. 11, number 8, pp. 42-47 (May 2000); J. Archer, “Sanders 101,” Education Week, (5 May 1998); J. Archer, “Putting Value Added to Good Use,” Education Week, (5 May 1998).