The Value-Added Achievement Gains of NBPTS-Certified Teachers in Tennessee: A Brief Report
J. E. Stone, Ed.D.
College of Education
East Tennessee State University
Introduction
The National Board for Professional Teaching Standards (NBPTS) was established in 1987. Funded principally by the Carnegie Foundation, major teacher unions, and the U. S. Department of Education, its mission is to set advanced proficiency standards for teachers and to certify teachers who meet those standards.
Teachers pay $2,300 to be evaluated by the NBPTS. They prepare a lengthy portfolio, make videotapes of themselves, and take an all-day written exam. Only fully licensed and experienced teachers may apply.
States and school districts have offered powerful incentives, such as salary increases in the $5,000 to $7,500 per year range for successful applicants. As of January 2002, the NBPTS has certified 16,037 teachers in 19 areas of teaching. Over 20,000 more have applied. The NBPTS says certified teachers are “highly accomplished,” and it anticipates that only 10% of all teachers will eventually be so designated.
NBPTS standards are closely linked to the National Council for the Accreditation of Teacher Education’s (NCATE) teacher training standards and the Interstate New Teacher Assessment and Support Consortium’s (INTASC) teacher licensure standards. All are based on research, teacher opinion, and expert judgment; however, all reflect the same ideas about teaching that have dominated education for decades.1
Several studies have attempted to show that NBPTS-certified teachers are truly superior in the classroom. The largest of these investigations–a study by the Center for Educational Research and Evaluation at the University of North Carolina-Greensboro–compared 31 teachers who were awarded certification to 34 teachers who applied but were unsuccessful.2 The results indicated that successful applicants were higher on 11 of the 13 qualities the NBPTS considers essential to good teaching. The student achievement outcome, however, was equivocal.
Neither the UNCG study nor any others have affirmed that NBPTS certified teachers are exceptionally effective in improving objectively measured student achievement. Its assessment of student learning was based on work samples chosen by the participating teachers themselves-hardly an unbiased measure. Moreover, despite the key role of standardized tests in state accountability plans, the UNCG report argued that such tests are narrow, inadequate, and arbitrary.
The purpose of the present report is to address whether NBPTS-certified teachers are exceptionally effective in bringing about objectively measured student achievement gains.
Method
Tennessee has more than 40 NBPTS certified teachers. Sixteen of the 40 teach in grades three through eight and therefore have value-added “teacher reports” in the state database. Tennessee’s teacher reports summarize the annual achievement gains exhibited by each teacher’s students.3 Using the “mixed model” statistical methodology developed by Dr. William Sanders, student progress is estimated on the basis of how much students gain in comparison to their achievement increases in previous years. The Sanders estimates of teacher impact on learning-called “teacher-effect” scores–are considered the most fair and exact available.4
The teacher-effect scores on which this report is based were furnished to the author by the Tennessee Department of Education.5 They were extracted from the year 2000 teacher reports of the NBPTS-certified teachers who are included in the Tennessee Value Added Assessment System (TVAAS) database. Names and school system identifiers were not included so as to preserve teacher anonymity.
The teacher-effect scores in this study are reported on a scale of zero to 50. They represent the estimated mean achievement gains of the students taught by each teacher, in each subject taught by that teacher. Although Tennessee tests its students annually with McGraw Hill’s “Terra Nova,” not every teacher generates a teacher-effect score in every subject every year; thus some teachers may have data available for only one year and/or only in selected subjects.
The central question to be answered by this report is whether Tennessee’s NBPTS certified teachers are exceptionally successful in improving the achievement test scores of their students. For the purposes of this report, “exceptional” teaching is defined as teaching that brings about an improvement in student achievement equal to 115% of one year’s academic growth in the local school system.
This definition is based on Tennessee’s performance standards for schools and school systems. Annual gain in a given subject equaling or exceeding 115% of the national norm gain is considered “exemplary” and awarded a grade of “A.” Not incidentally, a gain of less than 85% is considered “deficient” and awarded a grade of “F.”
A recently created Chattanooga, Tennessee program uses the 115% standard for identifying “high performing” teachers. Teachers whose students gain 115% of the local average in 3 core subjects earn a $5,000 bonus.6
Results
The TVAAS teacher-effect scores for the 16 NBPTS-certified teachers are displayed in Appendix A of this report. The teachers (rows) are numbered 1-16, and each row contains a date, a grade level, and the teacher-effect scores for a given teacher. With the exception of the rows highlighted in blue (3-year averages), each row labeled “Teacher #” contains average student achievement gain scores for one year and in one or more of the five tested subjects: mathematics, reading language, social studies, and science. The rows labeled “System #” (immediately following each teacher row) contain the relevant school system average and standard error.
Each teacher data cell contains the teacher-effect score followed by its standard error of measurement (in parenthesis) and by the percentage of annual achievement growth that it represents (in brackets). The percentage of annual growth is the critical indicator of teacher effectiveness in this study. It is the ratio of the teacher-effect score to the average annual achievement growth for the school system multiplied by 100. The percentage of average annual achievement growth is the teacher performance indicator used in the Chattanooga program.
In the interest of enhancing the accessibility of this report, the use of statistics has been minimized. The reader is encouraged to inspect the data in Appendix A. The scores that would warrant a grade of “A” (i.e., scores that represent 115% or more average annual growth; and ones that would warrant a bonus in Chattanooga if achieved in mathematics, reading, and language) are highlighted in green. The scores that would warrant a grade of “F” (i.e., scores that represent 85% or less of annual growth) are highlighted in red.
Even if one considers only the data available for each teacher, none of the 16 teachers would qualify for the bonus awarded teachers in the Chattanooga program. All fall short of the standard either in one of the required subjects (mathematics, reading, and language) or they fail to meet it for three consecutive years (where three years of data is available). Teacher 11 comes the closest to qualifying, with exceptional teaching in two out of the three subjects for two out of three years.
Considering the 16 teachers collectively, there are 123 teacher-by-subject-by-year teacher-effect scores. Only 18 (15 percent) of these scores reach the “exemplary” or “A” level and 13 (11 percent) would be designated as “deficient” and given a grade of “F.”
Discussion
The 16 NBPTS certified teachers for whom TVAAS data is available cannot be considered exceptionally effective in terms of their ability to bring about student achievement. With the exception of the above noted highs and lows, the achievement gains made by their students are no greater than those made by students who had other teachers. None would have qualified for the bonus offered in Chattanooga. Plainly, these findings are distinctly at odds with that which policymakers and the public have been given to understand about the quality of NBPTS certified teachers.
In an Education Week article titled “National Certification Found Valid for Teachers,” NBPTS President Betty Castor said of the UNCG study: “It gives us-parents, elected officials, and policymakers-the absolute highest confidence that national-board-certified teachers are providing students with a high-quality learning experience.”7 Clearly her confidence was misplaced.
Is there some chance that the teacher-effect scores earned by the teachers in this report are misleadingly low?
Inevitably, some of the scores reported in any study are underestimates and some are overestimates. Annual estimates of the impact had by teachers on student achievement can be unstable but the errors in estimation tend to balance out when scores are considered collectively.8
As a practical gauge for judging how many scores are needed to make a reasonable estimate of a given teacher’s performance, consider the number of test scores that enter into a student grade for a semester-long college course. Assuming that a teacher in grades three through eight would have fifteen to twenty students, the estimated annual achievement gains for the teachers in this study are probably based on three or four times as many test scores as the typical college course grade. Stated a bit more technically, gain scores have a larger error component but TVAAS teacher-effect averages compensate by including a larger number of scores.
Because Tennessee seeks to afford teachers a much higher degree of protection against inaccurate assessments than is typically assured by college grading practices, Tennessee’s use of “teacher effect” scores in annual teacher evaluations is statistically very conservative. An individual teacher is classified as above or below average only if his or her three-year rolling average is more than two standard errors above or below the system mean.
Is it necessary for a study of this kind to consider only three-year averages?
Given that the precise classification of individual teachers is not the primary objective of the present analysis, the restrictions appropriate to clinical use of these data are not necessary. It should be noted, however, that none of the teachers for whom three years of data is available have an average of 115 percent, in any subject. Moreover, with the exception of the Social Studies scores of Teacher 1, none of the teachers with less than three years of data would attain the 115 percent criterion without substantially improved second or third years. In other words, it is unlikely that any of the teachers in this study would have been classified as exceptional had 3 years of data been available.
Are the 16 teachers in this study representative of all NBPTS-certified teachers?
In any study, the available sample may misrepresent the population. In other words, it is at least statistically possible that while none of these NBPTS teachers appear to be exceptional, the sixteen thousand or so others for whom teacher-effect scores are unavailable would be so classified.
How plausible is this conclusion? Not very. One would have to presume that the present sample is an anomaly–perhaps a group of teachers who test well but perform poorly in the classroom. In truth, it is more likely that the initial applicants for NBPTS certification would be above-average representatives of their group.
Again, it is useful to consider the data from a practical standpoint: If a grocer bought several cases of premium, hand-wrapped, large apples and found the first box to be 85 percent mediums and smalls, he would certainly have good reason to be skeptical about the rest. Moreover, if the apples cost as much as NBPTS certified teachers, the grocer would be entirely justified in demanding a refund and finding a new supplier.
There is one final point regarding the statistical limitations of this study: Most of the studies that have drawn favorable conclusions regarding NBPTS-certified teachers have had similar limitations. For example, the UNCG study discussed above included only 31 NBPTS-certified teachers. A 1995 study sponsored by the NBPTS compared three NBPTS-certified teachers to three non-certified teachers.9 Plainly, a limited number of subjects was not considered a bar to scientific and practical importance in these earlier studies.
Conclusion
The findings of this study present a serious challenge to NBPTS’s claims regarding its teacher quality standards and certification process. At the very least, they suggest that public expenditures on NBPTS certification and teacher bonuses should be suspended until it can be clearly and independently established that NBPTS certification delivers what it promises.
The number of teachers in this study is small but comparable to the numbers of teachers in the studies on which the NBPTS has based its claims. Significantly, however, this study was independently conducted and it links NBPTS certification to the outcome of greatest interest to policymakers and the public. Although the findings are not definitive, they clearly indicate that the NBPTS standards and certification process–at least as presently constituted–are not serving the teacher quality aims of public policy.
Are NBPTS certified teachers exceptional in some respect other than their ability to improve objectively measured student achievement?
It may be that the teachers in this study are exemplars of the teaching practices idealized by NBPTS, NCATE, and INTASC and that their teacher-effect scores are only average because NBPTS, et al, treat measured student achievement as something less than a top priority.10 To the contrary, Tennessee and most other states treat measured achievement as an unrivaled priority; thus teachers who fail to produce exceptional student achievement would not be considered “exceptional” regardless of whatever else they are able to do. Contrary to the view expressed in the UNCG study, parents, policymakers, and the public regard gains in objectively measured student achievement as indispensable to good teaching.
Can efforts to correct and revise the NBPTS initiative succeed?
The discrepancy between the National Board’s portrayal of its standards and the findings of this study may reveal critical flaw in the conceptual scheme on which NBPTS is founded. Medicine, law, and other professions have advanced certifications. Members of those professions, however, work in a marketplace where cost and quality considerations influence consumer behavior and revenue. By contrast, teachers work in a marketplace where the cost of added certificates, licenses, and credentials creates no competitive disadvantage even if they are meaningless. Under such conditions, the consumer’s interest is virtually unprotected.
May 1, 2002
Endnotes
1 John E. Stone, George K. Cunningham, and Donald B. Crawford, Improving Teacher Quality in Oklahoma: A Closer Look, (Oklahoma City, OK: Oklahoma Council of Public Affairs), Policy Paper 01-7 (October 2001) available: http://www.ocpathink.org/sites/ocpathink/uploads/documents/policypaper01-7.pdf.
2 Lloyd Bond, Tracy Smith, Wanda K. Baker, and John Hattie, The Certification System of the National Board for Professional Teaching Standards: A Construct and Consequential Validity Study, Center for Educational Research and Evaluation, University of North Carolina at Greensboro, September 2000 (http://www.nbpts.org/UserFiles/File/validity_1_-_UNC_Greebsboro_D_-_Bond.pdf).
3 J. E. Stone, “Value Added Assessment: An Accountability Revolution” in Marci Kanstoroom and Chester E. Finn, Jr. (Eds.), Better Teachers, Better Schools. (Washington, DC: Thomas B. Fordham Foundation, 1999), available: http://www.education-consumers.com/articles/value_added_assessment.shtm
4 For more on the use of student achievement effects as a measure of teacher effectiveness, see Education Consumers ClearingHouse, “Teacher Evaluation and Student Achievement by James H. Stronge & Pamela D. Tucker,” From a Consumer Perspective, Briefings on Educational Research from the Education Consumers Consultants Network, 1, no. 3 (March 2001), available: http://www.education-consumers.com/briefs/march2001.shtm
5 Tennessee Department of Education, Evaluation & Assessment Division, Benjamin Brown, Ph.D., Executive Director (Ben.Brown@state.tn.us). By law, reports pertaining to individual teachers are available only to the teacher and his or her school system. In order to preserve teacher anonymity, the data on which the present report is based was furnished without teacher and school system identifiers.
6 See the Tennessee Grade Scale: http://www.k-12.state.tn.us/rptcrd01/gradescale.htm; Also, under terms of a recently instituted program, teachers in selected Chattanooga, Tennessee schools will earn annual bonuses of $5000/year for producing achievement gains equal to 115% of the school system’s annual achievement increase in 3 subjects–math, reading, and language: See Duane W. Gang, “Teachers at At-risk Schools may get Bonuses,” Chattanooga Times-Free Press (March 13, 2002). Also see, personal communication, (April 10, 2002), Jack Murrah, Lyndhurst Foundation, (jmurrah@lyndhurstfoundation.org).
7 Julie Blair, “National Certification Found Valid for Teachers”, Education Week, October 25, 2000.
8 See Thomas J. Kane, Douglas O. Staiger, and Jeffrey Geppert, “Randomly Accountable,” Education Next, 2, no. 1, (Spring 2002): 57-61, available: https://www.educationnext.org/randomly-accountable/
9 J. Hattie, J. Clinton, M. Thompson, & H. Schmidt-Davis, Identifying Highly Accomplished teachers: A validation study, A Research Report of the National Board for Professional Teaching Standards Technical Analysis Group (Greensboro, North Carolina: Center for Educational Research and Evaluation, University of North Carolina-Greensboro, 1995).
10 Stone, Cunningham, and Crawford, Improving Teacher Quality in Oklahoma
Appendix A
| Teacher-Effect Scores for TN NBPTS-Certified Teachers | |||||||
| Year | Grade | Mathematics | Reading | Language | Social Studies | Science | |
| Teacher1 | 1999 | 6 | 8.6 ( 2.0) [239] | 8.6 ( 2.1) [110] | 21.0 ( 2.8) [140] | ||
| System1 | 1999 | 6 | 3.6 ( 0.4) | 7.8 ( 0.4) | 15.0 ( 0.4) | ||
| Teacher1 | 2000 | 6 | 5.2 ( 2.1) [ 82] | 24.0 ( 3.4) [364] | |||
| System1 | 2000 | 6 | 6.3 ( 0.4) | 6.6 ( 0.4) | |||
| Teacher2 | 1999 | 7 | 16.2 ( 4.1) [109] [1] | ||||
| System2 | 1999 | 7 | 14.9 ( 1.5) | ||||
| Teacher3 | 2000 | 8 | 13.3 ( 4.1) [ 76] | ||||
| System3 | 2000 | 8 | 17.5 ( 1.6) | ||||
| Teacher4 | 1999 | 6 | 27.3 ( 3.7) [100] | 6.9 ( 2.6) [ 98] | 8.2 ( 2.9) [ 76] | ||
| System4 | 1999 | 6 | 27.2 ( 1.6) | 7.0 ( 1.6) | 10.8 ( 1.6) | ||
| Teacher4 | 2000 | 6 | 26.9 ( 3.4) [106] | 10.1 ( 2.8) [112] | 14.0 ( 2.8) [ 91] | ||
| System4 | 2000 | 6 | 25.4 ( 1.6) | 9.0 ( 1.5) | 15.3 ( 1.5) | ||
| Teacher5 | 1998 | 6 | 19.0 ( 2.4) [104] | 14.0 ( 2.6) [113] | 6.0 ( 2.2) [ 79] | 16.4 ( 2.3) [ 97] | |
| System5 | 1998 | 6 | 18.2 ( 1.3) | 12.4 ( 1.3) | 7.6 ( 1.4) | 16.9 ( 1.5) | |
| Teacher6 | 1998 | 7 | 10.7 ( 2.5) [ 93] | 15.1 ( 2.8) [ 98] | |||
| System6 | 1998 | 7 | 11.5 ( 2.5) | 15.4 ( 2.8) | |||
| Teacher6 | 1999 | 8 | 14.1 ( 2.3) [107] | 12.5 ( 2.6) [102] | |||
| System6 | 1999 | 8 | 13.2 ( 2.6) | 12.3 ( 2.9) | |||
| Teacher6 | 2000 | 8 | 14.4 ( 2.4) [100] | 23.0 ( 2.7) [126] | |||
| System6 | 2000 | 8 | 14.4 ( 2.7) | 18.3 ( 3.0) | |||
| Teacher7 | 2000 | 4 | 33.7 ( 3.9) [135] | 13.6 ( 3.0) [115] | 10.3 ( 2.7) [ 84] | 13.7 ( 3.1) [109] | 18.9 ( 2.9) [104] |
| System7 | 2000 | 4 | 25.0 ( 0.8) | 11.8 ( 0.8) | 12.3 ( 0.8) | 12.6 ( 1.0) | 18.1 ( 0.9) |
| Teacher7 | 1998 | 5 | 19.5 ( 2.8) [104] | 13.1 ( 2.2) [116] | 10.4 ( 2.7) [127] | 15.8 ( 2.3) [103] | 13.1 ( 2.4) [125] |
| System7 | 1998 | 5 | 18.8 ( 0.8) | 11.3 ( 0.8) | 8.2 ( 0.8) | 15.3 ( 0.9) | 10.5 ( 0.9) |
| Teacher7 | 1999 | 5 | 14.4 ( 3.2) [ 75] | 14.8 ( 2.2) [110] | 10.2 ( 2.6) [110] | 13.5 ( 2.5) [120] | 19.4 ( 2.7) [100] |
| System7 | 1999 | 5 | 19.2 ( 0.8) | 13.4 ( 0.9) | 9.3 ( 0.9) | 11.2 ( 0.9) | 19.4 ( 0.9) |
| Teacher8 | 2000 | 4 | 29.4 ( 4.7) [106] | 10.0 ( 3.3) [ 93] | 9.4 ( 3.9) [ 67] | 14.7 ( 3.8) [100] | 19.6 ( 3.9) [101] |
| System8 | 2000 | 4 | 27.6 ( 1.6) | 10.7 ( 1.6) | 14.1 ( 1.5) | 14.7 ( 1.8) | 19.5 ( 1.7) |
| Teacher9 | 1998 | 7 | 2.5 ( 2.3) [ 76] | ||||
| System9 | 1998 | 7 | 3.3 ( 1.5) | ||||
| Teacher9 | 1999 | 7 | 4.3 ( 2.2) [ 93] | ||||
| System9 | 1999 | 7 | 4.6 ( 1.6) | ||||
| Teacher9 | 1999 | 8 | 14.7 ( 3.4) [104] | ||||
| System9 | 1999 | 8 | 14.2 ( 1.7) | ||||
| Teacher10 | 1998 | 4 | 28.4 ( 4.8) [110] | 21.2 ( 2.7) [109] | 21.9 ( 3.5) [113] | 22.2 ( 2.9) [ 94] | 25.9 ( 3.7) [100] |
| System10 | 1998 | 4 | 25.8 ( 1.5) | 19.4 ( 1.5) | 19.3 ( 1.5) | 23.7 ( 1.7) | 25.9 ( 1.7) |
| Teacher10 | 1999 | 4 | 34.8 ( 3.6) [108] | 14.3 ( 2.6) [103] | 15.1 ( 3.2) [ 93] | 21.3 ( 3.1) [102] | 23.1 ( 3.3) [101] |
| System10 | 1999 | 4 | 32.3 ( 1.6) | 13.9 ( 1.6) | 16.3 ( 1.5) | 20.9 ( 1.9) | 22.8 ( 1.7) |
| Teacher10 | 1998 | 5 | 27.2 ( 4.6) [ 97] | 15.2 ( 2.7) [ 98] | 14.8 ( 3.4) [110] | 21.4 ( 2.9) [101] | 16.7 ( 3.2) [103] |
| System10 | 1998 | 5 | 27.9 ( 1.5) | 15.5 ( 1.5) | 13.5 ( 1.5) | 21.2 ( 1.6) | 16.2 ( 1.7) |
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Each teacher effect score (cell) is accompanied by a standard error of measurement ( in parenthesis) and (in brackets) the percentage of annual achievement growth represented by the teacher effect score. The percentage of annual growth is the ratio of the teacher effect score to the average annual achievement growth for the school system (shown in the following row) times 100.