Evaluating the Research Validity and Reliability... How High are the Stakes? By what standards is the TAAS test considered valid? How is the reliability of the test checked? How often are the standards of validity and reliability examined? How are the TAAS test scores used?

By what standards is the TAAS test considered valid?

Content validity refers to the extent that a test reflects and measures attainment of the objectives or content to be learned. TEA asserts that the TAAS has high content validity in that the test is aligned directly with the Texas curriculum, TEKS (TEA). According to the Educator’s Guide to TEKS-Based Assessment: Exit Level TAAS 1999-2000, the alignment is as follows:

 

Reading and Writing	Grade 8 English Language Arts English I English II (page 6 and 22)
Mathematics	Grade 8 Mathematics Algebra I (page 12)

The TAAS, a criterion-referenced test, is based on an extensive definition of the content it assesses. While many standardized tests are designed to recognize excellence, the TAAS was designed to merely assess minimum skills. Because of this, TEA asserts that for TAAS Content Validity and Construct Validity are indistinguishable. “The construct tested is the academic content required by the statewide curriculum” (TEA, page 47). Therefore, “The construct validity is grounded in the content validity of the test" (TEA, page 47).

The state contends that the primary evidence for the validity of the TAAS and end-of-course tests lies in the content being measured. Since the tests assess the Texas state-mandated curriculum, which is required to be taught to all students, the tests are not more or less valid for use with one sub-population over another sub-population In other words, because the TAAS and end-of-course tests are measuring what is required to be taught to all students and are given under the same standardized conditions to all students, the tests have the same validity for all students (TEA, page 48).

Criterion-related validity indicates the relationship between performance on the test and performance on some other measure. TEA has conducted many correlation studies on criterion-elated validity with regards to the TAAS test and assorted other measurement instruments. These tests are extensively described in their web site, but the only results that they site are of a study in 1992-1993 that shows a moderately positive correlation (0.32) (TEA). The referenced Student Performance Results 1992-1993, 1993-1994, 1994-1995, 1995-1996, 1996-1997, 1997-1998, and 1998-1999 are not available through the TEA web site.

How is the reliability of the test checked?

“TAAS and end-of-course tests provide observed scores, which serve as a proxy for direct measurement of achievement levels. Scores always contain some measure of error. Test reliability quantifies this error” (TEA, page 42). The correlation between the observed scores and the true scores would be the reliability of the test. “TAAS reliability's is based on internal consistency measures, specifically on Kuder-Richardson Formula 20 (KR20)” (TEA, page 42). All of the internal consistency computations are presented in Appendix 7 of the Technical Digest; this Appendix is not available online.

The internal consistency or reliability of the TAAS test as measured using Kuder-Richardson Formula 20 (KR-20) should be determined at each sitting of the test. At this time, one can only surmise that this is what TEA does, as they release a Technical Digest on the assessment program – TAAS and end-of-course – each year. However, as the appendices with the statistical information in them are not readily available this was not confirmed.

According to TEA, written composition score reliability is ensured with several methods. For one, each essay receives two or more readings. Secondly, daily checks using what is called validity packets are conducted to test for reader drift and reader consistency. If reader drift is detected all affected papers will be rescored. According to TEA, the scoring for the exit TAAS written composition was:

A district may appeal any failing grade on a composition and the composition in question will receive an individual analysis. According to TEA, “Score appeals yielded score changes on only .004% of the more than 827,000 papers scored during the 1999 school year” (TEA, page 45).

Another issue is that of "equating," which is the consistency in difficulty in the test from administration to administration of the total test and at the objective level (TEA, page 49). TEA checks for both pre-equating and post-equating.

Pre-equating is done using the following process:

1. New questions are chosen “from a pool of items that have been equated to either the original form [spring 1994] on which the State Board of Education established the passing standard or other forms linked to this original anchor form” (TEA, page 45).
2. The new test is then constructed by TEA staff, psychometricians [1 : a person (as a clinical psychologist) who is skilled in the administration and interpretation of objective psychological tests 2 : a psychologist who devises, constructs, and standardizes psychometric tests Merriam-Webster Collegiate Dictionary http://www.m-w.com/dictionary.htm] from National Computer Systems, and test writing experts from Harcourt Educational Measurement and Beck Evaluation and Testing Associates (BETA) Inc.
3. The test is then reviewed by TEA.
4. Each new item is field-tested. Each new test question “is embedded in a test booklet among the base-test items" (TEA, page 50). The example given in the Technical Digest for the Academic Year 1998-1999 shows that while the base-test items stay the same, different tests may have different field-test items. These are not included in the student’s score.

Post-equating is “a comparison of the item difficulties…to see whether the form constructed from the pre-equating data is the same difficulty as the original form.” According to TEA this has been done, at every administration of TAAS since 1990 (TEA, page 50). Additionally, “only on rare occasions do the post-equating data not agree with the pre-equating data as designed” (TEA, page 50). There is a comprehensive explanation of the mathematics involved in post-equating on pages 51 through 53 of the Texas Student Assessment Program Technical Digest for the Academic Year 1998-1999.

How often are the standards of validity and reliability examined?

As was stated above, inter-scorer reliability on the writing portion of the test is checked daily using what the State calls validity packets.

The TAAS was shifted from testing the former statewide curriculum, Essential Elements, to the TEKS in the 1998-1999 school year. As a part of this transition “the process of aligning TAAS to the new curriculum was carefully approached and included numerous committees of Texas educators.” (TEA, page 46). After this initial alignment for content validity, the State uses the process of pre and post-equating on a continuing basis to ensure that each subsequent test is a valid as all previous tests.

The State has performed some testing of concurrent validity. These tests are not performed for every test every year. To date the TEA has only studied the concurrent validity of five of the TAAS tests.

 

Test/Course Grade Comparison	School Year(s) Done
Exit Level TAAS Mathematics scale scores compared with the course grades for several different mathematics courses	1992-1993 1993-1994
Exit Level TAAS Reading and Writing compared with student performance – pass/fail – in English II	1994-1995
Grade 8 TAAS Mathematics compared with course grades for Grade 8 Mathematics	1995-1996
Grade 8 TAAS Social Studies Test compared with course grades for Grade 8 Social Studies	1996-1997
Grade 3 TAAS Reading Test compared with student performance – pass/fail – in Grade 3 reading “course”	1997-1998

(TEA, page 46-47)

How are the TAAS test scores used?

How are the TAAS test scores used by students, parents, and families?

According to Cheryl Wright, Curriculum Specialist with TEA, students can use their TAAS score to see where they are academically and what skills they need to work on in order to achieve mastery on the test (personal conversation, November 4, 2000). It is doubtful that many students use their test scores this way. More often than not, if students look at their TAAS grades they use it to compare themselves with other students. Who did better than whom? Who is the 'smartest'? Who is the 'doofus'?

TAAS testing can affect students in two ways. Some students are barely affected by the process. The three days when the testing occurs are merely days when they do not have to go to class. While the exit level TAAS is used to determine whether or not students will graduate from high school, the administration of exit level TAAS in Grade 10 blunts the stress many students might feel. This is because graduation is so far in the future.

Other students are keenly aware of the pressure on them to pass the grade level tests (Grades 3 though 8). This is because their teachers have told them all semester how important it is that they do well on TAAS. While the student may be unaware of why there is such emphasis on TAAS, they do know that TAAS is seen as the be-all and end-all for most of the teachers and other adults in the school. High school students who do not pass the exit level TAAS on the first sitting often experience ever increasing stress each time they sit for the test.

This push for students to achieve mastery is one that comes to consume not only the student's academic life, but also one that affects the whole family as a whole. According to the parent of a student in the Pearland Independent School District the elementary school that the child attended sent home directions every year to parents on how to prepare the child for the TAAS testing days. The letter advised parents to:

• Put the child to bed between 8 and 9 p.m.
• Make sure that the child had a well-balanced meal for supper.
• Get the child up early enough in the morning so that they are not rushed.
• Make sure that the child has a well-balanced, preferably warm, breakfast. 
                (B. Ardoin, personal communication, December 5, 2000)

Parents can use the TAAS score as a guide to tell them how their child is progressing from year to year in mathematics and reading using the Texas Learning Index (TLI).

(indent the following) The TLI is a score that describes how far a student's performance is above or below the passing standard. TLI is provided for both the TAAS reading and mathematics tests at grades 3 through 8 and at the exit level….The TLI was developed to allow students, parents, and schools both to relate student performance to passing standard and to compare student performance from year to year." (TEA, Texas Learning Index)

Parents can use the released Scoring Guides to determine how their child did on the writing portion of the TAAS which is administered in Grades 4 and 8, and at the exit level. The Scoring Guides for both the Spring 1999 and Spring 2000 TAAS are available online at http://www.tea.state.tx.us/student.assessment/guides/score.htm.

Parents can also access information about how particular schools, districts, and regions performed on the TAAS to help them determine how their child's school compares to other schools in the state. This can be particularly useful when parents are moving. Parents can look at district and campus TAAS results, available online for Grades 3 through 8 and at the Exit Level. To this end, parents can refer to http://www.tea.state.tx.us/student.assessment/results/swresult/campdist/campdist.htm, which currently has Spring 2000 TAAS results. Additionally, the Academic Excellence Indicator System (AEIS) provides a comprehensive listing of TAAS results and other statistical information about individual schools, school districts, regions of the state. AEIS is accessible online at http://www.tea.state.tx.us/perfreport/aeis/.

It is unclear how many parents use their child's TAAS scores in the ways described above. Anecdotal evidence suggests that most parents use TAAS scores for bragging rights if their child did well. Often parents have come to see TAAS as testing such low-level skills that their child's TAAS scores matter only if they are lower than the parent's expectations.

How are the TAAS test scores used by teachers, administrator, school districts and TEA?

TEA rates schools and districts based on the following criteria:

Schools or districts that are ranked as "low performing" for two years are at risk of being taken over by TEA as stipulated in the Texas Education Code § 13.054.

Because of this, school districts are very concerned with the TAAS scores district wide. The higher the district is ranked the easier it becomes to recruit new teachers. As mentioned above, higher rankings attract new homeowners and business to the district, which increases the tax base. In addition to this additional revenue, the higher the district's ranking the better the community feels about how their tax dollars are being spent.

Principals are under pressure to meet the standards set by the district. According to Linda Ware, Principal of Hunter's Glen Elementary School, TAAS drives the curriculum because the goal of every school in the state of Texas is to become "Exemplary" (personal interview, November 3., 2000). Principals have a vested interest in the ranking of their school. The state provides a financial incentive in form of the Principal Performance Incentive Program, through which a principal may receive up to $5,000. The school itself may also receive monies through the Texas Successful Schools Award System, which awarded a maximum of $5,000 to 2,300 schools in 1997 - the last available numbers (TEA, Incentive Grant Program).

The push to maintain or increase TAAS scores starts in the fall when administrators receive the previous spring's TAAS scores from TEA. At that time, according to Linda Ware, the school principal along with the Dean of Instruction (DOI) disaggregates and examines the TAAS passing rate of the school. Once this is completed, strategies for the coming year can be planned (personal interview, November 3, 2000).

A popular strategy schools use to improve TAAS scores is in-service training on methods used by other districts that have consistently received "Exemplary" status. The most popular model, according to Charles Polk, former principal of Willowridge High School, is the from Brazos Port Independent School District (personal conversation, December 7, 2000).

Since there is such a strong emphasis on obtaining "Exemplary" status, teachers are encouraged to structure their curriculums around spelling, sentence structure, math formulas, and other learning tools that have been used on past tests. These tools are used to familiarize each student with the format of the test, so that errors can be avoided while taking the test. The goal is for students to achieve mastery on all portions of the test.

According to Linda Ware, the Dean of Instruction along with selected teachers is given the task of structuring test preparation activities. A calendar is set up for the fall and spring semesters for tutorials in the tested areas and Mock TAAS tests. Tutorials are offered after school and on Saturdays. To ensure that students attend Saturday tutorials, some districts provide food and transportation. After students have taken the Mock TAAS, the student's scores are broken down by competency to see which skills need additional attention to assure mastery on the real test. This cycle continues each year.

The main criticism has been, from both parents and teachers, that so much emphasis has been placed on TAAS scores that real learning does not take place. These complaints received national attention during the Presidential Campaign of Texas Governor George W. Bush when, on September 10, 2000, the CBS news magazine 60 Minutes ran a story titled "Do Good Scores Mean Learning?" which featured several teachers discussing their displeasure with teaching to TAAS. A synopsis of this story is available online at http://cbsnews.com/now/story/0,1597,231464-412,00.shtml.

Return to other questions on Evaluating the Research