Monday, February 24, 2014

Truth and Literacy at UNC

Taunts from the home crowd are to be expected in college basketball, but on January 20th the taunts from Virginia’s student section were more degrading than usual. UNC was the visiting team that night, and anytime a UNC player stepped to the foul line to shoot his free throws, his concentration was challenged by Virginia fans singing the ABCs and reading children’s books aloud. The implication, of course, was that UNC players are illiterate.

Such insults came less than two weeks after CNN released a report on college athletes’ reading levels, highlighting the claims of a former UNC reading specialist who presented data on the athletes with whom she worked at UNC. Mary Willingham, the former reading specialist, claimed that 60% of the 183 athletes she sampled between 2004 and 2012 read between a fourth and eighth grade reading level, and that another 10% were functionally illiterate.

Over the past month or so, I have watched with dismay as the local and national media have created a spectacular melodrama out of what should have been a brief and rational debate over verifiable claims. This melodrama has been particularly poignant for me because I am the current reading and writing specialist for student-athletes at UNC. As such, I work every day with the student-athletes whom this controversy has concerned.

Although questions of truth have been latent in the media’s reporting, the media have been unconcerned with such questions. For higher education professionals, however, such questions are the questions that matter most. Truth and discovery, as one UNC professor has argued, form the foundation on which both education and research are built. Therefore, the truth about UNC athletes’ literacy rates should be of the highest concern for UNC faculty, administrators, athletics staff, and all stakeholders in college athletics across the country.

Whither Methodology?

Since the CNN report, a series of claims and counter-claims have been made. On January 17th, in a presentation to the Faculty Council, UNC Provost Jim Dean challenged Willingham’s findings, claiming she had misinterpreted her data. Willingham had used results from the Scholastic Abilities Test for Adults (SATA) (not to be confused with the SAT) to assess athletes’ reading levels, but according to the Provost, she used only the reading vocabulary subtest. Citing the SATA manual, he explained that the vocabulary subtest is not sufficient to determine overall reading performance. Furthermore, according to the Provost, Willingham apparently confused standard scores with grade equivalents, leading to a major error in interpretation.

Willingham was not in attendance but defended her findings to the media. “I’m telling the truth,” she asserted, “My data is 100% correct.” In the N&O the following day, Willingham explained that she used more than just a vocabulary test. She was quoted saying, “It was a combination of the SATA reading and writing (tests) and the SAT and ACT scores.”

Absent from the media’s coverage of this story (except for one article published two days ago) has been the insight of experts in educational assessment and literacy. As the media have presented the story, the dispute seems to be over two different interpretations of data and will remain unresolved without an independent review. Yet Willingham’s simple statement in defense of her findings revealed enough about her methodology to provide anyone who understands educational assessment all they need to know to determine the accuracy of her findings.

First, as the Provost pointed out, the SATA manual states that overall reading performance should be assessed using both the reading vocabulary and reading comprehension subtests, neither of which involve writing. Both subtests consist exclusively of selected-response items such as multiple-choice questions. The SATA does include a writing mechanics and a writing composition subtest, but those are used to assess writing—not reading. The instructions and explanations for assessing reading performance are found in the first chapter of the SATA manual. Willingham stated she used both the reading and writing tests to determine reading levels, but that is clearly contrary to what the SATA manual instructs.

Second, nowhere in the SATA manual are test administrators instructed to combine results from SATA subtests with ACT or SAT scores. Most educators familiar with the basics of educational assessment could explain why combining scores from two distinct tests to calculate students’ grade levels—called grade equivalents by educators—does not make sense. Every test that can determine grade equivalents goes through an extensive validation process specific to that test, and every test determines grade equivalents slightly differently from other tests. Thus, accurate interpretation of measures like grade equivalents is dependent on following the exact instructions from the test makers. That is why a competent educator would never combine the results from two tests to calculate grade equivalents.

Furthermore, the SATA subtests on reading and writing fall under a different category of tests than the ACT and SAT. The reading and writing subtests are achievement tests, whereas the ACT and SAT are aptitude tests. Achievement tests, as the very first paragraph of the SATA manual explains, measure students’ knowledge of content matter (e.g., social studies) or their abilities to perform basic skills (e.g., reading). Aptitude tests, on the other hand, measure general cognitive abilities, such as verbal and quantitative reasoning, which are expected to predict academic success. When screening a student for potential learning disabilities, an educator will look for discrepancies between the student’s aptitude and achievement results, but those results would never be combined to determine a grade equivalent. They cannot be combined, because achievement and aptitude are different assessment constructs.

Whatever methodology Willingham used to come to her conclusions was certainly not found in the SATA manual, nor was it based on sound educational assessment principles. Therefore, as the Provost stated, her findings are virtually meaningless. Willingham’s methodology has rendered her data profoundly incorrect.

Unfair Representation

Willingham has also cited two anecdotes to support her claims, but I believe her anecdotes are unfair representations of UNC athletes and reflect a reductionist understanding of academic preparedness. In the CNN report, she told stories of an athlete who could not sound out the word Wisconsin and of another athlete whose goal was to improve his reading enough to read newspaper articles about himself. By citing those anecdotes to demonstrate some athletes’ lack of academic preparedness, without providing any other information about those athletes’ academic abilities, she has ignored the multi-faceted nature of academic preparedness, reducing it to a single aspect—reading fluency.

In my nearly four years as a reading and writing specialist, at two universities, I have had only three student-athletes whose fluency was as limited as those Willingham described. All three had been diagnosed with a reading disability, and I highly suspect the two athletes to which Willingham was referring also had a reading disability. What is important to understand about reading disabilities is that they do not indicate overall impaired cognitive functioning. Reading disabilities are disorders that often involve difficulty decoding certain letters or performing other fluency-related tasks, but those with a reading disability can nonetheless engage in other complex cognitive tasks. Undoubtedly, the majority of people with reading disabilities are as intellectually capable as anyone reading this essay.

In fact, those three student-athletes all developed extraordinary capacities for retrieving and retaining information in ways other than reading. One student compensated for his struggle with reading by developing the ability to remember almost everything he heard during a lecture. Another student became very resourceful on the internet, finding lectures and videos he could listen to or watch through iTunes U, YouTube, and other online sources. Additionally, all three used technological accommodations (e.g., audio books) to help them read their assigned texts. Moreover, all three have either graduated or are on track to graduate soon.

My point is not that all students with reading disabilities, or even the majority, are otherwise prepared for college. My point is that academic preparedness is more than reading fluency, and that limited fluency due to a disability can be overcome. Those three student-athletes with reading disabilities were capable of succeeding because they were highly prepared in other ways. Academic preparedness, like merit, includes standard achievement and aptitude factors, but it also includes non-cognitive factors such as motivation, resilience, and resourcefulness. The extent to which those three students displayed such qualities was inspiring, and the three of them are the kinds of success stories a reading specialist should be telling.

A False Analogy

Willingham further misrepresented UNC athletes with another statement she made during her interview with CNN, which has been repeated in other articles since. Commenting on some UNC athletes’ inability to succeed at UNC, she stated, “We may as well just go over to Glenwood Elementary right up the street and just let all the fourth graders in here, the third graders in here.”

Such flippancy will no doubt be appalling to adolescent and adult literacy educators everywhere. Not only does she reduce academic preparedness to reading fluency, she compares UNC athletes—young adults—to elementary students, supplying Virginia fans with the substance of their taunts. Essentially, her statement equates reading fluency with cognitive or intellectual ability. According to her logic, if someone can read at only a third or fourth grade level, he is on the same intellectual level as a third or fourth grader.

(In fact, Willingham’s offensive analogy is what initially led me to start questioning her findings. When the CNN report was first published, I read it, but I did not watch the interview. After just reading it, I was surprised by the statistics, because my experience does not confirm Willingham’s findings, but I gave her the benefit of the doubt and was supportive of the truth coming out. However, a couple days later, I watched the interview, and I was so appalled I started examining her claims more critically.)

A basic axiom of adolescent and adult literacy is that reading ability is not the same as cognitive or intellectual ability. Visit any adult literacy center and ask the instructors what reading materials they use to teach basic reading to adults. They will tell you, most assuredly, they do not use children’s books. That is because even when adults read at only an elementary level, their overall intellectual abilities are nonetheless likely fully developed. They work in adult jobs and deal with adult situations, and have often developed sophisticated strategies to compensate for their lack of reading ability. The same is true of high school and college students who struggle with reading. They are not the equivalent of elementary students, and a reading specialist’s publicly suggesting such is irresponsible.

Research Ethics

Willingham’s findings are not the only verifiable claims that have been obfuscated and unchallenged by the media. The other claims involve the classification and status of her research with the Office of Human Research Ethics.

Several media outlets reported that her approval to conduct her research was suspended by UNC on January 16th. The next week Dr. Daniel K. Nelson, Director of the Office of Human Research Ethics at UNC, released a statement explaining that Willingham never actually had approval to conduct research on human subjects. According to Dr. Nelson, when Willingham submitted her research proposal to his office, she indicated that she would only be analyzing de-identified, secondary data (i.e., data without the names of the athletes attached). Therefore, his office had determined she would not need approval and oversight from the Institutional Review Board (IRB). However, Dr. Nelson explained, since her findings were reported by CNN, several sources, including her own statements, have confirmed she did have access to the names and academic records of the athletes she was studying. Therefore, his office took action to halt her research until she obtains the proper approval and oversight to conduct research on human subjects.

Willingham has countered by claiming Dr. Nelson’s office knew she had the athletes’ names and academic records from the beginning. In a January 17th article published in The Daily Tar Heel, Willingham was quoted saying, “How would I do the research if I didn’t have the names? The study included how they were doing in school, their GPA. From what I understand, the primary investigator can have access to that and you wouldn’t share that with the public because that would be unethical.” In the same article, she suggested the university was trying to “squash” her research on a “technicality.” In an article two weeks later, she accused UNC of conspiring against her. “I think they’re all in bed together, that’s been the problem all along,” she said, “They all report to each other, there’s no independent agency—they report to the provost, and he’s a void.”

Before addressing the truth of the matter, I want to explain why the matter is not just a “technicality” as Willingham claims. On the contrary, the matter is one of ethics and of federal law.

One of the most fundamental ethical principles of human subjects research is that researchers must obtain consent from the people (i.e., human subjects) being studied before carrying out the research. Federal law requires every university to establish an IRB to ensure researchers are obtaining consent and following other important ethical guidelines when conducting human subjects research. As a researcher learns when completing the prerequisite ethics training to do human subjects research, history provides all too many examples of unethical research on human subjects. Thus, the IRB is established to prevent ethical violations from happening in the future. When research does not involve human subjects, the researchers do not need approval and oversight from the IRB because the potential for unethical conduct is minimal.

Willingham’s research, by her own admission, involved human subjects. Willingham stated one of the purposes of her research was to monitor how the athletes performed in school (their GPAs), which would require accessing their academic records. In her testimony for the O’Bannon case, she even reported aggregate grades for a “cohort” of 17 football players (how she established this cohort is unclear).

Such research would have required consent from the athletes for two reasons.

First, it would have required consent because, as already explained, consent is a fundamental ethical principle of all research on human subjects. Second, her research would have required consent because students’ records are protected by federal law. The specific law, FERPA, states that university employees may only access students’ records without students’ consent when the purpose is for “legitimate educational interests.” For example, when an advisor is helping a student plan a course of study, the advisor has a legitimate educational interest in the student’s records because the advisor needs information from those records to effectively advise. Although the law allows each university to determine the criteria for legitimate educational interests, my understanding is that research interests do not typically fall under legitimate educational interests.

In other words, just having the ability to access students’ records does not give one the right to do so. One must either have consent from the students or a legitimate educational interest, as defined by the university.

That is not just some “technicality.”

The question is whether Willingham accurately completed the research application for the Office of Human Research Ethics, indicating she would be accessing students’ records and explaining how she would obtain consent to do so. UNC claims Willingham never stated she would have access to the names and academic records of the athletes; she claims the opposite. Although the media have allowed the issue to seem like a gridlock, the truth could be established if Willingham would publicly release her completed research application.

At UNC, when a researcher submits a research application to the Office of Human Research Ethics, the researcher must answer a series of questions that helps the office determine whether the research requires IRB approval and oversight. (Click here to see screen shots of some of those questions from an application I recently submitted.) Several of the questions are especially relevant to Willingham’s research:
  1. On the very first page, the researcher is asked to state the purpose, participants, and methods of the proposed research. (Willingham should have stated under the methods question that she would be accessing academic records.) 
  2. On a later page, the researcher is asked, “Will you be using identifiable private information about a living individual collected through means other than direct interaction? This would include data, records or biological specimens . . . .” (Willingham should have answered “Yes.”) 
  3. The researcher is then asked to “Check which of the following identifiers you already have or will be receiving [regarding the subjects you are studying] . . . .” A list of identifiers is provided, and the first is “Names.” (Willingham should have selected “Names.”) 
  4. On one of the last pages, the researcher is asked, “What existing records, data or human biological specimens will you be using?” The fourth response is “Student records,” and it is accompanied by this note: “You will need to satisfy FERPA requirements.” (Willingham should have selected “Student records.”) 
  5. When the research involves human subjects, the researcher is also asked to submit the consent form he or she will be using and explain how the form will be distributed to the subjects. (Willingham should have attached the consent form.) 
After the researcher has submitted her or his responses, the completed application is saved on the website, in .pdf form, for the researcher’s records.

As I already suggested, Willingham could publicly release her completed research application and thereby dispel any doubt about whether she indicated to the Office of Human Research Ethics that she would have the names of the athletes and be accessing their academic records. If she answered the above questions accordingly and supplied a consent form, we have no reason to question the ethics of her research. Rather, if she submitted the application properly, we should be questioning why the Office of Human Research Ethics classified her research as secondary data analysis. If she did provide the necessary documentation, was the Office of Human Research Ethics negligent in their duty to evaluate her application? Or did the university, as Willingham suggested, conspire to “squash” her research by putting pressure on the Office of Human Research Ethics? Such questions would need to be asked.

However, if she did not answer the above questions accurately, we should be questioning whether she actually obtained consent from the 183 athletes in her study and what the implications are if she did not. If he did not obtain consent, can she otherwise demonstrate a legitimate educational interest? If she cannot demonstrate legitimate educational interest, did she violate FERPA? If she violated FERPA, is her testimony in the O’Bannon case still permissible? Such questions would need to be asked.

Again, the question of what Willingham submitted to the Office of Human Research Ethics could be settled by her publicly releasing her completed research application (including the consent form). I would even make space available on my blog for her to make her application public.

The issue at hand is not one of just some technicality. Until the public can view her completed research application and the consent form she used for her study, we are left asking questions—questions with verifiable answers. Anyone with an interest in this story should be demanding those answers.

Improving Admissions

Notwithstanding Willingham’s inaccurate findings and the unresolved questions about the classification and status of her research, the university is not without fault. UNC has in the past admitted athletes who were considerably underprepared and incapable of succeeding at UNC. I have no doubt the past problems in the African and African American Studies Department were at least partly attributable to admitting athletes who could not succeed at UNC.

However, that does not mean those athletes’ reading abilities were as low as Willingham claims. The difference between an inability to read and write sufficiently for UNC academics and, on the other hand, reading and writing at an elementary level or lower is significant. Overall, UNC is a rigorous university, with high standards of academic performance. Succeeding at UNC requires not only reading, writing, and study skills but also high levels of motivation and resilience, the combination of which the average student graduating high school today does not have. Many students who succeed elsewhere would not be able to succeed at UNC. Therefore, referring to some athletes’ inability to succeed at UNC as evidence they read at an elementary level or lower would be unfair and untrue.

Moreover, the university has taken noteworthy steps to improve the admissions process for athletes. After I sent an email to Chancellor Folt last summer expressing my concern about admissions for athletes, the Vice Provost for Admissions met with me to explain what improvements his office had recently made. Specifically, the admissions office developed a program that can accurately predict a potential UNC athlete’s first-year GPA. Using that program, the university will no longer admit potential athletes whose first-year GPA is predicted to be less than 2.0.

Furthermore, with support from the athletics director and coaches, the admissions office has been making an effort to reduce the number of first-year athletes whose predicted GPA is less than 2.3. This year that number was reduced to 14, approximately half of what it was in the past. Next year the number is expected to be reduced even further. In the wake of the scandals threatening UNC’s integrity, UNC’s recently improved admissions process demonstrates a remarkable commitment to establishing an appropriate balance between academics and athletics.


In conclusion, I want to be clear about what I am and am not arguing. First, I am not arguing that UNC has never admitted athletes who read at elementary levels or lower, nor am I making any judgments of Willingham’s character. In fact, I believe she has had good intentions the whole time. Nevertheless, I am arguing that her methodology and anecdotes are not sufficient to substantiate her claims, and her choosing to publicize those claims has been irresponsible. Additionally, I am addressing questions about the classification and status of her research, though regardless of the answers to those questions, her methodology was still unsound. Last, I am acknowledging that UNC has admitted athletes who were considerably academically underprepared, but being underprepared for UNC does not necessarily mean reading at an elementary level or being illiterate.

Most importantly, I am arguing that truth matters. Last April, UNC convened a forum to present various perspectives on the role of intercollegiate athletics in higher education. Jay Smith, a distinguished professor of history at UNC, provocatively argued that faculty and athletics departments must become more adversarial with each other. Whereas athletics departments are committed primarily to winning, he averred, faculty are committed primarily to “truth and discovery,” to teaching and modeling “rigorous analysis.” Although Willingham’s faulty methodology demonstrates that Professor Smith, as a co-investigator with Willingham, has neglected to model rigorous analysis for the study under discussion, I hope he does not forget his commitment to truth. Indeed, as he argued last April, the integrity of the university depends on it.