Monday, May 19, 2014

A Fool's Argument: Jay Smith's Poor Defense of Willingham's Research

“The fool who knows his folly
Becomes wise by that fact.
But the fool who thinks he's wise—
He's called 'a fool' indeed!”
― The Buddha 
“We are all born ignorant, but one must work hard to remain stupid.” ― Benjamin Franklin

Jay Smith should stop speaking for Mary Willingham. Although he is clearly more articulate than she in general, he nonetheless reveals his glaring ignorance of educational assessment whenever he comments on the subject. I would never engage Smith in a debate about French monsters. For his own sake, he should likewise refrain from engaging me in a debate about educational assessment. His attempts thus far have only made him look like a fool. More clearly than any of Smith’s previous statements, his latest blog entry illustrates his failure to grasp the nuances of the debate over Willingham’s research, and his pretense of understanding amounts to nothing more than sophistry.

Smith’s latest, of course, is an attempt to defend Willingham after her research application became available through public records requests and revealed she has misrepresented the nature of her research to the public. The foundation on which his entire argument is built is a one-paragraph narrative of Willingham and Johnson’s research activities, which warrants full quotation:
The actual record, alas, fails to conform to the fantasy visions of Mr. Bethel and company. The fact of the matter is that Johnson and Willingham answered the “coded data” questions correctly. The data was indeed coded, and Johnson–who tested the students and subsequently carried out the bulk of the analysis–never had access to the key that would have allowed her to identify the people she tested. She followed protocol to the letter. Willingham kept the names of the students because she believed–and still believes to this day–that it was her duty to retain access to the initial screening results of athletes that she, as the athletic department’s chief learning specialist, needed to remediate. She never sought to hide from anyone her retention of identifiable information. Why would she hide it? She was doing her job and using the test scores to help determine what form and degree of help athletes X, Y, and Z would need in order to improve their classroom performance. Although she and Johnson of course conferred on the analysis Johnson was performing, Willingham never revealed names to her research partner. She never once divulged identifiable information to anyone–until January 13, 2014, when Provost James Dean coerced her into doing so.
A house built on sand, a famous parable explains, is a house doomed to collapse. As the cornerstone of Smith’s argument, the paragraph quoted above is alarmingly sandy, and his argument, consequently, is about to fall.

Smith’s Transparent Smokescreen


Before addressing the inanity of Smith’s narrative, I first want to address how misleading it is, because his attempt at diversion is the most obvious vulnerability in his argument’s foundation. Again, attempting to justify Willingham’s possession of identifiable information, Smith wrote, “She was doing her job and using the test scores to help determine what form and degree of help athletes X, Y, and Z would need in order to improve their classroom performance.” Smith is actually right about that. Willingham, while working for the Academic Support Program for Student-Athletes (ASPSA), certainly had a legitimate educational interest in reviewing athletes’ academic records, as well as a reason to request consent to review their psycho-educational evaluations.

However, Smith excludes a significant fact from his narrative: Willingham left ASPSA in January 2010, yet her study extended through 2012.

Before elaborating on the significance of that fact, I must make some important points about psycho-educational evaluations. Even when intended for educational purposes, an adult’s psych-educational evaluation cannot be released to educators without the client’s consent, because the results from psychological assessments are considered protected health information under HIPAA. Furthermore, if an educator intends to use information from the psycho-educational evaluation for research purposes, the educator must specify that intent in the consent form or obtain additional consent. Otherwise, the educator has no right to maintain information from the psycho-educational evaluation after the educator leaves the capacity in which she provides educational support to that client. (Likewise, without consent, an educator has no right to maintain information from academic records after the educator leaves the capacity in which she provides educational support to that client/student.)

Therefore, beyond January 2010, without consent from the athletes, Willingham had no right to maintain their protected academic and health information, nor did she have any right to acquire the protected academic and health information of future UNC athletes.

Yet statements from Willingham in recent months indicate that she did maintain protected academic and health information (e.g., GPAs and WAIS results) from previous years, and she did acquire protected academic and health information from the three cohorts of first-year athletes (2010, 2011, and 2012) following her departure from ASPSA. By maintaining and acquiring that protected information, Willingham violated the privacy rights of those athletes.

Oops.

Willingham’s Nonconsensual Research


Using the same reasoning, later in his blog entry, Smith further attempts to justify Willingham’s mishandling of protected information by suggesting she may have honestly confused her dual roles as educator and researcher. Such may be the case (though the required FERPA training for university employees and the required ethics training for researchers should have sufficiently explained those roles for Willingham), but, again, her confusion would have only extended to January 2010, when she left ASPSA. After that date, Willingham had only one role in relation to athletes, that of researcher.

Thus, again, Willingham’s maintenance and acquisition of athletes’ protected information after January 2010 appears to constitute a blatant violation of those athletes’ privacy rights. Willingham is not the innocent, na├»ve educator Smith contends she is.

Accordingly, Willingham has no justification for having accessed and publicized information about the 2013 football team’s grades. On her private blog and then in her written testimony for the O’Bannon case, Willingham reported on the aggregate grades of a “cohort of seventeen starters and other regular players” who played in the Belk Bowl last December. As I explained in my February 24th essay, “Truth and Literacy at UNC,” Willingham would have only had a right to access those athletes’ academic records if she had obtained consent from the athletes to use their protected academic information for research purposes. She suggested she had done so, when she told The Daily Tar Heel, “The study included how they were doing in school, their GPA.” However, we now know from her research application that GPAs were not part of the study, nor did she obtain consent from the athletes to access their academic records and use their protected information for research purposes.

If Willingham’s statement above was not a deceitful attempt to justify her unauthorized maintenance and acquisition of athletes’ protected academic information, then I must not understand the meaning of deceit.

Of course, the circumstances surrounding her acquiring and publicizing that information remain a mystery for which an explanation is on order. Willingham’s actions raise weighty questions: When did she access those athletes’ academic records? Did she do so over Winter Break, when fewer faculty and staff were on campus to notice? Did anyone assist her? Can the university’s IT office verify whether Willingham and anyone else were accessing those records over Winter Break? Does the university have other means by which they can verify when Willingham and anyone else were accessing those records? Should Willingham (and anyone who may have assisted her) have faced consequences for accessing those records without consent to do so?

Most importantly, will Willingham demonstrate the same transparency she demands of the university, by answering those questions herself?

Such questions are similarly raised over the obscured circumstances regarding her infamous tweet about the 2005 men’s basketball team:


When did she access those basketball players’ academic records? Did she have assistance? Can the university verify when she accessed those records? Should she have faced consequences? Will she provide an explanation herself?

Regardless of the answers to those questions, Willingham’s research application makes clear the fact that she did not obtain consent from the athletes to use their protected information for research purposes (or tweeting purposes), and by that omission she violated not only their privacy but also the most fundamental principle of research ethics. Furthermore, her claim that the study included the athletes’ grades was, in Smith’s terms, a bald-faced lie, and her accompanying accusations of conspiracy against the Provost were baseless distractions meant to obfuscate the fact that she had operated outside the scope of her research.

Smith’s Stupid Idea


Returning to Smith’s blog entry, perhaps the most inane part of it is his suggestion that Willingham would have been the one to code the identifiable data for Johnson, the psychologist. Again, Smith wrote,

The data was indeed coded, and Johnson–who tested the students and subsequently carried out the bulk of the analysis–never had access to the key that would have allowed her to identify the people she tested. She followed protocol to the letter. Willingham kept the names of the students because she believed–and still believes to this day–that it was her duty to retain access to the initial screening results of athletes that she, as the athletic department’s chief learning specialist, needed to remediate.

No, Jay.

The coding should have been the other way around, especially after Willingham left ASPSA in January 2010. In other words, Johnson should have coded the data for Willingham so that Willingham could perform a secondary data analysis, the study’s stated procedure on all five iterations of the research application. Johnson was the licensed psychologist, meaning she was the one who administered and interpreted the tests, composed the psycho-educational evaluations, and met with the athletes privately to review the results. As the only licensed psychologist/healthcare provider between them, Johnson was the one who had the right to the athletes’ private health information, and Johnson was the one who had the primary obligation to protect that information. In fact, that is why Johnson’s consent form, which would have been signed by every athlete, specifically states, “I have been informed that the data from these [psychological assessments] may be used for the purpose of research . . . . My identity, however, is detached from these data before they are ever used, and can never be discovered or revealed.”

By the stipulations of Johnson’s own consent form, she should not have shared any of the athletes’ protected health information with Willingham for research purposes, without first coding that information. Hence, Smith’s assertion that Willingham coded the data for Johnson amounts to nothing but a fatuous attempt to obfuscate the reality of Willingham’s dishonest claims about the nature of the research.

More of Smith’s Failed Attempts


Without his narrative of Willingham’s research procedures, Smith’s argument collapses on its sandy foundation. Therefore, I will not take the time to address all his supplemental attempts to defend Willingham, but I will address three.

First, both he, in his blog entry, and Willingham, in a recent radio interview, have attempted to refute my point that the purpose of Willingham’s study was limited to researching learning disabilities (LD) and ADHD, not reading levels. Their argument has been that screening for LD requires the administration of a reading assessment, and, therefore, assessing reading levels was within the scope of the research. Their argument sounds reasonable to anyone unfamiliar with educational assessment, but those of us familiar understand the nuance Smith and Willingham conceal. Yes, one uses a reading assessment to screen for LD, but reading assessments can yield several measures, only one of which is necessary to the screening process: the standard score. By comparing a student’s standard score on a reading achievement test with the student’s standard score on an aptitude or intelligence test, the educator or psychologist can determine whether a discrepancy may suggest the presence of a learning disability. Grade equivalents, the measure Willingham reported to CNN, are extraneous to the screening process and are determined by achievement tests alone. Therefore, grade equivalents would be outside the scope of a study focused exclusively on LD and ADHD. Willingham would know that, but Smith, having no background in educational assessment, probably would not. That is why he should stick to the subjects about which he is knowledgeable.

Second, Smith accused me of a “bald-faced lie” for my statement that Willingham originally reported only using the SATA to determine grade equivalents. On this issue, the CNN report speaks for me:

Of 183 athletes in revenue-generating sports admitted to UNC between 2004 and 2012:

• Between 8% and 10% were reading below a third grade level.
• About 60% were reading between the fourth and eighth grade reading levels.

That's based on [Willingham’s] review of scores on the Scholastic Abilities Test for Adults (taken after they're admitted to the university).

In addition, Willingham looked at SAT and ACT scores, and found:

• 92% scored below the low average for admissions standards at UNC
• 22% scored below a 400 on the SAT verbal test
• 37% had a 200 point gap in SAT verbal scores, compared to the average student at UNC

Moreover, as I have explained ad nauseum, anyone familiar with educational assessment would have no reason to believe Willingham would have used and reported additional tests, because grade equivalents are determined by individual achievement tests alone, not by a combination of tests. In fact, one need not just take my word for it. After Willingham complained the three independent reviewers could not replicate her findings because they did not have the ACT/SAT scores and WAIS results, I emailed the reviewers to ask whether having those test results would have enabled them to determine reading grade equivalents and thereby confirm Willingham’s claims. Two of the reviewers responded, and both said no. Dennis Kramer, an assistant professor at Virginia, explained the problem most concisely: “There is not any known methodology (to my knowledge) where you could accurately combine each of the three assessments to create a psychometrically sound estimate of grade-level equivalents.” Again, if Smith were familiar with educational assessment, he would have known that. Unfortunately, his ignorance has not stopped him from speaking as if he were otherwise.

Third, the cutest little rant of Smith’s was his attempt to refute my criticism of Willingham’s claim that she worked with 500 athletes during her seven years with ASPSA. He wrote,

[Bethel] said in an audio interview with WCHL, for example, that Willingham “claimed that she worked with 500 student-athletes,” which he considers a ridiculously inflated number. He provided no reference to his source on the 500 claim, though it is true that Willingham has said on several occasions that she worked with “hundreds” of athletes in her seven years in the ASPSA. . . . The number may well have approached 500 (though it appears that she has never used that number herself.) Yet Bethel, consumed by his desire to denounce, took an especially petty and off-the-mark stab at undermining her credibility.

Had Smith taken five seconds to google “Willingham 500 athletes,” the second listing in the search results would likely have been a WUNC article, from January 31, that states, “[Willingham] estimates working with about 500 athletes between 2003 and 2010.” (To understand my full critique of that claim, see my essay “SATA-gate.”) Apparently, however, Smith is as ignorant of basic internet search procedures as he is of educational assessment.

Conclusion


In Aaron Sorkin’s Newsroom (one of my two favorite television series, the other being Mad Men), the members of an elite news team mismanage a major story, by failing to vet and verify their sources thoroughly enough for the gravity of those sources’ claims. After learning of their blunder, several of the senior members of that news team volunteer to resign from their positions in embarrassment. For them, accuracy and truth are of the utmost importance, and verification is a journalist’s utmost responsibility. Thus, to have failed at verification is to have failed at their jobs.

After five months of watching CNN, BusinessWeek, and many other news outlets continually report Wilingham’s false and duplicitous claims, I have realized Newsroom is actually a fantasy rather than a real life drama.

Two people are especially to blame for the past five months of unnecessary debate over Willingham’s inflated statistics and embellished anecdotes. Of course, Willingham and Smith themselves bear much of the responsibility, but I have two others in mind: Sara Ganim, of CNN, and Paul Barrett, of BusinessWeek.

Had Ganim and Barrett observed the most fundamental standard of journalism—verification—they would have identified Willingham as an attention-seeking fraud and subsequently refrained from propagating Willingham’s unsubstantiated and now-invalidated claims. Unfortunately for those of us interested in accuracy and truth, Ganim’s and Barrett’s complacency in January and February allowed Willingham to appear credible, leading several other media outlets to feature her in the months since. Because Ganim and Barrett did not verify from the outset, we are now, months later, having to wade through the minutiae of IRBs and research methodology, topics in which most people understandably have no interest. As a society, we must demand more from our journalists. Journalism should inform, not scandalize.

In the meantime, as long as Willingham and Smith continue to use casuistry to defend Willingham’s false claims, and as long as media outlets continue to give Willingham a voice, tedious polemics over educational assessment and research ethics will likely persist. Perhaps, however, Willingham and I could settle our disagreements conclusively and end the ongoing contest through our blogs and media outlets, by agreeing to a live, public debate.

After interviewing with BusinessWeek, CNN, ESPN, HBO, HuffPost, the N&O, and others, Willingham must certainly feel confident enough in her claims to subject them to the live scrutiny of someone knowledgeable enough to challenge them. Let her demonstrate her confidence in a live, public debate with me.

To be clear, I am proposing a debate between Willingham and me, not Smith and me. Although I am not in the least bit intimidated by Smith, this debate should have never involved him in the first place. It is not a debate about imaginary monsters. Rather, it is a debate about imaginary numbers—Mary Willingham’s imaginary numbers. Again, if she is so convinced those numbers are real, let her defend them in a live, public debate with me.

Otherwise, the dispute over IRBs and research methodology will persist until the media stops paying attention to Willingham’s attention-seeking antics or until she ceases to grasp for the spotlight. I am happy to accept a truce, to end the discord, to see this 15 minutes come to a halt. However, I am also prepared to continue the debate, whether in writing or in a live, public venue.

I await Willingham’s response.