Virginia Tech STAR Ratings For Football Helmets: Helpful But Come With Limitations

A total of fifteen football helmets have earned five stars in the latest Virginia Tech Helmet Ratings,^TM a test of helmets in a laboratory environment intended to measure their effectiveness in reducing some of the forces that cause concussion.

Schutt’s AiR XP, Pro VTDII, and Vengeance VTD II continue to hold down the top three spots on the list of five-star helmets (awarded to helmets Virginia Tech considers the “best available” (1), with two from Riddell (the Speedflex and 360), three from Xenith (Epic, , X2, X2E), two from Rawlings (the Quantum Plus and Tachyon), and two from SG (Version 2.0 and 2.5) rounding out the five-star group. The inclusion of the two SG helmets comes with caveat, advising consumers to monitor the padding of the helmets throughout the season because of permanent cracks in the padding observed for all tested SG helmets; reminding them that the two year lifespan of the SG helmets is much shorter than the 10-year lifespan for most other helmets; and that the sizing of the SG helmets was unique, so careful measuring of head size was required.

Nine helmets were rated 4 stars (classified by Virginia Tech as “very good”): three by Schutt, two by Riddell and Rawlings, one by Xenith, and one by SG.

As in prior years, Schutt continues to hold down all three spots in the 3-star (good) category with its Air XP, DNA-Pro+ (which inexplicably dropped from 4-stars to 3 in 2013; see NOCSAE’s “concern” about this change below), and the Air XP Ultralite.

Extensive tests

Virginia Tech’s current evaluation process involves performing 120 impact tests on each helmet model at multiple locations and impact energies utilizing the STAR (Summation of Tests for the Analysis of Risk) Evaluation System, which it developed based on data collected from over 1.8-million head impacts experienced by football players over an eight-year period.

According to Virginia Tech, the STAR value is an estimate of the number of concussions that one collegiate player might experience over the course of a season wearing a specific helmet. The lower the STAR value, the better the helmet is believed by Virginia Tech to be in reducing the risk of concussion.

“We recommend any of the four or five star helmets for players. The specific helmet a player chooses will be dependent on other factors such as a fit and comfort,” said project director Stefan Duma, a professor of engineering and head of the Virginia Tech — Wake Forest School of Biomedical Engineering and Sciences.

Limitations

While the Virginia Tech ratings provide parents with valuable biomechanical data intended to help them make educated decisions about which helmet to purchase, parents and athletes need to be aware that the ratings come with a number of significant limitations:

Ratings currently based only on linear acceleration: Because the current STAR values are based solely on linear (i.e. straight-line) acceleration and do not reflect rotational (i.e. twisting) acceleration, Virginia Tech’s rating methodology has come under heavy criticism in recent years by the National Operating Committee on Standards for Athletic Equipment (NOCSAE).

In a July 2013 statement issued after the 2013 STAR ratings were published, NOCSAE noted that, “A concussion in football is a very complex event involving different and changing forces, linear and rotational accelerations, helmet fit, player position, impact duration, player concussion history and overall health, and potentially even genetics.” It went on to state that “the Virginia Tech Helmet Ratings system approaches the very broad and complex issue of concussion protection from a narrow vantage point of linear accelerations only and does not address other biomechanical variables such as rotational accelerations, particularly where rotational accelerations precede the linear acceleration in a hit. The consensus of scientific experts (1) agree that rotational accelerations are involved in many, if not almost all, concussive events, although no correlated injury threshold for rotational accelerations has been found,” the NOCSAE statement continue

In response to the NOCSAE criticism, Virginia Tech has continued to promise that its ratings would take into account rotational (e.g. twisting) acceleration as well, based on a research paper published in 2013, but, as of September 2016, they do still do not.

NOCSAE criticism continues

The release by Virginia Tech of its second set of STAR ratings in 2014 was met with continued criticism from NOCSAE. Although applauding and encouraging the growing research in the area of concussion protection for athletes, NOCSAE was once again quick to caution coaches, consumers, and parents that scientific evidence “does not support [Virginia Tech’s] claim that a particular helmet brand or model is more effective in reducing the occurrence of concussive events,” to point out, as it had in 2013, that the STAR ratings approached the very broad and complex issue of concussion protection from a narrow vantage point of linear accelerations only, and to again remind consumers that the ratings only applied to size large adult helmets, not to smaller adult helmets.

NOCSAE’s longtime Executive Director, Mike Oliver, noted that the STAR ratings were “not standards.” Rather, he said, they were a “theoretical method of comparing one helmet against another,” and claimed that “many have misunderstood the purpose and limitations of the STAR ratings. A 5-STAR rating does not mean,” Oliver said, “that the helmet is great at preventing concussions. It simply means that it might be better than another helmet with a lower rating.”

For its part, NOCSAE has continued to consider updates to its helmet standards, which have remained essentially unchanged since first issued in 1976 and which were designed to ensure that helmets protected against skull fractures (which they do) to require helmets to limit certain concussion-causing forces, both linear and rotational. Unfortunately, as with Virginia Tech’s promises to include tests of rotational acceleration in its standards, NOCSAE’s promises have been turned out to be just as empty: while the NOCSAE Board voted at its June 2014 meeting to issue new standards for public comment, and promised that it would finalize a new standard in June 2015 and require implementation by manufacturers by June 2016, as of September 2016, the new standard, according to the NOCSAE website, remains in draft status, with the earliest it can be moved to final status delayed until January 2017, at the earliest.

Oliver went on to claim in the NOCSAE statement that, because of this alleged misunderstanding, the effectiveness of helmets in protecting against concussions has become “exaggerated,” taking focus away from steps he said were known to have a more immediate and much greater effect on concussion reduction, such as making sure equipment fits properly, ensuring that young athletes are taught proper blocking and tackling techniques and demanding enforcement of rules that prohibit players from leading with their helmets to hit other players (all of which, it should be noted, are emphasized by MomsTEAM in the Six Pillar approach to concussion risk management featured in its PBS documentary, The Smartest Team: Making High School Football Safer (streaming on this website), and for which we have been advocating for many years).

In addition, NOCSAE pointed to a University of Wisconsin study [8] showing no difference in the rate of concussions or severity of concussions by helmet brand. (see further discussion of this study below; for an article discussing the study in detail, click here.]

Values should be used for comparative purposes only: The STAR value is based on injury probability and should not be interpreted as a literal number of injuries for any one athlete.

Wearing a highly rated helmet will not prevent a player from sustaining a concussion. The Virginia Tech researchers concede that any player can sustain a head injury even with the very best protection, that a “specific person’s risk of concussion may vary as a result of a number of factors” having nothing to do with the helmets, including genetic differences (some athletes may be genetically predisposed to concussion), age, health history (e.g. history of migraines, depression or other mental health disorders, attention deficit hyperactivity disorder, learning disabilities, sleep disorders, and/or previous history of concussion), impact factors (e.g. neck muscle strength/weight), style of play, etc.

The ratings are based on laboratory tests: While the drop tests performed by the Virginia Tech researchers are designed in theory to measure the ability of a helmet to absorb the energy from hits coming from various directions, and the STAR value is based on the theory that lowering head acceleration is likely to result in fewer injuries on the field, Virginia Tech did not test helmets under game conditions.

In its July 2013 statement, NOCSAE criticized the STAR rating system on that basis: “There are a near infinite number of ways to test helmets (varying temperatures, impact location, helmet size, drop height, etc.) and, therefore, generalizations were made so that the helmets could be tested in a practical manner. Helmets were not tested under game conditions. For example, air bladder fitting and protection systems were not inflated to achieve it, even though the NOCSAE standards require that manufacturer fitting instructions be followed.”

At the time the 2013 STAR ratings were issued, there were no methodologically sound peer-reviewed studies finding that helmet design can reduce risk of concussion in practices and games, with the one that did find such a reduction, published in 2006, heavily criticized as methodologically flawed or marred by conflicts of interest. (1)

Since that time, however, a study published in January 2014 in the Journal of Neurosurgery (9) has reported a 53.9% reduction of concussion risk associated with the Riddell Revolution helmet compared to another Riddell helmet, the VSR4. The difference in concussion risk reported between the two helmets matched almost exactly the estimated concussion risk between the two helmets predicted by the 2013 STAR ratings, which estimated that a Riddell Revolution, a 4-STAR helmet introduced in 2000, would result in a 54% reduction in risk of concussion compared to the Riddell VSR4, a 1-star helmet employing 20-year-old technology which, although no longer on the market, was still being worn by as many as half of all collegiate and professional players as recently as 2011.

“We would never have guessed the risk reduction would have matched so closely,” said Duma. “We think that this is excellent validation” of the STAR system, and, in our opinion makes the overall STAR ratings very useful.” [For the full interview between MomsTEAM’s Lindsay Barton Straus and Duma, click here.]

Other helmet companies and NOCSAE were quick, however, to point out limitations in that new study.

Said Ashley Quintero, National Sales Manager for SG Helmets, “We believe the most informative research is comprehensive and unbiased. In our opinion, the recent Virginia Tech study was not comprehensive, as the study limited its exploration to the products of just one manufacturer.” Quintero expressed concern that, unless those limitations were considered, “there exists the potential for coaches, parents, and athletes to assume these results reflect all football helmets.”

She cautioned “consumers against expanding the findings of one study and generalizing that its outcome holds true for all products,” and said SG Helmets “recommends that coaches, parents and athletes make a well-informed decision based on the advantages and disadvantages of every available product before making a purchase.”

“It would be misleading to suggest that the data [in the new study] supports the proposition that one helmet reduces the incidence of concussion more than another. There is a need for an abundance of caution in this respect,” said Schutt CEO, Rob Erb. “Stripped of the headline grabbing claims of double-digit concussion reduction, there is little here to get excited about. The best that can be said about the published Technical Note, based upon the limited data set, is that it suggests that players who wore large standoff shelled Riddell Revolution helmets were diagnosed with fewer concussions than their counterparts who wore the much smaller standoff VSR helmets. However, even this conclusion would have to be taken with a grain of salt. The statistical differences can be accounted for by a number of factors not discussed by the authors.” [For the full text of Erb’s lengthy email to Brooke de Lench on the Rowson study, click here.]

Given those comments, it was not the least bit surprising that Erb, despite the fact that two Schutt helmets topped the list of 5-star helmets in the Virginia Tech 2014 STAR rating (and continues to hold down the top three spots in the current ratings), was quoted in NOCSAE in its 2014 statement as being quick to acknowledge that the ratings do not support a conclusion that the helmets will limit or prevent concussions. “Schutt Sports would never represent to somebody that they’re not going to get a concussion if they wear one of our helmets,” he said. “As a manufacturer of a helmet considered by this rating system to be the best available, I believe that telling people that an athlete is less likely to get a concussion if they use a 5 STAR helmet is irresponsible. The best helmet is the one that carries NOCSAE certification, fits the position, is configured with the proper mask and the player is comfortable with it,” Erb said.

For his part, Mike Oliver, Executive Director of NOCSAE told MomsTEAM’s Brooke de Lench at the time the 2014 study was issued that, “We are still reviewing the study and its conclusions, and it is hard to tell at this point whether the conclusion is that the Riddell Revolution helmet model is better than the Riddell VSR 4 model, or whether that conclusion also applies to other helmet brands and models. Those two helmets are based on different design concepts, with the main difference being that the Revolution model has a larger shell with more padding on the inside. This type of design difference also exists in almost all other brand helmet models introduced since 2000 when the Revolution was first made available, and this new design type has replaced the older style helmets over time, but we see an increase in concussions, which is not consistent with the study findings. So there are lots of questions and information that the study does not address.” [For Brooke de Lench’s complete email interview with Oliver, click here]

Tests only performed on adult helmets: As the Virginia Tech researchers themselves admit, the STAR evaluation system was developed based on head impact exposure of collegiate football players, which research shows is different from that of high school football players, which, in turn, is different from youth football players. (3)

Said NOCSAE in its July 2013 statement, “Only adult-size large helmets were evaluated. There are no data to support applying the same ratings to other sized helmets of the same model, including youth football helmets. No adult X-Large, Medium, Small, X-Small or any youth-size helmets were tested. The STAR value itself is a theoretical calculation that is based on a probablistic analysis of impact exposure and injury risk at the collegiate level.” NOCSAE repeated that criticism in its May 2014 statement.

The Virginia Tech researchers agree: “It is not safe to extrapolate the[ir] findings to youth football helmets,” they admit. “Youth football players are likely to experience a different head impact exposure, which will require a modified evaluation system.”

“There are many questions left to be answered regarding youth helmets,” conceded Duma in a recent email to MomsTEAM. “In a general sense, lower head acceleration is better. I think that everyone would agree to that statement. However, we still have much research to perform on the youth population.”

Sales of 5-STAR helmets soaring

The criticism by NOCSAE and the helmet manufacturers of the STAR ratings system, however, has recently been replaced, at least according to the July 2014 Times article (12)with grudging support. “After initially voicing skepticism, football helmet manufacturers [have] embraced the STAR system’s ratings, and the N.F.L. posts the ratings in every team’s locker room,” the article reports.

The reason? Apparently, because money talks. “There has been a dramatic change in sales in football helmets since 2011 [when the first STAR ratings were issued],” Oliver told The Times’ Jeff Klein. “I see high school football athletic directors submitting purchase orders for 500 five-star helmets. Parents are saying, ‘I don’t want a four-star helmet, I want the best for my kid.'”

Absence of proof of protective effect not same as no protective effect

To be fair, all the major football helmet manufacturers (Riddell, Schutt, Rawlings, and Xenith), and newcomer, Simpson-Gnassi, are clearly attempting to engineer helmets that will reduce the risk of concussions, for which they deserve credit. As the 2013 international consensus statement on concussions (4) observes, although there is “no good clinical evidence that currently available protective equipment will prevent concussions, … biomechanical studies [such as those performed by the engineers at Virginia Tech] have shown a reduction in impact forces to the brain with the use of helmets.” In other words, that there aren’t more studies showing a concussion reduction effect under field conditions, is not the same as saying that helmets aren’t reducing that risk; just that proving that they do with the methodological and statistical rigor required by a peer-reviewed journal is is “complex, challenging methodologically, resources intensive and expensive.” (5)

Are 5-star helmets actually more protective?

In apparent reference to the Virginia Tech ratings, the American Academy of Neurology concluded in its 2013 guidelines on concussion management (6) that the data was “insufficient to support or refute the superiority of one type of football helmet in preventing concussions. In other words, while Virginia Tech’s rating system would lead one to conclude that a 5-star helmet is “better” than a 4-star helmet, and so forth, the AAN wasn’t buying into the ratings as providing that kind of information (which is no doubt why Virginia Tech itself says any 5- or 4-star helmet is recommended).

NOCSAE’s statement echoed the AAN’s view, asserting that, “According to an independent statistical review of the scores and categories upon which the STAR number is based, there is no significant statistical difference between helmets in the 5 STAR, 4 STAR, and 3 STAR categories.”

Indeed, NOCSAE suggested in 2013 that the STAR rating methodology might be fundamentally flawed, expressing “concern” that Xenith’s X2 helmet jumped after re-testing from a 3-STAR rating to a 5-STAR rating without a change in design or materials, and about the fact that not all brands/models were retested.

NOCSAE also noted that another helmet model retested in 2013 (the Schutt DNA-Pro) went from a 4 STAR to a 3 STAR with no design changes.

From these results, NOCSAE surmised that “If all helmets had been retested, it appears reasonable to believe that other helmets also may have moved into and out of different STAR categories.”

In an email to concussion blogger Dustin Fink published on his website in August 2013, (7) Duma responded to NOCSAE’s criticisms, characterizing them as “incorrect and misleading” in two respects (facts that he claims to have told NOCSAE before it released its statement but that they chose not to make): “First, only one helmet, the Xenith X2 increased two stars, and most importantly it did have substantial design changes. The 2013 [version] added much more padding and it performed much better. I have no idea why NOCSAE is misleading the public about this.”

“Second, we did re-test all of the helmets. I have no idea why NOCSAE is implying that we did not. Each year we re-test samples from all makes and models. If the values are within the range of previous years, we kept them the same. But, if the values are outside of the previous range, we use the new values. In this light, some helmets moved down. When you examine those, there are clear differences in materials. I relayed all of this to NOCSAE, and am disappointed they made the statement so misleading and inaccurate.”

NOCSAE urges parents to get all the facts

In its July 2013 statement, NOCSAE said that parents, players, coaches, and athletic directors needed to recognize the many limitations in the Virginia Tech helmet ratings, and expressed the fear that, unless such limitations were considered, “the potential exists for players, parents, coaches, and administrators to overemphasize the role of the helmet in protecting against concussions. This overemphasis increases the likelihood that less attention will be given to other steps that have a more immediate and much greater impact on concussion reduction,” such as teaching proper tackling and blocking technique minimizing helmet-to-helmet contact, neck strengthening, proper helmet fit, and better rules enforcement).

[For more on ways to minimize head injuries in contact and collision sports, click here; for MomsTEAM’s Six Pillars of concussion risk management, featured in its new high school football documentary, “The Smartest Team,” click here].

NOCSAE has thus urged parents of athletes and athletes to get all the facts about football helmets and concussion protection because “the safety of young athletes is too important to rely on only one partial measurement of helmet performance.”

To that end, NOCSAE encourages athletes and parents to carefully review:

Hang tags that come with all new football helmets that address the helmet’s abilities and limitations
Informational booklets developed by manufacturers that contain critical information about the helmet’s abilities and limitations
Warning information that is prominently and permanently affixed to the exterior of every helmet; and
Free downloadable resources created by the Centers for Disease Control regarding concussion recognition, response and prevention. Those resources can be found at www.cdc.gov/concussion/sports.

Indeed, if the finding in the 2013 and 2014 research papers by researchers at the University of Wisconsin ^[8,11] that newer helmets are not appreciably more protective against concussion than older helmets is replicated in future studies,it may mean, says lead author McGuine, that “parents shouldn’t be alarmed if their kid is being asked to wear a helmet that is three years old, and administrators shouldn’t be worried about purchasing the most expensive and/or newest helmets.”

Most important, some experts say, than the helmet brand is helmet fit. As not every helmet is going to fit every player, selecting a helmet that fits correctly – snugly almost to the point of being uncomfortable – will allow the helmet to perform at its best in protecting the player from injury. which includes staying on the player’s head and not coming off during play. (2)

Stay tuned for developments in this continuing story.

Sources: Virginia Tech (Virginia Polytechnic Institute and State University): and NOCSAE

Footnotes:

1. Daneshvar DH, Baugh CM, Nowinski CJ, McKee AC, Stern RA, Cantu RC. Helmets and Mouth Guards: The Role of Personal Equipment in Preventing Sport-Related Concussions. Clin Sports Med 2011; 30: 145-163.

2. Torg J, Boden B, Hirsch H, Fowler J, Gaughan J, Comstock RD, Tierney R, Kelly P. Athletic Induced mTBI and Catastrophic Intracranial Injuries: Determining Helmet Efficacy and Predisposing Injury Profiles. Presentation Paper AOSSM 2012.

3. Daniel R, Rowson S, Duma S. Head Impact Exposure in Youth Football. Annals of Biomed Eng 2012: DOI:10.1007/s10439-012-0530-7 (accessed February 15, 2012).

4. McCrory P, et al. Consensus statement on concussion in sport: the 4th International Conference on Concussion in Sport held in Zurich, November 2012. Br J Sports Med 2013;47:250-258.

5. Benson B, McIntosh A, Maddocks D, Herring S, Raftery M, Dvorak J. What are the most effective risk-reduction strategies in sport concussion. Br J Sports Med 2013;47:321-326.

6. Giza C, Kutcher J, Ashwal S, et al. Summary of evidence-based guideline update: Evaluation and management of concussion in sports: Report of the Guideline Development Subcommittee of the American Academy of Neurology. Neurology2013;DOI:10.1212/WNL.0b013e31828d57dd (published online before print March 18, 2013)

7. E-mail, Stefan Duma to Dustin Fink, unknown date, as reported in “Virginia Tech Responds to NOCSAE Press Release,” The Concussion Blog (July 24, 2013)(http://theconcussionblog.com/2013/07/24/virginia-tech-responds-to-nocsae-press-release/)(accessed August 5, 2013)

8. McGuine T, Brooks A, Hetzel S, Rasmussen J, McCrea M. “The Association of the Type of Football Helmet and Mouth Guard With the Incidence of Sport-Related Concussion in High School Football Players.” Presentation Paper AOSSM, July 13, 2013. Presentation Paper, American Academy of Pediatrics National Conference & Exhibition, October 28, 2013; American Medical Society for Sports Medicine. “Brand/type of helmet, mouthguard may not significantly reduce risk of sport-related concussion in high school football players.” ScienceDaily. ScienceDaily, 14 April 2014.

9. Rowson S, Duma SM, Greenwald RM, Guskiewicz KM, Mihalik JP,Can helmet design reduce the risk of concussion in football?J Neurosurg. 2014; DOI: 10.3171/2014.1.JNS13916(published online January 31, 2014).

10. Rowson S, Duma SM. Brain Injury Prediction: Assessing the Combined Probability of Concussion Using Linear and Rotational Head Acceleration. Ann. Biomed Engineering 2013;41(5):873-882. DOI: 10.1007/s10439-012-0731-0

11. National Operating Standards NOCSAE Board Approves First Helmet Standard to Address Concussion Forces. June 14, 2014 press release. (accessed at http://nocsae.org/wp-content/uploads/2011/10/NOCSAE-June-Board-Meeting-release-FINAL-6-20-14.pdf)

12. Jeff Z. Klein. “Spartan Hockey Helmets Going Under Microscope.” New York Times. July 22, 2014 (http://www.nytimes.com/2014/07/23/sports/hockey/for-safety-hockey-helmets-going-under-microscope.html?_r=0)(accessed July 23, 2014).

13. McGuine TA, Hetzel S, McCrea M, Brooks AM. Protective Equipment and Player Characteristics Associated With the Incidence of Sport-Related Concussion in High School Football Players. Am J Sports Med. 2014;20(10)(published online ahead of print, July 24, 2014 as doi:10.1177/036354651541926.