When Paul Bissonnette recently came out and openly criticized certain facets of the way we use advanced metrics in hockey – something that he did for The Player’s Tribune earlier in September – the response was, well… I mean. Did we expect anything else?
I would challenge @BizNasty2point0 to name *one* player the Hawks signed due to inflated possession stats but I bet he’d chirp me instead.
— pb13 (@petbugs13) September 21, 2015
paul bissonnette seems like a guy who went to the barber for a shave once and considers himself a gentleman because of it
— Gráinne (@wholegrainne) September 22, 2015
Knowing the kind of comments that Paul Bissonnette has made in the past – and the kind of reactions that people have had to him and his opinions – actually kept me from reading his piece for a good long chunk of time. I didn’t want to fight with people over whether a traditionally controversial social media mogul (yeah, wrap your head around THAT one, genX) had said something opinionated that we simply disagreed with, or if he was actually being a lug-headed jackass. Quite frankly, I didn’t care one way or another; then I read the piece that Benjamin Wendorf over at HockeyGraphs put out, and decided that it was worth my time to dive into this a bit.
To start, I think that Wendorf does a masterful job of breaking down the right and the wrong of what Bissonnette says – there’s plenty of intelligence to acknowledge in what Bissonnette offers via his Player’s Tribune piece, and then there’s plenty that seems to be the byproduct of misunderstanding what stats should really be used for. For example:
“One of the conundrums when it comes to developing performance metrics is this: if your data doesn’t suggest that obviously good players are good, you need to make sure there is a justification for it. The same thing goes for obviously bad players. So in some respects, metrics have to have some of these “no duh” assessments, to make sure we’re aligning with the primary focus: success. Conversely, the whole purpose for teams is to gain competitive advantage, and recent metrics do in fact draw out players that are better or worse than they appear to most hockey fans. By the numbers, Patrice Bergeron is almost without equal, rather than a good defensive center. Cody Franson, like Anton Stralman before him, was a decent top-6 defenseman acquired for a pittance. Marc Methot is an unheralded defenseman. Eric Fehr and Alexander Semin’s combined cap hit is just short of Deryk Engelland’s. Diamonds in the rough will always be rare, but there’s gold in dem hills, pardner.” -Benjamin Wendorf, ‘Paul Bissonnette is Wrong and Right’
This resonated with me particularly when it came to one position: goaltending.
In this off-season in particular, there’s been a lot of breakdown of goaltending using the methods that statisticians used to break down the skaters in front of the goaltenders in years prior. We’ve seen the metrics that various outlets track goaltenders with plotted and graphed, then goaltenders are labeled by various stats gurus as ‘elite’, ‘struggling’, ‘declining’, and – by far the most concerning – ‘likely to bounce back’ or ‘overrated’.
Part of what both Wendorf and Bissonnette mention throughout the Player’s Tribune article and the HockeyGraphs rebuttal is that, in order for these ever-evolving stats to be effective, they’ve been tweaked over time to best support true indicators of good versus bad on-ice talent.
Bissonnette may describe it as useless because you ‘don’t need stats to tell you that’ Jonathan Toews is one of the best possession drivers in the sport, while Wendorf describes it as a good way to take a stat that clearly lines up with obvious talent and helps identify more underrated talent – but both men are clear in their assertion that the stats agree with very obvious shining beacons of skill. We may not need a stat to tell us that Patrice Bergeron is above average in all three zones, but that stat then helps us realize that Tobias Rieder, Brad Marchand, and Mikael Backlund are actually above-average players in all three zones as well – that’s the point of the stats.
What makes stats so frustrating with goaltending, though, is that we’re moving forward with stats (in other words, we’re trying to evolve advanced goaltending metrics) without a clear understanding of the position itself.
A good friend of mine has been doing a lot of this data evaluating for goaltenders recently, and came to the conclusion today that ‘Mike Smith is likely to bounce back’, based on the way that patterns were lining up in a particular chart of goaltending data. The support – which many of us agree with – was the way that the data proved elite talent for a specific tier of NHL netminders; if data agrees that Henrik Lundqvist and Carey Price are elite, it’s indicator that Mike Smith will regress upwards again is then believed to be true.
Numbers don’t lie, right?
The problem with this, though, is that we don’t truly know why Lundqvist and Price are elite; we also have trouble understanding why a stat makes Pekka Rinne look overrated, Corey Crawford look underrated, and Ben Bishop look like a sustainably tier-one NHL talent.
Depending on the goaltender you talk to, certain attributes are valued more than others; that’s likely a combination of the position being so highly nuanced and the lack of organization goaltending development itself has been given over the years. I was introduced to the position by a believer that Johnny Bower was an unparalleled talent and Patrick Roy was the product of a mutually understood system between himself and the defensive corps he skated behind; I value lateral fluidity and track-first superiority over strong puck-handling and aggressive skating abilities. Other goaltenders I know believe that you can’t teach reflexes; another is a slave to structure and will die on the mountain of a solid subconscious level of competence surpassing any natural agility or speed as a sustainable asset in net.
With so many different goaltender philosophies, though, the skating structure in front of netminders has remained as disorganized as ever; the NHL is a jumble of talent, development, structured defensive systems, and goaltender-skater interactions that has resulted in the numbers telling us far less about goaltending talent than any other position on the ice.
Talk to a forward, and many will tell you that the goaltender has it easy; you don’t have to do the job with grace or flair, you just have to get it done. Where a player has to skate well, pass smartly, and shoot with speed, strength, and precision, the forward is sometimes convinced that the goaltender only has to stop the puck – it doesn’t matter how, it just matters that it happens.
Talk to a goaltender, though, and you’ll see why the numbers may lie.
For example, I like to hold up Pekka Rinne – the picture of consistency and strong tracking, the Finnish netminder favors his glove hand to a fault. When playing behind a defensive system that fails to recognize this and play to that weakness (or strength, depending on the shot faced), Rinne’s numbers will fall – but he’s consistently reliant on that side, which makes him an easier netminder to develop a structured system in front of. Give him that structure on the blue line, and he’ll easily outperform netminders who possess stronger blocker sides than him – meaning that his stats can look ‘lucky’ or ‘overrated’, but are actually extremely easy to replicate and build around.
The league has a handful of other goaltenders who are easy to pick out patterns for and establish systems in front of; Jonathan Quick (who possesses the dual strength of having an extremely good read on the game itself and a pretty easy to identify set of parameters to his game), Corey Crawford, and Tuukka Rask are all strong system goalies as well.
Then, you’ve got the goaltenders who lack those rigid structural characteristics; Mike Smith, Jake Allen, and Marc-Andre Fleury are three very good examples of netminders who are hard to build a mechanical system around. They’re hard to pick out patterns for, which means – in theory – that they’re less likely to benefit from defensive systems built around their games. That makes them less likely to see poor results just because of a poor blue line corps in front of them, but also more likely to see periods of sub par (or even absolutely awful) performance with no seeming solution available but ‘waiting it out’.
Finally, you’ve got guys who stay farther in their net and are reliant on tracking the puck rather than a specific skill set (such as flexibility, athleticism, or a quick glove hand). Braden Holtby, Henrik Lundqvist, and Carey Price are all easy examples of this; I personally consider that style of goaltending the only one that’s truly elite, but that’s just my opinion. They too have their own limitations that certain stats reveal and others can falsely mask.
Obviously, we still need a way to evaluate goaltenders – despite a lack of structure in the position league-wide, there does need to be a way to determine whether a team is retaining the right goaltender or not. A lot of stats overlap – for example, workload for a butterfly starter is going to have a different impact than workload for a track-first backup – but they do need to be used. Otherwise, every team would be in the same boat that Vancouver is (sorry, Ryan Miller), where we simply use personal perceptions of ‘quality’ or ‘sustainable’ goaltending to sign big-money deals or walk away from players due to strange sample sizes.
The way we do that, though, is inherently going to be flawed without proper understanding of the position – and there, ultimately, is where I see what Bissonnette says and I somewhat agree.
The development of stats based on incorrect evaluations of who’s good can be frustrating – which is where we see reactions like Bissonnette’s. I don’t think he was reacting for the same reason I am here, but it’s a good launching point for an argument that many are failing to consider; as Wendorf said, Bissonnette was as wrong as he was secretly right.
*** author’s note: the tweets posted at the top are from writers whose work with stats I both often agree with and frequently share, and the goaltender data tracking is being done by someone I respect and admire as a friend and a colleague. By sharing these examples within the piece, I’m in no way asserting that any of the individuals being cited are stupid, inherently wrong, or worthless. There are only smart people doing the gord lord’s work with numberz being shared on this blog, and that will always be the case. We don’t smear people here; we just learn how to adapt what we’re developing together in a constructive way. ***