The thing is, many of us have been around the block on more than one occasion with poorly thought out metrics which lead to unintended and down right bad consequences. A few years ago I was rallying against a system used to rate and rank relative risk of software systems based on a score. Many times I was told: "at least it's better than nothing." Ultimately it was discovered that there was nearly an inverse relationship to the scoring. Those bits of software which ranked as the most secure ended up being the least, with the least secure being the most. This happened because developers unaware of security concepts assumed security was happening elsewhere rather than taking the initiative to practice secure coding. Interestingly enough this entirely invalidated the argument that doing something was better than nothing as the system drove misplaced focus and overall reduced the security of the overall environment as the number of bad bits of software out numbered the good, thus leading to poor hiring practices. Management saw the score and figured everything was getting better and the environment needed fewer security resources rather than the reality which was that many more were needed.
Complex systems are challenging & one study isn't going to provide the miracle solution. Surgery is not a refrigerator and to think every problem can immediately be solved by a trivial metric can be a dangerous game, especially when it comes to lives. When it comes to health care I tend to look at the macro, rather than the micro. Which hospitals have higher ratings, lower insurance claims and rates, etc., however, I understand that information can be gamed as well and take it with a grain of salt. At the end of the day I hedge my bets in this space by eating well, working out daily, and otherwise trying to avoid the medical industry at all costs.
So we measure nothing? As pointed out in your comment, the fix is the consumer of the information understanding it and using it correctly, not avoiding the data entirely.
This reminds me of the debate about improving schools in the U.S. We hear objections from teachers unions how it's literally impossible to measure the efficacy of schools and teachers.
What other industries can afford to not rate their employees' effectiveness, outside of the broken markets of primary education and healthcare?
You have to be careful what you measure. Imagine you have a 6th grader who is reading at a 2nd grade level. At the end of 6th grade you test her and find out now she's reading at a 5th grade level. Wow! A big improvement! Her teacher must be great.
Unfortunately the test shows she's reading below grade level and the teacher is put on an improvement plan.
I agree that bad metrics can be worse than none. I've certainly seen other examples in books on management.
In this case, though, it sounds like they've put a lot of effort into coming up with good metrics that are hard to game and, most importantly, are convincing to the surgeons themselves. The article talks about that at some length.
(BTW thanks for plugging static code analysis -- I work in that field :-)
> I'm having a hard time understanding how this could be. How could attempting to meet a list of security requirements result in less security than doing nothing?
IME, people given a list of "X requirements" often assume that the appropriate experts have addressed subdomain X and that it is no longer necessary (or even appropriate) to expend further attention on that subdomain beyond meeting those requirements. Thus a standard security checklist that isn't well tailored to a particular project may decrease the quality of the result for that project, because the people on the project are less likely to consider security in the context of the particular project, and instead inspect efforts in crossing boxes off the checklist.
"Have you implemented input validation and output encoding?"
What could go wrong, right? A naive developer with little security understanding will look at that question and might even ask other members of the team about it. After a short bit of research the developer answers the question as "yes the application implements input validation and output encoding" based on finding that the framework utilized does in fact have input validation and output encoding functionality built in.
Of course while the framework might have such controls and they may even be implemented for the particular code base, it turns out that relying on the framework alone isn't sufficient because there are many places in the code where that framework can be subverted or entirely bypassed. Rather than stopping at the framework controls the developer actually needed to perform a code audit and find everywhere user data might go to a sink which could result in unauthorized code execution, SQLi, or XSS.
This is a particular scenario I've seen from the days of PHP magic quotes all the way to the latest ASP.NET request validation. It turns out the question was too simplistic. If we look at solving this problem we then need to dive into a great level of detail not only pertaining specifically to the framework controls available within the environment, but we must also look at how the code is implemented, ultimately scrapping the check-list only approach in favor of static code analysis. Of course if a management team only has a checklist and they are getting false answers as exampled above, they have no idea why they need static code analysis or should pay for the software and resources to use it.
Complex systems are challenging & one study isn't going to provide the miracle solution. Surgery is not a refrigerator and to think every problem can immediately be solved by a trivial metric can be a dangerous game, especially when it comes to lives. When it comes to health care I tend to look at the macro, rather than the micro. Which hospitals have higher ratings, lower insurance claims and rates, etc., however, I understand that information can be gamed as well and take it with a grain of salt. At the end of the day I hedge my bets in this space by eating well, working out daily, and otherwise trying to avoid the medical industry at all costs.