In comments to my “Why I Don’t Like CRISC” article, Oliver writes:
CobIT allows to segregate what is called IT in analysable parts. Different Risk models apply to those parts. e.g. Information Security, Architecture, Project management. In certain areas the risk models are more mature (Infosec / Project Management) and in certain they are not (software distribution). That is for the risk modelling (sic) part.
Oliver: I’m very glad that others in our industry are preaching the concept of model selection & fit. And because you’ve demonstrated that at least you believe this is an important aspect of IRM, I’m ready to believe what you’re saying there. But before I do so, I spent a good deal of time in Missouri, so I need you to show me:
- Define “mature” – what makes a mature information risk model? In fact, show me the industry standards for gauging model maturity, so that I can examine different models, similarly.
- Show me, oh please show me, an information risk model that has even been tested (publicly) for repeatability and accuracy, more or less been shown to provide repeatability and accuracy to a measurable degree of confidence.
Now my thought is that you can’t have a mature risk model without having a measurable notion of repeatability (two analysts with the same data and same model go into separate rooms and come out with reasonably similar results) and accuracy (model outcomes have been tested to be correct some degree of the time). Maybe I’m not subscribing to the right scientific journals out there, but I’ve yet to see the data sets and the published models or model maturity tests for IRM.
For risk identification and KRIs (note to readers: I’m assuming Oliver means Key Risk Indicator – a useful but loaded phrase itself), an internal control framework which is based on cobit allows an adequate and comprehensive net of indicators for risk assessment based on operational performance.
You’re assertion is that COBIT’ is proven to be an “adequate” and “comprehensive” internal control framework. Can you show me evidence of this? What documentation for this has ISACA released? How was it proven? Where’s the study? How did they seek to falsify COBIT’s adequacy and comprehension? How was comprehensive measured? At what point was it shown that more COBIT effort decidedly into the realm of diminishing returns?
If you think that “some things can’t be measured” will prove your thesis, you don’t know Risk Management at all.
I never said that, and due to the fact that I’ve taught courses based on Hubbard’s “How To Measure Anything” to risk analysts, I’m going to offer that you don’t know me well enough to come to any conclusion about my knowledge around Information Risk Management.
What I’m saying is that ISACA, COBIT, and RiskIT aren’t mature enough to certify practitioners in a meaningful manner – where “maturity” is an ability to consistently, repeatably, and accurately show a change in risk using ISACA’s own documentation. If you can’t show me how COBIT measurably (again, where the concept of measurement requires known accuracy and repeatability – just drilling the point home, here) modifies exposure to risk or capability to manage risk in these ways, I don’t think ISACA is ready to say that we, as an industry, are more than isolated alchemists trying to find our own, individual ways to turn lead into gold. To carry the analogy, the attestation that CRISC would provide has nothing to do with knowledge of chemistry, but everything to do with the alchemists ability to repeat a known means of trying to turn lead into gold.
There is no mathematical voodoo to model a risk exposure which is 100% correct.
We’re in agreement about modeling risk exposure. To paraphrase Jaynes (poorly), probabilistic models are hypothesis and therefore we should expect (hope!) for them to be frequently falsified. In addition – just to complete the picture for you, Oliver, I’m also on record as stating that arriving at a state of knowledge for capability to manage risk is similarly difficult (and this is the whole crux of the COBIT/RISKIT/CRISC request for proof – understanding capability in a measurable way is a key dependency to understanding exposure, and therefore, ISACA is silly for trying to certify that someone can discuss exposure if they can’t even show me how COBIT reduces risk) .
You have to keep the purpose in mind and also use professional judgment based on your experience (which CRISC by the way tries to attestate)
Fascinating, so CRISC tries to provide clear evidence that an individuals experience and professional judgment is of some quality? My whole point in this series is that any individual with experience in information risk management should know enough to know that a certification around Information Risk Analysis and management is goofy. As for documenting an individual’s professional judgment skills, I’d love to see how the test does that in a rational manner.
You fight against an attestation which takes into full consideration your own challenge.
Nope. Not even close. You have no CLUE what I stand for. I’m all for good attestation. As I said the other day:
(…I’d argue that IRM shouldn’t be part of an MIS course load, rather it’s own tract with heavier influences from probability theory, history of science, complexity theory, economics, and epidemiology than, say, Engineering, Computer Science or MIS.)
My position is that given the difficult nature of risk analysis (as I’m saying above), there’s no way CRISC can attest to any competency around Information Risk Analysis, and if ISACA can’t show me how COBIT changes exposure or capability in a measurably way, then CRISC can’t possibly even attest to competency around Information Risk Management. Maybe it can serve as a RiskIT test, sure and I’m fine with that. From the same blog post as my quote above:
IRM is not (just one) “process”. Now obviously certain risk management standards (document a simple) process. In my opinion, most risk management standards are nothing BUT a re-iteration of a Plan/Do/Check/Act process. And just to be clear, I have no problems if you want to go get certified in FAIR or OCTAVE or Blahdity-Blah – I’m all for that. That shows that you’ve studied a document and can regurgitate the contents of that document, presumably on demand, and within the specific subjective perspective of those who taught you.And similarly if ISACA wants to “certify” that someone can take their RiskIT document and be a domain expert at it, groovy. Just don’t call that person “Certified in Risk and Information Systems Control™” because they’re not. They’re “Certified in our expanded P/D/C/A cycle that is yet another myopic way to list a bajillion risk scenarios in a manner you can’t possibly address before the Sun exhausts it’s supply of helium.” “TM”
I’ll state it again, if they want to change the certification’s title and meaning to simply state that an individual can do the above for RiskIT – have a day, good on you. Just don’t expect me to believe that this certification means that the individual knows anything about information risk analysis, or risk analysis in general.
