The notion of SMART targets is a well understood one, and is often deemed to be a vital concept behind creating the kind of key performance indicators that so many BI projects exist to support. And yet the SMART criteria are, perhaps, less often applied to measuring the success of BI projects themselves.
There is a tendency for the business case for BI projects to be based around a core set of information-related benefits. These often cover areas such as:
- Improvements to data quality
- Reduction in manual processing/rekeying etc
- Improved reporting capability
- Making more information available to decision-making
- Removing duplication of data.
There is nothing wrong with any of the above, and a successful BI initiative should certainly make progress with each of them. But without further refinement and focus it will not be easy to demonstrate if, when and how your BI project has helped your organisation to achieve any of the above.
Improving reporting capability is a laudable aim, but what do you really mean? Which data sources are you referring to and what reporting requirements? What do you mean by ‘improved’ and how are you going to measure it? What is the current level of performance and what have you defined as the level of improvement you are looking for? When must this improvement be achieved by?
It is by fleshing out each of the above, and similar, that you start to move from a series of sensible, but rather nebulous, benefits into targets that can be quantified, measured and reported on.
In addition, the measurement of success in BI can also be closely linked to the achievement of specific operational and/or strategic goals. For the Open University, for example, the success of their RETAIN project would be measured by a reduction in student dropout rates.
It is worth identifying such targets wherever possible, as not only are they likely to automatically lead you someway towards the formation of SMART targets, but are clearly inherently linked to the strategic performance of the organisation; thereby cementing the relationship between the two.
There can be no absolutes and what is a vital connection or measurement of success for one organisation will not be for another. What is instead important is to work out what is acceptable to your organisation and particularly to the people who will invest in it and rely upon it. Not only is this an important part of the overall system design if you wish to develop a BI solution that is meaningful for your organisation, but it is also essential if you are to be able to measure and demonstrate its eventual success.
Using generic models
Another approach to measuring the performance of your BI capability (or elements of it) is to compare your situation against generic descriptive models of what is perceived to be represent good, or best, practice in the area.
Externally developed ‘maturity models’ and the like have the advantage of having been developed impartially and drawn from the experience of multiple perspective and experiences. They can help provide some welcome distance between you and your own organisation and allow you to examine its strengths and weaknesses from new perspectives and incorporating aspects and elements which might otherwise have not occurred.
By using the same set of metrics it also becomes possible for different organisations to benchmark their performance, or aspects of it, against their peers thereby giving you a further basis for assessment.
They are not foolproof, however, and their generic nature by necessity means that not all aspects of them will be right for your organisation and that they may need to be adapted and tailored accordingly. They can, without careful handling, also encourage the risk of ‘paralysis by analysis’ as for all the information they can give you about your current condition they will not, on their own at least, automatically help you to progress from it.
During the next two sections we present two different models which have been developed as resources to help you measure different aspects of your BI capability which you may well find useful.