Improving an organisation's ability to use data better has been a long-standing goal of CIOs. With the exponential growth in data - which has led to the "big data" phenomenon - there's now an even more compelling need to use the management of information as a competitive advantage.
The term "business intelligence" or BI was defined by an IBM employee in 1958 as the capability of collecting, integrating, and analysing internal and external data to generate knowledge and value for the organisation. These days BI is an umbrella term for all data-related activities - data integration, data warehousing, reporting, and analysis - designed to provide decision makers with accurate and on-time information to help them attain complete and timely insight into their business. This encompasses business process decision support at the strategic, tactical, and operational levels of an organisation; supported by so-called "underwater" activities to manage the data and architecture (such as data governance, data security, and master data management).
In a fast-moving information-based economy, many organisations can't generate clear, current, or appropriate information; being able to do so can become a key competitive advantage. Respondents to a 2006 Accenture survey of U.S. and U.K. middle managers reported that they felt only about half of the information they received was valuable, that they regularly use the wrong information (on average, once or twice a week, depending on which country was reporting), and that they consistently missed valuable information every day (four to five times a week in the United States, and five to six times a week in the United Kingdom).
A 2009 Gartner report stated that because of a lack of information, more than 35 percent of the top 5,000 global companies regularly fail to make insightful decisions about significant changes in their businesses and markets. Reuters reported in 2006 that 43 percent of managers believe that important decisions are delayed and the ability to make decisions is affected by too much information. A 2008 survey conducted by the Economist Intelligence Unit found that only one in 10 executives felt they received the information they needed, and over seven in 10 believed that management decision making was only moderately efficient or worse.
The exogenous pressures to manage data more effectively are also increasing. New regulations such as the Basel III rules in banking and Solvency II Directive for insurance are demanding that organisations manage their corporate data much more effectively and transparently (and there are many more examples). Cost and competitive pressures added by the global economic downturn are forcing companies to be more effective and efficient with what they have; and capital markets are putting them under more scrutiny as a result.
The amount of corporate data is also growing exponentially. According to a 2012 Forrester Research hardware survey, between 2010 and 2012 the average overall company data storage footprint grew by 60 percent. In 2009 IDC reported that enterprise data stores would grow by an average of 60 percent annually, with considerable variation among industries. An Accenture study in 2010 found that structured data and semi-structured data (including databases and messaging applications) were growing at a rate of between 30 and 50 percent every year.
It's this exponential data growth that has generated the big data phenomenon. Big data usually refers to having to manage rapid streams of data sets that go way beyond the capacity of commonly used BI tools and practices. Gartner defines it this way: BI uses descriptive statistics with data that has a high information density to measure things, detect trends, etc.; big data uses inductive statistics on data that has a low information density but whose huge volume provides a statistically significant sample to apply laws (such as regression analysis) for an element of prediction.
BI capability therefore is desirable to have. An integrated capability facilitates consistent sourcing, cleansing, storing, collating, and delivery of critical information in a timely manner.
The B is for Business
A successful deployment of BI capability has the potential to transform an organisation. BI has been used to identify cost-cutting ideas, uncover new business opportunities, roll enterprise resource planning data into accessible reports, react quickly to retail demand changes, and optimise pricing.
BI offers multiple opportunities to save money by optimising business processes and driving better decisions. For example, with the help of BI tools a leading global manufacturing firm realised that it had been double-paying its suppliers and had lost nearly a million dollars in 2000. Employees of the city of Albuquerque used BI software to identify opportunities to cut mobile phone usage, overtime, and other operating expenses - saving the city $2 million over two years.
Although the potential rewards from deploying and operating a BI capability can be great, the challenges are also considerable. According to a 2011 survey by Decision Path Consulting, many if not most organisations find putting BI into operation difficult, more complicated, and more costly than they had initially anticipated; likewise, they falter in quantifying the business benefits realised from the deployment; and they're generally only partially satisfied with the results at best. Gartner has reported that between 70 and 80 percent of BI projects fail.
These project failures are marked by common problems: user resistance, lack of adoption, incorrect use and understanding of BI tools, inaccurate source system data, lack of organisational understanding of the business processes to be affected, and a lack of leadership. These problems can be summarised into four categories:
- Lack of direction;
- Lack of organisational readiness;
- Poor project execution; and
- Lack of business engagement or user impact.
When undertaking a BI initiative, the first question that must be answered is, why are we doing this? Many BI projects fail because the companies go into them lacking a compelling and well-communicated value proposition agreed on by all key stakeholders. The business case must be supported by a BI strategy to answer not just why it will be delivered, but also how and when, and how success will be measured. The organisation must take strategy seriously, attain a broad base of support among key stakeholders, and align goals for the BI investment with corporate goals to avoid multiple, uncoordinated initiatives within departments.
Many organisations embark on expensive, sophisticated BI projects without taking the time to assess whether they're ready. BI readiness involves understanding current business processes and data within the organisation, examining the challenges and risks inherent in BI implementations, ensuring people with the right skills are in place, and locking down sponsorship among senior executives from the C level downwards.
Even if the company has well-articulated requirements, a sound BI strategy, and a solid tool set, other aspects can go haywire, such as delivery schedule or budget, or the promised benefits may not materialize. For example, perhaps user queries may run too slowly or BI reports don't get delivered because the jobs that fed them failed with no support, or requests for enhancements to the system may go nowhere. A lack of quality BI project execution can lead, at best, to dissatisfaction and resistance among users, or at worst, a complete refusal to use the new technology (an outcome that leads to the next typical problem).
A common scenario is that organisations may have invested in and deployed a BI solution, but they're not sure how the business is benefiting from it or whether it's actually using the new system. This lack of user impact, a result of poor engagement with the business, can be difficult to recover from; once the credibility is gone, so are the benefits of the BI initiative. There should be a strong change management initiative in parallel with the development of the BI technology; and from the initial requirements gathering process, requirements should be explicitly linked to information needs for business processes with clear metrics on use and benefits.
How do organisations successfully deliver BI projects then? We can distill the common issues and resolutions into 10 best practices.
- Be ready. Take inventory of BI skills and competencies in the organisation, assign clear roles and responsibilities, understand business processes, and understand current systems and the data they generate.
- Focus on the business. The organisation should treat any IT initiative as a business-led - not IT-centric - project. IT and business teams should work together from day one. The new BI technology should be aligned with business processes and current needs to be pragmatic and useful. Focus on the business needs. Be flexible and pragmatic to build relationships and to drum up support for the new BI solution.
- Business case. There should be a compelling, solid business case for the BI initiative that has a broad base of support within the organisation and is aligned with corporate goals. Soft benefits are as important as hard benefits for BI projects (for example, easy access to "single version of the truth" data).
- Executive sponsorship. There needs to be clear leadership from the C-level on the need and benefits of the BI initiative, a single sponsor is preferred for clarity of direction but a broad base of support needs to be developed. Realistic expectations have to be set by the executive sponsor, but at the same time the overall objective should be to transform the business to maximise the return from investment.
- Communicate early and often. Use newsletters, seminars, dashboards, intranet sites, and anything else that's highly visible to evangelise the benefits of BI. Find BI successes and keep explaining the value. Provide trophies for the best projects. Provide training and help for any and all changes.
- Think big, but start small. Target roll-out of BI to 100 percent of the organisation, but deliver BI in quick, incremental releases (quick wins) to demonstrate the benefit and maintain interest within the organisation.
- Focus on people, not technology. People skills can make or break projects. In 2008 SAP reported that three-quarter of project success was determined by elements outside of data and technology. Provide incentives and reasons for people to use the system. Link BI goals to what executives care about and include objectives in performance management systems.
- Data quality. Without good source data it doesn't matter how good the rest of the project is - the BI project will fail. Ensure that current data is understood - especially the data integrity and use in business processes.
- Execute well. Use specific BI methodology and governance, ensure that the delivery team has the right skills and profile, and ensure that BI is built on a solid data architecture foundation.
- Consider alternative resourcing models. Hire people with technical expertise or use consultants to "jumpstart" the BI initiative. A long start-up time can frustrate and stall the best efforts early. Consider commoditising common tasks to deliver the BI initiative in a cost-effective way (such as through outsourcing or offshoring).
BI can be challenging to do well, but the potential benefits make it worth the effort. Many organisations don't have the internal capability to deliver BI solutions well at the start of the journey. To these entities we suggest building up capability to evolve and maximise the return on investment.