Three main categories
Also, the BI systems are derived from the operational information systems. The obstacles refer to the ten most common and most important forces and risks that can make or break a Business Intelligence project, which are divided into three main categories:
- Culture: four ‘wrong’ behavioral cultures;
- Technology: the complexity of ETL and ‘technology’-push;
- Content: the amount, the quality, and/or the absence of data.
It is often the case that not just one or two obstacles stand in the way of a successful project result, but almost all of them. This is why managing Business Intelligence projects is such a huge challenge.
Culture and politics
Often, the ‘fixed’ patterns and traditional ideas of employees and managers within the organization are limiting factors when it comes to making data (in the organization) flourish. We distinguish four behavioral cultures that have an inhibiting effect on the success of Business Intelligence:
1. Fragmented culture
A culture in which each department operates individually, does business in its own specific way, and uses its own methods of measurement. This is also known as the ‘island culture’. Because everyone uses his (or her) own definitions and measurement tools, the organization is unable to obtain an overall picture of how things (actually) stand. The upside is that the developers of these ‘island’ databases can often respond quickly to the changing information needs of their managers. The downside, however, is that there are many different developers (one for each department) and together they invest a lot of time in information supply (for each department). Still, sometimes it takes years for an organization to decide to build a Central Data Warehouse and to phase out the ‘old’ (disparate) systems. This is often because project managers (and line managers) who are involved in the BI project lack decisiveness and courage. organizational politics and mutual trust also play a role.
2. Culture of power and gut feeling
Managers who make decisions purely based on gut feeling and not based on facts, often find it difficult to accept the new reality that Business Intelligence represents. A survey on a website for the BI community asked people to respond – true or false – to the following statement: “The managers in our organization manage on the basis of gut feeling”. The result: nearly 80% of the respondents agreed (or partly agreed). For this reason (amongst others), managers are likely to question the Business Intelligence system, come up with their own (little) systems, and nervously avoid more transparency. They may also fear that their status and authority will be affected (and diminished) because, with the arrival of the BI system, they are no longer the only source of information.
3. Financial culture
Managers and shareholders too often still assess the organization purely based on financial indicators. This is often due to short-term focused bonus systems at the top level of the organization. It is important to see through, monitor, and manage all sides (and aspects) of the organization instead of focusing solely on the balance sheet and the profit and loss statement.
4. Traditional IT culture
We all know the type: they are brilliant in the operation and optimization of ‘old’ technologies and are very efficient programmers: the traditional IT nerds who have worked for the same company for years and years. These IT people date back to the time when punch cards were commonplace and memory was very scarce. They understand the mainframe inside out. At the time, they had to program very efficiently, simply because the disk capacity of the machines was limited. This explains their unequivocal aversion to anything that is being stored twice; both data and software. They are against redundancy, because their motto is that data should only be stored in one place. Data redundancy, however, is important in BI systems. This is entirely opposite to traditional thinking, and thus blasphemy in the eyes of the traditional developer. This is why BI projects often take off slowly or even fail. Fortunately, the overly traditional IT expert is a dying breed.
Technology
We distinguish two important problem areas with regard to obstacles of a technical nature:
5. The complexity of ETL
Extracting and integrating data from different source systems is difficult. According to a renowned research institute, ETL takes up 70 to 80 percent of the total (technical) costs involved with building a BI system. This is sometimes due to aspects such as (poor) quality of source data, large amounts of data, or so-called ‘back-dated transactions’, but may also be due to the closed character of the source systems or the complexity of the transformation. In some cases, we simply want to do too much at one time, while we should in fact divide the ETL process into small steps. A balanced architecture is also important. We can minimize the risks somewhat by deploying an ETL tool, but working with bulldozers (ETL tools) requires a solid surface (balanced architecture) too. During the definition of the extraction process, we will often also need to collaborate with many different administrators of source systems, each of which has its own priorities and interests.
6. The technology push
The IT department forces a data warehouse and BI technology onto the organization without taking into account the wishes and requirements of users and without properly aligning technology and information needs. In some cases, they set out to work without a (proper) business case. IT builds the data warehouse almost without consulting managers, users and experts in the business. The result: a BI system that is hardly used and which delivers no results. This should not be happening anymore! The days in which IT departments could operate with impunity have gone. We can prevent this ‘technology push’ by aligning IT projects with both the operations and the policies in the organizations and through fully exploiting existing assets. Resources that were hidden behind the thick walls of the data center are now getting the attention of the business itself. Today, data is seen as an asset, which we must apply and employ in accordance with economic principles.
Content
Obstacles regarding content are the least difficult to get around because most Business Intelligence tools today take these into account. This is particularly true for the following three obstacles:
7. Polluted data
In most cases, polluted data is a result of errors or problems in the production system that registers and stores the data: the data entered is not sufficiently monitored and validated. Consequently, the data quality depends highly on either the accuracy of internal staff or the reliability of external data suppliers. Because of polluted data, the ETL process becomes much more complex and more difficult to maintain. This in turn can lead to long(er) data warehouse load times and sometimes even to loss of confidence with regard to the data warehouse.
8. Data without context
If we have large amounts of data at our disposal but we do not know the meaning (or context), then we have a major problem. If the data and its structure and meaning are not clearly described – metadata -, we cannot determine what we want to measure, and modeling the data in the data warehouse coherently is even more problematic. The better the data is described, the faster we can implement BI projects.
9. Large volumes of data
Large amounts of data can form a barrier when we start using Business Intelligence. The production systems of larger organizations often deal with very large volumes of data (hundreds of millions) on a daily basis. All this data needs to be processed in the data warehouse within a specific period. It is not always possible to transfer these large volumes of data directly, in which case we must come up with a clever solution, such as: incremental loading; leaving out metadata that is already known; compressing data or, if possible, widening the frame within which the processing can take place. Additionally, large volumes of data may cause delays during the development and testing of the BI system. It is then recommended to work with ‘balanced’ test sets (or developing sets), in order to develop and test functionalities faster.
10. Absence of data
If an organization does not register the signals that are important for effective business operations, thenits senses malfunction. If this is the case, the organization must return to previous steps in the major Business Intelligence cycle in order to record the required signals in their source system. Unfortunately this happens all too often while in fact this issue should not be resolved within the BI project: registering operational data is not a primary responsibility of Business Intelligence as the BI cycle shows. Business Intelligence ‘only’ determines what data is required for (more) effective decision making.
Personal or organizational risks
In addition to these common obstacles and risks, we sometimes need to deal with specific personal or organizational risks. These may relate to a lack of experience in the fields of Business Intelligence or data warehousing: project members who re-invent the wheel; selecting the wrong advisors; developing and applying an architecture that does not fit, etc. It is therefore essential that (business) managers, the project manager, and employees all have the necessary (behavioral) Business Intelligence skills so that each of them can fulfill his (or her) role and carry responsibility during the project-based development of Business Intelligence systems, whilst gradually creating an Intelligent organization.
Daan van Beek MSc, speaker, trainer and author of the BI & AI book 'Data Science for Decision-Makers and Data Professionals'. He is the CEO of the Passionned Group and you can follow Daan on Twitter.
