Data Integration: challenges and overview of the tools

Author: Daan van Beek
Senior Data Integration consultant

Table of Contents

Featured image | Data Integration - advice, tools & training

Manage your data flow

Data Integration helps organizations manage their data and data flows efficiently. Organizations seldom use one system to capture all their company data. They frequently use different systems for specific purposes like customer relationship management (CRM), Enterprise Resource Planning (ERP), finance, Human Resources, and so forth. In many cases, the same customer, product, or employee data is captured and stored in different systems in different formats for different purposes.

Many organizations face 3 big data integration challenges

The use of different systems gives organizations a lot of flexibility. On the other hand, the huge amount of relevant (and also irrelevant) data outside the company creates three big data integration challenges for every company:

How to make sure that data elements are not getting obsolete? Changes in one system are not automatically transferred to other systems.
How to create one view of the customer or employee data? Data in the different systems is not automatically combined in one database.
How to combine external (big) data with company data easily? Read more about Big Data.

Different technologies: EAI & ETL

These three challenges are addressed in Data Integration. To get rid of the first challenge, organizations have to implement Enterprise Application Integration (EAI) technology. Often a Service Oriented Architecture (SOA) is used. A data warehouse has to be implemented to overcome the second challenge. In both cases, you should consider using ETL tools. To deal with the third challenge you also need a Business Intelligence tool with Big Data capabilities.

The fundaments of Business Intelligence

To be able to make better decisions using KPIs and key insights across the enterprise, the data needs to be integrated and combined into one single system. Business Intelligence can’t be successful without a proper data integration layer. That layer takes care of unified definitions, improving the data quality, and combining data from different sources into a single table or cube, thus facilitating very fast response times. Data integration is the beating heart of the Intelligent Organization and it serves several vital functions. Also, read our article ‘9 reasons to build a data warehouse’.

Overview of Data Integration tools

Since 2007, we’ve been investigating a large number of data integration tools based on over 190 criteria that play a major role in the quick and successful development of a BI infrastructure. In our study, we concentrated on products that could be (relatively) easily used by either business or IT professionals with a minimum of specialized training. In our fully independent analysis (updated regularly) we have divided the products into four separate categories to avoid comparing apples with oranges.

1. Pure ETL Products

These data integration tools are independent of the database and the Business Intelligence tool with which they will probably be used. The companies do not rely on another product providing the functionality they offer, and they allow you to migrate to another (relational) database or another Business Intelligence tool without having to substantially change your integration processes.

List of pure ETL products:

2. Database Integrated

These data integration products are supplied as an option when you buy the database software; some of the functionality is built into the database and not available separately in the ETL tool itself. The ETL tool is database-dependent and will not work with another database, which is an intended part of the design. Replacement of the database would entail completely rewriting the integration processes.

List of database-integrated ETL products:

Oracle Warehouse Builder
SQL Server Integration Services

3. Business Intelligence Integrated

These are products from the same supplier as the BI software. In many cases these are separate products and the supplier will claim that they can be used independently of the BI tool. This is probably true but because of their use of proprietary metadata, and sometimes the same scripting tools as the BI product, they are seldom used separately.

List of BI-integrated ETL products:

SAP Data Services
SAS Data Integration Studio

4. Niche Products

These are products that don’t fit well into any of the above-mentioned groups, but still have considerable ETL functionality in them. In most cases they’re not as complete as the other data integration tools we’ve looked at, but they’re often in a totally different price class and are very usable to solve certain problems. Traditionally we have included the Open Source products in this category since they lacked the full functionality and customer experience to qualify for the main categories.

Click here to see a complete list of data integration tools.

Interested in our data integration services and independent advice?

Would you like to be informed about Data Integration and get professional, 100% vendor-independent advice? Contact us and we will get back to you as soon as possible.

About Passionned Group

We are Passionned, experts in data integration and Business Analytics. Our experienced consultants help companies and other institutions build an intelligent organization. Every other year we present the Dutch BI & Data Science Award™ to the Smartest organization in the Netherlands.