Data Integration is defined as, “the combining of fragmented data residing in different sources and locations which are aligned to support business goals”. There are many reasons to bring data of different types (flat file, DB2, or even spreadsheets), possibly residing on different servers, to one main location to be integrated together. If you do Electronic Data Interchange (EDI), translating data between an EDI fixed format and application variable format files, then you have already been doing a piece of the data integration puzzle.
EDI may supply data for receivable or payable applications within your ERP (Enterprise Resource Planning) package but what about the warehousing and shipping data? This data may reside on a different server, different locations even a different software package. This is one example and opportunity where Data Integration would be employed.
Understanding the terminology and role in application (data) integration is an important step in planning. Let’s look at a few common integration terms and examples as they apply to the EXTOL Business Integrator (EBI) data integration process.
Middleware: This is the software or management tools that connect a data source (or application) to a data target (or application). EBI is considered a middleware product that has configurable business processes capable of “listening” or being executed to isolate the needed data and perform the integration. This data could be spread across disparate systems and maintained in many different formats/applications. This source data is then transformed to the format needed by the target format and/or presentation.
Endpoints, Connectors and Adaptors: “Endpoints” are a configuration object that identifies the point or location where a process ends or is considered complete. “Connectors” connect two or more Endpoints. “Adaptors” are used by Connectors and carry information such as the database, database tables, table relationships, and database driver (if a database endpoint), the URL, the ID and Password, and other information necessary to link the endpoints.
Schemas: Knowing the location of data is important, but equally critical is to understand the format – whether a database, flat-file, spreadsheet, EDI or numerous others. “Schema” refers to the technical description and organization of that data. But, schemas go a step further…they must be usable electronically. For example, EDI specifications identify transactions, segments, elements, and syntax rules, but if the rules can’t be interpreted by the transformation tool then it really is not schema. Schema is “meta-data” – the technical layout that can be used during transformation of data. To do this, the transformation tool must be capable of importing schema definitions (or provide the ability to be manually created). Consider schema or meta-data as “data that describes the data”. Through schemas, EBI can link a source to a target endpoint and define rules that are interpreted using the schemas.
A Source schema is the format of the data in its’ current form. A Target schema defines the format of the data in its desired ending format. To transform the data from Source to Target, rules for manipulating the source should exist. These rules are commonly defined as Maps, Rule Sets, Message Classes, and other references.
Data Integration includes numerous possibilities with advanced functions to support the many facets of emerging businesses today. Having the right integration tool capable of reducing (or eliminating) manual (and custom) programming is a necessary trend to meet requirements and remain competitive.