Tag Archives: Design-time Automation

Data Transformation Mapping – Can it be Automated?

In my previous post on this subject, I wrote about the neglected problem of data transformation mapping productivity, and its impact on integration project costs, particularly for B2B integration in companies with many customer relationships (and therefore many documents pairs to map).

Although attempts have been made to automate aspects of mapping, it remains largely a manual activity.  Gartner and other industry analysts have published research on automated mapping methods, particularly Dictionary-based mapping, but real-world implementations haven’t put a serious dent in the mapping productivity problem.  Why is that?

Mapping is a large and complex problem space.   It accounts for the single largest part of integration time and effort, encompassing data validation, translation, value derivation, enrichment, aggregation, and routing.  Defining these outcomes involves two kinds of specification, not one:  source-target matching and transformation.  Most attempts to automate mapping have focused intensively on the first part (matching), but very little on the second (data transformations).  But the biggest obstacle to mapping automation is that specifying the right mapping action may require an understanding of business and data contexts and requirements.  And that requires human decision-making.

So to the degree that mapping automation is possible, it must occur in the context of a broader, human-guided mapping process.  Simply defining a source-target dictionary and “lighting the fuse” won’t produce a map that can be used in a production environment.

Integrating automated mapping methods with the human-guided mapping process imposes critical implementation requirements for integration technology providers, including:

  • Unobtrusive integration of automation methods and human mapping decisions in the UI
  • Support for both source-target matching and data transformation aspects of mapping
  • Automation that works with both familiar and unfamiliar document types and combinations
  • Results that can be verified quickly and easily by humans
  • Configurability to suit different circumstances and preferences
  • Ability to “learn” from and adapt past decisions to future mapping situations

Each of these requirements could be subjects for their own blog posts.  The bottom line, however, is that past attempts to automate mapping have ignored or fallen short in all of these areas.

Automating data transformation mapping isn’t easy, but it is possible.  The next post in this series will examine the technology solution strategy taken by EXTOL, with the Smart Mapping feature introduced in EXTOL Business Integrator v2.5.

Data Transformation Mapping – Is it a “Solved Problem”?

Mapping.  We all know how important it is to all kinds of business integration – it defines the data result of an integration process.  Despite the apparent simplicity of drag-and-drop mapping tools, we also know what a pain it can be to get right.  And how the time required to specify and test data transformation maps contributes to the cost of business integration.

Based on years of implementation engagements, EXTOL has determined that mapping can account for up to two-thirds of the time in a medium-to-large project.  The percentage is highest in large conversion projects conducted by companies with many customer relationships.  And the percentage is lower in smaller projects (like onboarding a single partner) and in companies that own value chains and have the power to set the terms of integration with partners.  But even in smaller projects, mapping is usually the single most time-consuming – and costly – design-time activity.

Despite the dominant influence that mapping exerts on life cycle costs, the state-of-the-art for mapping technology has stalled at the classic “line mesh” mapping model, introduced during the early 1990s.  At that time, the simplicity with which data associations could be specified by simply drawing lines between source and target documents was hailed as a marked improvement over coding (which it was), but the line mesh model did not deliver dramatic productivity improvements for complex mapping cases, in which hundreds or thousands of associations and transformations must be specified.

While industry experts concede that the cost of integration remains an important issue, the most time-consuming and costly integration activity – mapping – is treated by most technology vendors as a solved problem.  With the exception of minor advances in user interface design, the problem of mapping cost and productivity has been largely ignored.

The irony here is that integration technology has played such an important role in boosting business productivity.   Automating B2B, application, and data integration processes has enabled businesses of every kind to reduce processing costs, increase business throughput, and reduce errors.  Yet the technologies that have enabled those improvements suffer from productivity deficits of their own.

In my next post in this series, I’ll examine ways in which automation can be applied to data transformation mapping, and how different automation approaches produce very different outcomes.