Category Archives: Integration Architecture

What is “Smart Mapping”?

In my two previous posts on the subject of automated mapping, I examined the importance of automation as a way to reduce design-time integration costs and delivery time, and the challenges of applying automation to the mapping process, in particular.

If you’ve used or seen a demo of our original EDI Integrator product for System i, you know that EXTOL has a long history of innovation in this area.  In 1994, we introduced an automated source-target matching feature called “Automapping”, and three years later, the “Advanced Automapping” feature debuted, giving customers the ability to generate large portions of new maps, based on previous mapping examples.

Two years ago, we renewed our research into automated mapping methods, with the goal of delivering a new automated mapping implementation for EXTOL Business Integrator, the modern, cross-platform integration broker we introduced in 2003.  But instead of simply replicating the Automapping and Advanced Automapping features, we set out to push the boundaries of state-of-the-art mapping automation, by targeting a much more stringent set of requirements:

  • Mapping must be applicable to any mapping situation, including both familiar and unfamiliar document types and source-target combinations
  • Both aspects of mapping – source-target element matching and generation of data transformations – should be supported by the automated mapping mechanism
  • The automated mapping feature should be able to “learn” from and adapt past decisions to future mapping situations, incorporating best practices that evolve, over time
  • Automated mapping should integrate unobtrusively in the UI, allowing a blending of human and automated mapping actions and results
  • Users must retain control over the application of results generated by the automated mapping feature, with the ability to selectively apply generated matches and transformations
  • The behavior of the automated mapping feature must be configurable to suit different business circumstances and user preferences

These requirements went far beyond the capabilities of available automated mapping implementations, and called for the invention of a new mapping architecture and new automated mapping methods.

The resulting implementation, “Smart Mapping”, was introduced in EXTOL Business Integrator v2.5, in October of 2010.  Smart Mapping is an automated mapping implementation that integrates with the EXTOL Integration Studio Ruleset Editor, the drag-and-drop, rule-based mapping tool introduced originally in 2003.  It consists of multiple automated matching and rule generation methods that are combined through a user-weighted fuzzy logic layer, and can generate mapping results at the element, structure, and document root levels.

What makes Smart Mapping stand out from past attempts to automate mapping is the ease with which powerful mapping methods can be brought to bear on virtually any B2B, data, or application mapping situation, without imposing new skill requirements or intruding on the user interaction model.

We believe that Smart Mapping is not only a boon for companies that need to deliver sophisticated integrations faster and more cost-effectively, but is also interesting and important technology, in its own right.  Over the coming weeks, we will be posting additional insights into the Smart Mapping approach and the technology that underlies it.  Stay tuned.

Data Transformation Mapping – Can it be Automated?

In my previous post on this subject, I wrote about the neglected problem of data transformation mapping productivity, and its impact on integration project costs, particularly for B2B integration in companies with many customer relationships (and therefore many documents pairs to map).

Although attempts have been made to automate aspects of mapping, it remains largely a manual activity.  Gartner and other industry analysts have published research on automated mapping methods, particularly Dictionary-based mapping, but real-world implementations haven’t put a serious dent in the mapping productivity problem.  Why is that?

Mapping is a large and complex problem space.   It accounts for the single largest part of integration time and effort, encompassing data validation, translation, value derivation, enrichment, aggregation, and routing.  Defining these outcomes involves two kinds of specification, not one:  source-target matching and transformation.  Most attempts to automate mapping have focused intensively on the first part (matching), but very little on the second (data transformations).  But the biggest obstacle to mapping automation is that specifying the right mapping action may require an understanding of business and data contexts and requirements.  And that requires human decision-making.

So to the degree that mapping automation is possible, it must occur in the context of a broader, human-guided mapping process.  Simply defining a source-target dictionary and “lighting the fuse” won’t produce a map that can be used in a production environment.

Integrating automated mapping methods with the human-guided mapping process imposes critical implementation requirements for integration technology providers, including:

  • Unobtrusive integration of automation methods and human mapping decisions in the UI
  • Support for both source-target matching and data transformation aspects of mapping
  • Automation that works with both familiar and unfamiliar document types and combinations
  • Results that can be verified quickly and easily by humans
  • Configurability to suit different circumstances and preferences
  • Ability to “learn” from and adapt past decisions to future mapping situations

Each of these requirements could be subjects for their own blog posts.  The bottom line, however, is that past attempts to automate mapping have ignored or fallen short in all of these areas.

Automating data transformation mapping isn’t easy, but it is possible.  The next post in this series will examine the technology solution strategy taken by EXTOL, with the Smart Mapping feature introduced in EXTOL Business Integrator v2.5.

Data Transformation Mapping – Is it a “Solved Problem”?

Mapping.  We all know how important it is to all kinds of business integration – it defines the data result of an integration process.  Despite the apparent simplicity of drag-and-drop mapping tools, we also know what a pain it can be to get right.  And how the time required to specify and test data transformation maps contributes to the cost of business integration.

Based on years of implementation engagements, EXTOL has determined that mapping can account for up to two-thirds of the time in a medium-to-large project.  The percentage is highest in large conversion projects conducted by companies with many customer relationships.  And the percentage is lower in smaller projects (like onboarding a single partner) and in companies that own value chains and have the power to set the terms of integration with partners.  But even in smaller projects, mapping is usually the single most time-consuming – and costly – design-time activity.

Despite the dominant influence that mapping exerts on life cycle costs, the state-of-the-art for mapping technology has stalled at the classic “line mesh” mapping model, introduced during the early 1990s.  At that time, the simplicity with which data associations could be specified by simply drawing lines between source and target documents was hailed as a marked improvement over coding (which it was), but the line mesh model did not deliver dramatic productivity improvements for complex mapping cases, in which hundreds or thousands of associations and transformations must be specified.

While industry experts concede that the cost of integration remains an important issue, the most time-consuming and costly integration activity – mapping – is treated by most technology vendors as a solved problem.  With the exception of minor advances in user interface design, the problem of mapping cost and productivity has been largely ignored.

The irony here is that integration technology has played such an important role in boosting business productivity.   Automating B2B, application, and data integration processes has enabled businesses of every kind to reduce processing costs, increase business throughput, and reduce errors.  Yet the technologies that have enabled those improvements suffer from productivity deficits of their own.

In my next post in this series, I’ll examine ways in which automation can be applied to data transformation mapping, and how different automation approaches produce very different outcomes.

Understanding Open Source Licenses (Part Two)

Part one of this blog series discussed permissive and weak copyleft open source licenses.  Permissive licenses allow you to use and make changes to the software without having to distribute the source code to your application.  The weak copyleft licenses allow you to use the software without distributing your source code, but require you to use their license once you make changes to the source code (meaning release whatever you’ve changed).  Part two focuses on the strong copyleft licenses, and I list some companies that contribute to the open source community. Continue reading

Understanding Open Source Licenses (Part One)

The past 14 years of my computer programming career have been a joy thanks to the existence of open source software.  Using open source libraries allow me to create software quickly without having to implement certain tasks other developers around the world have completed.  I’m able to dig through the source code to learn how certain technologies work.  I’ve even become a better programmer because of reading other people’s code.  When building a commercial product that uses open source software, educating yourself on the various open source licenses is mandatory to avoid litigation over how you use that open source software.  The purpose of this two-part blog series is to give a brief overview of the major open source licenses and list some applications that use an open source license and companies that contribute and distribute open source software using the discussed licenses.  The focus of these blogs is for companies that wish to sell and distribute their applications that use open source software.  If your application will not be distributed to customers (such as in a Software as a Service scenario), then all but the last license (discussed in part two) do not apply. Continue reading