Author Archives: Jim OLeary

Data Transformation Mapping – Can it be Automated?

In my previous post on this subject, I wrote about the neglected problem of data transformation mapping productivity, and its impact on integration project costs, particularly for B2B integration in companies with many customer relationships (and therefore many documents pairs to map).

Although attempts have been made to automate aspects of mapping, it remains largely a manual activity.  Gartner and other industry analysts have published research on automated mapping methods, particularly Dictionary-based mapping, but real-world implementations haven’t put a serious dent in the mapping productivity problem.  Why is that?

Mapping is a large and complex problem space.   It accounts for the single largest part of integration time and effort, encompassing data validation, translation, value derivation, enrichment, aggregation, and routing.  Defining these outcomes involves two kinds of specification, not one:  source-target matching and transformation.  Most attempts to automate mapping have focused intensively on the first part (matching), but very little on the second (data transformations).  But the biggest obstacle to mapping automation is that specifying the right mapping action may require an understanding of business and data contexts and requirements.  And that requires human decision-making.

So to the degree that mapping automation is possible, it must occur in the context of a broader, human-guided mapping process.  Simply defining a source-target dictionary and “lighting the fuse” won’t produce a map that can be used in a production environment.

Integrating automated mapping methods with the human-guided mapping process imposes critical implementation requirements for integration technology providers, including:

  • Unobtrusive integration of automation methods and human mapping decisions in the UI
  • Support for both source-target matching and data transformation aspects of mapping
  • Automation that works with both familiar and unfamiliar document types and combinations
  • Results that can be verified quickly and easily by humans
  • Configurability to suit different circumstances and preferences
  • Ability to “learn” from and adapt past decisions to future mapping situations

Each of these requirements could be subjects for their own blog posts.  The bottom line, however, is that past attempts to automate mapping have ignored or fallen short in all of these areas.

Automating data transformation mapping isn’t easy, but it is possible.  The next post in this series will examine the technology solution strategy taken by EXTOL, with the Smart Mapping feature introduced in EXTOL Business Integrator v2.5.

Data Transformation Mapping – Is it a “Solved Problem”?

Mapping.  We all know how important it is to all kinds of business integration – it defines the data result of an integration process.  Despite the apparent simplicity of drag-and-drop mapping tools, we also know what a pain it can be to get right.  And how the time required to specify and test data transformation maps contributes to the cost of business integration.

Based on years of implementation engagements, EXTOL has determined that mapping can account for up to two-thirds of the time in a medium-to-large project.  The percentage is highest in large conversion projects conducted by companies with many customer relationships.  And the percentage is lower in smaller projects (like onboarding a single partner) and in companies that own value chains and have the power to set the terms of integration with partners.  But even in smaller projects, mapping is usually the single most time-consuming – and costly – design-time activity.

Despite the dominant influence that mapping exerts on life cycle costs, the state-of-the-art for mapping technology has stalled at the classic “line mesh” mapping model, introduced during the early 1990s.  At that time, the simplicity with which data associations could be specified by simply drawing lines between source and target documents was hailed as a marked improvement over coding (which it was), but the line mesh model did not deliver dramatic productivity improvements for complex mapping cases, in which hundreds or thousands of associations and transformations must be specified.

While industry experts concede that the cost of integration remains an important issue, the most time-consuming and costly integration activity – mapping – is treated by most technology vendors as a solved problem.  With the exception of minor advances in user interface design, the problem of mapping cost and productivity has been largely ignored.

The irony here is that integration technology has played such an important role in boosting business productivity.   Automating B2B, application, and data integration processes has enabled businesses of every kind to reduce processing costs, increase business throughput, and reduce errors.  Yet the technologies that have enabled those improvements suffer from productivity deficits of their own.

In my next post in this series, I’ll examine ways in which automation can be applied to data transformation mapping, and how different automation approaches produce very different outcomes.

Small- versus Large-Scale Provisioning

As applied to business integration, the term “provisioning” has many meanings, but overall, it refers to the process of defining integration endpoints and establishing connections and integration processes between them.   If an integration service that connects a pair of endpoints is simple and tightly constrained – for example, a data syndication service with a fixed process model and limited output options – provisioning can be as simple as selecting from a fixed list of connection and data format delivery options, and specifying the delivery endpoint’s address.

In most cases, however, business integration provisioning involves more steps, because the business problem to be solved requires tailored integration between some set of sources and targets, e.g., integration of an XML transaction set with a Warehouse Management System.  Those steps might include definition or specification of endpoints, communication and interface connections, documents / messages and envelopes, data routing, business processes, and data transformations.  By combining building blocks that implement such object types, you can solve most kinds of business-to-business, application, and data integration problems. Continue reading

Networked vs. Hosted Managed Services – What’s Your Religion?

Having just returned from a 2-week vacation, I’ve had plenty of time to think about things that my work schedule normally pushes to the background – music, hobbies, investments (don’t ask), and our surreal national political scene. After listening to some of the bizarre preconceptions and arguments about health care reform and US economic policy (throws shoe at TV), it’s hard to avoid the conclusion that many of our beliefs are so ingrained that they approach the status of “religious” tenets.

As business and IT professionals, we like to think that we operate in a reality-based world, where facts and good sense dominate. But the same factors that muddy public policy decisions – incomplete and inaccurate information, the lack of a contextual model that can assist in predicting outcomes, and inherent biases toward or away from certain solution approaches – are just as influential in the decisions we make, every day.

One example that’s top-of-mind, for me, is the debate over managed services delivery models.  EXTOL and other companies in the B2B integration space have offered managed services for years, but cost-cutting measures and our shrinking economy have pushed managed services to the fore, of late.  We are still seeing greater demand in the market for self-managed B2B integration than for 3rd party managed services, but battered budgets, scarce IT resources, and shifting business priorities make managed services more attractive than ever.

Deciding between self-managed B2B integration and managed services integration usually comes down to financial and budgetary factors (capex vs. operating budgeting, front-loaded vs. ongoing expenses, etc.). While “religion” can come into play here, decisions at this level are usually based on clear-cut business priorities. Continue reading

B2B to Be

For my inaugural post on this blog, I want to revisit one of those “solved problems” that still dogs many of the companies we talk with, namely, how to handle B2B integration requirements that don’t involve standard EDI. Companies still find it difficult to cope with the full range of B2B connections and content types needed to integrate with large and small trading partners, including:

  • Standard EDI (and in some cases, EDI that does not fully conform to standards)
  • “Standard” XML, which ranges from well-developed, horizontal standards like RosettaNet to hundreds of loosely-defined vertical transaction sets
  • EDI-like flat file standards (most of these are older, vertically-focused cases)
  • EDI-based web forms
  • Proprietary, partner-defined flat files
  • Proprietary, partner-defined spreadsheets
  • Proprietary, partner-defined web portals
  • Proprietary, partner-defined documents sent by email or fax

Did I miss any? Probably. But the point is that standard EDI is just one of numerous conventions used for B2B integration.  Of course, standard X12 and EDIFACT EDI are still the mainstay of B2B integration. And there is little evidence to suggest that companies are ready to invest in replacing all of their EDI connections with something “better”.  In fact, EDI adoption is increasing.

Continue reading