Tag Archives: extol

What is “Smart Mapping”?

In my two previous posts on the subject of automated mapping, I examined the importance of automation as a way to reduce design-time integration costs and delivery time, and the challenges of applying automation to the mapping process, in particular.

If you’ve used or seen a demo of our original EDI Integrator product for System i, you know that EXTOL has a long history of innovation in this area.  In 1994, we introduced an automated source-target matching feature called “Automapping”, and three years later, the “Advanced Automapping” feature debuted, giving customers the ability to generate large portions of new maps, based on previous mapping examples.

Two years ago, we renewed our research into automated mapping methods, with the goal of delivering a new automated mapping implementation for EXTOL Business Integrator, the modern, cross-platform integration broker we introduced in 2003.  But instead of simply replicating the Automapping and Advanced Automapping features, we set out to push the boundaries of state-of-the-art mapping automation, by targeting a much more stringent set of requirements:

  • Mapping must be applicable to any mapping situation, including both familiar and unfamiliar document types and source-target combinations
  • Both aspects of mapping – source-target element matching and generation of data transformations – should be supported by the automated mapping mechanism
  • The automated mapping feature should be able to “learn” from and adapt past decisions to future mapping situations, incorporating best practices that evolve, over time
  • Automated mapping should integrate unobtrusively in the UI, allowing a blending of human and automated mapping actions and results
  • Users must retain control over the application of results generated by the automated mapping feature, with the ability to selectively apply generated matches and transformations
  • The behavior of the automated mapping feature must be configurable to suit different business circumstances and user preferences

These requirements went far beyond the capabilities of available automated mapping implementations, and called for the invention of a new mapping architecture and new automated mapping methods.

The resulting implementation, “Smart Mapping”, was introduced in EXTOL Business Integrator v2.5, in October of 2010.  Smart Mapping is an automated mapping implementation that integrates with the EXTOL Integration Studio Ruleset Editor, the drag-and-drop, rule-based mapping tool introduced originally in 2003.  It consists of multiple automated matching and rule generation methods that are combined through a user-weighted fuzzy logic layer, and can generate mapping results at the element, structure, and document root levels.

What makes Smart Mapping stand out from past attempts to automate mapping is the ease with which powerful mapping methods can be brought to bear on virtually any B2B, data, or application mapping situation, without imposing new skill requirements or intruding on the user interaction model.

We believe that Smart Mapping is not only a boon for companies that need to deliver sophisticated integrations faster and more cost-effectively, but is also interesting and important technology, in its own right.  Over the coming weeks, we will be posting additional insights into the Smart Mapping approach and the technology that underlies it.  Stay tuned.

EXTOL’s Migration Assistant

EXTOL’s Migration Assistant provides a path for the users to upgrade the schemas they’re using in a Transformation in place.  The Migration Assistant is a first class archivable object that can be found in its own node in the Workbench.  This feature works for all syntax categories.  Use cases would be an upgrade of an EDI version (i.e. 4010 850 to 5010 850), upgrade of an .xsd (i.e. elements/attributes added or removed from an xml schema or other structural changes), changes made to a database (i.e. tables or columns added or removed), upgrade of a delimited flat file (i.e. fields added or removed), or the upgrade of a Spreadsheet (i.e. cells or tabs added or removed).

The Migration Assistant is a two step process.  First the user is required to create a map called a Schema Association Map between the “Old Schema” and the “New Schema”.  This map is created by using a Ruleset like editor called the Schema Association Map Editor.  This can be done manually or it can use the Auto Association Algorithm.  The Auto Association Algorithm is used by toggling the “Auto-Associate Children” button in the toolbar and dragging a desired source node to a desired target node.  All child nodes of the desired source will be auto associated with all child nodes of the desired target.  The recommended Associations are displayed in an Approval Editor where the user can review and change Associations.  The suggested Associations are ranked by a confidence level.  It is recommended for schemas that are drastically different that if it is desired to use the Auto Association Algorithm that the user start at lower level source elements that they are confident belong with lower level target elements and work their way up to the root.  For schemas where the changes between the “Old” and “New” schemas are reasonable (no huge structural changes) it is permissible to associate the two roots and let all of their children be auto associated, but one should carefully review the results.

The second step to the process is to apply the Schema Association Map to the desired Ruleset.  This can be applied in a batch mode (used when a schema is being used in multiple Rulesets) or it can be applied to a single Ruleset.  The Schema Association Map can be applied to a single Ruleset by right clicking on a Ruleset in the Rulesets tab and clicking Convert Ruleset->Source/Target.  A dialog will be displayed asking the user to select which Schema Association Map they want to apply.  Once the proper Schema Association Map is selected the Ruleset Conversion Editor will be displayed.  This is a diff like editor that will allow the user to review and apply their desired changes as dictated by the Schema Association Map.  To initiate the batch conversion mode the user can right click on the desired Schema Association Map and select Convert Rulesets.  A dialog will be displayed listing all of the Rulesets that the selected Schema Association Map can be applied to.  For each desired conversion the user must select whether they want to review their changes with the Ruleset Conversion Editor or if they want to automatically apply the changes dictated by the Schema Association Map to the Ruleset.

The hope is that the solution is flexible enough so that for simple cases there is very little for the user to do but is able to handle the really difficult cases as well.  Hopefully this has been a helpful and informative guide to upgrading your schemas in EBI.

Increasing Productivity With a Local Eclipse Mirror

We use Eclipse pretty heavily here at EXTOL. In addition to using it as our development environment, we also use a lot of the Eclipse platform and tools in our own product. It really helps our productivity both by making it easier for us to write our code, and by allowing us to reuse 3rd party code instead of writing it ourselves.

However, when new releases are made available, there is a whole lot of downloading going on. Each of our developers has to download the new Eclipse install, install it, and then download whatever other features they need from the update sites. Our Internet connection is pretty fast, but it still takes a lot of time. Time that would be better spent doing our actual work.

First, we started keeping the installs in a shared folder. That worked for just the installs. One person would download it first, and then the rest of us copy it from the shared folder. But that involves having  someone do the initial download and save first. No specific person is responsible for that, so it isn’t done consistently. Or, sometimes someone does it only to find that someone else already did. It also doesn’t solve the problem of the update sites. After installing, each developer still has to download all their features from them directly.

The next thing we tried was to set up a local copy of the update sites. Eclipse has a mirroring tool that can replicate an update site to another location. That was progressing pretty well, but then we thought about how disjointed it was doing manual downloads for the installs, this mirror for the update sites, and having to manage both of them.

I started looking into becoming an actual Eclipse mirror site. Hard drives are cheap, so storage space wouldn’t be a problem. But, we did not want to actually be a public mirror, since that would defeat the purpose of trying to save bandwidth. Then, after looking around a bit, I found this page. I saw that they accepted requests for internal only mirrors. That was perfect. I signed up, and a short time later we had access.

Then came the setup. They make it easy for you by using rsync to do the task, and they also provide a configuration script to get you started. Since the total size was over 100GB, I had to break it into chunks to download over a few nights. In that script, I would enable a few more pieces each night until it was fully synced. From then on, it would just download the changed content each night.

One problem we ran into, though, was that once the whole site was synced, we noticed it was missing some pieces. After further investigation, and an email to the Eclipse webmaster, we found out that what they were calling the full sync was not actually a true full sync. They cut out some of the subprojects because some of the mirror sites were worried about the space that was needed. The webmaster offered to put us on the true full sync, which was much larger. But, by today’s standards, it is still not all that much data. It is currently about 360GB. My laptop that is a year old has more than enough space to contain that. I don’t know what people were worried about.

In any case, our true full mirror is now up and running, and it saves a huge amount of time for us. Now when a new release is available, the rsync automatically gets it. Our developers then download it from our local mirror in seconds, instead of the minutes it used to take. The update sites are also pointed to our local mirror, so those downloads also complete in a fraction of the time it used to take. Finally, our automated build process can also use the local mirror. That lets us set up those builds to automatically get updates, but still have a fast turnaround time.

Teach Camel to work with your data

Camels can be stubborn and angry animals if you don’t take care of them. Lucky for you the EXTOL development team has figured out how to tame them. And we even taught them how to work with data!

Everywhere we look today we can see patterns. They’re in your shirt or tie. You witness traffic patterns (big or small) on your way to work. There are even patterns of integration – Enterprise Integration Patterns (EIP). These patterns allow you to define standard ways of dealing with messaging systems. Examples of these patterns include content-based routing and wiretapping.  Continue reading

Best Practices for Mapping: Application Files and Fields

Successful EDI implementations must begin with the development and employment of efficient object naming conventions using “best practices”.  This will avoid aggravation and redevelopment at a later time.  “Doing it correctly the first time” is a most-relevant piece of advice.  This is of particular advantage when creating files (tables) to store EDI data (the implementation and deployment of EDI interface / staging files and in support of both inbound and outbound EDI transactions).

Continue reading