Many IT professionals believe that data transformation and mapping are “solved problems”. After all, mapping tools have been around for over 20 years, and thousands of IT organizations use them in integration projects every day. “If it ain’t broke, don’t fix it”, right?
What belies that attitude are the missed integration project deadlines, runtime exceptions, customer chargebacks, vendor scorecard deductions and other business problems that can be traced to data transformation mapping practices. Mapping is also the single most costly integration activity, accounting for up to 75% of some integration project costs. Yet few project teams focus attention on ways to improve mapping efficiency and accuracy. Read more…
When talking about integration, whether it is application to application (A2A), business to business (B2B) or data and cloud integration, the goal is to exchange data, documents, information, and/or messages between a source and a target. To accomplish this goal, we identify business patterns related to the sources and targets of these “integrations” and then apply the identified patterns to similar scenarios that meet specified criteria. Most, if not all of these patterns will require data mappings, also known as “maps” or “rulesets”, at some point during the integration process. These maps are ultimately used to perform one of two operations: to transform or translate data. I often use the terms, transform and translate, interchangeably. But, is there more to them? Are these terms synonymous, or do their meanings differ?
Let’s look at this a little closer to see if there is a difference. Data translation can be defined as the process of converting data from one syntax to another and performing value lookups or substitutions from the data during the process. Translation can include data validation as well. One example of data translation is to convert EDI purchase order document data into purchase order database files or even flat files while performing data validation on the source data. Read more…
Some of the most difficult challenges that integration projects face are what I like to refer to as “artifacts of black-box implementations.” These artifacts include incomplete specifications (both from external sources such as vendor specification or internal sources due to a lack of understanding of the interrelationships between systems), opaque processes from lack of visibility and undocumented integrations. Opaque processes and undocumented integrations present the greatest challenge because necessary information is not available due to ineffective communication, absence of the original source (often a single person) or integrations with legacy systems where there is no more vendor support. Most processes enabled by commercial applications are opaque. If you can’t determine (because it’s undocumented) (a) the identities and locations of all data that is added or updated by the process; (b) the rules under which data changes and additions occur; and (c) the identities and locations of all downstream applications, modules, and services that are invoked, then the process can be considered, to some degree, opaque. Read more…
In my last blog entry here, I started a discussion of the technical challenges of using Spreadsheets as electronic business documents. I want to continue that discussion, but instead of focusing on how data is located predictably in the spreadsheet, I want to talk about separating the formatting from the business data.
As you might imagine, consuming Spreadsheets and producing them presents two different sets of challenges. When consuming them, the problem of locating the data in the spreadsheet predictably is the most significant challenge, next to data content concerns. The formatting (fonts, colors, shading, graphics, etc.) generally does not matter when consuming the Spreadsheets, whereas when producing them, it can matter almost as much as the data content. Read more…
Spreadsheets are a little bit like hammers: you can use them for almost anything. And, they’re a relatively simple mechanism so what could be hard about them…right? Humans seem to have a propensity to find new ways of complicating technology, and I think most of you have probably guessed that that applies to spreadsheets, too. I don’t expect that will ever change.
Spreadsheets are a natural choice for people who don’t want (or don’t have the time) to invest in more formalized options like EDI or XML. They’ve gotten a bad rap, perhaps because they’re arguably less sophisticated than the alternatives, but also because of the complication of dealing with them in an automated fashion.
Before we decided to add support for spreadsheets to our integration platform, we were asked by several prospects and even given some examples of what they needed to handle. These cases were, of course, quite varied and we thought about it for a while, wondering if we could come up with something that would be useful. Read more…
