Dealing with Black Box Integration

Some of the most difficult challenges that integration projects face are what I like to refer to as “artifacts of black-box implementations.”  These artifacts include incomplete specifications (both from external sources such as vendor specification or internal sources due to a lack of understanding of the interrelationships between systems), opaque processes from lack of visibility and undocumented integrations.  Opaque processes and undocumented integrations present the greatest challenge because necessary information is not available due to ineffective communication, absence of the original source (often a single person) or integrations with legacy systems where there is no more vendor support.  Most processes enabled by commercial applications are opaque. If you can’t determine (because it’s undocumented) (a) the identities and locations of all data that is added or updated by the process; (b) the rules under which data changes and additions occur; and (c) the identities and locations of all downstream applications, modules, and services that are invoked, then the process can be considered, to some degree, opaque.

Incomplete specifications are another challenge and can exist for many reasons but typically occur due to a lack of committed source participation to maintain the current information set or time-pressures to complete the specification. Employee movement (domain expertise leaving as a key employee retires or transfers), outdated software, vendor displacement or early project termination are common contributors to incomplete reference information about existing systems. These are problems that can exist through the entire lifecycle of the integration, not just the initial Build phase.  It becomes apparent at the start of the project, but can hamper efforts during the Maintain phase as the integration changes and evolves.  Communication and documentation is the key to taming the challenge of incomplete specifications.

Integrations should be thought of as complete processes and not just finite tasks that transform data.  Opaque processes and undocumented integrations pose similar challenges and actually contribute to incomplete specifications.  Again, the key here is communication and documentation (or lack thereof).  Opaque processes can exist due to vendor-controlled software assets that do not expose any information about how they operate or from legacy integrations that have been long forgotten.  It is important to dig deeper until you have a clear understanding of any change impacts on existing systems, applications or dependent processes.

Undocumented integrations often indicate a reactive approach to a specific tactical challenge.  They can be the most troublesome because their scope tends to be obscured.  For example, consider an integration that creates a file from a database and then transmits the file via FTP to another internal system and continues to do a little more processing locally (maybe updating a data warehouse with statistics). But, is that truly the entire integration process?  The answer is no.  Let’s pick up the story after the file has been FTP’d to the target system.  Some process (read: opaque process) picks up the file and does “something” else with it.  Without knowledge of the entire process chain, concepts like failure handling become impossible to consider and could easily create data integrity problems.  Consider that after the file is transferred (and presumably picked up by another process) a downstream failure occurs in the main process flow requiring a rollback.  Unfortunately, the file was transferred and has initiated other processes.  A successful rollback cannot possibly happen cleanly.

These types of challenges can wreak havoc on integration projects. So, what are some things that we can do to help eliminate some of these problems?  Tools that allow our integrations to self-document are a good start.  The tools should also handle changes to existing integrations (offering Change Management) and support Business Process Re-Engineering best practices.

Being able to adapt to changing specifications is a key capability in handling Integration project challenges.  Reusability of project assets is another significant way to tame integration complexity and maintain a single version of assets, eliminating integrity issues.

When interfaces changes, as is the case with an ERP upgrade, sometimes the data formats are completely redesigned.  A good example of this is a flat-file interface that is upgraded to XML.  Another integration capability that supports maintainability is the generation of assets from examples, such as interface data files or XML schema definition documents (XSD).  But, what if you have an existing integration and only one “side” changes formats?  It would also be useful to have a tool that can build a relationship between the original schema and the new schema and update any affected maps to incorporate the new schema.  For more interesting reading, check out the blog on Managing the Impact of Unexpected Document Change by Tammy Moyer.

Integration is not easy and requires up-front time really dissecting the stated requirements, applications affected and any existing processes involved.  Look for the areas that are not clearly defined and dig deeper.  And remember; it may be easier to re-design a process rather than to shoehorn new requirements into an existing process flow.


Leave a Reply

Your email address will not be published. Required fields are marked *