In most cases, however, business integration provisioning involves more steps, because the business problem to be solved requires tailored integration between some set of sources and targets, e.g., integration of an XML transaction set with a Warehouse Management System.  Those steps might include definition or specification of endpoints, communication and interface connections, documents / messages and envelopes, data routing, business processes, and data transformations.  By combining building blocks that implement such object types, you can solve most kinds of business-to-business, application, and data integration problems.

The scope, complexity, and duration of the provisioning process are very much dependent on your choice of implementation tools.  While integration provisioning can be implemented using conventional programming languages and tools, model-driven integration tools offer considerably more power and productivity, because they enable specification of integration applications to occur at a higher level, using predefined object types like business processes, events, transformations, and adapters.

Even so, there are limits to the productivity one can achieve with such object-level modeling environments.  Data transformations, for example, are commonly produced using visual mapping tools that support drag-and-drop associations between source and target elements and enable specification of functions (arithmetic, string manipulation, database lookup, external program or service call, etc.) that produce the desired output result in each target element.  A similar level of detailed effort may be needed when defining source and target documents, endpoints, and other integration object deliverables.

Scaling up this kind of object-at-a-time provisioning approach to very large projects, or even worse to large scale conversions (e.g., upgrading or replacing an ERP system for which a great deal of integration time and effort have been invested), while less arduous than conventional programming, can still consume very large amounts of IT staff time, and can introduce significant deadline risk.

That’s where “large-scale provisioning” comes into play.  The idea behind large-scale provisioning is to apply automation to parts of the provisioning process, operating on collections of integration objects at one time, rather than operating on individual objects, for example:

• Endpoint provisioning tools that can generate many endpoint definitions at once, by analyzing EDI instance data (sample interchanges from partners)
• Automated metadata importers and generators that produce document definitions containing hundreds or thousands of typed and structured elements
• Intelligent mapping utilities that can generate many transformation rules at a time, based on a combination of mapping history, property matching heuristics, and predefined source-target element associations
• Migration utilities that can automate modification of multiple transformation rules in multiple maps at a time, when versions, structures, or syntax changes occur in source or target documents

In a way, large-scale provisioning does for integration design-time processes in IT what business process automation does for operational processes in the business – by applying automation to tedious, low-level tasks, it increases organizational productivity and efficiency, reduces errors, and speeds the delivery of integration results.  Depending on the integration project at hand, large-scale provisioning can save up to 70% of your project time (even more, in some cases).  That translates to the ability to deliver more projects with less resource and time, reduced project risk (less chance of delivering projects late and less chance that projects, when delivered, no longer meet current business requirements), and earlier onset of business benefits (e.g., faster onboarding of customer transactions tied to new revenue).

In a few weeks, you’ll hear more about large-scale provisioning from EXTOL, as we launch EXTOL Business Integrator v2.5.  Stay tuned for more!

However, when new releases are made available, there is a whole lot of downloading going on. Each of our developers has to download the new Eclipse install, install it, and then download whatever other features they need from the update sites. Our Internet connection is pretty fast, but it still takes a lot of time. Time that would be better spent doing our actual work.

First, we started keeping the installs in a shared folder. That worked for just the installs. One person would download it first, and then the rest of us copy it from the shared folder. But that involves having  someone do the initial download and save first. No specific person is responsible for that, so it isn’t done consistently. Or, sometimes someone does it only to find that someone else already did. It also doesn’t solve the problem of the update sites. After installing, each developer still has to download all their features from them directly.

The next thing we tried was to set up a local copy of the update sites. Eclipse has a mirroring tool that can replicate an update site to another location. That was progressing pretty well, but then we thought about how disjointed it was doing manual downloads for the installs, this mirror for the update sites, and having to manage both of them.

I started looking into becoming an actual Eclipse mirror site. Hard drives are cheap, so storage space wouldn’t be a problem. But, we did not want to actually be a public mirror, since that would defeat the purpose of trying to save bandwidth. Then, after looking around a bit, I found this page. I saw that they accepted requests for internal only mirrors. That was perfect. I signed up, and a short time later we had access.

Then came the setup. They make it easy for you by using rsync to do the task, and they also provide a configuration script to get you started. Since the total size was over 100GB, I had to break it into chunks to download over a few nights. In that script, I would enable a few more pieces each night until it was fully synced. From then on, it would just download the changed content each night.

One problem we ran into, though, was that once the whole site was synced, we noticed it was missing some pieces. After further investigation, and an email to the Eclipse webmaster, we found out that what they were calling the full sync was not actually a true full sync. They cut out some of the subprojects because some of the mirror sites were worried about the space that was needed. The webmaster offered to put us on the true full sync, which was much larger. But, by today’s standards, it is still not all that much data. It is currently about 360GB. My laptop that is a year old has more than enough space to contain that. I don’t know what people were worried about.

In any case, our true full mirror is now up and running, and it saves a huge amount of time for us. Now when a new release is available, the rsync automatically gets it. Our developers then download it from our local mirror in seconds, instead of the minutes it used to take. The update sites are also pointed to our local mirror, so those downloads also complete in a fraction of the time it used to take. Finally, our automated build process can also use the local mirror. That lets us set up those builds to automatically get updates, but still have a fast turnaround time.

A possible starting point of a process could be to develop a written problem statement outlining the issue at hand.  This can be done by interviewing a customer or whoever the requirements are coming from.  Once the problem statement is developed you will find that the nouns in the statement will turn into classes and the verbs in the statement will turn into operations.  From there relationships between the classes can be derived.  Statements like “is a” can denote inheritance, “has a” can denote composition, “uses” can denote delegation, etc.

A UML diagram should be drawn modeling what was discovered by the problem statement.  It is also good to keep a catalogue of design patterns at hand so that a given pattern may be applied to a module of the software system.  Remember that Object Oriented Software is about loose coupling, encapsulation, and reuse, which design patterns help enforce.  Once the initial diagram is developed it is really more of a guideline than concrete fact at this point.  When the implementation process begins new facts will come to light that haven’t been considered before which will ultimately force the redesign some components of the software.  Once this happens the diagram should be updated to reflect the new functionality.  This will create a software cycle that will bounce back and forth from implementation to redesign approaching a final version of the software.

This is really only a generalized overview of an absolutely monstrous topic in which there are many different paradigms and many different ways to go about tackling the problem.  As a closing thought, keep in mind that a design is never really complete, that it can always be improved upon and made more robust, clearer, and more efficient.

The one recurring theme is the amount of implementation work that is performed without equal efforts applied to testing that work.  One would expect that any configuration with impact on company reputation and revenue would be tested to its’ fullest potential before it is ever promoted to a production status.  Schedules and deadlines do not always allow for intense sessions of testing and debugging, however, there are many important yet unrecognized advantages to the testing process.

Let’s first look at some of the obvious reasons why a company might opt NOT to implement a testing plan as part of their implementation:

1. Maintaining a separate test system is expensive.
2. Migrating or “promoting” work from the test to production system is time consuming and could be a bit overwhelming, particularly if the configuration is simple and on a tight schedule.
3. Keeping the test and production systems “mirrored” could be difficult to maintain, especially if many resources are tied to the same project.

Perhaps these statements are good reasons to bypass an adequate testing phase but let’s consider the repercussions of such actions.

Scenario

An EDI Analyst has been given the task of implementing a new Trading Partner and Transaction into their application system.  The addition of this new partner will yield high profits for the company (and maybe positive internal exposure for the analyst).

The company’s IT management informs the analyst that a test system has been made available, where the analyst could build and test the processes necessary for this implementation without concerns for interfering with day-to-day operations.  Instead, the analyst exudes his/her confidence in this seemingly repetitive task and decides that testing would delay completion by forcing him/her to coordinate with other production operations and personnel.  The analyst proceeds by building the work directly into the production system.

Fast-forward a few weeks.

The analyst is now one day from the deadline and the work is about to go live!  He/she places the finishing touches on the implementation and, the very next day, they “flip the switch” to start receiving the EDI information from the Trading Partner.  Aghast, the process does not work.  Reacting quickly, the analyst begins to search for the problem.  After a few intense moments, he/she discovers that the problem is occurring in a part of the process deemed the “heart” of the application.  Making any changes at this point would require that all other portions of the implementation would either need to be redone or, at minimum, re-evaluated and retested.  Meanwhile, the partner relationship has moved forward; data is flowing in but can’t be processed.  The partner activation is placed on hold, revenue is lost, and the partner relationship has been strained.

This describes a worst-case scenario but is not uncommon, particularly when little emphasis is placed on the testing process.  Perhaps this situation could have been easily avoided?

Let’s look at some of the advantages of having a test system:

1. Building/testing a process on a test system will identify immediate areas for concern and expose any mistakes made during the implementation.
2. Having a test system can also serve as backup in the event configurations would need to be restored to the production system.
3. Building a test system opens the opportunity for creative thinking as different methods for implementation can be tested without affecting production.

In our scenario, had the EDI Analyst decided to use a test system and prepared testing procedures, maybe he/she would not have found themselves in this awful predicament.  Unfortunately all potential benefits (both company and personal) were lost; a lack of confidence sets in and unneeded stress ensues.

Often, we do not realize the importance of the testing phase of an implementation.  From personal experience I have seen this make or break projects.  The next project you are assigned to, do not underestimate the importance of developing and including a thorough testing process.

For many years, relational databases were the cornerstones of applications. Organizations used them as the backend store for their thick-clients as well as being an integral part of the LAMP stack used in early web applications. Only recently has our software needed a little extra oomph.

The NoSQL movement promises to fulfill requirements of high availability, horizontal scaling, replication, schemaless design, and complex computational capabilities. It contests the notion that RDBMS are always the best place to store your data and opens the doors to greater freedom when choosing your storage mechanism.

The framework used to evaluate these systems is based on consistency, availability, and partition tolerance (CAP). The CAP theorem was developed by Eric Brewer to formally talk about tradeoffs in highly scalable systems¹. Like other decisions made in the software world, you can only pick two out of the three criteria.

The NoSQL movement doesn’t subscribe to a particular data model like RDBMS do. There are three other models that are part of the crowd:

key-value: much like a map that supports put, get, and remove (Redis, Dynamo)

column-oriented: still uses tables like the relational model, but without joins (BigTable)

document-oriented: stores structured documents like JSON or XML (CouchDB)

You may be thinking, “Ok, so what is the best one?” I only wish the answer was that simple. Many different factors go into choosing and you are not limited to one mechanism per application. You can choose different stores for different types of data and functionality.^{2, 3}

Structuring your application to take advantage of these data store capabilities requires analysis of your data requirements. You may need fast access or maybe your data is written more than it’s read. Perhaps you need to perform calculations such as map/reduce or graph manipulations. Maybe your data is of the binary variety. And of course, the availability rabbit hole – do you trust your server not to fail when you’ve just been featured on the 6 o’clock news (or Digg)?

While this is a lot to think about, the benefits of charting your way through the NoSQL forest are worth the effort in the long run. Your application will be better suited to expandability and your maintenance efforts may be decreased. However, there’s no cause to throw out your SQL books…just yet.

More info & references:

1. http://www.julianbrowne.com/article/viewer/brewers-cap-theorem

2. http://blog.nahurst.com/visual-guide-to-nosql-systems

3. http://blog.heroku.com/archives/2010/7/20/nosql/

4. http://architects.dzone.com/news/nosql-old-wine-new-bottle

EDI would save your company money by providing an alternative to data streams that require a great deal of human interaction and costly materials.  Overhead costs will be reduced by eliminating such tasks as mailroom sorting, circulation, data entry, the manual reconciliation of different documents, document mailing, faxing of information, distributing, filing and storage of documents and transportation costs.  You will save money on paper, envelopes, business forms, stamps, printing and processing fees.   Excess inventory costs would be lowered by reducing inventory levels achieved by shortening the order processing cycle time.  Companies could save money by taking advantage of early payment discounts offered by vendors for quicker supply chain turnaround.  Other cost reductions would include the handling of errors and exceptions – this could be many times more expensive than correctly processing the document in the first place.

EDI improves customer satisfaction and strengthens supplier relations.  EDI improves customer satisfaction because information is available “real time” and is more accurate. Concerns and questions of customers can be addressed in a more timely fashion.  EDI will help strengthen relations with suppliers. The mutual work required to implement EDI tends to build a good working relationship between trading partners.  The sharing of EDI information strengthens the ties between partners and encourages stronger levels of commitment.

When information is copied from one location to another there is an opportunity for error.  Electronic data transfer eliminates the need for copying information from a paper document to another format, or re-keying it into your system.  Once the information is entered correctly, there is no additional opportunity for error since it is electronically entered into all other applications without further human intervention.

One very important advantage of EDI over paper documents is the speed in which the trading partner receives and enters the information into their system.  This greatly reduces cycle and turnaround times.  For this reason, EDI can be an important component of “Just-in-Time” production systems.  Better responses to customer concerns may lead to increased sales.  Transmitting data via EDI across the country or around the world will only take a few seconds or minutes as opposed to days, when sent through manual delivery service.  Once the data is received, the information is available immediately without any time-consuming human intervention.

EDI gives companies the ability to exchange business documents electronically via a wide array of communication protocols.  This form of e-commerce is widely used throughout the world.  EDI carries millions of dollars of transactions every day for all types of businesses and government agencies.  EDI plays a very important role in the world’s economic make-up.  Many companies will not do business with vendors that cannot exchange business documents through EDI.  (Good luck trying to do business with the big retailers without EDI.)

EDI has been available for decades.  It is reliable, proven, stable, and supported by long-established standards developed and maintained by respected governing bodies (such as ANSI, EDIFACT, and ISO…to name a few).  It is highly unlikely that this will change in this lifetime.  Being EDI-capable is no longer an option; it is a business necessity!

We’ve learned many lessons while “dogfooding” our internal operations. Before using our own product − EXTOL Business Integrator (EBI) − to streamline those operations, the first (and most daunting) task was to untangle the jungle of existing processes in an attempt to understand their ultimate goal.

Lesson #1: Whether you prefer a flowchart, a whiteboard, or a corkboard with colored pushpins and string, you’d better have a solid overview of the forest before you walk amongst the trees.

At EXTOL, three separate shrink-wrapped applications handle our Customer relationship management (CRM), financial (FIN), and customer support (SUP) information; each resides on its own server. Previously, when a “prospect” signed a contract to purchase our software, their CRM status was immediately changed to “customer” and their software shipped that very day. But we waited a day or so until the requisite paperwork arrived from our sales office to manually enter that entity into our FIN app. Some time later (for various operational reasons), that customer would be manually added to our SUP system.

We were traveling the paper trail through that customer management forest, and there were many opportunities to get lost (or run headfirst into a tree or three!) along the way. Here’s how we could easily end up with this tangled mess:

• The CRM system’s customer name/address is “ABC, Inc.” in West Virginia.
• But when the contract arrives at our financial office, it reads “XYZ Industries” (of which ABC, Inc. is a subsidiary) in Ohio. So, that’s how that customer record is keyed into the FIN app.
• The next day, our support team gets a call from an individual with “ABC” in West Virginia, but we have no record of them in our SUP system. So, the support rep calls our financial group inquiring about that customer − a quick search in the FIN app for names staring with “ABC” results in no hits (because they’re in that database as “XYZ Industries”), and a subsequent search of all customers in West Virginia shows none starting with “ABC” (as that app has them located in Ohio). So, to provide our customary timely, award-winning customer support, we hastily enter them into the SUP system erroneously as “BAC” located in Virginia.

We now have one actual customer in three different databases, but they have three distinct names and three different addresses! This contrived scenario demonstrates how a manual customer data integration (CDI) effort can quickly disintegrate!

Lesson #2: When the answer to “Why is it done this way?” is “We always did it this way”, throw that way away!

That manual CDI method − fraught with time-delays, potential (if not probable) data entry errors, and cracks the largest of customers could fall through − had to go. This diagram illustrates how we used EBI to automate the exchange of information between these applications via five relatively small, sequenced, three-stage processes designed to insert/update data in real-time.

Lesson #3: Best-practice CDI projects start with (1) an ambitious vision, (2) a realistic understanding of its business impact, and (3) a small, controlled project that delivers value quickly.

Pre-EBI development work involved:

• disabling the ability to create customer records via the UIs in both the FIN app and SUP system − we only wanted one automated way to add new customers

• creating a “CUSTOMER KEYSTORE” database to house (for cross-reference purposes) the IDs for customer records from the CRM, FIN, and SUP databases

• laboriously synchronizing by hand the existing customer records in the three systems − tedious and time-consuming

Each process features three stages: a DB trigger to detect specific activity, a set of EBI objects (put into play by an EBI Event) designed to affect the desired result, and a resulting DB action (creating/updating customer records with pertinent data, namely consistent name and address information).

Specifically:

Now, mere seconds after a CRM prospect becomes a customer, both downstream systems (FIN and SUP) are populated with fully-synchronized new customer records. This automated method not only saves data entry time, eliminates error-prone re-keying, and emails alerts to key personnel, but most importantly ensures that all systems have identical name/address information.

This solution, which spearheaded our on-going EXTOL Business Process Initiative (EBPI), has been synchronizing our customer records for several years now. It has survived EBI platform/version changes, multiple application upgrades, relocation of databases, virtual machine migrations, and several modifications/additions of EBI sets to synchronize supplementary customer information.

Incidentally, this very project placed the development of the Database Monitors feature (coming in the soon-to-be-released EBI v2.5) on our must-do list. It also impressed upon me EBI’s power and ease-of-use, and emboldened me to tackle the next EBPI project.

One factor helping this shift is that technology has been developed and priced so that even small “Mom and Pop” shops can obtain affordable AS2 and/or FTP solutions.  The shift in cost of direct connections will fuel the move away from traditional Value Added Network (VAN) trading partners to having more AS2 and/or FTP partners (connection types).  This shift will increase more rapidly as a result of the constant need for companies to become more efficient to remain competitive.

Large retailers, such as Walmart, require a “direct” AS2 connection to do EDI business. This represents a key indicator that this shift to AS2 (and/or FTP) is becoming more prominent and recognized.

It is interesting to see this relationship go though another major change.  What we’re actually seeing is communications between Trading Partners coming full circle.  Initially there were leased lines and individual modems for connecting to trading partners.  Over time this became too expensive and, out of necessity to reduce communication costs, the VAN was born.  More recently the Internet was introduced, which provided a new and inexpensive means to communicate.  With expanding Internet capabilities, VANs became too expensive and too time consuming to manage.  Better, cheaper software was designed to utilize these new communication methods.  As a result, we now see more direct-connections being established in place of moving data through the traditional VANs.

This shift will require more time…it will not happen overnight.  Instead, it will be a slow migration that will occur over the next decade (possibly longer).  As new business relationships are formed, they will take advantage of these newer technologies; older methods are likely to remain with the VAN service (although the VANs often do provide other solutions and services besides merely the moving of data).  Going forward, implementations will generally see a mix of these connection options until costs and efficiency eventually eliminate those methods that restrict growth.

The series of 1980s TV commercials for Alpo^® dog food featuring Greene closed with the veteran actor claiming that Alpo is so good he feeds it to his own dogs. Hence, the notion of using one’s own product became known as “eating your own dog food”. Or so the legend goes. [A less-tasteful alternative account of the phrase’s origin has the president of a rival pet food corporation eating a can of his company's dog food at every shareholders meeting.]

During my first eight years with EXTOL, I was blissfully unaware of this term. But when I assumed the newly-created role of Special Projects Coordinator four years ago, Tony Baran (our co-founder, President, and CEO) caught me lingering at the coffeemaker and directed me to lead the effort to “eat our own dog food”. I mustered my best blank stare, nodded assuredly, scurried to my cubicle, and Googled that phrase (actually, I had one of my younger officemates show me how to Google it). Only then did Tony’s mandate to “streamline our processes by eliminating error-prone re-keying of transactional and/or persisted data by automating the exchange of information between applications, databases, spreadsheets, and other sources” − I’m paraphrasing here − make sense. And oh yeah, “use EXTOL Business Integrator to do it”. Aha! − we would “eat our own dog food”.

While “eating your own dog food” now has been elevated to acronym status, EYODF is more than a mouthful − I prefer the much more palatable and trendy “dogfooding”. While we don’t make (nor eat, for that matter) dog food at EXTOL, we do produce some seriously efficient business-to-business (B2B) and application-to-application (A2A) integration software that coordinates external partner interactions and internal business activities. And now we’re dogfooding.

In subsequent posts, I’ll share our experiences in using EXTOL Business Integrator (EBI) to solve our company’s data integration, manipulation, and migration needs. By telling our story and noting the lessons learned (mostly the hard way − those are the best-remembered lessons), we hope you can use EBI to your best advantage to tackle similar projects in your organization. You have them − just look around.

And please, if you care to share your stories with us, do. Who knows: maybe we’ll send you a can of Alpo.

EXTOL Customer Zone can be accessed via link in the top right corner of the EXTOL home page.

Customer Zone is a registered-user destination with information on many customer-related topics. From the Customer Zone page, you can link to information on topics such as:

• EXTOL University Presentations
• Customer Education Webinars
• HIPAA Migration Information
• Portal Migration
• Technical Documentation
• Webcasts
• Whitepapers
• Knowledge Base/Incident Views

Many of the above topics provide access to historical product information as well as currently active topics such as HIPAA Migration Information for EXTOL’s support of the X12 5010 standard.

In addition to product information, you can access information on incidents you have submitted. This information, available via the Knowledge Base/Incident Views link provides a current status of your submitted incidents.

I could further detail all of the content available on Customer Zone, however my goal is not to detail each piece of information available, but rather make you aware of its existence.

New to Customer Zone? No problem. Simply click the Register link on the Customer Zone login page. On the subsequent Webpage, provide all required information and click Register. Within 24 hours, and often much quicker, an account will be created for you and an account notification e-mail sent to the e-mail address provided during registration.