EXTOL Technology Blog

We use Eclipse pretty heavily here at EXTOL. In addition to using it as our development environment, we also use a lot of the Eclipse platform and tools in our own product. It really helps our productivity both by making it easier for us to write our code, and by allowing us to reuse 3^rd party code instead of writing it ourselves.

However, when new releases are made available, there is a whole lot of downloading going on. Each of our developers has to download the new Eclipse install, install it, and then download whatever other features they need from the update sites. Our Internet connection is pretty fast, but it still takes a lot of time. Time that would be better spent doing our actual work.

First, we started keeping the installs in a shared folder. That worked for just the installs. One person would download it first, and then the rest of us copy it from the shared folder. But that involves having  someone do the initial download and save first. No specific person is responsible for that, so it isn’t done consistently. Or, sometimes someone does it only to find that someone else already did. It also doesn’t solve the problem of the update sites. After installing, each developer still has to download all their features from them directly.

The next thing we tried was to set up a local copy of the update sites. Eclipse has a mirroring tool that can replicate an update site to another location. That was progressing pretty well, but then we thought about how disjointed it was doing manual downloads for the installs, this mirror for the update sites, and having to manage both of them.

I started looking into becoming an actual Eclipse mirror site. Hard drives are cheap, so storage space wouldn’t be a problem. But, we did not want to actually be a public mirror, since that would defeat the purpose of trying to save bandwidth. Then, after looking around a bit, I found this page. I saw that they accepted requests for internal only mirrors. That was perfect. I signed up, and a short time later we had access.

Then came the setup. They make it easy for you by using rsync to do the task, and they also provide a configuration script to get you started. Since the total size was over 100GB, I had to break it into chunks to download over a few nights. In that script, I would enable a few more pieces each night until it was fully synced. From then on, it would just download the changed content each night.

One problem we ran into, though, was that once the whole site was synced, we noticed it was missing some pieces. After further investigation, and an email to the Eclipse webmaster, we found out that what they were calling the full sync was not actually a true full sync. They cut out some of the subprojects because some of the mirror sites were worried about the space that was needed. The webmaster offered to put us on the true full sync, which was much larger. But, by today’s standards, it is still not all that much data. It is currently about 360GB. My laptop that is a year old has more than enough space to contain that. I don’t know what people were worried about.

In any case, our true full mirror is now up and running, and it saves a huge amount of time for us. Now when a new release is available, the rsync automatically gets it. Our developers then download it from our local mirror in seconds, instead of the minutes it used to take. The update sites are also pointed to our local mirror, so those downloads also complete in a fraction of the time it used to take. Finally, our automated build process can also use the local mirror. That lets us set up those builds to automatically get updates, but still have a fast turnaround time.

Class and object design can be a tricky thing.  It’s not really something that can be taught or quantified.  This skill is developed through years of experience.  This may be more of an art than it is a science.  No expert can come up with an exact formula on how this should be done but all experts can pretty much agree on what is a bad design.

A possible starting point of a process could be to develop a written problem statement outlining the issue at hand.  This can be done by interviewing a customer or whoever the requirements are coming from.  Once the problem statement is developed you will find that the nouns in the statement will turn into classes and the verbs in the statement will turn into operations.  From there relationships between the classes can be derived.  Statements like “is a” can denote inheritance, “has a” can denote composition, “uses” can denote delegation, etc.

A UML diagram should be drawn modeling what was discovered by the problem statement.  It is also good to keep a catalogue of design patterns at hand so that a given pattern may be applied to a module of the software system.  Remember that Object Oriented Software is about loose coupling, encapsulation, and reuse, which design patterns help enforce.  Once the initial diagram is developed it is really more of a guideline than concrete fact at this point.  When the implementation process begins new facts will come to light that haven’t been considered before which will ultimately force the redesign some components of the software.  Once this happens the diagram should be updated to reflect the new functionality.  This will create a software cycle that will bounce back and forth from implementation to redesign approaching a final version of the software.

This is really only a generalized overview of an absolutely monstrous topic in which there are many different paradigms and many different ways to go about tackling the problem.  As a closing thought, keep in mind that a design is never really complete, that it can always be improved upon and made more robust, clearer, and more efficient.

After having supported many customers for the past several years, I look back at some of the more critical issues and quickly recognize that most of these could have been resolved (or certainly avoided downtime) had the customer employed reasonable and simple testing.

The one recurring theme is the amount of implementation work that is performed without equal efforts applied to testing that work.  One would expect that any configuration with impact on company reputation and revenue would be tested to its’ fullest potential before it is ever promoted to a production status.  Schedules and deadlines do not always allow for intense sessions of testing and debugging, however, there are many important yet unrecognized advantages to the testing process. Read more…

There’s a great deal of data floating around and it has to be put somewhere. In 1970, the relational database (RDBMS) was invented to help store it. Data that was *related* could be grouped together in a table. A schema was used to define the structure of the data within the database and the relationships among it. SQL was used to manipulate the database and the data it contained. There are many databases that use this model, such as MySQL, PostgresSQL, and Oracle.

For many years, relational databases were the cornerstones of applications. Organizations used them as the backend store for their thick-clients as well as being an integral part of the LAMP stack used in early web applications. Only recently has our software needed a little extra oomph.

The NoSQL movement promises to fulfill requirements of high availability, horizontal scaling, replication, schemaless design, and complex computational capabilities. It contests the notion that RDBMS are always the best place to store your data and opens the doors to greater freedom when choosing your storage mechanism.

The framework used to evaluate these systems is based on consistency, availability, and partition tolerance (CAP). The CAP theorem was developed by Eric Brewer to formally talk about tradeoffs in highly scalable systems¹. Like other decisions made in the software world, you can only pick two out of the three criteria.

The NoSQL movement doesn’t subscribe to a particular data model like RDBMS do. There are three other models that are part of the crowd:

key-value: much like a map that supports put, get, and remove (Redis, Dynamo)

column-oriented: still uses tables like the relational model, but without joins (BigTable)

document-oriented: stores structured documents like JSON or XML (CouchDB)

You may be thinking, “Ok, so what is the best one?” I only wish the answer was that simple. Many different factors go into choosing and you are not limited to one mechanism per application. You can choose different stores for different types of data and functionality.^{2, 3}

Structuring your application to take advantage of these data store capabilities requires analysis of your data requirements. You may need fast access or maybe your data is written more than it’s read. Perhaps you need to perform calculations such as map/reduce or graph manipulations. Maybe your data is of the binary variety. And of course, the availability rabbit hole – do you trust your server not to fail when you’ve just been featured on the 6 o’clock news (or Digg)?

While this is a lot to think about, the benefits of charting your way through the NoSQL forest are worth the effort in the long run. Your application will be better suited to expandability and your maintenance efforts may be decreased. However, there’s no cause to throw out your SQL books…just yet.

More info & references:

1. http://www.julianbrowne.com/article/viewer/brewers-cap-theorem

2. http://blog.nahurst.com/visual-guide-to-nosql-systems

3. http://blog.heroku.com/archives/2010/7/20/nosql/

4. http://architects.dzone.com/news/nosql-old-wine-new-bottle

Being somewhat of a concerned environmentalist, I think that helping save the planet for future generations should be the priority for everyone. One small step would be to switch your company to the ecologically friendly technology called “EDI” (Electronic Data Interchange).  Workers around the world discard millions of tons of office paper every year.  Producing this paper kills an innumerable amount of trees each year.  Converting this huge quantity of paper to an electronic form would help save paper and energy.  So think green: Reduce your carbon footprint by using less paper while saving money.

EDI would save your company money by providing an alternative to data streams that require a great deal of human interaction and costly materials.  Overhead costs will be reduced by eliminating such tasks as mailroom sorting, circulation, data entry, the manual reconciliation of different documents, document mailing, faxing of information, distributing, filing and storage of documents and transportation costs.  You will save money on paper, envelopes, business forms, stamps, printing and processing fees.   Excess inventory costs would be lowered by reducing inventory levels achieved by shortening the order processing cycle time.  Companies could save money by taking advantage of early payment discounts offered by vendors for quicker supply chain turnaround.  Other cost reductions would include the handling of errors and exceptions – this could be many times more expensive than correctly processing the document in the first place.

EDI improves customer satisfaction and strengthens supplier relations.  EDI improves customer satisfaction because information is available “real time” and is more accurate. Concerns and questions of customers can be addressed in a more timely fashion.  EDI will help strengthen relations with suppliers. The mutual work required to implement EDI tends to build a good working relationship between trading partners.  The sharing of EDI information strengthens the ties between partners and encourages stronger levels of commitment.

When information is copied from one location to another there is an opportunity for error.  Electronic data transfer eliminates the need for copying information from a paper document to another format, or re-keying it into your system.  Once the information is entered correctly, there is no additional opportunity for error since it is electronically entered into all other applications without further human intervention.

One very important advantage of EDI over paper documents is the speed in which the trading partner receives and enters the information into their system.  This greatly reduces cycle and turnaround times.  For this reason, EDI can be an important component of “Just-in-Time” production systems.  Better responses to customer concerns may lead to increased sales.  Transmitting data via EDI across the country or around the world will only take a few seconds or minutes as opposed to days, when sent through manual delivery service.  Once the data is received, the information is available immediately without any time-consuming human intervention.

EDI gives companies the ability to exchange business documents electronically via a wide array of communication protocols.  This form of e-commerce is widely used throughout the world.  EDI carries millions of dollars of transactions every day for all types of businesses and government agencies.  EDI plays a very important role in the world’s economic make-up.  Many companies will not do business with vendors that cannot exchange business documents through EDI.  (Good luck trying to do business with the big retailers without EDI.)

EDI has been available for decades.  It is reliable, proven, stable, and supported by long-established standards developed and maintained by respected governing bodies (such as ANSI, EDIFACT, and ISO…to name a few).  It is highly unlikely that this will change in this lifetime.  Being EDI-capable is no longer an option; it is a business necessity!

Integrating with customers, suppliers, service providers, and applications is important to EXTOL’s daily internal operations, just as it is with your company. And probably just like your company, to manage those operations we initially relied on various disparate manual and/or home-grown semi-automated yet labor-intensive processes that inevitably involved no-longer-necessary and overlapping (if not outright redundant) activities. It wasn’t planned like that − it just sort of happened.

We’ve learned many lessons while “dogfooding” our internal operations. Before using our own product − EXTOL Business Integrator (EBI) − to streamline those operations, the first (and most daunting) task was to untangle the jungle of existing processes in an attempt to understand their ultimate goal. Read more…

With the ever-changing economy, especially now, businesses are looking for ways to reduce costs and become more efficient.  The new technologies (as previously discussed in Part 1 of this blog), coupled with the immediate need for cost cutting, have created the perfect environment to fuel another major shift in business communication.

One factor helping this shift is that technology has been developed and priced so that even small “Mom and Pop” shops can obtain affordable AS2 and/or FTP solutions.  The shift in cost of direct connections will fuel the move away from traditional Value Added Network (VAN) trading partners to having more AS2 and/or FTP partners (connection types).  This shift will increase more rapidly as a result of the constant need for companies to become more efficient to remain competitive.

Large retailers, such as Walmart, require a “direct” AS2 connection to do EDI business. This represents a key indicator that this shift to AS2 (and/or FTP) is becoming more prominent and recognized.

It is interesting to see this relationship go though another major change.  What we’re actually seeing is communications between Trading Partners coming full circle.  Initially there were leased lines and individual modems for connecting to trading partners.  Over time this became too expensive and, out of necessity to reduce communication costs, the VAN was born.  More recently the Internet was introduced, which provided a new and inexpensive means to communicate.  With expanding Internet capabilities, VANs became too expensive and too time consuming to manage.  Better, cheaper software was designed to utilize these new communication methods.  As a result, we now see more direct-connections being established in place of moving data through the traditional VANs.

This shift will require more time…it will not happen overnight.  Instead, it will be a slow migration that will occur over the next decade (possibly longer).  As new business relationships are formed, they will take advantage of these newer technologies; older methods are likely to remain with the VAN service (although the VANs often do provide other solutions and services besides merely the moving of data).  Going forward, implementations will generally see a mix of these connection options until costs and efficiency eventually eliminate those methods that restrict growth.

For those old enough to remember this reference (and for those young enough to know how to Google it), sometimes I feel like the Lorne Greene of EXTOL. Not in his role as the patriarch of Bonanza’s Cartwright clan, but rather as the dog food pitchman.

The series of 1980s TV commercials for Alpo^® dog food featuring Greene closed with the veteran actor claiming that Alpo is so good he feeds it to his own dogs. Hence, the notion of using one’s own product became known as “eating your own dog food”. Or so the legend goes. [A less-tasteful alternative account of the phrase’s origin has the president of a rival pet food corporation eating a can of his company's dog food at every shareholders meeting.]

During my first eight years with EXTOL, I was blissfully unaware of this term. But when I assumed the newly-created role of Special Projects Coordinator four years ago, Tony Baran (our co-founder, President, and CEO) caught me lingering at the coffeemaker and directed me to lead the effort to “eat our own dog food”. I mustered my best blank stare, nodded assuredly, scurried to my cubicle, and Googled that phrase (actually, I had one of my younger officemates show me how to Google it). Only then did Tony’s mandate to “streamline our processes by eliminating error-prone re-keying of transactional and/or persisted data by automating the exchange of information between applications, databases, spreadsheets, and other sources” − I’m paraphrasing here − make sense. And oh yeah, “use EXTOL Business Integrator to do it”. Aha! − we would “eat our own dog food”.

While “eating your own dog food” now has been elevated to acronym status, EYODF is more than a mouthful − I prefer the much more palatable and trendy “dogfooding”. While we don’t make (nor eat, for that matter) dog food at EXTOL, we do produce some seriously efficient business-to-business (B2B) and application-to-application (A2A) integration software that coordinates external partner interactions and internal business activities. And now we’re dogfooding.

In subsequent posts, I’ll share our experiences in using EXTOL Business Integrator (EBI) to solve our company’s data integration, manipulation, and migration needs. By telling our story and noting the lessons learned (mostly the hard way − those are the best-remembered lessons), we hope you can use EBI to your best advantage to tackle similar projects in your organization. You have them − just look around.

And please, if you care to share your stories with us, do. Who knows: maybe we’ll send you a can of Alpo.

In addition to the EXTOL Blog and EXTOL Forum destinations, EXTOL also provides another valuable destination from the www.extol.com Web page– EXTOL Customer Zone.

EXTOL Customer Zone can be accessed via link in the top right corner of the EXTOL home page.

Customer Zone is a registered-user destination with information on many customer-related topics. From the Customer Zone page, you can link to information on topics such as:

• EXTOL University Presentations
• Customer Education Webinars
• HIPAA Migration Information
• Portal Migration
• Technical Documentation
• Webcasts
• Whitepapers
• Knowledge Base/Incident Views

Many of the above topics provide access to historical product information as well as currently active topics such as HIPAA Migration Information for EXTOL’s support of the X12 5010 standard.

In addition to product information, you can access information on incidents you have submitted. This information, available via the Knowledge Base/Incident Views link provides a current status of your submitted incidents.

I could further detail all of the content available on Customer Zone, however my goal is not to detail each piece of information available, but rather make you aware of its existence.

New to Customer Zone? No problem. Simply click the Register link on the Customer Zone login page. On the subsequent Webpage, provide all required information and click Register. Within 24 hours, and often much quicker, an account will be created for you and an account notification e-mail sent to the e-mail address provided during registration.

When setting up a new trading partner or business integration process, thinking through the process beforehand can greatly decrease the implementation time and help minimize problems. Many integration projects can come to a grinding halt in the middle of implementation if it’s only later realized that required data is not configured in your backend application, or if your system doesn’t support the particular communication protocol your new trading partner intends to use.  “Looking before leaping” will help alleviate potential problems later.

Read more…