Increasing Productivity With a Local Eclipse Mirror

We use Eclipse pretty heavily here at EXTOL. In addition to using it as our development environment, we also use a lot of the Eclipse platform and tools in our own product. It really helps our productivity both by making it easier for us to write our code, and by allowing us to reuse 3rd party code instead of writing it ourselves.

However, when new releases are made available, there is a whole lot of downloading going on. Each of our developers has to download the new Eclipse install, install it, and then download whatever other features they need from the update sites. Our Internet connection is pretty fast, but it still takes a lot of time. Time that would be better spent doing our actual work.

First, we started keeping the installs in a shared folder. That worked for just the installs. One person would download it first, and then the rest of us copy it from the shared folder. But that involves havingĀ  someone do the initial download and save first. No specific person is responsible for that, so it isn’t done consistently. Or, sometimes someone does it only to find that someone else already did. It also doesn’t solve the problem of the update sites. After installing, each developer still has to download all their features from them directly.

The next thing we tried was to set up a local copy of the update sites. Eclipse has a mirroring tool that can replicate an update site to another location. That was progressing pretty well, but then we thought about how disjointed it was doing manual downloads for the installs, this mirror for the update sites, and having to manage both of them.

I started looking into becoming an actual Eclipse mirror site. Hard drives are cheap, so storage space wouldn’t be a problem. But, we did not want to actually be a public mirror, since that would defeat the purpose of trying to save bandwidth. Then, after looking around a bit, I found this page. I saw that they accepted requests for internal only mirrors. That was perfect. I signed up, and a short time later we had access.

Then came the setup. They make it easy for you by using rsync to do the task, and they also provide a configuration script to get you started. Since the total size was over 100GB, I had to break it into chunks to download over a few nights. In that script, I would enable a few more pieces each night until it was fully synced. From then on, it would just download the changed content each night.

One problem we ran into, though, was that once the whole site was synced, we noticed it was missing some pieces. After further investigation, and an email to the Eclipse webmaster, we found out that what they were calling the full sync was not actually a true full sync. They cut out some of the subprojects because some of the mirror sites were worried about the space that was needed. The webmaster offered to put us on the true full sync, which was much larger. But, by today’s standards, it is still not all that much data. It is currently about 360GB. My laptop that is a year old has more than enough space to contain that. I don’t know what people were worried about.

In any case, our true full mirror is now up and running, and it saves a huge amount of time for us. Now when a new release is available, the rsync automatically gets it. Our developers then download it from our local mirror in seconds, instead of the minutes it used to take. The update sites are also pointed to our local mirror, so those downloads also complete in a fraction of the time it used to take. Finally, our automated build process can also use the local mirror. That lets us set up those builds to automatically get updates, but still have a fast turnaround time.

Leave a Reply

Your email address will not be published. Required fields are marked *


*