The case for using open source tools for test automation

One of the strong signals coming from the test automation community lately is the increasing popularity and adoption of open source tools and libraries. And you’d be wrong to think it’s only because they come for free. Read on, to find out about some of the less obvious benefits of open source.

Some years ago I stepped into the intriguing world of test automation as the technical lead in a project that involved end-to-end functional testing of a mobile application and a proprietary point of sale system. Although I had 15 years of software engineering practice, my experience with test automation was very limited. There was also an additional complication we had to deal with: one of the applications we were supposed to test was a Windows native application using custom drawn controls, so we could not rely on object selectors for automation. We realized that whatever tool we were to settle on, there was going to be some heavy custom logic involved. There wasn’t too much room for error either - changing our mind one year down the line, would mean throwing away all the investment we made in time, effort and money, damaging the team’s morale, our credibility and impacting the timelines of the projects involved.

This is when we first started to throw around the idea of building our own automation tool and, after going back and forth a couple of times, we decided this was indeed our best shot at success. We were very much aware of the amount of hard work involved, but we were optimistic, because we had a few good things going for us: we had good software engineering and test automation expertise in our team and the leadership was very supportive, which is a crucial factor in large companies. The outcome of this effort was the OpenTest functional test automation framework. Luckily, the product proved to be a success and, after the first year, it started to see adoption from numerous teams across four continents. I know this wouldn’t have been possible had we not made — at the very start of the project — the strategic choice to only use dependencies released under a liberal open source license. The core projects we incorporated into the product were: Selenium for web testing, Appium for mobile testing, Apache HttpClient for API testing, OpenCV for visual validation and Tesseract for OCR capabilities. It’s amazing to think about all the work that went into these projects, that we can just download and use, no strings attached. We later decided to pay it forward and release the OpenTest framework itself as open source.

If you’ve been spending time around a test automation team for a while, I should have no problem convincing you that automation is hard work. But doing test automation while dealing with license-related issues and the artificial limitations imposed by vendors is like trying to do your manicure while being chased by wolves. Please don’t do put your testers through that. Wouldn’t it be nice if they could install the automation tool in about 15 minutes, on however many physical or virtual machines they need to, without having to ask anyone for permission and without having to get anyone to agree to a $3000+ expense? That would free up time and mental resources that can be redirected towards dealing with the real difficulties of test automation.

One other thing I witnessed first-hand is that teams using commercial tools tend to hold very strong opinions about the tool of their choice. And it makes perfect sense: when you spend thousands of dollars on a test automation tool, you’re going to do everything in your power to defend the soundness of that decision long after you’ve admitted to yourself the tool is not all that great. The net effect is that different teams end up using different tools which stifles collaboration and encourages the formation of silos.

To be sure, using open source software is not without its risks and perils, so you need to do the due diligence to vet a project before adopting it. I will not elaborate on this, as there are many good resources out there that you can reference.

Here’s a non-comprehensive list with the benefits of open source that mostly focuses on the aspects relevant to test automation:

• Zero cost. Everybody likes free stuff, but the zero cost comes with even more benefits: when your teams can install the automation tools for free, they will be able to do their jobs better and easier (e.g. run more parallel jobs to minimize execution time, install all components of the test tool on both their Mac and their Windows machine, etc.)
• Reduced friction. The fact that anybody can get started with the tool right away eliminates useless red tape and allows teams to move faster. With commercial tools, even after the expense is approved and you have the installation kit in your hands, you still have to worry about obtaining and setting up the license (and do it again every time it expires), or put in a request with your networking people to open up some obscure port to connect to the license server. This doesn’t sound like much, but it’s enough to discourage and demoralize your team before the real work even starts.
• Promotes collaboration between teams. When you make a behavior less expensive in terms of money, time and effort, you are motivating people to adopt that behavior. When using a free and open source tool that takes 15 minutes to install, it’s much easier to standardize on that tool across an organization. Once your teams speak the same “language”, they can share test assets, test infrastructure, best practices, etc.
• Harnesses the power of the community. It’s hard to overestimate the value of having a thriving community around a software product. Large and diverse communities are far more effective at innovating and uncovering bugs and best practices than teams working on closed, proprietary solutions. There’s something magical about posting a question on Stack Overflow and getting a complete, high-quality answer that solves your problem in the time it takes for you to top off your coffee mug. Not to mention that most responses are way better that some of the commercial support I’ve seen.
• Prevents vendor lock-in. With proprietary tools, you are at the mercy of vendors, whose primary objective is to maximize revenue. What happens when your vendor decides to raise the price significantly after you’ve spent the last five years polishing your test suite written on top of their tool? How about if you need a new feature or integration? How much time will that take to add and how much will it cost you, if they even agree to do it? With open source tools, you at least have some options: you can write that feature yourself or pay a qualified third party. If you feel generous, you can even make a pull request to the original project and allow everyone else to make use of that new shiny feature.

I hope I convinced you to give some serious thought on using open source for your next test automation project. Whatever the case, one thing is sure: open source is no longer the orphan kid it used to be until not so long ago, and that just fills my heart with joy!

It would be great if you can share your own experience with open source software. What projects and libraries have you found the most useful and what mistakes did you make that the rest of us can avoid?