Version Compatibility Test

How many times have you run into version compatibility issues that takes up a majority of your troubleshooting time? Wouldn’t it be nice if we could come up with a specification? Who reads the details buried in the readme files? Even if you diligently did read it, is it possible to juggle all those compatible version numbers for different version of the library/framework in your head while troubleshooting? Wouldn’t it be nice to make it an executable script? Read this article to find out how you can cut down time spent in troubleshooting.

We spend 90% of our troubleshooting time in finding the cause of the problem.

Software Compatibility

Software compatibility is the ability of software components to work together. Write tests for forward compatibility and backward compatibility. A library or framework must create and maintain a list of:

1. Common Compatibility Issues
2. Resolving Application Incompatibilities

This will save lot of time troubleshooting broken systems due to version incompatibility issues. It is a good practice for frameworks and libraries to ship a version compatibility script that developers can run to find the cause of the problem quickly.

Forward Compatibility

Forward incompatibility due to minor release is caused by regression bugs. In a real world project example, I found it very difficult to fix a bug due to RSpec regression bug. I had to downgrade RSpec to fix this difficult problem. Forward compatibility due to major release cannot be taken for granted. Example, I had to work with a custom test automation framework that broke when the Chrome browser was upgraded. The Chrome driver was not compatible with the version of the Chrome browser that was released after the Chrome driver release date.

Forward and Backward Compatible Assumptions

We all know that assumptions is the mother of all fuckups. Assuming a certain version is forward or backward compatible is one of the biggest mistake and will cost you time when your application breaks. It is just a time bomb waiting to blow up when you accidentally upgrade either due to bundler or OS. That's why it is a good idea to lock the gem version in the bundler. Instead of mentioning the minimum version of dependent software required for the library or framework to function in the readme file that gets ignored, wouldn't it be nice if it shipped with a version compatibility test script that documents the version compatibility? This also prevents the developer from wasting their time hunting and reading the release notes for all the dependent software to narrow down the culprit.

For instance, Rails can ship with a script that checks for the compatible MRI Ruby version, OS version, RVM, testing framework version etc. It is not enough to run automated tests on a continuous integration server. We need to write a version compatibility test that must also be run to quickly find any backward compatible and forward compatible issues. This will prevent shipping software that causes headache for early adopters. The script should ideally take a software compatibility matrix as the input and output the success/failure message with detailed resolution.

Version Compatibility Gem Specification

If we take this one step further, it should become part of the gem specification. It should be run before the gem is installed on the system. Does the gem require native libraries? Does it compile only on certain version of the compilers? By running the version compatibility test, we can catch the errors even before it begins to create issues. This version compatibility test can be exposed as a gem command to be run when developers need to troubleshoot and write scripts for different environments.