Code Coverage is a measurement of how many lines, statements, or blocks of your code are tested using your suite of automated tests. It’s an essential metric to understand the quality of your QA efforts. Code coverage shows you how much of your application is not covered by automated tests and is therefore vulnerable to defects. Code coverage is typically measured in percentage values – the closer to 100%, the better. And when you’re trying to demonstrate test coverage to your higher-ups, code coverage tools (and other tools of the trade, of course) come in quite useful.
Over the years, many tools, both open source, and commercial, have been created to serve the code coverage needs of any software development project. Whether you’re a single developer working on a side project at home, or an enterprise with a large DevOps team, or working on QA for a start-up, there’s a code coverage tool to suit every need. Code coverage tools are available for many programming languages and as part of many popular QA tools. They are integrated with build tools like Ant, Maven, and Gradle, with CI tools like Jenkins, project management tools like Jira, and a host of other tools that make up the software development toolset.
Below, we’ve listed 25 code coverage tools, including options for:
The tools in each section are listed in alphabetical order for easy reference.
There are a number of open source code coverage tools, but they’re not all the same. Some are deprecated, some actively developed, and each takes a different approach to code coverage.
Based on the now-retired jCoverage, Cobertura is another popular open source code coverage tool. It lets you execute tasks via Ant, and Maven, or via the Cobertura CLI. Just like EMMA, Cobertura is also not actively maintained and doesn’t support the current Java versions. It also is embedded into multiple QA tools, and still lives on in other forms.
Programming languages: Java
Developer: Steven Christou
Last release: 2015, version 2.1.1
Cost: Free
CodeCover is an open source glass-box testing tool for Java and COBOL. Glass box testing is a “testing technique that examines the program structure and derives test data from the program logic/code.” CodeCover measures statement, branch, loop, and strict condition coverage.
CodeCover is well integrated with a host of development and testing tools including Ant, Jenkins, JUnit, Eclipse, and more. It is licensed under the Eclipse Public Licence (EPL).
Programming languages: Java & COBOL
Developers: Team at University of Stuttgart
Last release: 2011, version 1.0.1.2
Cost: Free
Coverage.py is a code coverage tool for Python. It monitors your Python programs, notes which parts of the code have been executed, and analyzes the source to identify code that could have been executed but was not.
Programming Languages: Python
Developer: Originally developed by Gareth Rees, maintained by Ned Batchelder (and others) since 2004
Last release: May 2017, Version 4.4.1
Cost: Free
Emma is one of the oldest and most popular of the code coverage tools. Do a Google search for code coverage tools, and EMMA is the first to show up. EMMA tests coverage of Java applications and supports many types of coverage like class, method, line, and block. EMMA can run on-the-fly or in offline mode. Despite its popularity and mature feature set, EMMA is not actively maintained and doesn’t support the current version of Java. However, it is still widely used, and its legacy still lives on in the many tools and products that it has inspired.
Programming languages: Java
Developer: Vlad Roubtsov
Last release: 2005
Cost: Free
Gretel is a residual test coverage monitoring tool developed by the University of Oregon. The most recent version (which was back in 2002) provides statement coverage monitoring, identifying which lines of Java have been executed and which code has not been touched by testing.
Programming languages: Java
Developer: University of Oregon
Last release: June 2002, Version 1.0rc2
Cost: Free
Hansel is a JUnit extension that adds code coverage to the testing framework. Much of Hansel’s source code comes from Gretel, discussed above, although Hansel is compatible with JUnit, which Gretel is not. Going beyond the typical code coverage tool, Hansel lets developers know how much of the code that a test was supposed to test is covered. This tutorial offers tips on using Hansel with JUnit4.
Programming languages: Java
Developer: Niklas Mehner
Last release: October 2006, Version 2.0a3
Cost: Free
With the depreciation of EMMA and Cobertura, there was a need for an actively maintained code coverage tool, and JaCoCo was created just for this reason. It can integrate with Ant, Maven, Gradle, Jenkins, Visual Studio, and many others. Part of the Eclipse Foundation, it replaced EMMA as the code coverage tool that powers the EclEmma plugin, though the name of the plugin stays the same. It’s one of the few open source projects that’s still being developed.
Programming languages: Java
Developers: EclEmma team
Last release: 2017, version 0.7.10
Cost: Free
JCov is another open source code coverage tool. It is well integrated with Oracle’s test infrastructure such as JavaTest and JTReg test harness. It supports offline and on-the-fly instrumentation, and is test framework agnostic. It’s not actively developed today.
Programming languages: Java
Developers: YP engineering team
Last release: 2013, version 1.1.1
Cost: Free
NoUnit is developed by FirstPartners.net, and it measures your JUnit tests in your projects using Java, XML, and XSLT, giving you a picture of your code to reveal how good and thorough your JUnit tests really are.
Programming languages: Java
Organization: FirstPartners.net
Last release: 2013
Documentation: https://firstpartners.net/category/nounit/
Cost: Free
PITest does mutation testing for Java and the JVM which it proclaims as the “gold standard test coverage.” Most code coverage tools listed here check for test coverage for each line, statement, or branch of code. This means they can only point out which lines of code aren’t tested. However, mutation testing can tell you not just how many lines of code are tested, but how well your automated tests are at discovering bugs.
PITest does this by modifying your code, and running unit tests on the modified code. If the unit test fails after a code change, the test is useful. However, if the unit test doesn’t fail, it hasn’t been able to detect the change in code and, therefore, needs to be changed.
PITest is open source and integrates with And, Maven, Gradle, Eclipse, and InteliJ.
Programming languages: Java
Developers: Henry Coles
Last release: 2016, version 1.1.11
Cost: Free
Quilt is a Java software development tool for measuring code coverage, optimized for use with the JUnit testing package. In addition to JUnit, it can be used with Ant, Maven, and more. The original Quilt branched off into three distinct versions. The most recent version, Quilt 0.6, contains mostly new code but is inspired by the original version.
Programming languages: Java
Developers: David Dixon-Peugh
Last release: 2003, Version 0.6a5
Documentation: https://sourceforge.net/projects/quilt/
Cost: Free
Serenity BDD is an automated acceptance testing tool that also includes code coverage as one of its features. It lets you write stories and epics for each behavior path, and monitors testing coverage for each of these stories and epics. Because code coverage is mapped to user behavior, the results are a lot more useful than seeing how many lines of code are covered.
It integrates with the usual build tools, and even other QA tools like Sauce Labs, BrowserStack, Appium, and even more tools like Jenkins, and Jira. Serenity BDD provides an integrated testing suite based on Selenium, and if you plan on using the entire suite, code coverage is a bonus you get along with it.
Programming languages: Java, Groovy
Organization: Wakaleo Consulting
Last release: 2016, version 1.2.3-rc.5
Documentation: https://github.com/serenity-bdd/serenity-documentation
Cost: Free
While the open source tools are very capable, sometimes you need the assurance of a paid tool for your important large-scale projects. There are a few good options here as well.
Clover is from Atlassian, maker of the popular project tracking tool, Jira. It tests code coverage for Java- and Groovy-based applications. It has numerous integrations with tools CI tools like Bamboo, Hudson, Jenkins, and build tools like Ant, and Maven. You can run Clover in any Eclipse-based IDE or IntelliJ IDEA.
Programming languages: Java, Groovy
Organization: Atlassian
Last release: 2015, version 4.1.x
Documentation: https://confluence.atlassian.com/display/CLOVER
Cost: $300 – $8,000+
BullseyeCoverage is a code analyzer for C and C++ that evaluates how much of your source code was tested. It’s used to improve the quality of software in vital systems, such as those in enterprise applications, medical, industrial control, communications, aerospace, and defense. It works with anything you can write in C and C++, including system-level and kernel mode.
Programming languages: C, C++
Organization: Bullseye Testing Technology
Last release: November 2016, Version 8.13
Documentation: http://www.bullseye.com/whatsNew.html#8_13
Cost: $800 for 1-year license and up
Coco tests code coverage for C, C++, C# and Tcl code. It complements FrogLogic’s UI testing tool – Squish. It integrates with all major build, CI, and test tools, and even has a Visual Studio add-in.
Programming languages: C, C++, C# and Tcl code
Organization: FrogLogic
Last release: 2017, version 4.0
Documentation: https://doc.froglogic.com/squish-coco/latest/
Cost: Not disclosed
CTC++, as its name suggests, is a code coverage tool for C, C+, Java, and C#. It provides coverage for line, statement, function, decision, multi condition, modified condition/decision coverage (MC/DC), and condition coverage. It is a mature product, in use for over 25 years now, and is still being actively developed.
Because of its early start, and enterprise-friendly features, it is used across industries like aerospace, healthcare, and transportation. It is easy to customize for a variety of needs and can be used in embedded applications using the host-target add-on.
Programming languages: C, C+, Java, and C#
Organization: Testwell, acquired by Verifysoft Technology in 2013
Last release: 2017, version 8.2
Documentation: http://www.testwell.fi/ctcdesc.html
Cost: Not disclosed
NCover offers three code coverage solutions: a Desktop version offering a powerful .NET code coverage solution, Code Central, NCode’s fully integrated, enterprise-class code coverage solution for the entire team, and Collector, which extends Code Central to QA and enables you to build servers with automated collection.
Programming languages: .NET
Organization: gnoso
Last release: NCover V5.5.3672.681, March 2017
Documentation: http://www.ncover.com/support/docs/index
Cost: Not disclosed (free trials available)
Code coverage is part of the QA process, and it makes sense that some of the popular QA tools provide code coverage as part of their feature sets. Here’s a look at a few such tools that include code coverage as part of their offering.
Intel offers a code coverage tool as part of its Intel C++ Compiler 15.0, providing developers with information on how much of an application’s source code is exercised under a specific workload applied to the application. This tool displays dynamic execution counts for each basic block of the application, provides differential coverage data, and presents code coverage information visually with a customizable coloring scheme.
Programming languages: C++
Organization: Intel
Last release: Not disclosed
Documentation: https://software.intel.com/en-us/node/522743
Cost: Not disclosed
Parasoft provides a suite of testing tools for a variety of purposes like load testing, security testing, and more. It supports C, C++, Java, and .NET. One of its products Jtest provides code coverage. Jtest does code coverage by line, statement, block, decision/branch, path, and MC/DC, and condition. This is a great option if you’re concerned about compliance and security for your application.
Jtest can breakdown code coverage by file, class, function/method. It doesn’t just report on unit tests, but also other kinds of tests like integration testing, plugin testing, server-side testing, automated end-to-end functional testing, scripted testing, and user acceptance testing.
As one of the older tools, it is mature and integrates with all key tools like build tools, CI tools, and IDEs. It reports on a wide range of metrics like errors, inefficiencies, and policy violations. However, it is rather closed in the way it is presented, requiring you to sign up to read the documentation or understand the product better. This is not developer-friendly; it’s better-suited for traditional corporations.
Programming languages: C, C++, Java, and .NET
Organization: Parasoft
Last release: 2017, Version 10.3.2
Cost: Not disclosed
SpiraTeam from Inflectra is an Application Lifecycle Management (ALM) tool that focuses on improving the quality of testing, and consequently, code. One of its key features is that it can track code coverage by requirement. Taking a project management approach to code coverage, it can track how well your code is tested against the requirements you start out with, and you can view code coverage metrics by requirement. This is a great option if you spend a lot of time planning your software projects (as you should). SpiraTeam also has a variety of pricing options and lets you manage your own instance on-premise or host it with SpiraTeam.
Programming languages: Any programming language
Organization: Inflectra
Last release: 2016, version 5.1
Documentation: https://www.inflectra.com/SpiraTeam/Documentation.aspx
Cost: $9.99/mo – $3,999.99 for the hosted option, and $99.99 – $29,999.99 for the on-premise option
Vector Software offers code coverage as part of its VectorCAST embedded software testing platform, a family of products that automates testing across the SDLC.
Programming languages: C, C++
Organization: Vector Software
Last release: Not disclosed
Cost: Not disclosed (free trial available)
Most of the tools discussed so far cater to Java, C, and C++ applications. However, there are a few other code coverage tools that are built for lesser used tools.
Devel::Cover is a code coverage tool for Perl applications. It tests code coverage by statement, branch, condition, subroutine, and pod. It is open source and is the only code coverage tool for Perl. It’s not as mature as the other tools listed here and doesn’t include integrations with other platforms. It is still being developed, but not very actively.
Programming languages: Perl
Developer: Paul Johnson
Last release: 2016, version 1.23
Documentation: https://github.com/pjcj/Devel–Cover
Cost: Free
dotCover also focuses exclusively on .NET. It is integrated into Visual Studio and analyzes unit test coverage. It can highlight code coverage right inside the code editor, which is very convenient. dotCover is not sold separately but as part of the larger ReSharper Ultimate product from JetBrains.
Programming languages: .NET
Organization: JetBrains
Last release: 2016, version 2016.3
Documentation: https://www.jetbrains.com/dotcover/documentation/
Cost: $11.90/mo – $39.90/mo for ReSharper Ultimate product.
OpenCover is an open source code coverage tool for .NET 2 and above. It works on Windows only. However, for Visual Studio, you have a couple of options for code coverage, including one that’s built-in.
Programming languages: .NET2 and above
Developer: Community/Shaun Wilde
Last release: 2016, version 4.6.519
Cost: Free
Visual Studio is the popular IDE from Microsoft. It has integrated tools to collect code coverage metrics; however, this feature is only part of the Enterprise edition of Visual Studio.
Programming languages: C, C++
Organization: Microsoft
Last release: 2017
Documentation: https://docs.microsoft.com/en-us/visualstudio/
Cost: $45/mo for the hosted option, $499 – $11,99 for licensing
When choosing a code coverage tool, there are many things to consider. First, you should consider your programming language. Most code coverage tools cater to Java, followed by C, and C++, and some .NET.
You should also consider whether an open source tool or a commercial tool is right for you. If you’re working on a small application, and code coverage is not a critical need, you could make do easily with an open source tool. However, make sure the tool you select is still being actively developed and has a healthy community of developers. If you’re using the tool for your organization’s mission-critical apps, you likely need a commercial tool that’s under active development for responsive customer support.
A code coverage tool should be well-integrated with a broad range of development and QA tools that you already use so that your team is likely to adopt it readily and the code coverage metrics it provides are useful.
When using a code coverage tool, you need easy access to reports and metrics. There should be a way to export metrics, integrate with other tools, share it across teams, and analyze results within the tool. Code coverage metrics are useful to the extent that you can share and analyze them for actionable decision-making.
Each tool takes a different approach to code coverage. Some take the traditional route of monitoring lines, statements, branches, and blocks of code, then reporting the percentage that’s covered by automated unit tests. However, many tools on this list take a unique approach to code coverage by focusing on requirements, mutation testing, and glass box testing. You need to consider which of these approaches is in line with your development philosophy.
Finally, look at code coverage as one aspect of your overall QA strategy. Code coverage alone won’t give you all the answers on how to improve your tests and reduce bugs, but it can point you in the right direction. If you see your code coverage hovering around the 50% mark, it’s time to sit up and take notice that your test suite needs to be expanded. This is vital for products that are in a nascent stage. However, even if you expand your test suite to cover 90% of your code base, you could still have inefficient tests that don’t surface bugs. This is when you need to look beyond code coverage to other tools to gauge the quality of your automated tests.
For more information on choosing the best code coverage tools for your needs and using code coverage to your advantage for better-quality software, visit the following resources:
In conclusion, we hope this list helps you quickly locate the right code coverage tool for the project at hand. What are your go-to code coverage tools? Share your favorites with us in the comments below.
Try Stackify’s free code profiler, Prefix, to write better code on your workstation. Prefix works with .NET, Java, PHP, Node.js, Ruby, and Python.
Stackify's APM tools are used by thousands of .NET, Java, PHP, Node.js, Python, & Ruby developers all over the world.
Explore Retrace's product features to learn more.
If you would like to be a guest contributor to the Stackify blog please reach out to [email protected]
