Code Coverage Tools

The Ultimate List of Code Coverage Tools: 25 Code Coverage Tools for C, C++, Java, .NET, and More

Stackify Developer Tips, Tricks & Resources, Insights for Dev Managers Leave a Comment

Code Coverage is a measurement of how many lines, statements, or blocks of your code are tested using your suite of automated tests. It’s an essential metric to understand the quality of your QA efforts. Code coverage shows you how much of your application is not covered by automated tests and is therefore vulnerable to defects. Code coverage is typically measured in percentage values – the closer to 100%, the better. And when you’re trying to demonstrate test coverage to your higher-ups, code coverage tools (and other tools of the trade, of course) come in quite useful.

Over the years, many tools, both open source, and commercial, have been created to serve the code coverage needs of any software development project. Whether you’re a single developer working on a side project at home, or an enterprise with a large DevOps team, or working on QA for a start-up, there’s a code coverage tool to suit every need. Code coverage tools are available for many programming languages and as part of many popular QA tools. They are integrated with build tools like Ant, Maven, and Gradle, with CI tools like Jenkins, project management tools like Jira, and a host of other tools that make up the software development toolset.

Below, we’ve listed 25 code coverage tools, including options for:

The tools in each section are listed in alphabetical order for easy reference.

Open Source Code Coverage Tools

There are a number of open source code coverage tools, but they’re not all the same. Some are deprecated, some actively developed, and each takes a different approach to code coverage.

1. Cobertura

@schristou88

Based on the now-retired jCoverage, Cobertura is another popular open source code coverage tool. It lets you execute tasks via Ant, and Maven, or via the Cobertura CLI. Just like EMMA, Cobertura is also not actively maintained and doesn’t support the current Java versions. It also is embedded into multiple QA tools, and still lives on in other forms.

Programming languages: Java

Developer: Steven Christou

Last release: 2015, version 2.1.1

Cost: Free

2. CodeCover

CodeCover is an open source glass-box testing tool for Java and COBOL. Glass box testing is a “testing technique that examines the program structure and derives test data from the program logic/code.” CodeCover measures statement, branch, loop, and strict condition coverage.

CodeCover is well integrated with a host of development and testing tools including Ant, Jenkins, JUnit, Eclipse, and more. It is licensed under the Eclipse Public Licence (EPL).

Programming languages: Java & COBOL

Developers: Team at University of Stuttgart

Last release: 2011, version 1.0.1.2

Cost: Free

3. Coverage.py

@nedbat

Coverage.py

Coverage.py is a code coverage tool for Python. It monitors your Python programs, notes which parts of the code have been executed, and analyzes the source to identify code that could have been executed but was not.

Programming Languages: Python

Developer: Originally developed by Gareth Rees, maintained by Ned Batchelder (and others) since 2004

Last release: May 2017, Version 4.4.1

Documentation: https://coverage.readthedocs.io/en/coverage-4.4.1/index.html

Cost: Free

4. EMMA

Emma is one of the oldest and most popular of the code coverage tools. Do a Google search for code coverage tools, and EMMA is the first to show up. EMMA tests coverage of Java applications and supports many types of coverage like class, method, line, and block. EMMA can run on-the-fly or in offline mode. Despite its popularity and mature feature set, EMMA is not actively maintained and doesn’t support the current version of Java. However, it is still widely used, and its legacy still lives on in the many tools and products that it has inspired.

Programming languages: Java

Developer: Vlad Roubtsov

Last release: 2005

Cost: Free

5. Gretel

@uoregon

Gretel

Gretel is a residual test coverage monitoring tool developed by the University of Oregon. The most recent version (which was back in 2002) provides statement coverage monitoring, identifying which lines of Java have been executed and which code has not been touched by testing.

Programming languages: Java

Developer: University of Oregon

Last release: June 2002, Version 1.0rc2

Cost: Free

6. Hansel

Hansel

Hansel is a JUnit extension that adds code coverage to the testing framework. Much of Hansel’s source code comes from Gretel, discussed above, although Hansel is compatible with JUnit, which Gretel is not. Going beyond the typical code coverage tool, Hansel lets developers know how much of the code that a test was supposed to test is covered. This tutorial offers tips on using Hansel with JUnit4.

Programming languages: Java

Developer: Niklas Mehner

Last release: October 2006, Version 2.0a3

Cost: Free

7. JaCoCo

With the depreciation of EMMA and Cobertura, there was a need for an actively maintained code coverage tool, and JaCoCo was created just for this reason. It can integrate with Ant, Maven, Gradle, Jenkins, Visual Studio, and many others. Part of the Eclipse Foundation, it replaced EMMA as the code coverage tool that powers the EclEmma plugin, though the name of the plugin stays the same. It’s one of the few open source projects that’s still being developed.

Programming languages: Java

Developers: EclEmma team

Last release: 2017, version 0.7.10

Cost: Free

8. JCov

JCov is another open source code coverage tool. It is well integrated with Oracle’s test infrastructure such as JavaTest and JTReg test harness. It supports offline and on-the-fly instrumentation, and is test framework agnostic. It’s not actively developed today.

Programming languages: Java

Developers: YP engineering team

Last release: 2013, version 1.1.1

Cost: Free

9. NoUnit

NoUnit

NoUnit is developed by FirstPartners.net, and it measures your JUnit tests in your projects using Java, XML, and XSLT, giving you a picture of your code to reveal how good and thorough your JUnit tests really are.

Programming languages: Java


Free Download

Organization: FirstPartners.net

Last release: 2013

Documentation: https://firstpartners.net/category/nounit/

Cost: Free

10. PITest

@0hjc

PITest does mutation testing for Java and the JVM which it proclaims as the “gold standard test coverage.” Most code coverage tools listed here check for test coverage for each line, statement, or branch of code. This means they can only point out which lines of code aren’t tested. However, mutation testing can tell you not just how many lines of code are tested, but how well your automated tests are at discovering bugs.

PITest does this by modifying your code, and running unit tests on the modified code. If the unit test fails after a code change, the test is useful. However, if the unit test doesn’t fail, it hasn’t been able to detect the change in code and, therefore, needs to be changed.

PITest is open source and integrates with And, Maven, Gradle, Eclipse, and InteliJ.

Programming languages: Java

Developers: Henry Coles

Last release: 2016, version 1.1.11

Cost: Free

11. Quilt

Quilt

Quilt is a Java software development tool for measuring code coverage, optimized for use with the JUnit testing package. In addition to JUnit, it can be used with Ant, Maven, and more. The original Quilt branched off into three distinct versions. The most recent version, Quilt 0.6, contains mostly new code but is inspired by the original version.

Programming languages: Java

Developers: David Dixon-Peugh

Last release: 2003, Version 0.6a5

Documentation: https://sourceforge.net/projects/quilt/

Cost: Free

12. Serenity BDD

@SerenityBDD

Serenity BDD is an automated acceptance testing tool that also includes code coverage as one of its features. It lets you write stories and epics for each behavior path, and monitors testing coverage for each of these stories and epics. Because code coverage is mapped to user behavior, the results are a lot more useful than seeing how many lines of code are covered.

It integrates with the usual build tools, and even other QA tools like Sauce Labs, BrowserStack, Appium, and even more tools like Jenkins, and Jira. Serenity BDD provides an integrated testing suite based on Selenium, and if you plan on using the entire suite, code coverage is a bonus you get along with it.

Programming languages: Java, Groovy

Organization: Wakaleo Consulting

Last release: 2016, version 1.2.3-rc.5

Documentation: https://github.com/serenity-bdd/serenity-documentation

Cost: Free

Commercial Code Coverage Tools

While the open source tools are very capable, sometimes you need the assurance of a paid tool for your important large-scale projects. There are a few good options here as well.

13. Atlassian Clover

@Atlassian

Clover is from Atlassian, maker of the popular project tracking tool, Jira. It tests code coverage for Java- and Groovy-based applications. It has numerous integrations with tools CI tools like Bamboo, Hudson, Jenkins, and build tools like Ant, and Maven. You can run Clover in any Eclipse-based IDE or IntelliJ IDEA.

Programming languages: Java, Groovy

Organization: Atlassian

Last release: 2015, version 4.1.x

Documentation: https://confluence.atlassian.com/display/CLOVER

Cost: $300 – $8,000+

14. Bullseye Coverage

Bullseye Testing

BullseyeCoverage is a code analyzer for C and C++ that evaluates how much of your source code was tested. It’s used to improve the quality of software in vital systems, such as those in enterprise applications, medical, industrial control, communications, aerospace, and defense. It works with anything you can write in C and C++, including system-level and kernel mode.

Programming languages: C, C++

Organization: Bullseye Testing Technology

Last release: November 2016, Version 8.13

Documentation: http://www.bullseye.com/whatsNew.html#8_13

Cost: $800 for 1-year license and up

15. FrogLogic Coco

@froglogic

Coco tests code coverage for C, C++, C# and Tcl code. It complements FrogLogic’s UI testing tool – Squish. It integrates with all major build, CI, and test tools, and even has a Visual Studio add-in.

Programming languages: C, C++, C# and Tcl code

Organization: FrogLogic

Last release: 2017, version 4.0

Documentation: https://doc.froglogic.com/squish-coco/latest/

Cost: Not disclosed

16. Testwell CTC++

CTC++, as its name suggests, is a code coverage tool for C, C+, Java, and C#. It provides coverage for line, statement, function, decision, multi condition, modified condition/decision coverage (MC/DC), and condition coverage. It is a mature product, in use for over 25 years now, and is still being actively developed.

Because of its early start, and enterprise-friendly features, it is used across industries like aerospace, healthcare, and transportation. It is easy to customize for a variety of needs and can be used in embedded applications using the host-target add-on.

Programming languages: C, C+, Java, and C#

Organization: Testwell, acquired by Verifysoft Technology in 2013

Last release: 2017, version 8.2

Documentation: http://www.testwell.fi/ctcdesc.html

Cost: Not disclosed

17. NCover

@ncover

nCover

NCover offers three code coverage solutions: a Desktop version offering a powerful .NET code coverage solution, Code Central, NCode’s fully integrated, enterprise-class code coverage solution for the entire team, and Collector, which extends Code Central to QA and enables you to build servers with automated collection.

Programming languages: .NET

Organization: gnoso

Last release: NCover V5.5.3672.681, March 2017

Documentation: http://www.ncover.com/support/docs/index

Cost: Not disclosed (free trials available)

QA and Other Tools with Code Coverage as a Feature

Code coverage is part of the QA process, and it makes sense that some of the popular QA tools provide code coverage as part of their feature sets. Here’s a look at a few such tools that include code coverage as part of their offering.

18. Intel C++ Compiler 15.0

@IntelSoftware

Intel Software

Intel offers a code coverage tool as part of its Intel C++ Compiler 15.0, providing developers with information on how much of an application’s source code is exercised under a specific workload applied to the application. This tool displays dynamic execution counts for each basic block of the application, provides differential coverage data, and presents code coverage information visually with a customizable coloring scheme.

Programming languages: C++

Organization: Intel

Last release: Not disclosed

Documentation: https://software.intel.com/en-us/node/522743

Cost: Not disclosed

19. Parasoft Jtest

@Parasoft

 

Parasoft Logo

Parasoft provides a suite of testing tools for a variety of purposes like load testing, security testing, and more. It supports C, C++, Java, and .NET. One of its products Jtest provides code coverage. Jtest does code coverage by line, statement, block, decision/branch, path, and MC/DC, and condition. This is a great option if you’re concerned about compliance and security for your application.

Jtest can breakdown code coverage by file, class, function/method. It doesn’t just report on unit tests, but also other kinds of tests like integration testing, plugin testing, server-side testing, automated end-to-end functional testing, scripted testing, and user acceptance testing.

As one of the older tools, it is mature and integrates with all key tools like build tools, CI tools, and IDEs. It reports on a wide range of metrics like errors, inefficiencies, and policy violations. However, it is rather closed in the way it is presented, requiring you to sign up to read the documentation or understand the product better. This is not developer-friendly; it’s better-suited for traditional corporations.

Programming languages: C, C++, Java, and .NET

Organization: Parasoft

Last release: 2017, Version 10.3.2

Documentation: https://parasoft.force.com/customerportal/CommunitiesDocuments

Cost: Not disclosed

20. SpiraTeam

@inflectra

SpiraTeam from Inflectra is an Application Lifecycle Management (ALM) tool that focuses on improving the quality of testing, and consequently, code. One of its key features is that it can track code coverage by requirement. Taking a project management approach to code coverage, it can track how well your code is tested against the requirements you start out with, and you can view code coverage metrics by requirement. This is a great option if you spend a lot of time planning your software projects (as you should). SpiraTeam also has a variety of pricing options and lets you manage your own instance on-premise or host it with SpiraTeam.

Programming languages: Any programming language

Organization: Inflectra

Last release: 2016, version 5.1

Documentation: https://www.inflectra.com/SpiraTeam/Documentation.aspx

Cost: $9.99/mo – $3,999.99 for the hosted option, and $99.99 – $29,999.99 for the on-premise option

21. Vector Software

@VectorSoftware

Vector Software

Vector Software offers code coverage as part of its VectorCAST embedded software testing platform, a family of products that automates testing across the SDLC.

Programming languages: C, C++

Organization: Vector Software

Last release: Not disclosed

Cost: Not disclosed (free trial available)

Code Coverage Tools for Other Programming Languages

Most of the tools discussed so far cater to Java, C, and C++ applications. However, there are a few other code coverage tools that are built for lesser used tools.

22. Devel::Cover

Devel::Cover

Devel::Cover is a code coverage tool for Perl applications. It tests code coverage by statement, branch, condition, subroutine, and pod. It is open source and is the only code coverage tool for Perl. It’s not as mature as the other tools listed here and doesn’t include integrations with other platforms. It is still being developed, but not very actively.

Programming languages: Perl

Developer: Paul Johnson

Last release: 2016, version 1.23

Documentation: https://github.com/pjcj/Devel–Cover

Cost: Free

23. dotCover

@jetbrains

dotCover also focuses exclusively on .NET. It is integrated into Visual Studio and analyzes unit test coverage. It can highlight code coverage right inside the code editor, which is very convenient. dotCover is not sold separately but as part of the larger ReSharper Ultimate product from JetBrains.

Programming languages: .NET

Organization: JetBrains

Last release: 2016, version 2016.3

Documentation: https://www.jetbrains.com/dotcover/documentation/

Cost: $11.90/mo – $39.90/mo for ReSharper Ultimate product.

24. OpenCover

OpenCover

OpenCover is an open source code coverage tool for .NET 2 and above. It works on Windows only. However, for Visual Studio, you have a couple of options for code coverage, including one that’s built-in.

Programming languages: .NET2 and above

Developer: Steven Christou

Last release: 2016, version 4.6.519

Cost: Free

25. Visual Studio

@VisualStudio

Visual Studio is the popular IDE from Microsoft. It has integrated tools to collect code coverage metrics; however, this feature is only part of the Enterprise edition of Visual Studio.

Programming languages: C, C++

Organization: Microsoft

Last release: 2017

Documentation: https://docs.microsoft.com/en-us/visualstudio/

Cost: $45/mo for the hosted option, $499 – $11,99 for licensing

How to Choose a Code Coverage Tool

When choosing a code coverage tool, there are many things to consider. First, you should consider your programming language. Most code coverage tools cater to Java, followed by C, and C++, and some .NET.

You should also consider whether an open source tool or a commercial tool is right for you. If you’re working on a small application, and code coverage is not a critical need, you could make do easily with an open source tool. However, make sure the tool you select is still being actively developed and has a healthy community of developers. If you’re using the tool for your organization’s mission-critical apps, you likely need a commercial tool that’s under active development for responsive customer support.

A code coverage tool should be well-integrated with a broad range of development and QA tools that you already use so that your team is likely to adopt it readily and the code coverage metrics it provides are useful.

When using a code coverage tool, you need easy access to reports and metrics. There should be a way to export metrics, integrate with other tools, share it across teams, and analyze results within the tool. Code coverage metrics are useful to the extent that you can share and analyze them for actionable decision-making.

Each tool takes a different approach to code coverage. Some take the traditional route of monitoring lines, statements, branches, and blocks of code, then reporting the percentage that’s covered by automated unit tests. However, many tools on this list take a unique approach to code coverage by focusing on requirements, mutation testing, and glass box testing. You need to consider which of these approaches is in line with your development philosophy.

Finally, look at code coverage as one aspect of your overall QA strategy. Code coverage alone won’t give you all the answers on how to improve your tests and reduce bugs, but it can point you in the right direction. If you see your code coverage hovering around the 50% mark, it’s time to sit up and take notice that your test suite needs to be expanded. This is vital for products that are in a nascent stage. However, even if you expand your test suite to cover 90% of your code base, you could still have inefficient tests that don’t surface bugs. This is when you need to look beyond code coverage to other tools to gauge the quality of your automated tests.

Further Reading on Code Coverage Tools

For more information on choosing the best code coverage tools for your needs and using code coverage to your advantage for better-quality software, visit the following resources:

In conclusion, we hope this list helps you quickly locate the right code coverage tool for the project at hand. What are your go-to code coverage tools? Share your favorites with us in the comments below.