How toxic is your code?

21 November 2008

Update: I have reimplemented the charts in HTML5. See Toxicity reloaded.

If you are somebody who writes code you probably know that moment when you look at some code you didn’t write, or some code you wrote a long time ago, and you think “that doesn’t look good.” Ok, more realistically, you probably think “WTF? I wouldn’t want to touch that with a barge-pole!” It is not even so much about whether the code does what it should do—that takes a bit longer to figure out—or whether the code is too slow. Even if it’s perfectly bug free and performs well, there’s something to the way it’s written. This is part of the internal quality of a software system, something that the users and development managers can’t observe directly; yet, it still affects them because code with poor internal quality is hard to maintain and extend.

Now, as a developer, how do you help managers and business people understand the internal quality of code? They generally want a bit more than “it’s horrible” before they prioritise cleaning up the code over implementing new features that directly deliver business value. Or even: how do you figure out for yourself how bad some code actually is in relation to some other code? These were questions that Chris Brown, Darren Hobbs, and myself were asking ourselves a couple of years ago.

The answer came in the form of a simple bar chart, arguably not the most sophisticated visualisation but a very effective one. And our colleague Ross Pettit had the perfect name for it: The Toxicity Chart. Read on to see what it is and how it’s created.

As usual, let’s start with an example. What follows is the toxicity chart for a version of the Hibernate framework.

Toxicity Chart for Hibernate

In a toxicity chart each bar represents a class and the height of the bar shows the toxicity score for that class. The score is based on several metrics (more on that below) and the higher the score the more toxic the class. The individual components of the score are coloured. For example, the contribution of the method length metric to the overall score is shown in orange. This makes it possible to see at a glance not only how toxic a codebase is but also how the problems are distributed. If there is a lot of purple and orange in the chart, this indicates long and complex methods, which means that the code is probably hard to test on a unit level. Lastly, classes that score zero points are left off the chart.

The calculation of the toxicity score is based on metrics and thresholds. For example, the method length metric has a threshold of 30. If a class contains a method that is longer it gets points proportional to the length of the method in relation to the threshold, e.g. a method with 45 lines of code would get 1.5 points because its length is 1.5 times the threshold. The score for all elements is added up. So, if a class comprises two methods, one that is 45 lines and another that is 60 lines long, the method length component of the score for the class will be 3.5 points. This means that elements are not just classified as toxic but the score reflects how toxic the element is.

The following table shows the metrics that make up the toxicity score and the corresponding base threshold on which the multipliers are based.

Metric Level Threshold
File Length file 500
Class Fan-Out Complexity class 30
Class Data Abstraction Coupling class 10
Anon Inner Length inner class 35
Method Length method 30
Parameter Number method 6
Cyclomatic Complexity method 10
Nested If Depth statement 3
Nested Try Depth statement 2
Boolean Expression Complexity statement 3
Missing Switch Default statement 1

At this point you might wonder where the selection of metrics and thresholds comes from. Well, we made them up. When we designed the toxicity chart we made a call on what constitutes “toxic” as opposed to just bad. Of course, staying with the method length metric, we normally wouldn’t want to see a 15-line method but that might be disputed. However, we hope that nobody thinks that a 30-line method is acceptable. And in case you’re really uncomfortable with the thresholds, you can obviously change them to build your own toxicity score. We do suggest, though, that you try the values presented here first.

Like many visualisations the creation of the toxicity chart falls into two steps: data acquisition and rendering.

Step 1: For Java projects Checkstyle is the easiest way to get the metrics. The score table above easily translates into a Checkstyle configuration file, which, by the way, is included in the Zip archive attached to this page. Assuming this file is named “metrics.xml” Checkstyle can be invoked like this:

java -jar checkstyle.jar -c metrics.xml -r <source_dir> -f xml -o toxicity.xml

Afterwards, the file “toxicity.xml” contains a list of all components for the toxicity score in an XML format.

Similar tools for other languages should be able to create similar output. It is not a huge problem if the output is somewhat different as it’s always possible to change step 2 to handle different input formats.

Step 2: What is required next is to read the data from XML, aggregate it on a per-class level, and then render the bar chart. Microsoft Excel is good at the latter two but it needs a bit of help to read the XML file. This help comes in the form of a small piece of VBA code. The attached Excel workbook contains a sheet to load the data, an “Open XML” button backed by said VBA code, a pivot table to do the aggregation, and the chart based on the table. So, step 2, really is to open the Excel workbook and load the XML file. That’s all.

Now, if you know me you know that I’ve been working with Macs for a very long time. So, naturally, I’d like to use Excel on the Mac but, alas, in its current version it does not support VBA anymore. So, unfortunately, it’s VMWare Fusion for this one.

This Zip archive contains everything needed to create a toxicity chart for Java. The spreadsheet can be adapted to read input file formats created by tools for different languages; just edit the VBA behind the “Data” sheet.

Update: I have reimplemented the charts in HTML5. See Toxicity reloaded.

23 comments

  1. Andy MacGilvery

    24 November 2008, 13:50

    When I try to import the xml into the excel worksheet I get a “User-defined type not defined” compile error pointing at code “Dim result As XlXmlImportResult”. Is this because I’m using Excel 2002?

  2. erik

    24 November 2008, 15:57

    It looks like XlXmlImportResult is not available to VBA. This could be your version of Excel or it could be a missing Microsoft XML DLL. To be honest, I haven’t seen this problem before.

  3. The Disco Blog » Blog Archive » Meaningful code metrics

    2 December 2008, 23:45

    […] my friend Erik Doernenburg, posted an interesting entry entitled “How toxic is your code?” in which he suggests utilizing a simple bar chart that measures the toxicity of each class […]

  4. Schauderhaft » How to Use Key Figures and How Not to Use Key Figures

    16 August 2009, 23:03

    […] Toxicity […]

  5. Schauderhaft » Developing for Maintainability

    20 December 2009, 21:06

    […] linked to maintainability. Cyclomatic Complexity being possibly the best known. I personally prefer toxicity, which is based on Cyclomatic Complexity and a couple of other measures, which contribute to the […]

  6. Schauderhaft » New Project’s Resolution

    10 January 2010, 21:44

    […] We gonna improve that. Appart from the knowledge I took from Clean Code I’ll plan to use the toxicity metric as an important tool for […]

  7. Ian Cartwright

    25 February 2010, 04:34

    Noticed a problem with Integer overflows for very large values of NPath complexity, the CheckStyle team fixed the same overflow issue themselves a while back.

    The fix is pretty simple, just need to change the declaration for ‘values’ from Integer to Long. See below

    Function CalculateScore(n As String) As Variant
    If (n = “score”) Then
    CalculateScore = n
    Else
    Dim values(0 To 2) As Long
    values(0) = -1
    values(1) = -1

  8. Digital Dim Sum » Blog Archive » Dealing with creaky legacy platforms

    4 February 2011, 07:11

    […] Toxicity, another aggregated metric, has been playing a prominent role in our “system health checks” at ThoughtWorks. Toxicity charts stack multiple static analysis metrics for classes, methods, or components within an application, providing a combined “toxicity” score for each area of the code base (see Figure 2). This gives our clients guidance on where to start looking to fix problems. […]

  9. Danny Arica

    28 October 2011, 03:13

    What does “stack size” mean?

  10. Erik Doernenburg

    29 October 2011, 02:44

    Danny, what “stack size” are you referring to? That article doesn’t mention it.

  11. What am I thinking about? Visualising your technical debt « On The Job Alliance

    21 November 2011, 22:23

    […] find out more about these visualisation methods check out Erik’s blog article and his excellent presentation of Software Quality – you know it when you see […]

  12. Schauderhaft » My Testing and Code Analysis Toolbox

    11 December 2011, 17:42

    […] CheckStyle: A static code analysis that finds lots of bad practices and can check lots of coding conventions too. Integrates in IDEs and CI server. There are other tools in this area which are worth considering: FindBugs and PMD. You can also use Checkstyle in order to gather simple metrics about your code. […]

  13. My Testing and Code Analysis Toolbox | The Agile Radar

    12 December 2011, 20:13

    […] CheckStyle: A static code analysis that finds lots of bad practices and can check lots of coding conventions too. Integrates in IDEs and CI server. There are other tools in this area which are worth considering: FindBugs and PMD. You can also use Checkstyle in order to gather simple metrics about your code. […]

  14. How Toxic is your Code? | OnTechnicalDebt | The Premiere Technical Debt Online Community

    16 December 2011, 09:25

    […] Erik Bornenburg has an excellent post on his blog about how you can visualize your code and understand which part of it is part, compared to the rest of the code. This can be extremely useful in helping managers and non-developers understand the internal quality of the code. This post shows us how Toxicity Charts can help you see the technical debt that has been built up in your software: http://erik.doernenburg.com/2008/11/how-toxic-is-your-code/ […]

  15. Srivatsa Katta

    1 January 2012, 16:23

    Excellent post, we wanted to calculate toxicity on every checkin and monitor how we are improving the legacy code base. The excel sheet import needed some manual effort, so I wrote a small java utility which runs based on the checkstyle report and generates the toxicity (for all the important checks mentioned in this post).

    Yes the report generated is just numbers doesn’t have fancy visual representation of toxicity using graphs and all, but serves the purpose of hooking it up to the continuous integration tool.

    More details : https://github.com/katta/toxicity

    Thanks Erik again for this post which helped me understand how toxicity is calculated.

  16. Metrics based Refactoring for cleaner code

    3 January 2012, 06:53

    […] Toxicity charts have proved very useful in quantifying the amount of technical debt in a code-base. The magnitude of the debt is quantified by comparing a value against an arbitrary threshold. The toxicity is expressed as a score against the threshold. […]

  17. Schauderhaft » What Metrics to Use

    1 April 2012, 20:04

    […] also used toxicity and I still like it. But I also learned from the book Making Software: What Really Works, and Why […]

  18. Schauderhaft » Plans for My Next Project

    20 May 2012, 16:55

    […] We gonna improve that. Appart from the knowledge I took from Clean Code I’ll plan to use the toxicity metric as an important tool for […]

  19. My Testing and Code Analysis Toolbox | Java Code Geeks

    23 October 2012, 04:05

    […] area which are worth considering: FindBugs and PMD. You can also use Checkstyle in order to gather simple metrics about your code.JDepend: does static code analysis of the dependencies of your code. I use it to write tests […]

  20. Cezar Coca

    28 November 2012, 09:53

    We found this visualization technique very useful on refactoring projects. In order to integrate it in our Continuous Integration environment we managed to create a Sonar Plugin to collect the violation and render the chart.

    More details can be found here:
    http://docs.codehaus.org/display/SONAR/Toxicity+Chart+Plugin

    The above plugin can be installed from the Sonar Update Center

    Thank you Eric for this inspiring post

  21. Christos Zoulas

    17 January 2014, 17:52

    The code has a bug; the parameter number message has the numbers reversed (first the base value and then the actual). Here’s a fix:

    diff -r 904d4ca1fc4f web/complexity.js
    — a/web/complexity.js Fri Jan 17 11:49:23 2014 -0500
    +++ b/web/complexity.js Fri Jan 17 11:49:47 2014 -0500
    @@ -43,6 +43,8 @@
    var check = $(enode).attr(“source”).split(“.”).slice(-1)[0].replace(/Check$/, “”)
    var matches = $(enode).attr(“message”).replace(/,/g, “”).match(/(\d+)/g)
    var score = (matches && matches.length > 1) ? (matches[0] / matches [1]) : 1;
    + if (check == “ParameterNumber”)
    + score = 1 / score;
    result[check] = (result[check] || 0) + score
    result.total += score;
    });

  22. No Fluff Just Stuff Austin 2014 | Octocado

    22 July 2014, 06:17

    […] Toxicity Chart – How Toxic is your Code? […]

  23. erik dörnenburg » Articles » H...

    13 September 2014, 21:37

    […]   […]

Leave a comment