SolCover

Code coverage for Solidity testing

For more details about what this is, how it works and potential limitations, see the accompanying article.

Install

$ npm install --save-dev https://github.com/JoinColony/solcover.git#truffle3

Run

$ ./node_modules/solcover/exec.js

Tests run signficantly slower while coverage is being generated. A 1 to 2 minute delay between the end of Truffle compilation and the beginning of test execution is possible if your test suite is large. Large solidity files can also take a while to instrument.

Configuration

By default, Solcover generates a stub truffle.js that accomodates its special gas needs and connects to a modified version of testrpc on port 8555. If your tests will run on the development network using a standard truffle.js and a testrpc instance with no special options, you shouldn't have to do any configuration. If your tests depend on logic added to truffle.js - for example: zeppelin-solidity uses the file to expose a babel polyfill that its suite requires - you can override the default behavior by declaring a coverage network in truffle.js. Solcover will use your 'truffle.js' instead of a dynamically generated one.

Example coverage network config

module.exports = {
  networks: {
    development: {
      host: "localhost",
      port: 8545,
      network_id: "*" // Match any network id
    },
    coverage: {
      host: "localhost",
      network_id: "*", 
      port: 8555,         // <-- Use port 8555  
      gas: 0xfffffffffff, // <-- Use this high gas value
      gasPrice: 0x01      // <-- Use this low gas price
    }
  }
};

You can also create a .solcover.js config file in the root directory of your project and specify some additional options:

• port: The port you want Solcover to run testrpc on / have truffle connect to.
• testrpcOptions: A string of options to be appended to a command line invocation of testrpc.
  • Example: --account="0x89a...b1f',10000" --port 8777".
  • Note: you should specify the port in your testrpcOptions string AND as a port option.
• testCommand: By default Solcover runs truffle test or truffle test --network coverage. This option lets you run tests some other way: ex: mocha --timeout 5000. You will probably also need to make sure the web3 provider for your tests explicitly connects to the port Solcover's testrpc is set to run on, e.g:
  • var web3 = new Web3(new Web3.providers.HttpProvider("http://localhost:8555"))

Example .solcover.js config file

module.exports = {
    port: 6545,
    testrpcOptions: '-p 6545 -u 0x54fd80d6ae7584d8e9a19fe1df43f04e5282cc43',
    testCommand: 'mocha --timeout 5000'
};

Known Issues

Hardcoded gas costs: If you have hardcoded gas costs into your tests some of them may fail when using SolCover. This is because the instrumentation process increases the gas costs for using the contracts, due to the extra events. If this is the case, then the coverage may be incomplete. To avoid this, using estimateGas to estimate your gas costs should be more resilient in most cases.

Events testing: Because Solcover injects events into your contracts to log which lines your tests reach, any tests that ask how many events are fired or where the event sits in the logs array will probably error while coverage is being generated.

Using require in migrations.js files: Truffle overloads Node's require function but implements a simplified search algorithm for node_modules packages (see issue #383 at Truffle). Because Solcover copies an instrumented version of your project into a temporary folder, require statements handled by Truffle internally won't resolve correctly.

Coveralls / CodeCov: These CI services take the Istanbul reports generated by Solcover and display line coverage. Istanbul's own html report publishes significantly more detail and can show whether your tests actually reach all the conditional branches in your code. It can be found inside the coverage folder at index.html after you run the tool.

Examples

• Metacoin: The default truffle project
  • HTML reports
  • Metacoin with Solcover installed (simple, without configuration)
• zeppelin-solidity at commit 453a19825013a586751b87c67bebd551a252fb50
  • HTML reports
  • Zeppelin with Solcover installed (declares own coverage network in truffle.js)
• numeraire at commit 5ac3fa432c6b4192468c95a66e52ca086c804c95
  • HTML reports
  • Numeraire with Solcover installed (uses .solcover.js)

TODO

• Turn into a true command line tool, rather than just a hacked-together script
• Release on NPM
• Support for arbitrary testing commands
• You tell me