Ethereum Smart Contract Testing Guidelines
Tags: testing, ethereum, blockchain, solidity
Published: 2019-12-19 21:40:36

In a way, developing smart contracts is a throwback to a time where software was disseminated on cds with code that, once installed, must be as bug free as possible and work every time. The mantra of “move fast and break things” does not apply when it comes to smart contracts. When you deploy a smart contract, you are mostly commiting to a set of features and implementation details. While upgrade mechanisms exist, there are still potential minefields because of complications such as the existing memory layout, the compiler version, etc. – it’s best to just get it right the first time. To quote Happy Gilmore, “Oh, man. That was so much easier than putting. I should just try to get the ball in one shot every time.” To ensure any smart contract is feature complete and executes as intended in all scenarios, be thoughtful in your design, get an audit if possible, and test exhaustively.

There have been a few studies to see what kind of bugs tend to exist in smart contracts and how to catch them. Trail of Bits, a premier security firm in the cryptocurrency space and an auditor of the PAX contract, posted [an excellent summary](https://blog.trailofbits.com/2019/08/08/246-findings-from-our-smart-contract-audits-an-executive-summary/) on different flaws they found in all of the smart contract audits they’ve ever completed. They estimate that 49% of the flaws or vulnerabilities they’ve found could only be caught by a human. This doesn’t diminish the importance of automated testing, but instead emphasizes the importance of human care and attention.

## The Programmer’s Mindset
Before you sit down to write a single line of code, develop some guiding principles and create a process that works for you. Here at Paxos, we have specific principles that help us to work quickly while maintaining thought and focus – we don’t code on auto-pilot. As Leonardo Da Vinci once said, “The painter who draws merely by practice and by eye, without any reason, is like a mirror which copies everything placed in front of it without being conscious of their existence.”

### Always Ask Questions
Whenever writing code that represents business logic or customer activity, always question assumptions and expectations. Your code will execute exactly what you write according to its instructions. The computer is never wrong. It is imperative to completely understand what the smart contract should do in different situations and that it matches expected business logic.  
**Remember: bugs arise when your mental model is out of sync with the code.**


Both before and during your smart contract creation, ask yourself the following questions:

1. What are the bounds of this method?  
    1. Consider overflows and underflows.  
1. How can this be simplified?  
    1. Less code = less bugs.  
       Discover how we at Paxos [designed the PAX stablecoin smart contract](https://www.paxos.com/designing-paxos-standards-minimal-smart-contract/).
1. Who can call this method? Should they be able to?  
    1. Remember other smart contracts are potential callers.  
    1. Consider front-running.  
    1. Consider authentication and access control.  

## Different Testing Strategies
Just like you wouldn’t buy a house based only on the layout of the rooms, you can’t consider testing done just by doing enough unit tests to ensure the internals of the code act as expected. There is so much more to consider, including how the house and its location fits into your day to day life (integration tests), what others think of it (human attention), a home inspection (audits) and other reports and information available such as previous natural disasters in the area (static analysis, and generative testing). Three major testing strategies we will discuss are Unit and Integration Tests, Human Attention and Audits, and Generative Testing. Each has its own domain of expertise and covers different aspects of your code.

![Venn Diagram of Testing](https://miro.medium.com/max/1248/1*5wh6pviwlBKyMM3H1AknCg.png)

### Unit Tests
The main purpose of unit testing is to ensure the business logic is correct and operates as expected. Solidity can be a bit weird sometimes and unit tests will help adhere to expected behaviors<sup>1</sup>. It is absolutely key that you question assumptions while coding. While obvious, it’s easy to forget that tests only evaluate what you tell them to – that means you need to have tests for edge cases and flows you don’t think will happen. All this being said, testing smart contracts is a joy. They are beautifully encapsulated little testable functions by default. [Truffle](https://www.trufflesuite.com/docs/truffle/testing/testing-your-contracts) is great to work with for unit testing contracts and is what we use at Paxos. Take a look at their documentation to get up and running.

#### Truffle
Truffle is a powerful test runner that sits on top of Mocha and uses Chai’s assertion library. You can do everything you can do in javascript testing with Mocha including running something before each test, embedding tests within tests, etc. We found it useful to wrap our smart contract in order to add a bit more functionality for testing. For example, we added an `initializeBalance` method in order to quickly assign new funds to an account. This proved especially useful for quick setup for each test. Think about other functionality you are often doing that can be abstracted away to a simple wrapper contract.

In order to interact with your contract in a way that is realistic, you may want another smart contract to interact with it. For example, if you are testing a DEX that has deposit wallets, you may want to mock out those deposit wallets for better integration tests. For more examples see our open source token contracts for [PAX](https://github.com/paxosglobal/pax-contracts/tree/master/test), [BUSD](https://github.com/paxosglobal/busd-contract/tree/master/test), and [PAXG](https://github.com/paxosglobal/paxos-gold-contract/tree/master/test).

### Application Integration Tests
It’s likely that your application is not entirely written in smart contracts and has more complex business logic that results in on-chain transactions. In order to truly test your application end to end, you should run these on-chain transactions in your automated test suite before pushing any changes upstream. We at Paxos often use golang and go-ethereum’s simulated backend works well for this. The simulated backend did not have enough features so we enhanced it and [opened a PR](https://github.com/ethereum/go-ethereum/pull/20208) to merge this upstream. It is now an extremely good drop in replacement for an ethereum client and enables us to write full integration tests with confidence.

### Human Attention
Code reviews by others in your organization are a no-brainer. They can help to determine clear and consistent naming conventions, to ask questions about business logic that you may forget, and to suggest refactorings that could simplify the codebase. While audits cover much of the same purpose, having an internal review helps ensure the implementation matches the logical expectation of the different features. More eyes on the code means more diverse knowledge bases and viewpoints to inspect and scrutinize different aspects that would otherwise go unnoticed.

### Smart Contract Audits
If you can get them, smart contract audits offer invaluable insight into potentially critical bugs that are nearly impossible to catch on your own. Even if you are diligent about tests, static analysis, and have researched known potential vulnerabilities, you can still introduce flaws due to the simple fact that you are human and have conceptual blindness while testing your own code. And, most likely, you’re not a true expert. For example, would you think about how front-running a particular call could impact the logical flow of another<sup>2</sup>? Or what would happen if a different token that your contract interacts with does an upgrade<sup>3</sup>? Or how potential opcode price changes in a future Ethereum hard fork could affect your contract<sup>4</sup>? All of these scenarios could lead to serious consequences if not properly handled. Auditors find smart contract bugs for a living and have a deep understanding of Solidity, the EVM and code analysis in general. Ethereum and the EVM is ever updating and true expertise takes time. Generative testing tools can be difficult to use at first and there are people who work with them for a living who can write more comprehensive tests. Even beyond the security concerns they may expose, auditors can suggest ways to improve your code, access control mechanisms, or upgrade strategy. We at Paxos have received multiple audits and found them tremendously helpful<sup>5<sup>. All this is to say, if you can, its worthwhile to get an audit.

### Static Analysis 
At the very least you should run some static analysis tools like [Slither](https://github.com/crytic/slither), [Mythril](https://github.com/ConsenSys/mythril/), [Securify](https://github.com/eth-sri/securify), and [Ethlint](https://github.com/duaraghav8/Ethlint) as part of your development cycle. They are extremely easy to use and will test your code against known vulnerabilities that are not difficult to catch as well as provide tips and suggestions on improving your code style. For example, to run Slither just run `slither .` in the smart contracts directory and within seconds you will see a full analysis of your contracts including linting for best practices and potential security issues. Mythril works in a similar fashion and will inspect your code by running `myth analyze <solidity-file>`. These tools can find reentrancy bugs, uninitialized storage, overflow and underflow bugs as well as many others. All these tools are updated frequently with new potential vulnerabilities and are a great addition to any testing story.

Manticore is a symbolic execution tool to help analyze your contracts. It can do a lot on its own if you just let it loose on your contract, but it is best when guided to expose further analysis via Python scripts. You can setup certain conditions, execute transactions and analyze the specific state transitions.

### Generative Testing
If you have the time, and especially if you are not getting an audit, there are a number of great tools that test smart contracts in ways that are otherwise impossible. Generative testing refers to the broad class of testing strategies where test case input data is generated rather than specified for some interesting input space. One kind of generative testing, fuzz testing, will provide pseudo random or truly random inputs to methods in order to try to break a program. Fuzz testing is best for finding overflow or underflow bugs, and really test the boundaries of a program. Property-based is another type of generative testing and can be thought of as a marriage between fuzz testing and unit testing. You define a set of properties that should exist, for example that a token’s supply should never be negative, and unleash fuzz testing to see if this property can be broken. This is especially helpful when you undoubtedly forget an edge case in your unit tests. [Echidna](https://github.com/crytic/echidna) is the only property based fuzzer for Ethereum smart contracts and is fairly simple to use. Simply define a set of methods in your smart contract that are prefixed by “echidna_” that assert properties and run echidna. For more in depth review of what Echidna can do and how you can work with it, see Trail of Bit’s blog post [State Machine Testing with Echidna](https://blog.trailofbits.com/2018/05/03/state-machine-testing-with-echidna/).

## Putting It All Together
When it comes to testing smart contracts, there are a lot of tools out there to help you. Trail of Bits has created an extremely useful docker image, [eth-security-toolbox](https://github.com/crytic/eth-security-toolbox), that includes all of their security tools as well as the ability to jump between different solc versions using `solc-select`. You can load your contracts repository as a volume to this docker image in order to quickly iterate and test your contracts in an environment with minimal setup. This image includes another tool that is self described as a “swiss army knife” for Ethereum called [Etheno](https://github.com/crytic/etheno) that helps you switch between Echidna, Manticore and even Truffle. It can also spin up parity and go-ethereum and enforce client calls to both nodes to ensure working tests across the different clients, which would likely have caught the recent Ropsten Constantinople hard fork upgrade bug<sup>6</sup>.

The Solidity and Smart Contract developer experience has improved leaps and bounds over the past couple years and has created an environment where critical on-chain bugs are less and less common. The wild west of contract security has many new sheriffs, all of whom are improving and protecting the great expanse of decentralized applications. While all these tools are phenomenal and we have never been in a better position to create strong resilient smart contracts, remember that all tests are only as good as the ones who write them. You must question assumptions and think critically about your design. Never assume your unit tests are enough, use the amazing additional tools this community has created, and if you can, please get an audit.

## References
1. [A list of Solidity Idiosyncrasies](https://github.com/miguelmota/solidity-idiosyncrasies)
1. [This could result in a double spend as shown in the audit of Flexa](https://github.com/trailofbits/publications/blob/master/reviews/Flexa.pdf)
1. [This could result in orphaned balances as shown in the audit of Compound](https://github.com/trailofbits/publications/blob/master/reviews/compound-2.pdf)
1. [This happened to a number of contracts after the Istanbul hard fork](https://eng.ambcrypto.com/ethereum-contract-calls-that-ran-out-of-gas-spiked-by-over-1/)
1. Paxos’ Contracts were audited by CertiK, ChainSecurity, Nomic Labs, and Trail of Bits
1. [Ropsten Constantinople Hard Fork Retrospective](https://ethereum-magicians.org/t/issues-we-discovered-in-the-ropsten-constantinople-hard-fork/1598)