Audit reports are technical documents and reading them can prove to be challenging. In addition, the nature of smart contract audits can also be confusing to understand. As an example, many look at audits as binary entities; if a project has been audited, it is considered secure. However, as we shall later see, this is not the case. In this article, we will clear many such misunderstandings and learn how to comprehend an audit report properly.
Gauging auditor reputation
Auditing smart contracts is a highly technical feat. It not only requires deep knowledge of the codebase being audited, but also of other projects within the blockchain ecosystem. As such, auditor reputation and pedigree is an important factor to consider. Generally, the auditor(s) should be well-known, have several years of auditing experience with a few dozen projects audited, and code audited by them should not be known to have been exploited. Having audited high use and high value projects is also a plus.
Reading the executive summary
Audit reports often include an executive summary at the beginning; this is an important piece of information to read. It tells us about the approach the auditing team took towards reviewing the code, the timeframe of the audit and general sentiment of the auditors towards the codebase, along with any auxiliary info. It further informs us of the auditing team’s confidence in the codebase and their approach toward the audit. As an example, if by reading the executive summary, we come to know that the audit period was very short, it would not be unreasonable for us to presume that the audit could have missed something important.
Reading the assessment overview
This section delimits the scope of the audit. It mentions the files that are within the audit-scope, and excludes those that are not. While developers ought to have the entirety of their project audited, it is sometimes not feasible for them to do so; as such, they may opt to have only some files audited. The implication of this is that if the un-audited files were to have a vulnerability, the entire project would suffer from the exploit. Notwithstanding such problems, due to reasons such as resource constraints, scope delimitation remains a reality of smart contract auditing.
Reading the system overview
Audit reports may also have a System Overview section. Therein, the auditing team writes a summarised and holistic overview of the codebase being audited. It informs us of the project structure and the interplay between the various smart contracts which make up the whole codebase. In addition, it also mentions any assumptions which the auditors worked under. Similar to the executive summary, this part is important as it equips us with the bare minimum of the knowledge needed to understand the vulnerability disclosures.
Doing the freshness check
The audited code and the deployed code must be confirmed to be the same. The latter can be found by visiting the smart contract address on a block explorer such as Etherscan, and then going over to the “contract” tab. A verified smart contract would have its code present on its Etherscan page. As a general rule, developers should verify their smart contracts on block explorers or provide compelling reasons for not doing so. In addition, it goes without saying that since block explorers like Etherscan are privately run, trust in them is implied. The decentralised way to fetch on-chain smart contract code is to run a node for the given blockchain, and query it for the relevant information.
Comparing the audited with the deployed smart contract(s) can prove to be infeasible if we do a line-by-line comparison. Most projects house their codebase on version control services such as GitHub; audit reports either mention the commit hash of the GitHub repository of the audited code or include a list of files and their respective hashes.
The commit hash is a unique string of characters and represents the state of a repository on GitHub; it is different for each state of the code, and thus changes when the code is changed, no matter how slightly. Therefore, we can compare the commit hash of the audited code with that of the deployed code, making sure that the two are identical to each other. For audit reports which instead list the hashes of the files, it often is necessary to ask the project team for the specific commit and manually verify that the file hashes match with the files present in that commit.
The reason why we should do this is important; even the slightest change in a smart contract can introduce a severe vulnerability. As such, it ought to be of primary concern to us that the code has not changed between the audit and the deployment.
Granted, it is not easy, or sometimes even possible, to do the above process. However, it is definitely worth a try if one plans to substantially rely on the secure and continued functioning of a project. Even an indication that the audit code and deployed code differ from each other is enough to show that the report is more or less outdated and does not comment on the present, on-chain state of the codebase.
Reading the audit findings
Vulnerability disclosures are made through listing out and explaining the issues with the audited code. The issues are usually divided into four categories of risk or severity: Critical, High, Medium and Low.
Critical issues result in a complete breakdown of function and/ or loss of funds. High and Medium issues handicap the functionality and/ or result in loss of funds, while Low issues point to problems that do not result in a loss of funds, or worsen the code’s execution efficiency.
It is a hallmark of a good audit report that all the issues, especially the critical and high ones, are explained comprehensively. It is also necessary that the quality of the issues highlighted is good and the team has not just copy-pasted the output of static analysers.
Another important thing to look out for is “Severity Inflation” whereby auditors misuse severity labels. Lower severity issues are willingly labeled as being of a higher severity, to the end of increasing the perceived value of the audit.
Understanding the project owners’ response to the audit findings
When reading an issue, we should fully understand what part of the codebase is affected, how it is affected and what can be done to exploit the vulnerability. In addition, we should pay particular attention to how the project ownership responded to the audit findings.
It should be checked if they fixed the issues or opted not to. If they did fix it, we should confirm whether the auditing team okayed the fix. It is possible that an issue is ignored as a non issue but later ends up being the means to an exploit, or a fix is not communicated to the auditors which results in a hack. As such, the project ownership’s attitude towards the audit findings is very important to observe.
Understanding the impact of post-audit exploits
It could happen that a project has been audited, perhaps even more than once, but still ended up being exploited. This can mean several things.
First, it could be that the auditing team missed the particular attack vector which ended up being exploited. The promise of an audit is not bug-free code, but an effort at finding as many vulnerabilities as possible in the given time and resources.
Second, it is possible that the project team made changes post audit. It is not uncommon for projects to make post-audit changes thinking that they would not cause any issues. One example is when developers upgrade their smart contracts and as a result introduce vulnerabilities which were not there at audit time. In such cases, the presence of an audit report does not entail that the current base is audited, only that a previous version of it was.
Third, some projects opt to have audits done at different points of the development cycle. In such cases, even though their website or public GitHub may show the presence of multiple audit reports, this does not mean that the same code was audited multiple times. It just means that different code was audited by different auditing teams at different points in time.
Fourth, there have been cases where projects chose to have different parts of their codebase audited by different teams; the scope of each audit is different. As a result, even though parts of a project have been audited separately, when the system is taken as a whole, it may end up being exploited by an expert team of hackers.
Another fact to consider is that the scope defined for the audit excludes interactions with external contracts. It is reasonable for the project developers to treat external contracts as secure; they cannot go about having every external smart contract they interact with audited. As such, it could happen that while the audited code itself was secure, an external dependency was not, which results in an exploit.
Thus, multiple audits should not be considered as reinforcing each other and thereby signaling a highly audited codebase, simply because the various audits could have dealt with different codebases, commits and/ or scopes.
Conclusion
In this article, we learnt that comprehending an audit report is essential for all stakeholders of a project. It not only tells us of the vulnerabilities that were discovered in the course of the audit, but also how the project ownership decided to deal with the disclosures. It also informs us of the auditing team’s understanding of the project and their confidence in this understanding. We also looked at the implications of an audit, what it entails for both the project users and owners. The nature of an audit report, and the interplay between multiple audits was examined as well.