Selection of penetration testing methodologies: a comparison and evaluation

Keywords 
Penetration Testing, Methodology, System Security. 
INTRODUCTION 
The rate of cyber security threats detected for business and government is increasing, with approximately 7,300 
incidents reported to CERT Australia in 2012 and approximately 8,500 incidents reported by August 2013 
(CERT, 2013). Consequently, the cyber security industry is growing at a rapid rate with worldwide spending 
expected to reach US$86 billion by the year 2016 (Gartner, 2012). Australian security and intelligence agencies 
have stated that Australia is experiencing an increase in sophisticated cyber-attacks in both government and 
business originating from an array of sources: individuals, organised criminals and foreign intelligence services 
(CERT, 2013). The 2013 Cyber Crime and Security Survey showed an overall increase in reported cyber 
security incidents from 56 organisations reporting incidents in 2012 to 76 in 2013 (CERT, 2013). Fortunately, 
mitigation strategies are available for organisations, governments, and individuals to minimise risk. One 
mitigation strategy commonly used within the cyber security industry is penetration testing, commonly referred 
to as pentesting (Tang, 2014).
Pentesting aims to evaluate information security measures through the eyes of a potential attacker with the aim 
of testing the effectiveness of security controls (Midian, 2003). Pentesting is often employed by organisations as 
a mitigation strategy to reduce the risk of an attack on computer resources or in some cases, critical 
infrastructure. Pentesting attempts to ensure weaknesses and vulnerabilities in a networked environment are 
detected and can be addressed before they are exploited in a real-world attack (Tang, 2014). A security 
practitioner tasked with penetration testing will conduct a series of security tests in an attempt to gain access to a 
system and exploit security flaws that exist using the same tools and techniques that simulate a malicious attack, 
but do so in a controlled manner (Yeo, 2013). A properly scoped and deployed pen test can be an invaluable tool 
to assess the ability of a system to survive malicious attack (Valli, Woodward, Hannay, & Johnstone, 2014). The 
cornerstone of a successful pen test is its underlying methodology. A well-defined methodology plays a critical 
role in achieving results that can be verified and studied to protect data, applications and underlying 
infrastructure. Without an established methodology or framework within which to conduct a pen test, identifying 
vulnerabilities accurately can become difficult or provide a false sense of security (Frankland, 2009). Wilhelm 
(2009, p. 154) asserts that penetration tests are projects that need to be developed using effective and repeatable 
processes for improvements to be made, businesses goals to be met, and quality improved, therefore a 
methodology is a crucial factor. This suggests that penetration testing is achieving some level of maturity, akin 
to software engineering, although the lack of attention paid to software vulnerabilities in initial system releases 
may be due to the fixation of project managers on visible functionality as noted by Johnstone (2009). 
Avison and Fitzgerald (2006, p. 418) discuss in detail the loose but extensive use of the term “methodology” and 
argue that there is very little agreement as to what it means other than at a very general level. Furthermore there 
is little in the literature addressing frameworks and methodologies for the purposes of penetration testing 
specifically. Consequently pentesting methodologies and frameworks appear to be poorly defined. Despite this 
65

---

confusion of terms there are many pentesting methodologies/frameworks available. Certain frameworks or 
methodologies are free to use whereas others require some form of membership, payment or contribution, for 
example; technical input to the framework or methodology. Several pentesting methodologies and frameworks 
widely available in particular include: Open Source Security Testing Methodology Manual (OSSTMM), 
Information Systems Security Assessment Framework (ISSAF), Open Web Application Security Project 
(OWASP), Metasploit Framework (MSF), and Building Security in Maturity Model (BSIMM) Penetration 
Testing Execution Standard (PTES). 
The purpose of this research is to evaluate a selection of currently available pentesting methodologies and 
frameworks (see above). We perform a gap analysis to determine if a pentesting framework is actually a 
framework, i.e., it has a sound underlying ontology. A subset of these frameworks is evaluated against quality 
criteria, which will determine their suitability for real world applications.