Selection of penetration testing methodologies: a comparison and evaluation

RELATED WORK 
The ISO/IEC 25010:2013 quality standard (Standards Australia, 2013) defines a product quality model 
composed of eight characteristics, further divided into sub-characteristics that relate to static properties of 
software. We consider the suitability of six different penetration frameworks and methodologies and discuss the 
above-mentioned quality standard as a means of selecting evaluation criteria for the frameworks. 
Penetration Testing Frameworks and Methodologies 
ISSAF is an Open Source, peer-reviewed, penetration testing framework created by the Open Information 
Systems Security Group (OISSG). ISSAF is described as a framework and encapsulates multiple methodologies 
in draft 0.2.1B. ISSAF attempts to cover all possible domains of a penetration test from conception to 
completion. The authors suggest that is easier to remove information rather than develop it from the ground up 
(OISSG, 2005). The penetration testing methodology embedded within the framework is divided into three 
primary phases, namely; planning and preparation, assessment, and, reporting and clean up. One advantage of 
ISSAF in particular is that the distinct relationship between the tasks and their associated tools for each task are 
shown. 
OSSTMM is an open source security testing methodology introduced in 2000 by the Institute for Security and 
Open Methodologies (ISECOM). OSSTMM was developed under peer-review and benefits from open source 
licensing, however, access to the latest version (v4), requires paid membership. OSSTMM (v3) is defined as a 
methodology that encapsulates modules and channels whereby channels represent different domain areas 
(ISECOM, 2000). OSSTMM is primarily an auditing methodology thus is not as comprehensive as ISSAF and 
does not provide tools or methods for completing modules however it is a valuable auditing resource that can be 
used to satisfy regulatory requirements for corporate assets provided security auditors have sufficient skills to 
complete each phase. 
OWASP is a not-for-profit organisation focused on improving software security. OWASP provides numerous 
tools, guides and testing methodologies for cyber security under open source licenses, in particular, the OWASP 
Testing Guide (OTG). OTG is divided into three primary sections, namely; the OWASP testing framework for 
web application development, the web application testing methodology, and reporting. The web application 
methodology can be used independently, or in conjunction with the testing framework; a developer can use the 
framework to build a web application with security in mind followed by a penetration test (web application 
methodology) to test the design. Therefore, OTG has a strong focus on web application security throughout the 
entire software development lifecycle as opposed to the ISSAF and OSSTMM, both of which are aimed at 
security testing an implementation. OTG is targeted specifically to a single domain area, that of web 
applications. 
Building Security in Maturity Model (BSIMM) is a software security framework licensed under Creative 
Commons and authored by McGraw, Migues, and West (2009). In developing BSIMM, its authors observed the 
security practices implemented in sixty-seven highly successful companies. BSIMM consists of 112 activities 
divided into twelve practices, supporting four domains mainly; governance, intelligence, SSDL touch points, and 
deployment. In comparison to ISSAF and OSTMM, BSIMM does not specify what tools to use or how to use 
them, but describes practices used by successful companies. Pentesting is one of the practices identified within 
BSIMM however pentesting is only one process of many recommended activities. 
Penetration Testing Execution Standard (PTES) is a penetration testing standard that was originally created in 
2009 by Nickerson et al. (n.d). PTES includes pre-engagement interactions, intelligence gathering, threat 
66

---

modelling, vulnerability analysis, exploitation, post exploitation, and reporting. PTES takes advantages of other 
resources with the approach of not reinventing the wheel, rather, incorporates other frameworks within it, for 
example; OWASP for web application testing is referenced and recommended for use when testing web 
applications. PTES attempts to create a baseline for penetration tests whereby a security practitioner and/or 
organisation have a reference for what to expect at a minimum concerning penetration testing requirements. 
Metasploit is a suite of penetration testing and intrusion detection tools designed to identify and exploit 
vulnerabilities on a target system. Metasploit was originally an open source project developed in 2003 but was 
acquired in 2009 by Rapid7 which is now responsible for its development and support (Holik, Horalek, Marik, 
Neradova, & Zitta, 2014).
Metasploit, or the Metasploit Framework (MSF), is available in four different 
versions. MSF is suitable for the advanced security professional who has a solid understanding of penetration 
testing and is competent using command line pentesting tools. In comparison to ISSAF and OSSTMM, MSF is a 
practical solution that provides a suite of tools rather than a documented outline of process and methods to 
follow. MSF could be considered an application that encompasses a suite of tools that facilitate a penetration 
test. 
In summary, there are a diverse range of methodologies and frameworks available. Each has unique 
characteristics and takes a distinct approach to penetration testing. The literature suggests a difference in the way 
terminology is applied to each concept, thus terms are used interchangeably (or incorrectly). For instance, ISSAF 
is defined as a framework however throughout the documentation it refers to methodology as the primary 
approach. MSF, on the other hand describes itself as a framework whereas it is a software application 
encompassing a suite of tools, therefore clarification on the classification of methodology vs. framework is 
essential to avoid confusion.