Selection of penetration testing methodologies: a comparison and evaluation

Quality Models 
Figure 1 describes a generic quality model. Standards Australia (2013) clearly state that “It is not practically 
possible to specify or measure all sub-characteristics for all parts of a large computer system or software product. 
Similarly it is not usually practical to specify or measure quality in use for all possible user-task scenarios. The 
relative importance of quality characteristics will depend on the high-level goals and objectives for the project. 
Therefore the model should be tailored before use as part of the decomposition of requirements to identify those 
characteristics and sub-characteristics that are most important, and resources allocated between the different 
types of measure depending on the stakeholder goals and objectives for the product.” Therefore, we have 
amended the ISO model to focus less on software quality evaluation and more on aspects of penetration testing 
framework evaluation (see figure 2). 
The ISO9126 standard contains a taxonomy that defines software by its functionality, reliability, usability, 
efficiency, maintainability and portability. Security, which is the primary focus of this research, is defined as a 
sub-characteristic of functionality. This is a departure from the commonly held belief that security is solely 
within the domain of non-functional requirements. The ISO25010 standard (Standards Australia, 2013) extends 
upon this idea and considers security a characteristic in its own right. ISO25010 is, therefore, a replacement for 
ISO9126. 
Figure 1: Abstract Quality Model (adapted from ISO/IEC 25010:2013). 
67

---

EVALUATION OF FRAMEWORKS 
First, six methodologies/frameworks were selected for potential use. The selections were classified into either a 
methodology or framework category respectively by way of gap analysis to establish whether or not framework 
or methodology characteristics were present focusing primarily on the field of penetration testing (see table 2).
Table 1 summarises the gap analysis. Original classification (pre-evaluation) and post evaluation classifications 
are shown. The evaluation classifications illustrated in table 1 are an ordinal scale, for example; framework 
encapsulates methodology and methodology encapsulates tools, techniques, and resources. In addition 
frameworks that have included methodology in a particular framework are identified. We arrived at an 
individual evaluation based on theoretical analysis of each framework/methodology. Each framework or 
methodology underwent analysis to determine whether or not framework or methodology characteristics were 
present (or absent). 
Table 1: Evaluation Matrix 
From the gap analysis it was possible to develop a taxonomy of penetration testing specific frameworks or 
methodologies whereby suitable candidate methodologies or frameworks can be identified to facilitate their use 
in practice or research. We found that some frameworks were indeed frameworks in the accepted sense of the 
word, whereas others were simple collections of tools without a discernible underlying ontology. We felt that 
this was an important distinction because novice pen testers would benefit from the additional support provided 
by a mature framework. 
Table 2: Classification of Penetration Testing Frameworks and Methodologies.