Selection of penetration testing methodologies: a comparison and evaluation

Measurement of Frameworks with Quality Characteristics 
Both OTG and ISSAF were measured by the quality characteristics denoted in table 3. In some cases this 
measurement was not direct, as the characteristic did not have a direct mapping to a concept/artefact used in 
pentesting. An example would be “maintainability”. 
Conventional measures of software maintainability are not suitable for pentesting frameworks as the artefacts in 
question are documents or risk matrices rather than software artefacts (such as code). However, some 
commonalities exist as penetration testing in its essence is testing software and/or hardware with software of 
some type in most cases.
Maintainability is the ease in which a framework can be understood, adapted, enhanced or modified. To measure 
this characteristic, consideration is given particularly to the number of revisions (frequency of change) a 
framework has undergone since inception. The frequency of change (actually a sub-characteristic of 
maintainability) can be quantified by the number of revisions a framework has undergone in its lifetime, the 
underlying assumption being that revisions reflect a measure of activity. The type of activity, whether bug fixes, 
documentation readability, addition of new features or other activity is not considered in isolation in this 
instance. Both OTG and ISSAF do not offer enough comparative information; therefore that level of detail is 
restricted. It is also worth mentioning that other sub-characteristics, namely; applicability, analysability, 
testability, and relevancy, can be incorporated as additional measures. Discussion of these other sub-
characteristics is outside the scope of this particular research, therefore we focus here on one sub-characteristic 
that can be quantified. 
Table 4: Total number of revisions (r) / framework lifetime in years (t) 
Revisions 
Years 
Date 
of 
Inception 
ISSAF 
5 
2 
December, 
2004 
OTG 3.0 + 4.0 
468 
(245 + 223) 
7 
May, 2008 
Table 4 shows that OTG averaged 66 revisions per year in comparison to ISSAF which averaged 2.5 revisions 
per year. While it is obvious OTG had far more revisions than its counterpart it does not necessarily prove that 
ISSAF lacks quality. Moreover, other factors come to the fore, for example; Does OTG have more contributors 
than ISSAF?, Were revisions for OTG related to bad design, or does it simply suggest the product is better 
70

---

overall? These questions are considerations for future research, however we conclude that OTG is more 
maintainable than ISSAF, due to its higher revision activity. 
Similarly, Usability is measured by its sub-characteristic, Readability. Readability is significant as a sub-
characteristic of usability primarily because the penetration testing frameworks evaluated in this research are 
documents, therefore if a document is not readable, usability is affected. Readability is concerned with the level 
of difficulty to read or comprehend written text (Ludger & Gottron, 2012). There are a number of ways to 
measure vocabulary difficulty and sentence length to predict the difficulty level of text resulting in various 
readability formulae in use today (DuBay, 2004). One such formula known as the Gunning Fog Index (GFI), 
was published by Gunning (1952, p. 36), developed specifically for adults. Fog index attempts to estimate the 
number of years of education that is required by the reader in order to understand the text at first reading, for 
example a GFI score of 10 would indicate that ten years of formal schooling is required to understand the text. 
The formula works with two variables; first, the average sentence length (ASL), and second, the number of 
words with more than two syllables for each one hundred words (PHW). Finally the result is multiplied by 0.4 
(DuBay, 2004), thus produces the formula: Grade Level (GL) = 0.4 (ASL + PHW).
Table 5: Fog Index Scores 
Framework 
ISSAF 
OTG 
GFI Score 
7.7 
11 
Results were obtained using an automated tool that calculates various readability scores, among them GFI score. 
The readability score automation tool is available for public use released as an open source project ("Readility-
Score," 2015).
From table 5 it is clear that ISSAF is more readable than OTG. Taking into consideration the intended users of 
both frameworks it can be safely assumed that a security practitioner would have a minimum of 11 years formal 
schooling therefore readability scores for both frameworks are sufficient.