Selecting candidates for the evaluation of quality was determined by two criteria. First, whether or not a
particular candidate classified as either framework or methodology. Candidates that fall under a methodology or
framework were deemed in-scope, thus eliminating all other candidates. The second criterion was to examine the
boundaries of a particular candidate for its scope, in other words, whether or not a particular candidate is focused
entirely on penetration testing as opposed to an overall security assessment. The research being undertaken has a
primary goal of evaluating penetration testing methodologies and frameworks explicitly rather than assessing the
security posture of an organisation in its entirety, therefore candidates that are categorised as penetration testing
explicitly are preferred over security assessment specific candidates. As a result the
two remaining candidates
are ISSAF and OTG.
Next, quality characteristics were nominated (see figure 2), for the purpose of evaluating the refined subset of
frameworks. Two factors were taken into consideration for selection of quality characteristics. First, the field of
study or context from which a characteristic definition was drawn (in particular that of information systems was
preferred); and second, whether or not a particular quality characteristic was directly
applicable to the field of
penetration testing.
Figure 2: Penetration Testing Quality Model (adapted from ISO/IEC 25010:2013).
From the revised taxonomy shown in table 3, the selected quality metrics are applicable to both frameworks.
Both frameworks display evidence of the quality characteristics applicable to penetration testing, therefore the
six quality characteristics selected
are considered suitable, thus will be used to facilitate this research in
evaluating efficiency
for the two chosen candidates, ISSAF and OTG. Note that reliability,
whilst a valid
characteristic, was not tested in this evaluation due to lack of delivery of an expected real-world case study.
Table 3: Quality Matrix of Selected Penetration Testing Frameworks.
69