Selection of penetration testing methodologies: a comparison and evaluation

DISCUSSION 
Gap analysis
The gap analysis showed in some cases that the classification of a particular framework and/or methodology can 
often be misleading, for instance; MSF is described as a framework; however, a subsequent evaluation of 
characteristics using factors outlined in table 1, showed that MSF more closely aligns with a suite of penetration 
testing tools, therefore is appropriately classified as an application suite that can facilitate a penetration test 
rather than a framework. In contrast, OTG was pre-classified as a standard or guide, however strong framework 
characteristics are identified throughout the documentation that suggest framework characteristics in contrast to 
its original classification, thus, the post-evaluation classification more appropriately aligns with framework. 
Turning to PTES, the characteristics do not illustrate enough properties to be considered either a methodology or 
framework, due to incomplete documentation or loose structure when compared to the more mature frameworks 
evaluated. It should be noted however that PTES has the potential to be further developed into a framework 
should future amendments be undertaken; as a consequence PTES classifies as a resource post-evaluation. From 
the six frameworks and/or methodologies reviewed, three (ISSAF, OSSTMM, and BSIMM), agree with the pre-
evaluation classification, in other words did not change classification post-evaluation. As can be expected, the 
three aforementioned frameworks are considered mature, therefore it is not surprising that the classification of 
these three in particular, do not change post evaluation. Although not all the frameworks and methodologies 
evaluated show disparity with relation to classification it is important to note that some do, of which 
consideration needs to be given. The consequence of inappropriate classification lends itself to the possibility 
that penetration testing practitioners risk implementing or become reliant on a framework or methodology that 
might not meet an organisations goals in relation to completing a penetration test in its entirety, moreover 
adapting an approach that could potentially fail.