Pen Testing Versus Vulnerability Assessments

At the end of the first day of plenary sessions at OWASP App Sec DC 2010, there was a session with a panel of notable pen testers and a moderator.  I went to the beginning of this talk while waiting for another talk to start.  Although I didn't stick around, they started off getting into a rather interesting topic: what is the difference between a penetration test and a vulnerability assessment (VA)?  I thought this was interesting actually mainly because it's a common topic throughout the community, and it spurred some thoughts on what is absolutely wrong about some of the discussion that is currently going on.

The panel discussed a bunch of different issues, but basically said that a penetration test involves the compromise of systems.  If you don't actually plant a flag on a system, then it's not really a pen test.  They also noted that they are often (80% of the time) asked to perform pen tests against production systems.  Conversely, a vulnerability scan is just a click-button exercise using Nessus or Retina.

The problem I have with this is that the conversation is focused on the tester.  Every so often you'll get a thread going on one of the related lists at, and although 99% of the discussions there are extremely helpful/informative, too often this topic turns to an appendage measuring contest.  Obviously there are those in the field who sell these types of services on the list so it's a touchy subject involving lots of money to be sure.  But here's two basic problem with this discussion:

  • Performing a security assessment does not involve a binary decision wherein pen test is the 1 and VA is the 0.
  • If we're talking about it from the perspective of the tester instead of the client, then we shouldn't even be having the conversation.

In order to illustrate the first point better, I think it is simpler to just list some different scenarios, and ask "which is it, pen test or VA?"

  1. A non-credentialed scan using Nessus or Retina.
  2. A credentialed (admin) scan using Nessus or Retina.
  3. A credentialed (admin) scan using Nessus or Retina, and then follow-up verification of the results using manual assessment techiniques.  However, you stop short of actually compromising systems.  This may mean making sure you can get the reverse shell on the box using metasploit, but not creating a new user account.
  4. The same as #3 above, but also including manual probing of services and protocols to identify other vulnerabilities.
  5. Full assessment using DoS attacks, creation of new accounts on servers, and then using those to try to leverage trust relationships with other systems, attempting to export data from compromised systems in order to prove the seriousness of the compromise, etc.

These are just a few of the different options you might put together when designing a security assessment for a client.  And my point is that from their perspective it is largely useless to worry about which of the 5 qualifies as a "pen test."  The key thing to remember is that just as the panel said- we are often doing these assessments against production systems/networks so it is very probable that while the client wants you to really "dig deep" they also want you to not destroy anything.  As a result, it usually makes sense to do something like #3 or #4, but to avoid some of what would generally be included in #5 as it can be dangerous.

Anecdotally, we had a client earlier this year we were performing a security assessment on, and I was able to gain admin access to the web interface for their SAN.  At that point, if I were a "real" pen tester according to some I would have probably done something in there like provide myself access to the SAN so I could access existing files or upload new ones, or locked out the real admins by changing the password.  However, if I've got a screen shot to prove I was in there do I really need to place 2TB of the client's data at risk?  No.  While I doubt anyone would disagree on that point, I use a somewhat extreme example to try to highlight the uselessness of this pen test versus VA discussion.

I think the pen test panel would agree as well.  One of the key comments made by Kevin Johnson was that "a pen tester can tell a story."  With a VA you get a list or spreadsheet with hundreds or thousands of vulnerabilities listed.  With a pen test you should get a better understanding of what those mean.  To me, that has always meant that you need to do a better risk assessment.  When Nessus tells me that FTP is running in one environment versus another the resulting risk is not always the same.  As a result, the basic click-button VA is limited in usefulness.  Your client will almost always benefit from more analysis into what that means to them.  How far you need to go in your testing to figure that out is something you need to determine as a service provider.

Lastly, your more detailed and in-depth security assessment work will almost always benefit from an understanding of your client.  In essence, if the person who will consume the results is someone who generally does not understand the need for security, or doesn't understand the impact of vulnerabilities, then you need to go farther to develop proofs of concept.  You may need to plant that flag on a number of servers, or ship some scrubbed data sample off to an external system.  This might be what is necessary for them to "get it."  But in other cases you may be taking unnecessary risks with their production systems and information assets in doing so.

The point is that generally, the purpose of a "pen test" is to tell a story of what attackers could do, not to prove the l33t hacking skills of the tester to someone.  Therefore, create the test methodology that is appropriate for each client.

As a final anecdote, I'll offer a story relayed to me by a colleague.  They work in a small sub-organization of a larger entity.  The headquarters security team contacted them one day to say "we are coming in to do a pen test."  When my colleague received the rules of engagement, basically everything was in scope: aggressive DoS attacks, password cracking, data theft, etc.  This was to be executed against a production network with highly sensitive data making a lot of the testing completely inappropriate.  When pressed on these issues, the team that was to perform the pen test started to do a little soft-shoe saying, "oh well we wouldn't actually do the DoS attacks, etc.  That's just the standard ROE language."  They were then asked what tools they would be using, and could not provide a straight answer.

Again, my point is that if you understand the goals of the client (or in this case, your own goals as the headquarters department initiating the test), then you are very likely to fail regardless of whether you perform a "real" pen test or not.  And if you understand the client's goals and have an idea of the scope you should always have a methodology for the work.  In fact, if you can't define one (it doesn't have to be overly detailed, but at least know what tools you'll use and the general process you'll follow!), then you aren't ready to start the work.

I get the feeling that people who really are performing detailed technical assessments don't have the pissing-contest style discussion on the topic.  In essence, if you do it as a profession, then you're likely to discuss the topic in a professional manner.  If not, then you are more likely to resort to saying "what you're doing is not a pen test!  My hack sk33lz are > than yours."

More from Our Cybersecurity Experts