Welcome to Vulnerability Research

Start HerePage 1 of 410 minutesLessonNo lab next

So, you want to be a wizard

When I was a kid, I wanted to grow up to be a wizard. From Harry Potter to Gandalf the Grey (and, unfortunately, to some extremely trashy fantasy books that are better left unnamed), I loved magic in just about every form I could find it. Finding out that magic wasn’t real was devastating, although it didn’t stop me from trying to move objects with my mind until an embarrassingly old age. Eventually, however, I had to consider doing something else with my life.

Years passed, and I’d given up on my dreams of wizardry when I came across Criminal Minds. Garcia moved through computer systems, found information, and shut off lights from behind her keyboard. Watching her, I started to think that maybe a version of the thing I’d wanted to do all along actually existed.

Obviously, Criminal Minds isn’t a particularly accurate guide to working with computers. But understanding how software works really does let you do things that can seem impossible when you’re only seeing the result. You can take something apart, figure out what makes it work, and learn how to change its behavior yourself.

Vulnerability research is one part of that world. Programs don’t always behave the way their developers intended. They can trust information they shouldn’t, make the wrong decision about who has access to something, or fail when given input nobody thought to check. A vulnerability researcher investigates those mistakes and works out what they mean for the security of the software and the people using it. We’ll usually shorten vulnerability research to VR.

That’s what we’ll do here, using code, debuggers, and experiments to work out why a program behaves the way it does. Then, if we’re lucky, we can use that understanding to make it work for us rather than its developer.

What kind of hacker do you want to be?

Television tends to give all the computer work to one person. Need to find someone, break into a system, or understand a mysterious piece of software? Get the hacker. In practice, those tasks involve different skills, and people can spend entire careers getting very good at one part of the picture.

I’d start by asking which part of that work interests you. Do you want to take a program apart and discover something nobody knew was wrong with it, or would you rather investigate how an organization’s systems hold up when someone tries to get past its defenses? You can do both, but they give you rather different problems to spend your day on.

Researchers and developers

Imagine you’re looking at a document reader that closes every time it opens one particular file. Most people would quite reasonably decide the application is broken and try a different one. A researcher might look at that same failure and want to know exactly what happened.

What is different about this file? Where does the program handle that difference? Did the developer assume a value would always fit in a particular range? If that assumption is wrong, what else can go wrong with it?

Could opening the file let someone take over the computer?

We don’t know yet, but we have somewhere to start. I’d want to compare that file with one the reader opens correctly, then follow the difference into the code that handles it. If changing one value makes the failure disappear, why does that value matter?

A debugger lets us stop while the program is still running and look at what it does with the input. This is the research part: working out what went wrong, then testing whether our explanation actually accounts for it. We may find a serious security problem, or we may spend all that time explaining a much less exciting bug.

You often end up writing software along the way. Changing the file by hand gets tedious if you want to try a hundred variations, so a small program that generates them would be useful. You might also need a tool to make the file’s contents easier to inspect. That’s the development side of the work.

Exploit development takes the question further: given a vulnerability, can we use it to make the program do something it shouldn’t, such as reveal information it was supposed to keep private? A researcher might do that themselves or work with an exploit developer; both need programming skills.

If the question that caught your attention was how one file could break the reader while another worked, you’re in the right place. We’ll spend a lot of this course following that kind of question into a program.

Operators, pentesters, and red teamers

Now suppose you’re looking after an organization that uses that document reader. Knowing about the bug is useful, but you have some more immediate questions. Are your machines running the affected version? Does the way you use the software make the problem reachable? Would anyone notice if something went wrong?

This brings us to the operational side. An operator is someone carrying out the work of an authorized security engagement. Penetration testers, usually shortened to pentesters, look for weaknesses in the systems they’ve been given permission to test and show the organization what those weaknesses allow. A software bug might be part of the answer, but so might a configuration mistake or an account with permissions it doesn’t need.

With our document reader, the researcher could explain the flaw in the application while the pentester investigates whether it creates a problem in this particular organization. The pentester doesn’t have to discover a new bug to find something worth fixing. If a known problem is still present on the machines being tested, the fact that somebody else discovered it doesn’t make it go away.

Red teamers take on agreed objectives in an authorized exercise that simulates an adversary. The organization may want to test its response as well as its technical defenses: did anyone detect the activity, did the alert reach the right people, and could they do something about it? Finding a weakness is useful here because of what it lets the team test. A long list of bugs wouldn’t, by itself, answer those questions.

These aren’t fixed groups of people. A pentester can discover a new vulnerability, a researcher can help with an operation, and an operator can write their own tools. I wouldn’t choose between them by asking which sounds more advanced. Think about whether you’d rather spend the time understanding that document reader or investigating the environment it’s running in.

Same program, different questions

Researcher
Inside the program
Why does this file break the reader?
Pentester
Inside the organization
Does this flaw affect our systems?
Red team
During an agreed exercise
Would we detect it and respond?

The document reader is the same in all three views. These are overlapping kinds of work, not ranks or steps in a career.

Why learn vulnerability research?

There’s a lot to enjoy about research if you like taking things apart. A program gives you a result that makes no sense, and you get to follow it down through the layers until it does. Something that looked mysterious becomes a sequence you can explain and reproduce. I want you to be able to look at an unfamiliar program and have somewhere to start figuring it out.

There is a practical reason to do this beyond satisfying your curiosity. The developer fixing the document reader needs to see what went wrong and be able to check whether the fix works. A small test file and an explanation of the mistake give them something to work with. If that mistake could expose someone’s documents, getting it fixed matters to people who will never open a debugger themselves.

Of course, a debugger will occasionally reward your curiosity with something that makes even less sense than the original problem. Perhaps the value you were sure was wrong is fine, or the program fails before it reaches the function you suspected. You can recover by going back to the last point where it did what you expected and following it from there. We’ll work through examples like that, including the parts where the first plausible answer doesn’t survive the next check.

You don’t have to recognize every instruction before you can start. Looking things up and making smaller examples are part of figuring out unfamiliar software, even when you already know the tools. It helps if you enjoy asking why, though, because you’ll be doing quite a lot of it.

What you’ll learn to do

An unfamiliar program can give you a lot to look at, so we’ll practice narrowing that down to a question we can answer. By the end of the course, you should be able to:

Find your bearings. Work out what the program normally does, what’s different about the behavior you’re investigating, and what you want to test first.
Look inside the program. Stop it in a debugger and connect what you see with the source code or compiled instructions that produced it.
Make a failure reproducible. Find the input and conditions that cause it, then remove the details you don’t need so someone else can reproduce it too.
Explain the mistake and its consequences. Work out why it happens and whether it’s a security problem, keeping track of which possibilities you’ve tested and which are still guesses.
Build tools to support the investigation. Write small programs to test an idea, make awkward data readable, or demonstrate a result in a controlled environment.
Communicate the finding. Give another person the evidence, reproduction steps, and explanation they’d need to check the problem and work on a fix.

You won’t come out knowing everything about every program. The aim is to have somewhere to start, and to be able to work through the parts you don’t understand instead of waiting for somebody else to explain them.

If you’d rather be on the operational side

If the operational side sounds more appealing, I’d look at these resources. They focus on the assessment work this course won’t cover in depth:

TryHackMe’s Jr Penetration Tester path. A place to start if you want a guided introduction to pentesting. It takes you through practical exercises and the process of an assessment, including how to report what you found.
Hack The Box Academy’s Penetration Tester path. Take a look at the module list if you want a substantial course covering an assessment from initial investigation through reporting. It assumes some foundations, so check those prerequisites before jumping in.
PortSwigger’s Web Security Academy. If websites are what interest you, I’d start here. The explanations come with interactive labs where you can try the ideas, and the material is free.

For red teaming, I’d get comfortable with systems and pentesting first, then look specifically at how an exercise is planned and how the defenders’ response is evaluated. Knowing how to find a weakness won’t tell you which scenario the organization needs to test or what would make the exercise useful to it.

And if both sides interest you, there’s no reason to pick one permanently. What you learn about programs here can still be useful when you’re assessing a system later.

What you need to get started

You don’t need previous vulnerability research experience. You should be comfortable using Linux, reading and modifying small Python and C programs, and using GDB to stop a program and inspect its state. A basic understanding of processes, memory, and machine instructions will give the explanations something to build on.

If those are unfamiliar, start with Systems Foundations. I’d rather you spend time working out why an example behaves strangely than be stuck trying to compile it. If you’ve already worked through that course, you have the starting point these lessons assume.

We’ll use provided or explicitly authorized targets, with boundaries on what we can do to them. Before starting the practical work, read Don’t Get Arrested, where we’ll go through what that permission needs to cover.

Practice: investigate a wrong preview

Imagine an image previewer shows the wrong picture after you open two files in sequence. You have permission to investigate it, but you don’t yet know why it happens.

Consider:

One question you’d ask as a researcher.
One question you’d ask as a pentester assessing the program’s use in an organization.
A research artifact you could produce to help someone else investigate the problem.
One security claim you can’t make from the wrong preview alone.

Compare your thinking

I’d start the research question with the order of the files: does opening the second one on its own show the right preview? If so, something left over from opening the first file is worth investigating. For a pentest, I’d want to know whether the application is used in a way that could show one user’s image to another user.

The two test files and the exact opening sequence would give someone else a way to see the problem. Once you’ve worked out which state is being kept, include that explanation too. We haven’t yet shown that one user can see another user’s image, though. The preview might just be showing you an earlier picture of your own, which is annoying but doesn’t demonstrate that disclosure.

If your answer was “use a debugger,” add what you’d look for once it stops. And if you wrote that the preview leaks private images, go back to the example: whose image did we actually see? That is the part we’d still need to test.

If you found yourself wanting to know why the preview was wrong before you’d even finished the example, there’s plenty here for you to enjoy. You already have a question worth investigating. Let’s learn how to answer it.