🛠️ OPSE Part 2: Developing an OSINT framework in Python

Introduction
In this article, we’ll explain OPSE, a tool that we developed during our school project to automate the OSINT process.
This article is part of a three-part series on the OPSE project. In this series of articles we will present different part of the project:
- Presentation of the tool w/ installation, usage and an example;
- A technical presentation of the tool, how it is thinked and coded;
- A guide to develop your own OPSE plugin !
Context
During our fourth year in engeneering school, we led a project on the OSINT theme. We asked ourselves how we could obtain as much personnal informations as possible on someone with few data at the beginning.
We started to imagine an automatic tool that, with few input data such as firstname and lastname, would be able to find many connections of this person on the internet. But we wanted it to be an help for anyone else in the future, so they can reuse it and modify it.
Our objectives were simple:
- automatic;
- fast;
- modular;
- open-source.
This project is now open-source and available on Github: https://github.com/OPSE-Developers/OPSE-Framework
You want to contribute on the OPSE project ? Join the Discord server by clicking on the logo !
⚠️ Disclaimer
OPSE project is made for educational & awareness use only !
Technical development
Core
The core is the most central part of the project. It links all the other parts, from the interface used to the different plugins called.
We have made it as modular as possible so that anyone can take it as a basis for their own project.
To do so, the core follows some principles from OOP (Object-Oriented Programming):
- abstraction;
- inheritance;
- polymorphism;
- etc.
Handling configuration
We needed to handle configurations. Not only from the core but also from the plugins, like if you want to limit a function or take a input or output path.
We started to develop our own configuration handler, that would manage YAML files and dictionnary like configurations.
Configurations handling are done by the Config
class at utils/config/Config.py
.
Parallelism with threads and asynchronous functions
We managed to parallelize the tasks we were going to launch. Indeed, when the script is launched, it performs several tens or hundred of requests on the internet, which can sometimes take a long time.
In order to reduce this execution time as much as possible, we decided to call each tool in a process in order to parallelize them all. Thus, it is not necessary to wait for the end of one of the tools to obtain the result of another.
To do so, let me introduce you the class Task
. The class is an overlay of Thread
and therefore inherits from it. The use of this class allows to normalize
the use of Thread in the program. Classes Tool
& Api
inherit from Task
.
Automatic requirements download
Because of the modularity with all the plugins and to improve the user experience, we added a piece of code executed at each launch to download all the requirements for the core but also for the plugins.
To realise it, we read the requirements.txt
files from the Framework and from each plugins to check if every dependency is installed.
|
|
Dependency version conflicts are a major issue between tools as we can’t do anything about their develpment. For now, this issue is bypassed by ignoring the tool causing the problem (the second one).
Among the list of missing packages, we also add pip
if it is missing. We can then download all the missing parts and start using OPSE.
|
|
Plugins
Plugins implementation
Thanks to the core structure, it is easy to implement new tools as components. The abstract class Tool
allows all tools to be consistent with each other, while adding custom methods and attributes.
Each plugins is called by the main script (the core). Then, each enabled plugins are launched. But first, we do a check that verify that required data for a plugin are present before launching the tool.
Once the checks are done, the script performs it’s task and returns a boolean indicating if the operation was successful.
Plugins structure
Plugins have been totally separated from the core, which allows to improve the modularity of the framework.
All plugins are considered as projects in their own right, as modules. The core has been improved to manage this modularity, in particular the import of tools which is done automatically when they are detected. It is therefore possible to add as many tools as you want.
The core is also able to dynamically install the dependencies of each plugin as explained in the Automatic requirements download section.
Automatic imports
All the plugins are imported automatically.
To do that, we use the builtin libraries pkgutil
and importlib
to check if a specific directory contains packages.
|
|
In our case, we check if tools/
directory contains subpackages with subclasses of Tool
.
|
|
Then, for each object in our directory, if this object is also a package, we can try to import each module it contains. As we want to create instances of those modules, we check if there is actually a correct subclass of our motherclass and then store it.
|
|
Finally, we can return the dictionary of successfully loaded modules and use it easily.
|
|
Feedback
Looking back on mistakes made
One of the mistakes we made during the development of the project was a lack of unit testing. While we were able to deliver the first PoC on time, we encountered a number of bugs and issues that could have been avoided if we had implemented more comprehensive testing earlier in the process.
Another mistake is that we did not dedicate enough time to project planning and did not clearly define our goals and requirements at the beginning. As a result. This led to delays development process and therefore a lack of time to develop more plugins. That could have been avoided with better planning.
Future of this project
The second version of OPSE have been pushed recently ! We have done several improvements. OPSE architecture has been reviewed:
- Review of imports;
- Management of requirements:
- Add a check on each requirements of OPSE, including loaded plugins;
- Show the user any missing requirements (if any) and prompt them to install the missing ones.
- Profile enrichment function has been reviewed and now works;
- Data visibility feature has been removed.
After the release of OPSE v2.0.0 we will be focused on the development of plugins. For the moment, we have not developed a large number of plugins. But now, as the core is fully functional and modular it is “easy” for us to integrate new plugins.
We are also open to any contributions. We create a Discord server for people that want to contribute or give some feedback on the tool. You can also submit your own plugins on the OPSE discord server.
You can join the OPSE Discord server by clicking on the logo !
📖 In the next article…
In this article, we describe how OPSE has been designed. This article was the technical part of the series and you now know how OPSE works.
In the next article, we’ll explain how developer can create their own plugins to be added to the OPSE core.