ICO wiki - User contributions [en]

ICS0018 Hands-on seminars

2023-03-14T20:42:38Z

Mgoroz:

=== The idea ===

The hands-on seminars are based on ScamLab materials. The goal is to learn about different scams in a safe(ish) but real environment. A little side quest is to waste scammers' time so that they can't use it on actual victims.
A successful presentation will result in passing the course if the attendance criteria (6 out of 8 lectures and seminars) is met. To register a presentation, '''please send an e-mail to Kristjan''', stating the chosen time slot. '''There are limited presentation slots - first come, first served!'''

=== The Task ===

Step 1: Create a fake identity and honeypot email account for engaging with scammers. Other platforms are also welcome, as long as you are able to protect your identity.

Step 2: Distribute the email address on shady or spammy sites, such as social media, online forums, etc. Some tips can be found here https://www.quora.com/How-can-I-get-scam-emails

Step 3: Wait for the scams to start rolling in.

Step 4: Engage! First select if you're going to use a naïve or aggressive approach.

If you don't manage to get any scammers to directly email your newly created address, go look in your regular mailbox, in the spam folder, ask friends & family, etc. NB! Before replying to any of those "crowdsourced" scam emails from your fake account, be sure to delete the address it was originally sent to.

Some tips for safety:
# Never reveal your (or anyone else's) real personal information to the scammers. Make up something realistic.
# Never open any links in emails unless you're in a protected sandbox environment.
# NEVER give out any real financial information, account information, or passwords.
# Always use Multi-Factor Authentication (MFA). Even on your fake accounts.

Here's the grand prize: if you manage to engage with at least 3 scammers for an email chain of 5 messages or more (they respond to at least 2 of your letters in the same thread), and present your findings at one of the seminars, '''you pass the course'''. It's not as easy as it might first seem.

An alternative way to pass is to educate people in your social network, friends, family, coworkers, etc about scams, how to spot them, how to avoid them, what to do if you're already a victim.

=== The Seminars ===

# March 1:
#* Homework discussion: fake identities, findings in honeypots
#* Petr Jelinek ✅
#* Phasha Davrishev ✅
#* Anton Višnevski, Denis Shadrin ✅
#* Vladyslava Shekula ✅
#* Ilya Nikolaev ✅
# March 8:
#* Scambaiting: aggressive approach
#* Helena Veebel ✅
#* Aleksandr Voronkov ✅
#* Maria Logberg ✅
#* Farid Azizov ✅
#* Rauf Gozal ✅
# March 15:
#* Scambaiting: naïve approach
#* Karmo Kütt
#* Alejandro Ballesteros Perez
#* Sanan Mammadli
#* Filip Tomeš
# March 21 8:15:
#* Allen-Kristjan Päll
#* Lorenzo Cavallini
#* Rashad Baghiyev
#* Edvin Toome
#* Maksim Gorozhanko
#* …
# March 21 10:00:
#* Dmitri Trubetskoi
#* Risto Remmel
#* Johannes Kodumäe
#* Semen Diev
#* Bendeguz Koszticsak
# March 22:
#* Scam prevention, how to educate others
#* Nicoleta Petrea
#* Hannes Kraavi* (scambaiting, naïve approach)
#* Talha Gesen
#* Georgi Tarassov
#* Daniil Lemberg

[[Social Engineering | Back to the course page]]

ICS0018 Hands-on seminars

2023-03-03T11:06:18Z

Mgoroz:

=== The idea ===

The hands-on seminars are based on ScamLab materials. The goal is to learn about different scams in a safe(ish) but real environment. A little side quest is to waste scammers' time so that they can't use it on actual victims.
A successful presentation will result in passing the course if the attendance criteria (6 out of 8 lectures and seminars) is met. To register a presentation, please send an e-mail to Kristjan, stating the chosen time slot. '''There are limited presentation slots - first come, first served!'''

=== The Task ===

Step 1: Create a fake identity and honeypot email account for engaging with scammers. Other platforms are also welcome, as long as you are able to protect your identity.

Step 2: Distribute the email address on shady or spammy sites, such as social media, online forums, etc. Some tips can be found here https://www.quora.com/How-can-I-get-scam-emails

Step 3: Wait for the scams to start rolling in.

Step 4: Engage! First select if you're going to use a naïve or aggressive approach.

If you don't manage to get any scammers to directly email your newly created address, go look in your regular mailbox, in the spam folder, ask friends & family, etc. NB! Before replying to any of those "crowdsourced" scam emails from your fake account, be sure to delete the address it was originally sent to.

Some tips for safety:
# Never reveal your (or anyone else's) real personal information to the scammers. Make up something realistic.
# Never open any links in emails unless you're in a protected sandbox environment.
# NEVER give out any real financial information, account information, or passwords.
# Always use Multi-Factor Authentication (MFA). Even on your fake accounts.

Here's the grand prize: if you manage to engage with at least 3 scammers for an email chain of 5 messages or more (they respond to at least 2 of your letters in the same thread), and present your findings at one of the seminars, '''you pass the course'''. It's not as easy as it might first seem.

An alternative way to pass is to educate people in your social network, friends, family, coworkers, etc about scams, how to spot them, how to avoid them, what to do if you're already a victim.

=== The Seminars ===

# March 1:
#* Homework discussion: fake identities, findings in honeypots
#* Petr Jelinek ✅
#* Phasha Davrishev ✅
#* Anton Višnevski, Denis Shadrin ✅
#* Vladyslava Shekula ✅
#* Ilya Nikolaev ✅
# March 8:
#* Scambaiting: aggressive approach
#* Helena Veebel
#* Aleksandr Voronkov
#* Maria Logberg
#* Farid Azizov
#* Rauf Gozal
#* ...
# March 15:
#* Scambaiting: naïve approach
#* Karmo Kütt
#* Alejandro Ballesteros Perez
#* Sanan Mammadli
#* Filip Tomeš
#* Maksim Gorozhanko
#* ...
# March 22:
#* Scam prevention, how to educate others
#* Nicoleta Petrea
#* Hannes Kraavi* (scambaiting, naïve approach)
#* Talha Gesen
#* ...

[[Social Engineering | Back to the course page]]

OSINT – theory and practice

2022-04-24T13:41:36Z

Mgoroz: /* Commercial subscriptions= */

= Framework =
[[File:Osint_example.png|thumb|Example of OSINT analysis]]
The framework for Open source intelligence is both sources for the searched data and ways to obtain and analyze it. The whole framework depends on the goal and capacities of the research in which the OSINT method is utilized. This means that two OSINT projects with different goals most likely would have completely different frameworks. This can even happen for researches with the same goals. For example, this year an emergence of OSINT techniques in tracking of the latest developments in Ukraine war can be observed.

While having the same general goal — looking as deep as possible into the fog of war — different researchers have their own subgoals, i.e. tracking weaponry losses like Oryx project or tracking movements of armies like Conflict Intelligence Team. In addition, the researchers use wide variety of methods from analyzing of social media publications, photos and videos, to using plane- and ship-tracking services and even traffic functions of Google Maps to track movement of the armies.
== Goal of research ==
In many cases OSINT research starts with a certain goal and this goal shapes the whole framework: which data needs to be acquired, where it is searched and how it is analyzed. However, there are cases when the framework is defined by the data. This can happen after different leaks of documents, personal information or any other data. Examples for this can be the whole WikiLeaks project, where investigators worked with leaked secret documents, or investigations that followed the leak of Yandex’s food delivery service clients, which among other things allowed to uncover properties owned by Putin’s close circle.
== Sources of information ==
As mentioned above, OSINT can work with any data that is open to the public. Generally the sources of information could be divided in a few categories:
*Internet
**Social media
**Blogs and forums
**Maps and tracking services
**Web analysis services like Google Analytics
**Other online publications
*Media
**Magazines and papers
**TV
**Radio
**Online outlets
*Government data
**Official declarations
**Land registries
**Government contracts
**Other documents
**Speeches of officials
*Academic publications
*Commercial data
**Databases
**Other services that can provide necessary data (i.e. satellite image sources, company information, etc)
All of those sources can be interlinked — as generally nowadays most of the government information, media, academic publications, etc are in the internet.
==Tools to collect data==
[[File:Tweetdeck.png|thumb|Example of TweetDeck request. Source: bellingcat.com]]
===Social media===
Tools to collect necessary data depend of the type of the data. Generally, the main sources for open source intelligence are social media and, most importantly, Twitter. The reason for this is the news and current events orientation of this website and powerful advanced search capabilities. Using TweetDeck researcher can formulate a search request for what they seek and get real time updates. There are also wide capabilities of using Twitter API to parse its data and structure it. There are also possibilities of using API of other social media but it is much more limited.
===Search engines===
OSINT also heavily utilizes search engines so it’s a good idea to learn advanced search tools. In addition, it might be useful to use more than one search engine as some of the information can be withdrawn from the results due to legal reasons or terms of service.
===Traditional media===
To fully utilize possibilities of traditional media for your research it would be useful to have subscriptions to the biggest agencies or outlets. As these subscriptions can be really expensive, especially when one might need all of them, it’s also good idea to learn how to surpass paywall — in most cases it can be done easily with the incognito mode of the browser or some kind of webarchive service. This should eliminate most of the costs and leave possibilities to subscribe to media with hard paywall like Der Spiegel
===Government information===
For the official government information in a lot of cases it is possible to subscribe to an RSS or email updates about new documents and press-releases. If this is not possible, one might write a script that parses a page that he is interested in and notifies about any updates of its content. Utilizing of some services like government contracts registers might need extensive training to analyze. Other services like land registries in many countries require payment for its information, so it might be not the best starting point for collecting data.
===Commercial subscriptions===
Researcher might also need subscriptions to commercial services that are needed for the analysis. Examples of such services might include Flightradar24, Similarweb, Himera Search and others. In addition, there are services, for example Telegram bots, that search through the know data breaches for a certain entry.

===VPN===
A virtual private network (VPN) should be used by the researcher both due to reasons of security and access to information, moreover, a possibility to choose servers in different countries might be useful. Different countries have different information laws and different services can restrict access for foreign users so not all needed data might be possible to acquire from researcher’s location.

==Tools to analyze data==
The analysis of the acquired data in most cases would be the most challenging part of the OSINT. In case when the information is mostly text it is easier as text is much easier to analyze using parsing, programming of just word search.

When dealing with photos or videos it would be most likely needed to analyze them personally — or double check after the used algorithm of analysis if there is one for the task.

A useful tool of analysis is visualization, especially when it comes to location-based research and big databases of structured information.
The exact methodology of analysis as well as of the data collection should be determined by the researcher at the start of the work.

= Data organization =
While an OSINT enthusiast may be adept at data collection, he or she will never develop the necessary data organization skills and tools to become a true professional.
There are numerous methods for storing data, including basic text files or notes.
However, using text files is impractical, as when there is a large amount of data, it becomes unmanageable.
Features desirable for OSINT data management include the ability to export and backup, as well as visualize data.
== Examples of software for OSINT data organization and their disadvantages: ==
* Simple Notes Apps (unmanageable when dealing with a large amount of data)
* Evernote (useful when paid for)
* Notion (notes cannot be accessed offline)
* Joplin (inconvenient organization for large projects)
* Obsidian.md Obsidian.md (a bit tricky to master)
== Obsidian.md ==
Obsidian.md, being perplexive in comparison to simple notes application, contains all the desirable features. It is a cross-platform, free application for organizing notes stored in markup (.md) files.
Notes and files are stored on a user's computer, and there is also a premium feature for syncing, which is superfluous given that backups using any online storage service, Syncthing software, or Git.
Given that OSINT specialists often work in teams, it is recommended to store the data in a Git repository in order to retain a history of modifications and increase collaboration capability.
== Vaults ==
Obsidian.md contains all data in what are referred to as "Vaults."
A vault is a project that houses all of it's associated notes and information.
== Plugins ==
Obsidian.md supports the installation of community plugins that extend the app's initial functionality.
=== Recommended plugins ===
# Dataview – Allows us to treat a vault as a database, querying and visualizing information from notes and files.
# BreadCrumbs – Adds link types and notes hierarchy.
# Juggl – Create mindmaps based on your notes and customize their looks with CSS and internal styling features.
=== Plugin installation ===
# Open Settings – the button is in the bottom-left corner of the application.
# Choose 'Community Plugins' from the 'Options' clause.
# Switch 'Safe Mode' to OFF and confirm it.
# Click 'Browse Community Plugins'.
# Find the plugin.
# Click 'Install'.
# Go back to 'Community Plugins' submenu.
# In the bottom section turn on the newly installed plugin.
== Folding vs Tagging and Linking ==
Simple folder structure is sufficient, when it comes to organizing data in nonoverlapping groups. It is enough to have just a couple of folders in your photogallery, for example. But in OSINT it is important to have a more sophisticated structure.
=== Tagging ===
Tagging adds structure because a piece of data can have several tags, as opposed to folders, which can only have one organizing unit per file.
==== Tag structure example: ====
# #people #processes #technology (part targeted)
# #primary #supportive #irrelevant (importance)
# #finished #unfinished (state of note/file)
# #web #registry #socialengineering (means of getting the information)

=== Linking ===
Linking enables the creation of relationships between notes and files.
This manner, one note can include connections to other notes and files, making it easier to handle.
For example, if John purchased the domain name legit.com, John's note can be linked to legit.com's note, which contains information about the domain.

==== Link types ====
Using link types opens up even more possibilities. Link types are included in Breadcrumbs Plugin for Obsidian.md.
In the aforementioned situation of John and legit.com, John is the domain's owner, thus, the domain is John's asset. These are called types of relations.
If it is later revealed that John purchased another domain name - fake.com – the new domain can be connected back to John.
This structure will be displayed in the notes by creating two relations of John's ownership:
# John – owner of legit.com, fake.com
# legit.com – asset of Johh, relative of fake.com
# fake.com – asset of John, relative of legit.com

== Dataview plugin ==
Dataview is, first and foremost, a data index, so it supports relatively rich methods of adding metadata to your knowledge base.
Dataview tracks information at the markdown page and markdown task levels, with each page/task able to contain an arbitrary number of complex (numbers, objects, lists) fields.
Each field is a named value of a specific type (like "number" or "text").

=== Example of notes with arbitrary metadata and a tag: ===

jason_statham.md
<pre>
---
name: Jason Statham
salary: 7500
department: Cyber Forensics
notes: [
"Potential phishing target",
"Mother has stage T4 cancer"
]
---
#employee
</pre>
bruce_lee.md
<pre>
---
name: Bruce Lee
salary: 8000
department: Developer Operations
notes: []
---
</pre>

== Querying dataview data ==
=== Options for querying data: ===
# Dataview query language
# Dataview Javascript API

Both can be used to, as an example, render a table from jason_statham.md and bruce_lee.md with four columns:
# File – contains a link to the file
# Name – metadata 'name'
# Salary – metadata 'salary'
# Department – metadata 'department'
It can also be sorted by 'salary'.

==== Dataview query language ====

The dataview query language is a straightforward, organized custom query language that enables you to quickly create views from data.
It enables the following:
* Retrieve pages related with tags, folders, and links, among other things.
* Simple actions on fields, such as comparison, existence checks, and so on, can be used to filter notes/data.
* Sorting results according to their fields.

The query language is capable of generating the view kinds, which are detailed below:

* TABLE: The standard view type; one row for each data point, with multiple columns of field data.
* LIST: A list of pages that correspond to the query. Each page can have a single linked value.
* TASK: A collection of tasks whose pages correspond to the specified query.
To query data with Dataview Query Language the 'dataview' language specification for a codeblock is used.

===== Example result of a data query =====
[[File:Dont_know_how_to_embed_images_yet]]
The queries leading to this result are listed below.
===== The general format of queries: =====
<pre>
```dataview
TABLE|LIST|TASK <field> [AS "Column Name"], <field>, ..., <field>
FROM <source> (like #tag or "folder")
WHERE <expression> (like 'field = value')
SORT <expression> [ASC/DESC] (like 'field ASC')
```
</pre>
==== Example with jason_statham.md and bruce_lee.md ====
<pre>
```dataview
TABLE name as "Name", salary as "Salary", department as "Department"
FROM #employee
SORT salary ASC
```
</pre>
=== Dataview Javascript API ===
The Dataview JavaScript API allows arbitrary JavaScript to be executed with access to the dataview indices and query engine, which is useful for complex views or interoperability with other plugins.
To query data with Dataview Javascript API the 'dataviewjs' language specification for a codeblock is used.
The API is accessible via the implicitly provided dv (or dataview) variable, which allows you to query for data, render HTML, and configure the view.
==== Example with jason_statham.md and bruce_lee.md ====
<pre>
```dataviewjs
let employees = dv.pages("#employee")
.sort(emp => emp.salary, "asc")
.map(emp => [emp.file.link, emp.name, emp.salary, emp.department])
dv.table(["File", "Name", "Salary", "Department"], employees)
```
</pre>

== Conclusion ==
There is no defined standard for OSINT data organization, because the data may come in different forms, including, but not limited to, web-pages, paper documents, online calendars, video and audio recordings. Due to this, it is nearly impossible to create a convenient tool for all use cases. If the operation is big enough, it might be feasible to create a devoted web application that stores all necessary data in a database. However, since OSINT itself it usually a highly confidential activity, publishing the application in Clear Web is a privacy and a security risk.
= References =
*Pastor-Galindo, Javier, et al. "The not yet exploited goldmine of OSINT: Opportunities, open challenges and future trends." IEEE Access 8 (2020): 10282-10304.
*Richelson, Jeffrey T. The US intelligence community. Routledge, 2018.
*Williams, Heather J., and Ilana Blum. Defining second generation open source intelligence (OSINT) for the defense enterprise. Rand Corporation, 2018.
APA
*[https://www.bellingcat.com/resources/2021/11/09/first-steps-to-getting-started-in-open-source-research/] First Steps to Getting Started in Open Source Research / Bellingcat
*[https://www.bellingcat.com/resources/how-tos/2019/06/21/the-most-comprehensive-tweetdeck-research-guide-in-existence-probably/] The Most Comprehensive TweetDeck Research Guide In Existence / Bellingcat

OSINT – theory and practice

2022-04-24T13:41:13Z

Mgoroz: /* Tools to collect data */

= Framework =
[[File:Osint_example.png|thumb|Example of OSINT analysis]]
The framework for Open source intelligence is both sources for the searched data and ways to obtain and analyze it. The whole framework depends on the goal and capacities of the research in which the OSINT method is utilized. This means that two OSINT projects with different goals most likely would have completely different frameworks. This can even happen for researches with the same goals. For example, this year an emergence of OSINT techniques in tracking of the latest developments in Ukraine war can be observed.

While having the same general goal — looking as deep as possible into the fog of war — different researchers have their own subgoals, i.e. tracking weaponry losses like Oryx project or tracking movements of armies like Conflict Intelligence Team. In addition, the researchers use wide variety of methods from analyzing of social media publications, photos and videos, to using plane- and ship-tracking services and even traffic functions of Google Maps to track movement of the armies.
== Goal of research ==
In many cases OSINT research starts with a certain goal and this goal shapes the whole framework: which data needs to be acquired, where it is searched and how it is analyzed. However, there are cases when the framework is defined by the data. This can happen after different leaks of documents, personal information or any other data. Examples for this can be the whole WikiLeaks project, where investigators worked with leaked secret documents, or investigations that followed the leak of Yandex’s food delivery service clients, which among other things allowed to uncover properties owned by Putin’s close circle.
== Sources of information ==
As mentioned above, OSINT can work with any data that is open to the public. Generally the sources of information could be divided in a few categories:
*Internet
**Social media
**Blogs and forums
**Maps and tracking services
**Web analysis services like Google Analytics
**Other online publications
*Media
**Magazines and papers
**TV
**Radio
**Online outlets
*Government data
**Official declarations
**Land registries
**Government contracts
**Other documents
**Speeches of officials
*Academic publications
*Commercial data
**Databases
**Other services that can provide necessary data (i.e. satellite image sources, company information, etc)
All of those sources can be interlinked — as generally nowadays most of the government information, media, academic publications, etc are in the internet.
==Tools to collect data==
[[File:Tweetdeck.png|thumb|Example of TweetDeck request. Source: bellingcat.com]]
===Social media===
Tools to collect necessary data depend of the type of the data. Generally, the main sources for open source intelligence are social media and, most importantly, Twitter. The reason for this is the news and current events orientation of this website and powerful advanced search capabilities. Using TweetDeck researcher can formulate a search request for what they seek and get real time updates. There are also wide capabilities of using Twitter API to parse its data and structure it. There are also possibilities of using API of other social media but it is much more limited.
===Search engines===
OSINT also heavily utilizes search engines so it’s a good idea to learn advanced search tools. In addition, it might be useful to use more than one search engine as some of the information can be withdrawn from the results due to legal reasons or terms of service.
===Traditional media===
To fully utilize possibilities of traditional media for your research it would be useful to have subscriptions to the biggest agencies or outlets. As these subscriptions can be really expensive, especially when one might need all of them, it’s also good idea to learn how to surpass paywall — in most cases it can be done easily with the incognito mode of the browser or some kind of webarchive service. This should eliminate most of the costs and leave possibilities to subscribe to media with hard paywall like Der Spiegel
===Government information===
For the official government information in a lot of cases it is possible to subscribe to an RSS or email updates about new documents and press-releases. If this is not possible, one might write a script that parses a page that he is interested in and notifies about any updates of its content. Utilizing of some services like government contracts registers might need extensive training to analyze. Other services like land registries in many countries require payment for its information, so it might be not the best starting point for collecting data.
===Commercial subscriptions====
Researcher might also need subscriptions to commercial services that are needed for the analysis. Examples of such services might include Flightradar24, Similarweb, Himera Search and others. In addition, there are services, for example Telegram bots, that search through the know data breaches for a certain entry.
===VPN===
A virtual private network (VPN) should be used by the researcher both due to reasons of security and access to information, moreover, a possibility to choose servers in different countries might be useful. Different countries have different information laws and different services can restrict access for foreign users so not all needed data might be possible to acquire from researcher’s location.

==Tools to analyze data==
The analysis of the acquired data in most cases would be the most challenging part of the OSINT. In case when the information is mostly text it is easier as text is much easier to analyze using parsing, programming of just word search.

When dealing with photos or videos it would be most likely needed to analyze them personally — or double check after the used algorithm of analysis if there is one for the task.

A useful tool of analysis is visualization, especially when it comes to location-based research and big databases of structured information.
The exact methodology of analysis as well as of the data collection should be determined by the researcher at the start of the work.

= Data organization =
While an OSINT enthusiast may be adept at data collection, he or she will never develop the necessary data organization skills and tools to become a true professional.
There are numerous methods for storing data, including basic text files or notes.
However, using text files is impractical, as when there is a large amount of data, it becomes unmanageable.
Features desirable for OSINT data management include the ability to export and backup, as well as visualize data.
== Examples of software for OSINT data organization and their disadvantages: ==
* Simple Notes Apps (unmanageable when dealing with a large amount of data)
* Evernote (useful when paid for)
* Notion (notes cannot be accessed offline)
* Joplin (inconvenient organization for large projects)
* Obsidian.md Obsidian.md (a bit tricky to master)
== Obsidian.md ==
Obsidian.md, being perplexive in comparison to simple notes application, contains all the desirable features. It is a cross-platform, free application for organizing notes stored in markup (.md) files.
Notes and files are stored on a user's computer, and there is also a premium feature for syncing, which is superfluous given that backups using any online storage service, Syncthing software, or Git.
Given that OSINT specialists often work in teams, it is recommended to store the data in a Git repository in order to retain a history of modifications and increase collaboration capability.
== Vaults ==
Obsidian.md contains all data in what are referred to as "Vaults."
A vault is a project that houses all of it's associated notes and information.
== Plugins ==
Obsidian.md supports the installation of community plugins that extend the app's initial functionality.
=== Recommended plugins ===
# Dataview – Allows us to treat a vault as a database, querying and visualizing information from notes and files.
# BreadCrumbs – Adds link types and notes hierarchy.
# Juggl – Create mindmaps based on your notes and customize their looks with CSS and internal styling features.
=== Plugin installation ===
# Open Settings – the button is in the bottom-left corner of the application.
# Choose 'Community Plugins' from the 'Options' clause.
# Switch 'Safe Mode' to OFF and confirm it.
# Click 'Browse Community Plugins'.
# Find the plugin.
# Click 'Install'.
# Go back to 'Community Plugins' submenu.
# In the bottom section turn on the newly installed plugin.
== Folding vs Tagging and Linking ==
Simple folder structure is sufficient, when it comes to organizing data in nonoverlapping groups. It is enough to have just a couple of folders in your photogallery, for example. But in OSINT it is important to have a more sophisticated structure.
=== Tagging ===
Tagging adds structure because a piece of data can have several tags, as opposed to folders, which can only have one organizing unit per file.
==== Tag structure example: ====
# #people #processes #technology (part targeted)
# #primary #supportive #irrelevant (importance)
# #finished #unfinished (state of note/file)
# #web #registry #socialengineering (means of getting the information)

=== Linking ===
Linking enables the creation of relationships between notes and files.
This manner, one note can include connections to other notes and files, making it easier to handle.
For example, if John purchased the domain name legit.com, John's note can be linked to legit.com's note, which contains information about the domain.

==== Link types ====
Using link types opens up even more possibilities. Link types are included in Breadcrumbs Plugin for Obsidian.md.
In the aforementioned situation of John and legit.com, John is the domain's owner, thus, the domain is John's asset. These are called types of relations.
If it is later revealed that John purchased another domain name - fake.com – the new domain can be connected back to John.
This structure will be displayed in the notes by creating two relations of John's ownership:
# John – owner of legit.com, fake.com
# legit.com – asset of Johh, relative of fake.com
# fake.com – asset of John, relative of legit.com

== Dataview plugin ==
Dataview is, first and foremost, a data index, so it supports relatively rich methods of adding metadata to your knowledge base.
Dataview tracks information at the markdown page and markdown task levels, with each page/task able to contain an arbitrary number of complex (numbers, objects, lists) fields.
Each field is a named value of a specific type (like "number" or "text").

=== Example of notes with arbitrary metadata and a tag: ===

jason_statham.md
<pre>
---
name: Jason Statham
salary: 7500
department: Cyber Forensics
notes: [
"Potential phishing target",
"Mother has stage T4 cancer"
]
---
#employee
</pre>
bruce_lee.md
<pre>
---
name: Bruce Lee
salary: 8000
department: Developer Operations
notes: []
---
</pre>

== Querying dataview data ==
=== Options for querying data: ===
# Dataview query language
# Dataview Javascript API

Both can be used to, as an example, render a table from jason_statham.md and bruce_lee.md with four columns:
# File – contains a link to the file
# Name – metadata 'name'
# Salary – metadata 'salary'
# Department – metadata 'department'
It can also be sorted by 'salary'.

==== Dataview query language ====

The dataview query language is a straightforward, organized custom query language that enables you to quickly create views from data.
It enables the following:
* Retrieve pages related with tags, folders, and links, among other things.
* Simple actions on fields, such as comparison, existence checks, and so on, can be used to filter notes/data.
* Sorting results according to their fields.

The query language is capable of generating the view kinds, which are detailed below:

* TABLE: The standard view type; one row for each data point, with multiple columns of field data.
* LIST: A list of pages that correspond to the query. Each page can have a single linked value.
* TASK: A collection of tasks whose pages correspond to the specified query.
To query data with Dataview Query Language the 'dataview' language specification for a codeblock is used.

===== Example result of a data query =====
[[File:Dont_know_how_to_embed_images_yet]]
The queries leading to this result are listed below.
===== The general format of queries: =====
<pre>
```dataview
TABLE|LIST|TASK <field> [AS "Column Name"], <field>, ..., <field>
FROM <source> (like #tag or "folder")
WHERE <expression> (like 'field = value')
SORT <expression> [ASC/DESC] (like 'field ASC')
```
</pre>
==== Example with jason_statham.md and bruce_lee.md ====
<pre>
```dataview
TABLE name as "Name", salary as "Salary", department as "Department"
FROM #employee
SORT salary ASC
```
</pre>
=== Dataview Javascript API ===
The Dataview JavaScript API allows arbitrary JavaScript to be executed with access to the dataview indices and query engine, which is useful for complex views or interoperability with other plugins.
To query data with Dataview Javascript API the 'dataviewjs' language specification for a codeblock is used.
The API is accessible via the implicitly provided dv (or dataview) variable, which allows you to query for data, render HTML, and configure the view.
==== Example with jason_statham.md and bruce_lee.md ====
<pre>
```dataviewjs
let employees = dv.pages("#employee")
.sort(emp => emp.salary, "asc")
.map(emp => [emp.file.link, emp.name, emp.salary, emp.department])
dv.table(["File", "Name", "Salary", "Department"], employees)
```
</pre>

== Conclusion ==
There is no defined standard for OSINT data organization, because the data may come in different forms, including, but not limited to, web-pages, paper documents, online calendars, video and audio recordings. Due to this, it is nearly impossible to create a convenient tool for all use cases. If the operation is big enough, it might be feasible to create a devoted web application that stores all necessary data in a database. However, since OSINT itself it usually a highly confidential activity, publishing the application in Clear Web is a privacy and a security risk.
= References =
*Pastor-Galindo, Javier, et al. "The not yet exploited goldmine of OSINT: Opportunities, open challenges and future trends." IEEE Access 8 (2020): 10282-10304.
*Richelson, Jeffrey T. The US intelligence community. Routledge, 2018.
*Williams, Heather J., and Ilana Blum. Defining second generation open source intelligence (OSINT) for the defense enterprise. Rand Corporation, 2018.
APA
*[https://www.bellingcat.com/resources/2021/11/09/first-steps-to-getting-started-in-open-source-research/] First Steps to Getting Started in Open Source Research / Bellingcat
*[https://www.bellingcat.com/resources/how-tos/2019/06/21/the-most-comprehensive-tweetdeck-research-guide-in-existence-probably/] The Most Comprehensive TweetDeck Research Guide In Existence / Bellingcat

OSINT – theory and practice

2022-04-24T13:36:15Z

Mgoroz: /* References */

= Framework =
[[File:Osint_example.png|thumb|Example of OSINT analysis]]
The framework for Open source intelligence is both sources for the searched data and ways to obtain and analyze it. The whole framework depends on the goal and capacities of the research in which the OSINT method is utilized. This means that two OSINT projects with different goals most likely would have completely different frameworks. This can even happen for researches with the same goals. For example, this year an emergence of OSINT techniques in tracking of the latest developments in Ukraine war can be observed.

While having the same general goal — looking as deep as possible into the fog of war — different researchers have their own subgoals, i.e. tracking weaponry losses like Oryx project or tracking movements of armies like Conflict Intelligence Team. In addition, the researchers use wide variety of methods from analyzing of social media publications, photos and videos, to using plane- and ship-tracking services and even traffic functions of Google Maps to track movement of the armies.
== Goal of research ==
In many cases OSINT research starts with a certain goal and this goal shapes the whole framework: which data needs to be acquired, where it is searched and how it is analyzed. However, there are cases when the framework is defined by the data. This can happen after different leaks of documents, personal information or any other data. Examples for this can be the whole WikiLeaks project, where investigators worked with leaked secret documents, or investigations that followed the leak of Yandex’s food delivery service clients, which among other things allowed to uncover properties owned by Putin’s close circle.
== Sources of information ==
As mentioned above, OSINT can work with any data that is open to the public. Generally the sources of information could be divided in a few categories:
*Internet
**Social media
**Blogs and forums
**Maps and tracking services
**Web analysis services like Google Analytics
**Other online publications
*Media
**Magazines and papers
**TV
**Radio
**Online outlets
*Government data
**Official declarations
**Land registries
**Government contracts
**Other documents
**Speeches of officials
*Academic publications
*Commercial data
**Databases
**Other services that can provide necessary data (i.e. satellite image sources, company information, etc)
All of those sources can be interlinked — as generally nowadays most of the government information, media, academic publications, etc are in the internet.
==Tools to collect data==
[[File:Tweetdeck.png|thumb|Example of TweetDeck request. Source: bellingcat.com]]
Tools to collect necessary data depend of the type of the data. Generally, the main sources for open source intelligence are social media and, most importantly, Twitter. The reason for this is the news and current events orientation of this website and powerful advanced search capabilities. Using TweetDeck researcher can formulate a search request for what they seek and get real time updates. There are also wide capabilities of using Twitter API to parse its data and structure it. There are also possibilities of using API of other social media but it is much more limited.

OSINT also heavily utilizes search engines so it’s a good idea to learn advanced search tools. In addition, it might be useful to use more than one search engine as some of the information can be withdrawn from the results due to legal reasons or terms of service.

To fully utilize possibilities of traditional media for your research it would be useful to have subscriptions to the biggest agencies or outlets. As these subscriptions can be really expensive, especially when one might need all of them, it’s also good idea to learn how to surpass paywall — in most cases it can be done easily with the incognito mode of the browser or some kind of webarchive service.

For the official government information in a lot of cases it is possible to subscribe to an RSS or email updates about new documents and press-releases. If this is not possible, one might write a script that parses a page that he is interested in and notifies about any updates of its content. Utilizing of some services like government contracts registers might need extensive training to analyze. Other services like land registries in many countries require payment for its information, so it might be not the best starting point for collecting data.

Researcher might also need subscriptions to commercial services that are needed for the analysis. Examples of such services might include Flightradar24, Similarweb, Himera Search and others. In addition, there are services, for example Telegram bots, that search through the know data breaches for a certain entry.

A VPN should be used by the researcher both due to reasons of security and access to information, moreover, a possibility to choose servers in different countries might be useful. Different countries have different information laws and different services can restrict access for foreign users so not all needed data might be possible to acquire from researcher’s location.

==Tools to analyze data==
The analysis of the acquired data in most cases would be the most challenging part of the OSINT. In case when the information is mostly text it is easier as text is much easier to analyze using parsing, programming of just word search.

When dealing with photos or videos it would be most likely needed to analyze them personally — or double check after the used algorithm of analysis if there is one for the task.

A useful tool of analysis is visualization, especially when it comes to location-based research and big databases of structured information.
The exact methodology of analysis as well as of the data collection should be determined by the researcher at the start of the work.

= Data organization =
While an OSINT enthusiast may be adept at data collection, he or she will never develop the necessary data organization skills and tools to become a true professional.
There are numerous methods for storing data, including basic text files or notes.
However, using text files is impractical, as when there is a large amount of data, it becomes unmanageable.
Features desirable for OSINT data management include the ability to export and backup, as well as visualize data.
== Examples of software for OSINT data organization and their disadvantages: ==
* Simple Notes Apps (unmanageable when dealing with a large amount of data)
* Evernote (useful when paid for)
* Notion (notes cannot be accessed offline)
* Joplin (inconvenient organization for large projects)
* Obsidian.md Obsidian.md (a bit tricky to master)
== Obsidian.md ==
Obsidian.md, being perplexive in comparison to simple notes application, contains all the desirable features. It is a cross-platform, free application for organizing notes stored in markup (.md) files.
Notes and files are stored on a user's computer, and there is also a premium feature for syncing, which is superfluous given that backups using any online storage service, Syncthing software, or Git.
Given that OSINT specialists often work in teams, it is recommended to store the data in a Git repository in order to retain a history of modifications and increase collaboration capability.
== Vaults ==
Obsidian.md contains all data in what are referred to as "Vaults."
A vault is a project that houses all of it's associated notes and information.
== Plugins ==
Obsidian.md supports the installation of community plugins that extend the app's initial functionality.
=== Recommended plugins ===
# Dataview – Allows us to treat a vault as a database, querying and visualizing information from notes and files.
# BreadCrumbs – Adds link types and notes hierarchy.
# Juggl – Create mindmaps based on your notes and customize their looks with CSS and internal styling features.
=== Plugin installation ===
# Open Settings – the button is in the bottom-left corner of the application.
# Choose 'Community Plugins' from the 'Options' clause.
# Switch 'Safe Mode' to OFF and confirm it.
# Click 'Browse Community Plugins'.
# Find the plugin.
# Click 'Install'.
# Go back to 'Community Plugins' submenu.
# In the bottom section turn on the newly installed plugin.
== Folding vs Tagging and Linking ==
Simple folder structure is sufficient, when it comes to organizing data in nonoverlapping groups. It is enough to have just a couple of folders in your photogallery, for example. But in OSINT it is important to have a more sophisticated structure.
=== Tagging ===
Tagging adds structure because a piece of data can have several tags, as opposed to folders, which can only have one organizing unit per file.
==== Tag structure example: ====
# #people #processes #technology (part targeted)
# #primary #supportive #irrelevant (importance)
# #finished #unfinished (state of note/file)
# #web #registry #socialengineering (means of getting the information)

=== Linking ===
Linking enables the creation of relationships between notes and files.
This manner, one note can include connections to other notes and files, making it easier to handle.
For example, if John purchased the domain name legit.com, John's note can be linked to legit.com's note, which contains information about the domain.

==== Link types ====
Using link types opens up even more possibilities. Link types are included in Breadcrumbs Plugin for Obsidian.md.
In the aforementioned situation of John and legit.com, John is the domain's owner, thus, the domain is John's asset. These are called types of relations.
If it is later revealed that John purchased another domain name - fake.com – the new domain can be connected back to John.
This structure will be displayed in the notes by creating two relations of John's ownership:
# John – owner of legit.com, fake.com
# legit.com – asset of Johh, relative of fake.com
# fake.com – asset of John, relative of legit.com

== Dataview plugin ==
Dataview is, first and foremost, a data index, so it supports relatively rich methods of adding metadata to your knowledge base.
Dataview tracks information at the markdown page and markdown task levels, with each page/task able to contain an arbitrary number of complex (numbers, objects, lists) fields.
Each field is a named value of a specific type (like "number" or "text").

=== Example of notes with arbitrary metadata and a tag: ===

jason_statham.md
<pre>
---
name: Jason Statham
salary: 7500
department: Cyber Forensics
notes: [
"Potential phishing target",
"Mother has stage T4 cancer"
]
---
#employee
</pre>
bruce_lee.md
<pre>
---
name: Bruce Lee
salary: 8000
department: Developer Operations
notes: []
---
</pre>

== Querying dataview data ==
=== Options for querying data: ===
# Dataview query language
# Dataview Javascript API

Both can be used to, as an example, render a table from jason_statham.md and bruce_lee.md with four columns:
# File – contains a link to the file
# Name – metadata 'name'
# Salary – metadata 'salary'
# Department – metadata 'department'
It can also be sorted by 'salary'.

==== Dataview query language ====

The dataview query language is a straightforward, organized custom query language that enables you to quickly create views from data.
It enables the following:
* Retrieve pages related with tags, folders, and links, among other things.
* Simple actions on fields, such as comparison, existence checks, and so on, can be used to filter notes/data.
* Sorting results according to their fields.

The query language is capable of generating the view kinds, which are detailed below:

* TABLE: The standard view type; one row for each data point, with multiple columns of field data.
* LIST: A list of pages that correspond to the query. Each page can have a single linked value.
* TASK: A collection of tasks whose pages correspond to the specified query.
To query data with Dataview Query Language the 'dataview' language specification for a codeblock is used.

===== Example result of a data query =====
[[File:Dont_know_how_to_embed_images_yet]]
The queries leading to this result are listed below.
===== The general format of queries: =====
<pre>
```dataview
TABLE|LIST|TASK <field> [AS "Column Name"], <field>, ..., <field>
FROM <source> (like #tag or "folder")
WHERE <expression> (like 'field = value')
SORT <expression> [ASC/DESC] (like 'field ASC')
```
</pre>
==== Example with jason_statham.md and bruce_lee.md ====
<pre>
```dataview
TABLE name as "Name", salary as "Salary", department as "Department"
FROM #employee
SORT salary ASC
```
</pre>
=== Dataview Javascript API ===
The Dataview JavaScript API allows arbitrary JavaScript to be executed with access to the dataview indices and query engine, which is useful for complex views or interoperability with other plugins.
To query data with Dataview Javascript API the 'dataviewjs' language specification for a codeblock is used.
The API is accessible via the implicitly provided dv (or dataview) variable, which allows you to query for data, render HTML, and configure the view.
==== Example with jason_statham.md and bruce_lee.md ====
<pre>
```dataviewjs
let employees = dv.pages("#employee")
.sort(emp => emp.salary, "asc")
.map(emp => [emp.file.link, emp.name, emp.salary, emp.department])
dv.table(["File", "Name", "Salary", "Department"], employees)
```
</pre>

== Conclusion ==
There is no defined standard for OSINT data organization, because the data may come in different forms, including, but not limited to, web-pages, paper documents, online calendars, video and audio recordings. Due to this, it is nearly impossible to create a convenient tool for all use cases. If the operation is big enough, it might be feasible to create a devoted web application that stores all necessary data in a database. However, since OSINT itself it usually a highly confidential activity, publishing the application in Clear Web is a privacy and a security risk.
= References =
*Pastor-Galindo, Javier, et al. "The not yet exploited goldmine of OSINT: Opportunities, open challenges and future trends." IEEE Access 8 (2020): 10282-10304.
*Richelson, Jeffrey T. The US intelligence community. Routledge, 2018.
*Williams, Heather J., and Ilana Blum. Defining second generation open source intelligence (OSINT) for the defense enterprise. Rand Corporation, 2018.
APA
*[https://www.bellingcat.com/resources/2021/11/09/first-steps-to-getting-started-in-open-source-research/] First Steps to Getting Started in Open Source Research / Bellingcat
*[https://www.bellingcat.com/resources/how-tos/2019/06/21/the-most-comprehensive-tweetdeck-research-guide-in-existence-probably/] The Most Comprehensive TweetDeck Research Guide In Existence / Bellingcat

OSINT – theory and practice

2022-04-24T13:34:52Z

Mgoroz:

= Framework =
[[File:Osint_example.png|thumb|Example of OSINT analysis]]
The framework for Open source intelligence is both sources for the searched data and ways to obtain and analyze it. The whole framework depends on the goal and capacities of the research in which the OSINT method is utilized. This means that two OSINT projects with different goals most likely would have completely different frameworks. This can even happen for researches with the same goals. For example, this year an emergence of OSINT techniques in tracking of the latest developments in Ukraine war can be observed.

While having the same general goal — looking as deep as possible into the fog of war — different researchers have their own subgoals, i.e. tracking weaponry losses like Oryx project or tracking movements of armies like Conflict Intelligence Team. In addition, the researchers use wide variety of methods from analyzing of social media publications, photos and videos, to using plane- and ship-tracking services and even traffic functions of Google Maps to track movement of the armies.
== Goal of research ==
In many cases OSINT research starts with a certain goal and this goal shapes the whole framework: which data needs to be acquired, where it is searched and how it is analyzed. However, there are cases when the framework is defined by the data. This can happen after different leaks of documents, personal information or any other data. Examples for this can be the whole WikiLeaks project, where investigators worked with leaked secret documents, or investigations that followed the leak of Yandex’s food delivery service clients, which among other things allowed to uncover properties owned by Putin’s close circle.
== Sources of information ==
As mentioned above, OSINT can work with any data that is open to the public. Generally the sources of information could be divided in a few categories:
*Internet
**Social media
**Blogs and forums
**Maps and tracking services
**Web analysis services like Google Analytics
**Other online publications
*Media
**Magazines and papers
**TV
**Radio
**Online outlets
*Government data
**Official declarations
**Land registries
**Government contracts
**Other documents
**Speeches of officials
*Academic publications
*Commercial data
**Databases
**Other services that can provide necessary data (i.e. satellite image sources, company information, etc)
All of those sources can be interlinked — as generally nowadays most of the government information, media, academic publications, etc are in the internet.
==Tools to collect data==
[[File:Tweetdeck.png|thumb|Example of TweetDeck request. Source: bellingcat.com]]
Tools to collect necessary data depend of the type of the data. Generally, the main sources for open source intelligence are social media and, most importantly, Twitter. The reason for this is the news and current events orientation of this website and powerful advanced search capabilities. Using TweetDeck researcher can formulate a search request for what they seek and get real time updates. There are also wide capabilities of using Twitter API to parse its data and structure it. There are also possibilities of using API of other social media but it is much more limited.

OSINT also heavily utilizes search engines so it’s a good idea to learn advanced search tools. In addition, it might be useful to use more than one search engine as some of the information can be withdrawn from the results due to legal reasons or terms of service.

To fully utilize possibilities of traditional media for your research it would be useful to have subscriptions to the biggest agencies or outlets. As these subscriptions can be really expensive, especially when one might need all of them, it’s also good idea to learn how to surpass paywall — in most cases it can be done easily with the incognito mode of the browser or some kind of webarchive service.

For the official government information in a lot of cases it is possible to subscribe to an RSS or email updates about new documents and press-releases. If this is not possible, one might write a script that parses a page that he is interested in and notifies about any updates of its content. Utilizing of some services like government contracts registers might need extensive training to analyze. Other services like land registries in many countries require payment for its information, so it might be not the best starting point for collecting data.

Researcher might also need subscriptions to commercial services that are needed for the analysis. Examples of such services might include Flightradar24, Similarweb, Himera Search and others. In addition, there are services, for example Telegram bots, that search through the know data breaches for a certain entry.

A VPN should be used by the researcher both due to reasons of security and access to information, moreover, a possibility to choose servers in different countries might be useful. Different countries have different information laws and different services can restrict access for foreign users so not all needed data might be possible to acquire from researcher’s location.

==Tools to analyze data==
The analysis of the acquired data in most cases would be the most challenging part of the OSINT. In case when the information is mostly text it is easier as text is much easier to analyze using parsing, programming of just word search.

When dealing with photos or videos it would be most likely needed to analyze them personally — or double check after the used algorithm of analysis if there is one for the task.

A useful tool of analysis is visualization, especially when it comes to location-based research and big databases of structured information.
The exact methodology of analysis as well as of the data collection should be determined by the researcher at the start of the work.

= Data organization =
While an OSINT enthusiast may be adept at data collection, he or she will never develop the necessary data organization skills and tools to become a true professional.
There are numerous methods for storing data, including basic text files or notes.
However, using text files is impractical, as when there is a large amount of data, it becomes unmanageable.
Features desirable for OSINT data management include the ability to export and backup, as well as visualize data.
== Examples of software for OSINT data organization and their disadvantages: ==
* Simple Notes Apps (unmanageable when dealing with a large amount of data)
* Evernote (useful when paid for)
* Notion (notes cannot be accessed offline)
* Joplin (inconvenient organization for large projects)
* Obsidian.md Obsidian.md (a bit tricky to master)
== Obsidian.md ==
Obsidian.md, being perplexive in comparison to simple notes application, contains all the desirable features. It is a cross-platform, free application for organizing notes stored in markup (.md) files.
Notes and files are stored on a user's computer, and there is also a premium feature for syncing, which is superfluous given that backups using any online storage service, Syncthing software, or Git.
Given that OSINT specialists often work in teams, it is recommended to store the data in a Git repository in order to retain a history of modifications and increase collaboration capability.
== Vaults ==
Obsidian.md contains all data in what are referred to as "Vaults."
A vault is a project that houses all of it's associated notes and information.
== Plugins ==
Obsidian.md supports the installation of community plugins that extend the app's initial functionality.
=== Recommended plugins ===
# Dataview – Allows us to treat a vault as a database, querying and visualizing information from notes and files.
# BreadCrumbs – Adds link types and notes hierarchy.
# Juggl – Create mindmaps based on your notes and customize their looks with CSS and internal styling features.
=== Plugin installation ===
# Open Settings – the button is in the bottom-left corner of the application.
# Choose 'Community Plugins' from the 'Options' clause.
# Switch 'Safe Mode' to OFF and confirm it.
# Click 'Browse Community Plugins'.
# Find the plugin.
# Click 'Install'.
# Go back to 'Community Plugins' submenu.
# In the bottom section turn on the newly installed plugin.
== Folding vs Tagging and Linking ==
Simple folder structure is sufficient, when it comes to organizing data in nonoverlapping groups. It is enough to have just a couple of folders in your photogallery, for example. But in OSINT it is important to have a more sophisticated structure.
=== Tagging ===
Tagging adds structure because a piece of data can have several tags, as opposed to folders, which can only have one organizing unit per file.
==== Tag structure example: ====
# #people #processes #technology (part targeted)
# #primary #supportive #irrelevant (importance)
# #finished #unfinished (state of note/file)
# #web #registry #socialengineering (means of getting the information)

=== Linking ===
Linking enables the creation of relationships between notes and files.
This manner, one note can include connections to other notes and files, making it easier to handle.
For example, if John purchased the domain name legit.com, John's note can be linked to legit.com's note, which contains information about the domain.

==== Link types ====
Using link types opens up even more possibilities. Link types are included in Breadcrumbs Plugin for Obsidian.md.
In the aforementioned situation of John and legit.com, John is the domain's owner, thus, the domain is John's asset. These are called types of relations.
If it is later revealed that John purchased another domain name - fake.com – the new domain can be connected back to John.
This structure will be displayed in the notes by creating two relations of John's ownership:
# John – owner of legit.com, fake.com
# legit.com – asset of Johh, relative of fake.com
# fake.com – asset of John, relative of legit.com

== Dataview plugin ==
Dataview is, first and foremost, a data index, so it supports relatively rich methods of adding metadata to your knowledge base.
Dataview tracks information at the markdown page and markdown task levels, with each page/task able to contain an arbitrary number of complex (numbers, objects, lists) fields.
Each field is a named value of a specific type (like "number" or "text").

=== Example of notes with arbitrary metadata and a tag: ===

jason_statham.md
<pre>
---
name: Jason Statham
salary: 7500
department: Cyber Forensics
notes: [
"Potential phishing target",
"Mother has stage T4 cancer"
]
---
#employee
</pre>
bruce_lee.md
<pre>
---
name: Bruce Lee
salary: 8000
department: Developer Operations
notes: []
---
</pre>

== Querying dataview data ==
=== Options for querying data: ===
# Dataview query language
# Dataview Javascript API

Both can be used to, as an example, render a table from jason_statham.md and bruce_lee.md with four columns:
# File – contains a link to the file
# Name – metadata 'name'
# Salary – metadata 'salary'
# Department – metadata 'department'
It can also be sorted by 'salary'.

==== Dataview query language ====

The dataview query language is a straightforward, organized custom query language that enables you to quickly create views from data.
It enables the following:
* Retrieve pages related with tags, folders, and links, among other things.
* Simple actions on fields, such as comparison, existence checks, and so on, can be used to filter notes/data.
* Sorting results according to their fields.

The query language is capable of generating the view kinds, which are detailed below:

* TABLE: The standard view type; one row for each data point, with multiple columns of field data.
* LIST: A list of pages that correspond to the query. Each page can have a single linked value.
* TASK: A collection of tasks whose pages correspond to the specified query.
To query data with Dataview Query Language the 'dataview' language specification for a codeblock is used.

===== Example result of a data query =====
[[File:Dont_know_how_to_embed_images_yet]]
The queries leading to this result are listed below.
===== The general format of queries: =====
<pre>
```dataview
TABLE|LIST|TASK <field> [AS "Column Name"], <field>, ..., <field>
FROM <source> (like #tag or "folder")
WHERE <expression> (like 'field = value')
SORT <expression> [ASC/DESC] (like 'field ASC')
```
</pre>
==== Example with jason_statham.md and bruce_lee.md ====
<pre>
```dataview
TABLE name as "Name", salary as "Salary", department as "Department"
FROM #employee
SORT salary ASC
```
</pre>
=== Dataview Javascript API ===
The Dataview JavaScript API allows arbitrary JavaScript to be executed with access to the dataview indices and query engine, which is useful for complex views or interoperability with other plugins.
To query data with Dataview Javascript API the 'dataviewjs' language specification for a codeblock is used.
The API is accessible via the implicitly provided dv (or dataview) variable, which allows you to query for data, render HTML, and configure the view.
==== Example with jason_statham.md and bruce_lee.md ====
<pre>
```dataviewjs
let employees = dv.pages("#employee")
.sort(emp => emp.salary, "asc")
.map(emp => [emp.file.link, emp.name, emp.salary, emp.department])
dv.table(["File", "Name", "Salary", "Department"], employees)
```
</pre>

== Conclusion ==
There is no defined standard for OSINT data organization, because the data may come in different forms, including, but not limited to, web-pages, paper documents, online calendars, video and audio recordings. Due to this, it is nearly impossible to create a convenient tool for all use cases. If the operation is big enough, it might be feasible to create a devoted web application that stores all necessary data in a database. However, since OSINT itself it usually a highly confidential activity, publishing the application in Clear Web is a privacy and a security risk.
= References =
*Richelson, Jeffrey T. The US intelligence community. Routledge, 2018.
*Pastor-Galindo, Javier, et al. "The not yet exploited goldmine of OSINT: Opportunities, open challenges and future trends." IEEE Access 8 (2020): 10282-10304.
*[https://www.bellingcat.com/resources/2021/11/09/first-steps-to-getting-started-in-open-source-research/] First Steps to Getting Started in Open Source Research / Bellingcat
*[https://www.bellingcat.com/resources/how-tos/2019/06/21/the-most-comprehensive-tweetdeck-research-guide-in-existence-probably/] The Most Comprehensive TweetDeck Research Guide In Existence / Bellingcat

OSINT – theory and practice

2022-04-24T13:29:15Z

Mgoroz: /* Tools to collect data */

= Framework =
[[File:Osint_example.png|thumb|Example of OSINT analysis]]
The framework for Open source intelligence is both sources for the searched data and ways to obtain and analyze it. The whole framework depends on the goal and capacities of the research in which the OSINT method is utilized. This means that two OSINT projects with different goals most likely would have completely different frameworks. This can even happen for researches with the same goals. For example, this year an emergence of OSINT techniques in tracking of the latest developments in Ukraine war can be observed.

While having the same general goal — looking as deep as possible into the fog of war — different researchers have their own subgoals, i.e. tracking weaponry losses like Oryx project or tracking movements of armies like Conflict Intelligence Team. In addition, the researchers use wide variety of methods from analyzing of social media publications, photos and videos, to using plane- and ship-tracking services and even traffic functions of Google Maps to track movement of the armies.
== Goal of research ==
In many cases OSINT research starts with a certain goal and this goal shapes the whole framework: which data needs to be acquired, where it is searched and how it is analyzed. However, there are cases when the framework is defined by the data. This can happen after different leaks of documents, personal information or any other data. Examples for this can be the whole WikiLeaks project, where investigators worked with leaked secret documents, or investigations that followed the leak of Yandex’s food delivery service clients, which among other things allowed to uncover properties owned by Putin’s close circle.
== Sources of information ==
As mentioned above, OSINT can work with any data that is open to the public. Generally the sources of information could be divided in a few categories:
*Internet
**Social media
**Blogs and forums
**Maps and tracking services
**Web analysis services like Google Analytics
**Other online publications
*Media
**Magazines and papers
**TV
**Radio
**Online outlets
*Government data
**Official declarations
**Land registries
**Government contracts
**Other documents
**Speeches of officials
*Academic publications
*Commercial data
**Databases
**Other services that can provide necessary data (i.e. satellite image sources, company information, etc)
All of those sources can be interlinked — as generally nowadays most of the government information, media, academic publications, etc are in the internet.
==Tools to collect data==
[[File:Tweetdeck.png|thumb|Example of TweetDeck request. Source: bellingcat.com]]
Tools to collect necessary data depend of the type of the data. Generally, the main sources for open source intelligence are social media and, most importantly, Twitter. The reason for this is the news and current events orientation of this website and powerful advanced search capabilities. Using TweetDeck researcher can formulate a search request for what they seek and get real time updates. There are also wide capabilities of using Twitter API to parse its data and structure it. There are also possibilities of using API of other social media but it is much more limited.

OSINT also heavily utilizes search engines so it’s a good idea to learn advanced search tools. In addition, it might be useful to use more than one search engine as some of the information can be withdrawn from the results due to legal reasons or terms of service.

To fully utilize possibilities of traditional media for your research it would be useful to have subscriptions to the biggest agencies or outlets. As these subscriptions can be really expensive, especially when one might need all of them, it’s also good idea to learn how to surpass paywall — in most cases it can be done easily with the incognito mode of the browser or some kind of webarchive service.

For the official government information in a lot of cases it is possible to subscribe to an RSS or email updates about new documents and press-releases. If this is not possible, one might write a script that parses a page that he is interested in and notifies about any updates of its content. Utilizing of some services like government contracts registers might need extensive training to analyze. Other services like land registries in many countries require payment for its information, so it might be not the best starting point for collecting data.

Researcher might also need subscriptions to commercial services that are needed for the analysis. Examples of such services might include Flightradar24, Similarweb, Himera Search and others. In addition, there are services, for example Telegram bots, that search through the know data breaches for a certain entry.

A VPN should be used by the researcher both due to reasons of security and access to information, moreover, a possibility to choose servers in different countries might be useful. Different countries have different information laws and different services can restrict access for foreign users so not all needed data might be possible to acquire from researcher’s location.

==Tools to analyze data==
The analysis of the acquired data in most cases would be the most challenging part of the OSINT. In case when the information is mostly text it is easier as text is much easier to analyze using parsing, programming of just word search.

When dealing with photos or videos it would be most likely needed to analyze them personally — or double check after the used algorithm of analysis if there is one for the task.

A useful tool of analysis is visualization, especially when it comes to location-based research and big databases of structured information.
The exact methodology of analysis as well as of the data collection should be determined by the researcher at the start of the work.

= Data organization =
While an OSINT enthusiast may be adept at data collection, he or she will never develop the necessary data organization skills and tools to become a true professional.
There are numerous methods for storing data, including basic text files or notes.
However, using text files is impractical, as when there is a large amount of data, it becomes unmanageable.
Features desirable for OSINT data management include the ability to export and backup, as well as visualize data.
== Examples of software for OSINT data organization and their disadvantages: ==
* Simple Notes Apps (unmanageable when dealing with a large amount of data)
* Evernote (useful when paid for)
* Notion (notes cannot be accessed offline)
* Joplin (inconvenient organization for large projects)
* Obsidian.md Obsidian.md (a bit tricky to master)
== Obsidian.md ==
Obsidian.md, being perplexive in comparison to simple notes application, contains all the desirable features. It is a cross-platform, free application for organizing notes stored in markup (.md) files.
Notes and files are stored on a user's computer, and there is also a premium feature for syncing, which is superfluous given that backups using any online storage service, Syncthing software, or Git.
Given that OSINT specialists often work in teams, it is recommended to store the data in a Git repository in order to retain a history of modifications and increase collaboration capability.
== Vaults ==
Obsidian.md contains all data in what are referred to as "Vaults."
A vault is a project that houses all of it's associated notes and information.
== Plugins ==
Obsidian.md supports the installation of community plugins that extend the app's initial functionality.
=== Recommended plugins ===
# Dataview – Allows us to treat a vault as a database, querying and visualizing information from notes and files.
# BreadCrumbs – Adds link types and notes hierarchy.
# Juggl – Create mindmaps based on your notes and customize their looks with CSS and internal styling features.
=== Plugin installation ===
# Open Settings – the button is in the bottom-left corner of the application.
# Choose 'Community Plugins' from the 'Options' clause.
# Switch 'Safe Mode' to OFF and confirm it.
# Click 'Browse Community Plugins'.
# Find the plugin.
# Click 'Install'.
# Go back to 'Community Plugins' submenu.
# In the bottom section turn on the newly installed plugin.
== Folding vs Tagging and Linking ==
Simple folder structure is sufficient, when it comes to organizing data in nonoverlapping groups. It is enough to have just a couple of folders in your photogallery, for example. But in OSINT it is important to have a more sophisticated structure.
=== Tagging ===
Tagging adds structure because a piece of data can have several tags, as opposed to folders, which can only have one organizing unit per file.
==== Tag structure example: ====
# #people #processes #technology (part targeted)
# #primary #supportive #irrelevant (importance)
# #finished #unfinished (state of note/file)
# #web #registry #socialengineering (means of getting the information)

=== Linking ===
Linking enables the creation of relationships between notes and files.
This manner, one note can include connections to other notes and files, making it easier to handle.
For example, if John purchased the domain name legit.com, John's note can be linked to legit.com's note, which contains information about the domain.

==== Link types ====
Using link types opens up even more possibilities. Link types are included in Breadcrumbs Plugin for Obsidian.md.
In the aforementioned situation of John and legit.com, John is the domain's owner, thus, the domain is John's asset. These are called types of relations.
If it is later revealed that John purchased another domain name - fake.com – the new domain can be connected back to John.
This structure will be displayed in the notes by creating two relations of John's ownership:
# John – owner of legit.com, fake.com
# legit.com – asset of Johh, relative of fake.com
# fake.com – asset of John, relative of legit.com

== Dataview plugin ==
Dataview is, first and foremost, a data index, so it supports relatively rich methods of adding metadata to your knowledge base.
Dataview tracks information at the markdown page and markdown task levels, with each page/task able to contain an arbitrary number of complex (numbers, objects, lists) fields.
Each field is a named value of a specific type (like "number" or "text").

=== Example of notes with arbitrary metadata and a tag: ===

jason_statham.md
<pre>
---
name: Jason Statham
salary: 7500
department: Cyber Forensics
notes: [
"Potential phishing target",
"Mother has stage T4 cancer"
]
---
#employee
</pre>
bruce_lee.md
<pre>
---
name: Bruce Lee
salary: 8000
department: Developer Operations
notes: []
---
</pre>

== Querying dataview data ==
=== Options for querying data: ===
# Dataview query language
# Dataview Javascript API

Both can be used to, as an example, render a table from jason_statham.md and bruce_lee.md with four columns:
# File – contains a link to the file
# Name – metadata 'name'
# Salary – metadata 'salary'
# Department – metadata 'department'
It can also be sorted by 'salary'.

==== Dataview query language ====

The dataview query language is a straightforward, organized custom query language that enables you to quickly create views from data.
It enables the following:
* Retrieve pages related with tags, folders, and links, among other things.
* Simple actions on fields, such as comparison, existence checks, and so on, can be used to filter notes/data.
* Sorting results according to their fields.

The query language is capable of generating the view kinds, which are detailed below:

* TABLE: The standard view type; one row for each data point, with multiple columns of field data.
* LIST: A list of pages that correspond to the query. Each page can have a single linked value.
* TASK: A collection of tasks whose pages correspond to the specified query.
To query data with Dataview Query Language the 'dataview' language specification for a codeblock is used.

===== Example result of a data query =====
[[File:Dont_know_how_to_embed_images_yet]]
The queries leading to this result are listed below.
===== The general format of queries: =====
<pre>
```dataview
TABLE|LIST|TASK <field> [AS "Column Name"], <field>, ..., <field>
FROM <source> (like #tag or "folder")
WHERE <expression> (like 'field = value')
SORT <expression> [ASC/DESC] (like 'field ASC')
```
</pre>
==== Example with jason_statham.md and bruce_lee.md ====
<pre>
```dataview
TABLE name as "Name", salary as "Salary", department as "Department"
FROM #employee
SORT salary ASC
```
</pre>
=== Dataview Javascript API ===
The Dataview JavaScript API allows arbitrary JavaScript to be executed with access to the dataview indices and query engine, which is useful for complex views or interoperability with other plugins.
To query data with Dataview Javascript API the 'dataviewjs' language specification for a codeblock is used.
The API is accessible via the implicitly provided dv (or dataview) variable, which allows you to query for data, render HTML, and configure the view.
==== Example with jason_statham.md and bruce_lee.md ====
<pre>
```dataviewjs
let employees = dv.pages("#employee")
.sort(emp => emp.salary, "asc")
.map(emp => [emp.file.link, emp.name, emp.salary, emp.department])
dv.table(["File", "Name", "Salary", "Department"], employees)
```
</pre>

== Conclusion ==
There is no defined standard for OSINT data organization, because the data may come in different forms, including, but not limited to, web-pages, paper documents, online calendars, video and audio recordings. Due to this, it is nearly impossible to create a convenient tool for all use cases. If the operation is big enough, it might be feasible to create a devoted web application that stores all necessary data in a database. However, since OSINT itself it usually a highly confidential activity, publishing the application in Clear Web is a privacy and a security risk.

File:Tweetdeck.png

2022-04-24T13:28:21Z

Mgoroz:

OSINT – theory and practice

2022-04-24T13:25:26Z

Mgoroz: picture

= Framework =
[[File:Osint_example.png|thumb|Example of OSINT analysis]]
The framework for Open source intelligence is both sources for the searched data and ways to obtain and analyze it. The whole framework depends on the goal and capacities of the research in which the OSINT method is utilized. This means that two OSINT projects with different goals most likely would have completely different frameworks. This can even happen for researches with the same goals. For example, this year an emergence of OSINT techniques in tracking of the latest developments in Ukraine war can be observed.

While having the same general goal — looking as deep as possible into the fog of war — different researchers have their own subgoals, i.e. tracking weaponry losses like Oryx project or tracking movements of armies like Conflict Intelligence Team. In addition, the researchers use wide variety of methods from analyzing of social media publications, photos and videos, to using plane- and ship-tracking services and even traffic functions of Google Maps to track movement of the armies.
== Goal of research ==
In many cases OSINT research starts with a certain goal and this goal shapes the whole framework: which data needs to be acquired, where it is searched and how it is analyzed. However, there are cases when the framework is defined by the data. This can happen after different leaks of documents, personal information or any other data. Examples for this can be the whole WikiLeaks project, where investigators worked with leaked secret documents, or investigations that followed the leak of Yandex’s food delivery service clients, which among other things allowed to uncover properties owned by Putin’s close circle.
== Sources of information ==
As mentioned above, OSINT can work with any data that is open to the public. Generally the sources of information could be divided in a few categories:
*Internet
**Social media
**Blogs and forums
**Maps and tracking services
**Web analysis services like Google Analytics
**Other online publications
*Media
**Magazines and papers
**TV
**Radio
**Online outlets
*Government data
**Official declarations
**Land registries
**Government contracts
**Other documents
**Speeches of officials
*Academic publications
*Commercial data
**Databases
**Other services that can provide necessary data (i.e. satellite image sources, company information, etc)
All of those sources can be interlinked — as generally nowadays most of the government information, media, academic publications, etc are in the internet.
==Tools to collect data==
Tools to collect necessary data depend of the type of the data. Generally, the main sources for open source intelligence are social media and, most importantly, Twitter. The reason for this is the news and current events orientation of this website and powerful advanced search capabilities. Using TweetDeck researcher can formulate a search request for what they seek and get real time updates. There are also wide capabilities of using Twitter API to parse its data and structure it. There are also possibilities of using API of other social media but it is much more limited.

OSINT also heavily utilizes search engines so it’s a good idea to learn advanced search tools. In addition, it might be useful to use more than one search engine as some of the information can be withdrawn from the results due to legal reasons or terms of service.

To fully utilize possibilities of traditional media for your research it would be useful to have subscriptions to the biggest agencies or outlets. As these subscriptions can be really expensive, especially when one might need all of them, it’s also good idea to learn how to surpass paywall — in most cases it can be done easily with the incognito mode of the browser or some kind of webarchive service.

For the official government information in a lot of cases it is possible to subscribe to an RSS or email updates about new documents and press-releases. If this is not possible, one might write a script that parses a page that he is interested in and notifies about any updates of its content. Utilizing of some services like government contracts registers might need extensive training to analyze. Other services like land registries in many countries require payment for its information, so it might be not the best starting point for collecting data.

Researcher might also need subscriptions to commercial services that are needed for the analysis. Examples of such services might include Flightradar24, Similarweb, Himera Search and others. In addition, there are services, for example Telegram bots, that search through the know data breaches for a certain entry.

A VPN should be used by the researcher both due to reasons of security and access to information, moreover, a possibility to choose servers in different countries might be useful. Different countries have different information laws and different services can restrict access for foreign users so not all needed data might be possible to acquire from researcher’s location.

==Tools to analyze data==
The analysis of the acquired data in most cases would be the most challenging part of the OSINT. In case when the information is mostly text it is easier as text is much easier to analyze using parsing, programming of just word search.

When dealing with photos or videos it would be most likely needed to analyze them personally — or double check after the used algorithm of analysis if there is one for the task.

A useful tool of analysis is visualization, especially when it comes to location-based research and big databases of structured information.
The exact methodology of analysis as well as of the data collection should be determined by the researcher at the start of the work.

= Data organization =
While an OSINT enthusiast may be adept at data collection, he or she will never develop the necessary data organization skills and tools to become a true professional.
There are numerous methods for storing data, including basic text files or notes.
However, using text files is impractical, as when there is a large amount of data, it becomes unmanageable.
Features desirable for OSINT data management include the ability to export and backup, as well as visualize data.
== Examples of software for OSINT data organization and their disadvantages: ==
* Simple Notes Apps (unmanageable when dealing with a large amount of data)
* Evernote (useful when paid for)
* Notion (notes cannot be accessed offline)
* Joplin (inconvenient organization for large projects)
* Obsidian.md Obsidian.md (a bit tricky to master)
== Obsidian.md ==
Obsidian.md, being perplexive in comparison to simple notes application, contains all the desirable features. It is a cross-platform, free application for organizing notes stored in markup (.md) files.
Notes and files are stored on a user's computer, and there is also a premium feature for syncing, which is superfluous given that backups using any online storage service, Syncthing software, or Git.
Given that OSINT specialists often work in teams, it is recommended to store the data in a Git repository in order to retain a history of modifications and increase collaboration capability.
== Vaults ==
Obsidian.md contains all data in what are referred to as "Vaults."
A vault is a project that houses all of it's associated notes and information.
== Plugins ==
Obsidian.md supports the installation of community plugins that extend the app's initial functionality.
=== Recommended plugins ===
# Dataview – Allows us to treat a vault as a database, querying and visualizing information from notes and files.
# BreadCrumbs – Adds link types and notes hierarchy.
# Juggl – Create mindmaps based on your notes and customize their looks with CSS and internal styling features.
=== Plugin installation ===
# Open Settings – the button is in the bottom-left corner of the application.
# Choose 'Community Plugins' from the 'Options' clause.
# Switch 'Safe Mode' to OFF and confirm it.
# Click 'Browse Community Plugins'.
# Find the plugin.
# Click 'Install'.
# Go back to 'Community Plugins' submenu.
# In the bottom section turn on the newly installed plugin.
== Folding vs Tagging and Linking ==
Simple folder structure is sufficient, when it comes to organizing data in nonoverlapping groups. It is enough to have just a couple of folders in your photogallery, for example. But in OSINT it is important to have a more sophisticated structure.
=== Tagging ===
Tagging adds structure because a piece of data can have several tags, as opposed to folders, which can only have one organizing unit per file.
==== Tag structure example: ====
# #people #processes #technology (part targeted)
# #primary #supportive #irrelevant (importance)
# #finished #unfinished (state of note/file)
# #web #registry #socialengineering (means of getting the information)

=== Linking ===
Linking enables the creation of relationships between notes and files.
This manner, one note can include connections to other notes and files, making it easier to handle.
For example, if John purchased the domain name legit.com, John's note can be linked to legit.com's note, which contains information about the domain.

==== Link types ====
Using link types opens up even more possibilities. Link types are included in Breadcrumbs Plugin for Obsidian.md.
In the aforementioned situation of John and legit.com, John is the domain's owner, thus, the domain is John's asset. These are called types of relations.
If it is later revealed that John purchased another domain name - fake.com – the new domain can be connected back to John.
This structure will be displayed in the notes by creating two relations of John's ownership:
# John – owner of legit.com, fake.com
# legit.com – asset of Johh, relative of fake.com
# fake.com – asset of John, relative of legit.com

== Dataview plugin ==
Dataview is, first and foremost, a data index, so it supports relatively rich methods of adding metadata to your knowledge base.
Dataview tracks information at the markdown page and markdown task levels, with each page/task able to contain an arbitrary number of complex (numbers, objects, lists) fields.
Each field is a named value of a specific type (like "number" or "text").

=== Example of notes with arbitrary metadata and a tag: ===

jason_statham.md
<pre>
---
name: Jason Statham
salary: 7500
department: Cyber Forensics
notes: [
"Potential phishing target",
"Mother has stage T4 cancer"
]
---
#employee
</pre>
bruce_lee.md
<pre>
---
name: Bruce Lee
salary: 8000
department: Developer Operations
notes: []
---
</pre>

== Querying dataview data ==
=== Options for querying data: ===
# Dataview query language
# Dataview Javascript API

Both can be used to, as an example, render a table from jason_statham.md and bruce_lee.md with four columns:
# File – contains a link to the file
# Name – metadata 'name'
# Salary – metadata 'salary'
# Department – metadata 'department'
It can also be sorted by 'salary'.

==== Dataview query language ====

The dataview query language is a straightforward, organized custom query language that enables you to quickly create views from data.
It enables the following:
* Retrieve pages related with tags, folders, and links, among other things.
* Simple actions on fields, such as comparison, existence checks, and so on, can be used to filter notes/data.
* Sorting results according to their fields.

The query language is capable of generating the view kinds, which are detailed below:

* TABLE: The standard view type; one row for each data point, with multiple columns of field data.
* LIST: A list of pages that correspond to the query. Each page can have a single linked value.
* TASK: A collection of tasks whose pages correspond to the specified query.
To query data with Dataview Query Language the 'dataview' language specification for a codeblock is used.

===== Example result of a data query =====
[[File:Dont_know_how_to_embed_images_yet]]
The queries leading to this result are listed below.
===== The general format of queries: =====
<pre>
```dataview
TABLE|LIST|TASK <field> [AS "Column Name"], <field>, ..., <field>
FROM <source> (like #tag or "folder")
WHERE <expression> (like 'field = value')
SORT <expression> [ASC/DESC] (like 'field ASC')
```
</pre>
==== Example with jason_statham.md and bruce_lee.md ====
<pre>
```dataview
TABLE name as "Name", salary as "Salary", department as "Department"
FROM #employee
SORT salary ASC
```
</pre>
=== Dataview Javascript API ===
The Dataview JavaScript API allows arbitrary JavaScript to be executed with access to the dataview indices and query engine, which is useful for complex views or interoperability with other plugins.
To query data with Dataview Javascript API the 'dataviewjs' language specification for a codeblock is used.
The API is accessible via the implicitly provided dv (or dataview) variable, which allows you to query for data, render HTML, and configure the view.
==== Example with jason_statham.md and bruce_lee.md ====
<pre>
```dataviewjs
let employees = dv.pages("#employee")
.sort(emp => emp.salary, "asc")
.map(emp => [emp.file.link, emp.name, emp.salary, emp.department])
dv.table(["File", "Name", "Salary", "Department"], employees)
```
</pre>

== Conclusion ==
There is no defined standard for OSINT data organization, because the data may come in different forms, including, but not limited to, web-pages, paper documents, online calendars, video and audio recordings. Due to this, it is nearly impossible to create a convenient tool for all use cases. If the operation is big enough, it might be feasible to create a devoted web application that stores all necessary data in a database. However, since OSINT itself it usually a highly confidential activity, publishing the application in Clear Web is a privacy and a security risk.

File:Osint example.png

2022-04-24T13:20:39Z

Mgoroz:

OSINT – theory and practice

2022-04-24T13:19:24Z

Mgoroz:

= Framework =
The framework for Open source intelligence is both sources for the searched data and ways to obtain and analyze it. The whole framework depends on the goal and capacities of the research in which the OSINT method is utilized. This means that two OSINT projects with different goals most likely would have completely different frameworks. This can even happen for researches with the same goals. For example, this year an emergence of OSINT techniques in tracking of the latest developments in Ukraine war can be observed.
[[File:https://i.imgur.com/H6wqKuC.png]]
While having the same general goal — looking as deep as possible into the fog of war — different researchers have their own subgoals, i.e. tracking weaponry losses like Oryx project or tracking movements of armies like Conflict Intelligence Team. In addition, the researchers use wide variety of methods from analyzing of social media publications, photos and videos, to using plane- and ship-tracking services and even traffic functions of Google Maps to track movement of the armies.
== Goal of research ==
In many cases OSINT research starts with a certain goal and this goal shapes the whole framework: which data needs to be acquired, where it is searched and how it is analyzed. However, there are cases when the framework is defined by the data. This can happen after different leaks of documents, personal information or any other data. Examples for this can be the whole WikiLeaks project, where investigators worked with leaked secret documents, or investigations that followed the leak of Yandex’s food delivery service clients, which among other things allowed to uncover properties owned by Putin’s close circle.
== Sources of information ==
As mentioned above, OSINT can work with any data that is open to the public. Generally the sources of information could be divided in a few categories:
*Internet
**Social media
**Blogs and forums
**Maps and tracking services
**Web analysis services like Google Analytics
**Other online publications
*Media
**Magazines and papers
**TV
**Radio
**Online outlets
*Government data
**Official declarations
**Land registries
**Government contracts
**Other documents
**Speeches of officials
*Academic publications
*Commercial data
**Databases
**Other services that can provide necessary data (i.e. satellite image sources, company information, etc)
All of those sources can be interlinked — as generally nowadays most of the government information, media, academic publications, etc are in the internet.
==Tools to collect data==
Tools to collect necessary data depend of the type of the data. Generally, the main sources for open source intelligence are social media and, most importantly, Twitter. The reason for this is the news and current events orientation of this website and powerful advanced search capabilities. Using TweetDeck researcher can formulate a search request for what they seek and get real time updates. There are also wide capabilities of using Twitter API to parse its data and structure it. There are also possibilities of using API of other social media but it is much more limited.

OSINT also heavily utilizes search engines so it’s a good idea to learn advanced search tools. In addition, it might be useful to use more than one search engine as some of the information can be withdrawn from the results due to legal reasons or terms of service.

To fully utilize possibilities of traditional media for your research it would be useful to have subscriptions to the biggest agencies or outlets. As these subscriptions can be really expensive, especially when one might need all of them, it’s also good idea to learn how to surpass paywall — in most cases it can be done easily with the incognito mode of the browser or some kind of webarchive service.

For the official government information in a lot of cases it is possible to subscribe to an RSS or email updates about new documents and press-releases. If this is not possible, one might write a script that parses a page that he is interested in and notifies about any updates of its content. Utilizing of some services like government contracts registers might need extensive training to analyze. Other services like land registries in many countries require payment for its information, so it might be not the best starting point for collecting data.

Researcher might also need subscriptions to commercial services that are needed for the analysis. Examples of such services might include Flightradar24, Similarweb, Himera Search and others. In addition, there are services, for example Telegram bots, that search through the know data breaches for a certain entry.

A VPN should be used by the researcher both due to reasons of security and access to information, moreover, a possibility to choose servers in different countries might be useful. Different countries have different information laws and different services can restrict access for foreign users so not all needed data might be possible to acquire from researcher’s location.

==Tools to analyze data==
The analysis of the acquired data in most cases would be the most challenging part of the OSINT. In case when the information is mostly text it is easier as text is much easier to analyze using parsing, programming of just word search.

When dealing with photos or videos it would be most likely needed to analyze them personally — or double check after the used algorithm of analysis if there is one for the task.

A useful tool of analysis is visualization, especially when it comes to location-based research and big databases of structured information.
The exact methodology of analysis as well as of the data collection should be determined by the researcher at the start of the work.

= Data organization =
While an OSINT enthusiast may be adept at data collection, he or she will never develop the necessary data organization skills and tools to become a true professional.
There are numerous methods for storing data, including basic text files or notes.
However, using text files is impractical, as when there is a large amount of data, it becomes unmanageable.
Features desirable for OSINT data management include the ability to export and backup, as well as visualize data.
== Examples of software for OSINT data organization and their disadvantages: ==
* Simple Notes Apps (unmanageable when dealing with a large amount of data)
* Evernote (useful when paid for)
* Notion (notes cannot be accessed offline)
* Joplin (inconvenient organization for large projects)
* Obsidian.md Obsidian.md (a bit tricky to master)
== Obsidian.md ==
Obsidian.md, being perplexive in comparison to simple notes application, contains all the desirable features. It is a cross-platform, free application for organizing notes stored in markup (.md) files.
Notes and files are stored on a user's computer, and there is also a premium feature for syncing, which is superfluous given that backups using any online storage service, Syncthing software, or Git.
Given that OSINT specialists often work in teams, it is recommended to store the data in a Git repository in order to retain a history of modifications and increase collaboration capability.
== Vaults ==
Obsidian.md contains all data in what are referred to as "Vaults."
A vault is a project that houses all of it's associated notes and information.
== Plugins ==
Obsidian.md supports the installation of community plugins that extend the app's initial functionality.
=== Recommended plugins ===
# Dataview – Allows us to treat a vault as a database, querying and visualizing information from notes and files.
# BreadCrumbs – Adds link types and notes hierarchy.
# Juggl – Create mindmaps based on your notes and customize their looks with CSS and internal styling features.
=== Plugin installation ===
# Open Settings – the button is in the bottom-left corner of the application.
# Choose 'Community Plugins' from the 'Options' clause.
# Switch 'Safe Mode' to OFF and confirm it.
# Click 'Browse Community Plugins'.
# Find the plugin.
# Click 'Install'.
# Go back to 'Community Plugins' submenu.
# In the bottom section turn on the newly installed plugin.
== Folding vs Tagging and Linking ==
Simple folder structure is sufficient, when it comes to organizing data in nonoverlapping groups. It is enough to have just a couple of folders in your photogallery, for example. But in OSINT it is important to have a more sophisticated structure.
=== Tagging ===
Tagging adds structure because a piece of data can have several tags, as opposed to folders, which can only have one organizing unit per file.
==== Tag structure example: ====
# #people #processes #technology (part targeted)
# #primary #supportive #irrelevant (importance)
# #finished #unfinished (state of note/file)
# #web #registry #socialengineering (means of getting the information)

=== Linking ===
Linking enables the creation of relationships between notes and files.
This manner, one note can include connections to other notes and files, making it easier to handle.
For example, if John purchased the domain name legit.com, John's note can be linked to legit.com's note, which contains information about the domain.

==== Link types ====
Using link types opens up even more possibilities. Link types are included in Breadcrumbs Plugin for Obsidian.md.
In the aforementioned situation of John and legit.com, John is the domain's owner, thus, the domain is John's asset. These are called types of relations.
If it is later revealed that John purchased another domain name - fake.com – the new domain can be connected back to John.
This structure will be displayed in the notes by creating two relations of John's ownership:
# John – owner of legit.com, fake.com
# legit.com – asset of Johh, relative of fake.com
# fake.com – asset of John, relative of legit.com

== Dataview plugin ==
Dataview is, first and foremost, a data index, so it supports relatively rich methods of adding metadata to your knowledge base.
Dataview tracks information at the markdown page and markdown task levels, with each page/task able to contain an arbitrary number of complex (numbers, objects, lists) fields.
Each field is a named value of a specific type (like "number" or "text").

=== Example of notes with arbitrary metadata and a tag: ===

jason_statham.md
<pre>
---
name: Jason Statham
salary: 7500
department: Cyber Forensics
notes: [
"Potential phishing target",
"Mother has stage T4 cancer"
]
---
#employee
</pre>
bruce_lee.md
<pre>
---
name: Bruce Lee
salary: 8000
department: Developer Operations
notes: []
---
</pre>

== Querying dataview data ==
=== Options for querying data: ===
# Dataview query language
# Dataview Javascript API

Both can be used to, as an example, render a table from jason_statham.md and bruce_lee.md with four columns:
# File – contains a link to the file
# Name – metadata 'name'
# Salary – metadata 'salary'
# Department – metadata 'department'
It can also be sorted by 'salary'.

==== Dataview query language ====

The dataview query language is a straightforward, organized custom query language that enables you to quickly create views from data.
It enables the following:
* Retrieve pages related with tags, folders, and links, among other things.
* Simple actions on fields, such as comparison, existence checks, and so on, can be used to filter notes/data.
* Sorting results according to their fields.

The query language is capable of generating the view kinds, which are detailed below:

* TABLE: The standard view type; one row for each data point, with multiple columns of field data.
* LIST: A list of pages that correspond to the query. Each page can have a single linked value.
* TASK: A collection of tasks whose pages correspond to the specified query.
To query data with Dataview Query Language the 'dataview' language specification for a codeblock is used.

===== Example result of a data query =====
[[File:Dont_know_how_to_embed_images_yet]]
The queries leading to this result are listed below.
===== The general format of queries: =====
<pre>
```dataview
TABLE|LIST|TASK <field> [AS "Column Name"], <field>, ..., <field>
FROM <source> (like #tag or "folder")
WHERE <expression> (like 'field = value')
SORT <expression> [ASC/DESC] (like 'field ASC')
```
</pre>
==== Example with jason_statham.md and bruce_lee.md ====
<pre>
```dataview
TABLE name as "Name", salary as "Salary", department as "Department"
FROM #employee
SORT salary ASC
```
</pre>
=== Dataview Javascript API ===
The Dataview JavaScript API allows arbitrary JavaScript to be executed with access to the dataview indices and query engine, which is useful for complex views or interoperability with other plugins.
To query data with Dataview Javascript API the 'dataviewjs' language specification for a codeblock is used.
The API is accessible via the implicitly provided dv (or dataview) variable, which allows you to query for data, render HTML, and configure the view.
==== Example with jason_statham.md and bruce_lee.md ====
<pre>
```dataviewjs
let employees = dv.pages("#employee")
.sort(emp => emp.salary, "asc")
.map(emp => [emp.file.link, emp.name, emp.salary, emp.department])
dv.table(["File", "Name", "Salary", "Department"], employees)
```
</pre>

== Conclusion ==
There is no defined standard for OSINT data organization, because the data may come in different forms, including, but not limited to, web-pages, paper documents, online calendars, video and audio recordings. Due to this, it is nearly impossible to create a convenient tool for all use cases. If the operation is big enough, it might be feasible to create a devoted web application that stores all necessary data in a database. However, since OSINT itself it usually a highly confidential activity, publishing the application in Clear Web is a privacy and a security risk.

OSINT – theory and practice

2022-04-24T13:13:34Z

Mgoroz: Tools to analyze data

= Framework =
The framework for Open source intelligence is both sources for the searched data and ways to obtain and analyze it. The whole framework depends on the goal and capacities of the research in which the OSINT method is utilized. This means that two OSINT projects with different goals most likely would have completely different frameworks. This can even happen for researches with the same goals. For example, this year an emergence of OSINT techniques in tracking of the latest developments in Ukraine war can be observed.

While having the same general goal — looking as deep as possible into the fog of war — different researchers have their own subgoals, i.e. tracking weaponry losses like Oryx project or tracking movements of armies like Conflict Intelligence Team. In addition, the researchers use wide variety of methods from analyzing of social media publications, photos and videos, to using plane- and ship-tracking services and even traffic functions of Google Maps to track movement of the armies.
== Goal of research ==
In many cases OSINT research starts with a certain goal and this goal shapes the whole framework: which data needs to be acquired, where it is searched and how it is analyzed. However, there are cases when the framework is defined by the data. This can happen after different leaks of documents, personal information or any other data. Examples for this can be the whole WikiLeaks project, where investigators worked with leaked secret documents, or investigations that followed the leak of Yandex’s food delivery service clients, which among other things allowed to uncover properties owned by Putin’s close circle.
== Sources of information ==
As mentioned above, OSINT can work with any data that is open to the public. Generally the sources of information could be divided in a few categories:
*Internet
**Social media
**Blogs and forums
**Maps and tracking services
**Web analysis services like Google Analytics
**Other online publications
*Media
**Magazines and papers
**TV
**Radio
**Online outlets
*Government data
**Official declarations
**Land registries
**Government contracts
**Other documents
**Speeches of officials
*Academic publications
*Commercial data
**Databases
**Other services that can provide necessary data (i.e. satellite image sources, company information, etc)
All of those sources can be interlinked — as generally nowadays most of the government information, media, academic publications, etc are in the internet.
==Tools to collect data==
Tools to collect necessary data depend of the type of the data. Generally, the main sources for open source intelligence are social media and, most importantly, Twitter. The reason for this is the news and current events orientation of this website and powerful advanced search capabilities. Using TweetDeck researcher can formulate a search request for what they seek and get real time updates. There are also wide capabilities of using Twitter API to parse its data and structure it. There are also possibilities of using API of other social media but it is much more limited.

OSINT also heavily utilizes search engines so it’s a good idea to learn advanced search tools. In addition, it might be useful to use more than one search engine as some of the information can be withdrawn from the results due to legal reasons or terms of service.

To fully utilize possibilities of traditional media for your research it would be useful to have subscriptions to the biggest agencies or outlets. As these subscriptions can be really expensive, especially when one might need all of them, it’s also good idea to learn how to surpass paywall — in most cases it can be done easily with the incognito mode of the browser or some kind of webarchive service.

For the official government information in a lot of cases it is possible to subscribe to an RSS or email updates about new documents and press-releases. If this is not possible, one might write a script that parses a page that he is interested in and notifies about any updates of its content. Utilizing of some services like government contracts registers might need extensive training to analyze. Other services like land registries in many countries require payment for its information, so it might be not the best starting point for collecting data.

Researcher might also need subscriptions to commercial services that are needed for the analysis. Examples of such services might include Flightradar24, Similarweb, Himera Search and others. In addition, there are services, for example Telegram bots, that search through the know data breaches for a certain entry.

A VPN should be used by the researcher both due to reasons of security and access to information, moreover, a possibility to choose servers in different countries might be useful. Different countries have different information laws and different services can restrict access for foreign users so not all needed data might be possible to acquire from researcher’s location.

==Tools to analyze data==
The analysis of the acquired data in most cases would be the most challenging part of the OSINT. In case when the information is mostly text it is easier as text is much easier to analyze using parsing, programming of just word search.

When dealing with photos or videos it would be most likely needed to analyze them personally — or double check after the used algorithm of analysis if there is one for the task.

A useful tool of analysis is visualization, especially when it comes to location-based research and big databases of structured information.
The exact methodology of analysis as well as of the data collection should be determined by the researcher at the start of the work.

= Data organization =
While an OSINT enthusiast may be adept at data collection, he or she will never develop the necessary data organization skills and tools to become a true professional.
There are numerous methods for storing data, including basic text files or notes.
However, using text files is impractical, as when there is a large amount of data, it becomes unmanageable.
Features desirable for OSINT data management include the ability to export and backup, as well as visualize data.
== Examples of software for OSINT data organization and their disadvantages: ==
* Simple Notes Apps (unmanageable when dealing with a large amount of data)
* Evernote (useful when paid for)
* Notion (notes cannot be accessed offline)
* Joplin (inconvenient organization for large projects)
* Obsidian.md Obsidian.md (a bit tricky to master)
== Obsidian.md ==
Obsidian.md, being perplexive in comparison to simple notes application, contains all the desirable features. It is a cross-platform, free application for organizing notes stored in markup (.md) files.
Notes and files are stored on a user's computer, and there is also a premium feature for syncing, which is superfluous given that backups using any online storage service, Syncthing software, or Git.
Given that OSINT specialists often work in teams, it is recommended to store the data in a Git repository in order to retain a history of modifications and increase collaboration capability.
== Vaults ==
Obsidian.md contains all data in what are referred to as "Vaults."
A vault is a project that houses all of it's associated notes and information.
== Plugins ==
Obsidian.md supports the installation of community plugins that extend the app's initial functionality.
=== Recommended plugins ===
# Dataview – Allows us to treat a vault as a database, querying and visualizing information from notes and files.
# BreadCrumbs – Adds link types and notes hierarchy.
# Juggl – Create mindmaps based on your notes and customize their looks with CSS and internal styling features.
=== Plugin installation ===
# Open Settings – the button is in the bottom-left corner of the application.
# Choose 'Community Plugins' from the 'Options' clause.
# Switch 'Safe Mode' to OFF and confirm it.
# Click 'Browse Community Plugins'.
# Find the plugin.
# Click 'Install'.
# Go back to 'Community Plugins' submenu.
# In the bottom section turn on the newly installed plugin.
== Folding vs Tagging and Linking ==
Simple folder structure is sufficient, when it comes to organizing data in nonoverlapping groups. It is enough to have just a couple of folders in your photogallery, for example. But in OSINT it is important to have a more sophisticated structure.
=== Tagging ===
Tagging adds structure because a piece of data can have several tags, as opposed to folders, which can only have one organizing unit per file.
==== Tag structure example: ====
# #people #processes #technology (part targeted)
# #primary #supportive #irrelevant (importance)
# #finished #unfinished (state of note/file)
# #web #registry #socialengineering (means of getting the information)

=== Linking ===
Linking enables the creation of relationships between notes and files.
This manner, one note can include connections to other notes and files, making it easier to handle.
For example, if John purchased the domain name legit.com, John's note can be linked to legit.com's note, which contains information about the domain.

==== Link types ====
Using link types opens up even more possibilities. Link types are included in Breadcrumbs Plugin for Obsidian.md.
In the aforementioned situation of John and legit.com, John is the domain's owner, thus, the domain is John's asset. These are called types of relations.
If it is later revealed that John purchased another domain name - fake.com – the new domain can be connected back to John.
This structure will be displayed in the notes by creating two relations of John's ownership:
# John – owner of legit.com, fake.com
# legit.com – asset of Johh, relative of fake.com
# fake.com – asset of John, relative of legit.com

== Dataview plugin ==
Dataview is, first and foremost, a data index, so it supports relatively rich methods of adding metadata to your knowledge base.
Dataview tracks information at the markdown page and markdown task levels, with each page/task able to contain an arbitrary number of complex (numbers, objects, lists) fields.
Each field is a named value of a specific type (like "number" or "text").

=== Example of notes with arbitrary metadata and a tag: ===

jason_statham.md
<pre>
---
name: Jason Statham
salary: 7500
department: Cyber Forensics
notes: [
"Potential phishing target",
"Mother has stage T4 cancer"
]
---
#employee
</pre>
bruce_lee.md
<pre>
---
name: Bruce Lee
salary: 8000
department: Developer Operations
notes: []
---
</pre>

== Querying dataview data ==
=== Options for querying data: ===
# Dataview query language
# Dataview Javascript API

Both can be used to, as an example, render a table from jason_statham.md and bruce_lee.md with four columns:
# File – contains a link to the file
# Name – metadata 'name'
# Salary – metadata 'salary'
# Department – metadata 'department'
It can also be sorted by 'salary'.

==== Dataview query language ====

The dataview query language is a straightforward, organized custom query language that enables you to quickly create views from data.
It enables the following:
* Retrieve pages related with tags, folders, and links, among other things.
* Simple actions on fields, such as comparison, existence checks, and so on, can be used to filter notes/data.
* Sorting results according to their fields.

The query language is capable of generating the view kinds, which are detailed below:

* TABLE: The standard view type; one row for each data point, with multiple columns of field data.
* LIST: A list of pages that correspond to the query. Each page can have a single linked value.
* TASK: A collection of tasks whose pages correspond to the specified query.
To query data with Dataview Query Language the 'dataview' language specification for a codeblock is used.

===== Example result of a data query =====
[[File:Dont_know_how_to_embed_images_yet]]
The queries leading to this result are listed below.
===== The general format of queries: =====
<pre>
```dataview
TABLE|LIST|TASK <field> [AS "Column Name"], <field>, ..., <field>
FROM <source> (like #tag or "folder")
WHERE <expression> (like 'field = value')
SORT <expression> [ASC/DESC] (like 'field ASC')
```
</pre>
==== Example with jason_statham.md and bruce_lee.md ====
<pre>
```dataview
TABLE name as "Name", salary as "Salary", department as "Department"
FROM #employee
SORT salary ASC
```
</pre>
=== Dataview Javascript API ===
The Dataview JavaScript API allows arbitrary JavaScript to be executed with access to the dataview indices and query engine, which is useful for complex views or interoperability with other plugins.
To query data with Dataview Javascript API the 'dataviewjs' language specification for a codeblock is used.
The API is accessible via the implicitly provided dv (or dataview) variable, which allows you to query for data, render HTML, and configure the view.
==== Example with jason_statham.md and bruce_lee.md ====
<pre>
```dataviewjs
let employees = dv.pages("#employee")
.sort(emp => emp.salary, "asc")
.map(emp => [emp.file.link, emp.name, emp.salary, emp.department])
dv.table(["File", "Name", "Salary", "Department"], employees)
```
</pre>

== Conclusion ==
There is no defined standard for OSINT data organization, because the data may come in different forms, including, but not limited to, web-pages, paper documents, online calendars, video and audio recordings. Due to this, it is nearly impossible to create a convenient tool for all use cases. If the operation is big enough, it might be feasible to create a devoted web application that stores all necessary data in a database. However, since OSINT itself it usually a highly confidential activity, publishing the application in Clear Web is a privacy and a security risk.

OSINT – theory and practice

2022-04-24T13:12:56Z

Mgoroz: Tools to collect data

= Framework =
The framework for Open source intelligence is both sources for the searched data and ways to obtain and analyze it. The whole framework depends on the goal and capacities of the research in which the OSINT method is utilized. This means that two OSINT projects with different goals most likely would have completely different frameworks. This can even happen for researches with the same goals. For example, this year an emergence of OSINT techniques in tracking of the latest developments in Ukraine war can be observed.

While having the same general goal — looking as deep as possible into the fog of war — different researchers have their own subgoals, i.e. tracking weaponry losses like Oryx project or tracking movements of armies like Conflict Intelligence Team. In addition, the researchers use wide variety of methods from analyzing of social media publications, photos and videos, to using plane- and ship-tracking services and even traffic functions of Google Maps to track movement of the armies.
== Goal of research ==
In many cases OSINT research starts with a certain goal and this goal shapes the whole framework: which data needs to be acquired, where it is searched and how it is analyzed. However, there are cases when the framework is defined by the data. This can happen after different leaks of documents, personal information or any other data. Examples for this can be the whole WikiLeaks project, where investigators worked with leaked secret documents, or investigations that followed the leak of Yandex’s food delivery service clients, which among other things allowed to uncover properties owned by Putin’s close circle.
== Sources of information ==
As mentioned above, OSINT can work with any data that is open to the public. Generally the sources of information could be divided in a few categories:
*Internet
**Social media
**Blogs and forums
**Maps and tracking services
**Web analysis services like Google Analytics
**Other online publications
*Media
**Magazines and papers
**TV
**Radio
**Online outlets
*Government data
**Official declarations
**Land registries
**Government contracts
**Other documents
**Speeches of officials
*Academic publications
*Commercial data
**Databases
**Other services that can provide necessary data (i.e. satellite image sources, company information, etc)
All of those sources can be interlinked — as generally nowadays most of the government information, media, academic publications, etc are in the internet.
==Tools to collect data==
Tools to collect necessary data depend of the type of the data. Generally, the main sources for open source intelligence are social media and, most importantly, Twitter. The reason for this is the news and current events orientation of this website and powerful advanced search capabilities. Using TweetDeck researcher can formulate a search request for what they seek and get real time updates. There are also wide capabilities of using Twitter API to parse its data and structure it. There are also possibilities of using API of other social media but it is much more limited.

OSINT also heavily utilizes search engines so it’s a good idea to learn advanced search tools. In addition, it might be useful to use more than one search engine as some of the information can be withdrawn from the results due to legal reasons or terms of service.

To fully utilize possibilities of traditional media for your research it would be useful to have subscriptions to the biggest agencies or outlets. As these subscriptions can be really expensive, especially when one might need all of them, it’s also good idea to learn how to surpass paywall — in most cases it can be done easily with the incognito mode of the browser or some kind of webarchive service.

For the official government information in a lot of cases it is possible to subscribe to an RSS or email updates about new documents and press-releases. If this is not possible, one might write a script that parses a page that he is interested in and notifies about any updates of its content. Utilizing of some services like government contracts registers might need extensive training to analyze. Other services like land registries in many countries require payment for its information, so it might be not the best starting point for collecting data.

Researcher might also need subscriptions to commercial services that are needed for the analysis. Examples of such services might include Flightradar24, Similarweb, Himera Search and others. In addition, there are services, for example Telegram bots, that search through the know data breaches for a certain entry.

A VPN should be used by the researcher both due to reasons of security and access to information, moreover, a possibility to choose servers in different countries might be useful. Different countries have different information laws and different services can restrict access for foreign users so not all needed data might be possible to acquire from researcher’s location.

= Data organization =
While an OSINT enthusiast may be adept at data collection, he or she will never develop the necessary data organization skills and tools to become a true professional.
There are numerous methods for storing data, including basic text files or notes.
However, using text files is impractical, as when there is a large amount of data, it becomes unmanageable.
Features desirable for OSINT data management include the ability to export and backup, as well as visualize data.
== Examples of software for OSINT data organization and their disadvantages: ==
* Simple Notes Apps (unmanageable when dealing with a large amount of data)
* Evernote (useful when paid for)
* Notion (notes cannot be accessed offline)
* Joplin (inconvenient organization for large projects)
* Obsidian.md Obsidian.md (a bit tricky to master)
== Obsidian.md ==
Obsidian.md, being perplexive in comparison to simple notes application, contains all the desirable features. It is a cross-platform, free application for organizing notes stored in markup (.md) files.
Notes and files are stored on a user's computer, and there is also a premium feature for syncing, which is superfluous given that backups using any online storage service, Syncthing software, or Git.
Given that OSINT specialists often work in teams, it is recommended to store the data in a Git repository in order to retain a history of modifications and increase collaboration capability.
== Vaults ==
Obsidian.md contains all data in what are referred to as "Vaults."
A vault is a project that houses all of it's associated notes and information.
== Plugins ==
Obsidian.md supports the installation of community plugins that extend the app's initial functionality.
=== Recommended plugins ===
# Dataview – Allows us to treat a vault as a database, querying and visualizing information from notes and files.
# BreadCrumbs – Adds link types and notes hierarchy.
# Juggl – Create mindmaps based on your notes and customize their looks with CSS and internal styling features.
=== Plugin installation ===
# Open Settings – the button is in the bottom-left corner of the application.
# Choose 'Community Plugins' from the 'Options' clause.
# Switch 'Safe Mode' to OFF and confirm it.
# Click 'Browse Community Plugins'.
# Find the plugin.
# Click 'Install'.
# Go back to 'Community Plugins' submenu.
# In the bottom section turn on the newly installed plugin.
== Folding vs Tagging and Linking ==
Simple folder structure is sufficient, when it comes to organizing data in nonoverlapping groups. It is enough to have just a couple of folders in your photogallery, for example. But in OSINT it is important to have a more sophisticated structure.
=== Tagging ===
Tagging adds structure because a piece of data can have several tags, as opposed to folders, which can only have one organizing unit per file.
==== Tag structure example: ====
# #people #processes #technology (part targeted)
# #primary #supportive #irrelevant (importance)
# #finished #unfinished (state of note/file)
# #web #registry #socialengineering (means of getting the information)

=== Linking ===
Linking enables the creation of relationships between notes and files.
This manner, one note can include connections to other notes and files, making it easier to handle.
For example, if John purchased the domain name legit.com, John's note can be linked to legit.com's note, which contains information about the domain.

==== Link types ====
Using link types opens up even more possibilities. Link types are included in Breadcrumbs Plugin for Obsidian.md.
In the aforementioned situation of John and legit.com, John is the domain's owner, thus, the domain is John's asset. These are called types of relations.
If it is later revealed that John purchased another domain name - fake.com – the new domain can be connected back to John.
This structure will be displayed in the notes by creating two relations of John's ownership:
# John – owner of legit.com, fake.com
# legit.com – asset of Johh, relative of fake.com
# fake.com – asset of John, relative of legit.com

== Dataview plugin ==
Dataview is, first and foremost, a data index, so it supports relatively rich methods of adding metadata to your knowledge base.
Dataview tracks information at the markdown page and markdown task levels, with each page/task able to contain an arbitrary number of complex (numbers, objects, lists) fields.
Each field is a named value of a specific type (like "number" or "text").

=== Example of notes with arbitrary metadata and a tag: ===

jason_statham.md
<pre>
---
name: Jason Statham
salary: 7500
department: Cyber Forensics
notes: [
"Potential phishing target",
"Mother has stage T4 cancer"
]
---
#employee
</pre>
bruce_lee.md
<pre>
---
name: Bruce Lee
salary: 8000
department: Developer Operations
notes: []
---
</pre>

== Querying dataview data ==
=== Options for querying data: ===
# Dataview query language
# Dataview Javascript API

Both can be used to, as an example, render a table from jason_statham.md and bruce_lee.md with four columns:
# File – contains a link to the file
# Name – metadata 'name'
# Salary – metadata 'salary'
# Department – metadata 'department'
It can also be sorted by 'salary'.

==== Dataview query language ====

The dataview query language is a straightforward, organized custom query language that enables you to quickly create views from data.
It enables the following:
* Retrieve pages related with tags, folders, and links, among other things.
* Simple actions on fields, such as comparison, existence checks, and so on, can be used to filter notes/data.
* Sorting results according to their fields.

The query language is capable of generating the view kinds, which are detailed below:

* TABLE: The standard view type; one row for each data point, with multiple columns of field data.
* LIST: A list of pages that correspond to the query. Each page can have a single linked value.
* TASK: A collection of tasks whose pages correspond to the specified query.
To query data with Dataview Query Language the 'dataview' language specification for a codeblock is used.

===== Example result of a data query =====
[[File:Dont_know_how_to_embed_images_yet]]
The queries leading to this result are listed below.
===== The general format of queries: =====
<pre>
```dataview
TABLE|LIST|TASK <field> [AS "Column Name"], <field>, ..., <field>
FROM <source> (like #tag or "folder")
WHERE <expression> (like 'field = value')
SORT <expression> [ASC/DESC] (like 'field ASC')
```
</pre>
==== Example with jason_statham.md and bruce_lee.md ====
<pre>
```dataview
TABLE name as "Name", salary as "Salary", department as "Department"
FROM #employee
SORT salary ASC
```
</pre>
=== Dataview Javascript API ===
The Dataview JavaScript API allows arbitrary JavaScript to be executed with access to the dataview indices and query engine, which is useful for complex views or interoperability with other plugins.
To query data with Dataview Javascript API the 'dataviewjs' language specification for a codeblock is used.
The API is accessible via the implicitly provided dv (or dataview) variable, which allows you to query for data, render HTML, and configure the view.
==== Example with jason_statham.md and bruce_lee.md ====
<pre>
```dataviewjs
let employees = dv.pages("#employee")
.sort(emp => emp.salary, "asc")
.map(emp => [emp.file.link, emp.name, emp.salary, emp.department])
dv.table(["File", "Name", "Salary", "Department"], employees)
```
</pre>

== Conclusion ==
There is no defined standard for OSINT data organization, because the data may come in different forms, including, but not limited to, web-pages, paper documents, online calendars, video and audio recordings. Due to this, it is nearly impossible to create a convenient tool for all use cases. If the operation is big enough, it might be feasible to create a devoted web application that stores all necessary data in a database. However, since OSINT itself it usually a highly confidential activity, publishing the application in Clear Web is a privacy and a security risk.

OSINT – theory and practice

2022-04-24T13:11:41Z

Mgoroz: Sources of information

= Framework =
The framework for Open source intelligence is both sources for the searched data and ways to obtain and analyze it. The whole framework depends on the goal and capacities of the research in which the OSINT method is utilized. This means that two OSINT projects with different goals most likely would have completely different frameworks. This can even happen for researches with the same goals. For example, this year an emergence of OSINT techniques in tracking of the latest developments in Ukraine war can be observed.

While having the same general goal — looking as deep as possible into the fog of war — different researchers have their own subgoals, i.e. tracking weaponry losses like Oryx project or tracking movements of armies like Conflict Intelligence Team. In addition, the researchers use wide variety of methods from analyzing of social media publications, photos and videos, to using plane- and ship-tracking services and even traffic functions of Google Maps to track movement of the armies.
== Goal of research ==
In many cases OSINT research starts with a certain goal and this goal shapes the whole framework: which data needs to be acquired, where it is searched and how it is analyzed. However, there are cases when the framework is defined by the data. This can happen after different leaks of documents, personal information or any other data. Examples for this can be the whole WikiLeaks project, where investigators worked with leaked secret documents, or investigations that followed the leak of Yandex’s food delivery service clients, which among other things allowed to uncover properties owned by Putin’s close circle.
== Sources of information ==
As mentioned above, OSINT can work with any data that is open to the public. Generally the sources of information could be divided in a few categories:
*Internet
**Social media
**Blogs and forums
**Maps and tracking services
**Web analysis services like Google Analytics
**Other online publications
*Media
**Magazines and papers
**TV
**Radio
**Online outlets
*Government data
**Official declarations
**Land registries
**Government contracts
**Other documents
**Speeches of officials
*Academic publications
*Commercial data
**Databases
**Other services that can provide necessary data (i.e. satellite image sources, company information, etc)
All of those sources can be interlinked — as generally nowadays most of the government information, media, academic publications, etc are in the internet.

= Data organization =
While an OSINT enthusiast may be adept at data collection, he or she will never develop the necessary data organization skills and tools to become a true professional.
There are numerous methods for storing data, including basic text files or notes.
However, using text files is impractical, as when there is a large amount of data, it becomes unmanageable.
Features desirable for OSINT data management include the ability to export and backup, as well as visualize data.
== Examples of software for OSINT data organization and their disadvantages: ==
* Simple Notes Apps (unmanageable when dealing with a large amount of data)
* Evernote (useful when paid for)
* Notion (notes cannot be accessed offline)
* Joplin (inconvenient organization for large projects)
* Obsidian.md Obsidian.md (a bit tricky to master)
== Obsidian.md ==
Obsidian.md, being perplexive in comparison to simple notes application, contains all the desirable features. It is a cross-platform, free application for organizing notes stored in markup (.md) files.
Notes and files are stored on a user's computer, and there is also a premium feature for syncing, which is superfluous given that backups using any online storage service, Syncthing software, or Git.
Given that OSINT specialists often work in teams, it is recommended to store the data in a Git repository in order to retain a history of modifications and increase collaboration capability.
== Vaults ==
Obsidian.md contains all data in what are referred to as "Vaults."
A vault is a project that houses all of it's associated notes and information.
== Plugins ==
Obsidian.md supports the installation of community plugins that extend the app's initial functionality.
=== Recommended plugins ===
# Dataview – Allows us to treat a vault as a database, querying and visualizing information from notes and files.
# BreadCrumbs – Adds link types and notes hierarchy.
# Juggl – Create mindmaps based on your notes and customize their looks with CSS and internal styling features.
=== Plugin installation ===
# Open Settings – the button is in the bottom-left corner of the application.
# Choose 'Community Plugins' from the 'Options' clause.
# Switch 'Safe Mode' to OFF and confirm it.
# Click 'Browse Community Plugins'.
# Find the plugin.
# Click 'Install'.
# Go back to 'Community Plugins' submenu.
# In the bottom section turn on the newly installed plugin.
== Folding vs Tagging and Linking ==
Simple folder structure is sufficient, when it comes to organizing data in nonoverlapping groups. It is enough to have just a couple of folders in your photogallery, for example. But in OSINT it is important to have a more sophisticated structure.
=== Tagging ===
Tagging adds structure because a piece of data can have several tags, as opposed to folders, which can only have one organizing unit per file.
==== Tag structure example: ====
# #people #processes #technology (part targeted)
# #primary #supportive #irrelevant (importance)
# #finished #unfinished (state of note/file)
# #web #registry #socialengineering (means of getting the information)

=== Linking ===
Linking enables the creation of relationships between notes and files.
This manner, one note can include connections to other notes and files, making it easier to handle.
For example, if John purchased the domain name legit.com, John's note can be linked to legit.com's note, which contains information about the domain.

==== Link types ====
Using link types opens up even more possibilities. Link types are included in Breadcrumbs Plugin for Obsidian.md.
In the aforementioned situation of John and legit.com, John is the domain's owner, thus, the domain is John's asset. These are called types of relations.
If it is later revealed that John purchased another domain name - fake.com – the new domain can be connected back to John.
This structure will be displayed in the notes by creating two relations of John's ownership:
# John – owner of legit.com, fake.com
# legit.com – asset of Johh, relative of fake.com
# fake.com – asset of John, relative of legit.com

== Dataview plugin ==
Dataview is, first and foremost, a data index, so it supports relatively rich methods of adding metadata to your knowledge base.
Dataview tracks information at the markdown page and markdown task levels, with each page/task able to contain an arbitrary number of complex (numbers, objects, lists) fields.
Each field is a named value of a specific type (like "number" or "text").

=== Example of notes with arbitrary metadata and a tag: ===

jason_statham.md
<pre>
---
name: Jason Statham
salary: 7500
department: Cyber Forensics
notes: [
"Potential phishing target",
"Mother has stage T4 cancer"
]
---
#employee
</pre>
bruce_lee.md
<pre>
---
name: Bruce Lee
salary: 8000
department: Developer Operations
notes: []
---
</pre>

== Querying dataview data ==
=== Options for querying data: ===
# Dataview query language
# Dataview Javascript API

Both can be used to, as an example, render a table from jason_statham.md and bruce_lee.md with four columns:
# File – contains a link to the file
# Name – metadata 'name'
# Salary – metadata 'salary'
# Department – metadata 'department'
It can also be sorted by 'salary'.

==== Dataview query language ====

The dataview query language is a straightforward, organized custom query language that enables you to quickly create views from data.
It enables the following:
* Retrieve pages related with tags, folders, and links, among other things.
* Simple actions on fields, such as comparison, existence checks, and so on, can be used to filter notes/data.
* Sorting results according to their fields.

The query language is capable of generating the view kinds, which are detailed below:

* TABLE: The standard view type; one row for each data point, with multiple columns of field data.
* LIST: A list of pages that correspond to the query. Each page can have a single linked value.
* TASK: A collection of tasks whose pages correspond to the specified query.
To query data with Dataview Query Language the 'dataview' language specification for a codeblock is used.

===== Example result of a data query =====
[[File:Dont_know_how_to_embed_images_yet]]
The queries leading to this result are listed below.
===== The general format of queries: =====
<pre>
```dataview
TABLE|LIST|TASK <field> [AS "Column Name"], <field>, ..., <field>
FROM <source> (like #tag or "folder")
WHERE <expression> (like 'field = value')
SORT <expression> [ASC/DESC] (like 'field ASC')
```
</pre>
==== Example with jason_statham.md and bruce_lee.md ====
<pre>
```dataview
TABLE name as "Name", salary as "Salary", department as "Department"
FROM #employee
SORT salary ASC
```
</pre>
=== Dataview Javascript API ===
The Dataview JavaScript API allows arbitrary JavaScript to be executed with access to the dataview indices and query engine, which is useful for complex views or interoperability with other plugins.
To query data with Dataview Javascript API the 'dataviewjs' language specification for a codeblock is used.
The API is accessible via the implicitly provided dv (or dataview) variable, which allows you to query for data, render HTML, and configure the view.
==== Example with jason_statham.md and bruce_lee.md ====
<pre>
```dataviewjs
let employees = dv.pages("#employee")
.sort(emp => emp.salary, "asc")
.map(emp => [emp.file.link, emp.name, emp.salary, emp.department])
dv.table(["File", "Name", "Salary", "Department"], employees)
```
</pre>

== Conclusion ==
There is no defined standard for OSINT data organization, because the data may come in different forms, including, but not limited to, web-pages, paper documents, online calendars, video and audio recordings. Due to this, it is nearly impossible to create a convenient tool for all use cases. If the operation is big enough, it might be feasible to create a devoted web application that stores all necessary data in a database. However, since OSINT itself it usually a highly confidential activity, publishing the application in Clear Web is a privacy and a security risk.

OSINT – theory and practice

2022-04-24T13:07:24Z

Mgoroz:

= Framework =
The framework for Open source intelligence is both sources for the searched data and ways to obtain and analyze it. The whole framework depends on the goal and capacities of the research in which the OSINT method is utilized. This means that two OSINT projects with different goals most likely would have completely different frameworks. This can even happen for researches with the same goals. For example, this year an emergence of OSINT techniques in tracking of the latest developments in Ukraine war can be observed.

While having the same general goal — looking as deep as possible into the fog of war — different researchers have their own subgoals, i.e. tracking weaponry losses like Oryx project or tracking movements of armies like Conflict Intelligence Team. In addition, the researchers use wide variety of methods from analyzing of social media publications, photos and videos, to using plane- and ship-tracking services and even traffic functions of Google Maps to track movement of the armies.
== Goal of research ==
In many cases OSINT research starts with a certain goal and this goal shapes the whole framework: which data needs to be acquired, where it is searched and how it is analyzed. However, there are cases when the framework is defined by the data. This can happen after different leaks of documents, personal information or any other data. Examples for this can be the whole WikiLeaks project, where investigators worked with leaked secret documents, or investigations that followed the leak of Yandex’s food delivery service clients, which among other things allowed to uncover properties owned by Putin’s close circle.

= Data organization =
While an OSINT enthusiast may be adept at data collection, he or she will never develop the necessary data organization skills and tools to become a true professional.
There are numerous methods for storing data, including basic text files or notes.
However, using text files is impractical, as when there is a large amount of data, it becomes unmanageable.
Features desirable for OSINT data management include the ability to export and backup, as well as visualize data.
== Examples of software for OSINT data organization and their disadvantages: ==
* Simple Notes Apps (unmanageable when dealing with a large amount of data)
* Evernote (useful when paid for)
* Notion (notes cannot be accessed offline)
* Joplin (inconvenient organization for large projects)
* Obsidian.md Obsidian.md (a bit tricky to master)
== Obsidian.md ==
Obsidian.md, being perplexive in comparison to simple notes application, contains all the desirable features. It is a cross-platform, free application for organizing notes stored in markup (.md) files.
Notes and files are stored on a user's computer, and there is also a premium feature for syncing, which is superfluous given that backups using any online storage service, Syncthing software, or Git.
Given that OSINT specialists often work in teams, it is recommended to store the data in a Git repository in order to retain a history of modifications and increase collaboration capability.
== Vaults ==
Obsidian.md contains all data in what are referred to as "Vaults."
A vault is a project that houses all of it's associated notes and information.
== Plugins ==
Obsidian.md supports the installation of community plugins that extend the app's initial functionality.
=== Recommended plugins ===
# Dataview – Allows us to treat a vault as a database, querying and visualizing information from notes and files.
# BreadCrumbs – Adds link types and notes hierarchy.
# Juggl – Create mindmaps based on your notes and customize their looks with CSS and internal styling features.
=== Plugin installation ===
# Open Settings – the button is in the bottom-left corner of the application.
# Choose 'Community Plugins' from the 'Options' clause.
# Switch 'Safe Mode' to OFF and confirm it.
# Click 'Browse Community Plugins'.
# Find the plugin.
# Click 'Install'.
# Go back to 'Community Plugins' submenu.
# In the bottom section turn on the newly installed plugin.
== Folding vs Tagging and Linking ==
Simple folder structure is sufficient, when it comes to organizing data in nonoverlapping groups. It is enough to have just a couple of folders in your photogallery, for example. But in OSINT it is important to have a more sophisticated structure.
=== Tagging ===
Tagging adds structure because a piece of data can have several tags, as opposed to folders, which can only have one organizing unit per file.
==== Tag structure example: ====
# #people #processes #technology (part targeted)
# #primary #supportive #irrelevant (importance)
# #finished #unfinished (state of note/file)
# #web #registry #socialengineering (means of getting the information)

=== Linking ===
Linking enables the creation of relationships between notes and files.
This manner, one note can include connections to other notes and files, making it easier to handle.
For example, if John purchased the domain name legit.com, John's note can be linked to legit.com's note, which contains information about the domain.

==== Link types ====
Using link types opens up even more possibilities. Link types are included in Breadcrumbs Plugin for Obsidian.md.
In the aforementioned situation of John and legit.com, John is the domain's owner, thus, the domain is John's asset. These are called types of relations.
If it is later revealed that John purchased another domain name - fake.com – the new domain can be connected back to John.
This structure will be displayed in the notes by creating two relations of John's ownership:
# John – owner of legit.com, fake.com
# legit.com – asset of Johh, relative of fake.com
# fake.com – asset of John, relative of legit.com

== Dataview plugin ==
Dataview is, first and foremost, a data index, so it supports relatively rich methods of adding metadata to your knowledge base.
Dataview tracks information at the markdown page and markdown task levels, with each page/task able to contain an arbitrary number of complex (numbers, objects, lists) fields.
Each field is a named value of a specific type (like "number" or "text").

=== Example of notes with arbitrary metadata and a tag: ===

jason_statham.md
<pre>
---
name: Jason Statham
salary: 7500
department: Cyber Forensics
notes: [
"Potential phishing target",
"Mother has stage T4 cancer"
]
---
#employee
</pre>
bruce_lee.md
<pre>
---
name: Bruce Lee
salary: 8000
department: Developer Operations
notes: []
---
</pre>

== Querying dataview data ==
=== Options for querying data: ===
# Dataview query language
# Dataview Javascript API

Both can be used to, as an example, render a table from jason_statham.md and bruce_lee.md with four columns:
# File – contains a link to the file
# Name – metadata 'name'
# Salary – metadata 'salary'
# Department – metadata 'department'
It can also be sorted by 'salary'.

==== Dataview query language ====

The dataview query language is a straightforward, organized custom query language that enables you to quickly create views from data.
It enables the following:
* Retrieve pages related with tags, folders, and links, among other things.
* Simple actions on fields, such as comparison, existence checks, and so on, can be used to filter notes/data.
* Sorting results according to their fields.

The query language is capable of generating the view kinds, which are detailed below:

* TABLE: The standard view type; one row for each data point, with multiple columns of field data.
* LIST: A list of pages that correspond to the query. Each page can have a single linked value.
* TASK: A collection of tasks whose pages correspond to the specified query.
To query data with Dataview Query Language the 'dataview' language specification for a codeblock is used.

===== Example result of a data query =====
[[File:Dont_know_how_to_embed_images_yet]]
The queries leading to this result are listed below.
===== The general format of queries: =====
<pre>
```dataview
TABLE|LIST|TASK <field> [AS "Column Name"], <field>, ..., <field>
FROM <source> (like #tag or "folder")
WHERE <expression> (like 'field = value')
SORT <expression> [ASC/DESC] (like 'field ASC')
```
</pre>
==== Example with jason_statham.md and bruce_lee.md ====
<pre>
```dataview
TABLE name as "Name", salary as "Salary", department as "Department"
FROM #employee
SORT salary ASC
```
</pre>
=== Dataview Javascript API ===
The Dataview JavaScript API allows arbitrary JavaScript to be executed with access to the dataview indices and query engine, which is useful for complex views or interoperability with other plugins.
To query data with Dataview Javascript API the 'dataviewjs' language specification for a codeblock is used.
The API is accessible via the implicitly provided dv (or dataview) variable, which allows you to query for data, render HTML, and configure the view.
==== Example with jason_statham.md and bruce_lee.md ====
<pre>
```dataviewjs
let employees = dv.pages("#employee")
.sort(emp => emp.salary, "asc")
.map(emp => [emp.file.link, emp.name, emp.salary, emp.department])
dv.table(["File", "Name", "Salary", "Department"], employees)
```
</pre>

== Conclusion ==
There is no defined standard for OSINT data organization, because the data may come in different forms, including, but not limited to, web-pages, paper documents, online calendars, video and audio recordings. Due to this, it is nearly impossible to create a convenient tool for all use cases. If the operation is big enough, it might be feasible to create a devoted web application that stores all necessary data in a database. However, since OSINT itself it usually a highly confidential activity, publishing the application in Clear Web is a privacy and a security risk.

OSINT – theory and practice

2022-04-24T13:07:07Z

Mgoroz: Goal of research

= Framework =
The framework for Open source intelligence is both sources for the searched data and ways to obtain and analyze it. The whole framework depends on the goal and capacities of the research in which the OSINT method is utilized. This means that two OSINT projects with different goals most likely would have completely different frameworks. This can even happen for researches with the same goals. For example, this year an emergence of OSINT techniques in tracking of the latest developments in Ukraine war can be observed.

While having the same general goal — looking as deep as possible into the fog of war — different researchers have their own subgoals, i.e. tracking weaponry losses like Oryx project or tracking movements of armies like Conflict Intelligence Team. In addition, the researchers use wide variety of methods from analyzing of social media publications, photos and videos, to using plane- and ship-tracking services and even traffic functions of Google Maps to track movement of the armies.
= Goal of research =
In many cases OSINT research starts with a certain goal and this goal shapes the whole framework: which data needs to be acquired, where it is searched and how it is analyzed. However, there are cases when the framework is defined by the data. This can happen after different leaks of documents, personal information or any other data. Examples for this can be the whole WikiLeaks project, where investigators worked with leaked secret documents, or investigations that followed the leak of Yandex’s food delivery service clients, which among other things allowed to uncover properties owned by Putin’s close circle.

= Data organization =
While an OSINT enthusiast may be adept at data collection, he or she will never develop the necessary data organization skills and tools to become a true professional.
There are numerous methods for storing data, including basic text files or notes.
However, using text files is impractical, as when there is a large amount of data, it becomes unmanageable.
Features desirable for OSINT data management include the ability to export and backup, as well as visualize data.
== Examples of software for OSINT data organization and their disadvantages: ==
* Simple Notes Apps (unmanageable when dealing with a large amount of data)
* Evernote (useful when paid for)
* Notion (notes cannot be accessed offline)
* Joplin (inconvenient organization for large projects)
* Obsidian.md Obsidian.md (a bit tricky to master)
== Obsidian.md ==
Obsidian.md, being perplexive in comparison to simple notes application, contains all the desirable features. It is a cross-platform, free application for organizing notes stored in markup (.md) files.
Notes and files are stored on a user's computer, and there is also a premium feature for syncing, which is superfluous given that backups using any online storage service, Syncthing software, or Git.
Given that OSINT specialists often work in teams, it is recommended to store the data in a Git repository in order to retain a history of modifications and increase collaboration capability.
== Vaults ==
Obsidian.md contains all data in what are referred to as "Vaults."
A vault is a project that houses all of it's associated notes and information.
== Plugins ==
Obsidian.md supports the installation of community plugins that extend the app's initial functionality.
=== Recommended plugins ===
# Dataview – Allows us to treat a vault as a database, querying and visualizing information from notes and files.
# BreadCrumbs – Adds link types and notes hierarchy.
# Juggl – Create mindmaps based on your notes and customize their looks with CSS and internal styling features.
=== Plugin installation ===
# Open Settings – the button is in the bottom-left corner of the application.
# Choose 'Community Plugins' from the 'Options' clause.
# Switch 'Safe Mode' to OFF and confirm it.
# Click 'Browse Community Plugins'.
# Find the plugin.
# Click 'Install'.
# Go back to 'Community Plugins' submenu.
# In the bottom section turn on the newly installed plugin.
== Folding vs Tagging and Linking ==
Simple folder structure is sufficient, when it comes to organizing data in nonoverlapping groups. It is enough to have just a couple of folders in your photogallery, for example. But in OSINT it is important to have a more sophisticated structure.
=== Tagging ===
Tagging adds structure because a piece of data can have several tags, as opposed to folders, which can only have one organizing unit per file.
==== Tag structure example: ====
# #people #processes #technology (part targeted)
# #primary #supportive #irrelevant (importance)
# #finished #unfinished (state of note/file)
# #web #registry #socialengineering (means of getting the information)

=== Linking ===
Linking enables the creation of relationships between notes and files.
This manner, one note can include connections to other notes and files, making it easier to handle.
For example, if John purchased the domain name legit.com, John's note can be linked to legit.com's note, which contains information about the domain.

==== Link types ====
Using link types opens up even more possibilities. Link types are included in Breadcrumbs Plugin for Obsidian.md.
In the aforementioned situation of John and legit.com, John is the domain's owner, thus, the domain is John's asset. These are called types of relations.
If it is later revealed that John purchased another domain name - fake.com – the new domain can be connected back to John.
This structure will be displayed in the notes by creating two relations of John's ownership:
# John – owner of legit.com, fake.com
# legit.com – asset of Johh, relative of fake.com
# fake.com – asset of John, relative of legit.com

== Dataview plugin ==
Dataview is, first and foremost, a data index, so it supports relatively rich methods of adding metadata to your knowledge base.
Dataview tracks information at the markdown page and markdown task levels, with each page/task able to contain an arbitrary number of complex (numbers, objects, lists) fields.
Each field is a named value of a specific type (like "number" or "text").

=== Example of notes with arbitrary metadata and a tag: ===

jason_statham.md
<pre>
---
name: Jason Statham
salary: 7500
department: Cyber Forensics
notes: [
"Potential phishing target",
"Mother has stage T4 cancer"
]
---
#employee
</pre>
bruce_lee.md
<pre>
---
name: Bruce Lee
salary: 8000
department: Developer Operations
notes: []
---
</pre>

== Querying dataview data ==
=== Options for querying data: ===
# Dataview query language
# Dataview Javascript API

Both can be used to, as an example, render a table from jason_statham.md and bruce_lee.md with four columns:
# File – contains a link to the file
# Name – metadata 'name'
# Salary – metadata 'salary'
# Department – metadata 'department'
It can also be sorted by 'salary'.

==== Dataview query language ====

The dataview query language is a straightforward, organized custom query language that enables you to quickly create views from data.
It enables the following:
* Retrieve pages related with tags, folders, and links, among other things.
* Simple actions on fields, such as comparison, existence checks, and so on, can be used to filter notes/data.
* Sorting results according to their fields.

The query language is capable of generating the view kinds, which are detailed below:

* TABLE: The standard view type; one row for each data point, with multiple columns of field data.
* LIST: A list of pages that correspond to the query. Each page can have a single linked value.
* TASK: A collection of tasks whose pages correspond to the specified query.
To query data with Dataview Query Language the 'dataview' language specification for a codeblock is used.

===== Example result of a data query =====
[[File:Dont_know_how_to_embed_images_yet]]
The queries leading to this result are listed below.
===== The general format of queries: =====
<pre>
```dataview
TABLE|LIST|TASK <field> [AS "Column Name"], <field>, ..., <field>
FROM <source> (like #tag or "folder")
WHERE <expression> (like 'field = value')
SORT <expression> [ASC/DESC] (like 'field ASC')
```
</pre>
==== Example with jason_statham.md and bruce_lee.md ====
<pre>
```dataview
TABLE name as "Name", salary as "Salary", department as "Department"
FROM #employee
SORT salary ASC
```
</pre>
=== Dataview Javascript API ===
The Dataview JavaScript API allows arbitrary JavaScript to be executed with access to the dataview indices and query engine, which is useful for complex views or interoperability with other plugins.
To query data with Dataview Javascript API the 'dataviewjs' language specification for a codeblock is used.
The API is accessible via the implicitly provided dv (or dataview) variable, which allows you to query for data, render HTML, and configure the view.
==== Example with jason_statham.md and bruce_lee.md ====
<pre>
```dataviewjs
let employees = dv.pages("#employee")
.sort(emp => emp.salary, "asc")
.map(emp => [emp.file.link, emp.name, emp.salary, emp.department])
dv.table(["File", "Name", "Salary", "Department"], employees)
```
</pre>

== Conclusion ==
There is no defined standard for OSINT data organization, because the data may come in different forms, including, but not limited to, web-pages, paper documents, online calendars, video and audio recordings. Due to this, it is nearly impossible to create a convenient tool for all use cases. If the operation is big enough, it might be feasible to create a devoted web application that stores all necessary data in a database. However, since OSINT itself it usually a highly confidential activity, publishing the application in Clear Web is a privacy and a security risk.

OSINT – theory and practice

2022-04-24T13:06:21Z

Mgoroz:

OSINT – theory and practice

2022-04-24T13:05:05Z

Mgoroz: framework

E-SPEAIT Participants

2022-01-30T21:55:39Z

Mgoroz:

Everybody should add his/her name and blog address here.

If you log in with your Uni-ID, you should have the 'edit' tab at the top of this page. Alternatively, you are allowed to e-mail the address directly to the lecturer (but this would keep others from reading - and possibly contributing to, by commenting - your blog), therefore publishing is preferred.

Attention: if you see that the list is already long, please add yourself to the '''bottom''' of it. Otherwise, later people will be really hard to spot.

Lecturerː
* Kaido Kikkas, https://jora.kakupesa.net/ (sorry, in Estonian only)

Students:
* Tashi Kamlaldin Rwalshrangpa https://www.blogger.com/profile/14026800162708146439
* Maria Logberg, https://mlogberg.wordpress.com/
* Semen Diev https://cybersecurityinfospeedrun.blogspot.com/
* Andres Naruson https://tln-cyberstalker.blogspot.com/
* Aleksandr Voronkov, https://cyberkotyara.blogspot.com/
* Johannes Kodumäe https://thisismyblogaboutcybersecurity.wordpress.com/
* Grzegorz Kmita, https://otisthescribe.wordpress.com/
* Anton Ostashkov, https://anosta1147.blogspot.com/
* Vladislav Suprun, https://vlsupr.blogspot.com/
* Can Çağlar, https://injapanheartsurgeonnumber1steadyhand.wordpress.com/
* Izolde Springe, https://cyberice3.wordpress.com/
* Ezel Erguden https://blog60239333.wordpress.com
* Sanan Mammadli https://taltech-sananm.blogspot.com/
* Alejandro Ballesteros Perez, https://ballesterosalx.wordpress.com/
* Farid Azizov https://cyber-faaziz.blogspot.com
* Artyom Davydik https://cyberkotleta.wordpress.com/
* Lorenzo Cavallini, https://locava.wordpress.com/
* Louis Alvin, https://speait.louis-alvin.eu/
* Helena Veebel, https://helenablog1.blogspot.com
* Nicoleta Petrea https://addicttech-nicoleta.blogspot.com/
* Aleksei Bahmatov, https://cybermonki.wordpress.com/
* Oskar Pikkov, https://r00m641a.blogspot.com/
* Rasmus Reigo, https://rareig.wordpress.com/
* Ilya Nikolaev, https://itisagoodidea.blogspot.com/
* Benedek Matveev, https://bematv.wordpress.com/
* Farkas Pongrácz, https://fapong.wordpress.com/
* Risto Remmel, https://riremm.blogspot.com/
* Pavel Rotov, https://pavelrotov-speait.blogspot.com/
* Nadine Jungermann, https://e-speait-nadine.blogspot.com/
* Sofia Bermudez : https://speaitsofia.wordpress.com
* Edvin Toome https://edvintoome.blogspot.com/
* Bendegúz Koszticsák https://bekosz.blogspot.com/
* Margus Valdre https://itobsevatoorium.blogspot.com/
* Akimbek Kurlys https://kazakhsecurity.wordpress.com/akimbek-kurlyss-blog/
* Artur Lykov https://roomno40.blogspot.com/
* Roman Krutsko https://siidirom.blogspot.com/
* Dmitri Trubetskoi https://zxclord.blogspot.com/
* Luca Maddaleno https://www.lucamaddaleno.me/ ([https://www.lucamaddaleno.me/feed rss feed])
* Aleksandrs Rimlins https://alexrimlin.livejournal.com/
* Hannes Kraavi https://hannesk2022.blogspot.com/
* Vladyslava Shekula https://vladyslavashblog.blogspot.com/
* Denis Shadrin https://denissh2022.livejournal.com/
* Phasha Davrishev https://phdavr.wordpress.com/
* Georgi Tarassov https://kot25blog.blogspot.com/
* Mark Samoilov https://msamoittu.blogspot.com/
* Aldous Waters https://aldouswaters.livejournal.com/
* Jáchym Líva https://livajach.wordpress.com/
* Adetunji Adeyimika https://wordpress.com/view/adetunjiadeyimika.wordpress.com
* Rauf Gozal https://raufgozal.blogspot.com
* Orkhan Hasanzade https://orkhanhasanzade.blogspot.com/
* Allen-Kristjan Päll https://tamrextulekustuti.blogspot.com/
* Yaroslav Bilobrov, https://cybergnida.blogspot.com/
* Daniil Lemberg, https://dlember068.livejournal.com
* Maksim Gorozhanko, https://mgorozhanko.wordpress.com

* The OPML file with the RSS feeds for the forum and all the blogs will be available when the course gets underway and all participants are onboard.

[[E-SPEAIT | Back to the course page]]

[[Category:ITSPEA]]