Recommendations for threat analysis

Recommendation 1 – A stronger focus on methods

In order to address the problems posed by the overreliance on analysts’ judgements and, to a certain extent the SAT, experts propose as an alternative better exploitation of scientific methods, coming from different disciplines, such as behavioural and social sciences, or data science methods.  Computational social science (e.g., the COMSES Network) has, for example, been put forth as a useful resource, which will allow analysts to more accurately model a range of scenarios that may produce surprises. This is, by far, not a new recommendation albeit still a very valid one. Already in 2011, in the ‘Intelligence Analysis for Tomorrow’ report elaborated by the National Research Council the importance of reviewing analytical methods and fostering those built on strong scientific foundations was mentioned as a key recommendation for intelligence analysis.

Recommendation 2 – Communicating probability with numbers instead of words

Research evidence has shown that people are more accurate and coherent when averaging and multiplying probabilities if they were presented numerically rather than linguistically.  Moreover, numeric probabilities have advanced applications, such as supporting Bayesian networks, which can be employed by analysts to pool and revise judgements. They can also be used to exploit accuracy-boosting post-analytic methods, much better than traditional methods used for this purpose (e.g. Analysis of Competing Hypotheses).  Scholars have also argued that numeric estimates can be mathematically recalibrated to correct analytical biases such as underconfidence or overconfidence.

The issue of probability communication is particularly relevant in relation to threat analysis for border security. A 2021 Report on the efficiency of CIRAM has shown that the data sets employed for calculating probability are lacking consistency, as different contributors employ different units of measurement. For example, some states report incidents on a case-by-case basis, while others provide only aggregated data. Moreover, some states create an incident report for each individual, while others create one incident report for several individuals, which means that statistical data is not consistent.

Recommendation 3 – Look inwards

Though difficult to detect weak signals, to the extent that they exist as brute facts can be identified. According to some experts, one way in which this can be achieved is by expanding the category of ‘mysteries’ that we are trying to understand. This could be achieved by fostering creative analysis when it comes to the possible coordinated use of non-military tools in the arsenals of adversaries against societal vulnerabilities.

Recommendation 4 – Using internal and external experts & academic expertise

Some countries employ a wide variety of experts when preparing national risk assessments. For example, in the case of the Netherlands, this is done by a network of analysts working in the field of national security (Dutch Network of Analysts for National Security), who possess multidisciplinary expertise together with external experts from intelligence services, civil services, members of academia, planning agencies, private companies and consultancy firms.  This approach meets at least some of the requirements outlined by some scholars as being key in carrying out good analysis, namely that ideas are being tested by independent clusters of researchers. This would ensure that analysts do not remain prisoners of their preconceptions.  Moreover, as scientists also pursue goals other than purely epistemic ones, it is vital for their ideas and findings to be subject to peer scrutiny.  Therefore, using panels of experts when drafting methodologies and/or in different steps of the analytical process has more than one benefit:

1. It ensures that the methodology and/or analysis is being carried out by individuals with the right knowledge and skillset;
2. It is a successful way of exploiting existing scientific knowledge;
3. It fosters multi and inter-disciplinarity as well as creativity;
4. The fact that Expert Panels are employed to review intermediate versions of the analysis in a quality control cycle is also a good practice in terms of validation of results.

Employing external experts (preferably also some coming from civil society) can also assist with countering the effects of ‘securitisation’, for example by challenging existing methodologies for threat identification and description.

Similarly, experts have also proposed the idea of using quantitative and qualitative datasets, such as those collected by independently funded academic research projects for threat analysis (especially in the field of migration). This could address some of the problems associated with training AI algorithms for migration management.

Recommendation 5 – Contextualize open source information

For open source information to be secure and useful, it needs to be continuously contextualized. The contextualization can be done either by cross-referencing with other sources and/or by using human experts familiar with the topic and the culture under analysis.

Recommendation 6 – Transparency of data sets

When using different technologies for collecting and analysing social media information, there are several good practices in terms of data set transparency, which should be followed:

• Why this particular data set? – It is important for the analyst to grasp the signification of the selected data in relation to the features to be observed;
• What are the limitations of the data? – This refers to data source reliability, data integrity, data set completeness, and representativeness. When it comes to representativeness this should be considered in two ways: representativeness of the sampling taken from a larger network and representativeness for the social network as it exists in the real world;
• What is the context? – This refers to the context of the data before its extraction, and this should be made available to the analysts.

Recommendation 7 – Using advanced visualisation techniques to improve analysis

There is emerging evidence from research that the visual representation of group processes and thinking on complex issues enables larger ideas to be shared. Therefore, visualisations can be a powerful negotiation and consensus-building tool.

Recommendation 8 – Analysing threats as interconnected elements

When carrying out threat analysis it is important to not only describe threats individually but also explain how different threats are interconnected. A good example in this respect can be found in the Lithuanian National Threat Assessment, which in addition to individual threat analysis also considers how different threats are interconnected. However, this is only a first step as more effort needs to be studied to not only point to potential interconnectedness but also include in the threat analysis information on how the link between different threats emerges and what is its impact.

Recommendation 9 – Better exploitation of new technologies for threat analysis

The meetings had with security practitioners within the MIRROR project have shown the need of such practitioners for more automation in the field of threat analysis, especially when it comes to open sources (especially as MIRROR research has shown that open source collection and analysis for border security is still being done manually in many EU countries). This being said practitioners were sceptical about the level of automation that can be achieved in analysis.

There is already a lot of evidence on the type of assistance that machine learning and artificial intelligence can provide to threat analysis. For example, machine learning can help with the classification and clustering of results, as well as the extraction of additional information using natural language processing. One can also employ de-anonymization techniques to connect different identifiers when collecting information on a particular target. Translation technology can also provide analysts with access to information in other languages as well as convert the meaning of more complex language structures, such as abbreviations or phrases.

Recommendation 10 – Using a human security approach

When addressing the issue of ‘securitisation’ in migration management, a number of experts have argued that by employing the threat perspective as a single variable in relation to migration (as it occurs in methodologies such as CIRAM) leads to a perception whereby migration is directly associated with a threat, without recognising the potential positive aspects of this phenomenon, both to the migrants themselves but also to the host societies. Moreover, frameworks such as CIRAM focus exclusively on state security and dismiss considerations related to human security.  As observed by some scholars, irregular migrants, including those with legitimate protection needs, are therefore first and foremost defined through their risk qualities in the technologies – as threats – rather than through their vulnerability and mortality.

The importance of changing the analytical framework is not restricted to the quality of the analysis itself but also to its wider implications, in terms of policy and strategy options made available to policy-makers. A number of authors have issued warnings as to the risk of preserving such a unilateral security-oriented approach to migration analysis. Van Munster (2009) argued that the current analytical framework employed by Frontex subjects migrants to categorical suspicion, making them vulnerable to targeted practices of precautionary risk management; while Ibrahim & Howarth (2018) argue that the type of analyses carried out by Frontex frames migration as a societal and economic problem and a source of fear and anxiety.

A reframing is, therefore, necessary for the way migration (including irregular migration) is considered in the analysis. On one hand, aspects other than security and threat need to be considered when building migration management models; and on the other, in the context of threat analysis, more attention should be given to (a) self-reflection on the predominant paradigm in which migration is considered (that of threat) including through the use of methods aimed at challenging bias and reducing preconceptions and (b) making efforts to examine the well-being of migrants and state security as interconnected phenomena instead of adversarial ones. In the current framework of defining irregular migration as a threat, migrants act as a source of information, which can be exploited to uncover smuggling and other crime-related activities. As such analysts may miss out on the risks associated with them (and which in turn are relevant for the state, societal and individual security), for example, whether they have been victims of trafficking, if they have serious health problems, if they are unaccompanied minors etc.