PGI’s Digital Investigations Team brings you the Digital Threat Digest, SOCMINT and OSINT insights into disinformation, influence operations, and online harms.
The potential of AI-generated disinformation and influence operations is picking up fast. An anonymous software engineer called ‘Neapaw’ recently showcased ‘CounterCloud’, a Large Language Model (LLM) project designed to automate a pro-US, anti-Russia, pro-Biden, anti-Trump influence operation in its entirety.
To do this, the LLM continually scrapes a set of predetermined sources such as RT, The Daily Beast, and Sputnik for content. Once it identifies an article containing a narrative it wishes to counter, it generates its argument, attributes it to a fake journalist, and posts it to the sandboxed CounterCloud website. It even creates a populated comment section to make the article appear more authentic. The AI can also carry out this same process on social media, identifying posts and generating responses autonomously. According to the developer, the project ran over two months and only required two engineers and $400 a month – if it wasn’t a sandboxed experiment, that’s an easy cost to recoup with a bit of AdSense on the website. Obviously, the immediate thought here is that a nation-state or malign group (or individual) with significantly more resources could easily scale this and that there might even be a non-sandboxed operation running right now, under our noses. But there are a few things about this video that need challenging, or pulling out of the ‘AI will destroy the world’ flames…
The developers claim that it creates ‘convincing’ disinformation articles 90% of the time, but they do not specify what metrics we are measuring ‘convincing’ against, nor provide any kind of context on the percentage statistics – a particular bugbear of mine. The video goes on to explain how they attempted to get the AI to generate hate speech and conspiracy, but that the AI struggled to understand how to insert these things seamlessly into news-style articles – it’s good, but it’s not that good yet. And, yes, the model was able to generate short social media posts about conspiracy theories and hate speech, but as we’ve said before – AI seems to always fall short of generating any kind of novelty in its creations, and when it comes to digital threats it’s in novelty and unknown unknowns where the threats truly lie.
However, it is important that we don’t just shrug off experiments like this and that we continuously consider the risks posed by LLM to the information environment. We still need to be developing tools to detect, flag, and hopefully even delete harmful AI-generated content across social media. We still need a top-down approach to AI trust and safety, governed by a clear set of regulations for both AI developers and users. And, I think most importantly, we need to advocate for greater public education around AI. Approaches to digital literacy must include teaching people how AI and LLMs operate and how to detect when they are being used – the percentage of how ‘convincing’ these campaigns are is wholly down to how educated we are.
More about Protection Group International's Digital Investigations
Our Digital Investigations Analysts combine modern exploitative technology with deep human analytical expertise that covers the social media platforms themselves and the behaviours and the intents of those who use them. Our experienced analyst team have a deep understanding of how various threat groups use social media and follow a three-pronged approach focused on content, behaviour and infrastructure to assess and substantiate threat landscapes.
Disclaimer: Protection Group International does not endorse any of the linked content.
Tuesday night saw the celebration of a major political event, a commemoration of political stability and continuity: Guy Fawkes Night.
What is a data breach? A data breach occurs when sensitive, protected, or confidential information is accessed, shared, or stolen by an unauthorised person.
In the mid-20th century, Gilbert Ryle threw sand in the eye of Cartesian dualism, calling the idea of a separate mind a 'category mistake' and dubbing it the 'ghost in the machine'—essentially suggesting that Descartes had outed himself as harbouring an imaginary friend.