Internet crooks and hackers using AI to aid malicious activities

Large language models

The widespread availability of large language models (LLMs) has given AI a massive boost. LLMs are neural networks made up of hundreds of millions of nodes trained using huge volumes of data gathered from the internet and other sources.

Although the technology is complex, the basic principle is simple: the models underpinning chatbots such as OpenAI's ChatGPT (GPT-3.5/4), Google's Bard (PaLM) and Meta's LLaMA-2 are trained on the vast text resources available on the internet. They are in effect very sophisticated next word predictors, which can hold (or simulate, if you prefer) a conversation by taking previous prompts (context window content) into account when generating replies (in-context learning).

Images and program code

Text-to-image generators [1] such as Stability AI's Stable Diffusion (SDXL), OpenAI's DALL-E and Midjourney work in much the same way. They are typically trained using web images in combination with the associated 'alt' descriptions, 'caption' texts and contextual text.

Finally, there is GitHub Copilot, a product of collaboration between Microsoft and OpenAI. Copilot is a generative programming assistant (text-to-code, code-to-text and code-to-code) trained on the content of GitHub itself, of the Microsoft Developer Network (MSDN) and of StackOverflow. The relationship between text and code is established by analysing things such as the comments in the source code and the relatively well-structured problems and solutions found in the MSDN and StackOverflow.

Prompt engineering

Computer criminals use AI tools in the same way as mainstream programmers and content creators, but for different purposes. So, for example, they'll use Copilot to improve their malware, and to identify vulnerabilities in other people's (open source) code. Or they'll ask ChatGPT to polish up the wording of their phishing messages so as to make them more convincing, or to add organisation-specific or job-related information (effectively combining deepfake technology and social engineering) [1]. Alternatively, they might ask a model trained on internet content to find leaked credentials (like searching for passwords on GitHub or using Google to look for accidentally published documents).

Various security firms have presented proofs-of-concept that demonstrate how ChatGPT can be used to generate polymorphic code [CyberArc, HYAS BlackMamba]. By constantly changing shape, such programs are much better at hiding from endpoint detection and response (EDR) solutions.

All that's needed to make effective use of AI tools is the ability to ask the right questions, a discipline known as prompt engineering. But, if you're struggling, you can simply use another LLM to write you a good prompt. Auto-GPT even goes a step further, dividing a set task into steps and performing them independently by generating new prompts for itself using the replies to earlier prompts. Auto-GPT is therefore more like an independent assistant or agent.

Cat-and-mouse game

AI service providers are now locked in a cat-and-mouse game with the crooks and hackers, adapting their models to prevent them being used for malicious purposes. That involves not only excluding information whose main uses are malicious (e.g. information about how to perpetrate crimes and acts of terror, how to make weapons or how security protocols work), but also filtering for extreme and hateful views, discrimination, violence and child sexual exploitation and other forms of pornography.

Meanwhile, the bad guys are working to sidestep the developers' safeguards by using carefully crafted prompts to trick the models into providing protected information or behave in unintended ways [1].

Malicious AI tools

However, that kind of 'jailbreaking' is increasingly unnecessary. The reason: computer criminals are now developing their own AI tools and offering them to other crooks on a commercial basis.

The development of such sophisticated tools is much easier than many people might think. Numerous models for text-to-text and text-to-code are available on the internet in open-source form. In other words, pre-trained models, supporting datasets and the associated tooling are all out there for anyone to make use of.

Open-source models, datasets and tools

The best-known open-source framework for both training and prompting LLMs is the Transformers Library maintained by Hugging Face [1]. Their Hub serves as a repository for a huge number of models, datasets, front-ends and tools. nVidia, the company that makes the video adapters (GPUs) that are typically used as AI processors, also operates a hub, where models and tools [1] are available.

A widely used open-source dataset for training English-language models is The Pile, a text corpus consisting of a couple of dozen sources. Meanwhile, anyone looking to use 'the whole web' no longer needs their own crawler, because there's now the Common Crawl.

Fine-tuning existing models

For mere mortals and hackers, training a new model from scratch is unaffordable. The task involves putting to work one of the many specialised AI cards (typically nVidia H100 modules [1]: sophisticated video adapters with very large VRAMs and no video output) for several weeks on one or more datasets in the AWS Cloud [1, 2, 3]. The cost of such an exercise can be millions or even tens of millions of euros.

What hackers (and others) do to make the development of a purpose-specific model affordable is to take an existing model and give it further training using special datasets (fine-tuning). After all, a 'pre-trained model' consists of little more than a set of weightings (expressions of the strength of inter-node connections) for the neural network, as configured by the end of the training session. However, there is nothing to stop anyone taking an open-source pre-trained model, and continuing its training by exposing it to additional or alternative datasets. Hence, pre-trained models are also known as checkpoints.

Copilot, for example, is based on the (closed-source) GPT-3 LLM. That generic (foundational) language model was then given further training specific to the task of programming (OpenAI Codex).

Although the publication of pre-trained models in open-source form facilitates abuse, model developers have good reason to publish their work. Indeed, publication can sometimes be essential. Openness is vital for the further development of basic models: if others cannot continue investigating a model's performance and building on the foundations provided, an AI project is dead on completion, scientifically speaking.

Locally installed open-source models

A hacker will typically instal an open-source pre-trained model on their own workstation, so they can then prompt it directly. To make local use of a modern language model like that, all you need is a top-end video card costing a few thousand euros. The advantage of taking that approach is that the prompts and replies are not filtered or supplemented – or saved in an account – as they invariably are if you use an AI service provider.

A hacker can therefore take a suitable pre-trained language model as their starting point, and then give it additional training to suit their own purposes.

As well as using ChatGPT to refine the communications used for phishing and business email compromise (BEC) attacks – possibly after jailbreaking a public AI service – profit-minded hackers have started training their own models and advertising the resulting specialised, malicious chatbots as paid services in dark web marketplaces and hackers' forums.

WormGPT

This summer, 2 such underground services were discovered by investigators. The first is WormGPT [1], based on the GPT-J open-source model, which is similar to GPT-3 and good with English language and program code.

According to the investigators that discovered it, WormGPT is a version of the GPT-J model that has received further training with a variety of malware-related data. The mail messages generated by WormGPT were amazingly good, the discoverers reported. They described it as "ChatGPT without ethical boundaries or limitations." What's more, the model was available to use for only 60 euros a month.

FraudGPT

The second recently discovered malware chatbot is FraudGPT [1], made available via darknet marketplaces and Telegram. Apparently offering greater capabilities than WormGPT, this chatbot also commands a higher price: 200 euros a month.

Info resource, assistant and sparring partner

Such tools are useful mainly for helping would-be black-hat hackers to get started. They are also attractive to people without a strong command of English, and to relatively inexperienced malware programmers [1]. For criminals, AI chatbots serve the same purposes that they do for legitimate users: as information resources, assistants and sparring partners[1].

Late last year, Check Point researchers showed how they could use ChatGPT and Codex to set up a multi-stage attack (involving a phishing mail, an Excel macro and ultimately a reverse shell) without having to write a single line of code themselves.

The FBI has also confirmed that AI can help cybercriminals and terrorists to operate more effectively. Indeed, the agency expects malicious use to increase as AI becomes more widely available. Malicious AI tools like the open-source models re-trained for malware development described above are likely to play a key role in that trend.

DarkBERT

South Korean security company S2W has trained its DarkBERT language model using dark web data [1, 2, 3]. DarkBERT is a version of S2W's RoBERTa language model (based on Google's context-sensitive BERT model), which has been retrained using a 6-gigabyte corpus of dark web data.

The retrained chatbot speaks the language of the dark web, and can therefore be used for the detection of ransomware breaches and threats, as well as for new developments in malware and newly discovered vulnerabilities.

Although the DarkBERT pre-trained model is available on a conditional basis for research purposes only, it seems that malicious actors have managed to get hold of it. The same hacker (group) that created WormGPT and FraudGPT is now offering a malicious version of DarkBERT (as well as a dark web version of Google's Bard, DarkBART). The adapted version can be used not only to prepare phishing/BEC attacks and other forms of social engineering, but also to identify vulnerabilities in software, systems and infrastructures, to produce and distribute malware (ransomware [1, 2]), and to discover 'zero-day' vulnerabilities.

New arms race

The emergence of new AI tools seems to have triggered an 'arms race' like the one between virus developers and virus scanner providers. For example, DarkBERT was originally developed for monitoring the dark web for ransomware breaches and threats, and with a view to detecting developments in malware and newly discovered vulnerabilities. And research is underway to assess the potential of AI chatbots for analysing attacks and drawing up incident response plans [1].

It remains to be seen whether defensive AI tools will prove a match for the AI applications used by internet criminals. However, the fact that such tools are currently barely able to distinguish authentic content from AI-generated content [1] is not an encouraging sign.

"Large-scale use of such technologies is making it increasingly difficult to establish the authenticity and authority of written information, images, videos and audio," according to the NCTV's Cyber Security Assessment Netherlands 2023.

Modern internet security standards

Like the FBI, the NCTV emphasises the importance of taking basic security measures. "Such measures still provide effective protection against many types of cyber-attack," the organisation writes. Against that backdrop, we also regularly seek to raise awareness of the value of infrastructural support for modern internet security standards.

As shown by the following incident – involving another AI modality, namely voice simulation – deepfake technology and spoofing can be combined to produce a convincing imposter scam.

In 2019, scammers managed to swindle a British energy company out of 220,000 euros using an AI-generated voice in combination with a spoofed e-mail message. When the CEO of the British company was asked to arrange an immediate payment to a Hungarian supplier, he believed that the caller was the CEO of the German parent company, whose voice he recognised. After he hung up, the payment details and confirmation were sent by e-mail. Only when the fraudsters called again with a second request did the British executive become suspicious. By then, however, the first payment had been forwarded to Mexico.

Known as 'vishing' (voice phishing), the fraudulent use of audio deepfakes is increasingly commonplace [1, 2]. Scammers use the technique not only for CEO fraud, but also to steal personal information and passwords. An instructive feature of the incident involving the British energy company is that the victim became suspicious when he noticed that the second call came from an Austrian number. The scam involved the combination of vishing, mail spoofing and caller ID spoofing to create an impression of authenticity via multiple channels. The implementation of e-mail security standards such as SPF, DKIM and DMARC would have made the fraud significantly less likely to succeed.