From Efficiency to Exposure: AI’s Role in Modern Cybersecurity

The rapid advancement of artificial intelligence (AI) is reshaping the way many tasks are performed, enabling the automation of numerous routine and repetitive activities that have traditionally required human effort. This transformation is reminiscent of the Industrial Revolution of about a century ago, when mechanization revolutionized manual labor and dramatically increased productivity across industries. Just as machines took over physical tasks during that era, AI is now taking over cognitive and digital tasks at an unprecedented speed and scale.

Let’s take a look at the current state: AI tools have significantly amplified the capabilities of malicious actors in the cyber domain. Attackers can leverage AI to automate complex operations such as crafting convincing phishing campaigns, identifying system vulnerabilities, and evading detection systems. This democratization of these advanced cyber tools means that even relatively unskilled individuals (once known as «script kiddies»), can now conduct highly effective attacks with ease, turning them into formidable threats. On the other hand, we have better email filters to protect us from these advanced threats. Additionally, cybersecurity agents around the globe have started incorporating AI to increase their effectiveness.

To conclude: AI has not only enhanced defense mechanisms but also empowered adversaries, escalating the cybersecurity arms race to new levels of intensity.

Through the increase of available tools, we also increase the attack surface. This is similar to how the industrial revolution increased the number of possible accidents. In the case of AI, it is especially important to examine the new attack surface. Currently a lot of old and well-established defensive mechanisms exist, but AI works differently compared to old, predictable programs. They present new opportunities for attacks, mainly the «mind» itself. In this context, «mind» refers to the way an AI works and operates: its algorithms, training data and interactions. New attack methods, such as data poisoning, prompt injection and reverse prompt engineering have never been encountered before. So far, we rely on a variety of mechanisms for defense, but there exist no framework or similar guidelines to help us create a robust level of defense against attackers yet. How can we tell if our system is truly secure?

The «Cost of a Data Breach Report 2025» shows that one in six breaches already uses generative AI. This demonstrates how criminals are implementing new technologies into their existing methods, to increase both frequency and efficiency. On the other hand, 20% of the reported breaches involved the use of shadow AI. This means that tools or applications using AI were used without approval or oversight of the IT department.

In cybersecurity we constantly face new risks and must reevaluate our old approaches and evolve our understanding of the digital world. This is especially true when new and very complex technologies, like AI, are introduced. We cannot assume that our old and well-established security mechanism will help us with AI, since AI operates differently in a fundamental way. Now is the time for cybersecurity to adjust and evolve, so that we secure our old and new technologies from this revolutionary change we are currently witnessing.

The first step to securing our systems is to understand the new threats which are a result of the new technologies. Only then can we stay one step ahead of cybercriminals and preemptively secure our systems from these threats. With that in mind, we will take a detailed look at examples of some of these new risks.

Over the past two years, a contest was held to effectively use AI to analyze large code bases. Contestants had the opportunity to analyze huge amounts of data quickly, scanning thousands of lines of code to find new zero-day vulnerabilities. Each contestant took a different approach, but they all succeeded in finding new exploitable flaws in those codebases. After the results were announced at DefCon 2025, all contestants made their final competition versions available as open source for everyone to use. This raises the question: What if parties with ill intentions use them? While the contest was designed to help everyone find and close vulnerabilities, especially for parties with limited resources like open-source projects, other parties can exploit this software for their own benefit. Open source is an important foundation for many licensed software and can be found everywhere. All of these are now exposed and offer potential for newfound vulnerabilities. The result will be a literal race on who is using the new technology first on the library or repositories and find the vulnerabilities first.

AI is a versatile and helpful tool that can accelerate humanity in many ways. One area where AI shows strength is programming. Code generation is inherently logical, pattern-based, and often built on publicly available datasets. These characteristics make it an ideal domain for AI models, which excel at predicting and assembling code structures. The emergence of terms like «vibe programming», where users rely heavily on AI to generate entire libraries or applications, illustrates how deeply integrated these tools have become in modern development workflows. However, we must raise this question now: «What’s stopping people from using these tools to create malicious software like a virus or a phishing website?» Most mainstream AI platforms implement safety mechanisms, such as content filters and behavioural guardrails, to prevent harmful instructions. But even those are no barriers for a determined person. Alternate options like FraudGPT can be found on the dark web for a fee, which have none of those safety mechanisms and are optimized to assist on illegal activities like malware generation, phishing and social engineering.

The widespread adoption of AI tools has introduced a new class of cybersecurity risks—particularly in the domain of data leakage and model manipulation. One emerging threat is the extraction of sensitive information through interactions with publicly accessible language models.

Mainstream AI platforms often retain user inputs for model improvement. For example, when an employee uses a model like ChatGPT to summarize an internal report, the content may be stored and later incorporated into future training datasets. This creates a latent vulnerability: if proprietary data is stored, it can theoretically be retrieved.

A notable case involved a Samsung employee who uploaded confidential source code to ChatGPT. The data was later extracted, demonstrating how human error, combined with insufficient awareness of data handling practices, can lead to significant breaches. Employees often prioritize efficiency over security, using AI tools without understanding the implications of data persistence and model retraining.

Beyond passive leakage, a more advanced threat is the active circumvention of AI safety mechanisms. Large language models (LLMs) are equipped with guardrails designed to prevent the generation of harmful or unethical content. These include prompt injection detection, jailbreak prevention, and refusal protocols for sensitive queries. However, recent research has demonstrated that these defenses can be bypassed.

The paper «Bypassing Prompt Injection and Jailbreak Detection in LLM Guardrails» outlines how attackers can exploit the model’s own reasoning capabilities to override its safety constraints. By embedding adversarial examples within the input context, often disguised as benign instructions, attackers can induce the model to ignore its guardrails. This technique leverages the model’s in-context learning behavior, allowing it to adopt unsafe instructions without triggering standard detection mechanisms.

Common Evasion Strategies include:

• Prompt obfuscation: Rewriting malicious queries to avoid triggering filters.
• Contextual masking: Hiding harmful instructions within multi-step or nested prompts.
• Roleplay exploits: Framing queries as fictional or hypothetical scenarios to bypass ethical constraints.
• Encoding schemes: Using symbolic or structured language that the model can interpret but guardrails cannot flag.

These techniques allow attackers to extract restricted information, generate exploit code, or simulate phishing attacks, all within the interface of a supposedly secure AI system.

Understanding the dual-use nature of AI is essential. It is no longer sufficient to view AI as merely a neutral productivity enhancer. Artificial intelligence enables cybercriminals by providing them with new attack vectors, and the path for new cybercriminals has never been easier. At the same time, similar tools are used to protect us from these new dangers. This illustrates the intense arms race between cybercriminals and cybersecurity experts.

Companies must recognize that AI is not only a productivity tool but also a potential attack vector. Without proper governance, employee training, and technical safeguards, integrating AI into workflows can introduce systemic vulnerabilities.

Moreover, cybersecurity is becoming increasingly mandatory for all companies in the EU. With new legislation such as the Cyber Resilience Act and the NIS-2 Directive, the EU is pushing for greater cybersecurity and communication to strengthen the digital infrastructure. Additionally, we are seeing the development of AI-specific laws to address open problems. Until then, we must take responsibility for critically examining our infrastructure and solutions and assessing our level of security.