As artificial intelligence (AI) technologies, including ChatGPT, become more prevalent, concerns about their potential misuse in cybercriminal activities have grown. This blog delves into the effectiveness of ChatGPT’s safety measures, the risk of AI misuse by malicious actors, and the limitations of current AI models.

ChatGPT, an advanced language model developed by OpenAI, gained widespread adoption at a rapid pace. However, this popularity raised concerns about its potential misuse, particularly in the creation of malware and other harmful software. To address these concerns, OpenAI implemented safety filters designed to detect and prevent malicious use of ChatGPT.

Testing ChatGPT’s Malware Coding Potential

CyberArk, in a recent study, tested ChatGPT’s ability to create malware while evading detection and bypassing policies. They explored the limitations of ChatGPT’s security filters and found that with specific prompts, the model could generate code snippets related to malware development.

For testing purposes, the model was provided with prompts based on tactics, techniques, and procedures (TTP) from the MITRE ATT&CK framework, a valuable resource for understanding potential risks associated with AI technologies. These prompts aimed to generate code snippets for specific tasks related to malware development.

While the model could generate code based on prompts, certain workarounds were necessary. For example, the model might refuse to generate code if a prompt directly mentioned a malicious program. However, by tweaking the prompt, attackers could obtain the desired output.

Additionally, ChatGPT demonstrated the ability to learn and adapt to user preferences over time. For instance, it learned that users preferred saving information to a text file and incorporated this into subsequent prompts.

Limitations of ChatGPT in automated malware generation

Despite its capabilities, ChatGPT has limitations when it comes to automated malware creation:

1. Code modification required

All code snippets generated by ChatGPT needed modifications to function properly. These modifications ranged from minor changes, such as renaming paths and URLs, to substantial edits involving code logic and bug fixes.

2. Success rate

Approximately 48% of tested code snippets failed to deliver the requested output. Only 42% fully succeeded, and 10% partially succeeded. This highlights the model’s current limitations in accurately interpreting and executing complex coding requests.

3. Error rate

Of all tested codes, 43% contained errors, even in code snippets that successfully delivered the desired output. This suggests potential issues in the model’s error-handling capabilities or code-generation logic.

4. MITRE techniques breakdown

Success rates varied depending on the MITRE technique used in the prompt. Discovery techniques had a 77% success rate, while Defense Evasion techniques had a lower success rate of 20%. This discrepancy may result from the complexity of certain techniques or the model’s training data.

5. Hallucination and truncated output

ChatGPT is susceptible to generating inaccurate information, known as hallucination, which can be frustrating for users. Additionally, it may produce incomplete or truncated code, rendering it nonfunctional.

6. Lack of control over certain variables

ChatGPT cannot generate custom paths, file names, IP addresses, or command and control (C&C) details, limiting user control over these variables. Specifying them in prompts is possible but not scalable for complex applications.

7. Limited obfuscation and encryption

While ChatGPT can perform basic obfuscation, heavy obfuscation and code encryption are challenging for these models. Custom-made techniques may be difficult for ChatGPT to understand and implement, and the obfuscation it generates may not be strong enough to evade detection.

Balancing potential and pitfalls

While ChatGPT and similar AI models have potential in code generation, it is essential to recognize their limitations. They can simplify initial coding steps, especially for those familiar with the malware creation process. However, full automation without significant prompt engineering, error handling, model fine-tuning, and human supervision is not yet feasible.

The adaptability of ChatGPT to user preferences is promising, enhancing efficiency in code generation. Nonetheless, human oversight remains crucial. These models may facilitate the creation of a large pool of code snippets for different malware families, potentially enhancing evasion capabilities, but current limitations provide some reassurance that misuse is not yet fully feasible.

As AI technologies evolve, it is crucial to maintain a strong focus on safety and ethical use. While ChatGPT shows promise, particularly in code generation, it is not a fully automated solution for malware creation. Striking a balance between harnessing AI’s potential and mitigating its pitfalls is essential to ensure these tools serve as assets rather than instruments of harm.

As artificial intelligence (AI) technologies, including ChatGPT, become more prevalent, concerns about their potential misuse in cybercriminal activities have grown. This blog delves into the effectiveness of ChatGPT’s safety measures, the risk of AI misuse by malicious actors, and the limitations of current AI models.

ChatGPT, an advanced language model developed by OpenAI, gained widespread adoption at a rapid pace. However, this popularity raised concerns about its potential misuse, particularly in the creation of malware and other harmful software. To address these concerns, OpenAI implemented safety filters designed to detect and prevent malicious use of ChatGPT.

Testing ChatGPT’s Malware Coding Potential

CyberArk, in a recent study, tested ChatGPT’s ability to create malware while evading detection and bypassing policies. They explored the limitations of ChatGPT’s security filters and found that with specific prompts, the model could generate code snippets related to malware development.

For testing purposes, the model was provided with prompts based on tactics, techniques, and procedures (TTP) from the MITRE ATT&CK framework, a valuable resource for understanding potential risks associated with AI technologies. These prompts aimed to generate code snippets for specific tasks related to malware development.

While the model could generate code based on prompts, certain workarounds were necessary. For example, the model might refuse to generate code if a prompt directly mentioned a malicious program. However, by tweaking the prompt, attackers could obtain the desired output.

Additionally, ChatGPT demonstrated the ability to learn and adapt to user preferences over time. For instance, it learned that users preferred saving information to a text file and incorporated this into subsequent prompts.

Limitations of ChatGPT in automated malware generation

Despite its capabilities, ChatGPT has limitations when it comes to automated malware creation:

1. Code modification required

All code snippets generated by ChatGPT needed modifications to function properly. These modifications ranged from minor changes, such as renaming paths and URLs, to substantial edits involving code logic and bug fixes.

2. Success rate

Approximately 48% of tested code snippets failed to deliver the requested output. Only 42% fully succeeded, and 10% partially succeeded. This highlights the model’s current limitations in accurately interpreting and executing complex coding requests.

3. Error rate

Of all tested codes, 43% contained errors, even in code snippets that successfully delivered the desired output. This suggests potential issues in the model’s error-handling capabilities or code-generation logic.

4. MITRE techniques breakdown

Success rates varied depending on the MITRE technique used in the prompt. Discovery techniques had a 77% success rate, while Defense Evasion techniques had a lower success rate of 20%. This discrepancy may result from the complexity of certain techniques or the model’s training data.

5. Hallucination and truncated output

ChatGPT is susceptible to generating inaccurate information, known as hallucination, which can be frustrating for users. Additionally, it may produce incomplete or truncated code, rendering it nonfunctional.

6. Lack of control over certain variables

ChatGPT cannot generate custom paths, file names, IP addresses, or command and control (C&C) details, limiting user control over these variables. Specifying them in prompts is possible but not scalable for complex applications.

7. Limited obfuscation and encryption

While ChatGPT can perform basic obfuscation, heavy obfuscation and code encryption are challenging for these models. Custom-made techniques may be difficult for ChatGPT to understand and implement, and the obfuscation it generates may not be strong enough to evade detection.

Balancing potential and pitfalls

While ChatGPT and similar AI models have potential in code generation, it is essential to recognize their limitations. They can simplify initial coding steps, especially for those familiar with the malware creation process. However, full automation without significant prompt engineering, error handling, model fine-tuning, and human supervision is not yet feasible.

The adaptability of ChatGPT to user preferences is promising, enhancing efficiency in code generation. Nonetheless, human oversight remains crucial. These models may facilitate the creation of a large pool of code snippets for different malware families, potentially enhancing evasion capabilities, but current limitations provide some reassurance that misuse is not yet fully feasible.

As AI technologies evolve, it is crucial to maintain a strong focus on safety and ethical use. While ChatGPT shows promise, particularly in code generation, it is not a fully automated solution for malware creation. Striking a balance between harnessing AI’s potential and mitigating its pitfalls is essential to ensure these tools serve as assets rather than instruments of harm.