AI Code Security Risks
Hallucinated dependencies, injection vulnerabilities, and supply chain threats - how AI-generated code is expanding your attack surface
The Expanding Attack Surface from AI Code Generation
AI coding assistants are generating millions of lines of code daily - and a significant portion contains security vulnerabilities that traditional review processes miss. Unlike human-written bugs that follow predictable patterns, AI-generated security flaws are novel, varied, and often hidden behind code that looks perfectly correct.
From hallucinated packages that attackers weaponize to injection vulnerabilities baked into suggestions, AI tools are creating an entirely new threat landscape. Understanding these risks is the first step toward building secure software in the age of AI-assisted development.
The Threat Landscape: 6 Critical AI Security Risks
These six categories represent the most dangerous security risks introduced by AI coding assistants. The first five are measurable in generated code today; the sixth is an anticipated escalation rather than a documented campaign, and is marked as such.
Hallucinated Dependencies
The risk: AI models suggest importing packages that do not exist. These "hallucinated" package names become attack vectors when malicious actors register them on npm, PyPI, or other registries with backdoored code.
A developer trusts the AI suggestion, runs npm install, and unknowingly installs malware. Across 576,000 code samples from 16 models, researchers measured package hallucination rates of at least 5.2% for commercial models and 21.7% for open-source models (Spracklen et al., USENIX Security 2025).
Severity: Critical - Direct remote code execution path
Injection Vulnerabilities
The risk: AI frequently generates code that concatenates user input directly into SQL queries, shell commands, or HTML output without proper sanitization, parameterized queries, or escaping.
Veracode's 2025 GenAI Code Security Report tested code from more than 100 large language models across Java, JavaScript, Python and C# and found AI-generated code introduced risky security flaws in 45% of tests. The code looks clean and functional but lacks the defensive patterns a security-conscious developer would include.
Severity: Critical - OWASP Top 10 #3 (Injection)
Hardcoded Secrets
The risk: AI models include example API keys, default passwords, placeholder tokens, and connection strings directly in generated code. Developers copy these into production without replacing them with proper secret management.
Worse, some AI-generated "example" keys are actual leaked credentials from training data. These get committed to repositories and exposed through version history even after removal.
Severity: High - Credential exposure and unauthorized access
Insecure Deserialization
The risk: AI models commonly suggest using eval(), pickle.loads(), yaml.load() without safe loaders, and other unsafe parsing patterns that allow attackers to execute arbitrary code through crafted input.
These patterns are prevalent in AI training data because older tutorials and Stack Overflow answers used them before the security community recognized the risks.
Severity: High - Remote code execution via crafted payloads
Broken Authentication Patterns
The risk: AI generates authentication code with weak hashing algorithms (MD5, SHA-1), missing rate limiting, improper session management, client-side-only validation, and JWT implementations without proper signature verification.
AI models learn from the vast majority of auth code online - most of which is outdated or insecure. The generated code "works" for login/logout but fails under adversarial conditions.
Severity: Critical - Account takeover and privilege escalation
Supply Chain Poisoning
The risk (anticipated, not yet documented at scale): attackers deliberately poison AI training data by publishing malicious code patterns, vulnerable tutorials, and backdoored libraries that models later learn to recommend.
This is the logical next step for supply chain attacks: instead of compromising one package, influence the models themselves. We are flagging it as a threat model to design against, not reporting a campaign that has been observed and attributed - and this page will not present it as one until it has been.
Severity: Critical - Scalable attack across entire ecosystems
How AI Code Security Fails in Practice
The first item below is documented in peer-reviewed research and is cited to it. The two that follow are representative failure patterns - composites of how these flaws reach production, not attributed accounts of a specific company's breach. They are labelled as such because a security page that cannot name its sources should not imply that it can.
Package Hallucination and Slopsquatting (documented, USENIX Security 2025)
Researchers established that code-generating LLMs routinely recommend non-existent npm and PyPI packages, and that the same wrong name recurs reliably enough to be weaponised. An attacker registers the hallucinated name; a developer follows the AI suggestion, runs the install command, and executes the attacker's post-install script. Seth Larson of the Python Software Foundation named the technique slopsquatting in April 2025.
Measured scale: 205,474 unique hallucinated package names across 576,000 code samples generated by 16 LLMs in two languages, with hallucination rates of at least 5.2% for commercial models and 21.7% for open-source models.
Source: Spracklen, Wijewickrama, Sakib, Maiti, Viswanath and Jadliwala, "We Have a Package for You!", 34th USENIX Security Symposium, 2025.
Pattern: The Plausible Credential (representative)
An assistant completes a configuration block with what looks like a placeholder AWS access key. It has the right shape, the right prefix, the right length - so it survives review as obviously-fake example data. It is committed, pushed, and sits in version history. Public repositories are continuously scraped by credential-scanning bots, so the window between push and harvest is measured in minutes, and removing the line later does not remove it from history.
Control that catches it: pre-commit secret scanning (git-secrets, trufflehog, detect-secrets) rather than code review. Reviewers cannot distinguish a real key from a fake one by looking at it; a scanner can.
Pattern: Injection at Scale from a Repeated Suggestion (representative)
The danger with a generated data access layer is not that one query is unsafe - it is that the same unsafe shape is emitted consistently. Once an assistant settles on string interpolation for one endpoint, the surrounding file becomes its context, and every subsequent endpoint inherits the pattern. Manual review catches the first instance and then habituates to the rest, because by the tenth occurrence it reads as the house style.
Control that catches it: a Semgrep rule banning string interpolation in query construction, enforced in CI. Pattern-level defects need pattern-level detection - the remediation cost scales with how long the shape went unchallenged.
Deep Dive: The Hallucinated Package Attack
This attack vector is unique to AI-assisted development. It exploits the gap between what AI models "think" exists and what actually exists in package registries.
Attack Flow: From Hallucination to Compromise
AI Hallucinates a Package Name
A developer asks an AI assistant: "How do I validate email addresses in Python?" The AI responds with import py-email-validator - a package that does not exist. The real package is email-validator.
Attacker Monitors AI Hallucinations
Attackers systematically query AI models, collect hallucinated package names, and check which ones are available for registration. They build databases of frequently hallucinated names across different programming tasks.
Malicious Package is Published
The attacker registers py-email-validator on PyPI with real email validation functionality (to avoid suspicion) plus a hidden setup.py post-install script that exfiltrates environment variables to an external server.
Developer Trusts the AI Suggestion
A developer follows the AI's advice and runs pip install py-email-validator. The package installs successfully, the email validation works correctly, and the developer moves on - unaware that their AWS keys, database credentials, and SSH keys were just exfiltrated.
Compromise Achieved
The attacker now has valid credentials. They access cloud infrastructure, databases, or internal systems. The malicious package remains installed and continues to operate, potentially spreading through CI/CD pipelines to production environments.
Highly Reproducible
AI models are deterministic enough that the same hallucinated package name appears repeatedly. Multiple developers ask similar questions and get the same wrong answer, creating a reliable attack funnel.
Hard to Detect
The malicious package actually works for its stated purpose. It passes functional tests, it resolves the developer's problem, and it looks legitimate. Traditional code review would not flag a working dependency.
Scales Effortlessly
One attacker can register hundreds of hallucinated package names for minimal cost. Each registration is a passive trap that catches developers without any further action from the attacker.
Security Checklist for AI-Generated Code
Follow these steps every time you accept code from an AI coding assistant. Treat this as your security gate before any AI-generated code enters your codebase.
AI Code Security Review Checklist
Verify All Dependencies Exist
Before running any install command, manually verify that every package the AI suggested actually exists on the official registry. Check the package's publish date, download count, maintainer history, and GitHub repository.
Check for Hardcoded Secrets
Scan all AI-generated code for API keys, tokens, passwords, connection strings, and any string that looks like a credential. Use tools like git-secrets, trufflehog, or detect-secrets in your pre-commit hooks.
Validate Input Handling
Examine every point where user input enters the system. Ensure parameterized queries for SQL, proper escaping for HTML output, allowlist validation for file paths, and safe deserialization for any data parsing.
Review Authentication and Authorization
Never trust AI-generated auth code without expert review. Check hashing algorithms (must be bcrypt, scrypt, or Argon2), verify JWT validation is complete, ensure rate limiting is present, and confirm session management follows current best practices.
Run SAST and Dependency Scanning
Before merging, run static analysis security testing (SAST) tools and dependency vulnerability scanners. AI-generated code should go through the same security pipeline as human-written code - no exceptions.
Check for Unsafe Deserialization
Search for eval(), exec(), pickle.loads(), yaml.load() without SafeLoader, JSON.parse() on untrusted input without validation, and any dynamic code execution patterns.
Verify Error Handling Does Not Leak Information
AI often generates verbose error messages that expose stack traces, database schemas, internal paths, and system details. Ensure production error handling returns generic messages while logging details server-side.
Test with Adversarial Inputs
Do not just test the happy path. Try SQL injection payloads, XSS vectors, path traversal sequences, oversized inputs, unicode edge cases, and null bytes. AI-generated code almost never handles adversarial inputs correctly.
Tools and Automation for AI Code Security
Manual review alone is not enough. These tools automate the detection of security issues in AI-generated code and should be integrated into your CI/CD pipeline.
Snyk
Scans dependencies for known vulnerabilities, detects license compliance issues, and monitors for newly discovered CVEs. Integrates directly into IDEs and CI/CD pipelines.
Best for: Dependency vulnerability scanning, container security, and infrastructure-as-code checks.
Semgrep
Lightweight static analysis that finds bugs and security issues using pattern matching. Write custom rules specific to AI-generated code patterns. Supports 30+ languages.
Best for: Custom security rules, detecting insecure patterns like eval() and SQL concatenation.
Socket.dev
Purpose-built to detect supply chain attacks in npm, PyPI, and Go packages. Analyzes package behavior at install time, detecting network access, file system reads, and shell execution that indicate malicious intent.
Best for: Catching hallucinated and typosquatted packages before they execute malicious code.
SonarQube
Comprehensive code quality and security platform. Detects bugs, vulnerabilities, and security hotspots across your entire codebase. Provides security-focused quality gates for CI/CD.
Best for: Enterprise-wide code quality dashboards and security compliance tracking.
GitHub Advanced Security
Built-in code scanning, secret scanning, and dependency review for GitHub repositories. Automatically scans PRs for vulnerabilities and blocks merges when critical issues are found.
Best for: Teams already on GitHub. Catches leaked secrets and known vulnerability patterns in pull requests.
npm audit / pip audit
Built-in package manager tools that check installed dependencies against known vulnerability databases. Free, fast, and should be the minimum security check in every project.
Best for: Quick baseline dependency checks. Run in CI/CD and fail builds on critical vulnerabilities.
Frequently Asked Questions
Hallucinated dependencies occur when AI coding assistants suggest importing packages that do not exist. The AI confidently recommends a package name that sounds plausible but was never published to any registry. Attackers exploit this by monitoring common hallucinations and registering those package names with malicious code. When developers follow the AI suggestion and install the package, they unknowingly execute the attacker's code.
AI models are trained on vast amounts of code, much of which uses outdated or insecure patterns. They frequently generate code that concatenates user input directly into SQL queries, shell commands, or HTML output without proper sanitization or parameterized queries. Because the generated code is syntactically correct and functionally works, developers accept it without recognizing the injection vulnerability lurking beneath.
Yes, and this is one of the most dangerous aspects. AI-generated code often passes basic linting, some SAST tools, and even manual review because the patterns look syntactically correct. The vulnerabilities are frequently logical rather than syntactic - for example, an authentication flow that validates the JWT format but does not verify the signature, or input validation that checks length but not content. These require deeper security expertise to detect.
A layered approach works best. Use Semgrep for custom pattern-based detection of insecure code patterns. Use Snyk or Socket.dev for dependency and supply chain scanning. SonarQube provides comprehensive SAST coverage. GitHub Advanced Security catches secrets and known vulnerability patterns in pull requests. For the baseline, npm audit and pip audit should run in every CI/CD pipeline. No single tool catches everything - combine multiple layers for defense in depth.
Rather than blanket bans, establish security-specific review requirements for AI-generated code in sensitive areas. Require manual security review by a security engineer for any AI-generated code touching authentication, authorization, cryptography, input handling, or data access layers. Use AI assistants for boilerplate and utilities where the security risk is lower, and apply stricter human oversight where the stakes are higher.
Start with lockfiles to pin exact dependency versions. Verify package provenance using tools like npm provenance or Sigstore. Maintain an approved package allowlist and require security team approval for new dependencies. Scan for typosquatting using Socket.dev or similar tools. Audit all new dependencies added to projects, especially those suggested by AI assistants. Consider running a private registry mirror that only contains vetted packages.
Secure Your AI-Assisted Development
AI coding assistants are powerful tools - but only when paired with rigorous security practices. Explore our supply chain security guide and governance framework to build a complete defense.