Security Journey Blog

A Complete Secure Coding Checklist for AI Apps

Written by Security Journey Team | Nov 20, 2026, 9:28:02 PM

AI coding agents can write features that work well but still leave them open to attacks. Researchers at the 2026 International Conference on Machine Learning (ICML) found that a coding agent delivered working code for 57% of feature requests, but only about one in five of those features was secure.1 A reviewer who only checks that code works would also approve the other four.

A secure coding checklist for AI apps is the set of checks your team completes before it ships AI-assisted code. The checklist covers invented packages, exposed secrets, agents with too much access, and large language model (LLM) features.

What Does Secure Coding Mean for AI Applications?

Secure coding for AI applications is the work your developers and application security (AppSec) team do to protect code an AI assistant helped write. That work also includes the LLM-powered features that code adds to your product. Your team is ultimately responsible for every line it ships, which is why your developers should review AI-assisted or AI-written code.

Developers who already validate input and limit permissions can keep those habits as they start working with AI. Your team can keep the standards it enforces today and add checks for the ways assistants and models fail. That combination is one of the important secure coding practices for AI applications Engineers and AppSec leaders can track those added checks in a secure coding checklist for AI apps, next to the requirements they already report against.

How Is Securing An AI Application Different from Securing a Traditional Application?

With AI applications, your team has to cover everything it would for a traditional app, plus the code an assistant suggests and any feature where your app sends a prompt to an LLM. A developer might get a working function from the assistant in seconds and merge it after a quick skim once it passes a few tests. If that function imports a package that doesn't exist, an attacker could claim that name and publish malicious code the developer would then install.

Your team has to treat those prompts and replies as a new trust boundary. Validate everything that crosses that boundary the way you'd validate form input.

Employees without an engineering background can create both risks when they use those same assistants to write scripts and automations. Your reviewers need to check that code too, since it ends up in the same repositories and environments.

What Should You Check Before Using AI-Generated Code?

A reviewer should check any AI-generated code before anyone on your team merges it. Fu and colleagues examined 733 AI-generated snippets found in public GitHub projects, 672 of them generated by Copilot, and spotted security weaknesses in 27.3% of them.2

That's why your reviewers should use an AI code security checklist on every pull request that includes code an AI assistant wrote. Start each AI-generated code security review with the weakness types that were most common in Fu's snippets, like weak random values and code injection. Work through these checks before approving the merge:

  • Pull random values from a cryptographic source: Confirm the code uses a cryptographically secure generator for session tokens and password reset links. Insufficient randomness was the most common weakness in Fu's data.2
  • Keep user input out of eval, exec, and shell calls: Search the code for eval, exec, and shell calls and confirm none of them receives text that came from a user.
  • Convert special characters in template output: Every template should turn characters like < and > into safe character codes so a browser displays any injected script as plain text.
  • Compare login and session code against the spec: The agent in the ICML study left a required session max-age unenforced. In another task, its login code responded at different speeds depending on whether a username existed, which could let attackers figure out valid usernames.1

What Dependency and Supply Chain Risks Come with AI Code Suggestions?

AI assistants sometimes import packages that don't exist and tend to invent the same names over and over. Spracklen and colleagues analyzed 576,000 code samples from 16 models and found that 19.7% of package references pointed to packages that didn't exist.3 A developer who accepted every import those 16 models suggested would have tried to install a made-up package about once in every five.

The researchers also found that when they gave a model the same prompt 10 times, it produced 43% of its invented names in all 10 attempts.3 An attacker who tries common prompts can learn which fake names your developers might see. Your team can take care of dependency security for AI apps with these checks in every project that uses an AI coding assistant:

  • Confirm the package exists by checking its publisher, age, and download history before you install it.
  • Commit a lockfile with version hashes so your team installs only the versions it approved.
  • Route new dependencies through an allowlist or internal mirror of vetted packages.
  • Scan dependencies on every build for known vulnerabilities and malware.

What Belongs on a Secrets and Access Checklist for AI-Assisted Projects?

Your team needs to control what an AI assistant can read and what an agent can do with its credentials. For example, an assistant that reads a .env file could copy an API key into new code. Agents are riskier because they can use their credentials to delete production systems. Add these items to your secrets management for AI code and your agent access policies:

  • Block the assistant from your credential files: Exclude .env files and credential stores from the folders the assistant can open.
  • Scan for secrets before every commit: A secret scanner in a pre-commit hook can flag a key the assistant added to a file before a developer pushes it.
  • Rotate any key that someone pasted into a prompt: Treat that key as exposed since the AI provider might store the conversation. Replace it with a new key and revoke the old one.
  • Give agents their own credentials: Set up separate credentials for each agent with no production access by default. An agent with development credentials can only change development systems.
  • Turn on deletion protection and test your backups: Cloud providers let you lock production databases against deletion. Schedule test restores so your team knows it can get deleted data back.
  • Have a developer approve every infrastructure plan: Tools like Terraform show a plan that lists every resource a change will create or delete. Store Terraform's record of your infrastructure remotely so an agent can't work from an outdated copy.

What LLM-Specific Risks Belong on This Checklist?

Check every feature that sends prompts to an LLM for prompt injection, unsafe output handling, and sensitive data exposure. The Open Worldwide Application Security Project (OWASP) lists all three in its OWASP Top 10 for LLM applications ranking. Add these checks to your LLM application security checklist:

  • Test for prompt injection: An attacker could hide instructions in a web page or file the model reads. The model might follow those instructions and ignore yours.
  • Validate output before your app acts on it: An attacker could slip code into a model reply that your app then passes straight into a query or web page.
  • Keep prompts out of your logs: Some monitoring tools record raw prompts, replies, and reasoning steps by default.4 Turn that logging off because anyone with log access could read customer data or API keys from those prompts.

How Do You Check for Prompt Injection and Unsafe Output Handling?

Treat everything the model reads as untrusted and validate everything it writes before your code or a user acts on it. Your developers need to put those checks in the app's own code because the model can't reliably tell your instructions apart from an attacker's. OWASP recommends these controls for prompt injection prevention:

  • Define a strict output schema: List the exact fields a model reply can contain and have your code reject any reply that breaks that format. Your team still needs to check the values within a reply that passes, though.
  • Remove invisible Unicode characters: Attackers can hide instructions in zero-width and tag characters that a person can't see but the model still reads. Have your code strip those characters from text going into or out of the model.
  • Keep injection tests in your test suite: Write tests that send known injection attempts to the model and check how your app responds. Repeat those tests whenever your team changes the model or a prompt.

How Do You Build These Checks Into Your Development Workflow?

Automate every check a tool can handle and save human review for the logic that tools can't judge. Faros AI tracked about 10,000 developers on 1,255 teams and found that engineers who used AI tools merged 98% more pull requests and pull request review time went up 91%.5

Your reviewers can spend their time on code that needs human judgment once tools take care of the routine checks. Add these automated checks to your commit hooks and continuous integration (CI) pipeline:

  • Scan for secrets in a pre-commit hook and again in CI.
  • Check dependencies for known vulnerabilities in CI and confirm every new package exists in the registry.
  • Run static analysis on every pull request to flag patterns like weak random number generators.
  • Add your injection tests to the pipeline so it tests every model or prompt change.

Focus your human reviewers on authentication, authorization, and business logic. A scanner can't tell whether a login flow matches your spec. The session and login flaws from the ICML study were exactly this kind of mismatch.

How Do You Keep This Checklist Current as AI Tools Change?

Revisit this checklist any time your team adopts a new assistant, model, or agent capability because each one changes what can go wrong. Your developers can also identify these flaws faster in a pull request once they've found and fixed them in real code.

Security Journey's hands-on Break/Fix labs and capture-the-flag challenges give developers that practice, with role-based training that we update as AI tools change. Employees without an engineering background who code with AI get hands-on practice through the AI Advantage Citizen Developers track. Your team can then institutionalize secure AI-assisted development habits any time it adopts new AI tools.

References

  1. Zhao, Wang, Zhang, Luo, Li, and Li. “Is Vibe Coding Safe? Benchmarking Vulnerability of Agent-Generated Code in Real-World Tasks.” ICML 2026. July 2026. https://proceedings.mlr.press/v306/zhao26ax.html
  2. Fu, Liang, Tahir, Li, Shahin, Yu, and Chen. “Security Weaknesses of Copilot-Generated Code in GitHub Projects: An Empirical Study.” ACM Transactions on Software Engineering and Methodology. 2025. https://doi.org/10.1145/3716848
  3. Spracklen, Wijewickrama, Sakib, Maiti, Viswanath, and Jadliwala. “We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs.” USENIX Security 2025. August 2025. https://www.usenix.org/conference/usenixsecurity25/presentation/spracklen
  4. OWASP GenAI Security Project. “OWASP GenAI LLM Top 10 2026.” August 2026. https://genai.owasp.org/resource/owasp-genai-llm-top-10-2026/
  5. Faros AI. “The AI Productivity Paradox.” July 2025. https://www.faros.ai/ai-productivity-paradox