AI Security

AI cybersecurity and AI-assisted attacks: what defenders need to know

An evidence-led guide to AI cybersecurity and AI-assisted attacks, including attacks on AI systems, AI-enabled offensive operations, agentic-AI risk and practical defensive priorities for Indian organisations.

Abstract defensive cybersecurity scene showing a protected AI network facing controlled streams of suspicious activity, with no text or logos.

AI cybersecurity and AI-assisted attacks** describe two connected but different realities. AI can help defenders secure AI systems and improve cyber operations, while threat actors can use AI to accelerate reconnaissance, vulnerability discovery, phishing, malware development, impersonation and other offensive work. The central lesson is not that every attack is autonomous. It is that AI can change the speed, scale, targeting and adaptability of activity that organisations already need to manage. The evidence reviewed for this draft shows active institutional attention rather than a quantified search trend. CERT-In, Google Threat Intelligence, Microsoft, NIST and CISA have all published relevant material dated 2025 or 2026. Their scopes differ, so their observations should not be combined into a claim about all attacks or all organisations. A sound security programme therefore separates observed activity from projected capability and treats AI risk as part of established governance, identity, application, data and incident-response work. This guide explains the distinction between attacks on AI systems and AI-assisted attacks against ordinary targets. It then translates the supplied guidance into practical priorities, with an India-specific section for organisations using CERT-In guidance and AI-enabled systems. > **Key Takeaways > - AI-assisted attacks can accelerate reconnaissance, vulnerability discovery, targeted phishing, malware development, impersonation and evasion, but the supplied sources do not support assuming that every technique is autonomous or has caused a confirmed breach. > - Attacks on AI systems include prompt injection, data poisoning, adversarial attacks, model inversion, model stealing, hallucination exploitation and backdoors, according to CERT-In’s 2025 advisory. > - Agentic AI can expand the attack surface through broad permissions, privilege creep, behavioural misalignment and less transparent event records; CISA recommends low-risk starting points and restricted access. > - Practical defence starts with governance, least privilege, approved tools, protected data, logging, vulnerability management, tested incident response and continuous validation. > - CERT-In’s 2026 blueprint is directly relevant to Indian organisations, but it is guidance for reducing exposure and improving readiness, not evidence of an India-specific incident count.

What does AI cybersecurity and AI-assisted attacks mean?

AI cybersecurity and AI-assisted attacks are best understood as two sides of one risk model: securing AI systems and preparing for attackers who use AI to improve familiar operations.** NIST’s preliminary Cyber AI Profile, released on 16 December 2025, frames the field in three areas: securing AI systems, conducting AI-enabled cyber defence and thwarting AI-enabled cyberattacks. NIST described that document as a preliminary draft for public comment, not a final standard.

Attacks on AI systems target the model, its data, its prompts, its tools or its surrounding application. CERT-In’s 26 March 2025 advisory lists data poisoning, adversarial attacks, model inversion, model stealing, prompt injection, hallucination exploitation and backdoor attacks among the risks of using generative-AI solutions. These are application and information-security concerns: an AI feature can expose data, follow a hostile instruction, produce an unsafe result or be manipulated in ways that affect a wider workflow.

AI-assisted attacks against ordinary targets use AI as an accelerator or amplifier. CERT-In’s 25 May 2026 blueprint describes rapid attack-surface mapping, automated vulnerability discovery and exploit development, personalised phishing and social engineering, AI-generated malware or malicious scripting, deepfake-enabled impersonation, automated orchestration and adaptive evasion. The practical distinction matters because a team may need one set of controls for an AI application and another for a conventional system targeted by an AI-supported operation. Organisations can also map the overlap through their [software supply-chain security](https://cyberoinfo.com/cybersecurity/software-supply-chain-attacks) and application-security processes.

How do AI-assisted attacks change familiar threats?

AI-assisted attacks make familiar activities easier to repeat, personalise and coordinate; they do not automatically make every operation fully autonomous.** Google Threat Intelligence Group reported on 12 February 2026 that it had observed threat actors using AI to gather information, create realistic phishing scams and develop malware. It also said it had not observed direct attacks on frontier models or generative-AI products from advanced persistent threat actors in the reporting described, while it had observed and mitigated frequent model-extraction attacks against private-sector entities. Those are the group’s reported observations within its stated scope.

Reconnaissance is one area where speed and breadth can matter. CERT-In says threat actors are leveraging AI to accelerate reconnaissance, automate vulnerability discovery and increase attack scale and speed. A defensive team should respond by keeping an accurate asset inventory, prioritising exposed services, applying patches based on risk, monitoring for unexpected discovery activity and validating that internet-facing systems behave as intended. The goal is not to guess an attacker’s exact tool; it is to reduce the opportunities that faster discovery can exploit.

Phishing and social engineering can also become more targeted. CERT-In describes personalised phishing, social engineering and deepfake-enabled impersonation, while Microsoft’s 2025 Digital Defense Report describes increased threat-actor use of AI to scale phishing and automate intrusions. These sources do not establish a universal success rate or a total number of victims. They do support a defensive shift toward strong identity checks, independent verification of unusual requests, phishing-resistant authentication where supported and clear escalation routes for suspected impersonation.

Malware and evasion deserve the same careful framing. CERT-In describes AI-generated malware or malicious scripting and adaptive evasion, while Microsoft identifies adversarial prompts, data poisoning and model manipulation as risks that can expand the attack surface. Security teams should therefore combine endpoint and network controls with secure software practices, protected build pipelines, useful telemetry and rehearsed response. For identity-focused controls, CYBERoinfo’s planned [identity attacks and MFA bypass](https://cyberoinfo.com/identity-security/identity-attacks-mfa-bypass) coverage can sit alongside this broader AI-risk explanation.

Which attacks target AI systems directly?

Direct attacks on AI systems exploit the model, its training or runtime data, its instructions, or the permissions around it.** Prompt injection attempts to influence how an AI application follows instructions. Data poisoning contaminates information used to train or ground a system. Model inversion and model stealing concern attempts to infer sensitive information or reproduce model behaviour. CERT-In groups these risks with adversarial attacks, hallucination exploitation and backdoors in its generative-AI advisory.

The correct control depends on the failure mode. Prompt injection requires careful separation of instructions and untrusted content, constrained tool use, output validation and a design that does not grant an AI component unnecessary authority. Data poisoning calls for provenance checks, controlled data pipelines and monitoring for unexpected changes. Model extraction concerns access controls, rate limits, authentication, monitoring and a response plan for unusual query patterns. These are defensive design principles; they should not be treated as proof that a specific attack has occurred.

AI applications also inherit ordinary security weaknesses. A model may be connected to a database, code repository, ticketing system, browser, cloud account or internal workflow. If those integrations are over-permissioned, a successful prompt manipulation or compromised credential can have effects beyond the model itself. CERT-In recommends using approved tools, avoiding sensitive information in uncontrolled public services and reviewing access rights when using generative-AI solutions.

The safe question is therefore not only “Is the model accurate?” It is also “What can this system see, change, send or approve?” Teams should document data flows, separate high-impact actions from low-risk assistance, require human review for consequential steps, preserve logs and test failure cases before deployment. This model is relevant to cloud-hosted AI as well as on-premises systems, and it should be reviewed whenever tools, data sources or permissions change.

What changes with agentic AI?

Agentic AI can introduce a wider and less obvious attack surface because a system may plan, call tools and act across connected services.** A joint CISA and international-partner guide released on 1 May 2026 identifies expanded attack surface, privilege creep, behavioural misalignment and obscure event records as risks of agentic AI. The guide recommends avoiding broad or unrestricted access, beginning with low-risk use cases and incorporating agentic-AI security into the organisation’s security model and risk posture.

Privilege creep is especially important. An agent that starts with a narrow task may gradually accumulate access to files, APIs, identities or administrative functions as teams add integrations. A safer design gives each agent the minimum permissions needed for a defined task, limits the duration of sensitive access and separates read, recommend and execute capabilities. Human approval should remain part of workflows where an action could change production systems, disclose sensitive information or create a material business impact.

Event records must also explain what happened. If an agent makes several tool calls, a conventional application log may not show the original user request, the model’s intermediate decision, the data it received or the action it triggered. Teams should preserve useful audit context, monitor unusual sequences, test for prompt manipulation and review whether a person can reconstruct and stop the workflow. These controls complement, rather than replace, established [cloud security and misconfiguration](https://cyberoinfo.com/cloud-security/cloud-security-misconfiguration) practices.

How can organisations reduce AI cyber risk?

The most reliable AI cyber-risk reduction programme combines governance with ordinary security fundamentals and AI-specific testing.** Start by cataloguing AI use: approved products, internal models, public services, data sources, integrations, owners and high-impact actions. Define what information may be entered, which tools are permitted and which decisions require human review. CERT-In’s 2025 advisory specifically recommends approved tools, avoiding sensitive information in uncontrolled public services and reviewing AI-tool access rights.

Next, reduce exposure before adding advanced detection. Keep internet-facing assets and AI endpoints inventoried, patch vulnerable components, protect credentials, enforce least privilege and separate development, testing and production. Apply strong authentication to administrative access, monitor for unusual model or API use and protect logs from unauthorised alteration. These measures help whether the immediate risk is prompt injection, account compromise, malicious code or an AI-assisted conventional attack.

AI-aware security operations should connect model activity with surrounding systems. A suspicious prompt may be harmless in isolation but important if it is followed by an unusual database query, a new outbound connection or an attempt to access a sensitive repository. Detection logic should be tested against false positives and false negatives, and response playbooks should identify who can disable an integration, revoke credentials, preserve evidence and communicate internally. CERT-In’s 2026 blueprint recommends AI-aware security operations, vulnerability management, incident response, continuous validation and workforce readiness.

Finally, validate the controls continuously. Test prompts and data-handling boundaries, review tool permissions, inspect model and application updates, exercise incident-response procedures and record lessons learned. Do not assume that a vendor’s safety statement, a model’s general performance or a single assessment proves ongoing security. A practical programme measures whether people can identify unsafe behaviour, contain it and restore trusted operation without relying on unsupported assumptions about attacker capability.

What should Indian organisations prioritise?

Indian organisations should use CERT-In’s AI guidance as a dated, practical input to their existing risk and incident-response programme.** CERT-In’s 25 May 2026 blueprint recommends governance, exposure reduction, technical controls, AI-aware security operations, vulnerability management, incident response, continuous validation and workforce readiness. It is directly relevant to CISOs, security operations teams, IT administrators, AI product owners, compliance leaders and public-sector or critical-infrastructure organisations.

CERT-In’s 26 March 2025 advisory addresses attacks against or through AI applications and recommends safe use of generative-AI tools. Organisations should translate that advice into an approved-tools register, data-classification rules, access reviews, secure integration patterns and escalation procedures. Where an AI system processes sensitive information or can take consequential actions, its owner should document the purpose, data, permissions, human oversight and recovery path before expanding its use.

The India angle should not be overstated. The supplied research does not provide an India-specific count of AI-assisted incidents, victims, losses or confirmed breaches. It also does not establish that every CERT-In recommendation applies identically to every organisation or situation. For broader reporting context and cyber-safety routing, readers can use CYBERoinfo’s [India cybersecurity hub](https://cyberoinfo.com/cybersecurity/india-cybersecurity), while keeping legal and operational decisions aligned with current official guidance and the organisation’s own advisers.

FAQ: AI cybersecurity and AI-assisted attacks

### Are AI-assisted attacks always autonomous?

No. The supplied evidence describes AI being used to accelerate, automate, personalise or scale parts of offensive activity, including reconnaissance, phishing and malware development. That does not establish that every operation is autonomous or that a human is absent from the workflow. Defensive planning should focus on the capabilities and access paths that matter to the organisation rather than assuming a single level of automation.

### What is the difference between an attack on AI and an AI-assisted attack?

An attack on AI targets a model, its data, its prompts, its integrations or its runtime behaviour. Examples in CERT-In’s advisory include prompt injection, data poisoning, adversarial attacks, model inversion, model stealing, hallucination exploitation and backdoors. An AI-assisted attack uses AI to improve an operation against an ordinary target, such as reconnaissance, phishing, malware development or impersonation.

### Does using an AI tool create a cybersecurity incident?

Not by itself. Risk depends on the tool, the data supplied, the permissions granted, the connected systems and the organisation’s controls. CERT-In advises using approved tools, not entering sensitive information into uncontrolled public services and reviewing AI-tool access rights. Teams should assess the facts, preserve relevant evidence and follow their incident and escalation procedures when suspicious activity is identified.

### What is the first defensive step for an AI application?

Start with visibility and boundaries: identify the application owner, data sources, integrations, users, permissions and possible actions. Then limit access, protect sensitive data, validate outputs, retain useful logs and test prompt and data-handling failures. A system that cannot be inventoried or audited is difficult to secure, regardless of the model’s capabilities.

Limitations

  • Evidence and guidance can change; verify current advisories and organisational applicability before acting.

Evidence sources

  1. cert-in.org.in — CERT-In Blueprint for Reducing Exposure and Defending against AI-Assisted Vulnerabilities Exploitation in Digital Infrastructure, 25 May 2026
  2. blog.google — Google Threat Intelligence Group: Our new report details the latest ways threat actors are misusing AI, 12 February 2026
  3. microsoft.com — Microsoft Digital Defense Report 2025
  4. nist.gov — NIST: Draft NIST Guidelines Rethink Cybersecurity for the AI Era, 16 December 2025
  5. cisa.gov — CISA and international partners: Guide to Secure Adoption of Agentic AI, 1 May 2026
  6. cert-in.org.in — CERT-In Advisory CIAD-2025-0013: Best Practices against vulnerabilities while using Generative AI solutions, 26 March 2025