The Human Factor in AI Security

Why Social Engineering Gets Smarter When Attackers Use AI

9 MIN Read

The phishing email that bypasses your filters wasn’t written by a human. And it knew things about your target that a human attacker would have taken weeks to gather.

INTRODUCTION

Introduction

Social engineering has always been the attack vector that bypasses the most sophisticated technical controls. A firewall cannot stop a well-crafted phishing email from convincing an employee to hand over their credentials. An endpoint agent cannot prevent a user from wiring funds to a fraudulent account because a voice on the phone convinced them it was their CFO’s emergency request. The human element — the cognitive biases, the social obligations, the desire to be helpful under pressure — has always been the most reliable path past technical defenses.

AI has not changed this dynamic. It has amplified it in ways that make the threat qualitatively different, not just quantitatively larger.

The barriers that historically constrained social engineering at scale — the time required to research targets, the language skill needed to craft convincing lures, the effort of producing personalized content, the cost of impersonating voices and faces — have been dramatically lowered by AI. Attacks that previously required skilled human operators working individually can now be produced at scale, with higher quality and greater personalization than was ever operationally feasible before.

This article examines how AI is changing the social engineering threat landscape, what it means for organizations’ defenses, and how security awareness programs and technical controls must evolve in response.

02 / AI SECURITY

AI-Generated Phishing: The End of ‘Obvious’ Attacks

Traditional phishing detection relied significantly on signals that were artifacts of attacker limitations: poor grammar and spelling that reflected non-native language construction, generic content that made no reference to the target’s specific context, implausible scenarios that did not match the target’s actual relationships and responsibilities.

Security awareness training has long taught employees to look for these signals. The training was effective because the signals were reliable. They are no longer reliable.

Large language models can produce phishing content that is grammatically flawless, contextually specific, and tonally matched to the communication style of the organization or individual being impersonated. More significantly, AI tools can dramatically accelerate the research phase of targeted phishing by aggregating information from LinkedIn profiles, public press releases, social media, corporate websites, and other open-source intelligence sources to construct detailed target profiles in minutes rather than the days or weeks that manual research required.

The result is spear phishing content that would previously have required a skilled human attacker significant time and effort to produce, now generated at scale with minimal marginal cost per target.

03 / AI SECURITY

Deepfake Voice and Video: When Seeing Is No Longer Believing

The impersonation attacks that have historically required physical proximity or significant technical sophistication — convincingly mimicking a specific person’s voice or appearance — are now accessible to a much broader range of adversaries through AI-powered deepfake tools.

The operational implications are significant and already being exploited:

Deepfake voice in BEC attacks. Business Email Compromise has evolved from email-only attacks to multi-channel attacks that combine convincing email with AI-generated voice calls that appear to come from executives, legal counsel, or financial counterparties. The combination of a well-crafted email followed by a voice confirmation from an AI-cloned executive voice has been used successfully to authorize fraudulent wire transfers at significant dollar values.

Video deepfakes in verification bypass. Organizations that use video verification as an identity confirmation step — for financial transactions, account access, or identity verification — are facing attacks in which AI-generated video of the target individual is used to satisfy the video check. Financial institutions and cryptocurrency platforms have been among the early targets of this technique.

Synthetic identity creation. AI can generate entirely synthetic identities — individuals who do not exist but have convincing photographs, plausible biographical details, and consistent digital footprints. These synthetic identities can be used to create fraudulent accounts, pass identity verification checks, and provide cover for social engineering campaigns.

04 / AI SECURITY

Business Email Compromise Evolution

BEC is already one of the highest-financial-impact attack categories in enterprise security. AI lowers the barriers to conducting BEC attacks in ways that will expand both the frequency and the scale of successful incidents:

• The language quality threshold that previously filtered out less sophisticated attackers has been eliminated. Any actor with access to an LLM can produce email content that matches the writing style of a specific individual.
• The research time required to understand organizational structures, relationships, and processes — information needed to craft convincing impersonation — has been dramatically reduced through AI-assisted open-source intelligence gathering.
• The cost of personalizing attacks at scale has collapsed. A campaign targeting 500 finance department employees with individually personalized emails referencing their specific managers, recent company news, and context-appropriate urgency scenarios is now operationally feasible in ways it was not before.
• Multi-channel attacks combining email, voice, and chat can now be coordinated with AI generating content across all channels simultaneously, increasing the apparent legitimacy of the attack through channel reinforcement.

05 / AI SECURITY

How Security Awareness Programs Must Evolve

Security awareness programs that teach employees to identify phishing by its surface characteristics — poor grammar, generic content, suspicious sender addresses — are providing defenders with a detection methodology that is increasingly unreliable against AI-generated attacks. The awareness curriculum must evolve to address the new threat model:

Shift from content-based to process-based defenses. Instead of teaching employees to identify phishing by how it looks, teach them to verify requests through process regardless of how legitimate they appear. A wire transfer request that follows the correct verification procedure is safe regardless of how convincing the accompanying email was. A request that bypasses verification is suspicious regardless of how legitimate it appears.

Establish out-of-band verification requirements for high-risk actions. Any action that carries significant financial, legal, or security consequences should require verification through a channel independent of the one through which the request arrived. An email requesting a wire transfer should be verified by calling the requestor on a known phone number — not the number provided in the email.

Educate employees about deepfake capabilities. Employees who do not know that AI can convincingly clone a voice or generate a realistic video call will not recognize that these capabilities create new attack vectors. Awareness content should explicitly cover what deepfakes are, how they are used in attacks, and why a voice or video call does not constitute identity verification.

Create and train on pre-agreed verification signals. For high-risk scenarios, pre-agreed safe words or verification phrases — established in advance through a trusted channel — provide a simple and effective defense against real-time impersonation attacks. If the ‘CFO’ on the phone cannot provide the pre-agreed verification phrase, the call is not the CFO.

06 / AI SECURITY

Technical Detection Approaches

While process-based defenses are the most reliable protection against AI-enhanced social engineering, technical controls provide complementary detection capabilities:

• AI-generated content detection tools that analyze text for statistical patterns associated with LLM-generated content, though these tools remain imperfect and are subject to evasion by adversaries who understand their detection logic
• Email authentication controls (DMARC, DKIM, SPF) that prevent domain impersonation — a necessary baseline that limits but does not eliminate the email attack surface
• Behavioral email analysis that identifies anomalous sending patterns, unusual request types, and communication patterns inconsistent with known sender behavior
• Voice biometric verification for high-value communications that provides authentication beyond the voice content itself, though this control is also subject to ongoing adversarial development
• Deepfake detection tools for video verification scenarios, used to flag potential synthetic media for human review before high-risk actions are approved

07 / AI SECURITY

Building Organizational Resilience

The deepest defense against AI-enhanced social engineering is organizational resilience: a culture and set of processes that make it structurally difficult for even highly convincing attacks to succeed, regardless of the quality of the lure.

Resilient organizations share several characteristics:

• They have clear, documented, and enforced procedures for high-risk actions that cannot be bypassed regardless of the apparent urgency or authority of the request
• They treat unusual urgency and pressure as a red flag rather than a reason to expedite, recognizing that urgency is one of the most reliable social engineering levers
• They create psychological safety for employees to pause, verify, and escalate without fear of being seen as obstructive or unhelpful
• They conduct regular exercises that expose employees to AI-generated social engineering attempts in a controlled environment, building recognition through experience rather than theory
• They measure the effectiveness of their awareness program against AI-generated attack content specifically, not just against legacy attack patterns

KEY TAKEAWAYS

Key Takeaways

Recap: AI has not changed the fundamental dynamics of social engineering. It has removed the constraints that previously limited attacker scale, quality, and personalization. Process-based defenses are now more important than content-based detection.

• AI-generated phishing eliminates the surface signals that employees have been trained to recognize as indicators of attack
• Deepfake voice and video have made impersonation attacks accessible at a scale and quality that was previously operationally infeasible
• BEC attack economics have been fundamentally altered: personalization at scale, reduced research time, and multi-channel reinforcement
• Security awareness programs must shift from content-based to process-based defenses: verify the request, not the requester
• Out-of-band verification, pre-agreed verification signals, and enforced high-risk action procedures are the most reliable organizational controls
• Technical detection tools for AI-generated content provide supplementary capability but are not a substitute for process-based resilience

The organizations that will be most resilient against AI-enhanced social engineering are not those with the most sophisticated detection tools. They are those that have built processes and cultures that make verification automatic, that treat unusual urgency as a threat signal rather than a reason to expedite, and that understand that the quality of an attack is no longer a reliable indicator of its legitimacy.

Security / Strategy / Trust

Security isn't just technology.
It's trust.

Exploring identity, security strategy, AI security, and the human side of cybersecurity.