In a recent incident, a finance professional in Hong Kong was deceived into transferring US$25 million to fraudsters using deepfake technology to mimic his company’s CFO on a video call. This alarming event underscores the increasing sophistication of generative AI and its potential for misuse.

Since the advent of platforms like ChatGPT in 2022, generative AI has made significant strides. These systems, developed from extensive training with large datasets, are adept at emulating human language patterns, often making it challenging to distinguish between machine and human interaction. But why do we frequently perceive these machine interactions as genuine human conversations?

This inclination is rooted in the fundamental rules of conversation embedded in human psychology. Typically, when humans communicate, they assume that their counterpart shares a mutual understanding of the world—an assumption known as anthropomorphism. Historically reserved for animals, this attribution of human-like qualities now extends to our interactions with machines.

Insights from Recent Studies

Kyle Mahowald, Anna Ivanova, and their research team have explored this tendency to confuse AI’s language proficiency with actual cognitive ability. They make a critical distinction between formal linguistic competence, which involves understanding language rules, and functional linguistic competence, which is about applying language in practical situations. AI often generates technically correct responses that miss the emotional or situational nuances of human interactions.

For instance, an AI might respond to someone’s anxiety about an upcoming event by explaining the biological mechanisms of stress, rather than offering empathy or practical advice, revealing its incapacity for genuine emotional engagement.

Grice’s Maxims and AI Communication

Philosopher Paul Grice identified four maxims that guide effective communication:

1. Quality: Be truthful and base information on evidence.
2. Quantity: Provide just enough information—neither too little nor too much.
3. Relevance: Keep contributions relevant to the current discussion.
4. Manner: Convey information in a clear and orderly way.

While AI tends to perform well regarding quantity, relevance, and manner, it often struggles with the quality maxim, sometimes producing responses that are inaccurate or misleading.

The Cooperative Principle in Human-AI Interaction

Grice’s cooperative principle suggests that communication relies on mutual cooperation, assuming that all parties adhere to these maxims. This principle explains why deceit can be effective—people interacting with AI might assume it is providing truthful responses.

Navigating AI Communications

As AI technologies increasingly permeate our daily interactions, it’s crucial to engage with them cautiously. Recognizing that AI merely simulates human conversation—lacking real understanding or ethical discernment—can help prevent potential miscommunications and protect against misinformation. It’s important to keep in mind that AI does not possess human-like thinking or ethical considerations.

Overall, while AI can enhance certain aspects of communication, being aware of its limitations is vital. Understanding that AI does not engage in genuine human thought or ethical reasoning is essential for maintaining the integrity of our communications and ensuring the security of our personal information.