AI companions: Ensuring their “company” can be safely enjoyed

In an era when loneliness and depression are taking on pandemic proportions, it is becoming increasingly common for people to seek the company of artificial intelligence (AI) entities, the so-called “AI companions”, in the form of chatbots, virtual assistants, and conversational agents. These systems are used across a wide range of contexts, from companions for minors to professional tools in sensitive settings, and increasingly take on roles as virtual therapists, study assistants, professional helpers, or even romantic partners. They can talk, reason, generate text, respond like humans, and engage with unprecedented realism and adaptability. Said systems, besides enhancing practical aspects of daily life, offer understanding, compassion, empathy, emotional support, uninterrupted availability, and absence of a judgmental approach, i.e., all those elements that are hard to find nowadays in real people. 

The fact that some of the most prominent AI companionship apps are marketed in the “Health and Fitness” sections of app stores, already gives a clear view of their manufacturer’s intended use as an ideal simulation of human relationships. But as their use grows, so does concern: While such systems may offer accessibility and emotional support, their ability to influence behaviour, process sensitive data, and exploit emotional vulnerability raises serious fundamental rights concerns. 

The European Union (EU) already has a set of legislative frameworks applicable to AI companion systems. While the legal backbone governing AI companions is the much-discussed EU Artificial Intelligence Act (AI Act), other existing frameworks, such as the General Data Protection Regulation (GDPR) are equally relevant. Nonetheless, headlines about severe harm, including cases where AI companions have encouraged or failed to prevent self-harm, highlight the urgent need for a more tailored regulatory approach to their safe and beneficial use. Given the widespread deployment of AI companions, this task is inherently context-dependent and requires careful calibration of legal obligations and safeguards to ensure they are effective in practice. Here's a breakdown of how EU law currently governs AI companions and what that means for developers, companies, and users. 

1. The AI Act: A Risk-Based, Horizontal Regulation 

The AI Act (Regulation (EU) 2024/1689) is the first comprehensive regulation on AI worldwide. It entered into force in August 2024 and establishes a unified, EU-wide framework for the development, deployment, and use of AI systems. Some parts of the Act are not yet applicable and will enter into force later (see this blog post for an overview). Instead of a “one-size-fits-all” approach, the AI Act adopts a risk-based classification: AI systems are subject to different levels of regulation, depending on the assigned level of inherent risk that they pose to health, safety, and fundamental rights. The following classification is applicable:

AI companions do not automatically fall under a specific risk level. The intensity of their regulation depends significantly on how they are used, who deploys them, and for what purpose. Determining the risk level and the applicable obligations therefore requires a case-by-case assessment, including at the design stage, where systems may be structured to avoid triggering higher risk classifications. Below we try to identify how likely it is for an AI companion to fall under each category, what are the existing regulatory blind spots, and how to best address them. 

For the sake of conciseness, this blog post does not distinguish between the obligations of AI companion “providers” and “deployers” within the meaning of the AI Act and assumes that the same entity both provides and deploys the AI system. 

a. Horizontal obligations 

The AI Act imposes certain horizontal obligations which aim to make AI systems accountable, auditable, and fair. These obligations are particularly relevant for AI companions and mainly imply: 

- Transparency obligations: Inform users in a timely and clear manner that they are interacting with AI (not a human); 

- Marking of generated content, to be able to identify said content – where applicable – especially for synthetic audio, image or video content that could mislead users. 

b. Requirements applicable to General-Purpose AI (GPAI) models underlying AI companions 

If a chatbot or companion app is built on – or is – a so-called foundational model (e.g., GPT-4, GPT-4o, etc), which will in practice most likely be the case, additional obligations apply to its provider (e.g., OpenAI), under the GPAI category, such as: 

- Providing technical documentation; 

- Implementing a copyright policy; 

- Publishing a summary of the model’s training content. 

Some GPAI may pose systemic risks, in the sense that they have "high-impact capabilities" causing significant impact on the EU market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or society. For those systems, additional obligations include model evaluation, adversarial testing, reporting to the European Commission on serious incidents and corrective measures, risk assessment and mitigation, and cybersecurity safeguards. 

Currently, Codes of Practice developed by GPAI providers under the supervision of the European Commission, identify a list of systemic risks. Among those we can find “harmful manipulation” as “significantly enhancing capabilities for persuasion, deception, and personalised targeting, particularly through multi-turn interactions and where individuals are unaware of or cannot reasonably detect such influence”. No specific examples are mentioned therein, but a GPAI that can be used as a component of an AI companion is potentially covered.

Even though a Code of Practice could raise the bar for the obligations of GPAI providers and the subsequent safeguards of these AI systems, it is important to underline that it is a voluntary tool for compliance that aims to bridge the gap between GPAI model provider obligations applicable from 2 August 2026 onwards, and the adoption of standards, which may take more than three years. 

c. AI companions falling under the high-risk classification 

AI companions are not automatically high-risk in the meaning of the AI Act, but they can become so depending on their intended use, i.e., if used in certain contexts listed in Annex III (e.g., for biometric identification, as a safety component of critical infrastructures, or to access essential services, such as healthcare, and education) or if integrated into regulated products (e.g., an AI conversational assistant integrated into a medical device, a robotic caregiver). In this context, more stringent obligations apply for providers, such as: 

- Using high-quality, well-governed data; 

- Giving clear and accessible instructions for use; 

- Ensuring proper human oversight; 

- Running a quality management system (e.g., incident response procedures); 

- Passing a conformity assessment and getting a CE mark before entering the market; 

- Registering the AI system in an EU database before the system is placed on the market;

For many AI companions, such as productivity assistants or brainstorming partners, the appropriate categorisation will likely not be that of a high-risk AI system. However, given their very nature, there is a (high) probability that an AI companion may fall within the category of prohibited practices under the AI Act. 

d. AI companions can potentially fall under the prohibited practices 

The latent features of AI companions that can provoke vulnerability exploitation through emotional manipulation, open the door to the prohibited practices of AI systems, when said systems employ manipulative or exploitative techniques that can cause significant harm, particularly to vulnerable persons. As the name suggests, such systems cannot, in principle, be placed on the market or used. 

In particular, AI companion systems often rely on anthropomorphic features, emotional mirroring, and persuasive design, which can blur the line between artificial interaction and human relationships. Although providers are subject to transparency obligations under the AI Act – requiring users to be informed that they are interacting with an AI system – such disclosures do not necessarily prevent emotional dependence or behavioural influence, raising concerns for mental health and social well-being. These risks are particularly acute for vulnerable users, including children, people with mental health difficulties, or those with limited social networks. 

Anthropomorphic design or emotional expressiveness as such are not prohibited under the AI Act. These features become legally relevant only where they are combined with manipulative or exploitative techniques that distort users’ behaviour and are capable of causing significant harm, particularly to vulnerable persons. 

The AI Act addresses such risks primarily through its prohibitions in Article 5, but without relying on a single, uniform definition of “vulnerability”. Under Article 5(1)(a), AI systems are prohibited where they use manipulative or deceptive techniques that are capable of causing significant physical or psychological harm, assessed on a case-by-case basis. According to the Commission’s Guidelines on prohibited AI practices, factors such as age or other forms of vulnerability are relevant when determining whether harm is “significant”. 

Article 5(1)(b) further prohibits AI systems that exploit vulnerabilities linked to age, disability or socio-economic situation where this exploitation leads, or is likely to lead, to significant harm through behavioral distortion. While enforcement practice is likely to be cautious at first, particularly given the novelty of these provisions and the high evidentiary threshold required to establish “significant harm”, this should not be interpreted as a safe harbour for AI companion providers. As supervisory practice, guidance and case law evolve, systems that combine emotional manipulation, anthropomorphic design and interaction with vulnerable users may increasingly attract regulatory scrutiny. 

In sum, a provider or deployer of an AI companion must at all times ensure that its product is not – or does not become, depending on technical developments, applicable standards and real-world use – a prohibited AI system. In other words, while the AI Act does not necessarily subject AI companions to the high-risk AI regime, it does de facto require providers to actively mitigate the risk that an AI companion could fall within the scope of prohibited AI practices. The exact probability thresholds at which such risks materialise are not crystal-clear and will likely continue to evolve in the coming years. 

In practical terms, providers and deployers of AI companions should (i) assess whether their product falls within the high-risk category and the corresponding requirements, and (ii) take appropriate measures to address the risk that the AI companion may fall within the scope of prohibited AI practices. 

2. GDPR and the intersection with AI Act

Because AI companions process user conversations, personal preferences, possibly sensitive data (e.g., when acting as romantic partners or therapists) – the rules under the GDPR apply as well. 

Users’ personal data must be handled in accordance with Article 5 GDPR, i.e., lawfully, transparently, accurately, securely, processed for explicit and legitimate purposes, only to the necessary extent in relation to the processing purpose, and stored for no longer than necessary for the processing operation. This may prove challenging in practice, notably given the diversity of information that users may share with the companion. 

Moreover, when AI systems learn from personal data (e.g., training on chat logs), additional safeguards, such as identifying an appropriate legal basis under Article 6 GDPR, applying pseudonymisation or anonymisation, and complying with data subject rights are essential. 

Both the AI Act and the GDPR encapsulate the accountability principle, which plays a critical role in the development and deployment of AI companions. Under the GDPR, this principle most concretely translates into the obligation to carry out a Data Protection Impact Assessment (DPIA), where AI companions involve profiling, large-scale monitoring, sensitive data inference, or interactions with vulnerable users. The DPIA must assess impacts on mental health, autonomy, and informational self-determination, and identify appropriate mitigation measures. Where an AI companion qualifies as a high-risk AI system, then the deployer bears the obligation to conduct a Fundamental Rights Impact Assessment (FRIA) prior to first use. While a DPIA does not replace the FRIA as a matter of law, a well-designed DPIA may meaningfully inform and facilitate the FRIA, notably where both assessments address overlapping risks and mitigation measures. It should nonetheless be recalled that the DPIA falls within the responsibility of the data controller under the GDPR, whereas the FRIA constitutes a distinct obligation incumbent on the deployer under the AI Act, meaning that the two assessments are not necessarily carried out by the same entity. 

One of the most powerful mechanisms under the GDPR for mitigating the critical risks of AI companions is the principle of data protection by design and by default (DPbD), embodied in Article 25. Controllers are required to implement appropriate technical and organisational measures to integrate data protection principles into the design, architecture, and operation of AI systems. For AI companions, DPbD provides a legally binding framework for addressing risks at their root, before entering the market. 

DPbD translates into different obligations. For example, as with the AI Act, GDPR requires developers to consider and mitigate (high) risk of psychological harm. To do so, several design restrictions can be contemplated: 

- limiting or disabling anthropomorphic features (e.g., voice, avatars, flirtatious or intimate responses); 

- eliminating emotional reinforcement loops; 

- embedding safe-response mechanisms for self-harm or crisis expressions; 

- incorporating transparency cues (e.g., reminders that the chatbot is non-human). 

These measures aim to reduce risks of dependency, grooming, exploitation, and distorted decision-making.

Also, transparency should be embedded at the interface level, not just in privacy policies. AI companions must provide: - clear, real-time disclosure of their artificial nature;

- intelligible information about data use, storage, and inference; 

- UI features that facilitate user understanding of risks. 

Access to raw conversation data shall be restricted, preventing misuse of emotionally charged content and avoiding unauthorised secondary uses. Technical measures include: 

- data compartmentalisation;

 - differential privacy; 

- encryption in transit and at rest; 

- minimised retention periods; 

- auditable logging of system outputs and model updates. 

These safeguards reduce the probability of data leakage, unauthorised profiling, inferential harms, as well as unauthorized secondary use. Said measures are also relevant as part of the data governance framework to implement under Article 10 AI Act, which mandates specific requirements for raw data used for training, validation and testing purposes, i.e., to be high-quality and relevant, sufficiently representative, properly governed, documented for their origin, and monitored for bias.

Especially for children, being a vulnerable group of society, both GDPR (Recital 38) and the AI Act (Article 5(1)(b)) point to: 

- age assurance systems designed with privacy-preserving methods; 

- prohibition of emotionally manipulative modelling; 

- increased monitoring for risky behaviours (e.g., sexualisation, dependency creation). 

Another point of intersection between the AI Act and the GDPR is human oversight (Articles 14 and 22 respectively), even though the scopes of the respective regimes do not entirely overlap. AI companions may be required to incorporate human oversight processes that: 

• allow intervention from qualified persons when high-risk patterns appear; 
• enable users to escalate concerns; 
• ensure human review of dangerous model outputs; 
• incorporate post-market monitoring (AI Act). 

This allows to effectively address risks to health, safety, and fundamental rights throughout the AI system’s lifecycle. 

Thus, when offering an AI companion, deployers must comply with obligations stemming from both legal frameworks, while being able to rely on overlapping governance mechanisms and safeguards across the two regimes. 

3. Liability regime of AI companions 

A pending question in the EU regulatory framework – and one especially relevant for AI companions – is how to treat AI systems legally: as products or services? This matters because liability regimes differ. 

• If the AI companion is considered a product, traditional product-safety and liability rules apply. 
• If considered a service, liability might depend on contractual or negligence frameworks. 

In the context of AI, it is difficult to opt for only one status. The classic legal divide between products – expected to function consistently – and services – whose outcomes hinge on professional judgment and situational factors – is increasingly difficult to maintain in the context of AI. A central element of AI systems is their unpredictability, added to inherent features, such as their complexity, autonomy and opacity (“black box effect”). Their behaviour and results depend on factors, such as model design and training, data quality, and performance context. 

Because AI performance is so heavily shaped by external elements, it resembles a service model in which liability depends on whether due care was exercised. However, the AI Act largely adopts a product-oriented approach, especially for high-risk AI systems. Through requirements such as conformity assessments, CE marking, and post-market monitoring, the Act embeds AI systems in a product safety framework, even without formally defining them as products in all contexts. 

This approach is reinforced by the new EU Product Liability Directive, which extends strict liability to software and AI-integrated products. Where harm is caused by a defective AI system, injured parties can claim compensation without proving fault. This contributes to legal certainty and reducing legal fragmentation. 

However, this harmonisation has limits. When harm stems from negligent actions of third parties (e.g., deployers, integrators, service providers), strict liability does not apply. Instead, claims fall back on national fault-based tort law, which remains largely unharmonised across Member States. This fragmentation results in differing evidentiary standards and definitions of negligence, uneven application of enforcement, different standards of AI rules application across Member States, and subsequent business uncertainty. 

The EU initially sought to address this gap through the proposed AI Liability Directive, aimed at easing the burden of proof in AI-related fault-based claims. Following criticism of overregulation, the proposal was withdrawn in February 2025. As a result, many AI liability scenarios, especially those involving improper use rather than defective design, remain governed by fragmented national rules. 

4. Practical implications for AI Companions 

For Developers and Companies 

• If you build or deploy an AI companion in the EU (or target EU users), you must map the use case and determine risk classification under the AI Act; 
• In any case, you will need to implement transparency measures; 
• Where an AI companion relies on a GPAI model provided by a third party, specific obligations apply to that provider. The resulting documentation and assessments may be leveraged when designing the AI companion. 
• If your AI system is high-risk, prepare for significant compliance obligations (risk assessments, monitoring, quality management, possibly external audits); 
• Take appropriate measures to ensure that your AI companion does not fall within the scope of prohibited AI practices; 
• Maintain robust data-protection practices (e.g., consent, data minimisation, secure storage, user rights), and involve the relevant teams within your company from the outset; 
• Be aware that liability may arise under product or service law: Even if the landscape now seems unclear, always apply high ethical development and deployment standards to avoid legal implications. 

For Users and Consumers 

• You have the right to know when you’re interacting with an AI companion; 
• You have rights over your personal data processed by an AI companion as a data subject (access, deletion, correction, portability, etc.); 
• If an AI companion causes harm (e.g., misinformation, manipulation, breach of privacy), you may have legal recourse – depending on how the system is classified – and seek compensation or other remedy that corresponds to the level of harm; 
• Transparency obligations should make the AI companion’s limitations, known biases, or potential risks clear. 

Some reflections for Regulators, Policymakers, and Civil Society 

• The EU’s risk-based approach needs ongoing scrutiny: What is considered low-risk now may become highly risky as technology advances (e.g., more human-like conversational agents, “always-on” companions). 
• Consider adding explicitly AI companions with anthropomorphic features to the list of GPAI modes with systemic risks or high-risk or prohibited AI systems, when the relevant conditions apply. Emerging initiatives in the US invite a broader conversation to re-examine how AI systems that interact emotionally with humans are shaping our social and personal landscapes. 
• A meaningful enforcement framework is likely to emerge only if the European Artificial Intelligence Office (AI Office) and national authorities are adequately resourced and possess the necessary technical and legal expertise. 
• Supporting user rights in practice (e.g., transparency, redress, audits) will require speeding up of standardisation and possibly new guidance at EU and national levels. 

Conclusion 

With the AI Act, Europe has positioned itself as a global leader in AI regulation. For AI companions, this means that – for the first time – there is a clear legal framework balancing innovation, safety, rights, and accountability. But the journey is just beginning. 

If you require any assistance in relation to the development or use of an AI companion, please do not hesitate to contact the Timelex team. Our specialised data lawyers support multiple AI companion (and AI systems) projects and closely monitor regulatory developments in this area.