A sweeping new piece of federal legislation quietly working its way through Parliament may fundamentally reshape how Canadians interact with artificial intelligence and not in the way the government is advertising.
Bill C-34, formally introduced as part of Ottawa’s Digital Safety Act, has been framed by its architects as a measured response to public concern over AI-facilitated harm, particularly in the wake of the Tumbler Ridge school shooting. But privacy advocates and legal experts are raising alarms about what the bill actually incentivizes once the fine print is examined: a surveillance architecture in which AI companies become de facto intermediaries between users and law enforcement.
“The optics are one thing. The mechanics are another,” said one legal researcher familiar with the bill, who spoke on background. “When you combine vague obligations with penalties reaching three percent of global revenue, you’re not regulating safety you’re creating enormous pressure for companies to monitor everything and report anything that might look risky.”
Section 51 sits at the heart of the controversy. It requires regulated chatbot services platforms like OpenAI’s ChatGPT, Google’s Gemini, and xAI’s Grok to immediately interrupt conversations in which users express suicidal ideation, intentions to self-harm, or intentions to harm others. These interrupted conversations must then be referred to what the legislation vaguely calls “crises intervention services appropriate to the situation.”
The problem, critics say, is what that phrase leaves unsaid. No framework is provided for determining which service is appropriate. The RCMP, under a plain reading of the bill, qualifies.
Section 58(1) makes things more explicit. It requires chatbot companies to submit safety plans to the Digital Safety Commission laying out their criteria and processes for escalating to the RCMP any conversations where “there is a risk that an individual will commit an act that would cause death or serious bodily harm to another individual.”
That standard risk of harm is extraordinarily broad. It could encompass discussions of mental health, historical atrocities, extremist ideology, fictional violence, or even journalism. And because the threshold for triggering a referral is never precisely defined, companies face an uncomfortable choice: build systems that are aggressively cautious and flag an enormous volume of false positives, or risk a regulatory penalty for under-reporting.
Most companies, when facing that kind of liability, will err toward the former.
To reliably catch conversations that cross a statutory threshold, any monitoring system must scrutinize far more conversations than it flags. That is the simple mathematics of surveillance: you cannot find the needle without sifting through the hay.
The practical result is that Canadians discussing sensitive topics mental illness, extremism for academic purposes, war, grief, self-harm in clinical contexts will be doing so under the assumption that their words are being evaluated and potentially escalated to a government body or police service.
This matters because of how profoundly Canadians’ relationship with AI has shifted. These tools are no longer novelties. Millions of Canadians now use chatbots for research, health inquiries, legal questions, and creative exploration. Placing a surveillance layer over those conversations has consequences that go far beyond the narrow category of dangerous speech the bill claims to target.
Legal precedent makes the privacy stakes even higher. The Supreme Court of Canada has long recognized that individuals retain a reasonable expectation of privacy in their electronic communications, even when those communications pass through third-party systems. Bill C-34 does not override that doctrine directly but it creates the conditions under which companies may voluntarily hand information to police that investigators would otherwise need judicial authorization to obtain.
If Section 51 is the bill’s surveillance engine, Section 53(e) is perhaps its most consequential long-term provision and its most quietly troubling.
Sections 49, 50, and 53 require chatbot services to implement measures mitigating harmful outputs: impersonating humans or professionals, fostering emotional dependency in users, encouraging violence or self-harm. These are defensible restrictions, even if broadly worded.
But Section 53(e) adds a catch-all: chatbots must also refrain from “any other type of behaviour specified in the regulations.” Those regulations do not yet exist. They will be written later, by a commission whose authority to define “harmful outputs” faces no statutory constraints.
This means that an increasingly critical gateway to information AI systems through which Canadians research, learn, and explore could be restricted in ways that Parliament has not debated and citizens cannot currently anticipate. It is a delegation of sweeping power over public discourse, wearing the modest clothing of a technical regulatory provision.
Here is where the incentive structure becomes self-reinforcing in ways the bill’s authors may not have fully anticipated.
Faced with unclear obligations and significant financial exposure, AI companies have several rational responses. They can monitor more aggressively and report more broadly, turning their platforms into surveillance tools to avoid any appearance of non-compliance. They can implement increasingly conservative generative models that restrict outputs in any area that might plausibly attract regulatory scrutiny. Or and this is the option Ottawa may least want to contemplate they can simply stop serving Canadian users altogether.
All three outcomes harm the Canadians the bill purports to protect. The first transforms a private conversation into a monitored channel. The second degrades AI as a tool for legitimate inquiry. The third removes access entirely.
What the bill does not create is a meaningful incentive to do any of this narrowly or proportionately. Precision costs money. Caution is cheaper. Over-compliance reduces liability. The market logic here does not point toward targeted, careful oversight it points toward surveillance as a business strategy.
If Bill C-34 passes in its current form, two sets of calculations change immediately.
For Canadians: any conversation with an AI chatbot about a sensitive topic violence, mental health, radicalization, political extremism, even certain works of literature should be conducted with the understanding that it may be reviewed, flagged, and potentially shared with law enforcement. This is not speculation. It is the logical consequence of what the bill requires.
For AI companies: the rational response to ambiguous obligations and severe penalties is not caution applied thoughtfully. It is caution applied broadly, even indiscriminately, to insulate the business from any possible charge of under-responding.
Parliament has not yet passed this bill. But the architecture it would create private AI companies as intermediaries between citizens and police, operating under vague mandates and substantial financial pressure is already visible in the legislation as written. The question Canadians may want to ask their representatives is not whether the government should regulate AI. It is whether this is the regulation they had in mind.
