Lessons from NZ Government AI Adoption
Across New Zealand’s public service, generative AI is moving from experiment to infrastructure. The Public Service AI Framework, sitting within the National AI Strategy launched in 2025, is guiding agencies toward AI use that is “innovative and trustworthy” and respects human rights and democratic values. The ambition is real: better citizen services, faster processing, and more efficient government. But the pace of adoption may be outrunning public trust, and that gap is exactly where security and privacy need to catch up.
The trust problem is already visible. According to the Office of the Privacy Commissioner’s 2026 annual privacy survey, 67% of New Zealanders are concerned about government agencies and businesses using AI to make decisions about them with their personal data—up five points on the previous year. Trust in government agencies to handle personal information responsibly sits at just 31%.
Why LLMs raise the stakes. Large language models don’t just process data the way traditional software does—they can memorise fragments of training or input data, be manipulated through prompt injection to reveal information or bypass controls, and produce confident-sounding but inaccurate outputs that get treated as fact. When these systems touch personal information—health records, benefit applications, tax details—the consequences of a security gap are not hypothetical. A leaked prompt, a poorly sandboxed integration, or an over-permissioned AI assistant could expose exactly the kind of sensitive data citizens are already anxious about.
What responsible use actually requires. The OPC’s guidance on AI and the Information Privacy Principles, first issued in 2023 and still the reference point for agencies, sets out clear expectations: senior leadership sign-off on AI projects informed by documented risk assessments, Privacy Impact Assessments and Algorithm Impact Assessments before deployment, and meaningful human review for decisions that affect individuals. Agencies remain accountable for outcomes—using an AI tool doesn’t transfer responsibility away from the agency under the Privacy Act. Where third-party AI providers are involved, contracts need to explicitly prevent those providers from retaining or repurposing citizens’ data beyond its intended use.
There’s also a distinctly local dimension: the guidance recognises the need to engage proactively with Māori communities on data sovereignty and te ao Māori perspectives on privacy—a reminder that “privacy” isn’t a one-size-fits-all concept, even within a single country’s framework.
The bottom line. AI can genuinely improve how government serves people, but only if security is built in, not bolted on. That means treating LLM security as a first-class requirement alongside functionality: securing data pipelines, testing for prompt injection and data leakage, keeping humans in the loop on consequential decisions, and being transparent with citizens about when and how AI is used. Public trust, once lost, is hard to rebuild—and in New Zealand’s case, the survey data suggests agencies are already starting from a deficit they can’t afford to widen.
CANDA’s AI engagement’s to date have greatly increased our experience in understanding the pitfalls of early adoption. Without a complete understanding of the security risks and considerations needed to be built into actionable design requirements, security certification can become very problematice, causing delays and extra cost. Let us share with you the benefits of our experience, to help enable AI for your organisation securely.
GenAI Security
