AI has changed how quickly we build. Has security testing kept up?

AI has already changed the pace at which software can be built and improved.

We can see that in our own business. From a development perspective, AI is helping us produce more code, ship updates faster, connect systems more quickly and make the software our teams and customers use easier to work with.

That is exactly the kind of progress businesses should be looking to take advantage of.

But when the pace of change increases, the amount that needs validating increases with it.

The question is not whether organisations should slow down their use of AI. It is whether the way we review, test and assure what we are building is keeping pace.

AI has not removed the need for human review

Anyone using AI seriously in development will already recognise this principle.

AI can help a capable developer get to an answer faster. It can suggest code, remove repetitive work, support debugging and make it easier to explore different approaches.

But the developer still needs to understand what has been produced, review it in context and validate that it does what was intended.

That human validation is not a sign that AI has failed. It is part of using the technology responsibly and effectively.

We think AI security testing needs to follow the same principle.

If AI is helping organisations produce more functionality, create more integrations and release changes more frequently, there is naturally more change to assess.

Independent testing becomes more relevant, not less.

More code is only part of the change

The increase in development pace is important, but AI is also changing the architecture of the systems businesses are building.

An AI chatbot may be connected to internal knowledge.

A copilot may be able to retrieve customer information.

An AI-powered application may call APIs.

An AI agent may be able to use tools, update records, execute a process or act on someone's behalf.

Each of those capabilities can make the experience more useful. They also create new trust boundaries, permissions, data flows and behaviours that need to be understood.

This is why AI security testing cannot stop at the model.

The model is only one part of the system

It is easy to think of LLM security as a question for the model provider.

In reality, organisations control a large part of the environment around the model: what information it receives, which systems it can reach, how users are separated, what happens to its outputs and which actions it is permitted to take.

Effective AI security testing therefore needs to look across the wider implementation, including:

• Prompts and context: Can manipulated instructions change behaviour in a way that matters?
• Data: What information can the AI retrieve, expose, retain or pass between users?
• Permissions and identity: Does the AI have more access than the user or task actually requires?
• APIs and integrations: Can weaknesses in connected services be reached or abused through the AI feature?
• Tools and actions: If the AI can do something, are approval steps and technical controls working as intended?
• The application and infrastructure: Are conventional web, API, cloud or configuration weaknesses present around the AI capability?

This is not about assuming every AI feature is unsafe.

It is about testing the assumptions that have been made as the technology is connected to more of the organisation.

Why human-led testing still matters

Automation is useful in security testing, just as it is useful in development.

It can improve coverage, speed up repetitive work and help identify areas that deserve closer investigation.

But real systems are contextual.

A finding that looks low impact in isolation may become much more important when it is combined with another behaviour.

An agent may react differently depending on its goal, memory, permissions or the tool it is using. A chatbot may appear well controlled until the tester understands how its knowledge source, user separation or API connections work.

Experienced human testers can adapt as the system responds, follow those chains and determine what a weakness actually allows someone to do.

That is why manual, consultant-led testing remains a core part of how we approach AI security testing at Secarma.

When should organisations be testing AI?

There is no single point at which AI security testing becomes relevant, but there are some useful triggers.

Testing is worth considering before an AI-enabled feature is launched, before it is connected to more sensitive information, when new APIs or tools are introduced, when an agent is given more autonomy, or when a significant change alters what the system can access or do.

The more capability an AI feature gains, the more useful it becomes to revisit the assumptions around its access and controls.

Why we introduced dedicated AI security testing services

This change in how systems are being built is why Secarma has introduced a dedicated suite of AI security testing services.

Our AI Integration Security Testing looks at the complete AI-powered feature together with the application, APIs, data, tools and infrastructure around it.

We can then go deeper where required through:

• LLM Application Security Testing
• AI Chatbot Security Testing
• AI Agent Security Testing

The right scope depends on how AI is being used, what it can access and what it is capable of doing.

Keep moving quickly, with confidence

AI is creating a genuine opportunity for organisations to build more, remove friction and improve the experience they provide to customers and employees.

Security should enable that progress, not become the reason it slows down.

The aim of good security testing is to give teams the evidence they need to keep moving: to understand where controls are working, where an assumption needs challenging and which improvements should be prioritised before the next stage of deployment.

If you are currently introducing AI into an application, chatbot, copilot or agent and are unsure what should actually be tested, get in touch with Secarma. We can help you work through the architecture and identify the right testing scope.

Or, if you would rather start with the practical principles, join our upcoming webinar, AI Is Now Part of Your Attack Surface. Are You Testing It?, on Wednesday 16 September at 2pm.