How AI Is Changing Software Testing and VAPT

Key Takeaways

• AI in testing has progressed from generating test cases toward generating, executing, analysing, and maintaining parts of the testing workflow.
• Industry survey data shows substantial adoption of AI-powered testing, but most organisations are still in experimentation or limited-use stages.
• Security testing benefits from AI in reconnaissance, test generation, evidence correlation, source-code analysis, and reporting.
• AI does not remove the need for testing expertise; it increases the value of validation, attack-path reasoning, false-positive elimination, and impact assessment.

The first phase: AI as a test-writing assistant

The earliest practical use of generative AI in testing was straightforward:

Requirement
    ↓
AI
    ↓
Test cases

A tester could provide:

Create test cases for login.

and receive:

Valid login
Invalid password
Invalid username
Empty credentials
Account lockout
Session timeout
...

This reduced the effort involved in repetitive documentation and brainstorming.

The limitation was equally obvious.

The human still had to:

understand the requirement
design the strategy
execute the tests
analyse the results

AI assisted the tester but did not fundamentally change the workflow.

The second phase: AI-generated automation

The next step was moving from:

AI → test cases

to:

AI → executable tests

For example:

Requirement
    ↓
AI
    ↓
Playwright / Selenium / API test
    ↓
Execute
    ↓
Report

This reduced the amount of manual scripting needed to automate repetitive scenarios.

But automation generation introduces a different problem:

A generated test can be executable
without being a good test.

An AI-generated script may technically run while:

asserting the wrong thing
missing an edge case
using brittle selectors
ignoring business logic
testing implementation instead of behaviour

The human still has to verify the test’s validity.

AI adoption is no longer theoretical

Katalon’s 2025 State of Software Quality Report surveyed more than 1,500 quality professionals. It reported that 76% were using AI-powered tools in their testing activities, while 82% considered AI critical to the future of testing. At the same time, the report found that 56% of QA teams still struggled to keep up with testing demands.

The World Quality Report 2025-26 presents a similar transition. It reported that 15% of respondents had reached enterprise-wide implementation of GenAI in Quality Engineering, while 43% remained in an experimental phase and 30% were using limited use cases.

That is important.

The industry is not at:

AI completely runs testing.

It is closer to:

AI is being integrated into more parts of testing.

The shift from outputs to inputs

An interesting change reported in the World Quality Report is that GenAI adoption is moving beyond analysing outputs such as defect reports toward shaping inputs such as test-case design and requirements refinement.

The workflow therefore becomes:

Old:

Requirement
   ↓
Human
   ↓
Test
   ↓
Result
   ↓
AI analyses result

Increasingly:

Requirement
   ↓
AI assists test design
   ↓
Test
   ↓
Execution
   ↓
AI assists analysis

AI is appearing at multiple points in the lifecycle.

Test data generation

Test data is another major area.

Traditional testing often requires:

Create users
Create transactions
Create edge cases
Create malformed data
Maintain datasets

AI can increasingly generate representative datasets based on defined constraints.

Katalon’s research also highlights AI-driven test generation and optimisation as active areas of adoption, while industry research continues to explore LLM-based test generation and constraint mining for APIs.

The important limitation is that:

syntactically valid data

does not necessarily mean:

business-valid data

A transaction generated by AI may satisfy a schema while violating a real business invariant.

AI-assisted defect analysis

Failure analysis is particularly suited to AI because the raw material is often highly repetitive.

A traditional process:

Test fails
   ↓
Read log
   ↓
Read stack trace
   ↓
Check recent commit
   ↓
Search related issue
   ↓
Form hypothesis

An AI-assisted process:

Failure
  +
Logs
  +
Stack trace
  +
Recent changes
  +
Environment information
       ↓
AI
       ↓
Candidate root cause
       ↓
Human verification

The critical distinction is:

AI-generated hypothesis
≠
proved root cause

AI is good at correlation.

Testing still requires proof.

AI enters security testing

The same progression is occurring in VAPT.

Reconnaissance

AI can help organise:

domains
subdomains
endpoints
technologies
headers
API specifications
source-code paths

The important change is not that AI replaces scanners.

It can help interpret the results from multiple scanners and connect them into a more coherent attack surface.

Test generation

A security Skill or agent can generate candidate tests for:

authentication
authorisation
input validation
session management
business logic
API security
mobile security
configuration

Evidence correlation

Security engagements frequently produce large amounts of data:

Burp traffic
Nmap results
Nuclei output
source code
logs
screenshots
API documentation
scanner findings

AI can correlate those sources into a single investigation.

The real shift: from tools to workflows

Traditional VAPT often resembles:

Tester
 ↓
Tool
 ↓
Output
 ↓
Tester

Agentic VAPT can look like:

Tester
 ↓
Security Agent
 ├── Proxy
 ├── Scanner
 ├── Browser
 ├── Source code
 ├── Logs
 └── Documentation
 ↓
Candidate findings
 ↓
Validation
 ↓
Report

The agent becomes the orchestration layer.

The tester increasingly becomes responsible for:

scope
hypotheses
test strategy
validation
impact
risk

Agentic testing is the next progression

The difference between an assistant and an agent is the number of steps the system can perform without requiring a new human instruction at every stage.

Assistant:

Write an API test.

Agent:

Assess this API's authentication workflow.

The agent may then:

Discover endpoints
    ↓
Read API documentation
    ↓
Inspect authentication flow
    ↓
Generate candidate tests
    ↓
Execute them
    ↓
Analyse responses
    ↓
Generate additional tests
    ↓
Correlate evidence
    ↓
Produce findings

This is where AI changes testing from a collection of isolated tools into an orchestrated workflow.

The danger of false confidence

Security testing has a fundamental rule:

Interesting behaviour
≠
Vulnerability

AI can produce a highly convincing explanation for something that is not actually exploitable.

Consider:

AI:
"Potential authorisation bypass detected."

That statement proves nothing.

The tester still has to establish:

Can an unauthorised identity reproduce it?
Can the protected resource actually be accessed?
Does the server enforce the correct state?
Is the behaviour intended?
What is the actual impact?

This may make validation more important as AI-generated findings increase.

A useful multi-agent validation model

A security workflow can separate discovery from validation:

Agent A
Candidate finding
      ↓
Agent B
Reproduction
      ↓
Agent C
Challenge / false-positive review
      ↓
Human
Final determination

The third stage is especially valuable.

Its job is not to prove the finding.

Its job is to ask:

What would make this conclusion wrong?

That is a useful defence against confirmation bias in automated workflows.

Testing is becoming more hybrid

Katalon’s 2025 report describes a growing “hybrid tester” model that combines manual testing, automation, and AI-augmented testing. The report also states that 68% of respondents consider automation scripting and programming skills essential to success.

That is consistent with the broader direction of testing.

The tester is not necessarily becoming less technical.

The technical skillset is changing from:

manual execution

toward:

manual reasoning
+
automation
+
AI orchestration
+
validation

What becomes more valuable

AI is particularly effective at repetitive operations.

Therefore the human advantage shifts toward areas where judgement is difficult to encode:

attack-path reasoning
business logic
contextual risk
novel abuse cases
false-positive elimination
impact analysis

An AI system can find:

403

more quickly.

The tester still has to determine whether the protected operation can actually be reached under the relevant threat model.

The progression

The evolution can be represented as:

AI writes test cases
        ↓
AI writes automation
        ↓
AI generates test data
        ↓
AI analyses failures
        ↓
AI assists test planning
        ↓
AI orchestrates tools
        ↓
AI executes multi-step workflows
        ↓
Agentic testing

The exact pace differs across organisations, but current surveys already show substantial adoption combined with relatively low enterprise-wide deployment, indicating that the transition is still underway.

Validation principles for AI-assisted VAPT

• Treat AI-generated findings as candidate findings.
• Reproduce security impact independently.
• Keep raw evidence separate from AI-generated interpretation.
• Verify that generated tests actually satisfy the security objective.
• Use adversarial review to challenge high-impact findings.
• Record the tools and actions performed by autonomous workflows.
• Keep final severity and business-impact determination under human control.

References

• Katalon — State of Software Quality Report 2025.
• Capgemini / Sogeti / OpenText — World Quality Report 2025-26.
• Katalon — Software Testing Research and LLM-based Testing Research.
• Katalon — Test Automation Statistics and Trends for 2025.