Certified Tester AI Testing (CT-AI)

This four-day course provides a comprehensive introduction to Artificial Intelligence (AI) and its application in modern systems. Participants will explore the foundational concepts of AI, including its types, technologies, and development frameworks, as well as the unique quality characteristics that distinguish AI-based systems—such as autonomy, adaptability, ethics, and transparency. The course also covers the essentials of Machine Learning (ML), from algorithm selection and data preparation to performance metrics and neural networks, equipping learners with a solid understanding of how ML models are developed and evaluated.

Building on this foundation, the course delves into the challenges and methodologies of testing AI-based systems. Learners will examine test strategies for AI-specific traits like bias, non-determinism, and concept drift, and gain hands-on insight into techniques such as adversarial testing, metamorphic testing, and A/B testing. The final sessions focus on test environments and the use of AI to enhance software testing processes, including defect analysis and regression optimization. By the end of the course, participants will be equipped to critically assess, test, and apply AI technologies in real-world scenarios.

By the end of this course, learners will be able to:

• Understand the foundational concepts of Artificial Intelligence, including its types, technologies, and development frameworks.
• Explore the quality characteristics specific to AI-based systems, such as adaptability, autonomy, ethics, and transparency.
• Gain a comprehensive overview of Machine Learning (ML), including its forms, workflows, and algorithm selection criteria.
• Learn the importance of data in ML, including data preparation, dataset types, and the impact of data quality on model performance.
• Evaluate ML performance using functional metrics and benchmark suites for classification, regression, and clustering tasks.
• Understand neural networks and their testing methodologies, including coverage measures and concept drift.
• Apply testing strategies tailored to AI-based systems, addressing specification, test levels, and automation bias.
• Examine challenges in testing AI-specific quality traits, such as bias, non-determinism, and explainability.
• Explore various testing techniques for AI systems, including adversarial testing, metamorphic testing, and A/B testing.
• Discover how AI can be leveraged to enhance software testing processes, including defect analysis, test case generation, and regression optimization.

The entry criterion for taking the Certified Tester AI Testing exam is that candidates have acquired the ISTQB® Certified Tester Foundation Level certification.

Target audience

The Certified Tester AI Testing is suitable for anyone who is involved in testing as well as anyone interested in AI-based systems. This includes people performing activities such as test analysis, test consulting and software development.

The syllabus provides testing knowledge for anyone working with Agile or sequential software development lifecycles.

Chapter 1: Introduction to AI

• Definition of AI and AI Effect
• Narrow, General and Super AI
• AI-based and Conventional Systems
• AI Technologies
• AI Development Frameworks
• Hardware for AI-Based Systems
• AI as a Service (AIaaS)
• Pre-Trained Models
• Standards, Regulations and AI

Chapter 2: Quality Characteristics for AI-Based Systems

• Flexibility and Adaptability
• Autonomy
• Evolution
• Bias
• Ethics
• Side Effects and Reward Hacking
• Transparency, Interpretability and Explainability
• Safety and AI

Chapter 3: Machine Learning (ML) – Overview

• Forms of ML
• ML Workflow
• Selecting a Form of ML
• Factors Involved in ML Algorithm Selection
• Overfitting and Underfitting

Chapter 4: ML – Data

• Data Preparation as Part of the ML Workflow
• Training, Validation and Test Datasets in the ML Workflow
• Dataset Quality Issues
• Data Quality and its Effect on the ML Model
• Data Labelling for Supervised Learning

Chapter 5: ML Functional Performance Metrics

• Confusion Matrix
• Additional ML Functional Performance Metrics for Classification, Regression and Clustering
• Limitations of ML Functional Performance Metrics
• Selecting ML Functional Performance Metrics
• Benchmark Suites for ML Performance

Chapter 6: ML – Neural Networks and Testing

• Neural Networks
• Coverage Measures for Neural Networks
• Chapter 7: Testing AI-Based Systems Overview
• Specification of AI-Based Systems
• Test Levels for AI-Based Systems
• Test Data for Testing AI-Based Systems
• Testing for Automation Bias in AI-Based Systems
• Documenting an AI Component
• Testing for Concept Drift
• Selecting a Test Approach for an ML System

Chapter 8: Testing AI-Specific Quality Characteristics

• Challenges Testing Self-Learning Systems
• Testing Autonomous AI-Based Systems
• Testing for Algorithmic, Sample and Inappropriate Bias
• Challenges Testing Probabilistic and Non-Deterministic AI-Based Systems
• Challenges Testing Complex AI-based Systems
• Testing the Transparency, Interpretability and Explainability of AI-Based Systems
• Test Oracles for AI-Based Systems
• Test Objectives and Acceptance Criteria

Chapter 9: Methods and Techniques for the Testing of AI-Based Systems

• Adversarial Attacks and Data Poisoning
• Pairwise Testing
• Back-to-Back Testing
• A/B Testing
• Metamorphic Testing (MT)
• Experience-based testing of AI-based Systems
• Selecting Test Techniques for AI-based Systems

Chapter 10: Test Environments for AI-Based Systems

• Test Environments for AI-Based Systems
• Virtual Test Environments for Testing AI-Based Systems

Chapter 11: Using AI for Testing

• AI Technologies for Testing
• Using AI to Analyze Reported Defects
• Using AI for Test Case Generation
• Using AI for the Optimization of Regression Test Suites
• Using AI for Defect Prediction
• Using AI for Testing User Interfaces

Exams and Assessments

Your course fee includes an iSQI voucher for the examination which you will book at a later date.

The format of the exam is multiple choice.

• Exam duration is 60 minutes. If the candidate’s native language is not the examination language, the candidate is allowed an additional 25% (exam duration = 75 minutes).
• There are 40 questions.
• To pass the exam, at least 65% of the total sum of points must be answered correctly.
• The total number of points for this exam should be set at 47 points. Therefore, a minimum of 31 points is required to achieve a passing score.

Hands-On Learning

Hands-on Machine Learning Concepts:

Learners engage in exercises that illustrate key ML concepts such as overfitting and underfitting. Activities include creating simulated datasets, training simple models (like linear regression), and visualizing model performance under different data conditions (e.g., limited data, weak feature-target correlations). Participants analyze results using metrics like Mean Squared Error (MSE) and R², and interpret graphical outputs to understand model behavior.

Test Design and Reduction Techniques:

One exercise focuses on combinatorial test design. Learners are tasked with defining a model with multiple parameters (e.g., model type, number of estimators, training rate, etc.), generating a large set of possible parameter combinations, and then applying pairwise testing to reduce the number of test cases. This introduces practical skills in test optimization and the use of tools (such as Microsoft PICT) for efficient test coverage.