ISTQB AI Testing

The Certified Tester AI Testing certification is aimed at anyone involved in testing AI-based systems and/or AI for testing. This includes people in roles such as testers, test analysts, data analysts, test engineers, test consultants, test managers, user acceptance testers, and software developers. This certification is also appropriate for anyone who wants a basic understanding of testing AI-based systems and/or AI for testing, such as project managers, quality managers, software development managers, business analysts, operations team members, IT directors, and management consultants.

The participants must hold the ISTQB/ISEB Foundation Certificate in Software Testing. As the course material and the certification exam are in English, the participants are expected to have good command of English language

Chapter 1: Introduction to AI

• Definition of AI and AI Effect
• Narrow, General and Super AI
• AI-based and Conventional Systems
• AI Technologies
• AI Development Frameworks
• Hardware for AI-Based Systems
• AI as a Service (AIaaS)
• Pre-Trained Models
• Standards, Regulations and AI

Chapter 2: Quality Characteristics for AI-Based Systems

• Flexibility and Adaptability
• Autonomy
• Evolution, Bias and Ethics
• Side Effects and Reward Hacking
• Transparency, Interpretability and Explainability
• Safety and AI

Chapter 3: Machine Learning (ML) –Overview

• Forms of ML and ML Workflow
• Selecting a Form of ML

Chapter 4: ML – Data

• Data Preparation as Part of the ML Workflow
• Training, Validation and Test Datasets in the ML Workflow
• Dataset Quality Issues
• Data Quality and its Effect on the ML Model
• Data Labelling for Supervised Learning

Chapter 5: ML Functional Performance Metrics

• Confusion Matrix
• Additional ML Functional Performance Metrics for Classification, Regression and Clustering
• Limitations of ML Functional Performance Metrics
• Selecting ML Functional Performance Metrics
• Benchmark Suites for ML Performance

Chapter 6: ML – Neural Networks and Testing

• Neural Networks
• Coverage Measures for Neural Networks

Chapter 7: Testing AI-Based Systems Overview

• Specification of AI-Based Systems
• Test Levels for AI-Based Systems
• Test Data for Testing AI-Based Systems
• Testing for Automation Bias in AI-Based Systems
• Documenting an AI Component
• Testing for Concept Drift
• Selecting a Test Approach for an ML System

Chapter 8: Testing AI-Specific Quality Characteristics

• Challenges Testing Self-Learning Systems
• Testing Autonomous AI-Based Systems
• Testing for Algorithmic, Sample and Inappropriate Bias
• Challenges Testing Probabilistic and Non-Deterministic AI-Based Systems
• Challenges Testing Complex AI-based Systems
• Testing the Transparency, Interpretability and Explainability of AI-Based Systems
• Test Oracles for AI-Based Systems
• Test Objectives and Acceptance Criteria

Chapter 9: Methods and Techniques for the Testing of AI-Based Systems

• Adversarial Attacks and Data Poisoning
• Pairwise Testing
• Back-to-Back Testing
• A/B Testing
• Metamorphic Testing (MT)
• Experience-based testing of AI-based Systems
• Selecting Test Techniques for AI-based Systems

Chapter 10: Test Environments for AI-Based Systems

• Test Environments for AI-Based Systems
• Virtual Test Environments for Testing AI-Based Systems

Chapter 11: Using AI for Testing

• AI Technologies for Testing
• Using AI to Analyze Reported Defects
• Using AI for Test Case Generation
• Using AI for the Optimization of Regression Test Suites
• Using AI for Defect Prediction
• Using AI for Testing User Interfaces

Course begins at 9.00 and ends at 16.00 - 16.30. Breakfast is served from 8.15 onwards.