Retrieval Evaluation & RAG Optimizationin Sydney

Module 1: Introduction to Retrieval Evaluation and RAG Optimization

• Understand the fundamentals of Retrieval-Augmented Generation (RAG).
  • Explore the role of retrieval systems in providing relevant context to Large Language Models (LLMs).
  • Identify common challenges in RAG applications, including irrelevant retrieval, incomplete context, and unsupported answers.
• Understand the importance of systematic RAG evaluation.
  • Differentiate retrieval quality from generated response quality.
  • Explore how evaluation supports continuous improvement of AI applications.

Module 2: RAG Architecture and Retrieval Pipeline Analysis

• Examine the components of an end-to-end RAG architecture.
  • Study document ingestion, preprocessing, chunking, embedding generation, indexing, retrieval, and response generation.
  • Understand how information flows between retrieval components and LLMs.
• Identify performance bottlenecks in retrieval pipelines.
  • Analyse query processing, search latency, context selection, and response generation.
  • Locate potential sources of retrieval errors and context loss.

Module 3: Evaluation Datasets and Test Case Development

• Design datasets for evaluating RAG applications.
  • Prepare representative questions, reference answers, relevant document passages, and expected retrieval results.
  • Include factual, contextual, multi-step, and domain-specific queries.
• Develop reliable evaluation test cases.
  • Create test scenarios for common user queries and challenging retrieval conditions.
  • Organise evaluation datasets to support repeatable experiments and regression testing.

Module 4: Retrieval Quality Metrics and Benchmarking

• Evaluate retrieval effectiveness using appropriate metrics.
  • Understand precision, recall, Hit Rate, Mean Reciprocal Rank (MRR), and Normalized Discounted Cumulative Gain (NDCG).
  • Interpret ranking results and identify gaps in retrieved information.
• Establish a retrieval benchmarking process.
  • Compare retrieval configurations using consistent datasets and evaluation criteria.
  • Analyse metric trade-offs based on application requirements.

Module 5: Embedding Models and Semantic Search Optimization

• Evaluate embedding models for domain-specific retrieval tasks.
  • Compare semantic similarity, query-document alignment, and retrieval effectiveness.
  • Examine the influence of embedding model selection on search quality.
• Optimise semantic search configurations.
  • Assess similarity measures, metadata filters, and vector search parameters.
  • Identify opportunities to improve relevant document discovery.

Module 6: Document Chunking and Context Optimization

• Evaluate document segmentation strategies.
  • Compare fixed-size, overlapping, sentence-based, and structure-aware chunking approaches.
  • Examine how chunk size and overlap influence retrieval precision and context completeness.
• Improve the quality of retrieved context.
  • Evaluate metadata enrichment and document structure preservation.
  • Reduce irrelevant, duplicated, or incomplete passages supplied to LLMs.

Module 7: Reranking and Advanced Retrieval Strategies

• Understand reranking techniques for improving search relevance.
  • Explore cross-encoder rerankers and other ranking approaches.
  • Compare initial retrieval rankings with reranked results.
• Optimise retrieval strategies for complex queries.
  • Examine hybrid search combining lexical and semantic retrieval.
  • Explore query rewriting, query expansion, and multi-query retrieval where appropriate.

Module 8: LLM Response Evaluation and Faithfulness Assessment

• Assess the quality of generated responses.
  • Evaluate answer relevance, correctness, completeness, and contextual alignment.
  • Identify responses that contain unsupported statements or omit important information.
• Measure faithfulness to retrieved evidence.
  • Examine whether generated claims are supported by the supplied context.
  • Use reference-based and reference-free evaluation approaches where suitable.

Module 9: RAG Evaluation Frameworks and Automated Testing

• Explore tools and frameworks for RAG assessment.
  • Understand evaluation workflows using suitable frameworks such as Ragas and other LLM evaluation tools.
  • Review the role of automated metrics, human review, and LLM-as-a-judge methods.
• Implement repeatable evaluation processes.
  • Configure evaluation datasets, metrics, test runs, and result tracking.
  • Understand the limitations of automated scoring and the importance of validating evaluation outputs.

Module 10: Root Cause Analysis and RAG Performance Tuning

• Diagnose common RAG quality issues.
  • Investigate poor retrieval, misleading context, incorrect answers, and inconsistent responses.
  • Trace quality problems to data preparation, indexing, retrieval, ranking, or generation stages.
• Apply targeted optimization techniques.
  • Compare alternative chunking, embedding, retrieval, and reranking configurations.
  • Document changes and evaluate their effects against a consistent baseline.

Module 11: RAG Regression Testing and Continuous Evaluation

• Establish regression testing for RAG applications.
  • Maintain benchmark datasets to detect changes in retrieval and response quality.
  • Test the impact of updates to documents, embedding models, prompts, and retrieval settings.
• Design continuous evaluation workflows.
  • Track evaluation metrics across development and deployment stages.
  • Define review thresholds and investigate quality regressions before release.

Module 12: Capstone Project — Evaluate and Optimize a RAG Application

• Build an evaluation plan for a practical RAG use case.
  • Define the problem statement, evaluation dataset, baseline configuration, and success criteria.
  • Select suitable retrieval and response-quality metrics.
• Implement and assess optimization strategies.
  • Experiment with chunking, embeddings, retrieval parameters, and reranking techniques.
  • Compare baseline and optimized results and document limitations.
• Present the final evaluation report.
  • Summarise findings, metric comparisons, recommended changes, and future improvements.
  • Demonstrate how the evaluation process supports more reliable knowledge-based AI applications through StepMerit's training in Sydney, Australia.