Retrieval Evaluation & RAG Optimization in New Caledonia

Problem Statement

An educational organization implemented a Retrieval-Augmented Generation (RAG) application to help students and faculty access information from academic regulations, course materials, examination guidelines, admission policies, and institutional documents. However, the knowledge assistant frequently retrieved irrelevant passages, missed important information, and generated incomplete or insufficiently supported answers.

Students sometimes received outdated policy information, while faculty members found it difficult to identify the source of inaccurate responses. The organization lacked a structured evaluation process to measure retrieval relevance, verify answer quality, and determine which components of the RAG pipeline required improvement.

The key challenge was to improve the accuracy, relevance, consistency, and reliability of the educational knowledge assistant through systematic retrieval evaluation and RAG optimization.

How Did We Overcome the Problem?

Learners applied the techniques covered in the Retrieval Evaluation & RAG Optimization course to investigate retrieval and response-quality issues systematically.

• Analysed Existing RAG Performance – Examined sample student queries, retrieved document passages, generated answers, and failure cases to identify weaknesses across the retrieval and generation pipeline.
• Created an Educational Evaluation Dataset – Prepared representative questions based on course syllabi, examination rules, admission policies, attendance requirements, and student support documents. Relevant source passages and reference answers were documented where available.
• Evaluated Retrieval Effectiveness – Applied metrics such as Precision, Recall, Mean Reciprocal Rank (MRR), and Normalized Discounted Cumulative Gain (NDCG) to assess whether the system retrieved relevant educational information and ranked it appropriately.
• Optimized Document Chunking and Metadata – Reviewed document segmentation, chunk size, overlap, and metadata fields to preserve important context, such as course names, academic years, policy categories, and document versions.
• Refined Search and Ranking Strategies – Compared suitable embedding models, semantic search configurations, metadata filters, and reranking techniques to improve the relevance of retrieved content.
• Assessed Generated Answer Quality – Evaluated answer relevance, contextual faithfulness, completeness, and factual support to identify responses that contained unsupported claims or omitted important details.
• Introduced Repeatable Testing – Established a consistent evaluation process to compare baseline and optimized configurations, record findings, and check for quality regressions when documents or retrieval settings changed.

Overall Implementation

The educational organization followed a structured approach to evaluate and optimize its RAG application.

Data Collection and Preparation

Academic handbooks, course materials, examination policies, admission guidelines, and frequently asked questions were collected from approved institutional sources. Documents were cleaned, organized, and prepared for indexing, with appropriate metadata and document-version information.

Baseline RAG Evaluation

A representative set of student and faculty queries was used to assess the existing system. Retrieval rankings, relevant passages, generated answers, and observed failure cases were recorded to establish a baseline for comparison.

Retrieval Pipeline Optimization

The team tested alternative chunking strategies, embedding configurations, metadata filters, and retrieval methods. Where appropriate, reranking was introduced to improve the ordering of candidate passages before context was passed to the language model.

Response Quality and Faithfulness Testing

Generated responses were compared with the retrieved evidence and available reference answers. The evaluation identified whether answers were relevant, supported by institutional documents, complete enough to address the question, and appropriately qualified when information was unavailable.

Regression Testing and Documentation

The evaluation dataset was retained for repeated testing. Changes to document collections, embedding models, prompts, and retrieval configurations were assessed against the established baseline to detect potential reductions in quality.

Review and Continuous Improvement

Evaluation findings were documented and shared with the development and academic teams. Recommendations were prioritized according to the severity of the observed issues, the importance of the educational information, and the results of comparative testing.

Learner Outcomes

• Design Evaluation Datasets – Develop evaluation datasets for education-focused RAG applications using representative queries, reference answers, and relevant source passages.
• Measure Retrieval Performance – Apply appropriate information retrieval metrics to assess search relevance, ranking quality, and retrieval effectiveness.
• Diagnose RAG Quality Issues – Identify problems associated with document chunking, embeddings, search relevance, ranking, and context selection.
• Optimize Retrieval Configurations – Compare retrieval approaches and recommend evidence-based improvements to RAG pipelines.
• Evaluate Generated Responses – Assess answer relevance, factual support, completeness, and faithfulness to retrieved context.
• Implement Repeatable Testing – Establish regression testing and continuous evaluation workflows to monitor changes in RAG performance.
• Communicate Evaluation Findings – Prepare technical reports that explain results, limitations, optimization decisions, and recommendations.

Evidence and Deliverables

• Problem Analysis Report – Documents the initial retrieval and response-quality challenges.
• Evaluation Dataset – Contains representative educational queries, reference answers, and relevant source passages.
• Baseline Evaluation Report – Records initial retrieval metrics, answer-quality observations, and identified failure cases.
• Optimization Experiment Log – Documents tested chunking strategies, embedding configurations, retrieval settings, and reranking alternatives.
• Comparative Results Report – Presents baseline and optimized results using consistent datasets and evaluation criteria.
• RAG Test Suite – Includes repeatable test cases for retrieval relevance, contextual faithfulness, and regression checking.
• Implementation Documentation – Describes the evaluation workflow, optimization decisions, limitations, and recommendations for future improvements.
• Final Project Presentation – Summarizes the problem, approach, implementation, evaluation evidence, and proposed next steps.

Expected Organizational Benefits

A structured evaluation and optimization process can help the educational organization identify retrieval weaknesses, improve access to relevant academic information, reduce unsupported responses, and maintain greater consistency as institutional documents evolve. It can also support more transparent review of AI-generated answers and help technical teams prioritize improvements based on evaluation evidence.