MLOps & LLMOps in French Southern Territories

Project Overview

The MLOps & LLMOps training equips learners with practical skills to deploy, monitor, evaluate, and maintain machine learning models and large language model (LLM) applications in production environments. In this banking scenario, learners apply these skills to develop a reliable AI-powered loan risk assessment and customer support solution, focusing on automation, model performance, operational monitoring, security, and responsible AI practices.

Problem Statement

A banking organisation faced challenges in managing loan risk prediction models and AI-powered customer support applications. Manual model deployment processes caused delays, while inconsistent experiment tracking made it difficult to reproduce results and compare model versions. Changes in customer behaviour and financial data also affected prediction accuracy over time. Meanwhile, the bank needed to ensure that its LLM-based customer support system generated accurate, relevant, and secure responses without exposing confidential customer information.

The organisation required a standardised operational framework to automate ML workflows, monitor model performance, evaluate LLM responses, and support secure, auditable AI deployments.

How Did We Overcome the Challenges?

Learners designed an MLOps and LLMOps workflow to address these operational issues through the following measures:

• Automated ML Pipelines: Established repeatable workflows for data validation, model training, testing, and deployment.
• Experiment and Model Tracking: Used tools such as MLflow to record model versions, parameters, evaluation metrics, and experiment artefacts.
• Containerisation and CI/CD: Applied Docker and automated testing pipelines to improve deployment consistency and reduce manual errors.
• Model Monitoring: Defined monitoring practices for prediction performance, data drift, service latency, and operational failures.
• LLM Evaluation: Created test datasets and evaluation criteria to assess response relevance, factual accuracy, consistency, and adherence to banking policies.
• Security and Governance: Incorporated access controls, sensitive-data protection, audit logging, and human review for high-impact lending decisions.

Overall Implementation

The learners implemented a demonstration banking workflow in which validated historical financial data was used to train and evaluate a loan risk prediction model. The selected model was versioned, packaged, tested, and deployed through a repeatable pipeline. Monitoring procedures were designed to identify performance degradation and changes in incoming data, triggering review and retraining when required.

For customer support, learners developed an LLM-based assistant that retrieves approved banking information and responds to common service enquiries. The application was evaluated using predefined test cases, with safeguards for sensitive information, unsupported responses, and escalation to human agents when necessary. Both workflows included logging, version control, evaluation procedures, and documented deployment steps.

Learner Outcomes and Evidence

By completing this project, learners demonstrated their ability to:

• Design and explain an end-to-end MLOps pipeline for a banking use case.
• Track experiments, manage model versions, and support reproducible deployments.
• Apply containerisation and CI/CD concepts to AI application delivery.
• Identify model drift and define monitoring and maintenance procedures.
• Evaluate LLM outputs using test datasets and predefined quality criteria.
• Apply security, governance, and human-review controls to banking AI workflows.

Evidence of Learning: Project architecture diagram, versioned model and experiment records, pipeline configuration, deployment documentation, monitoring dashboard or sample metrics, LLM evaluation results, test reports, and security checklist.

Conclusion: This banking scenario demonstrates how MLOps and LLMOps practices can help learners build more consistent, observable, and maintainable AI workflows. The project provides practical evidence of skills relevant to production AI engineering while reinforcing the importance of security, governance, and responsible decision-making in financial services.