Fine-tuning Open Source LLMs in Mauritania

• Problem Statement: Healthcare organisations manage large volumes of patient education materials, hospital guidelines, frequently asked questions, clinical information documents, and administrative resources. General-purpose large language models may not consistently follow healthcare-specific terminology, response formats, or approved communication guidelines. Healthcare teams need AI systems that can provide clear, accessible, and consistent general health information while protecting sensitive data and avoiding unsupported medical claims. The organisation wanted to explore fine-tuning open source LLMs to support healthcare education and information services.
• How We Overcame the Problem: Learners designed an AI-powered healthcare information assistant using an open source pretrained LLM and fine-tuning techniques. They explored supervised fine-tuning (SFT), Parameter-Efficient Fine-Tuning (PEFT), Low-Rank Adaptation (LoRA), and Quantised LoRA (QLoRA), selecting suitable methods according to model requirements, dataset quality, computing resources, and project objectives. They prepared healthcare education examples and evaluated model outputs against predefined quality and safety criteria.
• Overall Implementation: Learners implemented an end-to-end LLM fine-tuning workflow covering use-case definition, dataset collection, data cleaning, instruction formatting, pretrained model selection, baseline evaluation, fine-tuning configuration, model training, output evaluation, and application integration. The prototype processed general healthcare education questions and generated structured responses about preventive care, hospital procedures, appointment preparation, and approved patient education materials.
• Evidence of Learning: Learners demonstrated the ability to explore open source model repositories, load pretrained LLMs, prepare instruction-response datasets, configure supervised fine-tuning, apply LoRA or QLoRA where appropriate, monitor training metrics, evaluate generated responses, and integrate a fine-tuned model into a Python application.
• Healthcare AI Workflow:
  Approved Healthcare Education Materials → Data Cleaning and Validation → Instruction Dataset Preparation → Tokenization → Pretrained Open Source LLM → Fine-Tuning (SFT/LoRA/QLoRA) → Model Evaluation → Safety Testing → Healthcare Education Application
• Healthcare Dataset Preparation Implementation: Learners prepared structured instruction-response examples using approved patient education materials, general health FAQs, and hospital procedure information. They removed duplicates, standardised formatting, checked data quality, and used synthetic or appropriately de-identified data. Training, validation, and test datasets were separated to support meaningful evaluation and reduce data leakage.
• Pretrained Model Selection Implementation: Learners explored suitable open source LLMs and reviewed model documentation, licensing terms, parameter sizes, supported languages, hardware requirements, and intended use cases. They selected a model appropriate for healthcare education tasks and established a baseline by testing its responses before fine-tuning.
• Supervised Fine-Tuning Implementation: Learners formatted healthcare education examples as instruction-response pairs and configured a supervised fine-tuning workflow using tools such as Hugging Face Transformers and PyTorch. They adjusted training parameters, including learning rate, batch size, epochs, and sequence length, and monitored training and validation loss to identify potential overfitting.
• Parameter-Efficient Fine-Tuning Evidence: Learners explored LoRA and QLoRA to adapt selected pretrained models while reducing the number of trainable parameters and, where supported, memory requirements. They compared the practical setup, resource consumption, and output quality of the chosen adaptation approach against the baseline or another suitable configuration.
• Healthcare Response Generation Implementation: Learners tested the fine-tuned model using representative questions about preventive healthcare, preparation for appointments, hospital admission procedures, and general patient education. They assessed whether the responses were understandable, relevant to the supplied material, consistently formatted, and appropriately limited to general information.
• Model Evaluation and Comparison Evidence: Learners compared the base model and fine-tuned model using held-out test examples and predefined evaluation criteria. They assessed relevance, readability, consistency with approved educational content, factual support, and appropriate handling of uncertainty. Quantitative metrics were recorded where suitable, while healthcare subject-matter review was used for qualitative assessment. Any claimed improvement was supported by actual test results rather than assumed in advance.
• Problem-Solving Evidence: Learners addressed challenges such as inconsistent training examples, lengthy input documents, GPU memory limitations, overfitting, repetitive responses, unsupported medical statements, and variation in output quality. They applied data cleaning, appropriate input-length management, parameter-efficient fine-tuning, training-parameter adjustments, checkpoint comparison, and structured evaluation to improve the prototype.
• Inference and Application Integration Evidence: Learners loaded the fine-tuned model or its adapter weights into a Python-based application and configured text-generation settings. They demonstrated how the model could support a basic healthcare education interface or API, while documenting inference latency, hardware requirements, model-loading considerations, and deployment limitations.
• Responsible AI Evidence: Learners considered healthcare data privacy, consent, dataset licensing, model bias, output reliability, and risks associated with inaccurate medical information. They used synthetic or appropriately de-identified data, restricted access to project artefacts, documented model limitations, and added safeguards to prevent the prototype from presenting itself as a diagnostic or treatment authority. Personal medical concerns and urgent symptoms were directed to qualified healthcare professionals or appropriate emergency services.
• Comparative Evidence: Learners compared the base and fine-tuned models, and where appropriate, different fine-tuning configurations, based on response quality, consistency, resource consumption, training time, inference speed, and suitability for the defined healthcare education task. Findings were recorded to justify model and technique selection.
• Overall Outcome: By completing the implementation, learners demonstrated practical skills in adapting open source LLMs for a specialised healthcare education scenario. The resulting prototype could support general health information responses, patient education content, and explanations of hospital procedures. The project also helped learners understand the importance of dataset quality, model evaluation, resource optimisation, and safety controls when developing domain-specific Generative AI applications.
• Final Evidence: A working healthcare education prototype, Python source code, model and tokenizer details, dataset preparation documentation, fine-tuning configuration, training logs, model checkpoints or adapter weights, sample generated responses, baseline-versus-fine-tuned evaluation reports, safety test cases, model comparison findings, and implementation documentation provide evidence of learners' practical competency in Fine-Tuning Open Source LLMs.

Important: This scenario demonstrates educational and technical model-development skills. The prototype is not a clinically validated medical system and should not be used for autonomous diagnosis, treatment recommendations, or other clinical decisions without the necessary clinical validation, professional oversight, and regulatory review.