MLflow is the industry-standard open-source platform for experiment tracking, and Azure Machine Learning has first-class native integration with it. When you use MLflow within an Azure ML job, parameters, metrics, and artifacts are automatically logged to the run history of the AML workspace, making every run reproducible and comparable. Option A (AML job output logs) only captures console output and lacks structured parameter and metric logging. Option C (Application Insights logs) is designed for application-level telemetry, not ML experiment metadata. Option D (Azure Monitor alerts) is a reactive notification tool, not a tracking system. MLflow ' s autologging capability means that for common frameworks such as scikit-learn, XGBoost, and PyTorch, parameters and metrics are captured without a single line of custom code, directly answering Fabrikam ' s requirement for consistent experiment tracking.

Microsoft Learn Reference Topic: Track ML experiments with MLflow – Azure Machine Learning MLflow integration

The first action should be to evaluate the model output . Fabrikam already has evaluation data containing manually defined expected responses, and the technical requirements explicitly state that the organization must use its existing evaluation datasets based on real data with input-output pairs . Microsoft Foundry supports evaluating a model against an existing curated dataset to establish measurable baseline performance before optimization. Evaluation results can identify weaknesses in accuracy, relevance, groundedness, safety, or other quality dimensions and provide objective evidence for subsequent changes.

This sequencing is critical because Fabrikam also requires that advanced fine-tuning be applied only when prompt engineering is insufficient . Therefore, immediately fine-tuning the model would bypass the required baseline evaluation and optimization process. Fine-tuning should be driven by demonstrated performance gaps, not applied automatically.

Deploying directly to production is also inappropriate because leadership requires low operational risk and gradual rollout. Production traffic should not be used as the first evaluation mechanism when curated evaluation data already exists. Synthetic interaction data is useful when evaluation coverage is insufficient, but Microsoft specifically positions synthetic generation as an option when an adequate test dataset is unavailable. Fabrikam already has suitable evaluation data.

Study Guide Reference: Optimize generative AI systems and model performance — evaluate model performance, benchmark against curated datasets, iterate on prompts, and apply advanced fine-tuning when justified.

Domain specialization: Supervised fine-tuning

Poor response accuracy of a RAG-based solution: Apply prompt engineering

For domain specialization , Supervised Fine-Tuning (SFT) is the correct first strategy. Microsoft describes SFT as the foundational fine-tuning technique for training a model from labeled input-output pairs , and specifically identifies domain specialization as one of its principal use cases. Microsoft also recommends starting with SFT for most customization projects because it supports task specialization, instruction following, style, and domain-specific behavior. Fabrikam already possesses evaluation data containing input-output pairs, which aligns directly with the SFT data model.

For poor RAG response accuracy , the first action is prompt engineering . The case explicitly requires advanced fine-tuning only when prompt engineering is insufficient. In a RAG system, prompt engineering determines how the model interprets retrieved context, constrains answers to grounding information, handles missing evidence, and formats responses. Microsoft notes that inadequate RAG prompting can produce false or incomplete answers even when retrieval returns appropriate content.

DPO is primarily appropriate for alignment using preferred versus non-preferred responses, while RFT targets complex reward-based reasoning optimization. Neither is the initial technique for domain specialization in this requirement.

Study Guide Reference: Optimize generative AI systems and model performance — prompt engineering, RAG optimization, supervised fine-tuning, preference optimization, and model customization strategy.

The correct solution is a GitHub Actions workflow that executes Microsoft Foundry evaluations as part of the CI process . Microsoft provides an AI agent evaluation GitHub Action specifically for incorporating Foundry agent evaluations into CI/CD workflows. The action can invoke the agent against an evaluation dataset, execute configured evaluators—including safety evaluators—and publish evaluation results before a change reaches production.

For pull-request gating, the workflow can be configured to run whenever relevant application files change. Evaluation commands can also enforce explicit thresholds and return a non-zero exit code when those thresholds are not satisfied. Microsoft documents evaluation gating through options such as --fail-on pass-rate= < threshold > or --fail-on any-failure; a non-zero result causes the CI job to fail, preventing unsafe changes from being promoted.

A blocklist or content filter provides runtime content protection but does not automate pull-request evaluation. A retrieval chunking strategy affects RAG retrieval quality, not CI safety gates.

Therefore, the required mechanism is GitHub Actions integrated with Foundry evaluation runs and safety thresholds .

Study Guide Reference: Implement generative AI quality assurance and observability — automated evaluations, CI/CD quality gates, safety evaluators, GitHub Actions, and pre-production validation.

===============

Correct sequence:

Collect retrieval logs.

Run RAG evaluators.

Adjust chunking strategy.

Re-index documents.

The process should begin by collecting retrieval logs . Retrieval diagnostics provide the evidence needed to understand which queries were issued, which document chunks were returned, and whether relevant content is being surfaced. Without retrieval telemetry, optimization becomes speculative rather than measurable.

Next, run RAG evaluators . Microsoft Foundry provides retrieval-oriented evaluators that assess how relevant retrieved context is to a query. The Retrieval evaluator measures contextual relevance without requiring ground truth, while the Document Retrieval evaluator can calculate metrics such as Fidelity and NDCG when labeled retrieval ground truth exists. These measurements help establish the retrieval-quality baseline before modifications are made.

After identifying retrieval deficiencies, adjust the chunking strategy . Microsoft specifically recommends reviewing chunk size and chunking methodology when retrieval returns irrelevant or incomplete passages. Chunks that are too small can lose context, while oversized chunks can introduce irrelevant material and reduce retrieval precision.

Finally, re-index the documents so the revised chunking configuration is reflected in the searchable corpus. Changing temperature or regenerating the prompt template primarily affects generation behavior rather than correcting the underlying retrieval pipeline.

Study Guide Reference: Implement generative AI quality assurance and observability — RAG evaluation, retrieval telemetry, retrieval-quality metrics, chunk optimization, indexing, and grounded-response assessment.

Azure Machine Learning ' s experiment tracking is built around two complementary concepts. First, experiments are named containers that group related runs. By calling mlflow.set_experiment with an experiment name at the start of your training code, all subsequent runs are grouped under that experiment name in the AML workspace, creating the centralized record required. Second, metrics are scalar values such as accuracy, loss, or AUC that represent model performance. Calling mlflow.log_metric with a metric name and value during training persists these values to the run ' s record in the experiment history. These values appear on the Azure ML Studio run detail page and can be compared across runs using the experiment comparison view. Without set_experiment, runs fall into a default experiment. Without log_metric, the run history has no performance data to display or compare.

Microsoft Learn Reference Topic: Track machine learning experiments with MLflow in Azure Machine Learning

Preference comparison data with chosen versus rejected response pairs is the input format for Direct Preference Optimization (DPO) or RLHF-style fine-tuning — an advanced fine-tuning technique available in Microsoft Foundry. A valid DPO dataset record must have three fields: a prompt as the input, a chosen field containing the preferred response, and a rejected field containing the less preferred response. The file must be in JSONL format with UTF-8 encoding, where each line represents one complete preference pair. When evaluating statements about this dataset, mark True if the dataset contains all three required fields and chosen responses represent higher-quality outputs than rejected ones. Mark False if the format is incompatible with DPO requirements, if the required rejected field is missing, or if the chosen and rejected responses appear to be of equivalent quality with no clear preference signal.

Microsoft Learn Reference Topic: Advanced fine-tuning with preference data in Microsoft Foundry – DPO dataset format

Median stopping and Truncation selection both evaluate a run relative to the broader population of training runs rather than comparing it only against the single best run.

The Median Stopping Policy calculates running averages of the primary metric across all runs . At an evaluation interval, a run can be terminated when its best primary-metric performance is worse than the median of the running averages across the population. Microsoft explicitly defines this policy as being based on running averages of the primary metric of all runs.

The Truncation Selection Policy periodically ranks runs by their primary metric and terminates a configured percentage of the lowest-performing runs . For example, a 20-percent truncation policy terminates runs falling within the lowest 20 percent at the applicable evaluation interval. It therefore compares each candidate with the population of comparable runs rather than only the current leader.

Bandit is specifically unsuitable because it uses an allowable slack relative to the best-performing run , which directly violates the requirement. The Default behavior applies no early termination policy and allows runs to execute to completion.

Study Guide Reference: Implement machine learning model lifecycle and operations — automated hyperparameter tuning, early termination policies, Median Stopping Policy, Truncation Selection Policy, and efficient experiment execution.