Learn the steps to perform A/B testing with AWS SageMaker to evaluate a new machine learning model’s performance in a production environment compared to an existing model. Discover how to create a new endpoint configuration with production variants and update the existing endpoint to use the new configuration.
Table of Contents
Question
A company’s data scientist has trained a new machine learning model that performs better on test data than the company’s existing model performs in the production environment. The data scientist wants to replace the existing model that runs on an Amazon SageMaker endpoint in the production environment. However, the company is concerned that the new model might not work well on the production environment data.
The data scientist needs to perform A/B testing in the production environment to evaluate whether the new model performs well on production environment data.
Which combination of steps must the data scientist take to perform the A/B testing? (Choose two.)
A. Create a new endpoint configuration that includes a production variant for each of the two models.
B. Create a new endpoint configuration that includes two target variants that point to different endpoints.
C. Deploy the new model to the existing endpoint.
D. Update the existing endpoint to activate the new model.
E. Update the existing endpoint to use the new endpoint configuration.
Answer
A. Create a new endpoint configuration that includes a production variant for each of the two models.
E. Update the existing endpoint to use the new endpoint configuration.
Explanation
To perform A/B testing in the production environment and evaluate the new model’s performance on production data, the data scientist must take the following steps:
A. Create a new endpoint configuration that includes a production variant for each of the two models. This allows the endpoint to route traffic to both the existing and new models based on specified weights.
E. Update the existing endpoint to use the new endpoint configuration. This activates the A/B testing setup, enabling the endpoint to direct traffic to both models according to the configuration.
By creating a new endpoint configuration with production variants for both models (A) and updating the existing endpoint to use this new configuration (E), the data scientist can effectively conduct A/B testing. This setup allows the evaluation of the new model’s performance on production data compared to the existing model, ensuring a thorough assessment before fully replacing the model in the production environment.
Amazon AWS Certified Machine Learning – Specialty (MLS-C01) certification exam practice question and answer (Q&A) dump with detail explanation and reference available free, helpful to pass the Amazon AWS Certified Machine Learning – Specialty (MLS-C01) exam and earn Amazon AWS Certified Machine Learning – Specialty (MLS-C01) certification.