Explore the most cost-effective solutions for processing streaming market data on AWS. Learn how services like Amazon Kinesis Data Firehose, Amazon S3, and AWS Batch with Spot Instances can significantly reduce costs while ensuring efficient data processing.
Table of Contents
Question
A company ingests and processes streaming market data. The data rate is constant. A nightly process that calculates aggregate statistics takes 4 hours to complete. The statistical analysis is not critical to the business, and data points are processed during the next iteration if a particular run fails.
The current architecture uses a pool of Amazon EC2 Reserved Instances with 1-year reservations. These EC2 instances run full time to ingest and store the streaming data in attached Amazon Elastic Block Store (Amazon EBS) volumes. A scheduled script launches EC2 On-Demand Instances each night to perform the nightly processing. The instances access the stored data from NFS shares on the ingestion servers. The script terminates the instances when the processing is complete.
The Reserved Instance reservations are expiring. The company needs to determine whether to purchase new reservations or implement a new design.
Which solution will meet these requirements MOST cost-effectively?
A. Update the ingestion process to use Amazon Kinesis Data Firehose to save data to Amazon S3. Use a scheduled script to launch a fleet of EC2 On-Demand Instances each night to perform the batch processing of the S3 data. Configure the script to terminate the instances when the processing is complete.
B. Update the ingestion process to use Amazon Kinesis Data Firehose to save data to Amazon S3. Use AWS Batch with Spot Instances to perform nightly processing with a maximum Spot price that is 50% of the On-Demand price.
C. Update the ingestion process to use a fleet of EC2 Reserved Instances with 3-year reservations behind a Network LoadBalancer. Use AWS Batch with Spot Instances to perform nightly processing with a maximum Spot price that is 50% of the On-Demand price.
D. Update the ingestion process to use Amazon Kinesis Data Firehose to save data to Amazon Redshift. Use Amazon EventBridge to schedule an AWS Lambda function to run nightly to query Amazon Redshift to generate the daily statistics.
Answer
B. Update the ingestion process to use Amazon Kinesis Data Firehose to save data to Amazon S3. Use AWS Batch with Spot Instances to perform nightly processing with a maximum Spot price that is 50% of the On-Demand price.
Explanation
Option B proposes the use of Amazon Kinesis Data Firehose for data ingestion, which is a fully managed service for delivering real-time streaming data to destinations such as Amazon S3. This eliminates the need for managing EC2 instances for data ingestion, thereby reducing costs.
For the nightly processing, it suggests the use of AWS Batch with Spot Instances. AWS Batch enables you to run batch computing workloads on the AWS Cloud. Spot Instances allow you to use spare EC2 computing capacity at up to a 90% discount compared to On-Demand prices. So, if you can be flexible about when your applications run and if your applications can be interrupted, then Spot Instances can significantly reduce your Amazon EC2 costs. Moreover, setting a maximum Spot price of 50% of the On-Demand price will further ensure cost-effectiveness.
The other options are less cost-effective for the following reasons:
- Option A: It also uses Amazon Kinesis Data Firehose for data ingestion, but it suggests using EC2 On-Demand Instances for processing, which are more expensive than Spot Instances.
- Option C: It suggests using EC2 Reserved Instances with 3-year reservations for data ingestion, which would require a significant upfront cost and may not provide the flexibility to scale or change as the business needs evolve.
- Option D: It involves using Amazon Redshift for data storage, which is a data warehousing service and may be more expensive and less suitable for this use case than Amazon S3.
Amazon AWS Certified Solutions Architect – Professional SAP-C02 certification exam practice question and answer (Q&A) dump with detail explanation and reference available free, helpful to pass the Amazon AWS Certified Solutions Architect – Professional SAP-C02 exam and earn Amazon AWS Certified Solutions Architect – Professional SAP-C02 certification.