Learn how to effectively deduplicate large volumes of continuously growing semi-structured data in AWS with minimal operational overhead, using AWS Glue FindMatches.
Table of Contents
Question
An investment company needs to manage and extract insights from a volume of semi-structured data that grows continuously.
A data engineer needs to deduplicate the semi-structured data, remove records that are duplicates, and remove common misspellings of duplicates.
Which solution will meet these requirements with the LEAST operational overhead?
A. Use the FindMatches feature of AWS Glue to remove duplicate records.
B. Use non-Windows functions in Amazon Athena to remove duplicate records.
C. Use Amazon Neptune ML and an Apache Gremlin script to remove duplicate records.
D. Use the global tables feature of Amazon DynamoDB to prevent duplicate data.
Answer
A. Use the FindMatches feature of AWS Glue to remove duplicate records.
Explanation
AWS Glue FindMatches is the best solution for deduplicating large volumes of semi-structured data that are continuously growing, with the least operational overhead.
FindMatches uses machine learning to identify duplicate or matching records in the data, even if there are variations or misspellings. It can then remove the duplicates without needing the data engineer to manually define all the subtle variations.
The other options have drawbacks:
B. Amazon Athena is a query service and cannot directly deduplicate data with its functions. It would require the data engineer to write complex queries to identify duplicates, which is operationally intensive.
C. Amazon Neptune is a graph database. While it could theoretically use Gremlin and ML to find duplicates in a graph structure, this would be a complex custom implementation with high operational overhead, and not well-suited for semi-structured data.
D. DynamoDB global tables only prevent future duplicates through multi-region replication. They cannot deduplicate existing data.
Therefore, AWS Glue FindMatches is the most appropriate choice to efficiently deduplicate a large, growing semi-structured dataset with minimal operational burden on the data engineer. Its machine learning capabilities make it well-suited to handle subtle variations and misspellings that commonly lead to duplicate records.
Amazon AWS Certified Data Engineer – Associate DEA-C01 certification exam assessment practice question and answer (Q&A) dump including multiple choice questions (MCQ) and objective type questions, with detail explanation and reference available free, helpful to pass the Amazon AWS Certified Data Engineer – Associate DEA-C01 exam and earn Amazon AWS Certified Data Engineer – Associate DEA-C01 certification.