Google Data Engineer Sample Questions:

1. You architect a system to analyze seismic dat
a. Your extract, transform, and load (ETL) process runs as a series of MapReduce jobs on an Apache Hadoop cluster. The ETL process takes days to process a data set because some steps are computationally expensive. Then you discover that a sensor calibration step has been omitted. How should you change your ETL process to carry out sensor calibration systematically in the future?

A) Introduce a new MapReduce job to apply sensor calibration to raw data, and ensure all other MapReduce jobs are chained after this.
B) Add sensor calibration data to the output of the ETL process, and document that all users need to apply sensor calibration themselves.
C) Modify the transformMapReduce jobs to apply sensor calibration before they do anything else.
D) Develop an algorithm through simulation to predict variance of data output from the last MapReduce job based on calibration factors, and apply the correction to all data.

2. You want to automate execution of a multi-step data pipeline running on Google Cloud. The pipeline includes Cloud Dataproc and Cloud Dataflow jobs that have multiple dependencies on each other. You want to use managed services where possible, and the pipeline will run every day. Which tool should you use?

A) cron
B) Cloud Scheduler
C) Workflow Templates on Cloud Dataproc
D) Cloud Composer

3. You operate an IoT pipeline built around Apache Kafka that normally receives around 5000 messages per second. You want to use Google Cloud Platform to create an alert as soon as the moving average over 1 hour drops below 4000 messages per second. What should you do?

A) Use Kafka Connect to link your Kafka message queue to Cloud Pub/Su
B) Use Kafka Connect to link your Kafka message queue to Cloud Pub/Su
C) If that number falls below 4000, send an alert.
D) If that number falls below 4000, send an alert.
E) Consume the stream of data in Cloud Dataflow using Kafka I
F) Compute the average when the window closes, and send an alert if the average is less than 4000 messages.
G) Use Cloud Scheduler to run a script every five minutes that counts the number of rows created in BigQuery in the last hour
H) Use a Cloud Dataflow template to write your messages from Cloud Pub/Sub to BigQuer
I) Consume the stream of data in Cloud Dataflow using Kafka I
J) Use Cloud Scheduler to run a script every hour that counts the number of rows created in Cloud Bigtable in the last hour
K) Set a sliding time window of 1 hour every 5 minute
L) Use a Cloud Dataflow template to write your messages from Cloud Pub/Sub to Cloud Bigtabl
M) Set a fixed time window of 1 hour.Compute the average when the window closes, and send an alert if the average is less than 4000 messages.

4. You are designing the database schema for a machine learning-based food ordering service that will predict what users want to eat. Here is some of the information you need to store:
The user profile: What the user likes and doesn't like to eat
The user account information: Name, address, preferred meal times
The order information: When orders are made, from where, to whom
The database will be used to store all the transactional data of the product. You want to optimize the data schem a. Which Google Cloud Platform product should you use?

A) Cloud Datastore
B) Cloud SQL
C) Cloud Bigtable
D) BigQuery

5. You are creating a new pipeline in Google Cloud to stream IoT data from Cloud Pub/Sub through Cloud Dataflow to BigQuery. While previewing the data, you notice that roughly 2% of the data appears to be corrupt. You need to modify the Cloud Dataflow pipeline to filter out this corrupt data. What should you do?

A) Add a SideInput that returns a Boolean if the element is corrupt.
B) Add a GroupByKey transform in Cloud Dataflow to group all of the valid data together and discard the rest.
C) Add a Partition transform in Cloud Dataflow to separate valid data from corrupt data.
D) Add a ParDo transform in Cloud Dataflow to discard corrupt elements.

Solutions: