Databricks Certified-Data-Engineer-Professional 問題集 : Databricks Certified Data Engineer Professional

A Databricks job has been configured with 3 tasks, each of which is a Databricks notebook. Task A does not depend on other tasks. Tasks B and C run in parallel, with each having a serial dependency on Task A.
If task A fails during a scheduled run, which statement describes the results of this run?

• A. Unless all tasks complete successfully, no changes will be committed to the Lakehouse; because task A failed, all commits will be rolled back automatically.
• B. Tasks B and C will attempt to run as configured; any changes made in task A will be rolled back due to task failure.
• C. Tasks B and C will be skipped; task A will not commit any changes because of stage failure.
• D. Tasks B and C will be skipped; some logic expressed in task A may have been committed before task failure.
• E. Because all tasks are managed as a dependency graph, no changes will be committed to the Lakehouse until all tasks have successfully been completed.

An upstream system is emitting change data capture (CDC) logs that are being written to a cloud object storage directory. Each record in the log indicates the change type (insert, update, or delete) and the values for each field after the change. The source table has a primary key identified by the field pk_id.
For analytical purposes, only the most recent value for each record needs to be recorded in the target Delta Lake table in the Lakehouse. The Databricks job to ingest these records occurs once per hour, but each individual record may have changed multiple times over the course of an hour.
Which solution meets these requirements?

• A. Use Delta Lake's change data feed to automatically process CDC data from an external system, propagating all changes to all dependent tables in the Lakehouse.
• B. Use MERGE INTO to insert, update, or delete the most recent entry for each pk_id into a table, then propagate all changes throughout the system.
• C. Deduplicate records in each batch by pk_id and overwrite the target table.
• D. Iterate through an ordered set of changes to the table, applying each in turn to create the current state of the table, (insert, update, delete), timestamp of change, and the values.

A data engineer wants to refactor the following DLT code, which includes multiple table definitions with very similar code.

In an attempt to programmatically create these tables using a parameterized table definition, the data engineer writes the following code.

The pipeline runs an update with this refactored code, but generates a different DAG showing incorrect configuration values for these tables.
How can the data engineer fix this?

• A. Move the table definition into a separate function, and make calls to this function using different input parameters inside the for loop.
• B. Convert the list of configuration values to a dictionary of table settings, using table names as keys.
• C. Wrap the for loop inside another table definition, using generalized names and properties to replace with those from the inner table definition.
• D. Load the configuration values for these tables from a separate file, located at a path provided by a pipeline parameter.

Which statement describes the correct use of pyspark.sql.functions.broadcast?

• A. It caches a copy of the indicated table on all nodes in the cluster for use in all future queries during the cluster lifetime.
• B. It caches a copy of the indicated table on attached storage volumes for all active clusters within a Databricks workspace.
• C. It marks a column as having low enough cardinality to properly map distinct values to available partitions, allowing a broadcast join.
• D. It marks a DataFrame as small enough to store in memory on all executors, allowing a broadcast join.
• E. It marks a column as small enough to store in memory on all executors, allowing a broadcast join.

A Databricks SQL dashboard has been configured to monitor the total number of records present in a collection of Delta Lake tables using the following query pattern:
SELECT COUNT (*) FROM table
Which of the following describes how results are generated each time the dashboard is updated?

• A. The total count of records is calculated from the parquet file metadata
• B. The total count of records is calculated from the Hive metastore
• C. The total count of rows is calculated by scanning all data files
• D. The total count of rows will be returned from cached results unless REFRESH is run
• E. The total count of records is calculated from the Delta transaction logs