The correct answer is B — No.

Lakeflow Connect is an ingestion service that physically copies data from external databases into Delta tables managed by Databricks. It ' s designed for scenarios where you want a replicated, writable Delta copy of external data — essentially a CDC-based ingestion pipeline.

That ' s the opposite of what ' s required here. The requirement states ' the data is NOT copied into Databricks-managed storage. ' Lakeflow Connect would create Delta tables in Databricks and copy DB1 ' s data into them — a direct violation.

Additionally, Lakeflow Connect creates Databricks-native Delta tables rather than exposing DB1 ' s original schemas and tables as virtual objects. Analysts querying through a Lakeflow Connect pipeline are querying a replicated copy, not the live source.

For zero-copy, live query federation of an external SQL Server into Unity Catalog, Lakehouse Federation (foreign catalog) is the correct tool.

For authentication, a Managed Identity (via a Databricks Access Connector) is the right choice. The Access Connector wraps an Azure-managed identity so Databricks can authenticate to Azure Storage without any credentials being stored in the workspace. The cloud security team controls the identity through Azure RBAC — there are no secrets to rotate or leak inside Databricks.

For the permission, READ FILES on the volume is exactly right. It allows users to read and list files through the volume path while blocking writes, deletes, and modifications. This is the minimum necessary access, honouring the principle of least privilege.

WRITE FILES would allow modifications, contradicting ' users can access but NOT modify. ' ALL PRIVILEGES grants far more than needed. Service principals with stored client secrets would mean credentials inside Databricks, violating the ' does not require storing credentials ' requirement.

Job1 should contain one ingestion task that acts as the common upstream dependency for two separate transformation tasks. Once ingestion succeeds, Lakeflow Jobs can start both downstream tasks concurrently because neither transformation depends on the other. This design represents the actual workflow, avoids duplicated ingestion, and reduces total execution time through parallelism. Creating two ingestion tasks would repeat the same source processing and could introduce inconsistent results or unnecessary costs. A single sequential task would prevent parallel transformation and make failures harder to isolate and retry. Defining three independent tasks without dependencies could allow transformations to start before ingestion has completed. An explicit directed task graph therefore provides the required execution order while preserving parallelism for independent downstream processing.

The correct answer is C — @dlt.expect_or_fail.

Lakeflow Spark Declarative Pipelines (SDP) offers three expectation decorators, each with a different violation response:

@dlt.expect — logs the violation as a metric but writes all records, including bad ones, to the table. Suitable for monitoring only.

@dlt.expect_or_drop — drops violating records and continues the pipeline. The table receives only clean rows, but the pipeline update commits successfully.

@dlt.expect_or_fail — fails the entire pipeline update when a violation is detected. The table update is never committed. This is the correct choice when data integrity is non-negotiable: ' Prevent an update to Table1 from being committed when data that violates rule1 is detected. '

@dlt.expect_all_or_drop takes a dictionary of rules and drops violating rows but still commits — it doesn ' t halt the pipeline.

Task-level retries allow the ingestion task to recover automatically from transient failures without rerunning unrelated tasks or restarting the complete workflow. Downstream tasks remain governed by their dependencies and start only after ingestion succeeds. This provides focused failure recovery and reduces unnecessary compute consumption. Disabling retries and relying on manual execution directly contradicts the requirement for resilient, automated pipeline operation. Setting the retry count to zero also prevents automatic retry behavior. Restarting the workflow from the first task whenever any task fails would repeat completed processing, increase costs, and potentially reingest data unnecessarily. Lakeflow Jobs supports individual retry policies for tasks, including the number of retries and delay between attempts, making option D the most controlled and operationally efficient configuration. Microsoft Learn

An Azure Storage account provides durable, comparatively low-cost retention for diagnostic and audit logs that are accessed infrequently. A lifecycle or retention policy can preserve the logs for 90 days and then remove them automatically. This matches an investigation-only access pattern without paying the ingestion and indexing charges associated with Log Analytics. Azure Monitor metrics stores numerical monitoring measurements, not the complete audit-event records required here. Azure Event Hubs is a streaming transport intended to forward events to consumers and is not the final long-term retention destination. Log Analytics is appropriate when teams need frequent querying, dashboards, and alerting, but those capabilities introduce unnecessary cost for logs reviewed only during occasional investigations. Storage therefore best satisfies centralized retention and cost requirements.

The correct answer is C — a Lakeflow Spark Declarative Pipelines (SDP) pipeline.

SDP is tailor-made for exactly this pattern: ingest from cloud storage, transform through cleansing and enrichment stages, and publish Delta tables to Unity Catalog. What sets it apart from the other options is built-in monitoring (the pipeline graph shows row counts, expectation metrics, and run history) and automatic retries (failed tasks retry automatically based on pipeline settings, without manual re-run triggers).

Option A (Structured Streaming job) gives you the streaming engine but nothing else — monitoring, alerting, and retry logic all have to be built from scratch. Option B (scheduled notebook job) is a batch approach that requires manual monitoring and lacks the declarative lineage tracking SDP provides. Option D (Azure Data Factory with data flows) works but adds a separate Azure service to manage, introduces ADF licensing costs, and doesn ' t integrate natively with Unity Catalog governance.

The correct answers are C and D. Together they deliver cost-efficient autoscaling:

D (Enable autoscaling) allows the cluster to grow when workloads increase and shrink when they ease off. This satisfies both scale-up and scale-down requirements without manual intervention.

C (Auto-termination after 30 minutes of inactivity) ensures the cluster stops entirely when no work is running, eliminating the cost of an idle cluster. This is the cheapest possible state.

Option A (disable Photon) reduces compute acceleration — that ' s a performance regression with no meaningful cost benefit for autoscaling. Option B (compute policy that lets users manage settings) adds governance overhead and doesn ' t address scaling behaviour. Option E (fixed number of workers) is the opposite of autoscaling — a static worker count that either over-provisions during quiet periods or under-provisions during peaks.

The correct answer is B — No.

This is a common Python-to-PySpark trap. In pure Python, comparing a value to None with != works as expected. In PySpark, null comparisons follow SQL null semantics: any comparison involving NULL returns NULL (not True or False). So df.filter(df.order_amount != None) doesn ' t evaluate to True for non-null rows — the comparison itself returns NULL for null values, and Spark interprets NULL in a filter as False, effectively dropping null rows. But the behaviour is undefined in edge cases and is not the documented approach.

More practically, Python ' s None and Spark ' s SQL NULL are different concepts. PySpark Column objects don ' t support Python ' s native equality/inequality semantics for null checking. The result is typically an empty DataFrame or incorrect filtering behaviour.

Always use .isNotNull() or .isNull() for null checks in PySpark column expressions. These methods are specifically designed for SQL-null-aware comparisons and produce correct, predictable results.

The Databricks CLI contains the bundle command group for validating, deploying, and running Declarative Automation Bundles. CLI commands can be executed consistently from developer terminals or CI/CD pipelines, making deployments automated, repeatable, and suitable for multiple target environments. The Databricks SDK for Python can manage workspace APIs programmatically, but it is not the standard bundle deployment interface required here. Git folders provide source-control integration inside the workspace but do not deploy bundle-defined resources. The Jobs UI supports interactive creation and management of jobs, which introduces manual steps and does not provide the same infrastructure-as-code workflow. Using commands such as databricks bundle validate and databricks bundle deploy -t dev directly satisfies the automated development deployment requirement. Microsoft Learn