If a data transfer is expected to take 48 hours, which Spark connector mode is recommended?

Master Snowflake Data Engineer Exam. Study with flashcards and multiple choice questions, each question includes hints and explanations. Prepare for your success!

Multiple Choice

If a data transfer is expected to take 48 hours, which Spark connector mode is recommended?

Explanation:
For long-running data moves, external transfer is the best fit. This mode uses a cloud storage staging area to bulk-load data into Snowflake, which lets the transfer run in parallel and decouples the Spark process from Snowflake ingestion. It reduces memory and resource pressure on the Spark cluster and avoids prolonged streaming of data through Spark, making multi-day transfers more reliable and scalable. Internal transfer, by contrast, streams data directly between Spark and Snowflake, which can become a bottleneck and is more sensitive to driver/executor memory and network constraints. So for a 48-hour transfer, External Transfer is the recommended option.

For long-running data moves, external transfer is the best fit. This mode uses a cloud storage staging area to bulk-load data into Snowflake, which lets the transfer run in parallel and decouples the Spark process from Snowflake ingestion. It reduces memory and resource pressure on the Spark cluster and avoids prolonged streaming of data through Spark, making multi-day transfers more reliable and scalable. Internal transfer, by contrast, streams data directly between Spark and Snowflake, which can become a bottleneck and is more sensitive to driver/executor memory and network constraints. So for a 48-hour transfer, External Transfer is the recommended option.

Subscribe

Get the latest from Passetra

You can unsubscribe at any time. Read our privacy policy