Migrating and Moving Data – Data Sources and Ingestion

The persistent batching, incremental loading, streaming, inserting, or updating of data increases the amount of data over time. If you want to move your in‐house data consisting of many gigabytes, terabytes, or petabytes of data to the cloud, then you need to use special migration products. Using FTP, copying a database backup over the network, or writing some custom code to move data to Azure are not optimal options. You need to use products like Azure Migrate, Azure Data Box, and Azure Database Migration Service to migrate and move large data sources from on‐premises to the Azure platform. All three products were introduced in Chapter 1. As shown in Figure 3.87, Azure Migrate provides not only database migration features but also VMs, web apps, and virtual desktop infrastructure (VDI) migration. Azure Data Box is also available from the Azure Monitor blade and asks for information like whether the transfer is inbound to Azure or outbound from Azure. Also, it is necessary to identify the target or source subscription, resource group, the location of the data source, and the target data location.

FIGUER 3.87 Azure Monitor and Azure Data Box

Azure Database Migration Service supports many different databases, including MySQL, PostgreSQL, MongoDB, and Azure Cosmos DB. If you are primarily focused on SQL Server, you can install an Azure SQL Migration extension into Azure Data Studio that will walk you through the steps necessary to migrate an existing database to Azure. Remember that copying large amounts of data over the Internet is risky, as it can fail, and can be costly. If you choose to move large data sources from your on‐premises datastores onto Azure or vice versa, consider setting up an ExpressRoute connection. This is a private connection and a direct link between your network and the Azure network. Not only will this improve the performance and stability, but it also will be more secure.