What is Azure Data Factory ADF?

ADF is a cloud-oriented data integration service that helps in the creation and management of data-driven workflows for automating data migration and conversion. ADF does not auto-store any data but only aids in the management of workflows via programmatic and UI modes.

Best Azure BI Training in Hyderabad

Azure Business Intelligence

With Azure services and Power BI, you can turn your data processing efforts into analytics and reports that provide real-time insights into your business. ... Power BI has a multitude of Azure connections available, and the business intelligence solutions you can create with those services are as unique as your business.

Azure Storage

Azure Storage is a Microsoft-managed cloud storage service, that provides highly available, durable, scalable and redundant storage, at a fraction of the cost, if you were to manage it manually.

• Storage Account
• Azure Data Lake Storage Gen1
• Azure Data Lake Storage Gen2

Storage Account

An Azure storage account contains all your Azure Storage data objects: blobs, files, queues, tables, and disks. The storage account provides a unique namespace for your Azure Storage data that is accessible from anywhere in the world over HTTP or HTTPS. Data in your Azure storage account is durable and highly available, secure, and massively scalable.

Azure Data Lake Storage Gen1

Previously known as Azure Data Lake Storeto create a hyper-scale, Hadoop-compatible repository for analytics on data of any size, type, and ingestion speed. Tutorials, API references, and other documentation show you how to set up, manage, and access a data lake repository for operational and exploratory analytics.

Azure Data Lake Storage Gen2

It is now generally available... Azure Data Lake Storage Gen1 is an enterprise-wide hyper-scale repository for big data analytic workloads. Azure Data Lake enables you to capture data of any size, type, and ingestion speed in one single place for operational and exploratory analytics.

Azure SQL Database

The best destination for fully managed SQL in the cloudSQL Azure is Microsoft's cloud database service. Based on SQL Server database technology and built on Microsoft's Windows Azure cloud computing platform, SQL Azure enables organizations to store relational data in the cloud and quickly scale the size of their databases up or down as business needs change.

Azure Synapse Analytics

Azure Synapse is a limitless analytics service that brings together enterprise data warehousing and Big Data analytics. It gives you the freedom to query data on your terms, using either serverless on-demand or provisioned resources-at scale. Azure Synapse brings these two worlds together with a unified experience to ingest, prepare, manage, and serve data for immediate BI and machine learning needs.

Azure Data Lake Analytics

Azure Data Lake Analytics is a distributed, cloud-based data processing architecture offered by Microsoft in the Azure cloud. It is based on YARN, the same as the open-source Hadoop platform. It pairs with Azure Data Lake Store, a cloud-based storage platform designed for Big Data analytics.

Unified SQL

U-SQL is a data processing language that unifies the benefits of SQL with the expressive power of your own code. U-SQL's scalable distributed query capability enables you to efficiently analyze data in Data Lake Store, Azure Storage Blobs and relational stores such as Azure SQL DB/DW.

PolyBase

PolyBase enables your SQL Server instance to process Transact-SQL queries that read data from external data sources. SQL Server 2016 and higher can access external data in Hadoop and Azure Blob Storage. Starting in SQL Server 2019, you can now use PolyBase to access external data in SQL Server, Oracle, Teradata, and MongoDB.

PolyBase pushes some computations to the Hadoop node to optimize the overall query. However, PolyBase external access is not limited to Hadoop. Other unstructured non-relational tables are also supported, such as delimited text files.

Data Factory

Hybrid data integration service that simplifies ETL at scale

Integrate data silos with Azure Data Factory, a service built for all data integration needs and skill levels. Easily construct ETL and ELT processes code-free within the intuitive visual environment or write your own code. Visually integrate data sources using more than 90+ natively built and maintenance-free connectors at no added cost. Focus on your data-the serverless integration service does the rest.

• No code or maintenance required to build hybrid ETL and ELT pipelines within the Data Factory visual environment
• Cost-efficient and fully managed serverless cloud data integration tool that scales on demand
• Azure security measures to connect to on-premises, cloud-based and software-as-a-service apps with peace of mind
• SSIS integration runtime to easily rehost on-premises SSIS packages in the cloud using familiar SSIS tools

Components of Azure Data Factory

Pipeline – A pipeline is a logical grouping of activities that performs a grouping of work. An example of an activity may be: you're copying on-premise data from one data source to the cloud (Azure Data Lake for instance), you then want to run it through an HDI Hadoop cluster for further processing and analysis and put it into a reporting area. The components will be contained inside the pipeline and would be chained together to create a sequence of events, depending upon your specific requirement

Linked Service – This is very similar to the concept of a connection string in SQL Server, where you're saying what is the source and destination of your data.

Trigger – A trigger is a unit of processing that determines when a pipeline needs to be run. These can be scheduled or set off (triggered) by a different event.

Control Flow – The control flow in a data factory is what's orchestrating how the pipeline is going to be sequenced. This includes activities you'll be performing with those pipelines, such as sequencing, branching and looping.

Azure Databricks

Fast, easy, and collaborative Apache SparkTM–based analytics service. The best destination for big data analytics and AI with Apache Spark

Unlock insights from all your data and build artificial intelligence (AI) solutions with Azure Databricks, set up your Apache Spark™ environment in minutes, auto scale and collaborate on shared projects in an interactive workspace. Azure Databricks supports Python, Scala, R, Java, and SQL, as well as data science frameworks and libraries including TensorFlow, PyTorch and scikit-learn.

• Fast, optimized Apache Spark environment
• Interactive workspace with built-in support for popular tools, languages, and frameworks
• Supercharged machine learning on big data with native Azure Machine Learning integration
• High-performance modern data warehousing in conjunction with Azure Synapse Analytics

Azure Stream Analytics

Discover Azure Stream Analytics, the easy-to-use, real-time analytics service that is designed for mission-critical workloads. Build an end-to-end serverless streaming pipeline with just a few clicks. Go from zero to production in minutes using SQL-easily extensible with custom code and built-in machine learning capabilities for more advanced scenarios. Run your most demanding workloads with the confidence of a financially backed SLA.

• Serverless real-time analytics, from the cloud to the edge
• Deliver powerful insights from your streaming data with ease, in real time

Azure Analysis Services

Use Azure Resource Manager to create and deploy an Azure Analysis Services instance within seconds and use backup restore to quickly move your existing models to Azure Analysis Services and take advantage of the scale, flexibility and management benefits of the cloud.