Azure Data Factory

Mapping Data Flows are visually-designed data transformation in Azure Data Factory. Data Flows allow data engineers to develop graphical data transformation logic without writing code. The resulting data flows are executed as activities within Azure Data Factory Pipelines using scaled-out Azure Databricks clusters. The intent of Azure Data Factory Data Flow is to provide a fully visual experience with no coding required. Your Data Flows will execute on your own execution cluster for scaled-out data processing. Azure Data Factory handles all of the code translation, path optimization, and execution of your data flow jobs.

Use this transformation for column selectivity (reducing number of columns), alias columns and stream names, and reorder columns.

Several Azure Data Factory Data Flow transformations support the idea of "Columns Patterns" so that you can create template columns based on patterns instead of hard-coded column names. You can use this feature within the Expression Builder to define patterns to match columns for transformation instead of requiring exact, specific field names. Patterns are useful if incoming source fields change often, particularly in the case of changing columns in text files or NoSQL databases. This condition is sometimes referred to as "Schema Drift". Column patterns Column patterns are useful for handling both Schema Drift scenarios as well as general scenarios. It is good for conditions where you are not able to fully know each column name. You can pattern match on column name and column data type and build an expression for transformation that will perform that operation against any field in the data stream that matches your  name  &  type patterns.

Use the Derived Column transformation to generate new columns in your data flow or to modify existing fields.

{     "name": "AzureStorageLinkedService",     "properties": {         "type": "AzureStorage",         "typeProperties": {             "connectionString": {                 "value": "DefaultEndpointsProtocol=https;AccountName=<accountName>;AccountKey=<accountKey>;EndpointSuffix=core.windows.net",                 "type": "SecureString"             }         }     } } ---------------------- Difference in Linked Service JSON New vs Old {   "name": "AzureStorageLinkedService",   "properties": {   "type": "AzureStorage",   "typeProperties": {   "connectionString": "DefaultEndpointsProtocol=https;AccountName=<accountname>;AccountKey=<accountkey>"   }   }  } New p...

Accessing Pipeline Variable Array elements in another activity by. @ variable ('NameOfPipelineVariable')[0]   -- 1st element @ variable('NameOfPipelineVariable')[1]   -- 2nd element @concat( variable ('NameOfPipelineVariable')[0], variable ('NameOfPipelineVariable')[1]) @{activity('Copy1').output.dataWritten} Accesses a property of the previous copy activity and passes the value of dataWritten. or  @activity('LookupTableList').output.value --- this will return full dataset i.e. all records returned by SQL in an activity. ------- @{activity('CopyBlobtoBlob').error.message -  For the failure case, pass the error output instead of  ----------------------------------------------------------------------- First Activity outputs a column named NewWatermarkvalue. select MAX(LastModifytime) as NewWatermarkvalue from data_source_table Below is how you access the output of the above activity in another activity in...