Data Management Framework

Home Data Management Framework

Data Management Framework

Index
  1. Data entities
  2. Data management framework
  3. SQL Server Integration Services (SSIS)
  4. Azure Data Factory
  5. Third party integration products
  6. Curogens ADF Estimator (Power App)

Data Management Framework for Dynamics 365 Finance and Operations is an in-built tool for the management of Data import and export, furthermore it comes with special functionality to copy configuration data between the legal entities (setup, configurations, references, master, and document data) or for the management of permanent integration scenarios (real-time, near-real-time, asynchronous integration patterns).

Data entities

DMF works with data entities (i.e., conceptual abstraction and encapsulation of underlying table) to quickly migrate reference, master, and document data from external systems to D365FO.

Each Data Entity uses an intermediate staging table for the data to be imported and has a target mapping showing the mapping of that staging table to target. The source file with the data to be imported must be provided with header fields that match the target fields.

It the source file delivers correct data also in terms of type of fields and correct data for each field (clean of duplicates) the data will be imported on staging table and all required validations (business logic) for the target fields will be performed. If validations are OK, the data is imported to target with the possibility to overwrite (= update) and insert new records into the target table.

DMF is very user-friendly and customizable to the specific needs and preferences of the business. 

Data management framework

By structuring the import and export processes in single “Data projects” with well defined hierarchies and sequences for each single data entity, it allows the business to have always full control and overview of the data migration and integration tasks.

Whilst the handling of the “Load” or “Extraction” is very handy and easy to manage also for power users, the “Transformation” capabilities of Data Management Framework are very reduced and require the application of program code by developers.

SQL Server Integration Services (SSIS)

SQL Server Integration Services is a Microsoft SQL Server in-built Data Warehousing tool to perform high-performance data integrations. It can be used for the extraction, loading and transformation (ETL) of structured data from multiple sources, such as SQL Server, Oracle Database, Excel files, flat files etc.

The primary use of SSIS is the migration of data from different sources to other destinations, and it is used mainly in on-premise environments.

SSIS is an ETL tool (extract-transform-load). It is designed to extract data from one or more sources, transform the data in memory – in the data flow – and then write the results to a destination.

Azure Data Factory

Azure Data Factory is a fully managed, serverless data integration service that allows businesses to automate data movement and transformation by means of  ETL/ELT workflows.

Azure Data Factory comes with an intuitive user interface, low code transformation possibilities, and more than 90 in-built connectors on the cloud and on-premise.

Whilst SSIS allows only for data transformation and integration mainly on on-premise systems, Azure Data Factory offers also features for Data cleansing and can detect and parse schema from many common file fomats, such as CSV, JSON etc.

ADF is typically used as an orchestrator of activities that go beyond the mere ETL/ELT process, like calling Azure functions, Azure Logic Apps, stored procedures etc.

Azure data factory manages

Like SSIS, Azure Data Factory manages ETL scenarios using data flows that visualize the transformations in a user-friendly format. Data flows allow Data Engineers to develop data transformation logic without writing code. For more complex transformations there is an Expression builder composed of column values, functions, operators and literals available.

The use of IntelliSense code completion for highlighting and syntax checking makes the building of data flows very easy and comfortable.

To complete the tool, there are features for debugging and previewing the data throughout the whole pipeline, which are a great help in troubleshooting the integration logics.

Third party integration products

Besides SQL Server Integration Services and Azure Data Factory there is a great market of Third Party Integration Tools. They all have in common that they use tools for selecting, preparing, extracting and transforming data to ensure that its form is compatible with the data structure in the destination system. They differ in terms of cost and in terms of available connectors, or transformation tools, and must hence be carefully analyzed in the selection process.

Curogens ADF Estimator (Power App)

It is difficult to estimate the end-to-end cost of a data migration since there are many variables and cost driving factors in play. Calculating this cost can be done by figuring out how much data you need to move (how many tables, how many fields, how many transformation are required?) and which tools you are going to use, and if or not those tools can access the Source data directly or there is need for an additional Data Extraction step.

Curogens has developed a handy tool for giving you a ROUGH IDEA about the cost to be expected for your Data Migration using Azure Data Factory as Integration tool (being the tool with the broadest possibilities for both on-premise and cloud destination systems).

Of course this tool cannot replace a concrete analysis of your particular situation with our experts, which takes into account many more aspects and special requirements. Request a free consultation with CuroGens