Power Query Dataverse connector - Power Query (2024)

  • Article

Summary

ItemDescription
Release StateGeneral Availability
ProductsExcel
Power BI (Semantic models)
Power BI (Dataflows)
Fabric (Dataflow Gen2)
Power Apps (Dataflows)
Dynamics 365 Customer Insights
Authentication typesOrganizational account
Service principal

Note

The service principal authentication type isn't supported when using an on-premises data gateway or a virtual network (VNet) data gateway.

Note

Some capabilities may be present in one product but not others due to deployment schedules and host-specific capabilities.

Prerequisites

You must have a Dataverse environment.

You must have read permissions to access data within tables.

To use the Dataverse connector, the TDS endpoint setting must be enabled in your environment. More information: Manage feature settings

To use the Dataverse connector, one of TCP ports 1433 or 5558 need to be open to connect. Port 1433 is used automatically. However, if port 1433 is blocked, you can use port 5558 instead. To enable port 5558, you must append that port number to the Dataverse environment URL, such as yourenvironmentid.crm.dynamics.com, 5558. More information: SQL Server connection issue due to closed ports

Note

If you're using Power BI Desktop and need to use port 5558, you must create a source with the Dataverse environment URL, such as yourenvironmentid.crm.dynamics.com,5558, in Power Query M. More information: Finding your Dataverse environment URL

Capabilities supported

  • Server URL
  • Import
  • DirectQuery (Power BI semantic models)
  • Advanced
    • Include relationship columns

Connect to Dataverse from Power Query Desktop

Note

The Power Query Dataverse connector is mostly suited towards analytics workloads, not bulk data extraction. More information: Alternative Dataverse connections

To connect to Dataverse from Power Query Desktop:

  1. Select the Dataverse option from Get Data. More information: Where to get data

  2. If you're connecting to this site for the first time, select Sign in and input your credentials. Then select Connect.

    Power Query Dataverse connector - Power Query (1)

  3. In Navigator, select the data you require, then either load or transform the data.

    Power Query Dataverse connector - Power Query (2)

  4. If you're using Power Query from Power BI Desktop, you're asked to select either the Import or DirectQuery data connectivity mode. Then select OK.

    Power Query Dataverse connector - Power Query (3)

Connect to Dataverse from Power Query Online

To connect to Dataverse from Power Query Online:

  1. Select the Dataverse option in the Choose data source page. More information: Where to get data

  2. In the Connect to data source page, leave the server URL address blank. Leaving the address blank lists all of the available environments you have permission to use in the Power Query Navigator window.

    Power Query Dataverse connector - Power Query (4)

    Note

    If you need to use port 5558 to access your data, you'll need to load a specific environment with port 5558 appended at the end in the server URL address. In this case, go to Finding your Dataverse environment URL for instructions on obtaining the correct server URL address.

  3. If necessary, enter an on-premises data gateway if you're going to be using on-premises data. For example, if you're going to combine data from Dataverse and an on-premises SQL Server database.

  4. Sign in to your organizational account.

  5. When you've successfully signed in, select Next.

  6. In the navigation page, select the data you require, and then select Transform Data.

    Power Query Dataverse connector - Power Query (5)

Connect using advanced options

Power Query Online provides advanced options that you can add to your query if needed.

The following table lists the advanced options you can set in Power Query Online.

Advanced optionDescription
Include relationship columnsIf checked, includes columns that might have relationships to other tables. If this box is cleared, you can’t see those columns. More information: Performance issues related to relationship columns

Once you select the advanced options you require, select Next to connect to Dataverse.

Finding your Dataverse environment URL

If you need to use port 5558 to connect to Dataverse, you have to find your Dataverse environment URL. Open Power Apps. In the upper right of the Power Apps page, select the environment you're going to connect to. Select the Power Query Dataverse connector - Power Query (6) settings icon, and then select Advanced settings.

In the new browser tab that opens, copy the root of the URL. This root URL is the unique URL for your environment. The URL is in the format of https://<yourenvironmentid>.crm.dynamics.com/. Make sure you remove https:// and the trailing / from the URL before pasting it to connect to your environment. Append port 5558 to the end of the environment URL, for example yourenvironmentid.crm.dyamics.com,5558.

Power Query Dataverse connector - Power Query (7)

When to use the Common Data Service (Legacy) connector

Dataverse is the direct replacement for the Common Data Service connector. However, there might be times when it's necessary to choose the Common Data Service (Legacy) connector instead of the Dataverse connector:

There are certain Tabular Data Stream (TDS) data types that are supported in OData when using Common Data Service (Legacy) that aren't supported in Dataverse. The supported and unsupported data types are listed in How Dataverse SQL differs from Transact-SQL.

All of these features will be added to the Dataverse connector in the future, at which time the Common Data Service (Legacy) connector will be deprecated.

More information: Accessing large semantic models

Limitations and issues

Dataverse performance and throttling limits

For information about performance and throttling limits for Dataverse connections, go to Requests limits and allocations. These limitations apply to both the Dataverse connector and the OData Feed connector when accessing the same endpoint.

Table retrieval rate

As a guideline, most default tables are retrieved at a rate of approximately 500 rows per second using the Dataverse connector. Take this rate into account when deciding whether you want to connect to Dataverse or export to data lake. If you require faster retrieval rates, consider using the Export to data lake feature or Tabular Data Stream (TDS) endpoint. For more information, go to Alternative Dataverse connections.

Alternative Dataverse connections

There are several alternative ways of extracting and migrating data from Dataverse:

  • Use the Azure Synapse Link feature in Power Apps to extract data from Dataverse into Azure Data Lake Storage Gen2, which can then be used to run analytics. For more information about the Azure Synapse Link feature, go to What is Azure Synapse Link for Dataverse?.

  • Use the OData connector to move data in and out of Dataverse. For more information on how to migrate data between Dataverse environments using the dataflows OData connector, go to Migrate data between Dataverse environments using the dataflows OData connector.

Note

Both the Dataverse connector and the OData APIs are meant to serve analytical scenarios where data volumes are relatively small. The recommended approach for bulk data extraction is “Azure Synapse Link”.

SQL Server connection issue due to closed ports

When connecting with the Dataverse connector, you might encounter an Unable to connect error indicating that a network or instance-specific error occurred while establishing a connection to SQL Server. This error is likely caused by the TCP ports 1433 or 5558 being blocked during connection. To troubleshoot the blocked port error, go to Blocked ports.

Using native database queries with Dataverse

You can connect to Dataverse using a custom SQL statement or a native database query. While there's no user interface for this experience, you can enter the query using the Power Query Advanced Editor. In order to use a native database query, a Database must be specified as the Source.

Source = CommonDataService.Database([DATABASE URL])

Once a database source is defined, you can specify a native query using the Value.NativeQuery function.

myQuery = Value.NativeQuery(Source, [QUERY], null, [EnableFolding=true])

Altogether, the query looks like this.

let Source = CommonDataService.Database("[DATABASE]"), myQuery = Value.NativeQuery(Source, "[QUERY]", null, [EnableFolding=true])in myQuery

Misspelling a column name might result in an error message about query folding instead of missing column.

Accessing large semantic models

Power BI semantic models contained in Dataverse can be very large. If you're using the Power Query Dataverse connector, any specific query that accesses the model must return less than 80 MB of data. So you might need to query the data multiple times to access all of the data in the model. Using multiple queries can take a considerable amount of time to return all the data.

If you're using the Common Data Service (Legacy) connector, you can use a single query to access all of the data in the semantic model. This connector works differently and returns the result in "pages" of 5-K records. Although the Common Data Service (Legacy) connector is more efficient in returning large amounts of data, it can still take a significant amount of time to return the result.

Instead of using these connectors to access large semantic models, we recommend that you use Azure Synapse Link to access large models. Using Azure Synapse Link is even more efficient than either the Power Query Dataverse or Common Data Service (Legacy) connectors, and is specifically designed around data integration scenarios.

Performance issues related to relationship columns

Similar to the SQL Server connector, there's an option available to disable navigation properties (relationship columns) in the Dataverse connector to improve performance. This option can be set in the advanced options available in Power Query Online, or it can be set using the CreateNavigationProperties=false parameter in the Dataverse connector function.

 Source = CommonDataService.Database("{crminstance}.crm.dynamics.com",[CreateNavigationProperties=false]),
Power Query Dataverse connector - Power Query (2024)

FAQs

What are the limitations of Dataverse? ›

Number of records per entity limit: each entity has a limit of 2 billion records. This means that you can only store 2 billion records for each custom entity. Field length limit: Dataverse has a limit of 2,000 characters for a single-line of text field and 32,768 characters for a multi-line of text field.

What is the 80mb limit for Dataverse? ›

Through official documentation, This error can occur with the Dataverse connector when you run or design a report with a result set greater than 80 MB. TDS has a result set size limit of 80 MB. To work around this limit, optimize the query adding filters and dropping columns so that the query returns less data.

What is the best practice for Dataverse table? ›

When naming tables in a Dataverse, it's best to use two or three meaningful and easily understandable words. Additionally, it's crucial to add descriptions when creating tables to help maintain applications. Using singular names for tables is recommended, as plural versions can be generated but modified later.

How much data can Power Query handle? ›

Power Query specifications and limits in Excel
FeatureLimitation
Size of data processed by the EngineLimited to available virtual memory (for 64-bit version) or about 1GB for 32-bit version, if data cannot be fully streamed, such as when sorting the data set locally before filling it
Number of columns per table16,384
12 more rows

How much data can Dataverse handle? ›

Power Automate Data Storage
Default per userAccrued per app
Dataverse Database capacity10GB50MB
Dataverse File capacity20GB200MB
Dataverse Log capacity2GB
Dec 6, 2023

Why is Dataverse better than SharePoint? ›

In conclusion, the choice between SharePoint and Dataverse depends on the specific requirements of your Power App. If your focus is on document management and collaboration, SharePoint might be the go-to option. For applications demanding a scalable and secure data platform, Dataverse is likely the better fit.

Can Dataverse handle big data? ›

Dataverse will normally try to access the file contents itself, i.e. for ingest (in future versions), full-text indexing, thumbnail creation, etc. This processing may not be desirable for large/sensitive data, and, for the case where the URL does not reference the file itself, would not be possible.

How many rows can you have in Dataverse? ›

Select Microsoft Dataverse to filter the search results to display only actions and triggers for Microsoft Dataverse. Select List rows. Select the Accounts table from the Table name list. Save and run your flow to confirm that no more than 5,000 rows are returned.

How much does Dataverse usage cost? ›

"Dataverse for Teams", each user of the application will need to pay the $5 per month to utilize.

How do you query data in Dataverse? ›

Connect to Dataverse from Power Query Online

If you need to use port 5558 to access your data, you'll need to load a specific environment with port 5558 appended at the end in the server URL address. In this case, go to Finding your Dataverse environment URL for instructions on obtaining the correct server URL address.

Why use Dataverse instead of SQL Server? ›

In contrast to SQL's limitation to canvas apps, Dataverse offers the flexibility to create both canvas and model-driven Power Apps. Model-driven apps, in particular, stand out as they are inherently mobile-responsive without requiring additional configuration.

How many apps can use the same Dataverse table? ›

Yes, you can use the same table on as many apps you need. Each app need a connection to this table.

Is Power Query a good skill? ›

Power Query is one of the most transformative tools available to any analyst working with data in Excel. Not only will it save analysts hours of time, but it will result in reduced manual errors, and a better ability to source data from a single source of truth.

Is Power Query good for ETL? ›

Because the engine is available in many products and services, the destination where the data will be stored depends on where Power Query was used. Using Power Query, you can perform the extract, transform, and load (ETL) processing of data.

What makes Power Query slow? ›

Certain operations require reading the full data source in order to return any results, and will thus be slow to preview in the Power Query Editor. For example, if you perform a sort, it's possible that the first few sorted rows are at the end of the source data.

What is the threshold limit for Dataverse? ›

Microsoft Dataverse limits the maximum number of rows that can be retrieved in a single query to 5000.

What is the limit of Dataverse environment? ›

Individual environment limits

Each Dataverse for Teams environment has 2GB of combined database and file storage. If the limit is reached, any existing resources (i.e. apps and flows) in that environment will continue to work, but it will not be possible to create or install new resources in that environment.

What is the limit of Dataverse license? ›

What Dataverse capacity is included with the Power Apps and Power Automate plans?
Power Apps capacity limitsPer license entitlement (Power Apps per app plan)Per license entitlement (Power Apps per user plan)
Dataverse Database capacity+ 50 MB+ 250 MB
Dataverse Log capacity+ 0+ 0
Dataverse File capacity+ 400 MB+ 2 GB
Apr 3, 2024

Top Articles
Latest Posts
Article information

Author: Arielle Torp

Last Updated:

Views: 6495

Rating: 4 / 5 (41 voted)

Reviews: 80% of readers found this page helpful

Author information

Name: Arielle Torp

Birthday: 1997-09-20

Address: 87313 Erdman Vista, North Dustinborough, WA 37563

Phone: +97216742823598

Job: Central Technology Officer

Hobby: Taekwondo, Macrame, Foreign language learning, Kite flying, Cooking, Skiing, Computer programming

Introduction: My name is Arielle Torp, I am a comfortable, kind, zealous, lovely, jolly, colorful, adventurous person who loves writing and wants to share my knowledge and understanding with you.