An Azure service for ingesting, preparing, and transforming data at scale.
Copy data from Sharepoint using Authorization: Bearer token getting status code 401 Unauthorized
ErrorCode=HttpRequestFailedWithUnauthorizedError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Http request failed with status code 401 Unauthorized, usually this is caused by invalid credentials, please check your activity…
Azure Data Factory
Data Factory: HubSpot Connector Out of Date — Errors with Certain Endpoints
Problem description I am experiencing errors when using the Azure Data Factory HubSpot connector 2.0, particularly when accessing certain endpoints like the marketing campaigns table during a source-data update pipeline. The behavior did not meet my…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
Error occurring during the SSH handshake with ADF
I am trying to find a solution on this error when I tried to copy data from a client who uses AWS for SFPT. I am using ADF (Azure Data Factory) to move data from this client's SFTP. Bottom is details of error…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
How to bypass locked temporary Office files (~$*) during bulk FileServer-to-ADLS copy in ADF without high cost?
Hi Everyone, I am building an incremental ingestion pipeline in Azure Data Factory (ADF) to copy ~70,000 files (across 47 different file types) from an on-premises FileServer into ADLS Gen2 using a Self-Hosted Integration Runtime (SHIR). The Challenge We…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
Unable to delete data workflows gen2 tasks under fabric workspace
Hi Team, Under fabric workspace, we have created Gen2 dataflow tasks. We would like to delete the Gen2 Workflow tasks, by removing the linages, but we are getting error. Then same is able to delete after some days, either it takes 1 week or 10days. In…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
How to Ingest a table of xml data from on-prem DB2 warehouse to ASDLGen2 using ADF
Hi, I am using ADF to incrementally load a table containing 2 XML columns from on-prem DB2 warehouse to ADSLGen2 in parquet format. In Copy activity, I use DB2 connector at source and parquet dataset at sink; I also use Query with XMLSERIALIZE for these…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
Setup ADF authen with personal github
I have error when I try setup repo git integration with new credential it's not working. check in tab network it return 401 but I don't see any step input credential or token???? why it have token in here?? where it come from??
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
Issues Azure data Factory SFTP connection ( SSH client)
HI MS Team, We encounter some issues with connecting a SFTP server of one of our financial counter parties. The issue is due to security level that they do not support rsa-sha but only the rsa-sha2 version for host key exchange. If i look at the error…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
Azure Data Factory Copy Activity fails extracting ZIP files to ADLS Gen2 with BadRequest for Unicode/Vietnamese filenames — possible ZIP filename encoding issue
We are using Azure Data Factory Copy Activity to extract ZIP files and write the extracted files to Azure Data Lake Storage Gen2. The Copy Activity fails for some files/directories inside the ZIP with the following error: The ZIP contains…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
Recommended technologies for creating a dashboard of ongoing jobs.
I'm looking for recommendations on technologies to use for the following project. I need to create a dashboard of summarized job data from a network of hundreds of users. Our current application is used in the field for inputting and tracking job costs…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
Ingest multiple xml files through ADF which has different schemas
Hi all, I have an Azure blob storage of xml files with at least two different schemas. I would like to ingest all of these files through ADF (probably using Dataflow) and only process one type of schema to sink to a SQL database. How is this done…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
Update of parentaccountid into CRM from ADF
I'm trying to update records from a .csv file through ADF copy activity into CRM. It's working fine for the GUIDs present for parentaccountid and updating accordingly. However, when a blank value is sent for this field, it's not removing the existing…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
How to fix: Request Id: fccd854a-cfc2-43a1-89dd-0403c2493700 Correlation Id: 4fb66761-4ec3-477e-8c3b-d9c2a5c8562e Timestamp: 2024-01-16T19:29:03Z Message: AADSTS900561: The endpoint only accepts POST requests. Received a GET request.
Request Id: fccd854a-cfc2-43a1-89dd-0403c2493700 Correlation Id: 4fb66761-4ec3-477e-8c3b-d9c2a5c8562e Timestamp: 2024-01-16T19:29:03Z Message: AADSTS900561: The endpoint only accepts POST requests. Received a GET request. I can´t login in the web anymore…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
Checkpoint Key on Azure Data Factory (ADF)
Hi, We have created Azure Data Flow with Azure SQL Database Managed Instance as source and Azure Data Lake Storage Gen 2 (ADLS) as a sink (parquet format). Change Data Capture is enabled at the source with SQL Server CDC type and "Full on the first…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
Fabric scan is ingesting Lakehouse schemas but not Warehouse schemas in my environment.
Hi All I am trying to scan my Fabric environment into Purview but I have hit a snag, I want to be able to drill down into the my datawarehouse to extract the Tables and Views, but I am not sure how to do it When I open the Lineage in the scanned…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
Trying to filter csv file based on content
I have an existing ADF pipeline that is creating an error due to some bad (or not acceptable data) displayed on the csv file. Basically, instead of having proper header on the first row with certain amount columns (with data), csv file displays data like…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
@microsoft/azure-data-factory-utilities: purpose of the factoryId parameter
Hi, I have recently started using the @microsoft/azure-data-factory-utilities package to generate ARM templates in an Azure DevOps pipeline. It is working, and I am able to generate the ARM templates successfully. :) However, I have a question about the…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
SFTP connection problem from Azure Data Factory to a server that only supports rsa-sha2-256/512
Hello, I am trying to connect from Azure Data Factory (both with Managed IR and Self-Hosted IR) to the SFTP server of a supplier. But i receive the following error: "Failed to connect to Sftp server 'xxxxxx'. The connection was closed by the…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
enable the access to local machine under self hosted integration runtime
enable the access to local machine under self hosted integration runtime
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
How to handle BigQuery mirroring when the source has more than 1,000 tables?
Hi, We are setting up database mirroring in a Microsoft Fabric workspace, with BigQuery as the source. Currently, our BigQuery source contains 1,193 tables, but a single mirrored database supports a maximum of 1,000 tables. Because of this limitation, we…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.