Post/Put huge volume of data to an api endpoint through ADF

I have few output files in my unity catalog volumes/adls gen2. I want to push the data of these files as a body for api endpoint. However the data from my table is huge which goes upto 2 million rows, which is then converted to the below formatted json file:

dataload:[

{

"Metadata": {

            "FinishExtract": true,

            "Timestamp": "20261016.124853",

            "OperationType": "Full"

           }

   },

   {

          Data: {

                    "Rows": [

                                 {

                                         "Values":[

                                                     "a",

                                                     "b",... "c"..,d,e,f,g,h,i]},

                                 {

                                          "Values":[j,k,l,m,n,o,p,q,r]}}} and so on.

this is a sample, I have total of 9 fields. The API only accepts this format as a downstream. Now, I want to understand what would be the right approach to send it in batches. The API can have 20 request and 0.5 million rows per request. The only option for me is to use ADF due to data restrictions.

I tried/explored the below approaches:

1. Lookup + web (works, but size limit)
2. Copy activity (content type conflict as I have to pass application/json;charset=UTF-8. I tried multiple options but nothing worked)
3. Dataflows - Due to complex json, it was not able to format.
4. Azure batch jobs/Azure functions/logic apps - Access restrictions, but this approach can be used if there's a workaround to solve the issue.
5. Databricks job/notebooks - Access restricted, but this approach can be used if nothing else works.
6. Databricks job + Web activity+ web activity - size limit.

I need my first request to go as a post request followed by put request. I can control the size and volumes my json files being generated. However, I can restrict the total volumes. I am looking for a scalable design.

Appreciate the help in advance!

Thank you.