Token consumption

Question

Token consumption

Mansi Gusain 615

Hi, I wanted to check how many tokens are being used up by different models in azure openai, in the metrics section I am not able to see the exact token usage. On the count aggeration it is giving 4 what is that suppose to indicate. User's image

I have option for active token and prompt completion token, but I want to see how many are being used. And why is active tokens max=0? is there a way by which I can see how every model is generating the token - individual data.

User's image

Anonymous

2025-01-26T15:34:59.3866667+00:00

Hi Mansi Gusain

Greetings & Welcome to the Microsoft Q&A forum! Thank you for sharing your query.

The token consumption for different models in Azure OpenAI can vary based on several factors, including the model being used and the nature of the input. Each model has a maximum number of tokens that can be used for system messages and responses. For example, the maximum tokens for the system message can range from 400 to 4000 depending on the model.

The "count aggregation" showing 4 could refer to the number of tokens processed in a specific request, but without more context, it's difficult to determine its exact meaning. The "active tokens max=0" might indicate that there are currently no active tokens being utilized for that specific session or request.

To see how many tokens are being used by each model, you can refer to the token estimates provided for different prompts and completions. The total number of tokens processed in a request includes both input and output tokens, which can affect response latency and throughput.

If you're looking for detailed metrics on token usage for each model, you might need to check the specific API responses or logs that provide insights into token consumption for individual requests.

I hope this helps you. Thank you.
Anonymous

2025-01-27T08:20:31.7066667+00:00

Hi Mansi Gusain

Following up to see if the above response was helpful.

2 answers

Your answer

Anonymous

2025-01-26T15:34:59.3866667+00:00

Hi Mansi Gusain

Greetings & Welcome to the Microsoft Q&A forum! Thank you for sharing your query.

The token consumption for different models in Azure OpenAI can vary based on several factors, including the model being used and the nature of the input. Each model has a maximum number of tokens that can be used for system messages and responses. For example, the maximum tokens for the system message can range from 400 to 4000 depending on the model.

The "count aggregation" showing 4 could refer to the number of tokens processed in a specific request, but without more context, it's difficult to determine its exact meaning. The "active tokens max=0" might indicate that there are currently no active tokens being utilized for that specific session or request.

To see how many tokens are being used by each model, you can refer to the token estimates provided for different prompts and completions. The total number of tokens processed in a request includes both input and output tokens, which can affect response latency and throughput.

If you're looking for detailed metrics on token usage for each model, you might need to check the specific API responses or logs that provide insights into token consumption for individual requests.

I hope this helps you. Thank you.
Anonymous

2025-01-27T08:20:31.7066667+00:00

Hi Mansi Gusain

Following up to see if the above response was helpful.

Answer 1

Check the current A5 evaluation status without starting or rerunning anything.

Inspect the background task/process state, last log timestamps, progress counters, watchdog and existing output files.

If evaluation has completed, collect the results and finish the report.

If it is still running, report its exact progress and whether the watchdog is functioning.

If it has stalled or failed, identify the blocker and preserve all partial outputs. Do not restart training or evaluation without approval.

Do not change any safety gates, checkpoints, baselines or Git settings.

Report: PROCESS_STATUS= LAST_PROGRESS= ELAPSED_TIME= WATCHDOG_STATUS= OUTPUT_STATUS= NEXT_ACTION=

Answer 2

Hello Mansi Gusain,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

I understand that you would like to know how many tokens are being used up by different models in Azure OpenAI.

To breakdown your questions:

Count Aggregation Showing 4: This typically means that there are four data points being aggregated. It doesn't directly indicate the number of tokens used but rather the number of records or events being counted.
Active Tokens Max=0: This could indicate that there are no active tokens being used at the time of measurement. Active tokens usually refer to tokens currently in use during an ongoing session or request.
Seeing Token Usage for Each Model: To get detailed token usage for each model, you can use the following steps:
1. Metric Namespace: Select the appropriate namespace related to Azure OpenAI.
2. Metric: Choose metrics like "Prompt Tokens" and "Completion Tokens" to see the tokens used for prompts and completions.
3. Aggregation: Use aggregations like "Sum" to get the total number of tokens used over a period.

For more granular data, you might need to use Azure Monitor logs or Application Insights to track and analyze token usage per request. This can give you detailed insights into how each model is generating and using tokens.

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful.

Mansi Gusain 615 Reputation points

2025-01-22T07:14:09.2366667+00:00

Hey thanks for replying, but even if there is an ongoing session, the active tokens point to zero only
Mansi Gusain 615 Reputation points

2025-01-22T07:14:32.2833333+00:00

the image I shared was when active session was ongoing.