Splunk Universal Forwarder Parsing Queue Blocked

Hello Farid Ali,

When a forwarder queue reaches 100% capacity, the issue is usually downstream, meaning the indexers are not consuming data as fast as it is being sent. Splunk recommends first determining whether the indexing tier is saturated by reviewing queue fill patterns, indexing rate, and indexer health, as full event-processing queues and blocked forwarding commonly indicate indexing performance bottlenecks.

Start by verifying CPU utilization, storage latency, and IOPS on the indexers, as insufficient CPU resources or slow storage are among the most common causes of sustained queue saturation. If the ingestion volume has increased, scale the indexing layer by adding indexers or increasing available indexing resources so that pipeline throughput matches the incoming data rate.

On the forwarder side, review the tcpout_* queue fill metrics and throughput patterns to confirm whether the backlog is caused by temporary ingestion bursts or a persistent downstream bottleneck. Avoid simply increasing queue sizes as a primary fix, since larger queues only buffer more data and do not improve indexing performance; Splunk specifically recommends addressing the root cause in the indexing or network path first.

After tuning, validate that outbound queues no longer remain at 100%, receiving ports on the indexers stay open, and ingestion delay returns to normal. Consistently monitoring queue fill ratios, indexing throughput, CPU utilization, and storage performance will help ensure forwarder throughput remains aligned with indexer capacity.

If my answer is useful for you, please hit Accept the answer for me please.

HL.

Hello Farid,

Thank you for posting your question on the Microsoft Windows Forum!

If the Universal Forwarder stops sending data when queues reach 100% capacity, the issue is usually not the forwarder itself but a bottleneck somewhere downstream:

• Indexers are not ingesting data fast enough.
• Parsing/indexing pipelines are saturated.
• Network throughput is constrained.
• Heavy searches are competing for indexing resources.

The first step is to identify where the backlog is occurring before increasing throughput settings.

Step 1: Check Queue Status

On the Splunk instance, review the ingestion queues:

index=_internal source=*metrics.log group=queue

or monitor:

index=_internal source=*metrics.log blocked=true

Look for queues that remain consistently full or blocked.

Step 2: Check Indexer Pipeline Utilization

On the indexers, review:

index=_internal source=*metrics.log group=pipeline

Pay attention to:

indexpipeline
parsingpipeline
aggregationpipeline

If parsing or indexing pipelines are consistently near capacity, the indexers may need additional resources or scaling.

Step 3: Review Forwarder Throughput

Check whether the forwarder is falling behind by reviewing:

splunk list forward-server

and:

Settings
→ Monitoring Console
→ Forwarder and Indexing Performance

Verify that:

• Connections remain established.
• Indexers are responding normally.
• No persistent acknowledgement delays exist.

Step 4: Verify Resource Utilization

On the affected indexers, monitor:

CPU
Memory
Disk I/O
Storage latency

Even if queues are filling, the underlying bottleneck is often:

• High CPU utilization
• Slow disk writes
• Insufficient IOPS
• Storage contention

Step 5: Scale the Environment

If the indexers are consistently saturated, consider:

• Adding additional indexers.
• Increasing indexer CPU and memory resources.
• Redistributing data ingestion.
• Reviewing indexing workloads and retention policies.

Additional Information Needed

To provide more specific guidance, could you please share:

1. Splunk version currently in use?
2. Universal Forwarder version?
3. Number of indexers?
4. Which queue is reaching 100%?
5. Are you seeing warnings in:

splunkd.log
metrics.log

related to blocked queues or indexing delays?

Vendor Documentation

• https://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/Aboutmetricslogs
• https://docs.splunk.com/Documentation/Splunk/latest/Monitoring/UseDMC
• https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Aboutforwardingandreceivingdata

If the information provided was helpful, please consider clicking Accept Answer and consider upvoting it. Doing so helps other community members identify useful solutions to similar issues