Splunk Search Head Cluster Out of Sync

Hello Chang Min,

When Search Head Cluster members return different search results, the usual cause is that the baseline configuration bundle on one or more members is out of sync. The correct remediation is to push the deployer bundle again from the deployer using the apply shcluster-bundle operation, which redistributes the configuration bundle to all cluster members. Splunk documents that the deployer is the authoritative mechanism for distributing baseline app and configuration changes to Search Head Cluster members, and that bundle distribution occurs when explicitly triggered from the deployer.

Before forcing synchronization, verify that the cluster is healthy and that the captain is reachable by reviewing the Search Head Cluster status. After the bundle push, validate configuration replication and member status to confirm that all nodes are running the same configuration generation and that no replication errors remain. Splunk also recommends reviewing the Search Head Clustering monitoring dashboards and replication status if synchronization continues to fail, as underlying captain, replication, or deployment issues can prevent members from applying the bundle successfully.

If my answer is useful for you, please hit Accept the answer for me please.

HL.

Hello Chang,

Thank you for posting your question on the Microsoft Windows Forum!

If members of your Splunk Search Head Cluster are returning different search results, and you've already identified a bundle replication/synchronization issue, the first step is to verify whether the deployer bundle has been successfully pushed to all search heads before investigating other causes.

Step 1: Check Search Head Cluster Status

Run the following on each search head:

splunk show shcluster-status

Verify:

• All members are listed.
• No members are marked as down or out-of-sync.
• The same configuration generation is reported across members.

Step 2: Force a Deployer Bundle Push

From the deployer, push the latest configuration bundle:

splunk apply shcluster-bundle </span>
-target https://<captain-name>:8089 </span>
-auth admin:<password>

After the bundle push completes, confirm that all search heads receive and apply the updated bundle.

Step 3: Validate Bundle Replication

On the cluster captain, review:

splunk show shcluster-status --verbose

Look for:

bundle replication status

conf replication status

Any failed or pending replication entries may indicate why cluster members are returning different search results.

Step 4: Check the Splunk Logs

Review:

/splunkd.log

Search for:

ConfReplication

apply shcluster-bundle

bundle replication

captain

These entries often reveal why a bundle was not successfully applied.

Step 5: Confirm Knowledge Object Consistency

Since search results differ between members, verify that:

• Apps
• Saved searches
• Macros
• Lookup files
• Knowledge objects

have successfully replicated to all members.

A failed bundle push may leave one or more members with stale configurations

I hope this answer has provided you with useful information. If so, please click "Accept answer" and consider upvoting it. This helps other community members find useful solutions to similar problems.