Validator Troubleshooting - Injective Docs

Troubleshooting reference and operational guide for Injective validators and node operators. Covers common failure modes, recovery procedures, upgrade checklists, and critical safety practices.

Before You Begin: Critical Safety Rules

What to never do

Never run two nodes with the same priv_validator_key.json simultaneously. This causes double-signing, 0% slash, but permanent tombstoning. The validator can never rejoin the active set.
Never use unsafe-reset-all on a validator node unless you fully understand the consequences. It wipes consensus state and can produce conflicting votes that lead to tombstoning.
Never back up priv_validator_state.json while the node is still running. The file changes continuously during consensus. Stop the node fully first, then copy it.
Never start a node after snapshot recovery without restoring your backed up priv_validator_state.json. Snapshots do not include this file. Starting without it resets your signing state to height 0, which causes double-signing.
Never rollback an archival or large history node without testing first. Rollback on large databases can take hours or brick the node entirely, especially with PebbleDB.

What to always do

Always back up priv_validator_state.json after fully stopping the node and before any recovery operation (rollback, snapshot restore, binary upgrade).
Always restore priv_validator_state.json to its original location after rollback or snapshot recovery, before starting the node.
Always verify priv_validator_state.json is intact before starting the node. Check that the height, round, and step values are sensible.
Always set --halt-height for coordinated upgrades. Nodes that skip the halt-height flag require a harder recovery path.
Always verify the binary version with injectived version before starting after an upgrade.
Always undo temporary consensus overrides (like --unsafe-consensus-timeout-precommit-delta=1ms) once the chain stabilizes.

Need a snapshot to recover? See Snapshot Resources at the bottom of this page for pruned snapshots and community providers. For archival segment snapshots, see the Archival Setup page.

Common Problems and Solutions

Node stuck at upgrade height

Symptoms:

Node logs show the chain halted at the expected upgrade height
latest_block_height does not advance past the halt height

Cause: The node is running the old binary. The chain requires the upgraded binary to proceed past the upgrade height. Solution:

Stop the node
Back up ~/.injectived/data/priv_validator_state.json
Install the new binary
Verify: injectived version
Start the node

Block header hash mismatch after upgrade

Symptoms:

ERR prevote step: consensus deems this block invalid; prevoting nil
err="wrong Block.Header.LastResultsHash. Expected 284C339C..., got 694706724..."

The node continuously prevotes nil and cannot finalize the next block. Cause: The node’s local state at the upgrade height is inconsistent with the new binary’s expected state. This happens when the node processed the halt-height block with the old binary logic before upgrading. Solution:

Stop the node
Back up priv_validator_state.json
Rollback one block:
injectived rollback
Restore priv_validator_state.json
Start the node with the new binary

Rollback on archival nodes with large databases (hundreds of GB) can take a very long time or fail. For these nodes, recover from a snapshot instead.

Conflicting vote warnings

Symptoms:

ERR Found conflicting vote from ourselves; did you unsafe_reset a validator?
height=181027006 module=consensus round=40 type=SIGNED_MSG_TYPE_PREVOTE

This message repeats continuously. Cause: The node’s priv_validator_state.json was reset or corrupted, so it is resigning at rounds it already signed for, producing conflicting votes. Common triggers:

Ran unsafe-reset-all on a validator
Restored an old or incorrect priv_validator_state.json
Backed up priv_validator_state.json while the node was still running (stale copy)
Rollback command reset the validator state

Solution:

If you have a correct backup of priv_validator_state.json (taken after fully stopping the node): restore it and restart.
If you do not have a correct backup: stop the node immediately and wait for block production to resume before restarting. Starting with incorrect state risks double-signing.

Check signing info:

injectived query slashing signing-info <injvalcons_address>

If conflicting votes are detected on-chain, the validator will be tombstoned (permanently jailed). There is no on-chain unjail for double-signing.

Validator tombstoned (double-signed)

Symptoms:

injectived query slashing signing-info injvalcons1...
  tombstoned: true
  jailed_until: "9999-12-31T23:59:59Z"

Cause: The validator signed two different blocks or prevotes at the same height. Common triggers:

Running two instances of the node with the same signing key
Restoring an old priv_validator_state.json that caused re-signing at a previously signed height
Rollback resetting signing state, followed by the node signing a conflicting block

Resolution: There is no standard on-chain unjail for tombstoned validators. Resolution paths:

A governance proposal or upgrade handler that explicitly un-tombstones affected validators
Creating a new validator with a new operator key (loses existing delegations)

If you were tombstoned during a coordinated security upgrade, reach out to the Injective team in the verified validators channels immediately. During coordinated upgrades, the team may include an un-tombstone handler in the upgrade binary to restore affected validators. Time sensitive, report as soon as possible so you can be included before the upgrade handler is finalized.

Prevention:

Use TMKMS or Horcrux for hardware-enforced single-signer semantics
Always back up priv_validator_state.json after stopping the node
Never run two nodes with the same signing key simultaneously

Recovering from a snapshot

When to use: When rollback fails, takes too long, or the node state is corrupted beyond repair (apphash mismatch). Procedure:

Stop the node completely
Back up ~/.injectived/data/priv_validator_state.json
Back up ~/.injectived/config/priv_validator_key.json if not already backed up elsewhere
Download a snapshot from a trusted provider:
- Injective Team may also share emergency snapshots before or during security upgrades
- Polkachu Injective Snapshots (typically goleveldb)
- Community validators may share emergency snapshots during incidents
Remove old data:
rm -rf ~/.injectived/data
Extract the snapshot into ~/.injectived/data/
Restore your backed-up priv_validator_state.json into ~/.injectived/data/
Verify priv_validator_state.json is present and correct
Start the node

Why you must keep your own `priv_validator_state.json`

Snapshots contain blockchain data (blocks, application state) but never include your validator’s signing state. The signing state tracks the last height, round, and step your validator signed at. If you lose it or replace it with a blank one, your node doesn’t know what it already signed and may sign a conflicting block — causing double-signing and permanent tombstoning. Here’s a practical example showing why the signing state must always be ahead of (or equal to) the snapshot height. Setup: A coordinated chain upgrade with halt-height at block 125.

Block:  100  105  110  115  120  125  126
         |    |    |    |    |    |    |
         |    |    |    |    |    |    Chain resumes with new binary
         |    |    |    |    |    Chain halts here (upgrade height)
         |    |    |    |    |
         |    |    |    Snapshot taken at block 115
         |    |    |
         Your node has been signing every block...

At the halt, your priv_validator_state.json reads:

{
  "height": "126",
  "round": 42,
  "step": 3,
  "signature": "...",
  "signbytes": "..."
}

Why does it say height 126 if the halt height was 125? The --halt-height flag prevents block 126 from being committed (finalized), but the CometBFT consensus engine still enters height 126 and begins its rounds — proposing, prevoting, and precommitting — before the application layer refuses to process the block and the node shuts down. The high round number (42) reflects the network churning through rounds as validators gradually stop and consensus can’t be reached. This is normal behavior: your validator cast real votes at height 126 even though that block was never finalized. This means your validator has already cast votes at height 126, round 42. The available snapshot is from height 115. What happens if you restore the snapshot WITHOUT your signing state: The snapshot comes with either no priv_validator_state.json or a blank one (height 0). If you start the node this way:

Your node loads blockchain data from height 115
It catches up to height 126 by replaying blocks 116-125
At height 126, it enters consensus and starts signing from round 0
But the network already has your votes from round 42 at this height
Your node signs a different block proposal at height 126 (because it replayed with the new binary and may produce a different result)
The network detects two conflicting signatures from your validator key
Your validator is double-signed and tombstoned. Permanently.

The correct procedure:

Stop the node
Back up your priv_validator_state.json (it says height 126, round 42)
Delete your data directory
Extract the snapshot (blockchain data from height 115)
Copy your backed-up priv_validator_state.json back into the data directory
Start the node

Now when the node starts:

It loads blockchain data from height 115
It catches up to height 126 by replaying blocks 116-125
At height 126, it reads the signing state: “I already signed up to round 42”
It skips rounds 0-42 and waits for round 43 before signing anything new
No conflicting votes. Your validator is safe.

Key rule: The signing state must always reflect the highest point your validator has ever signed at. It is safe for the signing state to be ahead of the snapshot (the node will catch up). It is never safe for the signing state to be behind what the validator actually signed on the network.

Scenario	Signing state height	Snapshot height	Safe?
Normal recovery	126	115	Yes — node replays 116-125, skips already-signed rounds at 126
Fresh snapshot, kept state	126	120	Yes — same logic, fewer blocks to replay
Snapshot without state	0	115	No — will re-sign at heights it already signed, causing double-sign
Stale backup (taken at 110)	110	115	No — will re-sign at heights 111-126

Database backend compatibility: Snapshots are backend specific. A goleveldb snapshot will not work on a node configured for pebbledb, and vice versa. Check your configuration:

grep db_backend ~/.injectived/config/config.toml

Rollback hangs or fails

Symptoms: injectived rollback runs but never completes, or produces errors. Causes:

PebbleDB backend: Rollback with pebble has known issues and may hang indefinitely
Very large database: Rollback time scales with database size
Corrupted WAL (Write-Ahead Log)

Solutions:

For PebbleDB nodes: recover from a snapshot instead
For large nodes: if rollback hasn’t completed in 30+ minutes, abort and use a snapshot
Consider switching to goleveldb. PebbleDB is more space efficient but less tested for rollback

Round regression errors during catchup

Symptoms:

ERR Failed signing vote err="error signing vote: round regression at height 181027006.
Got 108, last round 141"

Cause: The node is catching up to the current consensus round. This is normal and expected when restarting a node during an active consensus round that has advanced through many rounds. Solution: No action needed. The node will catch up. To speed up round catch-up:

# Flag
--unsafe-consensus-timeout-precommit-delta=1ms

# Or environment variable
INJECTIVED_UNSAFE_CONSENSUS_TIMEOUT_PRECOMMIT_DELTA=1ms

Remove this override once the chain stabilizes and the node is caught up. Revert to the default (100ms) and restart.If left on during normal operation, the 1ms delta causes rounds to advance far faster than votes can propagate across the network. Your validator moves to the next round before other validators’ precommits arrive, so its votes miss the actual commit round. The result is a steadily climbing missed-block counter, and if enough blocks are missed (typically 500 of the last 10,000), the validator will be jailed for downtime.

AppHash mismatch

Symptoms:

panic: Tendermint state.AppHash does not match AppHash after replay

Cause: The application state diverged from the consensus state. Can result from an incomplete upgrade, a crash during block execution, or a corrupted database. Solution:

Try rollback: injectived rollback
If rollback fails or the error persists: recover from a snapshot
Always restore priv_validator_state.json after recovery

Sentry nodes stuck after upgrade

Symptoms: The validator node is upgraded and signing, but sentry nodes remain stuck at the old block height. Cause: Sentry nodes also need the new binary to process blocks past the upgrade height. Solution: Upgrade sentry nodes to the same binary version as the validator. Sentries do not have priv_validator_state.json (they don’t sign), but they do require the correct binary.

Chain stalled after upgrade (insufficient voting power)

Symptoms:

The chain does not produce new blocks after the upgrade height
Consensus round numbers climb continuously (round 50, 100, 200+)
Nodes log prevotes and precommits but no block is committed
latest_block_height remains stuck at the halt height

Cause: CometBFT requires 2/3+ of total voting power to be online and participating with the correct binary to finalize a block. During a coordinated upgrade, there is a window where validators are upgrading at different speeds. Until enough voting power is running the new binary, the network cannot reach consensus. What happens during this window:

The chain halts at the upgrade height (ex: block 125)
Validators that have upgraded start participating in consensus at height 126
Each round, the network attempts to finalize block 126 but fails because less than 2/3 of voting power is voting
The round number increments and the cycle repeats, round 1, 2, 3, … 50, … 100+
This continues until enough validators complete the upgrade and come online

This is normal during upgrades. The chain is not broken, it is waiting for quorum. What to do:

If you have already upgraded: No action needed. Your node is participating and will automatically finalize the next block once quorum is reached. You will see round regression errors in the logs as your node catches up through rounds. This is expected (see Round regression errors during catchup).
If you have not yet upgraded: You are part of the missing voting power. Complete the upgrade as quickly as possible to help the chain resume.

Monitor progress: Check how many validators are online and participating:

curl -s localhost:26657/consensus_state | jq '.result.round_state.votes'

Coordinate in validator channels: During major upgrades, validators typically coordinate in the verified validators channels to track upgrade progress and voting power percentage.

The longer this window lasts, the higher the round numbers climb. Validators that come online later will need to catch up through all those rounds, which is why the --unsafe-consensus-timeout-precommit-delta=1ms flag exists but remember to remove it after the chain stabilizes.

Jailed for downtime (not tombstoned)

Symptoms:

Validator shows as inactive or jailed
tombstoned: false in signing info
Missed blocks counter is high

Cause: The validator missed too many consecutive blocks (typically 500 of the last 10,000). Solution:

Ensure the node is running and fully caught up to the latest block

Unjail:

injectived tx slashing unjail \
  --from=<key_name> \
  --chain-id=injective-1 \
  --gas=auto \
  --gas-adjustment=1.5

Verify the validator is active again:

injectived query staking validator <injvaloper_address>

Coordinated Upgrade Checklist

Before the upgrade height

Confirm the target halt-height from the governance proposal or chain team
Download and verify the new binary (check sha256 checksums)
Set --halt-height=<target_height> in node config or CLI flags
If using Cosmovisor: place the new binary in the correct upgrade directory
Back up priv_validator_state.json
Identify snapshot providers in case rollback is needed
Monitor chain progression toward the halt height

At the upgrade height

Confirm the node has halted at the expected height
Stop the node fully (verify the process is dead)
ps aux | grep injectived
Back up priv_validator_state.json
Install the new binary
Verify the version:
injectived version

After the upgrade

Start the node with the new binary
Monitor logs for consensus participation
Confirm prevotes are correct (not voting nil on every round)
Monitor voting power on an explorer (Mintscan)
If seeing wrong Block.Header.LastResultsHash: stop, rollback 1 block, restore priv_validator_state.json, restart
If using round-catchup overrides (--unsafe-consensus-timeout-precommit-delta): remove them once stable

Verify the validator is signing:

curl -s localhost:26657/consensus_state | jq '.result.round_state["height/round/step"]'

Monitoring Reference

Consensus and sync status

# Current consensus state (height, round, step)
curl -s localhost:26657/consensus_state \
  | jq '.result.round_state["height/round/step"] | split("/") | {height: .[0], round: .[1], step: .[2]}'

# Latest block height
curl -s localhost:26657/status | jq '.result.sync_info.latest_block_height'

# Whether the node is still catching up
curl -s localhost:26657/status | jq '.result.sync_info.catching_up'

Validator health

# Signing info (missed blocks, jail status, tombstone status)
injectived query slashing signing-info $(injectived tendermint show-validator)

# Validator status and jail state
injectived query staking validator <injvaloper_address> --output json | jq '.status, .jailed'

Node configuration

# Current binary version
injectived version

# Current priv_validator_state (check height/round/step)
cat ~/.injectived/data/priv_validator_state.json | jq

# Database backend
grep db_backend ~/.injectived/config/config.toml

Pruned vs Archival Nodes

	Pruned Node	Archival Node
Rollback	Fast (minutes)	Slow (minutes to hours), may fail
Snapshot recovery	Fast (10-30 min)	Very slow (hours, snapshots 100+ GB)
Typical DB size	10-50 GB	500+ GB
Recommended recovery	Rollback first, snapshot as fallback	Snapshot preferred, rollback is risky

Additional Resources

Snapshot Resources

Injective Provided Snapshots

Injective maintains pruned mainnet snapshots in two regions. Check the status endpoint for the latest available height and download URL:

Region	Status	Download
EU (OVH)	status.json	`http://injective-mainnet-snapshots.s3-website.gra.io.cloud.ovh.net/`
Asia (GCS)	status.json	`https://storage.googleapis.com/injective-mainnet-snapshots-asia/`

To download the latest snapshot, check the status endpoint for the current filename:

# Check latest available snapshot (Asia example)
curl -s https://storage.googleapis.com/injective-mainnet-snapshots-asia/status.json | jq

# Download the snapshot
wget <url from status.json>

# Extract
lz4 -d <snapshot_file>.tar.lz4 | tar xf - -C ~/.injectived/data/

During coordinated security upgrades, the Injective team may also share emergency snapshots in the verified validators channels. For archival segment snapshots, see the Archival Setup page.

Community Snapshot Providers

Polkachu Injective Snapshots (typically goleveldb, pruned)
HighStakes Injective Snapshots
Community validators may share emergency snapshots during incidents in the validator channels

Database backend compatibility: Snapshots are backend specific. A goleveldb snapshot will not work on a node configured for pebbledb, and vice versa. Check your configuration before downloading:

grep db_backend ~/.injectived/config/config.toml

You must restore your backed up priv_validator_state.json into the data directory before starting the node after a snapshot recovery. See Recovering from a snapshot above for the full procedure.

​Before You Begin: Critical Safety Rules

​Common Problems and Solutions

​Node stuck at upgrade height

​Block header hash mismatch after upgrade

​Conflicting vote warnings

​Validator tombstoned (double-signed)

​Recovering from a snapshot

​Why you must keep your own priv_validator_state.json

​Rollback hangs or fails

​Round regression errors during catchup

​AppHash mismatch

​Sentry nodes stuck after upgrade

​Chain stalled after upgrade (insufficient voting power)

​Jailed for downtime (not tombstoned)

​Coordinated Upgrade Checklist

​Before the upgrade height

​At the upgrade height

​After the upgrade

​Monitoring Reference

​Consensus and sync status

​Validator health

​Node configuration

​Pruned vs Archival Nodes

​Additional Resources

​Snapshot Resources

​Injective Provided Snapshots

​Community Snapshot Providers

Before You Begin: Critical Safety Rules

Common Problems and Solutions

Node stuck at upgrade height

Block header hash mismatch after upgrade

Conflicting vote warnings

Validator tombstoned (double-signed)

Recovering from a snapshot

Why you must keep your own `priv_validator_state.json`

Rollback hangs or fails

Round regression errors during catchup

AppHash mismatch

Sentry nodes stuck after upgrade

Chain stalled after upgrade (insufficient voting power)

Jailed for downtime (not tombstoned)

Coordinated Upgrade Checklist

Before the upgrade height

At the upgrade height

After the upgrade

Monitoring Reference

Consensus and sync status

Validator health

Node configuration

Pruned vs Archival Nodes

Additional Resources

Snapshot Resources

Injective Provided Snapshots

Community Snapshot Providers