> ## Documentation Index
> Fetch the complete documentation index at: https://docs.injective.network/llms.txt
> Use this file to discover all available pages before exploring further.

# Validator Troubleshooting

Troubleshooting reference and operational guide for Injective validators and node operators. Covers common failure modes, recovery procedures, upgrade checklists, and critical safety practices.

***

## Before You Begin: Critical Safety Rules

<Callout icon="warning" color="#FF8C00" iconType="regular">
  **What to never do**

  1. **Never run two nodes with the same `priv_validator_key.json` simultaneously.** This causes double-signing, 0% slash, but permanent tombstoning. The validator can never rejoin the active set.
  2. **Never use `unsafe-reset-all` on a validator node** unless you fully understand the consequences. It wipes consensus state and can produce conflicting votes that lead to tombstoning.
  3. **Never back up `priv_validator_state.json` while the node is still running.** The file changes continuously during consensus. Stop the node fully first, then copy it.
  4. **Never start a node after snapshot recovery without restoring your backed up `priv_validator_state.json`.** Snapshots do not include this file. Starting without it resets your signing state to height 0, which causes double-signing.
  5. **Never rollback an archival or large history node without testing first.** Rollback on large databases can take hours or brick the node entirely, especially with PebbleDB.

  **What to always do**

  1. **Always back up `priv_validator_state.json` after fully stopping the node** and before any recovery operation (rollback, snapshot restore, binary upgrade).
  2. **Always restore `priv_validator_state.json`** to its original location after rollback or snapshot recovery, before starting the node.
  3. **Always verify `priv_validator_state.json` is intact before starting the node.** Check that the height, round, and step values are sensible.
  4. **Always set `--halt-height` for coordinated upgrades.** Nodes that skip the halt-height flag require a harder recovery path.
  5. **Always verify the binary version** with `injectived version` before starting after an upgrade.
  6. **Always undo temporary consensus overrides** (like `--unsafe-consensus-timeout-precommit-delta=1ms`) once the chain stabilizes.
</Callout>

<Callout icon="info" color="#22C55E" iconType="regular">
  **Need a snapshot to recover?** See [Snapshot Resources](#snapshot-resources) at the bottom of this page for pruned snapshots and community providers. For archival segment snapshots, see the [Archival Setup](/infra/archival-setup) page.
</Callout>

***

## Common Problems and Solutions

### Node stuck at upgrade height

**Symptoms:**

* Node logs show the chain halted at the expected upgrade height
* `latest_block_height` does not advance past the halt height

**Cause:** The node is running the old binary. The chain requires the upgraded binary to proceed past the upgrade height.

**Solution:**

1. Stop the node
2. Back up `~/.injectived/data/priv_validator_state.json`
3. Install the new binary
4. Verify: `injectived version`
5. Start the node

***

### Block header hash mismatch after upgrade

**Symptoms:**

```
ERR prevote step: consensus deems this block invalid; prevoting nil
err="wrong Block.Header.LastResultsHash. Expected 284C339C..., got 694706724..."
```

The node continuously prevotes nil and cannot finalize the next block.

**Cause:** The node's local state at the upgrade height is inconsistent with the new binary's expected state. This happens when the node processed the halt-height block with the old binary logic before upgrading.

**Solution:**

1. Stop the node
2. Back up `priv_validator_state.json`
3. Rollback one block:
   ```bash theme={null}
   injectived rollback
   ```
4. Restore `priv_validator_state.json`
5. Start the node with the new binary

<Warning>
  Rollback on archival nodes with large databases (hundreds of GB) can take a very long time or fail. For these nodes, recover from a snapshot instead.
</Warning>

***

### Conflicting vote warnings

**Symptoms:**

```
ERR Found conflicting vote from ourselves; did you unsafe_reset a validator?
height=181027006 module=consensus round=40 type=SIGNED_MSG_TYPE_PREVOTE
```

This message repeats continuously.

**Cause:** The node's `priv_validator_state.json` was reset or corrupted, so it is resigning at rounds it already signed for, producing conflicting votes.

**Common triggers:**

* Ran `unsafe-reset-all` on a validator
* Restored an old or incorrect `priv_validator_state.json`
* Backed up `priv_validator_state.json` while the node was still running (stale copy)
* Rollback command reset the validator state

**Solution:**

* If you have a correct backup of `priv_validator_state.json` (taken after fully stopping the node): restore it and restart.
* If you do not have a correct backup: **stop the node immediately and wait for block production to resume before restarting.** Starting with incorrect state risks double-signing.
* Check signing info:
  ```bash theme={null}
  injectived query slashing signing-info <injvalcons_address>
  ```

<Warning>
  If conflicting votes are detected on-chain, the validator will be tombstoned (permanently jailed). There is no on-chain unjail for double-signing.
</Warning>

***

### Validator tombstoned (double-signed)

**Symptoms:**

```
injectived query slashing signing-info injvalcons1...
  tombstoned: true
  jailed_until: "9999-12-31T23:59:59Z"
```

**Cause:** The validator signed two different blocks or prevotes at the same height. Common triggers:

* Running two instances of the node with the same signing key
* Restoring an old `priv_validator_state.json` that caused re-signing at a previously signed height
* Rollback resetting signing state, followed by the node signing a conflicting block

**Resolution:**

There is no standard on-chain unjail for tombstoned validators. Resolution paths:

* A governance proposal or upgrade handler that explicitly un-tombstones affected validators
* Creating a new validator with a new operator key (loses existing delegations)

<Warning>
  **If you were tombstoned during a coordinated security upgrade**, reach out to the Injective team in the verified validators channels immediately. During coordinated upgrades, the team may include an un-tombstone handler in the upgrade binary to restore affected validators. Time sensitive, report as soon as possible so you can be included before the upgrade handler is finalized.
</Warning>

**Prevention:**

* Use TMKMS or Horcrux for hardware-enforced single-signer semantics
* Always back up `priv_validator_state.json` after stopping the node
* Never run two nodes with the same signing key simultaneously

***

### Recovering from a snapshot

**When to use:** When rollback fails, takes too long, or the node state is corrupted beyond repair (apphash mismatch).

**Procedure:**

1. Stop the node completely
2. Back up `~/.injectived/data/priv_validator_state.json`
3. Back up `~/.injectived/config/priv_validator_key.json` if not already backed up elsewhere
4. Download a snapshot from a trusted provider:
   * Injective Team may also share emergency snapshots before or during security upgrades
   * [Polkachu Injective Snapshots](https://polkachu.com/tendermint_snapshots/injective) (typically goleveldb)
   * Community validators may share emergency snapshots during incidents
5. Remove old data:
   ```bash theme={null}
   rm -rf ~/.injectived/data
   ```
6. Extract the snapshot into `~/.injectived/data/`
7. **Restore your backed-up `priv_validator_state.json` into `~/.injectived/data/`**
8. Verify `priv_validator_state.json` is present and correct
9. Start the node

#### Why you must keep your own `priv_validator_state.json`

Snapshots contain blockchain data (blocks, application state) but never include your validator's signing state. The signing state tracks the last height, round, and step your validator signed at. If you lose it or replace it with a blank one, your node doesn't know what it already signed and may sign a conflicting block -- causing double-signing and permanent tombstoning.

Here's a practical example showing why the signing state must always be ahead of (or equal to) the snapshot height.

**Setup:** A coordinated chain upgrade with halt-height at block 125.

```
Block:  100  105  110  115  120  125  126
         |    |    |    |    |    |    |
         |    |    |    |    |    |    Chain resumes with new binary
         |    |    |    |    |    Chain halts here (upgrade height)
         |    |    |    |    |
         |    |    |    Snapshot taken at block 115
         |    |    |
         Your node has been signing every block...
```

At the halt, your `priv_validator_state.json` reads:

```json theme={null}
{
  "height": "126",
  "round": 42,
  "step": 3,
  "signature": "...",
  "signbytes": "..."
}
```

**Why does it say height 126 if the halt height was 125?** The `--halt-height` flag prevents block 126 from being *committed* (finalized), but the CometBFT consensus engine still enters height 126 and begins its rounds -- proposing, prevoting, and precommitting -- before the application layer refuses to process the block and the node shuts down. The high round number (42) reflects the network churning through rounds as validators gradually stop and consensus can't be reached. This is normal behavior: your validator cast real votes at height 126 even though that block was never finalized.

This means your validator has already cast votes at height 126, round 42. The available snapshot is from height 115.

**What happens if you restore the snapshot WITHOUT your signing state:**

The snapshot comes with either no `priv_validator_state.json` or a blank one (height 0). If you start the node this way:

1. Your node loads blockchain data from height 115
2. It catches up to height 126 by replaying blocks 116-125
3. At height 126, it enters consensus and starts signing from round 0
4. But the network already has your votes from round 42 at this height
5. Your node signs a different block proposal at height 126 (because it replayed with the new binary and may produce a different result)
6. The network detects two conflicting signatures from your validator key
7. **Your validator is double-signed and tombstoned. Permanently.**

**The correct procedure:**

1. Stop the node
2. Back up your `priv_validator_state.json` (it says height 126, round 42)
3. Delete your data directory
4. Extract the snapshot (blockchain data from height 115)
5. **Copy your backed-up `priv_validator_state.json` back into the data directory**
6. Start the node

Now when the node starts:

1. It loads blockchain data from height 115
2. It catches up to height 126 by replaying blocks 116-125
3. At height 126, it reads the signing state: "I already signed up to round 42"
4. It skips rounds 0-42 and waits for round 43 before signing anything new
5. No conflicting votes. Your validator is safe.

<Warning>
  **Key rule:** The signing state must always reflect the highest point your validator has ever signed at. It is safe for the signing state to be ahead of the snapshot (the node will catch up). It is never safe for the signing state to be behind what the validator actually signed on the network.
</Warning>

| Scenario                    | Signing state height | Snapshot height | Safe?                                                                    |
| --------------------------- | -------------------- | --------------- | ------------------------------------------------------------------------ |
| Normal recovery             | 126                  | 115             | Yes -- node replays 116-125, skips already-signed rounds at 126          |
| Fresh snapshot, kept state  | 126                  | 120             | Yes -- same logic, fewer blocks to replay                                |
| Snapshot without state      | 0                    | 115             | **No** -- will re-sign at heights it already signed, causing double-sign |
| Stale backup (taken at 110) | 110                  | 115             | **No** -- will re-sign at heights 111-126                                |

**Database backend compatibility:** Snapshots are backend specific. A goleveldb snapshot will not work on a node configured for pebbledb, and vice versa. Check your configuration:

```bash theme={null}
grep db_backend ~/.injectived/config/config.toml
```

***

### Rollback hangs or fails

**Symptoms:** `injectived rollback` runs but never completes, or produces errors.

**Causes:**

* **PebbleDB backend:** Rollback with pebble has known issues and may hang indefinitely
* **Very large database:** Rollback time scales with database size
* **Corrupted WAL** (Write-Ahead Log)

**Solutions:**

* For PebbleDB nodes: recover from a snapshot instead
* For large nodes: if rollback hasn't completed in 30+ minutes, abort and use a snapshot
* Consider switching to goleveldb. PebbleDB is more space efficient but less tested for rollback

***

### Round regression errors during catchup

**Symptoms:**

```
ERR Failed signing vote err="error signing vote: round regression at height 181027006.
Got 108, last round 141"
```

**Cause:** The node is catching up to the current consensus round. This is normal and expected when restarting a node during an active consensus round that has advanced through many rounds.

**Solution:** No action needed. The node will catch up. To speed up round catch-up:

```bash theme={null}
# Flag
--unsafe-consensus-timeout-precommit-delta=1ms

# Or environment variable
INJECTIVED_UNSAFE_CONSENSUS_TIMEOUT_PRECOMMIT_DELTA=1ms
```

<Warning>
  **Remove this override once the chain stabilizes and the node is caught up.** Revert to the default (100ms) and restart.

  If left on during normal operation, the 1ms delta causes rounds to advance far faster than votes can propagate across the network. Your validator moves to the next round before other validators' precommits arrive, so its votes miss the actual commit round. The result is a steadily climbing missed-block counter, and if enough blocks are missed (typically 500 of the last 10,000), **the validator will be jailed for downtime**.
</Warning>

***

### AppHash mismatch

**Symptoms:**

```
panic: Tendermint state.AppHash does not match AppHash after replay
```

**Cause:** The application state diverged from the consensus state. Can result from an incomplete upgrade, a crash during block execution, or a corrupted database.

**Solution:**

1. Try rollback: `injectived rollback`
2. If rollback fails or the error persists: recover from a snapshot
3. Always restore `priv_validator_state.json` after recovery

***

### Sentry nodes stuck after upgrade

**Symptoms:** The validator node is upgraded and signing, but sentry nodes remain stuck at the old block height.

**Cause:** Sentry nodes also need the new binary to process blocks past the upgrade height.

**Solution:** Upgrade sentry nodes to the same binary version as the validator. Sentries do not have `priv_validator_state.json` (they don't sign), but they do require the correct binary.

***

### Chain stalled after upgrade (insufficient voting power)

**Symptoms:**

* The chain does not produce new blocks after the upgrade height
* Consensus round numbers climb continuously (round 50, 100, 200+)
* Nodes log prevotes and precommits but no block is committed
* `latest_block_height` remains stuck at the halt height

**Cause:** CometBFT requires 2/3+ of total voting power to be online and participating with the correct binary to finalize a block. During a coordinated upgrade, there is a window where validators are upgrading at different speeds. Until enough voting power is running the new binary, the network cannot reach consensus.

What happens during this window:

1. The chain halts at the upgrade height (ex: block 125)
2. Validators that have upgraded start participating in consensus at height 126
3. Each round, the network attempts to finalize block 126 but fails because less than 2/3 of voting power is voting
4. The round number increments and the cycle repeats, round 1, 2, 3, ... 50, ... 100+
5. This continues until enough validators complete the upgrade and come online

**This is normal during upgrades.** The chain is not broken, it is waiting for quorum.

**What to do:**

* **If you have already upgraded:** No action needed. Your node is participating and will automatically finalize the next block once quorum is reached. You will see round regression errors in the logs as your node catches up through rounds. This is expected (see [Round regression errors during catchup](#round-regression-errors-during-catchup)).
* **If you have not yet upgraded:** You are part of the missing voting power. Complete the upgrade as quickly as possible to help the chain resume.
* **Monitor progress:** Check how many validators are online and participating:
  ```bash theme={null}
  curl -s localhost:26657/consensus_state | jq '.result.round_state.votes'
  ```
* **Coordinate in validator channels:** During major upgrades, validators typically coordinate in the verified validators channels to track upgrade progress and voting power percentage.

<Note>
  The longer this window lasts, the higher the round numbers climb. Validators that come online later will need to catch up through all those rounds, which is why the `--unsafe-consensus-timeout-precommit-delta=1ms` flag exists but remember to [remove it after the chain stabilizes](#round-regression-errors-during-catchup).
</Note>

***

### Jailed for downtime (not tombstoned)

**Symptoms:**

* Validator shows as inactive or jailed
* `tombstoned: false` in signing info
* Missed blocks counter is high

**Cause:** The validator missed too many consecutive blocks (typically 500 of the last 10,000).

**Solution:**

1. Ensure the node is running and fully caught up to the latest block
2. Unjail:
   ```bash theme={null}
   injectived tx slashing unjail \
     --from=<key_name> \
     --chain-id=injective-1 \
     --gas=auto \
     --gas-adjustment=1.5
   ```
3. Verify the validator is active again:
   ```bash theme={null}
   injectived query staking validator <injvaloper_address>
   ```

***

## Coordinated Upgrade Checklist

### Before the upgrade height

* [ ] Confirm the target halt-height from the governance proposal or chain team
* [ ] Download and verify the new binary (check sha256 checksums)
* [ ] Set `--halt-height=<target_height>` in node config or CLI flags
* [ ] If using Cosmovisor: place the new binary in the correct upgrade directory
* [ ] Back up `priv_validator_state.json`
* [ ] Identify snapshot providers in case rollback is needed
* [ ] Monitor chain progression toward the halt height

### At the upgrade height

* [ ] Confirm the node has halted at the expected height
* [ ] Stop the node fully (verify the process is dead)
  ```bash theme={null}
  ps aux | grep injectived
  ```
* [ ] Back up `priv_validator_state.json`
* [ ] Install the new binary
* [ ] Verify the version:
  ```bash theme={null}
  injectived version
  ```

### After the upgrade

* [ ] Start the node with the new binary
* [ ] Monitor logs for consensus participation
* [ ] Confirm prevotes are correct (not voting nil on every round)
* [ ] Monitor voting power on an explorer ([Mintscan](https://www.mintscan.io/injective/validators))
* [ ] If seeing `wrong Block.Header.LastResultsHash`: stop, rollback 1 block, restore `priv_validator_state.json`, restart
* [ ] If using round-catchup overrides (`--unsafe-consensus-timeout-precommit-delta`): remove them once stable
* [ ] Verify the validator is signing:
  ```bash theme={null}
  curl -s localhost:26657/consensus_state | jq '.result.round_state["height/round/step"]'
  ```

***

## Monitoring Reference

### Consensus and sync status

```bash theme={null}
# Current consensus state (height, round, step)
curl -s localhost:26657/consensus_state \
  | jq '.result.round_state["height/round/step"] | split("/") | {height: .[0], round: .[1], step: .[2]}'

# Latest block height
curl -s localhost:26657/status | jq '.result.sync_info.latest_block_height'

# Whether the node is still catching up
curl -s localhost:26657/status | jq '.result.sync_info.catching_up'
```

### Validator health

```bash theme={null}
# Signing info (missed blocks, jail status, tombstone status)
injectived query slashing signing-info $(injectived tendermint show-validator)

# Validator status and jail state
injectived query staking validator <injvaloper_address> --output json | jq '.status, .jailed'
```

### Node configuration

```bash theme={null}
# Current binary version
injectived version

# Current priv_validator_state (check height/round/step)
cat ~/.injectived/data/priv_validator_state.json | jq

# Database backend
grep db_backend ~/.injectived/config/config.toml
```

***

## Pruned vs Archival Nodes

|                          | Pruned Node                          | Archival Node                         |
| ------------------------ | ------------------------------------ | ------------------------------------- |
| **Rollback**             | Fast (minutes)                       | Slow (minutes to hours), may fail     |
| **Snapshot recovery**    | Fast (10-30 min)                     | Very slow (hours, snapshots 100+ GB)  |
| **Typical DB size**      | 10-50 GB                             | 500+ GB                               |
| **Recommended recovery** | Rollback first, snapshot as fallback | Snapshot preferred, rollback is risky |

***

## Additional Resources

* [Run an Injective Node](/infra/run-node)
* [Cosmovisor Setup](/infra/cosmovisor)
* [Upgrade a Node](/infra/upgrade-node)
* [Cosmos Validators FAQ](https://github.com/cosmos/cosmos/blob/master/VALIDATORS_FAQ.md)
* [CometBFT Running in Production](https://docs.tendermint.com/v0.34/tendermint-core/running-in-production.html)

***

## Snapshot Resources

### Injective Provided Snapshots

Injective maintains pruned mainnet snapshots in two regions. Check the status endpoint for the latest available height and download URL:

| Region         | Status                                                                                        | Download                                                              |
| -------------- | --------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- |
| **EU (OVH)**   | [status.json](http://injective-mainnet-snapshots.s3-website.gra.io.cloud.ovh.net/status.json) | `http://injective-mainnet-snapshots.s3-website.gra.io.cloud.ovh.net/` |
| **Asia (GCS)** | [status.json](https://storage.googleapis.com/injective-mainnet-snapshots-asia/status.json)    | `https://storage.googleapis.com/injective-mainnet-snapshots-asia/`    |

To download the latest snapshot, check the status endpoint for the current filename:

```bash theme={null}
# Check latest available snapshot (Asia example)
curl -s https://storage.googleapis.com/injective-mainnet-snapshots-asia/status.json | jq

# Download the snapshot
wget <url from status.json>

# Extract
lz4 -d <snapshot_file>.tar.lz4 | tar xf - -C ~/.injectived/data/
```

During coordinated security upgrades, the Injective team may also share emergency snapshots in the verified validators channels. For archival segment snapshots, see the [Archival Setup](/infra/archival-setup) page.

### Community Snapshot Providers

* [Polkachu Injective Snapshots](https://polkachu.com/tendermint_snapshots/injective) (typically goleveldb, pruned)
* [HighStakes Injective Snapshots](https://tools.highstakes.ch/snapshots/injective)
* Community validators may share emergency snapshots during incidents in the validator channels

<Warning>
  **Database backend compatibility:** Snapshots are backend specific. A goleveldb snapshot will not work on a node configured for pebbledb, and vice versa. Check your configuration before downloading:

  ```bash theme={null}
  grep db_backend ~/.injectived/config/config.toml
  ```

  **You must restore your backed up `priv_validator_state.json`** into the data directory before starting the node after a snapshot recovery. See [Recovering from a snapshot](#recovering-from-a-snapshot) above for the full procedure.
</Warning>