Troubleshooting
Common errors, what they mean, and how to fix them.
Start with the database's Pulse tab: Health lists every problem, worst first, with Apply fix when Rowsafe can fix it (Apply fix).
archive_command is already set
archive_command is already set to "...": another archiver may be running; re-run with force to replace itAnother tool (WAL-G, Barman, an older pgBackRest) already archives WAL. Same for archive_library is set to "...".
- Find out what it is first. If it's your current backup, replacing it stops it.
- To let Rowsafe take over:
rowsafe apply app --force. Keep the old backups until Rowsafe has proven itself.
In Docker, a plan also refuses an archive_command that runs pgBackRest itself; a server install refuses a Docker spool command. --force doesn't override these. Only if archive_mode is off (otherwise see Rollback), reset it and plan again:
ALTER SYSTEM RESET archive_command; SELECT pg_reload_conf();Restart pending
The status says awaiting PostgreSQL restart, or rowsafe verify says archive_mode is still off. The settings need one restart, and Rowsafe never restarts on its own.
Click Restart PostgreSQL now on the database's page (or Apply fix on Health), now or at a time you pick. Rowsafe restarts, checks changes reach your bucket, and starts the first full backup.
After 24 hours the awaiting_restart alert reminds you.
Not enough disk for a drill
not enough disk for a drill: 8.1 GiB free in /var/lib/rowsafe/drills, need about 14.0 GiBProof restores a full copy, so it needs about 1.3 × the database size + 1 GiB free. Either:
- free or add disk on the filesystem holding
/var/lib/rowsafe/drills, or - set
ROWSAFE_DRILL_DIRin/etc/rowsafe/agent.envto a bigger filesystem (short absolute path, outside/home, owned bypostgres), then restart the agent, or - run Proof on another server: Proof → Where restore tests run.
Until Proof passes again, the database isn't protected.
Agent offline
You got agent_offline. PostgreSQL keeps archiving, but backups, Proof and monitoring have stopped. On the server:
sudo systemctl status rowsafe-agent
sudo journalctl -u rowsafe-agent -n 50| You see | Fix |
|---|---|
| The service is stopped or crash-looping | Check the journal. curl -fsSL https://rowsafe.sh | sudo sh runs the self-test and names what's wrong. |
repository not configured, missing: ... | Add the missing settings to /etc/rowsafe/agent.env, then run the installer again. |
not enrolled: set ROWSAFE_ENROLL_TOKEN | Run the install command again with --account and approve the server in your browser, or pass a token from rowsafe hosts enroll-token. |
the control plane rejected this agent's token: the host was removed from Rowsafe | Enroll it again with a new token, or uninstall the agent. |
| Connection errors | The server must reach https://api.rowsafe.sh on outbound port 443. Check firewall, proxy and DNS. |
In Docker: docker compose ps and docker compose logs rowsafe-agent.
When a backup step fails
The failed task says why in plain words, with a fix button under it when there is one. Open Activity and expand the task; Details shows pgBackRest's output with secrets removed (or rowsafe task ID).
Storage didn't accept the key
The key or secret is wrong, expired or deleted (403, InvalidAccessKeyId, SignatureDoesNotMatch). Nothing changed. Keys never leave your server, so fix them there; the setup tests them first:
curl -fsSL https://rowsafe.sh | sudo sh -s -- --setup-storageThen click Try again (or Test storage again).
Storage refused access
The key works but can't use the bucket (403 (Forbidden), AccessDenied). Give it read, write, list and delete on the bucket (Storage), then Try again. New key? Enter it with the storage setup above first.
Bucket not found or wrong region
NoSuchBucket(404): create the bucket, or fix the name with the storage setup above.PermanentRedirectorAuthorizationHeaderMalformed: wrong region; fix it the same way.
Then Try again.
Server can't reach the storage
DNS failed, the connection was refused, or the certificate isn't trusted (self-hosted storage like MinIO). Allow outbound HTTPS to the storage endpoint, or add the certificate authority with the storage setup above. Then Try again.
The backup folder already has files
backup directory and/or archive directory not emptyLeftover files from an earlier setup. Rowsafe won't mix them with new backups, and changes nothing.
Click Start fresh in a new folder (for example /rowsafe/app-20260926-044600). Rowsafe never deletes the old files; delete them yourself when done.
Backups from another database or key
backup and archive info files exist but do not match the databaseThe folder holds another database's backups, or ones encrypted with another passphrase (is or was the repo encrypted?). Nothing changed.
- Start fresh in a new folder, as above. Old backups stay.
- Same database, older PostgreSQL version? Use the backups already there keeps them.
- Have the old passphrase? Enter it with the storage setup above, then Try again.
PostgreSQL isn't handing over changes
WAL segment ... was not archived before the 120000ms timeout
archive_mode must be enabledUsually PostgreSQL still needs its restart, or the storage is unreachable or refuses the key (WAL archiving failing). Then click Test storage again.
Another backup job was running
unable to acquire lock on file ...: wait for it to finish, then Try again.
Disk full
No space left on device: free space (the Pulse tab shows what uses it), then Try again.
Other pgBackRest errors
Details has pgBackRest's message. Its exit code: 39 storage request failed, 40 folder not empty, 50 lock, 82 timeout. pgBackRest missing or too old? Run the Rowsafe installer again on the server.
WAL archiving failing
You got wal_archiving_failing. Restore to any second stops at the last archived change, and pg_wal grows until the disk fills. Fix it soon.
Common causes:
- Bucket keys rotated or revoked, or the key's IP allowlist no longer matches. Update
/etc/rowsafe/agent.env, restart the agent, then click Check again on Health (orrowsafe verify app). - Bucket unreachable: firewall, DNS or provider outage. Archiving catches up by itself.
- Full disk on the server.
- Docker: the agent container is stopped or can't reach the bucket, so WAL waits in the spool. Check
docker compose logs rowsafe-agent. Don't delete spooled files.
The alert resolves by itself once archiving works.
Stuck sessions, slots and cleanup
Idle-in-transaction sessions, blocking queries, inactive replication slots, tables needing vacuum, unused indexes: open the finding on Health and click Apply fix (or rowsafe fix app). Ending a session or removing an index or slot asks you to type the database name; removing an index sets a Mark first. Apply fix.
Rewind
Rewind checks everything before changing anything, and says why it can't go ahead.
Not enough free disk for a copy
not enough free disk for a copy: 8.1 GiB free in /var/lib/rowsafe/rewind, and a copy of 10.0 GiB needs about 14.0 GiBA copy needs about 1.3 × the database size + 1 GB free. Free disk, delete an old copy, or set ROWSAFE_REWIND_DIR in /etc/rowsafe/agent.env to a bigger filesystem (short absolute path, owned by postgres; restart the agent).
Not enough free disk for a rewind
A whole rewind keeps current data aside for undo, so it needs about the database size + 10% free next to the data directory. Free disk, click Delete the old data now to drop an earlier rewind's data, or restore a copy and bring back rows instead.
Rewind the whole database isn't available
The disabled button says why:
| It says | What to do |
|---|---|
| Rowsafe isn't allowed to stop PostgreSQL on this server | On the server: sudo rowsafe-allow restart (permissions). Installed before rowsafe-allow: curl -fsSL https://rowsafe.sh | sudo sh -s -- --allow-restart |
| The helper that restarts PostgreSQL is from an older Rowsafe | Run the install command again there. It keeps your answer and updates the helper. |
| PostgreSQL runs in Docker | Add the container control service with the data volume writable for the agent, or restore a copy and bring back rows. |
| This PostgreSQL is a read-only replica | Rewind the primary. |
| This database uses tablespaces | Not supported in place yet. Restore a copy and bring back rows. |
| The data directory is a symbolic link, or a mount point of its own | Not supported in place yet. Restore a copy and bring back rows. |
A table was skipped
- No primary key: rows can't be matched.
- Too big: over 5 GB or 20 million rows.
- Dropped since: not in production any more.
For all three: rewind the whole database, or read the copy yourself (Do it yourself).
Too many rows
too many rows: that is 12.4 million rows, over the limit of 10 million at a time; nothing was changedBring back fewer tables per run: up to 50 tables and a row limit per plan.
Rows that point to another table
120 of the rows for public.order_items point to rows of public.orders that aren't in production. Bring back public.orders too (in the same run); nothing was changedSelect both tables and run again. Parents come back first.
Conflicts
Some rows "were left out because another row now has the same unique value" (for example, the same email). Rowsafe won't overwrite the new row. Check them in the copy and decide by hand.
Other Rewind messages
| It says | What to do |
|---|---|
| another session held a lock on the table for more than 5s, so Rowsafe stopped; nothing was changed | Try again in a moment. |
| the column ... was added since and needs a value, which the copy doesn't have | Give the column a default, or bring the rows back by hand. |
| ... already has a copy ...; delete it first | One copy per database. Delete it or let it expire. |
| the copy no longer exists (it expired or was deleted) | Restore a new one. |
| the backup ... is no longer in the repository | Retention removed it. Pick a later moment. |
| if the time is after the last change that reached the backups, pick an earlier one | Pick a moment inside the recovery window. |
| another rewind is running on this server | Wait for it to finish. |
| only the latest rewind can be undone | Undo the latest one first. |
Other messages
Still stuck? Email [email protected] with rowsafe status output and the failing task's ID.