Repository: restic/restic

Issue #4832: Getting "Fatal: Fatal: config cannot be loaded: ciphertext verification failed"
Created: 2024-05-29T13:39:46Z
Output of restic version
This is a version with with I get the errors still:
restic 0.16.4 compiled with go1.21.6 on linux/amd64
This was the version I was running prune (while B2 hit the capability cab), that started the problem:
restic 0.13.1 compiled with go1.18 on linux/amd64
What backend/service did you use to store the repository?
Regular B2.
Problem description / Steps to reproduce
Not entirely sure.
I was running a prune of a big backup (about 40TB of size), just during the prune, the B2 backend hit the download limits and stopped providing data.  Attached a log while this happened:

After this probably a faulty initialization of a snapshot, see another screenshot below.
After this I am just getting  ciphertext verification failed for all operations I try.
Expected behavior
Any interupted operations should not corrupt the repository.
Actual behavior
Getting "Fatal: Fatal: config cannot be loaded: ciphertext verification failed"
Even if I try unlock or unlock --remove-all from a different host.
Do you have any idea what may have caused this?
The B2 hit the download capability limit, and didn't provide anymore downloads during the prune operation.
I was using resticker container, which does some weird initialization try if operations fail, so it could have tried to re-initialize, see below:

So a combination of these could have caused this. Ie first downloadcap reached, then resticker trying to-reinitialize without an ability to download anything.
Did restic help you today? Did it make you happy in any way?
Love restic anyway, I hope there is a way to recover.

Duplicate issue #4848: Restic repository corrupted with error: ciphertext verification failed 
Created: 2024-06-04T17:50:03Z
Output of restic version
restic 0.16.4
What backend/service did you use to store the repository?
We are using GCP bucket as backend to create backup for below services:
Mongodb, postgresql, zookeeper and Schema Registry dump
Problem description / Steps to reproduce
We are facing issue with restic backup wherein 2 of our repositories have been corrupted leading us to not access  backup snapshots and also we are not able to access existing snapshots.
This is not reproducible and we ourselves are facing this issue with 2 of our backup jobs mongodb backup and schema registry backup.
We are using gcs bucket as our repository and running restic as a kubernetes cronjobs.
Exception:
open repository
lock repository
load index files
ciphertext verification failed
github.com/restic/restic/internal/crypto.init
	/restic/internal/crypto/crypto.go:30
runtime.doInit1
	/usr/local/go/src/runtime/proc.go:6740
runtime.doInit
	/usr/local/go/src/runtime/proc.go:6707
runtime.main
	/usr/local/go/src/runtime/proc.go:249
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1650

Steps tried to recover:

restic snapshots ls -r 
Ignoring "<id>": failed to load snapshot <id>: ciphertext verification failed
Ignoring "<id>": failed to load snapshot <id>: ciphertext verification failed
Ignoring "<id>": failed to load snapshot <id>: ciphertext verification failed
Ignoring "<id>": failed to load snapshot <id>: ciphertext verification failed
Ignoring "<id>": failed to load snapshot <id>: ciphertext verification failed
Ignoring "<id>": failed to load snapshot <id>: ciphertext verification failed
Ignoring "<id>": failed to load snapshot <id>: ciphertext verification failed



restic unlock --remove-all -r gs:<gcs-bucket-name>:/<foldername>/schema-registry 
enter password for repository:
repository 7400edb4 opened (version 2, compression level auto)

╰─ restic check --read-data -r gs:<gcs-bucket-name>:/<foldername>/schema-registry  
using temporary cache in /var/folders/l2/<id>/T/restic-check-cache-<>
enter password for repository:
repository <repo> opened (version 2, compression level auto)
created new cache in /var/folders/l2/<indexId>/T/restic-check-cach<id>
create exclusive lock for repository
load indexes
[0:44] 100.00%  463 / 463 index files loaded
error: error loading index <indexId>: ciphertext verification failed
error: error loading index <indexId>: ciphertext verification failed
error: error loading index <indexId>: ciphertext verification failed
....
....
Fatal: LoadIndex returned errors


Tried:
restic repair index --read-all-packs -r gs:<gcs-bucket-name>:/<foldername>/schema-registry 

Command worked but still was not able to get restic snapshots list or push a new snapshot

Documentation and Issues Checked:
https://restic.readthedocs.io/en/latest/077_troubleshooting.html#find-out-what-is-damaged
https://forum.restic.net/t/help-with-fatal-load-index-invalid-data-returned/4147/2
#4168
Expected behavior
Restic should not get corrupted and we should be able to access existing snapshots and take new snapshots
Actual behavior
Currently restic is blocked and we are not able to take backup
Do you have any idea what may have caused this?
Went through docs and older issues not able to investigate the issue as the same thing work for other backups
Please help and we are not able to build confidence into using this in production

Closing comment (2024-06-05T20:32:19Z): When were the config file and any file in the keys folder of the repository last modified? Were there any errors right before the errors started to show up? Did the automation for some reason call the init command of restic?
The errors look like the used key file does not match the key used to encrypt the repository content. That is somewhat similar to #4832 , but the GS backend is not affected by the same bug. So this is likely a separate problem.