Debugging pass-tomb on Qubes 3.2
The Problem
pass open
(do some stuff)
pass close
tomb [E] Tomb is busy, cannot umount!
The workaround (not a solution)
Aha. So I wrote here about pass. Specifically I wrote
"For whatever reason, it throws an error when closing,
I haven't figured out why, yet, but the tomb closes nevertheless and detaches.
If you found a way to verbose the output of pass close please let me know and send me an email ;-)"
So I found a way to verbose my output. It is really simple (duh..):
pass close -v
That gives me:
tomb . Closing tomb [.password] mounted on /home/user/.password-store
tomb . Closing tomb bind hook: /rw/home/user/.password-store
umount: /home/user/.password-store: not mounted
tomb [E] Tomb is busy, cannot umount!
Wait, why is there a tomb bind hook to /rw/home/user ? Qubes exposes the private read-write partition for each VM to /rw . So far, so good. But then it binds /home to /rw/home.
But what does that have to do with the error? Looking in the source of pass-tomb you see the opening operation:
_tomb open "$TOMB_FILE" -k "$TOMB_KEY" -g "${PREFIX}/${path}"
so TOMB_FILE is obvious, the -k option is for the location of the TOMB_KEY. The -g tells tomb to use GPG instead of a symetric key and the last option is the most interesting. It teels tomb (which tells cryptsetup) where to mount. Omiting it leads to a mounted tomb filesystem at /media/.password. Umounting with
tomb close
yields no error! So where is the difference?
Quite easy: Mounting at /home/user/.password-store mounts inside a mount (a bound mount to be exact) which leads to a double entry in mount:
mount
[...]
/dev/xvdb on /rw type ext4 (rw,relatime,discard,data=ordered)
/dev/xvdb on /home type ext4 (rw,relatime,discard,data=ordered)
/dev/xvdb on /var/spool/cron type ext4 (rw,relatime,discard,data=ordered)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=31284k,nr_inodes=39100,mode=700,uid=1000,gid=1000)
/dev/mapper/tomb..password.1500372740.loop10 on /home/user/.password-store type ext4 (rw,nodev,noatime,data=ordered)
/dev/mapper/tomb..password.1500372740.loop10 on /rw/home/user/.password-store type ext4 (rw,nodev,noatime,data=ordered)
as you can see, /dev/xvdb is mounted three times. This is because two of those are bound directories. Just looking at the output of mount (or /proc/mounts) will not help you. --bind is a mount option and is ommited in the final display.. Very helpful..
Looking at the source code of tomb, you'll see:
# check if there are binded dirs and close them
bind_tombs=(`list_tomb_binds $tombname $tombmount`)
for b in ${bind_tombs}; do
bind_mapper="${b[(ws:;:)1]}"
bind_mount="${b[(ws:;:)2]}"
_message "Closing tomb bind hook: ::1 hook::" $bind_mount
_sudo umount "`print - ${bind_mount}`" || {
[...]
done
_verbose "Performing umount of ::1 mount point::" $tombmount
_sudo umount ${tombmount}
[[ $? = 0 ]] || { _failure "Tomb is busy, cannot umount!" }
So you see, appreantly tomb thinks that we bound our tomb to another directory and tries to umount it. Unfortunatly, this directory is not a bound directory but appears twice because it is mounted within a bound directory (and therefore can be accessed via /home/user/.password-store and /rw/home/user/.password-store).
Sadly, the detection of bound directories in Linux is a pain. Tomb iterates through the list generated by mount and looks for /dev/mapper/tomb* mounted more than once. This works (normally), but not when it actually isn't a bound directory, but is mounted within one.
So the solution would be a more sophisticated method to determine, whether we bound our password tomb.
One would have to implement this check into tomb instead of a readline of mount output to determine a bound directory.
This is easier than it sounds. You could try:
findmnt --list -o TARGET,SOURCE| grep "\[/"
But this will only show bound directories, when they bind to a subdirectory of a mounted volume. It won't catch a bound directory to /home/user/.password-store.
Other options are basically not available or not reliable (see here for a discussion on stackexchange.)
So, what to do?
Well I opted for a simple solution. I just commented out the section about the umounting of bound directories in tomb:
# check if there are binded dirs and close them
################################################
#Commented that in QUbes 3.2 to avoid umounting the
#correct directory with this umount of the binded directory
# bind_tombs=(`list_tomb_binds $tombname $tombmount`)
# for b in ${bind_tombs}; do
# bind_mapper="${b[(ws:;:)1]}"
# bind_mount="${b[(ws:;:)2]}"
# _message "Closing tomb bind hook: ::1 hook::" $bind_mount
# _sudo umount "`print - ${bind_mount}`" || {
# [[ -n $SLAM ]] && {
# _success "Slamming tomb: killing all processes using this hook."
# slam_tomb "`print - ${bind_mount}`" || _failure "Cannot slam the bind hook ::1 hook::" $bind_mount
# umount "`print - ${bind_mount}`" || _failure "Cannot slam the bind hook ::1 hook::" $bind_mount
# } || {
# _failure "Tomb bind hook ::1 hook:: is busy, cannot close tomb." $bind_mount
# }
# }
# done
This way, tomb will only umount the original mount and the error disapered.
But what, if you actually --bind mounted a password store?
Well this happens:
sudo mount --bind /home/user/.password-store/ /mnt/test/
pass close -v
. pass Closing the password tomb /home/user/.password.tomb
. tomb . Closing tomb [.password] mounted on /home/user/.password-store
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. Device tomb..password.1500372740.loop10 is still in use.
. tomb [E] Error occurred in cryptsetup luksClose tomb..password.1500372740.loop10
[x] Error : Unable to close the password tomb.
Ok, tomb can't close, because something is still mounted at /dev/mapper/tomb... Let's just try again:
. pass Closing the password tomb /home/user/.password.tomb
. tomb . Closing tomb [.password] mounted on /mnt/test
. tomb (*) Tomb [.password] closed: your bones will rest in peace.
(*) Your password tomb has been closed.
. Your passwords remain present in /home/user/.password.tomb.
Ah, so now the remaining mount was umounted and everything works fine.
Conclusion
There doesn't seem to be a clear solution in detecting bound mounts in Linux, currently. So to have pass work correctly in my vault-VM (and close the /dev/mapper which holds my password-tree) I did a hack to the tomb script to not check for bound directories.
This is far from ideal, but the only workaround I came up with today.
If you have a better idea, please let me know (or even better, submit a patch to tomb).
Regards,
Florian
Tags: howto, linux, pass, security, tomb, qubes
How I backup in Qubes 3.2
The scenario
So I want to backup my AppVms in Qubes. Obviously it makes sense, to encrypt the resulting backup file. Not just so a third party will hae problems decoding my private digital life but also to ensure integraty. See also here
The catch: You should use a high entropy passphrase. But entering that on a regular basis didn't seem like it would work for me.
The setup
I use an external 1 TB HDD attached via usb. The HDD is attached as a block device to a special "backup" AppVM. backup is obviously not connected to a NetVM and there is nothing special in it.
How I use the backup Manager
First, I use the Qubes VM Manager to attach the external HDD as a block device to the backup VM.
In it I run:
sudo mount /dev/xvdi1 backup/
to mount the HDD in the backup folder (which obviously must be created first).
Then, from a dom0 console, I run
qvm-run --pass-io vault-nw 'cat /home/user/.backup_key'
So, wait, what?
qvm-run --pass-io
is used to copy file(content) between VMs. It can also be used to copy to dom0.
vault-nw
is the source VM. In this case, it is my password manager VM.
'cat /home/user/.backup_key'
This command is executed in the source VM (so in this example vault-nw). And the output is displayed in the dom0 Terminal. Within the vault-nw I exported my laptop backup key to a plain text file, called .backup_key.
The content is displayed in dom0 Terminal (which is considered ultimatly trusted) and can be copied through a simple right click.
In the Qubes VM Manager I shutdown every VM that I want to save, then go to System->Backup VMs. From there I choose all the VMs to backup. Clicking on Next will show the Dialog, where Qubes wants to know, where to backup to. Here comes the backup VM handy. Here I choose the backup VM as a target and
/home/user/backup/
as the backup folder. (You remember? The external HDD was mounted here within the backup VM)
Next, I copy/paste the backup key to the dialog and click next. The backup starts.
Pitfalls
So, the worst case happend and you need to restore your backup. But wait, what was that horrible long, high entropy passphrase again? Yeah, right, it is in my password-vault VM. But where is that? In my backup....
So you see, that sucks.
You should therefore have your passphrase somewhere seperate than within your backup.
I personally use pass and setup a git repository on my homeserver to sync to. But that is a story for another blog.
If you have questions or remarks, just drop me an email.
Regards,
Florian