Debugging pass-tomb on Qubes 3.2
The Problem
pass open
(do some stuff)
pass close
tomb [E] Tomb is busy, cannot umount!
The workaround (not a solution)
Aha. So I wrote here about pass. Specifically I wrote
"For whatever reason, it throws an error when closing,
I haven't figured out why, yet, but the tomb closes nevertheless and detaches.
If you found a way to verbose the output of pass close please let me know and send me an email ;-)"
So I found a way to verbose my output. It is really simple (duh..):
pass close -v
That gives me:
tomb . Closing tomb [.password] mounted on /home/user/.password-store
tomb . Closing tomb bind hook: /rw/home/user/.password-store
umount: /home/user/.password-store: not mounted
tomb [E] Tomb is busy, cannot umount!
Wait, why is there a tomb bind hook to /rw/home/user ? Qubes exposes the private read-write partition for each VM to /rw . So far, so good. But then it binds /home to /rw/home.
But what does that have to do with the error? Looking in the source of pass-tomb you see the opening operation:
_tomb open "$TOMB_FILE" -k "$TOMB_KEY" -g "${PREFIX}/${path}"
so TOMB_FILE is obvious, the -k option is for the location of the TOMB_KEY. The -g tells tomb to use GPG instead of a symetric key and the last option is the most interesting. It teels tomb (which tells cryptsetup) where to mount. Omiting it leads to a mounted tomb filesystem at /media/.password. Umounting with
tomb close
yields no error! So where is the difference?
Quite easy: Mounting at /home/user/.password-store mounts inside a mount (a bound mount to be exact) which leads to a double entry in mount:
mount
[...]
/dev/xvdb on /rw type ext4 (rw,relatime,discard,data=ordered)
/dev/xvdb on /home type ext4 (rw,relatime,discard,data=ordered)
/dev/xvdb on /var/spool/cron type ext4 (rw,relatime,discard,data=ordered)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=31284k,nr_inodes=39100,mode=700,uid=1000,gid=1000)
/dev/mapper/tomb..password.1500372740.loop10 on /home/user/.password-store type ext4 (rw,nodev,noatime,data=ordered)
/dev/mapper/tomb..password.1500372740.loop10 on /rw/home/user/.password-store type ext4 (rw,nodev,noatime,data=ordered)
as you can see, /dev/xvdb is mounted three times. This is because two of those are bound directories. Just looking at the output of mount (or /proc/mounts) will not help you. --bind is a mount option and is ommited in the final display.. Very helpful..
Looking at the source code of tomb, you'll see:
# check if there are binded dirs and close them
bind_tombs=(`list_tomb_binds $tombname $tombmount`)
for b in ${bind_tombs}; do
bind_mapper="${b[(ws:;:)1]}"
bind_mount="${b[(ws:;:)2]}"
_message "Closing tomb bind hook: ::1 hook::" $bind_mount
_sudo umount "`print - ${bind_mount}`" || {
[...]
done
_verbose "Performing umount of ::1 mount point::" $tombmount
_sudo umount ${tombmount}
[[ $? = 0 ]] || { _failure "Tomb is busy, cannot umount!" }
So you see, appreantly tomb thinks that we bound our tomb to another directory and tries to umount it. Unfortunatly, this directory is not a bound directory but appears twice because it is mounted within a bound directory (and therefore can be accessed via /home/user/.password-store and /rw/home/user/.password-store).
Sadly, the detection of bound directories in Linux is a pain. Tomb iterates through the list generated by mount and looks for /dev/mapper/tomb* mounted more than once. This works (normally), but not when it actually isn't a bound directory, but is mounted within one.
So the solution would be a more sophisticated method to determine, whether we bound our password tomb.
One would have to implement this check into tomb instead of a readline of mount output to determine a bound directory.
This is easier than it sounds. You could try:
findmnt --list -o TARGET,SOURCE| grep "\[/"
But this will only show bound directories, when they bind to a subdirectory of a mounted volume. It won't catch a bound directory to /home/user/.password-store.
Other options are basically not available or not reliable (see here for a discussion on stackexchange.)
So, what to do?
Well I opted for a simple solution. I just commented out the section about the umounting of bound directories in tomb:
# check if there are binded dirs and close them
################################################
#Commented that in QUbes 3.2 to avoid umounting the
#correct directory with this umount of the binded directory
# bind_tombs=(`list_tomb_binds $tombname $tombmount`)
# for b in ${bind_tombs}; do
# bind_mapper="${b[(ws:;:)1]}"
# bind_mount="${b[(ws:;:)2]}"
# _message "Closing tomb bind hook: ::1 hook::" $bind_mount
# _sudo umount "`print - ${bind_mount}`" || {
# [[ -n $SLAM ]] && {
# _success "Slamming tomb: killing all processes using this hook."
# slam_tomb "`print - ${bind_mount}`" || _failure "Cannot slam the bind hook ::1 hook::" $bind_mount
# umount "`print - ${bind_mount}`" || _failure "Cannot slam the bind hook ::1 hook::" $bind_mount
# } || {
# _failure "Tomb bind hook ::1 hook:: is busy, cannot close tomb." $bind_mount
# }
# }
# done
This way, tomb will only umount the original mount and the error disapered.
But what, if you actually --bind mounted a password store?
Well this happens:
sudo mount --bind /home/user/.password-store/ /mnt/test/
pass close -v
. pass Closing the password tomb /home/user/.password.tomb
. tomb . Closing tomb [.password] mounted on /home/user/.password-store
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. Device tomb..password.1500372740.loop10 is still in use.
. tomb [E] Error occurred in cryptsetup luksClose tomb..password.1500372740.loop10
[x] Error : Unable to close the password tomb.
Ok, tomb can't close, because something is still mounted at /dev/mapper/tomb... Let's just try again:
. pass Closing the password tomb /home/user/.password.tomb
. tomb . Closing tomb [.password] mounted on /mnt/test
. tomb (*) Tomb [.password] closed: your bones will rest in peace.
(*) Your password tomb has been closed.
. Your passwords remain present in /home/user/.password.tomb.
Ah, so now the remaining mount was umounted and everything works fine.
Conclusion
There doesn't seem to be a clear solution in detecting bound mounts in Linux, currently. So to have pass work correctly in my vault-VM (and close the /dev/mapper which holds my password-tree) I did a hack to the tomb script to not check for bound directories.
This is far from ideal, but the only workaround I came up with today.
If you have a better idea, please let me know (or even better, submit a patch to tomb).
Regards,
Florian