Linux, Software and play

A blog about all kinds of things concerning software and Linux

Debugging pass-tomb on Qubes 3.2

July 18, 2017 — Florian Brandes

The Problem

pass open
(do some stuff)
pass close
tomb [E] Tomb is busy, cannot umount!

The workaround (not a solution)

Aha. So I wrote here about pass. Specifically I wrote

"For whatever reason, it throws an error when closing, 
I haven't figured out why, yet, but the tomb closes nevertheless and detaches. 
If you found a way to verbose the output of pass close please let me know and send me an email ;-)"

So I found a way to verbose my output. It is really simple (duh..):

pass close -v

That gives me:

tomb  .  Closing tomb [.password] mounted on /home/user/.password-store
tomb  .  Closing tomb bind hook: /rw/home/user/.password-store
umount: /home/user/.password-store: not mounted
tomb [E] Tomb is busy, cannot umount!

Wait, why is there a tomb bind hook to /rw/home/user ? Qubes exposes the private read-write partition for each VM to /rw . So far, so good. But then it binds /home to /rw/home.

But what does that have to do with the error? Looking in the source of pass-tomb you see the opening operation:

_tomb open "$TOMB_FILE" -k "$TOMB_KEY" -g "${PREFIX}/${path}"

so TOMB_FILE is obvious, the -k option is for the location of the TOMB_KEY. The -g tells tomb to use GPG instead of a symetric key and the last option is the most interesting. It teels tomb (which tells cryptsetup) where to mount. Omiting it leads to a mounted tomb filesystem at /media/.password. Umounting with

tomb close

yields no error! So where is the difference?

Quite easy: Mounting at /home/user/.password-store mounts inside a mount (a bound mount to be exact) which leads to a double entry in mount:

mount
[...]
/dev/xvdb on /rw type ext4 (rw,relatime,discard,data=ordered)
/dev/xvdb on /home type ext4 (rw,relatime,discard,data=ordered)
/dev/xvdb on /var/spool/cron type ext4 (rw,relatime,discard,data=ordered)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=31284k,nr_inodes=39100,mode=700,uid=1000,gid=1000)
/dev/mapper/tomb..password.1500372740.loop10 on /home/user/.password-store type ext4 (rw,nodev,noatime,data=ordered)
/dev/mapper/tomb..password.1500372740.loop10 on /rw/home/user/.password-store type ext4 (rw,nodev,noatime,data=ordered)

as you can see, /dev/xvdb is mounted three times. This is because two of those are bound directories. Just looking at the output of mount (or /proc/mounts) will not help you. --bind is a mount option and is ommited in the final display.. Very helpful..

Looking at the source code of tomb, you'll see:

# check if there are binded dirs and close them
    bind_tombs=(`list_tomb_binds $tombname $tombmount`)
    for b in ${bind_tombs}; do
        bind_mapper="${b[(ws:;:)1]}"
        bind_mount="${b[(ws:;:)2]}"
        _message "Closing tomb bind hook: ::1 hook::" $bind_mount
        _sudo umount "`print - ${bind_mount}`" || {
      [...]
    done
    _verbose "Performing umount of ::1 mount point::" $tombmount
    _sudo umount ${tombmount}
    [[ $? = 0 ]] || { _failure "Tomb is busy, cannot umount!" }

So you see, appreantly tomb thinks that we bound our tomb to another directory and tries to umount it. Unfortunatly, this directory is not a bound directory but appears twice because it is mounted within a bound directory (and therefore can be accessed via /home/user/.password-store and /rw/home/user/.password-store).

Sadly, the detection of bound directories in Linux is a pain. Tomb iterates through the list generated by mount and looks for /dev/mapper/tomb* mounted more than once. This works (normally), but not when it actually isn't a bound directory, but is mounted within one.

So the solution would be a more sophisticated method to determine, whether we bound our password tomb.

One would have to implement this check into tomb instead of a readline of mount output to determine a bound directory.

This is easier than it sounds. You could try:

findmnt --list -o TARGET,SOURCE| grep "\[/"

But this will only show bound directories, when they bind to a subdirectory of a mounted volume. It won't catch a bound directory to /home/user/.password-store.

Other options are basically not available or not reliable (see here for a discussion on stackexchange.)

So, what to do?

Well I opted for a simple solution. I just commented out the section about the umounting of bound directories in tomb:

    # check if there are binded dirs and close them
    ################################################
    #Commented that in QUbes 3.2 to avoid umounting the
    #correct directory with this umount of the binded directory

#        bind_tombs=(`list_tomb_binds $tombname $tombmount`)
#        for b in ${bind_tombs}; do
#            bind_mapper="${b[(ws:;:)1]}"
#            bind_mount="${b[(ws:;:)2]}"
#            _message "Closing tomb bind hook: ::1 hook::" $bind_mount
#            _sudo umount "`print - ${bind_mount}`" || {
#                [[ -n $SLAM ]] && {
#                    _success "Slamming tomb: killing all processes using this hook."
#                    slam_tomb "`print - ${bind_mount}`" || _failure "Cannot slam the bind hook ::1 hook::" $bind_mount
#                    umount "`print - ${bind_mount}`" || _failure "Cannot slam the bind hook ::1 hook::" $bind_mount
#                } || {
#                    _failure "Tomb bind hook ::1 hook:: is busy, cannot close tomb." $bind_mount
#                }
#            }
#        done

This way, tomb will only umount the original mount and the error disapered.

But what, if you actually --bind mounted a password store?

Well this happens:

sudo mount --bind /home/user/.password-store/ /mnt/test/
pass close -v
 .  pass Closing the password tomb /home/user/.password.tomb
  .  tomb  .  Closing tomb [.password] mounted on /home/user/.password-store
  .  device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
  .  device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
  .  device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
  .  device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
  .  device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
  .  device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
  .  device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
  .  device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
  .  device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
  .  device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
  .  device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
  .  device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
  .  device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
  .  device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
  .  device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
  .  device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
  .  device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
  .  device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
  .  device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
  .  device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
  .  device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
  .  device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
  .  device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
  .  device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
  .  device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
  .  Device tomb..password.1500372740.loop10 is still in use.
  .  tomb [E] Error occurred in cryptsetup luksClose tomb..password.1500372740.loop10
 [x] Error : Unable to close the password tomb.

Ok, tomb can't close, because something is still mounted at /dev/mapper/tomb... Let's just try again:

.  pass Closing the password tomb /home/user/.password.tomb
  .  tomb  .  Closing tomb [.password] mounted on /mnt/test
  .  tomb (*) Tomb [.password] closed: your bones will rest in peace.
 (*) Your password tomb has been closed.
  .  Your passwords remain present in /home/user/.password.tomb.

Ah, so now the remaining mount was umounted and everything works fine.

Conclusion

There doesn't seem to be a clear solution in detecting bound mounts in Linux, currently. So to have pass work correctly in my vault-VM (and close the /dev/mapper which holds my password-tree) I did a hack to the tomb script to not check for bound directories.

This is far from ideal, but the only workaround I came up with today.

If you have a better idea, please let me know (or even better, submit a patch to tomb).

Regards,

Florian

Tags: howto, linux, pass, security, tomb, qubes