Debugging pass-tomb on Qubes 3.2
The Problem
pass open
(do some stuff)
pass close
tomb [E] Tomb is busy, cannot umount!
The workaround (not a solution)
Aha. So I wrote here about pass. Specifically I wrote
"For whatever reason, it throws an error when closing,
I haven't figured out why, yet, but the tomb closes nevertheless and detaches.
If you found a way to verbose the output of pass close please let me know and send me an email ;-)"
So I found a way to verbose my output. It is really simple (duh..):
pass close -v
That gives me:
tomb . Closing tomb [.password] mounted on /home/user/.password-store
tomb . Closing tomb bind hook: /rw/home/user/.password-store
umount: /home/user/.password-store: not mounted
tomb [E] Tomb is busy, cannot umount!
Wait, why is there a tomb bind hook to /rw/home/user ? Qubes exposes the private read-write partition for each VM to /rw . So far, so good. But then it binds /home to /rw/home.
But what does that have to do with the error? Looking in the source of pass-tomb you see the opening operation:
_tomb open "$TOMB_FILE" -k "$TOMB_KEY" -g "${PREFIX}/${path}"
so TOMB_FILE is obvious, the -k option is for the location of the TOMB_KEY. The -g tells tomb to use GPG instead of a symetric key and the last option is the most interesting. It teels tomb (which tells cryptsetup) where to mount. Omiting it leads to a mounted tomb filesystem at /media/.password. Umounting with
tomb close
yields no error! So where is the difference?
Quite easy: Mounting at /home/user/.password-store mounts inside a mount (a bound mount to be exact) which leads to a double entry in mount:
mount
[...]
/dev/xvdb on /rw type ext4 (rw,relatime,discard,data=ordered)
/dev/xvdb on /home type ext4 (rw,relatime,discard,data=ordered)
/dev/xvdb on /var/spool/cron type ext4 (rw,relatime,discard,data=ordered)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=31284k,nr_inodes=39100,mode=700,uid=1000,gid=1000)
/dev/mapper/tomb..password.1500372740.loop10 on /home/user/.password-store type ext4 (rw,nodev,noatime,data=ordered)
/dev/mapper/tomb..password.1500372740.loop10 on /rw/home/user/.password-store type ext4 (rw,nodev,noatime,data=ordered)
as you can see, /dev/xvdb is mounted three times. This is because two of those are bound directories. Just looking at the output of mount (or /proc/mounts) will not help you. --bind is a mount option and is ommited in the final display.. Very helpful..
Looking at the source code of tomb, you'll see:
# check if there are binded dirs and close them
bind_tombs=(`list_tomb_binds $tombname $tombmount`)
for b in ${bind_tombs}; do
bind_mapper="${b[(ws:;:)1]}"
bind_mount="${b[(ws:;:)2]}"
_message "Closing tomb bind hook: ::1 hook::" $bind_mount
_sudo umount "`print - ${bind_mount}`" || {
[...]
done
_verbose "Performing umount of ::1 mount point::" $tombmount
_sudo umount ${tombmount}
[[ $? = 0 ]] || { _failure "Tomb is busy, cannot umount!" }
So you see, appreantly tomb thinks that we bound our tomb to another directory and tries to umount it. Unfortunatly, this directory is not a bound directory but appears twice because it is mounted within a bound directory (and therefore can be accessed via /home/user/.password-store and /rw/home/user/.password-store).
Sadly, the detection of bound directories in Linux is a pain. Tomb iterates through the list generated by mount and looks for /dev/mapper/tomb* mounted more than once. This works (normally), but not when it actually isn't a bound directory, but is mounted within one.
So the solution would be a more sophisticated method to determine, whether we bound our password tomb.
One would have to implement this check into tomb instead of a readline of mount output to determine a bound directory.
This is easier than it sounds. You could try:
findmnt --list -o TARGET,SOURCE| grep "\[/"
But this will only show bound directories, when they bind to a subdirectory of a mounted volume. It won't catch a bound directory to /home/user/.password-store.
Other options are basically not available or not reliable (see here for a discussion on stackexchange.)
So, what to do?
Well I opted for a simple solution. I just commented out the section about the umounting of bound directories in tomb:
# check if there are binded dirs and close them
################################################
#Commented that in QUbes 3.2 to avoid umounting the
#correct directory with this umount of the binded directory
# bind_tombs=(`list_tomb_binds $tombname $tombmount`)
# for b in ${bind_tombs}; do
# bind_mapper="${b[(ws:;:)1]}"
# bind_mount="${b[(ws:;:)2]}"
# _message "Closing tomb bind hook: ::1 hook::" $bind_mount
# _sudo umount "`print - ${bind_mount}`" || {
# [[ -n $SLAM ]] && {
# _success "Slamming tomb: killing all processes using this hook."
# slam_tomb "`print - ${bind_mount}`" || _failure "Cannot slam the bind hook ::1 hook::" $bind_mount
# umount "`print - ${bind_mount}`" || _failure "Cannot slam the bind hook ::1 hook::" $bind_mount
# } || {
# _failure "Tomb bind hook ::1 hook:: is busy, cannot close tomb." $bind_mount
# }
# }
# done
This way, tomb will only umount the original mount and the error disapered.
But what, if you actually --bind mounted a password store?
Well this happens:
sudo mount --bind /home/user/.password-store/ /mnt/test/
pass close -v
. pass Closing the password tomb /home/user/.password.tomb
. tomb . Closing tomb [.password] mounted on /home/user/.password-store
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. device-mapper: remove ioctl on tomb..password.1500372740.loop10 failed: Device or resource busy
. Device tomb..password.1500372740.loop10 is still in use.
. tomb [E] Error occurred in cryptsetup luksClose tomb..password.1500372740.loop10
[x] Error : Unable to close the password tomb.
Ok, tomb can't close, because something is still mounted at /dev/mapper/tomb... Let's just try again:
. pass Closing the password tomb /home/user/.password.tomb
. tomb . Closing tomb [.password] mounted on /mnt/test
. tomb (*) Tomb [.password] closed: your bones will rest in peace.
(*) Your password tomb has been closed.
. Your passwords remain present in /home/user/.password.tomb.
Ah, so now the remaining mount was umounted and everything works fine.
Conclusion
There doesn't seem to be a clear solution in detecting bound mounts in Linux, currently. So to have pass work correctly in my vault-VM (and close the /dev/mapper which holds my password-tree) I did a hack to the tomb script to not check for bound directories.
This is far from ideal, but the only workaround I came up with today.
If you have a better idea, please let me know (or even better, submit a patch to tomb).
Regards,
Florian
Tags: howto, linux, pass, security, tomb, qubes
How to manage passwords with pass
Background
Why a password manager?
Well, if you found this page and are interessted in pass, you must already have your reasons to look into password managers. For me, it's a basic concept: Use password only once. Don't (ever) reuse passwords or passphrases for other services. If one service gets compromised, you won't automatically have to worry about your other services. This makes remebering passwords a bitch, especially if you don't iterate through numbers of your favorite, easy-to-guess, passwords. Speaking of which, yes, there are tools out there, that can generate very good dictonaries based on a bit of social engenieering. So you really should use generated passwords.
Why pass?
Well, there is a ton of really good password-managers out there. I mean a lot. I used gorilla a long time, but at some point, I wanted to sync my passwords with my home computer, my laptop and my work computer. (Yes, arguably, keeping your password safe sync'd to your work computer bears its risks, but it all comes down to whether you trust your employer to not backdoor your work computer. I use FDE (full-disk-encryption) and my password safe is password protected and encrypted, too, but that won't save you, if your employer is determined to eavsdrop on you.)
Anyway, if I change a password on one system and maybe add another on my second system, I will run into problems using gorilla (gorilla uses a single, encrypted file).
So something new was needed.
First, I looked into online, cloud serving password managers. I ran across lastpass, which worked really well with all my systems. Unfortunatly, I was left in the lurch, because I didn't want to use the browser plugin for firefox, but rather a standalone app. Lastpass provides such a solution, even for Linux, but it has two drawbacks:
It doesn't work on a reasonably new Ubuntu Linux (see below for my ticket to lastpass)
"I am using Ubuntu 17.04 and pocket_x86 does not start, since it complains about the version ob libpng. libpng was updated to v1-6 but your application needs v1-2.
The same problem also applies to the oder version of ubuntu, since it also uses the newer version of libpng. The problem was also reported on your forum (see here: https://forums.lastpass.com/viewtopic.php?f=14&t=232835&p=780625&hilit=pocket_x64#p780625) where the exact same problem was described more than half a year ago!
Could you please update the dynamically linked application to also allow the new version of libpng? (or give me the source code and I'll update your application..) I do not use FIrefox (or Chrome or any other supported browser) and I do not use windows, so right now I'm stranded and can only use your online website to access my passwords."
It is closed source.
They replied that they forwarded it to the development branch, but I never heard back and they closed the ticket. I didn't try it since then.
You need to trust lastpass. I do have trust issues, but I was willing to give it a try. I stoped using lastpass since then, because I couldn't even debug the code myself.
So I decided I needed something else. Since I love my Terminal I got intrigued by the idea of pass. It has a built-in git support, which works really well and combined with pass-tomb provides a reasonably secure password store.
What makes pass different?
Well, pass uses GPG to encrypt/decrypt your passwords. It saves all your passwords in a directory tree. Each directory works like a folder in your previous password manager. The directory is stored at ~/.password-store . You can just enter
pass
to get an overview or, for example:
pass Email/yahoo
to get your email password for yahoo. To get a nice overview see here.
I didn't like that everyone with access to my (decrypted) HDD can see, where I signed up, so I use the pass-tomb extension, as well. It encrypts your whole .password-store folder into a tomb file with your GPG key.
How to set pass up
First I installed pass. It can be found in most package systems of mondern ditros, but I found it won't work (out of the box) with pass-tomb. So I opted to download it manually and install it manually:
You will need git and gpg for this to work
First, lets import the public signing key
gpg --fetch-keys http://www.zx2c4.com/keys/AB9942E6D4A4CFC3412620A749FC7012A5DE03AE.asc
Now lets download pass in a temporary directory (or wherever you like):
mkdir tmp
cd tmp
git clone https://git.zx2c4.com/password-store
cd password-store
To verify the integraty of the git branch, run the following command:
git tag -v 1.7.1
If version 1.7.1 isn't the curent one, run git tag to get an overview of all the tags.
It shoud read somewhere:
gpg: Good signature from "Jason A. Donenfeld <Jason@zx2c4.com>"
Now, you are ready to set up pass. Just run:
sudo make install
Thats it. Now you could already start using it. I opted to install the pass extension for tomb. Acording to the pass-tomb website:
A pass extension that helps to keep the whole tree of password encrypted inside a tomb.
So, first intsall the dependencies:
sudo yum install zsh sudo cryptsetup pinentry-curses
or
sudo apt-get install zsh sudo cryptsetup pinentry-curses
Next, install tomb:
wget https://files.dyne.org/tomb/Tomb-2.4.tar.gz
wget https://files.dyne.org/tomb/Tomb-2.4.tar.gz.asc
gpg --keyserver pool.sks-keyservers.net --recv 4ACB7D10
gpg --verify Tomb-2.4.tar.gz.asc Tomb-2.4.tar.gz
It shoud read somewhere:
gpg: Good signature from "Denis Roio (Jaromil) <jaromil@dyne.org>"
Next, untar, cd into the directory and install
tar xvfz Tomb-2.4.tar.gz
cd Tomb-2.4
sudo make install
So, to add it to tomb, do the following (in your tmp directory):
git clone https://github.com/roddhjav/pass-tomb/
cd pass-tomb
sudo make install
Thats it.
How does it work?
Pass, by itself, uses GPG to encrypt your password. That automatically means, you will need gpg. I assume, you already have gpg installed. If you haven't got a gpg private key, or want to create a seperate one, just for your passwords (what I did) you will need to generate one:
$ gpg --gen-key
gpg (GnuPG) 1.4.21; Copyright (C) 2015 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Please select what kind of key you want:
(1) RSA and RSA (default)
(2) DSA and Elgamal
(3) DSA (sign only)
(4) RSA (sign only)
Your selection? 1
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 4096
Requested keysize is 4096 bits
Please specify how long the key should be valid.
0 = key does not expire
<n> = key expires in n days
<n>w = key expires in n weeks
<n>m = key expires in n months
<n>y = key expires in n years
Key is valid for? (0) 0
Key does not expire at all
Is this correct? (y/N) y
You need a user ID to identify your key; the software constructs the user ID
from the Real Name, Comment and Email Address in this form:
"Heinrich Heine (Der Dichter) <heinrichh@duesseldorf.de>"
Real name: Your name
Email address: email@provider.com
Comment: Password key
You selected this USER-ID:
"Your name (Password key) <email@provider.com>"
Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O
You need a Passphrase to protect your secret key.
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
gpg: key ABCD1234 marked as ultimately trusted
public and secret key created and signed.
gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u
pub 4096R/ABCD1234 2017-07-17
Key fingerprint = ABCD EFGH 1234 5678 3C7D 8558 0B49 5647 CD6F 83A4
uid Your name (Password key) <email@provider.com>
sub 4096R/EFGH6789 2017-07-17
With that out of the way, you can initialize the password store. I do recommend using pass-tomb extension. So with pass-tomb, you first need to create a tomb:
pass tomb <gpg-id>
where gpg id is the ke id of your GPG key you generated earlier.
If you run into an error like this:
[x] Error : Unable to forge the password tomb.
you will need more information. Do the following:
rm .password.tomb
pass tomb -v <gpg-id>
Unfortunatly on my system, tomb detects a swap partion (inside an LVM voulme group which is encrypted) but won't continue, since in would pose a security risk, if the swap would be unencrypted. Tomb cannot distingues between that, so it complains:
. tomb . tomb lock /home/user/.password.tomb -k /home/user/.password.tomb.key
. tomb . An active swap partition is detected...
. tomb [W] This poses a security risk.
. tomb [W] You can deactivate all swap partitions using the command:
. tomb [W] swapoff -a
. tomb [W] [#163] I may not detect plain swaps on an encrypted volume.
. tomb [W] But if you want to proceed like this, use the -f (force) flag.
. tomb [E] Operation aborted.
[x] Error : Unable to forge the password tomb.
so, I turned off my swap partion for a lack of a better option (you can't pass the -f flag to tomb using pass-tomb extension and I was too lazy to hack pass-tomb..)
sudo swapoff -a
now, it should work.
You will need to enter your passphrase from your GPG key. Tomb initializes and is ready to work.
From now on, you will need to enter
pass open
before accessing your passwords. When open, you can work with it like described above. When you're done just enter
pass close
For whatever reason, it throws an error when closing, I haven't figured out why, yet, but the tomb closes nevertheless and detaches. If you found a way to verbose the output of pass close please let me know and send me an email ;-)
Syncing
So, to setup a git repository, I used my raspberry pi at home. I started a blank repository
git init --bare ~/.password.git
Next, on your work PC init a git repository:
pass open
pass git init
pass git remote add origin ssh://username@raspberry_ip/absolute/path/to/.password.git/
pass git push origin master
(Yes I use ssh to connect to my pi. You will therefore need to set it up. I assume you know how to do that. I reccommend adding your ssh pulic key to the authorized_keys file of your raspberry to avoid needing to enter the password everytime you snyc).
From your second PC, you just need to do the same, but replace the last line with:
pass git pull
Remember to first open your pass tomb, otherwise you will sync the wrong directory!
(pass tomb mounts the tomb over your ~/.password-store directory. So when you forget to open your tomb before, it will complain about not initilizing the git repository first and obviously there won't be any passwords, because pass uses the empty not-mounted ~/.password-store directory)
You can further automate the procedure with a git hook to automatically pull when opened and push when you change something. But, again, I was lazy and just do pass git pull and pass git push by hand.
Hope that helped,
Florian
Tags: linux, pass, security, tomb, sync, howto
How I backup in Qubes 3.2
The scenario
So I want to backup my AppVms in Qubes. Obviously it makes sense, to encrypt the resulting backup file. Not just so a third party will hae problems decoding my private digital life but also to ensure integraty. See also here
The catch: You should use a high entropy passphrase. But entering that on a regular basis didn't seem like it would work for me.
The setup
I use an external 1 TB HDD attached via usb. The HDD is attached as a block device to a special "backup" AppVM. backup is obviously not connected to a NetVM and there is nothing special in it.
How I use the backup Manager
First, I use the Qubes VM Manager to attach the external HDD as a block device to the backup VM.
In it I run:
sudo mount /dev/xvdi1 backup/
to mount the HDD in the backup folder (which obviously must be created first).
Then, from a dom0 console, I run
qvm-run --pass-io vault-nw 'cat /home/user/.backup_key'
So, wait, what?
qvm-run --pass-io
is used to copy file(content) between VMs. It can also be used to copy to dom0.
vault-nw
is the source VM. In this case, it is my password manager VM.
'cat /home/user/.backup_key'
This command is executed in the source VM (so in this example vault-nw). And the output is displayed in the dom0 Terminal. Within the vault-nw I exported my laptop backup key to a plain text file, called .backup_key.
The content is displayed in dom0 Terminal (which is considered ultimatly trusted) and can be copied through a simple right click.
In the Qubes VM Manager I shutdown every VM that I want to save, then go to System->Backup VMs. From there I choose all the VMs to backup. Clicking on Next will show the Dialog, where Qubes wants to know, where to backup to. Here comes the backup VM handy. Here I choose the backup VM as a target and
/home/user/backup/
as the backup folder. (You remember? The external HDD was mounted here within the backup VM)
Next, I copy/paste the backup key to the dialog and click next. The backup starts.
Pitfalls
So, the worst case happend and you need to restore your backup. But wait, what was that horrible long, high entropy passphrase again? Yeah, right, it is in my password-vault VM. But where is that? In my backup....
So you see, that sucks.
You should therefore have your passphrase somewhere seperate than within your backup.
I personally use pass and setup a git repository on my homeserver to sync to. But that is a story for another blog.
If you have questions or remarks, just drop me an email.
Regards,
Florian