So, it turns out, if you try and run pip while in my schools network, it recognises a self signed ssl certificate and refuses to connect due to security concerns, forcing us to dopip install [package] --trusted-host pypi.org --trusted-host files.pythonhosted.org every single time we want to install a package, which is really annoying. So, why not fix this?
It turns out pip has a semi-universal config system, meaning that I can simply quickly shop up a script to permanently trust specific hosts. Turns out there are 3 ‘levels’ of config file (See here for the full docs), that load in this order, overwriting eachother:
- Enviroment Variable - Always overwrites everything else
- Global (All users on system) - Only overwrites User and Global
- User (just the current user) - Only overwrites Site
- Site (per enviroment) - Cannot overwrite
Now ideally, I’d write to the Enviroment Variable, to prevent a user accidently messing up the fix, however, I want this solution to work for all users, and writing to the Env Var seems to require admin permissions, so I’ll settle for the Global config file (/Library/Application Support/pip/pip.conf on MacOS).
So by simply doing
1 | mkdir -p /Library/Application\ Support/pip && printf "%s\n" "[global]" "trusted-host = pypi.python.org" " pypi.org" " files.pythonhosted.org" > /Library/Application\ Support/pip/pip.conf || echo "Failed to create conf file, please run with root permissions and try again" |
I can write the correct config options.
Lets break this down, first I do mkdir -p, making a new directory, with intermediate folders automatically being created if needed (thus the -p option). After that, if the folder creation runs successfully, print the following string:
1 | [global] |
And then output it to pip.conf (using the > redirection character), else if it fails (||), then print an error msg.
Ok, so now pip is fixed, but theres still 1 annoying issue, anaconda never likes to start up while connected to wifi, on this specific network. To figure this out, I simply opened wireshark before opening anaconda, and tried to figure out what was going wrong.
Turns out, it tries to call home to see if there are any updates avalible, however, if the server does not respond for any reason (say for example, an ssl certificate being self signed and causing an error), it has an insanely long timeout period.
So, let’s fix this!
The first thing I did, was try to see if there was any ssl documentation, turns out, there is! So, all I needed to do, was make a new certificate, append my schools self signed certificate, and I’d be home free!
Easier said than done it turns out…
First, I had to download the current, full ssl certificate, I opted to use curl for this:
curl https://curl.se/ca/cacert.pem -o $PWD/.conda.ssl.pem
After downloading it, I next had to fetch woodleighs certificate… once again, easier said than done!
To do this, I ended up using the fantastic openssl client’s feature of being able to see the certificates you send and recieve.
This is the line i came up with:
1 | openssl s_client -showcerts -servername "curl.haxx.se" -connect curl.haxx.se:443 | pcregrep -M -e "----.*(\n.*){19}" | pcregrep -M -v -e "---\nServer certificate" >> $b || echo "Failed to append to new certificate" |
Now, how does that work?
Getting into the openssl man page (which is STUPID long BTW), we can see the s_client option does the following:
1 | The s_client command implements a generic SSL/TLS client which connects to a remote host using SSL/TLS. |
So next, we add the -showcerts and -servername options, opting to connect to the curl page (ironic, I know), followed by the -connect option, specifying what port we would like to connect to in specific (443 for https).
I then pipe (|) this to a bunch of pcregrep statements, designed to filter down the last certificate, which, conveniantly, is the school one. I then append (>>) that to the certificate sheet (here, stored in the $b variable), otherwise, echo an error command.
I then copy the file to a new location for permanent storage, in particular the $HOME location, named with a dot . to make sure the file is non-visable.
I then run 2 sed commands
1 | sed -i '' "s~ssl_verify: true~$c~" $HOME/$a || sed -i '' "s~ssl_verify: True~$c~" $HOME/$a |
This repleaces the content ssl_verify: true (the default value) with the directory to my custom certificate, basically saying “use that certificate for ssl verification”. I also run another copy of this searching for a capital T in ‘true’ just be to sure.
After that, everything is patched and the follow should have been fixed:
- Anaconda taking forever to load when connected to the internet
- Pip Needing an overly complex install cmd (now it should just be
pip install package)
I am currently investigating the occasional nature of curl to fail, ironically due to a self signed ssl certificate, although I may simply switch to wget instead to avoid this issue.
On top of that, some installations of anaconda seem to be in different places, without any rhyme or reason, and i’ll be investigating this further soon.
The full script:
1 |
|