OpenSSL and the Rails-Breaking Arch Upgrade

My development Rails server would't start after upgrading Arch today.

$ rails server
bin/rails:6: warning: already initialized constant APP_PATH
/home/john/.../bin/rails:6: warning: previous definition of APP_PATH was here
$ ruby -v
ruby 2.2.1p85 (2015-02-26 revision 49769) [i686-linux]
$ rails -v
Rails 4.2.5.1

I tried to run my tests:

$ rake
rake aborted!
LoadError: /usr/local/rvm/rubies/ruby-2.2.1/lib/ruby/2.2.0/i686-linux/openssl.so: undefined symbol: SSLv3_method - /usr/local/rvm/rubies/ruby-2.2.1/lib/ruby/2.2.0/i686-linux/openssl.so

Interesting, so I thought I'd bundle, just in case:

$ bundle
Error loading RubyGems plugin "/usr/local/rvm/gems/ruby-2.2.1@global/gems/executable-hooks-1.3.2/lib/rubygems_plugin.rb": /usr/local/rvm/rubies/ruby-2.2.1/lib/ruby/2.2.0/i686-linux/openssl.so: undefined symbol: SSLv3_method - /usr/local/rvm/rubies/ruby-2.2.1/lib/ruby/2.2.0/i686-linux/openssl.so (LoadError)

Could not load OpenSSL.
You must recompile Ruby with OpenSSL support or change the sources in your Gemfile
from 'https' to 'http'. Instructions for compiling with OpenSSL using RVM are
available at http://rvm.io/packages/openssl.

Even more interesting. So, it's an OpenSSL issue. What do I have?

$ openssl version
OpenSSL 1.0.2h  3 May 2016

Ok, so check the Pacman log:

$ grep openssl /var/log/pacman.log 
[2016-01-29 09:22] [PACMAN] installed openssl (1.0.2.e-1)
[2016-02-03 15:34] [ALPM] upgraded openssl (1.0.2.e-1 -> 1.0.2.f-1)
[2016-05-20 20:49] [ALPM] upgraded openssl (1.0.2.f-1 -> 1.0.2.h-1)

Let's check the Arch site for any news (I did before upgrading and found nothing, but let's check again...). No announcements since the Pacman 5.0.1 upgrade requirement on 2016-03-04.

So I haven't missed anything, or have I???

I searched the interwebs for "archlinux openssl.so: undefined symbol: SSLv3_method" to see what it threw up. Lo, and behold, I found this commit.

Let's try without SSLv3, too

Ok, so SSLv3 was disabled in the ArchLinux OpenSSL package. What to do?

Well, for a short term fix, downgrade SSL.

$ ls -l /var/cache/pacman/pkg/openssl*
-rw-r--r-- 1 root root 2609292 Jan 28 17:30 /var/cache/pacman/pkg/openssl-1.0.2.f-1-i686.pkg.tar.xz                                                                     
-rw-r--r-- 1 root root 2610288 May  3 17:17 /var/cache/pacman/pkg/openssl-1.0.2.h-1-i686.pkg.tar.xz

Ok, so downgrade to version 1.0.2.f-1 which was the last one that worked.

$ sudo pacman -U /var/cache/pacman/pkg/openssl-1.0.2.f-1-i686.pkg.tar.xz
$ openssl version
OpenSSL 1.0.2f  28 Jan 2016

And try Rails again...

$ rails server
=> Booting WEBrick
=> Rails 4.2.5.1 application starting in development on http://localhost:3000
=> Run `rails server -h` for more startup options
=> Ctrl-C to shutdown server
[2016-05-20 21:32:22] INFO  WEBrick 1.3.1
[2016-05-20 21:32:22] INFO  ruby 2.2.1 (2015-02-26) [i686-linux]
[2016-05-20 21:32:22] INFO  WEBrick::HTTPServer#start: pid=3951 port=3000

Bingo! But I still need to fix things up so they work with the latest OpenSSL package. How do we do that?

Well, as I use RVM, it's just a matter of reinstalling the Ruby instances:

$ rvm reinstall all

Of al the rubies that I had installed, the only one that built successfully was version 2.3.1. All of the others terminated with an error:

Error running '__rvm_make -j1',
showing last 15 lines of /usr/local/rvm/log/1463820181_ruby-2.2.1/make.log
ossl_ssl.c:143:27: error: ‘SSLv3_client_method’ undeclared here (not in a function)
     OSSL_SSL_METHOD_ENTRY(SSLv3_client),
                       ^
ossl_ssl.c:119:69: note: in definition of macro ‘OSSL_SSL_METHOD_ENTRY’
 #define OSSL_SSL_METHOD_ENTRY(name) { #name, (SSL_METHOD *(*)(void))name##_method }
                                                                     ^~~~
Makefile:293: recipe for target 'ossl_ssl.o' failed
make: *** [build-ext] Error 2
++ return 2
There has been an error while running make. Halting the installation.

The failing rubies were ruby-1.9.3-p194, ruby-2.0.0-p247 and ruby-2.2.1. They are now, along with their gemsets, gone.

$ curl https://github.com/ruby/ruby/commit/801e1fe46d83c856844ba18ae4751478c59af0d1.diff > openssl.patch
$ rvm install --patch ./openssl.patch 1.9.3-p194

Installing the rubies with this patch succeeded ad I have posted this information on Stack Overflow.

Oh, and the Gemsets weren't gone - they just weren't displayed by rvm list gemsets when the rubies upon which the depend weren't available.