Thursday, July 26, 2012

Building PyCrypto on Amazon EC2

I setup a new AMI Linux instance in the EC2 cloud today primarily for playing around with Python and possibly building some small web apps.  Shortly after firing up the instance, I tried to build and install PyCrypto and ran into some problems.  It was a bit of an adventure.  Here's how I got it working:


Step 1 - Install gcc/make

$ sudo yum install gcc
$ sudo yum install make

That was easy.

Step 2 - Install the GNU MP Arithmetic Library

$ wget ftp://ftp.gmplib.org/pub/gmp-5.0.5/gmp-5.0.5.tar.bz2
$ bunzip2 gmp-5.0.5.tar.bz2
$ cd gmp-5.0.5
$ sudo ./configure
$ sudo make
$ sudo make check
$ sudo make install
$ cd

Still not too bad.

Step 3 - Install MPIR

$ wget http://www.mpir.org/mpir-2.5.1.tar.bz2
$ bunzip mpir-2.5.1.tar.bz2
$ cd mpir-2.5.1
$ sudo ./configure
$ sudo make
$ sudo make check
$ sudo make install
$ cd

Everything worked up through here.

Intermission

There are two problems you'll run into if you try to build PyCrypto at this point: errors from missing header files and, once you resolve those, runtime errors complaining about an undefined symbol for rpl_malloc.  Let's fix that.

Step 4 - Install the Python development headers

$ sudo yum install python-devel

Easy fix.

Step 5 - Download PyCrypto

$ wget http://ftp.dlitz.net/pub/dlitz/crypto/pycrypto/pycrypto-2.6.tar.gz
$ tar xzvf pycrypto-2.6.tar.gz
$ cd pycrypto-2.6

Step 6 - Edit the configure script

$ [favorite editor] configure

Find the this section of the script (I added line numbers):

3865:   if test $ac_cv_func_malloc_0_nonnull = yes; then:
3866: 
3867:   $as_echo "#define HAVE MALLOC 1" >> confdefs.h
3868: 
3869:   else
3870:       $as_echo "define HAVE_MALLOC 0" >> confdefs.h
3871:
3872:   case " $LIBOBJS " in
3873:   *" malloc.$ac_objext "* ) ;;
3874:   *) LIBOBJS = "$LIBOBJS malloc.$ac_objext"
3875:  ;;
3876:  esac
3877:
3878;
3879:  $as_echo "define malloc rpl_malloc" >>confdefs.h
3880:
3881:  fi
 
Keep line 3867, but comment out the rest.  Save and exit.

The problem is that when you try to build PyCrypto, autotools thinks that we're using rpl_malloc instead of malloc.  I'm not sure how to fix this problem the right way, but this hack will get rid of the check and just have it use malloc.  If someone knows the right way to fix this, please say something in the comments and I'll update this post.

Step 7 - Build PyCrypto


$ python setup.py build
$ sudo python.setup.py install

Step 8 - Test PyCrypto

$ python
>>> from Crypto.Cipher import AES

If this works, you should be good to go.

If it fails, you did something wrong.  You can either rm -r pycrypto-2.6 and re-extract the archive or just rm -r the build folder then grep for anything that says "rpl_malloc" and comment it out.  Also edit config.status and change 'D["HAVE_MALLOC"]=" 0"' to say '=" 1"'.  This is what I did the first time, but when I decided to write this up I started over to try to get it right from the start.

Have fun.

11 comments:

  1. From https://github.com/jtriley/StarCluster/issues/138:

    ...I believe you can fix this by doing:

    $ export ac_cv_func_malloc_0_nonnull = yes

    and then forcing a reinstall of PyCrypto:

    $ easy_install -U PyCrypto

    ReplyDelete
  2. Today, I had to rebuild PyCrypto from scratch into a brand new EC2 - (config: Amazon Linux AMI 2013.09.2 - ami-ccf297fc (64-bit) / ami-def297ee (32-bit)) with nothing in it -- fresh install. I've done it about a year ago, however lost my notes : ( .

    So thank you very much for providing the steps above! Very handy.. However, on a truly fresh EC2 instance per above, here are some additional notes I had to do to do a full end2end clean/error free install of PyCrypto:

    STEP2: After step 2.2:
    $ tar -vxf gmp-5.0.5.tar
    $ sudo yum install m4
    I had to install m4 otherwise sudo ./configure gave me errors.

    STEP3: After step 3.2
    $ tar -vxf mpir-2.5.1.tar

    After STEP4, just run: export ac_cv_func_malloc_0_nonnull=yes as suggested by https://github.com/jtriley/StarCluster/issues/138:

    If you do: export ac_cv_func_malloc_0_nonnull=yes THEN you don't have to do any of STEP6 as suggested. This worked for me perfectly.

    Pretty minor stuff, the only show stopper for me was to install m4 at the right time per above.

    Thanks again this saved me on all those compile errors.

    Cheers

    ReplyDelete
  3. Thanks so much for providing these clarifications and corrections.

    ReplyDelete
  4. hi,
    i am having same problem. I follow what you have mentions above but below is the error

    >>> from Crypto.Cipher import AES
    Traceback (most recent call last):
    File "", line 1, in
    File "build/bdist.linux-x86_64/egg/Crypto/Cipher/AES.py", line 50, in

    File "build/bdist.linux-x86_64/egg/Crypto/Cipher/_AES.py", line 7, in

    File "build/bdist.linux-x86_64/egg/Crypto/Cipher/_AES.py", line 6, in __bootst
    rap__
    ImportError: /root/.python-eggs/pycrypto-2.6-py2.7-linux-x86_64.egg-tmp/Crypto/Cipher/_AES.so: undefined symbol: rpl_malloc

    additional note I am having python 2.7 install on non standard location and for that i am doing below stuff

    C_INCLUDE_PATH=/opt/python2.7/include
    export C_INCLUDE_PATH

    CPLUS_INCLUDE_PATH=/opt/python2.7/include
    export CPLUS_INCLUDE_PATH

    LIBRARY_PATH=/opt/python2.7/lib
    export LIBRARY_PATH

    before running setup.py

    can you please help me or mail me the solution on garade.shahu@gmail.com


    ReplyDelete
    Replies
    1. Did you read Jomell's comment above? Step 6 of my instructions was meant to avoid the problem you're having. Jomell shared a better way to handle it. Either should work.

      Quote:

      "After STEP4, just run: export ac_cv_func_malloc_0_nonnull=yes as suggested by https://github.com/jtriley/StarCluster/issues/138:

      If you do: export ac_cv_func_malloc_0_nonnull=yes THEN you don't have to do any of STEP6 as suggested. This worked for me perfectly."

      Delete
    2. I tried it on both way. with editing file and using export ac_cv_func_malloc_0_nonnull=yes still no success.
      when I export ac_cv_func_malloc_0_nonnull=yes still showing same issue.

      Is your Python installed on non standard location?? and is that python2.7 with non standard location?

      I can install Python2.6 on EC2. The problem is with Python2.7 with non standard location.

      Non Standard location means I am having it on /opt/python2.7/include.

      And Python2.6 is on normal location i.e. /usr/lib64/python2.6

      can you please try with Python2.7 on your EC2 with above installation path.

      We should solve this as our deployment is pending.

      Thanks in advance for your replay and help.



      Delete
    3. The EC2 instance that I had when I wrote this is long gone. I don't have an answer for you. Sorry.

      Delete
  5. no prob bro. thnks for your help on this. will workaround on this.

    one more thing you could answer me if possible.

    see the error i am geting. i realy dont understand when i am imprting ASE why it accessing heating below path

    ImportError: /root/.python-eggs/pycrypto-2.6-py2.7-linux-x86_64.egg-tmp/Crypto/Cipher/_AES.so: undefined symbol: rpl_malloc

    i.e. /root/.. though it install successfully and export rpl_malloc to null. it should hit lib.. right?? do you have any idea.

    I think its different issue with EC2 not pycrypto nor cross compiling.

    your help on this was realy appricated

    thanks buddy

    ReplyDelete
  6. This comment has been removed by a blog administrator.

    ReplyDelete
  7. This comment has been removed by a blog administrator.

    ReplyDelete

Understanding Scope in Go

As per my New Year's resolution, I've been learning to program in Go and reading  The Go Programming Language .   On page 141 of the...