Pyrit setup

Pyrit is a tool used to pre-compute all possible SSID:PASSWORD combinations to generate a list of PMK tokens.

I ran this setup on a 'homebrew' cluster consisting of 4 mainboards stacked in height using threaded rods. Bottom 2 boards are the Datanodes, 3rd is the MySQL Server, while the upper-most is the actual GPGPU Server with slots for 4x Cuda/Stream acceleration cards.

• Pyrit cluster photo gallery
• Pyrit MySQL Cluster Batch video
• WPA Attack

Setup

This cluster is installed on a set of 32-bit Ubuntu Server Edition (9.10) database servers and 64-bit Ubuntu Desktop Edition for the ATI Stream GPU accelerator.

• ATI Stream GPU accelerator - i5 750 2.66 GHz, Asus P7P55 WS SC, 4GB memory, 2x Gigabyte GV-R585OC-1GD Radeon 5850 - 10.0.0.2
• MySQL NDB cluster master - P4 2.4 GHz, Supermicro P4SAA, 2 GB memory - 10.0.0.10
  • MySQL NDB data storage #1 - AMD64 3000+ 1.8 GHz, DFI NF4 SLI-DR, 4GB memory - 10.0.0.11
  • MySQL NDB data storage #2 - AMD64 3000+ 1.8 GHz, A8N-SLI Premium, 4GB memory - 10.0.0.12

All boards are connected through a 5-port Netgear gigabit desktop switch.

Pre-requisite

Pyrit version 0.3.0 and up depend on Scapy2 (package inspection) and SQLAlchemy 0.5.6 (database interface).

Scapy: On a Debian/Ubuntu bases distribution, get the package using aptitude. Backtrack 4 final include the Scapy2.

sudo aptitude update
sudo aptitude install python-scapy scapy2

Install

Main App

Download the latest Pyrit version from pyrit.googlecode.com.

wget http://pyrit.googlecode.com/files/pyrit-0.3.0.tar.gz

Extract, build and install the package.

tar zxvf pyrit-0.3.0.tar.gz
cd pyrit-0.3.0
python setup.py build
sudo python setup.py install

If errors like this occurs:

cpyrit/_cpyrit_cpu.c:21:20: error: Python.h: No such file or directory

On Ubuntu, try to install the python2.6-dev package along with the other dev packages.

sudo aptitude install python2.6-dev

The 0.3.1-dev r242 svn repository version requires 'libpcap0.8-dev installed, else this error hits:

cpyrit/_cpyrit_cpu.c:26:18: error: pcap.h: No such file or directory

Test to verify the core modules work.

pyrit eval
pyrit list_cores
pyrit -e linksys create_essid

If these operations execute without error, everything should be set.

To try the latest experimental svn version, try the command below. I recommend reading the pyrit svn changelog over at the project page.

sudo aptitude install subversion
svn checkout http://pyrit.googlecode.com/svn/trunk/ pyrit_svn
cd pyrit_svn
# And compile as normal for pyrit and cpyrit_opencl

GPU Cores

Now, to install the GPU accelerated modules. A good tip is to pick a desktop distro with the graphical interface already configured, as the cpyrit modules require X Window to be running when in use. Also X11vnc comes in handy when remotely administering the system. Put /usr/bin/x11vnc -display :0 -noxdamage in /etc/X11/Xsession.d/55gnome-session_gnomerc.

OpenCL (Nvidia/AMD/Cell)

OpenCL support requires the OpenCL SDK; Nvidia OpenCL SDK or ATI Stream SDK.

But before installing the SDK, the (proprietary) graphics drivers for the video card has to be installed and set-up beforehand. This is normally a procedure covered on great detail in the distro wiki or help forum, as it's something most Linux users need to do to get GPU acceleration up and running (games). For Ubuntu, see Check the ATI Binary Driver Howto and cchtml Installation Guide.

Here is a summary, for ATI Catalyst 10.3 (8.702) on a 64-bit setup:

Install binary drivers under System->Administration->Hardware Drivers
Enable the Source Code repository from the Synaptic Package Manger
wget https://a248.e.akamai.net/f/674/9206/0/www2.ati.com/drivers/linux/ati-driver-installer-10-3-x86.x86_64.run
sudo apt-get update
sudo apt-get install libqtgui4
sudo sh ./ati-driver-installer-10-3-x86.x86_64.run --buildpkg Ubuntu/karmic # will install dependencies and build packages
sudo apt-get install ia32-libs # only for 64-bit systems
sudo dpkg -i *.deb
If errors about libdri.so.xlibmesa occur, try removing xorg-driver-fglrx, apti-get remove --purge xorg-driver-fglrx and install the package again
In case of broken packages, open Synaptic Package Manager and click on Fix Broken Packages from Edit menu
sudo aticonfig --initial -f # generates /etc/X11/xorg.conf
fglrxinfo # when logged into X window to see the adapters details

Later, to update the drivers; first remove all the driver files and then install the new package as mention above (minus the libqtgui4, ia32-libs, etc).

sudo apt-get remove --purge xorg-driver-fglrx 

To install the ATI SDK, follow these steps to set up the paths (there are no packages or anything to compile).

Download the ATI SDK package from the official ATI Stream SDK page.

wget http://download2-developer.amd.com/amd/Stream20GA/ati-stream-sdk-v2.01-lnx64.tgz
tar zxvf ati-stream-sdk-v2.01-lnx64.tgz
mv ati-stream-sdk-v2.01-lnx64 ~/ati-stream-sdk

As per the installation notes, set the paths and create the /usr/lib/OpenCL/vendors symblinks.

pico ~/.bashrc
# paste the following lines and correct the username/64-bit path:
export ATISTREAMSDKROOT=/home/ivc/ati-stream-sdk
export ATISTREAMSDKSAMPLESROOT=/home/ivc/ati-stream-sdk/samples
export LD_LIBRARY_PATH=$ATISTREAMSDKROOT/lib/x86_64:$LD_LIBRARY_PATH
sudo mkdir -p /usr/lib/OpenCL/vendors
sudo ln -sf /home/ivc/ati-stream-sdk/lib/x86/libatiocl32.so /usr/lib/OpenCL/vendors/libatiocl32.so # full path, not relative
sudo ln -sf /home/ivc/ati-stream-sdk/lib/x86_64/libatiocl64.so /usr/lib/OpenCL/vendors/libatiocl64.so # full path, not relative

If Pyrit reports an error like the one below, it might be an issue with the Stream SDK setup / symblinks.

Failed to load Pyrit's OpenCL-driven core ('Failed to enumerate OpenCL-platforms (Unknown CLresult)')

It might be necessary to install libstdcc++5 on Ubuntu using the method below, as Ubuntu 9.10 only ships with libstdcc++6:

sudo pico /etc/apt/sources.list
# add the following lines
deb http://ppa.launchpad.net/jason-scheunemann/ppa/ubuntu karmic main
deb-src http://ppa.launchpad.net/jason-scheunemann/ppa/ubuntu karmic main
wget -q 'http://keyserver.ubuntu.com:11371/pks/lookup?op=get&search=0x932062C9CD30EE56' -O - | sudo apt-key add - # add the ppa key
sudo aptitude update
sudo aptitude install lib32stdc++5

• libstdc++.so.6 installed but not found
• Karmic No Longer Includes 'libstdc++5'
• libstdc++5 removal breaks non-ubuntu applications

And finally, to build the cpyrit OpenCL module:

sudo aptitude install python3 python3-dev libssl-dev
wget http://pyrit.googlecode.com/files/cpyrit-opencl-0.3.0.tar.gz
tar zxvf cpyrit-opencl-0.3.0.tar.gz
cd cpyrit-opencl-0.3.0
# Edit setup.py as mentioned below to add the OpenCL include path
python setup.py build
sudo python setup.py install

The cpyrit setup.ph script doesn't know where to find the cl.h header files and will report this error:

_cpyrit_opencl.c:23:19: error: CL/cl.h: No such file or directory

Edit setup.py and at the top add the path for /home/ivc/ati-stream-sdk/include to the list:

for path in ('/usr/local/opencl/OpenCL/common/inc', \
           '/opt/opencl/OpenCL/common/inc', \
           '/home/ivc/ati-stream-sdk/include', \
           '/usr/local/opencl/include'):

I had problems with the build and ld reporting it couldn't find the OpenCL library:

/usr/bin/ld: cannot find -lOpenCL
collect2: ld returned 1 exit status

A quick workaround was to create a symblink of libOpenCL.so to /lib64:

sudo ln -s /home/ivc/ati-stream-sdk/lib/x86_64/libOpenCL.so /lib64/

X Window has to be attached to at least one display out for each card installed in the system. Enabled them in /etc/xorg.conf, that is, one "Device" + "Monitor" + "Screen" combination for each card installed. Check my xorg.conf for an example for 2x HD 5950 cards setup.

I've successfully generated a working xorg.conf using these commands:

sudo mv /etc/X11/xorg.conf /etc/X11/xorg.conf-previous
sudo cp /etc/X11/xorg.conf.failsafe /etc/X11/xorg.conf
sudo aticonfig --initial -f --adapter=0,1 --resolution=0,1024x768 --resolution=1,1024x768
sudo stop gdm; sudo start gdm

Finally, to test and verify that the new module is ready, run the following commands in a remote ssh session or console. Using an X session will result in only the CPU cores and/or (n-1) graphics cores appearing, as the X session is busy running the termial/pyrit.

pyrit list_cores
pyrit benchmark

The core list and benchmark results on 2x HD 5850 graphics cards, 4x core i5 2.66 GHz processor, 4 GB memory setup:

Pyrit 0.3.0 (C) 2008-2010 Lukas Lueg http://pyrit.googlecode.com
This code is distributed under the GNU General Public License v3+

Running benchmark (72963.2 PMKs/s)... \

Computed 73196.00 PMKs/s total.
#1: 'OpenCL-Device 'Cypress'': 34107.9 PMKs/s (RTT 1.7)
#2: 'OpenCL-Device 'Cypress'': 34179.9 PMKs/s (RTT 1.8)
#3: 'CPU-Core (SSE2)': 776.6 PMKs/s (RTT 2.8)
#4: 'CPU-Core (SSE2)': 785.4 PMKs/s (RTT 2.8)
#5: 'Network-Clients': 0.0 PMKs/s (RTT 0.0)

The reason only two processors appear in the list above, is that for each single graphics core, a CPU core is used to feed and process the data to/from the GPU. An example with X Window shut down, thus only CPU cores used, is shown below.

Pyrit 0.3.0 (C) 2008-2010 Lukas Lueg http://pyrit.googlecode.com
This code is distributed under the GNU General Public License v3+

Running benchmark (3217.4 PMKs/s)... \

Computed 3362.03 PMKs/s total.
#1: 'CPU-Core (SSE2)': 791.1 PMKs/s (RTT 2.8)
#2: 'CPU-Core (SSE2)': 788.4 PMKs/s (RTT 2.8)
#3: 'CPU-Core (SSE2)': 791.2 PMKs/s (RTT 2.8)
#4: 'CPU-Core (SSE2)': 791.4 PMKs/s (RTT 2.8)
#5: 'Network-Clients': 0.0 PMKs/s (RTT 0.0)

A worthy tool to mention is aticonfig; it's a tool to configure, control, and get status reports from supported ATI cards. Run the tool in an X session, remote ssh terminal doesn't work/supported.

• sudo aticonfig --adapter=0,1 --odgt - get the temperature from adapter 0 and 1, mine range from 58 to 79 degrees
• sudo aticonfig --adapter=0,1 --odgc - see clocks speed and gpu load, mine hovers around 86% when pyrit loaded
• sudo aticonfig --pplib-cmd 'get fanspeed 0' - check the current fan speed on adapter 0, mine stays in the 58-63% range
• sudo aticonfig --pplib-cmd 'set fanspeed 0 50' - adjust the fan speed, 50%, mine only seems to work on adapter 0, atioc below can set both though
• sudo aticonfig --od-enable - for the adventurous, enable Overdrive to release the overclocking potential
  • sudo aticonfig --odsc 775,1125 - set the new core and memory clock speeds, had little effect in the benchmark results, default 765,1000
  • sudo aticonfig --odcc - commit, or save, the new clock speeds
  • sudo stop gdm and sudo start gdm - restart X to take advantage of the new speeds
• aticonfig --help - find all the other goodness, or see this aticonfig help output

Another, GUI, tool I like is ATI Overclocking Utility by sabby. A simple interface showing the current clock speeds, temperature, fanspeed, and also Overdrive panel for overclocking.

sudo aptitude install libqt4-core
wget http://beta.ivancover.com/wifi/pyrit/atioc105_x64.tgz # 64-bit version
tar zxvf atioc105_x64.tgz
./atioc # from an X session of course

To make changes persistent over reboots, check off 'Reload Overdrive settings on startup' under Settings and put /home/ivc/pyrit/atioc & where it's executed during login.

References:

• Overclocking ATI cards in Linux
• Controlling the fan speed for ATI/AMD GPUs

Nvidia CUDA

This module install in a similar way. I have not tested this method. Make sure the Nvidia graphics drivers work with the graphics card. Next install the CUDA-Toolkit to get the required Nvidia compiler.

To install the cpyrit CUDA module:

wget http://pyrit.googlecode.com/files/cpyrit-cuda-0.3.0.tar.gz
tar zxvf cpyrit-cuda-0.3.0
python setup.py build
sudo python setup.py install

Test to verify the new module is ready.

pyrit list_cores

Database Setup

The default Pyrit file container is a blobspace on the local filesystem, but with the -u mysql://host Pyrit parameter an external database can be used for storage.

I will not cover the configuration and setup of a MySQL cluster, see this Howtoforge.com guide and my list of cluster links for more details.

To begin, the current SQLAlchemy version, 0.4.5, is too old to be used with pyrit. Version 0.5.6 or later is required for pyrit 0.3.0.

Get the latest stable release, extract, build and install.

wget http://downloads.sourceforge.net/project/sqlalchemy/sqlalchemy/0.5.8/SQLAlchemy-0.5.8.tar.gz
tar zxvf SQLAlchemy-0.5.8.tar.gz
cd SQLAlchemy-0.5.8
sudo aptitude install python-setuptools
sudo easy_install SQLAlchemy

Next, to use a MySQL database, install the python-mysqldb package.

sudo aptitude install python-mysqldb

Or else an error like this will let you know:

    return __import__('MySQLdb')
ImportError: No module named MySQLdb

After upgrading to SQLAlchemy 0.5.8, pyrit might complain with a warning that Binary type has been replaced by LargeBinary. Doing the suggested edits from Binary to LargeBinary will make the warning messages disappear.

/usr/local/lib/python2.6/dist-packages/cpyrit/storage.py:741: SADeprecationWarning: The Binary type has been renamed to LargeBinary.
  sql.Column('essid', sql.Binary(32), nullable=False),
/usr/local/lib/python2.6/dist-packages/cpyrit/storage.py:750: SADeprecationWarning: The Binary type has been renamed to LargeBinary.
  sql.Column('collection_buffer', sql.Binary(2**24-1), \
/usr/local/lib/python2.6/dist-packages/cpyrit/storage.py:762: SADeprecationWarning: The Binary type has been renamed to LargeBinary.
  sql.Column('results_buffer', sql.Binary(2**24 - 1), \

On a default MySQL server install, max_allowed_packet is 1M. For pyrit to work on large datasets/blobs increase this to f.ex. 32M in /etc/my.cnf (cluster) or /etc/mysql/my.cnf (standard).

[MYSQLD]
max_allowed_packet=32M

Lastly, for a NDB Cluster I modified the Pyrit table sql queries to reflect that it's a NDB engine.

• Pyrit NDB SQL Queries

Test 2010-04-12:

On a MySQl NDB Cluster 7.0.13 on 2 data nodes, 4 GB memory, gigabit-network, and NoOfReplicas=1 (no redundancy but for speed), the performance is good but a undetermined bottleneck (probably network latency, neither memory, cpu, disk topped) and falling efficiency hinders full GPU utilization. I haven't looked into the MySQL Cluster setup too much.

Pyrit 0.3.0 (C) 2008-2010 Lukas Lueg http://pyrit.googlecode.com
This code is distributed under the GNU General Public License v3+

Connecting to storage at 'mysql://root:@10.0.0.10/pyrit'...  connected.
Working on ESSID 'dlink'
Processed all workunits for ESSID 'dlink'; 33229 PMKs per second.d.

Working on ESSID 'wifi'
Processed all workunits for ESSID 'wifi'; 15719 PMKs per second.nd.

Working on ESSID 'linksys'
Processed all workunits for ESSID 'linksys'; 11186 PMKs per second.

Working on ESSID 'internet'
Processed all workunits for ESSID 'internet'; 8864 PMKs per second.

Working on ESSID 'free_internet'
Processed all workunits for ESSID 'free_internet'; 7864 PMKs per second.

Running the MySQL node on the same machine as the GPU didn't change the performance issue.

Update #1 2010-04-12:

Upgrading to the latest SVN version, 0.3.1-dev r242, helped tremendously. Now it tops out at 63000 PKMs per second (ranging form 50k-63k) on the NDB cluster! With a little tweaking I guess it's possible to hit the maximum 72000 PMKs/s (which is the benchmark score from the test further up).

Pyrit 0.3.1-dev (svn r242) (C) 2008-2010 Lukas Lueg http://pyrit.googlecode.com
This code is distributed under the GNU General Public License v3+

Connecting to storage at 'mysql://root:pass@10.0.0.10/pyrit'...  connected.
Working on ESSID 'wifi'
Processed 752/768 workunits so far (97.9%); 61275 PMKs per second.

I made a video showing the PMKs per second and network activity, check it: Pyrit MySQL Cluster Batch.

Update #2 2010-04-17:

As a last test I would like to find out if running a local MySQL Server would help the performance compared to the cluster.

Pyrit 0.3.1-dev (svn r242) (C) 2008-2010 Lukas Lueg http://pyrit.googlecode.com
This code is distributed under the GNU General Public License v3+

Connecting to storage at 'mysql://root:pass@localhost/pyrit'...  connected.
Working on ESSID '2WIRE591'
Processed 761/768 workunits so far (99.1%); 63357 PMKs per second.

Another run..

Pyrit 0.3.1-dev (svn r242) (C) 2008-2010 Lukas Lueg http://pyrit.googlecode.com
This code is distributed under the GNU General Public License v3+

Connecting to storage at 'mysql://root:pass@localhost/pyrit'...  connected.
Working on ESSID '2WIRE592'
Processed 757/768 workunits so far (98.6%); 82363 PMKs per second.

The results were better, in fact, at short moments topping out at over 80k (a best). It fluctuated in the 55k-75k range.

Commands

A list of commands used during the testing:

• pyrit eval
• pyrit list_cores
• pyrit -i passwords.txt import_passwords
• pyrit -e 2WIRE590 create_essid
• pyrit batch
• pyrit -r wpapsk-linksys.dump attack_db
• pyrit -u mysql://root:pass@10.0.0.10/pyrit -r wpapsk-linksys.dump attack_db

Check the reference manual for great details on all the commands.

References

• Pyrit installation
• Pyrit tutorial
• Reference manual