Pyrit setup
Pyrit is a tool used to pre-compute all possible SSID:PASSWORD combinations to generate a list of PMK tokens.
I ran this setup on a 'homebrew' cluster consisting of 4 mainboards stacked in height using threaded rods. Bottom 2 boards are the Datanodes, 3rd is the MySQL Server, while the upper-most is the actual GPGPU Server with slots for 4x Cuda/Stream acceleration cards.
Setup
This cluster is installed on a set of 32-bit Ubuntu Server Edition (9.10) database servers and 64-bit Ubuntu Desktop Edition for the ATI Stream GPU accelerator.
- ATI Stream GPU accelerator - i5 750 2.66 GHz, Asus P7P55 WS SC, 4GB memory, 2x Gigabyte GV-R585OC-1GD Radeon 5850 - 10.0.0.2
- MySQL NDB cluster master - P4 2.4 GHz, Supermicro P4SAA, 2 GB memory - 10.0.0.10
- MySQL NDB data storage #1 - AMD64 3000+ 1.8 GHz, DFI NF4 SLI-DR, 4GB memory - 10.0.0.11
- MySQL NDB data storage #2 - AMD64 3000+ 1.8 GHz, A8N-SLI Premium, 4GB memory - 10.0.0.12
All boards are connected through a 5-port Netgear gigabit desktop switch.
Pre-requisite
Pyrit version 0.3.0 and up depend on Scapy2 (package inspection) and SQLAlchemy 0.5.6 (database interface).
Scapy: On a Debian/Ubuntu bases distribution, get the package using aptitude. Backtrack 4 final include the Scapy2.
sudo aptitude update sudo aptitude install python-scapy scapy2
Install
Main App
Download the latest Pyrit version from pyrit.googlecode.com.
wget http://pyrit.googlecode.com/files/pyrit-0.3.0.tar.gz
Extract, build and install the package.
tar zxvf pyrit-0.3.0.tar.gz cd pyrit-0.3.0 python setup.py build sudo python setup.py install
If errors like this occurs:
cpyrit/_cpyrit_cpu.c:21:20: error: Python.h: No such file or directory
On Ubuntu, try to install the python2.6-dev package along with the other dev packages.
sudo aptitude install python2.6-dev
The 0.3.1-dev r242 svn repository version requires 'libpcap0.8-dev installed, else this error hits:
cpyrit/_cpyrit_cpu.c:26:18: error: pcap.h: No such file or directory
Test to verify the core modules work.
pyrit eval pyrit list_cores pyrit -e linksys create_essid
If these operations execute without error, everything should be set.
To try the latest experimental svn version, try the command below. I recommend reading the pyrit svn changelog over at the project page.
sudo aptitude install subversion svn checkout http://pyrit.googlecode.com/svn/trunk/ pyrit_svn cd pyrit_svn # And compile as normal for pyrit and cpyrit_opencl
GPU Cores
Now, to install the GPU accelerated modules. A good tip is to pick a desktop distro with the graphical interface already configured, as the cpyrit modules require X Window to be running when in use. Also X11vnc comes in handy when remotely administering the system. Put /usr/bin/x11vnc -display :0 -noxdamage in /etc/X11/Xsession.d/55gnome-session_gnomerc.
OpenCL (Nvidia/AMD/Cell)
OpenCL support requires the OpenCL SDK; Nvidia OpenCL SDK or ATI Stream SDK.
But before installing the SDK, the (proprietary) graphics drivers for the video card has to be installed and set-up beforehand. This is normally a procedure covered on great detail in the distro wiki or help forum, as it's something most Linux users need to do to get GPU acceleration up and running (games). For Ubuntu, see Check the ATI Binary Driver Howto and cchtml Installation Guide.
Here is a summary, for ATI Catalyst 10.3 (8.702) on a 64-bit setup:
Install binary drivers under System->Administration->Hardware Drivers Enable the Source Code repository from the Synaptic Package Manger wget https://a248.e.akamai.net/f/674/9206/0/www2.ati.com/drivers/linux/ati-driver-installer-10-3-x86.x86_64.run sudo apt-get update sudo apt-get install libqtgui4 sudo sh ./ati-driver-installer-10-3-x86.x86_64.run --buildpkg Ubuntu/karmic # will install dependencies and build packages sudo apt-get install ia32-libs # only for 64-bit systems sudo dpkg -i *.deb If errors about libdri.so.xlibmesa occur, try removing xorg-driver-fglrx, apti-get remove --purge xorg-driver-fglrx and install the package again In case of broken packages, open Synaptic Package Manager and click on Fix Broken Packages from Edit menu sudo aticonfig --initial -f # generates /etc/X11/xorg.conf fglrxinfo # when logged into X window to see the adapters details
Later, to update the drivers; first remove all the driver files and then install the new package as mention above (minus the libqtgui4, ia32-libs, etc).
sudo apt-get remove --purge xorg-driver-fglrx
To install the ATI SDK, follow these steps to set up the paths (there are no packages or anything to compile).
Download the ATI SDK package from the official ATI Stream SDK page.
wget http://download2-developer.amd.com/amd/Stream20GA/ati-stream-sdk-v2.01-lnx64.tgz tar zxvf ati-stream-sdk-v2.01-lnx64.tgz mv ati-stream-sdk-v2.01-lnx64 ~/ati-stream-sdk
As per the installation notes, set the paths and create the /usr/lib/OpenCL/vendors symblinks.
pico ~/.bashrc # paste the following lines and correct the username/64-bit path: export ATISTREAMSDKROOT=/home/ivc/ati-stream-sdk export ATISTREAMSDKSAMPLESROOT=/home/ivc/ati-stream-sdk/samples export LD_LIBRARY_PATH=$ATISTREAMSDKROOT/lib/x86_64:$LD_LIBRARY_PATH sudo mkdir -p /usr/lib/OpenCL/vendors sudo ln -sf /home/ivc/ati-stream-sdk/lib/x86/libatiocl32.so /usr/lib/OpenCL/vendors/libatiocl32.so # full path, not relative sudo ln -sf /home/ivc/ati-stream-sdk/lib/x86_64/libatiocl64.so /usr/lib/OpenCL/vendors/libatiocl64.so # full path, not relative
If Pyrit reports an error like the one below, it might be an issue with the Stream SDK setup / symblinks.
Failed to load Pyrit's OpenCL-driven core ('Failed to enumerate OpenCL-platforms (Unknown CLresult)')
It might be necessary to install libstdcc++5 on Ubuntu using the method below, as Ubuntu 9.10 only ships with libstdcc++6:
sudo pico /etc/apt/sources.list # add the following lines deb http://ppa.launchpad.net/jason-scheunemann/ppa/ubuntu karmic main deb-src http://ppa.launchpad.net/jason-scheunemann/ppa/ubuntu karmic main wget -q 'http://keyserver.ubuntu.com:11371/pks/lookup?op=get&search=0x932062C9CD30EE56' -O - | sudo apt-key add - # add the ppa key sudo aptitude update sudo aptitude install lib32stdc++5
- libstdc++.so.6 installed but not found
- Karmic No Longer Includes 'libstdc++5'
- libstdc++5 removal breaks non-ubuntu applications
And finally, to build the cpyrit OpenCL module:
sudo aptitude install python3 python3-dev libssl-dev wget http://pyrit.googlecode.com/files/cpyrit-opencl-0.3.0.tar.gz tar zxvf cpyrit-opencl-0.3.0.tar.gz cd cpyrit-opencl-0.3.0 # Edit setup.py as mentioned below to add the OpenCL include path python setup.py build sudo python setup.py install
The cpyrit setup.ph script doesn't know where to find the cl.h header files and will report this error:
_cpyrit_opencl.c:23:19: error: CL/cl.h: No such file or directory
Edit setup.py and at the top add the path for /home/ivc/ati-stream-sdk/include to the list:
for path in ('/usr/local/opencl/OpenCL/common/inc', \
'/opt/opencl/OpenCL/common/inc', \
'/home/ivc/ati-stream-sdk/include', \
'/usr/local/opencl/include'):
I had problems with the build and ld reporting it couldn't find the OpenCL library:
/usr/bin/ld: cannot find -lOpenCL collect2: ld returned 1 exit status
A quick workaround was to create a symblink of libOpenCL.so to /lib64:
sudo ln -s /home/ivc/ati-stream-sdk/lib/x86_64/libOpenCL.so /lib64/
X Window has to be attached to at least one display out for each card installed in the system. Enabled them in /etc/xorg.conf, that is, one "Device" + "Monitor" + "Screen" combination for each card installed. Check my xorg.conf for an example for 2x HD 5950 cards setup.
I've successfully generated a working xorg.conf using these commands:
sudo mv /etc/X11/xorg.conf /etc/X11/xorg.conf-previous sudo cp /etc/X11/xorg.conf.failsafe /etc/X11/xorg.conf sudo aticonfig --initial -f --adapter=0,1 --resolution=0,1024x768 --resolution=1,1024x768 sudo stop gdm; sudo start gdm
Finally, to test and verify that the new module is ready, run the following commands in a remote ssh session or console. Using an X session will result in only the CPU cores and/or (n-1) graphics cores appearing, as the X session is busy running the termial/pyrit.
pyrit list_cores pyrit benchmark
The core list and benchmark results on 2x HD 5850 graphics cards, 4x core i5 2.66 GHz processor, 4 GB memory setup:
Pyrit 0.3.0 (C) 2008-2010 Lukas Lueg http://pyrit.googlecode.com This code is distributed under the GNU General Public License v3+ Running benchmark (72963.2 PMKs/s)... \ Computed 73196.00 PMKs/s total. #1: 'OpenCL-Device 'Cypress'': 34107.9 PMKs/s (RTT 1.7) #2: 'OpenCL-Device 'Cypress'': 34179.9 PMKs/s (RTT 1.8) #3: 'CPU-Core (SSE2)': 776.6 PMKs/s (RTT 2.8) #4: 'CPU-Core (SSE2)': 785.4 PMKs/s (RTT 2.8) #5: 'Network-Clients': 0.0 PMKs/s (RTT 0.0)
The reason only two processors appear in the list above, is that for each single graphics core, a CPU core is used to feed and process the data to/from the GPU. An example with X Window shut down, thus only CPU cores used, is shown below.
Pyrit 0.3.0 (C) 2008-2010 Lukas Lueg http://pyrit.googlecode.com This code is distributed under the GNU General Public License v3+ Running benchmark (3217.4 PMKs/s)... \ Computed 3362.03 PMKs/s total. #1: 'CPU-Core (SSE2)': 791.1 PMKs/s (RTT 2.8) #2: 'CPU-Core (SSE2)': 788.4 PMKs/s (RTT 2.8) #3: 'CPU-Core (SSE2)': 791.2 PMKs/s (RTT 2.8) #4: 'CPU-Core (SSE2)': 791.4 PMKs/s (RTT 2.8) #5: 'Network-Clients': 0.0 PMKs/s (RTT 0.0)
A worthy tool to mention is aticonfig; it's a tool to configure, control, and get status reports from supported ATI cards. Run the tool in an X session, remote ssh terminal doesn't work/supported.
- sudo aticonfig --adapter=0,1 --odgt - get the temperature from adapter 0 and 1, mine range from 58 to 79 degrees
- sudo aticonfig --adapter=0,1 --odgc - see clocks speed and gpu load, mine hovers around 86% when pyrit loaded
- sudo aticonfig --pplib-cmd 'get fanspeed 0' - check the current fan speed on adapter 0, mine stays in the 58-63% range
- sudo aticonfig --pplib-cmd 'set fanspeed 0 50' - adjust the fan speed, 50%, mine only seems to work on adapter 0, atioc below can set both though
- sudo aticonfig --od-enable - for the adventurous, enable Overdrive to release the overclocking potential
- sudo aticonfig --odsc 775,1125 - set the new core and memory clock speeds, had little effect in the benchmark results, default 765,1000
- sudo aticonfig --odcc - commit, or save, the new clock speeds
- sudo stop gdm and sudo start gdm - restart X to take advantage of the new speeds
- aticonfig --help - find all the other goodness, or see this aticonfig help output
Another, GUI, tool I like is ATI Overclocking Utility by sabby. A simple interface showing the current clock speeds, temperature, fanspeed, and also Overdrive panel for overclocking.
sudo aptitude install libqt4-core wget http://beta.ivancover.com/wifi/pyrit/atioc105_x64.tgz # 64-bit version tar zxvf atioc105_x64.tgz ./atioc # from an X session of course
To make changes persistent over reboots, check off 'Reload Overdrive settings on startup' under Settings and put /home/ivc/pyrit/atioc & where it's executed during login.
References:
Nvidia CUDA
This module install in a similar way. I have not tested this method. Make sure the Nvidia graphics drivers work with the graphics card. Next install the CUDA-Toolkit to get the required Nvidia compiler.
To install the cpyrit CUDA module:
wget http://pyrit.googlecode.com/files/cpyrit-cuda-0.3.0.tar.gz tar zxvf cpyrit-cuda-0.3.0 python setup.py build sudo python setup.py install
Test to verify the new module is ready.
pyrit list_cores
Database Setup
The default Pyrit file container is a blobspace on the local filesystem, but with the -u mysql://host Pyrit parameter an external database can be used for storage.
I will not cover the configuration and setup of a MySQL cluster, see this Howtoforge.com guide and my list of cluster links for more details.
To begin, the current SQLAlchemy version, 0.4.5, is too old to be used with pyrit. Version 0.5.6 or later is required for pyrit 0.3.0.
Get the latest stable release, extract, build and install.
wget http://downloads.sourceforge.net/project/sqlalchemy/sqlalchemy/0.5.8/SQLAlchemy-0.5.8.tar.gz tar zxvf SQLAlchemy-0.5.8.tar.gz cd SQLAlchemy-0.5.8 sudo aptitude install python-setuptools sudo easy_install SQLAlchemy
Next, to use a MySQL database, install the python-mysqldb package.
sudo aptitude install python-mysqldb
Or else an error like this will let you know:
return __import__('MySQLdb')
ImportError: No module named MySQLdb
After upgrading to SQLAlchemy 0.5.8, pyrit might complain with a warning that Binary type has been replaced by LargeBinary. Doing the suggested edits from Binary to LargeBinary will make the warning messages disappear.
/usr/local/lib/python2.6/dist-packages/cpyrit/storage.py:741: SADeprecationWarning: The Binary type has been renamed to LargeBinary.
sql.Column('essid', sql.Binary(32), nullable=False),
/usr/local/lib/python2.6/dist-packages/cpyrit/storage.py:750: SADeprecationWarning: The Binary type has been renamed to LargeBinary.
sql.Column('collection_buffer', sql.Binary(2**24-1), \
/usr/local/lib/python2.6/dist-packages/cpyrit/storage.py:762: SADeprecationWarning: The Binary type has been renamed to LargeBinary.
sql.Column('results_buffer', sql.Binary(2**24 - 1), \
On a default MySQL server install, max_allowed_packet is 1M. For pyrit to work on large datasets/blobs increase this to f.ex. 32M in /etc/my.cnf (cluster) or /etc/mysql/my.cnf (standard).
[MYSQLD] max_allowed_packet=32M
Lastly, for a NDB Cluster I modified the Pyrit table sql queries to reflect that it's a NDB engine.
Test 2010-04-12:
On a MySQl NDB Cluster 7.0.13 on 2 data nodes, 4 GB memory, gigabit-network, and NoOfReplicas=1 (no redundancy but for speed), the performance is good but a undetermined bottleneck (probably network latency, neither memory, cpu, disk topped) and falling efficiency hinders full GPU utilization. I haven't looked into the MySQL Cluster setup too much.
Pyrit 0.3.0 (C) 2008-2010 Lukas Lueg http://pyrit.googlecode.com This code is distributed under the GNU General Public License v3+ Connecting to storage at 'mysql://root:@10.0.0.10/pyrit'... connected. Working on ESSID 'dlink' Processed all workunits for ESSID 'dlink'; 33229 PMKs per second.d. Working on ESSID 'wifi' Processed all workunits for ESSID 'wifi'; 15719 PMKs per second.nd. Working on ESSID 'linksys' Processed all workunits for ESSID 'linksys'; 11186 PMKs per second. Working on ESSID 'internet' Processed all workunits for ESSID 'internet'; 8864 PMKs per second. Working on ESSID 'free_internet' Processed all workunits for ESSID 'free_internet'; 7864 PMKs per second.
Running the MySQL node on the same machine as the GPU didn't change the performance issue.
Update #1 2010-04-12:
Upgrading to the latest SVN version, 0.3.1-dev r242, helped tremendously. Now it tops out at 63000 PKMs per second (ranging form 50k-63k) on the NDB cluster! With a little tweaking I guess it's possible to hit the maximum 72000 PMKs/s (which is the benchmark score from the test further up).
Pyrit 0.3.1-dev (svn r242) (C) 2008-2010 Lukas Lueg http://pyrit.googlecode.com This code is distributed under the GNU General Public License v3+ Connecting to storage at 'mysql://root:pass@10.0.0.10/pyrit'... connected. Working on ESSID 'wifi' Processed 752/768 workunits so far (97.9%); 61275 PMKs per second.
I made a video showing the PMKs per second and network activity, check it: Pyrit MySQL Cluster Batch.
Update #2 2010-04-17:
As a last test I would like to find out if running a local MySQL Server would help the performance compared to the cluster.
Pyrit 0.3.1-dev (svn r242) (C) 2008-2010 Lukas Lueg http://pyrit.googlecode.com This code is distributed under the GNU General Public License v3+ Connecting to storage at 'mysql://root:pass@localhost/pyrit'... connected. Working on ESSID '2WIRE591' Processed 761/768 workunits so far (99.1%); 63357 PMKs per second.
Another run..
Pyrit 0.3.1-dev (svn r242) (C) 2008-2010 Lukas Lueg http://pyrit.googlecode.com This code is distributed under the GNU General Public License v3+ Connecting to storage at 'mysql://root:pass@localhost/pyrit'... connected. Working on ESSID '2WIRE592' Processed 757/768 workunits so far (98.6%); 82363 PMKs per second.
The results were better, in fact, at short moments topping out at over 80k (a best). It fluctuated in the 55k-75k range.
Commands
A list of commands used during the testing:
- pyrit eval
- pyrit list_cores
- pyrit -i passwords.txt import_passwords
- pyrit -e 2WIRE590 create_essid
- pyrit batch
- pyrit -r wpapsk-linksys.dump attack_db
- pyrit -u mysql://root:pass@10.0.0.10/pyrit -r wpapsk-linksys.dump attack_db
Check the reference manual for great details on all the commands.
