Showing posts with label Python. Show all posts
Showing posts with label Python. Show all posts

Wednesday, April 24, 2013

A Simple Python Plugin Framework

This framework is somewhat different than those currently out there, most of those either:
  • don't allow dynamic plugins
  • require you to use nasty hooks
  • require specifically formatted imports
  • require plugins to all be in one place
This script however allows plugins to:
  • be dynamically loaded
  • be a subclass of a master plugin class
  • have any name
  • be in many different folders
The general idea is that you give the function a path, and the base-class that all plugins are subclasses of, then it iterates through the path, finding python files, loading them, and then searching them for any classes that are subclasses of the class you want. When it gets done, it returns them:

Code

import inspect
import os
import sys
def load_plugins(plugin_path, instance):
'''Loads a set of plugins at the given path.

Arguments:
plugin_path - the OS path to look for plugins at.
instance - classes of this instance will be returned
'''
plugins = []
plugin_dir = os.path.realpath(plugin_path)
sys.path.append(plugin_dir)

for f in os.listdir(plugin_dir):
if f.endswith(".py"):
name = f[:-3]
elif f.endswith(".pyc"):
name = f[:-4]
elif os.path.isdir(os.path.join(plugin_dir, f)):
name = f
else:
continue

try:
mod = __import__(name, globals(), locals(), [], 0)

for piece in inspect.getmembers(mod):
if isinstance(piece, instance):
plugins.append(piece)

except ImportError as e:
print(e)
pass # problem importing

return plugins

Wednesday, April 17, 2013

Things I Learned From Working A 2 Year 50,000 Line Project

This is mostly to serve as a brain dump for a piece of software I've been writing for around two and a half years, it was comprised of a large database (read: ~five hundred million rows), a cross platform Java application, a large Python web-server/API, a phone app, lots of small scripts that fetched and transformed data, and globs of glue. (I wrote all of them in the two year period!)

Architecture

  • It pays to fully think through your APIs first
  • All web APIs should have a version in the URL e.g. /application/api/2.0/...
  • Profile early and often
  • A few good caches are worth many bad ones
  • Make everything a plugin that extends one class; you may think it's a big hairy mess, but it's better than multiple plugin systems.
  • Use libraries wherever possible
  • Ditch libraries when you only use one function from them

Debugging

  • Users write awful bug reports
  • You can help them write better ones by giving them a short form to fill out.
  • Automatically attach debugging info to all bug reports
  • Make version numbers easily available (not just in the executable's name)
  • Auto-update your apps so you don't have users lagging behind

Upgrades

  • Your requirements will constantly change, make sure you have an easy upgrade path in your database that won't force hacky solutions.
  • Make sure your ORM supports incremental changes.

User Experience

  • Start fast, even if you're not done processing yet
  • Never trust an engineer to do design (including software engineers)
  • If the user doesn't see it, it doesn't exist
  • Make it very hard for the user to do stupid things
  • Do automatic backups/saves

Sanity

  • Clients like saying "I did say that, didn't I? What I meant was..."
    • all features should be plugins so you can enable/disable when they do this
  • SVN and Eclipse are a nightmare to get playing nice
  • Java 7 is really awesome!
    • auto-closing streams
    • multiple catches in a try block
    • string in switch statements
  • JNLP sucks, but so does embedding apps in-browser; just make it easy for the user to download it, and provide auto upgrading

Monday, February 4, 2013

HOWTO Use Python "for in" and "for else" Loops

This is a short guide to the subtleties of Python for loops, especially for those programmers from Java/C++ that may miss some of the best features because those languages don't have any equivalent, it covers:
  • The basic for loop
  • Looping over dictionaries
  • Using the for else loop

The Basic for loop

If you come from a Java/C++/PHP background, you're used to two different for loops foreach and for. Python only has one of these, the more powerful foreach.
In Java, your for loop would look something like this:
for(int i = 0; i < 10; i += 2)
{
System.out.println(i);
}
In Python, the same loop would look like;
for i in range(0, 10, 2):
print(i)
The range function returns a list of numbers 0 to 10 incremented by 2.
If you wanted 0-9 inclusive, the loop would be even more simple.
for foo in range(10):
print(foo)
In this same manner, you can replace the range() with any list:
for j in [1,1,2,3,5]:
print(j)

Looping over Dictionaries (maps)

Maps are a very handy Python builtin feature, allowing you to create arbitrary key->value pairs.
{
"key":"value",
"bob":"smith",
"age":13,
"dob":(2012,01,01)
}
You can quickly loop over key->value pairs in a dictionary by using the following:
for key, value in a.items():
print("{} -> {}".format(key, value))
Note that this is much faster than looping over the keys, then looking up the value for each key inside the for loop. The items function returns a list of tuples where the first value is the key, and the second a value.

In fact, you can loop over any list of tuples in a similar manner:
b = [("Perro", 1, ["Canis", "lupis"]),
("Gato", 2, ["Felis", "catus"])]

for name, id, scientific in b:
print("Name: {} ID: {} Breed: {}".format(name, id, " ".join(scientific)))

The Magic of the for else Loop

If you've ever wanted to check if something happened within a for loop you can use an else block after your for loop:
for i in ["foo","bar"]:
if i == "baz":
break
else:
# happens if break is not called, meaning "baz" was not found
return -1

# i is set to the location of "baz"
return i
The example above tries to find the location of a specified string in a list of strings, if found it returns the index, otherwise -1. Of course there are far easier ways to do this, but you get the idea.

Tuesday, January 29, 2013

Extended Euclidean Algorithm in Python

The Extended Euclidean Algorithm is a simple extension to the Euclidean Algorithm that in addition to returning the greatest common denominator also gives numbers s and t such that gcd = s * a + t * b.
Specifically, you may want to find 1 = sa + tb which is very useful in cryptography.

The Code

def extended_euclidean(a,b):
origa = a
origb = b

x = 0
lastx = 1
y = 1
lasty = 0

while b != 0:
q = a // b
a, b = b, a % b

full = "%d = (%d * %d + %d * %d) - %d * (%d * %d + %d * %d) " % ( b, origa, lastx, origb, lasty, q, origa, x, origb, y )
short = "%d = %d * %d + %d * %d" % ( b, origa, lastx - x * q, origb, lasty - y * q)
print("%s\t%s\t%s\t%s" % (a, b, full, short))

x, lastx = (lastx - q * x, x)
y, lasty = (lasty - q * y, y)

return (lastx, lasty)

The Output

Take the numbers 120 and 23:
>>> extended_euclidean(120,23)
23 5 5 = (120 * 1 + 23 * 0) - 5 * (120 * 0 + 23 * 1) 5 = 120 * 1 + 23 * -5
5 3 3 = (120 * 0 + 23 * 1) - 4 * (120 * 1 + 23 * -5) 3 = 120 * -4 + 23 * 21
3 2 2 = (120 * 1 + 23 * -5) - 1 * (120 * -4 + 23 * 21) 2 = 120 * 5 + 23 * -26
2 1 1 = (120 * -4 + 23 * 21) - 1 * (120 * 5 + 23 * -26) 1 = 120 * -9 + 23 * 47
1 0 0 = (120 * 5 + 23 * -26) - 2 * (120 * -9 + 23 * 47) 0 = 120 * 23 + 23 * -120
(-9, 47)

Sunday, January 20, 2013

Extract Email Attachments With Python

If you archive your email messages, like me, you may find that you want to pull out all of the attachments for those files so your desktop search will parse them better, or so you can quickly search through them.

This is a simple script that just recurses through your .eml messages in a directory and pulls out all of the base64 encoded attachments.

For those of you that are wondering what base64 is, it's an encoding that only uses sixty-four different characters to transmit information. The email system uses this to send documents around so that the protocol didn't have to be reconfigured to account for stuff that wasn't text.

Code

#!/usr/bin/env python

import email.parser
import os
import sys
import base64


fileList = []
rootdir = "/path/to/.eml/messages/"
for root, subFolders, files in os.walk(rootdir):
for file in files:
fileList.append(os.path.join(root,file))

id = 0

for path in fileList:
if not path.endswith(".eml"):
continue

fp = email.parser.FeedParser()
fp.feed(open(path).read())

message = fp.close()

for message in message.walk():
fn = message.get_filename()
if fn == None:
continue

try:
with open(fn, 'wb') as out:
out.write(base64.b64decode(message.get_payload()))
except TypeError:
with open(fn, 'wb') as out:
out.write(message.get_payload())

Extensions

  • This script isn't very efficient being that it uses python to decode.
  • It would be nice to pull arguments from the command line using sys.argv

Update 2013-09-04 Python 3


#!/usr/bin/env python3

import email.parser
import os
import sys
import base64
import binascii
import sys


def extract(rootdir):
fileList = []

for root, subFolders, files in os.walk(rootdir):
for file in files:
fileList.append(os.path.join(root,file))

for path in fileList:
if not path.endswith(".eml"):
continue

fp = email.parser.BytesFeedParser()
fp.feed(open(path, "rb").read())

message = fp.close()

print("Checking {}".format(path))

for message in message.walk():
fn = message.get_filename()
if fn == None:
continue
try:
try:
with open(fn, 'wb') as out:
out.write(message.get_payload(decode=True))
except (TypeError, binascii.Error):
with open(fn, 'wb') as out:
print(message.get_payload())
out.write(bytes(message.get_payload(), message.get_charset()))
except Exception:
print("Error extracting item from {}".format(path))

if __name__ == "__main__":
if len(sys.argv) == 1:
print("usage: {} path/to/.eml/files".format(sys.argv[0]))
exit(1)
extract(sys.argv[1])

Tuesday, January 1, 2013

ODT Text Mining in Python

The ODT format is slowly gaining popularity, as governments worldwide begin to either use FOSS software, or transition to formats that are open enough to still be usable in at least a hundred years.

The Format

Within all .odt files (which are just zipfiles) there is an XML file that holds all of the document's text named content.xml. All of the characters between the XML tags are the document's text, so by cutting out all of the tags, all that remains is the text (what the script below does).

The Code (Python)

#!/usr/bin/env python3
'''

Copyright 2012 Joseph Lewis <joehms22@gmail.com> | <joseph@josephlewis.net>

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:

* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
* Neither the name of the nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

This software recieves an odt file from the command line, and returns the text
from its path.

'''

import zipfile
import xml.etree.ElementTree
import sys


def extract_odt_text(fp):
myFile = zipfile.ZipFile(fp)

share = xml.etree.ElementTree.fromstring(myFile.read('content.xml'))

text_nodes = []

for elt in share.iter():
if elt.text != None:
text_nodes.append(elt.text.strip())

return " ".join(text_nodes)

if __name__ == "__main__":
if len(sys.argv) == 1:
print "usage: odt.py FILENAME [FILENAME...]"
else:
for arg in sys.argv:
try:
print(extract_odt_text(arg))
except zipfile.BadZipfile:
pass

Sunday, December 30, 2012

DOCX Text Mining in Python and Java

In recent years Microsoft has been kind in making its formats easier to parse than binary lumps like the old .doc format. Instead, they now use XML within zip files with the extension .docx. This format can be quickly and easily parsed in Python (see my earlier xlsx parser.)

The Format

Within all new .docx files there is a folder named word and within this folder is a file named document.xml. This has a bunch of <w:t> tags, which hold text. To extract the text, just find those and pull out the contents!

The Code (Python)

#!/usr/bin/env python3
'''

Copyright 2012 Joseph Lewis <joehms22@gmail.com> | <joseph@josephlewis.net>

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:

* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
* Neither the name of the nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

This software recieves a docx file from the command line, and returns the text
from its path.

'''

import zipfile
import xml.dom.minidom
import sys


def extract_docx_text(fp):
myFile = zipfile.ZipFile(fp)

share = xml.dom.minidom.parseString(myFile.read('word/document.xml'))
text = share.getElementsByTagName('w:t')

return " ".join([node.childNodes[0].nodeValue for node in text])

if __name__ == "__main__":
if len(sys.argv) == 1:
print "usage: docx.py FILENAME [FILENAME...]"
else:
for arg in sys.argv:
try:
print(extract_docx_text(arg))
except Exception:
pass # not a valid zipfile, probably.

The Code (Java)

There is an issue with this Java implementation, but it should work for most purposes: special characters that are escaped in xml, like &amp; will not be unescaped in processing.
/**

Copyright 2012 Joseph Lewis <joehms22@gmail.com> | <joseph@josephlewis.net>

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:

* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
* Neither the name of the nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

This software recieves a docx file from the command line, and returns the text
from its path.

**/
import java.io.IOException;
import java.io.InputStream;
import java.util.Scanner;
import java.util.zip.ZipEntry;
import java.util.zip.ZipFile;

public class DOCXExtracter {
private static String convertStreamToString(java.io.InputStream is) {
try(Scanner s = new java.util.Scanner(is).useDelimiter("\\A")) {
return s.hasNext() ? s.next() : "";
}
}

public static String extractFromPath(String path) throws IOException
{
ZipFile zipFile = new ZipFile(path);
String ret = extractFromFile(zipFile);
zipFile.close();
return ret;
}

public static String extractFromFile(ZipFile f) throws IOException
{
ZipEntry e = f.getEntry("word/document.xml");
InputStream is = f.getInputStream(e);
String contents = convertStreamToString(is);
is.close();
return contents.replaceAll("(?s)\\<.*?\\>", " ");
}

public static void main(String[] args)
{
for(int i = 0; i < args.length; i++)
try {
System.out.println(extractFromPath(args[i]));
} catch (IOException e) {
e.printStackTrace();
}
}
}

A Python Hopfield Network

Quick Background

Hopfield networks are general purpose neural networks that take a set of input values (boolean) and produce an output of same length. They are particularly useful in signal processing and cleaning up images where noise is an issue, (think smudged characters).

I imagine they would be particularly useful in normalizing inputs from different hardware (e.g. creating color profiles, removing static or echos from some microphones, etc.)

This Implementation

This implementation is a tanslation of the algorithm found in the Clever Algorithms book from Ruby to Python.
#!/usr/bin/env python3
# A translation of http://www.cleveralgorithms.com/nature-inspired/neural/hopfield_network.html
import random

def random_vector(minmax):
return [row[0] + ((row[1] - row[0]) * random.random()) for row in minmax]

def initialize_weights(problem_size):
return random_vector([[-0.5,0.5] for i in range(problem_size)])

def create_neuron(num_inputs):
return {'weights' : initialize_weights(num_inputs)}

def propagate_was_change(neurons):
i = random.randrange(len(neurons))
activation = 0

for j, other in enumerate(neurons):
activation += other['weights'][i] * other['output'] if i != j else 0

output = 1 if activation >= 0 else -1
change = output != neurons[i]['output']
neurons[i]['output'] = output
return change

def flatten(nested):
try:
return [item for sublist in nested for item in sublist]
except TypeError:
return nested

def get_output(neurons, pattern, evals=100):
vector = flatten(pattern)

for i, neuron in enumerate(neurons):
neuron['output'] = vector[i]

for j in range(evals):
propagate_was_change(neurons)

return [neuron['output'] for neuron in neurons]

def train_network(neurons, patters):
for i, neuron in enumerate(neurons):
for j in range((i+1), len(neurons)):
if i == j:
continue

wij = 0.0

for pattern in patters:
vector = flatten(pattern)
wij += vector[i] * vector[j]

neurons[i]['weights'][j] = wij
neurons[j]['weights'][i] = wij

def to_binary(vector):
return [0 if i == -1 else 1 for i in vector]

def print_patterns(provided, expected, actual):
p, e, a = to_binary(provided), to_binary(expected), to_binary(actual)
p1, p2, p3 = ', '.join(map(str, p[0:2])), ', '.join(map(str, p[3:5])), ', '.join(map(str, p[6:8]))
e1, e2, e3 = ', '.join(map(str, e[0:2])), ', '.join(map(str, e[3:5])), ', '.join(map(str, e[6:8]))
a1, a2, a3 = ', '.join(map(str, a[0:2])), ', '.join(map(str, a[3:5])), ', '.join(map(str, a[6:8]))
print( "Provided\tExpected\tGot")
print( "%s\t\t%s\t\t%s" % (p1, e1, a1))
print( "%s\t\t%s\t\t%s" % (p2, e2, a2))
print( "%s\t\t%s\t\t%s" % (p3, e3, a3))


def calculate_error(expected, actual):
return sum([1 for i in range(len(expected)) if expected[i] != actual[i]])

def perturb_pattern(vector, num_errors=1):
perturbed = [v for v in vector]
indicies = [0 for i in range(random.randrange(len(perturbed)))]

while len(indicies) < num_errors:
index = random.randrange(len(perturbed))
if not index in indicies:
indicies.append(index)

for i in indicies:
perturbed[i] = -1 if perturbed[i] == 1 else 1

return perturbed


def test_network(neurons, patterns):
error = 0.0

for pattern in patterns:
vector = flatten(pattern)
perturbed = perturb_pattern(vector)
output = get_output(neurons, perturbed)
error += calculate_error(vector, output)
print_patterns(perturbed, vector, output)

error = error / float(len(patterns))
print("Final Result: avg pattern error=%s" % (error))

return error

def execute(patters, num_inputs):
neurons = [create_neuron(num_inputs) for i in range(num_inputs)]
train_network(neurons, patters)
test_network(neurons, patters)
return neurons

if __name__ == "__main__":
# problem configuration
num_inputs = 9
p1 = [[1,1,1],[-1,1,-1],[-1,1,-1]] # T
p2 = [[1,-1,1],[1,-1,1],[1,1,1]] # U
patters = [p1, p2]
# execute the algorithm
execute(patters, num_inputs)

Sunday, January 15, 2012

INITV Style Startup Script in Python

Projects like Upstart and systemd are incredibly nice with the ease that they provide for creating system startup processes, but nothing can beat initv for portability and widespread use/knowledge.

The major problem with most init style scripts is they run lots of processes and can be a mess to get working the first time: here is a plug n' play solution, just replace the ARGS with the program you want to run and the program's arguments. In the provided example, it opens chromium (the open source version of Google Chrome) in kiosk mode to this site (press Alt + F4 to exit).

#!/usr/bin/env python
'''
2012-01-15 Joseph Lewis <joehms22@gmail.com>

Provides init-like handling of a program.
'''

import os
import sys
import signal
from subprocess import Popen


ARGS = ["chromium-browser", "--kiosk", "http://onehourhacks.blogspot.com"]
PIDFILE = "/tmp/%s.pid" % (ARGS[0])


try:
cmd = sys.argv[1]
except IndexError, e:
print("Usage: %s (start|stop|restart)" % (sys.argv[0]))
exit(2)


def start():
try:
j = open(PIDFILE)
j.close()
print("Already running!")
return
except IOError:
pid = Popen(ARGS).pid
j = open(PIDFILE, 'w')
j.write(str(pid))
j.close()


def stop():
try:
j = open(PIDFILE)
os.kill(int(j.read()),signal.SIGTERM)
j.close()
os.remove(PIDFILE)
except IOError, e:
print("Cannot stop: the program is not running")
except OSError, e:
os.remove(PIDFILE)
print("The program has quit before it was killed")


# Very simple stuff here, initv style!
if cmd.lower() == "start":
start()

if cmd.lower() == "stop":
stop()

if cmd.lower() == "restart":
print("Restarting")
stop()
start()

Saturday, October 29, 2011

.exe file String extractor.

I recently had the need to quickly and dirtily find some strings from a Windows .exe file on Linux.

Going through with HexEditor was a pain in the arse. There seemed to be a few programs for this on windows, which would work messily under Wine. It was easier to write my own.

The Rules:
  • Strings must be ASCII.
  • Strings must be greater than 4 characters.
  • Strings must be at least 50% different characters. (i.e. www.com.com fails, while www.google.com succeeds)
This still leaves you with a lot of junk, but at least you can easily see what is junk and not.

#!/usr/bin/env python
'''A program that extracts strings from binary files.

Usage: reader.py path/to/file
'''

import sys

numstrs = 0
with open(sys.argv[1]) as j:
j = j.read()
mystr = ""
for i in range(0,len(j)):
if 31 < ord(j[i]) < 127:
mystr += j[i]
else:
# If the string isn't that long discard it.
if len(mystr) > 4:
uniqs = set()
for char in mystr:
uniqs.add(char)

# If duplicate chars are less than half the string
if len(uniqs) > .5 * len(mystr):
print mystr
numstrs += 1

mystr = ""

print "==========\n\nOutput: %s strings" % numstrs

Some of you may be thinking, why do you care? Well, this is a great way to quickly see the human interactions going on within a binary: what libraries are being loaded, what websites are being accessed, what programs are being called, or even give you cheats by allowing you to look at some dialogs not normally shown.

Sample: SimAnt
Extended sections of junk have been replaced by "...".


...
Copyright (C)1990, Daniel Goldman
...
96|yr
y, <Xw
<st'<ntQ<pt`<Et
...
eov0001:
Cannot find overlay file "
eov0002:
eov0003:
eov0007:
Commit error - section
eov0009:
eov0004:
-- use RELOAD to increase size
...
Expanded Memory
Extended Memory
Conventional Memory
eca0001:
.RTLink CACHE -
Handling
Function code
, Error code
RTVMEXP
RTVMEXT
RTVMCONV
RTOVEXP
RTOVEXT
RTOVCONV
eov0010:
evm0019:
Environment syntax error --
(last char is erroneous)
Fatal Error $
Press any key to terminate.$PQWV
...
pError!!
pNOTE: SimAnt
runs radically
p faster in 16-color mode!
pPlease refer to the
pSimAnt
manual.
pSave Game As:
pThe name 'SimAnt
' is reserved
pfor this application.
...
"'*'%"
pSimAnt
Saved Game
how to edit/create one.
HIGHLIGHTED
CHITIN
MAXILLAE
OCELLI
INFRABUCCAL
MIDGUT
TROPHALLAXIS
CASTE
LARVAE
PUPATE
CASTES
PUPAE
ECLOSE
CALLOWS
PHEROMONES
BROOD
HONEYDEW
APHIDS
BREEDERS
RECRUIT
FORAGE
FORAGING
QUEENCHAMBER
NURSERIES
MS Run-Time Library - Copyright (c) 1990, Microsoft Corp
...
DACsample
FREELIST > 38
winheaders
Sound Driver reports driver: %d
%d DAC channels
%d FM channels
%d TANDY PSG channels
%d PS1 PSG channels
%d COVOX PSG channels
and %d MIDI channels
MUSIC INIT!
UPDATEEVERYTHING!
WAIT YES
result=%d
Yardmode=%d, newMode=%d
i=%d, j=%x
= %x
EMS version: %x
total pages: %d
free pages: %d
TILESXXX
emstiles
Tile >= 256 in PutLifeAndTile
Tile >= 256 in PutLifeAndTile
PutTile grndTile > 0x100
Y>MAXAY
SimAnt
Vis message %s at %d,%d
balbuf
Cannot GetResource (HEX, %d), ResErr %d
MemDeath
Warning: Can't load strings: %d
strptrs
NewPtr argument too large
%-ld%%
BAD SWITCH
USAGE: SimAnt [/d{EeTHMm2V}] [/s{NABCI}]
SimAnt configuration file missing
simant.cfg
install.exe
DOS Error %d: %s
sound
SimAnt cancelled.
BAD font in MakeBaloon
balloon
@tdyballoon
clrbln
Balloon Size=%d, %d
ePtr->mouse_flags=%x, MRPRESSED=%x
Map window image
Generated map window
ITEM=%x, startTool=%x
tileNum=%d
ANTMENU
L: item=%x, openSub=%x
E=%x, openSub=%x
Last sub state = %x
X: item=%x, openSub=%x
~zwsolhea^[
<@DHLPTX\`dhlptx|
[^aehloswz~
|xtplhd`\XTPLHD@<
TPLHD@<84.,
,.48<@EINRV
<@DIII]c
]IID@
CHEAT %d
Load Game
game not loaded
CITYMCRP
Save Game
lastFileName==%s
OVERWRITE
TOTALLEN=%u
COPYING DATA INTO BUFFER!
WRITING
Saved correctly
File list
Cannot read drive %c
Can't read disk
Aborting
Cannot read drive %c
<PATH=%s>
%s*.*
*.ant
%-12s
0No Files
%d:%s
-> %s
Event=%x,%x
DEFAULT:Event=%x,%x
%c:%s
CHDIR(%s)
Filename:%s
PathName=%s, iniPath=%s
:;,.=+-_\/*
Unpause
Pause
FREE+SOFT AT INITMAP=%ld
Generated buffer window
Map window image
Generated map window
Draw map - yard
YMAXCOUNT =%d!!!!!
GOT Scenario %x
StrnID=%d
Pict=%d
line %d:%s
PictStrnDialog: %d
MePLane=%d
Malloc
()*+,-./0
1234567
EVENT=%x
FLUSH
User Request
Keypress=%x
COMMAND KEY:%c
%ld bytes @ startup
%ld bytes free now
%ld bytes discardable stuff
%ld AVAILABLE
KEYEVENT=%x, %x
ePtr->mouse_flags=%x
MapPlane=%d
len = %d
%d(%d)(%d): [%s]
ERROR: Cannot file text rez(%s)
DisplayCard(cardRez)(%d)
Card Resource not found.
INFO TEXT REC: %d
Unknown Resource.
AnimYellowInsane
MemPunt @%s after %s
RALLOCXX
DISCARDED
SYSTEM
Free illegal type
ralloc.dmp
Ralloc dump at %s, %s
By handles:
By location:
%p->%p:
size=%x, %ld, type=%c%s
age=%5ld name=%s
DiscardEntry
ALL MEMORY HANDLES USED
OLDESTH=%p
<SPLIT>
<WHOLE>
Cannot find a big enough space! (%ld bytes)
<FO2>
ALLOC 0 BYTES
NOEMS - memPtr=%p
RL5: Invalid handle index
LOCKED FREED %s
RL5: Invalid handle index
REALLOC 0 BYTES
{DIS}
Lock depth exceeded 100
Locked discarded!!
RU: Bad handle in unlock
RL5: Invalid handle index
malloc
_ffree: Could not find memory ptr
_frealloc: Could not find memory ptr
EMS Error
%s.ndx
Index file missing
WINLABEL
BITMAP
CSRMASK
CSRPIC
ANIMDLT
SCREEN
PALETTE
CARDTITLE
STYLE
U%s,%d
%d:%d
DBRecall Unpack error!!! - object=%d, type=%d
Attempt to ADD during a READ-ONLY run
Attempt to DELETE during a READ-ONLY run
Attempt to PACK during a READ-ONLY run
DBEMSXXX
DBEMSLIST
Handle mismatch
Database error
Out of handles.
%s.dat
Cannot create data file.
for more information.
Dos error: %d: %s
%s.dat
Cannot open database %s
Purge attempt with database closed
Release %d not in cache!
Unhook attempt with database closed
cachetable
no memory over 0 in age!
MouseHide < 0
HPKMGOIQLRS9;
HotBox overflow
Continue
Continue
%s: Are you sure?
Cancel
Retry
Cancel
Unknown unit
Drive not ready
Unknown command
Data error (bad CRC)
Bad request structure length
seek error
unknown media type
sector not found
printer out of paper
write fault
read fault
general failure
abort request
GSaveRect
Color picture in mono file
PutPackedBuf
GPutPacked
PutPackedBuf
WARNING: i>=MAXWINDOWS-1 in KillWin(%d)
SetWin: Window open but clip is NULL
ERROR MEMNULL winClipHandle
tmprects
clipout
winClipList
C097: Clip overflow %d
C098: Clip overflow %d
winClipList
C099: Clip overflow %d
subinclude
CL074:Temp clip overflow in SubInclude
Sub include NULL rect!!
Clip out of memory!!!
subinclude
CL074:Temp clip overflow in SubInclude
Sub include NULL rect!!
Clip out of memory!!!
subexclude
CL074:Temp clip overflow in SubExclude
Sub exclude NULL rect!!
Clip out of memory!!!
include
CL174:Temp clip overflow in SubExclude
include NULL rect!!
clip_Push
Cannot allocate memory in clip_Push
Error CL98463: Pushed it too far
Clip_Pop w/ nothing on the stack: E9073
Could not find fonts
Could not find fonts!
Font loaded!
Could not find fonts
Could not find fonts!
FATAL: %s
%%-%ds
%%c%%-%ds
Menu data too long
MSMOUSE
language.dat
language
shared
lrshare
Cannot load menu
bmcmBad 'Display Mode' in configuration file
Hires EGA
Tandy
Hercules
Lores EGA
Mono EGA
MCGA/VGA Color
MCGA/VGA mono
VGA Color
hcega
tdyga
lcega
hcega
MDA system detected. Cannot run graphics.
Couldn't load resource %x,%x
CANNOT LOAD WINDOW %03x
Cannot load resource
please try another
Could not load purge list
!!W:%x
QQW:%x
##W:%x
**W:%x
w:%x, i:%x
xxw:%x, i:%x
zzw:%x, i:%x
ppw:%x, i:%x
%%w:%x, i:%x
!!i:%x
@@i:%x
formatStr
formatStr
Illegal win num %x at lock
win_Lock > 10 levels deep!!
Attemp to unlock when discarded win %x
Bad object type in win_DrawElevator
eleTop= %d
PrevListLine punt
SS=%d, line=%d
Bad object type in win_ProcSliderEvent
fontBuf
fontBuf
fontBuf
font1
font2
font3
font4
WINDOW %x NOT LOCKED DURING CALL TO win_WinRectAddr!!
WINDOW %x NOT LOCKED DURING CALL TO win_ObjAddr!!
GGetPic - trans @ %d, %d (%d, %d), bytes=%u
GPutPic size AA %d, %d, tag=%x
GPutPic size %d, %d, tag=%x
Couldn't find resize icoN!!
FontHeader
FONTIMAGE
locTable
owTable
Printset @ %s %d objects
Buffer in set too small!!
animbufs
anim_Remove objects not found
anim_hide object not found
anim_hide object not found
anim_Remove objects not found
animHandle
animobjs
to flush
Bad pic type in ANIM.C
...
music failure
Seg=%x, buf=%p, handle=%p
_C_FILE_INFO=
0PX
000WP
(null)
USunMonTueWedThuFriSat
JanFebMarAprMayJunJulAugSepOctNovDec
Error 0
No such file or directory
Arg list too long
Exec format error
Bad file number
Not enough core
Permission denied
File exists
Cross-device link
Invalid argument
Too many open files
No space left on device
Math argument
Result too large
Unknown error
...
P<<NMSG>>
R6000
- stack overflow
R6003
- integer divide by 0
R6009
- not enough space for environment
run-time error
R6002
R6001
- null pointer assignment
...
06/13/90
==========

Output: 828 strings

It is interesting looking at some of the common problems that plagued early computer programs.

Friday, September 30, 2011

Conway's Game of Life

Conway's Game of Life is a classic computer game invented to simulate a self-replicating machine on a very simple grid. I decided to write a simple implementation, it is after the break.

Just as in life, simple structures form to generate complex behaviours in these simple systems that can be generated very quickly on modern computers.

The unofficial open source icon is even inspired from the game:

The "Glider" a structure that "flies" around in the game.



The program in action.




This is a simple implementation of the Game of Life:

#!/usr/bin/env python
''' Conway's game of life in python.

Copyright 2011 Joseph Lewis MIT License

1. Any live cell with fewer than two live neighbours dies.
2. Any live cell with two or three live neighbours lives.
3. Any live cell with more than three live neighbours dies.
4. Any dead cell with exactly three live neighbours becomes alive.
-- Wikipedia.
'''

import random
import time


ROWSCOLS = None # Number of rows and columns in the game.
grid = None # A grid with 0s and 1s, 0 is no cell, 1 is a cell.

def printgrid():
'''Prints out the given grid after blanking the screen.'''

print 10 * "\n"
print "+"+ "-" * ROWSCOLS + "+" # Border top

for r in grid:
line = "|" # Border Left
for c in r:
if c:
line += "0" # Cell
else:
line += " "
print line + "|" # Border Right

print "+"+ "-" * ROWSCOLS + "+" # Border Bottom


def do_life():
'''calculates the next grid'''

grid_len = range(0,ROWSCOLS)
grid2 = [[0 for i in grid_len] for i in grid_len]

for r in grid_len:
for c in grid_len:
if grid[r][c]:
increment(grid2, r, c)

for r in grid_len:
for c in grid_len:
g = grid2[r][c]
if g < 2 or g > 3:
grid[r][c] = 0
elif g == 3:
grid[r][c] = 1


def increment(grid, row, col):
'''Increments the life count for all cells around this one.'''
for r,c in [(row, col+1), (row, col-1),
(row+1, col), (row-1, col),
(row+1, col+1), (row-1, col-1),
(row+1, col-1), (row-1, col+1)]:
grid[r % ROWSCOLS][c % ROWSCOLS] += 1


def start_life():
'''Randomly fills the game.'''
for i in range(0, ROWSCOLS**2 / 4):
r = random.randint(0, ROWSCOLS-1)
c = random.randint(0, ROWSCOLS-1)
grid[r][c] = 1


if __name__ == "__main__":
try:
print("=Conways Game of Life=")
k = int(input("How many turns should I run before I reset (200 is nice)? "))
ROWSCOLS = int(input("How many rows and columns of life should there be (35 is nice)? "))
t = float(input("How many frames per secould should I run (10 is nice)? "))

except Exception:
print("Hey! Play nice, enter a number!")
exit(1)

grid = [[0 for i in range(0, ROWSCOLS)] for i in range(0, ROWSCOLS)]

j = 0
while True:
if not j:
start_life()
j = k # Number of turns to run before restart.
j -= 1
do_life()
printgrid()
time.sleep(1/t)

Tuesday, September 27, 2011

Automatic Summary of a Document

It is the primary dogma of computer science that computer scientists are lazy. That is why they work with tiny stupid slaves all day, forcing them to do their bidding. Another strange and seemingly unrelated fact is that reading documents and providing summaries is mind numbingly boring.

Some years ago the folks at Microsoft made a little addition to their word processor that would allow it to automatically generate summaries. This feature is little used and little needed. However it is still fun, and could have the potential to provide some nice summaries to sites like Manybooks.net which would definitely benefit from some documents of theirs (especially lesser known books) being summarized.

Thinking about having your computer generate nice summaries and actually having it do them are two entirely different things it turns out. However in more structured documents, such as non-fiction some simple math can work out the problem of which sentences are important and which are not.

Before we can worry about that, let's first find the important words in a document. One of the better ways, is simply counting every occurrence of every word, then removing fairly common words; I removed the top 100 in the example code given by setting their scores to -1, although different values can be tried to attempt to prove a sentence better or not. For example the sentence, "Call me Ishmael." would likely score rather low, though it is one of the most known sentences of all time. If the word "me" were set to a higher value (as it is in the top 100) the sentence may be included in a summary. 

Sentences can then be split up at punctuation marks (. ! ?) and a total for the sentence can be generated by adding all of the word scores and dividing by the number of words in the sentence, thus creating an average word score for the sentence.

Once all sentences have been scored, the top ones are taken out, and put in order again, thus preserving some of the structural integrity of the original document, then can be returned.

In a few of my tests, this was not enough however. I also added a multiplier that took in to account the average length of words in the sentence; as either main points are generally made well, or supporting evidence doesn't get its credit.
Talk is cheap, show me the code!

#!/usr/bin/env python
# -*- coding: utf-8 -*-
'''Provides An Automated Summary of a document.

Copyright 2011 Joseph Lewis

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:

* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
* Neither the name of the nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
'''

import sys
import re
import argparse

__author__ = "Joseph Lewis"
__copyright__ = "Copyright 2011, Joseph Lewis"
__license__ = "BSD"

TOP_HUNDRED_SCORE = 0
EN_TOP = '''the be to of and a in that have i it for not on with he as you do at
this but his by from they we say her she or an will my one all would
there their what so up out if about who get which go me when make can
like time no just him know take person into year your good some could
them see other than then now look only come its over think also back
after use two how our work first well way even new want because any
these give day most us'''


text = """Fourscore and seven years ago our fathers brought forth upon this continent a new nation, conceived in liberty, and dedicated to the proposition that all men are created equal. Now we are engaged in a great civil war, testing whether that nation, or any nation so conceived and so dedicated, can long endure. We are met on a great battlefield of that war. We have come to dedicate a portion of that field as a final resting-place for those who here gave their lives that that nation might live. It is altogether fitting and proper that we should do this. But in a larger sense we cannot dedicate, we cannot consecrate, we cannot hallow this ground. The brave men, living and dead, who struggled here, have consecrated it far above our poor power to add or detract. The world will little note, nor long remember, what we say here. But it can never forget what they did here. It is for us, the living, rather to be dedicated here to the unfinished work which they who fought here have thus far so nobly advanced. It is rather for us to be here dedicated to the great task remaining before us, that from these honored dead we take increased devotion to that cause for which they gave the last full measure of devotion; that we here highly resolve that these dead shall not have died in vain; that this nation, under God, shall have a new birth of freedom, and that government of the people, by the people, and for the people, shall not perish from the earth. """


def chars_per_word(sentence):
'''Returns the average characters per word.'''
return float(len(sentence)) / float(len(sentence.split(' ')))

def clean_text(text,leavepunct=False):
# Strip to a-z A-Z (TODO: I18N).
text = text.replace("\n", " ")
if leavepunct:
return re.sub("[^a-z.!\? ]", " ", text.strip().lower())
return re.sub("[^a-z ]", " ", text.strip().lower())

def word_frequency(text):
'''Counts the frequenc of words in the piece of text, and returns
a dict for each word (a-z) lowercased where the key represents the
word and the number of times the word is found the value.

'''
words = {}


tmp = clean_text(text)

# Cut to words.
for word in tmp.split():
if word in words:
words[word] += 1
else:
words[word] = 1

return words

def set_words_to_value(word_dict, top=EN_TOP, value=TOP_HUNDRED_SCORE):
'''Sets the given words to the given value in the given word_dict.
The default is to set the top hundred words in english to the
TOP_HUNDRED_SCORE.

'''

j = word_frequency(top).keys() # get the words in the top hundred text.
# remove the top 100 words
for w in j:
words[w] = value

def sentences_in(text):
'''Returns a list of sentences in the text.

'''
text = text.replace("\n", " ")
return re.split('[\?.!]', text)


def score_sentence(sentence, words):
'''The scoring function, given a dictoinary of word:value pairs,
creates a score for each sentence.

'''
# Score value based upon words and frequencies of those words.
tmp = clean_text(sentence)
total = 0

for word in tmp.split():
if word in words:
total += words[word]

# Make the total in to a percentage.
try:
total /= float(len(tmp.split()))

# Secret ingredient, higher characters per word generally means
# more important sentence.
total *= chars_per_word(tmp)

return total
except ZeroDivisionError:
return -100


def top_sentences(text, word_freq_dict):
'''Returns a sorted list with the top rated sentences first, that
contains the tuples (score, sentence_location, sentence_text)

For example, the sentence "Call me Ishmael" would come back:
(1.8304283, 0, "Call me Ishmael.")

0 would mean it was the first sentence in the text, and it had a
score of 1.83...

'''
sentences = [] # array of tuples (total score, sentence num, sentence text)
known_sentences = set()
currs = 0
for s in sentences_in(text):
currs += 1 # Increment the current sentence.
total = score_sentence(s, words) # Don't add duplicate sentences.

s = s.strip()
if s not in known_sentences:
sentences.append((total, currs, s+"."))
known_sentences.add(s)

# Sort highest rated sentences to lowest.
sentences = sorted(sentences)
sentences.reverse()

return sentences


def __combine_summary(summary):
'''Combines a summary in to a meaningful paragraph. A summary is an
array of tuples with values of (position of sentence, text). This
creates better summary paragraphs by ordering important sentences
in the order they were originally.

'''
summary = sorted(summary)

paragraph = ""
for l, s in summary:
paragraph += s + " "

return paragraph

def percentage_top(percentage, sorted_sentences):
'''Returns the top rated percentage of the given sentences, in
paragraph form.

i.e. to get the top 25 percent of a text, call with:
percentage_top(25, )

'''

percentage = percentage / 100.0

num_sentences = int(len(sorted_sentences)*percentage) + 1

if num_sentences >= len(sorted_sentences):
num_sentences = len(sorted_sentences)

# Create summary (top x sentences)
summary = []
for j in range(num_sentences):
t,l,s = sorted_sentences[j]
summary.append((l,s))


return __combine_summary(summary)



def top_words(num_words, sorted_sentences):
'''Returns x number of words from the top sentences.'''

summary = []
words = 0
try:
for t,l,s in sorted_sentences:
if words >= num_words:
break

words += len(s.split())
summary.append((l,s))


except IndexError:
pass

return __combine_summary(summary)


if __name__ == "__main__":

parser = argparse.ArgumentParser(description='Creates summaries from documents by magic.')
parser.add_argument('-p','--percent', type=int, default=None,
help='the percent of the original document to give as a summary')
parser.add_argument('-w', '--words', type=int, default=None,
help='the number of words to output as a summary')

parser.add_argument('PATH', help='the path of the textfile to read from')

args = parser.parse_args()

try:
if args.PATH:
text = open(args.PATH).read()
words = word_frequency(text)

set_words_to_value(words)
sentences = top_sentences(text, words)

if args.words:
print top_words(args.words, sentences)
if args.percent:
if args.words:
print "\n\n"
print percentage_top(args.percent, sentences)

except IOError:
print "File given can't be found."

Monday, September 19, 2011

XLSX Parsing in Python

A while ago I had to throw together a program that read from .xslx files. Knowing essentially nothing about them, other than the fact that they were xml files in a zip, I was able to hack together something that worked kind of like the default CSV reader in Python. For every row in the file, it returns an array of cell contents.

In this case only Text and Numeric cells will render properly, formulas and such won't work. Also, the only sheet read is sheet1.

Okay, a little about the file structure of xlsx: Sheet info is stored in xl/worksheets/sheetX.xml, which looks essentially like this:
<sheetdata>
<row collapsed="false" customformat="false" customheight="false" hidden="false" ht="12.8" outlinelevel="0" r="1">
<c r="A1" s="0" t="s"><v>0</v></c>
<c r="B1" s="0" t="n"><v>123</v></c>
</row>
</sheetdata>

Cells (c tags) have several types (denoted by the t attribute) "s" is a string, and "n" is a number. The string's attribute is an index for a string in the xl/sharedStrings.xml file, which removes duplicate strings from the saved data.


''' Opens .xlsx files as if they were .csv files. '''
import zipfile
import xml.dom.minidom
import re

class XLSXReader:
rows=[]

def _nodeText(self, node):
return "".join(t.nodeValue for t in node.childNodes if t.nodeType == t.TEXT_NODE)

def _get_col_num(self, col):

strpart = col.attributes['r'].value
colnum = re.sub('[^A-Z]', '', strpart.upper().strip())

c = 0
for char in colnum:
c += ord(char)

c -= (65) # ASCII to number

print("Colnum for '%s' is %s" % (strpart, c))

return c


def __init__(self, filename):
shared_strings = []
self.rows = []
myFile = zipfile.ZipFile(filename)

# Read the shared strings file.
share = xml.dom.minidom.parseString(myFile.read('xl/sharedStrings.xml'))
j = share.getElementsByTagName("t")

for node in j:
shared_strings.append(self._nodeText(node))

sheet = xml.dom.minidom.parseString(myFile.read('xl/worksheets/sheet1.xml'))
sheetrows = sheet.getElementsByTagName("row")
for row in sheetrows:
cols = row.getElementsByTagName("c")

largest_col_num = 0
for col in cols:
colnum = self._get_col_num(col)
if colnum > largest_col_num:
largest_col_num = colnum

thiscol = ['']*(largest_col_num + 1)

for col in cols:
value = ""
try:
value = self._nodeText(col.getElementsByTagName('v')[0])
except IndexError:
continue

#Get col number (A=0, B=1, etc. up to AA)

colnum = self._get_col_num(col) # ASCII to number

try:
if col.attributes['t'].value == 's':
thiscol[colnum] = shared_strings[int(value)]
else:
thiscol[colnum] = value
except KeyError:
thiscol[colnum] = value
self.rows.append(thiscol)

myFile.close()

def __getitem__(self, i):
return self.rows[i]

To use the code, call the class with a file name, then use a for - each loop to iterate over the rows.

Update 2012-09-14: Mati noted that in some situations the xlsx format didn't have the proper attribute to indicate a number, and sent in a patch. Thanks!