Showing posts with label Java. Show all posts
Showing posts with label Java. Show all posts

Wednesday, April 17, 2013

Things I Learned From Working A 2 Year 50,000 Line Project

This is mostly to serve as a brain dump for a piece of software I've been writing for around two and a half years, it was comprised of a large database (read: ~five hundred million rows), a cross platform Java application, a large Python web-server/API, a phone app, lots of small scripts that fetched and transformed data, and globs of glue. (I wrote all of them in the two year period!)

Architecture

  • It pays to fully think through your APIs first
  • All web APIs should have a version in the URL e.g. /application/api/2.0/...
  • Profile early and often
  • A few good caches are worth many bad ones
  • Make everything a plugin that extends one class; you may think it's a big hairy mess, but it's better than multiple plugin systems.
  • Use libraries wherever possible
  • Ditch libraries when you only use one function from them

Debugging

  • Users write awful bug reports
  • You can help them write better ones by giving them a short form to fill out.
  • Automatically attach debugging info to all bug reports
  • Make version numbers easily available (not just in the executable's name)
  • Auto-update your apps so you don't have users lagging behind

Upgrades

  • Your requirements will constantly change, make sure you have an easy upgrade path in your database that won't force hacky solutions.
  • Make sure your ORM supports incremental changes.

User Experience

  • Start fast, even if you're not done processing yet
  • Never trust an engineer to do design (including software engineers)
  • If the user doesn't see it, it doesn't exist
  • Make it very hard for the user to do stupid things
  • Do automatic backups/saves

Sanity

  • Clients like saying "I did say that, didn't I? What I meant was..."
    • all features should be plugins so you can enable/disable when they do this
  • SVN and Eclipse are a nightmare to get playing nice
  • Java 7 is really awesome!
    • auto-closing streams
    • multiple catches in a try block
    • string in switch statements
  • JNLP sucks, but so does embedding apps in-browser; just make it easy for the user to download it, and provide auto upgrading

Monday, April 1, 2013

HowTo Calculate an MD5 hash in Java

If you're dealing with networking, or just want to make sure your files don't get corrupted/modified between times your app runs.

It is critical to use the algorithm.reset() command before using a MessageDigest instance, as if there are bytes still remaining from a prior digest, it will completely ruin your output.

Code

import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;

// calculates an MD5sum for a given bit of text
public class MD5
{
public static void main(String[] args)
{
try
{
MessageDigest algorithm = MessageDigest.getInstance("MD5");
algorithm.reset(); // clear the digest back to starting state.

// Get selected digest of the given text.
byte[] digest = algorithm.digest( "Hello, world!".getBytes() );

System.out.println(byteArrToHex(digest));
}
catch (NoSuchAlgorithmException e)
{
// no such algorithm, MD5
}
}

// converts bytes to text.
private static String byteArrToHex(byte[] bytes)
{
StringBuilder sb = new StringBuilder();
for(byte b : bytes)
sb.append(String.format("%02X", b));
return sb.toString().toLowerCase();
}
}

Other Notes

  • To calculate SHA1 hashes instead, use SHA1 as the instance type.
  • The NoSuchAlgorithmException likely won't be thrown on desktop platforms, but it is possible that certain algorithms won't be implemented in certain areas due to them being considered military grade encryption; SHA1 and MD5 shouldn't have this problem because they are just simple hashes.

Tuesday, February 12, 2013

HOWTO Make a Quick and Dirty (Pseudo)Random Number Generator

If you ever need to build your own quick and dirty (pseudo-)random number generator it actually isn't that hard. This generator is used in various real-world systems:
  • Visual Basic (to version 6)
  • A few ANSI C implementations, including glibc
  • Java's Random library
The implementation will iterate over all numbers [0,m) when using proper a, m ,c's. 1

Implementation

#include <stdio.h>
#include <unistd.h>

const long m = 4294967296; // 2 ^ 32
const long a = 1103515245;
const long c = 12345;

long lastX = 0;

long nrandom() {
lastX = (a * lastX + c) % m;
return lastX;
}

void seed(int num)
{
lastX = num;
}

int main()
{
int i;

seed(getpid());

for(i = 0; i < 1000; i++)
printf("%li\n", nrandom());
}
Note that this implementation only generates m pseudorandom numbers, in this case 4,294,967,296; so if you need more, you'll want to use a different algorithm.

  1. These conditions require some discrete mathematics that are too long to go in to here; it involves coprime numbers and recurrence relations. (Please point out if I've missed some simple explaination in the comments, I'm fairly new to this!)↩

Sunday, December 30, 2012

DOCX Text Mining in Python and Java

In recent years Microsoft has been kind in making its formats easier to parse than binary lumps like the old .doc format. Instead, they now use XML within zip files with the extension .docx. This format can be quickly and easily parsed in Python (see my earlier xlsx parser.)

The Format

Within all new .docx files there is a folder named word and within this folder is a file named document.xml. This has a bunch of <w:t> tags, which hold text. To extract the text, just find those and pull out the contents!

The Code (Python)

#!/usr/bin/env python3
'''

Copyright 2012 Joseph Lewis <joehms22@gmail.com> | <joseph@josephlewis.net>

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:

* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
* Neither the name of the nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

This software recieves a docx file from the command line, and returns the text
from its path.

'''

import zipfile
import xml.dom.minidom
import sys


def extract_docx_text(fp):
myFile = zipfile.ZipFile(fp)

share = xml.dom.minidom.parseString(myFile.read('word/document.xml'))
text = share.getElementsByTagName('w:t')

return " ".join([node.childNodes[0].nodeValue for node in text])

if __name__ == "__main__":
if len(sys.argv) == 1:
print "usage: docx.py FILENAME [FILENAME...]"
else:
for arg in sys.argv:
try:
print(extract_docx_text(arg))
except Exception:
pass # not a valid zipfile, probably.

The Code (Java)

There is an issue with this Java implementation, but it should work for most purposes: special characters that are escaped in xml, like &amp; will not be unescaped in processing.
/**

Copyright 2012 Joseph Lewis <joehms22@gmail.com> | <joseph@josephlewis.net>

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:

* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
* Neither the name of the nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

This software recieves a docx file from the command line, and returns the text
from its path.

**/
import java.io.IOException;
import java.io.InputStream;
import java.util.Scanner;
import java.util.zip.ZipEntry;
import java.util.zip.ZipFile;

public class DOCXExtracter {
private static String convertStreamToString(java.io.InputStream is) {
try(Scanner s = new java.util.Scanner(is).useDelimiter("\\A")) {
return s.hasNext() ? s.next() : "";
}
}

public static String extractFromPath(String path) throws IOException
{
ZipFile zipFile = new ZipFile(path);
String ret = extractFromFile(zipFile);
zipFile.close();
return ret;
}

public static String extractFromFile(ZipFile f) throws IOException
{
ZipEntry e = f.getEntry("word/document.xml");
InputStream is = f.getInputStream(e);
String contents = convertStreamToString(is);
is.close();
return contents.replaceAll("(?s)\\<.*?\\>", " ");
}

public static void main(String[] args)
{
for(int i = 0; i < args.length; i++)
try {
System.out.println(extractFromPath(args[i]));
} catch (IOException e) {
e.printStackTrace();
}
}
}

Sunday, December 23, 2012

Tiny Embeddable Java Webserver

This is a tiny embedded Java webserver that'll serve pages from a local drive or a .jar (if it is within the jar file).
While it may not be completely useful to do this a very easy extension would include a Map from Strings to Objects, and call the .toString() method on the object and use that as output.
By creating something like this, you would get a simple CGI like server that you could use to write difficult GUI code, using styling from CSS and Javascript.

Source Code

/**
* A local embedded webserver, will run from a .jar file, and serve files
* from the root.
*
* @author Joseph Lewis <joehms22@gmail.com>
*/

import java.io.BufferedOutputStream;
import java.io.DataInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
import java.io.PrintStream;
import java.net.ServerSocket;
import java.net.Socket;
import java.util.NoSuchElementException;
import java.util.StringTokenizer;


public class Webserver extends Thread
{
private static final FileNotFoundException FILE_NOT_FOUND = new FileNotFoundException();
private static final byte[] ERROR_404 = "HTTP/1.0 404 Not Found\r\nContent-type: text/plain\r\n\r\n404 Error, page not found.".getBytes();
private static final int PORT_NUM = 8_000;
public static Webserver SERVER = null;
private ServerSocket serverSocket;

public static Webserver getInstance()
{
// Make sure we only ever have one server running.
if(SERVER == null)
SERVER = new Webserver();

return SERVER;
}

private Webserver()
{
try
{
serverSocket = new ServerSocket(PORT_NUM);

System.err.println("Starting server on port: " + PORT_NUM);

// If you're running this embedded, you will probably want to
// make the thread a daemon so it quits when your main program
// quits rather than serves forever.
//setDaemon(true);

start();

} catch (IOException e) {
e.printStackTrace();
}
}

@Override
public void run()
{
// Starts a new thread for every request we get.
while (true)
try {
new RequestHandler(serverSocket.accept());
} catch (Exception e) {
e.printStackTrace();
}
}

public static void main(String[] args)
{
Webserver.getInstance();
}


private class RequestHandler extends Thread
{
private final Socket socket;
private final ClassLoader m_loader;
private static final String DEFAULT_INDEX_PAGE = "index.html";

// Start the thread in the constructor
public RequestHandler(Socket s) {
socket = s;
m_loader = Webserver.getInstance().getClass().getClassLoader();
start();
}

@Override
public void run()
{
try
{
DataInputStream in = new DataInputStream(socket.getInputStream());
PrintStream out = new PrintStream(new BufferedOutputStream(socket.getOutputStream()));


// Parse the HTML request
StringTokenizer st = new StringTokenizer(in.readLine());

try
{
String requestType = st.nextToken();
String filename = normalizeFilePath(st.nextToken());

System.err.println("Request: " + requestType + " " + filename);


InputStream f = m_loader.getResourceAsStream(filename);

if(f == null)
throw FILE_NOT_FOUND;

out.print("HTTP/1.0 200 OK\r\nContent-type: " + getMimeType(filename) + "\r\n\r\n");

// Send file contents to client, then close the connection
final byte[] a = new byte[2048];
int n;
while ((n = f.read(a)) > 0)
out.write(a, 0, n);


} catch (NoSuchElementException | FileNotFoundException e) {
out.print(ERROR_404);
}

out.close();
in.close();
} catch (IOException e1) {
e1.printStackTrace();
}
}

public String getMimeType(String filename)
{
String ext = filename.substring(filename.indexOf(".") + 1);

switch(ext)
{
case "html": case "htm":
return "text/html";
case "css":
return "text/css";
case "java":
return "text/x-java";
case "class":
return "application/x-java";
case "png":
return "image/png";
case "jpg": case "jpeg":
return "image/jpeg";
case "js":
return "application/javascript";
default:
return "application/octet-stream";
}
}

public String normalizeFilePath(String filename) throws FileNotFoundException
{
// Remove GET params.
if(filename.contains("?"))
filename = filename.split("?", 2)[0];

// Make requests to a folder show the index.html file.
if(filename.endsWith("/"))
filename += DEFAULT_INDEX_PAGE;

// Remove leading /
while (filename.indexOf("/") == 0)
filename = filename.substring(1);

// Check for illegal characters to prevent access to superdirectories
if (filename.contains("..") || filename.contains(":") || filename.contains("|"))
throw FILE_NOT_FOUND;

return filename;
}
}
}

Friday, December 21, 2012

Java 7's Best New Features

Java 6 is approaching its end of life release in February 2013, so here are some new features of Java 7 that will help you hit the ground running with the new JDK. A few of them are sure to be huge timesavers, like using Strings in switch statements.

Binary Literals

int j = 0b10101010101;

Underscores in Numbers

Let you quickly see how large a number really is.
int k = 1_000_000;

String Switch Statements

Use the new string switch statements to apply .equals() on the string:
switch(someString)
{
case "Saturday": case "Sunday":
return "weekend";

default:
return "weekday";
}

Automatic Resource Closing

When you open streams in try blocks, they are now automatically closed.
try(InputStream in = socket.getInputStream();
PrintStream out = new PrintStream(new BufferedOutputStream(socket.getOutputStream())))
{
byte[] b = new byte[1024];
int k = 0;
while((k = in.read(b)) > 0)
out.write(b,0,k);
}

// in and out will be cleaned up after this.

Multiple Catch

You can now catch more than one exception at a time:
try{

// do stuff here

}catch(IOException | NullPointerException ex)
{
// Handle either exception
}

New Diamond Operator

No longer do you have to declare the same type in the constructor and variable when you're providing types:
HashMap<String, Integer> myMap = new HashMap<>();

NIO

This is a huge topic, too long to be covered here; but there is a new library for working with various filesystems that abstracts away their differences, so you can use the same code to work remotely, locally, within zip files, etc.

Compute/Fork Join

Java now provides an interface for embarrassingly parallel fork-join tasks (such as sorting), just implement Compute.

Swing Improvements

  • Draw on top of components, then erase the drawing, using JLayer
  • Make translucent windows with .setOpacity(<float>) on a JFrame
  • New HSV tab on JColorChooser.

Thursday, December 20, 2012

A Tiny Java Mime Lookup Library

I couldn't find a nice and very tiny solution to looking up a mimetype based on file extension in Java, so I built one, it uses /etc/mime.types as the source for the information, after the first run, the library should be pretty fast due to the use of a HashMap for lookups.

Get a Mime by Extension

lookupMapping("doc");

Get a Mime by Path

getMime("/path/to/some/file.tar.gz"); // looks up extension "gz"

Full Code Listing

import java.util.HashMap;

/**
* Looks up a mimetype based upon a file extension.
*/
public class MimeType {
public static final String DEFAULT_MIME = "application/octet-stream";

/**
* A list of extensions and their appropriate mime mappings.
*/
private static final String[][] MIME_MAPPING = {{"323","text/h323"},
{"3gp","video/3gpp"},
{"7z","application/x-7z-compressed"},
{"abw","application/x-abiword"},
{"ai","application/postscript"},
{"aif","audio/x-aiff"},
{"aifc","audio/x-aiff"},
{"aiff","audio/x-aiff"},
{"alc","chemical/x-alchemy"},
{"amr","audio/amr"},
{"amr","audio/amr"},
{"anx","application/annodex"},
{"apk","application/vnd.android.package-archive"},
{"appcache","text/cache-manifest"},
{"~","application/x-trash"},
{"%","application/x-trash"},
{"art","image/x-jg"},
{"asc","text/plain"},
{"asf","video/x-ms-asf"},
{"asn","chemical/x-ncbi-asn1"},
{"asn","chemical/x-ncbi-asn1-spec"},
{"aso","chemical/x-ncbi-asn1-binary"},
{"asx","video/x-ms-asf"},
{"atom","application/atom+xml"},
{"atomcat","application/atomcat+xml"},
{"atomsrv","application/atomserv+xml"},
{"au","audio/basic"},
{"avi","video/x-msvideo"},
{"awb","audio/amr-wb"},
{"awb","audio/amr-wb"},
{"axa","audio/annodex"},
{"axv","video/annodex"},
{"bak","application/x-trash"},
{"bat","application/x-msdos-program"},
{"b","chemical/x-molconn-Z"},
{"bcpio","application/x-bcpio"},
{"bib","text/x-bibtex"},
{"bin","application/octet-stream"},
{"bmp","image/x-ms-bmp"},
{"book","application/x-maker"},
{"boo","text/x-boo"},
{"brf","text/plain"},
{"bsd","chemical/x-crossfire"},
{"c3d","chemical/x-chem3d"},
{"cab","application/x-cab"},
{"cac","chemical/x-cache"},
{"cache","chemical/x-cache"},
{"cap","application/vnd.tcpdump.pcap"},
{"cascii","chemical/x-cactvs-binary"},
{"cat","application/vnd.ms-pki.seccat"},
{"cbin","chemical/x-cactvs-binary"},
{"cbr","application/x-cbr"},
{"cbz","application/x-cbz"},
{"cc","text/x-c++src"},
{"cda","application/x-cdf"},
{"cdf","application/x-cdf"},
{"cdr","image/x-coreldraw"},
{"cdt","image/x-coreldrawtemplate"},
{"cdx","chemical/x-cdx"},
{"cdy","application/vnd.cinderella"},
{"cef","chemical/x-cxf"},
{"cer","chemical/x-cerius"},
{"chm","chemical/x-chemdraw"},
{"chrt","application/x-kchart"},
{"cif","chemical/x-cif"},
{"class","application/java-vm"},
{"cls","text/x-tex"},
{"cmdf","chemical/x-cmdf"},
{"cml","chemical/x-cml"},
{"cod","application/vnd.rim.cod"},
{"com","application/x-msdos-program"},
{"cpa","chemical/x-compass"},
{"cpio","application/x-cpio"},
{"cpp","text/x-c++src"},
{"cpt","application/mac-compactpro"},
{"cpt","image/x-corelphotopaint"},
{"cr2","image/x-canon-cr2"},
{"crl","application/x-pkcs7-crl"},
{"crt","application/x-x509-ca-cert"},
{"crw","image/x-canon-crw"},
{"csd","audio/csound"},
{"csf","chemical/x-cache-csf"},
{"csh","application/x-csh"},
{"csh","text/x-csh"},
{"csm","chemical/x-csml"},
{"csml","chemical/x-csml"},
{"css","text/css"},
{"csv","text/csv"},
{"ctab","chemical/x-cactvs-binary"},
{"c","text/x-csrc"},
{"c++","text/x-c++src"},
{"ctx","chemical/x-ctx"},
{"cu","application/cu-seeme"},
{"cub","chemical/x-gaussian-cube"},
{"cxf","chemical/x-cxf"},
{"cxx","text/x-c++src"},
{"dat","application/x-ns-proxy-autoconfig"},
{"davmount","application/davmount+xml"},
{"dcm","application/dicom"},
{"dcr","application/x-director"},
{"deb","application/x-debian-package"},
{"diff","text/x-diff"},
{"dif","video/dv"},
{"dir","application/x-director"},
{"djv","image/vnd.djvu"},
{"djvu","image/vnd.djvu"},
{"dll","application/x-msdos-program"},
{"dl","video/dl"},
{"dmg","application/x-apple-diskimage"},
{"dms","application/x-dms"},
{"doc","application/msword"},
{"docm","application/vnd.ms-word.document.macroEnabled.12"},
{"docx","application/vnd.openxmlformats-officedocument.wordprocessingml.document"},
{"dot","application/msword"},
{"dotm","application/vnd.ms-word.template.macroEnabled.12"},
{"dotx","application/vnd.openxmlformats-officedocument.wordprocessingml.template"},
{"d","text/x-dsrc"},
{"dvi","application/x-dvi"},
{"dv","video/dv"},
{"dx","chemical/x-jcamp-dx"},
{"dxr","application/x-director"},
{"emb","chemical/x-embl-dl-nucleotide"},
{"embl","chemical/x-embl-dl-nucleotide"},
{"eml","message/rfc822"},
{"ent","chemical/x-ncbi-asn1-ascii"},
{"ent","chemical/x-pdb"},
{"eot","application/vnd.ms-fontobject"},
{"eps2","application/postscript"},
{"eps3","application/postscript"},
{"eps","application/postscript"},
{"epsf","application/postscript"},
{"epsi","application/postscript"},
{"erf","image/x-epson-erf"},
{"es","application/ecmascript"},
{"etx","text/x-setext"},
{"exe","application/x-msdos-program"},
{"ez","application/andrew-inset"},
{"fb","application/x-maker"},
{"fbdoc","application/x-maker"},
{"fch","chemical/x-gaussian-checkpoint"},
{"fchk","chemical/x-gaussian-checkpoint"},
{"fig","application/x-xfig"},
{"flac","audio/flac"},
{"fli","video/fli"},
{"flv","video/x-flv"},
{"fm","application/x-maker"},
{"frame","application/x-maker"},
{"frm","application/x-maker"},
{"gal","chemical/x-gaussian-log"},
{"gam","chemical/x-gamess-input"},
{"gamin","chemical/x-gamess-input"},
{"gan","application/x-ganttproject"},
{"gau","chemical/x-gaussian-input"},
{"gcd","text/x-pcs-gcd"},
{"gcf","application/x-graphing-calculator"},
{"gcg","chemical/x-gcg8-sequence"},
{"gen","chemical/x-genbank"},
{"gf","application/x-tex-gf"},
{"gif","image/gif"},
{"gjc","chemical/x-gaussian-input"},
{"gjf","chemical/x-gaussian-input"},
{"gl","video/gl"},
{"gnumeric","application/x-gnumeric"},
{"gpt","chemical/x-mopac-graph"},
{"gsf","application/x-font"},
{"gsm","audio/x-gsm"},
{"gtar","application/x-gtar"},
{"hdf","application/x-hdf"},
{"hh","text/x-c++hdr"},
{"hin","chemical/x-hin"},
{"hpp","text/x-c++hdr"},
{"hqx","application/mac-binhex40"},
{"hs","text/x-haskell"},
{"hta","application/hta"},
{"htc","text/x-component"},
{"h","text/x-chdr"},
{"h++","text/x-c++hdr"},
{"html","text/html"},
{"htm","text/html"},
{"hwp","application/x-hwp"},
{"hxx","text/x-c++hdr"},
{"ica","application/x-ica"},
{"ice","x-conference/x-cooltalk"},
{"ico","image/vnd.microsoft.icon"},
{"ics","text/calendar"},
{"icz","text/calendar"},
{"ief","image/ief"},
{"iges","model/iges"},
{"igs","model/iges"},
{"iii","application/x-iphone"},
{"info","application/x-info"},
{"inp","chemical/x-gamess-input"},
{"ins","application/x-internet-signup"},
{"iso","application/x-iso9660-image"},
{"isp","application/x-internet-signup"},
{"ist","chemical/x-isostar"},
{"istr","chemical/x-isostar"},
{"jad","text/vnd.sun.j2me.app-descriptor"},
{"jam","application/x-jam"},
{"jar","application/java-archive"},
{"java","text/x-java"},
{"jdx","chemical/x-jcamp-dx"},
{"jmz","application/x-jmol"},
{"jng","image/x-jng"},
{"jnlp","application/x-java-jnlp-file"},
{"jpeg","image/jpeg"},
{"jpe","image/jpeg"},
{"jpg","image/jpeg"},
{"js","application/javascript"},
{"json","application/json"},
{"kar","audio/midi"},
{"key","application/pgp-keys"},
{"kil","application/x-killustrator"},
{"kin","chemical/x-kinemage"},
{"kml","application/vnd.google-earth.kml+xml"},
{"kmz","application/vnd.google-earth.kmz"},
{"kpr","application/x-kpresenter"},
{"kpt","application/x-kpresenter"},
{"ksp","application/x-kspread"},
{"kwd","application/x-kword"},
{"kwt","application/x-kword"},
{"latex","application/x-latex"},
{"lha","application/x-lha"},
{"lhs","text/x-literate-haskell"},
{"lin","application/bbolin"},
{"lsf","video/x-la-asf"},
{"lsx","video/x-la-asf"},
{"ltx","text/x-tex"},
{"ly","text/x-lilypond"},
{"lyx","application/x-lyx"},
{"lzh","application/x-lzh"},
{"lzx","application/x-lzx"},
{"m3g","application/m3g"},
{"m3u8","application/x-mpegURL"},
{"m3u","audio/mpegurl"},
{"m3u","audio/x-mpegurl"},
{"m4a","audio/mpeg"},
{"maker","application/x-maker"},
{"man","application/x-troff-man"},
{"mbox","application/mbox"},
{"mcif","chemical/x-mmcif"},
{"mcm","chemical/x-macmolecule"},
{"md5","application/x-md5"},
{"mdb","application/msaccess"},
{"me","application/x-troff-me"},
{"mesh","model/mesh"},
{"mid","audio/midi"},
{"midi","audio/midi"},
{"mif","application/x-mif"},
{"mkv","video/x-matroska"},
{"mm","application/x-freemind"},
{"mmd","chemical/x-macromodel-input"},
{"mmf","application/vnd.smaf"},
{"mml","text/mathml"},
{"mmod","chemical/x-macromodel-input"},
{"mng","video/x-mng"},
{"moc","text/x-moc"},
{"mol2","chemical/x-mol2"},
{"mol","chemical/x-mdl-molfile"},
{"moo","chemical/x-mopac-out"},
{"mop","chemical/x-mopac-input"},
{"mopcrt","chemical/x-mopac-input"},
{"movie","video/x-sgi-movie"},
{"mov","video/quicktime"},
{"mp2","audio/mpeg"},
{"mp3","audio/mpeg"},
{"mp4","video/mp4"},
{"mpc","chemical/x-mopac-input"},
{"mpega","audio/mpeg"},
{"mpeg","video/mpeg"},
{"mpe","video/mpeg"},
{"mpga","audio/mpeg"},
{"mpg","video/mpeg"},
{"mph","application/x-comsol"},
{"mpv","video/x-matroska"},
{"ms","application/x-troff-ms"},
{"msh","model/mesh"},
{"msi","application/x-msi"},
{"mvb","chemical/x-mopac-vib"},
{"mxf","application/mxf"},
{"mxu","video/vnd.mpegurl"},
{"nb","application/mathematica"},
{"nbp","application/mathematica"},
{"nc","application/x-netcdf"},
{"nef","image/x-nikon-nef"},
{"nwc","application/x-nwc"},
{"o","application/x-object"},
{"oda","application/oda"},
{"odb","application/vnd.oasis.opendocument.database"},
{"odc","application/vnd.oasis.opendocument.chart"},
{"odf","application/vnd.oasis.opendocument.formula"},
{"odg","application/vnd.oasis.opendocument.graphics"},
{"odi","application/vnd.oasis.opendocument.image"},
{"odm","application/vnd.oasis.opendocument.text-master"},
{"odp","application/vnd.oasis.opendocument.presentation"},
{"ods","application/vnd.oasis.opendocument.spreadsheet"},
{"odt","application/vnd.oasis.opendocument.text"},
{"oga","audio/ogg"},
{"ogg","audio/ogg"},
{"ogv","video/ogg"},
{"ogx","application/ogg"},
{"old","application/x-trash"},
{"one","application/onenote"},
{"onepkg","application/onenote"},
{"onetmp","application/onenote"},
{"onetoc2","application/onenote"},
{"orc","audio/csound"},
{"orf","image/x-olympus-orf"},
{"otg","application/vnd.oasis.opendocument.graphics-template"},
{"oth","application/vnd.oasis.opendocument.text-web"},
{"otp","application/vnd.oasis.opendocument.presentation-template"},
{"ots","application/vnd.oasis.opendocument.spreadsheet-template"},
{"ott","application/vnd.oasis.opendocument.text-template"},
{"oza","application/x-oz-application"},
{"p7r","application/x-pkcs7-certreqresp"},
{"pac","application/x-ns-proxy-autoconfig"},
{"pas","text/x-pascal"},
{"patch","text/x-diff"},
{"pat","image/x-coreldrawpattern"},
{"pbm","image/x-portable-bitmap"},
{"pcap","application/vnd.tcpdump.pcap"},
{"pcf","application/x-font"},
{"pcf.Z","application/x-font"},
{"pcx","image/pcx"},
{"pdb","chemical/x-pdb"},
{"pdf","application/pdf"},
{"pfa","application/x-font"},
{"pfb","application/x-font"},
{"pgm","image/x-portable-graymap"},
{"pgn","application/x-chess-pgn"},
{"pgp","application/pgp-encrypted"},
{"pk","application/x-tex-pk"},
{"pls","audio/x-scpls"},
{"pl","text/x-perl"},
{"pm","text/x-perl"},
{"png","image/png"},
{"pnm","image/x-portable-anymap"},
{"potm","application/vnd.ms-powerpoint.template.macroEnabled.12"},
{"pot","text/plain"},
{"potx","application/vnd.openxmlformats-officedocument.presentationml.template"},
{"ppam","application/vnd.ms-powerpoint.addin.macroEnabled.12"},
{"ppm","image/x-portable-pixmap"},
{"pps","application/vnd.ms-powerpoint"},
{"ppsm","application/vnd.ms-powerpoint.slideshow.macroEnabled.12"},
{"ppsx","application/vnd.openxmlformats-officedocument.presentationml.slideshow"},
{"ppt","application/vnd.ms-powerpoint"},
{"pptm","application/vnd.ms-powerpoint.presentation.macroEnabled.12"},
{"pptx","application/vnd.openxmlformats-officedocument.presentationml.presentation"},
{"prf","application/pics-rules"},
{"prt","chemical/x-ncbi-asn1-ascii"},
{"ps","application/postscript"},
{"psd","image/x-photoshop"},
{"p","text/x-pascal"},
{"pyc","application/x-python-code"},
{"pyo","application/x-python-code"},
{"py","text/x-python"},
{"qgs","application/x-qgis"},
{"qtl","application/x-quicktimeplayer"},
{"qt","video/quicktime"},
{"ra","audio/x-pn-realaudio"},
{"ra","audio/x-realaudio"},
{"ram","audio/x-pn-realaudio"},
{"rar","application/rar"},
{"ras","image/x-cmu-raster"},
{"rb","application/x-ruby"},
{"rd","chemical/x-mdl-rdfile"},
{"rdf","application/rdf+xml"},
{"rdp","application/x-rdp"},
{"rgb","image/x-rgb"},
{"rm","audio/x-pn-realaudio"},
{"roff","application/x-troff"},
{"ros","chemical/x-rosdal"},
{"rpm","application/x-redhat-package-manager"},
{"rss","application/x-rss+xml"},
{"rtf","application/rtf"},
{"rtx","text/richtext"},
{"rxn","chemical/x-mdl-rxnfile"},
{"scala","text/x-scala"},
{"sce","application/x-scilab"},
{"sci","application/x-scilab"},
{"sco","audio/csound"},
{"scr","application/x-silverlight"},
{"sct","text/scriptlet"},
{"sd2","audio/x-sd2"},
{"sda","application/vnd.stardivision.draw"},
{"sdc","application/vnd.stardivision.calc"},
{"sd","chemical/x-mdl-sdfile"},
{"sdd","application/vnd.stardivision.impress"},
{"sdf","application/vnd.stardivision.math"},
{"sdf","chemical/x-mdl-sdfile"},
{"sds","application/vnd.stardivision.chart"},
{"sdw","application/vnd.stardivision.writer"},
{"ser","application/java-serialized-object"},
{"sfv","text/x-sfv"},
{"sgf","application/x-go-sgf"},
{"sgl","application/vnd.stardivision.writer-global"},
{"sha1","application/x-sha1"},
{"sh","application/x-sh"},
{"shar","application/x-shar"},
{"shp","application/x-qgis"},
{"sh","text/x-sh"},
{"shtml","text/html"},
{"shx","application/x-qgis"},
{"sid","audio/prs.sid"},
{"sig","application/pgp-signature"},
{"sik","application/x-trash"},
{"silo","model/mesh"},
{"sis","application/vnd.symbian.install"},
{"sisx","x-epoc/x-sisx-app"},
{"sit","application/x-stuffit"},
{"sitx","application/x-stuffit"},
{"skd","application/x-koan"},
{"skm","application/x-koan"},
{"skp","application/x-koan"},
{"skt","application/x-koan"},
{"sldm","application/vnd.ms-powerpoint.slide.macroEnabled.12"},
{"sldx","application/vnd.openxmlformats-officedocument.presentationml.slide"},
{"smi","application/smil"},
{"smil","application/smil"},
{"snd","audio/basic"},
{"spc","chemical/x-galactic-spc"},
{"spl","application/futuresplash"},
{"spl","application/x-futuresplash"},
{"spx","audio/ogg"},
{"sql","application/x-sql"},
{"src","application/x-wais-source"},
{"srt","text/plain"},
{"stc","application/vnd.sun.xml.calc.template"},
{"std","application/vnd.sun.xml.draw.template"},
{"sti","application/vnd.sun.xml.impress.template"},
{"stl","application/sla"},
{"stw","application/vnd.sun.xml.writer.template"},
{"sty","text/x-tex"},
{"sv4cpio","application/x-sv4cpio"},
{"sv4crc","application/x-sv4crc"},
{"svg","image/svg+xml"},
{"svgz","image/svg+xml"},
{"sw","chemical/x-swissprot"},
{"swf","application/x-shockwave-flash"},
{"swfl","application/x-shockwave-flash"},
{"sxc","application/vnd.sun.xml.calc"},
{"sxd","application/vnd.sun.xml.draw"},
{"sxg","application/vnd.sun.xml.writer.global"},
{"sxi","application/vnd.sun.xml.impress"},
{"sxm","application/vnd.sun.xml.math"},
{"sxw","application/vnd.sun.xml.writer"},
{"t","application/x-troff"},
{"tar","application/x-tar"},
{"taz","application/x-gtar-compressed"},
{"tcl","application/x-tcl"},
{"tcl","text/x-tcl"},
{"texi","application/x-texinfo"},
{"texinfo","application/x-texinfo"},
{"tex","text/x-tex"},
{"text","text/plain"},
{"tgf","chemical/x-mdl-tgf"},
{"tgz","application/x-gtar-compressed"},
{"thmx","application/vnd.ms-officetheme"},
{"tiff","image/tiff"},
{"tif","image/tiff"},
{"tk","text/x-tcl"},
{"tm","text/texmacs"},
{"torrent","application/x-bittorrent"},
{"tr","application/x-troff"},
{"tsp","application/dsptype"},
{"ts","video/MP2T"},
{"tsv","text/tab-separated-values"},
{"txt","text/plain"},
{"udeb","application/x-debian-package"},
{"uls","text/iuls"},
{"ustar","application/x-ustar"},
{"val","chemical/x-ncbi-asn1-binary"},
{"vcd","application/x-cdlink"},
{"vcf","text/x-vcard"},
{"vcs","text/x-vcalendar"},
{"vmd","chemical/x-vmd"},
{"vms","chemical/x-vamas-iso14976"},
{"vrml","model/vrml"},
{"vrml","x-world/x-vrml"},
{"vrm","x-world/x-vrml"},
{"vsd","application/vnd.visio"},
{"wad","application/x-doom"},
{"wav","audio/x-wav"},
{"wax","audio/x-ms-wax"},
{"wbmp","image/vnd.wap.wbmp"},
{"wbxml","application/vnd.wap.wbxml"},
{"webm","video/webm"},
{"wk","application/x-123"},
{"wma","audio/x-ms-wma"},
{"wmd","application/x-ms-wmd"},
{"wmlc","application/vnd.wap.wmlc"},
{"wmlsc","application/vnd.wap.wmlscriptc"},
{"wmls","text/vnd.wap.wmlscript"},
{"wml","text/vnd.wap.wml"},
{"wm","video/x-ms-wm"},
{"wmv","video/x-ms-wmv"},
{"wmx","video/x-ms-wmx"},
{"wmz","application/x-ms-wmz"},
{"woff","application/x-font-woff"},
{"wp5","application/vnd.wordperfect5.1"},
{"wpd","application/vnd.wordperfect"},
{"wrl","model/vrml"},
{"wrl","x-world/x-vrml"},
{"wsc","text/scriptlet"},
{"wvx","video/x-ms-wvx"},
{"wz","application/x-wingz"},
{"x3db","model/x3d+binary"},
{"x3d","model/x3d+xml"},
{"x3dv","model/x3d+vrml"},
{"xbm","image/x-xbitmap"},
{"xcf","application/x-xcf"},
{"xcos","application/x-scilab-xcos"},
{"xht","application/xhtml+xml"},
{"xhtml","application/xhtml+xml"},
{"xlam","application/vnd.ms-excel.addin.macroEnabled.12"},
{"xlb","application/vnd.ms-excel"},
{"xls","application/vnd.ms-excel"},
{"xlsb","application/vnd.ms-excel.sheet.binary.macroEnabled.12"},
{"xlsm","application/vnd.ms-excel.sheet.macroEnabled.12"},
{"xlsx","application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"},
{"xlsx","application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"},
{"xlt","application/vnd.ms-excel"},
{"xltm","application/vnd.ms-excel.template.macroEnabled.12"},
{"xltx","application/vnd.openxmlformats-officedocument.spreadsheetml.template"},
{"xltx","application/vnd.openxmlformats-officedocument.spreadsheetml.template"},
{"xml","application/xml"},
{"xpi","application/x-xpinstall"},
{"xpm","image/x-xpixmap"},
{"xsd","application/xml"},
{"xsl","application/xml"},
{"xspf","application/xspf+xml"},
{"xtel","chemical/x-xtel"},
{"xul","application/vnd.mozilla.xul+xml"},
{"xwd","image/x-xwindowdump"},
{"xyz","chemical/x-xyz"},
{"zip","application/zip"},
{"zmt","chemical/x-mopac-input"}};

private final HashMap<String, String> m_mapping;
private static MimeType instance;

private MimeType()
{
m_mapping = new HashMap<String, String>();

for(int i = 0; i < MIME_MAPPING.length; i++)
{
m_mapping.put(MIME_MAPPING[i][0], MIME_MAPPING[i][1]);
}
}

private static MimeType getInstance()
{
if(instance == null)
instance = new MimeType();

return instance;
}

/**
* Returns a mime given the extension, or the default if none is found.
* @param ext - the extension to lookup the mime based upon.
* @return
*/
private String lookupMapping(String ext)
{
String mime = m_mapping.get(ext);

if(mime == null)
return DEFAULT_MIME;

return mime;
}

/**
* Looks up a mimetype.
*
* @param path - the path to look at the mime for.
* @return - the mime of the path.
*/
public static String getMime(String path)
{
String ext = path.substring(path.lastIndexOf('.') + 1);
return getInstance().lookupMapping(ext);
}
}

Saturday, October 27, 2012

A Java eval() Implementation

I recently came across a situation where I had to evaluate expressions that the user had input, in Java there are quite a few ways to do this, but the solutions had several downsides:
  • Resource hungry (3mb libraries that add to JAR file size greatly)
  • RAM hungry, some store strings indefinitely
  • Insecure, compile code using JVM then load it as a class
  • Difficult to integrate
After working with Mozilla's Rhino for around eight months, it became burdensome and I wrote my own interpreter that:
  • Can do standard operations */+-()
  • Can optionally run "functions" or use "variables"
  • Is fast, as fast as Rhino for small one-off expressions
  • Is small (< 7Kb compiled, before being put in a JAR)
The code is on Github, under a BSD license.

Running it is quite simple, just download the jar, add it to your project, and use the following examples as guides

Simple Example:

import net.josephlewis.jeval.Eval;
import net.josephlewis.jeval.MalformedExpression;

public class BasicExample {
public static void main(String[] args)
{
// A simple expression.
try {
System.out.println( Eval.eval("2 * 3 + 5") );
} catch (MalformedExpression e) {
System.err.println(e.getMessage());
}

// Check to see if an expression is valid or not.
System.out.println("is valid? (should be true)");
System.out.println(Eval.isErrorFree("2 * 3 + 5", null));

System.out.println("is valid? (should be false)");
System.out.println(Eval.isErrorFree("2 * 3 +", null));
}
}

Example With Variables/Function Calls:

import net.josephlewis.jeval.Eval;
import net.josephlewis.jeval.MalformedExpression;
import net.josephlewis.jeval.VariableStore;

public class VariableStoreExample implements VariableStore {

/**
* Not a very useful getValue, just summs the ASCII value of the given
* string and returns it.
*/
@Override
public double getValue(String variable) throws MalformedExpression {
double b = 0;

for(char c : variable.toCharArray())
b += Character.getNumericValue(c);

return b;
}

public static void main(String[] args)
{
VariableStore vse = new VariableStoreExample();

try {
System.err.println(Eval.eval("$a - $a", vse));
System.err.println(Eval.eval("$abc - $cba", vse));

// Spaces can be included in variable names, if they're enclosed
// within ( and ) or "s
System.err.println(Eval.eval("$(hello world) - $(world hello)", vse));
System.err.println(Eval.eval("$hello(world) - $world(hello)", vse));
System.err.println(Eval.eval("$hello(\"world\") - $world(\"hello\")", vse));

} catch (MalformedExpression e) {
e.printStackTrace();
}
}
}

And there you have it, a custom parser for expressions as simple as just */+-() or as complex as you want through extensions using dollar signs.

After using the old JS parser for over a year in the before-mentioned project this was a breath of fresh air, the JAR file that held Rhino shrunk by a few MB providing relief for my users and the server that had to send it around.

Monday, April 9, 2012

Serializable Hash Map in Java

A simple HashMap implementation that is both serializable, and provides methods to convert it to/from a Base64 representation (serialize function).

The definition of this class seems to be lacking in the Java standard library, and I was unable to find a small stand-alone that was easy to integrate with my HSQLDB that fits over CSV files, so I made one!

import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.ObjectInput;
import java.io.ObjectInputStream;
import java.io.ObjectOutput;
import java.io.ObjectOutputStream;
import java.io.Serializable;
import java.util.HashMap;

/**
* A HashMap that serializes in to a base64 representation.
* @author Joseph Lewis <joehms22@gmail.com>
*
* @param <E>
* @param <T>
*/
public class SerializableHashMap<E extends Serializable,T extends Serializable> extends HashMap<E,T> implements Serializable
{
private static final long serialVersionUID = 1L;

/**
* Creates an empty serializable hash map.
*/
public SerializableHashMap()
{

}

/**
* Creates a new map from a pre-serialized representation.
* @param init
* @throws IOException
* @throws ClassNotFoundException
*/
public SerializableHashMap(String init) throws IOException, ClassNotFoundException
{
SerializableHashMap<E,T> et = deserialize(init);

putAll(et);
}

private void writeObject(ObjectOutputStream out) throws IOException
{
// Write number of objects to follow
out.writeInt(size());

// Write Key:Value pairs.
for(E key : keySet())
{
out.writeObject(key);
out.writeObject(get(key));
}
}


private void readObject(ObjectInputStream in) throws IOException, ClassNotFoundException
{
// Read number of objects to follow
int num = in.readInt();

// Read key value pairs.

for(int i = 0; i < num; i++)
{
@SuppressWarnings("unchecked")
E key = (E) in.readObject();
@SuppressWarnings("unchecked")
T val = (T) in.readObject();

put(key, val);
}
}

public String toString()
{
String output = "SerializableHashMap\n";

for(E key : keySet())
{
output += "\t" + key.toString() + " -> " + get(key) + "\n";
}
return output;
}

/**
* Creates a serialized representation of this object.
*
* @return
* @throws IOException
*/
public String serialize() throws IOException
{
ObjectOutput out;
ByteArrayOutputStream bos = new ByteArrayOutputStream() ;
out = new ObjectOutputStream(bos) ;
out.writeObject(this);
out.close();

// Get the bytes of the serialized object
byte[] buf = bos.toByteArray();

return new sun.misc.BASE64Encoder().encode(buf);

}

/**
* Same as serialize, but returns the given string if serialization fails.
*/
public String safeSerialize(String given)
{
try
{
return serialize();
} catch (IOException e)
{
return given;
}
}

/**
* Returns a deserialized representation of an object.
* @param s
* @return
* @throws IOException
* @throws ClassNotFoundException
*/
@SuppressWarnings("unchecked")
private SerializableHashMap<E,T> deserialize(String s) throws IOException, ClassNotFoundException
{
byte[] buf = new sun.misc.BASE64Decoder().decodeBuffer(s);

ByteArrayInputStream bosin = new ByteArrayInputStream(buf);
ObjectInput in = new ObjectInputStream(bosin);

return (SerializableHashMap<E, T>) in.readObject();
}
}

Wednesday, October 19, 2011

Real Time Header Editing with Paros

Sometimes it is advantageous to modify headers as they are sent by a website, let's say you're looking for SQL injection vulnerabilities on your server or want to see what happens when non validated information escapes your javascript validation.

But what to use to easily view/modify/drop headers and incoming connections?

Introducing Paros, a web proxy written entirely in Java so it is cross platform, just set your browser or operating system to use 127.0.0.1:8080 as a proxy after firing it up. Basic functionality shown after the break.





Paros has many advanced features, I'll just discuss trapping requests and responses and using the history.


When you begin using Paros, any websites you visit will be left in the sites pane on the left hand side, from there you can click on each requested item for a site, and view the reqest and response for that item in the appropriate tabs.

If you want to modify the request or response, head over to the Trap tab, and check the appropriate boxes. If there is POST information sent along with the HTTP request, then it will be displayed below; I changed the view to "tabular" to more easily see/edit the data.

Aside: In this instance my browser is pointed at a site called conjuguemos.com teachers us it to test their students' ability to conjugate verbs, when students are done, unencrypted javascript headers are sent containing the session information, not very bright in my book, as any student could quickly and easily modify their scores to have 10,000 words right in thirty-eight seconds.

Once a header/response is trapped you may edit it by hand in the panel given, or drop it (say you wanted to test a timeout for your web widget, like a Twitter reader if Twitter were down).

Sessions can also be saved and restored through the File menu. Overall the tool is easy to use, fast to learn, and very effective.

Saturday, October 23, 2010

Java Source Code Scraping

I am taking a course in Java development whose textbook provides example programs.  Unlike many other books the source code for these programs is trapped inside of HTML pages.  Being a programmer I don't like doing work, and therefore spent some time making a nice Python script that will take in an URL, and folder path, and will scrape the given page for source code, remove all formatting, and output them properly named to the folder specified.



With a little work this script could scrape whatever you wanted, as long as it is between certain tags.

#!/usr/bin/python

import re
import urllib2
import xml.sax.saxutils

numnones = 0

def unescape(string):
'''Unescape the & escapes in html, like &quot; returns a string '''

string = string.replace("&quot;", "\"") #Not done by xml lib
string = xml.sax.saxutils.unescape(string)

return string

def remove_HTML_tags(text):
'''Removes html tags from a supplied string.

Warning: This is accomplished using regular expressions that simply cut
out all text in between and including less than and greater than
characters.

'''

regex_html = re.compile(r'<.*?>')
return regex_html.sub('', text)

def fetch_page(url):
'''Returns the html for the webpage at the supplied url.'''

page = urllib2.urlopen(url)
pagetext = ""

for line in page:
pagetext += line

return pagetext

def get_name(class_text):
''' Returns the name of the abstract, class, or interface, when given the
text of a Java file.

Returns None if the name can not be determined.

Warning: Although Java classes may be named with unicode, this function
will only return characters A-Z, a-z, and 0-9.

'''

#Compile the regular expressions, they all ignore case and accept all chars.
class_ = re.compile(r'public class (.*?) ', re.DOTALL | re.IGNORECASE)
interface = re.compile(r'public interface (.*?) ', re.DOTALL | re.IGNORECASE)
abstract = re.compile(r'public abstract class (.*?) ', re.DOTALL | re.IGNORECASE)

#Find the name of the class/interface/abstract
name = class_.findall(class_text) #Returns [] if none found

if name == []:
name = abstract.findall(class_text)

if name == []:
name = interface.findall(class_text)

#If no name is found return None.
if name == []:
return None

#Remove any remaining non a-z A-Z characters
accepted_chars = re.compile(r'[^A-Za-z0-9]*')
return accepted_chars.sub('', name[0])


def return_between(first_tag, second_tag, text):
'''Returns an array of the text between the delimiters given. All text
between the end and beginning delimiters will be discarded.

Arguments:
first_tag -- The tag to begin text harvesting. (string)
second_tag -- The tag to end text harvesting. (string)
text -- The string in which the tags are to be found. (string)

'''

basic_split = text.split(first_tag)

#select only the sections which contain the close tags, discard the rest.
second_split = []
for i in basic_split:
if second_tag in i:
second_split.append(i)

#Cut out the text before the close tag
between = []

for line in second_split:
value, end = line.split(second_tag, 1)
between.append(value)

return between

def write_file(name, content, location, ext=".java"):
'''Writes a file with the name given, with the content provided, to the
location given, with the given extension.

Arguments:
name -- The name of the file to be written.
content -- The content of the file to be written.
location -- The folder where the file will be created/overwritten.

Keyword Arguments:
ext -- The extension for the file. (default ".java")

Warning: This function will overwrite any pre-existing files without
giving warnings.

'''

if not location.endswith("/"): #FIXME This is not cross platform!
location = location + "/"

print("Location: "+location)

f = open( location + name + ext, "w" )
f.write( content )
f.close()

def main(url, output_folder):
'''Fetches the text at the supplied url and outputs .java files of code
between pre tags.

Arguments:
url -- The url of the page to scrape code from. (string)
output_folder -- The directory that the java files will be created in.

'''
global numnones
page_text = fetch_page(url)

#Get the text between the pre tags
code_and_garbage = return_between("<pre>", "</pre>", page_text)

#Walk through each entry clean it, and write the source file
for i in code_and_garbage:

print i

i = remove_HTML_tags(i)
i = unescape(i)

name = get_name(i)

if name == None:
write_file("None"+str(numnones), i, output_folder)
numnones += 1
else:
write_file(name, i, output_folder)

if __name__ == "__main__":
location = raw_input("Enter the url: ")
output_folder = raw_input("Enter the folder to write to: ")

main( location, output_folder )