Wednesday, April 24, 2013

A Simple Python Plugin Framework

This framework is somewhat different than those currently out there, most of those either:
  • don't allow dynamic plugins
  • require you to use nasty hooks
  • require specifically formatted imports
  • require plugins to all be in one place
This script however allows plugins to:
  • be dynamically loaded
  • be a subclass of a master plugin class
  • have any name
  • be in many different folders
The general idea is that you give the function a path, and the base-class that all plugins are subclasses of, then it iterates through the path, finding python files, loading them, and then searching them for any classes that are subclasses of the class you want. When it gets done, it returns them:

Code

import inspect
import os
import sys
def load_plugins(plugin_path, instance):
'''Loads a set of plugins at the given path.

Arguments:
plugin_path - the OS path to look for plugins at.
instance - classes of this instance will be returned
'''
plugins = []
plugin_dir = os.path.realpath(plugin_path)
sys.path.append(plugin_dir)

for f in os.listdir(plugin_dir):
if f.endswith(".py"):
name = f[:-3]
elif f.endswith(".pyc"):
name = f[:-4]
elif os.path.isdir(os.path.join(plugin_dir, f)):
name = f
else:
continue

try:
mod = __import__(name, globals(), locals(), [], 0)

for piece in inspect.getmembers(mod):
if isinstance(piece, instance):
plugins.append(piece)

except ImportError as e:
print(e)
pass # problem importing

return plugins

Friday, April 19, 2013

Friday Link List -- Ubuntu 13.04 Backend Updates and Good News for the Web

Real Dev News for Ubuntu 13.04

This is way better than the stuff you'll find on OMGUbuntu!
  1. gvfs updates with MTP Support (you can now connect your Android 4 device and it just works)
  2. LibreOffice 4.0; a huge number of bugfixes went in to this
  3. cups updates to now auto-detect shared printers on your network, and auto-share them with other devices including iPads/iPhones etc.
  4. Much of the backend has been ported to Python 3 in hopes of totally removing Python 2.
  5. Lots of work has gone in to sandboxing applications from one another (most likely due to the phone release)
  6. A potential new feature (if it makes it in time) is automatic removal of old kernels, which can take up a few gigs of space over the course of six months if not cleaned manually.
  7. New upstart version allowing userspace (beta) jobs, and launch on file/folder change, hopefully meaning fewer background services and less clunky code.
  8. Fixes for quite a few RAM hogs that persist across sessions.

Web

  • jQuery 2.0 has been released it drops support for IE 6-8, it looks like old Android (2x) is going to get the chop next.
  • Chrome forked WebKit to form Blink, they say they were able to drop about 2.5 million lines of code right off the bat, hopefully it'll speed up their iteration time.
  • Firefox has released it's baseline compiler tl;dr version, it cuts down on complexity, and does a much better job of supporting IonMonkey being it's based on the same backend giving a huge speed boost.
  • ASM.js promises a good future for porting C/C++ apps to the browser with support from Firefox, and Chrome in the works.

Wednesday, April 17, 2013

Things I Learned From Working A 2 Year 50,000 Line Project

This is mostly to serve as a brain dump for a piece of software I've been writing for around two and a half years, it was comprised of a large database (read: ~five hundred million rows), a cross platform Java application, a large Python web-server/API, a phone app, lots of small scripts that fetched and transformed data, and globs of glue. (I wrote all of them in the two year period!)

Architecture

  • It pays to fully think through your APIs first
  • All web APIs should have a version in the URL e.g. /application/api/2.0/...
  • Profile early and often
  • A few good caches are worth many bad ones
  • Make everything a plugin that extends one class; you may think it's a big hairy mess, but it's better than multiple plugin systems.
  • Use libraries wherever possible
  • Ditch libraries when you only use one function from them

Debugging

  • Users write awful bug reports
  • You can help them write better ones by giving them a short form to fill out.
  • Automatically attach debugging info to all bug reports
  • Make version numbers easily available (not just in the executable's name)
  • Auto-update your apps so you don't have users lagging behind

Upgrades

  • Your requirements will constantly change, make sure you have an easy upgrade path in your database that won't force hacky solutions.
  • Make sure your ORM supports incremental changes.

User Experience

  • Start fast, even if you're not done processing yet
  • Never trust an engineer to do design (including software engineers)
  • If the user doesn't see it, it doesn't exist
  • Make it very hard for the user to do stupid things
  • Do automatic backups/saves

Sanity

  • Clients like saying "I did say that, didn't I? What I meant was..."
    • all features should be plugins so you can enable/disable when they do this
  • SVN and Eclipse are a nightmare to get playing nice
  • Java 7 is really awesome!
    • auto-closing streams
    • multiple catches in a try block
    • string in switch statements
  • JNLP sucks, but so does embedding apps in-browser; just make it easy for the user to download it, and provide auto upgrading

Friday, April 12, 2013

Covertly Sending Messages Using DNS Queries

I won't be the first to post about this, but I read it a while back and thought it sounded fun: note, this post will be more technical in nature than most, but I hope it will be a fun read, and a fun toy.

Let's say you want to send short messages that are very covert and not persistent. You could, of course use Twitter, sending strange cryptographic tweets, but that may be too obvious for your tastes, plus they can more easily be traced back to an IP/Person if just one slip-up is made. Somehow using DNS would be an awesome way to do this because:

  • It can be accessed by anyone
  • Messages expire over known intervals
  • No accounts are necessary
  • There are huge amounts of data to go through and junk requests that will look just like yours if someone wanted to find you out
  • Google has a very nice public DNS you can use for free at 8.8.8.8 or 8.8.4.4

Specifics

The two types of queries we'll be using here are standard and non-recursive DNS queries.

Standard queries are the ones that your computer normally does to change a hostname in to an IP address it can contact, to do this from a command line you'd type:

$ dig @8.8.8.8 www.google.com
To which the DNS server responds:

; <<>> DiG 9.8.1-P1 <<>> @8.8.8.8 www.google.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 5459
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.google.com. IN A

;; ANSWER SECTION:
www.google.com. 300 IN A 74.125.225.179
www.google.com. 300 IN A 74.125.225.180
www.google.com. 300 IN A 74.125.225.176
www.google.com. 300 IN A 74.125.225.177
www.google.com. 300 IN A 74.125.225.178

;; Query time: 53 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Apr 12 10:18:31 2013
;; MSG SIZE rcvd: 112
This shows you all of the numerical IP addresses you can reach google.com at, if you copy and paste the first one in to your browser, you'll see Google's home page.

That's all well and good, but how would you actually store information using DNS? For this you'll need to generate a domain name that doesn't exist, we'll be using www.testdnsflagsetting.com for the purposes of this article:

Now do a dig on it:

$ dig @8.8.8.8 www.testdnsflagsetting.com +norecurse
The response looks different this time, note that ANSWER: 0 and AUTHORITY: 0, this means the request is not authoratative (8.8.8.8 doesn't know this informatoin for sure), and there was no answer.

This is because we passed in the +norecurse flag to dig, asking the server to just reply with what was it had stored rather than asking higher up in the chain of command:

; <<>> DiG 9.8.1-P1 <<>> @8.8.8.8 www.testdnsflagsetting.com +norecurse
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 23556
;; flags: qr ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.testdnsflagsetting.com. IN A

;; Query time: 57 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Apr 12 10:04:10 2013
;; MSG SIZE rcvd: 44
Now we'll do the same thing, but without the +norecursive:

$ dig @8.8.8.8 www.testdnsflagsetting.com

; <<>> DiG 9.8.1-P1 <<>> @8.8.8.8 www.testdnsflagsetting.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 22889
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;www.testdnsflagsetting.com. IN A

;; AUTHORITY SECTION:
com. 900 IN SOA a.gtld-servers.net. nstld.verisign-grs.com. 1365782647 1800 900 604800 86400

;; Query time: 82 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Apr 12 10:04:21 2013
;; MSG SIZE rcvd: 117
Note that the AUTHORITY: 1 now is true, meaning that 8.8.8.8 checked with the server that knows about all .com addresses to see if ours existed. There is also a new section, note the number 900 in it, that means this response is going to be saved in Google's DNS (8.8.8.8) for 900 seconds.

If we go back and do the +norecurse version again, we now get the cached response, but with 890, meaning there are only 890 seconds left until the cache is cleared:

$ dig @8.8.8.8 www.testdnsflagsetting.com +norecurse

; <<>> DiG 9.8.1-P1 <<>> @8.8.8.8 www.testdnsflagsetting.com +norecurse
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 14832
;; flags: qr ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;www.testdnsflagsetting.com. IN A

;; AUTHORITY SECTION:
com. 890 IN SOA a.gtld-servers.net. nstld.verisign-grs.com. 1365782647 1800 900 604800 86400

;; Query time: 39 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Apr 12 10:04:31 2013
;; MSG SIZE rcvd: 117
Put these together now, and say you want to send a true or a false to someone on the other side of the globe, you both just have to agree on a time to check, and a domain, then if you want to say true you do a recursive query, and the other person will get back an authoritative record when they do a non-recursive; if you don't do the query, they will not and they'll know you sent false

Sending a Whole Message

Now say you want to send a whole message! Convert the text to ASCII, and break it down to bits, use the first byte to tell how long the message will be (up to 255 characters).

Say you just wanted to send the message Hi, that means you'd send a 2 then Hi, but you can't use the same domain the whole way through, you'll have to agree on a bunch of domains, or a way of randomly generating them. Here we'll just append a number to the end:

Letter:       2        H        i
Binary: 00000010 01001000 01101001
POsition: 111111 11112222
01234567 89012345 67890123
Your queries would look something like this:

dig www.testdnsflagsetting6.com
dig www.testdnsflagsetting9.com
dig www.testdnsflagsetting12.com
dig www.testdnsflagsetting17.com
dig www.testdnsflagsetting18.com
dig www.testdnsflagsetting20.com
dig www.testdnsflagsetting23.com
You only actually dig for the 1s, because they are represented by true.

Getting The Message Back Out

Now to get the message back out, you just need to do a dig for positions 0-7, and convert that in to the number of letters in the message, multiply that by 8 to get the number of bits you need to check, and check the domains ending with all of those numbers.

First 8
dig www.testdnsflagsetting0.com

dig www.testdnsflagsetting1.com

...

dig www.testdnsflagsetting7.com


This would give us the number 2, meaning there were 2 more sections of 8, then you read on.

A program to do it!

This is a short Python script to do it, you may want to change the DOMAIN_NAME. in case other people are using the script too.

#!/usr/bin/env python3
''' A program to send messages via DNS queries.
Copyright 2013 Joseph Lewis <joehms22@gmail.com> | <joseph@josephlewis.net>

MIT License
'''

import subprocess

DNS_SERVER = '8.8.8.8'
DOMAIN_NAME = "www.testdnsflagsetting{}.com"
NORECURSE_OPT = "+norecurse"

msg = raw_input("Enter a message, or blank to receive: ")

def read_byte(byteno):
byte = "0b"
for i in range(byteno * 8, (byteno + 1) * 8):
output = subprocess.check_output(['dig','@{}'.format(DNS_SERVER), DOMAIN_NAME.format(i), NORECURSE_OPT])
if ";; AUTHORITY SECTION:" in output:
byte += '1'
else:
byte += '0'

return int(byte, 2) # converts binary to an int

def write_byte(byteno, byte):
to_write = bin(byte)[2:].zfill(8) # gets binary representation of a byte
for loc, b in enumerate(to_write):
if b == '1':
i = (byteno * 8) + loc
subprocess.check_output(['dig','@{}'.format(DNS_SERVER), DOMAIN_NAME.format(i)])
print "Wrote 1 at: {}".format(i)

if len(msg) == 0:
message = ""
for byte in range(1,read_byte(0) + 1): # first byte is length of message
message += chr(read_byte(byte))

if len(message) > 0:
print message
else:
print "[No Message]"

else:
total = len(msg)
write_byte(0, total)
for loc, char in enumerate(msg):
write_byte(loc + 1, ord(char))

print "Message written"

Tuesday, April 9, 2013

HOWTO Not Worry About IE6 Ever Again

While IE6 market share is dwindling, we are far from being out of the woods yet. Gecko, Trident, WebKit and Blink, are likely to never convirge on a full implementation of W3C specifications, leaving some browsers more equal than others.

Not to worry though, there is a nice little script called Modernizr, that can auto-detect features of a browser and automatically load fixes for browsers that don't have the technologies you need.

Shortlist of Features Modernizr Can Detect

  • HTML5 Canvas
  • IndexedDB
  • WebWorkers
  • Lots of CSS3 Stuff
  • Geolocation
  • WebSockets

Monday, April 8, 2013

Coding is a Drug

Coding is highly addictive. I suspect it has to do with the combination of the brain being the bottleneck, like all good puzzles; rapid iteration allowing you to see a result slowly emerge, and the fact that once you get done, you'll have something tangable as a reward that will serve you for a long time.

It's somewhat like combining crossword puzzles, woodworking, and sketching.

Monday, April 1, 2013

HowTo Calculate an MD5 hash in Java

If you're dealing with networking, or just want to make sure your files don't get corrupted/modified between times your app runs.

It is critical to use the algorithm.reset() command before using a MessageDigest instance, as if there are bytes still remaining from a prior digest, it will completely ruin your output.

Code

import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;

// calculates an MD5sum for a given bit of text
public class MD5
{
public static void main(String[] args)
{
try
{
MessageDigest algorithm = MessageDigest.getInstance("MD5");
algorithm.reset(); // clear the digest back to starting state.

// Get selected digest of the given text.
byte[] digest = algorithm.digest( "Hello, world!".getBytes() );

System.out.println(byteArrToHex(digest));
}
catch (NoSuchAlgorithmException e)
{
// no such algorithm, MD5
}
}

// converts bytes to text.
private static String byteArrToHex(byte[] bytes)
{
StringBuilder sb = new StringBuilder();
for(byte b : bytes)
sb.append(String.format("%02X", b));
return sb.toString().toLowerCase();
}
}

Other Notes

  • To calculate SHA1 hashes instead, use SHA1 as the instance type.
  • The NoSuchAlgorithmException likely won't be thrown on desktop platforms, but it is possible that certain algorithms won't be implemented in certain areas due to them being considered military grade encryption; SHA1 and MD5 shouldn't have this problem because they are just simple hashes.