New Location

As of September 7th, 2013 this blog will be moved to http://onehourhacks.com!

Simple Credit Card Validator

Validating credit cards numbers is a fairly simple job, and there are no short supply of algorithms out there you can easily download to do it. That being said, it is a perfect algorithm to write in Haxe for the following reasons:

• Speed isn't super critical
• It doesn't involve the system in any way
• You can use the same code across PHP, JS, Java, C++, C#, Actionscript etc.
• The same code would be useful in all contexts.

Algorithm

The algorithm realized here is the Luhn algorithm. You can do the formal proofs with some discrete math, or trust the Wikipedia Article that it is useful because:

• It detects any one digit error e.g. 1235 instead of 1234
• It detects any one transposition e.g. 1324 instead of 1234

Both errors are exceedingly common for human typists.
(Note that the ISBN10 algorithm holds some of these same properties.)

The Code

/**
A simple credit card validator that just runs the luhn algorithm against the 
given number. It doesn't even check to see if the string is long enough!
**/
class Validator
{
    /**
        Call this function with a string of digits that represent a credit card
        number.
    **/
    public static function validate(cardNumber : String) : Bool
    {
        return (luhn(cardNumber) == 0);
    }

    private static function luhn(numericString : String) : Int
    {
        var output = stringToDigitArray(numericString);
        var sum : Int = 0;
    
        var isEven : Bool = true;
        for( i in output.iterator())
        {
            if(isEven)
            {
                sum += sumOfDigits(i * 2);
            }
            else
            {
                sum += i;
            }
        
            isEven = !isEven; // Invert sign for next iteration
        }
    
        return sum % 10;
    }

    private static function stringToDigitArray(numericString : String) : Array<Int>
    {
        var output = new Array<Int>();
        for( i in 0...numericString.length)
        {
            output.push(Std.parseInt(numericString.charAt(i)));
        }
    
        return output;
    }

    private static function sumOfDigits(number : Int) : Int
    {
        var sum : Int = 0;
    
        while(number != 0)
        {
            sum += number % Std.int(10);
            number = Std.int(number / Std.int(10));
        }
    
        sum += number;
    
        return sum;
    }

    /* Examples */
    public static function main() : Void
    {
        // False
        trace(Validator.validate("1111111111111111"));
        
        // All true below this point
        trace(Validator.validate("4271058012370682"));
        trace(Validator.validate("4052047247466840"));
        trace(Validator.validate("4250112315345738"));
        trace(Validator.validate("4302220036446780"));
        trace(Validator.validate("4271382676044056"));
        trace(Validator.validate("4027883785016825"));
        trace(Validator.validate("4125875428334483"));
        trace(Validator.validate("4231380166327600"));
        trace(Validator.validate("4820020713741757"));
        trace(Validator.validate("4013112216053363"));
    }
}

HOWTO Find and Restore a Deleted SVN File/Folder

1. Change directories to your SVN project.
2. Run svn log -v > all_history.txt
3. Open all_history.txt and find the file you want (Ctrl + F is your friend!)
4. Make sure the line of the file you want begins with a D, this means it was deleted.
5. Look up above that file for the revision number, subtract one from it.
6. Run a new checkout for the version right beforehand i.e.: svn co http://hostname/path/to/files_parent_directory@revision#_right_before

Example

We want to find DataFeedList.java, here is the svn log:

------------------------------------------------------------------------
r24843 | joslewis | 2013-04-14 10:24:28 -0600 (Sun, 14 Apr 2013) | 1 line
Changed paths:
   D /trunk/software/src/bmod/gui/widgets/DataFeedList.java
   D /trunk/software/src/bmod/gui/widgets/FilterableComponentList.java
   D /trunk/software/src/bmod/gui/widgets/JValidatingTextField.java
   D /trunk/software/src/bmod/gui/widgets/WebButton.java
   D /trunk/software/src/bmod/gui/widgets/WebMenuItem.java
   M /trunk/software/src/bmod/plugin/generic/gui/ReportBugMenuItem.java

Modified ReportBugMenuItem to now work without WebButton. -~400 lines.
------------------------------------------------------------------------

We can see it was deleted (D) during revision r24843 by me on 2013-04-14 10:24:28 -0600. That means we want to actually get the version right before it was deleted, so we'll check out version 24842.

It was in the folder: /trunk/software/src/bmod/gui/widgets/, so we'll craft a new checkout with the information to get it back:

svn co http://svn.host.com/trunk/software/src/bmod/gui/widgets@24842

Run this in another folder so you don't overwrite the existing source, and you'll find the deleted item in it!

The Haxe Programming Language

This is a language that has seemed to slip by under the radar for the most part. Haxe, commonly pronounced "hex" or "hacks" is a language that targets many virtual machines, instead of the more common virtual machine that targets many languages.
It has a reasonable standard library that feels designed, much like Python's. The syntax is Go like, with identifiers following names:

class WebFetch 
{
    static function main()
    {
        var output = haxe.Http.requestUrl("http://www.google.com/");
        Sys.print(output);
    }
}

When built, this program is a ~68Kb JAR file when targeting Java or a 550Kb ELF when generating C++.
Compilation is straightforward, consisting of an internal build system that resolves libraries well, and a global repository that you can pull new packages from.
One of the biggest benefits of this language is its ability to compile and run on so many platforms; it easily handles:

• Android
• Flash
• Linux
• Mac
• Windows
• HTML + JS
• PHP
• Java (in Beta)

Example Uses

• You're a web-developer and have some business code that needs to target language of the day. Haxe can compile to language of the day without the need to translate.
• You want to code a cross-platform game to reach as broad of an audience as possible without re-writing lots of code.
• You want to develop a client for an API you publish so developers will actually use it without learning all of the languages it is going to be used in or maintaining the libraries.
• You want to develop an app that is future proof, but don't mind it being a little ugly.

Pitfalls

• There is no standardized UI system.
• The coding conventions are a strange to begin with.
• There are some obvious deficiencies in the standard library (no string compare)
• Code is slightly larger than it would be natively.

Conclusion

Overall, Haxe is a good language that I see myself using in the long run because it is so future-proof and well designed. Plus, the ability to do things easily cannot be underestimated. The example above would have taken over 100 lines of Java and about the same in C++, instead it took eight and thirty seconds to put together after knowing the language less than twenty-four hours.

Finding Strings in Raw Data

Occasionally, you'll find yourself wishing you could pull out human-readable strings from binary formats, like JPEGs, EXEs or even raw disk dumps. This may be for a variety of reasons, such as:

• Performing malware analysis
• Finding lost files
• Efficient searching

Algorithm

Actually extracting interesting text from a file is somewhat difficult because,

• you don't know what format the file is in
• you don't know what language the text is in
• you don't know how relevant a snippet is

Therefore, we can only rely on if a character is printable or non-printable, whatever that means. Then we can use information to try and weed out the junk without compromising things like IP addresses, which may look very much like junk, but are not.

The method I found that works relatively well is this:

1. Look for runs of printable characters in a file with length greater than 5
2. Look for runs of uppercase letters, lowercase letters, or numbers.
3. For each character of each run, set the value of the nth letter to n.
4. Sum the run totals
5. Sort the scores and return.

This algorithm is implemented in the code below.

Outputs

The software below strips special terminal characters (like newlines and carriage returns), and prints out items in the following format:

<score><tab><file><tab><string><newline>

Example:

00000250    dd.img  &'()*56789:CDEFGHIJSTUVWXYZcdefghijstuvwxyz
00000176    dd.img  .IEC 61966-2.1 Default RGB colour space - sRGB
00000176    dd.img  .IEC 61966-2.1 Default RGB colour space - sRGB
00000138    dd.img  ;CREATOR: gd-jpeg v1.0 (using IJG JPEG v62), quality = 95 
00000136    dd.img  ,Reference Viewing Condition in IEC61966-2.1
00000136    dd.img  ,Reference Viewing Condition in IEC61966-2.1
00000099    dd.img  Copyright (c) 1998 Hewlett-Packard Company
00000078    dd.img  <?xpacket begin='
00000075    dd.img  Hhttp://ns.adobe.com/xap/1.0/
00000075    dd.img  Adobe Photoshop CS2 Macintosh
00000063    dd.img  $54545,,4,44455,554,,4,,,4,55456,,,,,,/5554-,,4,24,
00000058    dd.img  IEC http://www.iec.ch
00000058    dd.img  IEC http://www.iec.ch
00000051    dd.img  bottomOutsetlong
00000046    dd.img  rightOutsetlong

It appears this unallocated space actually has files!

Operation

Save the below code to reader.py and provide it paths to the files you want it to read. There are no parameters.

Source

#!/usr/bin/env python
'''
A program that extracts strings from binary files.

Usage: reader.py file [file ...]

Copyright (c) 2013, Joseph Lewis III <joseph@josephlewis.net> | <joehms22@gmail.com>
All rights reserved.

Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:

    Redistributions of source code must retain the above copyright notice, this 
    list of conditions and the following disclaimer.
    
    Redistributions in binary form must reproduce the above copyright notice, 
    this list of conditions and the following disclaimer in the documentation 
    and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND 
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES 
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON 
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

'''

import sys

ALPHABET_UPPER = frozenset("ABCDEFGHIJKLMNOPQRSTUVWXYZ")
ALPHABET = frozenset("abcdefghijklmnopqrstuvwxyz,-. \t\n")
NUMBERS = frozenset("0123456789")
OTHER_CHARACTERS = frozenset("!\"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~ \t\n\r")
PRINTABLE = frozenset('0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~ \t\n\r')
READ_BLOCK_SIZE_B = 2048

def type_of_char(c):
    if c in ALPHABET_UPPER:
        return 'A'
    if c in ALPHABET:
        return 'a'
    if c in NUMBERS:
        return 'n'
    return 'o'

def value_of_string(string):
    ''' A heuristic to give an approximate value to a string. '''

    # make sure the string isn't total junk
    if all([c in OTHER_CHARACTERS for c in string]):
        return 0

    types = [type_of_char(c) for c in string]
    runs = [0 for c in string]

    # higher runs are better
    for i in range(1, len(types)):
        if types[i] == 'o':
            runs[i] = 0
        elif types[i] == types[i - 1]:
            runs[i] = runs[i - 1] + 1
        else:
            runs[i] = 0

    return sum(runs)

def process_string(string, extracts):
    if len(string) < 5:
        return

    score = value_of_string(string)

    if score > 0:
        extracts.append((score, path, string))

def extract_strings(path, extracts):
    '''Extracts "interesting" strings from a file'''

    mystr = []

    with open(path, 'rb') as fd:
        chars = fd.read(READ_BLOCK_SIZE_B)

        while chars != "":
            for char in chars:
                if char in "\r\n":
                    mystr.append(" ")
                elif char in PRINTABLE:
                    mystr.append(char)
                elif len(mystr) == 0:
                    continue
                else:
                    process_string("".join(mystr), extracts)
                    mystr = []
            chars = fd.read(READ_BLOCK_SIZE_B)

    process_string("".join(mystr), extracts)


if __name__ == "__main__":
    if len(sys.argv) == 1:
        print("Usage: {} file [file ...]".format(sys.argv[0]))
        exit(1)

    # a list of tuples (score, document, string)
    extracts = []

    for path in sys.argv[1:]:
        extract_strings(path, extracts)

    extracts = sorted(extracts, reverse=True)
    for score, path, string in extracts:
        print("{:08.0f}\t{}\t{}".format(score, path, string))

Friday Operating System News

Operating Systems

NixOS is a neat little OS that is built around a single config file. The file can be moved to a different computer, and it'll set itself up as a clone, or as you're more likely to use it, as a single file that does version control on your system (so next time you upgrade, you can roll it back right away!)

Jolla has released it's first phones running Sailfish OS.

Finding Deleted JPEGs in Disk Dumps

If you've ever accidentally deleted items from your digital camera, or want to find them from a drive that has been lost, this script could be for you!

It works by looking through a disk image for a sequence of bytes that signify the beginning and end of a JPEG image.

Running It

First save the below script as ImageCarver.java

Then compile it:

javac ImageCarver.java

Afterwards, run it with the path to the disk image you want it to search:

java ImageCarver disk.img

It will report images it found, as well as the locations in the disk that it found them:

1.jpg 0xe3fa 0x10272
2.jpg 0xe000 0x20865
3.jpg 0x1000 0x2557e
4.jpg 0x41000 0x4437d
5.jpg 0x26000 0x654d4
6.jpg 0x25000 0x96816
7.jpg 0x86000 0x9eca0
8.jpg 0x6d000 0xdc614
9.jpg 0xe2000 0xe7ac9
10.jpg 0xa4000 0xeb28a
11.jpg 0xed000 0xfc9a9
12.jpg 0x136000 0x13b08e
13.jpg 0x1b5144 0x1b629e
14.jpg 0x1b5000 0x1b794a
15.jpg 0x1b67f0 0x1bea75
16.jpg 0x1c9000 0x1cfada
17.jpg 0x22d000 0x237824
18.jpg 0x281000 0x288928

The Script

import java.io.IOException;
import java.io.OutputStream;
import java.nio.ByteBuffer;
import java.nio.channels.SeekableByteChannel;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.StandardOpenOption;
import java.util.LinkedList;

/**
 * Carves images from a file given from the command line.
 * 
 * @author Joseph Lewis <joehms22@gmail.com>
 *
 */
public class ImageCarver 
{
    byte[] STARTING_BYTES = new byte[]{(byte)0xFF, (byte)0xD8,(byte)0xFF};
    byte[] ENDING_BYTES = new byte[]{(byte)0xFF,(byte)0xD9};
    int imgno = 0;


    /**
     * Carves images out of a file.
     * @param sbc - the byte channel to read from.
     * @throws IOException - on a file error
     */
    public ImageCarver(SeekableByteChannel sbc) throws IOException
    {
        // find all potential starting and ending points in the file.
        LinkedList<Long> startingPoints = ringMatch(sbc, STARTING_BYTES);
        LinkedList<Long> endingPoints = ringMatch(sbc, ENDING_BYTES);

        // keep writing out we're out of images
        while(startingPoints.size() > 0)
        {
            carveImage(sbc, startingPoints, endingPoints);
        }
    }

    /**
     * Carves an image out of the file.
     * @param sbc - the channel to read from
     * @param starts - the start positions of the image
     * @param ends - the end positions of the image
     * @throws IOException - on error
     */
    public void carveImage(SeekableByteChannel sbc, LinkedList<Long> starts, LinkedList<Long> ends) throws IOException
    {
        // clear off no longer useful end points
        while(starts.peek() > ends.peek())
        {
            ends.pop();
        }
    
        // If ends is empty, add the literal file end.
        if(ends.size() == 0)
        {
            ends.push(sbc.size());
        }
    
        // get our starting position.
        long sp = starts.pop();
    
        // Check to see if there's an image inside of us.
        if(starts.size() > 0 && starts.peek() < ends.peek())
        {
            carveImage(sbc, starts, ends);
        }
    
        // get our closest end point.
        long endloc = ends.pop();
    
        imgno++;
        createImage(imgno + ".jpg", sbc, sp, endloc);
    }

    /**
     * Extracts an image from the SeekableByteChannel.
     * @param name - the image name
     * @param src - the channel to read from
     * @param begin - the offset to the beginning of the image
     * @param end - the offset to the end of the image
     * @throws IOException - on error
     */
    public void createImage(String name, SeekableByteChannel src, long begin, long end) throws IOException
    {
        // alert the user about the image we're outputting
        System.out.println(String.format("%s 0x%x 0x%x", name, begin, end));
    
        // write out the given slice of bytes
        try(OutputStream os = Files.newOutputStream(Paths.get(name), StandardOpenOption.CREATE))
        {
            src.position(begin);
            ByteBuffer b = ByteBuffer.allocate((int) (end - begin));
            src.read(b);
            os.write(b.array());
        }
    }

    /**
     * Searches through a channel looking for a given array of bytes to match.
     * 
     * @param sbc - the channel to read from
     * @param toMatch - the bytes to match in the stream
     * @return a list of all offsets where the given bytes are found
     * @throws IOException - on file read error
     */
    public LinkedList<Long> ringMatch(SeekableByteChannel sbc, byte[] toMatch) throws IOException
    {
        // holds all found vars
        LinkedList<Long> finds = new LinkedList<Long>();
    
        for(long i = 0; i < sbc.size() - toMatch.length; i++)
        {
            ByteBuffer b = ByteBuffer.allocate(toMatch.length);
            sbc.position(i);
            sbc.read(b);
        
            boolean matches = true;
            for(int j = 0; j < toMatch.length; j++)
            {
                if(b.array()[j] != toMatch[j])
                {
                    matches = false;
                    break;
                }
            }
        
            if(matches)
            {
                finds.add(i);
            }
        
        }
        return finds;
    }

    public static void main(String[] args) throws Exception 
    {
        // Make sure we have an arg.
        if(args.length != 1)
        {
            System.out.println("Must supply exactly one argument.");
            return;
        }
    
        Path p = Paths.get(args[0]);
    
        try (SeekableByteChannel sbc = Files.newByteChannel(p)) 
        {
            new ImageCarver(sbc);
        }
        catch (IOException x) 
        {
            System.out.println("Caught exception: " + x);
        }
    }
}

Doing a Diff on a Specific Git Version

Sometimes you just need to look up something that changed between versions of a repository instead of reverting a full file, here two git commands come in handy.

The first shows the log of your recent commits, along with hex numbers which represent the versions:

git log

When you find the revision you want, just type:

git show af78ec673f46db23ae5537a8dbea1253e6e844f3

replacing the revision number with the one you want, and you'll get a diff from that version!

If A Robot Makes Your Makefile

If a robot makes your Makefile, it is no better than pressing that magic green arrow at the top of an IDE and watching the project compile.

LIST=CPU
ifndef RECURSE
RECURSE=recurse.mk
ifdef CONFIG
RDIR=$(dir $(CONFIG))
endif
endif
include settings.mk
include $(RDIR)$(RECURSE)

While a robot may easily understand this, it isn't any better than using g++ by hand.

Makefiles, in this programmer's humble opinion, should be clean, elegant, and easy to understand. They are the easiest way to immediately jump in to a project and figure out what is important and the way it is structured.
Even more importantly, they can point to dead ends and code that is never actually accessed by the program, keeping wasted days at bay.

A Simple Python Plugin Framework

This framework is somewhat different than those currently out there, most of those either:

• don't allow dynamic plugins
• require you to use nasty hooks
• require specifically formatted imports
• require plugins to all be in one place

This script however allows plugins to:

• be dynamically loaded
• be a subclass of a master plugin class
• have any name
• be in many different folders

The general idea is that you give the function a path, and the base-class that all plugins are subclasses of, then it iterates through the path, finding python files, loading them, and then searching them for any classes that are subclasses of the class you want. When it gets done, it returns them:

Code

import inspect
import os
import sys
def load_plugins(plugin_path, instance):
    '''Loads a set of plugins at the given path.

    Arguments:
    plugin_path - the OS path to look for plugins at.
    instance - classes of this instance will be returned
    '''
    plugins = []
    plugin_dir = os.path.realpath(plugin_path)
    sys.path.append(plugin_dir)

    for f in os.listdir(plugin_dir):
        if f.endswith(".py"):
            name = f[:-3]
        elif f.endswith(".pyc"):
            name = f[:-4]
        elif os.path.isdir(os.path.join(plugin_dir, f)):
            name = f
        else:
            continue

        try:
            mod = __import__(name, globals(), locals(), [], 0)

            for piece in inspect.getmembers(mod):
                if isinstance(piece, instance):
                    plugins.append(piece)

        except ImportError as e:
            print(e)
            pass # problem importing

    return plugins

Friday Link List -- Ubuntu 13.04 Backend Updates and Good News for the Web

Real Dev News for Ubuntu 13.04

This is way better than the stuff you'll find on OMGUbuntu!

1. gvfs updates with MTP Support (you can now connect your Android 4 device and it just works)
2. LibreOffice 4.0; a huge number of bugfixes went in to this
3. cups updates to now auto-detect shared printers on your network, and auto-share them with other devices including iPads/iPhones etc.
4. Much of the backend has been ported to Python 3 in hopes of totally removing Python 2.
5. Lots of work has gone in to sandboxing applications from one another (most likely due to the phone release)
6. A potential new feature (if it makes it in time) is automatic removal of old kernels, which can take up a few gigs of space over the course of six months if not cleaned manually.
7. New upstart version allowing userspace (beta) jobs, and launch on file/folder change, hopefully meaning fewer background services and less clunky code.
8. Fixes for quite a few RAM hogs that persist across sessions.

Web

• jQuery 2.0 has been released it drops support for IE 6-8, it looks like old Android (2x) is going to get the chop next.
• Chrome forked WebKit to form Blink, they say they were able to drop about 2.5 million lines of code right off the bat, hopefully it'll speed up their iteration time.
• Firefox has released it's baseline compiler tl;dr version, it cuts down on complexity, and does a much better job of supporting IonMonkey being it's based on the same backend giving a huge speed boost.
• ASM.js promises a good future for porting C/C++ apps to the browser with support from Firefox, and Chrome in the works.

Things I Learned From Working A 2 Year 50,000 Line Project

This is mostly to serve as a brain dump for a piece of software I've been writing for around two and a half years, it was comprised of a large database (read: ~five hundred million rows), a cross platform Java application, a large Python web-server/API, a phone app, lots of small scripts that fetched and transformed data, and globs of glue. (I wrote all of them in the two year period!)

Architecture

• It pays to fully think through your APIs first
• All web APIs should have a version in the URL e.g. /application/api/2.0/...
• Profile early and often
• A few good caches are worth many bad ones
• Make everything a plugin that extends one class; you may think it's a big hairy mess, but it's better than multiple plugin systems.
• Use libraries wherever possible
• Ditch libraries when you only use one function from them

Debugging

• Users write awful bug reports
• You can help them write better ones by giving them a short form to fill out.
• Automatically attach debugging info to all bug reports
• Make version numbers easily available (not just in the executable's name)
• Auto-update your apps so you don't have users lagging behind

Upgrades

• Your requirements will constantly change, make sure you have an easy upgrade path in your database that won't force hacky solutions.
• Make sure your ORM supports incremental changes.

User Experience

• Start fast, even if you're not done processing yet
• Never trust an engineer to do design (including software engineers)
• If the user doesn't see it, it doesn't exist
• Make it very hard for the user to do stupid things
• Do automatic backups/saves

Sanity

• Clients like saying "I did say that, didn't I? What I meant was..."
  • all features should be plugins so you can enable/disable when they do this
• SVN and Eclipse are a nightmare to get playing nice
• Java 7 is really awesome!
  • auto-closing streams
  • multiple catches in a try block
  • string in switch statements
• JNLP sucks, but so does embedding apps in-browser; just make it easy for the user to download it, and provide auto upgrading

Covertly Sending Messages Using DNS Queries

I won't be the first to post about this, but I read it a while back and thought it sounded fun: note, this post will be more technical in nature than most, but I hope it will be a fun read, and a fun toy.

Let's say you want to send short messages that are very covert and not persistent. You could, of course use Twitter, sending strange cryptographic tweets, but that may be too obvious for your tastes, plus they can more easily be traced back to an IP/Person if just one slip-up is made. Somehow using DNS would be an awesome way to do this because:

• It can be accessed by anyone
• Messages expire over known intervals
• No accounts are necessary
• There are huge amounts of data to go through and junk requests that will look just like yours if someone wanted to find you out
• Google has a very nice public DNS you can use for free at 8.8.8.8 or 8.8.4.4

Specifics

The two types of queries we'll be using here are standard and non-recursive DNS queries.

Standard queries are the ones that your computer normally does to change a hostname in to an IP address it can contact, to do this from a command line you'd type:

$ dig @8.8.8.8 www.google.com

To which the DNS server responds:

; <<>> DiG 9.8.1-P1 <<>> @8.8.8.8 www.google.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 5459
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.google.com.            IN  A

;; ANSWER SECTION:
www.google.com.     300 IN  A   74.125.225.179
www.google.com.     300 IN  A   74.125.225.180
www.google.com.     300 IN  A   74.125.225.176
www.google.com.     300 IN  A   74.125.225.177
www.google.com.     300 IN  A   74.125.225.178

;; Query time: 53 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Apr 12 10:18:31 2013
;; MSG SIZE  rcvd: 112

This shows you all of the numerical IP addresses you can reach google.com at, if you copy and paste the first one in to your browser, you'll see Google's home page.

That's all well and good, but how would you actually store information using DNS? For this you'll need to generate a domain name that doesn't exist, we'll be using www.testdnsflagsetting.com for the purposes of this article:

Now do a dig on it:

$ dig @8.8.8.8 www.testdnsflagsetting.com +norecurse

The response looks different this time, note that ANSWER: 0 and AUTHORITY: 0, this means the request is not authoratative (8.8.8.8 doesn't know this informatoin for sure), and there was no answer.

This is because we passed in the +norecurse flag to dig, asking the server to just reply with what was it had stored rather than asking higher up in the chain of command:

; <<>> DiG 9.8.1-P1 <<>> @8.8.8.8 www.testdnsflagsetting.com +norecurse
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 23556
;; flags: qr ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.testdnsflagsetting.com.    IN  A

;; Query time: 57 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Apr 12 10:04:10 2013
;; MSG SIZE  rcvd: 44

Now we'll do the same thing, but without the +norecursive:

$ dig @8.8.8.8 www.testdnsflagsetting.com

; <<>> DiG 9.8.1-P1 <<>> @8.8.8.8 www.testdnsflagsetting.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 22889
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;www.testdnsflagsetting.com.    IN  A

;; AUTHORITY SECTION:
com.            900 IN  SOA a.gtld-servers.net. nstld.verisign-grs.com. 1365782647 1800 900 604800 86400

;; Query time: 82 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Apr 12 10:04:21 2013
;; MSG SIZE  rcvd: 117

Note that the AUTHORITY: 1 now is true, meaning that 8.8.8.8 checked with the server that knows about all .com addresses to see if ours existed. There is also a new section, note the number 900 in it, that means this response is going to be saved in Google's DNS (8.8.8.8) for 900 seconds.

If we go back and do the +norecurse version again, we now get the cached response, but with 890, meaning there are only 890 seconds left until the cache is cleared:

$ dig @8.8.8.8 www.testdnsflagsetting.com +norecurse

; <<>> DiG 9.8.1-P1 <<>> @8.8.8.8 www.testdnsflagsetting.com +norecurse
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 14832
;; flags: qr ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;www.testdnsflagsetting.com.    IN  A

;; AUTHORITY SECTION:
com.            890 IN  SOA a.gtld-servers.net. nstld.verisign-grs.com. 1365782647 1800 900 604800 86400

;; Query time: 39 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Apr 12 10:04:31 2013
;; MSG SIZE  rcvd: 117

Put these together now, and say you want to send a true or a false to someone on the other side of the globe, you both just have to agree on a time to check, and a domain, then if you want to say true you do a recursive query, and the other person will get back an authoritative record when they do a non-recursive; if you don't do the query, they will not and they'll know you sent false

Sending a Whole Message

Now say you want to send a whole message! Convert the text to ASCII, and break it down to bits, use the first byte to tell how long the message will be (up to 255 characters).

Say you just wanted to send the message Hi, that means you'd send a 2 then Hi, but you can't use the same domain the whole way through, you'll have to agree on a bunch of domains, or a way of randomly generating them. Here we'll just append a number to the end:

Letter:       2        H        i
Binary:   00000010 01001000 01101001
POsition:            111111 11112222
          01234567 89012345 67890123

Your queries would look something like this:

dig www.testdnsflagsetting6.com
dig www.testdnsflagsetting9.com
dig www.testdnsflagsetting12.com
dig www.testdnsflagsetting17.com
dig www.testdnsflagsetting18.com
dig www.testdnsflagsetting20.com
dig www.testdnsflagsetting23.com

You only actually dig for the 1s, because they are represented by true.

Getting The Message Back Out

Now to get the message back out, you just need to do a dig for positions 0-7, and convert that in to the number of letters in the message, multiply that by 8 to get the number of bits you need to check, and check the domains ending with all of those numbers.

First 8

dig www.testdnsflagsetting0.com

dig www.testdnsflagsetting1.com

...

dig www.testdnsflagsetting7.com

This would give us the number 2, meaning there were 2 more sections of 8, then you read on.

A program to do it!

This is a short Python script to do it, you may want to change the DOMAIN_NAME. in case other people are using the script too.

#!/usr/bin/env python3
''' A program to send messages via DNS queries.
Copyright 2013 Joseph Lewis <joehms22@gmail.com> | <joseph@josephlewis.net>

MIT License
'''

import subprocess

DNS_SERVER = '8.8.8.8'
DOMAIN_NAME = "www.testdnsflagsetting{}.com"
NORECURSE_OPT = "+norecurse"

msg = raw_input("Enter a message, or blank to receive: ")

def read_byte(byteno):
    byte = "0b"
    for i in range(byteno * 8, (byteno + 1) * 8):
        output = subprocess.check_output(['dig','@{}'.format(DNS_SERVER), DOMAIN_NAME.format(i), NORECURSE_OPT]) 
        if ";; AUTHORITY SECTION:" in output:
            byte += '1'
        else:
            byte += '0'

    return int(byte, 2) # converts binary to an int

def write_byte(byteno, byte):
    to_write = bin(byte)[2:].zfill(8) # gets binary representation of a byte
    for loc, b in enumerate(to_write):
        if b == '1':
            i = (byteno * 8) + loc
            subprocess.check_output(['dig','@{}'.format(DNS_SERVER), DOMAIN_NAME.format(i)])
            print "Wrote 1 at: {}".format(i)

if len(msg) == 0:
    message = ""
    for byte in range(1,read_byte(0) + 1): # first byte is length of message
        message += chr(read_byte(byte))

    if len(message) > 0:
        print message
    else:
        print "[No Message]"

else:
    total = len(msg)
    write_byte(0, total)
    for loc, char in enumerate(msg):
        write_byte(loc + 1, ord(char))

    print "Message written"

HOWTO Not Worry About IE6 Ever Again

While IE6 market share is dwindling, we are far from being out of the woods yet. Gecko, Trident, WebKit and Blink, are likely to never convirge on a full implementation of W3C specifications, leaving some browsers more equal than others.

Not to worry though, there is a nice little script called Modernizr, that can auto-detect features of a browser and automatically load fixes for browsers that don't have the technologies you need.

Shortlist of Features Modernizr Can Detect

• HTML5 Canvas
• IndexedDB
• WebWorkers
• Lots of CSS3 Stuff
• Geolocation
• WebSockets

Coding is a Drug

Coding is highly addictive. I suspect it has to do with the combination of the brain being the bottleneck, like all good puzzles; rapid iteration allowing you to see a result slowly emerge, and the fact that once you get done, you'll have something tangable as a reward that will serve you for a long time.

It's somewhat like combining crossword puzzles, woodworking, and sketching.

HowTo Calculate an MD5 hash in Java

If you're dealing with networking, or just want to make sure your files don't get corrupted/modified between times your app runs.

It is critical to use the algorithm.reset() command before using a MessageDigest instance, as if there are bytes still remaining from a prior digest, it will completely ruin your output.

Code

import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;

// calculates an MD5sum for a given bit of text
public class MD5 
{
    public static void main(String[] args)
    {
        try 
        {
            MessageDigest algorithm = MessageDigest.getInstance("MD5");
            algorithm.reset(); // clear the digest back to starting state.

            // Get selected digest of the given text.
            byte[] digest = algorithm.digest( "Hello, world!".getBytes() );

            System.out.println(byteArrToHex(digest));
        } 
        catch (NoSuchAlgorithmException e) 
        {
            // no such algorithm, MD5
        }
    }

    // converts bytes to text.
    private static String byteArrToHex(byte[] bytes)
    {
        StringBuilder sb = new StringBuilder();
        for(byte b : bytes) 
            sb.append(String.format("%02X", b)); 
        return sb.toString().toLowerCase();
    }
}

Other Notes

• To calculate SHA1 hashes instead, use SHA1 as the instance type.
• The NoSuchAlgorithmException likely won't be thrown on desktop platforms, but it is possible that certain algorithms won't be implemented in certain areas due to them being considered military grade encryption; SHA1 and MD5 shouldn't have this problem because they are just simple hashes.

The Four Keys To Good Design

The points here are stolen from The Design of Everyday Things which you should definitely purchase from the publisher.

There are four major design aspects that all interfaces should have, many already exist in the physical world due to physical processes, but need to be replicated in the visual world for users to feel in control and comfortable.

Visibility

At all times, the user should be able to quickly see what is going on and what options they have.

• Change icons when a process is running 
• Give timers estimating how much time is left 
• Gray out options that can't be chosen 
• Give context-sensitive menus for a given context one of the best I've seen here is the Adobe Flash Designer whose entire bottom toolbox changed based upon what you had selected, yet it was still predictable.

A Good Conceptual Model

This is one of the biggest issues with design, especially in complex systems that could never be made in the physical world.

Email is a nice example, you open an "Inbox" and read "Mail" that was "Sent" to you. Of course, you don't want to tell your users that email is mail, otherwise you end up with the dreaded "Please email me back the file, I sent you the only copy I have".

But the abstraction does remove doubt from the user's mind that the process is stable and will work, even if it doesn't work in the way that they believe or you portray.

Good Mappings

Every action should have an immediate and visible response, if the user presses "Send", don't wait on the message screen until the message is sent to return the user to the Inbox. Show the user what happened whenever they do something so they can build relations in their mind about what that is doing in software.

Feedback

It doesn't matter so much that your software does computation, it just needs to inform the user of that or keep the user occupied while it is doing it; much in the same way that Mac OSX 10.6 shows an intro video: All new operating system installs need to do routine checks and get themselves in order, but there is surely a better way than this:

HOWTO Quickly Create PHP Forms with phpMyEdit

One of the biggest hassles of building a site in PHP is that there is no way to easily generate forms for your database. The same is true for most systems, CRUD mechanisms take most of the design time, even when the core logic is simple.

In PHP there is a simple program called phpMyEdit that takes the time out of creating these forms. Unfortunately, it is rather old and only works with MySQL, but for most projects that is what you need.

To Setup

1. Download phpMyEdit
2. Extract the contents in to the directory of your site.
3. Go to: http://example.com/phpMyEdit-5.1/phpMyEditSetup.php replacing 5.1 with your version of phpMyEdit.
4. Enter your database credentials and server.
5. Choose your database.
6. Choose your fields.
7. Create the form!

You'll want to secure the forms, of course, behdind something that checks for administrator rights, but that is about it!

Hacking Respondus LockDown Browser 2.0

It has been about a year since I've visited this topic, and I've decided that it is indeed worth revisiting. But first, a few reasons that I'm doing this so nobody gets the wrong idea:

• To prove Respondus LockDown Browser does not circumvent cheating
• To encourage anyone left using this technology to change their assessment methodology to other means, such as projects because:
  • They are more effective
  • It is just as easy to determine if a student cheated
  • They test ability to apply instead of memorization (life is open book, what you need to know is how to apply what you know!)
• To boost blog viewership (I write to be read, and Respondus is one of my most read posts)

The Technique

I have devised a better way of "hacking" Respondus that does not involve the use of monitoring system calls, although that would be equally viable and was what I had originally intended to do.
This version works better, is cleaner, and anyone can edit it when it stops working. Essentially, it just modifies Respondus' window and opens other programs with button presses.

How To Run

1. Download the code, and save it to a file on your desktop called Respondus.ahk
2. Download and install Auto Hot Key
3. Close all other windows on your desktop.
4. Right click the Respondus.ahk file and choose "Run As Administrator"

Code tested on Windows 8/XP. You might need to change the first line in 64 bit versions of Windows 7/Vista to reflect another path to LockDown.exe.

Code

Run, C:\Program Files\Respondus LockDown Browser\LockDown.exe

WinWait, Respondus LockDown Browser
WinSet, AlwaysOnTop, Off, Respondus LockDown Browser
WinSet, Enable, , Respondus LockDown Browser


Gui, Add, Button, default, &Show IE
Gui, Add, Button, default, &Hide Respondus
Gui, Add, Button, default, &Show Respondus
Gui, Add, Button, default, &Maximize All
Gui, Show,, Subversion Menu

WinSet, AlwaysOnTop, On, Subversion Menu
return

ButtonShowIE:
Run, IEXPLORE.EXE http://onehourhacks.blogspot.com
WinWait, One Hour Hacks - Windows Internet Explorer
WinMove, One Hour Hacks - Windows Internet Explorer, , (A_ScreenWidth/2), 0, (A_ScreenWidth/2), (A_ScreenHeight),,
return

ButtonHideRespondus:
WinHide, Respondus LockDown Browser
return

ButtonMaximizeAll:
WinGet, WindowList, List
Loop, %WindowList%
    {
    WinMaximize, % "ahk_id " . WindowList%A_Index%
    }

WinHide, Respondus LockDown Browser
return

ButtonShowRespondus:
WinShow, Respondus LockDown Browser
return

ButtonLeftSideRespondus:
WinMove, Respondus LockDown Browser, , 0, 0, (A_ScreenWidth/2), (A_ScreenHeight),,
return

ButtonFullScreenRespondus:
WinMove, Respondus LockDown Browser, , 0, 0, (A_ScreenWidth), (A_ScreenHeight),,
return

GuiClose:
ExitApp

The Case Against Customizability

Three words on building customizability in to your product: don't do it. This is to the products that allow users to change colors, organize panels, choose icons, etc.

Users have a hard enough time with finding functionality

Watch your technophobic family members try to attach an image to an email. While you may have no problem getting through menus to do it, they often do. Remember: there are more of them than us.
Customizability just gets in the way for super-users because they have better ways of doing it.

Your Product is a Tool, Not a Piece of Art

Your goal as a designer should be to minimize the amount of time on your software because it should just work.
Get your users back to the real world where they can go create art or enjoy it.

Troubleshooting/Documenting is Hard With Customizability

When you write tutorials, it is easy when everything is the same. Customizability makes it harder to follow tutorials, especially if the writer has a custom setup.
Users will also find it harder to help one another, which means more support is needed from you!

It Leads to Messy Code

Just think about rearranging components on the web, you need a place to save the information, a new server interface for submitting it, security procedures over that to make sure it isn't an attack vector, another library user-side to test in every browser.

Most of the Users Will Never Do It

Take BlackBoard for example, I TA a bit, and of all the students I've seen pull up blackboard, not one of them has changed the style or layout as they're allowed to; nobody wants to spend the time on a product they use twice a day.

Your time is better spent on usability

iGoogle was a great example of this, instead of needing a homepage like they used to, users can now just type what they want in to the search box and get information relevant to them.
Spend your time figuring out why users would need to customize, and eliminating that need.

Final Note

I'm not advocating that you throw everything in to code, no no! I fully support MVC, that is how superusers can change your environment to suit them best. If you must allow user tweaks to your software, like moving around menus, put it in a plugin so when your code gets updated, the whole software doesn't break.

HOWTO Find the Number of Pixels in a MM with Javascript

This is a simple trick to find the number of pixels in a millimeter using jQuery.

Code

function pixelsPerMM()
{
    $("body").append("<div id='onebyone' style='width:1mm;height:1mm;display:hidden;'></div>");
    var pixels = $("#onebyone").width();
    $("#onebyone").remove();
    return pixels;
}

How It Works

1. Adds a new element that is 1mm x 1mm to the page.
2. Requests the width (in pixels) of that element.
3. Deletes the element.

It is probably a good idea not to call this repeatedly in your code, as changing the web document in this manner is slow.

Why Use It

If you have an interface that you want to be exactly the same on every device, but you can't lay out the entire thing using CSS i.e. some elements are dynamically generated using Javascript, this would be a good method.

When You Are Justified in Complaining About FOSS

LibreOffice 4.0 has recently been released, and has received a lot of unfounded criticism over its new ability to be themed using Firefox Personas. Now would be a good time to go over when it is acceptable or not to complain about FOSS, as there seems to be quite a lot of confusion about that.

Most of the time, you should not complain about FOSS like this because:

• You got it for free.
• If you don't like it, you don't have to use it.
• You are encouraged to fix the things that annoy you.
• You did not do any work.

However, there are a few cases where you are allowed to complain about FOSS:

• When the product destroys your data.
• When there is no way to contribute to the project other than forking it. (e.g. Android)
• When you have contributed code/patches that are widely desired, but the maintainers will not accept them.
• When you are paying the developers to make changes, and they are not.

Back to Personas!

All of the complaints the LO team have received over this issue are unfounded because the software is free, does not force you to update to the newest version, has a remarkably open ecosystem (1500 merges from 3.0 to 4.0), and the product still works without ever using that feature.

Not only does LO 4.0 work better than the 3.X branches, it is vastly superior in speed, supported formats, and code maintainability.

This whole issue will seem silly in two months when Ubuntu releases a branded LO using a theme that adds subtleties to the document editor that perfectly integrate it with the platform, but until then hopefully these guidelines will suffice for indignant posters.

HOWTO Make a Quick and Dirty (Pseudo)Random Number Generator

If you ever need to build your own quick and dirty (pseudo-)random number generator it actually isn't that hard. This generator is used in various real-world systems:

• Visual Basic (to version 6)
• A few ANSI C implementations, including glibc
• Java's Random library

The implementation will iterate over all numbers [0,m) when using proper a, m ,c's. ¹

Implementation

#include <stdio.h>
#include <unistd.h>

const long m = 4294967296; // 2 ^ 32
const long a = 1103515245;
const long c = 12345;

long lastX = 0;

long nrandom() {
    lastX = (a * lastX + c) % m;
    return lastX;
}

void seed(int num)
{
    lastX = num;
}

int main()
{
    int i;

    seed(getpid());

    for(i = 0; i < 1000; i++)
        printf("%li\n", nrandom());
}

Note that this implementation only generates m pseudorandom numbers, in this case 4,294,967,296; so if you need more, you'll want to use a different algorithm.

---

1. These conditions require some discrete mathematics that are too long to go in to here; it involves coprime numbers and recurrence relations. (Please point out if I've missed some simple explaination in the comments, I'm fairly new to this!)↩

HOWTO Fix Flash in Chrome + Ubuntu 12.10

After a recent Chrome update, I was left without flash for this browser, which by the way handles Pandora way better than Firefox does on the same platform.
A quick fix is all that is needed:

1. Type chrome://plugins/ in to your URL bar.
2. At the upper right, press "Details"
3. Scroll down to the Adobe Flash Player entry, there should be two sub-entries, these are versions of Flash Chrome can use.
4. Disable the entry that looks like this (should be the first one):
   

   Name:   Shockwave Flash
   Location:   /home/user/.config/google-chrome/PepperFlash/11.5.31.138/libpepflashplayer.so
   Type:   PPAPI (out-of-process)
   
   MIME types:
   MIME type                       Description         File extensions
   application/x-shockwave-flash   Shockwave Flash     .swf
   application/futuresplash        Shockwave Flash     .spl




5. Restart the browser.

For some reason the flashplayer using the pepper API appears not to be working, so you'll just have to stick with the one that was installed to your platform through the package manager (a safer bet anyway).

HOWTO Use Python "for in" and "for else" Loops

This is a short guide to the subtleties of Python for loops, especially for those programmers from Java/C++ that may miss some of the best features because those languages don't have any equivalent, it covers:

• The basic for loop
• Looping over dictionaries
• Using the for else loop

The Basic for loop

If you come from a Java/C++/PHP background, you're used to two different for loops foreach and for. Python only has one of these, the more powerful foreach.
In Java, your for loop would look something like this:

for(int i = 0; i < 10; i += 2)
{
    System.out.println(i);
}

In Python, the same loop would look like;

for i in range(0, 10, 2):
    print(i)

The range function returns a list of numbers 0 to 10 incremented by 2.
If you wanted 0-9 inclusive, the loop would be even more simple.

for foo in range(10):
    print(foo)

In this same manner, you can replace the range() with any list:

for j in [1,1,2,3,5]:
    print(j)

Looping over Dictionaries (maps)

Maps are a very handy Python builtin feature, allowing you to create arbitrary key->value pairs.

{
    "key":"value",
    "bob":"smith",
    "age":13,
    "dob":(2012,01,01)
}

You can quickly loop over key->value pairs in a dictionary by using the following:

for key, value in a.items():
    print("{} -> {}".format(key, value))

Note that this is much faster than looping over the keys, then looking up the value for each key inside the for loop. The items function returns a list of tuples where the first value is the key, and the second a value.

In fact, you can loop over any list of tuples in a similar manner:

b = [("Perro", 1, ["Canis", "lupis"]),
 ("Gato", 2, ["Felis", "catus"])]

for name, id, scientific in b:
    print("Name: {} ID: {} Breed: {}".format(name, id, " ".join(scientific)))

The Magic of the for else Loop

If you've ever wanted to check if something happened within a for loop you can use an else block after your for loop:

for i in ["foo","bar"]:
    if i == "baz":
        break
else:
    # happens if break is not called, meaning "baz" was not found
    return -1

# i is set to the location of "baz"
return i

The example above tries to find the location of a specified string in a list of strings, if found it returns the index, otherwise -1. Of course there are far easier ways to do this, but you get the idea.

HOWTO Debug Crashes in C/C++ Applications on Ubuntu

In this howto we'll cover:

• Compiling C/C++ code for debugging
• Allowing debugging
• Viewing errors
• Fixing common gdb issues

Your computer is happily humming along and your program is progressing fine, suddenly disaster strikes! Segmentation fault (core dumped) your computer yells before it plunges back to darkness and your friendly shell prompt reappears.

The way of debugging in Ubuntu when coding by hand is not nearly as nice as popping up an IDE, but when done right can be much faster.

Our Problematic Code

For demonstration we'll be using this bit of code that is written to cause crashes.

int main()
{
    int* p = 0x0000007b; // The cause of many a Windows XP BSOD
    int j;

    for(j = 0; j < 10000000; j++)
    {
        p++;
        *p = j;
    }

    return 1;
}

In order to get the best debugging results you'll have to compile your code with the -ggdb flag, in this instance: gcc -ggdb killer.c.
If you run this program, it will die nearly right away with the error: Segmentation fault (core dumped).

The -ggdb flag to the compiler instructs it to include lots of debugging information. However, you won't want this on production executables because it takes up a lot of space. This example program was 9.6K with symbols, and 6.2K without!

Enabling the Dump

By default, Ubuntu 12.10 won't output crash information for programs you make yourself, to fix this you'll need to run the command:

ulimit -c unlimited

This will need to be run every time you log back on to your system.

Debugging

Once you have logging enabled and your program crashes a file called core should appear in the directory from which you ran the program. Use the gdb command to see what information it contains about the crash.

gdb a.out core

Where a.out is the name of your program that crashed and created the core file.

Analyzing the Debug Output

GNU gdb (GDB) 7.5-ubuntu
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/joseph/Desktop/a.out...done.

[New LWP 13663]

warning: Can't read pathname for load map: Input/output error.
Core was generated by `./a.out'.
Program terminated with signal 11, Segmentation fault.
#0  0x00000000004004ed in main () at breakme.c:9
9           *p = j;

This is the output you'll get from the gdb command, all you want is the last three lines, from the top they tell you:

1. What happened.
2. Where the error happened (in main in file breakme.c on line 9)
3. What the line was.

Common gdb Errors

• warning: exec file is newer than core file. means the core file you're using wasn't made by your executable, you'll need to delete it and run your program again.
• warning: core file may not match specified executable file. means the core file you're running against probably wasn't made by the program you specified, gdb will probably report wrong results.

Extended Euclidean Algorithm in Python

The Extended Euclidean Algorithm is a simple extension to the Euclidean Algorithm that in addition to returning the greatest common denominator also gives numbers s and t such that gcd = s * a + t * b.
Specifically, you may want to find 1 = sa + tb which is very useful in cryptography.

The Code

def extended_euclidean(a,b):
    origa = a
    origb = b

    x = 0
    lastx = 1
    y = 1
    lasty = 0

    while b != 0:
        q = a // b
        a, b = b, a % b

        full = "%d = (%d * %d + %d * %d) - %d * (%d * %d + %d * %d) " % ( b, origa, lastx, origb, lasty, q, origa, x, origb, y )
        short = "%d = %d * %d + %d * %d" % ( b, origa, lastx - x * q, origb, lasty - y * q)
        print("%s\t%s\t%s\t%s" % (a, b, full, short))

        x, lastx = (lastx - q * x, x)
        y, lasty = (lasty - q * y, y)

    return (lastx, lasty)

The Output

Take the numbers 120 and 23:

>>> extended_euclidean(120,23)
23  5   5 = (120 * 1 + 23 * 0) - 5 * (120 * 0 + 23 * 1)     5 = 120 * 1 + 23 * -5
5   3   3 = (120 * 0 + 23 * 1) - 4 * (120 * 1 + 23 * -5)    3 = 120 * -4 + 23 * 21
3   2   2 = (120 * 1 + 23 * -5) - 1 * (120 * -4 + 23 * 21)  2 = 120 * 5 + 23 * -26
2   1   1 = (120 * -4 + 23 * 21) - 1 * (120 * 5 + 23 * -26)     1 = 120 * -9 + 23 * 47
1   0   0 = (120 * 5 + 23 * -26) - 2 * (120 * -9 + 23 * 47)     0 = 120 * 23 + 23 * -120
(-9, 47)

HOWTO Setup Automatic Code Hilighting for Websites (and Blogger)

This is actually the method that this blog uses! The scripts below quickly find all code on a page, auto-detect what language it is, and highlight it accordingly.

Setting Up On Blogger

To set up on Blogger all you need to do is the following:

1. Open the "Template" menu on the left side of your blog's dashboard.
2. Press the "Edit HTML" button below your template.
3. Just copy and paste the following code right before the </html> tag.

<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js" type="text/javascript"></script>
<script src="http://balupton.github.com/jquery-syntaxhighlighter/scripts/jquery.syntaxhighlighter.min.js" type="text/javascript"></script>
<script type="text/javascript">
$('pre').each(function() {
$(this).addClass( "highlight" );
});
$.SyntaxHighlighter.init();
</script>

On Your Website

Alternately, you can add this to your website for automatic code finding and hilighting! Just make sure to add it to the very bottom of your page, or add the third Javascript code block that runs once the page is loaded.

If all of your code is in <code> blocks, just change the pre on line four to code.

Side Note: You probably want to use your own versions of jQuery and jQuerySyntaxHilighter so you can update them yourself.

Extract Email Attachments With Python

If you archive your email messages, like me, you may find that you want to pull out all of the attachments for those files so your desktop search will parse them better, or so you can quickly search through them.

This is a simple script that just recurses through your .eml messages in a directory and pulls out all of the base64 encoded attachments.

For those of you that are wondering what base64 is, it's an encoding that only uses sixty-four different characters to transmit information. The email system uses this to send documents around so that the protocol didn't have to be reconfigured to account for stuff that wasn't text.

Code

#!/usr/bin/env python

import email.parser
import os
import sys
import base64


fileList = []
rootdir = "/path/to/.eml/messages/"
for root, subFolders, files in os.walk(rootdir):
    for file in files:
        fileList.append(os.path.join(root,file))

id = 0

for path in fileList:
    if not path.endswith(".eml"):
        continue

    fp = email.parser.FeedParser()
    fp.feed(open(path).read())

    message = fp.close()

    for message in message.walk():
        fn = message.get_filename()
        if fn == None:
            continue

        try:
            with open(fn, 'wb') as out:
                out.write(base64.b64decode(message.get_payload()))
        except TypeError:
            with open(fn, 'wb') as out:
                out.write(message.get_payload())

Extensions

• This script isn't very efficient being that it uses python to decode.
• It would be nice to pull arguments from the command line using sys.argv

Update 2013-09-04 Python 3

#!/usr/bin/env python3

import email.parser
import os
import sys
import base64
import binascii
import sys


def extract(rootdir):
 fileList = []
 
 for root, subFolders, files in os.walk(rootdir):
  for file in files:
      fileList.append(os.path.join(root,file))

 for path in fileList:
  if not path.endswith(".eml"):
      continue

  fp = email.parser.BytesFeedParser()
  fp.feed(open(path, "rb").read())

  message = fp.close()
  
  print("Checking {}".format(path))

  for message in message.walk():
      fn = message.get_filename()
      if fn == None:
          continue
      try:
          try:
              with open(fn, 'wb') as out:
                  out.write(message.get_payload(decode=True))
          except (TypeError, binascii.Error):
              with open(fn, 'wb') as out:
                  print(message.get_payload())
                  out.write(bytes(message.get_payload(), message.get_charset()))
      except Exception:
          print("Error extracting item from {}".format(path))

if __name__ == "__main__":
 if len(sys.argv) == 1:
  print("usage: {} path/to/.eml/files".format(sys.argv[0]))
  exit(1)
 extract(sys.argv[1])

HOWTO Redirect A Page Using Javascript

Occasionally you need to move a page on a website from one location to another, but don't have the ability to update all of the links that point to the old one or access to the server software to do the same thing.

All you need is a page with a little Javascript to solve the problem.

Example

<html>
    <head>
        <title>Redirecting...</title>
        <script type='text/javascript'>
            window.location.replace("http://your.site.here");
        </script>
    </head>
    <body>
        <p>Redirecting...if you are not redirected click 
        <a href='http://your.site.here'>here</a>.</p>
    </body>
</html>

Notes

• Include a link and Javascript to redirect for users that have JS turned off.
• Use window.location.replace instead of window.location.href so the user doesn't get stuck in a loop when they press the back button.
• Use window.location.replace instead of document.location
• Insert if(top != self) top.location.replace("http://your.site.here"); in the top of the <script> tag to escape from frames.

Ubuntu and the Art of Deception

I am, and have been an Ubuntu user for almost six years; over the past four, I've given up Windows entirely (skipping over Vista and 7, save for rescuing family members' PCs occasionally). However, the more and more Ubuntu grows the more I notice discrepancies between the Linux that I loved originally, and the current incarnation.

The Deception

In the most recent release, the biggest change was the addition of a web API that interfaces with the desktop. Don't get me wrong, this is great, but a very underwhelming thing to take SIX MONTHS to develop. Had I wished to do so, I could have perfected the same thing (because the back-end is already totally there with DBus) in about forty hours of work. So, if Canonical had one developer working on it, for one week, it would have been done. What happened the rest of the time?

When Shuttleworth gets out and states all of the great things that came to us in 12.10 the major ones were these:

• Webapps - as discussed about 40 hours of time
• Online accounts - already integrated in Gnome, a simple port was all that was needed
• Dash Previews - essentially a few more plugins to the dash, which should be about a week of programming, especially if you use the utilities already available, like the "file" command
• Easy full disk encryption - Essentially a single change to the installer
• Ditching Unity 2D - this should have freed up a lot of developer time!
• And the hotly contested Amazon Search Results - advertisements for your PC that I'm sure Amazon would have implemented due to all the free advertising

The News

So, what did the news sites report on for six months? Theme changes, new button gradients, new backgrounds, etc. All the real work I could have done quicker, and not included the ads.

When all of this came out, webapps were touted as a great fusion of desktop and web, where in reality, they work on when the pages are already open, saving a few clicks at best, and opening your system to security flaws at worst. Why not support W3C apps? That would be a huge boon to the software overall, and a push to help Tizen. What about supporting Firefox apps? Quick to develop, quick and open to deploy.

The Amazon advertisements are a huge privacy concern, according to the EFF, rather than some magical new way to shop.

The Future

So, what does this all mean? Ubuntu has reached the feature creep stage, because it's founder won't take risks; maybe he's outpaced himself with a six month development cycle instead of continuous integration, but not enough is changing other than seeming regressions that users have to fight with each subsequent "release", but a great number of users refuse to admit this, probably because they don't know enough how quickly these changes could be made to a well designed system.

Perhaps Shuttleworth has spread himself too thin; rather than shoring up upstream apps and using them (i.e. docky could have easily been used instead of unity) he has created his own; rather than creating a unified configuration file set, we have a hack that works with each individual application.

It would be nice to see a desktop, particularly the most powerful one, contribute far more to the small projects that make it up, rather than fork them, until that happens, Ubuntu will become more and more of an impenetrable island that will eventually grind to a halt where a finite set of resources is used to do everything.

If You Think Your Users Are Stupid, You're Probably Doing it Wrong

I was recently in an office where a programmer and a designer were complaining about their users being too stupid.

The users weren't using the program they were tasked with developing "properly". If you develop an application that the users cannot figure out, you have failed to do your job, and are asking the users to do it for you.

As an application developer, it is your job to be invisible, and to make the software invisible. The user, novice or expert, should be able to get in to your system, figure out how it works immediately, do what they need to, and leave easily.

One of the most valuable things I've found out as a developer is that users rarely think in the same models as you. As programmers, we're taught to divide the world in to neat little problems and objects that flow through pipes and get transformed.

The user doesn't understand that the box that helps them compose email is a text pane wrapped in a scroll pane that has buttons that apply styles to the data inside that passes it to a renderer, and has a side process that is activated whenever a whitespace token is pressed that does a spelling check on the document. They probably don't imagine that the place to enter names that looks them up in their contacts database is a separate piece of software altogether. To the user, the object they see is a form.

This leads me to a few, simple rules for usability design that I think everyone should benefit from:

Anticipate the User's Needs

If you are writing an email app, auto-suggest contacts to use as recipients, offer printer-friendly versions of everything (preferably that auto-format for printers).

Consistency

Just because you're designing a website for My Little Pony watchers doesn't mean purple links is a good idea (forgive me if I've gotten this entirely wrong), blue links are standard, and they are what users look for.

• Buttons are not links, and links are not buttons (links navigate and buttons change the current page)
• If two things look the same, they should do the same thing.
• If something is dangerous

Navigation

Or, "where the hell am I?" generally you don't need folders in folders in folders; if you do, you're doing something horribly wrong. Even desktop applications benefit from clear location awareness.

If you have a confusing area in your software, use contextual popups (when the user is editing a form) or help can help ferry the user through potentially confusing areas.

Speed

This one is probably the most useful. Find out what 90% of your users are doing (it is probably either not what you intended, or they are doing what you wanted in a long and laborious way), so try doing the following:

• Remove all items in a form that aren't required.
• Allow sign-in from existing accounts, like Google/Twitter/Facebook if applicable.
• Show the user the status of their most used items right away, using colors if possible.
• Use consistent icons.
• Use consistent navigation.
• When the users are working around somthing you've made, fix your software to facilitate their way (ideally you'd make whatever way you had in mind the easiest so it would be quickly discovered and used)

End of Blogging For A Month

Over the past month I made it a goal to write a post every day. It was tough trying to find something to write about every day. Even if it seemed easy to begin with.

The hardest part was trying to write well, rather than just send links (and yes, I cheated on Fridays with a link list, but so does everyone else, I regret nothing!)

Overall, the experience has been good, I've nearly now got a hundred posts and roughly fifty page views a day, and my most popular articles have exploded while others don't, I'm not entirely sure why yet, but I'm determined to find out.

In future months, I plan to cut back on the schedule, probably to three times a week, now that I know it isn't as bad as I imagined.

What's up next? Hmm, I'm not sure yet; I may write a book about something, exercise or whatever, I'll think about it for a day.

Here is the video that originally made me decide to try something new for thirty days every thirty days:




My last endeavor was to take a picture every day for a month, and it was great!

Three New Interesting Operating Systems

AROS Icaros

Not an OS per se, but a desktop environment that allows you to run old Amiga software; it allows full emulation for Amiga software without the need for ROMs, in the latest version.

Icaros Desktop

Whonix

This is an interesting concept of an OS, the user runs inside of a virtual machine where all the internet traffic is automatically routed through TOR. Somehow I doubt this will come to as great of use as it should, simply because running a VM is very costly, especially through VirtualBox (which this is), maybe if it were running on Xen. The project just got released so there is still time for improvement!

FreeBSD 9.1

The latest FreeBSD has been released (FreeBSD is the thing Apple based OSX on). New in this release are

• Xen ethernet driver
• IPv6 improvements
• C++11 stack
• New Intel Drivers

Visualizing REGEXs

The excellent REGEXPER generates beautiful representations of Javascript regexes you send its way.

Example (IP Address)

\d\d?\d?\.\d\d?\d?\.\d\d?\d?\.\d\d?\d?

ODT Text Mining in Python

The ODT format is slowly gaining popularity, as governments worldwide begin to either use FOSS software, or transition to formats that are open enough to still be usable in at least a hundred years.

The Format

Within all .odt files (which are just zipfiles) there is an XML file that holds all of the document's text named content.xml. All of the characters between the XML tags are the document's text, so by cutting out all of the tags, all that remains is the text (what the script below does).

The Code (Python)

#!/usr/bin/env python3
'''

Copyright 2012 Joseph Lewis <joehms22@gmail.com> | <joseph@josephlewis.net>

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:

* Redistributions of source code must retain the above copyright
  notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
  copyright notice, this list of conditions and the following disclaimer
  in the documentation and/or other materials provided with the
  distribution.
* Neither the name of the  nor the names of its
  contributors may be used to endorse or promote products derived from
  this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

This software recieves an odt file from the command line, and returns the text 
from its path.

'''

import zipfile
import xml.etree.ElementTree
import sys


def extract_odt_text(fp):
    myFile = zipfile.ZipFile(fp)

    share = xml.etree.ElementTree.fromstring(myFile.read('content.xml'))

    text_nodes = []

    for elt in share.iter():
        if elt.text != None:
            text_nodes.append(elt.text.strip())

    return " ".join(text_nodes)

if __name__ == "__main__":
    if len(sys.argv) == 1:
        print "usage: odt.py FILENAME [FILENAME...]"
    else:
        for arg in sys.argv:
            try:
                print(extract_odt_text(arg))
            except zipfile.BadZipfile:
                pass