Devin Venable

Badges

2012-02-10T12:01:00.000-08:00

We've been experimenting with badges for the SayAh web site. For example, the following badge---which is just a hyperlink with an embedded image---directs visitors to the CSOS web site to Bryan Hawkin's SayAh Review Site.

I'm not sure if the badge fragment will render properly within a blogger post, thus this post, which tests behavior.

git differences between working copy and stashed copy

2011-09-21T08:50:00.000-07:00

Shows the differences between copy and third item in the stash stack...

git diff stash@{3}

Vim search and replace from current location to end of file

2011-09-12T08:53:00.000-07:00

Instead of searching and replacing the entire file with...

:s/old/new/g

Try inserting ".," just after the colon. This means "from current position to end of file".

:.,s/old/new/g

The comma actually specifies a range to make the substitution, from left and right of comma. For example...

:5,10s/old/new/g

...makes the substitutions on lines 5 through 10.

Notice to .NET developers

2011-09-09T09:02:00.000-07:00

Word is there are about 30 more JAVA positions that need to be filled in the Tulsa area than there are JAVA capable developers. This is driving up the rate that JAVA developers can demand.

Meanwhile, Tulsa is flooded with .NET developers, which drives the rate .NET developers demand, downward.

So a word to the wise: don't get married to one vendor's technology. Any software engineer worth his salt can switch from .NET to JAVA or visa-versa without much pain.

Vim tags

2011-08-31T07:14:00.000-07:00

Continuing on with my exploration of Vim features I've yet to use, let's explore tags.

Tags are like bookmarks. They let you jump around to different places in your file that you might want to reference, like a class definition for example.

Create a tag like so: ":tag tagname". This tags the current line number with the name you provide. Alternatively, you can use the shortcut CTRL + ]. The tag name will be taken from either the character under your cursor, or if you've highlighted a word or phrase, it will be used.

":tags" shows you your list of tags.

:tags

# TO tag FROM line in file/text

> 1 1 LongTextAnswer 8 q = LongTextAnswer()

Each time I show my :tags I get this error:

E433: No tags file.

This may also explain why my tags queue only contains the most recent tag I create. Further reading reveals that the tags file is created by the program ctags. I'll try installing exuberant-ctags and see what affect that has on my operation.

sudo apt-get install exuberant-ctags

Restarting vim and trying the :tags command again showed that my tags file could still not be found. I found a nice little page that describes the rest of the process that one must go through to get your tags file set up. Follow the link to learn more.

http://amix.dk/blog/post/19329

On a related note, you can read ctags in python:

http://code.google.com/p/python-ctags/

Vim Execute Mode

2011-08-31T06:51:00.000-07:00

I use vim and gvim every day. I'm so productive using the small subset of available commands that I know that I tend not to explore the question, "what else can I do with Vim?"

Take Execute Mode for example. In 10 years of Vim use, I've never tried it. Let's fix that now.

You enter Execute Mode by pressing capital Q (shift + q) when in normal mode. You'll see this at the bottom of your window.

Entering Ex mode. Type "visual" to go to Normal mode.

This seems familiar enough, but this is different than when you simply type ":" to enter a command at the bottom of the screen in normal mode. In normal mode, typing ":10 " + enter takes you to the tenth line in the document. In Execute Mode, ":10" + enter displays the tenth line just below your cursor. Hitting return again shows the eleventh line, and so on. You are not editing the document in this mode, simply accessing it. Interesting, but not yet terribly useful. Let's see what else you can do.

:w filename

This writes the file, just as it would if you had typed :w filename in normal mode, except again, instead of returning you to editing the document, you remain in Execute mode, meaning that the ":" appears again, awaiting your next command.

I'm afraid I can't imagine too many good reasons to use Execute Mode. The best use-case I can find is that you could use the :n, where n is a line number, feature view a section of the document you are not working on without leaving the section you are working on. If you have any tips on how Execute Mode makes your life easier, please leave a comment.

Safari URL opens new window

2011-08-08T08:41:00.000-07:00

Here's a strange quirk I encountered while working with Safari mobile: HTML anchors (<a href="http://www.blogger.com/something.html">click</a>) open a new Safari browser instance---every time.

Now that can't be right. As it turns out, this is indeed the standard behavior when the target attribute is not explicitly set in the anchor. Apple does provide a work around setting:

http://www.naquah.net/blog/dennis/2008/03/21/how-to-enable-single-window-mode-in-safari

There are many instances when web designers don't need targets, and they are omitted. For most browsers, when the target is not explicitly set, the browser default behavior is _self. See the table from w3cschool:

http://www.w3schools.com/tags/att_a_target.asp

Attribute Values

Value	Description
_blank	Open the linked document in a new window or tab
_self	Open the linked document in the same frame as it was clicked (this is default)
_parent	Open the linked document in the parent frameset
_top	Open the linked document in the full body of the window
framename	Open the linked document in a named frame

The kicker is that Safari uses _blank as the default instead of self. The solution for web designers? Explicitly set the target attribute to _self for every anchor. A bit of a hassle, but the best way to ensure that a new window does not open up with each hyperlink click.

Huge gvim annoyance

2011-07-13T07:30:00.000-07:00

Since upgrading to Ubuntu 11.01, when I open a document using gvim, I get a very unresponsive file browser window. In the past, I would double click a folder and it would quickly open. Using gvim, when I double-click, I get a wait cursor. I often have to click four or five times before the folder opens. Very irritating. Anyone else have this problem?

Same thing happens whether using vim-gnome or vim-gtk package.

Other programs do not share this quirk. For example, the text editor program behaves as expected when opening the files.

Can't ignore IE

2011-05-24T07:50:00.000-07:00

I've been using Ubuntu as my operating system for the past five years. So long that I almost forgot that you must always check your web site for IE compliance. Well, I didn't forget so much as I wasn't concerned.

So since I've been shopping for contracts---I decided to return to consulting if I can line up the right opportunities--I thought it would be wise to review the user experience of my site, devinvenable.com, using IE. I fired up a virtual machine running Windows XP, opened up IE 8, and was hit with Javascript errors. Doh!

It's simple enough to conditionally remove code from your page that won't run on IE, or conditionally include code that does. Even if you use cross-platform javascript libraries like jQuery, as I do, you are still bound to run into issues from time to time.


<!--[if IE]>
   You are using IE (IE5+ and above), so include include script that is specific for IE here.
<![endif]-->

<![if !IE]>
   You are NOT using IE, so feel free to use HTML 5 features.
<![endif]>

So don't get lazy and assume that IE is history. While it's true that many savvy technical users have ditched IE for Chrome or Firefox, there are always users who will use IE because it is installed by default on their Windows OS.

Writing this entry made me curious about the current browser usage statistics. Amazingly, Firefox is way out in front with 42.9% this month, followed by Chrome at 25.6%. Microsoft's IE is still hanging in there with a respectable 24.3%. Still a contender but far from being the market leader in 2011.

http://www.w3schools.com/browsers/browsers_stats.asp

Job search fun!

2011-05-20T07:36:00.000-07:00

I've decided to make a transition. New opportunity, here I come!

Looking for a new gig is always a lot of hard work. One must find the right fit: good people, exciting technology, and room for growth. I've talked to one prospective employer already and was asked, "What would be your ideal job if you could have anything you dream of, the perfect environment?"

It's a tough question. As a technologist, I'm probably at the top of my game. With 14 years of software development, I've dappled in just about everything. I don't encounter new technologies or languages often for which I've not already built products. Because of this, I tend to stay in the trenches---doing hands on technical work.

I'm 42. At my age, some get out of the trenches and into management. I'm afraid I would be bored silly if my primary work tool was Microsoft Project. That said, I am a people person, and often find myself wishing for more socialization when working with other developers, as so often they are introverts.

I like building new products versus extending the work others did before me. Who wouldn't? So often you inherit a bug infested mess. Building systems from scratch is fun for me.

I love brainstorming and sharing ideas. I'm a dreamer and I always have been, and this tends to work for me in the creativity department.

I love open source projects and all things Linux. This doesn't always work to my benefit in the Tulsa market where many mid-sized corporations go with Microsoft solutions. Perhaps I ought to go after these markets, selling CTO's on the joy of open source and living in a license free world. For a fraction of what companies pay for licensing fees I could switch them over to free solutions. That's a gift that would keep on giving. Many mid-sized corporations would be well served ditching Oracle, for example, as the scalability and reliability of open source alternatives is easily demonstrable. And what about the hundreds of thousands they spend on DBA's? Database administration doesn't have to be this hard, folks.

I'm no closer now than when I started writing this blog entry to knowing what my ideal job would be. But at least the creative juices are flowing. :)

Adding information to Django's default error page

2011-04-26T13:12:00.000-07:00

Django's default error page is a handy way find out what went wrong when your view code runs into an exception that is not explicitly handled. But sometimes you don't get the info you really want to see. For instance, today I kept running into sql errors, something I don't see too much of using Django because they are normally abstracted away by the model framework.

In my case I needed to resort to using raw sql. The query was complex and the variables many, and thus I encountered a number of different database (cx_oracle) errors while working on the view and I didn't have a quick and dirty way to see the sql that was generated.

Getting the most recently generated sql is possible by importing django.db.connections. You can then tack it into your exception and see it on the default error page using the following technique.



try:

    """
    This is where you build your view.  Do whatever you need to do.  In my case
    I was inserting a row into the database here.
    """
except Exception as inst:
    
    """
    inst.args is a tuple of arguments that were passed when the exception
    was raised.  Django will turn this variable into the main error message
    you see atop the default error page.  It's a tuple, so you can't append
    to it, but you can replace it with another tuple and add any information
    that would serve you well.
    """

    inst.args = ("%s sql:%s" % (inst.args[0], " ".join(connections[db].queries.pop()['sql'].split())),)

    """
    Sure that's ugly, but it served my debugging purpose.  Use pop to get the most 
    recent query from the queries list.  I wanted to get rid of the while space and
    newline characters, which is the reason for the split and join.  Now you just 
    raise the exception.
    """
    raise

If you are going to do this kind of thing frequently you should probably explore the process-exception middleware API.

http://docs.djangoproject.com/en/dev/topics/http/middleware/?from=olddocs#process-exception

Best Django debugging technique ever

2011-04-25T11:34:00.000-07:00

Start up your test app using the Development Server. Then inject the following into your code where you want to examine everything, perhaps in an exception handler.


import pdb
pdb.set_trace

Now when you hit the exception, pdb will take over the console window you started the test app. Type help if you are unfamiliar with pdb. All of your variables are available for review!

Sweet! I can't believe I didn't know about this technique before.

Dansguardian and Youtube on Ubuntu 10.10

2011-03-28T11:49:00.000-07:00

If you want content filtering for your kids on Ubuntu 10.10, Dansguardian does the trick, but be prepared to do some tweaking.

Installation is not difficult. Follow this recipe:

https://help.ubuntu.com/community/Servers/DansGuardian

My kids love youtube but the default configuration renders it useless. Locate the configuration files in /etc/dansguardian. You'll primarily be interested in the content under the "lists" directory.

I added the following file extensions to exceptionextensionlist:


.js
.flv
.mp3
.ico

From bannedregexpurllist, I removed the keyword "naked", which seemed a bit over the top to me.

For video playback, add this line to exceptionurllist:


youtube.com/videoplayback

In exceptionsitelist, add the following. What the heck is ytimg.com? Well apparently it serves the main static css off youtube. I don't know if this changes and if it does how often, but I do know that the addition of this line alone had the most positive change on the overall youtube experience under Dansguardian.


ytimg.com

Similar steps may need to be repeated for other sites that fail to render properly. Watch the log at /var/log/dansguardian/access.log to see what's getting blocked, as that will provide the clues as to what you need to make exception for.

Color highlight columns in Vim

2011-03-17T12:41:00.000-07:00

Vim 7.3 has a new, useful feature that allows you to highlight (set the background color) of one or more columns of the document you are working on. This comes in handy if you are working with a file that uses a fixed width format. For example, if you want to quickly see the 44th column for every row as you scan through a file, this might assist in the task.

Vim 7.3 is not yet in the Ubuntu repositories officially, but a PPM is available here:

http://askubuntu.com/questions/7283/where-can-i-find-vim-7-3

Once installed, use vim help to get details about colorcolumn.


:help colorcolumn

This will show you:


'colorcolumn' 'cc'      string  (default "")
                        local to window
                        {not in Vi}
                        {not available when compiled without the |+syntax|
                        feature}
        'colorcolumn' is a comma separated list of screen columns that are
        highlighted with ColorColumn |hl-ColorColumn|.  Useful to align
        text.  Will make screen redrawing slower.
        The screen column can be an absolute number, or a number preceded with
        '+' or '-', which is added to or subtracted from 'textwidth'. 

                :set cc=+1  " highlight column after 'textwidth'
                :set cc=+1,+2,+3  " highlight three columns after 'textwidth'
                :hi ColorColumn ctermbg=lightgrey guibg=lightgrey

        When 'textwidth' is zero then the items with '-' and '+' are not used.
        A maximum of 256 columns are highlighted.

In practice you only need to :set cc=n, as there are default colors.

History in Python shell

2011-03-03T10:56:00.000-08:00

I never could get used to ipython, but still wish for command history in my shell. You too? I'm reposting this solution here that not only saves your command history between sessions but also gives you auto-completion.

Here's what you do. Store the following file in ~/.pystartup

Don't forget to export PYTHONSTARTUP as noted in the comments.


# Add auto-completion and a stored history file of commands to your Python
# interactive interpreter. Requires Python 2.0+, readline. Autocomplete is
# bound to the Esc key by default (you can change it - see readline docs).
#
# Store the file in ~/.pystartup, and set an environment variable to point
# to it:  "export PYTHONSTARTUP=/home/user/.pystartup" in bash.
#
# Note that PYTHONSTARTUP does *not* expand "~", so you have to put in the
# full path to your home directory.

import atexit
import os
import readline
import rlcompleter

readline.parse_and_bind('tab: complete')
historyPath = os.path.expanduser("~/.pyhistory")

def save_history(historyPath=historyPath):
    import readline
    readline.write_history_file(historyPath)

if os.path.exists(historyPath):
    readline.read_history_file(historyPath)

atexit.register(save_history)
del os, atexit, readline, rlcompleter, save_history, historyPath

Please note that this will only work on *nix systems. As readline is only available in Unix platform.

Thank God for Tripper McCarthy

2010-11-24T07:28:00.000-08:00

A long time ago, in our galaxy, Tripper McCarthy posted this:


Tripper McCarthy  Tue Jun 24 2003 19:19:31 GMT-0500 (CDT) 

 There have been several posts asking how to change the body of a message
(not the header) from within a handler. Here is a way to pull it off. It's a
little strange, but it works. 

    public void invoke(MessageContext msgContext) throws AxisFault {
        try {
            javax.xml.soap.SOAPMessage soapMessage =
msgContext.getRequestMessage();
            String oldRequest =
soapMessage.getSOAPPart().getEnvelope().toString();
            
            int index = oldRequest.indexOf("            index = oldRequest.indexOf(">",index);
            String newRequest = oldRequest.substring(0,index+1) +
                "5" +
                oldRequest.substring(index+1,oldRequest.length());
            
            ByteArrayInputStream istream = new
ByteArrayInputStream(newRequest.getBytes());            
            Message msg = new Message(istream, false);
            msgContext.setRequestMessage(msg);

        }
        catch(Exception e) {
            e.printStackTrace();
            throw new AxisFault(e.getMessage());
        }
    }

Thanks Tripper. I've been beating my head for hours trying to modify the message using the deeply-nested object model in Handler. Your solution cut to the chase.

RedHat Enterprise 5 and Python

2010-10-22T11:01:00.000-07:00

Update:

I found it difficult to make Python 2.6 the default distribution for RHEL. It jacks up YUM when you do it. So I'm abandoning that approach altogether. The following notes were taken before I decided to take a different approach.

Python 2.4 is the python that ships with RedHat Enterprise Linux 5. This will never do.

I didn't remove 2.4 because some suggested against it, saying that RedHat needs to keep this version around.

Suppose you want to go to version 2.6. Here's what to do: either install from repo (yum install python26), which requires that you enable EPEL, or build from Python from source. It will install along side 2.4. When you next run Python you will find that the OS is still defaulting to 2.4. To fix this run update-alternatives.


update-alternatives --config python

Run Python again and your system wide default should now be the latest version you installed.

Alert: After building my own 2.6 and following the process above, I discovered that I had inadvertently screwed up YUM. Beware this process until I post final comments.

Shrink PDF file on linux (command line)

2010-10-22T10:59:00.001-07:00

gs -sDEVICE=pdfwrite -dCompatibilityLevel=1.4 -dPDFSETTINGS=/screen -dNOPAUSE -dQUIET -sPAPERSIZE=a4 -dBATCH -sOutputFile=output.pdf input.pdf

Convert 32 to 16 bit WAV (Linux) recipe

2010-10-14T17:15:00.000-07:00

Check file info:



devin@studio:~/Music/samples$ sndinfo funky.wav 
virtual_keyboard real time MIDI plugin for Csound
PortMIDI real time MIDI plugin for Csound
PortAudio real-time audio module for Csound
util sndinfo:
funky.wav:
 srate 44100, stereo, 32 bit WAV, 9.835 seconds
 (433737 sample frames)

Convert to 16 bit:

sox funky.wav -b 16 funky16.wav

cx_Oracle and out cursor

2010-09-09T09:23:00.000-07:00

If you need to call an Oracle stored procedure using cx_Oracle, and one of the arguments is an "out" cursor, this is how it is done.



connection = cx_Oracle.connect(connstr)
cursor = connection.cursor()
out = cursor.var(cx_Oracle.CURSOR)

items = cursor.callproc('EXAMPLE_PKG.get_some_data',
                ['02-SEP-2010',
                '03-SEP-2010',
                332123.0,
                896798.0,
                68567.0,
                'xyz',
                out
                ]

print items[6].fetchall()

A list of objects will be returned from callproc containing each of the arguments passed in. If any of the arguments passed are "out" arguments, they will have been modified to hold the results. In my case the seventh argument (items[6] --- zero indexed) returned a cursor. This cursor is then used to retrieve the result set.

I had trouble finding a good example of this usage pattern for cx_Oracle on the net, so here's one for the next lucky guy or girl who is digging for this answer.

OpenGL Intellivision Man

2010-07-30T11:48:00.000-07:00

Not long ago I set out to turn an animated gif I found of the Intellivision Running Man into an OpenGL generated video clip. My goal was to shrink the images into tiny bitmaps, a.k.a. 8-bit, and then use the resulting bitmap data as positioning information within a 3D space. Another way of saying it: I wanted to render each pixel from the original images as a 3D block. Jump to the video to get a better idea about what I mean:

http://www.youtube.com/watch?v=E1rXt0l2pxY

Along the way I took a few turns. I never did reduce the image data to its smallest possible representation. I did reduce the image size for each image to 24x24 pixels and I converted the image data to grayscale. Using the handy PIL library, I was able to pull in the bitmap file data as a tuple of pixel values. I used each "pixel" to offset the drawing position in pyglet's on_draw handler.

The code I used to render the animation is a hack of an example found here:

http://code.google.com/p/pyglet-hene/source/browse/trunk/

This is a fairly ugly hack, meaning I didn't work to make the code beautiful. It's just the original code, chopped, altered and enhanced as needed to accomplish what I wanted to see rendered on the screen.

Though not provided here, to run this code an image directory needs to be provided named "data" containing a series of bitmaps. My directory contains this:


ls -ltr
total 32
-rw-r--r-- 1 dvenable dvenable 1654 2010-07-20 14:06 9.bmp
-rw-r--r-- 1 dvenable dvenable 1654 2010-07-20 14:06 8.bmp
-rw-r--r-- 1 dvenable dvenable 1654 2010-07-20 14:06 6.bmp
-rw-r--r-- 1 dvenable dvenable 1654 2010-07-20 14:06 5.bmp
-rw-r--r-- 1 dvenable dvenable 1654 2010-07-20 14:06 4.bmp
-rw-r--r-- 1 dvenable dvenable 1654 2010-07-20 14:06 3.bmp
-rw-r--r-- 1 dvenable dvenable 1654 2010-07-20 14:06 2.bmp
-rw-r--r-- 1 dvenable dvenable 1654 2010-07-20 14:06 1.bmp

If you are so inclined to try your own little experiment, find a favorite animated gif on the web, extract each frame, reduce size, convert to grayscale and drop your own images in a directory similar to the one I created here. Your animation result should be similar to what I've produced here.

And here's the complete source:


from pyglet.gl import *
import pyglet
from pyglet.window import *
from pyglet import image
import os
from PIL import Image
import glob
from math import sin, cos

window = pyglet.window.Window(width=640, height=480, resizable=True)

y=0.0
x=-10.0     
z=10.0    
xspeed = 0.5
yspeed = 0.0
lx=ly=0
lz=-2
angle=ratio=0.0

boxcol = [ [1.0, 0.0, 0.0], # bright: red
           [1.0, 0.5, 0.0],        # orange
           [1.0, 1.0, 0.0],        # yellow
           [0.0, 1.0, 0.0],        # green
           [0.0, 1.0, 1.0],        # blue
                        ]

# Dark: red, orange, yellow, green ,blue
topcol =[ [0.6, 0.0, 0.0],
          [0.6, 0.25, 0.0],
          [0.6, 0.6, 0.0],
          [0.0, 0.6 ,0.0],
          [0.0, 0.6, 0.6]]
                        


box = None # display list storage
top = None #display list storage

yloop = None # loop for y axis
xloop = None # loop for x axies

bmpdata = None 
nextimg = 0
files = None

def load_image_data():
    global bmpdata, bmpdatalen, files 

    files = glob.glob('data/*.bmp')
    files.sort()
    bmpdata = map(lambda x: Image.open(x).getdata(), files.__iter__())
    bmpdatalen = len(bmpdata)

 
def build_lists():
  global box, top
  box = glGenLists(2)

  glNewList(box, GL_COMPILE)    # new compiled box display list

  # draw the box without the top (it will be store in display list
  # and will not appear on the screen)
  glBegin(GL_QUADS)
  
  # front face
  glTexCoord2f(0.0, 0.0);       glVertex3f(-1.0, -1.0, 1.0)
  glTexCoord2f(1.0, 0.0);       glVertex3f(1.0, -1.0, 1.0)
  glTexCoord2f(1.0, 1.0);       glVertex3f(1.0, 1.0, 1.0)
  glTexCoord2f(0.0, 1.0);       glVertex3f(-1.0, 1.0, 1.0)
  # back face
  glTexCoord2f(1.0, 0.0);       glVertex3f(-1.0, -1.0, -1.0)
  glTexCoord2f(1.0, 1.0);       glVertex3f(-1.0, 1.0, -1.0)
  glTexCoord2f(0.0, 1.0);       glVertex3f(1.0, 1.0, -1.0)
  glTexCoord2f(0.0, 0.0);       glVertex3f(1.0, -1.0, -1.0)
  # right face
  glTexCoord2f(1.0, 0.0);       glVertex3f(1.0, -1.0, -1.0)
  glTexCoord2f(1.0, 1.0);       glVertex3f(1.0, 1.0, -1.0)
  glTexCoord2f(0.0, 1.0);       glVertex3f(1.0, 1.0, 1.0)
  glTexCoord2f(0.0, 0.0);       glVertex3f(1.0, -1.0, 1.0)
  # left face
  glTexCoord2f(0.0, 0.0);       glVertex3f(-1.0, -1.0, -1.0)
  glTexCoord2f(1.0, 0.0);       glVertex3f(-1.0, -1.0, 1.0)
  glTexCoord2f(1.0, 1.0);       glVertex3f(-1.0, 1.0, 1.0)
  glTexCoord2f(0.0, 1.0);       glVertex3f(-1.0, 1.0, -1.0)
  glEnd()

  glEndList()   # Done building the list

  top=box+1

  glNewList(top, GL_COMPILE)    # new compiled top display list
  # Top face
  glBegin(GL_QUADS)
  glTexCoord2f(0.0, 1.0); glVertex3f(-1.0, 1.0, -1.0)
  glTexCoord2f(0.0, 0.0); glVertex3f(-1.0, 1.0, 1.0)
  glTexCoord2f(1.0, 0.0); glVertex3f(1.0, 1.0, 1.0)
  glTexCoord2f(1.0, 1.0); glVertex3f(1.0, 1.0, -1.0)
  
  glTexCoord2f(1.0, 1.0);       glVertex3f(-1.0, -1.0, -1.0)
  glTexCoord2f(0.0, 1.0);       glVertex3f(1.0, -1.0, -1.0)
  glTexCoord2f(0.0, 0.0);       glVertex3f(1.0, -1.0, 1.0)
  glTexCoord2f(1.0, 0.0);       glVertex3f(-1.0, -1.0, 1.0)
  glEnd()
  glEndList()


     
def load_gl_textures():
        # load bitmaps and convert to textures
        global texture, texture_file, texture_surf
        #texture_file = os.path.join('data', 'cube.bmp')
        texture_file = files[nextimg]
        texture_surf = image.load(texture_file)
        texture = texture_surf.get_texture()
        glBindTexture(GL_TEXTURE_2D, texture.id)

        glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR)
        glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR)


        
def init():
    """
    Pyglet oftentimes calls this setup()    
    """
    glEnable(GL_TEXTURE_2D)

    load_image_data()
    load_gl_textures()
    build_lists()

    glShadeModel(GL_SMOOTH) # Enables smooth shading
    glClearColor(0.0, 0.0, 0.0, 0.0) #Black background
    
    glClearDepth(1.0)               # Depth buffer setup
    glEnable(GL_DEPTH_TEST)         # Enables depth testing
    glDepthFunc(GL_LEQUAL)          # The type of depth test to do

    glEnable(GL_LIGHT0) # quick and dirty lighting 
    
    #glEnable(GL_LIGHTING)  # enable lighting
    glEnable(GL_COLOR_MATERIAL)     # enable coloring

    glHint(GL_PERSPECTIVE_CORRECTION_HINT, GL_NICEST)       # Really nice perspective calculations
       
    
     
@window.event
def on_draw():
    global nextimg, bmpdata, x, y, z, lx, ly, lz

    # Here we do all the drawing
    glClear(GL_COLOR_BUFFER_BIT |GL_DEPTH_BUFFER_BIT)

    # Select the texture
    load_gl_textures()
    #glBindTexture(GL_TEXTURE_2D, texture.id)

    xloop = 1
    yloop = 1
    
    mandata = bmpdata[nextimg]
    for idx in range (0, len(mandata)):
        if (idx+1) % 24 == 0:
            yloop += 1
            xloop = 1
        else:
            xloop += 1
        if mandata[idx] < 100:

            glLoadIdentity()        # reset our view
            gluLookAt(x,y,z, x+lx , y+ly, z+lz, 0.0, 1.0, 0.0)

            glTranslatef( xloop*1.8 - 30 ,
                                      28 - yloop*2.4 ,
                                      -60.0)
            glColor3f(*boxcol[xloop % 4]) # select a box color
            glCallList(box) # draw the box

            glColor3f(*topcol[1])
            glCallList(top)  # draw the top


    return pyglet.event.EVENT_HANDLED
      
def moveMeFlat(direction):
    global x, z, y, lx, lz, ly 
    x = x - direction*(lx)*0.75;
    y = y + direction*(ly)*0.5;
    z = z + direction*(lz)*0.5;
 
def orientMe(ang):
    global lx, lz
    lx = sin(ang)
    lz = -cos(ang)
  

def update(dt):
    global z, angle
    angle +=0.005
    orientMe(angle)
    moveMeFlat(0.5)

def update2(dt):
    global nextimg
    if nextimg < bmpdatalen-1:
        nextimg += 1
    else:
        nextimg = 0 


pyglet.clock.schedule_interval(update2, .1)
pyglet.clock.schedule(update)
        
@window.event
def on_resize(width, height):
        if height==0:
            height = 1
        glViewport(0, 0, width, height)
        glMatrixMode(GL_PROJECTION)
        glLoadIdentity()

        # Calculate the aspect ratio of the window
        gluPerspective(45.0, 1.0*width/height, 0.1, 100.0)

        glMatrixMode(GL_MODELVIEW)
        glLoadIdentity()
        return pyglet.event.EVENT_HANDLED
        
init()  

pyglet.app.run()

Cross-compile ActiveMQ-cpp on Centos 5

2010-07-20T07:16:00.000-07:00

To date an RPM for activemq-cpp is not available for Centos 5. I found RPMs in Fedora 13 and 14 repositories, but due to dependencies on fresher libs, they won't install.

Building for 32 bit architecture on 64 bit Centos 5 was a bit tricky. Here's the recipe I came up with that worked for me:

Install dependencies:


yum install expat-devel zlib-devel uuid-c++-devel openssl-devel

Download sources from here or other mirrors:


wget http://www.carfab.com/apachesoftware/activemq/activemq-cpp/source/activemq-cpp-library-3.2.1-src.tar.gz
wget http://mirror.candidhosting.com/pub/apache/apr/apr-1.4.2.tar.gz

Extract archives and build APR first. Note PKG_CONFIG_PATH, as this is one of the keys to ensuring that lib64 libraries are not found during link step.


[root@myserver] cd apr-1.4.2
[root@myserver] ./configure --prefix=/usr --libdir=/usr/lib CXXFLAGS="-m32" LDFLAGS="-m32" CFLAGS="-m32" --build=i686-redhat-linux-gnu PKG_CONFIG_PATH=/usr/lib/pkgconfig 
[root@myserver] ./make
[root@myserver] ./make install

Next build and install ActiveMQ-cpp:


[root@myserver] cd ../activemq-cpp-library-3.2.1
[root@myserver] ./configure --prefix=/usr --libdir=/usr/lib CXXFLAGS="-m32" LDFLAGS="-m32" CFLAGS="-m32" --build=i686-redhat-linux-gnu PKG_CONFIG_PATH=/usr/lib/pkgconfig --with-apr=../apr-1.4.2/apr-1-config
[root@myserver] ./make
[root@myserver] ./make install

qemu-img

2010-07-13T06:08:00.000-07:00

qemu-img is a handy tool for converting one virtual machine image format into another.


qemu-img convert windowsxp.img -O vdi windowsxp.vdi

I had need for its use after running into errors creating a virtual machine with virt-manager. KVM managed by virt-manager has been my recent virtualization solution of choice, that is, until last Friday when I hit a snag creating a new virtual machine for WindowsXP. The installation process got stuck and I wasn't able to recover. Following advice from somewhere it the net-realm, I opted to create the image using this easy command line two-liner.


 dd if=/dev/zero of=vm8.img bs=2048k count=12000
 kvm -m 512 -cdrom winxp.iso -boot d v8.img

The installation was successful and running the image from the command line using kvm was a snap, but what I really wanted was to place this image under the control of virt-manager.

I couldn't find a nice option to import an existing image, so if it exists it is not very intuitive. Apparently to place it under virt-manager's control I would need to create an xml file under /etc/libvirt/qemu/ and in addition I would probably need to convert the image from raw format to qcow2. But after messing around with this for too long, a colleague recommended I use virtualbox.

I used virtualbox in the past, so I figured a different approach might be worthwhile and should get me beyond the headache in the short term. But what about the work I did creating the original image? Installing WindowsXP is a hassle, not to mention the effort I'd put into installing subsequent software on the image.

And that's where qemu-img came in. virtualbox only reads vmi formatted images, thus I converted my raw image to vmi.

Updated: FAIL

Importing the new vmi was easy enough, but once I created the new virtual machine and started it...I got this error from virtualbox-osi:

"Failed to start the virtual machine windowsxp. VirtualBox can't operate in VMX root mode. Please disable the KVM kernel extension, recompile your kernel and reboot."

Okay...but I will silently resent you.

Updated: Original image hung after removing KVM. At this point I'm going to throw up my hands and create a new image and fresh install. I'm at the point of diminishing returns...

Going really 8-bit

2010-07-07T14:18:00.001-07:00

Back when I was a kid, my nerd friends and I would create graphics for our Commodore computers by filling filling in squares on graph paper and calculating the bitmaps. A single character on the Commodore 64 and Vic 20 was an 8x8 grid.

I recently found an 8-bit image fondly remembered from my youth, the Intellivision Running Man, in animated GIF format. I converted the GIF to a short video, which I embedded in a video project.

Some of my video conversion notes were covered in another post and some are on my wiki.

I want to extract the Intellivision Running Man bitmaps for use in an opengl project that I'm thinking about doing. Though it would probably be simpler for me to just sit down with pen and paper to graph and calculate the bitmaps, I'm hoping to find tools that will allow me to extract the data directly from raw video.

I first needed to reduce the screen resolution as much as possible by shrinking the video and dropping frames. Here I shrank the video output to 24x24 and reduced the frame rate to 4 frames per second.

ffmpeg -i intel.avi -r 4 -s 24x24 intel8bit.avi

Next I extracted individual bitmap images:

ffmpeg -i intel8bit.avi -f image2 foo_%5d.bmp

I initially took one image into Gimp and used Desaturate to convert it to grayscale, and then I used Levels to reduce image to pure Black or White. But I didn't want to repeat this again and again. So I wrote a small Python script which achieved the same goal and along the way got to play with the PIL library, which I've not used before.


from PIL import Image
import sys, os

def bw(pt):
    if pt>126:
        return 255
    else:
        return 0

for infile in sys.argv[1:]:
    f, e = os.path.splitext(infile)
    outfile = f + "_mod" + e

    im = Image.open(infile)

    im = im.convert('L')
    out = im.point(bw)

    out.save(outfile)
    print outfile

This little script loads a bitmap then uses convert() with the 'L' option to make it grayscale. (I couldn't find a comprehensive list of options for convert---the documentation could use some work.)

Finally, I used the point method that passes each pixel in the bitmap data to a custom function. Your function can do anything, but mine just looks at the value, makes higher level grays absolute black and lower level grays absolute white. point() returns a copy of the bitmap file (headers updated and intact) with the transformed result, which is then saved.

At this point I have true black and white bitmaps.

I wanted to quickly visualize my bitmap, to get a feel for how much more I want to reduce the size of my bitmaps. This little script renders my images to screen as a series of X's.


from PIL import Image
import sys, os


for infile in sys.argv[1:]:

    im = Image.open(infile)
    data = im.getdata()
    for idx in range (0, len(data)):
        i = data[idx]
        if i == 255:
            print 'X',
        else:
            print ' ',

        if (idx+1) % 24 == 0:
            print ''

The output looks like this:


devin@studio:~/src/py$ python showdata.py /home/devin/Video/bitmaps/foo_00041_mod.bmp 

X X X X X X X X X X X X X X X X X X X X X X X X 
X X X X X X X X X X X X X X X X X X X X X X X X 
X X X X X X X X X X X X X X X X X X X X X X X X 
X X X X X X X X X X X X       X X X X X X X X X 
X X X X X X X X X X X X     X X X X X X X X X X 
X X X X X X X X                   X X X X X X X 
X X X X X                   X X X X X X X X X X 
X X X X X                   X X X X X X X X X X 
X X X X X                   X X X X X X X X X X 
X X X X X                     X X X X X X X X X 
X X X X X X                       X X X X X X X 
X X X X X X X X         X X X X X X X X X X X X 
X X X X X X X X             X X X X X X X X X X 
X X X X X                         X X X X X X X 
X X X X X     X X X X X X X X     X X X X X X X 
X X X X X     X X X X X X X X     X X X X X X X 
X X X X X     X X X             X X X X X X X X 
X X X X X     X X X     X X X X X X X X X X X X 
X X X X X     X X X X X X X X X X X X X X X X X 
X X X X X     X X X X X X X X X X X X X X X X X 
X X X X X X X X X X X X X X X X X X X X X X X X 
X X X X X X X X X X X X X X X X X X X X X X X X 
X X X X X X X X X X X X X X X X X X X X X X X X 
X X X X X X X X X X X X X X X X X X X X X X X X

Animated gif to video

2010-06-25T06:49:00.000-07:00

This little recipe worked for the basic job of converting the giv to an avi.


mplayer animated.gif -vo yuv4mpeg
ffmpeg -i stream.yuv -an -r 24 -b 640 -s vga -aspect 4:3 test.avi

However I wanted to loop the gif, but ffmpeg's -loop_input and -loop_ouput had no affect. Decided to make a copy and cat a few times...but this didn't work as expected. (Save yourself time and don't copy this section!)


mencoder -oac copy -ovc copy -forceidx test.avi test2.avi -o test.avi
mencoder -oac copy -ovc copy -forceidx test.avi test2.avi -o test.avi
mencoder -oac copy -ovc copy -forceidx test.avi test2.avi -o test.avi

Tried this with no luck:


 mplayer -loop 10 test.avi -vo yuv4mpeg

Finally found a good way:


avimerge -i test.avi test2.avi test3.avi - o mergedfile.avi

Sweet!

Logitech webcam issues with Ubuntu 10.04 LTS

2010-06-23T04:14:00.000-07:00

Update:

Fixed (well, a workaround): Setting environment variable LD_PRELOAD prior to running video problem solves issues on my box.

$ LD_PRELOAD=/usr/lib/libv4l/v4l1compat.so cheese

---Original Text follows---

cheese and other video programs have been working rather poorly on my AMD64 bit Ubuntu Studio install.

Errors like the following show up:


libv4lconvert: Error decompressing JPEG: unknown huffman code: 0000ffd9
libv4lconvert: Error decompressing JPEG: unknown huffman code: 0000ffff
libv4lconvert: Error decompressing JPEG: unknown huffman code: 0000ffec
libv4lconvert: Error decompressing JPEG: unknown huffman code: 0000ffff
libv4lconvert: Error decompressing JPEG: unknown huffman code: 0000ffd9
libv4lconvert: Error decompressing JPEG: unknown huffman code: 0000fffd

Rebuilding the latest drivers from linuxtv.org didn't help. I'm still looking for a solution.

Note: Same camera produces same error on a second computer running same version of Ubuntu.

lsusb

Bus 005 Device 004: ID 046d:089d Logitech, Inc. QuickCam E2500 series

gstreamer-properties is a great test tool for this kind of thing.

I'm going to log a bug on this one.

New RSS feed

2010-06-20T20:13:00.000-07:00

If you are feeding this blog into Google Reader, update your settings and pull from here instead:

http://www.devinvenable.com/rss/feed.xml

I pull my activity from several sources (including blogger.com) to display a nicely formatted aggregate of my online activity on my home page. But it hasn't been possible to subscribe to the aggregate until now. I use Universal Feed Parser to combine my various feeds and had hoped that there would be a nice method to easily turn the parsed values back into serialized ATOM. Poking around, I didn't find any such method, so I used PyRSS2Gen to pack the parsed dictionary back into RSS. It probably would have been just as easy to load an XML library and just write out XML, but I didn't want to have to read specifications or, really, do much work on this. Still took me at least an hour of my life, but that's not so bad. If it wasn't kind of fun I wouldn't be monkeying with it on my day off!

Allow Sendmail on Centos to accept connections on port 25

2010-06-04T13:25:00.001-07:00

Need to use sendmail as an MTA? It's running but your box is not accepting incoming connections on port 25?

Perhaps you need to do this:

The default sendmail.cf file does not allow Sendmail to accept network connections from any host other than the local computer. To configure Sendmail as a server for other clients, edit the /etc/mail/sendmail.mc file, and either change the address specified in the Addr= option of the DAEMON_OPTIONS directive from 127.0.0.1 to the IP address of an active network device or comment out the DAEMON_OPTIONS directive all together by placing dnl at the beginning of the line. When finished, regenerate /etc/mail/sendmail.cf by executing the following command:


    m4 /etc/mail/sendmail.mc > /etc/mail/sendmail.cf

Then restart the service.

A little something I need to remember...

2010-06-02T11:14:00.000-07:00

To turn off expandtab for editing makefiles, put the following in your vimrc:


autocmd FileType make setlocal noexpandtab

Shrink PDF linux (command line)

2010-05-30T21:17:00.001-07:00

gs -sDEVICE=pdfwrite -dCompatibilityLevel=1.4 -dPDFSETTINGS=/screen -dNOPAUSE -dQUIET -sPAPERSIZE=a4 -dBATCH -sOutputFile=output.pdf input.pdf

Just for fun

2010-05-20T14:04:00.001-07:00


 .file "tiny.cpp"
 .local _ZStL8__ioinit
 .comm _ZStL8__ioinit,1,1
 .section .rodata
.LC0:
 .string "Hello World"
 .text
.globl main
 .type main, @function
main:
.LFB957:
 .cfi_startproc
 .cfi_personality 0x0,__gxx_personality_v0
 pushl %ebp
 .cfi_def_cfa_offset 8
 movl %esp, %ebp
 .cfi_offset 5, -8
 .cfi_def_cfa_register 5
 andl $-16, %esp
 subl $16, %esp
 movl $.LC0, 4(%esp)
 movl $_ZSt4cout, (%esp)
 call _ZStlsISt11char_traitsIcEERSt13basic_ostreamIcT_ES5_PKc
 movl $_ZSt4endlIcSt11char_traitsIcEERSt13basic_ostreamIT_T0_ES6_, 4(%esp)
 movl %eax, (%esp)
 call _ZNSolsEPFRSoS_E
 movl $0, %eax
 leave
 ret
 .cfi_endproc
.LFE957:
 .size main, .-main
 .type _Z41__static_initialization_and_destruction_0ii, @function
_Z41__static_initialization_and_destruction_0ii:
.LFB966:
 .cfi_startproc
 .cfi_personality 0x0,__gxx_personality_v0
 pushl %ebp
 .cfi_def_cfa_offset 8
 movl %esp, %ebp
 .cfi_offset 5, -8
 .cfi_def_cfa_register 5
 subl $24, %esp
 cmpl $1, 8(%ebp)
 jne .L5
 cmpl $65535, 12(%ebp)
 jne .L5
 movl $_ZStL8__ioinit, (%esp)
 call _ZNSt8ios_base4InitC1Ev
 movl $_ZNSt8ios_base4InitD1Ev, %eax
 movl $__dso_handle, 8(%esp)
 movl $_ZStL8__ioinit, 4(%esp)
 movl %eax, (%esp)
 call __cxa_atexit
.L5:
 leave
 ret
 .cfi_endproc
.LFE966:
 .size _Z41__static_initialization_and_destruction_0ii, .-_Z41__static_initialization_and_destruction_0ii
 .type _GLOBAL__I_main, @function
_GLOBAL__I_main:
.LFB967:
 .cfi_startproc
 .cfi_personality 0x0,__gxx_personality_v0
 pushl %ebp
 .cfi_def_cfa_offset 8
 movl %esp, %ebp
 .cfi_offset 5, -8
 .cfi_def_cfa_register 5
 subl $24, %esp
 movl $65535, 4(%esp)
 movl $1, (%esp)
 call _Z41__static_initialization_and_destruction_0ii
 leave
 ret
 .cfi_endproc
.LFE967:
 .size _GLOBAL__I_main, .-_GLOBAL__I_main
 .section .ctors,"aw",@progbits
 .align 4
 .long _GLOBAL__I_main
 .weakref _ZL20__gthrw_pthread_oncePiPFvvE,pthread_once
 .weakref _ZL27__gthrw_pthread_getspecificj,pthread_getspecific
 .weakref _ZL27__gthrw_pthread_setspecificjPKv,pthread_setspecific
 .weakref _ZL22__gthrw_pthread_createPmPK14pthread_attr_tPFPvS3_ES3_,pthread_create
 .weakref _ZL20__gthrw_pthread_joinmPPv,pthread_join
 .weakref _ZL21__gthrw_pthread_equalmm,pthread_equal
 .weakref _ZL20__gthrw_pthread_selfv,pthread_self
 .weakref _ZL22__gthrw_pthread_detachm,pthread_detach
 .weakref _ZL22__gthrw_pthread_cancelm,pthread_cancel
 .weakref _ZL19__gthrw_sched_yieldv,sched_yield
 .weakref _ZL26__gthrw_pthread_mutex_lockP15pthread_mutex_t,pthread_mutex_lock
 .weakref _ZL29__gthrw_pthread_mutex_trylockP15pthread_mutex_t,pthread_mutex_trylock
 .weakref _ZL31__gthrw_pthread_mutex_timedlockP15pthread_mutex_tPK8timespec,pthread_mutex_timedlock
 .weakref _ZL28__gthrw_pthread_mutex_unlockP15pthread_mutex_t,pthread_mutex_unlock
 .weakref _ZL26__gthrw_pthread_mutex_initP15pthread_mutex_tPK19pthread_mutexattr_t,pthread_mutex_init
 .weakref _ZL29__gthrw_pthread_mutex_destroyP15pthread_mutex_t,pthread_mutex_destroy
 .weakref _ZL30__gthrw_pthread_cond_broadcastP14pthread_cond_t,pthread_cond_broadcast
 .weakref _ZL27__gthrw_pthread_cond_signalP14pthread_cond_t,pthread_cond_signal
 .weakref _ZL25__gthrw_pthread_cond_waitP14pthread_cond_tP15pthread_mutex_t,pthread_cond_wait
 .weakref _ZL30__gthrw_pthread_cond_timedwaitP14pthread_cond_tP15pthread_mutex_tPK8timespec,pthread_cond_timedwait
 .weakref _ZL28__gthrw_pthread_cond_destroyP14pthread_cond_t,pthread_cond_destroy
 .weakref _ZL26__gthrw_pthread_key_createPjPFvPvE,pthread_key_create
 .weakref _ZL26__gthrw_pthread_key_deletej,pthread_key_delete
 .weakref _ZL30__gthrw_pthread_mutexattr_initP19pthread_mutexattr_t,pthread_mutexattr_init
 .weakref _ZL33__gthrw_pthread_mutexattr_settypeP19pthread_mutexattr_ti,pthread_mutexattr_settype
 .weakref _ZL33__gthrw_pthread_mutexattr_destroyP19pthread_mutexattr_t,pthread_mutexattr_destroy
 .ident "GCC: (Ubuntu 4.4.3-4ubuntu5) 4.4.3"
 .section .note.GNU-stack,"",@progbits

Inversion of Control pattern

2010-05-04T13:48:00.000-07:00

I'm not a huge fan of the so-called Inversion of Control pattern. One blogger at theburningmonk.com writes, "You have Macaroni code when your application is chopped up into many little pieces and it's difficult to see the big picture which may exist only in your (or some one else's!) head."

Another blogger says, "In software design, you can often end up with Macaroni code when you overuse/misuse/abuse abstractions, and it's one of the main dangers of using Inversion of Control."

This echoes my own thoughts on the matter. It's all well and good to work toward code reuse, and encapsulation is a worthy goal. In my mind the DRY principal is the most important thing. Many ills of poor software design can be cured simply by keeping the code compact, elegant and devoid of redundancy regardless of whether or not the code is procedural, object-oriented, or tiny atomic classes bound together by an XML configuration file.

Procedural code gets a bad rap, as if code is inferior simply because it has a well defined entry and exit point. It's beautiful thing when a debugger can be used to examine a code path from end to end. It's beautiful when you can see the whole picture with your own eyes on a single page. No diagrams needed to comprehend...no scrolling through hundreds of lines of XML to find the relationships. It's all right there in the source code where it, very often, belongs.

Modes matter for password-less login

2010-04-28T10:54:00.000-07:00

I typically setup keys to allow myself password-less access to remote development servers that I use all the time. Today the typical ssh-keygen/deploy public-key routine didn't work as expected. After deploying my public-key to the remote's authorized_keys, I still was getting prompted for a login.

Found this in /var/log/secure.


Apr 28 12:51:35 theserver sshd[16285]: Authentication refused: bad ownership or modes for directory /home/theuser

It turns out the the failure was due to the user's group permissions on the remote machine for two important folders. Both the home folder and the .ssh folder had the following permissions:


drwxrwx--- 36 theuser thegroup 4096 Apr 28 11:56 ..

chmod 700 for both /home/theuser and /home/theuser/.ssh fixed the problem.

Tulsa Developers tied to vendor technology

2010-04-26T17:31:00.000-07:00

From time to time I take a look around to see what kind of programming talent is available in the Tulsa area. Most of what I find is tied to Oracle/Sun (Java) or Microsoft.

It's no secret that Tulsa has been heavy into Microsoft technology for years now. The community colleges and trade schools teach it, the recruiters can get their heads around it, and many conservative businesses would rather go with a name they know.

I remember a Williams employee steering me away from my Borland c++ compiler many moons ago, assuring me that Visual c++ was the future.

Well it was and it wasn't. I embraced Visual c++ and talked my first (serious) employer into letting me use it to write an application for a Phillips petroleum project. That work experience propelled me to my next development job, one I kept for almost a decade.

I wrote a heck of a lot of code using Visual c++ and the MFC framework, until it became my job to port the code to Unix flavors. For days, weeks, months I chased the not-quite-regular constants, the libraries that were similar but rarely identical to the standards. It was then that I started going cold on Microsoft.

I moved away from Microsoft development tools for a lot of reasons including a strong preference for open-source. There's so much free and truly open support on the web. When combined with Linux, open-source makes it possible to examine every nook and cranny of source code down to the kernel level.

Why would new developers gravitate to vendor tools when

they can't examine what is going on under the covers,

they can't control how long the technology will be supported before the vendor deprecates (or abandons) it in favor of new vendor tools,

they can't really have any substantial influence over the evolution of the technology?

I was thinking about these questions as I reviewed search results for "tulsa developers". It turns up these sites and not much more.

www.tulsadnug.org

tulsajava.com

Tulsa seems to have a flourishing Microsoft and Java community. Who is representing everything else that's happening in computer science?

According to Tiobe.com, the most popular programming language is JAVA. Microsoft-based technology doesn't rank until the fifth position, and then it falls off except for position eight. In Tulsa, I'm sure Visual Basic and C# would contend for two of the first three slots.

1. Java - 19.1%
2. C - 15.2%
3. C++ - 10.1%
4. PHP - 8.7%
5. Visual Basic - 8.4%
6. Perl - 6.2%
7. Python - 3.8%
8. C# - 3.7%
9. JavaScript - 3.1%
10. Ruby - 2.6%
11. Delphi - 2.1%

We all know that Tulsa, Oklahoma is a conservative city and not well known for risk taking. And perhaps there is more to the story that a simple web query immediately reveals. Ping.fm is Tulsa-based. Python-friendly Vidoop was founded here, though they relocated and then folded. Perhaps they should have stuck around.

Switch Primary Monitor in Ubuntu 10.4

2010-04-21T16:12:00.000-07:00

The default Monitor Preferences dialog, while sweet, does not allow you to make your secondary monitor your primary. This is a problem if you want the top and bottom Ubuntu Panels displayed on the secondary monitor. Thankfully I found this sweet little script.


#!/bin/sh
#
# Change Primary Monitor for Gnome
# ver 1.0
# 
# Copyright (c) 2010 michal@post.pl
# 
# This file is free software. You can redistribute it 
# and/or modify it under the terms of the GNU
# General Public License (GPL) as published by
# the Free Software Foundation, in version 3.
# It works for me. I hope it works for you as well.
# NO WARRANTY of any kind. 
#


# get list of top-level gnome panels
getTopPanels() {
 gconftool-2 --all-dirs /apps/panel/toplevels
}

# get monitor number for this panel
getMonitor() {
 local PANEL=$1
 gconftool-2 --get $PANEL/monitor
}

# set monitor to display on for given top-level panel
setMonitor() {
 local PANEL=$1
 local NEW=$2
 gconftool-2 --set --type int $PANEL/monitor $NEW
}

# return number of connected monitors
getConnectedMonitors() {
 xrandr --query | grep -c '^.* connected'
}

# compute next monitor
nextMonitor() {
 # number of monitors
 local CURRENT=$1
 local MONITORS=$2
 awk 'BEGIN{ print ('$CURRENT' + 1) % '$MONITORS'; }'
}

# logging finction
log() {
 echo $@ 1>&2
}

# main logic below #############

MONITORS=`getConnectedMonitors`
log "Detected $MONITORS connected monitors"

getTopPanels | while read PANEL
do
 MONITOR=`getMonitor $PANEL`
 NEW=`nextMonitor $MONITOR $MONITORS`
 log "Panel $PANEL is displayed on $MONITOR. Switching to monitor $NEW."
 setMonitor $PANEL $NEW
done

mod_security setup on Centos 5.4

2010-04-15T07:06:00.000-07:00

Enable the EPEL repository.


 rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-3.noarch.rpm

Install via yum.


 yum install mod_security

This will load your basic mod_security configuration including the core rules.

Next I had to set SecDataDir in the config. This was not initially set and errors in the following form appeared in the log file.


ModSecurity: Unable to retrieve collection (name "", key ""). Use SecDataDir to define data directory first.

Fixed this up by creating SecDataDir and creating a directory for this purpose, making sure to give apache permission to use it.


vim /etc/httpd/modsecurity.d/modsecurity_crs_10_config.conf 
( Added SecDataDir /usr/local/apache/modsec_data )
mkdir /usr/local/apache
mkdir /usr/local/apache/modsec_data
chown apache:apache /usr/local/apache/modsec_data
chown apache:apache /usr/local/apache

After a restart modsecurity successfully began applying rules, but rather than blocking problem requests (my intention) it merely logged warnings. I changed the SecDefaultAction in vim /etc/httpd/modsecurity.d/modsecurity_crs_10_config.conf from


SecDefaultAction "phase:2,pass"


SecDefaultAction "phase:2,deny,log,status:403"

vim /etc/httpd/modsecurity.d/modsecurity_crs_10_config.conf

Trouble using sockets in Django view

2010-03-30T12:11:00.000-07:00

I ran into an issue while writing socket communication in code that was called by a Django view handler. After a Twisted-based and socket-based solution failed mysteriously, I discovered a telnet-based solution that worked.

Read more about it the details here:

http://wiki.devinvenable.com/mediawiki/index.php/Interesting_socket_behavior_exhibited_while_processing_Django_view

Comments or insights as to the cause of the problem behavior would be helpful.

git merge

2010-03-25T08:32:00.001-07:00

I used to work with a guy named Jay who, when encountering a particularly complex or difficult programming challenge, would state aloud, "Now that's powerful."

git is a powerful tool. And it really is but, my God, it sure has a way with obscure error messages.

At some point you'll not be able to pull changes from the master (I want to call it the HEAD via cvs, svn concepts) because you are in the middle of a conflicted merge. You'll know this because you'll see this message:


You are in the middle of a conflicted merge.

Unlike svn update, which will pull the latest changes and automatically merge what can be merged and insert diffs where it can't, you need to manually merge the changes. You can open the diff tool with this command:


git mergetool

mergetool doesn't actually work out the diffs---you do. It opens a default diff editor for each file in need of merging, which may be vimdiff if nothing else is installed on your system. You compare the differences, edit to make any changes, and save the file. mergetool then asks you if the diff is complete, and if so, opens the next file. This continues until you have resolved all conflicts.

By the way, mergetool works with a number of different diff tools. See the man page for a complete list. I went with meld, the gnome diff editor. Once installed via apt-get, mergetool opened meld without any additional configuration. Not quite sure how it knew which one I wanted, but it guessed and got it right.

So now you can pull the latest changes, right? Not so fast. You might see an error like this:


fatal: You have not concluded your merge. (MERGE_HEAD exists)

Ah, what does it all mean? I find this to be common thought when using git. Clemens Buchacher had the answer:


The following is an easy mistake to make for users coming from version 
control systems with an "update and commit"-style workflow. 

        1. git merge 
        2. resolve conflicts 
        3. git pull, instead of commit

Yes, that would be me: a fellow used to the "update and commit" style workflow.

For my case, I just had to commit once and then I could, finally, pull changes I had committed on another server.

Automounting removable eSata drive on Centos 5.4

2010-03-08T13:46:00.000-08:00

So my boss says, "I don't get it. When I have a gnome desktop open on the Centos box, my new eSata II removable mounts. But if I don't have a desktop open it doesn't get mounted when I reboot the box."

I ask The Google about it, and it tells me:

http://wiki.centos.org/TipsAndTricks/HAL

But try as I might, gnome-volume-manager fails to detect and mount the eSata drive. The idea of running gnome daemons when in fact I'm working with a server in which gnome is rarely utilized (except for when my boss runs a remote desktop) gives me pause. Hmmm, I think, wouldn't this be a good job for udev rules?

Running dmesg tells me that the device is present and that it's getting mounted as /dev/sdb1. (Or you can tail /var/log/messages) Running the following gives me a list of attributes that are visible to udev on my device.


# udevinfo -a -p $(udevinfo -q path -n /dev/sdb)

With this is see several attributes which can be used for matching when the udev rules are applied. I take the following two:


    SYSFS{rev}=="ST6O"
    SYSFS{model}=="Hitachi HDT72101"

I want to mount the block device as /media/removable each time the device is plugged. I also want to unmount when the device is removed. The following worked for me.


ACTION=="add",SYSFS{rev}=="ST6O",SYSFS{model}=="Hitachi HDT72101",KERNEL=="sd?1",NAME="REMOVABLE", RUN+="/bin/mount /dev/REMOVABLE /media/removable"
ACTION=="remove",SYSFS{rev}=="ST6O",SYSFS{model}=="Hitachi HDT72101",RUN+="/bin/umount /media/removable"

For more information, refer to this helpful page.

http://reactivated.net/writing_udev_rules.html

strptime

2010-02-22T07:08:00.000-08:00

Can't find strptime in datetime because you're working in a Python 2.4 environment?


Python 2.4.3 (#1, Jul 27 2009, 17:56:30) 
[GCC 4.1.2 20080704 (Red Hat 4.1.2-44)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from datetime import datetime
>>> d = datetime.strptime("21-JAN-08", "%d-%b-%y")
Traceback (most recent call last):
  File "", line 1, in ?
AttributeError: type object 'datetime.datetime' has no attribute 'strptime'

strptime was added to datetime in 2.5. Prior to 2.5 you pull it from the typically lower-level library 'time'.


>>> import time
>>> time.strptime("21-JAN-08", "%d-%b-%y")
(2008, 1, 21, 0, 0, 0, 0, 21, -1)

Ah, there it is. Of course, you wanted a datetime, didn't you?


>>> t = time.strptime("21-JAN-08", "%d-%b-%y")
>>> datetime(*t[0:6])
>>> datetime.datetime(2008, 1, 21, 0, 0)

mod_mono, Centos 5 64 bit, and SELinux Part 2

2010-02-09T10:38:00.001-08:00

The trick to creating a SELinux policy is setting the mode to be permissive, which prevents nothing but logs all of the infractions to audit.log, and then using the log to generate the policy. After running my mod_mono based application for a bit in permissive mode, I used this command to generate a local policy.


egrep 'http|mono' /var/log/audit/audit.log | audit2allow -M myhttp

Here is the result:


module myhttp 1.0;

require {
 type httpd_tmp_t;
 type device_t;
 type initrc_t;
 type httpd_t;
 type httpd_sys_script_t;
 type http_port_t;
 type port_t;
 type inotifyfs_t;
 class process { execstack execmem getsched ptrace };
 class unix_stream_socket connectto;
 class chr_file { read write ioctl };
 class tcp_socket name_connect;
 class file execute;
 class sem { unix_read write unix_write associate read destroy };
 class shm { unix_read read write unix_write associate };
 class dir read;
}

#============= httpd_sys_script_t ==============
allow httpd_sys_script_t http_port_t:tcp_socket name_connect;
allow httpd_sys_script_t httpd_tmp_t:file execute;
allow httpd_sys_script_t inotifyfs_t:dir read;
allow httpd_sys_script_t self:process { execmem getsched ptrace };
allow httpd_sys_script_t self:sem { unix_read write unix_write associate read destroy };

#============= httpd_t ==============
allow httpd_t device_t:chr_file { read write ioctl };
allow httpd_t httpd_sys_script_t:unix_stream_socket connectto;
allow httpd_t initrc_t:shm { unix_read read write unix_write associate };
allow httpd_t port_t:tcp_socket name_connect;
allow httpd_t self:process { execstack execmem };

mod_mono, Centos 5 64 bit, and SELinux

2010-02-08T08:28:00.001-08:00

Getting mod_mono up and running on Ubuntu 9.10 is relatively simple. Install the packages, drop in a test asmx file, browse to the URL and you are done.

 
apt-get install libapache2-mod-mono mono-apache-server2

My experience getting the same demo file with Centos 5 running SELinux was a bit more involved. First off, here's the complete simple web service. You should be able to drop it into your document root and browse to the appropriate URL, once mod_mono is properly installed.


<%@ WebService Language="c#" Codebehind="TestService.asmx.cs" Class="WebServiceTests.TestService" %>

using System;
using System.Web.Services;
using System.Web.Services.Protocols;

namespace WebServiceTests
{
    public class TestService : System.Web.Services.WebService
    {
        [WebMethod]
        public string Echo (string a)
        {
            return a;
        }

        [WebMethod]
        public int Add (int a, int b)
        {
            return a + b;
        }
    }
}

On Centos 5, install these packages:


yum install mod_mono xsp mono-web

To enable mod_mono for Apache and run the xsp demo programs, add something like the following to the tail end of your http.conf file. Be sure to check that the paths used here are the same on your machine. (Note that I'm using a 64 bit Centos installation.)



AddType application/x-asp-net .aspx
AddType application/x-asp-net .asmx
AddType application/x-asp-net .ashx
AddType application/x-asp-net .asax
AddType application/x-asp-net .ascx
AddType application/x-asp-net .soap
AddType application/x-asp-net .rem
AddType application/x-asp-net .axd
AddType application/x-asp-net .cs
AddType application/x-asp-net .config
AddType application/x-asp-net .Config
AddType application/x-asp-net .dll
AddType application/x-asp-net .asp
DirectoryIndex index.aspx
DirectoryIndex Default.aspx
DirectoryIndex default.aspx


Alias /demo /usr/lib64/xsp/test
MonoApplications "/demo:/usr/lib64/xsp/test"
MonoServerPath /usr/bin/mod-mono-server

You are likely to run into myriad problems if using SELinux. Start with giving permissions to run mono to httpd.


chcon -t httpd_sys_content_t '/usr/bin/mono'

Each time you hit your URL you will likely encounter another SELinux error. You can repeat this process again and again until you come up with a final policy that will allow apache access to mono, its directories, and dependencies. My final policy looked like this.


module mymono 1.0;

require {
        type lib_t;
        type tmp_t;
        type mono_exec_t;
        type httpd_t;
        type httpd_sys_script_t;
        class process ptrace;
        class sock_file { write create };
        class sem create;
        class file { read execute_no_trans };
}

#============= httpd_sys_script_t ==============
allow httpd_sys_script_t self:sem create;

#============= httpd_t ==============
allow httpd_t lib_t:file execute_no_trans;
allow httpd_t mono_exec_t:file { read execute_no_trans };
allow httpd_t self:process ptrace;
allow httpd_t tmp_t:sock_file { write create };

Mono makes extensive use of a temp directory known as the wapi directory. It is possible for you to specify your own temp directory in your http.conf file or else the default will be used: /tmp/.wapi.

It took awhile to discover that /tmp/.wapi needed different permissions. The best clue I could get from messages was:


Feb  8 08:43:32 carbon setroubleshoot: SELinux is preventing the mono from using potentially mislabeled files (mod_mono_server_global). For complete SELinux messages. run sealert -l a00a5946-cec1-4291-a410-e74c5f96edfd

This was corrected by running...


restorecon -R -v /tmp/.wapi

...as suggested by sealert.

Just as I thought I was finished, as the mono test application was finally working, I found additional errors in the /var/log/audit/audit.log. This policy was the fix:


module mynotify 1.0;

require {
        type httpd_t;
        type inotifyfs_t;
        class dir read;
}

#============= httpd_t ==============
allow httpd_t inotifyfs_t:dir read;

Are we done yet? I sure hope so. I read elsewhere on the web that there is a plan to get the proper SELinux configuration into the mod_mono RPMs. Until that happens, I hope that this info will help you to get your mod_mono setup working.

Note: After rebooting, I had to relabel the temp and bin directory with these two commands:


restorecon -R -v /tmp/.wapi
chcon -t httpd_sys_content_t '/usr/bin/mono'

I'm currently looking for a better, permanent solution.

ISO 9797 algorithm 3

2010-01-25T11:56:00.000-08:00

Last Friday I set out to implement ISO 9797 algorithm 3 using the OpenSSL library. I did not have the specification handy, so I decided to to the best I could with what I could find by way of examples on the net.

I came across a description of the algorithm in this 2005 thread (http://www.derkeiler.com/Newsgroups/sci.crypt/2005-02/0374.html). This was posted in a query by someone named Christian. He also posted his keys, data and the expected answer.


H0 = 0
stages 1 to n: Hj = Enc(K, Dj XOR H{j-1})
MAC = Enc(K, Dec(K', Hn))

Francois Grieu replied with, "This is very likely ISO/IEC 9797-1, using DES as the block cipher,
padding method 2, MAC algorithm 3." He provided an answer by sharing sample code in "some near-extinct dialect".


set m0 72C29C2371CC9BDB #message
set m1 65B779B8E8D37B29
set m2 ECC154AA56A8799F
set m3 AE2F498F76ED92F2

set pd 8000000000000000 #padding

set iv 0000000000000000 #initialisation vector

set k0 7962D9ECE03D1ACD #key
set k1 4C76089DCE131543

set xx {iv} # setup
for mj in {m0} {m1} {m2} {m3} {pd} # for each block including padding
 set xx `xor {xx} {mj}` # chain
 set xx `des -k {k0} -c {xx}` #encrypt
end
set xx `des -k {k1} -d {xx}` #decrypt
set xx `des -k {k0} -c {xx}` #encrypt
echo {xx} #show result

5F1448EEA8AD90A7

I've implemented the same in c for the purpose of research.


#include openssl/des.h
#include memory
#include string.h

//message + padding
const unsigned char msg[40] = { 0x72, 0xC2, 0x9C, 0x23, 0x71, 0xCC, 0x9B, 0xDB,
                              0x65, 0xB7, 0x79, 0xB8, 0xE8, 0xD3, 0x7B, 0x29,
                              0xEC, 0xC1, 0x54, 0xAA, 0x56, 0xA8, 0x79, 0x9F,
                              0xAE, 0x2F, 0x49, 0x8F, 0x76, 0xED, 0x92, 0xF2,
                              0x80, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 };

//initialization vector
unsigned char iv[8] = { 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 };

unsigned char k0[8] = { 0x79, 0x62, 0xD9, 0xEC, 0xE0, 0x3D, 0x1A, 0xCD };
unsigned char k1[8] = { 0x4C, 0x76, 0x08, 0x9D, 0xCE, 0x13, 0x15, 0x43 };


void print_hex(const unsigned char *bs, int n) {

    for (int i = 0; i < n; i++)
        printf("%02x", bs[i]);
    printf("\n");
}

void des_ecb_crypt(unsigned char* input, unsigned char* output, int encrypt, unsigned char* key) {

    des_key_schedule sched;
    des_set_key((des_cblock *) key, sched);

    DES_ecb_encrypt((const_DES_cblock *)input,
                     (const_DES_cblock *)output,
                     &sched,
                     encrypt);
}

void xor_block(unsigned char* src, unsigned char* dest) {

    for (int x = 0; x < 8; x++) {
       src[x] =  src[x] ^ dest[x];
    }
}

int main(int argc, char* argv[]) {

    unsigned char output[8];
    unsigned char xx[8];
    unsigned char block[8];
    int offset = 0;

    memcpy(xx, iv, 8);

    // Chain and encrypt 5 8-bit blocks
    for (int x = 0; x < 5; x++) {

        memcpy(block, &msg[offset] , 8);
        offset+=8;

        //set xx `xor {xx} {mj}` # chain
        xor_block(xx, block);

        //set xx `des -k {k0} -c {xx}` #encrypt
        des_ecb_crypt(xx, output, DES_ENCRYPT, k0);
        memcpy(xx, output, 8);
    }


    des_ecb_crypt(xx, output, DES_DECRYPT, k1);
    memcpy(xx, output, 8);

    des_ecb_crypt(xx, output, DES_ENCRYPT, k0);
    memcpy(xx, output, 8);

    print_hex(xx, 8);
    return 1;
}

Finding hidden characters in file using Vim

2010-01-15T08:52:00.000-08:00

Show newlines and tab location:


:set list

Back to normal:


:set nolist

View open file as hex:


:%!xxd

Apache, mod_wsgi, Django, SELinux

2010-01-05T09:13:00.000-08:00

I've come to rely on the fact that if something isn't working after an installation on Centos 5, it is probably due to bad SELinux permissions. I've learned to live with SELinux and generally can handle any complication it throws at me. I've learned to keep an eye on /var/log/messages and /var/log/audit/audit.log. I've learned to use sealert and how to make my own local policies. Yet still it finds a way to throw me under the bus.

This time around I'm installing Django under Apache. I finally moved from mod_python to mod_wsgi. With mod_wsgi, less configuration is needed in httpd.conf, but more is needed in an external config file. For example, my original mod_python configuration looked like this:


<Directory "/var/www/html/python">
    AddHandler mod_python .py
    PythonHandler mptest
    PythonDebug On
</Directory>

<Location "/">
    SetHandler python-program
    PythonHandler django.core.handlers.modpython
    SetEnv DJANGO_SETTINGS_MODULE hsm.settings
    PythonDebug On
    PythonPath "['/home/stuff/src/python', '/home/stuff/src/python/hsm'] + sys.path"
</Location>

The second section is the Django deployment. (The first I included to remind myself how simple it is to deploy a very basic test program using mod_python. )

With mod_wsgi, the apache config looks like this. A single line identifies the location of the wsgi configuration, and the Directory element is used to give Apache permission to access the script.


WSGIScriptAlias / /usr/local/www/wsgi-scripts/hsm.wsgi

<Directory /usr/local/www/wsgi-scripts>
Order allow,deny
Allow from all
</Directory>

My custom wsgi script includes these lines. The last line keeps the wsgi handler from puking on
print lines that might be included in your file.


    import os
    import sys

    os.environ['DJANGO_SETTINGS_MODULE'] = 'hsm.settings'

    import django.core.handlers.wsgi
    application = django.core.handlers.wsgi.WSGIHandler()

    sys.stdout = sys.stderr

For this particular Django application, I decided to use sqlite3 instead of mysql. But attempting to launch proved problematic.



[Tue Jan 05 11:02:08 2010] [error] [client 10.8.8.62]     return query.execute_sql(return_id)
[Tue Jan 05 11:02:08 2010] [error] [client 10.8.8.62]   File "/usr/lib/python2.4/site-packages/django/db/models/sql/subqueries.py", line 320, in execute_sql
[Tue Jan 05 11:02:08 2010] [error] [client 10.8.8.62]     cursor = super(InsertQuery, self).execute_sql(None)
[Tue Jan 05 11:02:08 2010] [error] [client 10.8.8.62]   File "/usr/lib/python2.4/site-packages/django/db/models/sql/query.py", line 2369, in execute_sql
[Tue Jan 05 11:02:08 2010] [error] [client 10.8.8.62]     cursor.execute(sql, params)
[Tue Jan 05 11:02:08 2010] [error] [client 10.8.8.62]   File "/usr/lib/python2.4/site-packages/django/db/backends/util.py", line 19, in execute
[Tue Jan 05 11:02:08 2010] [error] [client 10.8.8.62]     return self.cursor.execute(sql, params)
[Tue Jan 05 11:02:08 2010] [error] [client 10.8.8.62]   File "/usr/lib/python2.4/site-packages/django/db/backends/sqlite3/base.py", line 193, in execute
[Tue Jan 05 11:02:08 2010] [error] [client 10.8.8.62]     return Database.Cursor.execute(self, query, params)
[Tue Jan 05 11:02:08 2010] [error] [client 10.8.8.62] OperationalError: attempt to write a readonly database

To read and write to the sqlite database, I had to:

1. Ensure the database file and its parent directory was owned by the apache user. (The location of the database file is specified in the Django settings.py file for your project.)

2. Add a Directory entry to the httpd.conf file for the location of my database file. By default, apache does not have access to directories not under DocumentRoot.

3. Apply SELinux level permissions to the database file and its parent directory.

It took me quite awhile to discover that I had SELinux level permissions because I did not receive notifications in the /var/log/messages file. Normally all security exceptions arrive there and it is easy enough to tail the file and look for events. For whatever reason, the alerts were not appearing in the log, at least not every time. For example, the settroubleshoot alert browser (sealert -b) showed the exact problem that occurred at 11:02 AM (below), but the /var/log/messages file had no corresponding entry. The log did have a similar message from 10:59. The log either does not receive duplicate messages for SELinux events, or that some kind of bug is to blame.


Summary:

SELinux is preventing the httpd from using potentially mislabeled files
./sqlite3.db (usr_t).

Detailed Description:

SELinux has denied the httpd access to potentially mislabeled files
./sqlite3.db. This means that SELinux will not allow httpd to use these files.
Many third party apps install html files in directories that SELinux policy
cannot predict. These directories have to be labeled with a file context which
httpd can access.

Allowing Access:

If you want to change the file context of ./sqlite3.db so that the httpd daemon
can access it, you need to execute it using chcon -t httpd_sys_content_t
'./sqlite3.db'. You can look at the httpd_selinux man page for additional
information.

The solution was to modify the SELinux permissions on the folder that contained the sqlite database:


sudo chcon -R system_u:object_r:httpd_sys_content_t database_folder

Hope this information will help the next unlucky person to deploy in a similar environment.

Compiling ZynAddSubFX on Ubuntu 9.04

2009-12-28T06:50:00.000-08:00

ZynAddSubFX does not compile from source by default and there is no configure file. Add the following packages and the build will succeed.


apt-get install fluid libmxml-dev libfftw3-3 mffm-fftw-dev libasound2-dev

I've never been happy with the appearance of ZynAddSubFX on linux. It seemed the fonts were all too big, crowding the screen and buttons. I was going to look into fixing this, which was the reason I downloaded the source version in the first place. But to my surprise, the version I compiled myself looked great. I'm guessing that fluid makes geometric decisions based on the video display it is used on and that the package maintainer uses a lower screen resolution than I do, thus the crappy display.

M-Audio / Freebob and Ubuntu 9.10

2009-12-24T18:04:00.001-08:00

In setting up a new Ubuntu 9.10 install with ubuntustudio, I had to do a few manual things. There is an UbuntuStudio distribution, but I typically opt to install what I need later, which is generally just the awesome audio stuff. If you go this route and want to use jack, be sure to also install the real time kernel.


sudo apt-get install ubuntustudio-audio
sudo apt-get install linux-rt

I use a M-Audio FiewWire Solo audio card and have used the freebob driver with success in the past. I was happy to see that it is installed by default now with qjackctl. But when I started jack initially I got errors suggesting the driver was not loaded. I had to install the driver and change permissions before I jack would start up.


sudo modprobe raw1397
sudo chmod a+rw /dev/raw1394

Chmod was transient however, and after rebooting /dev/raw1394 didn't have the correct permissions. I added a file called 40-permissions.rules and gave the audio group permission to access the device. I failed to mention earlier that I also had to add my personal user to the "audio" group before getting jack to start up.


sudo vim /etc/udev/rules.d/40-permissions.rules
# EEE1394 (firewire) devices
KERNEL=="raw1394",          GROUP="audio"

So I was done right? No. After reboot I still didn't have permission to access the firewire device. ls told me why.


ls -altr /dev/raw1394 
crw-rw---- 1 root video 171, 0 2009-12-24 19:55 /dev/raw1394

I guess by default /dev/raw1394 belongs to the video group. Well my user doesn't, and I'm not using firewire for video, so I switched it to belong to audio.


dextron@dextron:~$ sudo chgrp audio /dev/raw1394
ls -altr /dev/raw1394 
crw-rw---- 1 root audio 171, 0 2009-12-24 19:55 /dev/raw1394

After another reboot, all is well. I'm wondering if I should have just left the group as "video" and added my user to the video group plus changing my udev rule (to use video instead of audio). I suspect there is no single right answer, but if you have any insights, please leave a comment. Thanks!

Creating digital synth sounds using Python

2009-12-07T08:15:00.001-08:00

I used to really be into electronic music. I had a bunch of keyboards, synths and digital samplers and I would use them, along with a computer, to compose techno compositions. In fact, I took up c++ programming in the 90's with the intention of building a drum sequencer for Windows. Well that project didn't get finished and I got sidetracked writing code for corporations; after all you have to pay the bills first and find time to play around later.

Now seems like a good time to resume play.

Since Python is my language of choice these days, I thought I'd see what's out there by way of using the language to generate digital sounds. I found a wave generation library and a few sample snippets on the web, including code which can be used to produce a simple sine wave. I've been sampling since the 80's using Roland, Ensoniq and EMU samplers, and I have a pretty good understanding of the basics. But it turns out I had more to learn at the lowest level. Consider the following snippet of code, which can be used to generate a 90 Hz tone.


import wave, random, math, numpy

noise_output = wave.open('tone.wav', 'w')
noise_output.setparams((2, 2, 44100, 0, 'NONE', 'not compressed'))

# num of seconds
duration = 4

# Hz per second
samplerate = 44100

# total number of samples
samples = duration * samplerate

# pulse per second
frequency = 90 # Hz

"""
The time of one sample is the inverse of the sample rate, and the period is the inverse of the frequency, so the number of samples is also the sample rate divided by the frequency.
"""
period = samplerate / float(frequency) # in sample points

"""
This is the phase increment.
"""
omega = numpy.pi * .2 / period

"""
Creates the x-axis set with 'period' number of items.  numpy.arange(int(period), dtype = numpy.float) produces {0..146}, but since each value is a factor of omega, the transformed set is in the range {0..0.627}.
"""
xaxis = numpy.arange(int(period), dtype = numpy.float) * omega

"""
This snippet calculates the sin for each value in xaxis and multiplies that value with 16384.  Here we're creating the y-axis data.
"""
ydata = 16384 * numpy.sin(xaxis)


"""
If we were to graph the sets now, we would see an inclining sin wave.  Resize creates an extended array which repeats the ydata chunk, the result being something that looks a bit more like a saw wave than a sin wave due to the omega calculation.
"""
signal = numpy.resize(ydata, (samples,))

for i in signal:
    packed_value = wave.struct.pack('h', i)
    noise_output.writeframes(packed_value)

noise_output.close()

http://en.wikipedia.org/wiki/Direct_digital_synthesis

Find all non-binary files in a directory

2009-12-04T06:41:00.000-08:00

You've got to love that there are so many different ways to accomplish this. Here's one:


find . -type f | xargs file | grep -v ELF

1. Recursively finds all files in current directory
2. Pipes to file, which displays information about each file type
3. Pipes to grep, which checks to see that ELF is NOT in result text

file is a handy command. On Linux, binary files with show something like the following.


dvenable@dvenable:~/src$ file bin.exe
bin: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.15, not stripped

Save with CTRL-S using VIM and copy/paste with CTRL-C, CTRL-V

2009-12-03T07:01:00.000-08:00

Oh now this has got to be the best VIM find in years. I love VIM but get sick of copying from a web browser and having to visual paste in strange ways. It's my one holdover from Windows, the desire to use CTRL-C to copy and CTRL-V to paste. It turns out that this is easily remedied by modifying your .vimrc file and adding VIM mappings.


nmap <C-V> "+gP
imap <C-V> <ESC><C-V>i
vmap <C-C> "+y

That's it. Now enjoy the most standard shortcuts in your favorite VIM editor. The mappings looked so easy that I thought I'd try one of my own. Another thing that causes my fingers to grow weary is saving. The typical sequence is: press escape, then :w enter. I save all the time because I'm paranoid and I miss CTRL-S which is so much easier on the hands.


map <C-S> <ESC>:w<CR>
imap <C-S> <ESC>:w<CR>

Oh yes it is a sweet discovery!

Multi-file search and replace one-liner

2009-11-30T11:55:00.000-08:00


find . -name  "*.ksh" -exec sed -i 's/oldtext/newtext/g' {} \;

Centos 5, SELinux, and Bugzilla

2009-11-17T07:34:00.001-08:00

I really don't like SELinux. There. I said it.

After installing bugzilla from the yum repository I found that it could not send email notifications. Why? SELinux. The solution? Here.


[dvenable@somecorporateserver ~]$ sudo chcon -R --reference=/var/www/html /usr/share/bugzilla

KVM on Centos 5

2009-11-13T06:14:00.000-08:00

I've been a XEN man for sometime now, but yesterday my employer asked me to set up KVM on one of our Centos 5 servers. It was a snap.

Installation


 yum install kvm qemu virt-manager libvirt

The next step is to modprobe the kvm module for your architecture. Use the module that is right for you:


 modprobe kvm-intel
 ...or...
 modprobe kvm-amd

If all goes well, you should have the kvm module loaded on your system by now. You can check this by running:


 /sbin/lsmod | grep kvm

Setup the Bridge

If you want to access the virtual machine from the LAN, you'll need to set up a bridge on one or more of your network interfaces. Once this step is complete, virt-manager will take care of the remaining details when setting up a virtual machine using the GUI.

Check that a default bridge is configured. Run brctl show and you should see something like this.


 [root@server ~]# brctl show
 bridge name bridge id  STP enabled interfaces
 virbr0  8000.000000000000 yes

What you should see is a line that includes vnet0. Don't screw the pooch like I did and use virbr0 if vnet0 is not listed. virbr0 is used by NAT and you will hose the system if you go this route. IF vnet0 is not listed, restart networking. If still not present, use Google to resolve before continuing. Your goal is to see this:


bridge name bridge id  STP enabled interfaces
virbr0  8000.000000000000 yes  
vnet0  8000.000000000000 no

Create a file in network-scripts called ifcfg-vnet0...


 vim /etc/sysconfig/network-scripts/ifcfg-vnet0

...and add the following if you want to configure the bridge for eth0:


 DEVICE=vnet0
 TYPE=Bridge
 BOOTPROTO=dhcp
 ONBOOT=yes

Now add the bridge:


  brctl addif vnet0 eth0

Now you'll see:


 [root@server ~]# brctl show
 bridge name bridge id  STP enabled interfaces
 virbr0  8000.000000000000 yes  
 vnet0  8000.003048625732 no  eth0

To make this sticky (available after reboot), add BRIDGE=vnet0 to /etc/sysconfig/network-scripts/ifcfg-eth0.

Create the Virtual Machine

It's possible to create a virtual machine from the command line like this:


 qemu-kvm -hda windisk.img -cdrom winimg.iso -m 1024 -boot d

However, it's probably easier to manage things using virt-manager. Launch from the UI or from the command line.


 virt-manager

Click on the first row to highlight it, right-click and select new to open the wizard. Follow the steps including choice of ISO and where to write the image. When you get to the network screen, choose Shared Physical Device and you will see your bridged eth0 interface. If you skipped bridging, you will not have this option so take a different one.

Once complete, start up the VM. Done.

Migrating schema from Oracle 11g Enterprise to 10g Express Edition

2009-11-06T06:04:00.000-08:00

Here's the situation: you are working in an environment that uses Oracle Enterprise 11g, you want to do some development on your personal computer using your own database, and for whatever reason you cannot simply install a full-blown non-free version of Oracle on your laptop.

Your best free option is to install Oracle 10g XE, especially if you are running Ubuntu because installation is a snap.

http://www.oracle.com/technology/tech/linux/install/xe-on-kubuntu.html

Of course you should know that XE is stripped down, so read up and make sure that you won't be losing features that you'll absolutely need to run your production or test database.

The first step is to export your database. If you're like me an you attempt to simply export using the exp tool that ships with 11g, you will discover that there are incompatibilities between version which will prevent you from importing with 10g XE.

The easy solution? Just use the 10g version of the tools that ship with XE to dump the 11g database. In my environment, everything is Linux, so my quick-and-dirty approach was to create a temp directory on the 11g server and scp the 10g version of exp over to it.


cd $ORACLE_HOME/bin
scp exp me@11gserver:~/temp

Of course I don't have the needed 10g libs on the 11g server, so I just copy the all Oracle libs over to the same directory.


scp /usr/lib/oracle/xe/app/oracle/product/10.2.0/server/lib/* me@11gserver:~/temp

Now I ssh over to the 11g server and set up my environment to add the current directory (temp) to the LD_LIBRARY_PATH.


export LD_LIBRARY_PATH=.:/usr/lib32:/usr/local/lib32

Now to dump...


exp SYSTEM/stuff@11gserver:1521/yeah.stuff.com FILE=export.dmp OWNER=me_the_owner

Now copy the resulting export.dmp file back to your target computer, the one running 10g XE. (You know how to do it!)

Okay, now cross your fingers and attempt the import, hoping to succeed though you just know that critical 11g features won't be supported on XE and will probably cause all your effort to be for naught. But do it anyway.

In my case I now yell "Doh!" because I get the same damn error I had before. Did I mention the error? Yes, the one I got after importing initially, the export file created using 11g's exp. This is what I see on my development laptop.


imp SYSTEM/xx FILE=/home/me/export.dmp FULL=y

Import: Release 10.2.0.1.0 - Production on Fri Nov 6 08:32:09 2009

Copyright (c) 1982, 2005, Oracle.  All rights reserved.


Connected to: Oracle Database 10g Express Edition Release 10.2.0.1.0 - Production

IMP-00010: not a valid export file, header failed verification
IMP-00000: Import terminated unsuccessfully

I check the head...aha I see my error. I was still using the darn 11g exp because I failed to put the 10g exp on the path on the 11g server before I ran exp.


 head export.dmp 
EXPORT:V11.01.00
DSYSTEM
RUSERS

Oh, and in case you are thinking you might just break out a HEX editor and change the EXPORT version in the header to the correct version, it won't work. Yes I thought of that too.

Okay, back to the 11g server to re-run. This time notice the ./ before exp to pick up the correct version of the executable.


./exp SYSTEM/stuff@11gserver:1521/yeah.stuff.com FILE=export.dmp OWNER=me_the_owner

Once you "get it right" move the export back to your development box. In my case, I had to create matching tablespaces and users before the importing with success. Once that was done, I ran the following...


imp SYSTEM/xx FILE=/home/me/export.dmp FULL=Y 
...
lots of output here, such as
 row rejected due to ORACLE error 12899
...
Import terminated successfully with warnings.

I believe my warnings are due to character set incompatibilities, and since I'm set to blow up this copy of the database in a development environment anyway, I'm not going to research them now.

I fired up SqlDeveloper, connected to my 10g XE instance, and selected some data. No problem! Who says you can't move an Oracle Enterprise 11g database to Oracle 10g XE?

Netbeans and Make

2009-10-30T09:18:00.000-07:00

Netbeans for c++ uses the standard make tools. The top level Makefile is always the same for every project and is the correct place to add custom build steps. The nbproject subproject files are modified by the NetBeans editor, so it is best to avoid making changes there.

There are hooks in the top-level make where various custom steps can be inserted.


#
#  There exist several targets which are by default empty and which can be 
#  used for execution of your targets. These targets are usually executed 
#  before and after some main targets. They are: 
#
#     .build-pre:              called before 'build' target
#     .build-post:             called after 'build' target
#     .clean-pre:              called before 'clean' target
#     .clean-post:             called after 'clean' target
#     .clobber-pre:            called before 'clobber' target
#     .clobber-post:           called after 'clobber' target
#     .all-pre:                called before 'all' target
#     .all-post:               called after 'all' target
#     .help-pre:               called before 'help' target
#     .help-post:              called after 'help' target
#
#  Targets beginning with '.' are not intended to be called on their own

You can also add your own targets, just as if you were building a makefile from scratch.

Generating HTML color syntax highlighting from PRE tag

2009-10-27T05:50:00.000-07:00

I'm working on a Django template tag that should transform code embedded within a PRE tag and convert it to nicely formatted HTML with color syntax. To test, I inserted this block of code here, the same code I wrote to do the conversion.


#!/usr/bin/python

from django import template
from django.template.defaultfilters import stringfilter
from django.utils.safestring import mark_safe
from BeautifulSoup import BeautifulSoup
from pygments.lexers import guess_lexer, guess_lexer_for_filename
from pygments import highlight
from pygments.lexers import get_lexer_by_name, TextLexer
from pygments.formatters import HtmlFormatter
import re

register = template.Library()

@stringfilter
def tocode(value):
    try:
        commentSoup = BeautifulSoup(value)
        c = commentSoup.findAll('pre')
        for all in c:
            brs = all.findAll('br')
            for item in brs:
                item.replaceWith('\n')
            joined = ''.join(all.findAll(text=True))

            if all.has_key('class'):
                lex = get_lexer_by_name(all['class'], stripall=True)
            else:
                try:
                    lex = guess_lexer(joined)
                except:
                    lex = BashSessionLexer
            formatter = HtmlFormatter(linenos=True, cssclass="source")
            result = highlight(joined, lex, formatter)
            all.replaceWith(result)

        return mark_safe(commentSoup)

    except:
        return value

register.filter('tocode', tocode)

This is how it works: You pull a feed from, for example, blogger, and look for PRE tags, assuming that there is something interesting (like a code snippet) inside. After discovering that Pygments' guess_lexer has a hard time identifying most of the snippets I feed it, I decided to make it possible to explicitly specify the PRE content type. I do this by tagging the PRE element with a class name. In this case, I use the class name verbatim to call the get_lexer_by_name method. So this...

<pre class="python" >

...will look up the python lexer and

<pre class="php" >...

will look up the php lexer.

The PHP lexer is very disappointing actually.

Originally, I set the code to use the TextLexer in the event that PRE class attribute was not present, but this was boring. I found that the python lexer produces more appealing results for almost all snippets, so now I'm using it as the default when the PRE attribute is not specified. Of course, not all content will be python lexerable, so in the case of exception I fail over to the TextLexer. The exception handling chain is a bit ugly but it gets the job done.

Common git operations

2009-10-08T11:02:00.000-07:00

Clone repository via ssh


git clone dvenable@dvenable:~/git/python

Show files that have changed:


git status

See all changes to a file:


git diff HEAD^ HEAD filename

See changes for a particular commit:


git show HEAD

Show commit log for a particular file:


git log filename.py

Show diff for all changes since branch master was created:


git diff master index.html

Ignite Tulsa

2009-09-18T06:05:00.000-07:00

Last night's Ignite Tulsa event was a blast. The beer was cold, the first one was free, and the talks were often amusing.

The best presentation had to be "If someone gives you roses you should be pissed off" by Matthew Galloway. I follow Galloway on twitter, but, as one not twitter obsessed, haven't really paid too much attention to him before. I simply knew him as the guy who had T-Shirts made up that said, "I'm following @mattgalloway #supergenius".

Well a super genius he is, or if that's overstating it, he is at least a super dynamic speaker. His observations, from roses to Harley Davidson motorcycles, were poignant and so damn true. The point of his five minute twenty slide rant: Have an original thought already.

Hopefully he'll post his slide deck or a video will be made available.

I also want to give props to my former colleague and fellow entrepreneur Geoffrey Simpson. His presentation "Challenging Yourself" was a real inspiration. Geoff is always trying new things, and presenting to crowd that size might have been one of those first steps he described. His conviction, energy, and positive message definitely won over the crowd.

There were other notable presentations, too many to mention. Here's a last one: @OKLibrarian was hilarious.

Verified By Visa

2009-08-21T13:38:00.000-07:00

I've been studying Verified By Visa from the Issuer perspective for an upcoming project. You can can get an official Visa overview by reading this document.

Verified By Visa is more-or-less the brand name for the 3-D Secure service. 3-D stands for "Three Domain", referring to the three parties (Visa calls them domains) that provide the software that comprise the service.

Issuer Domain

Implementor: Card holder account; Card issuer or processor
Servers: Access Control Server, Authentication Enrollment Server (or pages)

Interoperability Domain

Implementor: Visa
Servers: Visa Directory Server, Authentication Server

Acquirer Domain

Implementor: Merchant
Servers: Web Server fitted with Merchant Server Plug-in

The issuer implementation is a relatively straightforward secure HTTPS web service. It is even possible (and permitted) to use a single web server instance to fulfill the roles of both Access Control Server and Authentication Enrollment Server. The Issuer server accepts requests from merchant web pages via web requests sent AJAX style and makes requests of Visa's Interoperability Domain servers. All communication is done using a straightforward XML protocol.

It's quite the network dance that goes on between all of the players. Visa apparently distributes a JavaScript library to each merchant for use on their web sites that abstracts the details of the interaction for them, hiding the complexities of the communication to and from the Issuer and Visa servers.

I created a nice diagram that details the communication flow between the 3-D parties, but I probably shouldn't publish it. Ask me if you've got questions.

cx_Oracle on Ubuntu 9.04 64 bit

2009-07-22T13:38:00.000-07:00

I had a heck of a time getting cx_Oracle for 64 bit. Using the 32 bit version didn't work either, possibly due to the ultimate problem that I eventually fixed locally.

Compounding the problem: Sourceforge was buggy as hell today and I couldn't download anything due to excessive redirects. What's up with Sourceforge?

Finally found the package here:

http://rpm.pbone.net/index.php3/stat/6/idpl/12200190

After installing, still no dice. Importing cx_Oracle failed with a message saying unable to locate the package. A search of my directories showed that the RPM dropped my site-package in the wrong location, at least wrong for Ubuntu 9.04. This fixed it:

sudo cp /usr/lib/python2.6/site-packages/cx_Oracle-5.0.1-py2.6.egg-info /var/lib/python-support/python2.6
sudo cp /usr/lib/python2.6/site-packages/cx_Oracle.so /var/lib/python-support/python2.6

/var/lib instead of /usr/lib... wtf

Startup Lessons #3 - What will it take to feed the founders?

2009-05-04T09:16:00.000-07:00

If you can afford to develop your business plan and prototype after hours and on the weekend--- while keeping your day job---do it.

If you have a little bit of startup capital, assume that you will never receive another dime.

Be realistic about what it is going to take to feed your founders. Can they live on the amount in-hand for one year? If not, don't quit the day job.

Our business was born of four founders, all of whom were fairly mature in their careers. It felt like we were making big sacrifices by paying ourselves approximately 60% of our former incomes. Still---the combined amount required was considerable. With what we raised we could have paid two of the founders full-time salaries for a year. But we all went in full-time, leaving us with only 6 months running room. It wasn't enough.

Six more months would have afforded us the opportunity to complete a pilot we began at a financial institution---a pilot with great promise.

So why did we decide the four founders should quit their day jobs up front? A number of reasons, including a strong belief that, to make the most of our combined energies, we must all be 100% committed to the effort. That creativity would best flow if we could all work together in the same room. There may have even been a "fun" factor. All wanted in on the excitement of rapidly building something from the ground up.

What is the take away? Don't let irrational exuberance into the equation. Plan for the worst. If you can't support your founders for a year with what you have in-pocket, think hard. Stick with the day job or find other creative ways to support yourself while you build the company.

Correcting VIM indention, Python

2009-04-28T09:26:00.000-07:00

First off, get your VIM editor up to speed.

http://henry.precheur.org/2008/4/18/Indenting_Python_with_VIM.html

Check that your filetype and settings are correct...


:filetype

filetype detection:ON  plugin:ON  indent:ON                   1,0-1         Top

Now issue the magic VIM command to re-tab the entire file.


gg=G

Or here's an alternative that converts all tabs to spaces


:set tabstop=4
:set shiftwidth=4
:set expandtab
:retab

Revisiting Pyshards

2009-04-27T09:09:00.000-07:00

So I had this idea last year after reading several white papers on database scaling techniques. I also had about a week's time between paid projects to spend however I liked. So I turned my ideas into pyshards, a quick and dirty horizontal database partitioning library. At the end of that week, I published my effort on Google Code for a number of reasons:

I really couldn't find a Python-based toolkit like the one I was imagining at the time, and I needed it.

I was looking for Python gigs and wanted to be able to easily refer hiring technologists to something I had written in Python. (Most of my previous work had been written in JAVA or C++, or could not be made public.)

After years of using great free and open source, I was ready to give something back.

I was curious if others would volunteer to help me build the library.

I received a number of messages from other coders saying they were looking a tool like the one I was building and would be interested in joining the project. But the messages were about as far as it went. Actual participation from the outside was nill.

I went on to use pyshards in my next project, but not quite in the capacity I had originally envisioned. I did use the tool to configure my shards and I used its distribution mechanism to evenly spread data across the many databases. I didn't end up use it for querying, as I needed something a little different. In the following months I went on to create a new page (in the Django sub-project) that visually communicated the shard organization and remaining capacity, but that was the only new work done.

Though the library was imperfect and incomplete, it certainly worked for my purposes. I gave it little thought over the next several months. My hands were full building a new system for the company I had started with my partners.

Jump ahead to PyCon 2009 in Chicago. I had a few hours to kill on the last day before catching my plane and decided to attend an OpenSpaces session called "Is my code Pythonic?" I had intended to simply listen in, but when no one offered up their code for review, I volunteered. A lot of my Python code is proprietary, so I decided to offer up a file from pyshards for review, since it was public. There was a LOT of feedback.

At this point you may be wondering, what do they mean by Pythonic? I generally understood it to mean that you should follow the Zen of Python coding principals and stick to the "pythonic" coding style.

In case you missed it, the Zen of Python is always near and dear if you are working in a Python interpreter.


me@mrroboto:~$ python
Python 2.5.2 (r252:60911, Jul 31 2008, 17:28:52) 
[GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import this
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

So that takes care of the Zen, but what is the Pythonic coding style? There are lots of opinions, but in general it is whatever the core developers and expert users say it is, and that evolves over time even as the fabric of the community evolves.

Jump to the present. This morning I'm finally sitting down to take a good look at the patch that Jack Diederich submitted as well as the notes I took while discussing the code with Moshe Zadka. And as I bring the project back up for testing, I remember that the setup and configuration steps were very incomplete. Okay Devin, quit blogging and get to work on it.

Startup Lessons #2 - It takes a customer

2009-04-24T20:46:00.000-07:00

There are startups and then there are startups. We were never going to be a "web-based" company. We had decided to use web-centric technology to address specific enterprise problems, so ultimately we would be an enterprise software company. The problem we set out to solve was summed up by one angel like this, "Everybody knows and understands that data behind the firewall is a mess." We hoped to fix that.

My partners worked on a number of large telco projects that ultimately failed, and for a variety of reasons. We understand what the corporation is left with after the legion of offshore programmers and consultants leave. We've watched the systems go live only to then crush under the weight of massive data. We got it that no matter how good the user interfaces, reports and controls seemed to be, what gets delivered somehow doesn't quite provide the value imagined months or years before getting into the project. There is often disappointment.

We came up with a concept would make data easy to access and integrate and that would be massively scalable. We decided to jump. We were going to launch a startup.

But this isn't your internet startup. There are a whole different set of problems when launching a software company that intends to sell to Fortune 1000 companies. With a web based company, you build it, and when you are ready to go, you make the site public. You are deployed. Typically the web startup's next problem is that no one shows for the party. You've got a whole different set of problems.

But for enterprise software, you have to build it, test it and then deploy it. That takes time---time to not only build the software, but to decide on a target market, to build customer relationships, to persuade someone at the company to let you in the door. It takes time to see through the initial deployments, to make it work in the real word, to seal a deal or two.

All of this comes out as you work through the business plan; you realize that it is going to take big money, bigger than you originally imagined, to make it through to those first customers. It's going to take VC money. It seemed to us, at the time, to be the the only way forward.

But there is a chicken and egg problem: The VCs won't give you the time of day until you have those first customers. Don't kid yourself into believing anything else. Seriously. This is not the late nineties and no one is lining up to throw money at every clever software idea. I swear.

Our team has a combined 50 years of experience in the telco space. We have a salesman with a proven track record, a versatile product manager, a data integration specialist with a knack for people, and a software architect. We knew we could build it and had good long-standing relationships in our space. The combination of our ideas, our relationships, and our technology gave us confidence that we would be able to get our product deployed and eventually sold. The VCs and angels liked hearing this, but they asked, what about the customer?

We know about big problems at Verizon and ATT, we told them. Are you in talks with them? Well, not them specifically, but we have talked to a number of other interested parties. You see, that's why we need your money. We need it to see us through until the software is mature enough and while we work up the relationship hierarchy looking for the best champion within these companies.

That's when they tell us to come back once we're deployed at a customer site.

Using our combined efforts we were able to raise some initial money. It wasn't much compared to what it would take to keep our team working full-time for a year (I'll get to this in another post), but it was a start and it gave us hope. We were able to take advantage of a government program to secure matching seed funds. But additional funding did not come as quickly as hoped and our rope grew shorter and shorter.

I built the software while my partners worked every possible existing relationship. The feedback was always positive. Yes, they really liked our technology. Yes, they wanted to work with us. Yes. Yes. Yes. By February we had developed a small sales pipeline, but we were out of money. With no customer, there could be no investment. With no investment, where does this leave the company?

I have omitted the fact, thus far, that the economy tanked during this our first 10 months. I suspect that had it not, we would have secured funding from one or more of the angels we were courting. There is of course no way to really know that.

At this point my partners are looking for services work. They will eventually find themselves engaged on location with one of our company's target customers. At that time, they will look for problems that our software, or new software, can solve. It may be that they'll learn of new problems that require a completely different solution, but once they find it, they'll be in a great place to recommend a software solution. They will have a technology foundation---that they own---that can be adapted to solve a number or data classification, access, performance and integration problems. They will present their ideas and have a much easier time of getting it deployed than they would if they were coming in from the outside.

This solves a number of problems. They will be able to articulate clearly the specific problem to the next investors, which was a bit of a problem during the first go-round. (Clearly they won't be hired by a client unless there is a real problem to solve in the first place!) They will be getting paid for their services work, eliminating the need to use investment to pay their salaries. And most importantly, they will be engaged with a real-world CUSTOMER.

Hindsight really is 20/20. We probably should have gone this route on month one instead of month ten.

Startup Lessons #1 - The Valley of Death

2009-04-15T08:37:00.000-07:00

If nothing else I've gained considerable insight into the startup process during these last 10 months of chasing the dream. The lessons are abundant.

You've heard it said that it's all about the journey, not the destination. Well, not when you are a startup. The destination is everything.

But there is much to be said about the journey as well. If you are an entrepreneur and thinking of going off on your own, you won't find a lot of war stories. I imagine that those who succeed guard their formula for success and those who don't prefer to pretend that nothing happened. The way I see it we've neither failed nor succeeded at this point, so why not reflect on a few of the lessons learned?

Personally,the journey has been both rewarding and costly. There were a lot of great experiences. I met fellow entrepreneurs. I learned about what it takes to look someone in the eye and ask them for money. I was able to practice the art of giving presentations to Fortune 1000 CTOs and VPs, and of course, I was able to build a substantial---though incomplete---enterprise software system using some of my favorite technologies. And there is so much more--- consensus building when working with partners, learning to recognize when you are self-deluding, finding the energy to take action on the days when it seems there is no hope.

I probably wouldn't be blogging about this today if everything with the business had gone as planned. Friends and former co-workers keep asking me, "Is the company dead?" Well, no. But it is true that we're not where we had hoped to be at this point, that's for sure.

We recently held a second meeting with an Oklahoma-based VC and he said it best,"You're in the valley of death," a chasm between the initial seed funding and a substantial VC round. It's a place where companies who have burned through their initial funding but have yet to make it to revenue reside. Word to the wise: A substantial revenue generating customer relationship is paramount; this must happen before any serious investment consideration will be given to you by a venture fund.

If you go back and read our business plan, at this moment in time we should have a development staff, a beta release, and we should be engaged with at least two paid pilots. But that didn't happen and for a lot of reasons. So what went wrong and is there any way to turn things around now? I'll reflect and blog on it over the next several weeks.

A List

2009-04-15T06:34:00.000-07:00

So many things are going on right now. Where to begin? Here is a list of things that I either need or want to do in no particular order.

Blog about the startup experience
Find gainful employment
Find the time to contribute to Catherine Devlin's sqlpython project
Find the time to test the diff that Jack Diederich contributed to the pyshards project, and then refactor everything else
Take my tax filing to the post office
Evaluate plone, drupal, wordpress and...
Revamp my home page (a simple resume currently)
Figure out what I want to be when I grow up
Find the time to finish recording a new electronica song I started writing a few weeks ago
Keep working on my contract job---the only item in here that pays---but without getting so buried in it that I can do nothing else
Watch out for typos or misspellings on my blog!

Brave?

2009-03-29T17:51:00.000-07:00

I did what I suspect was a fairly brave thing at PyCon. I attended an Open Spaced entitled "Is This Pythonic?" No one else was willing to put their code in front of the Python core developers (Jack Diederich and Moshe Zadka), so I offered up on of the files from my PyShards project. Now, keep in mind that I only decided that Python would be my language of Primacy about 6 months before writing the file that they reviewed. I tend to use the common subset of programming language features so that I can hop back and forth between programming languages without making my brain explode. Nonetheless, I wanted to see what it would take to make my code acceptable to the most pedantic of Python programmers, so I volunteered.

Check out the result:

http://jackdied.blogspot.com/2009/03/is-this-pythonic.html

Jack could not pythonicify the code to his satisfaction without relying on newer and trunk Python features. At the time of his writing, he was still not satisfied with his result.

It was an interesting, if not a bit nerve racking, exercise.

Django templating

2009-03-22T06:07:00.001-07:00

One of my contacts, John, wrote me yesterday, asking for some conceptual help with the Django templating system. I broke down template inheritance for him, using the following very simple example.

I told John...

I typically use Django's template inheritance in this way. A base page handles most of the boiler plate stuff and then I specialize in the inherited pages. URLs and the views (python functions) also get involved.

Here's a super simple page. (Imagine that each code snippet lives in its own file.) This one is base.html.

<html>
<body>
Part One
Part Two
</body>
</html>

I need two more things to display a page using the Django system, the URI routing expression in urls.py and a view. (I'm sure you get this, but to be clear I just want to call out all of the parts.)

The URI expression:

(r'^base$', base_function),

base_function is the view function, typically defined in another file. Something like this will display the page without really doing anything fancy with data.



def base_function(request):
    return render_to_response("base.html")

From a browser, you go to /base/, and you see...
Part One Part Two

Now you want to either extend or replace a section of the template. First, let's extend. We modify the original template and add markers for the areas we will be extending or replacing.

<html>
<body>
{% block one %} Part One{% endblock %}
{% block two %} Part Two{% endblock %}
</body>
</html>

Suppose you just want to just change the second block. So you create a file called mod2.html.

{% extends "base.html" %}
{% block two %} A different thing to render in Part Two{% endblock %}

You add another URL expression in urls.py. So now it looks like...

(r'^base$', base_function),
(r'^mod2$', mod_alt_two_function),

and you add a second view function...



def mod_alt_two_function(request):
    return render_to_response("mod2.html")

From a browser, you go to /mod2/, and you see...
Part One A different thing to render in Part Two

A third view might replace the first and second section....

(r'^base$', base_function),
(r'^mod2$', mod_alt_two_function),
(r'^all$', all_mod_function),



def all_mod_function(request):
    return render_to_response("all.html")

{% extends "base.html" %}
{% block one %} Much different than
{% endblock %}
{% block two %} the original{% endblock %}

Producing this when you hit /all/ in the browser:

Much different than
the original

So that's the basics with template inheritance. You can also just include chunks of other templates by including them right in the middle of the page. And you can wrap the includes in logic, so that based upon the data you are passing to the template from the view function, you can make decisions about what to include. Here is a small snippet from one of my pages that includes one of four different HTML template 'chunks' based upon the values of the variables "result.format" and "result.render_hint".



                    {% ifequal result.format 'pipe' %}
                        {% ifequal result.render_hint 'email' %}
                            {% include 'email_result.html' %}
                        {% endifequal %}
                        {% ifnotequal result.render_hint 'email' %}
                            {% include 'table_properties.html' %}
                        {% endifnotequal %}
                    {% endifequal %}
                    {% ifequal result.format 'text' %}
                        {% include 'text_result.html' %}
                    {% endifequal %}                    
                    {% ifequal result.format 'tag' %}
                        {% include 'tag.html' %}
                    {% endifequal %}

Android Pt. 3: Photo uploading

2009-01-17T13:50:00.000-08:00

I found enough bits and pieces of code to hobble together a photo uploader. This blog post was helpful:


http://itp.nyu.edu/~dbo3/blog/?p=122

Speaking of helpful, once running an Android app with the emulator, you might wonder where Log statements are going. Standard out doesn't show them in the Eclipse editor, but you can tail them with this command:


adb logcat *:W

Where do I want my uploaded photos to go? I have been thinking about writing a small appengine app which could be used for coordinating, managing, and searching for the photographs. Maybe I'll use flickr for photo storage, at least initially.

Next up I need to find the code to upload the photos to flicker. Knowing that I will need an API key, I requested and received one from Amazon. I had heard that that flickr uses a hybrid rest api, and sure enough, after searching around I found no pure REST interface for authenticating and uploading.

I've decided to test the basic uploading steps in Python first. Once I get it working, I'll go back and write the more verbose version using JAVA.

I found that the first step in uploading is going through an authentication handshaking sequence.
First up, grab a frob from flickr.


import md5
import xmlrpclib

uri = "http://api.flickr.com/services/xmlrpc/"
key = YOUR_FLICKR_API_KEY
secret = YOUR_FLICKR_SECRET
proxy = xmlrpclib.ServerProxy( uri )
secretstr = secret + "api_key" + key 
hexsig = md5.new( sigstr ).hexdigest()
resturi = "http://api.flickr.com/services/rest/?method=flickr.auth.getFrob&api_key="+key+"&api_sig="+sig
result = proxy.flickr.auth.getFrob( {'api_key': key, 'api_sig': sig} )

print result

At this point, I must take a break and rejoin the human race. More on this next time.

Android Pt. 2: Crisis averted

2008-12-29T21:49:00.000-08:00

I made another attempt at deploying the test application, this time using the command line tool.


me@mrroboto:~/src/android-sdk-linux_x86-1.0_r2/tools$ ./adb -d install ~/picthere.apk
203 KB/s (12207 bytes in 0.058s)
   pkg: /data/local/tmp/picthere.apk
Success

I'm not sure why I was getting the permissions problem when deploying from inside of Eclipse.

But in any case, the application has been deployed to my G1 and works as expected.

Android Pt. 1: Developing and deploying first steps

2008-12-29T17:57:00.000-08:00

To get things rolling I took the Hello World app that I had initially created using the Eclipse plugin and modified it to show a couple of buttons that pop up dialogs. Most of the code was taken and adapted from the alert dialog example programs.

Everything worked out when running the program under the emulator. My next step was to see what it would take to deploy to the G1.

When I opened up the manifest.xml file in Eclipse I noticed a link that read, "Export the unsigned apk". I clicked the link but didn't see output in the Eclipse editor, nothing to suggest that anything had happened. But, when I checked my home directory, I did find a new .apk file.

Note: If you right-click your project in the Eclipse package view you will find a menu item under Android tools that performs the same export function.

According to Google's documentation, my next step was to sign the apk. Since I'm only testing a deployment to my personal phone, I was hoping that this would be fairly painless: I didn't want to mess around with the details of key generation right then.

I read that there is a debug.keystore that Eclipse uses when automatically signing debug builds for deployment to the emulator. I hoped I would be able to skip the keytool step because of this and use the same key generated by Eclipse. Skipping to the jar signing step, I ran the following.

jarsigner -verbose -keystore ~/.android/debug.keystore picthere.apk  PictHere

When prompted for a password, I used the android default which is "android" according to this page.


me@mrroboto:~$ jarsigner -verbose -keystore ~/.android/debug.keystore picthere.apk  PictHere
Enter Passphrase for keystore: android
jarsigner: Certificate chain not found for: PictHere.  PictHere must reference a valid KeyStore key entry containing a private key and corresponding public key certificate chain.

Whoops. Wrong alias for the last argument. After correcting, all worked as expected.


me@mrroboto:~$ jarsigner -verbose -keystore ~/.android/debug.keystore picthere.apk  androiddebugkey
Enter Passphrase for keystore: android
  adding: META-INF/MANIFEST.MF
  adding: META-INF/ANDROIDD.SF
  adding: META-INF/ANDROIDD.RSA
 signing: res/drawable/alert_dialog_icon.png
 signing: res/drawable/icon.png
 signing: res/layout/main.xml
 signing: AndroidManifest.xml
 signing: resources.arsc
 signing: classes.dex

I verified that the apk (actually a jar file) had been signed as suggested by Google.

I had a bit of a hiccup during the next step: using DDMS to deploy the app to the actual device.

There is a DDMS perspective in Eclipse that is really very, very nice. Unfortunately, after plugging the G1 into my computer via USB, the emulator showed up and not my actual device. A number of posts suggested that the "Unknown Sources" checkbox must be enabled on the G1 (found under Settings/Applications). After checking the box, I plugged the G1 back in and restarted Eclipse, but no device listed---still just the emulator.

I decided to try it from the command line:


me@mrroboto:~/src/android-sdk-linux_x86-1.0_r2/tools$ ./adb devices
List of devices attached
emulator-5554    device

Nope...nothing there but the emulator.

This page suggested that I add udev rules. Made sense...need to give the USB device the right permissions. (I probably should have mentioned that I'm developing on an Ubuntu box.)


If you're developing on Ubuntu Linux, you need to add a rules file:Login as root and create this file: /etc/udev/rules.d/50-android.rules. For Gusty/Hardy, edit the file to read:
SUBSYSTEM=="usb", SYSFS{idVendor}=="0bb4", MODE="0666"
For Dapper, edit the file to read:
SUBSYSTEM=="usb_device", SYSFS{idVendor}=="0bb4", MODE="0666"
Now execute:
chmod a+rx /etc/udev/rules.d/50-android.rules

After unplugging and re-plugging the G1, it showed up for the party...


me@mrroboto:~/src/android-sdk-linux_x86-1.0_r2/tools$ ./adb devices
List of devices attached
emulator-5554    device
HT849GZ16642    device

At this point I was confident that I would be able to see the device in the DDMS perspective in Eclipse. Sure enough, it was there.

But sadly, when I chose the "push file to device" and selected my apk, I got the following error:


[2008-12-29 21:35:22 - ddms]transfer error: Read-only file system

Could it be that I am not going to be able to copy my app directly to the phone? Apparently this is the reason I should have picked up the Android Dev Phone, at least according to this post, which states that the Dev Phone is the same as the G1 but "there are no software restrictions on the device, users can directly load their own applications and test code straight to the unit rather than just relying on emulators to test code."

We shall see...

Developing for the G1

2008-12-29T12:05:00.000-08:00

I received a G1 for Christmas and can't resist the urge to write my first Android application. I'll share my experience here.

So far I've downloaded the SDK, installed the Eclipse plugin, wrote my first Hello World application, and examined many of the sample applications that come with the SDK. The framework is quite simple and elegant.

My next step: try out the deployment process. My simple program runs fine under the Emulator, but how difficult will it be to package and deploy it to my phone?

Crap. Gmail IMAP doesn't support searching by header Message-ID

2008-11-09T20:03:00.000-08:00

I just spent the last hour trying to figure out why my IMAP query is failing. I'm testing an IMAP integration with gmail. Here is my query:

>>> type, data = mail.search(None, '(HEADER Message-ID "<799d9ef60811090712j14c90d46hf9939af135dc5a19@mail.gmail.com>")')
>>> data
['']

I should have mentioned that I'm using python IMAPLIB.

Even though the message-id is valid (there is a message with this ID), nothing is returned. Why? It's just not supported. From google:

http://mail.google.com/support/bin/answer.py?answer=78761&topic=12762&disablechatbrowsercheck=1

Does Gmail support all IMAP features?

Gmail IMAP is a fairly complete implementation of IMAP, but the following features are currently unsupported:

\Answered and \Recent flags on messages.
Substring search. All searches are assumed to be words.
Searching arbitrary headers. Only some headers are available for searches: From/CC/BCC/To/Subject.
There is no SIEVE interface to Gmail filters.
Only plain-text LOGIN over SSL tunneled connections are supported.

Yep, that's pretty much the feature I needed. Poop!

The other Devin Venable

2008-08-17T03:36:00.000-07:00

The Internet is a lot like your brain in that it remembers a great deal of your past. That goofy question you asked another developer back in 1998 via a mailing list---a copy is still floating around on the net. Or what about that videographer friend who asks for your help on a project? Do you really want the world to see you through the prism of a music video on youtube? How about the online petition you signed when you were all worked up over a political issue?

Unlike your brain's recollection of that great moment twenty years ago, degraded and made fuzzy over time, the internet can quote your words verbatim. Like your brain, some memories will be lost---or buried so deep that they will never resurface.

The Internet is different in that it may have false memories about you. As a brain, it has more in common with the Star Trek Borg than one belonging to a human. This is the collective memory of millions. Fortunately, millions are not interested in you personally---unless you are Paris Hilton. But what about those who do write about you? And if something is attributed to you, is it the right you?

For the first several years on the Internet, I was the only Devin Venable around---as far as I could tell. Just as I had convinced myself that my name was one of a kind, a high school kid popped up, posting things that were sensible for a kid his age.

I'm sure as he posted that the last thing on his mind was keeping up my professional image on the net.

If your name is John Smith or David Lee, you don't have to worry about this. You are granted a certain anonymity because so many share your...John Doe.

So this other Devin is still around. He is in his twenties now. How do I know this? Because he talks about himself on sites like bebo and Facebook. I know a lot about him. I know that he likes to wear New Era baseball hats, listen to rap music, and---not that I wanted to know---that he wears boxers instead of briefs.

Then again, I really don't know him at all. I can't even be sure that he is one other person. I don't know anything other than the public details that he (or they or someone) is projecting on the net. Will he/she/it/they be pleased with the picture they have painted of Devin Venable in ten years?

This gets me to thinking about metadata. Devin Venable is a very distinctive attribute (or tag). But it's not a particularly useful attribute for grouping. Here are some seemingly common attributes of the members of this group (based upon what I know of the Devins on the Internet): [human, male, American, white]. This isn't very specific and, as the name Devin is sometimes given to females, non-whites, and non-Americans, the only truley unique attribute is that we're all human.

Name your cat Devin Venable and that goes out the window as well.

Jacob Kaplan-Moss In Tulsa

2008-06-11T06:16:00.000-07:00

Last night Jacob Kaplan-Moss came to speak at the Tulsa Python User's Group meeting for June. He is one of Django's founders and one of the pair of Django's Benevolent Dictators for Life, the other being Adrian Holovaty.

Jacob gave a little background on the Django project and talked about the upcoming 1.0 release. He gave a date for the release, a news flash that he had yet to announced on the user group lists. I think he said September, but you probably ought to check the mailing lists to get the real dope.

Most interesting comments:

He called Kent Beck's unit test approach either "crap" or "bullshit"---I can't recall the exact term, saying that his problem with it was sitting down to design tests before writing functions, and testing for inputs that you will never realistically be pushing into the functions. (Hopefully I paraphrased him correctly here.) The gist of his point was the suggestion that the programmer should be trusted to do the right thing.

Jacob also told us that he was a surprised as anyone when Google announced that AppEngine was Django based (or inspired). Google had approached him several months before to discuss "something" but the conversation didn't happen because Jacob wouldn't sign an NDA.

Of course most of the conversation was about new features, but you can go out and find that information yourself.

After the meeting a few of us accompanied Jacob to McNellie's, a local pub, for a few beers.
My impression was that he is a reasonable, down-to-earth kind of guy.

JAVA today

2008-05-15T05:08:00.000-07:00

(I came up with this today while reviewing source.)

JAVA today: take a 40 line function, explode it into 80 objects, and call it an improvement.

__

JAVA is not a bad language. But the bloat and cruft is driving me crazy. If only I could do every project in Python.

Mobile Phone Site Detection

2007-12-26T20:28:00.000-08:00

I got a Palm Centro mobile device for Christmas and it got me to thinking about the number of potential employers/clients that might use their mobile phones to view my resume or company site online. I've been quite happy with Google's mobile-versioned sites, like www.google.com/calendar/m and decided that I'd better follow the leader.

It wasn't as easy as you might think to find a good mobile-phone detection script online, at least not one that worked for the Centro. After four failed attempts, I found this one on brainhandles.com.




function checkmobile()
{
        if(isset($_SERVER["HTTP_X_WAP_PROFILE"])) 
             return true;

        if(preg_match("/iphone/i",$_SERVER["HTTP_USER_AGENT"])) 
             return false;

        if(preg_match("/wap\.|\.wap/i",$_SERVER["HTTP_ACCEPT"])) 
             return true;

        if(isset($_SERVER["HTTP_USER_AGENT"]))
        {

             $uamatches = array("midp", "j2me", "avant", "docomo", "novarra", "palmos", "palmsource", "240x320", "opwv", "chtml", "pda", "windows\ ce", "mmp\/", "blackberry", "mib\/", "symbian", "wireless", "nokia", "hand", "mobi", "phone", "cdm", "up\.b", "audio", "SIE\-", "SEC\-", "samsung", "HTC", "mot\-", "mitsu", "sagem", "sony", "alcatel", "lg", "eric", "vx", "NEC", "philips", "mmm", "xx", "panasonic", "sharp", "wap", "sch", "rover", "pocket", "benq", "java", "pt", "pg", "vox", "amoi", "bird", "compal", "kg", "voda", "sany", "kdd", "dbt", "sendo", "sgh", "gradi", "jb", "\d\d\di", "moto");

        foreach($uamatches as $uastring)
        {
            if(preg_match("/".$uastring."/i",$_SERVER["HTTP_USER_AGENT"])) 
                 return true;
        }
    }
    return false;
}
#example
if(checkmobile())
{
  header('Location: http://www.yoursite.com/m/index.html');
  exit;
}

if(!checkmobile())
{
  header('Location: http://www.yoursite.com/');
  exit;
}
?>

It's a PHP script, of course. For my purposes, I just created an index.php file and set "DirectoryIndex index.php index.html". The first thing the visitor hits is the php file, which either redirects to the main index.html file or the mobile version in the "m" directory.

Open Source Project Management

2007-11-06T05:19:00.000-08:00

Do's and Don't

Use Open Source Judiciously

We all know the benefits of using open source, but what are the pitfalls? If you have been asked to build a professional, production-ready, mission-critical, customer-facing application, you probably don't want to go bleeding edge.

You need to ask yourself if the project is ready for prime-time. Don't be fooled by a glossy, over-hyped project page. Dig deep. Find out
how many people are really using the project. Look at the mailing list archives. Is it active enough? Is is full of unanswered questions? Check out the source from subversion. Review the check-in history. If time permits, read the code.

For short-timeframe, mission-critical projects, I use what I call The Book Test. Any project for which an O'Reilly book has been written has been vetted and tested to my satisfaction. If three or more books on the subject have been written and are available from your local bookstore, you can bet that the technology is ubiquitous. Tried and true open source projects include Linux, Apache web server, Python, PHP, and Ruby. This litmus test will eliminate a majority of the projects you find on source forge, so this shouldn't be your only test. But it's a good place to start.

Wireless Problems Upgrading Ubuntu to Gutsy

2007-11-03T09:03:00.000-07:00

I ran into trouble this week while installing Ubuntu Gutsy on some older hardware for my five year old son Kadin. He'd been using a prism2 based wireless card with success under older releases of Ubuntu for over a year, so I knew that the card was supported. But after completing the fresh install, network manager couldn't find my home network.

Matt Hartley suggested that the problem was with Network Manager in this post, but I followed his recipe, winstalling wicd as a replacement, and it didn't help my situation. Wicd had the same problems detecting my home network.

ifconfig reported two interfaces for my wireless card---wifi0 and wlan0. Some suggested blacklisting hostap_pci (and others) to remove conflicting drivers which might be confusing network manager. The best suggestion I found early on---which didn't work for me at the time---was to install the linux-wlan-ng package. This was the right thing to do (the needed prism2 driver is installed), but unfortunately, this didn't work either. Perhaps some additional configuration was needed?

Since, according to some, the real problem was with Network Manager, and since it was working well for us under Feisty, I decided to pin back those packages using apt.

I added the following to /etc/apt/preferences:

Package: *
Pin: release a=gutsy
Pin-Priority: 700

Package: network-manager
Pin: release a=feisty
Pin-Priority: 800

Package: network-manager-gnome
Pin: release a=feisty
Pin-Priority: 800

Then I duped all of the repository entries in /etc/apt/sources.list and changed each "gutsy" to "feisty" so that apt can find modules in both repositories. Then I ran:

sudo apt-get install network-manager network-manager-gnome

This successfully downgraded and pinned the two Network Manager packages to the versions that are available in Feisty Fawn.

I was really pleased that the downgrade went so smoothly, as I've gotten trapped in dependency hell attempting similar feats in the past. Sadly, though, my problem remained! Perhaps Network Manager was not at fault after all.

linux-wlan-ng seemed to be the right solution for my card, so I decided to build from source from the trunk.

svn co svn://svn.shaftnet.org/linux-wlan-ng/trunk

Then I built and installed it and followed the directions in the README to setup, which included setting values in /etc/wlan/wlan.conf. I added my home network's SSID and changed the following entry:

# for some reason, this value was set to N --- seemed wrong to me
WLAN_SCAN=y

I also added the suggested values to /etc/rc.local (for modprobing the correct driver---prism2_pci in my case).

I remembered seeing wlan0 aliases under /etc/modprobe.d, so I grepped for matches and found this file:
/etc/modprobe.d/linux-wlan-ng

The contents are shown below. The "alias wlan0 prism2_pci" line was commented so I removed the comment.

# Aliases to tell insmod/modprobe which module to use when bringing up the
# wlan0 interface.

# Uncomment the line corresponding to the type of prism2 device you have.
alias wlan0 prism2_pci
#alias wlan0 prism2_usb
#alias wlan0 prism2_cs
#alias wlan0 prism2_plx

# this allows network manager to manipulate the device
options p80211 wlan_wext_write=1

I rebooted and network manager was able to find my home network, as well as a number of my neighbors wireless network. In hindsight, I wonder if, after installing linux-wlan-ng as a package, I had just removed the comment from this file and WLAN_SCAN=Y, my troubles would have washed away? If you are having similar problems, I would try that first before compiling the latest driver from source.

2007-11-01T12:39:00.000-07:00

I posted a question to the Wicket users group a few days ago on how to best deal with frames, in particular how to pass arguments between a tree view in one frame and a list view in another. I didn't get any takers.

I finally determined that there was no good way other than to use plain-old javascript to push values from the tree up to the parent frame, where they are dispersed to the other frames.

I wanted to use LinkTree, but really didn't need most of the features other than the look and feel. I just needed to be able to insert an onclick link that would pass a value to my parent frame.

After a lot of digging I came up with a solution that worked, but seems verbose for the job I'm attempting. My question for the reader is this: Can you propose a more concise solution?


public class CategoryTree extends LinkTree
{
  @Override
  protected Component newNodeComponent(String id, IModel model)
  {
      return new LinkIconPanel(id, model, CategoryTree.this)
      {
          private static final long serialVersionUID = 1L;

          protected void onNodeLinkClicked(TreeNode node, BaseTree tree, AjaxRequestTarget target)
          {
              super.onNodeLinkClicked(node, tree, target);
              CategoryTree.this.onNodeLinkClicked(node, tree, target);
          }
       
          protected Component newContentComponent(String componentId, BaseTree tree, IModel model)
          {
              Label l = new Label(componentId, model)
              {
                  private static final long serialVersionUID = 1L;
               
                  @Override
                  protected void onComponentTag(ComponentTag tag)
                  {
                      super.onComponentTag(tag);
                      tag.put("onclick", "parent.notifyViews(this.getAttribute('catalogid'))");
                   
                  }
              };
       
              DefaultMutableTreeNode n = (DefaultMutableTreeNode)model.getObject();
           
              if (n.getUserObject() instanceof EcCategoryTreeNode)
              {
                  EcCategoryTreeNode ec = (EcCategoryTreeNode)n.getUserObject();
                  l.add(new SimpleAttributeModifier("catalogid", "" + ec.getId()));
              }
           
              return l;
          };
      };
  }

  public CategoryTree(String s, TreeModel tm)
  {
      super(s,tm);
      this.setLinkType(LinkType.REGULAR);
  }

}

Wicket and HTML CSS attributes

2007-10-24T15:17:00.000-07:00

JAVA open source web frameworks are everywhere. And the documentation is everywhere, which is not always a good thing. Especially when what you want is to go to a particular somewhere to find all of the documentation. Mailing lists are great, but you may end up the first to post on your problem. As long as you don't need to get it done---you have no problem.

I didn't find a post or documentation that addressed the problem I was facing today.

In this case, the task I had to complete was simple: render an HTML table, alternating the background color for each row. I've implemented this pattern a half dozen times before using Java Servlets, ASP, PHP, OpenLaszlo and even a Struts-like home-grown system when working for a previous employer. In the end, it has always come down to plain old HTML and CSS, programatically assigning one of two CSS classes to a table row. (OpenLaszlo was an exception.)

This time around I'm using Wicket. I must admit that I'm not to crazy about heavy, rule-wielding web frameworks because they often improve on the way you accomplish big tasks, but make it exponentially more difficult to do the simple ones that don't fit nicely into their design. Is this true of Wicket? It's too early for me to say, but I'm certainly interested in finding the answer to that question.

The early verdict: So far, Wicket seems to do a pretty decent job doing what a framework ought to do.

Wicket keeps the presentation concerns separate from the model. In practical term this means that the web stuff (javascript, CSS, HTML) is kept independent from the JAVA source. No mixing-and-matching within a single file. Wicket does a lot more, but I'll leave it to others to sell you on it.

For a non-Wicket web app, the type of logic required to provide alternating rows is normally applied in a for loop, using mod to determine if you are dealing with an even or odd row. In a scripting language, the whole operation takes all of two lines of code.

So I set out to do the same for Wicket. Fifteen minutes of googling and I found How to modify an attribute on a HTML tag. Sounded promising, but upon reading, looked verbose. (Must we really use a separate model and view to create a simple attribute pair?)

Reading this page, it wasn't apparent how the example would lend itself to my case. However, with some experimentation, I was able to retrofit the familiar pattern onto the Wicket frame.

<tr wicket:id="catalog" class="unknown">
<td wicket:id="name"></td>
<td wicket:id="sku"></td>
<td wicket:id="manufacturer"></td>
...
</tr>


  ListView listview1 = new ListView("catalog",    catalogItems.getProductWindow())
  {
          public final String style1 = "row_light";
          public final String style2 = "row_dark";
       
          protected void populateItem(ListItem item)
          {
              Map rec = (Map) item.getModelObject();
           
              item.add(new Label("name", (String)rec.get("cttm_display")));
              item.add(new Label("sku", (String)rec.get("cttm_partnumber")));
              item.add(new Label("manufacturer", (String)rec.get("mfg_name")));
              ...
           
              if (item.getIndex() % 2 == 0)
                  item.add(new AttributeModifier("class", new PropertyModel(this, "style1")));
              else
                  item.add(new AttributeModifier("class", new PropertyModel(this, "style2")));

          }
   }

I used PropertyModel to get around having to create beans to hold the two strings representing the CSS classes. Does the reader see a better solution?

Indexing and Ranking

2007-05-05T07:34:00.000-07:00

It is not easy to get a high Google ranking.

A few weeks back I revived my auction website, www.fleahaus.com. I say revived because I originally developed the site a few years back. It was born, it floundered, it died a slow and agonizing death.

So I got a case of the what-if-I-had-done-it-a-little-differentlies and decided to administer CPR. For example, I initially designed the site show items by region, a feat accomplished by using subdomains. Criagslist uses this very concept currently---based on the geographical data you submit, you see a city based portal. For example---when I visit craigslist, I am directed to http://tulsa.craigslist.com because I live in the Tulsa area. My original site used the same pattern.

This time around I threw out the city based portal idea and introduced a local search button. This ultimately accomplishes the same goal---giving users a way to find items that are geographically near to them---something very important to those who want to sell items too large or heavy to ship by mail. This gave me a change to try out Google's GeoLocate APIs.

After the face-lift and new features were done, it took three weeks before Google indexed my site. This is apparently not too bad.

The delay in indexing during that three weeks was a major concern for me--- especially because I had so much trouble getting the original site indexed. I partially attribute the failure of the original site (www.communitybuy.com for the record) to my inability to get Google to index the darn thing.

I'm still not quite sure why communitybuy.com failed to make it into the index. Perhaps because I didn't put effort into back-linking at that time? Formatting issues? Only the crawler knows for sure.

Google's webmaster tools have been beneficial this go-round. I have been able to check crawl rates, get indexing status, and more.

So one milestone met: www.fleahaus.com is indexed. Next, find a way to move up the rankings.

Tunneling VNC over SSH

2005-12-15T07:10:00.000-08:00

Connecting a Windows Cygwin client to a Linux SSH/VNC server across the public Internet

There are times when I need to connect to my computer at home from work, most often because I need to retrieve a file or program stored there. A common scenario is one where I can (sort of) remember the name of the file I'm after, but not its location on my home computer's hard drive. This computer is a Linux server and what I end up doing is opening a SSH shell into my home box, and then I use tools running on my home machine to locate the file I'm after. Sometimes I find myself changing directories and listing files like mad until it comes up, particularly if the slocate database is out of date. Once I find the file I'm after, I open a second Cygwin shell on the client side (at work that is), and issue SCP to retrieve it.

Sometimes I'm just not in the mood to work with shells. Recently, while in one such mood, I thought I might try perusing the files on my home machine using Gnome’s Nautilus file manager. After some fiddling around with environment variables, I did manage this via X over SSH, but the performance was dismal. I've heard people say that the X protocol is too chatty, and that you shouldn't expect good client/server performance over the Internet in most cases. I’d been advised that I would have better luck using VNC and so I decided to give it a try.

I needed to run the VNC server on the machine whose desktop I wished to view---my home computer---a scenerio that would seem to necessitate that I poke a hole in my home firewall to let the VNC traffic through. Nay, nay.

For reasons I hope are totally obvious, only a single incoming port opens into my home network, and that's SSH port 22. I lock this port down as tight as I can using configuration rules, of course. (Wow…even admitting that I have an open port feels like a security risk.) Instead of exposing the VNC server's listening port to the public Internet, I preferred tunneling VNC traffic over my SSH connection.

It took a few hours of research before I was able to pull this off. I’ll retrace my steps here for the benefit of others of similar need and configuration.

Client Requirements

As I began my research, I found a lot of information on how to accomplish VNC tunneling using the Putty SSH program for Windows on the client side. I personally use Cygwin tools, am quite happy with them, and didn’t see the sense in downloading yet another SSH client (thank you very much). And so I decided to use my existing Cygwin SSH client, one of the two tools you will need to have on your Windows client box if you intend to reproduce these steps for yourself, using my cookbook recipe.

The second of the two Windows programs you’ll need is a VNC client, and I suggest you use TightVNC. I think you can download an executable for the viewer only, but I just grabbed a download that included both the client and server. I’m sure I’ll use the server on my work box at some point down the road.

(By the way, if you google for “SSH VNC Tunneling” and find a lot written about setting an AllowLoopback value in a registry key for TightVNC---disregard it. It’s not needed for what we’re going to do.)

Server Requirements

You’ll need a running server that accepts SSH connections and that has VNC installed. Many modern Linux distributions install it by default, so you may already have it.

1. Open a SSH shell into your remote machine. If you’ve not used VNC before, you should set a password. You’ll provide this password when opening your VNC client later on.

[me@aldous ~]$ vncpasswd
Password: *****
Verify: *****

2. Start a vncserver. You should be able to just type ‘vncserver’ and use the defaults.

[me@aldous ~]$ vncserver

New 'aldous:1 (me)' desktop is aldous:1

Starting applications specified in /home/me/.vnc/xstartup
Log file is /home/me/.vnc/aldous:1.log

3. Use the tail command to peek at your log. We’re interested in which port VNC will be listening on.

[me@aldous ~]$ tail /home/me/.vnc/aldous:1.log

Xvnc version 4.0 - built Oct  6 2004 08:11:33
Underlying X server release 60801000, The X.Org Foundation


Thu Dec 15 09:17:00 2005
vncext:      VNC extension running!
vncext:      Listening for VNC connections on port 5901
vncext:      Listening for HTTP connections on port 5801
vncext:      created VNC server for screen 0

We’re interested in port 5901 for this example. That’s it for the server side of things.

4. Open a new Cygwin window on your Windows client and type something like the following, replacing USERNAME with the username you use to SSH into your remote machine, replacing myhomebox.net with the IP address or DNS lookup-able name of your home box. Also replace the trailing 5901 with the port reported to you in step three (from the Listening for VNC connections… line).

ssh -l USERNAME myhomebox.net -L 5900:127.0.0.1:5901

A little explanation here is warranted.

First, let’s see what the ssh man page says about the –L option.

-L [bind_address:]port:host:hostport

Specifies that the given port on the local (client) host is to be
forwarded to the given host and port on the remote side. This
works by allocating a socket to listen to port on the local side,
optionally bound to the specified bind_address. Whenever a con-
nection is made to this port, the connection is forwarded over
the secure channel, and a connection is made to host port
hostport from the remote machine. Port forwardings can also be
specified in the configuration file. IPv6 addresses can be spec-
ified with an alternative syntax:
[bind_address/]port/host/hostport or by enclosing the address in
square brackets. Only the superuser can forward privileged
ports. By default, the local port is bound in accordance with
the GatewayPorts setting. However, an explicit bind_address may
be used to bind the connection to a specific address. The
bind_address of ``localhost'' indicates that the listening port
be bound for local use only, while an empty address or `*' indi-
cates that the port should be available from all interfaces.

Let's look at the command once again.

ssh -l USERNAME myhomebox.net -L 5900:127.0.0.1:5901

The first number after the –L specifier, 5900, is the port your local client will be listening on that will be forwarded across the tunnel to your remote box. 5900 is the port that the TightVNC client tries to contact when establishing a VNC connection by default, so this is a pretty good number to use. The second number, 127.0.0.1, is the address of the remote machine from the perspective of the remote machine. Attempting to use the public IP address of your home machine, the one that you see from your client box, probably won’t work because of firewall rules, etc. I won’t get into an explanation here…just use 127.0.0.1. The third number, 5901, is the port your VNC server is listening on, on the remote machine.

Note that the above SSH command will not only create your tunnel, but it will also open a SSH command shell. To prevent this, add –T and –N to your list of command arguments.

ssh -l USERNAME myhomebox.net -L 5900:127.0.0.1:5901 -T -N

This will disable pseudo-tty allocation and prevent you from opening a shell. See man ssh for more information.

5. Start your TightVNC viewer (Fast compression). A dialog will pop up asking you to enter the IP address of a VNC Server. Enter 127.0.0.1. You will then be prompted for the password you created for your VNC server back in step one.

If all went well, you should now be looking at your remote machine’s desktop.

To avoid confusion about the use of the 127.0.0.1 IP address (we gave it to the TightVNC Viewer as the VNC server's address, and to the SSH command as the host address) , let me describe the tunnel from the perspective of the VNC client. For the TightVNC viewer you specified that the VNC server was running on 127.0.0.1. In other words, you specified your localhost...the box at your physical location (a.k.a. the client). The VNC client then looked for the default port that VNC servers run on (5900) and said (or would if it could speak), “Ah ha! There is indeed a VNC server running on this same box.” But of course you fooled it. Due to the nifty SSH command and the –L option, you created a tunnel that forwarded the traffic on the 5900 port on to, through SSH, your home (or remote) machine. The VNC client was never any the wiser.

Something similar happened on the remote side. The request to speak with the VNC server came out of the tunnel on the remote side and it asked (again, if machines could talk like us), “Any VNC servers around here that want to respond to a VNC client? I’m looking for one that is running here locally…it’s addressed to 127.0.0.1:5901.”

The point I’m getting around to is that both the client and the server side of VNC think that they are talking to services running on their own machines. SSH is acting as a proxy on both sides of the tunnel. Pretty cool, eh?

And if it didn’t work for you…

Look at your vncserver log on the remote machine. If an attempt was made and failed for any reason, you’ll read about it there.


[me@aldous ~]$ tail /home/me/.vnc/aldous:1.log

Also make sure your problem is not with making the connection to your remote box. You might check the secure log.


[me@aldous ~]$ tail /var/log/secure