Migrating schema from Oracle 11g Enterprise to 10g Express Edition

Here's the situation: you are working in an environment that uses Oracle Enterprise 11g, you want to do some development on your personal computer using your own database, and for whatever reason you cannot simply install a full-blown non-free version of Oracle on your laptop.

Your best free option is to install Oracle 10g XE, especially if you are running Ubuntu because installation is a snap.

http://www.oracle.com/technology/tech/linux/install/xe-on-kubuntu.html

Of course you should know that XE is stripped down, so read up and make sure that you won't be losing features that you'll absolutely need to run your production or test database.

The first step is to export your database. If you're like me an you attempt to simply export using the exp tool that ships with 11g, you will discover that there are incompatibilities between version which will prevent you from importing with 10g XE.

The easy solution? Just use the 10g version of the tools that ship with XE to dump the 11g database. In my environment, everything is Linux, so my quick-and-dirty approach was to create a temp directory on the 11g server and scp the 10g version of exp over to it.

cd $ORACLE_HOME/bin
scp exp me@11gserver:~/temp

Of course I don't have the needed 10g libs on the 11g server, so I just copy the all Oracle libs over to the same directory.

scp /usr/lib/oracle/xe/app/oracle/product/10.2.0/server/lib/* me@11gserver:~/temp

Now I ssh over to the 11g server and set up my environment to add the current directory (temp) to the LD_LIBRARY_PATH.

export LD_LIBRARY_PATH=.:/usr/lib32:/usr/local/lib32

Now to dump...

exp SYSTEM/stuff@11gserver:1521/yeah.stuff.com FILE=export.dmp OWNER=me_the_owner

Now copy the resulting export.dmp file back to your target computer, the one running 10g XE. (You know how to do it!)

Okay, now cross your fingers and attempt the import, hoping to succeed though you just know that critical 11g features won't be supported on XE and will probably cause all your effort to be for naught. But do it anyway.

In my case I now yell "Doh!" because I get the same damn error I had before. Did I mention the error? Yes, the one I got after importing initially, the export file created using 11g's exp. This is what I see on my development laptop.

imp SYSTEM/xx FILE=/home/me/export.dmp FULL=y

Import: Release 10.2.0.1.0 - Production on Fri Nov 6 08:32:09 2009

Copyright (c) 1982, 2005, Oracle.  All rights reserved.


Connected to: Oracle Database 10g Express Edition Release 10.2.0.1.0 - Production

IMP-00010: not a valid export file, header failed verification
IMP-00000: Import terminated unsuccessfully

I check the head...aha I see my error. I was still using the darn 11g exp because I failed to put the 10g exp on the path on the 11g server before I ran exp.

 head export.dmp 

EXPORT:V11.01.00
DSYSTEM
RUSERS

Oh, and in case you are thinking you might just break out a HEX editor and change the EXPORT version in the header to the correct version, it won't work. Yes I thought of that too.

Okay, back to the 11g server to re-run. This time notice the ./ before exp to pick up the correct version of the executable.

./exp SYSTEM/stuff@11gserver:1521/yeah.stuff.com FILE=export.dmp OWNER=me_the_owner

Once you "get it right" move the export back to your development box. In my case, I had to create matching tablespaces and users before the importing with success. One that was done, I ran the following...

imp SYSTEM/xx FILE=/home/me/export.dmp FULL=Y 
...
lots of output here, such as
 row rejected due to ORACLE error 12899
...
Import terminated successfully with warnings.

I believe my warnings are due to character set incompatibilities, and since I'm set to blow up this copy of the database in a development environment anyway, I'm not going to research them now.

I fired up SqlDeveloper, connected to my 10g XE instance, and selected some data. No problem! Who says you can't move an Oracle Enterprise 11g database to Oracle 10g XE?

Netbeans and Make

Netbeans for c++ uses the standard make tools. The top level Makefile is always the same for every project and is the correct place to add custom build steps. The nbproject subproject files are modified by the NetBeans editor, so it is best to avoid making changes there.

There are hooks in the top-level make where various custom steps can be inserted.

#
#  There exist several targets which are by default empty and which can be 
#  used for execution of your targets. These targets are usually executed 
#  before and after some main targets. They are: 
#
#     .build-pre:              called before 'build' target
#     .build-post:             called after 'build' target
#     .clean-pre:              called before 'clean' target
#     .clean-post:             called after 'clean' target
#     .clobber-pre:            called before 'clobber' target
#     .clobber-post:           called after 'clobber' target
#     .all-pre:                called before 'all' target
#     .all-post:               called after 'all' target
#     .help-pre:               called before 'help' target
#     .help-post:              called after 'help' target
#
#  Targets beginning with '.' are not intended to be called on their own

You can also add your own targets, just as if you were building a makefile from scratch.

Generating HTML color syntax highlighting from PRE tag

I'm working on a Django template tag that should transform code embedded within a PRE tag and convert it to nicely formatted HTML with color syntax. To test, I inserted this block of code here, the same code I wrote to do the conversion.

#!/usr/bin/python

from django import template
from django.template.defaultfilters import stringfilter
from django.utils.safestring import mark_safe
from BeautifulSoup import BeautifulSoup
from pygments.lexers import guess_lexer, guess_lexer_for_filename
from pygments import highlight
from pygments.lexers import get_lexer_by_name, TextLexer
from pygments.formatters import HtmlFormatter
import re

register = template.Library()

@stringfilter
def tocode(value):
    try:
        commentSoup = BeautifulSoup(value)
        c = commentSoup.findAll('pre')
        for all in c:
            brs = all.findAll('br')
            for item in brs:
                item.replaceWith('\n')
            joined = ''.join(all.findAll(text=True))

            if 'class' in all:
                lex = get_lexer_by_name(all['class'], stripall=True)
            else:
                try:
                    lex = guess_lexer(joined)
                except:
                    try:
                        lex = get_lexer_by_name('python', stripall=True)
                    except:
                        lex = TextLexer
            formatter = HtmlFormatter(linenos=True, cssclass="source")
            result = highlight(joined, lex, formatter)
            all.replaceWith(result)

        return mark_safe(commentSoup)

    except:
        return value

register.filter('tocode', tocode)

This is how it works: You pull a feed from, for example, blogger, and look for PRE tags, assuming that there is something interesting (like a code snippet) inside. After discovering that Pygments' guess_lexer has a hard time identifying most of the snippets I feed it, I decided to make it possible to explicitly specify the PRE content type. I do this by tagging the PRE element with a class name. In this case, I use the class name verbatim to call the get_lexer_by_name method. So this...

<pre class="python" >

...will look up the python lexer and

<pre class="php" >...

will look up the php lexer.

The PHP lexer is very disappointing actually.

Originally, I set the code to use the TextLexer in the event that PRE class attribute was not present, but this was boring. I found that the python lexer produces more appealing results for almost all snippets, so now I'm using it as the default when the PRE attribute is not specified. Of course, not all content will be python lexerable, so in the case of exception I fail over to the TextLexer. The exception handling chain is a bit ugly but it gets the job done.

Ignite Tulsa

Last night's Ignite Tulsa event was a blast. The beer was cold, the first one was free, and the talks were often amusing.

The best presentation had to be "If someone gives you roses you should be pissed off" by Matthew Galloway. I follow Galloway on twitter, but, as one not twitter obsessed, haven't really paid too much attention to him before. I simply knew him as the guy who had T-Shirts made up that said, "I'm following @mattgalloway #supergenius".

Well a super genius he is, or if that's overstating it, he is at least a super dynamic speaker. His observations, from roses to Harley Davidson motorcycles, were poignant and so damn true. The point of his five minute twenty slide rant: Have an original thought already.

Hopefully he'll post his slide deck or a video will be made available.

I also want to give props to my former colleague and fellow entrepreneur Geoffrey Simpson. His presentation "Challenging Yourself" was a real inspiration. Geoff is always trying new things, and presenting to crowd that size might have been one of those first steps he described. His conviction, energy, and positive message definitely won over the crowd.

There were other notable presentations, too many to mention. Here's a last one: @OKLibrarian was hilarious.

Verified By Visa

I've been studying Verified By Visa from the Issuer perspective for an upcoming project. You can can get an official Visa overview by reading this document.

Verified By Visa is more-or-less the brand name for the 3-D Secure service. 3-D stands for "Three Domain", referring to the three parties (Visa calls them domains) that provide the software that comprise the service.

Issuer Domain

• Implementor: Card holder account; Card issuer or processor
  
• Servers: Access Control Server, Authentication Enrollment Server (or pages)
  

Interoperability Domain

• Implementor: Visa
• Servers: Visa Directory Server, Authentication Server
  

Acquirer Domain

• Implementor: Merchant
• Servers: Web Server fitted with Merchant Server Plug-in
  

The issuer implementation is a relatively straightforward secure HTTPS web service. It is even possible (and permitted) to use a single web server instance to fulfill the roles of both Access Control Server and Authentication Enrollment Server. The Issuer server accepts requests from merchant web pages via web requests sent AJAX style and makes requests of Visa's Interoperability Domain servers. All communication is done using a straightforward XML protocol.

It's quite the network dance that goes on between all of the players. Visa apparently distributes a JavaScript library to each merchant for use on their web sites that abstracts the details of the interaction for them, hiding the complexities of the communication to and from the Issuer and Visa servers.

I created a nice diagram that details the communication flow between the 3-D parties, but I probably shouldn't publish it. Ask me if you've got questions.

cx_Oracle on Ubuntu 9.04 64 bit

I had a heck of a time getting cx_Oracle for 64 bit. Using the 32 bit version didn't work either, possibly due to the ultimate problem that I eventually fixed locally.

Compounding the problem: Sourceforge was buggy as hell today and I couldn't download anything due to excessive redirects. What's up with Sourceforge?

Finally found the package here:

http://rpm.pbone.net/index.php3/stat/6/idpl/12200190

After installing, still no dice. Importing cx_Oracle failed with a message saying unable to locate the package. A search of my directories showed that the RPM dropped my site-package in the wrong location, at least wrong for Ubuntu 9.04. This fixed it:

sudo cp /usr/lib/python2.6/site-packages/cx_Oracle-5.0.1-py2.6.egg-info /var/lib/python-support/python2.6
sudo cp /usr/lib/python2.6/site-packages/cx_Oracle.so /var/lib/python-support/python2.6

/var/lib instead of /usr/lib... wtf

Startup Lessons #3 - What will it take to feed the founders?

If you can afford to develop your business plan and prototype after hours and on the weekend--- while keeping your day job---do it.

If you have a little bit of startup capital, assume that you will never receive another dime.

Be realistic about what it is going to take to feed your founders. Can they live on the amount in-hand for one year? If not, don't quit the day job.

Our business was born of four founders, all of whom were fairly mature in their careers. It felt like we were making big sacrifices by paying ourselves approximately 60% of our former incomes. Still---the combined amount required was considerable. With what we raised we could have paid two of the founders full-time salaries for a year. But we all went in full-time, leaving us with only 6 months running room. It wasn't enough.

Six more months would have afforded us the opportunity to complete a pilot we began at a financial institution---a pilot with great promise.

So why did we decide the four founders should quit their day jobs up front? A number of reasons, including a strong belief that, to make the most of our combined energies, we must all be 100% committed to the effort. That creativity would best flow if we could all work together in the same room. There may have even been a "fun" factor. All wanted in on the excitement of rapidly building something from the ground up.

What is the take away? Don't let irrational exuberance into the equation. Plan for the worst. If you can't support your founders for a year with what you have in-pocket, think hard. Stick with the day job or find other creative ways to support yourself while you build the company.