Deploying Bitcoin-Qt on OSX...

My goal: create a .dmg file that anybody running OSX version 10.5 or later on an Intel-based Mac can use to install and run Bitcoin-Qt (aka Bitcoin version 0.5).

How hard could it be? There must be plenty of people who are using the QT cross-platform graphical user interface toolkit to create Mac applications... right?

With some help and hints and a lot of googling I did finally succeed, but not before giving up at least three separate times. I'm writing this to, hopefully, save somebody else lots of frustration.

Side note: Bitcoin-Qt doesn't run on PowerPC chips, doesn't use more than 4 gigabytes of memory, and doesn't contain any CPU-performance-critical code. So there is no reason to create and deploy a "universal" app bundle, and I saved myself a lot of recompiling time by compiling just 32-bit i386.

Problem 1: I've got a 64-bit Mac running Snow Leopard (10.6).

Solution 1: compile everything with these flags:
-mmacosx-version-min=10.5 -arch i386 -isysroot /Developer/SDKs/MacOSX10.5.sdk

Compiling Bitcoin-Qt with those flags is easy when you know how; just add something like this to the Bitcoin-Qt.pro file:

macx:QMAKE_CXXFLAGS += -mmacosx-version-min=10.5 -arch i386 -isysroot /Developer/SDKs/MacOSX10.5.sdk

Problem 1a: I'm using macports for dependencies, so it wasn't obvious how to compile everything with those flags.

Solution 1a: Edit /opt/local/etc/macports/macports.conf and set macosx_deployment_target and the build_arch:

# Building for lowest-common denominator OSX 10.5, 32-bit
# (see build_arch i386 later in this file)
macosx_deployment_target 10.5

# CPU architecture to compile for. Defaults to i386 or ppc on Mac OS X 10.5
# and earlier, depending on the CPU type detected at runtime. On Mac OS X 10.6
# the default is x86_64 if the CPU supports it, i386 otherwise.
# GAVIN: overrode to default to 32-bit i386 build
build_arch                      i386

I then ran:

port list installed | cut -f 1 -d' ' | uniq > /tmp/portsinstalled
sudo port remove installed
sudo port install $(cat /tmp/portsinstalled)

... to remove and recompile/reinstall everything. Yes, that took several hours.

Problem 2: macdeployqt refuses to deploy one of our dependencies.

Solution 2: file a bug (QT-21913), and patch my copy of macdeployqt.

How'd I patch? I found the macdeployqt source inside /opt/local/var/macports/build/ (find /opt/local/var/macports/build -name macdeployqt -print), copied it to $HOME/src/, and ran qmake/make to compile after editing its .cpp files.

Problem 3: macdeployqt created an App bundle that crashed on startup.

Solution 3: When you 'port install qt4-mac', it installs a shared-library version of Qt. macdeployqt seems to work with a shared-library version of Qt... but it really doesn't. It wants the 'Framework' version. So the solution: 'port deactivate qt4-mac' then 'port install qt4-mac +framework'.

Problem 4: Images were not being displayed by the deployed App.

Solution 4: macdeployqt wasn't deploying plugins and the images are JPEG format which requires the JPEG plugin.

It wasn't deploying them because it thinks the plugin path is "/Developer/Applications/Qt/plugins"... but since I'm using macports, the actual pluginPath is "/opt/local/share/qt4/plugins".

I took the easy way out and changed the path in my patched version of macdeployqt. I'll send the patch to the macports qt4-mac maintainer.

---

Did I make things hard for myself by following the Qt documentation for deploying on OSX-- is there some magical XCode or commercial application for creating App bundles that I don't know about that would have saved me days of frustration and work?

Yahoo Pipes Yay!

This morning I decided to take a little time and look at tools I could use to filter my RSS feeds; for example, I have some feeds written by multiple authors, and I might want to get entries posted by just one of the authors.

A little searching turned up some small companies that look like they're struggling (ZapTxt, Blastfeed, FeedRinse) and Yahoo Pipes. I've already got a Yahoo account, and Yahoo is the least likely of the services to disappear tomorrow, so that's what I used.

And wow! In less than five minutes I have a new rss feed that does exactly what I want! Now I'm going to see if I can come up with a filtering rule that nukes posts that are nothing but recycled content from somewhere else... (and I may have to write a filter for any post by Brad deLong that has the phrase "a better press corps" in it...).

SoyLatte and GWT: Retreat!

After much struggle and consternation trying to do development on my Mac with SoyLatte (version 1.0.2-- it's a JDK1.6 implementation for the Mac) and GWT (version 1.5), I've decided to retreat back to JDK1.5.

So, I've recompiled lots of stuff (we had compiled our own and third-party JARs with JDK1.6), rewrote a couple of little bits of code (surprisingly little-- one use of Deque that was a straight-up replacement with LinkedList, and one use of java.text.Normalizer that was easy to replace with sun.text.Normalizer), and I'm much happier.

Just in case it helps somebody else, my dead ends were:

SoyLatte wouldn't load the libgwt-ll.jnilib . Turns out simply creating a copy named "libgwt-ll.dylib" makes SoyLatte happy...

... but trying to run the gwt hosted mode browser results in an error about not being able to open the X11 Display :0.0. No problem, I've got X on my mac, so I startx and run from an xterm window...

... and I get some weird exception deep in the SoyLatte code. That's the point where I decided it would be less painful to go back to JDK1.5.

Oh, also, another bizarre-ness:

I'm using ant's os.name built-in property to read a operating-system-specific .antproperties file (so the build can use the GWTCompiler in either the gwt-dev-mac or gwt-dev-linux jar files), like this:

<property file="${os.name}.antproperties" />

Depending on which version of ant I'm using, os.name is either "Darwin" or "Mac OS X".

I don't like to check in filenames with spaces in them-- in my experience it's just not a good idea. So my solution is to check in the Darwin.antproperties file, and just manually create a symbolic link from that file to "Mac OS x.antproperties"

Spring Security+GWT Rant

One day, I may come to love and appreciate Spring (aka acegi) Security.

I'll appreciate the elegance of being able to inject security into web pages and methods that know nothing authentication and authorization. Maybe one day I'll embrace editing XML configuration files and restarting my web server as a Better Way than writing code in a scripting language like PHP or Python and poking the refresh button on my web browser.

That day is not today.

My task seems like it should be reasonably straightforward:
we've got a web site that you can browse either logged-in (in which case you'll see Edit buttons, and can submit edit/updates) or not-logged-in (in which case you see "Login to Edit" links).

We're re-implementing an old version of the site using GWT and Spring. I'm rewriting the login/logout code to use Spring Security... and MAN! It's been PAINFUL!

It's been painful for a few reasons:

1. I'm new to Spring, XML configuration files, and GWT. Oh, and Java and Eclipse, too (but I feel like I've started to master those in the last couple of months).

2. There's lots of documentation and examples available online, but it's not easy to figure out what's relevant and what's not. A lot of the documentation that's out there is for the old, deprecated, previous version (the acegi security library). A lot of the new documentation assumes that you were using the old, deprecated version-- so it contains all sorts of stuff that I don't need to know. But I don't know that I don't need to know it...

3. The edit/compile/debug turnaround time is pretty bad. Editing the xml file with the security stuff in it, then patching it into the war.d/ directory is quick, but starting up Jetty and refreshing my test web page takes about 45 seconds.

ANYWAY, in the hopes that describing the snags I encountered might help somebody else, here are a few random things that are probably obvious to somebody who knows a lot more about this stuff than I do:

Here's the security-context.xml file I'm using:

<?xml version="1.0" encoding="ISO-8859-1"?>

<!-- Spring Security settings -->

<beans:beans xmlns="http://www.springframework.org/schema/security"
    xmlns:beans="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd
                        http://www.springframework.org/schema/security http://www.springframework.org/schema/security/spring-security-2.0.2.xsd">

  <http auto-config='true'>
    <intercept-url pattern="/foo/**" access="ROLE_USER" />
  </http>

  <beans:bean id="myJdbcDAO"
        class="org.springframework.security.userdetails.jdbc.JdbcDaoImpl">
    <beans:property name="dataSource">
      <beans:ref bean="varDataSource"/>
    </beans:property>
    <beans:property name="usersByUsernameQuery">
      <beans:value>SELECT email AS username, LOWER(password),'true' AS enabled FROM users WHERE email=?</beans:value>
    </beans:property>
    <beans:property name="authoritiesByUsernameQuery">
      <beans:value>SELECT email AS username, 'ROLE_USER' AS authority FROM users WHERE email=?</beans:value>
    </beans:property>
  </beans:bean>
  <authentication-provider user-service-ref="myJdbcDAO">
    <password-encoder hash="sha"/>
  </authentication-provider>

</beans:beans>

In no particular order, this is different from what the Spring Security tutorials describe in the following ways:

1. For initial testing, I'm protecting a non-existent URL ( /foo ). I want to make sure that I get a login page if I go there.
2. Usernames and passwords are stored in an already-existing database (the varDataSource is a JdbcTemplate database doo-hickey that is defined in another xml configuration file).
3. I had to step through the code for org.springframework.security.providers.dao.DaoAuthenticationProvider to figure out why I was getting "Bad credentials" when trying to login. It turns out the SHA password comparison code inside Spring is fussy about upper and lowercase (you've gotta give it a lowercase hex string, which explains the LOWER(password) in the usersByUsernameQuery).
4. I've hardcoded all users to have ROLE_USER; eventually, I'll teach our old code to handle multiple roles
5. I only have a vague idea of why I have to say <beans:bean> (instead of just <bean>); if I understood XML namespaces I'm sure it would be easy and obvious.

There's still one big security hole here-- passwords are sent across the network in the clear (I know that's true because I saw my clear-text password as I was stepping through the DaoAuthenticationProvider code on the server). One solution is to require logins go over an https connection, and Spring Security supports that, but I'm surprised that this hole wasn't mentioned in any of the Spring Security tutorials or documentation that I came across.

CMS Made Simple: very nice!

I've volunteered to help the Amherst League of Women Voters with their web site (here). Just about the first thing they mentioned is that they want an easier way to update content.

I happen to know quite a bit about that problem; I implemented much of Gravity Switch's "content management system" (what an awful term for "software that lets you create a web site"). The Gravity Switch CMS would be overkill for what they need, and it had been a while since I'd looked at the open source solutions, so I decided to take another look at open source solutions.

The usual suspects-- Drupal, Joomla, WordPress-- are all still around, and they all have more than enough functionality. But they all fail the simplicity test; any software for creating a web site that doesn't have a "Create a new page" button is too complex.

Then I found CMS Made Simple, and played around with it. Finally, an open source CMS with a reasonable user interface, where simple things are simple and complex things are possible! One of these days I'm going to get around to converting my old, hand-coded HTML home page to a CMS Made Simple-powered site...

Python SIGALRM oddness

I usually don't stop debugging my code until I've found the root cause of a problem. And the cause just about always turns out to be me being an idiot and doing something boneheaded.

Anyway, I ran into a hard-to-reproduce, hard-to-debug problem with some Python code, and still haven't figured out what's going on.

The code in question downloads PDF files from the Web, using Python's httplib2. I was using signal.alarm() to setup a timeout-- if httplib took more than four minutes to download the file, I wanted to abort the transfer and move on to the next file (otherwise a very slow web server could make the process take, essentially, forever).

The relevant code started out like this:

# How long to wait?  Lets do a "typical worst case":
#   5 megabyte paper, through a fairly busy DSL connection (20Kbytes/second upload)
#   works out to 250 seconds (downloads have a little over 4 minutes before timing out)
alarm_timeout = 250

def _alarm_handler(signum, frame):
    """Bail out of HTTP reads if server takes too long."""
    raise IOError(60, "fetch timed out")

def _start_alarm(timeout=alarm_timeout):
    signal.signal(signal.SIGALRM, _alarm_handler)
    signal.alarm(timeout)

def _stop_alarm():
    signal.alarm(0)

def _remember_and_write(reader, db, save_to_directory):
    _start_alarm()
    data = reader.read()
    _stop_alarm()

The problem was that every once in a while the alarm wouldn't go off and the code would hang. I'm fetching tens of thousands of pdf files, and maybe one in a thousand would hang-- and, of course, if I tried fetching that particular file again everything would work just fine. Python wasn't hung-- if I sent the process a SIGINT signal it would wake up and generate a KeyboardInterrupt, showing a stack trace somewhere deep in the httplib fetching code.

That stack trace gave me a clue, and helped me fix bug number 1: I was timing out only around the reading of the body of the HTTP request, but sometimes fetching (for example) the robots.txt file or the HTTP header could take a very long time. So, confident that I'd figured out the problem, I re-arranged the code so that an alarm was always set to go off if anything took too long. I re-run a big fetch... and it still hangs after several hours.

I realize my code has another bug just thinking about the problem during my morning shower. What if something in httplib is catching (and silently dropping) the IOError that I raise in the alarm handler? So I rewrite my code to raise a custom exception, and re-run a big fetch... and no dice, it still hangs. But I can still interrupt it by sending a KeyboardInterrupt.

Maybe httplib2 is catching ALL exceptions somewhere deep in it's code, so the alarm goes off, throws the exception, which is caught by httplib insted of interrupting out of whatever operation is hanging... I briefly contemplate digging into it's source, and decide it would be easier to just change my entire approach.

Which is what I ended up doing. Now I do a fork() to create two processes-- one of which does the downloads (as before), updating the timestamp on a file every time it starts a file download. And another which checks that file every four minutes, and if it hasn't changed in over 4 minutes, sends the main process a SIGINT (which Python turns into a KeyboardInterrupt exception) to tell it to proceed to the next file.

Works like a charm. I wrapped up the "fork and monitor" functionality into this nifty little routine:

#!/usr/bin/env python
"""
Routine that monitors a log file, and, if it hasn't been modified in a certain amount of time,
sends a signal to the process that called the routine.

Gavin ran into problems using SIGALRM, so came up with this solution instead.
"""

import atexit
import os
import signal
import sys
import time

def monitor_and_signal(filename, timeout, signal):
    "If file doesn't change in timeout seconds, send this process {signal}.  Returns monitoring process pid."

    parent_pid = os.getpid()
    # Strategy is:  fork a child process, which mostly sleeps.
    pid = os.fork()
    if pid == 0:
        # We're the child process; monitor the file:
        while 1:
            time.sleep(timeout)
            try:
                info = os.stat(filename)
            except OSError:
                sys.exit(0)  # Nothing to monitor any more, exit.
            if (time.time() - info.st_mtime) > timeout:
                try:
                    os.kill(parent_pid, signal)
                except OSError:
                    sys.exit(0)  # Parent process goes away: exit.
    return pid
        
if __name__ == '__main__':
    fname = "/tmp/pretendLog.log"
    fp = open(fname, "w")
    fp.write("Testing\n")

    monitor_and_signal(fname, 3, signal.SIGINT)

    for i in range(1,8):
        print("%d second delay (interrupt at >3)"%i)
        try:
            time.sleep(i)
        except KeyboardInterrupt:
            print("Interrupt, was sleeping %d seconds"%i)
        fp.write("%d\n"%i)
        fp.flush()
    fp.close()

Mac, gwt, SoyLatte Java 1.6 == unhappiness

I'm setting up a development environment for GWT (the Google Web Toolkit) on my Mac... and ran into "issues."

It seemed to be going well, until I actually tried to run a sample application, and got:

Warning: The Mac OS -XstartOnFirstThread: is currently unsupported
Unable to load required native library 'gwt-ll'.  Detailed error:
Can't load library: /usr/local/lib/gwt-mac-1.4.61/libgwt-ll.dylib)

Your GWT installation may be corrupt
Exception in thread "main" java.lang.UnsatisfiedLinkError: Unable to load required native library 'gwt-ll'.  Detailed error:
Can't load library: /usr/local/lib/gwt-mac-1.4.61/libgwt-ll.dylib)

Your GWT installation may be corrupt
        at com.google.gwt.dev.shell.LowLevel.init(LowLevel.java:107)
        at com.google.gwt.dev.shell.mac.LowLevelSaf.init(LowLevelSaf.java:262)
        at com.google.gwt.dev.BootStrapPlatform.go(BootStrapPlatform.java:29)
        at com.google.gwt.dev.GWTShell.main(GWTShell.java:318)

A couple of hours of googling, searching, reading release notes, scratching my head...

... and I found the cause, and a workaround.

The cause: I was running SoyLatte (a version of Java 1.6 for the Mac).
The workaround: unset JAVA_HOME, and reset my PATH so I run the java that came with my Mac (version 1.5).

Grumble, grumble... why does it seem like EVERYTHING I try to do with Java is about three times as hard as it should be? Maybe I've just been spoiled by PHP and Python...