We ran into an interesting problem when trying to start up an Oracle instance using dbstart whereby we were getting the following error:

-bash-3.2$ dbstart
 
ORACLE_HOME_LISTNER is not SET, unable to auto-start Oracle Net Listener
Usage: /u01/app/oracle/product/11.2.0/dbhome_1/bin/dbstart ORACLE_HOME
Processing Database instance "orcl": log file /u01/app/oracle/product/11.2.0/dbhome_1/startup.log

Ignoring the usage message we thought that setting the environment variable was what we needed to do, but…

-bash-3.2$ export ORACLE_HOME_LISTNER=$ORACLE_HOME
-bash-3.2$ dbstart
ORACLE_HOME_LISTNER is not SET, unable to auto-start Oracle Net Listener
Usage: /u01/app/oracle/product/11.2.0/dbhome_1/bin/dbstart ORACLE_HOME
Processing Database instance "orcl": log file /u01/app/oracle/product/11.2.0/dbhome_1/startup.log

We ended up looking at the source of dbstart to see what was going on:

# First argument is used to bring up Oracle Net Listener
ORACLE_HOME_LISTNER=$1
if [ ! $ORACLE_HOME_LISTNER ] ; then
  echo "ORACLE_HOME_LISTNER is not SET, unable to auto-start Oracle Net Listener"
  echo "Usage: $0 ORACLE_HOME"

The usage message does explain that you’re supposed to call it like this:

-bash-3.2$ dbstart $ORACLE_HOME

But it still seems a bit weird/misleading to me that you’d override the value of a global variable from inside a script which doesn’t suggest that it’s going to do that!

Such is life in Oracle land..

As I mentioned in a post last week we’ve been automating the setup of our developer machines with puppet over the last week and one thing that we’ve learnt is that you need to be careful about how you define dependencies.

The aim is to get your scripts to the point where the outcome is reasonably deterministic so that we can have confidence they’re going to work the next we run them.

We noticed two ways in which we haven’t quite achieved determinism yet:

Accidental Dependencies

The first few times that we ran the scripts on top of a vanilla image we were doing it on a virtual machine which had VMware tools installed on it.

We’d forgotten that VMware tools had been installed on those VMs and ran into a problem with Oracle dependencies not being satisfied when we ran puppet on some machines which had CentOS installed directly (i.e. not on a virtual machine).

Those dependencies had been satisfied by our VMware tools installation on the VMs so we didn’t realise that we hadn’t explicitly stated those dependencies, something which we have done now.

External Dependencies

We couldn’t find the Firefox version that we wanted install on the default yum repositories so we created a puppet task which linked to a Firefox RPM on an external server and then installed it.

It worked originally but at some stage over the last couple of weeks the URI was changed as a minor version had been upgraded, breaking our script.

We also came across another way that external dependencies can fail today – if a corporate proxy blocks access to the URL!

We’re trying to get to the stage where we’re only relying on artifacts either coming from a yum repository or an internal repository where we can store any libraries which aren’t available through yum.

Don’t assume determinism

While trying to solve these dependency problems in our puppet scripts I made the mistake of assuming that if the script runs through once and works that it’s always going to be that way in the future.

Since we had achieved that previously in my mind it was impossible for it to fail in future which stopped me from properly investigating why it had stopped working.

Over the last 3/4 months I’ve been playing around with the idea of using pomodoros to track all coding/software related stuff that I do outside of work.

I originally started using this technique while I was doing the programming assignments for ml-class because I wanted to know how much time I was spending on it each week and make sure I didn’t run down rabbit holes too often.

One interesting observation that I noticed from keeping the data of these pomodoros was that while during the early programming assignments it would take me 7 or 8 pomodoros to finish, by the end it was down to around 4.

I think this was due to the difficulty of the assignments decreasing as time went on, I didn’t improve that dramatically!

As I mentioned a few weeks ago I’ve also been using pomodoros in combination with a yak stack to make sure I don’t go off track and it’s been interesting applying the technique while trying to solve a problem I’m having with using the Jersey client on Android.

It’s such a fiddly problem and splitting my time into 25 minute slots has forced me to create a plan for what I’m going to try and do in that pomodoro, whether it be ruling out an approach or trying to understand the underlying code that isn’t working.

I haven’t been successful in solving my problem but I’m pretty sure that I’ve spent much less time trying to solve it than I would have otherwise. I can certainly imagine spending hours aimlessly trying things that have no chance of working.

One thing I’ve been experimenting with is reducing the length of the pomodoro to 15 minutes when I know there’s something specific that I want to investigate and I’m fairly sure it won’t take a full length pomodoro.

Previously I would end up just killing time for 10 minutes or just resetting the pomodoro because I didn’t have anything else to do.

I generally enjoy coding much more by applying this time constraint and I think the reason for that is explained by The Progress Principle, which I’m currently reading:

If people are in an excellent mood at the end of the day, it’s a good bet that they have made some progress in their work. If they are in a terrible mood, it’s a good bet that they have had a setback.

To a great extent, inner work life rises and falls with progress and setbacks in the work. This is the progress principle

Using a pomodoro seems to reduce the amount of time that is spent dealing with setbacks and it creates frequent opportunities to discard an approach you’re taking if it’s clear that it’s not going anywhere.

A disadvantage that I’ve sometimes felt when working on the Jersey/Android problem is that I really don’t want to spend 25 minutes working on it because I’ve been getting absolutely nowhere with it for about 6/7 pomodoros now.

I’d rather delude myself that I’m going to magically fix it just by fiddling around with the code for an indeterminate period of time!

In a way constraining coding in this way does take some of the fun out of it as well because it’s now more structured and you tend to have fun when you’re just randomly doing stuff and lose track of time.

On the other hand I probably end up doing a lot more of the stuff I want to do when I constrain it in this way!

Decisions, decisions…

We’ve been spending some time trying to setup our developer environment on a Oracle Linux 5.7 build and one of the first steps was to install Puppet as we’ve already created scripts which automate the installation of most things.

Unfortunately Oracle Linux builds don’t come with any yum repos configured so when you run the following command…

ls -alh /etc/yum.repos.d/

…you don’t see anything

We eventually realised that there are a list of public yum repositories on the Oracle website, of which we needed to download the definition for Oracle Linux 5 like so:

cd /etc/yum.repos.d
wget http://public-yum.oracle.com/public-yum-el5.repo

We then need to edit that file to enable the appropriate repository. In this case we want to enable ol5_u7_base:

[ol5_u7_base]
name=Oracle Linux $releasever - U7 - $basearch - base
baseurl=http://public-yum.oracle.com/repo/OracleLinux/OL5/7/base/$basearch/
gpgkey=http://public-yum.oracle.com/RPM-GPG-KEY-oracle-el5
gpgcheck=1
enabled=1

I made the mistake of enabling ol5_u5_base which led to us getting some really weird problems whereby yum got confused as to which version of libselinux we had installed and was therefore unable to install libselinux-ruby as its dependencies weren’t being properly satisfied.

Calling ‘yum list installed’ suggested that we had libselinux 1.33.4.5-7 installed but if we ran ‘yum install libselinux’ then it suggested we already had 1.33.4.5-5 installed. Very confusing!

After trying to uninstall and downgrade libselinux and pretty much destroying the installation in the process, another colleague spotted my mistake.

We also found that we had to add the epel repo which gave us access to some other packages that we needed:

rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm

After all that was done we were able to run the command to install puppet:

yum install puppet

That installs puppet 2.6.12 as that’s the latest version in that repo. The latest stable version is 2.7.9 but I think we’ll need to hook up a puppet specific repo to get that working.

I recently came across Carl Erickson’s ‘small teams are dramatically more efficient than large teams‘ blog post which reminded me of something which my colleague Ashok suggested as a useful way for determining team size – the application footprint.

As I understand it the application footprint is applicable for an application at a given point in time and determines how many parallel tasks/streams of work we have.

In the case of the project that I’m currently working on there are 3 separate components which need to interact with each other via an API but otherwise are independent.

We can therefore have 3 pairs working – one on each component – and won’t have to worry about them stepping on each other’s toes.

One interesting thing about the application footprint is that it doesn’t stay the same size all the time.

More often than not once a team has gained trust by getting a release out the product owner will start prioritising more independent features which don’t necessarily overlap.

At this stage it might not be such a bad idea to add people to the team if we want to try and finish more quickly.

If we’re already at the point where we have the same number of pairs as parallel pieces of work then adding people is going to be problematic because we’ll struggle to find work for everyone to do.

Stories in the same stream will have dependencies on each other and although it’s theoretically possible to start on something which has a dependency, the likelihood of having to rework it is higher.

One way to get around that problem if we decide that we don’t want to reduce our team size is to have a pair assigned to working on bugs, cross functional requirements such as performance testing/tuning or doing some technical analysis on upcoming stories.

It’s easy enough to remember all this when you’re starting out building an application but I think it’s something that we need to keep in mind so that if there’s pressure to add people to ‘go faster’ then we can determine if that will actually be the case.

As an aside

Obviously there are times when we decide that we’re happy to put more people on a team than it’s footprint might suggest in order to get an overall gain.

For example with 5 pairs we may finish 50 points in a week but if we increase to 10 pairs then perhaps we now get 60 points.

We’ve nearly halved the efficiency of each pair but overall we’ve got a marginal gain which sometimes makes sense. We also need to be aware of the collective unresponsibility that we might introduce by doing this.

Photo courtesy of farlane

A few weeks ago a slide deck from an Esther Derby presentation on retrospectives was doing the rounds on twitter and one thing that I found interesting in the deck was the suggestion that a retrospective needs to be focused in some way.

I’ve participated in a few focused retrospectives over the past 7/8 months and I think there are some things to be careful about when we decide to focus on something specific rather than just looking back at a time period in general.

Victimisation

In a retrospective about 6 months ago or so we focused on the analysis part of our process as we’d been struggling to know when a story was complete and what exactly its scope was.

The intention wasn’t the victimise the people working in that role but since there were very few of them compared to people in other roles they were forced onto the defensive as people criticised their work.

It was a very awkward retrospective and it felt like a retrospective was probably the wrong place to address the problem.

It might have been better for the analysts to have been given the feedback privately and then perhaps worked on a solution with a smaller group of people.

Looking for a problem when there isn’t one

I had an interesting conversation with a colleague about whether with very focused retrospectives we end up looking for something to change rather than having any specific pain point which necessitates change.

The problem with this is that there’s a thin line between following the status quo because it works and getting complacent and not looking for ways to improve.

It is interesting to keep in mind though that if it doesn’t seem like there is something to change in an area then perhaps that’s the wrong thing to be focusing on at the moment, which nicely leads into…

Let the team choose the area of focus

There can be a tendency in the teams I’ve worked on for people in managementy roles to dictate what the focus of the retrospective will be which makes sense in a way since they may be able to see something which the team can’t.

On the other hand it can mean that we end up focusing on the wrong thing and team members probably won’t be that engaged in the retrospective since they don’t really get to dictate what’s talked about.

Esther points this out out on slide 23 of the presentation – “Choose a focus that reflects what’s going on for the team“. This perhaps can be determined by having a vote before hand based on some topics that seem prominent.

In summary

There’s lots of other useful tips in Esther’s slide deck which are worth having a look at and I’m sure most of the potential problems I’ve listed probably don’t happen when we have a highly skilled/experienced facilitator.

I like using Wireshark to have a look at the traffic going across different interfaces but because it shows what’s happening across the wire by the packet it’s quite difficult to tell what a request/response looked like.

I’ve been playing around with restfulie/Vraptor today so I wanted to be able to see the request/response pair when something wasn’t working.

I didn’t know it was actually possible but this post on StackOverflow describes how.

First we need to select the row which contains any part of our request/response – in this case I just selected the row representing the request – and then we go to the Analyze menu and click ‘Follow TCP Stream’:

We can then see the requests/responses which happened all next to each other:

The keyboard shortcut to get to that menu is ‘Alt-A F’ but for some reason the ‘Alt’ key wasn’t working for me by default so I had to follow the instructions on Francis North’s blog to get it working.

I spent a bit of time this afternoon trying to export an Oracle test database so that we could use it locally using the exp tool.

I had to connect to exp like this:

exp user/password@remote_address

And then filled in the other parameters interactively.

Unfortunately when I tried to actually export the specified tables I got the following error message:

EXP-00008: ORACLE error 904 encountered
ORA-00904: "POLTYP": invalid identifier
EXP-00000: Export terminated unsuccessfully

I eventually came across Oyvind Isene’s blog post which pointed out that you’d get this problem if you tried to export a 10g database using an 11g client which is exactly what I was trying to do!

He explains it like so:

The export command runs a query against a table called EXU9RLS in the SYS schema. On 11g this table was expanded with the column POLTYP and the export command (exp) expects to find this column.

I needed to download the 10g client so that I could use that version of exp instead. I haven’t quite got it working yet but at least it’s a different error to deal with!

In my last post I showed how I’d been able to write a test around saved preferences in my app by making use of a ShadowPreferenceManager but it seemed a bit hacky.

I didn’t want to have to do that for every test where I dealt with preferences – I thought it’d be better if I could wrap the preferences in an object of my own and then inject it where necessary.

Another benefit of taking this approach is that the interface of exactly what I’m storing as user preferences.

I wanted the class to be roughly like this:

public class UserPreferences {
    public String userKey() {
        return getDefaultSharedPreferences().getString("user_key", "");
    }
 
    public String userSecret() {
        return getDefaultSharedPreferences().getString("user_secret", "");
    }
 
    private SharedPreferences getDefaultSharedPreferences() {
        return PreferenceManager.getDefaultSharedPreferences(getContextHereSomehow());
    }
}

Initially it wasn’t entirely obvious how I could get a Context to pass to getDefaultSharedPreferences but I came across a blog post explaining how to do it.

What we need to do is inject a Context object via the constructor of the class and decorate the constructor with the @Inject attribute so that Roboguice will resolve the dependency:

public class UserPreferences {
    private Context context;
 
    @Inject
    public UserPreferences(Context context) {
        this.context = context;
    }
 
    public SharedPreferences getDefaultSharedPreferences() {
        return PreferenceManager.getDefaultSharedPreferences(context.getApplicationContext());
    }
 
    public String userKey() {
        return getDefaultSharedPreferences().getString("user_key", "");
    }
 
    public String userSecret() {
        return getDefaultSharedPreferences().getString("user_secret", "");
    }
}

We never have to explicitly setup a binding for Context in our Roboguice because it’s already been done for us in RoboModule which is instantiated by RoboApplication which we extend like so:

public class TweetBoardApplication extends RoboApplication {
    private Module module = new RobolectricSampleModule();
 
    @Override protected void addApplicationModules(List<Module> modules) {
        modules.add(module);
    }
 
    public void setModule(Module module) {
        this.module = module;
    }
}

We then hook TweetBoardApplication up in the AndroidManifest.xml file like this:

<manifest xmlns:android="http://schemas.android.com/apk/res/android"
          package="com.pivotallabs"
          android:versionCode="1"
          android:versionName="1.0">
    <application
            android:label="@string/app_name"
            android:theme="@android:style/Theme.Light.NoTitleBar"
            android:icon="@drawable/app_icon"
            android:name="TweetBoardApplication">
   </application>
</manifest>

And that’s it!

I’ve been writing some tests around an app I’ve been working on using the Robolectric testing framework and one thing I wanted to do was check that an OAuth token/secret were being saved to the user’s preferences.

The code that saved the preferences looked like this:

public class AuthoriseWithTwitterActivity extends RoboActivity {
    @Override protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(intent);
        ...
        save("fakeToken", "fakeSecret");
        ...
    }
 
    private void save(String userKey, String userSecret) {
        SharedPreferences settings = PreferenceManager.getDefaultSharedPreferences(getBaseContext());
        SharedPreferences.Editor editor = settings.edit();
        editor.putString("user_key", userKey);
        editor.putString("user_secret", userSecret);
        editor.commit();
    }
}

This is an outline of what I wanted to do in the test:

@RunWith(InjectedTestRunner.class)
public class AwesomeTest {
    @Test public void shouldSaveOAuthDetails() {
        activity.onCreate(null);
 
        ShadowIntent shadowIntent = shadowOf(activity).getNextStartedActivity();
 
        // Get SharedPreferences and check 'fakeToken' and 'fakeSecret' are stored. 
    }
}

In Robolectric it’s possible to replace classes with shadow versions of themselves which get used in the test so I first created a shadow version of PreferenceManager:

@Implements(PreferenceManager.class)
public class ShadowPreferenceManager {
    private static SharedPreferences  preferences = new TestSharedPreferences(new HashMap<String, Map<String, Object>>(), "__default__", Context.MODE_PRIVATE);
 
    @Implementation
    public static SharedPreferences getDefaultSharedPreferences(Context context) {
        return preferences;
    }
 
    public static void reset() {
        preferences = new TestSharedPreferences(new HashMap<String, Map<String, Object>>(), "__default__", Context.MODE_PRIVATE);
    }
}

I had to make preferences a static variable here so that it’ll retain state. It’s a bit hacky but it’ll do for now.

Then to hook it up I had to change my test to read like this:

@RunWith(InjectedTestRunner.class)
public class AwesomeTest {
    @Test public void shouldSaveOAuthDetails() {
        Robolectric.bindShadowClass(ShadowPreferenceManager.class);
        activity.onCreate(null);
 
        ShadowIntent shadowIntent = shadowOf(activity).getNextStartedActivity();
 
        SharedPreferences defaultSharedPreferences = PreferenceManager.getDefaultSharedPreferences(activity);
        assertThat(defaultSharedPreferences.getString("user_key", ""), equalTo("fakeToken"));
        assertThat(defaultSharedPreferences.getString("user_secret", ""), equalTo("fakeSecret"));
 
        ShadowPreferenceManager.reset();
    }
}

The InjectedTestRunner class used here is pretty much like the one in the Robolectric code base.

There is actually a ShadowPreferenceManager in the Robolectric library but it doesn’t seem to store preferences anywhere as far as I can tell so it wasn’t quite what I wanted.

In the app I’m working on I make use of the ViewPager class which is only available in the compatibility package from revisions 3 upwards.

Initially I followed the instructions on the developer guide to get hold of the jar but now that I’m trying to adapt my code to fit the RobolectricSample, as I mentioned in my previous post, I needed to hook it up as a Maven dependency.

I added the dependency to my pom.xml like this:

<dependency>
  <groupId>android.support</groupId>
  <artifactId>compatibility-v4</artifactId>
  <version>r6</version>
</dependency>

But when I tried to resolve the dependencies (via ‘mvn test’) I ended up with this error:

Downloading: http://repo1.maven.org/maven2/android/support/compatibility-v4/r6/compatibility-v4-r6.pom
[WARNING] The POM for android.support:compatibility-v4:jar:r6 is missing, no dependency information available
Downloading: http://repo1.maven.org/maven2/android/support/compatibility-v4/r6/compatibility-v4-r6.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.878s
[INFO] Finished at: Sun Jan 08 20:42:17 GMT 2012
[INFO] Final Memory: 8M/554M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project tweetboard: Could not resolve dependencies for project com.markhneedham:tweetboard:apk:1.0.0-SNAPSHOT: Could not find artifact android.support:compatibility-v4:jar:r6 in central (http://repo1.maven.org/maven2) -> [Help 1]

A bit of googling led me to a demo project showing how to hook up the compatibility package. It linked to the Maven Android SDK Deployer which is:

The Maven Android SDK Deployer is a helper maven project that can be used to install the libraries necessary to build Android applications with Maven and the Android Maven Plugin directly from your local Android SDK installation.

I had to first clone that git repository:

git clone git://github.com/mosabua/maven-android-sdk-deployer.git

And then find the compatability-v4 package and install it:

$ cd maven-android-sdk-deployer
$ cd extras/compatibility-v4
$ mvn clean install

I initially made the mistake of not setting $ANDROID_HOME to the location of the Android SDK on my machine, which led to the following error:

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.484s
[INFO] Finished at: Sun Jan 08 20:51:07 GMT 2012
[INFO] Final Memory: 3M/81M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:properties-maven-plugin:1.0-alpha-2:read-project-properties (default) on project android-extras: Properties file not found: /Users/mneedham/github/maven-android-sdk-deployer/extras/${env.ANDROID_HOME}/extras/android/support/source.properties -> [Help 1]

Setting it solves the problem:

$ export ANDROID_HOME=/Users/mneedham/github/android/android-sdk-macosx

There are more detailed instructions on the home page of the github project.

I’ve been trying to adapt my Android application to fit into the structure of the RobolectricSample so that I can add some tests around my code but I was running into a problem when trying to deploy the application.

To deploy the application you need to run the following command:

mvn package android:deploy

Which was resulting in the following error:

[INFO] UNEXPECTED TOP-LEVEL ERROR:
[INFO] java.lang.OutOfMemoryError: Java heap space
[INFO] 	at com.android.dx.rop.code.PlainInsn.withNewRegisters(PlainInsn.java:152)
[INFO] 	at com.android.dx.ssa.NormalSsaInsn.toRopInsn(NormalSsaInsn.java:121)
[INFO] 	at com.android.dx.ssa.back.SsaToRop.convertInsns(SsaToRop.java:342)
[INFO] 	at com.android.dx.ssa.back.SsaToRop.convertBasicBlock(SsaToRop.java:323)
[INFO] 	at com.android.dx.ssa.back.SsaToRop.convertBasicBlocks(SsaToRop.java:260)
[INFO] 	at com.android.dx.ssa.back.SsaToRop.convert(SsaToRop.java:124)
[INFO] 	at com.android.dx.ssa.back.SsaToRop.convertToRopMethod(SsaToRop.java:70)
[INFO] 	at com.android.dx.ssa.Optimizer.optimize(Optimizer.java:102)
[INFO] 	at com.android.dx.ssa.Optimizer.optimize(Optimizer.java:73)

I’d added a few dependencies to the original pom.xml file so I figured on of those must be causing the problem and eventually narrowed it down to be the twitter4j-core library which I had defined like this in the pom.xml file:

<dependency>
  <groupId>org.twitter4j</groupId>
  <artifactId>twitter4j-core</artifactId>
  <version>[2.2,)</version>
</dependency>

I found a bug report for the maven-android-plugin which suggested that increasing the heap size might solve the problem.

That section of the pom.xml file ended up looking like this:

<plugin>
  <groupId>com.jayway.maven.plugins.android.generation2</groupId>
  <artifactId>android-maven-plugin</artifactId>
  <version>3.0.0-alpha-13</version>
  <configuration>
    <sdk>
      <platform>10</platform>
      <path>/path/to/android-sdk</path>                
    </sdk>
    <dex>                                
      <jvmArguments>
        <jvmArgument>-Xms256m</jvmArgument>                                        
        <jvmArgument>-Xmx512m</jvmArgument>
      </jvmArguments>                        
    </dex>
    <undeployBeforeDeploy>true</undeployBeforeDeploy>
  </configuration>
  <extensions>true</extensions>
</plugin>

That seemed to get rid of the problem but I also tried changing the plugin version to the latest one and that seemed to solve the problem as well without the need to add the JVM arguments:

<plugin>
  <groupId>com.jayway.maven.plugins.android.generation2</groupId>
  <artifactId>android-maven-plugin</artifactId>
  <configuration>
    <sdk>
      <platform>10</platform>
      <path>/path/to/android-sdk</path>                
    </sdk>
    <undeployBeforeDeploy>true</undeployBeforeDeploy>
  </configuration>
  <extensions>true</extensions>
</plugin>

The latest version is 3.0.2 from what I can tell.

As I mentioned in a previous post I recently wrote some code in my Android app to inform a BroadcastReceiver whenever a service processed a tweet with a link in it but in implementing this I managed to freeze the UI every time that happened.

I made the stupid (in hindsight) mistake of not realising that I shouldn’t be doing a lot of logic in BroadcastReceiver.onReceive since that bit of code gets executed on the UI thread.

The service code which raises the broadcast message is the same as in the previous post:

public class TweetService extends IntentService {
    ...
    @Override
    protected void onHandleIntent(Intent intent) {
        StatusListener listener = new UserStreamListener() {
           // override a whole load of methods - removed for brevity
 
            public void onStatus(Status status) {
                String theTweet = status.getText();
                if (status.getText().contains("http://")) {
                    Intent tweetMessage = new Intent(TweetTask.NEW_TWEET);
                    tweetMessage.putExtra(android.content.Intent.EXTRA_TEXT, status.getText());
                    sendBroadcast(tweetMessage);
                }
 
            }
        };
 
        // code to connect to the twitter streaming API
    }
}

That is then handled like this by the BroadcastReceiver:

public class MyActivity extends Activity {
    protected void onPause() {
        super.onPause();
        if (dataUpdateReceiver != null) unregisterReceiver(dataUpdateReceiver);
    }
 
    protected void onResume() {
        super.onResume();
        if (dataUpdateReceiver == null) dataUpdateReceiver = new DataUpdateReceiver();
        IntentFilter intentFilter = new IntentFilter(TweetTask.NEW_TWEET);
        registerReceiver(dataUpdateReceiver, intentFilter);
    }
 
    private class DataUpdateReceiver extends BroadcastReceiver {
        @Override
        public void onReceive(Context context, Intent intent) {
            if (intent.getAction().equals(TweetTask.NEW_TWEET)) {
                Pattern p = Pattern.compile("(http://[^\\s]+)");
                String theTweet = intent.getStringExtra(TweetTask.NEW_TWEET);
                Matcher matcher = p.matcher(theTweet);
 
                int startIndex = -1;
                int endIndex = -1;
                while (matcher.find()) {
                    startIndex = matcher.start();
                    endIndex = matcher.end();
                }
 
                if (startIndex != -1 && endIndex != -1) {
                    String resolvedUrl = resolveUrl(theTweet.substring(startIndex, endIndex));
                    saveToDatabase(resolvedUrl);
                    updateUI(resolvedUrl);
                }
            }
        }
    }
}

In particular the ‘resolveUrl’ line was probably the one one causing the problem since it makes a network call to resolve URLs from link shorteners.

To stop the screen freezing up I just needed to move most of the code from BroadcastReceiver into the TweetService:

public class TweetService extends IntentService {
    ...
    @Override
    protected void onHandleIntent(Intent intent) {
        StatusListener listener = new UserStreamListener() {
           // override a whole load of methods - removed for brevity
 
            public void onStatus(Status status) {
                String theTweet = status.getText();
                if (status.getText().contains("http://")) {
                    Pattern p = Pattern.compile("(http://[^\\s]+)");
                    Matcher matcher = p.matcher(theTweet);
 
                    int startIndex = -1;
                    int endIndex = -1;
                    while (matcher.find()) {
                        startIndex = matcher.start();
                        endIndex = matcher.end();
                    }
 
                    if (startIndex != -1 && endIndex != -1) {
                        String resolvedUrl = resolveUrl(theTweet.substring(startIndex, endIndex));
                        saveToDatabase(resolvedUrl);
 
                        Intent tweetMessage = new Intent(TweetTask.NEW_TWEET);
                        tweetMessage.putExtra(android.content.Intent.EXTRA_TEXT, resolvedUrl);
                        sendBroadcast(tweetMessage);
                    }
                }
            }
        };
 
        // code to connect to the twitter streaming API
    }
}

And then the code for BroadcastReceiver becomes much simpler which means we’re doing less work on the UI thread:

    private class DataUpdateReceiver extends BroadcastReceiver {
        @Override
        public void onReceive(Context context, Intent intent) {
            if (intent.getAction().equals(TweetTask.NEW_TWEET)) {
                String url = intent.getStringExtra(TweetTask.NEW_TWEET);
                updateUI(url);
            }
        }
    }

And the freezing up of the UI is gone!

In the app I’m working on I created a service which runs in the background away from the main UI thread consuming the Twitter streaming API using twitter4j.

It looks like this:

public class TweetService extends IntentService {
    String consumerKey = "TwitterConsumerKey";
    String consumerSecret = "TwitterConsumerSecret";
 
    public TweetService() {
        super("Tweet Service");
    }
 
    @Override
    protected void onHandleIntent(Intent intent) {
        AccessToken accessToken = createAccessToken();
 
        StatusListener listener = new UserStreamListener() {
           // override a whole load of methods - removed for brevity
 
            public void onStatus(Status status) {
                String theTweet = status.getText();
                if (status.getText().contains("http://")) {
                    // do something with the tweet
                }
 
            }
        };
        ConfigurationBuilder configurationBuilder = new ConfigurationBuilder();
        configurationBuilder.setOAuthConsumerKey(consumerKey);
        configurationBuilder.setOAuthConsumerSecret(consumerSecret);
 
        TwitterStream twitterStream = new TwitterStreamFactory(configurationBuilder.build()).getInstance(accessToken);
        twitterStream.addListener(listener);
        twitterStream.user();
    }
}

That gets called from MyActivity like so:

public class MyActivity extends Activity {
    @Override
    public void onCreate(Bundle savedInstanceState) {
        ...
        super.onCreate(savedInstanceState);
        Intent intent = new Intent(this, TweetService.class);
        startService(intent);
    }
}

I wanted to be able to inform the UI each time there was a tweet which contained a link in it so that the link could be displayed on the UI.

I found a post on StackOverflow which suggested that one way to do this would be to raise a broadcast message which could then be listened to by a BroadcastReceiver in the activity.

It is possible for any other apps to listen to the broadcast message as well if they wanted to but in this case the information isn’t very important so I think it’s fine to take this approach.

I first had to change the service to look like this:

public class TweetTask {
    public static final String NEW_TWEET = "tweet_task.new_tweet";
}
 
public class TweetService extends IntentService {
    String consumerKey = "TwitterConsumerKey";
    String consumerSecret = "TwitterConsumerSecret";
 
    public TweetService() {
        super("Tweet Service");
    }
 
    @Override
    protected void onHandleIntent(Intent intent) {
        AccessToken accessToken = createAccessToken();
 
        StatusListener listener = new UserStreamListener() {
           // override a whole load of methods - removed for brevity
 
            public void onStatus(Status status) {
                String theTweet = status.getText();
                if (status.getText().contains("http://")) {
                    Intent tweetMessage = new Intent(TweetTask.NEW_TWEET);
                    tweetMessage.putExtra(android.content.Intent.EXTRA_TEXT, document);
                    sendBroadcast(tweetMessage);
                }
 
            }
        };
        ConfigurationBuilder configurationBuilder = new ConfigurationBuilder();
        configurationBuilder.setOAuthConsumerKey(consumerKey);
        configurationBuilder.setOAuthConsumerSecret(consumerSecret);
 
        TwitterStream twitterStream = new TwitterStreamFactory(configurationBuilder.build()).getInstance(accessToken);
        twitterStream.addListener(listener);
        twitterStream.user();
    }
}

I then had to define the following code in MyActivity:

public class MyActivity extends Activity {
    protected void onResume() {
        super.onResume();
        if (dataUpdateReceiver == null) dataUpdateReceiver = new DataUpdateReceiver(textExtractionService);
        IntentFilter intentFilter = new IntentFilter(TweetTask.NEW_TWEET);
        registerReceiver(dataUpdateReceiver, intentFilter);
    }
 
    protected void onPause() {
        super.onPause();
        if (dataUpdateReceiver != null) unregisterReceiver(dataUpdateReceiver);
    }
 
    private class DataUpdateReceiver extends BroadcastReceiver {
        private CachedTextExtractionService textExtractionService;
 
        public DataUpdateReceiver(CachedTextExtractionService textExtractionService) {
            this.textExtractionService = textExtractionService;
        }
 
        @Override
        public void onReceive(Context context, Intent intent) {
            if (intent.getAction().equals(TweetTask.NEW_TWEET)) {
                // do something with the tweet
            }
 
        }
    }
}

Now whenever there’s a tweet with a link in it my BroadcastReceiver gets notified and I can do whatever I want with the tweet.

This seems like a reasonably simple solution to the problem so I’d be interested to know if there are any other drawbacks other than the one I identified above.

A couple of years ago I used to write a blog post reflecting on what I’d worked on in the preceding year and what I’d learned and having read 2011 reviews by a couple of other people I thought I’d have a go.

Am I actually learning anything?

A thought I had many times in 2011 was ‘am I actually learning anything?‘ as, although I was working with languages that I hadn’t used professionally before, the applications that we I worked on were very similar to ones that I’ve worked on previously.

Often I’d work on something and know exactly how it should be designed and where we could go wrong since I’d done the same thing several times before and the challenge of not knowing what to do had disappeared somewhat.

Now and then…

I certainly failed to learn one thing a day as I suggested in a blog post a couple of years ago although eventually I managed to learn a bit about node.js and clojure by building some toy applications with my colleague Uday.

We decided to rewrite part of our Scala application in clojure in our own time to see what it’d look like which provided us with an interesting insight into what it’d be like to build a system for the second time when you know exactly what to do.

I also completed ml-class which was fun as it was the type of programming that I’ve never done before. Obviously I’m still a novice at the whole machine learning thing but it’s given me an idea of the sorts of things you can do.

Learning is doing

From February until April I was in Bangalore working as a trainer/coach for one of the ThoughtWorks University batches where we tried as much as possible to reduce the amount of ‘teaching’ done.

Sumeet has previously written about the new style of ThoughtWorks University which is more focused on people working on a real project than sitting in workshops and we tried to take this even further.

Previous groups had spent about 2 weeks doing workshop style sessions and then 4 weeks working on a project but we got it to the point where we spent just over a week in workshops and the rest working on the project.

In general I think it worked reasonably well and the skill level of the group seemed reasonably high by the end. We were lucky that there were only 13 people in the group – it would be interesting to see how our approach would scale.

I’ve also noticed this last year that when I’m learning something new it’s not enough to just do toy exercises anymore, I actually have to build something to retain interest.

During the Christmas holidays I decided to try and build a Flipboard style application for my Android phone so I can (yet again) capture the links that people post on twitter.

Actually having a real problem to solve has made me much more engaged than following a tutorial or hello world demo would have done.

Remembering the value of blogging

My rate of posting on here has decreased a lot over the last year which I think is partly down to the fact that I’ve written about a lot of the stuff I see on projects before but also because I started filtering what I thought was interesting enough to write about.

In hindsight the latter approach doesn’t necessarily make sense – the most read posts on this blog are the ones which I thought were the most pointless when I wrote them.

I got stuck in the mindsight that I wasn’t actually learning anything by writing blog posts, which has been proved wrong multiple times both in terms of what I learn in writing the post and from what I learn from people’s comments.

Expressing opinions in big groups/public

I spent 10 months in late 2010/early 2011 working in India and one of the most interesting things I remember observing was that people seemed very reluctant to express their opinion in big groups.

I thought that was something specific to India but on coming back to the UK I’ve noticed the same thing here as well which means we need to adjust our approach in retrospectives if we want everyone to participate.

I also learnt that expressing strong opinions in public in isn’t necessarily the most effective way of making change happen. I probably should have learnt this already but it became increasingly evident how ineffective this approach was in 2011.

Going at my own pace

A couple of years ago I was advised by a couple of colleagues that the way to get to the ‘next level’ was to become more knowledgeable about the overall architectural design of systems but at the time I wasn’t that interested in that.

It’s only more recently that I’ve found it interesting to read about different architectures on High Scalability or Systems We Make.

Another interesting way for me to learn in this area is to try and understand the architectures used in other ThoughtWorks projects that I didn’t work on and see how they compare to the ones I’ve worked on.

I generally can’t force myself to be interested in something if I’m not but once I am interested then I want to learn every detail about it so it’s better to wait until I become interested naturally.

The next thing which I’m sure I’ll eventually become interested in is tech leading a team which several of my peers (in terms of years of experience) are doing now or have been doing for a year or two. Right now though I want to focus on coding!

Overall…

I’m not sure 2011 was a year where I learned as much as I did in previous years – the learning did seem to taper off a bit which in a way is inevitable unless you completely change your role/the types of things you’re building.

In 2012 I plan to keep learning about Android development and I’m going to be doing algo-class to try and get better at another aspect of programming which I’m not very good at right now.

I want to be able to get the tweets from my timeline into my app which means I need to authorise the app with Twitter using OAuth.

The last time I tried to authenticate using OAuth a couple of years ago was a bit of a failure but luckily this time Honza Pokorny has written a blog post explaining what to do.

I had to adjust the code a little bit from what’s written on his post so I thought I’d document what I’ve done.

We’re using the signpost library for which we need to download the following two jars and put them into the app’s ‘libs’ directory.

wget http://oauth-signpost.googlecode.com/files/signpost-commonshttp4-1.2.1.1.jar
wget http://oauth-signpost.googlecode.com/files/signpost-core-1.2.1.1.jar

I created a button that had to be clicked to fire the first step of OAuth authentication with twitter. The code looks like this:

public class MyActivity extends Activity {
  private String CALLBACKURL = "app://twitter";
  private String consumerKey = "TwitterConsumerKey";
  private String consumerSecret = "TwitterConsumerSecret";
 
  private OAuthProvider httpOauthprovider = new DefaultOAuthProvider("https://api.twitter.com/oauth/request_token", "https://api.twitter.com/oauth/access_token", "https://api.twitter.com/oauth/authorize");
  private CommonsHttpOAuthConsumer httpOauthConsumer = new CommonsHttpOAuthConsumer(consumerKey, consumerSecret);
 
  @Override
  public void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
 
    ImageButton oauth = (ImageButton) findViewById(R.id.oauth_button);
    oauth.setOnClickListener(new View.OnClickListener() {
      public void onClick(View v) {
        try {
          String authUrl = httpOauthprovider.retrieveRequestToken(httpOauthConsumer, CALLBACKURL);
          Intent intent = new Intent(Intent.ACTION_VIEW, Uri.parse(authUrl));
          v.getContext().startActivity(intent);
        } catch (Exception e) {
          Log.w("oauth fail", e);
          Toast.makeText(v.getContext(), e.getMessage(), Toast.LENGTH_LONG).show();
        }
      }
    });
  }
}

The CALLBACK URL is called by Twitter when the user authorises the application (and therefore the request token). Usually it would be a HTTP URL but in this case we need to define a special URL which gets handled by our application.

The consumerKey and consumerSecret are values assigned by Twitter for the application.

The definition of the callback URL in the manifest file looks like this:

<activity android:name="MyActivity" android:label="@string/app_name" android:launchMode="singleInstance">
  ...
  <intent-filter>
    <action android:name="android.intent.action.VIEW"/>
    <category android:name="android.intent.category.DEFAULT"/>
    <category android:name="android.intent.category.BROWSABLE"/>
    <data android:scheme="app" android:host="twitter" />
  </intent-filter>
</activity>

We can change app://twitter to be anything we want but it needs to match what’s defined in the data element in our manifest file.

I found that I needed to define Callback URL in my application’s settings on Twitter otherwise I ended up getting this error:

oauth.signpost.exception.OAuthCommunicationException: Communication with the service provider failed: https://api.twitter.com/oauth/request_token
        at oauth.signpost.AbstractOAuthProvider.retrieveToken(AbstractOAuthProvider.java:214)
        at oauth.signpost.AbstractOAuthProvider.retrieveRequestToken(AbstractOAuthProvider.java:69)
...
 Caused by: java.io.FileNotFoundException: https://api.twitter.com/oauth/request_token
        at org.apache.harmony.luni.internal.net.www.protocol.http.HttpURLConnectionImpl.getInputStream(HttpURLConnectionImpl.java:521)
        at org.apache.harmony.luni.internal.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:258)
        at oauth.signpost.basic.HttpURLConnectionResponseAdapter.getContent(HttpURLConnectionResponseAdapter.java:18)
        at oauth.signpost.AbstractOAuthProvider.handleUnexpectedResponse(AbstractOAuthProvider.java:228)
        at oauth.signpost.AbstractOAuthProvider.retrieveToken(AbstractOAuthProvider.java:189)

I ended up just setting my callback URL on Twitter to the URL of this blog. It doesn’t seem to matter what you put the URL as since it’s going to be overridden by our callback URL anyway but it does need to be set.

The callback then gets handled by the following code in MyActivity…

public class MyActivity extends Activity {
  ...
 
  @Override
  protected void onNewIntent(Intent intent) {
    super.onNewIntent(intent);
 
    Log.w("redirect-to-app", "going to save the key and secret");
 
    Uri uri = intent.getData();
    if (uri != null && uri.toString().startsWith(CALLBACKURL)) {
 
        String verifier = uri.getQueryParameter(oauth.signpost.OAuth.OAUTH_VERIFIER);
 
        try {
            // this will populate token and token_secret in consumer
 
            httpOauthprovider.retrieveAccessToken(httpOauthConsumer, verifier);
            String userKey = httpOauthConsumer.getToken();
            String userSecret = httpOauthConsumer.getTokenSecret();
 
            // Save user_key and user_secret in user preferences and return
            SharedPreferences settings = getBaseContext().getSharedPreferences("your_app_prefs", 0);
            SharedPreferences.Editor editor = settings.edit();
            editor.putString("user_key", userKey);
            editor.putString("user_secret", userSecret);
            editor.commit();
 
        } catch (Exception e) {
 
        }
    } else {
        // Do something if the callback comes from elsewhere
    }
  }
}

…which makes another call to Twitter to get the user’s key and secret (the access token) which it then stored in shared preferences so we can use it in future without having to re-authenticate with Twitter.

We can then query Twitter like so:

HttpGet get = new HttpGet("http://api.twitter.com/1/statuses/home_timeline.json");
HttpParams params = new BasicHttpParams();
HttpProtocolParams.setUseExpectContinue(params, false);
get.setParams(params);
 
try {
  SharedPreferences settings = getContext().getSharedPreferences("your_app_prefs", 0);
  String userKey = settings.getString("user_key", "");
  String userSecret = settings.getString("user_secret", "");
 
  httpOauthConsumer.setTokenWithSecret(userKey, userSecret);
  httpOauthConsumer.sign(get);
 
  DefaultHttpClient client = new DefaultHttpClient();
  String response = client.execute(get, new BasicResponseHandler());
  JSONArray array = new JSONArray(response);
} catch (Exception e) { 
  // handle this somehow
}

Here we retrieve the user’s key and secret which we saved on the previous step and then set them on our OAuth consumer which we use to sign our request to Twitter.

There is a nice explanation of how OAuth works about half way down this StackOverFlow post, Eran Hammer-Lahav has a pretty good “beginner’s guide to OAuth” on his blog and the OAuth spec is surprisingly readable as well.

In the Android application that I’ve been playing around with I wrote a service which consumes the Twitter streaming API which I trigger from the app’s main activity like so:

public class MyActivity extends Activity {
    ...
    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
 
        Intent intent = new Intent(this, TweetService.class);
        startService(intent);
        ...
    }
}

Where TweetService is defined roughly like this:

public class TweetService extends IntentService {
    @Override
    protected void onHandleIntent(Intent intent) {
      // Twitter streaming API stuff goes here
    }
}

Unfortunately when I tried to deploy the app the service wasn’t starting and I got this message in the log:

01-01 03:10:31.758: WARN/ActivityManager(106): Unable to start service Intent { cmp=com.example/.TweetService }: not found

What I hadn’t realised is that the service needs to be specified in the AndroidManifest.xml file but not inside the activity definition:

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android" package="com.example" android:versionCode="1" android:versionName="1.0">
    <application android:label="@string/app_name">
        <activity android:name="MyActivity" android:label="@string/app_name" android:launchMode="singleInstance">
            <intent-filter>
                <action android:name="android.intent.action.MAIN"/>
                <category android:name="android.intent.category.LAUNCHER"/>
            </intent-filter>
        </activity>
        <service android:name="TweetService"></service>
    </application>
    <uses-permission android:name="android.permission.INTERNET"/>
</manifest>

After adding the service definition it works fine.

I have a bit of Java code for working out the final destination of a URL assuming that there might be one redirect which looks like this:

private String resolveUrl(String url) {
  try {
    HttpURLConnection con = (HttpURLConnection) (new URL(url).openConnection());
    con.setInstanceFollowRedirects(false);
    con.connect();
    int responseCode = con.getResponseCode();
 
    if (String.valueOf(responseCode).startsWith("3")) {
      return con.getHeaderField("Location");
    }
  } catch (IOException e) {
    return url;
  }
 
  return url;
}

I need to cast to HttpURLConnection on the first line so that I can make the call to setInstanceFollowRedirects which isn’t available on URLConnection.

I wanted to write some similar code in Clojure and my first thought was that I needed to work out how to do the cast, which I didn’t know how to do.

I then remembered that Clojure is actually dynamically typed so there isn’t any need – as long as the object has the method that we want to call on it everything will be fine.

In this case we end up with the following code:

(defn resolve-url [url]
  (let [con (.. (new URL url) openConnection)]
    (doall
     (.setInstanceFollowRedirects con false)
     (.connect con))
    (if (.startsWith (str (.getResponseCode con)) "3")
      (.getHeaderField con "Location")
      url)))

Which can be simplified to this:

(defn resolve-url [url]
  (let [con (doto (.. (new URL url) openConnection)
                  (.setInstanceFollowRedirects false)
                  (.connect))]
  (if (.startsWith (str (.getResponseCode con)) "3")
    (.getHeaderField con "Location")
    url)))

While I’ve been learning how to write an android application there’s been plenty of opportunities for me to go off shaving yaks, it’s pretty much Yakville Central.

Typically I’d end up spending hours trying to work out some obscure thing which I didn’t really need to know so I wanted to try and avoid that this time.

I started keeping a track of the ‘yak stack’ which I was currently following and mentally noting exactly where I was up to.

An example of a yak stack I kept while trying to authorise a user of the app with Twitter using OAuth is shown in the photo on the right hand side.

It ended up looking like this:

• Get the home timeline working
  • OAuth blowing up
    • Not actually capturing redirect back to app
      • Launch mode in Android manifest

Once I realise I’m heading down the stack I’ve been giving myself one pomodoro to try and dig myself one level up.

If I still haven’t managed to solve the problem I might keep going for one more pomodoro or just find another way around the problem.

I’m sure I’ll come across problems where I need to spend more than an hour trying to solve it but for now it’s working ok as a rule of thumb.

It’s definitely fun chasing yaks but I get to the end of the day and haven’t really achieved anything which isn’t fun.

A few weeks ago Chris Matts wrote an interesting blog post ‘the language of risk‘ in which he describes an approach he used to explain the processes his team uses to an auditor.

Why did the auditor like what I said?

Because I explained everything we did in terms of risk. When they asked for a “process”, I explained the risk the process was meant to address. I then explained how our different process addressed the risk more effectively.

This seems like a pretty cool idea to me and it got me thinking of the different ‘processes’ we’ve used in teams I’ve worked on and what risks they might be addressing:

• Pair Programming
  • Becoming dependent on one person with respect to knowledge of part of the code base.
  • Having someone new working on an area of the code that they don’t know well and making a mistake.
• Retrospective
  • Making the same mistakes repeatedly/working in a way that (indirectly) wastes money.
• Story Kick Off
  • Building the wrong thing
  • Solving the business problem in an inefficient way
  • Building something which is very difficult to test
• Stand Up
  • Someone getting stuck on something which someone else in the group might be able to help with.
  • People going down rabbit holes and getting stuck on things that don’t really matter
• Show Case
  • Building the wrong thing for too long
• Automated testing
  • The application regresses as new functionality is added
  • Humans make mistakes when manually going through scenarios

That’s just a first attempt at this, I’m sure others could come up with something better!

In coming up with the list I’ve been working from a process which I’ve seen used and trying to work out what risk that might be addressing.

Chris seems to look at risks/processes the other way around to i.e. we think about what risks we need to address and then work out whether we need a process to address it and if so which one.

Taking that approach would help to explain why some teams don’t necessarily need a lot of process – the risks might be catered for in different ways or maybe they just don’t exist in specific contexts.

For example a lot of risks around communication go away if the product owner and the team are sitting in the same physical location and can easily just turn and talk to each other if they have any questions.

Even with this new way of looking at risks/process I still think it’s useful to keep checking whether or not a process is still necessary because as our team/product changes the risks we face probably do as well.