Skip to content

Feed aggregator

Neo4j: Building a topic graph with Prismatic Interest Graph API

Mark Needham - Sat, 02/14/2015 - 01:38

Over the last few weeks I’ve been using various NLP libraries to derive topics for my corpus of How I met your mother episodes without success and was therefore enthused to see the release of Prismatic’s Interest Graph API

The Interest Graph API exposes a web service to which you feed a block of text and get back a set of topics and associated score.

It has been trained over the last few years with millions of articles that people share on their social media accounts and in my experience using Prismatic the topics have been very useful for finding new material to read.

The first step is to head to interest-graph.getprismatic.com and get an API key which will be emailed to you.

Having done that we’re ready to make some calls to the API and get back some topics.

I’m going to use Python to call the API and I’ve found the requests library the easiest library to use for this type of work. Our call to the API looks like this:

import requests
payload = { 'title': "insert title of article here",
            'body': "insert body of text here"),
            'api-token': "insert token sent by email here"}
r = requests.post("http://interest-graph.getprismatic.com/text/topic", data=payload)

One thing to keep in mind is that the API is rate limited to 20 requests a second so we need to restrict our requests or we’re going to receive error response codes. Luckily I came across an excellent blog post showing how to write a decorator around a function and only allow it to execute at a certain frequency.

To rate limit our calls to the Interest Graph we need to pull the above code into a function and annotate it appropriately:

import time
 
def RateLimited(maxPerSecond):
    minInterval = 1.0 / float(maxPerSecond)
    def decorate(func):
        lastTimeCalled = [0.0]
        def rateLimitedFunction(*args,**kargs):
            elapsed = time.clock() - lastTimeCalled[0]
            leftToWait = minInterval - elapsed
            if leftToWait>0:
                time.sleep(leftToWait)
            ret = func(*args,**kargs)
            lastTimeCalled[0] = time.clock()
            return ret
        return rateLimitedFunction
    return decorate
 
@RateLimited(0.3)
def topics(title, body):
    payload = { 'title': title,
                'body': body,
                'api-token': "insert token sent by email here"}
    r = requests.post("http://interest-graph.getprismatic.com/text/topic", data=payload)
    return r

The text I want to classify is stored in a CSV file – one sentence per line. Here’s a sample:

$ head -n 10 data/import/sentences.csv
SentenceId,EpisodeId,Season,Episode,Sentence
1,1,1,1,Pilot
2,1,1,1,Scene One
3,1,1,1,[Title: The Year 2030]
4,1,1,1,"Narrator: Kids, I'm going to tell you an incredible story. The story of how I met your mother"
5,1,1,1,Son: Are we being punished for something?
6,1,1,1,Narrator: No
7,1,1,1,"Daughter: Yeah, is this going to take a while?"
8,1,1,1,"Narrator: Yes. (Kids are annoyed) Twenty-five years ago, before I was dad, I had this whole other life."
9,1,1,1,"(Music Plays, Title ""How I Met Your Mother"" appears)"

We’ll also need to refer to another CSV file to get the title of each episode since it isn’t being stored with the sentence:

$ head -n 10 data/import/episodes_full.csv
NumberOverall,NumberInSeason,Episode,Season,DateAired,Timestamp,Title,Director,Viewers,Writers,Rating
1,1,/wiki/Pilot,1,"September 19, 2005",1127084400,Pilot,Pamela Fryman,10.94,"Carter Bays,Craig Thomas",68
2,2,/wiki/Purple_Giraffe,1,"September 26, 2005",1127689200,Purple Giraffe,Pamela Fryman,10.40,"Carter Bays,Craig Thomas",63
3,3,/wiki/Sweet_Taste_of_Liberty,1,"October 3, 2005",1128294000,Sweet Taste of Liberty,Pamela Fryman,10.44,"Phil Lord,Chris Miller",67
4,4,/wiki/Return_of_the_Shirt,1,"October 10, 2005",1128898800,Return of the Shirt,Pamela Fryman,9.84,Kourtney Kang,59
5,5,/wiki/Okay_Awesome,1,"October 17, 2005",1129503600,Okay Awesome,Pamela Fryman,10.14,Chris Harris,53
6,6,/wiki/Slutty_Pumpkin,1,"October 24, 2005",1130108400,Slutty Pumpkin,Pamela Fryman,10.89,Brenda Hsueh,62
7,7,/wiki/Matchmaker,1,"November 7, 2005",1131321600,Matchmaker,Pamela Fryman,10.55,"Sam Johnson,Chris Marcil",57
8,8,/wiki/The_Duel,1,"November 14, 2005",1131926400,The Duel,Pamela Fryman,10.35,Gloria Calderon Kellett,46
9,9,/wiki/Belly_Full_of_Turkey,1,"November 21, 2005",1132531200,Belly Full of Turkey,Pamela Fryman,10.29,"Phil Lord,Chris Miller",60

Now we need to get our episode titles and transcripts ready to pass to the topics function. Since we’ve only got ~ 200 episodes we can create a dictionary to store that data:

episodes = {}
with open("data/import/episodes_full.csv", "r") as episodesfile:
    episodes_reader = csv.reader(episodesfile, delimiter=",")
    episodes_reader.next()
    for episode in episodes_reader:
        episodes[int(episode[0])] = {"title": episode[6], "sentences" : [] }
 
with open("data/import/sentences.csv", "r") as sentencesfile:
     sentences_reader = csv.reader(sentencesfile, delimiter=",")
     sentences_reader.next()
     for sentence in sentences_reader:
         episodes[int(sentence[1])]["sentences"].append(sentence[4])
 
>>> episodes[1]["title"]
'Pilot'
>>> episodes[1]["sentences"][:5]
['Pilot', 'Scene One', '[Title: The Year 2030]', "Narrator: Kids, I'm going to tell you an incredible story. The story of how I met your mother", 'Son: Are we being punished for something?']

Now we’re going to loop through each of the episodes, call topics and write the result into a CSV file so we can load it into Neo4j afterwards to explore the data:

import json
 
with open("data/import/topics.csv", "w") as topicsfile:
    topics_writer = csv.writer(topicsfile, delimiter=",")
    topics_writer.writerow(["EpisodeId", "TopicId", "Topic", "Score"])
 
    for episode_id, episode in episodes.iteritems():
        tmp = topics(episode["title"], "".join(episode["sentences"]).json()
        print episode_id, tmp
        for topic in tmp['topics']:
            topics_writer.writerow([episode_id, topic["id"], topic["topic"], topic["score"]])

It takes about 10 minutes to run and this is a sample of the output:

$ head -n 10 data/import/topics.csv
EpisodeId,TopicId,Topic,Score
1,1519,Fiction,0.5798245566455255
1,2015,Humour,0.565154963605359
1,24031,Laughing,0.5587120401021765
1,16693,Flirting,0.5514098189505282
1,1163,Dating and Courtship,0.5487490108554022
1,2386,Kissing,0.5476185929151934
1,31929,Puns,0.5375100569837977
2,24031,Laughing,0.5670926949850333
2,1519,Fiction,0.5396488295397263

We’ll use Neo4j’s LOAD CSV command to load the data in:

// make sure the topics exist
LOAD CSV WITH HEADERS FROM "file:///Users/markneedham/projects/neo4j-himym/data/import/topics.csv" AS row
MERGE (topic:Topic {id: TOINT(row.TopicId)})
ON CREATE SET topic.value = row.Topic
// make sure the topics exist
LOAD CSV WITH HEADERS FROM "file:///Users/markneedham/projects/neo4j-himym/data/import/topics.csv" AS row
MERGE (topic:Topic {id: TOINT(row.TopicId)})
ON CREATE SET topic.value = row.Topic
// now link the episodes and topics
LOAD CSV WITH HEADERS FROM "file:///Users/markneedham/projects/neo4j-himym/data/import/topics.csv" AS row
MATCH (topic:Topic {id: TOINT(row.TopicId)})
MATCH (episode:Episode {id: TOINT(row.EpisodeId)})
MERGE (episode)-[:TOPIC {score: TOFLOAT(row.Score)}]->(topic)

We’ll assume that the episodes and seasons are already loaded – the commands to load those in are on github.

We can now write some queries against our topic graph. We’ll start simple – show me the topics for an episode:

MATCH (episode:Episode {id: 1})-[r:TOPIC]->(topic)
RETURN topic, r

Graph

Let’s say we liked the ‘Puns’ aspect of the Pilot episode and want to find out which other episodes had puns. The following query would let us find those:

MATCH (episode:Episode {id: 1})-[r:TOPIC]->(topic {value: "Puns"})<-[:TOPIC]-(other)
RETURN episode, topic, other

Graph  1

Or maybe we want to find the episode which has the most topics in common:

MATCH (episode:Episode {id: 1})-[:TOPIC]->(topic),
      (topic)<-[r:TOPIC]-(otherEpisode)
RETURN otherEpisode.title as episode, COUNT(r) AS topicsInCommon
ORDER BY topicsInCommon DESC
LIMIT 10
==> +------------------------------------------------+
==> | episode                       | topicsInCommon |
==> +------------------------------------------------+
==> | "Purple Giraffe"              | 6              |
==> | "Ten Sessions"                | 5              |
==> | "Farhampton"                  | 4              |
==> | "The Three Days Rule"         | 4              |
==> | "How I Met Everyone Else"     | 4              |
==> | "The Time Travelers"          | 4              |
==> | "Mary the Paralegal"          | 4              |
==> | "Lobster Crawl"               | 4              |
==> | "The Magician's Code, Part 2" | 4              |
==> | "Slutty Pumpkin"              | 4              |
==> +------------------------------------------------+
==> 10 rows

We could then tweak that query to get the names of those topics:

MATCH (episode:Episode {id: 1})-[:TOPIC]->(topic),
      (topic)<-[r:TOPIC]-(otherEpisode)-[:IN_SEASON]->(season)
RETURN otherEpisode.title as episode, season.number AS season, COUNT(r) AS topicsInCommon, COLLECT(topic.value)
ORDER BY topicsInCommon DESC
LIMIT 10
 
==> +-----------------------------------------------------------------------------------------------------------------------------------+
==> | episode                   | season | topicsInCommon | COLLECT(topic.value)                                                        |
==> +-----------------------------------------------------------------------------------------------------------------------------------+
==> | "Purple Giraffe"          | "1"    | 6              | ["Humour","Fiction","Kissing","Dating and Courtship","Flirting","Laughing"] |
==> | "Ten Sessions"            | "3"    | 5              | ["Humour","Puns","Dating and Courtship","Flirting","Laughing"]              |
==> | "How I Met Everyone Else" | "3"    | 4              | ["Humour","Fiction","Dating and Courtship","Laughing"]                      |
==> | "Farhampton"              | "8"    | 4              | ["Humour","Fiction","Kissing","Dating and Courtship"]                       |
==> | "Bedtime Stories"         | "9"    | 4              | ["Humour","Puns","Dating and Courtship","Laughing"]                         |
==> | "Definitions"             | "5"    | 4              | ["Kissing","Dating and Courtship","Flirting","Laughing"]                    |
==> | "Lobster Crawl"           | "8"    | 4              | ["Humour","Dating and Courtship","Flirting","Laughing"]                     |
==> | "Little Boys"             | "3"    | 4              | ["Humour","Puns","Dating and Courtship","Laughing"]                         |
==> | "Wait for It"             | "3"    | 4              | ["Fiction","Puns","Flirting","Laughing"]                                    |
==> | "Mary the Paralegal"      | "1"    | 4              | ["Humour","Dating and Courtship","Flirting","Laughing"]                     |
==> +-----------------------------------------------------------------------------------------------------------------------------------+

Overall 168 (out of 208) of the other episodes have a topic in common with the first episode so perhaps just having a topic in common isn’t the best indication of similarity.

An interesting next step would be to calculate cosine or jaccard similarity between the episodes and store that value in the graph for querying later on.

I’ve also calculated the most common bigrams across all the transcripts so it would be interesting to see if there are any interesting insights at the intersection of episodes, topics and phrases.

Categories: Blogs

Cross-Platform AutoMapper (again)

Jimmy Bogard - Fri, 02/13/2015 - 17:04

Building cross-platform support for AutoMapper has taken some…interesting twists and turns. First, I supported AutoMapper in Silverlight 3.0 five (!) years ago. I did this with compiler directives.

Next, I got tired of compiler directives, tired of Silverlight, and went back to only supporting .NET 4.

Then in AutoMapper 3.0, I supported multiple platforms via portable class libraries. When that first came out, I started get reports of exceptions that I didn’t think should ever show up, but there was a problem. MSBuild doesn’t want to copy referenced assemblies that aren’t actually being used, so I’d get issues where you’d reference platform-specific assemblies in a “Core” library, but your “UI” project that referenced “Core” didn’t pull in the platform-specific assembly.

So began a journey to force the platform-specific assembly to get copied over, no matter what. But even that was an issue – I went through several different iterations of this before it finally, reliably worked.

Unless you’re on Xamarin, which doesn’t support using this method (of PowerShell scripts) to run install scripts on Mac.

Then I had a GitHub issue from Microsoft folks asking for CoreCLR support. And with vNext projects, the project itself describes the platforms to support, including all files in the directory. Meaning I wouldn’t be picking and choosing which files should be in the assembly or not. So, we’re back to square one.

A new path

With CoreCLR and the vNext project style that is folder-based rather than scattershot, pick-and-choose file based, I could only get CoreCLR support working by using conditional compiler directives. This was already in AutoMapper a few places, but mainly in files between the platform specific assemblies. I’ve always had to do a little bit of this:

image

Not absolutely horrible, but now with CoreCLR, I need to do this everywhere. To keep my sanity, I needed to include every file in every project. Ideally, I could just have the one portable library, but that won’t work until CoreCLR is fully released. With CoreCLR, I wanted to just have one single project that built multiple platforms. vNext class libraries can do this out-of-the-box:

image

However, I couldn’t move all platforms/frameworks since they’re not all supported in vNext class projects (yet). I still had to have individual projects.

Back when I supported Silverlight 3 for the first time, I abandoned support because it was a huge pain managing multiple projects and identical files. With vNext project files, which just includes all files in a folder without doing any explicit adding, I could have a great experience. I needed that with my other projects. The final project structure looked like this:

image

In the root PCL project, I’ll do all of the work. Refactoring, coding, anything. All of the platform-specific projects will just include all the source files to compile. To get them to do this, however, meant I needed to modify the project files to include files via wildcard:

image

My projects automatically include *all* files within folders (I needed to explicitly specify individual folders for whatever reason). With this configuration, my projects now include all files automatically:

image

I just have to be very careful that when I’m adding files, I only do this in the core PCL project, where files ARE added explicitly. There seems to be strange behavior that if I added a file manually to a project with wildcard includes, all of the files will be explicitly added. Not what I’d like.

Ultimately, this greatly simplified the deployment story as well. Each dependency only includes the one, single assembly:

image

At the end of the day, this deployment strategy is best for the users. I don’t have to worry about platform-specific extension libraries, GitHub issues about builds breaking in certain environments or the application crashing in cloud platforms.

If I had to do it over again, I’m not unhappy with the middle step I took of platform-specific extension assemblies. It forced me to modularize whereas with pure compiler directives I could have accepted a spaghetti mess of code.

Eventually, I’d like to collapse all into one project, but until it’s supported, this seems to work for everyone involved.

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

Categories: Blogs

CSR, Knowledge, and a Student Conference

Dear Junior
At Omegapoint we have for a long time struggled with the question: How can we contribute towards society? Or, put otherwise, how can we pay back to the society to which we owe so much? To use the business lingo phrase: How do we exercise our Corporate Social Responsibility (CSR)?
There is the obvious non-answer: We already contribute to society though our business. We pay taxes without fuzzing or hiding. What we do for our customers must be good in a broader sense, otherwise they would not pay for it. So, obviously we contribute.
But our moral standard is somewhat higher than that kind of waterline mark.
Of course there is always the easy way out: Donate money to some beneficiary organisation. And of course we have done so. But it does not feel completely satisfactory. Anyone can donate spare money: should we not pay those money as salaries and let our employees donate at their own discretion? What could we as a company contribute with?
Also we have done pro bono work: helping NGOs with their web-sites and support systems. Also we have given pro bono aid and coaching to local startups. Of course this is better, but we were still not completely satisfied. Lots of other companies could do this, it did not feel unique to us.
Slowly has the realisation dawned on us. We define ourselves as a knowledge company with an attitude of openness. The epitome of this is our semi-annual two-days competence conference, affectionately named OPKoKo where we run presentations, workshops, and discussion - all powered by stunning colleagues and some occasional guest. This is the soul of Omegapoint, this is what we should share.
Sharing with whom became pretty obvious. At Omegapoint almost everyone has studied at a technical university, apart from some remarkable autodidacts. In Sweden university education has no tuition fee at any of the universities - it is funded through taxes instead. This means that even the best universities have a steady flow of talented young people from all corners society. Omegapoint as a company is build on this foundation. We should pay back to Swedish Academia, and its education of students in particular.
Once question phrased, answer was simple. This spring we will run our first student conference: Studentkonferens 2015 by Omegapoint Academy.
We have put together a one-day conference with three tracks: programming, security, and testing. We will bring our experts from the field to spend a day presenting, discussing and socialising with the students. 
We have tried to pick subjects that are as relevant to students as possible - not far-away is about "strategic architectural enterprise business process modelling", but rather about stuff they can relate to and hopefully use immediately. We ended up choosing "Craftsmanship" as the theme.
Now, "Craftsmanship" is a term that has been thrown around our community left and right, and I am definitely sceptical to some uses of the word. Still, I think the original message of craftsmanship holds: love the craft, pay attention to what matters, be pragmatic.
Yesterday we announced the conference to the public during the Datatjej2015 conference run by female-CS-student association Datatjej. This is not a coincidence. 
At Omegapoint we always hire on merit. The idea to select from non-relevant attributes as gender or ethnicity is completely foreign to us. However, we also do know that women have had (and still have) a harder time in our industry than men. Thus, we want to encourage initiatives that encourage young women to make a move in our industry. Hence, it makes sense to draw some attention to Datatjej2015 by selecting that event for announcing our own initiative.
Now, this is going to be fun. Also, I have been given the honour to present the opening keynote, presenting my view of Craftsmanship in software development. I am really looking forward to all of it.
Yours
  Dan
PS To pay credit where credit is due, I really must praise my colleagues Lotta Hammarström and Daniel Deogun. They where both part of the original idea, and have really worked hard to it come alive.


Categories: Blogs

The Fun Zone - The Sweet Spot Of Resilient Learning

Agile Thinks and Things - Oana Juncu - Fri, 02/13/2015 - 12:31
LEGO SERIOUS PLAY TrainingWe all learn all the time, and we enjoy  it . Learning is exciting !  ... when we succeed .  Remember when you first learned to ride a bike ? Wan't it a little bit like a victory whet you first succeeded to make the bike move without someone discreetly holding your ? The big secret of learning with joy , though, is that we at, we don't play mental games like conceptualisation or arguing . Scientific research have shown that we exclusively learn from acting , never from listed concepts . To remember what we learned , the key sweet secret is to have fun during the learning process. Here is what we can do about it .
Let's  Go And Play !
We love challenges. Or don't we ? If you think a little bit , there are challenges that we enjoy and others that we fear and want to avoid . What is the difference ? You might say, ones are not important , others are crucial and may change your life.  Think of a challenge from the second category that you faced and had a good outcome ? How did you feel after ?
Now think of other "unimportant" challenges that you successfully solved? How did you feel after ?
What was the difference of emotions ? Relieved vs excited ? Powerful vs joyful ? Proud vs ... proud ?
At the end there is little ( no ) difference between the  emotional experience of overcoming a challenge of stake and  "superficial" little challenge.
If you were asked to name a type of "low stake challenge" experience, you probably name "playing games".
"Low stake" set-up is an interesting environment for learning, because we are in a safe zone "by design".  As we remember the rewarding feeling of solving a challenge ,  the best way to create resilient learning is to put ourselves in a "low stake" environment  to acquire desired skills. This environment is will be the "learning playground" . We are ready to play games to learn.

Good Games For Fun Learning
ATD2014 - Playing for the Next Demo Episode of
Our Killing AppDeciding to go play games is not  - at least not always - enough !  We enjoy playing a game because it is a good game. We learn effectively form a game when it is a goo ( learning ) game .
What is a good game ? Here is what games theory says about  good games :

  • Have  a goal ( an outcome to reach)
  • Have clear rules ( if not players won't know how to reach the outcome ) 
  • Give a sense of control of the game to the participants
  • Give a sense of progress in the game .


The Fun Zone Of  Learning
Comfort Zone For Learning
From the LSP Certification Training The sweet spot of  fun learning is the right balance between the level of difficulty of the challenge  in respect with the skills you have. Did you enjoy a game that was way too complex for you from the beginning ? Would you play a video game at level 7 when you never played it ?
But what about starting at level 1 and playing only level 1 over time ? Do you have the same "fun level" over time ? When we enhance our skills , we expect higher challenges. Because challenges that   are just over our skill level are the one that give us the best quality entertainment.
We have a sense of progress. We learned . We played together. And playing IS collaboration.

The graph of the "Learning Fun Zone" was presented by the Lego Serious Play Trainers during my Facilitation Certification Training. Credits to Jean Semo and Marie-Christine Dupont for it. 


Linked Materials :

JanetMcGonigal : Reality is Broken 
Categories: Blogs

Can You Mandate Your Agile Transformation?

Leading Agile - Mike Cottmeyer - Fri, 02/13/2015 - 09:00

Well… it depends.

If you view agile as a system of beliefs, or a way of looking at the world, or as a culture your company is expected to adopt…I’d suggest that it’s impossible to mandate an agile transformation. There is no way to force people to believe in something they don’t believe in or to feel something they don’t feel.

If you view agile as a set of practices, or as a way of performing your day-to-day activities, or as a set of ceremonies and artifacts and roles that people are required to perform… I’d suggest that, while probably not impossible to mandate, at best you’ll get malicious compliance if you try.

If you view agile as a system of delivery predicated upon the notion of small cross-functional teams, and you mandate those teams have everything necessary to deliver a working, tested increment of the product… and you mandate the organization gives those teams extreme clarity around what you are asking them to build… and you mandate those teams deliver an increment of the product for inspection every couple of weeks, just so we can make sure they are on the right track, give them feedback, and validate they are making measurable progress against our business goals…

I’d suggest that it’s irresponsible NOT to mandate your agile transformation.

Once you mandate the right kind of agile transformation, now we can explore the wide palette of tools and techniques and practices that make that kind of system work, and we can invite the team to choose the tools and techniques and practices that work best for them in their particular context.

Once you mandate the right kind of agile transformation, and the team has everything they need to be successful, autonomy to make local decisions, and the safety to decide how to do the work and how much work can be done, you can then invite them to change their mind about what they believe.

Mandating an agile transformation and inviting people to participate are not mutually exclusive. We just have to be clear on what’s negotiable and what isn’t.

The post Can You Mandate Your Agile Transformation? appeared first on LeadingAgile.

Categories: Blogs

An open letter about unit tests

George Dinwiddie’s blog - Fri, 02/13/2015 - 04:33

An open letter to a programmer who thinks that code coverage by integration tests eliminates the need for unit tests.

Why do we want tests? Typically, we want functional or acceptance test to demonstrate that the system, as a whole, functions the way we (or “the business”) wants it to function. We want integration tests to demonstrate that the subsystems talk to each other properly, as often errors creep in with differing expectations at these major system boundaries. And we want unit or micro tests to demonstrate that the code works the way the programmer thinks it should work. This is the business view of tests.

From a personal point of view, as a programmer I want tests to make my life easier as a programmer. Tests, particularly unit tests, give me quick feedback when my changes to the code have violated previous expectations of the code. They let me know when I’ve accomplished the functionality that lead me to make changes. They let me reorganize the code without worrying about making a mistake, because the tests will immediately alert me if I do. They let me move quickly, because the tests can analyze the holes in my design much more quickly than I can. And they save me from lots of embarrassment that comes from delivering buggy code.

You might think that we only need one level of testing. As long as a line of code is covered by a test, why cover it again? In a perfect world, shouldn’t this be sufficient?

In a perfect world, we wouldn’t need to write tests at all. We’re not in a perfect world, and neither are our tests. Our functional tests can look at the whole system, but has a hard time exercising the edge conditions deep in the code. The fact that a line of code has been exercised by a test does not tell us that the line works properly under various conditions. Our unit tests can easily check the boundaries of a small piece of code, but can’t assure us that the pieces are connected together properly. We need multiple levels of testing to get the assurance we need.

In addition, tests have other attributes. Unit tests run much faster (much less than a second) than integration or functional tests. This lets us run them much more frequently (several times a minute) and quickly get feedback on the effect of our latest code change. They can also diagnose the errors much more precisely than larger scale tests. They can codify our assumptions about the way a piece of code works, so if someone else makes a change that breaks those assumptions it becomes immediately obvious. Done well, unit tests give us confidence to move fast and stay out of the debugger.

If you’re not getting these benefits from unit testing, call me and I’ll work with you on them. Life is too short to struggle with unit tests and not get the value they can offer.

Categories: Blogs

Python/gensim: Creating bigrams over How I met your mother transcripts

Mark Needham - Fri, 02/13/2015 - 01:45

As part of my continued playing around with How I met your mother transcripts I wanted to identify plot arcs and as a first step I wrote some code using the gensim and nltk libraries to identify bigrams (two word phrases).

There’s an easy to follow tutorial in the gensim docs showing how to go about this but I needed to do a couple of extra steps to get my text data from a CSV file into the structure gensim expects.

Let’s first remind ourselves what the sentences CSV file looks like:

$ head -n 15 data/import/sentences.csv  | tail
5,1,1,1,Son: Are we being punished for something?
6,1,1,1,Narrator: No
7,1,1,1,"Daughter: Yeah, is this going to take a while?"
8,1,1,1,"Narrator: Yes. (Kids are annoyed) Twenty-five years ago, before I was dad, I had this whole other life."
9,1,1,1,"(Music Plays, Title ""How I Met Your Mother"" appears)"
10,1,1,1,"Narrator: It was way back in 2005. I was twenty-seven just starting to make it as an architect and living in New York with my friend Marshall, my best friend from college. My life was good and then Uncle Marshall went and screwed the whole thing up."
11,1,1,1,Marshall: (Opens ring) Will you marry me.
12,1,1,1,"Ted: Yes, perfect! And then you're engaged, you pop the champagne! You drink a toast! You have s*x on the kitchen floor... Don't have s*x on our kitchen floor."
13,1,1,1,"Marshall: Got it. Thanks for helping me plan this out, Ted."
14,1,1,1,"Ted: Dude, are you kidding? It's you and Lily! I've been there for all the big moments of you and Lily. The night you met. Your first date... other first things."

We need to transform those sentences into an array of words for each line and feed it into gensim’s models.Phrase object:

import nltk
import csv
import string
from gensim.models import Phrases
from gensim.models import Word2Vec
from nltk.corpus import stopwords
 
sentences = []
bigram = Phrases()
with open("data/import/sentences.csv", "r") as sentencesfile:
    reader = csv.reader(sentencesfile, delimiter = ",")
    reader.next()
    for row in reader:
        sentence = [word.decode("utf-8")
                    for word in nltk.word_tokenize(row[4].lower())
                    if word not in string.punctuation]
        sentences.append(sentence)
        bigram.add_vocab([sentence])

We’re used nltk’s word_tokezine function to create our array of words and then we’ve got a clause to make sure we remove any words which are punctuation otherwise they will dominate our phrases.

We can take a quick peek at some of the phrases that have been created like so:

>>> list(bigram[sentences])[:5]
[[u'pilot'], [u'scene', u'one'], [u'title', u'the', u'year_2030'], [u'narrator_kids', u'i', u"'m", u'going', u'to', u'tell', u'you', u'an_incredible', u'story.', u'the', u'story', u'of', u'how', u'i', u'met_your', u'mother'], [u'son', u'are', u'we', u'being', u'punished', u'for', u'something']]

gensim uses an underscore character to indicate when it’s joined two words together and in this sample we’ve got three phrases – ‘narrator_kids’, ‘met_you’ and ‘an_incredible’.

We can now populate a Counter with our phrases and their counts and find out the most common phrases. One thing to note is that I’ve chosen to get rid of stopwords at this point rather than earlier because I didn’t want to generate ‘false bigrams’ where there was actually a stop word sitting in between.

bigram_counter = Counter()
for key in bigram.vocab.keys():
    if key not in stopwords.words("english"):
        if len(key.split("_")) > 1:
            bigram_counter[key] += bigram.vocab[key]
 
for key, counts in bigram_counter.most_common(20):
    print '{0: <20} {1}'.format(key.encode("utf-8"), counts)
 
i_'m                 4607
it_'s                4288
you_'re              2659
do_n't               2436
that_'s              2044
in_the               1696
gon_na               1576
you_know             1497
i_do                 1464
this_is              1407
and_i                1389
want_to              1071
it_was               1053
on_the               1052
at_the               1035
we_'re               1033
i_was                1018
of_the               1014
ca_n't               1010
are_you              994

Most of the phrases aren’t really that interesting and I had better luck feeding the phrases into a Word2Vec model and repeating the exercise:

bigram_model = Word2Vec(bigram[sentences], size=100)
bigram_model_counter = Counter()
for key in bigram_model.vocab.keys():
    if key not in stopwords.words("english"):
        if len(key.split("_")) > 1:
            bigram_model_counter[key] += bigram_model.vocab[key].count
 
for key, counts in bigram_model_counter.most_common(50):
    print '{0: <20} {1}'.format(key.encode("utf-8"), counts)
 
do_n't               2436
gon_na               1576
ca_n't               1010
did_n't              704
come_on              499
end_of               460
kind_of              396
from_2030            394
my_god               360
they_'re             351
'm_sorry             349
does_n't             341
end_flashback        327
all_right            308
've_been             303
'll_be               301
of_course            289
a_lot                284
right_now            279
new_york             270
look_at              265
trying_to            238
tell_me              196
a_few                195
've_got              189
wo_n't               174
so_much              172
got_ta               168
each_other           166
my_life              157
talking_about        157
talk_about           154
what_happened        151
at_least             141
oh_god               138
wan_na               129
supposed_to          126
give_me              124
last_night           121
my_dad               120
more_than            119
met_your             115
excuse_me            112
part_of              110
phone_rings          109
get_married          107
looks_like           105
'm_sorry.            104
said_``              101

The first 20 phrases or so aren’t particularly interesting although we do have ‘new_york’ in there which is good as that’s where the show is set. If we go further we’ll notice phrases like ‘my_dad’, ‘get_married’ and ‘last_night’ which may all explain interesting parts of the plot.

Having the data in the Word2Vec model allows us to do some other fun queries too. e.g.

>>> bigram_model.most_similar(['marshall', 'lily'], ['ted'], topn=10)
[(u'robin', 0.5474381446838379), (u'go_ahead', 0.5138797760009766), (u'zoey', 0.505358874797821), (u'karen', 0.48617005348205566), (u'cootes', 0.4757827818393707), (u'then', 0.45426881313323975), (u'lewis', 0.4510520100593567), (u'natalie.', 0.45070385932922363), (u'vo', 0.4189065098762512), (u'players', 0.4149518311023712)]
 
>>> bigram_model.similarity("ted", "robin")
0.51928683064927905
 
>>> bigram_model.similarity("barney", "robin")
0.62980405583219112
 
>>> bigram_model.most_similar(positive=['getting_married'])
[(u'so_glad', 0.780311107635498), (u'kidding', 0.7683225274085999), (u'awake', 0.7682262659072876), (u'lunch.', 0.7591195702552795), (u'ready.', 0.7372316718101501), (u'single.', 0.7350872755050659), (u'excited_about', 0.725479006767273), (u'swamped', 0.7252731323242188), (u'boyfriends', 0.7127221822738647), (u'believe_this.', 0.71015864610672)]
 
>>> bigram_model.most_similar(positive=['my_dad'])
[(u'my_mom', 0.7994954586029053), (u'somebody', 0.7758427262306213), (u'easier', 0.7305313944816589), (u'hot.', 0.7282992601394653), (u'pregnant.', 0.7103987336158752), (u'nobody', 0.7059557437896729), (u'himself.', 0.7046393156051636), (u'physically', 0.7044381499290466), (u'young_lady', 0.69412761926651), (u'at_bernie', 0.682607889175415)]

I’m not quite at the stage where I can automatically pull out the results of a gensim model and do something with it but it is helping me to see some of the main themes in the show.

Next up I’ll try out trigrams and then TF/IDF over the bigrams to see which are the most important on a per episode basis. I also need to dig into Word2Vec to figure out why it comes up with different top phrases than the Phrases model.

Categories: Blogs

Some Thoughts on Self-Organization in Agile Teams

Leading Agile - Mike Cottmeyer - Fri, 02/13/2015 - 01:04

I think some of us are getting it wrong when it comes to self-organization.

Self-organization doesn’t mean that the team doesn’t have managers or that they get to decide what problems to solve or what product to build.

Self-organization doesn’t mean that the team doesn’t have any tooling constraints, or architectural constraints, or budget constraints, or timing constraints.

Self-organization doesn’t mean that the team decides who is on the team, how big the team is, or how much money everyone on the team should make.

Self-organization DOES mean that once the team is formed, and given a problem to solve, and a set of constraints to operate within, the team gets to decide how the work is done.

While any given team, in any given company, may have have latitude to make any or all of these decisions, in my opinion, that is not what is meant by self-organization as it pertains to agile.

I’m interested in your thoughts.

The post Some Thoughts on Self-Organization in Agile Teams appeared first on LeadingAgile.

Categories: Blogs

Creating an Environment of Confidence

Illustrated Agile - Len Lagestee - Fri, 02/13/2015 - 00:45

When I enter an organization as a coach, I often find people (and sometimes whole teams or departments) suffering from low confidence. For individuals, this could stem from being in a new role, a history of not being given a chance to speak what is on their mind, or a mistreatment after they made a […]

The post Creating an Environment of Confidence appeared first on Illustrated Agile.

Categories: Blogs

Psssst! I Can Get You Fixed Cost AND Fixed Dates!!

Agile Management Blog - VersionOne - Thu, 02/12/2015 - 23:50

130614-110928-2486x1687

I have an offer you can’t refuse…

You don’t have to be afraid, just because I am Sicilian.

I am talking about Product Development here, not “garbage collection”.

I know it frustrates you that all this Agile stuff talks about uncertainty and fluffy stuff. I have a secret for you, however. It’s one of the most overlooked aspects of Agile. I will even let you in on this secret for absolutely FREE.

Here it goes…

In Scrum, there are fixed timeboxes or iterations we call Sprints. You probably knew that. However, what you probably didn’t realize is that if your Scrum Teams establish fanatical discipline and rigor around only releasing things that are in line with their strong and comprehensive Definition of Done every Sprint, you will have…

FIXED DELIVERY DATES!!!

What will undermine this is if they compromise on the various criteria in the DoD, effectively cutting corners and introducing risk into the product. Also, if they extend the Sprints, change the duration repeatedly, or have nonsensical practices like magical mystery Sprints where hardening, innovation, and planning suddenly take place then all bets are off in terms of having…

FIXED DELIVERY DATES!!!

So, let the Scrum Teams be responsible and empowered to make the critical decisions that no one else can truly and effectively make. They will make the product sound in accordance with changing customer needs and technological advancements by baking quality in, integrating along the way, practicing emergent design, improving by coming up with new ideas, and doing smaller increments of ongoing planning, as Scrum intended. The result will be…

FIXED DELIVERY DATES!!!

Now, we all know that a Development Team in Scrum is supposed to be 5-9 (7±2) people, right? If we use a blended rate of, say, $100k to represent the average salary for team members (including consultants), then we know for certain that a Development Team will cost $500-900k / year. Voila! We have…

FIXED COSTS!!

Now, we can figure out what a Sprint costs by doing simple division. Let’s say a Sprint is two weeks. That gives us ~$19,000-$35,000 / Sprint depending on the Development Team size. Further, let’s assume our releases are every 3 Sprints (6 weeks). Now we know that a release costs us ~$57,000-105,000. That’s a beautiful thing. That’s…

FIXED COSTS!!!

You can’t ask for more!!

No, I mean literally, you can NOT ask for more; like Fixed Scope, for instance. In order to get fixed costs and fixed delivery dates in Scrum, the trade-off here is that the Scope is flexible. This is good, don’t freak out.

Having flexible scope ensures that we are able to roll with the punches and change as customer needs change, technology changes, and most importantly, business priorities change. To help us with this, we want the Sprints to be as short as possible. If we have one week Sprints, then we can formulate smaller increments in planning and ultimately have very granular refinements in our strategy rather than very drastic course corrections which are costly.

We still have higher level elements of planning that map to overall strategy: Vision, Roadmap, Release level planning, and insisting upon a Sprint Goal for every Sprint. This helps to keep us on target and focused with our longer term strategy.

So, not having fixed scope is a good thing. We could still have releases that are structured around fixed scope instead of fixed delivery dates. But it’s simply not realistic or REAL WORLD to expect to have more than one element fixed, one element firm, and one element flexible from among Scope, Cost, and Time. Those who demand to have all three fixed (so-called “firm fixed price”) are best served in taking this up by seeking an audience with OZ in the Emerald City, since they are indeed in fantasy land…

So, there it is:
FIXED DELIVERY DATES and FIXED COSTS

Happy weekend!

Peace and blessings-

Daniel

Categories: Companies

Mixing Lean UX and Agile Development

Scrum Expert - Thu, 02/12/2015 - 21:39
Carbon Five has been using Agile XP from our very beginnings 14 years ago. Six years ago we started on a deep dive into Design Thinking inspired by collaborations with the Stanford d.school. We then extended those learnings, integrating Lean UX techniques, to help our clients focus the team’s development power in a direction more closely aligned to a viable product market fit. Using two specific case studies and learnings from several other projects, this presentation walks attendees through the weekly cadence used at Carbon Five during product development. This ...
Categories: Communities

3 Reasons IT Ops Uses Lean Flow (Kanban for DevOps part 1 of 3)

In this three-part series on Kanban for DevOps, Dominica DeGrandis, Director of Training and Coaching at LeanKit, explains three key reasons why IT Ops teams and those implementing a DevOps value chain use a lean flow approach to product development. Here’s part one. Reason #1: The work isn’t done until it’s working right in Production. There’s a joke in […]

The post 3 Reasons IT Ops Uses Lean Flow (Kanban for DevOps part 1 of 3) appeared first on Blog | LeanKit.

Categories: Companies

Setting a Good Example

TV Agile - Thu, 02/12/2015 - 21:28
To get the most out of Behavior Driven Development (BDD), Specification by Example (SBE) or Acceptance Test-Driven Development (ATDD), you need much more than a tool. You need high value specifications. How do we get the most out of our specification and test writing effort? How do we write testable scenarios that business people will […]
Categories: Blogs

SonarQube Java Analyzer : The Only Rule Engine You Need

Sonar - Thu, 02/12/2015 - 17:17

If you have been following the releases of the Java plugin, you might have noticed that we work on two major areas for each release: we improve our semantic analysis of Java, and we provide a lot of new rules.

Another thing you might have noticed, thanks to the tag system introduced by the platform last year, is that we are delivering more and more rules tagged with “bug” and “security”. This is a trend we’ll try to strengthen on the Java plugin to provide users valuable rules that detect real problems in their code, and not just formatting or code convention issues.

What you might wonder then is: where do we get the inspiration for those rules?  Well, for starters, the SonarSource codebase is mostly written in Java, and most SonarSource developers are Java developers. So in analyzing our own codebase we find some patterns that we want to detect, turn those patterns into rules, and provide the rules to our users. But that is not enough, and that is why we are taking inspiration from other rule engines, and more specifically FindBugs. We are in the process of deprecating FindBugs rules by rewriting them using our technology.

Our goal is that at some point in 2015 we’ll stop shipping the FindBugs plugin by default with the platform (we’ll still support it and provide it through the update center) because out of the box, the Java Plugin will provide at least as much (hopefully more!) value as FindBugs.

This might seem pretentious, but there are a couple of reasons we are moving in this direction:

  • This is a move we already made with PMD and Checkstyle (and we are still supporting the sonar-pmd-plugin and sonar-checkstyle-plugin).
  • FindBugs works only at the bytecode level: the analysis only runs on compiled classes. The Sonar Java Plugin works with both sources and bytecode, and is thus able to be more precise in its analysis, eliminating false positives and detecting patterns that cannot be detected by FindBugs.
    For instance consider the following code run against the Java Plugin rule “Identical expressions should not be used on both side of a binary operator”, which deprecates multiple FindBugs rules:

    //...
    if(a == a) { //self comparison
      System.out.println(“foo”);
    }
    if( 2+1*12 == 2+1*12 ) { //selfcomparison
      System.out.println(“foo”);
    }//...
    

The approach used by FindBugs, which relies only bytecode, will not be able to detect the second issue because the second if will be erased by the compiler and thus will not be visible in bytecode.

  • FindBugs project activity: The activity on the project is quite low and thus the value coming out of it does not come fast enough to satisfy our users.
  • Documentation: One thing we really value at SonarSource, and that we think has made our products great, is that for each issue we raise we provide a clear explanation of why we raised the issue and an indication of how to fix it. This is something that FindBugs clearly lacks in our view, and we are confident we can offer better value in this area.

As we reimplement the FindBugs rules, our goal is also to remove some useless  or outdated rules, merge close-in-meaning rules, and report fewer false positives that FindBugs does.

However, this is going to take some work: we are still one step behind FindBugs regarding an essential part of what makes it valuable, the Control Flow Graph (CFG). Briefly: CFG allows tracking the value of a variable through the execution paths your code. An example of its use is to be able to detect NullPointerException without executing the code. This feature is not implemented yet in the SonarQube Java Plugin, but a first version was shipped in the latest version (3.3) of the C/C++ plugin. It’s in the roadmap of the Java plugin to embed this feature and deprecate the FindBugs rules requiring it.

This rewriting of FindBugs rules has already started, with a huge effort on documenting and specifying them properly. Out of the 423 rules provided by FindBugs we have decided to reimplement 394, and have already specified replacements for 286. At the time of this writing, 157 rules have already been reimplemented using our own technology (so about 40% of the implementable rules).

Don’t get me wrong: FindBugs is a great tool providing a lot of useful feedback to Java developers. But within the year, we will be at a point in the development of the SonarQube Java plugin where we can deliver even better feedback, and detect issues that even FindBugs can’t find.

Categories: Open Source

Socrates Canaries, Tenerife, Spain, February 26 – March 1 2015

Scrum Expert - Thu, 02/12/2015 - 17:05
Socrates Canaries is the Spanish stage of the International Software Craftsmanship Gathering, a series of events for open-minded software developers who want to improve their craft and the software industry as a whole. The local Software Craftsmanship community in the Canary Islands organizes it. The agenda of the Socrates Canaries International Software Craftsmanship Gathering follows the rules of the open conference where the participants create themselves the schedule of the event. Proposals are presented during the event itself in the mornings, and voted by the participants right after. This event is ...
Categories: Communities

February Meetup Recap

Agile Ottawa - Thu, 02/12/2015 - 15:53
Here’s the play-by-play recap of Tuesday’s meetup, in case you missed it or want to take the fun back to your organization. Agile 101: Scrum Meetings: Myth or Fact “The Scrum Master should be in charge of running the Sprint … Continue reading →
Categories: Communities

Scrum und ERP-Systeme – das passt zusammen!

Scrum 4 You - Thu, 02/12/2015 - 09:22

In der Rolle der ScrumMaster und Coaches betreuen meine Kollegen und ich seit mehr als zwei Jahren intensiv diverse ERP-Projekte. Alle sind wir mittlerweile der Meinung, dass Scrum im SAP- bzw. ERP-Umfeld nicht nur funktioniert, sondern dass Scrum und ERP sogar eine besonders vorteilhafte Kombination ergeben können. Aber Moment, wie kommen wir überhaupt zu dieser Aussage?

Was sind die größten Herausforderungen, wenn es um ERP-Systeme geht:

  • Weil sie die Geschäftsprozesse unterstützen, sind ERP-System nicht weniger als das Rückgrat moderner Unternehmen.
  • ERP-Systemlandschaften nehmen oft gewaltige Dimensionen an und passen sich daher oft nur schwerfällig an geänderte Rahmenbedingungen des Marktes an.
  • Gleichzeitig lastet ein hoher Druck auf den IT-Abteilungen: Bei gleichbleibender Flexibilität für Erweiterung die Kosten weiter zu reduzieren.

In den 1990er- und 2000er-Jahren, in denen agile Ansätze populär wurden, war es gerade die Softwareentwicklung, die als Erste die Vorteile zu nutzen wusste. Heute führt bei Web-Anwendungen und eCommerce-Produkten kein Weg mehr an agilen Methoden vorbei. Anders sieht das hingegen bei ERP-Systemen aus, was auch die aktuellste Status Quo Agile 2014 Studie belegt. Doch woran liegt das? Ist es die Komplexität und die „Größe“ moderner ERP-Systemlandschaften? Ist es der Mix aus Einführung, Customizing und Programmierung, der Fragen zur Anwendbarkeit von Scrum aufwirft?

Während der Begleitung der Projekte konnten wir zahlreiche Stimmen und Meinungen von ERP-involvierten Personen einfangen. Wir haben versucht, die größten Bedenken in punkto Nutzung von Scrum in ERP-Projekten zu extrahieren. Drei der am öftesten genannten Bedenken und unsere Antwort darauf will ich hier anführen (die übrigen sind in unserem Whitepaper ERP nachzulesen):

„Wieso sollten bei standardmäßigen ERP-Einführungen agile Methoden verwendet werden? Es wird doch meistens nichts entwickelt, sondern die Software nur an die bestehenden Prozesse angepasst.“
ERP-Einführungen greifen in die Arbeitsweise vieler Mitarbeiter ein, und ob Standard oder eigene Entwicklung: Veränderungen bewirken Unsicherheit vor allem bei jenen Mitarbeitern, die tagtäglich die zukünftige Software nutzen müssen. Widerstand ist dann oft vorprogrammiert. Agile Methoden wie Scrum haben den klaren Vorteil, dass sie die Betroffenen, die Anwender, frühzeitig in den Entwicklungsprozess einbeziehen. Deren Feedback wird ernst genommen und sie bekommen so kein System übergestülpt, das nicht zu ihren Arbeitsabläufen passt. Einbeziehung steigert die Akzeptanz und führt so auch eher zu einem Projekterfolg.

„In ERP-Systemen kann nur schwer agil entwickelt werden, da alle Anforderungen am Anfang bekannt sein müssen. Darauf aufbauend können das Datenmodell und das Fachkonzept erstellt werden.“
Im Grunde ist es doch so: Die meisten ERP-Systeme unterstützen ihre Nutzer nicht auf die effizienteste Art und Weise, die systemtechnisch möglich wäre. Ursache dafür ist aus unserer Sicht das Festhalten an Anforderungen und Fachkonzepten, die als vollständig und unumstößlich betrachtet werden. Der Fachbereich als Anforderer kann sich zu Beginn eines Projekts meist gar nicht vorstellen, wie das IT-System den Prozess unterstützen soll. Das von der IT vorgelegte Fachkonzept wird durchgewinkt, ohne es komplett verstanden zu haben. So steuern einige Projekte auf nicht bedienbare Systeme zu, die anschließend mit unzähligen Change Requests an die Realität angepasst werden müssen. Scrum kann hier helfen, sich dem für den Anwender „richtigen“ System von Anfang an anzunähern. Denn was die Anwender tatsächlich brauchen, erkennen sie durch Ausprobieren. Das passiert durch das kontinuierliche Liefern in kurzen Zyklen. Die Nutzer haben etwas in den Händen, können es testen und Rückmeldung geben, was bereits passt und was noch angepasst oder weitert werden sollte.

„In ERP-Systemen ist es nur beschränkt möglich, alle zwei Wochen fertige Software mit zusätzlicher Funktionalität zu liefern und produktiv zu setzen. Es besteht die Gefahr, dass bestehende Funktionalität nicht mehr einwandfrei funktioniert und operative Prozesse blockiert werden.”
Scrum selbst ist keine Entwicklungsmethode, sondern eine Vorgehensweise, um komplexe Aufgaben in überschaubaren Teilgrößen zu zerlegen und umzusetzen. Im Zusammenhang mit agilen Management-Frameworks für die Softwareentwicklung haben sich aber auch agile Entwicklungspraktiken etabliert. Es geht darum, die Qualität des Quellcodes konstant hoch zu halten und Funktionalitäten mit automatisierten Tests kontinuierlich zu überprüfen. Langsam aber sicher setzen sich auch im ERP-Bereich intelligente Entwicklungsumgebungen und nützliche Tools (Eclipse Unterstützung, eCATT, Mock-Frameworks) durch, mit deren Hilfe die Stabilität eines gewaltigen ERP-Komplexes nachhaltig abgesichert werden kann.
Was sind also auf den Punkt gebracht die größten Vorteile von agilen Methoden im ERP-Umfeld?

  • Treffsichere Erfüllung von Kundenbedürfnissen, weil die Nutzer einbezogen werden und laufend Feedback zum aktuellen Entwicklungsstand geben.
  • Schnelle Lieferung durch kurze Entwicklungszyklen und transparente Kommunikation der Ergebnisse.

Mehr zur Frage, ob Scrum und ERP zusammenpassen, lesen Sie in unserem Whitepaper ERP.

Categories: Blogs

The Simplest Systems Thinking Exercise - How to Make Toast.

Agile Complexification Inverter - Thu, 02/12/2015 - 04:27
For many years one example of process thinking, resource gathering, requirements, implementation and acceptance criteria has been the exercise - make PB&J sandwiches.  I've done this with groups to discuss the simple task that we typically overlook as "experts" in sandwich making, that perhaps a 5 year old will find difficulty glossing over the - get bread - instruction.


Here's a TED Talk by Tom Wujec who has analyzed a similar exercise and draws some powerful conclusions from many iterations.  Watch it and then rethink the simple acts in your life.




So tell me again why group collaboration is important when you are solving wicked problems?

See Also:

Visual Thinking


Categories: Blogs

You Don’t Know Jack (or Jill)

Tyner Blain - Scott Sehlhorst - Wed, 02/11/2015 - 15:25

Candles

You’ve got some shiny new segmentation data about prospective customers; how much they earn, where they are located, how old they are. How does that help you make decisions about your product? You know this information, but you don’t really know your audience, or why they might become your customers.

Customer Needs Are A Function of Intent

Healthy Food

Last month we looked at a single example of a customer expressing their wants instead of their needs – as most customers do.  The main point is to keep in mind, that what a customer “wants” is really the satisfaction of a need – which reflects an intent. In last month’s example, the desire to sort a list is really a reflection of a need to select healthy food items, arising from an intent to be healthy.

This perspective is useful when thinking about your product, and questioning / prioritizing investment in particular features.  Collecting these individual “what is the underlying goal?” insights informs your understanding of your market.

When you don’t know your customer’s intent, you don’t know your customer.

Kano for Product Managers

Who cares about ease of use?

I did a presentation in 2009 for the Product Management View, on applying Kano analysis as a product manager. Kano helps you understand the nature of value associated with features and incremental improvements in features.  Good stuff.

As a product manager, the main benefit of using this approach comes from appreciating that any given feature – like ease of use of an upright vacuum – will be of greater or lesser importance to different users.

Combine these two ideas (understanding the underlying goal, and realizing that different users value that goal differently) to get an understanding of what is important to whom, and why.  This definitely helps, but still leaves you with a bit of a mess, as you realize that not everyone wants every thing.

Segmentation Based On Intent

acrylic hammer hitting nail

On a walk the other day, I was listening to Brian Shea‘s interview with Susan Baier of Audience Audit, who really hit the nail on the head.  Susan helps agencies understand the intent of their client’s customers and prospects.  In the interview, Susan describes this in terms of attitude, opinions, and motivations.  My inner monologue was rattling off intent, goals, context, jobs-to-be-done.  Different language, describing the exact same thing.  This is a great listen for any product manager or marketer, who is responsible for, or relies upon, an understanding of their market.

Susan provides a great example, describing work she did for one of her clients, in helping them understand the customers for scented candles.  Check out the interview if you want more specifics, but here’s one aspect I remember (paraphrasing, and all mistakes are mine):

Some people buy scented candles because the aroma helps them manage their mood (relaxation / aromatherapy).  Their intent, when making a purchase decision, is to become relaxed – and duration of scent is an important criteria for satisfying their needs.

Some people buy scented candles because they are decorating.  Their goal is aesthetic – and often they would desire a completely unscented candle, because it is part of an ensemble achieving a particular appeal, and introducing an additional scent would detract from the appeal.

Some people buy scented candles as gifts for others.  Their buying choices are influenced by the packaging (“Oh what a lovely gift!”) and the brand perception (“Wow, you really splurged, getting me ‘the best,’ thank you!”) because their job is to give a great gift.

All three of the people above could make $120,000 per year, be 43 years old, and live in the same subdivision; they could all be (otherwise) identical members of any demographic analysis.

This is exactly the kind of thing that matters for us as product managers.

Companies Have Identities Too

I’m not talking about the corporation as a legal “person” thing we do here in the USA.  Companies are driven by goals and intents.  I’ll save a discussion of Bowman Clocks for another post, and for now will contrast two hypothetical companies.

Two companies are competing in the hair-cutting market.  They are both opening chains of salons, but there is a big difference in terms of the strategies the companies are pursuing – specifically, “who” the companies are trying to be.  One company is trying to be the best low-cost and low-value provider (cheap cuts).  The other company is trying to be the best high-cost and high-value provider (a pampered experience).  Both companies need to do many of the same things, and each company will value different investments differently.

Just as different candle-buyers have different goals, so do these hair-cutting businesses.  Imagine you’re selling software to these hair cutters, and you need to know what to prioritize for the next release of your software.

  • The low-cost salon would love to reduce the time spent scheduling appointments – maybe you could build an online reservation / check-in capability.  This increases their throughput and/or reduces their cost structure.
  • The high-value salon would love to “remember” how best to pamper their clients on return visits, so building out some capabilities for the stylist to surreptitiously glance at his phone and “remember” that Fifi the Pomeranian is recovering from surgery. This increases their word-of-mouth referrals by improving the customer experience, allowing them to grow their customer base or charge higher prices.

While the concept of identifying intent-based personas is easy to describe in terms of individual consumers (B2C), it applies just as effectively when companies are your customers (B2B).  Don’t make the mistake of assuming that intent only matters for consumer products.

Connecting the Dots

When we connect the dots from the sections above, to create a successful product, you must solve the right problems for the right users. Intent is a useful lens for identifying users (or buyers) who will value the same solution to the same problems.

Intent-based market-segmentation drives a tighter, better product.

That’s it in a nutshell.  Thanks again Brian and Susan for inspiring this article.

 

 

Categories: Blogs

Mini XP Day Benelux 2015

April 3, 2015 in Mechelen, Belgium

Each spring we rerun the best sessions of the previous XP Days Benelux at the “Mini XP Day“.

This year we have 11 sessions that cover:

  • Improving testing, design and coding skills by “programming people”
  • Honing your Test Driven Development skills
  • Recruitment techniques
  • Going from idea to business value
  • Using feedback loops in Scrum
  • Improving the way you work and the way you work with others
  • Managing agile transitions
  • How students and teachers apply “EDUScrum” in school
  • Defeating the dark forces that hold you and your team back from taking action

There’s something for everybody who’s interested in agile, development and systems thinking.

Don’t take our word for it

Have a look at the  feedback from participantsphotos and articles about the conference.

Don’t wait too long to register for the conference, as it usually sells out.

See you there!

XP Days sketchnote

Categories: Blogs

Knowledge Sharing


SpiraTeam is a agile application lifecycle management (ALM) system designed specifically for methodologies such as scrum, XP and Kanban.