Skip to content

Blogs

Understanding the Indian 'Yes'

Scrum Breakfast - Mon, 03/02/2015 - 12:12
My experience with the word 'yes' in India is that it doesn't seem to mean what I think it means. And the word 'no' does not seem to exist.Doing business with India for me has always been a somewhat strange experience. As native speaker of North American English, there is a language barrier. Yes, it's English, but.... I was never really sure when negotiating with my business partners whether they were really going to deliver what they said they would. Often there would be substantial differences between what I thought we had agreed to and what was actually delivered.

Since I agreed to a four city tour in November of Scrum Trainings, I thought this would be an excellent opportunity to explore what yes really means, and why it remains such a stumbling block. What did I find out about 'yes' and other aspects of Indian culture?
Raising the questionMy Scrum courses devote substantial attention to working agreements. At the start of each course, I facilitate a discussion among the participants. “What agreements do we need among ourselves so that we can work together effectively between now and the end of the course?” The participants propose their topics, and then agree to concrete actionable proposals. I might add a topic or two. Sometimes we have to modify the proposal before we can come to an agreement that everyone can really support.

Agreements among the participants make for an effective course and agreements among colleagues are an excellent tool for transforming corporate culture. You can't change corporate culture directly, but you can influence your meeting culture quite strongly, simply by making working agreements with your colleagues. This in turn influences and changes your corporate culture.

Nicolas Jene, one of my participants last October was retaking the CSPO course he had taken a year ago. Back then, we had made a number of working agreements, including being on time and ready to go after breaks, devices off or on silent, no crosstalk, the attention protocol, how to deal with questions, etc.
People are the same everywhere...Nicolas noticed that the working agreements this time around were essentially the same as the agreements as in his last course, so he asked, 'Are the working agreements always the same? Even in different cultures?” I thought back to my the last year of courses in Switzerland, in Russia, in Portugal, in Italy, and in Texas, and replied, 'yes, pretty much. But I am not sure how that will be when I get to India, because the culture there is quite different compared to where I have been before.

So in all of my India courses, I facilitated the same discussion. What do we need to work together effectively?

All of my participants in India asked for and agreed to more or less the same the agreements, just like at every other place else I have been to. They did have some novel suggestions, but the topics and actual agreement were essentially the same.

My conclusion? People are the same everywhere. They want the same things from each other. They need the same things to work effectively: Focus, Commitment, Courage, Openness and Respect. Sound familiar? These are the Scrum values. You find all of them reflected in people's working agreements.
But cultures are differentIf people are the same everywhere, does that mean everyone thinks alike? Of course not! Case in point: during my last product owner class, I showed Henrik Kniberg's fantastic video “Agile Project Ownership in a Nutshell” (watch it here). One highlight of this video is the importance of saying 'No.' “Pat [the product owner] practices [saying No] every day in front of a mirror.” After watching the video, the class wrote down their “Eureka!” moments of what they would take home from the video. Most of the participants found that 'No!' was a highlight of the film. “We are allowed to say no!?” “We're expected to say no!?” “Wow!”

One person, of Indian origin, had a completely different lesson from the video: “Sometimes it is better to say yes and prioritize down.” What?! That was exactly the opposite of what Henrik's video was trying to say, or so I thought. I even asked him about it. Yes, that was his a-ha! moment. After getting past my disbelief, I reflected on why his answer might have been different from mine and the others.

If the inputs seem the same, but the outputs are different, then there must be some other inputs I don't know about. I call these inputs 'culture' and I decided I wanted to explore that further during my trip to India.
Why is 'no' so difficult to say?At each course in India, during the section on working agreements, I introduced the topic of yes and no, and explained why I thought there was an issue. My participants agreed that this misunderstanding occurs quite regularly when working with western companies. So I invited the participants to discuss in pairs why saying no is so difficult, then report back on to the group.

To my surprise, there was no hesitation to discuss the issue (though I had done a lot to lower the social risks in the room before asking question). The opinions of all four classes – over 100 people – centered on three issues:

  • Our upbringing discourages us from saying no. It is impolite to say no.
  • If I say no (to the customer), my company might lose the business.
  • If I say no (to a manager), I might lose reputation and risk losing my job.

In short, it's about fear. It's about the challenges of living in a developing nation. It's about the fear of becoming part of the 60% of the population that lives on $2/day or less. Saying yes is an accommodation. You may lose the business or your job later, but you do have it today, and that is perceived as much lower risk than saying no right away. Given that upbringing appeared so often in the list of reasons, it looks like the accommodation is deeply ingrained in the society.
What does 'yes' really mean?What does 'yes' really mean in India? It depends. It might really mean yes. It might also mean “I'm saying yes to satisfy your expectation” (but without committing to actually deliver on the expectation). More likely, it means, “I'm going to do my best, but there may be outside forces that I cannot control which will prevent me from succeeding.”
How to bridge cultural differences?I don't pretend to have the definitive answer to this question. I found being in India, talking face to face with people and learning to recognize their body language were both very helpful. I found it easier to work with people while I was there than over the phone from Switzerland.

Is there really a conflict between being polite and setting realistic expectations? I think this is a false dilemma. Henrik Kniberg pointed out that Autonomy and Alignment are not opposites, but independent of one another. People can be aligned and autonomous. Steve Biddulph wrote that parents can be both firm and loving with their children. How can you set realistic expectations and be polite? I think that is a fair question for any team to ask of itself.
Conflict avoidance is not evilMany facilitation techniques avoid using the word No. In a retrospective, we first identify possible topics for improvement. Then we identify the most important items, and say yes to a few of them. We don't actually say no to anything. Just yes. Dot-voting is classic consensus-building tool. Identify the most important, start there. And Rod Collins' Work-Thru also focus on identifying where we agree, so we can start here, and deferring points of conflict until later, when trust is higher and the situation is clearer. (Read more about Work-Thru's...)

Are there patterns here which could be applied for other kinds of collaboration? I have seen evidence that working agreements can be extremely effective at transforming corporate culture. Can working agreements be the basis for bridging cultural differences within a project? I believe so. You create your own culture through your working agreements (among other things – good stories are vital!) within your project so that you and your on-shore and off-shore collaborators can work together effectively.

Finally I believe that one day, people in India will let go of their fear. It is not weakness that makes people afraid, it is fear that makes them weak. India has strong abilities and great prices. There will always be a demand for that combination. If this customer gets away, there is always the next one. Once people realize they don't need to be afraid, they will have the courage to say no and to say a yes that really means, Yes!
Categories: Blogs

Bend the Spoon

Leading Agile - Mike Cottmeyer - Mon, 03/02/2015 - 11:00

Bend the spoon is a phrase we use quite a bit here at LeadingAgile. I don’t want to hear what’s happening, I want to hear what we need to make happen… and what we are doing to make it happen. I don’t want to hear why we can’t do something, I want to talk about what we are doing to make reality conform to our will.

If it’s impossible… bend the spoon.

Some people are wired to bend the spoon, some are not. When I walk down a busy sidewalk, I kinda expect traffic to flow around me. When my wife walks down the same sidewalk, she dodges the oncoming traffic. My wife sees the world as a set of rules that have to be followed. I think that rules are made to be broken.

If you believe you can bend the spoon in your organization, you are going to have a much greater chance of actually making change. If you see your organization as a set of immutable laws, laws set in motion by powers that are beyond your control, it’s going to be much more difficult.

Bending the spoon isn’t about imposing your will on the organization… it’s about believing that change is possible. That somehow, someway… all the crap that is getting in the way of really adopting agile can be changed. If you don’t believe you can change your organization and decide to change agile to accommodate dysfunction… you are going to fail.

Bend the spoon.

The post Bend the Spoon appeared first on LeadingAgile.

Categories: Blogs

Python: scikit-learn – Training a classifier with non numeric features

Mark Needham - Mon, 03/02/2015 - 09:48

Following on from my previous posts on training a classifier to pick out the speaker in sentences of HIMYM transcripts the next thing to do was train a random forest of decision trees to see how that fared.

I’ve used scikit-learn for this before so I decided to use that. However, before building a random forest I wanted to check that I could build an equivalent decision tree.

I initially thought that scikit-learn’s DecisionTree classifier would take in data in the same format as nltk’s so I started out with the following code:

import json
import nltk
import collections
 
from himymutil.ml import pos_features
from sklearn import tree
from sklearn.cross_validation import train_test_split
 
with open("data/import/trained_sentences.json", "r") as json_file:
    json_data = json.load(json_file)
 
tagged_sents = []
for sentence in json_data:
    tagged_sents.append([(word["word"], word["speaker"]) for word in sentence["words"]])
 
featuresets = []
for tagged_sent in tagged_sents:
    untagged_sent = nltk.tag.untag(tagged_sent)
    sentence_pos = nltk.pos_tag(untagged_sent)
    for i, (word, tag) in enumerate(tagged_sent):
        featuresets.append((pos_features(untagged_sent, sentence_pos, i), tag) )
 
clf = tree.DecisionTreeClassifier()
 
train_data, test_data = train_test_split(featuresets, test_size=0.20, train_size=0.80)
 
>>> train_data[1]
({'word': u'your', 'word-pos': 'PRP$', 'next-word-pos': 'NN', 'prev-word-pos': 'VB', 'prev-word': u'throw', 'next-word': u'body'}, False)
 
>>> clf.fit([item[0] for item in train_data], [item[1] for item in train_data])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/markneedham/projects/neo4j-himym/himym/lib/python2.7/site-packages/sklearn/tree/tree.py", line 137, in fit
    X, = check_arrays(X, dtype=DTYPE, sparse_format="dense")
  File "/Users/markneedham/projects/neo4j-himym/himym/lib/python2.7/site-packages/sklearn/utils/validation.py", line 281, in check_arrays
    array = np.asarray(array, dtype=dtype)
  File "/Users/markneedham/projects/neo4j-himym/himym/lib/python2.7/site-packages/numpy/core/numeric.py", line 460, in asarray
    return array(a, dtype, copy=False, order=order)
TypeError: float() argument must be a string or a number

In fact, the classifier can only deal with numeric features so we need to translate our features into that format using DictVectorizer.

from sklearn.feature_extraction import DictVectorizer
 
vec = DictVectorizer()
X = vec.fit_transform([item[0] for item in featuresets]).toarray()
 
>>> len(X)
13016
 
>>> len(X[0])
7302
 
>>> vec.get_feature_names()[10:15]
['next-word-pos=EX', 'next-word-pos=IN', 'next-word-pos=JJ', 'next-word-pos=JJR', 'next-word-pos=JJS']

We end up with one feature for every key/value combination that exists in featuresets.

I was initially confused about how to split up training and test data sets but it’s actually fairly easy – train_test_split allows us to pass in multiple lists which it splits along the same seam:

vec = DictVectorizer()
X = vec.fit_transform([item[0] for item in featuresets]).toarray()
Y = [item[1] for item in featuresets]
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.20, train_size=0.80)

Next we want to train the classifier which is a couple of lines of code:

clf = tree.DecisionTreeClassifier()
clf = clf.fit(X_train, Y_train)

I wrote the following function to assess the classifier:

import collections
import nltk
 
def assess(text, predictions_actual):
    refsets = collections.defaultdict(set)
    testsets = collections.defaultdict(set)
    for i, (prediction, actual) in enumerate(predictions_actual):
        refsets[actual].add(i)
        testsets[prediction].add(i)
    speaker_precision = nltk.metrics.precision(refsets[True], testsets[True])
    speaker_recall = nltk.metrics.recall(refsets[True], testsets[True])
    non_speaker_precision = nltk.metrics.precision(refsets[False], testsets[False])
    non_speaker_recall = nltk.metrics.recall(refsets[False], testsets[False])
    return [text, speaker_precision, speaker_recall, non_speaker_precision, non_speaker_recall]

We can call it like so:

predictions = clf.predict(X_test)
assessment = assess("Decision Tree", zip(predictions, Y_test))
 
>>> assessment
['Decision Tree', 0.9459459459459459, 0.9210526315789473, 0.9970134395221503, 0.9980069755854509]

Those values are in the same ball park as we’ve seen with the nltk classifier so I’m happy it’s all wired up correctly.

The last thing I wanted to do was visualise the decision tree that had been created and the easiest way to do that is export the classifier to DOT format and then use graphviz to create an image:

with open("/tmp/decisionTree.dot", 'w') as file:
    tree.export_graphviz(clf, out_file = file, feature_names = vec.get_feature_names())
dot -Tpng /tmp/decisionTree.dot -o /tmp/decisionTree.png


The decision tree is quite a few levels deep so here’s part of it:

DecisionTreeSection

The full script is on github if you want to play around with it.

Categories: Blogs

Etikettenschwindel oder was Führung ausmacht

Scrum 4 You - Mon, 03/02/2015 - 08:28

Es ist immer wieder faszinierend zu beobachten, wie intensiv die Auseinandersetzung mit dem Thema Führung in Unternehmen seit vielen Jahrzehnten geführt wird. Mal sind Führung und Hierarchie die Buhmänner für Ineffektivität, Demotivation und das Elend in Unternehmen und Organisationen, mal sollten sie ganz beseitigt werden, mal spricht dann doch wieder viel dafür, dass es ohne Führung einfach nicht geht. Deutlich erkennbar gibt es die Tendenz, den Begriff Führung und damit seine Funktion durch freundliche Etiketten zu ersetzen. Da ist die Führungskraft mal der Coach, mal der Gastgeber, mal der Servant Leader etc. und ich bin sicher, es werden noch mehr fantasievolle Ersatzbegriffe dazukommen. Um mit Herbert Grönemeyer zu sprechen: „Was soll das?“

Auch der alte Fritz war ein Servant Leader

Zweifellos können solche Sprachbilder und Metaphern sinnvoll und nützlich sein. Sie differenzieren ein frederick-the-great-258986_1280Thema und beschreiben gut nachvollziehbar besondere Charakteristika eines Phänomens. Auf Führung bezogen sind es häufig Elemente, die aktuell als nicht ausreichend vorhanden oder zeitgeistig, aber als zwingend notwendig gesehen werden. So gibt z.B. ein Coach nichts vor, er begleitet und aktiviert Ressourcen. Er hat so gut wie keine Macht, sondern setzt auf Vertrauen und höchste Freiwilligkeit. Der Gastgeber seinerseits lädt ein und stellt Leckerli auf den Tisch (Motivation), anstatt zu bestimmen. Er macht Angebote und bietet guten Service. Der Servant Leader ist in erster Linie Diener, orientiert sich stark an den Bedürfnissen der Geführten, er ist demütig und zurückhaltend. Dieses Etikett lässt sich bis zu Friedrich II. von Preußen zurückverfolgen, der postulierte: „Der Herrscher ist der erste Diener des Staates.“ Gerade dieses Beispiel zeigt die Grenzen des Gedankens in Bezug auf Führung: Denn der alte Fritz konnte seine Armeen gleichzeitig sehr direktiv führen, seine Vorhaben – wenn auch aufgeklärt – absolutistisch durchsetzen und den Diener ganz weit hintanstellen.

Führung ist in erster Linie Führung und sollte Führung bleiben. Gerade in den komplexen Prozessen der Selbstorganisation eines Systems ist Führung in ihrer hierarchischen Funktion das „Herz“ und damit der ursprünglichste Sinn und Nutzen für die geführten Menschen. Dazu gehört z.B. die legitimierte Macht, die Notwendigkeit, Settings zu schaffen und auch nichtdemokratische Entscheidungen treffen zu können.

Ich plädiere dafür, Menschen in disziplinarischen oder lateralen Führungsfunktionen nicht dadurch zu schwächen, dass man die ursprünglichste Komponente von Führung klein macht, verniedlicht oder gar negiert. Nichts spricht allerdings dagegen, Elemente anderer Rollen und Funktionen zu übernehmen und so für Führung ein zeitgemäßes, flexibles und unterstützendes Werte-, Einstellungs-, und Handlungsspektrum anzubieten.

Wenn die Welt agiler wird, wenn neue Generationen von Mitarbeitern in den Arbeitsprozess eintauchen, braucht es das eindeutige Etikett „Führung“. Und es braucht Führungskräfte, die sich zum Führen legitimiert fühlen, wo es im Sinne von Selbstorganisation angesagt erscheint.

Genau deswegen heißt das Buch, das ich mit Boris Gloger veröffentlicht habe, nicht Selbstorganisation braucht Coaches, Gastgeber oder Servant Leader, sondern „Selbstorganisation braucht Führung“.

Categories: Blogs

The Great Positive Thinking Quotes Collection

J.D. Meier's Blog - Mon, 03/02/2015 - 03:28

I revamped my positive thinking quotes collection on Sources of Insight to help you amp up your ability to generate more positive thoughts. 

It’s a powerful one.

Why positive thinking?

Maybe Zig Ziglar said it best:

“Positive thinking will let you do everything better than negative thinking will.”

Positive Thinking Can Help You Defeat Learned Helplessness

Actually, there’s a more important reason for positive thinking: 

It’s how you avoid learned helplessness.

Learned helplessness is where you give up, because you don’t think you have any control over the situation, or what happens in your life.  You explain negative situations as permanent, personal, and pervasive, instead of temporary, situational, and specific.

That’s a big deal.

If you fall into the learned helplessness trap, you spiral down.  You stop taking action.  After all, why take action, if it won’t matter.  And, this can lead to depression.

But that’s a tale of woe for others, not you.   Because you know how to defeat learned helplessness and how to build the skill of learned optimism.

You can do it by reducing negative thinking, and by practicing positive thinking.   And what better way to improve your positive thinking, than through positive thinking quotes.

Keep a Few Favorite Positive Thinking Quotes Handy

Always keep a few positive thinking quotes at your fingertips so that they are there when you need them.

Here is a quick taste of a few of my favorites from the positive thinking quotes collection:

“A positive attitude may not solve all your problems, but it will annoy enough people to make it worth the effort.” –  Herm Albright

"Attitudes are contagious.  Are yours worth catching?" — Dennis and Wendy Mannering

"Be enthusiastic.  Remember the placebo effect – 30% of medicine is showbiz." — Ronald Spark

"I will not let anyone walk through my mind with their dirty feet." — Mahatma Gandhi

"If the sky falls, hold up your hands." – Author Unknown

Think Deeper About Positivity By Using Categories for Positive Thinking

But this positive thinking quotes collection is so much more.  I’ve organized the positive thinking quotes into a set of categories to chunk it up, and to make it more insightful:

Actions
Adaptability and Flexibility
Anger and Frustration
Appreciation and Gratitude
Attitude, Disposition, and Character
Defeat, Setbacks, and Failures
Expectations
Focus and Perspective
Hope and Fear
Humor
Letting Things Go and Forgiveness
Love and Liking
Opportunity and Possibility
Positive Thinking (General)

The distinctions you can add to your mental repertoire, the more powerful of a positive thinker you will be.

You can think of each positive thinking quote as a distinction that can add more depth.

Draw from Wisdom of the Ages and Modern Sages on the Art and Science of Positive Thinking

I've included positive thinking quotes from a wide range of people including  Anne Frank, Epictetus, Johann Wolfgang von Goethe, Napoleon, Oscar Wilde, Ralph Waldo Emerson, Robert Frost, Voltaire, Winston Churchill, and many, many more.

You might even find useful positive thinking mantras from the people that you work with, or the people right around you. 

For example, here are a few positive thinking thoughts from Satya Nadella:

“The future we're going to invent together, express ourselves in the most creative ways.”

“I want to work in a place where everybody gets more meaning out of their work on an everyday basis.

I want each of us to give ourselves permission to be able to move things forward.  Each of us sometimes overestimate the power others have to do things vs. our own ability to make things happen.

Challenge Yourself to New Levels of Positive Thinking through Positive Quotes

As you explore the positive thinking quotes collection, try to find the quotes that challenge you the most, that really make you think, and give you a new way to generate more positive thoughts in your worst situations. 

In the words of Friedrich Nietzsche, "That which does not kill us makes us stronger."

You can use your daily trials and tribulations in the workplace as your personal dojo to practice and build your positive thinking skills.

The more positivity you can bring to the table, the more you’ll empower yourself in ways you never thought possible.

As you get tested by your worst scenarios, it’s good to keep in mind, the words of F. Scott Fitzgerald:

"The test of a first-rate intelligence is the ability to hold two opposed ideas in the mind at the same time, and still retain the ability to function.  One should, for example, be able to see that things are hopeless and yet be determined to make them otherwise."

I’ll also point out that as you grow your toolbox of positive thinking quotes and you build your positive thinking skills, you need to also focus on taking positive action.

Don’t just imagine a better garden, get out and actually weed it.

Don’t Just Imagine Everything Going Well – Imagine How You’ll Deal with the Challenges

Here’s another important tip about positivity and positive thinking …

If you use visualization as part of your approach to getting results, it’s important to include dealing with setbacks and challenges.   It’s actually more effective to imagine the most likely challenges coming up, and walking through how you’ll deal with them, if they occur.   This is way more effective than just picturing the perfect plan where everything goes without a hitch.

The reality is things happen, stuff comes up, and setbacks occur.

But your ability to mentally prepare for the setbacks, and have a plan of action, will make you much more effective in dealing with the challenge that actually do occur.  This will help you respond vs. react in more situations, and to stay in a better place mentally while you evaluate options, and decide a course of action.  (Winging it under stress doesn’t work very well because we shut down our prefrontal cortex – the smart part of our brain – when we go into flight-or-fight mode.)

If I missed any of your favorite positive thinking quotes in my positive thinking quotes collection, please let me know.

In closing, please keep handy one of the most powerful positive thinking quotes of all time:

“May the force be with you.”

Always.

You Might Also Like

The Great Inspirational Quotes Revamped

The Great Happiness Quotes Collection Revamped

The Great Leadership Quotes Collection Revamped

The Great Love Quotes Collection Revamped

The Great Motivational Quotes Revamped

The Great Personal Development Quotes Collection Revamped

The Great Productivity Quotes Collection Revamped

Categories: Blogs

Python: Detecting the speaker in HIMYM using Parts of Speech (POS) tagging

Mark Needham - Sun, 03/01/2015 - 04:36

Over the last couple of weeks I’ve been experimenting with different classifiers to detect speakers in HIMYM transcripts and in all my attempts so far the only features I’ve used have been words.

This led to classifiers that were overfitted to the training data so I wanted to generalise them by introducing parts of speech of the words in sentences which are more generic.

First I changed the function which generates the features for each word to also contain the parts of speech of the previous and next words as well as the word itself:

def pos_features(sentence, sentence_pos, i):
    features = {}
 
    features["word"] = sentence[i]
    features["word-pos"] = sentence_pos[i][1]
 
    if i == 0:
        features["prev-word"] = "<START>"
        features["prev-word-pos"] = "<START>"
    else:
        features["prev-word"] = sentence[i-1]
        features["prev-word-pos"] = sentence_pos[i-1][1]
 
    if i == len(sentence) - 1:
        features["next-word"] = "<END>"
        features["next-word-pos"] = "<END>"
    else:
        features["next-word"] = sentence[i+1]
        features["next-word-pos"] = sentence_pos[i+1][1]
 
    return features

Next we need to tweak our calling code to calculate the parts of speech tags for each sentence and pass it in:

featuresets = []
for tagged_sent in tagged_sents:
    untagged_sent = nltk.tag.untag(tagged_sent)
    sentence_pos = nltk.pos_tag(untagged_sent)
    for i, (word, tag) in enumerate(tagged_sent):
        featuresets.append((pos_features(untagged_sent, sentence_pos, i), tag) )

I’m using nltk to do this and although it’s slower than some alternatives, the data set is small enough that it’s not an issue.

Now it’s time to train a Decision Tree with the new features. I created three variants – one with both words and POS; one with only words; one with only POS.

I took a deep copy of the training/test data sets and then removed the appropriate keys:

def get_rid_of(entry, *keys):
    for key in keys:
        del entry[key]
 
import copy
 
# Word based classifier
tmp_train_data = copy.deepcopy(train_data)
for entry, tag in tmp_train_data:
    get_rid_of(entry, 'prev-word-pos', 'word-pos', 'next-word-pos')
 
tmp_test_data = copy.deepcopy(test_data)
for entry, tag in tmp_test_data:
    get_rid_of(entry, 'prev-word-pos', 'word-pos', 'next-word-pos')
 
c = nltk.DecisionTreeClassifier.train(tmp_train_data)
c.classify(tmp_test_data)
 
# POS based classifier
tmp_train_data = copy.deepcopy(train_data)
for entry, tag in tmp_train_data:
    get_rid_of(entry, 'prev-word', 'word', 'next-word')
 
tmp_test_data = copy.deepcopy(test_data)
for entry, tag in tmp_test_data:
    get_rid_of(entry, 'prev-word', 'word', 'next-word')
 
c = nltk.DecisionTreeClassifier.train(tmp_train_data)
c.classify(tmp_test_data)

The full code is on my github but these were the results I saw:

$ time python scripts/detect_speaker.py
Classifier              speaker precision    speaker recall    non-speaker precision    non-speaker recall
--------------------  -------------------  ----------------  -----------------------  --------------------
Decision Tree All In             0.911765          0.939394                 0.997602              0.996407
Decision Tree Words              0.911765          0.939394                 0.997602              0.996407
Decision Tree POS                0.90099           0.919192                 0.996804              0.996008

There’s still not much in it – the POS one has slightly more false positives and false positives when classifying speakers but on other runs it performed better.

If we take a look at the decision tree that’s been built for the POS one we can see that it’s all about POS now as you’d expect:

>>> print(c.pseudocode(depth=2))
if next-word-pos == '$': return False
if next-word-pos == "''": return False
if next-word-pos == ',': return False
if next-word-pos == '-NONE-': return False
if next-word-pos == '.': return False
if next-word-pos == ':':
  if prev-word-pos == ',': return False
  if prev-word-pos == '.': return False
  if prev-word-pos == ':': return False
  if prev-word-pos == '<START>': return True
  if prev-word-pos == 'CC': return False
  if prev-word-pos == 'CD': return False
  if prev-word-pos == 'DT': return False
  if prev-word-pos == 'IN': return False
  if prev-word-pos == 'JJ': return False
  if prev-word-pos == 'JJS': return False
  if prev-word-pos == 'MD': return False
  if prev-word-pos == 'NN': return False
  if prev-word-pos == 'NNP': return False
  if prev-word-pos == 'NNS': return False
  if prev-word-pos == 'POS': return False
  if prev-word-pos == 'PRP': return False
  if prev-word-pos == 'PRP$': return False
  if prev-word-pos == 'RB': return False
  if prev-word-pos == 'RP': return False
  if prev-word-pos == 'TO': return False
  if prev-word-pos == 'VB': return False
  if prev-word-pos == 'VBD': return False
  if prev-word-pos == 'VBG': return False
  if prev-word-pos == 'VBN': return True
  if prev-word-pos == 'VBP': return False
  if prev-word-pos == 'VBZ': return False
if next-word-pos == '<END>': return False
if next-word-pos == 'CC': return False
if next-word-pos == 'CD':
  if word-pos == '$': return False
  if word-pos == ',': return False
  if word-pos == ':': return True
  if word-pos == 'CD': return True
  if word-pos == 'DT': return False
  if word-pos == 'IN': return False
  if word-pos == 'JJ': return False
  if word-pos == 'JJR': return False
  if word-pos == 'JJS': return False
  if word-pos == 'NN': return False
  if word-pos == 'NNP': return False
  if word-pos == 'PRP$': return False
  if word-pos == 'RB': return False
  if word-pos == 'VB': return False
  if word-pos == 'VBD': return False
  if word-pos == 'VBG': return False
  if word-pos == 'VBN': return False
  if word-pos == 'VBP': return False
  if word-pos == 'VBZ': return False
  if word-pos == 'WDT': return False
  if word-pos == '``': return False
if next-word-pos == 'DT': return False
if next-word-pos == 'EX': return False
if next-word-pos == 'IN': return False
if next-word-pos == 'JJ': return False
if next-word-pos == 'JJR': return False
if next-word-pos == 'JJS': return False
if next-word-pos == 'MD': return False
if next-word-pos == 'NN': return False
if next-word-pos == 'NNP': return False
if next-word-pos == 'NNPS': return False
if next-word-pos == 'NNS': return False
if next-word-pos == 'PDT': return False
if next-word-pos == 'POS': return False
if next-word-pos == 'PRP': return False
if next-word-pos == 'PRP$': return False
if next-word-pos == 'RB': return False
if next-word-pos == 'RBR': return False
if next-word-pos == 'RBS': return False
if next-word-pos == 'RP': return False
if next-word-pos == 'TO': return False
if next-word-pos == 'UH': return False
if next-word-pos == 'VB': return False
if next-word-pos == 'VBD': return False
if next-word-pos == 'VBG': return False
if next-word-pos == 'VBN': return False
if next-word-pos == 'VBP': return False
if next-word-pos == 'VBZ': return False
if next-word-pos == 'WDT': return False
if next-word-pos == 'WP': return False
if next-word-pos == 'WRB': return False
if next-word-pos == '``': return False

I like that it’s identified the ‘:‘ pattern:

if next-word-pos == ':':
  ...
  if prev-word-pos == '<START>': return True

Next I need to drill into the types of sentence structures that it’s failing on and work out some features that can handle those. I still need to see how well a random forest of decision trees would as well.

Categories: Blogs

Eat Risks for Breakfast, Poop Awesomeness All Day!

Leading Answers - Mike Griffiths - Sun, 03/01/2015 - 00:03
I have submitted a presentation for Agile 2015 Conference about team based risk and opportunity management that may well get rejected based on its title alone! It has always been a good practice to engage team members in the estimation... Mike Griffiths
Categories: Blogs

A Stakeholder goes to St. Ives

lizkeogh.com - Elizabeth Keogh - Sat, 02/28/2015 - 20:32

As I was trying to resolve my problem, I met a portfolio team with seven programmes of work.

Each programme had seven projects;

Each project had seven features;

Each feature had seven stories;

Each story had seven scenarios.

How many things did I need to resolve?


Categories: Blogs

The Role of PI Objectives by Eric Willeke

Agile Product Owner - Sat, 02/28/2015 - 18:43

We are always grateful to the SPCs in the field who are transforming enterprises to a new and better Lean and Agile way of working with SAFe. It’s heavy lifting, but the results (see Case Studies, many more on the way): higher quality and productivity, faster time to market, higher quality of work life, are so worth it. Many of the SPCs are thought leaders in their own right, and contribute directly to the SAFe body of knowledge.

Eric Willeke from Rally is a good example. He recently opined on the critical role that the interesting mechanism of “PI Objectives” play in aligning the business and helping Agile Release Trains achieve better outcomes. His post is here: Role of PI Objectives in SAFe.

And a while back, he contributed another neat bit of work, highlighting how SAFe applies WIP limits, both explicitly and simplicity at the Portfolio level, where it really counts. Just for reference, that post is Perspective on SAFe Portfolio-WIP-Limits.

Both of these belong in Guidance, so I’ll talk to Eric and see if we can get that accomplished.

Thanks Eric!

Categories: Blogs

Summary of five years

Thought Nursery - Jeffrey Fredrick - Sat, 02/28/2015 - 11:05

“What have you been doing these last few years?” was the question Péter Halácsy asked me during my visit to Prezi. I was there for the CTO equivalent of a developer exchange: learning how things were done at Prezi, sharing my observations, and then speaking at the Budapest Jenkins meetup. Prior to my visit Péter had come to this blog to learn more about me, only to learn that I’d not been blogging. I’m resolved to get back into the blogging habit this year and I decided I’d take the time to fill in the gap for any future Péters. In part this will recapitulate my LinkedIn profile but also describe some of what I felt was most significant.

The primary reason I only posted a single post after 2009 was that I joined Urbancode in a marketing/evangelism role and I posted almost everything I had to say under their masthead. In my two and half years there I had a great time spreading the word about build automation, continuous delivery and Devops. I was able to visit a wide range of companies, learn first hand about the challenges of enterprise organization, and then turn this information into new content. At Urbancode we developed a very good flow of information and almost every month we had a new webinar, a newsletter, and maybe a white paper. My primary content collaborator was Eric Minick and he has kept up those evangelizing ways at IBM following their acquisition of Urbancode.

After I left Urbancode we made a family decision to try living in London for a few years. I reached out to Douglas Squirrel and he brought me into TIM Group to do continuous delivery, infrastructure and operations. In my time there I’ve become CTO and Head of Product and I’ve really enjoyed the opportunity to apply what I know, both about product development and about organizational change. I’ve been almost equally as absent from the TIM Group development blog, but I have managed to share some of our experiences and learning at a few conferences including GOTO Conference 2012 (talk description & slides: A Leap from Agile to DevOps), London Devops Days 2013 (video of talk: Crossing the Uncanny Valley of Culture through Mutual Learning),  and XPDay London 2014.

During my time in London Benjamin Mitchell has been one of the biggest influences on my thinking and approach to organizational change. Benjamin has been a guide to the work of Chris Argyris and Action Science. It has been what I’ve learned from and with Benjamin that has inspired me to start the London Action Science Meetup.

Finally, I couldn’t recap the last few years without also mentioning Paul Julius and CITCON. Since I last mentioned CITCON North America in Minneapolis on this blog in 2009 we’ve gone on to organize 16 additional CITCON events worldwide, most recently in Auckland (CITCON ANZ), Zagreb (CITCON Europe), Austin (CITCON North America), and Hong Kong (CITCON Asia). For PJ and I this is our 10th year of CITCON (and OIF, the Open Information Foundation) and it has been fantastic to continue to meet people throughout the world who care about improving the way we do software development.

Categories: Blogs

Fueling Delivery Teams

Leading Agile - Mike Cottmeyer - Fri, 02/27/2015 - 20:52

I’ve started using an analogy to illustrate the importance of product owner teams in larger organizations.  When working with organizations to do an agile transformation, almost always, a tiered model is used for scaling across the organization. The model looks something like this:

tiered-model

The top tier is portfolio management which is responsible for investment decisions and what initiatives continue to move forward. The middle tier is representative of the Product Owner role in Scrum and is where we often create program teams, sometimes called product owner teams. The bottom tier is where stable cross-functional teams reside that are delivering increments of value in some cadence such as every two weeks.

Let me describe this engine analogy to see if it resonates with you. Stable Engines

I think of the delivery tier as the engine of the organization. Each delivery team is designed to operate at a sustainable pace in a very predictable manner. If we think of each team as an engine then we may find some operate at 2500 RPM, others 3500 RPM, regardless, they each operate at a RPM that is sustainable. If we can operate with little variance then we can start to become predictable and make promises on what can be delivered over time. If a team is running at 2500 RPM then I can determine how far that team will travel if that engine powers a car.

But, if we push the engine past a sustainable pace, or have them operate past the redline, then the engine becomes stressed, will wear out more quickly, and likely fail to deliver as expected and certainly become unpredictable.

Fuel for the Engines

Product owner teams represent refineries responsible for providing good fuel. They must understand the business goals and problems to be solved, collaborate with delivery teams to define capabilities to build, and further decompose those into clear backlog items that provide clean burning fuel needed by the engines.

Provide bad fuel – the teams sputter and are unable to run at a consistent RPM. Bad fuel will not allow us to make promises to the market or our customers.

Drilling Locations

I haven’t completely vetted this out in my mind, but when thinking about portfolio management I believe this is where we are making investments on where to drill for crude oil. We increasingly learn more about the potential initiatives, collaborate with the product owner teams, and possibly delivery teams to progressively gain more knowledge so we can make funding or cut decisions about where we will invest.

Wrapping Up

This analogy seems to resonate when I’ve used it over the last few months to describe the responsibilities of delivery teams and product owner teams. Turning crude into clean burning fuel is a big job and really highlights the challenge most organizations have in providing clarity to their delivery teams.

I would love to get your thoughts on how I’m thinking about this.

The post Fueling Delivery Teams appeared first on LeadingAgile.

Categories: Blogs

Gastblog: Ervaringen met Agile en Scrum Certificeringen

Ben Linders - Fri, 02/27/2015 - 11:02
In deze gastblog deelt Erik Philippus van ImprovemenT zijn ervaringen met diverse agile en Scrum certificeringen. Continue reading →
Categories: Blogs

What Is An Agile Team and How Do You Form Them?

Leading Agile - Mike Cottmeyer - Fri, 02/27/2015 - 11:00

An agile team is not just any random group of people. An agile team is not a group of business analysts doing a daily standup to coordinate their work. It’s not a group of developers that meet every other week to do sprint planning. It’s not a project team with folks matrixed across two or more other agile teams.

An agile team is a cross-functional group of people that have everything, and everyone, necessary to produce a working, tested increment of product. These people are dedicated to the team, and as a rule, do not move between or across teams as demand ebbs and flows.

I’m going to suggest that the very definition of an agile team… a cross-functional group of people that have everything, and everyone, necessary to produce a working, tested increment of product… is getting in the way of forming agile teams, mostly because we misunderstand what a product actually is.

In the small, this guidance is clear… it’s the system all the way from user interaction to data and back and all the abilities to deploy and install said product into production. In the large, forming a team like this isn’t usually possible, and often if possible, not advisable.

In the large, a product is actually a sub-system of a larger systems integration.

When you look at a product this way, it is often very possible to create a small cross-functional group of people that has everything, and everyone, necessary to produce a working, tested increment of product.

The idea is that you organize around products or features where you can, and you organize around subsystems where you have shared functionality. We call these collectively business capabilities. Once you have the business capabilities understood, you align the business capabilities with the technical architecture and ultimately the organizational architecture.

The intersection and alignment of business architecture, technical architecture, and organizational architecture is where you form a complete cross-functional group of people that have everything, and everyone, necessary to produce a working, tested increment of their part of the product.

Because your business architecture, technical architecture, and organizational architecture are probably broken, you are going to have dependencies between these teams that are going to have to be managed. For now.

Over time, the job of the transformation initiative is to break these dependencies.

Dependencies are evil and they need to be broken.

The more you manage dependencies the less agile you will be, period.

Over time, as you break these dependencies, you will be able to treat each of these teams as pure play agile teams.

Until you start forming teams that align business capabilities with technical architecture and organizational architecture, and begin the hard work of breaking dependencies… all you can do is go through the motions of Scrum. You will never get the value you are working towards.

I’m telling you guys… the reason you’re doing agile and not feeling very agile is because you don’t have these kinds of teams and you have way too many dependencies.

No amount of daily standup meetings is going to fix this problem.

An agile culture won’t do the hard work for you.

The post What Is An Agile Team and How Do You Form Them? appeared first on LeadingAgile.

Categories: Blogs

Ten Outsourcing Considerations

Bobtuse Bobservations - Bob MacNeal - Fri, 02/27/2015 - 04:07
Software developer testing with JUnitTen outsourcing considerations when choosing a software development team: Are they local?Are they recommended by someone you trust?Are they demonstrably...

Bobtuse can be wildly informative
Categories: Blogs

R/ggplot: Controlling X axis order

Mark Needham - Fri, 02/27/2015 - 02:49

As part of a talk I gave at the Neo4j London meetup earlier this week I wanted to show how you could build a simple chart showing the number of friends that different actors had using the ggplot library.

I started out with the following code:

df = read.csv("/tmp/friends.csv")
top = df %>% head(20)
 
ggplot(aes(x = p.name, y = colleagues), data = top) + 
  geom_bar(fill = "dark blue", stat = "identity")

The friends CSV file is available as a gist if you want to reproduce the chart. This is how the chart renders:

2015 02 27 00 41 08

It’s in a fairly arbitrary order when it would be quite cool if we could get the most popular people to show from left to right.

I had the people’s names in the correct order in the data frame but annoyingly it was then sorting them into alphabetical order. Luckily I came across the by using the scale_x_discrete function which does exactly what I needed.

If we pass in the list of names to that function we get the chart we desire:

ggplot(aes(x = p.name, y = colleagues), data = top) + 
  geom_bar(fill = "dark blue", stat = "identity") + 
  scale_x_discrete(limits= top$p.name)

2015 02 27 00 45 01

Categories: Blogs

Updated: Full-Day Product Owner Simulation

Learn more about our Scrum and Agile training sessions on WorldMindware.com

The Product Owner Simulation that I shared last summer has some minor updates based on a stronger emphasis on product vision.  In particular, two 5 minute exercises before and after the Product Box exercise help to frame the concept of product vision and make it stronger.

Try out our Virtual Scrum Coach with the Scrum Team Assessment tool - just $500 for a team to get targeted advice and great how-to informationPlease share!
facebooktwittergoogle_plusredditpinterestlinkedinmail
Categories: Blogs

The One Immutable Law of Adopting Agile

Leading Agile - Mike Cottmeyer - Thu, 02/26/2015 - 10:00

I have three non-negotiables when it comes to running an agile transformation. If we are unwilling to create clear, well-articulated backlogs… if we are unwilling to form complete cross-functional teams… and if we are unwilling to produce working, tested software on regular intervals… we probably shouldn’t even get started. Without these three pre-conditions it is unlikely that adopting agile practices (or even an agile culture) will make much of a dent in terms of how you put product into market.

That said, the number one thing that gets in the way of creating clear, well-artciulated backlogs, forming complete cross-functional teams, and producing working tested software on regular intervals is dependencies. Let’s just say it… dependencies are evil. Dependencies break agile. You have two choices when faced with a dependency… you can either break it or you can manage it. The more dependencies you choose to manage instead of breaking, the less agile you will be over time.

This might just be the one immutable law of adopting agile.

The post The One Immutable Law of Adopting Agile appeared first on LeadingAgile.

Categories: Blogs

Why LeadingAgile?

Leading Agile - Mike Cottmeyer - Thu, 02/26/2015 - 03:47

We are working on more video for the site to tell our story. This is the first stab.

The post Why LeadingAgile? appeared first on LeadingAgile.

Categories: Blogs

R: Conditionally updating rows of a data frame

Mark Needham - Thu, 02/26/2015 - 02:45

In a blog post I wrote a couple of days ago about cohort analysis I had to assign a monthNumber to each row in a data frame and started out with the following code:

library(zoo)
library(dplyr)
 
monthNumber = function(cohort, date) {
  cohortAsDate = as.yearmon(cohort)
  dateAsDate = as.yearmon(date)
 
  if(cohortAsDate > dateAsDate) {
    "NA"
  } else {
    paste(round((dateAsDate - cohortAsDate) * 12), sep="")
  }
}
 
cohortAttendance %>% 
  group_by(row_number()) %>% 
  mutate(monthNumber = monthNumber(cohort, date)) %>%
  filter(monthNumber != "NA") %>%
  filter(monthNumber != "0") %>% 
  mutate(monthNumber = as.numeric(monthNumber)) %>% 
  arrange(monthNumber)

If we time this function using system.time we’ll see that it’s not very snappy:

system.time(cohortAttendance %>% 
  group_by(row_number()) %>% 
  mutate(monthNumber = monthNumber(cohort, date)) %>%
  filter(monthNumber != "NA") %>%
  filter(monthNumber != "0") %>% 
  mutate(monthNumber = as.numeric(monthNumber)) %>% 
  arrange(monthNumber))
 
   user  system elapsed 
  1.968   0.019   2.016

The reason for the poor performance is that we process each row of the data table individually due to the call to group_by on the second line. One way we can refactor the code is to use the ifelse which can process multiple rows at a time:

system.time(
cohortAttendance %>% 
  mutate(monthNumber = ifelse(as.yearmon(cohort) > as.yearmon(date), 
                              paste((round(as.yearmon(date) - as.yearmon(cohort))*12), sep=""), 
                              NA)))
   user  system elapsed 
  0.026   0.000   0.026

Antonios suggested another approach which involves first setting every row to ‘NA’ and then selectively updating the appropriate rows. I ended up with the following code:

cohortAttendance$monthNumber = NA
 
cohortAttendance$monthNumber[as.yearmon(cohortAttendance$cohort) > as.yearmon(cohortAttendance$date)] = paste((round(as.yearmon(cohortAttendance$date) - as.yearmon(cohortAttendance$cohort))*12), sep="")

Let’s measure that:

system.time(paste((round(as.yearmon(cohortAttendance$date) - as.yearmon(cohortAttendance$cohort))*12), sep=""))
   user  system elapsed 
  0.013   0.000   0.013

Both approaches are much quicker than my original version although this one seems to be marginally quicker than the ifelse approach.

Note to future Mark: try to avoid grouping by row number – there’s usually a better and faster solution!

Categories: Blogs

Cultural Judo – How to Change Culture Without Changing Culture

Leading Agile - Mike Cottmeyer - Thu, 02/26/2015 - 01:21

There are a bunch of us out there that are super passionate about this notion of agile as a cultural framework… or a value system… or a mindset. If you’ve been following this blog for any length of time, I suspect you’ll know where I stand on this. I think that culture, values, and mindset are important… but they aren’t why we adopt agile. We adopt agile to build better software, software that our customers want to use, and to get better economic outcomes for our companies.

To the extent that culture, values, and mindset lead to those outcomes… great… but culture, values, and mindset are not why we adopt agile. They are a byproduct.

If we allow for a moment that culture, values, and mindset are a necessary precondition for adopting agile… let’s think through what that actually means. Let’s say we have a traditional organization… functionally soloed… tightly coupled legacy architecture… matrixed, project based delivery system… overlapping, intertwined value streams… power struggles between executives… compensation models that reinforce and incentivize around the existing system… and all the cultural toxicity that you might expect comes along with this… what do you do?

Do you come in and give everyone a pep talk on the value of agile? Do you invite them to participate and hope that they all agree and self-organize into a new, rational delivery model? Do you run them through ScrumMaster training and have them bring back a bunch of practices that are incongruent with their existing culture and delivery framework? How do you create safety for people to step out and try new things? What if the team level folks all see the value, but those in power with the most to loose don’t?

I’d suggest that you can’t attack culture by attacking culture. You have to meet the organization where it is today and use it’s own weight against it.

Most people in these kinds of organizations recognize that something is broken and that something needs to be fixed… they just don’t trust everyone around them to necessarily work in their best interest to fix it. Our approach is to use the organizations need for predictability and control to actually create the conditions where agile practices and culture can take hold. We know that agile creates more visibility, more insight into what’s going on, better ability to make and meet commitments than any sequential process.

We have to use agile to get the organization what it needs and to create the conditions for change. We create the conditions for agile by creating the org patterns that enable agility.

When you are promising the organization what it wants now… predictability, quality, early return on investment… and can make the case that all this agile stuff gets them there faster… you can begin the change without even mentioning culture, values, or mindset. Instead, you talk about backlogs, teams, and working tested software delivered on regular intervals. You talk about metrics and planning. You talk about the ability to realize revenue earlier, you talk about creating clean breakpoints where we can change direction, you talk about having greater ability to make and meet commitments.

None of this is incongruent with where we want to go… none of this is incongruent with evolving a healthier culture, living agile values, or adopting an agile mindset… those things just aren’t where we start. We start by adopting an agile system of delivery, a system that delivers the business results your company so desperately needs, while creating the conditions for more agile practices to take root… and a more agile culture to emerge over time. You are never going to change culture by telling everyone they are doing it wrong and need to change.

My belief is that to effectively change culture, you have to create safety… you have to create alignment… you have to create congruence between your desired outcomes and rewards. You have to have a rational system of delivery that allows people to show up for work, to be able to do what they are being asked to do, and really see and feel what it is like to be successful in that new model. In an organization with a highly toxic culture, you have to lead with a point of view and a plan and push the organization to change… I don’t believe pull will work.

I think culture, values, and mindset are important, but I tend to see them as a byproduct of a rational system of delivery… a system of delivery that creates safety for everyone involved… a system of delivery that allows those cultural norms, applied values, and mindset to emerge over time. People ask me all the time how to sell agile to your executives… the answer is to simply stop talking about what we want to accomplish with agile and start aligning agile to the business drivers that are most important to your executives.

Use the organizations weight against it. Sell the organization a way to solve it’s business problems… you’ll get the culture change for free.

The post Cultural Judo – How to Change Culture Without Changing Culture appeared first on LeadingAgile.

Categories: Blogs