Skip to content

Feed aggregator

Major SAFe LSE Update with Big Implications

Hello Folks,

As you probably know, I’ve moved most of my blogging activity to the Scaled Agile Framework Blog. For a major update on SAFe LSE and the future of SAFe, check out this post.

I’ll be maintaining this blog for just a while longer. As the SAFe blog is dedicated to SAFe content, I plan to start my personal blogging again on my website at deanleffingwell.com. I’ll be describing a few other topics I’ve been researching lately, and some great books and background reading, all of which is too early for SAFe.

–Dean

Categories: Blogs

How Agile Helped a Business Analyst

TV Agile - Mon, 05/18/2015 - 17:23
As companies introduce agile practices, the business analyst (BA) role is often left by the wayside. The BA title doesn’t exist in Scrum and other agile implementations, leaving many BAs wondering where—or if—they fit in. But fear not! The skills of a good BA are even more valuable in an agile environment. Diane Zajac-Woodie tells […]
Categories: Blogs

Customer Spotlight: MongoDB

Rally Agile Blog - Mon, 05/18/2015 - 17:00

Occasional stories about Rally customers who are doing cool and interesting things.

Fast, flexible, and scalable is an apt description for MongoDB, the leading next-generation NoSQL database. The document-oriented database is used by the world’s most sophisticated organizations, from cutting-edge startups to the largest companies, to create applications never before possible at a fraction of the cost of legacy databases. Think Internet of Things, realtime analytics, and mobile apps — all part of a rapidly changing technology landscape that requires a thoroughly modern database.

Fast, flexible, and scalable is also an apt description for MongoDB, the company.

With customers all over the world, MongoDB provides 24/7 support and services from offices in Dublin, Palo Alto, New York, Amsterdam, London, Sydney, Hong Kong, and Singapore, just to name a handful. That’s 200 people, from solution architects and consultants to TechOps, IT, and Support teams.

And they’re doing an incredible job staying patched into the hive mind. Teams are collaborating across time zones, using Rally® Flowdock® as a kind of clearinghouse for support tickets, bug fixes, troubleshooting, and more, with integrations to GitHub and social media channels. We love that they used the Hubot integration to automate much of this back-and-forth through a bot called GladOS.

As Engineering Manager Jeff Yemin told us, “We use Flowdock as one of our primary communication tools across teams and offices to manage and discuss how our various systems are performing. It has enabled us to raise issues and discuss solutions faster, becoming a core solution we depend on.”

Great example of a company living out the principles they build into their product. Want to learn more? Read the case study.

  Jen Page
Categories: Companies

Is Agile Working for Your Project?

Johanna Rothman - Mon, 05/18/2015 - 15:59

My column is up on projectmanagement.com. It’s called Is Agile Working for Your Project?

I hope you enjoy it.

Categories: Blogs

Agile Coach Camp Denmark, Dragør, Denmark, May 28-30 2015

Scrum Expert - Mon, 05/18/2015 - 14:24
Agile Coach Camp Denmark is a three-day event that serves as an unconference for Agile coaches of Denmark and other countries. The Agile Coach Camp Denmark is a free, not-for-profit, practitioner-run event. Agile Coach Camp Denmark follows the open space format for conferences. Open space is a simple methodology for self-organizing conference tracks. It relies on participation by people who have a passion for the topics to be discussed. There is no preplanned list of topics, only time slots and a space in the main meeting room where interested participants propose ...
Categories: Communities

Organisier dich oder geh!

Scrum 4 You - Mon, 05/18/2015 - 12:05

Als ich vor ein paar Wochen auf Fast Company den Artikel über den Versuch von Zappos-Gründer Tony Hsieh las, seine Mitarbeiterinnen und Mitarbeiter per Dekret zur Selbstorganisation zu zwingen, musste ich schmunzeln. In den Zeilen seiner E-Mail konnte ich förmlich spüren, wie genervt und hilflos er ist. Psychologen nennen das wohl „Übertragung“: Ich bezog mich auf ihn, ich konnte mich in ihn hineinversetzen, weil ich vor drei, vier Jahren genau so hilflos war. Ich wollte unser Unternehmen, Boris Gloger Consulting, zu einem Ort der Selbstorganisation machen, den Kolleginnen und Kollegen die Verantwortung für unser Unternehmen und so viel Entscheidungsspielraum wie nur irgend möglich geben. Aber sie nahmen das Angebot nicht an. Die Menschen in unserem Unternehmen wollten Anleitung, ließen sich von vermeintlichen Chefs kommandieren und die Hierarchie schlug ständig zu. Aber ich will hier gar nicht von uns sprechen, sondern nur deutlich machen, dass ich die Lage von Tony Hsieh zu verstehen glaube.

Neue Organisation nach altem Muster

Schmunzeln musste ich, weil Hsieh meiner Auffassung nach in seiner Hilflosigkeit genau den Fehler macht, den er bekämpfen will: Er schlägt als CEO nach Chefmanier zu und fordert die Veränderung ein. Übersetzt sagt er eigentlich: „Wer nicht mitmachen will, darf gerne gehen!“ Das erzeugt keine Sicherheit oder das Gefühl: „Mein Chef versteht mich und ist für mich da.“ Es erzeugt nur noch mehr Druck. Druck, weil ich als einzelner Mitarbeiter – so zeigt es mir mein Boss Hsieh gerade – offenbar nicht dazugehöre, wenn ich nicht folge. Gleichzeitig ist Hsiehs Impuls verständlich, er legt damit aber auch offen, dass er selbst die neue Klaviatur der Führung im Kontext der Selbstorganisation noch nicht perfekt beherrscht. Also lebt er in seiner Firma die alte, traditionelle Struktur. Das soll kein Vorwurf sein, sondern eine Analyse: Der Fisch beginnt am Kopf zu stinken. Die Veränderung zu einer anderen Organisation fängt immer und ausschließlich bei der Führung an. Klar gibt es Grassroot-Bewegungen und die agile Community hat so begonnen, aber in einer Organisation braucht es den so oft zitierten Buy-in der Top-Führung. Dieser Buy-in ist aber nur ein Anfang. Wie es funktionieren kann, lernt man am Beispiel von Bodo Janssen, CEO und Eigentümer der Hotelkette Upstalsboom.

Im Video “Wertschöpfung durch Wertschätzung” zeigt er wunderbar, dass die Veränderung seiner Organisation mit ihm selbst angefangen hat. Das Ändern seiner Haltung war entscheidend. Sicher, er als CEO musste einen Weg finden, um seine Mitarbeiter mitzunehmen, aber das gelang nur durch sein aktives Engagement. Sein ständiges Arbeiten an seinen Ideen und sein ständiges Kommunizieren darüber veränderte das Unternehmen innerhalb von zwei Jahren. Das bestätigt auch die Forschung von Frederick Laloux. Obwohl einige sicher mit Recht sagen, dass sein Buch “Reinventing Organizations” keine großartigen neuen Erkenntnisse enthält, macht Laloux sehr deutlich, dass die Veränderung mit der Haltung des CEO steht und fällt und es dann beinahe egal ist, wie der Weg zur Teal-Organisation aussieht. Ich erwähne Laloux an dieser Stelle, weil er diesen Aspekt in dem von Hsieh zitierten Skype-Call vergessen zu haben scheint. Denn er hätte Hsieh vielleicht raten sollen, an seiner eigenen Haltung zu arbeiten anstatt einen Mechanismus einzuführen und die Nichteinverstandenen gehen zu lassen.

Achtung: Es liegt mir fern, Hsieh oder Laloux zu kritisieren, denn ich kann das absolut nachvollziehen. Ich möchte nur etwas deutlich machen: Selbstorganisation braucht einen Rahmen und einen Menschen, der diesen Rahmen – Nonaka nennt das Ba (Raum) – aufspannt. Dieser Mensch muss diesen Raum so erfüllen, dass die Kollegen darin beginnen können, sich selbst zu organisieren und Entscheidungen selbst zu fällen. Diesen Raum zu erzeugen und auszuhalten, dass es nicht sofort so funktioniert wie man es sich als Chef wünscht, immer wieder aufs Neue die eigenen Verhaltensweisen zu hinterfragen und zu einer neuen Form von Führung und des Arbeitens zu kommen – das ist ein langer Weg. Ich weiß aber, dass sich dieser Weg auszahlt.

Categories: Blogs

Retrospective Prime Directive in nine languages

Ben Linders - Mon, 05/18/2015 - 09:21
The retrospective prime directive is a sentence that is used by facilitators to establish safety in a retrospective meeting. Safety is crucial if you want people to speak up and be open, which is an important precondition to reflect and learn which is what agile retrospectives are all about. Since our book is being translated into many languages we now have translations of this unique and important statement. Continue reading →
Categories: Blogs

Are incorrectly applied KPIs the root of all evil?

ScrumSense.com - Peter Hundermark - Mon, 05/18/2015 - 08:46

We have all heard it before “show me how you measure me and I show you how I behave” (I think the original quote is from Eliyahu M. Goldratt). It’s been quoted and requoted so many times it’s hard to tell, and it’s true.

I see the effects of this issue every day, which leads me to ask the question…

Are incorrectly applied KPIs the root of all evil?

I see dysfunction in organisations all the time, as do many people I am sure. I often ask the question, “is that part of your KPI’s?” The resulting answer “yes” just reinforces this more often than not.

I was speaking to the product owner of an interesting project. He was complaining about the number of bugs logged that were essentially duplicates. I asked how the testers were measured, and he said “bug counts.”
It feels to me that KPIs are trying to solve multiple problems, and that we have lost sight of the goal of encouraging a behaviour of self critical improvement, instead of informing a behaviour of compliance.

Trust / Performance

The first problem I see, is that people don’t trust each other to do a good job or rather that organisations don’t trust the awesome people they hire. We create performance bonuses and remuneration around objectives, because we believe that people need external motivators to get work done. There is an underlying belief system that says people need carrots and sticks in order to be their best. I believe that if you give people a clear common goal, and the resources to achieve that goal, remove the impediments, then they will try their best to achieve the goal, regardless of bonuses.

Instead we link specific outcomes to people’s value.

By doing this we are essentially linking their value and success in the organisation to these KPIs. People want to feel like they are adding value. The resulting behaviour is that they will do everything in their power to achieve the stated KPI outcome. This, in and of itself is not a bad thing. The problem arises when this goal is at odds with another person’s goal (or KPI) or with the overall business strategy. How we set these up is part of the problem.
Creating an environment of visibility and openness is a far better way to encourage trust and performance. People will more often than not do their best to be committed to a team and to deliver on what they have promised. No one wants to be the person that let the team down. A collaborative team environment that encourages honesty and trust is far better than a group of people with conflicting goals. Rating people individually while trying to foster a team culture will likely create dysfunction, competition and unhealthy conflict.

Lack of direction

Closely linked to the lack of trust is a lack of direction. What I see is that companies have not clearly defined their strategic objectives and created a clear common goal for the people in their organisation to move towards. Setting up Global KPIs as an indicator of progress or as an overall goal, makes more sense to me than individual competing KPIs.

An example of this I have seen is that, we have 100+ projects that need to be delivered and no clear alignment about what we are trying to achieve with these projects and which ones are most valuable to the organisation. Project Managers are rated on getting Project X in on time and this is linked to their performance bonus, whilst teams need to deliver on all 100 which is frequently an unachievable goal. Teams are forced to make priority decisions on a daily basis about what is the most important thing. They make these decisions while lacking valuable context and information.

How the Project Manager or Project Sponsor is rated, drives her behaviour. She will do whatever it takes to make sure that Project X is delivered on time. If Project X is not the best for the business, that is not her focus. If Project X gets executed poorly and shipped with loads of bugs, that is not her primary concern. The KPI was to get the project done on time and that will inform her behaviour, her goal is to add value and meet her targets.

By setting her KPI and linking it to a monetary incentive, we are basing her value and reward on a single goal over which she has no direct control. The resulting behaviour is to conceal the truth about progress, to compete with other project managers or project sponsors, and to not always share valuable information, especially if it does not serve her end goal. This has large impacts on the global strategy that most people are unaware of.
Setting a clear, common goal and ensuring that people have the necessary resources (yes, I mean actual resources) to achieve that goal is far better than creating competing KPIs. Setting goals that align to business strategy across all areas will help teams and individuals make better decisions about what to do next. Giving those individuals enough information about where we are going and why we need to get there will encourage them to work smarter and with more collaboration. So it seems that this starts with better prioritisation and alignment at all levels (Brad Swanson has a great post on this).

Creating Competition

Some organisations believe that creating competition between divisions and departments is healthy and that it helps the business to grow. I have seen instances where this has created a large amount of dysfunction. What often happens is that each division has KPIs that compete with other divisions, except of course the service departments, like “IT”, who often have to service all of these divisions, and generally at the same time.

Now you have different divisions vying for “resources” (if you’ll excuse the term) and the result of this is that each team member is 20% allocated to this and 30% allocated to that and 45% allocated to the next thing, and 5% allocated to learning. These individuals spend their days context-switching between competing stakeholder requests or in project update meetings. Likely less than 50% of their work day is spent doing the making that everyone is hoping for. As a result, each person is actually only able to contribute minimal time to the project. The transaction costs are so high that it ends up taking years for a team to ship anything.

Having alignment is better than competition, especially when there are only one or two teams that can deliver to the business. If the Project Managers and Product Owners are aligned in their goals, then it will be much easier to ship the most important things, because it will be easier to get focus for the teams on one specific thing. Setting people against each other will only result in more context switching and higher transaction costs. The knock-on effect is that everything takes longer to get done.

The current incarnation of KPIs is NOT, I believe, effectively solving the problems we want to solve. We pit people against each other and encourage them to compete; for time, for resources, and for people. We create environments where people are fighting to get their needs met and not for the overall needs of the business. In an environment like that, it is the loudest person or bully that will win every time, and that doesn’t always make for the best business decisions.

According to Gerald Weinberg, it’s useful to apply the rule of threes when thinking about any problem. Gerald suggests that if we can’t come up with at least three ways to solve the problem, then we don’t understand the problem effectively.

Perhaps in the case of KPIs we have lost sight of the problem we were trying to solve.

Some organisational KPIs add value and direction. At agile42 we use KPIs to help create focus, direction and self critical improvement. The current incarnation of KPIs in most organisations is reinforcing the dysfunction and not solving the right problems. We apply KPIs as a solution to create performance goals because that is the way we have always done it. To quote Gerry again “Things are the way they are because they got that way”. Maybe it is time to go back to the beginning, and ask ourselves,

“What is the problem we are trying to solve?”

If we did that, we could find better ways of solving these problems without the resulting dysfunction.

 

The post Are incorrectly applied KPIs the root of all evil? appeared first on ScrumSense.

Categories: Blogs

I Prefer This Over That

A couple weeks ago I tweeted:

I prefer: - Recovery over Perfection - Predictability over Commitment - Safety Nets over Change Control - Collaboration over Handoffs

— ElisabethHendrickson (@testobsessed) May 6, 2015

Apparently it resonated. I think that’s more retweets than anything else original I’ve said on Twitter in my seven years on the platform. (SEVEN years? Holy snack-sized sound bytes! But I digress.)

@jonathandart said, “I would love to read a fleshed out version of that tweet.”

OK, here you go.

First, a little background. Since I worked on Cloud Foundry at Pivotal for a couple years, I’ve been living the DevOps life. My days were filled with zero-downtime deployments, monitoring, configuration as code, and a deep antipathy for snowflakes. We honed our practices around deployment checklists, incident response, and no-blame post mortems.

It is within that context that I came to appreciate these four simple statements.

Recovery over Perfection

Something will go wrong. Software might behave differently with real production data or traffic than you could possibly have imagined. AWS could have an outage. Humans, being fallible, might publish secret credentials in public places. A new security vulnerability may come to light (oh hai, Heartbleed).

If we aim for perfection, we’ll be too afraid to deploy. We’ll delay deploying while we attempt to test all the things (and fail anyway because ‘all the things’ is an infinite set). Lowering the frequency with which we deploy in order to attempt perfection will ironically increase the odds of failure: we’ll have fewer turns of the crank and thus fewer opportunities to learn, so we’ll be even farther from perfect.

Perfect is indeed the enemy of good. Striving for perfection creates brittle systems.

So rather than strive for perfection, I prefer to have a Plan B. What happens if the deployment fails? Make sure we can roll back. What happens if the software exhibits bad behavior? Make sure we can update it quickly.

Predictability over Commitment

Surely you have seen at least one case where estimates were interpreted as a commitment, and a team was then pressured to deliver a fixed scope in fixed time.

Some even think such commitments light a fire under the team. They give everyone something to strive for.

It’s a trap.

Any interesting, innovative, and even slightly complex development effort will encounter unforeseen obstacles. Surprises will crop up that affect our ability to deliver. If those surprises threaten our ability to meet our commitments, we have to make painful tradeoffs: Do we live up to our commitment and sacrifice something else, like quality? Or do we break our commitment? The very notion of commitment means we probably take the tradeoff. We made a commitment, after all. Broken commitments are a sign of failure.

Commitment thus trumps sustainability. It leads to mounting technical debt. Some number of years later find themselves constantly firefighting and unable to make any progress.

The real problem with commitments is that they suggest that achieving a given goal is more important than positioning ourselves for ongoing success. It is not enough to deliver on this one thing. With each delivery, we need to improve our position to deliver in the future.

So rather than committing in the face of the unknown, I prefer to use historical information and systems that create visibility to predict outcomes. That means having a backlog that represents a single stream of work, and using velocity to enable us to predict when a given story will land. When we’re surprised by the need for additional work, we put that work in the backlog and see the implications. If we don’t like the result, we make an explicit decision to tradeoff scope and time instead of cutting corners to make a commitment.

Aiming for predictability instead of commitment allows us to adapt when we discover that our assumptions were not realistic. There is no failure, there is only learning.

Safety Nets over Change Control

If you want to prevent a given set of changes from breaking your system, you can either put in place practices to tightly control the nature of the changes, or you can make it safer to change things.

Controlling the changes typically means having mechanisms to accept or reject proposed changes: change control boards, review cycles, quality gates.

Such systems may be intended to mitigate risk, but they do so by making change more expensive. The people making changes have to navigate through the labyrinth of these control systems to deliver their work. More expensive change means less change means less risk. Unless the real risk to your business is a slogging pace of innovation in a rapidly changing market.

Thus rather than building up control systems that prevent change, I’d rather find ways to make change safe. One way is to ensure recoverability. Recovery over perfection, after all.

Fast feedback cycles make change safe too. So instead of a review board, I’d rather have CI to tell us when the system is violating expectations. And instead of a laborious code review process, I’d rather have a pair work with me in real time.

If you want to keep the status quo, change control is fine. But if you want to go fast, find ways to make change cheap and safe.

Collaboration over Handoffs

In traditional processes there are typically a variety of points where one group hands off work to another. Developers hand off to other developers, to QA for test, to Release Engineering to deliver, or to Ops to deploy. Such handoffs typically involve checklists and documentation.

But the written word cannot convey the richness of a conversation. Things will be missed. And then there will be a back and forth.

“You didn’t document foo.”
“Yes, we did. See section 3.5.1.”
“I read that. It doesn’t give me the information I need.”

The next thing you know it’s been 3 weeks and the project is stalled.

We imagine a proper handoff to be an efficient use of everyone’s time, but they’re risky. Too much can go wrong, and when it does progress stops.

Instead of throwing a set of deliverables at the next team down the line, bring people together. Embed testers in the development team. Have members of the development team rotate through Ops to help with deployment and operation for a period of time. It actually takes less time to work together than it does to create sufficient documentation to achieve a perfect handoff.

True Responsiveness over the Illusion of Control

Ultimately all these statements are about creating responsive systems.

When we design processes that attempt to corral reality into a neat little box, we set ourselves up for failure. Such systems are brittle. We may feel in control, but it’s an illusion. The real world is not constrained by our imagined boundaries. There are surprises just around the corner.

We can’t control the surprises. But we can be ready for them.

Categories: Blogs

Neo4j: Refactoring the BBC football live text fouls graph

Mark Needham - Sun, 05/17/2015 - 13:04

Yesterday I wrote about a Neo4j graph I’ve started building which contains all the fouls committed in the Champions League game between Barcelona & Bayern Munich and surrounding meta data.

While adding other events into the graph I realised that I’d added some duplication in the model and the model could do with some refactoring to make it easier to use.

To recap, this is the model that we designed in the previous blog post:

The duplication is on the left hand side of the model – we model a foul as being committed by one player against another and then hook the foul back into the match. By doing that we’re not using the ‘appearance’ concept which links a player and a match together.

We can make the ‘COMMITTED_IN_MATCH’ relationship redundant by connecting the foul to appearance rather than to player. The match the foul was committed in can then be found by navigating through the appearance node.

This is what we want the graph to look like:

2015 05 17 10 40 44

We’ll move towards this new model in 3 steps:

  • Introduce the new structure alongside the existing one
  • Rewrite our queries to use the new structure
  • Remove the old structure
Introducing the new structure

First up let’s write a query to introduce the new structure.

match (foul:Foul)-[:COMMITTED_AGAINST]->(fouledPlayer),
      (foul)<-[:COMMITTED_FOUL]-(foulingPlayer),
      (foul)-[:COMMITTED_IN_MATCH]->(match:Match {id: "32683310"}),
      (foulingPlayer)-[:MADE_APPEARANCE]-(foulingPlayerApp)-[:IN_MATCH]->(match),
      (fouledPlayer)-[:MADE_APPEARANCE]-(fouledPlayerApp)-[:IN_MATCH]->(match)
MERGE (foul)<-[:COMMITTED_FOUL]-(foulingPlayerApp)
MERGE (foul)-[:COMMITTED_AGAINST]->(fouledPlayerApp)

Remember we’re not going to delete the old structure yet so that’s why there aren’t any delete statements in here.

Rewriting our queries

Now we need to update our queries to work against the new graph structure:

Where do the fouls happen?
match (match:Match {id: "32683310"})<-[:COMMITTED_IN_MATCH]-(foul)
RETURN foul.location AS location, COUNT(*) as fouls
ORDER BY fouls DESC

becomes

match (match:Match {id: "32683310"})<-[:IN_MATCH]-()<-[]-(foul:Foul)
RETURN foul.location AS location, COUNT(*) as fouls
ORDER BY fouls DESC
Who fouls the most?
match (match:Match {id: "32683310"})<-[:COMMITTED_IN_MATCH]-(foul:Foul)<-[:COMMITTED_FOUL]-(fouler:Player)
RETURN fouler.name AS fouler, COUNT(*) as fouls
ORDER BY fouls DESC
LIMIT 10;

becomes

match (match:Match {id: "32683310"})<-[:IN_MATCH]-(appearance)-[:COMMITTED_FOUL]->(foul:Foul),
      (appearance)<-[:MADE_APPEARANCE]-(fouler)
RETURN fouler.name AS fouler, COUNT(*) as fouls
ORDER BY fouls DESC
LIMIT 10
Who was fouled the most?
match (match:Match {id: "32683310"})<-[:IN_MATCH]-(appearance)-[r:COMMITTED_FOUL]->(foul:Foul),
      (appearance)<-[:MADE_APPEARANCE]-(fouler)
RETURN fouler.name AS fouler, COUNT(*) as fouls
ORDER BY fouls DESC
LIMIT 10

becomes

match (match:Match {id: "32683310"})<-[:IN_MATCH]-(appearance)<-[:COMMITTED_AGAINST]->(foul:Foul),
      (appearance)<-[:MADE_APPEARANCE]-(fouled)
RETURN fouled.name AS fouled, COUNT(*) as fouls
ORDER BY fouls DESC
LIMIT 10
Who fouled who the most?
match (match:Match {id: "32683310"})<-[:COMMITTED_IN_MATCH]-(foul:Foul)-[:COMMITTED_AGAINST]->(fouled:Player),
      (foul)<-[:COMMITTED_FOUL]-(fouler:Player)
RETURN fouler.name AS fouler, fouled.name AS fouled, COUNT(*) as fouls
ORDER BY fouls DESC
LIMIT 10

becomes

match (match:Match {id: "32683310"}),
      (match)<-[:IN_MATCH]-(fouledApp)<-[:COMMITTED_AGAINST]->(foul:Foul)<-[:COMMITTED_FOUL]-(foulerApp)-[:IN_MATCH]->(match),
      (fouledApp)<-[:MADE_APPEARANCE]-(fouled),
      (foulerApp)<-[:MADE_APPEARANCE]-(fouler)
RETURN fouler.name AS fouler, fouled.name AS fouled, COUNT(*) as fouls
ORDER BY fouls DESC
LIMIT 10;
Which team fouled most?
match (match:Match {id: "32683310"})<-[:COMMITTED_IN_MATCH]-()<-[:COMMITTED_FOUL]-(fouler),
      (fouler)-[:MADE_APPEARANCE]-(app)-[:IN_MATCH]-(match),
      (app)-[:FOR_TEAM]->(team)
RETURN team.name, COUNT(*) as fouls
ORDER BY fouls DESC

becomes

match (match:Match {id: "32683310"})<-[:IN_MATCH]-(app:Appearance)-[:COMMITTED_FOUL]->(),
      (app)-[:FOR_TEAM]->(team)
RETURN team.name, COUNT(*) as fouls
ORDER BY fouls DESC
Worst fouler for each team
match (match:Match {id: "32683310"})<-[:COMMITTED_IN_MATCH]-(foul)<-[:COMMITTED_FOUL]-(fouler),
      (fouler)-[:MADE_APPEARANCE]-(app)-[:IN_MATCH]-(match),
      (app)-[:FOR_TEAM]->(team)
WITH team, fouler, COUNT(*) AS fouls
ORDER BY team.name, fouls DESC
WITH team, COLLECT({fouler:fouler, fouls:fouls})[0] AS topFouler
RETURN team.name, topFouler.fouler.name, topFouler.fouls;

becomes

match (match:Match {id: "32683310"})<-[:IN_MATCH]-(app:Appearance)-[:COMMITTED_FOUL]->(),
      (app)-[:FOR_TEAM]->(team),
      (fouler)-[:MADE_APPEARANCE]->(app)
WITH team, fouler, COUNT(*) AS fouls
ORDER BY team.name, fouls DESC
WITH team, COLLECT({fouler:fouler, fouls:fouls})[0] AS topFouler
RETURN team.name, topFouler.fouler.name, topFouler.fouls;
Most fouled against for each team
match (match:Match {id: "32683310"})<-[:COMMITTED_IN_MATCH]-(foul)<-[:COMMITTED_FOUL]-(fouler),
      (fouler)-[:MADE_APPEARANCE]-(app)-[:IN_MATCH]-(match),
      (app)-[:FOR_TEAM]->(team)
WITH team, fouler, COUNT(*) AS fouls
ORDER BY team.name, fouls DESC
WITH team, COLLECT({fouler:fouler, fouls:fouls})[0] AS topFouler
RETURN team.name, topFouler.fouler.name, topFouler.fouls

becomes

match (match:Match {id: "32683310"})<-[:IN_MATCH]-(app:Appearance)<-[:COMMITTED_AGAINST]->(),
      (app)-[:FOR_TEAM]->(team),
      (fouled)-[:MADE_APPEARANCE]->(app)
WITH team, fouled, COUNT(*) AS fouls
ORDER BY team.name, fouls DESC
WITH team, COLLECT({fouled:fouled, fouls:fouls})[0] AS topFouled
RETURN team.name, topFouled.fouled.name, topFouled.fouls

The early queries are made more complicated by the refactoring but the latter ones are slightly simpler. I think we need to hook some more events onto the appearance node to see whether this refactoring is worthwhile or not.

Removing the old structure

Holding judgement for now, let’s look at how we’d remove the old structure – the final step in this refactoring:

match (match:Match {id: "32683310"})<-[oldRel:COMMITTED_IN_MATCH]-(foul:Foul)
DELETE oldRel
match (player:Player)<-[oldRel:COMMITTED_AGAINST]-(foul:Foul)
DELETE oldRel
match (player:Player)-[oldRel:COMMITTED_FOUL]->(foul:Foul)
DELETE oldRel

Hopefully you can see how you’d go about refactoring your own graph if you realise the model isn’t quite what you want.

Any questions/thoughts/suggestions let me know!

Categories: Blogs

Neo4j: BBC football live text fouls graph

Mark Needham - Sat, 05/16/2015 - 23:13

I recently came across the Partially Derivative podcast and in episode 17 they describe how Kirk Goldsberry scraped a bunch of data about shots in basketball matches then ran some analysis on that data.

It got me thinking that we might be able to do something similar for football matches and although event based data for football matches only comes from Opta, the BBC does expose some of them in live text feeds.

We’ll start with the Champions League match between Barcelona and Bayern Munich from last Tuesday.

2015 05 16 23 10 43

Our first task is to extract the events that happened in the match along with the players involved. After we’ve got that we’ll generate a Neo4j graph and see if we can find some interesting insights.

I find the feedback cycle with this type of work is dramatically improved if we have the source data available locally so the first step was to get the BBC web page downloaded:

$ wget http://www.bbc.co.uk/sport/0/football/32683310

Next we need to write a scraper which will extract all the events. We want to get an array containing one entry for each event, where the following is an example of an event:

2015 05 16 22 19 00

HTML-wise it looks like this:

2015 05 16 22 20 28

4709393221 bddd85c64e z

Image courtesy of William Brawley

I do most of my scraping work in Python so I used the Beautiful Soup library with the soupselect wrapper to get the data into CSV format ready to import into Neo4j.

It was mostly a straight forward job of finding the appropriate CSS tag and pulling out the values although the way fouls are described in the page is a bit strange – sometimes the person fouled comes first row and the fouler comes on the next line and sometimes vice versa.

Luckily the two parts of the foul can be joined together by matching the time which made life easier.

The full code for the scrapper is on github if you want to play with it.

This is what the resulting CSV file looks like:

$ head -n 10 data/events.csv 
matchId,foulId,freeKickId,time,foulLocation,fouledPlayer,fouledPlayerTeam,foulingPlayer,foulingPlayerTeam
32683310,3,2,90:00 +0:40,in the defensive half.,Xabi Alonso,FC Bayern München,Pedro,Barcelona
32683310,9,8,84:38,on the right wing.,Rafinha,FC Bayern München,Pedro,Barcelona
32683310,12,13,83:17,in the attacking half.,Lionel Messi,Barcelona,Sebastian Rode,FC Bayern München
32683310,15,14,82:43,in the defensive half.,Sebastian Rode,FC Bayern München,Neymar,Barcelona
32683310,17,18,80:41,in the attacking half.,Pedro,Barcelona,Xabi Alonso,FC Bayern München
32683310,22,23,76:31,in the defensive half.,Neymar,Barcelona,Rafinha,FC Bayern München
32683310,25,26,75:03,in the attacking half.,Lionel Messi,Barcelona,Xabi Alonso,FC Bayern München
32683310,31,30,69:37,in the attacking half.,Bastian Schweinsteiger,FC Bayern München,Dani Alves,Barcelona
32683310,36,35,63:27,in the attacking half.,Robert Lewandowski,FC Bayern München,Ivan Rakitic,Barcelona

Now it’s time to create a graph. We’ll aim to massage the data into this model:

2015 05 16 22 50 32

Next we need to write some Cypher code to get the CSV data into the graph. The full script is here, a sample of which is below:

// match
LOAD CSV WITH HEADERS FROM "file:///Users/markhneedham/projects/neo4j-bbc/data/events.csv" AS row
MERGE (:Match {id: row.matchId});
 
// teams
LOAD CSV WITH HEADERS FROM "file:///Users/markhneedham/projects/neo4j-bbc/data/events.csv" AS row
MERGE (:Team {name: row.foulingPlayerTeam});
 
LOAD CSV WITH HEADERS FROM "file:///Users/markhneedham/projects/neo4j-bbc/data/events.csv" AS row
MERGE (:Team {name: row.fouledPlayerTeam});
 
// players
LOAD CSV WITH HEADERS FROM "file:///Users/markhneedham/projects/neo4j-bbc/data/events.csv" AS row
MERGE (player:Player {id: row.foulingPlayer + "_" + row.foulingPlayerTeam})
ON CREATE SET player.name = row.foulingPlayer;
 
// appearances
LOAD CSV WITH HEADERS FROM "file:///Users/markhneedham/projects/neo4j-bbc/data/events.csv" AS row
MATCH (match:Match {id: row.matchId})
MATCH (player:Player {id: row.foulingPlayer + "_" + row.foulingPlayerTeam})
MATCH (team:Team {name: row.foulingPlayerTeam})
 
MERGE (appearance:Appearance {id: player.id + " in " + row.matchId})
MERGE (player)-[:MADE_APPEARANCE]->(appearance)
MERGE (appearance)-[:IN_MATCH]->(match)
MERGE (appearance)-[:FOR_TEAM]->(team);
 
// fouls
LOAD CSV WITH HEADERS FROM "file:///Users/markhneedham/projects/neo4j-bbc/data/events.csv" AS row
 
MATCH (foulingPlayer:Player {id:row.foulingPlayer + "_" + row.foulingPlayerTeam })
MATCH (fouledPlayer:Player {id:row.fouledPlayer + "_" + row.fouledPlayerTeam })
MATCH (match:Match {id: row.matchId})
 
MERGE (foul:Foul {eventId: row.foulId})
ON CREATE SET foul.time = row.time, foul.location = row.foulLocation
 
MERGE (foul)<-[:COMMITTED_FOUL]-(foulingPlayer)
MERGE (foul)-[:COMMITTED_AGAINST]->(fouledPlayer)
MERGE (foul)-[:COMMITTED_IN_MATCH]->(match);

We’ll use neo4j-shell to execute the script:

$ ./neo4j-community-2.2.1/bin/neo4j-shell --file import.cql

Now that we’ve got the data into Neo4j we need to come up with some questions to ask of it. I came up with the following but perhaps you can think of some others!

  • Where do the fouls happen on the pitch?
  • Who made the most fouls?
  • Who was fouled the most?
  • Who fouled who the most?
  • Which team fouled the most?
  • Who’s the worst fouler in each team?
  • Who’s the most fouled in each team?
Where do the fouls happen?
match (match:Match)<-[:COMMITTED_IN_MATCH]-(foul)
RETURN foul.location AS location, COUNT(*) as fouls
ORDER BY fouls DESC;
 
+----------------------------------+
| location                 | fouls |
+----------------------------------+
| "in the defensive half." | 12    |
| "in the attacking half." | 12    |
| "on the right wing."     | 3     |
| "on the left wing."      | 3     |
+----------------------------------+
4 rows
Who fouls the most?
match (match:Match)<-[:COMMITTED_IN_MATCH]-(foul)<-[:COMMITTED_FOUL]-(fouler)
RETURN fouler.name AS fouler, COUNT(*) as fouls
ORDER BY fouls DESC
LIMIT 10;
 
+------------------------------+
| fouler               | fouls |
+------------------------------+
| "Rafinha"            | 4     |
| "Pedro"              | 3     |
| "Medhi Benatia"      | 3     |
| "Dani Alves"         | 3     |
| "Xabi Alonso"        | 3     |
| "Javier Mascherano"  | 2     |
| "Thiago Alcántara"   | 2     |
| "Robert Lewandowski" | 2     |
| "Sebastian Rode"     | 1     |
| "Sergio Busquets"    | 1     |
+------------------------------+
10 rows
Who was fouled the most?
// who was fouled the most
match (match:Match)<-[:COMMITTED_IN_MATCH]-(foul)-[:COMMITTED_AGAINST]->(fouled)
RETURN fouled.name AS fouled, COUNT(*) as fouls
ORDER BY fouls DESC
LIMIT 10;
 
+----------------------------------+
| fouled                   | fouls |
+----------------------------------+
| "Robert Lewandowski"     | 4     |
| "Lionel Messi"           | 4     |
| "Neymar"                 | 3     |
| "Pedro"                  | 2     |
| "Xabi Alonso"            | 2     |
| "Andrés Iniesta"         | 2     |
| "Rafinha"                | 2     |
| "Bastian Schweinsteiger" | 2     |
| "Sebastian Rode"         | 1     |
| "Sergio Busquets"        | 1     |
+----------------------------------+
10 rows
Who fouled who the most?
match (match:Match)<-[:COMMITTED_IN_MATCH]-(foul)-[:COMMITTED_AGAINST]->(fouled),
      (foul)<-[:COMMITTED_FOUL]-(fouler)
RETURN fouler.name AS fouler, fouled.name AS fouled, COUNT(*) as fouls
ORDER BY fouls DESC
LIMIT 10;
 
+--------------------------------------------------------+
| fouler              | fouled                   | fouls |
+--------------------------------------------------------+
| "Javier Mascherano" | "Robert Lewandowski"     | 2     |
| "Dani Alves"        | "Bastian Schweinsteiger" | 2     |
| "Xabi Alonso"       | "Lionel Messi"           | 2     |
| "Rafinha"           | "Neymar"                 | 2     |
| "Rafinha"           | "Andrés Iniesta"         | 2     |
| "Dani Alves"        | "Xabi Alonso"            | 1     |
| "Thiago Alcántara"  | "Javier Mascherano"      | 1     |
| "Pedro"             | "Juan Bernat"            | 1     |
| "Medhi Benatia"     | "Pedro"                  | 1     |
| "Neymar"            | "Sebastian Rode"         | 1     |
+--------------------------------------------------------+
10 rows
Which team fouled the most?
match (match:Match)<-[:COMMITTED_IN_MATCH]-(foul)<-[:COMMITTED_FOUL]-(fouler),
      (fouler)-[:MADE_APPEARANCE]-(app)-[:IN_MATCH]-(match),
      (app)-[:FOR_TEAM]->(team)
RETURN team.name, COUNT(*) as fouls
ORDER BY fouls DESC
LIMIT 10;
 
+-----------------------------+
| team.name           | fouls |
+-----------------------------+
| "FC Bayern München" | 18    |
| "Barcelona"         | 12    |
+-----------------------------+
2 rows
Worst fouler for each team?
match (match:Match)<-[:COMMITTED_IN_MATCH]-(foul)<-[:COMMITTED_FOUL]-(fouler),
      (fouler)-[:MADE_APPEARANCE]-(app)-[:IN_MATCH]-(match),
      (app)-[:FOR_TEAM]->(team)
WITH team, fouler, COUNT(*) AS fouls
ORDER BY team.name, fouls DESC
WITH team, COLLECT({fouler:fouler, fouls:fouls})[0] AS topFouler
RETURN team.name, topFouler.fouler.name, topFouler.fouls;
 
+---------------------------------------------------------------+
| team.name           | topFouler.fouler.name | topFouler.fouls |
+---------------------------------------------------------------+
| "FC Bayern München" | "Rafinha"             | 4               |
| "Barcelona"         | "Pedro"               | 3               |
+---------------------------------------------------------------+
2 rows
Most fouled against for each team
match (match:Match)<-[:COMMITTED_IN_MATCH]-(foul)-[:COMMITTED_AGAINST]-(fouled),
      (fouled)-[:MADE_APPEARANCE]-(app)-[:IN_MATCH]-(match),
      (app)-[:FOR_TEAM]->(team)
WITH team, fouled, COUNT(*) AS fouls
ORDER BY team.name, fouls DESC
WITH team, COLLECT({fouled:fouled, fouls:fouls})[0] AS topFouled
RETURN team.name, topFouled.fouled.name, topFouled.fouls;
 
+---------------------------------------------------------------+
| team.name           | topFouled.fouled.name | topFouled.fouls |
+---------------------------------------------------------------+
| "FC Bayern München" | "Robert Lewandowski"  | 4               |
| "Barcelona"         | "Lionel Messi"        | 4               |
+---------------------------------------------------------------+
2 rows

So Bayern fouled a bit more than Barca, the main forwards for each team (Messi/Lewandowski) were the most fouled players on the pitch and the fouling was mostly in the middle of the pitch.

I expect this graph will become much more interesting to query with more matches and with the other event types as well but I haven’t got those scraped yet. The code is on github if you want to play around with it and perhaps get the other events into the graph.

Categories: Blogs

The Art of Splitting User Stories

Agile Game Development - Sat, 05/16/2015 - 17:52
Splitting stories—the act of breaking up large stories into smaller ones—is as much art as practice.

The strategy of splitting stories is to split them as late as possible to fit in a sprint just before the sprint that will take them on.  Think of it as playing the classic arcade game Asteroids: you only want to break up the large asteroids when they are an immediate threat to you or when you've cleared out the smaller, more dangerous rocks…break up too many of them too soon and the resulting cloud of small asteroids will kill you.

Similarly, if you break up a bunch of epics into a large number of sprint-sized stories, the resulting backlog bloat will cloud vision and diminish your response to change.

There are a lot of good tips on how to split stories, like this one.  In this post, I'm going to suggest some of the more common splitting techniques used for video game development.

A couple goals exist for splitting stories:
  1. Each of the split stories should have some value to the user.
  2. The split stories should be able to be prioritized against one another.
Below are a few of the common strategies and examples for splitting epics along with some reasoning for them.  The examples illustrate single suggested splits, but aren't the only ways the epics could be split.

Split along research/prototype dependencies: 
Example: As a player, I want to react to impacts.
A suggested split:
⁃ As a designer, I want to know where an impact occurs.
⁃ As a player, I want to react based on the impact location.

Reasoning: The first story allows some research on collision physics and helps the team and designer experiment with different strategies. (e.g. would a simple strategy of having high, mid and low collision volumes be good enough, or do we need something more complex?)

Split along conjunctions:
Example: As a player I want to smash wooden crates and doors open.
A suggested split:
⁃ As a player, I want to smash wooden crates open.
⁃ As a player, I want to smash wooden doors open.

Reasoning: Do the simplest or most important first.  Also note that the first of anything is usually more expensive, so this makes it easier to see costs by sizing these stories individually.

Split by progression:
Example: As a player, I want to be coached on the rules of poker and ways I can improve my home game.
A suggested split:
⁃ As a player, I want the game to have some first-time user experience (FTUE) tips for playing poker.
⁃ As a player, I want the game to give me tips for improving my home game.
⁃ As a player, I want the game to evaluate my hand strength and suggest the next move.
Reasoning: This split details some of the "ways" the player can improve with a progression of stories that are a bit more specific.

Other splitting strategies include:
⁃ Split by acceptance criteria.  Acceptance criteria often make perfect stories themselves.
⁃ Split by discipline capacity.  E.g. if you don't have enough animation support, then you might have to split off a "Add polished animations to..." story for the next sprint (after you've done everything else to avoid this).

There is one consistent splitting technique that teams new to agile have a hard time avoiding.  This is splitting stories along component design or discipline effort.  Teams new to agile are used to breaking up features into a plan to develop all the “parts”, which will become useful sometime in the future.

Example: As a player, I want a third person camera that can transition smoothly between a loose camera in open spaces and a tight camera in small spaces that avoids collision with the surrounding geometry.  

A poor split would be:

⁃ As a programmer, I want to create a camera manager that will handle transitions between all future cameras.
⁃ As a programmer, I want to implement a camera collision system that will avoid camera collisions or occlusions from the surrounding geometry.
⁃ Etc.

The reason these are poor splits reflects agile values and principles that building emergent, working software is better than following a never-perfect-yet-detailed plan which delays that emergence.  A better approach to splitting might be:
⁃ As a player, I have a simple polar camera that follows me around.
⁃ As a player, I want the distance between me and the polar camera to be based on how close the surrounding geometry is.
⁃ As a player, I want the the camera to have a collusion box around it to avoid going into surrounding geometry.
⁃ As a player, I want the camera to linearly interpolate its position as a rotate.
⁃ Etc.

There are several advantages to this:
  1. The rest of the team gets a camera sooner.  This allows work and feedback to continue (rather than finding “something else” to work on in parallel).  
  2. Unless you are in the position of being 100% certain of the requirements of the camera system (something I’ve rarely seen on a team), there’s less of a chance that the system developed will need major rework.
  3. Emergent development will often save a lot of time by identifying what we don't have to build.   I relate a story of this in another article.
I’m fully aware that there are some out there rebelling at this idea.  They’ll say “just give me the spec, let me go into my office, put on my headphones, and bang out the camera system…it’ll be much faster”. This is a very common attitude from cultures with more silo’d disciplines.  It’s very hard to for such organizations to shift the mindset from one whose primary goal is to crank out components at the highest possible velocity, to one which cranks out features of the highest possible value.

This is not to say there aren't features that can be fully planned and executed without iteration.  The trick is to not fool ourselves into the belief that we can plan away uncertainty.


Categories: Blogs

How fast you can change?

Manage Well - Tathagat Varma - Sat, 05/16/2015 - 04:42
Speed is not how fast you can drive and deliver. It is how fast you can change and adapt. And life and product development have more hairpin bends than you think...
Categories: Blogs

Agile goes beyond Epic Levels

Xebia Blog - Fri, 05/15/2015 - 18:10

IMG_5514A snapshot from my personal backlog last week:

  • The Agile transformation at ING was frontpage news in the Netherlands. This made us even more realize how epic this transformation and assignment actually is.
  • The Agile-built hydrogen race car from the TU Delft set an official track record on the Nurburgring. We're proud on our guys in Delft!
  • Hanging out with Boeings’ Agile champs at their facilities in Seattle exchanging knowledge. Impressive and extremely fruitful!
  • Coaching the State of Washington on their ground breaking Agile initiatives together with my friend and fellow consultant from ScrumInc, Joe Justice.

One thing became clear for me after a week like this: Something Agile is cookin’.  And it’s BIG!

In this blog I will be explaining why and how Agile will develop in the near future.

Introduction; what’s happening?

Human kind is currently facing the biggest era change since the 19th Century. Our industries, education, technologies and society are simply not compliant anymore with today’s and tomorrows needs.  Some systems like healthcare and the economy are that broken they actually should be to be reinvented again. Everything has just become too vulnerable and complex. Just Quantitative-easing“lubricating the engine” like quantitive easing the economy, are no sustainable solutions anymore. Like Russell Ackoff already said, you can only fix a system as a whole not only by separate parts.

This reinvention will only succeed when we are able to learn and adjust our systems very rapidly.  Agile, Lean and a different way of organizing our selfs, can make this reality.  Lean will provide us with the right tools to do exactly what’s needed, nothing more, nothing less.  But applying Lean for only efficiency purposes will not bring the innovations and creativity we need.  We also need an additional catalyst and engine: Agile. It will provide us with the right mindset and tools to innovate, inspect and adapt very fast.  And finally, we must reorganize ourself more on cooperation not on directive command and control. This was useful in the industrial revolution, not in our modern complex times.

Agile’s for everything

Unlike most people think, Agile is not only a software development tool. You can apply it to almost everything. slider_Forze_VI_ValkenburgFor example, as Xebia consultants we've successfully coached Agile and Lean non-IT initiatives in Marketing, Innovation, Education, Automotive, Aviation and non-Profit organizations. It just simply works! And how. A productivity increase of 500% is no exception. But above all, team members and customers are much happier.

Agile's for everybody

At this moment, a lot of people are still unconsciously addicted to their patterns and unaware about the unlimited possibilities out there.  It’s like having a walk in the forrest.  You can bring your own lunch like you always do, but there are some delicious fruits out there for free!  Technology like 3D printing offer unlimited possibilities straight on your desk, where only a few years a go, you needed a complicated, million dollar machine for this. The same goes for Agile. It’s open source and out there waiting for you. It will also help you getting more out of all these new awesome developments!

The maturity of Agile explained

Until recently, most agile initiatives emerged bottom up, but stalled on a misfit between Agile and (conventional) organizations.  Loads of software was produced, but could not be brought to production, simply because the whole development chain was not Agile yet. Tools like TDD and Continuous Integration improved the situation significantly, but dependencies were still not managed properly most of the time.

Screen Shot 2015-05-13 at 7.53.15 PM

The last couple of years, some good scaled agile frameworks like LeSS and SAFe emerged. Managing the dependencies better, but not directly encouraging the Agile Mindset and motivation of people.  In parallel, departments like HR, Control and Finance were struggling with Agile. There was a scaled agile framework implemented, but the hierarchical organization structure was not adjusted creating a gap between fast moving Agile teams and departments still hindered by non-Agile procedures, proceses and systems.

Therefor, we see conventional organizations moving towards a more Agile, community based model like Spotify, Google or Zappos.  ING is now transforming towards to a similar organization model.

Predictions for the near future

My expectation is that we will see Agile transformations continue on a much wider scale.  For example, people developing their own products in an agile fashion while using 3D printing.  Governments will use Agile and Holacracy for solving issues like the reshaping the economic system together with society. Or like I have observed last week, the State of Washington government using these techniques successfully in solving the challenges they're facing.

For me, it currently feels like the early Nineties again when the Internet emerged.  In that time I explained to many people the Internet would be like electricity for them in the near future.  Most people laughed and stated it was just a new way of communication.  The same applies now for the Agile Mindset.   It's not just a hype or a corporate tool. It will reshape the world as we know it today.

 

 

 

 

Categories: Companies

Failure Modes of an Agile Transformation: Workflow

Rally Agile Blog - Fri, 05/15/2015 - 14:00

In my previous post on the first three Agile transformation failure modes, I focused on the LEADERSHIP aspects of failure: lack of real executive sponsorship, failure to transform leader behaviors, and refusal to change organizational structures.

These next three fail modes are failures in WORKFLOW: effectively making work decisions flow, setting work boundaries, and factoring in the costs associated with work across distributed teams.

4. No Business View of the Value Stream

 photo via Flickr Commons 

Several years ago, I had the good fortune to work with Hendrik Esser in a weekend workshop. Hendrik and I converged on the value stream: What does it look like in traditional organizations? What might we want to be different? And how does an Agile transformation support or impede this new perspective? What we’d seen in our respective careers was frighteningly similar. Most organizations embraced silos, to the degree that even as they were adopting Agile practices, they did so in “agile silos.”

There was no view across the value stream. That is, these organizations weren’t tracking the “concept-to-cash” flow of value through a system.

Successful Agile transformations ask us to accept a few basic truths and practices about value streams:

  • You’ve got one; go find it.
  • Map your system at whatever level of detail best articulates your sense of handoffs and bottlenecks.
  • Start where you are in the value stream; taking on an entire system will exhaust you and your teams, which will lead to abandonment of the Agile transformation effort.
  • Every system has one primary constraint; find yours, crush it, and then look for the next primary constraint that emerges. Rinse and repeat.
  • Be ruthless in your vision to expand the boundaries of your value stream: upstream from the neighbor processes that feed your work, and downstream where you feed your work into your neighbor process.
  • Include everyone in identifying the value stream, its bottlenecks, and its potential flow.
  • Broaden commitment up and down the stream, not in localized silos.
5. Failure to Decentralize Control

 photo via Flickr, CC

In my previous post about leadership-themed failure modes, I mentioned the work of Robert Greenleaf. Among his 10 attributes of a servant leader, two of my favorites are acceptance and withdrawal. What do these mean to you and how to do they impact the success or failure of an Agile transformation? To me, these make me think of the experience as a manager or being managed. Here’s a little story about managing me.

Across my tech career I’ve worked for a variety of managers. In one job, I worked in a small group analyzing various methodology approaches for software development. This was exciting: it was my first foray into a break-up with waterfall approaches. Though rather shy (yes, me!) I was a sponge, passionate to contribute and engage. In one meeting, a particular issue was raised that related to the work I’d been doing and I offered up what I thought was a relevant solution. At a break in the meeting, my manager pulled me aside into a separate room. “Why did you disagree with me in there?” he inquired. “Don’t you trust that whenever I make a decision I’ve thought of all other possibilities? You’re disrespecting me.”

Huh?

I’d like to say that was the only manager who has ever needed to have complete control over decisions, but you know better. In most cases, there are a few dynamics at work here:

  • The manager’s sense of significance is somehow wrapped up in the decision-making control.
  • Due to this strict, centralized control, decisions are not as informed as they could be.
  • Waiting for a decision because of centralized control wastes time and human potential.
  • This control style ultimately lowers morale and fosters cynicism.

Agile transformations fail when we don’t pay attention to whether we centralize or decentralize control. Greenleaf would characterize this as an inability to recognize when to accept the decisions of the team and when to withdraw your control so that the team owns decisions that impact their way of working. Servant leaders don’t relinquish all control; rather, they recognize the value in releasing control when all concerned are better served. They maintain centralized control when they see that this is in the best service to their teams.

Don Reinertsen talks and writes well about this (see The Principles of Product Development Flow.) You can use some simple guidelines to steer you towards appropriate control modes. Long-term decisions with extended impact that include dependencies across many constituents warrant a centralized mode of control. Short-term and low-impact decisions merit decentralized control. In Don’s words, think of these two principles:

  • The Scale Principle: “Centralize control for problems that are infrequent, large, or that have significant economies of scale.”
  • The Perishability Principle: “Decentralize control for problems and opportunities that age poorly.”

Agile transformations require continuous learning, and servant leadership is no exception. When Don talks about these domains of control, one thing is certain: this is not about controlling the worker. As he says,

“Watch the work, not the worker.”

Decentralize to move decisions closer to the worker, the one who best knows the work. Centralize control as a service to the flow of value. A bias toward decentralization helps leaders and teams better converge.

6. Unwillingness to Address Illusions Around Distributed Teams

More and more I’m seeing organizations with distributed teams: wide dispersement of teams across many time zones, and even 80/20 rules that guide projects to employ 80% offshore teams with only 20% of teams located in the same building (or the same city.)

Want to fail at your Agile transformation? It’s easy. Follow these simple rules for distributed teams: set up a complex geographic maze based on the economics of resource utilization; ensure a time zone difference between 7-11 hours; rely heavily on emails and large documents (especially detailed test plans) for your communication; and definitely don’t invest in bringing people together to collaborate or train.

Organizations in this distributed bind have essentially made deals with the devil, trading off fundamental Agile success principles like face-to-face collaboration for the promise of speed and lower costs (but don’t worry, it’s still Agile!)

 photo via Flickr Commons

Really?

How do you monitor the value stream and the flow of work? What working agreements and communication approaches have you implemented that work across cultures? How fast is the feedback? Where is the cost-of-delay calculation in your ROI equation? And perhaps most importantly, who’s representing the challenges of the distributed teams to the rest of the organization? Who listens to them when the constraints become overwhelming?

Here are some changes you can embrace to deal with the distress introduced by the distributed model:

  1. Hire a coach who’s well-steeped in distributed team success.
  2. Ensure all team members receive the same Agile training; for maximum effectiveness, get everyone in the same room for the training.
  3. Invest in high-definition, large-screen video technologies. Accompany these with a top-notch sound system that figuratively brings all the voices into the room.
  4. Have a facilitator in each location when teams plan their dependent delivery commitments.
  5. Whether using audio-only or adding video, use facilitation techniques that ensure all insights are welcome (small-group brainstorming, round robin check-ins, frequent breaks.)
  6. Invest in technologies that support transparent workflow communication — for example, Flowdock.
  7. Maintain a regular cadence of visits across all geographies and all roles.
  8. Build working agreements that support core hours for availability, or alternative solutions for quick turn-around feedback.
  9. Trade or share the burden of dealing with broad time zone differences.
  10. Engage the executive sponsor in regular visits to all locations.

Finally, here’s the one, most important practice I believe is critical to a distributed team delivery model:

Eliminate distributed teams.

That’s it. This yields the biggest success stories I’ve seen with distributed teams. Nearly everyone I talk to says this isn’t a choice for them; yet if they knew the amount of waste built into the distributed delivery model, they’d realize the irony of thinking it’s saving them value.

Stay tuned for the next in the series on the 12 Agile Transformation Failure Modes!

  Jean Tabaka
Categories: Companies

Growing Kanban in Three Dimensions

Improving projects with xProcess - Fri, 05/15/2015 - 10:00
Kanban systems can work at different scales and in widely different contexts. Indeed any organisation that delivers discrete packages of value ("work items") and which is interested in maximising the value and timeliness of its delivery, can analyse and improve its performance using the Kanban method. 
Kanban systems can grow - in fact in most cases it's much better that they grow than a massive process change is made suddenly across a whole organisation. "Big bangs" tend to be quite destructive, even if they could clear the way for something new. There are three dimensions in which Kanban systems grow:

  • Width-wise growth: encompassing a wider scope of the lifecycle of work items than the typical "to do - doing - done" a single division of the process. It can cover from the idea to real value - or "concept to cash", though cash may come before or after the realisation of real value.
  • Height-wise growth: by considering the hierarchy of items that make up valuable deliveries, each level of the hierarchy having differing flow characteristics. (This dimension use the "scale-free" nature of Kanban, the same principles and practices apply whatever the size of the work item.)
  • Depth-wise growth: not only depth of understanding but depth of penetration through the full set of services required by the organisation to deliver value. (Sometimes referred to as "Scaling by not scaling" or "service-oriented Kanban", the approach here connects multiple services at the same level through feedback loops that balance the capacity of the various kanban systems.)

We'll look at each of these dimensions in upcoming articles. Which dimension to grow first will depend on context and the motivations for change. Any change needs to pay for itself with improvements in the flow of value, so asking "why?" is a more important first question than "what?".

When you come across a good idea ("agile" in general springs to mind at this point) it is very tempting to sweep away whatever you were doing before you were converted to the new idea, and start doing it everywhere. It should not come as a surprise to those who do this, that very soon a new idea will come along. With the poor results from mass conversion to the caricature of the original idea you adopted, the same cycle will be repeated. Instead grow the changes organically.

Try this: start small; understand the ideas as you assimilate them; grow what works and understand what doesn't work; work out why. Success will follow.

Acknowledgement: Thanks to +Pawel Brodzinski for the discussions on Portfolio Kanban... and one of the graphics on the top floor of the above diagram.
Categories: Companies

Introducing: Pivotal Tracker for iOS 3.0! (Part 1)

Pivotal Tracker Blog - Thu, 05/14/2015 - 21:00

We hope you’re sitting down, because this one’s a doozy of an update. In this post, we’ll take the wide-angle view of what you can expect in the iOS 3.0 update. In Part 2, we’ll zoom in to the finer points of the new features.

phone-sidemenu (1)For now, let’s get down and dirty. There are three main screens you’ll see while working through the new Tracker app: the Side menu, the Project and its panels, and the Story.

Side menu

The Side menu—aptly named, as it sits off to the left side—holds Settings, Notifications, and all of your Projects. You can access the Side menu from the Project.

Project

The Project panels have gone all topsy-turvy. You can now access them from the top of the screen instead of the bottom. From these panels, you can see all your Stories. You can still press and hold a Story to move it from panel to panel (e.g., from the Icebox to Backlog) and search the project. Slide, slide, slippety slide the panel names right and left to access all panel types.phone-panelpicker

Story

We’ve created three tabs within the Story to ease scrolling—Details, Activity, and Tasks. The Details tab will show you the description, Story type, estimate, and similar information. Activity is where comments, attachments, and git commits live. And Tasks . . . well, that’s where you’ll find tasks for the Story. As an aside, Notifications will now drop you into the relevant tab of a Story to be more direct in context; if you receive a mention, the Notification will drop you into that Story’s Activity for easy access to the conversation.

Auto updatesphone-story (1)

And on top of all that, auto updates! That’s right, the app now keeps itself up to date, like any good sentient helper. If you get anxious, feel free to pull-to-refresh—nobody likes to wait.

Boom!

We’re looking forward to expanding on this update frequently; it’s a fresh start, and one we’re excited about. You should also know that adding functionality to Search, adding gestures, squeezing in Workspaces, and a few new ideas are primed in our heads and will be at your fingertips soon.

Check it out in the App Store! We appreciate your usage and your feedback. As you get familiar with this new version, please report bugs, give feedback, and send digital beers to tracker@pivotal.io. And don’t forget to rate us in the App Store!

The post Introducing: Pivotal Tracker for iOS 3.0! (Part 1) appeared first on Pivotal Tracker.

Categories: Companies

The future of agile: changing the world of work

Agile For All - Bob Hartman - Thu, 05/14/2015 - 20:07

I gave a presentation at the Scrum Gathering in Phoenix AZ about the historic context of Agile and Scrum, and where we are headed next. While agile practices like Scrum and XP are fairly mainstream in software companies, Agile as a mindset is still in the early adopter phase in the business world at large. What can we do to help it “cross the chasm” to broader adoption?

Below are the slides and the talk track. The presentation was in Pecha Kucha format – 20 slides, 20 seconds each on an auto-advance timer, which was a fun challenge to put together!

The Context For an Agile Mindset 1. So Much Shouting

Every day, Social Media and Email Lists are overrun with agilists virtually shouting at each other about why their favorite techniques are the best, and this makes me sad. We are in the middle of a complete transformation of our economy and the role of people in it, and these debates often end up splintering our community and confusing newcomers.

2. The Economic Shift

Technology has had a major impact on the world’s workforce. In the last 100 years we have seen factory automation and robotics displace millions of physical laborers, leading to the knowledge work era. In the last 20 years, the proliferation of computers and the internet have displaced millions of knowledge workers, causing the current shift to what is referred to as the Creative Economy.

3. The Creative Economy

The creative economy is based upon the uniquely human capability to generate new ideas, new technology, and works of art. In the creative economy, workers are driven by autonomy, mastery and purpose. When these conditions are present, innovation thrives, customers are delighted, and workers collaborate to unite science and art to solve deep human problems.

Agile: The Past 4. The History

Unfortunately, most organizations are not set up with the needs of creative workers in mind. Our management approach comes from the 100 year old concepts of Frederick Winslow Taylor, and our org structures go back even further to the hierarchy of Julius Caesar’s armies. These approaches were breakthrough ideas at the time, but are not effective approaches in Creative Economy organizations.

5. The Background of Lean

In the second half of the last century, the big three auto makers dominated the market with massive workforces, inventories, and bottom lines. Toyota, a scrappy automated loom company in Japan was just entering the auto industry. There was no way they could afford to match the economies of scale of the U.S. manufactures. Their only competitive advantages could be speed and quality.

6. The Birth of Lean

The American expert W. Edwards Deming was working with several Japanese business leaders teaching them the Total Quality Management approach. A rising executive at Toyota named Taiichi Ohno embraced Demings’ ideas and expanded upon them to create the Toyota Production System, what we now refer to as Lean Manufacturing.

7. Lean Principles

Three key concepts from Lean that are helpful in the Creative Economy are: 1. Trusting the workers to decide how to do the work, rather than relying on managers to choose the process, 2. Focusing on continuous improvement over relying on prescribed best practices. 3. Viewing value from the customer’s perspective, instead of from the company’s profit motive.

8. The Context for Agile

Let’s fast forward to the 90s, and transition to the world of software. The Chaos reports from that time do a nice job of illustrating the somewhat disappointing state of the industry. The large majority of the features we were building were rarely or never used, and most projects were at least challenged and at worst completely failing to deliver on their stated goals.

9. The Birth of Scrum

In 1984, two business professors published a paper sharing the results of their study of several lean companies that were getting great results. Jeff Sutherland combined their findings with ideas from Complexity science and other areas to give birth to scrum. Notice that Lean was a direct ancestor of agile. Scrum incorporates and builds on several lean concepts.

10. Scrum Principles

Three key ideas from scrum that are helpful in the Creative Economy are: 1. Focusing on building small, high performing, cross-functional teams, rather than relying on and rewarding individual heroics. 2. Developing products incrementally, where each increment is potentially releasable so that we can get good feedback. 3. A new type of role that uses a servant leader approach to help organizations to continuously improve.

Agile: The Present 11. The Context of Lean Startup

Scrum and related agile approaches did help many organizations get better at delivering working software. One such company called IMVU was using full XP practices. However, they were still failing to succeed in the market. While agile had helped them build things fast, and build them the right way, it hadn’t helped them build the right thing and deliver value to their customers.

12. The Birth of Lean Startup

Eric Ries, the CTO at IMVU at the time, was taking some classes at Stanford from noted business professor Steve Blank, whose approach had helped many startups pivot towards a great product/market fit. Eric combined the ideas from Blank’s customer discovery process with Lean, Agile, and Open Source, into what he called “Lean Startup”.

13. Lean Startup Principles

Three key ideas from Lean Startup that are helpful in the Creative Economy are: 1. A focus on working directly with potential customers to develop empathy for their problems and better understand their unmet needs, 2. A rapid iterative approach to discovering what solutions we might provide to meet those needs, 3. And a shift from thinking about requirements, to testing our hypotheses in the market prior to investing in developing the product.

14. Alignment of Ideas

So, while we burn energy debating the merits of these approaches, they are in fact not competing ideas. Lean begat Agile, which begat Lean Startup, they are fully aligned, and, used together, are creative economy enablers, that create an environment where autonomy, mastery, and purpose are not just posters that HR puts up on the wall.

15. Generational Waves

These approaches have emerged in generational waves, about every twenty years. Each new evolution of this mindset enabled us to deal with increasing levels of complexity. When I noticed this 20 year pattern, I began to wonder what might be the next evolution. I discovered some interesting research that shows some emerging and aligned patterns at the organization level. Let’s look at some metaphors:

 

16. Organizational Metaphors

Some organizations see themselves as armies, with strict hierarchy & processes. Others, like machines, where leaders pull levers and the org spits out a result. Agile organizations, like families, strive to balance the needs of everyone involved. A new pattern is emerging in organizations that see themselves as living systems, evolving without hierarchy towards a shared purpose.

Agile: The Future 17. Teal Principles

Three key concepts from this emerging organizational paradigm are: 1. Anti-fragile organizational patterns, usually completely flat or based on interlocking circles with no managers, 2. A primary focus on achieving a shared, evolutionary purpose, 3. A focus on wholeness, where workers don’t feel like they have to be a different person when they’re at work.

18. The Engagement Problem

A 2014 Gallup polls show that only 31% of US workers are actively engaged at work, and Gallup estimates that the 18% that are actively disengaged cost the US $550 Billion dollars a year in lost productivity. The numbers internationally are worse, and I believe that the engagement problem is a root cause of the world economic crisis.

19. A New Perspective

The organizations of the world are struggling, people are hurting, and it is our responsibility as members of the Agile community, and just as human beings, to broaden our perspective. Combining the concepts of Lean, Agile, Lean Startup, and Organizational Agility provides us with a framework for dramatically improving the level of engagement at work.

20. Change the World of Work

Old thinking does not create new results; we need to shift from ideas of exerting control and achieving financial profit towards enablement and achieving a shared purpose. Only such a mindset shift will give us a fighting chance to fulfill the Scrum Alliance’s Mission to Change the World of Work. I don’t know if the world can wait another 20 years.

What do you think?

Let me know what you think – what can we do to help spread the mindset of Agile – which is all about engagement, wholeness, and adding value in our communities? If you liked this post, please share it using one of the social media buttons below!

The post The future of agile: changing the world of work appeared first on Agile For All.

Categories: Blogs

Another Agile Article on Slashdot – Andy Hunt has Failed, not Agile

Learn more about our Scrum and Agile training sessions on WorldMindware.com

For reference, here is the link to the article on Slashdot called Is Agile Development a Failing Concept?

This article will generate lots of great discussion, but most of it will be ignorant.  My biggest problem with this is that one of the authors of the Agile Manifesto, Andy Hunt, has asserted that Agile just isn’t working out.  My opinion: Andy has failed to have the necessary patience for a decades-long cultural change.  This is a lot like a leader at Toyota saying that lean has failed because 50 years after they started doing it, not everyone is doing it properly yet.  One organization that I know of has been working on changing to Agile for over 10 years and they still aren’t “done”.  That’s actually okay.

Try out our Virtual Scrum Coach with the Scrum Team Assessment tool - just $500 for a team to get targeted advice and great how-to informationPlease share!
facebooktwittergoogle_plusredditpinterestlinkedinmail
Categories: Blogs

Earned Value Management and Agile Processes

Improving projects with xProcess - Thu, 05/14/2015 - 16:40
I've recently been working with a client whose customer requires project reporting using Earned Valued Management metrics (EVM). It made me realise that, since they are also wishing to use agile methods, a paper I wrote back in 2008 could be relevant to them, and maybe a few others. When I looked for it online it was no longer available, so I thought I'd remedy that here. You can access the paper by clicking this link: EVM and Agile Processes – an investigation of applicability and benefits.

EVM is a technique for showing how closely a project is following both its planned schedule and planned costs. It's a superior method to simply reporting time and cost variance, since if the project has slipped but also underspent you cannot tell from the simple variances the degree to which the underspend has caused the slippage. EVM's cost efficiency and schedule efficiency (nothing to do with efficiency by the way!) can tell you this.

However agile methods do not have a fixed scope during their lifecycle and this can make EVM reporting effectively meaningless. The paper explains a technique for using the substitutability of User Stories, estimated in points, for overcoming this problem. If this is relevant to your business environment, I hope you find it useful.

Agile EVM has continued to develop since this paper and you can find more details and further references in the Wikipedia entry here: Earned value management: Agile EVM.

Citation: Andy Carmichael (2008). EVM and Agile Processes – an investigation of applicability and benefits, The 2nd Earned Value Management Conference, NEC, Birmingham UK, 12 March 2008.
Project Manager Today Events. www.pmtoday.co.uk.
Categories: Companies

Knowledge Sharing


SpiraTeam is a agile application lifecycle management (ALM) system designed specifically for methodologies such as scrum, XP and Kanban.