Skip to content

Feed aggregator

Performance Testing in Agile Contexts

TV Agile - Mon, 06/30/2014 - 23:05
Performance Testing has long been conducted as a single experiment against a fully complete and assembled system. Architecture, software, hardware, networks, data states, and workload models are all combined to try and create the most “accurate” simulation possible to produce test results predictive of production experience. Performance testing with this approach can be helpful for […]
Categories: Blogs

New Content: Continuous Integration

Agile Product Owner - Mon, 06/30/2014 - 22:11

For those following SAFe, you are aware that the next release will occur on July 28, 2014. Many of the upcoming changes focus on the expanded portfolio level, as we have described through this recent series of blog posts. This is because we are seeing SAFe applied in larger and larger contexts, and we are seeing new patterns of effective lean-agile portfolio management emerge. (As we learn new things, we record them. As we understand these things, we create better pictures!)

But we’ve been working on the Teams and Program areas as well, because if the teams aren’t agile, nobody is. You can see some of these changes implied in the expanded Code Quality section below. (We described Test-First earlier). We’ve just completed some guidance on Continuous Integration, so we’ve posted here, in SAFe 2.5 guidance, as well.

SAFe 3.0 team with Continuous Integration highlighted.

SAFe 3.0 team with Continuous Integration highlighted.

A special thanks to the Lean Samurai, Inbar Oren, who contributed much of this content.

Categories: Blogs

Hansoft Expands Financing to $10 million with Hasso Plattner Ventures - Mon, 06/30/2014 - 19:44
Hansoft, with its eponymous tool for team collaboration and management in Agile software development today announced it has partnered with venture capital firm Hasso Plattner Ventures to further accelerate growth worldwide.
Categories: Communities

Hansoft Gets $10 million Financing

Scrum Expert - Mon, 06/30/2014 - 19:41
Hansoft, vendor of a tool for team collaboration and management in Agile software development today announced it has partnered with venture capital firm Hasso Plattner Ventures to further accelerate growth worldwide. Hansoft is used for development of products and services by the most demanding teams in industries such as Electronics, Aerospace, Space, Game Development, and Telecom. These are companies who need to accelerate time to market, while improving overall productivity and quality, and Hansoft helps them achieve this by enabling better practices for boosting innovation and scaling Agile software and hardware ...
Categories: Communities

So you’re going to be a ScrumMaster?

Agile Management Blog - VersionOne - Mon, 06/30/2014 - 17:58

Team“Well I am a PM now but they are making me a Scrum Master.” I have heard that statement countless times. Some PMs are happy about the new challenge, some aren’t and some just want to keep making those Benjamins. I have to admit even though being an embedded software engineer by trade the ScrumMaster role is my favorite role on a Scrum team. When asked what is a ScrumMaster I always respond with a servant leader and Agile champion for your team. Yes there are more details but I think simple is the best approach sometimes.

I thought it might be a nice time to talk about some of the qualities that a great PM has and how those look in an Agile environment. PMs tend to get a bad wrap from the Agile world so lets clear the air. I just so happen to have a fiancee that is a PM for a large company. She tries to give me a gantt chart for the week and I hand her story cards daily at breakfast. I asked her to take a survey of her peers which consisted of development, product management and other PMs. They came up with five qualities a PM has, here they are in no particular order.

Issue / Risk Assessment
Issue Resolution
Listening Skills
Manage Expectations of the Business
Process Oriented
On the topic of issue/risk assessment I can help your stress level by letting you know that issue assessment and risk assessment take different routes. Issue assessment gets the chance to happen everyday in team’s standup meeting. Of course if this is urgent it may be brought to your attention sooner. Risk assessment is happening all the time by way of the entire team. Is the sprint going to be completed, what stories might be slipping and what can we do as a team to deal with that.

Issue resolution is a key quality that will carry over. As a ScrumMaster you should do everything in your power to resolve issues or impediments for the team. This not only keeps the team adding value all the time it also builds their trust in you as the ScrumMaster. Issue resolution also requires a good listening skill.

Listening is a skill or quality that I think everyone is challenged with in life. Listening before you speak, listening before you start to think of your response etc. ScrumMasters have to fine tune this skill, even when it makes things uncomfortable at times. Some ScrumMasters tried to lead the team too often with words. I am a fan of posing the question and then letting the silence get uncomfortable enough that someone from the team answers. Which breaks the ice and conversation begins to flow. This also starts the baby step path to the team becoming more and more self organized.

Our next quality, managing expectations of the business, which is easier now! You get to manage it two weeks at a time or however long your sprints are. You will work with product owner to sprint plan and release plan if you are making use of that planning method. Better yet, let’s remove manage and add rank. Ranking is what a product owner is doing with the pieces of the value the business is looking for. Your job as a ScrumMaster may be to help the product owner provide details necessary to get the conversation and collaboration going! Those specs are you are used are gone and I challenge you to ensure the story doesn’t turn into a spec.

Being process oriented is good thing since you have new processes to learn. Scrum as a concept isn’t hard to understand but mastering it can take quite some time. That being said a Scrum team can take a year or more to fully mature. A good ScrumMaster is more than a certification, not to knock them. You will have times when you have to figure out the best way to passively guide the team towards better Agile values. You have to truly believe that what you are saying has value and can be successful.

So I go back to my definition, a servant leader and Agile champion. Live everyday for those around you and remember the values that Agile stands for. Go have fun with the rest of the pigs! (Search for it) I’m sure I didn’t paint rainbows and unicorns for the PMs out there reading this. I can assure you the ScrumMaster role will be one of the most challenging and enjoyable roles you’ve ever played on a team.

Categories: Companies

Five links on Agile Planning

Most of the dependencies that cross-discipline teams can’t handle can be addressed in release planning.






I don’t claim these articles to be the best on this subject, but I have enjoyed reading them and they have made my knowledge grow. I recommend you to have a look if you are interested in the subject. Happy reading!

Follow me on twitter @hlarsson for more links on a daily basis. You may also subscribe to weekly newsletter of my links. Thanks to writers of these articles and Kirill Klimov for collecting the agile quotes: Please follow @agilequote for more quotes.

Categories: Blogs

Fun Retrospectives

Scrum Expert - Mon, 06/30/2014 - 15:54
If you are following an Agile approach to project management like Scrum, you should have adopted a continuous improvement practice. Retrospectives are the name of the meeting when the Scrum team makes a pause to think on how to improve its current. Fun Retrospectives is a book that should help you to animate these meetings.The material of this book can however be applied to many types of team project meetings like kick-off meetings or risk management meetings. The book discusses the following types of activities that you can use for ...
Categories: Communities

LAST Conference, Melbourne, Australia, July 11 2014

Scrum Expert - Mon, 06/30/2014 - 15:20
The LAST conference is a one-day conference about Lean, Agile, Systems Thinking that takes place in Melbourne, Australia. It is a event for practitioners that encourages participation and interaction via talks, workshops and activities. In the agenda up you can find topics like “Soft Skills for Software”, “My take on SAFe”, “How we delivered more with Test Coverage Mondays, No Meeting Tuesdays and Innovation Fridays”, “Scaling Agile”, “Solving Agile-at-Scale issues on an even larger scale: Crowd sourced!” or “Goodbye Requirements, Hello Hypotheses”. Web site: Location for the 2014 conference: Swinburne University of ...
Categories: Communities

The Next 7 Habits of Highly Effective Coaches: Habit 12 – Dare to Be Different

Portia Tung - Selfish Programming - Mon, 06/30/2014 - 11:00
Habit 12 – Dare to Be Different

Nothing stays the same but one thing’s for certain. Things change. Choose to challenge and change the status quo for better. Be the beacon that encourages people to stand up and stand out. When you’re bold and brave, you can transform the grey into a rainbow.

Exercise: Be a Super Hero

If you were a super hero, who would you be and why? Or perhaps, if you had a superpower what would it be and why? Take turns answering either or both of these questions over lunch with your friends at work. How can you leverage all your superpowers to transform the way you work for the better?

For more information, see: Watch a film or read a comic about a super hero (eg Kick-Ass). Ideally, do this with friends. It’s more fun and so much more empowering! Repeat the exercise and see what new ideas you can come up with. The more “out there” the idea, the better!

About This Blog Entry

The Next 7 Habits of Highly Effective Coaches is part of a mini series inspired by the style of Paul Coelho‘s “Manual of the Warrior of Light“. You can find the first 7 habits here.

Categories: Blogs

Going Solo

AvailAgility - Karl Scotland - Mon, 06/30/2014 - 10:00

There is often a time when singers and songwriters who have always been part of a band, decide to pursue a solo project. Well, I feel I can empathise with that situation, and having always worked for someone else, I’ve decided its time to go out on my own.

I’m now an independent Lean Agile Consultant, providing services to help businesses become Learning Organisations. I hope to be able to focus more time on developing Kanban Thinking as something which develops problem solvers. More on this very soon!

If you think I might be able to help you, and would like to discuss and explore possibilities, then please contact me via


Categories: Blogs

Scrum Framework, or how do you do Scrum?

Scrum Breakfast - Mon, 06/30/2014 - 09:30
Last post I wrote, "Scrum is a simple, team-based framework for solving complex problems." How does the Scrum framework actually work?

Scrum is modelled on successful patterns for product development. The classic phase-driven approach, or "relay-race model" was found to be less effective than the team-oriented, "rugby model", in which an interdisciplinary team solves the whole problem together, without explicit phases or formal leadership roles. (See "The New, New Product Development Game" at HBR for the research which inspired Scrum).

The Scrum FrameworkThe Scrum Framework ensures the core principle of Inspect and Adapt can take place at regular intervals. To achieve this, Scrum introduces a few constraints on the development process:
  1. Time-boxing triggers inspect-and-adapt cycles. So a Sprint is time-boxed to four weeks or less.
  2. An interdisciplinary Development Team solves the problem. Outsiders do not tell the Team how to organize or do its work.
  3. Work taken on at the beginning of the Sprint must be finished by the end of the Sprint. So the team only takes on what it reasonably believes it can finish on time.
  4. No changes to the assignment before the end of sprint. Work that is started but not finished is waste, so this should be avoided.
If you accept and believe in Scrum's principles and constraints, then doing Scrum will come naturally to you.
Roles"The Scrum Team" refers to an interdisciplinary, self-organizing group of people, who collaborate to define and solve the problem as effectively as possible. Within the Scrum Team, Scrum defines three roles:
  • Product Owner - represents the voice of the customer or stakeholder. The Product Owner is responsible for answering the question "Why?" S/he is empowered to cancel a feature or even the development of the product if the answer is not satisfactory. The Product Owner guides development from the first formulation of the Vision to the (final) release of the product. 
  • Development Team - solves the problem presented by the Product Owner. The Dev Team has all the skills need to bring an idea to the completion. It is responsible for answering the question "How?" and for creating something potentially valuable to the customer or user at least by the end of each sprint. 
  • Scrum Master - represents the voice of common sense and is responsible for the process. The Scrum Master acts as trainer and coach to help the Development Team, Product Owner and indeed, the rest of the organisation, become more effective. The Scrum Master recognizes all kinds of impediments and works to eliminate or at least mitigate them.
There are no hierarchies, further divisions or sub-teams within the Scrum Team.ActivitiesScrum is an iterative, incremental approach to development. So development is broken into a large number of fixed-length "sprints", each of which produces an increment of "potentially shippable" customer visible value, which is integrated with the previously created increment to create a seamless whole.
  • Sprint -- the basic feedback cycle Scrum. The Development Team takes on a problem to solve at the beginning of the Sprint and commits to do its best to complete that mandate by the end of the Sprint. A Sprint should strive to achieve a clear business goal, which should be between one and three Sprints in the future.
  • Sprint Planning 1 -- The Scrum Team collaborates to decide what problem(s) should be solved in the current sprint. This should represent the best step forward toward the ultimate goal, given the Scrum Teams knowledge at the time of the planning.
  • Sprint Planning 2 -- The Development team collaborates to decide how to solve the problem they took on in Sprint Planning 1. The result is usually a technical concept and task planning. Neither needs to be definitive, but should be enough that the development team can start work.
  • Daily Scrum -- The Development Team inspects and adapts its own work on a daily basis. The team report to each other: What have I accomplished? What is my goal for today? What is slowing me down? Brief, clarifying questions are allowed, but longer discussion must be postponed to (immediately) after the Daily Scrum.
  • Sprint Review -- The Scrum Team inspects the product it created in this sprint so it can better decide how (or even whether) to move forward in the next sprint. A potentially shippable Product Increment should be available at the review, consisting exclusively of "done" functionality.
  • Sprint Retrospective -- The Scrum Team inspects and adapts itself, so it can be more effective in the future.
  • Backlog Refinement -- This is the process of keeping the backlog up to date and preparing the backlog entries for implementation. This includes ensuring that the Team understands upcoming backlog entries; estimation; "slicing" this stories into small enough pieces; and discarding or de-prioritising unneeded entries.
All activities in Scrum are time-boxed.

The sprint length must always be the same, and should be short compared to the length of the project. Two weeks is common in software development. The maximum length may be limited by the Product Owner's ability to hold his/her word not to change the mandate: fast moving businesses generally require shorter Sprints. Four weeks is the maximum allowed by Scrum, because longer sprints have been shown to lower performance.

The Daily Scrum is time-boxed to 15 minutes. This discourages excessive talk and teams from getting too big. The remaining mandatory meetings are time-boxed to 1 hour per week of Sprint (e.g. 2 hours for a 2 week sprint, 4 hours for a 4 week sprint).
Note: the authoritative literature often combines the time-boxes for Sprint Planning 1 and 2 into one time-box of two hours per week of Sprint. I have found that the time-boxes for Sprint Planning 1 and the Sprint Review are critical. If you cannot finish Sprint Planning 1 in the allotted time, you need to invest more effort in Backlog Refinement. If you cannot finish the Sprint Review in the allotted time, your Team is probably finishing most backlog items at the last minute, and/or the items are not really Done.  ArtifactsI don't like the term 'Artifact' very much. To me, this implies objects that were created by people who died thousands of years ago and were dug up by archaeologists.  The so-called artifacts in Scrum are living entities to guide and direct the project.Scrum defines three artifacts: the Product Backlog, the Sprint Backlog, and the Product Increment:
  • Product Backlog -- this is the single list of ideas for the product under construction. The Product Backlog consists of entries which represent some value to the customer or user. Each entry is:
    • Sequenced - a strong form of prioritization in which each backlog item has a unique sequence number: 1, 2, 3, 4.... Sequencing is most important for the items at the top of backlog, which may be implemented in the next 2 or 3 Sprints.
    • Estimated - not all Scrum Teams estimate, but if you do estimate, the Development Team does the estimating.
    • Emergent - the product backlog is not a binding definition of the product, just the current best understanding of what the product should be.
  • Sprint Backlog -- the Team's detailed plan on how to solve the problem. It consists of the selected backlog items from the Sprint Planning, a solution concept (how to solve the problem) and a task planning (the steps needed to complete the work involved). The task planning is often represented on a task board.
  • Product Increment -- Scrum achieves results incrementally, that is one slice at a time. Each backlog item defines a slice. At least once per Sprint, but possibly more often, the team must integrate all the slices of the current sprint into the previous Increment to create a new Increment. This increment and each slice in the increment must be "done" according to the Definition of Done. Should the Product Owner wish to release the Increment, there should be no impediments to doing so.
AgreementsCore Scrum introduced the notion that the Definition of Done is an Agreement, not an Artifact. At first I found this a bit strange, because the DoD was usually posted on our Wiki somewhere. But now I find the notion quite powerful, and believe that Agreements are the basis for many context specific enhancements to Scrum.Planning in Scrum is commitment based. There is no hierarchical command authority in Scrum. Scrum defines explicitly one agreement, the Definition of Done implies a second agreement, which I call the Sprint Contract, which govern the expectations during a Sprint.
  • Definition of Done. The constraint that a feature be done before it can be included in the Increment and that the Increment be potentially shippable requires that everybody mean the same thing when they call something "Done." Generally, the Definition of Done ensures that the Team has built the right thing, that they have built in right, and if something was Done in a previous Sprint stays Done in this Sprint.
The Sprint Contract is implied by Scrum, but not explicitly named. I have found it useful to give it a name, discuss its conditions, and secure agreement among the concerned parties.
  • Sprint Contract -- An agreement between Product Owner and Development Team covering the current Sprint: scope (forecast), quality (Definition of Done), cost (Effort), and time (Sprint Length). The team commits to do its best to achieve the forecast. Product Owner and Development Team commit to support each other during the Sprint. Time, cost and quality remain fixed, so if the team overestimates its capacity, it will deliver less functionality than forecast. When that is the case, it should leave low-priority stories unstarted. To prevent wasted effort and unfinished work, the Sprint Contact cannot be changed unilaterally (though it can be cancelled in an emergency). The entire Scrum Team and the rest of the organisation must respect the Sprint Contract.
The following is not defined in the authoritative Scrum literature but I have found additional agreements useful in many contexts:
  • Team Working Agreement -- An agreement among the (Development) Team members to enable them to work together more effectively. This often covers core hours, punctuality at meetings, whether reading email is allowed during a meeting, when to update the task board, and anything else the team finds useful. 
  • Additional agreements -- As development teams work with their Product Owners they learn to work together effectively. This know-how is reflected in the agreements. One such agreement is the "Definition of Ready" when is a backlog item ready for execution. How often does the Scrum Team get together to do Backlog Refinement? 

Categories: Blogs

Team-Heterogenität als Chance

Scrum 4 You - Mon, 06/30/2014 - 07:30

Es ist anscheinend eine ganz selbstverständliche Annahme, dass Teams leistungsfähiger sind, wenn sie möglichst heterogen aufgestellt sind. Und tatsächlich findet man in einschlägigen Publikationen eine Vielzahl von Beispielen und Nachweisen. Aus eigener langjähriger Erfahrung im Umgang mit Teams und Teamarbeit kann ich dieser Annahme durchaus zustimmen. Heterogene Teams sind zwar meist anspruchsvoll (gerade auch in punkto Führung), aber in ihrer Performance tatsächlich vielversprechend.

Heterogenität kann als die Uneinheitlichkeit und Vielschichtigkeit der Elemente einer definierten Menge, oder der Bestandteile eines (sozialen) Systems bezeichnet werden. Als heterogen wird das übergeordnete Ganze, das System als solches (also in unserem Fall das Team) bezeichnet, nicht seine einzelnen Teile.
In der Teamarbeit kann man verschiedene Ebenen von Heterogenität konstatieren, z.B. bezüglich Alter, Geschlecht, Persönlichkeitsmerkmalen, fachlichen Kompetenzen, formalen und informellen Rollen und Funktionen usw. Ich will hier speziell auf Heterogenität in Bezug auf sogenannte individuelle Typologien der Teammitglieder eingehen – also ihre typischen individuellen Rollen- und Verhaltensmuster.

Die Nützlichkeit von Typologien

Typologien sind schlicht ziemlich nützlich. Menschen brauchen vereinfachende Strukturen, in die sie unterschiedlichste Umweltfaktoren einordnen können, um mit Komplexität fertig zu werden und handlungsfähig zu sein. Auch menschliche Typologien sind als eine solche Vereinfachung zu sehen, und werden von jeder Person automatisch und meist unbewusst entwickelt. Erst über diese Schubladen registrieren und definieren wir Unterschiede in den persönlichen Mustern unserer Mitmenschen und im Bezug zu uns selbst. Die Auseinandersetzung mit einem strukturierten Typologiemodell bietet die Möglichkeit, sich diese Unterschiede bewusst, transparent und nutzbar zu machen. Dies kann zum einen helfen, sich selbst besser kennenzulernen und zum anderen ein differenziertes Verständnis für andere und ihre Eigenarten im Team zu entwickeln. Gerade in der Teamarbeit, in der direkte Kontakte, enge Beziehungen und eine gewisse Abhängigkeit voneinander herrschen, kann die Reflexion der persönlichen Typologien Verständnis für einander schaffen und so die Heterogenität bewusst einzusetzen und zu fördern.

Die LIFO®-Methode

Aus der Vielzahl der auf dem Markt angebotenen Modelle wie z.B. MBTI™-Typenindikator, DISG-Persönlichkeits-Profil, Enneagramm, Team-Management-System™ (TMS™), möchte ich mich hier auf das Modell der LIFO®-Methode beziehen. Ursprünglich als Selbstmanagement-Tool gedacht, eignet sich LIFO auch hervorragend für die Teamentwicklung. Die LIFO®-Methode wurde 1963 von Stuart Atkins und Allan Katcher in den USA auf der Basis der Theorie von Erich Fromm entwickelt, und wird inzwischen weltweit auch in vielen großen Unternehmen in der Personalentwicklung eingesetzt. Sie verwendet, wie viele andere Modelle auch, einen detaillierten Fragebogen zur Analyse der persönlichen Verhaltensstile. Daraus ergibt sich ein konkretes Verhaltensprofil im Sinne einer je individuellen Typologie.

Die LIFO®-Methode geht von der Annahme aus, dass unsere individuellen Verhaltensmuster im Wesentlichen das Ergebnis von Erziehung und Erfahrungen, d.h. biographischen Lernens sind. Wir haben gelernt, uns anderen gegenüber so zu verhalten, dass unsere unterschiedlichen psychischen und physischen Wünsche möglichst befriedigt werden. Daraus resultiert die individuelle Kombination unserer komplexen Verhaltensstile. Diese Kombination ist die Quelle unserer Stärken und unserer Ressourcen.
Dieses Typologie-Modell streicht als Besonderheiten heraus, dass

  • es ganz pragmatisch um beobachtbares, konkretes Verhalten geht, nicht um Eigenschaften.
  • alle vier Verhaltensstile sinnvoll und nützlich sind, in welcher Rolle man auch agiert, d.h. Stilflexibilität möglich und erwünscht ist.
  • zwischen günstigen und ungünstigen Bedingungen unterschieden werden muss, unter denen sich Menschen verhalten.
  • sinnvollerweise nicht von Schwächen sondern vom Einsatz übertriebenen Stärken zu sprechen ist

Die vier LIFO Verhaltensstile sind:

  1. Unterstützend / hergebend (hilfsbereit, leistungsorientiert, lebt Werte und Ideale, anspruchsvoll u.a.)
  2. Bewahrend / festhaltend (analytisch, methodisch, detailorientiert, vorsichtig, pragmatisch u.a.)
  3. Anpassend / harmonisierend (emphatisch, kooperativ, ausgleichend, harmonieorientiert u.a.)
  4. Bestimmend / übernehmend (Führung, einflussnehmend, innovativ, risikobereit u.a.)

Die Ergebnisse des Fragebogens ergeben das individuelle Verhaltensprofil einer Person mit der jeweiligen Ausprägung der vier Stile. Im persönlichen Stilprofil zeigen sich ein eher bevorzugter und ein eher vernachlässigter Stil. Der bevorzugte Verhaltensstil wird häufiger, ja in der Regel standardmäßig, eingesetzt, während auf den vernachlässigten Stil eher weniger zugegriffen wird. Weitere Detailinformation sind unter oder zu erfahren.

Vergleicht man in der Teamentwicklung die individuellen LIFO-Profile der einzelnen Teammitglieder, ergeben sich in der Regel differenzierte und unterschiedliche Verhaltensschwerpunkte, die im Sinn der Heterogenität gezielt analysiert und eingesetzt werden können. Gerade in Teamkonstellationen, in denen Selbstorganisation einen zentralen Stellenwert hat, braucht es eine gezielt gestaltete Teamstruktur.
In der Teamzusammenstellung oder Rekrutierung einzelner Mitarbeiter kann die LIFO®-Methode als Unterstützung zur Personalauswahl für Teams genutzt werden. Welche Ressourcen sind aktuell gut vertreten? Welche Stärken ergänzen das Team? Wieviel Homogenität / Heterogenität ist für das Team und die Aufgabe hilfreich? In der Zusammenarbeit in Teams müssen immer wieder Aufgaben und Situationen analysiert und effektive Lösungen gefunden werden. Durch die unterschiedlichen Stilausprägungen geht jeder Mitarbeiter unterschiedlich an die Problemlösung heran. Sind die unterschiedlichen Verhaltensmuster untereinander transparent, ausgetauscht und akzeptiert, können sich die Synergien des Teams optimal entfalten.

Fazit: Typologiemodelle sind in ihrer Funktion durchaus nicht unumstritten. Sie können zu Vereinfachungen, ja zu Stigmatisierungen einzelner Personen führen. Andererseits können sie, wie oben beschrieben, durchaus einen konstruktiven Beitrag zur selbstorganisierten Teamentwicklung leisten. Ganz gleich welches Modell gewählt wird, kommt es darauf an, es funktional einzusetzen, höchstmögliche Transparenz herzustellen und die Teammitglieder nicht nur auf die Ergebnisse der Fragebogenergebnisse zu reduzieren. Hoch lebe die Heterogenität als wesentliches Element guter Teamarbeit!

Related posts:

  1. Team Booster
  2. Die 3 Säulen der Teamarbeit – zentrale Elemente der Selbstorganisation in Teams
  3. Zertifizierung

Categories: Blogs

Neo4j/R: Grouping meetup members by join timestamp

Mark Needham - Mon, 06/30/2014 - 02:06

I wanted to do some ad-hoc analysis on the join date of members of the Neo4j London meetup group and since cypher doesn’t yet have functions for dealings with dates I thought I’d give R a try.

I started off by executing a cypher query which returned the join timestamp of all the group members using Nicole White’s RNeo4j package:

> library(Rneo4j)
> query = "match (:Person)-[:HAS_MEETUP_PROFILE]->()-[:HAS_MEMBERSHIP]->(membership)-[:OF_GROUP]->(g:Group {name: \"Neo4j - London User Group\"})
RETURN membership.joined AS joinDate"
> meetupMembers = cypher(graph, query)
> meetupMembers[1:5,]
[1] 1.389107e+12 1.376572e+12 1.379491e+12 1.349454e+12 1.383127e+12

I realised that if I was going to do any date manipulation I’d need to translate the timestamp into an R friendly format so I wrote the following function to help me do that:

> timestampToDate <- function(x) as.POSIXct(x / 1000, origin="1970-01-01")

I added another column to the data frame with this date representation:

> meetupMembers$joined <- timestampToDate(meetupMembers$joinDate)
> meetupMembers[1:5,]
      joinDate              joined
1 1.389107e+12 2014-01-07 15:08:40
2 1.376572e+12 2013-08-15 14:13:40
3 1.379491e+12 2013-09-18 08:55:11
4 1.349454e+12 2012-10-05 17:28:04
5 1.383127e+12 2013-10-30 09:59:03

Next I wanted to group those timestamps by the combination of month + year for which the aggregate and format functions came in handy:

> dd = aggregate(meetupMembers$joined, by=list(format(meetupMembers$joined, "%m-%Y")), function(x) length(x))
> colnames(dd) = c("month", "count")
> dd
     month count
1  01-2012     4
2  01-2013    52
3  01-2014    88
4  02-2012     7
5  02-2013    52
6  02-2014    91
7  03-2012    12
8  03-2013    23
9  03-2014    93
10 04-2012     3
11 04-2013    34
12 04-2014   119
13 05-2012     9
14 05-2013    69
15 05-2014   102
16 06-2011    14
17 06-2012     5
18 06-2013    39
19 06-2014   114
20 07-2011     4
21 07-2012    16
22 07-2013    20
23 08-2011     2
24 08-2012    34
25 08-2013    50
26 09-2012    14
27 09-2013    52
28 10-2011     2
29 10-2012    29
30 10-2013    42
31 11-2011     2
32 11-2012    31
33 11-2013    34
34 12-2012     7
35 12-2013    19

I wanted to be able to group by different date formats so I created the following function to make life easier:

groupBy = function(dates, format) {
  dd = aggregate(dates, by= list(format(dates, format)), function(x) length(x))
  colnames(dd) = c("key", "count")

Now we can find the join dates grouped by year:

> groupBy(meetupMembers$joined, "%Y")
   key count
1 2011    24
2 2012   171
3 2013   486
4 2014   607

or by day:

> groupBy(meetupMembers$joined, "%A")
        key count
1    Friday   135
2    Monday   287
3  Saturday    80
4    Sunday   102
5  Thursday   187
6   Tuesday   286
7 Wednesday   211

or by month:

> groupBy(meetupMembers$joined, "%m")
   key count
1   01   144
2   02   150
3   03   128
4   04   156
5   05   180
6   06   172
7   07    40
8   08    86
9   09    66
10  10    73
11  11    67
12  12    26

I found the ‘by day’ grouping interesting as I had the impression that the huge majority of people joined meetup groups on a Monday but the difference between Monday and Tuesday isn’t significant. 60% of the joins happen between Monday and Wednesday.

The ‘by month’ grouping is a bit skewed by the fact we’re only half way into 2014 and there have been a lot more people joining this year than in previous years.

If we exclude this year then the spread is more uniform with a slight dip in December:

> groupBy(meetupMembers$joined[format(meetupMembers$joined, "%Y") != 2014], "%m")
   key count
1   01    56
2   02    59
3   03    35
4   04    37
5   05    78
6   06    58
7   07    40
8   08    86
9   09    66
10  10    73
11  11    67
12  12    26

Next up I think I need to get some charts going on and perhaps compare the distributions of join dates of various London meetup groups against each other.

I’m an absolute R newbie so if anything I’ve done is stupid and can be done better please let me know.

Categories: Blogs

Neo4j: Set Based Operations with the experimental Cypher optimiser

Mark Needham - Sun, 06/29/2014 - 10:45

A few months ago I wrote about cypher queries which look for a missing relationship and showed how you could optimise them by re-working the query slightly.

To refresh, we wanted to find all the people in the London office that I hadn’t worked with given this model…

…and this initial query:

MATCH (p:Person {name: "me"})-[:MEMBER_OF]->(office {name: "London Office"})<-[:MEMBER_OF]-(colleague)
WHERE NOT (p-[:COLLEAGUES]->(colleague))
RETURN COUNT(colleague)

This took on average 7.46 seconds to execute using cypher-query-tuning so we came up with the following version which took 150 ms on average:

MATCH (p:Person {name: "me"})-[:COLLEAGUES]->(colleague)
WITH p, COLLECT(colleague) as marksColleagues
MATCH (colleague)-[:MEMBER_OF]->(office {name: "London Office"})<-[:MEMBER_OF]-(p)
WHERE NOT (colleague IN marksColleagues)
RETURN COUNT(colleague)

With the release of Neo4j 2.1 we can now make use of Ronja – the experimental Cypher optimiser – which performs much better for certain types of queries. I thought I’d give it a try against this one.

We can use the experimental optimiser by prefixing our query like so:

cypher 2.1.experimental MATCH (p:Person {name: "me"})-[:MEMBER_OF]->(office {name: "London Office"})<-[:MEMBER_OF]-(colleague)
WHERE NOT (p-[:COLLEAGUES]->(colleague))
RETURN COUNT(colleague)

If we run that through the query tuner we get the following results:

$ python
cypher 2.1.experimental MATCH (p:Person {name: "me"})-[:MEMBER_OF]->(office {name: "London Office"})<-[:MEMBER_OF]-(colleague)
WHERE NOT (p-[:COLLEAGUES]->(colleague))
RETURN COUNT(colleague)
Min 0.719580888748 50% 0.723278999329 95% 0.741609430313 Max 0.743646144867
MATCH (p:Person {name: "me"})-[:COLLEAGUES]->(colleague)
WITH p, COLLECT(colleague) as marksColleagues
MATCH (colleague)-[:MEMBER_OF]->(office {name: "London Office"})<-[:MEMBER_OF]-(p)
WHERE NOT (colleague IN marksColleagues)
RETURN COUNT(colleague)
Min 0.706955909729 50% 0.715770959854 95% 0.731880950928 Max 0.733670949936

As you can see there’s not much in it – our original query now runs as quickly as the optimised one. Ronja #ftw!

Give it a try on your slow queries and see how it gets on. There’ll certainly be some cases where it’s slower but over time it should be faster for a reasonable chunk of queries.

Categories: Blogs

Neo4j’s Cypher vs Clojure – Group by and Sorting

Mark Needham - Sun, 06/29/2014 - 04:56

One of the points that I emphasised during my talk on building Neo4j backed applications using Clojure last week is understanding when to use Cypher to solve a problem and when to use the programming language.

A good example of this is in the meetup application I’ve been working on. I have a collection of events and want to display past events in descending order and future events in ascending order.

First let’s create some future and some past events based on the current timestamp of 1404006050535:

CREATE (event1:Event {name: "Future Event 1", timestamp: 1414002772427 })
CREATE (event2:Event {name: "Future Event 2", timestamp: 1424002772427 })
CREATE (event3:Event {name: "Future Event 3", timestamp: 1416002772427 })
CREATE (event4:Event {name: "Past Event 1", timestamp: 1403002772427 })
CREATE (event5:Event {name: "Past Event 2", timestamp: 1402002772427 })

If we return all the events we see the following:

$ MATCH (e:Event) RETURN e;
==> +------------------------------------------------------------+
==> | e                                                          |
==> +------------------------------------------------------------+
==> | Node[15414]{name:"Future Event 1",timestamp:1414002772427} |
==> | Node[15415]{name:"Future Event 2",timestamp:1424002772427} |
==> | Node[15416]{name:"Future Event 3",timestamp:1416002772427} |
==> | Node[15417]{name:"Past Event 1",timestamp:1403002772427}   |
==> | Node[15418]{name:"Past Event 2",timestamp:1402002772427}   |
==> +------------------------------------------------------------+
==> 5 rows
==> 13 ms

We can achieve the desired grouping and sorting with the following cypher query:

(def sorted-query "MATCH (e:Event)
WITH [e IN events WHERE e.timestamp <= timestamp()] AS pastEvents,
     [e IN events WHERE e.timestamp > timestamp()] AS futureEvents
UNWIND pastEvents AS pastEvent
WITH pastEvent, futureEvents ORDER BY pastEvent.timestamp DESC
WITH COLLECT(pastEvent) as orderedPastEvents, futureEvents
UNWIND futureEvents AS futureEvent
WITH futureEvent, orderedPastEvents ORDER BY futureEvent.timestamp
RETURN COLLECT(futureEvent) AS orderedFutureEvents, orderedPastEvents")

We then use the following function to call through to the Neo4j server using the excellent neocons library:

(ns neo4j-meetup.db
  (:require [clojure.walk :as walk])
  (:require [ :as cy])
  (:require [ :as nr]))
(def NEO4J_HOST "http://localhost:7521/db/data/")
(defn cypher
  ([query] (cypher query {}))
  ([query params]
     (let [conn (nr/connect! NEO4J_HOST)]
       (->> (cy/tquery query params)

We call that function and grab the first row since we know there won’t be any other rows in the result:

(def query-result (->> ( db/cypher sorted-query) first))

Now we need to extract the past and future collections so that we can display them on the page which we can do like so:

> (map #(% :data) (query-result :orderedPastEvents))
({:timestamp 1403002772427, :name "Past Event 1"} {:timestamp 1402002772427, :name "Past Event 2"})
> (map #(% :data) (query-result :orderedFutureEvents))
({:timestamp 1414002772427, :name "Future Event 1"} {:timestamp 1416002772427, :name "Future Event 3"} {:timestamp 1424002772427, :name "Future Event 2"})

An alternative approach is to return the events from cypher and then handle the grouping and sorting in clojure. In that case our query is much simpler:

(def unsorted-query "MATCH (e:Event) RETURN e")

We’ll use the clj-time library to determine the current time:

(def now (clj-time.coerce/to-long (clj-time.core/now)))

First let’s split the events into past and future:

> (def grouped-by-events 
     (->> (db/cypher unsorted-query)
          (map #(->> % :e :data))
          (group-by #(> (->> % :timestamp) now))))
> grouped-by-events
{true [{:timestamp 1414002772427, :name "Future Event 1"} {:timestamp 1424002772427, :name "Future Event 2"} {:timestamp 1416002772427, :name "Future Event 3"}], 
 false [{:timestamp 1403002772427, :name "Past Event 1"} {:timestamp 1402002772427, :name "Past Event 2"}]}

And finally we sort appropriately using these functions:

(defn time-descending [row] (* -1 (->> row :timestamp)))
(defn time-ascending [row] (->> row :timestamp))
> (sort-by time-descending (get grouped-by-events false))
({:timestamp 1403002772427, :name "Past Event 1"} {:timestamp 1402002772427, :name "Past Event 2"})
> (sort-by time-ascending (get grouped-by-events true))
({:timestamp 1414002772427, :name "Future Event 1"} {:timestamp 1416002772427, :name "Future Event 3"} {:timestamp 1424002772427, :name "Future Event 2"})

I used Clojure to do the sorting and grouping in my project because the query to get the events was a bit more complicated and became very difficult to read with the sorting and grouping mixed in.

Unfortunately cypher doesn’t provide an easy way to sort within a collection so we need our sorting in the row context and then collect the elements back again afterwards.

Categories: Blogs

Data Science: Mo’ Data Mo’ Problems

Mark Needham - Sun, 06/29/2014 - 01:35

Over the last couple of years I’ve worked on several proof of concept style Neo4j projects and on a lot of them people have wanted to work with their entire data set which I don’t think makes sense so early on.

In the early parts of a project we’re trying to prove out our approach rather than prove we can handle big data – something that Ashok taught me a couple of years ago on a project we worked on together.

In a Neo4j project that means coming up with an effective way to model and query our data and if we lose track of this it’s very easy to get sucked into working on the big data problem.

This could mean optimising our import scripts to deal with huge amounts of data or working out how to handle different aspects of the data (e.g. variability in shape or encoding) that only seem to reveal themselves at scale.

These are certainly problems that we need to solve but in my experience they end up taking much more time than expected and therefore aren’t the best problem to tackle when time is limited. Early on we want to create some momentum and keep the feedback cycle fast.

We probably want to tackle the data size problem as part of the implementation/production stage of the project to use Michael Nygaard’s terminology.

At this stage we’ll have some confidence that our approach makes sense and then we can put aside the time to set things up properly.

I’m sure there are some types of projects where this approach doesn’t make sense so I’d love to hear about them in the comments so I can spot them in future.

Categories: Blogs

The Love Contagion

Doc On Dev - Michael Norton - Sat, 06/28/2014 - 19:04
We have an application internal to Groupon called the "Love Monster". It was written, in large part by Devin Breen. There are other contributors, but Devin is the one that made it happen. He didn't do it because it was on a roadmap or because it's part of our quarterly objectives. He did it because he and others wanted something like this to exist. So he willed it into existence.

The idea is simple - send people love. If somebody on your team goes above and beyond, send them some love. If somebody helps you through a tough time, send them some love. If somebody does something that exemplifies our company values, send them some love. When you send love, the recipient gets an email and the love note is recorded for all to see. Anyone can give love. Anyone can receive love.

This makes me happy. Not only do I enjoy getting a little love now and then, but I really enjoy giving love. I have a recurring reminder to reflect on the past couple of days and to give out some love.

The science on appreciation tells us that gratitude is not only lacking in most work environments, but even a little bit of appreciation can lead to a better work environment and higher performance. People start to feel better about themselves, perform better as a result, and are more likely to show appreciation to others, creating a virtuous cycle of performance and appreciation.

Let's Start an Epidemic by Doc Norton from Groupon Engineering on Vimeo.

Let's Start and Epidemic!Let a family member, friend, or co-worker know how much you appreciate them. Tell us about the awesome way you show appreciation at work. Share a quote related to showing appreciation, leading with gratefulness, or brotherly love. If you use social media, tag it with #LoveContagion.

Categories: Blogs

On Asiana, Driverless Cars, and Factories

Evolving Excellence - Sat, 06/28/2014 - 15:41

I've always been interested in technology - its use, advantages, and disadvantages.  Regular readers know that I often advocate for manual solutions first, such as whiteboards to run factories and a handwritten notebook for notes.

Some believe that I'm against technology, but nothing could be further from the truth.  I'm more like the guy that stands in line at the Apple Store to spend money on the latest iPhone that is supposedly but in reality indistinguishably better than the previous version.  A few weeks ago there was a long discussion on this topic in the Lean CEO LinkedIn group, and after over 100 comments we all had to basically agree to disagree.

My belief is that you need to first understand and improve the underlying process, and only then apply technology to further refine and automate it.  Otherwise you may simply automate waste, which is not true improvement.  Manual methods also create more learning and understanding, which creates the foundation for improvement.

Holman Jenkins penned an opinion piece for The Wall Street Journal last week that brings up another downside of technology: losing the understanding of the underlying process, and how to control it, in the first place.

As nobody disagrees, Asiana Flight 214's crew failed in a basic task, keeping track of the plane's airspeed on final approach. Before they could correct their error, the plane's tail smacked a sea wall, breaking off.

The chief pilot later claimed "it was very difficult to perform a visual approach with a heavy airplane," according to the NTSB, which would seem to indicate the real problem: The crew was nonplused, perhaps nearly panicked, at the prospect of having to maintain a proper glidepath without help from the airport's sophisticated landing aid.

That's basic flying.  Yes, there were also cultural issues involved. The downside of a culture deeply respectful of elders and superiors was in play when communication between pilot and copilot was delayed.

We were warned.

Diligent annotators of this column will recall Captain Malcolm Scott from nearly a decade ago, who criticized a British Airways decision to ban manual thrust control (which Asiana's pilots should have employed to maintain the plane's airspeed) by its Airbus pilots. Flying skills would atrophy, he warned, suggesting that the industry's implicit goal was to remove the human factor from the cockpit altogether.

Mr. Jenkins goes on to draw a link to the current fascination with Google’s driverless cars. The nascent technology is working surprisingly well, even at this early stage, and many are predicting full adoption by 2020.

But what will happen to our skills to actually... drive?

Even Porsche, builder of driving machines for purists, has their PDK doppelkupplung automatic transmission that is so refined that 0-60 times are faster in an automatic than with a stick. They are even considering eliminating manual transmissions as an option.  But is the purpose of driving a Porsche to go faster off the line, or to experience driving? An automatic does make it easier to hold your latte, which I guess is why cup holders also had to be installed. Is that what driving a Porsche has devolved into?  Unfortunately, for many, yes.

What will happen if the software or system fails? Will we just sit there staring at each other hoping we don't go into a wall, or worse?

An analogous situation exists with many forms of technology in factories and organizations.  We feed data into the machine and trust that the supposed best practice algorithms created thousands of miles from the gemba will optimize operational performance.  We're confident that the software geeks at SAP, Microsoft, and Oracle must know best how to deal with the nuance of our situations, even to the point of limiting the ability to modify and improve.  Ya...

In practice, though, all cars would likely have to be driverless—or at least capable of taking control away from a driver in heavy traffic situations—for any cars to be driverless. Otherwise, effectively one jerk in a '74 Buick would own the only right of way.

Maybe that’s actually the opportunity. An organization that has a team of folks that actually know how and why the operation works, instead of relying on pre-programmed algorithms, can take the right of way. They can find the weaknesses in the algorithms used by others, improve, and exploit them as a competitive advantage.

Systems are great, but so is the joy of actually driving a car, the safety of understanding how to fly, and the opportunity of truly understanding and running an organization. Not just letting technology do it.

Categories: Blogs

Neo4j: Cypher – Finding movies by decade

Mark Needham - Sat, 06/28/2014 - 13:12

I was recently asked how to find the number of movies produced per decade in the movie data set that comes with the Neo4j browser and can be imported with the following command:

:play movies

We want to get one row per decade and have a count alongside so the easiest way is to start with one decade and build from there.

MATCH (movie:Movie)
WHERE movie.released >= 1990 and movie.released <= 1999
RETURN 1990 + "-" + 1999 as years, count(movie) AS movies
ORDER BY years

Note that we’re doing a label scan of all nodes of type Movie as there are no indexes for range queries. In this case it’s fine as we have few movies but If we had 100s of thousands of movies then we’d want to optimise the WHERE clause to make use of an IN which would then use any indexes.

If we run the query we get the following result:

==> +----------------------+
==> | years       | movies |
==> +----------------------+
==> | "1990-1999" | 21     |
==> +----------------------+
==> 1 row

Let’s pull out the start and end years so they’re explicitly named:

WITH 1990 AS startDecade, 1999 AS endDecade
MATCH (movie:Movie)
WHERE movie.released >= startDecade and movie.released <= endDecade
RETURN startDecade + "-" + endDecade as years, count(movie)
ORDER BY years

Now we need to create a collection of start and end years so we can return more than one. We can use the UNWIND function to take a collection of decades and run them through the rest of the query:

UNWIND [{start: 1970, end: 1979}, {start: 1980, end: 1989}, {start: 1980, end: 1989}, {start: 1990, end: 1999}, {start: 2000, end: 2009}, {start: 2010, end: 2019}] AS row
WITH row.start AS startDecade, row.end AS endDecade
MATCH (movie:Movie)
WHERE movie.released >= startDecade and movie.released <= endDecade
RETURN startDecade + "-" + endDecade as years, count(movie)
ORDER BY years
==> +----------------------------+
==> | years       | count(movie) |
==> +----------------------------+
==> | "1970-1979" | 2            |
==> | "1980-1989" | 2            |
==> | "1990-1999" | 21           |
==> | "2000-2009" | 13           |
==> | "2010-2019" | 1            |
==> +----------------------------+
==> 5 rows

Alistair pointed out that we can simplify this even further by using the RANGE function:

UNWIND range(1970,2010,10) as startDecade
WITH startDecade, startDecade + 9 as endDecade
MATCH (movie:Movie)
WHERE movie.released >= startDecade and movie.released <= endDecade
RETURN startDecade + "-" + endDecade as years, count(movie)
ORDER BY years

And here’s a graph gist for you to play with.

Categories: Blogs