Skip to content

Feed aggregator

Neo4j: Using LOAD CSV to help explore CSV files

Mark Needham - Fri, 06/12/2015 - 01:15

During the Neo4j How I met your mother hackathon that we ran last week one of the attendees noticed that one of the CSV files we were importing wasn’t creating as many records as they expected it to.

This is typically the case when there’s some odd quoting in the CSV file but we decided to look into it.

The file in question was one containing references made in HIMYM. The first 5 lines look like this:

$ head -n 5 data/import/references.csv
ReferencedEpisodeId,ReferencingEpisodeId,ReferenceText
168,184,"Marshall will eventually hear back from the New York State Judicatory Committee in Something New, which will become a main plot point of Season 9."
168,169,Barney proclaiming to be done with Robin will be the focal point of Lobster Crawl.
58,57,"Barney finally confronts his saboteur (Abby, whom he slept with in Ten Sessions) in Everything Must Go."
58,63,"Barney finally confronts his saboteur (Abby, whom he slept with in Ten Sessions) in Everything Must Go."

And this is how many lines the Unix ‘wc’ command sees:

$ wc -l data/import/references.csv
     782 data/import/references.csv

So we might expect that there are going to be 782 records created if we import that file into Neo4j. Let’s run a quick query in Neo4j to see what it thinks:

LOAD CSV WITH HEADERS 
FROM "file:///Users/markneedham/projects/neo4j-himym/data/import/references.csv" 
AS row
return count(*)
 
==> +----------+
==> | count(*) |
==> +----------+
==> | 636      |
==> +----------+
==> 1 row

So we have 146 less records than we expected which means Neo4j is treating multiple lines as one CSV line in some cases.

Let’s go back to the Unix command line to try and work out which lines those are. There must be some lines which start with part of the ‘ReferenceText’ rather than a ‘ReferenceEpisodeId’ so let’s extract the first column and see what’s going on there:

$ cat data/import/references.csv | cut -d"," -f1 | grep -v  '[0-9]\+$'| head -n 10
ReferencedEpisodeId
This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny
This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny
This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny
This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny
This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny
This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny
This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny
This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny
Also

We’ve extracted the first column and then filter the output to only keep rows which don’t contain all numbers which will be our rogue rows.

Let’s switch back to Neo4j land to see which rows it thinks contains these fragments of text:

LOAD CSV WITH HEADERS FROM "file:///Users/markneedham/projects/neo4j-himym/data/import/references.csv" AS row
WITH row WHERE row.ReferenceText =~ ".*This is the Mother's first.*"
RETURN row.ReferencedEpisodeId, row.ReferencingEpisodeId, row.ReferenceText
 
==> +--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
==> | row.ReferencedEpisodeId | row.ReferencingEpisodeId | row.ReferenceText                                                                                                                                                                                                                                                                                                                                                                     |
==> +--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
==> | "45"                    | "37"                     | "This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9." |
==> | "45"                    | "184"                    | "This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9." |
==> +--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Interestingly it only returns two rows containing that phrase whereas we see it at least 8 times. Initially I thought this was an issue with the LOAD CSV command but if we filter the rows to only return ones that have a ‘ReferencedEpisodeId’ of ’45’ then we do see them returned:

==> +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
==> | row                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
==> +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "53", ReferenceText -> "The website counting down to the next slap (slapcountdown.com) that Marshall sends Barney reaches zero in Slapsgiving, when the third slap is delivered."}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "55", ReferenceText -> "Ted gets rid of his butterfly tramp stamp through ten weekly sessions of laser tattoo removal between The Platinum Rule and Ten Sessions, over the course of which he meets, asks out, and eventually starts dating his dermatologist, Stella Zinman."}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "57", ReferenceText -> "Ted gets rid of his butterfly tramp stamp through ten weekly sessions of laser tattoo removal between The Platinum Rule and Ten Sessions, over the course of which he meets, asks out, and eventually starts dating his dermatologist, Stella Zinman."}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "56", ReferenceText -> "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton.
==> This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9."}  |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "200", ReferenceText -> "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton.
==> This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9."} |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "100", ReferenceText -> "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton.
==> This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9."} |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "86", ReferenceText -> "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton.
==> This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9."}  |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "113", ReferenceText -> "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton.
==> This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9."} |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "161", ReferenceText -> "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton.
==> This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9."} |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "37", ReferenceText -> "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton.
==> This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9."}  |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "184", ReferenceText -> "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton.
==> This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9."} |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "37", ReferenceText -> "This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9."}                                                                                                                                                                                                                                                                                                                                                                         |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "184", ReferenceText -> "This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9."}                                                                                                                                                                                                                                                                                                                                                                        |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "63", ReferenceText -> "Marshall makes other home-made websites in Everything Must Go (lilyandmarshallselltheirstuff.com) and The Sexless Innkeeper (itwasthebestnightever.com), where Lily and Future Ted mention it being a problem."}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "92", ReferenceText -> "Marshall makes other home-made websites in Everything Must Go (lilyandmarshallselltheirstuff.com) and The Sexless Innkeeper (itwasthebestnightever.com), where Lily and Future Ted mention it being a problem."}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
==> +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

So the actual problem is that the regex matcher doesn’t deal with the new line in the string.

Our next step is therefore to get rid of new lines within strings. I spent ages trying to find the appropriate command before coming across the following use of awk which does the job:

$ cat data/import/references.csv | awk '(NR-1)%2{$1=$1} {print $0}' RS=\" ORS=\" | wc -l
637
 
$ cat data/import/references.csv | awk '(NR-1)%2{$1=$1} {print $0}' RS=\" ORS=\" > data/import/refs.csv

Let’s try the LOAD CSV command again:

LOAD CSV WITH HEADERS FROM "file:///Users/markneedham/projects/neo4j-himym/data/import/refs.csv" AS row
WITH row WHERE row.ReferenceText =~ ".*This is the Mother's first.*"
RETURN row.ReferencedEpisodeId, row.ReferencingEpisodeId, row.ReferenceText
 
==> +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
==> | row.ReferencedEpisodeId | row.ReferencingEpisodeId | row.ReferenceText                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
==> +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
==> | "45"                    | "56"                     | "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton. This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9." |
==> | "45"                    | "200"                    | "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton. This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9." |
==> | "45"                    | "100"                    | "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton. This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9." |
==> | "45"                    | "86"                     | "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton. This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9." |
==> | "45"                    | "113"                    | "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton. This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9." |
==> | "45"                    | "161"                    | "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton. This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9." |
==> | "45"                    | "37"                     | "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton. This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9." |
==> | "45"                    | "184"                    | "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton. This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9." |
==> | "45"                    | "37"                     | "This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9."                                                                                                                                                                                                                                                                                                                                                                        |
==> | "45"                    | "184"                    | "This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9."                                                                                                                                                                                                                                                                                                                                                                        |
==> +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

And there we go!

Update

Michael pointed out that I could have used the dotall regex flag at the beginning of the regular expression in order to search across new lines without having to remove them! In that case the query would read like this:

LOAD CSV WITH HEADERS FROM "file:///Users/markneedham/projects/neo4j-himym/data/import/references.csv" AS row
WITH row WHERE row.ReferenceText =~ "(?s).*This is the Mother.*"
RETURN row
 
==> +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
==> | row                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
==> +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "56", ReferenceText -> "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton.
==> This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9."}  |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "200", ReferenceText -> "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton.
==> This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9."} |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "100", ReferenceText -> "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton.
==> This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9."} |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "86", ReferenceText -> "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton.
==> This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9."}  |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "113", ReferenceText -> "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton.
==> This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9."} |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "161", ReferenceText -> "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton.
==> This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9."} |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "37", ReferenceText -> "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton.
==> This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9."}  |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "184", ReferenceText -> "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton.
==> This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9."} |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "37", ReferenceText -> "This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9."}                                                                                                                                                                                                                                                                                                                                                                         |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "184", ReferenceText -> "This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9."}                                                                                                                                                                                                                                                                                                                                                                        |
==> +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Categories: Blogs

Mac OS X: GNU sed – Hex string replacement / replacing new line characters

Mark Needham - Thu, 06/11/2015 - 23:38

Recently I was working with a CSV file which contained both Windows and Unix line endings which was making it difficult to work with.

The actual line endings were HEX ‘0A0D’ i.e. Windows line breaks but there were also HEX ‘OA’ i.e. Unix line breaks within one of the columns.

I wanted to get rid of the Unix line breaks and discovered that you can do HEX sequence replacement using the GNU version of sed – unfortunately the Mac ships with the BSD version which doesn’t have this functionaltiy.

The first step was therefore to install the GNU version of sed.

brew install coreutils
brew install gnu-sed --with-default-names

I wanted to replace my system sed so that’s why I went with the ‘–with-default-names’ flag – without that flag I believe the sed installation would be accessible as ‘gs-sed’.

The following is an example of what the lines in the file look like:

$ echo -e "Hello\x0AMark\x0A\x0D"
Hello
Mark

We want to get rid of the new line in between ‘Hello’ and ‘Mark’ but leave the other one be. I adapted one of the commands from this tutorial to look for lines which end in ‘0A’ where that isn’t followed by a ‘0D':

$ echo -e "Hello\x0AMark\x0A\x0D" | \
  sed 'N;/\x0A[^\x0D]/s/\n/ /'
Hello Mark

Let’s go through the parts of the sed command:

  • N – this creates a multiline pattern space by reading a new line of input and appending it to the contents of the pattern space. The two lines are separated by a new line.
  • /\x0A[^\x0D]/ – this matches any lines which contain ‘OA’ not followed by ‘OD’
  • /s/\n/ / – this substitutes the new line character with a space for those matching lines from the previous command.

Now let’s check it works if we have multiple lines that we want to squash:

$ echo -e "Hello\x0AMark\x0A\x0DHello\x0AMichael\x0A\x0D"
Hello
Mark
Hello
Michael
 
$ echo -e "Hello\x0AMark\x0A\x0DHello\x0AMichael\x0A\x0D" | \
  sed 'N;/\x0A[^\x0D]/s/\n/ /'
Hello Mark
Hello Michael

Looks good! The actual file is a bit more nuanced so I’ve still got a bit more work to do but this is a good start.

Categories: Blogs

Quality Gates Work – If You Let Them

Sonar - Thu, 06/11/2015 - 21:20

Some people see rules – standards – requirements – as a way to hem in the unruly, limit bad behavior, and restrict rowdiness. But others see reasonable rules as a framework within which to excel, a scaffolding for striving, an armature upon which to build excellence.

Shortly after its inception, the C-Family plugin (C, C++ and Objective-C) had 90%+ test coverage. The developers were proud of that, but the Quality Gate requirement was only 80% on new code, so week after week, version by version, coverage slowly dropped. The project still passed the Quality Gate with flying colors each time, but coverage was moving in the wrong direction. In December, overall coverage dropped into the 80′s with the version 3.3 release, 89.7% to be exact.

In January, the developers on the project decided they’d had enough, and asked for a different Quality Gate. A stricter one. They asked to be held to a standard of 90% coverage on new code.

Generally at SonarSource, we advocate holding everyone to the same standards – no special Quality Profiles for legacy projects, for instance. But this request was swiftly granted, and version 3.4, released in February, had 90.1% coverage overall, not just on new code. Today the project’s overall coverage stands at 92.7%.

Language Team Technical Lead Evgeny Mandrikov said the new standard didn’t just inspire the team to write new tests. The need to make code more testable “motivated us to refactor APIs, and lead to a decrease of technical debt. Now coverage goes constantly up.”

Not only that, but since the team had made the request publicly, others quickly jumped on the bandwagon. Six language plugins are now assigned to that quality gate. The standard is set for coverage on new code, but most of the projects meet it for overall coverage, and the ones that don’t are working on it.

What I’ve seen in my career is that good developers hold themselves to high standards, and believe that it’s reasonable for their managers to do the same. Quality Gates allow us to set – and meet – those standards publicly. Quality Gates and quality statistics confer bragging rights, and set up healthy competition among teams.

Looking back, I can’t think of a better way for the C-Family team to have kicked off the new year.

Categories: Open Source

Scrum Day Germany, Filderstadt, Germany, June 16-17 2015

Scrum Expert - Thu, 06/11/2015 - 14:45
Scrum Day Germany is a two-day conference about Scrum and Agile project management that propose an international line-up of Agile experts. It provides a multi-track sessions and full day workshops. The conference sessions are mainly in German. In the agenda of Scrum Day Germany you can find topics like “Product Owner Types: What You Should Know”, “Status Quo Agile – Scrum: Rather Tailor-made than According to the Textbook”, “Measuring Agile to Prove the Success of the Adoption”, “Agile Enterprise – Growing Agile beyond Tech”, “HardScrum – Software Development in the Embbeded ...
Categories: Communities

Do You “Really” Know Your Team? How to “Turbocharge” Interactions

Agile Management Blog - VersionOne - Thu, 06/11/2015 - 14:30

Have you ever stated, “I didn’t know you could do that?” If so, you were not only stating the obvious, but also exposing a flaw in team formation. What would the result have been if you had known this characteristic about your teammate earlier?

Agile recognizes that people are unique individuals, instead of replaceable resources, and the highest value is achieved through interactions. People have unique physiognomies, however there are also internal characteristics such as motivational gifts, natural talents, learned skills, developmental environment, and personal passion, which can provide a much richer team experience.

Discovering these individual talents and sharing them at the team level can turbocharge interactions and lead to higher productivity. Here is an activity I have used with many people which is designed to expose these characteristics and facilitate better conversations.

Create a Personal Purpose Profile (PPP)

PPPStarting at the Individual level, each person creates a Personal Purpose Profile (PPP) or triple P. The triple P helps people understand their uniqueness in light of the five areas of focus. Each area reveals a different aspect of individuality and presents opportunities to connect with people.

Write PPP at the top of a paper and get ready to create a deliverable which will be a panoramic look at your life. It can be text-oriented or graphical, black and white or more colorful.

Motivational Gifts
PresentThe first area is to determine your motivational gifts. This will help develop understanding and relevant perspectives as a foundation for interaction.

In researching how people communicate and collaborate over 25 years, there are 7 responses or motivations that are regularly displayed during events, scenarios, or interactions. In addition, these reactions show up in people from an early age more like a gift rather than learned behavior.

Each person will typically have one main motivational gift, but can demonstrate some or all of the 7 gifts outlined below. This helps answer “Why don’t others see things the way I see them?” Here is a working definition – an individual’s first reaction when presented with an event, scenario, or interaction.

The 7 gifts and brief definitions are as follows:

  • The Perceiver sees life as all or nothing, right or wrong, and there is no gray area.
  • The Servant seeks to help with any situation.
  • The Teacher desires to share learned information.
  • The Encourager wants to give courage to people.
  • The Giver looks to invest time, talent, and money.
  • The Ruler understands the big picture and needs to control it.
  • The Compassionate is sympathetic to people’s issues.

Do any of these sound familiar? Take a few minutes to examine these gifts and make a few notes, drawings, or doodles on the paper, which identifies your top motivational gifts.

Natural TalentsTalent

The next portion of the triple P is natural talents. These are talents that have been a part of you for as long as you can remember. For example, some people are talented in a form of the arts like drawing, dancing, singing. Others might have outstanding academic aptitude in math or science, etc.   Others may have outstanding athletic ability in a sport. Statements like “you are a natural” or “this comes easy for me” will help to illuminate these as you think about them.

Take a few minutes to continue your triple P by documenting any of your natural talents.

SkillsLearned Skills

Next, is the area of learned skills. During life, there are many opportunities to gain knowledge and occasionally these will develop into valuable learned skills. Examples of these are computer skills, language skills, team skills, and even agile skills. Skills can be learned formally or informally and validated by degrees, certifications, or accreditations. Learned skills might listed on a resume or CV.

List your learned skills on your triple P.

Developmental EnvironmentGlobe

Each person was born and raised in a particular part of the world with it’s own culture, language, food, clothing, and overall lifestyle. These make up a developmental environment. Although it might not be where you are living today, the background will potentially influence your interactions with others.

Look back into your past and write down some of the things that describe your developmental environment.

HeartPersonal Passion

In our busy world, there are many activities that consume time. Most of these are “need to” type scenarios, however somewhere in the 3D arrangement of life is a personal passion waiting patiently to be done.

This could be called a hobby or form of relaxation. However it will usually be related to the question: What would you do if could do anything in the world? Examples could be spending time with pets, athletics, social endeavors, kids, community, etc., etc., etc.

Now that you are dreaming of doing it, draw a picture or write a story about your personal passion on your triple P.

Share Your Uniqueness

Celebrate how unique you are and start sharing your triple P. Don’t wait for someone to say, “I didn’t know you could do that?”  Start with your co-workers and begin to recognize the benefits of knowing your team better. With each passing day, there are interactions that could be improved if the participants knew more background information. It’s amazing how the smallest piece of information can spark a large conversation leading to enhanced communication and collaboration.

TurbochargeHuddle

Agile recognizes that people are unique individuals, instead of replaceable resources, and the highest value is achieved through interactions. Creating and sharing a Personal Purpose Profile (PPP) containing motivational gifts, natural talents, learned skills, developmental environment, and personal passion should be a part of your agile transformation activities.

It’s time to have a team meeting and turbocharge your interactions.

Find out how VersionOne can support your turbocharged teams. 

versionone-coaches-kelly-keyAbout the Author
Kelly Key
SAFe Agilist, PSM1, MOL, PSD

Kelly has more than 25 years of experience in the software industry and 19 years working with agile practices. As an agile program manager, ScrumMaster, agile coach, and VersionOne administrator, Kelly has helped create successful products in the desktop, dot com, and mobile space for the airline, hotel, rental car, rail and cruise industries. As a thought leader, he is always looking to refine processes with continuous improvement while stimulating innovation.

 

Categories: Companies

[UPDATE] Version R6#14.1

IceScrum - Thu, 06/11/2015 - 12:47
Hello everybody, Here comes a new version of iceScrum and iceScrum Pro! This version brings many changes, we hope that you will like them! This post lists all the changes brought by iceScrum R6#14. It is updated when minor versions are released to fix bugs and add small improvements on top of this major version.…
Categories: Open Source

An Introduction to Cost of Delay

Leading Agile - Mike Cottmeyer - Wed, 06/10/2015 - 22:20

I was recently watching an episode of Shark Tank. I loved the unfiltered statement from Kevin O’Leary (Mr. Wonderful) toward an entrepreneur seeking an investor in his company.

I’m here to make money!

If you’re a fan of Shark Tank, you’ll notice something about Mr. Wonderful.  He keeps the conversation focused on the money.  When will he get his money back? How many multiples of his investment should he expect to get back? Other investors (and many of our stakeholders) don’t focus enough on the money.  Particularly, what is the cost of delaying the implementation of one feature over another.

I tell customers that if they want to save or make the most money, they need to prioritize their backlog… by money.  My colleague Isaac recently wrote something similar in his blog post about how to prioritize work.

One way you can prioritize work by money is by using cost of delay.  Sounds kind of weird, right?  Aren’t we always telling people to prioritize their backlogs by customer value?  When you ask customers (or the business) which features are the highest priority, all too often they say all of them.  (Jim Hayden makes reference to this in a recent podcast, when teams don’t prioritize or limit their work in process.  People are really good at starting things but not finishing them.)  Don’t just ask what is the most valuable. Ask the question, “what will cost us the most, by delaying its delivery?”  That’s really what we’re doing.  We’re not profiting from a feature that is not in production, so therefore, we are losing money every day it’s not out there.

If I have 3 features to choose from, each with a different worth to the business and each taking a different amount of time to implement, how do I make the best economic decision on what to finish first?  I use Cost of Delay.

Cost of Delay Step 1 – Compare Features

Let’s put the 3 features in a table and compare them.  I intentionally tried to keep this simple by multiplying the value of the features by $1,000. Imagine what this would look like if your features were worth tens of thousands or hundreds of thousands of dollars?

Duration Value CD3 Feature A 3 weeks $3000 1 Feature B 4 weeks $7000 1.75 Feature C 6 weeks $9000 1.5 Step 2 – Visualize Scenarios

Taking what we have learned in the table of Step 1, let’s visualize different scenarios, showing when we could get a return on our investment, given a choice of priority.

  1. No priority at all. Do all at the same time.
  2. Do the features that take the shortest amount of time first.
  3. Do the features that are the most valuable first.
  4. Lastly, do the features that have the highest CD3 (value divided by duration) score.

Remember, regardless of our choice of priority, all of the features are done by the 14th week.

Visualize Prioritization Scenarios

For every week features are not making us money, they are costing us money.
Let’s do some math!

Step 3 – Priority Impact on Cost of Delay

Using the three features we can look at the financial impact of the four alternatives.

All at the same time (No Priority)

If we started all of the features at the same time, we would get any and all of that $19,000 of value on our 14th week.  For the 13 weeks we are working, we incur the Cost of Delay of all three features: $3000 + $7000 + $9000 per week. Delay Cost incurred is $247,000.

Do the Shortest Job First

If we prioritized based on shortest to longest length of time to complete a feature, it would take us until our 4th week to get our initial ROI ($3,000), until the 8th week until we get our next ROI ($7,000), and not until our 14th week would we get our final ROI (which happens to have the greatest value of $9,000).  For the 3 weeks we are working on Feature A, we incur the Cost of Delay of all three features: $3000 + $7000 + $9000 per week. This adds up to $19,000 per week times 3 weeks giving us a total Delay Cost incurred so far of $57,000.  We then move on to developing Feature B. For the 4 week this takes us to deliver we incur the Cost of Delay of Features B and C: $7000 + $9000 per week = $16,000 per week. So the Delay Cost is an additional $64,000, bringing us to a total of $121,000 worth of Delay Cost incurred so far.  At last, we can start working on Feature C. incurring the Cost of Delay of C during it’s development of $9000 per week for the 6 weeks it takes to build Feature C. This is another $54,000 of Delay Cost to add to our previous of $121,000 for a total of $175,000 Delay Cost incurred.

Do Most Valuable First

If we prioritized based on most to least valuable feature, it would take us until our 7th week to get our initial ROI ($9,000), until the 11th week until we get our next ROI ($7,000), and not until our 14th week would we get our final ROI ($3,000).  For the 6 weeks we are working on Feature C we incur the Cost of Delay of all three features: $3000 + $7000 + $9000) per week. This adds up to $19000 per week multiplied by 6 weeks giving us a total Delay Cost incurred so far of $114,000.  We then move on to developing Feature B. For the 4 week this takes us to deliver we incur the Cost of Delay of Features B and C: $7000 +$9000 per week = $16,000 per week multiplied by 4 weeks giving us a total Delay Cost is an additional $64,000, bringing us to a total of $178,000 worth of Delay Cost incurred so far.  At last, we can start working on Feature A. incurring the Cost of Delay of A during it’s development of $3000 per week for the 3 weeks it takes to build Feature A. This is another $9,000 of Delay Cost to add to our previous of $178,000 for a total of $187,000 Delay Cost incurred.

Use Cost of Delay Divided by Duration

If we develop the features based on whichever has the highest CD3 score we would do Feature B first, followed by Feature C, and finally Feature A.  For the 4 weeks we are working on Feature B we incur Cost of Delay of $3000 + $7000 + $9000 per week. Delay Cost = $76,000.  For the 6 weeks we are working on Feature C we incur Cost of Delay of $9000 + $3000 per week. Delay Cost = $72,000.  For the 3 weeks we are working on Feature A we incur Cost of Delay of $3000 per week.  Delay Cost = $9,000.  Total Delay Cost is $157,000.

Summary

Surprisingly, doing the most valuable feature first is not the best economic decision.
Next time you prioritize your portfolio, don’t just try to maximize value delivered. Limit your cost of delay.

Priority Method Cost of Delay All at the same time $247000 Shortest first $175000 Most valuable first $187000 CD3 $157000

The post An Introduction to Cost of Delay appeared first on LeadingAgile.

Categories: Blogs

My Journey: Agile, Culture & Transformation

Agilitrix - Michael Sahota - Wed, 06/10/2015 - 20:10

It has been 3 years since I wrote “An Agile Adopt […]

The post My Journey: Agile, Culture & Transformation appeared first on agilitrix.com - Michael Sahota.

Categories: Blogs

Group Vs team

Agile World - Venkatesh Krishnamurthy - Wed, 06/10/2015 - 19:07

image

I have seen many “Agile teams” working quietly without talking to their teammates. Every day they are at work sharp 9 AM, pick up a user story from the backlog, finish it and go.  Their interaction with other team members is limited. But each one is really happy as the are achieving something. This is where the line separating the “groups” and “teams gets blurred.

A team is a group of people who cannot work without depending on each other. That is, they have high interdependence on each other. However, a group need not have interdependence.  A good example of a group is a call center.  Typically in call centers, each attends the customer requests on their own and solves the problem. If one of the individual’s in the call center group is blocked with an issue, the rest are not affected. They could still continue working.

However, a team has shared the responsibility of delivery, and their work should be interdependent. In other words, team members have an agreed goal and the only way to achieve the goal is to work together. My thumb rule is, a user story cannot be moved to “Done” without the help of at least four other people :-)  

I believe that if your team room is very quiet you might want to check if there is anything wrong there. You should ask why teams are not talking to each other ? do they have shared responsibility of work ?  do they have a common goal to achieve or individual goals?  How many people do you need to complete an user story?

Photo courtesy: http://www.teams-forsuccess.com/working-groups-and-teams-are-they-the-same/

Categories: Blogs

Nexus Reifies

Learn more about our Scrum and Agile training sessions on WorldMindware.com

Nexus Scrum

I had the privilege of attending Scrum.org‘s 2-day seminar on Scaled Professional Scrum. The Nexusa connection or series of connections linking two or more things (direct translation from Latin a binding together), is the recommended scaling framework. The purpose of the Nexus is to manage dependencies between 3-9 Scrum Teams towards “reification”, to make an abstract idea real or concrete. This is ensured mostly through a single Product Owner, single Product Backlog, integrated (Nexus) Sprint Planning, Review and Retrospective and the addition of a Nexus Integration Team whose membership is made up mostly of Scrum team members internal to the Nexus, but often also includes other support personnel. The structure is very similar to LeSS, but perhaps even less prescriptive and is certainly much less prescriptive than SAFe. This is probably my favourite thing about the Nexus – the fact that it has just enough structure to be a model for scaling Scrum, but is light and flexible enough to accommodate all of the nuances that “just depend” on your situation. Like the other two above-mentioned scaling models, it places emphasis on the need for strong technical practices, continuous integration and the synchronization of events to facilitate integration. There is flexibility around synchronization, in that if the Nexus Sprint is 4 weeks in duration and teams within the Nexus want to do 2 or even 1 week Sprints, the model accommodates – as long as all of the teams’ work is combined into a fully integrated (reified) increment of potentially shippable product by the end of the Nexus Sprint.

 

Try out our Virtual Scrum Coach with the Scrum Team Assessment tool - just $500 for a team to get targeted advice and great how-to informationPlease share!
facebooktwittergoogle_plusredditpinterestlinkedinmail

The post Nexus Reifies appeared first on Agile Advice.

Categories: Blogs

Meet Our Sponsors at RallyON 2015

Rally Agile Blog - Wed, 06/10/2015 - 16:00

You’ve heard the saying, “It takes a village.” At Rally, we believe it takes an integrated network of technology and services partners to make the most of your Agile transformation.

That’s why the Expo floor at this year's RallyON!™ Agile conference pulls together top organizations such as CA Technologies, QA Symphony, and Scaled Agile, Inc. — all industry-leading Rally partners that can help you deliver at the new pace of change.

Whether you’re just starting with Agile or you’re ready to launch release trains across your enterprise, the RallyON 2015 sponsors will be there to help you:

  • Accelerate your transformation

  • Reduce testing time by 40 percent

  • Enable code sharing so you can build amazing things

  • Trace data from requirements to defects

  • Unify your development practices

  • Automate DevOps and continuous delivery

  • Get a mobile app for your Rally platform implementation

The RallyON Agile conference is just one week away, so register today.

Don’t miss this opportunity to mingle with the industry’s most innovative movers and shakers.

Learn more about all the sponsors at RallyON 2015.  

Platinum

CA Technologies

Gold

Scaled Agile, Inc.

QA Symphony

Silver

GitHub

Tasktop

XebiaLabs

Jama Software

Spotlight

Fullback Solutions / ASA for Rally

Services Booth and Expo Happy Hour

Eliassen Group

Icon ATG

Morgan Campbell
Categories: Companies

Growing Leaders

ScrumSense.com - Peter Hundermark - Wed, 06/10/2015 - 14:18

In the 21st century organisation where digital knowledge work replaces physical manufacturing work, the role of leaders and leadership has become a core need of organisations.

In fact the need for leadership and the role of leaders has been long discussed and understood by management experts. Amongst the most forward-thinking of these were Edwards Deming with his Fourteen Points for Management and Peter Drucker who practically invented modern management.

The Toyota Way to Lean Leadership includes a strong focus on leadership development. Jeffrey Liker and Gary Convis have documented that Toyota invests 25 years to grow a leader. This contrasts strongly with the quarterly-target based management prevalent in most traditional western organisations.

Robert Greenleaf’s famous 1970 essay The Servant as Leader started a revolution around the role of leaders in modelling how their followers should behave. The servant leader puts the needs and behaviour of the operative personnel first, over his or her own needs. The traditional organizational pyramid is literally flipped over. The subordinates are not supporting their manager in reaching his goals: the leaders are supporting their people in finding and aligning goals, in reaching their goals, and in growing as persons and employees.

This is congruent with the Chaordic Leadership approach from Dee Hock, the founder of Visa. He states: “Lead yourself, lead your superiors, lead your peers and free your people to do the same. All else is trivia.”

Motivation of employees features strongly in this frame. Dan Pink provides us with a helpful framework in his TED talk video and book “Drive”. Dan Pink explains that motivation is highly connected to three important matters. These are: Autonomy, Mastery and Purpose, which all goes hand in hand with the important matters of being a servant leader:

  • Respect and foster the self-organization. As stated in the fifth agile principle: “Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done”. A servant leader trusts the employees – even when they are in trouble. You might feel encouraged to intrude in the work of a challenged team (your good intention is probably to help them sort things out), but if you are taking over and directing the problem solving, the self-organization is actually injured for a while.
  • Give the employees the room and freedom to learn and improve their skills and mastery.
  • Give the employees the vision of where we are going. Knowing the direction, helps them make decisions.

A modern leader is someone who succeeds through the achievements of her employees. A great leader is someone that sets the direction and supports the environment in which the employees can act and evolve. A great leader is someone that sets the boundaries for the team, but leaves space and freedom for the employees to take responsibility and improve.

To grow into a modern leader you will likely need to unlearn what you were taught at business school and by traditional managers who mentored you. You can do worse than going back to the essentials taught by Deming and Drucker.

The post Growing Leaders appeared first on ScrumSense.

Categories: Blogs

Boundaries Determine Our Identities

Growing Agile - Wed, 06/10/2015 - 13:35
I attended a great talk and presentation recently, at t […]
Categories: Companies

2nd edition of Scrum & XP from the Trenches – “Director’s Cut”

Henrik Kniberg's blog - Wed, 06/10/2015 - 13:11

Guess what – I’ve updated Scrum and XP from the Trenches!

Scrum and XP from the Trenches 2nd edition

Eight years have passed, and this book is still really popular. Wow! I never could have imagined the impact this little book would make! I still bump into teams, managers, coaches, and trainers all over the place that use it as their primary guide to agile software development.

But the thing is, I’ve learned lots since 2007! So the book really needed an update.

Since publishing the book, I’ve had the opportunity to work with many agile and lean thought leaders; some have even become like personal mentors to me. Special thanks to Jeff Sutherland, Mary and Tom Poppendieck, Jerry Weinberg, Alistair Cockburn, Kent Beck, Ron Jeffries, and Jeff Patton – I can’t imagine a better group of advisors!

I’ve also had the chance to help a lot of companies implement these ideas in practice: companies in crisis as well as super-successful companies that want to get even better. All in all, it’s been a pretty mind-blowing journey!

When I reread this old book, I’m surprised by how many things I still agree with. But there are also some pages that I’d like to rip out and say “What the *&€# was I thinking? Don’t do it like that! There’s a much better way!”

Since the book is a real-life case study, I can’t change the story. What happened is what happened. But I can comment on it!

So that’s what the second edition is – an annotated version of the original book. Like a director’s cut. Think of it as me standing behind your shoulder as you read the book, commenting on stuff, cheering you on, with the occasional laugh and groan.

Here it is. Spread the word.

PS – the Amazon version is still 1st edition. Will be updated in a few weeks I think. But InfoQ provides both the printed version and the free online version.

Categories: Blogs

Morning Routines

I’ve come across a number of articles lately that have talked about the importance of a good morning routine (like this one), so I have been trying to practice some of these techniques.

One practice is to start the day with meditation, not checking any electronic devices. I'll admit, sometimes when I'm working from home I'll turn on the TV while making my morning coffee before heading up to the office, but I don't start responding to email first thing in the morning.

Another recommendation is to journal. Your mind is the most sharp in the morning so that is not the time to waste responding to emails. Instead figure out what your top two or three priorities are and try to get those accomplished first thing in the morning. Journaling can help you figure out where to start your day.

I have also found that I can change my evening routine to be more productive as well. I find when I'm traveling and staying at a hotel, that if I skip the TV and put on music instead, that I'm more focused and get more work done.

So is it time to change up some of your routines?
Categories: Blogs

Lead for Greatness

Portia Tung - Selfish Programming - Tue, 06/09/2015 - 22:07

Grow Leaders

Everyone a Leader

I have a dream. I dream of my two year old little girl all grown up, doing what she loves and loving what she does. She tells me how much fun she’s having at work. And that she’s great friends with the people there. And of how what she does makes the world a better place.

When I explain how far mankind’s relationship with work has come in the past twenty years alone, she laughs. She shakes her head at ideas such as “putting people first”, “Intent-Based Leadership“, “self-actualisation” and “collaboration”.

She tells me, “It all sounds like common sense, Mama. Why would people want to live and work any other way?”

I reply, “Because all too often, common sense isn’t common practice, Darling. And even though some ideas sound simple, they’re not easy to implement.”

She asks me to tell her the Ship story, of how long ago I made a wish to learn how to lead. And how one day, I met a former U.S Navy submarine captain who gave his power away to turn his ship around, taking his crew and boat from poor performance to greatness.  And of the leadership course that transformed the way I live and lead.

My grownup little girl turns to me and says with a bold and brave smile, “I intend to… make the most of my life because anything less would be a waste of the gift you have given me.”

And with that, we hug and I look forward to a brand new adventure of making more dreams come true.

Categories: Blogs

All new SAFe Foundations PowerPoint & Video to sync with SAFe 3.0 updates

Agile Product Owner - Tue, 06/09/2015 - 21:22

Hi Folks,

One of our most popular PowerPoint decks—SAFe Foundations—just got a major refresh to sync up with the most recent developments in SAFe 3.0. The previous version has been downloaded over 20 thousand times, so we know that it’s an important resource heavily used by our community.

thumb_foundations_2This slimmed down 26-slide deck is a great way to introduce SAFe and its underlying concepts to your executive leadership, team members, clients, etc.—anyone who is looking to get a high-level, easily digestible overview of SAFe, and what steps to take to achieve success through implementation.

What’s New

Here’s what you can expect from this update (version 3.0.2):

  • Streamlined content to more easily fit in a one-hour briefing
  • Updated House of Lean and Lean-Agile Values, which includes Innovation and Respect for Culture (in addition to Respect for People)
  • New SAFe Lean-Agile Principles—the immutable, fundamental tenets and economic underpinnings that drive the roles and practices in SAFe
  • Updated implementation guidance on how to incrementally achieve success one Agile Release Train at a time

In addition to the PowerPoint deck, I’ve recorded a 45-minute video with Scaled Agile’s Chief Technology Officer, Drew Jemilo, where we walk you through each slide, providing the backstory and thinking that went into each topic.

Go here to find both the video and PowerPoint deck: scaldagileframework.com/foundations (scroll to the bottom of the page).

So please download it, view it, present it, use it to support your SAFe journey. If you have comments, we’d love to hear them, ideally through responding to this blog post.

Stay SAFe!
–Dean and Drew

 

Categories: Blogs

April Dallas Recap: What Makes a Good Team Room?

DFW Scrum User Group - Tue, 06/09/2015 - 16:13
A big thank you to everyone who attended and contributed to this meeting!  This session was all about sharing our experiences with team rooms—the challenges in creating them, characteristics that make for good team rooms, and even having breakout spaces … Continue reading →
Categories: Communities

ScrumMaster: Servant Leader or Secretary?

Agile Management Blog - VersionOne - Tue, 06/09/2015 - 14:30

SecretaryIn Certified ScrumMaster (CSM) courses, many Scrum myths are busted. One such myth is that the ScrumMaster is somehow an administrative assistant to a development team, to a product owner or to an organization.

The Scrum Guide notes that the ScrumMaster is the servant leader to a development team, to the product owner and to the organization: http://scrumguides.org/

The guide describes this service as coaching, guiding, enabling understanding, enabling outcomes and so on. It does not talk about the ScrumMaster typing or writing.

Get Your Hands Off the Team’s Work

When ScrumMasters describe writing down “notes” for the team or scribing the team’s tasks at sprint planning this is what I advise ScrumMasterss everywhere: “Get your hands off of the team’s work!”

Will the team ever learn accountability and collective ownership if there’s an admin who will simply scribe everything for them? What will the team do if their ScrumMaster / Secretary is off for the day? Typically nothing is noted in these situations because the team has not assumed any level of accountability or ownership for tracking their own work.

I have the pleasure of working with high performing Scrum teams. Rarely is there a case where the team members expected someone to write down what they were saying or doing. The development team members make notes, type and do all that is necessary. They see it as just “work they have to do,” as opposed to falling into dogma from traditional system development lifecycle roles. The ScrumMasters serving in those situations, truly understand that they are focusing on the outcome, listening for impediments and pulling interactions back on track. When ScrumMasters feel like they have to “scribe,” their focus is split and they rarely pick up on the items that a great ScrumMaster notice.

Get Your Hands Off the Product Owner’s Work

A great ScrumMaster also needs to serve the product owner. I recently performed an assessment in which the ScrumMaster “kept” the product backlog. The product owner had not only never touched the product backlog, they did not even have access to it. Yikes! Once again ScrumMasters who are falling into this dysfunction, I advise:“Get your hands off of your product owners work!”

The product backlog is for the product owner. It is the way they manage the work that is needed for the product. While anyone can add items, this should not be interpreted as “just send your items to the ScrumMaster and they will take care of adding them to the product backlog.” Many dysfunctions are created out of this type of activity. Not only is the product owner not taking responsibility for the list of items to work on for the product, they are also more than likely are not actively refining and getting items to a state of ready by sprint planning time either.

Service to the Organization

Many associate the ScrumMaster with only a development team and a product owner. This tells me that these ScrumMasters are missing a third of the job. The ScrumMaster has responsibilities in the organization to teach, coach and guide the right outcomes.

In Certified Scrum Product Owner (CSPO) classes, I hear product owners describe interactions with steering committees or leaders that get heated, go down rat holes and never have a definitive outcome. “Where was your ScrumMaster when this was going on?” I ask. They go on to explain that no note taking was necessary so the ScrumMaster was probably not needed for the meeting.

The ScrumMaster is a neutral. They are a facilitator. Who better to come along when the product owner expects emotional discussions? As a skilled, neutral facilitator, a great ScrumMaster can keep the discussion fact-based and focus on the outcomes. If good ideas are generated during the discussion, he or she can suggest that these ideas be captured in the product backlog, so that ideas that can be discussed at a later time and don’t detract from the focus of the conversation at hand.

Conclusion

The ScrumMaster is not an administrative assistant or a secretary. They are the master of the Scrum framework. A process enabler. An advocate for the development team, the product owner and the organization. Great ScrumMasters who focus on servant leadership and outcomes will enable delivering of business value with each sprint.

Check out VersionOne’s upcoming CSM and CSPO training classes.

Categories: Companies

ScrumMaster: Servant Leader or Secretary?

Agile Management Blog - VersionOne - Tue, 06/09/2015 - 14:30

SecretaryIn Certified ScrumMaster (CSM) courses, many Scrum myths are busted. One such myth is that the ScrumMaster is somehow an administrative assistant to a development team, to a product owner or to an organization.

The Scrum Guide notes that the ScrumMaster is the servant leader to a development team, to the product owner and to the organization: http://scrumguides.org/

The guide describes this service as coaching, guiding, enabling understanding, enabling outcomes and so on. It does not talk about the ScrumMaster typing or writing.

Get Your Hands Off the Team’s Work

When ScrumMasters describe writing down “notes” for the team or scribing the team’s tasks at sprint planning this is what I advise ScrumMasterss everywhere: “Get your hands off of the team’s work!”

Will the team ever learn accountability and collective ownership if there’s an admin who will simply scribe everything for them? What will the team do if their ScrumMaster / Secretary is off for the day? Typically nothing is noted in these situations because the team has not assumed any level of accountability or ownership for tracking their own work.

I have the pleasure of working with high performing Scrum teams. Rarely is there a case where the team members expected someone to write down what they were saying or doing. The development team members make notes, type and do all that is necessary. They see it as just “work they have to do,” as opposed to falling into dogma from traditional system development lifecycle roles. The ScrumMasters serving in those situations, truly understand that they are focusing on the outcome, listening for impediments and pulling interactions back on track. When ScrumMasters feel like they have to “scribe,” their focus is split and they rarely pick up on the items that a great ScrumMaster notice.

Get Your Hands Off the Product Owner’s Work

A great ScrumMaster also needs to serve the product owner. I recently performed an assessment in which the ScrumMaster “kept” the product backlog. The product owner had not only never touched the product backlog, they did not even have access to it. Yikes! Once again ScrumMasters who are falling into this dysfunction, I advise:“Get your hands off of your product owners work!”

The product backlog is for the product owner. It is the way they manage the work that is needed for the product. While anyone can add items, this should not be interpreted as “just send your items to the ScrumMaster and they will take care of adding them to the product backlog.” Many dysfunctions are created out of this type of activity. Not only is the product owner not taking responsibility for the list of items to work on for the product, they are also more than likely are not actively refining and getting items to a state of ready by sprint planning time either.

Service to the Organization

Many associate the ScrumMaster with only a development team and a product owner. This tells me that these ScrumMasters are missing a third of the job. The ScrumMaster has responsibilities in the organization to teach, coach and guide the right outcomes.

In Certified Scrum Product Owner (CSPO) classes, I hear product owners describe interactions with steering committees or leaders that get heated, go down rat holes and never have a definitive outcome. “Where was your ScrumMaster when this was going on?” I ask. They go on to explain that no note taking was necessary so the ScrumMaster was probably not needed for the meeting.

The ScrumMaster is a neutral. They are a facilitator. Who better to come along when the product owner expects emotional discussions? As a skilled, neutral facilitator, a great ScrumMaster can keep the discussion fact-based and focus on the outcomes. If good ideas are generated during the discussion, he or she can suggest that these ideas be captured in the product backlog, so that ideas that can be discussed at a later time and don’t detract from the focus of the conversation at hand.

Conclusion

The ScrumMaster is not an administrative assistant or a secretary. They are the master of the Scrum framework. A process enabler. An advocate for the development team, the product owner and the organization. Great ScrumMasters who focus on servant leadership and outcomes will enable delivering of business value with each sprint.

Check out VersionOne’s upcoming CSM and CSPO training classes.

Categories: Companies

Knowledge Sharing


SpiraTeam is a agile application lifecycle management (ALM) system designed specifically for methodologies such as scrum, XP and Kanban.