Skip to content

Blogs

Context, rationale, and overview of Story / Impact Mapping

Small batch software development is goodThere are many advantages to working in small batches in software development:
Naive small batch software development is not quite so goodThere is also a significant disadvantage when small batches are done naively: disconnection from overall purpose.
Jeff Patton described this phenomena using the metaphor of a tree versus a bag of leaves:In my head I see a tree where the trunk is built from the goals or desired benefits that drive the system; big branches are users; the small branches and twigs are the capabilities they need; then finally the leaves are the user stories small enough to place into development iterations. After all that work, after establishing all that shared understanding I feel like we pull all the leaves off the tree and load them into a leaf bag - then cut down the tree.
The "bag of leaves" approach to backlogs is a context-free prioritised list of work items, that is, the naive, and I'd argue typical for Agile novices, understanding of "backlogs".  Although it nominally reflects a small batch approach, the "bag of leaves" discourages connection with underlying rationale and purpose, and therefore undermines autonomy.
Where I propose 5 attributes that indicate an effective approach to defining small batches in software developmentInstead, a more effective approach goes beyond work broken down into smaller pieces.  A more effective approach...
  • Highlights overall intention in the form of goals
  • Highlights the logical relationship between goals and the work items that need to be done
  • Visually represents goals, work items, and relationships in order to expose potential disagreement or misunderstanding
  • Is designed to be done in a collaborative way
  • Is iterative, that is, it does not require full completeness at the start
This is how I describe Story / Impact MappingBoth Jeff Patton's User Story Mapping and Gojko Adzic's Impact Mapping are (mostly equivalent) answers to naive approaches to defining small batches in software development (aka creating backlogs).  This is how I would describe how to do it:

Note: This process should be iterative.  It is only presented sequentially because that is easier to understand.  It is not necessary, nor advised to try to fully complete every step in order.  Instead, start with what you have and jump around and revisit steps as useful.
1. Identify GoalsWhat are you trying to accomplish?

This should be something observable and therefore measurable.  For this reason, I like the term "impact" given that it implies something that is observable.  However "goal" is more familiar to most people and I generally don't like introducing new jargon unnecessarily.

There may be more than one goal.
2. Identify Roles and ActivitiesWho has to engage in some Activity to achieve the Goals? OR What Activities need to be done by someone to achieve the Goals?

I generally prefer starting with Activities but in practice, this tends to be a very iterative process.

Note that Role is distinct from Person.  The Person may play other Roles that don't have anything to do with your Goals but rather their own Goals.  This typically means that you will need to add additional Goals that need to be supported.
3. Identify CapabilitiesWhat needs to exist in order for the Role to do an Activity?

A Capability may not need to be something that is built in software.
4. Identify Features and StoriesWhat needs to be done to create and/or enable a Capability?

Stories should be smaller pieces of work activity; Features are more to allow discussion at a higher level of granularity.
Categories: Blogs

Controlling Chaos with Kanban

TV Agile - Wed, 06/17/2015 - 17:04
Many Agencies suffer from the same problems: Too much to do, looming deadlines, dependencies on a few specialists, and insufficient predictability. On top of that, there are customers that want to know a fixed price and yet want to modify their decisions all the time. Is there a cure? Yes, Kanban! Kanban provides 6 practices […]
Categories: Blogs

Trust, Accountability, and Where Does the Time Go?

Johanna Rothman - Wed, 06/17/2015 - 08:56

As more of my clients transition to agile, many of them have a fascinating question:

How do I assess who is doing what on my team?

When I ask why they want to know, they say it’s all related to reviews, rewards, and general compensation. They are still discussing individual compensation, not team compensation.

When I ask why they want to reward individuals instead of the team, they say, “I am sure some people do more work than others. I want to reward them, and not the other people.”

Interesting idea. And, wrong for agile teams. Also wrong for any innovation or learning that you want to happen as a team (regardless of whether you are agile or not).

Agile is a team-based approach to work. Why would you want to reward some people more than others? If the team is not sure that they are working well together, they need to learn to provide each other feedback. If the team doesn’t know how to manage team membership, a manager can facilitate that membership discussion and problem-solving. Then, the managers can transition team membership issues to the team, with manager as backup/facilitator.

What I see is that the managers want to control team membership. Instead, why not let the team control its membership?

I often see that the managers want to control feedback: who provides it and who receives it. Instead, why not train everyone in how to provide and receive feedback?

When managers want to reward some people more than others, they imply that some people are less capable than others—something agile is supposed to fix with teamwork. Or, they wonder if some people are wasting time.

If managers trust their teams, managers don’t need to worry about accountability. They don’t have to worry about people “wasting time.” Agile creates transparency, which helps people to learn to trust each other and know when they are working on relevant work or not. If you encourage the team to add pairing or swarming (or both), you have the recipe for whole-team work and team momentum.

I don’t know anyone who goes to work thinking, “How can I waste my time today?” Do you? (If you do, why is that person on your team?)

I know plenty of people who do waste their time, because of technical debt, or experts creating bottlenecks, or team members who don’t want to work as part of a team. I bet you know many of these people, too.

But I don’t know anyone who wants to go to work to waste time and collect a paycheck without doing anything. Sure, there might be some people like that. I don’t know any.

In an agile team, the team members know who works hard and who doesn’t. A manager could trust the team to handle their compensation.

And, that would mean the manager has to relinquish control of much of what managers have done recently.

  • The manager would have to provide feedback and meta-feedback to people who want to learn how to provide and receive feedback.
  • The manager might provide coaching as to how to work in a team effectively. (This can be tough if a manager has never been part of a team that worked well.)
  • The manager would allow the team to control their compensation.

There’s more, and I’ll stop there.

What would managers do? They would stop interfering with the team’s progress. The manager might read “If Managers Don’t Give Performance Reviews, What Happens?

Managers control much less in agile. Managers enable more. They have to trust the teams. This is a culture change.

If you can’t trust your agile team or its team members, there is something wrong. You can investigate and either fix (manage the project portfolio) or ask the team to fix it (maybe with retrospectives as a first step). If you need to know who is accountable for what, you are asking the wrong question. The answer might be you.

If you are managing an agile team and you want to know about individual work or accountability, ask yourself what you really need. Ask yourself if there is another way to obtain the results you want. Maybe you won’t waste quite as much time.

Categories: Blogs

5 Key Skills Great Product Owners Have

Agile For All - Bob Hartman - Tue, 06/16/2015 - 20:18

Being Product Owner is hard. Actually, that’s not quite true. Anybody can make a list of things to build, call it a backlog, and bring it to a few meetings every month.

Being a great Product Owner is hard. Development skills are essential, but it doesn’t matter how fast your team delivers and how good your quality is unless you’re delivering the right thing every day. Identifying and expressing that “right thing” at every level of detail is a big job.

To improve our CSPO classes and our PO coaching, Bob and I made a tree of the skills that the best Product Owners use. While there are over two dozen skills on the tree—see, it’s not an easy job!—here are 5 you can work on to have an impact on your team right away.

Skill #1: Clearly express the connections between larger business goals and small backlog items

Many POs are good at making a business case for a project, but end up with a backlog of items that seem disconnected from that larger goal. By the time the team gets a few sprints into the project, half the team (and maybe the PO) have forgotten the business case and are just delivering on the details. They can no longer see the forest for the trees.

A great PO distills the business case into a short vision, makes it visible, and can point to it to explain every little thing on the backlog.

There are lots of ways to write a vision. One of my favorites is a variation on the elevator pitch template from Geoffrey Moore’s Crossing the Chasm. It goes like this:

FOR target customer
WHO need or dissatisfaction
product name IS A category
THAT key benefit (not key features, but what the features allow the customer to do).
UNLIKE alternative way of meeting the need
OUR PRODUCT how it’s better for the customer than the alternative.

Can you answer all the questions posed in this template? Could any member of your team? If not, you’d do well to fill in the template.

This doesn’t always make the most pithy, literate vision statement—it can sometimes be too formal or clumsy—but I love how it makes you answer some key questions about your product in a couple sentences. A slightly clumsy vision statement is better than none at all. And once you have a draft in this format, you can revise it read better.

Here’s an example for our Humanizing Work Conference:

FOR Agile For All alumni
WHO want to get better at their work
Humanizing Work IS AN Agile conference
THAT allows Agile For All alumni to learn advanced content from us and from other like-minded Agile practitioners.
UNLIKE a public Agile conference,
OUR CONFERENCE is built from the ground up on brain-friendly accelerated learning methods and has attendees who share a common language and understanding of the principles behind Agile (which lets every session go deeper).

This vision gives us a filter for every decision we make about the product. It prompts us to ask questions like:

  • Is that session relevant to Agile For All alumni?
  • Is it advanced content, or would they already have learned it in one of our classes?
  • Is ______ brain-friendly or just how conferences always work? Could we make it more brain-friendly?

Sometimes you have more than one kind of target customer. For the Behavior-Driven Development with Cucumber book I’m writing with Paul Rayner, we’re targeting people in four different roles who have different needs. BDD is a practice for collaborating across roles, so we have to meet the needs of all these roles. So, we drafted four parallel visions, like this:

BDD with Cucumber Book Vision

We had a client in education software who had to do this for students, teachers, and school administrators.

Once you have a vision statement, make it visible. If you’re on a colocated team, make a big poster and stick it on the wall. As a distributed team, we tend to use tools like Google Docs, Slack, and email. But when we’re together, we collaborate with paper.

Having a visible vision statement is great. But take it a step further: explicitly connect the vision to the details in your backlog. Say things like, “We’re doing this user story because it helps differentiate us from [alternative],” and, “Remember, we’re not building this story for everybody, it’s just for [your target customer], so we don’t have to do [some variation your target customer doesn’t need].”

You can practice this today:

  1. Write a vision statement if you don’t have one already. Don’t worry about making it perfect, just fill in the blanks for a first draft.
  2. Make it visible to your team. Incorporate their feedback to improve the vision.
  3. Go through the top items on your backlog. For each one, write a sentence or two explaining how that backlog item moves your product towards the vision.

Skill #2: Write user stories in a way that allows the team to contribute to the “what”

I was once coaching a PO and a couple of developers as they groomed their backlog. I noticed that all their user stories looked like this:

As a user, I want some new feature for the team to build.

Yes, their stories actually said “as a user,” but that’s not what struck me. I noticed none of their stories said why the user wanted the thing. There was no “so that” or “in order to.” So, I asked them why.

“Because they don’t need to know,” responded the PO, “I tell them what to build, and they build it.”

I looked over at the developers, expecting to see shock and dismay at being treated like short order cooks, and was suprised to see them nodding.

“Yep,” one said, “she tells us what the user wants and we build it.”

Just to see if it might make a difference, I asked them to try adding a “why” statement to their top story. We quickly realized that the PO had a similar relationship to her stakeholders—they tell her what they want and she goes to the team to get it built. But we talked about it for a few minutes and came up with a plausible “why.”

The moment the PO finished typing the change into their backlog tool, one of the developers said, “Oh, that’s what you’re trying to do. We can do it this other way for a tenth of the cost of what you asked for.” So they changed the story to accomodate the new “what” and went on to deliver the value for a much lower cost.

By adding the “why” to their story, this team was able to collaborate with their PO to make a better “what” than either could come up with on their own. Writing stories in a way that enables this collaborate is a key PO skill.

Here are two simple ways to do this:

  1. Make sure each story has a clear “why” somewhere that explains the larger value or goal for the user. Explicitly connect the story to the product or release vision if you can (as I described in the previous skill).
  2. For the “what” in the story, try describing the task the user is trying to accomplish rather than the feature in the system. Ask yourself, “Am I telling the team what to build, or am I inviting them to empathize with and solve a problem for a user?” The more your language in the story trends towards the latter, the more opportunity you have for the team to solve the problem in a way you haven’t imagined.
Skill #3: Modify the backlog in response to change (without having to throw out lots of work)

Whether it’s from customer feedback, your competitor’s latest release, a new technology, or a new regulation, the world changes in a way that affects your backlog.

Great POs respond to change by changing their backlog so it always reflects the current reality and your current knowledge. Merely competent POs add new backlog items and reprioritize in response to change, but they tend to leave out-of-date cruft in their backlogs. (Bad POs ignore change and treat the backlog as a big up front project plan, but you wouldn’t do that.)

The key to being able to respond to change without having to throw lots of work away is avoiding detail in your backlog until it’s unlikely to change. I know, that almost sounds too easy to be a real thing, but we have several clients doing it well.

Here’s how you do it:

  1. The top of your backlog should be user stories, sized small enough that 6–10 could fit into a sprint and with enough detail that the team feels comfortable committing to them in sprint planning. Most backlogs should have about 2 sprints worth of stuff like this on the top of the backlog and no more. When you get down to this level of detail beyond a few weeks in the future, you’re very likely to have to throw something away when the world changes. Small details are fragile to small changes.
  2. Beyond 2 sprints, don’t use user stories. Instead, use minimum marketable features, or MMFs. An MMF is a small (minimal) change in system behavior (feature) that’s valuable enough actually get in somebody’s hands (marketable). You can prioritize these in your backlog. You can size them against each other just like you do with stories. And you can measure the relationship between feature size and story size as you split these MMFs into stories. MMFs are usually stable enough to use in your backlog out to about 6 months.
  3. Beyond 6 months, even MMFs are too volatile, so just keep a cloud of themes you want to address over the next year or two, with the most likely ones at the top. This will let you keep your product strategy in mind without wasting time on detail that’s likely to change.
  4. Finally, keep a list of things you’re intentionally leaving out of your backlog. You can revisit this list quarterly to see if they need to get promoted to the backlog, but you won’t need to waste energy on these items every time you look at the backlog. In his Getting Things Done personal produtivity method, David Allen calls this the “Someday/Maybe List.” I have one that includes things like, “Learn German.” It’s something I might like to do someday, but don’t want to think about today.

Your backlog ends up looking like this, with small items on top, larger items further down, and eventually a cloud of options to draw from:

backlog structure

(I teach more about how to structure your backlog in this way, how to visualize it on a kanban board, and how to forecast releases over it in my 80/20 Product Ownership online course.)

Skill #4: Say “no” appropriately when necessary to maximize the value of your product

We give out a lot of NO buttons at conferences and in our classes. They say “no” 10 different ways, and Product Owners love them. They’re a joke, but they highlight a real issue.

no button

If you don’t say “no” intentionally, you can’t say “yes” intentionally. You still don’t get everything done, but instead of choosing what to say “no” to, you let it happen by accident or let others choose.

Great POs have the skill of saying “no” without being a jerk. Here are a few ways to say “no” when someone wants to add something to your backlog:

  1. “I’d like to add that item to the backlog, too. What should we pull out to make room for it?”
  2. “That’s not a priority for us at this time. We’re choosing to focus on [current focus] right now. Let’s consider it again [next month/quarter/year/whatever].”
  3. “Sorry, that doesn’t fit our current vision.”
  4. “Help me understand why you want this. Maybe there’s another way to solve this problem.”
  5. “No, because [reason why you’re choosing not to put this item in your backlog at this time].”
  6. “Here’s another thing you can do in the product that might help…”
  7. “Here’s another similar item in the backlog…”
  8. “Maybe we can include part of that in [another closely-related backlog item].”

You’re the Product Owner because your business leaders trust you to make the best decisions about the direction of your product. Accept that responsibility and be deliberate about what you say “yes” to and what you say “no” to.

You can practice this right away by looking at your backlog for things you’d like to say “no” to now. Either remove them if you the authority to or plan how you’re going to talk to the stakeholder if you don’t.

(If you’re not actually trusted to make decisions about what goes in your backlog, by the way, that’s an issue to address directly. No amount of skill in saying “no” fixes that one.)

Skill #5: Split big slices of value into small slices of value

Of all the skills a PO can have, this is the one that makes the great POs stand out. Great POs can take a big idea for a project and find early high-value, high-learning MMFs. They can take an MMF and find the essential user stories that make it marketable. They can take a big user story and split it into several small stories. And they can take even a small story and help the team build it in small, valuable slices.

Getting good at splitting has several benefits:

  • It helps you identify the low-value parts of every project, feature, and story so you can avoid ever building them.
  • It helps you find the high-value parts so you can build them first.
  • It helps you avoid building components, infrastructure, and technical tasks that don’t directly deliver value.
  • It reduces the risk in each sprint because you’re not betting the whole sprint on one or two stories. Getting 9 of 10 stories done is better than getting 1 story 90% done.
  • It gets you to done much faster, which enables earlier feedback and more frequent releases.

I’ve written a lot about how to split a big user story into smaller stories. My articles and posters on story splitting account for almost half the traffic on our website.

"How to Split a User Story" Flowchart

If you have stories near the top of your backlog that are larger than ⅙ of your team’s velocity, try using my story splitting patterns to find some small, valuable slices inside those stories.

If you’ve gotten value out of my story splitting material and you’d like to learn more about splitting at the other levels of detail (or just hone your story splitting skills), check out my new online course, 80/20 Product Ownership. In this self-paced online course, I share a set of splitting and backlog management techniques that, until recently, were only available through in-person training and coaching with me.

The post 5 Key Skills Great Product Owners Have appeared first on Agile For All.

Categories: Blogs

Agile Transformation … Learning What Doesn’t Work Is The Key To Success

Leading Agile - Mike Cottmeyer - Tue, 06/16/2015 - 13:55

I was teaching a private class a few weeks ago and a student in the class mentioned that his company had “already failed at Agile three different times.”

The way the statement was phrased really struck me. It made me very aware of my own opinions about Agile adoption and taught me a little about how this particular student, and possibly the organization, felt about adopting Agile.

What struck me was how easily the previous efforts were declared failures. If they truly had failed, there was no way I’d have been in the room teaching a private CSM class. It is possible that the company did not fully reach the perfect state of Agile nirvana and was not able to enjoy the bliss of a friction-less work environment populated with completely realized, high-performing teams of enlightened humans who delivered shippable product every sprint in a self-organized, cross-functional manner. Instead, they learned how not to transform the company… which is not a bad thing, because every misstep gets them one step closer to the solution that will open the lock (or at least a part of it.)

I have not failed. I’ve just found 10,000 ways that won’t work.
~Thomas Edison

Let’s say you have a guy who loves to sit on the couch binge watching old TV shows while jamming Ho Hos and Mountain Dew down his throat all day. He hears about people running marathons and decides this is for him. He goes to the mall… buys some expensive running shoes, a sweat suit, special moisture wicking socks… the whole bit. Then he stops by the “24 Hour Buffness” gym (which, shockingly is having a new membership special) and he signs up for a yearly membership. Then he goes home, puts on his new running shoes with the special moisture wicking socks, his new track suit, grabs his 24 Hour Buffness Membership Card, plops himself down on the couch, cracks open 1a3fa-rockford_filesa fresh can of Mountain Dew and a brand new Ho Ho. Then he fires up Netflix and puts on The Rockford Files, Season 5, Episode 4… The one where Tom Selleck guest stars as horribly annoying Lance White. Two days and 12 episodes later, our man realizes he’s not been to the gym, is not in shape, but seems to have consumed his weight in Ho Ho’s. If you ask him about it, he’ll explain how his situation is different, he can’t just start exercising like everyone else, The Rockford Files is not going to watch itself and let’s face it, that Lance White episode is a critical moment in Detective TV History. No Lance White, no Magnum. No Magnum, no Simon and Simon/Magnum mash-up episode… and then where would we be?

The real question is, has the “get in shape and run a marathon” project f2961982487_9b50b93f4bailed at this point? Or has he just not figured out the right way to get him motivated to change his lifestyle yet?

Now… imagine you had a building with 1,000 people just like him. Has the entire building failed? No… we just haven’t figured out the right way to make it work yet.

Change of any kind is difficult. Just because it doesn’t completely stick on the first try doesn’t mean it has failed or it didn’t work. It just means you did not figure out how to make it work YET.

And if you happen to work in an organization that has coaching and/or training for Agile, take that as a very positive sign. If the organization is willing to commit at that level, it’s a very good start. There are many places out there that are still convinced they don’t need help figuring this out.

Whether you want to call it transformation, adoption, “Project Nimble”, or whatever… the journey from a traditional approach to Agile is a long one. Transformation has to occur at multiple levels and it requires deep change to work practices and work culture. It may not happen all at once, it may be too tall of an order to just completely go Agile all at once. You may need to do it in stages (Basecamps). Your organization, like many others, may go through a number of attempts before things begin to stick.

If you know the old way doesn’t work, you have to make a choice. This choice takes place at an individual as well as an organizational level. Do you want to keep working in the old way, with annoying, but familiar pain? Or do you want to take a chance at trying something different that just might offer you a path to a better world of work? If you want the latter, you need to be willing to work for it, be patient and commit to keeping at it until you reach your goals. And, like any good sales person will tell you, No just means you haven’t figured out the right way to ask the question yet. When you do find it, they’ll say, “Yes”.

The post Agile Transformation … Learning What Doesn’t Work Is The Key To Success appeared first on LeadingAgile.

Categories: Blogs

News update 2015/06 – Creative Visualisation Workshops

ScrumSense.com - Peter Hundermark - Tue, 06/16/2015 - 09:33

A modern leader is someone who succeeds through the achievements of her employees. A great leader is someone that sets the direction and supports the environment in which the employees can act and evolve. A great leader is someone that sets the boundaries for the team, but leaves space and freedom for the employees to take responsibility and improve.

To grow into a modern leader you will likely need to unlearn what you were taught at business school and by traditional managers who mentored you. Read more from Peter Hundermark’s latest blog post on ‘Growing Leaders‘.

Creative Visualisation workshops

 

Leading Self-Organising Teams courseLeading Self-Organising Teams course Masterclass for Agile leaders and professionals
  • Understand the principles of Agile Leadership
  • Learn and apply state-of-the-art leadership tools
  • Enrich your skill set through practical role plays
  • Cope with tricky situations and identify blind spots

Only a few spots left, so be sure to book your place and get equipped to face the complex leadership challenges of the 21st century.

When: 07-08 July 2015
Where: FNB Conference & Learning Center, Sandton

Upcoming Courses

Certified Scrum Master (JHB)
23-24 June 2015

Certified Scrum Product Owner (CPT)
29-30 June 2015

Leading Self-Organising Teams (JHB)
07-08 July 2015 (Only 6 spots left!)

Visual Facilitation Foundation Workshops
27 July 2015 (CPT)
30 July 2015 (JHB)

Visual Facilitation Advanced Workshops
28 July 2015 (CPT)
31 July 2015 (JHB)

Course Schedule and Book Online

The post News update 2015/06 – Creative Visualisation Workshops appeared first on ScrumSense.

Categories: Blogs

Three Debugging Tricks for Lego Mindstorms

My daughter’s FFL Lego Club is over for the year, but I wanted to two helpful debugging things I learned along the way, that were especially helpful for newer kids.

This was the first year for a group of 5th and 6th graders and I noticed many of them ended up with some long procedural blocks of code to navigate the robot through obstacles across the board and back to the base. A really common problem was they’d make some progress, and then come back to the table only to discover the robot didn’t do what they expected after a “small change.”

The first fix for this was actually saving a copy off before making significant changes, just giving it a new name. The Lego LabVIEW environment didn’t make this easy as you have to cut and paste the blocks by hand into a new program, but it saves a lot of time with the kid’s longer programs. I’ve used Git for the same effect, but source control isn’t the focus here.

A second quick win was testing with the robot connected. Our boys and girls were very set on the program, download, run to the table loop, but I eventually had a few of them persuaded to test with the robot just connected via USB. The best part of this is you can select a single block and then run just that block with the robot. We used this several times to troubleshoot issues with arms that swung back too far and got stuck on lego pieces, which halts further execution and doesn’t give much feedback on what’s wrong.

Finally was the one I had the most fun with. Many kids’ programs had dozens of blocks connected. Often they would forget where they added tweaks to turn left, spin, or adjust the shooting arm. As a lover of sound effects I suggested adding sound effects before and after they added their tweaks so they’d know several things:

  • Did my changes actually get loaded on the robot?
  • Did my changes improved the program?
  • Exactly where to tweak the program again if it just needed another adjustment.

The sounds were fun on their own and they can easily be deleted when the program is ready to go.

Categories: Blogs

When the team says, “we need detailed requirements before we can estimate”…

Learn more about our Scrum and Agile training sessions on WorldMindware.com

The next time the team says “we can’t estimate without better requirements” what they actually mean is, “this is crazy, but hey…if you think you can accurately predict all the exact requirements and you can guarantee that nobody in this company will change their minds about those requirements between this moment and forever, then we’ll give you an estimate to hang your hat/noose on.”

Every group responsible for the creation and delivery of software (or any complex/creative product for that matter) will experience dissonance between the need to plan and the need to obey the laws of nature which prevent us from travelling through time and future-telling.  Business leaders have to finance the development of product; creative and technical leaders have to solve complex problems amidst dynamic, unpredictable, circumstances.  These conditions manifest as a dichotomy which is difficult to mediate (at best) and/or downright toxic (at worst).

On one hand, a common sentiment among project managers is: “The problem I have with the release planning stage is that without clear requirements, the developers don’t like to give estimates, even with story points.”

On the other hand, a common sentiment among developers is: “Stakeholders don’t understand what they’re asking for, if they knew the complexity of our technology they wouldn’t be asking those questions.”

If developers don’t like to provide estimates, it is likely because others in the organization have used their estimates as though they are accurate predictions of future. Thus, when said estimates turn out to be inaccurate they are used as punitive metrics in conversations about “commitment” and “velocity” and “accountability”.

Point of order: NOBODY CAN PREDICT THE FUTURE.

Try out our Virtual Scrum Coach with the Scrum Team Assessment tool - just $500 for a team to get targeted advice and great how-to informationPlease share!
facebooktwittergoogle_plusredditpinterestlinkedinmail

The post When the team says, “we need detailed requirements before we can estimate”… appeared first on Agile Advice.

Categories: Blogs

Northwind: Finding direct/transitive Reports in SQL and Neo4j’s Cypher

Mark Needham - Tue, 06/16/2015 - 00:53

Every few months we run a relational to graph meetup at the Neo London office where we go through how to take your data from a relational database and into the graph.

We use the Northwind dataset which often comes as a demo dataset on relational databases and come up with some queries which seem graph in nature.

My favourite query is one which finds out how employees are organised and who reports to whom. I thought it’d be quite interesting to see what it would look like in Postgres SQL as well, just for fun.

We’ll start off by getting a list of employees and the person they report to:

SELECT e."EmployeeID", e."ReportsTo"
FROM employees AS e
WHERE e."ReportsTo" IS NOT NULL;
 
 EmployeeID | ReportsTo
------------+-----------
          1 |         2
          3 |         2
          4 |         2
          5 |         2
          6 |         5
          7 |         5
          8 |         2
          9 |         5
(8 ROWS)

In cypher we’d do this:

MATCH (e:Employee)<-[:REPORTS_TO]-(sub)
RETURN sub.EmployeeID, e.EmployeeID 
 
+-------------------------------+
| sub.EmployeeID | e.EmployeeID |
+-------------------------------+
| "4"            | "2"          |
| "5"            | "2"          |
| "1"            | "2"          |
| "3"            | "2"          |
| "8"            | "2"          |
| "9"            | "5"          |
| "6"            | "5"          |
| "7"            | "5"          |
+-------------------------------+
8 rows

Next let’s find the big boss who doesn’t report to anyone. First in SQL:

SELECT e."EmployeeID" AS bigBoss
FROM employees AS e
WHERE e."ReportsTo" IS NULL
 
 bigboss
---------
       2
(1 ROW)

And now cypher:

MATCH (e:Employee)
WHERE NOT (e)-[:REPORTS_TO]->()
RETURN e.EmployeeID AS bigBoss
 
+---------+
| bigBoss |
+---------+
| "2"     |
+---------+
1 row

We still don’t need to join anything so the query isn’t that interesting yet. Let’s bring in some more properties from the manager record so we have to self join on the employees table:

SELECT e."FirstName", e."LastName", e."Title", manager."FirstName", manager."LastName", manager."Title"
FROM employees AS e
JOIN employees AS manager ON e."ReportsTo" = manager."EmployeeID"
WHERE e."ReportsTo" IS NOT NULL
 
 FirstName | LastName  |          Title           | FirstName | LastName |         Title
-----------+-----------+--------------------------+-----------+----------+-----------------------
 Nancy     | Davolio   | Sales Representative     | Andrew    | Fuller   | Vice President, Sales
 Janet     | Leverling | Sales Representative     | Andrew    | Fuller   | Vice President, Sales
 Margaret  | Peacock   | Sales Representative     | Andrew    | Fuller   | Vice President, Sales
 Steven    | Buchanan  | Sales Manager            | Andrew    | Fuller   | Vice President, Sales
 Michael   | Suyama    | Sales Representative     | Steven    | Buchanan | Sales Manager
 Robert    | King      | Sales Representative     | Steven    | Buchanan | Sales Manager
 Laura     | Callahan  | Inside Sales Coordinator | Andrew    | Fuller   | Vice President, Sales
 Anne      | Dodsworth | Sales Representative     | Steven    | Buchanan | Sales Manager
(8 ROWS)
MATCH (e:Employee)<-[:REPORTS_TO]-(sub)
RETURN sub.FirstName, sub.LastName, sub.Title, e.FirstName, e.LastName, e.Title
 
+----------------------------------------------------------------------------------------------------------------+
| sub.FirstName | sub.LastName | sub.Title                  | e.FirstName | e.LastName | e.Title                 |
+----------------------------------------------------------------------------------------------------------------+
| "Margaret"    | "Peacock"    | "Sales Representative"     | "Andrew"    | "Fuller"   | "Vice President, Sales" |
| "Steven"      | "Buchanan"   | "Sales Manager"            | "Andrew"    | "Fuller"   | "Vice President, Sales" |
| "Nancy"       | "Davolio"    | "Sales Representative"     | "Andrew"    | "Fuller"   | "Vice President, Sales" |
| "Janet"       | "Leverling"  | "Sales Representative"     | "Andrew"    | "Fuller"   | "Vice President, Sales" |
| "Laura"       | "Callahan"   | "Inside Sales Coordinator" | "Andrew"    | "Fuller"   | "Vice President, Sales" |
| "Anne"        | "Dodsworth"  | "Sales Representative"     | "Steven"    | "Buchanan" | "Sales Manager"         |
| "Michael"     | "Suyama"     | "Sales Representative"     | "Steven"    | "Buchanan" | "Sales Manager"         |
| "Robert"      | "King"       | "Sales Representative"     | "Steven"    | "Buchanan" | "Sales Manager"         |
+----------------------------------------------------------------------------------------------------------------+
8 rows

Now let’s see how many direct reports each manager has:

SELECT manager."EmployeeID" AS manager, COUNT(e."EmployeeID") AS reports
FROM employees AS manager
LEFT JOIN employees AS e ON e."ReportsTo" = manager."EmployeeID"
GROUP BY manager
ORDER BY reports DESC;
 
 manager | reports
---------+---------
       2 |       5
       5 |       3
       1 |       0
       3 |       0
       4 |       0
       9 |       0
       6 |       0
       7 |       0
       8 |       0
(9 ROWS)
MATCH (e:Employee)
OPTIONAL MATCH (e)<-[rel:REPORTS_TO]-(report)
RETURN e.EmployeeID AS employee, COUNT(rel) AS reports
 
+--------------------+
| employee | reports |
+--------------------+
| "2"      | 5       |
| "5"      | 3       |
| "8"      | 0       |
| "7"      | 0       |
| "1"      | 0       |
| "4"      | 0       |
| "6"      | 0       |
| "9"      | 0       |
| "3"      | 0       |
+--------------------+
9 rows

Things start to get more interesting if we find the transitive reporting relationships that exist. I’m not an expert at Postgres but one way to achieve this is by writing a recursive WITH query like so:

WITH RECURSIVE recursive_employees("EmployeeID", "ReportsTo") AS (
        SELECT e."EmployeeID", e."ReportsTo"
        FROM employees e
      UNION ALL
        SELECT e."EmployeeID", e."ReportsTo"
        FROM employees e, recursive_employees re
        WHERE e."EmployeeID" = re."ReportsTo"
)
SELECT re."ReportsTo", COUNT(*) AS COUNT
FROM recursive_employees AS re
WHERE re."ReportsTo" IS NOT NULL
GROUP BY re."ReportsTo";
 
 ReportsTo | COUNT
-----------+-------
         2 |     8
         5 |     3
(2 ROWS)

If there’s a simpler way let me know in the comments.

In cypher we only need to add one character, ‘*’, after the ‘REPORTS_TO’ relationship to get it to recurse as far as it can. We’ll also remove the ‘OPTIONAL MATCH’ so that we only get back people who have people reporting to them:

MATCH (e:Employee)<-[rel:REPORTS_TO*]-(report)
RETURN e.EmployeeID AS employee, COUNT(rel) AS reports
 
+--------------------+
| employee | reports |
+--------------------+
| "2"      | 8       |
| "5"      | 3       |
+--------------------+
2 rows

Now I need to find some relational datasets with more complicated queries to play around with. If you have any ideas do let me know.

Categories: Blogs

Championing P3 on a panel at DevTalks Bucharest

thekua.com@work - Mon, 06/15/2015 - 13:52

On Thursday I was presenting at DevTalks Bucharest, a 550+ developer conference with four different stages. I shared the panel with a Sabin Popa (Cloud Strategy Leader at IBM) and there was supposed to be another panelist but they had withdrawn. The topic of the panel was, “Innovation and data privacy – Keeping innovation alive in Cloud!” We first presented a bit about ourselves and our companies, where I was talking about our three pillars: Sustainable Business (P1), Software Excellence (P2), and Social Justice (P3).

Image courtesy of Image courtesy of Phillipp Krenn (@xeraa)

I talked a lot about observations we see around the world with our clients – these stories really resonated with people, particularly because:

  • It is based in reality (and not just sales demonstrations or fancy presentations)
  • People are genuinely interested in what other people are doing around the world.

During the panel, I talked about the responsibilities that we have as developers for privacy, and the responsibilities that we have as educated citizens to get this on the agenda of our parliaments. I touched upon the idea of Datensparsamkeit and that we can use our knowledge to start raising awareness among our friends and families.

Although not meaning to, I found that I probably spent a lot of time talking on the panel – but mostly because both the moderator and the other panelist wanted to keep asking questions of me. I had suggested that Sabin also give his own thoughts about what they could be done about data privacy.

When we touched the topic of innovation in the cloud, the topic of certification came up – something that didn’t really surprise me. One statement was that all platforms would be certified in the future (for security) and that would be considered one form of innovation. Although useful, I challenged the position, talking about how certification gives false confidence – particularly in services and products where people are involved. I think certification is definitely useful for testing mechanical parts, for testing platforms and products that never change – but software is soft. It constantly involves and once a platform is certified, doesn’t mean it will continue to pass the same tests. I see a lot of companies sell certification as an easy answer and I believe it gives companies a false sense of confidence.

An interesting question posed to the panel is what would we do if we are asked by our company to do something that is borderline unethical but not doing the task puts our job at risk and the mention that there are many more people to do our role. This, for me, was an easy answer. I talked about our responsibility of being digitally-educated and responsible citizens of the world and talked about the bravery and confidence of people like Snowden. I challenged everyone we should think through the consequences of mindlessly doing tasks that we don’t believe in and not think about just the consequences of the job right now, but question the consequences for our family, friends and the world we are creating for future generations.

Categories: Blogs

Working Towards Perfection

Leading Agile - Mike Cottmeyer - Mon, 06/15/2015 - 13:45

As Roy McAvoy (played by Kevin Costner) said in the movie Tin Cup, “Tempo is everything; perfection is unobtainable…” McAvoy was describing a golf swing with these words, and anyone who has played a round, been to a driving range, or even observed the professionals on a televised tournament can attest, the perfect swing is not out there. Yes, there are many swings that are close, but none are perfect and even if you catch lightening in a bottle for one swing you can’t recreate that swing consistently. At least this is my experience. I can’t get that ball to go where I want it to every time. I don’t hit each green. And definitely don’t sink every putt. But enough with golf.

My point for bringing up the golf swing was to point out it’s similarities to an organizational transformation. Whether you are trying to shift to a paperless office, become a more collaborative organization, or transform your organization into an entirely agile enterprise, I can tell you now, the perfect picture state you have in your head about what your organization should look like after this endeavor, is not going to be what your organization looks like. Also, it will probably take longer than you are hoping for.

Perfection is Unobtainable

You are not going to be perfect, no one is. I use the goal of the perfect end-state as a gauge to identify which areas we need to focus our efforts to improve upon in order to move us towards the perfect picture. To act as though we are failing because we are not perfect is not a sustainable solution. Strive for perfection, but know that you are not going to obtain it, and you can rest assured knowing that the drive towards perfection, not being perfect, is what is going to set you apart from your competitors.

Tempo is Everything

I have come to realize that we can only change at a certain pace. I have not been a part of a team that was able to successfully take on everything they needed to change and complete it in a day. I have observed many who have tried but they only ended up making things worse for themselves. Through observation, research and experience I have come to believe that in order to adopt a concept and make it part of your everyday life you will need to go through the 5 levels of adoption: Aware, Learning, Practicing, Proficient and Mastered.

We will walk through these in a different episode, the important thing to remember is that this takes time to work through. There will be times when the organization is going to be stuck in an Aware or Learning stage for longer than you thought it should be. My advice to you, if you are in this situation, is to review all aspects of the environment and controls that you have manipulated to get to this point. Chances are that you have inadvertently introduced multiple concepts that everyone must adopt and each concept requires them to go through the 5 levels. Granted these levels might only take half a day for some concepts, but it might take a week, a month, or longer per level depending on the complexity.

The next time you start a transformation of any magnitude, think of Roy and remember this one last bit.

Patience is Required.

Thanks for reading, I’m off to buy a new set of clubs to try to fix my golf game. Maybe we’ll talk about that conundrum after we review the levels of adoptions.

The post Working Towards Perfection appeared first on LeadingAgile.

Categories: Blogs

Portfolio Management

Notes from a Tool User - Mark Levison - Mon, 06/15/2015 - 09:16

Portfolio Management

(Presented as Part 3 in the Scrum Alone is Not Enough series.)

As mentioned in the introduction to the Scrum Alone is Not Enough series, Scrum is simply the framework and, to work best, other tools and patterns need to be incorporated to build the most effective systems.

Portfolio Management is one of those things that should be seriously considered.

Agile Portfolio Management is the art of deciding which big picture items the business wants its Scrum Teams to invest in over the next few months, so a Product Owner can understand the priorities and the Team can work on them in appropriate order.

There are several key underpinnings:

  • Scrum Teams are an indivisible unit – in organizations where it is hard to create permanent, effectively sized (5-9 people) Scrum Teams, effective Portfolio Management can be the first step to making this happen
  • All of the work flows through the Teams – there are no special projects done on the side
  • Product Owners collaborate with their Development Team (aka Doers) to achieve the business goals
  • Our focus is on delivering value and delighting the customer, not on maximizing the utilization of individual team members
  • Incremental funding/budgeting versus annual cycles
  • The organization has already visualized the state of the work using a Portfolio Kanban Board or similar tool

When your organization has only two or three Scrum Teams, it is entirely possible for Portfolio Management to be conducted by the Team Product Owners. They get together to discuss the big picture and where they want to steer their products over the next few months.

But when the number of teams becomes too large for informal coordination to be sufficient, or when Product Owners frequently find themselves outgunned by stakeholders with more clout, we need Portfolio Management to help keep the focus in the right place.

In the context of the World’s Smallest Online Bookstore we have a Portfolio Kanban board in this state (click for full version):

SmallestOnlineBookstore_Portfolio-before-review-meeting

Managing the Portfolio can be as simple as holding a review meeting every six to eight weeks. The meeting needs to include the individual Team Product Owners, Chief Product Owner, executives who have a significant stake in the direction of the business, and other key stakeholders (internal and possibly external). In each Portfolio Review Meeting the assembled group will need to ask itself these things:

What is the current state of the Product(s)? This should involve a demonstration of each Product as it currently stands. As with Scrum at the team level, all decisions need to be grounded by seeing the actual working software.

The oft used traffic light Project Status Reporting – Green for all good; Yellow for needs some help, and Red for in trouble – are worse than useless, and are dangerously misleading. Instead of focusing on value delivered, they focus on schedule and a percentage of work done, versus actual features complete and value delivered. This leads to unrealistic beliefs about progress in a project because, right up and until the last moment, people believe that the project is healthy.

All of Scrum is based around the idea that value is only accrued or measured when it is truly done. In the above case of the Smallest Online Bookstore, that would mean demonstrating the items in the “Ready for Deployment” column. Perhaps we also demonstrate the features that are missing Online Help but are otherwise Ready for Deployment, such as Ship a Book Home and Star Reviews.

Are the current features that have been committed or are being worked on still the most important?

In the case of the Smallest Online Bookstore, that means asking whether the next items in the Product Backlog (“Find a book similar to…,” “Fancy Gift Card Graphics,” and “Support Payment Options Beyond MasterCard”) are still the most important. 

Reviewing the current Sprint should be a safety check, and not an alteration. Work that has already started isn’t quite sacrosanct, but it should be very, very unusual for it to be cancelled/changed now. If changes happen once the work has been started, that implies a failure at both the PO and Portfolio level. The effect of cancelling work that is in progress will be quite devastating and will affect the Team’s morale for some Sprints to come. In the case of the Bookstore there is nothing I can imagine that would justify cancelling a Sprint. From personal experience, Sprints might get cancelled because the company got bought and the product no longer makes sense, or a major client whose needs we’re working on fires the company. We’re talking major and unforeseeable stuff, not just someone changing their mind.

Are there any impediments to achieving the current Portfolio goals? What can this group do to resolve them?

Having stakeholders, executives, and Product Owners jointly agree to a direction and assume accountability for sticking to the plan will remove most, if not all, common obstacles. At the Bookstore, John’s team is having trouble getting daily pickups scheduled with Fedex. It turns out this is a contract issue and beyond the ability of the Scrum Team to handle on their own. Sarah’s team is being asked to do some work around displaying purchase histories by a stakeholder. Since the work doesn’t fit inside of the features they’ve been asked to work, and since their PO wasn’t able to solve the problem alone, we need someone above the team level to remind the stakeholder of the team’s focus.

Are there any feature areas that the Scrum Teams have invested enough time and effort? Have we validated/tested the current features with the Market? Do we want to Keep, Release or Transform (aka Pivot)?

Will the current work around Gift Cards, Premium Shipping, and Book Returns likely be sufficient?

Are there open slots for work in front of the Scrum Teams?

If so in what feature areas would the Portfolio group like them to invest? What would we like to see finished next? Is there sufficient capacity in the next 6-8 weeks that it is worth asking the teams to build new items?

Note this is not a commitment on behalf of the Scrum Teams – instead it’s an invitation to make these the next work items if there is capacity. This is where features in the Idea column of the board can be considered.

What are ballpark estimates for the next few features that are likely to considered?

Instead of Story Point estimates, simply ask whether the feature is likely to be bigger or smaller than a well-known feature that we’ve built in the past. At the Bookstore, that would mean asking the team if “Publishers can add books” is bigger or smaller than “Buy a Book” (large) and “Write Reviews” (smaller). To be clear, we’re not looking for precision – simply “bigger” or “smaller”. Some people use Tshirt sizing (S, M, L, XL) to illustrate the goal and keep it generalized.

Portfolio Management is really about handing out a budgetary envelope for a certain feature area, and then asking the Scrum Teams to build the best feature they can inside that budget.

But as you can imagine, when a group hands out a budget envelope, they want to believe that they will get value in return. This can sometimes cause conflict in the case of idle workers. We will address that in the next post in this series.

Categories: Blogs

The Willpower Instinct: Reducing time spent mindlessly scrolling for things to read

Mark Needham - Sat, 06/13/2015 - 01:12

I recently finished reading Kelly McGonigal’s excellent book ‘The Willpower Instinct‘ having previously watched her Google talk of the same title

My main takeaway from the book is that there are things that we want to do (or not do) but doing them (or not as the case may be) isn’t necessarily instinctive and so we need to develop some strategies to help ourselves out.

In one of the early chapters she suggests picking a habit that you want to do less off and write down on a piece of paper every time you want to do it and how you’re feeling at that point.

After writing it down you’re free to then follow through and do it but you don’t have to if you change your mind.

I was quite aware of the fact that I spend a lot of time idly scrolling from email to Twitter to Facebook to LinkedIn to news websites and back again so I thought it’d be interesting to track when/why I was doing this. The annoying thing about this habit is that it can easily eat up 20-30 minutes at a time without you even noticing.

I’ve been tracking myself for about three weeks and in the first few days I noticed that the first thing I did as soon as I woke up was grab my phone and get into the cycle.

It was quite frustrating to be lured in so early in the day but one of the suggestions in the book is that feeling guilty about something is actually detrimental to our progress. Instead we should note why it happened and then move on – the day isn’t a write off because of one event!

Kelly suggests that if we can work out the times when we’re most likely to fall into our habits then we can pre-plan a mitigation strategy.

From looking over my notes the following are the reasons why I want to start mindlessly scrolling:

  • I’m stuck on the problem I’m working on
  • I’m bored
  • I’m tired
  • I’m hungry
  • I’m getting distracted by notifications
  • I want to not think for a while

The notifications bullet is easy to address – I turn off notifications on my phone for 4 hours at a time so I don’t even know there’s anything to read.

I was intrigued to note that I got distracted when stuck on a problem – the main take away here is to check whether the urge to scroll mindlessly is being driven by having to think hard. If it is then I can choose to either get back to it or go for a short walk and then come back. But definitely don’t start scrolling!

I often find myself bored on my commute to work so I’ve addressed this by working out a book/paper I’m going to read the night before and then having that ready for the journey. Lunch time is prime time for mindless scrolling as well so I’ve filled that time with various computer science/data science videos.

Since I started tracking my scrolling I’ve found myself sleeping earlier so my assumption is that the extra hours awake were being spent mindlessly scrolling which led to being more tired so a win all around in that respect.

Something I’ve noticed is that I’m sometimes wasting time on other activities which I’m are not ‘forbidden’ but are equally unconstructive e.g. chat applications / watching music videos.

The former are obviously useful for communicating with people so I’ve been trying to use them only when I actually want to chat to someone rather than mindlessly looking for messages to read.

I also find myself not wanting to write down the times I’ve mindlessly scrolled when I’m doing it a lot on a given day. Being aware of this is helpful as I just write it down anyway and get on with the day.

The summary of my experience so far is it seems beneficial – I don’t think I’ve lost anything by not checking those mediums so often and I’ve definitely read a lot more than I usually do and been more focused as well.

Now I need to go and try out some of the other exercises from the book – if you’ve read it / tried out any of the tips I’d love to hear what’s worked well for you.

Categories: Blogs

Neo4j: Using LOAD CSV to help explore CSV files

Mark Needham - Fri, 06/12/2015 - 01:15

During the Neo4j How I met your mother hackathon that we ran last week one of the attendees noticed that one of the CSV files we were importing wasn’t creating as many records as they expected it to.

This is typically the case when there’s some odd quoting in the CSV file but we decided to look into it.

The file in question was one containing references made in HIMYM. The first 5 lines look like this:

$ head -n 5 data/import/references.csv
ReferencedEpisodeId,ReferencingEpisodeId,ReferenceText
168,184,"Marshall will eventually hear back from the New York State Judicatory Committee in Something New, which will become a main plot point of Season 9."
168,169,Barney proclaiming to be done with Robin will be the focal point of Lobster Crawl.
58,57,"Barney finally confronts his saboteur (Abby, whom he slept with in Ten Sessions) in Everything Must Go."
58,63,"Barney finally confronts his saboteur (Abby, whom he slept with in Ten Sessions) in Everything Must Go."

And this is how many lines the Unix ‘wc’ command sees:

$ wc -l data/import/references.csv
     782 data/import/references.csv

So we might expect that there are going to be 782 records created if we import that file into Neo4j. Let’s run a quick query in Neo4j to see what it thinks:

LOAD CSV WITH HEADERS 
FROM "file:///Users/markneedham/projects/neo4j-himym/data/import/references.csv" 
AS row
return count(*)
 
==> +----------+
==> | count(*) |
==> +----------+
==> | 636      |
==> +----------+
==> 1 row

So we have 146 less records than we expected which means Neo4j is treating multiple lines as one CSV line in some cases.

Let’s go back to the Unix command line to try and work out which lines those are. There must be some lines which start with part of the ‘ReferenceText’ rather than a ‘ReferenceEpisodeId’ so let’s extract the first column and see what’s going on there:

$ cat data/import/references.csv | cut -d"," -f1 | grep -v  '[0-9]\+$'| head -n 10
ReferencedEpisodeId
This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny
This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny
This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny
This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny
This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny
This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny
This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny
This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny
Also

We’ve extracted the first column and then filter the output to only keep rows which don’t contain all numbers which will be our rogue rows.

Let’s switch back to Neo4j land to see which rows it thinks contains these fragments of text:

LOAD CSV WITH HEADERS FROM "file:///Users/markneedham/projects/neo4j-himym/data/import/references.csv" AS row
WITH row WHERE row.ReferenceText =~ ".*This is the Mother's first.*"
RETURN row.ReferencedEpisodeId, row.ReferencingEpisodeId, row.ReferenceText
 
==> +--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
==> | row.ReferencedEpisodeId | row.ReferencingEpisodeId | row.ReferenceText                                                                                                                                                                                                                                                                                                                                                                     |
==> +--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
==> | "45"                    | "37"                     | "This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9." |
==> | "45"                    | "184"                    | "This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9." |
==> +--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Interestingly it only returns two rows containing that phrase whereas we see it at least 8 times. Initially I thought this was an issue with the LOAD CSV command but if we filter the rows to only return ones that have a ‘ReferencedEpisodeId’ of ’45’ then we do see them returned:

==> +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
==> | row                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
==> +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "53", ReferenceText -> "The website counting down to the next slap (slapcountdown.com) that Marshall sends Barney reaches zero in Slapsgiving, when the third slap is delivered."}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "55", ReferenceText -> "Ted gets rid of his butterfly tramp stamp through ten weekly sessions of laser tattoo removal between The Platinum Rule and Ten Sessions, over the course of which he meets, asks out, and eventually starts dating his dermatologist, Stella Zinman."}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "57", ReferenceText -> "Ted gets rid of his butterfly tramp stamp through ten weekly sessions of laser tattoo removal between The Platinum Rule and Ten Sessions, over the course of which he meets, asks out, and eventually starts dating his dermatologist, Stella Zinman."}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "56", ReferenceText -> "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton.
==> This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9."}  |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "200", ReferenceText -> "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton.
==> This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9."} |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "100", ReferenceText -> "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton.
==> This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9."} |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "86", ReferenceText -> "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton.
==> This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9."}  |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "113", ReferenceText -> "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton.
==> This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9."} |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "161", ReferenceText -> "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton.
==> This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9."} |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "37", ReferenceText -> "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton.
==> This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9."}  |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "184", ReferenceText -> "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton.
==> This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9."} |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "37", ReferenceText -> "This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9."}                                                                                                                                                                                                                                                                                                                                                                         |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "184", ReferenceText -> "This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9."}                                                                                                                                                                                                                                                                                                                                                                        |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "63", ReferenceText -> "Marshall makes other home-made websites in Everything Must Go (lilyandmarshallselltheirstuff.com) and The Sexless Innkeeper (itwasthebestnightever.com), where Lily and Future Ted mention it being a problem."}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "92", ReferenceText -> "Marshall makes other home-made websites in Everything Must Go (lilyandmarshallselltheirstuff.com) and The Sexless Innkeeper (itwasthebestnightever.com), where Lily and Future Ted mention it being a problem."}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
==> +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

So the actual problem is that the regex matcher doesn’t deal with the new line in the string.

Our next step is therefore to get rid of new lines within strings. I spent ages trying to find the appropriate command before coming across the following use of awk which does the job:

$ cat data/import/references.csv | awk '(NR-1)%2{$1=$1} {print $0}' RS=\" ORS=\" | wc -l
637
 
$ cat data/import/references.csv | awk '(NR-1)%2{$1=$1} {print $0}' RS=\" ORS=\" > data/import/refs.csv

Let’s try the LOAD CSV command again:

LOAD CSV WITH HEADERS FROM "file:///Users/markneedham/projects/neo4j-himym/data/import/refs.csv" AS row
WITH row WHERE row.ReferenceText =~ ".*This is the Mother's first.*"
RETURN row.ReferencedEpisodeId, row.ReferencingEpisodeId, row.ReferenceText
 
==> +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
==> | row.ReferencedEpisodeId | row.ReferencingEpisodeId | row.ReferenceText                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
==> +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
==> | "45"                    | "56"                     | "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton. This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9." |
==> | "45"                    | "200"                    | "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton. This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9." |
==> | "45"                    | "100"                    | "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton. This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9." |
==> | "45"                    | "86"                     | "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton. This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9." |
==> | "45"                    | "113"                    | "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton. This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9." |
==> | "45"                    | "161"                    | "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton. This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9." |
==> | "45"                    | "37"                     | "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton. This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9." |
==> | "45"                    | "184"                    | "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton. This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9." |
==> | "45"                    | "37"                     | "This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9."                                                                                                                                                                                                                                                                                                                                                                        |
==> | "45"                    | "184"                    | "This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9."                                                                                                                                                                                                                                                                                                                                                                        |
==> +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

And there we go!

Update

Michael pointed out that I could have used the dotall regex flag at the beginning of the regular expression in order to search across new lines without having to remove them! In that case the query would read like this:

LOAD CSV WITH HEADERS FROM "file:///Users/markneedham/projects/neo4j-himym/data/import/references.csv" AS row
WITH row WHERE row.ReferenceText =~ "(?s).*This is the Mother.*"
RETURN row
 
==> +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
==> | row                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
==> +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "56", ReferenceText -> "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton.
==> This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9."}  |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "200", ReferenceText -> "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton.
==> This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9."} |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "100", ReferenceText -> "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton.
==> This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9."} |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "86", ReferenceText -> "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton.
==> This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9."}  |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "113", ReferenceText -> "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton.
==> This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9."} |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "161", ReferenceText -> "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton.
==> This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9."} |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "37", ReferenceText -> "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton.
==> This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9."}  |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "184", ReferenceText -> "Ted eventually acquires the yellow umbrella in No Tomorrow (after the Mother leaves it behind at the St. Patrick's Day party, as seen in How Your Mother Met Me), and leaves it in Cindy's and the Mother's apartment in Girls Versus Suits. The umbrella is also seen/referenced in many other episodes, including Right Place, Right Time, Big Days, and Farhampton.
==> This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9."} |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "37", ReferenceText -> "This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9."}                                                                                                                                                                                                                                                                                                                                                                         |
==> | {ReferencedEpisodeId -> "45", ReferencingEpisodeId -> "184", ReferenceText -> "This is the Mother's first on-screen appearance with the yellow umbrella. Previously she appeared in Lucky Penny, with her head obscured by a bridal veil. She is seen again in No Tomorrow, again hidden by the umbrella, her ankle is seen briefly in Girls Versus Suits, and she gets her first proper appearance in Something New, after which she appears throughout Season 9."}                                                                                                                                                                                                                                                                                                                                                                        |
==> +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Categories: Blogs

Mac OS X: GNU sed – Hex string replacement / replacing new line characters

Mark Needham - Thu, 06/11/2015 - 23:38

Recently I was working with a CSV file which contained both Windows and Unix line endings which was making it difficult to work with.

The actual line endings were HEX ‘0A0D’ i.e. Windows line breaks but there were also HEX ‘OA’ i.e. Unix line breaks within one of the columns.

I wanted to get rid of the Unix line breaks and discovered that you can do HEX sequence replacement using the GNU version of sed – unfortunately the Mac ships with the BSD version which doesn’t have this functionaltiy.

The first step was therefore to install the GNU version of sed.

brew install coreutils
brew install gnu-sed --with-default-names

I wanted to replace my system sed so that’s why I went with the ‘–with-default-names’ flag – without that flag I believe the sed installation would be accessible as ‘gs-sed’.

The following is an example of what the lines in the file look like:

$ echo -e "Hello\x0AMark\x0A\x0D"
Hello
Mark

We want to get rid of the new line in between ‘Hello’ and ‘Mark’ but leave the other one be. I adapted one of the commands from this tutorial to look for lines which end in ‘0A’ where that isn’t followed by a ‘0D':

$ echo -e "Hello\x0AMark\x0A\x0D" | \
  sed 'N;/\x0A[^\x0D]/s/\n/ /'
Hello Mark

Let’s go through the parts of the sed command:

  • N – this creates a multiline pattern space by reading a new line of input and appending it to the contents of the pattern space. The two lines are separated by a new line.
  • /\x0A[^\x0D]/ – this matches any lines which contain ‘OA’ not followed by ‘OD’
  • /s/\n/ / – this substitutes the new line character with a space for those matching lines from the previous command.

Now let’s check it works if we have multiple lines that we want to squash:

$ echo -e "Hello\x0AMark\x0A\x0DHello\x0AMichael\x0A\x0D"
Hello
Mark
Hello
Michael
 
$ echo -e "Hello\x0AMark\x0A\x0DHello\x0AMichael\x0A\x0D" | \
  sed 'N;/\x0A[^\x0D]/s/\n/ /'
Hello Mark
Hello Michael

Looks good! The actual file is a bit more nuanced so I’ve still got a bit more work to do but this is a good start.

Categories: Blogs

An Introduction to Cost of Delay

Leading Agile - Mike Cottmeyer - Wed, 06/10/2015 - 22:20

I was recently watching an episode of Shark Tank. I loved the unfiltered statement from Kevin O’Leary (Mr. Wonderful) toward an entrepreneur seeking an investor in his company.

I’m here to make money!

If you’re a fan of Shark Tank, you’ll notice something about Mr. Wonderful.  He keeps the conversation focused on the money.  When will he get his money back? How many multiples of his investment should he expect to get back? Other investors (and many of our stakeholders) don’t focus enough on the money.  Particularly, what is the cost of delaying the implementation of one feature over another.

I tell customers that if they want to save or make the most money, they need to prioritize their backlog… by money.  My colleague Isaac recently wrote something similar in his blog post about how to prioritize work.

One way you can prioritize work by money is by using cost of delay.  Sounds kind of weird, right?  Aren’t we always telling people to prioritize their backlogs by customer value?  When you ask customers (or the business) which features are the highest priority, all too often they say all of them.  (Jim Hayden makes reference to this in a recent podcast, when teams don’t prioritize or limit their work in process.  People are really good at starting things but not finishing them.)  Don’t just ask what is the most valuable. Ask the question, “what will cost us the most, by delaying its delivery?”  That’s really what we’re doing.  We’re not profiting from a feature that is not in production, so therefore, we are losing money every day it’s not out there.

If I have 3 features to choose from, each with a different worth to the business and each taking a different amount of time to implement, how do I make the best economic decision on what to finish first?  I use Cost of Delay.

Cost of Delay Step 1 – Compare Features

Let’s put the 3 features in a table and compare them.  I intentionally tried to keep this simple by multiplying the value of the features by $1,000. Imagine what this would look like if your features were worth tens of thousands or hundreds of thousands of dollars?

Duration Value CD3 Feature A 3 weeks $3000 1 Feature B 4 weeks $7000 1.75 Feature C 6 weeks $9000 1.5 Step 2 – Visualize Scenarios

Taking what we have learned in the table of Step 1, let’s visualize different scenarios, showing when we could get a return on our investment, given a choice of priority.

  1. No priority at all. Do all at the same time.
  2. Do the features that take the shortest amount of time first.
  3. Do the features that are the most valuable first.
  4. Lastly, do the features that have the highest CD3 (value divided by duration) score.

Remember, regardless of our choice of priority, all of the features are done by the 14th week.

Visualize Prioritization Scenarios

For every week features are not making us money, they are costing us money.
Let’s do some math!

Step 3 – Priority Impact on Cost of Delay

Using the three features we can look at the financial impact of the four alternatives.

All at the same time (No Priority)

If we started all of the features at the same time, we would get any and all of that $19,000 of value on our 14th week.  For the 13 weeks we are working, we incur the Cost of Delay of all three features: $3000 + $7000 + $9000 per week. Delay Cost incurred is $247,000.

Do the Shortest Job First

If we prioritized based on shortest to longest length of time to complete a feature, it would take us until our 4th week to get our initial ROI ($3,000), until the 8th week until we get our next ROI ($7,000), and not until our 14th week would we get our final ROI (which happens to have the greatest value of $9,000).  For the 3 weeks we are working on Feature A, we incur the Cost of Delay of all three features: $3000 + $7000 + $9000 per week. This adds up to $19,000 per week times 3 weeks giving us a total Delay Cost incurred so far of $57,000.  We then move on to developing Feature B. For the 4 week this takes us to deliver we incur the Cost of Delay of Features B and C: $7000 + $9000 per week = $16,000 per week. So the Delay Cost is an additional $64,000, bringing us to a total of $121,000 worth of Delay Cost incurred so far.  At last, we can start working on Feature C. incurring the Cost of Delay of C during it’s development of $9000 per week for the 6 weeks it takes to build Feature C. This is another $54,000 of Delay Cost to add to our previous of $121,000 for a total of $175,000 Delay Cost incurred.

Do Most Valuable First

If we prioritized based on most to least valuable feature, it would take us until our 7th week to get our initial ROI ($9,000), until the 11th week until we get our next ROI ($7,000), and not until our 14th week would we get our final ROI ($3,000).  For the 6 weeks we are working on Feature C we incur the Cost of Delay of all three features: $3000 + $7000 + $9000) per week. This adds up to $19000 per week multiplied by 6 weeks giving us a total Delay Cost incurred so far of $114,000.  We then move on to developing Feature B. For the 4 week this takes us to deliver we incur the Cost of Delay of Features B and C: $7000 +$9000 per week = $16,000 per week multiplied by 4 weeks giving us a total Delay Cost is an additional $64,000, bringing us to a total of $178,000 worth of Delay Cost incurred so far.  At last, we can start working on Feature A. incurring the Cost of Delay of A during it’s development of $3000 per week for the 3 weeks it takes to build Feature A. This is another $9,000 of Delay Cost to add to our previous of $178,000 for a total of $187,000 Delay Cost incurred.

Use Cost of Delay Divided by Duration

If we develop the features based on whichever has the highest CD3 score we would do Feature B first, followed by Feature C, and finally Feature A.  For the 4 weeks we are working on Feature B we incur Cost of Delay of $3000 + $7000 + $9000 per week. Delay Cost = $76,000.  For the 6 weeks we are working on Feature C we incur Cost of Delay of $9000 + $3000 per week. Delay Cost = $72,000.  For the 3 weeks we are working on Feature A we incur Cost of Delay of $3000 per week.  Delay Cost = $9,000.  Total Delay Cost is $157,000.

Summary

Surprisingly, doing the most valuable feature first is not the best economic decision.
Next time you prioritize your portfolio, don’t just try to maximize value delivered. Limit your cost of delay.

Priority Method Cost of Delay All at the same time $247000 Shortest first $175000 Most valuable first $187000 CD3 $157000

The post An Introduction to Cost of Delay appeared first on LeadingAgile.

Categories: Blogs

My Journey: Agile, Culture & Transformation

Agilitrix - Michael Sahota - Wed, 06/10/2015 - 20:10

It has been 3 years since I wrote “An Agile Adopt […]

The post My Journey: Agile, Culture & Transformation appeared first on agilitrix.com - Michael Sahota.

Categories: Blogs

Group Vs team

Agile World - Venkatesh Krishnamurthy - Wed, 06/10/2015 - 19:07

image

I have seen many “Agile teams” working quietly without talking to their teammates. Every day they are at work sharp 9 AM, pick up a user story from the backlog, finish it and go.  Their interaction with other team members is limited. But each one is really happy as the are achieving something. This is where the line separating the “groups” and “teams gets blurred.

A team is a group of people who cannot work without depending on each other. That is, they have high interdependence on each other. However, a group need not have interdependence.  A good example of a group is a call center.  Typically in call centers, each attends the customer requests on their own and solves the problem. If one of the individual’s in the call center group is blocked with an issue, the rest are not affected. They could still continue working.

However, a team has shared the responsibility of delivery, and their work should be interdependent. In other words, team members have an agreed goal and the only way to achieve the goal is to work together. My thumb rule is, a user story cannot be moved to “Done” without the help of at least four other people :-)  

I believe that if your team room is very quiet you might want to check if there is anything wrong there. You should ask why teams are not talking to each other ? do they have shared responsibility of work ?  do they have a common goal to achieve or individual goals?  How many people do you need to complete an user story?

Photo courtesy: http://www.teams-forsuccess.com/working-groups-and-teams-are-they-the-same/

Categories: Blogs

Nexus Reifies

Learn more about our Scrum and Agile training sessions on WorldMindware.com

Nexus Scrum

I had the privilege of attending Scrum.org‘s 2-day seminar on Scaled Professional Scrum. The Nexusa connection or series of connections linking two or more things (direct translation from Latin a binding together), is the recommended scaling framework. The purpose of the Nexus is to manage dependencies between 3-9 Scrum Teams towards “reification”, to make an abstract idea real or concrete. This is ensured mostly through a single Product Owner, single Product Backlog, integrated (Nexus) Sprint Planning, Review and Retrospective and the addition of a Nexus Integration Team whose membership is made up mostly of Scrum team members internal to the Nexus, but often also includes other support personnel. The structure is very similar to LeSS, but perhaps even less prescriptive and is certainly much less prescriptive than SAFe. This is probably my favourite thing about the Nexus – the fact that it has just enough structure to be a model for scaling Scrum, but is light and flexible enough to accommodate all of the nuances that “just depend” on your situation. Like the other two above-mentioned scaling models, it places emphasis on the need for strong technical practices, continuous integration and the synchronization of events to facilitate integration. There is flexibility around synchronization, in that if the Nexus Sprint is 4 weeks in duration and teams within the Nexus want to do 2 or even 1 week Sprints, the model accommodates – as long as all of the teams’ work is combined into a fully integrated (reified) increment of potentially shippable product by the end of the Nexus Sprint.

 

Try out our Virtual Scrum Coach with the Scrum Team Assessment tool - just $500 for a team to get targeted advice and great how-to informationPlease share!
facebooktwittergoogle_plusredditpinterestlinkedinmail

The post Nexus Reifies appeared first on Agile Advice.

Categories: Blogs