Skip to content

Feed aggregator

Luigi: Defining dynamic requirements (on output files)

Mark Needham - Tue, 03/28/2017 - 07:39

In my last blog post I showed how to convert a JSON document containing meetup groups into a CSV file using Luigi, the Python library for building data pipelines. As well as creating that CSV file I wanted to go back to the meetup.com API and download all the members of those groups.

This was a rough flow of what i wanted to do:

  • Take JSON document containing all groups
  • Parse that document and for each group:
    • Call the /members endpoint
    • Save each one of those files as a JSON file
  • Iterate over all those JSON files and create a members CSV file

In the previous post we created the GroupsToJSON task which calls the /groups endpoint on the meetup API and creates the file /tmp/groups.json.

Our new task has that as its initial requirement:

class MembersToCSV(luigi.Task):
    key = luigi.Parameter()
    lat = luigi.Parameter()
    lon = luigi.Parameter()

    def requires(self):
        yield GroupsToJSON(self.key, self.lat, self.lon)

But we also want to create a requirement on a task that will make those calls to the /members endpoint and store the result in a JSON file.

One of the patterns that Luigi imposes on us is that each task should only create one file so actually we have a requirement on a collection of tasks rather than just one. It took me a little while to get my head around that!

We don’t know the parameters of those tasks at compile time – we can only calculate them by parsing the JSON file produced by GroupsToJSON.

In Luigi terminology what we want to create is a dynamic requirement. A dynamic requirement is defined inside the run method of a task and can rely on the output of any tasks specified in the requires method, which is exactly what we need.

This code does the delegating part of the job:

class MembersToCSV(luigi.Task):
    key = luigi.Parameter()
    lat = luigi.Parameter()
    lon = luigi.Parameter()


    def run(self):
        outputs = []
        for input in self.input():
            with input.open('r') as group_file:
                groups_json = json.load(group_file)
                groups = [str(group['id']) for group in groups_json]


                for group_id in groups:
                    members = MembersToJSON(group_id, self.key)
                    outputs.append(members.output().path)
                    yield members


    def requires(self):
        yield GroupsToJSON(self.key, self.lat, self.lon)

Inside our run method we iterate over the output of GroupsToJSON (which is our input) and we yield to another task as well as collecting its outputs in the array outputs that we’ll use later.
MembersToJSON looks like this:

class MembersToJSON(luigi.Task):
    group_id = luigi.IntParameter()
    key = luigi.Parameter()


    def run(self):
        results = []
        uri = "https://api.meetup.com/2/members?&group_id={0}&key={1}".format(self.group_id, self.key)
        while True:
            if uri is None:
                break
            r = requests.get(uri)
            response = r.json()
            for result in response["results"]:
                results.append(result)
            uri = response["meta"]["next"] if response["meta"]["next"] else None


        with self.output().open("w") as output:
            json.dump(results, output)

    def output(self):
        return luigi.LocalTarget("/tmp/members/{0}.json".format(self.group_id))

This task generates one file per group containing a list of all the members of that group.

We can now go back to MembersToCSV and convert those JSON files into a single CSV file:

class MembersToCSV(luigi.Task):
    out_path = "/tmp/members.csv"
    key = luigi.Parameter()
    lat = luigi.Parameter()
    lon = luigi.Parameter()


    def run(self):
        outputs = []
        for input in self.input():
            with input.open('r') as group_file:
                groups_json = json.load(group_file)
                groups = [str(group['id']) for group in groups_json]


                for group_id in groups:
                    members = MembersToJSON(group_id, self.key)
                    outputs.append(members.output().path)
                    yield members

        with self.output().open("w") as output:
            writer = csv.writer(output, delimiter=",")
            writer.writerow(["id", "name", "joined", "topics", "groupId"])

            for path in outputs:
                group_id = path.split("/")[-1].replace(".json", "")
                with open(path) as json_data:
                    d = json.load(json_data)
                    for member in d:
                        topic_ids = ";".join([str(topic["id"]) for topic in member["topics"]])
                        if "name" in member:
                            writer.writerow([member["id"], member["name"], member["joined"], topic_ids, group_id])

    def output(self):
        return luigi.LocalTarget(self.out_path)

    def requires(self):
        yield GroupsToJSON(self.key, self.lat, self.lon)

We then just need to add our new task as a requirement of the wrapper task:

And we’re ready to roll:

$ PYTHONPATH="." luigi --module blog --local-scheduler Meetup --workers 3

We’ve defined the number of workers here as we can execute those calls to the /members endpoint in parallel and there are ~ 600 calls to make.

All the code from both blog posts is available as a gist if you want to play around with it.

Any questions/advice let me know in the comments or I’m @markhneedham on twitter.

The post Luigi: Defining dynamic requirements (on output files) appeared first on Mark Needham.

Categories: Blogs

Spark the Change – Montreal

Agile Ottawa - Mon, 03/27/2017 - 22:38
Spark the Change brings together change leaders from all industries and walks of life, with one mission only: to work together to create positive and lasting change. If you have a vision for change within your organization then this event … Continue reading →
Categories: Communities

AgilePath Podcast Up

Johanna Rothman - Mon, 03/27/2017 - 18:42

I’ve said before that agile is a cultural change, not merely a project management framework or approach. One of the big changes is around transparency and safety.

We need safety to experiment. We need safety to be transparent. Creating that safe environment can be difficult for everyone involved.

John LeDrew has started a new podcast, agilepath.fm. I had the pleasure of chatting with John for the podcast. He wove a story with several other interviewers and it’s now up, In search of Safety.

I hope you enjoy it.

Categories: Blogs

Article 10 in SAFe Implementation Roadmap series: Launch More ARTs and Value Streams

Agile Product Owner - Mon, 03/27/2017 - 18:00
Click to enlarge

When you’ve done all the hard work—planned, prepared, trained, launched the first ART, and then put in the effort to make it even better—there’s this moment in a SAFe rollout when the early results are coming in and you start to fully realize the potential that has been unleashed for the organization. It’s an exciting time, as enthusiasm from the first ART is making its way into other parts of the organization, and more people are getting on board with the new way of working.

To put that work into the context of the SAFe Implementation Roadmap, that means you’re ready for the tenth ‘critical move’ in this series, Launch More ARTs and Value Streams. You can think of this stage as opening up the throttle on your car. The larger business opportunity has arrived, enabling the enterprise to ‘consolidate gains and produce more change’  by launching more Agile Release Trains (ARTs) and Value Streams. This allows the business to start realizing the fuller benefits of SAFe, effectively shifting to the next ‘higher gear’ of the transformation.

In this article, we discuss:

  • Launch More ARTs
  • Implement Value Streams Roles, Artifacts and Events
  • Launch More Value Streams
  • The SAFe Implementation Railway
  • Managing Impediments
  • Moving Forward

Take note, even if you’re a field-seasoned SAFe veteran, and think you’ve seen it all, this article is well worth reading as it introduces a great new BVIR for managing a transformation rollout: the SAFe Implementation Railway. It’s an idea generously shared by our friends at Northwestern Mutual. They manage one of the larger SAFe rollouts to date, and use something called a “cow board” to manage ARTs in each state of development.

SAFe Implementation Railway BVIR, courtesy of Northwestern Mutual

If you’re wondering how cows and SAFe ended up together, read the full article here.

As always, we welcome your thoughts so if you’d like to provide some feedback on this new series of articles, you’re invited to leave your comments here.

Stay SAFe!
—Dean and the Framework team

Categories: Blogs

End to End Kanban for the Whole Organization

Scrum Expert - Mon, 03/27/2017 - 15:47
If shorter release cycle could be considered as a success for Agile software development teams, they might be considered as an issue if the other parts of the organization are not ready to handle...

[[ This is a content summary only. Visit my website for full links, other content, and more! ]]
Categories: Communities

New Book: The Agile Enterprise: Building and Running your Agile Enterprise

Imagine an enterprise where everyone focuses on the highest customer value.  Where strategy to tasks are visible so everyone knows if their work is aligned with the highest value work. Imagine an enterprise where a discovery mindset wins over certainty thinking. Where experimentation with increments and feedback help define the way toward customer value. Imagine a company where employees use 100% of their brain power to self organize around the work and be trusted to think of better ways to work. Where leaders encourage employees to put customer value first.  Imagine an enterprise where customers embrace the products and services being built because they are engaged in the building of the work all along the way. If you can imagine it, it can be yours!  In this unique and cutting edge Agile book, veteran Enterprise Agile Coach Mario Moreira, will guide you to build and run an Agile enterprise at every level and at every point from idea to delivery. Learn how Agile-mature organizations adapt nimbly to micro changes in market conditions and customer needs and continuously deliver optimal value to customers.  Learn cutting-edge practices and concepts as you extend your implementation of Agile pervasively and harmoniously through the whole enterprise for greater customer value and business success. Readers of The Agile Enterprisewill learn how to:
  • Establish a Customer Value Driven engine with an enterprise idea pipeline to process an enterprise’s portfolio of ideas more quickly and productively toward customer value and through all levels of the enterprise
  • Incorporate the Discovery Mindset; experimental, incremental, design, and divergent thinking; and fast feedback loops to increase the odds that what you build aligns more closely to what customer wants.
  • Leverage Lean Canvas, Personas, Story Mapping, Cost of Delay, Discovery Mindset, Servant leadership, Self-organization, and more to deliver optimum value to customers
  • Use continuous Agile Budgeting and enterprise idea pipeline at the senior levels of the enterprise to enable you to adapt to the speed of the market.
  • Reinvent Human Resources, Portfolio Management, Finance, and many areas of leadership toward new roles in the enablement of customer value. 
  • Establish a holistic view of the state of your Agile Galaxy from top-to-bottom and end-to-end allowing you to understand where you are today and where you’d like to go in your Agile future.
  • Be truly Agile throughout the enterprise, focusing on customer value and employees over all else.

This book is geared for: Sponsors of Agile Transformations; Executives and Senior Management; Agile Coaches, Consultants, and Champions; Portfolio Management; Project Management Offices (PMOs); Business and Finance; Human Resources (HR); Investors and Entrepreneurs; Scrum Masters, Agile Project Managers, and Product Owners.  This book concludes with an adventuring through an Agile Enterprise story that shows you how an enterprise may transform to Agile in an incremental manner with materials in this book.  Let the material in The Agile Enterprise help you achieve your successful customer value driven enterprise.

Thank you to the contributors JP Beaudry (on Story Mapping) and David Grabel (on Cost of Delay)!  A special Thank you to the Apress editing team!
Categories: Blogs

Dash off a Fiver to the ACLU

Agile Complexification Inverter - Mon, 03/27/2017 - 02:28
What can you do to save the world with an Amazon Dash Button?

Has a new era of enablement reached the hockey stick curve of exponential growth?  I think it has.  I've been picking up this vibe, and I may not be the first to sense things around me.  I've got some feedback that I very poor at it in the personal sphere.  However, on a larger scale, on an abstract level, in the field of tech phenomena I've got a bit of a streak going.  Mind you I'm not rich on a Zuckerberg level... and my general problem is actualizing the idea as apposed to just having the brilliant idea - or recognizing the opportunity.

A colleague told me I would like this tinker's Dash Button hack.  It uses the little hardware IoT button Amazon built to sell more laundry soap - a bit of imaginative thinking outside of the supply chain problem domain and a few hours of coding.  Repurposing the giant AWS Cloud Mainframe, that the Matrix Architect has designed to enslave you, to give the ACLU a Fiver ($5) every time you feel like one of the talking heads (#45) in Washington DC has infringed upon one of you civil liberties.


Now I think this is the power of a true IoT the fact that an enabling technology could allow the emergent property that was not conceived of in it's design.  No one has really tried to solve the problem of the democrat voice of the people.  We use the power of currency to proxy for so many concepts in our society, and it appears that the SCOTUS has accepted that currency and it's usage is a from of speech (although not free - do you see what I did there?).  What would the Architect of our Matrix learn if he/she/it could collect all the thoughts of people when they had a visceral reaction to an event correlate that reaction to the event, measure the power of the reaction over a vast sample of the population and feed that reaction into the decision making process via a stream of funding for or against a proposed policy.  Now real power of this feedback system will occur when the feedback message may mutate the proposal (the power of Yes/AND).

I can see this as enabling real trend toward democracy - and of course this disrupts the incumbent power structure of the representative government (federal republic).  Imagine a hack-a-thon where all the political organizations and the charities and the religions came together in a convention center.  There are tables and spaces and boxes upon boxes of Amazon Dashes Buttons.  We ask the organizations what they like about getting a Fiver every time the talking head mouths off, and what data they may also need to capture to make the value stream most effective in their unique organization.  And we build and test this into a eco-system on top of the AWS Cloud.
"You know, if one person, just one person does it they may think he's really sick and they won't take him."What would it take to set this up one weekend...  I've found that I'm not a leader.  I don't get a lot of followers when I have an idea... but I have found that I can make one heck of a good first-follower!

"And three people do it, three, can you imagine, three people walking in singin a bar of Alice's Restaurant and walking out. They may think it's an organization. And can you, can you imagine fifty people a day, I said fifty people a day walking in singin a bar of Alice's Restaurant and walking out. And friends they may thinks it's a movement."I will just through this out here and allow the reader to link up the possibilities.


Elmo From ‘Sesame Street’ Learns He's Fired Because Of Donald Trump’s Budget Cuts.  Would this be a good test case for a Dash Button mash up to donate to Sesame Workshop.

See Also:

GitHub Repo Donation Button by Nathan Pryor
Instructables Dash Button projects
Coder Turns Amazon Dash Button Into ACLU Donation Tool by Mary Emily O'Hara
Life With The Dash Button: Good Design For Amazon, Bad Design For Everyone Else by Mark WilsonHow to start a movement - Derek Sivers TED Talk
Categories: Blogs

Luigi: An ExternalProgramTask example – Converting JSON to CSV

Mark Needham - Sat, 03/25/2017 - 16:09

I’ve been playing around with the Python library Luigi which is used to build pipelines of batch jobs and I struggled to find an example of an ExternalProgramTask so this is my attempt at filling that void.

Luigi - the Python data library for building data science pipelines

I’m building a little data pipeline to get data from the meetup.com API and put it into CSV files that can be loaded into Neo4j using the LOAD CSV command.

The first task I created calls the /groups endpoint and saves the result into a JSON file:

import luigi
import requests
import json
from collections import Counter

class GroupsToJSON(luigi.Task):
    key = luigi.Parameter()
    lat = luigi.Parameter()
    lon = luigi.Parameter()

    def run(self):
        seed_topic = "nosql"
        uri = "https://api.meetup.com/2/groups?&topic={0}&lat={1}&lon={2}&key={3}".format(seed_topic, self.lat, self.lon, self.key)

        r = requests.get(uri)
        all_topics = [topic["urlkey"]  for result in r.json()["results"] for topic in result["topics"]]
        c = Counter(all_topics)

        topics = [entry[0] for entry in c.most_common(10)]

        groups = {}
        for topic in topics:
            uri = "https://api.meetup.com/2/groups?&topic={0}&lat={1}&lon={2}&key={3}".format(topic, self.lat, self.lon, self.key)
            r = requests.get(uri)
            for group in r.json()["results"]:
                groups[group["id"]] = group

        with self.output().open('w') as groups_file:
            json.dump(list(groups.values()), groups_file, indent=4, sort_keys=True)

    def output(self):
        return luigi.LocalTarget("/tmp/groups.json")

We define a few parameters at the top of the class which will be passed in when this task is executed. The most interesting lines of the run function are the last couple where we write the JSON to a file. self.output() refers to the target defined in the output function which in this case is /tmp/groups.json.

Now we need to create a task to convert that JSON file into CSV format. The jq command line tool does this job well so we’ll use that. The following task does the job:

from luigi.contrib.external_program import ExternalProgramTask

class GroupsToCSV(luigi.contrib.external_program.ExternalProgramTask):
    file_path = "/tmp/groups.csv"
    key = luigi.Parameter()
    lat = luigi.Parameter()
    lon = luigi.Parameter()

    def program_args(self):
        return ["./groups.sh", self.input()[0].path, self.output().path]

    def output(self):
        return luigi.LocalTarget(self.file_path)

    def requires(self):
        yield GroupsToJSON(self.key, self.lat, self.lon)

groups.sh

#!/bin/bash

in=${1}
out=${2}

echo "id,name,urlname,link,rating,created,description,organiserName,organiserMemberId" > ${out}
jq -r '.[] | [.id, .name, .urlname, .link, .rating, .created, .description, .organizer.name, .organizer.member_id] | @csv' ${in} >> ${out}

I wanted to call jq directly from the Python code but I couldn’t figure out how to do it so putting that code in a shell script is my workaround.

The last piece of the puzzle is a wrapper task that launches the others:

import os

class Meetup(luigi.WrapperTask):
    def run(self):
        print("Running Meetup")

    def requires(self):
        key = os.environ['MEETUP_API_KEY']
        lat = os.getenv('LAT', "51.5072")
        lon = os.getenv('LON', "0.1275")

        yield GroupsToCSV(key, lat, lon)

Now we’re ready to run the tasks:

$ PYTHONPATH="." luigi --module blog --local-scheduler Meetup
DEBUG: Checking if Meetup() is complete
DEBUG: Checking if GroupsToCSV(key=xxx, lat=51.5072, lon=0.1275) is complete
INFO: Informed scheduler that task   Meetup__99914b932b   has status   PENDING
DEBUG: Checking if GroupsToJSON(key=xxx, lat=51.5072, lon=0.1275) is complete
INFO: Informed scheduler that task   GroupsToCSV_xxx_51_5072_0_1275_e07372cebf   has status   PENDING
INFO: Informed scheduler that task   GroupsToJSON_xxx_51_5072_0_1275_e07372cebf   has status   PENDING
INFO: Done scheduling tasks
INFO: Running Worker with 1 processes
DEBUG: Asking scheduler for work...
DEBUG: Pending tasks: 3
INFO: [pid 4452] Worker Worker(salt=970508581, workers=1, host=Marks-MBP-4, username=markneedham, pid=4452) running   GroupsToJSON(key=xxx, lat=51.5072, lon=0.1275)
INFO: [pid 4452] Worker Worker(salt=970508581, workers=1, host=Marks-MBP-4, username=markneedham, pid=4452) done      GroupsToJSON(key=xxx, lat=51.5072, lon=0.1275)
DEBUG: 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   GroupsToJSON_xxx_51_5072_0_1275_e07372cebf   has status   DONE
DEBUG: Asking scheduler for work...
DEBUG: Pending tasks: 2
INFO: [pid 4452] Worker Worker(salt=970508581, workers=1, host=Marks-MBP-4, username=markneedham, pid=4452) running   GroupsToCSV(key=xxx, lat=51.5072, lon=0.1275)
INFO: Running command: ./groups.sh /tmp/groups.json /tmp/groups.csv
INFO: [pid 4452] Worker Worker(salt=970508581, workers=1, host=Marks-MBP-4, username=markneedham, pid=4452) done      GroupsToCSV(key=xxx, lat=51.5072, lon=0.1275)
DEBUG: 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   GroupsToCSV_xxx_51_5072_0_1275_e07372cebf   has status   DONE
DEBUG: Asking scheduler for work...
DEBUG: Pending tasks: 1
INFO: [pid 4452] Worker Worker(salt=970508581, workers=1, host=Marks-MBP-4, username=markneedham, pid=4452) running   Meetup()
Running Meetup
INFO: [pid 4452] Worker Worker(salt=970508581, workers=1, host=Marks-MBP-4, username=markneedham, pid=4452) done      Meetup()
DEBUG: 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   Meetup__99914b932b   has status   DONE
DEBUG: Asking scheduler for work...
DEBUG: Done
DEBUG: There are no more tasks to run at this time
INFO: Worker Worker(salt=970508581, workers=1, host=Marks-MBP-4, username=markneedham, pid=4452) was stopped. Shutting down Keep-Alive thread
INFO: 
===== Luigi Execution Summary =====

Scheduled 3 tasks of which:
* 3 ran successfully:
    - 1 GroupsToCSV(key=xxx, lat=51.5072, lon=0.1275)
    - 1 GroupsToJSON(key=xxx, lat=51.5072, lon=0.1275)
    - 1 Meetup()

This progress looks 🙂 because there were no failed tasks or missing external dependencies

===== Luigi Execution Summary =====

Looks good! Let’s quickly look at our CSV file:

$ head -n10 /tmp/groups.csv 
id,name,urlname,link,rating,created,description,organiserName,organiserMemberId
1114381,"London NoSQL, MySQL, Open Source Community","london-nosql-mysql","https://www.meetup.com/london-nosql-mysql/",4.28,1208505614000,"

Meet others in London interested in NoSQL, MySQL, and Open Source Databases.

","Sinead Lawless",185675230 1561841,"Enterprise Search London Meetup","es-london","https://www.meetup.com/es-london/",4.66,1259157419000,"

Enterprise Search London is a meetup for anyone interested in building search and discovery experiences — from intranet search and site search, to advanced discovery applications and beyond.

Disclaimer: This meetup is NOT about SEO or search engine marketing.

What people are saying:

  • ""Join this meetup if you have a passion for enterprise search and user experience that you would like to share with other able-minded practitioners."" — Vegard Sandvold
  • ""Full marks for vision and execution. Looking forward to the next Meetup."" — Martin White
  • “Consistently excellent” — Helen Lippell

Sweet! And what if we run it again?

$ PYTHONPATH="." luigi --module blog --local-scheduler Meetup
DEBUG: Checking if Meetup() is complete
INFO: Informed scheduler that task   Meetup__99914b932b   has status   DONE
INFO: Done scheduling tasks
INFO: Running Worker with 1 processes
DEBUG: Asking scheduler for work...
DEBUG: Done
DEBUG: There are no more tasks to run at this time
INFO: Worker Worker(salt=172768377, workers=1, host=Marks-MBP-4, username=markneedham, pid=4531) was stopped. Shutting down Keep-Alive thread
INFO: 
===== Luigi Execution Summary =====

Scheduled 1 tasks of which:
* 1 present dependencies were encountered:
    - 1 Meetup()

Did not run any tasks
This progress looks 🙂 because there were no failed tasks or missing external dependencies

===== Luigi Execution Summary =====

As expected nothing happens since our dependencies are already satisfied and we have our first Luigi pipeline up and running.

The post Luigi: An ExternalProgramTask example – Converting JSON to CSV appeared first on Mark Needham.

Categories: Blogs

New Case Study: Sony PlayStation Network Stays Top of its Game with SAFe

Agile Product Owner - Sat, 03/25/2017 - 15:33

“I personally believe we have delivered more in the two years we’ve been using SAFe than we did in the four years prior—not in raw code, but in value. Our downtime went down and that saved the company about 30 million over the course of the year. That’s real money and a really positive outcome.”

Tripp Meister, Director of Technology, PlayStation Network

PlayStation, made by Sony Interactive Entertainment (SIE), currently leads the gaming console market with more than 150 million users globally.

When millions of gamers eagerly await your product releases, every day matters. For PlayStation Network, delivering a quality product on time requires tight collaboration across more than 1,000 SIE engineering team members.

In meeting its targets, the SIE engineering organization found Waterfall and Agile Scrum fell short in bringing together hundreds of team members cohesively and in addressing the many dependencies across the organization. What’s more, disparate teams were able to plan only one or two iterations in advance.

In early 2014, SIE leadership turned to SAFe to bring greater organization and collaboration to development. The Framework allowed them to bring top-down prioritization, pull disparate groups together into common timeframes, and enabled the group to manage dependencies much better.

Two years into their implementation, SAFe has fundamentally changed the culture of the engineering organization:

  • Greater visibility/transparency – Developers have more insight into broader company initiatives and activities
  • Better coordination – Now, from Tokyo to San Diego, everyone speaks a common language when it comes to Agile
  • More predictability – In an environment with many dependencies, SAFe serves as a dependency management system, improving predictability
  • Clearer priorities – With weighted shortest job first (WSJF), SAFe brought a new approach to prioritizing

Since 2014, the company has launched six trains globally, shipped more than 350 production releases, and completed 22 PSIs (potentially shippable increments, now known as Program Increments), over 125 sprints and 250 features.

They also cut initial planning time by 28 percent. Instead of 1550 man-days to plan, it now takes 1125. With new efficiency, PlayStation Network puts a multi-million-dollar value on cost savings.

Check out the full case study, which is based on a public presentation by Tripp Meister of PlayStation Network. To watch in full, click here.

Many thanks to Tripp Meister, former Director of Technology, PlayStation Network, for sharing PlayStation Network’s SAFe story.

Stay SAFe!
—Dean

Categories: Blogs

Monitor Your Mesos Cluster with StackState

Xebia Blog - Fri, 03/24/2017 - 10:28

This post is part 2 in a 4-part series about Container Monitoring. Post 1 dives into some of the new challenges containers and microservices create and the information you should focus on. This article describes how to monitor your Mesos cluster. Apache Mesos is a distributed systems kernel at the heart of the Mesosphere DC/OS and is designed for […]

The post Monitor Your Mesos Cluster with StackState appeared first on Xebia Blog.

Categories: Companies

Book Review:: Agile Noir by Lancer Kind

Agile Complexification Inverter - Thu, 03/23/2017 - 23:40
First, allow me to layout some ground rules and a touch of the backstory...

I'm not a professional book reviewer, nor paid in anyway to read.  But if I could get that gig... I'd be a happy camper.  I've never written a book, but I've hacked out some code, a few articles, some of which might be considered book reviews.  I've worked in the Agile industry for more than a decade (but who's counting), and so - I may be a little close to the topic to have proper literary impartial bias.  In fact let me just go ahead and be explicit - I've done this, been there, got the t-shirt; I shit you not - this shit is for real!
Agile Noir by Lancer Kind
Now the ground rules...  I think this review will be written ... what's the word... while I'm reading, at the same time, without much delay in the reading - writing phases....in situ.... iteratively... oh I give up...

So don't be surprised - dear reader - if I just drop off in the middle...
                       ... maybe check back every week until I finish
Mar 22,
I've studied the cover... quite a nice graphic - to bad the whole novel isn't a graphic novel; oh - maybe it would be too bloody,  I could see Agile Noir becoming a Tarantino film.  As I sat looking at my book to-do stack... I skipped a few levels down the stack and pulled out Lancer Kind's 2016 Agile Noir.  I have read some of his previous comics titled Scrum Noir (vol 1, 2, 3).  So maybe I know what to expect - should be a fun romp in the fast lane with lots of inside the industry puns, innuendo and metaphors.

Well the damn dedication just reeks of an Agile Coach - Servant Leader (puke, barf.... moving on).

The High Cost of Schedule Slip
Now you may not find the situation Kartar finds himself in funny...  allow me to add some overtones of irony....  I'm going to go out on a racist limb and suggest that Kartar is an Indian.  That he is working in the heart of the Indian nation (Los Wages, NV), perhaps on a job for an Italian crime boss.  And none of these circumstances have anything to do with one of the world of science's biggest failures - Columbus's discover of the New World - which the thought was India, and named it's inhabitants there by creating the confusion we will have to deal with evermore.  Now Columbus was of course searching for a way to reduce the schedule required for shipping spices.

Kartar appears to be very emerged in planning and the art/science/pesdo-truth of planning and predicting the future of projects.  And he may be a master with the Gantt chart (which is footnoted on page 18).

This is all ringing just too true ... and I'm envisioning it in the style of a 1956 black and white film...

Kartar is the metaphor of his project... it seems that it's not quite on schedule... he's late to a just announced meeting with some superior and is driving at break neck speed on loose sand in the Vegas out skirts creating over bumps and ditches in his car with the accelerator pinned to the floor - because some people in a van might be trying to kill him.  Happens ALL - THE - TIME.





See Also:
Scrum Noir - several volumes of graphic novel about scrum masters and the projects they encounter - also by Lancer Kind
I will have a Double Expresso - Amazon review of Scrum Noir.
Categories: Blogs

7 Sins of Scrum and other Agile Antipatterns

Scrum Expert - Thu, 03/23/2017 - 22:01
This is about agile “anti-patterns”: “something that looks like a good idea, but which backfires badly when applied” (Coplien). The presenter has been around Agile development from before it was...

[[ This is a content summary only. Visit my website for full links, other content, and more! ]]
Categories: Communities

Kanban: The Key to Unlocking SAFe® (Scaled Agile Framework)

The inclusion of Kanban in the Scaled Agile Framework 4.0® enables orgs to improve business flow at every level of the business. See how LeanKit can help.

The post Kanban: The Key to Unlocking SAFe® (Scaled Agile Framework) appeared first on Blog | LeanKit.

Categories: Companies

Using Self-Selection for Happy Scrum Teams

TV Agile - Thu, 03/23/2017 - 19:38
Amber King is a fan of team self-selection, which, in her experience, yields happier, more empowered, more productive teams. Amber shares her story working at Opower where the company decided to do an experiment: the goal would be to do what’s best for the company, but individuals get to choose what they work on and […]
Categories: Blogs

Favro Will Spin-Off from Hansoft

Scrum Expert - Thu, 03/23/2017 - 18:35
Favro, a cloud-based planning and collaboration app for agile businesses, has announced plans to separate from Hansoft, a project management software firm. The separation of the two businesses will...

[[ This is a content summary only. Visit my website for full links, other content, and more! ]]
Categories: Communities

Dialogue on Prerequisites for Collaboration

Agile Complexification Inverter - Thu, 03/23/2017 - 15:08
IDEO-University 'From Ideas to Action' Lesson 1.

Join the dialogue on G+ Agile+ group.

Dialogue on Collaboration on Facebook (PDF)


Collaboration starts with who we are and our story - not the technology or the data
"The Future of Work Is Social Collaboration from Inside Out, where people connect around the why of work from who they really are as individuals in community.
They collaborate in generative conversations and co-create what’s next, i.e. their unique Contribution of value to society – what we might call Social Good.
They collaborate by taking the time to appreciate and align each other’s unique, hard wired, natural strengths, creating new levels of authentic and trusting relationships to take the Social Journey."Jeremy ScrivensDirector at The Emotional Economy at Work




What does dialogue mean... what does it contribute to collaboration?  Here's what the inventor of the internet Al Gore had to say about this:

Audie Cornish speaks with former Vice President Al Gore about the new edition of his book, The Assault On Reason.
Well, others have noted a free press is the immune system 
of representative democracy. And as I wrote 10 years ago, American democracy is in grave danger from the changes in the environment in which ideas either live and spread or wither and die. I think that the trends that I wrote about 10 years ago have continued and worsened, and the hoped-for remedies that can come from online discourse have been slow to mature. I remain optimistic that ultimately free speech and a free press where individuals have access to the dialogue will have a self-correcting quality. -- Al GoreExcerpt from NPR interview with Al Gore by Audie Cornish March 14, 2017. Heard on  All Things Considered.

See Also:
Mob Programming by Woody Zuill

[View the story "Dialogue on Prerequisites for Collaboration" on Storify]
Categories: Blogs

Cycle Time and Lead Time

Agile Complexification Inverter - Wed, 03/22/2017 - 22:57
Our organization is starting to talk about measuring Cycle Time and Lead Time on our software engineering stories.  It's just an observation, but few people seem to understand these measurement concepts, but everyone is talking about them.  This is a bad omen...  wish I could help illustrate these terms.  Because I doubt the measurements will be very accurate if the community doesn't understand when to start the clock, and just as important - when to stop it.

[For the nature of confusion around this terms compare and contrast these:  Agile Alliance Glossary; Six Sigma; KanbanTool.com; Lean Glossary.]

The team I'm working with had a toy basket ball goal over their Scrum board...  like many cheep toys the rim broke.  Someone bought a superior mini goal, it's a nice heavy quarter inch plastic board with a spring loaded rim - not a cheep toy.  The team used "Command Strips" to mount it but they didn't hold for long.

The team convinced me there was a correlation between their basketball points on the charts and the teams sprint burndown chart.  Not cause and effect, but correlation; have you ever stopped to think what that really means?  Could it mean that something in the environment beyond your ability to measure is an actual cause to the effect you desire?

I asked the head person at the site for advice, how could we get the goal mounted in our area?  He suggested that we didn't need permission, that the walls of the building were not national treasures - we should just mount it... maybe try some Command Strips.  Yes, great minds...  but what about getting fired after putting holes in the walls scares one from doing the right thing?  How hard is it to explain to the Texas Work Force Commission when they ask why you were fired?

The leader understood that if I asked the building facilities manager that I might get denied - but if he asked for a favor... it would get done.  That very day, Mike had the facilities manager looking at the board and the wall (a 15-20 minute conversation).  Are you starting the clock?  It's Dec 7th, lead time starts when Mike agreed to the team's request.

The team was excited, it looked like their desire was going to be granted.  Productive would flourish again.

Over the next few days I would see various people looking up at the wall and down at the basketball goal on the floor.  There were about 4 of these meetings each very short and not always the same people.  Team members would come up to me afterwards and ask...  "are we still getting the goal?"... "when are they going to bring a drill?"...  "what's taking so long?"

Running the calendar forward a bit... Today the facilities guy showed up with a ladder and drill.  It took about 20 minutes.  Basketball goal mounted (Dec 13th) - which clock did you stop?  All of the clocks stop when the customer (team) has their product (basketball goal) in production (a game commences).

I choose to think of lead time as the time it takes an agreed upon product or service order to be delivered.  In this example that starts when Mike, the dude, agreed to help the team get their goal mounted.

In this situation I want to think of cycle time as the time that people worked to produce the product (mounted goal) - other's might call this process time (see Lean Glossary).  And so I estimated the time that each meeting on the court looking at the unmounted goal took, plus the actual time to mount  the goal (100 minutes).  Technically cycle time is per unit of product - since in the software world we typically measure per story and each story is some what unique - it's not uncommon to drop the per unit aspect of cycle time.

Lead time:  Dec 13th minus Dec 7th = 5 work days
Cycle time:  hash marks //// (4)  one for each meeting at the board to discuss mounting techniques (assume 20 m. each); and about 20 minutes with ladder and drill;  total 100 minutes

Lead Time 5 days; Cycle Time 100 minutes
This lead to a conversation on the court - under the new goal with a few team members about what we could do with these measurements.  How if one's job was to go around and install basketball goals for every team in the building that a cycle time of 100 minutes with a lead time of 5 days might make the customers a bit unhappy.   Yet for a one off, unusual once a year sort of request that ratio of 100 minutes to 5 days was not such a bad response time.  The customer's were very happy in the end, although waiting for 5 days did make them a bit edgy.

But now what would happen if we measured our software development cycle time and lead time - would our (business) customers be happy?  Do we produce a once in a year product? (Well yes - we've yet to do a release.) Do our lead times have similar ratios to cycle time, with very little value add time (process time)?

Pondering...

Well it's January 5th and this example came up in a Scrum Master's Forum meeting.  After telling the tale we still did not agree on when to start and stop the two watches for Lead Time and Cycle Time.  Maybe this is much harder than I thought.  Turns out I'm in the minority of opinions - I'm doing it wrong!

Could you help me figure out why my view point is wrong?  Comment below, please.

LeanKit just published an article on this topic - it's very good but might also misinterpret cycle time.  I see no 'per unit' in their definition of cycle time.  The Lead Time and Cycle Time Debate: When Does the Clock Start? by Tommy Norman.

An Experiment in measuring the team's cycle time:
After a bit of time reflecting, debating, arguing with colleagues and other agilitst online I've decided to publish a little experiment in measuring cycle-time on a scrum team.  Here's the data... what does it say?  How do you think the team should react?  What action should be next?  What should the team's leadership feel/think/do?

The Story:  This team has been working together for a while.  The sprints are numbered from the start of the year... an interesting practice, this team uses 2 week sprints, is practicing Scrum.  Took a nice holiday and required some priming to get back in the swing of things after the first of the year (you see this in the trend of stories completed each sprint).  Cycle Time for a story on trend is longer than the sprint, this correlates with typical story "carry-over" (a story started is not finished in one sprint and is carried over to the next sprint).  Generally a story is finished in the sprint but not in sequence or priority - they all take at least the full sprint to get to done.  There is no correlation of story size to cycle time.

Now those are the facts more or less -- let us see what insights we might create from this cycle time info.  With no correlation of story size to cycle time AND little consistency of number of stories finished in a sprint (trend of # of stories: 1, 6, 7, 2, 2). The question arrises - what is the controlling variable that is not being measured that effects the time it takes to get from start to finish with a story?  Now that the team can see that the simplest things we could track do not have a strong effect on the length of time (or the through-put) a story requires... and that means the process is not under good control - we can start to look around for some of the uncontrolled (invisible factors) -- if we a courageous enough!

We reflected that many of the stories that carry over and are virtually unpredictable in size/time/effort appear to have large delays or multiple delays within their implementation phase.  So we devised a quick and dirty way to track this delay.  The assumption that this delay inherent in the work will perhaps be the unmeasured / uncontrolled variable that throws the correlation of story size with cycle-time out of kilter.

Our devised technique for tracking delay per story - a yellow dot on the task with a tick mark for every day the task is stuck in-process (delayed).





See Also:

LeanKit published this excellent explanation of their choices in calculating cycle time within their tool:  Kanban Calculations: How to Calculate Cycle Time by Daniel Vacanti.
LeanKit Lead Time Metrics: Why Weekends Matter
Elon Musk turns a tweet into reality in 6 days by Loic Le Meur
The ROI of Multiple Small Releaseshttps://en.wikipedia.org/wiki/Frank_Bunker_Gilbreth_Sr.
https://en.wikipedia.org/wiki/Cheaper_by_the_Dozen


The Hummingbird Effect: How Galileo Invented Timekeeping and Forever Changed Modern Life
by Maria Popova.  How the invisible hand of the clock powered the Industrial Revolution and sparked the Information Age.

Categories: Blogs

The New Lean: Beyond the Assembly Line

Enjoy this excerpt from the latest Lean Business Report. Download the full report here.
The new Lean...

The post The New Lean: Beyond the Assembly Line appeared first on Blog | LeanKit.

Categories: Companies

The New Lean: Beyond the Assembly Line

Enjoy this excerpt from the latest Lean Business Report. Download the full report here.
The new Lean...

The post The New Lean: Beyond the Assembly Line appeared first on Blog | LeanKit.

Categories: Companies

ProjectManagementParadise Podcast Up

Johanna Rothman - Wed, 03/22/2017 - 17:51

Johnny Beirne over at projectmanagementparadise podcast interviewed me, specifically about my post, Visualize Your Work So You Can Say No. See Episode 38: “How to visualise your work so that you can say no” with Johanna Rothman.

Hope you enjoy it!

Categories: Blogs

Knowledge Sharing


SpiraTeam is a agile application lifecycle management (ALM) system designed specifically for methodologies such as scrum, XP and Kanban.