Skip to content

Blogs

R: dplyr – Group by field dynamically (‘regroup’ is deprecated / no applicable method for ‘as.lazy’ applied to an object of class “list” )

Mark Needham - Sun, 11/09/2014 - 00:29

A few months ago I wrote a blog explaining how to dynamically/programatically group a data frame by a field using dplyr but that approach has been deprecated in the latest version.

To recap, the original function looked like this:

library(dplyr)
 
groupBy = function(df, field) {
  df %.% regroup(list(field)) %.% summarise(n = n())
}

And if we execute that with a sample data frame we’ll see the following:

> data = data.frame(
      letter = sample(LETTERS, 50000, replace = TRUE),
      number = sample (1:10, 50000, replace = TRUE)
  )
 
> groupBy(data, 'letter') %>% head(5)
Source: local data frame [5 x 2]
 
  letter    n
1      A 1951
2      B 1903
3      C 1954
4      D 1923
5      E 1886
Warning messages:
1: %.% is deprecated. Please use %>% 
2: %.% is deprecated. Please use %>% 
3: 'regroup' is deprecated.
Use 'group_by_' instead.
See help("Deprecated")

I replaced each of the deprecated operators and ended up with this function:

groupBy = function(df, field) {
  df %>% group_by_(list(field)) %>% summarise(n = n())
}

Now if we run that:

> groupBy(data, 'letter') %>% head(5)
Error in UseMethod("as.lazy") : 
  no applicable method for 'as.lazy' applied to an object of class "list"

It turns out the ‘group_by_’ function doesn’t want to receive a list of fields so let’s remove the call to list:

groupBy = function(df, field) {
  df %>% group_by_(field) %>% summarise(n = n())
}

And now if we run that:

> groupBy(data, 'letter') %>% head(5)
Source: local data frame [5 x 2]
 
  letter    n
1      A 1951
2      B 1903
3      C 1954
4      D 1923
5      E 1886

Good times! We get the correct result and no deprecation messages.

If we want to group by multiple fields we can just pass in the field names like so:

groupBy = function(df, field1, field2) {
  df %>% group_by_(field1, field2) %>% summarise(n = n())
}
> groupBy(data, 'letter', 'number') %>% head(5)
Source: local data frame [5 x 3]
Groups: letter
 
  letter number   n
1      A      1 200
2      A      2 218
3      A      3 205
4      A      4 176
5      A      5 203

Or with this simpler version:

groupBy = function(df, ...) {
  df %>% group_by_(...) %>% summarise(n = n())
}
> groupBy(data, 'letter', 'number') %>% head(5)
Source: local data frame [5 x 3]
Groups: letter
 
  letter number   n
1      A      1 200
2      A      2 218
3      A      3 205
4      A      4 176
5      A      5 203

I realised that we can actually just use the group_by itself and pass in the field names without quotes, something I couldn’t get to work in earlier versions:

groupBy = function(df, ...) {
  df %>% group_by(...) %>% summarise(n = n())
}
> groupBy(data, letter, number) %>% head(5)
Source: local data frame [5 x 3]
Groups: letter
 
  letter number   n
1      A      1 200
2      A      2 218
3      A      3 205
4      A      4 176
5      A      5 203

We could even get a bit of pipelining going on if we fancied it:

> data %>% groupBy(letter, number) %>% head(5)
Source: local data frame [5 x 3]
Groups: letter
 
  letter number   n
1      A      1 200
2      A      2 218
3      A      3 205
4      A      4 176
5      A      5 203

And as of dplyr 0.3 we can simplify our groupBy function to make use of the new count function which combines group_by and summarise:

groupBy = function(df, ...) {
  df %>% count(...)
}
> data %>% groupBy(letter, number) %>% head(5)
Source: local data frame [5 x 3]
Groups: letter
 
  letter number   n
1      A      1 200
2      A      2 218
3      A      3 205
4      A      4 176
5      A      5 203
Categories: Blogs

Airbnb and Uber – From Batch to One Piece Flow

Evolving Excellence - Sat, 11/08/2014 - 21:42

Some of the more interesting internet-driven companies these days are the likes of Airbnb and Uber. They call themselves part of the “sharing economy.”

But let’s take a look at the word “share.” From the MacMillan dictionary, share is “to allow someone to have something you own.” Is that what’s happening here? Not really. It’s not free.

There is a small unit of demand in the form of a single room or a single car ride, and there’s a small unit of availability in the form of a room in a house, an apartment, an empty cab, or even someone with a car going in the same direction. The are matched, a value is being transferred, and there’s also a financial transaction representative of that value. That’s really micro supply and demand management.

Previously such small units of supply and demand were never taken advantage of, let alone optimized, unless aggregated. Now internet connectivity and computational power can dynamically and efficiently track, match, and transact such small units.

The results are rather astounding.

Since its founding, in 2008, Airbnb has spearheaded growth of the sharing economy by allowing thousands of people around the world to rent their homes or spare rooms. Yet while as many as 425,000 people now stay in Airbnb-listed homes on a peak night, the company’s growth is shadowed by laws that clash with its ethos of allowing anyone, including renters, to sell access to their spaces.

Over 400,000 rooms – on a single night. That’s the equivalent of over 3,000 average size hotels. Empty space that was being wasted, now going to good use, eliminating the need to build 3,000 hotels. Think about the positive impact on the environment, urban sprawl, energy use, and so forth.

Similarly, in March 2014, seven months ago and a lifetime in the timescale of a hypergrowth company, Uber was providing 1.1 million rides per week. In this case it is only partially displacing the required capacity of the old business model, taxis, as many taxi drivers are switching to the Uber platform. Still, think about the impact of that optimized micro capacity and demand utilization on the required supply of taxis – and hence steel, plastics, and gas.

I happen to be a big fan of Uber, and use their service almost every time I travel. The speed, convenience, and ease of transaction creates significant value.  Yes, their business practices may make me hold my nose a bit.

Other companies are looking at similar concepts. Amazon is looking at using micro units of delivery capacity in the form of taxis and Uber competitor Flywheel to provide same day – and perhaps same hour – delivery.

Very large numbers of previously wasted supply units being matched with demand in a very efficient manner. The batch unit of a large delivery truck, a bus, or a hotel is being broken down into units approaching one.  Obviously any change like this doesn't come easily, and cities - often dependent on bed taxes - are pushing back on Airbnb while traditional taxi services are pushing back on Uber.  But value is being created in an efficient and popular manner, therefore change will occur.

Where have I heard about that concept before?

Categories: Blogs

Here Be Dragons – Scaling Agile

ScrumSense.com - Peter Hundermark - Sat, 11/08/2014 - 08:38

I gave a talk at the global Scrum Gathering in Berlin and two weeks later this revised (and improved?) version at the regional Scrum Gathering in Cape Town.

Scaling agility in any organisation requires attention to both cultural and structural dimensions. The culture aspect rests in providing modern leadership and change management. I provide only a passing reference here to this dimension. My presentation here is focussed on the structural and process dimension.

I describe 3 “laws” of scaling. These are observations or empirical laws. Then I set out 10 patterns that I have found helpful over nearly a decade of doing this with many organisations as a consultant and coach.

The post Here Be Dragons – Scaling Agile appeared first on ScrumSense.

Categories: Blogs

Dual-Speed IT Drives Digital Business Transformation and Improves IT-Business Relationships

J.D. Meier's Blog - Fri, 11/07/2014 - 19:24

Don’t try to turn all of your traditional IT into a digital unit.  

You’ll break both, or do neither well.

Instead,  add a Digital Unit.   Meanwhile, continue to simplify and optimize your traditional IT, but, at the same time, add a Digital Unit that’s optimized to operate in a Cloud-First, Mobile-First world.

This is the Dual-Speed IT approach, and, with this way, you can choose the right approach for the job and get the best of both worlds.

Some projects involve more extensive planning because they are higher-risk and have more dependencies.

Other projects benefit from a loose learning-by-doing method, with rapid feedback loops, customer impact, and testing new business waters.

And, over time, you can shift the mix.

In the book, Leading Digital: Turning Technology into Business Transformation, George Westerman, Didier Bonnet, and Andrew McAfee, share some of their lessons learned from companies that are Digital Masters that created their digital visions and are driving business change.

Build Digital Skills Into One of Your Existing Business Units

You can grow one of your existing business units into a Digital Unit.  For example, marketing is a pretty good bet, given the customer focus and the business impact.

Via Leading Digital:

 

“Changing the IT-business relationship is well worth the effort, but doing so takes time.  Your company may not have the time to wait before starting your digital transformation.  Rather than improving the IT unit, some companies try to build digital skills into another unit, such as marketing.  They try to work around IT rather than with it.”

Don’t Mix Your Digital Unit with Your Traditional IT

Don’t throw away your existing IT or break it by turning it into something it’s not, too quickly.   Instead, leverage it for the projects where it makes sense, while also leveraging your new Digital IT unit.

Via Leading Digital:

“Although building digital skills is useful, trying to work around IT can be fraught with challenges, especially if people do not understand the reasons for IT's systematic, if sometimes ponderous, processes.  This kind of flanking action can waste money, make the digital platform more complex, and even worse, open the company to security and regulatory risks.”

Create a Dual-Speed IT to Support Both Traditional IT and Faster-Speed Digital Transformation

You can have the best of both worlds, while both evolving your traditional IT and growing your Digital Unit to thrive at Cloud speed.

Via Leading Digital:

“A better approach is to create a dual-speed IT structure, where one part of the IT unit continues to support traditional IT needs, while another takes on the challenge of operating at digital speed with the business.  Digital activities--especially in customer engagement--move faster than many traditional IT ones.  They look at design processes differently.  Where IT projects have traditionally depended on clear designs and well-structured project plans, digital activities often engage in test-and-learn strategies, trying features in real-life experiments and quickly adding or dropping them based on what they find.”

Optimize the Digital Unit for Digital World

Your Digital Unit needs to be very different from traditional IT in terms of the mindset and the approaches around the people, processes, and technology.

Via Leading Digital:

“In a dual-speed approach, the digital unit can develop processes and methods at clock-speeds more closely aligned with the digital world, without losing sight of the reasons that the old IT processes existed.  IT leaders can draw on informal relationships within the IT department to get access to legacy systems or make other changes happen.  Business leaders can use their networks to get input and resources.  Business and IT leaders can even start to work together in the kind of two-in-a-box leadership method that LBG and other companies have adopted.”

Choose the Right Leadership Both in the Business and in IT

To make it work and to make it work well, it takes partnerships on both sides.   The business and IT both need skin in the game.

Via Leading Digital:

“Building dual-speed IT units requires choosing the right leadership on both sides of the relationship.  Business executives need to be comfortable with technology and with being challenged by their IT counterparts.  IT leaders need to have a mind-set that extends beyond technology to encompass the processes and drivers of business performance.  Leaders from both sides need to be strong communicators who can slide easily between conversations with their business- or IT-focused people.”

Great IT Leaders Know When to Choose Traditional IT vs. the Digital Unit

With both options at your disposal, Great IT Leaders know how to choose the right approach for the job.   Some programs and projects will take a more traditional life-cycle or require heavier planning or more extensive governance and risk management, while other projects can be driven in a more lightweight and agile way.

Via Leading Digital:

“Dual-speed IT also requires perspective about the value of speed.  Not all digital efforts need the kind of fast-moving, constantly changing processes that digital customer-engagement processes can need.  In fact, the underlying technology elements that powered LBG's new platform, Asian Paints' operational excellence, and Nike's digital supply chain enhancements required the careful, systematic thinking that underpins traditional IT practices.  Doing these big implementations in a loose learning-by-doing method could be dangerous.  It could increase rework, waste money, and introduce security risks.  But once the strong digital platform is there, building new digital capabilities can be fast, agile, and innovative.  The key is to understand what you need in each type of project and how much room any project has to be flexible and agile.  Great IT leaders know how to do this.  If teamed with the right business leaders, they can make progress quickly and safely.”

Dual-Speed IT Requires New Processes within IT

It takes a shift in processes to do Dual-Speed IT.

Via Leading Digital:

“Dual-speed IT also takes new processes inside IT.  Few digital businesses have the luxury to wait for monthly software release cycles for all of their applications.  Digital-image hosting business Flickr, for example, aims for up to ten deployments per day, while some businesses require even more.  This continuous-deployment approach requires very tight discipline and collaboration between development, test, and operations people.  A bug in software, missed step in testing, or configuration problem in deployment can bring down a web site or affect thousands of customers.”

DevOps Makes Dual Speed IT Possible

DevOps blends development and operations into a more integrated approach that simplifies and streamlines processes to shorten cycle times and speed up fixes and feedback loops.

Via Leading Digital:

“A relatively new software-development method called DevOps aims to make this kind of disciplined speed possible.  It breaks down silos between development, operations, and quality assurance groups, allowing them to collaborate more closely and be more agile.  When done properly, DevOps improves the speed and reliability of application development and deployment by standardizing development environments.  It uses strong methods and standards, including synchronizing the tools used by each group.”

DevOps Can Help IT Release Software Better, Faster, Cheaper, and More Reliably

DevOps is the name of the game when it comes to shipping better, faster, cheaper and more reliably in a Cloud-First, Mobile-First world.

Via Leading Digital:

“DevOps relies heavily on automated tools to do tasks in testing, configuration control, and deployment—tasks that are both slow and error-prone when done manually.  Companies that use DevOps need to foster a culture where different IT groups can work together and where workers accept the rules and methods that make the process effective.  The discipline, tools, and strong processes of DevOps can help IT release software more rapidly and with fewer errors, as well as monitor performance and resolve process issues more effectively, than before.”

Driving Digital Transformation Takes a Strong Link Between Business and IT Executives

In order for your Digital Transformation to thrive, it takes building better bridges between the business leaders and the IT leaders.

Via Leading Digital:

“Whether your CIO takes it upon himself or herself to improve the IT-business relationship, or you decide to help make it happen, forging a strong link between business and IT executives is an essential part of driving digital transformation.  Strong IT-business relationships can transform the way IT works and the way the business works with it.  Through trust and shared understanding, your technology and business experts can collaborate closely, like at LBG, to innovate your business at digital speeds.  Without this kind of relationship, your company may become mired in endless requirements discussion, filing projects, and lackluster systems, while your competitors accelerate past you in the digital fast lane.”

If you want to thrive in the new digital economy while driving digital business transformation without breaking your existing business, consider adding Dual-Speed IT to your strategies and shift the mix from traditional IT to your Digital Unit over time.

You Might Also Like

10 High-Value Activities in the Enterprise

Cloud Changes the Game from Deployment to Adoption

Drive Business Transformation by Reenvisioning Operations

Drive Business Transformation by Reenvisioning Your Customer Experience

How To Improve the IT-Business Relationship

Management Innovation is at the Top of the Innovation Stack

Think in a Series of Sprints, Not Marathons

Categories: Blogs

NeuroAgile Quick Links #6

Notes from a Tool User - Mark Levison - Fri, 11/07/2014 - 17:52
NeuroAgileInteresting reports from the world of Science that can be applied (or not) to Agile Teams

 

 

Categories: Blogs

Retrospective Exercise: Vital Few Actions

Ben Linders - Fri, 11/07/2014 - 15:03
The aim of an agile retrospective is to define actions for the next iteration that will improve the way of working and help teams to deliver more value to their customers. This retrospective exercise can be used within agile frameworks like Scrum, SAFe, XP or Kanban to have teams agree upon the vital few improvement actions that they will do. Continue reading →
Categories: Blogs

The Agile Reader – Weekend Edition: 11/07/2014

Scrumology.com - Kane Mar - Fri, 11/07/2014 - 05:59

You can get the Weekend Edition delivered directly to you via email by signing up http://eepurl.com/0ifWn.

The Weekend Edition is a list of some interesting links found on the web to catch up with over the weekend. It is generated automatically, so I can’t vouch for any particular link but I’ve found the results are generally interesting and useful.

  • Agile Data Scientist, A Disciplined Hiker or Reckless Hunter? #agile #scrum http://t.co/TuCFkpqwAA
  • Leading Scrum Experts, Braintrust Consulting Group, Return to Memphis to Host … – IT Business Net #Agile #Scrum
  • RT @yochum: The Guide on the Side #agile #scrum
  • RT @MasterScrum: Agile Data Scientist, A Disciplined Hiker or Reckless Hunter? #agile #scrum http://t.co/TuCFkpqwAA
  • RT @MasterScrum: Agile Data Scientist, A Disciplined Hiker or Reckless Hunter? #agile #scrum http://t.co/TuCFkpqwAA
  • 8 Ways To Overcome Resistance To An #Agile Process Rollout via @jonathanlevene #scrum
  • RT @MasterScrum: Agile Data Scientist, A Disciplined Hiker or Reckless Hunter? #agile #scrum http://t.co/TuCFkpqwAA
  • Are you #Agile? Seriously who puts a scrum shirt on a baby #geekout
  • Planning, Tracking & Managing Agile Web Development Sprints w/ Scrum & Intervals by @intervals http://t.co/nhBGUjXO3R
  • RT @iranfleitas: Infographic about “The Scrum framework in 30 seconds” #agile #scrum @ScrumAlliance http://t.co/1Ypa…
  • Less than 20 tickets left for the 6th annual GIVE THANKS FOR SCRUM event 11/25 in #Boston: #scrum #agile #innovation
  • #Scrum was born in #Boston. And that is why we GIVE THANKS FOR SCRUM every year here: #agile #lean #collaboration
  • RT @DanielMezick: #Scrum was born in #Boston. And that is why we GIVE THANKS FOR SCRUM every year here: #agile #lean…
  • RT @MasterScrum: Agile Data Scientist, A Disciplined Hiker or Reckless Hunter? #agile #scrum http://t.co/TuCFkpqwAA
  • Lookin for a #ProductOwner in #WilmingtonDE that has #agile and #scrum experience check out for info #career #job
  • The dangers inherent when Key Performance Indicators (KPIs) are used as a target to drive behavior #Scrum #Agile
  • Xebia Blog: Mutation Testing: How Good are your Unit Tests? #agile #scrum
  • SolutionsIQ: Automating Application Development, SAFe, and Other Takeaways from AgilePalooza Seattle #agile #scrum
  • RT @yochum: SolutionsIQ: Automating Application Development, SAFe, and Other Takeaways from AgilePalooza Seattle #ag…
  • RT @yochum: Xebia Blog: Mutation Testing: How Good are your Unit Tests? #agile #scrum
  • RT @yochum: SolutionsIQ: Automating Application Development, SAFe, and Other Takeaways from AgilePalooza Seattle #ag…
  • RT @yochum: Xebia Blog: Mutation Testing: How Good are your Unit Tests? #agile #scrum
  • RT @yochum: SolutionsIQ: Automating Application Development, SAFe, and Other Takeaways from AgilePalooza Seattle #ag…
  • Confirmado @dbassi no #AgileTourBH 2014 #Agile #PMOT #AgileBrazil #AgileBR #ExtremAgile #SCRUM http://t.co/QHqbq0MCY1
  • Scrum Expert: User Stories for Agile Requirements #agile #scrum
  • RT @AgileTourBH: Confirmado @dbassi no #AgileTourBH 2014 #Agile #PMOT #AgileBrazil #AgileBR #ExtremAgile #SCRUM http…
  • Studie: Agile Methoden im Höhenflug – #Scrum #Kanban #DesignThinking via @heiseonline
  • FIRST LEGO League Team Sponsored By Scrum Alliance In Virginia – PR Newswire (press release) #Agile #Scrum
  • RT @scrum_coach: #Agility In All Things #scrumterms #agile #mentalagility #physicalagility #strategicagility
  • RT @yochum: Xebia Blog: Mutation Testing: How Good are your Unit Tests? #agile #scrum
  • RT @MasterScrum: Agile Data Scientist, A Disciplined Hiker or Reckless Hunter? #agile #scrum http://t.co/TuCFkpqwAA
  • RT @iqberatung: Studie: Agile Methoden im Höhenflug – #Scrum #Kanban #DesignThinking via @heiseonline
  • RT @yochum: On Software Development, Agile, Startups, and Social Networking – Isaac Sacolick: Agile Data Sci… #agi…
  • Uzility now prompts you on your team’s activity, so you can track progress even easier. Check it out #agile #scrum
  • Agile by McKnight, Scrum by Day is out! Stories via @BLupano @AgileUniversity
  • Continuing the mission… and continually improving #Scrum #Agile
  • RT @MasterScrum: Agile Data Scientist, A Disciplined Hiker or Reckless Hunter? #agile #scrum http://t.co/TuCFkpqwAA
  • RT @MasterScrum: What Does QA do on the First Day of a Sprint? #agile #scrum http://t.co/180ciIpkkt
  • RT @MasterScrum: Agile Data Scientist, A Disciplined Hiker or Reckless Hunter? #agile #scrum http://t.co/TuCFkpqwAA
  • RT @MasterScrum: Agile Data Scientist, A Disciplined Hiker or Reckless Hunter? #agile #scrum http://t.co/TuCFkpqwAA
  • Categories: Blogs

    People over Process

    Agilitrix - Michael Sahota - Fri, 11/07/2014 - 04:44

    Here are the slides from my Keynote at Lean Into Agile Conference:


    Overview

    Agile and Lean are a means to an end. Once we are clear what our goals are and our approach is consistent with what we truly value, then we may hope for success.

    When we simplify the Agile Manifesto’s “Individuals and Interactions over Processes and Tools” we get “People over Process”. Agile is about people. It’s about a people-first culture. Lean is simliar.

    Sadly, many organizations are mired in organizational debt: mistrust, politics and fear. Changing the process won’t fix this. We need to go to the root of it – to find a way to talk about and shift to a healthier culture: one that values people.

    The VAST (Vulnerability, Authentic Connection, Safety and Trust) model makes the dynamics of human systems visible and clarifies where we may apply leverage to foster lasting change.

    We outline a fundamentally different approach for organizational change: one where valuing people is integral to building lasting success.

    Why we need to pay attention Agile Mindset vs Practices

    The post People over Process appeared first on Catalyst - Agile & Culture.

    Related posts:

    1. Letting Go of Agile (Culture) “If you want something very, very badly, let it go...
    2. The Business Case for an Authentic Workplace People are messy: they have personalities and emotions. In this...
    3. WholeHearted Manifesto: We Value People The WholeHearted Manifesto consists on one value statement: We Value...

    Related posts brought to you by Yet Another Related Posts Plugin.

    Categories: Blogs

    R: Joining multiple data frames

    Mark Needham - Fri, 11/07/2014 - 03:29

    I’ve been looking through the code from Martin Eastwood’s excellent talk ‘Predicting Football Using R‘ and was intrigued by the code which reshaped the data into that expected by glm.

    The original looks like this:

    df <- read.csv('http://www.football-data.co.uk/mmz4281/1314/E0.csv')
     
    # munge data into format compatible with glm function
    df <- apply(df, 1, function(row){
      data.frame(team=c(row['HomeTeam'], row['AwayTeam']),
                 opponent=c(row['AwayTeam'], row['HomeTeam']),
                 goals=c(row['FTHG'], row['FTAG']),
                 home=c(1, 0))
    })
    df <- do.call(rbind, df)

    The initial data frame looks like this:

    > library(dplyr)
    > df %>% select(HomeTeam, AwayTeam, FTHG, FTAG) %>% head(1)
      HomeTeam    AwayTeam FTHG FTAG
    1  Arsenal Aston Villa    1    3

    And we want to get it to look like this:

    > head(df, 2)
                    team    opponent goals home
    HomeTeam     Arsenal Aston Villa     1    1
    AwayTeam Aston Villa     Arsenal     3    0

    So for each row in the initial data frame we want to have two rows: one representing each team, how many goals they scored in the match and whether they were playing at home or away.

    I really like dplyr’s pipelining function so I thought I’d try and translate Martin’s code to use that and other dplyr functions.

    I ended up with the following two sets of function calls:

    df %>% select(team = HomeTeam, opponent = AwayTeam, goals = FTHG) %>% mutate(home = 1)
    df %>% select(team = AwayTeam, opponent = HomeTeam, goals = FTAG) %>% mutate(home = 0)

    I’m doing pretty much the same thing as Martin except I’ve used dplyr’s select and mutate functions to transform the data frame.

    The next step was to join those two data frames together and with Nicole’s help I realised that there are many ways we can do this.

    The functions that will do the job are:

    We decided to benchmark them to see which was able to transform the data frame the fastest:

    # load data into data.frame
    dfOrig <- read.csv('http://www.football-data.co.uk/mmz4281/1314/E0.csv')
     
    original = function(df) {
      df <- apply(df, 1, function(row){
        data.frame(team=c(row['HomeTeam'], row['AwayTeam']),
                   opponent=c(row['AwayTeam'], row['HomeTeam']),
                   goals=c(row['FTHG'], row['FTAG']),
                   home=c(1, 0))
      })
      do.call(rbind, df)
    }
     
    newRBind = function(df) {
      rbind(df %>% select(team = HomeTeam, opponent = AwayTeam, goals = FTHG) %>% mutate(home = 1),
            df %>% select(team = AwayTeam, opponent = HomeTeam, goals = FTAG) %>% mutate(home = 0))  
    }
     
    newUnion = function(df) {
      union(df %>% select(team = HomeTeam, opponent = AwayTeam, goals = FTHG) %>% mutate(home = 1),
            df %>% select(team = AwayTeam, opponent = HomeTeam, goals = FTAG) %>% mutate(home = 0))  
    }
     
    newJoin = function(df) {
      join(df %>% select(team = HomeTeam, opponent = AwayTeam, goals = FTHG) %>% mutate(home = 1),
           df %>% select(team = AwayTeam, opponent = HomeTeam, goals = FTAG) %>% mutate(home = 0),
          type = "full")  
    }
     
    newMerge = function(df) {
      merge(df %>% select(team = HomeTeam, opponent = AwayTeam, goals = FTHG) %>% mutate(home = 1),
           df %>% select(team = AwayTeam, opponent = HomeTeam, goals = FTAG) %>% mutate(home = 0),
           all = TRUE)  
    }
    > library(microbenchmark)
     
    > microbenchmark(original(dfOrig))
    Unit: milliseconds
                 expr   min    lq  mean median    uq max neval
     original(dfOrig) 189.4 196.8 202.5    201 205.5 284   100
     
    > microbenchmark(newRBind(dfOrig))
    Unit: milliseconds
                 expr   min    lq  mean median    uq   max neval
     newRBind(dfOrig) 2.197 2.274 2.396  2.309 2.377 4.526   100
     
    > microbenchmark(newUnion(dfOrig))
    Unit: milliseconds
                 expr   min    lq  mean median   uq   max neval
     newUnion(dfOrig) 2.156 2.223 2.377  2.264 2.34 4.597   100
     
    > microbenchmark(newJoin(dfOrig))
     
    Unit: milliseconds
                expr   min    lq  mean median   uq   max neval
     newJoin(dfOrig) 5.961 6.132 6.817  6.253 6.65 11.95   100
     
    > microbenchmark(newMerge(dfOrig))
    Unit: milliseconds
                 expr   min    lq  mean median    uq   max neval
     newMerge(dfOrig) 7.121 7.413 8.037  7.541 7.934 13.32   100

    We actually get a 100 time speed up over the original function if we use rbind or union whereas with merge or join it’s around 35 times quicker.

    In this case using merge or join is a bit misleading because we’re not actually connecting the data frames together based on any particular field – we are just appending one to the other.

    The code’s available as a gist if you want to have a play.

    Categories: Blogs

    Continuing our journey together to focus on YOUR One Shiny Object…

    Mike Vizdos - Implementing Scrum - Thu, 11/06/2014 - 21:25
    www.implementingscrum.com -- Cartoon -- May 6, 2008

    Hi.

    It’s been a while [again] since posting anything new at www.implementingscrum.com.

    I see people are sharing the information (all around the world) in cubicles and Scrum team rooms.  People are linking comic strips from their blogs, using them in presentations, and including them in books.

    The Chicken and Pig cartoons here are now firmly established as folklore in the Scrum and Agile communities today.

    People either love it or hate the story; however, an amazing thing is happening — people still talk about them AND they are still used to help start tough conversations in software development.

    Cool.

    And yet I ask myself, why am I not posting *regularly* here anymore?

    The answer is pretty simple.  I’ve moved on and evolved my interests over the years.  The comic strips are obviously not my area of *passion* anymore; they don’t get me out of bed excited to take on the world daily anymore.

    And.

    That. Is.  OK.

    There is still a boat load of information out here to share, consume, and learn with one another.

    So.

    Heading into 2015 (and beyond), I’ll still be maintaining this site and will continue to push out regular reminder postings with the old comic strips (and possibly some new information as a topic is relevant).  If you want to help me on this, please let me know (the ball is in your court on this one, and YES I am serious with this invitation).

    Where else am I headed into 2015?

    A concept I am calling, “One Shiny Object.”

    I’ve been on working with people (all around the world) for many years and have found that my passion today is helping people figure out their own concept of a, “One Shiny Object.”

    It’s about keeping things simple.  Removing complexity.

    Very similar to the concepts we live in the Scrum and Agile world.  It’s beyond that now (and probably has always been… I’ll be using the tools in that box heading out of just the software development world).

    Out on twitter (@mvizdos there) you can catch me doing this (almost daily) now.  At conferences, this is something I am passionately talking to audiences about.

    With clients, it’s all about this:

    Focus.  #deliver

    Interested in following me (Mike Vizdos) on this new journey?

    Please jump over to www.OneShinyObject.com and let me know your name and e-mail address; from there, we’ll branch off from the topic of this blog about Implementing Scrum and then you’ll see what we can do together to focus on your One Shiny Object.

    Amazing things are already happening.  Join me (still FREE!).

    You’ll still be getting information here about Implementing Scrum, and I reiterate my humble request that you to get involved here to keep things relevant in your world of Scrum today.

    It’s all a journey!

     

    Categories: Blogs

    Advanced Topics in Agile Planning

    TV Agile - Thu, 11/06/2014 - 18:27
    Velocity is perhaps the most useful metric available to agile teams. In this session we will look at advanced uses of velocity for planning under special but common circumstances. We will see how to forecast velocity in the complete absence of any historical data. We will look at how a new team can forecast velocity […]
    Categories: Blogs

    How To Improve the IT-Business Relationship

    J.D. Meier's Blog - Thu, 11/06/2014 - 17:38

    It’s possible to change IT from a poorly respected cost center to a high-functioning business partner.

    Driving business transformation is a people, process, and technology thing.

    Some people think they can change their business without IT.   The challenge is that technology is the enabler of significant business change in today’s digital landscape.  Cloud, Mobile, Social, and Big Data all bring significant capabilities to the table, and IT can hold the keys.

    But the business doesn’t want to hear that.

    Business Leaders Want to Hear About the WHY and WHAT of the Business

    Business leaders don’t want to hear about the HOW of technology.

    Business leaders want to hear about the impact on their business.   They want to hear about how predictive analytics can help them build a better pipeline, or target more relevant offers.   Business leaders want to hear about how they can create cross-sell/upsell opportunities in real-time.   And, business leaders want to hear about the business benefits and the KPI that will be impacted by choosing a particular strategy.

    The reality is that the new Digital Masters of the emerging Digital Economy bring their IT with them, and in many cases, their IT even helps lead the business into the new Digital Frontier.

    In the book, Leading Digital: Turning Technology into Business Transformation, George Westerman, Didier Bonnet, and Andrew McAfee, share some of their lessons learned from companies that are digital masters that created their digital visions and are driving business change.

    How IT Can Change Its Game

    While it takes work on both sides, IT can change it’s game by creating transparency around performance, roles, and value.  This includes helping employees think and talk differently about what they do.   IT can show very clearly how it delivers value for the money.  And IT can change the way IT and business leaders make investment decisions and assess the returns.

    IT Needs to Speak Business

    The CIO and everybody in IT, needs to speak the language of business.

    Via Leading Digital:

    “Poor relations between IT and business leaders can have many causes.  Sometimes it's the personality of the IT leader.  A common complaint among senior executives is that their CIO seems to speak a different language from the business.  Another is that the CIO doesn't seem to understand what's really important.  For example, a chemical company CIO we interviewed described how he communicates regularly with business executives about the innovative possibilities of digital technologies.  Yet none of his business executive peers (whom we interviewed separately) seemed to find the discussions credible.”

    IT Needs to Deliver Better, Faster, and More Reliably than Outsourcing

    It’s a competitive world and IT needs to continuously find ways to deliver solutions in a way that makes business sense.

    Via Leading Digital:

    “Sometimes the issue arises from IT's delivery capability.  According to Bud Mathaisel, who has served as CIO in several large companies, 'It starts with competence in delivering services reliably, economically, and at very high quality.  It is the absolute essential to be even invited into meaningful dialog about how you then build on that competence to do something very interesting with it.'  Unfortunately, some IT units today do not have this competence.  One business executive we interviewed said, 'IT is a mess.  It's costs are not acceptable.  It proposes things in nine or ten months, where external firms could do them in three to nine weeks.  We started offshoring our IT, and now our IT guys are trying to change.' A legacy of poor communication, byzantine decision processes, and broken commitments is no foundation on which to build a strong IT-business relationship.”

    IT Needs a Good Digital Platform to Be High-Performing IT

    In order to bet on IT, it needs to be high-performing.  And in order for IT to be high-performing, it needs to have a good digital platform.

    Via Leading Digital:

    “However, the fault doesn't always rest only with IT leaders.  In many cases, business executive share some of the blame ... high-performing IT requires a good digital platform, and good platforms require discipline.  If your approach to working with IT can be characterized by impatience, unreasonable expectations, or insisting on doing things your way, then you'll need to think about how to change your side of the relationship.”

    CIOs Can Lead Digital Business Transformation

    Key business transformation takes technology.  CIOs can help lead the business transformation, whether it's through shared goals with the business, creating a new governance mechanism, or creating a new shared digital unit.

    Via Leading Digital:

    “Regardless of the case, if your IT-business relationships are poor, it's essential to fix the problem.  A bank executive stated, 'IT has been brought closer to business during the last five years.  It is very important to success because man of the important transformations in our business are enabled by technology.'  With strong relationships, IT executives can help business executives meet their goals, and business executives listen when IT people suggest innovations.  Executives on both sides are willing to be flexible in creating new governance mechanisms or shared digital units.  At Codelco, Asian Paints, and P&G, the CIO even leads digital transformation for the company.”

    CIOs Can Help Drive the Bus with the Executive Team

    CIOs can help drive the bus, but it takes more than senior sponsorship.

    Via Leading Digital:

    “So, how can you start to improve your IT-business relationship?  Angela Ahrendts, CEO of Burberry, told her CIO he needed to help drive the bus with the executive team.  However, leadership changes or top-down mandates are only the start of the change.  Few CIOs can change the business by themselves, and not all business executives will climb on the bus with the CIO, even if the CEO demands it.”

    Fix How You Communicate to Fix the IT-Business Relationship

    Start by fixing how you communicate between the business and IT.

    Via Leading Digital:

    “Fixing the IT-business relationship can take time, as people learn how to trust each other and redefine the way they work together.  As with any struggling relationship, the best starting point is to fix the way you communicate.  Does IT really cost too much, or are costs reasonable, given what IT has to do? Is the IT unit really too bureaucratic, or do all of those procedures actually serve a useful purpose?  Are you a good partner to IT or a difficult one?  How can IT make it easier for you to get what you need, while still making sure things are done correctly?  What investments can help IT improve its technology, internal processes, cost-effectiveness, quality, or speed?”

    Change IT from a Poorly Respected Cost Center to a High-Functioning Business Partner

    It’s possible to change IT from a low performing cost center to a high-performing business partner.  Companies do it all the time, and MIT has the research.

    Via Leading Digital:

    “MIT research into IT turnarounds has identified a series of steps that can change IT from a poorly respected cost center to a high-functioning business partner.  The key change mechanism is transparency--around performance, roles, and value.  The first step is to help IT employees think, and talk, differently about what they do.  The second step proceeds to showing very clearly how well (or how poorly) IT delivers value for money--the right services at the right quality and right price, and where problems still exist.  And then the third step moves to changing the way IT and business leaders make investment decisions and assess the returns that projects deliver.  Through transparency around roles, performance, and investments, both sides can make smoother decisions and work together to identify and deliver innovation.”

    If you’re part of a business that wants to change the world, start by reimagining how IT can help you achieve the art of the possible.

    You Might Also Like

    Building Better Business Cases for Digital Initiatives

    Drive Business Transformation by Reenvisioning Your Customer Experience

    Drive Business Transformation by Reenvisioning Your Operations

    How Digital is Changing Physical Experiences

    The Future of IT Leaders

    Categories: Blogs

    Replacing Backlog Grooming

    Leading Agile - Mike Cottmeyer - Thu, 11/06/2014 - 17:37
    The Problem

    Over the last few years, I’ve worked with numerous teams. One thing they all struggle with is backlog grooming.  They all know they need to do it.  Unfortunately, they all seem to struggle with when to do it or who should do it.  The most interesting struggle with backlog grooming happened two years ago.  The “story time” meetings took place at the beginning of a month-long sprint. The manager stated, the work to be completed and delivered during that sprint had to be refined within the same sprint.  This helped explain why the team thought they needed month-long sprints.  When I asked why they would try to refine work the first two weeks of the sprint and then complete that work the second two weeks, you know what their answer was?  “It said to do it like that in the Scrum Guide!” After I clarified their misunderstanding, we established a cadence to continuously mature the backlog.  A few select people would participate in the scheduled meetings.  We would reserve capacity from each sprint to get that work ready for future sprints.  The team was able to shorten their sprints to 2 weeks.  They more than doubled their delivery rate without increasing defect rates.  With that as an example, over the last few years, I have evolved my practice of backlog grooming.  Let’s look at some key dates in the evolution of backlog grooming.

    Evolution of Backlog Grooming

    2005: “grooming the product backlog” is mentioned by Mike Cohn on what is now the Scrum Users Yahoo Group;

    I always have teams allot some amount of time to “grooming the product backlog” to make sure it’s ready for the next sprint.

    2008: A formal description of “backlog grooming” is given by Kane Mar, under the name Story Time, and recommending it as a regular meeting

    I call these meetings “Story Time” meetings….Although they are not a formal part of Scrum, I’ve found that Story Time greatly improves project planning and reduces confrontational planning meetings, which are all too common for many teams.  A Story Time meeting should be held at the same time and location every single week and involve the entire team, including the Product Owner and ScrumMaster. The sole intention of these weekly meetings is to work through the backlog in preparation for future work.

    2011: The practice of “backlog grooming” is called “backlog refinement” and promoted to an “official” element of Scrum with its inclusion in the Scrum Guide

    Product Backlog refinement is the act of adding detail, estimates, and order to items in the Product Backlog. This is an ongoing process in which the Product Owner and the Development Team collaborate on the details of Product Backlog items. During Product Backlog refinement, items are reviewed and revised. The Scrum Team decides how and when refinement is done. Refinement usually consumes no more than 10% of the capacity of the Development Team. However, Product Backlog items can be updated at any time by the Product Owner or at the Product Owner’s discretion.

    2014: Derek Huether from LeadingAgile evolved the practice of backlog grooming with one of his clients, to allow the practice to work better at scale, calling it a “Progression” workshop.

    When operating at scale, my client deals with different problems than a standard Scrum team.  They’re dealing with separate lines of business. They’re dealing with multiple delivery teams for each line of business, to include external vendors. They’re dealing with a portfolio roadmap that has annual plan items and budgets. Our strategy encapsulated the entire product delivery value stream, while ensuring we had enough architectural runway. We progressed work to be consumed by delivery teams, via a series of workshops.

    Progression Workshops

    Our progression workshops differ slightly from the story time meeting detailed by Kane Mar and the refinement meeting mentioned in the Scrum Guide.  Counter to Story Time, we don’t invite the entire team. Instead, we have elevated the workshop to a group some have come to know as a Product Owner (PO) team.  The group of people within the PO team will vary, depending on the line of business.  Yes, there will be a Product Owner (Product Lead) and facilitator, but from there we’ll include the development lead, testing lead, and an architect.  We’ve found two key challenges when operating at scale. First, is there a well defined backlog that is ready enough to be consumed by different delivery teams?  Second, is the work being queued up to the delivery teams free and clear of other teams.  That is, have we decomposed it in such a way that we’ve minimized dependencies on other teams? Beyond that, in order to achieve some degree of architectural runway, we continually refactor existing platforms. Architectural changes are not only made incrementally but we require an architect to be present at every progression workshop.

    Counter to the Scrum Guide, I’m not going to be proscriptive as to how much capacity the PO Teams do/should commit to progression workshops.  The goal is to have enough work ready for delivery teams to consume for a few sprints.

    When progressing work, we do expect some artifacts to be generated, to contribute to the teams understanding of what will be developed, tested, and delivered. Below is a partial list of potential artifacts. To be clear, we do not expect all of these to be generated.

    Potential Deliverables
    • System Context Diagram
    • Dependency and Risk Work Items
    • System Architecture Guidance Acknowledgement
    • Use Case Diagram and document
    • Business Process Flow
    • Known Business Rules
    • High Level Technology Alignment
    • Architecture Backlog for Planned Work
    • As-Is Data Contracts
    • Feature Work Items Assigned to Delivery Team
    • Feature Business Value and Acceptance Criteria
    • Feature Stack Rank
    • Test Strategy

    So, that’s the high-level view of the Progression Workshop.  Most of the time, a feature will require two or more progression workshops before work is ready to be consumed by a delivery team.  Once features progress to a defined level of shared understanding, the delivery teams assist in the decomposition of features to user stories.  In this way, work is decomposed to the right level of detail for each delivery team.

    I’m curious, how have you scaled feature development and backlog grooming in your organizations? What mechanics outside of the standard Scrum process have you found useful to refine work to be completed by delivery teams?  Have you evolved Story Time or backlog refinement?

    The post Replacing Backlog Grooming appeared first on LeadingAgile.

    Categories: Blogs

    [POSTPONED] Portland, Oregon: Agile Smack-Down (Debate)

    James Shore - Thu, 11/06/2014 - 10:02
    06 Nov 2014 James Shore/Calendar

    This event is postponed due to inclement weather. I will update the listing when a new date is chosen.

    The Technology Association of Oregon (TAO) is hosting a debate about Agile between me and Frank D'Andrea next week. Adam Light is hosting. It's going to be a fun, lively event with plenty of thought-provoking moments. Here's the blurb:

    Join us for the TAO Agile Smack-Down, a thought-provoking discussion and debate of the good, bad, and ugly ways to implement Agile methodologies. Adam Light will moderate a rousing dialog between Frank D'Andrea, VP Software Development at Tater Tot Designs, and James Shore, author of The Art of Agile Development.

    Our panel will discuss real-world lessons from applying and misapplying Agile to application development, with time set aside for Q&A from the audience.

    The event is 5:30-8:30 on Thursday, November 13th at Urban Airship in downtown Portland. It's $25 for TAO members and $45 for non-members. Registration and details here.

    Categories: Blogs

    Reddit "Ask Me Anything"

    James Shore - Thu, 11/06/2014 - 10:01
    06 Nov 2014 James Shore/In-the-News

    The Agile subreddit on Reddit invited me to do an "Ask Me Anything" (AMA) earlier this year. We had a great turnout and discussed many interesting topics.

    Read it here.

    Categories: Blogs

    trust, but verify …

    Derick Bailey - new ThoughtStream - Wed, 11/05/2014 - 17:52

    My dad was recently involved in a very large project, working as the project manager. It was a construction project with a handful of contracting companies that he had worked with in the past. One of the crews, though, decided they wanted to get lazy on the job and did about an hour worth of work in about 5 hours, one day.

    surveilance-coding

    The real issue was that this was the first day on the job for this crew – there was no history of good work on this job site, and immediately obvious reason why it was taking so long. So my dad called their manager and talked to him about what was happening – very friendly, offering the manager opportunity to explain, and wanting to find a reason why this was ok for the first day.

    A few days later, my dad was walking with one of his project leads and passed through the area where this team was working. One of the crew members looked up and said “coming to check up on us?” with a grin on his face. My dad stood silent for a moment, while his project lead said “why? do I need to?” The construction worker immediately became aware of himself and tried to back out of the conversation, awkwardly saying “well, no. Of course not!”

    That crew never had any issues afterward – they did great work, got it done on time, and built a professional relationship with my dad and the other project leads.

    Trust The Team

    I have to admit, I would not have been quite so astute in my response if I were the project lead. I probably would have blurted out, “Yes.” with a very serious tone of voice, had I been asked if I were checking up on the crew. But that’s not the kind of thing that I really want to do, and I certainly wouldn’t want a project manager to do that to me. It creates an air of distrust, as if you were spying on people with cameras and accusing them of not working, constantly.

    No one wants to work in an environment like that. We need freedome, flexibility, autonomy and independence to do our jobs – the same as the construction crew needed.

    Now, this doesn’t mean that my dad never checked the work they did. There are a lot of regulations that have to be met, a lot of plans that need to be accounted for, and a lot of safety issues that need to be addressed when doing commercial construction. It was absolutely necessary to verify the work, just as it is necessary to verify the software we build before shipping to customers. That doesn’t mean there is no trust. It only means that we recognize human fallibility and the need to verify work, in spite of the trust that we place in people.

    Trust, But Verify

    There’s a saying that I learned a long time ago from some ex-military friends and co-workers: trust, but verify. The idea is simple. You default to trusting someone for whatever it is… work or otherwise. But that doesn’t mean you have to be blind and wreckless with your trust. It’s ok to verify the information provided. In fact, it’s usually a good idea. No one is perfect. Mistakes are made. It happens.

    But when mistakes are made, we only need to educate and correct the mistake. If we trust the person that made the mistake, we can help them learn and they will improve.

    Trust vs Assistance

    It’s also important to understand the difference between blind trust and assistance.

    I once assigned a team member some work that I knew was just beyond their current ability. I did this to help them stretch and grow. But I made a mistake and blindly let them go at it for nearly 6 weeks. Sure, I asked for updates on progress but I never once offered to do a design or code review, and I left it up to them to ask questions and seek help. The results were … a mistake. I squarely blamed myself in the retrospective for the work he did. I knew I should have spent more time offering assistance, and helping him to grow in to the design skill that I wanted him to reach.

    My desire to trust, blindly, created a lost opportunity for someone to grow faster and ended up costing a lot of time and money for the project.

    Trust, Assist, and always Verify

    There are two extremes, here: absolute trust, and never trusting. A great leader will understand how far to lean toward either end of this spectrum, in nearly any situation. That doesn’t mean mistakes won’t be made, but that they will be caught early and corrected.

    Default to a high level of trust, in your team and your coworkers. Assist when someone is having a hard time or is given a task that they must grow in to. And always verify the work they do, to ensure it will perform as needed.

    – Derick

    Categories: Blogs

    Rally Software Announces Enhanced Support for SAFe 3.0

    Agile Product Owner - Wed, 11/05/2014 - 03:30

    Rally Software, Scaled Agile’s  first gold partner, recently announced enhanced support for SAFe V 3.0 in both their product and service offerings:

    See https://www.rallydev.com/community/agile/expanding-rally-support-safe-30

    I’m particularly happy to see the new release Planning Board, which helps agile programs create realistic release plans that consider feature size, cost of delay, and development capacity. These are the primary elements you need to take into account to help assure that your demand matches your capacity, and that you are working on the right priorities. (Unless both are true, many bad things are about to happen.)

    Thanks Rally!

     

     

    Categories: Blogs

    Certified Scrum Product Owner (CSPO) in Edmonton—March 25-26

    Notes from a Tool User - Mark Levison - Tue, 11/04/2014 - 20:51
    Agile Pain Relief presents a two-day Certified Scrum Product Owner (CSPO) workshop in Edmonton—March 25-26 taught by certified ScrumMaster Trainer Mark Levison.
    Categories: Blogs

    Announcement: Certified Scaled Agile Framework Program Consultant (SPC) – Mishkin Berteig and Travis Birch

    Learn more about our Scrum and Agile training sessions on WorldMindware.com

    I’m happy and proud to announce that both myself and Travis Birch have passed our SAFe SPC test after our training in early October.  This means that Berteig Consulting will start offering a number of SAFe related training sessions both publicly (on World Mindware) and as in-house.  We are also experienced in large scale applications of Agile and SAFe is a great framework to have in our toolkit along with our Real Agility Program.  If you are considering using Agile at scale or if you are having trouble with making Agile at scale work effectively, I hope you will contact us.

    Try out our Virtual Scrum Coach with the Scrum Team Assessment tool - just $500 for a team to get targeted advice and great how-to informationPlease share!
    facebooktwittergoogle_plusredditpinterestlinkedinmail
    Categories: Blogs