Skip to content

Feed aggregator

Lean Leadership, Culture and Tools

Watch this webinar to learn how Lean leadership, culture and tools form the foundation of lasting change. About This Webinar Chris Hefley, CEO at LeanKit, explains how without three elements — leadership, culture and tools — our efforts at Lean transformation are little better than building castles in the sand. During this session, Chris discusses: – The […]

The post Lean Leadership, Culture and Tools appeared first on Blog | LeanKit.

Categories: Companies

Agile Metrics That Matter

Scrum Expert - Thu, 09/24/2015 - 16:04
This session is born of the hard lessons learned from years of working with diverse companies using metrics to improve their software development processes. Come learn from our successes and be warned by our failures. This session covers what makes an effective Agile metric, the pitfalls in designing metrics, and the basic principles for using metrics appropriately. We cover many general principles, but dive deeply in ...
Categories: Communities

Agile Metrics

TV Agile - Thu, 09/24/2015 - 15:57
This presentation discusses the good the bad and the ugly of Agile metrics: * What makes a good metric? * Why should you use metrics? * What are some examples of metrics? * Are there any majors to avoid? Video producer:
Categories: Blogs

Interactive mind map for meetings and retrospectives

TargetProcess - Edge of Chaos Blog - Thu, 09/24/2015 - 10:40

We wrote some time ago about our initiatives for visualizing data and project information for smarter and more effective decision making. Today we would like to share an online Ishikava (fishbone) diagram:

This is a kind of mind map that can help you better understand a problem by visualizing its causes. It can be also used in retrospectives and work meetings where a group of people can analyze different aspects of a process or a situation for a multi-perspective view of the problem and a well-thought-out solution.

We have made it interactive so that it is possible to track a brainstorming flow as a structured text list on the left and immediately get a visual diagram on the right. Also you can drill-down to a particular aspect by double-clicking it if you want to explore an aspect of a precedent separately.

The concept of the diagram was developed by Kaoru Ishikawa in 1968 to show the causes of a specific event ( We were also inspired by the “Rule of 5 why” (, an iterative interrogative technique used to understand the root cause of the problem or a defect.

Any feedback is welcome!

Categories: Companies

Agile Business Conference, London, UK, 6-7 October 2015

Scrum Expert - Thu, 09/24/2015 - 10:30
The Agile Business Conference is a two-day conference that takes place in London where meet and engage with the experts and thought leaders who have implemented focused, efficient Agile change programmes in a wide variety of industries. In the agenda of the Agile Business Conference you can find topics like “The Inexorable Rise of Agile”, “Monitor Agile Progress and Health – A Common Ground”, “How to ...
Categories: Communities

Agile Cambridge, Cambridge, UK, September 30-October 2 2015

Scrum Expert - Thu, 09/24/2015 - 10:12
Agile Cambridge is a three-day conference that is focused on Agile software development. The conference mixes workshops, experience reports and case studies. In the agenda of the Agile Cambridge conference you can find topics like “Becoming an agile leader, regardless of your role”, “Scaled portfolio experience report”, “Intro to Lean UX”, “The ABCD of database development: always be continuously delivering”, “Test-driving the Internet of Things”, “Agile ...
Categories: Communities

October 26-27th in Vancouver, Canada: Agile Fluency Presentation and Workshop

James Shore - Thu, 09/24/2015 - 09:29
23 Sep 2015 James Shore/Calendar

I'll be at Agile Vancouver's "Much Ado About Agile" conference in October!

I'm presenting two sessions on the Agile Fluency™ Model at the conference. On Monday the 26th, I have a presentation at 1:30:

Leading Best-Fit Agile

As teams grow in their understanding of Agile, their perspective of Agile shifts and changes, and so do the challenges they must overcome. Whether it's team cohesion, technical skills, or organizational politics, the right investment to make in your teams depends on where they're at, what they need, and what your organization is willing to provide.

Where are your teams along the Agile path? What investments do they need? We'll share the latest developments in the Agile Fluency model and what it tells us about leading Agile teams.

Then, on Tuesday the 27th, I'm presenting a full-day workshop elaborating on the topic:

Bringing Fluency to your Agile teams: Coaching for Best-Fit Agile

Learn how to tailor your coaching efforts to best fit your context. You'll learn how to evaluate your teams' proficiencies, organizational needs, and how to balance benefits and investments. Using the Agile Fluency Model, a proven model for understanding team capabilities and opportunities, you'll be able to articulate what benefits to expect from your teams, what opportunities exist, and what efforts will be needed in order to take advantage of those opportunities.

Learn more and sign up here.

Categories: Blogs

November 4-6th in Malmö, Sweden: Agile Engineering, Estimates (Presentations)

James Shore - Thu, 09/24/2015 - 09:23
23 Sep 2015 James Shore/Calendar

I'm going to be at Øredev in Malmö Sweden in November! I have two sessions I'm presenting at the conference, both which promise to be good ones. And of course, I'll be around all week, so be sure to look me up.

On Wednesday the 4th, I'm digging deep into advanced web development:

Agile Engineering for the Web

Test-driven development, refactoring, evolutionary design… these Agile engineering techniques have long been established for back-end code. But what about the front-end? For too many teams, it's dismissed with a "JavaScript sucks!" and unreliable, brittle code.

In this session, we look at what it takes to bring the best of Agile development to front-end code. Test-drive your JavaScript and refactor your CSS.

Then, on Friday the 6th, I'm discussing the always contentious subject of estimates:

Estimates or No Estimates?

There's a debate raging in the Agile world: should we estimate, or not? Lost in the noise are more important questions: When should we estimate, and why? When should we not estimate, and why not?

As with so many Agile questions, the answer to "should we estimate?" isn't a clear-cut "yes" or "no." Instead, the answer depends on what your team is capable of, what your organization needs, and how to best balance the two.

We'll take a deep look at how estimates work, why they work, and when and why to discard them. Along the way, we'll encounter surprising insights about the nature of Agile development and the teams that use it.

Learn more and register here.

Categories: Blogs

February 17th in Portland, Oregon: Agile Engineering (Presentation)

James Shore - Thu, 09/24/2015 - 09:23
23 Sep 2015 James Shore/Calendar

I'm going to reprise November's Øredev talk on Agile engineering at AgilePDX's evening session in February. If you couldn't make it to Sweden, this will be the next best thing:

Agile Engineering for the Web

Test-driven development, refactoring, evolutionary design… these Agile engineering techniques have long been established for back-end code. But what about the front-end? For too many teams, it's dismissed with a "JavaScript sucks!" and unreliable, brittle code.

In this session, we look at what it takes to bring the best of Agile development to front-end code. Test-drive your JavaScript and refactor your CSS.

AgilePDX is the Portland Agile user group. Learn more about AgilePDX here.

Categories: Blogs

How To Get The Difference Between Two Arrays, As A Distinct List, With JavaScript ES6

Derick Bailey - new ThoughtStream - Wed, 09/23/2015 - 13:30

I recently found myself in a situation where I had a non-distinct list of things that were invalid (it contained duplicates), which came from a larger list of things. What I needed was a distinct list of things that were valid. But I only had the full list and the list of invalid things.

Array filtering

My thought process took me down the pat of saying “I wish I had a Set and a simple way to get the difference between two lists”. A short search later, I was looking at Dr Axel Rauschmayer’s Exploring ES6 book (highly recommended reading!), with my answer right there on the page!

ES6 Sets

If you’re not familiar with the idea of a set, it’s a math concept that says there is a list of distinct things. There can be no duplicates in a set. Integers are a good example of a set: 1, 2, 3, 4, 5

In this example, there is a list of integers that count from 1 to 5 with no duplication. Each item is distinct. You won’t find “1” in the list from 1 to 5 more than once.

This is a set – and I wanted these semantics in a JavaScript list of things. Fortunately, ES6 provides a Set object for just that purpose.

Array-Like, Distinct List Of Items

Think of a Set as an array-like (iterable) object, but with the semantics and behavior of ensuring a distinct list of items. In other words, it is not possible to have duplicates. 

There isn’t a “set literal” syntax, like there is for an array (the [square, bracket, syntax]) but the API for a set is fairly simple. 

In this example, I’m creating a set and adding numbers to it – including some duplicates. The resulting list of items does not contain any duplicates. 

This solves my first problem – getting a distinct list of the objects that I have, which are invalid. Now I need to solve the second half of my problem.

ES6 Array Filter To Get The Difference

One of the new features of ES6 Array’s is the filter method. This method lets you decide which items to exclude from the original array, returning a new array with only the leftover items in it. 

In my case, I needed to filter the set of invalid objects out of the original array. To do this, I used the .has method on my “invalid” set. This method is used in the “filter” callback to determine which objects should be removed.

Note that the filter method keeps everything that returns “true” from the callback. Since I want to remove items that are in my other list, I need to invert the boolean result of the has method. That way my code says

If I have this object in the invalid list, return false so the object is removed.

If I do not have this object, return true so that the object is kept.

Having filtered the array down to a “valid” object list, I then converted it into a Set. This probably isn’t necessary, but made me a little more comfortable knowing that I would have a distinct list of objects with no duplicates.

At this point, I have both my list of valid and invalid objects!

It’s Like A Left Outer Join… For JavaScript!

Ok, that probably sounds dumb… but that’s how I was thinking about this problem. I was using a set-based, SQL mindset to think through the problem and found a way to perform the operations with my already in-memory arrays. 

If you have a database, use a database with a query language that knows how to deal with sets, like this. If you don’t have the database query available, you may be able to take advantage of that mindset, still. JavaScript does not natively work with set based operations in the way SQL does – but it does provide a functional, iterative approach to solving many of the same problems. 

(… and yes, there are a lot of potential improvements for this code – including the use of objects that store their own valid/invalid state. These objects don’t keep track of that information in their attributes. They just get put in to an invalid bucket when they are invalid, and taken out when they are valid.)

Categories: Blogs

New TeamForge Provides Tighter Integrations with JIRA

Scrum Expert - Tue, 09/22/2015 - 17:32
CollabNet has announced a new release of TeamForge. TeamForge now allows enterprise software teams to easily manage a wider range of tools and processes to streamline the increasingly complex software development lifecycle. Organizations leveraging open source, scaling Agile and adopting DevOps now have deeper levels of traceability and reporting, and tighter integrations with JIRA, Git, Subversion and other leading tools to create a unified software ...
Categories: Communities

Managers Don’t Have to Hate Agile

Danube - Tue, 09/22/2015 - 17:00

Earlier this year Forbes published an article titled “Why Do Managers Hate Agile?”  The author, Steve Denning, builds a case for managers hating Agile due to “management” and “Agile” being defined as two different worlds. It’s like Men are From Mars and Women are from Venus, only we’re talking about the IT world and management and developers, instead of men and women. The article caught my attention for the obvious reason that CollabNet sells products and services to help support Agile development efforts, sparking the question, “Why would managers hate Agile?”

In the article, the first world of “Management” is referenced as “The vertical world of hierarchical bureaucracy”.  Management is seen as the boss at the top of the totem pole and individuals at the bottom. This world is governed by roles, rules, plans and reports.

The second world of “Agile” is referenced as “The horizontal world of Agile,” and described as one operating horizontally with a focus on the customer instead of a vertical dynamic with people reporting upwards to bosses. Therefore, there is tension between the two worlds and the perception that managers hate Agile because it turns their world on its side and creates confusion. Management may not understand the needs of Agile developers and vice versa. The two entities speak a different language and don’t know how to communicate with each other. Sound familiar?

So what is the solution to relieve the tension for Agile and enable these two different worlds to not only get along, but also to work collaboratively to produce innovative products and services for their customers? The answer is…transparency. It sounds so simple…transparency, visibility, awareness. Yes, this is the key for managers to truly understand Agile developers and begin to work together.


The concept of transparency is one that CollabNet wholeheartedly embraces. Our TeamForge platform is specifically designed to give developers the tools they want to use and managers the end-to-end visibility and traceability they need – across the entire tool chain – and advanced data analytics and trending reports. As a result, transparency creates trust between management and developers.

When managers have visibility into the work of their developers and access to the reports they need, then the two worlds can come together not as enemies, but as partners. And the result is that managers don’t have to hate Agile.

What is your experience with the two Worlds of “Management” and “Agile”?


The post Managers Don’t Have to Hate Agile appeared first on

Categories: Companies

The Responsibility Process, Context, and Safety

Agile For All - Bob Hartman - Tue, 09/22/2015 - 16:15

The Responsibility Process™ is a practice (some may argue it’s more or less than a practice) that helps us move towards more self-mastery. Being able to facilitate ourselves is all about emotional intelligence – our ability to recognize and react appropriately in the moment to our emotions.

The Responsibility Process has been around for quite some time, so I’m still surprised that so few people are aware of it. The Responsibility Process is a structure to become aware of the states we all move through or get stuck in, as we aim to move to a mindset of responsibility. The Process was created by Christopher Avery who spent years researching the idea (you can read more at Christopher Avery’s website or just search Google!).

A Summary

Through years of research, Avery determined that responsibility is not a character trait that some people have and others don’t.


Falling out of the mental state of responsibility does not look quite like this, but it might feel this way!

Responsibility is something that we can see and learn — it is a mental state.

That alone is an interesting concept, since we often say ‘that person is irresponsible.’ Maybe we mean, ‘based on my perception, that person is acting irresponsibly.’?

There are a number of mental states that we go through when something interrupts our movement towards a goal. Ideally, we move through them to get to responsibility. These are emotions or feelings that, if we can work from responsibility and be aware and engaged with this process, we are better able to manage.

I’m going to explain the states from bottom to top (referencing the poster), since these are the states we typically move through sequentially. When we are able to ‘figure out’ one state, we are effectively moving on to the next one.

When we fall out of the mental state of responsibility, we end up searching for answers and end up in these other states.

  • Denial – We are not aware of the issue. Some say this is ignoring the problem, but it’s really not even knowing there is an issue to contend with.

    The Responsibility Process

    The Responsibility Process Poster — you can download official posters at Christopher Avery’s website above.

  • Lay Blame – When we are here we believe that external forces are at fault. This is all about giving away our power because we are convinced that someone else is in control here. If we realize we have no power to effect any change here, we can move to the next state.
  • Justify – Justify is about rationalizing why the situation makes sense. You might hear “it has always been this way” as one type of justification. Something ‘out there’ must change for the issue to go away. Moving to this state from blaming someone for a problem might be a solid step forward, but realize that if you are now justifying why that person caused a problem (e.g. they have never been trained) you still are not at a place of ownership. In this justify, you are still in a powerless state, believing again that external forces must change.
  • Shame – Shame is self-blame. As we move past believing that external forces are at fault, we turn the fault inward on ourselves. We might say “I can’t believe I did that again” or “I’m such an idiot of falling for that one.”
  • Obligation – This state is about believing you are doing what you must. We make commitments because we have to do ‘this!’ But if we are stuck doing things where we feel we do not have any choice, we don’t have ownership.
  • Quit – this is simply a way to attempt to avoid the issue. You might be trying to avoid the drama or trauma of the situation or you might simply be exhausted by obligation or shame. In any case, we check out. This can feel a lot like Stonewalling.
  • Responsibility – Avery says this means “I have the power and the ability to find and resolve the real problem.”  When I have the mindset that this is mine to solve or to act on, I can access new options to make changes — often that simply did not exist in the other states of the process.

An example: I was in a situation a while back where I volunteered to help someone with a class. While I was only a volunteer, it was an important part of the class. I got a call that a client needed me at the same time. I felt obligated both to take the work and to help with the class. I was mad at myself for getting into the situation and quite upset that I was going to let this person down. The person I was going to volunteer for said they understood, but was also clearly not happy about the situation (which I understood!). I could justify why I needed to take the client work, since it paid and volunteer position did not — or say “I need to be “responsible” for my family!” So many options! When I actually shifted to responsibility, I realized that I did need to take the client work, but also needed to go out of my way to find someone who could step in to replace me. Finding a replacement truly seemed impossible when I was in the coping states of blame, justify, shame, and obligation. I did find someone to replace me and they were very excited to do it. In the end, shifting allowed me to see more options and press through them even though they were challenging.


Ah, awareness. If we all had it all the time perhaps we could gleefully ride to responsibility and simply wave to the coping states on the way there!

We may only be in a state for a second or even less. You might find that as you start to blame you switch mid-sentence to justify. So they are not all necessarily resting places where we stay for a long period of time. Imagine yourself riding a bike towards responsibility and notice off to the side that blame and justify are there, sitting on the curb. They ask you to stop and chat for a while, but you simply smile and wave to them and continue on your journey to responsibility.

Doing the self-work to move to a responsibility mindset is a critical part of growing as a human and is a requirement for good leadership (regardless of your role or title).

Context and Safety

While I’ve used The Responsibility Process for myself and within organizations, as I started to work on this article, I realized I tended to use it with certain assumptions in place. I look at it and talk about it with a focus on personal responsibility (I’ll talk about teams in a future article). I also tend to use it in situations where people have options to shift their mindset, not extremely oppressive situations.

I have tried not to use The Responsibility Process in contexts where safety was clearly missing or where there was major privilege, rank, and/or power at play. To shift, you need to be at choice to move along the states of the process to get to responsibility. You need to see options. Some may argue that we always have a choice, but if you are in a place of rank or privilege making that argument, your judgement may be impaired. Beyond business situations (where this can occur as well) consider this idea and how it might apply to many of the situations out in the world today.

An example: If you are in a situation with a terrible boss who yells at you and belittles you, with all power aligned to that person, you can’t find another position, and need to support yourself or family — the idea of simply choosing to move out of justify or blame would likely strike you as absurd. That said, lets say you were able to move beyond blame and justify. You decide you are done with shame (I’m terrible and that’s why I get yelled at) and have moved past obligation (to do the job and help all the organizations end customers). The responsibility mindset allows you to theoretically move past those — and you continue to look for another job, but also stay in the hostile environment because you are responsible for your family. Saying this is no easy task would be an understatement!

If you don’t have safety, getting to responsibility is a lot more challenging. If people are not safe, they will have less options and less ability to change mindsets. Increasing safety may be something to focus on if you want people to exercise more responsibility. That may increase peoples options to choose from.

Being Responsible

Write the steps down and paste it on your wall or your laptop. When you run into a situation ask yourself where you are. Assess what it means for you to be in each step and what it would be like to move to the next one in the process. Focus on yourself and your mindset. Work on being in a responsibility mindset and see what it does for you! Love to hear how it turns out for you!

Subscribe to receive emails with my new posts and Agile Safari cartoons and Follow me on Twitter or checkout my page for more ways to connect!

The post The Responsibility Process, Context, and Safety appeared first on Agile For All.

Categories: Blogs

Using Strength-based Questions in Retrospectives

Ben Linders - Tue, 09/22/2015 - 15:29
Agile retrospectives help teams to learn from how they are doing and find ways to improve themselves. In stead of learning from what went wrong, from problems, mistakes or failures, teams can also learn from things that went well, by asking strength-based questions in their retrospectives. Such questions will help them to use their existing skills and experience to become great in doing things that they are already good at. Continue reading →
Categories: Blogs

Agile Bodensee, Constance, Germany, September 30 – October 1 2015

Scrum Expert - Tue, 09/22/2015 - 15:21
The Agile Bodensee conference is a two-day event focused on Agile and Scrum and that takes places on the shore of the Lake Constance (the Bodensee in German) in the southern part of Germany. The first day is dedicated to workshops and the second day to an open space discussion. Everything is in German. The Agile Bodensee conference follows the open space format for conferences. Open ...
Categories: Communities

Dependencies Are Evil

Leading Agile - Mike Cottmeyer - Tue, 09/22/2015 - 12:00

It seems inarguable to me that dependencies limit agility.

If I am able to make a decision on my own that only involves me, I have full freedom of movement. If I want to go to the local pub for dinner, and on my way, decide that I want to go to a steakhouse, I go to the steakhouse. I don’t have to call anyone, rearrange a schedule, broker agreement or anything. I just go to the steakhouse.

Now let’s say that I am meeting my wife for dinner. She is on her way to the pub, I am on my way to the pub. I decide that I want to go to the steakhouse. I can call her. I may get a hold of her, I might not. She may be up for steak, she might not. Either way, I have now lost some of my ability to decide.

I’ve created a dependency on my wife around dinner.

Now let’s say my wife and I are going to dinner with our in-laws. I love my in-laws but they are getting older and don’t like plans to change. I know they aren’t up for the pub, but they love steak and really, really like to make plans in advance. If I am going to have steak with my in-laws there is no room for plans to change… ever.

My wife and I have created a rather hard dependency with my in-laws.

In short, when it’s just me, I am autonomous. The more people I have to coordinate with, the less I am able to change direction. Depending on who is going with me, I may not be able to change at all.

Dependencies will inevitably limit my ability to act autonomously. Dependencies have limited my personal agility.

Again, this just seems inarguable to me.

When it comes to dealing with dependencies, I have two choices. I can either choose to break dependencies or I can choose to manage them. I cannot pretend they don’t exist.

If I go to the steakhouse, and ditch my wife waiting for me at the pub, there are consequences.

If I back out on my in-laws and go to the pub, there are consequences.

The same thing works with agile teams.

If I am a small Scrum team, I have 6-8 people, operate off a single backlog, have a single Product Owner, have a single ScrumMaster, and don’t share any of my people with any other Scrum teams, I can be totally agile. I can burn my backlog at my own rate, change direction, and basically do what we need to do.

Let’s say I start sharing team members between Scrum teams. Now I have to accommodate during Sprint planning their partial participation on my team. If their other team gets behind, that impacts my team, because the shared team member might not be available as planned to do my work.

Let’s say that one of the Scrum teams is a back-end team and the other is a front-end team. If I have to change the front-end and the back-end in order to get ready for the release, not only do I create a requirements dependency, but I also have a scheduling dependency. Both have to be ready on-time for the release to happen.

I know I said at the beginning of this post that dependencies were evil, but I’m not being judgmental here.

If you have dependencies like this, you have to manage them.

That said, they will limit your agility.

If you pretend otherwise, you are fooling yourself.

The thing about dependencies…

Many dependencies are self-inflicted. They get introduced by how we architect the product. They get introduced by how we staff teams. They get introduced by how we bring work into the organization, by how we create roadmaps, and by how we sell changes to our customers.

Very few dependencies can’t be broken with sufficient time and money and attention.

Here is the deal though… from YOUR point of view, dependencies might not be able to be broken. In your role, YOU might not be able to break them. That said, SOMEONE can choose to invest the time, money, or attention necessary to break them.

We just have to find those people, and they have to decide if it’s less expensive to break dependencies, or to keep managing them.

So again… dependencies are evil.

Dependencies reduce freedom of movement, thus agility.

They won’t stop you from doing Scrum, but they will stop you from getting benefit from Scrum.

You can choose to manage dependencies.

You can choose to break dependencies.

If you can’t break them, I guarantee someone in your organization can.

It just depends on if it’s worth it to break them.

What you can’t do is pretend Scrum will work well in their presence.

Dependencies break Scrum.

Dependencies are evil.

The post Dependencies Are Evil appeared first on LeadingAgile.

Categories: Blogs

Publishing ES6 code to npm

Xebia Blog - Tue, 09/22/2015 - 08:58

This post is part of a series of ES2015 posts. We'll be covering new JavaScript functionality every week!

Most of the software we work with at Xebia is open source. Our primary expertise is in open source technology, which ranges our entire application stack. We don’t just consume open source projects, we also contribute back to them and occasionally release some of our own. Releasing an open source project doesn’t just mean making the GitHub repository public. If you want your project to be used, it should be easy to consume. For JavaScript this means publishing it to npm, the package manager for JavaScript code.

Nowadays we write our JavaScript code using the ES6 syntax. We can do this because we’re using Babel to compile it down to ES5 before we run it. When you’re publishing the code to npm however you can’t expect your package consumers to use Babel. In fact if you’re using the Require Hook, it excludes anything under node_modules by default and thus will not attempt to compile that code.


The reality is that we have to compile our code to ES5 before publishing to npm, which is what people writing libraries in CoffeeScript have been doing for a long time. Luckily npm provides a helpful way to automate this process: the prepublish script. There’s a whole list of scripts which you can use to automate your npm workflow. In package.json we simply define the scripts object:

  "name": "my-awesome-lib",
  "version": "0.0.0",
  "scripts": {
    "compile": "babel --optional runtime -d lib/ src/",
    "prepublish": "npm run compile"
  "main": "lib/index.js",
  "devDependencies": {
    "babel": "^5.8.23",
    "babel-runtime": "^5.8.24"

This will make npm automatically run the compile script when we run `npm publish` on the command line, which in turn triggers babel to compile everything in ./src to ./lib (the de-facto standard for compiled sources in npm packages, originating from CommonJS). Finally we also define the main entry point of our package, which denotes the file to import when require(‘my-awesome-lib’) is called.

An important part of the above configuration is the runtime option. This will include polyfills for ES6 features such as Promise and Map into your compiled sources.

Ignoring files

It’s easy to simply publish all of our source code to npm, but we don’t have to. In fact it’s much nicer to only publish the compiled sources to npm, so package consumers only have to download the files they need. We can achieve this using a file called .npmignore:


This file is very similar to .gitignore, in fact npm will fall back to .gitignore if .npmignore is not present. Since we probably want to ignore ./lib in our git repository because it holds compiled sources, npm would normally not even publish that directory, so we have to use .npmignore to get the desired result. However if you want your library to also be directly consumable via a git URL you can commit ./lib to git as well.


Now that everything is set up, the only thing left to do is publish it. Well, not so fast. First we should specify our version number:

npm version 1.0.0

This will update package.json with the new version number, commit this change to git and make a git tag for v1.0.0, all in one go. This of course assumes you're using git, otherwise it will just update package.json. An even better way is to have npm automatically determine the next version number by specifying the scope of change. We should also specify a commit message:

npm version patch -m "Bump to %s because reasons"

For more options check out the npm documentation. Now finish up by pushing your new version commit and tag to GitHub and publish our package to npm:

git push --follow-tags
npm publish
Categories: Companies

Lunch for Integrating Teams

Lunch is a funny thing for many developers. With a tendency towards introverts often people get into habits of eating alone either with food from home or just running out to pick something up and bring it back. Lunch as a social outing is a bit unusual and unexpected.

I’ve viewed eating with others as one of my favorite parts of the working day for a long time. Even when I work remote I often schedule lunches with former colleagues or friends to keep up with them. At my present gig I work in the office Mondays, Tuesdays, and Thursdays. Tuesdays are a company sponsored team lunch which has been a great way to get everyone out eating together. Mondays and Thursdays I typically get a bite with another one of our developer teams. I enjoy the conversations and it’s a free way to keep up with another team and explain what’s going on with our project. I know there’s a small cost involved in eating out, but it’s only two days a week and far outweighed by sharing knowledge across the organization. And I do like to get out of the office.

So if as a developer you tend towards the lunch from home and eating at your desk, try out lunch out with some co-workers at least once a week. It’s a cheap experiment and a great way keep up with your organization.

Categories: Blogs

SparkR: Add new column to data frame by concatenating other columns

Mark Needham - Tue, 09/22/2015 - 00:30

Continuing with my exploration of the Land Registry open data set using SparkR I wanted to see which road in the UK has had the most property sales over the last 20 years.

To recap, this is what the data frame looks like:

./spark-1.5.0-bin-hadoop2.6/bin/sparkR --packages com.databricks:spark-csv_2.11:1.2.0
> sales <- read.df(sqlContext, "pp-complete.csv", "com.databricks.spark.csv", header="false")
> head(sales)
                                      C0     C1               C2       C3 C4 C5
1 {0C7ADEF5-878D-4066-B785-0000003ED74A} 163000 2003-02-21 00:00  UB5 4PJ  T  N
2 {35F67271-ABD4-40DA-AB09-00000085B9D3} 247500 2005-07-15 00:00 TA19 9DD  D  N
3 {B20B1C74-E8E1-4137-AB3E-0000011DF342} 320000 2010-09-10 00:00   W4 1DZ  F  N
4 {7D6B0915-C56B-4275-AF9B-00000156BCE7} 104000 1997-08-27 00:00 NE61 2BH  D  N
5 {47B60101-B64C-413D-8F60-000002F1692D} 147995 2003-05-02 00:00 PE33 0RU  D  N
6 {51F797CA-7BEB-4958-821F-000003E464AE} 110000 2013-03-22 00:00 NR35 2SF  T  N
  C6  C7 C8              C9         C10         C11
3  L  58      WHELLOCK ROAD                  LONDON
4  F  17           WESTGATE     MORPETH     MORPETH
                           C12            C13 C14
1                       EALING GREATER LONDON   A
2               SOUTH SOMERSET       SOMERSET   A
3                       EALING GREATER LONDON   A
6                SOUTH NORFOLK        NORFOLK   A

This document explains the data stored in each field and for this particular query we’re interested in fields C9-C12. The plan is to group the data frame by those fields and then sort by frequency in descending order.

When grouping by multiple fields it tends to be easiest to create a new field which concatenates them all and then group by that.

I started with the following:

> sales$address = paste(sales$C9, sales$C10, sales$C11, sales$C12, sep=", ")
Error in as.character.default(<S4 object of class "Column">) :
  no method for coercing this S4 class to a vector

Not so successful! Next I went even more primitive:

> sales$address = sales$C9 + ", " + sales$C10 + ", " + sales$C11 + ", " + sales$C12
> head(sales)
                                      C0     C1               C2       C3 C4 C5
1 {0C7ADEF5-878D-4066-B785-0000003ED74A} 163000 2003-02-21 00:00  UB5 4PJ  T  N
2 {35F67271-ABD4-40DA-AB09-00000085B9D3} 247500 2005-07-15 00:00 TA19 9DD  D  N
3 {B20B1C74-E8E1-4137-AB3E-0000011DF342} 320000 2010-09-10 00:00   W4 1DZ  F  N
4 {7D6B0915-C56B-4275-AF9B-00000156BCE7} 104000 1997-08-27 00:00 NE61 2BH  D  N
5 {47B60101-B64C-413D-8F60-000002F1692D} 147995 2003-05-02 00:00 PE33 0RU  D  N
6 {51F797CA-7BEB-4958-821F-000003E464AE} 110000 2013-03-22 00:00 NR35 2SF  T  N
  C6  C7 C8              C9         C10         C11
3  L  58      WHELLOCK ROAD                  LONDON
4  F  17           WESTGATE     MORPETH     MORPETH
                           C12            C13 C14 address
1                       EALING GREATER LONDON   A      NA
2               SOUTH SOMERSET       SOMERSET   A      NA
3                       EALING GREATER LONDON   A      NA
6                SOUTH NORFOLK        NORFOLK   A      NA

That at least compiled but all addresses were ‘NA’ which isn’t what we want. After a bit of searching I realised that there was a concat function that I could use for exactly this task:

> sales$address = concat_ws(sep=", ", sales$C9, sales$C10, sales$C11, sales$C12)
> head(sales)
                                      C0     C1               C2       C3 C4 C5
1 {0C7ADEF5-878D-4066-B785-0000003ED74A} 163000 2003-02-21 00:00  UB5 4PJ  T  N
2 {35F67271-ABD4-40DA-AB09-00000085B9D3} 247500 2005-07-15 00:00 TA19 9DD  D  N
3 {B20B1C74-E8E1-4137-AB3E-0000011DF342} 320000 2010-09-10 00:00   W4 1DZ  F  N
4 {7D6B0915-C56B-4275-AF9B-00000156BCE7} 104000 1997-08-27 00:00 NE61 2BH  D  N
5 {47B60101-B64C-413D-8F60-000002F1692D} 147995 2003-05-02 00:00 PE33 0RU  D  N
6 {51F797CA-7BEB-4958-821F-000003E464AE} 110000 2013-03-22 00:00 NR35 2SF  T  N
  C6  C7 C8              C9         C10         C11
3  L  58      WHELLOCK ROAD                  LONDON
4  F  17           WESTGATE     MORPETH     MORPETH
                           C12            C13 C14
1                       EALING GREATER LONDON   A
2               SOUTH SOMERSET       SOMERSET   A
3                       EALING GREATER LONDON   A
6                SOUTH NORFOLK        NORFOLK   A
1                             READING ROAD, NORTHOLT, NORTHOLT, EALING
3                                      WHELLOCK ROAD, , LONDON, EALING
4                           WESTGATE, MORPETH, MORPETH, CASTLE MORPETH

That’s more like it! Now let’s see which streets have sold the most properties:

> byAddress = summarize(groupBy(sales, sales$address), count = n(sales$address))
> head(arrange(byAddress, desc(byAddress$count)), 10)
                                                            address count
1                          BARBICAN, LONDON, LONDON, CITY OF LONDON  1398
8                             QUEENSTOWN ROAD, , LONDON, WANDSWORTH  1153
10                      QUEENSTOWN ROAD, LONDON, LONDON, WANDSWORTH  1079

Next we’ll drill into the data further but that’s for another post.

Categories: Blogs

SparkR: Error in invokeJava(isStatic = TRUE, className, methodName, …) : java.lang.ClassNotFoundException: Failed to load class for data source: csv.

Mark Needham - Tue, 09/22/2015 - 00:06

I’ve been wanting to play around with SparkR for a while and over the weekend deciding to explore a large Land Registry CSV file containing all the sales of properties in the UK over the last 20 years.

First I started up the SparkR shell with the CSV package loaded in:

./spark-1.5.0-bin-hadoop2.6/bin/sparkR --packages com.databricks:spark-csv_2.11:1.2.0

Next I tried to read the CSV file into a Spark data frame by modifying one of the examples from the tutorial:

> sales <- read.df(sqlContext, "pp-complete.csv", "csv")
15/09/20 19:13:02 ERROR RBackendHandler: loadDF on org.apache.spark.sql.api.r.SQLUtils failed
Error in invokeJava(isStatic = TRUE, className, methodName, ...) :
  java.lang.ClassNotFoundException: Failed to load class for data source: csv.
	at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.lookupDataSource(ResolvedDataSource.scala:67)
	at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:87)
	at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:114)
	at org.apache.spark.sql.api.r.SQLUtils$.loadDF(SQLUtils.scala:156)
	at org.apache.spark.sql.api.r.SQLUtils.loadDF(SQLUtils.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(
	at java.lang.reflect.Method.invoke(
	at org.apache.spark.api.r.RBackendHandler.handleMethodCall(RBackendHandler.scala:132)
	at org.apache.spark.api.r.RBackendHandler.channelRead0(RBackendHandler.scala:79)
	at org.apache.spark.api.r.RBackendH

As far as I can tell I have loaded in the CSV data source so I’m not sure why that doesn’t work.

However, I came across this github issue which suggested passing in the full package name as the 3rd argument of ‘read.df’ rather than just ‘csv’:

> sales <- read.df(sqlContext, "pp-complete.csv", "com.databricks.spark.csv", header="false")
> sales
DataFrame[C0:string, C1:string, C2:string, C3:string, C4:string, C5:string, C6:string, C7:string, C8:string, C9:string, C10:string, C11:string, C12:string, C13:string, C14:string]

And that worked much better! We can now carry on and do some slicing and dicing of the data to see if there are any interesting insights.

Categories: Blogs

Knowledge Sharing

SpiraTeam is a agile application lifecycle management (ALM) system designed specifically for methodologies such as scrum, XP and Kanban.