Skip to content

Feed aggregator

How to Become a Software Craftsman

Agile Management Blog - VersionOne - Thu, 08/06/2015 - 14:30

Craftmanship

 

 

 

 

 

 

 

How to become a Software Craftsman has become a huge subtext in the software community and the development conversation. One of the things that I’ve been exploring is how do we get there? How do we go from where we are to becoming  true software craftsmen?

It’s not this magical “Oh, we’re agile, we put posters of the manifesto everywhere, so now we’re agile and we’re software craftsmen.” It takes work, and it takes activities. I’ve been doing a lot of exploration into this and believe I have discovered three paths to the summit of software craftsmanship.

Software Craftsman Defined

Let’s first define what I mean by a software craftsman. Everybody has their own views, but I think of a software craftsman as someone who has practiced the techniques of XP, agile, and DevOps till those techniques have worked themselves into the person’s subconscious. This software craftsman creates software using these techniques almost through muscle memory. They no longer have to think about what they need to do to create beautiful code, they just execute in their relentless pursuit of creating amazing software products.

How You Get to be a Software Craftsman

So now that we have defined what being a software craftsman means to me, let’s explore how we get there. I’ve found that there are three paths to becoming a software craftsman. These paths are comprised of software development skills that need to be developed. The first path develops the people skills, the second path develops the technical skills and the third path explores the principles derived from the other two paths.

Let’s survey what each of these paths covers.

The People Path

One of aspects that I’ve been exploring a lot is that there are people problems and there are technical problems to DevOps and software craftsmanship. Of course, the technical skills are critical, but no more so than the people side. Just like it does a person no good to only strengthen their right arm, while their left atrophies, it does us no good to only strengthen our technical skills, while our people skills go to waste.

We must learn and apply technical tools from a people perspective. Craftsmanship means doing things by hand and knowing how to execute with an artist’s touch. It’s not enough to say, “I expect you to be doing test-driven development.” You have to be able to help people understand what test-driven development is.

Those of us in the software development community must help foster craftsmanship because it certainly isn’t being taught in school. The typical computer science college graduate does not understand how to do test-driven development or agile. We’re making some progress, but it’s still not good enough.

The Technical Path

There is also the technical side to keep in mind. Practices include test-driven development, refactoring and continuous integration. It is also important to know when and how to write acceptance tests and how to apply these as well as how to automate these. These are the nuts and bolts of a solid software craftsman. The idea of refactoring becomes part of daily life, not just using the buzzword but intuitively building it into everything you do.

These are the steps you’re going to continue with, and at some point that will lead you to DevOps and continuous delivery. These technical practices and methodologies are more organizational than individualized, but DevOps and continuous delivery do require a discipline that only a software craftsman can really, truly supply.

The Principles Path

I’ve found that you can break this down into certain areas of foundational skills that need to be developed. The first of those foundational skills is coding. Sounds kind of silly to say it, but it’s worth saying; a programmer needs to be excellent at coding.

Coding

By coding I don’t mean one language. You must be astute in multiple languages. No true craftsman only knows one way of doing anything. It doesn’t matter what languages, but there needs to be at least two, preferably more.

Design

Designing is an important aspect, but is tricky in the agile world because we say, “don’t get caught up in big, up front designing.” I believe that, but you do need to understand design. Whether it be unit design, large architectural pieces or systems design, you do need to understand and be able to apply good design.

Applying Agile Principles

Learning the 12 principles of the Agile Manifesto isn’t very difficult; applying them is much harder. You have to understand when they apply, when simplicity really is essential, what simplicity is and how to apply simplicity to a particular problem.

Tooling

We have many tools available to us. Mastering these tools as a true craftsman is not about simply using them, it’s about knowing how to use them wisely. Like the saying goes, “If you need a hammer, whatever tool is handy is a hammer.” That’s not necessarily the best approach. A craftsman seeks the right tool for the right job and uses that tool masterfully.

Work Habits

You need to be able to establish strong work habits. You need to be able to, not just by yourself, but in a team, practice these habits. Test-driven development and continuous integration are tools to help us with practice our work habits. Having the work habits and the discipline around those work habits to be able to say yes or no and to be able to say, “This is what I need to do, and it’s what I will do” is crucial.

Professionalism

Professionalism is something you notice more often when it’s not there than when it is. There is a quiet confidence and understanding in professionals. There is the ability to know where you’re going and what you’re doing without conscious thinking. You have confidence in professionals because you know that they will do great work. That’s what I mean by professionalism, it’s very difficult to define, but it’s absolutely vital to a strong development shop. Especially, as we aspire to craftsmanship.

Traditional Craftsman Education

To learn how to teach software craftsmanship, we have to look no further than the trades where craftsmanship originated. Traditionally, craftsmen were created through apprenticeships. By going through an apprenticeship program, young craftsmen, no matter what their background or their education, learn not just what they should do but how to do it. They’d learn the tricks and techniques that don’t necessarily come just from reading a book, taking a class or passing a test. It’s about learning from doing, and it’s learning by doing things together.

The next component of how craftsmanship has been historically taught is recognizing progress. To recognize progress, you need a path to follow. It’s no secret that there’s no really well-defined career path for software developers.

The typical path of a developer is to start as a junior software engineer, progress to a senior software engineer and, if you’re really good, you become a tech lead. As a tech lead, you have to now tell other people how to do it. Then, if you’re really good at that, they take you completely out of the thing you love, which is programming, and make you a manager. Then, you get to try to figure out how to make other people do what you love to do most. That path has never really worked.

Micro-certifications are becoming very popular in the world of education and development. Think of micro-certifications as similar to boy scout badges. You could have a badge in test-driven development, concrete data systems or web design. By obtaining these badges, you can recognize progress and simultaneously you can be recognized for that progress. When taking an approach such as this you should start associating some of the compensation and programs with the development of these badges.

When you are done with your apprenticeship, you are, of course, not done learning. At this stage, you grow into a journeyman. A journeyman is a very time-honored tradition. The idea of a journeyman is that now you are good enough to go out on your own. In the traditional craftsmanship model, a journeyman would have wandered from village to village practicing their craft.

In the software world, this might mean you work on a team for a year or maybe two. Then, you go to another team. The wandering part doesn’t have to be quite as frequent as the traditional journeyman, but the idea is that you need to continue to develop your skills and to develop them not in a single place but to explore other areas.

If you’ve been doing nothing but data mining for six months, then maybe for the next six months you should be focused on webpages so that you are building a broad base of skills. That’s what a journeyman’s life is. We should be spending the majority of our time as journeymen.

Conclusion

These are the steps. It’s not the easiest path in the world, but it’s absolutely worth it as you go along. We should all aspire to be great at our craft and be true craftsmen in our discipline. I hope this has inspired you to take a look at what areas you can develop to become a stronger software craftsman.

What other skills do you think are important for a software craftsman to develop?

About the Author

versionone-coaches-steve-ropaSteve Ropa
CSM, CSPO, Innovation Games Facilitator, SA
Agile Coach and Product Consultant, VersionOne

Steve has more than 25 years of experience in software development and 15 years of experience working with agile methods. Steve is passionate about bridging the gap between the business and technology and nurturing the change in the nature of development. As an agile coach and VersionOne product trainer, Steve has supported clients across multiple industry verticals including: telecommunications, network security, entertainment and education. A frequent presenter at agile events, he is also a member of Agile Alliance and Scrum Alliance.

Categories: Companies

Agile Hotline – Is agile more efficient than waterfall?

Growing Agile - Thu, 08/06/2015 - 13:43
Question Recently someone asked for advice about a debate he was having with a co-worker about whether agile and Scrum can work in any software development scenario. The co-worker argued that it couldn’t. This was his scenario Say we are working on building a software product that allows user’s to take assessments while reading an […]
Categories: Companies

Article published on Information Age: Why expert developers make the worst tech leads

thekua.com@work - Thu, 08/06/2015 - 09:46

Promoting your best or most experienced developer to lead a technology team may seem like a logical move, but it could prove to be a disastrous decision

I recently published a new article on the Information Age website: “Why expert developers make the worst tech leads.” You can read more here.

Categories: Blogs

Hugh MacLeod’s Illustrated Guide to Life Inside Microsoft

J.D. Meier's Blog - Wed, 08/05/2015 - 20:32

imageIf you remember the little blue monster that says, “Microsoft, change the world or go home.”, you know Hugh MacLeod.

Hugh is the creative director at Gaping Void.  I got to meet Hugh, along with Jason Korman (CEO), and Jessica Higgins, last week to talk through some ideas.

Hugh uses cartoons as a snappy and insightful way to change the world.  You can think of it as “Motivational Art for Smart People.”

The Illustrated Guide to Life Inside Microsoft

One of Hugh’s latest creations is the Illustrated Guide to Life Insight Microsoft.  It’s a set of cards you can flip, with a cartoon on the front, and a quote on the back.  It’s truly insight at your fingertips.

image

I like them all … from “Microsoft is a ‘Get Stuff Done’ company” to “Software is the thing between the things”, but my favorite is:

“It’s more fun being the underdog.”

It’s a reminder how you can take the dog out of the fight, but you can’t take the fight out of the dog, and as long as you’re still in the game, and you are truly a learning company, and a company that continues to grow and evolve, you can change the world … your unique way.

Tweaking People in the Right Direction

Hugh is an observer and participant who inspires and prods people in the right direction …

Via Hugh MacLeod Connects the Dots:

“’Attaching art to business outcomes can articulate deep emotions and bring things to light fast,’ said MacLeod. To get there requires MacLeod immersing himself within a company, so he can look for what he calls ‘freaks of light’—epiphanies about a company that express the collected motivations of its people. ‘My cartoons make connections,’ said MacLeod. ‘I create work in an ambient way to tweak people in the right direction.’”

Via Hugh MacLeod Connects the Dots:

“He’s an observer and a participant, mingling temporarily within a culture to better understand it. He’s also a listener, taking your thoughts and combining them with his own to piece together the puzzle he is trying to solve about the human condition and business environment.”

Check out the Illustrated Guide to Life Inside Microsoft and some of the ideas just might surprise you, or, at least inspire and motivate you today – you smart person, you.

Categories: Blogs

AutoMapper 4.0 Released

Jimmy Bogard - Wed, 08/05/2015 - 18:46

Release notes here: https://github.com/AutoMapper/AutoMapper/releases/tag/v4.0.0

On NuGet of course.

This was a big release – I undertook the exciting challenge of supporting all the new platforms from VS 2015, and in the process, collapsed all of the projects/assemblies into exactly one assembly per target framework. It’s much easier to manage on my side with just the one project instead of many different ones:

I have to use compiler directives instead of feature discovery, but it’s a tradeoff I’m happy to make.

There’s a ton of small bug fixes in this release, quite a few enhancements and a few larger new features. Configuration performance went up quite a bit, and I’ve laid the groundwork to make in-memory mapping a lot faster in the future. LINQ projection has gotten to the point where you can do anything that the major query providers support.

Enjoy!

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

Categories: Blogs

Pitfall of Scrum: ScrumMaster as Contributor

Learn more about our Scrum and Agile training sessions on WorldMindware.com

The ScrumMaster is like a fire-fighter: it’s okay for them to be idle – just watching the team – waiting for an emergency obstacle. Taking on tasks tends to distract the ScrumMaster from the job of helping the team follow the rules of Scrum, from the job of vigorously removing obstacles, and from the job of protecting the team from interruptions. Let’s look at each of these aspects of the ScrumMaster role in turn:

The ScrumMaster Helps the Team Follow the Rules of Scrum

The ScrumMaster is a process facilitator. The Scrum process, while simple to describe, is not easy to do. As the Scrum Guide says:

Scrum is:

Lightweight

Simple to understand

Difficult to master

The ScrumMaster helps the Scrum Team and the organization to master the Scrum framework. Helping everyone understand Scrum and respect its rules is a first step. Some of the rules are particularly challenging. In some companies, being on time for meetings and ending them on time is hard. Scrum requires this. The ScrumMaster helps the team do this. In some companies, meeting deadlines, even short ones, is difficult. Scrum requires this every Sprint. The ScrumMaster helps the team do this. In some companies, giving time to improving things is hard. Scrum Teams do retrospectives. The ScrumMaster ensures that the team takes the time for this.

Of course, following the rules is hard for people. Even just the concept of “rules” is hard for some people. Everyone has the right to do whatever they want. Well, if you aren’t following the rules of Scrum you aren’t doing Scrum. So for some teams, just getting to the point of being willing to follow the rules of Scrum is a big step. The ScrumMaster needs to help with motivation.

The ScrumMaster is Vigorously Removing Obstacles

The Scrum Team is going to be working hard to meet a goal for the Sprint. As they work, they are going to work through many challenges and problems on their own. However, the team will start to encounter obstacles as well. These obstacles or impediments come from a few sources:

  1. Dependencies on other people or parts of the organization outside the Scrum Team.
  2. Skill gaps within the team.
  3. Burdensome bureaucracy imposed by the organization.
  4. Lack of resources such as tools, equipment, licenses, or even access to funds.

The ScrumMaster needs to work through these.

On a panel talk on Saturday one person said “the scrum master is an administrator, moving cards, updating the burn down. It is an easy job, I think my son could do it.” I then rebutted his remarks….

The ScrumMaster will tackle enterprise operations for their slow error prone deployment process, tackle Sarbox [Sarbanes-Oxley] compliance policy that has been way over-engineered to the point of slowing dev to a crawl, telling the PMO that 3 sets of reports is waste, exhorting the team to try to do unit tests in ABAP (SAP cobol), etc.

Robin Dymond, CST – (Scrum Training and Coaching Community Google Group, Sep. 23, 2009)

The ScrumMaster is Protecting the Team from Interruptions

Every organization seems to have more work than their staff have the capacity to deliver. Staff are often asked to task switch repeatedly over the course of a day or even in a single hour. Sometimes people are “allocated” to multiple projects simultaneously. This breaks the Scrum value of focus. The ScrumMaster needs to protect the team from interruptions or anything else that would break their focus.

But what should the Scrum Team members be focused on? Simply: the goal of a single Sprint. And a single Scrum Team is focused on a single product. The Product Owner should be the point of contact for any and all requests for the time and effort of a Scrum Team. The ScrumMaster needs to re-direct any interruptions to the Product Owner. The Product Owner decides if:

  • the interruption results in a new Product Backlog Item, OR
  • the interruption is irrelevant to the product and simply discarded, OR
  • the interruption is important enough to cancel the current Sprint.

There are no other options in Scrum for handling requests for work from the Scrum Team (or any member of the Scrum Team).

Contribution as Distraction for the ScrumMaster

Any time the ScrumMaster starts to contribute to the product development effort directly, the ScrumMaster is distracted from the other three duties. Although simple, following the rules of Scrum is not easy. Getting distracted from the duty of helping the team follow the rules of Scrum means that the team is likely to develop bad habits or regress to non-Scrum behaviour. Vigorously removing obstacles is usually a huge job all on its own. Most Scrum Teams have huge organizational obstacle that must be worked on. Some of these obstacles will take years of persistent effort to deal with. The ScrumMaster cannot become distracted by tactical details of product development. Protecting the team from interruptions means the ScrumMaster must have broad awareness, at all times, of what is happening with the team. If a team member is interrupted by a phone call, an email, or someone walking into the Scrum team room, the ScrumMaster needs to notice it immediately.

Whenever a ScrumMaster takes on a product development task, focus on the role is lost and a condition of a simple conflict-of-interest is created. If the team has “committed” to deliver certain Product Backlog Items at the end of a Sprint, then that feeling of commitment may lead a ScrumMaster to focusing on the wrong things.

The time of a ScrumMaster is an investment in continuous improvement. Letting a ScrumMaster contribute to the work of the team dilutes that investment.

This article is a follow-up article to the 24 Common Scrum Pitfalls written back in 2011.

Try out our Virtual Scrum Coach with the Scrum Team Assessment tool - just $500 for a team to get targeted advice and great how-to informationPlease share!
facebooktwittergoogle_plusredditpinterestlinkedinmail

The post Pitfall of Scrum: ScrumMaster as Contributor appeared first on Agile Advice.

Categories: Blogs

What is Flow Debt?

Improving projects with xProcess - Wed, 08/05/2015 - 15:36
+Daniel Vacanti's excellent treatise on Actionable Agile Metrics [vaca] introduces a term that may be unfamiliar, even to those with an interest and experience in managing flow systems. The term is Flow Debt - for a definition and explanation read on.

My own particular interest in flow systems is the management of agile software development teams, usually using some variant of Scrum and/or Kanban, and other agile practices such as test-driven (or automated-test intensive) build-test-deploy processes. However the discussion is relevant in many other domains, such as one I've recently been involved in discussing, the flow of patients through diagnosis, treatment and convalescence in healthcare systems.

In managing these systems we need ways to look at the mass of data that emerges from them to focus on the useful information rather than the noise; information in particular that indicates when intervention is appropriate to improve flow, and when the attempt would be as futile as trying to smooth the waves on an ocean. Flow systems in knowledge work contain variability. That variability, within certain bounds (much wider bounds than in manufacturing for example), is desirable to allow innovation, responsiveness and minimising wasteful planning activities.

In this context Flow Debt is a measure that provides a view of what is happening inside our system. This is in contrast with other important measures such as Throughput (Th) and the time an item stays in the process (I call this "Time in Process", TiP [macc], though other terms may be used). These measures provide information only after items have left the system, which may be too late to avoid problems accumulating.

Having Flow Debt roughly translates as: delivering more quickly now at the cost of slower times later. It is calculated by comparing the time since the number of arrivals into the systems was equal to the current number of deliveries with the average time in the process for the most recent deliveries. It is easiest to visualise this on a Cumulative Flow Diagram.
At the point highlighted in the the diagram it is a little over 2 weeks since the cumulative number of items entering the system equalled the cumulative number of deliveries on that date. If the items were delivered in the precise order they arrived, and if all the items were delivered (neither assumption is true!), then we would be able to say that the time the last item spent in the process was also a little over 2 weeks. Furthermore if arrivals and deliveries were smooth over the period, the Average Time in Process for the items would also be this same time.
What was the actual Average Time in Process though? Well you can't read this off the diagram. You have to look at the average TiP for the items delivered in the recent period. Each one has a known TiP, so take the average of them. Exactly how long the period you select for this average is up to you - a day or a week seems reasonable. The shorter the period you take the more noise there will be in the signal. Take too long a period though and there is insufficient time to act on the information. 
With this information we can calculate Flow Debt using Dan's method[vaca]:Flow Debt = (Time since number of arrivals equalled deliveries) - Average TiPIf you plot this quantity for the data above you get a graph like this. Note I've reversed the sign on this graph to show Flow Debt as negative.The plot of Flow Debt in this case is quite normal showing a fluctuation around zero and maxima and minima of around the value of the average TiP for the whole period. If you plotted the same data with a monthly average, most of this fluctuation would disappear. I certainly wouldn't want managers rushing down to this team to radically change their process!
There is one point highlighted which is interesting, where the Flow Debt goes from highest debt to highest credit in a few days. What do you think is going on here? Well, if you go back to the informal definition of Flow Debt (delivering more quickly now at the cost of slower times later), we should surmise that before this point the delivered items had been in the process for only a short time. Those delivered at or after this point had a longer time in the process. That's exactly what happened, as the Control Chart below shows.Another useful indicator here is the "average age" of the work in progress. Here is the plot of that and you can see the significant drop in this metric at the same point.Just by way of balance let's look at another data set of a team delivering software much less frequently, where their work in progress is increasing over the process, and where the items are not being delivered in age order. All these factors are likely to effect the efficiency and predictability of the flow system... and this is borne out by their plot of Flow Debt.Seeing a plot like this is a indication to management (and flow management specialists in particular) to take a much closer look at the process being used here.
References
Categories: Companies

focus

Derick Bailey - new ThoughtStream - Wed, 08/05/2015 - 12:00

The month of July was almost entirely unfocused for me. I spent more than a week on “vacation” with my family, and I spent another week sick and stuck in my bed. With more than two weeks of downtime, essentially, I feel lost. I feel like I’m not sure of what I’m doing … and I’m scared of that feeling.

focus

It’s more than a bit scary, honestly – the feeling of being unfocused and not knowing what I’m supposed to be working on. I panicked a bit, too, and on the Entreprogrammers podcast, I ended up talking with Josh and John for almost 3 hours because I was panicking about being unfocused.

An Existential Crisis

The kind of focus I’m talking about isn’t the “put twitter down, get back to work” kind. No, I’m talking about the big picture… the “What is my purpose? Why am I here? What is the real value that my business, my blog and my efforts bring to the community?”

It turns out that the 2+ weeks of down time were what I actually needed – even if they weren’t what I wanted. Having that much down time gave me a chance to step back and think for a bit. And the questions I came up with while I was thinking scared me because I was seriously asking “What am I doing? Why?”

I was having a mild existential crisis.

I wasn’t sure what I was doing and I almost deleted my twitter account and stopped working entirely. I didn’t and I’m glad I didn’t. But I was close. Really close.

I Am Unfocused, But Now I Can Fix That

I’ve known that I have been unfocused for a long time now. This is why I shut down SignalLeaf – because I was stretched in too many directions. I had no time to do anything I wanted, and was constantly rushing around in a panic trying to keep things floating. SignalLeaf took a back seat when it should have been a top priority. WatchMeCode languished in the site design and features I wanted to add. My blogging and writing have been almost non-existent for a while.

When I shut down SignalLeaf and had those 2 weeks of down time in July, I realized how completely unfocused I was with everything else.

And now, I have the time to fix that. Now, I don’t have three separate businesses that I’m trying to run. I only have my consulting and my screencasting now. And my consulting feeds my screencasting, to a large degree. So really, I have a much more focused business already.

But I still don’t have the kind of focus I want.

Finding Focus

In the next few weeks, I’m going to spend some more time thinking. The thought process is going to be aimed at creating new focus for WatchMeCode and DerickBailey.com. I am going to put a new emphasis on the direction that I am heading with JavaScript and all of the writing that I am doing. That new heading will be centered around architecture and messaging based patterns in JavaScript – both on the client and on the server.

These next few weeks will be used to determine how I want to approach this renewed focus. I’ll be setting goals for myself. I’ll be generating metrics that I want to capture. I’ll be looking at what I need to do in order to accomplish these things. I’ll be creating a singular focus for my work as an entrepreneur in the JavaScript space.

Resurrecting Architecture

Messaging and architecture aren’t new territories for me. I’ve got the RabbitMQ For Developers package which is all about messaging. But I’ve also got years of experience writing about architecture and messaging patterns for all kinds of applications, over at my old DerickBailey.LosTechies.com blog.

Architecture is what I did for many, many years – including JavaScript architecture. I brought messaging patterns to the world of Backbone.js when I built Marionette.js on top of it. I helped bring these same patterns to Windows applications and mobile devices when I built projects on those platforms, as well.

I want to bring these patterns of architecture, of scalable (not just large scale – scalable from small, all the way out) systems, and of messaging both in memory and across distributed systems, in to the world of JavaScript. This little language that could is finally doing it for more people than ever, and it’s about time the JavaScript community embraced the architecture it needs.

Join Me In My New Focus

I’m getting ready to refocus my blogging and screencasting, to center around JavaScript based architecture and messaging patterns, and I want you to come along with me. Clearly, you’re here now on my mailing list. You’ll get the blog posts as I restart this journey. You’ll be on the edge of what I am working on, already. But, there’s so much more to be found in WatchMeCode, already.

Take advantage of the 100+ screencasts and other videos on WatchMeCode.

This new direction will take a while to gain momentum. But I’m already moving. And I want you to join me in this new, better and more focused direction.

Categories: Blogs

Agile leads to technical debt?

Does Agile lead to technical debt?

The phrase "technical debt" was coined by Ward Cunningham in 1992.  Ward is known for a few significant things: wikis, CRC cards, and influencing Extreme Programming (XP).

XP was by far the dominant "lightweight methodology" when the Agile Manifesto was created.

So it seems rather odd to suggest that Agile leads to technical debt.  All the talk about technical debt came from Agile people.

Here's Ward talking about the history of "technical debt":

Here's a couple articles by Martin Fowler on technical debt:
Ward and Martin are signatories of the Agile Manifesto, which makes sense because they participated in creating it.

It's really ironic that people today might believe that Agile leads to technical debt.

But I can understand the belief.

XP is no longer what people typically think of when they think "Agile".  There has been a shift to Scrum and Kanban and management issues in general.  I can understand that people exposed to Flaccid Scrum might believe that Agile equates to poor technical practice.

The more controversial, though unoriginal, claim would be that XP practices lead to technical debt.  In other words, claiming that the community that developed and embraced the concept of being mindful of technical debt advocates practices that lead to it.
Categories: Blogs

Agile is for extroverts?

Is Agile for extroverts?
And given most programmers are introverts, is Agile just stupid?

Given that I'm an introvert and most people I've worked with are introverted, I find this question weird every time it's asked.

As long as you are building things for other people, you cannot be effective unless you engage with those people to understand the problem to be solved.  As long as you do not have perfect focus and infinite perspective, you cannot be effective unless you learn to work with other people in a close, collaborative way.

I will acknowledge that engaging with people in this way might initially feel uncomfortable.

But I'll suggest that you should be more interested in being effective than not being uncomfortable.  If  you can't make that shift, then yes, Agile is not for you.
Categories: Blogs

Agile is not sustainable?

Does Agile produce a working environment that is unsustainable?

Extreme Programming (XP) originally had an explicit practice called "40 Hour Week", which eventually became "Sustainable Pace".

Part of this was because of an intent to make software development humane; part of this was because people were interested in what was actually effective and not what just looked effective.  Working excessive, unsustainable hours is not effective.

As Ward Cunningham said, in response to a request to get a team to work more hours: "We would if it would help."

There are other practices that address sustainability:
I will acknowledge that there are some things that may seem like they are designed to increase unsustainability:
  • The use of the word "sprint"
  • Daily stand-ups
  • The continuous flow approach of kanban
  • Continuous Delivery
All the principles and practices designed for increased pace and speed can easily lead to unsustainability if you hold a particular mistaken assumption.

The mistaken assumption is that Agile encourages a one-way communication where a Product Owner tells a development team what to do rather than a n-way collaboration between everyone because that's just a better way to do software development.
Categories: Blogs

Guest Blog: The Value of Continuous Agile Retrospectives

Ben Linders - Tue, 08/04/2015 - 16:58
In this gust blog post on BenLinders.com David Horowitz, CEO and Co-Founder of Retrium, explores why you should do continuous retrospectives and how you can do them to establish continuous improvement. Continue reading →
Categories: Blogs

Run, Don't Walk, to Scale Agile

Rally Agile Blog - Tue, 08/04/2015 - 16:00

Rally customers have always been front and center at RallyON conferences — filling the audience and the speaking agenda with their experiences, knowledge, and ideas. But at this year's RallyON 2015 conference, our customers were so engaged they nearly blew up the conference app. Since their commentary did such a great job of capturing the essence of what we heard and learned at RallyON, we thought we'd share a few of the conference’s customer stories about Agile at scale — in our customers' own words.

The Seagate Story

Seagate is a leading producer of data storage solutions. To keep up in this industry, Moore’s Law alone dictates that you have to move fast. Seagate knew it needed a better way to predictably and reliably get products out the door, but it wasn’t initially familiar with Agile approaches.

At RallyON 2015, Seagate Agile coach Iky Chan and Rally Solutions Architect Andy Carlson co-presented a talk on Seagate’s journey from a waterfall shop to one that now assembles all 100+ members of its firmware group for release planning on a regular cadence.

Their story starts out describing how Seagate worked up from piloting a few Agile teams to selling leadership on the first big room planning event.

“What is big room planning? It's mid-range planning with all the people connected to a value stream. You need to get more than 7 plus or minus 2 people in a room together if you want to solve complex problems."

As the main ceremony for an Agile delivery group to successfully deliver on its release, big room planning naturally requires an investment of time, energy, and preparation. The Seagate talk did a great job detailing and demystifying the process of running a big room planning event. Chan even included a photo of her “big room planning suitcase.”

“Prep is required for a successful big room event.” "What’s in your big room planning kit? Big Post-Its, Sharpies, tape, markers, snacks . . . What else?” “Big room planning is often scary and intimidating — the first time! Each time it gets easier and easier.”

Halfway into their talk, Carlson and Chan put up a photo of several developers looking at a whiteboard full of stickies, an intimidating “wall of WiP.” 

The success of Seagate’s investment in Agile might be summed up in this quote:

“These guys used to work weekends. Since they started big room planning, not anymore. #bigroomplanning helped Seagate address bottlenecks and become more predictable.”

Several audience members noted during the Seagate story that big room planning isn’t valuable just for the work it produces, but for how it brings people together to collaborate:

“The outcome of big room planning isn't just a plan. It's a group prepared to execute on a plan.”

Chan closed the talk by exhorting the audience to “run, don’t walk” to the scaling agile booth in the lobby, where we demo’d how Rally’s new features can be used to improve your release planning.

(Read the Seagate case study or watch the video.)

The PayPal Story

When PayPal VP of Technology Kirsten Wolberg opened her keynote about PayPal’s Agile transformation, it made #BigBang a trending hashtag.

“If we didn't do a #bigbang the fear was we'd get all of the pain and none of the benefit because there were simply too many dependencies.”

As with many transformations, PayPal went all-in on change because its status quo had become untenable. A system of 15,000 people, left to its own devices, had devolved into bottlenecks and a lack of trust. There was too much effort on projects with not enough output, so the company committed to unraveling its problems and fixing them.

“Learning about the ‘why’ is important so that you can figure out the ‘what’ when it comes to transformations.” “Transformations usually are not about the development process. They are about mindset.”

Wolberg pointed out that executive support — a vital component of any transformation — isn’t the same as buy-in. Case in point: she heard,

"I trust you, but think you will fail spectacularly."

Executive support means leaders may not agree with every aspect of a transformation, but they will stand behind you to make change happen. She commented that leaders at every level need to play together for successful transformation: a mature leadership team has a willingness to look for and find problems, for example, then listen to advice about how to fix them.

As Wolberg describes in her keynote, PayPal mapped out a seven-month change agenda with four key pillars:

  1. Customer-driven innovation (CDI.) “Let’s figure out what the customer wants, not what we want.” This meant talking to customers, walking in the customer’s shoes, to understand their problems. This is in contrast to “executive-driven innovation,” which, as Wolberg said, has a greater chance of missing the mark since executives typically are not “in the same headspace” as customers.
  2. Core product operation model. PayPal implemented a product model centered around core product teams providing basic functionality (the “chassis”), with regional teams customizing for local needs. This approach was introduced to foster alignment and focus around core products (working on the right thing at the right time), while eliminating unnecessary complexity.
  3. Agile transformation. PayPal made a concerted effort to build Agile teams and ensure they had strong Agile practices. They re-organized teams — who previously had been siloed in 83 different product groups — around a vastly smaller set of value streams. In the process of assembling Scrum teams, they actually identified 200 “missing” people. And they gave teams ownership of the products, which fostered a sense of “having skin in the game.” Additionally, PayPal enlisted coaches in every region to provide back-up and support so teams could focus on delivering their work.
“Fragile (really bad agile) is a way to doom agility transformation.”

PayPal gave its Agile teams lots of flexibility, but asked every team to adhere to just two clear rules: follow a synchronized sprint cadence (start and finish sprints at the same time) and use a single, shared platform (Rally) for tracking work. There were knowing chuckles in the room as Wolberg described the fervor with which teams can get trapped in the "tool debate" — when the real challenge of an Agile transformation isn’t a tool at all, she pointed out, but resistance to change.

  1. KPIs. Key Performance Indicators are vital, explained Wolberg. Product and technology teams need to be accountable, and if you don't measure the value of your actions you won't know if they’re working. In particular, the company was keen to measure engagement — which it believes can lead to better solutions and better quality.

“KPIs help illustrate the right behaviours. Lip service isn’t possible if the right things are measured.”

There were nods around the room as Wolberg called out middle management as most resistant to transformational change.

“Permafrost, typically known as middle management, ice that never melts. Lol.”

She pointed out the importance of having open and honest conversations with development managers, to celebrate successes and identify opportunities, and called on us to use discipline in changing our own all-too-human behaviors.

“You have to hear something 7 times to remember it, and you have to do something 21 times to make it a habit. (Very true! )”

Wolberg ended her talk by summarizing some lessons learned at PayPal, and our customers in the audience had their own a-ha moments:

“The value of portfolio planning & roadmapping … Agility is a top-down mindset.” “Forming teams around the work/product is more effective than forming teams around the organisation chart.” "Fundamental agility disciplines and practices need to be built into the workflow; they are vital to moving forward well.”

Watch the PayPal keynote.

 

The Physicians Mutual Story

Physicians Mutual, an insurance company that’s been in business for more than 100 years, makes its headquarters in Omaha, Nebraska. So it seemed fitting when Physicians keynote speaker — Project Manager and Certified ScrumMaster Joan Bohannon — cited the famed Omaha businessman Warren Buffett near the beginning of her talk.

"Great quote: ‘Should you find yourself in a chronically leaking boat, energy devoted to changing vessels is likely to be more productive than energy devoted to patching leaks.’" “All hail the Oracle of Omaha!”

Like PayPal, Physicians told the story of a pilot turned #bigbang Agile transformation; but that wasn’t initially how its Enterprise Technology Group got started. The group had been using RUP and waterfall approaches before piloting Agile methods, and its first attempt at Agile adoption failed — for reasons that seemed all too familiar to some in the audience:

“The usual story ... No team, process or product integration … this is pain! Lack of integration creates bottlenecks and dependencies!” “[They] did Agile with one or two teams and one or two projects, but it didn't become sticky; Did Agile 101 training, but not role-based training." “Limited roll out = limited results.”

So the group took a deep look at how it was doing things, and decided to make some serious changes — cue Warren Buffett’s advice to get a new vessel rather than patch the leaks. At Rally we often talk about transformational change through platform, process, and people, and Physicians set out to improve all three of these areas.

Platform. Like many growing companies, Physicians had integrated various technologies in an effort to harness its growing complexity — only to end up with a vast (and complex) catalog of tools. When it overhauled its catalog and settled on Rally as its central hub, Physicians went from 23 different tools down to 5, at a savings of $300,000. “We looked at many tools,” said Bohannon, “but we quickly realized that Rally was the only organization that could provide training and coaching as well as the platform we needed.” Standardizing on a single platform had other benefits, too:

“As soon as the software got in the door, people were so excited about it that they didn’t want to wait for the pilot. that’s when we realized we should be going with a big bang approach.”

Process. "Boy, did we have process,” said Bohannon. “Click on any one of these and you'll get more process," she said, standing in a front of a slide with a spider web of interconnected people, departments, and requirements.

Physicians actually reviewed all of its legacy processes and got rid of the ones that weren’t working, leaving them “just enough process to be successful.” With the Rally platform in place Physicians can plug directly into its other four tools, further minimizing process maintenance and context-switching.

“Rally integrations streamline processing!” “I envy having five tools.”

People. Unlike the first time around, when it only did “intro to Agile” courses for some pilot teams, Physicians was keen this time to include comprehensive and role-based training — as well as Scaled Agile Framework® (SAFe®) training to help them implement and manage the transformation. “We trained everyone,” said Bohannon. “Project managers, testers, developers … and we coached some in-house coaches to provide ongoing support.”

“Very impressive investment in people!”

With the platform, process, and people changes in flight, Physicians began to use its Rally data for conversations about performance and collaboration. Rather than seeing metrics as a weapon or punishment, the increased transparency helped teams drive important discussions — “the why behind the what.”

“Early conversations allow course corrections. Late conversations can become screaming matches!” “Transparency for team health checks - what a bonus!” “Rally helps provide transparency for end to end process improvements. Love this!”

One of the early conversations at Physicians was about estimates, points, and timesheets. Physicians made an effort to move away from timesheets and learn to estimate and plan based on points, instead. At a business level, cost by points turned out to be more accurate than timesheets — whose value was questioned by many in the audience:

“Timesheets are a drag! Wow!” “??? Could this ever work for our organization ???”

Physicians uses Rally and Planview for its portfolio planning and management, and Physicians Program Manager and co-keynote speaker Thomas Hall described how its product owners and portfolio managers plan and track progress — from the team level to the program level to the portfolio level.

Hall explained that portfolio management is hard, but critical for enterprise success. A “Business Council,” comprised of senior executives, is charged with setting strategic priorities for the company. Portfolios steer the business to the highest-value priorities; programs align the development work to those business priorities; and delivery teams plan and track their work according to program direction. Feature definition is key to successful prioritization. Product owners gather as a team to review and assess. When estimates change, that information rolls back up the hierarchy so they can re-evaluate the plan.

“Business council is the way to go! Keep the enterprise focused! Realise the value stream! Love this!”

Because everyone in the group — from developers to executives — is using the same platform, they can all see progress in realtime and make necessary adjustments. Teams have access to important information about prioritization and capacity, while managers can identify risks and do more accurate planning.

“Would love to get out of Excel and PowerPoint and into Rally.” “My program managers are not looking at this sort of detail. It would really up the game. A cool view to assess the health of the backlog.”

In addition to decreasing its tool footprint and maintenance costs, Physicians’ Agile transformation has yielded other substantial benefits: the Enterprise Technology Group has increased its major releases from four to six per year, and now delivers at the end of every iteration — nearly 1,000 stories per year — without waiting for a major release. More importantly, given it’s the group that maintains the entire company’s business-critical systems, it has improved its ability to respond faster to market demands and is considered a trusted, predictable partner for the business. Customers sure loved the outcome.

“Great insights from actual practitioners!” “Big Bang approach to agile transformations hurts my head, but I'm thinking it through …” “Love the hard conversations this will evoke!”

Watch the Physicians Mutual keynote or read the case study.

Want more stories? Check out some of our other scaling Agile customer stories from RallyON, including another perspective on Physicians Mutual’s transformation, “What Pilot? Physicians Mutual's ‘Big Bang’ Approach“;  “Scaling Scaled Agile: Lessons Learned at UnitedHealth Group” and “Heartland’s Scaled Agile Journey.

 

Rally
Categories: Companies

10th Annual State of Agile Survey is Open!

Agile Management Blog - VersionOne - Tue, 08/04/2015 - 14:30

300x250It’s hard to imagine that we’re celebrating the 10th year of the State of Agile™ survey! The annual survey has become the largest, longest-running, and most comprehensive survey serving the agile community.

Last year nearly 4,000 of your peers shared what they’ve learned through their agile experiences. The survey gives agile software professionals around the world the opportunity to provide insights on a wide range of agile topics including the benefits of agile, top tips for scaling agile, how to measure agile success, and the most popular agile project management tools. This year we will be conducting an even deeper analysis of the trends over the past 10 years.

To help celebrate the 10th anniversary, VersionOne is giving away 10 Apple Watches in a random drawing in each of the 10 weeks that the survey is open. The survey, which takes about 10 minutes to complete, is open until Oct. 2. The full report will be available in early 2016.

For the past nine years, the results from the State of Agile survey have been helping organizations realize the benefits of agile faster, easier and smarter. Help the agile community make the 10th annual State of Agile survey the most valuable report yet!

State of Agile is a registered trademark of VersionOne Inc.

Categories: Companies

Who Should be Your Product Owner?

Johanna Rothman - Tue, 08/04/2015 - 14:01

In agile, we separate the Product Owner function from functional (development) management. The reason is that we want the people who can understand and evaluate the business value to articulate the business value to tell the people who understand the work’s value when to implement what. The technical folks determine how to implement the what.

Separating the when/what from how is a great separation. It allows the people who are considering the long term and customer impact of a given feature or set of features a way to rank the work. Technical teams may not realize when to release a given feature/feature set.

In my recent post, Product Manager, Product Owner, or Business Analyst?, I discussed what these different roles might deliver. Now it’s time to consider who should do the product management/product ownership roles.

If you have someone called a product manager, that person defines the product, asks the product development team(s) for features, and talks to customers. Notice the last part, the talking to customers part. This person is often out of the office. The product manager is an outward-facing job, not an internally-focused job.

The product owner works with the team to define and refine features, to replan the backlogs, and to know when it is time to release. The product owner is an inward-facing function.

(Just for completeness, the business analyst is an inward-facing function. The BA might sit with people in the business to ask, “Exactly what did you mean when you asked for this functionality? What does that mean to you?” A product owner might ask that same question.)

What happens when your product manager is your product owner? The product development team doesn’t have enough time with the product owner. Maybe the team doesn’t understand the backlog, or the release criteria, or even something about a specific story.

Sometimes, functional managers become product owners. They have the domain expertise and the ability to create a backlog and to work with the product manager when that person is available. Is this a good idea?

If the manager is not the PO for his/her team, it’s okay. I wonder how a manager can build relationships with people in his/her team and manage the problems and remove impediments that the team needs. Maybe the manager doesn’t need to manage so much and can be a PO. Maybe the product ownership job isn’t too difficult. I’m skeptical, but it could happen.

There is a real problem when a team’s manager is also the product owner. People are less likely to have a discussion and disagree with their managers, especially if the organization hasn’t moved to team compensation. In Weird Ideas That Work: How to Build a Creative Company, Sutton discusses the issue of how and when people feel comfortable challenging their managers. 

Many people do not feel comfortable challenging their managers. At all.

We want the PO and the team to be able to have that give-and-take about ranking, value, when it makes sense to do what. The PO makes the decision, and with information from the team, can take all the value into account. The PO might hear, “We can implement this feature first, and then this other feature is much easier.” Or, “If we fix these defects now, we can make these features much easier.” You want those conversations. The PO might say, “No, I want the original order” and the team will do it. The conversations are critical.

If you are a manager, considering being a PO for your team, reconsider. Your organization may have too many managers and not enough POs. That’s a problem to fix. Don’t make it difficult for your team to have honest discussions with you. Make it possible for people with the best interests of the product to have real discussions without being worried about their jobs.

(If you are struggling with the PO role, consider my Product Owner Training for Agencies. It starts next week.)

Categories: Blogs

Scrum Without Removing Impediments Isn’t Scrum

Notes from a Tool User - Mark Levison - Tue, 08/04/2015 - 13:00
Mechanical Scrum Versus True Scrum – What’s the Difference?

Speed BumpsRecently I was talking to a friend about their company’s implementation of Scrum. They don’t see the point. Before Scrum was implemented, they often had to wait an hour or more to access a test machine.  After several years of using Scrum, it’s still a problem. The same company has its test servers in another country and there are often network issues that cause the running tests to fail. These were problems before they started using Scrum, and nothing has been done to address them since.

Scrum only works when we use it to address the problems that exist in our environment, and then work to resolve them.

Consider the basic principle that a Scrum Team is required to produce high quality working software (or hardware) at the end of every Sprint. Shippable quality.

Anything that stops the Team from achieving Shippable Quality Software every Sprint is an impediment, and must be resolved.

Here are some of the common reasons that impediments don’t get resolved:

  • Definition of Done[1] doesn’t exist, isn’t honoured, or isn’t published in a truly visible place (usually the Team room wall).
  • Definition of Done isn’t frequently updated and improved until the Team is at least able to ship at the end of every sprint; preferably able to ship continuously.
  • There are no Component or Feature Teams[2]. Scrum doesn’t require Feature Teams explicitly, but it does require that the Team has a sufficient supply of skills that it can get a small vertical slice (not just one component) of software to truly done at the end of every Sprint. Component Teams, while legitimate in Scrum, increase the coordination effort required to get to done every Sprint.
  • Pressure exists to deliver more every sprint. Many Teams feel a constant pressure to deliver significantly more than they’re able to. The pressure is so intense, that they constantly take shortcuts to try to meet the demand. Scrum was created to give the doers more freedom in deciding how much work they could achieve and how they could achieve it. It only works when the Team takes a stand and makes clear their capacity. Real improvement is only possible when the Team has sufficient slack to pause, reflect, and improve.
  • Organizational impediments are not removed. For example, network issues between one office and another, coordination issues between Teams, or anything else that can’t be resolved at the Team level.
Remember, Scrum is a problem-finding tool, not a problem-solving tool.

Your organization has to solve the problems that Scrum highlights, for it to work effectively. Since Scrum doesn’t solve the problems for your organization, you’re not actually practicing Scrum if you don’t follow through and resolve the issues that it finds. Instead, you’re practicing Mechanical Scrum.

 

[1] Definition of Done – a checklist used by a Scrum Team to test if a Product Backlog Item is truly completed.
[2] Feature Teams – A feature team is “a long-lived, cross-functional, cross-component team that completes many end-to-end customer features—one by one.” Source: Larman and Vodde: http://www.featureteamTeams.org
Categories: Blogs

Scrum Without Removing Impediments Isn’t Scrum

Notes from a Tool User - Mark Levison - Tue, 08/04/2015 - 12:00
Mechanical Scrum Versus True Scrum – What’s the Difference?

Speed-BumpsRecently I was talking to a friend about their company’s implementation of Scrum. They don’t see the point. Before Scrum was implemented, they often had to wait an hour or more to access a test machine.  After several years of using Scrum, it’s still a problem. The same company has its test servers in another country and there are often network issues that cause the running tests to fail. These were problems before they started using Scrum, and nothing has been done to address them since.

Scrum only works when we use it to address the problems that exist in our environment, and then work to resolve them.

Consider the basic principle that a Scrum Team is required to produce high quality working software (or hardware) at the end of every Sprint. Shippable quality.

Anything that stops the Team from achieving Shippable Quality Software every Sprint is an impediment, and must be resolved.

Here are some of the common reasons that impediments don’t get resolved:

  • Definition of Done[1] doesn’t exist, isn’t honoured, or isn’t published in a truly visible place (usually the Team room wall).
  • Definition of Done isn’t frequently updated and improved until the Team is at least able to ship at the end of every sprint; preferably able to ship continuously.
  • There are no Component or Feature Teams[2]. Scrum doesn’t require Feature Teams explicitly, but it does require that the Team has a sufficient supply of skills that it can get a small vertical slice (not just one component) of software to truly done at the end of every Sprint. Component Teams, while legitimate in Scrum, increase the coordination effort required to get to done every Sprint.
  • Pressure exists to deliver more every sprint. Many Teams feel a constant pressure to deliver significantly more than they’re able to. The pressure is so intense, that they constantly take shortcuts to try to meet the demand. Scrum was created to give the doers more freedom in deciding how much work they could achieve and how they could achieve it. It only works when the Team takes a stand and makes clear their capacity. Real improvement is only possible when the Team has sufficient slack to pause, reflect, and improve.
  • Organizational impediments are not removed. For example, network issues between one office and another, coordination issues between Teams, or anything else that can’t be resolved at the Team level.
Remember, Scrum is a problem-finding tool, not a problem-solving tool.

Your organization has to solve the problems that Scrum highlights, for it to work effectively. Since Scrum doesn’t solve the problems for your organization, you’re not actually practicing Scrum if you don’t follow through and resolve the issues that it finds. Instead, you’re practicing Mechanical Scrum.

 

[1] Definition of Done – a checklist used by a Scrum Team to test if a Product Backlog Item is truly completed.
[2] Feature Teams – A feature team is “a long-lived, cross-functional, cross-component team that completes many end-to-end customer features—one by one.” Source: Larman and Vodde: http://www.featureteamTeams.org
Categories: Blogs

Spark: pyspark/Hadoop – py4j.protocol.Py4JJavaError: An error occurred while calling o23.load.: org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot communicate with client version 4

Mark Needham - Tue, 08/04/2015 - 08:35

I’ve been playing around with pyspark – Spark’s Python library – and I wanted to execute the following job which takes a file from my local HDFS and then counts how many times each FBI code appears using Spark SQL:

from pyspark import SparkContext
from pyspark.sql import SQLContext
 
sc = SparkContext("local", "Simple App")
sqlContext = SQLContext(sc)
 
file = "hdfs://localhost:9000/user/markneedham/Crimes_-_2001_to_present.csv"
 
sqlContext.load(source="com.databricks.spark.csv", header="true", path = file).registerTempTable("crimes")
rows = sqlContext.sql("select `FBI Code` AS fbiCode, COUNT(*) AS times FROM crimes GROUP BY `FBI Code` ORDER BY times DESC").collect()
 
for row in rows:
    print("{0} -> {1}".format(row.fbiCode, row.times))

I submitted the job and waited:

$ ./spark-1.3.0-bin-hadoop1/bin/spark-submit --driver-memory 5g --packages com.databricks:spark-csv_2.10:1.1.0 fbi_spark.py
...
Traceback (most recent call last):
  File "/Users/markneedham/projects/neo4j-spark-chicago/fbi_spark.py", line 11, in <module>
    sqlContext.load(source="com.databricks.spark.csv", header="true", path = file).registerTempTable("crimes")
  File "/Users/markneedham/projects/neo4j-spark-chicago/spark-1.3.0-bin-hadoop1/python/pyspark/sql/context.py", line 482, in load
    df = self._ssql_ctx.load(source, joptions)
  File "/Users/markneedham/projects/neo4j-spark-chicago/spark-1.3.0-bin-hadoop1/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 538, in __call__
  File "/Users/markneedham/projects/neo4j-spark-chicago/spark-1.3.0-bin-hadoop1/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py", line 300, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o23.load.
: org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot communicate with client version 4
	at org.apache.hadoop.ipc.Client.call(Client.java:1070)
	at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
	at com.sun.proxy.$Proxy7.getProtocolVersion(Unknown Source)
	at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
	at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
	at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203)
	at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
	at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:176)
	at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:208)
	at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:203)
	at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219)
	at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217)
	at scala.Option.getOrElse(Option.scala:120)
	at org.apache.spark.rdd.RDD.partitions(RDD.scala:217)
	at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:32)
	at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219)
	at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217)
	at scala.Option.getOrElse(Option.scala:120)
	at org.apache.spark.rdd.RDD.partitions(RDD.scala:217)
	at org.apache.spark.rdd.RDD.take(RDD.scala:1156)
	at org.apache.spark.rdd.RDD.first(RDD.scala:1189)
	at com.databricks.spark.csv.CsvRelation.firstLine$lzycompute(CsvRelation.scala:129)
	at com.databricks.spark.csv.CsvRelation.firstLine(CsvRelation.scala:127)
	at com.databricks.spark.csv.CsvRelation.inferSchema(CsvRelation.scala:109)
	at com.databricks.spark.csv.CsvRelation.<init>(CsvRelation.scala:62)
	at com.databricks.spark.csv.DefaultSource.createRelation(DefaultSource.scala:115)
	at com.databricks.spark.csv.DefaultSource.createRelation(DefaultSource.scala:40)
	at com.databricks.spark.csv.DefaultSource.createRelation(DefaultSource.scala:28)
	at org.apache.spark.sql.sources.ResolvedDataSource$.apply(ddl.scala:290)
	at org.apache.spark.sql.SQLContext.load(SQLContext.scala:679)
	at org.apache.spark.sql.SQLContext.load(SQLContext.scala:667)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
	at py4j.Gateway.invoke(Gateway.java:259)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:207)
	at java.lang.Thread.run(Thread.java:745)

It looks like my Hadoop Client and Server are using different versions which in fact they are! We can see from the name of the spark folder that I’m using Hadoop 1.x there and if we check the local Hadoop version we’ll notice it’s using the 2.x seris:

$ hadoop version
Hadoop 2.6.0

In this case the easiest fix is use a version of Spark that’s compiled against Hadoop 2.6 which as of now means Spark 1.4.1.

Let’s try and run our job again:

$ ./spark-1.4.1-bin-hadoop2.6/bin/spark-submit --driver-memory 5g --packages com.databricks:spark-csv_2.10:1.1.0 fbi_spark.py
 
06 -> 859197
08B -> 653575
14 -> 488212
18 -> 457782
26 -> 431316
05 -> 257310
07 -> 197404
08A -> 188964
03 -> 157706
11 -> 112675
04B -> 103961
04A -> 60344
16 -> 47279
15 -> 40361
24 -> 31809
10 -> 22467
17 -> 17555
02 -> 17008
20 -> 15190
19 -> 10878
22 -> 8847
09 -> 6358
01A -> 4830
13 -> 1561
12 -> 835
01B -> 16

And it’s working!

Categories: Blogs

Knowledge Sharing


SpiraTeam is a agile application lifecycle management (ALM) system designed specifically for methodologies such as scrum, XP and Kanban.