I’ve used a lot of different architectures, patterns and implementations that revolve around the core concept of command-query separation (CQS) and the more recent label of command-query responsibility separation (CQRS). The ideas behind these principles help us create code that targeted to a single purpose, generally side-effect free and easier to work with and maintain. In the last few days, though, I’ve begun to see how CQRS can be used for performance engineering as well.

Performance Problems With A Common Pattern

A few weeks ago, our product owner reported a performance problem with a control that is used on two screens in our handheld / Compact Framework application. This control is not terribly complicated – it has 4 drop down lists, each one loaded based on the data selected in the previous one. I’m pretty sure every developer has created a series of drop down lists like this at some point in their career. It’s not difficult… it just takes a little time and effort to handle all the cases of no items found, auto-selecting if there’s only a single item in the list, having a “Select One” or other default option, etc.

After digging into the offending control, I found that it was doing the following for every drop down list on the control:

Data Load / Display:

1. Load all data from the database into a DataTable
2. Convert each row of data into the full object it represents
3. Convert each object into a simple Name/Value Lookup object
4. Bind the Lookup objects to the drop down list

Data Select / Use (on selected index changed):

1. Get Value (ID) of the selected lookup item and load the full object for that ID
2. Run the Data Load / Display for the next drop down list based on the ID of the object
3. Publish the selected object on an event so the parent form could respond to it as needed

This is a pattern that I see a lot of – whether it’s WinForms or WebForms development. It’s especially common in a WebForms environment, though, where there is not state on the view implementation. Unfortunately, this pattern and implementation is very problematic when it comes to performance. The actual performance on the control in question was so bad that we resorted to using asynchronous commands to retrieve the data for the drop down lists. This let us keep the UI “responsive” to the user – it prevented the screen from locking up with strange artifacts for the 3 to 5 seconds that it took to load any given drop down list.

 

Separation Of Concerns

Why would I want to load the entire set of data from the database and deserialize that into the full object model just so I can bind the name and id of the objects to a drop down list and then re-load the same object from the database again? That doesn’t make much sense to me – even in a web environment where I should bind nothing more than the name and id in the form. In a WinForms environment, though, I guess I can see “the easy way out” by loading up the objects with my existing data access infrastructure… but that just doesn’t make any sense other than being lazy.

Here’s the crux of a read-only or view model in this situation: if I’m only going to display the name and id of the objects, then that’s all I should load.

 

Load View Model, Lazy Load Full Object When Its Needed

To solve the performance problems in this control, I decided to use the basic CQRS tenants of separating my view model, which is a read-only representation of my data, from the object model which is a read/write representation. Here’s the new approach I took to solve the performance problems, with each of the drop down lists:

Data Load / Display

1. Load the name and id only, from the database using a DataReader
2. Populate a generic Lookup object with the name / id of each record
3. Bind the drop down list to the Lookup objects

Data Select / Use

1. Get the the id of the selected item in the drop down list
2. Run the Data Load / Display for the next drop down list based on the id of the selected item

Data Collection

1. After the entire selection process has been performed, then and only then load the full object that was selected and publish it to the parent form

There are a couple of key things to note in this solution… namely, I’m only loading the name and id for the drop down lists. I only need that information for the drop down list to work, so I’m not going to bother loading anything else. And I’m not loading the full object model until I’m actually ready to use it. If the user is constantly switching the drop down lists to figure out what they need, then loading the full object model after each individual selection will just use up a bunch of time and resources for no good reason. I’m waiting until some level of confidence in the selection can be established and the code is ready to use the object model before loading the full model.

 

The Performance Improvements

I don’t have any scientific performance metrics for this, yet. I’m not sure if I’ll need to do that, actually. I do have first hand experience with the existing performance and the new performance, though.

The original code tended to take anywhere from 3 to 5 seconds, on average, to load any given drop down list. The worst performance, though, was one particular query that returned nearly a thousand items for the drop list to display. This would take closer to 6 or 8 seconds to load. … again, these are all based on my experiences, not actual timers… I can say with certainty, though, that I was never able to use keypad up/down arrows to select items in the drop down list. The control was simply too slow in responding so I would sit there and wait for it to finish loading before clicking the down arrow again.

With the new implementation in place, the control’s performance is significantly enhanced. The average time it takes to load the drop down list has dropped to far below a second. Again, I haven’t done any real timer / performance testing with this… but I can say with certainty that I can now use the up/down arrow keys on the keypad and the control keeps up with me no matter how fast I’m able to click the keys. Furthermore, the performance is good enough that I have not yet needed to use any asynchronous processing to load or display any data. Even with the one query that returns nearly a thousand records to the drop list, the time to load is less than a second – a barely noticeable stutter in the list being available for selection.

 

Conclusions And Other Considerations

The principles and patterns that comprise CQRS can be used for a number of different reasons – not the least of which is performance improvements in your code. Whether you are working on Winforms, Webforms, Compact Framework or another system or platform that has read vs. read/write needs, keeping CQRS in mind at all levels of the system can have a significant impact in many different ways.

Of course, this does not come free. There is an increase in the amount of code you have to maintain when you go down this path. You may end up writing two or more different types of data access code and you will have the same data represented in multiple objects and queries in your system. These costs are not to be taken lightly. However, when used judiciously and understood by the entire team the impact of these costs can be mitigated. Keep your data access methods simple and have a clean separation between your full object model and your read only models. Constantly communication with team members and work on well named and organized code. Its your team’s communication, collaboration and standards that will help to cut the costs, keep your system clean and maintain it’s performance over time.

I do most of my UI development – in ASP.NET WebForms and in WinForms – with a Model-View-Presenter setup. It helps me keep my application logic separate from my view implementations, makes it possible to unit test the presenters, etc. I also like to use custom controls – often with their own presenter - to help encapsulate UI related process and keep my UI implementations clean. The challenge with custom controls is getting them to converse to each other and getting the parent form to converse with the controls. My favorite way of solving this challenge is through simple messaging patterns. This gives you a lot of control and ensures your system is nice and decoupled. Of course, there is a cost/benefit tradeoff that needs to be considered. There may not need the indirection and potential complexities that come along with those solutions. The system in question may not need a messaging system, event aggregator, command pattern or whatever else. There are times when its easier and makes more sense to forego these patterns and have the presenters talk directly to each other.

 

Role Specific Interfaces

When the cost of the messaging pattern architecture out-weighs the benefits, stick to simple abstractions that still  keep the presenters decoupled by one layer. This can easily be done with an interface or abstract base class in static languages like C#, Java and C++. However, don’t take the easy way out in this abstraction and creating a one-to-one mapping between the abstraction and the implementation. Doing so will create a semantic coupling between the two presenters.

For example, the IProductCodeSelectionPresenter may have the following definition:

   1: public interface IProductCodeSelectionPresenter

   2: {

   3:   void Initialize();

   4:   void ProductCodeSelected(ProductCode code);

   5:   ProductCode GetSelectedProductCode();

   6:   void SelectionCancelled();

   7:   void SelectionConfirmed();

   8: }

   9:  

Which of these methods should another presenter call in order to retrieve the ProductCode? Should GetSelectedProductCode be called? Does this method guarantee the view to select a product code was run and that the product code has been specified by the user? Or maybe the ProductCodeSelected method should be called instead, or Initialize or … This easy-to-create interface may cause semantic coupling by forcing another developer to look at the implementation in order to know which methods should be called, when.

It would be better to define a role that the presenter is playing in the communication and create an interface that is specific to that role. In this situation, the name of the presenter provides some insight to what role the presenter is playing - product code selection. A simple role specific interface for this presenter may look like this:

   1: public interface IProductCodeSelector

   2: {

   3:   ProductCode GetProductCode();

   4: }

With an interface defined like this, a developer calling this code will not have any confusion on what needs to be called. There is no need to look at the implementation of the interface, and semantic coupling has been avoided. Making a call to this interface is easy.

 

The Interface Segregation Principle (ISP)

The driving principle in making the decision to create the role specific interface is often the Interface Segregation Principle (ISP). This principles says that we should not force a client – the code that is calling out to our interface – to know about methods and properties that it does not need.

In this case, the client code does not need to know that the interface sits on top of a presenter. Therefore, take the name “presenter” out of the interface that the client calls. This gives the interface more flexibility for the future and prevents the client code from knowing that a view and user input is likely to be the implementation. The client code also doesn’t need to know about the Initialize, ProductCodeSelected and other methods that the presenter has. These methods are specific to the interactions between the View implementation and the Presenter – a different role that the presenter is playing. By removing these methods from the interface, the client code is no longer bound to the knowledge of which methods should and should not be called, when.

 

The Dependency Inversion Principle (DIP)

The Dependency Inversion Principle (DIP) may also be at play in this scenario. DIP is not just about creating an abstraction and passing it into a constructor. That would only be dependency abstraction and dependency injection. Rather, DIP talks about abstraction ownership. In the case of a role specific interface, the owner of that interface is the code that depends on it – the client code that calls out to it.

If another presenter, such as a ProductDefinitionPresenter, is the driving force behind the need to create the IProduceCodeSelector interface, then this presenter should own that abstraction. This means that the ProductDefinitionPresenter determines what that interface looks like. What methods and properties are available, and the name of the interface are all driven by the needs of the ProductDefinitionPresenter.

   1: public class ProductDefinitionPresenter

   2: {

   3:   private IProductCodeSelector ProductCodeSelector;

   4:  

   5:   public ProductDefinitionPresenter(IProductCodeSelector productCodeSelector)

   6:   {

   7:     ProductCodeSelector = productCodeSelector;

   8:   }

   9:  

  10:   public void SelectProductCode()

  11:   {

  12:     var productCode = ProductCodeSelector.GetProductCode();

  13:     //do something with the productCode, here

  14:   }

  15: }

There is not syntax or markup that declares ProductDefinitionPresenter as the owner of this interface, in this example. That responsibility is left to the standards, conventions and organizational means of the system in question and the team that maintains it.

 

Other Considerations

Model-View-Presenter scenarios are not the only place that roles need to be considered. Any time two or more objects interact and there is a need for them to be decoupled, the roles that the objects are playing need to be considered. There are likely other principles and patterns that come into play when considering a role specific interface, as well. Each scenario’s needs must be considered for their own reasons, and ISP and DIP may not always be at play when defining an interface for an object. And role specific interfaces are not always needed. There are other benefits to creating interfaces or other abstractions that can be referenced in place of concrete implementations such as dependency injection, general decoupling, creating service layer or other context specific barriers, etc.

Lately I’ve been trying to return IEnumerable<T> whenever I need a collection that will only be enumerated or databound to something. This prevents me from making changes to the collection outside the context of the collection’s parent entity. The problem with doing this is that I might need to write a unit test that looks for a specific item in the collection, checks the count of the collection or otherwise needs to do something that the IEnumerable<T> interface doesn’t provide.

With tools like Resharper, It’s easy to change the return types of the methods that you’re getting the collection from and use an IList<T> or some other collection type that allows you to get at the information I want. However, this can lead to broken encapsulation and other potential problems in code. After all, I wanted to keep the collection encapsulated within the parent entity which is why I chose to use the IEnumerable<T> in the first place.

The good news is that there’s a super simple solution to this situation that does not require changing the IEnumerable<T> return type. Have your test code wrap the IEnumerable<T> in an IList<T>.

   1: IEnumerable<MyObject> myEnumerator = someService.GetStuff();

   2: var myCollection = new List<MyObject>(myEnumerator);

   3:  

   4: [Test]

   5: public void my_test()

   6: {

   7:   myCollection.Count.ShouldBe(1);

   8:   myCollection[0].ShouldEqual(myObject);

   9:   //etc.

  10: }

 

If you’re doing interaction testing with an interface and a mock object, where the interface receives an IEnumerable<T>, you can still use this trick. For example, if I have this method on an interface defintion:

   1: void ShowProductCodes(IEnumerable<Lookup> productCodes);

I can grab the output of this method via a stub and convert it to an IList<T>. Here’s one way to do it via RhinoMocks:

   1: var view = Mock<IAssetClassificationView>();

   2: view.Stub(v => v.ShowProductCodes(Arg<IEnumerable<Lookup>>.Is.Anything))

   3:     .Callback((IEnumerable<Lookup> lookups) =>

   4:     {

   5:         DisplayedProductCodes = new List<Lookup>(lookups);

   6:         return true;

   7:     });

   8: return view;

Line 5 wraps up the IEnumerable<Lookup> into an IList<Lookup> object, letting me test the contents/count/etc on the collection.

Now you never need to worry about whether you can test the IEnumerable<T> when you are passing it around in your code. Just wrap it in an IList<T> at test time and call your tests the way you need to.

I’ve been working in Ruby for my Albacore project over the last 6 or 8 months, and taking every chance I can find to learn how to really use the language effectively. One of the benefits I’m seeing in a dynamic language like Ruby is the ability to really DRY up your code through it’s dynamic/duck type system, and through metaprogramming.

I’ve noticed in my ruby code that I tend to see repeated patterns of implementation in a different light. Rather than seeing the things that make each repetition of the pattern different, I tend to see the things that make each repetition of the pattern the same. I notice the same structure used with different variable name, the same method calls used with different parameters, and context specific method calls as the outliers that made me duplicate the code in the first place. When I see these patterns, my mind begins to run down the path of “this code is duplicated… how can I eliminate that duplication?” Whereas in C#, I almost immediately see the differences as “these are different calls based on the context and I can’t eliminate this repeated pattern of code because of the unique calls each has to make.”

I’m not sure why my mind has been operating this way with C#, but I know that is has been doing this for a very long time. I’ve often written the same pattern of code 6 or 8 times, or more in some cases – especially when it comes to UI code and event handlers from UI controls. I wrote a prime example of repeated patterns in C# just today, on a UI that has 4 ComboBox controls on it. Each combobox has a SelectedIndexChanged event handler that gets the selected value and pushes it to the presenter via a presenter method that is specific to the value being pushed. Here’s the code in all it’s glorious duplication:

   1: void cboTypes_SelectedIndexChanged(object sender, EventArgs e)

   2: {

   3:     TeardownComboBoxEvents();

   4:  

   5:     var lookup = cboTypes.SelectedItem as Lookup;

   6:     if (lookup != null)

   7:     {

   8:         _presenter.TypeSelected(lookup);

   9:     }

  10:  

  11:     SetupComboBoxEvents();

  12: }

  13:  

  14: void cboGroups_SelectedIndexChanged(object sender, EventArgs e)

  15: {

  16:     TeardownComboBoxEvents();

  17:  

  18:     var lookup = cboGroups.SelectedItem as Lookup;

  19:     if (lookup != null)

  20:     {

  21:         _presenter.GroupSelected(lookup);

  22:     }

  23:  

  24:     SetupComboBoxEvents();

  25: }

  26:  

  27: void cboCategories_SelectedIndexChanged(object sender, EventArgs e)

  28: {

  29:     TeardownComboBoxEvents();

  30:  

  31:     var lookup = cboCategories.SelectedItem as Lookup;

  32:     if (lookup != null)

  33:     {

  34:         _presenter.CategorySelected(lookup);

  35:     }

  36:  

  37:     SetupComboBoxEvents();

  38: }

  39:  

  40: void cboCodes_SelectedIndexChanged(object sender, EventArgs e)

  41: {

  42:     TeardownComboBoxEvents();

  43:  

  44:     var lookup = cboCodes.SelectedItem as Lookup;

  45:     if (lookup != null)

  46:     {

  47:         _presenter.CodeSelected(lookup);

  48:     }

  49:  

  50:     SetupComboBoxEvents();

  51: }

When I wrote this code and looked back at it, I had my usual feeling of “well, these presenter calls are specific the the context of the combox being selected, so I can’t really do anything to eliminate this repeated pattern of code.” I even went so far as to think “man, if this were Ruby, I wouldn’t have any issue killing this repeated pattern.”  That’s when a little voice in the back of my head started shouting at me and I realized that I could eliminate the duplication in C# just as easily as I could in Ruby with the use of anonymous delegates.

 

Method Blocks And Anonymous Delegates

One of the techniques I often use in Ruby to help dry up repeated code is ruby’s method blocks - basically an anonymous delegate in C#. These two code samples are functionality equivalent…

Ruby Method Block With Named Parameter

   1: def my_method(&block)

   2:   name = "derick"

   3:   block.call(name) unless block.nil?

   4: end

   5:  

   6: my_method do |name|

   7:   puts "the name is: #{name}"

   8: end

C# Anonymous Delegate (Lambda) With Named Parameter

   1: public void MyMethod(Action<string> block)

   2: {

   3:   string name = "derick";

   4:   if (block != null)

   5:     block(name);

   6: }

   7:  

   8: MyMethod(name => {

   9:   Console.WriteLine("the name is: " + name);

  10: });

 

Eliminating This Repeated Pattern

After I finally decided to listen to that little voice shouting at me and use my tools to their full extent, I rewrote the event handlers into the following code, using an Action delegate and anonymous lambda block to execute the context specific presenter calls.

   1: private void LookupSelected(ComboBox comboBox, Action<Lookup> presenterCall)

   2: {

   3:     TeardownComboBoxEvents();

   4:  

   5:     var lookup = comboBox.SelectedItem as Lookup;

   6:     if (lookup != null)

   7:     {

   8:         presenterCall(lookup);

   9:     }

  10:  

  11:     SetupComboBoxEvents();

  12: }

  13:  

  14: void cboTypes_SelectedIndexChanged(object sender, EventArgs e)

  15: {

  16:     LookupSelected(cboTypes, l => _presenter.AssetTypeSelected(l));

  17: }

  18:  

  19: void cboGroups_SelectedIndexChanged(object sender, EventArgs e)

  20: {

  21:     LookupSelected(cboGroups, l => _presenter.GroupSelected(l));

  22: }

  23:  

  24: void cboCategories_SelectedIndexChanged(object sender, EventArgs e)

  25: {

  26:     LookupSelected(cboCategories, l => _presenter.CategorySelected(l));

  27: }

  28:  

  29: void cboGroups_SelectedIndexChanged(object sender, EventArgs e)

  30: {

  31:     LookupSelected(cboGroups, l => _presenter.GroupSelected(l));

  32: }

 

Lessons Learned

This certainly isn’t anything extraordinary, mind you. I’ve written methods with delegates and lambda blocks more often than I can remember. This code is not complex, it’s not difficult to write, it’s not difficult to read or understand. But that’s the beauty of it. It’s simple, elegant, and eliminates the repeated pattern that I was creating. There are probably some additional tweaks I could make, honestly, but I also want to keep in mind the readability and understandability of the code – not just how often a pattern is repeated.

The significance of this is not in the code that I wound up writing, but in how I came to that decision. My exposure to ruby and my predisposition to see repeated patterns of code in ruby as duplication that should be eliminated finally made a jump across the neuro-pathways of my brain into C# land. I was able to take a paradigm from a different language and different set of optimizations and capabilities, and redefine my own understanding of the current paradigms and capabilities of this situation. That kind of cross-breading and transfer of knowledge is critical to our ability to come up with new and creative solutions in situations where we believe we already have mastery.

Do yourself a favor – learn a new paradigm of development or whatever your job entails. You’ll never truly be able to say “use the right tool for the job” unless you actually know how to use the tools available, and you never know when the paradigms of one tool will cross the boundaries of your experience and begin to show you new solutions to existing problems.

I’m writing a spec with a mock object that mock object returns data to the class under test. In these situations, I don’t bother asserting that my mock object’s method was called because I know that if it’s not called the data I need in the class under test won’t be there and I’ll end up having other unexpected behavior. This falls under the general guideline of ‘test output and interactions, not input’.

In this specific situation, I am taking a value from user input and using that value to load up some data from a mock repository. I find myself wondering if I should specify the value that is being passed into the mock object’s method so that the mock will only return the data I need if the method is called with the right value. To illustrate in code, here are the two different ways I could do this.

1. Always return the data from the stubbed method call:

   1: private IAssetGroupRepository GetAssetGroupRepository()

   2: {

   3:     var repo = Mock<IAssetGroupRepository>();

   4:     AssetGroups = new List<Lookup>();

   5:     repo.Stub(r => r.GetGroupLookups(Arg<int>.Is.Anything)).Return(AssetGroups);

   6:     return repo;

   7: }

2. Only return the data from the stubbed method when the right argument is found:

   1: protected override IAssetGroupRepository GetAssetGroupRepository()

   2: {

   3:     var repo = Mock<IAssetGroupRepository>();

   4:     AssetGroups = new List<Lookup>();

   5:     repo.Stub(r => r.GetGroupLookups(1)).Return(AssetGroups);

   6:     return repo;

   7: }

The difference is on line 5 – the use of Is.Anything vs. a literal value of 1. It seems that the arguments should be specified when it matters what the arguments are… when the arguments are going to determine whether or not the right thing is being done. In this situation, it seems to me that the argument is important. If I’m not specifying the value that was selected when calling the GetGroupLookups, then my code has failed to account for the user’s input and it will likely produce the wrong behavior. The counterpoint to this is that the test where this stub definition lives becomes a little more brittle.

So the question is when should I return the data no matter what arguments are used vs. when should I only return the data when the right arguments are used? I know the answer is “it depends”, as that’s the only valid answer to any code question. :) But I’m looking for some input from the rest of the world on when they do / don’t require the right arguments and why.

Given my recent experiences with Ruby, my cursory knowledge of Java, and my past experiences with other object oriented systems, I find myself asking a lot of questions about why we do things the way we do in the C#/.NET space. Today’s questioning is about one of those fundamental things that I have been preaching for a long time, yet suddenly find myself unable to answer ‘why’:

field vs property: does it really matter?

As I sit and think about in the back of my head while writing code or this post, I can’t really say that I have any good reason for saying “you should use a property instead of a field” other than the defacto answer of encapsulation. But what if encapsulation doesn’t matter when you just want to store and retrieve a simple piece of data? Why is oh so important to use a property as the public API for a class?

Ruby doesn’t really have properties. It just has methods that allow you to use an = sign, but those methods are not even required to “set” a value. It’s only the conventional use of = methods that say you should. Java doesn’t have properties at all. It’s convention to use getWhatever and setWhatever methods, but it’s better design to not use mutators and I like the general reasons behind that.

So why does it matter if it’s a field vs. a property? Someone convince me that it’s important. Someone convince me that it’s not.  Better yet – someone explain the contexts in which it is important and the contexts in which it’s not, and someone point out where it’s important to hide the data behind the process through a mutator-less API of service methods.

…

Question everything – especially your own assumptions.

Of course that statement on it’s own can obviously be shown to be fallacy. When you consider the context of continuous improvement, learning or generally advancing our own capabilities and understanding, though, this statement can be quit liberating. Why? Because without failure, you are not learning anything.

Let’s say you are faced with a problem, a challenge, or a need that you are trying to fulfill. If your first attempt and creating a solution is successful you have learned nothing. You already knew how to provide the solution. it only took some thinking to analyze the situation and apply your existing knowledge. If, however, you stretch yourself as far as you can, put every last effort that you currently have into the solution and employ all of your existing knowledge, capabilities and resources but you still fail the first time, fail again and fail some more before finding a solution, then you are learning. You are stepping outside of your own knowledge and capabilities, learning new things and gaining new insight and experience that leads to new solutions for situations that you did not know how to previously solve.

Consider any endeavor to learn, from this light. Whether it’s adopting a new process or methodology such as an agile process or lean toolkit, using a new tool or technology, contributing to an open source project, or learning how to ride a bicycle - having the correct mentality, that failure is required, will undoubtedly help you succeed.

Mike Rother said it quite well in his book, Toyota Kata (p.138-139), when faced with people who are merely capitulating because they were told to do so, or because they want to prove that some change or new way of doing things won’t work:

Eventually it dawned on me how to deal with this question. Now, when arms fold up and people say, “Let’s see if this will work,” I say, “I can save you the time. We already know it probably won’t work. Despite our best efforts to plan this, we know that within a short time there will be ‘charred and glowing pieces’ lying around. We just don’t know in advance when, where, or why it will fail.”

Think about the last time you delivered any fragment of functionality to a customer or customer representative for feedback. Did you expect that they would be 100% satisfied and would accept it as is? Not if you were asking for feedback. When the customer responds with changes or updates that they would like, you have effectively failed. Hopefully you have failed within a very short cycle, though, and are able to incorporate the feedback of the customer into the next delivery or demonstration.

They key is not learning from mistakes and failures. The key is failing fast, failing cheap, and responding to those failures in a timely manner so that you can learn quickly and still reach your objectives. Failure is critical to success and learning, and short feedback cycles are critical to the effective use of failure as a learning tool.

Branching and merging are never free operations. Even if you are using a source control system that makes the mechanical process of branching and merging negligible, there are other costs that need to be accounted for than just the button clicks or commands that are required for a developer to commit changes.

If you were to do your current work directly in the main source line, what impact would that have? Would it be possible for your work to be committed in an incomplete or unstable state? Could you possibly prevent another team member from continuing their work because you accidentally created a problem that broke the build or otherwise hampered their efforts? Would you be introducing an immediate need for downstream team members, such as testers or business analysts to do reviews and testing, when those team members may not be immediately available?

Of course there will be times when the cost of branching and merging is far greater than changing the main line directly. For example, you may need to change a label on a form, or update an email address for logging purposes. Simple changes such as this may not warrant a branch, but there is no hard and fast rule on this. Any change to the source code has the possibility of creating more work downstream. You can’t look at the source code only, when calculating the cost of working directly in the main code line. There are always other considerations and other possible side effects for your team and it’s processes. 

 

The Mechanical Cost

There is, at minimum, an inherent cost in branching and merging found in the time that it takes to create the branch, switch to the new working branch, merge the branch to it’s target location and test the merge to ensure nothing broke. This cost is highly variable depending on the source control system that you are using, though.

With centralized systems where the actual branching takes place on the server, the cost may be fairly high. For example, with Subversion it only takes a moment to create a branch but it may take many minutes (I’ve seen a few hours for some repositories over slow connections) to switch your working branch and to do the merge. With distributed versioning systems (DVCS) like Git or Mercurial, though, the cost of branching and merging may be significantly lower. In fact, a DVCS is by it’s nature a branching scheme in it’s own way. You are never working directly against the master repository or code line. Rather, you are working against a local repository. (This is one of the reasons that working with git + svn is easier than working with svn alone.) And there are some systems that just make branching very painful and difficult. Visual SourceSafe being the prime example of that, and from what I hear (I have never used, though), Team Foundation Server is not much better at branching and merging than VSS was Understanding the cost is important, though. It helps us to know when we should be using a branching strategy.

 

The Management / Intellectual Cost

The more branching you have in your system, the more management it takes to keep track of them. When you start working with sub-branches, it becomes even more expensive to track them.

Every branch has a source line that it came from. In most cases, this source line will be the target of the merge as well. However, remembering where each branch came from so that you can merge it back to the correct location can be difficult. A good source control system will offer tools that help you understand where a branch came from, though. Subversion has a commit log that will tell you when and where a branch was made, and there are other tools available that can visualize where the branches came from. Git also has tools to visualize and otherwise represent the branching built right in, such as “gitk”. These tools can help you understand where a branch came from and where it went. They can also help you to know where it should go when you are making the decisions to merge. However, these tools don’t come free from intellectual cost. You must understand how to read the information that the tools provide, and you have to make intelligent decisions based on the information you are seeing.

As you begin to use sub-branches or hub-and-spoke branching, the cost of understanding where these branches came from and where they need to go increases. Better tools will make this easier to process in your mind, but they will not supplant the need to understand.

 

The Stability And Testing Cost

How and when your system is tested has a tremendous impact on the cost of branching and merging.

If testing is done after the software has been built, as a distinct phase before release, then the impact on branching is minimal. Whether or not you branch your code to maintain stability during change, the stability of the system will not be known until after the system has been completely integrated and shipped to the testers. (While this may sound beneficial based only on these points, the long term cost of a test after the build strategy is far more than a test as you go strategy. But that’s another story…)

If testing is done as soon as possible, and done manually – that is, tests plans are written and executed by testers that are using the software directly – then the cost of every branch and merge can be enormous. Imagine having an enterprise logistics and maintenance system with 5+ years of functionality baked into it, and only having manual test plans for this system. If it takes a team of testers 2 weeks to exercise the entire suite of tests, then every branch and merge operation could require up to 2 weeks of testing to ensure nothing was broken. Now imagine working with a large team of developers and 10 or 15 active branches, each taking up to 2 weeks to test. Stability of the system is going to be much higher than a test at the end strategy. However, the cost is quickly rocketing out of the solar system, let alone the realm of possibility. (Note that I am not saying “manual testing is bad.” You need manual testing at some point. It will find things that test automation cannot find. How much manual vs. test automation your system needs is entirely contextual – there is no single correct answer for that.)

If testing is done as soon as possible with test automation either immediately after the changes are made – or better yet, with while changes are being made – then the cost of branching can be balanced with the stability of the system with greater ease. If the entire suite of automated tests takes 1 hour to run, then the cost of every branch and merge operation is potentially 1 hour plus the time it takes to do manual, exploratory testing around the changes. This doesn’t mean that the testing cycles are free, though. If the change requires 2 testers to work 4 hours each, then that is the cost of testing – whether or not the change was made on a branch or directly into the main source line. (Note that the long term cost/benefit of automated testing is almost always better than other testing strategies. There are exceptions to this, of course. Once again, that’s another story…)

 

The Cost Of Merging Too Much, Too Soon

When a branch is merged – whether into the main source line or another branch – you are effectively coupling the merged branches together. This can be useful at times, and can be detrimental at times. In one team I worked with a while back, we had a policy of merging branches together when they had been tested by the development staff and were ready to go on to testing by the business analyst / testing staff. Once the branches were merged, we did not want to un-merge them or go back to the originating branches because we did not want to introduce yet another variable in an already confusing process. The effect of this policy was to have multiple stories that would travel between “ready for test”, “tested” and/or bug fixes at the same time. At one point, we had nearly 20 stories and defects that were travelling between various steps at the same time. The effect was essentially a waterfall like process where we handed off large amounts of work and re-work to the next step. It was a hard lesson for us, but taught us the value of keeping our branches independent of each other for as long as possible.

So, saying that a branch should be merged when it is “done, done” is saying that the branch should live independently for as long as it can without impeding the flow of work to the next part of your process. How and when work flows to the next part of your process is a policy that you and your team decide within the confines and contexts of your company, customer, and project.

 

Other Costs And Consideration

As I said in my first Branching Strategies post, there are many aspects of a system to be considered when branching your code, like a team’s organization, the system’s architecture, etc. Understanding your options, the context in which you are working, the problems you are facing, and the cost of all this will help you choose the right option at the right time. But, while the cost-benefit ratio is often a driving factor, it should not be the only factor that we consider.

At a very high level, all branching strategies have the same core policies: create the branch when you are confident that the cost of branching and merging is less than the cost of committing to the main source line, and merge when you are “done, done” making changes in that branch. There are many branching strategies that are very useful. You can branch per iteration or sprint, branch per sub-team, branch-per-release, branch per feature or defect, etc. Most likely, though, you will find yourself in scenarios where a policy of a single branching strategy is not feasible. The real world is full of fun surprises and little nuances that can make a single strategy good, bad or otherwise – all within the same project.

Having multiple branching strategies available in your toolbox will undoubtedly create new opportunities to sustain or improve the performance of your team. For example, my current team tends to run on a branch per iteration basis. However, we have recently found ourselves using a feature branch in our centralized source control system. Additionally, many of the individual team members run local branches for their day to day development work, synchronizing with the centralized server only when necessary. This emergent set of strategies allows us to solve some rather complex problems that are found in our environment, rather easily.

 

Branching Strategies

Here are some of the branching strategies that I’ve used in the last few years. I know there are many others out there and I can almost guarantee that I’ll end up using more than just this list at some point. I don’t intend this to be an exhaustive list of branching strategies – only an introductory list. For a much more in-depth discussion of other branching strategies check out Brad Appleton’s “Branching Patterns for Parallel Software Development”.

 

Branch Per Feature

If you’re reading this, you’re probably familiar with branching by feature. :) If not, you may want to head back to the beginning of my series to get a complete understanding of what feature branches are, why, when and how to use them. As a quick review, though, a feature branch is exactly what is sounds like. A branch is created for a feature to be developed from the start of work on that feature, to it’s completion. Once the feature is ready to go you merge it back into the main source line or into another branch for continued work. Multiple feature teams can work in parallel without immediately affecting each other. When FeatureTeamA completes their work and merges back into the main source line, FeatureTeamB is then required to merge the changes into their branch so that they have the correct foundation of code to work from.

 

Branch Per Iteration/Sprint

Branching per iteration or sprint is used to separate the individual iterations from each other. This allows you to have potential release candidates in one branch while continuing to work on new features and development efforts in other branches. You can either create a branch at the beginning of the iteration and work out of there, entirely, or you can work out of the trunk throughout the iteration and branch it at the end.

If your customer or other stakeholders are not immediately available for review or demonstration of the iteration’s features, then an iteration branching strategy can be very beneficial. You can continue working on the next set of features and functionality while you wait for the customer availability. Once they have reviewed the work and if they suggest any changes, those changes can either be incorporated into the original iteration branch as a potential release, or can be worked into the current or future iterations.

 

Branch Per Team/Sub-Team (Hub And Spoke)

If you’re working on a team that has multiple projects, feature groups, or other separated areas of functionality, you can divide the work into teams and sub-teams and have each team working in their own branch. This can be done with feature branches, project branches, functional areas, or any other line of division in the code. The important factor here is that the branch is maintained by multiple developers who may, in turn, have their own sub-branches during development efforts. The effect of this branching strategy is that you will have a multi-tiered hub-and-spoke layout in your source control system. There will be a hub where multiple sub-teams synchronize their own changes, and each hub will synchronize to a further upstream hub.

If your team is using feature breakdown and rollup techniques that allow individual stories to be implemented independently, you can merge to a hub and show the current state of a feature to a customer. For example, if FeatureA has Story1, Story2 and Story3, you may be able to merge Story1 and Story2 into a “FeatureA” branch when they are done. Let the customers and/or other stakeholders see the current progress of the feature, while Story3 is still being worked on. Then when the Story3 is ready, it can be merge into the FeatureA branch for final testing, and the FeatureA branch can be merged into the main source line when it is ready for delivery.

Distributed versioning systems like Mercurial and Git make this very easy to do. Chances are, if you are using a distributed system then you are working in a manner that is similar to this already. It’s a very natural branching schema for distributed source control systems. Centralized source control, on the other hand, requires more work and more coordination for this to be effective. It can be done, though. For example, with Subversion you may find that you need to create sub-folders in your primary branches folder. Each of these folders may represent a team, and would contain multiple branches – one per sub-team (or per person).

 

Merging

Whatever branching strategy you decide to use, the merging strategy tends to follow right behind. For example, a branch per feature strategy typically has merges done along feature completion lines. That is, when a feature is started, a branch is made and when that feature is done, the branch is merged back in to the main source line. This isn’t always the case, though. There are some good reasons for decoupling the branching strategies from the merging strategies, allowing your team to react to the real world and it’s every changing landscape. Your team’s definition of “done” will have a direct impact on when you should merge your code. Feature or bug-fix aggregation can be done to simplify testing. Inter-branch dependencies can cause problems that may solved by merging branches together.

In a continuous deployment environment, “done” means that is has been thoroughly tested by all automated and human testers, verified by the stakeholders, demonstrated to potential end users, etc. etc. It is production worthy, and ready to be shipped. In other environments – like many of the teams that I have worked with in the last few years - “done” means something much earlier in the lifecycle of a feature or piece of functionality. For example, a team I worked with in 2009 had a “done” definition that stopped just prior to demonstration and formal testing by our customer. Another team I worked with defined “done” as “ready for the test lab”. And still other teams have other definitions that match their specific circumstances.

There are still other reasons why you might not merge along the same lines as you originally branched.

 

Other Strategies And Considerations

There are many aspects of a system to be considered when branching your code – more than what I’ve stated here. You need to consider a team’s organization, the system’s architecture, etc. These, together with the rest of the circumstances that create the context in which you work, will play into which branching strategies you decide to use, when. In the end, the goal is to provide a sandbox for the context in which code is being written. Understanding the available options, when each option is best suited to the situation at hand and the cost of these options will help you in deciding how and when to branch.

I had a fun conversation over instant messenger yesterday. The original subject matter revolved around a problem that the other person was having and how that problem was consuming a number of minutes per day with repetitive, tedious work. At the start of the conversation the focus was on a specific solution implementation and some general direction on how to get there. After a while, though, we started talking about the complexities of the proposed solution and potential high cost. At that point the other person mentioned that they probably don’t have the ROI to justify working on the solution during business hours.

The mentioning of ROI in this manner made me stop and re-think the conversation a little… it made me ponder how we are using ROI in situations like this and how it’s a flawed approach. It seems that we often want to jump straight to the solutions and then back ourselves down to a reasonable cost so that we can make some arbitrary ROI sound good.. Rather than backing an assumed solution that we want into a cost, though, what we should really be doing is determining the conditions that a solution must meet – including any specific timeframe and budget – and then find solutions that fit within those constraints.

To help illustrate this, here’s the rest of the conversation. I’ve consolidated a few of the times and removed a few lines, but the remaining text is from the conversation log, directly.

me: on the subject of ROI... there's an interesting perspective change that you can use to help solve these types of problems. rather than using ROI to determine whether or not you can take on a task like this, use it to determine what the right solution for the problem is

me: you obviously have a problem. we've established that. rather than specify the implementation of the solution (i.e. a tool to auto-generate the stuff you need), look at it from a “process to solve the problem” perspective

me: the process to solve the problem is probably something more like "reduce the amount of time that we spend creating new solutions for these small projects". you may even want to attach a metric to "time"... "reduce it by x%" or "reduce it to 10 minutes" or something like that. once you have a target like this in mind, you can also add the ROI bit to the solution: "must not spend more than x hours / days on the solution". from there, you have a better perspective on what you can really do

them: Excellent points. Focusing on the process will obviously yield the proper solution, versus focusing my energy on a specific implementation. i.e., "it takes 20 minutes to do this. Let's cut it in half and see if we still need/want to reduce it more"

me: exactly! here's the fun part... when you make the statement "let's cut the time in half" and "don't spend more than x hours on the solution to cut it in half", you introduce some interesting challenges to the people creating the solution. "this is our target... what are the challenges that we face in meeting this target?" list out the challenges... then brain storm solutions to each of the challenges, ensuring that the solutions lead you toward your target. those challenges will spark a lot of conversation, a ton of creativity, and you'll end up with some extremely simple solutions to very complex problems

them: Which is obviously an approach that's pretty universal. In fact, that perspective would certainly prevent a lot of "over-designing" of implementations, as well

me: yup

them: I feel like it's all coming back to the YAGNI principle, no matter how I look at it

I was thinking about target conditions and the Plan-Do-Check-Act (PDCA) cycle earlier today when a code related issue that I was having popped into head and decided to meld with the current string of thoughts. The resulting thought was leading me toward wondering if a PDCA approach to test-first development (TDD/BDD/etc) would be of significant benefit.

The basic idea behind PDCA is to work in small steps toward a defined goal – not an specific implementation as a goal, but a process as a goal. We would need to find a goal that defines a process and find small steps that can provide feedback quickly, allowing us to adjust our approach to the goal as needed.

It seems that user stories and acceptance criteria are already a fit for this type of goal and the PDCA cycle. A user story and it’s acceptance criteria does not describe an implementation, but a process that the software should provide.

• Plan: discuss the story and criteria detail
• Do: implement a criteria or two
• Check: get feedback from customer rep on the increment of functionality
• Act: adjust as needed based on feedback, and start the PDCA cycle again

I’m also wondering if the test-first process itself could benefit from a PDCA cycle… this is my first attempt and fleshing out the idea, so it’s probably a bit off, still.

• Plan: write a failing test to begin the implementation of an acceptance criteria (Red)
• Do: make the test pass (Green)
• Check: validate the full business needs technical standards against the implementation
• Act: adjust the implementation as needed (Refactor)

Is there any value in this? is it just a bunch of uber-geekery babble or nonsense? Is it painfully obvious and doesn’t need to be restated as PDCA? I’m interested in honest feedback on the idea and whether or not you think there could be any value in taking a PDCA based approach to test-first development.

Anyone that follow me on twitter likely knows that I’m a big fan of Git these days. I’ll spare you the gushing heart felt nausea of how it’s so awesome  and skip right to the point, though: I don’t always have the luxury of being able to use Git. For example, my current team has been using Subversion for quite some time. Changing source control systems is not an easy thing to do when your system is as large as this one and has several key points of the development / build process tied directly to the existing source control. So, rather than be forceful and pushy and tell everyone on the team that we need to use git (most of them are already using or learning git, so there’s not much need to preach the good news), I decided to approach the situation a little differently.

 

Git-SVN? No, Git+SVN

I’ve never had any luck getting git-svn to work. I’ve tried several times on several different repositories, and it always bombs on me for one reason or another. But I still want to use git to manage my local subversion checkout. Rather than fight against the built in functionality in hopes that I would get it working, I decided to follow up on a comment that I Scott Bellware made about putting a git repository inside of an svn checkout. It’s not the “git-svn” functionality of git… it really is “git + svn”… and it turns out that this is easy to setup, fairly easy to manage and provides a lot of flexibility in working on a large codebase.

I want to let you know up-front, though, that managing git + svn in the manner I’m describing here does add some administrative overhead. This is not a “free” solution. It has a cost associated with it, so it must be evaluated as one possible option for your situation. Like most other tools that we have in our toolbox, understanding where your situation sits on the cost-benefit curve is important.

 

Setting Up Git+SVN

Getting setup to run git + svn is very straightforward. There are only a few things you need to do. I am assuming that you are already familiar with both git and subversion, though. If you’re not, there are some great tutorials out there on the inter-webs (I’m particularly fond of Jason’s Git For Windows Developers series. He did a great job with that series.)

 

1. SVN Checkout

There’s nothing special about this. Follow your team’s standard practices for what to checkout from your subversion repository. For my team, we checkout the entire trunk. It’s a massive amount of information to pull down, but we seem to use most of it on a regular basis.

 

2. Initialize A Git Repository

Run a “git init” from the root folder of your subversion checkout but DO NOT add / commit to the git repository, yet! You need to complete steps 3 and 4 before doing that. This will create your .git repository. I’ve got a .git and a .svn folder sitting right next to each other in my svn trunk checkout, now.

 

 

3. Gitignore The .svn Folder

This step is very important. You do not want your git repository to be littered with constantly changing .svn folder contents. For one thing, it makes the management of your git commits difficult because you have to sift through more data than you want to make sure you’ve staged / committed the right files. More important, though, is that you can very easily corrupt your svn checkout if you don’t do this!

Imagine having git make modifications to your .svn folder contents while you are switching between branches, then doing an svn update and getting a merge conflict in the .svn folder. That does not sound like fun to me.

To keep things clean and simple, add .svn to your .gitignore file.

You should also take the time to add your standard .gitignore settings at this point. My list has grown a little, but the basics are still the same as when I posted them, a while back.

 

4. SVN Ignore The .git Folder

Subversion doesn’t need to know about your git repository, either. More specifically, you’re team members don’t want to know about it… ever… imagine 3 team members managing their branches with git and one of them accidentally commits their .git folder to subversion. The next time you update from subversion, it would fail because the folder exists. Now imagine that someone does a clean checkout from subversion and receives your git repository in the mix. If they want to use git to manage branches, they would just start using git because a repo exists. The next time either of the two git users commits to subversion, they will update / overwrite the git repo in subversion, and the next person to update from svn will have a clobbered git repo. That’s one mess I don’t want to clean up.

Do yourself and your teammates a favor: tell subversion to ignore the .git folder.

 

 

5. Add Everything To Git

At this point you can add the contents of your svn checkout to your git repository. There’s nothing special to do or remember here… just go about adding the files in whatever manner you prefer.

 

Managing Git+SVN

Since subversion is the system of record in this setup (it’s the ‘upsteam’ source where all of the team members commit), you need to make sure that you can always do a clean update from subversion. You want to avoid having subversion clobber your local changes and you definitely don’t want your use of git to clobber anything in subversion. To do this, hold fast to this simple rule in how to use the branches in git:

the git ‘master’ branch is a clean, stable subversion checkout

The git ‘master’ branch is almost always a clean checkout from subversion. It may not be up to date with the subversion server, but it is clean for whatever revision it is on. I very rarely work directly in the master branch, in git, actually. The only time I will work directly in the master branch is when I know I’m making a very small change that will be completed before my next food/drink/bathroom/whatever-else break. I do this so that I always have a known good copy of the subversion checkout. Without this, it’s very easy to get your git revisions out of sync with your subversion revisions, and cause problems when committing to subversion.

Really, it really doesn’t matter if its the master branch or not. It can be any branch you want – just make sure you have one that is a clean, stable checkout from subversion. I recommend using the master branch for this, though, as the significant of ‘master’ in git being correlated to the stable subversion checkout makes it a natural fit.

This is likely the most important rule in managing the interaction between git and subversion, though. The reasons will become apparent, later.

 

Just Another Git Repository

If the git ‘master’ branch is always a stable subversion checkout, then it follows that all work is done in a git branch (other than the master branch). With this in mind, we can fall back on the already known patterns of managing feature branches and managing a local git repository.

After getting your git+svn setup, branch off the master in git and start working. Other than the one rule about the master branch, you are managing a git repository like any other git repository. Branch, merge, rebase and do whatever else that you want to do with your local git repository. This is where the real flexibility of this process comes into play, honestly. You don’t have to worry about subversion very much because you are not dealing with subversion right now.

 

Committing Changes To Subversion

Once you have a set of changes ready to go and you’ll want to commit your changes to subversion. The first thing you need to do is update your master git branch with the latest changes from subversion. After that, you can merge your changes into the master branch and then commit to subversion.

Here’s how I manage this process:

 

1. Commit Or Stash In Git

Get all of your changes in your current git branch committed or stashed, and change your current branch back to master.

git add –A 

git commit –m “some description of my changes”

 

2. Update The Master Branch From Subversion

Once you’re in the master git branch, do an svn update. If you’re already working in a git bash prompt, you should be able to run ‘svn up’ from there and have it pull all the latest changes from the central subversion repository.

git checkout master 

svn up 

 

3. Commit SVN Updates To Git

After you update from subversion, you need to commit the changes to your git repository. I typically comment the git commit with “update from svn” so that I know why all these files changed.

git add –A 

git commit –m “update from svn” 

 

4. Merge Your Working Branch’s Changes Into ‘master’

Once you have all the latest and greatest changes from subversion, you can dump all of your working branch changes into the master branch. Remember that you’re dealing with git at this point – you have as many options as git provides for getting your working branch changes into your master branch.

This is the one time that I allow my master git branch to have any significant changes in it – but only because it’s temporary. After the working branch changes are in the master branch, I immediately move on to the next step.

git checkout myworkingbranch 

git rebase master 

git checkout master 

git merge myworkingbranch 

 

5. Commit To Subversion

Now that you have your master git branch up to date in relation to subversion, with your git branch’s changes also in place, you can commit these changes to subversion. Be sure to add some descriptive comments of why you are making this commit (and remember – git / subversion will tell you what changed, it’s your job to say why you changed it.)

svn commit –m “some description of my changes”

 

Lather, Rinse, Repeat

I don’t expect everyone to use this exact set of commands, though. I don’t even do it this way all the time, honestly. This is just one example of how you can manage the git+svn process. There are probably a dozen or more methods of managine your git+svn setup. I’m very interested in hearing about how you do it, too. Please drop a comment here or put up your own blog post on how you approach this situation.

But that’s all there is too it, really. The process is fairly straightforward and can be repeated as often as needed.

 

Benefits, Drawbacks, and Lessons Learned

There are a few things that I really like about this git+svn setup, a few things that bug me, and some lessons that I’ve learned as well.

 

All The Benefits Of Branch-Per-Feature, Lower Subversion Overhead

This is a complete branch-per-feature implementation running on an individual team member’s computer. All of the benefits (and drawbacks) that have been discussed around branch-per-feature are in play, here. The real benefit, though, is that you get branch-per-feature with a lowered subversion cost. You can quickly snap off branches from the git master branch, make changes, dump them back into master and commit to subversion – all while having another branch or ten sitting in the git repository, waiting for you to get back to them.

 

Don’t Worry About Keeping Up With Subversion Changes

Since you have the master branch in git stable, as a clean checkout of a subversion revision, you don’t need to keep updating it constantly. You can wait until you are ready to merge your changes into subversion and commit them. I’ve gone for several days (almost a week, once) without pulling down any updates from subversion, with no problems. This is one of the reasons why it’s important to keep your master branch as a clean, stable subversion checkout.

 

No More Subversion Merge Conflicts

I don’t have subversion merge conflicts anymore!!! WOO HOO!!! :)

Even when pulling down a significant amount of changes from subversion, I don’t have merge conflicts anymore. Once again, this is because the ‘master’ branch in git is a clean, stable checkout of the subversion repository.

You will still have merge conflicts, mind you – just not from subversion. Your changes in git are subject to the same rules and processes as any other git repository, though, and we all know that merge conflicts happen. Don’t fool yourself into thinking that you’ll never have a merge conflict again – that’s crazy talk.

 

Administrative Overhead

As I mentioned earlier in this post, there is an overhead associated with this git+svn setup. Specifically, the process of moving commits into the master branch so that they can be committed to subversion does add some more work. There are likely other things that add a bit of time, too. You need to evaluate this process (and potentially improve upon the steps I’ve outlined) for yourself to see if/when the cost is outweighed by the benefits.

 

One Git Repo Per SVN Checkout

When I first started out, I was managing git repositories in specific sub-folders of my subversion checkout. For example, we have a Trunk/Source/TA folder and a Trunk/Source/TAMobile folder, each of which contains a different solution. I originally set up a git repository in each of these folders thinking that it would be easier to manage them independently. This turned out to be a bad idea and a management headache. I had to wipe out my git repositories and recreate them more than once because I accidentally left them sitting in branches when I did an svn update. It took a few tries, but I finally learned the lesson: one git repository per svn checkout.

Your mileage may vary on this point, though. You might be able to manage multiple git repos in a single svn checkout – but I wouldn’t recommend it.

 

!!!Caveat Emptor!!!

Any time you mix and manage multiple source control systems manually, like this, you are opening up the possibilities of clobbering either or both of the systems. This is not a fool proof solution. There are no safeguards here. This is raw, high risk source code management. With such high risk, though, comes a very high potential for making your life easier. If you’re not comfortable with git or subversion (or worse yet, both) to the point where you know how to fix your mistakes without the assistance of others, then I would recommend that you not yet take on an endeavor as risky as this. Get comfortable with managing git and subversion, first.

 

A Step In The Right Direction, But Not The End Goal

I’ve been working with git+svn for more than 2 months now and I don’t know if I’ll go back to managing branches in Subversion – at least not for this team. I imagine there will be times when I need to do a push into a subversion branch for one reason or another, but this is likely going to be an exception to how I work now.

In the end, though, the git+svn setup is definitely not the goal. There are enough potential issues with it and there is enough of an administrative overhead, that is make me cringe at times. Ultimately, I would like to run git as the source control system for our team. But until that time arrives, I have a working solution for my needs.

Don’t confuse activity – even when it has a visible, measurable effect – with productivity. Without a clear picture of where we are going and why, our best efforts at improvement (though they may be ‘continuous’ efforts) are likely to be change for the sake of change. At best, change without direction has a chance of doing something useful and making part of our system easier to work, reducing cost relatively soon. At worst, change without direction has is a direct cause of lost revenue as we are spending our time and money to change areas of the system that are not causing any immediate financial difficulties. Worse yet, even when we have a goal set, we can still do financial damage to the project and company.

 

Flailing About, Believing We Are Improving

At a previous job, I was working on a product with a team that had been together for a few years. We had a rapidly evolving set of standards and were making constant technical improvements to the architecture and infrastructure of the application. At one point, we needed to add some functionality to a part of the application that was used by two processes. The architecture and code for that part of the app was ‘old’ by our standards and we needed to update it to support the new functionality. After discussing what needed to be done with the management / technical leadership of the team, we sent a developer down the path of rewriting the screens and process involved in this part of the app.

6 weeks later, we had a rewrite of the functionality that gave us nothing in terms of adding the new functionality. We had flailed about with our eyes closed, frantically rearranging puzzle pieces, cutting them apart and gluing them back together in a manner that let us use them easier. Yes, the code was easier to read and understand. Yes, the code was well tested (done entirely with test-first development). But the goal of being able to add the new functionality, easily, was missed entirely. We were effectively sitting at ground zero, staring at the same implementation problems that we started with from the perspective of the needed functionality.

After some digging we realized that the problem was directly the fault of our management efforts, relating to goals and direction. We gave the developer the task of re-writing the functionality but we never set any goals for how the rewrite should be done, to support the new functionality that we needed. In the end, we realized that we had wasted thousands of dollars and hundreds of man hours because we didn’t have the right goal set for the change we were engaging in.

 

What Technical Debt Can We Pay Down?

My current team is working on a product that is now 7+ years old. It has a lot of “legacy” code in it, to say the least. The amount of technical debt that we encounter on a daily basis is staggering. But that’s too be expected on a product of this size and complexity, with a continuously evolving landscape of ‘best practices’ in the .NET arena. The good news is we have a continuous effort to improve the system instilled in the team – and it’s been there for at least the last few years (I don’t know the entire history or timeline because I’ve only been here for a few months). However, there are still some very dark, disturbing creatures that like to raise their heads out of the depths now and then, demanding the sacrifice of one’s soul in order to get the current task done. But, all snarky commentary aside, we manage to change the system a little more every day and we are slowly moving forward to a … well… toward what? A system with less technical debt, I suppose. But for what purpose? To allow what to happen?

And now add Dave Laribee’s first article on Technical Debt in MSDN Magazine into the equation. My team – management included – is now gung-ho about paying down technical debt, after reading this. There’s plenty of reason to be gung-ho about it, too. The article paints a very telling picture of technical debt and why it’s bad. There’s a lot of good info in there and I highly recommend reading it. However, I believe Laribee got it wrong when it comes to choosing what technical debt to tackle, by introducing the use of Luke Hohmann’s “Buy A Feature” Innovation Game.

In the article, there is an example of the New York City streets vs. the Atlanta, Georgia streets. It is a very compelling and a good example of long term, goal oriented thinking. We can make some sweeping assumptions about how New York seems to have had a goal of creating a systematic, simple to navigate street layout and naming / numbering convention. They likely wanted people to get to and from their destinations with relative easy. With Atlanta, though, it seems that the goal was to use the existing “cattle paths” as the road layout – to simply grow a new infrastructure of roads on top of an existing, outdated system of paths. There’s a wealth of metaphor that can be drawn from this, but the example is misleading in the context of the article.

Shortly after the street layout example, the article suggests the use of the Buy A Feature game to determine what debt to pay down. The problem is not the game itself – there are plenty of circumstances where this game is appropriate – but that it is being used in a circumstance where it does not serve it's purpose. This game is essentially a more interesting variation on a standard voting system for what to do next. My team has basically been doing for the last week or so – but using a vote count instead of a dollar  amount. We’ve gathered together a list of problem areas, prioritized that list a little, and set about voting on the items in the list. The problem with this approach – whether we call it Buy A Feature or not – is that it inherently pits the team against each other and facilitates a flailing about in terms of what should be worked on. In some cases, we end up with various team members that are very persuasive, taking control over the conversations and convincing other less persuasive team members that they need to vote a certain way. In other cases, we have developers voting independently of each other, with no rhyme or reason in terms of the big picture, creating a scattershot of “I had to touch this part recently so I want to fix it” throughout the system.

To illustrate the problem, take a look at the current voting from our technical debt list. (I’ve scrubbed the names of the voters and a few of the items)

Though there is a relatively close grouping (due to a sort that was done  on Vote Totals just prior to all votes being cast), we can clearly see a scattershot effect taking place. There is not one item that is unanimously agreed upon and there are several items where only one or two people feel that it is important enough to warrant a vote. Even if the entire team (or the majority in the case of some of these items) agrees on what should be done, we still run the risk of flailing about in our improvement efforts… just because we want to do these things, doesn’t mean we should.

 

What Technical Debt Do We Need To Pay Down?

From what I gather, the MSDN article assumes that the elimination of technical debt is the goal in itself. If that’s true then the Buy A Feature game may be an appropriate tactic for determining what should be worked on, next. However, I’m betting that your company doesn’t want to spend 6 weeks of effort re-arranging the puzzle pieces with your eyes closed, the way my previous team did. You, like everyone else in the software development business, have customers and managers and product owners and everyone else with a stake in the product wanting delivered functionality that people will pay for.

Mike Rother paints a very powerful picture of how to approach a situation with many different options, in his book Toyota Kata. He correctly points out that the problem with most improvement efforts is that we spend all of our time focusing on what we can change, instead of what we need to change. When I first read this, I had no clue what the real difference was – or at least I didn’t understand how you could know what needed to be done vs. what could be done. It was when he started talking about the long term vision and target conditions, that it all started to make sense. I had previously seem similar work in Patrick Lencioni’s book, Silos Politics and Turf Wars. In this book, Lencioni talks about how we can eliminate the organizational problems listed in the book’s title by creating a unifying vision of where we need to go. Lencioni calls this the “Thematic Goal”. Rother’s calls this a “Target Condition”. The purpose is to provide focus and direction for our efforts. They give us something to strive for, something to accomplish, and a way of knowing that we are making progress and enacting change with a purpose, not just changing for the sake of change.

The scope of a target condition (my personal favorite of the two names) is to specify a state at which we want to be, within a given time period, budget and other constraints that we may need to account for. Typically, we should look at something that is a few weeks to a few months out and does have a readily apparent solution. (If we already know the solution then we don’t need to plan, prioritize or question what should be done. We would skip right to implementing the solution.) By doing this we provide a focus that our efforts can be aligned to. With our efforts aligned, the question of what to work on next easily moves away from “what can we work on?”, to “what do we need to work on?”. We can use the target conditions to help eliminate the organizational problems of us vs. them, pursuasive vs. passive employees, and the other issues associated with voting mechanisms and directionless action.

There are additional benefits to working within target conditions, as well. For example, it is well known that people are motivated by accomplishment of goals that they see as challenges – especially when they help to define and implement the solutions for the challenge. When setting up a target condition, we should introduce a challenging goal… something that can’t be immediately solved. We need a stretch goal with issues that we need to work out from a technical, human and/or process perspective. If we setup a target condition like this and turn people loose on brainstorming the challenges and solutions, we end up with the full potential of human creativity and ingenuity being tapped. Job satisfaction comes quickly when the solutions people implement to the challenges that face them, are the ones they suggested.

 

Refocusing The Payment Of Technical Debt

The problem that our team faced in the effort to reduce technical debt is that we had no team focus – no unifying theme that will tell us what we need to work on, next. When faced with a technical debt problem, we assumed that “eliminate technical debt” was in itself, a valid goal. If we were in the business of writing clean code, then this may be valid. However, we’re in the business of providing features and functionality to our customers so that their jobs can be done faster and more accurately with less effort. 

The good news is that we do have a target conditions – we just didn’t know it or didn’t realize that we needed something more specific than the elimination of technical debt. The following is from an email that our manager sent out, with a few of my own edits to clarify some points and remove some information.

Complicating our situation today is how we do testing.  [edit: we are staggering sprints between development and testing]  Where we want to be is that items get finished by the developer and then get tested as soon possible.  This leaves ample time for developers to make needed fixes should there be bugs, and also leaves time at the end of the iteration for the PMs to do the final acceptance review.  We have not been doing final acceptance review, and we want to change this.

The way we operate today, we only update [our internal test environment] on Monday mornings (or ad-hoc during the week if there's an update needed after a bug fix).  So testing can't begin until Monday, because the new code isn't available.  The dev team will start working on figuring out how to get completed code into a testable state more quickly so that testing can commence.

However, I think we can say that it is in everyone's best interest to get the testing done as soon as possible, and to not leave it until the end of the last week before a release.

I would very much like to see us getting more testing done earlier, and I would really like to see us not leave the testing of both iterations/milestones until the second week.  We can't go easy on the week in which we don't have a release.  Doing so puts a lot of pressure on, well, everyone when it gets left so late.

 

Setting Up A Target Condition

This email provides a perfect opportunity to clearly define a target condition. Having a target condition then allows us to answer the question of what technical debt needs to be paid down, now.

Let’s define the target condition based on the information in the email.

The Target Condition

• Delivery of new functionality and bug fixes to the internal testing environment, immediately after development work is done
• Formal testing immediately after deployment to internal testing environment
• Project Manager / Product Owner “final acceptance review” prior to delivering functionality to production environment

Some Of The Work To Be Done

(Note that this is only an sample of some work that needs to be done to achieve the target condition)

• Deployment of completed features without bringing along incomplete features
• Full automation for deployment process
• Isolation of feature development
• Better integration of / communication between dev and test team
• Coordination between dev / test / PM for final review
• May need a better CI server to support deploying to many different environments, easier

Notice how many of the items from the current voting list of technical debt didn’t make it into this list. This is because not everything that we want to work on will have an immediate impact on us moving toward the target condition. There may be other target conditions that will be affected by items that dropped off the list, though. The technical debt items will make their way back into the list of what we need to work on, as our target conditions are met and new ones are specified. It’s also important to note that not everything in this list is “technical debt”. There are some process and human interaction issues that must be addressed, as well. The target conditions that we choose will determine what needs to be worked on – not the other way around.

 

Pulling Principles From The Illustrations

The focus of my illustrations has been technical debt, in this post. However, the principle that underlies the issue – using currently target conditions to drive our improvement efforts – is universal. The effort we put into change does not have to go to waste. We don’t have to end up wasting 6 weeks of effort, as in my first example. We can ensure improvement is happening, rather than guess or vote and hope we are not just changing, by setting target conditions for our teams and working toward those conditions, continuously.

 

Other Conditions For Paying Down Debt

Since the illustration in this post is technical debt, I wanted to address some concerns that I know will come up.

I want to point out that it doesn’t always take a target condition to set up the circumstances in which technical debt should be paid down. There are other conditions, such as a defect report, that can create the necessary circumstances. For example, when a team receives a defect report the surrounding code should be scrutinized and a determination of whether it is cost effective (from a long term perspective) to invoke the boyscout rule should be made.

There going to be times when a voting process is appropriate for managing technical debt, too. For example, if a team knows that a significant amount of cleanup must be done before the project can move forward with its current goals, then there may be a need to prioritize what is cleaned up, when. There may also be a need for management to allow the satisfaction of some technical itch in the development team. It may be beneficial to morale and personal pride in the work being done to allow a team to have some Buy A Feature money, giving them the opportunity to express their own interest in solving their own pain points. There may even be a target condition to improve morale or solve some architectural issues that would account for this…

The point is, we need to remember that context is king and your circumstances need to be evaluated for what they are.

I found the quote that title’s this blog post in Mike Rother’s Toyota Kata book – the origin is simply stated as “a Toyota person”.

 

Kanban: We’re Doing It Wrong

The title quote plays hand in hand with a quote from The Toyota Way:

“kanban is something you strive to get rid of, not to be proud of”

What does this really mean? It means we are probably doing Kanban wrong or for the wrong reasons.

If we’re setting our WIP limits to the team’s capacity currently or something above current capacity, then we’re likely doing it wrong. If we’re not using a Kanban system as a tool, a means to identify and solve systemic issues, then we’re likely doing it wrong. If we are using Kanban and WIP limits as ends in themselves, then we are certainly doing it wrong.

 

One Step In A Long Journey

According to Rother, the reason most kanban implementations fail is because we expect to be able to implement the system and have it stabilize immediately. This is wrong. We must instead use the desired state of the Kanban system as a target goal. When we first define our process and our WIP limits, they must hold fast as the target we are aiming for – not a hard and fast rule of what must happen now. By changing our focus from “this is what we must be doing” to “this is what we are striving for”, we allow ourselves the opportunity to truly improve the system that we are working in. When we try to enforce the WIP limits, the Kanban system will hold up a bright spotlight and reflecting mirror, exposing the real problems that prevent the team from achieving the stated process and WIP limits. It is then our responsibility to face those problem head on and overcome them – not work around them, not ignore them, not give up because we don’t think Kanban can work for us, but to eliminate the problems by addressing and removing their root causes.

Therefore, a Kanban system is not something that should model our team’s current process and capacity. Rather it should model the process and capacity that we want to have, next. It should represent the output of the current improvement efforts as well as the input into the next improvement efforts. It should be a “target condition” that moves us toward our “vision”, as Rother states it. Or a “defining objective” that moves us toward a “thematic goal”, as Patrick Lencioni states it. The point of these authors is the same and is one that I have touched on briefly in my blogging and more extensively in my coaching and change management efforts. A Kanban system is not an end in itself, but only one of many tools to help us see our system’s problems and eliminate them by giving us focus and direction.

 

Enforcing WIP Limits May Not Be The Right Thing To Do

When we are using a specified state for a Kanban system as a goal that to strive for, then we need to recognize that enforcing the WIP limits within the current goal may be dangerous. It is possible for an artificial WIP limit enforcement to cause financial damage by neglecting the needs of the business and/or the customers. We need to understand the cost of delay for priority situations that come up unexpectedly.

For example, when an urgent request comes in – say a production bug that must be fixed immediately, or risk loss of business and revenue – there are at least a few options to consider: queuing the work to be done (likely at the top of the queue if it is a priority), having one or more team members stop what they are doing and get it done immediately, or possibly bringing in additional resources to work on the issue. The right answer, of course, depends on the situation and understanding of the cost of delay for the priority issue vs the cost of delay for the current work in progress. 

There will be some scenarios where the work should be queued and done when capacity is available. There will also be some scenarios where the work needs to be done immediately, recognizing that we are not yet able to meet our WIP limit goals. When we run into scenarios that need immediate attention and WIP limit violations, we need to ask ourselves “what problems are causing us to not be able to meet our WIP limit goals?”, and then work to correct those problems.

 

The Continuous Improvement Cycle

By solving the root causes of problems that prevent us from working to your WIP limit goals, we will eventually be able to meet those goals. If we find ourselves and/or our team in a position to routinely meet the process and WIP limit goals, then we are now back at square one. We need to set new goals that we know are not currently attainable and begin the hunt for and elimination of our new problems.

The goal is not to use a Kanban system. The goal is to eliminate the need for a Kanban system.

I was notified yesterday that my ‘Decoupling Workflow’ submission for the Lean Software & Systems Conference in Atlanta was accepted!

This will be a complete re-write of the presentation, by the same name, that I gave at the 2009 Austin CodeCamp. I’ve learned a lot since then, had a chance to implement this architecture in a few more systems, and have recently expanded on the subject with my ‘Object Messaging’ post. The emphasis of this rewrite will be enabling concurrent and incremental feature development with one possible architecture that supports release per feature (including some branch per feature).

I’m honored to have this presentation selected and to be able to participate along side of some tremendously talented, intelligent, and influential people.

You can read the full abstract and get more information about the conference at http://atlanta2010.leanssc.org/ 

I hope to see you there!

Update: There’s a now apparent disconnect between what I intended in this post, and what it came across as. The poor choice of labels and tasteless examples are getting in the way of what I was trying to say. My intention was to cry out against celebrity, opportunism, and abuse of status. Turns out, I came across as doing those things that I was trying to say were bad.

The point I was trying to make is that the headline making news of individual’s “shocking” or “extreme” actions are not usually the right way to enact change (there are times when it is appropriate, though). It was supposed to be a call to action, to live the example of the ideals that you espouse, to influence those around you toward better practices by showing them and teaching them, to be a part of the solution and not just someone who wants to get their 15 minutes of fame in the limelight of extremist action.

I think @derekgreer undertood my intent best: “are you just someone that likes to stir up trouble or are you trying to help a cause along?”

 

The Original Post

------------------

 

A long time ago, I heard someone ask and heard another person respond:

Q: “Are you an environmental activist?”

A: “No. I’m an active environmentalist.”

I honestly don’t remember where I heard this, or when, but it has stuck with me for many years.

  What’s the difference?

I don’t know what the original respondent intended as the difference, but here’s what I take from this:

An movement activist is someone who gets a lot of public attention by doing outlandish and even crazy or illegal things. An activist may bomb an abortion clinic, tackle the Pope, or chain themselves to some large object to prevent ‘bad’ things from happening to it. These people tend to do these things for attention – attention for themselves and ‘for 'the movement’ (though this can be questioned). They make the headlines and people pay attention for a few minutes. These people are the ‘celebrities’ of ‘the movement’. Once the story is old news, though, people go back to what they were doing, ignoring whatever movement this scene was promoting.

By contrast, a person who is active in a movement is pretty much the opposite of that. They aren’t in it for attention or glory. They don’t necessarily care about the name or ‘the movement’ as a formal organization. Rather, they see the underlying goals and ideals and are actively working within them. They are out in the real world every day, making a difference in someone else’s life by _living_ the examples and ideals of ‘the movement’. They promote the principles and practices of ‘the movement’ not through voice and showmanship, but through example, by serving and through real leadership. They are not celebrities. They are servants, enriching the lives of those that ‘the movement’ is claiming to benefit. … this doesn’t mean the active person gets no attention. There are plenty of authorities and leaders that are making waves and getting attention who I don’t consider to be activists. But they are getting attention for the right reasons – for doing good and helping others do better.

 

Activism and Active People In Software Development

Recently, there has been some interesting conversation around twitter and the blogosphere around the state of certain movements in software development. There are a number of people who love to be the activist in various software movements. This can be said of the Alt.NET movement, the Software Craftsmanship movement, and probably others. Unfortunately, those people are also the ones doing the most damage to the movements.

So, am I an activist or someone active in any of these movements? … does anyone care? I doubt that you do, and you shouldn’t if you do. The real question is whether _you_ are an activist in ‘the movement’, or are active in promoting better principles and practices by living the example. I sincerely hope that you are the active person, living and teaching by example.

…

the sad thing is that this blog post is likely going to wither in the echo chamber with no real effect. for better or worse, I am a part of the echo chamber by association if not by action.

Earlier in the year, I posted a few times on the Application Controller pattern that I was implementing, including some workflow service related posts, all leading up to the presentation on decoupling workflow from forms that I gave at Austin Code Camp ‘09. I’ve been working with this style of architecture in my winforms apps since then, and have really grown to love it. And now with my new job and my new team, I’ve recently had a chance to take the same patterns and port them over to the .net compact framework. The good news is that it took zero changes to the core architecture and code. The 2 things I had to do were re-write the form implementations in compact framework forms, and replace structuremap with a ninject version compiled for the compact framework. It was an easy port and the code is available on github, along with the original winforms version.

One thing I don’t think I did very well with my original posts, or in the presentation I gave, was explaining when you should use which parts of this architecture. There’s nothing terribly difficult about understanding when to use what, honestly, but it may not be so obvious to someone that is new to the patterns and implementation details that I’ve included in the sample applications. Before we get into the detail on when to use what, though, I want to discuss the underlying patterns that make up the App Controller. This will help to facilitate the discussion on when each part of the App Controller should be used.

 

The Code Patterns

These patterns have been covered in depth, but I wanted to consolidate them here for ease of reference.

Command Pattern

Originally outlined by the infamous "Gang of Four", the Command Pattern is described as an object that represents an action - a command that will be executed.

The command patterns is implemented in the AppController via the Execute<T>(T args) method. You can pass any arbitrary command parameter information into this message as the T args parameter. The type of the args is then used to find an instance of an ICommant<T> interface and the ICommand<T> has the .Execute<T>(T args) command called on it, to execute the actual command object. This allows you to decouple the knowledge of work to be done from the worker doing it.

 

Event Aggregator

A system with lots of objects can lead to complexities when a client wants to subscribe to events. The client has to find and register for each object individually, if each object has multiple events then each event requires a separate subscription. An Event Aggregator acts as a single source of events for many objects. It registers for all the events of the many objects allowing clients to register with just the aggregator.

The Event Aggregator in the sample code is based on Jeremy Miller’s Build Your Own CAB series (here, here and here), and is hidden behind the Raise<T> method of our AppController. We can easily create and publish events by calling the AppController.Raise<T> method, and we can easily subscribe to those events by implementing an IEventHandler<T> interface. This allows you to have multiple parts of the system respond to an event without having to be coupled directly to the part of the system that raises the event.

 

Application Controller

“Some applications contain a significant amount of logic about the screens to use at different points, which may involve invoking certain screens at certain times in an application. This is the wizard style of interaction, where the user is led through a series of screens in a certain order. In other cases we may see screens that are only brought in under certain conditions, or choices between different screens that depend on earlier input.

To some degree the various Model View Controller input controllers can make some of these decisions, but as an application gets more complex this can lead to duplicated code as several controllers for different screens need to know what to do in a certain situation.

You can remove this duplication by placing all the flow logic in an Application Controller. Input controllers then ask the Application Controller for the appropriate commands for execution against a model and the correct view to use depending on the application context.”

Now I know that my AppController doesn’t take on all of these responsibilities. That decision was made on purpose, to further decouple the workflow decisions. In keeping with the Single Responsibility Principle, I decided to move the “logic about the screens to use at different points” out of the AppController itself and into objects that can control the specific flow of screens for a specific need. The movement between screens, coordinating all of the resources that are needed for a given process, are encapsulated in what I call Workflow Services.

 

Workflow Services

I’m not sure if this is a legitimately recognized pattern in software development (hence no link). However, it’s a term that I’ve been using for over a year now and it fits with the intent very well. My definition of a Workflow Service, if I had to give one, is something a long the lines of:

An object that encapsulates the entire flow of work, from beginning to end, for a given action or sequence of actions. This includes the coordination of all required resources, such as the user interface, user input handling, object and domain model calls, repository and data access calls, and/or any other resource that is required to complete the action in question.

A workflow service is an object that represent the logic of how something happens from a 1,000 foot view. Imagine making a flowchart diagram that has a start, some work and some decisions, and a stop. Rather than haphazardly coding this flow of work as a tightly coupled set of forms which makes it difficult to change the flow of work, you can code the knowledge that is represented by the flowchart into an object. This object is a workflow service.

One thing you’ll notice is that the AppController has no explicit IWorkflow or IWorkflowStep or anything else related to worklow in it’s model. This is because workflow is a very application and task specific process. You cannot be hamstrung to a specific style of workflow in your application. Rather, you should be free to use any type of workflow process that you need: wizard style, single screen pop up style, or however else you want to flow.

 

Correlating Code Patterns To Messaging Patterns

I’ve done a lot of work with messaging systems in the last two years and it’s had a tremendous amount of influence on me. I’ve changed how I see the architecture of large scale, disconnected systems and even how I see small scale, in process systems. The idea of passing around various types of messages for various purposes has really settled well into my thinking and has helped me to create some systems that are very well abstracted, decoupled, and pieced back together at runtime.

There is a direct correlation between the code in the App Controller and a messaging system. Many of the principles and patterns that I’ve used to implement the App Controller are directly based on various patterns found in messaging systems. These are the core messaging patterns that I’ve used, and how they correlate. I haven’t really talked much about messaging patterns, even though it’s become an important part of what I do. You can find all you need to know and more in the Enterprise Integration Patterns book and website. (Note: all the descriptions and images for these patterns come from the Enterprise Integration Patterns website)

 

Publish / Subscribe Channel

A Publish-Subscribe Channel works like this: It has one input channel that splits into multiple output channels, one for each subscriber. When an event is published into the channel, the Publish-Subscribe Channel delivers a copy of the message to each of the output channels. Each output channel has only one subscriber, which is only allowed to consume a message once. In this way, each subscriber only gets the message once and consumed copies disappear from their channels.

The publish / subscribe channel lets you broadcast a message to any client that is listening to the channel. The publisher of the message shouldn’t care who is listening or how many listeners there are. There may be zero and the publisher does not do anything different than if there are many listeners. There is no durability in the messages published to this channel. If you are not actively listening to this channel when a message is published, then you will not receive the message.

A common use of the publish / subscribe channel is to broadcast event messages (see below) between systems. This allows systems that are not otherwise connected to respond to things that are happening in other places. The Event Aggregator in our App Controller provides us with a publish / subscribe mechanism. 

 

Event Message

When a subject has an event to announce, it will create an event object, wrap it in a message, and send it on a channel. The observer will receive the event message, get the event, and process it. Messaging does not change the event notification, just makes sure that the notification gets to the observer.

The event message is the data that describes the event that occurred. It allows the system that is responding to the event to know the details of what happened, so that it can make the appropriate choices in how to respond. In our AppController, the generics parameter <T> of the Raise<T>(T args) represents the event message. This allows the Event Aggregator to determine which event handlers need to be executed – based on the IEventHandler<T> interface implementation – so that the appropriate actions can be taken for the type of event that has occurred.

 

Point To Point Channel

A Point-to-Point Channel ensures that only one receiver consumes any given message. If the channel has multiple receivers, only one of them can successfully consume a particular message. If multiple receivers try to consume a single message, the channel ensures that only one of them succeeds, so the receivers do not have to coordinate with each other. The channel can still have multiple receivers to consume multiple messages concurrently, but only a single receiver consumes any one message.

The point to point channel is commonly used when a publisher knows that there is one very specific end point that needs to receive a message, and the message needs to be delivered even if the subscriber on the other end is not currently available. For example, if you have two separate systems that need to talk to each other and exchange very specific information with each other, you will likely have Point To Point Channels open between the two systems. This allows each system to send a message specifically to the other system, without having to be coupled to the other system’s API.

In our AppController, the command pattern with the ICommand<T> interface acts as the Point To Point channel. The simplified command pattern implementation that I provide in the example code may not provide the durability of a full Point To Point messaging system, but it does provide most of the other constructs and uses. It allows your code to say “I need this to happen” and have another piece of code get it done, without coupling the two parts of the system together.

 

Command Message

There is no specific message type for commands; a Command Message is simply a regular message that happens to contain a command. In JMS, the command message could be any type of message; examples include an ObjectMessage containing a Serializable command object, a TextMessage containing the command in XML form, etc. In .NET, a command message is a Message with a command stored in it. A Simple Object Access Protocol (SOAP) request is a command message.

The command message is essentially an order to do something, describing what is to be done. The “how” of the work, though, is handled entirely by the code that is processing (executing) the command. In our AppController, the generics parameter T in the .Execute<T>(T args) method is the command message. The T parameter also has a second duty, though, helping to define the point to point channel. This argument is used by the AppController to determine which ICommand<T> should be executed – in other words, it determines which “channel” the message is sent through. The code that resides in the channel (the ICommand<T>) may or may not be the final executing code for the command. You can have an object register itself to execute for a given command message and then have it call out to another object as the receiver (subscriber) of the command, doing the actual work. In most cases, though, I prefer to combine the channel and subscriber into a single object – to have the object that implements the ICommand<T> be the object that executes the final code for that command.

 

When To Use Events Vs. Commands

Keep in mind that these are only suggested reasons and situations for use. From a technical standpoint, you can use any part of the AppController for anything you want. That doesn’t mean you should, though. It is good to keep a general set of guidelines in mind when using the AppController’s functionality vs. when to use other functionality and more direct calls into other pats of your system. This will help to create consistency in your system, which will help to make the entire system easier to learn.

 

Commands / Execute<T>(T args)

The command pattern is like a monitor in the kitchen of a fast food restaurant. When orders come in, the show up on the monitor. Each of the items on the monitor can be considered the parameter data – the “what to make” parameter - while the order itself showing up on the screen is the command “go make this”. The person executing the command – that is, the person who makes the food – is irrelevant to the customer. They don’t care who makes it as long as it is made correctly. Similarly, your code that needs a command executed may not care what code executes it. When this is the case, you can ask the AppController to execute the command that is registered for your parameter data.

An object that implements the ICommand<T> interface does not need to be ‘live’ for the AppController to execute it. The AppController uses the underlying IoC container to find the registered instance of the command handler and will instantiate it if the handler is not already alive.

When To Use It

A good rule of thumb for executing a command is when you need an action to occur that is not part of the current process or workflow. Some good example of this are buttons or menu items that kick off another workflow or sub-workflow. In the example code for the AppController, I have a button on the main form that kicks off the process to add a new employee. Behind the scenes, this button causes an Execute(new AddNewEmployee()); call – telling the AppController to execute whatever command can handle the AddNewEmployee command message. The main form / presenter in this case know that the user is requesting for a new employee to be added to the system. However, the main form / presenter don’t know (and shouldn’t know!) how that actually occurs. They want to delegate the responsibility of adding the new employee to a part of the system that does know how to handle it, but they don’t want to be coupled directly to that part of the system.

When Not To Use It

There are time when you know you need to call out to another process and you expect that process to return a known type of information to you. In situations like this, you may not want to use the commands. Commands are set up so that they do not provide any immediate response to the caller. This is done so that the commands be parallelized and / or otherwise handled out-of-process. The knowledge of whether a command is handled in-process or out-of-process should make no difference to the caller.

An example of when not to use a command may be saving an object to a database. You probably need to know if the save was successful or not – if any foreign key violations occurred, if any validation failed, etc. Most systems expect this type of information to be returned from the call, directly. Unless you are working in a larger scale or distributed system, a command may not be suitable here because it would introduce a lot of complexity in getting the return data to the caller, asynchronously.

Another example of when not to use a command would be when you expect multiple handlers for the message and don’t care whether one or all of them are ‘live’ to handle it. If you want to broadcast a message for multiple, unknown subscribers, you should use an Event message.

 

Events / Raise<T>(T args)

The Event Aggregator is used to notify many parts of the system when something has happened, so that those parts and each take the appropriate action. An example of an event would be a customer walking into a restaurant. At the time a customer walks in a host or hostess may respond by greeting them, a server may respond by grabbing some menus and silverware, and a cook may respond by prepping their workstation to make the orders for the customer. All of these responses are occurring because of the ‘event’ – a customer walking in. If no customers walk in, you won’t see the host or hostess greeting no one because they haven’t said their greeting for a while, and you won’t see the cook finishing up a bunch of food for orders that haven’t been placed (well, maybe at some fast food places). Rather, you will see these persons doing tasks that happen on a regular or scheduled basis, not an event-driven basis.

The event aggregator does not do any form of object lifecycle management and has no references to any IoC containers (service locator or dependency injection). If an object needs to be notified of an event being raises, that object must be alive and must have itself registered with the event publisher as a handler for the event in question, at the time that the event is raised. If any of these conditions is not met, then the object that needs to be notified will not be notified. The event aggregator is a true publish/subscribe model in this respect. It does not care if there are zero or many listeners – it publishes the event to any interested listener.

When To Use It

The Event Aggregator is best used in situations where a domain or application level event has occurred, and various parts of the already-running system need to react. The AppController’s wrapping up of the Event Aggregator does suggest that we are dealing with application level events, for the most part. These types of events are needed to coordinate the various application specific parts of the system – the parts that are there entirely for the user or other systems to interact with – and not the domain level events. Of course there is nothing wrong with raising domain level events from the Event Aggregator, either. You may wish to reference the IEventPublisher directly from your domain layer, though, rather than pushing the domain level events from the application layer.

A good example usage is at the end of the add new employee process in the sample application. The event aggregator raises a NewEmployeeAdded even which is caught by the main form’s presenter, and the presenter then refreshes the tree view of the org chart. This may de considered a domain level event, depending on how you see the add new employee service (if it’s part of the application layer or the domain layer).

Another great example of how an event aggregator can help decouple UI from process is found in the main form’s functionality. When you click on an employee in the tree view, the main form’s presenter raises an EmployeeSelected event. This event is then caught by the presenter of a custom user control on the same form – the display of the employee information. This example illustrates how power the event aggregator can be by helping us clean up our presenter->presenter communications, and keep them decoupled from each other. This is definitely an application level event. The entire purpose of this event is to facilitate a part of the application and it’s functionality – displaying the details of the selected employee.

When Not To Use It

The first thing to remember when using the event aggregator is that it is not intended to be used for simple button click handlers and text_changed events, or other simple UI->CodeBehind “events”. The event system built into .net is perfect for handling these simple UI level events. A domain level or application level event is one that allows the system’s functionality to be kicked off, not just at a UI level, but at any level of the system.

Secondly, the event aggregator should not be used when you want to guarantee the execution of something. An event is transient information that is only handled by objects that are currently alive. If you need to guarantee that a message is handled, you should consider using a point to point message like a command.

 

Do Something vs Something Happened

A good way to distinguish between an event and a command is to think about a command as an order to do something (future tense), while an event is a notification that something happened (past tense). If you need something to be done, you want a command. If you want to respond to something that happened, you want an event / event handler.

 

When To Use Workflow Services

Workflow services are the 1,000 foot view of how things get done. They are the direct modeling of a flowchart diagram in code.

When To Use It

Any time you need to coordinate various resources through a specific flow of work, to complete an action or sequence of actions, a workflow service should be used.

When Not To Use It

Technically, a single UI form can have a work flow all on it’s own. How you should move throughout the completion of a single form is a type of work flow. However, a workflow service is geared toward a higher level view of the system than that. In a scenario where a single form is sufficient to accomplish the task, you may be looking at a scenario where a presenter or controller is sufficient.

 

Download The Code

Just in case you missed the links all the way back up at the top, here are the locations of the source code that I am referring to throughout this article.

For standard WinForms applications: http://github.com/derickbailey/appcontroller

For Compact Framework applications: http://github.com/derickbailey/appcontroller.cf

The only significant differences between these two solutions are the forms implementation (full windows vs compact framework) and structuremap is used in the winforms version while ninject is used in the compact frmaework version.

There’s been a lot of fun things happening with Albacore over the holiday weekend.

The Discussion Group

First, there was the Google Groups announcement to coordinate the community. It’s taken off fairly well with our first call for help, and an answer from Steve.

Albacore Dev The Albacore Website

Next, Ben Hall was kind enough to register and host the new url:

http://albacorebuild.net

We’ve got it set to point at the Albacore Github page, and I have some ideas of what we can do with this… it’ll grow over time so be sure to keep that url in mind as the central location to find all things Albacore.

The Albacore Twitter

Then there’s a new twitter account to keep the twitterverse up to date with all the latest and greatest happenings in Albacore.

@albacorebuild The Albacore Tag Line

Don’t forget about the new tag line! I’m quite proud of this one… it’s kinda dumb, but funny at the same time. :)

Albacore: Dolphin-Safe Rake Tasks For .NET Systems The Albacore Logo

And last, but certainly not least for the weekend, is the new Albacore logo that incorporates the tag line. I managed to pull out my rusty old design skills and put together this little gem with the help of InkScape.

As you can see, it’s been a busy holiday weekend for the Albacore crew. I’m hoping that this is just the beginning of a good community building effort, and a lot of support and contributions for the project.

There’s been a whirlwind of activity around the Albacore rake tasks recently. Steve Harman has started using it at VersionOne. Ben Hall has been contributing some great patches and tasks. My friends and former coworkers at my previous job are using it on a number of projects, and there’s still others who are looking at contributing and using the framework. I’m really excited about the possibilities and directions that this is heading.

With all this in mind, keeping everyone informed on what’s going on is becoming more and more challenging, and more and more important. So, I decided to put together a mailing list for the framework over at Google Groups.

Come join the discussion! Get involved! Ask your questions, talk about issues that you’ve found, submit ideas, and send in some patches while your at it.

AlbacoreDev: Discussions, development, and contributions for the Albacore framework.