In my continued playing around with the R data.table package I wanted to find the maximum row based on one of the columns, grouped by another column, and then return back the whole row.
We’ll use the following data table to illustrate:
> blogDT = data.table(name = c("Property 1","Property 1","Property 1","Property 2","Property 2","Property 2"), price = c(10000, 12500, 18000, 245000, 512000, 1000000), date = c("Day 1", "Day 7", "Day 10", "Day 3", "Day 5", "Day 12")) > blogDT[, lag.price := c(NA, price[-.N]), by = name] > blogDT[, diff := price - lag.price] > blogDT name price date lag.price diff 1: Property 1 10000 Day 1 NA NA 2: Property 1 12500 Day 7 10000 2500 3: Property 1 18000 Day 10 12500 5500 4: Property 2 245000 Day 3 NA NA 5: Property 2 512000 Day 5 245000 267000 6: Property 2 1000000 Day 12 512000 488000
I wanted to find the biggest difference in ‘price’ and ‘lag.price’ grouped by the ‘name’ column.
If we just want to get the max ‘diff’ grouped by ‘name’ it’s quite easy:
> blogDT[!is.na(diff), .(max = max(diff)), keyby = name] name max 1: Property 1 5500 2: Property 2 488000
However now we’ve lost the information about which ‘date’ that was on and what the ‘price’ and ‘lag.price’ were which is a bit annoying.
If we only want to keep the rows which have the highest ‘diff’ grouped by ‘name’, one way to go about this is to add a ‘max.diff’ column to each row and then filter appropriately. e.g.
> maxDT = blogDT[!is.na(diff)][, max := max(diff), by = name] > maxDT name price date lag.price diff max 1: Property 1 12500 Day 7 10000 2500 5500 2: Property 1 18000 Day 10 12500 5500 5500 3: Property 2 512000 Day 5 245000 267000 488000 4: Property 2 1000000 Day 12 512000 488000 488000 > maxDT[diff == max] name price date lag.price diff max 1: Property 1 18000 Day 10 12500 5500 5500 2: Property 2 1000000 Day 12 512000 488000 488000
I’ve only been playing with data.table for a few days so if there’s a better way do let me know in the comments.
In case you did not attend Agile2015 Conference this year and your organization is in the process of adopting Agile you may find a recording of this panel discussion inspiring and insightful. Just several conclusions:
Problems at the Agile adoption stage arise generally in one of the two basic situations:
A) The organization is primarily focused on increasing sales & profits, then the problem is a misalignment of objectives of Agile teams and the non-Agile organization. The problem is more at the organizational level.
B) If the organization is totally focused on delivering value and delighting the customer, then the problems might be at the team level.
While all companies are turning into software companies, Agile is about so much more than just software development. Agile is a solution for the whole organization to become more innovative and successful by sincerely focusing on delivering value and delighting the customer. Frequently it denotes that changes are needed in the way the organization is run. However, as soon as there is a sync at all the levels in understanding that the traditional top-down approach is less effective in the long-term than the outside-in (customer-driven) one, this will not only foster the Agile adoption in a particular organization it will also benefit the whole economy:
– the customers for whom the work is being done
– the people who do the work
– the organization itself
– the society
Watch the recording of the panel discussion for more conclusions from:
Melinda Ballou, IT Industry Analyst, IDC
Steve Denning, Author & Management Consultant
André Grard, Senior Analyst, VDC Research Group
Tom Grant, Senior Consultant, Cutter Consortium
Ron Jeffries, Extreme Programming (XP) Innovator, Author, Coach, and Trainer
Jim Newkirk, VP of Service Engineering, CenturyLink Cloud (Moderator)
Someone I don’t know offered to teach me Agile Best Practices recently.
I tend to think there are “generally good practices,” some of which are broadly applicable. In my experience, the search for Best Practices is often a search for Silver Bullets, and a reflect a desire for easy solutions to complex problems. It would be nice to short circuit the difficult work of seeing and evolving a system, building capacity–but I seldom see it succeed.
And, one of my practices is to challenge assumptions. So I challenged my assumptions that there are no best practices. And I came up with some, which I suspect are not what the dear fellow who offered to school me had in mind.#1 Think deeply about the problem you are trying to solve.
First, understand which problem you are trying to solve, for which people, to create which benefit. If you don’t understand this, you are relying on luck for your chosen “solution” to work.#2 Question your assumptions about the causes of the problem.
Assumptions about how work works and how people work will determine the solution space you explore. For example, if someone assumes people aren’t finishing stories by the end of sprint is because they are not sufficiently accountable, he or she will probably not consider the way work flows into the teams, or dependencies between teams.
Notice that I wrote the causes, not the cause. In a complex system, there are likely multiple, entangled influences that result in the problem you observe. Everything touches everything, change one thing and you’ll change many. You can’t anticipate all, but you can predict some, which may change your choice of action.#3 Understand your current system and how it contributes toward the problem, and in what ways it might contribute to solving the problem.
Systems drive behavior. The patterns you observe emerge from the system you have. Sketch what you know about the system, using CDE, DOE, influence maps, reward maps, value stream maps… what ever set of diagrams will help you reason about the system. Use these diagrams to consider which which factors you can change, and what effects they might have.#4 Research at least three candidate actions to improve the situation. Don’t rely on claims by people who are selling “solutions.”
If you can’t think of at least three different possible approaches, you haven’t thought enough. Identifying more candidate approaches almost always improves your understanding of the situation.
Look at how you could influence different factors that contribute to the pattern. Don’t limit your self to comparisons of three similar approaches (should we use Tool A, Tool B, or Tool C?).
Learn from what has worked for other people, but don’t follow slavishly. You need to understand enough that you know where you can modify, and where you need to follow strictly. This sort of learning comes from studying theory, and practicing theory.
There are no silver bullets.#5 Develop experiments to move towards a more effective way of working and improved outcomes.
Big changes feel like existential threats. Small changes support learning.#6 Run experiments, and examine the results. Adjust based on what you observe.
When I took chemistry classes in high school and at university, our “experiments” had an expected correct outcome. Real experiments are about learning. You may learn that you need to increase some skill or create a different level of understanding in order to apply a particular technical practice effectively. You may learn that your architecture is preventing your from benefiting from autonomous teams. What ever you learn, it will help you refine your approach.
Look confirming data, and disconfirming data. Consider what you’ll do to amplify results if things are headed in a good direction, and how you’ll dampen the effect if it reduces function.#7 Work incrementally and iteratively to solve the problem(s).
It’s really not very agile to choose one big bang solution and then roll it out. Try something, learn from it, see how the system changes. You’ll learn more about the issue, have a chance to observe side-effects. Be open to the different possibilities and opportunities that emerge.
These best practices are likely to lead you to an approach that fits your context, your organization, your people. Which will be the best practice for you, not something that worked for some other group in some other context, to solve some other problem. And, since you and your people refined the approach it will be theirs, and they will support it.
Many people and companies have talked about Scaling Agile to DevOps and have tried adopting this approach to software development at different scales. While many companies have been successful in adopting the technique, only a few have been able to scale.
The term DevOps (Development + Operations) was born around 2008 when some used the term to describe “Agile Infrastructure”. But ever since then, the two terms “Agile” and “DevOps” have been used selectively, either together or separately. In recent years some companies have successfully mastered Agile Execution at scale, making DevOps a hot new topic on every CIO’s list.
The Agile Manifesto had it all defined back in 2001 – “continuous delivery of software”, “deliver working software frequently”, etc. In my view of this space, DevOps and Agile are one and the same. Scaling Agile beyond project management, into the realms of operations is DevOps. If you viewed software development as a supply chain, DevOps is Downstream Agile. To effectively be an Agile enterprise it’s important to have working software in frequent releases. How can software work without being deployed into a server and tested?Are you DevOps ready?
Many companies have jumped into the DevOps bandwagon today. These companies have spun initiatives around Agile adoption and DevOps. While it’s a good sign that these enterprises are wanting to embrace change, it’s scary to see so much money spent without a clear plan. DevOps is not a magic potion. There are 5 things that these enterprises need to consider before jumping into the Agile and / or DevOps bandwagon:
- Is there a commitment from the top leadership to become an Agile enterprise?
- Does the organization have shared goals across departments?
- Has the organization identified the key processes that deliver the most business value?
- Have bottlenecks in these key processes been identified?
- Take an Enterprise Agility Assessment
Scaling Agile to DevOps requires enterprises to take a deeper look at the driving need. Business value for enterprises can vary. For cloud-based consumer businesses the business value could come from delivering software to smartphones or the cloud. For enterprise businesses the value could come from delivering high quality software to the cloud or made available for download. In both cases, the frequency at which business value is created could vary anywhere from a few minutes to a few months.First steps to DevOps
If your enterprise is convinced to pursue a DevOps or Agile Execution strategy, and if there is a strong leadership commitment, here are a few things you should consider doing:
- Identify key processes that create the most business value
- Consider re-aligning your business functions around delivering business value
- Define your DevOps or Agile Execution strategy with clear milestones
- Commit the necessary budget to establish the infrastructure needed
If your enterprise creates business value by delivering world-class software, it’s important to realize that business value consistently and predictably. An organization’s functions have to contribute to adding value to the business, and not to the functional goals. For example, the role of Operations or Quality Engineering teams should look at getting high quality software our of the door as the business demands.
DevOps or Agile Execution can come in different sizes, and one enterprise’s needs could be very different from another’s. Depending on how mature your enterprise is with Agile, you could have shorter milestones to push the envelope. Good unit testing, functional testing and performance testing is one tiny step towards getting Agile.
Having dedicated test environments that run automated tests for continuous testing, having continuous integration (CI) jobs to create build software is the next stage. Based on what the business demands, your enterprise could choose to further the DevOps pipeline to continuous delivery (CD) or go further and do continuous deployment to stage or production environments.Challenges and Opportunities
As enterprises scale across multiple projects, teams and geographies, one of the most important challenges you face is one of tool chain complexity and data. With developers making commits by the minute, CI jobs creating builds every few minutes, CD pipelines deploying into a multitude of servers every few minutes, enterprises will soon end up with a problem of too much of data.
It’s important that enterprises get into a DevOps or Agile Execution initiative with the longer term vision and invest in the right set of tools. With the right strategy in place and the right tools, enterprises can actively engage with data to improve overall agility. The ability of these tools to offer clear end-to-end traceability and insight across requirements, commits, builds, deployments and servers is key to your enterprise’s continued success with DevOps.
These tools are primarily used to:
- Build the sprint backlog
- Update the estimates during the sprint
- Display sprint backlog details during the daily scrum
- Spit out the sprint burn down
One of the primary aims of Scrum is the boost in productivity seen with teams that have a sense of ownership in how they run their sprints. Teams with ownership are accountable for their commitment. They enjoy their work more and they explore ways to improve.
Sprint tracking tools can get in the way of that by:
- Not allowing an "all hands" approach to managing the sprint backlog: one mouse, one keyboard, one operator.
- The data in the tool is only brought out once a day because there are not enough licenses, developers don't want to learn the tool, or the extra effort isn't worth it.
- The team's ability to customize the artifacts (task board, burn down, etc) are limited by what the tool can do.
- The metrics of the sprint (sprint velocity, burn down), and even individual developer progress can be monitored by management. Guess what happens when a developer or team is asked why their burn down is not diagonal enough?
- Making the daily standup a status reporting meeting by focusing on individuals.
Also, teams need their sprint backlog to be radiated at them. Tools refrigerate the backlog by storing it in some cloud.
When I bring this up, I often hear from the Scrum Master that the tool makes it easier to do their job; tracking the sprint and producing the burn down or that there is no wall space. It's often news to such Scrum Masters that their role is not to be efficient creating the artifacts, but helping the team build ownership and trust...or just tracking down solutions, like a portable task board. I don't blame them necessarily. Adopting a tracking tool for sprints are often an interpretation of how Scrum should work in an amber or orange organization. Teams in these orgs who are new to Scrum might even want tools in the sprint so that the "Scrum fad" has minimal impact on them.
The sprint backlog is owned by the developers on a team to help them organize and manage their shared commitment and forecast for achieving the sprint goal. It serves no purpose external to the team.
Courage, commitment, focus, openness and respect aren't always easy to instill, but it can start with a bit of tape and some index cards.
At the SonarQube Geneva User Conference last week I learned that 7 of the Fortune 10 companies and 47 of the Fortune 100 use the SonarQube platform. We’ve got an estimated adoption of 50,000 companies overall (based on the number of unique IP’s that hit the update center). The figures really struck me because I’d never realized before how mainstream the platform is.
At my previous job, there was a lot of discussion about adopting a standard tool set. The logic went that standard tool sets are easier to hire for than home-grown frameworks. Not only that, but it’s easier to keep up with the pace of change if you’re using the standard; for any given technology change, you simply do the normal upgrades everyone else is doing, rather than having to re-engineer your stack, and wonder if you’ve covered all the bases.
That’s why an adoption of 50,000 companies – more than 20 times the adoption rate of our all our competitors put together, I’m told(!) – caught my attention. If you assume an average of 40 developers and development managers at each company, that’s 2 million people using SonarQube today(!). And that doesn’t even count the contributors to open source projects that track their code quality on nemo, the demonstration instance of SonarQube.
That means that if your company is using SonarQube, your chances are good that any new hires will already be familiar with it. You’ll have to train them on your expectations, but not on the tool itself.
At my last job we wanted to adopt the standard tools. At SonarSource, we are the standard tool. That’s pretty cool.
One of the problems that I have often run in to with building re-usable components in an application, is the ugly require statements that I have to use:
The deeper the folder structure, the worse it gets! So how do we get rid of this ugliness?
Reduce The Ugly
My goal is to reduce require statements shown above, into a simple statement like this:
This is a problem I’ve already tackled in my node apps and I want to solve it for my browser based modules with Browserify, too.But, I Don’t Wanna NPM It!
At first glance, I thought my only option was to publish a module into NPM. While this is a valid option, it’s not one that I want to do most of the time.
Typically, when I’m building a component within my application, it is for this application only. I don’t want other people to see it, and I don’t want other applications that I’m building to see it, either.
I know I could use private NPM modules for this, but doesn’t really fit either. My components for an application are only for that application.And I Don’t Want node_modules, Either
There are other options for having NPM “install” a module in to the node_modules folder, by pointing at a file system… but this presents another problem of semantics. In this case, I’m writing code that is strictly for the front-end browser. This isn’t node code, nor is it unimorphic isoversal JavaScrode.
I don’t want my front-end components in my node_modules folder because they don’t belong there. I want my front-end components in a folder structure that is organized for my front-end code.Browserify’s “paths” To The Rescue
Fortunately for my sanity and my code, I found my answer in the “paths” setting of Browserify. I was able to adjust my Grunt file to include this settings, like this:
With Browserify pointed at the “common” folder of my application, I can place a named sub-folder with an index.js file in that sub-folder, and include the component in my app easily!
Now, instead of using using all those ugly “../../../common/whatever” relative paths, I can just require("myComponent"); and I’m done.
After a recent IntelliJ upgrade I’ve been running into the following error when trying to attach the sources of any library being pulled in via Maven:
Unable to import maven project
And this is where you need to add it:
Cmd + , -> Build, Execution, Deployment -> Build Tools -> Maven -> Importing
As agile has grown in popularity, so have the misconceptions about it. In a recent introduction to agile webinar we asked our audience which of these common myths they’d encountered — here’s what they said:
Do any of these look familiar to you? Let’s dive into these a bit and separate truth from fiction.Agile = Scrum
Quick: what’s the first thing you think of when you hear the word agile? For many of us it’s a “standup” meeting. Or maybe it’s creating a “backlog” of user stories that get delivered in a two-week “sprint.” Fact is, these are all elements of Scrum, a popular project management methodology used by many agile teams. So Scrum is a framework for developing and managing work, while agile is an umbrella term for approaches like Scrum that share a common set of principles — but this isn’t about the semantics.
Scrum is just one of many methodologies that build on agile principles (others include Lean, Kanban, Test-driven Development, and Xtreme Programming). Simply “doing Scrum” doesn’t make you agile.
To “be agile” means focusing on customer value and quality, delivering value incrementally, limiting work in process (WiP), enabling cross-functional collaboration, and striving to continuously improve. When agile fails to take hold or improve results within organizations, as Rally VP of Product (and former coach) Ryan Polk has discussed, it’s often because it only lives in pockets at the team level or because it’s executed poorly. Sure, scaling agile more broadly means mastering the fundamentals of Scrum, but it also means adopting an agile mindset in how your teams, programs and departments work together and communicate with each other.
If you haven’t already read the Agile Manifesto and its 12 principles, and thought about what “being agile” (not just “doing agile”) would look like in your own organization, there’s no time like the present.Agile Means “We Don’t Have a Plan”
Equating agile with a lack of planning is one of the more insidious agile myths. We know its likely origins: the misguided observation that agile at the developer level seems allergic to functional specs, and the belief that adapting to change means abandoning whatever plan you did have.
Well, maybe you’ve seen the iconic “planning onion” before?
While it’s true that agile approaches trade the lengthy, upfront planning process for a more adaptive, iterative process, planning is a very important agile tenet. It’s just that instead of making one grand plan, once a year, and hoping that it’s still on target six months later, you’re dividing your planning (like the work) into smaller chunks and committing to reviewing your plans on a regular basis. Why is it better to plan, and then plan to re-plan?
- Short-range planning helps you control WiP, which means you can deliver more value, faster.
- Midrange planning helps you communicate, make visible, and prioritize the most important work, ensuring small iterations add up to high-value increments.
- Long-range planning lets you take the feedback from your incremental delivery and use it to steer the business in the right direction.
This might be one of the most fear-provoking misconceptions about agile — at least if you’re a project manager in a non-agile environment — and it reflects a fundamental misunderstanding about the roles on agile teams, which skills are most valued, and the fact that changing your collaboration dynamic can make people work more efficiently, happily, and productively.
The truth is that agile needs project managers — but it’s likely you won’t be called a project manager (you’ll probably have a title like Scrum Master or Product Owner), and it won’t be your job to tell people what to do or when to do it. That’s because agile approaches value self-managing teams — whose success relies on collaboration, localized decision making, regular communication, and tools for visualizing and sharing WiP.
For example: are you good at working with stakeholders, managing expectations, and balancing competing priorities? Do you have a strong sense for business value? Are you comfortable driving vision and strategic direction, while collaborating with a team to make day-to-day tactical decisions? Then you might be a good Product Owner. Do you understand people – their motivations, fears and desires — and group dynamics (what makes a team better than the sum of its members)? Do you enjoy problem solving for people so they can do their best work? Then you might be a good Scrum Master. (Learn more about how traditional project management maps to agile project management.)
Multiple analyst firms predict that agile approaches will dominate other methodologies in the coming years, and software-driven innovation means agile is gaining a foothold across nearly every industry. The Project Management Institute has found that the use of agile methodologies is up 27 percent from just two years ago, and a PricewaterhouseCoopers survey found that nearly two-thirds of these agile projects are managed by certified agile practitioners. And even though it’s a bit of a misnomer often used by companies transitioning to agile from waterfall practices, search for “agile project manager” on indeed.com and you’ll see that this is an in-demand role.Agile Doesn’t Believe in Documentation
It’s true that the Agile Manifesto values "working software over comprehensive documentation." However, this doesn’t mean documentation has no place in agile approaches. It’s just that instead of assuming a project and its associated processes require comprehensive documentation, you should undertake and design project artifacts because they provide value. Rather than drafting pages and pages of requirements and use cases and process rules, for example, think about the minimum viable information that needs to be captured, with whom it needs to be shared, how to document it in a collaborative way, and how that documentation might help you continuously improve.
Think about the roles on a delivery team — how people spend their time and how the tasks they perform correlate to a project’s success — and you’ll probably conclude that success maps directly to the time spent designing, building, testing, and shipping … not the time spent documenting all those processes.
Some roles will rely more heavily on documentation than others: for example, our UX designers do a lot of ethnographic research and customer interviews before they work with teams to design and build features, and their results need to be codified and shared. And when we do sprint or planning meeting retrospectives, we like to document what we liked, learned, lacked, and longed for, because capturing that information helps us improve over the next cycle. The point is that documentation isn’t the point in agile; it’s a byproduct of the actual work, and like the work, it should be designed to deliver value.Agile Only Works for Developers / Software
Agile may have started out in the technology realm, but like any set of effective practices, it has evolved and taken hold across a much broader audience. From finance and healthcare to communications and manufacturing, non-software companies — now finding themselves driven by software — are using agile approaches to improve their delivery, customer experience, and innovation. Organizations buoyed by the success of agile in their IT or engineering groups also have invested in extending agile principles and approaches into the executive suite.
Agile principles are now being applied to marketing, legal and human resources departments, and McKinsey recommends that companies should borrow agile approaches from their IT departments to build agility into their company DNA. No matter your organization type or your current way of working, wouldn’t you like to be faster and more transparent? Wouldn’t it be great to know that you’re delivering what customers really want? Don’t you want to know that you’re doing the highest-priority work and that you can reprioritize when your competitive, economic, or strategic landscape changes? Agile companies do this.Agile Will Fix All Our Problems
You know better than to fall for this one, don’t you? No one thing is going to fix ALL your problems. When leaders have problems that need fixing, however, they tend to search for a silver bullet. Pinning all your hopes on a “magical methodology,” however, is going doom you to disappointment. Just as doing standups or buying an agile tool doesn’t make you agile, integrating agile practices into your IT group isn’t, by itself, going to turn your enterprise company into a nimble startup.
If you think of your company as having a fitness goal, then adopting agile approaches is like a workout. Exercising your agile muscles takes discipline and commitment, and if you do it on a regular basis, it can make you much stronger. But to meet your larger fitness goal you also need to change other habits: let’s say, eat well and get enough sleep. Similarly, your agile muscles become even more powerful and effective with supportive infrastructure — like the right training, a commitment to change, and engaged leadership.Bust Your Own Myths
If you’ve encountered any of these myths about agile, it might be time to brush up on what you think you know. We’ve put together some resources for you to do this easily on our Discover Agile Toolkit page. Learn how to make the business case for agile in your organization, hone your agile practices, get the scoop on role-based training, and take agile to the next level. Or check out the Discover Agile webinar recording to get an introduction to agile, Scrum, and the ROI you can expect at your organization.Jenny Slade
Why engineers and organizations are turning to agile methodologies to build the connected device
Every day, every hour, every minute, devices are becoming connected. The Internet of Things (or Iot) has gone from a buzzword floating through the echoes of Silicon Valley to a tangible and quickly advancing movement in the world of hardware. But, like a stone tossed into a pond, the effects of the Internet of Things are quickly rippling outward, causing immense changes to the environment surrounding it. And, as it turns out, the IoT is a rather big stone and it’s making a large splash. The IoT revolution is necessitating enormous changes to companies who want to play in this brave new world – and those companies are not few in number.
In fact, organizations across the globe are looking to leverage the power of the IoT revolution for their products. Indeed, companies are looking at their products both new and old, and how they add value in this new environment. The IoT is finding a comfortable spot in product roadmaps worldwide.“software now accounts for over 50% of embedded system value and development costs.”
While there are many benefits to the IoT and the way it is changing the world of embedded systems, it is also shifting the focus of product development. In a recent survey of over 500 embedded system engineers, responders said that software now accounts for over 50% of embedded system value (as well as over 50% of development cost).
One of the most significant impacts of the IoT is that organizations are seeing a noted change in consumer behavior and expectations. For nearly all consumers, their first experience with a “connected device” started with their iPhone, Android or other smartphone. With this as their first experience with the connected world, users are demanding their other smart and/or connected devices deliver experiences and services similar to those of a smart phone. This means updates, upgrades and a slick and sleek user experience.
Not only do consumers expect “more,” but they are coming to expect “more, faster.” Rate of change or upgrade is increasing rapidly. Consumers have an expectation of frequent product releases and feature upgrades – and this expectation puts a pressure on both the fundamental product design and the support system behind it.
A complication of this change in consumer expectations is the unintentional increase in product and security flaws as the complexity in designs increase. This can happen due to many manual steps that are often skipped during the development process and low-priority items fall off the radar.“Now with IoT, you can’t just ship it and forget it.”
In the past, the development process stopped with deployment. Meaning that once a product was released into the wild, that product was finished from a development standpoint. But now with IoT, you can’t just ship it and forget it. The growing expectation for post deployment content and updates mean that organizations need enhanced change and management solutions – and – they need to think about testing over a greater development cycle. All the traditional mindsets of bringing a product to market are changing, blending and extending.
At this point, you might be thinking “this sounds like a lot of problems, but no answers.” Fortunately, there are solutions to the challenges presented by the Internet of Things. One of the main ways organizations are trying to navigate the IoT is through the implementation of agile development methodologies. While serial and waterfall methods have been widely used for decades, those methodologies are crumbling under the enormous weight of the Internet of Things.
Agile methodologies – on the other hand – including continuous development and continuous delivery, are particularly well equipped for dealing with the demands of the connected device. With Agile, frequent updates are essentially a requirement – which satisfies both the end users’ desire for a constantly updated device as well as the developers need for a manageable development schedule.
Agile methods also support the use of application release automation. This is useful in the world of connected devices for several reasons. First off, it means that many tests that are required for security purposes in IoT can be automated, which leads to a reduced risk of security flaws in the end product. Furthermore, by automating updates, it frees up developers to create new features and functions for their products.“Agile methods give Developers freedom to create new features and functions.”
Another issue engineers are seeing is that, in traditional development, development tends to happens in silos, which each team focusing of their specific function necessary to carry the project to completion. However, in the IoT world, consumer expectations are fueling the need for more and more cross-collaboration between project teams. The agile methodology truly focuses on collaboration between teams. By harnessing the power of newly available tools – mainly the idea of manipulating and debugging system virtually – teams are able to collaborate on products in never-before-seen ways. This leads to shorter lead times, faster overall development, and more product updates and releases.
This ability to simulate tests is critical to the performance of agile within the Internet of Things. That is because it is difficult to test a connected device within the context of the full system. But with the power of simulations, organizations are able to automate any function within the simulator and then test fully. They can scale and reconfigure as needed and then send the context of test failures to developers so they can solve the issues at hand.
As the demands of the Internet of Things continues to increase and consumers expectation follow suit, it is becoming more and more imperative for organizations to operate as efficiently as possible. Agile methodologies – and continuous delivery and development – are ideally positioned solutions to address these increasing challenges and difficulties. With the tools and best practices of agile, developers and embedded engineers alike are fully-equipped to jump headfirst into the dynamic world of the Internet of Things.
Programming for Immutability: https://schneide.wordpress.com/2015/09/28/thinking-in-immutability/
Systems Thinking: http://www.samihonkonen.com/14-habits-of-a-systems-thinker/
I regularly delete some of my unit tests, sometimes within minutes of writing them. Even as a TDD fanatic I’ve come to realize tests are just a means to an end. If I decide to write a constructor test when first designing a bit of code and delete it a few minutes later, nothing’s wrong. These sorts of initial tests can become redundant quickly since to you have to construct the object for all the later tests.
As you adjust to TDD there’s a tendency to see the tests as important code. Deleting them is the last thing you’d think of doing. It turns out just like with production code less tests are better easier to maintain and easier to refactor. And as an added benefit you test suite runs faster. Tests are path to good code and you get an added benefit of a regression suite. And besides everyone’s favorite checkins are ones where you remove more lines of code then you add.
In the production industry, software is very often the breaking point in the product lifecycle. Of course, the question raised is how to optimize software development and how to shorten time to market. The amount of software contained in “classic” products is substantial.
A luxury car contains up to 100 million lines of code! Even a simple, computer controlled household appliance can contain up to 1 million lines of code. Medical devices contain up to 3 million lines of code and a jet fighter contains up to 6 million lines of code. Surely you are using a Product Lifecycle Management System (PLM-System) like Dassault Enovia, PTC Windchill or Siemens TeamCenter for controlling the product LifeCycle. But how do you control the development of the software, which is part of your product?
Do you use your PLM-System for managing your product’s software? Maybe you are using an ALM-System (Application Lifecycle Management-System) for managing your software? If yes, this is great! In case your company is part of that 3% of enterprises who have connected the ALM-System to the PLM-System then you can stop reading now.
If you are still reading, this means that there is a very high chance that your enterprise belongs to the remaining 97% of companies who handle Product Lifecycle Management and Application Lifecycle Management isolated from each other – even though both outputs will become part of one product.
To achieve a synchronized effort for both, use the best of breed tools and connect software development with the production processes of your product.
The image above showcases where the production of hardware and software is separated in the PLM process. The processes for producing hardware and software are very different. It begins with the project methodology: Hardware is produced in projects using the waterfall or V-Model as project methodology usually. Producing hardware is highly automatable using CAD/CAM systems and CNC machines, but the software development process contains a lot of manual interactions even though the right tools for automation are available.
Software development in isolated environments with lots of manual interaction not only harms traceability (Important for archiving compliance!), but also the protection of intellectual property and quality of work, and therefore the quality of the entire product. Manual interactions always lead to a high risk of human failure, hence why this automation in the software development processes is duty, not luxury. For archiving this, a co-operation of tools is unavoidable. This starts with requirement management and moves from coding and the build process to automated testing and deployment of the software product back into the chain of the product lifecycle.
Most important for automation is the connection of point tools used for software development. This is where CollabNet’s TeamForge comes into the picture. TeamForge connects the point tools and creates an automatable and traceable production chain for software. Permission control and role based access control containing single sign on for protecting your intellectual property and an inter project reporting are wanted side effects. This allows the developer to continue working with the tools they are experienced with without affecting their day-to-day tasks. TeamForge’s interfaces allow the connection of most of the point solutions and their integration into the PLM process.
TeamForge becomes the dashboard for the software development process and monitors sub processes and progress. With TeamForge, it doesn’t matter if software is developed following agile or waterfall and if there are one or multiple Git or Subversion repositories.
Following this example, the most important integration is the one from TeamForge into the PLM system. A bidirectional data transfer for transmitting requirements from the PLM system to TeamForge as the ALM system and vice versa is key here. This includes- and is not limited to- the deployment of the produced software, status reporting, amounts of effort, etc. The information transferred from TeamForge to the PLM system are key for a successful integration of PLM and ALM processes, and therefore optimizing the product lifecycle.
An example process could be
- A requirement for a new product (part) of type of “Software” is created inside the PLM system.
- Either a new project will be created in TeamForge for this requirement or information is provided for which project the software part should be assigned to. The specification, a definition of an interface for example, will become a part of a User Story or an Epic in TeamForge and this can be assigned to a Project Manager or a Software Architect.
- The usual software development process containing coding, build and test, is started in TeamForge .
- If the created software passes the final tests it will be deployed to a binary repository and a message will be sent to the PLM system that the task is completed with details of where to find the software. The source code will be deployed to a source code repository for protecting the intellectual property.
- The task will be marked as “completed” in the ALM System. In the PLM system the software can be burnt on to a chip to build into the product and bring the new “part” to production.
Because a PLM system is implemented differently in every Enterprise, the adaptation of TeamForge to the PLM system is always an individual solution. This allows a perfect integration and top level support for the PLM process.
Through the implementation of ALM by TeamForge into the product lifecycle, software development is accelerated and time to market is decreased, saving budget and time are wanted side effects. Automation of processes allows more effective planning and shorter release cycles for software. Project planning becomes more reliable using TeamForge’s reporting and because of tool integrations, each step in the process is much more reasonable.
As a consultant I am doing a lot of things, so to keep up I have always used some form of a TODO list. The reason why I did this is because it helped me break down my tasks in to smaller ones and keep focusing, but also because I kept remembering the quote I once heard “smart people write things down, dumb people try to remember it”.
Years ago I read the books “Seven habits of highly effective people” and “Switch”, in my research in to how to become more effective I came in to contact with GTD and decided to try it out. In this post I want to show people who have heard about GTD how I use it and how it helps me.
How do I use GTD?
Because of my experience before GTD I knew I needed a digital list that I could access from my phone, laptop or tablet. It was also crucial for me to be able to have multiple lists and reminders. Because of these requirements I settled on using Todoist after having evaluated others as well.
I have couple of main groups like Personal, Work, Read-me, Someday & Maybe, Home etc.. In those lists I have topics like Xebia, CustomerX, CustomerY etc.. And in those topics I have the actual tasks or lists with tasks that I need to do.
I write everything in to my Inbox list the moment a task or idea pops up in to my mind. I do this because most of the time I don’t have the time to figure out directly what I want to do with it so I just put it in to my Inbox and deal with it later.
As a consultant I have to deal with external lists like team scrum boards. I don’t want to depend on them so I write down my own tasks with references to them. In those tasks I write the things that I need to do today, tomorrow or longer if it is relevant in making decisions on the long run.
Every day I finish my work I quickly look at my task list and decide on priorities for tomorrow, then every morning I evaluate my decisions based on the input of that day.
As a programmer I like to break down a programming task in to smaller solutions so therefore sometimes I use my list as a reference or micromanagement of tasks that take couple of hours.
To have a better overview of things in my list I also use tags like tablet, PC, home, tel so it helps me pick up tasks based on context I am in on that moment.
Besides deciding on priorities every day I also do a week review where I decide on weekly priorities and add or remove tasks. I also specify reminders on tasks, I do this day based but if it is really necessary I use time based.
Because I want to have one Inbox I am sticking to zero mail in my mail Inboxes. This means every time I check my mail I read it and then either delete it or archive it and if needed make a task to do something.
What does GTD do for me?
The biggest win for me is that it helps me clear my mind because I put everything in there that I need to do, this way my mind does not have to remember all those things. It is also a relaxing feeling knowing that everything you need to do is registered and you will not forget to do it.
It gives me structure and it allows me to make better priorities and decisions about saying yes or no to something because everything is in one place.
Having a task registered so that you can cross it off when it’s done gives a nice fulfilling feeling.
Having a big list of things that you think you need to do can be quite overwhelming and in some cases it can also feel like a pressure because you haven’t done a list of tasks that is there for a long time. That’s why I keep evaluating tasks and remove things from it after some time.
Task lists and calendars can sometimes collide so it is importent to keep your agenda as a part of your list although it’s not a list.
What am I still missing?
Right now from the GTD’s point of view I miss a better time management. Besides that I would like to start a journal of relevant things that happend or people I spoke to but also incorporate that with my GTD lists (for example combining Evernote/OneNote with my Todoist lists).
Everything else I miss is technical:
I miss a proper integration of all my mail where all the calendar invites are working without having to connect all accounts on all different devices. I also miss a proper integration between my mail and Todoist for creating and referencing mail.
Because I write down every task/idea that pops up in to my mind it would be great if voice recognition would work a little better for ‘me’ when I am in a car.
GTD has helped me structure my tasks and gave me more controle in to making decision around them. By writing this post I am hoping to trigger somebody else to look in to GTD and maybe have an impact on his or hers effectiveness.
I am also curious in what your opinion is on GTD or if you have any tips for me regarding GTD?