Team John, part 2

Yesterday's post ended a bit abruptly while I had a lot more to say on the subject of Team John. The reason for this is a silly one, but this is just a silly blog, so I'll do with it however I will, including using run-on sentences as I feel like it: I am trying to get back to using 750words.com. It's a great site that simply tracks the number of words as you write them to encourage you to write at least 750 words each day. I'm picking this back up, as I have been reading Stephen King's marvelous "On Writing". King encourages people that want to write to start with at least 1000 words a day. Being more of an underachiever, I'm starting by going back to 750 each day. I'll have more to say on Mr. King and his book hopefully in coming days and weeks. The 750 word goal also works to make sure that I don't spend an inordinate amount of time writing drivel and sometimes even forces me to stop when I have more to say.

So I have more to say on Team John. In the previous post I talked a little about how we implemented it with the current team. Other than what I think were good ideas of tying it to our oncall schedule and having two developers on in a rolling fashion, it probably wasn't exactly helpful in understanding much beyond the message that I think Team John is a good idea for teams. I did hit on the primary purpose that Team John shields the team from interruptions and interruptions are very costly to teams. As an Agile, or more specifically a Lean practice the idea of why the Team John works goes beyond just funneling interruptions  away from the main development team and it goes to queuing theory.

Queuing theory is really the study of how queues or "waiting in lines" affects system throughput. As a study it was first used in telecommunications and then was adopted by Toyota in manufacturing and then came from there to software development through Lean teachings. As a practice it is as old as when people first began lining up to wait for something.It's helpful if you think about waiting in line at Wendy's or at the local grocery store. Will opening more registers help people waiting in line move faster? Yes, it will, but up to a point, right? I can open up four registers at Wendy's, but if I only have one person back on the grill, I can only move people as fast as burgers come off the grill. Or think about those videos you've seen of an automotive assembly line. The line moves along at a regular pace with each man or woman doing the job at their station. Highly repetitive, but also very precise - not only in the placement of each bolt but also in the time the entire task takes.

Applying just those two ideas, it's easy to see that the entire process can only move as fast as it's slowest component (we call that a bottleneck) and that a sustained pace is better than randomly throwing work at people. Like I said, these aren't new concepts but have been around at least as long as people have been waiting around for hamburgers. So how does Team John help the process?

First, when a Scrum team has planned out a sprint, they have planned an amount of work that is sustainable if they work a regular pace throughout the sprint. Unplanned work coming to a development team means a) it will be of some volume that the team couldn't previously deduce and b) it will have some deadline that the team couldn't previously expect (presumably sometime within that sprint, however). There is a rhythm to a sprint, particularly when a team has a sprint goal and is doing a similar group of repetitive tasks towards that goal. Unplanned work throws that cadence off kilter.

Secondly, Team John helps teams deal with those bottlenecks. That guy on the grill can become a bottleneck. If you picture the auto assembly line again, it's easy to see that the line can only move as fast as the slowest person working it. The other thing to realize, particularly in software development but also in fast food service, is that you can never completely predict how a relatively short run on the system will unfold. One hour I might have everyone order singles with cheese, a Frosty, and a small fries. The next I may have people each ordering custom orders ("Hold the ketchup, hold the pickles!"), one of a variety of beverages, and all wanting large fries. What if I have one or two people just waiting, ready to place toppings or fill beverages or put down more fries as needed? I wouldn't need to always anticipate exactly where those bottlenecks are going to be.

I probably have more on the subject, in fact I know I have more, but I'm at almost 850 words now and need to leave some for tomorrow.

Team John, part 1

I was introduced to Agile concepts around 2004 or 2005, primarily eXtreme Programming ideas from articles by Kent Beck and Ward Cunningham. Fairly shortly thereafter, 2007 or 2008, I attended my first Mile High Agile conference here in Denver. At either that first one, or possible the next year (my memory is starting to go), I was introduced to the very energetic V. Lee Henson and a number of his really great ideas in implementing Agile including the "Team John" concept. Briefly, a software development Team John is an engineer who works separately from the rest of the development team on bugs, production incidents that arise, work that comes in sideways last minute ("Hey, can you pull some numbers from the database for me?"), or tech debt. The idea is that developer works on all the other shit that isn't part of the team's sprint work that would otherwise lead to interruptions. The responsibility rotates to a new developer with a new week or sprint.

It seems counterintuitive that removing a developer from the mix would allow the rest of the team to go faster, but I think any development team that regular needs to deal with incidents or gets hit with work sideways, from outside the Scrum process will immediately see the value. Interruptions kill a developer's productivity. That loss in productivity is attributed to context switching: The very act of switching from one task to another results in a tremendous amount of lost productivity. Psychological research has shown that context switching can result in 40% productivity loss, and the more complex the tasks, the greater the loss. 

In his blog, Joel Spolsky, the creator of Trello and one of the founders of StackOverflow, puts it this way, "The trick here is that when you manage programmers, specifically, task switches take a really, really, really long time. That’s because programming is the kind of task where you have to keep a lot of things in your head at once. The more things you remember at once, the more productive you are at programming." Spolsky goes on to sum up, "In fact, the real lesson from all this is that you should never let people work on more than one thing at once."

The primary purpose of Team John is to shield the team from interruptions or at the very least funnel them all to a single person. With one of the teams I currently support we were getting a number of ad hoc requests coming into our sprints sideways, whether from our business partners ("Hey, can you get some numbers for us so that we can see how the latest campaign is working?"), or elsewhere within the technology department ("These three servers have critical updates and need to be patched by Friday"), or from within our own group ("This service is causing alerts. Does anyone even know what it does?"). Then you had production bugs and incidents which had to be addressed right away and suddenly we would have three developers dropping what they were doing to look at a single bug. I brought up the idea of using the Team John concept - we already  had an oncall rotation for who would handle incidents outside of business hours. We decided whomever was oncall would also handle the interruptions that arose day-to-day. They could work on regular sprint work if there were no interruptions, but at least each individual developer wasn't being pulled away almost hourly.

We quickly discovered that there was a lot of Team John work once we all started funneling it to a single person. Not enough work to keep that one developer busy the entire time, thankfully there weren't that many bugs and incidents, but enough that they really ended up contributing very little on the actual sprint. Meanwhile the rest of the team was able to accomplish more work that it had in previous sprints. I think there were six or seven developers on the team at the team. By taking one out of the mix to handle interruptions our velocity increased by roughly 15%. 

Rather than have Team John work on sprint work intermittently, we had the developer begin to pick up some of the technical debt we had accumulated and try to pay that down in between other interruptions. As we began doing that, some Team John work would carry over to the next week (our oncall schedule rotates weekly) leaving the next person in the rotation without much context for what was being worked on. After a while, and after adding a couple more developers, we decided that both the primary and secondary, which was always the previous week's primary, oncall developers would both be on Team John. This way as the primary rolled over to be secondary, they could bring the new primary up to speed on what was being worked on. With the addition of the second developer on Team John I don't believe the velocity increased, but it definitely did not decrease.

Why You Shouldn't Point Tech Debt, Take 2

I want to expand on not pointing technical debt stories. As I finished the last blog post, I realized that I had made the presumption that teams should only assign story points to stories based on business value. Then I proceeded to basically lay out the argument for why a team should (or in some cases should not) pay down their technical debt, kind of skirting the entire question of what not assign points in the first place. A classic case of begging the question: We only point things that are valuable because they are things that are valuable. If you already have bought into that statement, you wouldn't be here asking the question of whether to point tech. debt or not.

So now I want to back the premise that a team should only point work that is of value to the business. One of the developers I support is a fan of both metaphors and the NBA, so I would like to present my argument in these terms: When you watch a basketball game rebounds, assists, steals, and turnovers are all very important to the teams. However, at the end of forty minutes, the one stat that matters is how many points each team scored. Just like having a lot of turnovers will hurt a team's chances to win, technical debt is going to drag down your development's ability to deliver new features. In the end, however, the business runs on the value of the features the team delivers.

I mentioned previously that a team may look to distinguish between technical debt points and business value and still somehow combine those to estimate output. I don't believe that this is really an apples-to-apples comparison, however. Some technical debt MUST be addressed and some subset MUST be addressed IMMEDIATELY (particularly if bugs and incidents fall into that technical debt bucket). For example, our organization recently moved its package management from a Nexus repository to Artifactory. Every development team had to change projects to point to Artifactory. No value was delivered to the customer, but this had to be done by a set deadline else project build pipelines would cease to work. We also have contractual obligations for SOC 2 and PCI compliance driven by hard deadlines. Managing API keys and updating network diagrams does not deliver value to our customers but must be done and must be done by particular dates. If that date is a couple years off the tradeoff with choosing to bring in business value may be relatively small. That cost increases, however, as the deadline approaches.

Another thing that makes estimations of technical debt unequal to estimating business value is that technical debt, in my opinion, is generally more difficult to accurately estimate. I say this because if it were easy to accurately assess the risk, effort, and value of paying down debt that teams would actively engage it rather than accumulating it. I believe technical debt accumulates, in most cases, precisely because we find it difficult to assess risk, effort and payoff. "If it were easy, we'd have done it." I know me saying this is my own opinion based on experience and that in some cases teams know exactly the risk and effort and the value simply isn't worth addressing the debt. That said, this is one more reason why, in my experience, I do not believe teams should point technical debt. To extend my metaphor further - a made basket in basketball is worth a set number of points: one for a free throw, two for a standard field goal, and three for those beyond the three point line. A steal, like paying off technical debt, may result in points, and it may not. A coach can't tell his team, "If you get this many steals we are sure to win the game." He can look Jimmy Chitwood in the eye though and say, "Jimmy, if you come off the picket fence and make this shot, we win the game." (That isn't exactly how it happens in "Hoosiers" and I realize I've taken this too far, but I can't get this far into a basketball metaphor and not bring up one of the greatest sports films ever made.)

A final reason why I do not see technical debt and business value estimates to be equivalent is that a team can timebox paying down technical debt. For user stories delivering business value there is some discrete level of effort involved in delivering that value. Generally a user story that is 80% done will deliver 0% of its value. (This in itself is probably another blog post regarding the INVEST model for writing user stories.) Paying down technical debt can be timeboxed, though. I could devote half of a sprint to moving my package management to Artifactory. That's not the same as being able to say how much I am delivering, only the time invested. Perhaps I will get two projects done, perhaps it will be ten - I know how long it will take and how much it is affecting the team's velocity in delivering business value, but not how much it will ultimately benefit the team internally. Still, it's perfectly fine to timebox addressing tech debt, whereas it generally is not with user stories.

Ultimately the business is requesting an estimate on when valuable code will be delivered to production. This is kept simple if your user point estimates correspond to business value delivered and technical debt is not pointed. It becomes an exercise in Sabremetrics if you insist on pointing technical debt as you do business value.

Why You Shouldn't Point Tech Debt

Let's suppose we have a team that does 20 points of work per sprint, but a tenth of that work (2 points) would wind up as technical debt. So, if our hypothetical team wanted to accumulate no debt, they could deliver 18 points of business value, paying off the two points of potential debt, just like paying off your credit card in full at the end of each month.

We could also have our hypothetical team deliver the full sprint of work toward business value and choose to carry the debt into the next sprint. (Of course, we will ultimately fall somewhere between the two hypotheticals, paying down debt as the opportunity arises, carrying it when the friction it causes is relatively less than the business value being delivered. Humor me, though.) So with the first sprint we deliver 20 points and carry two points of debt into the next sprint. The thing we know about debt is that it comes with a price, interest charges! For the sake of argument, let's say our team is paying Lucky Luciano type rates on their interest so that every two points of debt costs us a point of work each sprint due to the friction it causes.

(Tech debt is like sand in the gears of your development process. It will cause things to move slower and slower until development grinds to a halt completely. Of course it works slowly over time, and the interest you pay on it is much lower than 50% per sprint. However, a system of any size that has been around for any length of time has accumulated much more than a tenth of a sprint's worth of debt. Your dev. team could probably work a couple months right now on nothing but technical debt.)

But back to our hypothetical --

So now, in the second sprint, our team can accomplish 19 points of value and accumulate two more points of debt. In the third sprint they are going to do 18 points, just as if they had been paying the debt with each sprint, but now they are six points worth of debt. Of course, in the fourth sprint the debt has caught up on our team: They begin doing less as if they had been paying it off with each sprint, and it will spiral down from there.

When this becomes a big deal is when the business comes to the team and says they would like to add 90 more points to the system and wants the team to estimate how long that work will take. In the first scenario the team can relatively confidently say the work will take exactly five sprints. (Amazing how that worked out to be exactly divisible by 18!) In the second scenario the estimate would depend on where the team was with their debt. A completely greenfield solution would take five sprints, yes, with an accumulated debt of about another half a sprint. If the team couldn't start greenfield, though, it could be six, seven, eight... sprints. While in the first scenario, it would be five sprints regardless of what point the team began developing.

So comparing the team to itself in the two separate scenarios (because we'd never compare velocity of different teams as apples-to-apples, right?), it is not accurate nor fair to say they are delivering the same amount of value in both cases. With an effort lasting fewer than five sprints they will deliver more in the second scenario, delivering value and accumulating debt. More than five sprints and they deliver more by paying down debt every time.

Now suppose another scenario where our team has accumulated 20 points worth of tech debt. Rather than deliver only ten points of value with the next sprint, they choose to take the hit and pay off the debt, delivering no business value. On the next sprint, however, they are back to delivering 18 points of value (accounting, once again, for immediately taking care of any debt). After the third sprint they've delivered 36 points of value compared to what would only be 27 if they continued stumbling carrying the debt. We can measure the difference in value, nine points after 3 sprints, being delivered by the decision to pay off the debt. If we measure debt and business value the same way, that comparison won't be made. It isn't that the comparison can't be made -- And I realized that as I wrote this post. With some bending over backwards while jumping through hoops and some algorithms written in python, comparisons using tech debt pointed as stories could be made. In my next post, though, I will argue why a comparison like that probably won't be made and shouldn't be made.

"How can I make my company realize that they are NOT agile?"

I just got an email from a company called agile42 that I talked with at Mile High Agile. I was really just looking for a good place to save this for my own use, but figerd that agile42 wouldn't mind if I gave them some free advertising by sharing it here.

Q: How can I make my company realize that they are NOT agile when they say they are?

agile42 Insight:
This is a great question and one we often get from the community. If a company is saying "We're all good, we're agile now!", it likely means they're looking at it as a binary measure - that they simply areor are not agile. In this case, convincing someone that they are not something they claim to beis difficult, because it feels like failure. Instead, we prefer to think of it like a spectrum. Some might think of it as a journey and most organizations are somewhere between "We need to get started" and "We've reached our destination".

One way to have this conversation at any level of the organization is to have people reflect on the 12 Agile Principles and then rank them in order of how strongly they are living them.

1. First, select the top 3 strongest and bottom 3 weakest.
2. Next, go through the 3 strongest principles and identify two or three concrete things the organization does that makes them good at those principles.
3. Finally, go through the weakest 3 (opportunities for improvement if you'd like a less negative connotation). Identify two or three concrete things that might improve these principles.

• Through this method, you get a list of improvements that either shore up weak spots or double down on strengths - either way, they all make your company more agile!

Easter At Home?

It's strange to be home at Easter. Jen and I are almost always in Vegas on this weekend for Viva Las Vegas. Because of Easter falling early this year, VLV doesn't begin for another few weeks. It isn't so strange that I wrote a very similar post to this four years ago when we were on our way to VLV.

It isn't strange because I feel this way every year this time, a feeling of hope and renewal. Maybe it comes from a Christian upbringing. Maybe it just comes because winter is over and spring is here and for eons human beings have held celebrations at this time of year, though most probably don't involve Rockabilly music, loud cars, tatted-up pinups, and Pabst Blue Ribbon.

I've always liked Mardi Gras because it was that one last Bacchanalian celebration before setting my sights on making some real changes come Easter. I would say that sometimes I was marginally better. Usually I just realized that I was another year older and my Rock-n-Roll lifestyle wasn't doing my body much good. (Three or four nights in Vegas will do that to you.)

This year has been different. The changes I have made have been "out-of-band" so to speak, not occurring in the regular course of events. Mardi Gras was less "Mardi Gras" this year, knowing, as really I always have, that true change comes little by little, day by day.

Easter time this year has me feeling the same feelings as always, hope and revitalization. There is still so much I need to work on as a person. Winter had me pretty down. (The meds help a lot to whitewash it.) Now the sun is up, the daffodils are blooming, and the Cubs are on the t.v. (Someone needs to re-energize the Cubs starting rotation, however!) I'm motivated. I hope you are too. For whatever challenges you are facing and whatever your faith, this is a great time to "get after it" as my old football coach used to say.

Happiness and Community

I'm listening to The Art of Happiness in a Troubled World. It's the third in a series, which started with The Art of Happiness, a Handbook for Living, that a psychiatrist, Howard Cutler, has written along with the Dalai Lama. Confession: I had started the first book several years back, and when I checked this audio book out of the library, that was the one I thought I was getting. This book actually seems more appropriate for the current time, though, in the wake of the Vegas and Parkland shootings.

Obviously Cutler and HHDL (that's His Holiness the Dalai Lama to the unintiated) have a good rapport if they are working together on their third book together, and that rapport does come through. The Dalai Lama is a funny dude. When Cutler asks him the simple question of how do we get humans to better see the humanity in each other, HHDL suggests that if Martians invaded Earth we'd all ban together pretty quickly. "Okay, well apart from that..," continues Cutler.

Im only a short way into the book, but it seems that the main premise is an important one, and I wanted to share. The Dalai Lama suggests that a good deal of the trouble that occurs amongst humans today is because we have lost our sense of community. Tibetans are shocked when they hear that Westerners can live for years alongside neighbors that they do not know. Cutler admits that he doesn't know his neighbors, and I admit that I don't know mine at all. And it isn't just our immediate neighbors that we've lost touch with. We don't socialize in the ways we once did. Television and the internet has removed a lot of the personal interaction we once used to have. Church attendence is down. Socializing with coworkers has been eaten into by longer and longer commute times.  Many of us live in different cities or even states from where we grew up in numbers that are unprecedented. We've become a lonely society. The average number of intimate friends that Americans said they had has decreased from three ten or twenty years ago to two now. Twenty-five percent of us say that we have no real intimate friends at all.

That loneliness is literally killing us. Sometimes it's showing up at the door to the school armed with an AR-15. Sometimes it shows in a suicide rate that is the highest it has been in thirty years (Is it just coincidence that the three states with the lowest population densities, Alaska, Wyoming, and Montana, have the three highest suicide rates amongst U.S. states?) . More often it shows in heart disease and cancer rates in lonely people. One health study showed that lonely, disconnected people are four times more likely to catch a cold than someone who isn't.

I struggle with making human connections. It's a lack of self-confidence and really a bit of cowardice generally that keeps me in my little bubble. For years I tried to overcome that with alcohol. Finally I've come to the harsh truth that breaking down my reserve with alcohol doesn't work for me. I've been struggling again for about the past five months of feeling disconnected once again. So listneing to this book has come at a good time. It came as a revelation that what I really need is to connect within a community. It's almost absurd that it comes as a revelation to me, since Jen has been saying as much for at least the past half-dozen years, though not in the same words.

As I sat and thought last night, I realized I am part of a great many communities. I'm a Rockies fan and last night we had our season ticket draft. Sure we only meet once a year, but that has been for the last dozen or so years. In a broader sense I connect with other Rockies fans online and connect with sports fans generally. Sports fandom is a community that is overlooked as a meaningful one to many people. There is a community of people I work with here at Nordstrom and beyond in a general sense of technical people in the Denver area. I'm a member of Mensa and the Triple Nine Society, not that that in itself is meaningful, as we have much less in common amongst ourselves beyond a high IQ. But within those groups I identify with subgroups. It gives me a group to sound off about running or books or politics. In quitting drinking I attended a handful of AA meetings. That is a community, and an important one for some people. It can be very lonely when you quit drinking and for the past number of years the only new friends you made were people at the bar.

Being a part of those communities is not the same as being connected to them, however, just as living next to people doesn't make you connected to your neighbors. I'm working on that. Just identifying all the opportunities I really have was an unexpected step for me. Maybe I'll actually make a Mensa meeting in the next 6 months.

I didn't start this intending to write so much about myself. I meant it more to get people thinking more about community and what they can do to be more connected. For those of you who struggle with it like I do, keep trying. Do little things to reach out. Be brave. Don't let rejection deter you.Think of the communities you are a part of and what you can do to be connected within them. For those of you who don't struggle, reach out to those of us who do. We'll likely reject you using crucifixes and garlic as if you've walked out of a monster movie. Don't let rejection deter you either. Eventually we'll run out of garlic.