Team John, part 2

Yesterday's post ended a bit abruptly while I had a lot more to say on the subject of Team John. The reason for this is a silly one, but this is just a silly blog, so I'll do with it however I will, including using run-on sentences as I feel like it: I am trying to get back to using 750words.com. It's a great site that simply tracks the number of words as you write them to encourage you to write at least 750 words each day. I'm picking this back up, as I have been reading Stephen King's marvelous "On Writing". King encourages people that want to write to start with at least 1000 words a day. Being more of an underachiever, I'm starting by going back to 750 each day. I'll have more to say on Mr. King and his book hopefully in coming days and weeks. The 750 word goal also works to make sure that I don't spend an inordinate amount of time writing drivel and sometimes even forces me to stop when I have more to say.

So I have more to say on Team John. In the previous post I talked a little about how we implemented it with the current team. Other than what I think were good ideas of tying it to our oncall schedule and having two developers on in a rolling fashion, it probably wasn't exactly helpful in understanding much beyond the message that I think Team John is a good idea for teams. I did hit on the primary purpose that Team John shields the team from interruptions and interruptions are very costly to teams. As an Agile, or more specifically a Lean practice the idea of why the Team John works goes beyond just funneling interruptions  away from the main development team and it goes to queuing theory.

Queuing theory is really the study of how queues or "waiting in lines" affects system throughput. As a study it was first used in telecommunications and then was adopted by Toyota in manufacturing and then came from there to software development through Lean teachings. As a practice it is as old as when people first began lining up to wait for something.It's helpful if you think about waiting in line at Wendy's or at the local grocery store. Will opening more registers help people waiting in line move faster? Yes, it will, but up to a point, right? I can open up four registers at Wendy's, but if I only have one person back on the grill, I can only move people as fast as burgers come off the grill. Or think about those videos you've seen of an automotive assembly line. The line moves along at a regular pace with each man or woman doing the job at their station. Highly repetitive, but also very precise - not only in the placement of each bolt but also in the time the entire task takes.

Applying just those two ideas, it's easy to see that the entire process can only move as fast as it's slowest component (we call that a bottleneck) and that a sustained pace is better than randomly throwing work at people. Like I said, these aren't new concepts but have been around at least as long as people have been waiting around for hamburgers. So how does Team John help the process?

First, when a Scrum team has planned out a sprint, they have planned an amount of work that is sustainable if they work a regular pace throughout the sprint. Unplanned work coming to a development team means a) it will be of some volume that the team couldn't previously deduce and b) it will have some deadline that the team couldn't previously expect (presumably sometime within that sprint, however). There is a rhythm to a sprint, particularly when a team has a sprint goal and is doing a similar group of repetitive tasks towards that goal. Unplanned work throws that cadence off kilter.

Secondly, Team John helps teams deal with those bottlenecks. That guy on the grill can become a bottleneck. If you picture the auto assembly line again, it's easy to see that the line can only move as fast as the slowest person working it. The other thing to realize, particularly in software development but also in fast food service, is that you can never completely predict how a relatively short run on the system will unfold. One hour I might have everyone order singles with cheese, a Frosty, and a small fries. The next I may have people each ordering custom orders ("Hold the ketchup, hold the pickles!"), one of a variety of beverages, and all wanting large fries. What if I have one or two people just waiting, ready to place toppings or fill beverages or put down more fries as needed? I wouldn't need to always anticipate exactly where those bottlenecks are going to be.

I probably have more on the subject, in fact I know I have more, but I'm at almost 850 words now and need to leave some for tomorrow.

Team John, part 1

I was introduced to Agile concepts around 2004 or 2005, primarily eXtreme Programming ideas from articles by Kent Beck and Ward Cunningham. Fairly shortly thereafter, 2007 or 2008, I attended my first Mile High Agile conference here in Denver. At either that first one, or possible the next year (my memory is starting to go), I was introduced to the very energetic V. Lee Henson and a number of his really great ideas in implementing Agile including the "Team John" concept. Briefly, a software development Team John is an engineer who works separately from the rest of the development team on bugs, production incidents that arise, work that comes in sideways last minute ("Hey, can you pull some numbers from the database for me?"), or tech debt. The idea is that developer works on all the other shit that isn't part of the team's sprint work that would otherwise lead to interruptions. The responsibility rotates to a new developer with a new week or sprint.

It seems counterintuitive that removing a developer from the mix would allow the rest of the team to go faster, but I think any development team that regular needs to deal with incidents or gets hit with work sideways, from outside the Scrum process will immediately see the value. Interruptions kill a developer's productivity. That loss in productivity is attributed to context switching: The very act of switching from one task to another results in a tremendous amount of lost productivity. Psychological research has shown that context switching can result in 40% productivity loss, and the more complex the tasks, the greater the loss. 

In his blog, Joel Spolsky, the creator of Trello and one of the founders of StackOverflow, puts it this way, "The trick here is that when you manage programmers, specifically, task switches take a really, really, really long time. That’s because programming is the kind of task where you have to keep a lot of things in your head at once. The more things you remember at once, the more productive you are at programming." Spolsky goes on to sum up, "In fact, the real lesson from all this is that you should never let people work on more than one thing at once."

The primary purpose of Team John is to shield the team from interruptions or at the very least funnel them all to a single person. With one of the teams I currently support we were getting a number of ad hoc requests coming into our sprints sideways, whether from our business partners ("Hey, can you get some numbers for us so that we can see how the latest campaign is working?"), or elsewhere within the technology department ("These three servers have critical updates and need to be patched by Friday"), or from within our own group ("This service is causing alerts. Does anyone even know what it does?"). Then you had production bugs and incidents which had to be addressed right away and suddenly we would have three developers dropping what they were doing to look at a single bug. I brought up the idea of using the Team John concept - we already  had an oncall rotation for who would handle incidents outside of business hours. We decided whomever was oncall would also handle the interruptions that arose day-to-day. They could work on regular sprint work if there were no interruptions, but at least each individual developer wasn't being pulled away almost hourly.

We quickly discovered that there was a lot of Team John work once we all started funneling it to a single person. Not enough work to keep that one developer busy the entire time, thankfully there weren't that many bugs and incidents, but enough that they really ended up contributing very little on the actual sprint. Meanwhile the rest of the team was able to accomplish more work that it had in previous sprints. I think there were six or seven developers on the team at the team. By taking one out of the mix to handle interruptions our velocity increased by roughly 15%. 

Rather than have Team John work on sprint work intermittently, we had the developer begin to pick up some of the technical debt we had accumulated and try to pay that down in between other interruptions. As we began doing that, some Team John work would carry over to the next week (our oncall schedule rotates weekly) leaving the next person in the rotation without much context for what was being worked on. After a while, and after adding a couple more developers, we decided that both the primary and secondary, which was always the previous week's primary, oncall developers would both be on Team John. This way as the primary rolled over to be secondary, they could bring the new primary up to speed on what was being worked on. With the addition of the second developer on Team John I don't believe the velocity increased, but it definitely did not decrease.

Why You Shouldn't Point Tech Debt, Take 2

I want to expand on not pointing technical debt stories. As I finished the last blog post, I realized that I had made the presumption that teams should only assign story points to stories based on business value. Then I proceeded to basically lay out the argument for why a team should (or in some cases should not) pay down their technical debt, kind of skirting the entire question of what not assign points in the first place. A classic case of begging the question: We only point things that are valuable because they are things that are valuable. If you already have bought into that statement, you wouldn't be here asking the question of whether to point tech. debt or not.

So now I want to back the premise that a team should only point work that is of value to the business. One of the developers I support is a fan of both metaphors and the NBA, so I would like to present my argument in these terms: When you watch a basketball game rebounds, assists, steals, and turnovers are all very important to the teams. However, at the end of forty minutes, the one stat that matters is how many points each team scored. Just like having a lot of turnovers will hurt a team's chances to win, technical debt is going to drag down your development's ability to deliver new features. In the end, however, the business runs on the value of the features the team delivers.

I mentioned previously that a team may look to distinguish between technical debt points and business value and still somehow combine those to estimate output. I don't believe that this is really an apples-to-apples comparison, however. Some technical debt MUST be addressed and some subset MUST be addressed IMMEDIATELY (particularly if bugs and incidents fall into that technical debt bucket). For example, our organization recently moved its package management from a Nexus repository to Artifactory. Every development team had to change projects to point to Artifactory. No value was delivered to the customer, but this had to be done by a set deadline else project build pipelines would cease to work. We also have contractual obligations for SOC 2 and PCI compliance driven by hard deadlines. Managing API keys and updating network diagrams does not deliver value to our customers but must be done and must be done by particular dates. If that date is a couple years off the tradeoff with choosing to bring in business value may be relatively small. That cost increases, however, as the deadline approaches.

Another thing that makes estimations of technical debt unequal to estimating business value is that technical debt, in my opinion, is generally more difficult to accurately estimate. I say this because if it were easy to accurately assess the risk, effort, and value of paying down debt that teams would actively engage it rather than accumulating it. I believe technical debt accumulates, in most cases, precisely because we find it difficult to assess risk, effort and payoff. "If it were easy, we'd have done it." I know me saying this is my own opinion based on experience and that in some cases teams know exactly the risk and effort and the value simply isn't worth addressing the debt. That said, this is one more reason why, in my experience, I do not believe teams should point technical debt. To extend my metaphor further - a made basket in basketball is worth a set number of points: one for a free throw, two for a standard field goal, and three for those beyond the three point line. A steal, like paying off technical debt, may result in points, and it may not. A coach can't tell his team, "If you get this many steals we are sure to win the game." He can look Jimmy Chitwood in the eye though and say, "Jimmy, if you come off the picket fence and make this shot, we win the game." (That isn't exactly how it happens in "Hoosiers" and I realize I've taken this too far, but I can't get this far into a basketball metaphor and not bring up one of the greatest sports films ever made.)

A final reason why I do not see technical debt and business value estimates to be equivalent is that a team can timebox paying down technical debt. For user stories delivering business value there is some discrete level of effort involved in delivering that value. Generally a user story that is 80% done will deliver 0% of its value. (This in itself is probably another blog post regarding the INVEST model for writing user stories.) Paying down technical debt can be timeboxed, though. I could devote half of a sprint to moving my package management to Artifactory. That's not the same as being able to say how much I am delivering, only the time invested. Perhaps I will get two projects done, perhaps it will be ten - I know how long it will take and how much it is affecting the team's velocity in delivering business value, but not how much it will ultimately benefit the team internally. Still, it's perfectly fine to timebox addressing tech debt, whereas it generally is not with user stories.

Ultimately the business is requesting an estimate on when valuable code will be delivered to production. This is kept simple if your user point estimates correspond to business value delivered and technical debt is not pointed. It becomes an exercise in Sabremetrics if you insist on pointing technical debt as you do business value.

Why You Shouldn't Point Tech Debt

Let's suppose we have a team that does 20 points of work per sprint, but a tenth of that work (2 points) would wind up as technical debt. So, if our hypothetical team wanted to accumulate no debt, they could deliver 18 points of business value, paying off the two points of potential debt, just like paying off your credit card in full at the end of each month.

We could also have our hypothetical team deliver the full sprint of work toward business value and choose to carry the debt into the next sprint. (Of course, we will ultimately fall somewhere between the two hypotheticals, paying down debt as the opportunity arises, carrying it when the friction it causes is relatively less than the business value being delivered. Humor me, though.) So with the first sprint we deliver 20 points and carry two points of debt into the next sprint. The thing we know about debt is that it comes with a price, interest charges! For the sake of argument, let's say our team is paying Lucky Luciano type rates on their interest so that every two points of debt costs us a point of work each sprint due to the friction it causes.

(Tech debt is like sand in the gears of your development process. It will cause things to move slower and slower until development grinds to a halt completely. Of course it works slowly over time, and the interest you pay on it is much lower than 50% per sprint. However, a system of any size that has been around for any length of time has accumulated much more than a tenth of a sprint's worth of debt. Your dev. team could probably work a couple months right now on nothing but technical debt.)

But back to our hypothetical --

So now, in the second sprint, our team can accomplish 19 points of value and accumulate two more points of debt. In the third sprint they are going to do 18 points, just as if they had been paying the debt with each sprint, but now they are six points worth of debt. Of course, in the fourth sprint the debt has caught up on our team: They begin doing less as if they had been paying it off with each sprint, and it will spiral down from there.

When this becomes a big deal is when the business comes to the team and says they would like to add 90 more points to the system and wants the team to estimate how long that work will take. In the first scenario the team can relatively confidently say the work will take exactly five sprints. (Amazing how that worked out to be exactly divisible by 18!) In the second scenario the estimate would depend on where the team was with their debt. A completely greenfield solution would take five sprints, yes, with an accumulated debt of about another half a sprint. If the team couldn't start greenfield, though, it could be six, seven, eight... sprints. While in the first scenario, it would be five sprints regardless of what point the team began developing.

So comparing the team to itself in the two separate scenarios (because we'd never compare velocity of different teams as apples-to-apples, right?), it is not accurate nor fair to say they are delivering the same amount of value in both cases. With an effort lasting fewer than five sprints they will deliver more in the second scenario, delivering value and accumulating debt. More than five sprints and they deliver more by paying down debt every time.

Now suppose another scenario where our team has accumulated 20 points worth of tech debt. Rather than deliver only ten points of value with the next sprint, they choose to take the hit and pay off the debt, delivering no business value. On the next sprint, however, they are back to delivering 18 points of value (accounting, once again, for immediately taking care of any debt). After the third sprint they've delivered 36 points of value compared to what would only be 27 if they continued stumbling carrying the debt. We can measure the difference in value, nine points after 3 sprints, being delivered by the decision to pay off the debt. If we measure debt and business value the same way, that comparison won't be made. It isn't that the comparison can't be made -- And I realized that as I wrote this post. With some bending over backwards while jumping through hoops and some algorithms written in python, comparisons using tech debt pointed as stories could be made. In my next post, though, I will argue why a comparison like that probably won't be made and shouldn't be made.

"How can I make my company realize that they are NOT agile?"

I just got an email from a company called agile42 that I talked with at Mile High Agile. I was really just looking for a good place to save this for my own use, but figerd that agile42 wouldn't mind if I gave them some free advertising by sharing it here.

Q: How can I make my company realize that they are NOT agile when they say they are?

agile42 Insight:
This is a great question and one we often get from the community. If a company is saying "We're all good, we're agile now!", it likely means they're looking at it as a binary measure - that they simply areor are not agile. In this case, convincing someone that they are not something they claim to beis difficult, because it feels like failure. Instead, we prefer to think of it like a spectrum. Some might think of it as a journey and most organizations are somewhere between "We need to get started" and "We've reached our destination".

One way to have this conversation at any level of the organization is to have people reflect on the 12 Agile Principles and then rank them in order of how strongly they are living them.

1. First, select the top 3 strongest and bottom 3 weakest.
2. Next, go through the 3 strongest principles and identify two or three concrete things the organization does that makes them good at those principles.
3. Finally, go through the weakest 3 (opportunities for improvement if you'd like a less negative connotation). Identify two or three concrete things that might improve these principles.

• Through this method, you get a list of improvements that either shore up weak spots or double down on strengths - either way, they all make your company more agile!

Rambling About Scrum, pt 1

I started this post a while back, meaning a couple weeks ago. I started it for the reasons I start any blog post, because I had both a flicker of inspiration and a few minutes to begin writing something. I realized after a while that I was sort of rambling, another good attribute of my blog posts. So instead of going on rambling I thought maybe I should begin breaking this into little doses, so that it is apparent that I'm just rambling. And now I'm rambling, so without further ado, part one on my thoughts bout applying Ssrum outside of the domain of software development:

So I'm already about halfway through my friend Jared's book dealing with sports psychology. I had said to him that I was looking to mine it for ideas for helping me as a ScrumMaster considering that a ScrumMaster is a sort of coach. We certainly coach teams, especially new team members in the Scrum process. We deal with monitoring the team performance and trying to find ways to improve. ScrumMasters help motivate and focus the development team. Finally, we shield them from negative outside influences that would detract from the team performance. So, there are ways where being a ScrumMaster is very much like being the coach of an athletic team, though I said previously that it may be a little more like coaching a golf team than a football team. (Of course, as I'll point out, a Scrum team is like a rugby team in more ways than one.)

Well, Jared turned the question back on me, saying he'd like to learn more about Scrum to see how it might apply to what he does in regards to sports psychology. I thought about that. How would I describe Scrum to someone completely outside the domain of where we use it? That idea certainly isn't foreign to anyone who has practiced Scrum for any length time: You need to get business managers and executives to buy into Scrum or Agile practices at least, you need to sometimes guide product owners in why it works, and there are sometimes when you need to explain it to the customer directly. Still, while they are all stakeholders in the process, besides the product owner they really only need as much understanding as needed to let the dev. team do its thing. Customers, upper management, they aren't really "practicing Scrum" (unless you are lucky). So where does one start to get relatively good understanding from someone outside the domain of software programming?

First off, I'll say as an aside, that if you are a business person interested in how Scrum can affect business practices, you are really better off looking into Lean practices. The Lean software development practice was adopted from the Lean manufacturing process which was built from the Toyota Production System. Frankly, I don't know Lean near as well as I know Scrum. Maybe someday I'll learn enough to feel comfortable writing a blog post about it.

While Jared writes from the point-of-view of practicing, playing, and coaching football, I work in the domain of voting machines and elections software. So that's where some of my examples will come from.

Any discussion of Scrum needs to start with a discussion of Agile methodologies. "Agile" has become such a buzzword in software development, and it's one that like many buzzwords, few people really understand before they bandy it about. Software development is a very, very recent and evolving practice. If you think about it, man has been building houses since he first moved out of the caves. The art of architecture is at least older than the Great Pyramids. The art of creating software is not so old, so new rules and practices are always being tried. Well, some years back, a bunch of programmers got together and (I like to think) had a bunch of pitchers of beer trying to solve all the problems inherent in this nascent practice and came up with a manifesto, the Agile Manifesto for Software Development:

We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:
Individuals and interactions over processes and toolsWorking software over comprehensive documentationCustomer collaboration over contract negotiationResponding to change over following a plan
That is, while there is value in the items on the right, we value the items on the left more. 

Kent BeckMike BeedleArie van BennekumAlistair CockburnWard CunninghamMartin FowlerJames GrenningJim HighsmithAndrew HuntRon JeffriesJon KernBrian MarickRobert C. MartinSteve MellorKen SchwaberJeff SutherlandDave Thomas 

© 2001, the above authorsThis declaration may be freely copied in any form, but only in its entirety through this notice.

To me an Agile software development methodology is one that values those things on the left sides of the statement over those on the right. That leaves a lot of room for interpretation on just how to accomplish that, but it also means that just saying that your team embraces responding to change doesn't make it an Agile team. There are three other lines in that manifesto. Certainly those values aren't only valuable to software development, and could easily be applied to other types of business. So I want to highlight in this discussion those areas where Agile practices and Scrum in particular could be generalized.

A Great Day to Run it Again!

Just had to buy an old friend and classmate's book after finally checking it out. It's been on my list of things to do. It's a sports psychology book by Dr. Jared Wood. (Doctor! Lord Jesus, help us all, Jared Wood is a doctor!) I bought it hoping that it may give me a few ideas in my own job. A ScrumMaster is a coach of sorts for a software development team. It's pretty unlike coaching a football team, maybe the furthest thing from while still being able to refer to oneself as a coach. Maybe more like coaching a golf team. But just in Jared's blog I saw enough similarity to think his book may help.

The stories Jared tells about high school football bring back a ton of memories. I have recurring dreams about high school football. It was one thing I just wasn't very good at, and I desperately wanted to be. To tell the truth, at the time, I desperately wanted to be good at everything. My friends, though, the kids I grew up with, my pals, were good at football. Jared, Ryan, Chad, Ryan (a different Ryan), Eric, Terry, Bill, did I mention Ryan? They did other things in the offseason - maybe basketball, maybe baseball, but football was all-important at Frankenmuth High. That's a slight exaggeration; it wasn't all-important... but really pretty damn important.

And truthfully, it wasn't so much the sport itself that was so important. For me, it was being with my friends that was important. Yes, Mom, if my friends jumped off a bridge I would probably do it too. I wanted to be just like them, which meant playing football. Genetics conspired against me mostly. I think I was still 5'4" at the end of my sophomore year, adding a whole 3 inches over the next year, and I was still the smallest guy on the varsity team. Also, truth is, as competitive a person as I am, I'm not so much competitive with other people; I was competitive with myself. I was okay with losing as long as I "did my best" as my mother would say. I wanted to set PR's with running. I was never going to be the biggest, fastest, or best athletically, but as long as I was better each day, I was... okay with it. Jared, on the other hand, hated to lose. When I hear about those great athletes like Michael Jordan and Tiger Woods who don't so much like winning as they just hate to lose, I think of Jared. I don't think he took pleasure in winning so much as it was a relief.

I guess I've come to experience that same feeling over time. There are times in some competition where you simply know you are better than your opponent. You know that the only way to lose is by beating yourself. That's when winning comes as a relief. Nowadays, I mostly only ever get to feel this directly while beating some poor souls at a trivia game. Still there is the indirect experience of rooting for the Broncos and the wash of relief as we pound the Raiders into oblivion again. (I'm sorry that my Lions fan friends cannot enjoy this. Every win is a God-damned celebration for you.) 

In his last post, Jared tells this story of Coach Munger, our high school football coach, yelling "Run it again!" at practice.... Over and over he'd yell it. Mind you, Coach Munger would yell this in a game. If we ran "sweep-right" and someone missed their block he'd go halfway out on the field and scream, "Run it again!" for us to hear, the opposing team to hear, the fans and all the rest of the world to hear. And we would, and we'd run it better. 

So at one practice we're running this play where the wing in the formation, normally lined up outside the tight end, is lined up off the line between the the tight end and tackle. His job is to come through the line and wipe out the inside linebacker with a block while the ball gets handed to the fullback, Jared in this case, on a dive following right behind the wing. I played back-up safety so I was the safety on the scout team. On this particular play when the inside linebacker gets wiped out, I'm supposed to fill in behind and tackle the ball carrier. Well Jared may have had seventy pounds on me at that time (not to mention Jared was also probably the only person on the team faster than I was), so the first time we run it, he just steamrolls me. 

For some reason, Jared turning me to pudding on the practice field isn't good enough for Coach. "Run it again!" he yells. Same play, linebacker wiped out, I fill in, and Jared goes over me like a Mack truck. From my vantage point on my back on the practice field  staring up into a slate gray, autumn Midwestern sky I can't tell what nit Coach has to pick with his first team offense, but I hear him yell, "Again!" So I haul my body up off the ground, cursing a bit under my breath and get ready for superstar fullback to come through again. This time, as he blows me up I grab some jersey or pad and hang on for dear life, eventually dragging him down twenty yards past the line. Coach is already yelling, "Run. IT. Again," as we are picking ourselves up. "Sorry, man," Jared mutters to me. Right then, I wanted to just say to him, "Fuck you. I don't want you to feel sorry for me. I want to knock you 'ass-over-tea kettle'," (as the coaches would say). I don't know what I actually said, but I know we ran it a couple more times because I remember being pretty steamed that he apologized to me. I'd love to say being angry made a difference; I'm pretty sure I thought it did at the time and that I was better at least at absorbing the impact. 

Then like every other person who has played the game, you realize that you won't ever play it again. I'd love to go back. Even to be as small as I was then, I'd love to have one more season or one more game. One more practice with those guys or even one more play, as a backup safety waiting for the fullback, Doctor Wood, to come barreling through that line, because this time it would be different.

(You can get Jared's book for yourself at his site. I'm only through the first couple chapters, but I know I will have much more to say about this as it pertains to Scrum and coaching.)