"Internal" Bugs

Software is full of issues. Tests explore software and uncover some of these issues. Sometimes we won't write enough tests, due to time constraints and reasonable effort already expended. So some of these issues are found by people outside our team - ie our customers.

And its perfectly reasonable of us to provide a bug tracking system for all external stakeholders so they get feedback on their bug as it goes through production.

But why do we record issues we find ourselves, as bugs?  

I've often heard that bugs are actually a good thing! It must mean the customer is using our software. Surely the customer is expecting a few bugs. Right?

Wrong. Bugs are bad. Always bad. Every time a customer finds a bug, the effect on our reputation from the customers point of view as software developers is negative. Even if we label the software an Alpha or Beta release just to lower their expectations, they still get a little disappointed when things don't work. Even if we write the bug ourselves before the customer finds it, we still suffer reputation loss. Bugs are expensive in terms of time and money for the customer and we are charging the customer more for they actually already expected.

Analogy: I employ a cabinet maker to make me a new chair. The apprentice and the master craftsman take the measurements, cut the pieces, carve the design and begin the assembly. At one point they figure out that piece X doesn't fit slot X. Do they ring me to tell me that there is an issue? Do they just fix it up? How about when the chair is delivered and I find that it wobbles? There is no difference for software.

So in software development, why do we make public our own sloppiness in what we deliver? Bugs are one indication of sloppiness.

In a scrum team, we must try and establish the culture of honest craftsmanship. If opening the code for a user story, uncovers an issue we need to fix that issue as a task on the story. We must ship a good quality product. So often this occurs right at the end of the sprint. And so often we force ourselves to write the bug, but ship the code anyway. When this happens you have to ask: "Why do we ship sh*t?" 

As scrum masters, we need to be disciplined and honest with our team and customers. We are done and ready to ship if we are done and there are no outstanding issues. Otherwise we are not done. 

To be true to software craftsmanship and our profession, we need to have a 0 tolerance to bugs.

Delivering too early...

This is a transcript of a conversation between 3 developers on a real life project.

Background: Dev 1 releases a product to the deployment without integration testing. As soon as deployed, the system is broken. The pressure is on to resolve and repair the deployment. So Dev 2 agrees to help Dev 1 troubleshoot with a view to sorting it out. Dev 3 is their other senior team member, who takes part in the conversation.

Dev 1: OK I have seen what the problem is. My merge wasn't completed like I thought it was and the impacts are missed in the master branch. So it's not in the RPM we delivered.

Dev 3: So lets back out the packages that include your changes.

Dev 1: I really want it to stay in.

Dev 3: Is that really a good idea. It's not tested. It's not really professional to keep it in there, is it?

Dev 1: Ya, well its tested locally and this is valuable feedback for me and the code is finished. You have to crack a few eggs to make an omelet and all that.

Dev 3: OK. Your call.

Dev 1: One other thing, Dev 4 actually made the update to this component. I'm not entirely sure what he did, but there are 4 unit tests failing now. He never updated the unit tests.

Dev 3: OK, so you are still leaving the component in the build? Are you going to remove them or fix them up?

Dev 1: Well I don't know how to fix them up right now. But I'm definitely leaving the rpm in.

Dev 3: So what are you proposing? Commenting out the test cases?

Dev 2: No, we are going to comment out the assert statement at the end of the test case.

Dev 3: ???WTF? So the tests will just pretend to pass?

Dev 2: we want to keep the coverage value high.

Dev 3: <Insert swear word here> lads, that can't be the right thing to do.

Dev 1&2: Code coverage is watched by management, it can't reduce!

Dev 3: I'm not sure I want to hear this. Not sure whether believing the lie or knowing the truth is better!! [laughter all around]

We see lots of dysfunction above. Dev 1 definitely isn't persevering to take total ownership of the product that they have assumed responsibility for (the person doesn't know how to fix the tests correctly). Dev 2 has probably suggested the worst possible solution to the failing tests problem, because he wants to make sure the teams' product code coverage statistic doesn't decrease as it's being watched by management (but he is doing it in a very unprofessional way). Dev 3, on the face of it might look like the best guy here, but that person hadn't the bravery to make sure the right thing is done. Dev 4, the guy who actually made the original code has ignored valuable test feedback, either by not running or ignoring the failing tests.

Ultimately we have three developers who come to the wrong conclusion, despite having the right conversation. There is little excuse for not backing out the package and re-introducing it to the main delivery when all testing is satisfactorily complete, either by Dev 1 or Dev 4. The main track delivery is not the place to be getting initial feedback.

It's interesting is that the management code coverage target drives probably the worst possible behavior with unit-tests - False positives just to keep a coverage figure. When coverage is a target, you have to work hard on the culture of your developers.

A brief look at component based teams...

This is an expression of my observations of the challenges I see teams facing.

For our organisation to be agile, we have an explicit need for the "Vertical slice" team. Where every team can make updates across the system, pretty much independently of other teams. Nothing, or at least no other team, gets in the way of progress of the "Vertical slice" team! But in the real world we often shy away from the Vertical team from both a line management and a technical leadership point of view and we continue to form component based teams.

Management are tasked with staffing teams with enough competence to get the job done. When focusing on a component, it is far easier to tick competence boxes on the fewer technologies. Also certain in vogue technologies' developers command a kings ransom. It's just financially impossible to have one of those guys in every team.

For developers, the "money" is in being an expert in whatever technology is currently "hot", so developers have a inherent incentive to become an expert in relatively few technologies. The best way to become an expert is to focus entirely on that technology.

The component based team:

Pros

• The most "natural" way for most organisations to setup its' teams.

• Scope of the teams' responsibility is small so they have excellent technical focus. There are a few technologies that the team works intimately with, all the time. Line mgt and developers are happy.

• Team members can become true experts in an area or technology.

• Can build out a "complex" component out of sophisticated "niche" technologies relatively quickly.

Cons

• That team only knows a small part of the system. They are effectively limited to working on that part of the system.
  
• The team can have the illusion of being successful in a product that is failing
  
• There is a big "tax" when moving people or other teams onto or off this component.

• Those people become key to the organization even though they might not be domain experts in the business. The team can make themselves indispensable to the organisation.

• Component becomes impossible to fundamentally change its technology as the business changes. A component team is loyal to the technology because they have so much expertise. The business adapts its solutions to fit the technologies employed.

• Long term waste/expense. Over medium to long term you end up building parts of a component that are "nice to have" and may never be used by the business.
  
• Perceived "God" complex develops over time. And Conways' Law applies.

Happy new year 2015

Its been a while... Had a busy 6 months since my last post. With the new year and all that I'm back and will be creating some articles. The main aim of this blog is to share some observations and knowledge, while re-enforcing my own learning.

The aim is at least 1 per month, as per last year. While I haven't been publishing, I have been gathering some ideas for articles. I hope to add some book and video reviews as I go.

Looking forward to a great 2015.

Scrum master tools: Effective Whiteboards

The whiteboard is one of the most powerful tools of a co-located scrum team. A picture speaks a thousand words for promoting understanding within a team. True, there are thousands of cool drawing applications out there. Some are free and some have some license cost. And these are great collaboration tools, especially for a non-collocated team. But for the co-located team nothing beats a good quality, fully equipped whiteboard.

There is no other tool with as low a competence barrier for use as the whiteboard. Whether you are the CEO or the technology expert, old or young, rich or poor, chances are you have basic writing and drawing skills. This means you can use a whiteboard. 

The whiteboard is a great aid to the creative process. You don't have to go searching for the "right" icon or diagram. There's no chance of it crashing just at the eureka moment. Its always on. It's easy undo/change the diagram. Its easy to save (take a photo using your phone!).

Plus its a great break from just sitting around the computer on the desk. 

In my opinion, one of the things a scrum master has to do is get their team using the whiteboard effectively. The following tips are vital

Guidelines

Make sure the whiteboard is near the team. Preferably right beside. Preferably only for use by your team. Unrestricted and easy access are vital.

A clean whiteboard is an invitation to the team for content. Make sure the whiteboard is always clean. If no one is using the whiteboard at that moment in time, it must be clean. . There are two ways you can achieve this. Firstly have a rule in your team to always wipe the whiteboard after they are done. And at the end of the day or start of the day, clean down the board yourself. Never tolerate those that write "Do not erase". Erase immediately.

Fully equip your white board: Make sure you personally source: Markers (various colors), Magnetic eraser and white board cleaning fluid or spray. Daily check the white board for markers and that it is clean.  

Practice with your team: It can be daunting to start using the whiteboard for some team members, even though they have the basic skills. Take time to help your team to articulate their ideas on the board.

Practical advice

I favour metallic based whiteboards. These tend to have a higher quality surface. From time to time, magnets are useful for planning with index cards etc.

Communication advice

Team need a common language to understand each others drawing. For me, working with mainly Java and OO based teams, its mainly UML diagrams (class, object, sequence diagrams. I can recommend "UML for Java Programmers" by Uncle Bob. Flow diagrams are also useful for algorithmic solutions. And there's always free form diagramming - come up with your own representations, just be sure to explain it to the rest of the team what the boxes and arrows represent! 

Happy white boarding.

Retrospective Planning Roulette

This is a planning game to help organize team retrospectives with a difference. Its a great activity to run in a scrum master community of practice when discussing retrospectives.

Required:
Copy of "Agile Retrospectives: Making good teams great" by Ester Derby and Diana Larsen
Queue Cards
Colored Markers
Whiteboard
Magnets or bluetack
4 Scrum masters (I had enough Queue cards for 4, if you have more just write more cards)

You will plan a number of different 5-stage retrospectives for all your teams at the same time. The book is organised into activities that support each stage of the retrospective.

Instructions:

• Write out the exercises from the chapters 4, 5, 6, 7 and 8. It helps if you use colors. These are exercises that support: Setting the scene, Gathering Data, Generating Insights, Deciding what to do and Closing the Retrospective.
• Now lay all the cards out on the table in columns according to chapter. See photo.

• Next you lay out the whiteboard. So across the top put the 6 columns. One row per team. Now you are ready to play. Here we have 4 teams.

• The next step is to get the scrum masters to choose from the laid out cards which activities they will use to run the retrospective. It doesn't matter if they don't know what that activity is as its described how to do it in the book. All they need is on the queue card.
• You end up with something like this: Here you have 4 teams retrospectives planned out. For example 

• Now each scrum master has an agenda put together. The instructions for each task are in the book quoted above. 
• Its good to follow up this activity to see how did the scrum masters get on. And you can easily re-run it to get another retrospective agenda filled.
• FYI after we had done this I found this website that does the same thing with an online version. http://www.plans-for-retrospectives.com/ 

Failure in scrum mastering and project management.

Project management typically tries to identify failures (risk) and takes actions to avoid those from happening.

In scrum mastering, we let the team fail and fail as fast as possible. We then help the team to learn from that failure.

What happens when you ignore scrum in an scrum based agile environment

Agile Chaos... Everyone tries to do everything and very little gets done and quality goes out the window. Worse is that developers will perceive scrum as a problem rather than a solution. As a scrum master and guardian of scrum, it's one of your biggest fears! Forget about the sprint failing, this is when agile fails.

Scrum

Scrum: a process where short, fixed scope iterative cycles are used to deliver software over a long period of time. Scrum promises the fixed space to developers to allow them to be creative, while offering the customer the flexibility to change the product. A win win.

So the team gets the X weeks space to work on something and during that X weeks the organization issues are supposed to go on the backlog and, if the PO agrees, be prioritised to the top to be tackled in the next sprint. So you could have a possible x weeks needed to turn around your issue. 

Reality

But how often do we see that managers or PO's just can't help themselves tinkering with the sprint backlog. As soon as the customer or stakeholder get off the phone with a question, suggestion or complaint, they have to have a developer that MUST look at this new revelation TODAY!

Well lets look at the effects this have on the developers in the team.

Context switching goes up. What the developer was working on yesterday, must be dropped to look at this new thing. Then when we have brought it to resolution, that developer switches back to what they were looking at. "Now where was I?"

Trust in the PO and Scrum master goes down. The team recognize they didn't get the space as promised by the PO and the scrum master couldn't prevent it. "I thought the scrum master had our backs"

Commitment excuse. We are now trying to fit more work into a short iteration. This means our original goal is probably unattainable unless overtime is called for. "Well we could have got that finished, but..."

Morale down. Trust is down, our goal looks out of reach. We have to work longer hours just to make up. "There goes my Saturday..."

Quality down. "If I have to work overtime, well dammit I'm gonna take a short cut. We can fix it up next sprint."

Amount of testing down. "We haven't got time to put in those negative scenarios. We'll pick a few to give us good coverage and leave it at that?"

What can a scrum master do prevent this from happening?
Scrum masters should have the power to stop a sprint and call a re plan. Also scrum masters have the power of informational at their hands. They can publicize the effects as observed so that Product owners realize the consequences of their decisions on the team and on the product. 

When reviewing the sprint, make sure you highlight what was added at an inappropriate times during the sprint. 

The advantage of using a tool like JIRA is that the story or task history gives you evidence of the amount of context switching in a team. It should be universally recognized that context switching is inefficient in the organization. So highlight the history of stories during a chaotic sprint as evidence.

Trust and Morale are much less tangible to highlight. But it also means you have to work harder to increase both when you recognize they are down.

Quality and testing are much more tangible to show. There are lots of statistics out there you can gather on your product. Be weary that teams will design towards quality metrics if the quality metrics are a stick to beat the team. Quality metrics are evidence for management to give the team room to take some action.

When was the last time you accepted work into the team during the sprint? When was the last time you stopped the sprint? A scrum master should have the power to do this. 

Scrum should be about sustainable pace. When you are in chaos due to too much flux, make sure you call it what it is: Chaos. If scrum is confused for chaos, people will not see scrum as a solution, but as a problem. 

Lesson

The ultimate realization here is that there is more than the Scrum master protecting the team. The product owner needs to realize how they fit into the picture of the team. This is easier to do if you consider the product owner as part of the team. However if your organisation doesn't consider the PO part of the team, it's a bit harder. The scrum master has to go to more effort to make them realize the consequence of the product owner not absorbing the majority of that flux driven by the stakeholders. 

Winston Wolf... A scrum master from the movies!

"I'm Winston Wolf, I solve problems..."

This is the introduction from Harvey Kietel's character in the movie Pulp Fiction.

Better quality version
Low quality, but longer clip

Here are some of the excellent Scrum master qualities Winston demonstrates in this clip.

1. Skin in the game. He puts "skin in the game" by showing up to the crime scene. He's now as guilty as the rest of them. 
2. Team. He forms the team. The four guys in the house are the team. They are all in together in the one situation. The husband is part of another team (husband and wife), but the Wolfman takes subtle control of him and takes him into this team using a clear vision, providing a service and providing compliments. 
3. Trust. He immediately builds the relationship and trust. 
   1. Reputation. He comes with a reputation and he makes sure they know it. Sets a seemingly impossible task to start with. Surpasses expectations. (He shows up at an impossible 20 minutes early.) 
   2. Commitment language. He states "He will solve the problems."
   3. Respect. He knows names. Respects his fellow team mates. Commands Respect from the guys.
   4. Forms a bond, establish a relationship immediately.
4. Belief. They believe he can solve their problem. He uses positively and commitment language to instill this belief.
5. Time lines. He understands the deadlines. He ensures everyone else knows the deadlines too by repeating them. 
6. Calmness. Cool head under tight deadlines. Understands the situation. Knows the facts of the case, but is never flustered or excited. Looking at the ugly crime scene, he looks for coffee... I'm so cool under this pressure, I want coffee. 
7. Complements & treats. Small things really help the situation given the pressure. When he gets the coffee he's thanks and compliments the guy who makes it. He makes some jokes at various times.
8. Disciplined. Enforces discipline. Comes across as curt, but then explains himself as he has no positional power.
9. Urgency. He "thinks fast", he "acts fast". All times he conveys the sense of urgency, while not being hasty or flustered. He is calculating and organised. He arrived 20 minutes early!
10. Everything is delegated. Notice that he actually does none of the "Hard Labour"... But note that he asserts that he has expertise in the area. Even though he never lays a hand on anything other than the coffee cup, the team believe he can help them.

---

Updated 22/11/2017

A short discussion on code coverage

What is code coverage? 

Code coverage is a measurement of the lines of code executed during a test. Usually this is aggregated together over a number of tests, to give a suite code coverage metric. It is a measurement that is easy to obtain automatically as there are lots of tools out there to measure it for you. It is usually expressed as a percentage of the lines of source code executed. It is can be used as an indication of quality of the test suite, however like all statistics it needs to be used in the correct context.

What does it indicate? 

Low levels of code coverage indicates that our automated test harness does not do a very good job verifying the entire components' source code. We cannot be confident that the changes we made in this release will not effect some of our users out there in the world.

Hmmm so a higher number is better? 

Yes. Higher numbers can mean we have a very good test suite, but does not guarantee it. When developers use code coverage as a targeted Key Performance Indicator (KPI) when writing tests, a small subset of the required tests would give an artifically high code coverage metric, while leaving little or no confidence in our test framework. It is amazing how many times managers put a coverage KPI on developers. This usually results in an artificially high coverage metric, typically 95+%

So we should have 100% then?

Absolutely. Why not? If the code is well structured AND you thought of all the tests required, it should be pretty near it. You will always have code, like logging, tracing and certain "aspect" type or cross cutting work that you may not test because its not actually business logic or effects business logic. If the code is legacy code the cost of introducing a test framework to reach high levels of code coverage may not be worth the effort. You should instead aim to write the most popular/common test cases and just use code coverage to see help indicate future refactorings/removal of code. 

100% code coverage means 100% confidence? 

Absolutely not. Not even close to 100% confidence. 100% code coverage does not mean we have written 100% of the tests that ensure our code is working. Only 100% of the tests required to test a component ensures confidence when future changes are made.

So I need 100% of tests? 

Exactly. Thats all that matters. Thats what we really should be striving to write, not to hit a coverage limit of 80%, 90% etc.

Can I automatically measure % tests I have written? 

No... not without a clever human involved. This is where a good automated tester earns their bread and they are worth their weight in Gold. Literally.