Scrum Master Class

I have a new mission at Acquia – I am taking over as Scrum Master for my team. Bwahahahaha …

I’ve been doing Scrum ever since I started in tech full time at Axeda. Stand-ups and sprints are all old hat, but for the first time thanks to Acquia, I have received formal training in Scrum and a certification from Scrum Inc in Boston.

Scrum Inc.

How does Scrum help? Why do it?

You use Scrum because you want to improve team velocity without increasing team resources.

I liked the fact that the Scrum Master class itself was organized as a Sprint using Scrum. This let us “live the example” and see first-hand the effectiveness of the techniques.

What is Scrum? What is its relationship to Lean and Agile?

Scrum is different from Lean and Agile but it derives from both. It came about as an evolution in workflow process based on techniques pioneered at Toyota. Scrum is an adaptation of manufacturing floor process to software engineering.

  • Eliminate waste
  • Understand Value Stream Analysis
  • Implement Single Piece Continuous Flow

Agile competition – rapid prototyping and fail fast mentality, permit the customer to determine what the product will be jointly with the producer

What do you have to have and do in order to be doing Scrum authentically?

In order to do authentic Scrum, you must satisfy these requirements – these particular 3 artifacts, 5 events, and 3 roles as part of the team’s process.

3 Artifacts

  • Product Backlog – vision, priorities
  • Sprint Backlog – known work, capacity
  • Product Increment – scrum board, burndown, velocity

5 Events

  • Backlog Refinement
  • Sprint Planning
  • Daily Scrum
  • Sprint Review
  • Retrospective

3 Roles

  • Product Owner
  • Scrum Master
  • Team

What do these workflow philosophies consider worthwhile?

Scrum inherits values from Agile. The five values of scrum are focus, courage, openness, commitment and respect.
Agile Manifesto

  • Individuals and interactions over processes and tools
  • Working software over comprehensive documentation
  • Customer collaboration over contract negotiation
  • Responding to change over following a plan

Happiness is important because it is a precursor to great team performance and high velocity. Team velocity will predictably go down after happiness does. Scrum practice works to improve happiness by relying on intrinsic motivators over external ones – purpose, mastery, and autonomy over money, power and status. The workplace should ensure the external rewards are present to the extent that they are no longer preoccupying thought, but the flow of the team should emphasize those internal motivators over the external ones.

Best Practices
What are the best practices included in Scrum that help the teams that adopt them?

There were numerous best practices discussed during the class.  The first one that I’ll be introducing to my team is the observance of strict scrum rules during our standup.

During our standup, we will kick it off with a short inspirational music clip and physically stand up to report our answers to “What did you do yesterday”, “What will you do today” and “What are your impediments if any.”

Since we are a remote team doing our standup over a call, we will determine the order of statuses according to the order that the team members join the call. Since not everyone joins right at 10:30, the standup will start at 10:35, and the first five minutes will be used for team sync and/or a single ticket triage. At 10:35 the alarm goes off, and the team gets up for standup! Post scrums optional, anyone with a post scrum says who they need and everyone else should drop off.

One further thing I learned that I will mention is about measuring velocity. While you can’t compare velocity across teams, you can compare acceleration. Subtle but important.

What Scrum terminology provides useful ideas?

Kaizen (evolution) and Kaikaku ( revolution).Kaikaku means the philosophy of challenging entrenched dogma, refusing to accept waste no matter how it is disguised

Forms of waste

  • Muda – work in progress but not finished
  • Mura – inconsistency
  • Muri – unreasonable demands

Types of Waste and their Mnemonic
DOWNTIME – Defects, Overproduction, Waiting, Not utilizing talent, Transportation, Inventory, Movement, Extra processing

“Don’t take my word for it” – any change we make should have a follow up to ensure that the data proves the change is helping team velocity

JJ Sutherland, Alex Sheive, Sara Jarjoura
JJ Sutherland, Alex Sheive, Sara Jarjoura

All in all, I had a fantastic time and learned a ton. I am stoked to bring this to the team and see how we do.

Sara Jarjoura-ScrumAlliance_CSM_Certificate
Certified Scrum Master

Resiliency and Game Day Exercises at Acquia

In March of 2017 I came across the idea of “Game Day” in the DevOps Handbook by Gene Kim and others. Game Day is brilliantly advocated by Jesse Robbins in his presentation from 2011. It’s the idea that deliberately staging periodic system outages forces engineers to think about and design for resiliency in those systems. The extreme programming example is Chaos Monkey, which operates under only the one constraint that the outages should happen during working hours. Other than that, the outages caused by Chaos Monkey can happen anywhere in the system (even production!) and at any time.

Game Day is a step removed from Chaos Monkey, conceived of as a planned activity for engineers to resolve systemic outages. The resiliency exercises held at Acquia were yet another step away from the extreme towards the approachable. Our exercise included two activities, one geared for non-support engineers and the other for support. The non-support engineers had to bring back up a down site, and the support engineers had to attack and compromise an insecure site. The idea was to challenge engineers to step outside their comfort zone, and attempt to resolve technical challenges beyond the requirements of their every-day work.

The Team
The personalities involved in Game Day were a strong influence on the event. There’s Amin Astaneh, an Ops manager with the temperament of the proverbial town crier, faithfully and urgently supporting us in our DevOps transformation. Then there’s Apollo Clark, expert in secure systems who contributed the idea of doing a security vulnerability exercise. Finally there’s James Goin, seasoned Ops warrior relentlessly invested in the improvement of systems administration, including resiliency and disaster recovery training.

It just so happened that the idea for Game Day came two months in advance of Acquia’s annual engineering-wide event called Build Week, a truly awesome gathering of the entire team at Acquia HQ in Boston (read more on Dries’ blog!). Holding our Game Day at the same time would allow it to reach a broader audience across the company, so we requested a slot on the calendar. We ended up with 8-9pm on the Tuesday during Build Week. We had our opportunity!

Build Week imposed two constraints that had a significant and positive influence on our interpretation of Game Day. The whole event needed to fit in a single hour, and the event had to be accessible to engineers other than just the Ops subject matter experts. A Game Day exercise typically involves only the core engineering team which works directly with critical systems, and it takes however long they need to bring the systems back up. These constraints made the whole thing more approachable, and inspired the introduction of an Easy Mode and a Hard Mode.

Game Day as Exercise
The original idea was to have a trouble-shooting session with an Acquia development installation of a Drupal site (managed Enterprise-grade Drupal being the chief product of Acquia). The site would have some failure that either smaller teams or the whole group would have to resolve. Since we needed to accommodate varying levels and areas of expertise in the product, we settled on two “modes”, Easy Mode and Hard Mode, that participants would opt into based on their familiarity with troubleshooting techniques. The difference between the modes would only be in the level of difficulty. Easy Mode would be for those who don’t handle troubleshooting support calls as part of their regular day-job, Hard Mode for those who do.

The Identity Crisis
At this point, it hit home for me that the exercise was not going to be what I had originally intended – it wasn’t going to be a cookie-cutter Game Day. Although this seemed disappointing at the time, looking back it was a blessing in disguise, since it motivated us to create a new idea instead of copying someone else’s.

Apollo’s suggestion which we ended up following was to stage a Hard Mode Capture the Flag exercise instead of a site outage. Capture the Flag in a security context is an exercise where teams gain access to privileged resources in a system by leveraging security vulnerabilities. We could hide hashes – randomized strings of a fixed length – throughout the site. The winner of the competition would be the team that found all the hashes first.

The exercise would demonstrate that a site that works from a user perspective can still need work to become secure and performant. We would have Easy Mode to include some troubleshooting, which would then flow directly into the Capture the Flag exercise.

Trying It Out
We ran through the whole event a few weeks before Build Week. Easy Mode troubleshooting took up the first half hour, transitioning to Hard Mode Capture the Flag for the second half hour. This was pure thought experiment at this stage, and shockingly for me, it worked really really well.

During Easy Mode, non-Ops engineers drove the resolution with Ops experts only acting as consultants. Once the site was back up, we switched over to Capture the Flag. For this run through we only had one shared site for all the Hard Mode participants. One mischievous participant who found the site credentials deliberately locked out everyone else. This incident motivated much of the end-game setup for prevention of cross-site hacking.

Game Day!
Our Game Day-inspired exercise followed the flow established in our run through, with the addition of the isolated environments for Capture the Flag.

The Easy Mode troubleshooting took less time than we had allowed for, putting the start of Hard Mode right on time. The teams dove in, probing their environment – a Drupal site – for weaknesses. The narrative revolved around a fictional user submitting a question to the forum about how to enable the PHP module in Drupal, which would allow access to the bash shell on the server. The fictional admin replied that she had enabled the module for him, and reset his login to a “temporary password”. These were the credentials the participants were expected to use to hack the site. Since the user had access to the PHP module, they could also use it to gain shell access. Using this shell access to the server, they had easy access to the privileged resources and opportunities to discover the hashes.

When time ran out at 8:55, three of our twelve teams and forty participants had found all five of their hashes. The first team with all five hashes won the grand prize, an invitation for morning coffee with our resident tech celebrity, Drupal founder and Acquia CTO Dries Buytaert. As an aside, when I thanked Dries for agreeing to have coffee with our winners, he graciously replied, “No, thank you – now I get to have coffee!”

The decision to pivot from the established Game Day resulted in a new kind of learning in the spirit of Game Day. This learning was more accessible for our engineers and bridged the gap between where we are and where we are headed. While this isn’t the end of the story, I think it’s a fantastic start. Game Day, Day 2, here we come …