Categories
devops

Celebrating 10 Years in Cloud Engineering

The year was 2014 and I was working at Internet of Things startup Axeda (later acquired by PTC). I was building prototypes of Internet of Things solutions using the Axeda platform, and this particular one needed a UI, customer facing component. My manager suggested using an EC2 instance on AWS, which I had never tried before. I went through the flow of choosing an AMI, tweaking the knobs, and launching it. The invisible computer appeared!

That very year, the first annual State of DevOps report was released Nicole Forsgren, Gene Kim, Jez Humble and others. After the acquisition of Axeda, I applied for my first “DevOps Engineer” job and worked at Teradata automating the provisioning of flavors of Hadoop clusters across various clouds. Although SRE would later become known as the “implementation of class DevOps”, I always preferred to consider myself working in DevOps as that embodied to me the principle of bridging gaps between siloed developers and ops engineers.

At Acquia, I was able to mature my engineering experience to encompass Enterprise-grade practices. I worked with compliance concerns, went on call for the first time, and used Puppet to automate complex tasks. However at that time they did not offer a DevOps career track, so I transitioned into a full time DevOps role at Tomorrow.io (formerly ClimaCell).

It was at Tomorrow.io that I was able to find my sweet spot of working with Kubernetes on a daily basis, adopting Terraform and Helm as orchestration go-to tools. I used GitOps to ensure that the state of the deployment always matched the code, and hooked it all up to a Gitlabs pipeline for on demand releases.

The basis for my current work at Sensible Weather relies on the foundation laid by my combination of Enterprise and Start-up experience. I have the depth to understand how mature companies need to work, and the breadth to implement solutions tailored to the particular challenges of a growing org.

Ultimately, I am extremely grateful for the opportunity to enjoy meaningful work that remains fresh and interesting to this day.

Here’s to another ten years!

Categories
devops

Eliminating NaNs in PromQL Histogram Quantile

Working with Prometheus and PromQL can be tricky. I found it challenging to define an SLO using sloth-sli for a service whose histogram had a lot of NaN (Not a Number) values. I searched around for an out of the box solution and didn’t find what I was looking for.

The key to the solution was the knowledge that in PromQL an NaN doesn’t equal itself: NaN != NaN.

With this in mind I could use the “unless” operator to filter out any value that didn’t equal itself.

p50 Latency Error Query

histogram_quantile(0.5, sum(rate(http_request_duration_seconds_bucket{job="$job", namespace="$namespace"}[{{.window}}])) by (le)) > 5 unless (histogram_quantile(0.5, sum(rate(http_request_duration_seconds_bucket{job="$job", namespace="$namespace"}[{{.window}}])) by (le)) != histogram_quantile(0.5, sum(rate(http_request_duration_seconds_bucket{job="$job", namespace="$namespace"}[{{.window}}])) by (le))) OR on() vector(0)



This query returns the 0.5 bucket values that are over 5, and if the values that it evaluates do not equal themselves (i.e., are NaNs), it turns those into nulls. The nulls are evaluated to false (while the NaNs are evaluated to true) and then OR’ed with the vector 0 to create a consistent graph with no gaps.

Hope this tidbit helps someone, SLOs are hard enough as it is without NaNs in your life!

Categories
devops

Startup Life with DevOps at Sensible Weather

During my time so far at Sensible Weather, I’ve built an infrastructure pipeline which moves code from Github to an EKS cluster via Argo CD, a project lovingly named Margaritaville, since once you’re done with it you can go sip margaritas … 🙂 This project leveraged Terraform in Github Actions to deploy Helm applications onto the cluster. The only gap here was the management of secrets, which I initially deployed using Velero as a stopgap. Velero is a powerful tool which allows you to back up Kubernetes resources from a cluster and then restore them either to the same or a different cluster. Using it for secrets management didn’t make sense in the long term however, so as soon as I was able I migrated us to the External Secrets project, which enables the syncing of Kubernetes secrets to an external secrets provider. In this case we used AWS Parameter Store since we didn’t need cross-region or cross-account secrets sync ( and hey, free ).

Once this was all set up we needed centralized metrics, logging, and tracing, so I made use of the excellent Grafana open source LGTM stack . I used Pomerium to provide SSO for these tools which I covered in a post featured on Sensible Weather’s official blog.

Next steps – setting up an official on call with integrations into our comms. #StartupLife !

Categories
devops

DevOps at ClimaCell

I joined ClimaCell as a Senior DevOps Engineer in January.  I love it!  I am working in Terraform spooling up Kubernetes infrastructure as well as using Helm to bootstrap applications.

JAJ is learning her letters and walking around opening doors wherever she can.  Super proud of her.

Categories
devops scrum

Scrum Master Class

I have a new mission at Acquia – I am taking over as Scrum Master for my team. Bwahahahaha …

I’ve been doing Scrum ever since I started in tech full time at Axeda. Stand-ups and sprints are all old hat, but for the first time thanks to Acquia, I have received formal training in Scrum and a certification from Scrum Inc in Boston.

board
Scrum Inc.

Effectiveness
How does Scrum help? Why do it?

You use Scrum because you want to improve team velocity without increasing team resources.

I liked the fact that the Scrum Master class itself was organized as a Sprint using Scrum. This let us “live the example” and see first-hand the effectiveness of the techniques.

History
What is Scrum? What is its relationship to Lean and Agile?

Scrum is different from Lean and Agile but it derives from both. It came about as an evolution in workflow process based on techniques pioneered at Toyota. Scrum is an adaptation of manufacturing floor process to software engineering.
Lean

  • Eliminate waste
  • Understand Value Stream Analysis
  • Implement Single Piece Continuous Flow

Agile competition – rapid prototyping and fail fast mentality, permit the customer to determine what the product will be jointly with the producer

Requirements
What do you have to have and do in order to be doing Scrum authentically?

In order to do authentic Scrum, you must satisfy these requirements – these particular 3 artifacts, 5 events, and 3 roles as part of the team’s process.

3 Artifacts

  • Product Backlog – vision, priorities
  • Sprint Backlog – known work, capacity
  • Product Increment – scrum board, burndown, velocity

5 Events

  • Backlog Refinement
  • Sprint Planning
  • Daily Scrum
  • Sprint Review
  • Retrospective

3 Roles

  • Product Owner
  • Scrum Master
  • Team

Values
What do these workflow philosophies consider worthwhile?

Scrum inherits values from Agile. The five values of scrum are focus, courage, openness, commitment and respect.
Agile Manifesto

  • Individuals and interactions over processes and tools
  • Working software over comprehensive documentation
  • Customer collaboration over contract negotiation
  • Responding to change over following a plan

Happiness is important because it is a precursor to great team performance and high velocity. Team velocity will predictably go down after happiness does. Scrum practice works to improve happiness by relying on intrinsic motivators over external ones – purpose, mastery, and autonomy over money, power and status. The workplace should ensure the external rewards are present to the extent that they are no longer preoccupying thought, but the flow of the team should emphasize those internal motivators over the external ones.

Best Practices
What are the best practices included in Scrum that help the teams that adopt them?

There were numerous best practices discussed during the class.  The first one that I’ll be introducing to my team is the observance of strict scrum rules during our standup.

During our standup, we will kick it off with a short inspirational music clip and physically stand up to report our answers to “What did you do yesterday”, “What will you do today” and “What are your impediments if any.”

Since we are a remote team doing our standup over a call, we will determine the order of statuses according to the order that the team members join the call. Since not everyone joins right at 10:30, the standup will start at 10:35, and the first five minutes will be used for team sync and/or a single ticket triage. At 10:35 the alarm goes off, and the team gets up for standup! Post scrums optional, anyone with a post scrum says who they need and everyone else should drop off.

One further thing I learned that I will mention is about measuring velocity. While you can’t compare velocity across teams, you can compare acceleration. Subtle but important.

Terminology
What Scrum terminology provides useful ideas?

Kaizen (evolution) and Kaikaku ( revolution).Kaikaku means the philosophy of challenging entrenched dogma, refusing to accept waste no matter how it is disguised

Forms of waste

  • Muda – work in progress but not finished
  • Mura – inconsistency
  • Muri – unreasonable demands

Types of Waste and their Mnemonic
DOWNTIME – Defects, Overproduction, Waiting, Not utilizing talent, Transportation, Inventory, Movement, Extra processing

“Don’t take my word for it” – any change we make should have a follow up to ensure that the data proves the change is helping team velocity

JJ Sutherland, Alex Sheive, Sara Jarjoura
JJ Sutherland, Alex Sheive, Sara Jarjoura

All in all, I had a fantastic time and learned a ton. I am stoked to bring this to the team and see how we do.

Sara Jarjoura-ScrumAlliance_CSM_Certificate
Certified Scrum Master

Categories
code devops geek

Resiliency and Game Day Exercises at Acquia

In March of 2017 I came across the idea of “Game Day” in the DevOps Handbook by Gene Kim and others. Game Day is brilliantly advocated by Jesse Robbins in his presentation from 2011. It’s the idea that deliberately staging periodic system outages forces engineers to think about and design for resiliency in those systems. The extreme programming example is Chaos Monkey, which operates under only the one constraint that the outages should happen during working hours. Other than that, the outages caused by Chaos Monkey can happen anywhere in the system (even production!) and at any time.

Game Day is a step removed from Chaos Monkey, conceived of as a planned activity for engineers to resolve systemic outages. The resiliency exercises held at Acquia were yet another step away from the extreme towards the approachable. Our exercise included two activities, one geared for non-support engineers and the other for support. The non-support engineers had to bring back up a down site, and the support engineers had to attack and compromise an insecure site. The idea was to challenge engineers to step outside their comfort zone, and attempt to resolve technical challenges beyond the requirements of their every-day work.

The Team
The personalities involved in Game Day were a strong influence on the event. There’s Amin Astaneh, an Ops manager with the temperament of the proverbial town crier, faithfully and urgently supporting us in our DevOps transformation. Then there’s Apollo Clark, expert in secure systems who contributed the idea of doing a security vulnerability exercise. Finally there’s James Goin, seasoned Ops warrior relentlessly invested in the improvement of systems administration, including resiliency and disaster recovery training.

Constraints
It just so happened that the idea for Game Day came two months in advance of Acquia’s annual engineering-wide event called Build Week, a truly awesome gathering of the entire team at Acquia HQ in Boston (read more on Dries’ blog!). Holding our Game Day at the same time would allow it to reach a broader audience across the company, so we requested a slot on the calendar. We ended up with 8-9pm on the Tuesday during Build Week. We had our opportunity!

Build Week imposed two constraints that had a significant and positive influence on our interpretation of Game Day. The whole event needed to fit in a single hour, and the event had to be accessible to engineers other than just the Ops subject matter experts. A Game Day exercise typically involves only the core engineering team which works directly with critical systems, and it takes however long they need to bring the systems back up. These constraints made the whole thing more approachable, and inspired the introduction of an Easy Mode and a Hard Mode.

Game Day as Exercise
The original idea was to have a trouble-shooting session with an Acquia development installation of a Drupal site (managed Enterprise-grade Drupal being the chief product of Acquia). The site would have some failure that either smaller teams or the whole group would have to resolve. Since we needed to accommodate varying levels and areas of expertise in the product, we settled on two “modes”, Easy Mode and Hard Mode, that participants would opt into based on their familiarity with troubleshooting techniques. The difference between the modes would only be in the level of difficulty. Easy Mode would be for those who don’t handle troubleshooting support calls as part of their regular day-job, Hard Mode for those who do.

The Identity Crisis
At this point, it hit home for me that the exercise was not going to be what I had originally intended – it wasn’t going to be a cookie-cutter Game Day. Although this seemed disappointing at the time, looking back it was a blessing in disguise, since it motivated us to create a new idea instead of copying someone else’s.

Apollo’s suggestion which we ended up following was to stage a Hard Mode Capture the Flag exercise instead of a site outage. Capture the Flag in a security context is an exercise where teams gain access to privileged resources in a system by leveraging security vulnerabilities. We could hide hashes – randomized strings of a fixed length – throughout the site. The winner of the competition would be the team that found all the hashes first.

The exercise would demonstrate that a site that works from a user perspective can still need work to become secure and performant. We would have Easy Mode to include some troubleshooting, which would then flow directly into the Capture the Flag exercise.

Trying It Out
We ran through the whole event a few weeks before Build Week. Easy Mode troubleshooting took up the first half hour, transitioning to Hard Mode Capture the Flag for the second half hour. This was pure thought experiment at this stage, and shockingly for me, it worked really really well.

During Easy Mode, non-Ops engineers drove the resolution with Ops experts only acting as consultants. Once the site was back up, we switched over to Capture the Flag. For this run through we only had one shared site for all the Hard Mode participants. One mischievous participant who found the site credentials deliberately locked out everyone else. This incident motivated much of the end-game setup for prevention of cross-site hacking.

Game Day!
Our Game Day-inspired exercise followed the flow established in our run through, with the addition of the isolated environments for Capture the Flag.

The Easy Mode troubleshooting took less time than we had allowed for, putting the start of Hard Mode right on time. The teams dove in, probing their environment – a Drupal site – for weaknesses. The narrative revolved around a fictional user submitting a question to the forum about how to enable the PHP module in Drupal, which would allow access to the bash shell on the server. The fictional admin replied that she had enabled the module for him, and reset his login to a “temporary password”. These were the credentials the participants were expected to use to hack the site. Since the user had access to the PHP module, they could also use it to gain shell access. Using this shell access to the server, they had easy access to the privileged resources and opportunities to discover the hashes.

When time ran out at 8:55, three of our twelve teams and forty participants had found all five of their hashes. The first team with all five hashes won the grand prize, an invitation for morning coffee with our resident tech celebrity, Drupal founder and Acquia CTO Dries Buytaert. As an aside, when I thanked Dries for agreeing to have coffee with our winners, he graciously replied, “No, thank you – now I get to have coffee!”

Epilogue
The decision to pivot from the established Game Day resulted in a new kind of learning in the spirit of Game Day. This learning was more accessible for our engineers and bridged the gap between where we are and where we are headed. While this isn’t the end of the story, I think it’s a fantastic start. Game Day, Day 2, here we come …