Automating WebPageTest with a Ruby Script

WebPageTest is a great tool for seeing how well your website is performing. I was just on a project where the speed of a particular results page loading was the focus. How long the server took to respond, how long the UI took to render, what it was rendering, and where there were places for optimisations. We also wanted to measure the users’ perceived page load time.
We wanted to create an automated test that would run regularly so that we could track changes in page load over time.

Reasons why we picked WebPageTest

  • Ability to test any URL we wanted (ourselves, competitors, etc.)
  • Multiple browsers that can be used to test pages
  • Multiple locations and line speeds to test from
  • Can be automated (they have a REST API)
  • Ability to pin point a particular DOM element to measure load time of. For example we might consider the user perception of the page load as when there are 10 results on the page.
  • Provides a waterfall of all the objects loaded onto the page so that we can look for optimisations.

We set up the following to prototype (read: not production quality)  a top line measurement tool that the business could use to see how performance was tracking from a users perspective. With only a week of these tests running we found 3 issues that had been causing major problems with the page load time.

Where to start?

WebPageTest provide a RESTful API that we can use to interact with them. You will however have to host your own  private instance of WPT to be able to automate public URLs. You will also have to set up an agent for that instance to talk to.

Follow this guide for more information Setting up a Private Instance of Web Page Test. We set up  a couple of Amazon EC2 boxes for ours. (Make sure you pick at least Small rather than Micro)

Once you have that setup you should have a private instance of WebPageTest and URL that you can use to start manually testing your website.

For the purposes of this example I am going to test the BBC news website… I also used Ruby for this script. This was my first ever time creating a Ruby script for something like this, so keep that in mind!

These are some steps you can follow to build up a scripted test, and then store the results in a MongoDB for analysis later.

Step 1:

Call your instance of WebPageTest with your URL of choice and get back a 200 response

#!/usr/bin/env ruby
require "net/http"
require "uri"

test_url = "http://www.bbc.co.uk/news/"
baseurl = "http://[WEB PAGE TEST SERVER]/runtest.php?runs=1&f=xml&fvonly=1&url=#{test_url}"

response = Net::HTTP.get(URI(baseurl))
puts response

At this point you should get a response that looks something like this:

<!--?xml version="1.0" encoding="UTF-8"?-->

  200
  Ok
  <data>
    130723_9Y_BM
    d93f0a316369c61a891428dfb8b071d97b3dd19b
    http://[WEB PAGE TEST SERVER]/xmlResult/130723_9Y_BM/
    http://[WEB PAGE TEST SERVER]/result/130723_9Y_BM/
    http://[WEB PAGE TEST SERVER]/result/130723_9Y_BM/page_data.csv
    http://[WEB PAGE TEST SERVER]/result/130723_9Y_BM/requests.csv
    http://[WEB PAGE TEST SERVER]/jsonResult.php?test=130723_9Y_BM/
  </data>

WebPageTest will asynchronously run the tests and dump the results into the locations returned above. As you can see there are lots of different formats that you can consume for the results data. Everything that I needed was in the summaryCSV.

Step 2:

Parse the URL to get the summaryCSV and then poll that until the results appear (as we have no idea of knowing when they will appear I checked every 5 seconds until I no longer got a 404 response). I then parse the CSV response into a 2 dimensional array, where the first array are the headers, and the second is their values. You can then see all the results that come back and which ones might be interesting for you.

#!/usr/bin/env ruby

require "net/http"
require "uri"
require "nokogiri"
require "csv"

test_url = "http://www.bbc.co.uk/news/"
baseurl = "http://[WEB PAGE TEST SERVER]/runtest.php?runs=1&f=xml&fvonly=1&url=#{test_url}"

response = Net::HTTP.get(URI(baseurl))
doc  = Nokogiri::XML(response)

csv_url = doc.at_xpath('//summaryCSV').content

uri = URI.parse(csv_url)
http = Net::HTTP.new(uri.host, uri.port)
req = Net::HTTP::Post.new(uri.path)

while((csv_content = http.request(req)).class == Net::HTTPNotFound)
	sleep 5
end

raw_results = CSV.parse(csv_content.body, {:headers => true, :return_headers => true, :header_converters => :symbol, :converters => :all})

puts raw_results

Step 3:

Great! Now I have test results coming back and I want to save the ones I’m interested in so that I can visually represent them later and look at the changes in performance over time.
I’m going to use mongoDB to store the results of the tests. I created a class called TestResult with the fields that I am interested in from my WebPageTest results.
At this point you will need mongoDB up and running and you will also need a .yml file to define your mongoDB setup.
mongoid.yml

development:
  sessions:
    default:
      database: web_page_test_results
      hosts:
        - localhost:27017

My scripts now looks like this:

#!/usr/bin/env ruby

require "net/http"
require "uri"
require "nokogiri"
require "csv"
require "mongoid"
require "chronic"

Mongoid.load!("mongoid.yml", :development)

class TestResults
	include Mongoid::Document

	field :timestamp_of_test, type: Time
	field :load_time, type: Integer
	field :time_to_first_byte, type: Integer
	field :csv_url, type: String
end

test_url = "http://www.bbc.co.uk/news/"
baseurl = "http://[WEB PAGE TEST SERVER]/runtest.php?runs=1&f=xml&fvonly=1&url=#{test_url}"

response = Net::HTTP.get(URI(baseurl))
doc  = Nokogiri::XML(response)

csv_url = doc.at_xpath('//summaryCSV').content

uri = URI.parse(csv_url)
http = Net::HTTP.new(uri.host, uri.port)
req = Net::HTTP::Post.new(uri.path)

while((csv_content = http.request(req)).class == Net::HTTPNotFound)
	sleep 5
end

raw_results = CSV.parse(csv_content.body, {:headers => true, :return_headers => true, :header_converters => :symbol, :converters => :all})

TestResults.create({
	:timestamp_of_test => Chronic.parse(raw_results[1][:time]),
	:load_time => raw_results[1][:load_time_ms],
	:time_to_first_byte => raw_results[1][:time_to_first_byte_ms],
	:csv_url => csv_url
	})

And my mongoDB happily contains the result of my first test, which will look something like this

db.test_results.find()
{ "_id" : ObjectId("51eef7cee055e6cee5000001"), "timestamp_of_test" : ISODate("2013-07-24T21:38:04Z"), "load_time" : 3566, "time_to_first_byte" : 317, "csv_url" : "http://[WEB PAGE TEST SERVER]/result/130723_RW_CN/page_data.csv" }

What next?

If I had had a chance to extend this, I would of loved to of added visualisation, using a graphing framework and deploying the results to a web service so that everyone can see the change over time and drill down into any of the suspicious looking results.

Advertisements

The fear of blogging about technical topics…

I rarely blog about technical topics, despite my main trade of being a developer. The prospect has always scared me, and from others that I’ve talked to I’m not the only one.

The irrational thoughts that tend to go through my head are:

  • What right to I have to blog about technical stuff?!
  • Major case of Imposter Syndrome
  • What do I know that the internet doesn’t already?!
  • Will people judge me about what knowledge I do have?
  • What if  I’m wrong?!

Wonder if anyone else finds this too?

Well, we have to embrace whats scary to grow. So I have set myself a target of at least one technical blog a month.

These are my mantras to help me feel more confident about it

  • We are all always learning. There will always be people that know more about something than you, but there will always be people who know less. Things that I find useful when I am learning WILL be useful for other people who are learning.
  • If I had to hunt about on the internet for half an hour to find out how to do something,  if I can give an example that might save someone  haf an hour of googling, then I have made a tiny difference.
  • I must embrace feedback! If people leave comments, and share other ways that I could do things better, I should embrace and learn from these.
  • I am not ever pretending to know everything… these are just things that have helped me and I hope they might help others.
  • I can put things down here, so I can remember them for next time.

Here goes!

I’d love to hear any thoughts if you feel the same about technical blogging, or even better if you used to, and how you got over it.

The Retrospective Handbook by Patrick Kua

The Retrospective Handbook

One of my colleagues at ThoughtWorks published his first book last year called the Retrospective Handbook. It’s a great tool for people thinking about facilitating or wanting to hone their skills facilitating Agile retrospectives.

You can buy it here on Amazon or on Leanpub

I thought it was great… I’ve listed out here some of the things that the book covers that I really liked.

  • The right context for Retrospectives
    • The effect that team dynamics and environment have on the success of a retrospective
  • Lots of practical advice
    • How to prepare and set up a retrospective. Right down to the kinds of pens and paper best to use.
    • Lots of examples of language to use in specific situations
  • The importance of independent facilitators and advice on what to do if thats not possible and a team member facilitates.
  • Great tips for facilitators
    • How to help achieve equal participation from everyone.
    • How to handle anti-patterns from participants
    • Importance of observing body language as a facilitator and realising that there are many more things to ‘listen to’ of you participants that just what they are speaking out loud.
  • A whole section on Distributed Retrospectives
  • Your job is not done once people have left the room. There are a bunch of necessary things that need to get done after the retro is over.
  • 8 Common retrospective smells

Stand-ups – We can make them better…

I’ve been thinking about how to get the most out of stand-ups. They are something that all the teams that I have been on do every day, but  how often do they actually meet the real goals of a stand-up? I spent some time thinking about how, instead of just rattling off the usual status update, we can really think about how to best use this time. Not only to communicate  with and learn from the team, but also how to provide that daily re focus on what the team needs to do today to get awesome features into the hands of users. I propose some different questions to think about and frame your updates with during stand-up.

While I was hunting about the internet looking for inspiration, I came across Jason Yipp’s article on Martin Fowler’s Blog – It’s Not just Standing Up. There’s loads of great stuff in there, but specifically what jumped at me was the goals of a stand-up.

 The Goals of the Daily Stand-up are GIFTS

There are several goals for a daily stand-up  meeting:

  • To help start the day well
  • To support improvement
  • To reinforce focus on the right things
  • To reinforce the sense of team
  • To communicate what is going on

As a mnemonic device think of GIFTS:

Good Start, Improvement, Focus, Team, Status

I also came across this from Agile Coaching by Rachel Davie and Liz Sedley. A good way to sanity check how your stand-ups are going.

Signs of a healthy stand-up

– Is everyone engaged, motivated and excited?

– Are they making progress and working on high-priority tasks?

– Are they working together and helping each other?

– Are they able to concentrate and do their job without disruptions?

Reality of what many stand-ups tend to be and some of the problems

I think that it’s easy to say that you are ‘doing standups’, everyone goes around giving their 3 question updates (what I did yesterday, what I am planning on doing today and do I have any blockers), but  you have to ask how effective they are. Was everyone listening to each other? Do you even remember what each person said? Did you learn anything from stand-up? Do you have a better sense of where in the lifecycle any particular story is?

Just like other processes that the team follows, stand-ups should be continually improved and tweaked.

So. I don’t think that the 3 questions above are enough. I think they are great to get the team started, but teams often grow out of them quickly. As long as you keep the goals in mind, you can change the structure to be whatever you want it to be.

Why aren’t they enough? Some reasons:

  • There is too much talk of ‘I’.
  • Lacks focus on getting stories to ‘Done’.  Ultimately the team should be focused on how to get the best, most useful features into the hands of it’s customers, I would like to see updates the revolve around how to achieve that.
  • Too easy for people to hide behind them, robotically answering and it’s easy to lose emphasis.
  • The language is quite passive. Rather than stating what your blockers are, why not focus on what you are doing to get rid of them?

The differences may seem subtle, but the language a team use can totally change the team’s approach.

How to make them better?

Don’t think of it as a status report, more as a chance to communicate with and update your whole team at once.

Explore different formats of standups. I find that ‘Walk the Wall’ works quite well to ensure that everyone is focused on what needs to get done to get  features out the door.

Start to move away from the 3 basic questions. Talk with your team about what kind of updates you would like to hear during stand-up that would be most useful to everyone.  Change the language to be more proactive.

Perhaps think about framing your updates using the language below:

  • What am I doing today to get cards moved across the wall towards “Done”?
    • Ensures that the focus is on activities that directly relate to helping to progress work.
  • What remaining work is left? When am I likely to move it to the next state?
    • Helps downstream roles to know when to expect new work, helping them plan better. Gives the whole team the sense of where the story is in its lifecycle.
  • Meetings? Would the team benefit from hearing a summary?
    • I forever hear “I was in meetings all day”,  hopefully the meetings were project related, so share what happened. People want to know…
  • Did you learn something yesterday that people might benefit from knowing?
    • Help the team become more efficient, by sharing things that you have learnt. It may prevent someone from making a similar mistake, or struggling with something someone has already figured out. If it prompts interest, have a post-standup huddle for people to hear more.
  • What am I doing to remove/fix blockers?
    • This is an important language difference from the standard update.  Don’t wait for people to remove obstacles for you, see what you can do yourself to work round them or move them yourself, or even other peoples. It’s more often than not quicker than waiting for someone else to do it.
 So I prepared the following ‘cheat sheet’ to help you think about your update. You could even stick this up on the story wall itself to act as a memory trigger.

StandupOther great stand-up resources

It’s Not Just Standing Up: Patterns for Daily Standup Meetings http://www.martinfowler.com/articles/itsNotJustStandingUp.html

The Standup Game – Great game to introduce standup concepts, but also to bring awareness to the effects that stand up anti-patterns have on the team.  http://loveagile.com/stand-up/stand-up-game

Some ways to address anti-patterns http://www.scrumalliance.org/articles/358-daily-standup-beyond-mechanics-a-measure-of-selforganization

Gender Diversity on Teams – Roundup

Wanted to keep a note of a bunch of articles and books that I found recently around studies about gender diversity on teams. Prompted by a discussion with a programme manager at an investment bank and then subsequently with my fellow ThoughtWorks colleague Nic Ferrier

I found these often quoted “facts” and was trying to hunt down their sources…..

National Center for Women in Technology – Resources
The Impact of Gender Diversity on the Performance of Business Teams 
Gender Diversity and the Impact on Corporate Performance – Credit Suisse Research Institute
Gender Diversity, Team Decision Quality, Time on Task, and Interpersonal Cohesion
LBS study shows addition of women to teams improves performance

Beginnings of an Agile Coach

So, I’ve not blogged in a while…  I’ve been at a new client since January as an Agile Coach, it’s been quite the learning curve,  leaving me little mental space left for blogging!  To celebrate my return I have got a new blog theme. Pretty!

Agile Coach?! What is one of those?

Teacher, Instructor, Coach, Trainer, Encourager, Enabler, Questioner, Confidente, Counselor, Advocate,  Life Coach…. (Not sure all of those are words) in all things Agile. Helping the team achieve the best that they can and create and deploy high quality software as quickly as possible.

Most ThoughtWorks projects have some inherent enablement as part of their remit. Projects where we deliver software alongside the client normally involved an enablement piece, teaching and mentoring them as we both deliver software together. Showing the client how to do TDD, continuous integration etc. The main point here though is that we normally are the ones setting up and driving the process that the team follows, and sometimes show whilst ‘doing’ when all else fails.

The enablement part had always come quite naturally to me, I’m fairly reasonable at communicating, people tend to naturally follow me,  I’ve been working in Agile environments for 5 years now and my ski instructing background gives me a leg up on the teaching aspect.

For a while now I had been asking to go on a pure coaching/enablement gig thinking that I’d be quite good at it, and how different could it be?!

Very. Turns out, when you don’t have full control over everything, as well as the ability to jump in a ‘just do it’, it’s not so easy any more! As with any new project it always takes me a while to settle in and really feel confident about what I’m doing. It’s been a very steep learning curve, but I can happily report that I’m now really enjoying it.

What advice would I have given myself  back at the start of this enagagement?

Learning about Learning

I’ve spent quite a bit of time learning about the ways that people learn. Funnily enough this is where a lot of the things that I learnt whilst learning to be a ski instructor were very familiar. Acknowledging that different people learn in very different ways and that you, as a coach, have to adapt your techniques and approaches to suit each kind of learner.

I also spent a lot of time learning about the Dreyfus Model, which is a model that talks about how individuals acquire skills and the different techniques that work best for people depending on where they sit on the Dreyfus Scale http://www.learninggeneralist.com/2009/08/using-dreyfus-model-to-engage-people-in.html

I still have tonnes more to learn about learning, but I find it fascinating. I think it’s something that you really need to invest in if you want to become a great coach.

Being coached on how to Coach

I was lucky enough to be working with a small group of very experienced consultants. Not only could they share their ideas and war stories with me, but I felt comfortable getting feedback from them. They made an effort to be approachable and open, so I always felt like I could go and ask them questions, and validate the ideas I had and the approaches that I wanted to take.  It’s important to have that supportive group as you are starting out. Unfortunately I don’t think that anyone can teach you how to be a coach, so it’s really a question fo getting regular feedback, failing fast and going with your instinct.

We all have own style

Following on from the point above. It’s important to recognise that we all have our own styles of coaching and teaching. What works for one person in a situation may not work for another person in the same situation. When you are starting out it’s very easy to watch other, successful coaches and think, “Wow they are amazing at XYZ when they do ABC, therefore if I do the same I will be successful too”. Not true in most cases. Every time I tried to emulate someone else I was never as effective as if I just went with what I would naturally do. For example one of my colleagues is great at asking very probing, thoughtful questions to people to get them to think about what they are doing and the affect that it has. I thought, ah brilliant, downloaded a bunch of  “consulty” type questions and tried them out on the next person I spoke to. I was so busy trying to remember the right questions in the right situations that I wasn’t even listening to what they responded with, I also sounded distinctly un-genuine. Turns out that when I stick to my normal mannor of talking to someone and asking they things, I achieve the same outcome but in a different way.

Pace

Things will move a lot slower than you are used to in a delivery situation. People take time to learn and change, and this shouldn’t be rushed. Step back, take a breath and let things happen in their own time. The sooner you come to terms with this the more enjoyable the experience will be.

It also may feel like things are a lot slower because you are not the one thats physically doing the work anymore (writing code, doing analysis etc.). That lack of control will seem like things are going slower, but they are probably not that different.

Don’t get overwhelmed

When  you are first in an Agile Coaching environment you will like see infractions of what you think is “proper Agile” everywhere you turn. Especially if it’s a new team and they are brand new to Agile. You might feel like there is so much to work on that you don’t know where to start. Recognise that you can only do one thing at a time, and some things may be easier to work on than others. Try and find the thing that is causing the team the most pain and start from there. We also found that trying to work on something that they are already ‘doing’ will be harder than introducing something that is brand new that will really help them. You are much less likely to get a defensive reaction.

I found just keeping track of the behaviours I was seeing, coming up with things that I thought might help, and then reveiwing them daily with some of my colleagues really helped. It’s also worth thinking about  whether or not a task is worth the effort that it requires. The image below is a Impact/Effort Analysis chart that you could use. Rate the potential exercise/piece of work on this scale and think about which order to then tackle things in.

Effort/Impact Analysis