In your own words, what is ResearchOps?
ResearchOps is about offering support and infrastructure to ‘people who do research’, which I've shorted to PwDR. This is because in a lot of bigger organisations like Atlassian it's not just researchers doing research, there's also PMs and designers doing research.
“Of course, the holy grail is to have only skilled, practised researchers focus on doing research, but in most big organisations it's just not a reality. And so, for better or worse, it means that the services and support I offer are not just for researchers but a broader customer base.”
The kinds of things that ResearchOps is looking at is participant recruitment and things like tooling, procurement, managing relationships with vendors and a whole lot more. The ResearchOps framework we developed last year is a good reference for just how much ResearchOps can cover.
A lot of the work that I do is dovetailing or bridging the gaps between the research team and other parts of the business. I very often think of ResearchOps as being an API.
“I very often think of ResearchOps as being an API.”
My unique position is being able to understand and share what researchers need with finance, procurement, legal and even with people who are taking part in research.
What's the structure of your research team like in Atlassian?
When I arrived, around eight months ago, there was one person here doing what you can call ResearchOps but it's just not possible to deliver operations with one person. You can deliver aspects of something, but one person cannot handle research participant recruitment for an organisation this size - they will never even touch sides on it, never mind actually designing systems to make things more efficient.
What you end up with is one person doing some administration for the team, and never really having a chance to get to the actual operations.
After 7 months, we've grown to 5 people, and that's just a start. At the moment, we have two people working on participant recruitment but I could have a whole lot more. My reality check on that is that one person can handle around 15 requests at any one time, and when you're looking after 300 researchers/designers/PMs, you're going to have a lot more requests coming in than that. So, it could very easily grow to a team of 4 or 5 people doing just participant recruitment. I know that Google has had for 800 researchers, a team of 50 just doing research customer recruitment.
We've got two people who just joined the team to focus on engagement, which sits between the researchers and the Research Insights team (R&I). They'll be making sure that the right people in the organisation are seeing the research and are able to engage with it and act on it. Their superpower will be that they’ll be one of the few people in the organisation to have an overview of all the research going on across the business and how it all knits together, or doesn’t. That’s pretty powerful stuff. It's a very interesting, new development in the team and also means that we can gather insights from other parts of the organisation, like marketing, support and social media, who also learn from engaging with customers.
“Having the unique and powerful position of being one of the few people in the organisation who know what the research team is doing and learning from their research - is a superpower.”
What's the next area you'd like to grow in the team/Who would your next hires be?
I'm very keen on getting someone focused on what I’m calling QuantOps. There's a lot of tooling around Quant, and the support needs around it are very different to Qual. We have vendor relationships with a variety of Quant vendors that need to be managed, and there’s a lot of integration and APIs work - basically building an IT stack - that looks after the full workflow, and there’s compliance involved too .
With Quant, often you want to integrate with internal databases, and the amount of data you're moving around between systems is a lot bigger. So you need someone more technical who is able to work across the various teams and products managing those integrations and the architecture of all your products together. That role would also be looking at lab technology, audio visual technology...one of my fields of interest.
The next person would be a Digital Librarian, to work closely with the engagement team, who are also part of the library, but more the storytellers or journalists of the insights that it holds.
“It's an emerging field, if anyone says that they know it all, they're lying or delusional. One of the two. We're all trying to figure it out.”
Over the past 8 months, have you seen any key results from growing the team?
Massive results! I'm pretty proud of how far we've come in the last 8 months. The R&I team's grown (and the Ops team has grown with it), and there’s a lot of infrastructure which was not there before.
We're using Jira Service Desk and have set up a one ticket service for things like research participant recruitment and thank you gifts. So at Atlassian, you shouldn't need to worry about how we get participants for you. You only submit a ticket to us through our Research Help Service Desk, and within the ticket there's a brief like an online form that you complete. Then, Sarit, who runs the desk, will look through that and decide if this is something that we do from our internal panel, or it's something that we use this particular vendor for.
“We've procured a variety of vendors or pathways for recruiting various kinds of people - and worked hard on tooling around our Atlassian research group and on our community of participants.”
My joke is, you can tell us you want five Jira users who knit and are from Mars, and it would be our deal to get in touch with NASA to see if we could find those Martians for you.
We've got that system in place and built some great relationships with the teams that look after our customer databases. This means we can start to access people based on their behaviour within our products and not just on demographic. That said, it’s early days and there’s still loads to do. None of this is short-term or fast work.
More generally, how does ResearchOps impact design?
What we're finding here in Atlassian, similar to other large organisations, is we've got a relatively minuscule research team for the size of the organisation. It's growing under Lisa Reichelt’s incredible leadership but, for the moment, it means that a lot of designers and PMs are doing research and that's just how it has to be.
As a result, the scale of good research to not-so-great research is going to be pretty broad. What's interesting is you can end up building these operational services that are supporting both good and bad research with not a lot of ability to be filtering what you really want to support and what you don't.
“Our main question right now is how do we improve the quality of research so that the impact we're having as operations is really valuable?”
A lot of the operational things are decentralised until an Ops team comes in. The business will do research with or without you - they are still spending the money on thank you gifts and recruiting to greater or lesser degrees - but are they getting really good participants?
As a result, are they wasting employee time by doing research with the wrong participants and without the best tools, or skills?
When you do introduce an Ops team and you start to centralise all these costs, you suddenly start to realise, “We're spending x amount of thousands of dollars on thank you gifts for participants, and we're spending x amount of thousands of dollars on recruitment, and so much on tooling.” You really start to see these costs and how they add up.
“As the Ops team grows, the conversation about how much research is actually costing changes from, 'hey, let's be close to customers' to 'hey, it's very expensive to be close to customers'. Now that we can see it's costing us hundreds of thousands of dollars a year, let's be skillfully close to customers because there needs to be some trackable return on investment.”
That has an interesting impact on how people start to look at the quality and the value of research that's happening and I think that that story is going to develop over time.
What has been one of the biggest challenges that you've faced implementing ResearchOps?
One of the big challenges for us right now, is scale. We're really wanting to get the word out about the services we're offering. Like our service desk, where people can ask for thank you gifts, swag boxes, e-gift cards and Atlassian University vouchers. Where we've set up relationships with vendors and internal teams to be able to offer those things.
The response from Atlassians has been amazing, they're like, “oh my gosh, there's this formal place I can go and just drop in a few details and then stuff gets sent out to the right people, and then I don't have to lift a finger beyond sending a ticket. That's pretty incredible.”
However, what happens is that as people get to know about your service, there's balancing the growing demand without having the scale yet. However, you need the demand in order to argue that you need more headcount.
Any kind of startup is going to have the same issue and they’re probably nodding their head. How we manage that scale, the growing demand and what kind of demand we want - is the next big thing. Along with what kind of research we actually want to support. That’s key.
How are you communicating your insights to the rest of the business at the moment?
At the moment, it's ad hoc. Each person and team is communicating in their own way. Some of the engagement team's work will be understanding, communicating and making sure that the people who need to see the research are seeing it and taking notice.
However, we also need to look at a Research Library - a place to store what we're learning. Often what happens is, people will think about Research Libraries first, it's one of the things that comes up the most often. Which I think is fascinating, because until you've sorted the quality of your research, participants and analysis...making a research library is really not in your interest. At least in the hierarchy of needs, it's not the most important.
There is no point building a library that requires a massive amount of management and is also expensive to run and to keep populated. It's very rare to speak to someone who's put lots of money into libraries, to hear that it all worked out without massive management and constant gardening.
You need to have the foundations of good participant recruitment, good quality research, along with good analysis and the outcome of trustworthy insights in order first. This will then develop a real understanding of the kind of things need to be saved for the long term. Sometimes a decision path needs to be locked down in case it's questioned in the future. But also, it might be an interesting insight to be used in 6 months, a year, two years and so on.
“Until you've got all that in place, including your consent forms, privacy policies, how you store and analyse data - nevermind insights - a library is the last thing to look at.”
What I do think is fascinating and is something I'm very excited to be working on, is one of the biggest problems of libraries and why they fail all the time - which is that people want to take stuff out, but they don't want to put stuff in. And if people don't put stuff in there's going to be nothing to take out. And that's why libraries fail. The fanciest, smartest, sexiest libraries fail because of that reason.
To begin tackling this I started looking at the research pipeline. Where we first step in with recruitment, then consent forms, then with thank you gifts and then with tools to store your data coming out of that. And then right at the end we're stepping in with the library.
Right at the beginning, when someone submits a research recruitment ticket to us, they're telling us everything we need to know to make a library: who they're researching, who's doing the research, when it'll take place, what they want to learn etc. With that data, we have the beginnings of a very robust library.
Then our Digital Librarian can go back to the person that made the ticket and collate how much they spent, gather their report and what their impact was - and backfill from there. It's a really good way to start a library and it solves the problem of nothing going in, which is just human nature.
As this is a new field with new challenges like that, where do you go for inspiration and support in the community?
I have a lot of conversations with people who have gotten into the field. I hold office hours on Monday evenings and have people booking into chat. Oftentimes it's to ask me questions about what I'm doing, not unlike this conversation, but they become two-way conversations where I'm learning about the problems they're having and they very often line up with what I'm experiencing.
And then I also have connections with people like Tim at Airbnb, who's got a ResearchOps team and are ahead of my time. So, we share stuff and I get to learn from them, it's a small world so oftentimes we have the same vendors and the same business owners.
“Operations is not new, we've had factory operations probably since Henry Ford's times. A lot of his famous sayings, whether they are mythological or not, are around operations or his innovations around operations. So you can look to regular business operations for inspiration and for learning.”
Looking back over your career into ResearchOps, is there any standout achievement, or something that you're most proud of?
The first thing that comes to mind was my work at Government Digital Service (GDS) in the UK, which is really how I got into Ops. I was a content strategist, and Leisa invited me in to look at the content the user researchers were creating at GDS. I think at that time it was a 30-person research team or slightly less, and it grew to 40 or more people over time. That's how I ended up researching researchers and becoming one myself, even though I was trying to convince everybody I wasn't a researcher.
I built them a research lab, which was very successful and I was super proud of that work. However, something I've learned over the years is that, just because something worked very well in one context doesn't mean it's going to be a triumph in another context. You could do it all the same and it could be a white elephant because it just wasn't needed.
“There's no hard and fast rules, you've got to be learning all the time, sensing, thinking and developing new strategies because you're in a different place, with different needs.”
What do you see for the future of ResearchOps?
I would hope that the conversation around leadership, and what it means to be a research leader, becomes a lot stronger in the field than it has been. It’s taken seven or eight years for it to go from in-house teams and having a researcher on your design team being pretty unusual - to a situation where now we’ve got companies with 800 researchers, 300 researchers, 50 researchers, 10 researchers at least.
I think what's happened is that over that time, people have risen up the ranks either through skill or through sticking around and being pushed up the ranks, and have become senior researchers. Then from there, moving into managerial position. Some people have been brilliant managers but there are other people who aren't and don't want to be managers - so there's this gap or empty space, I think, in really understanding what it means to be a researcher leader.
It's typical as a researcher to ask, what is my journey? Is it junior, mid level, senior? And then ... where do I go? Is it manager? What if I don't want to manage? I don't know that that's been properly explored and defined.
From an Ops perspective, I could never do my job without an excellent leader/strategist on the team. Building pretty significant investments in staffing and procurements and money, it would be very difficult for me to make such sure-footed decisions if I didn't have someone who was really good on strategy and where the research team were going and where we're tracking.
What advice would you give to someone who is facing the same task as yourself, setting up ResearchOps in their business?
Become close to your team. It's understanding where the stresses are and how to alleviate them before they become problematic. I remember us looking at growth in December last year and seeing that by February we were likely to have a big problem with scale, and not being able to manage the tickets that are coming in. So we procured a contractor then and there to double up our research recruitment capacity now and it’s going to pay off.
I introduced this thing called the zenometer in my team which we use to measure how stressed we're feeling. We'll ask each other how's your zenometer tracking? If they say it's a 2 - that's terrible, it should be a 10, ideally.
It's like being a really cool seismometer that is feeling the vibrations in the ground and trying to understand when there is something coming down the line that could actually break our service and our sense of happiness. A team of stressed, overworked people is not fun. .
You need to have your feet on the ground, you need to be feeling what's going on ahead of time - a little bit of intuition, ear to the ground is what I'm getting at. Speaking to people and having an open conversation about what's working and not working, when they're stressed and when they're not, finding out what their worries are and then start dealing with those things straight away. Not thinking, 'Oh, I'll meditate on it for a while and maybe we'll have someone in 3 months time'. The moment you sense that something is coming down the pipeline, deal with it, and get something working.
Enjoy this article? Let us know on Twitter and subscribe to the newsletter so you don’t miss next weeks interview.
