Our Research Journey: from the Wooden Spoon to Running Research Operations at Scale
This post is a trip down memory lane for our UX research team at UX studio. We'll reminisce about our incredible journey, from shoestring budgets and DIY research setups to running Research Operations at scale, not just for our internal projects, but for our global clients as well.
The Very Beginnings: A Focus on Core Usability
Five years before I joined, in 2013, UX studio began its UX research journey, focusing on usability testing for websites and web apps to improve user interactions. During these initial stages, we already started to employ a variety of research methods, mostly focused on evaluative techniques, including analytics to support our findings and card sorting to assist in designing intuitive user flows.
After the first years, which were dominated by delivering web projects, the team expanded its expertise in researching mobile products, integrating usability testing into more complex digital environments, and influencing strategic decisions.
Growing into Being Strategic Partners
As we grew more experienced, our focus shifted toward a more strategic role, informing product strategies and integrating our research into the agile workflows of our clients. We could only make this shift slowly, with much training, knowledge sharing, and experimentation. We gradually got better and better at speaking the language of stakeholders in different roles and levels and ultimately became better facilitators as well with a lot of practice.
After years of trying out different ways to best integrate UX design and research processes, getting experience in leading strategic workshops, and having a large workshop toolbox, we achieved what we wanted: we were there for our clients, helping them shape business strategies and value propositions. Through these efforts, we became more than just researchers; we became trusted partners, actively participating in and shaping our clients’ decision-making processes.
From Basic Usability Testing to Mixed Methods Research
UX Studio’s scope of research services expanded significantly over time. We moved from standard usability testing to conducting contextual inquiry, diary studies, building up a panel for beta testing, and more.
We aim to triangulate the data and pair insights from qualitative research with quantitative data. This way we can successfully inform decisions in every stage of the product development cycle, from strategic decisions and discovery to defining and monitoring post-release UX metrics.
Building up a Research Function from Scratch
Our work with clients often starts at a crucial point, when they’re bringing structured UX research into their projects for the very first time. We're typically the first researchers to join these organizations, and we set up their research functions and processes from scratch. As we work together, we figure out how to make research the most impactful and effective for their business goals.
These efforts, though not always labeled as such, are essentially what define Research Operations: activities designed to scale and streamline research within organizations. We not only help set up these systems but also continue to support our clients as their teams grow.
During one of my collaborations, we were localizing an application. After assuming the role of UX writers along with designers, we led a few meetings on what UX writing is, why it is important, and how it will contribute to usability, user experience, and user satisfaction. In the end, the team decided to hire an in-house UX writer.
We consider organizations hiring their in-house UX talent a success, and we remain engaged through consultancy and training, helping to optimize their ResearchOps and re-engaging as new needs emerge.
Our DIY Approach to Running Mobile Usability Tests
Tamás Virágh, currently Studio Lead at UX Studio, was amongst the very first UX researchers at the company, having joined in 2015. When I mentioned my idea to make a fun lookback article on our research practices, he made me swear to include the infamous wooden spoon.
As I have learned from Tamás, people from UX studio have built 4-5 versions of these mounts to find the best experience. They experimented with mounting the camera to the table which allowed for more free movement with the phone, but also hacking together a car mount and a HD webcam. Then, intending to optimize the experience and raise the comfort levels of test participants, we purchased a device specifically made for this purpose, a fairly light magnetic strip attached to the back of the smartphone, called Mr. Tappy.
Mr. Tappy was certainly an upgrade to the wooden spoon, although the setup and the use of it were still far from easy and comfortable. Smartphones got wobbly and couldn’t be held naturally. Even though mobile testing software (such as Lookback) that register taps started to enter the market, we believed in the added information coming from recording the hand and finger movements. Screen recorders were able to record taps, but they were not able to record hesitation, avoidance of tapping, or intricate hand gestures like swiping and pinch zooming, we stuck with Mr. Tappy for some time while also experimenting with DIY solutions, like having a 4k document camera attached above the table in our usability lab.
Smartphones were and are advancing at a dizzying pace, so looking back, it’s very amusing to see the hand-screen ratio on one of our in-person tests from 2016.
Up until 2020, we did a fair amount of in-person tests in our Budapest-based office. We had a couple of iOS and Android smartphones, so we could provide the participants with the system they were used to. We created a usability lab, and most of the time, it was fully booked for in-person or remote sessions.
Many more usability tests and user interviews were happening in person. Talking to my colleagues who have been researchers for 6-8 years, I would say that in 2016 the percentage of in-person tests was about 80%, probably 100% for mobile, as sending a prototype and screen sharing from mobile while having a selfie camera on was not possible, or would have excluded not so tech savvy participants.
Nowadays, it’s very straightforward and mostly frictionless to organize a remote testing session, and participants have no trouble opening up a high-fidelity prototype optimized for their screen size in a matter of seconds. If we look at our research collaboration with clients in 2023, 100% of user tests were remote. We have clients from all over the world, but even for local clients and Hungary-based audiences, the default is very much remote interviews and tests: no one wants the added effort of being in the same place physically.
Client Collaboration and Research: From Face-to-Face Interactions to Screens
Benefits of being in the same room when starting a new collaboration
Before the COVID-19 pandemic hit and lockdowns were in place, we had a “rule” when starting to collaborate with clients. Wherever they are in the world, we’d do an ideally 3-day long kick-off workshop in person, to get a fast start, build relationships, show our way of working, and essentially build trust.
Everyone, including me, had to have a valid passport at all times because you might have traveled to Beijing or Paris for the start of your new project. As a researcher, these were especially valuable interactions for me. In those 3 days, we had time to run stakeholder interviews and talk to customer success teams and sales - even though they are often not tightly involved in UX projects. I got to learn as much as I could about their business challenges, their target audience, and their way of working. It helped me craft informed and ready-to-go research plans.
Today, we still value collaboration and conduct moderated user research sessions just as much, but these are almost exclusively online. We had to react to the pandemic and post-pandemic realities of companies and their way of working.
Luckily, this shift also broke down some barriers since remote work and collaboration are tried and tested, being in a different county and timezone is not seen as a barrier to prospective clients working with us. And not only client collaborations have been affected, but interview, and usability testing sessions as well. Working and interacting from home is the new norm.
The power of in-person research sessions
If I’m being honest, I miss those in-person sessions, collaborative workshops, and trainings the most. You can just feel the full focus, people reacting to each other's ideas in real-time, being able to read all those non-verbal cues from participants. It has added value, personability, and fun to research. I’m a huge advocate for work-life balance and I can comfortably do my research work remotely as well, which made me extremely privileged during the lockdown period of the pandemic. However, I’m a bit disappointed that this has become the norm to such an extent that bringing up the idea of a guerilla testing session or arranging international travel for in-person workshops is often met with shock.
Also, please note the in-person card sorting conducted in the image below — a completely different experience for both the researcher and the participant. While it does have its drawbacks, I would assume that participants are more immersed in the process and give more thought-out and authentic feedback, while you cannot be sure of that with an asynchronously completed card sort of a “professional tester” online. In-person card sorts give you a lot of extra information by being able to observe the process real-time, and you can even ask a lot of follow-up questions or apply a think-out-loud protocol during the sorting process.
Is testing paper prototypes a thing of yesterday?
Before starting my career in UX, like many others, I took as many opportunities as I could to learn about this field, gain experience, and find out whether this was a path I should pursue. That’s how I first got to know about UX Studio: back in 2016, I participated in a paper prototyping workshop, where we sketched out flows and tested the paper prototypes with each other.
I won’t dive into the evolution of design tools and processes over the span of the last 10 years, but naturally, these affect research methods, techniques, and processes as well. Testing a paper prototype requires a completely different approach compared to testing a high-fidelity or programmed prototype. It used to be a topic of many blog posts to encourage UXers to produce and test paper prototypes because it was a fast and cheap way to get feedback on ideas, flows, basic layouts, and even functionality. However, no one seems to give that advice anymore, since designers can now whip up something mid-fidelity in Figma even quicker than on paper.
Championing Safe and Ethical Research
Tales of the paper consent forms and the year of GDPR
Many of the researchers at UX studio, including me, have completed a Psychology, Sociology, or other Social Science degree before turning to UX research. I believe that resulted in us, as a company, allotting large importance to ethical research practices and data privacy.
Even before GDPR, we had always applied Consent forms, anonymized the data, or cleaned it not to contain personally identifiable information. As we mostly did in-person research, we explained to participants what the session was about, basic rules, like they could pause or stop at any time, and how we would use it and what data to collect. Then they signed the paper-based consent form.
Later, when we conducted more and more remote sessions we started to use e-signature tools for sending consent forms, but making sure the consent form was sent, signed, and returned before the session on top of scheduling took a lot of work. But then GDPR came in 2018 and we knew that all kinds of research we do, may it be surveys, or usability tests will fall under this regulation.
We knew we had to include lawyers in mapping out what we are able to do when conducting research, and also, what are our obligations. It’s one thing that someone has an elaborate T&C, but the other part of it is that researchers have to be informed and trained on how to abide by the regulations and the policies that bind us. That’s why it was a collaborative effort, and I worked together with lawyers to draft the policies and T&C of research practices. Then we trained the whole team in it, created a manual, and automated processes to ensure safe data management. We made the “GDPR Presentation” part of the onboarding of any new researcher joining UX studio.
Participant Management
We did and still do recruitment ourselves. That means that with the use of different tools and platforms, we manage everything that has to do with recruitment: ads, posting on social channels, setting up screeners, obtaining informed consent, selection process, and scheduling. We often share our best practices and help each other out.
Often, we could not find quality participants who fit the criteria in Facebook groups, got banned from Reddit, got no reaction Slack groups, or had targeted Instagram ads performing poorly. But because we had a large collective experience with different platforms we were always able to share best practices.
My colleague, Fanni Zsófia Kelemen collected all commonly used recruitment methods - and someone who has years of experience working at UX studio will have tried all of their approaches at one point. I prefer not to “hack” Reddit and scour through scammers, but sometimes that gets the most authentic people to test with. Sometimes, you have to go outside the scope of people who are tech-savvy and signed up to the panel of a recruitment vendor.
Over the years we have tried many vendors with large panels of participants. We now know what are the product types, cases, methods, and geographies where we can harness these panels for more efficient and fast research. Having spent 10 years in the industry and having worked with 200+ partners, we could consistently meet recruitment needs in the vast majority of cases, even with niche and hard-to-reach audiences. Why? Because collectively, we have experience in all recruitment methods, know the larger panels, and when something doesn’t work out we can quickly move on to the next approach.
We are familiar with several vendors; we know which vendor works well for a specific region, for B2B or B2C audiences, for different platforms, and different methods — survey recruitment often can be less precise than that for a diary study. We can confidently help our clients build out their own participant management system, and ensure ethical and safe data governance practices in research.
From a Simple Toolstack to an Individualized One
The maze of research tools
The landscape of research tools is exploding. Every day, it seems, a new platform emerges promising to revolutionize the way we gather data, analyze results, and synthesize insights.
Personally, my feeling of excitement over a new tool turned into feeling overwhelmed. It’s extremely hard to keep up and know all the tools that are out there that integrate into research processes. Can’t say that we have it all covered, but having 15 fellow UX researchers gives us better “coverage” and knowledge on these different tools we have tried on different projects gives an edge that would not be the case if someone were to work for the same company for years in a small team.
Being in an agency gives us plenty of opportunities to try out different tools. With the experience gained, we can provide individualized recommendations to clients on what tools to work with: for example, a mobile behavioral data analytics platform, a recruitment vendor for large-scale surveys, or a centralized source of research findings and insights.
Everyone’s knowledge management and repository look different
Nowadays, we could say that more and more knowledge management systems — or in other terms, research repository tools — pop up in the market.
Years before dedicated repository tools gained traction, we made it a standard for every researcher to be able to build up a research system in Airtable based on atomic research principles. You can read our guide on the topic from 2019 where Tamás Virágh explains how we moved from systematically storing observations in Google Sheets to a database tool, where we organized objective evidence-backed observations into insights.
Fast forward four years and we released a comprehensive ebook titled ‘How to Build Research Repositories from Scratch’. During these years, some of us experimented with using dedicated repository tools, and others have gained experience in managing the politics of advocating for building a knowledge management system.
We always strive to bring the most out of research, and wherever appropriate, we rally to decision-makers and put the initial extra effort into building a research repository, because we believe in its benefits. Knowledge Management as one pillar of Research Ops is something that requires researchers to educate themselves, experiment, fail, iterate, and then reap the rewards of a centralized source of knowledge. We always like to share what we have learned with each other and other UXers, so if you are based in Budapest, look for our event where our UX Researcher experts share how they find the right tool for managing insights effectively, get stakeholder buy-in, build and scale research repositories.
Research Operations as an External Service
The evolution from foundational research to handling operations
Since UX studio’s founding in 2013, when user experience design was guided purely by foundational research, to now setting up Research Operations (ReOps) at scale, we as a team have significantly evolved. Initially, our work did not explicitly focus on ResearchOps roles; instead, we integrated these practices within our broader UX research and consultancy projects. When we discussed it amongst ourselves, we realized we hadn’t done it in the context of fulfilling the dedicated role of ReOps specialist, but in reality, we have already covered all areas of Research Ops in our collaborations with long-term clients, especially multiple-year-long collaborations.
Partnering with an agency for research operations
Many companies struggle to establish research practices from the ground up. Integrating research seamlessly into existing workflows, building strong stakeholder relationships, and establishing efficient operations (ResearchOps) can feel like a daunting task. We lead the building of entire research functions, and that's where we can be a valuable partner for you.
Our established processes in participant recruitment and management, along with data governance protocols, mean you can initiate projects rapidly without the typical delays of setting up operations from scratch. With our processes, we can run research sessions with members of your target audience right away.
Flexible solutions for varied client needs
Our approach adapts to each client's specific requirements, providing flexibility whether you choose to utilize our tools and processes for the short term or over extended periods. We cater to a diverse client base, from small startups to large corporations, adjusting our services to match their evolving research needs.
We also provide training and support, focusing on embedding efficient research practices within your team's workflow. With our research training, your team can make the first steps to incorporate research into the product development process, or to scale up already existing processes. This helps minimize the operational load on your researchers, allowing them to concentrate on delivering insightful, quality research.
Building a robust research infrastructure
Partnering with us goes beyond transactional support; it involves developing a deep, collaborative relationship that enhances your research capabilities. Our experience across various industries enables us to offer tailored advice and solutions that improve research outcomes and decision-making processes.
As we look to the future, our commitment remains to support organizations in developing their research functions to better understand and meet user needs. Read more about how we can help your team raise the effectiveness and impact of research practices with our research experts.
