Moderated user interview

Moderated user interview

Interview

Task

  • Help a neighboring product team conduct interviews for "Personal Account AI Gateway", a web app for internal AI agent developers
  • The service was meant to track agent implementation status, provide training materials, and enable work with DevOps
  • The product's key goal was to reduce time to market and the number of support tickets

What was done

  • Defined the research goal — to identify the current pain points teams face during the AI agent rollout process
  • Prepared a research guide: expected outcome, decisions the team could make based on the results, research questions, hypotheses to test, research method, respondent requirements, and the list of interview questions
  • Aligned the guide with the team and got contacts for suitable respondents
  • Conducted the research and, based on the insights gathered, formulated conclusions and proposals for improving problem areas

Methodology and respondents

  • Method: in-depth interviews, 40–60 minutes, remote, with video and audio recording
  • Respondent criteria: priority given to those who had already gone through the rollout process, or those preparing to do so and familiar with the documentation
  • Conducted 8 interviews with POs and analysts responsible for the AI agent rollout process

Hypotheses

  • Developers don't fully understand the structure and requirements described in the existing documentation
  • When facing difficulties, developers turn to colleagues for help more often than to the documentation
  • Developers don't understand what stage their agent is at (submitted, reviewed, approved, etc.)
  • The lack of a unified interface makes it harder to track progress and fragments the rollout process

Result

  • Conducted 8 interviews with team members (POs, analysts) responsible for the rollout process
  • Confirmed 3 out of 4 hypotheses
  • Identified key pain points in the rollout process and interface wishes
  • Proposed solutions to the identified problems
  • Presented the research results to the team