direct-file/docs/design/Direct File Research Templates and Protocols.md

Research Templates and Protocols

Protocols and Strategies for Research Sessions

Observers and Notetakers Protocols

• As a general user research best practice, research participants shouldn't feel outnumbered by research staff. Having more people than just the moderator can affect the participant's behavior, rapport, and impact findings. This becomes more important when addressing sensitive topics.
• Taxes are a difficult, sensitive topic for most taxpayers, depending on the taxpayer's situation. The team prioritizes participant psychological comfort when including observers and notetakers in research sessions. We also prioritize including IRS Direct File team members in research sessions to increase empathy with taxpayers.
• For each research session, we recommend no more than 3 research staff members max (including the moderator) to ensure participant psychological safety while allowing exposure to users for DF team members.
• Reflect if the research study will include sensitive topics such as complex family situations (e.g., a hostile divorce resulting in disagreement in dependent custody that affects tax filing), low or no income, transiency, and more.
• Participants can decide if they want to allow observers and notetakers in their research session, or if they want to talk to the moderator only. This check can happen during recruitment screening and again as the research session starts. If the research topic is sensitive, consider whether notetakers and observers should be reduced in number or even just the moderator (with the aid of video recordings for post-session notetaking). Make sure this is made clear to participants during the recruitment process (screening survey and phone screening questions and recruitment comms).
  • If recordings are possible and/or the moderator is confident to handle notetaking on their own, give participants the ability to choose whether they want to have observers/notetakers in the session during the recruitment process. Reiterate this option to them as the research session starts, in case they change their mind. This option will make participants the most comfortable and result in better rapport and findings, as it gives them the option to have a 1:1 with the moderator.
  • If the topic is sensitive or Federal Tax Information is involved (like in end-to-end observation sessions), it may be better to not record. If in this case the moderator needs a notetaker, make sure the participant is aware of this in advance and that this is contemplated in the recruitment screener. A notetaker may also be required in studies when the moderator isn't sure recordings are going to work.
• To help make participants comfortable, ensure any notetakers and observers present are on-camera at the start and introduced by full name and organization. Being on camera is not required beyond this initial introduction.
• Consider making video recordings available to team members after the session if research staff attendance must be limited. Videos should only be made available to team members working directly on the topic of research study and only for the purpose of research analysis.

Guidelines for Observers and Notetakers

• Please be on time and on camera when we start. The moderator will quickly introduce you by name and organization. Being introduced at the start helps make the participant more comfortable (talking with the IRS can be intimidating!)
• Observation of research sessions may not be guaranteed. You may have to wait a few minutes in the web conferencing waiting room as the moderator checks with the participant if they will allow observers/notetakers. Talking about taxes is extremely personal and sensitive, this practice helps increase participant psychological safety. If you're not able to join a session, please ask the research lead for access to research session recordings and/or notes before the research study ends.
• If possible, please stick around for 15 minutes after the participant leaves the room so we can debrief the session.
• If you have any questions that you would like to ask during the session, please send them in a Slack group chat with the moderator and notetaker. The moderator will have the chat open during the session and will ask the question when there is a good opportunity to. Do not include ANY PII (personal identifying information) in this chat.

Guidelines for Notetakers

• Note what participants say and also what they don't say--things they complete successfully, things they misunderstand or do wrong, where they try to click if relevant.
• Try to capture what they said, close to verbatim — not your interpretation of what they meant. E.g., “I’m not sure about this checkbox” does not equal “Doesn’t understand checkbox”
• Protect participant's PII (personal identifying information): Do not capture any sensitive details or PII. It's ok to capture details like what state they live in but if they start to get really specific on some detail, it’s ok to only capture a generic version.
  • “I’m a waiter at the Copacabana on Miami Beach” -- it’s ok to just write “waiter”.
  • “I had a tricky tax situation because I had 29 dependents” -- it’s ok to just say “I had a tricky tax situation because I had [much higher than average] dependents”

Research Moderation Protocols

We do dry-runs of the moderation guides before the real sessions

Find a colleague who's not involved with the project to do a dry-run, even if they have to make up a situation. This will help ensure moderation feels natural and that the interview can be completed in time. Make time after the session to get their and a notetaker's feedback on the questions and the notetaking templates. Before each dry-run, make sure to warn the "participant" that the moderation guide is work-in-progress and some things can be messy, but their feedback is important to help shape the moderation guide.

We do not provide tax advice to research participants

Participants may have questions about their tax situation or seek clarification on some aspect of how taxes work. We should never give tax advice.

If we're observing someone actually filling out their taxes as part of user research, and notice they made an error, what we can say will depend.

• Typos: Not tax advice. That's just noticing an error. The numbers are what the numbers are. If you see something, say something.
• Inaccurate selection: Be cautious -- we can't tell someone how to answer a question. But an "are you sure" or equivalent application of brakes would be fine. Possibly just reminding them there's a chat service if they have questions.
• Doing something that could result in a rejection like guessing their last year's AGI: Telling them that's not what the question asks and explaining what will happen if the number is wrong is a statement of fact, not tax advice.

When in doubt, don't say anything.

We protect participant PII (personal identifying information)

We follow these IRS mandated best practices for protecting participant PII.

• Participant screening data has to be in an encrypted file within the recruitment vendor’s secured network, limited to the project team working on the project – NO cloud based tool when collecting screening data.
• Recruitment partners must remove participant names, contact info and other linkable PII before sharing participants to the IRS (they will only be identified by participant ID numbers).
• Any summary of aggregated and de-identified participant characteristics from the recruitment screener will use secure email with password protected attachments.
• Any documents that contain participant PII should be deleted upon study completion.
• All other anonymized documents with participant notes to be deleted within 1 year.

Additionally, it's Direct File best practice to:

• Make sure participants don't enter any of their own PII into the dev site during a session. We do this by using prototypes that can't be entered into, we share our screens so they can't enter any data, or we give them fake data to use during sessions.
• Make sure any notes we capture during a session don't include PII or other sensitive information. See guidelines for notetakers above.

When the participant fails and gets frustrated during usability testing

Sometimes participants can feel like they're failing something when they don't successfully complete a task. Even if the moderator gives advance warning of that happening in user research and framing it as a normal thing, there are a few things the moderator can say to calm participants down without guiding them towards a solution.

Make sure to frame and describe the activity before it starts so the participant knows you want from them

• “We're testing the website and not you, which means there's no right or wrong answers we're looking for - we're interested in your general thoughts and feedback.”
• “As you're going through the prototype, please think aloud. You don't need to read from the screen, as that can take time, but I'd like you to say whatever comes into your mind. I'm interested in what's grabbing your attention on the screen, what questions you have, and your general thoughts as you review a page.”
• “If at any point you feel uncomfortable or confused, please let me know. We are testing the site and gathering feedback and testing not you or your tax knowledge. We’re here to support and learn from you.”
• “If you have any questions as we go along, please feel free to ask me. I may not be able to answer them right away since we’re interested in how people would navigate this website without a helper around. But if you still have any questions when we’re done, I’ll try to answer them then.”
• “And please feel free to be as honest as possible – you will not hurt our feelings.”
• When a participant get frustrated, reassure them they're doing well. “If you stumble on this, then a lot of people will, so you just found something for us to fix.”

Research Session Debrief Protocols

The research moderator may gather with the observers/notetakers and others who might be interested in joining a research session debrief. The following questions are recommended:

• How is everyone feeling? Does anyone need anything (moderator especially)?
  • If this was a difficult session, process what made it difficult, what went well, what could've been better, what you can do moving forward, etc. Remember that sometimes sessions are difficult and there's not much you can do about it. Be kind to yourselves.
• Does the research moderation need any changes? (e.g., phrasing of questions to avoid leading, changing of content in screens)
• Is there any PII (personal identifying information) we need to delete? Anything sensitive the participant shared that we need to clean up from the notes? (must be done the within 24 hours of the research session)
• What were the participants' needs and goals?
• What were the pain points in the experience?
• What went well in the experience?
• What are ideas and areas of opportunity for this participant?
• Next steps?

Sharing Research Insights Templates and Best Practices

	As research sessions are ongoing	After research sessions have concluded
Actions	Share updates and in-progress, preliminary findings with your study team and stakeholders via Slack. You can also post in a team Slack channel for broader team visibility.	Hold an interactive synthesis workshop with designers and other members of your study team to collectively form initial findings and recommendations. If needed, hold a broader share-out of finalized insights and recommendations with the study team and additional stakeholders.
Goals	Keep the team abreast of tidbits and interesting facts during research Give team members who were not able to participate in sessions a chance to respond to notes and ask questions	Deliver actionable insights and recommendations to individuals who can effect change on the product Keep a record of research findings for future reference
Audience	Study team (designers, content, PMs, other relevant stakeholders)	Study team (designers, content, PMs, other relevant stakeholders) Broader team
Dos	Share quotes, things participants did	Surface key themes Identify potential additional research opportunities Highlight study limitations (e.g., biases, audiences missing, etc.) Add all recommendations to backlog for future
Don'ts	Turn these research nuggets into themes prematurely Allow these research nuggets to be interpreted as themes prematurely, before synthesis has been conducted	Frame qualitative research "representative"—no qualitative research can be representative of all taxpayers. However, we can aim for a diversity of experiences and backgrounds
Template	[Title, research goals, context into study] Sessions completed: x out of x One participant did/said [something] [highlight pain points, positives, or questions the participant had, keep it brief] "[Quote, if there's a relevant one. We don't have transcripts but paraphrased notes can make up for it, as long as they are highlighted as paraphrased. Try to reduce paraphrasing to a minimum by reducing quotes to short sentences or even single words you know for sure were used by the participant. Make sure no PII is included in quotes]" (paraphrased) [2-5 bullets as needed. If no sessions today, highlight that, along with any prospect of future sessions:] No sessions conducted today / X no-shows / Prepping for 3 interviews Thursday / Conducting synthesis today and tomorrow	Presentation Share-outs to audiences outside of the immediate study team may benefit from a presentation Sprint demo: At the end of every sprint, the research pod shares an update on our work in-progress as well as the top 3 insights from any studies completed during the sprint. Use the template above for sprint demo share-outs. Document all finalized insights and recommendations within the study's study brief page for documentation purposes.

Moderation Guide Template: Usability Testing

This is the template that the Direct File research team used to facilitate usability testing.

Intro

allow participant in from the waiting room, keep observers/notetakers in the waiting room (you may allow a notetaker in at this point if the study plan required it). Ensure Slack and notifications are paused if preferred.

Good morning/afternoon. How are you today? I’m [moderator name], a [insert role] from [insert organization].

[if a notetaker is part of the plan] I wanted to introduce you to [name notetaker by their full name and what organization they are affiliated with and their role. Ask them to be camera on, wave, then turn camera off to make the participant comfortable].

I’m/we are part of a team that's building Direct File, a website for people to file taxes directly with the IRS.

Before we start, I want to let you know more about what you can expect during today's conversation:

Time: It’ll take no longer than [insert time]. I’ll be watching the time to make sure we don’t go over.

Session:

• I'll start with a few questions to learn a bit more about your past experiences filing taxes. Then I'll have you try out a prototype of the website we've been working on. [describe specific topics that will be covered, including any sensitive areas or lines of questioning.]
• We're testing the website and not you, which means there's no right or wrong answers we're looking for - we're interested in your general thoughts and feedback.
• Taxes are hard [if covering other sensitive topics, name those as well, e.g. “talking about shared custody can be difficult and bring up a lot of feelings”]. If some topics become difficult to talk about please let me know, I'm here to support you. You can let me know if you need to take a break or stop at any point. If there are any questions you don’t want to answer just let me know and I’ll move on.
• Unfortunately, we're not tax experts and we can't give you tax advice. I also don't have access to your tax information.

Privacy:

• We'll only use what we learn from you to help improve Direct File.
• Everything you say will be anonymous and be kept strictly private. We are taking notes — but those won’t include your name or any other personal identifying information. If there's anything else you want us to remove, you can always let us know, even at the end of the session.
• To help protect your personal and tax information, personally identifying information like your full name and your email is not shared with the IRS or other government agencies. The recruitment agency deletes that information after the study is completed.
• Your participation today is entirely voluntary. You'll receive compensation as a token of our appreciation for your participation. You do not have to answer any questions that you do not wish to answer but please keep in mind, there are no wrong answers.

Notetakers [if they opted in for observers and notetakers]:

• You mentioned you’d be ok with having a [additional] notetaker in our session today. Are you still ok with that, or would you prefer to have the session without them today? Either way is fine with us.
• [if participant accepts to have notetakers, or an additional notetaker, let them in the Zoom from the waiting room] Ok, I will let them in. I wanted to introduce you to [name notetakers/observers by their full names, role, and what organization they are affiliated with. Ask them to be camera on, wave, then turn camera off to make the participant comfortable]. Most of the conversation today will be between you and me and they will be off camera, but I wanted to make sure they had a chance to say hello.
• [if participant declines to have notetakers, or an additional notetaker, quietly remove them from the waiting room] Ok, no problem.

If recording:

I would like to record this session. The recording is for research purposes only and your name won't be associated with it. Recordings will be kept in a secure location and shared only with the people running the study for the purpose of improving Direct File. Are you ok with recording this session?

[when using Zoom] Before I start recording, I’m going to change your name on Zoom so that the recording doesn’t include any full names. [change name and start recording]

• Do you understand these guidelines?
• Are you ok to continue?
• Do you have any questions before we start?

Prototype Introduction - Moderator controls the prototype

Next we’re going to look at a prototype of the website. There are a few things I’d like you to keep in mind:

• I’ll describe a scenario and then give you a set of tasks to complete using the prototype.
• This is a prototype and not the real website. [Invision: It closely matches the real site, but some things may not work as expected.] You may see placeholder information that doesn't match your own situation exactly and that's ok.
• I'm going to share my screen and you can tell me what to do, what to click, where to scroll.
• There are questions that will ask you to input information. When we get to these points, you can tell me what you would input, and then we'll move on.
• As you're going through the prototype, please think aloud. I'd like you to say whatever comes into your mind. I'm interested in what's grabbing your attention on the screen, what questions you have, and your general thoughts as you review a page.
• If at any point you feel uncomfortable or confused, please let me know. We are testing the site and gathering feedback and testing not you or your tax knowledge. We’re here to support and learn from you.
• If you have any questions as we go along, please feel free to ask me. I may not be able to answer them right away since we’re interested in how people would navigate this website without a helper around. But if you still have any questions when we’re done, I’ll try to answer them then.
• And please feel free to be as honest as possible – it won’t hurt our feelings.

Pull up testing environment and share screen

Prototype Introduction - Participant controls the prototype

Next we’re going to look at a prototype of the website. There are a few things I’d like you to keep in mind:

• I’ll describe a scenario and then give you a set of tasks to complete using the prototype.
• This is a prototype and not the real website. It closely matches the real site, but some things may not work as expected. [Invision: It closely matches the real site, but some things may not work as expected.] You may see placeholder information that doesn't match your own situation exactly and that's ok.

I'm going to send you a link to the prototype in the chat and ask you to open the prototype and share your screen so I can follow along as you go. Does this sound okay to you? [If participant is uncomfortable - Alternatively, I can share my screen and you can tell me where to click and what to do. Switch to moderator controls prototype intro].

• Send link to prototype in chat, confirm has it open

• Can you share your screen now? (Zoom) You can share your screen by clicking on the green "Share screen" button in the bottom bar. Then, select on the option that has the website open. Let me know if you need any help with this.

Before we continue, I have some more information for you.

• As you go through the site, there'll be places where you can type in information. Please do not enter ANY real information about yourself. I'll give you fake data to enter.
• As you're going through the prototype, please think aloud. I'd like you to say whatever comes into your mind. I'm interested in what's grabbing your attention on the screen, what questions you have, and your general thoughts as you review a page.
• If at any point you feel uncomfortable or confused, please let me know. We are testing the site and gathering feedback and testing not you or your tax knowledge. We’re here to support and learn from you.
• If you have any questions as we go along, please feel free to ask me. I may not be able to answer them right away since we’re interested in how people would navigate this website without a helper around. But if you still have any questions when we’re done, I’ll try to answer them then.
• And please feel free to be as honest as possible – it won’t hurt our feelings.

Global questions/instructions/observation points throughout the session

• What are your takeaways from this screen?/Can you describe what's going on in this page?
• What would you do next?
• You click [button label] and this is what you see next.
• What do you think will happen next?
• What are your questions or concerns so far?
• How would you go about answering this question?
• What questions do you have about how to answer this?
• What might make this hard or confusing to answer?
• What kind of help would you be looking for to answer this question?
• How confident are you that you're answering this question accurately?
• Do you have any other questions not otherwise answered on the page?
• How do you feel about your understanding of [insert term]? How concerned or not concerned are you about your understanding?

When looking at help content:

• What are your thoughts on what you're reading?
• How helpful or not helpful is this information?
• Do you have any additional questions?

Wrap-Up Questions

That's all the questions I have for you today.

• Before we end, do you have any questions for me?
• Any last thoughts or feedback we didn't get to discuss already?

[Confirm incentive payment steps as needed]. Thank you so much for taking the time to share your thoughts and feedback. I'll bring your input to our team to help improve Direct File.

Moderation Guide Template: End-to-End Observation Session

Intro

Allow participant in from the waiting room, keep observers/notetakers in the waiting room (you may allow a notetaker in at this point if the study plan required it). Ensure Slack and notifications are paused if preferred.

Good morning/afternoon. How are you today? I’m [moderator name], a [insert role] from [insert organization].

[if a notetaker is part of the plan] I wanted to introduce you to [name notetaker by their full name and what organization they are affiliated with and their role. Ask them to be camera on, wave, then turn camera off to make the participant comfortable].

I’m/we are part of a team that's building Direct File, a website for people to file taxes directly with the IRS.

Before we start, I want to let you know more about what you can expect during today's conversation:

Time: We have 3 hours together today. If we end before 3 hours you will still get paid and if you aren’t finished by the 3-hour mark, we will stop our session, and you will still be paid for your time.

Session:

• I'll start with a few questions to learn a bit more about your past experiences filing taxes.
• [if using DF to file taxes] Since you chose to file your taxes using Direct File, you’ll be using the Direct File website while we observe.
• [if using real tax forms] Do you have the documents the recruiter asked you to gather?
• We're testing the website and not you, which means there's no right or wrong answers we're looking for - we're interested in your general thoughts and feedback.
• Taxes are hard [if covering other sensitive topics, name those as well, e.g. “talking about shared custody can be difficult and bring up a lot of feelings”]. If some topics become difficult to talk about please let me know, I'm here to support you. You can let me know if you need to take a break or stop at any point. If there are any questions you don’t want to answer just let me know and I’ll move on.
• Unfortunately, we're not tax experts and we can't give you tax advice. I also don't have access to your tax information.

Privacy:

• We'll only use what we learn from you to help improve Direct File.
• Everything you say will be anonymous and be kept strictly private. We are taking notes — but those won’t include your name or any other personal identifying information. If there's anything else you want us to remove, you can always let us know, even at the end of the session.
• Your participation today is entirely voluntary. You'll receive compensation as a token of our appreciation for your participation.
• We may ask you to show us your tax documents on occasion to compare it with what's showing up on the screen. When we do that, we'd ask you to hide SSNs, DOB and other information you deem sensitive before you show it to us. Would you be ok with that? If you prefer not to show us this information that's fine.

Notetakers [if they opted in for observers and notetakers]:

• Note that observers are referred to as notetakers throughout the experience for the sake of simplicity
• You mentioned you’d be ok with having a [additional] notetaker in our session today. Are you still ok with that, or would you prefer to have the session without them today? Either way is fine with us.
• [if participant accepts to have notetakers, or an additional notetaker, let them in the Zoom from the waiting room] Ok, I will let them in. I wanted to introduce you to [name notetakers/observers by their full names, role, and what organization they are affiliated with. Ask them to be camera on, wave, then turn camera off to make the participant comfortable]. Most of the conversation today will be between you and me and they will be off camera, but I wanted to make sure they had a chance to say hello.
• [if participant declines to have notetakers, or an additional notetaker, quietly remove them from the waiting room] Ok, no problem.

Do you understand these guidelines?

Are you ok to continue?

Do you have any questions before we start?

Intro to End-to-End Tax Filing Task

Ok, now we’ll dive into Direct File. I’ll start by setting up a scenario for you – let’s imagine it’s the 2025 tax season and you’re starting your 2024 federal taxes. You’ve heard about a new website called Direct File where you can file your federal taxes directly with the IRS online. You’ve decided you’re now going to file your federal taxes using Direct File.

Please take out your personal tax documents and use those as a reference when you fill out your tax information with Direct File. Next, please pull up the directfile.irs.gov on your device to start filing your taxes.

• [if remote session] Once you have successfully logged in, I would like you to share your screen.
• In order to share your screen on Zoom please press the green share screen button at the bottom center of the Zoom page. It will give you several options to share, please just share the Direct File website page. You can also click on the zoom window to make it active and:
  • Enter Alt+S on Windows or Command+Shift+S on Mac to start and stop sharing.
  • Enter Alt+T on Windows or Command+Shift+T on Mac to pause and continue sharing.
• Do everything you would normally do.
• During your tax filing process we may remind you to make sure we can clearly see your screen.

These questions may be asked depending on whether the participant has hit a roadblock or pain point.

• Can you describe what's going on in this page?
• What would you do next?
• What are your questions or concerns?
• Do you understand the way the information is being presented?
• How confident do you feel about this choice?

When looking at help content:

• What are your thoughts on what you're reading?
• Does this answer the questions you had?
• Do you have any additional questions?

Session Wrap-Up

Thanks. I just have a few follow-up questions before we finish.

• Is there anything you saw today that you have questions about?
• From a scale of 1 to 10—1 being terrible and 10 being the best—how would you score the experience of filing your return?
• What makes you give it that score?
• [if >1 or <10] Why not a 1/10?
• [if not 10] What would have made it a 10?

That's all the questions I have for you today.

• (Optional) Is there anything you said you'd want us to delete from our notes?
• Before we end, do you have any questions for me? Or something to share that we didn't talk about?
• The recruitment firm will be in touch about your incentive.
• Thank you so much for your help and for taking the time to share your thoughts and feedback. I will bring your input to our team to help make using the Direct File website a better experience.