direct-file/docs/adr/adr-fact-dictionary-tests.md

parent	nav_order	title	status	date
Decisions	100	Fact-dictionary Testing	Proposed	2023-07-21

Fact-dictionary testing

Context and problem statement

The core logic for how we calculate taxes is expressed through the Fact Dictionary, a configuration of the Fact Graph akin to a database schema (including something akin to views, via our derived facts). Currently, however, we have no direct way to test these configurations; we can test the client and then get at values, but the dictionary is used in more than just the client, and tests for it don't naturally belong there.

Desirable solution properties

We'd like the solution to this problem to

1. Create test artifacts that can be verified by non-programmer tax experts
2. Allow for quick feedback cycles for developers editing the fact dictionary
3. Alert developers to regressions or new bugs while editing the fact dictionary
4. Identify calculation differences between the frontend and backend

Considered options

1. Testing via the client directly, by writing vitest tests
2. Testing via calls to the spring API
3. Testing via spreadsheets, using the client test runner as a harness
4. Tests via spreadsheets, using a new scala app that runs tests against the JVM and Node

Decision Outcome

Chosen option: Test via spreadsheets that are parsed by the client, using the client test runner as a harness. The spreadsheets will test common and edge tax scenarios, to identify possible errors in the fact dictionary.

In some detail, we will

• Create a series of workbooks that have a sheet of "given facts" full of paths to writable facts and their values and a sheet of "expected facts" of derived facts and their expected values.
• Create a test harness (using the JS libraries we use for the client) to watch and execute tests against these spreadsheets, effectively setting everything in the given sheet on a factgraph and then asserting each path in the expected sheet equals its value

Consequences:

1. Good: If we get the spreadsheet format to be readable enough, we should be able to have the spreadsheets reviewed by someone at the IRS to check for correctness
2. Good: The client APIs are pretty quick to work with, and the vitest runner is familiar and fun
3. Good: Most of the fact dictionary edits happen while building out client code, so the feedback happens where we want it, during the normal development workflow
4. Good: Assuming we get a format we like, it wouldn't be too outlandish to run the same tests against the JVM by rewriting the test harness --- creating the test cases is the really hard part
5. Less good: We're going to be abusing vitest
6. Less good: We won't identify differences between the backend and frontend
7. Neutral but I don't like it: coming up with examples is both subjective and difficult, since you have to do taxes correctly

Discussion

During the discussion in despair, we decided that looking for differences in the frontend and backend was distinctly secondary and could come later.

Chris made two helpful suggestions:

• We can mine VITA Volunteer Assistors test (pub 6744) for scenarios to test. Sadly, there is no answer key included.
• We will likely want to be able to base scenarios on one another (which I called spreadsheet inheritance to make everyone sad), so that we can test variations

Pros and Cons of other options

Testing via the client directly, by writing vitest tests

Pros

• Really straightforward, since we just write some tests
• Gets us tests

Cons

• Not easily reviewable by non-programmers
• Checking the tests on the JVM requires rewriting all of them

Testing via calls to the spring API

Pros

• Probably good tooling exists --- it's spring
• Checks the test cases against the JVM

Cons

• Requires new Java APIs just for this
• Requires running the backend just to run the tests, which isn't where most fact dictionary changes happen

Tests via spreadsheets, using a new scala app that runs tests against the JVM and Node

Pros

• All the pros of the chosen approach, plus gives us Node and JVM testing

Cons

• Just literally harder --- we have a lot of what we need in the client already, we'd be spinning up another scala app just for this
• Outside of the usual workflow, since devs would need to run a separate scala build tool just for testing fact dictionary changes