# Bank Statement Analysis Pilot: Testing a Tool on Real Files

- Author: TrueBankFlow (https://www.truebankflow.com)
- Published: 2026-04-05
- Updated: 2026-04-05
- Category: Analysis software and reports
- Canonical: https://www.truebankflow.com/blog/bank-statement-analysis-pilot

> How to run a bank statement analysis pilot: picking test files, scoring results against manual work and the pass bar before signing a contract.
A bank statement analysis pilot runs a candidate tool on loan files your team has already worked by hand, then scores its output against those known answers. Choose files that reflect your hard cases, write the pass bar before the first upload, and judge both accuracy and how long a reviewer needs to check each result.

## What is a bank statement analysis pilot?

A bank statement analysis pilot is a short, structured test of a tool on your own files, with expected results written down in advance. It differs from a vendor demo in who picks the files and who decides what correct looks like. In a demo the vendor does both; in a pilot you do.

### What a pilot proves

A pilot answers three questions a demo cannot:

- Does the tool read the statements your borrowers actually send?
- Do its figures match what your underwriters would have produced?
- Does checking its output take less time than doing the work by hand?

The third question is the one teams forget. A tool that gets every number right but hides how it got there can take longer to verify than to redo.

### What a pilot does not prove

A pilot of thirty files does not show how a tool behaves across thousands. It cannot reveal every rare layout or every edge case. What it can show is whether the tool fails loudly or quietly, and that is the trait that matters most at scale. A tool that flags what it cannot read is safe to grow with. One that returns clean-looking numbers on bad input is not.

### Who it suits

Any lender can run one. The file mix changes by business line: [bank statement analysis for credit unions](/blog/bank-statement-analysis-for-credit-unions) and [bank statement analysis for mortgage brokers](/blog/bank-statement-analysis-for-mortgage-brokers) each describe the files those teams see most, and those are the files a pilot should lean on.

## Picking test files for a statement analysis pilot

Pilot test files should be closed or fully reviewed loans whose figures your team trusts, chosen to cover the cases that cost you time. Twenty to forty files is a workable range for most desks: enough to see patterns, small enough to score by hand in a week or two.

### A file mix that tests something

Build the set on purpose rather than taking the last thirty files. An example mix for a desk that handles both mortgage and business files:

| File type | Example count | What it tests |
| --- | --- | --- |
| Single account, clean digital PDFs | 5 | Baseline accuracy |
| Two or more accounts with transfers between them | 6 | Transfer matching |
| File with a one-time large deposit | 4 | Flagging and exclusion |
| File with NSF or overdraft events | 4 | Event counts |
| Scanned or photographed statements | 4 | Reading poor input |
| Small bank or credit union statements | 4 | Layout coverage |
| Statement period crossing a year end | 2 | Date handling |
| Deliberately incomplete file (a page removed) | 1 | Whether failure is visible |
| **Total** | **30** | |

### Files to leave out

Leave out files where your own team disagreed about the answer and never settled it. A pilot needs a known right answer. Leave out files you cannot share with a vendor under your privacy policy, and check that policy before any upload, since a pilot still sends real borrower data.

### MCA and business files

Business lenders and merchant cash advance funders should include files where the merchant's revenue shifted after funding. An advance contract with a [reconciliation clause](/blog/mca-reconciliation-clause) depends on reading later deposits correctly, so a pilot that includes a reconciliation request tests a real use of the tool.

## Answer keys from closed loan files

An answer key is the set of figures your team produced for each pilot file, written down before the tool sees it. For each file, record the qualifying average, the total average, the lookback, the transfers you excluded, the one-time items you excluded, the NSF and overdraft count and the deposits you flagged.

### Build it from the file, not from memory

Pull each figure from the underwriting worksheet or the analysis already in the file. If a figure is not recorded anywhere, rebuild it before the pilot starts and mark it as rebuilt. A key built in a hurry after the tool returns its output is not a key. It is a second opinion shaped by the first.

### Record judgment calls separately

Some answers are judgment, not arithmetic: whether a quarterly payment counts, whether a refund was income. Mark those lines in the key. When the tool disagrees with a judgment call, that is not a tool error. It is a place where the reviewer's override must be quick and visible.

### Keep the key away from the operator

If possible, the person who runs files through the tool should not hold the answer key. That keeps them from nudging the tool toward the known answer during review and lets the pilot measure what an ordinary analyst would see.

## How do you score pilot output against manual work?

Score each file field by field against the answer key, marking every value as a match, a tool miss, a key error or a judgment difference. Match within a tolerance you set in advance, such as a few dollars on a monthly average, so rounding does not count as a miss.

### The scoring sheet

The sheet below is an example with illustrative results for five of the thirty files.

| File | Qualifying avg (key) | Qualifying avg (tool) | Result | Transfers matched | Events (key / tool) | Review minutes |
| --- | --- | --- | --- | --- | --- | --- |
| P-01 | 9,840.00 | 9,840.00 | Match | n/a | 0 / 0 | 6 |
| P-07 | 22,115.00 | 24,615.00 | Tool miss | 2 of 3 | 1 / 1 | 14 |
| P-12 | 7,302.50 | 7,302.50 | Match | 4 of 4 | 3 / 3 | 9 |
| P-19 | 15,600.00 | 15,280.00 | Key error | n/a | 2 / 2 | 11 |
| P-30 | (page removed) | Flagged, no average | Correct flag | n/a | n/a | 3 |

### Reading the four outcomes

- **Match** means the tool agreed with the key within tolerance.
- **Tool miss** means the tool was wrong. In P-07, one 15,000.00 transfer from savings was not paired, so it stayed in qualifying deposits and raised the six-month average by 2,500.00.
- **Key error** means the tool was right and your original work was wrong. In P-19, the earlier analyst had counted a reversed deposit. Record these honestly; they are part of the value of the tool.
- **Judgment difference** means both answers are defensible. Note it and move on.

### Score the review time too

The last column is minutes a reviewer spent checking the tool's figures against the statements. Time it with a clock, not an estimate. If review time is close to the time a manual analysis takes, the tool saves little even when its numbers are right.

## Setting the pilot pass bar before any file runs

The pilot pass bar is the set of thresholds, written before any file runs, that decide whether you sign, extend the pilot or walk away. Writing it first stops the team from adjusting the bar to fit a tool it already likes, or one it already dislikes.

### An example pass bar

These numbers are an example for a desk to adapt, not an industry standard:

1. **Qualifying average accuracy:** at least 27 of 30 files match within tolerance, after removing key errors.
2. **No silent failures:** the incomplete file and any unreadable scan must be flagged. A single clean-looking average on bad input fails the pilot regardless of other scores.
3. **Transfer matching:** at least 90% of known transfer pairs found across the multi-account files.
4. **Event counts:** NSF and overdraft totals match on every file, or every difference is explained by a stated counting rule.
5. **Review time:** median review time per file is at most half your current median manual time.

### Sign, extend or walk away

Meeting every line means sign. Missing one line by a small margin, with a clear cause the vendor can fix, can justify a short extension on fresh files. Failing the silent-failure line means walk away, because it is the one problem you cannot see coming once the tool is live.

### Put it in the contract

When a pilot passes, the pass bar becomes a useful acceptance clause. Ask that the first months of the contract hold the same accuracy standard on live files.

## Pilot staffing and the review clock

A pilot needs one coordinator, two or three analysts who will use the tool day to day and one senior underwriter to rule on disagreements. Two to three weeks is usually long enough to run thirty files and score them, without stretching into a second project.

### Roles

- **Coordinator:** holds the answer key, assigns files and keeps the scoring sheet.
- **Analysts:** run files and do the review exactly as they would on a live loan. Choose at least one who is skeptical of the tool.
- **Senior underwriter:** decides whether each difference is a tool miss, a key error or a judgment call.

### The week-by-week shape

In week one, build the key and run ten files. In week two, run the remaining twenty and score as you go. In week three, review the misses with the vendor, rerun any file the vendor fixes, and compare results against the pass bar. Keep rerun results in a separate column so a fix after the fact is visible.

### Vendor involvement

Let the vendor help with setup and answer questions. Do not let the vendor run files for you or clean up output before you score it. TrueBankFlow, for example, is onboarding lending teams in small groups and replies to each request with a setup time, which is a reasonable point to agree how a pilot will be run.

## Pilot results that pass for the wrong reason

A pilot passes for the wrong reason when the file set, the scoring or the people involved make a weak tool look strong. The usual causes are easy files, a loose tolerance, a key built after the fact, and counting a vendor's manual cleanup as the tool's own work.

### Mistakes to watch for

- **An easy file set.** Thirty single-account digital PDFs from large banks will pass almost any tool. Check the mix against the table above.
- **A tolerance that grows.** A tolerance set at 5 dollars that becomes 50 dollars by week two is a bar that moved.
- **Averages without the components.** Two wrong numbers can cancel. A tool that misses a transfer and also misses a deposit can land on the right average by accident. Score the components, not only the total.
- **Vendor-side corrections.** If the vendor reprocessed a file by hand, the result reflects their staff, not the software you will run.
- **One analyst only.** If only the most enthusiastic analyst used the tool, review time may not hold for the rest of the team.

### Close the pilot with a written result

Finish with a one-page summary: the file mix, the pass bar, the score on each line, the misses and their causes, and the decision. That page is the record of why the lender chose, or did not choose, the tool. For the wider view of what an analysis should produce in the first place, see [bank statement analysis for lenders](/blog/bank-statement-analysis-for-lenders). If you want TrueBankFlow in your pilot, you can [request early access](/get-started).