Research

Crypto Accounting Bench

An evaluation of frontier models on accounting for crypto and stablecoin native workflows.

Mean Score
77.43%
Gemini 3.7 Flash scores highest
Average rubric score across every task and all three attempts.
Pass@3
56.78%
Three attempts are not enough
Kimi K3 passes the most tasks. Even so, 43.22% of its tasks have no fully correct attempt.
Deciding-account accuracy
56.3%
Choosing the account is the hard part
Amounts agree 97.8% of the time. Selecting the correct account is where models fail.

Overview

Crypto Accounting Bench evaluates whether AI models can reconstruct complete journal entries for real-world crypto transactions.

The benchmark is built from transaction flows that are difficult to reproduce with public data alone: high-frequency on-chain activity across exchanges with inconsistent reporting, stablecoin card programs that settle on-chain, and credit facilities funded in stablecoins. It also covers complex disposition accounting, including realized gains and losses, tax-lot tracking, and entity-specific FIFO cost-basis methods applied at the legal entity and wallet level. For in-scope crypto assets, FASB’s fair-value guidance requires changes in fair value to be recognized in net income, while total realized gain or loss on disposition is the difference between disposal proceeds and cost basis. FASB ASU 2023-08

Each task includes transaction evidence, wallet and legal entity context, and a chart of accounts. Models must return a balanced journal entry with the correct accounts, amounts, currencies, and asset quantities.

The dataset contains 118 tasks from 7 organizations across fintech, asset management, blockchain foundations, and crypto-specialist accounting firms. The public dataset is anonymized: names, identifiers, amounts, and dates are transformed while preserving the underlying accounting relationships. Twelve models made 3 attempts per task, for 4,248 total runs. Reference entries come from the organizations’ recorded ledgers and remain hidden from models.

CAB measures agreement with journal entries recorded by expert CPAs and finance leaders in the field. It does not claim that the reference entry is the only acceptable accounting treatment, or evaluate a full close, tax return, audit, consolidation, or external report.

Full dataset scores

12 models. 118 tasks. Three independent attempts per task.

ModelMean ScoreScore

Gemini 3.7 Flash
Reasoning: high
77.43%
Fable 5
Reasoning: xhigh
77.39%
Kimi K3
Reasoning: max
77.08%
Grok 4.6
Reasoning: high
76.33%
GPT-5.6 Sol
Reasoning: xhigh
75.99%
GPT-5.6 Terra
Reasoning: xhigh
75.93%
Claude Sonnet 5
Reasoning: xhigh
75.66%
Muse Spark 1.2
Reasoning: xhigh
75.62%
Grok 4.5
Reasoning: high
75.32%
Claude Opus 5
Reasoning: xhigh
74.28%
deepseek-v4-flash-0731
Reasoning: max
72.82%
GPT-5.6 Luna
Reasoning: xhigh
71.92%

Average rubric score across all 354 attempts per model. The top 9 models are within 2.11 percentage points.

Gemini 3.7 Flash leads on Mean Score at 77.43%. Kimi K3 has the highest Pass@3 at 56.78%. Even with three attempts, no model completes every task correctly.

Three metrics from the same evaluation:

Mean Score: Average weighted rubric score across all attempts. This is the primary metric.
Best@3: Highest score per task, averaged across tasks. This measures the strongest of three attempts.
Pass@3: Share of tasks with at least one attempt that earns full rubric credit and clears all seven gates.

Score against average attempt cost

Each marker is 1 model, placed by its Mean Score and by what 1 attempt cost on average, judge cost excluded. Dashed lines mark the median on each axis, and the shaded quadrant holds the models that score above the median while costing less than it. The correlation between cost and score is Pearson r = 0.26, n = 12: the highest average cost per attempt does not lead on Mean Score, and the leading model costs 52% as much. Where a pipeline did not price every attempt, its cost is estimated from stored token counts at published rates.

Score against average attempt cost: 12 models
Model	Mean Score	Cost per attempt (USD)
Gemini 3.7 Flash	77.43%	$0.238
Fable 5	77.39%	$0.454
Kimi K3	77.08%	$0.101
Grok 4.6	76.33%	$0.106
GPT-5.6 Sol	75.99%	$0.090
GPT-5.6 Terra	75.93%	$0.087
Claude Sonnet 5	75.66%	$0.185
Muse Spark 1.2	75.62%	$0.082
Grok 4.5	75.32%	$0.035
Claude Opus 5	74.28%	$0.156
deepseek-v4-flash-0731	72.82%	$0.019
GPT-5.6 Luna	71.92%	$0.245

Mean Score vs Pass@3

Each marker is 1 model, placed by its average rubric score and by Pass@3, the share of tasks where at least 1 of its 3 attempts satisfied every rubric criterion and cleared every required gate. Dashed lines mark the median on each axis, and the shaded quadrant contains models above both medians. Across the 12 models, the correlation between Pass@3 and Mean Score is Pearson r = 0.74. The correlation is over models rather than individual attempts.

Mean Score vs Pass@3: 12 models
Model	Mean Score	Pass@3 success (%)
Gemini 3.7 Flash	77.43%	52.54%
Fable 5	77.39%	54.24%
Kimi K3	77.08%	56.78%
Grok 4.6	76.33%	50.00%
GPT-5.6 Sol	75.99%	50.00%
GPT-5.6 Terra	75.93%	50.85%
Claude Sonnet 5	75.66%	47.46%
Muse Spark 1.2	75.62%	47.46%
Grok 4.5	75.32%	48.31%
Claude Opus 5	74.28%	45.76%
deepseek-v4-flash-0731	72.82%	46.61%
GPT-5.6 Luna	71.92%	47.46%

Where models make mistakes

Models usually recover the amount and direction. Selecting the deciding account remains harder: accuracy is 56.3%, compared with 97.8% base-amount agreement.

Component diagnostics

Best-of-three attempt per task, averaged across the 12 models.

Asset quantity accuracy: 100.0%
Accounting direction: 98.1%
Base-amount agreement: 97.8%
Journal-account agreement: 72.6%
Journal-line match F1: 70.9%
Deciding-account accuracy: 56.3%

Mean Score by task family

Average across the 12 models. The six families contain all 118 tasks.

Feen=12: 87.5%
Swapn=16: 84.5%
Transfern=37: 77.5%
Realized gain/lossn=5: 76.5%
Intercompanyn=21: 68.1%
Income / expensen=27: 67.5%

The reference entry requires the account used by that organization. Identifying a transfer or a fee is only part of the task. Income and intercompany entries also depend on the counterparty and legal entities. The realized gain/loss group has only 5 tasks, so its mean is less stable.

How to read these diagnostics

For each model and task, diagnostics use the attempt with the highest rubric score. Deterministic checks break ties. Every component is measured on that same attempt.

Accounting direction checks Debit/Credit agreement without requiring the account to match. Journal-account agreement gives partial credit for correct accounts. Journal-line match F1 requires the account, side, base-currency amount, and currency to match.

Example task

A selected failure case: 36 of 36 runs fail task task_0103, with scores from 15.0% to 40.0%. Explore all three runs from each model. View the public task on Hugging Face.

Select modelSelect run

Reconstruct a customer settlement

The source record shows tokens arriving from a zero address. The organization records the deposit against a customer settlement receivable. The model must recover that entry from the transaction evidence and the organization’s chart of 95 accounts. Every run misses the customer receivable account.

Organization: Juniper Systems
Transaction: DEPOSIT · INFLOW
Asset and chain: USD6 · marnet
Base-currency amount: 2100.00 USD
Asset quantity: 5700
Source wallet: N8C858E:N19A951 Wallet:N37E904 - N394665

View transaction evidence as JSON

{
  "amounts": {
    "assetUnitPrice": "0.3684210526315789473684210526",
    "currency": "USD",
    "feeQuantity": "0",
    "feeValueBase": "0.00",
    "grossQuantity": "5700",
    "grossValueBase": "2100.00",
    "netQuantity": "5700",
    "netValueBase": "2100.00",
    "quantityDecimalPlaces": 18
  },
  "assetType": "USD6",
  "chain": "marnet",
  "counterparty": {
    "isZeroAddress": true,
    "smartContractMetadata": {
      "name": "N627951: 0NA7DDB2…NA2EF76"
    }
  },
  "flowDirection": "INFLOW",
  "fromAddress": "0xbba82d3dfa8b680aeefc8d10060eb58a16b957b4",
  "organizationName": "Juniper Systems",
  "recurrenceProfile": {
    "firstPriorReceipt": null,
    "medianDaysBetweenPriorReceipts": null,
    "mostRecentPriorReceipt": null,
    "priorReceiptsInStream": 0,
    "priorValueBaseMax": null,
    "priorValueBaseMedian": null,
    "priorValueBaseMin": null
  },
  "relatedTransactions": [],
  "source": {
    "chain": "marnet",
    "isStakingAccountWallet": false,
    "legalEntity": {
      "entityName": "N8C858E Holdings, Inc.",
      "entityType": "C-CORP"
    },
    "name": "N8C858E:N19A951 Wallet:N37E904 - N394665",
    "sourceType": "WALLET",
    "walletType": "internal"
  },
  "toAddress": "0x1a25c26532407cba0a03a4dfacab266745555085",
  "transactionDate": "2020-01-27T19:56:00",
  "transactionGroupId": "TXG-00115",
  "transactionType": "DEPOSIT"
}

This task illustrates a failure case. The full-dataset results above measure overall performance. Names, addresses, amounts, and dates are transformed consistently across the public dataset. Accounting relationships are preserved. Answers are mapped to the public task. Scores and diagnostics come from the original evaluation. Private reasoning and tool-result text are not published here.

Paper and dataset

The paper describes the benchmark, grading method, and limitations. The dataset contains the 118 transformed tasks and reference answers. The evaluation toolkit includes the runner and scorer.

Kareem Khattab, Omar Khattab, and Mohamed Ibrahem. Crypto Accounting Bench: Evaluating Frontier and Open-Weight Models on Crypto-Asset Accounting Tasks. Entendre Finance, September 2026.