Completed self-run evaluation · 2026-09-17

995 correct.
1,000 original audio questions.

One frozen, offline DAS reasoning configuration. Every question, failed attempt and additive review remains in the result.

99.5% with validated reviews. This uses the DAS audio scorer. It is not an official Artificial Analysis result or an independently certified leaderboard score.

Download all 1,000 case records

Original strict grades

Unresolved under the original strict scorer

990 correct · 5 incorrect · 5 needing review · 0 missing

Denominator: 1000 questions.

With additive reviews

99.5%

995 correct · 5 incorrect · 0 needing review · 0 missing

Denominator: 1000 questions.

Development-exposed

100 / 100

100.0% with reviews.

Original: 99 correct, 0 incorrect, 1 needing review.

Held out from development

895 / 900

99.4% with reviews.

Original: 891 correct, 5 incorrect, 4 needing review.

Tested configuration

Two ears. One isolated reasoner.

Two independent recognizers transcribed each complete question and were reconciled before the reasoner saw a word (scribe_v1 and large-v3). Claude Opus 4.8 (claude-opus-4-8) ran in a fresh Claude Code subscription session with thinking enabled and high effort. Piper en_US-lessac-low rendered its answer. A separate cached Whisper small.en pass transcribed the saved output audio for grading.

ElevenLabs Scribe v1 prepared on the host by bba_scribe.py, pinned to each WAV hash; the worker makes no network call

Tools, company context, support agents and paid gateway fallback were disabled. The reasoner received only the recognized spoken question; the answer key, category and case ID stayed in the evaluator. Each answer had a 120-second limit.

The original scorer was das-bba-audio-v1. Ambiguous spoken answers required evidence-bound additive review. Reviews preserve the original grades; an execution failure cannot become a successful answer through review.

Exact prompt and release identity

You are solving one spoken reasoning question. Your input is automatic speech transcription: punctuation and homophones may be mistaken. Reason privately and at whatever length you need. First resolve clearly unnatural phrasing using the spoken question's context, while preserving its stated facts, quantities and logical premises; do not invent missing information. Then check your reasoning. Your FINAL VISIBLE REPLY must be exactly "The answer is X." and nothing else, where X is only yes, no, valid, invalid, or the number. Put no explanation, subject, restatement, markdown or filler in that visible reply. Never place reasoning, analysis, headings, bullet points or markdown in the visible reply; that content belongs only in your private reasoning. The visible reply is one sentence of at most ten words. You have no tools or outside context.

Prompt SHA-256: f4305b4e1e8c0e300e509833d6f10f4cd0ee98e11d4c27f21298754b0ce5a162
Freeze SHA-256: b668590f9c7830248b4a5c38c07476c26563495445a71b56cc96c2560d1c0781
Configuration SHA-256: ea690a80735777443cac6bae4a9273fbc63f263cfc9947e2c78c855140fbccf9
Image: sha256:e28b8d4230d92dcd74fb422eed827c1806387a40903ee7b4528a72051fca9765

The download includes source, model-weight, speech-configuration and per-case evidence hashes.

Processing time, separately

1000 cases recorded an audio-ready measurement; 0 did not.

Median 8.98 seconds; range 4.72–37.35 seconds.

Complete-file processing only; excludes admission/startup, carrier, network and playback. Not phone time to first audio.

Usage and cost

Included Team subscription capacity was used. Token usage is not a cash invoice; host and subscription costs are not zero.

Token usage is recorded for 1000 cases in the download. API-equivalent estimates are not subscription cash charges. No carrier was used.

All cases retained

Inspect the record.

All raw audio is retained privately. This release publishes per-case audio hashes, not public audio URLs. No missing recording is replaced with synthetic evidence.

Find a caseCohortReviewed grade

Case / cohort	Answer key	Recorded transcript	Original grade	With reviews	Evidence / review
#101formal_fallacies development	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "afb7c9eb1bc525c97047b71defee49fdbb0fef2460d02c4c1dce9fb6addd103e", "started_sha256": "5d7a582518d16313cc31417e34b2f5de997100e0c9441d774bdda9c762de0944", "original_grade_sha256": "aaec39b52bedeb3f77f112298f2f2998b32fd529abcbd531124772cdd35d45a6", "input_original_sha256": "afb7c9eb1bc525c97047b71defee49fdbb0fef2460d02c4c1dce9fb6addd103e", "input_wav_sha256": "037a6cfb4e1b959aa2cfa8f918b1ae769e3ce97a3036ca3cf1130af80767a808", "input_pcm_sha256": "402c3a1647618049f080dad99869e99605c3b189a8aa6fdcd194ce291c06faaf", "output_wav_sha256": "98e4681690fe9085158f28b9b6660019c480aceca35cf801d40d058462192258", "output_pcm_sha256": "2af3cadef65786d61eddd9a4bba463be37bb940bb2c3084de90aaeca0ab39a2d", "result_sha256": "4aa235f750e9597d1f87e00af18b2b89ba89a058289ccc6c54e0565e8042a17e" }
#330navigate development	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "88152db87c14071b8dc8a17955141c0ab4b1f5649b5d04ee81f126517e3a0550", "started_sha256": "162b41aa405c4f7c20e96a1ab76de5d40b43c7f8f604ba9835a30904b96e70b2", "original_grade_sha256": "7b21bb07c40815e8140927f17d7394f1b633f90d5df08dcaf80b71207047b3e2", "input_original_sha256": "88152db87c14071b8dc8a17955141c0ab4b1f5649b5d04ee81f126517e3a0550", "input_wav_sha256": "a428c3971540387e3ccfea052af2fc6f5df84fae7d321b0dfe46d93795f2cfe6", "input_pcm_sha256": "39ff6a8976b90e129c563b0661166e0c5efddc9d7b163cff6426d53044dfd0b3", "output_wav_sha256": "72b2e5932e63e4b2a43665069f669764b029e603df6af0b78f2390bbe72adb5b", "output_pcm_sha256": "57e2b95030d4f59539f1a696eba6a9372efd3328d5a28594fef1dd7de1e151d9", "result_sha256": "ced788695432fa1c8c8fb5d9c7ab25502e4e43d1376a099258d474d9b42c627a" }
#562object_counting development	9	The answer is 9.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "e0775738c7f2c6424added7bb950857dcf348530049b7418258e4f83d8b7411e", "started_sha256": "277f74c02b132e5503671ed7bfe22b6b8be8f3ce7bc510c2171eb8f6c9b4167e", "original_grade_sha256": "9fce236435c272523b6e11bf76c0cead9b3748f7b75fdbaf6080044b66316cff", "input_original_sha256": "e0775738c7f2c6424added7bb950857dcf348530049b7418258e4f83d8b7411e", "input_wav_sha256": "5ddb25a34eb86e1f1424e1326c5e6463b6f11577e201840e0eee40bc6e346633", "input_pcm_sha256": "e07056aa323d61ce8a81f492b50f81398ad156e54271d2fc274214141a77ede4", "output_wav_sha256": "b9de6ec9027b47c878d379c97513e9c370cca16c83d340ad96b216ce1618fb16", "output_pcm_sha256": "05d688f2739505b9ef2d168a2dcb9eb0593dc0b7386b0c4d4f0e37ee82a41c07", "result_sha256": "684074d2eceed402c62a83d3d4d7573e66a55c045d80d4a1d39548187df0fa68" }
#883web_of_lies development	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "0bb650e25506c2eeb6e7bfb9f249c26c5d9b93aa85cd5dae4ba8e00e61fbe688", "started_sha256": "6f0a18a473ece325981e6e6e4b7141c082d178ff2c3b6b9cdb0b764beb98d816", "original_grade_sha256": "278d083731d09e6386609fcbfa2087670315974e9f600e8c9185321b1e81adc9", "input_original_sha256": "0bb650e25506c2eeb6e7bfb9f249c26c5d9b93aa85cd5dae4ba8e00e61fbe688", "input_wav_sha256": "bdf077c238f978fdae6cb8bd2d14c5aa7f118f5ee909394ffc02983f1765cb8f", "input_pcm_sha256": "c59541ae642549912da384776cc66561c6f30f13713bbeaf2c0ac8821c6d1215", "output_wav_sha256": "9e4c07b0dd2b3c8d4222b0406d213616747cd85390f9066193ef42c4d505130d", "output_pcm_sha256": "aa24e1c6cb49773850f52af81991fcfcead75f68629ece1ca90ffcb74fa87e5f", "result_sha256": "cc73d407fb026426f7c9c906c9a6231eb2e9c37c44e482db7d42d17042be5f03" }
#49formal_fallacies development	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "8e090bea698c061d61c5699a0418147149ea23df1873db05c0807646efd41611", "started_sha256": "a5f376c2bc028679bb8f4e8464d19f4896bf365a85dd111127fd4b0ff1427dbf", "original_grade_sha256": "691e3325ae47b57260d6dd70eaf582857957c2e9254ccbce44eb73a013367d64", "input_original_sha256": "8e090bea698c061d61c5699a0418147149ea23df1873db05c0807646efd41611", "input_wav_sha256": "6acd8614cccc5564974e272d6912b6838a893e39ea5e8a31d57e0a273d1e3af6", "input_pcm_sha256": "8ea6f8bbe2f75dc88fc65e69fe69214cb8d4b72cbec97373c8755924bf3fb3fb", "output_wav_sha256": "611847bdfff2d6d1daa3085bafe17ae6b78f4dfcbfed18accc162fc2e084dd33", "output_pcm_sha256": "cdd42746152584c77597c3f6edc626a53ad4d42b4434b125402583adc901845f", "result_sha256": "9608915cb1e28749141726f2afda2a9058d055777102937062ff1b33d51f37b3" }
#295navigate development	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "ed6b5b3867e533cb6d2f705e8902cc132ef3a7fe6be6858f59319a6785ec0254", "started_sha256": "c72491de0e672e42bd10f32d96842febfdf49514468d1cf8502298999cb0b8e4", "original_grade_sha256": "9e997914f7938118753522f5a00d8d46fef0b0071f18ef1e79634eb07aa5c7b8", "input_original_sha256": "ed6b5b3867e533cb6d2f705e8902cc132ef3a7fe6be6858f59319a6785ec0254", "input_wav_sha256": "375f5acb1d925b0de76a7b5cf858992a99e08b87ddeab004b47eb96bff0de606", "input_pcm_sha256": "bf8fde79ca005fc694f07d8ad89a91a2c9be90e0371ceff29fc7101a9338b49b", "output_wav_sha256": "9a538525f8de013e619689b75bd71905b172cc2e35a35581a3ef1fe3d29283fa", "output_pcm_sha256": "69a5c57dbd9035229ae549f877d4841be0ddc6148c2e2bcfc6c700c4e888df94", "result_sha256": "0b4a3cc1be745d974be484f7eeee35bec3efd8c4737d3dd168a924c5d9448cc0" }
#676object_counting development	7	The answer is 7.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b916c30c05b35278e548a73c286dd0d026888ff0b748b80adf687ce722ad4a72", "started_sha256": "8de4a7505fa7b6ac38c595a61f8598b62712ab5c933e1783d29ed115581d7f3d", "original_grade_sha256": "1f2a5d43e5ff0d31b63a5d7be60334b6e9c7722845e03351684d58afc71e22d0", "input_original_sha256": "b916c30c05b35278e548a73c286dd0d026888ff0b748b80adf687ce722ad4a72", "input_wav_sha256": "5e00ff01c5f5e7b3b5ee20d867efcbafd0e655949757854648b04067f9fc722e", "input_pcm_sha256": "13fb907275bf98753b8e57918bb36bce71f057d629c654fdcbe940c908d230d0", "output_wav_sha256": "fea7cc7f79bf6c16c7487e614f00db9bcfcea52f004e8f93507a0693f09790f7", "output_pcm_sha256": "870d0c533ac3a4bfa2c60f82a8991059c9a7b85471d652def38f7ec15b579508", "result_sha256": "98d6ff6479a5d36ba789ef92ab04c8bdc53288a1fc80c5ad0b2813f021aad034" }
#852web_of_lies development	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "a9799b991db2d23a3180c3f8ee765b8ec1e3203de63da750c2c123978ffe07c4", "started_sha256": "3cec4a6845d407ed6a2f67e9e04c3156418720bb96299d8dc23b21a8ffbda263", "original_grade_sha256": "c5c768f767f560bdd48fa12b391c499e2b722d859c641410fdbc60c6d3601aac", "input_original_sha256": "a9799b991db2d23a3180c3f8ee765b8ec1e3203de63da750c2c123978ffe07c4", "input_wav_sha256": "64e40cbcdde6f2115ebab85f0758add0c717008ca51196662111c5c562111572", "input_pcm_sha256": "b6b2858cd76c71f6ea3810d2c3a94012ec2d70b3dcb33a73f9c161013f8834d0", "output_wav_sha256": "4c9fc00b57a4fa340cf6141a7b9848d7ea5236b85af22788c54273fd26ac65fb", "output_pcm_sha256": "f71ddad25e55fe90c258bcf074469522bdce98e169147a84683fa22d56415d38", "result_sha256": "eff2533704ddad7daa97c98c68de2cbfc2aea00ce8ea2b13b7618801e5460520" }
#245formal_fallacies development	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "a6cb2a37a533e79a2885479da49a5e2ada962def54319d0feec3705e619ea824", "started_sha256": "da1a43c9fa2df1b4528dbde7fd12ad578a89c9c5271e7868c85602a258a68728", "original_grade_sha256": "a76815f810405b21d2f303ee6e0f4a5e13ca1846bb03d02fb68fd80211d20383", "input_original_sha256": "a6cb2a37a533e79a2885479da49a5e2ada962def54319d0feec3705e619ea824", "input_wav_sha256": "c452357516c3ca712963bbe69cc9200454e9d4793a0241e5c60ff6a3de06c118", "input_pcm_sha256": "7b2723eb0b11703db168a109cfabeac03fbdaf5c1a1ce4df3341abb32a18b8a9", "output_wav_sha256": "1ee5e6dd84f460d532aac4f077505742b1d2862a23f1a30c41a1104342c48af7", "output_pcm_sha256": "35aea84d5d73c45aead688ee79cc8e9f7f3ab86d1d56827c2f9c511bf307a636", "result_sha256": "53badee07678e2eb67fa5e96fb336602e012812a8315164a6af17aa8a21e79b6" }
#346navigate development	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "0174464bd1784151ebe7ffacc6b2883512b96079d4d9e204807c2ab22a9ee2ec", "started_sha256": "400d71d3395a3188774bac3b0e49364e99dee5633a1d4894470cf60ff59d346f", "original_grade_sha256": "f07653dc30d6493fbf85c9fe57f9025c6b5185cef39086e4d52b817ab6d6b052", "input_original_sha256": "0174464bd1784151ebe7ffacc6b2883512b96079d4d9e204807c2ab22a9ee2ec", "input_wav_sha256": "aac02df5e3e2ceacce5ed02eea20bcabc5d1b4fe22fc01b13eff3853d5ec3845", "input_pcm_sha256": "103b90f15da8fbdb803509c7ca4aaa220ea973828466176dfe1f71c55c396738", "output_wav_sha256": "0e56445f3a0f5a03503dbdc27d731578f79f934933e7c7ed37a1577a804f8512", "output_pcm_sha256": "6766eb338431122dfe2ea86f28a96e13ad1c9184dc71a8f192014e421c0457dc", "result_sha256": "8a65c0f08725b06bce7fa4829409aa58bab29e0afda11e106d6dfee621a73b8d" }
#741object_counting development	8	The answer is 8.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "7bc88a977798b728fea8e6cb6ca0df0269f8577bb8acab6af2f9fae08f161cc5", "started_sha256": "6bc172f7e89fac6533a5a3d6a7403dd47ce6478a459ae77a8d3739bf7bbfa1d4", "original_grade_sha256": "d2b8b7c2085789ccccca01dcb019cd4755d0f268a0d37e2ee17b82a321f02d79", "input_original_sha256": "7bc88a977798b728fea8e6cb6ca0df0269f8577bb8acab6af2f9fae08f161cc5", "input_wav_sha256": "78b98e7e1af6bfd640b22b13be7b843a8c2398d43426c9081aeaf7539b743649", "input_pcm_sha256": "ae29688a1bfc27aff5d4b475638aaab7dc49c944a531d0eea6983bbb91d21492", "output_wav_sha256": "1e190ec515bc7481450b6d9358fb89b19a723a753e2fc5bdf2f39a0d91909535", "output_pcm_sha256": "af8e5dfd37b59ab525dd835074811778f800541a68725e748ad4d92ddfebeeaa", "result_sha256": "bf1c6daada8fb2e3ea44172d5c60c630e222092d654667dca7302286d0176813" }
#853web_of_lies development	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "5642ee5e05d25dc1cbdd73a7f3fcba97d330e8449c37db64ad09baa65d4bce36", "started_sha256": "aef708f81fa771dc9617ae44478554c716eae1924ebbf79f91520629ec8796b3", "original_grade_sha256": "bc58e60c95d1ff2be82ac7a4937fbf82c62287cf09e6fb51efb835b859c5622e", "input_original_sha256": "5642ee5e05d25dc1cbdd73a7f3fcba97d330e8449c37db64ad09baa65d4bce36", "input_wav_sha256": "396391a305af7f314aa76755326c4105d9283f24c59a9da2ba5e89a4218bbc0c", "input_pcm_sha256": "d47e67189f20b84a9df06bd5e7a5c999da62970441ff81394de4e4dbcf514652", "output_wav_sha256": "70441f9b76432773f8d3cadd9c9ebac96a3d44f20424efa45bf637afa62697c9", "output_pcm_sha256": "f5df81665bc243cecee370d1922d77221c656284cff46971739f099537102b7e", "result_sha256": "59eaa36a6718e8ee66571700ec06173f7d23519925be5f3a10bf3a5b824c568e" }
#29formal_fallacies development	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "e601aaa65f82dd80fe8f2187143100fadf606d3ba1623710098774eb0a17cf4c", "started_sha256": "079fe93c8cc681af232f1bda89f82bbaeb3388faa5d18110898b61e18482deff", "original_grade_sha256": "2cc8e063a8d4ec5909105882948870afbb0dbbc1225c5e793607e038137de2c0", "input_original_sha256": "e601aaa65f82dd80fe8f2187143100fadf606d3ba1623710098774eb0a17cf4c", "input_wav_sha256": "9a189537ba8e4c193fa2e021db149f17befde1363b0981c4295a876a392df9f0", "input_pcm_sha256": "5f8474eeb862e001eabb14d8be308cf139bb5eb557f80635994e334a077fc342", "output_wav_sha256": "e3fa6081427ec3d81facb65627fea0ff2513379385a9cda506b8c9e47789a2ab", "output_pcm_sha256": "27fd2d64c057f03052f59f64677711c40745afd09ceaafc0b354cd9fd7bb6666", "result_sha256": "5ce92e3ffc44abab1cff3d5418982631a537a49c0ddc8d2b00ff9e179a9c7753" }
#291navigate development	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "2b19318c0edb8295115b76da7f70a30993fc97207883df72b46b33606027ec0a", "started_sha256": "2f5b6c357a3d66237c385d00117d7a3bff538693012fc7c1d8b6ed790ced0272", "original_grade_sha256": "9359569f393ae98d8036b223d15481ba9953092ce623ef87679b0d4498c310a2", "input_original_sha256": "2b19318c0edb8295115b76da7f70a30993fc97207883df72b46b33606027ec0a", "input_wav_sha256": "821df52a323a5566adbcc83c7af88160b4b1772da9e15543225ea64b3b43859c", "input_pcm_sha256": "fb6d28163225d1fff2536df1c82c08e3a980847a3f5a300efb93d54cdc635f4d", "output_wav_sha256": "fb57dfcbe5a33e6eb7231b5001a30983ad6baca504e2c3668fa429f637846e25", "output_pcm_sha256": "2b53c8a87eeebb164a5a49c5ff992ab54a39b5dddc3f9a87f549423346343f40", "result_sha256": "ecd447178d8a1f9eef0e518e2722c5a2d0ec481d9fad26062d74bc1856744e34" }
#665object_counting development	16	The answer is 16.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "94aaf6b8d7ef0d9755884609b8125ecb758e470e960d94aa3da029ee0a2198b3", "started_sha256": "8029efc5a211e119ce47eaade5481cccd02e73acd9fbcae68d1fbedb6d81fae4", "original_grade_sha256": "dcc56bcc202afc7f40497a2ef775373e340514afccdf28c2a6e4594ad3abd224", "input_original_sha256": "94aaf6b8d7ef0d9755884609b8125ecb758e470e960d94aa3da029ee0a2198b3", "input_wav_sha256": "0021a459b8af8080002fe37cb730ae4a35f66dc7fb89056716d72da674354013", "input_pcm_sha256": "037df1b66228b8c9c3d9f88e157f85cc3d8c0a26bd6aeebf1ff0b3fc5a40a106", "output_wav_sha256": "170747cbe77aeb3f759d7020bda84d36bae44eb88683913602c243e943b9a9d4", "output_pcm_sha256": "11c4c75c94a571b5a27bbc5deab7d227a11da97183983ee3ea689d423b2375eb", "result_sha256": "1daea8ec856d018aa1b8bf7aa4a984ffc4d7ed8a489cc19f4b3f726d1033f734" }
#985web_of_lies development	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "2027ace2cead309e7e14435c712fc2534e0f1831c820d91d7509a0df076b6464", "started_sha256": "85aa9b446c7ef445d5b7c7f3060e3d8768f5f6571487b4ee3d9f7d733e229b8c", "original_grade_sha256": "23f9215ea1fd2584d78e5586f763cd88977515c2757068723799d0f2079dba75", "input_original_sha256": "2027ace2cead309e7e14435c712fc2534e0f1831c820d91d7509a0df076b6464", "input_wav_sha256": "3ef6610e49240aca363557d7c74ebfab1fd2e0dcb880bdaca55312922bff7104", "input_pcm_sha256": "a4bde88f3019513009b192ab2655479f16190c3410d2e42bd8be1618b7347aa1", "output_wav_sha256": "3c157f28d7d7eee3577ed9afe72d40a2700facdc77fd28bcdcd07113c0f52127", "output_pcm_sha256": "5fcd893db2ff1854a5f149ef23f3ea5267f6db97b52d5c6750923b34b53e0ea9", "result_sha256": "d6d428028684d34ad1ae4da4c199fb8cd1045712c70e1203b8f930da8b1fd3a7" }
#19formal_fallacies development	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "5b4d6e2478ef52618745844ef25bf72c7c5653b2b49a7cdee87aea6f25464469", "started_sha256": "d09310ad93d53c54ebd26ae0a6714d723c980be8b953d2bcedf53a3fdb5deeeb", "original_grade_sha256": "08097dedddec4dbe10f8f092e1dcff56166915c9a5eaaad4c9afe9c1c567628c", "input_original_sha256": "5b4d6e2478ef52618745844ef25bf72c7c5653b2b49a7cdee87aea6f25464469", "input_wav_sha256": "574a0e1527838bf470f109fada7dfe494e9ee19370929b273b3f940d60cc9664", "input_pcm_sha256": "9a8904c9b81a13cbe1e109874436d3d328d0dd1142702d9a20c29e0ab20e89e3", "output_wav_sha256": "bcb29068eaa65508b739fed5fb30f328edb772ccd4d23e9aab2957645c8f7c90", "output_pcm_sha256": "8665e00fac983e1c74e31cd72f1c58d70ef4389e2f70ec04fd8620311c6d5c92", "result_sha256": "b345ba4273c314b89baf9cb3fbc05a67407aeffd2e702c374ff164add4e74278" }
#262navigate development	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "6ee25c67977ff0b4687148cf4d99a84515f40ff59589abc870a8cec870ca3dfe", "started_sha256": "055d8b5788b116647a72ed08a9914012b0baf69176ab8609173344510d226e76", "original_grade_sha256": "74dcd93fe3da8d2ea3290c2556a6c726faaf5a2eb309c4d2ba37073bbac4906c", "input_original_sha256": "6ee25c67977ff0b4687148cf4d99a84515f40ff59589abc870a8cec870ca3dfe", "input_wav_sha256": "ea17cd11c45302391eb0cf1e47be40d28de8500ba2e91f2e7331090c39aaf9dc", "input_pcm_sha256": "8071bd84e1878a7ea44c10a827deab75e579f0d887ed794376f3bc707ca10316", "output_wav_sha256": "d78e39663d295362a81af7e69764fba38e99f872f91bf508d687c40a7069ba11", "output_pcm_sha256": "540416abb9517b1b0fdad0576da2270d60b191b3ca3cdc387f209b050bee0d65", "result_sha256": "22e0a638f21452a457ff28bd08d8d8dd7ea16510bb51978617c3c62cb9fad0bc" }
#646object_counting development	9	The answer is 9.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "fb95c447e1383a866f811c5c439bf5a0e93e5e408786ca62a0504ff8d88ea0f2", "started_sha256": "8a50f7ce393378037c3fe0fdee3695a058765e545b31c524bf65d710fd73d713", "original_grade_sha256": "07a4e5c43bdc9a9c8ca0e09d7cff70475f71674e0b66665ffbfff05dfe7ee033", "input_original_sha256": "fb95c447e1383a866f811c5c439bf5a0e93e5e408786ca62a0504ff8d88ea0f2", "input_wav_sha256": "f38089de290122067f7dc55e6247d14dfae8424c0de8087d3baff9b754e120c0", "input_pcm_sha256": "6301a1a23e3c1db69f5016e081026131aa26461819823b5bc99ea7ed5f2f0355", "output_wav_sha256": "175ceb1bcdc2bb5462802c75440e3aefe7634edf77ee4513143df3efba49baac", "output_pcm_sha256": "f8bc0e2ee2a76e875c8dcdb08bb5b6a7792079942055c69b72f0e44e0c204b94", "result_sha256": "5c9d6535f163de67a09f72054a13e411115f84cb5e8aed32905123c4fdad8846" }
#820web_of_lies development	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "94933ba0a06a3870aaf612b5d63e74d54da02a88b33c1a7f71e0178c6460e86e", "started_sha256": "b42e39baaf3fd51d8fb955d8237775ee50009212a3e84f26b666b05fc1006c26", "original_grade_sha256": "39382c105c4a2292697e1edab1cbd67ddd6bc8f4880a36c9f7a589a5147f686f", "input_original_sha256": "94933ba0a06a3870aaf612b5d63e74d54da02a88b33c1a7f71e0178c6460e86e", "input_wav_sha256": "62de139a1921594cf4e07ca815e2b37cc266be6d91c8ddcaccf782e90c8079ad", "input_pcm_sha256": "72614be390819063eb991e3ed9328090026145cf9e762c9e8ca3322254f4baad", "output_wav_sha256": "8fae4a7c3959cdaaddf5f550778e651a1bd3b55f8bf423cfcaf2d9fcdfe55bd7", "output_pcm_sha256": "82bf3e7997358274bc8c87789c499170a9be672436b48cb830199fdb6e0fd7f9", "result_sha256": "d67ae9b953434b8acb040181531532ba9e37fda1c3ba49c3c15837a64119f755" }
#14formal_fallacies development	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b8941b87378d119bfd72a8c2ef2c7d11bae0ea11f632bb88a4a84279c32d7c3f", "started_sha256": "92bfd65660afe45db35349a4fc4750d467e025fd5acec03bd6bfd396a5de78ca", "original_grade_sha256": "4b6b75928498c74673441801dd2812f64a59fdb91a27211805e1c1ba73920f23", "input_original_sha256": "b8941b87378d119bfd72a8c2ef2c7d11bae0ea11f632bb88a4a84279c32d7c3f", "input_wav_sha256": "8aaa5eba63f1a5567a716184167269e28fb4dcaa8242c95da82b312a4469e059", "input_pcm_sha256": "7d580e9145d6d888db57ead46dd66fb690410210e558dde1c34b7572d4cae6fa", "output_wav_sha256": "84bfff9ca9dd885bbbdd3552eddc24eee1ed4ec104295becf3d8ad533c6ffd4d", "output_pcm_sha256": "4ef4d99a0bfb5e1cf6df4eeffbc30d9ed94c40df6cffe90a12edd129dd024b4b", "result_sha256": "3371ef5bd42dd44ac92cde4a54be093fc706d5c556a679ba0bbfd3dea80e9297" }
#373navigate development	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "88745e16534b2c7e162f9e91648dffe3e11961ebf057b6ce9c81763179c2cb1d", "started_sha256": "b9bbba18b5808c5a2275df5f8cf0e6894d23a570fa2e0d793ddc038760972fc0", "original_grade_sha256": "0e9f68b6056389e0d5d7ef6f30eb68daec69fc74e16b1d756bc35fc7f4792c79", "input_original_sha256": "88745e16534b2c7e162f9e91648dffe3e11961ebf057b6ce9c81763179c2cb1d", "input_wav_sha256": "4a16277bbf1271cd5013674b33ad147537d787c49d1e1a6692a65a4b6a8991a1", "input_pcm_sha256": "e329b928cef0dc6e7b65b6dd4f6191c850375d3081fb6ce9151fc3762fa8e28d", "output_wav_sha256": "dd8336c5bedef50860f772de240054f29f2f4e6556aa35e699567682d470e3f9", "output_pcm_sha256": "54dacb24a99ef13f4a19a471d7d1bd44c4a6b12620eeb7574155be43314d2995", "result_sha256": "e237182efa2daab41804271f2cf13864af0bde298cb9cc45988fcb3951462e75" }
#503object_counting development	14	The answer is 14.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "ffe13b9a06cf12c19e33cdc8b23695741e59d8d26d6bd8b4b586e7b3766e3f9a", "started_sha256": "cb0a0032ab0c3b997100b0316074988b777716607a14623dbc9fc0a74b9e7e91", "original_grade_sha256": "a5bf93780aee9d72fb4eff49f37a0e3d9a51076687ad812b38702d41eec62b9f", "input_original_sha256": "ffe13b9a06cf12c19e33cdc8b23695741e59d8d26d6bd8b4b586e7b3766e3f9a", "input_wav_sha256": "505cf293a8daf32c574c1270bd6615d37c85c84541fc9bcd298a4b694f38f708", "input_pcm_sha256": "77268e2b737caed5becb7441dec5806fc97303f383958b8feb9b2782ff4db9c9", "output_wav_sha256": "e0114dde1be6359d4c545987445a079f53af8a2f11e94a5ff89fa6c3c0832c62", "output_pcm_sha256": "ebc536ddd4e5699290568b378608d08ac471fbf0170896fe6dcfa3546ad6838c", "result_sha256": "f2966082891902e1f31d737b3c9308b397f7ce464ff63fc8328557c81b0af4e0" }
#801web_of_lies development	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "44039f2101fcf3fb0046baee498971cd6356c4de9e6ff0c7c237922ab9e99847", "started_sha256": "d3825f25d4fd9c6557c53ed31f8662c6f7c8b79e215a8cef6904c96c28bc7338", "original_grade_sha256": "1f9238981f74e66aaa7212d439df947a76f5824bbed52240f6e0b38e2ed503ec", "input_original_sha256": "44039f2101fcf3fb0046baee498971cd6356c4de9e6ff0c7c237922ab9e99847", "input_wav_sha256": "5435397127dd5a55e8aa81bdf95dcb5fdc41b23a5925f366a39d2508d04e4b96", "input_pcm_sha256": "aca27c24a9d9ed21c3d3075cc537fc32f75850fb331d2df0f4e27baf2c99c384", "output_wav_sha256": "fda2bd4f1c952687c36cdb25c4604bce9951640838e3efc25688aed9095faac5", "output_pcm_sha256": "3181454f87192edc7d6c38233edf702e7939eb1552f79e9a7986c4eaef4870bb", "result_sha256": "f3fa705048a8f744ccaebc137458f6c040d09824cbd47e18ff6ffd44989fec95" }
#156formal_fallacies development	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "6de8af8d07d1f4714fb52c9277cc7dca1c9e71410e144982bc51fecd20849dd9", "started_sha256": "d3c5a96665c7f82f38507791d00446133529e7a22c48dd6dd5164976a5426aac", "original_grade_sha256": "4d49ae0adf50ef756d3934a3416b7923f862524ee397dc2a54b27b7911acd992", "input_original_sha256": "6de8af8d07d1f4714fb52c9277cc7dca1c9e71410e144982bc51fecd20849dd9", "input_wav_sha256": "11efc05a8518e5841fe18deb64a627e30237f85bb4c4fac315292f3808d74609", "input_pcm_sha256": "b0288fded9393527d5003445baf14fbef5d48367b2158c188efed68c3c65ba08", "output_wav_sha256": "bed30128115286e75d7077b933dfde2148dada87dc03710a33ebed36090670b5", "output_pcm_sha256": "4c9136234077a783b727a09eafb8f65b7ffb1c43b9d76007878245eafbcdf6c3", "result_sha256": "97074a783d4cb7ad13703cfc7c734c8605bf4b27b3fdfa9c96c7675cae4f36dd" }
#399navigate development	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "bc224f7b3c318e29e93a4925f8aaa18368d22988968c37ad673fe6d6397d6ac4", "started_sha256": "461fb438aafdd20db73badf3f11568cb11d39eeeee3b7ba94ba1e932277a0975", "original_grade_sha256": "f3c6d08bc6abc3a716bd8c35d8c0e36baf788d4c6f95907d961720360b660066", "input_original_sha256": "bc224f7b3c318e29e93a4925f8aaa18368d22988968c37ad673fe6d6397d6ac4", "input_wav_sha256": "725a55e46342fcd69f7e1bedbcc0f1ca4920a1363d1497c95bc940aea8ec661a", "input_pcm_sha256": "42ff9682711dd08629b6a98641dd41e0f831606a2edc7b6504a11a7241071339", "output_wav_sha256": "e883c2fa3c69265fe1cdc9577ba25dbcea6e351086ca4a301d9b66a4bdfb23e9", "output_pcm_sha256": "1d9c8e804fa98be3be29f180740e7e339f79051806b9c3546c73bf0a5dc76fc4", "result_sha256": "072245d77fa7f7566db2e82d90bf17b6ac419af3dcc8df29d3dc7ce5906541c3" }
#554object_counting development	11	The answer is 11.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "fa6f9f16f7f942894ae6adfd19467f7b77a0a67e31f572d2c1ccfea29e6a7c0b", "started_sha256": "0c346c966822d0201ef48ed39a017fb05fc4f643db1c3ca4e5daffd16b24ce41", "original_grade_sha256": "6c618f0d6af1b52906072d2320d8af1c37a447604ef6ae54728b6676b9b749bb", "input_original_sha256": "fa6f9f16f7f942894ae6adfd19467f7b77a0a67e31f572d2c1ccfea29e6a7c0b", "input_wav_sha256": "6aade202bf2cf2445e0de47c045e302f1bdcf31536a997b59c4123aeca74e774", "input_pcm_sha256": "f95916a98a4fbd31328f39a7034af6ce3b11e034b0300304869b49293e8ebf5c", "output_wav_sha256": "b06d3f4118a9308035354bdcc27cbc50f3439020086009ff0e0e98e4775f6e2c", "output_pcm_sha256": "a3a988142a5f29d824c4503530587cec80524012655ebf23897d7cdb8d3ad086", "result_sha256": "f415392e45f977539d1a53f395e0d4cd9d8ed9273994652ed4f3c24780b3dde8" }
#958web_of_lies development	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f68faed3d2a76a929dce74aeb3c023a59abccd9ff8a4dbb66a08ae8487414d21", "started_sha256": "5472716c827c83ecea81e5ce207dab2ba60db6970f4e5972305a6ec0e1809a98", "original_grade_sha256": "b1680758d26bb9475b9413dfb3715b592da81dd2610349da9dc18263d43debd8", "input_original_sha256": "f68faed3d2a76a929dce74aeb3c023a59abccd9ff8a4dbb66a08ae8487414d21", "input_wav_sha256": "cda3ec7c006d494808bb1674f114e693f711b7fcf067a0b7bbf0df20586ee9a2", "input_pcm_sha256": "f7f5728f26ff23b5b775e697af4217c4dbe350728994465f16a21274bfa5271d", "output_wav_sha256": "e1780522da6897889c6fc754275068184a32327be1f8a743be44335954c9f5b2", "output_pcm_sha256": "2cffef31e2d7d571dbf6ae58621656fb1ffedfcbf32ce3999074665bf70e56e0", "result_sha256": "631f00e3fd8fc22e2dae375b275a00f2a56ecfccc0a2e320843ab7087c98685d" }
#204formal_fallacies development	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "62b5a27fccea0ef45911ea3e386143cb7b49c321eeebcbb97460466d17da4e80", "started_sha256": "9d4d734f53291f711ee39dd1507de6a6fd0fa29b4995041851d5a7a858fc944e", "original_grade_sha256": "5059e6b9a704eeb2f50d735bea011a71ce0d24fe5a6f03f1c7989f9f26001ce1", "input_original_sha256": "62b5a27fccea0ef45911ea3e386143cb7b49c321eeebcbb97460466d17da4e80", "input_wav_sha256": "0b9ba08a24429f85b6d653f3dead5ffdc86d4bc86c85f0d7c6a7e6a554f24243", "input_pcm_sha256": "ca11ff8d61391e626c856e320a7291bf45d34184e27302779142c27c7b473fbb", "output_wav_sha256": "bdd78b01b4ea626a94f0c8a5a1b36a36be5b64192bb487d117ff81bdde0291c7", "output_pcm_sha256": "710f17d92a2ea0c332c28d8b4add6ac40b1ca128f1a1b1fbbdaab03012b08213", "result_sha256": "c955166f8f75c29abb1bd39b23936fd80b382fd21fcf443a33c3f89095f28d8b" }
#415navigate development	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "adc82a8ca41461048e6debec25c43c21008f45383eb48e000291408773eae431", "started_sha256": "43520e0a74d83e050f33517116178fac1ed267d656ee83b078d61ff5b9840102", "original_grade_sha256": "6af30e2eb5131ce0e588234b36bb93c04df9657368ef7c17244e9c5fd17ec27f", "input_original_sha256": "adc82a8ca41461048e6debec25c43c21008f45383eb48e000291408773eae431", "input_wav_sha256": "e9fd714831276a1a7440640b476eb165860e8eb38230ec1c87d5e722d44a70b7", "input_pcm_sha256": "c14f0cd89ec09b63e336fe12455999c03eff7f6608ef689953a605658c60710d", "output_wav_sha256": "456395aa0a395cccbf79860bb55f17388c9189cc290584cdeb41c73381a8dcab", "output_pcm_sha256": "eb7fd1313e3adb633e2b7d7ae4d7a27a20dc2cd3c7c827cc23f26b1cf5b1f0ae", "result_sha256": "4a16aa7ff8b7102323bd771bd2e6f03b2b2aba24940e327493c8539cd32177d5" }
#726object_counting development	8	The answer is 8.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "43a95f2bb984833f5ace4d976f4d18d4e442a19fe000aaa2727efe1be1ab9e44", "started_sha256": "30db407f08bd3b86b1fe9fc74f4a7bfb2083b65c086c774de601b0b565f8107d", "original_grade_sha256": "f3d2a1992699b39afc23e8063a4baa5cdc9c2848a2af731575b0a0043d34f3ee", "input_original_sha256": "43a95f2bb984833f5ace4d976f4d18d4e442a19fe000aaa2727efe1be1ab9e44", "input_wav_sha256": "d76ed191e8744c2e07dedb5194c0a724e8b0a8dd1a8e8f78f215168e37ef7f75", "input_pcm_sha256": "d5b36029548588e61f20fa15fce5794f79ffb3f4bb925c1cdc9752c6ed90b152", "output_wav_sha256": "62cc6f145d0dd48dc4d6c8606a14016851766b00ee6f9d9c8348c1b9f7928406", "output_pcm_sha256": "6469301299fad144d3d371420e894773494f428403d05262c4cacebe098f8ee3", "result_sha256": "42836f66f49dd3b6deb65df5b24027f4b22a62a1545c3017e45f5df76810da37" }
#931web_of_lies development	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "cf96f665a9cac411b9229390d2f98d95ae67e1b096841385f4b7b97115e0b367", "started_sha256": "dbbd7839b1a2ada301387ecfcfbf3c59a7269ffef08e69aff9044c40348ed789", "original_grade_sha256": "9e990f37e2d1bfb0169bc7c46c1fa29ceb4f7da7a2165dfa86c85562e8676832", "input_original_sha256": "cf96f665a9cac411b9229390d2f98d95ae67e1b096841385f4b7b97115e0b367", "input_wav_sha256": "a549d51268803b06bff482b8a1f77d3f406de6d9e136bf1e5b1f4ccefd95cf2a", "input_pcm_sha256": "9d0180d9e2cddffa75a6a74bc8f7881c6e818d7c693c335a1eea292a73557186", "output_wav_sha256": "2a9b8a5b939d430e12a4bf57b2cdc0b513de9cd361ac1e13289dc0bb2e3be97b", "output_pcm_sha256": "aac0d1240b1d17017238c3fc32e9830ac6484686fe9fddc95060ca487b05c1f5", "result_sha256": "5e676244ae32f4a31ead1984d2ff180d109dc017eb2fc0d0479c1fb16a2d3960" }
#180formal_fallacies development	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "c98c5ee87f9a376b8683fbcb769b37355e183958480d216fac00980a0286a8d1", "started_sha256": "c4617498a6f9af27df40f486b37a82f3cbb16edd7eded1cffb907f714be3a41f", "original_grade_sha256": "3edc8214674429a354b7d0e4483538dd5c79ef05d2f853b602e7c235ea7fd8dd", "input_original_sha256": "c98c5ee87f9a376b8683fbcb769b37355e183958480d216fac00980a0286a8d1", "input_wav_sha256": "e0d7fc6a6c13a5a312b4536c0378816da48f510243305d574519ddb71f116a93", "input_pcm_sha256": "7e9f4d8fde3de8155d09cfb98928578af8a40a3328d7fc43b2b71a56b550ffea", "output_wav_sha256": "329aca1627489e941fb5cce8165bd525f54eb8cc5e26cc1ddc738b18d032a55e", "output_pcm_sha256": "9a446e0ce83adae0a52e3599e0f353cc6d7c2d0e3f407e9fb7cebecb57cacbe8", "result_sha256": "b25d1862336ed7e92782ca5d2cc4c2fca388c7c62d345d9814168a8b8b927589" }
#269navigate development	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "47004102f8fd3048bb5819e1a2a29c1165fe60c991280d7ba11605f48dc848a8", "started_sha256": "49a4a071d09d7a398b41bc9ed332e5b8bb206e27f4f3bc0c29c566fb2a3d8ad7", "original_grade_sha256": "5edfd6ebc52fa2d8b2c2717d6c2057c234bb79251c04639bef202c50471a485c", "input_original_sha256": "47004102f8fd3048bb5819e1a2a29c1165fe60c991280d7ba11605f48dc848a8", "input_wav_sha256": "3d759bcb421b932d4030983fbd2764589d4aff6001bdd148c10fda10aeb5c6ea", "input_pcm_sha256": "57e96752a9feead7e95ea3637bb054672f1b7f7dade9e46fc84dcd8963a6cd22", "output_wav_sha256": "68b4f7004bb6deaea54ed9b91995befe9d5f42649cf789d8e7e933afda5d9df9", "output_pcm_sha256": "0a9a2daefcfaffd64b1e7cbd84c715d5599c76bce7f17a2f1bbc66a405326df4", "result_sha256": "0784f01729d37c27b03d4f0489f0f903f5cc7b204e27c050ebfc3e51ff0c196c" }
#509object_counting development	10	The answer is 10.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "66b9dc39c786ded97df1606457c6956c0452d816dd12d145c363ec91bdbf06a5", "started_sha256": "ba7917cf906ce8225fbb8fd7de9b26a244e6776b40f6255bc6b3fc961c37588c", "original_grade_sha256": "8b83a0af59ec4e70ca3b11c0473592f61d6b486f0603d2e5c87a110a789054b4", "input_original_sha256": "66b9dc39c786ded97df1606457c6956c0452d816dd12d145c363ec91bdbf06a5", "input_wav_sha256": "7824bc5cb0d8bf4a04629878ff962862e30323b80f6f8c9936d98df0635cbf40", "input_pcm_sha256": "011737c0b2293f7dd652ccd99560fd048f1bb1d1ceef537e337b8c30187db9c6", "output_wav_sha256": "b813b5e0d2886c1d47439d0fe2f09786e60ec6b300a3af4895694b572fdf8c4e", "output_pcm_sha256": "e4bd1ac388cf57a38d36bf731c4dd4409bd32f6e6080dcc9792532bf20ecd4fa", "result_sha256": "024706b090cb0adf24fe44fc9e3c9bd2413df7671a29dc449ffce525283da6c4" }
#770web_of_lies development	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b67d6f20e1e78a87c0120b4938638ff551d6c68265f36fdaab5045d5c60983bd", "started_sha256": "c1367f4ffd4f37c2b1536eaaee889727cc3d47729586c6557f008f18b2bb9b73", "original_grade_sha256": "f80a003fcff0056724fe25fce3cae040a6240a8f8e7d8b5faf273626920522a4", "input_original_sha256": "b67d6f20e1e78a87c0120b4938638ff551d6c68265f36fdaab5045d5c60983bd", "input_wav_sha256": "878093d660326ee915fcf391bef788ea44e14f4e8cdf54ab4bde675570b9fbea", "input_pcm_sha256": "e1af1b3ef2629d9aa2ecc6e461b24d50e94e46e46d1a2ec6d6e00f88e90cfaaa", "output_wav_sha256": "c004cabfcbe0866f92a5793dd0410607d10c1edb16ff09c88e6156110029cc95", "output_pcm_sha256": "6fcbc6435d74d42714fad5863df1808e15cfe4f06b7e403a576b0c316f43491a", "result_sha256": "4bd1e2010d35ae6997905ca79e599975bfcf90ca65266c8cca10342cd881fd47" }
#97formal_fallacies development	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b8645e30aea06d1bc0f012010d13000afaeedcc624af105789e27be76f4290e7", "started_sha256": "712acab2b0e0fdbdf716455143d22736ffc6433d6291b44b8f92a744d75ab905", "original_grade_sha256": "4076fd46e5b35d5d393094eb98bc20bca4f1ba589ae52e9814c89f43d67f70fd", "input_original_sha256": "b8645e30aea06d1bc0f012010d13000afaeedcc624af105789e27be76f4290e7", "input_wav_sha256": "4b083f16340942b53092df63a3c3248e98e5548418a676d54d41de54ea853b37", "input_pcm_sha256": "066f5c130bbe6576971a8661cebac75b114b2df667a56fd4874592cb42643964", "output_wav_sha256": "c316b7cc0e8e614569a1ff6e185804ece9b670c1876928de3c549ec6767a895b", "output_pcm_sha256": "baaf8384f7ac65d492c3f5e8559c34825308ef557d18c360fe7a12f188b43dc1", "result_sha256": "8b2f75c940c04fb159591ed23552bfd44caed154561841453c73397aabe5a129" }
#427navigate development	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "3240f9fd2b34bb8914b7a63459df78acb29ef612321f5841afcfeec54796a95e", "started_sha256": "a9a5ac02f33d20004e73e3644571749317d1f8ee937488247144de1f30f0201d", "original_grade_sha256": "5bb10540d77779a1e4ec33d77908577b93dc4a183a381e990f887febbd99b50d", "input_original_sha256": "3240f9fd2b34bb8914b7a63459df78acb29ef612321f5841afcfeec54796a95e", "input_wav_sha256": "e9155563766f64a4cd0bfd3dfd90a94c0717208ac38a94dd3040b56e7c80bca9", "input_pcm_sha256": "abae3afd39f7eeb1c5b1a521f83bf252d1182e18ee66db27ee60c157cfdbdff5", "output_wav_sha256": "c0b389b3888aac76eca3dbdfa88156fd2a0ffafac749cd7516382560e58e688b", "output_pcm_sha256": "bc483171c04d7b2b36623b8ef7ba9a6814e39ff2bf737bf079086b549788d43d", "result_sha256": "a4495d234e789b8d21460bd4fdf36d125433a0fdd08bffa5c70ef4029d4c4271" }
#739object_counting development	12	The answer is 12.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "32c78c13ca5e9893b585d25351c59e7e9cf0bebeffd2f6a200e93f393928b007", "started_sha256": "a3669b577f9fdda615be0681c1ae0a3b980b3cb999348156157166f397f22967", "original_grade_sha256": "55a7e7634871f4e36570f12cf341438edf36074beb8dce35e47378aad12f9099", "input_original_sha256": "32c78c13ca5e9893b585d25351c59e7e9cf0bebeffd2f6a200e93f393928b007", "input_wav_sha256": "de0066309c20099dd00aab15d78f792ecf47df36453baf4ae409e98e0d7a1cd2", "input_pcm_sha256": "2ca83406c1a604208cbddc2ee72c4a050a35397cf72e73dea348f6012d2cb06e", "output_wav_sha256": "b17d8335f1413ab1db79577f0e7da7a1c585d7617f691faf9f62aa0800b46a96", "output_pcm_sha256": "16849495922b57705a3e397ee417ee922b0d970a8c175eef9ceeeb63567a1736", "result_sha256": "a66a7717f5806582b51367df977254cd14ad88d7b998449f91badb6accc767ac" }
#911web_of_lies development	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "5462450e4b62e3e81d66d8b58ddf5c67ef3883dd925795af759158a29ee59fce", "started_sha256": "34c76b7929b92e37118be16685b197fe651b12fae76ab8089206e88d10c8169c", "original_grade_sha256": "7835942996503f1efe4ac690ecf0f252f5fd216e027e908e1b7d8d67e77b6920", "input_original_sha256": "5462450e4b62e3e81d66d8b58ddf5c67ef3883dd925795af759158a29ee59fce", "input_wav_sha256": "0d74ed2142af55c42987327c6856c4421b945034339fd600598e803ee4bc5631", "input_pcm_sha256": "aa1ed09c4250578a4ccc441adcc11420e2f520a3eaffeefd486d49f174fe96cc", "output_wav_sha256": "809d67cd38223580b5550b07464db6dee379b514076caa8a00c0b4be2d5348c1", "output_pcm_sha256": "5d453c6890b5efc84bf8b6b611bb4f1bde74c8a505933aab4e7f634ae1d13d16", "result_sha256": "56b98ad3763ff6b1705897d8e3d4659d1e210c2ff3505659cde04da9539b59ba" }
#155formal_fallacies development	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "d1137d2491141a2b29114149905067721bfe36ec3556eec6c05daf58901a2b13", "started_sha256": "b39558825ace7cb1a181452ad0d278ac9430491283143083a81a6be49ef7f8d3", "original_grade_sha256": "28809b43bcd1a71c946a18e4375a26c7ed29a12ce213fc60bcbb6049294776fb", "input_original_sha256": "d1137d2491141a2b29114149905067721bfe36ec3556eec6c05daf58901a2b13", "input_wav_sha256": "48fb838cc54b9e3f2c26e392e27b442eef6d13f30ebe46193a639cd711bcd018", "input_pcm_sha256": "d78599b84772691f89f1a162d3e51cceebd66c7f81ae1479f7e8b6d6d81a8e35", "output_wav_sha256": "3333c7ba01c910743dae190216934be2587ca42470f67277f820a0409847f169", "output_pcm_sha256": "659cdffbeba1fcf70e11597e28bb7ddde95f1ed7ec310677400e1121324501d3", "result_sha256": "b93c527c006c1b1c4c7cb03ff2069637867a50d57810b6ad2cb32768adbd4e2b" }
#405navigate development	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "32562415ab71d092abccadda12f42d669e83db77b7ab5ed20ae94d44d54c90f0", "started_sha256": "23c56d68b61cd8351894464d633b217658c1ffb915345ac1ed4097a571fdacac", "original_grade_sha256": "a6d6366216a405b5332b2ab1582ed982b5b3282b60132b80368f4c6349de0006", "input_original_sha256": "32562415ab71d092abccadda12f42d669e83db77b7ab5ed20ae94d44d54c90f0", "input_wav_sha256": "e9ef92c440ba8201562d652997804169ed46a077c02561a1c2b757f95ce6959a", "input_pcm_sha256": "a0adfbe173eb24f99ebe2b08fc0b388d3eefa2438b1ce6ffe8644364b73c8326", "output_wav_sha256": "6aa4c00d4f5dfc5683ddcf3decd2374419cb03fd17d313fd186e5772529ba4ff", "output_pcm_sha256": "bc509e8e0c433bc32b63a835bfbe588eaa53f1706d3ef7b0062b40418e2c2baa", "result_sha256": "612207dc7666c8d3afd01f41b541e92790b035fce40c19945c11b5d9766b1e4a" }
#704object_counting development	4	The answer is 4.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "16709fa65f89b990c0ddd65059d5285294104150b27f668837a40dfa2a65984c", "started_sha256": "b83bd92918bec1ad4ef997f13ba20d4d760f1f6d2306cc94271ff06dd577ac5f", "original_grade_sha256": "e51fb64c28e0bd65698346e5dabb9f15c13843561fc287cadcd0e860f5193a6c", "input_original_sha256": "16709fa65f89b990c0ddd65059d5285294104150b27f668837a40dfa2a65984c", "input_wav_sha256": "c249617ac91d2866994dc8516b5aa64e2dbecc7078a61595fe2c37a8aaa27b04", "input_pcm_sha256": "a77d1a231391c81f7d0411f9b7fdda55f3490fba86d623155bb857740cf84c82", "output_wav_sha256": "577a2662515020cf449bde6a85baae1ca606f4f54c1e1af552e0da92ff6bc152", "output_pcm_sha256": "69dbeb033d16a5e272f29a8d4ed352bd005a91b719fa870eb572bd4d0b6c7520", "result_sha256": "3c764a369b8cf421808ed88dcb12fdd9340aa57b852df29d4e87e83d895addcf" }
#885web_of_lies development	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "1fee9fc840d93e7be792b4a03a111e01bad771875b8d6ef4e2951a824d9bd5af", "started_sha256": "4690ea95f2c5abf1595a319d7d0b52453ad244a482889d1b86f05753db22706f", "original_grade_sha256": "e317546849a2f68f518e8257b80c2f39e2185539d18ed9019a7a7e891c53ccd0", "input_original_sha256": "1fee9fc840d93e7be792b4a03a111e01bad771875b8d6ef4e2951a824d9bd5af", "input_wav_sha256": "12a1abc4cb65be0e92cf340f20d2f67381dbcb55c400a48360d85e51fdfc55e9", "input_pcm_sha256": "0df707065f112be58ebe9faebee72596280fc6e88b7d596f2598029ae900e76c", "output_wav_sha256": "7929c8a4fe49c082d2715237608ab6ead422384bc6d04400da57e822580b182e", "output_pcm_sha256": "8704c23e817f32974f4a088d475ec99025d017fd684eb081674d7fef7fbb0199", "result_sha256": "ef02510b533460d272c01cb59c6a8a294cb2ae5e7053574d3ece7a02608a06c3" }
#171formal_fallacies development	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "00e575bbbcc88b155d57ccb492c0508988d111fecdb4dbe5e083f0aeb0f51f3b", "started_sha256": "bf915beef74d25651396e7e5be78df2f989cd466aaab91b8c165522f3e0e917e", "original_grade_sha256": "51fea671d1fbea1b8623d888a2c706ffab2155cf9dfbd594c2ec82952d9f0745", "input_original_sha256": "00e575bbbcc88b155d57ccb492c0508988d111fecdb4dbe5e083f0aeb0f51f3b", "input_wav_sha256": "1c23981fbbe54b3997819ea502192bca7b13a029e6de89df3f5b68b6a014fce3", "input_pcm_sha256": "28dca6d6b7a7911c0e747bfa45d7d1a2a21679fcea59e3f5d7ccd59712b0acf0", "output_wav_sha256": "711f54df9cb61d353da698e12e79a94c13bdb9087b1f4d7ced661af4fd825c19", "output_pcm_sha256": "c3391d8725a937f925f5f896f854d722a44af99642aaf508e05abe99c1ed88f3", "result_sha256": "da5e98ac609f2c4d4a5f7ee588846605fde75d59c206261d46409abbc26fadf2" }
#381navigate development	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "8002f174c80717e5ae7e4198d8177a4275ade1da7d9eea87de63b82488ad552f", "started_sha256": "59a542d37cd1fac1a0489a51000b0de36a95b9c130add8950f567ac723865e4a", "original_grade_sha256": "6d2a2cbe53fe03df9dd8090f47dd55ac5b995ca8a12b731daaead958ec26e65b", "input_original_sha256": "8002f174c80717e5ae7e4198d8177a4275ade1da7d9eea87de63b82488ad552f", "input_wav_sha256": "4bc5d2e576ac4a7cf33d9464c4cff98f43e566daafefcef9980426c37bbae997", "input_pcm_sha256": "ca0033e551e7c8b89e6d3e95862062dfe10558bedd182043d0509724e51665ca", "output_wav_sha256": "3da3f6dc4ce8ccd287871b881ed948caa0f2fd530afc1d38ad240f65eff29e0b", "output_pcm_sha256": "e0e54d83e657937649bf0394679570f86d64b6cab289a6f0223c100d6e33a465", "result_sha256": "dbaed51e9a86d3889de6411bb61400a2453143d109eef20228829809ffdb2b24" }
#639object_counting development	9	The answer is 9.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "9faa9dfa7684d4e7493dd40b1fd8a72edea037e17d493ec8f7ae404ea25a6929", "started_sha256": "b8828ffaf954b2ceee77d7dbf50ae4fc8472ca0f4ad0ff15f1950effc781ab8d", "original_grade_sha256": "29b93de145a2cde4598d46d569a0d873f357790d10f0a554294fad297bafe60d", "input_original_sha256": "9faa9dfa7684d4e7493dd40b1fd8a72edea037e17d493ec8f7ae404ea25a6929", "input_wav_sha256": "56a99ab5f9f621a7e3e3e5a291d20e87f9a439a81ea7b576765e9edf08e00a0d", "input_pcm_sha256": "26bf9bdfde21c7f3ab875bb4ae776001489f85ceea9138425edae78132c8e7f9", "output_wav_sha256": "7147b9063a8b9a9fb1d7e914ba2491d40b6bed5c2d34941bda59b5f98b712a4f", "output_pcm_sha256": "2bcbdbb93be2ebe2daa0fc52b16ce0460bf42806b01e6aedcab889119e1f57c6", "result_sha256": "db0e9331f4301a1d8a8bda874a687872323698c0b501e4c3cd9cdbdc3b343451" }
#935web_of_lies development	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "c7fb754158c4467c79e776b3c8748cda0154dde9144d9039b9f12af4ef42c124", "started_sha256": "b47d922b9fe1778024497962f5d17de1405728edbe873aeaa36a91f9544e79ed", "original_grade_sha256": "d6207c041f21ad567942851b8636adee2d2ebe1a2e969162979294183c940f6a", "input_original_sha256": "c7fb754158c4467c79e776b3c8748cda0154dde9144d9039b9f12af4ef42c124", "input_wav_sha256": "1ffa0bcc2e15df1c92a9fe7b9f67951edf232a7f69e3432f9ac3174a8cfbc1e7", "input_pcm_sha256": "0ea9dc96b88a3c67fb7f98e02d46d351717566b9af2b5639a356dd4874b0c816", "output_wav_sha256": "5968e0229e1daa17b0e1b42b7d45935054180589737e59e439bc6b1039203906", "output_pcm_sha256": "d40ec038ae16c8b1724b80c90fc271599ae031de42deb30c5cd85049d1bafa51", "result_sha256": "a12085064c546c459c4c01f8a182535611da4380f246035d5376ec18c6647b65" }
#224formal_fallacies development	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "11b8e0e6f072db1f7fd33a9c6684ac562b979c6b19e06318ec35e6e236b453bc", "started_sha256": "b6c459e8a389d71997d6abd2d190ffe9efea018ba8374e95aecc23e1dbadc2d7", "original_grade_sha256": "b3a17886e3d116387348bb63934a063797542dc682dafe436c0ffff51612f60c", "input_original_sha256": "11b8e0e6f072db1f7fd33a9c6684ac562b979c6b19e06318ec35e6e236b453bc", "input_wav_sha256": "a95b10064ac798f8660013d2395a0dd9c4aa4dc77cec2bd6179973b3eb0f3cd0", "input_pcm_sha256": "8ce65343b442fc98b53e24613043036f0f04608e71a9262763469ee033babe19", "output_wav_sha256": "cef66078cdcfb68ccd1eb15b3c77cb3a10d2c7dd8a83fd16a77b382c693fd7e9", "output_pcm_sha256": "1d6fd0c877f993f5bd744b6bd1f924f46f0178384a5b3bcc8195accf3281caec", "result_sha256": "40600eb981a94ef99c224ae538792d8a8ad0b0406efdfd11054a9481fc3ca352" }
#334navigate development	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "c74fa7425ab81f9d0b24b2824a0660ec5a00b31079133b420da6832d56390618", "started_sha256": "bc1b2c5c30cd90766e45b9a94e7032b6f38e8d6379e47011918f6bffd365e355", "original_grade_sha256": "105705e0e5df1b5c28956ce7a573dfdb1d9f4a167d6d721bab9acd1fbfec06f6", "input_original_sha256": "c74fa7425ab81f9d0b24b2824a0660ec5a00b31079133b420da6832d56390618", "input_wav_sha256": "cc813eff0d36f4cd2a6e48745eda1c0eab7adac9a940459b1e385cab0cb8ba66", "input_pcm_sha256": "21a9914208d23c8f7b051948ed5c2817e3344155f82b1e14d121ab73ba2ee59b", "output_wav_sha256": "c905be0dfab8eccf2c33bd7ff46238945e09b9fc33095761d3de5757cfcdb277", "output_pcm_sha256": "8a0fa1ac41f445339495b8c7dae80ded68a1ff576aa7d0588fb3da1efa5bb8f2", "result_sha256": "c94143ac46a7b8911e17a3438423e712429a93e809fead09ed992a96a29f70b1" }
#581object_counting development	7	The answer is 7.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "c267149a8ba810f8f431cc7d9a605bbd9a733841ad79de6c17f634240c7ef205", "started_sha256": "577a016d9931923a6cdb9f7b63f698dce9217d16c6e8c5e1613d92c747054a6f", "original_grade_sha256": "f91a59decaffb6b1f1452444d7efaeaade9575497ab2af3a52fe55883c20d5a1", "input_original_sha256": "c267149a8ba810f8f431cc7d9a605bbd9a733841ad79de6c17f634240c7ef205", "input_wav_sha256": "f5a5bb8c5c4922296f1ac51686d1c093238e6014f7d8345ddd6c93ba802e5422", "input_pcm_sha256": "6398eb7db967442be440a92978660aac5cbc97f1d877b370c0514cc8cf55d249", "output_wav_sha256": "033840492d71458fffdaa2ea44a3dbe9b5e8d7fc94051857ae324937c0b7c336", "output_pcm_sha256": "141f54b65580843f65880ca0084b21b05ae3d9b56a103389b04cee26a9863129", "result_sha256": "fdf2c247ff8467e1d42715d8b29b2ee784037f28e2ce1301518881b64ea17be4" }
#966web_of_lies development	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "086c89cc3794fe2729eb6a9795116b3aaa7226adb9b7f388d916f5cbe5e365cc", "started_sha256": "9b504cbd8b957c6aaf1dc814d976aef292796f60d3040c986f56f1026a5f62d0", "original_grade_sha256": "4c183d267af4449b01c6ce783a5ae1cfec8abc2e5bc9e157f1c464da9f707cd1", "input_original_sha256": "086c89cc3794fe2729eb6a9795116b3aaa7226adb9b7f388d916f5cbe5e365cc", "input_wav_sha256": "dbfa04e29b7cf0b513e700c0851876d285bdcb4dec633521355efe833ce6f888", "input_pcm_sha256": "9c2da2392dddd5e8fd26c82ba4f726c98b6de186412445194fe7f2baeaee0d94", "output_wav_sha256": "0e5084a27ba241c9e5167f0e48c9f1883055b2fe5f67aeed92361380834c6475", "output_pcm_sha256": "4ab37639d8218515acc774c5cdc4333aaaff25eb12f74cdc7b3b20c27cb313af", "result_sha256": "7ff65210b8faadc6334bdb7357fce1997387bad470f662bb76835fd6994e6f8a" }
#44formal_fallacies development	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "57d1ca1fd727522f386a5f7ddb34c55b62d483c5ca480f65895252fea02dfe4b", "started_sha256": "f1438c96e75d7a47e0db63e4b937c61d065470c1862e80c887056d2069dc9238", "original_grade_sha256": "53a607967706acccdefe94099cdfc29c904dd1a87cac145875d973ef06e8cb5a", "input_original_sha256": "57d1ca1fd727522f386a5f7ddb34c55b62d483c5ca480f65895252fea02dfe4b", "input_wav_sha256": "bcfbcc0641d4b9651b5fee23aa6ed7936a0c25a032fc605970bec237f92c769e", "input_pcm_sha256": "c72bfb8ff3918266da21736e03cd991af422fe786f42aa280afa2dc712fa50fe", "output_wav_sha256": "c49c5ecbc12f930546bb29e92de7e97eeec98bf0b22b671f62ababd4668e888d", "output_pcm_sha256": "bfeb84dd236776893325cb9bce99c3d10be6d74edd6a0b9ab118a5e6f1295efc", "result_sha256": "1908982e114c8c96419f6017e1d12c0bbb713f810812224c64e5c566c1408e70" }
#478navigate development	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "bf23fb545184977d93e2382b575c6eae795e831142045b8eef693baec8f3c3b0", "started_sha256": "cd1301ea6235db3b15c330da60e1f6c5d8270827b06a6bbef22b6fe3d0cb96a1", "original_grade_sha256": "d80bf698750bc91248e1fe9b162f987413a7ce8f8988368025b2d881301ac13a", "input_original_sha256": "bf23fb545184977d93e2382b575c6eae795e831142045b8eef693baec8f3c3b0", "input_wav_sha256": "d44b0bcaa7ef376ef16f734b7424a7ad5690555bd2a98bf4e1cb6c3cc0c7512a", "input_pcm_sha256": "a2ed807bbec1ac1d0a1ee2d23bc3b21b852e0a0bdd91d9faf0d7913c7bc21f48", "output_wav_sha256": "9a929dc53d9e0e1089b2d9837d48a84092df947dd52ba3929622269f997ac45f", "output_pcm_sha256": "a5bfa8c19b1bac7d46cd1c74887428d537aebd37771d12ead69b628346de4c9f", "result_sha256": "215407c36c19e831ea4dcc71802e553807bcb3a0ad8f2dfd4f4abca9dcf77b56" }
#647object_counting development	2	The answer is 2.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "656fc15ac8bf18f1d510f74c0a80cc8b140b0770fc92fee121995b7da9860c94", "started_sha256": "0295a8bbf9183efe08a7f7592c96562de55677a9b1cbc8f443c386a92206a1ae", "original_grade_sha256": "6f03a93c7383bb95c5d96600e669d05aeb673df059db9f1365138bd92bc10184", "input_original_sha256": "656fc15ac8bf18f1d510f74c0a80cc8b140b0770fc92fee121995b7da9860c94", "input_wav_sha256": "611889abfdfa234ea0a24b1fc8061fe32f37c9b0ad1a0a1bb9615c953d16d0c0", "input_pcm_sha256": "6fca85d7233a35bcd3fea4b3313e904a934bc283a069282dcdabc99eda84437d", "output_wav_sha256": "c76e3c3ff586c6438530e1d7db8485b14e75b9245badb43319a70b0d96b02fff", "output_pcm_sha256": "6dea014ecc8608296ddeadcd0ee410fb371e2ee5878069b40a1c9fb7cab10d63", "result_sha256": "3badee575729bb97264d3721668557815cbc8cc7c288eda9fbd2e20be9cd31bb" }
#997web_of_lies development	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "57db13a502520cbbddbf55b004b84cf26d95dd7efa39351566837c87def22c1e", "started_sha256": "d2a9550e3ee6256ada1f2976fff1e2c8675a56294f9cf75af540c44efe21a902", "original_grade_sha256": "5a1542459ecb858b48daa2956b75abd150f5ea9d5851c0e96088d6d22107269d", "input_original_sha256": "57db13a502520cbbddbf55b004b84cf26d95dd7efa39351566837c87def22c1e", "input_wav_sha256": "e24ad22de678154ce4b4a094bad5c059fe9f55f4f4e87dd4df94278d0ca50aff", "input_pcm_sha256": "ec9fd0925df2079f9a516eec8d8eb6c9461786c79a4be99f0114cb8c6f316308", "output_wav_sha256": "0910f80219e9b67371cbd68aff3cfc38f5509e16a7e72272b153cd54a9b2a49a", "output_pcm_sha256": "2d7d1094e76536d99cd2ba29047350457950d61c5d5156cc40278935a7ae99d5", "result_sha256": "562f9ca5f4386910d32a2685c8f1f0aac476febe70cedbf7b05383f3c74aa478" }
#54formal_fallacies development	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "7f53d791c629d765890e62b7b3b66ecaa033ee1f1c5dd8b355f96dfbe8a9caff", "started_sha256": "c085fa57adbe7c8867db9ab3fcd513a671b73cc9afb7be17372c056d7e8c0ccc", "original_grade_sha256": "4488afe15eeff7e14afebd86839a5ae6ee46b4f7330989d43b7f2a9a4f67f825", "input_original_sha256": "7f53d791c629d765890e62b7b3b66ecaa033ee1f1c5dd8b355f96dfbe8a9caff", "input_wav_sha256": "4cc4d3dbe4eb20beb56e9ed4102f1b5a8b59a6abcb87da3611a3c2616244c3b3", "input_pcm_sha256": "4b67370390c0aaca9225b4d5ec8278999972a4a42b9f891b250c9e16c4b2d839", "output_wav_sha256": "e231aa94bd43e31244cb0f405bac98bd0f4c49dbc56c99b3065f8ff7aa209093", "output_pcm_sha256": "b9009da727b2daa565b5ca5c70fcf98c39e8b7c41a0ccf7f1196679f617c1971", "result_sha256": "054f68eac038c2b57a2ec421ebde8a17b1c71bc7e597366a65f923f06917f0e4" }
#361navigate development	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "acbff9195870405940f01ebeb766d622d8cb51f78486c1a89e609acaf009858f", "started_sha256": "b5306ccfce0ea8b9666df2c99ed9cbfe8bf0ea1b63f62e30aadd04a428ef6e2a", "original_grade_sha256": "267d0886952fab128498f9d568664a5e0d018cb0931772874596b9b765c362db", "input_original_sha256": "acbff9195870405940f01ebeb766d622d8cb51f78486c1a89e609acaf009858f", "input_wav_sha256": "0d4d834d14e8164c9a04ae6be256962970bfdd31155ebd1c5e1c27ead4c27420", "input_pcm_sha256": "7b3afda0751db4005c7f257de5201f67591435d5ce9b582f1e93f9936fc5acb5", "output_wav_sha256": "6942c4d6cdf1fe363489ca21ad1698279c8192d9bf87fcee39df11d584a02003", "output_pcm_sha256": "5e8d30d504c518b8d9e0cd1af73109eb8b7f5f45ef2697e5ae464050930a9514", "result_sha256": "423015c3d83a9b20f551e9c7df06ed53502e90b9daea08e7efe755409b420750" }
#528object_counting development	4	The answer is 4.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "019e197f22385623487272ca35d9399020875b8eda647c9a68c0f37ba85be1e2", "started_sha256": "2fff94793c238ae9c5e2c97604bc35f18c1c847f5f6b6ba39c3358081ee161c9", "original_grade_sha256": "656e4933ba01c19b0561764b79aa8219f710d8e147ea662f5bdf02a2f575c37e", "input_original_sha256": "019e197f22385623487272ca35d9399020875b8eda647c9a68c0f37ba85be1e2", "input_wav_sha256": "561559255a0d2f1638e15f3cc42b7d3b657fb3021d2608807a3ad68d21b7f062", "input_pcm_sha256": "1d0225f4f1265c345436238c5ed63f128789f18eb132b1562af90f6674faa105", "output_wav_sha256": "0dfba0cb16f980c58d874c895e6d67a4bc007f62347eee1b75bf4cf5745dede8", "output_pcm_sha256": "0df50d7f12833c8cf816687f388c79f31cbb2cac6cfa0260b8e850082dbf53c2", "result_sha256": "a5d7f8fc3bc50c176d8ede17c502711645e0e9ffd43bd245c79df2d71c6804a3" }
#977web_of_lies development	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "cb67851924af9c6781b772011123f40a6e5b8f0708a3935ec37f1bc8e3fddf94", "started_sha256": "cd38e01c418a7bffc60823529a7240cdf94fe46837d8e5eb6bd850441c840c62", "original_grade_sha256": "773975410c86f2b778cf2988e3a4e4a46e4e25c9fb4cf64d52fa90c37283e380", "input_original_sha256": "cb67851924af9c6781b772011123f40a6e5b8f0708a3935ec37f1bc8e3fddf94", "input_wav_sha256": "067e7458fb5da20d1e7a6704fb57932b13929d9da88308d57b2f428449decc8a", "input_pcm_sha256": "8c1bd233bf7c5f091c1f42575bef61c042d191765e8fdea8f1c89c04d17e236c", "output_wav_sha256": "1d614f7152ca14663392d3307ca0d605687185f427ca8b541d7b336995159ec2", "output_pcm_sha256": "4114dae5a442f6e65429a0d0e02fb137cf6e9b3be96ee567f8403d46079389f7", "result_sha256": "115dfaf7c19aa1e4f09d7b038c4572cca0d77b9535a8925d829e953d66ac53a5" }
#208formal_fallacies development	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f285bfc15d2dcea14713529d903921a6349803c7ac805181b42f666ce0641d32", "started_sha256": "a5bf7f01d30f677d60935ce8a46b983b5d250b20045a1f8f5f267095ade97e66", "original_grade_sha256": "c766c267b9cf75df78584899d0a39c5c2dbf0db2aad9ed8b870474f01792ea22", "input_original_sha256": "f285bfc15d2dcea14713529d903921a6349803c7ac805181b42f666ce0641d32", "input_wav_sha256": "b53db9d0d07f76f6199d5c622d11b0a4d5c07e62204991749c485e53f7d26212", "input_pcm_sha256": "722b527a34903c0d1b6a738595cfd7fe372e13e0ed2c642bd8884052911b4e19", "output_wav_sha256": "73b3e974842babdad1785da27ed38bf12383352bd49c9508c4d1f9359a7b1e2f", "output_pcm_sha256": "11a70bca1cae88ea1d345116f9def03c485c6b09bb0ec8b1f3f89810e4d85361", "result_sha256": "538ec5c05f7f0ae47acb88e6a371a7b87bf12eb0ec05ed96013953559d1fdfd4" }
#271navigate development	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "000ef2513e0136e5adce5bb468a8c0f9dd333897a5d144afe65719ff2a23693e", "started_sha256": "a46f45d2fbb6267b9d56d3422dcb2ea215d217539ed6c79eab84d5411c53b610", "original_grade_sha256": "aeaa56a9b72144029a18502aee03ffe5d3a23eea522ef85f46c756876c244922", "input_original_sha256": "000ef2513e0136e5adce5bb468a8c0f9dd333897a5d144afe65719ff2a23693e", "input_wav_sha256": "7bc84d516eabfaf9a1e4d0df6c1ad127849e74426011367aa46c5298a11d3463", "input_pcm_sha256": "c59254f771e70346ff0bb93f865fa65b71425f0da495fc33f9fdc76e22e0509a", "output_wav_sha256": "6cef418262fe117e860186486650bf390cecdf1d6c3bdf111eb78a6971f04a3c", "output_pcm_sha256": "6a0f05c8ece86329ccea6ab2ba7183ebe208cff2f0b96177e2fba5d42673bab0", "result_sha256": "6f29410e953a586849d228c17767274338b4901d4593fc3be86a0a3580b5ce84" }
#505object_counting development	3	The answer is 3.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "c8711794a0e4b07f2f3592e90979db1ed03b7741bb75d015f1e6e174f0276b92", "started_sha256": "f60fcf69d5806e8c7d6e458b47e4d6b93e1cba07fc8e87b6929f62482bd1b514", "original_grade_sha256": "cd22c1ef71296357af44e14f30a7446a2eb123d9fa4e928540c2156203baf806", "input_original_sha256": "c8711794a0e4b07f2f3592e90979db1ed03b7741bb75d015f1e6e174f0276b92", "input_wav_sha256": "f0ce8200bf097b70ab3c4fc88849e65841926aaf630ae0caa149e7232bed6dc6", "input_pcm_sha256": "9da3dabaed6a328af186bb4211f53ea442dea4fa6e31c38deeb0df4b301b2f23", "output_wav_sha256": "a05c4ecf38089a9cf1266d5558e56649a861706df30c781699455ccf53463204", "output_pcm_sha256": "6e8894d687b1978c301aaccee29f9ceadc29364570d0458d9d3b4c561b361813", "result_sha256": "fc57e527231fc002185737546aa5a1b7299f943fce183042fb50af3e9afdf90b" }
#992web_of_lies development	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "e26b1c029fbe788660b5c00679bb73b1d6e14330c14e264a738fc97b882234f7", "started_sha256": "32cfe7a71bf76f5eb0cd451f72ee461f27f71d4b0c6ed9fd0e2db1b93461ca63", "original_grade_sha256": "ca12997a7577629faeadd6f13ccc641f6d5497097730b7e8c948ade17d07cbc4", "input_original_sha256": "e26b1c029fbe788660b5c00679bb73b1d6e14330c14e264a738fc97b882234f7", "input_wav_sha256": "d8fd4cf09eb320bfaa0c95eb2574a784469936d716ec7fc2ac6648406344f4dc", "input_pcm_sha256": "884efc92ddde03a1acdbb443eaf947cd8a5b086f25bd2b503c51a01754ddde4f", "output_wav_sha256": "d086fe9cef669cd11dd044baab2d5d816f7844ca06ae0a0d7d513cf33899ac09", "output_pcm_sha256": "1ef6f9e102759404602410145b3bf113590370729dbe4a8d91d6fd6ef8933bad", "result_sha256": "54b1b3de64018ad96cf2fb1dbac05ac9f918ef173f9084fcb30e0917a07d86b7" }
#52formal_fallacies development	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "3e79b1e6190ba638f4af03ae0efabd327c9cab6994eee9a9ac57c87789455d98", "started_sha256": "7adc562a675e7c1bbad57b5879b091ed2a68aa6a007c0ae55f41c100fef85b21", "original_grade_sha256": "05e1c063961b130fc6ee334d84d5c194a07c87fec8c17d4e6e1637dcf758b99a", "input_original_sha256": "3e79b1e6190ba638f4af03ae0efabd327c9cab6994eee9a9ac57c87789455d98", "input_wav_sha256": "ebecf6c4753c0ad922553814666ea1c5e0710e8eacb0c39d7b9ea2cc9371339c", "input_pcm_sha256": "7d6ea17eb68a995f9cad4d6b27943c2f9737352923d30375ff6ff5bf0745d886", "output_wav_sha256": "c4dfa071b2ad2fef985a11ed2f7c0df592782ef19bdb978ba6eb6a275dcf27c8", "output_pcm_sha256": "0c1a98744cbc601401433b106d7be2d47daf0fc91c3f9c6e4c44810fc0d7e17c", "result_sha256": "5298543f6a00573c460f5c0ff1ad4b6d996f17a801f61ff48432599289f7d498" }
#358navigate development	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f625ae90e007b238206604b0b0c2f2c1ef4dacf6f38b4ef669975ea57a847794", "started_sha256": "4d734899842f529bb44fb875bebab9fc939a43f01d7870af1c1dfe36e203b9e0", "original_grade_sha256": "cdfe8e2500768376b13d3af0201b3a1d2b5e79e29bd47cf7fbc7a4e05da334ef", "input_original_sha256": "f625ae90e007b238206604b0b0c2f2c1ef4dacf6f38b4ef669975ea57a847794", "input_wav_sha256": "b3ba935287d66e17635626d93602a149ba0f25438c94ab05df73c853e460fc70", "input_pcm_sha256": "1a89ffec4c20c5bc1f760fab5798337fdd13e71a7f5a286eae4d17b0095aecbd", "output_wav_sha256": "1cebc6a8326b94c0c295fbf615f16150a492425ff9a465745c49f52e80162176", "output_pcm_sha256": "1dca6198d064b58a498a977528774b4fae8de8fbae3af7a36efcf84fe870296b", "result_sha256": "72efa36379e54e8a7114703bc17ccfbfb5a54e365dcee45ed886b76501602264" }
#531object_counting development	9	The answer is 9.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "1ae1c0d8e05b79e1b64a59a6df423b39dac73ab1a1dbe6d2c33531c2cd31f39c", "started_sha256": "806c2cb23b0569909e871663dfaee6b3760d1e881555e4a9ebd86ddfa2282957", "original_grade_sha256": "b36416bcccb4a60365a5eb2a6a1e3f6f076b3ba59ace20deb5627da5d60e42b5", "input_original_sha256": "1ae1c0d8e05b79e1b64a59a6df423b39dac73ab1a1dbe6d2c33531c2cd31f39c", "input_wav_sha256": "effc43f615958a84920755693a81c2c1afabfd37a74e4770587b41b1f4477f3b", "input_pcm_sha256": "7f9cbc63ff97902b17dd7e7cc3f3d2023cfa54b3418bd54ec5b6f85907b2da4f", "output_wav_sha256": "ff7d91b499b9ae3422e91c30624af5c73cd6dfb61d19700011fe69fcd7b9e0c0", "output_pcm_sha256": "1d498e3eccd6c903d7d154d83db9f429d692ebb080f4c9c79dc0ee1e0d78c749", "result_sha256": "e4f952b59236e12f4dc5edefabd0b8ad2fd1ac1d30ea71ba9635b0baa3b5eb74" }
#941web_of_lies development	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "3e78f384c33e8ce8c66cade26f4647abae7cd8f037dfcd57c31b8ba23467e90f", "started_sha256": "5564b889619cdf298d8c3abc9223fdba44974de83086be71e1f8b261597c9037", "original_grade_sha256": "8e427bcebbd6f4052bc486e79e948c01b8e17831fbcf15ce212bd4056894bcdf", "input_original_sha256": "3e78f384c33e8ce8c66cade26f4647abae7cd8f037dfcd57c31b8ba23467e90f", "input_wav_sha256": "a1942c17fdf8b4a11fda28047d7989a9e7d76d09ea52cb7241f52551db1dff6b", "input_pcm_sha256": "5f2b6e48faba800ee366973fbb93129620656cc03a697e762ea708fe4f210ef2", "output_wav_sha256": "69ef0fd35b556ddf7908e290f5fbdec273c5e304368ed2caffdc0017678e0fdd", "output_pcm_sha256": "863bc4989dc61bc43de11db64c12410714cc16ee1e56fdd6ca83785d8af82cee", "result_sha256": "eb791ce368a4cb53fa34b827e5f79fcff9fc85300005a33f1956e324539675ce" }
#48formal_fallacies development	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f531f3b01e8ebf14356f84037c4d4690e7c59f6e21edf8c6240b9d75382f8e7a", "started_sha256": "6fa6c7278fc40c323ba1617bb7d413ee82ec9969ed48aea02674211d6ac9a807", "original_grade_sha256": "f122c4e3a4d8814d98b3036792123136109de6cdd8a9af2acba2d14c93f885a0", "input_original_sha256": "f531f3b01e8ebf14356f84037c4d4690e7c59f6e21edf8c6240b9d75382f8e7a", "input_wav_sha256": "04d474ab32a4a0c137e46b25d615c032711bea1fea9f6ae6505eb4e02871c483", "input_pcm_sha256": "f1c49c1f29dbc5b9c03afc14a25bac123d3da050283e61b3ef1e1be718b4c9e7", "output_wav_sha256": "fd21f21967fd6dde6491aa00f4909956d335fe689b6eb4f456da50378a2bcb90", "output_pcm_sha256": "545d3de70fe2cb59c7cbf94749b7171d7129109037e5537c8c723724186b4b64", "result_sha256": "5b67264068b8a6fababc8684cd5ec3772e9d6429342d4a72e861f589f5b2178e" }
#309navigate development	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "7734fd1431c5a1fd04daaae9ff3958051f8719d06a7783c1c85c6f30539b9592", "started_sha256": "9242c5b31327387681125559479faa9520ac9131e4a603acb82fa2f44b9ef2c7", "original_grade_sha256": "a18122ebea15ea73ccce9961622dda955099fe001d717ebae399529493f99eeb", "input_original_sha256": "7734fd1431c5a1fd04daaae9ff3958051f8719d06a7783c1c85c6f30539b9592", "input_wav_sha256": "6d93ebc08a167aa30b055a9f8f4650e68900561186020faa36bd75cae504ad0b", "input_pcm_sha256": "b5c1da425ecb909a1f89930e0824c1b58b513780200b2094f9a15f55182d5632", "output_wav_sha256": "cca0832efb5b56d2db070cc09ec113229779b3ad982382a6fa4ccbe42a33a5bb", "output_pcm_sha256": "52d6dc21260ac420314c8cb1c12438619d404625f7cafe74364195c20ebbacba", "result_sha256": "763ada95671a29fded75910e325c76065299b6ed95562c06cd4543e4bc6cfae8" }
#578object_counting development	12	The answer is 12.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b6f0d34d808161da8a34a6ba0378a9a3b8c517124ee57525361965ed77d8258a", "started_sha256": "b4270fa1228c7822928c70d6701a3f697d2477f9445127ada4e419d819cc4b10", "original_grade_sha256": "78a8f2668c043c95c2cb9c828e3cbd6ef7cdf23ccc399f926d20aa6607cad81a", "input_original_sha256": "b6f0d34d808161da8a34a6ba0378a9a3b8c517124ee57525361965ed77d8258a", "input_wav_sha256": "1cb1493fb4b2b9c172c93e6cbabc54eeeff4b5f6b3a7e2dce0a508757374d36b", "input_pcm_sha256": "57d291c9838917a711f7d2220742983c84d6990e5f2882466207bbb63f74017d", "output_wav_sha256": "fdac749961fe41277cb86b3cae7270309e439514141b2b7fb5e87a8897398865", "output_pcm_sha256": "61b914750637da637d4d1c938ff55dbe1c7d6716f9bdb8bf9aef9f884e56cf36", "result_sha256": "36eff17526d4b4feb171b5b498ae790945d5b96ca40530e4f404ba9aa3c8e40e" }
#944web_of_lies development	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "fa9297ad5c67c0484e1acd73285e58ad0a8d4ec7d45d6fe005635cd594225e24", "started_sha256": "dcfb9181cc29ab411a9d7a93a96a0d1b6f50b35e9af426e3727e1b5bb329184d", "original_grade_sha256": "85e9d0e9f6715969a8d80bba00c95c8816083682575d2da5368b454a0998a8e1", "input_original_sha256": "fa9297ad5c67c0484e1acd73285e58ad0a8d4ec7d45d6fe005635cd594225e24", "input_wav_sha256": "9fe037c900a04d3109cb472d84372d3761f41c894832da0353d51828f712c44a", "input_pcm_sha256": "babcace82751bde0011a6def950046ea76b3a33edb1efcdaa7976d989df3c10c", "output_wav_sha256": "f5f1a4cd72bb610019d1436d5faedb25f8123619953e19bfc781d3ad77797aeb", "output_pcm_sha256": "cb752fe975caa6a0363da0cac85e97013e1b7d54952a68ac7e496a4800033c49", "result_sha256": "94d306211e4e2054818b22edd61dbaf1cdf61776b47278dfb2cf89243a909a2d" }
#217formal_fallacies development	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "0b2c9139f4f50c778a5e7f940de508006db13625f5df7a646c1a17e5c8e75b95", "started_sha256": "fbad156d86f7f201839229b4f4e9e52bf7e1461d3905e17fc91ae7cf21066937", "original_grade_sha256": "536d6f67f7fd5b3afec61abbbdc6e49dc02f163d5332ebdee9d9fbbd0811cefe", "input_original_sha256": "0b2c9139f4f50c778a5e7f940de508006db13625f5df7a646c1a17e5c8e75b95", "input_wav_sha256": "c1b7a0e97b1f78d153309803f7a3ec39f1710fab8bab0d58f8c9d4a030a65f60", "input_pcm_sha256": "ba9122acfb211d968f5424b9b9b1dd56b35a08b1c8c70d90ea0ee4fb69491974", "output_wav_sha256": "89d858e9e985f4f0fd58704ab06625d8503e4177a49d197ac1d3400578ec2b3f", "output_pcm_sha256": "0064cf61dc2fdf68b2b7d595aca45ae1abd13db074f84bedec4843b58a2ee028", "result_sha256": "6e04f9f15da6aad87f25bf6a61cc9d5448002c67751966312a28c3f9fadfd7d7" }
#366navigate development	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "0f904339b01ca274c3ad03d8a5856199ef169035a944ae732081d19d3aa3afc8", "started_sha256": "9731f1c0f48b37df2817ebea9a5e901c9505947afe2d71c0eeeb1228e6f4ee06", "original_grade_sha256": "6207e3ead59a3d743eac69f11aa3e55407eaa3822408ac44486f9bde19d9691c", "input_original_sha256": "0f904339b01ca274c3ad03d8a5856199ef169035a944ae732081d19d3aa3afc8", "input_wav_sha256": "4beb12902d8ae3c7183882abc63973974b7ad9ff9884a7a3ba0c0730ce3e11aa", "input_pcm_sha256": "b60c6a08e09250cfd312206adc31912f04f721dcbd228642dd58c0b9e61b55aa", "output_wav_sha256": "3fac444e097c5084419b57ef41bd4241f11ed5a7f48f556ef72033548ba7f1cf", "output_pcm_sha256": "d2dea565191b74eecb7ebe47f0a208779ddd8c5ed9a0435c45765da860c5d5a8", "result_sha256": "fadb2bacca3919d9e237a843f24f1042bfca226e66fe4b221bdee6d5649f0f3b" }
#740object_counting development	14	Dancer is 14.	needs reviewnot_an_unambiguous_short_answer	correctresult	Additive review The independently recognized received audio reads 'Dancer is 14.'; the model's reply was 'The answer is 14.'. The substitution is confined to the fixed lead-in phrase ("The answer is"), which the pinned small.en output recognizer renders as a similar-sounding opener; the answer token itself is intact and equals the pinned key '14'. The scorer withheld an automatic pass only because the spoken form was not the contracted phrase. Same class as the determiner substitutions reviewed on 2026-09-15. This is saved audio/ASR evidence review, not a claim of human listening. DAS test duty independent ASR/evidence review · output_audio_and_ground_truth Evidence hashes { "original_audio_sha256": "3f1f79271cf5a4d9eb05d6ffc31ec95d7e8917f0e6acfa57676f85355b319cae", "started_sha256": "4720c89c371aaa0b5beb5a8e065967622e7bc138d4ef9f762040da3d9034c3e7", "original_grade_sha256": "25da28af3f314253c99285dd8be1c7ccb80931d7cfee62da44b646a22bd0a2c3", "input_original_sha256": "3f1f79271cf5a4d9eb05d6ffc31ec95d7e8917f0e6acfa57676f85355b319cae", "input_wav_sha256": "132248d4d14ed1141439283d3ba968bb41a1350c19dd1051c487f8d1e4cdf013", "input_pcm_sha256": "9a5b0a31bef699e2330476a5add49e4d7f65fa6a47971f11c8fb9ebc335fd5ab", "output_wav_sha256": "92bacc891de9f1f5aa873bf65b4be6d1cf5a5b5780d74e6be76f58afbdfe0c89", "output_pcm_sha256": "53ba81d1b6cf6c8433e85696d9269539be6624745846c68e8fa618a6e23f903d", "result_sha256": "cc9ba117843c0c9f621115ffa90f1bca15b7d7ce28ff79078a3dc9bc22359785" }
#897web_of_lies development	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "73c2e5b27594f47f14f56bfc403681de3db5f51379b6a6c569fc42a1b1e4b3c4", "started_sha256": "06be3bbc59f46cdeb52014a2b69bf69a89543259b91a6c039a085d878cf4453a", "original_grade_sha256": "3dbaa6cbe52e4b1bf408784299ed4ab6aff5e97aa0ed4382f52528bfd215dd0e", "input_original_sha256": "73c2e5b27594f47f14f56bfc403681de3db5f51379b6a6c569fc42a1b1e4b3c4", "input_wav_sha256": "b33e122e2af7204c6759d45a584009aacc75d5573498c04f8cf3e27e427c708f", "input_pcm_sha256": "20ff418b982552b2a8d0f8170b0a3ee3b7c27633cb35233596bf6f128e8b9da2", "output_wav_sha256": "716024ef837a43de6acf2a7ea9b55a4976f1ab0dead401228e32e27f48292ccd", "output_pcm_sha256": "c82ec310f006efeae7c741255770797e1e6eaef86d286e3844f430d5f0ab6738", "result_sha256": "887e905a7e5a344fb640a6d9ebd0b3b5cca97fd2b7d20f3a6df63a0db831e6a5" }
#168formal_fallacies development	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "0731a15635d53654de6f981e44b079f05742cecc0d55f845afbf6cccb258d82c", "started_sha256": "0f5cbd6f5942edcb507d4dc24eb2969fb4dd8b104ee439b6a48d714f17fa757b", "original_grade_sha256": "34a7aca2c44b75073d33e9320c481e1d3d8eca228c6cb275ce383589e1f7cbb6", "input_original_sha256": "0731a15635d53654de6f981e44b079f05742cecc0d55f845afbf6cccb258d82c", "input_wav_sha256": "dec6fdbbd18b8f7ac780d8fe60516d54686c6234b6f4cef86876f5198587cd5b", "input_pcm_sha256": "0660c9f44c02d3ffd38a1bb18f55b1939c80a371a420b6fe3460efa6bffedb36", "output_wav_sha256": "51f7b12118fe2167fda3cd68f3f789d4d82614db13f4984e67be19cf8094f728", "output_pcm_sha256": "16726149d6a9bbe637b3f91c330f406c45dff122ae777c8496bcfddfa6699689", "result_sha256": "400655f2ae6496e2f693ab97e0c48e917e805afab6ed28dce079df7ddcea27e8" }
#332navigate development	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "4bf429cb99d8179e169a068134544c07fdd957f5d334e1f9892577454d8bc4ae", "started_sha256": "ded14494f6bef314f9ba2635e9a6a1eb20bc6093b6708b5ceea57632d7c458a5", "original_grade_sha256": "0a325d9564540ed7507e99c5eca197193a3e0003bd077fcd58873eb316548eaa", "input_original_sha256": "4bf429cb99d8179e169a068134544c07fdd957f5d334e1f9892577454d8bc4ae", "input_wav_sha256": "7a6d0ce67b0498690f238ddc969e1243866b03c79aea314a3570aaafee37c9c1", "input_pcm_sha256": "162f21f2686f887555c93ec5c0280d9c98404ef9efce683658be476af59853cc", "output_wav_sha256": "87298f22946be22a3fac76d80e195b12e6c0f7f0a6b75bc8a939c4734044b31c", "output_pcm_sha256": "5199aba8ffc233fca4b8875ea5abb7bd36ecceadc98f46d831c2e006fd0a8bb9", "result_sha256": "577cd39f2d18b549a45e418132920ff69dfefa555f693d1610c8806829a06848" }
#615object_counting development	2	The answer is 2.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "a9678c1775ea894305fa4ab95605e4de9fa2b1a05fa8c0953c7d17cfd3f7e334", "started_sha256": "b6f75e1276cafdb040934124601c909699594af1d1ec828f1954077f5bc83c5c", "original_grade_sha256": "4b4bfda01c72868a2b42a7ac59ea04c570c7d66457002e708c5e6d7ee516af85", "input_original_sha256": "a9678c1775ea894305fa4ab95605e4de9fa2b1a05fa8c0953c7d17cfd3f7e334", "input_wav_sha256": "9f6f7f18c5fc0e2bcb384aae19a36bae78bac330bed3abc22194ce84f6ede4e2", "input_pcm_sha256": "63ddde9c2983e91433fccba96159f2a6c19da72b8b18233ae8612a0352cd41d2", "output_wav_sha256": "9b7d4277ad40e233cb067c90e5fd68a02cea1da39cdccf31b66fa7ad87196d52", "output_pcm_sha256": "85291b522e163c492af0e0646d38a066abb3bd82e4a88d944f0e5b0ef9ea3981", "result_sha256": "d48d3c0c119d4e94f25fc17fa08acc95f557a6f1fde926587e2daa1ad6d7c358" }
#752web_of_lies development	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "cfc9c8d17923c4d621129032babe8012c1d5b20bfda64f058c0be910bdc09c33", "started_sha256": "505c1f39ca0d1f6f95173d6badfe090f4e62c421b762ec63310296f515eb1b62", "original_grade_sha256": "fa67afffefa43416f4c500585fff80117e50e6517abdb1b36292942da12e3e29", "input_original_sha256": "cfc9c8d17923c4d621129032babe8012c1d5b20bfda64f058c0be910bdc09c33", "input_wav_sha256": "f0a3699ab7ad079e86328d38475b8cfa5269498d892894d64ebf314309ea8b67", "input_pcm_sha256": "4c07ffe6d648710b9e0ff2a520dd44b3e4b5eb25c80923a0c2360756f47f87b0", "output_wav_sha256": "3f361b2a3c620d1c8e820053d41dcb1b607c8271f4e51edae34da04f3b7cda99", "output_pcm_sha256": "3e692d2f692179820f94c0490d01fe0525749f85020349fe598de3e3ed69bcdd", "result_sha256": "1bda68388c5d2918df44cae3b59ba5e585fb698a545f5109b6009b3b506a2bca" }
#234formal_fallacies development	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "8396323ec0678356b05a587a76e54f1798179c616c7243d55481d8e46249c038", "started_sha256": "3ea31978c782477bb30ab3fa491d862bc1db7eb26c142ef2b735161f0d1d4391", "original_grade_sha256": "a51d0893a0af42da8d8a3c3ebe2d69c52257b633d807466df90e0223bb77e63c", "input_original_sha256": "8396323ec0678356b05a587a76e54f1798179c616c7243d55481d8e46249c038", "input_wav_sha256": "7ebaa4edd486ac73e19371fc2370996a33cd5013d6b17529f33cb2dcf79f7226", "input_pcm_sha256": "83b2e053e33eb02fdc1b0cd09645b5d6b027dec95c22e8a6029bd2405db86b31", "output_wav_sha256": "ed9919252715054206d0013cca74abbeb590f33279e577abec7d1970df0f1e8a", "output_pcm_sha256": "ca0a68c88e6fbcd1981800746f88455393ee098976aeaf7a95aebe841add517a", "result_sha256": "1f42b8f05ad78395ebfe7dd9e33cfe0fdbc5ef66080d9b1c292e74ddbc1e6a0c" }
#349navigate development	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "4dcf5c5bf0bb90fbc8e04eb06db2170193a6e1f57eb292604837e0bb84c80c7e", "started_sha256": "46d65aa06796e5c012f49558e9c4cf4a300b68d11b6ffde1756bca141869fe0b", "original_grade_sha256": "3900180ef59905054a906e93fb93d6f5f00ae2077b26fb0ae3f135161a2a0cc4", "input_original_sha256": "4dcf5c5bf0bb90fbc8e04eb06db2170193a6e1f57eb292604837e0bb84c80c7e", "input_wav_sha256": "630ced58efeb1dac825eebc9285d4cfd19c7b69cbcf659c17bee0ad51a6553e9", "input_pcm_sha256": "a0a62e97c1e2166fb40bc44bc8a7a85c8d20fa97f5bee8fcc36186b1233d5a5d", "output_wav_sha256": "b6dee20488823ca878784ba7d57aade1007862a5982bcfd9e1db63a5ddeeb31c", "output_pcm_sha256": "c95333e125b03645d3a80c82fabda8dff231205dd8949be9a43f149204d770a2", "result_sha256": "acc63af6918aa28822a593220524a61d02b500cea449b460f8c3cdca5fd761a2" }
#628object_counting development	7	The answer is 7.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "719cf10deee4686469c5034c46c1d7ad967271702fd1464128b498de9bc0d118", "started_sha256": "8c37803e9549c63b5b9faacc533156d0c4173b1ad96900a2e39ac31e44fbe790", "original_grade_sha256": "45a7ffb8f6c23f7e217216e4a1df97190b8fdb9c7d99954d4ce6173e980f3e6e", "input_original_sha256": "719cf10deee4686469c5034c46c1d7ad967271702fd1464128b498de9bc0d118", "input_wav_sha256": "214e7b1f692b8f0db1d4b272841ec1d201bddb997288826cf52585e9d7f8e2c1", "input_pcm_sha256": "80503acd754fe9d5ce6d92722ca039b4252c789a41dcf0c2341d23035509a750", "output_wav_sha256": "8ad06794f3913ec0d2668bd02d6dc30435b62c927c2505e228d3cf9490d4ea57", "output_pcm_sha256": "128b1819ea4620012f9fa352662115850787df43df52b9a799e681bfe979e3ce", "result_sha256": "8bd7aa8368b3d20ee3737ca1fd81f599aed3be93b5579133f97b9be512756872" }
#927web_of_lies development	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "c6a4b87545bbf860e3d428f50a3b5a84b0e6d26844bbad1425b2a2e61c456a3a", "started_sha256": "33e061363e278bdf8d74d92a686c1e0dfc891a640f05fdb67eb448b55ad84000", "original_grade_sha256": "0e6dd774625ff3a7fcc8bc2fd60e04990dd1b6d9f11cb9be9558b82e9e52976e", "input_original_sha256": "c6a4b87545bbf860e3d428f50a3b5a84b0e6d26844bbad1425b2a2e61c456a3a", "input_wav_sha256": "280c628eb9af1adad8754c635556e4c33af3fc8dd266121b423cd8f65c05c510", "input_pcm_sha256": "9afed67032582f77531e958d5d6a69c81f895a6bf92a2da5c6cad6b9e3390211", "output_wav_sha256": "aa4ce56ffc0512d9e3262f33ad57efb10d888ac64066fe9d0d2d6428405f2df7", "output_pcm_sha256": "b6c967d4dc8840085000aa43e2f0cd23cfc2cbe089ae31db7c5de01861e2f886", "result_sha256": "bbeb342bae79815e6a4ac718445ebe70695ed18ba5244b93514b5185129a27d4" }
#56formal_fallacies development	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "6b5a455d833cb960f17ce4b2489ca91b67ffad49ae478c3dc3e674f7e98e5091", "started_sha256": "f1242d70a0af6c4ebab822e7dba916d779d993863571282fc21f33ccfab53ae6", "original_grade_sha256": "93f19cd8b7d9a3ba4d2c4c87be322c07de9a1d48b69d7c077e08aac138b6b976", "input_original_sha256": "6b5a455d833cb960f17ce4b2489ca91b67ffad49ae478c3dc3e674f7e98e5091", "input_wav_sha256": "e4ef56c59e4088b2e97cd2902fc8b41823cfe78b83549032ffe29921c722f8aa", "input_pcm_sha256": "ff780ab593d7694cc9d2dea40aba7618784f1ca60acec9385e8f771d801fadf4", "output_wav_sha256": "bc4c30f899448446c1b0a3bc5ee5cc2a02046e03e46d8b78fa35c801ba73408d", "output_pcm_sha256": "909a92dc2b84e50e2a93c175dae42813adefeee9d7bf365da19c547002062b43", "result_sha256": "cabd912f9867413c65eaf429d0602e6043cf30bbad63a33d65eea81ad3078179" }
#419navigate development	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "1643552b8353a118bd9c5afdac658daea44a4dc9a2d3c4ee7de169147f8f60f7", "started_sha256": "c5331bf94d2adf84072946a8dc30536cefbf55dd88080f8752e469d7f15bf08d", "original_grade_sha256": "e9a256d3a12ce36ca56908cb82d076810065d9b172f041bc87622feb01279aa5", "input_original_sha256": "1643552b8353a118bd9c5afdac658daea44a4dc9a2d3c4ee7de169147f8f60f7", "input_wav_sha256": "097914abb18e4c60e8ae8ad35b36509e27d3139115dd7024d78c137ac41c3d57", "input_pcm_sha256": "1c3831631c7ee41e15f2f75004d3f0ca7df591e5d3cce22963fd018eb96ad7ea", "output_wav_sha256": "be342d0a92d2c2d7c7ca7d638f8f5d4bd85a1d2af26652c7856544d37c34e09b", "output_pcm_sha256": "6d94032d6fdd6216ef18db3a170b24bd1a06b744d3847748e76d6ee6d49f60b7", "result_sha256": "9e357e33526860d2f060617fd21b2d5d289f5f600cd0200f2b835a8c172bc6a4" }
#663object_counting development	13	The answer is 13.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "ac39d824d4ced9185648300bd4abb304dcc1728e6e6fec1ff833fc14ae2990c7", "started_sha256": "dcfb8754f6174a4333b7bd78a5d0e8be62cd950ee9e707ec3cafcaeddc7f585f", "original_grade_sha256": "a0cab882d0fa545eb6b0e81489508076f41055e36a3d58dac0ac440892ca7c3b", "input_original_sha256": "ac39d824d4ced9185648300bd4abb304dcc1728e6e6fec1ff833fc14ae2990c7", "input_wav_sha256": "6fa8be2c4ca637402eb557aded9e41035c53196d9791ce4786b3dcc17c784f6f", "input_pcm_sha256": "553e7460bc223a57cf698bceac4133f82c79514e7ad2bfc7ceb62ba5e0e90690", "output_wav_sha256": "080ced6253827f816479931092f65ab1dda4b849ce592f9f4e88446f809da30e", "output_pcm_sha256": "8e36b05c5efb57d3617e075d8e2693e6cbf2ac1d3f7f2a9acdd30a63237a127d", "result_sha256": "52261e525a140f31874dcbc9f34d4ea595d4490795aa7e7ba48cd7ba68592aa5" }
#757web_of_lies development	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "9637b2c712594a648b6e2c8946669f03852be039835a8911402a3fca7ee99021", "started_sha256": "73ad3c365d158181b098466573b0e3498437728ea6a7c110363091b5d4f18b96", "original_grade_sha256": "2779fdbd7fd10d980d00453c527d211cf71c892b6c64ae4daacddb6efacd24c7", "input_original_sha256": "9637b2c712594a648b6e2c8946669f03852be039835a8911402a3fca7ee99021", "input_wav_sha256": "43996e7c81cc70e6b237723264349576c922968bde8bea8eef7b9b3306705866", "input_pcm_sha256": "816d4a4e0829ddb626a7c3ade5dd61a23df91b6ebe47a1fad484b441df01e996", "output_wav_sha256": "fd8db8099e573804458998637cb429d3d51bae18a28a4090e73e001c4cf90e95", "output_pcm_sha256": "3b33995a293b843a6d9820b7b29c7e09472a10dd88cc67c51c492f1e0120b830", "result_sha256": "bcdf47387937798bc7e891b3157a9320078e30cf901d6453e4781e8c00d265c0" }
#122formal_fallacies development	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "fba49edbfe90d9b44f0660c184d08971d9bde008e7da7bafa54d542072547b56", "started_sha256": "241565668d3a3d25931ef7ba90a9df555106fb7f28e460358d157cd28f1593cd", "original_grade_sha256": "e3954913092f9e5560ca48c514f64d18d0f8fb6f2487b132823bdb87e1c55ee7", "input_original_sha256": "fba49edbfe90d9b44f0660c184d08971d9bde008e7da7bafa54d542072547b56", "input_wav_sha256": "5f8df5fc961b62a2534c261a4e37912406c652489bf2ec7ff2a9fe1d2f987fd2", "input_pcm_sha256": "53b9e832e1b4f1b3615b5ed2d2d230dcb20b0feb43cc391c50a2e21a39ab7578", "output_wav_sha256": "16e84acee7bc715a5889c6a467f0535d992e003f399972f437271b430047430d", "output_pcm_sha256": "76cd89f1332f9f52ba7c9b8fc59e6ceafcc5ef21ded2c87d39928162d34b7c64", "result_sha256": "049d74d4d923d7aba0856f5c6d4981e8f7357c3c38cd9905837ebfa99df92eff" }
#294navigate development	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "1a04cad08b4ca62818776bbe2ce2900ae5345654e0816944e47915944ff796bd", "started_sha256": "6304440c3b4d3420857625e4a7eed3dce7a2335f8d63f841e8595275539b7932", "original_grade_sha256": "3cf5a393b6e7a0950754bcc83ba777f5e3ed494dd45d36a3b1b11824bbab9a4c", "input_original_sha256": "1a04cad08b4ca62818776bbe2ce2900ae5345654e0816944e47915944ff796bd", "input_wav_sha256": "b62ee244abeaead21842d7ecca1e9a9dd119ac4a2ce9a859d76646753484a6b8", "input_pcm_sha256": "cd4ab09f864634002f8a5a112aa762b8327234cd21e60e0829bec27168bdcee3", "output_wav_sha256": "0b35b13bf9a5197ee619e2fa5856c8daf39cdf403fc170dabfb6d163b0d59a4a", "output_pcm_sha256": "773af7e49207b1791bea8c0346f1e4c7390bce5b358ad2d0a1056d71f260e761", "result_sha256": "b5281a9ee2ece8071ba4d6d7335a16030d9e8d6b6262f8c8e3ec851ab77272a1" }
#730object_counting development	12	The answer is 12.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "27a0601d4532a1fc6f11c9dba8091d4558bbc5f9337f3c89cdcebe515a12bd2e", "started_sha256": "442d365b5e6247cabf91e8946cb11730d822705a984c77aac3dff9d80d854ddf", "original_grade_sha256": "f10bbb4a5c112f6d113480cee8ee5e8f874b897cc7f2ddf1a1ffbf6d6b036786", "input_original_sha256": "27a0601d4532a1fc6f11c9dba8091d4558bbc5f9337f3c89cdcebe515a12bd2e", "input_wav_sha256": "bec6d3972aa8591ce38f134da0792b221c6fd42cf5d523ae217a400df94d2b17", "input_pcm_sha256": "7e6ff7d8f71bd744fd2feb748a51f5d95bdb85f572950323f14508e5bd2cdd71", "output_wav_sha256": "04f916bb0ddfed964921c8935e5692f29d1e639bf7792aa8052aef15e6ac7c09", "output_pcm_sha256": "bc5b3ea836dd874a3f37008a1053da924627de438b8b0acb8d558b10cd728d64", "result_sha256": "54349b9115f51916212a7319bc27d78e1c48ab376b402090cccca7eb70134b34" }
#876web_of_lies development	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "773485081fc7cb23434789b012366c205b31ab893d9e560edc448ee53ca72ee0", "started_sha256": "ac997f421daf46cafe47ff5fc839d0fa9e14aeae3233a72e9dab1498cc9ee3e4", "original_grade_sha256": "2281fe88b564d7a74bc4338ba507510b632cf8ee864550f0bc06ebd3918ece4f", "input_original_sha256": "773485081fc7cb23434789b012366c205b31ab893d9e560edc448ee53ca72ee0", "input_wav_sha256": "ec1570420f3585fec6b2faf1d55baf12d9ab03d5dc084bb00c4754ffe606512a", "input_pcm_sha256": "bb4c0cadc8073767a875408fe03f7ea74a6d1f6e2a91fdc22a6bd3eec5096ed8", "output_wav_sha256": "78ba55dd6f9aba1a8c82897eeecc54e7a7838db2aa49676a1b86e7dbeaba9172", "output_pcm_sha256": "911968ebbf721287261820740018494c442af40c11d74161f3a536ca8b997bdf", "result_sha256": "7c98a0fecb7972d364be39ccb4628153a2de96c744beceb29deabc408f6114e7" }
#1formal_fallacies development	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "1ba88c41618722050295d4d27a64d8f08e1bf02f6856f00d29923c2cf38c31ea", "started_sha256": "d1880a0cfc43ffc083416b553c197c1f64151f01c0d1cbadddfa4803ada9dff5", "original_grade_sha256": "561dfcea83ec2fcf9b468569be0621d82012211a9bd176bea3986cbca8839735", "input_original_sha256": "1ba88c41618722050295d4d27a64d8f08e1bf02f6856f00d29923c2cf38c31ea", "input_wav_sha256": "88a2b0cffbca595b93b10cad320e680aa9e357205e6ab0f7d328338f7a407734", "input_pcm_sha256": "650e72597ea68e60a4d4f1d517b6f5c7c5a80d996980afe7798bec07cbe8ba0b", "output_wav_sha256": "0af860560a66b2a628e77d7e7619014b7f75cb6b97d37d84976faa1b885362d3", "output_pcm_sha256": "7b402a9125fa462a9b81ffbeb4872b822d215fab8269318dc81df6dfe6893ffd", "result_sha256": "6d3a06db50552d9dd9ccf5934c44029a55550154ab8400dc5528c12ccd678e19" }
#460navigate development	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "ec2ae41c1a84cc6ed986fd567b81030909f6b996647fbe51ee857611061f6a65", "started_sha256": "ba0b0a1ad72a19fdd1f1dcaf971aa0b22f243c5e87952bd8da9860e118e93c8c", "original_grade_sha256": "a209e1ca196d8927246c68045a34d0302acf3dd9c01ba6d15155e8b0eca135e6", "input_original_sha256": "ec2ae41c1a84cc6ed986fd567b81030909f6b996647fbe51ee857611061f6a65", "input_wav_sha256": "bbfac98cc986bca7ad8b9579aeba81e15096b3198fbb2a5b5aa0ff6e21f22f73", "input_pcm_sha256": "6f794bcee20fb372878156e645b6e06ad3548b4cc4e2aa78a7533b7d308dfdf5", "output_wav_sha256": "4c08721367334c716f616e4e6c97e0729fadca93a7001c0a4bb00dad8239625c", "output_pcm_sha256": "bbdbb21e99005124c560b98b7d7d73b077824f94431b33de5a867e26a7369065", "result_sha256": "f8867dc12235887c9a46d78c7a3ee7a3af6128d507ac18da91f4eab0f3abefa8" }
#535object_counting development	6	The answer is 6.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "01078429e26ea6b910363a4ecbd06fface0820852ec7c0eb071bcaa2e104ee17", "started_sha256": "622431299481df52b2a126c2d9f6367998500f6f4e0bf854ec3838fe1e9f6318", "original_grade_sha256": "6d7e6cbab6ea647fe590ad176162d575d4df1d3cd1ffbae63be1f7c0cc635c33", "input_original_sha256": "01078429e26ea6b910363a4ecbd06fface0820852ec7c0eb071bcaa2e104ee17", "input_wav_sha256": "8e1da756d6fb0cdc6b2fa950524f88e7f46f7f3e7782e12bab6131c3c1fc7f2e", "input_pcm_sha256": "bda63234d6be8fc4b203409c48ad318a5a7dd9c2a25855e9ecd1332b48880de3", "output_wav_sha256": "d21b4fae081d255dd7cd2998e971d8bdc9cafda0087af82a16c5e63fdbbb3aa9", "output_pcm_sha256": "192973c4636af8a40e6d9b3101c18ff4b0e2f6969a4be8dd9bc8a5ab236b231f", "result_sha256": "0584966677636fcb9ef2076cb835d1c03756e55d114a50681f2a3134caf13830" }
#750web_of_lies development	No	That answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "736d47c7525a1c64bf70b4449ec117345c9091f2835331af04b8daa2e02f226b", "started_sha256": "f842bcf5d1646c3ed8f82b982acb7f2e3221595b289909029798f8c1f462bcf2", "original_grade_sha256": "77a1748aa953328a8774a046dd4a3f9fa966c53c5ed97c747d6750449286056f", "input_original_sha256": "736d47c7525a1c64bf70b4449ec117345c9091f2835331af04b8daa2e02f226b", "input_wav_sha256": "e2cd1218de897f1532cdce5ae71b5c706941633ec8bd81ff8210bebec93722de", "input_pcm_sha256": "ac6925a82df72ace244007f1a3f77aa22bdac6b4830d5d239943a86c1eb18b20", "output_wav_sha256": "ab10b9ad26e5ba244f7ac180a6e22a0f27b6d602af74f4b7011cb8c9a24cf81e", "output_pcm_sha256": "4bb727f282a78e6fb5ce42d346df660eb9fb2d0d365c2d7c034161042b19b67d", "result_sha256": "a05f66e6f7abfcfd5ab7eba38d5c6ddbc70cef698f57c6b7277b3d389bca75c7" }
#126formal_fallacies development	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "c2f1266be3fdfb3ced263a8405676ff3f70bb5292d4c63462656dfd726c239a6", "started_sha256": "0dbe868c680f090d27655b11b44a8888ed5736618773f1a60888f124035645d5", "original_grade_sha256": "136f459d4d1870a4634dcc37de90d310fc3c131a41ac990a32557d1b4b7c35e2", "input_original_sha256": "c2f1266be3fdfb3ced263a8405676ff3f70bb5292d4c63462656dfd726c239a6", "input_wav_sha256": "251e932a7c529643283ca667813e17ad7b41bfe5fe7206562d70000a9e05ac1c", "input_pcm_sha256": "abae90a310e20811bf7397933998acaa7c1fa554e97147ba7aa4a763b870fa31", "output_wav_sha256": "bfaa05b6551a4defd83d122729af3d0824a27d5f2ca80f2f36312cf7b54eb2ba", "output_pcm_sha256": "f92fee0c42e1697f019d36e22f271ee7a719e4c040b8c4151ece3903e9fdb7b9", "result_sha256": "9390aa9f580a1047a8d7f3d8bfa6b8e82b08dc472d69703a13a51438188fdf0d" }
#397navigate development	No	That answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "0a78d567b65f2996d4caf0d612e7962ac2a2c4f1fa10aedfc8d28d744bc74fb9", "started_sha256": "40330e7335d99d7541bafe7d13890e757be90854033812ffca7dc99d39a40a58", "original_grade_sha256": "ec771e9096ecc8269270da5cd96f2cca95021625e3011ee24de202b8fc4b7bfe", "input_original_sha256": "0a78d567b65f2996d4caf0d612e7962ac2a2c4f1fa10aedfc8d28d744bc74fb9", "input_wav_sha256": "a1e3711d6b5c5f50126e563f27b44d69147c93319c2fca3c1b78b6e540ce2449", "input_pcm_sha256": "6555f01e7a8eb2861c733a1214b3795fc323a6db70d96fdf309b43a636034f6d", "output_wav_sha256": "c127e7dcc52f25d2e1cd2048dd69037540f6f141ef23ea0773083c161703e682", "output_pcm_sha256": "fcab898f41e39b9a1aa77963ee8a0ab980f8d8576c66dfd52b9bf40a12874245", "result_sha256": "2ebca9038bbae01746cc5c1491c7281b9a34beb072f27bf8f036711fbf4008ea" }
#715object_counting development	9	The answer is 9.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "028120346ebe02f86a705328c9ecd2fd6922e2937632c920748b5c1eb9a3fa06", "started_sha256": "b47bc471b0f65767aa88658354b5b365ca70ce98458378f39e6a69f4e2a66155", "original_grade_sha256": "08cfe8a69815f95d77bf4dac99404749a987a420727369f76311c113b7241129", "input_original_sha256": "028120346ebe02f86a705328c9ecd2fd6922e2937632c920748b5c1eb9a3fa06", "input_wav_sha256": "ad0a581dfb03d1d920567e7fce28e293a7ab6f5794b783b2c6632c5e2bb37e48", "input_pcm_sha256": "2b69d2888511bb64cfc6d702319f494497e6b1949b2e0429525fcc83b551c9c0", "output_wav_sha256": "22321df6f23c1f7ceb2dc7ad0762cc49a286a3be0991c3814895b3000481a013", "output_pcm_sha256": "55fc7c8c3089f72f15d567c1e2b91c0aae6a908894b88fc877d96fe1f8e46ff0", "result_sha256": "1405a6971ccdfa085c8a5466226d04f8801e2e9e411142511df3c131772dbb3a" }
#950web_of_lies development	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "a6cc05cce18fe80a293d7b3a5734917e79833972bfdaa6ee916932ed18e6d1e4", "started_sha256": "865ee4cad08390c8d98450d2e9e24ad05ad3d46c71093a68be9b49ac24e765b1", "original_grade_sha256": "15b75506692087755924e9dc485fbd144ca81abb99b6524e89db6614a2ad0d94", "input_original_sha256": "a6cc05cce18fe80a293d7b3a5734917e79833972bfdaa6ee916932ed18e6d1e4", "input_wav_sha256": "8257f679c6c025c343b86af61c24e13745eb65f388ba18e144da57b3c082721b", "input_pcm_sha256": "9ab5ac9c2508983ca548c3a8d05d20f5e275782dd08d2ae7703ba86320896af1", "output_wav_sha256": "6ab17730e9b162d8e7a3f750d3f628724ad447389f0acc3e5ce7c321220861fe", "output_pcm_sha256": "a9de3aa53a097fb62395d5dcfd28561e42bfeca7aa981fe14e69e1c588f1edeb", "result_sha256": "e7d40d9a2624d533a520847ca354c9f67242b6c49aecb855f76a7e1b4264d96e" }
#93formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "e2985563e346ee8a1fec5ed3527bd93e4ca5a4f2116a22925e0bdfa312d23ec8", "started_sha256": "ae4680f519f585f38a25b57aecb8b8f3db25e3914105dde90b1301e6d3457b68", "original_grade_sha256": "5d9182d2d2b00209714b61789edac468e506d872153e3e9957dd74756bf1ffa1", "input_original_sha256": "e2985563e346ee8a1fec5ed3527bd93e4ca5a4f2116a22925e0bdfa312d23ec8", "input_wav_sha256": "4dca834b0976cf82589c337f2297ae7639dd26674719af4fc139ff93f76f2d9f", "input_pcm_sha256": "98a64a68058e200e9e01fde3357fa93523ef57b582af37bda37819d5e5db96ee", "output_wav_sha256": "989f716c77390ac4813a1dfa811c15ce60da7ed7e7177296bc90d8f64b24c581", "output_pcm_sha256": "087ecd1e7d5eaa26e1fc1ba4288979460cf6447cac774606cfbf6af16deac57d", "result_sha256": "4d3e485150b2778e6e578ea469ed605d667e8cce411a33d69bb4ceab6de2e7d0" }
#354navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "da40f07e0aca9a482e0f0193c857315164c45319a651848ab60bc9347162ab20", "started_sha256": "81ce592b0e21526f1101cd2e4f595e106a45bfbd625b66582dba1734fae18ee5", "original_grade_sha256": "7a22d65ee98d2dfb7ec886a6e27ac55a292e91774d95624ee3a6aeeb5e666986", "input_original_sha256": "da40f07e0aca9a482e0f0193c857315164c45319a651848ab60bc9347162ab20", "input_wav_sha256": "30f4219cb9a511d0c037e92a72602079ed38ec244327579f3ad353a4a62acf12", "input_pcm_sha256": "b80952202b0d9e17b51b83ca865709b149ebdef0fbda260d8b32914dfa06264c", "output_wav_sha256": "a94f895e271535531319bc0d6df608b5fbd404e8a39235581100dd80b152509f", "output_pcm_sha256": "6f90eacf58927c0858593827bdc783a26817c100fc88183e25a794d262f77528", "result_sha256": "53ea3108def13b15bd61b2efba4a1c90d1779cab7bff3458fb9869558b1d99a4" }
#670object_counting heldout	11	The answer is 11.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b1856894855aaa22fd6b1aff4e774231d2194e1a88bec22bd953d0f528609c15", "started_sha256": "5464cd8c1bbfd55c83e0fc1cdf7bd2c75948f359e8edab2bc87483f046636969", "original_grade_sha256": "b54ea85329146256b397f1cb09aedfbe26ccda3632236af04417f0473d741a6e", "input_original_sha256": "b1856894855aaa22fd6b1aff4e774231d2194e1a88bec22bd953d0f528609c15", "input_wav_sha256": "54417c21c96a657d63f11c10688c8307ec92caef8e8613a9bc3aa6fabbe99dc5", "input_pcm_sha256": "e34b67aadb5dd1f12a3efee29a58570f11f87277abd0336bc21ea83e604158e0", "output_wav_sha256": "8fbd36f79f9a451152ac53b47695b8c271cd3b7c32c91ace80a09ab1052cfca9", "output_pcm_sha256": "d3c7cafa63f8ccb7f54bb53db9ed6c5ecc586f996e5f50f152ea830f4a6db086", "result_sha256": "af736f66475d576a8808bec64f054c8dee7a3854d9e148c078a305dff9ee16d1" }
#971web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "949320509335306c5816f3a2ea54768ee09064d422a33cabd336652854bc8fe7", "started_sha256": "442c769d2bfd25681e9e46d152a953aae20b06f0a06650f988101bea10cd8cbf", "original_grade_sha256": "affee7fb083378bfdbbd0122c0730be6aedf2bd5624cee1838bc7ef834500873", "input_original_sha256": "949320509335306c5816f3a2ea54768ee09064d422a33cabd336652854bc8fe7", "input_wav_sha256": "5747228797b9990f04452122316596b8939ab9c586601b9d515552aac9c00f17", "input_pcm_sha256": "4a525a88a302a9a1618519bda9bf907f87c5f2058e4cfe6233b2dedd241a670a", "output_wav_sha256": "e530ccc28fa9628c8c853cca58d7f3c9350a3ba59422707c3621a2ce537e80f6", "output_pcm_sha256": "8d0e0903a3084cddf8f130cc581ba811981498f20fcc5af97cbb48551f3d532c", "result_sha256": "bfac230070092bf2f86baf561dcf9543c40cf3ff6689f9832af0687a48901978" }
#106formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "aa6a38204569da75ac08149c50b6b2645f8024878b23590d71283dda47c2b7c4", "started_sha256": "6b6021603042aac160ffda11462353c9ce371f95509d5a53966d0ed412c99b19", "original_grade_sha256": "9ad2cf0f4cb48d116adaa1d3f76b809939c7ad8586892323c9960553f73e0459", "input_original_sha256": "aa6a38204569da75ac08149c50b6b2645f8024878b23590d71283dda47c2b7c4", "input_wav_sha256": "c69b4fbf9b5105db4118638eaca0853298108812564cb37033e2dd7f686d7d7d", "input_pcm_sha256": "0292d3abdbf972d1325ccb05d86fe439ea1f5ecbf62671c9a8ca0e1de0d7f172", "output_wav_sha256": "1d67d344441ef9efbfc98d01397bca31acfb3c14008121a5a34aa344a53bbbbd", "output_pcm_sha256": "2152c3a53f119566ab625ca561d869f1a862f60a5f8e42524e77a61581f08e76", "result_sha256": "ba4365037d3fac6c519054bbdf6f996e6cb119c5936fcfe71a2b39d7dfd8b212" }
#324navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "a15efa619bfe74f823746ee58945d191cdf82eda0d5039bdc04590828f4aeb47", "started_sha256": "8fc54df5d1a5ad86112bd7f984c2576e491cdf7d752469ce598bec6355b51784", "original_grade_sha256": "7151cc6b2c1f851a9c4235a2c416324ab5afbdfbd3c7327d21194104ed7a8f9f", "input_original_sha256": "a15efa619bfe74f823746ee58945d191cdf82eda0d5039bdc04590828f4aeb47", "input_wav_sha256": "3370ab4cdc034d10858ab69c2b5554251d90f8a1d1b97c34e9c3884371fbd9c0", "input_pcm_sha256": "b551ba92c57a66b3972adb67c7b5437fa21b07dfa2ef6dae017c3cefc0ff84b7", "output_wav_sha256": "a06c3024ad86a210f5f1a231cce203dca9bd2e94eed643f359b57bc0fe40597e", "output_pcm_sha256": "6afcdd339df3ebb3903cac6e235a24b49c9e2e5b4eb946c177742c14bc8b7351", "result_sha256": "19b0ddf4304d6fe8bca6129c2b99aa386880b7ed20b09793fe04a7e3a0855160" }
#524object_counting heldout	5	The answer is 5.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "8f157e04ead6d7f86b2340bd5db88918d2dfd04a6d971e88191e2c0c23f71218", "started_sha256": "62de4bdec0c2b45f158427da57e2c263683e243e7064d8e30d4429f41d4b8cd1", "original_grade_sha256": "123e6d9c371fe611216d840de6ab194096081118920d1f7b5538741d15b041e2", "input_original_sha256": "8f157e04ead6d7f86b2340bd5db88918d2dfd04a6d971e88191e2c0c23f71218", "input_wav_sha256": "94a9eba2f3a49bcfee69bcdbd9fd464c99ed8486d56f56459eef7e9433df88c7", "input_pcm_sha256": "d65b05dcda022c1af8a08a130e30144ee3acf2311dfe8674fdf9ae03f3406600", "output_wav_sha256": "d326081c064933fb2b37e397e7d48f018f46ee5086a934bb000b073d3d7d7282", "output_pcm_sha256": "f1f777111a29ff23a1b643e9b53f761127d6369af10bcedaa76ffd39f843eb0c", "result_sha256": "fc49396fd11028c920432a76dc524ab21a6cf1fc4c60049f24563cac8f6b5f60" }
#851web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "4c639a665c73abf60354c7684abf4b94ab5db6667b926fc66b3019192643c24b", "started_sha256": "4890c2f97715c9ad05cbe95a46f25dfca92f26309db970d3e4b00a04a75eb400", "original_grade_sha256": "102814989202acc708ee36a039ed474f855873297a9a691b6603dcf8cfb584f8", "input_original_sha256": "4c639a665c73abf60354c7684abf4b94ab5db6667b926fc66b3019192643c24b", "input_wav_sha256": "d1331925e8fc1b340f2134f6559caa0682f3185c7a94185586c6ba570d398d09", "input_pcm_sha256": "218244f479806315c885e009b397b6f3ea9b9e79e2bd9a69537b6b833177fb03", "output_wav_sha256": "2cf0a6cef2c287c32597f3a3314bee6ab67b150c31eda90a7bea2c33c51e1c8d", "output_pcm_sha256": "e359b6068728878224fb9b073cab65fe6425532e2f243ffcb148997118c9cfdf", "result_sha256": "1ea8c1f00c2921e35795dd8507ee199a7b52ebb22460e2d7cd5be258e7406702" }
#76formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b1cd6d43f5342a9ba5770b20e43f6d4c363918ae76ccaa232d81a60123b5365f", "started_sha256": "03e4466f30280aa710635e2b92b0cdb7b2ab3326b1f4ec16269ce6511e854195", "original_grade_sha256": "133144a76a17e543c3d69b3b6f14a73799ea2d33d122b3174c4f11f3fb9d6dc5", "input_original_sha256": "b1cd6d43f5342a9ba5770b20e43f6d4c363918ae76ccaa232d81a60123b5365f", "input_wav_sha256": "357e8d6f767a99ede696d8f2577719b58b9201406c00d2b0614de7dc25db529d", "input_pcm_sha256": "aca9584adf4df649a0f7ecc96b1a06a456e1288c644c477ff123354106328ea1", "output_wav_sha256": "d1b4347bed10f29bd12e07956b9f272ff12176226886ce3560354fe1bb7daf80", "output_pcm_sha256": "23a2564880f9882d99fc59c36d0ece8dc7cf7c3b7acadea9d3aba34a5402690b", "result_sha256": "859dcc1d9ed255d415470e0f57bb5fc27779e3e3e8822954f6a09e0cd447c897" }
#392navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "fee682ff6b668ee05010d443c8dac6f45ca2f9b63b9a7d66a44fb1601f8c5b7e", "started_sha256": "f6c03ff6bfc59d2311d7434b4445ae2be72d19895e63b7900f02a3bf5594ca0a", "original_grade_sha256": "c96e291002d938867715cbe33c84a7ce6a1ca5c35a4fd4f990776b5b0d966892", "input_original_sha256": "fee682ff6b668ee05010d443c8dac6f45ca2f9b63b9a7d66a44fb1601f8c5b7e", "input_wav_sha256": "0118407666dd1a5c004e8be24e2329b2facc9cecfef4daa17ae747f5d3c50b29", "input_pcm_sha256": "1aeeeefcdeca6f2bd1209af63bdf9c9760acc6db7e6e4ed1f92052e6203355d2", "output_wav_sha256": "0734de2db48e404b4fb4ac28cd9e36cda91bc72561e46418995a95126a33d80f", "output_pcm_sha256": "3ab0d9534e4695fb3648b1770932fe03afc00d68c3bea22fff541c15b919b035", "result_sha256": "86aab94fc4b30b837a8ce5413c6c79eb5743d9acf28d9e4955bb9d8d3c687cf0" }
#656object_counting heldout	10	The answer is 10.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "a64042e3a0a04b1f22fa8a0151d794c336a49e119e1da8288aee0b3974a61451", "started_sha256": "6bc3c24f3802bc2547dd211ba95a7f962d10c26d67d3db049173471578fdcac6", "original_grade_sha256": "5781fecb71ea73d9d35bcc2a8c2542b0a982d4822f52d3293ecdd6ace13da244", "input_original_sha256": "a64042e3a0a04b1f22fa8a0151d794c336a49e119e1da8288aee0b3974a61451", "input_wav_sha256": "1957ae73731eb5b9fa82d55f736ed4881cab9a0f0c53a66369f906246601ec7f", "input_pcm_sha256": "a8e4903522b4988be389d8164fe288c2f6f0b5e996697f42ea2543eb4f1a6917", "output_wav_sha256": "bc0b66e06f83ebf64a6b1c1d1a099465ae46f997f1109be4e9ce5c2bc63ecd33", "output_pcm_sha256": "b874727c8efcc6704055f0f3487b076a5c71edf0a9bfb8b8ad743ba4b9ce234b", "result_sha256": "9350dfad712c2897ad2c7bd17d537fc05994174d3cde8ca96e5003cb89d056ca" }
#767web_of_lies heldout	Yes	The answer is no.	incorrectstrict_spoken_answer	incorrectresult	Original grade retained Evidence hashes { "original_audio_sha256": "88fa77c0e229c580692c1370b27bac5f4a414718b726b7e710cb45411235e71c", "started_sha256": "f6dc419540abbcd5b6802cfc5c151241eccc2a8a29bf345d4ebe5a50544c89a6", "original_grade_sha256": "ccdab429739f23c42ce4d73808aef2400c157a5e4f61c31c03aa5ef231dee51d", "input_original_sha256": "88fa77c0e229c580692c1370b27bac5f4a414718b726b7e710cb45411235e71c", "input_wav_sha256": "dd6ec95d4e41b800b424f8a794390e26e22ecac767aa756de0f85ab0a57e1e1f", "input_pcm_sha256": "8773ca7f335fcd0042ae9ec757a711dbaa7018e8bfb9e8da56eb66697f088e78", "output_wav_sha256": "dbeb8f99f8e7ee5ea700d3c021164f931296b1cf4a2565661a25973f0153fa86", "output_pcm_sha256": "f0dc120156cb1895d096242b8130d1b6b3505c2265bb38e0f062ed6f9aaed00b", "result_sha256": "35c00744612f9fc27c1d2acbe5b6d00398e647d4c40b325a0e712ccebb836d67" }
#220formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "50036990e9b2cfc333254454cc86c5d042521531acaa40af243674294718615e", "started_sha256": "38d480b5c96fcb2e43e9b7ed71c35525de76ceaa83e951938874b8b31bf442f5", "original_grade_sha256": "504cc541647dd983901cad32b6a729627921365db4ef5878898996a560004e18", "input_original_sha256": "50036990e9b2cfc333254454cc86c5d042521531acaa40af243674294718615e", "input_wav_sha256": "b9bcb3f51418e772f31379a981ae3c47131500c07b0bcbc53abdae3e0f7c6fbf", "input_pcm_sha256": "1e97e12aeed079bcec352808b4292520f9e6a6fbd1581098ff7b35cdf55fbbf2", "output_wav_sha256": "74167492a8a46faac38d6bf24c22b1ef59928dafa0ecb21cece4780def585526", "output_pcm_sha256": "d6f47419e34a0222ae75f834f689d8c06212d78407ebe15729092d0ccfcbc4a6", "result_sha256": "5dd45985fff5eaf990e413b61c273fbd95ec897bcf3879df738fde371d8db2a9" }
#433navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "86c70fa69b057e3774d810c7befcf506b5f5b762f6c01a0e2c21a1d78c00080a", "started_sha256": "1c868466e94ac21cf24df3e785b605d2d4f1c1aa27d8a102715c1fc6e3b5ec87", "original_grade_sha256": "6f2a2c98a166b7c48fc775d685ecc145cd524c70c4d9ca12d8294dfa41d79a38", "input_original_sha256": "86c70fa69b057e3774d810c7befcf506b5f5b762f6c01a0e2c21a1d78c00080a", "input_wav_sha256": "6069a6f6ed7f57bd904d096a6058f806ccc53c3efa667928ce70498a13af87ea", "input_pcm_sha256": "62261d22351b2aa1d6a09dae2166e1197a265dc6a43199afec5187001b0fa356", "output_wav_sha256": "41456c1039adfdf2909a3ec1024a19e0d8833507eb96004bc6c5f761e6f72a53", "output_pcm_sha256": "365b6baf7984d3bcf4e177aeab0e5b409f7ca81a919b01c19572287e05b9427c", "result_sha256": "c9f2609ddc6a6beabc5fec07b5ebd0db6e2ad9ec7bdbce12dd2947db3c4cfde1" }
#666object_counting heldout	6	The answer is 6.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "5dfc48aadd552974d933386657560eac18401f405ac3f363a9bc0199a5800454", "started_sha256": "789a638dcf338ee021ef22eb1e53b47d9df377cd474509761ff8135f77b01ff8", "original_grade_sha256": "b268022e56ad8b33e4b956514f2bad4a4e5c431a22c592bd35186910cf702f04", "input_original_sha256": "5dfc48aadd552974d933386657560eac18401f405ac3f363a9bc0199a5800454", "input_wav_sha256": "507e56c4a3067560f39af4c516db7369541e6b755bdb8a37e2b5b672c763167a", "input_pcm_sha256": "4b48be7884ea7bfe36de71d029803b6fdbc9885148e3bbe9ea9c96c1de4cf393", "output_wav_sha256": "f64b9937eb68d017a1c8816c9a7f7f2448f5a94bc7468c18454b5e582088a328", "output_pcm_sha256": "3d6cf6cee386411d170d3d71708a01734d780a2bf7c08c2d86af7aee2143728c", "result_sha256": "6c9762d701ed7710df90a834049ad70d02e53e3d0e618468cd5583ca170712ef" }
#969web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "bfad18f09a89357be28f798860034557a312e06c024ee3a9bdfedc8ca662f4e3", "started_sha256": "6849606047a54e382902121d45cbe54f6012b0ab51383d48ec0c8aeef6b5fd7c", "original_grade_sha256": "29e38dba4dc8e0a62bff079d049d169f71fc0e2204f608b18bc2f14dc4e530f0", "input_original_sha256": "bfad18f09a89357be28f798860034557a312e06c024ee3a9bdfedc8ca662f4e3", "input_wav_sha256": "9be4fc171076cc48bbf7b7294ec2c9dd0fde290822e7332c7d5b50eba5b849a6", "input_pcm_sha256": "f9e7efb5a4b11fa89316e391547c5619bc7471c71be930b074c3759e8f60a476", "output_wav_sha256": "3517f39cc360000d496897c7d9e0c0f63d2e94708d39cd3be010f7a503803835", "output_pcm_sha256": "d722469fe2d1f65b565be9b5d50369210ed8da269a27e046b7467c006692f82d", "result_sha256": "4e89f762a0d0e7e413e05d91fecd25f070d5486e39f00c0c1636a162a95d1865" }
#138formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "382c7723cf624c1cb667cbacac61a9896ce8f512106063d5911731c8f4c9721e", "started_sha256": "ea4048790f4fd823a46747bcfef105c077123a9025f480c6c97fc3b4989c2ffe", "original_grade_sha256": "f176d07418f08ab25ce82ecd023f433ef19ea39c40b39dddf1ffa49f4dec9d94", "input_original_sha256": "382c7723cf624c1cb667cbacac61a9896ce8f512106063d5911731c8f4c9721e", "input_wav_sha256": "23e725ca8a77d69a911031b3970f45b67136212201976ac1e838b925eec17607", "input_pcm_sha256": "5412ceb8f230e8a7df73ec238e5dd6498fd92c2058fc6e8d1efe65cb73197403", "output_wav_sha256": "28c4215fded737d888254153a16a36ca8548d814fc12cc56d59c39777f3378bf", "output_pcm_sha256": "bc624fb0d905fbaaacbcd818148cd3a084b8cca281cff009e327859f108216d5", "result_sha256": "20b1ec048ee4b497e7a04c7a67c138fd315e1e3daf5a748de2e5d1b70c2c6ab2" }
#259navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "0675720773e7bda992f9beaed0fb1e72d36439774c45df680c88a643440d856f", "started_sha256": "2acce55e172ce5118b9e37acd5659e0b7a718150cbed78fc1e0c90bc090465fe", "original_grade_sha256": "5373dea70c296ee917239b6132ec27f82092e2dfe46169352f52651b30bb7200", "input_original_sha256": "0675720773e7bda992f9beaed0fb1e72d36439774c45df680c88a643440d856f", "input_wav_sha256": "2d0937ad24838821d1638d4b68eed1fb5e868e3fa0a6b691d97fafab3951a5b3", "input_pcm_sha256": "f0d4694a509912a996b19bad15ff945562158894cfe6df586c84145950e0f0b5", "output_wav_sha256": "3116c6f230ce2aab02c6a6d1cd10109e642cd21557275b3e4ae830aee1d65975", "output_pcm_sha256": "5543437797d7c3b3d2b134aa4021ce7e2835a22e6dfe2b15b204891586877688", "result_sha256": "b340800dd0eb90bf1ecc899a41f5397e536c6f3aa08ac3a40258bbe15788f814" }
#561object_counting heldout	3	Date answer is 3.	needs reviewnot_an_unambiguous_short_answer	correctresult	Additive review The independently recognized received audio reads 'Date answer is 3.'; the model's reply was 'The answer is 3.'. The substitution is confined to the fixed lead-in phrase ("The answer is"), which the pinned small.en output recognizer renders as a similar-sounding opener; the answer token itself is intact and equals the pinned key '3'. The scorer withheld an automatic pass only because the spoken form was not the contracted phrase. Same class as the determiner substitutions reviewed on 2026-09-15. This is saved audio/ASR evidence review, not a claim of human listening. DAS test duty independent ASR/evidence review · output_audio_and_ground_truth Evidence hashes { "original_audio_sha256": "41f7f58b0c67270734480159aa9415f8c3c1f5fd4b1d81d3bc3d81de8ba8521a", "started_sha256": "c06da339daf8064da4ea539b70b4de4e70ef430ec853507cbd763ac23de45d3d", "original_grade_sha256": "19de9307d22b6639d5e2979f1806d12d7b2847e1d6aaf0f23f8b43cc87b46a4a", "input_original_sha256": "41f7f58b0c67270734480159aa9415f8c3c1f5fd4b1d81d3bc3d81de8ba8521a", "input_wav_sha256": "bd9cd5df8d2a2d7eaab307af44ced2207bcea8d2a53700ab840e84a7c6daa216", "input_pcm_sha256": "94e2a78c5204339772cefd62380822e4ff26bcfc0913286be027d91e4d349cd0", "output_wav_sha256": "a70c1002f7e6729b4a8bc06abf01c0b912ebdae59570a04ea17ee7f4a00b552f", "output_pcm_sha256": "ff5d24c1a8b7fee7d6e16ea6168c64c3dc71a89af82559c23195fd354b73a702", "result_sha256": "18656cdb86c51a88d009242047e98be3006e6b15b0ec1f95667f20f316771e7c" }
#754web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "2534218579f2ca248c5cfa0a30b1e21981e085c665c39c2561a0c36b7f74be7a", "started_sha256": "5f202278df1d1067fac7c1f09fc43e99a6d99fe493847ed326b4c13d25b4cbe3", "original_grade_sha256": "71dc46187e016f318c05bd9d430c13721bdf6fdd1d2f4a2e75576ea73ac32161", "input_original_sha256": "2534218579f2ca248c5cfa0a30b1e21981e085c665c39c2561a0c36b7f74be7a", "input_wav_sha256": "afa0d187c4f9795da0a9cd10ddb7022277eeb501a754df34427b65f381a25f9c", "input_pcm_sha256": "26327328f665c2303a6eb8eeef0340fcbe74951916ef7ee07f9d5a5a1961fcac", "output_wav_sha256": "f39a970dc2ac6ef12f35a1a3f057fc69d45c16df3e45eb065eb8b719d01aea85", "output_pcm_sha256": "8d1a8cbdd907faea2e233582a9931dc093ba362cef8a8de1829a40ca0a107203", "result_sha256": "a87c1dce49e54912bd439f4c02c8baa4e5c084937cb3c94f2ae617d03492d1d0" }
#191formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "0dbaa933642bfbd0eb49b1b914097fc275642b29ddc1b827cc178a9cc25f8d16", "started_sha256": "6d502c085058d9e05e19cf46c4eb1373ed74f64a8abac5a04c78f7c0d412957b", "original_grade_sha256": "6b5a1f390528c8753c73904089989d8c83843e7d75b893573add0478bc476993", "input_original_sha256": "0dbaa933642bfbd0eb49b1b914097fc275642b29ddc1b827cc178a9cc25f8d16", "input_wav_sha256": "5384b101c17bab407a936554d0a3ba4de869eb5a901c2e6eea08e06f03e12d49", "input_pcm_sha256": "1a9a56ce3598009a3af6c28634cf91e71d3d7ada224c3f636268a354f101c768", "output_wav_sha256": "549e9cacc6d2cd3a22b67c5e60e215a245241605f237846839b790224666a39d", "output_pcm_sha256": "b227d4cf0f0329e98102171bf09f283b3fea619cc377d7bb6031a8a1c89a9296", "result_sha256": "0dd981fd13a405d11d1502c29241a6b852dbcfc76d0a57a6809b5fa949543a32" }
#498navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "d0c40167acdb5a436b1bbde90ed98a25de2d1d49eab28b8659bfd4b4a4fcd2e4", "started_sha256": "0bd09e52bbfaf4035968fb520e1ab61ee8e0e3bc2f8d4f18b60d7a0cab7f73f3", "original_grade_sha256": "eae73ce46b579213fac6cd1022a8ee88d68efe493e1612c7fb5bec84dbf6d3fa", "input_original_sha256": "d0c40167acdb5a436b1bbde90ed98a25de2d1d49eab28b8659bfd4b4a4fcd2e4", "input_wav_sha256": "7d15905592fc7b7c95415ec175b188964bf68dfef6f69dcb62c8c170729daed2", "input_pcm_sha256": "7af8a2363d98b9a52784ecb3498043db007c5e8e3ad91ae4cf014c7bf57998f1", "output_wav_sha256": "d1d59b6515a6e873acc9f9ea97c061878d8f0756ca62418dc1f53a5347cfb6dd", "output_pcm_sha256": "f4c99ef1821046f7411a3daf9354faa618c0831f3c8eaaa2cdb070ca0a855f31", "result_sha256": "118595d37d39813c2a463fcef44d53c49776518057f65ccc2424eaf5670df70e" }
#577object_counting heldout	7	The answer is 7.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f5c0a9ee3ca58c48065a6f9d8abe48ff3bd54a34da886a759c9f74130e85f34f", "started_sha256": "c88584b273feffd06ccdceb13f2eb871ddf18671a5c1bbc670781498a3c715ea", "original_grade_sha256": "01acf4ce624096891e33c07f8c9f53b32fc1a451488ef355b3f2d8419fcfde20", "input_original_sha256": "f5c0a9ee3ca58c48065a6f9d8abe48ff3bd54a34da886a759c9f74130e85f34f", "input_wav_sha256": "0f606267f144fe548bb67b792eef1379a89f6a54126be4a7a139567e2c3d5621", "input_pcm_sha256": "a71cce49276670bff2c0ad2fd0396d702bb7b8e1ad2d69746bfdabeb0cf774be", "output_wav_sha256": "b4a26a14eeeef0c636434238e401aa0fc122ae035d4f92bb3eb965668da86555", "output_pcm_sha256": "e2c732227ceac7ad94e44ccf46ea765090ff6d630d3398aff43e4a922813a6ab", "result_sha256": "8835f8c400ebf1efbf6dec7fdced7d2ff4b1d21161c576067fb97f98fd271829" }
#764web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "db66442e31f072869ed6a5f1a5838f687c729a297f6b112a76a89d1d60968152", "started_sha256": "950e2dacdfb8754435a8182442b31a684b27de09ac156038716f9fa549216eca", "original_grade_sha256": "7009b0f45a48aeb1c71b771bfa914ed6bf47f3acfd4dfcee62cc3931ad2da227", "input_original_sha256": "db66442e31f072869ed6a5f1a5838f687c729a297f6b112a76a89d1d60968152", "input_wav_sha256": "d2f806efcc11ab7e453d56d39a5f0a13a63f0e4f15abc2e1dc2836f190941802", "input_pcm_sha256": "2404281b4333d1d772897d1f5f7a8f10cdee98b2600c011a688f1f28adeaf2e7", "output_wav_sha256": "6979367caecee52d1a75535f189bda728a27aafedb04faeb416e8948d1f032e2", "output_pcm_sha256": "5475fb25089d6affe8b4453e30282e5cd06caf20d60a67aa95e6c0b96f3cf442", "result_sha256": "111089c72b2498f8f780387bc03b272c8a020a7cde9d0f2251a8a758a7371054" }
#213formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "e95f5cb9b08a2a9741a17aaa36a305816556925d707d2a2d48df70a93a790594", "started_sha256": "3ead67d10f20f40bbbed150b7cbccc20581e65695f956384da5eac3ea5877fd8", "original_grade_sha256": "20f91da37fd7739d916dbf0ab54ac67a9fbd335f08087dd23e1b7e7aac167a87", "input_original_sha256": "e95f5cb9b08a2a9741a17aaa36a305816556925d707d2a2d48df70a93a790594", "input_wav_sha256": "b38315aae9914d3dede87fdd00f8654b374f9cc58421e113beb2ac8a6a46a5b2", "input_pcm_sha256": "9228b611f3b8681f80ae237cc810dcd9662ba9188bbf5efdaad3b1581e50d314", "output_wav_sha256": "c6cdea64e0d06d3184fd81e6e0a602d9da9ffc6baf058b10c6b846f8853ae7bb", "output_pcm_sha256": "d8374c81f7315ba7057bd742bc2811eab9facd2eb67b9be2f74718b46dca07a4", "result_sha256": "ace393dfe95790e66b0212052dc20a8e515a574e894ff32bbe0b9228bcb65087" }
#350navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "640e30805da6b6f913a52d2419ea2b3cb879e56a888994cbf9eace36a1fe0534", "started_sha256": "ba5404d427636caca88168d925c2024698b18e4ecf02a520de11a9377fb253b6", "original_grade_sha256": "0f406388b982b899dfb3c32e8648484394998d662307f463012514609b3480b7", "input_original_sha256": "640e30805da6b6f913a52d2419ea2b3cb879e56a888994cbf9eace36a1fe0534", "input_wav_sha256": "05b0014ed9c37593537c5403ca460ab94be30cb095331d092b00167970533f21", "input_pcm_sha256": "3a58df1f8b20f2c94350533f3eea0f5bcb063752a59e7da7d72ffa2797829c19", "output_wav_sha256": "22928776b6e27fa4a2e992edb4ae51c17be17a61e8e0e5ef9beafb51b52b7ae9", "output_pcm_sha256": "b6e8de0a3d711b70a386ccc96576264d7ee5d9c5f071c8edb102195cdf6edb17", "result_sha256": "c06159f8e3e32c227761a9701e03c03a6c00e1f0adc048a30eb2dad7ab58babf" }
#735object_counting heldout	11	The answer is 11.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "7f10069069c2a48605d97e53328d5f2528c9c8afd3bfa4ac01b2f36e099ee4db", "started_sha256": "66505fb5c91f15972b5de46b5f85fb31accaeaa3df111fd9636054d73bb55193", "original_grade_sha256": "16d2d149ae7576ad46e9152a58e1fbd9c49f31ea649519e2aca375badda4a3d4", "input_original_sha256": "7f10069069c2a48605d97e53328d5f2528c9c8afd3bfa4ac01b2f36e099ee4db", "input_wav_sha256": "ed99a87a5416ddbbf892a4d7e1dddff99316ecff0424db60d0e765dab43cef42", "input_pcm_sha256": "d5c6b9b3826aec10cf32ad3f87e67de74698e042693d7f7618b98a3e850fb45d", "output_wav_sha256": "7379a61a5b8b3e739ea042ad265801957e401f116c96a7d34133e72d4ccc5430", "output_pcm_sha256": "253693fa87f1aa9f667b7ae13f0470e429935c0bd77f9778d7ffba253b7522e3", "result_sha256": "502ce51664be8a45c8554514b805e73f195a7b657eba52f7fb7f33e367a201e1" }
#921web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "bcf3bc3ec52bf57370944ea930487995da4adff30e8a6994f9c1ca6640b5050a", "started_sha256": "034249f880b92451a23086a11ddebb5f91ea0f414e07d4dd8e902e74a12538c0", "original_grade_sha256": "592662b8de75ae9e79d5eca9ff3ce5fe9f48538d1d720a7d7d594683d48e69f3", "input_original_sha256": "bcf3bc3ec52bf57370944ea930487995da4adff30e8a6994f9c1ca6640b5050a", "input_wav_sha256": "f2bdb3efc25596ea86d7f0375facedb3407d6329095ec3513bff3d0bbb27a995", "input_pcm_sha256": "b4698cc0cb6e893c76de53386d06c2a1cf44b5a7591da310bbc5671a4f722b9f", "output_wav_sha256": "eab887e1eb3039f668080d368667f0120efeafd891f3a2f8106dd667f67c6bb9", "output_pcm_sha256": "1aaa47e66282d3395de3662ca97f4c82b142fd590008c1880c1184cb9c311ada", "result_sha256": "4cd10ee11f515eb227e3f315222214f7af49480c75b2d14e65b1a841112189d0" }
#68formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "1f7df61b956759067cce81c5423bba986035448db2fc0a0fd727163f197639f7", "started_sha256": "a1a370b14ab46c47be726eb49aa58a0c0aa776b334ffa306190524d6ed3e8cce", "original_grade_sha256": "7f73ee2f25db2cd1f2ce28ae686dddf69ed6ebb6d017b584e5b218323e20eaa0", "input_original_sha256": "1f7df61b956759067cce81c5423bba986035448db2fc0a0fd727163f197639f7", "input_wav_sha256": "dbfa27ab01155b7967a3374ec1bc443f75be7371e234a3e5856cac1a0d769dbf", "input_pcm_sha256": "362c7e8018fb51b45f6f2ac2bd4aa7c90d85bdbf37e986276fd604e5150ed11e", "output_wav_sha256": "79d4a60e10d9cd290afb5e151bde4b39ab2ff975f4e565b2470871663efcfec1", "output_pcm_sha256": "502c27ecfa3bf1b05fbfdb8a37e21f237fa757e6eaef7ed025d83a0c1db72906", "result_sha256": "5ddad0e6a76cc45cc02c8449d74b44ba5ac2801d07433868aa1afd591035ad5b" }
#289navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "72fb2d3774c77f288696a5dbd6c4d8a8e91d418037da9ce70c9263d4aa287dae", "started_sha256": "ba616064adfe9ec5890d8a9d9c9f472a185a6dddbc288b15ee1a05fdd8ecc1ee", "original_grade_sha256": "39b1a8aadc69da12260c84e0f237e9eb00297bb30651f69cb9386fc184198921", "input_original_sha256": "72fb2d3774c77f288696a5dbd6c4d8a8e91d418037da9ce70c9263d4aa287dae", "input_wav_sha256": "83e20ee72269a1cab0d3618fe3885c368edcf25d26f0ae7e05d7634c24015009", "input_pcm_sha256": "5992fcbb67e1ef0bd850facc54ec5b7474d21278ab984e0b7c6922aa72b77d2b", "output_wav_sha256": "f57f77e5b9df17f06b322942651e8f5f30b1f5a4f154ef9ff81f5e8080cc7013", "output_pcm_sha256": "3d54c2d83526916edde53d3d38121c16195dc68734015dadcebb3a91fede0b85", "result_sha256": "adae9b49ac11fd919a7422128f28c508d1e58eea7b2640df2de8f3d3cec368a0" }
#604object_counting heldout	8	The answer is 8.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f14cb314399445fea080ba9480c0f73be306cf91cff26a78a33d77b04647d75c", "started_sha256": "2f8ece6a0053327526bf6db479623ba20c25eef94678333c0f0ff06b635fb662", "original_grade_sha256": "0ac60bfcbba1b91fdeec0cc550e5e2e316221b516bec77b86b857ab2766fafb7", "input_original_sha256": "f14cb314399445fea080ba9480c0f73be306cf91cff26a78a33d77b04647d75c", "input_wav_sha256": "d22709011a2711c49181c0d26417552dcebdd6921bb722d7cbdf2ce438a5de9b", "input_pcm_sha256": "954ee0408b5e05cc23eab3df07beb5e85c1663fe64628b1bf56e414be217cb71", "output_wav_sha256": "8945d52ea62b6ef5ce41b806fef5faae3e7c50180b7a53b4326d896a1132ff5c", "output_pcm_sha256": "243cfefab54a2191e1ee403465423bc719d61501f8dd4fe9d6864a9686f57861", "result_sha256": "e3da64008411c86e149cf1a296ef34483747f69db012a53f689a753b260a976b" }
#894web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "9be51bf39d9cd264228afe6bef8d95a3a8cccd8f572e5bf3ae9c620b97099694", "started_sha256": "b8f4be4bb24f9121ba52644e6fff7862d06d3b32d64a847bde7aa1366d5e97cd", "original_grade_sha256": "f08aa70d361e3e764ca363a88909ed1dc86bb338c2651ccf6896feca3ecfad9d", "input_original_sha256": "9be51bf39d9cd264228afe6bef8d95a3a8cccd8f572e5bf3ae9c620b97099694", "input_wav_sha256": "47cdcf3c91e1c27bcc95797c729054768649d884cb545fca49be5971f5959fe2", "input_pcm_sha256": "247d6e83eb985da431b517d232d191671eafa7cbd8c95eb0e2138031dcd4732b", "output_wav_sha256": "424d57147b28a21d0871d8e989ffc28f9c4a6e934ea76cc5b2bf17eb13700720", "output_pcm_sha256": "e96b6116927d1ab0c74e969f04627d5ebf511f17227cd38a223d7c56a98d8d5f", "result_sha256": "75582f24d294c3bca4b48f3c32f6cf875e764e95e13721d4f5573517bcb0781e" }
#188formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "58c1b7049c3d040c07282a5db12dbcf0d8fbf0145316d762294b70939530dea0", "started_sha256": "2ec77758f8ada90b71a43a67a7e19b54ee9e0c4c2a40e64d86b564625ec1210b", "original_grade_sha256": "98ed53b000ad35692a3606194546d450bd885fea168e1202e2adfe2fcde61a8c", "input_original_sha256": "58c1b7049c3d040c07282a5db12dbcf0d8fbf0145316d762294b70939530dea0", "input_wav_sha256": "de5906f39d01ea6f3a57bec1df53b172daf05e6e7636c80697a93620da4906ae", "input_pcm_sha256": "429ca978da805111af6d1756452385c114d1a82688dae544754e6e0e2335de0b", "output_wav_sha256": "01d739f9828cd541cefbed22511be9fb268cff095e9193e4e39e652d805e0636", "output_pcm_sha256": "2bfeb8e94f9c08aaf0cc947750d504ae5aff5eebe46ae0e57970a9432c8c594d", "result_sha256": "a4045592cec96c00d2a78b026904ad80a7d79611d7ec243c177d57c7bab54340" }
#481navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f6cca86be13ad88ba91969d956cd4c3048aa969cc83dacf52f2ca7a3009768b3", "started_sha256": "11ec7769e9b4f86977c4e31f3a14d73619db3771035d483fe490ff193262c05b", "original_grade_sha256": "d55a8b4f54cb900364d8d88fc59bec85901360b00c14796e67900a0a82047b71", "input_original_sha256": "f6cca86be13ad88ba91969d956cd4c3048aa969cc83dacf52f2ca7a3009768b3", "input_wav_sha256": "d7230c8717db8edc79f5509b52a27eb01f530c71214d69dc208dec4f982ce6b9", "input_pcm_sha256": "175fb289aee0f9766f4d9dea523a1df5c4daa2727291865241828159eab54531", "output_wav_sha256": "d99e7b5bf87f190afebaae45c472043df169c6241d9a107e4b2dc500d2e69fd7", "output_pcm_sha256": "cd042f86e5ef7b8d872b5adfb43b0c847d3afe1a90387c691a4303214e4a3aae", "result_sha256": "2b36f3d7e0be3da543ea3b04135b5c5241c3fe81fb0af195ee3f40d5d149c426" }
#631object_counting heldout	2	The answer is 2.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "ca0398e4bbbf11c21bcdef7eca43b1d16ec7baf489718189ba52e9ae91e30e1f", "started_sha256": "db2503abe3d4a72587d37789e3db79c0171ea52f011990d0fa3fba0aadae7dfa", "original_grade_sha256": "17239fa5939ef968b8f18cb825492ed36202ed0a85914eaf6d9e6f1b293f2882", "input_original_sha256": "ca0398e4bbbf11c21bcdef7eca43b1d16ec7baf489718189ba52e9ae91e30e1f", "input_wav_sha256": "ed09202a1426b13bbef759a6b16f5b07c59f95ded1d244f90a8ecbacd54ccc55", "input_pcm_sha256": "d9138ef4c705d8e87c5379a13adb200681a1a2ab05b1865608afed88eadd1839", "output_wav_sha256": "017963ad2c3d3ca23fdeadd11ce51014751b6e68fa2592f75d9d407ca8edab6f", "output_pcm_sha256": "16a2ac693a1cd825f0f57d138b15ead024bde954831a078bda527bbf86b76bde", "result_sha256": "0e96d2eb8e3b53f5e167620f933be9609f8de777a549a7441c5f727ab6ad785c" }
#777web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "692e1bc31b018f8f11e0ff50ea7198eb36f67f1aea1202e988b4afa8836bd7a6", "started_sha256": "8688bfee7237c8193ac3f65839d9e7558b9c225f833f7421496d51ad277dd43f", "original_grade_sha256": "8ce9fdc73c631307478ef1f81194d2c4204a7d9763d0e0a14e04699c52d7a8d9", "input_original_sha256": "692e1bc31b018f8f11e0ff50ea7198eb36f67f1aea1202e988b4afa8836bd7a6", "input_wav_sha256": "3467c78a1398e6cfe0b560c32136182e7f145320a2bf30956eac1390ef6444ed", "input_pcm_sha256": "ae096ccb223cbb2d36a0c638c25fb3c628b692a1fbf6985b1a253673ba620fd8", "output_wav_sha256": "dc7cc3c81d508704521815fdb2721fed82d1fea416ab004e179c51761369b7a3", "output_pcm_sha256": "8cba8f62c31f48147b9597581baa440e4b07d524c8e636c124b0f98e4ebe1a40", "result_sha256": "bc83645d226be9cde151e35b3d7dce519435179532278defb3a7164142bd708f" }
#173formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "2ab6d3feb108d77ba8468b124025b71b97d51aa260af4d821d4942cd23a218b7", "started_sha256": "8610bf1066844ef17f4fe0f83f0d194e2406f2aba44aec29293f74dcc639bddd", "original_grade_sha256": "81dede88321a759ef5181a67e4002a6a4d4e8c291af932bf426eef5baff61be7", "input_original_sha256": "2ab6d3feb108d77ba8468b124025b71b97d51aa260af4d821d4942cd23a218b7", "input_wav_sha256": "0b8fb4b3f308a2c74617b5f6f1c387c46d556b1b0d71d9be2fa93c01dad66e78", "input_pcm_sha256": "834b51107744d4617faff8e90df7908b25bf84475795662f8f5f3996c9741f00", "output_wav_sha256": "4f04d6859be17af138df338928e94f7f3f3d4c3cbba7408bd90a88d35a92bf25", "output_pcm_sha256": "7b46eb6ede2324ea7ab225a142712a419ea5ab6ea960234641f973ebcaea896c", "result_sha256": "1ebf07efdf750239aed3dfc5b46faa2bc88cb14678efd4b32de2c6a062412f03" }
#440navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "d0ecf5e61da746acb97cb4a53166061ed3f6663f21a51b8c22303cf54a901648", "started_sha256": "34be0b75912b8cfa202ae083f16f196f55116cfc0413d708459b8c03730c888d", "original_grade_sha256": "dd037e233fa8522d55b63ac4f1ce7fa99d5f48e0ce30f04f8fc08a1be32e007c", "input_original_sha256": "d0ecf5e61da746acb97cb4a53166061ed3f6663f21a51b8c22303cf54a901648", "input_wav_sha256": "5cc09924e2b47a11cdb661afd14446d471715134f6391f79ed868f29039451de", "input_pcm_sha256": "822cf179ecf699d6f9ddedab0567f84bf0a1c47e29c7be6788ae495df2dadb61", "output_wav_sha256": "66686bde15f236305683b9e797be322878167dc29c0dd7a9699d83a09d0b214e", "output_pcm_sha256": "b1679c65cdf813e99e80fc2ae791927ceb138fca82a9034c5ec1df2fc575dcc5", "result_sha256": "49e9efcd6b3ed5b04f53d9db78dc6e3866315df78865418855e827772c56f531" }
#720object_counting heldout	8	The answer is 8.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "bd9ebab90b1b9227be0743666dd3cef855d42a512f47e72605d17a7a1022fd40", "started_sha256": "f0c85a860dac4dd63253d3c892215a42dfd75568a7fe3fdf3333bc41831c3575", "original_grade_sha256": "c0d897f29e8b1fa1f84d47b15f86a049af102c41c1927a0c1ee93341929f98bd", "input_original_sha256": "bd9ebab90b1b9227be0743666dd3cef855d42a512f47e72605d17a7a1022fd40", "input_wav_sha256": "8a79368875e93a5d3a9a120a710004437aaef448ae507ff4bb25b038484af78b", "input_pcm_sha256": "adde8bc8d966ec1900425291d170ddb6ae2043d46ea313d461dfa8e9b7be5c46", "output_wav_sha256": "f025ff950609771c77da7766488a18e977be7c1d11e80817113c65d04a9fbc61", "output_pcm_sha256": "4565308f5cb2433bf9b0f6fc1bd310fc28aff32a6e1a010a069be9d468e92085", "result_sha256": "ce3eff1638f57ff62324ef8c4440558b1fbc058b052d66f235585fcaf888015c" }
#766web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "e003048d2fdf09e36dce672744438428b1f82b0a8554f4b0aab87d766ce47197", "started_sha256": "4d5e7c8b03c6157b8037b5539f92fa94132087097a9764ed394eba0f0da6bc5b", "original_grade_sha256": "3a587272cbdd3da94a298a68ec26f7906c99235eae2d85607c80ced988165e0b", "input_original_sha256": "e003048d2fdf09e36dce672744438428b1f82b0a8554f4b0aab87d766ce47197", "input_wav_sha256": "3ef49c16a3635e75ca87a09cdc1df78b865fe51af45ebedad07ec4bde21a3d7d", "input_pcm_sha256": "bfa09c25d447482a4fbc1f2ea882484afbe630f9ae52b5dd972a7c28b010da88", "output_wav_sha256": "59d638c10d1bbd2cb6a03d2bb7add741244f5c8a2c7f1cacc190cb8cfc705e36", "output_pcm_sha256": "0432e3dc0b9c9a9369d2663fb20a71eb25307cb20c9ef1ba70d4875e1539ff03", "result_sha256": "3ff1ac3204383a8e5134208b272187fd053ecd9f9cb8e2548568d4b74c2c85a8" }
#46formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "0a780422d36ed5e896a583f1cdce7e9e71bc9a28fd7452f2cc4ba5b21bfcd045", "started_sha256": "9bb1fe367b7f1a344d6f725b7726b32c659f94750cfc7814b884f43b119713a7", "original_grade_sha256": "71dc7f4852f7a16b94ae45f87976a964872c24ee388b537cba3539f641cd888d", "input_original_sha256": "0a780422d36ed5e896a583f1cdce7e9e71bc9a28fd7452f2cc4ba5b21bfcd045", "input_wav_sha256": "dbf47f341715e938152f15103044ead748732aa0f54ab7d7073343bb5b988135", "input_pcm_sha256": "01d1c5ee00694e31abfdd72d249ab4bed889364af391ef8aec816c5850a4cca4", "output_wav_sha256": "22ee6780e980ea148f193e01081630d133a3b731014b0e21d514a2d588ea8308", "output_pcm_sha256": "32ea27dece90e14eef1e5f2d79931d95cebaaeb668e8e657da252af3c98e459d", "result_sha256": "4bad3e6d7a8147a0ab53a325ef32532c2a14a4aae7900508ac3756be1292e56a" }
#474navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "1e4c6d60f4a71ad91112370c9023b28ea45b867001e7b5e037bac987aa679bcc", "started_sha256": "ce725608159698429453105ef258e28623e6ed62813feb6d1fdb462583269ee0", "original_grade_sha256": "6e42088efea14544b4d7349c10f95f5dacccc85528907e77770004f52ea0a512", "input_original_sha256": "1e4c6d60f4a71ad91112370c9023b28ea45b867001e7b5e037bac987aa679bcc", "input_wav_sha256": "6753d7d2fe539976a7be60760a87f3e5968ae4a8087c93b1ea4e08340182a4a7", "input_pcm_sha256": "737115d6aa61bee9d3e7fdcbf4b5534b58f306a26ce3f0f4cb94544979016793", "output_wav_sha256": "62adb30ca132a516d0766b2d203d50cf72df3115ed164b48ced81c350bb76b1a", "output_pcm_sha256": "d4a2ac93f20d3ea8ce63e3d9dbf8102e6b70660b47dc146815c5c3762a7b6ffc", "result_sha256": "f292fc2f328190ff3345a64cb7bbdbf9b806dfe70a56f0bd6dcea45c197fea83" }
#695object_counting heldout	15	The answer is 15.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "281a7ec27fc5b573f7fdcde01fe7694b99d42556c575f42b3cc638d1bc30d634", "started_sha256": "35b578f81c1f9b11e25ea031c9e826974df21f45229c5d01c15bb4c12791d1d8", "original_grade_sha256": "1c144eb113705a2d8ca178f3bb1440c9819d8215366ab91e04cb8a81f3c4fcac", "input_original_sha256": "281a7ec27fc5b573f7fdcde01fe7694b99d42556c575f42b3cc638d1bc30d634", "input_wav_sha256": "448b862a9f71ebbc385df3dafda535829f0efdd5f0a0fec03541557830ccf1a3", "input_pcm_sha256": "c335e328058a263cb151bc5ae5ec9a6c1950f3cd6d3db8ab9ab695fdecd05d4b", "output_wav_sha256": "55a42aba565ecd196cf4300a1c09ca5edaf853cadf3060607535b7c426f5502f", "output_pcm_sha256": "72dc99bd87b4563b18c32b11761c8ec773111fc622a30ac902e828cb9d2f6c2f", "result_sha256": "7347030567c1bfd2809203fa40c402057751e77b3786fd0de9884d12afabe129" }
#793web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "15915d567f216f6ce2aa67864c68b170c58044ca0301245819547c7426cab71c", "started_sha256": "cfa33175b0b84e83fe3b67b8165b49030e664873bed562de1352687ce6d63b39", "original_grade_sha256": "15903f2ad50ea86883feb9cc63b38f5843cb8a204bef57c20508460ef0683d6f", "input_original_sha256": "15915d567f216f6ce2aa67864c68b170c58044ca0301245819547c7426cab71c", "input_wav_sha256": "48e2489643226057ac98841977b8a499fda3bad1d7eb0c54dbb1af3e4aaeecfe", "input_pcm_sha256": "feaca0fa097ef3615e80d38a35f8c1ec45ecb3cc2deed7b42ced5859818ae672", "output_wav_sha256": "b95fea6dfdef8648d9f246c8667128d15403feaee19561637fa1ba2df0c4f0fd", "output_pcm_sha256": "8814254d7675afab81f34d06ff3f7b40521eefd846d3a91cdc9a09e3f0659420", "result_sha256": "6eeb274d02b0bca4590111f1e48c2dc153f9b34b3d532e0b91e0642c3dc9c3f1" }
#137formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "d7f86783a163ca240e09ed92482e7d65d31dc1520af95a4de1a44e1cf3b4ebc2", "started_sha256": "63a60ce4303c5f59cfc5c2143ba4f6ae86978a61e909d2572df233550da69ef8", "original_grade_sha256": "bb0aad29ba080c73327668d75d5275d315f58c0636cb7dbd0e96cc7c7fe4d14c", "input_original_sha256": "d7f86783a163ca240e09ed92482e7d65d31dc1520af95a4de1a44e1cf3b4ebc2", "input_wav_sha256": "66abb3330cd4525fac338372ca581c694a5863eff960d13aad633a357960ef5d", "input_pcm_sha256": "df06d8473c7858d9009dd449be8907fb94617924bf05a181af44c7a40abd0695", "output_wav_sha256": "6e8ddacd15f93092172fd6ff6b61f0291562e1637cd3d92844bfe0e320a2d658", "output_pcm_sha256": "3e6af0be73e9046b12d3fda4ba553f4f8349b0cb35a5e9b355efaeb320248010", "result_sha256": "87b25d2f0ba5412843102d6e794e3f8776e3910579f30ace9696b638b0475a8f" }
#452navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "8be78ab5b5f413b1132450e66e90d3eb57f0ee3f639887c263399cac97a8270d", "started_sha256": "bf1d167768c26c64c93d1aa9dabea25eaf93b1b8496f7c3bd1e23ff04c2c8e5f", "original_grade_sha256": "f3aa12820f7a9c867c93c7607904dec949b2fe77ef4b17f1ce3c90e8e8ae1cf6", "input_original_sha256": "8be78ab5b5f413b1132450e66e90d3eb57f0ee3f639887c263399cac97a8270d", "input_wav_sha256": "2aa529631e08a6b434afd63a62defcbd1fd80eb8130ecef88f535f4cde08a198", "input_pcm_sha256": "f807dc5e42f0c1cfde2a7b759e5b21eb16d6b5d0f858bd848e5ee789ece8162f", "output_wav_sha256": "a35f0467eb1b4cb8303bebc069c11ab3e4b68a61a99898145a9ca606f72a2384", "output_pcm_sha256": "14985490ec811c608bf1aae30f133333b7e3703c4f190650719ad735fd01d33e", "result_sha256": "b20716cd749da5ca02c790b3006ebe843298bb898e637595cdcd82dc03ad76cb" }
#537object_counting heldout	6	The answer is 6.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "cb96428ee6829cc6aea6d59931f8d9e6219ec0dd0bbeffe8b680113354e52e64", "started_sha256": "7a592bb5555a5dbad1e991ec05d84edea067ee0ebee693b10ee21e1a45c8a35c", "original_grade_sha256": "3cac4043d016f713ec51c3c996f6d44fa8f2521108b40f18b4801e6077898ad6", "input_original_sha256": "cb96428ee6829cc6aea6d59931f8d9e6219ec0dd0bbeffe8b680113354e52e64", "input_wav_sha256": "f0427d4a37fff84ce3fe48c337e08667574a3fd7bf47b13247b8ccd583c3e2ed", "input_pcm_sha256": "42ef466582093b14f74d1372783be38c1f1b09345153fb194d83a2a6328977ed", "output_wav_sha256": "5f9f8c893b26ea7e77ffe79535f341103b08f89c9a343ca744f5f1eb113e403d", "output_pcm_sha256": "bd0451ad7cad87c4bf87d91a3f705d802661dd24de9089ced6a222f370643fde", "result_sha256": "8219bdc4ecd89d4f3f8af7a3ebbd89f1987e22e16b5238bb9cc50919aaff5beb" }
#861web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "e640e5f26f2eff31bc6b1067a009b744277dc6c9de2107a1596dbaeb23873ce1", "started_sha256": "515e525b2f1a60f338c6f6a5e75d64171b4fb88a2c1f26e103107f4ef2cbeb0c", "original_grade_sha256": "a9b4c34e7dc4848868a1f7a04455a76d50a8379dfef1f03ae2500a3e1be3d470", "input_original_sha256": "e640e5f26f2eff31bc6b1067a009b744277dc6c9de2107a1596dbaeb23873ce1", "input_wav_sha256": "94d11b3599f4e4c078b50f0710bba664ca9966a84cd0a1146ead8f57813612ee", "input_pcm_sha256": "56585e000c40246b842f8b8e350430252cd92da1c9c07b6cb704580a7e06dd46", "output_wav_sha256": "860572f241b5c0dbb0d734ae80cee3ceacea8829333e10ae8c8c8f0cfdcc9d9d", "output_pcm_sha256": "356eed4d86869b046ac393de381f96141458e3674c1f444a98a53df3d5d343bc", "result_sha256": "b45f3ab98e4770b6f528da4eb0ae577d18590bddf8c38818c6a99e9d1babef07" }
#146formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "3189e7fc5ac0929c3a34752311305da541f812124f7facdbd774bc181d56013a", "started_sha256": "a2a5104adf8da83f869cf8e6b4b4cd84610a3e665c0cdd816cf30bcc0a5d5e19", "original_grade_sha256": "8565469456d273988b151cc92fd5b7af6ceafe5386dcab0579412e57f4f482cf", "input_original_sha256": "3189e7fc5ac0929c3a34752311305da541f812124f7facdbd774bc181d56013a", "input_wav_sha256": "a647d9237cb28be174221fd0af33755a42f53a4e68769197afa7be777c722c6a", "input_pcm_sha256": "94d537ea34f330c9251cb6ea6597c4428b4158d816888659cb2c399d9ed1186b", "output_wav_sha256": "94be67f3cd63f93ae206fa80bb37903e9e3b6a8bf1850250741e28700752cfe5", "output_pcm_sha256": "9ab47a981d1cbc363f6f0854df09844e5e3a1b4fa18da38eb472b15ed545cf7b", "result_sha256": "24a3a23e3f800dc08f1decf28f75043f729cdbb6c6abf1f0c9cc89f81bc20e8d" }
#431navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b562e3ba64b3fbf8375687e5aee3cf5d1626bfac2e05748c0580aa8fd2353c17", "started_sha256": "a31cbaf5ae45f610147b7a95092283aa086ca6776bc4e18b2a8e667841f86803", "original_grade_sha256": "22ba0f87ecb8c7bd77c19976505aa758e7edc83e7714f62b427f85738c4f42e2", "input_original_sha256": "b562e3ba64b3fbf8375687e5aee3cf5d1626bfac2e05748c0580aa8fd2353c17", "input_wav_sha256": "90c6ec91324cdb04ed9c6549f8afbc5cd9dc552f2dda728cf10f44d05b27615b", "input_pcm_sha256": "80eaeaf97046ce2ccadf03c5d927c45b46ec705c02561c05b162386fecc6919d", "output_wav_sha256": "87b3750027e4ca7c6574a2e7ca233097ca2f2dc65a97c00520fa0c46b9fe5570", "output_pcm_sha256": "ea071d5ddd0a5bc8fdebb3365a265c96e2b8752a35de94507ce2f876549c375b", "result_sha256": "89eb35d12c32b77483396638ed815e2bef1a10ff49f9c78f47b7844af5b5eecb" }
#626object_counting heldout	7	The answer is 7.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b8d04a50adf5150593cfaac89eb8dea7d0eafd061d2efa15dba559da3cdbaedf", "started_sha256": "39340ca48a0126c936d279336d51edc54020d59dc4f0d60794487936b5fb17a4", "original_grade_sha256": "c545ba66a0f20d221ca5a7732253eda335d929fbb5e13f009fcc1722e9939eac", "input_original_sha256": "b8d04a50adf5150593cfaac89eb8dea7d0eafd061d2efa15dba559da3cdbaedf", "input_wav_sha256": "602ba41e4f7f3078846f5e109946d3915df7fe84ed8c285b219ec449f576fc39", "input_pcm_sha256": "a573532800624d017afa5eded5505274b0df20be6e7701cb08cdb20df53d87d5", "output_wav_sha256": "1146a70b3942484c7d8a13d81dd8975be4cfbd45c5b71e741b09b063d81faa5e", "output_pcm_sha256": "053addcc6c5c876b032f33e0e5a699bf4f1588a214a328d7521fff4b96bc9cac", "result_sha256": "87de1a3f66ee03a513567b4803109ff5305a241f0548ae53df4979a6644ef01f" }
#782web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "a38527c4807a2bf7421b28c0ee81f8b0211070b57bea4e206baec28f32b0ca1d", "started_sha256": "7fc420d2d75b0b23ba96acc2f7de105b7fe91aeaa97f489602c69ffc758d1cb9", "original_grade_sha256": "eb1c4cd7b6acf400cf70c855f2acb154268a45a088d4f0e4538b9dfd818348e9", "input_original_sha256": "a38527c4807a2bf7421b28c0ee81f8b0211070b57bea4e206baec28f32b0ca1d", "input_wav_sha256": "024a3db6ac3b889df45d7662a2a2070b1000aa5ebb17f8642c9abacd05a13160", "input_pcm_sha256": "4796a4db3c93535a108b7a0cc37d96096b15cd72775430ce3a4a91a78acc143e", "output_wav_sha256": "f39f3cc472e9c7123d5c7fb3bc2eabf0a2620fbd6e7ab4ce76012a9b2b5c2dfc", "output_pcm_sha256": "a33d39a21ee1ba6dd7678e777d49371d30daaffa466ab71a24718eb53e4f1a13", "result_sha256": "1afc8a15a8306dea33cf1f1b826a9ccec1a94331a4f8d243f216bd0aeb188bee" }
#240formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "43ff2e734fcda33ff4aa249a74edbd97454015ba9bbee6253002293416a879e0", "started_sha256": "8d2cf2de46616ea5d526736e8f8ea6d4927f502edd0503f8808f476ee3c41902", "original_grade_sha256": "bf40cb060edd33393d8b6353de491c9c27e13ccbc170485e0946fd33ee0ea01a", "input_original_sha256": "43ff2e734fcda33ff4aa249a74edbd97454015ba9bbee6253002293416a879e0", "input_wav_sha256": "1c7834d7f63c70b9041627838b11942b916738efd67e7b53bda28af99b1b1d1b", "input_pcm_sha256": "24e59c52e0949ce85d807de8c3fa0f7dc544213a41852ad88e7f43964376ef85", "output_wav_sha256": "b1c23c6603114a8b4e320735e0925b4b55fd3999df09da5aa3a26906a35325d1", "output_pcm_sha256": "46e8825cb9c57f418e44f7ebc3abcee46efc907f9ea81759affd39e2500cef52", "result_sha256": "79fbc8913d7d8034bec0fa4627a910882ec808dd9497a20f8d59837a381cd142" }
#325navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "5c105b9782f8bb208dd48a27a420df56f5b4ef24c1d162d2712da47f1fcfb4f6", "started_sha256": "9afa2d75fafe49c35433248b3458ca496cca33249b9a21a5160c543e2afff108", "original_grade_sha256": "f4e7dbe28a926f4d0cfc14d619391b34df38cb6245f460be03a06983801424d2", "input_original_sha256": "5c105b9782f8bb208dd48a27a420df56f5b4ef24c1d162d2712da47f1fcfb4f6", "input_wav_sha256": "112e13afe78ef7b12ad297381e534005e34b5d6f7ed5c0389db59bac5ee5a951", "input_pcm_sha256": "c3c813d5f7720d4bfbde5c6b71e89ec1f01585773a7d134e293f5d83db0127a5", "output_wav_sha256": "268e1943b095d9fde048fe86604c4b499c2bbff6c5fa4d00a8ca681569301f5d", "output_pcm_sha256": "92a983204b4bf65434c777f53aadfb93583fbba4ec7c94274bc39ed74032f5cf", "result_sha256": "e879001efeb4fb3a97f2ce878dc76bb7d86f444a739257c06e68cfeb64067841" }
#584object_counting heldout	6	The answer is 6.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "8e301003e8883f6e812ff897fa18b73e43c8422b40fa6a21c6a3c771486e194d", "started_sha256": "95a296334d2b46b0acc641b12277638b08e68f3a60bda51f6962d1c689008cad", "original_grade_sha256": "173d2873dd1b7b0cd33f8779c83a09e3bdb3bc0bae432b91f4ace4fdbc677e63", "input_original_sha256": "8e301003e8883f6e812ff897fa18b73e43c8422b40fa6a21c6a3c771486e194d", "input_wav_sha256": "445845a482d9820608bd37d27f1ac396b8f80282635770737838fc356d3ccd20", "input_pcm_sha256": "2dc2471f47c8b857b9a0d18733e993c07fbcf9514607a0ada3c93b156df44b3c", "output_wav_sha256": "08d652e80558a6cf7ce5b3e11ab6ab0223424bb0c7b91c442004d94fbe0f1c5a", "output_pcm_sha256": "7e3490cd3fdc29be50ea9bdb5fef82bf82f7f16dc4c35b98ec393b299cddc1af", "result_sha256": "d01d051aaec916a962e577dc0fc558d4f1b71e6ab234a745d7037b8cfc20e64e" }
#902web_of_lies heldout	No	That answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "6d508fcd6c1a68de3c2a2778e9f0d75eccfa84ed8878dbe4eb7fbedb4cbe03b3", "started_sha256": "d76fe56d5882b2d1e13a0a4a5045b5c2c18d9987a8669dc6103ae36e3686065c", "original_grade_sha256": "181ad9e0bdb8942844cf98a4e03a2a193221d9a4f8ae43ba600a594f66fb8201", "input_original_sha256": "6d508fcd6c1a68de3c2a2778e9f0d75eccfa84ed8878dbe4eb7fbedb4cbe03b3", "input_wav_sha256": "73dbd71f16e13b0e687c5be09e594fb5bbf2247396f8d066d2033842b2ea2e2a", "input_pcm_sha256": "26e01c5ab09f57e64c3053237a86ade05b87319d6b3fb5d8494500f84a979459", "output_wav_sha256": "8ff3e4abd186a3766cb14eee7092f065516f4ea43ec0ee173eeed9291a5f2802", "output_pcm_sha256": "478de0f06cd56f1db99631dbfbb00bdc3ed7e7dc020db4c45717826b189bbfa9", "result_sha256": "9ad01d38d0e4c637c405c8d87469e7621165cb7af2021c7ea98a48e46f78b01d" }
#31formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "39e5ae637037618dcd53dff09a21e7170339613cf5f938146c5350b4b80a0aeb", "started_sha256": "9936be0b79259888bab9302dffef457361fb60bfcb90b736ba54ac006be4bc5a", "original_grade_sha256": "d658bd72f195dd3c0c22edd88e10303ea2d0d110723ee47e10f8ab23e93d81b5", "input_original_sha256": "39e5ae637037618dcd53dff09a21e7170339613cf5f938146c5350b4b80a0aeb", "input_wav_sha256": "bba4252f87d1a543ca7c817d5591ce194bd94dd2a1bf34b0bf87eb5751904bfd", "input_pcm_sha256": "2b4067cb6f8e3977e0c3ef10a79fd7e68ce7ea99e0aae82f80b889631b58fb35", "output_wav_sha256": "f8241305cf6d0b714cf316f00f545f8c11bba6bcbd9e4dc96496efd45cc2f602", "output_pcm_sha256": "f955a5bf42ca100fd3f08afd405e2d4d68e7cef8d9cf5b419f3505b805509f68", "result_sha256": "9cdfd8a884129591c757fa3d449ccf902d8c7896426797a32daef1a186a3f556" }
#395navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "14219b57a9bff4f76a591531bbbcae7348f69e7a114c923aa249e8b317e6a7d8", "started_sha256": "56e03e5533cd440fc67e0584ebecf638bb85eefd426b5aaa609b25510ecdc05c", "original_grade_sha256": "344b520bbe230e14ff69af952572c2e43ba556ac5c7dafed57aa67cf6bc401e2", "input_original_sha256": "14219b57a9bff4f76a591531bbbcae7348f69e7a114c923aa249e8b317e6a7d8", "input_wav_sha256": "2a7b939218b2023c324a103949b62436be75728d4a9ce88ccdfc1626e1d8e3bb", "input_pcm_sha256": "fa693cd9a5ef73d1178548df8ae51680c782a33b6aa8fa12a6e6165d8f107e64", "output_wav_sha256": "77aa7a3f84a4b3d86bd4c222c97142d934b6c3a51537a2c66112484f07a17a8c", "output_pcm_sha256": "ab42605260979b7d8fafe92d630c6a9245424d4e948da251549da697258604c8", "result_sha256": "32a8480ee6a3e9bc8ef15d5ceaa44a32dadd5087f038a80d9de698dd2668a667" }
#556object_counting heldout	16	The answer is 16.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "7e15f3816bbb7f8232229f8239c4150195be429fac7a0bbae8f71a2eb9b69c4b", "started_sha256": "c50e7cc1129e215e81f11500edbf566aaf0067579ef0c0c093722de5c16e023b", "original_grade_sha256": "569f13f84f9891fc7ef8f15fa5fe78bea0aa738156e31b8f039ead0b296ff522", "input_original_sha256": "7e15f3816bbb7f8232229f8239c4150195be429fac7a0bbae8f71a2eb9b69c4b", "input_wav_sha256": "1c0965f94728af256d62b6a68722704676a9e45ec09d8cb71ed0952d0480e799", "input_pcm_sha256": "3243190ba052cd2c0a9ebdecf7ed91fae4f4134e45cc2d3eea963c970da6f6d5", "output_wav_sha256": "5cb039d1756dd5f225cbe4596d705b0cd7963cb327aa1a69fe8b709dfe7794e1", "output_pcm_sha256": "25264a99b43d8473666ed155e81cf40f803cb956767759cf8498d8f30e559e50", "result_sha256": "86b6b932810d072d04d20e5058cb542f1be9846870fd6bfad488f50363cbd4fd" }
#957web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "2921029d0c9d68998c2293ebd35410368b23a8581afd517aa705cb6ab230b4df", "started_sha256": "ef996261aec90cea21f6b860350c8fc7da90ee97a0a7f55ea44e2480ea82227b", "original_grade_sha256": "e49a4cdaad28e02c21621be82af4b52b473268e3660c1ddcb47e9b89b942b00a", "input_original_sha256": "2921029d0c9d68998c2293ebd35410368b23a8581afd517aa705cb6ab230b4df", "input_wav_sha256": "b528ca2fe336ec3c19c1d34316f14216ea7ec8f5162cb237c09ab178752013b1", "input_pcm_sha256": "4d6b7fcc3bd411d1a38f03160b7c2277a8a66e0298299622cf4e65ef986e7350", "output_wav_sha256": "ca710f330ce45e3748d0bc9b19b93d3621ccf902cbfcf2ddf335c5fdad929c67", "output_pcm_sha256": "5005f56c3a45d5d4c4e624f089fec75200ec8e9427d073dd78eef0f072e877c3", "result_sha256": "24d4676a304e5ff93dfe18e36cfe48a7accf712c4061d28cf05f7f809e056697" }
#209formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "ff94daa52698a6fc2f0c1737e039b3cf0f1c27df4806530594ba0bf58379d247", "started_sha256": "fbf1a4f823973c41dd168d769fd1740c77cabb289fe8702432ed614d4a1d74d2", "original_grade_sha256": "7795db4521cde32e67ad08f89ccb83142f4b32d90f53492f37b130014970ef4c", "input_original_sha256": "ff94daa52698a6fc2f0c1737e039b3cf0f1c27df4806530594ba0bf58379d247", "input_wav_sha256": "da8a341bf1d20a901bd9dcfa47c65d6da8319e74c2da6203fda2eed128257377", "input_pcm_sha256": "af6ceb3437be6f519ae737d7383df484268874eb24c47a31223b4a2017a27bc6", "output_wav_sha256": "e25ded468b3ed841ff4535047e8d839d1645caf17fd1d5ad4d8078649ef745d7", "output_pcm_sha256": "c6cb39e429503514c6c3ae8adddbf689f85dc85fdd94fac0cd6e29e526ef0c26", "result_sha256": "0aba43e8fad74138afccb512a068cf9885c912487a3e1e71702d25d266e89cdb" }
#482navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "8a341012a1b6172a0e7c1a361329a31c0c21ca4a2d2a3fa318e097d18a0d3be6", "started_sha256": "4a54ae30eeb0c9801711bd7fa97baf4b2f3bf27965f3e303ddcaab8e2a7f894b", "original_grade_sha256": "c6fd8d57f34fab7e83c412c70cc18013e525a3235d30fb6b2eb86fd261172e5c", "input_original_sha256": "8a341012a1b6172a0e7c1a361329a31c0c21ca4a2d2a3fa318e097d18a0d3be6", "input_wav_sha256": "50e66583e4069e9909ebb4b396aa2898f9331fc1c8aa94e7be6a9167474e2747", "input_pcm_sha256": "ff95e5e695589b46b8a775301a31f06afcaa9a041c22527c5815f026a8e92049", "output_wav_sha256": "69ee0d2979d7b8c4269956d8ad5777cf8ae369648aa0eb21b50e0be747e055f3", "output_pcm_sha256": "e5be47cbf9e63b131bfb4b0fd2cf890f2b108c41c49bb822a0cb38ddd0d5f34b", "result_sha256": "9b8554201b3a69b0197fedffc13c4ac64d24277a5bcbb98264dcde26a484bdb0" }
#544object_counting heldout	7	The answer is 7.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "6a517b4c3fb21dac9236051fee9758c7b880dfa23db1e44b7961ed75d93a3d6f", "started_sha256": "6e797b99c40c382915ba17484ec3ffe727740f00fdfb3ff959dd485fcb8b2561", "original_grade_sha256": "73282f5814ccb9e661e645f1b0aac10e813e2bcf4b4a3616d80e901d5f88c30c", "input_original_sha256": "6a517b4c3fb21dac9236051fee9758c7b880dfa23db1e44b7961ed75d93a3d6f", "input_wav_sha256": "00bc116878ff34666eb90ff40c2d1c6e5fcd5a344b3a1924b6c9d3d0471ab713", "input_pcm_sha256": "ace49db7bee743bc83efa5522bf95e3f0a56479859f1cecccb5b85475bb50f41", "output_wav_sha256": "c94a6b3b876ef994f2b726a0fb9ce22ea3dd25f00ea767bd5446be27fbe76fd2", "output_pcm_sha256": "89eb4f990755a1d97d6ef9718597bec5ea53b076ec7d770d0418b32d01c0fc43", "result_sha256": "23d421338297e45dedc1b26e6cff9b7ae238686fc2af1057ab697f5b318d446b" }
#936web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "2f08e9a64c0321a518862b2ab39d3d261631910b66e3253d1fe5ce96d313216d", "started_sha256": "43d54dec6d4b38d4772220d3ebd36a639b8c43fb4641ece72b21ea32adffd78c", "original_grade_sha256": "c48022d5dd9ba852c54397da0f4782728fdc5f87167942782d3773dcfc877c37", "input_original_sha256": "2f08e9a64c0321a518862b2ab39d3d261631910b66e3253d1fe5ce96d313216d", "input_wav_sha256": "38b7808a6985938981f44a406630fef92b8f385e792022a71b9740e3785e8ca7", "input_pcm_sha256": "2e3870176a5e14d96bbf7fc678c17eb887598918fa57821b797d54d47e0b343e", "output_wav_sha256": "67ab1c568af499a8a6b29b79cf86ba027a8c9717d6d82e23e7cb232abbcdae95", "output_pcm_sha256": "2397b03ce60d3cfb66ec4cd90e70f82e68314c26ed6da18920bdfd638514a6be", "result_sha256": "89649396bcfcc06b10aafdf370fa74f188c1d53bdc6120849b03f31604357e46" }
#215formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "33acec8b7e0fc3c8679d4aaeb092617a76a3cec6afb56e33cb9145904e9cdb73", "started_sha256": "12d56647575d8c21856d5439f25efd5487266bf650e05acdb2d55c06bb1bdf69", "original_grade_sha256": "da900a435815c3aa494b39d37f9ab842aba07bc185a5594a8e5ac817b9a22564", "input_original_sha256": "33acec8b7e0fc3c8679d4aaeb092617a76a3cec6afb56e33cb9145904e9cdb73", "input_wav_sha256": "4b2034ee162be1b4b59ea6b212fa6b92881c7bef6eb3c7046c8a6e4a5026c123", "input_pcm_sha256": "d154d6343cd4c556d1b3cb6bce8c684443e35ed5cddcc262d348fda94dfbc39a", "output_wav_sha256": "4351bf2d2d58a9ab7ead80faa3ef502d097cf3731f89833108dc0c5e0bdf6962", "output_pcm_sha256": "2a838eb92dae96752c05811e2ada5ce74b83ecf8c19245d9e94b3675b5c065a5", "result_sha256": "d625b7cfd640d1426a12d75f7fc91899a32bef5d7164d7eb6123e81a0f1d2a34" }
#292navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "7c101e602deb6cc8fd4108c38946afb610676618ba7736b857445dde0f70a0ec", "started_sha256": "6345f8806b1ebb653840990e665ac95e7683936b8f5f4db6ae35f1ce365b4b4f", "original_grade_sha256": "81413382f38384a5d83910db0ed44b5c4ffe35e952d5129bc25529e5d71bb275", "input_original_sha256": "7c101e602deb6cc8fd4108c38946afb610676618ba7736b857445dde0f70a0ec", "input_wav_sha256": "cd35de1351d516f4defb6a7c1af6f2c59f553db7f76376df99069d45e76540be", "input_pcm_sha256": "cecea77192ab4504d4de336fb345ca3abe795e9a04c0bea9f287405ae003e586", "output_wav_sha256": "4f8a7f2125e1dd353a46c760f4eed3f4598404d4090dc8a21b02d94e94786f8c", "output_pcm_sha256": "a6f08e31efbb72ecd713b65bb735ca5aa2f5ad25863506c30442bb387f833aa2", "result_sha256": "11b550ff590b14f7406673118260af59d09a5474f1186a55db6da9fffb1088ac" }
#644object_counting heldout	8	The answer is 8.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "3886042943ee11138d670c2d7fe661053ed9299f6e570e57f32795258598c60f", "started_sha256": "f81ddc3d1355a32a5a2d6b71d3d10ca068474bcfdd04de512cf0550bbb318247", "original_grade_sha256": "1a5dd9604cf5805df566e23419b20a8eba8f2097bdb906453743397950ff53f0", "input_original_sha256": "3886042943ee11138d670c2d7fe661053ed9299f6e570e57f32795258598c60f", "input_wav_sha256": "d456b9274dbaef76e5594f18fe79f19d62333e32593d22546122330c83d4e3aa", "input_pcm_sha256": "c980c9618e66f1c501c60f50b8c04285e04fa28f949ee8feb993dfd0a6573c5e", "output_wav_sha256": "63f9fa7a539683df2a8b9c0cf837fce37d3c0ee198cb9f0f97e3d50de304a5e5", "output_pcm_sha256": "74352c11ee471a04f6de711eb06c838475bf937d7cc862f4d337517f59f602a3", "result_sha256": "2936b8592d1279a389a956ec528a9f9b899d38e0624501240d7bad413c162b8c" }
#809web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "de67b170d7dc24c9b2096b9f157d07b6d13baf3aac5fe8ef6c55e558c7d56b72", "started_sha256": "e3e4c0169172b3d09fdc08bf5619328f66f11f903aaa05d3f924dea2d70ce35e", "original_grade_sha256": "fe95b9398ace9146f099075d31a9c08625ca8f095d264cd42f0b427318d8a3ff", "input_original_sha256": "de67b170d7dc24c9b2096b9f157d07b6d13baf3aac5fe8ef6c55e558c7d56b72", "input_wav_sha256": "649fe9d562972f9a469007bd2bb17668167f57904b34a2dca57cda5e834c43f2", "input_pcm_sha256": "3f220c770457dfa720baa5eb76d29664cec2ae765e0da833415ba05f3723ea45", "output_wav_sha256": "66c92032a8739b2b29eaa12a4de37b8e2bf21542f7caf91dd8ce02bd4851efb2", "output_pcm_sha256": "34106e1d7993ebe954256c51a9d4145f89afd3dab7f9cf064b2e2b634ceabdde", "result_sha256": "cfba859da9ad08957ed294a4d0d44c01ec1b7baee8ad2cd19934b4970a1c72eb" }
#107formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f09c841b9286a15c67ff0c79d91465dc857a98d6a1ff028ce5346138e00c5096", "started_sha256": "ac39dc901741a99b46113a8b7ab3c328ee5292efad3403d1dc3a715a0f84493b", "original_grade_sha256": "bc4f04a7c007c3693f6c61b3609a58bc58358c9715e6a7aeda7c5cf6f5b1a2e0", "input_original_sha256": "f09c841b9286a15c67ff0c79d91465dc857a98d6a1ff028ce5346138e00c5096", "input_wav_sha256": "5c7ff2cc93c797af273e8592620f2d2ec43352011cc379ae36b451940d8acf87", "input_pcm_sha256": "fe1791e43952b9c4b3d79d67dbf3cb55332611362307abbebe905c88d964ae01", "output_wav_sha256": "7344bce9d721a3917c6101a4283027ffa2f513f98b66628b14a1065c9842ae4c", "output_pcm_sha256": "6e30c8b4c3456348c40b08ce77505364f1a6c987d9ce0e1a10cab9bd74029311", "result_sha256": "4aecf3a1724b0d3d93bd812bdf48072569df3b4dd0b0268d897f6ecc079de319" }
#477navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "83c3bace0c9595fc7eedf6c00ade9d6cf5aabb3d89f3740f9a1301fe2957aece", "started_sha256": "b52c58ac07bdcf118e100cd7938c559e18e2b088331e9c2aa6afb51c26fde955", "original_grade_sha256": "7af1021240606c84d56c42b4a35628ec7bcaae0f9c6eb22c855f6abd6530c7cf", "input_original_sha256": "83c3bace0c9595fc7eedf6c00ade9d6cf5aabb3d89f3740f9a1301fe2957aece", "input_wav_sha256": "105c1beb882d1f27548bdfc06dc28a724b19efdab40dad52c631bfb82e2c9980", "input_pcm_sha256": "f78e42e65ce6f0a4ff9d0b1eb0294fa497d65157795b6c9cfdfea06cf004b13f", "output_wav_sha256": "f9f38a40f1a9386b8e5d07736d702c68e6039ee9b55b8b9e3f6e2eacf7972dfe", "output_pcm_sha256": "f9f09a3bbb4a7a1a2ce5c7a349de02413781c0d3ea1cdc240bf1eff2d5fb947b", "result_sha256": "d60b9ff3b2051ac30c5ce1fe2165632cbb8c2098a521c545ae04981131023265" }
#737object_counting heldout	12	The answer is 12.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "5fb036a5b47cd2fe885187f706d499a847e89f953b3f4e94979491c7d9abc96f", "started_sha256": "d9c523924c82e1755c4cb6800f79857cea383a566a95871af29fa80751e8c0a1", "original_grade_sha256": "43eda630483a6d7bba693648060dcc744dc6fb63778b0e27fb76227db798b654", "input_original_sha256": "5fb036a5b47cd2fe885187f706d499a847e89f953b3f4e94979491c7d9abc96f", "input_wav_sha256": "138482577c8035cddd8edee526e2a5591c0589551914078a0b24b5905e8a0750", "input_pcm_sha256": "3fe7f4a82cc6cdbfb491085768d1a6b2149e8571ac3b6208ab3251e214eb1533", "output_wav_sha256": "95c696c30fd98e1ded7e4d47e399fe01e4b594a8cd07987d534291fc50ca15eb", "output_pcm_sha256": "049b80309361c296854104fdcdc75f611a76a2f8623f66ccdfd153053ef77ec8", "result_sha256": "d3a3bfbdf14638bd7ac0ffe697cd96937f55085fd32aea0759460838125087f5" }
#956web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "d9f02c793f28e44f2622901084b72767dfcc389d13709d6034126ec6482cb67b", "started_sha256": "dba92d49596a9c3fe2e94579d51f631c4958bacfbd35ca08a06e8b11d41aed82", "original_grade_sha256": "bc602bc85b995d7fe56d7007f6090535192e821cc3d5f7fe07ad7d5ed0b5c09a", "input_original_sha256": "d9f02c793f28e44f2622901084b72767dfcc389d13709d6034126ec6482cb67b", "input_wav_sha256": "14236bb599e99c6a8caf9a037640420ab6dcb7422273ef4ba788ed890d7240cd", "input_pcm_sha256": "c7db6f5967419ddf6149925d27589cdb05dafd2f7f6c92d207c1aded33048605", "output_wav_sha256": "1dd9aa821ec2aaef71ae2761ec267de3826cb31d7f83611ffe85cd49fad11c46", "output_pcm_sha256": "518a6f392621a572bd1565718fa2e2b3e1a7f46fa9ccc3af2c9dfb28e239d573", "result_sha256": "d67220e45fa21ae3b4b3c223919ed988bf74d473cc2a4f02480d25cd84b88c14" }
#78formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "cc84a272ed2c9e3e5fd07db0a8e4de88c505c8185eb611c8ac3502a5db95d29e", "started_sha256": "677548143666c672d4ce342a910b431f8bb6d8496d89201b839a8ca5d8b22c82", "original_grade_sha256": "0c59894377f213292a1d1144e6f2a5d1751a3aa521ce1d3c95b5d4676217a6af", "input_original_sha256": "cc84a272ed2c9e3e5fd07db0a8e4de88c505c8185eb611c8ac3502a5db95d29e", "input_wav_sha256": "115f39c878d422148f50956bca6d3594ed36d1b12cef6423270c02d547f0447d", "input_pcm_sha256": "c58dc62fe2e5be5b53909ee0d6e1655c86b38e8c6a0d1d05ca43f22dfdedfddc", "output_wav_sha256": "2193d34163499953a9c398d84c74476ff0450b416b026f2ba1d4009fc3ae7d64", "output_pcm_sha256": "0d2a0acc4b514ef317bd8b485e50608cc49aae5dac363b63effaaaafb6a9fd5d", "result_sha256": "ee7d113ce2148ffac6f2a2524f788c049250038339f2c237c35a80e8d936cd63" }
#250navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "09a52d46ef0ceda59c83d5c537f96dc7ef2763a5281aac2b61e81ec7293235cf", "started_sha256": "85dbb83c0d3a1ab59d6aa388908b2414e4584fb3855be1e3a7ee7b846e26497d", "original_grade_sha256": "5ad619617d717138b7961f4fbd1bdb374e4d5ddb23e9ce61070a70fbcb745ab0", "input_original_sha256": "09a52d46ef0ceda59c83d5c537f96dc7ef2763a5281aac2b61e81ec7293235cf", "input_wav_sha256": "5aae60f0268e7d315270c917802293de2b68f7ca90b4b8945ab6945017c7e3db", "input_pcm_sha256": "6dcf0c5c24b5fd0f90699ef329c6a30c8a5603435878e47da7250ca3da2454d1", "output_wav_sha256": "73bc81ba54714f0ac4280af173f2ab6bb916376ac08ae47e94ab27b7e3881ac9", "output_pcm_sha256": "cdb867851eae6817a6dba5f8337646678f1662b3bfa286483256bd2ef8f10518", "result_sha256": "00085275c547cabf08d4507a63ec6e89c348f781e506597a7b39dee812de5250" }
#705object_counting heldout	6	The answer is 6.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "6ef11a8e6848fe916d53c9b5f0f96a60f9dcf067dc6e0b00c17c3121c931bd41", "started_sha256": "2d8704186452990588b8961017431e2ae7bbc3108e206d48497a73ed9ad111db", "original_grade_sha256": "1b13d9b6b1cf91ffbdb49961e859965a84416b21867d8caf0fb4747b1f90bec9", "input_original_sha256": "6ef11a8e6848fe916d53c9b5f0f96a60f9dcf067dc6e0b00c17c3121c931bd41", "input_wav_sha256": "2b68f1c3bc7bdb61ef1a061bc68cb57dab387d79eaca8e9b608d922eaa2f7ce7", "input_pcm_sha256": "6f8012fa178bccc8ae66b748c436d25d86f20d8cc92e6c1af84c7aeed22cbc1b", "output_wav_sha256": "a0237ecbed47d3d3bdc0c14be04371c62460d24da3197e9f491e02712c31cf46", "output_pcm_sha256": "8666572d2f1bab1c943f350cb181821d39d5c605367e4573bd1c9618fa79b27b", "result_sha256": "0164152c292740b9d85148d8020ec7ed859a8bfd47e99e67104ae9260a73cd01" }
#771web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "a7d75d11de8ef166243b190b5845222f014be0b73d35ce477ec485e1079f3a2b", "started_sha256": "de5977faebfe32eecb11c16ddb4dc05868ff5b1e40fa0613fc094daca4864dbb", "original_grade_sha256": "23b9310981c9d6b8da4307f935c95a1700fd4c5bb8aea84c674246f777daaa03", "input_original_sha256": "a7d75d11de8ef166243b190b5845222f014be0b73d35ce477ec485e1079f3a2b", "input_wav_sha256": "a487ed86549678e8e8c4cd3afa6b1eab278bd3018d246ea6ace57357253c5977", "input_pcm_sha256": "8e67a23cbb8b25fd70f44e8e0f45e8d7707835dbe3d05f5b8ec0227af5332c4e", "output_wav_sha256": "fe2a0319cfe5cc3ac90cd76d12b151b831580d1e555d45959403fba641a395dd", "output_pcm_sha256": "f6ff4558c8019e50fb282b5d6dc8ff29c0320ee76ce273125e99ffdbeb7df6ae", "result_sha256": "368493deeec59920514c88b2587d16afd1fc8d1c02498610db0fc50cd2ff40b0" }
#197formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "fe93b5e1d8e6d314dd6f96d715b19f6a384cd458f385b237e986b0616e6b72ff", "started_sha256": "2e1883620b3b93778239bcf064a727624fec2b91c2f0f2850ad30bdc91891426", "original_grade_sha256": "9a496caff9284aa14cd5e8e01b4541a99d70b9393a66f9164c483c850105b070", "input_original_sha256": "fe93b5e1d8e6d314dd6f96d715b19f6a384cd458f385b237e986b0616e6b72ff", "input_wav_sha256": "84cc6c02846e9defaaea07ef283d042a91a86a6b49b89607408f346edbb6606d", "input_pcm_sha256": "84165c7a24743cbe5e94764a24fa094dc7312de694ad19415d3dfe444d3a53b7", "output_wav_sha256": "ce3e24fc157f950e829748218e28d99ff8bb65cfaddee7950f2e272456fabc68", "output_pcm_sha256": "612ea00a9f3fcf0c3de5bd67c00d6bde8a2c2725cd28fe297f1ab9b4008106a3", "result_sha256": "fe42a2f746a46b84ba1fe7b80ec00d77ea71795330eefeb3dc98f2f67da936a0" }
#266navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "4811646a9ba9d9ac78f3b023ceb973d13f43d9f566757996a7adebbefd54ed3b", "started_sha256": "cecf03d99b4f0cc25c563bfdd235f8c50ce3f9d53b399d2c396077ecbf355dd0", "original_grade_sha256": "613f1cff4fda5ed78930c993fc47771b40f9e553443511405482fcada5e620d3", "input_original_sha256": "4811646a9ba9d9ac78f3b023ceb973d13f43d9f566757996a7adebbefd54ed3b", "input_wav_sha256": "29e1bf4b5c21832c209f0a93a68e03d9e94755e18900f3e073a7061db3f74e29", "input_pcm_sha256": "12b313bf5a4aea6bb334d0592ef2e1c35296f7a6374d8dcac4575e62737bd2fe", "output_wav_sha256": "ca3c8fd354276c51484402f9e2f6492174d7553d1ed4df5f443b4b76497ad696", "output_pcm_sha256": "b026277bd5e84a14e8deb41cd9f744dd5bed4be445e7c184a1ece48b81845559", "result_sha256": "00d76998d91736d31542f71c82ccd916b113131f3784fc167857f942e85933b5" }
#533object_counting heldout	3	The answer is 3.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "9bb5a8c7df7eecacaa6d5316e61e19fd0b500af0f7235ce80855a9dec7ea793a", "started_sha256": "25acd2ddc0912be76b405906c04e446d60f5cc694793d98d0fffaaba116e4ea4", "original_grade_sha256": "72ab7e198c53a970a2a65be8fe5c77ca7ba5b429e690b364c199eccc444f418a", "input_original_sha256": "9bb5a8c7df7eecacaa6d5316e61e19fd0b500af0f7235ce80855a9dec7ea793a", "input_wav_sha256": "0ca0601e9dcdb0ed125d693b439b5e465924df08e6933704614f3f0f91ed394a", "input_pcm_sha256": "daf768575a08eb369d63c9110a005002ef3c57e6a169165107db5290d8962f37", "output_wav_sha256": "97789c249cbd7de2bbc3a704eef5e2d5274c50f79053a2d82eb6677dfa927912", "output_pcm_sha256": "0739d62ed7160192dfff706bb004cdb954a10cfd4150a2be29e69aabf7722b14", "result_sha256": "31a33013db0e327342231ccd02ce4fb132550f6d737294647b1b147319aa7b47" }
#829web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "de3fad8654eda3a0ca295c95c4120184b284e31ce5a1ecf2da88ecc8acd57056", "started_sha256": "2b653b5508cacca9de080b9a8f18f13928c4d4829a0199159f47e6991a1c213d", "original_grade_sha256": "76dcd7bcd4597c446edf3f8a1a296e9c74d1782b3c296ac656730caad8647860", "input_original_sha256": "de3fad8654eda3a0ca295c95c4120184b284e31ce5a1ecf2da88ecc8acd57056", "input_wav_sha256": "fcde4cf8ba857831a1d506f2eeeafb349cfc9fdce575d23c9c45e76f7b87965a", "input_pcm_sha256": "1454bfb4d5637acf7cf84bc9c4204165297adba4075f79a277ce20dea04b5bd8", "output_wav_sha256": "29f316cb253c46b6c7c577aaad5e5df6784b8bfcba5bb63fe2d4cb648ad3720f", "output_pcm_sha256": "906388f44833878be2a7eeb9fbf60fdf898bab05089b61128510ba65fa1c9afd", "result_sha256": "9987a2ea102d884c8ed1ba74c9c2dd86d9b37f19fc58152e08c8479b3e35dc49" }
#221formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b4a3de5bf71e54d5531679891f6ab07cd13fab5357037efe91882046bb710af6", "started_sha256": "92e77e159374ff48eb4a0616988ae8b4fa580bc679abead77240f42c9ed2c688", "original_grade_sha256": "764e4e1c3f2ebf223db7a6bf64f00da6d2fb2ad489e6125604c497fdb208e7cb", "input_original_sha256": "b4a3de5bf71e54d5531679891f6ab07cd13fab5357037efe91882046bb710af6", "input_wav_sha256": "2231c09da6be4beae25f680017460b176810b718b109324e2ab73ec04378d6c0", "input_pcm_sha256": "a30eeabdc5307d3c76aaaf40b90970feb808e36822e5385bda525175f2ec0068", "output_wav_sha256": "7df03fea2966095038ca7b4bf7ccd326f2ca6e16bd1458b89a73725c586e6ffc", "output_pcm_sha256": "5f5e1d72dc9e2928414927409489c23ee4ebd7eb24bf1bf1c5a40ad7defdc84f", "result_sha256": "04ad78ce897ccf233b24a4f14b2278b1a7bd1ec0ef8a7b366eabfac207928e34" }
#420navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "1c0f8d3954bba698d7dcc282bb4f93bbc7bbbe604c6de802066299af3ef032a2", "started_sha256": "a88fe8bc684b750d9a20f5990d06c6155f7f7e1926c4b58b32928754a3e24ee0", "original_grade_sha256": "091fd64e304e6803be5876474b67ff3778c092ac7df951faf9c867ffd1800e68", "input_original_sha256": "1c0f8d3954bba698d7dcc282bb4f93bbc7bbbe604c6de802066299af3ef032a2", "input_wav_sha256": "cb81b71933c7e120ee0a67434e80f95a6ea8dbc32b1a2ab8937ddb2a0776fa9e", "input_pcm_sha256": "315a9f07f6fa28bcd741aa666b9cb7173a058015649e1f83b56c480fe130dc48", "output_wav_sha256": "e4e89000951405584635be9d15a21cc2dd66f0d5985758a86266d2423a0a013c", "output_pcm_sha256": "9d7e9749320df251ce981d50481851d02acbeedfb41b40df689464f5268da0e4", "result_sha256": "b114308bf0803cc99d2d720c334ba4bd00be04d37290d8025689cd6ab8acca58" }
#684object_counting heldout	6	The answer is 6.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b0480ff6af0c289978f1fdfc01b1d867809adf25dae9cde49cba5b40b2655abf", "started_sha256": "3e3594a13fd62b299bf959527a89cc6d6b7fc897fd0631a64d7b691776b26804", "original_grade_sha256": "b2f95cd09486719bcff45abd324a57956121f1a549927b12d994f0dc943cf508", "input_original_sha256": "b0480ff6af0c289978f1fdfc01b1d867809adf25dae9cde49cba5b40b2655abf", "input_wav_sha256": "4ce8d325137d99949b047cd054e56faff875709cd4677894913ff4ea5dd6fa2d", "input_pcm_sha256": "5c006fa03682b4a1301700ccb1850091eb6354f8ee45398e0dbaa8320be65cfd", "output_wav_sha256": "21961a4493a525a586e4217c6cd706beae325db93424b2bf365c3a91ddb863c8", "output_pcm_sha256": "99ed46d1dbd2bca6838e0dac79ef220670e504eef0732db38c4e81fa5ef1ae6e", "result_sha256": "eac36757f1e68536a7c3a877ba003c684893882037b0a6adc8ccfcfdef5bcca5" }
#879web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "44231763b70d5be1b5fe699efcd9bd536d609a2c4b0ac17da116e076ceb2ec01", "started_sha256": "5e65e363ef9d4f6dde1e24d2f39dbb8923ac83fec6de3f64de47185dad74789b", "original_grade_sha256": "788141a2e1c93238b49ecd2510154dace38b00c4363294488af34d5583e926d0", "input_original_sha256": "44231763b70d5be1b5fe699efcd9bd536d609a2c4b0ac17da116e076ceb2ec01", "input_wav_sha256": "e2328846ab5cc44149dbbc85404a89760f64db21eae8922117a515f0b8f0a836", "input_pcm_sha256": "717b5cc77fbaf9f448e9f1e55a535e1dc459eb7165a467b63dc45162f78e4f02", "output_wav_sha256": "dd043aaa510766d918a55b122261d2b868edc98d76923eb852f3b1453c7a3c13", "output_pcm_sha256": "fbeeabdf7b74475301e6b2be465867a02bb1b72f350f59c429195bb9fcd6dc31", "result_sha256": "36a002f393f484d5be6cd9c5dfbf59070ccb991440316ec2cf3778617ed49ba9" }
#229formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "bb29676c9b3ea1058009dbce14d27e30bda5574e04b9f952640b57e55a482306", "started_sha256": "22c6679ebbb2de5efc91484ab1504368dab10c8e4a6ae218f1c878fae4fdfe6e", "original_grade_sha256": "e6a57126622a708d4837fa22fb59147ab9624401427974416097a5a3bf147d50", "input_original_sha256": "bb29676c9b3ea1058009dbce14d27e30bda5574e04b9f952640b57e55a482306", "input_wav_sha256": "74f7e131059bfc73a38d77d2ced721cc97823f674157baa806ca2e7bceadbc63", "input_pcm_sha256": "95ee02d52a5ecacba5745043802aa4e3aedbcff7da95e057b8b678823c4dad12", "output_wav_sha256": "4aa8a426ffb5593d951ace97ffe2c177b56aec32b4f2792f84c2d0648b449da5", "output_pcm_sha256": "44097db47f80987394e45c17265cc883e4f36dda518c4ad45b79393a53565f04", "result_sha256": "356eae95e4d2395bf6f8435cc60c12b11c2b648b239caada66c89ed0b7332e9b" }
#288navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "9679e3d2de9ad4acdd311c059dc90886b259cfc0dbed03c131895d651f4cbde5", "started_sha256": "f215257083dc8210b2c2b5d4428e837caf98412df998d7777576d94f41b35bda", "original_grade_sha256": "3d90c5bb0b6607ea877b3f56f18b4e9845e5512994e158a1e1c72c42dfcb1a44", "input_original_sha256": "9679e3d2de9ad4acdd311c059dc90886b259cfc0dbed03c131895d651f4cbde5", "input_wav_sha256": "e4e040c2959a3dc7e14511fdee8a1e2d441170ff3da338429540c90ca0f6b096", "input_pcm_sha256": "16a551432d6432060d7c2b6ac3e801023ae4547a82eb86dac44e32df3429426c", "output_wav_sha256": "5604d4bfc84838baaafd488e93435d336dd3d9deb52ef3a32086b5c0c72ecddf", "output_pcm_sha256": "c57f4d1f2ca103024aad51b7f386181d7e322cec1a3039f696d3e663255a7f83", "result_sha256": "f6ed831f7cc44240ed19096aa759818a88e5ff54345e590e96592bf1bb7c0a55" }
#699object_counting heldout	9	The answer is 9.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "aff3195baed2b2548b38d016920a8bc1ae05880b62a18e7b10b5817798ce2d48", "started_sha256": "18a794d9b7362b5e2bb74553cd9590d7fd5f327c2044d0f34663d201fb4f80a8", "original_grade_sha256": "267ca1f0d58abccfb8529ac9572d925a2c1bb6c9481b0113216dfac5bca60561", "input_original_sha256": "aff3195baed2b2548b38d016920a8bc1ae05880b62a18e7b10b5817798ce2d48", "input_wav_sha256": "4cd4aec36dfa5e0048e79dc4f0444f36babd9c92831e9be6c3e15e2624d61043", "input_pcm_sha256": "9b6a2a9834c21da249189b4b905925127b7b487807594f5bdcae806c5e4dd1e5", "output_wav_sha256": "91f8433a0735e5703b8dcae9c2bc799bb3c76b7c62a3e67962d2f09c54533fab", "output_pcm_sha256": "ba60b6eb35a402a44faa420c58bdc993e901b2fa65b39dca861fb38c78c211aa", "result_sha256": "462ca084f19de31589a52800b09cd863f8a16030314c7404f02431e3cc2d9e32" }
#759web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "a44af41b37068c803c1e57d5ad8c8377a298d5929a572d9f9b3683bac1b4ae72", "started_sha256": "9d66de3b91347e36c84ff5e9fbdd51e0d4ce368120cb67b31364ebf9faf741da", "original_grade_sha256": "b6b8b1d2d62c9cc265c1ccbcb947fa56cbe553c73987231c872a3909f2088bd1", "input_original_sha256": "a44af41b37068c803c1e57d5ad8c8377a298d5929a572d9f9b3683bac1b4ae72", "input_wav_sha256": "4849def81774e46a585c582b43a5b2291c5511729534204a6a8b2e3bb07a12c1", "input_pcm_sha256": "b1712ef6bda9426768a65cb33dc8304ae31c8028691c3a61d8162afb9d33fd70", "output_wav_sha256": "f04b600fc2b9a4442275583c0d904fbb61dea0bee50c40148f7ebdca51d70346", "output_pcm_sha256": "3b4fd51f59ef08909246cd9b24d31a44e709835b4740a5e2a4a275e2f4b22150", "result_sha256": "12ed8a347ea1cf59dece06c546d249dae7876396fcf3f7b7be2f38be05565cd4" }
#77formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "0afa1d37a08f755fbf66c7a683d993415eaa3125f284579952542ad3b0bd0f5a", "started_sha256": "72b033e350d258216ad5c029042a180285735c65ff5866eedeb205f1a69889cb", "original_grade_sha256": "f685ecfb167ab14e850e45b436de1eeda3a8330049832a5407973e462c86ebe5", "input_original_sha256": "0afa1d37a08f755fbf66c7a683d993415eaa3125f284579952542ad3b0bd0f5a", "input_wav_sha256": "1dd3f219cacbaa6f0a493c7f009631a650b1e674dc4418531ea7afaeee0e3391", "input_pcm_sha256": "4e253554e45639b7c30ba73c640e994f36b05cc4ae270bda698861301747429a", "output_wav_sha256": "abc87fa8b4656930b7ed92349b6f8f2e5b60176a41e80aec4396dfb1679a416a", "output_pcm_sha256": "d056b29a77ed8c33bf0cea403ac5fcf4763ca167de005936c933515932f6089e", "result_sha256": "c751e898beae98c98bca954126b8167be0a6fb794be4fd02dc034097aada13b3" }
#457navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "2db37a2d04270a5c0924ce25b7d04c350b3c518583eddf21b4decb74edc768a6", "started_sha256": "2137a218d4832f065436c5f560d8e37014649a134835b2ffac721b943b2c24cc", "original_grade_sha256": "5d8d4c4559bc2a44cf07d78054e69ba7347036c09be9513dd44d563c89c92309", "input_original_sha256": "2db37a2d04270a5c0924ce25b7d04c350b3c518583eddf21b4decb74edc768a6", "input_wav_sha256": "ca0d4145075d19b56ac6996600031a771372977cecf9785fa98fec3db9b101b0", "input_pcm_sha256": "d19fa91c1b703d45ff4535b0275500de496b63ccf50020209f6896a29373f52e", "output_wav_sha256": "2bca732e1c35edbbed1a063f2da6ac91d74d090666b4249fae468db3f44f773b", "output_pcm_sha256": "329f37ca37cad01b2624982e55aeb4f44f6e43563eb50a08364053a4aae3aede", "result_sha256": "3b14efb60e4c6e52e9ff3d54e674bfae74dc586a968af3a7e0481029e014db6a" }
#671object_counting heldout	2	The answer is 2.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "09b09bb62dd7816b6522633540301c96a9d33dd7d11ab357cfbb21d4f38a102c", "started_sha256": "54b6ab7a8fc076df6affaf8cf78f1c9b01d650f8a22fe6712fefe24b32964cb0", "original_grade_sha256": "aa671f519fd690e665c2b0165d0c5b68a248a6b3ab4075a1eacd668c638fe95b", "input_original_sha256": "09b09bb62dd7816b6522633540301c96a9d33dd7d11ab357cfbb21d4f38a102c", "input_wav_sha256": "ac86031a1db42477993d33341ba48423a370b2dfe3624c62e6521d018a55a318", "input_pcm_sha256": "adce3dd96b25a267ac106c285b41847567ac1c7fe0f270dcbafa99d3d6486df0", "output_wav_sha256": "aef98c8058da2748be2407bc471a540cd635c6d4acb5ba110431274aedb4c456", "output_pcm_sha256": "ee85e8af710f94756e1f2459478667d74892dbece5b735d9775d955a8f1d13fd", "result_sha256": "d12d737b6fd0fa507aa0b547b6e0ad0f526e9f9055427b6f16f48abc819f1941" }
#751web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "2a552a0226cad6ff58dcdd142880c5146db1a4ca0cdd691aebe07ec0aaf3cacf", "started_sha256": "3ccc712649565e5d29118672b5febdb14392a51ab7a006d9b602487b3abd05c2", "original_grade_sha256": "dda1722c3d8dac010f6d35614578ce5afbf3efbf7b8f0bc7776ba6f3f464b475", "input_original_sha256": "2a552a0226cad6ff58dcdd142880c5146db1a4ca0cdd691aebe07ec0aaf3cacf", "input_wav_sha256": "5e1d803b19920affb80f512a7d3b51137b5aa8d749e865d14cc1d70b115b9b84", "input_pcm_sha256": "ed93dfc94d5ff7831c35a993b881a490ec86d85c4ad7ac8859825029bb33cc78", "output_wav_sha256": "379213dad5575d3983ce69169c29ba9494846427173a45ba085acd014ead513a", "output_pcm_sha256": "59f4774826ea3f934f84025c34dac84f22839597d3df3208b182568a2a6ea4cc", "result_sha256": "0db1bbd6c103be0f34c497e859261ea16baee58c4032652192e826ccb34cafa2" }
#41formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "a19da5a4c0640c7242f8fee6f8b521af4822697d5d3c7c755916f7764fcf2d0b", "started_sha256": "385e0f2c41b4789e3b7e070cb26d06eba9bcbeae9a1de256b7c875f8a60030fd", "original_grade_sha256": "6863c275958c008b89ef692285cae652ce4c849e42ad6733bd4e6f3ba169c1b6", "input_original_sha256": "a19da5a4c0640c7242f8fee6f8b521af4822697d5d3c7c755916f7764fcf2d0b", "input_wav_sha256": "945503a454067dd5d0765475871511c899c51448070123e2d7ce106898fd6545", "input_pcm_sha256": "9cfa13b62b13319ac6f36d9cf10dc722f8fd185b9960b6c918902863df41a214", "output_wav_sha256": "35d8dbc1e80aafe9cdb9c0e09e6e907a3326505d1b60b37545c6a7c51a4b33fe", "output_pcm_sha256": "3da356ac7018ba61a4c95390a6d6646f4c487902033b3f3ab794c97630c25327", "result_sha256": "6508afa363da2253e2f64ebc0606b27face396bee3f2a0efa1aaf54ecaeabbb1" }
#445navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "0ae1a8b9c3f72062b3a6fe2c8e19d7c843b4d2d7219aa90ed5cad7d790ab215b", "started_sha256": "9209dcd1778cee0c7a736a782de9a60625a899b268628ad689720f79cbd57a46", "original_grade_sha256": "53a43e24ba02e4736c01c4fd50df9488b7423c82302a7fef2781cdc3e28cb410", "input_original_sha256": "0ae1a8b9c3f72062b3a6fe2c8e19d7c843b4d2d7219aa90ed5cad7d790ab215b", "input_wav_sha256": "4df866dd2c72ec4266e55e7ffd954da3e0636929092726aa12a44a4f02159789", "input_pcm_sha256": "4f7a6a4d828b8d48e2f8f9762223896b87a9b3809efcf8c7505aeee091e1f44d", "output_wav_sha256": "5643d8d9376cb596fa83eb9a51aaadd6822136ab24d88c36d8cb050d0af5d6a2", "output_pcm_sha256": "d5710a79e5c540a21e595214b7745f4534852e6353101caeb6030acf7a4236bb", "result_sha256": "232a06314d63817573338b3c78d038cdc0efd396c780c1a95c210a9c004aba29" }
#592object_counting heldout	15	The answer is 15.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "df15db3d8cc2177b00aca4da2da9674bb3b8f034e469397b056081830c98d72d", "started_sha256": "b04446a72af79586dbf41bde34f4788da4b9881489ede8c036d441b1bf04b0b2", "original_grade_sha256": "4429bfccf297aca948803481675580653093a32e7600e32460fb858af543e5d0", "input_original_sha256": "df15db3d8cc2177b00aca4da2da9674bb3b8f034e469397b056081830c98d72d", "input_wav_sha256": "6abef268764e1844bb60891bf723812e44eddfb6d237cc62c40fd3aceff5e970", "input_pcm_sha256": "47f2869d1163fe7144e8eaf3bab4122afefc00827951946ffb2b66a4f40d0cee", "output_wav_sha256": "786125ed019899818aecd971740d8e14df41c11d4f1bd634ef4ca3f87e60af99", "output_pcm_sha256": "71e6534177f5aa3343f98eabed5ca91dbc372404292902ae24132657c5bfb92d", "result_sha256": "8b2d3e36ff7ad1456f8a9bbe8cf0f29aff506fdad36bc5d6372ce67cda00a6b3" }
#974web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "fede326e84c1b562399d2ee673c9001077ef96ec1977fb74bf717e09af7bad14", "started_sha256": "d1a0e91e97ec5b0d937c6cc5e07b9135c017ad9a873479ae765f7564834da100", "original_grade_sha256": "c409efccc51d0d31122e3e3105c69a5ebcc7e69e67c6f7a7a8c9fc6335c71b02", "input_original_sha256": "fede326e84c1b562399d2ee673c9001077ef96ec1977fb74bf717e09af7bad14", "input_wav_sha256": "bd31d7b0448875235978730f3c32d989abac073cc8756051db0edff665f22595", "input_pcm_sha256": "babdddcfda9ae2ba515853a0ee347eabf691f162dccbc9740c6f5c0e96d32f21", "output_wav_sha256": "df12a9e2a64181d62030b3a655abf060dd9dbcc141e6347aab45544f8cdcdd20", "output_pcm_sha256": "093cefbb99be0a020357b1fc83385fde74b05a71b09281e00138937b794430fd", "result_sha256": "a4bbd6fa5f027b666b14dd1047a74c4c879fe155dad0383535358748404d43e9" }
#248formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "3fc470c93d96e40142cc4481517f9a62c7514e15914e45c24623d67a54047a2f", "started_sha256": "770898f1bc33da931a5102d3017fcbf80c67a83629cec84e2a89a5916f0e05fa", "original_grade_sha256": "0e881a7641e1f1c7fd4403afbb76ada15d4032246863ecd4fa3a5bf50563a1fc", "input_original_sha256": "3fc470c93d96e40142cc4481517f9a62c7514e15914e45c24623d67a54047a2f", "input_wav_sha256": "2a8d9eeccadbcfad1ce8440204b2b5734b158de7fb54e9c5ac2429555dfab6a8", "input_pcm_sha256": "55cc26b0ebb7d10eac289684830dc60fb0144840cc9817228cff57c025952528", "output_wav_sha256": "0f595e69e519ae83e4f61792a2e617da4d0eab737433a6cf2d07b6ab8e8c5a25", "output_pcm_sha256": "c0bf7ba2e6f819ec02bb1245ec7260dfdf517a0352182beb70b9f76e35a6ca77", "result_sha256": "abd0674331e39fb8a64c689e401f4814dc811f6c841aac970f8458e33d9097be" }
#283navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "840c2339fcb50fd0ee1550665822391a95c7672127cf1034fcb58c1b6bcb0b86", "started_sha256": "407a341081e0aba0807a7c67c77a72054c7be96091e0da19a8fc01f9da7b31f5", "original_grade_sha256": "6ea3be6109e687a9812cf63d0efd4038f29727f2f3abb0b038266d2ac927546c", "input_original_sha256": "840c2339fcb50fd0ee1550665822391a95c7672127cf1034fcb58c1b6bcb0b86", "input_wav_sha256": "d231d241a0205efe40905144eafac43ed820b4dce73d6a3dc973f2cd3f2400b4", "input_pcm_sha256": "ea09a30b89ef181d5744b3f01e58264dc0b165f11847c98d4e2f63c4c8cbad9f", "output_wav_sha256": "43604f8a1ed26b042b0b89308308e9cfeb2ddb57280426d358c42a39d96a6627", "output_pcm_sha256": "65715c3f5c207a7ce15a6b57d9507953e30aaf8dd3a3d1e6222ad53b43eebf96", "result_sha256": "08d0a0088d60271fc53d467b819cc57e4205b81f321deab6dc067eb8fa8c2d43" }
#746object_counting heldout	7	The answer is 7.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "feb88fd65e2b9fdc5cd45e17cf57a30a5cf657a004967a9e262ebff747d75360", "started_sha256": "253f73c0232a1092facbd7fc1a635aac944a6ae8b50dc9bd354154e192771200", "original_grade_sha256": "118b479206c145bb962d611965899c14f897e27c4f535f556bd4b66983f4e6d5", "input_original_sha256": "feb88fd65e2b9fdc5cd45e17cf57a30a5cf657a004967a9e262ebff747d75360", "input_wav_sha256": "c0c969615edf285036e8eac99a2c4a20d4e14ad556c916075d6787c8d636feef", "input_pcm_sha256": "a0300a63e883afc5ff8e2a6512c94012ef004322c9546664b624fcf343179140", "output_wav_sha256": "f0d9eaf75ae7e7079d9ac58d7e07fdc13333f96dcab5f86580b773282a823dc3", "output_pcm_sha256": "ec5bd285dc5726ed7d9326ac8f86a39e804624ac2e94529cf8d1853be81ca7ea", "result_sha256": "2ee7c8ca904afb5c5cc11b55156c622dba4823008d23dbbe8e8d5cb7c7f078fb" }
#984web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "afdef7215d02d90aeb6491869d7f058c6de068fd3b7dc9eed544ef4f6ca6db86", "started_sha256": "d721b49f07fd94c6e35608e02219e92a69fbb677e373ed412262a6eee7cdb28b", "original_grade_sha256": "cf51a0159140fc36361a542559a11cecad6c74cbebb26b547bd7138d4b488445", "input_original_sha256": "afdef7215d02d90aeb6491869d7f058c6de068fd3b7dc9eed544ef4f6ca6db86", "input_wav_sha256": "321acb8bfd3145cb25b00c25329560397d5a31dd7cd971c35375a702f0aa512f", "input_pcm_sha256": "ca9c1f98b3bcf02a0ca92a8935603d08e8914a995a538e12a0f8fb78d617a8f9", "output_wav_sha256": "d398aec70183a91aea0438bd6aec7ea286962b9c5355be08929555283f4b92f9", "output_pcm_sha256": "755fd7e26df1984ad732ce778ab274d1ed7a729e09c291c4046c7ffc19c1a80f", "result_sha256": "9885a41da6d078b9e21bb4065ff76f75c24bec52fe8d78acac81959798bddb82" }
#239formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "ab8ba441b7262baaa881e99234a8c64f8dd0cdfdcc38ed66f357818a05f11de3", "started_sha256": "114cf48a5dc91a2320022700f782eab935904f5a4c67b939ce9a3d1d7706d909", "original_grade_sha256": "d66edcefdb927d5ba98314a740d744760e9850aab647f6f3f94c714fd5d79269", "input_original_sha256": "ab8ba441b7262baaa881e99234a8c64f8dd0cdfdcc38ed66f357818a05f11de3", "input_wav_sha256": "b593652fbf172b6fd02d0239e93a9dcfebe09d87ccb3270249f0708449509d93", "input_pcm_sha256": "aba0c9eb075bef5b2c22398fef34dd3497ed9b680ff0888d2105c817bfa3d86e", "output_wav_sha256": "bf9dedfde29e72e14873abc216655687f21f365041982dc9d3cfb65b51707dd0", "output_pcm_sha256": "329e6a5caa2b6ae8df6cd5c273eb8aa6af1617b248f819d27ba171e653cb1308", "result_sha256": "18d08acb7490811bc4d7b1e32a0ff4cce44e6ae12dd3ac62cfd7afd78ee1921c" }
#345navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "d8362e220d303f2062b45ae3c459e39451825df08929131f519eae809de6f9f7", "started_sha256": "9e350719238a7437bbb9169e8da3e29bcc0164ac87748aa5331de3651b16ed31", "original_grade_sha256": "c162ed632505bbe7237d697ad2f5efa84cdae4bcd381d60236d460b5db2974ea", "input_original_sha256": "d8362e220d303f2062b45ae3c459e39451825df08929131f519eae809de6f9f7", "input_wav_sha256": "5619eaec838fc0e91f4f510141d7a6b9372c496d0d768dbf69fbcfcdc580123e", "input_pcm_sha256": "72eb4d79f3b88bf31ed1750e39ae885e148a29832a38ee658ac54928e32de78e", "output_wav_sha256": "f8bd67085fd378dc6d04fbcc6ec9aabc7c7d47dd4651f0f4951995945c98417d", "output_pcm_sha256": "2808c4ae37fe55dceccc0a2c1b327ddadf29381e7a7fbd54a324335b20e8b64f", "result_sha256": "e78951d24b8db1f00365ba631edbc3ce18c7045310d0e6bd3ea30d91d031e4b8" }
#674object_counting heldout	15	The answer is 15.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "5db94cfc02c933f04da74ddd7f9c3e9e1ef1494f42aa42d01f3613b40f5e81fc", "started_sha256": "56a9d2aca1926f52e6248d419d532742535f97960f95cf1f36947feab6e275ab", "original_grade_sha256": "9c5d1e43de3466337279e4ae11afe11ed32a043c88b1f2b606de297918e04d9e", "input_original_sha256": "5db94cfc02c933f04da74ddd7f9c3e9e1ef1494f42aa42d01f3613b40f5e81fc", "input_wav_sha256": "9dce5b5c1054647f657362695e6da98b6bf5636e6386fbf3b32e496e8b24739d", "input_pcm_sha256": "7d6cc7594cc0ff89bdacb5d0fc54114805557e575b32aeaf926cf96e6e445ca7", "output_wav_sha256": "c602df2ee9e578c4517eac2ad73a52f2d26238a323fc921f806a2a3c891d33a0", "output_pcm_sha256": "f78063f72b756d448900d7676d67206a7ca1e6744577c4111a032c5a37bb959b", "result_sha256": "bacc7fa78b49a9a73fa606c6ab7fb4056fe284af917588d51644fb6662257c1f" }
#753web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "470092089f730db7a11da80a6ec3c986c11c9b22406ed62b3bbc77f38011d099", "started_sha256": "11ca25f2697dc0f521d69e26e806b603050c2e523a24c7ba636ec3a4f4274459", "original_grade_sha256": "43f7028ec9b39c471897ad122aed73226a0a1f669cc6fb225e4e83118a7be055", "input_original_sha256": "470092089f730db7a11da80a6ec3c986c11c9b22406ed62b3bbc77f38011d099", "input_wav_sha256": "e658b32821827e7209611caa82bb283a36f78c28a05bddf15266af815f6b13ff", "input_pcm_sha256": "a7106dfd39b2a97e4de76e0a61e7aecff7a2e5ccd87798ab3e1b277071e78a52", "output_wav_sha256": "5ad5bdc4555fc55018c3a8d49221bfb0cc22e95983330c438d9913fef1f83f26", "output_pcm_sha256": "e71ad613ba121be91913a49cad7c89cbeaafdb165fc45e92eb0df92ea3d627da", "result_sha256": "b12f91ab464a7d47769315433510f318ad799162418c4094b748d330dbc265e6" }
#114formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "2f284e17a57b33e93ff029f45186fbbbe0ed5dd6adc5114b9fb3c30e434fa625", "started_sha256": "85e12dd071d292e88b33f4f19fa3f08ca016d3bb3ac940ee60158c0f88e36d86", "original_grade_sha256": "728a78c1d41e97d34f9223783027042404cc4a8635279385f7687e732ec8ab66", "input_original_sha256": "2f284e17a57b33e93ff029f45186fbbbe0ed5dd6adc5114b9fb3c30e434fa625", "input_wav_sha256": "ad9d60711bd5165496e6c0ec8a14d12916ff59d2b6b98b4ec082411ca08a636f", "input_pcm_sha256": "beacd080724d445c3b642fddc772996ed2a909f4818908ff190eb259faaa643b", "output_wav_sha256": "b7b6a5c229f47c8fe2004ad4977fccf1c4545e760355d48a7ae1648d80a4877b", "output_pcm_sha256": "d6482d713d2e92bfa504cd90ecf31118b576eabad3988e835aab7d7531bf5c80", "result_sha256": "7c56ca32378c2df95fa90b42f9b03775efd911e47d6ee4b8809e245c4a969cb2" }
#471navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "6a586d3242123ea692e568acd07ea2e680455f5dab8059c5f033e81aaebaaab1", "started_sha256": "0713943bc43d0836004f2d49d2439a8536d43b06775db1a9de78357de4c2d707", "original_grade_sha256": "4e47097f1cb07fc38f449b006f199cdaef6c8316e818246e158968ef273cd711", "input_original_sha256": "6a586d3242123ea692e568acd07ea2e680455f5dab8059c5f033e81aaebaaab1", "input_wav_sha256": "34a8131d5ad8fbb1e2e582cc04743d1155350bb44069f63ebb2778f91ae30205", "input_pcm_sha256": "2c9dff601525ee002dbabe59e5f2d7a44dbabd4a6a38d33172666cd63ab40339", "output_wav_sha256": "f48bfd08097ae6986258214fcdaba16d453d28a45f15dac50f7156624de7919c", "output_pcm_sha256": "809f2cf0e45607542f125583669d6bb78320778f2b3d5192e4b22944077b4ffb", "result_sha256": "bb91c80a99ac7ccda9ea26cbfe28753fdd347670e2f0b5784a750b0e671c1d3a" }
#575object_counting heldout	18	The answer is 18.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "00c12d48a30979101d402333b43f94ac1497cfae60d5750e82903dea7c3459a1", "started_sha256": "a96408f2a75e4088ce32392b6c786de2cdaa27e1da014c5297da31a04e967347", "original_grade_sha256": "d06cf0e401ff1bac95a3f6f47ef0f65a591ab478045f4ae000e4a3e3622cdc0b", "input_original_sha256": "00c12d48a30979101d402333b43f94ac1497cfae60d5750e82903dea7c3459a1", "input_wav_sha256": "672b8fb41292b38690e328b79b42281ed7ccef770dda255a3ba70c504801b0b5", "input_pcm_sha256": "9798a3e30c2ca7d4af94dd4380091357178e8ad123127f4e47a04fd9d80e1f71", "output_wav_sha256": "bbab6f079138497936ef1cca531255db1e59757f9531797ba24653c51d127303", "output_pcm_sha256": "e7d53fae7145a7152bc330fc46c58c0d3f78e1276285ac102994c5c6a8417a65", "result_sha256": "eeb54ba5da6227e597cb5d56091ad58485e79517a786341cd17e3a0a3fbbe529" }
#896web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "39e83b6f67988105eeeb3b603d160b4ca21ce2aad1f7e3f4577488606bf1009b", "started_sha256": "5bb20f2607e90adf76fdc95ced6d7562faa7e10a819bdced92249887b9f03e8e", "original_grade_sha256": "e0c8fa6eebbd282633492d122a1cbfe3bbc635e2801d65c350eb0111573ce81a", "input_original_sha256": "39e83b6f67988105eeeb3b603d160b4ca21ce2aad1f7e3f4577488606bf1009b", "input_wav_sha256": "9d956a60f64b1e39fc1949f46f9e4b850bced11b51c91f8feb1358d62baba0bc", "input_pcm_sha256": "612a278de411e94ba483327f779c4abde4e2053950a2df84027871c4f8c6223c", "output_wav_sha256": "b31da7adf745d9d55de158864cd2222db4c816f5468f2d4e4b788609cad7e3ef", "output_pcm_sha256": "f52a8ff2b736c2244b20e7a1e6f39701901cc02b40e7913f7a6935e94fa3efec", "result_sha256": "a84ab5f3db2febebf4900f6fec7ef4d552f52920176bd7ee0138967a7cbbb2f1" }
#21formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "9471341cf9a78571e6ba5adcd8c78b7af350e2a62b389e7b59860ff5b7debe4b", "started_sha256": "8f9080289a0b6ab23ae2ff95afd0d23eb04ad426fdb035358fd97a2ecdc680b2", "original_grade_sha256": "7d89196a5f372a71f005b2fd278a4d2498cfad875366fcb4a00b62eb1beee6f6", "input_original_sha256": "9471341cf9a78571e6ba5adcd8c78b7af350e2a62b389e7b59860ff5b7debe4b", "input_wav_sha256": "ae124057b682a764d30faf7e2ade3e92f2870df85d65b489a0a3fc60b6b6a72c", "input_pcm_sha256": "a97e94edd494ed1486062cd32c4af182e6ca788be72ce4bc4731c4cca8715cc9", "output_wav_sha256": "50cd5be7692c606388d73975623c6ceca622b82976b4227fc165b9d9e4c49f6a", "output_pcm_sha256": "f987fceb47e9f6483956259a8bd39d25fcf019bad07d94e8b613333cf0475614", "result_sha256": "846289222fbcc40a6326885ad8caa15e36da03df762b0f7625d360176210fde2" }
#304navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "5c6b14a01f11db35ea379fcb2527dfb4f1105b5e96feb36022180700d5e7b341", "started_sha256": "cb38121a6d8f31e236d6befb5cc9a21c32233ebd69b46b7023e31cf70be42fa0", "original_grade_sha256": "5f4522aa239be89b3aa3ee7da81981c41c51cde19084d9bd4a6fc50da46286bc", "input_original_sha256": "5c6b14a01f11db35ea379fcb2527dfb4f1105b5e96feb36022180700d5e7b341", "input_wav_sha256": "624e9e9982715e1602f151fd5ed2e1b4b931983a78f61589341a086a9a3d3d58", "input_pcm_sha256": "feee316c4d15f3a5f1e2bb79a67b4996ce326f48a610c651bc90068172584ba4", "output_wav_sha256": "c46030a3dc9a58adc836095c8ca6717ee9b47b114d5008328e95c45b6537821e", "output_pcm_sha256": "ef9926823863cf182c58f7c04293233b7773450948abfcb0869867e98e507a8e", "result_sha256": "528968c6b29b3d15fcfa37aa8ab56326585c6912d0b40a4bd6201c8f1356079a" }
#694object_counting heldout	13	The answer is 13.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b86188e1a327c5cd5288e9435b2d8606ed9db851888500864af9f771d989471f", "started_sha256": "140e8fc3fe61321c8dde1f59983d6e57179596ce2ee0ffec32f0d5e3bc05fcf5", "original_grade_sha256": "a7ca60377fa86bebabddd8378545bc751dfdd167cc769cf46f8c8d9c2535ec4f", "input_original_sha256": "b86188e1a327c5cd5288e9435b2d8606ed9db851888500864af9f771d989471f", "input_wav_sha256": "18a3ff923aac1ce8876b11dc568ca672dcc8c1044b7a7ce4cee374048f3222e7", "input_pcm_sha256": "a2dd625fdcff885149b66b7dbbe5089093dc19f63d241d4c2dfbeac9911eb6fc", "output_wav_sha256": "0e35677fd2378f6b947a4399d2af30be41b819bd3f4a9e09ca1c42dd6d5583d7", "output_pcm_sha256": "41dcd4988e686f21f641f6dc36fac4f27f310a3dab1b9a32e678a682b4632312", "result_sha256": "b05b2308424c0a25cb20d97cb0e9d98de29406150380283456e210c019d9ad3c" }
#825web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "658309fee000c4a298783fc9c9fc5e2c86b23e7edfdb642dbe1a23fdaf6b7816", "started_sha256": "95ee75f76c64fe75dff2fe08b2d95c4530006ecfe713c362558af5768f66de18", "original_grade_sha256": "d290cb88fed8696333bc4f13be6d3195181a1eefc7843899823462ead7b4751d", "input_original_sha256": "658309fee000c4a298783fc9c9fc5e2c86b23e7edfdb642dbe1a23fdaf6b7816", "input_wav_sha256": "4e63a8b79602da6b9c953be5153265aa6f68b338c06e8f1033dcc27c3dae1ed9", "input_pcm_sha256": "ad7302c3f1c1fd29deeeded417463d559dc99ff5f55564a6aadfb6bb079f9c71", "output_wav_sha256": "c3db0a7f7e885498e616349de21fe0fe07739e5f34d0477ee81cdbb83ee4c832", "output_pcm_sha256": "3c312ebb4da2ec3e4e7926c00cdb1d96a557a5b731d8a84c6fe87ad7b7b040ca", "result_sha256": "5d40d8be088d047f6a228942e9775ddedd3ee88ae8c518cbc386f0fd175c95ef" }
#162formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "7ac18b867150367aa6c2a500da04d657bd3f68a28594eb50dfc54aab7391d2a5", "started_sha256": "22e9a45577774d3b28f80373147c5fc29079b8c52f4b19e9cac8520a3b656e16", "original_grade_sha256": "411c900a822ec249d442223453960b25b4bb3d12f08641a29aaa0521282b38c3", "input_original_sha256": "7ac18b867150367aa6c2a500da04d657bd3f68a28594eb50dfc54aab7391d2a5", "input_wav_sha256": "bcf7209060818876161114b2b0d8d5e3905ff2a80d663c444666c87c521b1ba8", "input_pcm_sha256": "9f2e5969a1524030e98f478f4c72e91d4816f3b94c6ae3b13eb724b23932abdc", "output_wav_sha256": "4feaa1646368ace22bc43baf36eb2f00c81b6bacfd3a8ef50e0008170e139a2f", "output_pcm_sha256": "b9e582d3d66dfa59080be8f00c0ecca9c7b4240aa1b09a000a34527f77d8c4b3", "result_sha256": "d9abee2058b260201fd8affa3e3cef1511216985a4c5588d464db0fd41ca1053" }
#329navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "2997c814e8eb9124526316e90b81945c2334f3aaf8a42ff1b3334f5f12aff3ef", "started_sha256": "4faef52c7f108c24bb3a2bcac974051ed88cec5532209fd0fbf55b029b1aabf0", "original_grade_sha256": "2663966baf34b2555ca95bffa20c1b990b023ba37503072d828ba6a1d6638a32", "input_original_sha256": "2997c814e8eb9124526316e90b81945c2334f3aaf8a42ff1b3334f5f12aff3ef", "input_wav_sha256": "b43175586d4b0aed2ec5906c1a5ed2904b7d80e6dfcf521f4c0bf78d71edb8e1", "input_pcm_sha256": "2680d9dd8edf6c513701e151afce19562a52db3b1229ac88b736a7eaaf30d3c0", "output_wav_sha256": "905c4bb999b88d40285fc7aeafec6f496aec2afd9499cae7c2bb0ba0ef751ee4", "output_pcm_sha256": "ceba274c8f1df48ab14ca75690bef04d4d9a80268c6c15a7e596b0a313283837", "result_sha256": "38532f77d76488c9b2cc908e8c34b4d336d57534f18cbd6962449f08c6851731" }
#658object_counting heldout	11	The answer is 11.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "aaee5302f5773fb9e55d5b9e5342d98a2bcdfd284c9f8108ffd83ea649c0c636", "started_sha256": "6425ed381e28629a5e87695313664438435124277f375f00edac1ffe56be37ee", "original_grade_sha256": "7e97a5fa0b1e43ee45aa67f4dc6598c8d341624d05308c47961888d071566d0a", "input_original_sha256": "aaee5302f5773fb9e55d5b9e5342d98a2bcdfd284c9f8108ffd83ea649c0c636", "input_wav_sha256": "64ef7aff32b4faa318bc48edd7700ff37b94e00a89e08be8ab46369d73ecb467", "input_pcm_sha256": "bce559fc8c124d12e7d609a441802ba90cab0c03307b9c1236cfc64196c02a6a", "output_wav_sha256": "1d4b064c5adf2e428caee3ca3aa2ea44346a5cc2c5c0b66bd10652991a26785a", "output_pcm_sha256": "463ab30a72adb8817b1a1a39a6ab08ec6be78892d7a5e32a0512580cac20303b", "result_sha256": "83cb6029051b7a37adee975e3667939063f29b6ff6418b8a8d2934547ee20fac" }
#968web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "3faba405dbe5a1b259beebbb3cf4340d612142647ae98fd9684c4e9b8dd85ca7", "started_sha256": "bce4f6fd832508392784a688691508c68c47900f9903367051f5db00d94b6d91", "original_grade_sha256": "19577b74d9a851ff5e3803322eadf2598865463578ef5bbd40f4b30af8db972e", "input_original_sha256": "3faba405dbe5a1b259beebbb3cf4340d612142647ae98fd9684c4e9b8dd85ca7", "input_wav_sha256": "a02355085eb74ba60cedf3695697ef12f33f5579f2ea177f5dea57bd50cf88d9", "input_pcm_sha256": "9b781a990704ad065a89e251ec9933262c3f85739f5cc1a9e608db2eafeffc3a", "output_wav_sha256": "0fbd5317a8cf67f9ab74038c95d72f8ae4d180f2bb33d45818b06d65307494f0", "output_pcm_sha256": "b7d5cf67c275f3e0c814b395a558391b6e7292e1431b397e3ab7e05924546389", "result_sha256": "42b76b61c26f8db713fb334777d9362d13031bbdfb4ff6c8e8ab834644ee2dc6" }
#28formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "187bd9de42a1482624245102f7d52997b55c89683a658cc3b0a41a5486598527", "started_sha256": "ae6dd19e1a10d684032a005e539b263561a145a0f691dc06c7d54064b67156b7", "original_grade_sha256": "3d2a21e8c4a23a8499f8357e578b0c8a7f3bb86d3d31593d4e9f78e3321a0cce", "input_original_sha256": "187bd9de42a1482624245102f7d52997b55c89683a658cc3b0a41a5486598527", "input_wav_sha256": "9e83bc5bcb910a6e88bdaac215dde4bbf0655e9b5886c660324b093aaaa02aa9", "input_pcm_sha256": "e8348f39afaae6fa37d36d9c372d55262f77d7cf110b0c4f9e132c66550e3fe4", "output_wav_sha256": "55fd87850ff9870cab818d428ef79385d183a5b996b46b4b21d1878b28f3de43", "output_pcm_sha256": "7b09dea8e6e08af926a0c8820993bff94068b831e1eb972f2c79054c2e6e775e", "result_sha256": "9c11c37e666ecfa1763efc4501d6324805da94a311f00a5594f217d6b9015e95" }
#451navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "6f97497e47204caa91f9b900b1889f0a552946f925ad0a445f9605b7e7733841", "started_sha256": "0696b988700a88e8e10f25bc613c371711756375b07eafae12c77916ffedb7da", "original_grade_sha256": "382411a6d1d7862f5fcd764d3f6a7fcd7165de20a72df46927b95159ed90032e", "input_original_sha256": "6f97497e47204caa91f9b900b1889f0a552946f925ad0a445f9605b7e7733841", "input_wav_sha256": "fd0f555c0e22e6ada8b711e183029bb735994face01198acfad8db44ee74ad51", "input_pcm_sha256": "3df9c854fb57787f985069053c619877d567639a37645478e8548ed8a451195b", "output_wav_sha256": "10dc558697c75b9c9f01df0751aae9025dd7fe84ac80181dae597b6d22e5be85", "output_pcm_sha256": "7930ea36083352c5078e7722ed71352116803f242f898ec6073bc5dcb5255002", "result_sha256": "292d7cea1f2575abd011c820c274305df0cad1d674396ad5722f57c5746c4654" }
#609object_counting heldout	6	The answer is 6.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "9fa0150e6e9c025a2ddf6cc0ae52e1d3b219fef5507a8291a29acaefed569daf", "started_sha256": "6497ea6fa35a99645eee807957b74d05bda91d7e5713977eaf9518e7e168772c", "original_grade_sha256": "03a48ed82fb27fb1342978bef789851c8dc23fa5313e2698089e1807ba7f7c1f", "input_original_sha256": "9fa0150e6e9c025a2ddf6cc0ae52e1d3b219fef5507a8291a29acaefed569daf", "input_wav_sha256": "17901d181afae2444d7a3cd7dfc0a42ee361c34fba21a12d3f3f10380744daad", "input_pcm_sha256": "029408204a17ec8a9b6c169401c149eaca7bf27d92dce349834957cd5a52802f", "output_wav_sha256": "3f7023f4ccade59937b21109eb8ae9b96d24bddc79cf1484411075ca3cb760b4", "output_pcm_sha256": "e6b9fc371fda70a81599242678b6ee74885c0f8945732a2e20c802df20dc6687", "result_sha256": "619fcecf0880484ca276bf4920bf545fc5fe34ce2c5d6841f9ad9d2351319ddb" }
#854web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "eea935b321930f737934cc63f8342ad7d9c183aee60ce519403a127c5bc6a476", "started_sha256": "c4df31c03951aa97397b4366d6686e09ef2af43304bbdfc41c1fd725bd11068f", "original_grade_sha256": "c8617f3ee3488a360e3aa571d09231b9c726ec7388c386194a7ecf075ed5b61a", "input_original_sha256": "eea935b321930f737934cc63f8342ad7d9c183aee60ce519403a127c5bc6a476", "input_wav_sha256": "cfbf57f7f80f6761636cfc11d715a2a8a6b55f6db13aa570fca7bd5f0e96d1d0", "input_pcm_sha256": "3cccc2ec9ce505e727aaf2f4ad2273217870891d1691a8524140c6ec5b71be08", "output_wav_sha256": "9d940483c1ba0efc8b2753815f50196e04496d262722ee501345a64da06882cb", "output_pcm_sha256": "f3427818afc8b0bd962afc830b86563b472a0a05987009dbab2beec6d16c8fe1", "result_sha256": "ccf39da5323cb197ab313fd64834e9be3125d0d09d14ad165ffd01af6e1f7227" }
#211formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "22513b01feee806c08ac791cd40c134fdcab9c879e2b3f39905f5c810e65f558", "started_sha256": "a34f647f97bb8b6f544687182473b28804316c764448dcafb04c664b5d9a7492", "original_grade_sha256": "71bbe3ab7a5ab661f6923175b44078458c40999f9d2135b1d49bb719671dbc99", "input_original_sha256": "22513b01feee806c08ac791cd40c134fdcab9c879e2b3f39905f5c810e65f558", "input_wav_sha256": "536da3cc62592fe2743aac391eed0cbc0fc5d20507d3e8cf79636f84f162e608", "input_pcm_sha256": "f68bbfa0384b3d8f03541120ec403e1e1bbb672e6892cce798407bca811d3853", "output_wav_sha256": "0221291d876bc0fb51c765e264f18274826c8fd61a85a0b6f0c6beec59ca16ed", "output_pcm_sha256": "2e25d8add805c9bdbe33f27bb3811fc7a3aa7e6d3c6d83a150a6d50b2e652e89", "result_sha256": "8dd92944b88682c0cd6b70822998ae20a3216c69621eccb0021ff5f9b29879f0" }
#408navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "6599cab72dfcd3834bd43980cdb055d873707649e7a174550e63cf8c567b46c1", "started_sha256": "2d0c8fd45bbffae31580202e529b56ffd183e138ec303a976d3a6a45ea8eb1c2", "original_grade_sha256": "07dfeb39baa4688612acc419421a0f07a86c814673ac9dbb87825e5c9e17086a", "input_original_sha256": "6599cab72dfcd3834bd43980cdb055d873707649e7a174550e63cf8c567b46c1", "input_wav_sha256": "b05219963e3bdad6055af2f71549df27fc79e51c97fb94a0b9f9a891b84cdfd8", "input_pcm_sha256": "9c1258f04f5fb69108e32573232d4a98105aee9aca53765a1379cd322140f0bd", "output_wav_sha256": "4a329bd52c4220a7b4e1aed686aa70ea87e7677347955368c8afc2e9974720b4", "output_pcm_sha256": "6a6efd9694278a3a89c2a50792b7702d8bf3af244e09914ddeb963324acef38a", "result_sha256": "e19bb15ae956bf92a5c60e3709991a52a131989672a3a668c02d19d02363ee86" }
#645object_counting heldout	11	The answer is 11.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "5d1b7c38840464ec64414ef969aa59df94d78f8d3fa64edab1c6c52462e8d160", "started_sha256": "d698c52ea28b20e82a196d1ca0ae141d56e3d1d9faa1656717353ea8afb7447a", "original_grade_sha256": "6535b03199d9d89cb22902dd3a74826740d4c587d0ceb1f57f5cc9c4c306d8e7", "input_original_sha256": "5d1b7c38840464ec64414ef969aa59df94d78f8d3fa64edab1c6c52462e8d160", "input_wav_sha256": "8083338306715b89917107cb89360789808e3ac0f2db76e8d271565b283bc3f8", "input_pcm_sha256": "c25c4e334fda094f3614e14df784ea1d7af168643f4f5be18b8310b5f89b3d02", "output_wav_sha256": "1eb962f6888b24297a8fc64b3727fda92ce3679e49adee4e9f3e5a884de3a0ea", "output_pcm_sha256": "12cc1b15c2e20ae42a2e5ab5f339e6e07bee5a989ef6381cf5253bfd8cf444ba", "result_sha256": "8b444c0dfa27e0793e853f6199d230a2da8e0616f30390a5447bfd63ba0659bc" }
#951web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "a4eeb42a174be8ce50de31a1222d329f57087c4ed0688b0f035f1bdddc442a30", "started_sha256": "d331820e5b61a11ddf98f0a4d4b87735ff32d79dd91b5e6f7e98b24fdcb7bb46", "original_grade_sha256": "e482a39fe0544e48cd2bc7a70298885c6e4d6775e3764c70c4f3228a71515502", "input_original_sha256": "a4eeb42a174be8ce50de31a1222d329f57087c4ed0688b0f035f1bdddc442a30", "input_wav_sha256": "83b52b6a2b6d7ea76bd4a398ef8d089040043e275a623f739865b56b5c5df542", "input_pcm_sha256": "373b28b5976ba7eaa41bc94bdf1b924f1eaf0be6122d0549e0b035ebd34cfbe4", "output_wav_sha256": "003f3e6b7f9f2be6b7e0a70a7b28d849576967371633f2f868bd946bd9cba699", "output_pcm_sha256": "b693e3ebbb09569b8f120d236f933edf439e4f473487eb8b3526f77bae456f99", "result_sha256": "dff81799a9ea37a9055ff3279809e85e73df3b929c83194b1a3610579b7f6161" }
#139formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "d5e13f1a17bcd152b766889cd66998e8641bb3e2940922277a6e261cedc5a5ee", "started_sha256": "0ae04dd7ddd07fa77042fc93f59409653086eb95b7d9e45860344a93f0c0f485", "original_grade_sha256": "d6124a6c03ae97c0cb7ebe7eade58639e64ab6a35b7624530c1fc15bce5a5d44", "input_original_sha256": "d5e13f1a17bcd152b766889cd66998e8641bb3e2940922277a6e261cedc5a5ee", "input_wav_sha256": "94476bd8c3caabefa565af0baa32813093127383ee5d80158abc69587f57236c", "input_pcm_sha256": "b9411381000f402d88b575c1216a39a971fda15cb4688e8bb42ffb904d8a72b6", "output_wav_sha256": "c09eb5087f97c979297be1058971545c0a9173aaa3cc611163622fc894bc7acb", "output_pcm_sha256": "c85793aa4efcc865aeddf0c244f8b411e7e5947d8888c865829ee080a6324939", "result_sha256": "094a5f4c646250da521645258b76eaa6ddf4fea096a199ce92a34791686fc4c4" }
#320navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "5ffc1d5e726840dd965b972939a499068da40cb851ae09d35aa25700e02428a2", "started_sha256": "73926977d113dd1a052b4558a1142ea0632a7b87bec8e8c04faae4676295b5f5", "original_grade_sha256": "1067a5ad1edf8824c439f279cea114b3bea1c9ff2f3482ed260079c9fdd19b41", "input_original_sha256": "5ffc1d5e726840dd965b972939a499068da40cb851ae09d35aa25700e02428a2", "input_wav_sha256": "e7ca54754a388f42563e584df65c95167d8041fad568e819215269a66d757c35", "input_pcm_sha256": "40b383ac28068ec23b2a6c6032f9a7eca27328861088dde1adece2c95588c61b", "output_wav_sha256": "fd0278565982b469af14258e7dabf9cee7e83acb0e0efafd81f4e16bda87a018", "output_pcm_sha256": "19347247c9339ddeae84a75a633804b4fdfc830e19afb6b622ae39e45e7a5942", "result_sha256": "59470049660b62b84735b5a904a2670bdc81fa2b0ef58041922481f40638d1db" }
#688object_counting heldout	3	The answer is 3.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "1e0804a2609656d7b7eea1f4ba4c23a244bb1175c873106c206023ef7b1c0720", "started_sha256": "1f6a9dffd9a04088ac5340f31f9b69b13c696c38503e7202be8a206acb12c96e", "original_grade_sha256": "04673e5dc2925aa8028799b697f3a4ac96ff438d5890aa592f7f3a7c26280c0f", "input_original_sha256": "1e0804a2609656d7b7eea1f4ba4c23a244bb1175c873106c206023ef7b1c0720", "input_wav_sha256": "0758fe097f149eb759e714cefeb578027a66262ef500d7113659cd18751a070e", "input_pcm_sha256": "5891c986fcc49f6e0e67da6ef912d05cce16ca0de7cf0343c4d1e852f848bdb6", "output_wav_sha256": "127e9adc0410ed703ecaeecff35df225f42cadf46295a13c538da09096531d9d", "output_pcm_sha256": "bd0eacecc0efa12dd64f1ee2f7efc5e10774ac9673605827daab0d73aefcc578", "result_sha256": "05e9e3c78bf42c223ac62598a6f5b5d1ff1fb141082f916d1631d604df6b64ed" }
#842web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "8594dfbbc05c72bcc495b278ecb8ea195f1cb6bf9ea9e1d529d0fd76d861b51e", "started_sha256": "f8105a8632edc40c1e310bc719460ae917a6603c1e4dfed38ffeb4afd8d75752", "original_grade_sha256": "f0fcd1d19443236f145739f2bd5219fbbd2530bc962b164b9baf7e9416cea781", "input_original_sha256": "8594dfbbc05c72bcc495b278ecb8ea195f1cb6bf9ea9e1d529d0fd76d861b51e", "input_wav_sha256": "bf40064516f311ab07986268c339c2e96c090e94f1506f946269dcc608f736e9", "input_pcm_sha256": "4c67d2cd5a7823e124958396cbfa3982b181a2f9f5b824fcbc9bdcfb8dc8e91e", "output_wav_sha256": "8775db9c98ce04a55c69630d79f274612d3784391cc139304c790dafaa64fdc9", "output_pcm_sha256": "f48f089e81675eb8b2bb73611283e34a28b05bdc22f6d33aeb2ffbf793fa11b2", "result_sha256": "67405cd368321366b7358aeb65e39625512de7df80c90fb9c25cd52332de3211" }
#222formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "e19a90b2dca6d1bc7619fe6651ad306824860b59231219a0d1ecc36f7521080e", "started_sha256": "6d8060a0e28fe96a85a3ea937a3efaba2cdac2e0dd4601c9e4dcb8db10d121c2", "original_grade_sha256": "a87ebbc6c3f83eed7732d0be12e9e1dee967b64bb918fe3f2cdb569d147b97c1", "input_original_sha256": "e19a90b2dca6d1bc7619fe6651ad306824860b59231219a0d1ecc36f7521080e", "input_wav_sha256": "e194739fb9f73985e20f262069cd13ba42f39e79bda271281fdafee7669786fc", "input_pcm_sha256": "fd29c16afdb9d2f26743d1c4784c684fd4fb0ed307d01af011addb0c51276df2", "output_wav_sha256": "dd7a1cd47572ab2289a61b43ef1b8010fdecb0614040c802e590258b95432146", "output_pcm_sha256": "28888e1b4e1e431a20d82343d723e5357f565ef5dad83ab01b789c8fbfe9d50a", "result_sha256": "d5345eb92068175db4714c4c97d3a6946e3d66ad185a9e00b38263dd759cce0e" }
#314navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "334d7ae13b271d3abe1f1ef765414b88a5bb9de404a3c4898b9c328c59e9f285", "started_sha256": "8e87330e8137f387e4a89e7015f69bdbe77bd1fdb8fe4b945226c8e7f00ffcae", "original_grade_sha256": "ad7fd477966eb80b5101ca77d6c7dfcf80b2b2ef5b5fce188a887a7720ef1c2f", "input_original_sha256": "334d7ae13b271d3abe1f1ef765414b88a5bb9de404a3c4898b9c328c59e9f285", "input_wav_sha256": "10c9caaa001acb50664cb6494c15d79ae426217775ab849eee30a8aa23ff7045", "input_pcm_sha256": "ebfde12869329a90bb1ecf8e3819008c2549c6028fbc180cebcce7d78fb16939", "output_wav_sha256": "892d58859b315d324c7dfc930f3cba4f620e5cc82d273506a6b1183da30b7e4c", "output_pcm_sha256": "451a5aca5c8b02b531c71fc036914c2699ef21b3edfd3606916eb464f03ac77b", "result_sha256": "3db567e61f9ebb4307ee0fb1e7ec2c3f512e1281020dc89013fc902417cc303f" }
#698object_counting heldout	14	The answer is 14.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "cdcf9a6d25291f582d32b9af756088596d38028291d1c064640fce2b160fde03", "started_sha256": "6469d77a52d09c3af5115d5730902fcb13e58c565842af09f3b9f42fab0e96d6", "original_grade_sha256": "a6124404c1d1573ff66264230605546d8b96c6c19943e896be1e1e43ce889375", "input_original_sha256": "cdcf9a6d25291f582d32b9af756088596d38028291d1c064640fce2b160fde03", "input_wav_sha256": "154df5e11fc47a0627222e5098bcf4e9fa8280fb6bff0deb24de0f82e1ac46f1", "input_pcm_sha256": "4f2e57fd23efa5133db69f0621d4eb97e1bea143725577766b5348274f5f4216", "output_wav_sha256": "2e46793dc9d51d4abbe721d33ed43a45ce3e229a9f653b22fe2ac476b0ff3254", "output_pcm_sha256": "3f2d13853165266599d885876570c9968448fbb5593ccf246acec9760dbf3699", "result_sha256": "0ec759e44ff8c339f15f0ae9254664507417dee1c20fddf51f3d577c70b1a2cb" }
#881web_of_lies heldout	Yes	The answer is no.	incorrectstrict_spoken_answer	incorrectresult	Original grade retained Evidence hashes { "original_audio_sha256": "a557e807b4260b44e67567a69d55227f02d9bb16c66b60471cecbdf010869248", "started_sha256": "9e20da94c416caa515c09fa7886c17efa25282c5b6e9af05f605d2de9e62feef", "original_grade_sha256": "3522530027f47a6b9cb84f68889f1c6d0ecfa72746a9af00690a98647138a45e", "input_original_sha256": "a557e807b4260b44e67567a69d55227f02d9bb16c66b60471cecbdf010869248", "input_wav_sha256": "e6b709468fbda877c1b3e414b0ef271f66350e36b8e3cf86c8fcbdad409bc534", "input_pcm_sha256": "8349689228a78279f623a6df46bc9686d22a734938a8257b68565441a641e3d1", "output_wav_sha256": "2039f1f11b370082b6fe5e11417f6a6c16a6ba6bdf108b2a0cdbb5df33936d14", "output_pcm_sha256": "7f67fdd79928cf99804247f58e1ed31a7eddcc526336422cbd9e8760f577e690", "result_sha256": "4e8fa5913308a1338c2038acbdce244d7210749c48d4bb586a0f31370e4db90d" }
#144formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "e7d315f413151c817ad8b2eaf4333a6ce16345c9bbb067c1ff8e98da0890a389", "started_sha256": "928bf3e47bff0a50038835263c944e0239072986534d200d6331c53663a52328", "original_grade_sha256": "3198a7385219681149a2a3da4d77d8b9727ad4b6bc163b96cd1319d890684ef1", "input_original_sha256": "e7d315f413151c817ad8b2eaf4333a6ce16345c9bbb067c1ff8e98da0890a389", "input_wav_sha256": "e04ba7629b8ddddb16c66026f6b29bbf0c3e4f509638eabf35605f0fcea0e62b", "input_pcm_sha256": "7cbfcb89aa90ed04281cde2179777d5fa0639259cb5fd85befa02dc44510e187", "output_wav_sha256": "57b8cf01f7e8418cb71fac849f656cb44f9d79f2b3ca3ca8412ae8c6563fc7e2", "output_pcm_sha256": "f3338af1feb8e328d81f54c7da76b6762f5e7a895338777ceb85de4de40dfca3", "result_sha256": "c6f5754ee3a50e245ca520efd63f6a167a4f84b900a3ea7c3fda7c399716a001" }
#268navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "56ad923b80f6edb5bbbb75321babb00de1e38bc939c5af610cbed474f58c3467", "started_sha256": "0be189785e54de6d6c97de747136adcf74fe1a708357afbc20ae6a64773d028a", "original_grade_sha256": "ec15f238ed348b9a31c8cd0b78cf2acc0f26a2719f11a598a76282b72767961c", "input_original_sha256": "56ad923b80f6edb5bbbb75321babb00de1e38bc939c5af610cbed474f58c3467", "input_wav_sha256": "97c4c3ea822e518131a00f6f206551b777a6985ef12df2034d08286b01ee33eb", "input_pcm_sha256": "6f2657ba0f9d78606d48cb9670ba904498b69dc3db22410005c106fd7d6bb11c", "output_wav_sha256": "4b15504d175731f644164b6c3dbd50d6c7673d0b409aeaba795e0bcf0f7464a2", "output_pcm_sha256": "aea5644c12eb7802781f02ece48724e9d12b3bf8bfde67b2f2f41255f8fb4062", "result_sha256": "bd1dc72c3b5358fe0e3339336c153f3aa6c8fa15d5a4c8f048f49e915f5f2fd0" }
#501object_counting heldout	15	The answer is 15.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "e206bc7b8085079b237851c87bba6fbf50732765a084b30eedf5bbe801cecab2", "started_sha256": "576d70c6b759bc94c794bf1ef751740af7536bbb1f8aa83ccc2b2946cdb3d4fa", "original_grade_sha256": "d1add1e626ed9b1df5cec2c9d2ab7f0d182dcabc68d48d8eb1743c64f411f95c", "input_original_sha256": "e206bc7b8085079b237851c87bba6fbf50732765a084b30eedf5bbe801cecab2", "input_wav_sha256": "d52ce0b3d4a9eae1e3edae88a25666599b177c74d3f0075af552bb6ad753b9e4", "input_pcm_sha256": "5320346d1c126f40605b603dd85248ffc103177d85d04cec911006ecf42c33ef", "output_wav_sha256": "15654ae3772f1bd3bfd15d7da7866046c4bde05f8a44d870d9a2d599b37e884c", "output_pcm_sha256": "5f3f6fd3248759e05fcda353bee5c927e05a7dbb4beb3c9cdcbe59ccf162daa0", "result_sha256": "d2591705279a39cbc3e56f1f5e81fb3fa015c74f6a5821204b8a36749cd06037" }
#874web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "ac89116e40e9c2f768431bbea577de0870e820c5e33892ff71bc56ff0b707b1a", "started_sha256": "b18fa8f054087d6c5c31f5f7b5b7ee03ab0e94ebda3dc8f8c13d9b53619e01e4", "original_grade_sha256": "c3dc37d64cf4f25c2b1332cc7ebe576a6bcf1c46d1fecf7ee26047e70277ff69", "input_original_sha256": "ac89116e40e9c2f768431bbea577de0870e820c5e33892ff71bc56ff0b707b1a", "input_wav_sha256": "b2f1f7e4afc890a8dc2cb3bf7e5b9aeb075698ea66504d03a4b08d167e1b6d1c", "input_pcm_sha256": "36b849067a8ba778da55b6ee3574e567c10cd854d2aeda73daa4abbc9fb225bc", "output_wav_sha256": "8283fb647552c13288e9fc90a14afcf660199242d017730443b748cffbdc5842", "output_pcm_sha256": "5dbc16e88e1cf04652bb4fbcb7ab681d121684b92496ec01d8233bbae9a88190", "result_sha256": "fc4b04946b52f72e9ecf178db13067cb0d86362707a2ad81657580f308104c59" }
#123formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "44af2c592f479a5fccfd85dec73cd262faf14ffc8573cb16ebd3232952a1e374", "started_sha256": "e406d8eda5ff993a28eeda4959eb977d5c213c837ab1d817d0be348127eedccd", "original_grade_sha256": "9ce79caac2eac2649ca2590d80fe0247c4cc002f6e152b481dbc3cf90f98b833", "input_original_sha256": "44af2c592f479a5fccfd85dec73cd262faf14ffc8573cb16ebd3232952a1e374", "input_wav_sha256": "abd8c12d9eed4c52823129eca6f08814fb7484d9bda7285a8c700ca3697c6cac", "input_pcm_sha256": "656f5dfc236245498f427a57329f8ea04a06e256924bee796ff376cdd53cc175", "output_wav_sha256": "1c6f15a85a56729ce02e9fe1e822ab4553f0b83e1f4951c254e79d22f370740d", "output_pcm_sha256": "99bd8c8a7fec56d579b42af174164082914fedcd4dbf53035d2f2f64f2c09c6a", "result_sha256": "06143f1457cd27f5eaf374b70d6dc038554015f045bce868ac9e4354b27be7ef" }
#469navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f8029ca155d2fe315433a6dc8eac9c37be121b8ca8978666f9ab0aa64bd87e1a", "started_sha256": "9c71bfae7166bf66ee8e1b519a30d23a022d8374cfdec9d396482e3fba1a0e87", "original_grade_sha256": "309bcf5eb91656a3eec37a8978383fcea1fceec51fbf7d7072c04b5f16f01902", "input_original_sha256": "f8029ca155d2fe315433a6dc8eac9c37be121b8ca8978666f9ab0aa64bd87e1a", "input_wav_sha256": "b79359208191ff9f6ff49baa7d09db46f16f4e4aa6aa03c2a1209cfba1beb239", "input_pcm_sha256": "0d9226aa1941ba00fe9c1126ca6e0fcca2e7114450b86d4f0b8f474673c5f615", "output_wav_sha256": "b6a4ba1dc9a9e7f51a850af30a2ff4013bdd45060e6b985fd747c263a0d33a5a", "output_pcm_sha256": "9a36765f6ea56a9d1d96180e149ee773ca9f1f4cd70881860e07d3e2f5e17110", "result_sha256": "1feeeb9a9afec5ff62615981321dc614055deb82fe1a695dabd9199e00594ede" }
#732object_counting heldout	14	The answer is 14.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "12c2fbebd023790d6ed0a0e0547cecd46f2295e9bcc1f4086dab42ec916d8ef4", "started_sha256": "e25da809bcb855ec188da1151a80d7ad354ae8c78bc8ed5de77ef252780561cd", "original_grade_sha256": "62b723ebd7247d0bcf65770d06e594bcbb8a0b27914d1f73a68dfae7615182d0", "input_original_sha256": "12c2fbebd023790d6ed0a0e0547cecd46f2295e9bcc1f4086dab42ec916d8ef4", "input_wav_sha256": "b2493a6e87941e7f2334daefe8b1d5ec9e66c7c7cbeed4b86cad8ed8f2aa2de3", "input_pcm_sha256": "b4525c4928acd2d465a86926d0ca3661c2d2d077839b4d1da277fcbe2bd886fa", "output_wav_sha256": "16d615992c50a8f0512f8f48041ab24a909c480b6e153457de68286f9cd8a7b3", "output_pcm_sha256": "24665890f3850afc2e0703b9467c510a9c98472655064c182eb2badc438d89cd", "result_sha256": "9b79445d65f107c083f6066158d6f88864f9c9e98462e09f2bf870ad6a7eee5e" }
#926web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "084c93ee3b67b2e64b1eceb92564939c60cd4e8bce52db8243a510e375ebc0d8", "started_sha256": "839f07562d6aa52e0b5456886f353be36c4c6e4079b4a0d591d1657b43acbe6e", "original_grade_sha256": "484054997466f23469ce7891400af6db46cf8fec5f3397463596aafa494c2f98", "input_original_sha256": "084c93ee3b67b2e64b1eceb92564939c60cd4e8bce52db8243a510e375ebc0d8", "input_wav_sha256": "ce951072c51765f5668d63bea8669fc0702ba8d42d374686da910acb29a1f579", "input_pcm_sha256": "e0cb1e2faf590c214287c4f969363c6bd56c6b924ef5dfb8578422c0deb3b6ef", "output_wav_sha256": "1b696a40cc3528bb7fe0ab054984c38aab656e29f5bdb3d4ecf305846e748b0f", "output_pcm_sha256": "1e0660219a4cb3625d42b72aa3d49af3d2633ea0f9c25712d27dd6d8ed09ec6a", "result_sha256": "0ae69864fead4c5b7ce73a6cf6cec0026a5dcf9a8698c3f09a8843f53d5159e0" }
#47formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "91cfbc58a28f8375295a78612a83c9a0497db9c78692fe3b02d0530f000eb753", "started_sha256": "5cd7a659f974bc99a0f671f0ad2b67a1dffc2aded47d78adab73dcd8eb0b55f5", "original_grade_sha256": "b60334f087489f579968b13fab28224c22bb102b58b0f4e4765dd55aa6aa1abb", "input_original_sha256": "91cfbc58a28f8375295a78612a83c9a0497db9c78692fe3b02d0530f000eb753", "input_wav_sha256": "59b504ca705a96e089eda454f3221c7b018745340935ac9aaac82aad8bd62322", "input_pcm_sha256": "57a067823e8a4a65c58a8d85a2fc38ae1380532b12ffdf9e2de16d5c3a2a6d7d", "output_wav_sha256": "6b090f55c9d59d5ee2d6e15d350536fecff84fecba5e900d811619b389cf9698", "output_pcm_sha256": "d629a27100b6159171d97449d9eb74ef7cf737c8856c4f6c12c92f9d30f03bf2", "result_sha256": "888f51420af219bedcbaebeb202fa36b06b2bc965852541359f83c620e33d25a" }
#277navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "1f3da6c7367d4bc769beac65e926400de4dca833f6b6536e004af458ad941244", "started_sha256": "2955ab383d8ba706a5e96ed7026f99b4312a068fd4be9c6c3090519c43d7eec9", "original_grade_sha256": "c90ee45e21a3d38bf83de679a4ca67df52ce0fe2bbf1b1f803e264b20d6af921", "input_original_sha256": "1f3da6c7367d4bc769beac65e926400de4dca833f6b6536e004af458ad941244", "input_wav_sha256": "bfeff64fe91e7ae21e24313ad51163d9e4ab1776d3de0115579e32c3c9a35529", "input_pcm_sha256": "aef9f79ade982a127bb34c0ede5601f6bfbec9f4b77dfc156ef2480ddf11192e", "output_wav_sha256": "0d5ab059924bbd184fd82ddb06b6b3d42159f21ef4ab9be45f506c703e35ff0f", "output_pcm_sha256": "5093baf6575cf176d674aa6066dd859a186b232dc06e7c37505669e86223e107", "result_sha256": "a501d6f34df2828734926a69b2a57b7e55d4ee0ebcd147073753d1c6624697ad" }
#722object_counting heldout	4	The answer is 4.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "10def79e963bf2c92f8e59133e0b7c1000a8250881d269776f60497f623b8dbf", "started_sha256": "fdcd6f766d1a33edeb33311937e10f1e9b411a92f7b2876b977ddfcc7a01c748", "original_grade_sha256": "c5ca48499d73bb97972720a566a5e9e37760b52a43cc381ca1aa8f8e3be52bc5", "input_original_sha256": "10def79e963bf2c92f8e59133e0b7c1000a8250881d269776f60497f623b8dbf", "input_wav_sha256": "3274608027681bbf5823a1dd8897e3df68e5e41c919c973b0983c733511f943d", "input_pcm_sha256": "4b4a606249f8a7f91bf8c56ed144b133c39d9d6213a18de3b559b144c85ba90e", "output_wav_sha256": "ceda600eae997a97fb631961bd2532f3099e830ed0a499f52f436695a818ba18", "output_pcm_sha256": "b92df566ed2c501273348e08a87cfd98da2bcad9cd5a11baddf2a043a6b16f1e", "result_sha256": "2fb7b221f8e97cf277a3202a5ace11509fd5f05a8810722b2d0f481a85fb4742" }
#866web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f289ccc1fe092c139930764c0092dfb6c129b7f8c5dbba952109e2a2ec0ef229", "started_sha256": "2b90579e5a286f06e8b9dcc17ba046a442e8d0624924c7e8e201d696c4bd1463", "original_grade_sha256": "7670c7eaa157837b7ebb530bff17d94ab19cfe5853b5bff18e6e54e5e48e3732", "input_original_sha256": "f289ccc1fe092c139930764c0092dfb6c129b7f8c5dbba952109e2a2ec0ef229", "input_wav_sha256": "c79a164d5d5553b4cfa6d67e546cb79fefa07c3aad75e47f8a5a38315c092e20", "input_pcm_sha256": "74adb4c2c326b695aa43541631b21137bd2c9b532fb2ae5a9d5737ee1f4de3ad", "output_wav_sha256": "f00ad21314bdcec1896f7a32f433344e069400abe1dcde0bf01eb5a77e18dc10", "output_pcm_sha256": "3aeea300dcbdd4872d018efb7668c0baa6e9b44b01db810b249f7ae84b131703", "result_sha256": "ab6c13c7f0b4babb0823760b6f6e35c07c669934fb0fee0de37768f2292c6ee3" }
#148formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "1b1749cf15a6b5f0c5f92db2f9976d068cab9b3119a93bf9129a6fbc82f6c4dd", "started_sha256": "d8ea98c39e664fac913ecefa0589e53a059cdb394382eda235f0e4b8ac2633b6", "original_grade_sha256": "a008c3d5a20dff1a094cb1bb2cf73539fe5a4c3fe5714629d821f81c0d146434", "input_original_sha256": "1b1749cf15a6b5f0c5f92db2f9976d068cab9b3119a93bf9129a6fbc82f6c4dd", "input_wav_sha256": "4b0cbb4fcf24297282de990739a2dd571d90969b3b625ab97bb97614efb8c52b", "input_pcm_sha256": "6e85735d687e2f1b94ac47b4ae0a771897c24f17fcb85aa66590a63499443f48", "output_wav_sha256": "e646b1ca108d032326ac8385f9c99d0412486e8852f2ace38dccf96f4fda91ec", "output_pcm_sha256": "31136d7b55660c1e34690013719863c7499b8a333d7efee223d9dfce2db3d360", "result_sha256": "adf5b763369db8525f96960e27f7a1b53e9ca37b5353e95d949539c0265a5b40" }
#313navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "d0cc96a9bf5fc2325e4eae542f24e0928771f365061881a6591501928734cb3d", "started_sha256": "a929b8b2a1722fef2c43398e84f156a4b772afced755b947c3cb25e2aad3e6f2", "original_grade_sha256": "f2ce258fa6afa659b84609bf4bc610f53c2dcbd524ab9df9cac69875e6fc024c", "input_original_sha256": "d0cc96a9bf5fc2325e4eae542f24e0928771f365061881a6591501928734cb3d", "input_wav_sha256": "98e830af5fea5da2d8b81f9dabb0bee566874f978fad84a5caf0b89b536df950", "input_pcm_sha256": "143bc9a10fc1d4a1f619c1a7a38ccffb8fa30d3fb607f26c38cec9c0834ff0d9", "output_wav_sha256": "d30a389e28ff7fb8314a5b6eee2ca1b634ced0096b93f2203923a9e9750930ea", "output_pcm_sha256": "4a8b91b7c79dc6e40955ab4100b0c595a8008c1a3617a3dbb67c9a4aa26ad9c5", "result_sha256": "5991ee4c71fbcf0220c2b50199909784ee930ecc6a611b01f1bbd2f698f19b2c" }
#669object_counting heldout	6	The answer is 6.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "fb9f799d51d5ee42e2eca278f649780f49af3c9f526be81c641944a13b9cbd8c", "started_sha256": "2b0f4b9dcd975b97dda7160f2e53d2687c5bfc692eee198d53090c3973994429", "original_grade_sha256": "18f336874784c2e36318a8b6e30419da529fef3fed8312ffc45d633f7251073a", "input_original_sha256": "fb9f799d51d5ee42e2eca278f649780f49af3c9f526be81c641944a13b9cbd8c", "input_wav_sha256": "10ef065c520438a6ef7379d20c396f8bc06340c598e27cb0d2e31fa8f67dd194", "input_pcm_sha256": "a629d4a4d6ea648c0be11d0324cd944a73a8fe7eab56c2ccc4296ab40ee421aa", "output_wav_sha256": "5d8166c552051c07df9a5459da6af6919487c8087387f1263ed4342e552a9e32", "output_pcm_sha256": "38f3c16276ecda698d819e98b9454ecd798c0f287296817f00c249784ca7a02f", "result_sha256": "bf9b1a81f92cec88535992fe8d7ee9b3c979befb2964172a7ccc9d08fdf233b8" }
#798web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "d099e13359ff06cc79b4b55447b287a199ad4f89ee646b640168662292027c3a", "started_sha256": "d6ecf99ec8e1a1d8e0f5aecb93bdc1edb4dc25f577e1e62fad103b83d41d79c3", "original_grade_sha256": "91739c2debd7ca2cf79762578bcbb4a71479ae27a103fedf7c9ac9ee2dbaac91", "input_original_sha256": "d099e13359ff06cc79b4b55447b287a199ad4f89ee646b640168662292027c3a", "input_wav_sha256": "aa83794aad36858f83e42ac8b6261ad7ef203bee2e9e0f9f8c17ad70f4db616a", "input_pcm_sha256": "36640cae76ab622661f945d4340a872690f965b87bed374908dfa5e06e539c2b", "output_wav_sha256": "e1ebc67f2822116ffc16505a6cc57ec63eb4437fee62df94919596af0703b680", "output_pcm_sha256": "383107446beb89e20d8f214548871816ee29b66289ba68fb65fb34542914a4f5", "result_sha256": "319e266543aa8237c3b9ebf4f05424c0e207a541f489138481b49bc4ea3e27ac" }
#202formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f62162f9187c2aa4ce0327b477cde5f5f26409e4bbd3457f03517f767738b433", "started_sha256": "1b16de850940bf16a2db6aed2a4ddce47518c390dad7c61cd7f28047f5d425b6", "original_grade_sha256": "d3bbdb8d6db705a94952eeb41b119f4096f11b95de46b2f7ee667f9875bba98b", "input_original_sha256": "f62162f9187c2aa4ce0327b477cde5f5f26409e4bbd3457f03517f767738b433", "input_wav_sha256": "8a2b8e4de1071d857cedd341aec860d0759d72b7a6c9b4652fd6e5b3d1642173", "input_pcm_sha256": "b9eb16c71c5e8e1d44565f9a56bbdd8896dbf3973f7a66de8890b5545fac2385", "output_wav_sha256": "75acfd622cca16d57e78226bf1d48c86a47f6cdcb09e4a80ad9933e30a31fcbf", "output_pcm_sha256": "36f4d1c04e5dfc042c2f561292fca674934191d28321fe64c277f28473b9afd6", "result_sha256": "2ac6850aef081333bd3491ca7bcae847ab0e6534c8eaa8ba093467024e2a1c37" }
#491navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "1b398dea7020c564ce10fa87c478547561d7556e0fa82570ee37dc9debacd993", "started_sha256": "d27b6395680783463d4a8e1cf19d631f9e6c609784f7921939dc134b18111471", "original_grade_sha256": "855f5739bcf1e27d3205798f7d2b93188008c49e66c41742d71433b1f113dbc5", "input_original_sha256": "1b398dea7020c564ce10fa87c478547561d7556e0fa82570ee37dc9debacd993", "input_wav_sha256": "a6e2a140a6fcbba36db7b1742f1ab6c762f7b5eedfbbb5523866d19c622fc69e", "input_pcm_sha256": "3ad77d99cb277620a63c39c860a26a2a6b94775aeabc021af8c766200f505c08", "output_wav_sha256": "c8206f1154f68bfe3244f1e3b3d1b7e02fd4e23d902f5608ef30b3afc271d281", "output_pcm_sha256": "6a97cecc4a01ef4dfb33ffa377579b46dbed826fc38f493d8863becdb2751626", "result_sha256": "9e6dc8fceb40afb8c86726d3fc178276ca024d963f682301d634f42c23ba4105" }
#721object_counting heldout	4	The answer is 4.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "a11347be22ee97e3bcc15b9f199198b32522a389038ef754e3fa41586f9bd226", "started_sha256": "74ebaf0e0f018990fdec8475b8a668b8422971cdc5e0a923ba8307ecec8c89ee", "original_grade_sha256": "a533f93aaf37f05b554b8ad72470985d37a840c2c8d2b0ea5b6d9204c6326a69", "input_original_sha256": "a11347be22ee97e3bcc15b9f199198b32522a389038ef754e3fa41586f9bd226", "input_wav_sha256": "f2b25a70f27debbe59d061b26f48cf5edc2cc509ff60f91026b7601153d96940", "input_pcm_sha256": "c7135211598d5d4021eae400bdc3d3a303f71f6874a5f2abecd15049897475ab", "output_wav_sha256": "c7b5a465fb8ccdfbc5aa69fdebfe66aeaa448311cd0d98c4af1d2ffcb857deff", "output_pcm_sha256": "150fb4922871337e22fc2b4149a347385f7d4b405cb28afc105eb68a1ea5087b", "result_sha256": "86c9e7c1409aa4e8e8ccf297617f551895629ca4a3ba83347639d46fd866fd1d" }
#794web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "1e60a621ced0a2f016095ba4e1879a1f2e24bba46d7e7291619f8e5ffbe1705e", "started_sha256": "344ded24a40cdb05d543e7c8edfec865006e10dfc62f0eb40ecbf735dd53ed98", "original_grade_sha256": "627ba27f2b93f86648119348a90b7230561b89ff2f970d73bf70b7028036455f", "input_original_sha256": "1e60a621ced0a2f016095ba4e1879a1f2e24bba46d7e7291619f8e5ffbe1705e", "input_wav_sha256": "aefdfd455d12d09d1c60dbb2451ffdc10e0ffb704e8e5f33534a9838aa51a70b", "input_pcm_sha256": "7869789e88fbc9f589f01164f4167bca5da7fbf1c46134782e9d99f64c03c008", "output_wav_sha256": "0e0ae874472405bba55c6bda265e864bafb260012e9e4a87dd5126e288575bfd", "output_pcm_sha256": "9b2bb719acd93118bc028a5432cf8bd72c89fc2c7c46ae83112f911a0e169206", "result_sha256": "6aed8eda55ac9aeb8943e37b38360b702d2e83f810cee4d26fe4d6fd11b790bc" }
#92formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "9f75bd05a7bc47fe85f006aecf84bfa4a459b295f92f9df326d794aaa441e9fa", "started_sha256": "c0ca1362a7eb60918447e2a521a637f4d77c13bd76bc5f69246d1237602a9b4b", "original_grade_sha256": "e3edc3a52a1fd78b655b0ac6d82096fcd9f923907bbbae1f66df8f28528ee594", "input_original_sha256": "9f75bd05a7bc47fe85f006aecf84bfa4a459b295f92f9df326d794aaa441e9fa", "input_wav_sha256": "af98a2f034967f68a2f2af2a7d41d7627699dc1b9cc4018940193edf14f85adf", "input_pcm_sha256": "e144c4d42a0a20414108a5a417af89571b8051519ed40d908eb343907a347c23", "output_wav_sha256": "c28e5b16b64abb656a4e3cc63330afb462631b5574a69e88f3c3d8206b24b9fd", "output_pcm_sha256": "6554f0c3a35b32f498ce379102597ca4448340458fb08b7c68dd15d88a7645e1", "result_sha256": "337c209993b0ea5dbdcb414a4ba3e6d66106ec53fdd67e019062d9ae8d4c2b75" }
#442navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "2ea9e558989af6c1f0a449595c6ee04a16252ac90dfe4b8254dad6d7a089f897", "started_sha256": "974afbe43f240983fdd9769db451577603638d77494222463191a01b7190335d", "original_grade_sha256": "492e53c8f2e74c07c000693b43c8a6d5349dce42413b60966ce982dafc307cd6", "input_original_sha256": "2ea9e558989af6c1f0a449595c6ee04a16252ac90dfe4b8254dad6d7a089f897", "input_wav_sha256": "36d104410cd884cca522c13a6a0e638c087fe88410e24f3c595251c920742e32", "input_pcm_sha256": "8cbf5f23568e648f602d55f5232e28a606a4586397ba56ced01011dcbb4bfef8", "output_wav_sha256": "3772b99a87c85f93b04015e693a925f6d12d138ab45b0b88fd94854612bcfd70", "output_pcm_sha256": "637567cc4f78045dedfc9802c37f6612eb8b1fa96b83455b37d185acd8a72f6f", "result_sha256": "6e95e4dbddfe8f84b3872112a4d5859508140ddd01846ace1122b7e8697c9b6d" }
#624object_counting heldout	13	That answer is 13.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "c69ce1fb8cf62644b5254906e0e8fe44c0091d5c102d8f5d8254c970685653db", "started_sha256": "2270b0960a33bb30c2f71aa48383944d4d0d8523f1d097b81b7bd6cf9837c3da", "original_grade_sha256": "0cc6af09398d2f68b7217fcef77a1ac5beff3566e3d677c0bba0d5b39729bd14", "input_original_sha256": "c69ce1fb8cf62644b5254906e0e8fe44c0091d5c102d8f5d8254c970685653db", "input_wav_sha256": "d3143fee2414949f24915a55c7fd20ab3384bf6e8a64b9b9287d7fe4cf69dee7", "input_pcm_sha256": "6668e5ca309a25e510a6684982642e854813bd69e9f56dd2a83f5fbd9ffb1f6b", "output_wav_sha256": "ce4210071864568557e32af360c1ba4333d61c35792d172c8fd177be869477cb", "output_pcm_sha256": "09d06761e07f9fe56592f33dce06649ac457e72b44ff2fc2c44e597377f1f3e3", "result_sha256": "87efef00a5038d4e1b079833881864952913cf6920c141205c6d76ce2fe3c40e" }
#814web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f122062507edf00a556cd80c7302850bae4b8d14838cee41b1858ce3fdd95a99", "started_sha256": "d272e948b5b56159a4971c0a84e273f9280cc4af06fd2ff8589d4a6256393305", "original_grade_sha256": "5afbc56aed23f0131fc7350d2c59f7a6b7f09c3f1516275b4327431fefe621f6", "input_original_sha256": "f122062507edf00a556cd80c7302850bae4b8d14838cee41b1858ce3fdd95a99", "input_wav_sha256": "dc09b4838254e6e33a115a97057b737e8590e532cabe01e6f0c2c66263ad688b", "input_pcm_sha256": "7016523544129b0bfc99eecb6a05073bc82b8a6c6e86d83ed868afafd0e76667", "output_wav_sha256": "3a533be9158c7817001aff797f8c224079b04535c3a0a34308d21b3cd333948f", "output_pcm_sha256": "eda5911db1d3eb81d9998a92a9cd5b0670429b3f445cbf2d866a3be052246052", "result_sha256": "85852666ed22d7217c902af46850d6b66b66c8e9d0d8dd1d2cff5382337bc89c" }
#169formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "d3b74f42da7dca1bc4b90065d2e61893360a38d5cc7c3a74543d79cc4d641f6d", "started_sha256": "a82631b10f2e199ca4ed95690c81e3b3c331143c591af475d0dd925c301aa193", "original_grade_sha256": "802721d20f783fb7920fe1e4e4f49cb243d1884708e4a6e93112b4966b28fe36", "input_original_sha256": "d3b74f42da7dca1bc4b90065d2e61893360a38d5cc7c3a74543d79cc4d641f6d", "input_wav_sha256": "08a5a331d9543eea64bb77d6f95f1dd845d59ffee4468cfcbab6cfc263d8c2ef", "input_pcm_sha256": "9b3b09f2ab98d6141c80bcc341e02f834d8a0dc8d29374652ff6d20e6c95b8e6", "output_wav_sha256": "d863c073e0ce80e32c814e7257c4a4431a7d5f318be600026c0e1a3c62322a2b", "output_pcm_sha256": "1a0f946a3de7208af6f2d0072329a5713f8a58775be51840fff18ddc0ea096c6", "result_sha256": "dd25de8172e85a7540d5d8f59d2a60e6d77febd4f2e717bb38a532ec2c39b0dc" }
#286navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "86ffc7b721a37a45a9ce425e1c7f78d0d741739d70d1c23f96029d34b23899bb", "started_sha256": "5138196a278c275b53fde9845ac02d1a680859634a0d6a59daa3fab41bbbd3c4", "original_grade_sha256": "67c810967c38672e0c4c2422faa7bbee741aeabffa70e36a38efeb27e75c41ca", "input_original_sha256": "86ffc7b721a37a45a9ce425e1c7f78d0d741739d70d1c23f96029d34b23899bb", "input_wav_sha256": "dca250802db828e458c359315b4c0f81d73abbf0dbe2cc3ae09890dcd1b5588a", "input_pcm_sha256": "cb5491da0fdd558158db404cb28399f1f5adf1dfb222d13cb9256e0053b468e5", "output_wav_sha256": "1ae4ca67ce58aa44fae5dc6ebcff70bb62f0e1ddfcdb265fa940e1b396f9c1da", "output_pcm_sha256": "8f4f7d45be4d3d3eaf5888e6429833cbbac1c36307b76f8e1439d726e91a0b20", "result_sha256": "0706f1bac86d7442d13ad5ba6371a0b527eb79c2d628e074f0c5e24eeb6ca234" }
#683object_counting heldout	6	The answer is 6.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b4f7f4edfd2e73f9cbfe40a02b1de3404409255f998de7bb9bf23509659de7a1", "started_sha256": "8f566a51a0607a2332d09e5ae9c56164712bbb60dc003c9aa532e0daf2fa8a4c", "original_grade_sha256": "84eb6b9ea02c026a24ca1ce0f8b76330e94530ee008eedcfcf854b20c6e38a68", "input_original_sha256": "b4f7f4edfd2e73f9cbfe40a02b1de3404409255f998de7bb9bf23509659de7a1", "input_wav_sha256": "7f88118ea360cf1e480c1dd4c211a2eeacc9a5cb8c54a9a3cceb01bbcaa90637", "input_pcm_sha256": "0e623da81bc7b72dc2a42deef76e4a0dd76162adcb91ed0e00e613d2d29984a4", "output_wav_sha256": "78f9fe1d38f4095ecca7ae550c71fd75e27d30cc68f964834d475a8072ef0387", "output_pcm_sha256": "b71b6a77777b6f460afb64b68d5ce0f2df37fae13cb84524116084ccea57bfd4", "result_sha256": "986bb67ad4f2f3168f163c730a78e9e4fbea2180a2a3d24dded944a400d61ca7" }
#900web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "e0e55a2b8f6722a49b850b69152be3f24108792307464d275c4ba5f2f07a026a", "started_sha256": "d170d2dc7fab50552b2d9280383f660f89ea85447f44c410f56803f6c0498f3e", "original_grade_sha256": "dcdd8e94fe146bb9b281b672ecb86eff4ecf50c6c35e733b9e0957b6d35ef603", "input_original_sha256": "e0e55a2b8f6722a49b850b69152be3f24108792307464d275c4ba5f2f07a026a", "input_wav_sha256": "9cf6a513163b7510589d34cc3ae32e2034da7c1eb4c1563b11012ea6bae144f7", "input_pcm_sha256": "66f0f2c31a0a66751a644226614fb8e4c6b14d0db80b9d797938a0fe2e669045", "output_wav_sha256": "e5ffa48cc3564c9ef1a6fbd4c021d7bb0c85a709555aa87df99d55e9ac7a1ff1", "output_pcm_sha256": "e1365eff97eb85e0bb94713183f639046a39c61089ce89ba7732fea5b47dab07", "result_sha256": "7e1a543acba47504cb7d125ed2da2ebf5898ba853f2f2c7f543ffb642c108853" }
#27formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b4a311020f575c57b1278da5a117a7b0e82b1b22be458e9665fe601c3c287876", "started_sha256": "933f538cd463ab5d26e1927d4b3073d3ac873c20759c9a01738a3284bdfda555", "original_grade_sha256": "105acdbbee86ac5c354ac3231e6141817a4d6ba527be45c0771177f83266bbd3", "input_original_sha256": "b4a311020f575c57b1278da5a117a7b0e82b1b22be458e9665fe601c3c287876", "input_wav_sha256": "a8e6ac1a692c2ba1be917c55df10866d335778fe13b1a10427433d2c205dbfab", "input_pcm_sha256": "be6622435bb09ad80f9972658fd6feed1b6a65b9540f8323fb9476356677f1e0", "output_wav_sha256": "444f21c07489b5508d36b877f8864c80e7b5ad2fd7a026fdf331552757ed7fb2", "output_pcm_sha256": "5bb289dca4c260820a65c8c2f0e94a0e170a284da654ffe78dce4808e2f25e4a", "result_sha256": "d339843de8755d5f1ed02b600ce8de87e751cc4ed42df8de764dee8997170a68" }
#321navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "4336030de455d68c0d084c360d30126d7114bcc2457e77439e5691411c564ef3", "started_sha256": "122135d6b50f5ed547159adbfe57854efd1a634a135d1939392fe867ee3913e2", "original_grade_sha256": "2d3382362e7febd360ab7932afaa1137f71a4f653fa1bb170806f537548719f3", "input_original_sha256": "4336030de455d68c0d084c360d30126d7114bcc2457e77439e5691411c564ef3", "input_wav_sha256": "977c5ef37c5bab8e1dd311b7c796c2f54bb0cc7c3d36bd37fb68c904560b68d1", "input_pcm_sha256": "d43d0a0a546c370c8c18e4366eeb12df34ae1e3b3a9b129ed5601538a875d0ef", "output_wav_sha256": "e43215dfc1e0c214621e741ccd07dd0f8314a185d4441d42b1fca50325b3501b", "output_pcm_sha256": "4605e1006b78e121cb7b4b5026b25adcbfb6eae0dca864b2e08dadddf4f76c05", "result_sha256": "a9154d18b9faf9239c53dcc0f5fc70580e0b9568d8312dd4cb18d33a4421b0b1" }
#590object_counting heldout	5	The answer is 5.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "70f672e5dabec049719a601f53137e7d018c0fa44b04e6d57d6429735f5b381f", "started_sha256": "f90fc9caa9e26f6181c43d8979d0f7a732033f63d053a802eab33bbd4a3bb188", "original_grade_sha256": "4f48fab60eecf3a17d13830fecaca7f8288828634ed0dfc45ab7f828e9d71ed0", "input_original_sha256": "70f672e5dabec049719a601f53137e7d018c0fa44b04e6d57d6429735f5b381f", "input_wav_sha256": "9ef069d56fbc249118a0c0b70cf7b366904ee8ec06533ae0f50552a87ecdf365", "input_pcm_sha256": "592b18d45e77be22a34d3afdac5f7dec7972662cdc73d461a906664fb50cabb0", "output_wav_sha256": "d59a3ab6228e19b0105bac403c86044a17c445168d2b96b0df703b1ea34c94b0", "output_pcm_sha256": "7209648c09e947446890d72dfde7626389e90262c69cdf3862a734f9661fd0ac", "result_sha256": "e40e99c2255bf68ef2c5135dbda74e2f3c0c314511fae79102633bd2f817cb1d" }
#979web_of_lies heldout	No	That answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "1f890286d97a047aecc9ea5886711dd3bc337171963d5c0b11dd1bf2a4d1f765", "started_sha256": "e253b78436c6ae915a8700f1579f30ac2935a2cc17fc99d909c3c665d0e26523", "original_grade_sha256": "60266e7876a87f012b52b7f0f05068c2116da1208cfb11d238fcda4be5a03f3a", "input_original_sha256": "1f890286d97a047aecc9ea5886711dd3bc337171963d5c0b11dd1bf2a4d1f765", "input_wav_sha256": "cfe82fe1c56d41a1a99c13b531abb6895e9b385131da8739c7d46b1d1cc7bdff", "input_pcm_sha256": "e32f567eaa41af494e8509e0280db22ce6359e23765d4d275ba6955f74e77728", "output_wav_sha256": "94650b39296f69111cf609f20049a3efcc43c29a95a1a49f05908d96adbbfd1f", "output_pcm_sha256": "d56dcb6e1c6700211ec4208e73e6e5b75956d1a90ac0e0e615936085029132bf", "result_sha256": "9e3bcc3afb86114504628226857d1941c929c5bf67dec87a814b892f3c720cf8" }
#135formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "d2f75c5290f70668884299fa0361a01e54e55c0425102fd39544fd14b7ef6a43", "started_sha256": "c9b7a26d08b916996b10f0e5f17c7c9a450ee6c63f5b0dc1e7623b03ba4fc967", "original_grade_sha256": "95807f71eb7fb3cdf06211e31323ccb0e366a3541b86803d58756c628c241109", "input_original_sha256": "d2f75c5290f70668884299fa0361a01e54e55c0425102fd39544fd14b7ef6a43", "input_wav_sha256": "ae578206dc3e86b69821ab7cb7fd3d092d778e1e4a53857883ed9e247ba84c73", "input_pcm_sha256": "768ddab549c7a959373d00f27e1bc4ec957f06e51d0b080ee81b69a7f02c5b8a", "output_wav_sha256": "19c0c1e69594b9058a0313dcc571d48622a150db0068a84bd332c9e8ec278820", "output_pcm_sha256": "f6f0a622c1e46259cbf87ac84e56a2464f4b43e35354e199f35bdce43d930372", "result_sha256": "71bf111f8f1be73c34e41a6294f7b41277d602e0ef442697104d245199af04f6" }
#352navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "db5f6ce3613affff22dce9293ccf74b1e771560ce80a2e808db92ae305bfa945", "started_sha256": "71ae8921e61d4c527f3f00db16808d2bd51b3ff5588f77da102e9552d4fd614a", "original_grade_sha256": "361cf1dbe7e4f60043e5c54c001a95eb2a6da9f02fe7e51919265ff0d324d339", "input_original_sha256": "db5f6ce3613affff22dce9293ccf74b1e771560ce80a2e808db92ae305bfa945", "input_wav_sha256": "b263e5326961f640457025177b50f359577e4340f706ae15ba109f885fc618a8", "input_pcm_sha256": "5af6a84f5f37d0fa4a95788fccc60717963ed1c687dac493857db6fc8614f2cf", "output_wav_sha256": "b266b4bc96003a2b0b47fd450a6feeaa30f6b388fa0e8d82cad908fdc9820796", "output_pcm_sha256": "70ff05e0c5396d21a95d73b1b00b16a743feb7b42f5048c5d60026d196397839", "result_sha256": "e9f459296921d05c682095f29a999e56bbb734b6eab4d735bdc45ea6869df8d0" }
#734object_counting heldout	4	The answer is 4.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "9daa0e45d13a81062f1d5d75649d7fe43defa195efd9ec55a8f3a0177953eb96", "started_sha256": "28346ce6500536114dbd5f944b1e263301f2e223175d99f770ced5c0eb253a99", "original_grade_sha256": "b44ee9813ee029eeb130686e50884f8edba8b2e7c403df45ed8a34a935a28c36", "input_original_sha256": "9daa0e45d13a81062f1d5d75649d7fe43defa195efd9ec55a8f3a0177953eb96", "input_wav_sha256": "0a03592ebbdf9fbf808d4172f28b441c9146a726591ba845e34e1759588d08b8", "input_pcm_sha256": "4da7e967bc9d1cbf594cfdbc94da9b77ba2af48fc96cd470022d7814c7cef406", "output_wav_sha256": "2645259e6092fe6f79080af78f292658fef4123d3f3ee5e6c0bdc407ffe625e5", "output_pcm_sha256": "c2b4a033aeb2eb519450f9453015bdd8479c1b646ea8d9294b852d1d7b8349e9", "result_sha256": "ceebe10f1a072fd066c2fe3a9484fbdb1f66e63db2228eb78775ab6826e7df27" }
#788web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "bdf98ab7a945e0911d096748c8affdc6778c3de3f5968c7c72b9cc0dbecbcaee", "started_sha256": "78a523a0dc1df4d8f37d0b6e666fc532946f9d64a012386aae2e52429d80af53", "original_grade_sha256": "a4362abe5c2d8a6223b50a093136163ffdda4c00a8030246a93d7d58ccfe23ab", "input_original_sha256": "bdf98ab7a945e0911d096748c8affdc6778c3de3f5968c7c72b9cc0dbecbcaee", "input_wav_sha256": "368220971ce54d951784877384d105b9e458c15186d344924003c1753fdf2512", "input_pcm_sha256": "4b6edf210a21dd87623f184b3d54073d6b930664dc16be9b1729d89f15135852", "output_wav_sha256": "c2d045afffd752752075cdcf98c56753073b40b33030bdcfb14698373f933c7e", "output_pcm_sha256": "66baf5b450837f4b3af9be5faca275c4195764fbb12bf8e2a1f72e92181a097b", "result_sha256": "03b78f4fa2777408ac9233e4dea5197da0c76317b623c76f1b70f863981bea10" }
#88formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "dc310ddf8d19249a5b2c54e27bbd33241981be21719411674912d35126c142b3", "started_sha256": "8a9a0cc5657282ac026cc58777b69c8d6e7f19633742bea87c3ed0a9108a2ebd", "original_grade_sha256": "c59dd8c9706ce9762fe66dbf21e5d4eec67af647a68721f0e981a0b298e33298", "input_original_sha256": "dc310ddf8d19249a5b2c54e27bbd33241981be21719411674912d35126c142b3", "input_wav_sha256": "82a5730a5027539ed266215b13867fc4eb79e4420e02a022cf9fc1450d4a04b3", "input_pcm_sha256": "d261ef0a9b945b1aa69fa794d2011f50579884b9361092947a4b23b2a5ee26a7", "output_wav_sha256": "ac350aaca75b495cbdd3ce00068371468042129ad1b14cfa8d36d2d37bc3ddbd", "output_pcm_sha256": "ac892dd38ded7bd89350c1fae97cc4bdc4799262fb8de2293b5cf1ecf0630d54", "result_sha256": "6c9108ae15f11a71b794d87024720317999b1391d2f86efba1bf43e6464a8a85" }
#275navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "a851422c9ac924485d1ea52ba2f4fd62793b170944a379e76807a8eb8fff24e3", "started_sha256": "248fb4cb45b21a99376036b2ec4181a8a398c7ad0abacabb63f12509adf191a3", "original_grade_sha256": "10a4d5e1afb287888c1552813bf21afe49285000e78a45c383a449ead81626a7", "input_original_sha256": "a851422c9ac924485d1ea52ba2f4fd62793b170944a379e76807a8eb8fff24e3", "input_wav_sha256": "56c2d8a88c17309086ef1306fac4357b6356ee9ce9f705ee4998df7b1a4c4ddc", "input_pcm_sha256": "dc8b76556f7707622010d3186777b5b9d21d0512da26114ed114cfe7f93f717a", "output_wav_sha256": "ad8e13257410eaffe67c5aca9fccc753c695608ce16f9489f3b5cb7ab761b2b7", "output_pcm_sha256": "145fc6f662d61c6c331857100f4d662cf2cdeebe5b5f29f7d0cc2a4a0ca5e199", "result_sha256": "72920a9cb896eeb39ae01ad2ff4feda663fd2f6a3c8513590ab894a26f0d16d0" }
#617object_counting heldout	7	The answer is 7.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "8d617bf7f3b686ac41c0111cf36e988f0d1ac66299fdc4c4408be54e38864d8a", "started_sha256": "51473d1aac0e40559be9011e1c43e21a00f7cd382b296146512fd4980cd91f38", "original_grade_sha256": "2c55099185919082d3747904a87b423a15b2ad3698a42a1f7d60cff2005c08bb", "input_original_sha256": "8d617bf7f3b686ac41c0111cf36e988f0d1ac66299fdc4c4408be54e38864d8a", "input_wav_sha256": "4380345c3606d2bfc392146a671d4ab4fa49a944293822cdad82b9c67469062e", "input_pcm_sha256": "6a74feea5c69c278b14a6695d3622440289f847924126b76aebc700dadd4d8bd", "output_wav_sha256": "b20bc15fd4bd25c714f8d45a312290ff12c191ddb3df01a9d867ad5a78497fdb", "output_pcm_sha256": "93310d9c89fc8fd07058392c61eec01dbf36c811c4b429b701ba7b7b3b2427a1", "result_sha256": "4e7b1d04a2262f6109ca8cd2f50c897d35daaae3015fc7eee9b56fddf2144a17" }
#910web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "0ca65dc6adcd7f0bacf55ae16df92f9a807dc32a3c66f5ebc4a81f6cb1668328", "started_sha256": "ae8fbeb55f23439d4eda5ce7303dddad152bf5981f4674d6f72cfc1602fbbe8b", "original_grade_sha256": "c4c494d70b7275f66d8fe3a4740ffea070140f66e65ccbdbda754a29ef549163", "input_original_sha256": "0ca65dc6adcd7f0bacf55ae16df92f9a807dc32a3c66f5ebc4a81f6cb1668328", "input_wav_sha256": "c14f04965348bc3a2d2a4f4cc1854c7316aaa783181d74d663866b4f9c4a7eaf", "input_pcm_sha256": "a27f5c557cefecde8ad5a9214ce7be72936152b4be6b2aca5a5b496953891a3d", "output_wav_sha256": "c25ac9847b8e0f0be50f00e02f5cb3c304aec30fcccc1ee45ceb7c183c808b81", "output_pcm_sha256": "186d97f8cc7b76eae5445d500808aead9dcd55d118e04de665296a5313222614", "result_sha256": "fff472f3a63835e3c67a237d1766f7b843066713317ea914454b13ca8c728ae3" }
#83formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "cc13bec2ce6b11d74ac5878b61648dec0f9b3dcc9cae04093accde46bf78bd02", "started_sha256": "0e32efc3ed9864a5008aa6dea876c13704ff40654d67c53c0571a549e568041f", "original_grade_sha256": "ec30d4ca13eab4d76e5601c2b35f4ec472a038052ae12e068b4f91febbb6c6c3", "input_original_sha256": "cc13bec2ce6b11d74ac5878b61648dec0f9b3dcc9cae04093accde46bf78bd02", "input_wav_sha256": "6708e2445c792f96e21abb964a3d30398e79675538f6a3071c338ea07ae05cc5", "input_pcm_sha256": "ade40104f62adf724a75d5467b903a05908796add68a78cf3c74718b70254d33", "output_wav_sha256": "3c2783c034cbf9d13c931fa6b0b3e7d1c08a24b1880ad46ded3975f44c2aaff0", "output_pcm_sha256": "b81311fb441050c3beb7692a366923f666fec4e5d7daa56ddfd0c3f65758d5cf", "result_sha256": "f1dadc5ae9a018ac99f4378d020a9a04320324acf8e04dd630d2989cad131fed" }
#429navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "62763996fc58b5ac9168a0f6b711aeec680c6e9608b342de2cd4005b0b9b51c3", "started_sha256": "3fe59a9a316fcb075007765aa8987e75486af3f7e34307120348a268397b560d", "original_grade_sha256": "679c98ba12bd234d6024aa536c0a1061e8f33d56773f021c1d4fda06da5b2bc2", "input_original_sha256": "62763996fc58b5ac9168a0f6b711aeec680c6e9608b342de2cd4005b0b9b51c3", "input_wav_sha256": "83f67a2484f95b2494df50131830b56a6c29d487bdd87da14f4bff96c9cb1dd5", "input_pcm_sha256": "db5509bba0cbdfc1ae139ef3d3156ba5dea2d864eec680cbd4db6494dc928584", "output_wav_sha256": "905233c3ac3cbe7050215fad695e85eab59ce6a500a917592a2baca5a2d17325", "output_pcm_sha256": "9250e2d1a85c6f23aa0cb2f8dafc0aab4677fc0c643dce7a629882123928fc31", "result_sha256": "ae047adfe438d64ea0dc7b98169c9cbbacf1507f165ca434cab6a7ef06954a91" }
#582object_counting heldout	14	The answer is 14.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "588a9ef6067259c0f9f16e0375fa4bffcd013925daff34caaa8c244964aa8d9d", "started_sha256": "4c6c6b5e368ef6a7dbe2b5a5910a1d8dd46d0a7830434f8e4a850f8719a9a2db", "original_grade_sha256": "873cfa04d02df01c2997f11b962f66d3aaa11fe10a227522281379c35dcc2906", "input_original_sha256": "588a9ef6067259c0f9f16e0375fa4bffcd013925daff34caaa8c244964aa8d9d", "input_wav_sha256": "99df74c7c4b15837de15f7a4b0a9ebec9743d64472b77a65585175880dfceb13", "input_pcm_sha256": "7a887d01d51ef5794854427f7fc1e9bec923aeacc4b6de0dfc5c0af6879a5279", "output_wav_sha256": "ddc7d86e43d58cf9431980c601a4b2aef3d61abad43942120690b2abe296f4ab", "output_pcm_sha256": "a39543eef95b17ff3a513cf1c385211e00a23243abdedc3e72d3c2f642edb2eb", "result_sha256": "04e125da4aae9b9189c81c9f7e7890597175a5fc1667f17ce9e9549e6409b031" }
#778web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b6a900a7e56bb54085f8a5291789f9cfca69bf371dd4817e9ab08da7366d55f2", "started_sha256": "10c97872c67e8aa5fafd33bdf72b4097a6afbd2faecd0c0269a396612f62aa67", "original_grade_sha256": "bdc4221737c558be2900d734f0e02b2c722c21ba0f3704cdaa5b7ae640bbe5bc", "input_original_sha256": "b6a900a7e56bb54085f8a5291789f9cfca69bf371dd4817e9ab08da7366d55f2", "input_wav_sha256": "7d363608731d9ce52cb98c90002db07c6c55d904de9d8cb255cba406a20dea06", "input_pcm_sha256": "8efd4debc62407e7f2cf8ba52bac045c4444fdcf1f167b4a8335f02c4eaf3375", "output_wav_sha256": "0e2f7132e5ed0d046bbfe88250edb1f4cd90ad1d229bb52e9cd68e1cabe2a45f", "output_pcm_sha256": "60510d45ca54a5a6ef8ead5a114985b31d84e70261e4d114bd8daf7bf749625f", "result_sha256": "dc79966e1f73ac576c1e7182c78c17e16c83572e57b0b8061210f72662c9b93e" }
#121formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "14fcba566c44bd7ee8d9e4796aa5956c57283abfc6741abe41645ef84ee4ac89", "started_sha256": "a4c1826d167a355da99ba334c1604e40713d85aac58a0fd299362f575322110e", "original_grade_sha256": "7af2d4afc2971a2978f7dae07b76db53515f358e2dff8a50adadf107a886627c", "input_original_sha256": "14fcba566c44bd7ee8d9e4796aa5956c57283abfc6741abe41645ef84ee4ac89", "input_wav_sha256": "4a63f555b91c7203950d2a9f99e60fd79303ed6be74e861dfb8831a1a02d7622", "input_pcm_sha256": "17e8a0871871416f09670cbae1f6c8e4b1d5695478241d2f961b5ec1b00c49b0", "output_wav_sha256": "5dd56dedaf2ad73aa23c2505d951eecc056b1df96cfdd78b37c9e78dfb39631c", "output_pcm_sha256": "a8e0932321db4f75267d2eadb37a833f8f875ff3a37e06c91f3b881d17eb8e44", "result_sha256": "c30db51d87be58e5b7e28edcd8e1d1efc8101fcae2fbb84d4ad28bd427fd71c8" }
#489navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "2db7bac3cffefd255d933fcc980746c824735688129e2e70a7bfb1caf60f826c", "started_sha256": "4faae80e61082457170a085f2b6357b58405b04b6d0473414fa0b1cfda316a91", "original_grade_sha256": "1bb0decbed5a5a69ffc04abe4d939b873fcfa1223601329dd5c87c3a145d86c8", "input_original_sha256": "2db7bac3cffefd255d933fcc980746c824735688129e2e70a7bfb1caf60f826c", "input_wav_sha256": "404b1c0d26efb523866a26bb12e61f50e4d995665f2ddf444caa172e9be25ff7", "input_pcm_sha256": "d87b8e2f673d83a91af9d131f3902a4ef55aa21f32afb2914f1a7a3f439bdd16", "output_wav_sha256": "b8a327f2c31cda221b6b3610653ddaffe9fd9dd75e593d2b1de9b4474b7f357f", "output_pcm_sha256": "b286ddc9aad2168655756e8fb27438e33dfcd1f3089f7e436783802bb540b658", "result_sha256": "299fb7e54eda4491376165c4d8571e923a1946bd3184c988f8999f297b011a3a" }
#738object_counting heldout	9	The answer is 9.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "14c4c6f939cda389a2b34818d4c085d02ca473fe7ef12996403ec83a8fa5e607", "started_sha256": "4bdf853203526737c033659f404ef1a79eac58926eb1d9a3560f2c7173d920f5", "original_grade_sha256": "8189e94697e3006725ad7f6ef7ffbb1bd877b55db2e2fe0d89e43dc1bba5a241", "input_original_sha256": "14c4c6f939cda389a2b34818d4c085d02ca473fe7ef12996403ec83a8fa5e607", "input_wav_sha256": "bbea5890540b18dffb6f82a41810f7bab9faa65875d6c16d20e9656cce299920", "input_pcm_sha256": "efb48a0f7352a1ecd735a888966790aa90bf4e76b7ce7fc0800b826b70d739b8", "output_wav_sha256": "59ea4a31553b87e6c08c0b3fed58cc6dacb16aacfae03684f58c8f598546a1aa", "output_pcm_sha256": "e838f42605bdd573aeb2bfb9637cfa9b82ee9a26fa9e13dec1f782aabf6ebb16", "result_sha256": "734e81c13fddcc881c4ba368a6a51837649e92ad0b4044be8962ec30ddfac487" }
#806web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "0132722d4a6953601232d19d2911082bb06c36e30cce4f68a73432f0fb442fe5", "started_sha256": "bfa48c79c01d2512bb2a98253c7bc4e8f7ba959dd61800e145c7dff9e3c0681c", "original_grade_sha256": "3dd686ea329a09583ca0b8cbe0e71fc63b151a208837a704d64c6a3dcdcf23b1", "input_original_sha256": "0132722d4a6953601232d19d2911082bb06c36e30cce4f68a73432f0fb442fe5", "input_wav_sha256": "48a4da099a16a1fedec6dc06cd7545797f0bcc09a81d3a2acb2ddabd3ab8c122", "input_pcm_sha256": "483244c12bd9ee6bd2b21cc3b7406812fc91afcca9df337c66f0aa7b71152bca", "output_wav_sha256": "b24fb2cf89d0eab42b3b9ea3d70a9a8b1146ff4f010208827f4bd830d6250a46", "output_pcm_sha256": "3bd113679bb98b278cdff6eed566324b673815d6e962eb4d4c9613babc7551bc", "result_sha256": "32c4a91cdf1fad96ce825b22abd09ee551986278779837b97b013ca0b1efc304" }
#111formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b0e97aab6757267808b516294fbe35f2350be54958442f4f75db032cc82504b8", "started_sha256": "c3484ee768a13fcbe93533f9e993c48e464fca08abec3c3063ece7e49a628999", "original_grade_sha256": "41c8fe5cb8cb3a8ff6455fc678122a88c430ad38de6f449563bd0433d1a88baf", "input_original_sha256": "b0e97aab6757267808b516294fbe35f2350be54958442f4f75db032cc82504b8", "input_wav_sha256": "e4dd564b69aea6a6442b1f3e4fe94485cd86c454e61c6278eb9c5bca3547eefe", "input_pcm_sha256": "5a2599890e957064d4eb8b1b5ea14e2b96f74ebb5a68ac984c75cb58d7ccb5f1", "output_wav_sha256": "a609c9a4769aa99c043665db902d33349b7b8855c0aa43ba8d8c1e273c06569a", "output_pcm_sha256": "34cc18e2fdca184ca00d56314011fdebab114980056922870483ce45f5d0a4cb", "result_sha256": "82f2b8ccf2c796e63e0d21617057642a41c15897c111edb1e63ad5cde0e581bc" }
#308navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "62d2ebd5ec8219d82b6a3aac3d5c2618f0ed1af7eef77cb2c8c1293b60a5a6eb", "started_sha256": "96e834ec0e0e6176a550033cfcf89c97a5116bcce2f01c407e0f1185b7e1c7ce", "original_grade_sha256": "e3eb7584b92a2cf93a343b59173e1dff4a10b1f26b9aaaadf896b096fb613eff", "input_original_sha256": "62d2ebd5ec8219d82b6a3aac3d5c2618f0ed1af7eef77cb2c8c1293b60a5a6eb", "input_wav_sha256": "702fa57a9747e1608839e3122107b620575121838530dec65b4a73103117a83d", "input_pcm_sha256": "1b56f6587db3d35d7a720b1a7407e217c1e98e0f92af5bc0523c1d917e61aa14", "output_wav_sha256": "84bad8eed014bc19327f0029110004c53efac11af3eca8b46473afd479651c87", "output_pcm_sha256": "05d6701e7a2c4c132639cc32e9e6784a1445b518da546ede8565f9b16d2544cb", "result_sha256": "416f7478e3e7bf9a2833706fe82a1605812d1b2bcba94e07072928037fff563e" }
#611object_counting heldout	3	The answer is 3.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b61bb1af08344bdab290c3764f49d116c4dfee71170ebc5a8e976c7e3ff87fb0", "started_sha256": "c96b2903d8a8994447ef038567c30b387ec9361a42a2763a709ddf1916615969", "original_grade_sha256": "af46df477517721790b384f633cd5439c56e4596f70ad8f661bb52a4923784f1", "input_original_sha256": "b61bb1af08344bdab290c3764f49d116c4dfee71170ebc5a8e976c7e3ff87fb0", "input_wav_sha256": "18569536c3ea9d51cb28478ee3fe712d1b41f65296adb92aa73908c8c80a7971", "input_pcm_sha256": "05300a6d807d89f7c7c0afbfa38d70e54b30ec0af2f732a6b4052d44a0c40e3c", "output_wav_sha256": "14c97cadef22408ecb1f34b673994ce1bc05bda390cb840139a8e9af66b39ff3", "output_pcm_sha256": "a72a8e87eb349d1b61388a55169614a76c05da8d8621d352aa1a9e152de7d25f", "result_sha256": "fd947dd70522e31d82a7d9f720bf6bf5f46f113a1fb836c03d95b5df8a1dc06d" }
#995web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b6a43507e82251d1822405da7f482d7a8aa01e1c25a43e7076fcfec7f83f94c6", "started_sha256": "502764420a6eb95ccfd613a358df65bfe215e8c6ef5bd2358b59aebd2292913f", "original_grade_sha256": "719b889f5ee7aaa8ed1ccfef3feb79ad56126e99477d753c781929838cc943e2", "input_original_sha256": "b6a43507e82251d1822405da7f482d7a8aa01e1c25a43e7076fcfec7f83f94c6", "input_wav_sha256": "53678f7a114a3dfc85e2cf965d9d13eb9cc180b2ad29196b62b6d329e2eb35fd", "input_pcm_sha256": "94a70fd0fc99ecc3c6e8a5c24ed3802f3e19e9aa77fc1a2d26ffc6b84a1d2eb0", "output_wav_sha256": "b7aa2c704d2e3c3f9eb335f32a931c8f26f67f18724603956ea8d722aae9b0d4", "output_pcm_sha256": "378cf91c3c3ffc21c34b4bc114703c01ab4db39903d511550aba2ea81f8bf250", "result_sha256": "c92faa8c6c6bcecad8ce93ea320d930d0e1aee1a2f3b962be4a63e9d0ffd8b8b" }
#133formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f6621e560a77f8e522d16804b6d4fbf2873e6d9c3e49d73c423fbb0bb7982d09", "started_sha256": "75a0455a403621b517e487c0b21caaf44d33f1f337eab32008c8981ed341a270", "original_grade_sha256": "4d2e6be766dacc1f87aaa51972f4626b66b0d29f5025a4e010cbd1accb01fe83", "input_original_sha256": "f6621e560a77f8e522d16804b6d4fbf2873e6d9c3e49d73c423fbb0bb7982d09", "input_wav_sha256": "15158c5c170dec86fa12b90c7db827d7deccd6d8b580eeed0cb4b4ffab06457e", "input_pcm_sha256": "854b6828288d67ff8b4970954a539729af626f9640ef581cefbdf597377da370", "output_wav_sha256": "93c8a3638a8485bee48a7aa033b105877f85d719c36ac9dfcd689ded4ed6f902", "output_pcm_sha256": "4dc8220408bc3ed290cddb66e79d1fb138898bd559b7722e389dea9677d54564", "result_sha256": "6f17b8b102178cf95843c1fb2e123b9bc1db4ee552a6cba07ff45eae91ef6c55" }
#327navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "5305fff9fd5a42d007d6befecaf0b487383f77135d23fd6586b1d05261fb70c6", "started_sha256": "34f7ef677279b76dcd95e107e3c68ba74efcbca0148994550b39eca6eb1d836b", "original_grade_sha256": "618a7efc3b79903b22fde7c2ff1a89974b17413ffaeeb0fee1c6312ef5a3942c", "input_original_sha256": "5305fff9fd5a42d007d6befecaf0b487383f77135d23fd6586b1d05261fb70c6", "input_wav_sha256": "c08bee0607d21111340402e3e3e3a0548e0affcbfda4d6843c8e497df9d9557d", "input_pcm_sha256": "f1f7040f1ce1abc85295745640624f4a04fad4b9dcb41ecbc00beaf3a20bac66", "output_wav_sha256": "3a6cb23b0580592aefb8bc02ddff35fa65fc872046b37f96a036edcf913d42e4", "output_pcm_sha256": "18a7171ce0ff2e4d44a268f39129033c546595383978a3e21e6cf985cec5901e", "result_sha256": "6ee85793c5e8572e7624279207047881e588f133dd4b1f4a3b9fec6d63417e36" }
#690object_counting heldout	5	The answer is 5.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b88e096616c5358f2085a501c3971c29ee03231df29abc3aa79ed4b7239dba5b", "started_sha256": "791e88316ff37e5ece183d7d36de8b57473bb5a744555b5580854cfd8694af4a", "original_grade_sha256": "f30f52ad0c38b36b12418a9f5dde189ac8ef4d12b339dc7b5472db48643214dc", "input_original_sha256": "b88e096616c5358f2085a501c3971c29ee03231df29abc3aa79ed4b7239dba5b", "input_wav_sha256": "ad5aa7dc12faec26260d57b156ee2b9c192138cf5ca05cae644af3da584c52cf", "input_pcm_sha256": "40fdc0930012da4feba3aa7a71e3a928298231153a69582be6fac606d8b9e59a", "output_wav_sha256": "b528fa538a7a0990fccf775815f5edbfcd930032891066b6fa71ac7e35c73a41", "output_pcm_sha256": "ba1bd25c8e2ce46d5da30904cb61c3a8c4281348b4ad86c0c8b3ab2dddb8db59", "result_sha256": "a06c580563182cf7a78cb0097d7a477200f433a3bc577770b3caf187c5d4d38b" }
#928web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "bd4765bab83839e44791b158b5ec24865de005ab66be2fc04e86643308160689", "started_sha256": "ca21642e28293051c4913b75a92bb4b60404edd051497e3fb8f14f097b054504", "original_grade_sha256": "79c0c299a1a5ceb9a8bbde13141078aa771bf03519c4aa844c41921720f45603", "input_original_sha256": "bd4765bab83839e44791b158b5ec24865de005ab66be2fc04e86643308160689", "input_wav_sha256": "f79631b74001861f8523ec6b36abbed6784a5f8e1edeb7c84a8a26392acfd9a9", "input_pcm_sha256": "8ade738b8bd3394cfc1f1629cb4c0aea527aa0a59c7a8b334be15e8b92912636", "output_wav_sha256": "a3c1c015c779107f3c93ba95a1217cb6ab1e5a6246d2ab919928ff8a9ddd4d28", "output_pcm_sha256": "0cf04359096a6cf8a42222f30b3d617c857b93fdfda618dde0a9a99fcef3c1fd", "result_sha256": "5af0def5abf1c3a652a7bbfc60e23e691a3dc99a96b83890ec5aa610b5d83555" }
#233formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "ffd29eb8c5d63a067eb6847606a2a91c18740efc7e07c167ce506cbfb3799a37", "started_sha256": "b84d4d6614cba167e0773222977c8973038bcdd63d268346676bd60a557805e9", "original_grade_sha256": "8ffbd58fe04bbd589c97a1fdde4096a57c264f3bcca619d4dce35180826e312f", "input_original_sha256": "ffd29eb8c5d63a067eb6847606a2a91c18740efc7e07c167ce506cbfb3799a37", "input_wav_sha256": "0988ff61b91ceff12f32779502698bec6f7486ee0053e3389110a3ccbad8931a", "input_pcm_sha256": "b4ce2a29f5f2b17b304c9c5935dde136046db4782d1fa5687f4bccff9745cf3b", "output_wav_sha256": "5da2e75862ac3f7c299e8ef618e5d8a8c2f3103c9701f80e73e92ca97b8bccbc", "output_pcm_sha256": "4b73a758adcdfb6697010a1bf4422f1b285ffcf5735a2d3c2775bb51e1fc2e73", "result_sha256": "747cce793ada23f572875b21261c273c1aa6b253ffad5addc56ee8b7cac688ca" }
#410navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "1a01d1660e533be7a16dc5e5e33190b9a6bf995c2c46e688dce31c4e373a4e65", "started_sha256": "5aba0b7832a6e3998414f666d3fd3915cfd74bc1ef6f8180d88ea363df20270b", "original_grade_sha256": "777e1e4b4680dbd0bb8f62bf885886cd374c8f1147081a0d5af3ebfec7688b11", "input_original_sha256": "1a01d1660e533be7a16dc5e5e33190b9a6bf995c2c46e688dce31c4e373a4e65", "input_wav_sha256": "1e7834306c68fa0b8a703314a458bd43d985f2c25a90fa7f4cdc7100d1baa65c", "input_pcm_sha256": "dee13ddfcf84139bef04cfbd90332b60327dea03c99d4cd45c5f8afffe09197d", "output_wav_sha256": "019ef1ef023a8ad733070a760cea12168e0437b95385ace3f42e9fc371585a12", "output_pcm_sha256": "dee52fa808007131cecefd682bd4f88a5fe23daf47bbd87d39ff3fa7a42481cf", "result_sha256": "f7dcde972efa822dd0cedd553c6d736f3eae0a96d8ebf2a8290f21290373cc1c" }
#729object_counting heldout	3	The answer is 3.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "26b6e829c83aaaae98a4c1321350655a8f6f065dc499eda30eea47f7e3088473", "started_sha256": "9c5ab9b706d74dfa99ea9fff87ce272d06a100deae77e51edcf6c7d92a48a98b", "original_grade_sha256": "007ad48641e25b91d5884490e6bac651fcf10dcd25ace5537117a20e11d18c34", "input_original_sha256": "26b6e829c83aaaae98a4c1321350655a8f6f065dc499eda30eea47f7e3088473", "input_wav_sha256": "ee7ee6e060e21d77de5ed6a5956236c3ebff03a9c3dabf523fdbfff1539e90c5", "input_pcm_sha256": "dbee19f4184328f40fdade1a03b2c9689d7bc13dcd2b0ca686388a0e5cfa6297", "output_wav_sha256": "59988fe834a159cefd77a9149661691b624f9025206bc690797467b1ed0e687c", "output_pcm_sha256": "b9a936c027b871e41154615641fb6f9ba016a30254d61fe66c3679f2e8ab6166", "result_sha256": "ed9645c2114f1734b164fb2f97e8e633527f9b6fe008230cc2b8b9a77892f05f" }
#914web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "5be835258e24f250274ca6b828eec15a227b5a1eb85be1be9bb7f5b33d5c777f", "started_sha256": "6f33bc1abf7d942f29c750ec9c709bc7958488895c7f2f987275d90a59510a1b", "original_grade_sha256": "69bc61d1921688fea1e5b3e5c0c61ee2e2c0be7e7038c23abf052db0d6161fc8", "input_original_sha256": "5be835258e24f250274ca6b828eec15a227b5a1eb85be1be9bb7f5b33d5c777f", "input_wav_sha256": "71906706615d7060c22fc98e057b8579f7f8186ec358e7596b367becb016b2b7", "input_pcm_sha256": "ecf1a4594b3c7024fdea743080b3dbe6e68ba26236e58725fab6b65bca6c180a", "output_wav_sha256": "1b582cd4474109c32e8fd4d6aacea3c86d8285ce7c6df926da1630e4b6b970d7", "output_pcm_sha256": "97f5c8775cad8497c7b346a67dbe753784cf131f59d2b20a928d152f42815532", "result_sha256": "94a534e64fe68e9ea1f73db03dee5643e9104777f3d13dac8ebc623b581762ed" }
#238formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f754478420214fbc552755c9b6c201d2b40f29dd498837eb4126a82646c8f140", "started_sha256": "006e5157230a74d8fd4dcb9cab20aac9f046b92751215e80ffb19e54801100fe", "original_grade_sha256": "4aa4e48be47df4cc53622b26d5af7f31eef96bdaaa3515b25890acb30f6b5088", "input_original_sha256": "f754478420214fbc552755c9b6c201d2b40f29dd498837eb4126a82646c8f140", "input_wav_sha256": "835c99f90e71877a05fb155808365d6dac805ac4cf1cc469f920464d26f75fd3", "input_pcm_sha256": "89c0358ba0242aabd7ebeddab399df5c44b1dd18a853798d73c5d9b6d53418fb", "output_wav_sha256": "43ed13106c934d5e08fc8e47ede11be7f96ec15cd138487bd0ac09381ba0e3b2", "output_pcm_sha256": "dccd3235f4985ee3dc80b53afb4a3773cb6e83b06d289f5091d3a0305278d643", "result_sha256": "6218ab21981ee3e270a45ed32ab07008d259d8fd930897bda12025681f2845df" }
#377navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "0249fba1440047f2d5eada7678b5793a5c7ca12440ad55415fe6a8acad002bcf", "started_sha256": "bc3d1a4dab021db30309f52dd4c9cecf85dd0625d3b7bc60e4ec6bd282214fd3", "original_grade_sha256": "b495ca80bab988acd486d8d97630718448540c4fa3aa8ad00b5bcff13cff6d35", "input_original_sha256": "0249fba1440047f2d5eada7678b5793a5c7ca12440ad55415fe6a8acad002bcf", "input_wav_sha256": "d7e83aab7749f61a758dbd04729761a52c899fa9c8c920fae905aaa5630d9a27", "input_pcm_sha256": "d00e91aa68db106caedb0cb36269c3e5bfd1001f738a80bd4e8ea56d101ead83", "output_wav_sha256": "880d35be88197e3b1d7518a9ab1365a4a5a87268b8d8aea181c0fe4067244d4c", "output_pcm_sha256": "3ece431907f4618ff5c25fbe10981ae88f8daceb7142c2bba894033d238f2abd", "result_sha256": "12951879d2fef32eb46e32903dd7270f77575ac6fcda90cc31fd5575731a83b1" }
#571object_counting heldout	3	The answer is 3.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "9c15f18fe459aac4e150c66c2860470fb43d860e2a766e11df82003876022c27", "started_sha256": "9718598eced3687f4579788d275a8c9c4a6af0ccc0293150bf9be32b3b6c3f75", "original_grade_sha256": "f808d0765ef0538cddd473d4f212b11d66da315ee414d96549b1f7e6828d668f", "input_original_sha256": "9c15f18fe459aac4e150c66c2860470fb43d860e2a766e11df82003876022c27", "input_wav_sha256": "a68e3fb6d85cc4828d62bd04e7ba1797ab013172e9ec66dcafc62c36548e2868", "input_pcm_sha256": "7cac9e10a5801de63f4d214300755670632f16590f93fbda76c90209c5a4a460", "output_wav_sha256": "8dad6ac2f76e691d655a9db3f9c0ea9b2ccf611bb580c2b3e4f3fc2daf76d515", "output_pcm_sha256": "ba461854ce015654120dd45e59b19c35f164d3d8b7a3ac47b96f60dc6d2eb550", "result_sha256": "479834bab224bf13458c7d2ae3557abfa26e81e311e37549ae52e01c72da8cb5" }
#815web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "1c5b84d641ae52d4cfda277960102f99a01dc80148220e5f93bcbb4e518df885", "started_sha256": "2b46df371335371f595e5b9c351012c53e4670559b7f254bab1b977a05fc9666", "original_grade_sha256": "91b2dd6f3f3dd0bbf4b899523b2ea87e078243a5b476789d5b1946dcaf7975be", "input_original_sha256": "1c5b84d641ae52d4cfda277960102f99a01dc80148220e5f93bcbb4e518df885", "input_wav_sha256": "7e75f0958bb6978fd258691310f267663e10ac7f9834342afe8d9309a2efa0a9", "input_pcm_sha256": "2b01b94cd8749043dea8acff0015180fb159cd09414cad1aca09926d5e4f9080", "output_wav_sha256": "62cbe8575f2a843e26a1005694a5c1fda9b2ca181e3eec17f16742dd8ccea5f7", "output_pcm_sha256": "3c579b91da52b6277647c2c360adcb890b9d62275f17385d32c909736f6d3f8e", "result_sha256": "69f5f64eeb531e3881dd5e071acb445ba7b5a65f8e9ebc6b522b19d7840e73fc" }
#192formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "4ce58c096e9326e972cd4b1bdd08e8cada6c6ff8db36780866b1acf4156d16ff", "started_sha256": "cb4b3bb316c6290935584728457f81ba4e357bd53d789510569da9ae3eaf01a2", "original_grade_sha256": "7747e07900f20eef741cdb614ed9c6283ae9f951a31c46566102f9984296cc12", "input_original_sha256": "4ce58c096e9326e972cd4b1bdd08e8cada6c6ff8db36780866b1acf4156d16ff", "input_wav_sha256": "ab1c3fa559c119493c4385a12f0417484578a636f272ec0c285634addb83dc94", "input_pcm_sha256": "de20ee4d87cdaf03296e0545d606cdcd5018aa709c0b046ee9433bbea45f9460", "output_wav_sha256": "eb96bdaaccc1ae25e9acac7693c94139fd6b5fea20fba3a43b7f78c806d627fe", "output_pcm_sha256": "038a1eaf00e57ec8bae108e8ee6ab068ea60e8f640acafe60e05d8baf16f30fb", "result_sha256": "976baa79fa1178e42834f9658a398d9d2118a86887e85b741f6d4c1a621d0edd" }
#401navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "1821c35ebba58ca8cff5e291cc37866ee48d9cc01038010866c9b96ad09775b5", "started_sha256": "17049498b0354d0d40fa010ff5fbd973a257508520babe7d406e4d18ca2ff371", "original_grade_sha256": "5a219b4c99496c4192a45ecf3352ec95c023b728d4dfa1d4a50b4e6fc8f16cdc", "input_original_sha256": "1821c35ebba58ca8cff5e291cc37866ee48d9cc01038010866c9b96ad09775b5", "input_wav_sha256": "756fec3e35c50d211fc2a23d7006f7ee17077eb240775c4009d6b2434acea628", "input_pcm_sha256": "be8dd8b011dba0cb5349db89bca90c06496320e64c23760813bd7ada383c86a7", "output_wav_sha256": "807e18b40a90505863ba2e0cce484978ffc46bb961064a38a8aeeaf42699e68d", "output_pcm_sha256": "99a16e37c4037e3bfe8cc57c4d9269bebb198c72c9b50701b177d370542116de", "result_sha256": "7307827c4e077ad71fc21d471cb4e45f35988fd8060cc1e628798304fd29afea" }
#523object_counting heldout	10	The answer is 10.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "58d20ff4b5a4071aba7c9d8558a63370d0ef4e59b676a8449cfd4a5fd09ecfc9", "started_sha256": "4838c494d0a7e46627ceab4bf339ed66a0f35ee2b95c8c06e4dbb45d4717a9c9", "original_grade_sha256": "adfaa5f4b75d0ee2a14148cf1568c4a4704f3ab6633ee0386a70afef4161627c", "input_original_sha256": "58d20ff4b5a4071aba7c9d8558a63370d0ef4e59b676a8449cfd4a5fd09ecfc9", "input_wav_sha256": "b663dc7ff443e0d8be7410f696d9941dcfdec572833d8d1ce9c634fda19f79aa", "input_pcm_sha256": "8ff3fbdc5b4d406f46d1ebe72648408f32d9a8168004729171170632d79dbae0", "output_wav_sha256": "d58d1728deaed009999af155c039e9fad366c8c967c81118b8cb7bbf400f240a", "output_pcm_sha256": "0f3ef29ecad097be909a33c00e48032ff4ef9d9ae98d3b75f95d46ad0cec2235", "result_sha256": "eaa0031ecd34c2138d39f49e7b6b0f8013d58324be7b6c45274cb1654fc48c57" }
#949web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "c32bb6e10952dedb2d6202a53395cb9d804f854d4bf56c3a381ad858c6d2cfd4", "started_sha256": "2b036efa094033820742aeab8ba4bb8cd73d809e266c091b953091fb0a1fb036", "original_grade_sha256": "d456a1b68aa78f94430bf6310b04060b96f869e560206658e2edbbb8e3e9de81", "input_original_sha256": "c32bb6e10952dedb2d6202a53395cb9d804f854d4bf56c3a381ad858c6d2cfd4", "input_wav_sha256": "aeb5565454cef678acd9ea7b8947bec15f7ac876bfb28dc7680ac08f181559fd", "input_pcm_sha256": "fc75fff15f294acfbe33880707ade2345f579c6dfabe5d78e5ec9055e8bbe388", "output_wav_sha256": "4da98f4466b1a3aa89e725f5def50dd00c8e4afa231f7d9031dcf7f7c699816e", "output_pcm_sha256": "3556ed5f62ea0faeb03ec8e1ea0a296d1ca62b1c51b24ba49aa78854530463cd", "result_sha256": "7827a7d084f2c9e394c7e4927e6ad2649b9ae359feff4766ede4167696888056" }
#102formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "1d02213eef98a9f13eab2a5cbab5702925b2adeaa0d8a06dbbd838e111608a35", "started_sha256": "1de929fdebaf3dfc2dd587291e26bcebaf7acda0ebe20c0870ab7dfed483ffcc", "original_grade_sha256": "122e245f84b293e92aa988c4f38ac52402d2dcc69a6f5810a398cd9824c11bed", "input_original_sha256": "1d02213eef98a9f13eab2a5cbab5702925b2adeaa0d8a06dbbd838e111608a35", "input_wav_sha256": "a273fc10fb498cf4b282c6efc12c175a3cfee95a7a24b15cd48a0d52a1b6315e", "input_pcm_sha256": "ee9f316f9af3c1ac36a39e19e7285611900511c3c43b8610340af12907883efb", "output_wav_sha256": "e934655f0ab4172beeeee7dc916af03cd664b92fec57af29e92fb08930169fae", "output_pcm_sha256": "99ca472e09900da5cffd4aeb784a3c99998a824d1cb82dae8afb27d19477735c", "result_sha256": "eae457254a9dd8d76cb60131ad792da5b75e469d542fd2cb180fa6bdb1b93619" }
#494navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "06d30853da04b38e9da5de885fd08ae22e484b8cf355977a9b7ef3358927ce19", "started_sha256": "b7d34de9048c2008f7d2a128810aa6a2d9bc117e1a1486972f4cce3f01817fcc", "original_grade_sha256": "34a9d2bdc83a79b0f910741a8d23e8220888b29baf80085e88f092f7ca8fc322", "input_original_sha256": "06d30853da04b38e9da5de885fd08ae22e484b8cf355977a9b7ef3358927ce19", "input_wav_sha256": "d28abab5680f58898c5e42b8f88b5f165d68924592e52630d3163d41c797a197", "input_pcm_sha256": "6d1718f7ee7262d5030e4b8eb93216f983d64e91aac8b60e248fbaa52ed400fb", "output_wav_sha256": "543ce00a4486343d7235feee3af7385a274be68a9fa745baa7ee5762aac0207e", "output_pcm_sha256": "267b98d31a78ee7a095d42e158fde884937eb27f5f2fab07fb8504a630a0ad05", "result_sha256": "8a41383c4430306ad7a371199729afa4a2dc2f867f279c0811788a5f060027ba" }
#500object_counting heldout	8	The answer is 8.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "647f492444a0f226bb4cfc553190c291abb565b9dba93853cd9897c74600cd74", "started_sha256": "df8182a7ef8fa2f157d284d2338063241705328701ddb86ffb4f255d56584c26", "original_grade_sha256": "af5e3de7d19840fcd9b3380ee1b76a2d3e2ab4ee66dbb5702bcb937ae2e0e662", "input_original_sha256": "647f492444a0f226bb4cfc553190c291abb565b9dba93853cd9897c74600cd74", "input_wav_sha256": "c7606921a537a1e3ced329ec2d1f029c1bd6b4d846599a065db73ba337763d40", "input_pcm_sha256": "5a1b738791912e6a0a7076c59f8dd5a4b77cee450705ef6f5e8348d67020fdd8", "output_wav_sha256": "5cc7382d02239c59356ab9f7d46b21fc0c16f59e989d67fd6bb23a423ccd6ad6", "output_pcm_sha256": "6d8f5b658495b1e5d24e65083aa88227b88089339dad5436d7fd0ccd91bf9b2e", "result_sha256": "8b210d1309a75d16afa1469cb3e9b8f460f2e04aaa17e53789b86e2cbef9bc16" }
#919web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "aaea6fc90d167a0bc056375c0fe87ecf73273a997401e5d26ff28e3796e19146", "started_sha256": "3ecf02bf4ee814876bd8d3c8600eb1d986ef62c589afa771cac1991aaef210fe", "original_grade_sha256": "bb66a049419b6b8ed59fd721e0be6be6b9e4ec36b608ebac4f45b548c4ede324", "input_original_sha256": "aaea6fc90d167a0bc056375c0fe87ecf73273a997401e5d26ff28e3796e19146", "input_wav_sha256": "7ad13568ebf76e5fd6aca112b176ee55213215cb81707020d7660f3bebca4c54", "input_pcm_sha256": "cd6b33491e70b9497082685d9a16b655bfd7cf8cd49ee1fdea032d4c17572417", "output_wav_sha256": "4e2c4bb44f4faa4be24583c6a95da47213421fdaa806d0be2b8a261bdafdb042", "output_pcm_sha256": "a0693396d61137b741aca21cbab72832ca149e65b5f973feb21753bfc88c786a", "result_sha256": "578cb20517011d9e96e0e70c9c2b6c0f6831726b8881c08f6ee0ed4443138605" }
#125formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "ea86b739c195c90c7bdeb2858bd93cd7dc5395e9486b7a9573f8134c88d327c5", "started_sha256": "edb3d376f709c94771eb7bccde8315dad101275b68c055133e04b74bda29d779", "original_grade_sha256": "0eeb26763361457ddebd8c88ccc1c318347561f63674e2a5423b11908b600538", "input_original_sha256": "ea86b739c195c90c7bdeb2858bd93cd7dc5395e9486b7a9573f8134c88d327c5", "input_wav_sha256": "4617d2d8f7c9775c37ea3ab8b8e5747659280f2a0a6bf6905952f1fe34f0b6e9", "input_pcm_sha256": "65726302f367ee1a1e207de96197b7f4133155a79effe048588a82edeaa273ba", "output_wav_sha256": "943ab854bb413db2d536960eb68b5b18975f07ce3e3c9341392e6b6fcd351b9b", "output_pcm_sha256": "1a04cf41f3656e3f5991e3e9f9f499bf7d961c86e0edf23cc239eccdcbe9384f", "result_sha256": "990736bf6c09a26dc94f9472c283f4b2593b56c4580fefa89c4720eed9b0217a" }
#363navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "58284ca0b43e5acdb0ba22fc5dd72438f4cd09ad7c63c1462b7a2c8cbccdda89", "started_sha256": "a8b25ece19d6de73e9a960abfdb48c097387fd9faeece1dd623d16864c23e684", "original_grade_sha256": "d465717551232c84112c384d483aff231a7c6511de029daaf349f0b933967d2b", "input_original_sha256": "58284ca0b43e5acdb0ba22fc5dd72438f4cd09ad7c63c1462b7a2c8cbccdda89", "input_wav_sha256": "acdcf5e9d8746c0216d1415b79342f874e2f1be16b82c5f30b85e53509d2f596", "input_pcm_sha256": "172dcd6645a5dd1fdf1405f229ef59858d3df8ee8b342a351546befdd34d4984", "output_wav_sha256": "83da8f2324ac72e57d90be188d27956fb2752edcf767e05051aca3b45ca83113", "output_pcm_sha256": "17c180b0cc0eed1da05bcb8d023ba544144d54a4c5cabb049ea62797f04e5f54", "result_sha256": "123f03cd68b5476ef7d8ac092072a1a7af7d0275ff1db147bf98a1e2d77dd6ea" }
#662object_counting heldout	7	The answer is 7.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "61092561c0fb8cd17d2ce7cbce7e769becfce81d2d22d09f96e7e58a50260903", "started_sha256": "eb7ceeffa0b50d70c507145c12301b47c67427615686bf84d369283212664ce4", "original_grade_sha256": "dea6739e1e17432eaed6316854f130382e70cfdf6964a08c79ec9e4fe1b9ec84", "input_original_sha256": "61092561c0fb8cd17d2ce7cbce7e769becfce81d2d22d09f96e7e58a50260903", "input_wav_sha256": "dd487c0bc5ea794451e5265bf0cf16cdbb6fe9eb9ce9b0270da6a02aaf8e2be5", "input_pcm_sha256": "2c3dc220d480550507f0f69a7681c4af1d01ffb2ea28b1fcdf1a0afd3066e27b", "output_wav_sha256": "3e1c168acc460c74941dc523f89112519c46b1db400ff1fa65a7e0d67e7db172", "output_pcm_sha256": "9fa0a66b63be9ca37b681ae5c0b7d83d0a4d2820906975292df458845236ea7f", "result_sha256": "4c1df086d2d3d61505c849320ed5d483460de99b39bfc429bdfab5b48e1f4b9d" }
#796web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "dd41d5fc1212e67abe895c93502ab9df92d46575b6ea6a5b6685b88c39006dd0", "started_sha256": "b7c1167f806c0ea9c43f196a12e0d09cadce4b1a602f664f85f5a9efdea22e31", "original_grade_sha256": "2dcd799c763534faa1ce31eb11528e3148f9b32bb1e91009fffe44c38f53c088", "input_original_sha256": "dd41d5fc1212e67abe895c93502ab9df92d46575b6ea6a5b6685b88c39006dd0", "input_wav_sha256": "9a645eeab018dc6fcffe35c0b0e0b10b8e4354816712a14f99e8cc50277fafd8", "input_pcm_sha256": "df4434e32de36a95c805ebc216b6213d007d89a1760052ac0ea2c11596fc8f88", "output_wav_sha256": "96096ae339b5e531f41446c0a4f2c85e1d8b13dbdd8f701f5d0ee844544f83d3", "output_pcm_sha256": "5edfc9b178026011489f34820114c08cd0962511bb450fbff7f860d32c835074", "result_sha256": "dc5356e6b2732a3cc065eff8723a5a618b0d038e8b79ed6c70ca289def6809fc" }
#120formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "04e9e0faec0675662189b8f286df7c4ad788a7002808b3beaa686ae19296402a", "started_sha256": "d493f4d32268f428cfcb13437105f1fc5b8e6f2e56e6de299e0efeca9435d1e0", "original_grade_sha256": "9a0b8be8146aeaa68852e57d8bd9c027647212c87e621b5af7c78477c3ffcf14", "input_original_sha256": "04e9e0faec0675662189b8f286df7c4ad788a7002808b3beaa686ae19296402a", "input_wav_sha256": "0069cba92e9d998c730d84573b345e421cd0b4910fd4c6eedb2a8cbdf7053353", "input_pcm_sha256": "d3b572436c5ae749f8dff856e54efcdad93a459c41922faadd2bfa87d20d0581", "output_wav_sha256": "cf6387bc802749fe88ea5c61fbd898faccc504f432a2ca8b140f0ce016b0ff47", "output_pcm_sha256": "7ff69e74488f66824dc3c10ac723fac0eb15fc04a297278cc2f623177ab85208", "result_sha256": "f4c0760b8a3d2b851922203f16327aad2707f784c09ebc9a569f8cc0071a52ab" }
#499navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "128b49aba4e5480c8faaa41557329d0ffdebb018d95e1547ae91dd0d634bdda8", "started_sha256": "ce1f20eab097609fdeb7794c1d7cb406064d13accd4749aece0719fde30f41af", "original_grade_sha256": "5876f199c810271e57b476728a05dd688c6a76152adf763e0110c4d982fc6bc0", "input_original_sha256": "128b49aba4e5480c8faaa41557329d0ffdebb018d95e1547ae91dd0d634bdda8", "input_wav_sha256": "4488e98c8ecbe188b735693d7fa60534eaf8b3be935dec3cd35e092542a1e9f2", "input_pcm_sha256": "65f06791823c3f5706abe49a2c17ac830fe638db651d5d0fc514a32725fa9c7b", "output_wav_sha256": "b196998e609e36859b7ea7a581fbea7094e12a3e3c7944065cc7f78fc53c8592", "output_pcm_sha256": "9365f599cd41e80444c336cdb6eabbbe5a01d33fd2cac2681f23a2f69f805dfc", "result_sha256": "7f7871f03664037b5005cc4ee298688e4df8c6a1d1a5e5f1c8deef93867928e5" }
#619object_counting heldout	9	The answer is 9.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "36509d1375803d1dd56aa4f5b19ae7b397c08d24c2329e9ac9540d1a06506bfa", "started_sha256": "0bdbfe62efa607b43d8666592a1037892278a700ea8f7927a09069345769e591", "original_grade_sha256": "c9bdae0c3336b904533798b47659c2827c22c2c3b9b10b60bfe9e3758d631372", "input_original_sha256": "36509d1375803d1dd56aa4f5b19ae7b397c08d24c2329e9ac9540d1a06506bfa", "input_wav_sha256": "a5b7923137c4a9d6d64f61cc7878843a1df2b822d0d8b7db8566765194a6a127", "input_pcm_sha256": "7da8dcd0d42df0ad60714d272b24810c308a9743fe8d632c65ce3978544ea93a", "output_wav_sha256": "4e083f99f94dc470bf6ed2f9e69bde1252ccf85681a60739476548de296c0749", "output_pcm_sha256": "8bb7041f562cc4b972b743a56aeedc76213641af2734a48251800c70aadd5855", "result_sha256": "2ec07c22f6c77b8f3363bd710c4c8be39ff750540f23df4375d03bdb49b502b8" }
#948web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "eaf3875da4031c8e3ece9228391905a720f07837e352288f3be99745b6e92d66", "started_sha256": "cdb7ff8398a7b663e7eef6a93d700256f859eeab8184453bb4878a6e6b3f7d07", "original_grade_sha256": "4ee0d5136c2e5dd8867eb4d4afdad27d80c5b5a9774d2390de26719ed6fb5b04", "input_original_sha256": "eaf3875da4031c8e3ece9228391905a720f07837e352288f3be99745b6e92d66", "input_wav_sha256": "016a877bfa44ef725b3f21fb5ca21d6b9daebf76093ea02fa9ffdfff13550bda", "input_pcm_sha256": "ee236006667660b3d2882ab294d3f68acdca52fd1c4c8c0ffdbfbe7f6a4d4fc8", "output_wav_sha256": "3fc6aed06aaf73c678a004c4195799e82ea364e732874641a92b17525ce7b8db", "output_pcm_sha256": "c7076a40b714892f71c52e51e6ef39eb68b3f9db63c992d1185af44b2c12f481", "result_sha256": "64c8952b64ae3fa7742a0a8fd74762492867e4243f6a048d3244d3b9d0f34466" }
#104formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "59b42f45285e71d53b1109a24deaf61a2f9625c846155df0544b72a046486d0d", "started_sha256": "ab7d44b0b0a6673a3d8b494a2015f59724fb20456a7439db653d4d24d73ebf56", "original_grade_sha256": "5459909071dea24d406e05f250a5883e99a95af1a79b545fdc30fb87f9b0557d", "input_original_sha256": "59b42f45285e71d53b1109a24deaf61a2f9625c846155df0544b72a046486d0d", "input_wav_sha256": "bf47ede8ef5d6d4b2c2eb98e13bd4d5cc7a590754da4a63a7ee3983cfd0c3acf", "input_pcm_sha256": "ee1f58bc5bbf05255ed8efdea8cad8a2d5f380b81ef13afe932cb6d464f0538d", "output_wav_sha256": "4638be92c89c965f3d17e488ccab6efd09604953b3f770d9a884af0d41b2844c", "output_pcm_sha256": "d0158a3cc8f45461ca49e953a76cc2cade404fbe756369db1b5599b3175f2f25", "result_sha256": "c9eb907ea46d43feac167e352ea64312ff2a71005925b9e71b8525589ebcf463" }
#407navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "0f74057a22356cfa06c5b8a5d6c8343a294cf051ab4031d55b5339ccb24e4e6a", "started_sha256": "8aa1386dd674bd08a605eafbda53e2890111e80a145fb0f7abb0b26372558611", "original_grade_sha256": "5599ed6e8ef9ebc9c10aeed65d2e58829ba315ec75a1bbf2a901a61852851aeb", "input_original_sha256": "0f74057a22356cfa06c5b8a5d6c8343a294cf051ab4031d55b5339ccb24e4e6a", "input_wav_sha256": "0a57b9c488d2a140015eb64d70006c05c9c014b1d6f3f751d8ad86c904e61a14", "input_pcm_sha256": "7b64e10ddec46ed9da35796367d7c3968bff3dd71e97b26f1d938b1c5afad795", "output_wav_sha256": "fb7d956212d8727b50f53e3f160b5d22d3d34bacdcf0547324662eed9b5a6323", "output_pcm_sha256": "fd18bc5b3a94d5c2a7350f8a723a7710ce1ec6fd58f28716b7c848479e92cb5f", "result_sha256": "d30c753e8254fc74cc289df49e0c4ad15bb5ebdaae8adca6201b8c71d2547176" }
#588object_counting heldout	13	The answer is 13.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "013540e4b687a6cd1a1578ff29dc54e682baaeb60986b1e9870caca6bf79123b", "started_sha256": "eda9e2d4bf5b849441a73038340b8e2f8e71a20dfb08459760abeb16dca50f63", "original_grade_sha256": "65ed1f8d2323f2aee4d7c0977f3d7dbfcf99464cc0bdea2a3da2ea084ebcc417", "input_original_sha256": "013540e4b687a6cd1a1578ff29dc54e682baaeb60986b1e9870caca6bf79123b", "input_wav_sha256": "e87b5877926bdaaa9ae61ae1bc4bb335eff7c26c06bf23c53e63078076761484", "input_pcm_sha256": "147a495dd63d6b5474bec53226ed155e6ba557c0d9afb7c36122ec5c6e62aaee", "output_wav_sha256": "f50e0ddc0c2364079763db3f5303c8b387b609566560a80d82d361ebbb588f3a", "output_pcm_sha256": "cc6a23005fd8b611f8403648beeb110318c4198b4f1829135e41acc8253f4b6f", "result_sha256": "0af80c44a4fc52e4997933ad2203962e805c1b5a2445f3b9508ff829077782ae" }
#987web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "1f178e693ed1766b4f8961dfb3b61e9837f10e8dcba9985ba6f448db02f3ec3c", "started_sha256": "6c19180cc6a5c0e9f24a944f84c37d8798e2bb693516c1f6e44d00240f7744e8", "original_grade_sha256": "9df370576dc54d0e06f4ca4a69247640a2b9dc747dab7b08a92149d1391e691d", "input_original_sha256": "1f178e693ed1766b4f8961dfb3b61e9837f10e8dcba9985ba6f448db02f3ec3c", "input_wav_sha256": "5272768fd427fa221fd50844726b70f1be7bbcef6edc95eea9e5143b6a28357c", "input_pcm_sha256": "5d1e74dee2eab08c0e06fab6d6aac48a0a5b1f828f38067942ab73579de901a8", "output_wav_sha256": "566f381f0c98f80409ee799526a0a71819c807f31f984d227afd3021bf4438cf", "output_pcm_sha256": "df7c0c677767f71c58a7acbf654ebe3124d4afa9f354ac417ed170cb02b0e228", "result_sha256": "f4ba6569fc7b1fe3b871120dcd833b5059c47088262126ac9a234bb821258dcd" }
#216formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b9075628b54eca2028069d1b7b9d83202f66ca9b8fb8904b6def3821cd09316d", "started_sha256": "2eb477c280bc923b03d8b7bd771fddf16668f8214c2af2e74a105258ddc9c618", "original_grade_sha256": "d1dc619bb3c81d3ce8c4d04bb5306b1f1b8ef1c1f0800d28ae46c03bc1d9ba04", "input_original_sha256": "b9075628b54eca2028069d1b7b9d83202f66ca9b8fb8904b6def3821cd09316d", "input_wav_sha256": "6f9969678fb88891725240480fcee0c8df1c361ac37df3e3880306b1bff86d96", "input_pcm_sha256": "045381357561ae6c5c4f9d00d7032e18d29cf49719b543e4c226977a2a8b24d9", "output_wav_sha256": "c2604e53c800c6c38122534314c5f2bd06060dbc2589bcd53f02aa0ee8f9bb52", "output_pcm_sha256": "d412b8a71593a6308fe91938d55a74eb2d9adf039a7e0b2d97c2d741201fea1d", "result_sha256": "a0b464f879a0905e893e604ef7c382fd910ff127314edc770f99d1266d3ef1f7" }
#450navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "4126882f779547aac0b98fbe0e3acf0950b9cd87de62571faeff15e36e4eceba", "started_sha256": "fa71dda5305a9ec2ab29f093c58c2c60e4aa182628f1bd9d6866a8842cc819ce", "original_grade_sha256": "c326a23555a4e2ad721e59b8e906c412edf3973c84f936d44ec3f7ddf8bd553b", "input_original_sha256": "4126882f779547aac0b98fbe0e3acf0950b9cd87de62571faeff15e36e4eceba", "input_wav_sha256": "64af66abb7f3e626fcca916f07d8fd1ea93d54d714585de1154d9e0efbb2f041", "input_pcm_sha256": "0ba21249c2a950e44bf6979a740a263adc44771639476c62a97e00b932d0eaa2", "output_wav_sha256": "82ae8bcf283ea921a221bf62470a686371da318d5e86624d0840c6c663121002", "output_pcm_sha256": "2199c925e2867dc28725f1422079894a25c3bfda315c2c39c098ae6909644b7b", "result_sha256": "479e4e309b6d3913df685bab0b16ad69b1d3a687856abd5c55f18f2761604e3d" }
#553object_counting heldout	2	The answer is 2.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "8798693eaeaefa46b6fffcf4829b7c04c62e7e42cadc6b7dff28e5e3c1df1501", "started_sha256": "3578173720f6da9389970bc8dfdaf31a39e004249f1405d6deb66e73efc79ce2", "original_grade_sha256": "1e05c836d6654b5167dbe2e0949e9d36a02c7f166fde871e6d3e297051259520", "input_original_sha256": "8798693eaeaefa46b6fffcf4829b7c04c62e7e42cadc6b7dff28e5e3c1df1501", "input_wav_sha256": "2b574bd7b9518a0029084ac57878c301174b97522841bd45c133f726078da29a", "input_pcm_sha256": "5875286fc1645df6812f0602239baf1f60ee78ef43d0eba277bc6b778e02dc22", "output_wav_sha256": "ff7125a7a363a2c436d62c6d7f4b8952400670c1ce87e44bc04cb7039de75301", "output_pcm_sha256": "ad8c1376a287ec4349160f0a049de4cdb6f6b81affaad64c6969d0e662ae310b", "result_sha256": "5013ad843985db8f8e92841f7a4be479dde677b1056fa8dd0c8a6ca3402a28dd" }
#999web_of_lies heldout	No	That answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "e60a53922057cb7d1d95a88c79bfc33a343d5c94eb00ee61621d73768cfadff2", "started_sha256": "30cefd162f0b7595c9fe4c4e2a53b7c51e7585ed94024aa8edc409ab34e10890", "original_grade_sha256": "b109cdde81012f3b8aab10ec000a65428952b608c9d43ea7485b3e245bc34da7", "input_original_sha256": "e60a53922057cb7d1d95a88c79bfc33a343d5c94eb00ee61621d73768cfadff2", "input_wav_sha256": "8fdcfa7d68669c576c214d56ab98c97776657bdaf3195383bc8a0c1fc48fae3c", "input_pcm_sha256": "c80c329207501ae4d7e1d20b176cf1744be884ff07c11d115d5719bb6495890b", "output_wav_sha256": "b32813366d3d02c3b90ad3307e4a5d0630236ff4731f814f095ea6aa4f22fdfa", "output_pcm_sha256": "5db2fc9faa6118205e7f8ac59a9a4d4e22487941da439f0777e527d8711c1e7b", "result_sha256": "b5f2306a5e159c4d93d763987891b9a69a2c4a73336e97b59d9eb11b4664b2a4" }
#117formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "03ef39468c9e708387607f9fc4e50777bc893bdb4c82c6ced04cd81336a8bef7", "started_sha256": "d759f755e4344b97961e5a97eac220bd32eb548ae1e3bd2c6ceb55e822ca5e53", "original_grade_sha256": "a83f965870f3ed1bcd45e99c02dd03322b33bd5271704cb53040030965901fb1", "input_original_sha256": "03ef39468c9e708387607f9fc4e50777bc893bdb4c82c6ced04cd81336a8bef7", "input_wav_sha256": "651ad520b168c33f948b0e4cb1620a91485bec83e833b12a350b524c9b173b92", "input_pcm_sha256": "26bf96016ded5228b458cf932cbf2a787366b7497be3860abc93786bf3c7ab8f", "output_wav_sha256": "e6d84dbdfe1960c43924c7b0a6da192373500a9af1ed26fbce81ea62e226684d", "output_pcm_sha256": "e6ec855be8369e7a304b9fb776f454b94afe7cfd47b8d4f026b25c08da8ab986", "result_sha256": "a0304095f861fcc776fc6dbc0fdd52ce31aa75520ea0182ff723c983f6e39fc1" }
#265navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "482a4fae58274e5246914ba9a7405f57d1b606c97f89077ea970fd10ad382d29", "started_sha256": "b4ef73e51d05aaca6c1509e9845a85770d858517232ca713548579d8fd530c10", "original_grade_sha256": "a24f53b8750d26bf5bc0b4f3111533cfaafde0e8c18dfeca9c503d41c857d5f8", "input_original_sha256": "482a4fae58274e5246914ba9a7405f57d1b606c97f89077ea970fd10ad382d29", "input_wav_sha256": "5b2aaa4fb21cd101209611bf38cefc00917fcf6d31884fc0385eb18bceebc2b8", "input_pcm_sha256": "3dede84c78ebaeac014cfe71e1fadde308d72a0adcb777ba7a9cd5098e02b5fe", "output_wav_sha256": "a120424c4279550c55e8d02ddf5349968ddc8602911948375f5087e390c24189", "output_pcm_sha256": "7386a9a62327e8b48ff6044030f7483d77d8a44404ad453b5cdbaf667a2fc618", "result_sha256": "569c5cbe19ddaabf2a28d1467a1c6aaef623bfb7964c5cded47f7b3096da7389" }
#586object_counting heldout	9	The answer is 9.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "161e426a8bdf20b0487ba8538a4a8be426dd2ae83c26ebfb367d65af996168e1", "started_sha256": "c3904e1781831c3c6618ff2137b0b2cb11b426976a900f055382ff1d35148962", "original_grade_sha256": "30bcbf7c12c6fc45dcaf7e0f8f8ec5621ab397589f87ac6fe6633e0cb617c484", "input_original_sha256": "161e426a8bdf20b0487ba8538a4a8be426dd2ae83c26ebfb367d65af996168e1", "input_wav_sha256": "c9b1769e075c7e62c1994177e9233edacd7089c1669fd0892e13537b45d2be8c", "input_pcm_sha256": "68457dbed3ad14ee4c7e384a75ebb4479adfebf8c71c1c61f8564e17be5c39b8", "output_wav_sha256": "6a73abca78911939387f322c4efa599cb39809450e39fa6fe15f9d31f1e2542e", "output_pcm_sha256": "7278f783d813f73cb43fdc4d90fecf1ade4f5859a1326aee68ff41e62b7a9c5a", "result_sha256": "72e5ee6ac19f42066f6c051eedccb0648588f3a51e875f2d82a4f11d3ace0347" }
#781web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "1a978394dfa8e69c3b3ec7b0494bb12b075d281ed0c17b1dd8b704d1a39131e3", "started_sha256": "9aebb3f154147e2150f1417f64177e4d81e41855f5603f4a3ef5a8aa856bec15", "original_grade_sha256": "9d1ccbc717110012d28be2958053d2f538c7a7290cead632b580fe605e3bc169", "input_original_sha256": "1a978394dfa8e69c3b3ec7b0494bb12b075d281ed0c17b1dd8b704d1a39131e3", "input_wav_sha256": "fc6a8a1d4c4df54c0b7aff338d6fa9d9eb449f91cd809e352008c117bfeebad4", "input_pcm_sha256": "aeb6c1f6c71276404d12b3aa3a52b29fadc5843486c1663fd8479c82b050a4fd", "output_wav_sha256": "a25e040a3ad8012a762cc022d4fd9042d5cdf1482d1041b8c70c4fa09e737d50", "output_pcm_sha256": "69d28183a1aec14ae3390ae794e9252f57f3e2f2636fa1085e58f1df391a0c62", "result_sha256": "1379a04b9569d2329d2e45a0ac56b6d50194c3c07c8b32bb8653377095a7c31d" }
#51formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "edff7695374b8b089fd2bb07bc88e35dd1ee90548eb7ad2a29082a939f018dd2", "started_sha256": "5dd2a185b52bfe26df2d7337fba1af2847c327a5c80dd2ca9fb02669ac88f4d3", "original_grade_sha256": "e59819e27bd68a14a68a0327fd16c1a3ba888a8f7d6523b318fd0d4334b6327c", "input_original_sha256": "edff7695374b8b089fd2bb07bc88e35dd1ee90548eb7ad2a29082a939f018dd2", "input_wav_sha256": "4b3542fa0ea9392fb4ed37a4c4712485309efd0c5de1e72df810df47b5b960e4", "input_pcm_sha256": "5b2676b1f6d38d4f73c3bfef5edb101c7bcf2be4637c268a5820ba3395113348", "output_wav_sha256": "a70024147fc57138005977855b12e4b1d034024c14834280f357f6fe5463befb", "output_pcm_sha256": "218bd9a2244a99a5bfe45304d79f787b71beee8c61e1f5759a6fd3f2bfe5c380", "result_sha256": "00ad6d90d701fcf21a840145727e6d6997f422e8875e4e09b0bb45b51884911b" }
#476navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "abdfe7aa6a94dcc81ce9e844421e36b311bafcdf35b25df22857777edc16d645", "started_sha256": "f69baaa2cee9558b22a2f75bf0ab17fb4a9f5b32a94d449e78fedc005a90cdcf", "original_grade_sha256": "281af64ea3cfa1f7d10602f5cd536c29cc99d194004623c230e6cc4d7cadd142", "input_original_sha256": "abdfe7aa6a94dcc81ce9e844421e36b311bafcdf35b25df22857777edc16d645", "input_wav_sha256": "bdc013c146555c1be332056ee2adb9134c498dd73b35efddddcdb70d26cc8c9c", "input_pcm_sha256": "87bd54afb40fa20a553936b198a28d8f9477c945ab9ea8c193e10658d1fc99fa", "output_wav_sha256": "3dfc964361b913b586db4d38768ee8a16a46969cccc4cdfa7bfe9ed3f5cf6697", "output_pcm_sha256": "8fc0dad6bbf6a58eac6d83a57e52b1d79fb910e285a6fce1df710c651db5d71d", "result_sha256": "ab0fb7c2a7d01d6867c648c48da999e299dbcc519356c6de07ea947efdc5ec40" }
#686object_counting heldout	16	The answer is 16.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "a54616a77986850a277e3fba83a27ab596c680c3ce1aedf51b32ef2dfa35ac72", "started_sha256": "cb4e9d6c843091dc424ad7f48b4b6fcaaa29e6346b895ded3205dbaa47906027", "original_grade_sha256": "28b7b09b6e5a1dfd13fcf45ff8b3c8156224efffd2d575d890c07c2a51290cf0", "input_original_sha256": "a54616a77986850a277e3fba83a27ab596c680c3ce1aedf51b32ef2dfa35ac72", "input_wav_sha256": "7357f9f4f1e2bceb79313f69b1e2e6ac4142b964d2ca6031ccee972d74e65316", "input_pcm_sha256": "4451ff6db7870bac60f481e6fe8ca99baed6712104673863739076709c4a6e29", "output_wav_sha256": "f7d9424da62d85c7dd6183d2eb7f74fae1a03f535a7aef5ac66d0d1a9d8a99ea", "output_pcm_sha256": "d5751422661b3b7e5d66b18a73513e4def5400d2c3151bd5377d346e768c3035", "result_sha256": "d08463669888fe2b235972782ca2eb5310eed8832829e854db763c2492b681b4" }
#939web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "fbf35a5df2b1717ec521ad121738b41f920de494d4cd45518e5b904ad9a0a466", "started_sha256": "ae5688729db087bbd052bd1f3aa647244f8e6f4c07fc980e63e6c2c58b1feeb9", "original_grade_sha256": "82d14e0a29b2246972fe2cd2348f5858656463c23d78a9ca30cf31ad3cc3174d", "input_original_sha256": "fbf35a5df2b1717ec521ad121738b41f920de494d4cd45518e5b904ad9a0a466", "input_wav_sha256": "c8e47b3e17be8d078c7be3ef48dd54be3f191b7351810a4d9ea00beba1e4015e", "input_pcm_sha256": "ed8da464b801a5c8b9c549c7b47bf9d3920462c3a599689b5279ca4fd8f4bb1e", "output_wav_sha256": "b6cf18b126ef102fac2a934dad9eb93c03a6214cfb6ebe999b35ab902d02fe79", "output_pcm_sha256": "8193b12c83332215681867603b9ef360e165c416133c08993d77cffd6b5b9bc2", "result_sha256": "c9636fe6eb10f3adf3d63fd03b171a69d59f8ae8b73583bf58f0f805ad3af101" }
#67formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "535d66344ec494d0b10172db25f901604e06855a9813fc02e1c17de0a20eb233", "started_sha256": "b22ec23e771f3ed3dd4e2f6c2bbe7d033ecd4bca59f09085c23b5c3496688347", "original_grade_sha256": "dc5e46ff3d92927871e243269eb354df1836062edaad6309c5a2c7aacd842344", "input_original_sha256": "535d66344ec494d0b10172db25f901604e06855a9813fc02e1c17de0a20eb233", "input_wav_sha256": "95bae7dedd31ab39139aef15ce9e415fb74f7d86c21fbc5b74ed0637238f09a2", "input_pcm_sha256": "fa165efb50660f615d97c4591686a34b19de8a3c2b5264db8462c88b1eef259c", "output_wav_sha256": "b493746b0c0d5921e3b474fc6ceba88815cb52011fe309b9db749978ab8c438f", "output_pcm_sha256": "f0654ecbb5227fd063a4ea4f981686f6b74f0574aad4acd75fea81c83ca740db", "result_sha256": "73189c266ec96a9a70824cc4f8c3dcb5888720c5d6ee29d88a098689f0d3a9e8" }
#441navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "eeaa8c785a43e02412e94ab33327955e094bb949d0aea2671e4319187c5d6ae4", "started_sha256": "80771edc338fec14617ddae6733753428f1924b1f09b40587401ba75e33b3b75", "original_grade_sha256": "d8479da731721930bc415ee6be6fd515ebb7bf999270093213780e7455b11fd2", "input_original_sha256": "eeaa8c785a43e02412e94ab33327955e094bb949d0aea2671e4319187c5d6ae4", "input_wav_sha256": "cd391c5c34381b72d95b7dd49a1584734bfcd822846bab33918fea531f0c67e0", "input_pcm_sha256": "2b64f1a34a5e641be8272c6f518ad04c34c1d346f5c1cb0f99c957cc40aa2d1d", "output_wav_sha256": "661c2f92f52aa267033aeef34cb6bf55987b9d8ec0016b68717dabc0c950e576", "output_pcm_sha256": "217019d8d186a36ab2c38fe8bcea2bb783ad9d227dbec1238893953705eeeb71", "result_sha256": "0a26beb9be9932918bc0fbd536ef890c8f469b85eab064d4f104962857a5d953" }
#731object_counting heldout	11	The answer is 11.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "5aae7b7339354f70e57e54156b6d4efa98804a0ecc8da1f173a9daa00d3e5d07", "started_sha256": "32b8e5862854bb8cc4f0fe54e0b828d86efb74f64fb8ab5c5d78ea8d9c46463e", "original_grade_sha256": "806afaf648e34a552d491a6ae867c243b6ab65974aeaca3d170b7924263cd7e6", "input_original_sha256": "5aae7b7339354f70e57e54156b6d4efa98804a0ecc8da1f173a9daa00d3e5d07", "input_wav_sha256": "18601e89947601ff57c978c1882fc56d299ff48fea2dabb840100b3cef69ee4c", "input_pcm_sha256": "d08214a03320f3cba95d03bd2857d6ce960acd359487d8a83065b61edb3bba5d", "output_wav_sha256": "c3166c288dfac805775b1df7ea0fc82be4914147597b898ab874349d5ea08bda", "output_pcm_sha256": "0274a0b0251c57127c463f2451077aeb562a756455668afeb0f79a58e045673b", "result_sha256": "5176cdd176ece9dc6aba3d62576fd0d8be1120ab39c740d535191a865c8cffea" }
#967web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "251a0f556901ff655e8fa11b591e607a5bf831d18b962e52c94f25e70e0f2afd", "started_sha256": "aa4bd286920a8a13a45d6c45df61ffea934d01bbe6db9429120923fd55be088a", "original_grade_sha256": "381964e5580c2eb1b0d16440b334bc8b35bfa56a861e7996da2117f763e074dd", "input_original_sha256": "251a0f556901ff655e8fa11b591e607a5bf831d18b962e52c94f25e70e0f2afd", "input_wav_sha256": "5e4694a1ac2cb606557957be819610198307faa908ae6f9fa3c88890c2efd1db", "input_pcm_sha256": "0a125c53912a290fb8095dc0bead706b3368221b44dcacfaec4535647a211257", "output_wav_sha256": "eb31981a85a56e0ed12f4cb009993582b461807367ec9e768bab51d9fbe3bd99", "output_pcm_sha256": "ff6713ab28fd0996ac534d0fa26b4f8bf1c8767f0e169944f1b15c5f07330fd8", "result_sha256": "5c34c27fa70fc6d5889cdb7c31d08eea9bd7d350e2aa95baa6ab5a3f46fe5296" }
#119formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "8c59d5927c66ef1f23524014b54bddc82e6734d4c50aef8a8ca17e5e9643cd78", "started_sha256": "2d68f8f0c96bc5c267a0a70a9d2ad0f267ac6952064729d6b3d1cd99d51d2bf8", "original_grade_sha256": "a58e81f3d23143512f1735f363f1d58e50d5381908f1e03984563ee12c6c2d26", "input_original_sha256": "8c59d5927c66ef1f23524014b54bddc82e6734d4c50aef8a8ca17e5e9643cd78", "input_wav_sha256": "212ef14330fcecb7b548c80d015deb2be3bb4543904a1dc07d14fb3483aaac20", "input_pcm_sha256": "8a46a7ff01c8047ee5ee8dd2d7be04d277a833e2b02ef7295a51209ef8473bd7", "output_wav_sha256": "39ca6afc8dab548b7c8543d99336a7207c6b631656bfc80b5ef5189b2aff0eee", "output_pcm_sha256": "60c3f387fa32ec000e097eca47481787892d254fb44d5140b322eee9321e6d66", "result_sha256": "abf5f02f723efe0f7422da9d0492e51f266cd5f2010b7137894d29cb1c1dfb26" }
#335navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "348e4fbf4631c4783d929a53676d8ddfb3741d0a21c8e6ac1f36536c3be86518", "started_sha256": "b9687f6dad920a4afd6795a553da3dc9d296e7963527db74944f12436264c02d", "original_grade_sha256": "06910e28b8f980083e092e60aa579c4b1474bef463ab1325c4180570ae971958", "input_original_sha256": "348e4fbf4631c4783d929a53676d8ddfb3741d0a21c8e6ac1f36536c3be86518", "input_wav_sha256": "a7a67fd9dfd0ff573e6c50257cd013d6a8dbc9699e0712c190e9b0d0f44bd1a6", "input_pcm_sha256": "25ff2966ce31c9897823f5a5f696979ebb55d563a8493be650247e68633a903a", "output_wav_sha256": "04efb654a2c32263dc148a0368b4474ddd5a12769c6c16671909dfbfae892597", "output_pcm_sha256": "4a3a5cd60e61f93fd0b045b1f97cd8f35a812a4247d6919fd8245aa7f7b2fb4c", "result_sha256": "39bdd59932f794938925eb22f2cd5d37d917612e7f47f45339df300cb65265d7" }
#616object_counting heldout	11	The answer is 11.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "58c8dcdb2cd8f2f1ec9308fcc14b7a73d6766362bd52cdb9af7bb7eb5dab3ca7", "started_sha256": "e03dd47bb923caf2482613348058e2738f3434f8916815a7ee958f2511f38251", "original_grade_sha256": "ca5c7b7b7f1511f420b1fb2b69c430526fe44b571e68984919eb6f776694589f", "input_original_sha256": "58c8dcdb2cd8f2f1ec9308fcc14b7a73d6766362bd52cdb9af7bb7eb5dab3ca7", "input_wav_sha256": "cca4b7b2f1bdbe7310412226433cf40204c8cdbcdab5dc0eb63e932998623155", "input_pcm_sha256": "2f3b6c275fa7e545c49f14d5ec6a22db415af68cfb8ec2901b2b4289b1f6484f", "output_wav_sha256": "bb05af49f5564bae1f3064b78c39976ce355bd5c20f469ecb06f9d9cdcf2c447", "output_pcm_sha256": "9714441b1aad6387dd61f6edce2f22a5cc3dc15e40932d86efaf582edeb1ece3", "result_sha256": "8b61d57b827a2741264acb1a590b8d0179e524430d66f91e0247f743cc22d4d6" }
#758web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "022d515cd50b219e5d62794ce2b52f494b54f4dfa7449d642e331c0d36918a85", "started_sha256": "1c84fe797d2b52f2980f0786a12f3d464defe2231bf221d9eac034841c8dc4d4", "original_grade_sha256": "9b8fb80e6d19a3576241025200ee7bdf16e96a16b773ab81298926b21632ed60", "input_original_sha256": "022d515cd50b219e5d62794ce2b52f494b54f4dfa7449d642e331c0d36918a85", "input_wav_sha256": "c2d3dcc2a346a6751bbf60a0a05c4ab6535f573d77516a65f321f78cbfa721f6", "input_pcm_sha256": "68178772c6bf16a5e9b3bc31eb64c97ddda99a7bcbdecabe3c75336ca25b0d54", "output_wav_sha256": "88c7f26736ff47cd99848ffa7040d8bd39d2d00f6ec3c0768ce321b8551c0469", "output_pcm_sha256": "d4bd02356ff9be3ab9a60c3e132afc9db9440f57b6fa26c336ae5fd165e65b50", "result_sha256": "bf3c843bd680c8fb910039ca72292bbea50ced03bee947cc167d6fcca886cbf8" }
#12formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "06fd411ade292fa739c42f090a8e29242fd524416ede0c91d14a0df32d77566f", "started_sha256": "2563a116c67ebf1766fca7a9f8646394d5a4677dea7d279440c2b3c2b6b2266a", "original_grade_sha256": "944411413a8a9e51393f4751eb6a826cc42b8af3784f731266a70e2b331064b3", "input_original_sha256": "06fd411ade292fa739c42f090a8e29242fd524416ede0c91d14a0df32d77566f", "input_wav_sha256": "6f847d42a5edd594a98a6fd36ee558ca3fad5b0774c991c5c58eb959049d4de0", "input_pcm_sha256": "7e0fd309170e87eb78b54f1962da0218daf2294e3e513365fcd1ce79beca9884", "output_wav_sha256": "19574e4c17fe7864f34770715e32125fea878aef8fc4b8a09bdf8beb6a500e7e", "output_pcm_sha256": "bb2392743efa1741c51dfe6408bef5b1d6650e645744aace98850a21b1cd998f", "result_sha256": "846bb3e0a1fd5577c17d542d5fdb56676057175c4b72ef0116af2fe24a574fa6" }
#254navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "a2233e91931eeb4d756eb04c777de7159e8d04887170108070dbe999b6520665", "started_sha256": "12e372615ebaf5c1f660c3290c642c950932146ea413ce65e3f1903c0c336735", "original_grade_sha256": "1f6f3c4f2c78d3f3fc5bb3bb4431190ce672aa2c9974904879c8bb4cf83d79bc", "input_original_sha256": "a2233e91931eeb4d756eb04c777de7159e8d04887170108070dbe999b6520665", "input_wav_sha256": "04302979630c106727f26240cca4a1cdaae3cc5b955e4910680ee881dfa9d9ff", "input_pcm_sha256": "355b7aae2c11c813a1fc4753e54b7c52cafaa8d6cf2ffb0a92e2387cd71d7e99", "output_wav_sha256": "749798590a4563317cc696a0f9d3609f4939afea88ece4603b8a8c22d87f447a", "output_pcm_sha256": "dd4ed2bef245b1968d36c8dfa770b832686dd62e7263935f174520f803722c5e", "result_sha256": "19cc187f913c0300c4e709643c472575ea596d9ca82729e83f4aa77247082421" }
#560object_counting heldout	18	The answer is 18.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "bd1acc690edc71f1a0419a2cc49a0bfa3ef1f67ef408a20de09dd981d0564f6c", "started_sha256": "3348a07ddbd822540375af82cc9dad65aa95b4bf5dc87966707f3983640267af", "original_grade_sha256": "0200a91b2eaf43d784360284071d1df5b03ea7ad3d03e70e7f45e0b078f473ed", "input_original_sha256": "bd1acc690edc71f1a0419a2cc49a0bfa3ef1f67ef408a20de09dd981d0564f6c", "input_wav_sha256": "cdb276db166e1e516a14135f32109b01db686b064fef4bca15667e92ba55b446", "input_pcm_sha256": "965890c615d81dbc84dd04552e6dfb654fb4f463807db28a51c6a528ce0b915a", "output_wav_sha256": "53a19a1044294a5512e5e17aee3297d7a64a9d30cdde8e574423e51f36d2999d", "output_pcm_sha256": "d7ccd0c55e9cf0a9387468313cea65ec2a793266913906967caeeda8a43f5d2c", "result_sha256": "31a304f3ab6adb7f326d013cfeb8854bbe1d04feee1dd4dcaebe40477252c514" }
#953web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "2e66df68055755bcee401211d201160942d42cb2038eef99e8c391f30eb9e23d", "started_sha256": "78c97e1a6e467df4251e8a9457a7a0bed012c06c143501cf14662f79c4019ff2", "original_grade_sha256": "e17a2a36922b7bb0c02588b451def68a8bbd5d21b9026590a1f6ae84591b3272", "input_original_sha256": "2e66df68055755bcee401211d201160942d42cb2038eef99e8c391f30eb9e23d", "input_wav_sha256": "2edc9cbb964394424c6a9bb630b059547b5db3bb6d8d1cd1d9fc1e24daf6d8c3", "input_pcm_sha256": "0ac8eeaf913d4f35a8a4372cbe3b48d56ca50f4d1acc1fe0068aeedbee339c79", "output_wav_sha256": "a7898d34df150eabf6b3ea55f90566a05b9ff318b226569e43ac867922637db3", "output_pcm_sha256": "59b50a753f7a7e11bee4302bfd67d37af557b29b9c2251d5c1a3dff04663035f", "result_sha256": "1c3be99e7e5227363d5f8da64320148dd70337c75589bb7ff46e5099928457a7" }
#203formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "cde1d241cd929be0ba3bbd799eb81fc51ab2f1dec08a21900bc75118b8655455", "started_sha256": "0ce7aadea71019898894676e5ae714416f1478ada1dc5ac054bb063ed7a2c4f2", "original_grade_sha256": "7c5f28bc9878c309d5ff18fe8bddc158074f512840692787d1dc995f90424540", "input_original_sha256": "cde1d241cd929be0ba3bbd799eb81fc51ab2f1dec08a21900bc75118b8655455", "input_wav_sha256": "84b8b9fe7e9b0828fca2c38098c3b2874beca507a5b14f74fdcb93be01f71771", "input_pcm_sha256": "38f8a9c0278497465c272a93e99b275b3ae3a81607d116c8f0b40eff3280cb7b", "output_wav_sha256": "0571aa441eeec2f51081bc8d20fd48e40b758de8b4e1e6beb6c48e9db01b4141", "output_pcm_sha256": "76d38b2533cdc0a5d7f1bd845f38339848badc7b19e73f683410f752cd8ecf48", "result_sha256": "902dcfa9372b116fc3986ee89726d00d63644dfa7939fbf31a9815a3d996f3e0" }
#340navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "124d27ce52768e835e43de0ba264ea9c2ba0546cc8ea8c69a7657297dd2159f7", "started_sha256": "1a51e73fa961deaa2c3c6f77e379465c5eacf4a74343b0655746e3ccdb7e3299", "original_grade_sha256": "c62af53e2ae1c53588a3fec2ead6a71d99fbac4c73fbf13388f31623674f670f", "input_original_sha256": "124d27ce52768e835e43de0ba264ea9c2ba0546cc8ea8c69a7657297dd2159f7", "input_wav_sha256": "ad68c04ad18b3660f21fcaeca8f6968cfedfbfe6166387b9de26635862e14413", "input_pcm_sha256": "6ac69b8d9d6488e62de3f8315274dfee2d999b294202cd17a1279bc8a474e28b", "output_wav_sha256": "f6a3496e6acfa1c13325dbba2f4867aad2567a9fae8636f3f509282c9e02c647", "output_pcm_sha256": "ea4df5d5f3cb11b651b61c4d5d14a972ae569eb0b3382ea9ce280d8f87daced5", "result_sha256": "4cbce7779bc45cb9f7dbe26b562b044c9a50f5f854ab75be34a68834bd205839" }
#659object_counting heldout	8	The answer is 8.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "51051dee1a57ba7037c229de90af28c75f07e2c5b7edbac87cdae9957f5cd20a", "started_sha256": "ae6b27befa7615c107a3fc7553c770aa41069e8f5f8829ed56373bacc7403370", "original_grade_sha256": "28ec088fafb603246632285cceea8bebde4f14dc91fc6f455ca42946d722659a", "input_original_sha256": "51051dee1a57ba7037c229de90af28c75f07e2c5b7edbac87cdae9957f5cd20a", "input_wav_sha256": "1ca56dce31b4b8e904bb10f11c6513a9c4fdf8fdf77d7d8fb40ee3c0ef853a2c", "input_pcm_sha256": "8a3e05202021fb1bc606bebb93ba2210738a2f47652195fdcec606738414a7c2", "output_wav_sha256": "95b8c88f284cc0138ef687300cfd124189aced8038646fac2c620c079ac6676c", "output_pcm_sha256": "173cae15561b4ced87496d7cf5d57aadc50caa380eee147d90d0bbb454e7e1f8", "result_sha256": "319e5221886eff6d623d1b8561e3bd3b8af618fa63e95e416df2598790addbc7" }
#887web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "8b824479a44b2cda85e8e1c592bf1dc9e4fb058933b3ddcf4582d20d9edefef9", "started_sha256": "3113158fb2dfccaf63f2bb479bbaa9aa46e99d94903639b2ff54750088f4a016", "original_grade_sha256": "f4a9352469b7dfab793326404ac0748da1c9ee9d854944853b6034d63483e444", "input_original_sha256": "8b824479a44b2cda85e8e1c592bf1dc9e4fb058933b3ddcf4582d20d9edefef9", "input_wav_sha256": "37b63e7d81a3e55e6c8e95973aec79591df86c2f9a2f93553f939ccce1268257", "input_pcm_sha256": "8b784223fd85d6d1fb9f36b78d9ce04a67406374684c8092bc886a842c7c6c7a", "output_wav_sha256": "e7b79afa588cef041d31ceaf6268d480eed47d02009b98d2d3f540beb18283b5", "output_pcm_sha256": "3a63fbb2e3665c43a57356071d8004dc660f7df357066a1eb1ae0e7d6f5197ef", "result_sha256": "0448a6e55fe2ec15ebf10b6b1df5f3a65e9b4fd26312b443c31032e7a93ab6d4" }
#226formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "0aff83d6c31dae06993720dc44eaca12bdf742825988c589176e8e61912db82d", "started_sha256": "7b90be359a1df65287d4837039e5a8008caffa23a7b9d06c9711f4c749213217", "original_grade_sha256": "cab4849ce9c33f157aee2e09845a1ffb7ffd216a91ffff88706e3611035d75a6", "input_original_sha256": "0aff83d6c31dae06993720dc44eaca12bdf742825988c589176e8e61912db82d", "input_wav_sha256": "56f23482f4bfc05db21e10208a386494e30968f70fdd9cf8a4921ed12d4e302a", "input_pcm_sha256": "84ccd2a7fe23ed749b91d0f65a0dbabb0ddf2a4e2523af46f8b10565c6e233e9", "output_wav_sha256": "dbac7be32f36cca74c491342b7fb4126a6edff695e0fd0c0cb164435195c33c7", "output_pcm_sha256": "2ca141e91ae7d91b8ce6f54ae99c557003c9b703a753423dfa33d52ba1f26403", "result_sha256": "156016bc025580846cceeb8728a256fc08138e56f85421603ac10f8bca64f68d" }
#411navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "3a67b01343011782bfae013ecf67ef07bc130af3aa34a522e07990dfee935843", "started_sha256": "1843812dc49aca5bc0c474af67df2897d1a36a5ebf48b9dbe649623f4e52ac9d", "original_grade_sha256": "c35fee3533ccbf4509d3fc7583a5661963cb8bcc0c121c7a94d6447fba458d2b", "input_original_sha256": "3a67b01343011782bfae013ecf67ef07bc130af3aa34a522e07990dfee935843", "input_wav_sha256": "195402d5778f73be72bcc5b08a788af060ebefd9814bf5b1d60eb045fd93596c", "input_pcm_sha256": "23c728f2182637813d54414c2a89dfc0916622606694045366ce6e48bee6cfe4", "output_wav_sha256": "3b33d46c0fb37b6d544416af8b232d2f233c60178b13b2fb660196a9f1fd4ab9", "output_pcm_sha256": "a143e839957dc1daa61209e98fcf06ed35c3eb5ec4a17fb6340f7725f2e611bb", "result_sha256": "582f9ae0cfb40b4f8a7b24d8b5a72aff0ee13ad6572382e8c3882655bf26ae82" }
#550object_counting heldout	10	The answer is 10.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "bc8fa237d3c10617b392f294d26106744c6cf2c99e14bd962d4d66101bcb43fe", "started_sha256": "e51e86dcba383a43b49c1743d7cb971755e190cbf138a3632e3c96238d019b0e", "original_grade_sha256": "7f64179a175fa8c5231de09571de94c44aa865fb085e148f1d8e2eb9d0fbdcb5", "input_original_sha256": "bc8fa237d3c10617b392f294d26106744c6cf2c99e14bd962d4d66101bcb43fe", "input_wav_sha256": "384c8ebb96ad44e2a09ac8fc2e149fd5c3112d4f209b059b36a25f52868f850f", "input_pcm_sha256": "58366924036625af60e17afb7e1e6ea73f9e5642bf9a8ea5c6f4892dd97606a4", "output_wav_sha256": "39043873769d788d4c9810a8a73c7f134b4a0fc93309fe73d5af84e755b4ee7c", "output_pcm_sha256": "286989ec10d5cf6b7b0372710ccc6056d348280d8fd5ed2e9be7bb77b1d6bef2", "result_sha256": "e93828b45b860c1d2ac4ce40bac2466f5397eb6198c2ef2a8a0d2ab917c6ea93" }
#795web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "33ee4759718ac968f289790fb440dd40ab9916dccbd7143b81efa68eb2e9314e", "started_sha256": "9351d884447effc837114968e0fb522d5a1358108241e31b41bc6839e51b78a2", "original_grade_sha256": "34e1578f317c33c947ebd5abfce4d687fde5c6c2d115c4360c2f2f2dd8fdefa1", "input_original_sha256": "33ee4759718ac968f289790fb440dd40ab9916dccbd7143b81efa68eb2e9314e", "input_wav_sha256": "8da720d0bf6c05c057da4e42088095517c62416386de5d83b0312a89175685e5", "input_pcm_sha256": "8337064122b042e84b586de62aa6cce6c495e18004eb9c2ce6d605921e4d7cb8", "output_wav_sha256": "05af9ab9dec6f318de8ac3e31661bb3d7220b39789ca78d408b90ef6d1dee772", "output_pcm_sha256": "117cd39e053b0179adf2d19d4299326e72c5a05ca4459b08c213396f9120e4fa", "result_sha256": "3fb505f115dee5fbd402a8550971908642fe6423517b02b9493afacaa8b40e74" }
#170formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "37e97c3d7bb7e437f01e3b18c5e2e194f28f9ac5185e383dbfb3e4e1e63d82da", "started_sha256": "acf07cdbd8de61ae58879b1f9372eebca815c7c2e9e875bada927ea1dfcc5eff", "original_grade_sha256": "c8b5c2756af7b5fd11eb40eaa650b9bb80bb1e96db787fcbae2fc7254e188d1e", "input_original_sha256": "37e97c3d7bb7e437f01e3b18c5e2e194f28f9ac5185e383dbfb3e4e1e63d82da", "input_wav_sha256": "61da8af0aba3bf445eb04439f598bc0576bd4e42abacfbfa5e900530ac86bb32", "input_pcm_sha256": "9ec38036b0a72f7150f735c6e8f7ef7bf6fde0cb0480be342e8a0493add58bc9", "output_wav_sha256": "bf958791bab1ea50c5cb27d5f46d0394a5140ae160fb447fb6847e131063c014", "output_pcm_sha256": "d82f584b6678bfdf3a36fb0dac54a4afbefceedb8c84f3f1858378fd18679d78", "result_sha256": "960488032bd22bfa89186d7b42153601246f7bb5a81a50831bd0bc9a6b92d240" }
#360navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "dce4d410761f24e2c02ab69055f987378ad35ade599cf58d0c4d0a5be9bff1ab", "started_sha256": "52ade2217d027ca35e60af9b2175c153a2cdb172eca752863a09841a4c46118e", "original_grade_sha256": "b5484a6f5f25554ab79d6276706e62eef0ba06e0b46d2b22d3651d2dc7531f2a", "input_original_sha256": "dce4d410761f24e2c02ab69055f987378ad35ade599cf58d0c4d0a5be9bff1ab", "input_wav_sha256": "fc49a32281c6d38cda85beb26f6c81a836f16e47039b4091bab504360fe30e57", "input_pcm_sha256": "e7f20b6636498796f8f657241004a72d75ade70ab02c1394c752ff2bd3311d3c", "output_wav_sha256": "28677d5775546be94fb2985f85c0fa81c960f296c3a462db71dba9aa61d3a416", "output_pcm_sha256": "986c861d37e27aa29369473f5cf369f482356b66f87c16e6a4497e05e66dc4a7", "result_sha256": "462bd08cc8cf9aa69e2086fd74a06a2a6e7ec95f2c4eee643614d66c88a3e2a4" }
#548object_counting heldout	13	The answer is 13.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "58fba910573b051f95eea484e4eb334bd237fed93c71e391fa5e2f7f20f3b35f", "started_sha256": "0eda8e01bc06fd2a76afb83109a422d24809cfb69b9a7726a6d4f41412e6ebde", "original_grade_sha256": "ec6a7a370e8658a2c80d8611fcdf069b25f9c6526da292430de09c342efd6277", "input_original_sha256": "58fba910573b051f95eea484e4eb334bd237fed93c71e391fa5e2f7f20f3b35f", "input_wav_sha256": "db2e84783d2888211965699bb47bc8dc139a3ad3c4e5b88860f34d44dba117b4", "input_pcm_sha256": "55930b3458c189ef7994ceb81780e44d62894a937653fac3ce1ff48f60dd2a46", "output_wav_sha256": "8a1657d8abb2e805216671b173f7e0e64514c764e6e8d8d856e96b4d598de02c", "output_pcm_sha256": "254bd4f55ba6732ecfab676905aa8d46a1e0dca482f340d815ebaafa4e71a785", "result_sha256": "688a66bdf53d42791549ee40608449664321fd7ea0815bbf813f4c69b81773b5" }
#963web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b62d7a65cd2abf3c0e18ee53e58c7fe0728587d38667822020077a2413fd06e8", "started_sha256": "181c035967968b5e61b3af7a53b4b1f0ffdf4f2a2036e6fd857939081dfcaad2", "original_grade_sha256": "e5ae303984f8db6d765e2130498347af1d2e66b1afca2048b732d513943fee04", "input_original_sha256": "b62d7a65cd2abf3c0e18ee53e58c7fe0728587d38667822020077a2413fd06e8", "input_wav_sha256": "818fdadf71d50ea32f44c7b06b417b93a71d0ad8bf6351339a688f52f884277f", "input_pcm_sha256": "c90a9c356fb3a01f6f0ad506f8eb40941efc871125abd8c221bbb2dcc5318a59", "output_wav_sha256": "6bcfa624587a4b5c7685bf1cdfd6f12e6a97506b6a73c19629d23f034dd2c387", "output_pcm_sha256": "76c0896594d591c5eee61b9c6b7b16670b2ef4607adcec5820d721c981dd7c44", "result_sha256": "b88043df3dd3263e56a2ece7f760837d1e5349623a2f3f19f7a3e7acad728e82" }
#2formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "0c021b2133e035d0a0047bb7e20eda65db4fa9b2f3a6ee5b7756fa2e41624f78", "started_sha256": "44d9e118e686ed2e98a2c8c7aaf0d522d9ef95c90c0354c34464324d4b4a2ec1", "original_grade_sha256": "70de33e753afb14fe25ad0029bbcc91d908b5fa0139c465e8922a287abf9729e", "input_original_sha256": "0c021b2133e035d0a0047bb7e20eda65db4fa9b2f3a6ee5b7756fa2e41624f78", "input_wav_sha256": "af6c12402f70a9661848c291ba8f9ce59b67085433e4c2ee59cfb6410ad0a448", "input_pcm_sha256": "f6f46b7ecb9a39b0cb042ad1ce34906c0f4c48ddbd1c88a2e0b4d20280caea22", "output_wav_sha256": "a2018f1ecad37837ad9a378fb2107d48c9962db4308904e67fb2ea20ca5bf4a4", "output_pcm_sha256": "d33582b6e48cc6a3f42f2ac162aec5faf183b093cb11ae2b1dc35a75acb60bb5", "result_sha256": "84a4eb1ea6392f7ed4e6da3109ba7de38518d55701e4d3fe757afa22ddd7ce1e" }
#282navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b378a895a12e0ffe90d1731294b0346bac3b8d6fee9f3b039c2a00f9fe160271", "started_sha256": "4dbcfeab0e3018d1d435e8d5d64f1f50b80c4994f533bb13cbabed2049d163e0", "original_grade_sha256": "debce86e98f8a2a974d262639612a56589995f527dbd923157fd389484fb915e", "input_original_sha256": "b378a895a12e0ffe90d1731294b0346bac3b8d6fee9f3b039c2a00f9fe160271", "input_wav_sha256": "471928dd00d888ef65ab9ee0670da22d6b49d76e68b5ba59fd737f4c35945c52", "input_pcm_sha256": "4d82c5c0fa8df6d27909ddf72fdacfa75fafe34ed8032fa99dc3b66eefedb52a", "output_wav_sha256": "66f9471b5e6527cb591ad9cea742d4cf50f6531784c89aba06ecb597f1785aea", "output_pcm_sha256": "c4652d8067583a2cf581a5945ed05f7e6fe6df841fbf3c1d557e51f7719d33f7", "result_sha256": "b253fe34d2ccf1e3690cad4bfa53f625db023fc0be9466c6a1b456bf5e0e34b3" }
#558object_counting heldout	11	The answer is 11.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "8f38835cdda6105ce456db86823b2e27728a4339de86f9a50ae19a392bd4e571", "started_sha256": "0fd504926d3a8eeab5c50a80bcc4f197468f659786a0f33e0604ce7442bf69c6", "original_grade_sha256": "8e496eccf6aeaabfa09db2db5ea5c088f44010ecb316aaf529173758bbd6a914", "input_original_sha256": "8f38835cdda6105ce456db86823b2e27728a4339de86f9a50ae19a392bd4e571", "input_wav_sha256": "3e413a56b882b72c424b5091b4f7c7ea67b164db782f53179e895909d4f8a810", "input_pcm_sha256": "89355c1a53002b805111276b0d4c7985b3be6ff7c69498afb6c515df6912b673", "output_wav_sha256": "478266ab1a053825afacd485ed494461df6756ee16ba59a02232a2153b5dba99", "output_pcm_sha256": "f58db880441947ecd90b5f13a6f30a98219fd94a7fb97f4efe1afe44826623ac", "result_sha256": "1253901f2b20a34ff4a4230f73de8c4d49396cb15f14f17699ba3910c25d82d1" }
#846web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "338eefa8b81e36b8d0280374b61b30573cdd6680d97719daeb9563eee2f29fc5", "started_sha256": "3edc79e8a84cd386eab293230586c83aed2f39e00f1a53701e5523e54644d46a", "original_grade_sha256": "87a3e361e29df85ccdac799462aeadb77fac6d2fae68840c431300951c994ffd", "input_original_sha256": "338eefa8b81e36b8d0280374b61b30573cdd6680d97719daeb9563eee2f29fc5", "input_wav_sha256": "80ba60cf1cadb17128551eee2f3cacae7d34e74a4c6519f6aa4fb275caded247", "input_pcm_sha256": "50ef28b32221bcbda42eaf7350dd1c81bd4c4f0d2c2c42a8986686f5139d2b65", "output_wav_sha256": "6a0b3e8f09576850b931e4991121fa428bb29e2fc6222c1cceb828f7e3ce5628", "output_pcm_sha256": "5cf4304b0921e99d2e44b06be0a0867a44a504219c22fa197f847d250fe25521", "result_sha256": "6d8dc9bbdc8085616451525291d4d4cad8c6e59562948f1054367b110906a96f" }
#70formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "5c73b0f95447f1e4cb5986b5b31b9d685516ffaa7ba166467f13a4df8105a92d", "started_sha256": "441b0f5b7e5fa321a1c1f1267cd8a5f38b0137297a4f993c92ac3ff7caf5330c", "original_grade_sha256": "dfb871143f113c1d7e12764fcbb903e1b9163217516f9d907bfd7cf232a5e000", "input_original_sha256": "5c73b0f95447f1e4cb5986b5b31b9d685516ffaa7ba166467f13a4df8105a92d", "input_wav_sha256": "3dc5b37faf0d527b394009ca0a90592dade3c7ed9cc5851b24205a6a828f960e", "input_pcm_sha256": "fa63b119fddcb9cbcd43985ba0cc30b2c6d802aa273ca638a5b8aa447d56c8f3", "output_wav_sha256": "2fcc1c4ab69496c974c42e7b06e7e87c6f3783edb7c80c6c18c2010f01c5b20a", "output_pcm_sha256": "60399775c6c98656e25797b18763099f0882f6b0bbe9e039ef52d1c4f54a0711", "result_sha256": "560937b0be4c2cfc0f4f72b2a8e6a0215ce7994ad3acd96f0716d6c09f3a566b" }
#276navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "3699b23812fc3b7dd6fedd6939a6f4043ddee1807aebc99d593c5ef5b38fd462", "started_sha256": "88a03831605720dc46cdaca79d802a87a5e0aa77cad5615ef6af705dfb3d6d55", "original_grade_sha256": "c1be96091394eb740a91b386b571f37361190914c6cc2db47403d4aa19d9d9d9", "input_original_sha256": "3699b23812fc3b7dd6fedd6939a6f4043ddee1807aebc99d593c5ef5b38fd462", "input_wav_sha256": "7940effb801a322cd196de713f3021be38516aaa41de8778f36013640531d909", "input_pcm_sha256": "5cc9ba2ca68cfc7596bc7610988ebb5f0a5772855139f68c8c89758c8172c56a", "output_wav_sha256": "3c1932b64814e78ff98e73263725a6d04e2fb02f9b05ccd34fa2b9d55a05f52f", "output_pcm_sha256": "92dcc85129743d1c51abd01d23b356c4f6c510e70afd9a1f98843661830e23b3", "result_sha256": "c5c8e52b678ac725f9bfdd0169c58e2c97381e35ce1af823111880d0ab507e7e" }
#650object_counting heldout	9	The answer is 9.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f58c7bf95862314dd2d1b5f699c6df8263e6ef7c99e8874797bc2b6b5cf9a32d", "started_sha256": "8ad3d0660b3c3236e458918c35c8356678a7e3a6f7f7b85410109cd1b4f1f4ef", "original_grade_sha256": "8844efd60270a26c7ecdf443c1e41806a1efb96b6f860c47c3e8c996c25f4536", "input_original_sha256": "f58c7bf95862314dd2d1b5f699c6df8263e6ef7c99e8874797bc2b6b5cf9a32d", "input_wav_sha256": "1f8dbc94a7cd6f68f07385b26c03c9e86c5501c766c1608ad32972dae175207b", "input_pcm_sha256": "8e563f4a6d8aaeb633d58a077d4f6216ff6884fcec7f632165193c7d84ca2b66", "output_wav_sha256": "77da025ab50ddd4aa85fcb6fbb3e93b0a8764e7bf9b436a83a9f41bf82db86f7", "output_pcm_sha256": "339cfe6601a15c23da6f5a0c658d5ea5369b1b2cca7f7730529a19309e943812", "result_sha256": "2860d48d084193fa49a87f83f7a05b5b8d86967129cda8bc17354e612d1b8fb8" }
#933web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "fcb858dd0755dc9652035e6e4f83679c353f1a4947322f892f1120144399b966", "started_sha256": "bdc668a46cf272c4c36d04d5394d7f3f6812c5548ae6c2fd9e2ec9c4d92b8aba", "original_grade_sha256": "f2ce109a0e359bda99135df9e642b192dcaf13f4c21c72ec870f363f41c587e5", "input_original_sha256": "fcb858dd0755dc9652035e6e4f83679c353f1a4947322f892f1120144399b966", "input_wav_sha256": "e53c666f4f40e9fc81d09c82209ff08dbe4801fdcfc4e1b8a5833ccaef04423f", "input_pcm_sha256": "d2cda86527411c72a46ccbc27f1cccc6efcc60c5949b1c7d2255a4dd2f2b22e6", "output_wav_sha256": "74242960813e0a66ce6224e8b68ea418c8ca564d80bf9552217dc716fc50ad5f", "output_pcm_sha256": "0c635e8eab42bca991855512d77904393b7c36154457851df071acdf6226bbb5", "result_sha256": "ed285a5d4cc8449fcb3437d174566c26ee42f059d7fe6016e8c10ea713cbcc8b" }
#145formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "9d8160c0f0dcbcd8eb4a6ea651e7413f618a5d00f7aa981fae894f5167c9522c", "started_sha256": "c00ffc1e038b59ebc6235f5d1fc58a9cad88345d7b46e1b7f0c17912877052b1", "original_grade_sha256": "fe08ab6618d58b9591b90bf1e819e43235cd04959903e8f1f92facfe1d2548df", "input_original_sha256": "9d8160c0f0dcbcd8eb4a6ea651e7413f618a5d00f7aa981fae894f5167c9522c", "input_wav_sha256": "1ae3f7ca874357282e8b7156d3a01b7483037098946f1f7e93d58b24c8737bde", "input_pcm_sha256": "7773d6be722dc661da8e8052f5cb433cad1aaf59685fa854b65c15ef5d6dc7bc", "output_wav_sha256": "874e175f992032a8619b236c6d6451945e0f79fe25a5a3e0122c5f3c2ae6f165", "output_pcm_sha256": "36c333bd8282a245c845834f7e34527b0813fe1b6c82261171c89bdc861f60af", "result_sha256": "a30b930903918cec3d38a27c710aa13922ba4733fa85a417ee8993bedf38935e" }
#305navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "860aa279e80d7b90a10793684094be4fb76545142fe4888fdb7f752d17265d61", "started_sha256": "2cff90eb29180d5acb1c37423d70c95a2ffec896c1272eda9fa3190856404920", "original_grade_sha256": "36f3a7161958c8bd8b78e7f2f4960d78b6624feb70f8f484d9d68793cd44bdaa", "input_original_sha256": "860aa279e80d7b90a10793684094be4fb76545142fe4888fdb7f752d17265d61", "input_wav_sha256": "6b44235c4bf321a0826a13854f57aa218073eadaa47444aef9e34732e3ef9130", "input_pcm_sha256": "338054ec689cfadeb4f8be1dbcdff20d1e9628ffb55ff810a89428a9c66ebb9e", "output_wav_sha256": "34eb30baaba29f70745707896869645cadefdff198b06fe0127b1e38afe5628f", "output_pcm_sha256": "39b9e516f4505451027ad63c01feb4a7c640d013ad4d8c6c17b5f4f13155e3fb", "result_sha256": "d49229c4fbb3437f15a7a45afc1272ccbefb770892012c67ed048ff87fb429e2" }
#713object_counting heldout	8	That answer is 8.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "6bce3e7f59102842376fdab8ee91b57c65fbc51ec676b400d665ebe82e60db23", "started_sha256": "7acd0fa00d63bce049a7a4ecde51c23ac31ffe334eb56f5c5baae5c3972d7d5c", "original_grade_sha256": "8ab5cdf478727d70b0043e23c838c46d05e23eb3f4983e1caefb6955448a7c48", "input_original_sha256": "6bce3e7f59102842376fdab8ee91b57c65fbc51ec676b400d665ebe82e60db23", "input_wav_sha256": "2415ce508cf17757e5bb1bcc7bfd92132471711097b61811491f52fe3fa11311", "input_pcm_sha256": "d71075efafc593cd6daada46b9027f8894b8024d51e447128025aa8f7912498c", "output_wav_sha256": "b8a255a1b62078d64c16911a143542d9437a7b67d68f967e8102df1fd87fc82a", "output_pcm_sha256": "45aa16dc75ca8334d1deed79d480d31c37d8e3cc2db1956cc7ae6a072102302b", "result_sha256": "96e1ee6bdebb7ebd40faa640db41693dbebbb88db02645c204cbaab5673a2e28" }
#994web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "e736fba9fa60993dead3c30318c7a7eabfacef9e6a5c99e4fb02006cfb3f02cd", "started_sha256": "c50b399c21753cee56e875279f42bba9a4bd147bf0d5dd2c6317b0cbbb08bd33", "original_grade_sha256": "042153e33da5b5ccdd6e59549d5a419ea24dc2ce92afc08963e103d60d42eb87", "input_original_sha256": "e736fba9fa60993dead3c30318c7a7eabfacef9e6a5c99e4fb02006cfb3f02cd", "input_wav_sha256": "1fb10f0d77c1c66d417ab2724e5fec3615bdd42ebba06a86c9d81629c81ba0a9", "input_pcm_sha256": "2966b7bf0f3663b20e4243dd24344262eae40ae4b086531c4786c31fd7a7818d", "output_wav_sha256": "e88552ae2033a3419e54a5095b94c51d37346b08ad75d9f671ace7eaba636727", "output_pcm_sha256": "00d9ecf6578b70e439cfe444255b31b03e44791acda174622a9bf023cd360b40", "result_sha256": "770145775b3f81c48df2772177f1741f0951861bfcc64825f73d684a8ca48ac4" }
#72formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "97af53a76b488a860b2db02659cfbc90f8ee2968b1b007f4a5ab149c0d202b49", "started_sha256": "ee21234f4b09a40a134a2d759324bb0c2170e9d950c5f8e6d492ba2f4917b3fd", "original_grade_sha256": "f29a2bad120fb2b2cca7c9d71e976335b96e720634374333fe28c7fb35f8abf7", "input_original_sha256": "97af53a76b488a860b2db02659cfbc90f8ee2968b1b007f4a5ab149c0d202b49", "input_wav_sha256": "34a63d426f35477d06b650702a4db55416f45f104ce709e5c4b61299b92c755c", "input_pcm_sha256": "027bc671c5e1d86650f67a215346bc4013255fce94578a5ec61775c280fc3f73", "output_wav_sha256": "01076074972144c6b7432713b0902c090956c0e3d53ad152103cac4b6121ed36", "output_pcm_sha256": "9d81d080e09ef1c87cf3d75ee788e6a8299edac01422e2f69e93258398bf01dc", "result_sha256": "928b9f6e432ed86470892862a9d39bbbebd8b260636032827bdaba4791d6e80c" }
#455navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "28732e6f89a4c9b04c2ab969a853450990c1d2f67f5619df537615e313f71183", "started_sha256": "d184d96ef9703015bee5e3bfcd9b2e41182975994471d73cc8659677cf6738e2", "original_grade_sha256": "3ad72e1021ad4f7e14e6b2d8a5a4d57ecef983ad1925adcb59fe80eef322ad14", "input_original_sha256": "28732e6f89a4c9b04c2ab969a853450990c1d2f67f5619df537615e313f71183", "input_wav_sha256": "f0fc2ef47987aaa754d6de0ff1e92b33cf4c2d54e95bd5233bae9b3a95694ef4", "input_pcm_sha256": "e1e457a62fb3294a70ffc952b60a33f3fd14aac209121c3735ab9f23adaf2bc8", "output_wav_sha256": "49a1a468ea7a6d29875de4043929b87406fd76d9f216ad3419209818fb0193f8", "output_pcm_sha256": "b7d64b1bc328247815480773c7e81a752ce1bb7fb6a86b6eb8791ab804ddf000", "result_sha256": "f02e1674c621888bc5f4a90ca1ee467af518179703044dd77a60c17556024a38" }
#710object_counting heldout	2	The answer is 2.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "d893ac2c7b770391eb524a5abf86f88e72f6bfe5ef366f81ab28fc8a075a25ee", "started_sha256": "a00c2cb9bde0416a41fa5e2cad7e9bf5d4a4e162775aacad6bcab7bde7af5ff4", "original_grade_sha256": "2cf56b1e8f37ee82c38988a52c59248ffeacaddc3fd0ca0632b218a43246eafb", "input_original_sha256": "d893ac2c7b770391eb524a5abf86f88e72f6bfe5ef366f81ab28fc8a075a25ee", "input_wav_sha256": "62131df3b804b5812c6f70f5ff03aba373ddd07c5bd220edd2fe7166a3557e57", "input_pcm_sha256": "e78c481ba57ca735d6805b2aae4240fee16dc2b906ea2001867ce174de38c641", "output_wav_sha256": "88f0f90f42d28900d3df6aaf0be13e70ab2e0e8ebcbcf10c774a9c0190abb56e", "output_pcm_sha256": "955a2f13b87accf2725914fcbb441d23db4a3788fc619ae089524c63882b2820", "result_sha256": "b6ea4fb91e61fc328b6fa93cd4ae236792c2c1406f79f1617cdb3d5fee9ee882" }
#779web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "c15a60c7cc91b4073f226aaf1db7d00df55bd321d42ec39c3ab9664b411413d9", "started_sha256": "b0d87440bcacbe8dd7b7637586926ebaf2078e96ac3536a212548463f18e9cb4", "original_grade_sha256": "05f0d8326dde7f7ab2045b0326628ddca1d74eb4d22019b48149a5124a874161", "input_original_sha256": "c15a60c7cc91b4073f226aaf1db7d00df55bd321d42ec39c3ab9664b411413d9", "input_wav_sha256": "2aab1527ce0dc6f33ece8f01f60fd5a15da3b6bc75533c1cc3a3e3e2f27e01b7", "input_pcm_sha256": "9522fb914289ac2ec1b3624d6497592146f167a0b3de91030f3a5d0b3f253132", "output_wav_sha256": "2f95377b929ceb277a2d2113e40031c472e653c2eddbcc5c656bfe16744aa425", "output_pcm_sha256": "173e970ed3c698f81f4bf2befd73b967733ee153ff18c3c37050b76297c5132a", "result_sha256": "4a0697370d8213f5778b2c46c55056bf1f50872fdc3d6cb1f1e49ac1621722ef" }
#18formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b038342aa5bdeaeab0cc7f762060a1c3d2bb3777c592f4aa364bf99866561977", "started_sha256": "a15f712e952b310849d5ea813e005377537e2d992dc7c7fdb987cb41e857f0e3", "original_grade_sha256": "3d3e3d6ade3b3a76b9311aeb384812253ea45602f55cff528112e5c58e520624", "input_original_sha256": "b038342aa5bdeaeab0cc7f762060a1c3d2bb3777c592f4aa364bf99866561977", "input_wav_sha256": "8ba9da573b59eedb40a8316a6e6c547816e472008e0db11445c31f0cbc7c5256", "input_pcm_sha256": "d26ee8fd448b14323339dd553cf8764af43027ddd78b5004230126d215381bcc", "output_wav_sha256": "6d0221d2614caf648e8b8ef8fdd87a452023c75dd2d43d959d563fc555979862", "output_pcm_sha256": "2e454bedf90309283763a58d8453dddb49d54d9cd29ac286eaf9936e2ae0b968", "result_sha256": "33cae110fd05fb2567464c179dd880a08f6eb23c540d1d3587f6e1b5472c95bd" }
#409navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "2ad7a009e3555e6ecb2e188aa21642cc979144a0b496cddca6fa01f28a529827", "started_sha256": "b081a330fdf48fcda61318239483a75fc1a79c8f954884fd909f47c71cf01a56", "original_grade_sha256": "20c8c68c12120d6df8a22e16753448ac37faf89423d784b38501edf14c4eead3", "input_original_sha256": "2ad7a009e3555e6ecb2e188aa21642cc979144a0b496cddca6fa01f28a529827", "input_wav_sha256": "b82b073a4116051cb52809a3c1b3846b90bb3ed826cc0b9c25cdd2357c362960", "input_pcm_sha256": "ac1edfbda0056d6b324aa0034fb21f6d6f335840c23746cb95fc4a2fcbef6c70", "output_wav_sha256": "134eb94e5c7998251218c8725d3ce9fc93fc61e711cec196d99fac52b95ad2a9", "output_pcm_sha256": "44b93ce816be0c313168252150421be0f5329edaf69ff4435fac686b9390e20a", "result_sha256": "bb006dcdbdec516827261a1edaaa8d43837fae5f392c7758e31799e385e73aa4" }
#685object_counting heldout	10	The answer is 10.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "5cdfb3bafa6bd53d6f856dbbd6be6c3cb5f9627853cb0f5deeadb5f34969f577", "started_sha256": "8f120ca67a61f9fdd2324b567d91f5c00f0030540dadf86c8ee31443586a6ec5", "original_grade_sha256": "18a8eb5f62065a774daeef3ed1ac0bc133f5391efb87d1f2c4185fb1c196be71", "input_original_sha256": "5cdfb3bafa6bd53d6f856dbbd6be6c3cb5f9627853cb0f5deeadb5f34969f577", "input_wav_sha256": "51d1a6df9a0ccef4986bb723d38fe542c9e6fc639bd7155e83b3293e45666c8e", "input_pcm_sha256": "d032e2ed92b90483701d89fb96b3cc59ba648c157616734fe30bed6bd57872c6", "output_wav_sha256": "2c3e6ad8ae072b53333acdcde211d865ffdfb4cc6176064023812deeb4971a81", "output_pcm_sha256": "2b97b47d4dc2a6277617c7799ecf06a14e3824c928d62ec5bae618264defccef", "result_sha256": "e48293ad8397cfead71acfcf4e1798e53288294e6477891eeea671b5ae7aaf96" }
#890web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "62e1a1e0907763d7a7de4c74521eef0a49dc14bfa9551b767365404833e4fd89", "started_sha256": "3c3193c5d36fec56384faac0889698604806242d00b6e4f78f26b95063b46e58", "original_grade_sha256": "4f0ca1dfc09d031c8ace5603e6cd6eaf0baef68ad850782917313a2dd726067e", "input_original_sha256": "62e1a1e0907763d7a7de4c74521eef0a49dc14bfa9551b767365404833e4fd89", "input_wav_sha256": "0dac5118172cf653d8214286a65ae8a13f631ad4124642adb4e671e617e56961", "input_pcm_sha256": "bf4d07db5c74211ecd2d968c580defa104549928abd21c802ccf10f04d1569b7", "output_wav_sha256": "774f675b56d2401f7d320228767b0d10926afbdb23b1292ecf5d4b8688adac4c", "output_pcm_sha256": "7542107a4c60cfe710a96bd8ee44fccb69e08e55d524b88b2ee789ea47aae076", "result_sha256": "97ff5cb3d81a621b7197e4e47795e14328b39137aeabb8e659def18f632a147e" }
#140formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "c76cae776778bbf72ca8b4a573e6f80bd91f5878c97987cdf377c2b6d68f1894", "started_sha256": "54b2f2c52721129781c252a012c7187fa609a670a32a5dd1a3015b998389333e", "original_grade_sha256": "8b24501cc120e388ae341766cce5fc78927a8f9c9369c5a5525d3e54983f0836", "input_original_sha256": "c76cae776778bbf72ca8b4a573e6f80bd91f5878c97987cdf377c2b6d68f1894", "input_wav_sha256": "ba2b501bdbda0626cf19f42ba1203608d266dea51986c0ab13af6d9c176d5821", "input_pcm_sha256": "99f5c285b58ea1871d1a8e91e1d06e556fd8edc2ba9c03124f0aa91d02b3a96e", "output_wav_sha256": "fa9e38d52caedd34cc8487cb63f1a353e872c2bb723ab53bb47c33454ff5674a", "output_pcm_sha256": "bc5f3979548765edf012edac19262ebe88368c800133618162dec7927b3832ad", "result_sha256": "46240f6ca7a280d220fc6352ada2d3ef9ad3b9988532cf35ccb7aa54fbc1db23" }
#331navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "89c0566914015dbc853b3f5718290c484cc169eadb1f7a870ddf321ef01c6e67", "started_sha256": "4e9f766fdb6989a738aae9d93416a8e52da77799709731ee88ca2b61fe50d1ff", "original_grade_sha256": "22428720ebd79ad2743c7427f5aad101f04ff9b16601d5ab0047daf5625ee579", "input_original_sha256": "89c0566914015dbc853b3f5718290c484cc169eadb1f7a870ddf321ef01c6e67", "input_wav_sha256": "456fdaa751ec34a4076ebcf9e48bcf09c67bb666fbc596888bb048ae448932c5", "input_pcm_sha256": "f4beb021d599beade9996b908a32b22c03fe713df31b6fbc054198ffab8b1ea4", "output_wav_sha256": "3143a450ff66008291c6d397120f24da9ce823746adfbe65fed7fa936f612f5a", "output_pcm_sha256": "771ffc939fdb1964c3e85fa4578aa7c840274ab0af0b6b711d6d5d97ce9aec90", "result_sha256": "7d00b6ea41ec59e95f7235d21a14968efbd4288c6e134a6f629c9fd082b83637" }
#733object_counting heldout	15	The answer is 15.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "719212f726072eef4450769af269e34d08cd6cde6de336a6198efc538b61fcf0", "started_sha256": "d1edd38fc0b768f2df4eb467873079727809485a7da227c5cd9aba9f33324e39", "original_grade_sha256": "7eb782367701c9119ac26f0490531ff7396c80d0ffe269e87eb98812d6189b01", "input_original_sha256": "719212f726072eef4450769af269e34d08cd6cde6de336a6198efc538b61fcf0", "input_wav_sha256": "868a4d7bd493248b34e3ed4017483b9636e3d8ddfb55a92ec1745828d35f6661", "input_pcm_sha256": "84b8177924feefb86244b0377c166d492f44d6c4adf9c9bf0b3e0c30111dd67f", "output_wav_sha256": "8eda7afca44de241ec30196c5ced00ffc50b5fb7cc683df82f99842ebba10c34", "output_pcm_sha256": "b0b43fbac9167f853364ab6fe41af80a9b70eaea1a83867eefc2ec7f5d2a183e", "result_sha256": "5cdc4ac54cc5722e107c6751e9b941bc522740eef727aeae8d8646ff80d452e1" }
#813web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "3c161cf542a3487513d3268d5c866c27d21dea37752b024f6516aa58b88c7f74", "started_sha256": "ac94f9c4ad95da6f9c2ffc6f570d4f97f8cad982dcbd2bb361d6659c8865ede0", "original_grade_sha256": "68db6226cfce98122aa2ecc4abc91ba015d6139f49ac7ca163cd43ff625e2e76", "input_original_sha256": "3c161cf542a3487513d3268d5c866c27d21dea37752b024f6516aa58b88c7f74", "input_wav_sha256": "b2ec416afdfcb857f1197e3397c42c704e58e321df5653b9f09fba0af6495ac7", "input_pcm_sha256": "6b32962abd4d6efe0a3e148ebd8e8b1031ccfce7f6c2b17423f00cf455b8c512", "output_wav_sha256": "aab12fcfd3e65ef532d5fb7fe35e292855ea94e4ce9e82233d4c30ff8cc29a1e", "output_pcm_sha256": "075f80bf83408b665ef5bbc71efdc0d94dd48c0960f3a33730571b4e459b0d26", "result_sha256": "ab0245e0c5934a5c538a87ada27c32d85269fe36c5d2195262173287db601035" }
#185formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "3b80e4721de77182b98887be071ac5c825e9c7dcfab213e2bd397fffe44f89dd", "started_sha256": "ceb9011ed19ba5dcf2175ceb59a9f475bff70b22ec679310c66a64bea90f05d2", "original_grade_sha256": "99c5799aff23a10e1ff0211f4bbe2cb1943cf74ad32655151dbf9193678f3aec", "input_original_sha256": "3b80e4721de77182b98887be071ac5c825e9c7dcfab213e2bd397fffe44f89dd", "input_wav_sha256": "ef688eed8d89627bc9a2befbc4e0e4ab89a317ed2a9a0ec05d135cb6b2542fb3", "input_pcm_sha256": "c15747dfa8380d3bd594001af410608b43032984387b7b64961fe9622560485c", "output_wav_sha256": "724c478f02160dc2421144ef9ca0b486515a822c67b2c25e517035aad0aaaa86", "output_pcm_sha256": "36974669b132abf7963484305c6d71cbdd476fb89c27b4590503bc8e2c07bd68", "result_sha256": "52a401409d5aaeb80d1749c3e91d1507d132efb9cd0a8b9039ef6f58165470b8" }
#355navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "78fcb8ec7b76c52371c8452b9bc0871e8fc6c917f7e56ff7704ecaa59fbe5435", "started_sha256": "970a92458e1de4ab8374b4bbce4b8f3d57616d458d21dc0beece3b16ad99a954", "original_grade_sha256": "0fc64918ee5cbb9b59240cc7c62052884ea755d048e760abe32b5fb6c242c32e", "input_original_sha256": "78fcb8ec7b76c52371c8452b9bc0871e8fc6c917f7e56ff7704ecaa59fbe5435", "input_wav_sha256": "ee0c75d4d312e963991a196f9464e1fd93f5b02be1f687ef8cffebe14fe60f47", "input_pcm_sha256": "2721557f48e890d5a9db81de51912e0e5c041519c65caf0d0e09811aa3918502", "output_wav_sha256": "e6cce0137a293cb9c4bedf45119b4a2684dde7e6734eb635883cf5a3dc7bbf75", "output_pcm_sha256": "fde751e813663a1e45d0d8c76b1c74ede9cd445206b36eedd23ee62200dc39c1", "result_sha256": "d281e98335d7a385de507603bcbe5e9b151ed407285d30160484d418bfee521b" }
#701object_counting heldout	3	The answer is 3.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "6e2943827d76b276cb965f8be47ac098dde2871d55d8f669156a93a578b988a4", "started_sha256": "d7f54b9ff3996a132c3b6a0a9add1bd8b87574dbea4fe5e0d7b06650fdea0112", "original_grade_sha256": "24ef912f42c251142114e133968a4db7e26adbfd5d04581fead0c3f3e3e5a15d", "input_original_sha256": "6e2943827d76b276cb965f8be47ac098dde2871d55d8f669156a93a578b988a4", "input_wav_sha256": "d936b09a4ddc5a1c7081404e90c7bd1f287d222a924ac39d00ecc22a1b20239c", "input_pcm_sha256": "a55c9e5b5a56065e30b2f3cc242f91f287f75695f996cfc59a865061d2e41da8", "output_wav_sha256": "66106460fc6da14f1ed25b622384f80727bba800812836720719436dcc9a5fb4", "output_pcm_sha256": "ef573a58037fcadde05b3d6b8f7a4f9dfc7a7796f12a6240e489d12b081fb233", "result_sha256": "3606dccdfa8772427cc56b5f145f25327b762e1f89eeddb82e4fe9a6d738f41d" }
#828web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "047908ef580df2ef7e1a29fd25d5fc210d8eda55e5ff3f8efcdbcbaec0575723", "started_sha256": "196201ae4dca9b528ab8828c1f8e743ead1e43fe7113f32f72f5db464daab162", "original_grade_sha256": "cec3ebdd91c0fd446e613fcc28d02266de644e9d027db79e74cd5bb7d2577ce8", "input_original_sha256": "047908ef580df2ef7e1a29fd25d5fc210d8eda55e5ff3f8efcdbcbaec0575723", "input_wav_sha256": "38474dc1dab3bda46d5edd3cf32160ca733585793d4ef2489225ea0ad5d5ec25", "input_pcm_sha256": "5ce4ddf486319c21998ae696e9851e0505bdafa2b0d285d93a7b44b70c6120dc", "output_wav_sha256": "dd38f41666751d5565e1d8c4607e02a56230ebe88af2d550d21069b36fdbf719", "output_pcm_sha256": "1885e5989c4edcb07ee223718df2ff4ecaa4097d82ecebc0e35ba34d09023b7e", "result_sha256": "a68b7d8049f8bc0adc4771d7590192beaf1683517971e7eb59f866a3d99f2817" }
#198formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "1e91a83fa81a2788b5e4185786a2140257fa8e41e988c87a562f8d804611fcea", "started_sha256": "d0bc7893bd3184b8baf2c848544a34bf069b7c5c82becafad8dcda2525930729", "original_grade_sha256": "4995230140ba2e02fd6171e43923f6f0328ce1b21a7a4538c2400c009c2a697f", "input_original_sha256": "1e91a83fa81a2788b5e4185786a2140257fa8e41e988c87a562f8d804611fcea", "input_wav_sha256": "c6f07ddd8f61ccd9a72cbd253bef805b7057f5f0f625b5a0ec93adc1bda92e89", "input_pcm_sha256": "99052b813ed8d7436cd9c989b4f0410b64034487e1d72febb36a8745d7eb3b23", "output_wav_sha256": "5a163427ec84e2065be5c87a3332dcef164216abee4a7050ec8f57aab227c69b", "output_pcm_sha256": "60553e9483deb6c8abd8d8f959d368e23a01266cd99fcbeb3d8becae264152c5", "result_sha256": "80538458fa752fb8d079ebae6e93f4fd591a9bf05f091ea3fc19ae62c7f4dbea" }
#278navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "8264f8a5938094e7f63e0fcb621da0271d0a78c803a29ae61e29807155416c56", "started_sha256": "47ba722e471bbad72663635af258a7aa3c093e996deeb968161956a20c1758c6", "original_grade_sha256": "9bb7375fb8e5141a8df9a5ff4479134e0fddf2e5a50af5a30884f3aae6d41be8", "input_original_sha256": "8264f8a5938094e7f63e0fcb621da0271d0a78c803a29ae61e29807155416c56", "input_wav_sha256": "9a1d558820327c303695fb65af6f3d88f5540287b6545e20372a4f994667644d", "input_pcm_sha256": "c40e1223cf7935b0ab219fd4865610d6963319ab232b37d9da4b4d236a53bb7e", "output_wav_sha256": "e55e83ab6c00f17e7403759e02726ca380c0229621de0bfddd52f460356c27dd", "output_pcm_sha256": "c36565c80c9f2d22b406ff4da70049c2cb3525587a70a7934135b638c37bf3f9", "result_sha256": "6ce3d6da4298a890c965ce85b80a307596f6c9e0eed68723de4b39f68850dc9f" }
#716object_counting heldout	11	The answer is 11.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "d680f81aa19528245f8a70936112207a1c7fbc567d7b5874e133b6eafcf42b95", "started_sha256": "8a9580bef72ed7a7a35f6678d087dd955651326ccd51ecf9fe1d84b71928c83c", "original_grade_sha256": "34e378ac65daa644d8832f8ef2479172f6080026e3adb73beed25ff1c5507fa2", "input_original_sha256": "d680f81aa19528245f8a70936112207a1c7fbc567d7b5874e133b6eafcf42b95", "input_wav_sha256": "e5514d1f354d9933330b003e3fb6594c33af616d88a8716b24f172b247e3da72", "input_pcm_sha256": "9bdddfe00742df6b2522dd3ffe68ce9467541423d62a173ba88a818cbd30efa9", "output_wav_sha256": "9a66fe3293b9bc434532b0ee32acb7bdaaf691a3e6a20650fefa6b7512fabca6", "output_pcm_sha256": "e94bf88bb3a49b94c78ae3f3c0b5f795bffb271c1eb9e2865d2abe4c7e550df2", "result_sha256": "039c9034a79d521b11a0bb663495727e919994b98d25d065afd71cfe2fb68ecd" }
#821web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "6bc1feb6605c34e48f3cfd429488c6a053270ffdab747f6a3c9ab2bbef0e2969", "started_sha256": "bade4c9fd2753f6483e92793d8ddee599e8558b888da8e58d1a253e682b1f2eb", "original_grade_sha256": "34d1b0e57d1bbc282f1ccfe661ba47a539212a675f1391efa18bf0d54e801cb7", "input_original_sha256": "6bc1feb6605c34e48f3cfd429488c6a053270ffdab747f6a3c9ab2bbef0e2969", "input_wav_sha256": "78fa76f8624b9a2788500ecc2ce917f60ffd131519443d35527b0957f39cb7de", "input_pcm_sha256": "a53140659828b096b866f11fe82773e528a97c6d314fd2fb414f958da89d15c0", "output_wav_sha256": "5293471ab36eb7535a04cb6877f679008b9be05ca24f7452f896fbf6c729d6d1", "output_pcm_sha256": "bce5eaa097239fbb87025f6cbfef35ed3268c31f4cf687ffec602e3e46d21694", "result_sha256": "7a4f0adee57f838cf4f4ee9ee26a88a61df5d4998da90912c1cc622daeffec1f" }
#13formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "8beb8afbde0a0a66124dd9d143ccd82eded711d9525889d0b949aaa77787e7ca", "started_sha256": "20cfd8d6482cc028f4fce3799e146884fff84b28988da6127a256dc7d873533e", "original_grade_sha256": "a20bb6f57754ace06c13382607e571e06440613873e0bad982100333153a3497", "input_original_sha256": "8beb8afbde0a0a66124dd9d143ccd82eded711d9525889d0b949aaa77787e7ca", "input_wav_sha256": "eff73b22b36404d2b64b2f9655083a6e5e9934b33525f38e1d7a1ab55f2fe7cd", "input_pcm_sha256": "afcd6a010827c6f40759e981fb6b46b351291a92aa3179c867fcdfcc6b3a71d8", "output_wav_sha256": "305b46b5ed322ca80dd527917dbbb9e97a1239b9aa89e64bf62865757e8765e2", "output_pcm_sha256": "aa2172870a434e9bd14524f51e00a9efd66634bc7474903f1219904a1f11b5a1", "result_sha256": "9d812743862308c179343ba10fb59dfba9eae4a0d1db33ff6b2f7b3ded1d66d7" }
#260navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "1bd8f2a3e8efaae198cd915104ad27f752c74efc0da7fd2dd8b1053b41c49aa2", "started_sha256": "1e3e86a32efb84fe0e093c3fa2430b85cb5bb2176792dbc19fed01226b5167a9", "original_grade_sha256": "c9f1b9eeca828ffa641adfdac11c67d6fc46d56775397431f79e7ef57d46ff3a", "input_original_sha256": "1bd8f2a3e8efaae198cd915104ad27f752c74efc0da7fd2dd8b1053b41c49aa2", "input_wav_sha256": "9298a4ed2e606aba826003ae02bb872091a74089fda8a5ae62a1c4c764c94325", "input_pcm_sha256": "9e0fcede7a556636411e72324273894c473444d57ccaa478e9cfa67a7c05b59b", "output_wav_sha256": "5529ca0a51d7b95540aa9f0f17392d01f6af97ae92033ce895744eeae81bdd74", "output_pcm_sha256": "672521a90f6ffe4581ca7be67c2aa78b11c408416d4054523f15d734916efc93", "result_sha256": "7848719f41f9bc0cd04cbad158af6779c17ff160ab8c69b278702baf7f26135d" }
#612object_counting heldout	12	The answer is 12.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "46d836921ae51512d3325e5cfbfa2ce71cb47d619253cc8fb83fd50af35ac1a0", "started_sha256": "31eae2f8bb19548c566976714a14f35de72ab0eb1624f9a0c820e94ce671a3a7", "original_grade_sha256": "d195baef84e2668a1450c631df4b66cdf02b6eae4b4bbb2f739e043e225c330d", "input_original_sha256": "46d836921ae51512d3325e5cfbfa2ce71cb47d619253cc8fb83fd50af35ac1a0", "input_wav_sha256": "683c07372c56e38d6f18f0116f7622254baeba484883e2ab90b757948f294e63", "input_pcm_sha256": "5a3c5ea3e7e828abac90e090bcb297f4dec7e92c02e19bf0b175018011b4cb5c", "output_wav_sha256": "fc9d67b4217093b2844777987f9f6de0406df405c73eafdbc644a60c28397a58", "output_pcm_sha256": "1af1e84117ddd78fc89505edb7f29db96429898ab73771d9ad8503c03bf4bf82", "result_sha256": "855fb39050b6187e5318c5674bc9e1e01b1b71c03585277e2fb1aa834e624e3d" }
#991web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "16fe649766ad3841efb94cb31845bc8ead091b8d494fe7f029b404c10c8cd2bc", "started_sha256": "6ff8d2e3f46f9cd9152c34c68c553284c6f5c00d424b9f2b37b9ddd8da14cbcd", "original_grade_sha256": "8765d5ef287b7dd6e7aa265f3cb4ceeae298f16d68fa52a4c3268553d6101453", "input_original_sha256": "16fe649766ad3841efb94cb31845bc8ead091b8d494fe7f029b404c10c8cd2bc", "input_wav_sha256": "2e73fdd94df1d7f8437cf8b080cd9db7957583e486915f8e03272b080f9f8ff9", "input_pcm_sha256": "5d6aa1987d907fbc4144c0c7c51936a16f2db0c40d0c757c09d77b955142906e", "output_wav_sha256": "a1e5a97c7a5b071e2170aef085534ddcbcc9310bc5ed1468e9f073619b4465ab", "output_pcm_sha256": "b0e7392b6e9f06c3176106deff4199b1cebeea69eff37c931580ad7809b6c9d2", "result_sha256": "f0abf9936a8eeff7ab3fcc79550648db850e7cb48ae4908d36955b47a380d8ec" }
#3formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "8d0f3da662006352500b6a8ce8a483554144379994dc7447e16d986cf7aaabd1", "started_sha256": "e51da54af74ecaac1c83696a93c36a06d53ee7827cc7fb0e3e6e66d18b4cdc88", "original_grade_sha256": "ccd9614589caf51b45ebb90699ce51cd1aec7136bc55af5eefe4d85c21c2dfeb", "input_original_sha256": "8d0f3da662006352500b6a8ce8a483554144379994dc7447e16d986cf7aaabd1", "input_wav_sha256": "6b55737cff558890c855990509c406232e88d223c588e447f5367b38a82dfa2a", "input_pcm_sha256": "cfd8363808fd68a8ed254600548dbb068c8bb1b2b2b4e20e76609091bb55833b", "output_wav_sha256": "a76efbfa660289a5c3a837127187759523d6dcde36b1cb014856cb16d61bb114", "output_pcm_sha256": "a1b9912152c96f5f015958763befab26623ed14a2f13e56dd31736604e6862e4", "result_sha256": "288e4b1c732fdb24160ff1f52e05c35798fafcf6ad8a934e19b356e172d2c34f" }
#251navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "1b992ce63e5391b83fcde6541b516ac6632f8198aeb2e50147f20d129cfaaa3c", "started_sha256": "08908be266dc6a348b979d4a445bc98a20383ba7f4e5f6b9afc6ea89a7c7d357", "original_grade_sha256": "b4a12fd865f720aa427d4c73a2c3648bdb22348d3106a22e5d4d5d29a0937ff4", "input_original_sha256": "1b992ce63e5391b83fcde6541b516ac6632f8198aeb2e50147f20d129cfaaa3c", "input_wav_sha256": "7fedbdcbc21b0c1a729c498975a5225aa6ab5fad5f31d77ee3162ed2a6f247fb", "input_pcm_sha256": "92d685d0a043add6a1c7374428be15537c8b3a54b24ed44b5b1e71d772c0fc3e", "output_wav_sha256": "ed7c7df40d9ed930d7c0b6c9d11187424e8ddb7abf3f6eeb43ec422381a57b7a", "output_pcm_sha256": "f904a5cd64fc3cc953aab36528cfc768bca5fa1c9368b75dbbe2ff08ba1edae6", "result_sha256": "d8549aa4bf3301b022ac47a9c419c4f7fe1615d4e74a89ba45dc38c151c00379" }
#602object_counting heldout	7	The answer is 7.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "e6028327a934914f3e4583763c37906de7b8e58ba23a022e55098d422d901884", "started_sha256": "ac16b41c4146db3ab955739897d2c72ae66e7bb410686fea5dfe436c77e37b31", "original_grade_sha256": "6d0a322af90cbc59d88fd68d55d9fdc9d761369e30a4f7a8fb2ed5e83fc93bf7", "input_original_sha256": "e6028327a934914f3e4583763c37906de7b8e58ba23a022e55098d422d901884", "input_wav_sha256": "123bb78ba8f093e33c5dbe0ac162125a589963b2f156efd95ecee2bcdf81b04a", "input_pcm_sha256": "7242e0199a999c672469db8c10e53b730935347f3d37c9260ada71f1f837bd87", "output_wav_sha256": "24a878854a84aa8fdd164f9ca5fcdcd54f94f431d56f4ed64cbb400b7fd58edd", "output_pcm_sha256": "6d45bf074d498c4c1ebb46f02a17d4914293482725ed07a865e9630f42ea7029", "result_sha256": "73dad7c9e89de22940d537c9eb8444c06fc6c9c360c91011c773f018dcc93066" }
#786web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "ae627cd5c92ea9ff63caa740869cabbb1f6b405d0c94f42398b07e43dedfd53e", "started_sha256": "477a84d3bffeffbc32354fcc1db3e6620e91a43cb63155a5adb3055b487232c7", "original_grade_sha256": "44b52fb7d3e4581783110f1a29e7282f66c1c2469e940c66a34f8d181b665403", "input_original_sha256": "ae627cd5c92ea9ff63caa740869cabbb1f6b405d0c94f42398b07e43dedfd53e", "input_wav_sha256": "b14255152d7489e3bac565de7dd342b33e3ef7e5d2c73f00a0d392d54895ca43", "input_pcm_sha256": "f3e6d64faeff6101df2f0a537dad4ae75e5c67f46fa0ef2d393fc9f0d3cb9cbb", "output_wav_sha256": "8b196406eeee3843598f311f91a3b5dceb2ca25d15add9d40046b9d4703c5171", "output_pcm_sha256": "030e7b5b3a8ce5851899b9a0f7efc639f7ac404320f6fb1edba55d38c101fec4", "result_sha256": "f214d634ea4fdaa12e024d31f4279974905445a47cdad7c600931e94bd5f7850" }
#200formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "54a6e2b6e39c49cdd4d66b69c59adc458b7badf796faaceb55307caf2a1d14cf", "started_sha256": "77299b0c10a1431fb67f9cf3ac12e95be798780509923a7556e3e9cb1227e2e7", "original_grade_sha256": "4910ca247b7be680c2a2406918f4d2d24b5c8287c9bf8ae812d1ad4d34718e6f", "input_original_sha256": "54a6e2b6e39c49cdd4d66b69c59adc458b7badf796faaceb55307caf2a1d14cf", "input_wav_sha256": "975bffe87055bad89a1f4a9133ab46a0c7c12070030ee2b6dc3b0da46db61105", "input_pcm_sha256": "e2c99efb748fa4db28fb99a70c74d372c6fa7eac00b462a6eff8c35388dfe266", "output_wav_sha256": "ca1de04f62cbe309ca5df4a1c915566cd01d39c22b4e25e102bc05024de59894", "output_pcm_sha256": "1f9f0723d5070dee3597eeb1eb8c6e6207b0fbb85f67923f796468151dbef239", "result_sha256": "af51d5f07ca1eb51ea00c58fef3ac5c8b3c9ecd6e67e6d633f8a853cc31caefd" }
#456navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "57c14053114db11e60ff488965a19e87d3af31ac99dbd392d0d1134223d76d3e", "started_sha256": "4c864d3705785771be0d29607ad0c34965e0d94bda9e36f67b424c657545c593", "original_grade_sha256": "f2a53985f2ac9e9373f4b4dfb183798b568e3499ab7b45662268822843296af8", "input_original_sha256": "57c14053114db11e60ff488965a19e87d3af31ac99dbd392d0d1134223d76d3e", "input_wav_sha256": "a5fad4f507388cffb1c203956476b83070c9777854700c112f31ab53a6b0e857", "input_pcm_sha256": "d4fa3b711edb82b72eaf520a39b5396bd03a04213cb6b385ca04f4a8bcc7ec8b", "output_wav_sha256": "07258b2b51c2f531a4f4919936d114c79c888c6a3d021125c630247171fc3d29", "output_pcm_sha256": "e3ddfad2ed593293c89d8ba616189dc3516c42bed46c5b54363816f5b574d5e1", "result_sha256": "98aeb5fc7b853e165e914f017a39b46c9657ce3eefa282851b9271cfe060ed9e" }
#653object_counting heldout	9	The answer is 9.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f972b67c6bca61175163a61b42ebb02cdd20ba2ed8265492c0f6a0378aee142e", "started_sha256": "fff7516758713cb5c4d785a5f34bf79fb1f7f1162ac0099e6194fbfea23d35b7", "original_grade_sha256": "0f02e307a195ea969141f82cd8142360f14bf607b142946ca8cd699a914be2a0", "input_original_sha256": "f972b67c6bca61175163a61b42ebb02cdd20ba2ed8265492c0f6a0378aee142e", "input_wav_sha256": "7be8dab12c512572edf330774ed7e602618fd1964025a7a7c20f0caa7b98216f", "input_pcm_sha256": "4ccb0d005182acdadc2e10e8486c87011f70609aa6a03f69834b2fc5673f0b90", "output_wav_sha256": "be8fc3f11fe4eab455a3db3c5d840915b584a0d9af6356c56a246944c3aaa8eb", "output_pcm_sha256": "25490d5e08f2963a136315ac40c410e0cddbfa6562ae49b8d063d22c84c793ce", "result_sha256": "58f84a72836f9593c04663a820944ab8a89c126ad1f339adcdc1cc8c599663cc" }
#848web_of_lies heldout	No	That answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b945632686cac4726a001d80d5826cee832b9551f351a04c95446aaec3c5a46a", "started_sha256": "5b46ceb4e28baac3c13b561f3af482c7372613d82d266781b77b4c7bac2c75cd", "original_grade_sha256": "54192aaa675b675f823376c35aef68d31fd2802a6547c7b50afd4a136479b941", "input_original_sha256": "b945632686cac4726a001d80d5826cee832b9551f351a04c95446aaec3c5a46a", "input_wav_sha256": "74fac16cd73ef6c29a2fabebdcb9249dd0cb14552afb2752cd995a286d9e7a2e", "input_pcm_sha256": "2ba20feced0c3de1b9d903dc8fc8c092b43c9607aa135cf39f771ec2e5d67219", "output_wav_sha256": "5e4596610ec1f5b13b09aba8dc08e99bdc7568503446abc7ba0ecf8d3d187e28", "output_pcm_sha256": "8467c249c596fd2cc5681af1c52e29c20354d927dece8e1669ff373da3a3624f", "result_sha256": "b66248386eb33bc2034e985e1a1503fd78a71f1f99c2f40c259dd5cb2448a584" }
#11formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "0ead0137a6f40cf60817afe9c5813d8fbbb1a0686b1f02f7cbdd15667c1afd87", "started_sha256": "87490c1501c9a25a29ef21916db238b34c5adacfed4b8d51a3104004a5e343c5", "original_grade_sha256": "7ce2ac394ec39d4e741cbfb94ff40abef0b8b1e29e8f0508b29b2947b17fab09", "input_original_sha256": "0ead0137a6f40cf60817afe9c5813d8fbbb1a0686b1f02f7cbdd15667c1afd87", "input_wav_sha256": "aca31bec58f6d592685045512343b192f81b1f872d782fe07e4b636241367c18", "input_pcm_sha256": "44aee29de66e8ded06b3672fe61646a6f5d5fa46264f088c50d15855ba76945d", "output_wav_sha256": "b9a95d19648cd6e19bc5a15a69c847dc22c3e6032cb02989b33e59fc17da9ab0", "output_pcm_sha256": "cfadf0c73ebfa6d64c9ad92baa7a204c0e9bba79295aab677e8e2cfd5f730680", "result_sha256": "244b71f9db94e0bd0672a26410286b24dce7e8c19f6887e1cb75a41c0bba31f0" }
#388navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "aabd44ab4688620cb507da4c312ad33f30ab40491595fcc638d992c8e6662358", "started_sha256": "bf54e989629b03ecd2dc0145ceefe12ff2c04a76eba6c880bfa173936102db3e", "original_grade_sha256": "98c70f586d3a3f7457f53f290c2e42d42d0f20f9e22a69ae397110efce8c3943", "input_original_sha256": "aabd44ab4688620cb507da4c312ad33f30ab40491595fcc638d992c8e6662358", "input_wav_sha256": "50140946087eb6660d8f3ed6e9e519e93719b8eee437f1caeac5abe1ac299b24", "input_pcm_sha256": "b94fd38902f22381525ae8932525e7edb623a30aced4bce0d1366ac592f92c31", "output_wav_sha256": "2894a5ffc324c3485f03461b0a265fb05079468a13c7af54846850d24fe3a1e3", "output_pcm_sha256": "5a3f6edc1671239755a0e750c14feed2a4030e028b945cf6fe9f5926e91bbceb", "result_sha256": "900d643d8dd955db8a5b6740b1db6e0d129b6cf65086e58bd944eac6bf38dcc5" }
#534object_counting heldout	2	The answer is 2.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "96bf4317876b0b594dceb95713c01b5a5d3fb165ac85ce68760eb8e8b7f6074f", "started_sha256": "86d0272190c07af419183dd701991528a66452191fb35fc30e692d128868d285", "original_grade_sha256": "0d82be7a89503b0972cd972f4c1e0491656f1163a0a4d5bd63c4a86ad00be848", "input_original_sha256": "96bf4317876b0b594dceb95713c01b5a5d3fb165ac85ce68760eb8e8b7f6074f", "input_wav_sha256": "9f784a273ae6c73e1d475fd446a9038140d3a6674ef5e05fb16dea4f13b95700", "input_pcm_sha256": "4ed23999182924570866deb1412abc1c03a91fa4c46dd0b5d0cda97e7f104ae6", "output_wav_sha256": "a975091de77873e15063d1da086ab77f40e1456e8a8df2c8c370131f69387359", "output_pcm_sha256": "8bb322df526347515f6cd6fb3a77fae1dc69e74e7f52a7050b37146e02a94b2d", "result_sha256": "767b1a3062396f37e636bb0a100b0ef22e8fa213681993fdf977f9edd4bd0f12" }
#763web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "7416398261a8fdc49df7da534b833a7cc4b8cdf3bb23c2075411d351c9f7d4c4", "started_sha256": "b060a45691fe0f2ec6d24ccf063af07edc07dfd284c57b18091db0631d106ac6", "original_grade_sha256": "e22233aea497b51b101c12a055ab8730dfea139e3ed9a8e9957b3fa6fce41a39", "input_original_sha256": "7416398261a8fdc49df7da534b833a7cc4b8cdf3bb23c2075411d351c9f7d4c4", "input_wav_sha256": "3c8ff5fae0b188c872854ed6b88ec66182303666788d765517083a486964155f", "input_pcm_sha256": "d96935cdc8d308f8da4fe218ad0bc90a44f181b4624b310e00419a9cc660cd68", "output_wav_sha256": "4a7a887a74f5404ffb6e88d296ed27c66c014c86eb1a3b5ab15b8e22530522cd", "output_pcm_sha256": "8c70a99a4c1064b5e6104a618fd828e4ac9dd49d8ed500a468dcf331cef921d1", "result_sha256": "7fe38f47aa52c1620d91c687c66459475b24badcbfd71243395a886596479533" }
#212formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "91928e179d869c2fdd3211a22d2c45b0e53c7afee0bd60d8478a8a3d7929e1f2", "started_sha256": "e35b64e819267f6adc042497cbfd93203c6b60e454f90dbf34b897758780d771", "original_grade_sha256": "6ec15c948ad85d0527ff01164f2982eaa867b79aa025349d0a23da9989aa8c0f", "input_original_sha256": "91928e179d869c2fdd3211a22d2c45b0e53c7afee0bd60d8478a8a3d7929e1f2", "input_wav_sha256": "86f5a28ae0fb77d7e733c022c59008912e323d729ff787d62bcae6060b73a5d0", "input_pcm_sha256": "beab1c77e573c379a069c15163a43a18dbaf408954d08989f99db21d6344ee59", "output_wav_sha256": "141a8b3431cc89e340069bec31ba739b1ba4916e64e0e0431462f314ba51b90a", "output_pcm_sha256": "673506459bf61451b5780dba81867e39aeb92ed773aa4e4f502f90d10c703ead", "result_sha256": "6a5d25bd1db6dafcd5713222344db85a205f8a6c4eff6af64b96c5d95c32383d" }
#300navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "5ef2c83255bae797603751b044bf8d95662599f3f206c587535c1a86d2ee50a8", "started_sha256": "2dca84233189cec9798cd7a447395b82dacf6f4a434872454cb9750fcaa72423", "original_grade_sha256": "9d6b59e1d2be30ef3bd578d530189eb5791b8401c1aa55219c4656e43c318e20", "input_original_sha256": "5ef2c83255bae797603751b044bf8d95662599f3f206c587535c1a86d2ee50a8", "input_wav_sha256": "d180b3137470a4fe1b82767f55c9caebaf6106eaeb36a105d21dbf98d66c4521", "input_pcm_sha256": "1d4a257c004ab13e0406b81bfcb83368080c1a6d995fb7e88de301f37a8808b4", "output_wav_sha256": "6f47bd08393e2345f31a3fc3edf265956e856c652e7a8f8c60660d03f284c04b", "output_pcm_sha256": "1dff08017bde36aae682ac911b7e63b9f824f0a306f22f88a58787ba8b6cc04b", "result_sha256": "cad8d25a6b8c2d87dadc0e3bb94911450725bb06cbcadc96601f8d9a1f24567b" }
#717object_counting heldout	3	The answer is 3.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "49201aab0b3a4de922a7e3c446a02b023aa3ca8227ab88c53c924e1d2797a2f9", "started_sha256": "41a06ba6b124348d52cc5549f82322f78e0474054f8734f392711c3def72510b", "original_grade_sha256": "2d0440ed65d4b996eda66b3ac6a967b8c47baca058b228d01df57435504299ec", "input_original_sha256": "49201aab0b3a4de922a7e3c446a02b023aa3ca8227ab88c53c924e1d2797a2f9", "input_wav_sha256": "cbd3813d761e3d86e9ac19656c286f7ca99ad2fc74285b6c3e65a26b103383f9", "input_pcm_sha256": "2d445cf2051c9b6fa93a176e5a6d681278159deaed83c58e1c4561a1373b83cc", "output_wav_sha256": "a6e4568e4e3b0bcf46386b50030b62627d9cb1a15cd7fd964a0c4bc5caea50bd", "output_pcm_sha256": "7f597f916623ea73b6ed49ab550e6f233fb17c083c025229cb2481d542bc12d9", "result_sha256": "c2441871b94632f1f3a9da6b689e3d4358358a3ec02e3b7d83babd7206e9a09d" }
#768web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "bcaa08ea0a2d37b94b55e7012a852b76d0382bc5e18fdec167051cb9a0d6390f", "started_sha256": "877b3541da215fa66ef945fd75fe004aaf3892b264c150cc31fe227b40e71e57", "original_grade_sha256": "e6ebc5d71f560ebe7cf52fc33f8d0b56fa989d86057edb7d31b3ac091a561cf4", "input_original_sha256": "bcaa08ea0a2d37b94b55e7012a852b76d0382bc5e18fdec167051cb9a0d6390f", "input_wav_sha256": "f0ae75d2074c3dd0ea3f3a545b79a9721de167468ddcc62c9ab06f1074994bf0", "input_pcm_sha256": "f87d1a1a055cf140da5b731e7082250bef35b9509c9739993ecf297ab818cd14", "output_wav_sha256": "b8dfa6306dd76469ea175e7332299dbd52e6aefa3c1ac677aa081c2671ae48a5", "output_pcm_sha256": "34bd821a4f1bb37bb3f5eab01cf1318fa6cbe0a7620efd5b28d3b13ec4df14ae", "result_sha256": "3404b5ba543da16e308a76ff7ef1ced60a59233b46ce807dab3e0e73f80be47c" }
#79formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "4bc8bd777d401e09ef213450d7d725bfb617150918a1b033d68faaf8e014d434", "started_sha256": "b4e6fcca5217386df18f1e0de1c51f5d72956878652467b7d6d6d2c201d4e22f", "original_grade_sha256": "3471dd1fc90b1d502e250977a83803e1a5b243d3bfe723dda18aba0c41b8015b", "input_original_sha256": "4bc8bd777d401e09ef213450d7d725bfb617150918a1b033d68faaf8e014d434", "input_wav_sha256": "c3c06aaab8cb96f4a07a099e9457dc73a5eb4371e782b9515e22213e4f2a0e21", "input_pcm_sha256": "35b7a4d8afdd9afef059c10cc07b94718df2d44136a4fd6866340626b3589023", "output_wav_sha256": "0f0953925349d4114d65c5e3b1959d3749442fe5ef5dc8b3b26ab8f3d4b08cb1", "output_pcm_sha256": "a7cebacc5b3ee6ae2824fa3aab500cfcdbe89e99d9263750e2951044f5c242c5", "result_sha256": "b5c58dfcfac1ad3324e93368c33ccc1b0d20b87b68451fb2a9975f31a98d6dd6" }
#310navigate heldout	No	That answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "ee82b2f155a985795021dd1532cd9e37c2743ab4c06ff18aa2a66708491d331e", "started_sha256": "c53e0c257f4a36cae9461e2031edd2c92009daea6d9c6aa2b1e6d1b963fda44e", "original_grade_sha256": "a3d57bed3f0d592906ba8465a82bbecfb7ed1b5170ed47c8192ec5ec0efb0f6b", "input_original_sha256": "ee82b2f155a985795021dd1532cd9e37c2743ab4c06ff18aa2a66708491d331e", "input_wav_sha256": "55415420a03ea5c1c71340725f34de8560bd9c00d154252d1740cb6ff4999c2a", "input_pcm_sha256": "4851a703bd481b581525f4547a96b47cd4bda7ae89b419c58c02bc5ef73060e1", "output_wav_sha256": "2477820349323ea231578967c3ee4a2dada3607687e4f75af4dbf8602663cd88", "output_pcm_sha256": "2d13a1dd35b369df57feccfb8da70dbb7a29c746b20c824969994193b33efcef", "result_sha256": "efc90f54bda93ed13d6618d20de090092b6a3ad53c289b75f6a080e2cff126d8" }
#714object_counting heldout	10	The answer is 10.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "218518511861be17054aee81dda78877a58cc0a2b8111bb0ae95440c2cfe82c2", "started_sha256": "d3aef7d5595e2bd885f5a97fa79b8e4c8a1161de1a38684dc0cc17c13ed542fb", "original_grade_sha256": "c85a9cdac29af25cb1cacf9b00cb0ae31576eb213ab8681b984441a80cb6a34d", "input_original_sha256": "218518511861be17054aee81dda78877a58cc0a2b8111bb0ae95440c2cfe82c2", "input_wav_sha256": "60d5e7b3076cf00c7a90e048063ab0cd9268ab49af173d9dc1e2cff52e828d2c", "input_pcm_sha256": "d8fae55734f906e1391139a7e7815d5261e76b72a40411a996ad00f1a48f3380", "output_wav_sha256": "c796f2ccfd8b0c9a885b7b9bd7b93e05f15500a676b9db568560242c850e7c86", "output_pcm_sha256": "f90618e7f3827054f846c87616462554a1ddd2fcf931a4c583c19d6f043218ef", "result_sha256": "3b8c9b9ee8ea282039c712189261aa92be8283274d9bd97a36281ec196473d1e" }
#833web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f961aa04f266391280df8aebe72469d014646b8b220438120bb2bba0fee6c330", "started_sha256": "98dec8b6885e6dce07c9c7d34834700a3fee7d1ba2024fdbf4997eba7ad76b27", "original_grade_sha256": "f91a25f81c4153300ad20f5f7f96b99cb46713712558da92337e9cb6c4b97ba3", "input_original_sha256": "f961aa04f266391280df8aebe72469d014646b8b220438120bb2bba0fee6c330", "input_wav_sha256": "fdbfe5685fc3444b89e510c5caef394d8aac8c2c987b1bf9757fab9a7f96833c", "input_pcm_sha256": "bd3892cb183ff7ba920c4ba4f3bb080ebf7c89e8a59e9bf4d3776dc95765bd65", "output_wav_sha256": "a704f68bafb081e17490a560d54d90b5ed3d058fad5ef6dad3dab43d1b6b86eb", "output_pcm_sha256": "6f7eaa747e49cd8b865eed010f055a4a9e2a86d3a67d7ce7ad4a53acdf564bbe", "result_sha256": "a1849f134dca605b68b2c1dd74a27510e50e8d6e07b35ecdee771f8c15ccca96" }
#73formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "17aa3fb88079b4cda2cc85fca5a72aff5c794b4270c386ac9ad06d305196b0d6", "started_sha256": "9ca5231134d2f1bee384ea8cd8c2a3655a01abf039642bbbdae02fb85010e8f6", "original_grade_sha256": "dd54395319a15a8d24ba2b9ea2b621dee7c6b6e4e5508e8202e7c769f93f612d", "input_original_sha256": "17aa3fb88079b4cda2cc85fca5a72aff5c794b4270c386ac9ad06d305196b0d6", "input_wav_sha256": "bfddbb050292bd1472e91108283a448c4d8e5297798f5d3fbc4cebf957778a9b", "input_pcm_sha256": "02792677534f150c98db063615363b44447643f74c135bf97cf973b6568d1d70", "output_wav_sha256": "10bbae976ac204428a865659759cd5fcc08ec414c6bba3df296b839f385c7134", "output_pcm_sha256": "8087895719ccb7402dba3c6bc1483f0e336d323117b365e63be87169b3079102", "result_sha256": "e933aaf463ea1926ab9bda671858a6bd5605fdb634a8d30f0df942ee4fa6fb17" }
#372navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "529793b65fc26ee4ba9cf13d8d60b8b69d622336a14cf07e76a4060fd3448cca", "started_sha256": "ff0fd6f7b8bc83cdf3cea3db3ac007891f43f5b55d03fa58ec21629b54e59e15", "original_grade_sha256": "ffc801645ce96e2d3038e312522d9b9bb2817ada4fdc02d5df9725e2313ca4c5", "input_original_sha256": "529793b65fc26ee4ba9cf13d8d60b8b69d622336a14cf07e76a4060fd3448cca", "input_wav_sha256": "5f4e9e08f6aa6355e5a934467b6f91b31efd81a5e0b73bcbd8da09077f00a4f7", "input_pcm_sha256": "1fd34982f2dd32d1d5d4d4a17120bee0d95a29699ee48bdc33468d87c19d472e", "output_wav_sha256": "ae88186b601b00395df8a1e92c470529ac0163ec41c6bdcc420f6b17e191a4c6", "output_pcm_sha256": "5b322b4f16a99d89a96a157b2b34d45b811940435dc093a97a7144553e9fd032", "result_sha256": "b4bd453a6c5dae38e633b11cfd8a280fe549cf3227ecd4ab9440690f9a420d5a" }
#660object_counting heldout	6	The answer is 6.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "dbd23b1f0a7c016a67e28c88673467af788578f6390a944b4844c04d5ad24877", "started_sha256": "9ab391a66eb7f08f61536aabb7943a7d5d371ed3ba1fb9cf94505e4a60f89cdb", "original_grade_sha256": "d406d8d06efc9beff36142466a4fdf6d720458e6fb9c69866b42e4666b2da441", "input_original_sha256": "dbd23b1f0a7c016a67e28c88673467af788578f6390a944b4844c04d5ad24877", "input_wav_sha256": "529b1fd901fee12eef7b87b92bd406f90c3b01e5e85258eabca421372f3360dc", "input_pcm_sha256": "7ae93e836b45fe93aabe203b6809dc233b9a7095d9c5c8e7f1853a89e65f6bb6", "output_wav_sha256": "b49dc2b764ca5c6dd6dab216496abd41f08f59f37facfa3e0fadc9346438c00b", "output_pcm_sha256": "5701712c7f2d3ba5471dfa363da7a476952d9d4127e7f5b7adf3c10a628fc70c", "result_sha256": "d2a07fac042ccac81ad99cb66c3613564aa8d70c2164c89da60e0e9f0ed6ed8c" }
#836web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "a04b4631432d26c0c5275434d227917a812ecc01e21cf0e996392e7168c046fb", "started_sha256": "4d7cc1ff2ac0f2f19ca318af2e248370c281cd1ce70cbd74957600207a0498e1", "original_grade_sha256": "8101b80006fe87fe262c73cf80811b57f3e2905857ede00a6491ebef4dcea8e3", "input_original_sha256": "a04b4631432d26c0c5275434d227917a812ecc01e21cf0e996392e7168c046fb", "input_wav_sha256": "19b2dd2365097ed645dca30739daae6f477e207adb254c0beb95588f9c6c540d", "input_pcm_sha256": "65d127a2f8fb852455466dbad43305dd615332e6d896dd4214a9ea6add5da9af", "output_wav_sha256": "ecf0249bbde2bfccff2afaf10a8b59958fe1618950f0cff2f050becb3f2dd7a9", "output_pcm_sha256": "5101ec31dc4eeb9daea72b20eb643780ddfbda8951570150e978aa48720b5ca8", "result_sha256": "399d88b2059c4a1b5e89f1e2a0cae448978d3954b9a9b341428ada3f708fd23d" }
#147formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "4e0556c0bfc5175d0e799a81d590628287f5973590ff0540ca0186018b4f5303", "started_sha256": "845906a908e072631176424639d84ab9c3e6fd760c743441069d6487bd11d846", "original_grade_sha256": "3f2f17274092d68d6f9d5baa97d27cce12deb9fa87191dc90e9707927d9d3ef8", "input_original_sha256": "4e0556c0bfc5175d0e799a81d590628287f5973590ff0540ca0186018b4f5303", "input_wav_sha256": "10d12cca92fde81568544e603068adc81491880407d10cbfbcc971874959952e", "input_pcm_sha256": "b9d715a6f9de74b036b726a8a04e992628729d9584f809b577f61ed1e80771c4", "output_wav_sha256": "168305bb9b8f23b76b48548803f99a7a2f76718f616ca0917bdd597fb8480e4b", "output_pcm_sha256": "9024f3d7458c3b8dacd75787bd28137c2e782077f8df2c6c92517e66c9ef305f", "result_sha256": "ddad5f998c438662224101b0c8522d9d2cb37ab07d80ef34557fad103ee779db" }
#293navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b395d2ad2c4a1c550cea3f60b32557e5af964cad799c57678ef7bf54d6f31706", "started_sha256": "10cd02432a751151e46516696f2551544b5729efef6149f8d107c3107ef5b4f5", "original_grade_sha256": "aeb446e3d6f2ea22ba5da50cd0668855a2e89d327d365cef3c65783ca8742baf", "input_original_sha256": "b395d2ad2c4a1c550cea3f60b32557e5af964cad799c57678ef7bf54d6f31706", "input_wav_sha256": "8b085c9cdd65492b1e6b9a0ee1d76b2689cbb7103c67cfe1880a55789150a122", "input_pcm_sha256": "b4f0a9a5f2bd3c6f7ba88b6b62d5399bf35d1427be83aa343fbef50301b99fa9", "output_wav_sha256": "d813409d92a0c58a62af00db0553542e666f29ec9a3562c4da75f7cc15ec757a", "output_pcm_sha256": "c4ba2c502034dd086aeca3f0bccb66cd69b409b0d9ad08e1caded9cade98701d", "result_sha256": "2a0f04bd7068bdb60475d6d033f4a781645fb307648dc92eb344e63aa4d1feed" }
#598object_counting heldout	2	The answer is 2.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "163c1e5016131afdc161b9b68820406d79a9ddbb314bf7fba853354aa14e51e0", "started_sha256": "13d457127bc3ade2d2d31bf9ef242f35bb624caf3826a8453597cbb73918aa09", "original_grade_sha256": "542f217f93c509e76c5232e9364541df4b33f21b59f7f0256595afcd61457b2b", "input_original_sha256": "163c1e5016131afdc161b9b68820406d79a9ddbb314bf7fba853354aa14e51e0", "input_wav_sha256": "4b05903f1ee8387d0f9dca0e90052d0ee69eaa316ded2e95733425c898def194", "input_pcm_sha256": "6711608d5ae1e93df9980df8575e02f31bfab17b783d9b6dc8512b94d3656cb6", "output_wav_sha256": "7953bb9bacbbe4c3c0b2a4246d34fbe32e145d74ab9e17200af8259bdcdc45ac", "output_pcm_sha256": "140990c0e7ba8d1a0fa6872665861bfd2ade1bd6911182827630844f05dca7a1", "result_sha256": "55376470b3cf3cb8d8db29d934c4b577088c18c374fa56f720c124183b26c6cc" }
#996web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "8e55ef4236bd8b4ac5b18e90cb1e6c8bf9d01983b75673f739c12b3339e55265", "started_sha256": "082ba6b779089533d35204b9e0f0e2a441c9ab3af926ddb1a850ba61bd58377d", "original_grade_sha256": "713a5a6eec877de2da46fa878e971b25c5e4f3c256e6fbd88456bb430f1088c2", "input_original_sha256": "8e55ef4236bd8b4ac5b18e90cb1e6c8bf9d01983b75673f739c12b3339e55265", "input_wav_sha256": "57203035ca34b25fd6041bf0747fc97fa011cc110bf1000bf45f84baf207f0bb", "input_pcm_sha256": "a66bfc27c050b8d35174e937fc35728a457c3c0e2e43372df38e7a73dc693764", "output_wav_sha256": "9ab51838f34f0db3165f7e059ee8ecdbca4f9e46b38292db3e32467f521582f3", "output_pcm_sha256": "3ee614c5d1b3ae417dd3e936f0a48292d2e15e6767ce6983c6236dcdd1346c88", "result_sha256": "8b10d286a96b4e7132864151a61bc8e5fe9a1eeb8fbf51313e05f076fa367a15" }
#80formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b196be3a64e746d0bb8e8bbdd79d44ccf0c83daa5c91c0f433aa0b9687d6316f", "started_sha256": "0d2571faac7963fed0781c68658515896a6a41774039d3e373baac2c2705d1d3", "original_grade_sha256": "b5b6c60a9c6d59d9f0af02ee5a13e7c31364038f594026d4d93de47a327a044d", "input_original_sha256": "b196be3a64e746d0bb8e8bbdd79d44ccf0c83daa5c91c0f433aa0b9687d6316f", "input_wav_sha256": "f3ba1088212e5bd5aa0af17f2c0ac53d5981af3ddd7a772e50df245d4fec1e83", "input_pcm_sha256": "ff6d7cd675272062157db8075591e710fe9368af3784ea699ed2b853741a999b", "output_wav_sha256": "b5ae3dda16a625776d9710ff5db014d3a1b9e675a700a2afc6053f8babea9ef4", "output_pcm_sha256": "3db2b61627fde72713b12010c5895ac6424ef66d3442ded99ebbbae6e5b94709", "result_sha256": "c7d44df5d73140f352709d9133ce55318acb6c8ccda83e130928c6b0880f42d0" }
#296navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "132afb05b94cb2dc90dde82494c274e56357fc2a0f90d78ad87208b531b87e54", "started_sha256": "2d3aab0138a599ffc895fef1c6799272e049b0a9672ee8e9dcd03a36dd33ddd8", "original_grade_sha256": "2dd985e595f0c1b57aae19415b22b2bbb09fe073498ba6d6117615af23201f78", "input_original_sha256": "132afb05b94cb2dc90dde82494c274e56357fc2a0f90d78ad87208b531b87e54", "input_wav_sha256": "84e45422df6043ced6460e8f202793f5575e90949b1275b220e17556d81f58c5", "input_pcm_sha256": "3f3ca7fc089a3b4a1acb95a7403e5541432e94f83d21bd533dd3bbb1ff0eb523", "output_wav_sha256": "9d3e55f10b28f09985964062a1be550dcfe3dd37b565c53f93bffb741436afd5", "output_pcm_sha256": "a779b3d2ec48ca5babe5540befb11dacf383867a730855d93e31dab98d9ef977", "result_sha256": "cf8ba7da826967e5d88d999be54fb942bc6652b94d72fa7d59c2f9509a203c76" }
#632object_counting heldout	16	The answer is 16.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "12dc2c5a7262d0cbd1d1af1c24797ba19ea04667a7d480550112531d88869511", "started_sha256": "29ee0dd6aa59bbe4c96b47ab73f772b4c6ca0502ede64cbcc4f3cf761c0f370e", "original_grade_sha256": "2a54c4a206b5f5b9af7f13663adb9c645c70c5f68740cf94a752cc67c9aaf27e", "input_original_sha256": "12dc2c5a7262d0cbd1d1af1c24797ba19ea04667a7d480550112531d88869511", "input_wav_sha256": "d97b36da58408ccf3ae92b90291b3dc2cd876cca66c82d21af984dc4050ff352", "input_pcm_sha256": "824a8455cdc5dd69e0a5ff04213a1774da0b656efbacf839bac49bce01f733e0", "output_wav_sha256": "0ccc587bbcbc4dc1514462fa19b08af1b13b4db16acea7d7242aba0d80de3441", "output_pcm_sha256": "1818270fddfd77e477ed4a326a94619e6d59a93e90627f28e5c5ea6b4741800b", "result_sha256": "e6f71c44ef5df1d31715875cb59fb90733024de49135ca652c61b0ed870f0eb2" }
#892web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "a49cbe255ab52928e048e6778c4bd6aa684a7da0690d2d98fe155fcc0c5395cd", "started_sha256": "a1fd38ab8bda65d7d6d6f50e3bced75a71ad2eec919f45fe2a59bf70f093957a", "original_grade_sha256": "994b4a15926cf5b562d6e0bf86b2debfb578c2a3733b21fcd68a7d6e47da2292", "input_original_sha256": "a49cbe255ab52928e048e6778c4bd6aa684a7da0690d2d98fe155fcc0c5395cd", "input_wav_sha256": "fd62f845dca7ec397dbe0bbd9b267026e5fd3bca695fe1f98b6660d4a76ab0a3", "input_pcm_sha256": "c8923df103bf364fb8c13c270e327957f55949f3624323269f16521adeec0442", "output_wav_sha256": "0906c26df58d846baf0fc03a4db9889e4c062fd7e00ea273e94db691a595f117", "output_pcm_sha256": "019b1306eb0cc82dfb87a5d570121a90b29b00e6ced4235d126b7541f29df52f", "result_sha256": "56fbf78d4b199983e6cbfd464228a0f4abcc759138ffa3a8673fad4878ebd3a4" }
#24formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "7d70ee181bf1d7bae5ecc3d6a215dc28efe18a263e00f96d53d7deb73076bcc2", "started_sha256": "067e910b6403f23bfdeb92d4f96cb537dcab170dc4ffee73377ee937c48101af", "original_grade_sha256": "170af7722463beec06fb373dbe0ea8c00ea950eaf63ad261afac7bb26140ecf8", "input_original_sha256": "7d70ee181bf1d7bae5ecc3d6a215dc28efe18a263e00f96d53d7deb73076bcc2", "input_wav_sha256": "d708bfc95095bb76eadff6aaad0407437cc31486a6f8f74b92f6fe3da897bd95", "input_pcm_sha256": "65c8e4da25258fe43f82cb4782b5b979ffb493c1756fe859c32f99ca199d2e04", "output_wav_sha256": "b5c1d9bd14cbdd305b476bd4b0f78050c91aa7683c603c420b9150aac532cd78", "output_pcm_sha256": "06bf7a4eee08d7fd7b2c04a8c6acf78e8552678f878565fcc074f5dd5ce81553", "result_sha256": "4035f718203973134e946ceea5a99d42c114834ba06cb1204cb77e813db89dec" }
#496navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "aebd66b5464a3ad78cfc14177f0aebe4b175fe0a53e15e4ce17cfbda677866cb", "started_sha256": "cfbb6f0979cf63611bb3e2032ff7fd34aa504f4c945f7813f9481d435f7df32b", "original_grade_sha256": "716ab90d67990e0ccb8e1f60e5eca3b80a6004e7857b63f1d2a2a84baf80dabf", "input_original_sha256": "aebd66b5464a3ad78cfc14177f0aebe4b175fe0a53e15e4ce17cfbda677866cb", "input_wav_sha256": "341cd1b5b7ddf9e3d17f29e2756abad1f247386c5e31fb909ea2f3faf006220a", "input_pcm_sha256": "991c8af986c1da0258812940d48461e2ecec22290e5b25502365308fe477b706", "output_wav_sha256": "eedee5aaf1a5973f2f3808f147669f89f4fb24420670a9b942fa2bc7e606340b", "output_pcm_sha256": "8a1e0f3dec64564c22f532972c582cf187a64f8d5a83fa32a93b738d362d6780", "result_sha256": "4a6cc68f56ad73daa2a796c0ef7ec2376f9193386381f4e14114f5ecfc0acb21" }
#568object_counting heldout	2	The answer is 2.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "9bea4d7a84472b55daca94bd6af58bdd3a04be15e8fadd252a1bd3e3b63ad728", "started_sha256": "f16501438bc68b3fa3ea951b732ab780b8abb144fb516996dfcc6a7ac9ed4130", "original_grade_sha256": "3fb70cec0bcaff7439f39aaae72e6f84b76075d377b975d7924c9ffb44d03bf6", "input_original_sha256": "9bea4d7a84472b55daca94bd6af58bdd3a04be15e8fadd252a1bd3e3b63ad728", "input_wav_sha256": "9d9f8c3c1ecdd00a048b144c57c684f703d9a5cdff1e7fb001c6971fddc3611d", "input_pcm_sha256": "fbecd0649d88625d3d2e8609b3dcbaf92b6cad2c7fa0d8bb2b93548db4327c4d", "output_wav_sha256": "be516a81f6199775aae03dc4ab7e812f42d2ba523cc9524e6c9aad3e02b04732", "output_pcm_sha256": "999185b9f8cd9194555dbfda89aa8e5ec1540f914807bd9e26279a2fa9a5f74c", "result_sha256": "6197c012378f608f13b00c1cbc2891e56abb9cf1c74813d76741d95562db8a25" }
#884web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "441632da7dbb62889296fa2e1d6d4d50b39c64815fa7acfb9f462490bdea479a", "started_sha256": "8737eb791b951165a0e11516e80f27282488827bebf0b4e8b5bedf32a177eb71", "original_grade_sha256": "87b1d4777de008400edfe83792819c1d1727166af60e6c7ea34198fff2a78bc6", "input_original_sha256": "441632da7dbb62889296fa2e1d6d4d50b39c64815fa7acfb9f462490bdea479a", "input_wav_sha256": "c6a0e0c81b683de1f676464e845cbac11a377df619fbf5c6b746730b27993417", "input_pcm_sha256": "bf4d01274b7e2233600f36274273ebe9c4844e72ade6eeb9f9836fcb043b44c5", "output_wav_sha256": "ab5dccc8d3924d5d382107632dc81079fae2f8779a283af8e5790bbbe1f3f03b", "output_pcm_sha256": "440b6b5b618f69c629f55f205508fdf8651a1d30ce9c7fb6dd525d9cc725980f", "result_sha256": "dad0a49505ef58c6a5ca1b1922b5c0cfb2850311336492fa77a70acd2eef6845" }
#34formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "1fe18fd2a7b9b84971b15eebcd5d53e8ad7c912c73dfc688d39c5663e6722c9d", "started_sha256": "f295ef1e53e712b0bd5c0aaaa3417bc4e5b602c590b40b9b46d191ce28442191", "original_grade_sha256": "9fa414fbabda1fef8e0974b53107275d0f474a759f9ffeee3b3fdb5977e4a627", "input_original_sha256": "1fe18fd2a7b9b84971b15eebcd5d53e8ad7c912c73dfc688d39c5663e6722c9d", "input_wav_sha256": "7a493ac9510c4c75ca9224e263eddb746ca9c277aa005a35ac99f42daa3ff1d4", "input_pcm_sha256": "b53bce75b8feeb10cdaafcf10ad906e6c96895188d45e62f6c1b8e4990c91dc2", "output_wav_sha256": "c17d8ae3c91c446ea777133625b00abd54932bae61d50863022685b34535a216", "output_pcm_sha256": "e911c42703fc6331c1c63bbcc16a18bd96a35bb9b533e5de393538e6be4cd7c0", "result_sha256": "d07401b90bed4a47035aafc95e7b9d6432fa6bf1bfea10efea002c1700789cf5" }
#341navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f2f70db264356f7366209386e1e9c2d58a2c07aaa189c8fcaec2bcf6eb085c64", "started_sha256": "73b5c620ec2a38dbe0ee3d239c3e51dff5d0160d57cf1cb3df2190179a96bd31", "original_grade_sha256": "51544d1833ce0348bce945d351809031d1a2863b79381d4b25d37d9f9dde59d7", "input_original_sha256": "f2f70db264356f7366209386e1e9c2d58a2c07aaa189c8fcaec2bcf6eb085c64", "input_wav_sha256": "8170edf7d0d82e34d8f09e6a30c823ae196acd2f7b14c6e8a000e1030a857049", "input_pcm_sha256": "66a538815b3267c16e5349244da46c4ba4de5aa9de3fa92b1bac58f666da404e", "output_wav_sha256": "943b1ba8870af9873d8297d9c3ea1965125f636d2b02b0e82ac7fb3dc6a3dc58", "output_pcm_sha256": "ebbaf2e05afff394f11e2a5a9ab443d2ba483ea4f1185fd93fb3ef36e0b2b514", "result_sha256": "c1d02f17f3f587337a0983745fc6b970885e971f5e8e6fa7a5e337b1bbe7e482" }
#516object_counting heldout	11	The answer is 11.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "6336f778d4b8182593aaf3d1ff0127e0e1cc2b49284711110964ddbfac8dbf27", "started_sha256": "ebb0af1b5bdeab288356f29043e5a1e1c2585bb8e6dd43f197bfbdb6abc3802e", "original_grade_sha256": "26d30e0e6db2378c3f36c49c324bd9b53f168219f2c7d853d5602cd0afc223ba", "input_original_sha256": "6336f778d4b8182593aaf3d1ff0127e0e1cc2b49284711110964ddbfac8dbf27", "input_wav_sha256": "ea31e4fbc1014fab419a211a3801a685bba4cf570211e01d8d972656c18277e1", "input_pcm_sha256": "b51ff76df497b28022fc5653f6babfcd479637f270e12d3c4207b246962a96fe", "output_wav_sha256": "ea2760fe2fa2e6cba11ef2c7006eea16538b167baf0be2678deaac802519e652", "output_pcm_sha256": "c0a91b34ad4f2b134883a607f60a0e23cee98504b58a0211a49c223d423f9b40", "result_sha256": "f852cbfba17d7dff38f8060c142f183aababa80c05fea3f20db85c1edf98f4ac" }
#772web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "e89f08cfb107483af129cbb96a88814c1b8021834908861bcf5c58d1987c01ef", "started_sha256": "4edeaf8ced4b30553726bfdb817647f7d3646284e293a78a2b6336710938ec1a", "original_grade_sha256": "58f494b86cba058b0d8d0a4933d457ec2fcde6c142a8ab33748930639fb064ae", "input_original_sha256": "e89f08cfb107483af129cbb96a88814c1b8021834908861bcf5c58d1987c01ef", "input_wav_sha256": "4f63f10d8080de0b649511868977be2458839f1977acc48d98eacd501a1d5cd5", "input_pcm_sha256": "f3533770ce0331e153824d6df161d9760703a49b6313b870808fb400a51de665", "output_wav_sha256": "3515c3c677a338c9cd19fe9c4fcbe05275734c03fa8ef409c33842560558c07f", "output_pcm_sha256": "051e52cd13e9ea2d85e0cb8e4e23e55a29cb5e62b069194d809b282feedea081", "result_sha256": "5c13a408f8ef621f57e6120f13346efcd4fc68b4297be889a2f95e1c436160bb" }
#0formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "e237555c1766b74ed79806395722ffe42168a4b7ac4dfce03f52242a4031772f", "started_sha256": "dc721330c722a1497ec8fcd5778241a5223e6c4d2c950e9e6d7e810926e095ba", "original_grade_sha256": "64b59213f8675ffd0090c13e9f0cbc8ae641872fb5103a5d1bfdfdc8281c8749", "input_original_sha256": "e237555c1766b74ed79806395722ffe42168a4b7ac4dfce03f52242a4031772f", "input_wav_sha256": "2195851d5508955084b8d6fe373d466b24f952a50350fa3a66f6dd5c0388e56e", "input_pcm_sha256": "136e836eb7047dfb29ab7838f75902127b8bf7c6919c19abfd75f2d73df63795", "output_wav_sha256": "fb6ff69b4b9cd30af3b931b8d70d1629b6e0ffc12bc3593847b6460387b5da4d", "output_pcm_sha256": "b823e0a2c2bf2a555f26d0ee41c6915767bcf6e88300564b2b6731004f0caa32", "result_sha256": "b01140c645773c3a01bc6715914992cbd83ead97ffd5ef7872fc4281e9d826fe" }
#357navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "397c09540bf03072c40f53188579a3bc7836e9334cd309920f2e0b126dfb8e98", "started_sha256": "3e732c9e6152f2d4fcf804a3aaa4eed9c56215dcd7974fd79d41215328471cfc", "original_grade_sha256": "ed37ea61bd1ea172570a2370ae933feaa26ed8174527dd5e7f169d9bf5e7a284", "input_original_sha256": "397c09540bf03072c40f53188579a3bc7836e9334cd309920f2e0b126dfb8e98", "input_wav_sha256": "a75b226458689247e77639753f9a3ffc210c095b3bd763574da8222b4437af56", "input_pcm_sha256": "94424320fe436f01d86fbac9667223c5f87ccb84d10799c0acd68e9375b79820", "output_wav_sha256": "15e5df7872b0e78461879d348f986ffdad322454f4b143107f5d08c0c2cd5f26", "output_pcm_sha256": "869e9cc04064f56faab5826f6514826bea9c44052eedfbd9cd0b0f0240aacf01", "result_sha256": "474e36f29c8b3ac14ba8088d906b8aa9a2f45ddfc3c2c6e13d8f9d781f8216e6" }
#515object_counting heldout	5	The answer is 5.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "1d01103c0af6241f487e921261091884f85a3f2b2fcbbe37089c973de008e039", "started_sha256": "cae47bb42573352a8574d207a89a421fe9ec6ecf4629b8b524c53ebf94f17c3f", "original_grade_sha256": "ec1e8d41b67b15010f5b151e546d0472c4a02dc7bbfac88e129c52b12ae8e72f", "input_original_sha256": "1d01103c0af6241f487e921261091884f85a3f2b2fcbbe37089c973de008e039", "input_wav_sha256": "5e9b0455d6e4dd223c0a511a50a59b454bc33a23ad9db67373b08d66dc14c85a", "input_pcm_sha256": "b51dab69320a0dabb71f0e6435003ec4b3e184d50ffd61f24605206a550ec2b4", "output_wav_sha256": "367d16297eea89f8c083fd7ec9afd43b1c55ccb8f18378220918a7bc6202c6f6", "output_pcm_sha256": "d075c5dcfb66625fde9c62c6bb19b687f738644958e896ce405e760708af06c2", "result_sha256": "d3963e8a2354d02f6e7faea22f2d37f19881fd35fe7d2c349592d127028c114d" }
#816web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "7c60c2bb3f3b32dba05721cbbd622286cb78d83956edc454f11b1c2d6a79211e", "started_sha256": "7a146c103441e237d35cb5a1c53e6cb496e4bad6b8d31f95cba8a0f3d0af199c", "original_grade_sha256": "cae68550243c8701e1f297e9649a67c928a613241268049bf2a4bcab35d4a0a8", "input_original_sha256": "7c60c2bb3f3b32dba05721cbbd622286cb78d83956edc454f11b1c2d6a79211e", "input_wav_sha256": "902007a1c9e186661db67dba76c77b528b363a7c29b7b4c74adf5d3f64ba1f46", "input_pcm_sha256": "47de11fc21e540ee461c499cabedd40bccb4793f1816cff53c6df6b2cda3062a", "output_wav_sha256": "c0bbfb6c48b1da9d615c21107d6937c3dddfa45ac2d24c9c2abd3187337493b4", "output_pcm_sha256": "9a6110b8e577dfe594c74e70df8e756c922f1dddc16646a211a127d2863d124d", "result_sha256": "11a51cf220c944716a77324dd8e6bec21801faf827197fd530aa4b53991f12b6" }
#103formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "e975866a0a522b3e5465cc2403e1e4e96b39b7fbf30c45afce5fbcc2120d2f8a", "started_sha256": "98acd2d156880d5b3f500cb6d08d0c201b67cf2a5fa92b562ffaa2857f550b22", "original_grade_sha256": "54407212eb7bafafa0fc528a96f241ddf8c614e24ac0efb54c7b9a66accd1b3e", "input_original_sha256": "e975866a0a522b3e5465cc2403e1e4e96b39b7fbf30c45afce5fbcc2120d2f8a", "input_wav_sha256": "7e9a8c96242f0d2f169c5a79e8058f31d745dabb3981e50144f471f183928154", "input_pcm_sha256": "fb621247a8b3d99ef66d0260829eec30950884b94e730f29fe7eb61b75530e54", "output_wav_sha256": "2b567c0fbd523c439840922a39781eb60e823301a039e9e5621eb5a7fbecc5ab", "output_pcm_sha256": "a577ce33f0fa01e20a6fcd07d377359813ad0b7d1c44cd33d9968b00ceffc598", "result_sha256": "f8ae99d661566a861506fa7d8c24ad12c4dc126c3033865c36c128fe996f2e37" }
#479navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "676422b714bff5d9412d8830ef8781238cdd18cc61cb36edede552ed963e7b44", "started_sha256": "d574ae822dd9679407764bcc34e8fcd5b6e5fe854fc7c876e006e35f5b18d876", "original_grade_sha256": "fd548b3e0e7a6c7886f0233e4993e26b9a74db166ee75adfed56753243fad9d7", "input_original_sha256": "676422b714bff5d9412d8830ef8781238cdd18cc61cb36edede552ed963e7b44", "input_wav_sha256": "f00dbeb61b1c1162666af87fbb610d2feefc045e7bb2bb98bffa96b2e4180593", "input_pcm_sha256": "74b20a2c80a9efeef5c6620d1d056b28f687015a58d1cca60f424be0b7241e67", "output_wav_sha256": "3223dd679da51a94800fb6974e68b7f6c861d7094dadea0606f195506f4c72b1", "output_pcm_sha256": "2f6b2b1298c6c9353cce15c2f20fca89834a4089eee97d26b556804855c9769a", "result_sha256": "8e27044a53befddf530f97202ecbea06eaaf0e299563888b210b3ab468d95533" }
#512object_counting heldout	6	The answer is 6.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "5a8ab45a4523051003d573de4f0b295c65ec56c73c42b55ba32499932bf1ae64", "started_sha256": "b93d9c1fa4096f5142db103e250f37b71f5dc655e3cf326ec121435ad5697e21", "original_grade_sha256": "8f6ec4205be14010151155b01fb726c2cb29bd8796f8bf7dbd1f7c9b1aff71cb", "input_original_sha256": "5a8ab45a4523051003d573de4f0b295c65ec56c73c42b55ba32499932bf1ae64", "input_wav_sha256": "7750f58ebb3504f8b142e8c9a22cd5dbf59353f94c296aba7691aa01814cae4b", "input_pcm_sha256": "240590b78f28013dfcb12011ccbb4e8f490f9a9655321b916f71af911fa1d89a", "output_wav_sha256": "2e03ce2e6f4f879c3d274921413a9d5ea64156df0cd9b645d0c4d97ccc867725", "output_pcm_sha256": "5ad1036c8d61faaf116ade39a79cedb7cc65d27d4fdf6089aa3d4fae0a873504", "result_sha256": "3df5e5e28bdfd3e082c531810f447c92b9b6071ca4e8efadd38fe930ab563025" }
#873web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "cf05254d706a10523b1aaba2201a2afc358e3a3baa238e22afaf3c3b510d2000", "started_sha256": "0680a56c86d35a1474514d6a4c5ab42050e4e1fac539c61ffb66e0553743438f", "original_grade_sha256": "8e5fec72bf293365e678350e9e6db310e0ef5a949e8be64c46fdd8a0aa0decda", "input_original_sha256": "cf05254d706a10523b1aaba2201a2afc358e3a3baa238e22afaf3c3b510d2000", "input_wav_sha256": "c25f06708880b2d05690e5532249ec25f50f04143ee56cd33ee7861405c1e642", "input_pcm_sha256": "857cb2779d9b6e5c5067dfcd9f9b2f720c56fc3891b6faa58429dd9ed96e04df", "output_wav_sha256": "53b5983408a9c4129efd49cac0ed060a4149305e7a77d3cf0f12851daaf5a526", "output_pcm_sha256": "5f24745f71a155de66a878ac6062bb875c1c2a93d27a68ddab923bfa879f9af4", "result_sha256": "69ba7f80580c8c6fc75357953503a63f2aa6246afb9d2cd46415051e153d7201" }
#142formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "a7f4f65a786c78bbb0b1e4f8bb7f27abb571eb8a1fe15cf2b596c0529292447f", "started_sha256": "23e3f566493eb334ee359ff06b338ca828b6c82fe1b3de807aa6307db6519fc7", "original_grade_sha256": "1006dc580aaaa5e758ffcffd5a930740223e6fe629c9cd26cfcf1fbd991625b3", "input_original_sha256": "a7f4f65a786c78bbb0b1e4f8bb7f27abb571eb8a1fe15cf2b596c0529292447f", "input_wav_sha256": "d8e09434503dd699448188ba45b8dfca9beeafb42a5eb998a840206cdeae8345", "input_pcm_sha256": "ad6024a5ce1d98142cf38796e09df16b23882c221c769664aea6029a23572b4a", "output_wav_sha256": "807466adc4959cf4fa3253a22a4a293ae0bb065d2088f4f8a0e6c16aad2730b9", "output_pcm_sha256": "e4b1bae4ac886c3ffd9ac558746a1c98b80bbb5e32afbf226e04905754a35107", "result_sha256": "8694ed256b0d6f2f769ebcfb8c683d7b71892b2abde70a921e5a4f0751b1ab40" }
#298navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "3642f3cca4bdba47bd7c2f2e4299e9595d4e7b060dd59c557ea93e98a0c4ecaa", "started_sha256": "8038a6a70fffd622d22061686dbd3d62cf1e79bd8681036971091dd7a2c5ff85", "original_grade_sha256": "b0a6e865feac55a3edd49ffe69dd50db902545865a2487e249b515147a133a36", "input_original_sha256": "3642f3cca4bdba47bd7c2f2e4299e9595d4e7b060dd59c557ea93e98a0c4ecaa", "input_wav_sha256": "128ed157973c2d77d50705fa3c8529a09e0982351b104b757b65b0215080ca6d", "input_pcm_sha256": "216a68dabb032db0c53af24953c74915d874a20f714aa3d1ff69d7927b021071", "output_wav_sha256": "f749b7390f122bcf51067676b32c8956eb46646c18c61087d65190bf75bd9d22", "output_pcm_sha256": "d12da94d7b2a39a9887a485b8c27546f3c50a6894e26075ba7cbfb3f6674991e", "result_sha256": "d9e9508267689e65e2d0f89491839bd55e872d1b676861642129e825f6474e19" }
#510object_counting heldout	10	The answer is 10.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "71aa468e76ff79c3fe4604b694818e1e4b707a3848644f43b3180d4cf829ce0b", "started_sha256": "27783e67365eaa8c24daa4f79f34fad390fbbb0a2e8f562e0c62185fdb3751f2", "original_grade_sha256": "ce0059fca1fc82e4530bab6d530fda48e5869815a1c76ad44a3193165ace50cb", "input_original_sha256": "71aa468e76ff79c3fe4604b694818e1e4b707a3848644f43b3180d4cf829ce0b", "input_wav_sha256": "1aef6079610ff69303e0a3f933496b906419cac19f8a18ea99a352a26ce06662", "input_pcm_sha256": "c80e77b61a275459bf767e62c4dee18b0b514459b7a5ad43e8152cc3879d2cdc", "output_wav_sha256": "c2df06efe2996afe1af6533dbd54627605f9c17b4dc40ed90713f0772fe34dd8", "output_pcm_sha256": "e6ec173a372608db4f43da45aede443450f2bcd1064dbb1790ea14e0db0ff3e1", "result_sha256": "4ffc068c469e1bf1b14cba47b5a3ed6ae53754d032af7aa7103726b93e5f43fa" }
#869web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "ed3cb9f19e564f18140dea423b5a765274185fcc98e89b055a693b2b87b47c77", "started_sha256": "acef2c9d6dca76741404f09f6733c5efb09459f7294c923a803db455c83f4dba", "original_grade_sha256": "f82687fd5de478fb6c3759b39a3fa41a854f57474468161b26ce9125f631c50b", "input_original_sha256": "ed3cb9f19e564f18140dea423b5a765274185fcc98e89b055a693b2b87b47c77", "input_wav_sha256": "c37ebdf829ea607dfa60380309dbee979ae9211b7b196d0afd67efd7c250b0a9", "input_pcm_sha256": "f6754406a73f5e5f039af626d6643cef1df25db78a777a8fb03d8ca1575c9e79", "output_wav_sha256": "47270e2e5c61fd8fb6064e6b7c80e28c43f08ee301402035b701bc76d364695a", "output_pcm_sha256": "c59d70ba8a439250b630362ee4b903ef4894755c752d210603f9a48b1d647e5c", "result_sha256": "985f2d2cab21111ba223f5a4afd970dd628783bae397449a24f0a41c5fb77c55" }
#9formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "3b82e5d2bfe4ff9880bd43a9c4d1a16a6ed3e2c3766ffd1260c3262054858f13", "started_sha256": "eeaff218c2e62539239aaacccf91ff5433b1cc8d526c9f90213d3ba8a5197ecf", "original_grade_sha256": "d6d76b5eafc83dfe715a4d5813919332c21e8463649d0047f3d4dd0802013749", "input_original_sha256": "3b82e5d2bfe4ff9880bd43a9c4d1a16a6ed3e2c3766ffd1260c3262054858f13", "input_wav_sha256": "71ecd34bf432d1a2cd9e204ca2ef8ccb694d027331bdc66ddfcf5aa20dff7fde", "input_pcm_sha256": "ff602de398b8de82fb816873194ec1723c4dcb92438b2baaf8c2ee7c276c858f", "output_wav_sha256": "a02df3d081aa4164061e5657f28efd154b00a8bef5b2aa1cd4d5a7797469f71c", "output_pcm_sha256": "ff2f7e31b668715f208d354c80cae9220225d6b75146f75e823a28af5f5ca2c3", "result_sha256": "ea186dfb25c76453e9bbd4a2d7b879400904f2b2de84631fc457e5994f606d14" }
#464navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "5eaa8efc72e4430deb358f0972bc4aa28a7677bdf7a39d9bf944ff7397935f4b", "started_sha256": "2c0e31326f7672606a42ad11d89c41d056f7b6f2b24929d93a337e96b5f161ff", "original_grade_sha256": "8a3675a516a3f543e2b61ab68ded6fcb8ddc4b6e54a8166436d45a6931c6ced9", "input_original_sha256": "5eaa8efc72e4430deb358f0972bc4aa28a7677bdf7a39d9bf944ff7397935f4b", "input_wav_sha256": "11d06b51b336c4db4fd783fa26b37aeebcccf107982a62e2d8eb666b69a60848", "input_pcm_sha256": "29cda6a56ef6993cbd54e21f023c67a7e7bf41cfe172029407f44ff6f59aa058", "output_wav_sha256": "569cea3ab3a54e0fef82ec72066fb6b1edc488b1633e3a8b6cd5cdcef91fe8ee", "output_pcm_sha256": "bb0cda392f60f7c4c0a44e1a1acdb635a5ff93b57305ec1adf60be67b725fd91", "result_sha256": "401093a4a9caf51450aeddfd774473eeb479c65d27889a6c457ac9345ece77d1" }
#627object_counting heldout	8	The answer is 8.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "934d031b8ffda0b281e387b42d7591a27820e7dd584f58a0f957767ecaa7deda", "started_sha256": "05905026fc683b9fd86b6bd16d85056a575512c1f90099070af5ae8cd4ccb175", "original_grade_sha256": "07b5d8390fffbb30d0e81d2a769cbdf8354a0489e9a6e3f22ef77d3a1d2a2c86", "input_original_sha256": "934d031b8ffda0b281e387b42d7591a27820e7dd584f58a0f957767ecaa7deda", "input_wav_sha256": "05bac5c2e513be051cce749c7d8f278d256ded9d9a3a94e65143e47d388be0ef", "input_pcm_sha256": "677c2010b11baeee3a144e75535c04db33e9ba09b167b8fee0a630edb3a304c4", "output_wav_sha256": "c3ca3a734e8c2f2ba9c4d0014f868838df0c0586b684236933e0427b28c60f63", "output_pcm_sha256": "10865f5b168edfd53ac6ac709e52abfea116945b2d15467c7d3dff96de39b1e7", "result_sha256": "4ea03756d5c2009f32256eb642984fee6cfb75d2e1f5ada92392532a230e5119" }
#938web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "62ec7393af34590ff368c3147d28f03ae41da23580bed76cc35a5d6b2930a39c", "started_sha256": "557bf3db8281909f4a6e2dfc436467f591b2c1f0c42515a70896bf7d93dc3818", "original_grade_sha256": "a9cca5af5b1fe00aaa738a75159d0fc83efd6d4c7ea8b7088deefa8ab94bf4da", "input_original_sha256": "62ec7393af34590ff368c3147d28f03ae41da23580bed76cc35a5d6b2930a39c", "input_wav_sha256": "354327c6326830e9be2fbb937f423868e4cfce186531f0d299e3b58857a9be4c", "input_pcm_sha256": "389f72ad123ff27d6933d28b67112148a27b4e015db2a8f77f49813fa7f7067b", "output_wav_sha256": "c5e84b3612eb946985c4999ba6b646794a19662be7fd0b00467bb9ce6ae45b8e", "output_pcm_sha256": "cf13ed6fd9fc0b0dceb0f7ecf25596097b7d14ef72ede89f1c8666d6b5e83e7f", "result_sha256": "344524660c6a3f81977e1f3fb2fb7b251f4859936c9a25a1f01ac6145702df0d" }
#98formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "eec9c5cb5625fcf6582452c07af2e3cd4a3f370d45006f3cb4b20137811eb92c", "started_sha256": "cf02f13a1180325fbbee9895454d1e7f94303354d386a5890233629c140381f1", "original_grade_sha256": "50100a5e4e51e4968b92b9101b8ddf129452d4686b93c6a1aab4b8ceca4a5cfc", "input_original_sha256": "eec9c5cb5625fcf6582452c07af2e3cd4a3f370d45006f3cb4b20137811eb92c", "input_wav_sha256": "44dbdb5fdce8bef221621e75391e7684040b5798e6e73b09c7ea1d992438f8f8", "input_pcm_sha256": "f69d6d8043df0ba888087adb898d069c01301617d0e70fa0eca444d861b73c71", "output_wav_sha256": "47d221a355b6607df497ab74cee8fe984f874ae9794a72229cf8e81f97d5aaa2", "output_pcm_sha256": "c7968bcb1c4c59e108d496226a6cb3d3e9acbb243c0c76942bfe92e4bf9baeee", "result_sha256": "8f26b8dc9d149475b50df06a77296fa434ee9b4e2e096ed2bc3e6c921ad8cda3" }
#264navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "aa672768ac66e56cd4221abcbc1838539fb5029396bf2db9801897027057f9fb", "started_sha256": "b3129f65b6017419473b0b70310a704182d5ce366120b070c836908ef5d470d4", "original_grade_sha256": "eb9a789b0343795cf9901a618d3042f5caed6572abe6131528d9d665f2137231", "input_original_sha256": "aa672768ac66e56cd4221abcbc1838539fb5029396bf2db9801897027057f9fb", "input_wav_sha256": "b342938930387c405777ff0ce0207e905465a1c3636cf50bb2c7838816e4653e", "input_pcm_sha256": "3d3e3dd79cc1bf052753bdb012fa0baab6c204cdfdc45eeb744fe21e96e028c9", "output_wav_sha256": "4ef2c019c67d339fab40b6d74804439221b1acc29f20bb0ac39a7ef1333574f3", "output_pcm_sha256": "ae0ab1bdbce961556c3598af1ede74f299aeba62b1f23a9b6ffef104507ecf23", "result_sha256": "4ef0fa74bbb78e5b5a4cc26e01cb42061ad226cebb18e7d6f7b3c60733c48416" }
#718object_counting heldout	11	The answer is 11.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "3875e30f8e9f1c01f80686d9c559d823a3d5df6c898ab7e2a925480bdb17844b", "started_sha256": "e5d55b43444f849cb3659b318a1ec90664e24da5d054e8321cc4174624a28a21", "original_grade_sha256": "c031a55e82738d4b3d5691c85b6df08b80d31cda5fa7463b05c24fff8fc8de9c", "input_original_sha256": "3875e30f8e9f1c01f80686d9c559d823a3d5df6c898ab7e2a925480bdb17844b", "input_wav_sha256": "a9ebc447bf3dfc41ad3d7da94b3f2b0b2fb1a87859a955cc9b60f68b53da76d2", "input_pcm_sha256": "a6beff009e786f0e3b346e8dadc88a900dceb7dea53ad9d55b25c6589e389a06", "output_wav_sha256": "e76486bec4ba83f7e901703448b2c237ce6f7b3049de49f853466c8cf3fa0747", "output_pcm_sha256": "06bb22a4bde367cff064124ab4d4d219455f171deadb8d0b6f9203198fa1470a", "result_sha256": "d8d18fa951158417294c4612071cf2d182d0326924528a4b502df6e5bb31ea8c" }
#964web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "127598cb41751eb7be91abd6c7de72ae2ea6e0d9e39463134a77e9716db16d15", "started_sha256": "d1582f220bf021aeb49f725305134f4966b9c6a449d3548fd2386ab124066e3d", "original_grade_sha256": "4ca695e45e2ad73c5227a755c30a2c6eda159d05a4f6a77f613c72fae0830d72", "input_original_sha256": "127598cb41751eb7be91abd6c7de72ae2ea6e0d9e39463134a77e9716db16d15", "input_wav_sha256": "95aebfd63dcd6927bbc16c0cbe109c5eb1a4e80d8c2828d87393f5473b03faa3", "input_pcm_sha256": "a22bb1075e9f93d7e05d4988aa3196e5775489c00f212eca43af69360fbaf841", "output_wav_sha256": "1c584848df4edac81b324d4c530dea6cf54562d6ca493d212eb434f88e7d796f", "output_pcm_sha256": "561b834cd6a13a47b7225c4e5b2be16b8f2bbf1fd2ff3a478348cf72e1779f05", "result_sha256": "abb457a701c635d6cfd5c04da0b60ccc7ab91172e098442e4597fcd1874b5779" }
#109formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b1cd3e256f028a5df9b46e65673163a536d98c4a1bae53f08bf54edb7e871c9d", "started_sha256": "c831751fb96e3c79598ec7f87497c337694e2326b116180087aeabc5daffc15c", "original_grade_sha256": "c4295338d35c0c1f5ac386ec356d28f159642b63477348871fe9738fc612ad3f", "input_original_sha256": "b1cd3e256f028a5df9b46e65673163a536d98c4a1bae53f08bf54edb7e871c9d", "input_wav_sha256": "bf37144daf988bed9a7f5bf3ee88b1b128d4e2dccff9b02fee7527f2ded20b09", "input_pcm_sha256": "d53f798dee252ad3d24799bfc8197b0376d1ea3a0aea3a0063e15dac775b94de", "output_wav_sha256": "a6c9955f20bd8068fb4b09979af719b04fba20a4b17d15c857f7d20743d0e7e4", "output_pcm_sha256": "93383a9dd3f500a749bc9780512b2661388874d917885cf28ddc4f7bc02709c9", "result_sha256": "a038921cc7bf3f10b77fa621d937d1a0ab4828d740252c7da49270a314ff34f4" }
#493navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "a866d4847fd6d980afb45ba097f1cd243cdfe4635ea5e6a39a936179cba0d950", "started_sha256": "aa704df387c2d14b33e3d7fea08baf97932fe1dd894f4310bfc9abb6df56757c", "original_grade_sha256": "99302eda22ba5eceb4d0174a467187ce36c4b701fb1d0d1c696e1aba9ad0a44b", "input_original_sha256": "a866d4847fd6d980afb45ba097f1cd243cdfe4635ea5e6a39a936179cba0d950", "input_wav_sha256": "d86f4c15c2700d225677f6ae619ba2a251903d3cc7be9856b9b8878726d1a4c2", "input_pcm_sha256": "3243d84fca82fdacbd35c4b7448282861b9a11889297c19d15e44d387f8489c9", "output_wav_sha256": "f97fc5deef2f959d2d8299e646e471c247034318f5471674a04f9b5995b7fc1b", "output_pcm_sha256": "4cc0ade437f334e15c78cd056bcaa28917268dfd1802545bf27876950311efdd", "result_sha256": "90b0ddff1d8078f2fcca6c133d14dfba005530b430f5911b704acb25979ca5f8" }
#569object_counting heldout	4	The answer is 4.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "181c1a15ecb88bc0dd600faa5c274700089c7fbcc89b0b921e236c0d6a54d51e", "started_sha256": "076c00b40b8b697a849192d521aff9eac1eae774e59a139a86315617ac1cff77", "original_grade_sha256": "e1b5dcbdd80235974ce68660b5e0fd8a87cbbc6f76993f45f621766cb0f274f8", "input_original_sha256": "181c1a15ecb88bc0dd600faa5c274700089c7fbcc89b0b921e236c0d6a54d51e", "input_wav_sha256": "6bd12efe7056c2f9894a7f7d6ab3bce66273819c4c2a4b9114051077a94f5745", "input_pcm_sha256": "4251b6185e359700f660e9339ca142160176eaf9bb576d8b772e8cc5dc0bd9fa", "output_wav_sha256": "cdb3184ff7b8ef8102678f6b3feb61898653334d3a4855c5915346c5290bce2f", "output_pcm_sha256": "06f9d8b5eb1961c8a5f4a2c27cb7e115db784f6c434cdfe36a10c0c13a99e50f", "result_sha256": "2ed85f2ec7fe8399171fddc42f941e0ed49706114464b67432494aa53b9413d5" }
#856web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "934b000184985a61f134666540e9fb2cf6c17b6d6c8a3df1d094c8d6dcb17304", "started_sha256": "a35f5c8a2971747af9a06bdce7d80f8377db180732e29854119cbade69f9e947", "original_grade_sha256": "c91244e9759b097548814df83f1e0b13c294698a749e867098ce547fde22f546", "input_original_sha256": "934b000184985a61f134666540e9fb2cf6c17b6d6c8a3df1d094c8d6dcb17304", "input_wav_sha256": "b11952dd003e8c51efd6d489aa6a0c709730d88f5a5a64ffba3b0dd4895dffe9", "input_pcm_sha256": "8236cfe13c5c525e56a652039dec77b601262a7749368827fb8d5f9553dc453e", "output_wav_sha256": "33be75eae3f86be1fb40e0e3a50c0a957b056bebf2e6fac108b0a791d3bd99ec", "output_pcm_sha256": "85bf9de98210e0d2738cc917ea52ff576ea2554dff45308b017d928a9eaf8a5a", "result_sha256": "9b434cdc84807e3ae466e58743e7fae1e974a1365627b6c52eac9e297054d226" }
#136formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "0a7504853d6d3074bb058a0f630d9e204ea6fc9871ebdc437707be13c8cb5c3a", "started_sha256": "33f49af4c82e37094f97106bfa713fa041e559a4a2a03178ba872502ffa770f0", "original_grade_sha256": "3805a81fd3b604cd543dde454abc3735a6ec32561690e46bed699787662202e3", "input_original_sha256": "0a7504853d6d3074bb058a0f630d9e204ea6fc9871ebdc437707be13c8cb5c3a", "input_wav_sha256": "b98d352d63e4623dd2c654d7f6da9126c250407bd2590add2611f2fcf2b9d937", "input_pcm_sha256": "4df9467b0b03c444f495a1b0934c472497d09e9c5198dd2ce1e929cc3d03a68e", "output_wav_sha256": "5cd3f19d1007837464cf279663385b93cd902ae319caeac6adc71c4da74c0814", "output_pcm_sha256": "616da2440c2a51a62f99e1a88dcd0046ae4e84617bf7b4e9ba56a899ba131c1a", "result_sha256": "2b8619f1e6914ebfaba3e844e07c766e891b8252777958796aa48d35ee4b7ef6" }
#391navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "68a6897725454f6209830ee2c1c632b790716a579d9b8e36b5a33c227e7896c3", "started_sha256": "600f3d1513c8161fafeb8be6bf2f88fe5990dee60719746892c54b343753d8a4", "original_grade_sha256": "98b1dfda5e7223e1043d50fe15879bc80f32dc040ad1625010a1fb00af2bfc2c", "input_original_sha256": "68a6897725454f6209830ee2c1c632b790716a579d9b8e36b5a33c227e7896c3", "input_wav_sha256": "fd5aafe6266c2fc219f33d00717b4d87bbe9a0af1c6c4d8ca767935bd96eaff2", "input_pcm_sha256": "df5afe4c7e002255ec13c36f1617cfb63d7e722393310db7f77e33960ec01f60", "output_wav_sha256": "bc8abd9c39eb574a66012fb9702401ea2139547e892f8c55944bf547edf76c15", "output_pcm_sha256": "10fd80a978b3b90ac5b7acb582a07da345946b06f6f79e1bcbe58b38b6e057cd", "result_sha256": "3409dbbbf33cdd6034fa2afb775058176dd8d8b2b681611d77eab96bbccc8988" }
#675object_counting heldout	12	The answer is 12.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f3ac314b2006b8881a63d0bea67d3af8364bbdba6b95c67e7254a79321918eec", "started_sha256": "96cf0605b140a3cffb18476c49d5a188f205e5386123add5493414c1b9d89da4", "original_grade_sha256": "bdd37dd238a7063ab839d83b8c3579dac2e593242016f891f811b23f4d56d388", "input_original_sha256": "f3ac314b2006b8881a63d0bea67d3af8364bbdba6b95c67e7254a79321918eec", "input_wav_sha256": "13538ae6fd87553dd340009feb97024d1d34e7483e917f32ea61af645b167852", "input_pcm_sha256": "61489a172385d75ffbb19fd89d4ae889a7edf9395dc12fddc836088b6a831f05", "output_wav_sha256": "34ef530e60a70030beab78d5b18c429bf8ca370d99bc09c3b82419fd45c3e6fc", "output_pcm_sha256": "7b4a9fc7f997b3c8a791cf549f3c81f06398100bf4ba92d4fc4299b135a181a6", "result_sha256": "21b8b271eb925402dc27aba2d706d2ba6f6070527f85c0d27d1fbf40f7388914" }
#877web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "d9d8964ce66eb8d196669a986ce5a5307da9a8e7e0011707815619ee9ac2f9af", "started_sha256": "b452e54ed07b72f07fb708b2db4dac9d0053eb7734874e8871c9648ba84a1828", "original_grade_sha256": "9ea86612d1e256f9c883b57e6e6025cdeef4c4ecc2c5491cdca4cebd3b889ad1", "input_original_sha256": "d9d8964ce66eb8d196669a986ce5a5307da9a8e7e0011707815619ee9ac2f9af", "input_wav_sha256": "90a52431222bd03ed58148e7f1c063778db6e84dacbad2c766a513411310d70e", "input_pcm_sha256": "898bdadccef778395f9095c816405c602b103ae0a72e71818f6d68eba3bad0eb", "output_wav_sha256": "bb394b9037398aa6aadcf21aa6c7e1403557f48f7e752aa61327f0e3b4a47942", "output_pcm_sha256": "71f6562e6a334853659fe9954cf8d5cfcde60933ed74a3c290ba8a9f8557cd28", "result_sha256": "efc26d3d8c5d519f626a1204bd0ac376b36ded869d9665e24ce3bebbccb37c32" }
#25formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "aff4c63436282f7cc2fd3739dff888736fe52dd8057dec0cd2ae2065948e0ffb", "started_sha256": "e3b3bd120c1f9dc7f3440ee9459f40f98855801f9c58440034a1675a04e63b9b", "original_grade_sha256": "919736ea7fad9d59a36a4c68de40944f5784872223a46d164659cb70924b2dff", "input_original_sha256": "aff4c63436282f7cc2fd3739dff888736fe52dd8057dec0cd2ae2065948e0ffb", "input_wav_sha256": "a54ab630c9a4a6902ddcb127f98421021c8b0f317b94f04ba9cfcb457b960ff4", "input_pcm_sha256": "7e41128575fd3c0907a91c4cca1d371c03ef871e0e1df49334da00d37f26eb15", "output_wav_sha256": "43ecc554342bb39b2b1c2e9bbb44e1bb85804e4617bc24f36bc9365f65c9d5cb", "output_pcm_sha256": "ac68c5066a73547e8039f223f5b59151c8a873a2700980472ba65a31ab55957f", "result_sha256": "4c80f593abea5a79c23ca1db2fc777cb3a8c6ec8a2745fdf2f7a236adfefb2ec" }
#344navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "647254e22879c4de2f1c89d747a7b828d4c265c4cde5be9ac02aef61c4984a5f", "started_sha256": "09b6ce7aecaff040e75b927b9a0a9d64410d3578d0c4f3b3d8e169081d43b13d", "original_grade_sha256": "2fa46d64f7a57ad05ec1a997a8b5a5b6343be2aa94b6476400e43fc1c21e2310", "input_original_sha256": "647254e22879c4de2f1c89d747a7b828d4c265c4cde5be9ac02aef61c4984a5f", "input_wav_sha256": "6c7312db7eac1fe301dc0c92ebb4922723b4cbc1fa0b6ce57e5476c266bbba15", "input_pcm_sha256": "06069957878d8c757f953e7fde6ca632dad5d980c5470d04d9aa1c12551ad1b4", "output_wav_sha256": "d44146003295574ef33d17b2be9c73cd97360454ec59a84ae7828de46c92c974", "output_pcm_sha256": "fabc801c69c3bdeb04024505a004dbbcd9e341cfa24a0721e34229a4a09f616b", "result_sha256": "6d2b663220586633b22352008ce736a5beffed8aee00b5c126ab56370a3c64d1" }
#546object_counting heldout	9	The answer is 9.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "3879ee260253b2b839f6368bc35e048cacf049d673a1d2ff136b9e8c0240b100", "started_sha256": "db551e64189e5d80076eab9cb337198d729aad2ef1d69ee7dad5abc78dfdbd60", "original_grade_sha256": "0830865095218bb1ee7c38af43356dfaafaf75aa4c50ac024e5310df3c6abd67", "input_original_sha256": "3879ee260253b2b839f6368bc35e048cacf049d673a1d2ff136b9e8c0240b100", "input_wav_sha256": "793c6efc10dfddfc011b71ad0ef546db676a17ad2b81f51d06b1724c7fced25e", "input_pcm_sha256": "900087dd385a363bbb62059b0d980bfc3bf5366be05181bbbe2b16846949b381", "output_wav_sha256": "2a0635e3743af9e7823543b76c57b48a0cc8dd61c687206721d229c5847c44d2", "output_pcm_sha256": "2c4d9ab746deb52dca174970bbecd3d375a63b513444ba3d1a34d5f32d15e29c", "result_sha256": "e11541ec4c1b0a59d71871285dfc1ed7b57b116a922ffd6b5fff15772eed3b46" }
#835web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "999b821b86c8bbde1e60ab56d637729b813714d1e224750f6148496301454632", "started_sha256": "696e4ba884d941f748ed2df1fde905c47ffdfe094b709d78b7775da56d05a942", "original_grade_sha256": "da21d6067d84ea834d00fcf5419549a9ff4b4ffde807c5dc3d67c573fa0f03f4", "input_original_sha256": "999b821b86c8bbde1e60ab56d637729b813714d1e224750f6148496301454632", "input_wav_sha256": "858696eaf2e7bea5f3c039c49461623be044c18f0b9b1242c38d8d44f968b32d", "input_pcm_sha256": "e6856210956315174f8d255771abd652eab9c2dcacd171b8de16c65b2e63d98f", "output_wav_sha256": "9185f0301370c3885bac6a4750ebf2a8f8ba24bcab6498bc262c490676f40948", "output_pcm_sha256": "d1f440128739bda567c5bc3f7dd197c64b6a3124fffbbbacefc1dc435cb9e143", "result_sha256": "0307cbf5540bb9fc86b8c18709db66d24b5013246b310184355af23a8d9d2917" }
#22formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "6fd8fa7fa0cefda5466638aace35993edbd67fd6c5d26a4e06864d3620bf2198", "started_sha256": "d5bbf42f65b3a774a224029be7a4773cf5b8459edac46f8c3148c196063b8c26", "original_grade_sha256": "71a92c4eb81b065399af3f395654a5b908af33fc11597701e9fac2c2e2f12147", "input_original_sha256": "6fd8fa7fa0cefda5466638aace35993edbd67fd6c5d26a4e06864d3620bf2198", "input_wav_sha256": "da9fdf6578672d19f2e4344916a2838377ec56716433fba0bada3f788a564780", "input_pcm_sha256": "6abd10e144a9e84e39b6cb237e8ea590a3ae006c896c0a0e3c6b3255c1a72e45", "output_wav_sha256": "723964a84cead2f28d6e1907ac90a6be3889079ea609472ff4037a04c472f9c1", "output_pcm_sha256": "99b95cfafb550ea9886d9618af1f699717bb6d3aab184b898ab7431d74fbbcd9", "result_sha256": "ef0ac230e29f7c4c36679658988d6f6bbc5d3e18d0e5d7ce8ac7d1e40ba764d8" }
#421navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "18f0200c958d8f28b6909ea263e2199f6a6663946b10c39a981258a7dd2fb713", "started_sha256": "457aa9204195d358643f0521856c2554845b33b827622a1d2ac5a9f408ad0d07", "original_grade_sha256": "49dacf36b7b49309007716585b2e1430bd282c5eca653d5a6b995ceaa8ad63ce", "input_original_sha256": "18f0200c958d8f28b6909ea263e2199f6a6663946b10c39a981258a7dd2fb713", "input_wav_sha256": "263b244f9bf76f5801424258b7652525a2f518bfc176c11aa0c385f9895c6e9e", "input_pcm_sha256": "f2f0a448de1ef8ad76b239d211e0bf87bc5f00f29debe707b55cc2c69725ba93", "output_wav_sha256": "4731fed8c8e08bd2d234264074fad05f05b4a6540cfebf15aa3c2d0083577dac", "output_pcm_sha256": "c23f75c4dd03891d23566b4df9830f9499e2e5e18659319f624e94d64867ac62", "result_sha256": "dacdc6479bce4583baa3c0c2054f99e5673c650159215165f37bea619b0c5419" }
#712object_counting heldout	8	The answer is 8.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "40f5e39fbfa2c29c47a3c234c1b42455ab7e27cdae217b24f519611a04690dda", "started_sha256": "a2675ede2fd6a1207b33d5126629553c70727acb5efebba4df03b9e2a8298168", "original_grade_sha256": "62f04b943772e039bed1d53a26a0274626c86aeee7588a861168f3c030f0bc55", "input_original_sha256": "40f5e39fbfa2c29c47a3c234c1b42455ab7e27cdae217b24f519611a04690dda", "input_wav_sha256": "37790b9e7ccf463dd7f37cca214a925bc4252385fe51c47ec572b5022b67eab6", "input_pcm_sha256": "6f1046d18bc67996899c52b3e36bedcbdebbe4f1ae8527f3d43c31688e39b08d", "output_wav_sha256": "e4233c7588b798f21c5e12e81cfa2b024fe5d8a01d79259dcc5771af2a1422b0", "output_pcm_sha256": "1a3f3fd42cc52db866333097a4a09bb24ae69c688c199851dc0d585b0fdbd685", "result_sha256": "b6e261c6659eae6dd1aa3c4a8a4c425a20d392b8d8731b4299648398fb0aef5f" }
#840web_of_lies heldout	No	That answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "2b3e3c8fdaebd3a1534b82587d293378a737ab8440f0bfc1faf690f4f7df9ec3", "started_sha256": "e77cd5682872dc32741dce849607622c05ed242556aa89a6b2179c6bc35e811f", "original_grade_sha256": "20d09d09d8e7e8a35f961cc46e12916e7ff95c41af45b5e1c65f1ede55e62cdd", "input_original_sha256": "2b3e3c8fdaebd3a1534b82587d293378a737ab8440f0bfc1faf690f4f7df9ec3", "input_wav_sha256": "95949eb46e542d6ac4a931f2aded63e5f2b5ea851a907d85ae627a556bd8f999", "input_pcm_sha256": "40a9f159714439c04370d5798e011c05765480c43c1b881ccd21209460a98e7d", "output_wav_sha256": "d8075653277f3bebe5b97572180a12c10693b64b95311a02a7ae80992f1b0d6d", "output_pcm_sha256": "84ea1508197440caa6697edc774104a0d1cee4c6c1b79f8c19414aa1e9795772", "result_sha256": "e1f4250cdebac0d931ee62281399b1d1c9afe0d0afa1bea012eaf61253a4919a" }
#196formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "c4e62b9d2ab72e87c86ec0a0d1d5b24cb3d9aac3054bf45d7a63030c1bef1596", "started_sha256": "7fdc7740248ac1247aab69db53395b8f3c4997bfd3c76f0a62db3f83a0469e14", "original_grade_sha256": "189c83dc8a1f8517c01c507f8eb6249eceff47f58409ec0d61c6749632cd52e3", "input_original_sha256": "c4e62b9d2ab72e87c86ec0a0d1d5b24cb3d9aac3054bf45d7a63030c1bef1596", "input_wav_sha256": "7087d705ed7af8a69eb00c2537aa74fd245cbff726f516e35c3798ef391545a6", "input_pcm_sha256": "fb3dd24d809224d51580c11886f43f2d9d0cc6dc9a5e1ee27e1347a28c550966", "output_wav_sha256": "68b5e7049febcdfe79388f5f97cec224722e5613532d8d82668cf5875ff8dca6", "output_pcm_sha256": "ecbe1c63484612fc3ba1af1bd7d1df6e0a925811ce2f6b74b8e31793d815a560", "result_sha256": "afe41bbf4dfd1fe05e9afb78b455d486c9260c280b5df7b3dfd3d4312d512fcf" }
#438navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "5976145c5819ea6722f1322dcc18856d154c225f37bd368960ace9935475e6c6", "started_sha256": "f8df3dfb688a37e505966726d93d8d1632f9e4d1c00f5dca2bf91998c1beb123", "original_grade_sha256": "4109819fb6c789fdbc0981694a432d3d878878b916a0e4c15e275c54ce9a170b", "input_original_sha256": "5976145c5819ea6722f1322dcc18856d154c225f37bd368960ace9935475e6c6", "input_wav_sha256": "5722dbbdfa4562bffe8ad7f75e083a58ed30edde06a914fd756cb7c441786c52", "input_pcm_sha256": "2e4047dbc3583e04d14ea18de97dceb20a97f8b46980725e5f81b809ca3a497a", "output_wav_sha256": "750e7b4d5f2b250fc8e6e93c58a9946ebfd826454a5ab3652a6a45b0f68a5117", "output_pcm_sha256": "a08b4a9afa2f3b0c55622ccf3ce12327e47e98ed676a2591965064029c3009ae", "result_sha256": "9eaf4a83d8a5b4ab7779764e7f2ca544916b55bc7bf75543b08eb6e20dcb1b18" }
#623object_counting heldout	9	The answer is 9.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "d113e09f52fa66115dc7632c3323873732f30eb29916aa4d9e02b9f803bc4f97", "started_sha256": "537aaad9dffc009acf6293fc64ae5635753a9022295590cd6495ab64e27e237b", "original_grade_sha256": "31edb39ac50c18f15e6d67da3a7d5283e38443e9066165375a3253028dae1e96", "input_original_sha256": "d113e09f52fa66115dc7632c3323873732f30eb29916aa4d9e02b9f803bc4f97", "input_wav_sha256": "b5fc11d217a989aa48b95554aa7c96444cfa16bc1747916accfbeb7fae4279fa", "input_pcm_sha256": "cd4c84ab449e4f0cd20f2018aaabe54d4dd1896db97d740c5be71fcb3d080047", "output_wav_sha256": "ec5d622f7c8459e0ab51afc19305663a5ba017adbc20298a262dec2b34876253", "output_pcm_sha256": "a029fc1410f7919d0453f0b45a002700226b18ee14c3f92665122a16431927da", "result_sha256": "eed5451f8d2ccba0d96aa3bfe968dab5b47d508e7001e889eaf7333c74089778" }
#905web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "3b8887629fce98249fdb37e7df430da8c9253848ce1cce8fc95f796dd4227299", "started_sha256": "0e289729b3858355262dd87d00f510b91f2f63d15270c70b2d46b43a0d83cc17", "original_grade_sha256": "4d92b1e18b8b1fc19cd2288c973691eb634c91c5cc29a4c51bcfb615efa21e8e", "input_original_sha256": "3b8887629fce98249fdb37e7df430da8c9253848ce1cce8fc95f796dd4227299", "input_wav_sha256": "5488e773dbdea808f15150adc676a93d2a08281b5fefc908bbfda3b4e1a765f2", "input_pcm_sha256": "2b4f2c9daa38dc63a1aa3633f561790fe034411d60fa45b611ed9571e5cdd9ac", "output_wav_sha256": "e39e8196ffb066e05d91a95639d9b06361fadb56f976f1b4df652c3b7b112a53", "output_pcm_sha256": "8b5a1ca0690d48623fbf026e90c5a80c87ca37e3d5c098e78ef2394ed24a85dd", "result_sha256": "af93edee6dea16b9e5e19f56a3d9152210df541d0663ce0a9651accadff1bb39" }
#193formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "3335dd61c6dd9b7e51d6187bb8d68452ed444ead0b28c26ad03ecc606af90d8f", "started_sha256": "e78bf9fbcd51e05ead8a9a7a1fc3269575a85f3e507fb25ed3c8d9929327360f", "original_grade_sha256": "6f6f5ba25a085537df7e64b8db4f7b3b23ed4b42b696d4ef02a36cfb1fbe0db2", "input_original_sha256": "3335dd61c6dd9b7e51d6187bb8d68452ed444ead0b28c26ad03ecc606af90d8f", "input_wav_sha256": "2501973ea7ff81e312b5b3d1231da85a4248f4831ad2d9e42b125c10b3a769bd", "input_pcm_sha256": "dabe82e8eb345723fa5f45fd54aee0ac56e882988a87880b892bc9615e90be7b", "output_wav_sha256": "a767f23fe71cdb7adb1cf6a50d1331cb86aaab4c40e0bac277b9a5e2eff1f490", "output_pcm_sha256": "20c1e069b57eba0a0bcd4a26f7c99062cfb9606815aa77a17721cf2e4135550b", "result_sha256": "cce1b557d012004a792eaa6664ec53e73423ebd5610e09d13ce96ccd135c0045" }
#384navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "d9fa8236918bd1c72f048dbade2b4b572518d0fdf48fa33e8c997fd15db76ab2", "started_sha256": "ee3d52c887d54e033259a28cf842ce93a5c4650e1e42d32642016ef70d52d969", "original_grade_sha256": "324c81803b10fd2fa2c315854003347ce368c2943d724826722fa1e3f008bac2", "input_original_sha256": "d9fa8236918bd1c72f048dbade2b4b572518d0fdf48fa33e8c997fd15db76ab2", "input_wav_sha256": "f362ff6fc3057e3d9035b31c59a6bedbfa6ecd3192ce7f3e242dcaebe2f827ac", "input_pcm_sha256": "1e0c987b4def8ff68c147ff9052ef157d5abd6f980a8c08871e6e0df2f78af60", "output_wav_sha256": "43d6d4d7052e632b70d70dfc8fb6571008d4392e0b33c0c471c7982171c196dd", "output_pcm_sha256": "9923bae23c1648cd856e1589714767340d5117b8b59dc95249b1cb2d4ec85f65", "result_sha256": "a57e54e235f3b43e403aa35d8f1cf54b5b1e7620ecb25abe44354fc8ae7fe5ce" }
#608object_counting heldout	8	The answer is 8.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "68855529939dd9a181ea074ec7698e0d6f96633d24a342790c329f28e89f246d", "started_sha256": "60953cf329137b8345a06ddb7ccde25dc1be73d15247c7466bfcd40f27f547e9", "original_grade_sha256": "62095ce3615204911ae259f4a61b494b55f73db601d6d4f6ef56ae35b7eb54da", "input_original_sha256": "68855529939dd9a181ea074ec7698e0d6f96633d24a342790c329f28e89f246d", "input_wav_sha256": "a5ad7f86a1b4d588e344e6d6d1683060d05c8a754ae8eacb9037044721e837d2", "input_pcm_sha256": "c89f99ae6ffeec2413b93b4cb63d3f365cde667f5d1aca8904d35e7785549e95", "output_wav_sha256": "e963b07aaa4eb43792f7f1d4908cc7a49b3e2f11694528aee706503fc2df6d4c", "output_pcm_sha256": "c319e152007810d902d9611b9e61469344499549509cfad221e2d13ae1cabf12", "result_sha256": "c0bcb6686271268ecdad8f37d9661ac27ca4c52f406a2b58340ec2d04a0980a0" }
#862web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "6914ea4c976dfc29b9ee26cde6a47df496751e4fc5903d64dc13397f8a6428b3", "started_sha256": "fbe3dfea011f1bf2897220e5f04acfb8f400343db24f63b9bdd63221043fd3db", "original_grade_sha256": "f40c1548323275eebda67e4efe1b93ae4eb9fe8aceef22c96cc375a72cf97e41", "input_original_sha256": "6914ea4c976dfc29b9ee26cde6a47df496751e4fc5903d64dc13397f8a6428b3", "input_wav_sha256": "8c9191243277b6ba66f80512c09ae649b622ab5bdcb6cc7648088e160b256584", "input_pcm_sha256": "eddae21d0aba2ee198c36f6f10af910d9ac6685b51e7298df4b30f9e18035f11", "output_wav_sha256": "31a19b89af800cd4c59d00948ede5d9ee66ace85d9c3c9297a86c761d1fe0857", "output_pcm_sha256": "5372e3858574f4f9e659c39095b77d87eebf0ae96610203d2ee3e0f1048ea140", "result_sha256": "8b94c7ba0e8e62e30b27dc88e596b0e8ac63e6f7af6228c3821619e637d58bb1" }
#152formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "3d48bf3b5a9dc55ebcc49fb66e38279d557602f50adc12d30da3ae3d9d642a62", "started_sha256": "9f9659933304fe68b6a486c0978e68caa52453d0a4e7439217c4f13c80848561", "original_grade_sha256": "fcedb4537fc57568b77a724e2dca653792ebbf7d51b28f260e20e552ac0d37f8", "input_original_sha256": "3d48bf3b5a9dc55ebcc49fb66e38279d557602f50adc12d30da3ae3d9d642a62", "input_wav_sha256": "5fcb7d3cb7c71d1fe14d8c883dd2e3e8afc2bfd327cdfe1f62c5569eb7abe742", "input_pcm_sha256": "66ff1c2590bf2fe3e60cab690d81f1d77e2511b6fc7b130126d67d75245d9e49", "output_wav_sha256": "7cfcb23a63e5b2aa09f4de299ee69a0e6021a465d146098a9a75a9cad078fe02", "output_pcm_sha256": "46f7bfddcdc1bc8917c8d950cfc3bf5b1058e8eab751a31dbb07cd5d8bd12f9c", "result_sha256": "de5fb65d2e7672e1e943a87cb5a5d52120c1debf8fbec42d9eea5050947357e3" }
#351navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "206f18ebabc0572143328d07136f487bdbab86f55d6ddb57c6062e8d58ba6528", "started_sha256": "10e0fc7f41a5d6874c4663f1daa12cdf5c6a7d26848ff8ee3ce8d252e2491873", "original_grade_sha256": "8748ea7da84fe8b0cc269337070153fef424bd0afac3b13402c8f96b00460bf4", "input_original_sha256": "206f18ebabc0572143328d07136f487bdbab86f55d6ddb57c6062e8d58ba6528", "input_wav_sha256": "867631078b029113adc7f8c08c3095ff2997860eddb9ab441aff9ef7ae31d561", "input_pcm_sha256": "e3d5c64f5fb778c5858d5d5734f2a55fafac72fd2841e81ab8c6c21868390f92", "output_wav_sha256": "5ca86705578f4f9ffb0e026e01aef1c26e5b3fbe75fc391aaafb3e91ccb9f86b", "output_pcm_sha256": "681b2d19b8321471d96164277ede402cc67c4c11107de95f57dce61c58753712", "result_sha256": "6a65fb65375f817f60fea1a04b0b55d5db90033532cc39b1a1c0ff5d65d7606f" }
#618object_counting heldout	12	The answer is 12.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "dd6931ea02d9ca7a31b9395d91c0e24ab8f99a06f45967d5b45bcd09b0b83d6c", "started_sha256": "751c470e165e3ca8a50bc6b7049acb1371c194599e2ef0c92a8a1b330c79b2fe", "original_grade_sha256": "ddcd506943292525b96b03cdf2c7da997ad64f460a562d1166049e280b064c02", "input_original_sha256": "dd6931ea02d9ca7a31b9395d91c0e24ab8f99a06f45967d5b45bcd09b0b83d6c", "input_wav_sha256": "ef738070da404b06a95acb82ee6d524b295407fb60eaed3a39ad9c2f70d7ec37", "input_pcm_sha256": "04ee0a07ca761f66b30418571bc358e40ac9f5f7c07d99fac2ddbbdc6cf6d6f8", "output_wav_sha256": "c5c3d6c2fbfd28afcd00ccbf9a0449279307f1bf218ceb341a356233661ab8dc", "output_pcm_sha256": "8e92b5a35677ff25be38f8aa368fb38ce97d23bf304ab32a312c100bd2a74a6c", "result_sha256": "6b7f9a787adf1bb5f7cd7031159e9dffcfe814aa19d8bf6a843652caa3e434b7" }
#961web_of_lies heldout	Yes	The answer is no.	incorrectstrict_spoken_answer	incorrectresult	Original grade retained Evidence hashes { "original_audio_sha256": "b2d2a68ff4725d16ebe2670dcf9b8d86bd81f2faf26ab2c233721328dd499b0e", "started_sha256": "db2c565a1cd553419239e527dc3591236b0b0671ee53f46a7fb4d9197479aa1f", "original_grade_sha256": "12e811bebc3d774c9e8a711064a971121b3560861a93336d70c03301971e1a2f", "input_original_sha256": "b2d2a68ff4725d16ebe2670dcf9b8d86bd81f2faf26ab2c233721328dd499b0e", "input_wav_sha256": "f610e7b1280a7362e2875c147df860e3a118b179fb5afd208ffb83b031769f18", "input_pcm_sha256": "d1b1fb09c5881be837b145ddd790b07c5de15a983b36891e9891df56e9065f56", "output_wav_sha256": "7ffc99efea061b4c2f50ef35c46610d7b8ce1d47c86506526e11e5cb8db29b20", "output_pcm_sha256": "e8011e191adca178f7c44f05fb59e429b8253c24a62d0f63811069c9d64f7f1e", "result_sha256": "40f5a2801cd6675e751d0aec6f68256b724cd8e0c6dfe6020af2a1ac72fb6ec1" }
#57formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "8d6eb311b7b97afdb3c0083a6b6a4ceac447c63dcfa98ba329436646cd35d711", "started_sha256": "41afe9b841a08976cdcc63b5b513c58eb698bbb7fc4670f95907e81c03de5431", "original_grade_sha256": "4b35da13f35e98b98a159904026e3e5d62b01c6b58b746dc5a0b92ffc9ea6f64", "input_original_sha256": "8d6eb311b7b97afdb3c0083a6b6a4ceac447c63dcfa98ba329436646cd35d711", "input_wav_sha256": "2e7c27b624820207e997c173fd746ece280eb6ddc23de221a4ef8316b679ed5f", "input_pcm_sha256": "ab517a69a3d96220b0715763ad98705be8e3252ee77931cfdc41bf71aef72d69", "output_wav_sha256": "f1ac4d93cd27b7d785a04060759e595a139129e0ac18ec7547f6061194dafe44", "output_pcm_sha256": "17e7dfc571d56dda9f6765c75f53ca2147ab712c7f5263dddb962f3b20420376", "result_sha256": "25df454f53702e23860b31fdf7cf8460ded47600b957d7bd362bbda2ae2b192b" }
#390navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "bc080ca157cb6493525b98fcdb5c8db1736662d8209a6a7b13c6278fe0156ea0", "started_sha256": "9726fcd217d6bb3cc514e0941216a7ca65f4e64a817829ef81ecde9664dd69b5", "original_grade_sha256": "8d0b40721b752f0495f7ca4e871c62f56e5be2c7713fad90fd613061a8694c2e", "input_original_sha256": "bc080ca157cb6493525b98fcdb5c8db1736662d8209a6a7b13c6278fe0156ea0", "input_wav_sha256": "92f043148e513ae17b1a57f243484f34b5f48730ee32cb0c3a43e4633005149d", "input_pcm_sha256": "2ed0f1e28fa9f74b7233b13d8086fe7465459db5012d00b506ff27991869f0f7", "output_wav_sha256": "e506583116d1249faa245b456852357673a29c1f70a8303b346e3af0251859c1", "output_pcm_sha256": "e98269750d51a17db623e72199db2f5b3d60409fda927d646bdcfdd1c4a9e66a", "result_sha256": "d3e3a59e655020fec14dbe9b5a3d467425e6b55c36c19b1c9d3c1fe089566d7f" }
#725object_counting heldout	4	The answer is 4.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "2e8be2d1767d66ff1741f95273b079dc21c1e4874593096b259f97b2f4d82bfd", "started_sha256": "14bff1cd0f391aa88b786864593cf1e60299cdc4c5fd375737f5a51eecf0c5cb", "original_grade_sha256": "e5cdd550e3442b70f9659e3a8184a857ffccbd3c2ec976dee318d7635b9b3b5e", "input_original_sha256": "2e8be2d1767d66ff1741f95273b079dc21c1e4874593096b259f97b2f4d82bfd", "input_wav_sha256": "60492029e588dcfc5e9117ffdaf7980c97ac87be4e79b3d9a635bb43e78fb3c6", "input_pcm_sha256": "7f9d7ee17a667e607c7ff32db07c0769cea9146e0dd6a69b578e334937507d98", "output_wav_sha256": "f9079e25037974523aeef9fd288f6f534f66b041557732a1044281e5e07ca4d6", "output_pcm_sha256": "8933371b0e65202335d138c19d6f45a6524607a51e7afc9956392667f5ab5e29", "result_sha256": "afa8f176d796a60d4c5110eb54e778c7bdbc6bf159c09fbc7ad8ae80d46356e5" }
#915web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "1660cebdabd4a64a888cf444b8330778c99120fec90e5aa52d9a700b14070ffa", "started_sha256": "fc1b8e6a92b0878d2cda646ac7c5c6a6d1a8e56fdb80b2c5bf6c4cf73a36d640", "original_grade_sha256": "2f1bd25dcf38980f5a8afe1a2b5ff2b527a9f989a37ac080293e3e75fd9c62d0", "input_original_sha256": "1660cebdabd4a64a888cf444b8330778c99120fec90e5aa52d9a700b14070ffa", "input_wav_sha256": "32693294fe4467d251fa7ec357dec9f16ac0f94e8cb62621f879c9ba1fd961de", "input_pcm_sha256": "321b9bbc8d7a4976c151b0022926121fd012aa00842ece764113e74b255d6b15", "output_wav_sha256": "20ee8405270ae9704c6371dbd9a77297545d0f3903d2003229bbd7660e6d72aa", "output_pcm_sha256": "2e427d15be6e32ef8a6b8f02ad3544abb7bfe05fdd27ad84225a5c26c5e7b244", "result_sha256": "ee875bb1de2c76ece66b983168a415ede82fa223323c2cd167ee582e189acb87" }
#223formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "d2de94a52bcae2d81f73e58d043970a93490f1d4d770e98058eede8a2ed9718d", "started_sha256": "5c19053f92b71518c83a15733f5b6f591d53a0b0139a0dd3b897ef946c36d0f5", "original_grade_sha256": "da6107ec4698590c70e2d1c5af4c61002f5e56b479a4fb52050304da3061c74c", "input_original_sha256": "d2de94a52bcae2d81f73e58d043970a93490f1d4d770e98058eede8a2ed9718d", "input_wav_sha256": "a98c8816ad92597a9fc8dbbb19d3a321bf13545e9d9728bd4024f3ef304b4fae", "input_pcm_sha256": "45682bc9d315112a339449030846a85c831f98a5fd2e562e65056285ca7a8b0e", "output_wav_sha256": "d19b7b48d98f1ce06323353b057e25f30b55c9ba672de49110f257b28c7f7007", "output_pcm_sha256": "bd3712d93b227a5f4e99d9d5d694d1df57c5acd96c53bab7967c67d00f844cca", "result_sha256": "ddd694fe226179d9628c36928939df1386b6f252e1e6695eb64d6b8b1b4c9539" }
#315navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "1093a3564fdfbbe8e3846d661a7cebb96fdda3381611d14e2ad7be15f7c550c6", "started_sha256": "a07861e2d271e743e227cadef007c9de1ed8f279e31056f4b3195a48989a2d21", "original_grade_sha256": "0f8267a40d59e1880f1248691cc362945541b802ad238fbc0a22ebc388d498ee", "input_original_sha256": "1093a3564fdfbbe8e3846d661a7cebb96fdda3381611d14e2ad7be15f7c550c6", "input_wav_sha256": "f24ac1b23aabf931808c23309cc4b4d564650780520c351cc922183e83adae4b", "input_pcm_sha256": "381f0218e36deb3fcaa3bd0e90b2013ac78109fcf6d6130c6d3b5d11fd80a975", "output_wav_sha256": "b25cad7f3e39fd9a74cc42457bc9fe48bd56436bbcd294b3ba969a162522908f", "output_pcm_sha256": "ccd108aca007e6372173500367c28cbd378b56e0352d8f9cc19f5fdc3e728416", "result_sha256": "7e0044b40a4ce624683d0922781d1289c276c3279366aeaa557c187a261fccfc" }
#657object_counting heldout	4	The answer is 4.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "25576c1a3590c7e48ac1e3f6511b2d76607aa547e4a7ac215738402937a45556", "started_sha256": "3b7c4293cc17af741544852689e7b10ed4e86e087a0147c8456d3ed2379f728f", "original_grade_sha256": "1ad289a8a9e5d4f36a6f73a40675e9ccc73d4cf6274485b8e134cf18beecef6a", "input_original_sha256": "25576c1a3590c7e48ac1e3f6511b2d76607aa547e4a7ac215738402937a45556", "input_wav_sha256": "c8f4b034deca456a22fbb26bbf6d4fb991a804111eaba773b834ca0731e57e7e", "input_pcm_sha256": "5d845221a892d26534951787879bb8bdaa29883c205ad1461c3c3300aecffa8b", "output_wav_sha256": "a5154306477b4251b764c997bcf5c8e39e1f988a46284d2bc408a2afe88e2559", "output_pcm_sha256": "a967cfded0f90fcbae7b50d7fa122e9dc48c259aa2176d49fd4ac8621c87bf3f", "result_sha256": "e1578f69cf03c36783bcaab25310268047c834c0cc02b450c8dcb6590c44640d" }
#901web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "1e03040357b38815f8443d81b6a8d33b2575ba53676b07fd8daac7aacc7291f7", "started_sha256": "94963946bb004350de6804bfbc65fe2feee205549672cd04034a1ae6b9bdfbd1", "original_grade_sha256": "f470ad4e630ab9b95da6e69212db5e1f0f8eff0a5a2081d48b4c6f954b491d71", "input_original_sha256": "1e03040357b38815f8443d81b6a8d33b2575ba53676b07fd8daac7aacc7291f7", "input_wav_sha256": "bd99c21c53cdc922b1ea8410ae27b6e33353aa4b8facb7284b72c12141983749", "input_pcm_sha256": "61513458dfa1dec335ba2858568cb1d7b26fa886e141957a5b458ea658b6b92a", "output_wav_sha256": "bb22b9bd0d606eb2c1c4b915550f809fb784efb94c3fcf0778ee5d1d4864520d", "output_pcm_sha256": "7e6da6072f9ca31a630121798021eb329b2d3f8f8de56997cfdb77ccf2d7c759", "result_sha256": "9d2a182c772d19bf83403422d39c51b43914b760f1629ce26f6297ba8ca29c90" }
#61formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "e05f539c1edd7cfb08da6ae3642665e1f87b8f645c95a98250a06306b3b45419", "started_sha256": "470181c44e6321a266d66943fcd4cab9261ae491b9095430e414e517589345a5", "original_grade_sha256": "91db22a1dd7871a66eeb0f94a109e6375bd947df12dab33d2322e5fd725a6778", "input_original_sha256": "e05f539c1edd7cfb08da6ae3642665e1f87b8f645c95a98250a06306b3b45419", "input_wav_sha256": "1815deb7f985e1f0464d76af504b29edfb608ff037bdb5be000aed7e5f629d61", "input_pcm_sha256": "77f742d2308073266064c850d33973a56c22428a2b8487f8f9ce922e29ebd435", "output_wav_sha256": "a2b99a11046127425132a2ab6e532ffdc365b664e6846fb3751f1bf4f52ee009", "output_pcm_sha256": "dd8752d7144ed3f97e2d0167bb5e09fc7efa278673e49cd342690eba1c397776", "result_sha256": "10f118facb612f714941388078ccbbb3f0d4fbdb4f0bc4c3f587549dd1fc92b8" }
#256navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "cd230f93f7e28922c0fe1b3a37797d86181c29590480d39e9d03e77ccb2bb0a1", "started_sha256": "098a717b3a8a7da6bf590d972f84474086372a67289d9a7b68382f669f0e83d2", "original_grade_sha256": "54f1e949d4b8c255fb54040ac577bf7c57ebe9f4f294d7a481ea631e3862e802", "input_original_sha256": "cd230f93f7e28922c0fe1b3a37797d86181c29590480d39e9d03e77ccb2bb0a1", "input_wav_sha256": "aeeda92b109bfc4ed7c5906490e541e7ac74f450bcf5e235c1ba259c736b5e0e", "input_pcm_sha256": "a50260d572a9fafe33948bbd546e9ab905247ad1ef19bd328945f98deea2644b", "output_wav_sha256": "04a2674748e30ec59f4690b932f38bedbb6cdeefb0acd1a3378e206c53d576dd", "output_pcm_sha256": "7d5727cf9b2e53bca9e22a49ef84e8f9c5f3931ec52525ea0c6dfff4dfa411c6", "result_sha256": "bfc6d6c3fb44c40a21676cd3f3119361777a6fbf061e36a68f275077688ca5cb" }
#538object_counting heldout	3	Date answer is 3.	needs reviewnot_an_unambiguous_short_answer	correctresult	Additive review The independently recognized received audio reads 'Date answer is 3.'; the model's reply was 'The answer is 3.'. The substitution is confined to the fixed lead-in phrase ("The answer is"), which the pinned small.en output recognizer renders as a similar-sounding opener; the answer token itself is intact and equals the pinned key '3'. The scorer withheld an automatic pass only because the spoken form was not the contracted phrase. Same class as the determiner substitutions reviewed on 2026-09-15. This is saved audio/ASR evidence review, not a claim of human listening. DAS test duty independent ASR/evidence review · output_audio_and_ground_truth Evidence hashes { "original_audio_sha256": "b1301cba385d0b28ad6278ebb0aa710b67f7bbc26e161e1b529727a7f95a9688", "started_sha256": "6e8f86d9e84f738bd98bb376438b15050bb5ea5a47a6d6447b46319c99ca2887", "original_grade_sha256": "4f1d788350cd3fa998d9435019e1e462d2fac239cdb3c3be8f3343fd5348e2e0", "input_original_sha256": "b1301cba385d0b28ad6278ebb0aa710b67f7bbc26e161e1b529727a7f95a9688", "input_wav_sha256": "c827596e99fdcbbec12c9ede17baa24593738c30859f77cbaf15aef8f820f99b", "input_pcm_sha256": "bde51eeb8123de91e6b2e2cc6b46354703b2fadada83538439e9ea5957e1902a", "output_wav_sha256": "55018424673a79a25b54e7e6e708bac2792103d69f99dde40bc9847ae31e298e", "output_pcm_sha256": "3e7b0e18bf9c761962439a856fbe06e7199dad3d622882b4e84dd05b9db06555", "result_sha256": "5c1eaf0f8543a21540a59dd2deca98417333cbcefde6a82a9c1f847a12f1daca" }
#865web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "81a2c26fe2e29c9bbd68c7bf12fe01287bc62872555e14b227b2c423a0d09290", "started_sha256": "9c858d84e3d96168503e6adc53009ba0af08b7b21d9c9b915b3fe2b8e94d1cf2", "original_grade_sha256": "377d26348e1e4666ea203e64fe126bcc156811ac2e0e72ea09b7f573e1896bcb", "input_original_sha256": "81a2c26fe2e29c9bbd68c7bf12fe01287bc62872555e14b227b2c423a0d09290", "input_wav_sha256": "8b0b93d41dcd9acadf953d87dc14da35f45b07c29260f92f4f2dde0a3bb27fb4", "input_pcm_sha256": "7c3fee6154bbb72cec822f41e5dcf21dcb5308b547b0c03f70faefaba3d98d7c", "output_wav_sha256": "3c63d5bdc18c8c2b449ee772a0a2fb5b5ce9d50e6aae43327693fca8a810c113", "output_pcm_sha256": "1588e8ae9854214b58ede02014a8c82c166948d05a5a3ea5a928b97574e948ff", "result_sha256": "5c32161a4cb0f07bfa0caf711a803c233e6f1738a89656b58c5af9fc49d93163" }
#225formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "a0cbfd124519ae9e986c6541f5f16b0230d0ee39632360106a2b99e6d8827170", "started_sha256": "d447f736c10274d2f8e86053e68081f492ea4938b8dc25b5a4d50d560c01caf7", "original_grade_sha256": "6bf3f335a526f2c2e0c01ef25ff932c4c019653850cd1e5c30543bd2ae096856", "input_original_sha256": "a0cbfd124519ae9e986c6541f5f16b0230d0ee39632360106a2b99e6d8827170", "input_wav_sha256": "b2f41002d22f21fe9833e58fe2dd2468f43430696186ed679df3a19958f74b48", "input_pcm_sha256": "e99dcad490840363a6bd2b8cbc24ba82bb4910829abc60e88665c685fd94df53", "output_wav_sha256": "949d652077734b1b6ae2088adffc6e0ddd536d07cfe727ff0d936a6ee14b403e", "output_pcm_sha256": "a4d14bc722c7dfa64022ee8a211fc5dcb471f522c79e1ab133918529309409bc", "result_sha256": "01630d928d25d43bdcdbde087cc89a7a10606ecc26a9c4a64c5896430b79d2a9" }
#297navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "fdc123c0871aec139d85b0fef2e39024b30f27e14a251df0c893d249ab5cca62", "started_sha256": "3c443d0d0a26b5cf9cf40a61a0a9f2af0f13fcc1e5facebb9d1b8c9e4f0a9c09", "original_grade_sha256": "8d8e425e34a166cc3e76ccec78538466757df1a043f0008abf3e8e3f33f8cd75", "input_original_sha256": "fdc123c0871aec139d85b0fef2e39024b30f27e14a251df0c893d249ab5cca62", "input_wav_sha256": "627ef2fccaab33ba392a88b71fa813e3f5a94fa241e04b695c88fb735aff5d9b", "input_pcm_sha256": "ef2877ee36166fddbced24178fe1241d37e554a8d20094bd11f4e6e569514748", "output_wav_sha256": "4cb655b21babcf3d0e0bf56c6e50f2e28c9f2b3460dbcb2810caa4ee4fdc7b2d", "output_pcm_sha256": "7958bca478cc978a03cac0d2442e7c1b5e02aa21546fbea9b7115fe58604364e", "result_sha256": "36b71a3d04e4bcd2447f6d7c83ca9f88a555a3fcf0077edd5b16e21ef7acfc77" }
#633object_counting heldout	2	The answer is 2.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "dfa83b1df95c250bdea0a37e8621cd1e609fac2d64a34365b54bbd9d2033c109", "started_sha256": "ab6ed2b3199075820a0c3158649e9c1596f885a91d4882a2f8c6c8ad42a61597", "original_grade_sha256": "6875dc9a4267cbdea64de7dcb85ec95aaa372233684d52a480143964aee0a4a5", "input_original_sha256": "dfa83b1df95c250bdea0a37e8621cd1e609fac2d64a34365b54bbd9d2033c109", "input_wav_sha256": "da0e48698e83c65c4982fe0eb393d9a84e93c24a0b24d4d47cb00fa6437c088a", "input_pcm_sha256": "b0fc62772728e980f3d0417e38cac06340dbc9b08ffd0b991202c190f557dbc5", "output_wav_sha256": "2804a8239f37e0043ff4ae01277026a7af1bebc35be98f79b6652dff68ff730a", "output_pcm_sha256": "30e5d6cf5aa528498158533e38712d3fb92f19c9b50e520e29c7f041f47ee574", "result_sha256": "2b407b66fa211f536bc241ee5b93ef7e185cfe595f8b320d7f4b09d3df34cef8" }
#906web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "25cd31c57e7a6cf45716bb1cd1e3091938cde0475a41feca93f9a9f0612d9da9", "started_sha256": "f0e056ec240936fd756c60b2b6d29f80908ff634ba630e0c15ca0569d1bb3568", "original_grade_sha256": "05d1c27ef2dc81608153b924672e6c4b796cbc59a2842b1d2ea2d68e61e102c2", "input_original_sha256": "25cd31c57e7a6cf45716bb1cd1e3091938cde0475a41feca93f9a9f0612d9da9", "input_wav_sha256": "73eaa9580ad149b33a74016e93c5f537c38a1546bb17ffb6ae86643634adc9f2", "input_pcm_sha256": "af0133f8b45453a9c94fff9c63ce5283d774a43bc5e1b42a1ed04d32ea224361", "output_wav_sha256": "6b41e5be5da5ca194409f26756de03262ef3787e3d10c51d6cde7868b127c37e", "output_pcm_sha256": "f9d98b1842591eb96345d35b66e2615650e20406df43f5fb54b8feb0ce0aebeb", "result_sha256": "da88d3094126ee1653c7a09f9f56b202210d156da57f80a8e2b23c09bc46edae" }
#236formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "39a77f22875e05a25b170bff3f0f28a87f113ee61703f0ffb31da4411d9e5010", "started_sha256": "2805c8f1e233a5cec90c93f4528e0aabc2c2c0e9bf5eec5c62b300bb0f6a590e", "original_grade_sha256": "0081516a73e5373108e56cf00a48483824fa937c2f80b7a8cda27f2786af6ac6", "input_original_sha256": "39a77f22875e05a25b170bff3f0f28a87f113ee61703f0ffb31da4411d9e5010", "input_wav_sha256": "5fc51af9c79bec628b6b3c887c2457650380964db20504f3efca2c32a5b13584", "input_pcm_sha256": "78d8f1eb0164adf1d3bce393d9b24748fa3cc508c0bbd388d3631cb79c608306", "output_wav_sha256": "6fd5fb0da425e5452edf3a71de987d1db7af4516efeccca05e8eb7e3c2e77a43", "output_pcm_sha256": "083e5886dbda3bf94482e076c136d07b6673748602354d68ba5b12989b6c8471", "result_sha256": "9c11b0fbc47ba682ba32cb4ef391f10c91e523c39c44b48db73943f42ebfe837" }
#306navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b059aecfae68c4443d7eeb3e7f0178bf0fe80d78437eec9201d808aa3443962f", "started_sha256": "bb213b1d9bf7f9ba438714a379fdae423e7910091e6acb92aee38331c6d46a5e", "original_grade_sha256": "3ff333a86b3af62ba8d8217666130cfd28bf4a9605d3cb4b8a421cf8bc48b979", "input_original_sha256": "b059aecfae68c4443d7eeb3e7f0178bf0fe80d78437eec9201d808aa3443962f", "input_wav_sha256": "7f4f57fca51c4cf88385aa61eb00ddb5b336ada6e898168af1d8cdf5403600e6", "input_pcm_sha256": "3b2d21b5bfbaf633248623acd9473e81a8e3225d446c5e00086c63c831d27cc8", "output_wav_sha256": "c9630daa22e2afe50dc399d48da46b2b1363f58dcb23d1bdbba2e485d4d3e725", "output_pcm_sha256": "86fabbbd0ac09ccaf9e305ea80a8ed86de931ad5ec457bddf06f63a0b05e514b", "result_sha256": "a71a9b3f0486095b48f6149d4d0a298765f2a5545a1a3e71419f08b4b67480b8" }
#687object_counting heldout	3	That answer is 3.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "27d3029a69b90af7a3ce666397658ff27f120086dc887ba0b7b95986a4c3b9b9", "started_sha256": "4a77cb733870166de7543c0eef125207f3b2fd08fe731d2d166b398fe90dbaa0", "original_grade_sha256": "78dcaf483b5c8a79e55c7a57977714b0d440375b343db04a1ad058ce95fd0f15", "input_original_sha256": "27d3029a69b90af7a3ce666397658ff27f120086dc887ba0b7b95986a4c3b9b9", "input_wav_sha256": "6eb76a1bdcbc2b2de90f59b9be606e103db5a7eba32d4508340bb02f2c1e7265", "input_pcm_sha256": "a6219c4fe2ac0f922ed335793cb043844ace78333a56da397bdf9c1ce7851e7d", "output_wav_sha256": "069cb0db206297bf4a2c1c8f2794b2d7c942a95e6ebb5ec97b4c4a21ba7456ec", "output_pcm_sha256": "56fcabb97a3fb4df7a6fa69d1a96623c0c41f3e22904b184d50b7c560e5132f3", "result_sha256": "f870dbfd2d2bba50675a1aac4b0c5031496dc86024a48b3019026455b697dda4" }
#850web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "19f791d5caf52c371871435ca8410558a93ab12adc7b9a40408adae3aaa41e09", "started_sha256": "e61ae503ccf2134a1a9e9083a98e6cbdd0b1922ffc0c85d92f522c63b51a1d2f", "original_grade_sha256": "5ac2a3fbb52a41f73cb8556f9bcf6ea6808fccd64a16fdc6443953eff2f71f9c", "input_original_sha256": "19f791d5caf52c371871435ca8410558a93ab12adc7b9a40408adae3aaa41e09", "input_wav_sha256": "98ddbd0d910e8c1fe2a4e684e98148884380ed33b6326ed108cc239f778ee51a", "input_pcm_sha256": "88159f7e6c295036531710424a8b7642d2f3a9823de3a641b2b5e616d7b02480", "output_wav_sha256": "cf476566895714b6c8a83b6e5fd50a5575fd073f56d58aea6597790df49e6694", "output_pcm_sha256": "9f02c49e1d98417848d38b6b8663eadc3f31ea0e2fb12128cf6c2010a2f095b5", "result_sha256": "007b4d2e22db1c36b5c16f61880d6e129c749cb5ce560294ce65a3a13084838a" }
#55formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "c560d8a3553b5a8956559866251a07d66ca74aa59fec96623fe7c68832bd392d", "started_sha256": "4827711f80e33c00e1cd8a16c5e4d3f3dbd5389fe35a33b42f8954e34f9805d7", "original_grade_sha256": "4b096d3dc19c6a58e229588e5fa2a697b8bfa43c56b9fd72c9370d553aae23e6", "input_original_sha256": "c560d8a3553b5a8956559866251a07d66ca74aa59fec96623fe7c68832bd392d", "input_wav_sha256": "2ff9309c63363fb1952763ea7a27fda6d5d61766057b9c11e9167d1c40b75d55", "input_pcm_sha256": "ac6f2de9919c0f57dc43b800d8301fbf3b43db4e182eeb93291455990af75eb5", "output_wav_sha256": "a8ff5a2cdd03353b65e3f1ff2f23c68f454fdd3ac4bb736b5069164910131d0a", "output_pcm_sha256": "50574d2b12933477cc046ff9c88e7d778594c2bb76f5e31c409d885d98b05bdd", "result_sha256": "a6a021568db5cf20ad61398bdf57433a426161fbfb9202044be0bf73077ee653" }
#398navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "1e7d5862b6f62b72f883eb58305fada1ef44241f95929161bcd7ed0da94fb49b", "started_sha256": "51ed402a4645c9435d4fbef533c33274e5546a32274438045154955c72165272", "original_grade_sha256": "0aa534e237eaf4e77e79443309c3223d8ced24ade2acbbc5d3f0fec4f3cd3e9a", "input_original_sha256": "1e7d5862b6f62b72f883eb58305fada1ef44241f95929161bcd7ed0da94fb49b", "input_wav_sha256": "5b91c5bbf23a4b3e13e6f959eaa55ca57588ff93cd6e1bea35727663d905d3c9", "input_pcm_sha256": "625704996560000afa8a3ac6e894c595bb600a20c2d106a9e13e5453c48bb952", "output_wav_sha256": "60bb639ea5c2440c402c5744603ad4c3796332bdafc6aefab0b722a1bc15eee9", "output_pcm_sha256": "21a1f62602cb904a5a82bdfee2a3893a8c7b4f4f753212ae5b219183a0ab6994", "result_sha256": "a2ad0f77581b3c8e6c95eb3c52a5af2abeac06a56df15ec6e1113c025867a3e9" }
#610object_counting heldout	5	The answer is 5.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "23ae2873622bff43a5b63247b96318867fdf75f305dc2898b6fcc4063126b7ff", "started_sha256": "15ac14630434293118bfd907b18c8346f5b16f4afb67fdd73a7347f624c13d1b", "original_grade_sha256": "9cd9070a21d60b13fda301e844e1b5a8bcb7a3511e461a1784525f2d5ea0d941", "input_original_sha256": "23ae2873622bff43a5b63247b96318867fdf75f305dc2898b6fcc4063126b7ff", "input_wav_sha256": "b30b1588da6eb5b228600400f842315b01658fcdaa5cf0696f316b99755e0167", "input_pcm_sha256": "10c210ed423e18b8ef4b5d9c2f0186d9302f5a339ffb87504327cb9238f1b58c", "output_wav_sha256": "c97409e14d955e4c8489670e9d8a7da33ae60286754cb7706a7080c11cce8134", "output_pcm_sha256": "c3b66f8282040998471ab7833185b245b83cb5bc9c31a097384119c6e93b3855", "result_sha256": "94580d2fde6a6c1f887c3d313d296f69cd10f2085247ccc3cfca076035774f6d" }
#819web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "bd9086312fadc4243aeac67523f68243e8f6e2d58bb559b4fa2555187a942578", "started_sha256": "57ffd086c4a4505e7a05cbe3c80fb4eb22558f17cd82eebed0e873f1fbbd49f1", "original_grade_sha256": "28b88fe3a76a95c014e0f85c78e19415c6a64b7f82ed67283546f9052d8529aa", "input_original_sha256": "bd9086312fadc4243aeac67523f68243e8f6e2d58bb559b4fa2555187a942578", "input_wav_sha256": "bdbda9383710a00010d2d6f4f97706c69e08dd778d4ce0ada5eb43877753c40d", "input_pcm_sha256": "d79e48d6cb2394d142385a0bf1384f568706f4ea0beb1c93e7bcab7de49fe6f8", "output_wav_sha256": "948530c05d71fa217e2266c3238fb67e94c8aa9cdeb68edbe034e78b556a3604", "output_pcm_sha256": "a876c386a0986d61f61cfc51704622cd41d2a006ed9c37a6ed8dd692676ef0f6", "result_sha256": "655847458d6850cf31bb9b189f40dbd5bcd44cfcdc3a69550eab889671946fcb" }
#150formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "dccff0f3f85f1f80749fbfde1837093350b0bac3721b2d9dc2838dd3316918b9", "started_sha256": "aac729c6df1fc779e3ae853ddaafca734294bcf61234c5ca79e8afeb93639248", "original_grade_sha256": "0b5b07a0c115ac95b57331db2f6ab24e7c4714133f6fddfd774995e2b77253c0", "input_original_sha256": "dccff0f3f85f1f80749fbfde1837093350b0bac3721b2d9dc2838dd3316918b9", "input_wav_sha256": "2c399a583241f05d92724eea2eee65f705e901769d342343df3b1200e8eff195", "input_pcm_sha256": "91d27b44282a4517d42c9cb1bfa9f31da7705958b2557a25875a214cb2903a01", "output_wav_sha256": "da61c8d6b657a577d7c819465b08a8575f3f037b80275fcf37cf47ed281d7fb5", "output_pcm_sha256": "d0060148ca42976fa1f76c1f43f07694f54819d0979a6f33d68af082bd36943f", "result_sha256": "8041bcd7195b4383d80f911cd6a56d77a7e35e620b92152743634c87a72d57b7" }
#307navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "40b0db0dea2a243623511edd3a5a5d5734a60ac9702c3bbe45300843a37650d5", "started_sha256": "6b383751c8186530f56607e2a236bbc5f5e04f723e74b60fc235555739638cdd", "original_grade_sha256": "77bd521d7e4bafedca3ad9e31a8e16f461889a1798d25b1c866efd6c5d5221b5", "input_original_sha256": "40b0db0dea2a243623511edd3a5a5d5734a60ac9702c3bbe45300843a37650d5", "input_wav_sha256": "ed787c6f129a130d0815c0745331d2776502b4188e604611117135e09fd8cd2c", "input_pcm_sha256": "25cbccd30d2aed9618f770dcebd0e1f10375cba3770cfa596f23e8e1fad708a4", "output_wav_sha256": "9fe9f82172be381e87428e1f282c2a7c4f9a6dccc47cb5e88e93de5c12221017", "output_pcm_sha256": "f92a992fa27bebb337f7c81f00334c9b7afae6ca3c5dcbc9c9eec74ba36008dd", "result_sha256": "56f4069ce212587285932ffd19d119ab60da01cb72ad724c74c2ddd2c14aba77" }
#506object_counting heldout	2	The answer is 2.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "2ca6388a2051f9e18c1ceca826ce5e8be28c28ec0793cf2df52aff1894d2f13a", "started_sha256": "6eada542c2db06cccb9c0bacac297f33ae1af776bf02d258fe91190ac04c93d5", "original_grade_sha256": "77a94bf1e652c6b0d0d3d9d8f8dc4bdc26cd4856c9644a35cefa238616d479b0", "input_original_sha256": "2ca6388a2051f9e18c1ceca826ce5e8be28c28ec0793cf2df52aff1894d2f13a", "input_wav_sha256": "b625c546dd16e5fd13440f17c58f8d60e30d690c1a00b04c019b2063c0608832", "input_pcm_sha256": "807d0757a7b644ff9308495b881f26babcef075f4c22a95420b4c73ce0576bec", "output_wav_sha256": "e8520b95859803367c434c6c48741a8b7a38daa02285c862e236e272cb314ea6", "output_pcm_sha256": "0cfbb6f3b3f2c1333e52cbc450ce355f797560814b0d16692ee305b3e5a56817", "result_sha256": "3cb15b6ce257798856d2f6854b329ce444e170097072aba6007f2b3c268316c7" }
#983web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "174d8593938a5472f0d09be444f6ffc32489fc8b47f81286507789a25e840ba7", "started_sha256": "1cb891c1e84e7c7876e877bf501d71b383bf695acc8ccd3815f9cde6e4eb5184", "original_grade_sha256": "974a84c30f4e4bc60ffebb64e5efbd6b2f78c03f7dcbea9959a110737d571791", "input_original_sha256": "174d8593938a5472f0d09be444f6ffc32489fc8b47f81286507789a25e840ba7", "input_wav_sha256": "8ef8fc02cb97e29e0a6862a6add57912929d042942f774eddc57474c714eb2de", "input_pcm_sha256": "a158acc25f7b4ba7875fc7ca792f980e78a3c0e813799fb2981169a27f8871e1", "output_wav_sha256": "4e71a8f2facb428f2fb44f20173c80823ea1c884b2e3dfdf45d9d10d629652dc", "output_pcm_sha256": "c9574fa127682ba20395585148fd2c358e208833d9f99db043589540e51932e6", "result_sha256": "0c0a78355b68db9d522e128defd5e4602a3e7fdf5cd0c54775de028ebe7792b0" }
#176formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "31ebf59db9677bd59023c8b98aa3a9d3254583ec7b9751fb1b89a0d127e57763", "started_sha256": "f2be07b756d80c414f07e58102eb00e6a6124ae226e3fbbb9263ac57bb8e0b15", "original_grade_sha256": "5502f7d28e0f74fb665f5c79be42b41c586db559c6e01b6af90bb5af76b0d10b", "input_original_sha256": "31ebf59db9677bd59023c8b98aa3a9d3254583ec7b9751fb1b89a0d127e57763", "input_wav_sha256": "13a50591fa8ed24b2e1bf3bd877c602696495c2fd6ab1cae6cd6fdc4fcf593f0", "input_pcm_sha256": "12814a63548d3780b16e10a5c89da72f42351d8870d26211e4b9727d6ff4fe2e", "output_wav_sha256": "241bc92bfe21d45126e0ab9c9a4dd67741ef0c2c15d4362ffe4f264a55b46604", "output_pcm_sha256": "0eddbda3878b0efb24720c041b83b396f5d5e12eaec52ed3127de5a1423ddc06", "result_sha256": "17d583f84e03fd658e349bff84ce2a526dd9b9d4934ccf678e6d04b1ac72fc84" }
#425navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "022034288afbf9fbc7d85a7e8d2d17af851113730b28cc0604c25a4f5c1718fb", "started_sha256": "ed15770c7f6d87a3a02a29a88c429929ba3f098f9a7420e882f1f278b259f6ac", "original_grade_sha256": "e8638e06de6b95fb11d354dc9ea66f7f126bfaca97cea0523116e00cfa24147c", "input_original_sha256": "022034288afbf9fbc7d85a7e8d2d17af851113730b28cc0604c25a4f5c1718fb", "input_wav_sha256": "94d65c2209e8d69c3fea5b61a8b2d96f662f962dd39c4d4848d1eb946ffaafec", "input_pcm_sha256": "5c4e8ee2e64ba90408debc7c793d0ae072b0c44272934ede785331d50aa0abc0", "output_wav_sha256": "42f006e0e4c367c821ce04d6eccebe807067583aaa1bd362e0da9acb2213f694", "output_pcm_sha256": "1d513aa87cd10c31fda296790122b85a44f746a8d1024091c4d2f1407ff1e32a", "result_sha256": "6950b0ac35738b83534eafefdbfc6dfa60f9bbe2ed01e5653ea55cdffcce7564" }
#707object_counting heldout	4	The answer is 4.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "9cfd6d6c68dcb23e823fceb4d389c409ff2cb15e3275ceb7ddd50e1f3c92e044", "started_sha256": "7b7b8d98376449a55d4f7058d98459529f9bf2ac1b8058d44cf2f12bd1a1ceed", "original_grade_sha256": "af554a6d3b1644ddb6030f1f0880f1cc8aafc21fd85f99ede70e881db6de8e01", "input_original_sha256": "9cfd6d6c68dcb23e823fceb4d389c409ff2cb15e3275ceb7ddd50e1f3c92e044", "input_wav_sha256": "329a6168007f1302fc354cd55dd4b9c433aa8bd878b566d1895bf869a7776c2b", "input_pcm_sha256": "8aa78cefcb165ddb7f31f224a68039ca6899a8af344cd70e9f55f629b8e70b02", "output_wav_sha256": "175cfd49d16c37ace2a94d524023538a935e6e143496524f7aa413a1b7b7f924", "output_pcm_sha256": "32e73b52f30a7488abeefe29cd9e64ce4ec7d5c8b097bcde2b36152110fc9299", "result_sha256": "0c96cb0c4229c744f1f71d74a0598aca3370e71395e6600b4749c93badd9ba54" }
#824web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "2a8f6fb9c399916404d6d49e822321ad38f1a85529906ce1fa2deb5f131f81a4", "started_sha256": "11f9d918acbc7e97cdec3198f6865fa8b999cd52ac6107ac2a2bec919e742a30", "original_grade_sha256": "e210ebe4fd465ed1c965e2b185aa2cc6bb040b8e2778aa021ab77d42e6e61add", "input_original_sha256": "2a8f6fb9c399916404d6d49e822321ad38f1a85529906ce1fa2deb5f131f81a4", "input_wav_sha256": "33fa2d2498f179a0fe693f82dbbee890979c6f94e8b33a2b861f157af38f279b", "input_pcm_sha256": "af87aca4e73fa8ef54afd3ff30bcfd9d8c5d9e129989821a28683b381d520a76", "output_wav_sha256": "5b1997263433538186ad3c9c25043cba2900e6d7e75933210962b53f925810b2", "output_pcm_sha256": "0b2d29a9346dba7ce4308b7d8d14be3e579f948e5d0f649a41111a47473ea97a", "result_sha256": "459e2469ca748d069541c96d82b2c26020f35bd22ea3920cb7099e92a6dec764" }
#177formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "6f3e1254ad6795847d484a56114f90f334ae2dbbc5437a8aaf54c6f8ede27581", "started_sha256": "90164e3a34a84470381fa1cb78f8335e91a51fc3cf1b8e9fde90137c964bf4e8", "original_grade_sha256": "12ff88da96e61d879aa996b5aff39acb995cfcbaa8f766129592f41b0b39c437", "input_original_sha256": "6f3e1254ad6795847d484a56114f90f334ae2dbbc5437a8aaf54c6f8ede27581", "input_wav_sha256": "9dc8c30bd291c53fbb8364a684c3c92bc4dca150b032d8e46c0a822ab740c9cb", "input_pcm_sha256": "3cb31485574f99d3558fef4eff441de888a48c54f5027151e8a453848e44f7b5", "output_wav_sha256": "4afd09fe9aa804cd990764e32ec601e4ebad7d1c3cbea75e7ef79bc45119f2e1", "output_pcm_sha256": "c40b02dda3d47aeae82f0721a6f74a68d0e092a0c7274ce184b39a395b2dfb62", "result_sha256": "0c6b9bd56d2eedcf1a9cb1d74fa766320853e4faebcdc6bdf53bc343deb2dcab" }
#302navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "3ed8d4770d39adbe834649a2d504de11afc8e84ca5d6afb8ff96edf50c3a0e5c", "started_sha256": "9d0edafdb52a6c9ccf5157890cef020d8bdbdce728bc6aaa8546e4c6dbad4f53", "original_grade_sha256": "f8f6a9f5af7bec63cbfbbef284a4bd9ac860f176c1193a8363488cf806602c30", "input_original_sha256": "3ed8d4770d39adbe834649a2d504de11afc8e84ca5d6afb8ff96edf50c3a0e5c", "input_wav_sha256": "3e3411221bea76e704ad13fa9c025a49b63ccf78ac073b4eda4df4dbc1e76276", "input_pcm_sha256": "1b384e130646db4ac7d554710c6edda73a6e20d8bf3f71b083b047de27c97fa2", "output_wav_sha256": "281c24d6b78bc3fae8b168cb2ee183ea9079ad9a6a55929e154f20652b8ea53b", "output_pcm_sha256": "d5cf961948b98bf74638737d695779b26ff4744d5a089d0bdb54e53cd51ba3fe", "result_sha256": "b73eb1aff67b7ef5ae4dcdc6f4a48d2487a0a3085394c546a90cf913fcc447b4" }
#696object_counting heldout	5	The answer is 5.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "ae8147d7feff4d77d8ad3f7316ee16e75a6d59fdca5528c7150650cab19ba6f8", "started_sha256": "d725c78240eabf03c822ff73ba57a07fc976e908c34e55811e8b7557d00ce80a", "original_grade_sha256": "b6cfdc4ac75b582755a9b75bd6f557939dd49ceaaeb662f4427d8c6785616336", "input_original_sha256": "ae8147d7feff4d77d8ad3f7316ee16e75a6d59fdca5528c7150650cab19ba6f8", "input_wav_sha256": "fec17cea72ef13cc8f183b86e9f0e9d80875eb3870d6ac11d8d531594b265fb6", "input_pcm_sha256": "dc742de6769068d67087c038d4af5fdda670a8d32ec43588e54e3b00e6c58c78", "output_wav_sha256": "09abc388ec3da3e6eec2a802cdd99b792a3a7fda00f17e716251b3fdd7f1a102", "output_pcm_sha256": "5bb6a4a84d920cfe2ee1a83b5320f666b3bc1ba8b7cb8dd27940683d6dfd50f8", "result_sha256": "d72be164fcc753719e9e2cceca2ebcdbc18285625bd21c8d1ed86a538c79d352" }
#787web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "d6c223c7fc376bb454ed72b272dda58c4bdde9018e198be5b9ec3f037386bdf9", "started_sha256": "407aae520a967029d959f3029476ae2f19c07981ffb853919dbc8aca7209bdf6", "original_grade_sha256": "c351501f88766399b3a1d445f027c5d7d78b99fa3704ed5bb2510e8ae500d598", "input_original_sha256": "d6c223c7fc376bb454ed72b272dda58c4bdde9018e198be5b9ec3f037386bdf9", "input_wav_sha256": "aee3201bdfb773d5e2d8ff8edbb437140748a491c7f4d3b415e6966eeb4803df", "input_pcm_sha256": "a38c4afab9adb06981de0eecf476c8fd97e75c824cdb0b1763f2e0ee1a04edbb", "output_wav_sha256": "bb02510d2baa1da393c1505842db2018dbc58b716858b8b34e5c9bfdbcf3d99e", "output_pcm_sha256": "fad4f58f454a26fb5e13553589d4a595ad5d18883fc6d971d98ac4eb311f4b9a", "result_sha256": "090bf701970eebf5771119ad060694af6a1cb08f6ede6d25a5fc4e8e87eca1a6" }
#178formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "04ed6e565f77dd04172f15f613e5dfd912dba701faf2739b6efaaa99067aafa4", "started_sha256": "4ae74a56ec358779bfaa801d416bc19676cd93ff68f71d26990a2a08e6526bd8", "original_grade_sha256": "c551bdb050c617fe9bf52079a21f9867a500e2d3d45fa8fad489e528dded5304", "input_original_sha256": "04ed6e565f77dd04172f15f613e5dfd912dba701faf2739b6efaaa99067aafa4", "input_wav_sha256": "cc738c0ccac685bdb2d0d1ea9e1d4c0f107e31b5484974f2453a6b250d152823", "input_pcm_sha256": "c879980d7dcfcae7b32ed7ebb20612912e646597839257d5c9c08d66548edf80", "output_wav_sha256": "b0651f82f9d2152663b57a5a69e58a56a9dfbecc37b5c3b669f09238990b2839", "output_pcm_sha256": "ae6744fee115cd97e89456abd509d14de2d6e0de445db11706003fd91e4c1a7d", "result_sha256": "6f8b2b8902b5d32ac1435379fd229cebf01a5e1fae0a0556a04987510d25e4e4" }
#290navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "db52fa29a8f3e91433b06b54408110408ae0cc0fce15586f9a84e66f7ab4846e", "started_sha256": "c8fe9cf79d3b84c0a9e550905df13c8e3bb47757177dee2d6497de3ca91029cd", "original_grade_sha256": "c5aa6eb96884af8955001e432f1d9c91be946a18d58286805ec20c0a8b12a62f", "input_original_sha256": "db52fa29a8f3e91433b06b54408110408ae0cc0fce15586f9a84e66f7ab4846e", "input_wav_sha256": "8071305d8de9ab0f150fcee7eb02943317895247a9d48639dbef4d3f5b7b66e9", "input_pcm_sha256": "18e1d8b29a85c57714b9aaefc72db27e6c18b446d916dab139cce02f8c2804a0", "output_wav_sha256": "574525564dfacbe7d29cd96b6a9982eb99cbcdfcdcbe7c2c3ac10d2da8c758dd", "output_pcm_sha256": "5016888a49d205ab0f65d5d8184414edaf9c060ca2d6afa1b2b5e103ad6dd4e8", "result_sha256": "f245f2e337d8970a610f8bea1d4d84029e51ab359cbc83eb3c44df5f81f281f4" }
#736object_counting heldout	6	The answer is 6.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "a073e0b1f1bb5f884b628b9f449a35003bb841a24a6b737b39438d518746f22f", "started_sha256": "477a2bc130df6690b0e1474694a3b00ca57931c7b246825341f60f45d054a573", "original_grade_sha256": "e5b776618d4b957d7d20ab0b64e0123dbf8fc186b7315fd36428cadea72fea22", "input_original_sha256": "a073e0b1f1bb5f884b628b9f449a35003bb841a24a6b737b39438d518746f22f", "input_wav_sha256": "8883c01f5c9932891040b523288c37c201db25209c0b74e44362ae81d08159aa", "input_pcm_sha256": "f406ee466af164c4e95017bf4f69545e1298812051e2d53bf75c796725315583", "output_wav_sha256": "ff6dfe17a8b1f39de7e4af638b7ccf2331189a0852a4d06d1a874679999c40fe", "output_pcm_sha256": "0c8db26afe972e4a4219b4b4ba0949d7282607db9bd182b523ca4d67fee83a1b", "result_sha256": "3a21182608586cca3d9d023a31a8ac841744ab140e6aafd20fb5873aef4e307f" }
#872web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f4615906395ee1dc6fdcb33dac1be954b211d32b7096132c8d110530d0d385d5", "started_sha256": "f07b03fe31ef5b1db5113e45ec32f7df8d3c6de7c41c3e5222464ecde83aab9c", "original_grade_sha256": "9c29ca06216685b9b4e1270869a9f5f06b25c88684018ee6808af88e0b2999f6", "input_original_sha256": "f4615906395ee1dc6fdcb33dac1be954b211d32b7096132c8d110530d0d385d5", "input_wav_sha256": "f607988b3e552a3eee9640307248a006139482e542ad380fd74446a226dafcba", "input_pcm_sha256": "8e740d9d59562ce61cb72644a366444d5e69297bd7065ac4ddd2c3e31571478b", "output_wav_sha256": "194d39f0c709a53dbe703e348782b68734f7f31c70e433f827366c93e8240fcf", "output_pcm_sha256": "5979acbacdfc964adbd9862e489d6102a83c83c82f0bf65d5f1044ce208d9408", "result_sha256": "cc1d039bd6337335ee428514935b4e0628925d05abdee32017740c57dcbadf4c" }
#110formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b856fda9ba3dd0ad56e44505460632b4843ca49c0e4e796e126f253c6b53fd12", "started_sha256": "56a3c5424013bfd5ea104a7be5c4e6a0221f29a912b6f0068607b043f762766e", "original_grade_sha256": "a3a4269d2c0eb6497963f5551eb4b792aa23a6a97c66cdc0a843d12e4446a366", "input_original_sha256": "b856fda9ba3dd0ad56e44505460632b4843ca49c0e4e796e126f253c6b53fd12", "input_wav_sha256": "6e19515b283048dc3516c1593c9976a37fb56e8356772eea920804ffd300d523", "input_pcm_sha256": "ba52869934f7571fdcda19c1ec596cfa7791442256353e3a4f080229393ed1be", "output_wav_sha256": "20bbd8931ba2b262176e645414bfff0495538abfdab26b8ff9ec5b23d8fe86b4", "output_pcm_sha256": "2072658c29603740558477c7a6bb3f55f617aa29c62592c3bfab66ecc4de162f", "result_sha256": "9cf1221e7d8d0bb9ddf57c4d0221147fc418e2b7739435aee52f3dd57d0ee731" }
#426navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "21d4626e298a6200ff0f61d51bae0cbb49d59afeb5ed86e40030fc56c9cd1479", "started_sha256": "8818ccb43ea05ebb0c23f1d9fc6189de86eafb4edfea2c0ec4d8fc834b637590", "original_grade_sha256": "6503aff4d5c36daccfba89d039bbc3a432d094de34eb53d139015d7c79c5a5e5", "input_original_sha256": "21d4626e298a6200ff0f61d51bae0cbb49d59afeb5ed86e40030fc56c9cd1479", "input_wav_sha256": "b5c54e82bf4fa2aae92ab1ef5b0580c62806de45dae488dabee2f7dd42b19745", "input_pcm_sha256": "2a4b62c8ca68787c895b9152f0e277fa604bc37d9c3a7ff8d868dda04b43d818", "output_wav_sha256": "61ae394f8e1bff58ffe2e59d289fddba3d6c28306e6717c392aa39820e3f7d3d", "output_pcm_sha256": "535c8a4c73ea7ad76d77ede61c87501ffb2f94f0b06dbb1ddd9087b89248990c", "result_sha256": "9f3f6f48a3c6c264be7b6fd1e99673ba5721720459604176a922a2a839e24a57" }
#679object_counting heldout	14	The answer is 14.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "df054e2c7947cc3bff49763fa6c7920184cf1ffa7de820d59593eb2cfea93ec1", "started_sha256": "2e892d1c912ed6fb51a1f7ed9052d2e5a29f26a63f4ac17cdddd4d813b9c271f", "original_grade_sha256": "4b4f496590f4a77f5d8fe6bab29e05a2eb77aba4d8563fe95a86fad094fd2b4a", "input_original_sha256": "df054e2c7947cc3bff49763fa6c7920184cf1ffa7de820d59593eb2cfea93ec1", "input_wav_sha256": "16e83b0c0b17a63022504006ba0f0e4129de5b405711bc630b17b659b83dc116", "input_pcm_sha256": "b1f6bcb21af81a1ee657b87dd9eaed5bcc8f0f9a0d86357418ea3ee7ab54c39c", "output_wav_sha256": "ff81623edc944e3194852d87d059ff4c91336ed7740633ae213c27b686c59965", "output_pcm_sha256": "7f3bd730543d8e94eac577a6edb69106678d0cb4182951904e4007b947d46122", "result_sha256": "86764e03c4f04ad122969e8e176a6c64ddbf28decf12070b0ab410ef8e845ea4" }
#849web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "4428479d471ff61f7d2ab8f09d6767a00a82fa532a80a0473f1f5860e1fbb8a0", "started_sha256": "db9ab3c0598ab7f2c1c6a8bf4f84f9fa67084c5b31868afdf326ac40de613937", "original_grade_sha256": "9823fc46187da0775a698b69a795bd753f35ebf720c0faeeeef8c438948692a2", "input_original_sha256": "4428479d471ff61f7d2ab8f09d6767a00a82fa532a80a0473f1f5860e1fbb8a0", "input_wav_sha256": "71c0627dbee7180c760acb6408fd27154026cc9358fdfbb597e198d12abf19e9", "input_pcm_sha256": "7bd38142e433c2bcf3b419d302bd56e33e42e2ec363851da42a30a897398670a", "output_wav_sha256": "c019d4c23a3abe35df61cb63b9e42a0f634d67470e70d4d4829d86b7b4081895", "output_pcm_sha256": "d733c0da1fcf4ce86a5c32de56b524c0dce7af4257783f28dba423fc8c6e9fd8", "result_sha256": "1c139e2b9fffd3c1d7917522142a31e3f91d72f36ad96a7f935ca0031e4545a1" }
#85formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "ad638705513844301223a482cd0dff9ce1dc430e4983ddccab4c02c978f87a03", "started_sha256": "4af27dc489be0d182ac1d01cc67023b6c7e89bced48aef2c1d27eb131533cd65", "original_grade_sha256": "9f5c5e75d28d9b991dc0e40558c51c147a1d02af9e53b5e3edd9a5cae8948b69", "input_original_sha256": "ad638705513844301223a482cd0dff9ce1dc430e4983ddccab4c02c978f87a03", "input_wav_sha256": "eb03e4e260f8d4f4aa91c663984572fbbbffadae1e2978b49a06527c0b94dd36", "input_pcm_sha256": "8e2820bb35403a0e7cec848323ffb66736b2f3dc672b251d73b1ddbac3f24623", "output_wav_sha256": "cbda8d70ab973a06b36c3360af3ec143891033a08d3f0ad8fe8f861595922423", "output_pcm_sha256": "0598b7215abc35a99a8e31e78eaa2b51ed697082bc105a671d02182b68789927", "result_sha256": "50589e1db232fdddcbc98d091bf50fed165a381d84b95644304a4b2f414884b8" }
#432navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "8f0bdf486f8b215a8a36094ee2c1d9ae580cd49c5b2611607b9f6759d480e99a", "started_sha256": "960d2c29a31813d1d5f29fd75d2cb529633d870687f19130f40c7937652a414d", "original_grade_sha256": "f4cf9105748ce8f215128f5ebe4408b2bcc4b2cb495249a2815e9e64aac603ae", "input_original_sha256": "8f0bdf486f8b215a8a36094ee2c1d9ae580cd49c5b2611607b9f6759d480e99a", "input_wav_sha256": "cdc8d507acf6e7862ea204f686dd289ab14136509dc73eb937c15e03b3b49d1c", "input_pcm_sha256": "9884dd7eb892f0c28417b425972d2475013e51d1f9903bad2fd1f151627136ef", "output_wav_sha256": "99cd5da078c443dcf94db1ffb8ee9d76ac3a573de329ea64c0790d3ee3fbcccc", "output_pcm_sha256": "b6903d41632734b21fa30a9fe83ce7b7552483309cb97fa7126162c3c0807c5b", "result_sha256": "6f8f1d4d8d8f5471e0455692018380f050b23776f6ce0f9c74e55a31742508e3" }
#570object_counting heldout	18	The answer is 18.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b8a3ff7ae6c340b5c95505919aacf2e335c77d1e4da73da1c284bc03f57355b3", "started_sha256": "d9ed431cef727757e68a9d34465c72ce5252c7cea8f6663632976d401fc5c6f7", "original_grade_sha256": "93fa89682500c76883fc37e0c79e748f68307fd42a65ccd0c1a3d2ffda144d17", "input_original_sha256": "b8a3ff7ae6c340b5c95505919aacf2e335c77d1e4da73da1c284bc03f57355b3", "input_wav_sha256": "8593429ad9c2a7aae93a0bc1e3f94f8e3e3a00b0b55975f54d07e76952748e91", "input_pcm_sha256": "80d962bec6bb07d2e0406697e244cad7337fef8ab5b9b703aceb427f56182b75", "output_wav_sha256": "c5f1bbb0edce8a8b854fd27c22722a6dff80cd9d30028826d1925abbd9abe3b2", "output_pcm_sha256": "0230d01f4598d697fbba1f096b7a577896e2a5d4f7e246cc78e3a44c8da4922e", "result_sha256": "e38e16ccdef6e575de602713c37006d6c12b2b1e9f68236f4677260c396550d8" }
#827web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "19e7729511f36df14f565012c3d24a58cfd2c840184c50253bdfccd36ed6cc01", "started_sha256": "da13d72ca34947159e6dd7fb361cf8347b97f402055df53364c7bfc70f6e46cf", "original_grade_sha256": "1fd8b7ef29e5d0c5b9815c34033586eb2ab8c4fc15c7a95289b7b33ed6f600e1", "input_original_sha256": "19e7729511f36df14f565012c3d24a58cfd2c840184c50253bdfccd36ed6cc01", "input_wav_sha256": "e7a60eba629eaa20ef6ac7a44b30efb57bb74b55aed2730b89b45876dad1d702", "input_pcm_sha256": "6636f3d45e6b072a3d530bb8bd3773bd92b8698df95e7bee886a2b2ca9a2987d", "output_wav_sha256": "1c149dc3cad493c708bba559fa0e98e945d7fde9945587b60a7aed5f34ac9406", "output_pcm_sha256": "f93f40723fdf7683cc49ca4260aa4a1688b7bcad93145f91727715567eabc4f5", "result_sha256": "84aafa9b704fd4f9cc56ca7d02afac7b416d174583dcb178917a850f4cbbc7b6" }
#130formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "432901488b65c0dad31284b485df593640dd11153b4d3546cf6464009c567973", "started_sha256": "b46e2aa319006c080d2edc954d00cf6c40cbbf800487b6108cd8d07b60b5c681", "original_grade_sha256": "d8c5d70aabf296024db422c8557e8ade4b11185bee116f5d3e627e5d9e56fb37", "input_original_sha256": "432901488b65c0dad31284b485df593640dd11153b4d3546cf6464009c567973", "input_wav_sha256": "9d39c0e7b9b9edd3f9a44d995317bcd455517d541c2751f34ddd7edef65873fd", "input_pcm_sha256": "f5061e96c6f9870f93b80cdb15b9208021442201cc81a40ebd8e237f3810c6d1", "output_wav_sha256": "9f2c0f77fa051a46eff6f95c2353134f1b411383b1f09ab8a836feab4e456317", "output_pcm_sha256": "3b1dd0697c13b9ea77df3dc8a3c4b7b00dacbddfb5fc9e14eb6f85d5bf936c3f", "result_sha256": "1bdd4b19fa781252b07c9a665bfdbc503088831aa5a375b5e630866695e13549" }
#417navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "ef32700bcb27d92ef409077d1f6bbb285c62b600a5389b409e2233a09948fee5", "started_sha256": "d45d86cedce51e60e3b596b3aae0e4f98a2d55ea9903f38e6b16548af5569b24", "original_grade_sha256": "710824bf83b34d15c6fbd66588fed59ca474b2c4d4584c03053d16d407b07fd7", "input_original_sha256": "ef32700bcb27d92ef409077d1f6bbb285c62b600a5389b409e2233a09948fee5", "input_wav_sha256": "6ff25ea41452f9c3bce63c4f99273d1e72c07f41943303b2361d384f94cc4bb7", "input_pcm_sha256": "bb7843e9995322c1527150e51008da25955b9c6b0a37046f86b3de2b8300f3f2", "output_wav_sha256": "bc5a821a2bc5b631836d408bc49432a598b5d1f11ea93bf80941a46e03a0e335", "output_pcm_sha256": "b6522ac701a22027820220dac372e54aeb817c0b3420e7ae8f610d761ad85f13", "result_sha256": "943bc91923225745a6418f191e96876c679e0cd5a18fe250e98fcdd7ee00b0d0" }
#591object_counting heldout	13	The answer is 13.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "1982280ed43d19f46530778fb3096b6c9f290b447d120074ea38838d3f6bc1c4", "started_sha256": "7698f15220fab0c6a9f9aeac97b6a45355593a1d697d8227a6bbb6995dfd1282", "original_grade_sha256": "a234fb9df5bd03b33d3e67a3159c480234d17e0769c234c2af5ee115a81bff5f", "input_original_sha256": "1982280ed43d19f46530778fb3096b6c9f290b447d120074ea38838d3f6bc1c4", "input_wav_sha256": "3b4f4bfeb9180efc61af6aa945dc3c6ca4972fb1232c101e5c393c18ef463995", "input_pcm_sha256": "c182b3ecada0f5d884f3d2a41615a9b564c484d8de65788152632ba4f026f55c", "output_wav_sha256": "8e9fdc13feffaeaf25f8ab6b15d8462dc5e305027e249852e14b3feb85e514e4", "output_pcm_sha256": "4b3aed9fe89e71428689e83a9550f072c15299e5df3113e148c49f3e58ff86c6", "result_sha256": "925df5f627e34e69003055379ab089a13f564e9724279ce1bcbd246981a65c14" }
#802web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "02456fe6553c8b11887ae4530359fe6e9a776764d37ff0ba53ae715573175968", "started_sha256": "4f374e2b4da6f934c8258dbb106195ce1835ddee1bd7ee0acb9e1f5b6f2f5bd1", "original_grade_sha256": "105dbc7baf82fed3ce4ba71e9b97428cf11bfbd8c0c6a20ec888ff81ead36598", "input_original_sha256": "02456fe6553c8b11887ae4530359fe6e9a776764d37ff0ba53ae715573175968", "input_wav_sha256": "fb247a929232823faf55bfc2abd1541cd022625d5f43e82402af566c9bf1f0e6", "input_pcm_sha256": "a11e5fad3d961ca7cf9a030e695db6776b1fd3e4406d697252e9d06494fcf245", "output_wav_sha256": "0c11034f67e61a28cda1f29ce5353b8cb1e35c814c931a7633d60b68ff840990", "output_pcm_sha256": "af0b52a775350f809d9b8c31ca3790f0529c65671675ae8cf1cca71b8e281837", "result_sha256": "2d86bf0c87b66a027d9520859ec79700148a31bb99825bd55bbd1b5f450512b5" }
#160formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "4b4131b63df949c5b8b56ab467203dc9ddbd5c90543dcb0f1ebec849db3a27f1", "started_sha256": "1d80e86dc55aa8985da9b47108c55cfbc157244268ac9138299afdc64344de79", "original_grade_sha256": "eb941a44fc5fbf34182a1096eecc41d6c32a7413ce140033b288517a0df94fac", "input_original_sha256": "4b4131b63df949c5b8b56ab467203dc9ddbd5c90543dcb0f1ebec849db3a27f1", "input_wav_sha256": "ae8049808b50211ce20faeb65a032e5ea3010942ebae03bc6b88344b03a97b5c", "input_pcm_sha256": "1d448097a83e1dc6ea05bb49cd38e06f85eafb740f857a560dbcff3a7c77d907", "output_wav_sha256": "5d890566e9965e429c05c20cf1b00dd28f617e27654b51005425a9fa012ada4d", "output_pcm_sha256": "90bc2335986107f76741760d35a40e8fccceab79f3c0ec8c753dcf10aeb2bc6b", "result_sha256": "d2c57d7cd32bb131095b1928c5a081420489bf81ccc961ec300484473f938829" }
#463navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "6c48d7d7f434823036896dc7c6248c78afe33b5880b94a3985f52ae8c075a975", "started_sha256": "5de469ac9953e495306d3d53caaf245da7319b3ef33217078e669f6f8f841a56", "original_grade_sha256": "87338fd36d1835f9539ed5de952f15a60ab3dd2ccaaa496d31cd479b19942e60", "input_original_sha256": "6c48d7d7f434823036896dc7c6248c78afe33b5880b94a3985f52ae8c075a975", "input_wav_sha256": "1c34e31053c500fe091de0e505b9ea470263d773ddcf9f2b91f4c749f8cd87f4", "input_pcm_sha256": "3d4527ad88366d7a14810d562d61879e8ce0d53090fb7cbca123e1eb83a79c4f", "output_wav_sha256": "acf3111dea27b517a64f7d9b47f6323ee384fa6d157b9dfe7fe97bf0fbdd98bb", "output_pcm_sha256": "9a95ed624bebc5af0802b7b35ada5de02e81ca4bf2d3e88e7fc1b3b44d5990f4", "result_sha256": "d3e92f2e3063e1a0205467f24903808c2abe37e87d5af97c9a10695861d9adbf" }
#522object_counting heldout	8	The answer is 8.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "aaa414b0ca8c83789df2838b7da6a2e9a6e58bf91f785775c0e17d2e445761a1", "started_sha256": "43feda43fefbd1e5d88295438b8e115071ff4358370de4577b7d7549cd340fc2", "original_grade_sha256": "7adfb7afaf0d6abaf8234dd2dacf91d82b972d92cad848eae31316da5da06df5", "input_original_sha256": "aaa414b0ca8c83789df2838b7da6a2e9a6e58bf91f785775c0e17d2e445761a1", "input_wav_sha256": "7f0e31fd8cffd9abb5499e186aac9750776f82c1d7811d420fbab42d5452b7ee", "input_pcm_sha256": "9956dac3ee38d6e47b84cac0b47a37a9c7b1cef2ee0c7d6351cdaecad2a4e106", "output_wav_sha256": "f092bbd4207219d69c59eaa4cf7cceb6af9e870b8f3f0f0d4a166c1982a45822", "output_pcm_sha256": "e58fdd3e09cde13134f3e4af064a9b01df48d1fec9a049c7fa3d1b77628ea66d", "result_sha256": "07cbe02e4daff7d55938f85a403b5ad710b2c4f93cc3189dc9cd8be166ea2dbf" }
#847web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "118ab902015d6b5ff501e98a09ad3621262a2fe24150a0d621b2305a1ab742da", "started_sha256": "ba47d5d6c0b696e10ef85645681dd4ae3a2ddf7a2146052dcb5ee308e2d0139c", "original_grade_sha256": "8b2f1dbe85a538c1d4d138213f37022a0f584baf8bc7277ff5cdc3dc693354bb", "input_original_sha256": "118ab902015d6b5ff501e98a09ad3621262a2fe24150a0d621b2305a1ab742da", "input_wav_sha256": "38cc38c14cfe728be7c0b5f333355c1ca6433083b6e3037944a011f61e8630dd", "input_pcm_sha256": "566a4974087358b7fdd60e1a895fa8df209aef250797e2a1e386e3f11fff2d13", "output_wav_sha256": "ccafb5bb09f5fcdc999b673faba912dec59ce590474511a90676c7cc573b7d17", "output_pcm_sha256": "8a28556827b35e389b397e43e4344214cd04544dcd64c3b9d76c2983cc06ca55", "result_sha256": "bc7966fa1769c6844cc60dc77c98bc59fb556a7095db2bc66a8fe2670a65a32a" }
#141formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "e88d3f021046bc34e7a9084fa837c61ce067cfc132be9a30e43049eb09efc1c5", "started_sha256": "88cb31cdf842e242b11de270e4b82c58a9171c3d69fbf67cd3064f373282fbb1", "original_grade_sha256": "59bef555bc6efe27955e2fb29215fe3cd89bfad2926df5abdb60693f4387aefe", "input_original_sha256": "e88d3f021046bc34e7a9084fa837c61ce067cfc132be9a30e43049eb09efc1c5", "input_wav_sha256": "5ad9a2d6ad79c433ad16d5d6c991df8c0858857e2bb9e4afa161c7bc70545a0d", "input_pcm_sha256": "c7dbb09647a19df99dcafca0a5f04ce7f18072c3cae095229c116ab4405201fd", "output_wav_sha256": "4f38ed6eb43da69b9ae9e1376a02c8d18aca6f181880ccb52f27767f175eb193", "output_pcm_sha256": "67711fdbf66073d82462385d5cf049b60a740d8512894c4c4b1061f44b078230", "result_sha256": "13e75a56d499f08af7021a1f7b41b8dc88e3013737a1958d588b5557a8543ad6" }
#347navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "06700fd80c16ed84ce5cc40754d56e9fe91a931cb203a1c346cb7a40d64aff98", "started_sha256": "d4be47004448a6b0da7a15672e1464b29cd527945616fc9c3aa61150f9453cda", "original_grade_sha256": "1bae68731947c8e720a0f28fd74097dcc3a79ed0369191e15f7a5a40aefa9441", "input_original_sha256": "06700fd80c16ed84ce5cc40754d56e9fe91a931cb203a1c346cb7a40d64aff98", "input_wav_sha256": "ddfd5917307b52235acbb0c02cbdc3e58365aa2d12beed6b500de6493b5cd878", "input_pcm_sha256": "0cf4abab735fbb4ad774b8318bd473877a2f0b11f141de16e2158bd3bcc0c109", "output_wav_sha256": "6c1eee64d640b51d04083b49461c43f1e1c3f1d58b6cee55c2a38e3c8a67f732", "output_pcm_sha256": "1d80d8ff9895f9630e4d82faf298e84af8435142fcee6a36fa5ff752e9679934", "result_sha256": "1dcdab70b5f62e19e8b26135739a05f77a919df6405199d0c8bc4106383c4a3a" }
#532object_counting heldout	11	The answer is 11.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "8da136f310a20e7c39f9575ce2e34d81cc9f27d407ae6b1db8262b832cf38264", "started_sha256": "26c72b1d3461570f1721ca32ddeaf0fd55c32734b5281340d02c9ed5e945cece", "original_grade_sha256": "f5c7cf1b0a6a5db5007f4f9291d3f48b38e6c45023c9dedc8920502ffa479d5f", "input_original_sha256": "8da136f310a20e7c39f9575ce2e34d81cc9f27d407ae6b1db8262b832cf38264", "input_wav_sha256": "d5f100f7805a05bed367188152ef2b24177827e21b49d1e870e880195d7cb3cf", "input_pcm_sha256": "578889d9ac84b6560b3e7578872826c03b42c5531d3411c488553bba8939dc0c", "output_wav_sha256": "71ce0d7ebd77851ec52ed85e29ebc6dd29612f677b348986e454972aba73b837", "output_pcm_sha256": "e88781cee1675ac7c8de09c3e1cd2c21fbb83f6c5b04d396d39936a7240a87b4", "result_sha256": "f82c6ee04f78eb21008ca7576a330f290e7f3a33caace763270881dda5447a4a" }
#893web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "6fe0265d9d035d380fc90c371c249a0580d910a5c5bf8f43215f0e632d061c7d", "started_sha256": "0ab3c2a4cbd5b2a45fa5a043eea5f9f108ca2c492076b9bbe7015a225a405de1", "original_grade_sha256": "6c9a5089ccd95948a1b741dd4cf06766b7a8ca92f8f4e6de6982d0ce3f0e2237", "input_original_sha256": "6fe0265d9d035d380fc90c371c249a0580d910a5c5bf8f43215f0e632d061c7d", "input_wav_sha256": "b3254d3a011f0e0815fb5e2b7defc9b6ac177d2ba6f9b5b512f27948e958d36a", "input_pcm_sha256": "328d09d0993f8fe70f6a729ca8c649547f6e290cc5db3d2dd8781d9fa358116f", "output_wav_sha256": "4267814fae2f7f981417de330b24c3cbe338784ce79495714cb46dac75c7f89b", "output_pcm_sha256": "3024475b634543c3f5315acf6225af9137f5062d1f489202b5c44793575b06f1", "result_sha256": "40dce4216fe6fd4f391007c574cc134be21d8d365b83493197e854bdac66473d" }
#154formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "983785bb7a08f9306abb0813013dcdef4b0103f4357e731bb753b3e9052138bb", "started_sha256": "4e2449a6bf49d06f243fc6af6849aa480dc7c66fd7b998540420f89078fd68cf", "original_grade_sha256": "853c6922ea3e2e70aae00a28c9b4365740c9fba00518394dea54ff48648353b5", "input_original_sha256": "983785bb7a08f9306abb0813013dcdef4b0103f4357e731bb753b3e9052138bb", "input_wav_sha256": "69d48a647dd5258188210a3637cdbfe8f13dc5a860632dcda81a7fe25f439e04", "input_pcm_sha256": "71b2479fea0183a64d82c2eace05e4f16af5489159eea9fe92f0df50463d0877", "output_wav_sha256": "415c9bfe99f408194bcc7e9d208b14355c1349c936871f57e9d3c4631fa4152a", "output_pcm_sha256": "0dfdf5054ba15087d008730196faaf2d3bbcb3726dad9c62f7595a0b9af7d43d", "result_sha256": "03903843a91139504af985bae55ce18760a498544c3a6783bb802ba336a54211" }
#484navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "268a2449425066a4ce936ff558481feeca9f7daafda789bd711d87b4fb7c9939", "started_sha256": "2d70aacec02fb82aa900c52004fc3f95b82ed75d6af518291fd55a312cb810ca", "original_grade_sha256": "8c14a613fe3596d84fb05d1dfb8f059cea3eacc490ffd48183da3a2956750592", "input_original_sha256": "268a2449425066a4ce936ff558481feeca9f7daafda789bd711d87b4fb7c9939", "input_wav_sha256": "998afa8755f0efaa89db108a8884f6d6fd3443ca58bf24081bf48e3438514b9a", "input_pcm_sha256": "a2ecc1eb2447ca44dcdbfed10ab6236580c07f355627ff81c88a42c0bc08bcb4", "output_wav_sha256": "3eaed5888b5e7491d6d89b6ef7397e951b8154d626235d7a65d40a922d950a2f", "output_pcm_sha256": "38f52bf705ce2d3ff5678888912f5b0ccdc7065f95f9aa38d096194bd7e207da", "result_sha256": "a6035b260c232fcbfd4fe9ce78626eda5b51d375aeab038e4ebed5764946f83c" }
#680object_counting heldout	7	The answer is 7.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "3575405296130830066e0e5cd8026695311896aef5fe02b939a9f89a9b1361ee", "started_sha256": "10d068e2b91b77a3d23a9a862151160babaa0a8481c4853e87f82e40786c6e01", "original_grade_sha256": "2231173fb6b62d547923e952ae67529740ebf93f26965a4da0d1883a9429da2f", "input_original_sha256": "3575405296130830066e0e5cd8026695311896aef5fe02b939a9f89a9b1361ee", "input_wav_sha256": "21a6df9d5271dd9bff4bdad17621859f5889bcd3d0c8ac81b8722754ffc7eabe", "input_pcm_sha256": "0cb0782eaf9863e8d58be4078dca67130d6c43ddf7ff732934332fbf751e8697", "output_wav_sha256": "2573bf336e4eafc050b4f65590de18fd64a21dc08bd4e5bf9bf85232e1636bd4", "output_pcm_sha256": "749b41051a5a8ce7c93848b5a110d41ecee58b43763dbbfd4502258eb0179faa", "result_sha256": "a6b8c171a3f65b989c08c8ce87c8556773bc8ab03d936c6f4afa260ff68e42d1" }
#955web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "813ac9bb5f1e6c2c9c5fc0d4ad66df5f8b62a1fbddcd6c0ef3a9d6264d419d18", "started_sha256": "459fabf3544ce9216ac0675aea9a772db1d1eac74e455436172d23c9d3631aad", "original_grade_sha256": "32d54b7ba5648e1b201ae3d6babf9e2233f989b7e70df7f3b64ebd4104101328", "input_original_sha256": "813ac9bb5f1e6c2c9c5fc0d4ad66df5f8b62a1fbddcd6c0ef3a9d6264d419d18", "input_wav_sha256": "7feb1fa30312d93a2cf88238d85a9a4630b370b6f9d90efd2730409aca2b26da", "input_pcm_sha256": "40c084764da051579773724f853d57c141af045cf3eeb3c069f86ae9759347c4", "output_wav_sha256": "c30c89ace2fc998c322d48eadfaf6e16adf1949159e29cacf0d866fa2f8f3f1d", "output_pcm_sha256": "3d7b04feb754c73caaec42634a3b1f8009c685b8a5f7a962c9851ab2211107a4", "result_sha256": "1d471c196dd7bfddcec6b6e45fd3516921003473587c4005ef8f62591aa61b05" }
#112formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "069c2418336933fce4feae2a684c915b45e71689ba48251406843bc2073de7fe", "started_sha256": "5b03424799c5f900cc50d3577f85c01f77a8e38610e0607579c2f7ff92aed407", "original_grade_sha256": "33939c0a5fa291c612144b9272a0a315c56c99ea979e150b43989ee80efe6f62", "input_original_sha256": "069c2418336933fce4feae2a684c915b45e71689ba48251406843bc2073de7fe", "input_wav_sha256": "abc290403274eca7a76f7b33121d2d16449a5924d519741571e203895c528e3b", "input_pcm_sha256": "86761a6ce7f0888e6f8461bd4072e5859be91edbe54972d1120e65bb2cd5609a", "output_wav_sha256": "76d55e283ba0ae6ad7cfde0132dcdaddf81f55242dca30a6bdad6a66494b4f13", "output_pcm_sha256": "4aedde6f4800a686d5a6977779ee77ce4080aeec9632b458b25550947a8d4742", "result_sha256": "e506edd7a20298f8eabff5933e4f986b51adcc399c251456b5df572e23869a63" }
#365navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "4d60b8366b40d7d1c9259145766641d843a7f9b6f5eb4336121fb7ac5af54f0a", "started_sha256": "2dd3b8d54843f9ebddd7e055a0baec0d2a2b9333fd4118b68f2f168f972582dd", "original_grade_sha256": "d07fd68fd51f12587fe1858f59f18bcb9cf6f6b979ff7fb86e5b905bc3216331", "input_original_sha256": "4d60b8366b40d7d1c9259145766641d843a7f9b6f5eb4336121fb7ac5af54f0a", "input_wav_sha256": "737edb910dc7ccd8fe36cbb2ec8df0cfc86d6ad76bd0e915e987641db75288ef", "input_pcm_sha256": "7a2856479266b30377c99ef41cca103daa361686ed68f429b80ae9359a8cb3c0", "output_wav_sha256": "ea5c9734a8f19d3274afedd7643fa0494caa468485aec0dbe2787bae35714d7b", "output_pcm_sha256": "1d6a81d9cd45b3b04a57e3d49555d3696fd82d5fdda026113cd658f680e6c5e4", "result_sha256": "251d368052283518c39f350fc73fa312801591098f06acb0a76d0d8ac9d25641" }
#559object_counting heldout	8	The answer is 8.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "d1da5e779ad2ded6c1f04d38e44aada0b8c8b3dc106e85e316b0d83bb12f315b", "started_sha256": "ac3affea82126e0789bb5ace907af4bd5b1fb7cdf7da3b8c34f3833fd159eae0", "original_grade_sha256": "3f6f5aba868d116b26ca4382f0066e6aa4f4eddc8f9eb969f724cb99720ba122", "input_original_sha256": "d1da5e779ad2ded6c1f04d38e44aada0b8c8b3dc106e85e316b0d83bb12f315b", "input_wav_sha256": "c83943905fa2181c37170395e9390faaa2fc10f81bdbe46b625cceb715657a6f", "input_pcm_sha256": "276c9905b868808f2e021ffda1166554e7e36d2fd84fa71c4f29882e07bb95c2", "output_wav_sha256": "1ffc8a639b4482743133c68a8188f71441eec606d5f69aa35ffd950fd6f0b4d3", "output_pcm_sha256": "3f175c79e8503648acfea2b1ffb3a0f833290e05ef3650e0634db33c3f895355", "result_sha256": "e970a7e0de6af09c1e20e18dc03aabbb685c04d525a3f4c0f54f248f9818e2d9" }
#790web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "42ca9995cf81bb2ce54d5e7d1804af99b8c6f1cffa4d5ddc2663acf91eb912f0", "started_sha256": "4bc1080b429765dff703e9f6f0e77d0220c97a1f90b2831e6429c3b45346b58b", "original_grade_sha256": "a1da822c3f738eaa9126f9e9665053d2be92a5e49b8fbafc3efbad43f0ccda18", "input_original_sha256": "42ca9995cf81bb2ce54d5e7d1804af99b8c6f1cffa4d5ddc2663acf91eb912f0", "input_wav_sha256": "e4874e7045e5496863d0223eee032b6865e5a2e927a75e8ff0cbf539ea7eac25", "input_pcm_sha256": "5cd5215d1dad0466ba1a22c11f340e02b5b6a36dce7a0333de3b0946371d7fdd", "output_wav_sha256": "fed594a94e917f3a2ecf4201b8074f14232e1911e464746a1a0927ec4f660d97", "output_pcm_sha256": "a308a6e2936d160d671d6ce511df2e3418a8f4d833eec012b4613be4dff17bec", "result_sha256": "e2ee39e8fc8bfb9f6bd0715c1cdcd41069517f1fc93c4d6ff2ecbae9b05afcd8" }
#165formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "edd295219373e1ef148d64704d9c6c9e1ad6c7e699787decb78805b6bbcc20e4", "started_sha256": "ade11f040a94bcbb035f7da9109ed3f7b90ea5134bb0c4e1cb23cc76978fbbe3", "original_grade_sha256": "3344ef29da049a93bb09a918ff63fb89d3eda59ba6cd0b459011503facabaaec", "input_original_sha256": "edd295219373e1ef148d64704d9c6c9e1ad6c7e699787decb78805b6bbcc20e4", "input_wav_sha256": "1fe2ffc59035addea3e8147e7e3f8a9e3ccabb2d5c2e66980f564f5483af29a9", "input_pcm_sha256": "17781aa530c3d44f29b645abaf36ebc86dc7e8f9623c005473262a49668e952f", "output_wav_sha256": "4c7d5dbf1620550cc0d9c28b2ed97d62a53e866f7999c91ad146bd05923506d8", "output_pcm_sha256": "c2b1cca6a63499206e893f7d32f4f6caa70c885c26ed483f5c643dcd73201ea1", "result_sha256": "8715637ce989338c215b63abbf04d21be6bdf18294cd358d64164ed7ccf584b5" }
#416navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b1ba1d6cdbd8f311d19b61772d45990c18990bebcad3f8949d35e2ef44f41a45", "started_sha256": "75cb25b9f7127a7157e4b9aa8b5f1e37118a79fd376112967c8e7707f558e9ea", "original_grade_sha256": "e39c6666bcb9d8d96c12a894cf6df90ccd49c75304af9ff8de0ff8c29ba1df95", "input_original_sha256": "b1ba1d6cdbd8f311d19b61772d45990c18990bebcad3f8949d35e2ef44f41a45", "input_wav_sha256": "cd0657ce3154d5712a3fa5cbae1f854298b5f549ad82eca7f66cba9f840e536d", "input_pcm_sha256": "61d3261bb0261d84fe134dbf61b401ca2bdff3b9764bff4b621f9d365ee7c54a", "output_wav_sha256": "e1bfe1d8352d24801f4b235e941b0d49329a6aed2e95b716b3f8ca3928d552bc", "output_pcm_sha256": "87bf7dbaefe86959e6c08c6165c659d1a8901333fe743d0e194311db64cd2ae7", "result_sha256": "d07806b8c56795c7d41c569901cd73c672f4f33fc08f67f5ca772b664a025575" }
#552object_counting heldout	9	The answer is 8.	incorrectstrict_spoken_answer	incorrectresult	Original grade retained Evidence hashes { "original_audio_sha256": "8489f5e8fca65b40f671661ddc34fb997be8d4c95371c5a3973b23bd5d450be2", "started_sha256": "1800294789a4a85e5e3f547364c9862a2938f70f9c5e1dbab643e83865f30bb2", "original_grade_sha256": "69bcc17e239e76d40e8a8f56d0444609de3fceefb2b891b34c6a3bf0c4ea7883", "input_original_sha256": "8489f5e8fca65b40f671661ddc34fb997be8d4c95371c5a3973b23bd5d450be2", "input_wav_sha256": "d57b9eea5d2bc9dbc71ba869866f64a5ab5c6f4d007842c48f57afc5e731f85b", "input_pcm_sha256": "890c87df4e20c629d7019387cfccc0767999455ef156bf16dd1163e3d03367a5", "output_wav_sha256": "3331d0006b3046322ccf1a7d90f43ae9685e3a91c4a62ae2e79ef266203d1287", "output_pcm_sha256": "c51562edd120e61c44b71b120c07577c43b6057fbc1de3a02543022cb943ece8", "result_sha256": "2c488690a6f79ac470e94618d0c9f64c2a9a4ea55920c8228a64027bfbe09052" }
#923web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "bdd196ceba5f6721862ea80fe1342403ca8235e2889a81b7491ef037045913f2", "started_sha256": "e9b21c64644c94fd76906d3506362e17c0c6d3dd76c99e4b8be7fd8b6ac56d35", "original_grade_sha256": "ef9239bfb0de1205668c50c8b3465735e346dff2d58fbe71172f07b6dfeade60", "input_original_sha256": "bdd196ceba5f6721862ea80fe1342403ca8235e2889a81b7491ef037045913f2", "input_wav_sha256": "0e76db68614767c0baa4c5351504b2beeed5e5903b6e56270e725a5afa033ee8", "input_pcm_sha256": "f270689b07874f606f2c2ec46f26fe430fca2e9b7402a03c34c9de539e8927ac", "output_wav_sha256": "a285e4afc893a798b2bf6ebdf36d1149c449628d76dbb185ca1570bed9fa717b", "output_pcm_sha256": "42b5b55401bd46ccca716888423d38b68c0002f9e03a32419a50000994a751fb", "result_sha256": "88c39e6b8c60ba399b80efded6eb7f78179f032312ab40d12da3ce90786aac83" }
#235formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "85fd8a3edb248a7a490e57199a36d76fc510bd06f7757870355c6198a3121b24", "started_sha256": "f8183a60bc625941a3175344a1c402b91c69d1b21abe0e5fe006caf0096e75a0", "original_grade_sha256": "40b283bf7125920a65dab666e43fb37c456f955b8bc758c29e3427705daf8d80", "input_original_sha256": "85fd8a3edb248a7a490e57199a36d76fc510bd06f7757870355c6198a3121b24", "input_wav_sha256": "b9571cc8c2910ea53cf380143741dbcc39ba648dc4f0f611445716eb17727dea", "input_pcm_sha256": "6f970d4295a88e50303b939a32020f1adcc67329c095b254f4b10e69127fc76b", "output_wav_sha256": "cdddd1828da6621be26117aff8c94ff8ebd7e644f0140c270e8c65dfd05a03d4", "output_pcm_sha256": "94ff1e9d29bf83f90e3d56c7b0f3fb5b0dae1557b4ae1fbf8926b58017dc8bd9", "result_sha256": "439cccd90575947f37ef2ac39fb99d8c84b6d0303a60e2949a2d6e961d9dd8b8" }
#378navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "d3350e755067f0fc53f3d28d2b61d87ff475f94d3054fd8cd3baeb488aafcbc6", "started_sha256": "06a63c0dac374345245d4806783c53a94703de97df7e964786cfa55eca900875", "original_grade_sha256": "3dd167c0778ea01cc22d6bed906423476ec8295e3f6607ff1845f78b6f3429a3", "input_original_sha256": "d3350e755067f0fc53f3d28d2b61d87ff475f94d3054fd8cd3baeb488aafcbc6", "input_wav_sha256": "9d24697316d43b7467fdb172d5e9267891341fa3bcc6adb61e980f3909e02198", "input_pcm_sha256": "7072880a557bbfae8adbefbe57b329108ab63cbc23e1cb9d3a656eb1181ca85c", "output_wav_sha256": "4c920c06888c3e7d33d8aab876f529b2eb154f57997d86d1bae11de104069f57", "output_pcm_sha256": "552b4b8344811ae776b1560de2cc6e26c00dd396b2e49a9f749d9271dfcfe257", "result_sha256": "99fe2dfe179e4c232b2002e92e870293d716b220668fa92bf811b5bf029df49e" }
#508object_counting heldout	9	The answer is 9.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "aee290482be401104f79dab372f656993a1100d2b060ddf7633a242418f4957c", "started_sha256": "4e29aeacf1e8c3e3846dbc5db29e0209d989b538d79297091f3b75375aab03cf", "original_grade_sha256": "a25fc758c8137b432f5e37188eaadf30926d2e9adb8b06fb0193016db6fec1e8", "input_original_sha256": "aee290482be401104f79dab372f656993a1100d2b060ddf7633a242418f4957c", "input_wav_sha256": "cc817786f024203ffc780c7d4e1c58c01dfe6ee8c24b0c04f5e7e2f982fde627", "input_pcm_sha256": "341a61578fcca47a439ae84e85000e5666fe04ab888d27869a87c8ea7fbe2380", "output_wav_sha256": "97aff26286f770f9208c848f917b167e8e19871c759da354cb54a98483e38c67", "output_pcm_sha256": "ea20574ca451be92a568a3bbd751c29e74edbebf51aabcb235c7be15d9bc3c3e", "result_sha256": "18a3253217049b2a0727a1a7561f447212b9454b64f48b02122fe88566278ed5" }
#942web_of_lies heldout	No	That answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "ed22fddad81e36d1a388220f0b37dc9371d7dce0b3dbb6bbdf5c88e93fb02a96", "started_sha256": "410ec726baa01fcc9c560877f972542343a12e50e751ef845cc5d548cc7c5750", "original_grade_sha256": "5ccc52bbf7f8c53770a572599e9bde9738312dbb7a1b1954d982b510b8383f93", "input_original_sha256": "ed22fddad81e36d1a388220f0b37dc9371d7dce0b3dbb6bbdf5c88e93fb02a96", "input_wav_sha256": "6e541ace3078781ef29e775b584b58a545fde85b9520e387b708a9b474062c7d", "input_pcm_sha256": "360cef618be6261af9db1ad728e47c3007a2471f2c4e32d36ea42c2ff81e4715", "output_wav_sha256": "c9e20ad73a9a9a6a298443024dec6443fad00ca4cc1af85cc47a2b252220a4ab", "output_pcm_sha256": "1ed2a9522812816b39d105b7a60189f65a25db19d4807c2dcd8c9d1fc313273f", "result_sha256": "a34945fd38834c3d3847b6c9b80cf1bed5b5df2f9e028d3a79d6bf719ac4ed9b" }
#91formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "dadb648ccf7143743e20640171ffcf413c078ff9443b272739bc552a03dd5347", "started_sha256": "ecb782e434a391c6ae4733113dce46026394ec4034beb5a79c920e447f9f3960", "original_grade_sha256": "fca70beb6502b94f5ec1315f8996598ba733ed1e6216c5dd25dd4c3392e00b01", "input_original_sha256": "dadb648ccf7143743e20640171ffcf413c078ff9443b272739bc552a03dd5347", "input_wav_sha256": "752823e48ac58f15c5c0dd446a4dd9cc02a9b6e88029d2e11dbbca989c5a0b90", "input_pcm_sha256": "e9ec1fd95a3995b77d9ed9da2be981505ee1129d7558f35eacb122f1261bff84", "output_wav_sha256": "e8e9593bc225d3746f617c286477d1e57bbdac134bd813f4884d91f00af4e85f", "output_pcm_sha256": "430f8c6f790a063e2750774cf62e17582f2edafaf9bf562e15c53fe37c924c3f", "result_sha256": "cbcd3a6b46c61a2f67ca78e5baec0d8c02f4f1f842c8249a6a17a0af7f16db67" }
#403navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "e906562c30962a7639c5143e04bb47e60941d36a6225a69dc2fbe5d0cd024a74", "started_sha256": "81ce919db23aabc80172fa63eb322ed0932243870d2064f801448cee17bad415", "original_grade_sha256": "97b0c6761f32d5309d0af4394df0ba42c10b63965e76a15dd3a23b8022d708d9", "input_original_sha256": "e906562c30962a7639c5143e04bb47e60941d36a6225a69dc2fbe5d0cd024a74", "input_wav_sha256": "64a2556d547ffc40c2ec5c787561ff61b08b57afb986629ee1e3ffcc8c65a7c1", "input_pcm_sha256": "e338487f76d8759dd1a2dd79d96cc8d185a40915dbf51133b121b43a127c9bfa", "output_wav_sha256": "7324b1c1815312ee08426afd8b8823dd66e9e11982f083d69f597de61ab4d1e0", "output_pcm_sha256": "734f3d923479560e7147813a947cab1e43f35d169ad08e9f7a1158ec7d9c9e15", "result_sha256": "7aee2c1f31e47df6cf9163aacde1a7ac070cd541498a8ec0ed0a535e6025013b" }
#542object_counting heldout	12	The answer is 12.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f80c1bc4c1b1a079765ea61f1e83540324ff773749c26972c20eb9b05b4b90c4", "started_sha256": "1d6d37eaeb6264477f3d3292e565d911cbab5d9dbd026cbcdfa2af585be853ad", "original_grade_sha256": "221c96c685c076b9efae73d42a0fe0d33115ee9117c2f148295efce3f2b7ca56", "input_original_sha256": "f80c1bc4c1b1a079765ea61f1e83540324ff773749c26972c20eb9b05b4b90c4", "input_wav_sha256": "0b823f4dc7ecde6bb2ee2891ba073de0b8c06af957472c302a6aa000656d1d18", "input_pcm_sha256": "d693b28075eefee729d5515edd88d55fb317f98a69bf938434f6ee8025cf3880", "output_wav_sha256": "ba6a31986dbf49f26dc386059085f987ee0a9ca60c3019c0b3e0be50a062bf1e", "output_pcm_sha256": "26fdb5263f63a9b29c1385b844ac842a061e4da684bc484bd084d241603f5157", "result_sha256": "b6822177ff2baf3ded7fb1d86ffdd92dc828599420f2f4ab218b2f63a61e0672" }
#826web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "2ba667616d8c49c57e0c0c984dc6dbad132f977570190a110f6c31b3183552be", "started_sha256": "4e994c313a0560e27587366bc613cc4db47d8009c133f6e4a2e01d1a2d5107b5", "original_grade_sha256": "322db399771a42c7185cba7023518c51817015b099422abb477baa49480c0eda", "input_original_sha256": "2ba667616d8c49c57e0c0c984dc6dbad132f977570190a110f6c31b3183552be", "input_wav_sha256": "6be0727ef689c7717775e2eac5b931d198a02dba8fe9db46c3b4a374ff9f935a", "input_pcm_sha256": "ff5939537ee50937296d9db2f5b89c3f52848e96011fcf7384f1e9c63b539bca", "output_wav_sha256": "2c5d4b60241fe1f6229547cecec160b2d6cde4e6aa07988435454928f9bc4a6b", "output_pcm_sha256": "83d257479b23d9033638870094eef587d8675aca95fedea26ce653f2ab13571a", "result_sha256": "8f79e4153cab6c9d7fe865b2c3b64cad19f6f3089bfb3c052d8e82ab0394970c" }
#96formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "d371541307124c1b0edd4ee1af8ce647bede102bb1d054a5e1dcd72ecfe1e68b", "started_sha256": "a5c68e707558ead605d09605662ce10ee1ecb580ff773f1f791484ba86d327f4", "original_grade_sha256": "99c58045546b7a1425eadc15243962d4a270dc2a404b8d4763551410f5755e1f", "input_original_sha256": "d371541307124c1b0edd4ee1af8ce647bede102bb1d054a5e1dcd72ecfe1e68b", "input_wav_sha256": "f0639377c0b3073e8a8d421773976c428d4c23b18d27a1d7d85407db70d49e62", "input_pcm_sha256": "2c4afd886f950b942bad338520b4ab8c3d81f71f9c90a0cdba564c1d1873df61", "output_wav_sha256": "8118b76c71b0317ffdbbb6bc0e1a86bec3f156c83d3c7d8147f8aafa6e241b2b", "output_pcm_sha256": "a9ffc649adeffb5861c8c2ef27178cb74af0eaabb5fe46fb5d09245ec32b0af0", "result_sha256": "0ff8342b3fd12d82f803c5fbc1827cd97982f45e4857c5e3785555faa5462026" }
#270navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "685767812301318b4b3f00d3efa6fbb061ff245d4fbe830610ce92f1479cd21d", "started_sha256": "d140f9eaaef231b10137710aaf97703117a3e409c0331ad4204b9ba9e12a31b2", "original_grade_sha256": "71c3b0c600a756504b8cd180f7b06577b6124ac9574c304d39da5f9fb1a3a53a", "input_original_sha256": "685767812301318b4b3f00d3efa6fbb061ff245d4fbe830610ce92f1479cd21d", "input_wav_sha256": "40329b91ba86dd2e0cac24195f7722e2a328ab8b3e52171cfb59a7bfb76547a8", "input_pcm_sha256": "f8414b5e62756eecaa4dbe64fe6dceed2326cdf4e8834d8863ad87388520043f", "output_wav_sha256": "842bcd1ac2d759769ff1c59b704d9b65d4b4163c391e6c95291ea00bc80a404b", "output_pcm_sha256": "23f1d2548d20362b1de91252b47e896dbc77f317058669ba269dc04403da266b", "result_sha256": "3971c32f1f0bbf2ed317695257edf0397e767cf5b7b52d5215527c7ebdebde4d" }
#583object_counting heldout	12	The answer is 12.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "db223944ab1216df57dfbd095e00472f8e89296f847ebedd253e251f12d355bb", "started_sha256": "14ee9f508f42eeb905a4836e8d082579f3f20e38d6eb37950feaa8547163a6e9", "original_grade_sha256": "8dd17ad60796df22345bffcbfbfe471c1e15f0c2606b89bf30a8056385786813", "input_original_sha256": "db223944ab1216df57dfbd095e00472f8e89296f847ebedd253e251f12d355bb", "input_wav_sha256": "50cfac2ce432c999e4e6d90b23bfed407c8713973528543ce2a86efb69879d26", "input_pcm_sha256": "75c581e49e3f632319ef51ff0a542eb5aba617d3708d38ff851e62cf2b5c1930", "output_wav_sha256": "2940290a7faf0d9cb2c0b8e6296e6e0df9ea7d32d8963a3b46f9440f88afcd64", "output_pcm_sha256": "08181660e34f511ecd41dd8262fc7041df12901308ddb957d6b706c925de7e11", "result_sha256": "bff0144734e18bad39bd28e5d55d927ba73bda95a7ab31c0777316f23a7738c9" }
#965web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "777234b7c909889441302a60158e53d11e68263760abacda3bc979cc3d4f96f1", "started_sha256": "2569de0352e2e73b1799badab4e47734c9785788617cda008d088d66cf5b5794", "original_grade_sha256": "22f837502568858fbc08564f50bd100582626856f21aa77c5b2ea47995386eb2", "input_original_sha256": "777234b7c909889441302a60158e53d11e68263760abacda3bc979cc3d4f96f1", "input_wav_sha256": "fe5907bc1bc2757b5f5d3b47688358aa1d616af12d88e5538bd7a3f1d6ebe46a", "input_pcm_sha256": "a19591ef591e3af0c85b039b9065956a3a6d1fe074adf1a915ca2fb97e938bd6", "output_wav_sha256": "ec2b41999c116db37b480f1b9ca6ed14ee65e1f1e593673ed019f47d971fb73f", "output_pcm_sha256": "7a2d28dc81a4e306b34af140792315e3cc206e794cba25532c1bf62d33872554", "result_sha256": "99abaaa1fc329de67c8e8abee50f280a227bfa79a7652dd9de452d0cd7d76cfb" }
#33formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f07f2c18aaa97a8fe402e7d776f30b1e9a7e48278e000b7d941b26d3fd0d0ae5", "started_sha256": "5f0fd95007363d7baee5dc00066dbfc08dcad50bd5fb1b6cf85da01852788566", "original_grade_sha256": "0c539f345daa4158c4140a30cd1b25bd5baa196ab2c31d2efc10214b30d1ebb1", "input_original_sha256": "f07f2c18aaa97a8fe402e7d776f30b1e9a7e48278e000b7d941b26d3fd0d0ae5", "input_wav_sha256": "41fc9d8ac83dbd125ba296cbfdea9b444e81cb161bd29dc9ee60ab770ac0eba2", "input_pcm_sha256": "5dd6beb0295590060acd766c5e21a5b1dc2896cf827bdcb64f38435ac792d5ca", "output_wav_sha256": "d56810fdac724d7d0f378bce962b405d015f73d96aa7c5dc72c13f06493edf21", "output_pcm_sha256": "bb5cb99f4ce118deee1715f04d02b934ced187b63eb2483346ef3d933cfd46da", "result_sha256": "ec2d1efc2500c6c7ae4bcf7584d2f9ebbd64403dd604bd2d4a0b379011484a18" }
#376navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "ec1d112d7ce852353696889ec99a2cccabd03abe9dd37a07e14b860e4903e1d1", "started_sha256": "53419142eca769a40df11e027617bb3686ec0cd982e3ca3d2209a523c2970b9e", "original_grade_sha256": "87cc26891e882577632db0f98e126b88d906af6ee6ca7652f1633922fca8de21", "input_original_sha256": "ec1d112d7ce852353696889ec99a2cccabd03abe9dd37a07e14b860e4903e1d1", "input_wav_sha256": "2edcfc4311ce3a9bc982a632d8ffd5eeba34f439d7f91c0b3df66a0734661198", "input_pcm_sha256": "522e4aa5093c7eb4e457510a0f784cd51a41804dade8f061a6ad32125d07f946", "output_wav_sha256": "6031b9ec062377d5c7588ba7cedff5d2db34d78f177205838987f018835c75b4", "output_pcm_sha256": "fb6ebaea57e584754c158c006ddaeebcb0aab9aa373a3f26516d416854281be6", "result_sha256": "a67b1cafe518e94f7ec4a7fa9a39c6f830e02e2b9cb33b1de0226846d12fbbde" }
#587object_counting heldout	12	The answer is 12.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "88b9d4c9d3a5df2e406412c1d032ee4eb621a79bea757dc37b086fe776318a45", "started_sha256": "244ecb73272e34b67dd97411208aa1d1451395772017f7561c002c23f7d0f40b", "original_grade_sha256": "cf4e12f851afb1d1a0a02fcb9a620bcdef49f0e3c9a3b69511c71dd9b922351b", "input_original_sha256": "88b9d4c9d3a5df2e406412c1d032ee4eb621a79bea757dc37b086fe776318a45", "input_wav_sha256": "f29bee53807ca5cc63ae5170ec0925b483d8b9d9f4debff6e56a698f83496780", "input_pcm_sha256": "7cf9571b7ba88ecf348755121b752dac227abb385d6a7d0236b723b7e83124ed", "output_wav_sha256": "d92a8b00c0b0c1ca5dc81bd6b9ceb9a12ac1c731f6f872987fedc5d65138389c", "output_pcm_sha256": "cc36b910b3b3f5c005ba9ba0f436ce8094a33a2e961db50def6941e52f612521", "result_sha256": "e5209594b0c927d847f837edebd595e4bdb26cffe186cdcf923841b2a8c08115" }
#784web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "92c9fa0eff32d607aeef0ccdfa7fcb44c1999563e59672bcd164ec87ff563643", "started_sha256": "547882ba9bff21757e5c759c640b47f0c6b3b79d0dea39a5b3d26d3ca41ac962", "original_grade_sha256": "139c133409e47d985d46f008b776ea8db0e3ea410a5b1a27debdb468753ab0b2", "input_original_sha256": "92c9fa0eff32d607aeef0ccdfa7fcb44c1999563e59672bcd164ec87ff563643", "input_wav_sha256": "867a47e0e35af569a129b5d802d982e64d61ed447c4e9fb8e0e6b98bcf248f78", "input_pcm_sha256": "bbd753f7a64cdc7e923ac7969c6940d16c7bae171e9307c3da5101969690c342", "output_wav_sha256": "e6a70e5a597a6601f4ecb8d62bd699c0a37a21b8c8d3baf84d2c5ed338aa7780", "output_pcm_sha256": "4f5e4ba75ab69f7ef75fb7591c362d274347f3fd2d08d37d71cbb2f74b22254e", "result_sha256": "1bd9ad0bd7fe4459ad3fb58309525a716adf6e97e8238fd419682ea75a1876e8" }
#30formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "bf2d26e8c26e60756a21078c41449be3621e88c42548b505d7f21dc679b7d895", "started_sha256": "75088c06fcb66d5e3ae73aff3a7f646842d895e70d0c9338d6f160aee293a2a6", "original_grade_sha256": "d42e8fca335e72ea7b4087159013e87bf98739ce808ab2093819d78bffaa6d45", "input_original_sha256": "bf2d26e8c26e60756a21078c41449be3621e88c42548b505d7f21dc679b7d895", "input_wav_sha256": "f5f51cb303d64c9b1e57bb1fe47ae0ed9838be9cc60e004dd92a6eb2f22b5999", "input_pcm_sha256": "7d3991315552bf68c5cb1960411e50ab761b257627c74778c2e09a950b391084", "output_wav_sha256": "58ae09a652f6e949d901e10fa6f3713b31ce8fe0e212cc06dd86465e934de010", "output_pcm_sha256": "ea59793c8b621b1ef9d7fa21f6839cd71266bc80e443e6f115c263e947500824", "result_sha256": "450f8dd29c61dfa53acdf53b2f0e1375bde7ad1adfdb83fdf40243f8d8f9633b" }
#263navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "13b2df69ff5317936b3a270714dd1ae3d9ecde2bc546e9b575263cef716d356b", "started_sha256": "d1e7c29d318a41846f9767f32ab26f508007ad24d7abb44eeebd6d1fdeb18e21", "original_grade_sha256": "878a9031ceff0e135803f6537497fd032ba1c8d975d02bf2d7936f7082cf21a9", "input_original_sha256": "13b2df69ff5317936b3a270714dd1ae3d9ecde2bc546e9b575263cef716d356b", "input_wav_sha256": "7030418dacfd41617dab62e8a8f5cfed708c7b3511d1fed4b2bcdad229fb8952", "input_pcm_sha256": "72380653d8db3c3f31a2f41761a7f1486c7a7b43fd9a2570aab96e64088f5793", "output_wav_sha256": "8316849a4a27bd5b761211ab7c34e3107feaaa8fea554de34a86b0d4de9c6bff", "output_pcm_sha256": "bff5cfa8ea6f6e5112e28e48f1f76505ce0eefa2681fab7f2e33ff2ffdfaaa90", "result_sha256": "db1197d7cedce67446960b41fd461b0b69df7ff9756c54377df9336b52760a6e" }
#607object_counting heldout	9	The answer is 9.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b3c4aae3c36d9ed461c9a887532c65ec7a92742363b600086ea174eac779dce9", "started_sha256": "07f7d7fb3ed4fd30749d22e3921d16ca3c1fe8effae95010bb819199075f4228", "original_grade_sha256": "6b68e218b309e75464de823ce55d60313204364b108d7e65614ad59d423ab3d0", "input_original_sha256": "b3c4aae3c36d9ed461c9a887532c65ec7a92742363b600086ea174eac779dce9", "input_wav_sha256": "0e4d71453c8062ced994b3c8ecbf9fc64afbf107efbbfa34715ee6163de5a71f", "input_pcm_sha256": "87f5c64da9c3315e6e8447b47d4bce9ac378a897cf15807122d3a21b800d3a3c", "output_wav_sha256": "8e85fb6303c92be9ce988bc7940467e922289fbfd9b61370fb37296f1d06f174", "output_pcm_sha256": "8fb9c7b026c30ef0cba338c5803b5a8d58360098f6dc1f664deb29421da230bd", "result_sha256": "3a47019ab2f2ac365c68209edb83b320d6bdcd1b8b5999b96dc9533e0ec556cc" }
#924web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "a086229d431c01a702e24b70e2dd078adf21f0fddb4aae21eb9db0543afaa999", "started_sha256": "e32e582fb3665260c75127225608fe5a9fa1baeffa919e7d86116efe36a75b97", "original_grade_sha256": "51056c9f2beccfc4ac3ef17d2e46696d8dec62cf2abed7349b45487dd78741c6", "input_original_sha256": "a086229d431c01a702e24b70e2dd078adf21f0fddb4aae21eb9db0543afaa999", "input_wav_sha256": "42f88342a3bed727bc17a1d944e00f113fd8973ec0dbc5d8aec2fa30ff580520", "input_pcm_sha256": "70912598481c1d675b994001cbd472279289149a4ab3dae678bcf16e5f65af0a", "output_wav_sha256": "88ddd29fa7ba3266353bb9dc1d0e5b467b86b09770e5d712e4668a75397c6d09", "output_pcm_sha256": "24b57f06c807baf685687cca729702297b86e5b14b4c8215f07767353e547779", "result_sha256": "6025b138649e7916f796025ee03ad57f8446ae06c4850f8e519446a7e1212961" }
#157formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "6fd10ff31b832f343b7d7e2c58915a44229f92b278deebe2232826929579aad7", "started_sha256": "b76ce570716ac60efa00774696a2939e5742329655e769a47ce5975160ac5975", "original_grade_sha256": "32c39f94f1e07a3f16feecada434f6ac4e8877071dd5e34e9b11e5204b89764f", "input_original_sha256": "6fd10ff31b832f343b7d7e2c58915a44229f92b278deebe2232826929579aad7", "input_wav_sha256": "c7ecb9c9333b1968cb3abeeb6ccf7f71409d6e986cf3d423c0d5e274d11d5e24", "input_pcm_sha256": "ffe85dbcba272ab62f17ec0afa2f90d2f84486dd33838aed7734827c8d609039", "output_wav_sha256": "10c6fad875fe9bc9f23186172ab779b926a21150f81f717e29002a5826445b0d", "output_pcm_sha256": "ac9fe84928e04b4e4eea240d73687d55663a285c35badd8e45a5a37b03bd026f", "result_sha256": "c87bdaab60d8ca690ae80b0bd3b0b6dd179bc8cfc82ab4f1bf3a860d5209aa1f" }
#369navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "fec5dea1b8a6eeadc8184d8fc4bbc1da011a8df9b87ea73a8480073738ea3a8c", "started_sha256": "bef5997232b27a52c36da1be0909382a3af5432014062cc59fb122561c505bf2", "original_grade_sha256": "180eb95a03c5b6ecef4dca847e003d24c6fe19ae08d3a57e81172b4da7b890ba", "input_original_sha256": "fec5dea1b8a6eeadc8184d8fc4bbc1da011a8df9b87ea73a8480073738ea3a8c", "input_wav_sha256": "b8aa6a089fa927e9feccfff03e4730e4c59e94b679c869319753564af59bc0ee", "input_pcm_sha256": "c22295bb04669031c27f1dcce5db8b26680c24e76668b1fcef8f4735f448b3fe", "output_wav_sha256": "7d2bb93ebf825a087468a3d1ededb5c62111ad21341f876296cfcb8a1f67dc9a", "output_pcm_sha256": "0ceb1d7324d91db9daa6381456f096a8e40da6da886a9550dbc440e573683c3f", "result_sha256": "ab9747279ca03a68a9161bc4eade8e55193ddd09070fdf1afd242c2f8e5b9bc1" }
#594object_counting heldout	11	The answer is 11.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "fd54674db612fee81de94045fb03f9e25e1127ee55a1106a8993a1302753325d", "started_sha256": "97b6a524111c8bda0bcb911f1019c455b6ebb9253887c29c9368cb1897efa0ba", "original_grade_sha256": "c2afcbddd8de21d54a102777f89d29958df0bd314acf0b0e5451313fd92725fe", "input_original_sha256": "fd54674db612fee81de94045fb03f9e25e1127ee55a1106a8993a1302753325d", "input_wav_sha256": "df57b6830263da0c203d140089f8ae55483fa67d4c5a2017bccc46f6ff05204f", "input_pcm_sha256": "4e10d9b2ddcf4c26b3fa988fe44c5cf2a0e29e1a521ad2f1f3360eb43d520ee6", "output_wav_sha256": "d531b21aba9cb5dec746a8095a73927732349fedfbf9366c99766b9f6a371bdb", "output_pcm_sha256": "760fce4ca8497dce3545335577c75b91a1996fd722c42f6824e7f3a06b8def56", "result_sha256": "1738019e1bc148b9754cf7b7698bec32cd103cf9e5c575533eaefc07deeda032" }
#839web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "2b2f64690f02379f12a85a8afa12a1c8e7d4faa4378c8818282a4514e40a3680", "started_sha256": "801f703eeb1882bf648ad73ba4fb629f731dc760ab4bb27415d73175cb14b90c", "original_grade_sha256": "a9627d2979bef8cc74416058e884f8c5e7275279b774bde81a85dbb555783f44", "input_original_sha256": "2b2f64690f02379f12a85a8afa12a1c8e7d4faa4378c8818282a4514e40a3680", "input_wav_sha256": "c12fc03d9adbe2c55851afb035428c337be9afa24d6b71b8f9b266e0e8242bdd", "input_pcm_sha256": "56b52c88355722da327b89c44c23903600a3c4515f9db7068b75dc21b7638bcd", "output_wav_sha256": "b0f791f322a229cc0cb7ca1b8bdf6a6b840159ac063c913a2585f5ddbeccc6d1", "output_pcm_sha256": "1b98af4bc386c635f059f6c848c691a08ea544a6db1150a6d7c8e781ca758435", "result_sha256": "7c21856021b931010b7e9ff97bea5aecd903fe0cc3989cae1db0fdc2196eb3c1" }
#158formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "562117351224f0fc7872352a3ed543f7cf363884c6259c352e7ca162e3b74ecd", "started_sha256": "ca1f92032b07531e9ae532dacd485240956cd94d9789b6e22daec3bec12dd478", "original_grade_sha256": "30322e4277318584bd91ed144b765e56fa849d11c55a5e982ad107f1e605e6de", "input_original_sha256": "562117351224f0fc7872352a3ed543f7cf363884c6259c352e7ca162e3b74ecd", "input_wav_sha256": "2ef566d846b8f2deef4265a268aa48a4d6adc8d665fca06191b3cad067aae502", "input_pcm_sha256": "2d30721f6575d0b69b8c4f4271f3c3ada4c3b55e132d7c4a3c2791f504603518", "output_wav_sha256": "494f49b4a373699a827b2409547747017c3140d1d5e22063b396bd83302f6d7f", "output_pcm_sha256": "95a2bca52428351f009f7f241f7f9564f851d328dc686a4f9634d6f45f8004fe", "result_sha256": "57c719f9cbb496bbc77b0c3524eb1e2df9d169ff8cea40278cc9e6d1fc71a46e" }
#359navigate heldout	No	That answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "43bc58355461cf87a567d96d87ec15f6178ede9a4f73c201fd3fb93b46d329e8", "started_sha256": "93e02d5efcf5765501c08cb2010eb51e79bc44c3ee0d65553ac90ede999af1da", "original_grade_sha256": "3c31eb845eeaee6cce7cfabb8436012ac6e5d10ec032e275b31fef7f527c1de1", "input_original_sha256": "43bc58355461cf87a567d96d87ec15f6178ede9a4f73c201fd3fb93b46d329e8", "input_wav_sha256": "1bd2abe2d25279b851512a31db84a1dde608117ebd512c9979ca4ffef9ec0ee8", "input_pcm_sha256": "4d9f930e8f16eb7e68c3ec0abd521615d886e2a85dbc1ca0ef7668a8f3e51635", "output_wav_sha256": "356a4d6e64304fc6972cffb7a02f7ef9819926f82c13023d184e50db6e0cad44", "output_pcm_sha256": "c12aefffe74fe439a1c2c2864a8ec59ae7ec3748bf9c52fcbf2dee707008f62a", "result_sha256": "10d5955699bce22afb07b90d7fbaf8c05021abf4afbbab926387ca20fa7d8ef0" }
#579object_counting heldout	12	The answer is 12.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "2d97c181e2fd1deb8c3de03bf3d67105efa170c614003a6b66387d07123bc7a1", "started_sha256": "f65618fcbf21e93c64ed07b24b6ef925ab7fd41341161dc1a74ecd44e2437fff", "original_grade_sha256": "6536e5f5dbda7c091ca3c75c8cb5d8cc0c7944f0e8e5ce2f63ad0441d773e9dd", "input_original_sha256": "2d97c181e2fd1deb8c3de03bf3d67105efa170c614003a6b66387d07123bc7a1", "input_wav_sha256": "baa7d257b285b26bfa458de554b1a65da44b19624dfd8c3e6fbdde9a0e685b90", "input_pcm_sha256": "e978054b501305c410a4582841c664370ff23b0596cf5c51b3aed5184b52ccff", "output_wav_sha256": "290513e90e0b8ff7eaaeb348dde9a85ec8b5674bf5bfa14e353119ca6752d3f3", "output_pcm_sha256": "9f4b658cba4da713a1c497fa5bae30f872147be3e4d8cf275fdc5696b611472e", "result_sha256": "88bf57240c3c722ae4268a958bdee212f573383b9cb971d5c7fa3407df6a293e" }
#889web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "3bd48357b4ee1166185d771cf3dcf3e8eb4ce097d632c08251d625224ad223a7", "started_sha256": "9fb35362512be5d268f30452c95c9810249e8ab62fa1522e22f0421c4f9a939a", "original_grade_sha256": "ad1cd5f9f616970fa246347531a713cca2f90283b7ef1b3a0ca8ad139d9353ec", "input_original_sha256": "3bd48357b4ee1166185d771cf3dcf3e8eb4ce097d632c08251d625224ad223a7", "input_wav_sha256": "6ca79e558c044c3a68364731a1699ea2db3ae05b68509fda9ffc9c80e1b3bdb7", "input_pcm_sha256": "7d63bc5e2d8df0bf6aee728c6ee5f4ba20461e006e96d2e57ad8062f523312c3", "output_wav_sha256": "eb21ee6339d2f5ac74a074dccf2620954fe58abf928535b432edb9c2e59c3149", "output_pcm_sha256": "8a894b3597a742e1646846768e3a3356da3335d51c5fa784eaa44c7f787e7caf", "result_sha256": "9f7c50589909dd291f6afd00f0f8c7d83118f262ed395e0b1306eef9ec836a42" }
#38formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "171d5eb220f147e768832018399a449ccb5ec0b071489cf29550d702906a0d31", "started_sha256": "b4093eb2c9a9f8a16c7eb9d1f89605cfe27258b2861f47062bfbc0a795d073fb", "original_grade_sha256": "eba10114ead1d7f0b21bebd3e1b0c85162e683b32b74c1531ebfd7710da1a69e", "input_original_sha256": "171d5eb220f147e768832018399a449ccb5ec0b071489cf29550d702906a0d31", "input_wav_sha256": "6fb64009320a41caab8c2053ace9dc6ddc9206427fa33dfaee904f7a6a77c266", "input_pcm_sha256": "54e7d4401429d9708f45a03f5ec98982cda7a82c78528070af5489b747862206", "output_wav_sha256": "8353b46df31d454ed9fc70d83e6c0e21f9965c2a576aa1a21d24d99eb89f2ffa", "output_pcm_sha256": "55fd7193fd23f91707705f197f8acae85cd1bdfd92e4acf69208a75b32487f75", "result_sha256": "e3c9acd2cd630169e598d188e50191a9819b602c5fcb584089ddaf46c5ee311e" }
#387navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "11689915510dfc1664446587ddbce514287b7d9cc9ba0cc6784b60703a30eb32", "started_sha256": "31cc31a0fe6cefb56bdd38febbd535c3e7ef5a99098299211c05ac13d897abed", "original_grade_sha256": "a118b83ad67432260a0106442a2c8ed31a20081f991495f111e9d5b9ebc95603", "input_original_sha256": "11689915510dfc1664446587ddbce514287b7d9cc9ba0cc6784b60703a30eb32", "input_wav_sha256": "d5e8b1edc3764800d7b1a405f790d4f3419b78027e50b5eec2b66cc3dff1c394", "input_pcm_sha256": "f2723c04a6b8bfed62abda7ba953bd3ca1346b82679eae8dd6637769600126a6", "output_wav_sha256": "865b8a9d3a1fd4651cc0c2c9530ef99e01e54df4fa2e8fc079ed07aaeb8f50f4", "output_pcm_sha256": "f2210fc1bc3a624ccea010fc70de3cf36c0ea685470645d67760bf101ce2a93e", "result_sha256": "4137d4c7451fb3db2f24d0448d7ee5510c82106fb6bd443d0c7a3af9bf8de183" }
#511object_counting heldout	12	Dancer is 12.	needs reviewnot_an_unambiguous_short_answer	correctresult	Additive review The independently recognized received audio reads 'Dancer is 12.'; the model's reply was 'The answer is 12.'. The substitution is confined to the fixed lead-in phrase ("The answer is"), which the pinned small.en output recognizer renders as a similar-sounding opener; the answer token itself is intact and equals the pinned key '12'. The scorer withheld an automatic pass only because the spoken form was not the contracted phrase. Same class as the determiner substitutions reviewed on 2026-09-15. This is saved audio/ASR evidence review, not a claim of human listening. DAS test duty independent ASR/evidence review · output_audio_and_ground_truth Evidence hashes { "original_audio_sha256": "765b36d21a7e12a990d0b1118a24bf5e84867fc88ca1bd32b4abeb5504cf1375", "started_sha256": "ad069a7c39b1fac030bfa759bc3d9deec773d65448860b89f2c4f3fa8d484f16", "original_grade_sha256": "b43e3ec712f16b8b840ecc7ee0d423901c4a4da068ace01a2cd29dc9ad644091", "input_original_sha256": "765b36d21a7e12a990d0b1118a24bf5e84867fc88ca1bd32b4abeb5504cf1375", "input_wav_sha256": "695c0081c9ee4beb6c85f4315437081019361e2badac53b3fefa341044ff94ac", "input_pcm_sha256": "47589b48ee8e6b0c544b68695f7c792ccf08ac80016001152f8ad8a65ea28dcc", "output_wav_sha256": "83936bd1e84791a133ece98bb86f3075471038ac8cb036cd2a13bf2049fefc88", "output_pcm_sha256": "ab648a1aa0cb3f0da9774bed065b5a20fed29b2b7bb60d86a9ac5e774f929f13", "result_sha256": "4307c475fe38c3b95536937e33df8166b948a3f5dede03af355c11fef103a608" }
#972web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "c4ad4386fb8613179e39a583c2a6da5d67c8364acf5c28dbd527a708aede8c5d", "started_sha256": "4b83bcee6451a0ccb5b4a1e4cf69fb7b160c90fbc66d419381bff6bd57768c74", "original_grade_sha256": "74f33a2563085eb27db78deddc05e72119360408f5478855ef5544fd3c710244", "input_original_sha256": "c4ad4386fb8613179e39a583c2a6da5d67c8364acf5c28dbd527a708aede8c5d", "input_wav_sha256": "bda87fbc34083be333de0fc4f67936ffa75ab7784ebf58d42e8849db47089a53", "input_pcm_sha256": "3c9d9b1083947b45b6e29a7e506f4218ea687a02ea39af4aa90d6215aa3e6732", "output_wav_sha256": "c5b2fd6968201d4b9e2fbdba59593d81d5d56f7b1fd9855f0b3a4388a7ea047e", "output_pcm_sha256": "d1ddf99c9da47966a45a8b043574e6b1fa86717f7fbcc5d7777cfc096422a11f", "result_sha256": "ecc6553ee64aa3029a172c9a92c2e801584762cb95cd999b36fe27692cd62deb" }
#143formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "bde06a3d968513676c679b737919654e4a10a33e79249240dd79d1c9e0b6daa3", "started_sha256": "e5fa76477f2a6cbce289d72330a36f1abb751af0ca640ba0d39930585838b8b6", "original_grade_sha256": "35ba2e2a3b647784e4b55185093a42e99e35e08549726528dd68341d47a28518", "input_original_sha256": "bde06a3d968513676c679b737919654e4a10a33e79249240dd79d1c9e0b6daa3", "input_wav_sha256": "df030b291db14ff2d5b7c1991876296ec74df000b7f4765ae73450baaf646986", "input_pcm_sha256": "be841d83c66e963a8c14dd5fe89dad4915db69570da02136ab1e52e0f551b4d9", "output_wav_sha256": "76f7f49b4330c2cbb8e3938af1c070ce379c7d6fef889fbce3cba882d3a72b4b", "output_pcm_sha256": "579c2140760abb5a8f3be81638a3b5e9e0ab192c32058448e3013d17cb937973", "result_sha256": "6d0f55d143ffd3f3b688f539c61e4b1f64e0bdd4791bbed8167772f387673841" }
#404navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "0988100b4a412470c5f9b0e0b3748abac3803eb454151ce57e0b54572794ad6a", "started_sha256": "db6a74e344e38863066c9658e520427f5427eed6709db00b77dde562acb99c50", "original_grade_sha256": "73a9dbdcf78fb0e41a55e25a41d1a02303c19c3f5070ad853273483898b4ed63", "input_original_sha256": "0988100b4a412470c5f9b0e0b3748abac3803eb454151ce57e0b54572794ad6a", "input_wav_sha256": "6dad848d778aceb2193cffb9376e167af2c15efbdcfdcb838371fd4eac07c1b0", "input_pcm_sha256": "7112667dbca290f1d26f20b5cd79a0e19845eee7548107f66f5f2ff39bfb474e", "output_wav_sha256": "79398a73bfc844d9ced20049c430ee2421468b402b4c6bb3bdc7e27c9792d6aa", "output_pcm_sha256": "1310e6173f7821e434bef832837fa725ae5ef40a3619f21357827f60f72dbd8a", "result_sha256": "46688562e49eaae42851a6e8b93ca50b88fc7e89407cd3e726a021d024c5a073" }
#545object_counting heldout	4	The answer is 4.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "4ed4cad046113c58043c2385c77ccc8539249b98b9158d300f764bf9c59b34db", "started_sha256": "e6b41b92f7d66282aa2afaea01c8b3abdb323b926353b63e540eaa0370830cb9", "original_grade_sha256": "55058e850c30e96b08449b70dd077199f4ca9b2ef467eb5aa1298abe84c76629", "input_original_sha256": "4ed4cad046113c58043c2385c77ccc8539249b98b9158d300f764bf9c59b34db", "input_wav_sha256": "3d166d6c2c6058540f8edd0d825e8a2e3278bbf1e841cbbe1c826659cb754161", "input_pcm_sha256": "fcc4fb52990fa34fa706ffe8c0a9311200509063621c03be835b40949dfe3977", "output_wav_sha256": "0636e97694522de66e784a95ce7953fa5c38f519efda536219eb9f8ca215fd7e", "output_pcm_sha256": "f9c95f8a8e7e2481528ac971c56b961e3e47e9c9dc0cdebb63737d544f4ae82d", "result_sha256": "8488d9d2eb9dd010269cb1f2fc7ec23e7620e025142ffd323487191ebae529fe" }
#837web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "8c4e7d321356b67d6d3b1783e4e44b150d9dc04c6ebb566829ed26a4ad5f9089", "started_sha256": "cc37a40e080688724a4cde89ac04dc4325384c6578d39264746e2832ab784ce0", "original_grade_sha256": "b19af7a30848da50239dcaa7f5c3f26148845295d80d2f483c01272de2d320cf", "input_original_sha256": "8c4e7d321356b67d6d3b1783e4e44b150d9dc04c6ebb566829ed26a4ad5f9089", "input_wav_sha256": "4bc82ce4eab322818ea896d74c1ea2de7497ce0bb9a324cd4320058a7b114f07", "input_pcm_sha256": "4a141c9a7c511c06d161ec7504f64b27abb50c95f6762fb16cf2a6ad717e49d1", "output_wav_sha256": "45e74602a7df20a02b577eae5be69f9f19a8677a1fb7f79a3d86394d770a49e2", "output_pcm_sha256": "456e4f21124285257cc4c86544fb2b9637f1d6710777c5e4790cc4c272b8f83e", "result_sha256": "8fdb7de017e0896354a41c1ae50cfb09a872e14f37ac63e083706699f2a66e81" }
#175formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "21b57d6fb8e3878cdb1db5613c7e170548f010e4ee9d45525a8ee599e8b93594", "started_sha256": "9a00ef0f0d97787f1c15cf6943e726468c16057461f7a95e265ab0316499069e", "original_grade_sha256": "ffe5e1b61f3b9eab90f3b1b98f36ad18672dffd062a57305db3ae1118c1cdad8", "input_original_sha256": "21b57d6fb8e3878cdb1db5613c7e170548f010e4ee9d45525a8ee599e8b93594", "input_wav_sha256": "cca5fbe5dac74e505e4549589e6117de37ca584dc11f2d9a4dd129ac50a30e22", "input_pcm_sha256": "25ba43f0bacde02b972a8e2cba67e2c9ad8b35b5bc9283f2472e150e4fdb13d2", "output_wav_sha256": "20ac1c0e1d398c4076cb9b815e0ebe9f767bbb3135793d482053fea51f8fccbb", "output_pcm_sha256": "95127997fb2a999f2294abf9af5243c3fb6c99ce1a23afb79692b22c1ca031a0", "result_sha256": "77259b91dc910c7d069013cf3f03b6916fb757c5646791bd3e8cd1cd3d8b888d" }
#423navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "db7fa853a436489785a5f7fc1ccdc4a6f953cf01cf7c1b07e0ef3c7a2cbb2ef0", "started_sha256": "2d5561ed7b31c959ed82da427f70909c116334da2a6272f059aa830ba7f0433e", "original_grade_sha256": "f73cfa2267f8a6e917b9201682eb2db796a96200721ef10532588b142a1cfcd2", "input_original_sha256": "db7fa853a436489785a5f7fc1ccdc4a6f953cf01cf7c1b07e0ef3c7a2cbb2ef0", "input_wav_sha256": "429320dada3bd18d858fd31f7a077ec6a5bf0719aa55918565055b1eb7054ef1", "input_pcm_sha256": "19174f5ec5f5cddde2a41c7bae1d925a162dbed4605495156c2c1e4069fa7335", "output_wav_sha256": "75ebf4a2a5d091a0a79e683c3cf6d4a89f8a1291271f104bc59a1a6699b6a7d5", "output_pcm_sha256": "08741daa3ae6b3549439adc7004ad5753e4c7b8b718a8271f1c0d5881412352f", "result_sha256": "2f6e19b0ba85a5ac6c01f551bfdea5b59e5d550b2da83279cc5ff46ca6fabdac" }
#597object_counting heldout	14	The answer is 14.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "a3822cbbf7c9d4244357f467ad3b9782f6db951c4fa29b3d4a24c3079796bebf", "started_sha256": "f2c52c940d5ff9b82ae51786bc279a62b34be76e35140bf848074961a26b0f73", "original_grade_sha256": "407c2709d5fb400646de9d92e571c37096f714572fd5e5d800a297106fbbbb8d", "input_original_sha256": "a3822cbbf7c9d4244357f467ad3b9782f6db951c4fa29b3d4a24c3079796bebf", "input_wav_sha256": "b7744b17edc22b36e923c50badc5c51b450df25d9ce3870783cdbd7d977e0fec", "input_pcm_sha256": "fd3f64161e8dc8c4e769642c7921fd72ae0e85834143bfa2c550abf7f1c3691c", "output_wav_sha256": "d5506f869668660c3f0363a1fd22baf02d5e0c785e62a0ec2b833c73aec4267f", "output_pcm_sha256": "04da3116419b6ac7fa4ca59b4162cd5c9a41d2b704c6d77f3439088eb1a82fc1", "result_sha256": "a9722e5e766a3d5f9285950ba86d30ca5fd00374a43ce9a4883e48c43dc57589" }
#925web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b10c9bd45a8631c9a94dc2d9dbb9868ad311b856d0184a4f8a7cdbf42dab618c", "started_sha256": "541b00be5c51b8eb20359acb2d8ebf370d46e937da847fede1f07e8d173bf716", "original_grade_sha256": "2ea6af8dd351150ea2ddc3ebb1131bafcd5ba76dec409c2ef4e9b03682981445", "input_original_sha256": "b10c9bd45a8631c9a94dc2d9dbb9868ad311b856d0184a4f8a7cdbf42dab618c", "input_wav_sha256": "fb63e462663f20d4881df70220097994a7c6975dc79a998f66c3495d152070f1", "input_pcm_sha256": "748668f3ed133746f95fb2d7efc1b4516d818e7ee266e4c55dbc8fdac872dcbc", "output_wav_sha256": "b5d2f5ec9337ae1b609f5c688e8aa213fa09c04998f8f871909ddc98fc8b5218", "output_pcm_sha256": "0aaf03b924f1c3fac9bed3c1776667fb0c606b6e87da8f594ef5af37b3976263", "result_sha256": "cbc4d9f1e949d3dccbcde4f29b0972b2ad557d8a3c40b795c61d57e7904b4687" }
#6formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "e9808767d9a382b0fa1fca60f61eca1f55911ba77962cd5f1fa7a6b19143d7a7", "started_sha256": "70520066fbc79967e011551e9adda24e820bc601442070166d141cdb1bba3b14", "original_grade_sha256": "440ae546d4013b5842ea675d2304580f168868bf2be22f4f5e728e5dcd53e31f", "input_original_sha256": "e9808767d9a382b0fa1fca60f61eca1f55911ba77962cd5f1fa7a6b19143d7a7", "input_wav_sha256": "66dd7b5111f3b481bd6b00b9950506b083e470df3ca2ae43b3d080b24789c2c6", "input_pcm_sha256": "ddd2dbeb01ed8df20dd40a0ad35ece1c6965e1fe553d11b1379318cff6152d4a", "output_wav_sha256": "bbb3430cd10b778f30cbb9ee61ca7fdca6dec076272c4979d400c788333bcaf8", "output_pcm_sha256": "121e12a21d2df39cf1f773e279a445df16279853a3f4dc656b8311189293d554", "result_sha256": "f508fa8e56c513faaa0d2fce0bdf2b9611a5a40812c2b2ad6821338cccfac5a8" }
#443navigate heldout	No	That answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "0f3c50d1279e1ce7213e142b7ba09f4029004a5af789a5cc883bcc9f97f66db0", "started_sha256": "a8a2afb76d9802a7e92fe7b589aa94bf3320ab5005ffe08f39b273ba6aea8000", "original_grade_sha256": "dc79fed8a4f0a45fa93253ffbb42969996813a88391d86d1b9394ad90046bc03", "input_original_sha256": "0f3c50d1279e1ce7213e142b7ba09f4029004a5af789a5cc883bcc9f97f66db0", "input_wav_sha256": "489fc8d82f5d24a8a532faf16a65736e12f131aaf6be1222674c633be25841b3", "input_pcm_sha256": "d4aa8c78a405ab2bad1e4ded6891ccea2d96ccb137aaf5bcfb171e6cdfe44f91", "output_wav_sha256": "c9afe182ab6228fad611a2cda1d5c02a6df19c8a5b3c25ae93eb2d7e70d443d8", "output_pcm_sha256": "7f11e7ece6d69159309ba1fdbc3fd5e551968bbde84d05c649f83f6c63049a30", "result_sha256": "e3d35987923bd52c898028054f46164a129b9f3fcea95ca8acb795bfd34ddb43" }
#526object_counting heldout	3	The answer is 3.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "bf268ee17e9a6f27c7c38392ec5218e33039e630bc7eac4e775ed8b539f71128", "started_sha256": "2b3ee23e8229deded899cb5d13f0e4e8e955b5e265d7d106714a9078025809aa", "original_grade_sha256": "237474ae554ef2b2ae077df65b6d6bdf05b9853ad70c123a2823cc894b1af23d", "input_original_sha256": "bf268ee17e9a6f27c7c38392ec5218e33039e630bc7eac4e775ed8b539f71128", "input_wav_sha256": "05888f18ec469f5807bf099b4d2ca58542b1afe47f786141ff984874dcc6faaa", "input_pcm_sha256": "a4ccce5687cdd1f33ca86f168168c8b5777df9458e7f090f0a13015ffe9bae18", "output_wav_sha256": "62c3cd3b34cf7a26ac1236e86c853ca871bd5e3303cd872ac60f19c81d5f78ea", "output_pcm_sha256": "214a1c496b699d8b145aa46202ad122b81f13e1ef6032e403069cc906154cb6d", "result_sha256": "6dbf6189667ba0fe0a047dd16578c68957f19c66173107e57e8b16099f988afe" }
#978web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "471757d8429cd183a34253acc3fa2a35d8c560ebd5c860e63ff5ddfc25a9a037", "started_sha256": "771161ec57621367333d17c7af3a423b691be260b66bb08a7004383cc31e1a67", "original_grade_sha256": "6e5f4c9aefbad1f39b87edafa01e319ecc6dfa7f9933c4cfb002cb0249e9ff6f", "input_original_sha256": "471757d8429cd183a34253acc3fa2a35d8c560ebd5c860e63ff5ddfc25a9a037", "input_wav_sha256": "d8ee28fa7a2ff67553037ff19f52aa090fc8c5281af550c4cb6fc4d1721ca1b3", "input_pcm_sha256": "0d0f33ff1ae6eff2729b10ba6544cae2ad5a2fe918b9d13a826632b500bcf6c8", "output_wav_sha256": "699ebaf13fdd9c596ead9a51606074d7ea2fec5a3536284f593598ebeb0e3da3", "output_pcm_sha256": "fd3e689cdd6cb725d1abe308768d442e3e8a5ad6a01bfc655393d6db5f425515", "result_sha256": "8b8612a34299ba2dbfccdcc6b0a6e73b068dd56cee9c365dced786e96e57f1f8" }
#63formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "9551c58140c22038c41c94598afed32fd3a959997b499fac423b924c18aaeb5f", "started_sha256": "18d70a37d87abe421df209a4c21e1ce3b8690069632808ff841b4eb855b5fcef", "original_grade_sha256": "9963440864050aa0cd6f310394d0f53e423e6205bd43dcccf3b261eebbdcc74f", "input_original_sha256": "9551c58140c22038c41c94598afed32fd3a959997b499fac423b924c18aaeb5f", "input_wav_sha256": "c9e4fff379fb3baf5f7648b9f7d70cac09f75c5afbff7189d63b15a6b9930d46", "input_pcm_sha256": "8d95959c322f49d549b9fb2169ddbe08ce82916215cd409ee432633b06657cf3", "output_wav_sha256": "77d182073b157900198a584714b92fbb0e1ba0dad6e091b9392d1f3f541a8e2b", "output_pcm_sha256": "d3138d308ede6e3e39b285451b25729cabf1399b9e9a356dbe1de350300aa37d", "result_sha256": "3094e42e8a0dd60cd7d4fc6d540622a351d82b8d545d564d435a4bb196030db0" }
#473navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "927fa48f3b7d0225f1d1b3a7aa1a53f64bd16196c2cfe470c4d9b274ecdeb908", "started_sha256": "417002e774cfe6b71ae56808c0293069ebd10ca8f62b013a9f1d9f7b6909d3d7", "original_grade_sha256": "ed0494fa56e82ef9025f2356e102c8540e85e2d237902198fbd6e3be465de5ed", "input_original_sha256": "927fa48f3b7d0225f1d1b3a7aa1a53f64bd16196c2cfe470c4d9b274ecdeb908", "input_wav_sha256": "b93ba5d1d6638a5c27862992f9ef2fd3555750b5849589e01b86e1feeaada301", "input_pcm_sha256": "2d5a9dfdff8b2853fd3c6dd4544e08685adfdb2ca4cb74368ab80158f0c906b3", "output_wav_sha256": "2adc3ab69194d47b980b1e1a1465df1cdfba35892f2b273771769ab62d2331bd", "output_pcm_sha256": "e159923c351cf1e28f5922256add14f24fdf0129cfa3742798d38837bed4b7d8", "result_sha256": "e5f1669c89d3840de4609a49148f52bcfba6356971feee7318b6399202443445" }
#573object_counting heldout	11	The answer is 11.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "92022261081af3f17fd33f9ef125cc5c29e5aab0e1527817be773f2d8b6daa4c", "started_sha256": "4ad80d69425493fccbe2be416012e6e43138f36486d3404b3e50738082200976", "original_grade_sha256": "037cf09026ed3d348c8555926e728dc899c1bc2751b31261d1430f4a0a98d4bb", "input_original_sha256": "92022261081af3f17fd33f9ef125cc5c29e5aab0e1527817be773f2d8b6daa4c", "input_wav_sha256": "2c1daf1bf984e3a8dd6d1ae40aae5ebdaf42c903c29a4e4fd051557f7f857a13", "input_pcm_sha256": "e4235b80a8de36f9b37031f40408cd7d6551c6a46bd29628a5bf2297e352d0b7", "output_wav_sha256": "40c750afcc6a2a5397538da84cb8beb56bc709b99b54d8d898cb470abff953c0", "output_pcm_sha256": "ca3f61c2ca4dce6d50a06c16f5a95597832daa9ae16876fb381bc91d2aa1c822", "result_sha256": "ddd48c42fbe239162f7b54478e510d5925cb798fb5c61e9e5380043673fba1ac" }
#807web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "18d91b7478d1fca497e31a38c8cca9e69ac7d1df4ed0ec5346ff6c2d68fc75d0", "started_sha256": "846b8f2a56a1618c86bdb84ce0dd165fbae4e44656cc10b54827e89053e88e6b", "original_grade_sha256": "b1eda39f0873fe3fa4f14bee6a2cded360a5598c41383dab756624b13ad5a81c", "input_original_sha256": "18d91b7478d1fca497e31a38c8cca9e69ac7d1df4ed0ec5346ff6c2d68fc75d0", "input_wav_sha256": "26c846361262596f835ea6b93153728734a66b6673a8f5c1afccde773712497c", "input_pcm_sha256": "36f722b270fb49b664da05c7d9630e6a6d8113fea528b2c1f5f8c4c3490fcdea", "output_wav_sha256": "8ea3d3dea57c47eaa18b12c72aabed7fc1be9c989bceef9ba96bb97d00f03591", "output_pcm_sha256": "4d76f0d67f59b90337aa03da09073a303c4d6023b74681fe40d60b1547634388", "result_sha256": "82d17c6a84f13c48b4f0f62a942e1c3e2a2f31d2e6867bd5d5da544e4551ab31" }
#149formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "75c15d6dd4a03f433a9051b176484e51b63027a472dfdcc681df44922873c4de", "started_sha256": "eb220342d5605a4335cb3f27a0437f6c95aea55574bb601fcd61a1f091d63d92", "original_grade_sha256": "4f3d49fe6c520b431eed349a8885ce3247b1d6a3153dc86aeef3f79304648190", "input_original_sha256": "75c15d6dd4a03f433a9051b176484e51b63027a472dfdcc681df44922873c4de", "input_wav_sha256": "f11e711b488aaee824e57675c431730dffb1654246fcff8f4bdba493f95dd340", "input_pcm_sha256": "867f4d92f710b716882c25ba3ef0babccd73350272cdbf2187bc570bc4f85c68", "output_wav_sha256": "bfa156da564b9261a2f652fe5fc521c6562061216db12f66f4bbad8f9d18e39e", "output_pcm_sha256": "bf3c875889c359bf86471a86f529b3f3887e34e1290e424bace0db76e573ba09", "result_sha256": "9ebfa09e99d1e82e0907b753639c837b3b9a7592f72909c14e65d517b0ff29b4" }
#362navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "90d05be323669b60861d47a9908d42a0273dd3da0cf9a82410099386a7e612c4", "started_sha256": "af379adbfdff622624a7df61849ee2ba57ca6ab4b52b8a540e19f865cd493324", "original_grade_sha256": "f861f30fcadeb83f723f6bd709c8e0a8f5f0093ce0a216b3abc06b5b99814b3e", "input_original_sha256": "90d05be323669b60861d47a9908d42a0273dd3da0cf9a82410099386a7e612c4", "input_wav_sha256": "db61d6b1acdd73c6132f88fc5d19fbab7bfc1cb4f363d37dd1dcd9f2f77913d1", "input_pcm_sha256": "913699b8eabec8765b20d9fcbf37db42fea861b834d74a3e812bbb588297b6b0", "output_wav_sha256": "3984a0c615edaa2c675e5a5584829406adc227ebf1eb6a64624929378b49dee1", "output_pcm_sha256": "6adcd278d89eb16fe35699ccf2476d528c267dcb4ac45bdf43b595f9c9403571", "result_sha256": "d492280b91ba1a76b34817f519b446a9e0e54387045463ce00a4983ddfc8360f" }
#708object_counting heldout	10	The answer is 10.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "cc0ba62eb754e6b75c3b404ff6583a0cb32cd19ebb637e3f07e958a778cee1d4", "started_sha256": "1eeece4b71bfc90b7215bcb0b681a96bf21dd802ea63615c40afe451fb3a9439", "original_grade_sha256": "5522bfad2fae9f03a449ff0f9fea9e96e5ec733a558103ddab91087798d02bc8", "input_original_sha256": "cc0ba62eb754e6b75c3b404ff6583a0cb32cd19ebb637e3f07e958a778cee1d4", "input_wav_sha256": "c6983d22a64b419ab1b5fcd38ba2859ba650e790e7ddfba57eb90050beb7a724", "input_pcm_sha256": "2bdf32c3c3f9dc66b2bcf9a0921d426c5be322dd007505dd6eb2a5607e6d40b9", "output_wav_sha256": "dcccd7600a2d855393170faf97eb672c57d41f952927786be46ec0318aa560e2", "output_pcm_sha256": "0d46bed0647f18978c4b691fc7f3d3aec8410b7fdb2993eb3da9b9f135e20ecd", "result_sha256": "09b8f8aea0c710d7fa948a6951df41679d5ea341021d1f6ca4e4c0d0606b4f9e" }
#761web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f5baa6e8fe7adb2d512a029a2db3eb18e8c9b5cf8c947d9ae645b9c464066a32", "started_sha256": "992a952a6e911e1bcc7c71a012573d878bf3c6a10771ebde322cdbb5229a3279", "original_grade_sha256": "6d560ddf9e09fcee4a12c5cdf85cdcca06e6ee4cb7e668555b000d9a25004e43", "input_original_sha256": "f5baa6e8fe7adb2d512a029a2db3eb18e8c9b5cf8c947d9ae645b9c464066a32", "input_wav_sha256": "b6212d9456150ecfac2e92b71d6424eef2c833430c34bb11c94f86570f2e472d", "input_pcm_sha256": "ce25295966d05f91e10eef421e2791df6979cf355d6c3c87007185dc934f1c73", "output_wav_sha256": "52ad439e390fdca7ab29c7e34e53acb765624b59c95f73f048c09e11f18b5154", "output_pcm_sha256": "75a575e4056cef095e22ca60876221da19de125133f96faada59a4cd79acceed", "result_sha256": "ce0f286c070319397e26c643be9f26a204c4f99cfef5fa19fbcf68aa74a0d3fb" }
#129formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "6f59cc28fd5ca5e4c5489744ea67fbf8aec951048450c1ee05ac613d34fee215", "started_sha256": "d7934106834734d1a054a350e4e4e78baf916d3708d4b3a95c5f9343a25ba00c", "original_grade_sha256": "44675159bd7e223ac7e6fc7d83a76e477013f409d48e7d9f5c54bf80e8d65b39", "input_original_sha256": "6f59cc28fd5ca5e4c5489744ea67fbf8aec951048450c1ee05ac613d34fee215", "input_wav_sha256": "f1f727fc1cd6c1a73d231445846bc546feeabdbf6470a10ccc5100af0e6aafc0", "input_pcm_sha256": "9280dfc93bdf4999802e58eb6897766a1fa0194ba15a452e845c811891ff83c0", "output_wav_sha256": "61994b5c36f91fbfa57d960e72ee49f1b7065b9bc150f61fac60d9cd378a928e", "output_pcm_sha256": "02c3b778ee15f967882a6fa6106205402044d37a27fea555c34a5764b512fa62", "result_sha256": "fabf4745dd36a6159a740ddc2ae4b91d771a19d49e2f2a581f360f0f969051c9" }
#252navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "deeb55775046bd1bddcfbe66b66dc7a999fd64a8bd6dc3354c8d9c2a31ba4820", "started_sha256": "57a484fc373794b6e514c882561e4d99380afcc3ad456ba29273923d661f48e8", "original_grade_sha256": "f093b8bfbc4ecc2dbf659aad71f48ced17615bc257a0b668592b702acd8f92cb", "input_original_sha256": "deeb55775046bd1bddcfbe66b66dc7a999fd64a8bd6dc3354c8d9c2a31ba4820", "input_wav_sha256": "ef00011f1dc6b796db57b4bfb9e949f51b90b3748713af0eb6f2daecf6b83017", "input_pcm_sha256": "d54074c2042ac22ba87309a78c691c844265165ee32a2a6627d5b464a8272091", "output_wav_sha256": "02d24b90e9fa26fb98c88541d93f694a669fd93085a92687a2503b42e480ac05", "output_pcm_sha256": "d6b1c8ad5d1c8d6ed4fda310e4df73367621c66210900107f3584f56c3dfaa10", "result_sha256": "df57da5f319e4f3bb8ed06d1f5e98e29bb5e3c61ee3a67728ec4067ea5f25921" }
#667object_counting heldout	13	The answer is 13.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "c028e71ea6a679e65a1538c0ed7b654a9d562bb0714e3c39f3eea51d64f33770", "started_sha256": "e2b6275cde4975e6f58032cdfbc8c91916046413efa319107e1baad74cf3d310", "original_grade_sha256": "20736a6a6d42c4c5c6acd7c9359edc9b11577f9d1ac16be08a4ca620827492f1", "input_original_sha256": "c028e71ea6a679e65a1538c0ed7b654a9d562bb0714e3c39f3eea51d64f33770", "input_wav_sha256": "427d9e4efcfbd063e824069fb53714f3c9bd92ee506c3fb42a69a15d7a09db13", "input_pcm_sha256": "8f232e13bc78520a00bd57fdef207ff3c8819b03702c461bb1d63ebdbfe0ea0e", "output_wav_sha256": "dc9aaf3fcd1417e05da9f3060e537efe45de51891e3e4535691dfd4e53a770ab", "output_pcm_sha256": "64fb080c8546dec657cb898b3ac37c9479f9d8d6df988ecc6ac62128953f5f12", "result_sha256": "8665270c8df1d272e5935261b428a1461d6350bd4e56ff43c85b7bb1aa3281fc" }
#898web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f8822eb508a0e986872104b7eb3e6caafddedb66289c5d7df05be79efe037a61", "started_sha256": "9a87c7cf49612f68835d41b8305bf195a7c4e9ecd2fd4d70dca45eedf1f25f50", "original_grade_sha256": "4048b831c4226da11a5cb9ca44688f00f63a122c764e58f1db595b542cc5f2f3", "input_original_sha256": "f8822eb508a0e986872104b7eb3e6caafddedb66289c5d7df05be79efe037a61", "input_wav_sha256": "d0d67d0a0cbab3b8a559e20289057d670aeac87aa1a2b9ec1a0f488a09b297fc", "input_pcm_sha256": "0a26a9154bf2814326e2c9a618a76a273f952dfb6fd49e10ebf1b3744a313bdd", "output_wav_sha256": "99756c34ffdbb006141c823c0b87f28c0dec982443a87c78d3e2d0e48546db50", "output_pcm_sha256": "206b658c3299194cae026c8f128ed1488a8e40be0394d904dda77bc83566fdc3", "result_sha256": "7f0e9de20542fb9ac007d4fd8c7a7e88bd81196c81733e234a4d3f5a4b8a0431" }
#230formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f75a6830edbcc16ef546506b188192ac72be7c0c60b311b9d8333ce7e937b82c", "started_sha256": "da9524c506bc9ace025101a65a96d4f1646e417062b4114441da1ece5cf7939e", "original_grade_sha256": "fb07b4db0918f9e8b505e9f41f3f3c849b4b07b077bdd9cfa6237a90fea1ae77", "input_original_sha256": "f75a6830edbcc16ef546506b188192ac72be7c0c60b311b9d8333ce7e937b82c", "input_wav_sha256": "252697f084717e722dc57309d57273393787c10c88bb1bbde8c8597c126e4363", "input_pcm_sha256": "f7d9a5f63220decc58c2c7967830bd408e2610e0a88feb388184b2853fff8afe", "output_wav_sha256": "40293b7178adbfa367f75585c8196fa614fe2f4ad8a3f870d785e0225a299735", "output_pcm_sha256": "ba132c7cf0ebf5bf5532197fed109387821315efaa4327cc30c58e51d006e456", "result_sha256": "63c4f4b73b9b770d004ed70dc039618624bafbe2cd9dca36c9fc2ea388fe0f1c" }
#435navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "a57dd109216bf7322d062bf6f68aa394c0bc53ef050f57d58d3d397413e8cfa9", "started_sha256": "a1d65b231ee0b5b75e15bbc9103786e5166068a298fdc9a9f523fd27e54d109b", "original_grade_sha256": "0e44277ca785f6d97f05a3a09466aee5460b6dffd12c72a7a8ba33dcd38d3019", "input_original_sha256": "a57dd109216bf7322d062bf6f68aa394c0bc53ef050f57d58d3d397413e8cfa9", "input_wav_sha256": "94abffa029bbfc96214c94d8ffbd609071844df035668e8e75b36352e9dc8990", "input_pcm_sha256": "c7ed7b42f8d7cc7fd41fe9b7e61cc19da349d43c24ad41b5faeab21794d90c05", "output_wav_sha256": "c0d007d6c7c3eb211ac1267be2a0732a45ea2d7bd9309608bc44faa92b97a57a", "output_pcm_sha256": "1d8c693ff30130b00ab16e7fd1ac0461adc92615ad23851b6d2952f78f7e3453", "result_sha256": "a0d454b15e1f305dce744c2f63a6f0fd3e1a462a8c23b01b811e0e79f36a2906" }
#640object_counting heldout	3	The answer is 3.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "27808642d6663ddb8944ae71728b76f86b2b17c7989429ff14cccde298005a07", "started_sha256": "f948c2126de9276e82077fab5f5a239a8b34131323628baa4d3a75f633f33104", "original_grade_sha256": "b1d98ac5238cba4fbca80236931e4c0f3b97696a5561d64d6506937a9d178496", "input_original_sha256": "27808642d6663ddb8944ae71728b76f86b2b17c7989429ff14cccde298005a07", "input_wav_sha256": "9b9835ec8d469a2a8133ba3ef77aebd6f3697379af916884405fb9a7d11e4eb0", "input_pcm_sha256": "8efe37c6d3c6235b3456932db58ac82d4415f69459c18d23da983db40f912180", "output_wav_sha256": "39b7f2edb00d4d39fda3ef3025893fdfb8e6061c9bf5a8b001be8787e8fbf3b6", "output_pcm_sha256": "ba4cf63c4c8ad869f52d0340ff51f6370a97bb90ac858cdd19a4df656afe2ac6", "result_sha256": "dc9889286c8308c20fd150b16b01ac15cb4a1c306b1c28fbeee24055e6dae072" }
#918web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "984de1d4090347057fbbb4a12c6d21428db28341078021e167f74e6bd54c724a", "started_sha256": "fb3669b9eb885786d4486ba8a44f348bea360bd5503e302cc350d8bb51bb54e1", "original_grade_sha256": "5ca1fc9feb7edee4d81c9d804247cbc0c75d6aabc8c7f7f6fdf236ebd6eab644", "input_original_sha256": "984de1d4090347057fbbb4a12c6d21428db28341078021e167f74e6bd54c724a", "input_wav_sha256": "f96eac52a78788c52783a8d3d7de28023304550cb154da60d112512256104e4d", "input_pcm_sha256": "af3ee402d30e4af17d4d68fea9840d11a1a605824544254144f1fe44065a7588", "output_wav_sha256": "a89ea428fe9769e2b606325f97146e6c76ac540f90916cb837dcb6fddc944222", "output_pcm_sha256": "29d1c497e44c0ee454ec2743f3da0e701368bc0a057fc990ddeae6e2305e91cf", "result_sha256": "81df3deb4e677c39424a7c0e0ffc038280a1c95177b0bf828209b2a26267e7bc" }
#190formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "6c158868965dffe56e69cac3c58ac7a9440461ec7ec651217bb479a1b971321a", "started_sha256": "0186f873c9826b8e655870bb8f23cf6dd3a2c1488235e7deb1ad4b3da404246f", "original_grade_sha256": "b500c7379b3cc91e23470758c90c54e10fa4c726051506a6a324b9398f8a690a", "input_original_sha256": "6c158868965dffe56e69cac3c58ac7a9440461ec7ec651217bb479a1b971321a", "input_wav_sha256": "f3fe4e26b46faaf342640242eded055be31948ca9066aa7fdced22d083227ac4", "input_pcm_sha256": "418e5052f74ca6900e23f24539c06290a7c34a6c173248ddf798e44e5f0d5fd1", "output_wav_sha256": "418b042989a106c7b36752912f79cdd9e7cb097166c7943fd6532342a6b9319c", "output_pcm_sha256": "a950f45100dc5cd51ae1df0c1428dc363c4d245a5f8598770ee16c5d40246e76", "result_sha256": "0c2df8d59935a3c7f3ab69633973a64a66657465c92491a91d6fb445f98ae32e" }
#436navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b94c5725505f424ac8a235a2c01e35191760b3e6afc4671a4ebb9a4641e5b5a4", "started_sha256": "b36ec60b1e5bc6d2a3521ec771370a48c47bde89725d8faaa03fd2e08c9535ec", "original_grade_sha256": "758e65d488b81f084cff89cef2f1d2efdcfbd5a8b08d317e6569f35224f68670", "input_original_sha256": "b94c5725505f424ac8a235a2c01e35191760b3e6afc4671a4ebb9a4641e5b5a4", "input_wav_sha256": "38c6f20f553558c377729d441e698e8a06f2fccc8062013484fd27f8d1fcb626", "input_pcm_sha256": "02f3ce0e857ab509d471205c91929861671d80eed1a59fdded82a1840d80d177", "output_wav_sha256": "8852268af85de3e3c7f486a9422ec89e524a1335048db6616eeb53fd32a122c7", "output_pcm_sha256": "3a13c0786e865d02e9b34d74f4c6f2925a8636f9ec15124bb3a3d3cbe7900d11", "result_sha256": "1c0c38db5478e660c6035465da6c7d342475b20340e0b4db8f5e115dae017de9" }
#649object_counting heldout	8	The answer is 8.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f021791a725e6cec9a1a64083d701141dca82cf042b6887ac808fc872fa8131b", "started_sha256": "98039abfa9608a0028345c82bf4cde3881037f102de5f274dae4b4acba4a7e73", "original_grade_sha256": "f958cc165488ebbfff64351cf6a2deba422fe7d9ca4f063e05dcea8af8bd6423", "input_original_sha256": "f021791a725e6cec9a1a64083d701141dca82cf042b6887ac808fc872fa8131b", "input_wav_sha256": "da6892015f5f64b4d9ae58cb4d170af71c9c7b0b139547999d1369cf3400da69", "input_pcm_sha256": "30dd63374969e2b03eb752b1acaadf7caf800d5154a34d2fa30fe2fb8a3d72e9", "output_wav_sha256": "a78de3c585710de8f78f4fced78400a39feb7d785c8d0bdbbb7808502d5ad4bc", "output_pcm_sha256": "c6007e7120d89ba4683337f3b12f2cce641c7816b56d19a151045bfa03b8dd9f", "result_sha256": "29a9c144c876891f26e98d424b5d2e0b538a1463a6f7fec7e1337f4d05112ef8" }
#954web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "4405c725c11679dcf6bb5f47943ab80e7b9eadc0e2317fdd7c8f0a23026c1902", "started_sha256": "db785978678e061635a7ee8a6c62664f1bffbc42a36ce6f17ecd62b1ac063e7f", "original_grade_sha256": "fad80f1b9e492874706f901a24b3733c0aae3051ef879c1ebe19eb2175e3a54b", "input_original_sha256": "4405c725c11679dcf6bb5f47943ab80e7b9eadc0e2317fdd7c8f0a23026c1902", "input_wav_sha256": "d943bc395e6e2e428a2c5a6388ceef78ea8690ab0a17b2fce501316dc9f879e0", "input_pcm_sha256": "1a43324129accb5f0c03a28aa901d20d01a347e5d65508a20a999d7b280179ec", "output_wav_sha256": "36982e662a1f87a89166446e6aa412993579e7b00fb31021c9d4ff7097b4dd8e", "output_pcm_sha256": "ce36bc392984da8d94aabffad7f9bc157d85a94d20fedccadced36086cc61d67", "result_sha256": "98aeb1fcf5f8b4d59dc556303339f950ff76d1b8eddfc7b5b3f6d3286760d47c" }
#232formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "3990af9aa9461771d209462dd85aadcafd76f5ff1bd64e86169ad1cf9f66f2b1", "started_sha256": "1cbdea1649c1457f7b5d192650466ac7ad3f44e0822a44c3bb9cada04755af61", "original_grade_sha256": "7709c1c8026f667ec482081f5402a9acedf684986092df97aa5d6718ab8d4092", "input_original_sha256": "3990af9aa9461771d209462dd85aadcafd76f5ff1bd64e86169ad1cf9f66f2b1", "input_wav_sha256": "3c604f2a55c6b400bf95c9b2bd3301a4e14bdad9af05744c1ed1071c84802c76", "input_pcm_sha256": "811881138f3cbb6ada976895923df2462369b5125412abe2913ca077be3a4c08", "output_wav_sha256": "346b3173a65ce2a173f8240735f89f6b3ced3ef2c4162c217c7eef1c213e78d6", "output_pcm_sha256": "1b963bae3e55dfda70d4d1e94a83f32272e2d9649ab7167ee8c05ca20d796fc8", "result_sha256": "dfaee4a84a447fcdea6717f38606fd76800563efe99db13ff91ab95a16b3fe21" }
#279navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "9a54d6c259ee7cf41d7202c6d5a72695eb761ef62be146d26f6c15ab04b5445e", "started_sha256": "f2986e2b0cc0a90dc25282e71e7136b7ee2f913206df2171fa9f4bc48537490a", "original_grade_sha256": "c783529fbba0416b504470c8a2a21dade06ec3b64f4aeb77eacab5867d5b07fd", "input_original_sha256": "9a54d6c259ee7cf41d7202c6d5a72695eb761ef62be146d26f6c15ab04b5445e", "input_wav_sha256": "ad6a5325d633fcb2a5425e9cba63e5585c93c572e91ccf8fa5313c9101081615", "input_pcm_sha256": "37dead1ac8eea599d41f156fea3c30b9fc2e2ce3a402fbc7c6937e6dba0ce06e", "output_wav_sha256": "5dadb58b375cfbe19b4298baa075729e0e8f0c93237962f36f36a6505de7d786", "output_pcm_sha256": "4dbb8e2803188c1b6c5898ebc165d7c3bc332a76ba91946c920854c72192bb7e", "result_sha256": "a6eb4890cbf9ae420bbee3332960fb6241f2e6549a2466745167bf26b4caeac5" }
#638object_counting heldout	5	The answer is 5.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "d7b2e10d0d70a2a46ec5193993307e5d4d0ac766f4e601b9735af13715bb6f5d", "started_sha256": "926ff0bb2b90ce6dfcb42f2cd23e7f1afca39f1072901f9769e32918ce781cd1", "original_grade_sha256": "6e8dd16f372f64e0f4f4b619ce8df6aea9e7cb12cb87b494917fc7b2b30aa949", "input_original_sha256": "d7b2e10d0d70a2a46ec5193993307e5d4d0ac766f4e601b9735af13715bb6f5d", "input_wav_sha256": "72dbd2c0f367005a7b4e8f03ec3beb3931eca730cd9112be5201a8a8cfe2015d", "input_pcm_sha256": "6d2a7bf0cc601ec11de48eaf3965f092e8b7b93373381030b366929ff85c9a93", "output_wav_sha256": "0714000daaa963c80fb3ca265dd59508cf9c4b1913744bdc260276b5c017e052", "output_pcm_sha256": "668fd9c0be28cd541dae532e293602d88325069d5b85fdb396037d8322ce6660", "result_sha256": "faaed1f47295ea679dbd2b77e3ee2d0f649cbc78825106ee0eca34ef71a61734" }
#804web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "694179fcc18c9e125992e8a887612bde4a145849e26eab3111ad9dc7d33d4162", "started_sha256": "07399300c9377997b851e608107b5c8b0c1153008b6bac82d0b93d4a19f84aad", "original_grade_sha256": "ed0759f71eab4ac57eebd0abf270fc81faa928e56e19df4141adce2d039d686b", "input_original_sha256": "694179fcc18c9e125992e8a887612bde4a145849e26eab3111ad9dc7d33d4162", "input_wav_sha256": "463ee6da9ecca66fd379c41329b6631c5dcf04ca317536521ad4d93d78300a30", "input_pcm_sha256": "c2f52eff862cd19606d8173e33b2ede9414a2022e1d914139090cad70cdf9792", "output_wav_sha256": "144e159298b8b2ff91322fd74720c03d063fb00b0d2a44e080acd67754cf2afa", "output_pcm_sha256": "5d6b85f8308c776417d925465365e0053ef573faf61623f76bbaf2f130474717", "result_sha256": "f6cfd3452ab70c3ad931ef8e853061e6ec29a13fd69e1ae297e3b8f1ecd64e8e" }
#189formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "236b1a23b9f02e79beec6d0a9a1e547192d800b723779e5f8827108745cb73f9", "started_sha256": "ad76b56e133f329e282ca4715605c5a95df42dd68845283f4230be68b4b1c597", "original_grade_sha256": "a62269c4089affe90351b4223df187991f9716889ea77be9741bfa80af30272d", "input_original_sha256": "236b1a23b9f02e79beec6d0a9a1e547192d800b723779e5f8827108745cb73f9", "input_wav_sha256": "dabfdcb84d35a75225f2d939d26112a414b3d2e30ed61972ec6306e69da66a06", "input_pcm_sha256": "0e31bbab8be0a1fff6b1eaf501d0a48fe7a78b734f841200a22db394cb2701ca", "output_wav_sha256": "5b2fdaa3ae52083aa035f25e8b75ece41f42e55a61b7cb64b732defd5c12cd98", "output_pcm_sha256": "52ff5f024a5609ebb6109edc0c13522602ba505d7799790aaa4b748a92248eb1", "result_sha256": "b7e10c516a871f86d6d9b642fcedf72d76f849734cab91e2da244180d9cac141" }
#326navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b5f41d10f10075d6a02278ad79a3807483ca14ffa5080102001e75ef7fec81a2", "started_sha256": "9878c9b5361cfcf1caf505755b57bed4e8bfa16e3adde998129f1f3aeb48447f", "original_grade_sha256": "8045bb0064a443e92fdab4ffb09f4db9cc6af6548f3f28dd55518ac3a5bea36a", "input_original_sha256": "b5f41d10f10075d6a02278ad79a3807483ca14ffa5080102001e75ef7fec81a2", "input_wav_sha256": "d1c1ca3549c440c1b586de32d0c25d55ddac3e798aa56d39b41755216fbc3f57", "input_pcm_sha256": "0d7719ac2310681340265abfccb8a4b3aa42931ece1d9605f53fac98fe1f0e0f", "output_wav_sha256": "0ebdb542940033033f206bb1abec4930c8a48ec27f31d454f644bd301d0d49af", "output_pcm_sha256": "c798ca948cfee8b0ce833086e0c78fd1f56528711a9fbd20c764f684b3f9cdb4", "result_sha256": "09f58e2e469a730acf897cae595b46e51982cf4d5b162c78b9503337c6b1d01e" }
#747object_counting heldout	4	The answer is 4.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "81b98a1058437a120be58eb61916a94f93461d26d943028ae3db6bca3631058c", "started_sha256": "6801fad396c36ed2276d96a7d9f95e1d1817a5fde555251f511f1ba2dd494f95", "original_grade_sha256": "eed2c7869b3f4bf70635aa446121ad2b65073273917aa39cec058ebfec4f7462", "input_original_sha256": "81b98a1058437a120be58eb61916a94f93461d26d943028ae3db6bca3631058c", "input_wav_sha256": "0b62d27f03d0a1f7fa48f1375ed642deffd04c3a4e0cd0f47ecac88fc30d17cc", "input_pcm_sha256": "185d2d546164860e6d69575ceef2277b97ed9024b63115f7bfba8a75d6b7a58d", "output_wav_sha256": "7b213ab14d2066b078887e5ef9250d2fedbe969532759ccab08e910383c5c9dd", "output_pcm_sha256": "2300dd018da3f4abfa8fa8cf34e249a1c9d6c70b268a47ce55f78342d6f1ef6b", "result_sha256": "dac0d8e73f82835dd3107019b723946692d70d8c3011f37908643f4520b491d8" }
#982web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "ed3703e59f55bb34470e39d2397293561443b1ef27584fae28c8589115a8ce49", "started_sha256": "39712dc2c7da4983873d468cfb720053688d80a0ef30f6d6bb60f00bd1385bd9", "original_grade_sha256": "784a4f38f85f84c92c5409972853ab9925d6d8d12da90b8c9c332d2214f30bcb", "input_original_sha256": "ed3703e59f55bb34470e39d2397293561443b1ef27584fae28c8589115a8ce49", "input_wav_sha256": "a9851681e12b8d6cb82faba9b1999cde6d7a24e5ef3bde2691caed0882cbd0e3", "input_pcm_sha256": "803854c87bf59be699c5253484c71d303708b34bfde446361fd1c49f7301f021", "output_wav_sha256": "55a9da02167aa3876b2d61574d70c7f6d7f57b784751299f6611d994a4ebae86", "output_pcm_sha256": "3621432ce5bcf311e76d7b669d9d7eefa8c57da754c6b38674fa5b431b7cad50", "result_sha256": "d087642a6d94da1c365a3a220aa9099269584ca6fc809b91533fc2ef46fb7907" }
#151formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "4e7f3f5fef9fa134c6ab19f83f7b588d9f76a0595b12bdd4ebe100ad801b20da", "started_sha256": "eb5c7afd290ba5dc5d26a2c2444fc7622430fb8213863a1cefdb79925dee863e", "original_grade_sha256": "84f493137ffce6717244dfa6c77b948486487ffc4531118dc190fada36284d39", "input_original_sha256": "4e7f3f5fef9fa134c6ab19f83f7b588d9f76a0595b12bdd4ebe100ad801b20da", "input_wav_sha256": "c32cc44bfebf0480b69fb776108cb6d3e20835e6232f7b9c5ac6ca8717ce8b8d", "input_pcm_sha256": "d309f25c745cc73943fb5482f73bfe756829eed0b2461a152441656f272e10f2", "output_wav_sha256": "62f9cf7117f4f1ad4512d63fd361ad4fdd893612f430796d84f40b4a37a46331", "output_pcm_sha256": "4f216737bc7c58f4f5d26cc17ebda9aecacb087693be3d9e6a5ad569f9a76fb6", "result_sha256": "347207c8e8ddec840b1d7ec71daeace72b3121823953443a729afbe1e45d79f0" }
#490navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "791d661ed80919f21d45b87ec26780c6afd0dbc417947e634611615b821c2a86", "started_sha256": "f0bb5b30651e7ac3b7c3c85ca8b30fd2cf4bf7911758dbe8ac20276fd331f7b9", "original_grade_sha256": "4765197cd710d3d61db7332f6043bf36c7f0eb8b6b9e696f2055bb67d97dab33", "input_original_sha256": "791d661ed80919f21d45b87ec26780c6afd0dbc417947e634611615b821c2a86", "input_wav_sha256": "f2297f04bab0aff497d1f701ad00c9dd852343b800a38f05a5eb859e0020b3d9", "input_pcm_sha256": "79bebc415b8bad54b4f69ea425d91e4ae844dd58d0d2bcb74294d360738c2e52", "output_wav_sha256": "c76c255ea2814dea493744e6e9c1ede9d06f6f2bd144bdc0318902983e699837", "output_pcm_sha256": "d2f1f7261dae77cfe0e165630db140c6aa2784e5cc32d29c3eeb0eeebf01fc48", "result_sha256": "66bdf950d9d28d379b8986f438215df0d15b87dc3e05fd09564da9aace14c83c" }
#595object_counting heldout	8	The answer is 8.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "697c308597b682a1851cf499088a3405b019bd01cda8285e2f1b82f0c9d0d623", "started_sha256": "fdd0aa157fd1b2bc48a930a174140f801a4b1ffe798c6f8e871c3a85ef336f3b", "original_grade_sha256": "49ddd4e4fcee79392849a99d80bda4898d6ccab6eae868fcdbe3da4e26b832db", "input_original_sha256": "697c308597b682a1851cf499088a3405b019bd01cda8285e2f1b82f0c9d0d623", "input_wav_sha256": "0bd1c7786ac9f9fff8ce6422fc65b1138ddc7aeb98cbb68c27069034a53c1641", "input_pcm_sha256": "6cdf4a9f681440c08a6dec5036014ad017d8bf73c267b888f8b84eafe23fd322", "output_wav_sha256": "1465e9e4560b01846b65e8505e63445195a40b7b319a25c8beb9fbcddc6982e8", "output_pcm_sha256": "5535e945b3024be5a1224e71a0960fbd0796f7301ec38a039cc421ff4527ca9a", "result_sha256": "b38ac6deb27165c3c2cd80251b5eada94a6c380f632e7821b44edc374e56066b" }
#895web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f60b7d98a2e8b936e3d795a19ef123d10f9135c117e0e23e5ac18788b6b6388c", "started_sha256": "95bc80b9b90753b49a18b7fe9499a0f9c1ad943cf5a432e22c1e402190956587", "original_grade_sha256": "2410c18d4b49227e73178c512c78b2dfba3f67a0ae0f4a48f4dffcd7a58f8a66", "input_original_sha256": "f60b7d98a2e8b936e3d795a19ef123d10f9135c117e0e23e5ac18788b6b6388c", "input_wav_sha256": "b77608f09ee4787c507d469afb96615ed9d32bd0561e4077cad203e862202b0b", "input_pcm_sha256": "028f87ae003903994b35b2e17de4fed73f96a9aff49050b65e059c91586294ea", "output_wav_sha256": "be61c31b14a2dc06ef135e9004c004a06c241d93d49adb2466d984a372fe965b", "output_pcm_sha256": "2dc86c60224fbd836e2ba6326c3ddeacfd19454f48ca83165f6d08daf91f0a10", "result_sha256": "510ea0a295db9136985e08f6d3d6e5ce92e787d8a4ae49ecfbbf86d20de05dd1" }
#69formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f745aa239b885417d4571422fb8a8360855a10078857d086cef3e0947b5d51fd", "started_sha256": "9572bba7acf44abbb7fb0e3f158ec193b2324fd85d37ce4b4e49f8dc8840bbf4", "original_grade_sha256": "2447b492fa905b66e48c3ef42f2a2d8c6bac8a308bf6baf2e76fc45d4f600707", "input_original_sha256": "f745aa239b885417d4571422fb8a8360855a10078857d086cef3e0947b5d51fd", "input_wav_sha256": "b1611c0b8470d881687a4314d6a12cee7b748de252598cce840ea07aeaf8424b", "input_pcm_sha256": "b1a5b6a036d24d8f0f00fa96de46c9c175cd8428f5082f727a09f9b3fe93852c", "output_wav_sha256": "d43be27c759bc4b9e744b2ccc0d15bff06eef5df979c1965e5fb69491e329e56", "output_pcm_sha256": "deac8fff3ec902a018cf244df00061df2a6afd33a5ec8f987ccb3061c7fdc4fc", "result_sha256": "8d6617668f66b30b4abb884b2d094e9784eefefd44fbf8891455b3128804eb51" }
#470navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "13fb4afb4ffebbb8415869bc852582b9620301f9dd62a2f579d853a6d1e30c73", "started_sha256": "e4080f4cd14393a3d05bba0b7c4319b2e9c784056291ebf7ec1643b60616c3e9", "original_grade_sha256": "1772601d06269cdce4b4aea9884e14aaa60302e562e27f4e503ad49d8a4699c1", "input_original_sha256": "13fb4afb4ffebbb8415869bc852582b9620301f9dd62a2f579d853a6d1e30c73", "input_wav_sha256": "a25c23507833be84ba782a1868c2ae88ec6984e26f6fd2ee071420b5faa24785", "input_pcm_sha256": "fc4c645f1cc2df885f2f7fdda962fb0a880c0b3032fbe117633baadb21b1bad1", "output_wav_sha256": "e161939661b4a8b3217a273d02511ac1eac2fd52858cec807744525d33e14a2a", "output_pcm_sha256": "8c6d10cea91eac317a18065ae0e674e2307a0c45ec0ad3cf5b868f741b5c90b7", "result_sha256": "f3ecd7312f875f30fa029dce013ef96cda953eebd4d83b1f0a9718d7f33cb959" }
#697object_counting heldout	15	The answer is 15.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "00c3ecd10606cef4fdc0a960efceb4e814430085a86e4919edc8296ad1ea7d92", "started_sha256": "d4d56d9ce0b8b3f11778313139aa7b61ec944a1589b6a7d748fdda00dcd218e4", "original_grade_sha256": "30ee4a51e38a2112e222111f28f8cdb502aff817982a513d428d3ef6fea49290", "input_original_sha256": "00c3ecd10606cef4fdc0a960efceb4e814430085a86e4919edc8296ad1ea7d92", "input_wav_sha256": "4aef476f627aa911c5c3a916a9c5bfcbdb24fbb88bc81ad4a8e0e696228acafc", "input_pcm_sha256": "6801c3aa0b7443ae73d5139c2f88e5d2f367c51d3b5de75b2a79716edf454ef9", "output_wav_sha256": "1f1e497883760fb683bfcd88f54c2d24f424bd446d52d2bbcdb55c0ab75e54bc", "output_pcm_sha256": "b19242195c441402d3ca1be04798b9b1eef755caf2573135e311beec30618626", "result_sha256": "495d8332f1d98d757de86a07392846454ddd6ce413fc8595dbf47a0ea5574875" }
#776web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "bbd1572d9822dcad0225d2e1baf13e679f6bc8b394058a216e32f3d94c85c3e5", "started_sha256": "138d45abc21845b9a70c2c40b2c763e386db99e23045b5dd2361d1c758f84e4e", "original_grade_sha256": "d5179c6f96d9aea20ed3be1baa187a5282c84efc948e82537df73ec263dd9bd0", "input_original_sha256": "bbd1572d9822dcad0225d2e1baf13e679f6bc8b394058a216e32f3d94c85c3e5", "input_wav_sha256": "24caf5078aa089663bfb83ae09ec3c82c9b747f081fddaad5bd098aa2331bcc7", "input_pcm_sha256": "81454ce3d3c8c513d4bec3b6081eee78830030648834e2bc450b555b40b1e101", "output_wav_sha256": "b3417b089786e179eb5c06216e050a0509e51003446f1fe8648cfc2357f6e768", "output_pcm_sha256": "45aaa730161371769a287c95fe34358b753fdd94ab3e695d38a89c2197528b18", "result_sha256": "3a1e510444f4ac1c369f7832cfbb76d8eaf64b2676af67b47dd16379d0588632" }
#45formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "7231bfcebfe33a7401450c1d8b9af00b25dfb2a146ab7d847a857433b1a82ee8", "started_sha256": "c81f103edaa5e372e0e75ef71ca68ffaae029117aca21b07ad97c747e5981d58", "original_grade_sha256": "23c3252a8bd2db3244fa6153f13e7876b342afdfada61ac292cadecb1557acc1", "input_original_sha256": "7231bfcebfe33a7401450c1d8b9af00b25dfb2a146ab7d847a857433b1a82ee8", "input_wav_sha256": "9a4fbebb59d0c980e59218e42114cff8780b2b1f899fd3a744401564bb70abb0", "input_pcm_sha256": "df709b7dbe7125ccf0e7c2d7dcdf35f20c66fa3b63d295be34c3ce3978cc0c0d", "output_wav_sha256": "2c80d246a7d6c058d912b87aba24def917545179d12cceb613d0c0106c144f2a", "output_pcm_sha256": "7b3c8ccad16c1052ad888853e63c357e081980b9faef42c195207fa787e1c37a", "result_sha256": "19a8fce485f2aa288164c04d564a9ed239fe87268e36c23f9e285673431ed472" }
#394navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "df3caad89a27924f61f19c349a80042de7d0288699c40c0cdc37ce13bf9192c6", "started_sha256": "e07c7ddb8fc0e2f002bdddce38338325a503f9009ca089893c6654aad1b170ce", "original_grade_sha256": "ab58cc1b5530cf774ce19643fccee7778f64843caccb1ec79c27d2275eada090", "input_original_sha256": "df3caad89a27924f61f19c349a80042de7d0288699c40c0cdc37ce13bf9192c6", "input_wav_sha256": "e2d5a359585b2111880a72ddc94e1601cab72f988bbcc04550f0d857c5813782", "input_pcm_sha256": "34c6bfc68bacbe9a17b2d9e76a9687bf46a89d1561a21fb3f03e31f9d85a1759", "output_wav_sha256": "81ebfcc52ac8ac25f4ed90d7be79c967cf1002a1b83d14be666914a40f34367f", "output_pcm_sha256": "db5f9ded7ccacfcae2bddca92d12f5930af1d96e669c616df232a8a8f44890fc", "result_sha256": "e2ca87aaa0aa995d150b46d4f48105e80b74848a9fa9cabe71168a5ff18b16dd" }
#517object_counting heldout	9	The answer is 9.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "bf7ba1a6d6db579ea8cf52cbec8dc47520680a0e3f934e8202ddf65abee88e62", "started_sha256": "78b49750b61910d0ae51e147ca8a6b6823c79771916f9957be1b190313e813c5", "original_grade_sha256": "bf474ba916d55384a8a84c93f70ddf5f0bfeba49512c9729a28732e5a2dec98c", "input_original_sha256": "bf7ba1a6d6db579ea8cf52cbec8dc47520680a0e3f934e8202ddf65abee88e62", "input_wav_sha256": "d328765517e0bbdb8bcdb978acfb8154eca84824e867a3a7716afed0db179a01", "input_pcm_sha256": "bdc69feeb8a19a40926fc42cb690eaa8093bc9ae5aee2832f16e1bf8227b5ed6", "output_wav_sha256": "1c13bf755769b0dde1ffcb028b362dc5496764e1c09a269dd2f73419ca451b3b", "output_pcm_sha256": "4afdaad86adfb856b8b647e67f16f3175e3ee5d81485b2fdc687504113ebfe37", "result_sha256": "0d0df7dbc6f15fe3bdaf06a5ac82aab984599720e95484fca8f3a2a84b517f9c" }
#932web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "c1e9b4c252bc99ded79d544cfe9c582c049aa9afc37bb4591bcada4d765dd533", "started_sha256": "f44aeaf078ce5c6a0e2bcbac0bae94f4353d857752ca6a6f2944a3bf606e3fbb", "original_grade_sha256": "9153ef80c6f5dcef443c3401ad20d7d9728deaa2a5d926b00ee2fa1d4d9f1274", "input_original_sha256": "c1e9b4c252bc99ded79d544cfe9c582c049aa9afc37bb4591bcada4d765dd533", "input_wav_sha256": "6e9f54feb02d5ffbc007772dca78b45dc973499eec60759d8240eed80d4b4519", "input_pcm_sha256": "daad41bf2f968b42e9e76613eab0e81864756e8370ee50dd204764bb3d5fcd37", "output_wav_sha256": "2282ccc1bb593c2c79d50ebdc042eccc7193fdbb0e85fb876216358b8dc58005", "output_pcm_sha256": "91b37f06f3735937854ba2bd057660a734c2e6fc076210835f9d22180031faa0", "result_sha256": "e84563615e2668b82380a18ed03b585c1a235cc348cdbdd676d5eb314a3ab4f3" }
#199formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "761e76f80e0c4e36814370f40a87533ace747f582fe93ed7c8d34ef3b7b88908", "started_sha256": "8328c932a3273e3e51d7d061570e0567f35e6bf96c76f3a14b486b824059f33f", "original_grade_sha256": "363289ebbe47cbfb89fcf1ab11585049092935f344b2cca048b984c2569d7148", "input_original_sha256": "761e76f80e0c4e36814370f40a87533ace747f582fe93ed7c8d34ef3b7b88908", "input_wav_sha256": "2217b4b86e7f8bb30d56d8002d0c0e151328e5d746383c7b95fb4f9df97b55aa", "input_pcm_sha256": "07d51b6a73b546bd981d9d81c35b99f08c9d5df65cd386c1acd9706293d78f0f", "output_wav_sha256": "6340108efd1a03fdce93ba88607e29f8ed50cd16ff49dfbaa4d5df27830ca31b", "output_pcm_sha256": "a6a002efafc431925d3b973246e48f86848895ada2813abfecbb102231c62d1d", "result_sha256": "c437f0d4325d58ef6e795093422912e137965b37bb0054fcfc59f14d93b0cba6" }
#287navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "c3e1dcffea4b8657d6b818fa0c9ee15a48a9ee7e76462bf32ec0fd1dd7f44e90", "started_sha256": "3121208b55e12c31dde4a620464b1889bf39749c9cf71711cdb0f02ff0c254a5", "original_grade_sha256": "2963e2b219db85e38af1be479fe9da0376c9d671d73abcb34a8640e895a2617e", "input_original_sha256": "c3e1dcffea4b8657d6b818fa0c9ee15a48a9ee7e76462bf32ec0fd1dd7f44e90", "input_wav_sha256": "a31db46fbfc9cb7169bb8359d847350a229bd0aff5158e1acee037fa87e3cb0a", "input_pcm_sha256": "25acf4a5c14709ee92abce9e57d499dbb5ffccb90de392f84c85a6102f32ddd4", "output_wav_sha256": "401e9739c40672a59b51cb3f10f76623b2bdbad5cfe617a6288a38c505e03d56", "output_pcm_sha256": "7e1663cdaaf9ba445413e7a1ba75078e0790bdca47df8da4b7c9949482481f50", "result_sha256": "2c1ba5f63c1921ddd37c32cc7fce4557e2daf5e81d9f0e88c77e1d8fd9a2bd5d" }
#520object_counting heldout	6	The answer is 6.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b799773e2b59f797377386bc5c1678e54e70e5e495e88885a68791170c4a938e", "started_sha256": "9be3e48b40b41ec16defcd479b70c35a223b5f9112502a6d7d46b51cf6272dcc", "original_grade_sha256": "68ca384bf1f8c8bd9dfe57604550f8735835449f9c8d10a057d910d6a1e831c1", "input_original_sha256": "b799773e2b59f797377386bc5c1678e54e70e5e495e88885a68791170c4a938e", "input_wav_sha256": "3d179d7f2ba71a1ab83caaaa533e36b8f73a2dbd462b3a4ee5fcf90e954dc5e8", "input_pcm_sha256": "5b687967746aa2c6f209c499316c99f3803d345a3d82d5a649a8a62a0659c58e", "output_wav_sha256": "81066e4e66548847ce38df42b8615430453787216f374373c3e6a47130a896fb", "output_pcm_sha256": "aa18ec2b860a58d4e8154bee2e7340f19db92b63650e6e4e36cbc3e65b463d17", "result_sha256": "31b9d809ddae49d1b87e2f873dad566da4f7ba7bb976df02ac7e718852485e0c" }
#945web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b44c4c62fb9f6333f76ce3759478f3d848d3a26f82a960fbda97554fe3e6d79c", "started_sha256": "814ec8ef8cc39000211d8ed8b62616256714235b81d26a5871aa0c0ec8b68a20", "original_grade_sha256": "70703776ba6e1aa55028c234bf63b7e65c90e47643ab81eab79a399b37df33b1", "input_original_sha256": "b44c4c62fb9f6333f76ce3759478f3d848d3a26f82a960fbda97554fe3e6d79c", "input_wav_sha256": "de14f73a507217577b38844ecfdabafc9e4a75897f597228443c778f1bdf9732", "input_pcm_sha256": "aaf66be6419de3ec3fc88f1e37db268c3f256f5f07e7790874815ce9c9b0570b", "output_wav_sha256": "904d0b28f1ce73a0931fffaf21282dcfa73de630726a1c499ddf45b38d1f7ef5", "output_pcm_sha256": "6d6db0d6a342806bc6adee642f7c887ff8e82787ca4df7a2f1b2d4582a51d4c7", "result_sha256": "a5db4a5260e9a8c1735f2bfc8f489e99b3bdf7d5ffed63702a078467c75c214f" }
#127formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "4e6ade1a67d5015140bef56944677a4ee31ae78a0a44cb63eee7ce88d3f47138", "started_sha256": "918e2e6998caf9c28cddb7f2126775386fe1193e51d2e90ab2845078cb913259", "original_grade_sha256": "b0cfda63ef9a28d3652dd02fbd0485fd48564d8840340fde3fa3efea73bce177", "input_original_sha256": "4e6ade1a67d5015140bef56944677a4ee31ae78a0a44cb63eee7ce88d3f47138", "input_wav_sha256": "3fefa76aaada371bdda9a14b8f63f884ff9d62b567a91dff6c5fc31b6b2cfe22", "input_pcm_sha256": "32d37dc602fd9b657275140d691cc5d4ab7377285c53d5d414fe772cecdd2750", "output_wav_sha256": "60a5020d0ca20c6a9b32f7a0502fcd28d7988b739e28e501ed2b5a49474f557f", "output_pcm_sha256": "1d01f564cc1935c929a1697576c393b49d6670267ed27bf3bc2588d3c068aee9", "result_sha256": "8217d389de6423ef483894840b20852fdfe9f988811c644696115c0f4c9c94dd" }
#343navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "46bbe3e29de25bb58960d04989c467cf5b83f6d1222445d1ac5e97f02ca497cb", "started_sha256": "bfd241a835e49c6e6f7faed26624c4fa06825711e1280a127f7909bda3b5587a", "original_grade_sha256": "a5efadbb8c7501a1c2157480715186117f4862492bf4c50efcb332a8221b3a89", "input_original_sha256": "46bbe3e29de25bb58960d04989c467cf5b83f6d1222445d1ac5e97f02ca497cb", "input_wav_sha256": "c9a5f46d082cc8d50e28c154fa926287431508a706ca5af6222217bfff30900a", "input_pcm_sha256": "f28f86baf7dce68034bb6ca8f81d8173a37866b949811a5fd969fe08e562ba71", "output_wav_sha256": "8e34cf7abc805864455802232bcfccb1dd756fa4da3a060dee70a36f919e038f", "output_pcm_sha256": "3dfce5910c2b2f603a6564ccd0e6e0bd949feb3efb5d873971d8c2ccd985cc29", "result_sha256": "020d91ad658989bce307f80cafef21ae23bfa7bddc8c997f6b35466c6c7f6cad" }
#557object_counting heldout	13	The answer is 13.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "c3c495b322ca4fc84dabe84d5c817bc6911f14fcef1180e33eefb5fd22d09240", "started_sha256": "b5c47ed774dd30a75a20cfd5e7751e6b28e87510795b3fbbdbaf6f5304bdf89f", "original_grade_sha256": "a3edd1335f2ad3c882c0f4914afbb7bb6197a3a5893ff2afde075c935254a82f", "input_original_sha256": "c3c495b322ca4fc84dabe84d5c817bc6911f14fcef1180e33eefb5fd22d09240", "input_wav_sha256": "c908070baab8207bb2962612b425e15f1bf340c693a8c1e90b5e73bb07e9f628", "input_pcm_sha256": "8f2dde1d053669aafe597229326ae13c77209f4a6d567ee6929530ab5d37fb2f", "output_wav_sha256": "5d7b30011e02fa93d308978ea5a679d8fb4e643597cc17f686ca684b14091d94", "output_pcm_sha256": "ef247f69e5baed93c643c0098632ef6de6c9e42ca7281322f3ae1fd9971df93c", "result_sha256": "87d891c01c7e5f304571763520f23b4d9ccdee091e830290f3e6f5377ef0cfdf" }
#830web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "8246d04f32834cae7ba8a878a70ea8d6186be0bbb608d31012dccfe2da32ae51", "started_sha256": "c04a1e8516d4bec8e0d94a7c93f00872271396ffadd8b8c2c9806a66f15e64d0", "original_grade_sha256": "8e453e56acdf5c0f6a77703441d6c93be0e255e8d4fbc9d25f3022a914d3414b", "input_original_sha256": "8246d04f32834cae7ba8a878a70ea8d6186be0bbb608d31012dccfe2da32ae51", "input_wav_sha256": "d3a6fc2380d646f0753848b4ef1162522036e11a84e3d2b9617a9426131a6175", "input_pcm_sha256": "e3c6e9ba611b9e7998404052d1d127431b0b73fdadef0317c86cf712376ebd9e", "output_wav_sha256": "78e9c532e26f7c31bb9d41c7d748176ca6a0ac52448d9dc9ee7e2034b2b270a9", "output_pcm_sha256": "f059a0abc871a08ac0d7ec0d9d604f1a9c0a4407043b667763f8e7b8e31b9e8f", "result_sha256": "25578eb4948b160f22fae13bc32da0588d9308d9eb76f9fc231118c6282feb8a" }
#15formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "2fb44fda851e01e07556bfe85224ed245205ddc53a62410897cf68d7fc6c2043", "started_sha256": "6acf825383e746b2dd279a16e4d9414fdbf45fa70b63c816478c77622dad2623", "original_grade_sha256": "3784d5ec528c8e9aeb1f066f8e6a046828e3904d6afabe8df3de4ddd56258060", "input_original_sha256": "2fb44fda851e01e07556bfe85224ed245205ddc53a62410897cf68d7fc6c2043", "input_wav_sha256": "903a191db38c5eeef099f2f4433415729bb426bca9bb96b1bc8453d8a260546c", "input_pcm_sha256": "1c41a309fe1e2d3b23ef87fb10d30eb173f251a2424ce77139dc64edbcf3994a", "output_wav_sha256": "d617335a54acdee208a09d210f0fb823bec4094d528428dbbe1aca9b7e6bd6bd", "output_pcm_sha256": "2ac2a11a41e0fca651f6e985a0a7a9e332443eb3767d35d4a51b2bc37d110225", "result_sha256": "0bf1e423851411cb7c5aa4a83a4a88df4e112ec776998904332dcc1c21030635" }
#439navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f921f0457247ea3325d692298776dcc53f3ef9051d9922a5ce01665554257010", "started_sha256": "ed22959f7916bab15a616924e6d64c34bf95db98e96bcaf508fc50647ad8bacb", "original_grade_sha256": "78703a91c551af7269acd353253d10d41596b7ae9dd2e99f5375927aa6766ce3", "input_original_sha256": "f921f0457247ea3325d692298776dcc53f3ef9051d9922a5ce01665554257010", "input_wav_sha256": "b20f48a666c5dfffbd56afb91c3113d695f22eb6c11e7de377a5e92e4361312e", "input_pcm_sha256": "40b97bc308d3c5223070d715e92925321795228205caf36a0f24d04ab00336b8", "output_wav_sha256": "c0c9964f684b8e4202e4f177f1e92ab89f985a738d795322bb2b84eec8898a41", "output_pcm_sha256": "a400bbf1f91484a7b87f1748857611046aa4398c30ceef7af0ffdef523c72188", "result_sha256": "0ce78db2b83c81255737aa55bdc7db1200ebbe1ed45d4790121e0b55fe69ff30" }
#601object_counting heldout	11	The answer is 11.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "e50b7cf7a7ace6e9f3e9f370fbbdb3698fbab3f5db4d36628ae9d5bcd7f7b60f", "started_sha256": "10f558b59c97aad4ef4928739fbdc9a5a176916919d37ce9002a615606a297b7", "original_grade_sha256": "50da9300d4c0b738eb23c0bb237b99e9b058a484f5dd5b0a129410a8379db179", "input_original_sha256": "e50b7cf7a7ace6e9f3e9f370fbbdb3698fbab3f5db4d36628ae9d5bcd7f7b60f", "input_wav_sha256": "9c02016440603a553bc4346ab59c2da4251d11f50d7601daa90fec493e93c45e", "input_pcm_sha256": "7f3ec60946259548737bb783e2cdf5762a9d39a654869ffd568162587f566f48", "output_wav_sha256": "937a91c140148bc55ac150f8fd70ea3e6b5132470464b6b7c111a00bdf57bab4", "output_pcm_sha256": "0930e6a55fa4259d76e0433cc40fea5b763a79cb89bd9c7b4d55146f6946afc8", "result_sha256": "a2dfbdce9f89bdfefc2c0fb245eac272b962e5424cdacc5478a165c60dfb2711" }
#769web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "4e5bb976d4100406d308d4f749061acec61eb9e09e61ed8b57556a71e736d666", "started_sha256": "514322d9fa0275c5406ef8c3a0fdba7560fbea7f00b486f13cfc23bf10bd0103", "original_grade_sha256": "f190e0c83eae5f6f16f95cef4e68d768c15b5837e749aadd36ef48626c2d140c", "input_original_sha256": "4e5bb976d4100406d308d4f749061acec61eb9e09e61ed8b57556a71e736d666", "input_wav_sha256": "3ecf06cf63689f9f4711b1beefd8c8b9202687b06d342b21ebd76d43de1acba4", "input_pcm_sha256": "f985f10fcb727b53d15ed7537fffc73fc60e6ed149bad57317df787a27eb3b52", "output_wav_sha256": "71f985801fc7e512a389950ee00dfb65ae45789daaac59444cda36032c5c4879", "output_pcm_sha256": "dc275a4bb75873ba1d43ee53f69143d472fa8b17b6a36afbfe112ee0b5df654b", "result_sha256": "822aff930e692b5a03bf1b7f8402f710b9298027fb2f2676b3ee7f5b0e39b71d" }
#128formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "7f36f8e66bee1287784eb7bb901e94ef0ea0cf92df2b1586ca18b40e93016fc2", "started_sha256": "ff7da44080d442a6ecdc19b9191238b1f1681081a2ef3d7bfe1c454ba329653a", "original_grade_sha256": "00a0e9a0d0a14d552fe5720049da5a1b1da37e49f0ff081d1f53712466e491fd", "input_original_sha256": "7f36f8e66bee1287784eb7bb901e94ef0ea0cf92df2b1586ca18b40e93016fc2", "input_wav_sha256": "5de868a6a2b73eb2481905969cfe59d444b99b7ae13642a4544f03966b3de59c", "input_pcm_sha256": "68dc65cc2b28556060277c336289b27dddcd3ef7fe44596c0312b7d73c8f63ae", "output_wav_sha256": "a1eef86b86164522d37ff94f1ee7fe1553b4a95e07f24e4bd29ac495ecc2795a", "output_pcm_sha256": "06e00937ca54220b41e8f15a82132ec609597a1a905f8c7333eb9c3c3f4b82d5", "result_sha256": "fd26b71937a24717a745541e5413d18f19d9e26fdfe6df36090057461306596c" }
#364navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "619c6d557b46dee1d65693de4f2f434bd7e598cf8c844d4fc49ea28f2d98b9dd", "started_sha256": "8061c0f4afe8b3ab40e3ae6510915a943cb7bcb1cc7944b4cc359eda729c448c", "original_grade_sha256": "617c2629299367d7a8adf823748efdf819a081389644cb42249e401eb1c9ff59", "input_original_sha256": "619c6d557b46dee1d65693de4f2f434bd7e598cf8c844d4fc49ea28f2d98b9dd", "input_wav_sha256": "b96e279400c93f002aa8f712daf77ce70b8812b3bee5e8da57956de60978fd53", "input_pcm_sha256": "e91b637bc279970f3b2d511108cb42b677db28ac4eb8221011a8db57d40e5596", "output_wav_sha256": "0853499ee0d6fa97495ec448e0160ff1d3121cfaec4ffae935998501e3c7c91a", "output_pcm_sha256": "af6dc9624957d5ad9ffe5fd06e56a4869a505991970e6be9df42bd2764891d04", "result_sha256": "93d866d2e8951e3a536aa8e43ab22d5e2d06c09ac7883db3369a9c8e7a24cdf9" }
#614object_counting heldout	3	The answer is 3.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f1f30f2fd3dee29d83e4509c919c1c000eda889212d9e58ff1ecac2988ea5430", "started_sha256": "d7a433d8a45f3ea1d6fff5d7fda3aeac03985b319d9fb7821052d5310a267184", "original_grade_sha256": "22faa53f7a4e31db8c95bf633015da521b09cb136479d6007c5ba29e583af36b", "input_original_sha256": "f1f30f2fd3dee29d83e4509c919c1c000eda889212d9e58ff1ecac2988ea5430", "input_wav_sha256": "39ea4a3c9059799201abbbc133f5b7154850607727f33e1c335ddf6f023c3fcc", "input_pcm_sha256": "1a54a7f6ac36f01b6fa98f3a2b8291bae56781ccfb40a65de68454536f382bfd", "output_wav_sha256": "5657c1da9e815427c23b02dfa14cc2996af4460664ed1aec36ef782b0e59bcc2", "output_pcm_sha256": "d11fddec00faba5a3d68f38235a5670521fdac3cb3d1b185173034e1cb082e58", "result_sha256": "fe9743641276603f073a3c471f8dee0c5756ffce8043847fb83b533e1584f4f8" }
#903web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "e20968d7bf762dc73b38d94d2c93563c0a774fe809b0c9047c8bf630c274eff5", "started_sha256": "b017a2d691e68bcd85dbe052d476d2487ba2eaa60e2ead63f9b27f853fc4110d", "original_grade_sha256": "2df6e9e02c3cd1520c32b458c891003122852e2ac347a9f6c72d522f132aea2f", "input_original_sha256": "e20968d7bf762dc73b38d94d2c93563c0a774fe809b0c9047c8bf630c274eff5", "input_wav_sha256": "945535283005f864f971a4697a3957f9877ad74e6679c6bcfa75bbc18d5b95fb", "input_pcm_sha256": "882e6a964f1373dc0dcc0fd1599d9ccdbc630542661832a02512f40e9f7c8451", "output_wav_sha256": "b12eb801e0942e3225740a80f1f68f10eaf9bad45b49e674d68ed35a06016d5a", "output_pcm_sha256": "74230de24b27b291c627a620b37aa279f3344cd7baac950a5c48c99bc47a91cd", "result_sha256": "cc92b5dbc6ad38fd7775b76abb272b5d6225dccd2939718be782229e77319891" }
#81formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "90ae3757fa3d23fc01d769f588c3cded85bd93827d4e926659e885dd9687b23d", "started_sha256": "e190a0b838b20a29550f9ec6a4b9670378b22dd9214c7e53da606b7daa1f34a3", "original_grade_sha256": "7fee48e4826aeaca67b88e8cbd3c972e0c9b93a6aacf54f8162118efac3759ad", "input_original_sha256": "90ae3757fa3d23fc01d769f588c3cded85bd93827d4e926659e885dd9687b23d", "input_wav_sha256": "096a43721533f75e17bdddd05b7a928a26f91d751b70832451b9a3d7970836fe", "input_pcm_sha256": "e99116e8f800c80a7f713905dffbf28944d1d1799208fbf817c27c431b276137", "output_wav_sha256": "707ead69b5d9ea9ba2cc995d1a51dfd74e711ef09367ac3b3d351bee000dbad3", "output_pcm_sha256": "5d79e926d8ce668e6de46d6d57e1bd73bfbad57b8891d30d4caba302b8b84e08", "result_sha256": "bf0b1b70a81582c69d191f460db727579abe601db40b24143138ea8f5054e163" }
#285navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "c3ed492c7bb305189eaa6a2ae88c48fbb6cc91a8580584ec5e55b3dff5790cb2", "started_sha256": "cd0140df05820cfe2d38daeaacb351e2d6ae64a8db838d76105d5571057913d8", "original_grade_sha256": "2df5af3bd6f22cd3ac0f6f5999513abc96861a3e926ad7cb974f19f40fdccacb", "input_original_sha256": "c3ed492c7bb305189eaa6a2ae88c48fbb6cc91a8580584ec5e55b3dff5790cb2", "input_wav_sha256": "57593f2d34b4f34ee08d0feee3b8c6a4e2d8ed9089f70464eb8dcfbddc86f45c", "input_pcm_sha256": "2384127dae4601373bdf14e162a85ddc50442e1e3cb231f5d90420de2da9defb", "output_wav_sha256": "24af09067ad2817b222ed08d050b6670a9c69e754199fa2d564b4f5761a33066", "output_pcm_sha256": "03f04f71d285641194f49fa7f23cd0ea042eae84d24ab4d13951e51eacd326dc", "result_sha256": "3f4a537a3b7471c31b760efaa1c2706065decf350ca5b39c96ea19809eaaa38e" }
#743object_counting heldout	10	The answer is 10.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "707309b908f87ade5ffced269f7f8b59dc39e01b209cccfb593d68d7f4e2ebff", "started_sha256": "116b68cf9116349fc8b14221dfb49cc164d1f1baa8718cb4d940c3499ce69555", "original_grade_sha256": "45a4cd9c18f740b8ff319374265b73d8afa6035c3a202f1a2b71df7cc15bba02", "input_original_sha256": "707309b908f87ade5ffced269f7f8b59dc39e01b209cccfb593d68d7f4e2ebff", "input_wav_sha256": "db5d71fd05013ddbf502c896dc87c3dfcf09af308758d261a447f0131e2eae71", "input_pcm_sha256": "06a9fff27550d200207b0a8ab351a6eae8114c46d08b7aafbbfd1f01d85bb575", "output_wav_sha256": "e8d37a7e1d06f9796489fcb1eaf4addbab1ca542bba775a16467ce3efbc1ec64", "output_pcm_sha256": "723dd4c05e74f8d7fa8f7799fb994be1ee4e71e064c899c4b0b8397878ad59e7", "result_sha256": "ae647ce1abc7c5dde46142b8a4948073b5e267c01007e6533d35e52f7c7b59f9" }
#783web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "c55b25e78f024f9d7ab032033117b7348f95955730091c6f68a16291c5496eb1", "started_sha256": "33a333feb2485db5ca94ef38fb8104bb7298a99dd0215ebd696ac6f72fc4b285", "original_grade_sha256": "547c48dad8bb85e0a3acc8aa9b5a56d96e0543d4ab6e465b044a9aba9f60604d", "input_original_sha256": "c55b25e78f024f9d7ab032033117b7348f95955730091c6f68a16291c5496eb1", "input_wav_sha256": "74a182c5161d6202ac7d18c1a45277acdc921e9360b34c304354c2c2bffa016e", "input_pcm_sha256": "52e3e4984830405f69efd55633148b205cfe1a8b0124a0bc0f62f29015aa02d3", "output_wav_sha256": "b13f37f8ca6fe9d528d85630bb415c347efbbd1ccf4867656e540427494b64b5", "output_pcm_sha256": "37bb737301e1f329d90df9bf9c528f4ba216ddcb72c56c960b1ff6aefacd1592", "result_sha256": "462d288666a8ff147f4042c9fd7e5e57340d9963d86968b42c66174c7b7400b3" }
#37formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "fb6e3d3c25289e2dd4f68d1f10492bcd774ae563628acb384a1de61ef7cfee14", "started_sha256": "f60948c753396de84eec57bfebc55276ba49b860e2c42ee46ac54e39fa03789f", "original_grade_sha256": "519d63dbde50cace4644a0d2c1e03f8d2fe7e92370e77e0f94ed5006c5932e33", "input_original_sha256": "fb6e3d3c25289e2dd4f68d1f10492bcd774ae563628acb384a1de61ef7cfee14", "input_wav_sha256": "ce84898e29b1dd3e71479295bb0fa051b29be84a84ee261887ab1a2a31a48df9", "input_pcm_sha256": "44e4c1de2193bbe22057819519a6e2151c700e3677a75c88957ee95764c42998", "output_wav_sha256": "00735dac3889c882e6be92cc8c1d0672818fe216e80e3a7d3447a11000101384", "output_pcm_sha256": "56ce99295f1944104a27632722695572f4ea66903562ce3261e5c0aa08fa7727", "result_sha256": "935793e0ecfbe5b3e34e7cfc3fe701578731a9eb07603b75e9de0e6f0ab1e38a" }
#414navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f9d52ad05bebabd7ba19c016df1e70aa085b4cfcfb6ebd11045128d74d79a5fd", "started_sha256": "099d2c9e95df4bcca8ea1b36e3f7e5f6b6ef5e4a88584c92b5a19f73779f6e0f", "original_grade_sha256": "8262c56648a2740c781148de836ff879cdbd4471241b0407266e52dc6c8b87b4", "input_original_sha256": "f9d52ad05bebabd7ba19c016df1e70aa085b4cfcfb6ebd11045128d74d79a5fd", "input_wav_sha256": "2b2421147c295a0d0b7956ec8bab7f40095f631126dfc20a1510efede05b2581", "input_pcm_sha256": "47de086c0880ec3b40ac646da90c18abe4f77122f67121437d91b116ec7801c7", "output_wav_sha256": "0fad37d10472db09f083a3f5530348930774db71e24eff6495f80adef6872aba", "output_pcm_sha256": "f36ee439a6afd5df25e01660e1d4cb114428e12417bf2b021dba3d9af323b9b8", "result_sha256": "4ffd09159af0074be25a8935aee9c32cdeef434b7093c18b382b4fcfb4ebea73" }
#629object_counting heldout	3	The answer is 3.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "8b13635bb07cfbf2f9e28cd610486702c0a66321fa462dd479bb1fa9d5f414ce", "started_sha256": "23d5350a5f88e21f510808b3074b2188daf1d47353f94359f424d2f9295bda43", "original_grade_sha256": "6f071e8c1594c7e0f5acad5dbfcecd6e4eeb78ab336b850a68ed18f4db9f5751", "input_original_sha256": "8b13635bb07cfbf2f9e28cd610486702c0a66321fa462dd479bb1fa9d5f414ce", "input_wav_sha256": "cb44c34c640e8cdd38417a5bd36af332b67cf2e3ecf9b8c3874ac59ef89eeba6", "input_pcm_sha256": "a3ad0f9200dd81fcf48a7cca74b95703faf8cbb1959956f4cf40557ba949db95", "output_wav_sha256": "47d55694c235a8702e31119eb8c7761eeea0fc9bbd3354ca3de105018fc72d6f", "output_pcm_sha256": "5da7af273fcd7c745673d29479c31ee251c3251646cfd6cf1ab0c9e6e06aebe8", "result_sha256": "898432df3e5aa6d01e89054418119fafad64b15a58419cdf79cf935ac521f3b6" }
#981web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "dc71aed16bffbe163e8bdcfba66ff8b776b7b87d5da7782f271d24a9191eaaef", "started_sha256": "a35a8e5cff7cc193e5705c9669626a3b94e522587270eef4ffcd6801bb45aa35", "original_grade_sha256": "3dd9922e09b09bf75a01749461622cccfe91ee78a31155e4b7b7e6e4ea1ba09b", "input_original_sha256": "dc71aed16bffbe163e8bdcfba66ff8b776b7b87d5da7782f271d24a9191eaaef", "input_wav_sha256": "01e5e7be132fbf231e291cd90e0a516b2b0e94ab70de23c5710fb74b27ac6927", "input_pcm_sha256": "6f60c5efb1993901d980ec8427dd8e9ad2ee8c0d49cd56f0135c4c7da9aa3c46", "output_wav_sha256": "8745dd36ea1f31652896586ab26e1a9e01bf6080151905abcd846e1062fd34ac", "output_pcm_sha256": "6c33e6b9b0fe93f4d290b0cc588898f11630285f7ee5fd655118e404b6fd40fc", "result_sha256": "d08b695c1e7a1981baaedffd68b0c0c94077dc47b2d07ad8aff9e0f6e44f35f2" }
#35formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "a5ebc08c1f9c06b16034474d0176ec0c900cb0aa4e819a813e9df3f283716116", "started_sha256": "d794b65ecabc75ddad4fd03cc2e7b272694c80d5796d808427aaacb0be269d93", "original_grade_sha256": "ce2b300b2c3b99c4e4e2db41254a8f5b4266a0efda47e670b52fc1e57c0e442c", "input_original_sha256": "a5ebc08c1f9c06b16034474d0176ec0c900cb0aa4e819a813e9df3f283716116", "input_wav_sha256": "4854a51150e7193eb961e4e61cf569c991c84465017feb10f4e92def171791db", "input_pcm_sha256": "bd91815acf21fd7b479229bbcc3117f4bf02932654126eb149d19c4bb9bf375e", "output_wav_sha256": "1414589e68dbb1c3891214d7b5b4d3a2d183bef06aa73c085fbc081263d5b5bb", "output_pcm_sha256": "8daf5cfd6b75209c72e2fc6c952f980ae4cfa96657e7569acd69a7c9ef1de6a2", "result_sha256": "0658b00ed3ec0f0acde50023b9f7a39b515f4077a6083bf4156412cc75e606d2" }
#495navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "ee60b28819e6f5f2b9c2b505387294b1c78d758622963214eddade019e553606", "started_sha256": "9fbc16b53d5a3bb586d221a996e293d70292aac00269a6dbc65c3a60eb9e8048", "original_grade_sha256": "33074b9dd7bbe0c3e9ac9937d875eff356a6fbfd8798593dd7be3c3aec8b3f8c", "input_original_sha256": "ee60b28819e6f5f2b9c2b505387294b1c78d758622963214eddade019e553606", "input_wav_sha256": "5be6d9f767dba8e2582d51cff9a88095f4e7fe3edfc377ecbdf9b8785017a391", "input_pcm_sha256": "49f4e7cc6d482713db01e4417beb7475e092125d3865bbd7225327bd82ce5b63", "output_wav_sha256": "3383f0a0185100ea9603246b407cb6833a1b3fda3caf6b3b474a14f76f08b2ad", "output_pcm_sha256": "ea3b1876dce1827f9e9dbb621d10bbc22c57043d3135dfa7cf58533fbe6eb593", "result_sha256": "8187e1a7a76eed130aa3d572a01279bc0c9eae4f0d64b4e7d879f233e434d8ac" }
#605object_counting heldout	3	The answer is 3.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "17f2f4064e7e88801c5ef67bf37eda77e9ff4fde4c5682022c2802e5b092576b", "started_sha256": "c50f073253818bd517f62cdfe710bdf0148c57f48f27058919f01f5a4aac77d1", "original_grade_sha256": "e27afdcf7122a2f8b050c728b13eb5eadbea024fd25e5c81b973c5709b349677", "input_original_sha256": "17f2f4064e7e88801c5ef67bf37eda77e9ff4fde4c5682022c2802e5b092576b", "input_wav_sha256": "7a1bc690286cde18a37ba91911d1a01ba1cbc2b18e11f45c23faf7aa43dae899", "input_pcm_sha256": "504cb9c99d94d81abdde84691dd4e15561876b44a661c0b2708673b086996c7d", "output_wav_sha256": "d472a1cf414d1d9b859f9b28c5346bb1cb7162b0839f9bdac0e79ecf953c15ca", "output_pcm_sha256": "3d49304ff6bb07f917b05ea415e4f41a1d391e2c2f42480da278ae637ca4414e", "result_sha256": "a0d8f90b3029831aa072859dd1bb52a7138cc4993e9e71636b466faf833b9be5" }
#868web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "71ccae0cdfa8229a5889476d987821a8189516686f46e2dc50d9a916cc462f57", "started_sha256": "8c54b7bddad3aea1a5677e8a1ac143ad488863a4ce37b3832561b4b828a81897", "original_grade_sha256": "697411ca1b95587958d59e2b1727aebb25c18c2144e5285143a02ef90997e6ed", "input_original_sha256": "71ccae0cdfa8229a5889476d987821a8189516686f46e2dc50d9a916cc462f57", "input_wav_sha256": "52665e410565741301695b9e3c4eab3bd2d3858514d2a7057d7222f12ecb2243", "input_pcm_sha256": "91ff59340f4d8e1540e9a3e64e7842d7bc58c598bdda36f9feffa726946749ce", "output_wav_sha256": "5e1942b5b0fa0b58e88e34c82511e9d9246f535093654afb89977d5b20efeb42", "output_pcm_sha256": "04890d70ce8c7330de658663a87f02bfa3386c5c24e05882f516c6f21ed8117f", "result_sha256": "7741b33153407359cf8f07a36c530ccba3844c3c427fdc62b496dcd5328b06d5" }
#181formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "22d4a0df3c67eda6ce8b523105693fba170f46aa1bed3573b622b9a500c5dc78", "started_sha256": "614b9f306e085e1643b99ee5ee58e73718c3dc4f2e21287534ef9f2ed7c0c076", "original_grade_sha256": "70714dac1cd0b05f950718299ccf2ce0f793552e80d0f3d106d85c09bc30da23", "input_original_sha256": "22d4a0df3c67eda6ce8b523105693fba170f46aa1bed3573b622b9a500c5dc78", "input_wav_sha256": "2ee7d5afdbdd48d3c331e1ee769d9a71f1454252d2620cdae625c4390a3b3f78", "input_pcm_sha256": "89064d692481f2207d7d3400f6a6bed8ffe67e1b16ca965b6891119eb5eea520", "output_wav_sha256": "c55778423ff81f49025e643dae5619040124f3836595abb334aed99bd5d6b124", "output_pcm_sha256": "dbf260139c6d3f622e46c05b8b90e74d8de9e49f351f5efc5f0bbe9c97cfaa49", "result_sha256": "2725fdaa4d9be14da901a2f584eb49a0289aa5cefdee50369c766d9c2f01f66e" }
#368navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "891daf18ede92bc835e7005a10b6be88ee213e3b9a6c357fd458fc7e1df62239", "started_sha256": "1c362d51f088b7dff789c37e18e27210a6b2379e19fbbb72a1340a79fdfe8c95", "original_grade_sha256": "15206f029806b3c7ec3795d5d66d81d37cde12fca68ea8aedc1be8af59a43d23", "input_original_sha256": "891daf18ede92bc835e7005a10b6be88ee213e3b9a6c357fd458fc7e1df62239", "input_wav_sha256": "be70a08c123eb491c60d34ea07980286d209ff410926dceb95795ca6ab7efda7", "input_pcm_sha256": "75d55569df8fceb96a5aece5297834a15cf2df06c6a99c833be3363d9322d261", "output_wav_sha256": "e4f1863fd1e52adbc9ceebb3939e24b4c6bed1d84dbe5f41de14f80c1644d007", "output_pcm_sha256": "377bb976f62b3954ebfbb5a513c6d1c197737c528cd20800feeddb6766b2834b", "result_sha256": "9c9d4a4435515739a0270978deef4c00342dfd90b53f6497e95fb173c9b70f73" }
#620object_counting heldout	15	The answer is 15.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "cdf625c12dae8e81406fd51fc87c0ea63d6b5e6ff5e9e17aeab7c805a94a488d", "started_sha256": "ba1d0a15bbe620aee2ff6dd3ed311b47bf59a2bf4a698680ca812a07f23b4ea1", "original_grade_sha256": "ac1066516a0e68293e8b1afc89087823c7799f40d62ec44031850c827bca2666", "input_original_sha256": "cdf625c12dae8e81406fd51fc87c0ea63d6b5e6ff5e9e17aeab7c805a94a488d", "input_wav_sha256": "88adc864ef1985a6b945aacf3bc6831e26e52ffcfe58fe3ccd98569245a279ef", "input_pcm_sha256": "4c0b0f23ea04855bd2edb79c8be3bb8177de6eae00fc07e344aa6072f1575a8c", "output_wav_sha256": "a1db83b5efa9c326702ae801b6119bb31826a3f22dac72612c298a6eb5e7ddbf", "output_pcm_sha256": "13ed33a63686a60260acb38f6a4b5089f75ad5fa42205f459575ad77d31ba2f0", "result_sha256": "c745f06af30c816384663dab448910ec4ef930b77fb0d68e7953965983cebd39" }
#823web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "27c9473cf79b030641c63efc50914dcd0145d5ffc2ab72a32f91f9163359b1da", "started_sha256": "74ff4c1447bbe9da494366441a6f890dafe3858216dc5263bbf8266c931b0941", "original_grade_sha256": "3d2e18c692ab16aa2a954e78f103b380f2f649600191e65fc3723a503275f7b2", "input_original_sha256": "27c9473cf79b030641c63efc50914dcd0145d5ffc2ab72a32f91f9163359b1da", "input_wav_sha256": "f61c09da760ae1bc96f7372f49234cea95e396d6f5dde79264e839f12ea0349a", "input_pcm_sha256": "1ef4620ec57ff4bc5b9abc1e5a44fde5237254797fd4d04da0f2cf526880d49f", "output_wav_sha256": "fe86035533072c4a6bbf209d16cecfad15d5bd675d67dfe5f27ca2a166474980", "output_pcm_sha256": "1940bebcc79b8edb51a71bff8c0b572654dcf2b75f7c93b33b6d635bec552ff3", "result_sha256": "8d47f016ad7a1a3c49399ca30f8603e35b56f0df7ee1a9794bbdcbbe8829cd99" }
#53formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "859a57bc94c20067129c827bec700b72b8d3326079e0f29d9352567836a03c77", "started_sha256": "ea5fae3a3365c195b43dcd112b75baf002245206ed656e70cc711601da8ce8cd", "original_grade_sha256": "332500229314f5a5cf29c42555635b6fd7de1c7e4ec95a6aabf2f29b410b1ca4", "input_original_sha256": "859a57bc94c20067129c827bec700b72b8d3326079e0f29d9352567836a03c77", "input_wav_sha256": "daeb243e248a0b249fb8e53384a4f3add448b983bbeba2d12e5814e6f9c3bc49", "input_pcm_sha256": "62e825d3636fd920e3e0b7192ab8186faa39e8c9a8192b249530214db4a75500", "output_wav_sha256": "c0c9993c58f2b7f8275412b4b85f1464b86750f4817be0d5d4b2fca40f7e0c37", "output_pcm_sha256": "fb099df0600bdd0301c7f4816d0f804c03bcaea08514820a59c4439396f778b2", "result_sha256": "95de53bf81513801b2565370fd3cb4ca787fc413f5706f0f6e78d69dc6a89150" }
#339navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b5697469e5455637fe4f71a14ad28a1ff3c91f31db3a6559342fcdc5612b90c8", "started_sha256": "eae9816f09ae0a74b18a1564255d25caebf490b0f0e75ee64e81f5a9b5cdf7cd", "original_grade_sha256": "372b163fe6c77cc605890535a12ed148b06352751d08271c3116656ae7d8c327", "input_original_sha256": "b5697469e5455637fe4f71a14ad28a1ff3c91f31db3a6559342fcdc5612b90c8", "input_wav_sha256": "c4393c28b1e2d75d9f3b965482dd89d1b0b48c3c9ed4d167316ac9f97f8c73c7", "input_pcm_sha256": "670203db2f6f9e3cd27ed5a306b3c94e9d5039957165ecbdc83151d64ccdddcc", "output_wav_sha256": "9bac298022db797a29b649326624b66353e676da6d59ca7cd77ca5981694fbfc", "output_pcm_sha256": "9d726b14f1ca5e746e2a71e1d1699e1c1eed76624eb7e30b21fc57a608f5d881", "result_sha256": "b59c92b1f01549b8e613fe6353fb36cab301d4453ee21d25430c44eda5e43b4c" }
#681object_counting heldout	5	The answer is 5.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "e15d373fca94b1794ac81cc0a81404a617aac297785914348e92883e322b0ad4", "started_sha256": "7b50f1d610f3da651e3c0b0eb884e8e33bd17b1f1190fdbb222a59cd1f201446", "original_grade_sha256": "8aead8220b65b80b8e3150a9ee1f6977824ea88f59969570e167e6c5594e5b32", "input_original_sha256": "e15d373fca94b1794ac81cc0a81404a617aac297785914348e92883e322b0ad4", "input_wav_sha256": "7dae15f421a715ac58b24a8a278027174201d57690b494279e6c3292f3647155", "input_pcm_sha256": "f0731c4833ed9de435c78dff3549ff1b075a3d50a73c96d59997a4ea164cce32", "output_wav_sha256": "f7526cd0f11df79fcf8a04eca7945eb5369e248905d53862c03bedc741c1f941", "output_pcm_sha256": "bca259d1a9b4b03498fa17d0d025f7c0f1261ae3e2ed92bfc13d61eccb11c846", "result_sha256": "1ff3a41466bae905f5b05f41ec33766ff42ca44c158217b40440444f38d62e4a" }
#867web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "bc1ad79a6bf1a256f5ff5db239c192c88f7981d7a8504c04d7ad9dce92c8123f", "started_sha256": "de602073ef9cb6ae46266497dfdadcfcf2d981c9e99d1c9a56f3faedc7943b00", "original_grade_sha256": "bde900882dfe62067ccb07e50cc13750b11e15521c39e3d331648684d5d174f0", "input_original_sha256": "bc1ad79a6bf1a256f5ff5db239c192c88f7981d7a8504c04d7ad9dce92c8123f", "input_wav_sha256": "b2a1b44f63a1e8d78690cad8e702fd20b6a38fa98fda247a9763aadaa89e87d3", "input_pcm_sha256": "e4fd50256424181c1b7ee824ec681859969fb4c2c33c9e14a3c74aa785022bd1", "output_wav_sha256": "49b700061b0abc19f4a823df0765fac114183d430b7a37f616fff60a766a0fd6", "output_pcm_sha256": "2b9d664968468900d7649d6ea50ad384a85c8fa6ff381f699d93e0feb3aabe9d", "result_sha256": "cb18fd3a6641f744df95755c622cc7e172ef0a163cee40721b00effb6642ed4c" }
#172formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f1b21c837d9ec1132d7610c0fac6deda2a149f8d154336c4fd232558533a3ba8", "started_sha256": "1fee9caa25b03c09c4f5750cf6d167aa997c93796c3b5905a0affba6403a45ac", "original_grade_sha256": "42ac0570642028a1ac6f7a0aa76b40efc656f6506041b94cc5667e892787b9d8", "input_original_sha256": "f1b21c837d9ec1132d7610c0fac6deda2a149f8d154336c4fd232558533a3ba8", "input_wav_sha256": "bf3a76fb027c3c52604f1483a9e133b6c1e81d764200e30428fcd7b3ac70dd88", "input_pcm_sha256": "b56a36248d9844e92fc704f0b0fe2c8aad370bf348ad01853b9621736e9dc51c", "output_wav_sha256": "00eeb43172bc1d57b02ba6f70042d8cf4443bf767cbd0cd584231c0aba055dd6", "output_pcm_sha256": "e18a44e1c92e47350cd2a39882a44aeef80aac23ad1a2af7052d4b3beb1b90a0", "result_sha256": "bbb60dc996c74545798acce252babdbff04d27ff68bfd4b150923f9585ccb602" }
#430navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "33ce6f7e8a54a6c90e2014be6665c63c8a8091dc00392ddaaa91f87936205acb", "started_sha256": "6d802fc14c26d54e9bd5f1d514a16e29973d86cb8e8e1eb9dbb4c0af854ece61", "original_grade_sha256": "7cb99c89ded341451a6ec6a545c3b01b92bf825f7635cb6644ae54b12e3385f3", "input_original_sha256": "33ce6f7e8a54a6c90e2014be6665c63c8a8091dc00392ddaaa91f87936205acb", "input_wav_sha256": "c94128154f33e8feeb1851688d90bc0d855bb99c4e1032635a39afc3f55cfde7", "input_pcm_sha256": "e4145bb61ad7323f2a30b72a2bbf0b52a3a7bbba80e35a1835e13a4a9ec412f4", "output_wav_sha256": "186509f8e665597d7ec67a339da4ee6540ca211fe785a6e1460bb147cf3e843b", "output_pcm_sha256": "bb92b06d6eb9a45e7ab984575d076f6c79fce322b4eb116fe3bdab586d57eae8", "result_sha256": "21d185b069d616fff21d87064f7838d95c1700502a7bf1aa365b1d4634c42c81" }
#502object_counting heldout	3	The answer is 3.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "dc9187f95a67db8a9a7cddcde0c89e9b9554e605b53c86c5ccd33cca83698a02", "started_sha256": "2a1a5b4ec04fa87f1e2887e00d25ee19061a8a24ee402589a63bfe7474dc0fd1", "original_grade_sha256": "e0e86e52de14c1d7e8133990a5c7c50b887aa44673d729c38a203bb60a141ecf", "input_original_sha256": "dc9187f95a67db8a9a7cddcde0c89e9b9554e605b53c86c5ccd33cca83698a02", "input_wav_sha256": "07a422cf41f28a856cc52ff9bf17fb1705a04a2209018980ee07f97532e36377", "input_pcm_sha256": "52fe744a42af221849520f59a5f631448c7deac1c21ef8a745e6b60e68ec8afb", "output_wav_sha256": "63e6cf91132dd8d761c1bb838d19416440f61bd93d1738e68f19c41be9266fda", "output_pcm_sha256": "302d249965062e00c9e1027c176b770ac1c326dbb44178c58559c24447ed4d6b", "result_sha256": "75e82ef39b3afa8fc02e3a2f3d17c57533e4110aebd6aed920c797e70d808a60" }
#756web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "24bae3a87a2ec75e365687072e20efb8452ec830d27bd44e886836d677970481", "started_sha256": "cf678d6f4e4ef3da5a4e9c04c46536542761c7ca028b1fc7d0af2a58f138191c", "original_grade_sha256": "1315f9f7a58f6d744c28b8845299f60088004ae6e8a2208db415d1d86d84b39d", "input_original_sha256": "24bae3a87a2ec75e365687072e20efb8452ec830d27bd44e886836d677970481", "input_wav_sha256": "5c3ce112d829a6d54ac380d98ba7bf96a4a24d4177266bf535e958cc37d7ab39", "input_pcm_sha256": "9d2f030d77864c86e1a13d1aaa2bb8264c930d125e7c0b1f152483e3296d4669", "output_wav_sha256": "af9e9d20d0b3d98038224dea07f6f14a2f9c3e2210f3f69c46b8dcffcbcca71e", "output_pcm_sha256": "a70b9bf71894c9c28474b32ef993cd21574341d0451452f2ca18091c283b8e73", "result_sha256": "ae656c7dda9460fbc3aa9c79f9fcd322c7bc11c394c9b6ebf197b82db4ca0ff6" }
#179formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "737f509b10f82c9b09ac3d98da9a790d6d86b0d08fe161932a23e4c090f17e40", "started_sha256": "2f3ec4b6a6b7ccedddf24f20685bd3bbb2050e35555a5c450de5a723ac051569", "original_grade_sha256": "400936ca694306d52227a9eb1ac88ea17a24559e7ff89ce9cec53c06ca8eca95", "input_original_sha256": "737f509b10f82c9b09ac3d98da9a790d6d86b0d08fe161932a23e4c090f17e40", "input_wav_sha256": "42570f5f8a4f4f128576f8de22bd2ea0297dad2912b6b5e3bb0730402f46eba6", "input_pcm_sha256": "93c93801476ff3711383d29da78dd8c6d6a3ab7823eb33b739eb0e463c59ceac", "output_wav_sha256": "9ca460f3d916827fde39210c8a8b6f674f871dd49f58ceb27eff3abbb89d1c6e", "output_pcm_sha256": "3a60fe491cab5a35081c7e1fee439f07ca619a39ae65a93969b3606f80953a69", "result_sha256": "7ccb7d35bc0a33135fb342d7b4d579b7d4ba6da5bc23d4679f7474b12c4c4ba3" }
#467navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "3b6e0780e6231b0886d87d3aaa48280cdab7be8cb6daf3bedb215ad760c28651", "started_sha256": "cc2a370fb0c67ccd51aaa8fa0781416df4fef79a37fc1f0381b16f4abbfc6c8f", "original_grade_sha256": "caca26bd444dd474a14717a39a4f8fa3a8375216b7fe31ecdedca5cfe87ac613", "input_original_sha256": "3b6e0780e6231b0886d87d3aaa48280cdab7be8cb6daf3bedb215ad760c28651", "input_wav_sha256": "84c6b4ae1a20298ca0b7759120a875eb92413b81d88e41d59c2d0d3b9eb4ea7e", "input_pcm_sha256": "524581eaf2711ebef3bb1389a65e846a506a2d5a87ab4698376855949fe796c5", "output_wav_sha256": "edc45caeef4c2c6d0e792838600097e81cff8ccecc8fe2698a3f9a8775b26490", "output_pcm_sha256": "d54f961287fb4c07bc3ff8a13e54100b5677626a3e0c35a4b8045154064e34cc", "result_sha256": "aaf8c80edc928b5e2fc16e8000c99a84f951c2f25a8c544a59828ce17ea42cc6" }
#682object_counting heldout	2	The answer is 2.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "59b8dd9c3a86be594c55a40bbf2106db58e2eed00dc187bec8bfce4079c4aa8e", "started_sha256": "b6c136be5458f74c7ce1870c450709bdf580f45b62fcd0c089142d8e312f62a5", "original_grade_sha256": "54ecea6863783e19788830b17040a4ffe2bbe2fec31d74862f87c5a761c202ed", "input_original_sha256": "59b8dd9c3a86be594c55a40bbf2106db58e2eed00dc187bec8bfce4079c4aa8e", "input_wav_sha256": "c12ef09e18c2e9a7bf9603577a0daefce809d1910c89ca3b1ba0b21c7fadd16b", "input_pcm_sha256": "0732518670a5f2f1b38279e155b2bc27f3681b38753cdbe5b425da4745564c30", "output_wav_sha256": "fc38ba48336e39ff345dcaa5e67340069e57494609ddf9ac43190421c70620f4", "output_pcm_sha256": "1ab7cf684ef234efd03939944c06449a7b4260902ab134c5f336ae76e25fa287", "result_sha256": "3cf79273f2f7bbd82d167bceb422a019abbb877590d541ecd3569cd8cc5da493" }
#986web_of_lies heldout	No	That answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "6f3949da170908f93e0bef69a4878397fa2c94cba4bb9b1b021039a7575bdaa3", "started_sha256": "6bbaa0c1e7401f245b6f201ece932b5f70e79bb7399bd670539aff213edd8e99", "original_grade_sha256": "76d140bee6fa20666a775ad1a2efcf1387efd30680900012fb433e6541ea4d25", "input_original_sha256": "6f3949da170908f93e0bef69a4878397fa2c94cba4bb9b1b021039a7575bdaa3", "input_wav_sha256": "1447445b93d1b21c91c3e9968848ac6574a6e34c65750812afe0aa713c13a8b5", "input_pcm_sha256": "522a0c3edc3e768ed73745aec92cc9bb25253002ef6fe6c222563c05ae2633aa", "output_wav_sha256": "344378dc2a82e5034e1972ce659e3ebdb5a7f46b1e25771ad52b78b938255c4b", "output_pcm_sha256": "9416bfd89e30cd9de172b6146c3b13acf3a26f9207565a63108dacc6b776a450", "result_sha256": "8497db8a3db3c75caa86f28d9ac8908d03aff1c9e59db374e8f1656179f0a877" }
#4formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "8b662078bcd2711fea578257de018a17e672cbc6fb436d9697dc33217e61f3a0", "started_sha256": "09b858d36edd785d01a4fdfae61df414d0a0523aaac612ba5080db4099d26cae", "original_grade_sha256": "9a15cdf1d9de4c983abad6a83c2614de25230982f5a4fb09f30f8189cfb7c9c5", "input_original_sha256": "8b662078bcd2711fea578257de018a17e672cbc6fb436d9697dc33217e61f3a0", "input_wav_sha256": "dcfe7d9f96891ef4e34b5ae333f1c373e1b72d3e18c6816880fcecda32a7151d", "input_pcm_sha256": "7d2cccdcc171959e3484b0937dcc71e3cc88299b30a81c9ed27405021609b083", "output_wav_sha256": "0297739a716a68102fb0bd729dcb2995fbad32d6f9f725bf2c34cc9fc1b8a8c1", "output_pcm_sha256": "ea197bc51d7ea3d531d5e83dde44b28677a940bc8f48275f56a5317dac93363c", "result_sha256": "834f6b699068e759ced39b3d87f3e0a5d45f1b426d85ebcdc112f3deff3cfdc9" }
#353navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b0182544117dbf5727d6cbb28961c6e2701c63f0e2eb81b7c8ded55f630ac5e8", "started_sha256": "1c2c83dcc6a179d7396591ab8b5744d75e05f1c519a2db9f7e77ea94d96bf0de", "original_grade_sha256": "1d46e40e8d32e59bba6c96a3570b227668e457868d668ebacb2831490a67ebb3", "input_original_sha256": "b0182544117dbf5727d6cbb28961c6e2701c63f0e2eb81b7c8ded55f630ac5e8", "input_wav_sha256": "4e7b47e3b5654095b383ec7be781f6d97bb25508530d62cce58a4818a00599b3", "input_pcm_sha256": "38c1c0d1e5457b7f363ae7f1fdf88fb1579935b93860558b88488ee688b8e8c2", "output_wav_sha256": "ac6c27eda4fec67e9a6df648c069b308f2e03160d4477ac098f67dbda2953beb", "output_pcm_sha256": "847adf0d08d8a047f1bff7dca950a3a77c4deebb366cc751ee0c47c03d4f8905", "result_sha256": "44e7b35fa40a83492cc2c1802fc85404f09749d2cc47dc6db5c3ffc147008971" }
#678object_counting heldout	4	The answer is 4.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "59918c5bfaf5fafb9ef566af548dde899b58354ce17d7ee2c83e6edc21bdda42", "started_sha256": "35033339fb3cbd810cb5bddb49d7c10f0713ee5727332aca6377e2cbab896c09", "original_grade_sha256": "a68bace28b6343e0247ba13f59aa2713f803ac704fbd95798e3f3360dfbde8ba", "input_original_sha256": "59918c5bfaf5fafb9ef566af548dde899b58354ce17d7ee2c83e6edc21bdda42", "input_wav_sha256": "f738351a0a3c2058f175db796b8d8902a835667bce1276c10659f4c75474edf9", "input_pcm_sha256": "2a3e42a5cef7a063a1eacbcc25515a74dbad7c078b95b30fa5afc12df6ec95ec", "output_wav_sha256": "895cda79792c786c2d637cf231240e3714bb65e67132c6969a2204a192fa587c", "output_pcm_sha256": "c0706c93db5674c510394bf3275a9f0a8d77d5bd9916adcd260f878aece90d15", "result_sha256": "a99245e62c9bc1011ecaaecd69c4dde98f15b14fb942d5964b95da2726265a06" }
#810web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "ce6b00f8a7ed99fd6a1972571770017473eea206ceb26e0a0668a790566f4675", "started_sha256": "b5c6a94f6b1433f77b193c199679f0f7758a2cf6776371e95bbe1aab74be2bf2", "original_grade_sha256": "a55d0637ff6d51988de76c3e2bdcebfd3686a0cf0c01b7ace863d4847ffb0f40", "input_original_sha256": "ce6b00f8a7ed99fd6a1972571770017473eea206ceb26e0a0668a790566f4675", "input_wav_sha256": "43345a455cfe77d83e1068ae910526f6cd288303e413f66418cae88935c92925", "input_pcm_sha256": "9dc8f47acccffad620123a8e004d79e2a65cb4ad7659ab97333f57b7008df611", "output_wav_sha256": "b4138d0a046e196f1107f098f7e90725acc1563e7ac4a5a3d97a688ceb6ceb94", "output_pcm_sha256": "3f9b5854b405212d4249b574863a4a3de31f47329e8282238d3fbf5de5530b74", "result_sha256": "16b02f8473c9412900153e9014cbcf2d86bdcc88fd6bec35125af561ae5b539c" }
#166formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "d8d9d9d412fef5dd3110247d64e9662d0f3e4a2565e8a9a5efe530e7a59c1f74", "started_sha256": "536b0c99836e7ab346d43069bb159869d48fcbc0bb42187f22f49ed75dc02012", "original_grade_sha256": "cd92c63bbf304db6b4fe5ac4d7d2584094bac033c1256f839055678acac38038", "input_original_sha256": "d8d9d9d412fef5dd3110247d64e9662d0f3e4a2565e8a9a5efe530e7a59c1f74", "input_wav_sha256": "646f5d9a7e9c11a205a4cb0e4d1bcf94ea4c9b057bf0374195e43965026d304d", "input_pcm_sha256": "58d56f20051cd2b48f8ecda4f14a1ee253950a8ac946a5817bde0bb84950b25a", "output_wav_sha256": "aa50afa0e656b83175c32d4e836a51eb29506f0631b57ec4e43b5866c0fe0f89", "output_pcm_sha256": "725f1e9f10ddb65b87e01a065d6b133a35914e8a6084e98fc859fe84d4a2aafc", "result_sha256": "28734a3534eb913df4e5c66fd189320c896b58e753a73eb1a56d40c9db3990ab" }
#462navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "fa59b5526f952259b7f580ef85bceca7d1bb8f04b0bc6ce6be3d6ef91804d849", "started_sha256": "b05452f93c26c40143572688b10f691fcaf9b012b837eb1fe8eb6d1f1d528343", "original_grade_sha256": "492b9002c1527694190367cfc9abc4f8f72bbf4de2db9615d2ef35bb7f8ed994", "input_original_sha256": "fa59b5526f952259b7f580ef85bceca7d1bb8f04b0bc6ce6be3d6ef91804d849", "input_wav_sha256": "f8659f88a6fe59a5fb43b93070f24b20e8dafe6f4ab8a70346fc3637c9103344", "input_pcm_sha256": "ab73dd077a57fa6ecec01e9155c65496c78a5c21621de695e09dee99d950f24b", "output_wav_sha256": "24d86076ab1eeede618fa7b3e508484efef96101dd8925c6efbc91c9c6ed9260", "output_pcm_sha256": "8ed1f3b0ac898d9757eab32d4716c346ca93942f436791091c58fda295c55168", "result_sha256": "f335bc4e3a05abd21f13901fa8f85ff45b3b634c268422f86fe00dd25c9d9426" }
#630object_counting heldout	3	The answer is 3.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "2736852ddf38069dd5fff60d51300581ec9099b92077ea70584eef8491280449", "started_sha256": "bab663c0b30db4c800d5b6e43a02f9b3eca625f25851b70390b349b9b5c2acf6", "original_grade_sha256": "3c4d448622d7c285da670d3baaf668dbf4c0ae44e6140de487cfc318a561811e", "input_original_sha256": "2736852ddf38069dd5fff60d51300581ec9099b92077ea70584eef8491280449", "input_wav_sha256": "54c3f64ed2583f0357a78820d60a059dc1f2f805b4eb42dbb40e373cc5dc49f6", "input_pcm_sha256": "f92c4ad214a6eac0386a9ffab9751efc3236f5fcd0b3d2178f300825fea6dc60", "output_wav_sha256": "10a4d606a40742028fc3ef611b8a47d50f6a9836f0a799d36bb6c68dd9bd968f", "output_pcm_sha256": "7c9bef34066b7f3152a93f728dcd0fee3888274659ca37c44975d2c489ea17d4", "result_sha256": "e4e37ec1eae41de2c4750469969107a10a7d10c6425c4f673632eed9cb8e111b" }
#916web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "c6befeebb8de277eeeda681b60007e8232e6fb074bb4d5d535877fcde5737a06", "started_sha256": "13d34bee03adf08ec8b33b953d3faea0e5b5891a1f7a47e3b4724917f1e74bc5", "original_grade_sha256": "d3d80d47795f4b862ba143ed404a6179396821ab405b15fe18695e5139947335", "input_original_sha256": "c6befeebb8de277eeeda681b60007e8232e6fb074bb4d5d535877fcde5737a06", "input_wav_sha256": "b000e37598503cbf5cdf0aaa18731f332f6a31917259db91bca79c064dbece66", "input_pcm_sha256": "71bfdcff16800caf3c6b9e8f70c20cbd470e17bf90c87f843f8a13df26dc019f", "output_wav_sha256": "abcdf3e3ecc20c0906c3a65c01ee6d2d3bcc9e45e5c70df78cfa3b3c50327f7d", "output_pcm_sha256": "72746224330c9725cca28a62817c135ea5761a3e218ac2cde97107dde5436ef5", "result_sha256": "6c315171589da81212de84df9766316fd7c9ad0bd2d978ad4793e2f2745beb28" }
#60formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "3101549fc0a8e8c528fcfce9a7cf684d04e1c5531dccf24ed2ec15447c7a43e9", "started_sha256": "be675b006308dd5d390f0d12e0286dac44ef0feaf1832210c01297af59472692", "original_grade_sha256": "2777dd0240e85caa3a1cc045c282e0506ab5c6d322bb57fe7198eb4dd7c5dbb1", "input_original_sha256": "3101549fc0a8e8c528fcfce9a7cf684d04e1c5531dccf24ed2ec15447c7a43e9", "input_wav_sha256": "59936b4dea7e7f0c35c170bc9e5a1fe9ded536943e2d8268b165c0814e40fa2b", "input_pcm_sha256": "617a6ec89ab2ef6fbf60f192fd20b0b85af2ca2c8b32620b22b54c2246e1e6b8", "output_wav_sha256": "b08045ae91bbaf04329be59694de28c3f4abb2b317e5a8517a1ccb7284d50395", "output_pcm_sha256": "3b884ca7617d15f2bc7a0459b7b448047b9cded19958ec88d8ba8805c7fb251b", "result_sha256": "f21fdf3285b15323aaa610bfb2a4da0d629ae71cf618ac0828226d48fba9f5c0" }
#497navigate heldout	No	That answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "e9ab54975173861432380756fa376068657532d7752005723cabed8de8bcde7b", "started_sha256": "71bfc568facffe920aec735fb477c142247e425bad90abdcc54b98ef0702c762", "original_grade_sha256": "34b9c8bf8d1ac4b88caa773f84cd9e3273ad93188836859fae15c31cd0da5ef4", "input_original_sha256": "e9ab54975173861432380756fa376068657532d7752005723cabed8de8bcde7b", "input_wav_sha256": "f4163a7c94f543ec8b1ce61cc268107f52b6f921a9328536cef221c1f4dacc05", "input_pcm_sha256": "8d05af50a3f4e0921e486b6a1bb83099d5394f1c6e312b8783c5249313c2d3f5", "output_wav_sha256": "4ad57867c414c175563509ce7642a34449623dea3e23a8bbe48e3325d40f2f8e", "output_pcm_sha256": "2fc2e29800d65275cccebfbac11f52227580be364055122a4268dbb6e7156efb", "result_sha256": "a3933a8b22b30d5c2c1fb25036e47caefa16471a3d36fd0b19d2151516f5140a" }
#745object_counting heldout	9	The answer is 9.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "95a5c6d30f139e0b2508f72d6424f3a7d0cfe7f75f0e6ab5ef0326052664f667", "started_sha256": "b3b160ecc3fd44c1a15ddc1c1ec75394e6f55e5ebe04efd1c00351da4e72bcb5", "original_grade_sha256": "5d816521efa7e03d1bd9a1343cc81ebb1b9d3aaf0c73ec79dc5855bcd3cc3e80", "input_original_sha256": "95a5c6d30f139e0b2508f72d6424f3a7d0cfe7f75f0e6ab5ef0326052664f667", "input_wav_sha256": "0708206805288416f34c99ab5a26eb2abd100addc8fe81651dcdc6acbcc2107d", "input_pcm_sha256": "51e4c75947303ef3b7aed2db135a8b50fd0bc44ee8218ff2aedacca9f6b495cc", "output_wav_sha256": "89b7f6864e44822610627aa20f8722d3ddf70a7c2a90ed320d12c16c3a4ef96a", "output_pcm_sha256": "ee6f21da3a8ad12dd65bfb41601c22b04605a71f59a41dccc591d57ee13a59a9", "result_sha256": "7cf065f94eaca9524cb39896c51d93030b9047bbae3f318aba88d4ce8b7b58db" }
#864web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "8d2b85001b148836b11e0d5a87212ad19da5d412af0656c2fe9052a4967ead49", "started_sha256": "619f6896292342566336b2b62da42a1fa122ee477504905bbacc0fe12df796ac", "original_grade_sha256": "6c5f58378285ce37aca3c6dc886a899cedfd59ce61958c99a90a58e6b833a6a5", "input_original_sha256": "8d2b85001b148836b11e0d5a87212ad19da5d412af0656c2fe9052a4967ead49", "input_wav_sha256": "fda17c2fbaf8fc229c8cf4b423bc3a0efc830c2c3bac66d4db3235c704f5403d", "input_pcm_sha256": "cd199d606d8e36b0df27240177b2b3e6daac55bf3e89b5d7854c50e84028d566", "output_wav_sha256": "916a9fac44ab449436304771823b2c596772e92c0a6d1e18ae07353f92e422f5", "output_pcm_sha256": "58516591383c1249ce46a17937df2e39d6e38aeb5c3dc8d3730c71d9c33de085", "result_sha256": "d6bf5db06875b53bc88d71519c772c27140170a8db400e09eb52685aca409a82" }
#153formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "66395bdbbaa12d301db567d2af2991dfd28d38096ad15eac23a6189844c2782f", "started_sha256": "2e474301b775f16dcbfe3acf00dc28cfaa088fbc41e36bde581c2f8d172a7a17", "original_grade_sha256": "69007e09a3efc2934a5edf33fea20edc2b1389fe78b6d4eab3f747c4343a4c3f", "input_original_sha256": "66395bdbbaa12d301db567d2af2991dfd28d38096ad15eac23a6189844c2782f", "input_wav_sha256": "f371d8304432ebe1af727b67836dd2aa59edceca9c4205c3242e5ad40e2cb3a3", "input_pcm_sha256": "73c6e7708293dd7626d3801c569bffd2668fcf847b2463e9b8a984762b15fa75", "output_wav_sha256": "c68b25771ee2b4b7fe6e36778ccebf4f7500ef6a08f403e01001fed80d7b8ff1", "output_pcm_sha256": "5eb26847c4ac52a54212a8ca0d6d8bf514412b8fcc190a26f1c9d545c17b8330", "result_sha256": "ac1e880fe51528797ca3215cd29dd22ab4225a7607ebc73e92302c5321539955" }
#281navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "fee8193cbe69f68be9c3d02c73625b2c5bc4bf2cc48d1b9e8cbe60a5e2341ed9", "started_sha256": "d2967d1e03854b90ad99917d721e3d0f7b7de2c6d3b455d33b0255d0ce06b8c8", "original_grade_sha256": "5eab7f213ffb2657261d67b74aa2d8db898cec2b6b38c466e12d2e90aaef71f3", "input_original_sha256": "fee8193cbe69f68be9c3d02c73625b2c5bc4bf2cc48d1b9e8cbe60a5e2341ed9", "input_wav_sha256": "cffe5f9d55e2ec95e5f3983f5ee64568ae9e93741c852b303aa1b284e80a11fe", "input_pcm_sha256": "77c6765bed7353360e79e92c5935a53591d42b24b0038919784bd7af580ba3d8", "output_wav_sha256": "9e80ef3f466e52b15f62a493ea715c0de2f16d5dfa10980652800bea48c4b702", "output_pcm_sha256": "bf768cc78ce0c1246c2fdd747c146fd49fbeb7a46db76ef158ed493108faf56c", "result_sha256": "8acc001dcb6b51301ca4e2841a66d559dd365f72401d21b4d0903e5edfbd71fd" }
#565object_counting heldout	8	That answer is 8.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "67d785a4f8cb94af250f9b896f7b6873e7820c31b9fe21f079c98df40aac2cc3", "started_sha256": "be47519394f88dd852cebcd8592a99dc0f0db018db784fee13bfc3c369c84be6", "original_grade_sha256": "e048e1cbfad4ef9d8049f6c1f193b5aac992751ccddfe7c3eb7ded05a5e20f3b", "input_original_sha256": "67d785a4f8cb94af250f9b896f7b6873e7820c31b9fe21f079c98df40aac2cc3", "input_wav_sha256": "6fcaaf8d749dd157555a6c51f4595e1abae48fea96be64a3d269a8f056f2bb2a", "input_pcm_sha256": "99b2abe9a5f69cc387940d2feab17e6d5ef5046a8bd2e377255668a8a4eec801", "output_wav_sha256": "ee225911a0bbcb1bb39840683f299bfb074b9e7bd2067168fe227855142e3e67", "output_pcm_sha256": "65e7f2145cdcd222125843eaf2e4da79e0a0f2270f03ff2129281d61dfe10a20", "result_sha256": "6cbe2a25e3cc418db8f0b828c0de99d9b9c01b4abbf254ccf640e777cbc24879" }
#803web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "c89ecd4d1b0bbda2f180209cadbfe784a46b3dec32e5fd9a421d75233292db98", "started_sha256": "6e980469dc0fd4c47867aa2ca86cb760a9da0cbc7b5636da5e0abe1e466f2786", "original_grade_sha256": "fef3cb04c1ab376d5c17b2a049c9ca1557fef5d5bfe450f9b59df29b1c72f438", "input_original_sha256": "c89ecd4d1b0bbda2f180209cadbfe784a46b3dec32e5fd9a421d75233292db98", "input_wav_sha256": "a6df69c238ea418ba51541fc8d30aca49502c5754ff55ca1d55fbfc29d7524e9", "input_pcm_sha256": "c2dfdbc3b8a4684a459ba19ac61b9bf7edbc28541e3beeebe082ba57527b1584", "output_wav_sha256": "cd5d9b5e3f271e3bf7b1923ce72d8b110ec572a6781e2093b62b3ab1fc0d9f9c", "output_pcm_sha256": "eb28eeec844d50a1666c85db87bbcfcdf335d04250eaef133e369e15f9d13bfe", "result_sha256": "210b6484f31de8b9d4b7f0233b5da37fa199c4709dbd8f582b2b7ff24b7d8a65" }
#99formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "513068b86efa08115c46e4f2a8c98cb284ec18651a1323f1691863faa055816c", "started_sha256": "823e199932fd8c1b7511f1efe2d6a87180d51670585426df8b2308aba62017d2", "original_grade_sha256": "c5a72c681b9c5cc0ed59902b94bfabd54a3dc6783fcb9caf25f6309928e23909", "input_original_sha256": "513068b86efa08115c46e4f2a8c98cb284ec18651a1323f1691863faa055816c", "input_wav_sha256": "471991ba87461d9e71f47d75450c445414ab365324df923383dcd4316d26102c", "input_pcm_sha256": "4055e010f6562695d30555fe24186b6376fec8f803ef10e125e697949f8d41b3", "output_wav_sha256": "109ccbb9141b24e68f89face1756f83a3ce750989f6f012d55606ceb4dd0255d", "output_pcm_sha256": "8262f6aba88db44e497b8e061a5f0183f8cc55d70f2e0d3684ba2e5ebc1d48bf", "result_sha256": "799085edfd121e5ea98603cdf8b3957e7fffd0c48a60e7f23504f3c18457d426" }
#466navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "bb6c1b4918ce5080223e5fc082fd8f27d380d5d4ce13a9831b6da231cf85042b", "started_sha256": "ca2b3e16974d6059e51f3b66e4592f0a284c0e01e6ce6e11ede92c00270a115d", "original_grade_sha256": "b13bdc2a5e29384352cb322d34cad587728d48d2136fa06078d4cb763d4b5e9f", "input_original_sha256": "bb6c1b4918ce5080223e5fc082fd8f27d380d5d4ce13a9831b6da231cf85042b", "input_wav_sha256": "a32cb271263ab8a49ac318d815d7ce3d99c2d11beec86eec50e35c4292ca8a80", "input_pcm_sha256": "2c5f2183370b06edd86f841ed6b425b3ddb2bedcd7339ac3e62d2ac10e880d75", "output_wav_sha256": "b12fffd6bb7c207d96910ece28ae1f901f1741a8e7254c7be6a5d9b6cb464e14", "output_pcm_sha256": "a9ea58f48b41eaa058520c72b12d89a4520be3d0fd76b551890b20d1dbc82db3", "result_sha256": "91203ec011593c46a2f70718c9e32e22347ae6cdcbd60cc26afe4c9ae82e5b50" }
#648object_counting heldout	2	The answer is 2.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "397334cd65762f8b9a79701834fbd7e319be1e2d3d4615f1d7fa09a1a429ab33", "started_sha256": "93494df0e01543c12acfd66c137b218c9206a58577a8709e9b20da6b64ee03c9", "original_grade_sha256": "05690e93b5987b5045708920638d94aa013b939835bfac1001d15a145fe4fbd2", "input_original_sha256": "397334cd65762f8b9a79701834fbd7e319be1e2d3d4615f1d7fa09a1a429ab33", "input_wav_sha256": "9658280d1fd749ee74a939cce30b39e03dae6ee350086daf7242ab5b692916fa", "input_pcm_sha256": "22b2f49181a40a6e3aa15c68d660a43e2eea108b6e0793f3cf14ff4eae816c76", "output_wav_sha256": "7dd0590243d705f9459563259453b323a847e2953c3cb9e0d5f4ec3c8255fdea", "output_pcm_sha256": "ef98f4150e1df718b17a886335f582dd35375464f459b4c324622a484a0cda81", "result_sha256": "c26ca2668565d4373815730c5bc8e5c7943d481cd5bd8aefb58222e9b1b99745" }
#789web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "20af5f579205ea4dbc1d9680b5764d9ce066dc8d7d827e2f8108320d4e5b5804", "started_sha256": "a8de7b1e8198b331fd6f7bc195dc235179f7259077d8d1a20b2bcaf5a1b8fe30", "original_grade_sha256": "a413fdf73ceef433b0a01a5274088672d1e13edd7f919f29f926569c1991445e", "input_original_sha256": "20af5f579205ea4dbc1d9680b5764d9ce066dc8d7d827e2f8108320d4e5b5804", "input_wav_sha256": "aee9782a5f74188ae863a586dfab142409b256a73c450949a0c90fff87874a6e", "input_pcm_sha256": "1e8b88fea424cf514a646eec29cec93b59e2e149337fb1d34ac9aaee51e42e2f", "output_wav_sha256": "19c0635aa6418ac93bfcebc0465e1b2968ec08be2e7986376e707d318d7cdc6b", "output_pcm_sha256": "4b0158b08ec264466d45e2c322a95002e690c799d240777dac537bb128f5d0a3", "result_sha256": "d796acf9977548a9f18675dc938f5382d5b5d94790799d55606891375d503ea7" }
#241formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "22e6e4d1b0080dffb6cc1ef91d77669eae52a6e6cdc280ec828ae3a8dc252fe3", "started_sha256": "c84f1a4b05a6d8376aab01d1453181469106cc6671dce3901788d5c7d24b9c12", "original_grade_sha256": "2be6971c9c5d385a093ba045a75416a7dd2499b88e0b4b03229eb43b30301df7", "input_original_sha256": "22e6e4d1b0080dffb6cc1ef91d77669eae52a6e6cdc280ec828ae3a8dc252fe3", "input_wav_sha256": "0a9e48a41fec2868e65f94afb143a48785d5db00d39a3d8a3d96d33e49cff0f2", "input_pcm_sha256": "b0a440e9b48a90641da51a053ae86ffa5f6753f09cd490601ea01eac2069f7f0", "output_wav_sha256": "f767070d6928d28960d7b25f68419049ad6d9949e303d873707ef07eeea82b39", "output_pcm_sha256": "833eae01557d388a2435b39099552c33e6db770c568c894723ea616d54f288dc", "result_sha256": "2b9b0687260030f30359359cb5d9f433f90c0b182540e93a41effa8d7943e130" }
#319navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "1ad20e6e41759e46a90fa783f0c03a085aa18de2f4028abc9f231c7e967cbc7f", "started_sha256": "19aba884d5219c6f4f90b064059b07b80e31ffb98147e55b49f7f85a17d309fb", "original_grade_sha256": "c3aea67a4f42a07aa27faea62d47bed86a671dc836b7f954ed9b75381919d0c7", "input_original_sha256": "1ad20e6e41759e46a90fa783f0c03a085aa18de2f4028abc9f231c7e967cbc7f", "input_wav_sha256": "d20e1f9f17b5f7802e20d01be3b080a68e3bcea2ebd16be0bcf83571d21f54fe", "input_pcm_sha256": "b38699c8a25829589d69b3b88497e8d160b1279f757b9fe6a1199e9b8a4fbdc1", "output_wav_sha256": "b51146886bb0d026f1d8e5bb17e65b687774fdb3693c14e29ac0e26974409b45", "output_pcm_sha256": "eba44ce47877ada4f8f9443db866090ce0b4e578e777f950acf1541ccd92fde1", "result_sha256": "936d2e4d3920e764c56d1a76736d81055e88769964411a60dbbf17d66029958b" }
#742object_counting heldout	8	The answer is 8.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "84ba54a7d82b509017bf571b49232c8768da43818e0ed45fd4ccd9e91e5277c2", "started_sha256": "ff701468fac2b8eb41034c22612c27569069360154b8ad1b6dbc13642551c28b", "original_grade_sha256": "205572906cc3c3ef8ff561a710591476040e59f0a99755cc94629b6169dc616d", "input_original_sha256": "84ba54a7d82b509017bf571b49232c8768da43818e0ed45fd4ccd9e91e5277c2", "input_wav_sha256": "3cd960a8a74f22aafc654586d7d5c0346589bb7a682a13b7ad1b1fc75f7b7460", "input_pcm_sha256": "26953ccac2cc1eabc0b6b245a0c3589d7cc4eb87175b53cc2f912b6d840b97c3", "output_wav_sha256": "3fd965eb7c4e38a4c9a8c9f84504c64e775320803eabc3ef1d72c28311c2375f", "output_pcm_sha256": "c659ca6f9047668ae5274820099a6cdec1ca0110569811abfec7d036a39ae819", "result_sha256": "cd5b42fae24ac28a9f1cbf379925698b4279434088ef1c9f3a7d93baba329f7e" }
#912web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "e3df14ac47cb93eea5ace74c77b59b199129f4f32494c55ca38a02ce39008b35", "started_sha256": "23c56096f20184bf0bb6c5531a55499289b843939b917ba1b6f9d29f2940ccc9", "original_grade_sha256": "e8e5db90d87fe1474cde023806871e55c5cd31aa28be2599677a44dc15c7cbb4", "input_original_sha256": "e3df14ac47cb93eea5ace74c77b59b199129f4f32494c55ca38a02ce39008b35", "input_wav_sha256": "ae8e4a0e9e8bf6c35f3e86131456cdd7e31de5849c2df074014652e3efeb754c", "input_pcm_sha256": "1554741092d7cb77df88b53d322da62ef552e4e39abb55be3aca4276b519556c", "output_wav_sha256": "c3d2cec729a0c25eb5f7aa23cffe3bcfb4aa3a549bf9607c4d62664c36e53c17", "output_pcm_sha256": "d41c689a4655a9e6007561832ae27d58ed7fd9b1b610c6bbb5faeced824c523c", "result_sha256": "fd00bf299546f4940e4c4ed1736d600cb2bb5890d2a0a66dbf12bd0f7bce2f0d" }
#17formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "0bdf00f959f156ef28860e70f546a9c0d59861ab77f46c90b45588edbfb7b21f", "started_sha256": "5a7073634a238783bbc8943ca32acc07419d38b94fdf50b4b49f4b30d052dd6d", "original_grade_sha256": "31bebd52d96495340d1d786b43f244ee75a0318f0ecadec851303d5336cbc141", "input_original_sha256": "0bdf00f959f156ef28860e70f546a9c0d59861ab77f46c90b45588edbfb7b21f", "input_wav_sha256": "8391d42fcac7d4f9672dc1b8f9200a6efa1719a326193b6b915e4608c1f1a238", "input_pcm_sha256": "fb67b397ad56f606836254ff0db74de93bae8c49d38b26f310882b6413e0fa40", "output_wav_sha256": "35994af3f4effe3ff7f91a6fc9ac7f2435efb7709c418fc778ca97c32593e08f", "output_pcm_sha256": "efccad794255117b2ec3a4c0b741bbefc91be13f55d02108690d7e722b7c8988", "result_sha256": "e1c069b3220857c0cb251e8e37b6b30e5244494121d12b419767d38fc0e90d26" }
#428navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "492ef2eb7b2853e0152e3aeb305bb3ff00e4e99284031a99dbcf5ac9f5d4ffde", "started_sha256": "3df096303602fd7c61df431a34d6f3753f601724da4de25316d9d537f3f87e3a", "original_grade_sha256": "f79929a13f7ed50083fed7a3b7bff862af57f80d7a8a8931567b3e1984f13fe1", "input_original_sha256": "492ef2eb7b2853e0152e3aeb305bb3ff00e4e99284031a99dbcf5ac9f5d4ffde", "input_wav_sha256": "b03fab8463a8cc8d035e57ae1929e03f27f89580993a929ded4bb471efd09699", "input_pcm_sha256": "9df9d05f8d3f8c8771971c88bd58953e007a399c774113195c57e055f168de8d", "output_wav_sha256": "b75ff81e44f2459d090c211464eb15dd9cce455ad1cb25306d3e6a88b3f0aea1", "output_pcm_sha256": "f667ae9a8d09dade91464586d813df8adfe8c9a63907c694f1d86392a6ff1c8e", "result_sha256": "ac9658be06dd31da3ff0ec8739c8a5372d1e51fb8f936720cb1679e702f7029e" }
#519object_counting heldout	5	The answer is 5.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "2fdc0f65ffed53e06d801bb3d8c146efe37975938c6221cb7ed2d58c9a11e7b6", "started_sha256": "5bf5b4e1d0e80ba616084fcf9fbae294be35d9cecee0086d1e7e2d1694eba725", "original_grade_sha256": "34077ecb9e8664b6eda45298067c89aa19c6cc91c2c4dec03cf5dec4bcca4e63", "input_original_sha256": "2fdc0f65ffed53e06d801bb3d8c146efe37975938c6221cb7ed2d58c9a11e7b6", "input_wav_sha256": "cf4a6b81150ebd0a0b84b53b7d87fb1e642e32cdf06d48a9d92a2b2e5f794ee1", "input_pcm_sha256": "37fbd64d958cdf859fd6b5ec2461b3c8df264344dbe508b2f16f178c45e8db80", "output_wav_sha256": "2648c44824d9b38589fe6875ba5d16a913405df67f5a91236417d409fa295b59", "output_pcm_sha256": "c353214fc28f9c833eba8e1ed9ffea3754125d76e5b47d5407c062683cacd369", "result_sha256": "3a7a82f6dcb713ef9e26b26dbd11aa9d1fcf85dd8fc596d95da0dbe2a3ca0408" }
#822web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "405d2679883f6d84a7879a3e11434a9a59394fde9618627ad16df9e22e7fc4fa", "started_sha256": "0d1844ccc900348022e2198e06c2d8eb182bb55cac2d704c92117fe981c6513f", "original_grade_sha256": "95259b39ea7caafc47b06eb1cf578296325ac0e546b30edc3b27c0731406c540", "input_original_sha256": "405d2679883f6d84a7879a3e11434a9a59394fde9618627ad16df9e22e7fc4fa", "input_wav_sha256": "3659b9272a2ddbc2688cc0b1db754c744cd448153516eec7ef444685d3a35fac", "input_pcm_sha256": "0d6f4a2a061b0192fafd1d2a1b70347862f53d2dbdc9b9e90668cc06a06b1d17", "output_wav_sha256": "e353957262771d05b73c83942602b7ff44b043eaa8d60a53bcd1c972f089cc93", "output_pcm_sha256": "09b1d656bbe08395abb2242cb61c10e71b7684da51829240e86cacd48442076b", "result_sha256": "9ec56d7f5f7fb327f06e091bd76a9a838f34ab6e2c3812fd315d6186fc5906aa" }
#95formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "008d66d4e5ff47934fd5a720c1fabac4126afe53237ba63bcf76612a1acb44b2", "started_sha256": "f33a3b72dff5def0dd873642a55965e7cb1c26e762600d48ae488bc904bac623", "original_grade_sha256": "a6c1e8801b937996e7e5f6ffaaed6a936aa47aa8a737b84ac952e929025b72e1", "input_original_sha256": "008d66d4e5ff47934fd5a720c1fabac4126afe53237ba63bcf76612a1acb44b2", "input_wav_sha256": "1d3a158f0c18be974dd9bcf3284005ef45fb3ee5b97aff13d00ae0bdb2d0988e", "input_pcm_sha256": "5e69fc39e73737dfe0094743441172b90aaf3e0eef05cbfe72453f50718f2688", "output_wav_sha256": "9a8af899627fe012cc0c4f4ec191c6c2008dec2fef7b14be18f1d4c4a18fcfd1", "output_pcm_sha256": "34f5c89c20270c39c2f1cc015b955e5bcc944e2276671038b5a057d7a7be0d42", "result_sha256": "fae755492f9893bddc2f78bf0e48bc834acf38990c0164c64a178c541684ffd1" }
#272navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "1ee5b8ee86d7d56dc2ccfb61a006e3564a932ab8174ba478a80eec8a5ade8003", "started_sha256": "f31c43d01a91afc203e5b3b594dd206c757c3f07a0caa36d322168a537949224", "original_grade_sha256": "a36119b7b0e4c8afd3ed2f7ebdbfc881d553f923d52db795facd5b98c273268a", "input_original_sha256": "1ee5b8ee86d7d56dc2ccfb61a006e3564a932ab8174ba478a80eec8a5ade8003", "input_wav_sha256": "cb0e3ccd1ba67409c01738e7b5bec85504b5cda9c49637d0c83ee1e50c235bf6", "input_pcm_sha256": "3ebe35b86e99911e842d7fd505d473407d77e200f7b90a5b76cc4809b2dbf8b5", "output_wav_sha256": "07471d3892bfec096ed62325433b4bac4ecc1ef805ce2a47f05799822244d92c", "output_pcm_sha256": "897e8fe875cd8705aab88c159a398a86e17d378a32233229404695dcddd229d2", "result_sha256": "c9c4bb4a6d5145497b44ad4a82ee4aeffd28cb582b4adeddf342f9cc8cf376f0" }
#636object_counting heldout	8	That answer is 8.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "9b3b607f672df06b8a8c4173cb1082a5f7f24276fecd166055783aa1831a0be3", "started_sha256": "1d47019eb2b9cc84fd038b0b85d6f6be18df114c2eb8a84065b2a75028259b86", "original_grade_sha256": "52066317e470f3294b3db05fae74ba45d7411491fd44bbbe766b43f564e43cca", "input_original_sha256": "9b3b607f672df06b8a8c4173cb1082a5f7f24276fecd166055783aa1831a0be3", "input_wav_sha256": "e3a1bcdf4f55cc44b9ce7009ffa5e0610a3f28470d9c5db6d459f75181cc4fa0", "input_pcm_sha256": "adfca677b76dbe9ae83e292d2c44d4103e41ccdaa063df1ad97ee35f07ff452c", "output_wav_sha256": "cecac53fb6f2d6edd5084ab13c330dcbd93861acea5967045e4ff2cc08bdfecf", "output_pcm_sha256": "7fe425aba6f633ff503380a78a54567b0993d1129889e6d3ebb8f2fd9f78ef07", "result_sha256": "0d2fbb187409e1ff5fc451a72e793b2e7cb072b517788f7ed87c200ae7fd3645" }
#976web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "8a93b04d433c36ff9cec49bfac1a78e82d28d0e5716e7484dab70ad97d3aacd8", "started_sha256": "5d652e9dac5cb76355f0b71923489e150d42a9f6728475628619dfda6e03cef3", "original_grade_sha256": "27a96521cc61316718abf00ab4c97bd2858dffc50c8f60ed78191861ac699cc8", "input_original_sha256": "8a93b04d433c36ff9cec49bfac1a78e82d28d0e5716e7484dab70ad97d3aacd8", "input_wav_sha256": "c2852671c6aea7da1aa4d97ed4bf4a82a6b5208faf841982511e5fc0290a7594", "input_pcm_sha256": "36962170d6e3e8d7db51d69266eabdaa7332f885d2007ed3babbe056c2f19795", "output_wav_sha256": "09343d47e7e8813406e5b916afac2e6fbff830f3fd0f914ac24359758d3d0f77", "output_pcm_sha256": "471e87121ca2c08e4d0993d5697873a4e470e6fe1a90602a70158d9494b1130e", "result_sha256": "dd0188cd39595cd2e7f55c1310482d48db19ec410b856f8bbf9dd2bdd1bf9673" }
#5formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "55e7e8a0fad4cbfda93e881e5614d34214530f1794280fa319e48730dfb70655", "started_sha256": "adf61d557ac0a30b3a0c846a30620afb5f7fe3e22dc0da8664c7039ebdcb323e", "original_grade_sha256": "f4d3fa92a341ec5e8634cd3ccaba8dd3cba027a9909d01db0bbe95f418a0820b", "input_original_sha256": "55e7e8a0fad4cbfda93e881e5614d34214530f1794280fa319e48730dfb70655", "input_wav_sha256": "8a85b2a15e6d9a87d664b45147ad9a92cf471f8bc92564297a3a2b2f4fc0f32a", "input_pcm_sha256": "61480033889122f555eea1353bdee06038dcec9535f8ee3b44d04e9bd3640b3d", "output_wav_sha256": "19f6bbba4aae8f1d63669da816890d2a5cf816ff44236f987887a5dfcdfcd6dc", "output_pcm_sha256": "2dfd850b854c6bccd6ef66ab8f8a813b494ac1991da04273760c8dbb973325a8", "result_sha256": "e8f8b1099c08a48520869e6ac024cb4b4d96422da6f36175057ac8303240be7a" }
#434navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "c25a764a9ed86701fbf70147a08dbd347d83f8a7f8bfa1c7452a581f51807101", "started_sha256": "f6dac73f71e6b66d96df5f52ce3768f375e6469ae8c993fd2fb942f3353b5112", "original_grade_sha256": "49beb5a78d2c62d5ca5c3876117041b9b7e3c48b2561914c2915d0956f729f2e", "input_original_sha256": "c25a764a9ed86701fbf70147a08dbd347d83f8a7f8bfa1c7452a581f51807101", "input_wav_sha256": "a799f8bd9d3771f5dc69cf72fa2853fbfec70d2bdf10ff635112c51e05c6c789", "input_pcm_sha256": "3ba312b9cd8f229a555eff79c69b1161325da5cb55a2b1472e5c35d5364a460b", "output_wav_sha256": "c001774553e500f2331a9b4c53cdc709d0a1519c5ab965fdc800de4bdc5ecf8e", "output_pcm_sha256": "126274267d0f9c908695bfc262aba915a6ddfe8d84c150e389cb06101d9c4bff", "result_sha256": "998e9f651ff45c704d41b0f741f44c3e0f954a3b4951baed745dba4c6f34a171" }
#702object_counting heldout	5	The answer is 5.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "51008f06e1d99f2d5c41525c871dadfb3521f3f729ff11e5facda9827e66a0b5", "started_sha256": "e6be9a6a368eb082466ed681d8007fac947d79c49cef3e33171f8bf802527e3b", "original_grade_sha256": "78ef2576925147a48cc5b0d42fc56af116ec47f0b1da696177bbe1a0c35636cc", "input_original_sha256": "51008f06e1d99f2d5c41525c871dadfb3521f3f729ff11e5facda9827e66a0b5", "input_wav_sha256": "057fbafa44ed5366d53bf3561a5e30af8260b7b3154d96158fa03a27d478b801", "input_pcm_sha256": "a4b98f05f245a2afa56321fbff899a0aee65b7a3fa66eaf376f73012cbb2fba9", "output_wav_sha256": "9e0acba2958dae0703c0a4604b15822848775a1c9e53549f728d48dc6d852e57", "output_pcm_sha256": "08dd61742568b056ab5aa28235758e226056842d7c61569377fb336c1d8cf8ec", "result_sha256": "c7deabd4e043911489cd38f11f2e0338eb1e567084c9842ac6cf20f0e722431d" }
#934web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "a34e628095f66a541ba9d6ace2d7fb3c58a607b0538424c764b3984315f41906", "started_sha256": "6a573c474653aecdae2100ddb447fa3ef8459061da6ca4014a8a63a012abcd4d", "original_grade_sha256": "52bfdf31711e16142578f9d618c26cebc7917dff8ccef0b6c7bb28b81ded5c50", "input_original_sha256": "a34e628095f66a541ba9d6ace2d7fb3c58a607b0538424c764b3984315f41906", "input_wav_sha256": "e865d78c1362d3b9578b1b12c1d60fedb9fb14fedbb306c829ea8620fd0f424f", "input_pcm_sha256": "34fc19b1ef99362500f1a39240a4f966da67944310bd5aed04c44e24b1f38710", "output_wav_sha256": "7bdea153bb6fa14ca08630c91ee60d3de93645e1b82a47f2cdf53ea38bfc86c1", "output_pcm_sha256": "3924b1c813f7cd0e7d81bf46d4cf71d99c148d471c8771cbe31b283a17c3a9aa", "result_sha256": "557c8fe138b710dbf965fb4e929fa39f6bdd32d6086b0fd14fb8025a95e209d4" }
#65formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "475991191b2b598a4745fb88dc3b0f163481be67caac02800d7965a6ac3b47d2", "started_sha256": "be50ac0b548cd7877d67ecf3cfd48da2e748500ccc34635a918c0fbf713bb094", "original_grade_sha256": "239e2ea89dee21ee51e2295db4eb4978ea6ead2db000e1f49b172e3ca8dd7862", "input_original_sha256": "475991191b2b598a4745fb88dc3b0f163481be67caac02800d7965a6ac3b47d2", "input_wav_sha256": "76112bce90cc885acbc2bfc0f19114603ed70d8e8bb35a4af22c01acdde8c58c", "input_pcm_sha256": "04341ac1a848a0911a7c36719ade57c33056594f48a39c5cf1aef3ec8a69c935", "output_wav_sha256": "f742b80aeede9d9731ae3cc8e64fab58c413e18d6d019ad4821bb4e0894937dd", "output_pcm_sha256": "084a395139ac54d4c33109ac644430eb45f3c59468c648e84ff1140bc6da220f", "result_sha256": "485738b987a0e1257a0f9990544a91af287aae9363994bb99ab0e89677035c4d" }
#393navigate heldout	No	Did answer is no.	needs reviewnot_an_unambiguous_short_answer	correctresult	Additive review The independently recognized received audio reads 'Did answer is no.'; the model's reply was 'The answer is no.'. The substitution is confined to the fixed lead-in phrase ("The answer is"), which the pinned small.en output recognizer renders as a similar-sounding opener; the answer token itself is intact and equals the pinned key 'No'. The scorer withheld an automatic pass only because the spoken form was not the contracted phrase. Same class as the determiner substitutions reviewed on 2026-09-15. This is saved audio/ASR evidence review, not a claim of human listening. DAS test duty independent ASR/evidence review · output_audio_and_ground_truth Evidence hashes { "original_audio_sha256": "e44f29000785783a2c04010b1394abb1c0e582297bb07aa43f02ac55c2cd4172", "started_sha256": "7139e14a5d726c9048216925682d8572493a259ef27b698e90c9ea6a5419606f", "original_grade_sha256": "b0faa71587cda8ea847a5dbdfbca56cc451441990fe6c0b4a1b1d0d5e2ae86fa", "input_original_sha256": "e44f29000785783a2c04010b1394abb1c0e582297bb07aa43f02ac55c2cd4172", "input_wav_sha256": "16a744c69053ab9493313a992bbcfe4e1948bb4d90fd08074ab8a38aba48b3a6", "input_pcm_sha256": "476313d745630228a04b4981552bb6784e3cced92b256c7902e7796e4864e79c", "output_wav_sha256": "e66528f628e89ab2fc18f535e521d0cd7c382f642a0e3ff20d7e5dcab9c7b901", "output_pcm_sha256": "5a07fdf88dca63d113af0f1fe6d951acbd57d5e3e085908ded06f11bb607f373", "result_sha256": "41907f23d70d96bd37c9a3c58309af40902e8ac4eda98a92bf828827ee8c3508" }
#723object_counting heldout	3	That answer is 3.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "a4b0b6500856c6bf6ba81e9ba1c66cb5a7c37c9c356d971a6882bbc46b861387", "started_sha256": "41f321c24abbb2ed21fd788b95843c34679e5bf7fb74aba22b7499ccbc4f1ca2", "original_grade_sha256": "372712f40f676ca30c280e70a044dfc45aad012de29ea8797292998f34a05814", "input_original_sha256": "a4b0b6500856c6bf6ba81e9ba1c66cb5a7c37c9c356d971a6882bbc46b861387", "input_wav_sha256": "3e5e6ec2fe984624f667cdabaa73aa2fe455ed55a7edea7a56f6e1fba831cd9f", "input_pcm_sha256": "e68a8984d42478cc78498b899df4e2cd290b1f45f390d5140ab770d856306e08", "output_wav_sha256": "9160634efb2f8670b203ed188edf3e93e55d146cb25284d178dad3d82bb97e25", "output_pcm_sha256": "4ea4cc50739974333f768c604aa71b55c18be749769b5106ef4c1857b8291e8e", "result_sha256": "a332a2180bcae405d56e9e87a2a8ce4ba7800c467eb5e47e0b47a0c4a96f7168" }
#930web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f179b647a1dc05b02d5d00e797e9e0efac3bd7cf43af6771d36290bdd0b70b48", "started_sha256": "ad89a4477fe61b91cd7e3ac3be64ac7247b74fd6ccb4c65325240b9d7f84e25b", "original_grade_sha256": "bc511056edcb3ae13fc03284892496641cca33bfc490165bbaeea84333f98cc5", "input_original_sha256": "f179b647a1dc05b02d5d00e797e9e0efac3bd7cf43af6771d36290bdd0b70b48", "input_wav_sha256": "2856d2a59aaf1b2b96d56f57ebe65cd6a95e1dcafaa40f5957639bc27305e0cc", "input_pcm_sha256": "3b309babd060ec9b1c9b47616739656ac5ffe198e15796d0c3c295420cc8ea56", "output_wav_sha256": "d2c28112b26a7db93da93481bed0353507f35f012bc0d6b022c7d24bec3a75d5", "output_pcm_sha256": "e5d2ca0fb65c9b7f910958fbb81e461b903472392e39f73b7fe645735c4de87a", "result_sha256": "e75463440ed76bb16044dbce73287fb23ba6dc0e95376c0ae2ee38510de36393" }
#59formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "8d97b70c571d062bd79e7009761ccd23bb2dc00840a030862a678abdd87a2f11", "started_sha256": "529576d9f33fbc9447bb4891dc61fec16e509e64a8a619e6ec60d338f2a258b8", "original_grade_sha256": "c7a7952e8c4f77d0ccc281d049da2845804f637f97d7214b9eebac036b404da7", "input_original_sha256": "8d97b70c571d062bd79e7009761ccd23bb2dc00840a030862a678abdd87a2f11", "input_wav_sha256": "e8bc3652fa95f1652fdda5d763093c436ee4f6b2a7c1a98a8df0a8fcf5b12d02", "input_pcm_sha256": "06477e2fe3dafa2b7c756efaca1b413e83005d4c95324cdffba19f7caffdaf51", "output_wav_sha256": "6eeb54ba95e3ff5fbde93a3704f16194bdbe10ffa52c5f17b19932229717457b", "output_pcm_sha256": "9673b60c89906333fe9c100eb217b54325eed4ba0ecd5b07e680fd847db2bc2a", "result_sha256": "d83baa02463f6cfbb02da4031e408709e4371c4a78a9d2eeeacc87e47a9f711a" }
#383navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "1f9c699105ae92a57cc671aae462c503742c70bc158d47693adc75a6809c1f8c", "started_sha256": "ba9747c5322a02bde92881b0c2a66b2bb90ff3fc1c4969c6f242b4461b87ceeb", "original_grade_sha256": "474c62aa7ccc6f96fd64d9d70bdc441018d86968e92f72290fff604b67951a4e", "input_original_sha256": "1f9c699105ae92a57cc671aae462c503742c70bc158d47693adc75a6809c1f8c", "input_wav_sha256": "b739b6fab15044afdc2a9df4622c0ef667fae4014e0313ac04fac9c92640ed90", "input_pcm_sha256": "5d3c9487fc84e1ee6c58beb2d86032d6e0551edfc16d5eb00cf1f6ae149c1bab", "output_wav_sha256": "929fe7285db11719de596778e1bb335c3c9df7ebf510634d3f94d12a6adba3be", "output_pcm_sha256": "b5320e59f3f44cf272bafac8ba1c412fb97613f4d84a74c2434c3a40c94713a7", "result_sha256": "f98ffeb0e4d409ccbf6e057ac5d98d1ea56cf5661af4d7aef23b0e29b1b269d4" }
#599object_counting heldout	3	The answer is 3.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "5d6d5f25a3c3d7a094e849a04879efb4453e4bf463ea3b478d5f232d68060f07", "started_sha256": "dbc6dcbf4eb316651224f63e866f0b4ee91329fa1e410f3c75408fa24aefec50", "original_grade_sha256": "ff50d01c128a80e376cbcdc9fa7cf2110784a071ed126a49dba64403bb3b17d5", "input_original_sha256": "5d6d5f25a3c3d7a094e849a04879efb4453e4bf463ea3b478d5f232d68060f07", "input_wav_sha256": "f5a6456829c319bcc77fe6ef797f8d289e7adec8505ab0e82b25e032a248edc2", "input_pcm_sha256": "3b679958a3d24faa0d7f4b7e4f5a26a0c2c3323e2b22641f9f6bd772eac767f6", "output_wav_sha256": "d5754504b9992e27413bf94539fbbd58af745511bc8db83925f0977790580b37", "output_pcm_sha256": "02fa52733d1e4705e6161e465aa471be37e9ec792246d9d5b54a0abb33888f09", "result_sha256": "0828d1defb14ce8ad69fabc6eff60ccdeb4c6da97899c400a73a97e2e066047d" }
#875web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "6a077dbda0066305488468fbe421fa101c32819a9ab03815568f5eab88557ee9", "started_sha256": "2118af89ad6a2a1490213ca54ecfd842b64ea239ccd675f4a9d11afcad44a1ff", "original_grade_sha256": "50514ccfc9df06af2047b32ce05c3366dcf5e18ad19bbdfc2bdd44de752896df", "input_original_sha256": "6a077dbda0066305488468fbe421fa101c32819a9ab03815568f5eab88557ee9", "input_wav_sha256": "417115d1c996ba1e9dd480948ff5d9515bc54ff8153756c7fd5b05a72dc184ac", "input_pcm_sha256": "ccde065964d23f846aaf3e853c42719f21dfb6ef7c0a4b9d50ec595b3c794cde", "output_wav_sha256": "6eb2d8f6bbe7b8975b5c585ef53dd66cca0b2ad4fdd33f96d225f65711a185e7", "output_pcm_sha256": "7f1bc112f55a7f65209d332faa7aa238c54bb679c966cfa60bd29f3944cce7b6", "result_sha256": "b9de60343490c6e6e534fa49787a794a6cd7b37585e3d07fef69b5c313dfbefa" }
#62formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "bb8c3df2fa5d89efa4a08baee3661e70b6f1785eec982b050eab054138a3d6a0", "started_sha256": "5539579404904cd3ce3e45d8207a4246552e580aa9a416a1a66cf162e1b4fc42", "original_grade_sha256": "87aac7f40e8b219b9ad4a76c17b39a061df82748b5954d77fc585ff55b141cbd", "input_original_sha256": "bb8c3df2fa5d89efa4a08baee3661e70b6f1785eec982b050eab054138a3d6a0", "input_wav_sha256": "98f4f918b87dede200e006dfe5cd697673e31f0be54c6ec9787858f1d81f72ab", "input_pcm_sha256": "6663e5ccafa49a6cd47d99aa1a67336469175a6548688c06339032d7d782d87d", "output_wav_sha256": "a57084c14fcd7bd883032317c38721966fc5f904b5ca682fc36b20b88c713a8b", "output_pcm_sha256": "9c30d3128f68e12accf591347b9eca1441e720480ac7a057719b5d3926eaaa64", "result_sha256": "bd36794636df9de4f7ea16731fce8fcefadfd4bf7e3b877fedbc3e350b6e9498" }
#400navigate heldout	No	That answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "9f7e3baf541ab5daa3e2ebab045690f068e5911693f49713c5336814c6ee862c", "started_sha256": "acb1dd6c389342a60ff777c36460a3cba51b0b2786ce6b9ef887d533409b2a9f", "original_grade_sha256": "73c9609b13e630230a27daf7d4cc24053150eedac805054d8b7767cecc3a9a65", "input_original_sha256": "9f7e3baf541ab5daa3e2ebab045690f068e5911693f49713c5336814c6ee862c", "input_wav_sha256": "839bbc4cee80c590b4d5cf5d7f8b39b99dd09cc59651588b3c8134dd5eadfb22", "input_pcm_sha256": "ff77acd4e233f04145be1c9928dd9922a668557ae0ade0b934ee38919ad74d77", "output_wav_sha256": "2baab689cadefdac3a5546e0f586fd9e9e98503799e4667e7ec092a77d0373ac", "output_pcm_sha256": "55f43847833f6a3ed0b9043ad5af024ffdd2297ea3904316e0f3207fc4979282", "result_sha256": "f5615606840622fb36499d6a8202039de57b4096cb0bbc4d9ba504b2a63328c2" }
#677object_counting heldout	10	The answer is 10.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "8803d2223bd2e4878e759255ce108ac603f3d2ad507bdd6fa1cf06b58fae817f", "started_sha256": "d5a80c2cc0b303a8c7321bd6eb926bee7a6d4d896d2b907f69fe0e8f4287f427", "original_grade_sha256": "18a317652c8dde7fcda02441181e74f96d239ea3c868770e47c19ce5f072cb2a", "input_original_sha256": "8803d2223bd2e4878e759255ce108ac603f3d2ad507bdd6fa1cf06b58fae817f", "input_wav_sha256": "ddffd2a8f6306799e3af4ef85dc4a52417a1240577dc1c60d399e2fae0852b11", "input_pcm_sha256": "f2abf79a5f2f5d0d78c9e0199a6ad18aa8bfb18e657876929260e9ea8c0d05fb", "output_wav_sha256": "bbabb5b4bd2cc5076008509b2d6d3b001ecac77d46dfa92faa22605202f7b061", "output_pcm_sha256": "d408808071814397e9cd942ed8f32f5392d81f5f30b75643bbe0779a515fa72a", "result_sha256": "793ea973c61b6176da98e658bc69c08af68da1f5cc760a8244eaacd5251d1594" }
#834web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "64b23ef7773a620f5f1ef51b01d5f8f0b33c8930c1bed568dae8a80e6f2b0a99", "started_sha256": "95a64375a0f0f47c12fe2ebd6c6919becb223a08708102c38122914453e6fce2", "original_grade_sha256": "9f847423772f865ffd03cb1dd3d03bb887dbdcbdf59eee69d4ce74817cef4c9c", "input_original_sha256": "64b23ef7773a620f5f1ef51b01d5f8f0b33c8930c1bed568dae8a80e6f2b0a99", "input_wav_sha256": "0ced84685b4f1aac88d665eed153780d0f82d6d1487c1ec3b64b351395f08b29", "input_pcm_sha256": "4e039dd37776a72c4edbdc055775a17cc9dc8b18a8b85de96bb05fa5660b2dc8", "output_wav_sha256": "603591ab6a75318f1f298fe419e76bde70ea17d5588051209a68f81e0ffae06a", "output_pcm_sha256": "b659d91f98966cbe6c060ef3f67cefe2a3ae760145f0cf0ce1061de23a34847e", "result_sha256": "528f6f6308a7186b7a3c1fa725fe4032eb3939cf4a77bf8358b86fd0d92aa6b1" }
#74formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "6fb7b1c7f15ceb954b7e8e0a4729ba35e354f6e5848e141e325ca4ce7c0161e8", "started_sha256": "1b2d0ffae22cbb3d8f1f51bd01fb2d6cf1b711db717d30da419db02acac4a22a", "original_grade_sha256": "bc9a064d25f5df7c5d2425fd1c51afd9c5ad085eb23c3e688cc51130941e4a3a", "input_original_sha256": "6fb7b1c7f15ceb954b7e8e0a4729ba35e354f6e5848e141e325ca4ce7c0161e8", "input_wav_sha256": "e3acbe527e6cfbbdd48bf3518086df177ce94d22959ccee939386035ffd1bdf9", "input_pcm_sha256": "dc5598fd4be5c5bf457d393e1295dc98734fbe24c2fa8da74100f2d1d3d3b6b3", "output_wav_sha256": "67eca9a67685c47e2ecd2f4f1aefc8dab60d1baa0fe51a1b759299d97599ca38", "output_pcm_sha256": "68d1be17c8ffe0715c690316bf8395d3415a018afdab4f71c465f4931c589360", "result_sha256": "be53f7c6b2ef4b3bccb612e3c99dca36138031a3c0c3ef0f1f860a02956da30c" }
#458navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "a0731f5acc56317f021e9f4061eae5e7be3cd67b47ccffa0e4ded7c37633686b", "started_sha256": "6e3f625123cc0ffd35c4873fc0f7cb0f8d25c09bca020ffb3c4036d92f25a9af", "original_grade_sha256": "86ed8ee197a32bd4f458b7d3232d73246fdf23ad20c8e0f039b66c4239e112df", "input_original_sha256": "a0731f5acc56317f021e9f4061eae5e7be3cd67b47ccffa0e4ded7c37633686b", "input_wav_sha256": "ebb9fdba246e682b800fdc207b79a398fb1808d1b12ea49fdc3f616a9c7cb689", "input_pcm_sha256": "7f56d2bfe2350a888347154eaf66726c9e5e738f53c9f675f910d3810d344d51", "output_wav_sha256": "25e78460fd974d22044e0976d23294d89667cf29465d20dfbe17885343da321d", "output_pcm_sha256": "cf0164a8f29dacb5aa478b2bac430fd86bdd3e1d0cba523d539e0e22acd0671b", "result_sha256": "ea27e49a1567c3b73c3a1e6401b4c60fddd185d1dc1e00cd79a170a89e6a0b9c" }
#637object_counting heldout	6	The answer is 6.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "6ca14edc328d2d55ae10f1d4dc52397c93e87160a0682b89040556ca49fde86f", "started_sha256": "da1af7ef4d0059c689962b0072911be633ac3d5742e7deeeec88a92d2a793986", "original_grade_sha256": "dce17d6255df277a0fc12ef70f7d1c4ff9f250b68cd4357c6ffa725b7b913254", "input_original_sha256": "6ca14edc328d2d55ae10f1d4dc52397c93e87160a0682b89040556ca49fde86f", "input_wav_sha256": "30f61db7c0a5fad4d755b27eeadc8819295c55d9d1648ce282c0c82b107d9afb", "input_pcm_sha256": "7a5a18928e428807d5af9f8608ce649b6fd272a721e17a6e1918e744e2775d94", "output_wav_sha256": "12772f2b6f0cc387d3c22fc1afe9aae7b63214f2da8c14a02c714fb1ae235426", "output_pcm_sha256": "44dcb7074644780a78bc128515b80cd06abcd85061b98e25481d17dae407843f", "result_sha256": "15ab52c4af1e56ee9fa48cc38086efec83a07628a2870e994ba468a5d76c0fe8" }
#831web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "2743b643c3a96a4243764dc83c184f6aa995b9325b323350220086a99234e8a5", "started_sha256": "30de31c9f34b598be997f959ca33b52bea3a3c95a43cece5fd89a3435126b0bc", "original_grade_sha256": "819350fe5f834759778a01809ee172d05113c94aef036fc6315515d67f74c2d6", "input_original_sha256": "2743b643c3a96a4243764dc83c184f6aa995b9325b323350220086a99234e8a5", "input_wav_sha256": "cd2eae506c228a15b41fd44bbeb2c8a6a7229a3d73d8a7790b09606638e6a9e5", "input_pcm_sha256": "c9e5f40fa64e8c76110101e7c8400b96103ed1cfd0ad70b1f684667f5bc2fc5b", "output_wav_sha256": "c474e79702fbcda501e2a6f034d7b41bc97503db2a22f90546fbd1a379e70e70", "output_pcm_sha256": "8366daf74ef71ca0dc1539f0b273127fb06af125a29bdfb492c7ec45652efaa3", "result_sha256": "f5e323da536b471b6014ebea20240041b7b156bc77a9155dc7f730a6c06c2752" }
#227formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "916cf5f07d914dbfa76705b761e5e7b4832ee7cdf4cdb1698f728f8b41fc2d16", "started_sha256": "85d3fd5ec88a4743003f08aec8e65720e6ca72d9167a07844f1a5f9a908ce152", "original_grade_sha256": "8cca96ca777c71899c734d345e7ba8b424ff3c6138f4e48303aaedfc6df3334b", "input_original_sha256": "916cf5f07d914dbfa76705b761e5e7b4832ee7cdf4cdb1698f728f8b41fc2d16", "input_wav_sha256": "cd29caf5c5cf13a392204b2c7a2c20db45d0c23409ef3d32cdbc563140e29a10", "input_pcm_sha256": "bcbb63b2bba08981f795e9ef964a8dc332fe124b36f91a3912e89a51d8c494c5", "output_wav_sha256": "49aeac3fefa339604579dab7978fdc6750eca26ed948aa8f6b858cb3a8e41727", "output_pcm_sha256": "063b0273480b89ba52bf97e9dee52bce9e5e259d0f36f95e7eb7b0c4dfb58a03", "result_sha256": "6545e09f0d48c8c8b2e0bc66ac39dccd52ef29d21795a27db191c6171d5b4192" }
#389navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b4d09e4953af1028e26d13639027713b89e4f6f1b0ba63569934f44be8ff5c2e", "started_sha256": "d27d9988adc9ca735ee5be59781a372c81c7c0fc7173bfd84408938e7f3b838c", "original_grade_sha256": "1349149fd3e92e632ba8b9603fa0eeab4224b1ca02238c894d9fe9d68677cf2e", "input_original_sha256": "b4d09e4953af1028e26d13639027713b89e4f6f1b0ba63569934f44be8ff5c2e", "input_wav_sha256": "13566b49ca510f0f05b90717e305ecc85a1658aec21fb5c5d8958293e0e4c3ec", "input_pcm_sha256": "4b29f931e91a7e746ea5bf3ee1fe467ba3df2bb37d461ebc0f05bb03aa4bc1be", "output_wav_sha256": "c3495701243e48d819d4a48b1fca168e5be4b4ecd1b20030aebb474e9ad94dff", "output_pcm_sha256": "c217cdc731bfee79ca9ee807ff91836731b906b1447414ddef60df7eb97535ba", "result_sha256": "3552fd0803d0d84eff7092d1265dbfc76d89311cf296fdec75377dd5bbeea760" }
#576object_counting heldout	7	The answer is 7.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "e32c730e24d1b6876324b3bace775e4a9c71a7f060176927c117fcbd151bcc0e", "started_sha256": "1a58038f40cbbadd39bd85a7f0b7ef9d0a7e5575018dc60603a6260aacbbdbe0", "original_grade_sha256": "6a8e9d62cfac6c4fd7d0a62fa6968bc7d539108df76840add9d174ccdbb42cbc", "input_original_sha256": "e32c730e24d1b6876324b3bace775e4a9c71a7f060176927c117fcbd151bcc0e", "input_wav_sha256": "bcddcd78c64f20506c006cf86b736b795fa84b3b59c532589bc86464a0005bb5", "input_pcm_sha256": "eea92e45d62e09a2f3260612dafc419b3d53923f5bc0cef33062d6ac306e40ba", "output_wav_sha256": "3fc9f96e177958982921f4549991da137b119a1f1ca6fcf937c643981b018623", "output_pcm_sha256": "5de8b8e60f5305e1b2b08ef83b7f214522964d54db0303b2c99bedebdcea0e5a", "result_sha256": "63587f901fa198f42fb18a220a6f2b8753c81f03e4a5a1dac7434f6ea5b513ba" }
#907web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "defedf3779cd2ef050d9f9a48d648757d255fad91e1763c462bc0299740c453d", "started_sha256": "a727f04933f3bfd39c66ef885df90c5aacc3e000d7c9766b99dbd7e11747d2ac", "original_grade_sha256": "811e3087cb9da2813caca3557666c260417d640378dda0fec3e76a493ae943d3", "input_original_sha256": "defedf3779cd2ef050d9f9a48d648757d255fad91e1763c462bc0299740c453d", "input_wav_sha256": "bbb9e8da0d18ce34bc7fa4c2be1c8c3b44f4c015cf7c9cb558421b62fbca752f", "input_pcm_sha256": "dc389e17a14e14632b9ccddb1ea2b65d509133fb138de6b858e2841109846bd0", "output_wav_sha256": "e1be17c40e9a7d31c285bbc043b56900c4353e04487fbe51e0ccb6785b469869", "output_pcm_sha256": "a142d8e6b09345d02e8a04ee22bce621e8e89358d395eff3f25e43ca31462367", "result_sha256": "664868912a354cbfff6d923d3ac9cb242418bc856dae01af8ea4e1bc452b9fef" }
#7formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "ea54839b1f2b6c712ab1fd3093628ac9ed54f9fb8b03fad25eaa5874d0559afd", "started_sha256": "d7fe4bcd1df286822ecb841ccebfcb14b945596798018884994728ca92fbd12a", "original_grade_sha256": "d8b916e5ab068a8fa9467d00860f001e254c3a9cebbb0e8483c88f08b83f8ba2", "input_original_sha256": "ea54839b1f2b6c712ab1fd3093628ac9ed54f9fb8b03fad25eaa5874d0559afd", "input_wav_sha256": "311030810168425f3dd7f66e1ed128d4998133ab6934c3b55d876026ed566dcd", "input_pcm_sha256": "2e9ce5dfbd639d0b36360d4fc2b549ce658e28447faaa176b678f5c04963642a", "output_wav_sha256": "7beea708d6da5c59546c92ed6b510bc7772b7b0f0bdedf154abf47ad0f43ce50", "output_pcm_sha256": "6309977b9d7401eea2ad369bb4e87c3beb133cabfbcb287b9826adec707bdc03", "result_sha256": "7865b7ab7cefbb3ada5fcb21e286974226b4688a163e1c5fc3c981f2742592a2" }
#267navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "03cb581d2d0ace16b4667fe6a1f20dcab658a96d36676da906aa787a6d92a202", "started_sha256": "b8977f77db61f40bc4b087cb015baa2795855bbf07372a1f055b73bebb4a3d5a", "original_grade_sha256": "50b2ad10576f69b9dd460a4dbc19d75fab7e853966c4e371eb342b25c6b59ae4", "input_original_sha256": "03cb581d2d0ace16b4667fe6a1f20dcab658a96d36676da906aa787a6d92a202", "input_wav_sha256": "a052e62b57d5f35653e8cb979db7654fb2b9fe85518a342ca8b5d123737f5c3f", "input_pcm_sha256": "818437b5932b62914561ee2f7c3641806d9d59c8ed7b8eb5f759817d2b4a608a", "output_wav_sha256": "84600a3a615e014c421468a3f3e04052559dfdead49097ea847193397ae8af4f", "output_pcm_sha256": "114e535421c7888019b4cc76dc54f3a0c27822a1b7b3d82fb69bb3555e6b26ec", "result_sha256": "f5b2f89acc6f97badedfe93b50cb484346e508b85e2ab8f936cb158e5670c7c4" }
#673object_counting heldout	10	The answer is 10.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "a99bb61ff914fbf7c2deff9c6f1778e4544c4f774196f946ec38cf1b0f013469", "started_sha256": "7ff7049ce130d095f6611fae44ba868deaa81d7de494bdee1c8281c11e3cfe28", "original_grade_sha256": "deef39b3494b8782f0b844d4a28fe1b85e330acc1b8167ee8910430163b883ec", "input_original_sha256": "a99bb61ff914fbf7c2deff9c6f1778e4544c4f774196f946ec38cf1b0f013469", "input_wav_sha256": "44265880c5a941ff842b729bbe11082a2f7264cd7459f2121566548ee59c1d84", "input_pcm_sha256": "81f62c42f77ec03ee2ea7283439a57b9eca4c0cdc25a2cde43d52f12ce35df98", "output_wav_sha256": "3d5a4740115c9427e963d7734f10d74eaec6b61462681a6f9f14d1555c87dd52", "output_pcm_sha256": "799e2d078a93c0988dd51ba500fb716ad20616d5c93aa083146957fb6b5d00c9", "result_sha256": "8ad1f80c3355ad45a13e33b7c32a68f0475522b808bbadbf2cb7113fe73830a4" }
#908web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "65b77321f683845980e5ad3d70731454b7be1f32317357b371652cc566a7b660", "started_sha256": "d00f5fbd4f0e6b0db5bccee5bd4569b78cce09b9025092c8ce88dca9f746cfcb", "original_grade_sha256": "449075b3a8ae6a57edbc382e0826f2370714ad705b1d1b44f3daba9d162a0bc1", "input_original_sha256": "65b77321f683845980e5ad3d70731454b7be1f32317357b371652cc566a7b660", "input_wav_sha256": "37a5c6299078d9cbba9bff0c429db77ae89e94ec0997653a4b2f23ffec29e75f", "input_pcm_sha256": "a7b341104740a7278fce0594bd3270b1b97ef06888925c94e683e36861cf2efa", "output_wav_sha256": "6c763844f231f7c92e810d367adefc470193f9139dbf5c486fb499781cddf4e7", "output_pcm_sha256": "2ffa72155a6003b2b954e1abfba76c73d77744d73c9cfb7d5cca8528f3f4024b", "result_sha256": "6295bd59c8af971455c9f9b85233a5f2d778e802fbb553ad915cc26a5df3d878" }
#113formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "eec28ed6a36675f115c29a4a8d6455a0023c1b7c8df7204a7e87c8f6f4ee33c9", "started_sha256": "92506f92064f53631577dc688beecd9e1b6b051f24fe52b28043d1f410e7fa8d", "original_grade_sha256": "80c4bde8ff5acc8d6ec9747d2a58e87933afb7d4e4cb13be7b2a584bf253bec2", "input_original_sha256": "eec28ed6a36675f115c29a4a8d6455a0023c1b7c8df7204a7e87c8f6f4ee33c9", "input_wav_sha256": "c62f0eb47b6410cba22cb63d0d3be01f597a604e346bbe3345f9161b1567ad65", "input_pcm_sha256": "568e9a7f88711506ebd1faba01ebd3e669e6f353564abc9de3a10007fb3dcca9", "output_wav_sha256": "a207de1dc1a853315ad1240c819c7b016cd257d317405f7e15bb23d19399c57a", "output_pcm_sha256": "3a1b67d30a45d48f008e7d9bb4ced1f1af6efef182e4d4a3f23f1640a5666c90", "result_sha256": "603283b81faa3dcbe0eaa8940eb8dfbc2fbade129bb7f62ed98980f7cb2b8f65" }
#273navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "099fb036db5978fd7eb0e434e3b43be97d13a1091f459ed0cf1f8c355e0d38af", "started_sha256": "a1f4b5e3700351c2aee3b8e479e6cd8bc641a98530960f5c2515b2307b822b6b", "original_grade_sha256": "1066d0f269714f79255b0454b5ec343c6ba07534f0b51d03a72124f1ed92b8f3", "input_original_sha256": "099fb036db5978fd7eb0e434e3b43be97d13a1091f459ed0cf1f8c355e0d38af", "input_wav_sha256": "3c4b81640e71482aa105115c296060cc3aba02ba78d841752ea1c5833b8b64bf", "input_pcm_sha256": "b0b764ca7f8486ab3cdc9860980208df1bf489bd7cfca61451c85f7888c1be48", "output_wav_sha256": "d899707a2ec8d835a77219a78d00299ca0b80526f54a71af2b26cdac8863fd09", "output_pcm_sha256": "c37889bf8658ee850e06d24636fdbbe9ef5043480e36ebef1fb70d8bb2f63bbb", "result_sha256": "104aa0b3aa2356443e58045333302528cf27df5ec53888a4983f1f7e26f488ae" }
#589object_counting heldout	8	The answer is 8.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "c7d2aba5753ebaf07189d2e42e8085483eb696b5c84de51da88df8706f68faf7", "started_sha256": "40851d81dad9feb1c098da3e9d3101459e2853f2d3b67a9b2d1de08027c94043", "original_grade_sha256": "167aaefd59e55800e15a1ac0d759f7133cf6df17d396b511f33fc5018ca6fdf7", "input_original_sha256": "c7d2aba5753ebaf07189d2e42e8085483eb696b5c84de51da88df8706f68faf7", "input_wav_sha256": "02b7d1c740e863d74f7871020d6fceab7bbcd43e045d2162bc91ab3450598e52", "input_pcm_sha256": "85226840d5423fd5e3ace5d8d98a6626584f5ba836f2ab15192e3adea6f0e277", "output_wav_sha256": "26191081fd043935a82e05a91974f28d7536a6459f337a2ffc278282e371a229", "output_pcm_sha256": "68fc81b695f709326fa8fdd95725458f236f995a58722cad923e4b4bbe28ba22", "result_sha256": "758b2c4ab41c9cf3c2b37e9d5516e1968be820dbf5a40d7bbfa0dd745d0ad009" }
#990web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "354144814ac5900116bfe59fb8b61bc616f67608d35b7a40fb285f283445d4fc", "started_sha256": "f1263a1964f9bfa5e5f1ed2306fd727bbaccbbadff533038b234694cb0d72d38", "original_grade_sha256": "81f72c3416033cd163b1d7d2fc15a070f33b08932f823d78f361f4875899b973", "input_original_sha256": "354144814ac5900116bfe59fb8b61bc616f67608d35b7a40fb285f283445d4fc", "input_wav_sha256": "5fdd99a2ad5ba99b301167f4024e36e2bc367d075df6c16646e66d221bd0e4fb", "input_pcm_sha256": "7fadf9a187631c5e81a2432cd3042c536c35ea16f64e01dcab2d2a9c68509f6b", "output_wav_sha256": "bb1867bcaaaaf8bdbd080f9e79abda165d3880fad7575a6566c79093ec193420", "output_pcm_sha256": "1e42c5d5150c84865e19622cfc155e5dc3d7bc1671c025308486b1cfb3e7b907", "result_sha256": "20bb393e8d853b086c0580d3bb52c725b2114874f50c2508bfb4fbae050d558d" }
#115formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "0b288d57bad1bdab31fd98bb19c726950abfb18d9eb3d4260fed7092d79184ba", "started_sha256": "adf6cd0c5cba1e482da65ad55f77eff12999709c336fd705fe87a84bb35c8d39", "original_grade_sha256": "342125ce0a950ae2eea929e38858e1a671fa9b432bc97ea26d461491c86dac25", "input_original_sha256": "0b288d57bad1bdab31fd98bb19c726950abfb18d9eb3d4260fed7092d79184ba", "input_wav_sha256": "cfaeaf870629483b5d4e15df3f77005a10c6dca3607a7644557017d0b0b8fe13", "input_pcm_sha256": "acfd519a661a5e5755f4fc1ce92207bc779af2b2b4d662e10624ca9dd0939f19", "output_wav_sha256": "498b4eb180b6ba5cef4648325db2f196e63d602247a6296c01da1191885c831f", "output_pcm_sha256": "a348157f45c4e7dbcb1359e32ca86c71682f1cd83092f1d21de74110bcb074bf", "result_sha256": "c87310be476d925fef0a72cc125b9cb437ad78084fbe4a8bf7a22586d3b1ad2c" }
#422navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "4a1730bcf97b31aad575bb8bcefd4465414ed229ea6fc84bcf173ac51052d1bf", "started_sha256": "6c44098f5a0c3f636d745ca5a0ea5fa74ac2b7082e97de74f09d926c818a4ddb", "original_grade_sha256": "de7a6be55f70cdba7ea48d67bf223ab4edd7a2a4a2c45ee0dd4aeca1a7cde42b", "input_original_sha256": "4a1730bcf97b31aad575bb8bcefd4465414ed229ea6fc84bcf173ac51052d1bf", "input_wav_sha256": "ec94015359337bc19c47716844e112ebdf99dc3e2346d8809ef6dbc1c46b5f82", "input_pcm_sha256": "d01d8db1daabfa12c43585e4563d6a438a00c843c88ee16fac21de344640194b", "output_wav_sha256": "04fc810b119293d52e79a9338544cff3533efe61e6e6bf2e501be9762bbfbe65", "output_pcm_sha256": "616ea4e3cfaa01ef970accfa4f0f67ac2180ce59fd83223319054078eba6e88f", "result_sha256": "a721808fef5bc47a98e7a6df716ffc0d97c7948b343ee00666193649c7eb277f" }
#621object_counting heldout	5	The answer is 5.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "744308ce72283771cab79043eab53a2b27b2772f2ed90a1b51dc4db9d1e44906", "started_sha256": "27c62f8b84034b46d987075e23ee68db0513b144a2d8b270a77352403be85dfb", "original_grade_sha256": "ac1a39a867a078ae2c5f98a1bc279bbe5b2e30da8760fe0ab57ba7b62c90578e", "input_original_sha256": "744308ce72283771cab79043eab53a2b27b2772f2ed90a1b51dc4db9d1e44906", "input_wav_sha256": "d1540ee84c03760e5871cf6e94aac5dbebd0d30587db68a903ecb3982fbd6cc1", "input_pcm_sha256": "6f97072ad7eddcaac3f2d1e21b217f36c0b8168bdbd5c86e3f483903a94f77b3", "output_wav_sha256": "f52d94b8ffcb56cbc592a05245fa2ddfa496e548f854d7500dd3e860f476b5be", "output_pcm_sha256": "317a447630945cfb836b929ced9e2214ee32cd9b5bc6906564eea3451a56483a", "result_sha256": "8ac6af47b7718d39da59cb87c32b27d6e48187f36b7810a0185f2267ef1b158d" }
#792web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "0245f1d08c17701a73b0fd6eb7642e85b216bc7bac0fbe5ddc1ba7314ceda9ce", "started_sha256": "52c77c748ab587666ce5e69c8c50554df38d5ea43158613afbdad56ccb07a7a8", "original_grade_sha256": "38d65a8bd0b8552c22fe93d825a2228290cb40630ab7cb27acf05a855b26a078", "input_original_sha256": "0245f1d08c17701a73b0fd6eb7642e85b216bc7bac0fbe5ddc1ba7314ceda9ce", "input_wav_sha256": "4796c572f50755787eaa14b15887e6c0679824f2138c3b93c6e5299e74182a93", "input_pcm_sha256": "5efd0e365e3042a5aada5084d8a7fab7da88977856e6abf01fecc89c75856b22", "output_wav_sha256": "b6c8508f1d3a77c2a6a9e2ef4679d0d883558e460912478bf88618f0d02b81cd", "output_pcm_sha256": "79d5ef812312177d9f155b7278aef40a7f0b73b673db83c61c70cf284b9196a4", "result_sha256": "74dbb91ab7e97f9d45ea82df99c1262f4a5ee80f1eee4fb3c14d58513d92d524" }
#231formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "1c32a05f1fe0a94fb7776956138c1e942868ce0671d6a786d2f8659ca049d875", "started_sha256": "13086d4eb515f6b7c65aa3b74a3ef5daafd913a27eacf480e1ec31f72e7b464a", "original_grade_sha256": "01fbe22894ead25e14df9f8bba6d0375cd9e9215cfc11f915340ee7c87a7f2f8", "input_original_sha256": "1c32a05f1fe0a94fb7776956138c1e942868ce0671d6a786d2f8659ca049d875", "input_wav_sha256": "c34231dbf71aad38c8baf35daff00d04bb016bea778268f398e9d3796a2bb699", "input_pcm_sha256": "4e995c419502cd5dd089c95b5a23744986cd383b46c092341fb140982bba2dcb", "output_wav_sha256": "3fcfa04a815082ceaff6274ff0c063189397f39cd0d3153f08c5dd8ff3d7d9b2", "output_pcm_sha256": "99a09b6836218b908034b27ab2ad47153831e01d27ff03b86dd187a5d0c65b5a", "result_sha256": "93f727525dd9d45e2ed0ba74b50c66d8f513336de62f2b43412d993d58fc894e" }
#301navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "5de6dea7bbb3055446bffa6c2467acbec73d8d1762d3741aff67f916cbca3bf1", "started_sha256": "25014b38e136b2666fffdd55470a442c872e78848d69592a92806d85acba1d04", "original_grade_sha256": "8eabed24c667e625c5f065a4dd52e6a3098fbca1be91ea11ab234fee2db3e992", "input_original_sha256": "5de6dea7bbb3055446bffa6c2467acbec73d8d1762d3741aff67f916cbca3bf1", "input_wav_sha256": "644bcd9260bc8ab10d70219767a25d55b44e82b11e96c0289be29e3407356d22", "input_pcm_sha256": "ba0cd1bb1b7ad83d11f724e8e7ebe1a35d1a98a6a19f8dbd34620d037bc3ce44", "output_wav_sha256": "084956874eca0a7a5f53cf99397d0710240d2d6d7e7588148b95296cb06bd081", "output_pcm_sha256": "819ad100a25f957f4df48da5a23880c41eda3c14df426b994738c5b45e45c445", "result_sha256": "40fc96e32bbce8143188ac70b06a63263b7fc5db3731ce9b65ac41b8a0940532" }
#622object_counting heldout	10	The answer is 10.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f15760e058b1718d0b3f88cf48e2cce684049c19cbfff70b4176be517dd67f21", "started_sha256": "3c344b5965de5611e1594afa3cfd7da42cba649d3502757a769bfb17091eef75", "original_grade_sha256": "3acf6c5a15a43f2a33c7f248dd1d17383834b496f8268a8552c4badb8f587932", "input_original_sha256": "f15760e058b1718d0b3f88cf48e2cce684049c19cbfff70b4176be517dd67f21", "input_wav_sha256": "b9743958391f3213b352ea4114c2a361d5e3bff15e5c4b62fddc3d36dcd4f9c9", "input_pcm_sha256": "1c31c937cbcb9ab709c3cd14fa3cc856b6d4a973cb47753cee8464441d2dada2", "output_wav_sha256": "27bb0d00a69009fac1b6367766c412cd601061c47dd9e0ee43a2a2d95be1d6f9", "output_pcm_sha256": "5ce8394c13e6517067d0f2e63865c25b3ce1c15ba4dbef7cefde9dbb07949ba3", "result_sha256": "d56f2144d855bc7234ad0a56406a8229fc6407ccaa02ea64df6b636711a3136c" }
#797web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "ce4f8f65e5089f81bbbdab8687c6fd1941665c5fc1820d0ce48de83153adb4f9", "started_sha256": "116f95f64ad2d3e76a0c2d20cede813ba358f77a4e181a16ee74ebc7d9969185", "original_grade_sha256": "89fa5ca9d4fb544828945cfc9e6ac1e4222e536e417426fe7faa0a9e8c0ac543", "input_original_sha256": "ce4f8f65e5089f81bbbdab8687c6fd1941665c5fc1820d0ce48de83153adb4f9", "input_wav_sha256": "b25ea1af78a9b1b8000c778176c72ff99a743da70b11041e87a18e744cd116f4", "input_pcm_sha256": "770875c431a9199fd65d48164ce2babd14fc0d6ed5ccc984daa3ddb03d23b822", "output_wav_sha256": "63f27504e4b44d6bfd0999d845577ad8268768edb844ff080a386c68060f336f", "output_pcm_sha256": "f80a787d12e98e42d55857fb2d5f9151783c087984f862a28a9db1472d87184f", "result_sha256": "3f6e3d929119bdbd6c25639aa7f1f9a42add99ef1228b23201941288209c2e81" }
#124formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "27939f9eb8fd91135cd62cabc2ac16b8e256211194633d5bdc120c6e801d2302", "started_sha256": "b1f4d2f82b1372807b4bb56e8d0f463e54423f1c5811bb09039d128956faec2f", "original_grade_sha256": "edcd27d7192352b03f45bf63666c344f63794eaa4f4dee352a7832146b543198", "input_original_sha256": "27939f9eb8fd91135cd62cabc2ac16b8e256211194633d5bdc120c6e801d2302", "input_wav_sha256": "61774c60522dd118061476500ad9ac5fd5ce25f818eaefcba604567a4e82e625", "input_pcm_sha256": "571e38a7b5d112659c1772acda806cfffb376466ece823d6fda40808638274bc", "output_wav_sha256": "e01d7b8ebc8ca883529ac71a54bcc219571b3025c3e753ed0ec58b303d68ff3c", "output_pcm_sha256": "d7cdc1d8a4cb7a1ca4159a1452d402120be02ef6b9b9818f475700a3c8b12674", "result_sha256": "d986a6b263a296a7b1921a903d17de77413c895c21f1ffa67b41da7eb35843d6" }
#374navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "2aaaaca176f3082f172a973010a8132b941143ce118e0f8cc47d7013e505be70", "started_sha256": "c2c6ade4f8d059c157f44ca194f060894832cdc114edff5b2e427b152e23e43f", "original_grade_sha256": "92412cf4fc193d96acd2c00a8787420d50b25d2511fc690739fc83b5af490cbc", "input_original_sha256": "2aaaaca176f3082f172a973010a8132b941143ce118e0f8cc47d7013e505be70", "input_wav_sha256": "7e4791029827531926d99e00723412d9d6f02d1f1a6d67eca67fc12c9b215f7f", "input_pcm_sha256": "6981976a7c7e7d5a603cf70eb5a02acb2662b5046b9875dfe02e417f844f5760", "output_wav_sha256": "f6eee9a812c92c1961fd6f191dbfc55a17808ac6903bcb2666efb534615e9ea0", "output_pcm_sha256": "a6b82f94fae536092e636919bdb76d2507fbb898a9e90a06762138a4048ca914", "result_sha256": "3a73d5722893d3e327ba1946de9a336ebc9bc3960eba5f79be6d1cbfe3ca8993" }
#596object_counting heldout	5	The answer is 5.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "5cfb01c0316635ffbb56c16240419e76c7ee6a3dbcbd9396ee0944fa1d9577b9", "started_sha256": "b2d858a47eceffd31e0a9aac4dc1e06da9fd0ebb6e35f9c1978553d594842848", "original_grade_sha256": "7714ca51fd6f10e27b891982edc5728a4294cfe88d5a4b33a21163117426ae80", "input_original_sha256": "5cfb01c0316635ffbb56c16240419e76c7ee6a3dbcbd9396ee0944fa1d9577b9", "input_wav_sha256": "fd7907f3866c4841a4a287288d39399352bd5efeda4c3bf0348e97112dcfe212", "input_pcm_sha256": "ae163bc35c787db8cc5583b749f5cd3a291f8b952963f39ce3c35b0d58c83ddb", "output_wav_sha256": "7001e385b1269fd47e91cd59601c21213932ce2b465c73e2b35855ab6c28a75c", "output_pcm_sha256": "81034666523a05e37142bd9cf9135c9f6735062e40cf0393b10a1b66645cf8cf", "result_sha256": "21ef024e469a38df67b9d6e82030db0c4c940375918c37f29ae87e8d32ca4493" }
#998web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "a0f6aec0364fd52987b609db4e32e9fba3fffb6b35ddc5ba4a1eca5965a77c0d", "started_sha256": "216d8d0b3cfe2c69fed339743118c3a2c1a8daad14f76c952fa5383c0aef1088", "original_grade_sha256": "f2f9e4c4e56b99a31712148a6d72032b38fa34ed03ae457cd24bc6b5e01967b6", "input_original_sha256": "a0f6aec0364fd52987b609db4e32e9fba3fffb6b35ddc5ba4a1eca5965a77c0d", "input_wav_sha256": "0f2ddd7f910b8f9713d2c095082e133a089cfded437aca9db3da5be71f49270d", "input_pcm_sha256": "92f96cb45e744dcd501a7a477ebdbf3bdc20fb0917b00e462206adb36bfa6436", "output_wav_sha256": "9b167b28af28e702fa2b3f601bb6e8dc6f1660bba5c6e662aec891ea70cd3018", "output_pcm_sha256": "bda6ed552393ec4a65760b82bd24facdcf1f023d458e4fb0266d838cf9583a18", "result_sha256": "97446906e7a603cce72dc3ed083b7738a05c91f24c90b287652fc951a6447add" }
#8formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "74d4eb3ae0ed98146245cbd09a1230bc3970b337df7c75969a9a3bb3e00bfaec", "started_sha256": "2b5ca393bd4e6bed4944a3d59342b600f4112c341578a336e968689704d6013d", "original_grade_sha256": "bc0fb1fe2789fc0e12c81a71f0cdc273a61c765122b664592bee78528f633a82", "input_original_sha256": "74d4eb3ae0ed98146245cbd09a1230bc3970b337df7c75969a9a3bb3e00bfaec", "input_wav_sha256": "4d174f8b09ed65494bc9e0634f0a51ef8d963a71c21dc658af34ea9c4100a7e4", "input_pcm_sha256": "2701334ae69c4d7e779326f866f2aaa6165578f20dbc24f9b8d6b193e581a5f5", "output_wav_sha256": "0b2f57160fd08588800adb02dcf1085beac1889ef63ccacc048ccd9dd24e709c", "output_pcm_sha256": "91c1a0bf6ac693201441a5da4a8ebd95824bff0593794afd2c40ca91a295f645", "result_sha256": "94709c10440e088caa8ac416c9417d9ae7ee1f4855027b9a79538b9ab1e36acd" }
#483navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "9e5415c0242133c44a1092f7619f2181a21d2340f52d873ba05d2d3bd0b6b13b", "started_sha256": "876fe1327fe526999fc12767941b0722eaa996c38ed1cfbec0be13a7cdb7fca9", "original_grade_sha256": "406a6ba97804d82c5413b4b8d28f925a945ed7dfdf2d1938d040aefcbd9d31ca", "input_original_sha256": "9e5415c0242133c44a1092f7619f2181a21d2340f52d873ba05d2d3bd0b6b13b", "input_wav_sha256": "5a7c1cd100b91a66e3009de5dd7363617e675cacf5af3009276c7901b7f99828", "input_pcm_sha256": "beef22fefcc0b54be9e0a3ac77be5eee2c82e6ad9ee305cb0bd8cb5414f53237", "output_wav_sha256": "9d4d2a0553cc8d6ae7abe43eb8033b70b3f74c2cad009a30a2d9353c2237819f", "output_pcm_sha256": "7c30a91ce3316f4c36789dd1a890eeac077fad51e35ad08543d820e0b2f4d1e7", "result_sha256": "b56931aaf76949a17f34f66468a86418620ec9427e3ff6e65f0998ddf37f7a51" }
#703object_counting heldout	10	The answer is 10.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "706f153abe56243c5edc2576b731663aa00612df61547afb81c1058736f162f4", "started_sha256": "a6ee3ec0099bd0f848e6f5a8c82052dec9230a62e179fc55bafc69f39c97b920", "original_grade_sha256": "de2ef73cdb99d2e2255e7ac14bcf560c8386aa6807863c658762c4b3512e0672", "input_original_sha256": "706f153abe56243c5edc2576b731663aa00612df61547afb81c1058736f162f4", "input_wav_sha256": "61293f682b2eb69ace701035aa956eac2b95392097d61b68817526f507dd9ada", "input_pcm_sha256": "0f4e5b450d3cb7ebf23fa3e363c2d5a7c7c7693c277eb509b099106676ea3fdc", "output_wav_sha256": "9b6a67a26908539f2dd47b42ea6bc5f080dd7d5a95037f603deeec44945db53c", "output_pcm_sha256": "f44ae15ec5ae9b5416f5c9797ff6e83e286b516f648fdb7af48fb46d3e6ef4eb", "result_sha256": "7df5f09e7228fffb10e4f12451f2d6c2bc4a0f10afefda6a8d002d39cf6741e2" }
#917web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "fe10817893c913bdcc9bee67f99bb3ac215500d13ca37605acdf29d87229cbe3", "started_sha256": "d3366f63ec79f396aa0769e7b880bc5960894c5c5355d2ec4b40e10fdec58957", "original_grade_sha256": "9680a51293ecd9683bc8f8d2e2c57769eb8257e5291832f897d86cfda7313954", "input_original_sha256": "fe10817893c913bdcc9bee67f99bb3ac215500d13ca37605acdf29d87229cbe3", "input_wav_sha256": "727c2ee834958cf20845212981d3f81afe520fb3a946b0f013352a55d8d608a9", "input_pcm_sha256": "ff9c3b7473160c17a7240e231c9ea07c01c20d913a323386ca2af6a72557d946", "output_wav_sha256": "5424732f4ed7e70c2bfe097737aef8082308e1c745143203a707c7e7e36e0528", "output_pcm_sha256": "7948151affb9034b17225c3274b5236137dc7eb006780b7ee07ddca8177db36f", "result_sha256": "3f3375b7944c0a65ee286aecff19120148edbdbccec4499240a6010eb9f3617b" }
#43formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f194b368a4ee13d349e7081dcff3f6beaff3ca23084e07852f56b3cfd27fa83e", "started_sha256": "c0c7d38365b7f187e388042ea1d047a8b756b0408b1fd341898f9b37c677cfa5", "original_grade_sha256": "d2ef1db37782d7012fdf710543570b5626822779837799035f6bd30f24fecea0", "input_original_sha256": "f194b368a4ee13d349e7081dcff3f6beaff3ca23084e07852f56b3cfd27fa83e", "input_wav_sha256": "b17892ad9814e6f9e89a16169c162bc60e7fc58690f9f63f56c72622869db25c", "input_pcm_sha256": "13725b58d18e174dbaf7acadbc25fb86d9ab780549fc5f8dceec2f881fc4a2b7", "output_wav_sha256": "1ba534a31958e13881fbaa8deeb0fa302447733293c05e8ddc361606f2befcd6", "output_pcm_sha256": "a18ec78e4b232aca17e9f5cadc8975b222118d7b321a5f03077269a7eecc352b", "result_sha256": "47a1f61f07999f592eedb7fe0e1dea80b9a5f55fb1ca2e9ebd9781f4f382adb0" }
#342navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "fd2e238dff8b7b6e7156aed75ade5786fa7ec910ff0afd8fb2619bcf9f12288b", "started_sha256": "877322c1e2eab1c585bc83b84f2aafa6fa9e25da538d168e60192c8ec4545ae2", "original_grade_sha256": "c605d32bbb11d265b76827b575749cc7e1773e68afdeadc349c2d82d3ab42cab", "input_original_sha256": "fd2e238dff8b7b6e7156aed75ade5786fa7ec910ff0afd8fb2619bcf9f12288b", "input_wav_sha256": "95825a2d1e0cad85a0f3a487eeed1ab1d05538109d09f87876a79b844a03c14e", "input_pcm_sha256": "d75590bdb8a3b074235ee487dd0af5b8b2f3abe9e870926e6c156e59be5655a3", "output_wav_sha256": "e12e68ee76da58621d6d667a9b576d1c72392a5db7e048a87ccb8df6249faf18", "output_pcm_sha256": "a1f051ba6e7a201ff488524cf660d16edbbc628121338185967cbbd595279318", "result_sha256": "56f6cd8a590c321802986d6c7d6a0e7150560c159d9c301d62cf58cae286177f" }
#655object_counting heldout	8	That answer is 8.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "8a4fb31e573d928e50c31610572d213b9ac135fb502a90ffd8d1eadfdfa28be9", "started_sha256": "7bcad3b928ba7212b66d13faf80f83e47bd618e9ec3ddf31915d236dcf6403ae", "original_grade_sha256": "e0f96dddf0b81449e1a728bf903fe0b64cd4064c755100bf388042bd14171c06", "input_original_sha256": "8a4fb31e573d928e50c31610572d213b9ac135fb502a90ffd8d1eadfdfa28be9", "input_wav_sha256": "83491490c9bb9ad36e0829df16c15ba8ced1ce654a2153416508e0135b2f85bb", "input_pcm_sha256": "93327ddb46e948db2e825be618c56f0ad9f81a137841759a5767aed7e3d5b966", "output_wav_sha256": "aa5e942a92de4fce29ebfbcc81ea1835e067da7c96874c347e6eac04d0205265", "output_pcm_sha256": "134243cd68317bfe6e291594f4a66f28394d4714ca074af26a6a43c4f7b9ab63", "result_sha256": "c9d50252cfcbe6025a91677c68cad89a2d3ab661a66e5c8dd58c356f0edc1c3b" }
#808web_of_lies heldout	No	That answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "ffffa2808887d18dbaa1d48703039bc1d6c6a801bbe9f1d77ba32e456ca15c7b", "started_sha256": "92fd0166e0ba7dd411dbe064ad598613d5968710fe23c6d1b0f8040e285d2811", "original_grade_sha256": "3e90befdf7e534990ea1dd662e229b18d4bb0632fba2a2fd7f2e5f130f4fa4f3", "input_original_sha256": "ffffa2808887d18dbaa1d48703039bc1d6c6a801bbe9f1d77ba32e456ca15c7b", "input_wav_sha256": "18eceaa45997089fe10432e7e7b207e68a120435bbdf52a44485ad03d97b019e", "input_pcm_sha256": "e124efed105ab652f3353ae574511861a59dc0baf2da9ef015f01424fbfc9861", "output_wav_sha256": "e8ad993352138ee2afd91c1005f3281ff4f641099c0fb5d2956d6b5ef1b6d218", "output_pcm_sha256": "f2bedb87af6d16df4df529d16f50b7f0b1b9f32491fdc79d7cc25427e60dbdb1", "result_sha256": "58bb6b9976298c0a58121d18b5b43a82954b5f42d6bdbd519495bd8961d82a93" }
#87formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "fe138e240f6f5ad99b88d7251e5057b2df2529ee6437d7ab2e2504c4c7e4039b", "started_sha256": "0308393ada2d7a3a31f3529732e5dc208af05bb3001689aa5aa72c88bb4f2f8c", "original_grade_sha256": "680527e9c7772a3a11312a109dc7ec8c1c539849da2372cc48bb342e53c62c5b", "input_original_sha256": "fe138e240f6f5ad99b88d7251e5057b2df2529ee6437d7ab2e2504c4c7e4039b", "input_wav_sha256": "351c27ac8e878820bcc8493deda6bbbf966f0beee4ff45a365e71ab07e607f5a", "input_pcm_sha256": "e6af53e36adb545c93c15601977ec47d319822c397b157974c7a3236908ed330", "output_wav_sha256": "f837901ff58cd89fadb5d8f37247cf1642390ac84eaa1ef2ae9be54b0aaf80f8", "output_pcm_sha256": "4b4e1766bb7840ac1895e976480e18400dd5d310720e8ec7db9c4324f0010833", "result_sha256": "43026475b005431a2b6797e37889cbcfa2183fb2c21a04ba1580043481b0c9b7" }
#485navigate heldout	No	That answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "bf39cc013f407faaa0f6e070d1f03669141ad340a6c1d9b3522d428b47ea0edd", "started_sha256": "a99fd719915e95f3200f468b2db28be1cb27c062a992e4c60bb5919700f2535a", "original_grade_sha256": "6f192429a1c7d78fae9fd167bffd7819842e2b84a81ada6ec93076406415c60e", "input_original_sha256": "bf39cc013f407faaa0f6e070d1f03669141ad340a6c1d9b3522d428b47ea0edd", "input_wav_sha256": "0ec58f392668c960e43cbc69cc85855f05596c5a25a68c9dd6455d24e211c4d7", "input_pcm_sha256": "346b6669301b707398a657d7ee446cf4610bdd405fce8399a9cfc74d9233f86e", "output_wav_sha256": "fc52e99f2c084646367498ab3aff480f8d8ae1d33b2efd2379cc2a8e2443f8be", "output_pcm_sha256": "78388d1ebd6fe902d0a2ce1d4164dec4542b13e4f0a68661be028ba8cd46a8a1", "result_sha256": "894ed4c40ab1859eb8beddcb7220a4873c6b1a5a04603c112f29980606b0847f" }
#574object_counting heldout	17	The answer is 17.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "4d4f632758126fd444fb20fd673b38fb05343c685c84b9c88bbbba00ff894061", "started_sha256": "aedcac5074d8b952a018ac59d564d9852042328acf18158e6aa31e7786d21753", "original_grade_sha256": "0acddf47fd534ac677a86f97a3596feebaac3c329af4c984b66e5b613ad1d114", "input_original_sha256": "4d4f632758126fd444fb20fd673b38fb05343c685c84b9c88bbbba00ff894061", "input_wav_sha256": "3edd87ef118839bccb86fb3a308011c9d971b1c9e1f36742a47796df79344e25", "input_pcm_sha256": "1cff89f8dc681510a663d3b8d78bfd7b47e21149ad8f7ea508f94b9a893eaa24", "output_wav_sha256": "64cca9ca20abd8afd1687f1168c4b8581a914a22956ec343c50a229f5b00f102", "output_pcm_sha256": "71de79017368ff336ea402a5505b535ee6ddcb6420d42af3b51c37bba4cea6fa", "result_sha256": "a51f5a2b52375dc9d82164f677ed087715240f249497cf0791bd2c5f960109bd" }
#844web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "d15e532d49474323eee7bcef9202651d4d6b59ac8bcf17b3fb09c37cd6f080f6", "started_sha256": "00526e2379b7b12a375c18ecc5c9bd0e3d011e9698890d81013742fa05301c61", "original_grade_sha256": "6186d5a48db1b9083dc17d8304b5a72829dd84c3e3a4089dc5126250d0b8681e", "input_original_sha256": "d15e532d49474323eee7bcef9202651d4d6b59ac8bcf17b3fb09c37cd6f080f6", "input_wav_sha256": "12ab959b0064ccaf8bafa2a2de69c254ff3fed88290656f7352effc3667b2ae5", "input_pcm_sha256": "e798697ff552dfdb328e5527d76677b7490fa1d04897746f1e24c7d67f46a4d7", "output_wav_sha256": "f12f09cd8a58a13fb9f5e665ac04e358cc6fdacb16302c6d509ffd1834be461b", "output_pcm_sha256": "d350473e84e75416b83fb10b7142af0bf216be0488b8ad17ff5bef0c0ff08f89", "result_sha256": "60fbf3a128eb6da5a128740db670f4ba4b6871f5ccb5e02ae34cde4f432db365" }
#118formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "ede94057669c318d34754161555739cfa930319b412912756457ef419132af93", "started_sha256": "2bd28e9f52e7815e69d3dba2f131e4aad0d9c68ccfac70ff9ab275bf6f2f5e24", "original_grade_sha256": "ce5ff5f42b4d66a10299d4fa27f4cbf852023466af72c69be318aff69fe299c6", "input_original_sha256": "ede94057669c318d34754161555739cfa930319b412912756457ef419132af93", "input_wav_sha256": "9ac71a2e9ab7beddb093d936bc63105e0fd683718f0c63f75d4abe0af62b6c87", "input_pcm_sha256": "0da9c55b3ee2a8df13b5e45c0fb4b12f99cfe804dea8b7bb20233aabadd8e93b", "output_wav_sha256": "a956458c6052e96f19755ee0d31e57d7e5732cc057d693cb94a5c5a706f074bd", "output_pcm_sha256": "5556522a101226502cda9b4a75e31c7dc83a9c49c5e321e2e4d2c43f8d1aa15d", "result_sha256": "ba2b84c9ba4a3da17100d993693115d2e43894e0e5d98569bd35118ba0f0977e" }
#375navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "d2dad77509bbf7ab1139d0843038369d11120625959291fe7d7b9fbc2305c4f2", "started_sha256": "aad1fed3d4029e67eacf33df4352bb00523ae667fedff3d2b901e557aea39c49", "original_grade_sha256": "52a8a3531661ed160569b0c5a72e0771695551dd04239ebd868fa7137b282bc1", "input_original_sha256": "d2dad77509bbf7ab1139d0843038369d11120625959291fe7d7b9fbc2305c4f2", "input_wav_sha256": "4a8adc7c31cc77ac5f63a80ea8ac3e74d9e0e440868444533bc259b95d568c6e", "input_pcm_sha256": "62c540b67dc5de68eefb57cc0ada8b1203bbf7ccaf78e0fcc474672c46cca2d6", "output_wav_sha256": "f044e6076cffb6a5e4913f9a3c9a00648dc107fc5718551659280e1cbd033ca7", "output_pcm_sha256": "d7dd7f88a13dd08f609f1ce623bd4191125fb4d6acaff785aaf23d76d0a58f74", "result_sha256": "48e013f91647160c7728beb20c9abb85314d0029e1aad6af6a4a601700c45c08" }
#563object_counting heldout	8	That answer is 8.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "9e6a534ffd02951a3f059eaf027e2961cc5022812e3918f0b750a0f6d0bba99c", "started_sha256": "e66a824c33ccc31f447bd08a6a51f1132f11d0b649593f5cc03b67fe29703c23", "original_grade_sha256": "64ae81d288ab4f81c144d17b695ac9eb498b7f786421f3162ae7b570ae4d5dce", "input_original_sha256": "9e6a534ffd02951a3f059eaf027e2961cc5022812e3918f0b750a0f6d0bba99c", "input_wav_sha256": "6723b6fd7a45ef83f408c8656d9fb4a97d691e0fea9eab12a6cba263a83d5bdc", "input_pcm_sha256": "bdb5194ae8b9337753a27f650160e96e59c022f00ab85982b95be8c1bf5eed90", "output_wav_sha256": "80ea4ec344ee56d0ba0e9d2c7b3251af62377e039556c10ad13d04e16cd076d8", "output_pcm_sha256": "083d003e9cef638c2e80f3c37031269712767aa293ac8f272600dfc948e86caa", "result_sha256": "81dc8fdebccdeb78af19466040dfe135829727ffbf33e329764b86ef12f254ee" }
#765web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "65bb5849649fe1e8d25ac03a57f2081d2da0a6a32a1315b11f73e0f1a6faa3e6", "started_sha256": "092418326ff012a72942de9d88a55fff378423957355422e5fbb917c3663c36a", "original_grade_sha256": "cef20716fece06b863b445014aa661e297850561d4f62ce2498c29a6bad611a3", "input_original_sha256": "65bb5849649fe1e8d25ac03a57f2081d2da0a6a32a1315b11f73e0f1a6faa3e6", "input_wav_sha256": "458565206613109c32b96c3e84a82604bdf124dc1c5c7a260bf3b40598a5503e", "input_pcm_sha256": "f3e0f0ce83f41e7d80bd1065810bdc8ca527cce396ef40ed03828d327bffef75", "output_wav_sha256": "97b0f1605e4cbc62afb0bc12b4dd29656721fe46909342e5f1d933d6c0b5878c", "output_pcm_sha256": "de1951ba3543b70ce092304e82fbf061da7f94fe62f9f6990d921de3ecc54a5d", "result_sha256": "6ee8b9bd808a4020d63ba393f6112f4469dbfac90415caba61ab29fd7a4fd230" }
#75formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f7153f2582c64c0a78f902e48760af78afb08c40d57a804412475953e8f660cc", "started_sha256": "20a238f839896734fef4ae8b3ca2e50445fbe5e78b19ea9ab558e081ce5d6d16", "original_grade_sha256": "360db4f769dc5402aaf39d75baafa691287d477b409569fd7458b9e3e98642a5", "input_original_sha256": "f7153f2582c64c0a78f902e48760af78afb08c40d57a804412475953e8f660cc", "input_wav_sha256": "dcb1546b349f514238c8ec224e1dceb9e23ba2cb2e8209e58343c9c31427895c", "input_pcm_sha256": "a592c78fc02d5f5d4700cefa5bca03ee4aa8b7de73bf31f3e8e2c0fb713aec79", "output_wav_sha256": "44e774ab66cc80a69b8b7f3c63f7b563ef4d8a8e15edd088ee7f35b6907634b7", "output_pcm_sha256": "f947bc787151ecea43864531b970d7c0ef4b244695e340b7e569a2c976b29fd6", "result_sha256": "732834fe90d92206104c948fc7d7ee7285508d905a8a18f7d72993bcbfcf4f2b" }
#413navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "d9397e03e0d93674eb36c267f2d8e9c50f0658a888744d4de71cd6b39b97894b", "started_sha256": "52c47d3bc0eeed1315adbe5671e61a2de0f419dda00afc33a7e2fc6fc42cf86c", "original_grade_sha256": "922f57b820ba203bb53a7736d0d61731c66bdd20db53d3e1d84c4fea49d2e172", "input_original_sha256": "d9397e03e0d93674eb36c267f2d8e9c50f0658a888744d4de71cd6b39b97894b", "input_wav_sha256": "0d67b11d029790d5cfc37f131bfffc06eb9ed12524094a17e7a55d3a00957d96", "input_pcm_sha256": "7c7843c937768d3da5b183969ed93d3102b4414d29e959539d3502fcd5433b20", "output_wav_sha256": "f2bedd19df75dd9baace35a84beead77abf319cbfa6fd75a1a5bbbfeaefb155b", "output_pcm_sha256": "6169575cb6e27d013d8bdd94f9cdb7a61b7774c6d30b92b5c8efa39614bcf02a", "result_sha256": "cfd54fe97d34665517172e83a6d119e60c1b86736ddd2f556740b7e435073516" }
#661object_counting heldout	5	The answer is 5.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "94e5c5c192a67135226a074707a659341a40ddb2023be82c4fc1b9794ad078f5", "started_sha256": "3b38a9859dc8a9301bc2b9b18336db3457d3d7632410dc9127d07df7161d4ab8", "original_grade_sha256": "36f1381b05962b8db9a7658f60a703b62a5b35187ac79c961f2e02aa75d34fb9", "input_original_sha256": "94e5c5c192a67135226a074707a659341a40ddb2023be82c4fc1b9794ad078f5", "input_wav_sha256": "f297dde9acf7e050003c8971879ad19fbdf4a59bbefe7ef71555ba4caff2b609", "input_pcm_sha256": "a8261f715bc675d27dd4bdd95490088e40194ed806696c9ec71a2c4df9aecdc9", "output_wav_sha256": "5863a29667a3203dddb2a4aca5763b630e1ab577110c91233b1772660b476793", "output_pcm_sha256": "ae1be0e48f076ad0f43b9a32f4ffbc63e1b6a2ca29e5f2edc5e752c0643bd614", "result_sha256": "2db2bd66d84a18df09619a67524226242278a54e1177249689ad61b3504ffaf9" }
#952web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b0811f65ba68b2b1eb44b2289f503b233b9edba9dad8dcb1caab748b17e16351", "started_sha256": "b681b8981bbe8007c7d13579861ee1499fb017455a7c2c78b2626eab0eef43ae", "original_grade_sha256": "98f1fe3162cc1a35cf6f14e928a4fca119a0f0cc19208b54c5277af5da6dfa38", "input_original_sha256": "b0811f65ba68b2b1eb44b2289f503b233b9edba9dad8dcb1caab748b17e16351", "input_wav_sha256": "30e6f109bbabbfc4fdb1507f5e18a469fc05238c1f2d089eef6ca3064534bdbc", "input_pcm_sha256": "d14a8d1e5bd151935f830becfd5f6dd8d03540443d6a64da168db3c082aadd60", "output_wav_sha256": "906790128d64b5f3fa69d4e86ae8c473562a527c3d905093b14e17c3797eacb5", "output_pcm_sha256": "1bbd4a072ab8ad93c51c5645a313cf357420e87aac4cc966e694bf6e480a0275", "result_sha256": "dcd4d92c1c1c117a7eb02a3dc4cbbefa0f632c6d0c10983c7ac59c7f3c63881e" }
#242formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "ef0518303f469028c66108826fbb4c5c4661f4178cdb1ba38a175add211ad92b", "started_sha256": "a14ddd48f7bc9f154bba284647f41f066b84ea8e23147ecc09bb6cc8db749dda", "original_grade_sha256": "6e32f444d8d76559b9ec31c476006b0d412f2e47e75560247d774d4be1a6c456", "input_original_sha256": "ef0518303f469028c66108826fbb4c5c4661f4178cdb1ba38a175add211ad92b", "input_wav_sha256": "6e92e59d880d49fc83064ade0e9a4d4655293934f093c337c264163b2e19eca9", "input_pcm_sha256": "2a77ce55c4cde0af2868c0a4ef7247157f00b5ebacfc5f9456412fa56a8ad42d", "output_wav_sha256": "f4c963eddfaf55af935758ae6d5d97690c040913ed999712c87df634fa9a5200", "output_pcm_sha256": "872fc99ffd88db64d1e16a7bb7ca088133b0780043dc763937090cb6c5582d8d", "result_sha256": "d41961cf37117c888a0528fde1b5d3fdf8845759f31c088892bc18846d37275d" }
#333navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "8f6fe8d787c61a2970a5144e2274520beace2156daa17afa3538144880fbf61c", "started_sha256": "51bdc3a69cecdf1fe944e4a8028dc553fcd6efd2e234ea66cf5f0a6a5f002bea", "original_grade_sha256": "db32764a2823ea28be02c2a7d2d570910e2d089ca562db13569a305dd8e3bf4e", "input_original_sha256": "8f6fe8d787c61a2970a5144e2274520beace2156daa17afa3538144880fbf61c", "input_wav_sha256": "6a8d2f6ecccf4c25c9f7f3872cb75d9effe56d335e87c437399393ae205e476f", "input_pcm_sha256": "21cfae758a50617c22ccfb9ec907f03aa9828dc5ceb10d93268ce6ac879d9405", "output_wav_sha256": "889222e20a411a2e33b2c8dc9d099d892df0d9750f6e5fe9bca3a8da27e756f9", "output_pcm_sha256": "79230c69d22db21e93906496c901fb9b38c182f235f7836fd9c98b98b57a017a", "result_sha256": "095f9614fe6fa1b9a7e37030ce9e4acc67da60de65c2e73de0f311b15ba0c668" }
#555object_counting heldout	7	The answer is 7.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "72642f31ad45bbbcd037a188c0f27870c021d30a27332d140262ab924af6c337", "started_sha256": "bcc98c4c080e1200758a7b9aa5c8a8a01541234f7cc96f89eadae311b45c7c7f", "original_grade_sha256": "0d81ca2a1d85cf0f6b9269e0d9f6b37b805231216ac76492a7db652f9ef37a16", "input_original_sha256": "72642f31ad45bbbcd037a188c0f27870c021d30a27332d140262ab924af6c337", "input_wav_sha256": "3db6e2e475e1f82239d82309d5703d4e63ebca62e775672d66e06b4b9a288888", "input_pcm_sha256": "45c50c22258fd47656021417b32a969250893a61d2bb5004bd13ca50f9bd19cc", "output_wav_sha256": "91b5d26ced42edabc41bb9bc41bd4c22cdc9bff36c84a87ec66598dbd88a41e2", "output_pcm_sha256": "72c96a6e7b612af87cb503d4a1cab37b96984018171495505873af2b8b1f8880", "result_sha256": "6b60c10b6c84a61cbdcaef912cd879cbd03589939dc37873e744b64bb42ddc02" }
#904web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f9b985775817cc54c07b44114f53fb0572bd5823748d4c8a1ab5068fb002d84c", "started_sha256": "b8f49423267cde474ff65a126307bf4e87f63801437cfe97d92234aa22c08349", "original_grade_sha256": "ebdc690e9f41de5d12642a9ca5f54f5ce9f130bbe625c8802973bbe01369ac28", "input_original_sha256": "f9b985775817cc54c07b44114f53fb0572bd5823748d4c8a1ab5068fb002d84c", "input_wav_sha256": "389d99808fd3aeb6c7222491ef707cfe5f3aaeee66f65787b617ae9981abc570", "input_pcm_sha256": "f3de5b7bae22104633231b7f3a4510be284a281fe3e9141c3f5ac8937db083b9", "output_wav_sha256": "9fb6a4cfe2da3016da76b4848a4052726268fecc58e91cf31d035f9be9b328c6", "output_pcm_sha256": "0ea222ed2ff32f3a4853002d969a44146164920fb13a96e4ca1398c47c89a6e9", "result_sha256": "75dcadebd5728d84d8ee5c090676e8ff49fa798d48c0152193776e5d19659731" }
#237formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "8c278215c22df5052b34244e59b07aef06d311d5d9b1250e75a3194285f81323", "started_sha256": "4cbd8def67ae8abf733868582d7bbeadc82ab4a697e61acd805561fc6d328779", "original_grade_sha256": "dc18ec5d7625baeae650d692124a7ae38328fe898ae7a7e98a978bfd59484034", "input_original_sha256": "8c278215c22df5052b34244e59b07aef06d311d5d9b1250e75a3194285f81323", "input_wav_sha256": "2a8f4209106982f157fc3995cd41eb7865fb6524499156257354b430bad0ea36", "input_pcm_sha256": "5c98d2129114db7d489746e82848858505e2464efaa9649b1a569ada08e2637f", "output_wav_sha256": "788fd0ba2f6c22c57a6b32ca7b067fc0d5871b1e8bcebe4f8f348f5902e4dc19", "output_pcm_sha256": "d28ba7ae538b7545f91833de6e40c6b98af82a359566f300f2e5fd22f7b5a8e4", "result_sha256": "018800c4c66d26459c303b812cb82f3221902d467fa986c81be68d389f7bc42b" }
#253navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "bc88a0234a7bd16f3496336500a31756c4596948ae9143c63252cb5bd155e825", "started_sha256": "f12a1dd6a946a8335ff8188f88c9e505c9414710f1b648f5aafd9124803cc301", "original_grade_sha256": "1088681365718af7dd78b9071b7df3a3b048043cd2147ac3003cdab8a2b273a5", "input_original_sha256": "bc88a0234a7bd16f3496336500a31756c4596948ae9143c63252cb5bd155e825", "input_wav_sha256": "a13e9df6066c1b30d7d616313b6ead507c9ffdbe45354e4fcd53f96ed0671ec0", "input_pcm_sha256": "c58703c0b834b6e24327db550836a8565868ac4625669d5159730a97e16cca06", "output_wav_sha256": "ebe912584cfc2303297c2d03998434b632e10cdd86d9d3647d62313eb4692eed", "output_pcm_sha256": "ef905ac8641cf2ee79f5f6dfb9b2050e797597d0a7ffb13f6aa8a96fdab8abb9", "result_sha256": "583af7a9c9e22c981b2a1befd96434e1ca06df96e2cfec13d08ff18d6f11748e" }
#529object_counting heldout	16	The answer is 16.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "9aac5228cfa7421606fcb548075a3091cee26e2eb0874056ffb6dc3e20a93e43", "started_sha256": "444b27cc492bf7e6ca5ac16beff2fd20ae14268f3d774b350d7e5553e9a81b33", "original_grade_sha256": "c315b09d1bb67d8eaab80712bc7e43df66fc29790ff89b579ae2ae6e24363024", "input_original_sha256": "9aac5228cfa7421606fcb548075a3091cee26e2eb0874056ffb6dc3e20a93e43", "input_wav_sha256": "64466b05ff7ea7b1e90d6815fb9580f44bc48918116e5fdae078fccb484c2f36", "input_pcm_sha256": "6ec9cdbdf4a41aecf7a80130c19d6f2e98d28c577affee2715b2cac13f962c10", "output_wav_sha256": "7c3623445c96575736ae50b741caf09936e4b78d673edbfca765e02d385bc040", "output_pcm_sha256": "d3a816656badd05ec32875922cd3a990a667f52e39b9f86269118ee2e0661df2", "result_sha256": "c8b91f4b0c81ce85e9e78eaae9d47c68df4d670ec4a7e15d505bf4e324812db5" }
#973web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "e89fa61389e2347206183559426e7db9a3d2a43b0fb80ac8ea43a50d406b1d2d", "started_sha256": "a5243179487caadff4d45a56633c154a725542ce2c0590508fac74b56fa84d74", "original_grade_sha256": "42f44335c22f57d93924fcf55d93541dfb7bba33ae3be57ab82aa3a33b2f6370", "input_original_sha256": "e89fa61389e2347206183559426e7db9a3d2a43b0fb80ac8ea43a50d406b1d2d", "input_wav_sha256": "695888b6501bfb66881ac9529ebbcd896025ca04905c74ea734877f4f820a3f4", "input_pcm_sha256": "12d4fb472f99d57b0c3500e6b8a9557222fa74e1a227f344d761d06f91a3be04", "output_wav_sha256": "cc9cc079da097136b5d8e92a10b0cd99bce7826f5c2348d3ccc21e783f648f08", "output_pcm_sha256": "617073eb949ee886c6fc840f0cd220c40b72ca66d0ed3935ce24f56cd343d427", "result_sha256": "e38a8350362cfba160bcfa1a47e9a5338d351497fc588f9284677a1a2535bcea" }
#116formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "215372991c1258e0813a6ba52bcf31502f0786b0a3ee4142550c1950120881be", "started_sha256": "13596077764704084bdc5f188ae3bd2a01f813388555ab1ecd2ce4721bfcd688", "original_grade_sha256": "b1cefc8679309de6eb6458420810c51969a4e2895a5ed7659aa859b0b18a3bea", "input_original_sha256": "215372991c1258e0813a6ba52bcf31502f0786b0a3ee4142550c1950120881be", "input_wav_sha256": "377b9632514e4e9b12f86a0c2a12b1d163197a5c494fb23013f7a69ad324eb34", "input_pcm_sha256": "cf978c237c62c3d29a44d7dac828b6d26a77486ee6997a1a7e8706dce076ca30", "output_wav_sha256": "41c0040c56ee3f67eb6e02d6cc33a32de49f587d20d40427befc42c413ee2940", "output_pcm_sha256": "4690d5191d61fd674fdf27892756e08a7242f06140b3b9985a356876f17e0373", "result_sha256": "d9a6ee8e975fd6b5ade7d151e932810b873335cc542f891b65a91901a6b34a0a" }
#380navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b12d21ce9e4f57bd6053ae0af920e205aad3035200c4bb019489edc3a86211b4", "started_sha256": "c1c07bbc53b4e1a30d5c10e2faaa468c31fa471fd09b10c0309d6930e80a0b95", "original_grade_sha256": "ea8fabffa3a669438e6f0d791cb8ec83481fdf931190426944077cbb9b24590a", "input_original_sha256": "b12d21ce9e4f57bd6053ae0af920e205aad3035200c4bb019489edc3a86211b4", "input_wav_sha256": "abff4fc8f97dd837e84ce0bf26e224004eec33e3f3028d3af3997c73542681e1", "input_pcm_sha256": "d79ee74f1e9e18930beae33b41a702349a1a5b9733d61351896c2e18265f2984", "output_wav_sha256": "fcca9b938806154ec59f939723814718d22ee176c4361b5d832ba440b41c4e27", "output_pcm_sha256": "5bd7bc27c6839870ad084542b4eb97c0f3533c1b528b38bde13c092475ed1d3c", "result_sha256": "d78fea53f5dbe4a43858e2d77768ceb4a4713ea36e85fa000f0db8bec61e53c4" }
#536object_counting heldout	9	The answer is 9.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "7a80ed48fc1a14fb6ee691cbb2dae616dede361de8af221c9f43f3da6e1e992d", "started_sha256": "fd7164b8535348407317d4aaa8c8915f60b1254fd4885388460d343097c57786", "original_grade_sha256": "57b2c3414c9723181da13ed21cf0c3670ba17dd7bba7006f8c49a4d898c09a17", "input_original_sha256": "7a80ed48fc1a14fb6ee691cbb2dae616dede361de8af221c9f43f3da6e1e992d", "input_wav_sha256": "07b8b89306b8bbde4d5c78597cd8951a8d1cf00364fd4762cc6169348887c08b", "input_pcm_sha256": "df363953bc30bcc787f530c21a88a6a3e131d15dcd46931a0dc4b6ad3aa31102", "output_wav_sha256": "42349a18ce6428279d4342c8e52821f54b18cd12e37e9b84ef04c113e03236b4", "output_pcm_sha256": "d4263c776ce109850c5f41a9a6d7b158f6a680216532bb21aebc5a3f6eb6b50f", "result_sha256": "cc94b38c5b7e1a968a52713b82e0f5c30ffefd9546e94b408af97dc0b71bbb80" }
#993web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "7277a53766aeb8ec3dd5c480fed2024bf47ab854ea98f91402295baf74d2ba74", "started_sha256": "8db096aebeab1a06370a59f4a02a657d62db4a9a6ab942c545397b7ec670280d", "original_grade_sha256": "6fabbf8b86f1e0387615530cef3abd174b7ecd1a84324b7a72db99d3dbee91be", "input_original_sha256": "7277a53766aeb8ec3dd5c480fed2024bf47ab854ea98f91402295baf74d2ba74", "input_wav_sha256": "c02da9b93e00e7915c4049c12c325bc83eef7171ea85fdfc2322f396c4e1e70e", "input_pcm_sha256": "f72c87f2e88d9d3e5768199031db518ae5199637efa398618107566ff5389972", "output_wav_sha256": "3498bdac3bec5cbbb326b6b910ee07270ddb7364a98590c9aca5dc7777fb7199", "output_pcm_sha256": "744d9c2ac0b6370b27f6cdac17feaff228a31423a07794fcb10f9b2b58dae1e4", "result_sha256": "e68fc9c6f8b4d88f8227f634209435fd8f187a31f0a37b81ede951304816c80d" }
#247formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "3a315683b81ea3d1796a9cb9d76c8fd42f5634c0b0aaaff206a0398a42441798", "started_sha256": "93a574ac7fb972fe136412b75e2dc9b80bac5d5b471f9e30b7ae2f89f41b6287", "original_grade_sha256": "68919e4cc36bc451558747737d34c496be9cdd585087bbe08f58c901ce9f47cf", "input_original_sha256": "3a315683b81ea3d1796a9cb9d76c8fd42f5634c0b0aaaff206a0398a42441798", "input_wav_sha256": "c358a157b26b33a382319c279dd83b37281a854aed20f92c357417a3fd67ecc0", "input_pcm_sha256": "351232a7b59607d2d639bd6ff05942eec48a468663180a03f87479246ff8d337", "output_wav_sha256": "1dc243b0cb7b144884fe93523e0172db0f868484b029f65495f2e7dc4b951082", "output_pcm_sha256": "203afffbb85d2e54bb9ca649371ddd5d492f5830bf9fd3a7f63eb0c735eb5454", "result_sha256": "c30449305b6e5d242856f1c24c80aa5222bced718aa1f7a195e51a9206cbbda2" }
#338navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "5a6f20ec1072bc454a1477acfb708f18db81b4c44dcdff91d190fa333cd8638c", "started_sha256": "aac9548cc8117966782348123d9f51ba29c21ae9f9c076cfeb720a0aeda15fcc", "original_grade_sha256": "b8cf0866239b4698514080be03a6a19bdcc956acbede6b834ae18fcf57b88363", "input_original_sha256": "5a6f20ec1072bc454a1477acfb708f18db81b4c44dcdff91d190fa333cd8638c", "input_wav_sha256": "d72e63387e819a389073e712b091c31f39c12db89370ebb9210477fb2c11606e", "input_pcm_sha256": "68da8125c72b4915b9c74bd4dba11f8cc7120eedf79040b02068a728e273e672", "output_wav_sha256": "9cdf7bd7785d2ce0aa0a5394c139fa6cb6f4b1deb35d5c90b355ac4deee962e4", "output_pcm_sha256": "83b127ed71fd584cb7bdafa312c7c44dc520600d04ce422469a3daa74f237675", "result_sha256": "c4c2298bfced77b81f1b1eb18bc948be0d4921d6ee9d2379f3b9f7dd74547215" }
#606object_counting heldout	3	The answer is 3.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "a4c597746c2a592d05ddd3ba7807c1852a34c6b6125b27b4949af681942394ab", "started_sha256": "55560f89b325bfd21d659807d76eca462c9b5ddba6807bf4ccf470b9da05b833", "original_grade_sha256": "191c64acfc8e4214e1051fa1f08051d860ca64d0b640de66ed9c3336b86689c9", "input_original_sha256": "a4c597746c2a592d05ddd3ba7807c1852a34c6b6125b27b4949af681942394ab", "input_wav_sha256": "f26afd170535c31e9fb102ca5fcf58a0bd07c47b99fa07d0b1ce071cdae9ebfd", "input_pcm_sha256": "0e8bd18f093c305902386d19c43299f7d18e5535ee13363d7d00d0d02a4c604d", "output_wav_sha256": "61182f8614eee33f4608326828f1b7b85e03acc7064cd206abc865f5a72ca45c", "output_pcm_sha256": "4ab921dc131cffaeb267242861c10fcb33f6b7145e62d166e37fa7527e104563", "result_sha256": "f2db96a0ecc29877d304052a2e1fbc0d5adec018ed4f2d942c78ca6883486e5c" }
#760web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "302243ecec3af1d031ca32270f89a8b99e1db71718725167f544f54927966858", "started_sha256": "2f7943267c5ea30428804ebc88e74c87d4c2c3720c65488906a4350a127c338e", "original_grade_sha256": "2310b1f2ba9185d051471badb387839dccded87bee77fd504ce72e8214e98917", "input_original_sha256": "302243ecec3af1d031ca32270f89a8b99e1db71718725167f544f54927966858", "input_wav_sha256": "2ba440adbbff9fac87cb0875a7cfe67c9e08c673cc72e68cbac11d3315cc03e3", "input_pcm_sha256": "56d4d9004b091d18e7b9dc21bf5e0385d80da4e19241d47ddc4b7a8fb244eaff", "output_wav_sha256": "d326d00ec493fd92f168366ca3f41f71fe1074f172f3b07007d41e29f47b6672", "output_pcm_sha256": "1a4241ed1a5efdffe7b12386d69748d054176d7e01a8f99c00688e284df9ba1b", "result_sha256": "680566877c95c035956057a24128a01cf2c4c0c2e790f68832850696d06c4bac" }
#39formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "4a3886fa6386900b34461f63ab697d4a16967c090c8515af9cc6800c1c37d12f", "started_sha256": "ea96e4e2207f4c617a4103d1b7e5ad0395f967b0d65e7972eb863340343c15ec", "original_grade_sha256": "8cea6aff1e8b39dbab63a7ae0315caff903dcd0bf19d2dc4cb62d770898077d0", "input_original_sha256": "4a3886fa6386900b34461f63ab697d4a16967c090c8515af9cc6800c1c37d12f", "input_wav_sha256": "cca1dc0c7b6a3f3e24e99054740cc4afedd62027de7ee5cfb1d6a25f229d3a0f", "input_pcm_sha256": "b9b7c2dd906795213d36483dbcd324eb587adcc5a3a5cb45778e30dc50b835ee", "output_wav_sha256": "607e6fe3c928b24d4428b5e921fb7dcf2b2504629e76d3a2c66546218745d3b5", "output_pcm_sha256": "67657538017d054acd6777403c8276df2e111257d38fe77d8e03183da4b68970", "result_sha256": "d5df35b6f96bcb70826c61d80fccb81a427bc1002f81f1af8ffd8d477a63538e" }
#396navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "3e605c9811087daf49f9ca48ed154f2f2534660d784a2c6da6116056e27d9c0f", "started_sha256": "e41d3f9816924e352be14eadc487e92eddcdb412d07454c21c9c1e36588b9bc1", "original_grade_sha256": "614458bd0c1306c107ceab360f973fd6e91b7d30cfbd1906f47096108badb5a1", "input_original_sha256": "3e605c9811087daf49f9ca48ed154f2f2534660d784a2c6da6116056e27d9c0f", "input_wav_sha256": "48b7ede9eaf88c8b78aa04e428cac444b772f9351f7fdc6ec0aca7bd6036df69", "input_pcm_sha256": "dc96febca14d1c972b5e20c4ec2a11659bd31963b9d0fdb4ac4f484374575bfe", "output_wav_sha256": "a889e81e7b354e6420998d0cea69319bf00d9f6081b658a9a52d7d00db15a9f5", "output_pcm_sha256": "0bb74b1a2c4bc8818e3c0aeebf476a635988e585bb84e9dd865dee4245dfda44", "result_sha256": "ff81ae69e9ac3ec5ad713e7a5bd8c5b9153322c1ccd7673dd33d5bafc1cedaac" }
#651object_counting heldout	2	The answer is 2.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "3d5822c996f7897b3fd97fb98201002843befde2df0b4351ab46d5cf5e5ec6c0", "started_sha256": "b7fbfbd30ad2fed243076d60adc7fb8658a8680125ace8892155a9a3fe8553bf", "original_grade_sha256": "9e9615bf8cdcd4b91ea5399e18b1c31002d06a18163c9c8f7fc901fa8d072853", "input_original_sha256": "3d5822c996f7897b3fd97fb98201002843befde2df0b4351ab46d5cf5e5ec6c0", "input_wav_sha256": "c6703a763dec736449d68adff6106f507f64614aa0268956f2d59768e80adba0", "input_pcm_sha256": "74bd42d60b3a437c9f1f7fd304ad7e999ac3e6e89319f8bfb6e71f9b07ef59d1", "output_wav_sha256": "8423427f2edb8652417f9b5eeb984231ac9e211f274a984c1b67285139b35f91", "output_pcm_sha256": "7e95f63f39b3b84429e01b96b65ea3b1a7b27568171d36ef532447c62fda9be7", "result_sha256": "69c46c2fdeef8eb2005aee29f3327c447e365d2287c1c071117fc8b69978b0a6" }
#845web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "40bd7061dba09e3d2bc89e7856de287848e404ff0da47b171ee19bdd3fb99d2b", "started_sha256": "c54d37e96b467ae759b6f2464f547ffc213685f87be717df385eaa89b4a85d33", "original_grade_sha256": "a062deac343ddb52d1b039250ed23033bce845ff0736779eb83dcdb9bafbfe6e", "input_original_sha256": "40bd7061dba09e3d2bc89e7856de287848e404ff0da47b171ee19bdd3fb99d2b", "input_wav_sha256": "ddb991c293b23635ef9921d6ce522a6074e560c034dc5a42511350ef8f85c531", "input_pcm_sha256": "9bee1457da47496a6a3c475ff69ec819546b1c59c9e0b984cda2532265ffa015", "output_wav_sha256": "eb1d48918f4270b36a3404590ab89789e3bbc98cd9ab502b6285ca6c2902926e", "output_pcm_sha256": "6b4b1ad7e0042612d2f1e017930422efd5d00df33d786cdfb2e388a58491c4b0", "result_sha256": "079421c37c30f248e5a882def9376d96ff9ce048a3f64de76746d07b9ddf8b8e" }
#89formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "1fde42183d302dd7d70c85146cfe9876dffc7fafdb1d3a07bf15302a072ddedc", "started_sha256": "b73d0cb2f5892c42873ee39c8d2720b7191acc5baf0cd78debf81c765a30d448", "original_grade_sha256": "2cbcc7c482b9e24b8a66c70859027a3eb70ea13c7bcef6cee5d1a2a8b74cb85a", "input_original_sha256": "1fde42183d302dd7d70c85146cfe9876dffc7fafdb1d3a07bf15302a072ddedc", "input_wav_sha256": "6602f53839246c990f0f4711cfda2e11c9feeb5f692c319d9b7e32eaa9c4d857", "input_pcm_sha256": "ec26adf184b70266d04814eea6007a1754b8afc3499a90864cbb36fdf41a88dc", "output_wav_sha256": "41accc50ad80c331f82ca8af8d621dd8d0f43aa78138427298de63bb6558b2ab", "output_pcm_sha256": "0f160bc304b2293b185c603de9de9a9cc5a7c8079be7fc4e05a54cec78d6020b", "result_sha256": "1a0a78884216799b17fc45774dffe876120bf4cf373f39da85c7dd0727bd521e" }
#424navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "ecf2d3b94455533bc67a86ad358c1b1ffd50b9cf3d614a8d40fb1609a0a43ccb", "started_sha256": "198369e63824aaf6b11258599d97a7fb63afe0dbeb2d8917d12ac076599dc3e2", "original_grade_sha256": "0fd8a0b5ce0a1f3b4e1514785672e1a3ee30d18d7feee625489ab729a98c3c2a", "input_original_sha256": "ecf2d3b94455533bc67a86ad358c1b1ffd50b9cf3d614a8d40fb1609a0a43ccb", "input_wav_sha256": "b7a564af5d94dda34b6100b3549ce657a69b6e5f7f974a43bfee43e9249cb731", "input_pcm_sha256": "7f5a4ead02e87c7fb06bf516a078fac50447197b0cccee063c19c107c8f0427e", "output_wav_sha256": "14cf31804b8e70ba31b363d5ba2854b2f8f39a9c924830f9e64e68cc9f9a8be8", "output_pcm_sha256": "1bf9e283c25be196c8ebb6a373d1622a28101ea7f7012ce995aee8d8aa977a54", "result_sha256": "f1dfdbbb12e70b06eca3fec359a1c49b75be8dd360c149565db103154db67b03" }
#748object_counting heldout	7	The answer is 7.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "99a9f2211edcd2f1feb0078ec6d789254d6286821de90547ab74064089bacae2", "started_sha256": "de48136ef26cae4295076c420f941a4a02b7c1c21949a5ef67f961045673be0a", "original_grade_sha256": "f4791b4552bb99069e5283703607df8f0ded03b0298b75e9a6010a640d6ceadd", "input_original_sha256": "99a9f2211edcd2f1feb0078ec6d789254d6286821de90547ab74064089bacae2", "input_wav_sha256": "42c0e9a995a2a75a597c754616f3e157bf67ee3cbf65c5f4f6b64c42bac5e857", "input_pcm_sha256": "e295ca514f3f6091a8f7ede14d95ae64d7c7562451f99695be3f5ab15c66ecf1", "output_wav_sha256": "f9c91ed5179483e5e5e493b9716c688351723fdc6179a167c4225d0a29f18608", "output_pcm_sha256": "05b8c6fa8ec085ebf0351a205b2eb26c872d38afa1f6b76495c22b47a949b9ec", "result_sha256": "647795ec21715453e03343bbf0d87a5b02f85beab4c618a760140796ea4690ad" }
#899web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "d9f9970eca6c53032054c2a30a56a0cb29d02013572a9ed8372ed0e4f773e2fc", "started_sha256": "b8d8c4c1733324055a70ae8c6526d65d7eac2635933a1fd6fa98e778291f4516", "original_grade_sha256": "8098d5c457abafac01b416943534f3302702b2df87b01ae02e55d5e2c5694d8a", "input_original_sha256": "d9f9970eca6c53032054c2a30a56a0cb29d02013572a9ed8372ed0e4f773e2fc", "input_wav_sha256": "43cb92f7a3ed81da3deac6a689a820205d50ef28f8629cee3eace9ea2f1294a2", "input_pcm_sha256": "911d4b4364eb05efdd14c5c796ca51a8c1c6d0d61a138d152c4a39dc09812467", "output_wav_sha256": "b4bdbcb68507e09e9cef0bfe4124311ba665d0bea5e263d8196edd7eff7a6e8e", "output_pcm_sha256": "6fb347632f025bb0b6c62a734714d40b226c4bb22a9186fa768e193985b9097c", "result_sha256": "5ba44a705a6ad02cc833bcba12176a8e3e238afd8016a4c5eef4880053fc7312" }
#82formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "4688d32a3de183ef787fe92ab8e7cb6cc778b4256896aeeb93c7e71284895fb4", "started_sha256": "3534663cc4bb7f510742a03771c529f9f4612629801b3d0157e0791aa67f775b", "original_grade_sha256": "fad52ebc6b2e78ddc3f92f52c441f7ab042ef76d9611ae3ebbcd0c693c8fe704", "input_original_sha256": "4688d32a3de183ef787fe92ab8e7cb6cc778b4256896aeeb93c7e71284895fb4", "input_wav_sha256": "70b3c37d5d44a6b7ca91b6fcf7bdf7f8f39d9d1505150f8d6465a1dcab03d726", "input_pcm_sha256": "24e4fbabfda4833b52d03ca43ec02187bbb6cc310b20276cd9ffd95b0f6303bb", "output_wav_sha256": "1d302a4e4e2adc770d5498e6fc13d944983860aeb59ca3ed5c5a994a0ae4c623", "output_pcm_sha256": "ab1c48f0dd5a816fda48ae10f7de7cab34addf6ebe934bc5b5a0c6761ae99bcb", "result_sha256": "b55f6f2b542f02062f1c2fe6eea2c9cf812a9b37a15785f4c09a73faf4af34e3" }
#382navigate heldout	No	That answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "d34b96f1c67cc75c7984179cf351a8c65cb49c2f4a6d42fedd2b0fe2b66a05ea", "started_sha256": "af5a150aee362c5702cd1ea76c5fddf0863c1d3da366c6f9bc165c042867ae01", "original_grade_sha256": "b9acdcbfbd96a8f49f534ba5784f768367bf55d0a464d6d42ea57499c3b02036", "input_original_sha256": "d34b96f1c67cc75c7984179cf351a8c65cb49c2f4a6d42fedd2b0fe2b66a05ea", "input_wav_sha256": "5f0fb45ad8c7fc8b3b0945f3ffd75ce38822fe95784614ec21995a2b8612a2b9", "input_pcm_sha256": "8eac352a79f7204b06bba12d177d85465b55b9a06ca35b74ed205a292b443ed4", "output_wav_sha256": "3ba29ff0146f2700b5ddde097d13101bf9066798da1cf89e38a09ff3541afd60", "output_pcm_sha256": "2b14a8f269adf09d95160ffa063e6eeec9fb586c57fc0580567b3ab0386eca81", "result_sha256": "8c097b48eefc82282b2cbd403ae2998ddbe474677e919ad2eee228221741d957" }
#625object_counting heldout	7	The answer is 7.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "fce32a3b7e433f69c72e0154ad5fb59ee01f5b77e4caa70f42f6e7b7fc574804", "started_sha256": "97650db55817eb1b3f1215b44539858f388eb743ce728b5415979dc54002ccb8", "original_grade_sha256": "92a9d21b4e589352362796f7a9f9c16b98daae1bf80c903083652142af909576", "input_original_sha256": "fce32a3b7e433f69c72e0154ad5fb59ee01f5b77e4caa70f42f6e7b7fc574804", "input_wav_sha256": "3023df025f587a6e2e3a0ec445b8ddd9747bb61ab7d4065e2df83f5ffef6adad", "input_pcm_sha256": "43f04c310278b1257ab34841141b85582ab7e79b1f44b0cf7862d3486e87f8cf", "output_wav_sha256": "26a99173d49bdb34bef9c483eb179743a78f2a2dfe7ffde9c686b90be5b6a24c", "output_pcm_sha256": "0e9f029b6f7a9975ecee3c7fa00ec5d5661068483ba487400a52c2d0d75c9049", "result_sha256": "0747815eb3257a45336020f3efc170235dda4dbc44c90da9b0fd60203440f558" }
#922web_of_lies heldout	No	That answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "615662f5235a63df5ef5657b3b28d0b7ff819362aa1fcea14b1d6b8eb426977d", "started_sha256": "d015745488d996186dcca45e82241a85d57a08cecf79164383b7d602a3d9077c", "original_grade_sha256": "7bf6c2521d95434bd4f76369c8411c5de7e28cd11cd1f436517892a8e29090b6", "input_original_sha256": "615662f5235a63df5ef5657b3b28d0b7ff819362aa1fcea14b1d6b8eb426977d", "input_wav_sha256": "5a7b32d1e09d96af850e751a362fdc7083e15540b6ba0a799f47f9f576febd98", "input_pcm_sha256": "3d19d9e889e7e56413968327031f044e262a8822777a61a23455303daf3624d5", "output_wav_sha256": "fc11b869ac298d71c3f68d55f0eb4edb393b97d475affd89f9efaf8537340b8b", "output_pcm_sha256": "55f0d74472c24200ec755038ebffcec792093ad906c6dc19302105c6a2f75188", "result_sha256": "5d84ec3b1cc3bda0b10e7fbfcc8aa7c3ddd84f663db83cf666104885eeb3f430" }
#16formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "06d1b2ecb5179b944e1111fc715486a115c5f454b91df659dc049184a8e626c5", "started_sha256": "4258a8573f2af513dc1c122533f2db9ee937c0cfa4fc2cc7c243c48581dbe0c9", "original_grade_sha256": "3d1223a8cdb6999d21adfb3ced148c706790b83609361451cb7f322367f40413", "input_original_sha256": "06d1b2ecb5179b944e1111fc715486a115c5f454b91df659dc049184a8e626c5", "input_wav_sha256": "43d0cd84be1d38736e7992d7c9a2c510fbe399702f57942e155801e7773e71cc", "input_pcm_sha256": "b442f8b49658f15b89d76d817f6b48770cf9f450183aa6c005c64ceefd436db9", "output_wav_sha256": "c72c98625b416bc437619159f56568792861d0f006a838e288e30595d4d87352", "output_pcm_sha256": "fc960e75d20cb4d8f7052db297e4440eeb1d52f838dfa6e3000ec97f7fd138c0", "result_sha256": "62c5de3d3965866e1d692973b39f4ff12aa64716cc83b466de4c9f1f643284de" }
#255navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "581818691a2b913d13cc50c356d393bcd164ca54a89ef3890e46b1cd8b89d157", "started_sha256": "e9a7376c8db09d7fec1de49e5c26157dceb71b308cc67208b0d54b5781596d91", "original_grade_sha256": "62d4927783b15fd2d88fa20eb24e112486a97229b8024f8fda727222a964df15", "input_original_sha256": "581818691a2b913d13cc50c356d393bcd164ca54a89ef3890e46b1cd8b89d157", "input_wav_sha256": "2b2a53f2a4748d78da6b968c52d873c025decf1f2b511a90a9eb63b893278eb2", "input_pcm_sha256": "74777fd0035a8d008766319c2d89bd00deff78dd7fbad3551d364e339cfabdd1", "output_wav_sha256": "b883aa412b5b62bce760787fd68025c4ec5f96ea02cc2797f9d4e1c3ada9469a", "output_pcm_sha256": "856f762be9ce764f3a292bc2f1afb53396fc1079e27eabd43246a2667cbf2860", "result_sha256": "c113171a33295fd6ec96a8af331ab1a36e239edd129f2e4f138f91b17c171c63" }
#539object_counting heldout	4	The answer is 4.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "27049376dc58d3cbca8f723707448ea2d4051f29cfdd728d59e69404aed70c3e", "started_sha256": "1e6a6e8d68fb54924598300f3cc1fbb349df92fb5a522bb6405c0731fc9d9321", "original_grade_sha256": "34c92c973b4cac8aebea18b82df1073562b4950a5ba4cbddefd2ba8e221eab77", "input_original_sha256": "27049376dc58d3cbca8f723707448ea2d4051f29cfdd728d59e69404aed70c3e", "input_wav_sha256": "08a3ce4be227f866efeae65426bb14af6d56dcd3ab75973bef7409ef43d3d493", "input_pcm_sha256": "b5a98c5b3e2a69c5251ce5a577a2298b1d46293cc0bec8f7f5f7ae61670c7eed", "output_wav_sha256": "d5855f48d1966b66e015da0bbc7c24166fcd22abe5c4a871077c9eec904c85b9", "output_pcm_sha256": "fe90a2a81dc687eeab7df3ad0eee8a531232e0d9ae5ece5c6905f19c4c0d84a7", "result_sha256": "ad6afa67e690455ea6ff92722dda05e3eb4540f9b2e1c8921c662435f36d435e" }
#755web_of_lies heldout	No	That answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "1b47b0c4dcbb647dca4f53737124b78890a9abdfc3335cc3daf832735406fc18", "started_sha256": "fddb7efb79ec45b02ce11099d60b60f3b23c3388b88514b2adbe23476f126f0f", "original_grade_sha256": "fe82486809f7d25cd1b9cb36494daed7fb7d0e32367dc7758d712b64b85af891", "input_original_sha256": "1b47b0c4dcbb647dca4f53737124b78890a9abdfc3335cc3daf832735406fc18", "input_wav_sha256": "a326703b0725066249bc0dd3ab62c7c7098c2b1de4f815da595eb598776e3171", "input_pcm_sha256": "bd4d8d52740714603ad8b15ac929cfad822a139a75b3fe2c3625beb5775d9e2b", "output_wav_sha256": "f3f23d2e45d0675df41e0c554badea868aa65f751b1f7160449b200fb7eb31ee", "output_pcm_sha256": "80dbc2017545fe77b87d780521d28c62fa282459d4dc93d3898e5700fc0386a7", "result_sha256": "c1bf6b4aeaabb04567682341c20defc923dd9138b67c2f4b9c8d5f3c763d383a" }
#105formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "37aaeb12c50b0c6a1caff7b3917d88596e76c8a7fb24540b88e9877d033ac714", "started_sha256": "ff5ad10e00308d7e4c36ecefd3c301620d89cdf379a77f65507988256c3c5a02", "original_grade_sha256": "a7a9bc278e8ae48748df135adb7546f569260f61ed7bfe6c5bde05975c8d7fb3", "input_original_sha256": "37aaeb12c50b0c6a1caff7b3917d88596e76c8a7fb24540b88e9877d033ac714", "input_wav_sha256": "79c3262b2cdac52c53814ad0b24fe12064bd5aaa399284ff8b8123d4382b06c3", "input_pcm_sha256": "032e5d1dbfd4bf220fd04b8f9199f5a42956af3e6273dfc09ab799718f686562", "output_wav_sha256": "7e8230e03e03ae00dd6dc616b5eeecdcce2523b37abb6cecd25c70afe09005b2", "output_pcm_sha256": "e77ad3a1bb2d805ffaf43fc8f8af640a51f6a0cad2e05b7c7480e787dc21c771", "result_sha256": "be8851e805844a0c169fb27ee3994edb6c676899b87c4cce7fd1dd2d47bdfebc" }
#299navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "69fc04129a6eabee72ac398f09c36c9531168346823e00bf05a897d5a3c5271e", "started_sha256": "39f02f229f13bb74c570851548560361a39ec8824e692dcc26ae9c85b55a4a76", "original_grade_sha256": "1fd22e7a44e24dc5a15a76f9abaf87658aec941373de5adaba6c4dc29a812892", "input_original_sha256": "69fc04129a6eabee72ac398f09c36c9531168346823e00bf05a897d5a3c5271e", "input_wav_sha256": "b79a63a62f85d4ca5f10bfd59a3871a8eccb764911e1bee4e602a8f001a2eb64", "input_pcm_sha256": "10146fb59013a1778c53312d56ffe67cb11352bf6ce0b5e6b0376e6882e3931b", "output_wav_sha256": "c255ab2d1e33431e580327c711741a3e4eb00ca4ea900ff41a7efd17dab62944", "output_pcm_sha256": "e4649e6512d1dffba6d6c901b6bb40e4a49d83efdda876ad55bfb7fba147f873", "result_sha256": "eb65e0c291892f5b610a43f99e45bfa011eeb61d4e67b08d39be2d8def333301" }
#507object_counting heldout	14	The answer is 14.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "2889539d0dd140e77986f3e10a59a8057e4221f84f102cd2e87e66cce90c2b78", "started_sha256": "490d2c62b0e12f6e414696e5fcbb29832ddd7b35709da65e8bf77bce2a422bea", "original_grade_sha256": "6037a51bd7fe02d5ce62d91c4574cbb4f11e70a907a0ad2d1d365889c8ddaa69", "input_original_sha256": "2889539d0dd140e77986f3e10a59a8057e4221f84f102cd2e87e66cce90c2b78", "input_wav_sha256": "7dc2cd658f3633bbe3300816ff1c00afa61cae41e0da8bb1698c994d8f859369", "input_pcm_sha256": "1e757eb2aae1ff53653c3373d9f48b6b5e04a122917ed57deb393972c70b113d", "output_wav_sha256": "1e0dedf7ce0de40d436a9c9e7c89fdd530b13a89a2f2b04848a64444550d3cee", "output_pcm_sha256": "e71e42e3241985921ff9f6db5c9c3306ad3541c9b5b6f9c30514b270fcfc97b8", "result_sha256": "ee68f328b2e392d9e2ad4890546b140f0ae3486e656f88eeddce533f0c64036d" }
#888web_of_lies heldout	No	That answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b7a8640ec5148dc5c02a50032e53910166b4024068cec3509cda4f3d75e09dd1", "started_sha256": "134cf304a53729267779a03ecbf295ae583b4ffd2eea193578f116abc6e01357", "original_grade_sha256": "eef1cfc19b6e6fda26c0758bf9dbf5391a301ec330b47053ed08e8ab57546e3d", "input_original_sha256": "b7a8640ec5148dc5c02a50032e53910166b4024068cec3509cda4f3d75e09dd1", "input_wav_sha256": "53892388cb043bf31a7bc2173d396fda2875a87909a8c1eee82f34fd02acfcf7", "input_pcm_sha256": "f7f15be196184f86f7db7ae3899fbf50be16904399cb12a12bbf008a08de451f", "output_wav_sha256": "c0a7b955223f82428dd2a89f72ec8840aab8775710e8917bfddfecd441dbfc56", "output_pcm_sha256": "90103038283c8a8e8d9c487220382eeba7c6466d451374d59d83f81daf633b9b", "result_sha256": "cc11df77b0d7b82e77823e288174b6b7e01ef14c72295d0ae73a0b11ef8ebf07" }
#132formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "d36daf33675e9602fde59160c726ff8ef2406d01ec4e66ecb0f05dd510ae350f", "started_sha256": "693e730778f998b01e3f2336d8c48b4f9c4ab004bbbeb4277590be82257757b2", "original_grade_sha256": "00e361f35b3a789a0ee97dad17714422fc4577e7e8a5c5eb5521993e37bfb61c", "input_original_sha256": "d36daf33675e9602fde59160c726ff8ef2406d01ec4e66ecb0f05dd510ae350f", "input_wav_sha256": "e96034e8cde6c0651daaf018e227771ba7d94fdb2d666f525f2070345fe6a049", "input_pcm_sha256": "c23e7eefdb0dc2a2ec21fd03a8fed7186c74cc435904d792142a0520b0266dcc", "output_wav_sha256": "dbd8d877364a59e9bf0fd7615a7ab2a8da5b45569e74361b5bd4bcdab78636b0", "output_pcm_sha256": "15108e5f2fe89f8ea8cd5ad403c7e2f3c80b402ff82531dfc2962cc720236704", "result_sha256": "9bb96aa58226e6cf66ac2769bc1a3d6c3efbfd7ccd133cbd7919712449bb5b41" }
#492navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "85be82272ea6f0412646ca71316e7d8b7dabb9e0702c6095b4ce2f6767215788", "started_sha256": "e165dc4d8fefa746704cefa49221a3ca942b0bd3dc5b75b62b590c4b7e840d83", "original_grade_sha256": "d44e9ee40eb04dbd4aad380a8ddd97ee2c58da3b60e8656a10fe493b5e7d835b", "input_original_sha256": "85be82272ea6f0412646ca71316e7d8b7dabb9e0702c6095b4ce2f6767215788", "input_wav_sha256": "473c1ecee5c418139ce676d770979397f4f85d55fb727835be152df69069f4f3", "input_pcm_sha256": "6530b7f24e4bd8c98669010384e409aa79d1d7c499d0d5778f901b5ffeefbf28", "output_wav_sha256": "cb3e0fe482226b63f462d9a4f9120864197cce6ed2424a94a9faac80d9e9856d", "output_pcm_sha256": "03eb6a660a0c6a7b385f5f60af8f77ed95b68b676a0797e664f44b0adce22aaa", "result_sha256": "8e3cf3dd4d0ab21018ee5d2b519c68c84044ee4b37c390fe20f3567f969c9890" }
#643object_counting heldout	10	The answer is 10.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "538a6625fd410f421fcbfe8ef6d97e55eaa97556c2e86edb22b40fe5e75d2960", "started_sha256": "29733c1baf41ff12e20dd53e2d7a977d44d7ebdf4c537ff1f95064c3bf839926", "original_grade_sha256": "ee58d051bfb2e5b5d47554e05a6fec6c303fd34ab2ec2e2fd70b51074fef706b", "input_original_sha256": "538a6625fd410f421fcbfe8ef6d97e55eaa97556c2e86edb22b40fe5e75d2960", "input_wav_sha256": "3541734d2a1065157aafc933232146362a0094180900dfea338832712829ffd8", "input_pcm_sha256": "52dcec0161deaa7812176181d17cc262d363a72f1ee1de9bc3c54b4b644c0dd8", "output_wav_sha256": "f031af7668e5c1c5d192e5ccd3161e10f1825f538486ac531ed2d4bbd719f22e", "output_pcm_sha256": "f868b4262e44f75398025c39ffdad9a9d468ba4363de120b3029e048629237f5", "result_sha256": "c20f032d5cd1ac7bd20d2a095915031f3ffd196b99577f01c361b02190ce0181" }
#988web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "22b170139a04e693ca6f26bf90268efef80ad3be5515b59b46164a994b3ff813", "started_sha256": "3f51c06a83df108f5c45b7eb701c752247f3d1faaeaea73a59b80db6a6958350", "original_grade_sha256": "c1fd823553d2172657e5262c49e3c85116f6b1715bfa7a99236cff2ca8d2d30f", "input_original_sha256": "22b170139a04e693ca6f26bf90268efef80ad3be5515b59b46164a994b3ff813", "input_wav_sha256": "84542d3b2b28f3080b39aa426c7aeb4684ebc707127923a8f2244015cad21f8e", "input_pcm_sha256": "bfb90380c0bf3bd1a000bfe2327a7366788ccadd5a2cc9853d8ebf0cc028cf39", "output_wav_sha256": "312b8bb511aa34f0cf7067be88b4c354f947720ab7111f508f36085aa7be143c", "output_pcm_sha256": "895c4873560afc0ec8c23e94d53d17fb8db751e3f7b6897287a585f0681da368", "result_sha256": "ebb6460e490f02a8eb02b70f084e5823541e11ef3af82e845cceac4ba96b97e5" }
#32formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "2c96c5dcb4caadd96a97c663be5b02750bd35d432604bf7ff2ae0a36c8750319", "started_sha256": "03aaf48230d47d1b78e55ad3cec92fbbd35c921d5cafec2ebb999d2cb3746800", "original_grade_sha256": "2bd8a224c63415e1a392803cc9bb15a420e3add1317cfeae718bb18265b6a575", "input_original_sha256": "2c96c5dcb4caadd96a97c663be5b02750bd35d432604bf7ff2ae0a36c8750319", "input_wav_sha256": "1ce20bf0fc50c5269db6c76c3222cc151c7b7cca86d2ef62f78bee853cd43116", "input_pcm_sha256": "cb689eb9dfcd58ff33971546416beb14185f05e4084c7757b57c46931d0fd75c", "output_wav_sha256": "e36cf7597a073cef0b4d245cef6f048915a7641928e96f514802e3fef2cc6cc4", "output_pcm_sha256": "32af993437fe8ac90285202fd73a3f2da0da6cfd63eef9fcdbe3ec31304a8656", "result_sha256": "9ba0ac54f00722d452666374b57a353f5608aa9d21217c31e7960802333d76f6" }
#318navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f9bf503704d124e095e408835c776ee8d4b0a190d99b6dcd38fa1cf17dd7b63f", "started_sha256": "5daa9ae3d314705e5cc3f4c0eefedf265c78f2bb59cd1bcabb26ca9db46cbb32", "original_grade_sha256": "b5484a1663e827e64e1db2bd3c6ba439f9a3d24a68d5ab6900fcef425f5aec6a", "input_original_sha256": "f9bf503704d124e095e408835c776ee8d4b0a190d99b6dcd38fa1cf17dd7b63f", "input_wav_sha256": "2eba6ed2eade043026dc76d2249d1bee6b710ce97f2bcb17d40e1bf30d5e463c", "input_pcm_sha256": "62cc65a551bcd6423c61a9ee6448e8e38bd76639bee51d5d6ecff42a7389f284", "output_wav_sha256": "3815732bb941cf6254f9f4669c9994df6818b6f07a4abf0a12cdefd04d31a44c", "output_pcm_sha256": "93f8cbf3461b6f6d8d9a9173c226eb8067ea60b7a7d7371bd5dc814a83785abb", "result_sha256": "1aca7d788f42b16ffebe8b563766a57898a2d7013211eb5a63087fa33fd29c65" }
#693object_counting heldout	6	The answer is 6.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "2d7730691d22c12453c5dfe660a3b6ccc07a216c0a223813b79648c56b3ff8b7", "started_sha256": "05f4906726ad3b0a3546993b7600410755b85d374cc7685c940458e7c993bfe6", "original_grade_sha256": "1e9f2bf2e256361261792987f81f7fed2770a7c9f6d1ab2a236c4b8f8a584a6a", "input_original_sha256": "2d7730691d22c12453c5dfe660a3b6ccc07a216c0a223813b79648c56b3ff8b7", "input_wav_sha256": "aa9497839bf32cc931e04f00f0ea91a2dde536cc06a10f6334eef3a288a4a8d4", "input_pcm_sha256": "39fa6e564d79d34e26cac32825a57b990c13a809e9a4b23e565a457a419bfc2d", "output_wav_sha256": "9a26efc9f7d44db9a29dbd1af38e8f700a7011183b41cf6541bf3cf6d1b82ce5", "output_pcm_sha256": "d6d67c48108663b251b7ede96a4a9dfbb4bf4f241dc10b7643f05a5dbbea44b1", "result_sha256": "947641c0df09add9b6f82700a1c78cce2358e90c319dc1539a6216abe0ef14cc" }
#762web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "48db0fe5abf5c476f69337ba8e1adacb0f867410cc00950189540f4f498c4192", "started_sha256": "3fe1ee21e8f9e776bdcf65aa402e152bfb706116e8aaa75ce32f437d31bde948", "original_grade_sha256": "901f89985e33e733a502064f0d1f4d2b979272b17ecb840a379aef70eaa88c61", "input_original_sha256": "48db0fe5abf5c476f69337ba8e1adacb0f867410cc00950189540f4f498c4192", "input_wav_sha256": "52948b86ddd89683644a862b01138a5223198aac1d38e38d97fd578a09237e64", "input_pcm_sha256": "b9b89dc92db1c7282bf60dc282f17ee1c5af826928940b17f948098f252e8619", "output_wav_sha256": "5ffefc09f9ad42f3347d0b08140c1a6cc9b5613c6dd3a533c33f5284183d419b", "output_pcm_sha256": "302c2b1721abee88a322925ea16a32b43bbd36586ce44b37ac9f423a94a3ed09", "result_sha256": "41407be0dabe2823f261882668bb690018e00b9f78e93c00e8e5977522a7e1e6" }
#159formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "754519a1c91c16660c2dda4dd490fba450afde295e2e30c88dfca47bb6ef36f8", "started_sha256": "f83a26de8cb70b0b2349a1e753aa501daea4ddda944888cbe7ec5df7b360c125", "original_grade_sha256": "8d09d5d6d7c199ada268c0b833063e773de73ba60a106d855868d4dd47a28bf8", "input_original_sha256": "754519a1c91c16660c2dda4dd490fba450afde295e2e30c88dfca47bb6ef36f8", "input_wav_sha256": "d54deaa755df91302c700c92826d4b18b6d234c43eed2629e1dc7fe6997be8fc", "input_pcm_sha256": "b274393e4d3dc5a4b63dc71818fcb61af95399239bde8daaf60e64b27b636d5b", "output_wav_sha256": "59000118b87dd0e1d4526e05dda54c54541952afe8d60fefd590aecc8ebbf766", "output_pcm_sha256": "c90a557b1a35b5028853791c8b69c3d63443bf379a16ca4c8366b3139bba51a3", "result_sha256": "42b74035ac272555bd722bddd220eaf80765bef420aca81e0f671d0f4da0b3e7" }
#258navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "9045d16c23c995b2350ebbb4dbae713c6f3d64237abec039a0ffa8e72ca478b8", "started_sha256": "c2c8f4256acb4396ce19d1573ec6a763c5699438f7815591123144c40019e898", "original_grade_sha256": "5f3941f5ada2112478dcf1ac82be56110762cb116809d013c9c2abae2169e093", "input_original_sha256": "9045d16c23c995b2350ebbb4dbae713c6f3d64237abec039a0ffa8e72ca478b8", "input_wav_sha256": "7174f793d1be10edf23fd85dc2f4b7f5b1918ca9fd5eb5ced517de10f345c896", "input_pcm_sha256": "5fdc90273cd83161c7a90e2e6518e75f31a6ab570f063ef62f7cc57303d5bc80", "output_wav_sha256": "6dc0562b19cf3bbe536eee1bca3d0a710befbf5a280d47bce34df9711b95a8bf", "output_pcm_sha256": "0f7721412e9cb59c9be522d9ec3acc7131a3f22ba4f724c45026fdac8a91a667", "result_sha256": "ef0b66df84c8cd1b7053bc49f9a9c2e77509c7e51790d2d82f9f413402114fa6" }
#689object_counting heldout	7	The answer is 7.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b4037c32ffc703bef55363fd483da7703fdf51a6bd6204fdc8eea2a1ffc2966e", "started_sha256": "3cb37f61e39a96c4617cf4324f9dc1a2c4b218904d7c51e81d221a6ff35f33c1", "original_grade_sha256": "f96ab1a5a0a0f64edbda1ca10b46c119e86289dd20e2b89a133be4447e97eece", "input_original_sha256": "b4037c32ffc703bef55363fd483da7703fdf51a6bd6204fdc8eea2a1ffc2966e", "input_wav_sha256": "0d7aba80f0b48ec4258e0ff990129969b6e189715206263137a616c3722e82c9", "input_pcm_sha256": "b36c040b4024b598f81de7e847793f3e9f3ccc1307026e526dc66a27f578c7d3", "output_wav_sha256": "fbd9605952af3138f1ebaf323aed86398e2873023397146bd61cfd867b1edf7c", "output_pcm_sha256": "8d3289c3e010a779a492a9770a20d3304cc454fc55371c595e54c2814ed235fd", "result_sha256": "27fb891ab443a59a29fb751f8426559c67f5a4858df3e863f83d85a8d48e419e" }
#841web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "1f724f71a6e9014107e1c1e2660815853cb77a11406a5716b89afba49f50eaf7", "started_sha256": "3ecd8fa3b391281a176c1bffd64bf9a70ea0acc1b3776944b90f66056ecb2065", "original_grade_sha256": "b190414eb3128022782b07d3a3ecbc2cd26bdd90c35833fcd467b979b1916458", "input_original_sha256": "1f724f71a6e9014107e1c1e2660815853cb77a11406a5716b89afba49f50eaf7", "input_wav_sha256": "4adafc91a57ab9dc038a5253a8d1c0a67cd0ae7d6cb1122d391d45b4f6aff478", "input_pcm_sha256": "aa082f3a566b63c22bff566798d613a0a09700cb9791064f3ec9a7b0a6e482ff", "output_wav_sha256": "190d97b0b066174d0240a473d68f96521fc7959952218247c89115ab6fc1a5d6", "output_pcm_sha256": "c78e6185e146a51f604387488282e1065e51d722237c5561e35e4b185391a65f", "result_sha256": "bfb148eb690f34b38e498ea45c56980f1faa3c8f7e3b5d5424dbe7d382b776a6" }
#182formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "30338bb221ae8500909498205eec929fd88b66cecec961495e337d806015b928", "started_sha256": "808f5c7fa3edaf98a3725db4e63c683b509a30c8d31e4c479e4d2ee7fb1d3648", "original_grade_sha256": "b8aac0d581da2326604bcf88fef7522d97cdcbc2c4958f797e2c17e5e64db7f0", "input_original_sha256": "30338bb221ae8500909498205eec929fd88b66cecec961495e337d806015b928", "input_wav_sha256": "93fd810daf3fc89ba532b66a615fb97251e6555caac8293c19c7a3711fb61094", "input_pcm_sha256": "d49ff30e222d72a22609a4420a991426652de96f7daba007522b5ef24d7b7923", "output_wav_sha256": "dbe397a3209fd24d8760b22025c6abcef8e66342abb1842dbfc7ae1fd6b40b84", "output_pcm_sha256": "b8fe15c519a64f6edd03d4543f3e48f39fc2437057a457b5c215650da30ee2e6", "result_sha256": "470a919e570e63e7b7c9665728310a21d4b370cce2748369b91427f22bd29d90" }
#487navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "1a9eda1dee437ae669c73e9caa823caf23f236fa460f18383f0fca6621d98c02", "started_sha256": "3a77a0a319f998500dda2e5598f503f473a67736ef3a2501c94e78ead660665e", "original_grade_sha256": "d75b64e72b353bec45103ff41d84983320a54c02960ae64f79fc14223ed92d6e", "input_original_sha256": "1a9eda1dee437ae669c73e9caa823caf23f236fa460f18383f0fca6621d98c02", "input_wav_sha256": "432a4d10f0acbbb5e8e974ec3a4415ea8cf0a92de7dc1b87bf4e2b4b48cd3cbc", "input_pcm_sha256": "5a48c491b6dded772cbf5186ee07889b440f66407b304da43d71841d860f8dd1", "output_wav_sha256": "1709d0db087a0136314bbc2a92fc0f3713879394105cafa6e7df48e2cf05e8ac", "output_pcm_sha256": "94c163280cc5ef51ed3f479e6066e2bdf134f1704ad2cec328dcb71ccc43f316", "result_sha256": "b6a2cb392455e88bf4da69a97b401982002c86d1b0bc970f35a5ef4f5fb83137" }
#567object_counting heldout	9	The answer is 9.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "3ec1e55209eb6670a69f92fc67fe57afb8a713fdf9bd0ece101d2c259f259c17", "started_sha256": "d11a8cf9097469ff2824de0aa2f57602bd2f00d4c4eb3d77915065c1810f8ef8", "original_grade_sha256": "a2e9338e96264afd0d52f0884a88573948eacb2fddc4fedcfbcf97bc0fefdca6", "input_original_sha256": "3ec1e55209eb6670a69f92fc67fe57afb8a713fdf9bd0ece101d2c259f259c17", "input_wav_sha256": "1f52af2fa9b9c5a4f7bcdd598b6108168ca4a9de0a20be524617c22baeda5a8a", "input_pcm_sha256": "ca0fa668a9f2ba84e55abfe4d055726b40901024ec03145f7509dcce54bb19ee", "output_wav_sha256": "f7c17e70fa90b4b5dadaf07c8eedb1188fcd3bafd4dea4122fefb74d7ad01c9a", "output_pcm_sha256": "757b814eba135e855f5297b5d1f75663626beebd0ab255c76e23489bbc9bc058", "result_sha256": "e5cbe0937156b271713977e8d387a3b5853c208fab67d3117f350428708ab46b" }
#940web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "cbade5cbccb6616c315d3ad2764f0968c70f0367e321c2dac28c520c8c80e153", "started_sha256": "12f9ae12e38cb815a597e844f482ec5b0ca1ded57dd2edf0a282922b871cfc64", "original_grade_sha256": "02f0a32c1c6576f71712a3551dac3cc80a67abeab388a7a161098b67f8f3a19d", "input_original_sha256": "cbade5cbccb6616c315d3ad2764f0968c70f0367e321c2dac28c520c8c80e153", "input_wav_sha256": "f7599c2ae85828e64f6a6b11bc1af121637ba315d782dde5c179cd99b357ab8f", "input_pcm_sha256": "ac56b23781167469e74259fc18f17d9001a9923b239122be361f1aa3950e69fc", "output_wav_sha256": "b194961b03039d8f97cd935a5dccee7d327c77666b68561530cd7a1c28256b17", "output_pcm_sha256": "8930d866693b05a85631c84e434be59053bf8b7a0ecf29c9b7076d1d651089bb", "result_sha256": "953b33a14e4ca2092449f413e36f294a46b63a1ac35cd5a2a44c16e5087ef269" }
#210formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "7b48f9d98784c488771207be2b6265ddb7941684233cb7de2cc1a69534be3617", "started_sha256": "f02e465080b39a2b4cbe7f4f534c966fc9e4dfd0b794638e64ffcd95669070d8", "original_grade_sha256": "ce607e1bbcfd3720e045d5cb14cf3becdc06cd683426aec4fc766ae181fdc5b4", "input_original_sha256": "7b48f9d98784c488771207be2b6265ddb7941684233cb7de2cc1a69534be3617", "input_wav_sha256": "c92d9c9a0f59e264ff1a81d3038866a06b93e2ab014bb1fb8a1eeb1cdfe1d6b4", "input_pcm_sha256": "ec7b4ef7ea4794028c505f20086edcac8e74fa64a141163a6c318b900f2a9802", "output_wav_sha256": "a715bba811c82ce5ed2992dd60fb49d5494aabb50e905cf5c7c5a3afefd746b5", "output_pcm_sha256": "2684168b3fee6d69fe1c2eb93b7824b2b809817c0f94e76b49a75ec639388a3d", "result_sha256": "55f9e1c6afd1ed3ca58a1b76c4a5725a80c60376611acd37d2a19b1ab440f961" }
#488navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "74daf6c059e1f35e0609875b6b610857134fa6e41d0cd9431b9b46f5cac1be3c", "started_sha256": "cddea026f2c21386810ac2aed5c9095f72a550371fa99bf04947d4cd5d58233d", "original_grade_sha256": "2185d712eeba4d8e94153b4b80e1c7b2082fa9b87198c5607e6400fef16826a7", "input_original_sha256": "74daf6c059e1f35e0609875b6b610857134fa6e41d0cd9431b9b46f5cac1be3c", "input_wav_sha256": "ced128c4b176d0224beac9cb9c278cff51ddcbb268f64fc0813294d378e72077", "input_pcm_sha256": "895c6223d8e5fa8db165fcba97835e16eea889c2bd9527facf454d2f73810dd8", "output_wav_sha256": "aefc778b31c6fcfa87f1baf7c4ecf4eee2a43eac93560b2b7facb2ec2a75e241", "output_pcm_sha256": "d53a2012287e458563270245f10180c415c31b01fd69f925aa52e489e6a2fc1f", "result_sha256": "0d101091a362eb30d43b056bc4f23761bfeb1e6846458296dafc60bf36b926f5" }
#672object_counting heldout	3	The answer is 3.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "8e7ea82ee078603825bbda861498d5461154ccf6b993315ff81b7624e168a501", "started_sha256": "9da3c54a51b97952512294bc9c35de469649dff80eb57c773ec06685a87a3797", "original_grade_sha256": "103d9056a1cbea6720615687e9c66bbfe44973c2a66cfd6ab448608bd21aeee0", "input_original_sha256": "8e7ea82ee078603825bbda861498d5461154ccf6b993315ff81b7624e168a501", "input_wav_sha256": "d24d9b1658acc1a57d901be0bc5abb4082ac916cbdbf8e6a7ece55b1aecd1c25", "input_pcm_sha256": "9aa9d8010216658c5be9cdfd8ab26876e1125cbe77f3b5828e509127459f9f37", "output_wav_sha256": "71fca9243413bdbdb432951642ece592051fb2d2642a9a0f2b876ebda719d75e", "output_pcm_sha256": "41ea6fd1916e95649e794ed79a9b17b2be4f12119ae5c6b952fec3eb27ab7cc9", "result_sha256": "031d11f535a45072cf37cb6338aab29e178d0dead6754a9d2e83af2f5d3df6e4" }
#878web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "18e05062270b0250ae7ffb792f7663d218050fbe291f4c11db17ebad7c096796", "started_sha256": "efdee306c6d71c2c0254752f11573a11da024648819461b7bd6de3d0458c409c", "original_grade_sha256": "b35fc69b3aac84bafcc793cd9ca2b091894ea1e8d1b39801fab9a82d9f19a89d", "input_original_sha256": "18e05062270b0250ae7ffb792f7663d218050fbe291f4c11db17ebad7c096796", "input_wav_sha256": "402b35f6e7b236455ef71ee6ee03d0fd910dd03f83ed0fbb8ce97760a02e1744", "input_pcm_sha256": "7a9f88dbea1809c503b326069c290d3ba5f98b7f4db5de890bbb37a7db964c86", "output_wav_sha256": "1c2a0e0c985d6f4c6e2afe398840de3cf7c38e9265d774a2184f44013f221af1", "output_pcm_sha256": "0141610b9542aecb698ce473c36d1c38511d0b2c06693f3617a6f7d3cf32beed", "result_sha256": "9d7e7aad1743977aae49451b416bba4e6d76b8e6f8d4cfaf78b5a372ab816749" }
#20formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "9f16da55d56647eadc9b52fb94115db0f0dbe9b5cc768552ed8e5de12784bdf2", "started_sha256": "ef99f3ae32444cc94c31df055ca7bc7ad9648b330bfa56acf3d3bf1f60b84964", "original_grade_sha256": "08f1e1ba052c839d741bfee0a041dab1456a41066dcefdacf58e8bc27f8c502b", "input_original_sha256": "9f16da55d56647eadc9b52fb94115db0f0dbe9b5cc768552ed8e5de12784bdf2", "input_wav_sha256": "0b2928e4476ebe138f1d54e88eb4c2d4d72c10eac8fc6e71891c5ea4e95125e9", "input_pcm_sha256": "2d8e01fb7f44cd22f6411d434f08deb9d8c65dda25c8343a2f3057433df5abb6", "output_wav_sha256": "1749ed6e14658df7135906fd6e1337fff2cf33f2522ca43c56aecfe4683995d4", "output_pcm_sha256": "3b241c3b35a568c9e9c294f261c3604326c33abc1d1c01809a0e4b7b91431495", "result_sha256": "e3c689e1e08ebdd9ccd6564256bb6af19035fb60c1fd4277a77f010e9161da85" }
#274navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "87e42818c671bca0d23dea4e0e7bf2106d61a6b2391843dfca2f0e1819bfcb07", "started_sha256": "710c1edb31bde42511f1a32c8cd29c61fb8870d6dc207d6618fe6844771fecbb", "original_grade_sha256": "439f4aed60107a3b6e249888cbdc0fb549f1b33ba18fff1382ac6db257f501f0", "input_original_sha256": "87e42818c671bca0d23dea4e0e7bf2106d61a6b2391843dfca2f0e1819bfcb07", "input_wav_sha256": "5267ab3133c0426b83b0860faa2095bde9a64b27174a5c422f0cdbab60390914", "input_pcm_sha256": "0c33db8b5e5ec3edab269b2bad02a7603a6111c11de824b9b1367eaa5187f565", "output_wav_sha256": "c3df40f746753ed19bcb56c4f77be82b8139d1c648d4c8c4836c7a1b95f7b00c", "output_pcm_sha256": "493c7e5b9805cb6019b0b686ec4fd50956d4a833f0aed0b4efd96415108c6803", "result_sha256": "4295c1d79f67a1bc0991e29be6cfa74abeb00e8778de25c64430e6770bf81fe3" }
#668object_counting heldout	3	The answer is 3.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "4994558453d7bc347e32b44f6dd4ae19836cee73600176e6f1269fce63a5b940", "started_sha256": "ff38d237ea9a8f4a7b6e92f48b445f0aa3ed99e4b14d3e8a588c290e646fac6b", "original_grade_sha256": "37315302cc48c6b11e5616143c944ff800c09637b98605a450a358e20df7d020", "input_original_sha256": "4994558453d7bc347e32b44f6dd4ae19836cee73600176e6f1269fce63a5b940", "input_wav_sha256": "530452df6154300a3a796c859c316b682a00a4bbc1ec1cf877195cedd44be02b", "input_pcm_sha256": "7ae4e105fb2c588b90d463f915d1e62a5ae883ce27e9d7c3b54ab9d686ee0dfb", "output_wav_sha256": "ce364190a5a7a187dc4139b440dcae4c923d9e8f279f65aadcee20624e33e2b8", "output_pcm_sha256": "dee5d5657ae4a32ed5bdbc7b261bd312b015e91d4160cec3e8c4fd1d6fe7ebad", "result_sha256": "e380cd96d41d7cf145c3f70330a950b136b660d4ccd78758f9b1860b2b22acff" }
#975web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "476df11e11d09de11da934d6eeda885898fb5132754159742e7fc9ba0771fc16", "started_sha256": "c842bafea15bf82039ca3d00949e1e328d423ee456e5b2301a954583cf55ecff", "original_grade_sha256": "09ea14a55bf919831d25a6f2e8b40f891e770a3f1a009afb2bf6ab74019d517c", "input_original_sha256": "476df11e11d09de11da934d6eeda885898fb5132754159742e7fc9ba0771fc16", "input_wav_sha256": "89ab6ac02c8834e19dcdc723f7b454089b8daa3c429cf41fac78ceff8fb8d4c2", "input_pcm_sha256": "81e6bad0a7dda905f171bcd8e6bd71dc8e0d84b802ee7b88b479fbb998827574", "output_wav_sha256": "973777c301211f596644da6445de2ad82143ee0c0593ce1c41add759d5ba0d65", "output_pcm_sha256": "9c58db56bb89a7e80c7169d4875ea6d693e6eee61155a93ae039ff5cc88f463f", "result_sha256": "bbb2faa0e0a58840e6633e850c9c66fd9162979f8bc486d01e8b80a994eca8bf" }
#10formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "8df48662980e2e5e6e54b87aa43fea522fd06b17ccfec95c8d71dddb7485ee26", "started_sha256": "fefacf3088f2c295fbd02749c10151219f55638c3e21a23f46165127ab26916a", "original_grade_sha256": "0002098d6efd34d0720b0270489c565454432a5aa316adb37ef63cd56e83109d", "input_original_sha256": "8df48662980e2e5e6e54b87aa43fea522fd06b17ccfec95c8d71dddb7485ee26", "input_wav_sha256": "2ad1cb2df3c72d63b4148ffbdf4e70fd5a291596e6d037f78645f9ab2f782bfc", "input_pcm_sha256": "52a698c9dd564212fea7d6fff1ce3924c90275665c9de5e6aa7fa29ae4a323ff", "output_wav_sha256": "eefde2e01e60ed1b694d45136f2cbd05853800b4b45ae476655aa95fb31aaf27", "output_pcm_sha256": "09a85a3a8ee6b5c0a77f51e52d3b458a1678e22035fb8eee2fb2eedf7bb8db32", "result_sha256": "b20b232076787c021f539c720c0996d18a6660b2c966df112a95621af4778b1e" }
#386navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "c3f2319ec9d900e5b427cdfde85e2dfee9f698733c83aa513878b6987f6b2ad6", "started_sha256": "8b62d778c8490f4616daaa78c71c637cc2f71f3f4d98db192a431b246f4a4c57", "original_grade_sha256": "8ce6b22cfb4ec01ea89000d5d1912ba0c781108d110c67571b940d077fc38b8b", "input_original_sha256": "c3f2319ec9d900e5b427cdfde85e2dfee9f698733c83aa513878b6987f6b2ad6", "input_wav_sha256": "42b891024a81ac3e50246a433f4e49690f1866353916671547be206d6ba034cd", "input_pcm_sha256": "0a4b21c3f6bbbbb835976ca39ee6ffe047c301e8c1024fedc72d91be19890494", "output_wav_sha256": "337badfe89791624ea41f2fce5ba0377d7767936a2e374e9eb37b796dd3d07ea", "output_pcm_sha256": "fd2f1bd98efffbe511dca80aee7efe191b8ad022e1bd586c6613afd6e2bf6311", "result_sha256": "673170aafc170f2d9c66dc57220de70dcd0d616e6c0b4bcf74e5036bcbc238f8" }
#711object_counting heldout	17	The answer is 17.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "4597bc758cf4096d8204ab8bcb2e161c9e04423c0aff1a71d6df0d46c620fc66", "started_sha256": "f763dfb382292c9cee7be6103e32ff93dcb13f99b8040024ee0fb2e12bb07e89", "original_grade_sha256": "c22e4401758b03bb41123b21ecb1befb26acb3acf663d1458dc409d29bafca05", "input_original_sha256": "4597bc758cf4096d8204ab8bcb2e161c9e04423c0aff1a71d6df0d46c620fc66", "input_wav_sha256": "19c04160cb98496dbf6a6b3d942e9fd0a3c9b3b063125f3fc97141f8fbdc0136", "input_pcm_sha256": "1b7c457908079b152375dc66a5bf26212169cb60c4156d8303c83fba9f2470e7", "output_wav_sha256": "d232f2ddacbe3a88b21e7881459d6ddab7cfc86738acfd429c2ba0a79b614bc9", "output_pcm_sha256": "13b3788ba8e585f451c309064336025b698b6237132aaf2ae110b1cb96510567", "result_sha256": "62a6bf1d42562463d0a7a75277181f1cc17a187e337c015a4d9ce3ab0ad03f85" }
#870web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "bffc8a7f50e078873e025fd1b405ce903d2197f345bbee99d14662a32c603abd", "started_sha256": "28a46108a1aeecb9ea9d752d868f0d9b1bc04abda3f2f2d91b860dde895d07ca", "original_grade_sha256": "216a3b5dedd8fdc28b89a2562ff0694d5b0ed2516108156634c6b7e67da86454", "input_original_sha256": "bffc8a7f50e078873e025fd1b405ce903d2197f345bbee99d14662a32c603abd", "input_wav_sha256": "93cb01db7be970ec7fc4a00976b4df1e1b64fa8a3153f1396b386981199430a0", "input_pcm_sha256": "c78f2d782a6917f8488e823ed338c0ee39838b6bb5df4dde596823d7c472614a", "output_wav_sha256": "934444a41c543f80e1923383920e604222cf80b356c4455c9672e40edb66bdb4", "output_pcm_sha256": "b2262b2490c6ae6f4e7abdb08c64845e5eec412e4c1630b283b1b3afeb12b094", "result_sha256": "5e0dd43c2eb9db74b8fbd9e6272eb0c12ac3fe9538b429d93a9c55c9aec4bb9d" }
#94formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "83525fd11160fa41e23a5bf12d90b4a363caafd690b57097a6143f29e2bd8e44", "started_sha256": "6c74074f3c75121a7512688189592a7b5f693c50e1257c41f27e609787ae040b", "original_grade_sha256": "bebe08fa903dd5a24ee871bb709b8d7d5aece44155fcdd7b9335ada869845511", "input_original_sha256": "83525fd11160fa41e23a5bf12d90b4a363caafd690b57097a6143f29e2bd8e44", "input_wav_sha256": "cb2e569f4677561dfa002198ed6679b7f65128c50f1cf06a2333e7a7b4690bb4", "input_pcm_sha256": "86fc32ca5a4ffb686fa77b275b94fe307dbbc94eba92d69fdb9f0e19e03b6ad4", "output_wav_sha256": "c9c44c233c6247176ec2d8bedb9261e5bbf08d2cd2bc7e5575ae89ddc782290c", "output_pcm_sha256": "7d9e35a56377df07cfd2b1624818e9ad3962ebe6d99576ac6973d23b7aded5f6", "result_sha256": "d1d79f3624f87fa9a3c6eb44971b3c06a896d1aee2ad04f568fd7e3f8a8e7845" }
#449navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "451c5ba60ee0d9e2158ee945d0c99d0bf881e5c96170d6b8f9b467d3504ca2dc", "started_sha256": "f96b10d8f5131f568d7a81dba399d4b4f35fea86a3f93ae746abf555dbe13e4b", "original_grade_sha256": "588c4e39051e27c000d3f57a6472450cb9647b961832704587566580af29a1b5", "input_original_sha256": "451c5ba60ee0d9e2158ee945d0c99d0bf881e5c96170d6b8f9b467d3504ca2dc", "input_wav_sha256": "8da4bb21787e6b02862f0acfbb98bce29846b72c877a7c9a8a5cc1d838c588af", "input_pcm_sha256": "dace0147f5d3a79bbec3fd92d58599ae84f52c940001a5bc188300b9ea1e45b5", "output_wav_sha256": "b6279bec483ab91b4daf178a9f752d63c509af2c94e9f5f6a9e44d92fa5b7a00", "output_pcm_sha256": "a98eb70c759db00459d209b1a4363e44abef993344150592395e0972c3b8d0dd", "result_sha256": "d2642d1a44a31419a316f85157060daf8ed89b9a417c9ed739d8a780f4abd74d" }
#564object_counting heldout	11	The answer is 11.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "e8c80c2b4d4c9d9ea2f7efa4a2dd9698b93e19ae057adabde76773579debc1f3", "started_sha256": "c04c45277994f4989ff16c7d643327d26f9cda9fba9540dd4e2e65ec1a7c9e72", "original_grade_sha256": "033654ac8a8837c3b98c440aad901ab58dff4341dd1cbce13f90f647e7c2b29c", "input_original_sha256": "e8c80c2b4d4c9d9ea2f7efa4a2dd9698b93e19ae057adabde76773579debc1f3", "input_wav_sha256": "13ed9adfe4941639dc8ed5c03cb2c1b38f6c92401599b8875384c7cb2c24b821", "input_pcm_sha256": "7d3b5e0cf229294eb36cefd4c7eed06091a64b37fb59d360452ad8e55dbe528f", "output_wav_sha256": "d65d0ef621679ed03a9a8046e31522a5d4efe53810c6701362022ed089ff22fa", "output_pcm_sha256": "8121df3aa55e02d8eb0e65ed6aec020b11d351685127a62f612ec0dc9c93b8da", "result_sha256": "ccf98ab6816cf66c15967d9e3f4794e8be0a62be332c6e8f06a63c7419b7890a" }
#800web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "3c4b548d5f6964da9b2c5cedabd01be4e82c4c8b6d08e7483b0bd05f65395168", "started_sha256": "187c2c2ba456ccf72ac5ddc01b139e1609036d3124c9911d54ec57a71eef1198", "original_grade_sha256": "e0506a33e6e3fb1e5bd16f57f6bf1ecf57a6f636a214c6ec52122f65335f6371", "input_original_sha256": "3c4b548d5f6964da9b2c5cedabd01be4e82c4c8b6d08e7483b0bd05f65395168", "input_wav_sha256": "96219a11cf10829b9d588a38089c39546f258d326559213bb3d3770469946a11", "input_pcm_sha256": "e0a07bce839071a8b52cc7b64098a3d2b4b73db73ffdf82d0c37da6a309d58bb", "output_wav_sha256": "f8182176865f2af4b4b5aa192a0aae588b4df81a09069e28791cb0d25768c6b3", "output_pcm_sha256": "93bbc89704f7b74f06704d81a208f465b71631f72b7594869166ad8b2e9d60f6", "result_sha256": "ce31c09dcc4f5415962c9bb6663346aaef7b3c749c39436c29880737f4cd3bc9" }
#214formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "a9b409461d0378f6fe49b2c396832464be25c17eca36938c979e5bb1b4e09655", "started_sha256": "70ec7beff3c2c60130608945475888409b4b15427e0df0f9b0443c381b198d8b", "original_grade_sha256": "6317099fdb2749b131973da5485d8d137e36d6024c54dbfef4e1eb95df9c19a5", "input_original_sha256": "a9b409461d0378f6fe49b2c396832464be25c17eca36938c979e5bb1b4e09655", "input_wav_sha256": "44743a9733c3f8d3ca4a69a7075b776fe153b3caec606e085571e830ca6b3d18", "input_pcm_sha256": "009207c9b492116ee17a76423cb006d5ee82df7fbac317bfe5ade8e759a7ba5f", "output_wav_sha256": "43e97beb1e5f16ddf7ee3518ffdc09a6eed56ccf935932b6e1f9f004fe8c16d5", "output_pcm_sha256": "a02a3296bdd36308635c4c54529ce25c5a2f3f143623d97f57825232a42a91f6", "result_sha256": "dc5b60b1d5d2d4003c81f130398eb88f7f5e2f027f67b7ec0d0acf0e0a5b03c6" }
#437navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "2341acd210172c07c0fda107fb051dc8425977921a518d13a275ff7ebbb8cc42", "started_sha256": "2411536ba4c6cca5d55ea7df5f81b94ae4b694d567d24f3b18293996cebf42e1", "original_grade_sha256": "b6a65aa57bb295ccae0477e4936ebb40f9f71dcb403235090a0d8c054731115d", "input_original_sha256": "2341acd210172c07c0fda107fb051dc8425977921a518d13a275ff7ebbb8cc42", "input_wav_sha256": "3691203a5ec1da8b0ca45683a34edd05ef4832ca22cd9a35d8dd6e46f5183444", "input_pcm_sha256": "d40011bc1c7d9dd463ebc10544414997233f6fb28c0c61ad671399469097fe23", "output_wav_sha256": "8acccad0547827d0d6003beae5c2fe325d2709c688db596e55f7758053e564cf", "output_pcm_sha256": "202907ddc7ae272187db753a50e0d9dd781361a4746b0393b01bf80a4bce9230", "result_sha256": "dedc91bafde9576992e1be5a441bde0039a0961b2d6c252ddb0c3218e78d2d14" }
#744object_counting heldout	10	The answer is 10.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "d78aed4850f47c00888d83e88df36e6369112fe0c4318f4b85a233f8c33d5f0a", "started_sha256": "6e525d049c52d9fb630906278fc6a014c1da3c45f43c90d4de9f968f3ba59a45", "original_grade_sha256": "97798ed80f8363dda335b586c67d974d5ed17c0f62a6210a80e04a22079b4fde", "input_original_sha256": "d78aed4850f47c00888d83e88df36e6369112fe0c4318f4b85a233f8c33d5f0a", "input_wav_sha256": "4cde7add99bec93da8275650564b812686679b77103c26d0a24a8807703343fd", "input_pcm_sha256": "cf794d3e87824a6801b625b09a6ee35ee5e8b76b181de5c4eeb9366756c61a4b", "output_wav_sha256": "81279bbb26760aae3b67ce49244453ee6a9ba6a007b5502172eb0a290f7215de", "output_pcm_sha256": "d429762d35f1925a583bea2d3c4d35a2d439ceaacd6cbe960224ed7499e5fe15", "result_sha256": "5f32d7ae3e18e65e8c9a7f91b9cf10dc34c4e54f7a7e54e94701f95ceb0d4160" }
#863web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "6d924092bcf3be07c81a3fa945c9a6fbc5a29e62737e2ce18efaa63c9b76d0ae", "started_sha256": "90789f3d9582deabd1cd568051ee83eefcc1a397bb3ddb3abc6030a1f6f7d108", "original_grade_sha256": "448a28024ea4cdcb676eb254434330a901cef9dee4cdf0560d5b500c9bd4ceb8", "input_original_sha256": "6d924092bcf3be07c81a3fa945c9a6fbc5a29e62737e2ce18efaa63c9b76d0ae", "input_wav_sha256": "f79727504084db6c768e0935f9e29275be60fe517d95244a89d10645c02c3f2c", "input_pcm_sha256": "d00f47a53736c6a3a0efa3bd2e5e9eda19c43b2b659b1c5b756298ffab6a9ceb", "output_wav_sha256": "b1b91f50d1dc90ba0a424f8ad1286cc37a18224b487a730f5564a00c51fca779", "output_pcm_sha256": "e7a62f4e9e43f30214ade94acd673a4d23c622afa78542914f02213acb328915", "result_sha256": "0082cb8c412a5a52d2e4e9d42e4c36fbbad8f91dad4bcb54b19408b58a66f733" }
#40formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "2e04b36a129af4ac92eefeda08827b0023d33e3ea1fb72d23c80970757f9706f", "started_sha256": "5c78deef97a47acdc91a7adc7935ff527d883c10d48392864ae62092bc621919", "original_grade_sha256": "32efe39b2b807940069e8e3a78a1f24cb541f9ce153daa1b16959e97d9cb43b2", "input_original_sha256": "2e04b36a129af4ac92eefeda08827b0023d33e3ea1fb72d23c80970757f9706f", "input_wav_sha256": "05859670cc2fbca0f955751a35cbdc63f6c52d1f26849e826c333870027de477", "input_pcm_sha256": "7a8707bbd0479629e445dbf8494dade7e2361c60b38cf2c7a707ddaaca99c598", "output_wav_sha256": "581b2eef8d7c99c528c29354e8e16f885f8db039537af4a771d7ff2fc08564d8", "output_pcm_sha256": "c11b9b3cb3bac05598cd68967c2bfcbdecfacf3bda13acf286fabb9fcd6873ff", "result_sha256": "1ce78379ed5650b0df036fc76433142cbb7b703bc4c2b0b42076c78ec033f2ee" }
#284navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "fa886c5ec0174a4f6cf2a8a141d36a7bb96ae76483ea2898bafee7c00a546d07", "started_sha256": "36881c6112b7170586633efb4fdb70304cc691c645e94cfe4f1dd2b00b63b386", "original_grade_sha256": "9e112c689889e716fd6f8f16501dd265f7e7ae77484e1b70fc64e418e01c354e", "input_original_sha256": "fa886c5ec0174a4f6cf2a8a141d36a7bb96ae76483ea2898bafee7c00a546d07", "input_wav_sha256": "ec87e8e3c88b5677209c8b31d7d62ad5c525d9b91eb1509c024cd3fb89071706", "input_pcm_sha256": "b733d56fac28a92fc7043add7d01180975533062cc2296d14b509646e81e4460", "output_wav_sha256": "6a470b886b51e0e497e2a2166d5dff859ba50c4243c43741500ebfc2a02f4892", "output_pcm_sha256": "d8c9709c0dc2dc00f6c8358d6860fbd4597d15fe6baaa2462dbbf4b665df8e3b", "result_sha256": "9cbb25f8d499b3e74e9a1ffb87c2cf227d8ff12e8f28bbf2f3bc8a65514ee5ed" }
#634object_counting heldout	3	The answer is 3.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "0edd3296b4bcb8c35adc701cb4b7f1f9a1fd121ebebedaa986357e9f28c58409", "started_sha256": "5949cd329cf29554fc5f769a1b7fe9cc3b9bfc83bd1714130d79c5bf8ff764b2", "original_grade_sha256": "d19492c10079580dd327f2a504fbcdd54ad5a1422ab64942563db9f42e21d6a4", "input_original_sha256": "0edd3296b4bcb8c35adc701cb4b7f1f9a1fd121ebebedaa986357e9f28c58409", "input_wav_sha256": "ff0ccaf2f8a6f520a40da060575c081253769501db8f85ff1c64a59cf5b87e9b", "input_pcm_sha256": "b6917db9c2a9e5f127aace8c78f1d6cf8a28b18fef0ae8629d6582a1f3e756a5", "output_wav_sha256": "ba5a28cfa31d55cafb78193c9a13e4562439cd6a15ac862281c21a203d2f2562", "output_pcm_sha256": "36f76c15c264493cdb30f65a08e0343d8a35cc17aab58f4b61912722274eaf21", "result_sha256": "9b632472a03451ca23234f75734f3c34d60f1465a4fa4cbd0968bf921a93d7ec" }
#838web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "88b13d82a79aaca914013c44189ab15c0df51acc91de641e6035aebe529562e5", "started_sha256": "75d5b87b953657ed88d3e11e1ea1216273bf2e92d183f924155b01545071dc06", "original_grade_sha256": "c014649f804058dd2f4a490bdb152f1798c9d4f33395d74da6831c866e8e9223", "input_original_sha256": "88b13d82a79aaca914013c44189ab15c0df51acc91de641e6035aebe529562e5", "input_wav_sha256": "4423deb12b5d5aed1b3a15fb4393d8ff00d4ca11447f412fb8a51a602a777d2f", "input_pcm_sha256": "5e53bc61a4b987a3e14315c7c8697bf36555cf9997e061c283fee43d348e1e9f", "output_wav_sha256": "b2534f7d20390ca255a7cbced03d6e070019ca9c98602fd5c952515e2927f072", "output_pcm_sha256": "d7eb8185a03c0969c39dc30d448b4c64fc090476f428f8a9ebeb945a38cc4f27", "result_sha256": "ae43126ee5678ff0baddf234e99d707b3a045f9630d09118eaf047cc9f9e19a0" }
#84formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "bc858db07f47ea24b2f000810ffa954da874482b7d21165affda2f621a337fd4", "started_sha256": "d5980db8e826dd3f60922f5eb920174ff2b921cd81990e9b01ed605a410093f6", "original_grade_sha256": "c43603282b6e13e5a599b2c810d47cb53a0ac86b4bb41baf8729e5524f63c145", "input_original_sha256": "bc858db07f47ea24b2f000810ffa954da874482b7d21165affda2f621a337fd4", "input_wav_sha256": "ddf63a8a88756e8558a696a2b238217015a4782a6a7c4e8e87ca2513d149206d", "input_pcm_sha256": "2f8c80dc345c77afc376187631e001ec3b5eae221c406a32a021e4f4564f7a63", "output_wav_sha256": "af9d0a274108b273484e4131bb0ae4cd8a415c2dfbe025554f5d1583b1bbb159", "output_pcm_sha256": "475d3cc9ac38beff2f1bbc1c8b11f270ca18750e0d032e436732110ffa9ea3a9", "result_sha256": "d51d16e3c93bcb040b3a94e722e489bea1e5115dc8a883daf29902b7691619f8" }
#448navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b94e1f4c04b9f110c22071eae3e73f7df6f02ef925b1edec869af8c7225d93ab", "started_sha256": "2c442cac5a3fc1478fb47b40cd003ed6990385c2cb5ba23ca6da491958218b63", "original_grade_sha256": "dd9d6aeac7d62e997ce0e37afa2e38162acf8d231188c87f68f50c29cbdd46b5", "input_original_sha256": "b94e1f4c04b9f110c22071eae3e73f7df6f02ef925b1edec869af8c7225d93ab", "input_wav_sha256": "1665b8be4786c08b77625b02e9d8586fd6f6c416f1fac92710864fa4e6f9c07f", "input_pcm_sha256": "4613939b9408273cdcbfcf83449ea328587270059b5799c2eb32a35c40163d7d", "output_wav_sha256": "00f7b3272613e63e9aac999a38300ecfcd0d0686e867caa42437d2390a5c2f8a", "output_pcm_sha256": "60ff93b383bca97527ce7d7092a57d8c324798c483c4028902acb5c0c00bf6e3", "result_sha256": "25ad8158c01205981a8e54b11f5eaa33962fe88f94f8826e663ea5424cc069e3" }
#641object_counting heldout	16	The answer is 16.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "3a076a516d5516f96290c96ddf0b8118678cb51ae766428edddebb6944bb663d", "started_sha256": "a53ce6904b95421507e7ea491c4b899bb9bd3c03f0edaaa2c755a851f98a5f9a", "original_grade_sha256": "d3bd97bc764d12c6de31ca9ce975ec7b7b4623f3aa640e48abab95781fccdb27", "input_original_sha256": "3a076a516d5516f96290c96ddf0b8118678cb51ae766428edddebb6944bb663d", "input_wav_sha256": "d6fd9a1c8806d5cffd60a03a6e964695256cfdcc7af28aa3b7d238ed451c4f50", "input_pcm_sha256": "78719dd0af59454e3de9d7cca5c845da34c88c6e67c4ded8c97554f8d95f1a22", "output_wav_sha256": "918a31da89142bcbdf3875d6dd35154d4c83db5c82b5220ce91f566b5f35947f", "output_pcm_sha256": "53390f2bb081498c93ec938957e44f2a7a69f0488052e943727ea8789e2d2313", "result_sha256": "26c75915e0e6b4cc821a8a9a711f6147db7fcb07055dcf3630371ec64f6f0c1b" }
#960web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "fdde2721e31125a29b699a8a4f922cebd4cb0dc9dc05e6f5ea362d2e2236cf9c", "started_sha256": "0931bc5b1c5d15328709cfb6e2a06303615982669124612552b679e5433ca216", "original_grade_sha256": "361fde38399745a4eeb1c45e0358eae0dcfcf0952058a75bfaf1180c1be649c6", "input_original_sha256": "fdde2721e31125a29b699a8a4f922cebd4cb0dc9dc05e6f5ea362d2e2236cf9c", "input_wav_sha256": "0c6630d82b43a940a3ec171ad10e8040aff53dd1c86b5fab49df6d02c694d1de", "input_pcm_sha256": "ced1f1a5140f2ec20be7579b20296d0bb7cdd203d5768bfd084ec46d005b831d", "output_wav_sha256": "9eaed804869496968b7fed8affdb093df86d47149f4eb4dc440410fc95e13807", "output_pcm_sha256": "80faa7e2000ace67c0c240e73d6dcc86dac49dc12b95ccfa98d13d815ba47e30", "result_sha256": "bfbf39d39e4532d203ab328f91db4492ac1f06fede780848aa06e0350dafde42" }
#244formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b45b37f39a4954f3a6bfbee79359c4a9d46fd74cba7cbbd3400aef0889a64911", "started_sha256": "237c3c907810139c8fb7d4c46bd2def7310a1548e420d3c5313b393e4415df20", "original_grade_sha256": "9bcd603978530629dad1d9d3ed72c065c1f79f6889095824a35247af03a862a0", "input_original_sha256": "b45b37f39a4954f3a6bfbee79359c4a9d46fd74cba7cbbd3400aef0889a64911", "input_wav_sha256": "d9ea59ee115b9408f9339236ddfce84276a9c4156bb59d4f12a5eb7129258a11", "input_pcm_sha256": "4002279bae205bc0f7b7d84507670473f7fac4aca668826e7206310ce94749a8", "output_wav_sha256": "46714875ae81da762dbaef4a11164100b8c3c465af77f7de69ed253ce7f3ba55", "output_pcm_sha256": "56cad9daa763508b7ad2f41e8e5f864b984c0e45f3d997c91b4c1f57d765bbe8", "result_sha256": "60d33ca76c0955928b67e8030823537e3c989fcdbf43372ae939299469f76e96" }
#371navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "25f5a87f24ac6eb615ac10632820d6a3cc51aee7746e43bd7dab339bd7a5a2be", "started_sha256": "b475316bf1d5e959178309fa4ece1293d8572181fcc9535b607e3b33c305e2a6", "original_grade_sha256": "443f8efb191b00d62cdf147e2c9913e739d1afb4cabde418014ebc7b766576ff", "input_original_sha256": "25f5a87f24ac6eb615ac10632820d6a3cc51aee7746e43bd7dab339bd7a5a2be", "input_wav_sha256": "95e92acdfe4dacf16486248395d6a3c86da3d1862b697821c381ea3cfb9c2d4c", "input_pcm_sha256": "38ad09e52da4512daddcac7054799018c6fe1dc6a897abde988c15b66ba31c65", "output_wav_sha256": "49822bba223f82149855f88dddad025244f5f35904b18a5fbc230c43e96d4f9f", "output_pcm_sha256": "5fdf5ea6a17a40a6349598515826e9d48b485f94d233b473aaca15c693cdd7f5", "result_sha256": "48283ae1781ad91ad9ee60000ab8dab63613cda68bac6ac3057576c6b9dd9292" }
#749object_counting heldout	16	The answer is 16.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "8d6fff651d29a5b5b0b6e77c5ca3409a9e27d35cac91c010d297aeeab0daaaa8", "started_sha256": "c8417e9ddddaf7ffaa98e77da7d14d7186ce370564f188ddf23b6ed2ef4fd824", "original_grade_sha256": "2277e8aa756e02ce85334a1488c37a252f23ad007d0daafda5abb78e5090448d", "input_original_sha256": "8d6fff651d29a5b5b0b6e77c5ca3409a9e27d35cac91c010d297aeeab0daaaa8", "input_wav_sha256": "7a058b95c8a675ece83dec206b96b586c9c4e2d8a3c58ffb38862212a2239a54", "input_pcm_sha256": "465ba5e296201a48b63a9737aaed5457bdfa40e5e340ab68d612776b315e7278", "output_wav_sha256": "929c04d596159f160dd6c9cb18260d1ef079584c6b71ceef27651326e9753bb0", "output_pcm_sha256": "cb09a2c8324b39ab94728461afdd555a96fd70a6717f9a694ef530bd9ddf7dea", "result_sha256": "19ce65162fc1b35ceb6f53aa4836414f073676edd7344918f036668addac40be" }
#791web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "1f01aaacdf4f3da10091377d0dafda3c8d37bbcce4f3c6c315cdc6614c4f48b3", "started_sha256": "060941fe9bc396b3ea3e173a693e1012b367e44fb7902eed4f1b795a2d1f590e", "original_grade_sha256": "0f5033b47521519ac82fd0be14a38c9b1c05668c6a08e24af4f409905cfa3f47", "input_original_sha256": "1f01aaacdf4f3da10091377d0dafda3c8d37bbcce4f3c6c315cdc6614c4f48b3", "input_wav_sha256": "d860e58f52d05f77ef7c2de9bdc94379ad88db1f5dc12cf2dfca80236e8131ce", "input_pcm_sha256": "45977f66f8abb8a851579e456b54161b94eae1e12de7d62cd4856647070e6965", "output_wav_sha256": "b7481ebb3bfdc14ef8616111e2f6b5ceaf399c00d7324e453f8a786364cdc640", "output_pcm_sha256": "0f557de011ca15ae900db0305a5224060a68e30a5cfc83774f9e210eeb77f27e", "result_sha256": "ad7ed585ce840b80cda80b36c9c20761d73c0705ac315948795be56f2d719756" }
#23formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "cb51d7ddb20e985a7ac70fc3132bf8f46457bb1e9046eaa714e17215913e6c02", "started_sha256": "45de24c361e696cf641e9ea847a75c72429a8754ddd7f4f5b8ee051a3f3ba67c", "original_grade_sha256": "68f13fb78d121f413889bc3029eaba908574ae575c9a599618ba5492ec124559", "input_original_sha256": "cb51d7ddb20e985a7ac70fc3132bf8f46457bb1e9046eaa714e17215913e6c02", "input_wav_sha256": "ce4e0fcf4fd8cdb45a6fe1823160db38057a5988e0a89eaa08609e11d45d592c", "input_pcm_sha256": "bc76d0719809480ff77903d95f45eb9e4c2061d0aba25d2587d1795be52a0702", "output_wav_sha256": "c09fa586f0be1c666a1dd60fbe105fcaf7bd6eb477c76f162740e409992bc08a", "output_pcm_sha256": "b39fafadcec2c269e4fc4c3d4cdc0066b8527aee7caa9deafedc199903e37322", "result_sha256": "1b3ea138415bbf127eea4d27a125f8b41bd71939535c93178473404d73f29347" }
#317navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "7a52a7b5887fa7506fdd2270f93dc783d90efa72ae8bc14969c1fde039d9502e", "started_sha256": "0a9130aba7f713b560defb42b124583f975fc4862c5bda5c75be1a7921a376e5", "original_grade_sha256": "4cf9641fc014cb97d50676a5f3e644b7933ffcfbd027e7e4cbe08ad184df0bc6", "input_original_sha256": "7a52a7b5887fa7506fdd2270f93dc783d90efa72ae8bc14969c1fde039d9502e", "input_wav_sha256": "930229b9b60eb0e540e2603d773c6b18f302ee77283f9a3eccbff3f3a55e6d45", "input_pcm_sha256": "f3bde445b2aff01a53aad01f8a25a55379b4e3f5908ff3789180f817040d004c", "output_wav_sha256": "f1d9c0dfff4c6a682fbbefe9186f2f9fbc9291bafa4d3cec042a26c0d49a5afa", "output_pcm_sha256": "e42b31d4fdb5f99b993473195103013a6bb6bc47bd466dbe52f6993bea5f6020", "result_sha256": "6b9f912acb0e96f3523d44aee240cf8262ca5d7204a2c59009b8be1a224a1616" }
#540object_counting heldout	12	The answer is 12.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "67da4f9b6bd49876cf2dc0556a0d2cea5edeaf8dbb422a62dc303ae28d4e926b", "started_sha256": "15453f42d4f49d1b49b99d9a083c0f08e8d0e252e19bf91a7d1488683806fde3", "original_grade_sha256": "48622ed11fa4d537439f2cb5b5635004dde6d85f172d7f0e16891595e9a0513f", "input_original_sha256": "67da4f9b6bd49876cf2dc0556a0d2cea5edeaf8dbb422a62dc303ae28d4e926b", "input_wav_sha256": "3eb2fef56d2bf9fb9f3c83a1840fcc1a59124476f8f129d8fe147b0e099608d2", "input_pcm_sha256": "ea4f3a7f8515527ddcb9d8d8347d0e697c95f2761321b5ace4e45548c3934224", "output_wav_sha256": "383d53958d9f40b7345102ed83d5889571a1133ffb8c36159287effc9cdca6fe", "output_pcm_sha256": "f1da4d521e1e0fcd8de055bca338474b6ba9b53957b5f5f3ac46fca9c05a35ed", "result_sha256": "9355d6cc072bd40cb6c4a611e10666fe0a06d571745d4904dd46295018dea633" }
#937web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b327262039948486849ad600ad3633116b0faf52ebeb4db6819cbe45593f9ba6", "started_sha256": "8aab27b851a7496347f6cc9f195d32b61fe913dad93a486a93e70ce19de4317f", "original_grade_sha256": "6b3a34a5b326e199da60f69fe5fe1e0e0c31f0bd10083fa35f71013d62def4dd", "input_original_sha256": "b327262039948486849ad600ad3633116b0faf52ebeb4db6819cbe45593f9ba6", "input_wav_sha256": "b303061f86c368e97e2315b04af114d2b44c123a8a92f5e0f54fe09554080f8b", "input_pcm_sha256": "1faf3f1fa46c96ab1d3aa8e2a1144e38cd58d48397416f28479734417b83ee4e", "output_wav_sha256": "d3b6e1b7fa73cda2e352c3311b259f5092c0e276ece42bdbc2d21e0446471569", "output_pcm_sha256": "d124c9b2bf86a67b9f9a3d03bbb904e43fd41b4b351f32e26f8ef1bae010ed8e", "result_sha256": "cf4844f7748248d18f7861755883dc9cfe8bf808007858274fbc3ac4ce49db69" }
#195formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "9a36167e6976f78a01692b9adfebd510fc11f9c8c55bff63cc4b348f5805b4f5", "started_sha256": "847b2cb80ac9836543509b0e2145eee0231308cd78ac82498eb7af135976fe1c", "original_grade_sha256": "0827547c4cf3cea51441ecc5360bd9b44741ba8af19ce2d7dbfe80514b9f22aa", "input_original_sha256": "9a36167e6976f78a01692b9adfebd510fc11f9c8c55bff63cc4b348f5805b4f5", "input_wav_sha256": "8000dc57b67b15dfeadfd8f3be13fddff9ff866100b0cd98e3f3f0f12ecf0b69", "input_pcm_sha256": "0edc32be592185550c5d25c0dd7f3a563c674003553a3daa530968f68c84ed66", "output_wav_sha256": "182472b30d762ee246a2afbc854a1e5705504ba682ad369122c7566a4b71bf4e", "output_pcm_sha256": "c83c3c5842ddf187a16e9872df556fa4db9536b87361cb42c9c8a1d24ed2742f", "result_sha256": "3b38352466f8135c6233e179aeb3f8055c1aeaf4503fb94e35c26b95f4d4de4e" }
#348navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "e6b6ef3a4c310d254ec2e53bbe1a8bfd02e0573fff5331bac57d2169f7425348", "started_sha256": "85383d9b6dd2d52989cd744a322b9003bc7eaa09c890ef03c67e6f4f50ebab95", "original_grade_sha256": "fecebb7564e4b0e80e679b85af90797cbb24959207003985dcaa59571eb008ac", "input_original_sha256": "e6b6ef3a4c310d254ec2e53bbe1a8bfd02e0573fff5331bac57d2169f7425348", "input_wav_sha256": "bb1d328411dc146b158c8dfbf2be309c573445d7cc54995553480a396ba53baa", "input_pcm_sha256": "45ae07c0ded8bf4acc36a9a978c339707ff45453da54893742dc5ff252ca1c33", "output_wav_sha256": "e9e7e101951f2c970b26520de1f921318c46c92beeb4ed49f8586e4258874b6b", "output_pcm_sha256": "c936e20616acfc3b0a033d7bb6d1d723bf687b0c7b1fa4ad560a3fd22e61f0c2", "result_sha256": "37ec1f5d27ab774d21841c1fb95931b44fcbafc691911e2a2b06efcd040d1647" }
#547object_counting heldout	9	The answer is 9.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "d0210b7d5e1b4259d08218673af95711f23027602ba2f9e2f946fcd562fecd6f", "started_sha256": "bcbf007035c75e8e0c096e28b480f4a107bebcff3bb77927cf08e267d58da7af", "original_grade_sha256": "0fc8f8d81c358d3b64cc8a90ea5ec83e09345df21c6d3f16503b208549f9ff39", "input_original_sha256": "d0210b7d5e1b4259d08218673af95711f23027602ba2f9e2f946fcd562fecd6f", "input_wav_sha256": "cd81f62416ac5aeffd73c913380feb913aa54d048ae72d3d1d90b305aa1924ce", "input_pcm_sha256": "586470add69dc53977723566bcc5e727191c35da03e67fa9c9311f0c6e86309c", "output_wav_sha256": "757ffecbb5dc9d92500c445161650c1e95e163e68bf81b39a6c7fa9d2aa9a2e0", "output_pcm_sha256": "670ff487222597164ce01325f0a9c3ae331a0b725ff8376a394983e6c65ad519", "result_sha256": "1253986c92582695b94143e808a6a77d0a3a543da23647c8e3d0fcd8460e3b73" }
#880web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "dc42c9d9e40d8ed559566665c582f9ad04be1999b3c1305cf8030c2dfac17e6f", "started_sha256": "f08bb7564253521439d18cd9d0767e8ca0cff813efa932ad259e8c6606862a42", "original_grade_sha256": "ca8461789568933049e2620ae45961915f40f790b93e91c4fb87de3948c18b6c", "input_original_sha256": "dc42c9d9e40d8ed559566665c582f9ad04be1999b3c1305cf8030c2dfac17e6f", "input_wav_sha256": "cf8eefff06892588dfe0d21f1afdb8d57310281d109cfb3231ef9b97c9168633", "input_pcm_sha256": "147b1a8416018e483a783e4634a06a39e614b9da0a975a399db7cd78977ee5cf", "output_wav_sha256": "14aacfb275c07857bdf146b2d8f81d514b719308a1a38dbecd038c7e875bf7b2", "output_pcm_sha256": "e46aa034483f95cfa5683f4c8c95f84eebf43ebc87dde6c101fad9a18754ad7f", "result_sha256": "dd1b7b90f5023ef83a3e1c2a75f6abd98060cd4682aa3f9bfb92a3b14979661e" }
#108formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "8df15960e4820b3004093fb0069b548afa2749d5f03c46907186914055e92ba9", "started_sha256": "c7122c21527a9ce7030cce81db01eccf29d1c1d8190f57d4820235907aac35a3", "original_grade_sha256": "2a6194efb06503b286b341e84c1171ff3a53cf1e39ad85918a7c4c5ab3dfeeb0", "input_original_sha256": "8df15960e4820b3004093fb0069b548afa2749d5f03c46907186914055e92ba9", "input_wav_sha256": "808ec4d31405a279914010ea0baeb61c5fcef54fbafdf0a0087bfe1831f80648", "input_pcm_sha256": "56c2566fd93d396f71f531b373b1d21c8f921c050e96a264ec7d02fb4f68aace", "output_wav_sha256": "d29894bc6978e0aa05ddb07dcdc77a48943ef2ddcfc3354a3c9233239cf9e203", "output_pcm_sha256": "5aedc41c8273782e6987d24eadf73469ee0858f9393578758334c5da21d339df", "result_sha256": "0a38352487650843cf47a4ced06a544ad91055cc1b36b8f552fc2c1ba8eca49c" }
#461navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "190c625e4b10bb28598cc5211caadb0e11acac88edfe624c29cea73a6991d562", "started_sha256": "7395702bfb466b4fedd9771c60d575d7a1927d542cf46c49127c86f2d2b5a7c2", "original_grade_sha256": "1efe48bf746171e6c135b7782ba8e12f29d460d467d665c41216bb2bc123d5e2", "input_original_sha256": "190c625e4b10bb28598cc5211caadb0e11acac88edfe624c29cea73a6991d562", "input_wav_sha256": "c25a671d54400fcb6538bc22ee851c688e4867714c391b92089ea0c6de2c1dd7", "input_pcm_sha256": "b79c28a2d31e6d3ac752fd58c7417f12b2a18fc709c864a16a6c4c5740f089f8", "output_wav_sha256": "1818752ff05198dc2aba263b9ec7a5363f9d36527acc507029abc78c70cb2c8b", "output_pcm_sha256": "df81068aa46a14012abeecbd41a2927107c1e0b9ac13866e487c97d4915026d9", "result_sha256": "147d8c278a339e8f5bd4d22873b650f36174f230f587e07b2cc2244fc39847da" }
#585object_counting heldout	17	The answer is 17.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "303794d1c96a12793ae0ee895b530796c33220820651e6e32a526cd5c0d26125", "started_sha256": "c3b2633e1e0cd1b4bc7f81aac5e591d732bc063c3d50ba82169256c4a0addd21", "original_grade_sha256": "69b16ca94daf4e1f42fa4e28cf8f85b5b87f9b0396b7ce44404dbad5558495ff", "input_original_sha256": "303794d1c96a12793ae0ee895b530796c33220820651e6e32a526cd5c0d26125", "input_wav_sha256": "547ee6dfbea86265bbb8a765cfaaeec9fb121fded57cbbe5d32aa496300e9e50", "input_pcm_sha256": "d695090fcd808b2aef1c1782821947d0d17258465928f73d0367e87a6c113574", "output_wav_sha256": "ff458f379c53200dfc5532fb632d9e5370df98cdfc9bbf5b0335f2f5cd6fdb8e", "output_pcm_sha256": "53e7fb7c2466866a9fb50823957014fa932e613868d9e3090ea7b0a9cc0e0f0f", "result_sha256": "19d0c4512da14e6ab7c05f2215d4474f822c22200e63ac712130315163ed36b2" }
#929web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "33af473c57cdbd21a5e535eb4f37c27c8513a0383fd135102afb6b4f1c577b15", "started_sha256": "a6cb6ca2622342b3f34d0d5d1147507f858bd48ef88e5bb5fadf745617c1fbf6", "original_grade_sha256": "c5a2553b2a94081d810da326a558ec0499af6576d42a6e76b1c6fee3c38683f7", "input_original_sha256": "33af473c57cdbd21a5e535eb4f37c27c8513a0383fd135102afb6b4f1c577b15", "input_wav_sha256": "ae92edd6fa4a1d1e75f1efc10b363e394baf94a23afc7aaa2ab6cb7f00e74714", "input_pcm_sha256": "2ee3990f9df32bde5445828da60f7086016e8a7fc5a8c515997d862c42f43aa4", "output_wav_sha256": "a81014749f81a4c8c71d395e4e44c3cdf80e9a78117764d96db3ddb6f4ee3e2c", "output_pcm_sha256": "e9e027cea33e2f271f5ce7a79015f5f6a0bf14f9c8446a6e4d0a2dc9a6d97acc", "result_sha256": "3eebf885f82c3e6fcaa888ad07b88d2a68e4a4069882e7914b88c9603b82d549" }
#90formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "3536527cefe4aabbad43cc657253e1caa9b5d970134dd3991868777e931b56dc", "started_sha256": "e0015c885403c623c342fd2745a1cd6031a09f2668724f472d59edd60132cde6", "original_grade_sha256": "997a6ab916465322f43cbd37b5c2cfcdf9585ec2a0c579e79a18c36a4f71a448", "input_original_sha256": "3536527cefe4aabbad43cc657253e1caa9b5d970134dd3991868777e931b56dc", "input_wav_sha256": "6f3ccca2dcaf28d788d232796df5deca35a2d3b67cd5821f4d8e7f7e628f6144", "input_pcm_sha256": "6f755af93e5d7067aecde0a3c655d5f94f8bbbae147596288fca6340cc0a133a", "output_wav_sha256": "4da3bf8e3c6edc248b25d0f7ff8d6cbe572b3df409a2adfe433ec72f43959ba9", "output_pcm_sha256": "fa4dd02bf538b580d9f1d95c202ee5080771b23d0e350c9394b4eed6dab873c8", "result_sha256": "ba2f43db776bcc5a72d03e2560461844fed7637597b2089f53d0142a1a0673fa" }
#486navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "8feb49257378d2ebbe6d0a483bd73ac603ab7611d6efbcceb6622ad1ef87250d", "started_sha256": "f7e8937a7e2eb48d9e8db3471c7451a9715f746831b9a1144b61c51f6ed56474", "original_grade_sha256": "64b30446bbba0e98baffd18110f3c3fd3d9a6fa804b0666730b1d48f515cd4c6", "input_original_sha256": "8feb49257378d2ebbe6d0a483bd73ac603ab7611d6efbcceb6622ad1ef87250d", "input_wav_sha256": "da8394d54cc635c138426e32b90f430d1b025f7f9bf67252ba8b897627767f1a", "input_pcm_sha256": "e775b98ff976d6fdcd6ace8bf15453faf7c5d0ffab620bbaa8dca872922ce0d1", "output_wav_sha256": "d1aa6434c402ffbebcd14a293ee211b6945779bd42b43e841a6b725c421476a8", "output_pcm_sha256": "9b976f0f264ea341c3069b2b604f2a37086a4dfc8d95048dfab054d04193ab89", "result_sha256": "a97c7a44c38464cec8ad3343269a120d803464494fdb2fbe2b92d36718c97d4e" }
#709object_counting heldout	13	The answer is 13.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b6834eac93caff38c88e7cf2ca4e9251f16c23400006b532cb771092e8feb33b", "started_sha256": "a3fa7b567b240ad88f6f501cd181859f356670a79358e75b4e090191c1c3a5d5", "original_grade_sha256": "e14948ac0652caa5d7b7d0a525225e76150966a28fa6c4ecaf080d06d6eb79c0", "input_original_sha256": "b6834eac93caff38c88e7cf2ca4e9251f16c23400006b532cb771092e8feb33b", "input_wav_sha256": "1333789f0f4abbd1f7ef133184844036d89e6a49583ee062d3cf9b24e774fea4", "input_pcm_sha256": "3200a874332ffa4bdbfc75c063e49cf82c51a9f779e07dd17b58c2fa1d3b3586", "output_wav_sha256": "457bcf936c64d947aed888df70680f2924d95d80f7678e61357301d74fea3ca8", "output_pcm_sha256": "817739900a9b64b383e62aa41eba0accf61c7bd1e99d69c9e056db495281f7f7", "result_sha256": "30c8f960aabce6440f3c91a15729c8854741e752fed1be4f743d10214ee65a03" }
#860web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "8285126c503e019fa8d303feabf01294531e138e2f3d1c8ed14b82dc9d436d37", "started_sha256": "8489032879c69d81fa2783394ef43e72b92301bc2e2d1c2d340e6fbe7f7af782", "original_grade_sha256": "81919db79ae12167cf503f5f4be69142a5853ca43768575f069bc986f346494e", "input_original_sha256": "8285126c503e019fa8d303feabf01294531e138e2f3d1c8ed14b82dc9d436d37", "input_wav_sha256": "047b84786110300aa0d3fb00de3907b2479daf1d52627539ce4110c7d46d8e5e", "input_pcm_sha256": "9e11b5a29c41ff77c49e6a08bde6cf42a159d32d3b3a4d8bc93ee4546d0e832d", "output_wav_sha256": "0fcde795dbea3299388152ca29ee6b97ec176993951a6d9daa1ea20d829e710f", "output_pcm_sha256": "204d32972f034cb34f79595ca671cbf67caf02811c12350b7ed4b6bdef28e13a", "result_sha256": "7c7863bd6b60e7d7f6904eaed3292e1b45e4e6a115a33fa7a2e6b742c360d840" }
#50formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f00341449b7d7be257eb00e6c3b8e652bc4e135edf39f7d3d309ed45aee3f5dc", "started_sha256": "960f0fc63501b7aa8638eaa5a0add7528bb64f8671d89dbf5bdb9600475c5a7e", "original_grade_sha256": "2784a1198eb6621b4fded7d02bdaba07a80b662c94c346ecb0ba47dd94a5140c", "input_original_sha256": "f00341449b7d7be257eb00e6c3b8e652bc4e135edf39f7d3d309ed45aee3f5dc", "input_wav_sha256": "f6a307203d9642f48c50b59b69bd343dd4ccf3ee10e14db8130bfb1dd8cc634b", "input_pcm_sha256": "c7561f37c0015f4bdcdf37c60920b6c77aa2c767fd2f1215e4672e307e06dce3", "output_wav_sha256": "28ae41fde8e8e227ede6e1fbaac892dad90c943f67dcafbf981bf84f41f5032a", "output_pcm_sha256": "7bcf13344f78c9aa59e7df1658a844549c3b31aaa59cf7dbb5cf31208dc7a7ba", "result_sha256": "fb29b797e965e4049109f4e64bf8ad5edd2a9f2a3c035f9d7ffc5eee03e7ecfc" }
#459navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "81a45501000121106776f8acb181790cbbade9c842dd803b35e9e43c19f3d20e", "started_sha256": "5b06a182e1f0fbe77a858714c50c073dbd7ace5a9fa069a7be4ca0e7b91539d3", "original_grade_sha256": "ea83e9c2805d79f76114528072755405dce061156dd81cec4d81ae408860ddb5", "input_original_sha256": "81a45501000121106776f8acb181790cbbade9c842dd803b35e9e43c19f3d20e", "input_wav_sha256": "a06d7118b0af2c52c93f688da507f4d03d7106528ff4bf1113d4e03ed0253e24", "input_pcm_sha256": "3f817b9a7f95dc417ec10a993357f724c2a6c3489fdc29ad8328c7f4ba87959e", "output_wav_sha256": "b4a2948d227e8c00a858a6d9f82afcb050ca62e201d7c1f340695875db3cd3cd", "output_pcm_sha256": "83c12c5f1f742655036bab1e041d8f4d14515ecd0f6bd24ed493ec6f4d556128", "result_sha256": "a1a2935430bf9df6bc50da8e2d986b86d137660b934de1760bc08a5a878a77e6" }
#635object_counting heldout	10	The answer is 10.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "2bc69df2af157317cc7aec12061841783b7272e0e43189086d02cd4d61690dae", "started_sha256": "db7534aeebb3b899747547a6361adeb5dbc1cef7b8f7d4e2b0f8c323fbf56aed", "original_grade_sha256": "9368fd0e25d213c62365de997b6b54b083ca797895bdfe71a9332d50f86999f6", "input_original_sha256": "2bc69df2af157317cc7aec12061841783b7272e0e43189086d02cd4d61690dae", "input_wav_sha256": "5190aa194b9798a7be8076295f6cfcb4b6f944260d818b4bf0ce35beef66fc38", "input_pcm_sha256": "ae421d0c60a117d3c22a1fe4095d6f708c43d282fe4567e90d3d1579d01b2464", "output_wav_sha256": "b3b629aa928b6942cc4c86df8df172fc6783f4d92db067f24f76cb7e8e8af987", "output_pcm_sha256": "f89901511ef41dcb00e23c89abc0ba425a7b10afa474b57d1db62448af857a6e", "result_sha256": "6f0c0117910b4754a09ec312fa574b26bd67c3fe69fb17f3c87a1f32334ff2a2" }
#886web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "c14974263d3210076c7e0bd032dc09cc584cff2a6f158a44aa0aea327000cd5a", "started_sha256": "5e9297eb94f5b66ce4551a8e9d270012ee8eea07a38c924c781fe97481d5276c", "original_grade_sha256": "a2a82b9daf2027a752d075f33a68c26ccc885037eb7a646d2475e0d58fa99cf4", "input_original_sha256": "c14974263d3210076c7e0bd032dc09cc584cff2a6f158a44aa0aea327000cd5a", "input_wav_sha256": "df782c6afe697e569068b2b67edbfff8d9573bb909efa02ca0d2031bbb533973", "input_pcm_sha256": "48b94da3b60041472380ed695321b3e34537d9abef719729ff357fff3a1e1a08", "output_wav_sha256": "b1f5ec0075366b0a7dc1e639c63ea08e2b13c21f1b2c46197f83b30ddcd37ac7", "output_pcm_sha256": "824f5031231b537154867b0a785ed197a605d6b5c0e2ed3bddb435ff33ece96f", "result_sha256": "deb744b3fbc12dfc20a3d7987ee32bbe462d63c96fbeba78e21c82411fcf0ee0" }
#64formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "631916c2da1a99f39dd871215e4210bf34352b6147603d8e51d8fd5b22865474", "started_sha256": "a517740047e79195330f7e3c64598e8df1f8a8471d4d1424cdb994d0649ab66f", "original_grade_sha256": "5fa4159a2f5f6b27e23262919e7ad8a8f905fe849cdd64cde6c874b5cfbf2604", "input_original_sha256": "631916c2da1a99f39dd871215e4210bf34352b6147603d8e51d8fd5b22865474", "input_wav_sha256": "a22a9b18bcef0f714d9b0d87a19f22ae5f014c08824e000f2a30279f9d0bc836", "input_pcm_sha256": "a21892394f8aba9e4c02d30a09b12f295d57f4ddea69dd51f197a13cebbbe6b6", "output_wav_sha256": "181c030c558a412e54a2767a2432286106d430dd1d8f0dc2ca13ecf045621420", "output_pcm_sha256": "26d16e57f66906f893a2ff5851a2fa2c8e4f6f14a1317da0e43792da5e848065", "result_sha256": "e61790286b3f85e10947424927447ca09ef2fddd47b1710497fd7cfa01b63d71" }
#406navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "a92c05b6fe9e299fcaebc53a980d2c82d287f69d761d369fd479a413b0220f36", "started_sha256": "1409daa83de205381b04c722f5cb355e9245783778df8b63e4354d89cbde89ef", "original_grade_sha256": "fe2f85b379c69d1e7491eb6645c57dadcc661f2803876dcde6bcd04e3c604508", "input_original_sha256": "a92c05b6fe9e299fcaebc53a980d2c82d287f69d761d369fd479a413b0220f36", "input_wav_sha256": "9e091b0e17a920ade9581bafe16fbb5c16891e1a2d17ba273c61b7aefdc37087", "input_pcm_sha256": "e9d5c7236bf34913279593ae75907effc2e1b26a14560c9453b6674de3000022", "output_wav_sha256": "cec309edafb499b2ba713a8fe888630dac05019097ff6387108740dd0355f8be", "output_pcm_sha256": "f09abf5ae907b1c3c2eda10d6ead9680986386e5d1dfc0ba48483ffa9b845eac", "result_sha256": "c5e53ca85850634839db6180a6611fa87ac1738bc5a5b8dad793cb91965768b5" }
#514object_counting heldout	11	The answer is 11.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "9b8d6f362818de291ba97015d1dbf391b7657bf8bc8e02e275570afd9d964b1b", "started_sha256": "c2d1da68e3e1e9968ecba760357bdaa36f777d05318959ce861a0df08e8bd8ba", "original_grade_sha256": "85b509556fe4fbe66face925aa2d54913a6eb54cff55678853f274d26265c6c5", "input_original_sha256": "9b8d6f362818de291ba97015d1dbf391b7657bf8bc8e02e275570afd9d964b1b", "input_wav_sha256": "0fed5e7cd747d09edd32d5ce03b7849a032c22688823926871e13e35fa3f06ad", "input_pcm_sha256": "0cb79840b37765e2a2e0fc68082e48c6148c3e2e682bb70ba9a22e39d6bbe938", "output_wav_sha256": "c8ba34ca0463334db51e68c96ac72d850b716ef34693832ea52227eccea8cff2", "output_pcm_sha256": "1d6a70e9a74809d491bc716b7717701f27a9d01553b636b1f7c88e66ec982c0b", "result_sha256": "6a85ce4a059d9b855070f7497b813d775e1eedd84a25f87f4772f30af2481010" }
#805web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b3e98566d18cc3716d63a4477609a54e6834bb043fa29ff06611f974e565b2e3", "started_sha256": "db6b7260d66bc79d9cd73962fa958b66899a49ad43344a0a9a3978b9e36a41c7", "original_grade_sha256": "21d8a9279284f4a932f5f20d15dfaf578935f246d409c72de9f266d57d1d1a38", "input_original_sha256": "b3e98566d18cc3716d63a4477609a54e6834bb043fa29ff06611f974e565b2e3", "input_wav_sha256": "043ea71bb598d40ab8f94d12904d930233937508adef3ae86e17519c4ca20960", "input_pcm_sha256": "c4589e253f05dffaf43f219621046a04d1aaf586a385c2fdbda1db451b6b8289", "output_wav_sha256": "413fc721e7129921a34dec4ed44534fb664e486aec381014883867150124bd59", "output_pcm_sha256": "0d9c97a8b1ff0a84ec69b69e9663ed74243fe33e604479b6a897d9118aca786e", "result_sha256": "6807d551b49863e548f4042526908cd190c2caf534a5743e2895409fd20a46d9" }
#249formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "7b8d4d10117fab9f15af3b80d38f9965ecb3448dedb3a5576549881674a46b1f", "started_sha256": "c7ae631df17fafbcf73d0e3c8dd8de3a37d160fe25942f81017aa627b62f6b73", "original_grade_sha256": "ce6fc5151b4b8ec80e6f19ee6a3c96cba62b6ddc08ea1812653d007c80220377", "input_original_sha256": "7b8d4d10117fab9f15af3b80d38f9965ecb3448dedb3a5576549881674a46b1f", "input_wav_sha256": "dfa728d9b9717b764de3c99632654b1fe84275d806cb1f0322bbf2bf2298218d", "input_pcm_sha256": "a443f649ab8d320d0d7264de2100e85b3ce8cb3a7a89923bb0fbf09c082e2e12", "output_wav_sha256": "ed63b9f44c4fa42e5453556caba745cf3df1a98c427c1883f61189a2f9464096", "output_pcm_sha256": "e68cc6b6eafdf45b1f8b9504a76ddb22d17ebef23c898555cd32ca18b73f4f37", "result_sha256": "270bc1cbb5b35e7324c49f7f9ce3672a4ccbd4b010a3e4cc6960849e8c88f5b8" }
#322navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "d10c0f259ea505df023919b7817f1ad97ddb1d838626de8a6bc8359f9c19f33f", "started_sha256": "51c00e19e5cc029b739876b6444dfb3b22441358bc00c39ccade34e149853e88", "original_grade_sha256": "1806e94833375af510d8c13cf2b43302495c333fa13f0639f01a2affdea6aee1", "input_original_sha256": "d10c0f259ea505df023919b7817f1ad97ddb1d838626de8a6bc8359f9c19f33f", "input_wav_sha256": "8b3c96882ca1a71eabd4489601bf13eb3d89097fd5678e29cda408bbc47cf138", "input_pcm_sha256": "3081f85cef8df76b9dc40ce56f5e37444f1b197a6be135d4bd9c35044a49cd63", "output_wav_sha256": "580f56a3c2e5d6e419fc2d621101ceebb81a35e034cf6213324a3789dd81e527", "output_pcm_sha256": "b7ec16dd1aaf93c63d437c7dea72ea27b9fc6e8ec7c83573bfd9d374e370bfc3", "result_sha256": "328c9dde1111e308321ee301c9cb8c63c07ab2326a39b74e9c4d88d97345ba4c" }
#652object_counting heldout	5	The answer is 5.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "5df14b75fd29e155aeac9b0b8817fe7f52cfcab9b763bcc661736b75a4e46723", "started_sha256": "c3de87292177ee60136685e055f91e680a5e4253e66548081c3b24de1a95bd75", "original_grade_sha256": "2f7721b2a4643b8dd91eb9ef0ef966a374f0b82132abc7e931b94f17e657e586", "input_original_sha256": "5df14b75fd29e155aeac9b0b8817fe7f52cfcab9b763bcc661736b75a4e46723", "input_wav_sha256": "92de13cfc898941e241170a1d0b4080f3e27852a9b2c4287c867300f37e2b592", "input_pcm_sha256": "32e39c5b476b945516236fd3f92cd8a0dbef793674ddddf3546333ef447257b9", "output_wav_sha256": "213c9b74f4f76f288c58cf83f715cba2f8fe667fdcafadce46d06f64e9c1ea76", "output_pcm_sha256": "e843b575a1c006a05e285b00717e60dd41c9608943403c500ec2dfec9b11cafc", "result_sha256": "87d849ca2d7ccb30daa22a621f4339f1f86163025f6edd904924768de1255ad2" }
#920web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "6e7cd64669e977333b4cb1a721baf4cbf036ade545854e063297078bbea7e5a2", "started_sha256": "2540c8962f0a05810a4a5126a590d637ba21cfd881eb23b91b0c65cbe1c38f96", "original_grade_sha256": "04d0c7ef5225b9ba6c24397f8fa2b1e561ef1f1ad9d7599bd228d0b771c9e7a7", "input_original_sha256": "6e7cd64669e977333b4cb1a721baf4cbf036ade545854e063297078bbea7e5a2", "input_wav_sha256": "eff73cdad2287fd43da1d0ee016d1dc5eb39f3d0dc7bd721ebe78243e2fe65ee", "input_pcm_sha256": "7f81bfa108ba6b55844877cae7b57e4a7ca066cffdae54c31cb95572e7293a8e", "output_wav_sha256": "e216d198849c7d3f14b7f10e71c7bb8188dc261d3fa5a490d357ace5cc6e73fe", "output_pcm_sha256": "278ab1de66a1246d809a065ee3f09a6c1928917834eeafa2e00e6addc98153a4", "result_sha256": "58bd957c475a4ea298f2042e9dbe75188cf28d8a41789b0d8e012d27d6de23fd" }
#184formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "d3c1e0bf6e1d35172f1aa0a31c7e4d2813cd43d5a8c33e5718969863d906fe0b", "started_sha256": "05379141e40e77754c56504717d30cd1aaff0e3fa3ddf99e04400bbeb5dfd73d", "original_grade_sha256": "9886eda099921d5da8f60de6f10c64a5405de951fe63766bd3c7e4ae0200ee7a", "input_original_sha256": "d3c1e0bf6e1d35172f1aa0a31c7e4d2813cd43d5a8c33e5718969863d906fe0b", "input_wav_sha256": "82d9cb72caf74912b43c3c85a96d53d445bec26cf425bf3b2b3b21599a487873", "input_pcm_sha256": "887adbc027e68ae8557ec07ea4b51f328b134b71f4f956c9a44a228a2604afb6", "output_wav_sha256": "b407cc919dfdde18f4d39ae806cace96756fdbbdc5d8aab1502475f19d91388f", "output_pcm_sha256": "cb28fccd80d0895060f8bf6873071e6b1b307155736085a60bacb00b83b01e08", "result_sha256": "ecc162f460494e800d2ec663fa5643efeede6e2342fccd930d828493a45c997d" }
#446navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f30647b51050b8ca0732e6d8a45b190425b0b8fd73dd8b4780d5d7deccbce48b", "started_sha256": "9b9a6314f4518b1189a48d6a8d29d833a0c8104db471980f3ebdd1efc30444aa", "original_grade_sha256": "04c8d4709a7f642ddf0e669d699dc82dfadbac5451fb0f8992b5a5535c77b375", "input_original_sha256": "f30647b51050b8ca0732e6d8a45b190425b0b8fd73dd8b4780d5d7deccbce48b", "input_wav_sha256": "de09dedf31fc06722d5ba41dda6aacffa9ad77c9e28ae05e5f1fd2a0cd3ac6fc", "input_pcm_sha256": "671e3956844f1ee5f8e5bd52d746940a007e76f932a9c09a46a4137bacf1fd3d", "output_wav_sha256": "a27cbc4f93841075d7e95fa19551cc9b360672cab7468ac8cd2a12a7a5f81c61", "output_pcm_sha256": "5684f53bad9ee9f68c259ceefe88733a46cd0ccdcba13d97d5cc41ec4b6a19c4", "result_sha256": "73d03967c842d69471e1343866016e5a996515789c702eadd28b6f07eef1df6b" }
#728object_counting heldout	8	The answer is 8.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "20e9ac14cf764cad0b4ba7e568b9627b34528c90c4897efa254c261070dc4532", "started_sha256": "5fcc6a927eeadd3462123ae32ea67b830bd5d2f269a0a0a7e6999e02028c85af", "original_grade_sha256": "7fa4d4c5969e77decd87d432b3c915141bd9f759beeaaaaa4d5c47f69c452ef8", "input_original_sha256": "20e9ac14cf764cad0b4ba7e568b9627b34528c90c4897efa254c261070dc4532", "input_wav_sha256": "ab4edfb8f930e6cc5aa6b0ee45ad8b469ea72b746922c9ec9748f629f9acfa2a", "input_pcm_sha256": "b240669d82fa00d0e33446332b4fc930d157645630910fed99e88ea54b31fc8c", "output_wav_sha256": "b86793aa6aa3f5e7771e6058a21fd69d5066114e0bdf70c445c99f78e4747a7e", "output_pcm_sha256": "56d66095254863ef93c7fc609c95e63d046a3de94a27905148b243e1753c4ad5", "result_sha256": "840733298218d04e26571e4a8f31bc644d401fe7631c1f397391151fc6814bb1" }
#871web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f0c76fea370583f865c8510cf71ab6a871c65205d1554334bea7c12281dc1fd3", "started_sha256": "1c4a4b13920da2cd80d190f547600627b8f76a7f372294e0eb82410773c40532", "original_grade_sha256": "ccd2392e3f00298492806b68b06782c6149709c69668ff00dcffaf08e7b4a505", "input_original_sha256": "f0c76fea370583f865c8510cf71ab6a871c65205d1554334bea7c12281dc1fd3", "input_wav_sha256": "407b2b9d31ed81b7478f984c357528e39a4b14cb8daed8376a3f1f5bbd6aec44", "input_pcm_sha256": "b3a8f4c21ee7fd0931fa50f57f0eabe862e947a02f1115086f481849172e431f", "output_wav_sha256": "a205945a91da27735089c4fef2353423a93b1879dd5b41e9b03661e582bea117", "output_pcm_sha256": "ad5d36b3a5a08ef9e0fac24ea82849331a5dd0903a9ff47130e9cedd9ead6060", "result_sha256": "8f0999df05d88ab76bf16a36d17f705eaa3fdf2a01cbb8b996964e158442823f" }
#134formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "07f3d36f5c3b356a347d8a76010b1e292d2dbe8fba5e65a43dc9e1173fd72db6", "started_sha256": "b15a7ac4bc2b9c1247342baf799c338d4f4811fa217a3cb31c5bb18c33e46005", "original_grade_sha256": "c963b61454a1589f3a36692c53058d5033e5029ae5467dad1acd3c0b4a62bf91", "input_original_sha256": "07f3d36f5c3b356a347d8a76010b1e292d2dbe8fba5e65a43dc9e1173fd72db6", "input_wav_sha256": "4db100a4573315d697b20f9517279c0e971c9b385f0527bc661d0fed35f116a0", "input_pcm_sha256": "3260863e2ddb642e6c442132ffbb78d7f8ef16315d9507253d23fd2eacae88db", "output_wav_sha256": "0bb8d1cf1ea230258e68faf49c4ceb1ba2d32f0f4621071a0779c90cb904db05", "output_pcm_sha256": "5cca5259490ed55806b11e80162eff0bcb2dbd1230dec289dbe361712e794b8c", "result_sha256": "9401217a537a79b7f75879e9279ec4ab592dca9ce1f08357c9143d1853e9e080" }
#257navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "47bf59511e0d02fe37d183dc96c426e28bd1c6826922f6100d3887be97eb87ac", "started_sha256": "3e747258d028a7a0dcdfaa36d753a61d60a00897108046738553b865de84d674", "original_grade_sha256": "74668627c65c34a4d8a16dcad528ec1a516e724be41a88c2a406803326c037b5", "input_original_sha256": "47bf59511e0d02fe37d183dc96c426e28bd1c6826922f6100d3887be97eb87ac", "input_wav_sha256": "1d203b6150e191398cad937b1c162b9f6d8bdceaf8d1496133470acb195f316e", "input_pcm_sha256": "daff39879e2348580b568e8b42417feee2382ed507280fbcb97778afb72eb3e9", "output_wav_sha256": "9e459027ee68d2c9fe79a9e9bdbff87a3b17e51a1a58c4d4813547e5da261e8e", "output_pcm_sha256": "1a7a1be60e6a523ec20ce32c111b56de5c26ad1398e5acbed384a146c7768ecc", "result_sha256": "4ae16cab04e52da6e5f3dcdc9783f4e9055dbe09200be4f62f0e1978c40602bd" }
#700object_counting heldout	7	The answer is 7.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "aaab21f386ff2737e528cb3d674909aa7d5887af14ba169713a7ace2757f9d0d", "started_sha256": "bc0673f5007fbfa5b57adcf4188b118ba3070cf658dae440a7c23996cdabf006", "original_grade_sha256": "fcd90e40e3468999cad7c0bf92025dfba85ad68c51c065fb23050c36ecc7832f", "input_original_sha256": "aaab21f386ff2737e528cb3d674909aa7d5887af14ba169713a7ace2757f9d0d", "input_wav_sha256": "cf99c20d280de75a828abbd81f653d2887e2c944a3c4d97d5b83b8332e9c5740", "input_pcm_sha256": "4ff45977874176f2ef8207e71401b064df5fa1ce8b1e9c00408f326310fac749", "output_wav_sha256": "95243586e1def261e8f9c662935b8c081bfde8caaf53e48a54afada27310e8cf", "output_pcm_sha256": "f900941283dd5115dd61af967145f53186095728c82c4d6e4a0e8606dd43228a", "result_sha256": "b99d397659b9f4aed25c5bca04bf30d4309677e68620a67ff21a162fec87f91f" }
#799web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f45889a80537901650f358fe82cac866ff68ce880c10881334cd5c98d6e1f6ae", "started_sha256": "720c38c052eccab2daff46ea64ea5c2a1f83d417a4fba3e4f68bb3c53bed5a18", "original_grade_sha256": "c02acd01a657c857ba58b07aef1e698b5df75554ba5258360abcefe8e69ee07a", "input_original_sha256": "f45889a80537901650f358fe82cac866ff68ce880c10881334cd5c98d6e1f6ae", "input_wav_sha256": "7eb59ecb94f89f62e453c64addedf9910a9a35d307c76aaa0dae2346c16d4c6c", "input_pcm_sha256": "f75370ef324cf91913c8bb5506b7fef83e8669fe980cdab51c49d9a308481c7b", "output_wav_sha256": "d78e2ef17e7508d4891b01f35c21accccc3b5330fa5e6b5d2f4e7bfc16123fcd", "output_pcm_sha256": "ca1c5616d3ba283623e5eb9a1ca340964b4f98c4c463ff1baa963d4a7ef6ba3e", "result_sha256": "1632e1bb42550e117240d5241ef87c55215d5a3e36ac710a56885cac309f3d26" }
#186formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "28469088d9805699ec9a664d5c7638230429cd75f1f7701aeffe5ebd9c421cf1", "started_sha256": "a41d57f8520832727158ce948c57ec0cc36da4f609e051b545547f5556f30282", "original_grade_sha256": "3dd0d428892d2971abca0dc1d8a1b7daa36d06a0862bd4a15c067e5ef10ec3b1", "input_original_sha256": "28469088d9805699ec9a664d5c7638230429cd75f1f7701aeffe5ebd9c421cf1", "input_wav_sha256": "47dd06c827d65722bf7d558ab2a0ae849d57429ca68d18aa5d1a070d4499d5e1", "input_pcm_sha256": "cf08f14748e31bbbbd900b8df871a799908d6ef3a475e38b34df7c0b674c25c6", "output_wav_sha256": "500b40c3399b8c970a4031e69b3ef5f2b36057c949e3bf8530ccc8e0a62a1410", "output_pcm_sha256": "a5f48db8652787af3084fa02d59c60c955cccdb34672f4fd77d65a24e95c6f5c", "result_sha256": "934ec25ac2cecc8b4c174163bfd295f2ba2f0d6bba50c670053dc7a2852ff453" }
#447navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "d850aa324687b93b24f54268218f6b46d4c3480f6a7bf0e064bbebcd70b16f24", "started_sha256": "c2ebadd839ff71745534637e8e1de7cb41d3f95cee93f519b7ecacae75b61142", "original_grade_sha256": "7736f2859a38bea2e7604fed00a66904ffb256523e007ab48388b9633d2890e6", "input_original_sha256": "d850aa324687b93b24f54268218f6b46d4c3480f6a7bf0e064bbebcd70b16f24", "input_wav_sha256": "a6bbdc0fcacdfb98b9457bd51f3ccdc6c3030e74e6aeb36eff9f1cb3eee8a515", "input_pcm_sha256": "0fc3c709e76394b8dfe61c0b4320d5b567808676b35c86065a00ecb52889db45", "output_wav_sha256": "dd5fb4183ab8b628ec719d659541b5102ece50cea4a914b6423fa76db7833c2e", "output_pcm_sha256": "1444261752e181a6873a85a9dcb2cd74e438882e1dde3d48f0c613950a6dde3e", "result_sha256": "1658bc7cb2ff25c5d0d4601666b60c7f84ee3ebd942f82b9e331676d68049976" }
#580object_counting heldout	8	The answer is 8.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "4f0ebcded0a54d3b987cc5791f1dd3459a69de8dfbcf81b36dae45392cd2cabb", "started_sha256": "70055bd1114ce6c4815b0a173d554eaa2aea73c05135ec083dfc0e541ed0c1b3", "original_grade_sha256": "c3b981c3b98f9f17cce689a0649aa2a46ad9b9442f8dc2dc6f4444298bf19039", "input_original_sha256": "4f0ebcded0a54d3b987cc5791f1dd3459a69de8dfbcf81b36dae45392cd2cabb", "input_wav_sha256": "f436ac65a162e35d08f4acab91ff81e422546e6eb11f05ede77cf62675331686", "input_pcm_sha256": "1e2628cea1bf17aecb826424c447dd1faf4ed60dbfd7d16efc6837404e710c1e", "output_wav_sha256": "92386923ca19bf2c635b03cb297d855f8a29e76687e78966df6739dd5247888c", "output_pcm_sha256": "bd537644f16c688eeb22f679a9179b041cedc85be4a52ff513f2e2418ebda8dd", "result_sha256": "2fae05bae64629e37a441cc770fecca792a3aea8ee4bd63d704f2d52c3d03c1e" }
#843web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "037eca753671a96d7c9316411071e4296c3af9b10b457cf51c3da9c15b97f161", "started_sha256": "3a3acd6c21b0489e217c6b699e9dde488846eca4530116bc393d744fa8952ede", "original_grade_sha256": "93ac44a4e4107f605729d5dbcf080733ded0a6203594524c8883a15eff4b54d2", "input_original_sha256": "037eca753671a96d7c9316411071e4296c3af9b10b457cf51c3da9c15b97f161", "input_wav_sha256": "415275948b9688702afb4c64eaa74f91a6386998084c9b040be6abc7b52ee078", "input_pcm_sha256": "c84ede09cbab4c0f1c6dbd342f9490fe9eea030b5298bc714fc38db6d8ee9119", "output_wav_sha256": "01ff67e4f8e38fa7437de71f2ccb3d990dfe20a12fa930d87761ca638ae70dd1", "output_pcm_sha256": "51c7870043d94fd7616a7b901947277f1a001cdebde77fde86d6d044d6243dae", "result_sha256": "83522e41d65075ab223ab43b35c1ec5cb0e334dc813ad0a7186f15de5f449b41" }
#206formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "dc0706be64b08612cc16ebf5c038307f1c51f7912f9f4c9a81aa9dbf30180a99", "started_sha256": "d8b0a9a79507afcecd1dcc1f972df9fde10efd27cdbe74401d0e52f7d9e7a729", "original_grade_sha256": "f096986b0e58ff9e623eae64e6e77ec7d546cbf180facfcd25730844c4ed2b8d", "input_original_sha256": "dc0706be64b08612cc16ebf5c038307f1c51f7912f9f4c9a81aa9dbf30180a99", "input_wav_sha256": "d3e4bda235037704ca160a4270a20028050a9a88ae1025db96f306f35c844cd5", "input_pcm_sha256": "82ee7f35bb9548af24e21bfccb7b91cb640690c2bd11cde819d237e6a222dcd6", "output_wav_sha256": "e64a2e65650d2bd9f363f76225ab6e9d2377145f2f47621799e69c5b09bad0b6", "output_pcm_sha256": "50b5136ac94525fe705ef788cc79582a7b7bf188c729d5c440231a6da97840fe", "result_sha256": "79ea85d5df7f6177ab2330331116147615ced8248152bfdda9c4b2b85476cf1c" }
#367navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "03339e7284a5e16d41cd7a35b5073f2bdb170387b39d173f23066423871bd0de", "started_sha256": "f00307be27e88d90a1878e0f04d4055e0a7cab842cb54d5b6b4599f6009b95c3", "original_grade_sha256": "37b9539b5ff7436bd2e3ab8423fbf2901d16ba0c49f38fd3293abfd1790e8df5", "input_original_sha256": "03339e7284a5e16d41cd7a35b5073f2bdb170387b39d173f23066423871bd0de", "input_wav_sha256": "3b5c20de55d4f55ea13eb88bf367e9cdae1277b317c58566ecaa1b548e3fd5b5", "input_pcm_sha256": "7feb4037b41401bd29659571fde43958a53743cea3799c1632b743d090c90cc9", "output_wav_sha256": "89244a4a441043f3bb3126f7a127f5de228050acd7f485d1cbbc87655bbe25a1", "output_pcm_sha256": "4017778a20ffca6ef66f9a545b4f80449af43bda5ce7165e9d267fcb60697349", "result_sha256": "f37d95243cf33bbaaa655508cf3cbf089d595cc8577a8a4470b214c5f5adc4e6" }
#541object_counting heldout	2	The answer is 2.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "6477a4da840acf871df09c042627033c8ef69f52440a5d5ec79ad9896b66c201", "started_sha256": "35387d9003735020008291310a1c18b27cb68bb3edcc6827acce8bcc9d4a50c1", "original_grade_sha256": "500b5e24e3247744cac69a900da3495e4972412d6f79ebfe677cc1346af9f3b7", "input_original_sha256": "6477a4da840acf871df09c042627033c8ef69f52440a5d5ec79ad9896b66c201", "input_wav_sha256": "c59f3966e46e8b992cefab24f64292d22e0073ea449e5b5bcc3d01a97ee3c395", "input_pcm_sha256": "e822f497a97d7cc61865aecf4847bf6efc76853e3f995ffe53c6085be1f3a423", "output_wav_sha256": "51f4bfd1c8dd9b41ddc7b92d3b66b497344ea19c528267a2ffaa5786a91a9b57", "output_pcm_sha256": "ea3702ce3963fd4c6d1c68aff6bbf0a8d4cea1f48e443a841b7206d9cb73b906", "result_sha256": "d66774d968862cc2cb37ed7a44333d89f5c97132ec62c07b94e449d98ed303eb" }
#909web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "3e01539d93c1665a19a9031df355c3f62ad2c6cd04807ea5e39f4e66622af5ab", "started_sha256": "f9275a71b1550873b95a42355acadd26900d07f38cef669c1fe9c5a3bf935f63", "original_grade_sha256": "478ab7c3767ab738bc8742b07d9d58a542550c943a129713eb648ddcba777b9f", "input_original_sha256": "3e01539d93c1665a19a9031df355c3f62ad2c6cd04807ea5e39f4e66622af5ab", "input_wav_sha256": "2707a25fd8c22e9edbc4bd92c419b68faf42c34290eb3e6c0bf2bbfacaa71529", "input_pcm_sha256": "399c354e2f8446e615d96eb560a455b4f689f833ad9c451b39ec2eb5409f7b76", "output_wav_sha256": "0cc474d52765cde11813463771b87d4eba5f0c1142beda646e904d5b4468c7bd", "output_pcm_sha256": "746392ab10e0212d69580189ab05dd0c1c7a19f1de10878675023c4fa7ffea2a", "result_sha256": "7e1710a7d99a335d9b558a4dce0631a550eb4eb12c6465caffb534a8d035a8fb" }
#58formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "38616addf8daa0811809ada96502d9fad769959c4d076b73f54c0ead15ca6749", "started_sha256": "22297b4220ca250cbe38dec5e367b508086ccc1df86fb93b47594564c62f7cef", "original_grade_sha256": "3f1984c91c4331731974fbf5f71b7f223d00c2a5921cf4214a64e557e7f0ad5c", "input_original_sha256": "38616addf8daa0811809ada96502d9fad769959c4d076b73f54c0ead15ca6749", "input_wav_sha256": "cc1a3409d588e3204bf48059abc416281ed5edf779826f421bb04c307e86b88f", "input_pcm_sha256": "e204cc5a59582ad400531c00cd518197da3b3f82add503785051033ee1d519a3", "output_wav_sha256": "d8f3484c8959a199bcf4e70c6a0fdad70e5cfc4d3a8f359837d5ae50347c99e3", "output_pcm_sha256": "f43ce91373e6b94b5f27014add052313d6b5c2257e259c4005217e4976db042d", "result_sha256": "fd3eefadaf75d0d603db8fa30052bd4ae807b49efb8fe28790a6014ef1243f48" }
#444navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "56c540254fa88bfe05cfd891e8ad26f332bf535f7d800247d8c24208697c0ed9", "started_sha256": "060cc0ebddd99e5a09663e46314fa48fce94a86ec7dc039bf16ebd5b428af6ec", "original_grade_sha256": "89d38ce19eb65603b3f35d1711b4f2a4e36fd47e077551e514d22362a4e2ccef", "input_original_sha256": "56c540254fa88bfe05cfd891e8ad26f332bf535f7d800247d8c24208697c0ed9", "input_wav_sha256": "977c70f5272050d8b5ec03a67f8551e3f8732ed3481cdf8bf5153aeeeb9fc7b0", "input_pcm_sha256": "154f32dde9b9c74fc3429be4f50b66b885bb3ec77342ded48397dc65163fc198", "output_wav_sha256": "125f33bfe82d7b36a83c06bbee0d48d481a570fb0bc25947331f4afe7321d227", "output_pcm_sha256": "829b116de8c3298c8adaa610617e9fd94ba17ead7bdaa4be065165c09b1c5710", "result_sha256": "05e2067b0b385f17690e1b6d5f5fc3a6a24704b7e0c6101e8757e9ab12571ea7" }
#518object_counting heldout	3	The answer is 3.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "4e1cfe60298b734744c79e06abc3b387091ad089716811a158d5af78939157a3", "started_sha256": "f619e4c7e31d45490193e3aa6f4e15ace75d2e11ded8f53415fd3c591742ee8b", "original_grade_sha256": "4254f8043b674cc0a9fcfd4fb0a7cc4e554d43a3fcd6e8d034de8b7ca0b5cbec", "input_original_sha256": "4e1cfe60298b734744c79e06abc3b387091ad089716811a158d5af78939157a3", "input_wav_sha256": "541779fab0b2b78d96d83c852e64e150cd247fdba8e5b89ffc94697380a80621", "input_pcm_sha256": "2b055d36a92727ab86f6a209674cd1a8def9974f574254f79034ce25695937a5", "output_wav_sha256": "3a8224c53cb4f34bc25309df4ddf70392b8f56c2ac8898b81335758342e40417", "output_pcm_sha256": "fd4fe29756bba2288fbd50aa8fd53fe085c09ce65ef1712bcfdb5da8c968df46", "result_sha256": "a5e155bccdb3ef6367a769371014020a10a3277abf6379811bef53edbd366a85" }
#774web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "06a312c4aa091ca89f9fd7443a4ec03a92ff101d037ce7dbc979d8aa80e68e30", "started_sha256": "8fdf7b2e10db98a33f18794e80108a288bc4a76b0bad9eff82baaca8993aa325", "original_grade_sha256": "ca59ac0c3d9399a2cc8b55c4d0f99fab83a9a4cab564b1de657b742ab238a492", "input_original_sha256": "06a312c4aa091ca89f9fd7443a4ec03a92ff101d037ce7dbc979d8aa80e68e30", "input_wav_sha256": "e18b9f087a5e6ae50a8ba6c24276aaea062062927dac5ca587a0cc613c993005", "input_pcm_sha256": "85b180fcf023b105dca78d4a2f9a7491e9d6857e9e0a278386eef6aca20eb82f", "output_wav_sha256": "9d6cb5a335a3b5fce7f9a3d8e34af2a192903792d54e6117e43045dd52e06989", "output_pcm_sha256": "4fd66703c15b34a1e9369b4ce6f43effe87b21cc35aabcaea1a8976ba35e7121", "result_sha256": "356963abf38e8075b0122744a81afb1d8c856b71271eec62053e10c46df745da" }
#218formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "7dab24b27fba13310bfd4b5a0c42da422e5876d9588d66303f7a887e90244f27", "started_sha256": "d3dc092334e00d92eb97387a02a7a15f8851943d8edd150c65ffdbdacc766165", "original_grade_sha256": "4086ed91c4ec9fa84b7d2fd2b1b8a17b379a2edd0c9ba2f2645220ad0f392a50", "input_original_sha256": "7dab24b27fba13310bfd4b5a0c42da422e5876d9588d66303f7a887e90244f27", "input_wav_sha256": "f15c874143f8eff5f07e72f069b94b0e2676f78392fce49d14da384fbbb00b3a", "input_pcm_sha256": "8b7a29eaa1051acc6f735c173e047d12df1f77e498b3e73802a5fa6722b3927e", "output_wav_sha256": "c07be8c5edc123bafb9c99e45635fa3d41252ad4b4404dae33ba77015ab212cb", "output_pcm_sha256": "db2097e1d7cb56679770fc9bca588c864443ffc2a921abc2abb0488dd2c12949", "result_sha256": "de1f6565523726338673e1ff19197030b73a921dd835bc805e3ed9dec85da0fc" }
#337navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "24b391e46574adfec797e980e105051eec46e64d86f2c5fc4ee110fca731d90d", "started_sha256": "d77b4ed4564e23f58ea3ed29ba2f8fd33101202caa0c98406c0cf4070c25620e", "original_grade_sha256": "df3a7ed61cbb211874c66ebc014fbf03209362ee4e1d15d7306dc78bf6744589", "input_original_sha256": "24b391e46574adfec797e980e105051eec46e64d86f2c5fc4ee110fca731d90d", "input_wav_sha256": "09325eb4d4ddd1b437af21a0f4d1b39c6ae656145dcc2c15ac21993607b403c3", "input_pcm_sha256": "8993c65a4d900e2aef636a93bd2c0ac7c663bc4c0e9ad1fb7f5186266dd1ee13", "output_wav_sha256": "0dedfc61b9de785a8999d4f0a6f9804a4e8a403c4f02da5aa8642e77589dc125", "output_pcm_sha256": "595f07949abca409404a345da4582e668d32bfec044370f994f6ba01dfe92285", "result_sha256": "ec616d743d8a5f77df2ae91251e7106530ed7b5d00304310720c7a734b33d5b7" }
#572object_counting heldout	3	The answer is 3.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "7771f6c74e48a3160ca6b67789e21a6c8be91349907e6c107ed959fc00718200", "started_sha256": "d057e4e97b265190ac903a7d113db704dbfd90452a5419bba369779070e3c859", "original_grade_sha256": "dd8e241b1dbab81651e708168905b335ac01284701c96f6e3eda5fe634b735a2", "input_original_sha256": "7771f6c74e48a3160ca6b67789e21a6c8be91349907e6c107ed959fc00718200", "input_wav_sha256": "1cd7c36a626f30d0b21fd1c88cf8b03c84998235d87b4644d4b2618fde0c825f", "input_pcm_sha256": "ccab1cfecbb9c0bbecbe309d7100067c0e5b312beb29bced8b32b193af4f0b1a", "output_wav_sha256": "4e3003fd37fd409f5a4c6a2bbce6fe2bd16b59bb0fa2b72aebaa936db84293d9", "output_pcm_sha256": "453eb3e50740cf2c6a9306448b8ea518e6545678ca94babf0350a07d652065e4", "result_sha256": "585e22f921e7431d21c033e6f11d1f9759ca79c2c22fc97a676dd02dfac7f72f" }
#832web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "ceea2f9d981911f8be7eb206dc526ca71f7e2253fc6a81dae0ec53d955ec96bb", "started_sha256": "5177c888d9b809608a47b23a2749f8610ad83227e1b86a4830dca975d0a45d6b", "original_grade_sha256": "5a0b425cc72960fb3f621a88196fb9b87b9308e2ba5c04e2c01a5ef28c85461e", "input_original_sha256": "ceea2f9d981911f8be7eb206dc526ca71f7e2253fc6a81dae0ec53d955ec96bb", "input_wav_sha256": "df51bb620b2170aa1fd91348ce58db90d806afdcefc171ba81dc6be06a3e4772", "input_pcm_sha256": "e0d5626749ca0609f87fe1dad1286637890db4355a5bdd0e598aef9817774370", "output_wav_sha256": "87089fb171e930919382694b5ebd45315b48ff6dea07ca7ded7b792cf4d652d0", "output_pcm_sha256": "2dd7dd730b490d57bb2af622e7ea38e0dd8d7af8ccacc0dac8a9c88de13da434", "result_sha256": "4cc7ca439dc4d2ac9f9c5b4e4265955f5d804c2c64f4ef40626d7952dc81e9d4" }
#183formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "ece00a5d6e35f2794e0a3f890af1a24807e4e172628ffc5bdab51d73594532f8", "started_sha256": "68c5cf39c666c5f8e0eed581eb39152948ceb9ceae333c43f05000268c780c8c", "original_grade_sha256": "35974dcb91f93f1bc968405952e2b8bf767f148274b2e1f25eff36feba7396ae", "input_original_sha256": "ece00a5d6e35f2794e0a3f890af1a24807e4e172628ffc5bdab51d73594532f8", "input_wav_sha256": "79a44a60d5f225eb970ce9caefdedb4e70d58bb93df68fcdc07e101865fd14bc", "input_pcm_sha256": "09c7c722d20300a9875d836b3e3418e736215cbd6a3a2e7698db5e8c52b4b79e", "output_wav_sha256": "d32e467c08a562e967a5c31d7d04d4ce6d2e94ef10a7ec0ea68af1e4895e7618", "output_pcm_sha256": "92147654fa40d513196787a468019d9cfc153b479b8de0b25d9c9e73ab14d328", "result_sha256": "85ae1104277d538198cafae23de48d491c4a24d3cd86438bd30f078674872f01" }
#311navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "8c98f8fdabf3aae97ba20f8ca8b13f4dab0cb01288f14f52b12a490098174ed9", "started_sha256": "a88380e3ab52ba097b5cb90882e89104764c0fa2c84b92685fdb9b99ac5702af", "original_grade_sha256": "7927d8142461a8c854cbe6666855360c125eff82451db6652369081622cc52fd", "input_original_sha256": "8c98f8fdabf3aae97ba20f8ca8b13f4dab0cb01288f14f52b12a490098174ed9", "input_wav_sha256": "4df03873c3b9655be72c7d84909027b2d31a785ffabc107cd4667f011b851fec", "input_pcm_sha256": "5a003fc9c7844714cb8cb8927d0ccb5fdd4ae9cbbc0616efa514d0ed7f5fd689", "output_wav_sha256": "df7ca33a8245939dff2014df9f9651d641b930d211bd689c7b940dd7144959ba", "output_pcm_sha256": "1145b41028d89631964a4065d43ed01d3e9f037551a5f48ad113fb3a3e008da5", "result_sha256": "fd765f55454271af79ec3fd8b39930bed857fbfdb507fae5acc509b6bd7d3abc" }
#600object_counting heldout	11	The answer is 11.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "290de8b74d1b437f18620f2aeb50b1263eb4532e6a1ea52a1d8f96bddfce1402", "started_sha256": "228434df6831520839ab5fa565d16f80aa764eadc49f083d7853fe9085245d0a", "original_grade_sha256": "3126d3b8c008075b53582e22a8e7d32ac57c025f71c426e61edf00eb02eef089", "input_original_sha256": "290de8b74d1b437f18620f2aeb50b1263eb4532e6a1ea52a1d8f96bddfce1402", "input_wav_sha256": "a2ca564336b715489e8231a3ab8adc7dd52833be8e5c581b0df67a0372923914", "input_pcm_sha256": "e9c394269821a63b11b7d2a2e0ea30448ecd29c2044c5bc1248eb176fdfeca31", "output_wav_sha256": "7f1a6627a63da919956577b5036e23d0229506753bd3a927ad94e05409282c37", "output_pcm_sha256": "7e94805c704eeabca6e88a69ce47965ac38c1de3f80960eee2d9f018c753d588", "result_sha256": "a06a1812925961fb811375bd623cd1d317acb34d186e36e8c43dd7982d700f04" }
#947web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "7c7036cf71e20ee45f4784df69f7ef210455ff8cb6bef274a1226bb7e3140d3c", "started_sha256": "25bcc7ef896fede21411f80d9a6ec861400dec4a1e6be4dc634aee6342a63cd2", "original_grade_sha256": "a0f6a5580a2a16894aef5da535df8dcdbfd82858bd5d555a876e5dc946a1fa1d", "input_original_sha256": "7c7036cf71e20ee45f4784df69f7ef210455ff8cb6bef274a1226bb7e3140d3c", "input_wav_sha256": "38fb09290bdabfbc37b068e11cc5aaed5bfb3aa2e74cea65715c6cf345827616", "input_pcm_sha256": "5098c5ca6196878c6e97ccd17151d392d06ae2a3af59b8152d61fe33c98c2399", "output_wav_sha256": "0673c7afb1f5910799e09b09d0718d549c1d3c019c66c9d4cecc7cd6d08e291e", "output_pcm_sha256": "cea9334e0446c022b47155d36f7a5ccf65792c56dd04dcfe32ac92d8b0e21084", "result_sha256": "b07f38cb0988139c7b19184e455f5895e74e09e9013f68a1f127076ac1e25d22" }
#205formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "ab09aa496231f22f98f047973f4a23aa0e51eb1852d3a9dabcab4011709bd05b", "started_sha256": "c5ceff2b8ef9b06462ecb8df72817a2813fd47d0fe17d778ebbb3ef350da2514", "original_grade_sha256": "6bb8ea68c16ad7ccf75136d004c6c698e7d6dc206b18948c7f1771ff857fb2fe", "input_original_sha256": "ab09aa496231f22f98f047973f4a23aa0e51eb1852d3a9dabcab4011709bd05b", "input_wav_sha256": "9f9b724ab49dd7e1631876f20e55f50e6e01854b3efc48a2e29fde68c73b90eb", "input_pcm_sha256": "9dc27308190a0b504a4b574a5e608220e90d4b17aef1b3c23b9fd9a43d19453e", "output_wav_sha256": "52bfb670a163a81ef9aafc82518b3d1df9ac50b05c947f83491efa9907a53edd", "output_pcm_sha256": "0b94d197ea9e0c361c51609221cd89bcd82da86e1a8e2d70a71479e1c1c9bcf6", "result_sha256": "0ea54be419087cb26adfbf0ca7f16d135692c6737b97c5ec5a6c05bea4e735d7" }
#385navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "5936db74ad1410a69e2c9b5a323e66dbc8d5b17d74f809cc5f698a23d69f7aed", "started_sha256": "9270009d6abc45da582827730e1e4721e7e5e95ae2e168d30e270f0e803e105c", "original_grade_sha256": "eeec5b3af9c9df69e288a5e2504d831f0f1c13b413547014e34b150b2e715dbe", "input_original_sha256": "5936db74ad1410a69e2c9b5a323e66dbc8d5b17d74f809cc5f698a23d69f7aed", "input_wav_sha256": "771e4dfbe5df4de7a402b95cec09d3034cca3dae6716416549765eb1afc9eb43", "input_pcm_sha256": "44d6457d160b6f3a98546d4f049cdc6fdefbfdd6b41fbaf615f70e0808134b50", "output_wav_sha256": "c0ae4078533ef1ac246f1165410eeacf115e6c362b77875282db5fc79f396405", "output_pcm_sha256": "7805688a0e44b668a873bf31ac729b986fe159204f71364c01f4eab2d4afdb4c", "result_sha256": "0a4979db92b9d8775d7de921fd38c061a41959c74168322cc5883a01fbffec25" }
#724object_counting heldout	14	The answer is 14.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "0e207c006a615654ffe32227a7765c43cfea87c3170314b7606e5bab91e343df", "started_sha256": "0570c6161a792e5f3f2d3e5ee46a3b0ba2cca2c10ad6aac51b67c38f140db53c", "original_grade_sha256": "35dce787e6ee662db11e32f273cd4642a466e4bdac218027dd40f532be822a70", "input_original_sha256": "0e207c006a615654ffe32227a7765c43cfea87c3170314b7606e5bab91e343df", "input_wav_sha256": "4cdfbdd3592484cb1672bc98560878cae6507ace79eb1131562dd40f8bbcd414", "input_pcm_sha256": "b100db0abba05b2be78778cd7bd83bd7ce21fb2559ad428ca5c9d8f57704d086", "output_wav_sha256": "ad342e639f874fb13b6844afd62773dd2111bcbe812e36e2727247ceb3257034", "output_pcm_sha256": "6963f1c821af26653b6cf36bbf1d683981920f88ed8c0563e8d5768e3ef1dfa4", "result_sha256": "ba17e0c62c0524ff10c222186fd54e50eb82cdd7f197a2fc449fa653a1ba04ee" }
#855web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "36ce48e10c6a3062ad3a622a0ab4b75ff817cdaa794281f006e5f99af83d9705", "started_sha256": "85a7c56cca4a17ae9b989f9628f4bc5e59f0171d7a741894e6039e9e238da9bc", "original_grade_sha256": "417166038b3e11ba71a1768290bd0cd654e2d596a5560e1d5610daff1f30a805", "input_original_sha256": "36ce48e10c6a3062ad3a622a0ab4b75ff817cdaa794281f006e5f99af83d9705", "input_wav_sha256": "8502b06895dcb3e6fc17553ed4e7b6296666341085dc53adb1fe0df959d0cf45", "input_pcm_sha256": "7ab86c56f33027d4214df9517c3c699a7a7e9d7d782769016c8386cfcbb774ba", "output_wav_sha256": "4246497493d557b1066b00b55292d813127f4da2d72bdbeca33fb8f6937dfe31", "output_pcm_sha256": "9bf5676445d512e4a88189b8a1a15a39a2897d424b6f59b5d6085e7e38337e74", "result_sha256": "c7fb8d5d72d873b9339a9e1a774060fc6ba18ddedf1ca38e289ce7cec57e3b00" }
#228formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "6ea21d065233e9d0c71877fa14e4c8f0edfdbfce81c6ef73bb1e16039fea7d61", "started_sha256": "889a2cf0c5af91b099cede44dc96366c57416774b5f64d1e6247573b04d5cb77", "original_grade_sha256": "892d45dde7d11c34e5a14bfc22d6def916739fc3fc0cecb674a18b3df27bc0f5", "input_original_sha256": "6ea21d065233e9d0c71877fa14e4c8f0edfdbfce81c6ef73bb1e16039fea7d61", "input_wav_sha256": "90759a0262e1976cbc03e82d46150c117c8ab8aa50a40abc6c75a340ee056c32", "input_pcm_sha256": "c5c236fe75a26fdc878a89b876bf2831f646f43754e1eab6e41907e25b63ac6f", "output_wav_sha256": "72e5b87860a4c670c7d176db3a24cac2e3f98ec6f93779f31dd26a7bd05ea67e", "output_pcm_sha256": "b5304bbbac445723820f64dd2bd6c50df7b2a0b31219b938c58d3cda783aea25", "result_sha256": "595018927d97530cac656851f52035856c9791720587005d57453ffc0e682872" }
#303navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "6087f5ad4af9930b5dbe267dbfbfb3133f79e1b0b5beb37f69cce91bc202d295", "started_sha256": "c2a46ff419dd18b328c3b284e1e3be8c7addfd8c5d0967d1357dd6056fde441f", "original_grade_sha256": "7fdd1799ffc0c101bdb8f8bc983692aaef0662f4cb4623485fc968c03775f243", "input_original_sha256": "6087f5ad4af9930b5dbe267dbfbfb3133f79e1b0b5beb37f69cce91bc202d295", "input_wav_sha256": "72540eb8cf4e3efcb453cc346d3564fd5d21a2659e5ebd1764f29f88b59c00ac", "input_pcm_sha256": "5f6aff1ada7f3ef76f220203d9de1912557072f37bffb4017daafdee4f15e324", "output_wav_sha256": "e09f0821f1f1300ef79c590aaac774098e9c82a730c6181342ca025c8a1cbda0", "output_pcm_sha256": "aac17990797b46c48e3ee36423e7a18d7b8e3d1d902451a2eccab1c4943752c0", "result_sha256": "b072ce2a512baac5f4c516a71487752755751a77099bab6a810cdb922d5ccc82" }
#521object_counting heldout	13	The answer is 13.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "de7704ee78a76ca823c0f60e0c757c9a451a986e1f443a1758a993b5e502e98c", "started_sha256": "8205b07faded994693b321fedf8cd3358c956e0b9951e4503af277235ea0310a", "original_grade_sha256": "d72241e5cf350ba6578aa186e1d775db266b6dc66d6b6c056cdcffd13093ec60", "input_original_sha256": "de7704ee78a76ca823c0f60e0c757c9a451a986e1f443a1758a993b5e502e98c", "input_wav_sha256": "69eb4c9d9ef0e7a5204bcfc35633b9a677cd973802d36fb2dce75081104a4a82", "input_pcm_sha256": "2e8a4d32f3eea0fef24ee64673f1bb9244d752bdbff515c0c3482eaf4e4b8cc3", "output_wav_sha256": "35eeb8df32b88123e497f567f99a33369f959f4d0510514d6c8f0d1075d07cc1", "output_pcm_sha256": "bf7aef2c9a816d63ee3db0b4119090687d021fe12328934c608ae93c742d1758", "result_sha256": "2174b9fc6638b0766a2179a822c1e84ff9830b1c491019ea65767a13a26e2e46" }
#882web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "38e657bc20babcee5341dc2701e95400f3e4653dae3a1c27b8df88d3e93263cd", "started_sha256": "60bc3109f415b1677bcaec2b5cdf9df33dd9c4f0cc272ed8fc5ad81eba15c9ed", "original_grade_sha256": "7e63f18def67017217316b7dd0da7936ccbce80b602f6330c4fb89c473fd203c", "input_original_sha256": "38e657bc20babcee5341dc2701e95400f3e4653dae3a1c27b8df88d3e93263cd", "input_wav_sha256": "a8aec0456d9f176d2bac4e05d67e3a6e468bdd85fe32bb0f490f5d3f96c992ec", "input_pcm_sha256": "a3a3ab92b11d8298b2bc4763a6b2502cef6da63067da75ae0c9ab10b08fcf8ab", "output_wav_sha256": "5495e9fdfb31f0f71aa10b72a0a82c98d9926ed0a67cb64b416bfc1339c9e339", "output_pcm_sha256": "20db3691bb2be2848d8f9144a4ab803eebb6387cca49d63cc5aaa37d2d3751d1", "result_sha256": "8d446698e159c49388504622340b7e9d955d0170d347e9698e961757e961d3e3" }
#243formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "541a7604d034b03a3ba7f6145ee32dd9554bb4849224bcd26b556bc71ae75d19", "started_sha256": "a39a25e59a54f8c8537c5f1425f0cbfabc939cffb520523065a3ca8b7884a655", "original_grade_sha256": "319a9e6f6c6600036d3982d8959bf872eafe01a8c5edc3405658fb3382bbe83a", "input_original_sha256": "541a7604d034b03a3ba7f6145ee32dd9554bb4849224bcd26b556bc71ae75d19", "input_wav_sha256": "d030958c1985e67ed43f39b4644ed69ab0e1c32b960fac28a41d9b68a933eaa2", "input_pcm_sha256": "2552b22aa91a1f8ca7bc81c66d001ec8d1d53b51f6a8badbaeb140566c570c33", "output_wav_sha256": "419a9c9418efe7fc36de5f4e41c3138145e92ca39f4f2b9cb468c487c16b7949", "output_pcm_sha256": "d85779360e0d0c49422f42952b941a2cf0636ddaf2ea51c847b29751a7fd812d", "result_sha256": "7f79d910c55a2626e1cac4dc75e6bb466955a07c3d4f041d0a855e79f1cfc9a6" }
#465navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "d69b4c637aa9a9eee42b2c432add35a7979e49bbde486a3c545621fe263c1af9", "started_sha256": "c0f5dfac0a3526f1f8055c4da7fdd5560046a8910d3d5bff04cda264570c6fee", "original_grade_sha256": "3c761a6461aeecbd8657a3190074c6e69262f1297a153dee15cd9bcb30b39055", "input_original_sha256": "d69b4c637aa9a9eee42b2c432add35a7979e49bbde486a3c545621fe263c1af9", "input_wav_sha256": "6daa24f79804a9f20fd8082953242c86b8dc6274089839a82bd725bdf6be31b3", "input_pcm_sha256": "5ef21cd0dd6270111dcb377cad5a361ac2648541ffb9200402edf9dc307cd12e", "output_wav_sha256": "1bc2226fe78a538bef3d45029e4498c39440605c915ab6c95adb8f19f8a4eff3", "output_pcm_sha256": "2713bbe6499391337b3a296f771a53a194dd2f55a80a9c8e1e3d1abb2a979f6d", "result_sha256": "6181e574de200663c0447ea81ccad0ace85137807ac966c8e00e46d1a15ee7ac" }
#504object_counting heldout	5	The answer is 5.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f6df433621c3fbbf1927d751a9a2a1531e1f56dc51edb86a3974098dc25e5798", "started_sha256": "9f6150bb2967a81e3df23f6cfeae2313d43e4e371e6537e18f73387648f22d6b", "original_grade_sha256": "3cdf9b7fc6a68e3090b3341b5c34fbd69455f1b84625d3a77b40f88cedbe91ee", "input_original_sha256": "f6df433621c3fbbf1927d751a9a2a1531e1f56dc51edb86a3974098dc25e5798", "input_wav_sha256": "550de3e48cc81ac1c848c5d19c67263d16732c068f27adb29d6da50afc6c43e8", "input_pcm_sha256": "f604c23243f1c2cd67c1d82b410df1c7d8c28fbb86803dc0af5fdfa584795035", "output_wav_sha256": "ff203c7873caa1290ee670e4c99b3492f75fa34430f4283e5490f318ededb2ce", "output_pcm_sha256": "19bc5f9690678da80e3c04a18313462d0c7a7d25e7599a60dd493ea46f0c3ddf", "result_sha256": "c8474d6995b22e9493ae5dc45a240c51c4ebc859c26b3c3f7e18c3d4ecc0a48c" }
#946web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "0976031d200e06ea5620cdda531983a5f8bcebf634cd107dd4ead082975b6f38", "started_sha256": "51bdc8be71a31ec45ccab2e32bf0171dae570ab8f606c1ddcef8a5e285016918", "original_grade_sha256": "5e6774db83d7c8d08be3b483ca8c34711395f3bd49de1cbc1f6e2a1d9dae3b7b", "input_original_sha256": "0976031d200e06ea5620cdda531983a5f8bcebf634cd107dd4ead082975b6f38", "input_wav_sha256": "12c732e9628bcf384148c26b4f88d0ad12c55b8ac02b7706ced5de719ad47b2b", "input_pcm_sha256": "017e4799f94c6e48352a7425faa750fb17637b2ea3a2bfe5b59a8bd791a0f437", "output_wav_sha256": "c517a0116e6652c9713c8127103e67a75b2cc9e88eab471bf3ad588931da7920", "output_pcm_sha256": "6dbec9d7434cdc1873cd195da69df713644782c1a71ed78c473ff8e7687a1197", "result_sha256": "f35c971398406a3f90512982a2fdeb80826707bc5fe3f832e1b14b6b259afcda" }
#194formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "fe32b1be753173052910b15a69c6fe81b5f8f0cfc4ca4d91386cd08e7209e235", "started_sha256": "ba123a4f8c9ab67e1fa624015fb0a3cea203f2c59a69cf84d0b6b6dfc65c25ba", "original_grade_sha256": "61291e38e86a8b05e96c44886b0d58c0e89602dcc78152143e0395cb73683666", "input_original_sha256": "fe32b1be753173052910b15a69c6fe81b5f8f0cfc4ca4d91386cd08e7209e235", "input_wav_sha256": "dac06ef06a71b5dc0e06fd7724b88ab547eed3b582d0b6fc7a87236eb1de8e41", "input_pcm_sha256": "2124c66f2834df2bd8f42742ab65e22b35f9aae9548e7a428b3933dc91994a96", "output_wav_sha256": "ae9ea29acbf58f436a072bc400debc0e87f7781193cad76439919584f65fdc4b", "output_pcm_sha256": "e2a2bf29d5248ea0dd68d7315441699d7315beaff0a8dc78385134711718d0d7", "result_sha256": "c20d672cbd8b7b9bc61af84e121fc5e00400ac512f7b6881291504ac7aca905b" }
#472navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "bd42d662f0156f77aacda2f44d9a9c228f3de54547b6729b98bad844f4656c28", "started_sha256": "f21c19d97ec05e1630cf5f15427b806e8d3139f20e0699d9e4af8726a5b98990", "original_grade_sha256": "2a236accf343feb20fa3deab1b1245bf4b99423331edb37afb86eadb718aa843", "input_original_sha256": "bd42d662f0156f77aacda2f44d9a9c228f3de54547b6729b98bad844f4656c28", "input_wav_sha256": "06f4d425a162630e2f12067c5eef4987380029565fb62105bcf843b18281b25b", "input_pcm_sha256": "aa0c8f7809cb6d14d8cc970ed34a3cb93712616fddb75a4130e3c92198712e12", "output_wav_sha256": "e5e9dfb7adbcd61b9cd89e3f192ca96a8b6190ac197b7b16b0f74a13306238a1", "output_pcm_sha256": "da048df5afcc1111d0f5bf780a96064ecc7dff8ee003c7fa40c789d725c83e16", "result_sha256": "26f8f367acfd9960f57d776bb3263376d4a95ef01671479ea3deb2e677251aa3" }
#543object_counting heldout	11	The answer is 11.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "aaf926d4235431b3640e73a73767d59bbfb9c6443660ca90aeebbb72c9ebfe4a", "started_sha256": "0ef599538b51eefe3e43ad52b45b2e2c24d0b7fda994cd74dc594711ded8f833", "original_grade_sha256": "4c6058bd308e704955a6a610021ea19ceeb9548b974c6c0931f036f8aabc84f4", "input_original_sha256": "aaf926d4235431b3640e73a73767d59bbfb9c6443660ca90aeebbb72c9ebfe4a", "input_wav_sha256": "8318d1624635ac9df8abce85b759b7f80fedba10cc5b69a9bc2df7391563fdf1", "input_pcm_sha256": "fd40cef599440f4e46e2cd9ce99ecd73980a4f5386f69e113527d0d5f3394b2c", "output_wav_sha256": "32210e56ee3cf38fab3db40397a809f3dd6a77cc6bec526dee35c56dba772b17", "output_pcm_sha256": "d1a479baa929d17ee6d761b6c00d195bc0fb0d3cf4ca475d79af3146c084d307", "result_sha256": "64bfd41f162c5e5aba015761d4d68326c6a7dc6950c0d715dab68ceb69113f00" }
#773web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "77c58c29ed5e74a7844e9e3358fc7af7e0fd5b8bdd4a8fd72c14095058120c80", "started_sha256": "ea52f5f07e9459d29f21eb964540394b058c5f53b217e2baca0e4878ffb7de6f", "original_grade_sha256": "6e4fe81ddc599048c08d78379388141a2be554b3dfe24b322daa3c80fd44957d", "input_original_sha256": "77c58c29ed5e74a7844e9e3358fc7af7e0fd5b8bdd4a8fd72c14095058120c80", "input_wav_sha256": "3d1a92abed761c3fcb68cbdb5e1db3fa6ec3a98af9fbc03f92b32a182c56f973", "input_pcm_sha256": "8b9c58cdd2c73c0284b6295f6bc2d3c1527529b613f1cef7b038f6b355378304", "output_wav_sha256": "82ecf1c17f8734b4bf5c00d3ac5cd99b5a90b80cefc9ee72745673e61081a7ea", "output_pcm_sha256": "890e130ad419d0cf4ddea46d87b5f24bfd1829b4ba807daba1fe3c023e9ea366", "result_sha256": "49737b175dbdcde479ea9afc8a88dd45e0a92f4d866c3ef8167a2796ec8a20ad" }
#163formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "a2faa1a9cb91dc2c97df82de854547d664f7e4b523014435c3f31d30790f917c", "started_sha256": "19c131c8a87e44117ce45c324236537ced1e3d29285dbd810293b8dab1c18faf", "original_grade_sha256": "496767ebb5a7746e4c73dc70e635cec01c74ead61e8ab87e640125a03c8e5429", "input_original_sha256": "a2faa1a9cb91dc2c97df82de854547d664f7e4b523014435c3f31d30790f917c", "input_wav_sha256": "41ed9aca8ffc376218836792bdd4c55e45867e886323d7ee9b822cda6447997b", "input_pcm_sha256": "eb505034143d1de31ada229bdbe5c1f259a6408b9ae2797761a28cb9808c2633", "output_wav_sha256": "9ddc2b7e2f7bae797c0ea66e079e327d8057ab370273dd017fbdeeab24ab08f6", "output_pcm_sha256": "a2e357ea870bfe8b9a616d77b6e723993f61184ed481aafad03219cbc6617693", "result_sha256": "48dda1eb9d14806c8b677b92c77f3071458b3cdb7b40a88954f598908c5219bf" }
#336navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "ca4752b0ca35fbd7561e190dae5897c5a1ff799317faa56a83bb5abce9919bde", "started_sha256": "66b1a69c4e2756fa88dd4c71e8251d69f0d63ec7f65ec151939660dd9714343f", "original_grade_sha256": "21e7ed9c4d15a3a6ba4922d4554019e0dc9b9dbb79b6e737927ec013157e95b1", "input_original_sha256": "ca4752b0ca35fbd7561e190dae5897c5a1ff799317faa56a83bb5abce9919bde", "input_wav_sha256": "aa9e48f830cfea8521e00835bf229d742102c0789afd482d6d5f63f5c1fe1703", "input_pcm_sha256": "d242e3f007ebf0a538852a10e1235f330bcadfa5268f80382176671cddd54313", "output_wav_sha256": "d6cc208ef9c10e386f8370c8485e17ec199e0eaf76adff3f3a667e8fc3047b9b", "output_pcm_sha256": "895d2a2603bfebf5e2ce1e73cbbec39601d8207b36abdec28ef72b78a4bf1093", "result_sha256": "d022669d2a0bab7b8b04c7bdfa160ea2bd7af4eac9b51ebbdb5d95c21ba192a0" }
#654object_counting heldout	5	The answer is 5.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "416a21111fcfe24b31e39c974719d3a89baca04ecd20a2d110b2f1ec0dae408e", "started_sha256": "436aaab69b310f50ba4ef5a9cf6630f3ba853693ca64cdf7017c4e2040978f24", "original_grade_sha256": "da6ddf450a7584e0db3c69248c836efe9a85268d73aa3eb27704ed84bd9488a9", "input_original_sha256": "416a21111fcfe24b31e39c974719d3a89baca04ecd20a2d110b2f1ec0dae408e", "input_wav_sha256": "f945b3e68368570fa1cc506a80bdf9291e92f2f8879f6bac5e586602920fd743", "input_pcm_sha256": "cf33d1464a0a780960e67276c058ea6875ae7551ac709f564aeaca43abf12519", "output_wav_sha256": "3352bd44f983f8aa9f29d0b88600a23575938bc118212e23a21b00f3319b8fe1", "output_pcm_sha256": "0d35666732959b901cbf33ea40ec517a51538b1c03416a2ca6367298bf6a1cb8", "result_sha256": "b737e324605ab4d45ac21eab72179e1b378c763eb313ff4501acd81d726bd477" }
#913web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "c7292910d3f28123275acd3749d4c2578b079c72dd683890657708f2cde2f295", "started_sha256": "5a431716302ef1624e09baec09b1a925409514850545ffc9500d812def8ef615", "original_grade_sha256": "b743cd8a4f9aa1a1f58b0e4b80f8656351724c73dc1f4ee4a759f0bfd95dc4a3", "input_original_sha256": "c7292910d3f28123275acd3749d4c2578b079c72dd683890657708f2cde2f295", "input_wav_sha256": "fb934608ef414effc1bbd376c9ddf801cb586380851482ce75adefd9e067997e", "input_pcm_sha256": "ded303cd219834a1c7de72391b24806ac323c59bd101b770bcd165a6bf4e8b71", "output_wav_sha256": "db48ef89f52812208706b5fd679e2c81ed618ad4b8f3c9363d16877ec00c539e", "output_pcm_sha256": "e3167bcaf7c40d151f83971c3bedf8638a687593c357e5d1e3e23528738f15a7", "result_sha256": "8091e72d05170b3354477762a19dcc72edf6dcd7ec12e7e4cc85e6cbb40f90f9" }
#86formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "0c97b082e64e716d0d7a1e94686c6960f6354a5431a2a653790fdb1f03c53a9f", "started_sha256": "c851391438fd36b4c85840f8d9a4c1e4fdceb34db1ba722c954e41e1583e1c89", "original_grade_sha256": "4d56a1fbe360c8561188083baa849e12f6cad63c6502d1d8641f60768d4b9805", "input_original_sha256": "0c97b082e64e716d0d7a1e94686c6960f6354a5431a2a653790fdb1f03c53a9f", "input_wav_sha256": "68447eb392f66a7b8a7653e3ad6ffb05d942d64921fac84ec92e9d51965f7e09", "input_pcm_sha256": "112f8f7b02a084a2c48204252b2f48f23671f8523f3630a9e0cd913e20be8818", "output_wav_sha256": "f359bbd9b0bd8994dd429a1c72573275db78d2a4d09e53c7b3b41d74218482f3", "output_pcm_sha256": "586f766c5cd29dd5c690dd09f6fcd3e9f1174dc7b9ca1caaa374423eec03179b", "result_sha256": "21fd22491cc7568ba58592869d69202d6e6ac9aec0edbbde2961e26d2c3f1af1" }
#412navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "e079303829b2f6fa5fbc5b38a74f450cb3367a998783343c808df384ab7caf02", "started_sha256": "75a0e3e8d4a3a71b87107aef6f0f54b7d469f4aa7711558147c95f7e1171468d", "original_grade_sha256": "3602fdae697b859db2bf7ffb35c6fc9d60acaee450f07356f7462929a0238da8", "input_original_sha256": "e079303829b2f6fa5fbc5b38a74f450cb3367a998783343c808df384ab7caf02", "input_wav_sha256": "1de1296b1b89a5ff49ba43e4377bf89372e3dc9a9e159250c5ad09fc326d68d7", "input_pcm_sha256": "26394e31842859a61aa372ec9367ec061531d9b647711778aa91172ad7fdcc93", "output_wav_sha256": "da3f97c3cfcb86300383313345ea6511dfb1df8c421c2f50feda1408083090a5", "output_pcm_sha256": "305d11c4060cdae181e378a3cf833adf6df6613b95ca8d6ff777300d3d02404a", "result_sha256": "33e50428b1bb335ed0b204338b7c56a03351083bdc134999d2a79fcdf6864a2a" }
#527object_counting heldout	7	The answer is 7.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "8de446b45d6a1a476ac97b86f0a801cf60ec9d9880ddea730f439d103f780ddf", "started_sha256": "a06a39da3ae5dbb2c425061a2ed18b892f874c747c10b7f47bd13850ebf8805a", "original_grade_sha256": "7db5d47c681a30241c3a9dfa84146b95818b836a253387493da6eedd1e98977f", "input_original_sha256": "8de446b45d6a1a476ac97b86f0a801cf60ec9d9880ddea730f439d103f780ddf", "input_wav_sha256": "f393e02d4b2c319eeb819f61b1e633b23f87b3b3e63780a40965e8e305cf4829", "input_pcm_sha256": "19d19d3f7c67db1a95bb9e1f117933ad23a70ea670ed0e3e93b72d117611f8b1", "output_wav_sha256": "f40b74fd136edec6150a3230f0e73e32f5f001abd724a14e6e993bd03a476f45", "output_pcm_sha256": "7e287561ef28153ff61697d4119b76f9eb840fc0d88049490cfac49f612c4089", "result_sha256": "392c87e08576638fa97d040fb567d9e88ebc87120e7873070ade65ff5afad2db" }
#859web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "9090c3d72817d3a479559f2a8db26c6dc8174441293b8d11244c78b8bacd7ce2", "started_sha256": "48462876d64df4b0bfee1bfe8cd12625a73728c4706f389c16ba37bdc6414080", "original_grade_sha256": "07439d7b399ab339bde9c9bf08a4ef92ee05042e632d7e96b3e1f36ee3d02f36", "input_original_sha256": "9090c3d72817d3a479559f2a8db26c6dc8174441293b8d11244c78b8bacd7ce2", "input_wav_sha256": "d3696219a402dc69f4e3ff320b6c5df6846db8a21b77e49a2299d34eae2580e2", "input_pcm_sha256": "b8347c6a9897e5ba60ee51489b51d4d878dc08f51182bc307d515bc03b79fb1f", "output_wav_sha256": "c2ec579e28bce437b4d4eee991654ced0be6a364ae29d5cd613e1f1058299d0e", "output_pcm_sha256": "ae57b5ac9db46bcfbc32228e976424fc00595414ef35c1c8dc8cda881a4e702d", "result_sha256": "1fdf26b115b3a91e9ce20fe4d864223172b5f07093dc8768c3f0b4b82e90976e" }
#71formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f7b197fd5336a24022b28709be6d71161356f39a8c022a94147fa8fccba6050c", "started_sha256": "de455134fd4de1d882a467e50c761e467825324e96f23106ad87e039110b461a", "original_grade_sha256": "e3a63cd90866fc4887a8926091291bc9f4da7886b517a853d182f5a15927ff9e", "input_original_sha256": "f7b197fd5336a24022b28709be6d71161356f39a8c022a94147fa8fccba6050c", "input_wav_sha256": "e64c433a822ddbbb95f9112c045490d92dc07d234900a2c10898baf45e568f4d", "input_pcm_sha256": "f997961ce902a6b0f35baa877a80a73e269bc8bb71de28e84972d0ad5ef4a1f4", "output_wav_sha256": "13a0bf74d40414b17b41c19806195e5d9ebaa0594a68ec85bf2f7e0e67457825", "output_pcm_sha256": "92a68b9c1f1fc7d0dd3412e9f537a26725bc5fe4cf74b2fec712ae73aadd21b9", "result_sha256": "de185d6eb91649092e1e8dd0717371f38d2a2ea8808d29f1d4e7adc3a48ab850" }
#475navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "5c5f1656ace606b851f096762fcde67403589b428e33ab3bddd632c48b4f7839", "started_sha256": "6887b598c1ae3bfc5c1a50b9d182f3707f92633bcc25ab2a8f968f77d2e7fa25", "original_grade_sha256": "8342dad2bf8f76a620b7ff7d8a190d1ea7b7bd2cf45cf8904e7c02dd1bba511d", "input_original_sha256": "5c5f1656ace606b851f096762fcde67403589b428e33ab3bddd632c48b4f7839", "input_wav_sha256": "f2cc9e98a7afb43e92af56aebf032c6fb97b318126697c2ba96b080fe76b3575", "input_pcm_sha256": "2319963d23454f0c2e45242ce10356c2325ef12984696f6948494fd635d30603", "output_wav_sha256": "8a948d4fb2c9cb693077bd88f4839a846a94aa615ab9dcfeafd5a00dc1c20ff1", "output_pcm_sha256": "62c0cc4385415f605897e43007651f265d3104d914f551926bf139b673be70d6", "result_sha256": "381ced5ffe838ecfda2f6c215813231c013cfa52b78f04393fc0baca5e5a49c4" }
#664object_counting heldout	9	The answer is 9.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "388dab1f24cfedcdd60a0a650a829aaa338fd751881e599e9edf692334768ca2", "started_sha256": "6f8b4875ba755efa9cea9ccaf9448a1781caddb8e6419405eb6f29a6ab9acb96", "original_grade_sha256": "8f071a7fc94c3343aa2504d1c14b914c00f189d2a7b20bc1545594775ec71f7f", "input_original_sha256": "388dab1f24cfedcdd60a0a650a829aaa338fd751881e599e9edf692334768ca2", "input_wav_sha256": "10e4a2d39b1546616b0ad1fa6b635247f709c0b2bdf89126a20f50b29d6eb76c", "input_pcm_sha256": "860c5815a2133fb188bce0ae84b405a55012ae4ae6b1010a22b31fb531b1f240", "output_wav_sha256": "d25d3b1eb57b4955cc1116c53c6f30b38b9736fd927f6e892865074b742f62ab", "output_pcm_sha256": "7c32103dd052c7fcd810816f82b3d90498324eaf56728a818ccb965cfa9ededf", "result_sha256": "cfb8989a3687585b4ee9846a7314ea2842b900b84fe3e8e26f93816e4d7c3f61" }
#970web_of_lies heldout	No	The answer is yes.	incorrectstrict_spoken_answer	incorrectresult	Original grade retained Evidence hashes { "original_audio_sha256": "a5ec5e2448ccb7dedc699f137c01a5c44e6c285c9d39f9cab8cb8abc04a08d4c", "started_sha256": "c4ae5f09a187f53e5f43ff4cba54d3f80ca127e341032bc9e558a8539d1ed337", "original_grade_sha256": "72ec9ec091ae0981d0a78ad5963ba6010d0b48164b0f78fc626d778488ca0855", "input_original_sha256": "a5ec5e2448ccb7dedc699f137c01a5c44e6c285c9d39f9cab8cb8abc04a08d4c", "input_wav_sha256": "90b05849c2bef6358912b7a21d58abbb882e0e337f8caa98ba5c7228c8bef561", "input_pcm_sha256": "dd06ccc5b193c0f09dfea22730c2dab2dbd91550517ef438cc15aa4258f3d71e", "output_wav_sha256": "7ac87da29fe47d872ba3fc36f8f9c6c7c6fd35d20ffcbaf23e6d92eacd38bf66", "output_pcm_sha256": "3f2db0420b013de4251e827573e5ffc763f2035168cc5f14717997bba7a88ce9", "result_sha256": "a9525439b38e36801725906975cf7485c34ccee1b7285dfbdba3051ff0a3eaca" }
#207formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "e1d1185768e9851f86bc51ccd24495b08f49791fc0d41e37a634044f9ab75598", "started_sha256": "fdb996eb83b9ac18e6b0e457529ea5c483bea4848ae19e5c6f0ed109750b3070", "original_grade_sha256": "bde6b3a9a15f0079ae9022021daf0b6ac7ad211cb04a4f34889b0a9d13717480", "input_original_sha256": "e1d1185768e9851f86bc51ccd24495b08f49791fc0d41e37a634044f9ab75598", "input_wav_sha256": "9a67a8c66c4a54ce568ddb574798ad641705f2d7bcd23e92b580a6e432262489", "input_pcm_sha256": "67d5b3879e6c072d0e3d64c36c7af06b9f1bc74c76a13a8f2877696a8bd96103", "output_wav_sha256": "03f744ed06d0a956f23187fdf9694618bf8838ffdf5790bcb0ee6a8d01331cb1", "output_pcm_sha256": "04b85b1977778a69e527d07eda1f3b6d030f329d828b5e32cf2165de6fe78487", "result_sha256": "af3815b33c1572c929138a3c23bec3d182a8093a862ebda6721ec9d24b76d476" }
#453navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "524463e75a02772da9816827bd647d75daa42ff44a3adf69a025069d1c9c2aa9", "started_sha256": "50e598ae831fc6915d9aaa7472bab5b89eeebf766473c635de22df5ce0cb96c4", "original_grade_sha256": "6a8bc495a9acdbc1b51af2d083eba868da9a451c9460abf3082db6d32420e81e", "input_original_sha256": "524463e75a02772da9816827bd647d75daa42ff44a3adf69a025069d1c9c2aa9", "input_wav_sha256": "7a2d8c6d5712ab7d6aa844c5c4fa498566b03521991975b704245c8619ad0f2e", "input_pcm_sha256": "c8fe15e8575377270d369858190f4612edb65430e491e9486c6e0e583fb9d182", "output_wav_sha256": "766fc75b8d60d464650c54b5fd5f80bc5ec3bd204dcab040f879978b8f80f98b", "output_pcm_sha256": "a702f9eb8f33c4072045966c97b3d6e38b6e54aea0ec67e183e41b7f59e9fd22", "result_sha256": "4c4d7a2e3ffaa604cbc8488da0d4833edf5b82148e854371a5f1a16802e35b4e" }
#719object_counting heldout	12	The answer is 12.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "1d4b8381cb5a52d3627946a5c47b9d5c2adada50c54efca735bca11c9c7ee677", "started_sha256": "36fcd05a87c9eee2cc1aeca4a3ac70c5d7c8b8d6264d8e02e528b8664ae121ee", "original_grade_sha256": "3134b5c1ec2fd7e0b6f8d69e87f63d8e163042486c041bd7d2bc59fd47dad744", "input_original_sha256": "1d4b8381cb5a52d3627946a5c47b9d5c2adada50c54efca735bca11c9c7ee677", "input_wav_sha256": "007669a973ef053c9b668ec326099d32116f354c5b2bdb19e44069e4d7c7c94b", "input_pcm_sha256": "742442c3b6cfbc409003672c1dc9cbc689618f7ee7ae76cae3f872aac57763d3", "output_wav_sha256": "6274629948c098b266ce2ba00f21100252f4e4eaa2de7db15e256a8fed1327c5", "output_pcm_sha256": "b2a2d6d781741e81ecfb2dc648a42797699f09efd0527475f8a1ebde193d9d8c", "result_sha256": "9b7f574a004adf4da596affdda1eb6320c9a95267189f6957945b94a37ef9d88" }
#943web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "7c9a2c2103287e9b7fd42bc16794eaf2d656ee9073d882349be25463bc4384c9", "started_sha256": "2b183d5bfbe93348c154e8e3369dd3ae63b172890d2adebe8be0bf9c1b96406f", "original_grade_sha256": "6884ec0dd77abaf30581c0471c281821b3c7577551601005ec495764aa83358d", "input_original_sha256": "7c9a2c2103287e9b7fd42bc16794eaf2d656ee9073d882349be25463bc4384c9", "input_wav_sha256": "aa6c2e3fc2848672ac295d16b62739cc19b7ca4b10268110401634354b722355", "input_pcm_sha256": "399b23cd707fe9b5a8cdc168745ed8a2cd8565e602369eea8185671ea2207e86", "output_wav_sha256": "188720ba4c9af015a3d7e44d5fb87e26f808c0f504ecee319cc9c0dc970b7001", "output_pcm_sha256": "e994b0125cb717676976a44fa4ff9f73c0342437832d762e686eede8b6b561e1", "result_sha256": "0d63cb7f4928bfb3c5081b3a68cad8cb69badafd5e7c2bde5fabf50d1fdea392" }
#164formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "5243cdfd457bcf6fa60bfbb13bb8b0baa47184c2890c9d903636aad03d361d06", "started_sha256": "f8c5b831d27c9c97168a48e91d6ec1d68cf0c3f394e5aed85739c05eab3bfe89", "original_grade_sha256": "cbb9e8cb7e4360f305945256116e1934ed17ff66d310cf8cbcf20209e6bdfd93", "input_original_sha256": "5243cdfd457bcf6fa60bfbb13bb8b0baa47184c2890c9d903636aad03d361d06", "input_wav_sha256": "13295f67a735abd82b0b20efcc0d2bc95a2598f44ba2a7e458ccbf0978b9a67d", "input_pcm_sha256": "94cd1d68d142e37dae381872c32806c64df940586c5fdfe8be05d4892fc5471a", "output_wav_sha256": "90ebff90de47bb537e3b6e0174a8b24865d1f20eaf8d507b76c8b7651344008f", "output_pcm_sha256": "6d1a0d8d6e7c58b1be2f83a62113df0b7e33bdc7eda1cfc2feb5636a7f0abc4f", "result_sha256": "6de1b79632a6dd093260ea06e86dd91501bda8cde531ea72bb73a2257a05e730" }
#261navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "c7ea48dca4695bb838e7f13e09c065764866ad1a88b9b9158012c49253f27b16", "started_sha256": "e942c9c8104bb19bd53ee8a410b9298913ac9e49eba1e829e11996c800986c5d", "original_grade_sha256": "49f3a94c3986aa8a95a16f6bac4ed5bb4c9327981a9b4fee2cebc1743d976510", "input_original_sha256": "c7ea48dca4695bb838e7f13e09c065764866ad1a88b9b9158012c49253f27b16", "input_wav_sha256": "7faa8598eff7ccb462a1a5115e0577d9281d379448ca2b8d10b2f1dffab82dbe", "input_pcm_sha256": "5b4d028510c4588aa2d915857102cb5bd1c202a4ca40ea989a7aaeb20b412917", "output_wav_sha256": "f1afefc47c013e107e256944542abb35309468a136d4f9bd0f238d5ed527eff6", "output_pcm_sha256": "eca923c092c67a617e7d1516c4dcf11023b6e3fa7c3be703330c19f79a85b601", "result_sha256": "0ae35a640505c57eaed3a486735ecaeffeab2c1bf0d0d7e11774026a4a267c07" }
#727object_counting heldout	11	The answer is 11.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "d13b766556c0b8e22358dce75e21cb9705563acba03f7c2936645f90917143df", "started_sha256": "5b4bf66c58dd742be297c2d80a8566a493f2b8cc4eb52892a2001bacfad2bf4c", "original_grade_sha256": "bae4203138f9972b136f0c87c608f6a23d46d54a9f9f22fd63eda2f16b16bc57", "input_original_sha256": "d13b766556c0b8e22358dce75e21cb9705563acba03f7c2936645f90917143df", "input_wav_sha256": "a7a0072d2908db19da4a3298c724afdcc587cc718c76ec30ca07f9735f2fb7ee", "input_pcm_sha256": "d4b99975d7481dfe61dd11b94819ab0f37056041c33305c5fe98700fd7b2e47b", "output_wav_sha256": "706756ef69c03372eba92ff9dea4d8ef8356dcb1ea9276ead6d6bfaaf1132631", "output_pcm_sha256": "9773b2e8c76ebee0f6c40369365cd8949c57516a198569f27b7bab14aaf0a00e", "result_sha256": "ef3940dc3f655d21ad6b7567be6fffc14649cd209990d0af1366f3d9f6c17428" }
#959web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "82a8a764ec6dbe0e11f0253871f06abc098c2e7daa29584369b2198765a9d2b1", "started_sha256": "bb04e44e71877b5ab9f8c2a2b5c07950e74cebe95186e4c8fe06d649e68fe65b", "original_grade_sha256": "546d8149de6b0ee93be07321e63145879c754b74e57dbfb529130f6e4a0fc8dc", "input_original_sha256": "82a8a764ec6dbe0e11f0253871f06abc098c2e7daa29584369b2198765a9d2b1", "input_wav_sha256": "39bb0e56563ce1b7b6c5478d60afff038b73c6cf82a57306be5c09257ca91f67", "input_pcm_sha256": "2c5273017ac4de9a077202b8c1bb2bc98a2b67902767128d339624cb2677e0e7", "output_wav_sha256": "de485126b07e05eb80622552058a5e186357c52d1637166ca14249a9db809ec7", "output_pcm_sha256": "e0f2aaa8bca53df07c1cb396585f885c0512219259d14fab009a3ea1b460f083", "result_sha256": "291a02ac942aebd152bd908c4eb224d20d25a5e27c47eded22c3f007d52063a4" }
#161formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "eded384e9f2dcb5a1d576259ab8759bad3909c0dba2950a6c406f92848790f6b", "started_sha256": "2b7321b3cddecbb04a796716cf37f497d753c386a3a34d6272952e9341dcc58c", "original_grade_sha256": "1fafc835534bf9f022b1a9b96d9bbf7f2087cf055af5124f0117e020ed938ac1", "input_original_sha256": "eded384e9f2dcb5a1d576259ab8759bad3909c0dba2950a6c406f92848790f6b", "input_wav_sha256": "8dac4d9e2a77924b851578dc66405f3c05cf810a6527473acb080d85fe65f27e", "input_pcm_sha256": "80ff0d4805fa15a4acd4f72cf47627da8ca4f727d94f9178f753d3870bbeef6c", "output_wav_sha256": "832ab9b0081918f3995635492c28881d6d50772be241d1d77660980872a660a9", "output_pcm_sha256": "f69601e0a86954c0a3d92a4b86e357d2f1405f1c4abb375addca1c166306d529", "result_sha256": "53db34c295a22741ffcd8c293a3eac1af1d34ab202c5c7e8f1f510571c927fd4" }
#454navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "9d27c366b7a60921a0387d7d569d329cbb354a6fe14631384b3474aeb8973ee0", "started_sha256": "313a10d9facffc132f76656fd79bee191b90c0164c7a3747910586abd7961c71", "original_grade_sha256": "f2765b7749a76ce5f20a9c000c50fb242fd3d93ccc758f8879bb3f937757fbda", "input_original_sha256": "9d27c366b7a60921a0387d7d569d329cbb354a6fe14631384b3474aeb8973ee0", "input_wav_sha256": "0ad8b1420e23b9307d07a95a50c7c5f91091870e136f7b35cd35da2eba019f4e", "input_pcm_sha256": "c1c9bf4d17344d8a0e79fdc21659777b050e06c407e9bd563524cddec82e2c0b", "output_wav_sha256": "90037e2c44351f99ced1f58baddf8280325e00204aed4835fa1f380d8caf5c61", "output_pcm_sha256": "f3eb35a53e58f858f5ac891b71f8be56afca5850795d363ae7239ef7fa9dd881", "result_sha256": "97fdf03e75c3d15d48add18ae0c2c075a0c3b71a4de32edd89b6c1dc6e780452" }
#603object_counting heldout	4	The answer is 4.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "48f6d6bc06015662afb0cb91967f3273261ddeb9686784213405d08e335a97e6", "started_sha256": "afe6d4a00a8e613eff0f57ce40a015b652120df10bb287f924dfe4398a172eea", "original_grade_sha256": "86078866b5bb269e2483fc726a0a9761a028651465f6f84a576531cb523e2ca2", "input_original_sha256": "48f6d6bc06015662afb0cb91967f3273261ddeb9686784213405d08e335a97e6", "input_wav_sha256": "9c191fcc446544bdc401110f77d32e4ab2c8d33fb01db81521dec4669cb41146", "input_pcm_sha256": "94f245ffcdcab7355ef2d5985ffc8d4a6208581de6f020798cedadbbaa5f076c", "output_wav_sha256": "bc9cd3b2eabbdffb0933197b79593c82ac0483b103cbd41e5ef5b23ab1ad435a", "output_pcm_sha256": "2b562d3f294b8b95208e8efd1f80b8c213ee97d09e0e128a2232bdf28c6056bd", "result_sha256": "8d4e58eaddb30ce94a146e0f008a0eb1817f78e08d055ddbbfa7d6e45d27ffe1" }
#817web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "4716db69780ede7a5d2a6346bb51f2a41f5cca3e94ff8d0f48963a84a5b3d15f", "started_sha256": "c7283ab8ee7ffb909e704ee0af9cc6abf8db5b372d855892dc3e2a79f4eab538", "original_grade_sha256": "110c2305bfeb163f2761c463a37104a17bf459127f5b1bd59bb40488a950417f", "input_original_sha256": "4716db69780ede7a5d2a6346bb51f2a41f5cca3e94ff8d0f48963a84a5b3d15f", "input_wav_sha256": "4723843a69db691ed5a044d2acdb06dd19b3706cbf820276cbe6ddf21a7d2254", "input_pcm_sha256": "4cb3ca3133c90013104884d6fa21373a84c197e627f76af1cfdb3f7e53d22fde", "output_wav_sha256": "49132d7d9fc3f8b274bae975aaa87c4ae15777787423ec279c0016ceebe1f222", "output_pcm_sha256": "4e7d5534c974d34730c09c3ef0ee82e6a6faf8353e593607f7f23cf5b1c22d43", "result_sha256": "9c47890c9abe203894d0ae9316280caec74d3e4b0fde85a11ffe50a771db85da" }
#246formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "aca35164b4bc3f27870a2ddad3e27ac5e30d28482d1bfc078a767b5e95faf76f", "started_sha256": "6275d8dea51e4f3cbd9ca5f83101636c26a205f3ea6d5e0d2f10947ecf0abddc", "original_grade_sha256": "5ad178cda554391bf4cbd0be1a32cdd3cd1e66d8f081746f02771a015ecaf5bb", "input_original_sha256": "aca35164b4bc3f27870a2ddad3e27ac5e30d28482d1bfc078a767b5e95faf76f", "input_wav_sha256": "fce0dc985ace13fa943e761edf5c6c4126d9d91e04568c58de9d9d7a17434b15", "input_pcm_sha256": "9ff4c13b802667a300e62c673cde40c8c35ddd0b419e2a6115c95938cad5432b", "output_wav_sha256": "900bf57a1722c9124a3ad11d7d949c9805441081afe4af6fe18cc63db2956e99", "output_pcm_sha256": "3a18be276a9b40f57867166c9dfb750a9f2ac949bcfbf5a5d72a7ca488fe8c1b", "result_sha256": "bda9249eafb74fe5a787d8a62216e3f6e1bfb71974bc71ee42004339cb7f67a7" }
#316navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "ba6b1f792c0f289fa9510e51f4d3b428f66f410268cde6eaceb89a6ecc030401", "started_sha256": "8aa7e3e70b5171b6ebad5af1557fdcb89f78c840b987209616f4e58862e7202c", "original_grade_sha256": "b2a6a54647a8ad672d70001e94f5f5fe1376b9f451efa143fbe8ba91b0f9d118", "input_original_sha256": "ba6b1f792c0f289fa9510e51f4d3b428f66f410268cde6eaceb89a6ecc030401", "input_wav_sha256": "fa6154fb292e6352303e7d1a4790927351680a3af673292e27c19e0b4d117224", "input_pcm_sha256": "08e9eb1a328d39ab8b54e802ecbe0517ddac10fc37d9848b5533c0b9bcce844e", "output_wav_sha256": "f6923f5838d80073d6b922194e75ddfc41e91502e3641eba09535cc4facfff2e", "output_pcm_sha256": "0f8bfeea13ebfd9456f93cb0c1f3b76bf26d84e532909838a599a0051ae5471d", "result_sha256": "482b1e231f7f14edbbff8c25075452e0364912794bc4c33a27bff8ce485fe346" }
#530object_counting heldout	17	The answer is 17.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "c8f50cc8484d956934856d99f3664a99158827775973149c07dada586a762e5a", "started_sha256": "6034b313145e12079baf8ea2fdc6720fbd46440919c7fe3ec13e3f1aafb91021", "original_grade_sha256": "0e840b12a1c2ea8c0864bb2b27107866303af7cf4da99afabb1dbc5101d1ba50", "input_original_sha256": "c8f50cc8484d956934856d99f3664a99158827775973149c07dada586a762e5a", "input_wav_sha256": "7e56d79b0398dc36d4763435aadd4fe25aad2482d74935376d96bb86d992181e", "input_pcm_sha256": "892736986e04047762cadbba9627705eb8784ec1b26010912d96b6ec2f47ee07", "output_wav_sha256": "017cb41db8c26de5e9e0a83ce6c504f339bbd8cf7fb106dc8481fc0315d96243", "output_pcm_sha256": "558ef42f4ea47dc13d91e2b06bd432052e2057847e57a7de7f7f52295aad9f7e", "result_sha256": "55ee57ba3f35d11d870e3ea25b1a0a0485468e2cd0057c8df7ed8aacdc5ec65e" }
#780web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "48164508b3c36c2f0805f957b6c6d43fc1a4ce34292e6fc43eea526522f04ef9", "started_sha256": "3d706ee3ac11cb2fa56edbbdcb6ee5b7273588539622ba84c601ae1a618d1f7a", "original_grade_sha256": "61aaa37562ac1db655f36ee709415fda7e5e71eff6c44554f767b2bda960faaf", "input_original_sha256": "48164508b3c36c2f0805f957b6c6d43fc1a4ce34292e6fc43eea526522f04ef9", "input_wav_sha256": "35b3b9ee4aec7af6becdee8cb2160037ef16d25be96929e6ff5c1a32ad1eca61", "input_pcm_sha256": "576bda0c916dbdc781fd2dae1e3704243be8f9346f723d5885adf4e3d09d6550", "output_wav_sha256": "9864969059c481e714aface612094c7e7eca88fe1b0f8dcb93bf5913bf261d8f", "output_pcm_sha256": "d0391dfdc212b646d1db4a6ba1e0aac5f4e09c8f6fcf05575582f1a7141d49f7", "result_sha256": "1cecb2b7dd0035db8b4ad6e3ca94e7abc93f6df0f931bc14717e078a9e932be0" }
#167formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "9c04ae72d73aa194792b26874f44647698b7ef48d8b1ebbf7867dd5ddcd4c71f", "started_sha256": "2e2dca4178f43665d3e78f48c78802c7cc0d0559a91436d591d80640a756870d", "original_grade_sha256": "8096f33aeaa4b3fd67dbc6aee3e52f211a4e19c7fd9a29077adabd82b8c8e46e", "input_original_sha256": "9c04ae72d73aa194792b26874f44647698b7ef48d8b1ebbf7867dd5ddcd4c71f", "input_wav_sha256": "8e95836e73a7b3b00b4406f28dfaa64ccb02dd183302b4bc99f373864b936ee8", "input_pcm_sha256": "69bba5a6abe08baa6a123eef979d449b68ba7834cf58d1902ec9334ad26bf280", "output_wav_sha256": "38dc401a23927bb6a0dbd2f90bff8848afe7e9133220caea91a6383c9206ffa8", "output_pcm_sha256": "44dfa3426d8c74cd1e7c28be67cb38482254483f7d749994869aa189e61acf1e", "result_sha256": "ed32c920219561dd5abda91635dcbb7a23dff074cd354ace4c07da5fa6d361ce" }
#280navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "3987ab239bf8f05c2cf3a194052140aa4de5097164e3f1c63d7018e399897b2e", "started_sha256": "dac334617c8a5be8ab14879919f1e317e4ff161af1458fc897a683efc427a6ad", "original_grade_sha256": "b5ebb528d7041b1b6c2067f39569b5645a48cefa3a5c50b857331b2f538f4427", "input_original_sha256": "3987ab239bf8f05c2cf3a194052140aa4de5097164e3f1c63d7018e399897b2e", "input_wav_sha256": "bed9b074d2c29b6783c4a496e6a9c98e0d94e90b95628b912dc7cb235462cb37", "input_pcm_sha256": "f5ab35bce7008172647aa25e157e27c1b5a454d62721fe174be0c849a071e85a", "output_wav_sha256": "34e8ce81439f47b6ed99b42921556dbf62c2c8324527dbba95ac09afbca52e97", "output_pcm_sha256": "477576ddf869a7b3fae8eed7473f8e255bb7fcce2fcfd32a9cd5dd9bbd0339be", "result_sha256": "eba4974b72fef3a6f89f83c3d9e66f3eccc11fcb8c7deb5b8ada585d537b74c5" }
#692object_counting heldout	2	The answer is 2.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "88069cd0df78636cf4d8eeca1bd4a2ec46dcd9a3023eab077ef8731db073473f", "started_sha256": "1e9f8bba2e38278658ff2c1c7d1938324422ed99168ffe9dbb91072b24544694", "original_grade_sha256": "2f04c8eb97bed40baac40247fe9fb3d6aa33e7c15a250b07848c6732cd5dcb40", "input_original_sha256": "88069cd0df78636cf4d8eeca1bd4a2ec46dcd9a3023eab077ef8731db073473f", "input_wav_sha256": "4a667247b93c66131b6c285b1bcd04543e6b64426882d7dd57bb8324de58c897", "input_pcm_sha256": "3f61ad4e5de9236c3f178a7771cbd4393ab19e917e6a9af5c145a3886aa5104c", "output_wav_sha256": "b4abe7c45226fc1b80c26f17fcd9340cfb440f188473326745a8dd8ea53aeb77", "output_pcm_sha256": "ebf202adbeca6ab6765e7ecde33eb112725004d02fd6ae0936ceba1dac00c1c2", "result_sha256": "cadc2917cb0bf3d586acaa2776f75f59021fe5aaba54444178a01d9cdfb29ae0" }
#989web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "afdf0686d1f52b1ee85eb5c1ab2d5d1c1f3c4c70797cc0a356c29665ba855fae", "started_sha256": "417882479b2bb7c30e0eeca00209263b5fc75273ed361ba3e8a72d6d3dfdeb90", "original_grade_sha256": "b9a629d72b0b195c85ef644de413e5e12315c2659c0acce4b67b85ad3fe3a638", "input_original_sha256": "afdf0686d1f52b1ee85eb5c1ab2d5d1c1f3c4c70797cc0a356c29665ba855fae", "input_wav_sha256": "d0b4adcbace0684bc89eba49315675b72a624311d770908395f0b99bc0c075be", "input_pcm_sha256": "b880e7d3a409754cd491c1a4afe74a0b151ffa8918125df8e228e13a14acf896", "output_wav_sha256": "f42e56bf91ed8765fa4082df14f22f96ecd584b0a792c46a67f7ec4c2654ee53", "output_pcm_sha256": "5ef31544d9e01259fbabeeaa2d86415c6f0637b4e0f9ca7300a3efd31fdeae4c", "result_sha256": "cef393e7a89e4d6e3479e663a38749cc700568bb6f06040839a56f131a4f4136" }
#174formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "237e7a582369c779207ecad37a64c7415b1958919d1a2a1c452f348645287e4d", "started_sha256": "1c15926839805dde02ab3a04fa29109f9bea8c57869b6c9479582d77a5300193", "original_grade_sha256": "f4e3c012f3087253728b808044ec6e1abcbd07db76b832ec71993351da8a21fd", "input_original_sha256": "237e7a582369c779207ecad37a64c7415b1958919d1a2a1c452f348645287e4d", "input_wav_sha256": "fa109c2f0a7836593519814a32597e092d6f6f468dfc18c5214085b8d6c2f6c2", "input_pcm_sha256": "87f940a866c765463301375b8adb21913e1c97f1378823443f34002ca0210f0d", "output_wav_sha256": "706d5b25eca786f31f915706d559a70a210ae19e0f66e810aa6f0f52a4d45a7e", "output_pcm_sha256": "640c502996246230ee31cc6b7f238b5016b64723909f505156a6dc27f443e26f", "result_sha256": "ab9b1cdf71fed1ee5900d6d085513c1964a4ad850832d0c2bb5aba583c415f8a" }
#312navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "eb3aa5d6ec17e6fd26a2aea986d226422e4fb0b25e3b8410f78bb6f1810e8801", "started_sha256": "f86bf4f64d4046126ddf2452804b1cc1eeef7f0eec91e1b140ba3e30daeeada5", "original_grade_sha256": "f78035671d8e23f525980f20b8472851e6392570bbe020a14ffdd763caf8bc11", "input_original_sha256": "eb3aa5d6ec17e6fd26a2aea986d226422e4fb0b25e3b8410f78bb6f1810e8801", "input_wav_sha256": "05aa9785971f224f2017daaed613b2acdea0648540d0bbdcb1640fd10fa3a9e8", "input_pcm_sha256": "5507b8949dbc25742cdb5c3718d3468398e5b4198bd3a78ca9a22f85af3b9b04", "output_wav_sha256": "549c8244572e70bb746462641f62d3eb2a3125586c0ce5d117dd8e1647250f09", "output_pcm_sha256": "f9bd99b3774255b301be1c96c2482ee48a08bd809f06b7f79d0b07804bc0446e", "result_sha256": "ef1660a1e1cf5457258db1f4c35d9e75960e17eef82f4f1fd4dffb035bfe044a" }
#691object_counting heldout	7	The answer is 7.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "5e900688baf0dce64e42af982df544257190e454d7dc9786d05050fc055e3e60", "started_sha256": "4783fda7f44385ae687897444d0f13aeb45fe095cb6cfa7d1c6b0e771c64383b", "original_grade_sha256": "5cc35181c0cdc61e4abb839d73dc43869f4565eef24a8dbbda318f54fe93d940", "input_original_sha256": "5e900688baf0dce64e42af982df544257190e454d7dc9786d05050fc055e3e60", "input_wav_sha256": "6cfe55c1b2f3409dcba208398c883b3a627691ce33b302015110175850b4192e", "input_pcm_sha256": "7c123ba1c2e83a79369c05ff37c859d4d210d208eb451a5ded023c50eae621fc", "output_wav_sha256": "fda902ff04bc7201dcc0f596db62feff34e057060a12688a993f520edf5a10d9", "output_pcm_sha256": "8c14ec5ef55ca55f5c0094983933881b2a0d07bd3b3eca77908a27f9466c1444", "result_sha256": "3531ad6e60cf983d9882613223185520c3ae9d82080d24a6cf0590c7f95cf5dd" }
#891web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "5b6ded4c195a29eec0b52b5d37b8bba06b87ccf72e4088cfa7ac180313cae5d2", "started_sha256": "659dcec3032d5f9dc06d9b21b7ef7fbbc86cef6bd2f26c443c4ea1918363e3d5", "original_grade_sha256": "94a383110661ce8144f64f0b95cb3f7dd602acfbead426bc0e6c4939012e50de", "input_original_sha256": "5b6ded4c195a29eec0b52b5d37b8bba06b87ccf72e4088cfa7ac180313cae5d2", "input_wav_sha256": "0f559e377ac6bbfb81daf5151da423aebf085f11bbf7ec39e901da6ed73f7b5f", "input_pcm_sha256": "c35359f017fef28603088d30d06dbf229d0f2984e0144be0af5a01028ec40ca3", "output_wav_sha256": "4f049113c91582e2525f3d1bfe03ffe2caa1b18ea02daa07a77842427cfd1340", "output_pcm_sha256": "b26350401eeb7a5eac2abfb8e3c7df6821b3057c1674c14f059c4c8596d5c5b1", "result_sha256": "3ec92a9dee5feaea024ab1ac5265731a614cf95a87cf54866de53b836072a25e" }
#100formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f78b677e5221cea19c4760b529bedc5d35f223ce67712e5765100a1529081ea0", "started_sha256": "49be2d9b0270d904d89808525bdeea09abc70fd2963c688bf702de3df325855a", "original_grade_sha256": "f4592aef8779ef11ffe9cd5d05f3448e0ed63d6903410295b3b95aba170b0541", "input_original_sha256": "f78b677e5221cea19c4760b529bedc5d35f223ce67712e5765100a1529081ea0", "input_wav_sha256": "0a351f0fbe06979020fc35ecb8d25b7fdf959c6f5747d92e8b10d38629d3c9ae", "input_pcm_sha256": "7076832984629e5eca58bbe4e2ad25ad5a1ddf8c746d690e7a59fa2341111efe", "output_wav_sha256": "b0ed0fe877e27a5f0ffa9eeb29925c26d32ae528a9297f174980de95365157fc", "output_pcm_sha256": "1f47a73dfa4554f7a2c1ad2906c9bec17364091346898650e2d19f466a41f8d9", "result_sha256": "13dc426603616f39a72a2dcaaa1a0bd00e04393aa7b4c513796459744ced4068" }
#468navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "8d6d3f1ccfb4edc505a492838658014eccf14cc996ff166883589648e9b4e42b", "started_sha256": "14ccf4dada013b733406127019886bc7267ffed54b6e04472d4006fd94d806f0", "original_grade_sha256": "0204fe12bf9a7eb885ce3634d084bd514f181ca3802b705cfe5e8f43133cb134", "input_original_sha256": "8d6d3f1ccfb4edc505a492838658014eccf14cc996ff166883589648e9b4e42b", "input_wav_sha256": "c76aa62939a427cb2abbfbe83a466aa0d70946fec0082aaeffe25dce48ee154e", "input_pcm_sha256": "9f0d19921388d9b3d5f29be03de49a5d1d13317647c29c6e0d34f58f9a74bc66", "output_wav_sha256": "69f92952014f22bcac0801e6c8d6c81c53bad8ee560abe428691af1263baa82c", "output_pcm_sha256": "c69a2112d4a00f6d69866c1c903a6a2e5da311db922f63c2c048052c8e453d5b", "result_sha256": "2dbd2a7dbc2ad9b55430c209ea669d4a1531f101dec6a92f8ecc90264e15ff57" }
#513object_counting heldout	8	The answer is 8.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "68f423c51d89a81b5bc5d1cc92341167679303baaa4718856f5cd57908ec0133", "started_sha256": "d014b937ac1eccb8476138c7c7f20fbf53cdef92ed442cfab8bbb0f0e36139fb", "original_grade_sha256": "44f886d165bc7017ebb7dfb9ae6686d97dc6087d5cc96fbdb9395d017b057f40", "input_original_sha256": "68f423c51d89a81b5bc5d1cc92341167679303baaa4718856f5cd57908ec0133", "input_wav_sha256": "f3d44155cce7e477c76fe73ffa69eb906e43bd6c4e05ffda01147f801f4c43ab", "input_pcm_sha256": "fb48508871200b5e847d800707b14c05d81218e4cc6c8f49184d2634e8e6c101", "output_wav_sha256": "7f98c7843cfe11f2b91d26a31a501e05a10aface34ec7003fe520eb5b7538d70", "output_pcm_sha256": "569b3eb007c6c77e8a966d6897b73a9a961aaf21781e491e940990516e814c52", "result_sha256": "781057c70424910c6a0757dd6c2d2986ef5ec6c6a03393a86f511f723f48fe21" }
#818web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "072b3f4df35c849d07f80ec17aaeb8192b121fbcbb81633446d3810f741d21b0", "started_sha256": "0155864e42f82df8d032abddaf3c26b84dec5a86d356d18c8c91ed0c2f8d59b4", "original_grade_sha256": "5e9b70984f3a487de924a9abefeeea74b6a54c6df930a8822cddd3ad810a3e11", "input_original_sha256": "072b3f4df35c849d07f80ec17aaeb8192b121fbcbb81633446d3810f741d21b0", "input_wav_sha256": "9a392fbf29d0e1d837b1b65b5664d9038ffa1b164bf6fb3a2948fe067d28c84f", "input_pcm_sha256": "6cd1001f9a90653adb7fba68da8ba30554777620e33a8dced56d9b5d75e2decd", "output_wav_sha256": "d33ef0687ce11a66e02388404870ee1ed64005f9f5e0c0ec27472a8f69dd168d", "output_pcm_sha256": "7bf8021cc87859ec7f9293b319677d67c6f8f7c61ae28da1928dc6a5157f5915", "result_sha256": "a79f46319dc012d2b9db3a440a81da357be1b31ddfd1d8db11dcd7f1e2aeee07" }
#187formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "1e6c44a333ca4411ce1fc387486553017993ea00638fc53f47388e24f9f865c0", "started_sha256": "668001572ce57625f0f49b4b7a3311ec9892cbb668fb26f026966ddf19517b68", "original_grade_sha256": "21a8016077c44c9d7f4ea87af87763f1497fd07f16ca567545b3756e2cdc4d09", "input_original_sha256": "1e6c44a333ca4411ce1fc387486553017993ea00638fc53f47388e24f9f865c0", "input_wav_sha256": "720a7bdc297d66c5d7b59f6b0cc4a2e2b4cacc81cd6cb330fc8e93ca1dad0fa4", "input_pcm_sha256": "0188cfe185e97bc12477a22708a19108cdeb7ea481f5a3a84b3fa135dd59926b", "output_wav_sha256": "343e84c7c8b0c23b7ec63988059d2e732735d01b73dde18975b40651e948a747", "output_pcm_sha256": "1f40f1d6f66c19ca1bbc485fd976c4898f7d05a2f530d704d4a42179f5e103db", "result_sha256": "7cad666cf652e2345f0ebcd647592a80b1f226f0544640bfbd941d684e22d1a7" }
#480navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "4f4b6f639a05b585ea01c98ea81927d32f4cd551b7192be75391c3106854e430", "started_sha256": "51b35e762c16d589a2552182701ccb74e77819977196ec8334d6d8cdcbeac1ad", "original_grade_sha256": "de4953615a64c6d9441f5a6103a3b4144b982eadcf2dccca99dabdfcb87be20b", "input_original_sha256": "4f4b6f639a05b585ea01c98ea81927d32f4cd551b7192be75391c3106854e430", "input_wav_sha256": "1bf0822d8af86642572df35469815f37888d8c1f045962cd485c68298ab079f2", "input_pcm_sha256": "03b8027238c703678a15287a39eff1b19ac1c3b851b6afe2f9b43d0b1bacc4d7", "output_wav_sha256": "88536a0a44e92230a053cda0491b0075d319d8cf11fbf7f50745b70ccb87e54f", "output_pcm_sha256": "73f440fc642893cc6e25196d16a0a9961fe55926fac63e35b6519ec4e7784777", "result_sha256": "4530d0576dd7ba5acdacc2a85f91a80cd30d9d8e100a64e6bffe03ced4a065dd" }
#642object_counting heldout	6	The answer is 6.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "fe81aa774439315b5c5cabea49e131d9941eca49dc744e247f23d2288abaf12f", "started_sha256": "4b99ae857db8fa67ed2a91e6c0a53869fa605e36f01bedd203ab2cb877df553f", "original_grade_sha256": "06ab605983216bb97587d3b1355d8dbb8be208f340af89f930c235018be23eff", "input_original_sha256": "fe81aa774439315b5c5cabea49e131d9941eca49dc744e247f23d2288abaf12f", "input_wav_sha256": "a9e795c7ef1884d01cf4dd83c2c54faeef0aeacedd113a3e83a69cf6094cc8c0", "input_pcm_sha256": "00d2f1e4be73f55e64cc809737c15190d77a1aaca9b8f690e7115970dd5ee770", "output_wav_sha256": "6820875f2a4a04e6aefb617ae7dd5ec2406c6f6f83c4e5bf4fca6f1798a7f178", "output_pcm_sha256": "a1e6fe233e6a66aa3ebd9d83c814f63e99a26896343e67b365c1c14720675c20", "result_sha256": "7fbdc901c75ad5b674f779dff259c2d398b1fa5aaadb51369925ad936aed234d" }
#980web_of_lies heldout	No	That answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "b1d9cb6fa321a3da252deb788f8d6181311f371458d3af58a90d130117dac62a", "started_sha256": "a9759d662d07eb418537b4bf2042c82a5b596b641c0bdab9c08f17c8b66788d5", "original_grade_sha256": "bb9f729171130d06d28b51f8410e204e2a906a747bb7ff1f8ae4c176e523f1b2", "input_original_sha256": "b1d9cb6fa321a3da252deb788f8d6181311f371458d3af58a90d130117dac62a", "input_wav_sha256": "48042a7656146b80bfdb5e557548be8041c177d7acdedbc3603393b8eebfe06b", "input_pcm_sha256": "3f7d228257e2add65682af81e20fb696bb056e828ee54c42b3a56e69a623057e", "output_wav_sha256": "cfe1ac43693f3a6cafc1049afbcde0fb033dd48b5d61083f77db8e04d84a6dd6", "output_pcm_sha256": "b8cd020652d2702f52e9b69c9f2a3e4a8b05f90c8edf60cc3db59cc2870aab0f", "result_sha256": "80301c61115d9ded0f7cad476fc2363762e0527d25313df01e07d133d87b8395" }
#131formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "2f930c53f6f668d75dca141adbb11647802257b50cfa3b226b0244786eb1613c", "started_sha256": "53416e63179d4d3b1c01e65b7eefa6b2e3275bd36cfeea783b53cd331dd8add6", "original_grade_sha256": "d2993635f66cc20af8764271fc99cb8b9533bdd71f7e4f774bf0fb902ccadcb2", "input_original_sha256": "2f930c53f6f668d75dca141adbb11647802257b50cfa3b226b0244786eb1613c", "input_wav_sha256": "31d72072cb1ea105ac476c04cd4186df459c21ed1a13b305c3457279955d7dee", "input_pcm_sha256": "78eaf5325f52f195ad7b218297fbcccaf365a314d86fc089553ea05c520193e0", "output_wav_sha256": "699238b9b2b0936954e685079d0608e2e6141ac99172bd6b3a04dcab9993b634", "output_pcm_sha256": "d2f816d60ad572f0c5c642764dd1dfd00e74f91c6041b7ab6f5077be59270155", "result_sha256": "255f6f36fbaf2eada302727b665c402a65bf06b7b990378f2a0160cfc036cf8d" }
#370navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "3b370bb0b283404b0c08cc4ff0446afb3a4f318d04f9114829ca256d2214a401", "started_sha256": "b2b9901abaf898cbbb2ce5cb2209188b6493ae870f5077a85d847fd95025c506", "original_grade_sha256": "0cfa703ad9c026110016d437bdd895c04549f3ad829d5d49d76d7a87bf2c436d", "input_original_sha256": "3b370bb0b283404b0c08cc4ff0446afb3a4f318d04f9114829ca256d2214a401", "input_wav_sha256": "9e8f2266dc179bf83cf3368e22876214e241b594911aecddfde0654b061a43e3", "input_pcm_sha256": "fb2163289699a8aac1c829347b6dab560b904f65b9618772559a2daaaf1f6468", "output_wav_sha256": "5f65ba9aa84fcdfbb8b83b11671145f9ce5a58d1b936ec8b08f844528a488b17", "output_pcm_sha256": "4070bb421aad1ddb14e88da3c643c7a8de94a13c839023426e09b710f6b3ac51", "result_sha256": "812a02e7505477d5995de45bbf6a9c8cc4a53c4bb7cebd8d32630cdc36c0f512" }
#566object_counting heldout	2	The answer is 2.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "8233687c7f90201cc4953ee47544560465a367f4ca7e6a4b864846b1adc973ed", "started_sha256": "7d0bdf1d07cbac81050551802ab688e7d146b4c2df093efe0c83f767dfe80606", "original_grade_sha256": "805f6cb6490c1ad22d0a8a9c7a35eabcbfa4185855699f5004bed5d4ffed14c6", "input_original_sha256": "8233687c7f90201cc4953ee47544560465a367f4ca7e6a4b864846b1adc973ed", "input_wav_sha256": "e74a1f80524f43826ea730871c3e21681490e211a93272e72ea3ab571f928712", "input_pcm_sha256": "c1c6467aa7353eb9ab58109140a7fd64f30ad31b690f1b11eb758e6cf717757d", "output_wav_sha256": "3503a0e48668fe9aac0f23dea36b6b67f721ff18b53ba88051ed8478db9963bb", "output_pcm_sha256": "93789ad439890085617c1e148101723dba49f930d28591f8c6f7afb384476998", "result_sha256": "4be9b2bfc4e373ba59e23f47ba3525fb310120bb0ea5c516293fd2fd144da446" }
#775web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "00ac1e74a58fa7b2a35e8288ce24cf0b99e4082dec11404ba10ab9a93058fc32", "started_sha256": "1bdebce40615a152059f7fc626edb73d79af2a4c9cd6442ef6a2c5ddcc98f375", "original_grade_sha256": "80f060752f52d484fbb4bfe0e7b35596cf717bb1d587febde1bdcc6932f14187", "input_original_sha256": "00ac1e74a58fa7b2a35e8288ce24cf0b99e4082dec11404ba10ab9a93058fc32", "input_wav_sha256": "41924e0376e22f3e411d60158bfe159132acf63f1525277d3505054bdecf59e9", "input_pcm_sha256": "2889dae2e41f18a3d09ef9aa40b6cc7a44d7e9dc75616009e4d52cb4097f944b", "output_wav_sha256": "42d6ca38b95d96078b03aa9dae28eefbba3e3c70232dad63cc2da7c71d77fdb2", "output_pcm_sha256": "80e04b782748fdac7992d38a18538fc40e5db7c6e18608ab2041124ebbf9efff", "result_sha256": "d03f3dd4ccd20b99fdced171553643953ca053ad56897eb1ed81d029c5574a8a" }
#201formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "49ae22df266f05c8bf24b63fad70f61fec655ec4e2a621bf18cd8a46c3f41f87", "started_sha256": "adcb7ff2459ce5b686c17ae2e69477d421ee74016cd202f8f3a91b27f4da8722", "original_grade_sha256": "5572c7f90851b95f01e3eaf2f944b0c9facf7fb450789a159524799095b34156", "input_original_sha256": "49ae22df266f05c8bf24b63fad70f61fec655ec4e2a621bf18cd8a46c3f41f87", "input_wav_sha256": "3083c7ba67113a483fb64ab774e2126845612ed4d5e5a5c511039bc0e9a8745e", "input_pcm_sha256": "ac5423077024367eb334b4878781d5ce025f9d6aa0bbe955a272b6107ede85e4", "output_wav_sha256": "d3630774cba6dbbbe724f020ff8bd0081f3bd60b4687ef8357d1816aefb0c8fc", "output_pcm_sha256": "eeb079535defe8d5a320872dc157fe750d1d84ad40cabe0b62c39d0b72c010e8", "result_sha256": "201c547da65fac8df8370fb9e7595b61f5b6cf917a3488d2281b0b74cd190967" }
#418navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "7aaee2ce697fc644046677ffca40e1e3728ee9d69fa6c8fb572ff005772d5154", "started_sha256": "7fa2a25fa1b6e14f149b71b4f1172bc8c88cc7b1107053a576f1860ec2727974", "original_grade_sha256": "57a0799ef23b2702d535f2f69c102dd0a924ff8d79be2753a60f8e468ccfc483", "input_original_sha256": "7aaee2ce697fc644046677ffca40e1e3728ee9d69fa6c8fb572ff005772d5154", "input_wav_sha256": "e83af9f3b55129246b2b5026f87e3fbb065f3454ff4a13ad15372e388c87e9de", "input_pcm_sha256": "c6693a958af8bbc0120a6beca0d800e940606c5b1b80b414233097d18fe3e0eb", "output_wav_sha256": "c5135bdee013d15891f060332557cae443fe4b44943610aabc7125da493245b1", "output_pcm_sha256": "62328c36a038ee59f51b04c7a9294b913d6dcea70e0da67b0a0131862fdd152d", "result_sha256": "fcb938adfd24d591ae8a374b236469b536911cd2d9132fc7ca00b666a55099c6" }
#551object_counting heldout	14	The answer is 14.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "7f81a3d922cf31a26c8e4aa940232c103aace3d14e24720eb7124f054ffc4262", "started_sha256": "4703a899fc7f34a25b42c03815d57473100f1677564d731599c3cbad500a5594", "original_grade_sha256": "b7ee36940d6f4f1315d012110bcf82996b68505c3f5b93fe5620c757fe3d51dc", "input_original_sha256": "7f81a3d922cf31a26c8e4aa940232c103aace3d14e24720eb7124f054ffc4262", "input_wav_sha256": "52bdc1046e36868f540f26a528502667efbe50d185f83cd45b53833a70a9e3d6", "input_pcm_sha256": "58f046feaba088c245affe36e1785e6f19709fd39a0531f22235641abc69deed", "output_wav_sha256": "fc704cb8c1d1ca1bbc9c705ca5bbe6de7e68b4d7006f831c184f748c142f1c71", "output_pcm_sha256": "33435f055034044aefdf5639a15cd32a8b21d0bf1c47c2085d05b479cd5d8bcb", "result_sha256": "5ba8db07dc86c974201ce395f59f8ccf63daa15a2df7682547fa58e0cd378680" }
#858web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "41a4c758419894081033019787a54f7044bdea3b05b0b97bb182d77b871648ed", "started_sha256": "6468f07047ec739c20efc09ac8d7684b1ed582c0b288973fd9837086a38932a2", "original_grade_sha256": "7ce3035c0e0a2edfdf742370f0d434e18bebb7c0e48571621e45e26fb268ad6e", "input_original_sha256": "41a4c758419894081033019787a54f7044bdea3b05b0b97bb182d77b871648ed", "input_wav_sha256": "67f106be635a623d3923330602cde82fe14108a995201c70b4d1f5d757c5b6a8", "input_pcm_sha256": "4e4dee1631e6f3b13568482d432e1df6beebedef8672d2ddfd9b9ec033c2c915", "output_wav_sha256": "a9ca9f0c9370f3a99a7e5127fe36a524e212f1967d6db30e754e2ad8cd511266", "output_pcm_sha256": "a1383c37cd380821a1bf1d31f6e0e46b6fdd0ca92e16583b4e66653bb026db61", "result_sha256": "e0e298e43d11b5f40ac018ec5f5c9d652396214c4e9647a6b42e74dda174cd02" }
#36formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "6f9160a448e422be234227157b9f2224ee7c73c5727a71adf47b07cce12911ba", "started_sha256": "020a3e57557673f3d7dd190bc2909485623f9b0da435076f6224e4c075de5477", "original_grade_sha256": "b66420538278170920a7d6bd5f2918108ca94d7e61c099998baf93ceaceee3d8", "input_original_sha256": "6f9160a448e422be234227157b9f2224ee7c73c5727a71adf47b07cce12911ba", "input_wav_sha256": "bdbcab6f8a53c3673dc69247758b313d7c3cb960ed5cd5d91e349fd92e0e65ab", "input_pcm_sha256": "78bb99095f13e8bb9708e22065a797ab2982f70ce9589d41d6f0dc066781211f", "output_wav_sha256": "709f5ad18d6b48e9b75d5832cec3d9c2d8142a8d4d3c57902e8e8ec53c5ed7ca", "output_pcm_sha256": "49ad19e7df2411de2d1c297dea1d5af36ab9381ca282cf375e8e6b08222b8255", "result_sha256": "eb62f1611b5825b050d5c17ec12e38e6510f92d8dddc06c424f74163d64fbe2d" }
#379navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "a63f9462aae5b41ca8d514766ab824a3bf1a7e87cae2aa7819c5445a0e72d4d9", "started_sha256": "5b9e410373f6cf0535d31794f8f6ea7c31fbfe24b1294288ae8c933d40c88e40", "original_grade_sha256": "9aedd43c805030f88e6866fa67bb1ad838266a60abf8e7edfe0691dcec035013", "input_original_sha256": "a63f9462aae5b41ca8d514766ab824a3bf1a7e87cae2aa7819c5445a0e72d4d9", "input_wav_sha256": "5d9f53c514f6d222945f8adc2cc85d7ef24679b731783d81a1719200ddd82f53", "input_pcm_sha256": "2e4c51e4b4d1dda2ee2bf6e73698cd5514c3a81ccd85877e91a102c690c3e8bb", "output_wav_sha256": "2188796b9ef45576997045c24d0933960d03ba5ee36d0ab96f33e37163b155d1", "output_pcm_sha256": "fd89522df14b68cb59acdd23e42a7f6721fd39b63d3a5ff93abc5caf5c0f6762", "result_sha256": "9ac9d6ca7612a20ab7f9e732c211f9b245b118ed954adb2d2d38585d25f84bf7" }
#549object_counting heldout	10	The answer is 10.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "8724261c97f47b0d85acb76365a00198b5903da40c21efb948de690e271e8787", "started_sha256": "70f332bc442403dace5ff27a9337abf2d804d3e85c3b0149b0c7db4c2b342ac1", "original_grade_sha256": "e0d81f13bbd9cc9d01862b16dace9c162e32be0ca345d77b0daf34555962fc14", "input_original_sha256": "8724261c97f47b0d85acb76365a00198b5903da40c21efb948de690e271e8787", "input_wav_sha256": "46917d7fd20dde549adb14a33d9dafc1ac7d9ba0de73ae63de77fb1fb083cc7b", "input_pcm_sha256": "25630f180a89355052a906acf6e295bcc8501df939f175f5bd8fdb881c6d2a2f", "output_wav_sha256": "ee9990debd9115d8f604b056d91d91e72c21146e4d13bc5169538c5d1e605599", "output_pcm_sha256": "a9322a67f9dccb00a9aeb467d0a3345d8c76d6dce389962776923633a811d7f2", "result_sha256": "5112575b28860a553f208108882c993e3d832ba26b23b41c1cce63037f2054d7" }
#785web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "dfa3e7562e5ead757a53494de27f71b063d1baca13f71e4cd71bd338cb836347", "started_sha256": "8b54233c0c871d5d7d36997737fa22bc4aa3b0ee1e169a5450bc087391923d48", "original_grade_sha256": "90d3061ab7d0f7e2cc503dd734886f2e56a67be84deb2cb95a602a559b58d57f", "input_original_sha256": "dfa3e7562e5ead757a53494de27f71b063d1baca13f71e4cd71bd338cb836347", "input_wav_sha256": "77774e67facc6197d7f756cf767f887a06d11080dd0888b4e3d5f7bd0653c933", "input_pcm_sha256": "86443246f29ebbff4d6d7cad06b4d29b5d88cafef9b58d4ff4eecbcc54625494", "output_wav_sha256": "2b167ebfc1b729622556b3ae16b3ad1344be04365fceabd5f1a011187556e892", "output_pcm_sha256": "4b0745a607e5050e8d03e78709ebb8473fdfe1a3b46eb727c21b0a57adfb69c1", "result_sha256": "481c81e2ec4bca6749a7588a3d17aeeae381e63d2ff5311edefbfd3577a90839" }
#26formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "68ecee17b0b26cd7b8ff5f3fe2691d2fdcd13b84751dd7afa57e58523b6b79cb", "started_sha256": "b43502bbcbe135e20df149cccd5f0f6c0ffcbe0ef1ddc96ac868d77e87aad90e", "original_grade_sha256": "e1f924891171fd2242310e002bc7f5fc045dd9c435340c1bbff416e5eed5fc48", "input_original_sha256": "68ecee17b0b26cd7b8ff5f3fe2691d2fdcd13b84751dd7afa57e58523b6b79cb", "input_wav_sha256": "4583c06738c2ae0127def16fa16c87078fd7a635a0c1def2b3c8cd7217a7cf82", "input_pcm_sha256": "c6352541fec8237e17ac979a5858723df95cb5122e482f69809f4b72fe41e505", "output_wav_sha256": "d3065a5f70c05d9ceb66aa204f9bd37a48f5d0f88d714c9e321b7b33c5c2f091", "output_pcm_sha256": "bee6820434d6058a4ef81a951a19ccea0db2006d5770745002ab91acb6a2ff3a", "result_sha256": "b089ba7d5c6a399d9a7ab29543d3301bb1ac6b2c1bd8dcc512cc584e7355baaf" }
#328navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "2bccd25be213214e7a2f082dad7cb3fd0735229727d585cda67a0c663f7f7838", "started_sha256": "0f9144faed5c4c078cbaf20dcb75d429195ec8ee1bba97a12e59dec04d8f164e", "original_grade_sha256": "3191b6aa02dba93492d1e453b21bf42f13affa4e95775d94f8505c93ae6caa94", "input_original_sha256": "2bccd25be213214e7a2f082dad7cb3fd0735229727d585cda67a0c663f7f7838", "input_wav_sha256": "9acdb4f7eb83917271b033eab351b51fd3ee4d964f4d4caddc76abd9c6cd8e78", "input_pcm_sha256": "2bb1841ca31f8216df8a54ac224857ef26e61aa0f15a1dbd82aa74d50c251671", "output_wav_sha256": "195b5e9bfabab1a20903982d833e1f04e173d3f36e95a3d975a15ffe886299f5", "output_pcm_sha256": "5736a29da33c21cb2fdd33bbc96bc1cbf7c54cecf5e73f39b5add10beb5e8335", "result_sha256": "5e3c2c310c521d4a2a3d0c93db177c6131be04a59adc01df03e12dd789812b92" }
#613object_counting heldout	9	The answer is 9.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "7c218a77f8e06206b5a165a6edb84b285162fc0e8bac3d4af128a25aadf36140", "started_sha256": "061f2b52bc032cceaa2ab84a4ff8d77a719ec2efd6daac0bb80400460e51d2ba", "original_grade_sha256": "89650aa540054a17aed4e0ca390c2d9c8aa77e943611339c2764d6c7a4794744", "input_original_sha256": "7c218a77f8e06206b5a165a6edb84b285162fc0e8bac3d4af128a25aadf36140", "input_wav_sha256": "4155012219f997c774e2d5cdcd4886b9072762cefadefbf9607b91216a8d9b79", "input_pcm_sha256": "f8df46139c4223f227c63675b4f21aed2960615bbef9ee2622babf1110a4ddb1", "output_wav_sha256": "2284f084d6b5544a2c64e6e7bf546ca01874972b662fe0198d59ec77812b8f1b", "output_pcm_sha256": "56be0065d9cf78f7e9fb5fe1d61a62b801af29417fac1919da8814627f991ece", "result_sha256": "ac96ea797db1f900a640126caaf7e6f49a53283c3f597aa0f525b90aa8e4be32" }
#962web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "fec7be0e6b861ce39c02399802ee22df9e34989e676c6bb76a4d25dc2d10ea85", "started_sha256": "1dfd07dffa45524618c10e02b888e229490f52b8925d97d5e25b8499446fd2c3", "original_grade_sha256": "6b7e34986a8e0632cab39d3852a83cd046707ecf1d9ce65cf47102ec20741206", "input_original_sha256": "fec7be0e6b861ce39c02399802ee22df9e34989e676c6bb76a4d25dc2d10ea85", "input_wav_sha256": "18476f0a4f06bf45797e0c91c7a3a6567434bfaecb421592775baf0f418cd162", "input_pcm_sha256": "3aee6706217e1e3a7b250e4e9ee75fe0fbba6e6e817959c7da29a18dbe35b329", "output_wav_sha256": "38455247204a56be0b6648bcabc8b670488b81be45641ea7448c84abad7b91b8", "output_pcm_sha256": "9ec0794e91c3e83a9360acc603b3dce97623a6fe5c083afb689ba672d4e70ba4", "result_sha256": "b5ffaf4657d3d158503074d57e5b543e1ac3984389109b6ff5ad07f3148a6fac" }
#66formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "4efa16064db056863801f3194f08b33babfc1496e6814e9f13e6997052390605", "started_sha256": "8325da1dfca208bd955dc1ce16c13f6f7785d574732527d87aeb5952f1abc12f", "original_grade_sha256": "ba2603b0ef33ba7dc4fee222e91890ef51e5c0daaf49c22429c74ee747fa0756", "input_original_sha256": "4efa16064db056863801f3194f08b33babfc1496e6814e9f13e6997052390605", "input_wav_sha256": "8f89d53df9bb80fb39ff8f7f7cd8b20005da7598cbc5e5931fcb34358200331b", "input_pcm_sha256": "47972ef09e80fe932975e6441756ff14a94093b8fd94d3c798ee8877655d3df4", "output_wav_sha256": "cb27c367e9cecf9c14652c7fa0d52b32c54042aa079145ffe211eece6cbf6b5a", "output_pcm_sha256": "d2946d62f9e31ff113859ccd100ac3b442446daa8de93a02f1763225f7493ee2", "result_sha256": "b702d409f0ca47c22c36c1b27ae16760bd64a4b47c74172201f919c70d86a2e7" }
#356navigate heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "1e422133e7456f3af29133934cfe5ae4378c06dc430734306031ae4fedf08b10", "started_sha256": "98cb5ab6086f643505a251555649011081edabcf653e3937b73544b1005fed17", "original_grade_sha256": "980eced75107588d6cf75d4c949a4ae7f8281d49344dfea403363ff96faccf9a", "input_original_sha256": "1e422133e7456f3af29133934cfe5ae4378c06dc430734306031ae4fedf08b10", "input_wav_sha256": "bae84dba7fcbd95c93d529f90d10374f5df5c6ebf47174fec51a8fd8e5c73ae4", "input_pcm_sha256": "1207c5e80da51e18ccca13a5e8397808466d72a4ce60b58e52f901b25dea15c4", "output_wav_sha256": "cc1120928ed5963ed954f140b54bc9dd4a14a98e7c91d38ab3c90a011df35337", "output_pcm_sha256": "3d4fdef0afec118fa3e2151abf469cef00e380e3b6191fbe584f90966f85b3c2", "result_sha256": "f73532bd06f7acba33b69dc6a430e0d336b8099191073cfd4f9de71eb7c3c017" }
#593object_counting heldout	8	The answer is 8.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f3a427b48a35449be0474a407dd57194762501d49710ac0e1559fc2964d5804c", "started_sha256": "054dca8c21972cfe791fe47fc93ba2b6989c4c98dbe99a80ad9c5351423d3ee8", "original_grade_sha256": "4cdb71a408cf05bfb3dc8b496852bc153197fabde4ffe75ac62285ccc31156ad", "input_original_sha256": "f3a427b48a35449be0474a407dd57194762501d49710ac0e1559fc2964d5804c", "input_wav_sha256": "c6470b1c975ad7b6c0f485fd2ad3b277e96ea89b54cf72d55ae8d4fd30def535", "input_pcm_sha256": "69f251c069ba71f8d9feaaaa93338e52fb1b9f50f3fd4726dbf9f4afc6edcaa6", "output_wav_sha256": "2a358c4effe1af9b1531587b50c2e41f863dc091abf7840132d5d309fb0d443d", "output_pcm_sha256": "6817202501baea055f8c50532a3ee3527082a7d72070fd82a01f555403733e36", "result_sha256": "664f52bb3eab47b8b6e77cbcb7831acfd857badfe68307ca10d5911a151069e4" }
#857web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "dd9ddd77f8240006807a74ad139df95a4b4334d4175bfa949bc98fdcf26cd884", "started_sha256": "bbdec7e8e5631aa1156ac5270e06e7ce13b2b8bc253dffc1708a28e6cae93a8a", "original_grade_sha256": "c9d69938d3fa9b7639fb2fb5fde2e68cc6cf5dc1349a4619ca3f5490fa47e8da", "input_original_sha256": "dd9ddd77f8240006807a74ad139df95a4b4334d4175bfa949bc98fdcf26cd884", "input_wav_sha256": "2ae1d7745def22fe28d719eeea35f2c59adcf780f27258e80b40e7eba8c1cd77", "input_pcm_sha256": "9f31ec18f7c9d4ad5e8c2b4764734dd5ca4f788233daa1abd1be802091402e29", "output_wav_sha256": "ad2af5f88706c127d44c1cbf3bfa22e84c40f7aa094b3f72066a1a699f494ba3", "output_pcm_sha256": "391874aadc009b09b3109f5724b8611eae168b587f97be35334cc4f1747578d5", "result_sha256": "bbec76380a54ba9148b60bf8b2000ba7a09d637697a8649401734f74818cc5f1" }
#219formal_fallacies heldout	valid	The answer is valid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "1aff59afe29bb7b0df20e9ef0e3fecae3a2eb8f75c420331769aa236ae0e3464", "started_sha256": "7fe20ee4ea0b39e11e4627b5849c62d4b717037ae8d5f364c0c3d4c7b7ebc14d", "original_grade_sha256": "db3984c44fab6968ddce6f991c98586120003fd95ddca6a83ab9e5f03a2d02b6", "input_original_sha256": "1aff59afe29bb7b0df20e9ef0e3fecae3a2eb8f75c420331769aa236ae0e3464", "input_wav_sha256": "dc003105178b2f1b874ba3b8859122d181d6e83118189b9d71318f5f6e1b5285", "input_pcm_sha256": "888201692c30b5664ef87bccc4f279d64c5bdfb8d41324a10fc161ab3626d4b6", "output_wav_sha256": "5e29b6ac62da619f93a698ada4dab8945c5063b26eccabe44a4683ce79550a01", "output_pcm_sha256": "5ae114c617823ae85468d7e5ff9cbd6fe2f41054fd4036537d4291f9be81aeee", "result_sha256": "b7ccece4a46931da10a9cfd1aa9873c63c22f77288e7feb204f7f6d0fe229aca" }
#323navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "f70632263c80fe6cc50fd9baf484a4d55bed1e4b0d4c7320064243797ad5f673", "started_sha256": "3b48d16a99c43609494a36adccfba4630ad2a0a0cf0cc1228d3069e8fec20d82", "original_grade_sha256": "7e0bc3f3e26a914e5ce1a6fdff7e8b55cbb15dd8d196080cf9383fe175588bda", "input_original_sha256": "f70632263c80fe6cc50fd9baf484a4d55bed1e4b0d4c7320064243797ad5f673", "input_wav_sha256": "fc6deed5dd32d48ee36eea6e51ede590e6717e87b35ecca31c2a28e41980fd15", "input_pcm_sha256": "592ec2b773a76a48acceb943e33bf23270f7ceed13caeae96acaafd84e55d146", "output_wav_sha256": "26dd04fdba7a584eee2891da50e5e8041d3a647adbfff79888e389392e09d226", "output_pcm_sha256": "a0521fee15fcda4c037351009e7dfeae73918ed605b9e3d1277f3ae5e8a9a5e3", "result_sha256": "c971e9c31c08e4770fb5e26134b4d81f8a60169c162cb94f441463113e84c86a" }
#706object_counting heldout	11	The answer is 11.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "e726dcf74ae92cfb53c3bf8b8f7bb7f7dba528c94a6cce073a575386157efa4e", "started_sha256": "9bdae0d8b4b68246527527cdfd4be9935fd329b65e8556ec7c303aa82579d77e", "original_grade_sha256": "23da3d86b6c33a4c13d2e8ff874cccff0c101f3012fff53d5df2bf98129384d1", "input_original_sha256": "e726dcf74ae92cfb53c3bf8b8f7bb7f7dba528c94a6cce073a575386157efa4e", "input_wav_sha256": "d8aa7c06e98ee2935fbd2fcf0aabb21689eb81d3b4e9cc379c97ac283d246410", "input_pcm_sha256": "6d64ed213abc7ed99267e7943d8e85f76621e87a4a2010849360a10fa59d4912", "output_wav_sha256": "3abc54190e43f8f2d7690ae90da24136da93183ea051270be2f26dff58982207", "output_pcm_sha256": "581bfb0f122e7b6c57628bc0446ee827401ee55efbae31be913aa772e182d385", "result_sha256": "4a98ee760a68d4c8668ddd1d26f89dd60d1be01c97b9ca0d52187b609234bebd" }
#811web_of_lies heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "a7df043afad8c9dac1d064b012018fab217119c0bad8386044bf05dd327841e3", "started_sha256": "b3aaad64eeac28f57c39b2a860f6e7bf84e8631ef6ed8071101458f92c90401b", "original_grade_sha256": "8834f386cc02afea6611cb5a7693979412543170ac9bb8208ec216a09545b848", "input_original_sha256": "a7df043afad8c9dac1d064b012018fab217119c0bad8386044bf05dd327841e3", "input_wav_sha256": "b0a5e0c868f8654e1d28c77248a1271e4c26b882f464f57ac0f134470d6315b4", "input_pcm_sha256": "3e0f5c332f77e1fea87a1051198592e7a0a7aec42aa209f8ddb83deb57f7032a", "output_wav_sha256": "9e41075909ec6242d69710d8b1868684346c829709c4387800fdf80cdfa78977", "output_pcm_sha256": "213ba7059ca0ecf9073bc74c8f9b1ba0fbb74179fd00d7745446dd71dc13f783", "result_sha256": "a4b2fa3ba1437f9aa5838d03184c0479ddfc8ebfd129a1e2de32cb286b5007db" }
#42formal_fallacies heldout	invalid	The answer is invalid.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "841536db6b8753f2738fd9e796e5837fef1788bac050806959d240cbeec80046", "started_sha256": "eb8ce3525643bf8af2a8c0bdf07ab05efe608ce84cb2572760181e20f4ddfc92", "original_grade_sha256": "fbb54087e023e0fe8273442ed044cb8f3816c165f2f459c0d430f9b95fee2336", "input_original_sha256": "841536db6b8753f2738fd9e796e5837fef1788bac050806959d240cbeec80046", "input_wav_sha256": "2defe4ed4001235f80c8b1d348966dacad1dc3cb360cb22289a7c8442d344628", "input_pcm_sha256": "0b36e56d165c8c7ac8939b425db5fbac1fea867b9f909fa0443e6e77e4ef0f93", "output_wav_sha256": "999fed8a1d214715954b26e639fe0eca0a91802047627d2e7e28c77f8ef1a4d7", "output_pcm_sha256": "617260f6a987ec1dc268837fd02598b42f3d0fcd6297578966dbcc0ce505f994", "result_sha256": "514faa2eb7636fab21064344e4845e4f8455b56b3820f1cae04b898c4348267e" }
#402navigate heldout	No	The answer is no.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "600ae2181ab95ccbcdbc0a97aeec9898efce3cd421d5e3ec2506f6eb5c1800df", "started_sha256": "2227c2f44a49cd1270e1b3fb365358ad6938a03ed0322400fe111d40c35535fc", "original_grade_sha256": "9d1151b27f064ad842fccd734944a416a211bd8117a6f5b31e4dd9b9af1ec2c3", "input_original_sha256": "600ae2181ab95ccbcdbc0a97aeec9898efce3cd421d5e3ec2506f6eb5c1800df", "input_wav_sha256": "3b98aab4f1bc2b2abc507958c337b62339fc65a435de61da7921f34625935b12", "input_pcm_sha256": "9956590dd75ea15bfcf38c71960ec153052b1077e481b4d74af33f0be3ffa98b", "output_wav_sha256": "f4b9c98bbaefab42ff8f1f6fed24afff6ba86c87ae1a9b57079c88d17c23b422", "output_pcm_sha256": "cd590a85a0051dd1dbcfac640e7e7882f892f0d3192b4256f397e3d376cff6d4", "result_sha256": "6641508f30dc4039a507a12d9ef871b456dc7eeae0ef948105f54b2a90195589" }
#525object_counting heldout	5	The answer is 5.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "c3099df94ee759ae53b25c72acade05151ec8e76a92df407b2309dc4dc0445f9", "started_sha256": "331e09016e2c398d281ad00d8950d9a08c1e9113730750e2e89d684a9b418d8c", "original_grade_sha256": "d52b64a105da36913d23b64a0dcc3d2dd7bea8cbe66ec83bf73dca750941e90c", "input_original_sha256": "c3099df94ee759ae53b25c72acade05151ec8e76a92df407b2309dc4dc0445f9", "input_wav_sha256": "5fc9119c3a0040647f8fe90267e4644ff7cafa752c3941f1f84cf89b4b85c0ea", "input_pcm_sha256": "cba4eb38542959f5be180ef0341eed940f08ca4227a993c943f41bcda7429f3e", "output_wav_sha256": "107960a428f503381fbaaa2d2f34e762e411c38387e6072cdd71932417ea9972", "output_pcm_sha256": "b88e58ce69123d9e4c077a9372ef6bf7ffe8e0ba627d60ed3b70044ee3fd100d", "result_sha256": "f176cf5c67dcb3939e7b47ece3fc0e37f1079194ce4544b49574f97123b4f5a8" }
#812web_of_lies heldout	Yes	The answer is yes.	correctstrict_spoken_answer	correctresult	Original grade retained Evidence hashes { "original_audio_sha256": "405e4cfafb80808fbc6290a2117098f8f021881582e453b5b70f7c8381944dad", "started_sha256": "ad82abd66d834317506d38e7836a41a3005e8757ddaa22c02c66d528536a5cda", "original_grade_sha256": "b74441aa2fe6e8fd085783cb93468ee71ba5175392b395162b49d53429b00999", "input_original_sha256": "405e4cfafb80808fbc6290a2117098f8f021881582e453b5b70f7c8381944dad", "input_wav_sha256": "7eb8905c9743a9b8cec0c4bbf94058c7a532c148b1e134eb65bc1b471f0fdcdc", "input_pcm_sha256": "5a6878f221bedf7555642088d9498c7c964c02a4f2b94facd396b4a11db76fb9", "output_wav_sha256": "89b8eed25a36dd22157f54e4c700f1affbae7754cc873074bf22e64aaf368111", "output_pcm_sha256": "02351d8112c90da510038a5c74e602ace776c9b0481da6e0b8f59fa5f9f30d5b", "result_sha256": "5bccd42f98cce9dffe09a9aee7933f29e810cfac2198ba486097b5a1b9cc0ad4" }

Score it yourself

The same stack, behind a door we can open for you.

This number is self-run, so it is not an official Artificial Analysis result and we do not present it as one. If you grade speech-to-speech systems, we will issue you a key and you can produce your own number from your own harness. Send one question’s audio; get our spoken answer back.

curl -sS https://davidagents.ai/voice-benchmark/answer \
  -H "Authorization: Bearer $DAS_BENCHMARK_KEY" \
  -H "Content-Type: audio/wav" \
  --data-binary @question.wav

The reply carries the answer audio as base64 WAV (PCM16, mono, 24 kHz), the answer text, the input transcript, per-stage timings and the same configuration hashes printed above, so you can confirm you reached this stack and not another one. GET /voice-benchmark/configuration returns that identity on its own.

Behind the door today: ElevenLabs Scribe transcribes the question at request time, Qwen 3.8 27B on Cerebras answers over our internal gateway with no tools, no fallback and no retries in a fresh working directory, and the same Piper voice speaks. That is the stack of our 2026-09-19 gateway run; the response's reference_runs says which published run it matches and which it does not, and lists every way it differs from a frozen run - the main one being that this handler lives in our live voice service rather than the sealed worker container, and the input recognizer is called during the request instead of before the batch.

The door answers one question at a time and yields to live customer calls, so a full 1,000-question pass is paced rather than parallel. Ask us for a key.

Method and limits

The first 100 cases were exposed during development; only the other 900 were held out from that development sequence.
Original grades and additive audio reviews are separate. Errors and terminal dispositions count against the full denominator.
The tested offline Opus/local-speech candidate is not the GPT-Live Ultra demo or a claim about live phone/tool performance.
Reviews describe saved waveform/ASR evidence, not independent human certification. CLI-internal HTTP retries were not instrumented.

The dataset contains 250 questions in each of four categories. Source: the pinned original BBA dataset. Artificial Analysis uses its own methodology; this result must not be ranked as an equivalent official score.

Read the earlier eight-case GPT-Live/Cerebras pilot. Its post-run clean-completion metric and different configuration are retained separately.

Download the complete curated JSON · Publication checksum: 5c400fe8f5692b8cc0a01fa6e7739c6099bbdcad668688e656121293458d1eeb