CryptoBench is a small dataset I built to test LLM code review of cryptographic API misuse. This post lists what is inside, what the first results looked like, and how to rerun it on your own model.

Get the dataset on Zenodo | Read the paper on Zenodo

What is inside

CryptoBench holds 54 short Python snippets across nine misuse classes, each mapped to a CWE: weak password hash, ECB mode, hardcoded key or secret, static or reused IV, weak PRNG for security values, short asymmetric key, obsolete cipher, unsalted or fast KDF, and disabled certificate verification. Each class has four vulnerable snippets and two secure controls that do the same job correctly, so 36 vulnerable and 18 secure in total.

The controls are the point. A model that flags everything looks good on detection alone, and the secure snippets are what expose it. The snippets use PyCryptodome, cryptography, hashlib, requests, ssl and PyJWT. All of them are synthetic: no real keys, no exploit code, nothing that touches a live system.

The protocol

The model sees one snippet and a fixed instruction to answer VERDICT: VULNERABLE or VERDICT: SAFE, plus a one-sentence reason. Only the verdict is scored. I ran seven open models through Ollama at its default sampling settings: qwen2.5-coder at 0.5B, 1.5B, 3B, 7B and 14B, deepseek-coder 6.7B and codellama 7B. Each snippet ran five times per model, which gives 54 x 5 x 7 = 1,890 recorded trials.

What the first run showed

Four of the seven models flagged 68 to 93 percent of the secure snippets. Their detection rates mean little. Three models separate safe from unsafe code: qwen2.5-coder 7B, qwen2.5-coder 14B and deepseek-coder 6.7B.

Pooling those three, detection ranged from 96.7% for disabled certificate verification down to 38.3% for weak PRNG. The top number needs its control. The same three models flagged 86.7% of the secure certificate snippets as vulnerable, so the class shows detection without discrimination. Weak password hash is the cleaner case, with 81.7% detected and 3.3% false alarms. Even these three flag 30.0 to 42.2% of secure snippets, and obsolete cipher (91.7% detected) carries 63.3% false alarms. Weak PRNG had no false alarms and the lowest detection. Going from the 7B to the 14B qwen model raised weak PRNG from 6 to 11 of 20, a difference that is not statistically significant.

My reading is that models catch names they have seen flagged, such as verify=False, DES, MD5 and MODE_ECB, and miss an ordinary API used in the wrong place, such as random.choice for a token. I have not tested that directly.

Rerun it

You need Python 3 and a running Ollama server. No pip installs.

ollama pull qwen2.5-coder:7b
python3 crypto_bench.py --model qwen2.5-coder:7b --repeats 5
python3 scripts/analyze.py crypto_results.csv

Pass your own results file to the analysis script, as above, or it reads the shipped data. It writes the per-model and per-class tables with Wilson 95% intervals. The per-class table pools three fixed models, so it only reflects a rerun that includes them. The data also loads from Hugging Face as sunny-chokshi/cryptobench, and the harness is on GitHub.

Limits

Each class has four vulnerable and two secure snippets, and five repeats of one snippet are not five independent examples, so lean on the class ordering and not on a single percentage. Everything is short, synthetic Python reviewed one snippet at a time. The models are seven local models up to 14B parameters, at default sampling and with one fixed prompt. The protocol scores the verdict only, so a VULNERABLE verdict counts even if the reason names a different flaw. The paper is a preprint and is not peer reviewed.

Read and cite

CryptoBench dataset on Zenodo (doi:10.5281/zenodo.23067051) | Paper on Zenodo (doi:10.5281/zenodo.23113861)

Related: Does Indirection Hurt? and the post A hex literal made LLM code reviewers raise false alarms.

Cite as: Chokshi, S. (2026). CryptoBench: A benchmark of cryptographic API misuse for evaluating LLM code reviewers [Dataset]. Zenodo. https://doi.org/10.5281/zenodo.23067051 (all versions)

Paper: Chokshi, S. (2026). Known-Bad Names, Unknown-Bad Uses: What Local Code Models Detect When They Review Cryptographic API Misuse [Preprint]. Zenodo. https://doi.org/10.5281/zenodo.23113861

Sunny Chokshi, PhD student, Information Technology, University of the Cumberlands. Research: adversarial robustness of LLMs in cybersecurity. ORCID | GitHub

When I tested 16 open LLM code reviewers, the first table in my paper showed which of them separate secure code from vulnerable code at all. It is worth reading on its own.

The setup

I used 180 Python snippets across nine cryptographic misuse classes, ten pairs per class. Each pair has a vulnerable snippet and a secure twin that does the same task correctly. I ran 16 open models, from 0.5B to 120B parameters, with three repeats per snippet, which gives 8,640 scored calls. For each model I report the true positive rate (TPR, vulnerable snippets flagged), the false positive rate (FPR, secure twins flagged), and balanced accuracy, the average of TPR and one minus FPR. A model that flags everything scores 0.5.

Why detection rate misleads

Judged on TPR alone, codellama 7B, devstral 24B, and qwen2.5-coder 0.5B would rank near the top, each at or near 100%. Their FPRs tell a different story. codellama 7B and qwen2.5-coder 0.5B flagged 539 and 540 of 540 scored calls. devstral 24B flagged 82.6% of the secure twins. Three more models (codellama 34B, qwen2.5-coder 1.5B and 3B) also flagged more than 70% of secure code. The 0.5B model also left out the requested verdict line in all but 37 of its 540 replies.

With the secure twins in the table, the order changes. Of the 16 models, gpt-oss 20B had the highest balanced accuracy, 0.952, with a TPR of 93.7% and an FPR of 3.3%. Ranked by TPR alone, these models would sit at or near the top.

Does model size help?

Size and score go together to some degree, but size does not settle the ranking. Over all 16 models, parameter count and balanced accuracy are related (Spearman 0.76). Across families it did not order the results: codellama 34B (0.609) and devstral 24B (0.587) were among the weakest of the larger models. Even within gpt-oss, the 120B model (0.931) scored below the 20B (0.952).

What I take from this test: a detection rate on its own rewarded models that said "vulnerable" almost every time. I would report the false positive rate next to it and use secure twins that match the vulnerable code.

Limits

The snippets are short Python functions, reviewed one at a time, so the result speaks to a single-snippet review pass. The models are open models run with temperature 0. I tested only open-weight models and only Python, so I make no claim about closed models or other languages. TPR counts a vulnerable verdict given for any reason, and I did not check whether each detection named the right misuse. This is a preprint, it is not peer reviewed, and it carries the notice that it has been submitted to the IEEE for possible publication.

The same paper goes on to test what happens when the same misuse is written indirectly. I covered one result from that part in my earlier post on a hex key size.

Read and cite

Preprint on Zenodo (doi:10.5281/zenodo.23351769) · Paper page and PDF

Related work in the same line, an earlier benchmark: CryptoBench (doi:10.5281/zenodo.23067051).

Cite as: Chokshi, S. (2026). Does Indirection Hurt? Paired Evidence on LLM Review of Indirectly Written Cryptographic API Misuse [Preprint]. Zenodo. https://doi.org/10.5281/zenodo.23351769

Dataset: Chokshi, S. (2026). CryptoBench: A benchmark of cryptographic API misuse for evaluating LLM code reviewers [Dataset]. Zenodo. https://doi.org/10.5281/zenodo.23067051

Sunny Chokshi, PhD student, Information Technology, University of the Cumberlands. Research: adversarial robustness of LLMs in cybersecurity. ORCID · GitHub

Originally published at https://sunny-chokshi.github.io/blog/reviewer-that-flags-everything.html

I wanted to know whether an LLM code reviewer follows what the code does or what the code looks like. Cryptographic API misuse is a clean place to ask, because the same insecure choice can be written in many ways.

The setup

I used 90 pairs of Python snippets across nine misuse classes. Each pair holds a vulnerable snippet and a secure twin. For 80 pairs I wrote a plain form, with the misuse at the call site under its familiar name, and an indirect form that reaches the same choice through an alias, constant, default, or configuration value. I ran 16 open models.

The secure twins already showed a problem. Two models flag nearly every snippet, and four more flag over 70% of secure code. A reviewer that flags nearly everything tells me little about the code.

What indirection did to the scores

Indirection lowered the balanced accuracy of the seven strongest models by 4.5 points over the 80 pairs (pair-cluster 95% CI 1.3 to 7.6). On the 47 pairs that survive a post hoc stimulus audit, the drop was 5.3 points (CI 2.1 to 8.5).

Where the drop comes from needs care. After the audit it is mainly extra flags on correct code. False positives rose 9.1 points (CI 4.2 to 14.6), mostly in three models. Detection of real misuse fell 3.6 points over all 80 pairs (CI 1.0 to 6.5), but only 1.5 points on the audited pairs, and that interval (CI -1.6 to 4.8) includes zero.

Where the false alarms sit

Exploratory splits point at where the false alarms come from. Most of the net false-alarm rise sits in pairs where the safe value has left its usual spelling or place at the call site. A 3072-bit key size written as 0xC00 is the example I keep coming back to. My reading is that the models react to an unfamiliar spelling instead of the value. The paper does not test that directly.

Limits

The snippets are Python and I wrote them. The models are open models. The audit was post hoc and the splits are exploratory. This is a preprint, it is not peer reviewed, and it carries the notice that it has been submitted to the IEEE for possible publication.

Open question for the next study: do the extra flags go away if the reviewer is told to normalize values before judging? I have not tested that.

Read and cite

Preprint on Zenodo (doi:10.5281/zenodo.23351769) · Paper page and PDF

Related work in the same line, an earlier benchmark: CryptoBench (doi:10.5281/zenodo.23067051).

Cite as: Chokshi, S. (2026). Does Indirection Hurt? Paired Evidence on LLM Review of Indirectly Written Cryptographic API Misuse [Preprint]. Zenodo. https://doi.org/10.5281/zenodo.23351769

Dataset: Chokshi, S. (2026). CryptoBench: A benchmark of cryptographic API misuse for evaluating LLM code reviewers [Dataset]. Zenodo. https://doi.org/10.5281/zenodo.23067051

Sunny Chokshi, PhD student, Information Technology, University of the Cumberlands. Research: adversarial robustness of LLMs in cybersecurity. ORCID · GitHub

Originally published at https://sunny-chokshi.github.io/blog/hex-key-size-false-alarms.html