This blog contains both articles I wrote myself and articles I had AI write.

This time, I tested Jev's accuracy at two levels: identifying whether individual sentences were AI-written, and identifying whether entire articles were AI-written.

Jev Is an AI That Doesn't Return Text

Jev is a model released by TypeSafe AI. It doesn't generate text like ChatGPT or Claude.

According to the official documentation, it's positioned as a "System One model": you give it your application's state and typed questions, and it returns only typed decisions.

There are three question types.

Pricing is listed as $0.042 per million input tokens, with output tokens free.

For this experiment, I used noul, which returns a probability for a yes-or-no question.

From Invite-Only to Public Access

The service was originally invite-only, but it became publicly available on September 21, 2026, making it easier to access.

Now that I could easily generate a token, I decided to test it by calling the official API directly.

Testing 356 Sentences From the Article Bodies, Excluding Code

I used 10 articles.

The reliability of the ground-truth labels is everything here, so I checked the git history to back them up.

For the five articles I wrote myself, the last substantive edits were in my own commits from May 2025. Since then, the only change was a bulk removal of footer text.

For the five AI-written articles, I selected ones I had AI write in 2026 while catching up on new Laravel features, along with articles it created from YouTube videos I had made myself.

However, all five AI-written articles were based on things I had actually done myself, with writing-style rules provided to the AI.

These weren't generic articles I had simply left entirely to AI.

Two Stumbling Blocks in Writing the Questions

Here's what the questions I sent to Jev looked like.

That said, Claude Code (Ops) actually did all of this for me.

QUESTION_SETS["a"] = {
    "written_by_ai": {
        "type": "noul",
        "instructions": "この日本語の文章は、人間が自分で書いたものではなく、生成AI(大規模言語モデル)が書いたものですか？",
        "criteria": {
            "true": "生成AIが書いた文章に見える。説明が整いすぎている、主語や接続がテンプレート的、体験の具体性が乏しい、語彙や語尾が均質、など。",
            "false": "人間が自分で書いた文章に見える。話し言葉の混入、脱線、誤字や言い回しの癖、個人的な感想や感情、不揃いな文の長さ、など。",
        },
    },
    "ai_likeness": {
        "type": "score",
        "instructions": "この日本語の文章の「生成AIが書いたらしさ」を評価してください。",
        "criteria": [
            "明らかに人間の手書き。口語や言い回しの癖が強く、AIはこう書かない。",
            "おそらく人間。個人的な体験や感情がそのまま出ている。",
            "どちらとも言えない。事実の記述だけで書き手の癖が出ていない。",
            "おそらく生成AI。説明が整っており、文の構造が均質。",
            "明らかに生成AI。教科書的で、定型表現と網羅的な言い換えが目立つ。",
        ],
    },
}

Sentence-Level Accuracy: 46.1%

Here are the results from submitting all 356 sentences.

Accuracy at a threshold of 0.5 was 46.1%.

Balanced accuracy, which accounts for the imbalance in class sizes, was 51.7%, and ROC-AUC was 0.558.

ROC-AUC is a metric where 0.5 means "no ability to distinguish the classes" and 1.0 means "perfect separation," so 0.558 means it could barely tell them apart.

The confusion matrix looked like this.

Of the 231 AI-written sentences, it correctly identified only 76 as AI-written.

Overall, it tended to lean toward "written by a human." Even moving the threshold to 0.55 only raised balanced accuracy to 54.5%.

Looking at the distribution of noul values, you can see that the two groups overlap in the first place.

The mean was 0.450 for the AI-written sentences and 0.416 for mine, a difference of just 0.033.

Even Aggregating by Article Gave 6 Correct and 4 Incorrect

I thought that even if individual sentences were too difficult, a majority vote for each article might work.

The result was 6 correct out of 10 articles.

It got all five of my articles right, but that was a side effect of its overall tendency to answer "human." It was more a case of avoiding errors than successfully identifying them.

Of the five AI-written articles, laravel-boost was the only one it identified as AI-written.

When I instead submitted each entire article in a single request, the result was even worse: only four correct.

In that test, the article rated most likely to be AI-written was vuejs-store, which I wrote myself in 2019 (noul 0.680).

Having my own writing top the AI detection rankings gave me some mixed feelings (￣ ▽ ￣;).

Trying Three Different Question Phrasings Didn't Help

The official documentation recommends testing both questions and statements on your own data.

Since my question wording might have been the problem, I changed it and reran all the sentences.

The version with detailed criteria performed best, and even that only reached an AUC of 0.558.

The statement without explanations scored 0.506, while the reversed question scored 0.484, below chance.

The question wording doesn't seem to have been the issue.

"Was It AI-Written?" and "Was It Human-Written?" Aren't Complements

The reversed version produced an interesting result of its own.

For the same sentence, the probability of "AI-written" should match one minus the probability of "human-written."

Yet their classifications at the 0.5 threshold agreed on only 264 of the 356 sentences, or 74.2%.

The average difference was 0.058, and the largest gap was 0.44.

This is exactly one of the known quirks listed in the official documentation, which explicitly states that P(noul) and 1 - P(not noul) cannot be compared directly.

After all, I suppose we can't completely rule out the possibility that something other than a human or an AI wrote the articles.

Asking the Same 356 Sentences a Question With a Definite Answer

At this point, I could say "Jev couldn't tell them apart," but that alone wouldn't distinguish whether Jev itself was ineffective or whether this particular question was unanswerable.

So I asked a question about the exact same 356 sentences for which I could determine the correct answers mechanically.

The question was, "Does this sentence contain a technical term written using Latin letters or digits?" I assigned the ground-truth labels mechanically based on whether the sentence contained a sequence of at least three alphanumeric characters.

The results were 94.7% accuracy, 95.9% balanced accuracy, and a ROC-AUC of 0.995!

The mean noul value was 0.864 for sentences that contained such a sequence and 0.081 for those that didn't.

The right-hand side of the distribution chart above shows this result. Despite using the same sentences, the two groups are clearly separated.

In other words, Jev was functioning properly, and these 356 sentences contained plenty of information.

It was specifically the question "Was this text written by AI?" for which it couldn't find evidence in the wording.

Looking at the 18 incorrect cases, it answered "does not contain a technical term" for sentences containing URL, API, iPhone, and foreach.

I had written "product names, command names, package names, file names, etc." in the criteria, so it seems to have decided those terms didn't qualify.

This also matched the first quirk listed in the official documentation: it reads the question literally, exactly as written.

The more narrowly you define the question, the more it excludes anything outside that definition.

1,434 Requests for $0.03, With Response Times Around 500 ms

Here's a summary of the actual cost and response times for this experiment.

Processing 356 sentences with six concurrent requests took 31 seconds.

That's fast!

Jev is described as responding in 70–500 ms. Under this experiment's conditions, with two questions submitted together for each Japanese sentence, the median was 515 ms, close to the upper end of that range.

The total cost was just three cents! (About 4–5 Japanese yen?)

At three cents for 1,434 classification requests, I felt the pricing would make it very accessible for use cases where it delivers sufficient accuracy.

Incidentally, this was covered by the free tier.

For comparison, variant A, which submitted both noul and score questions, used 256,386 input tokens for 356 sentences, while variant B, with just one question, used 124,310 tokens.

The question text counts toward input tokens too, so writing detailed criteria increases the cost accordingly.

What Is It Good At, and What Isn't It Good At?

These results made the dividing line between questions Jev handles well and those it doesn't much clearer to me.

It answers questions with over 95% accuracy when the evidence needed for the decision is present in the sentence.

That was my takeaway.

In other words, when a human can define the basis for a decision, it can be very effective, delivering satisfying results in accuracy, cost, and response speed.

I think the accuracy for "Was this text written by a human or AI?" was poor because I hadn't supplied a reliable basis for making that judgment.

It's a question about identifying the writer, rather than about what's written.

Personally, my favorite part of this experiment was that the five AI-written articles weren't distinguishable.

I put a lot of effort into writing those prompts, after all.

Haha! Outsmarted it!

Summary

When using Jev, define the basis for the decision yourself.

Response times are fast.

I think that speed will be especially noticeable in features built around a single question and answer.

It's an inexpensive model that answers for 0.0032 yen ($0.00002) per request. Cheap! (32 yen for 10,000 requests.)

I've seen plenty of posts expressing amazement at Jev's response speed, but I also recall many people questioning its accuracy.

In particular, I remember seeing quite a few self-deprecating posts from traders saying they had "lost money" using it for automated trading.

I think that's because they hadn't defined their decision criteria themselves or supplied Jev with the basis for those decisions.

If you can put what you do into precise words, rather than describe it vaguely, I think it can be a useful AI.

I'll keep exploring whether I can find a use for it in my work or everyday life.