Gemini 4 Neon is the new #1 on the Models Engineering Index

(git-aarvin.github.io)

36 points | by hn-aarvin 1 hour ago

5 comments

  • YuechenLi 12 minutes ago
    Ok, so I can see that this is a benchmark the submitter made and submitted themselves, which is kinda misleading and should at least be disclosed.

    But my question is, what does this benchmark actually measure about the LLM's engineering abilities? It just looks like a page of random bar graphs to me.

  • quantisan 24 minutes ago
    When and why should I trust this index over others?
    • ummonk 20 minutes ago
      I wouldn't trust any particular one, but obscure / indie indexes are going to be less affected by benchmaxxing than the well known ones, so it's a positive sign for Neon that it is performing so well on a lot of them including this one.
  • js4ever 19 minutes ago
    Is it just me or all Google models are terribly bad? They all seems benchmaxed to hell and producing total crap compared to Claude or openai models from my own experience (mostly coding)
    • hadlock 12 minutes ago
      After hearing some good things from my tokenmaxxing buddies (or whatever you want to call them, AI enthusiasts), I got a $20/mo sub to try out Neon and it's just breezed through some hardware graphics bring-up problems I've been struggling with on Sol/Astra and Fable. I'm gonna drop my $200 OpenAI sub this coming month in favor of Google's $200 plan just for more Neon tokens.
    • asadm 3 minutes ago
      this week I ran out of codex astra quota and switched to antigravity, boy is Gemini Carbon amazing model! It actually helped achieve a breakthrough I was stuck on for weeks with Astra.
    • SwellJoe 15 minutes ago
      I think it's just you. Carbon is excellent but extremely slow. If Neon is better than or equal to Carbon while also being a lot faster, that'd be a great model. I haven't tried Neon, yet, but they seem to have understood the assignment with the latest round of models.
    • mjburgess 16 minutes ago
      And tokens/task is very high. They don't seem to have figured out making the models usable in a typical interactive workflow, ie., they're in a Opus-4.7 era.

      Very high hallucination rates too.

    • Yahivin 10 minutes ago
      With Neon Gemini is good now.
    • copperx 17 minutes ago
      Are you saying Neon is just as bad?
    • azan_ 14 minutes ago
      Yes. Every Google model so far has been disappointment. And their biggest moat, the integration into other google products has been laughable. Today I've asked gemini in gmail to download all attachments from one e-mail address and it turned out it's unable to do it. It's unbelievable how incompetent they are, Google has such a massive edge over OAI and Anthropic and it's still losing.
      • andunieee 5 minutes ago
        You mean they are not providing you with their best models for free?
  • andunieee 5 minutes ago
    How do I use it? Do I have to create a Google Console account and navigate through 15 menus in order to generate an API key? Or are there another 7 different ways to achieve similar outcomes?