Sometimes I get into arguments about AI and people comment weird stuff here on Lemmy. Like we have ethical AI models (meaning they’re not made up of unethically sourced training material). Or we got plenty “libre” or “free and open-source” AI models.

Am I missing something? Or is this just a bunch of people trying to bullshit me? Is anyone here aware of such large language models and would be so kind as to educate me? I mean we do have a ton of open-weights models. But proper open-source or ethical??? I’ve been following the news for quite a while and I’m aware of a very select few I’d call LLMs with training datasets available. I’m aware of:

  • Pythia by EleutherAI (up to 12B, from 2023) and Dolly 2 by Databricks (12B, also from 2023)
  • BLOOM by BigScience (176B, from 2022)
  • GPT-2 by OpenAI (1.5B, from 2019)
  • The recreation of LLaMA by the RedPajama-Project
  • Apertus by the ETH Zürich

Edit: In the comments we got:

  • LLM360 with K2-65B, Crystal-7b, Amber-7B
  • the Soofi-Project
  • OLMoE

That list needs a lot of footnotes, though. Some of them come with restrictions, terms and conditions… And none of them can do modern tasks like AI-assisted coding. The only one getting close is the Swiss model. Sadly the earlier version I tried didn’t perform well. Seems they released an updated v1.5 version this summer, I wonder if that’s better and anybody uses it for real-world applications.

So… Back to my initial question: Are there models out there with datasets available? I mean except the exactly one somewhat usable one I already found?

    • hendrik@palaver.p3x.deOPEnglish
      4·
      22 days ago

      Huh. It doesn’t come with all the datasets, though. Does it? The model card contains a long list of “Private Non-publicly Accessible Datasets”, both by NVIDIA and by third parties. And some of the stuff just says “Undisclosed”.

      • e0qdk@reddthat.comEnglish
        2·
        22 days ago

        “Private Non-publicly Accessible Datasets”

        Huh. I thought they were fully open. I missed that. Most of the data sets and methodology is open though. (Maybe I’m remembering an older model? Or maybe I just fell for their marketing a while back… 😔️)

        NVIDIA’s own proprietary stuff looks largely like machine translation examples, and some sort of game logs (Sudoku and Minesweeper 🤨️), plus a few things I’m not quite sure I understand (“Multitool Typewriter Hard”? Is that for OCR maybe? Edit: or maybe they mean ‘multitool’ in an agentic sense… and that’s for tool call reasoning?).

        The third-party stuff is these six entries:

        • Global Regulation
        • TAUS Translation Memory
        • Scale HLE
        • HackerRank Coding
        • RL data for Search
        • Mercor SWE-AgentsV1

        I wonder how hard it would be to rebuild something equivalent that is fully open. 🤔️

        • hendrik@palaver.p3x.deOPEnglish
          2·
          22 days ago

          Sudoku and Minesweeper could very well be some internal project names and have nothing to do with what the words mean. I have no clue.

          Seems to me their announcements ooze with marketing speech. I bet they put it all in one sentence to mislead about the datasets. Put a comma there and omitted the word “partly”. It’s technically borderline correct. I’ll have to read up on it, whether that’s a good amount of data, or just some tiny fraction. It’s not looking good, though.

          • e0qdk@reddthat.comEnglish
            2·
            22 days ago

            It could be, but using the games as logical puzzle solver/reasoning tests might have value – I don’t know.