e

  • 1 Post
  • 62 Comments
Joined 1 year ago
cake
Cake day: June 15th, 2023

help-circle



  • AdrianTheFrog@lemmy.worldto196@lemmy.blahaj.zoneRulecles
    link
    fedilink
    English
    arrow-up
    11
    ·
    28 days ago

    I checked the list of 370k english words I downloaded from github a while ago and yeah, its true other than the variants of homeowner (homeowners, homeownership)

    I was looking at some other random words, heres some I found:

    • self: weaselfish, damselfish
    • eye: greyer, honeyed, journeyed, etc
    • bear: beard

    this got me interested so I wrote a program to find each time a small word bridges the gap between two larger words in a compound word, honestly the funnier part of its outputs is the weird ‘compound words’ its finding, like “asp: aspirating: as, pirating” or “at: deepseated: deepsea, ted” (ted, apparently, meaning ‘to scatter hay for drying’). Occasionally it finds good ones, like “ices: apprenticeship: apprentice, ship” or “hen: archenemy: arch, enemy”, and it did find the meow one. It does allow the small word to contain the first word in a compound word, because that can still give some interesting ones like “warp: warplanes: war, planes”. It probably would have been a lot better if I had actually used a list of compound words, it tries to find its own very slowly which does allow it to find any possible combination for any word

    anyways, here’s the list






  • AdrianTheFrog@lemmy.worldto196@lemmy.blahaj.zoneThe Rule
    link
    fedilink
    English
    arrow-up
    5
    ·
    2 months ago

    I don’t have access to llama 3.1 405b but I can see that llama 3 70b takes up ~145 gb, so 405b would probably take 840 gigabytes, just to download the uncompressed fp16 (16 bits / weight) model. With 8 bit quantization it would probably take closer to 420 gb, and with 4 bit it would probably take closer to 210 gb. 4 bit quantization is really going to start harming the model outputs, and its still probably not going to fit in your RAM, let alone VRAM.

    So yes, it is a crazy model. You’d probably need at least 3 or 4 a100s to have a good experience with it.






  • AdrianTheFrog@lemmy.worldto196@lemmy.blahaj.zoneRule elitism
    link
    fedilink
    English
    arrow-up
    7
    ·
    2 months ago

    Unions would probably work, as long as you get some people the company doesn’t want to replace in there too

    Maybe also federal regulations, although would probably just slow it because models are being made all around the world, including places like Russia and China that the US and EU don’t have legal influence over

    Also, it might be just me, but it feels like generative AI progress has really slowed, it almost feels like we’re approaching the point where we’ve squeezed the most out of the hardware we have and now we just have to wait for the hardware to get better