Hungary 🇭🇺🇪🇺

Developer behind the Eternity for Lemmy android app.

@bazsalanszky@lemmy.ml is my old account, migrated to my own instance in 2023.

  • 3 Posts
  • 23 Comments
Joined 1 year ago
cake
Cake day: July 2nd, 2023

help-circle
  • From what I’ve seen, it’s definitely worth quantizing. I’ve used llama 3 8B (fp16) and llama 3 70B (q2_XS). The 70B version was way better, even with this quantization and it fits perfectly in 24 GB of VRAM. There’s also this comparison showing the quantization option and their benchmark scores:

    1000029570

    Source

    To run this particular model though, you would need about 45GB of RAM just for the q2_K quant according to Ollama. I think I could run this with my GPU and offload the rest of the layers to the CPU, but the performance wouldn’t be that great(e.g. less than 1 t/s).


















  • Thank you for the fast response!

    1. The saved post functionality is not implemented yet. I want to add this functionality in the following days.
    2. The “karma” score should be removed, I agree
    3. The subscription part is interesting. Is this on the main page or the subscriptions page?
    4. I’ve also experienced slow loading times, mostly on the main page. I believe this part of the code causes this issue
    5. Sadly, the “cannot fetch user info” error is shown right now when It fails to get the access token (for any reason). If you can provide me with some logs or open an issue about it, I can take a look at it.

    Thanks again for your quick review. If you spot any major issues with this build please open an issue on codeberg.




  • I am planning on releasing an alpha version in a couple of days. It won’t be anything fancy, just some basic functionality.

    I got access to Codeberg’s CI yesterday, so I will set up a pipeline for this project, and I think you will be able to download it with Obtainium this way. I also want to push for an F-Droid or IzzyOnDroid release as well.