Open LLMsMoiz GhansarMay 13, 2024, 10:28 p.m. 
The AI boom has long since arrived and sure, it's exciting and all. Language tasks in particular have become a lot easier. Just the day before I heard someone used ChatGPT to write a letter to the passport office (and their application was granted). I myself use GitHub copilot on projects where I need to code stuff that is more or less common knowledge to me. The typing out is the overhead and copilot helps me overcome that. (People get triggered when they hear students using copilot. I do not use it when I need to solve a problem. The process is important.)


Restriction

Many student devs like me built projects wrapped around OpenAI's GPT API. Pretty cool stuff. But-
There's always a feeling of restriction

    

  1. There's a rate limit
    2. 

  2. These tools, services, etc. are all on the cloud
    3. 

  3. Most are paid services
    4. 



Discovery

I browse Reddit a fair bit. It's user base was made to be diverse because of the subreddit structure that houses communities for all kinds of people. Really nice place.
One day I stumbled across r/LocalLLaMA. It's a subreddit which I assume started as a place for people to talk about Meta AI's Llama model. But now, you can talk about pretty much any LLM there. The most common subject of talk is normally Open LLMs.

Most orgs that have released their commercial closed LLMs that are quite capable and useful. In addition to that, many have also released the weights of the smaller versions of those LLMs (as small as 128k params and as large as 70b params, as per my info). After reading through a few of the posts on this subreddit, I was intrigued. Look at this post!.
LLMs are amazing. OpenAI has showed us that. But openly available LLMs that anyone with even a 8GB RAM, i3 9th gen, GT 1030 can run is where I started to realize that maybe I did want to work with LLMs.


Conclusion

Ended up making something. It's very low effort. Not much really. It's just for fun, afterall. Take a look. This is where I start.