I have always felt like the network/security and integration with internal it systems was worse than the physical maintenance. Like people should expect that they have to invest time into integrating into a on-prem data center environment or physical maintenance stuff. I think small teams are benefited by a small gpu cluster with a fixed budget over large cloud gpu training costs. Mid-large companies do better with cloud than on-prem bc they can have better separation of environments but they cost more.

I'd do GLM-130B

> With INT4 quantization, the hardware requirements can further be reduced to a single server with 4 * RTX 3090 (24G) with almost no performance degradation.

https://github.com/THUDM/GLM-130B

I'd also look into pruning/distillation and you could probably shrink the model by about half again.

Thanks! I was just curious for future reference. I’ll need to first read the papers to see if it can help with my projects.

I think Ray is great! But Ray will not click your GPUs into a motherboard, install linux on all the machines, setup nvidia-docker, power cycle if there are issues, periodically clear up space on hdds, etc. Its the non-software part of cluster management that ends up being the most annoying and time consuming.

If u use ray u can setup a gpu cluster in less than 30 min

vscode extension pls

perfect answer is perfect

Ah, I see. I didn’t know. I guess you could do it that way.

Check out this comment. Some things that we take for granted from low-wage humans are incredibly hard for computers and robots. Think about valet parking. Our society doesn't think "Oh my goodness, valet parkers are geniuses!!!" But it's really really hard to build a robot that can do what they do.

I had this exact thought of using VAE or BYOL etc to generate powerful representation for text/sentences and then train a diffusion model on continuous latent data.

There are some things that are incredibly hard. Imagine you work on a farm. You toss the keys to the ATV to a 17yo farmhand who's never worked for you before. You say, "Head over to field 3 and tell me if it's dry enough to plow. You can see where it is on this paper map. Radio back using this handheld." The farmhand duly drives the ATV to field 3, sees that it's muddy, picks up the radio, and says, "Sorry boss, field 3's a no-go."

We're a long way from a robotic farmhand being able to perform those skills, certainly not for a price comparable to a farm laborer.

You could definitely train an application-specific AI to monitor fields and report on their moisture levels. You could even have an algorithm that schedules all of your farm equipment based on current conditions and other factors. So it's not that AI can't revolutionize how we work, it's just that it'll be different from true AGI.

Why don't you think AGI will ever happen?

What I mean is that faiss can compute knn for a variety of metrics including cosine distance. So you can just directly index by cosine distance instead of L2

Cloud is almost always better imo. At the small scale you can prototype quicker and spend less time messing with hardware by using cloud services. Once you actually need to scale your product then using a cloud solution makes it really easy. The "but its cheaper" argument gets less and less valid every year, and it often doesn't account for the time and effort spent setting up a local cluster.

Mine’s in Tensorflow 2.11 — I’m sure writing a PyTorch version wouldn’t be hard. The extra lines of the algorithm are three lines in my paper. I can share my code though?

Yeah, I see a lot of goalpost-moving, but in the end, it depends on how you define "AGI", some people have varying definitions. I think even a language model can become AGI eventually.

It should be able to capture some transformations of the original images, but maybe I should think about measuring that. Thanks for the idea!

Well, if you're storing 1 million images in the database, it's going to take a long time to do the cosine distance for all 1 million images. FAISS will give you very roughly the 1000 nearest and you can do the cosine distance from there. My usage was anybody could enter any text phrase and search my dataset. I can't precompute the cosine distance for every query somebody might make.

>If you do this approach, I would recommend asking Faiss to retrieve a few more images than you need, then calculating cosine similarity yourself on the images Faiss retrieves to get the 'best' matched images.

Why not just index by cosine distance in the first place?

Would love this to be a VSCode plug-in! Happy to drop our OpenAI api key in there.

Same license as the 32 bit version so commercial usage is fine (apache-2.0 - see the page for details) Should give similar results and scaling (according to the link above is 1-10% slower inference).

I don't think AGI will ever happen, but with enough task-specific applications, the difference may become academic.

What made you decide to run an on-prem server instead of going to the cloud? I'm a data science manager and I'm currently looking at our options. I like self-hosting for most things, but I'm up in the air about training deep learning models.

So you want to create a business that allows users to write stories by just "using the API"?

Tell me your background in business....

Which model are you using from openAI ?