Sry, what's the "active" part here? Is the model actually generative? I'm aware of Karl Friston and the free-energy principle. Is the active part the input stream selection? I thought that the active part refers to learning, in a sense that I get to pick my training data along the way. Sounds like what you're doing is akin to Gato from DeepMind with tokenization and is about multi-modal policies (modulo the hierarchical processing and attention).

Is there a math writeup somewhere?

Hire an engineer or depending on your business plan a team of engineers. It is not something that can be solved in a laptop, especially finetuning and without ML background

It's a model that "chooses" its input stream from a 2d array of sensor data (cam, mics, and servo encoders) in real time using policies decoded from predictions of the bottom layer. Then, it processes this input up the hierarchy of identical layers. Higher layer predictions are used to modulate attention.

It may qualify as a general intelligence (idk) as any data can be encoded into the format of its input stream. What I mean is that I have a particular way of encoding video, audio, anything really, into a universal format which preserves the salient semantics.

Currently, it is greatly inhibited in what it can learn because I cannot feed it experiences at the rate it could take them. It has far more potential than realized knowledge.

Thanks a lot for the detailed reply! I will try these suggestions.

Can you contribute datasets?

That would help so many!

GridSearchCV?

Can you share the problem and solution?

Probably. I'm still getting fired if I do something like that without permission.

Hey thanks for sharing all this - it's all very fascinating.

I'm interested in what kind of work you've been doing with models that use active inference.

Attention is still pretty confusing for me. I find diffusion much more intuitive fwiw.

VAEs are running under the hood in stable diffusion. Instead of denoising a 512x512x3 image directly, the image is encoded with a VAE to a smaller latent space (i think 64x64x4). The denoising steps happen in the latent space, and finally the VAE decodes the result back to color space. This is how it can run relatively quickly and on machines that don't have tons of VRAM.

So it's not necessarily the case that these techniques die. We can learn and incorporate them in larger models.

Yes, it's mentioned in the post

is it possible to fine tune GPT for static code analysis ? if yes...what would be the training set looks like?

I had no problem fine-tuning GPT-3 on a million sequences using an rtx-2060 super with 8GB ram. Just follow the huggingface tutorials…

That said for non-nvidia GPU’s I’m not sure how it would work.

Forgive me for my ignorance but isn’t TTTS opensource?

I think worth looking at for sure. The math behind isn’t “that” complex and the idea is pretty intuitive in my opinion. Take that from someone who took months to wrap their head around attention as a concept lol.

How should I think about the way a large language model gains new specific knowledge? For example, suppose you have a model trained on hundreds of gigabytes of text and then want to continue its training to gain knowledge of a single specific fact it has not yet encountered such as “Steven Pinker is the author of The Language Instinct.”

I imagine that presenting it with a single sentence such as this embedded in a training set would contribute very little to its ability to subsequently answer the question “Who was the author of The Language Instinct?” Is that correct?

Is there some heuristic for how many exposures a model like GPT3.5 would need to a new fact, as such, before its weights and biases were adjusted enough to embody this fact?

Thanks for the reply and a solution :)

If you access the instance through browser, then yes it will be close when you close your browser. You need to SSH in using terminal or browser and use screen command first. Then inside that screen, start your model training or anything else that is using GPU. Now don't terminate screen(google it) and close your SSH session or browser window and your program inside screen will be still running

Ok, thanks for the reply

It depends. For colab, only highest tier pro account allow you to turn off the computer. The machine still runs at GCP. Different providers have different setups. I think you don't have to leave browser open if the machine is just port forward to your local machine port. The best is to contact your provider about your instance.

Oh ok, what service do u recommend that stay on when I shutdown my laptop and is fairly cheap?

If you’re planning on doing anything serious don’t use vast.ai. I had bad reliability experience with them

Depends upon which platform you use. Services like GCP, Paperspace, AWS, Azure do not close down after you shutdown your PC. However, it also depends upon which platform you're using to access the remote server.

I'm thinking vast.ai