Found relevant code at https://github.com/ridgerchu/SpikeGPT + all code implementations here

--

To opt out from receiving code links, DM me

• Whats [R] and [N] in title ?

• Whats a dropout??

TNN is like convolution, while RWKV can be written as a CNN too (RWKV v1 is a CNN). So there's some similarity, though not much :)

Yes a paper is coming. Meanwhile you can read https://arxiv.org/abs/2302.13939 (SpikeGPT) which is inspired by RWKV and has plenty of explanations :)

Will the 150 lines help? Please read the code first :)

https://github.com/BlinkDL/ChatRWKV/blob/main/RWKV_in_150_lines.py

This is ALL you need for RWKV inference.

And you can read https://arxiv.org/abs/2302.13939 (SpikeGPT) which is inspired by RWKV and has plenty of explanations :)

Bro sounds like the discussion comments in some of my university courses

Neat! What's early s.d. in the tables in the github repo?

I don't think the author of the post took a position on the original argument; rather they just presented ways to explore the latent space and make comparisons that are reasonable so that we might derive better distance metrics.

I see it as a potential way to probe for evidence of mode collapse.

People said similar things about deep learning a long time ago.

If you can use supervised learning, then you should, because it means you have tons of data with ground-truth labels for each decision. But many real-world problems are not like that. Even humans don't know if each of their decisions is optimal or not.

If you are successful in getting noticed, you may get sued. If you are just one guy (not a company) may be not. But tread carefully. There may be a restricted licensing way you could show your work if you want to, but I am not an expert there.

And lazy

Like come on at least have a landing page

Great comment 👍

>Do GANS really model the true data distribution...

I find their argument to be pretty weak. Of course these images look semantically similar; they ran a semantic similarity search to find them.

They are clearly not memorized training examples. The pose, framing, and facial expressions are very different.

hmm I very much doubt it couldve ran 100x faster for the same parameter count, as you are memory bandwith bound (both GPT and RWKV have to load the parameters n times to generate n tokens). Also Im somewhat skeptical that you only need 3GB for 14B parameters *without offloading the model*, as even 4-bit quantization is 14B/2 = 7GB needed. and offloading the model is slow to the point of being unusable as you need to do CPU<->GPU transfers.

> people who discount GANs due to their lack of a likelihood

I was going to ask you to expand on this a little, but instead found a post that describes it pretty well for anyone else who is curious:

Do GANS really model the true data distribution...

For further nuance on this topic, Machine Learning Street Talk discussed interpolation vs extrapolation with Yann LeCun regarding interpolation vs extrapolation, which Letitia Parcalabescu summarizes here.

>Same way as one wouldn't use RL to multiply two numbers

🤔 https://www.nature.com/articles/s41586-022-05172-4

IDK did you read it?

Totally agree.

I have been following this from some time but I can't fully understand it and explain it to my collaborators.

I work in ML and I have quite some experience with transformers and I still can't fully get it. Let alone convince some of my collaborator that is worth pursuing it.

It is paramount that we have a paper that explains this in more detail if we want the community to consider this seriously.

Please do it!

How can authors be confident that this phenomenon is generally true?

[deleted]

Yes. In the first two lines of the abstract:

> Introduced by Hinton et al. in 2012, dropout has stood the test of time as a regularizer for preventing overfitting in neural networks. In this study, we demonstrate that dropout can also mitigate underfitting when used at the start of training.

For sure it can be taught. But I don't think the way to teach it is to give it a bunch of sentences from the internet and expect it to figure out advanced reasoning. It has to be explicitly tuned into the objective. A more interesting question is, then how can we do this for all domains of knowledge in a general manner? Well, that is the question. In other words, what is that master algorithm for learning? There is one (or a collection of them) for sure, but I don't think we are much close to it. ChatGPT is simply pretending to be that system, but it's not.

People will say anything in hope of drawing attention. Reframing an unexplored MDP into a supervised learning problem makes no sense.

Indeed.

It once said 100 times faster and 100 times less (V)RAM here. However, it now says that RWKV-14B can be run with only 3 GB of VRAM, which is regardless a massive improvement, because a 14B model normally requires about 30 GB of VRAM or thereabouts.