ChatGPT may be not on the level of AGI yet (even though some think it is -
https://www.lesswrong.com/posts/HguqQSY8mR7NxGopc/2022-was-the-year-agi-arrived-just-don-t-call-it-that)

But the preogress of AI training does not show signs of slowing down, and there is a very big possibility that we will reach it soon.

Also, even without being AGI, AI can be extremely dangerous.

I wrote find a masters thesis from a notable deep learning person because they will have laid out the related work since the “beginning of time“, and it will be organized and each sub group will be related

Is that meaningfully better than just googling?

Talking Nets from MIT press has some nice interviews with pre-90s neural net researchers. Also iirc there’s some commentary on those early papers in the Neurocomputing 1 and 2 collections, also from MIT

Thank you!!!

Going all the way back to the Perceptron .. or maybe even earlier?

You'll be very, very busy!

Complementing your answer...

I've actually used ChatGPT before posting this thread, and the answers were all unsatisfying, just like in the comment above yours. Actually looking into the resources recommended by ChatGPT will quickly show that most of them simply do not meet the criteria I specified in my post. The only useful reference is the book by Haykin, but it looks like the historical references are somewhat scattered throughout the book, which makes it not so easy to read.

Also, I've not come here after just opinions. I've come here hoping for informed opinions from experienced researchers, who may have already read and evaluated many of the references on NN history that I'm asking for. This is something ChatGPT is very far from being able to provide.

Great question and you're right we did not cover this (alas, we could not cover everything even with 3 hours). You can unroll NN training as a sequence of gradient updates. The gradient updates involve nonlinear additions to the set of weights at initialization (e.g. the first update is w -> w - grad_w(L), where w is randomly initialized). Unrolling the entire graph is a large composition of such nonlinear functions of the weights at initialization. The Master Theorem, from a bird's eye view, is precisely the tool to handle such a computation graph (all such unrolls are themselves tensor programs). This is how Greg's work covers NN training.

Note: This is just a cartoon picture of course. The updated weights are now highly correlated in the unrolled computation graph (weight updates in a given layer depend on weights from all layers), and one has to do a careful analysis of such a graph.

Update: Actually, Greg did discuss this unrolling of the computation graph for NN training. https://www.youtube.com/watch?v=1aXOXHA7Jcw&t=8540s

[deleted]

I’m a bit confused, are you unable to use their approach in this case? It seems like using embeddings is the work around to get more context when the text is too large.

The author of the paper (Schmithuber) has more works on the history of nns. Check out the second link for more.

Here it is accurate, but for more technical questions chatGPT very often invents references!

Having spoken to Greg (who may or may not be chiming in), it appears that the authors of PDLT were only considering one kind of infinite width limit (as evidenced by your use of the word "the"). But Greg considers a general family of them. The NTK limit indeed has no feature learning, whereas Greg analyzes entire families, some that do have feature learning, in particular, one that has maximal feature learning. So there is no contradiction with respect to past works.

Automating anything that currently requires a human to work I assume

Nice, thanks!

I'll make a counter argument. If you are going to compare ChatGPT to Google, then why do you feel it is appropriate to directly paste the results of ChatGPT on Reddit?

I ask because Google can have factually incorrect search results that must then be evaluated by whoever is performing the search. By posting the raw output, without applying any of your own critical perception onto the suggestions, you are merely dilluting the level of intelligence in the conversation.

So on Reddit, a forum for humans, I might remind you, why would we post the results of an automatic query of whatever kind, that then requires effort to determine if the results of the query are actually useable themselves?

I think it's ok for you or anyone else to use ChatGPT to inform your answer, the same way its ok to use Google or Wikipedia. And yes, mentioning that your answer came from ChatGPT is nicer than just pasting from GPT directly.

Another suggestion, if you have read enough ChatGPT responses, you'll notice a pattern in them, that they are full of filler or have a hard time just getting to the point. This filler is not desirable for a quick response (such as on a forum, or in this case, getting a list of references). Compare the length of your comment to the other comments here. Do you see how much longer your comment is?

Lastly, I know that there are a lot of easily Google-able questions on Reddit, but for something subjective, such as this forum thread in particular, then the kind of information being sought is not objective or measurable. The OP is asking for opinions. Let's assume they've already Googled "Books on History of Artificial Neural Networks". Since ChatGPT is trained on the same dataset as what Google accesses (the Internet), there is no benefit in copy-pasting the output of ChatGPT. It does not have novel opinions, it simply aggregates known existing ones.

I look forward to your introspection.

Neural tangent kernels as an idea are old. They predate deep learning. To my knowledge not a single practically useful fact came out of these analysis yet.

I'm not at all up to speed on this, but I followed most of the presentation. I was left with this question, though.

Up to the latter part of the video, I was left with the impression that this was building a rigorous theory of what happens if you forget to train your neural network. That is, the assumption was that all the weights were taken from independently sampled Gaussian distributions. The "master theorem" as stated here definitely assumed that all the weights in the network were random. But then suddenly about 2.5 hours in, they are talking about the behavior of the network under training, and as far as I can tell, there's no discussion at all of how the theorems they have painstakingly established for random weights tell you anything about learning behavior.

Did I miss something, or was this just left out of the video? They do seem to have switched by this point from covering proofs to just stating results... which is fine, the video is long enough already, but I'd love to have some intuition for how this model treats training, as opposed to inference with random weights.

>Copyright

Thank you for taking the time to write this up! I am super glad, if my writing spurs a discussion.

On 3) I think that their brand partnership would not necessarily need to an in- and out-group. If we take the example of Nike investing heavily into breaking the record of running a 2h marathon. They were not really creating two groups either. They were injecting themselves into the narrative of a community aspiring the do something great. This gave them loads of brand lift. If NVIDIA would have been part of the Alpha Fold project, it could have been the same. I think they would be happy if they could replace "Deepmind pushed the envelope of protein folding problem" with "DeepMind and NVIDIA...". Obviously, replace protein folding with any upcoming breakthrough.

On 4) I fully agree. I do also not see any alignment organization adding this much value. Not to throw any shade, but if we compared this to Web3 investment in 2022, this prediction from the report does not seem too far off. In the Web3 case heaps of money were poured onto crypto founders that did not know the difference between a mutual and an exchange-traded fund, but were certain that their immutable database would "obviously" disrupt legacy banking by the next week.

On 5) I can't argue with the low interest in artists' well-being in the broader political dialogue. I would think that a lot of the low-tier creative work will change significantly in the next years. Imho, this does not just include fine arts, but also things like copywriting, product photography, and design.

6. Agreed. OpenAI is not open source. Thank you for pointing out the case of Microsoft. Seems like their prediction already came true. :)

7. Hell yeah! That would be so cool!

​

Thanks again for taking the time and contributing your insights!

Also I think vulkan compute shaders does not support more than 3d or 4d tensor, not sure

While reviewing works concerning automatic fake news detection, I discovered that some papers tend to divide the topic into two approaches: data-mining-oriented and NLP-oriented (both using ML).

I'm not sure what's the difference, since NLP too can use hand-crafted features obtained by data mining. Can someone explain me how these approaches differ?

Thanks for the suggestion, this is a good reference. I've actually taken a look at it already and noticed that it doesn't delve much into the early developments (there's barely any mention of McCulloch & Pitts's model). Do you know of any other references that go into more detail on each paper, like in "Evolution of Artificial Neural Networks" by Averkin and Yarushev?

Why would it be used? It doesn't begin to compare to CUDA and cuDNN. Nothing really does. And Vulkan specifically is made for graphics pipelines, not for general purpose compute. To be cross compatible, it usually sends compute to be done on the CPU.

It's not that there is a consipiracy to use proprietary nvidia software - there just isn't anything better than it.

[removed]

What are the best resources to get into AI as someone who is already a professional software engineer? A lot of the courses and tutorials seem targeted at complete beginners.