Didn't this get posted last week?

I am very impressed by the underlying GPT3.5 LLM and the capabilities that alignment via RLHF has unlocked in LLM, but I don't believe any serious NLP researchers or practitioners think that NLP is solved.

There are still tonnes of challenges and limitations that needs to be solved before this tech is ready. E.g The very convincing hallucinations, failure on simple math problems, and second order reasoning tasks amongst others. And many other areas that remains unresolved in NLP as well.

Having been in the NLP field for close to 10 years and having experienced several other developments and paradigm shifts in the past (RNN/LSTM, Attention, Transformer Models, LLMs with emergent capabilities) , I am more optimistic than fearful of this development's impact on our job.

Each of these past developments made obsolete certain expertise, but also expanded the problem space that NLP can tackle. The net effect however has been consistently positive with the amount of money and demand for NLP expertise increasing.

You are a lousy researher then. The trend of using incredibly large models was there a long time ago, so individual researchers couldn't produce SOTA NLP models for years already. And Chat GPT isn't even a great model compared to something like Chinchilla - you should know that, actually

Neuro-symbolic models

Can you link to an example online or from a book? I need to understand this!!!

That makes sense. Convolutional neural networks were just deep learning with convolution.

They have a sequence of models ranging from 6B params up to 175B largest, so you can work on smaller variants if you don’t have gpus. There’s def some papers working on inference efficiency and benchmarking their failure modes if you look around.

Ok I’ll admit to only having used neural models not trained them. AWS Polly is incredibly monotoned last I used it.

Megawatt sounds right for training. But kilowatts for inference. Take a look at tim dettmer’s work (he’s at UW) on int8 to see some of this kind of efficiency work. There’s definitely significant work happening in the open.

There are tools to work around this limitation such as LangChain with its support of summarization of previous context https://github.com/hwchase17/langchain

You can represent any neural network as a decision tree, and I believe you can represent any decision tree as a series of if statements...

But the interesting bit about neural networks is the training process, automatically creating that decision tree.

For small tabular datasets I found gradient boosting methods (xgboost, lightgbm, catboost) to outperform typical deep learning architectures.

Fusion of LLM and vision models is something I’m noticing more work on. Also, embodied feedback with human in the loop, especially towards robotics applications. The vision field def seems to be co-opting language models and there is research on making inference with them faster (recurrent-transformers) and bringing back recurrence into the transformer which is interesting since transformers succeeded them naturally once the power of attention came to light.

Also a lot of work to be done on using them for mission critical applications (healthcare) as well as “robustifying” them (transformers using raw byte sequences showing much more robustness to noise.)

So I guess a lot of the native NLP tasks that LLM were made for are being used more for non-NLP tasks, especially now in reinforcement learning.

Posted there too:)

There is a gpt3 subreddit. You should probably post it there too

brilliant

Sounds like a god of the gaps argument. Machines may not be better at humans for any given task today, but it seems to be moving pretty quickly in that direction. With that said, it seems humans become a lot smarter with the aid of machines and vice verse.

Domain specific LLM's need not to be huge like these LLM's like chatgpt. They have world knowledge. In most of the settings, we don't need that.

I'm convinced. Brilliantly explained. Constructing the problem with respect to a bound makes a ton of sense.

Now I suppose we just code it up and throw it at CPLEX.

Do you have a link to the paper? I’ve been working in Stan pretty recently on a similar problem.

Or they could just be quite recent topics? NTK, for example, seems to have been introduced in 2018. If you're not actively reading ML research papers, you'll probably have a hard time getting exposed to those topics.

Your data doesn't need to be linear to use a linear regression. The linearity is in the parameters. Ex: Y = Bo + B1X + B2X**2 Is a linear regression You can do non linear transformations to the X

Of course SVM’s work better over simple models, they are pretty heavyweight solutions involving the solution of a convex function.

Thanks! I was wondering when neural networks were going to go beyond the current trial and error alchemy, and it looks like this is a big step forward for the mathematical foundations.

Excerpt from pages 8 and 9:

>Unfortunately, the formal infinite-width limit, n -> ∞, leads to a poor model of deep neural networks: not only is infinite width an unphysical property for a network to possess, but the resulting trained distribution also leads to a mismatch between theoretical description and practical observation for networks of more than one layer. In particular, it’s empirically known that the distribution over such trained networks does depend on the properties of the learning algorithm used to train them. Additionally, we will show in detail that such infinite-width networks cannot learn representations of their inputs: for any input x, its transformations in the hidden layers will remain unchanged from initialization, leading to random representations and thus severely restricting the class of functions that such networks are capable of learning. Since nontrivial representation learning is an empirically demonstrated essential property of multilayer networks, this really underscores the breakdown of the correspondence between theory and reality in this strict infinite-width limit.
>
>From the theoretical perspective, the problem with this limit is the washing out
of the fine details at each neuron due to the consideration of an infinite number of incoming signals. In particular, such an infinite accumulation completely eliminates the subtle correlations between neurons that get amplified over the course of training for representation learning.