I have tried many cases. For example. It gives correct proof of one of my technical lemmas in my own paper which make me quite amazine. It is a simple lemma, but it is very specific to my question. I also tried to search with google but do not find the answer.

Based on the techniques ChatGPT uses we cannot formally prove that it can generalize without infinite width. Even our training process amounts to mostly teaching the model to compress knowledge. ChatGPT made some strides by partially introducing something similar to reinforcement learning, but reinforcement learning itself is not enough to extrapolate or come up with new concepts.

All the big names in AI claim that stochastic gradient descent techniques and our current direction are fascinating, but ultimately a dead end. Certainly the area has been stale for several years and has degenerated into a dick measuring contest, only instead of dicks you measure parameters, TPUs and metrics on benchmark datasets. Blame transformers which were in a sense us getting a taste of the forbidden fruit, but you know what followed after that.

Of course, out of this you do get some advances useful for the industry, but nothing really of note in the general picture. And it seems to me that lately all these big models that imitate knowledge really well are generating negative sentiment in the population, which may ruin AI.

Is this true? What type of inference would it be capable of?

But that's the current state, we know there will be a v.next to infinity, no? Would there be a state where it can train itself, similar to how Deepmind trains itself in games?

When I used it it kept saying it had no connection to the internet and was trained on a large amount of text and data. I tested in two languages.

At the time I used it the answers to most questions were structures in the same way: paraphrasing the question, weighing a few pros/cons or facts, summary. Almost every answer to a question that required a decision was inconclusive and ChatGPT usually said it was difficult to answer the question.

As an interface for human-machine-communication it was great. But the conversations were simple and lacked depth. It can write short stories and expand these stories. And it creates poems and jokes. I’d say you are lucky if it comes up with something that is above middle school level.

The next version will be far better I believe.

I work at one of the big techs doing research on this. Frankly LLMs will be the leading edge of the field for the next 2 years imo. Join one of the big techs and get access to tens of thousands of dollars of compute per week to train some LLMs. Or in academia, lots of work needs to be done to characterize inference-time capabilities, understand bias, failure modes, smaller scale experiments w/ architecture, etc.

Again, not in the field so don't laugh at me, but would there be opportunity / value to apply a Meta layer on top of ChatGPT? We know that it needs to be prompted certain ways, so would there be an opportunity to tune the prompting and also to evaluate the responses? Maybe you can apply your skills on this Meta layer?

[removed]

Not by a long shot.

ChatGPT in practice is a politically-biased conversational Google and Wikipedia summarizer with a bit of polite talk. And it is less broad than both of them.

It is truly fascinating how DEEP it can go, ex. translating arbitrary code in almost correct assembly, even recent one like M1, but that's that. It cannot reason fully, it cannot extrapolate, and most importantly, it has fairly old training data to compete with the speed of NLP research.

But it's nifty to chat with if none of your colleagues have the time.

I have tried and found it is a huge advance in this area. It not just googling. It has some inference capability.

I looked for it once years ago, but couldn't find any. I don't think it's needed anymore. Current TTS systems based on neural networks are really good at producing speech with the right intonation from just the text.

It's not just googling. I can summarize the information it has and write a good answer to the questions. It can even have some inference capability.

Tried to join the waitlist. Works now

This.

It basically is just a good google searcher, that can articulate results in a helpful way.

It may be useful to save time researching things... But it has had some laughable failure results as well.

I'm not in the field, but would be curious. Since you are in the field, why don't you try it out yourself and tell us. FWIW, majority of everyday problems can be solved by putting Googlable elements together properly.

How did you run into this error?

By default it's true that DL algorithms are truly unoptimized on this point because modelers usually don't really care about optimizing the number of parameters.

For example Resnet50 uses 23 million parameters, which is much more than efficient net B0 which uses 5 million parameters and have a better accuracy (and is harder to train). But when you try to further optimize algorithms which were already optimized on their number of parameters you quickly see these limits. You would need models that would be even more efficient than these DL models which are already optimized regarding their number of parameters.

A DL model could probably solve this handwriting problem with a very low number of parameters if you build it specifically with this goal in mind.

I agree of course, you can't compress more than some hard limit, even in lossy compression. I just think DL finds very poor compression schemes compared to what's possible (compare DL for that handwriting problem above to the solution constructed by human experts).

I have heard its essentially Googling with extra steps, are you certain its actually creating novel solutions to novel problems or is it just scraping together Googlable elements? Maybe I have subconsious bias as I develop for a living

>Imo there's no reason why we can't have much smaller models

It depends on how much smaller they would be. There are limits to how much you can compress information. If you need to represent 4 states, you can't use one binary value 0/1, you need two parameters 00/01/10/11.

A large image of the real world contains a lot of information / details which can be hard to process and compress. We can compress it of course, that's what current DL algorithms and compression softwares do, but they have limits otherwise they loose too much information.

Usual models are far from being perfectly optimized but when you try to optimize them too much you can quickly loose in accuracy. Under 1.000.000 parameters it's hard to have anything that could compete with more standard DL models on the tasks I've described... at least for now. Perhaps people will have great ideas but it would require to really push current limits.

Is there a summary of that criticism somewhere, I wouldn't want to read a full book

Imo there's no reason why we can't have much smaller models that do well on these tasks, but I admit it's just a hypothesis at this point. Specifically for images, an inverse graphics approach wouldn't require nearly as many parameters: http://sunw.csail.mit.edu/2015/papers/75_Kulkarni_SUNw.pdf

Well it doesn't really count because you can also "solve" these tasks with SVM / RandomForests, etc. MNIST, OCR and other tasks with very small images are not great benchmarks anymore to compare a random algorithm with a deep learning algorithm.

I was more thinking of getting 90% top 1 on ImageNet or generating 512x512 images from text or learning on billions of texts to answer questions. You either need tons of parameters to solve these or an unbelievable amount of compression. And even DL algorithms which do compression need a lot of parameters. You would need an even bigger way to compress an information, perhaps it's possible but it's yet to invent.

Ah ok, I didnt know that was an issue. Extensions are really important so you should definitely look into that

Yeah we don't currently have extensions implemented(not sure it's possible in a 3rd party web-app actually).

​

I've been using Copilot a ton so was curious if you were also using it