One dark outlook on this is the compute cost reduces very slowly (or does not reduce at all), the large models become the ones that only the rich can run. And using the capital that they earn using the large models, they reinvest and further accelerate the model development to even larger models and the models become inaccessible to most people.

"In 70 years" feels extremely cautious. I would say it's in the next few years for regular ML, perhaps 20 years for robotics

Yeah I have a pretty dystopian outlook on the future because of this.

They really do. Just take any (somewhat reputable) book on ML.

Bitter lesson 3.0: The entire idea of fine-tuning on a large pre-trained model goes out of the window when you consider that the creators of the foundation model can afford to fine-tune it even more than you because fine-tuning is extremely cheap for them and they have way more compute. Instead of providing API access to intermediaries, they can simply sell services to the customer directly.

With enough scale we get crude compositionality, yes. That trend will probably continue, but I don't think it'll take us to the moon.

Did this ever get any traction?

Would you recommend any of those or is there one in particular that introduces well such techniques?

Edit : I'm asking this because I'm not very familiar with what tree search means in the first place

It's appropriate for Google Colab.

125m parameters already sounds like it wouldn't be fun on most GPUs that people actually own, I'm imagining a student with like a 1650 laptop waiting days with their laptop sounding like an F-15 with full afterburner

To the general public, absolutely, yes

Regarding that Winoground paper: Isn't compositionally what DALLE-2, Image, Parti, etc are famous for? Like the avocado chair, or some very specific images like "a raccoon in a spacesuit playing poker". SOTA vision language model are the only models that actually show convincing compositionally, or am I wrong?

Username checks out hehe. Jk, don’t get why people are downvoting you for being a lil wrong!

This sort of reads like an advertisement….

They really do not without further context.

Yes, I guess feeding more data to larger models will be better in general.
But what should we (especially who do not have access to large computing resources) do while waiting for computation to be cheaper? Maybe balancing the amount of inductive bias and the improvement in performance to bring the predicted improvements a bit earlier?

Artificial intelligence

>Artificial intelligence (AI) is intelligence—perceiving, synthesizing, and inferring information—demonstrated by machines, as opposed to intelligence displayed by non-human animals and humans. Example tasks in which this is done include speech recognition, computer vision, translation between (natural) languages, as well as other mappings of inputs. The Oxford English Dictionary of Oxford University Press defines artificial intelligence as: the theory and development of computer systems able to perform tasks that normally require human intelligence, such as visual perception, speech recognition, decision-making, and translation between languages.

^([ )^(F.A.Q)^( | )^(Opt Out)^( | )^(Opt Out Of Subreddit)^( | )^(GitHub)^( ] Downvote to remove | v1.5)

I know that in France a lot of university labs still have teams working on old (https://en.wikipedia.org/wiki/Artificial_intelligenc) techniques.

I think AI is the general term that machine learning falls to. For example minimax is AI but it has nothing to do with machine learning.

Not sure on what AI is from a technical standpoint, I always thought it was a concept.

No it's not strictly needed, though I haven't seen a course that teaches ML starting from the application and working backwards to the fundamentals. In teaching that's sometimes called "top down" as opposed to starting from fundamentals.

If you're taking courses, you may need to pick up a bit of math along the way. If you're self-taught, you might try starting with tutorials of ML libraries like scikit-learn and keeping a journal of any terms you need to look up later.

Interesting! Will have look at it! Thanks

Thanks!

So, there’s a trick where you write a differentiable program and swap out expensive bits with a neural network, which I think is probably related to this. Looking at the article, I think you would very quickly run into some hard problems in differential geometry if you tried to make this formal.

No, 60 problems are not enough, probably not even for fine tuning. I would also strongly advise against starting from scratch.

The best approach in this case would be to fine-tune a pre-trained LLM which was trained on both natural language and code, something like GPT-Neo with 125M parameters. I'm mentioning the small version because the you'll have trouble fitting in memory larger models with billions of parameters!

Personally this is what I used for my Bachelor's where I made a tool to automatically generate input code from competitive programming statements.