>the bias of DL towards low-dimensional smooth manifolds

What is this? Got all the rest but that

[removed]

[removed]

What is "bounded interval" here?

It's that entirely accurate though? There's all kinds of explicit dimensionally reduction methods. They can be combined with traditional ml models pretty easily for supervised learning. As I understand, the unique thing DL gives us just a massive embedding that can encode/"represent" something like language or vision.

This is a great point. Thank you. So do you mean that DL work for language models only when they get a large amount of data?

This one is pretty good: https://arxiv.org/abs/2207.08815

"for example on tabular data where discontinuities are common, DL performs worse than alternatives, even if with more data it would eventually approximate a discontinuity." True. Is there references on this issue?

There are two aspects, scalability and inductive bias. DL is scalable because compositions of differentiable functions make backpropagation fast, and those functions being mostly matrix multiplications make GPU acceleration effective. Combine this with stochastic gradients, and you can train on very large datasets very quickly.
Inductive biases make DL effective in practice, not just in theory. While the universal approximation theorem guarantees that an architecture and weight-setting exist that approximate a given function, the bias of DL towards low-dimensional smooth manifolds reflects many real-world datasets, meaning that SGD will easily find a local optimum with these properties (and when it doesn't, for example on tabular data where discontinuities are common, DL performs worse than alternatives, even if with more data it would eventually approximate a discontinuity).

?

Inventing them

"Artificial neural networks are often (demeneangly) called "glorified regressions". The main difference between ANNs and multiple / multivariate linear regression is of course, that the ANN models nonlinear relationships."

https://stats.stackexchange.com/questions/344658/what-is-the-essential-difference-between-a-neural-network-and-nonlinear-regressi

Before why, ask if. GBDTs are very widely used.

You seem confused as to what you yourself are saying.

It's not from a paper, but it's pretty uncontroversial I think - though people like to forget about the "bounded interval" part, or at least what it implies about extrapolation.

Will make a good story when you accept your Nobel prize. Well done.

Residual refers to the fact that the NN/bottleneck learns the residual left over after accounting for the entire input. Anyone calling the skip connections “residual connections” should stop though lol

>Whether the algorithms for training them do that as well as the ones for deep NNs in practice is a separate issue.

For supervised learning the big problem with decision trees (RFs, GBTs etc) seems to be representation learning

I feel like recently ML boosters come this Reddit, make large claims, and then use the ensuing discussion, time, and energy from others to correct their click content at our expense

I have been an applied data scientist for 10 years. I have built over 100k models using python, databricks and DataRobot. I have never seen a DL model out compete all the other algorithms. Granted I am largely working with structured business data, but nonetheless DL isn't really competitive.

[deleted]

I like your wording, did you come up with that definition yourself or is it from a paper?

[deleted]

The statement stand-alone that you have there is very vague. Can I assume is taking about NLP or CV projects ?

On tabular data even with non linear relationship normal boosting, ensemble algorithms can scale and be top of the game.

Do you teach online?