Recent comments in /f/MachineLearning

I-am_Sleepy t1_j7ybb41 wrote

Reply to comment by Ulfgardleo in [D] Critique of statistics research from machine learning perspectives (and vice versa)? by fromnighttilldawn

I don’t think ML researcher didn’t care about model calibration or tail risks. Just it often doesn’t came up in experimental settings

It also depends on the objective. If your goal is regression or classification, then tail risk and model calibration might be necessary as supporting metrics

But for more abstract use case such as generative modeling, it is debatable if tail risk and model calibration actually matter. For example GANs model can experience mode collapse such that the generated data isn’t as diverse as the original data distribution. But it doesn’t mean the model is totally garbage either

Also I don’t think statistics and ML is totally different, because most of statistical fundamentals is also ML fundamentals. And such many of ML metrics is directly derive from fundamental statistics and / or related fields

13

avocadoughnut t1_j7yaq8w wrote

Reply to comment by Sm0oth_kriminal in [D] Using LLMs as decision engines by These-Assignment-936

I'm considering a higher level idea. There's no way that transformers are the end-all-be-all model architecture. By identifying the mechanisms that large models are learning, I'm hoping a better architecture can be found that reduces the total number of multiplications and samples needed for training. It's like feature engineering.

8

WikiSummarizerBot t1_j7y9nn5 wrote

Reply to comment by [deleted] in [D] Critique of statistics research from machine learning perspectives (and vice versa)? by fromnighttilldawn

All models are wrong

>All models are wrong is a common aphorism in statistics; it is often expanded as "All models are wrong, but some are useful". The aphorism acknowledges that statistical models always fall short of the complexities of reality but can still be useful nonetheless. The aphorism originally referred just to statistical models, but it is now sometimes used for scientific models in general. The aphorism is generally attributed to the statistician George Box.

^([ )^(F.A.Q)^( | )^(Opt Out)^( | )^(Opt Out Of Subreddit)^( | )^(GitHub)^( ] Downvote to remove | v1.5)

1

Ulfgardleo t1_j7y8hdg wrote

Reply to [D] Critique of statistics research from machine learning perspectives (and vice versa)? by fromnighttilldawn

The difference between stats and ml is as large as between math and applied math. They aim to answer vastly different questions. In ml you don't care about identifiability because you don't care whether there is a gene among 2 millions that cause a specific type of cancer. This is not what ml is about. In ML you also very rarely care about tail risk (you should) and almost nothing about calibration (you really should). And identifiability is out of the window as soon as you use neural networks and that prevents you from interpreting your models.

22

sunbunnyprime t1_j7y86w7 wrote

Reply to [D] Critique of statistics research from machine learning perspectives (and vice versa)? by fromnighttilldawn

Good question.

An ML Researcher is typically trying to find models which are more powerful in terms of output behavior - whether that be predictive power, generative ability etc.

A Statistical Researcher is typically trying to understand the dataset, the underlying generative distribution, and really dig into what the model’s innards are saying about the data and what you can conclude from it. They’re more likely to want to extract insight about the data itself.

Statisticians tend to be more rigorous about data and more well grounded in my experience, while ML Scientists tend to want to push boundaries and be the person who’s read the latest ML journal piece.

There’s so much you can say and know about something as simple as linear regression. There’s really a lot of fascinating math in there that goes so much deeper than you might expect.

If you’re interested in just using models to predict, there’s not that much of interest in a linear model. If you really want to know what meaning you can extract from what’t going on inside - exactly why it learns the coefficients it does, what the learning dynamics are, what the results mean etc - then you might end up writing 10 papers on Lasso.

Both sides are valid. Most ML scientists suck at their jobs I must say though.

31

Sm0oth_kriminal t1_j7y6wv6 wrote

Reply to comment by avocadoughnut in [D] Using LLMs as decision engines by These-Assignment-936

This is probably only the case in which there’s a very low “compression ratio” of model parameters to learned entropy.

Basically, if the model has “too many” parameters it can be distilled but we’ve found that, empirically, until that point is hit, transformers scale extremely well and are generally better than any other known architecture.

Another topic is sparsificafion, which takes a trained model and tries to cut out some percentage of weights that have a minimal output effect, then fine tuning that model. You can check out Neural Magic online and associated works… they can run models on CPUs that normally require GPUs

4

Jemimas_witness t1_j7y68en wrote

Reply to comment by [deleted] in [D] Critique of statistics research from machine learning perspectives (and vice versa)? by fromnighttilldawn

This is only correct for certain problems, like everything it has best use cases. When you only have a hammer everything looks like a nail.

In medicine the backbone of clinical trial results that change the field relies often on 2000-3000 patients (datapoints) and often groundbreaking achievements in medical practice are made by simple statistics and simple methods. Go to the New England journal of medicine and pick any trial and the weight of their conclusions are based off of survival functions, hazard ratios, and chi squared statistics. Then go look at the funding section - these projects are funded by millions. The only disciplines in medicine with ML datapoints are epidemiology and claims level data which strays way into econometrics.

I myself study rare diseases as well as AI/ML applications in medicine and for some projects I’d be stoked to get 80 patients because there just simply aren’t that many around.

2

currentscurrents t1_j7y4073 wrote

Reply to comment by [deleted] in [D] Critique of statistics research from machine learning perspectives (and vice versa)? by fromnighttilldawn

>Right now basically all progress is with large models,

You mean all progress... in machine learning. A lot of scientific fields necessarily must make do with a smaller number of data points.

You can't test a new drug on a million people, especially in early phase trials. Even outside of medicine, you may have very few samples if you're studying a rare phenomena.

Statistics gives you tools to make limited conclusions from small samples, and also measure how meaningful those conclusions actually are.

6

Borrowedshorts t1_j7xz19r wrote

Reply to [D] Using LLMs as decision engines by These-Assignment-936

Yes, and this example actually isn't all that impressive. Google demonstrated a LLM can significantly improve decision making for a real world robot, giving it a type of 'common sense'. Check out Palm-SayCan which is a collaboration of two models that can perform real world robotic tasks through the assistance of a language model.

18