finBERT maybe?

This was a fascinating read, thanks a lot!

In my experience NLP models are released as public science when trained on datasets scraped from the web.

Things like "models that solve this problem in finance" or "datasets of annotated football matches" or "medical records of millions of people" are not likely to follow the publication patterns of open science.

If you have a model like the one you asked for you likely have a way to profit from it and you are unlikely to publish it.

It’s a retrieve/rerank model

More accurate, as in it can more accurately capture the semantic meaning of jargon that is typical in financial documents.

> a model that can generate more accurate embeddings in general

What do you mean by this? Accuracy is an evaluation metric that doesn't really mean anything "in general" but only with respect to a specific prediction being made. I think this is a slightly ill-posed question

What is your ML model supposed to do?

Another interesting idea might be to start training with smaller context len (and bigger batch size - together with random sampling)

If you think about it, people also learn the noun-verb pairs first and then go to sentences and then to longer paragraphs/articles, etc. And it's also good if we have a lot of variance at this early stages.

So it makes some sense, BERT MLM is also very similar to what people do when learning languages :)

but this is likely going to take forever because of LR decay, right?

For noobs like me: GBDT=gradient boost decision tree

What you accomplished is wonderful enough. I agree wholeheartedly with your 1:n mapping

You will not be able to train huge models like ViT or BERT from scratch, but for fine tuning a 4090 is waaaaaaay enough. Even with my 2080Ti I achieve really fine results with a decent amount of time.

Your best friend is Huggingface, because transformer are life with a nice GPU

With your new config here would be my first try:

• ViT
• BERT
• Donut because it's hell of a thing
• Everything you can find on huggingface and makes you curious

>Everybody lies.

In medicine, patients often say X but mean Y.

It's not really lying.

As a practitioner, it's your job to drag this info out of them.

> Do you think data augmentation should also be disabled in that test?

Yes. I've never actually experimented w/ stuff like image augmentation. But in most examples I looked up, augmentation is a training-only computation which may make training loss look higher than it actually is. In general the rule is just this: to unbiasedly estimate training loss, apply the exact same code you're using to estimate validation loss to training data.

Same hahaha, would've linked it otherwise xD

Ive also had this idea and think there is a decent market for it. DM me

Doesn't that mean that if you include inference costs, and the model will be used extensively, you may actually get much better bang for your bucks by training much more than chinchilla-optimal?

[removed]

>Notice that "significantly lower" can't actually be defined.

True. I guess I would say that over-fitting is a spectrum, and that there's generally some amount of over-fitting happening (unless your training set happens to be significantly more challenging than your test set). So the bigger the gap between train and test, the more over-fitting.

>It's tempting to think "test error is 3x train error, we're overfitting". This may or may not be right; there absolutely could be a (more complex) model B with, e.g., training error rate 0.05, test error rate 0.27.

Maybe it's semantics, but in my view, I would say model B is indeed overfitting "more" than model A. But I don't think more overfitting guarantees worse test results, it just increases the likelihood of worse test results due to increased variance. I may still choose to deploy model B, but I would view it as a highly overfitting model that happened to perform well.

Appreciate the response. I also liked your CrossValidated post. I've wondered about that issue myself. Do you think data augmentation should also be disabled in that test?

My dentist uses very little new tech, it would seem to me if you could build a robot that could inspect the mouth visually, perhaps perform parts of a cleaning or other repair much faster and more comfortable than todays procedures. This robot could be deployed remotely or even home. Toothache? Buy this robot at Walgreens and a AI will tell you what the issue is(get a human opinion for only $99 more!). Collecting lots of images would let you at some point create a ml model of defects that could then be identified by the machine. While the science of ml is complicated, collection and creating models is much less complicated. The tech is great but this has the potential to solve the real problems and bring automation.

There are so many ways to tackle this, I guess I would start with a use case. Since I know absolutely nothing about dentistry, let me ask the question - what is the absolute dead simple work that seems like it could be automated? Knowing nothing, I suspect that things like teeth cleaning could be performed by a machine for most cases. You can imagine in your home having some kind of thing you stick your face in, open your mouth and a large amount of data is collected visually about your teeth. If there is a way to see plaque and calculus, ML would highlight that and say, “remove?” And you would say yes, then it would gently pick at your teeth, visually inspecting the whole way. You are probably pressing some kind of trigger the whole time, and seeing the machine work the whole time. This is so you have some sense of control and in an emergency you could just release the trigger. Of course this would only work for the standard cases, anything else would need to see a human specialist. Eventually in 40 years no one would trust the human - all use cases covered.

The thing about medical expertise is that so little if it is conveyed through the literature, because of the limitations of text and even images/recordings. Things like symptoms and disease progression have very particular looks and feel that can’t be easily conveyed. Also, the full explanation of what normal and healthy is is also poorly expressed. In your profession, there is probably a lot of nuance to exactly the right amount of pressure to exert on teeth in different circumstances.

If I were you I would catalogue where the literature falls short, and where/how you did gain your expertise, without trying to explain exactly what to do. In other words, tell the developers where they need to focus their efforts.

I agree with your assessment that a lot of what medical professionals do can and arguably should be supplanted by technological solutions with deep and subtle expertise built in. I think besides the software we are waiting on robotics better capabilities, particularly things like sensory feedback as detailed as human skin.

i think you've got the right idea about getting involved. i would say that your expertise would be wasted if you try to do the ML stuff yourself. you should definitely leverage your biggest asset as a domain expert and find some ml researchers who can be the tech experts. others have pointed out how reaching out to universities can be a good way to start. maybe you can look into some medtech startups that would like to have a chat with your about this stuff

in terms of getting to know the tech itself, you should have some layman's understanding of how the ml works. for that i'd recommend finding some good ml content geared towards enthusiasts rather than professionals. stuff like computerphile's videos on models and concepts like GPT, diffusion models, encoder decoder networks, and deep learning in general. you may also want to find videos and demos in to things like Word2Vec, google's BERT, and zero shot learning in NLP in general. also look at the e-palm model google just released a demo of. that some of task tranlation is probably your biggest lead

> New robot dentist, now with only a 1 in 1000000 chance to mistakenly drill into your bones!

Everybody lies.

The AI tools we have now seem too trusting and take users input at literal value. I wonder what research would suggest about making AIs more skeptical towards humans that need their help.

Also, trusting an AI that doesn't trust its users sounds sketchy.