Depends on who you’re talking to. Most people don’t know the difference. If you’re talking with those that do - you’d better acknowledge the difference or they think you’re dumb.

What is the cheapest way to pre-train a generic LLM with my own data?

FYI, using output from first stage model is called model stacking

Are you trying to model time series classification (many-to-one)? I don't know if making it a 2 stage model is appropriate i.e. using 0 and 1 as an intermediate representation

The hierarchical classification error will propagate through multiple stage if using raw prediction from previous stage alone. For example, if first stage model is 0.9 in accuracy, and second stage is also 0.9. The maximal accuracy two stage model will be 0.9*0.9 = 0.81 (performance degrade)

Are you trying to do research, or solve a problem? Building expert systems out of neural networks is still a new, experimental idea. If you just want to get the job done you may want to pick more proven methods.

Kaggle would like to know your location

Crazy that this has even a single upvote and proof that this subreddit is no longer the community for academic discourse it once was. Do you know who Paul Smolensky is? He practically invented the term "neuro-symbolic" and was virtually the only researcher seriously working on it in the 20 years leading up to the deep learning revolution. Harmonic Grammar, Optimality Theory, Tensor Product Representations. Please pick up perhaps any article on connectionism before 2010.

No, this is not a new term for for neuro-symbolic computing (which is now just a buzzword applicable to half of the field), it's a specific theoretical take on how compositional structure could be captured by vectorial representations.

We're exploring some data augmentation approaches right now (see my response to u/giga-chad99) but how would you propose generating those problems with compositional structure?

>gradient of probability

Nicely put.

Thank you, 'perturbation' is what I was looking for.

Text2image generation models do anecdotally appear to be better than image-text matching models at compositional tasks, but if you look closely at some generated images, you'll notice compositional failures. They often apply properties to entities on which the text did not describe them as applied to, or misunderstand the described relation between entities as a more common relation between those entities.

Try a prompt like "man with dog ears running in the park", and it'll generate images of a man with a dog (sometimes with amplified ears) running in the park. Why? Because models don't have the underlying ability to create compositional representations, they instead simply approximate their training data distribution.

Examples like "a raccoon in a spacesuit playing poker" often do well because spacesuits are only ever worn and poker is only ever played (i.e. relations that are common in the training distribution). Try a prompt like "a raccoon sitting on a poker chip and holding a spacesuit" and you'll see pretty drastic failures.

All this being said, generative models *still* appear better than discriminative models for vision-language compositionality tasks, and our current work is exploring approaches to impart this ability onto discriminative models to solve tasks like Winoground.

It's meaningful right now because there's a threshold where LLMs become awesome, but getting there requires expensive specialized GPUs.

I'm hoping in a few years consumer GPUs will have 80GB of VRAM or whatever and we'll be able to run them locally. While datacenters will still have more compute, it won't matter as much since there's a limit where larger models would require more training data than exists.

[removed]

Hot take: foundation models is pure branding, so if they say it's foundation models it will be foundation models that we're all using.

Or you could use ml to predict the error of the rule based system or simulation.

That’s not bitter lesson, that’s just Capitalism.

You don't have a dedicated tpu cluster in your lab? Pffftt

Im looking into something similar, essentially combining data driven ML with a knowledge base, but in the context of explainable AI and predictive maintenance.

I have stumbled across something called "Logic Tensor Networks" (search for the paper) which might help in your situation. I need to look into it more, but it combines ML + knowledge bases + fuzzy logic.

Hope you find a solution!

I‘ve used it in production as well, but thought that there is quite few research published about it lately. So I kind of assumed it’s similar in practice.

In real world; stacking is used. It's usually behind the scenes -- as you have seen in this AutoML application.

Which is and has been the Status Quo for the entire history of computing, I don't see how that's a new development?

I’ll look into it, thanks a lot! The main topic/question is still open for me tho.

For now, yeah. I'm the guy building their fancy hodgepodge theoretical linear algebra functions into efficient PyTorch backend code so it can actually do something. And the CI/CD pipelines, the serving systems and all of that. You could even say I'm contributing to the demise of those 10 engineers. Especially all the Javascript bootcamp CRUD engineers flooding NPM with god-knows-what these days.

Gotta back the winning side, not fight them. If foundation models get replaced by something else, I'll go build software for those guys and gals too.

Hi! There is relatively new paradigm, 'Physics informed Machine Learning"

Here is a nice review of the different techniques:

https://www.nature.com/articles/s42254-021-00314-5

The most popular approach is based on physics regularization on neural networks.

PS: link for the paper is offered by the autor:

https://www.researchgate.net/publication/351814752_Physics-informed_machine_learning

I guess so, there's nothing bitter in this so-called "bitter lesson 2.0"

does anyone have any experience using Youtube videos for text to speech/speech to text data?

I can get the subtitle data for videos, although they don't make it easy. While the subtitles themselves are accurate I also need accurate timestamps and the timestamps from Youtube (which seem to be designed for close captioning rather than accuracy) seem to be just inaccurate enough to make them not useful. Am I just doing things wrong and there is a way you guys use to get an accurate timed Youtube transcripts?