Ok, I'll bite. What's uptrain?

We currently support LLMs, Vision models, Recommendation systems, etc., and are working to integrate it seamlessly with any of the major MLOps frameworks or cloud providers.

This is going to do amazing things for GIF reactions when it's fast and cheap.

> Whenever I see music generation models, I immediately go to the "classical" examples (or as close to classical as are provided). The reason for this is that while some genres such as techno, drum 'n' bass, 8-bit, and hip hop are "simple" (from a music theory perspective), and other genres such as ambient, relaxing jazz, swing, and dream pop are vague enough that the model can get by just from spitting out the right general timbre, generating classical music requires understanding of structure, style, and form.

> Frankly, I'm not particularly impressed. […]

> […]

> This is not to say that the model is not impressive in other ways. Its ability to mimic the styles of different genres is quite good (although the "swing" example in the Long Generation section loses focus halfway through), and the style transfer elements are quite interesting as well. However, music generation models have a long way to go when it comes to idiomatic understanding of the structural elements of music.

It feels similar to earlier LLMs: It is, by today's standards, extremely easy to generate a model that generates vaguely correct looking text in the sense that the words have reasonable length and the characters have a reasonable distribution. Only at later stages do the models manage to output vaguely correct words with minor spelling mistakes. At that point the grammar is still complete nonsense, as well as the semantics. Only very recently did LLMs manage to stay coherent over larger blocks of text.

Relatedly, diffusor-based image generation has a similar thing going on: Textures are frighteningly great. Image composition and logic not so much.

I think for music generating models we are at the stage where they get the texture and syllables right, that is the overall sound, but not at the stage where image composition and grammer is quite there, that is chord progression, melody, themes and overall composition.

would be interested as well do you have your progress on GitHub or something?

> I’m not sure a 2.8 trillion token dataset actually exists

DeepMind's Massive Text is assumed to be 10TB large, the largest publically available dataset is The Pile and weighs in at about 820GB.

A 2.8 trillion token dataset would need to be more than 20TB large, which could be possible by including more of Common Crawl - weighing in at 380TiB - or non-English resources. I have a suspicion that training LLMs on more languages, especially outside of the Indo-European family, will improve performance within the Indo-European family.

Wait how do you cross the channel on foot? Did it freeze over at some point?

Where can I do this right now?

> I think most people nowadays use RNN-T.

Isn't slightly finetuning Whisper the go to?

Don’t bother. mps support is terrible. Tensorflow GPU support is better in comparison.

However, the MBA is good for fast and efficient cpu prototyping which you should ship off to a Linux running workstation or cloud with discrete Nvidia GPUs.

Yeah, there is some stuff published out there. It's related to pruning (A link to a ton of papers on it); the lottery ticket method solves this one well, because you're re-training from scratch, just with "lucky" selection of the initialized weights. Results-wise, I never got anything to improve because of the distributional changes caused by trying to re-randomize a subset in the middle of training. Still saw the same level of performance as without re-randomizing, but that basically just showed that the way that I was re-randomizing wasn't helping or hurting b/c those neurons weren't important...

Then you shouldn't be training on a laptop, and should train on a remote server, in which case your laptop doesn't matter.

Perhaps answer with a better subreddit, but autodelete without any kind of a message is rude and not helpful on the long run.

Here is what I do. For laptops I always want to have a good battery life. I don't need my laptop to be particularly powerful. I have a normal business laptop running Linux. I personally like Lenovo laptops because they play nice with Linux and I just love the keyboard.

I work remotely. We have our own compute server. But you could just as easily work on an AWS instance.

To me it makes more sense to use a laptop for what it is good at (Mobility) and a stationary server for what it is good at (Power). In the past I've tried to make compromises and it always is a lousy compromise.

Can we do a bot to autodelete these kind of posts?

>Text-To-4D Dynamic Scene Generation
>
>Abstract
>
>We present MAV3D (Make-A-Video3D), a method for generating three-dimensional dynamic scenes from text descriptions. Our approach uses a 4D dynamic Neural Radiance Field (NeRF), which is optimized for scene appearance, density, and motion consistency by querying a Text-to-Video (T2V) diffusion-based model. The dynamic video output generated from the provided text can be viewed from any camera location and angle, and can be composited into any 3D environment. MAV3D does not require any 3D or 4D data and the T2V model is trained only on Text-Image pairs and unlabeled videos. We demonstrate the effectiveness of our approach using comprehensive quantitative and qualitative experiments and show an improvement over previously established internal baselines. To the best of our knowledge, our method is the first to generate 3D dynamic scenes given a text description. github.io

Thanks for the exhaustive answer.

Google collab

I don’t even think it’s as much about research but data collection and labeling

Don't train on a laptop. Being portable and using lots of computational power are essentially opposite goals. You're not going to be able to train anything more than toys on battery, at which case if you're going to be tethered to a power cord you might as well be tethered to a desktop. You're also going to be limited in performance, due to a combination of efficiency-focussed laptop hardware as well as the thermal constraints imposed by a laptop form factor. You're far better off getting a highly portable, long battery life, but low power machine, and using cloud resources (even free ones like Colab or Paperspace) to do the heavier lifting.

If you absolutely must use a laptop because you're living out of your car or something and have nowhere to set up a desktop, then the rest depends on what you're doing:

If you're doing "deep learning" (anything involving neural networks more than a layer or two deep) you'll need a discrete GPU from NVIDIA specifically. AMD and AS support exist but are far from mature or feature complete in most frameworks. CPU need only be powerful enough to keep the GPU fed, a modern i5 or equivalent AMD will do the job, although you may find that specs with a suitable GPU aren't offered with less than an i7 / R7.

If you're not doing deep learning, you probably don't need a GPU. In that case, stick with integrated graphics and look for a higher end i7 or i9 (or equivalent AMD).

As a rule, you'll get better support on Linux than Windows or MacOS. You can skirt this in Windows via the WSL.

Finally, this post reads like you haven't even started doing whatever it is you're trying to do. I'm guessing you're a beginner just starting out, and I'd strongly advise anyone at that stage to delay purchasing hardware as long as humanly possible. Everything I noted is a generalisation, none of it specific to what you're doing because you haven't been (and likely can't be) specific to what you're doing. If you get started first using free or cheap cloud resources, you'll get a much better idea of which things you need, and which things you don't.

Who’s annotating music with these weird, non-intuitive text descriptions for training?

Throughput matters for me. How is it affected?

It looks like you don't care about throughput so it doesn't matter.

Most machine learning technologies for music are like telling an music theorist to design a circuit board. the results are hilarious. you need a synth architect.

Hi there!

I have trained ImageNet several times myself, using both local and cloud resources.

I would recommend starting with a tutorial on how to get it running locally - there are many out there. As for the cloud resources, I found Google Cloud to be the best option in terms of cost/performance. In terms of expenses, the cost of training on ImageNet can be quite high, depending on the resources you use. As for the model, I would recommend running the model at least three times, so that you can get an accurate estimate of the performance. As for early stopping, I would recommend using the validation set - this will give you a more accurate representation of the model's performance.

Hope this helps!