Gradient accumulation is tricky for contrastive methods that rely on having lots of negatives in a batch.

For minor polishing, I use quillbot.

Also, stay away from grammarly.

If you are willing to trade time for batch size you can try with gradient accumulation

Are you a native English speaker? Is your advisor a native speaker? If one of you are, just have that person check for English grammar, and have your advisor check for content, flow and argument.

[deleted]

Google supposedly has better models based on benchmarks, but few people outside of Google has used them (and those that have used it don't seem to be giving good reviews).

AnthropicAI's Claude model seems promising as a ChatGPT competitor.

https://blog.google/technology/ai/lamda/

It's supposed to be ahead of OpenAI's current GPT. Seems only logical to me since it was Google that invented and open-sourced the Transformer model and they likely have much more and much higher quality data than OpenAI for training.

Yum Lecum

[deleted]

[removed]

chatGPT is probably a very good model on the task it had to solve (to be a great conversational agent based on openAI data), but there are better models regarding the broad task of language understanding. You could adapt these models to be conversational agents, and they could probably beat chatGPT if they had access to the same dataset. But it would still be this specific task of being a great conversational agent. It's not the task of "thinking by itself like humans".

So it depends on what "more advanced" means. There are probably more "advanced" tasks towards AGI. But towards being a great conversational agent perhaps openAI has the best task-dataset combo today. At least I'm quite sure that there aren't systems which would be "significantly" more advanced than that, because I think the current limit is that it's "just" a very good conversational agent.

NOW it does, after a massive public beta-test :)

How about mocov2? That should work on a single gpu

[removed]

[removed]

I think single GPU for SSL contrastive learning is a research direction to pursue, I'm not sure if anyone published papers on it, but if there's none, I'm surprised.

Meh - now OpenAI has the best chat dataset available to mankind. Not sure if saying one is more advanced now will make any difference in a few years.

For 224x224 images, sure. But for images with large sizes, for example satellite images, it is hard to get 200+ batch size for a single gpu.

Google has few systems that would beat current public SOTA models. PALM/Minerva/Med Palm is the best but Flamingo, Chinchilla/Sparrow would also best chatGPT.

Dunno about anything from meta. They have open source GPT models released but they're not as good as Open AI's stuff.

I managed to use SwAV on 1 GPU (8GB), batch size 240, 224x224 images, FP16, ResNet18.

Of course it works, the problem isn't just the batch size but the accuracy - batchsize trade-off, and the accuracy was quite bad (still usable for my task though). If 50% top5 on imagenet is ok for you, you can do it. But I'm not sure there are many tasks where it makes sense.

Perhaps contrastive learning isn't the best for single GPU. I'm not sure about the current SOTA on this task.

Can you not train in python and dump the results to a file and run analysis on that? Either you have to be an expert in the details of the implementation or you have to use the setup everyone else is using.

[deleted]

[removed]

Thanks. This is useful.

Ok, yeah that's what I was thinking. That totally makes sense. Good luck!