Recent comments in /f/MachineLearning

debrises t1_j3itq4j wrote

Reply to comment by i_likebrains in [D] Simple Questions Thread by AutoModerator

Larger batch sizes lead to a better gradient estimation, meaning, optimizer steps tend to be in the “right” direction, thus leading to faster convergence.

Run a test epoch to see when your model converges, and then use slightly more epochs so that your model can try to find different minimum points. And use model checkpoint callback.

As for loss, just use an Optimizer from the Adam family, like AdamW. It handles most of the problems that can happen pretty well.

The learning rate heavily depends on what range of values your loss has. Think about it this way: if your loss is equal to 10 then using the lr of 0.01 will get us 10 * 0.01 = 0.1. We then compute partial derivatives of this value with respect to each weight and backpropagate that and update our weights. Usually, we want our weights to have small values and to be centered around zero, updating them by even smaller values every step. The point is that your model doesn't know what values your loss takes and thus, you have to optimize the learning rate to find that nice value that connects your loss signal to your weights.

1

debrises t1_j3irf9s wrote

Reply to comment by CygnusX1 in [D] Simple Questions Thread by AutoModerator

>What are techniques or best practices for detecting/segmenting large objects in high resolution images? Some problems I run into are training with large image chip sizes

The first thing that came to my mind was gradient accumulation if you have limited GPU memory. Fitting an image of that size on a single GPU could result in a very small batch, which is not so good for training speed and stability.

PyTorch lightning offers such a feature if you're using PyTorch.

1

NichtMarlon t1_j3ikrkw wrote

Reply to [D] Have you ever used Knowledge Distillation in practice? by fredlafrite

Yes its very useful for text classification tasks. Big transformers get highest accuracy, but we can't deploy them because they are too slow. So we distil knowledge from bigger transformers into smaller transformers or CNNs. If you have a decent amount of unlabeled data to pseudo-label with the teacher, there is barely any loss in accuracy for the student model.

4

madmax_br5 t1_j3ik7wh wrote

Reply to comment by mandogbeer in [Project] Major drawback/limitation of GPT-3 by trafalgar28

It kind of depends on what the use case is. If it's simply to query against a large amount of information, you can just create embeddings of the information in chunks and add these together in a vector store index (https://gpt-index.readthedocs.io/en/latest/guides/index_guide.html). Then you embed your query using the same model and basically the relevant chunks are returned, and then you can synthesize a response from those chunks.

So let's say the use-case is to create a conversational tutor assistant for a textbook. Obviously, you can't put the whole textbook in the prompt. So you feed it in one paragraph at a time into the embeddings model, and store all these embeddings (along with the text they relate to) in a vector database like weaviate or pinecone. Then, when the user asks a question, you embed the query using the same embeddings model, and do a cosine similarity search using your vector database (a common function of vector DBs). And you say, return me the top 5 relevant chunks. Now you have some short context you can feed into normal GPT-3, with a prompt like "given the following context, create a bullet point summary" or "given the following context, create a simplified analogy using real-world examples."

Embeddings are basically the first half of the transformer. Language transformers essentially have two halves - the first half understands the input and encodes it into a set of numbers the model can understand. The second half takes that understanding and predicts a probable next word. When you think about this from a computational perspective, the first half only runs once, and the second half runs hundreds of times (once per output token). So you end up with only a fraction of a percent of the computation time spent on understanding (embedding) the input, and most of the time iteratively generating tokens. What semantic search in vector space lets you do is essentially compare items after only step 1, and THEN produce an output once you've gathered the necessary context. But of course you perform the embedding on your data ahead of time, so the only real compute that is needed at runtime is the embedding of the user's query, which is cheap.

7

universal_explainer OP t1_j3ij9jz wrote

Reply to comment by [deleted] in [P] searchthearxiv.com: Semantic search across more than 250,000 ML papers on arXiv by universal_explainer

Hey, thanks for trying it out!

First, do you mind sharing an example of different queries that return the same results? I have not been able to reproduce that (unless, of course, the queries are semantically similar, in which case that would be expected).

Also, of course exact search is far superior if you know the title of the paper you are looking for! In that regime, Google Scholar wins every time. However, semantic search might be better if you either a) can't remember the title but do remember some of the content or b) are simply looking to explore papers based on a handful of keywords.

Finally, the size of the database has no bearing on the quality of the embeddings, since I'm using the pretrained model by OpenAI. There is no notion of "popularity" except to rank the 10 papers with the highest cosine similarity to the query embedding according a citation score (if it's available).

3

universal_explainer OP t1_j3if5d5 wrote

Reply to comment by coumineol in [P] searchthearxiv.com: Semantic search across more than 250,000 ML papers on arXiv by universal_explainer

Might be in some cases, maybe not in others. Anecdotally, a query like "model using only attention mechanism site:arxiv.org" on Google doesn't bring up "Attention Is All You Need", while it does here. Aside from that, it might be a useful resource for finding similar papers based on an arXiv link.

3

trnka t1_j3i3vk4 wrote

Reply to comment by throwaway2676 in [D] Simple Questions Thread by AutoModerator

If your input is only ever a single word, that's right.

Usually people work with texts, or sequences of words. The embedding layer maps the sequence of words to a sequence of embedding vectors. It could be implemented as a sequence of one-hot encodings multiplied by the same W though.

2

aaaasd12 t1_j3i3ea8 wrote

Reply to [D] Have you ever used Knowledge Distillation in practice? by fredlafrite

It's like transfer learning?

In the company that I'm work only use the normal things like classification tasks/ segmentation with clusters.

Maybe the use case that i see is in NLP with topic modeling using bertopic and tuning the hyperparameters.

But in general simple models are perfect for the tasks that se have.

0