Recent comments in /f/MachineLearning

jimmymvp t1_j4fcjly wrote

Reply to comment by chaosmosis in Why is Super Learning / Stacking used rather rarely in practice? [D] by Worth-Advance-1232

Hm, I'm not sure about that. There's the mixture of experts idea that does not exactly stacking, but rather specializes multiple models to parts of the data so each data point gets assigned to a specific shallow model. What you need then is an assignment rule, mostly done by a classifier and it's been shown that this is cheaper in terms of compute at evaluation time. I'm not sure if the idea is abandoned by now, but Google Brain published a paper on this and there were subsequent works.

2

nullbyte420 t1_j4f72e8 wrote

Reply to comment by NoPause9252 in [D] Is MusicGPT a viable possibility? by markhachman

Guy doesn't know anything about it. There are many famous copyright claim lawsuits in music. Chuck Berry vs The beatles is a cool one I think. Lana del Rey vs I can't remember is a more recent case 🙂 I'm sure you can find a list of famous copyright cases in music.

1

eyeofthephysics t1_j4f2w85 wrote

Reply to comment by IamTimNguyen in [R] Greg Yang's work on a rigorous mathematical theory for neural networks by IamTimNguyen

>u/IamTimNguyen

Hi Tim, just to add on to your comment, Sho Yaida (one of the co-authors of PDLT) also wrote a paper on the various infinite width limits of neural nets, https://arxiv.org/abs/2210.04909. He was able to construct a family of infinite width limits and show that in some of them there is representation learning (and he also found agreement with Greg's existing work).

1

m98789 t1_j4f135j wrote

Reply to comment by GoodluckH in [D]: Are there models like CODEX but work in a reversed way? by GoodluckH

Got it, this is how I believe it was implemented:

  • Stage 0: All code was split into chunks and had their embeddings taken, and saved into one table for lookups, e.g., code in one field and embedding in the adjacent field.
  • Stage 1: semantic search to find code. Take your query and encode it into an embedding. Then apply dot product over all the code embeddings in the table to find semantically similar code chunks.
  • Stage 2: combine all the top-K similar chunks into one string or list we can call the “context”.
  • Stage 3: stuff the context into a prompt as a preamble, then append the actual question you want to ask.
  • Stage 4: execute the prompt to a LLM like gpt-3 and collect the answer and show it to the user.
2

m98789 t1_j4eutfz wrote

Reply to comment by GoodluckH in [D]: Are there models like CODEX but work in a reversed way? by GoodluckH

Can you please link me to the tweet you are referring to?

From my understanding of Q&A from LangChain is it can answer “what” questions like “What did XYZ say…” but not “why” because the “what” questions are really just text similarity searching.

But maybe there is more to it, so I’d like to see the tweet.

1

GoodluckH OP t1_j4espcc wrote

Reply to comment by m98789 in [D]: Are there models like CODEX but work in a reversed way? by GoodluckH

Wow, that's really cool. But I can actually ask things like "what does XYZ do?", and it can give me some explanations like ChatGPT.

Clearly, they are using more than OpenAI's embedding to make this possible. I read if from Twitter that GPTDuck also uses LangChain which I'm not so familiar with.

Any idea how they're able to go from advanced search to conversational?

thank you for your insight!

1

aidv t1_j4e6ezn wrote

Reply to comment by nullbyte420 in [D] Is MusicGPT a viable possibility? by markhachman

Why are you so interested in me? You seem a bit obsessed.

That’s weird. People on the internet are weird. You are hella weird that’s for sure.

Oh well, that’s just what I have to deal with.

Weird mfs on the internet 🤷‍♂️

0