Number of tokens in the input perhaps?

I'm assuming u/imaginethezmell is referring to the context length. Indeed, if there is a need for longer context lengths, then OpenAI outcompetes SentenceTransformer which has a default context length of 128.

That’s cool. I assume you’re going to apply this to memories for the agent. There’s already relevant research on how to do that. Here’s one from Facebookresearch: https://ai.facebook.com/blog/retrieval-augmented-generation-streamlining-the-creation-of-intelligent-natural-language-processing-models/

My main problem is Lightning itself. I don’t find the flexibility of it like pytorch. Tried migrate an old code, gave up along the way, and still using pytorch now.

What are we looking at though? t-SNE?

I was lazy and used the model they show in their code example found here https://platform.openai.com/docs/guides/embeddings/what-are-embeddings.

Also on that page, they show that Ada outperform Davinci (BEIR score) and is cheaper to use.

So the attention mechanism has N^2 space and time complexity relative to sequence length. However, if you are memory constrained it is possible to get the memory requirement per token down to O(N) by computing only 1 token at a time and caching the previous keys and values. This is only really possible at inference time and requires the architecture was implemented with caching in mind.

Why not try Davinci instead?? Why is it on ada??

si if I understood it well, this lib "explains" git repo. Am I right?

This is really fascinating, thanks for sharing. I'm also working on generating natural language representations of Python packages. My approach is:

1. Extract a call graph from the package, where each node is a function and two nodes are connected if one contains a call to the other.
2. Generate natural language summaries of each function by convolving over the graph. This involves generating summaries of the terminal nodes (i.e. functions with no dependencies), then passing those summaries to their dependents to generate summaries, and so on. Very similar to how message passing works in a GNN. The idea here is that summarizing what a function does isn't possible without summaries of what its dependencies do.
3. Summaries of each function within a file are chained to generate a summary of that file.
4. Summaries of each file within a directory are chained to generate a summary of that directory, and so on until the root directory is reached.

I'd love to learn more about the differences/advantages of your approach compared to something like this. Thanks again for your contribution, this is insanely cool!

/u/NovelspaceOnly Can you verify this?

As to /u/Main_Mathematician77 , you are effectively a software developer with the ability to dabble with machine learning. Are you located in the states or elsewhere? It would be very confusing as to how you are broke yet have that skillset.

> Don’t @ me saying this is a waste of compute, I know what I’m doing and idgaf.

That is extremely unnecessarily antagonistic/combative

[deleted]

8K? I'm not sure what you're referring to.

openai is 8k

how about sentence transformer

Yesterday I set up a paid account at OpenAI. I have been using the free sentence-transformers library and models for many months with good results. I compared the performance of the two by encoding 20K vectors from this repo https://github.com/mhjabreel/CharCnn_Keras. I did no preprocessing or cleanup of the input text. The OpenAI model is text-embedding-ada-002 and the SentenceTransformer model is all-mpnet-base-v2. The plots are simple UMAP(), with all defaults.I also built a very generic model with 3 dense layers, nothing fancy. I ran each model ten times for the two embeddings, fitting with EarlyStopping, and evaluating with hold out data. The average results were HF 89% and OpenAI 91.1%. This is not rigorous or conclusive, but for my purposes I'm happy sticking with SentenceTransformers. If I need to chase decimal points of performance, I will use OpenAi.

Edit - The second graph should be titled "SentenceTransformer" not HuggingFace.

It is a bit like the daleks. Created by a mastermind but he forgot about stairs. Tall with cameras will get stuck under low hanging fruit trees. Ultrasound fine for walls but flower beds less so.

There is a lot of unknowns even for one application.

Another example..I had one with a rain sensor so it avoided cutting wet grass. Sounds good but if there are weeks of rain then the grass does not get cut until gets too long for the mower to cut.

Lawn robots work inside a border wire. They change direction randomly at the border. No machine learning but in practice it works really well. You can over think things.

I can't say for sure whether it is the best combination for research in the long run, but if you do go down that route I have found this template very useful

Open source?

If you’re that broke, aren’t there better uses of your time than this project lol

I have some preliminary generation scripts for SMILES chemical graphs, Feynman diagrams, storytelling with interleaved images, and testing compilation rates. sorry for switching accounts. this one is logged on my laptop lol..

Nvidia Jetson Nano and Raspberry Pi can run 2 FPS of AI object detection, on Yolo NN code. A Yolo model can differentiate 80 different objects, and you can run 20-50 different Yolo models, to detect 10,000 different objects.

The traditional programming copies the AI identification objects to a 3D map of the zone.

Yes exactly! That’s the a major goal of this project. I plan on incorporating the inference server that Yannic set up for open assistant.

Thank you! Yes I thought the topic tree would be a great complement to the commit tree. Would be great for stale repos with little to no documentation.

Also the option to mix in multiple repositories and message pass between them to help with brain storming new features. Or message passing between your repo and its dependencies.

I also plan on applying the basic idea of a GNN with prompting to the thought loop of an cognitive entity (basically open assistant). I believe if you take the tree your outputting for code, but use it to aid CoT reasoning, that could be pretty powerful