ruliad/deepthought-8b-llama-v0.01-alpha
Text Generation
•
Updated
•
35.1k
•
130
ruliad
Enterprise
company
AI & ML interests
None defined yet.
Recent Activity
View all activity
ruliad's activity
pharaoukÂ
updated
a
model
15 days ago
DhanOSÂ
updated
a
model
16 days ago
alpindaleÂ
updated
a
model
17 days ago
Add Deepthought chat template
1
#2 opened 17 days ago
by
alpindale
Fix casing on Python script name
#1 opened 17 days ago
by
alvarobartt
pharaoukÂ
updated
a
Space
about 1 month ago
Alignment-Lab-AIÂ
posted
an
update
about 2 months ago
Post
1003
remember boys and girls, always keep all your data, its never a waste of time!
pharaoukÂ
updated
a
dataset
about 2 months ago
Post
8396
Okay, first pass over KAN: Kolmogorov–Arnold Networks, it looks very interesting!
Interpretability of KAN model:
May be considered mostly as a safety issue these days, but it can also be used as a form of interaction between the user and a model, as this paper argues and I think they make a valid point here. With MLP, we only interact with the outputs, but KAN is an entirely different paradigm and I find it compelling.
Scalability:
KAN shows better parameter efficiency than MLP. This likely translates also to needing less data. We're already at the point with the frontier LLMs where all the data available from the internet is used + more is made synthetically...so we kind of need something better.
Continual learning:
KAN can handle new input information w/o catastrophic forgetting, which helps to keep a model up to date without relying on some database or retraining.
Sequential data:
This is probably what most people are curious about right now, and KANs are not shown to work with sequential data yet and it's unclear what the best approach might be to make it work well both in training and regarding the interpretability aspect. That said, there's a rich long history of achieving sequential data in variety of ways, so I don't think getting the ball rolling here would be too challenging.
Mostly, I just love a new paradigm and I want to see more!
KAN: Kolmogorov-Arnold Networks (2404.19756)
Interpretability of KAN model:
May be considered mostly as a safety issue these days, but it can also be used as a form of interaction between the user and a model, as this paper argues and I think they make a valid point here. With MLP, we only interact with the outputs, but KAN is an entirely different paradigm and I find it compelling.
Scalability:
KAN shows better parameter efficiency than MLP. This likely translates also to needing less data. We're already at the point with the frontier LLMs where all the data available from the internet is used + more is made synthetically...so we kind of need something better.
Continual learning:
KAN can handle new input information w/o catastrophic forgetting, which helps to keep a model up to date without relying on some database or retraining.
Sequential data:
This is probably what most people are curious about right now, and KANs are not shown to work with sequential data yet and it's unclear what the best approach might be to make it work well both in training and regarding the interpretability aspect. That said, there's a rich long history of achieving sequential data in variety of ways, so I don't think getting the ball rolling here would be too challenging.
Mostly, I just love a new paradigm and I want to see more!
KAN: Kolmogorov-Arnold Networks (2404.19756)
- 5 replies
Post
5748
Benchmarks!
I have lately been diving deep into the main benchmarks we all use to evaluate and compare models.
If you've never actually looked under the hood for how benchmarks work, check out the LM eval harness from EleutherAI: https://github.com/EleutherAI/lm-evaluation-harness
+ check out the benchmark datasets, you can find the ones for the LLM leaderboard on the about tab here: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard, then click the dataset and actually peak at the data that comprises these benchmarks.
It feels to me like benchmarks only represent a tiny portion of what we actually use and want LLMs for, and I doubt I'm alone in that sentiment.
Beyond this, the actual evaluations of responses from models are extremely strict and often use even rudimentary NLP techniques when, at this point, we have LLMs themselves that are more than capable at evaluating and scoring responses.
It feels like we've made great strides in the quality of LLMs themselves, but almost no change in the quality of how we benchmark.
If you have any ideas for how benchmarks could be a better assessment of an LLM, or know of good research papers that tackle this challenge, please share!
I have lately been diving deep into the main benchmarks we all use to evaluate and compare models.
If you've never actually looked under the hood for how benchmarks work, check out the LM eval harness from EleutherAI: https://github.com/EleutherAI/lm-evaluation-harness
+ check out the benchmark datasets, you can find the ones for the LLM leaderboard on the about tab here: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard, then click the dataset and actually peak at the data that comprises these benchmarks.
It feels to me like benchmarks only represent a tiny portion of what we actually use and want LLMs for, and I doubt I'm alone in that sentiment.
Beyond this, the actual evaluations of responses from models are extremely strict and often use even rudimentary NLP techniques when, at this point, we have LLMs themselves that are more than capable at evaluating and scoring responses.
It feels like we've made great strides in the quality of LLMs themselves, but almost no change in the quality of how we benchmark.
If you have any ideas for how benchmarks could be a better assessment of an LLM, or know of good research papers that tackle this challenge, please share!
- 3 replies
Post
Working through the Reddit dataset, one thing that occurs to me is we pretty much always train LLMs to be a conversation between 2 parties like Bot/Human or Instruction/Response.
It seems far more common with internet data that we have multi-speaker/group discussions with a dynamic number of speakers. This also seems to be more realistic to the real world too and requires a bit more understanding to model.
Is there some research into this? I have some ideas of how I'd like to implement it, but I wonder if some work has already been done here?
It seems far more common with internet data that we have multi-speaker/group discussions with a dynamic number of speakers. This also seems to be more realistic to the real world too and requires a bit more understanding to model.
Is there some research into this? I have some ideas of how I'd like to implement it, but I wonder if some work has already been done here?
- 5 replies
Post
Hi, welcome to my first post here!
I am slowly wrangling about 5 years of reddit comments (2015-2020). It's a total of billions samples that can be filtered as comment-reply pairs, chains of discussion, filtered by subreddit, up/down votes, controversy, sentiment, and more.
Any requests or ideas for curated datasets from here? I'll also tinker with uploading the entire dataset potentially in chunks or something, but it's quite a few terabytes in total, so I'll need to break it up still. I have some ideas for datasets I personally want too, but curious if anyone has something they'd really like to see that sounds interesting too.
I am slowly wrangling about 5 years of reddit comments (2015-2020). It's a total of billions samples that can be filtered as comment-reply pairs, chains of discussion, filtered by subreddit, up/down votes, controversy, sentiment, and more.
Any requests or ideas for curated datasets from here? I'll also tinker with uploading the entire dataset potentially in chunks or something, but it's quite a few terabytes in total, so I'll need to break it up still. I have some ideas for datasets I personally want too, but curious if anyone has something they'd really like to see that sounds interesting too.
-
- 7 replies
Post
hello world!
we're starting a new recurring event/club where we read and implement cool ai papers on skunkworks discord. first paper we chose is self-play as there are a lot of opportunities to expand on this framework, here's the link for the event: https://discord.gg/eAgBr7Fy?event=1194392774905172030
im plannin my next post to be a technical deepdive of PCN and ProspectiveConfiguration algo as ive been spending the last few days getting a good grasp at this promising alternative to BP, stay tuned.
we're starting a new recurring event/club where we read and implement cool ai papers on skunkworks discord. first paper we chose is self-play as there are a lot of opportunities to expand on this framework, here's the link for the event: https://discord.gg/eAgBr7Fy?event=1194392774905172030
im plannin my next post to be a technical deepdive of PCN and ProspectiveConfiguration algo as ive been spending the last few days getting a good grasp at this promising alternative to BP, stay tuned.
- 3 replies
alpindaleÂ
authored
a
paper
over 1 year ago