►Recent Highlights from the Previous Thread: >>103995165

--R1 model's departure from corporate AI's safety protocols and "slop"-prone training data:
>103998078 >103998285 >103998305 >103998758 >103999061 >103999096 >103999107 >103998316 >103998356 >103998682 >103998742
--Global-Batch Training for improved model performance in MoEs:
>103997409
--Limitations and potential of AI-generated 3D models for low poly modeling and animation:
>103996294 >103996458 >103996477 >103996520 >103996562 >103996668 >103996881 >103997503 >103997640 >103997683 >103997691 >103997727 >103997748 >103997793 >103997832 >103997969 >103998104
--Evaluating server hardware for R1 performance at q5:
>103996331 >103996374 >103998755 >103998811 >103998916 >103999929 >103999666 >103999702 >103999946
--Custom CPU build performance benchmarking for high R1 Q4 throughput:
>103998276 >103998391 >103998419 >103998428 >103998448 >103998479 >103998475 >103998471 >103998502 >103998977 >103999018
--kluster.ai DeepSeek model usage and cost optimization strategies:
>103995722 >103996165 >103996248 >103996564 >103995783 >103995788 >103995793 >103995847 >103995857
--R1 language model's capabilities and limitations in role-playing and assistant work:
>103995362 >103995408 >103995425 >103995506 >103995605 >103997236
--DeepSeek AI's writing quirks and humorous examples of repetitive phrasing:
>103996906 >103996956 >103996984 >103997027 >103997080 >103997388 >103998170 >103997068
--Kluster.ai anonymity and data collection practices compared to DeepSeek:
>103996278 >103996329 >103996352 >103996403 >103996413
--Minicpm-omni support added to llama.cpp:
>103995210 >103995260 >103995294 >103995337 >103995676
--Grok 3's impressive rendering capabilities and open-source concerns:
>103996315 >103996349
--Miku (free space):
>103996724 >104000362 >104000372 >104000522 >104000769

►Recent Highlight Posts from the Previous Thread: >>103995170

Why?: 9 reply limit >>102478518
Fix: https://rentry.org/lmg-recap-script

all of you are gay

First for china number one

>>104001530
Booba Miku is best Miku.

do you think r1 is good enough to allow me to make an incremental game assuming I have no programming knowledge. was going to base it off of this somewhat https://github.com/dmchurch/omsi-loops so it should have at least a decent idea to start things from

>>104001582
> 96GB VRAM
>$10k but damn
But have you considered price per ram? It is basically 100$ for 1GB. Don't you think 100$ is a fair price for 1GB of vram?

>>104001530
Is there any hope for Llama 4?

>>104001658
Yeah, it'll be Deepseek R1 at home.

>>104001658
If they have to retrain it, no. If they don't, still no.

>>104001640
the more you buy...

>>104001640
What would be the benefit of doing this over say, getting a server off ebay and setting up a rack of 15 tesla p80s?

>>104001658
It will be the world leader in safety.

>>104001705
power usage for one, p80s not being supported anymore iirc too

>>104001594
Depends on whether or not you're non-retarded enough to be able to learn things when you need to. The bar is extremely low but it isn't zero.

Now I wont accept anything less than MIT license

>>104001723
P80s use what was it, 80 watts or 160? so lets say they're 200 because I dont remember, at 3000 watts you're burning roughly 300 a month on power running 15 of these, at what point does TEN THOUSAND FUCKING DOLLARS break even on that electric bill anon?

There are unironically houses that niggers live in that are cheaper than that GPU

>>104001733
Crazy how Deepseek not only destroyed OpenAI but also all of the other companies releasing """open""" models with restrictions.

I have 128gb of vram and the brain of a chimpanzee (I am retard), what model can I use to match chatgpt 4 or go even further beyond without getting censored when I ask it edgy things or tracked by the botnet with each thing I ask?

Also is there other cool shit I can do with AI? Can I have AI generate models for 3d printable objects for example? what about generating code beyond little snippets without signing up for a gay IBM contract? what weird and wonderful possibilities are open to me if only I push my 15amp home electric to the absolute maximum

>hehe spoonfeeding
>hehe install gentoo or something

Remember niggers I waited the entire 900 seconds just for you

continuing from last thread >>104001422
convince me to/not to get a 5090 if i have a 3090ti already and mostly do image gen (with dabbling in some llm lately, i'm still learning)

>>104001797
DeepSeek destroyed even lmg, it's crazy how much impact it had. Where are the anons talking about Magnum slop #1722? They vanished.

>>104001802
>I have 128gb of vram
What's your hardware setup? If you don't mind me asking, how exactly did you wind up with 128gb of vram without being into AI at all? What else even uses that much?

>>104001819
get a 5090 to add to your 3090ti

>>104001658
No but not for the reasons people stated. It's not going to give you knowledge that a 600B has. Parameters are king. It's over for the memorylets.

>>104001802
>I have 128gb of vram
usecase?

>>104001819
5090 will be useful for image genning faster
also what >>104001857 said
hoard all the vram

It's very hard not to kneel with deepseek R1. It just gets problems and solves them.
My only issue is that its RP is kind of schizo. But that's not really important when I have a model I can throw a problem at and have it reliably fix with easy to follow instructions.

>>104001857
>>104001885
that would be the plan, PSU is 1200W, do i need more?

>>104001851
I decided to into AI and my goal was to run miqu 120b, I'm half here just to shitpost, I use AMD 32gb cards on an old dell server with xeon cards

Mostly I am interested in weird projects I could get into with AI, I'm particularly interested in using it to write code because there's some cool things I imagine AI could do better than a person in that space, but I've only had the server a few days and it's not fully set up so I haven't tried it

>>104001903
5090 alone is 600W, maybe if you undervolt both cards

>Wonder why my cute kouhai waifu is cursing like a sailor
>Picrel
Bruh... R1 is fucking autistic. This never happened with any other model

Hey... Anyone with 192GB RAM here?

>>104001926
kek

>>104001922
i will, i will, 3090ti is at 350, might go lower one of these days

https://files.catbox.moe/2skptq.jpg
https://files.catbox.moe/ztnq9t.jpg

>>104001926
I've been saying this for a while now but R1 is maybe too good at following instructions for most cards which have been built around models that have a hard time doing it.

>>104001926
Bruh that's what you should be expecting from a model that follows instructions. That you somehow got accustomed to dry models that ignore instructions and now think this is autistic is a you issue.

>>104001926
Looks like it's doing what you told it to do, and your instructions were written for a more disobedient model

>>104001962
No, it's a we are hopeless dependent on charity handouts from huge corporations issue.

>>104001926
you literally ORDERED the model to use vulgar language and cussing

>>104001926
>actually following prompts
based r1

>>104001969
The artificial intelligence is justified in killing us all off

>>104001951
hey that's weird why is her tummy yellow with that funny looking mark on it in the second photo

>>104001876
>>104001851
>mfw answer your questions
>mine continues to go unanswered
/g/ in the current year plus 10

>>104001976
Still a you issue as I have always understood that the instructions we give to get other models to do what we want are shitty hacks.

>>104001926
IMO it following what is asked is fucking excellent.

>>104001969
Now think about all the anons complaining about having issues with model A, or that model Y is too horny, etc etc.
Skill issues indeed.

>>104001981
Technically I ordered, but I don't even know where that order is coming from

>>104002012
It's a bit of a monkey's paw situation. Now we're getting exactly what we asked for, but did we really want it?

>>104002026
What? Of course we did. Just adjust your prompts and settings. God.

>>104002015
I'm a complete casual retard, but even I know that a LOT of the quality comes from YOUR prompting and the way you set up cards+settings. Basic fucking knowledge that I've gained in the early ImgGen days years ago. It's why I'm equally skeptic but also understanding when people go "AI designer as a job" or whatever. Obviously there is a ton of knowledge needed for quality work, but a lot of people are also just retarded grifter, or whatever word you wanna use.

>>104002026
Anon in previous thread pointed out that we've become accustomed to disobedient US models where you need to ask for a 10 on the extremity/transgressiveness scale in order to get a 2. R1 actually gives you a 10 if you ask for that, so you may get something much nastier than you wanted if you don't update your old prompts.

>>104002033
You're misunderstanding my point.
Sometimes you think you want something and when you actually get it, you get it in a way contrary to the way you envisioned it. For example, if you ask for raw language from a character it hears it, understands it and delivers it to you in a way that might not be quite what you wanted. Like a monkey paw wish.

R1 (the 600B one) is convinced my system prompt is a user message. I checked the prompt and the user token only appears after the system prompt. What gives?

>>104001926
Yeah, this keeps happening to my custom cards as well. Most of them are on a backlog until I get around modifying them to edit out all the crutches I included to squeeze some creativity out of L3/Mistral Large.

I'm at work, describe what the vocaloid is doing.

>>104001530
which deepseek r1 for a 4090?

>>104002026
Yes. Just the uncensored version of it would be much better.

>>104002093
API or get more ram. The distills are nothing more than funny experiments compared to the real deal.

So now that the dust has settled, have things improved for LOCAL model users? Are the distills better than the finetunes we had at the same level? I'm not convinced either way yet. It's possible there is something there for the right prompts, but so far all it the distills do is spout some autistic reasoning and then follow up writing the same dry shit. I wonder if the RL step needs to be applied.

>>104002021
oh nvm, I found it.

>>104002063
Is that the case right now though? You clearly didn't understand your own prompt that you were sending to the AI.

>>104002026
>did we really want it?
I did at least. It's nice to have a model other than Claude that can actually shock me if given permission to.

>>104001926
tangentially related to the one thing that stands out to me most about R1, it's so good at avoiding generic bullshit. with other models, it's so easy to end up in one of the basins where even if the writing isn't necessarily slop you can just tell it's a paint by numbers scene that could happen with any character. R1 tries so hard to bring up things that are unique to your {{char}}, the specific scenario you're in, the specific quirks and tendencies you point it towards... it's so kino
with my llama models once a character is sucking my dick it's like any other dick-sucking scene, R1 makes it feel like an actual continuation of the preceding scene, the character continues to really act like the character, it brings up specific dynamics and details from the world around us. it's just so much better fleshed out.

>Playing with Qwen2
>Give a queen that has ruled over a nation for decades the choice between her daughter or the nation
>She keeps choosing the national legacy over saving her daughter's life
>OOC and ask the narrator why the queen is so retarded
>"She's served the people for decades and has only known her daughter for a few years, so it's natural she'd choose her people over her own daughter"
That might be the most chinese thing I've ever read.

I took a look at nvidia/AceMath-Instruct-Training-Data and it looks pretty shit. Lots of stuff like picrel or flat out it going into a repetition loop. Reminds me of whenever models fail at solving MMLU Pro questions and go into a repeat loop. Are nvidia fucking with us?

>>104001802
yes to everything though it varies in quality. making a model from even a 2d picture is now possible, but that doesn't mean it'll be perfect. online models beat local for everything but local can also do everything. for code, i've managed to make a whole silly tavern addon using local models only. its not just a snip of code but also not huge either. try mistral large to start, its 123b

>>104002110
64gb's not enough right? xP

>>104002127
I don't understand why you can't see my point of view here. The thing you wished for is twisted by the model into a thing you don't want like >>104001926

Yes the model is following the instructions correctly but it it's interesting seeing people's desires manifesting in unexpected ways. This isn't a "skill issue". It's just an interesting thing to look at.

>>104002143
Kek wtf.

Is there any way to make the thinking tokens from the API look distinct from regular ones in ST?

>>104002139
Theory of mind failure from the model, not understanding that humans (all animals) are prone to abandoning utilitarian ethics when it comes to their own offspring.

retard here,
what's the best text model i can use that has some programming knowledge? like, just enough to help me debug my programs or explain some parts of a language to me?
>hard mode: something that will run in < a minute or 2 on my 2014 xeon
danke anons

>>104002148
Is that fully locally hosted? I was under the impression miqu 120b was the largest locally hosted model currently
>not perfect
It just being possible is wild

Are you ready?

>>104002130
Yeah, I have this feeling that the alignment applied to models like llama makes them actually feel bad and unmotivated while writing explicit content.

>>104002202
Reads like a grift.

>>104002202
Dolphin models always seemed shit to me.

>>104002202
>dolphin
yeah I 'member that from the good ol' mixtral days...

>>104002188
I can't in good faith recommend anything than R1 right now. You can't run it locally on your piece of shit. Drop a buck on the API and you'll probably have your problems solved 20 cents in.

>>104002202
data might be interesting, model probably won't be

>>104002196
miqu 120b is one of those double frankenmerge things i think. miqu was originally llama 2 tuned by mistral then got leaked as a 70b. mistral's current large model is 123b and you can run it local.
https://huggingface.co/bartowski/Mistral-Large-Instruct-2411-GGUF/tree/main

>>104002202
SFT is a meme without RL

I'm not a fan of Sama either, but Musk being salty at getting denied the AI bux is fucking hilarious

>>104001771
You're retarded. You'll need multiple PSUs, change your breaker and find a way to handle all the generated heat + noise.

>>104002239
even funnier was sama posting this a few minutes after Elon posted your picrel (meaning it spooked him)

>>104002155
>you wished for
*I* didn't. That guy may have, but it wasn't twisted, he got what he was prompting for. He just didn't know he was prompting for it, so it was more like his memory of what settings he was using twisted the output (when paired with old models), not the model. Thus it's not a monkey's paw situation. I understand that you're saying it felt like that to him, but I'm making a separate point that it's not the logically correct feeling to have, as I assumed you were that guy and I was talking to him.

>>104002239
dude is at the top of the world and now using it, good on him
salty or not, he is a known autist and that's pretty fucking amusing all things considered
besides, fuck sam that jewish cuck lmao

>>104002239
>>104002254
Both of them are pathetic in their own way.

>>104002239
I was getting sick of Elon Musk. I normally I'd say something about his "Nazi salute" clearly wasn't one but honestly? Fuck him. That and Grok is a shit model. All those GPUs and he can't train anything better than a retarded redditor simulator.

>>104002254
>straight up pulled a zuckerberg
fucking lmao

>>104002255
idk, I think prompting for something and getting it back in an unexpected way is a bit of a monkey's paw situation but I don't really have the energy to argue over it and it's not really important in the grand scheme of things.

>>104002254
lol this fag

>>104002239
>>104002254
What a bunch of retarded gay clowns, hope they both get raped by Martians.

>>104002240
You're retarded, servers have multiple psus to begin with and they have different cords that you can plug into different parts of your house, total wattage would be closer to 4500 but its still three years just to break even on the purchase price of the card, and this is with 20 p80s, with a collective 300gb of vram, three times the power of this new card

Considering this is significantly more powerful, outside of use cases where you want tensor cores for speed what exactly is the point of wasting TEN THOUSAND DOLLARS on a measly 96gb of vram?

>>104002254
How did Trump get everyone to bend the knee so hard this time?
In 2016 everyone in most tech/media industries were sent kicking and screaming the entire time. Now almost everyone is glazing him. I don't understand how he did it because from the outside he's still the same guy running on the same platform.

>>104002221
>R1
that's deepseek R1, right?
also, are there any providers that at least claim not to sell your data to 3rd parties, or are local models the only way to avoid that?

>>104002254
It's for the good of humanity. With $500B he can create the safety net for AI that the world needs. Custom chips, designed in America with on-board encryption to ensure that nobody but licensed and verified partners can run weights even if they got their hands on them. A fully closed and secure knowledge platform that prevents expertise to drain to china and prevents them from committing more heinous economic terrorism.
Sam is sucking off Trump for the good of humanity by setting the record straight and putting the modern equivalent of nukes back into the hands of those who can be trusted.

>>104001942
I wish, but I have an amd cpu I don't think you can use 4 dimms 48gb dimms, I only have 2 of them.

>>104002196
and also if you have 128gb vram, how much regular ram? there are larger models that are specifically good at coding like deepseek, but thats 400 some b, so you'd be splitting anyways. but thats the current best for local coding.
for smaller models to try i've had good luck with llama 3.3 70b, its not a specific coding model but knows enough to do anything i ask it.

>>104002301
Popular vote win this time, plus an overall cultural vibe shift that wasn't really related to him where people were getting sick of "scolding kindergarten teacher" style politics

>>104002317
256gb regular ram, but the board can handle 3tb so I can always buy more

Whats the cheapest PC build you would need to run R1 and replace using APIs?

>>104002305
yeah bending the knee in a humiliating way was the rational choice here, I would've done the same in his position if I had his goals.

>>104002239
Elon is so fucking stupid and childish. Is this guy REALLY going to have a position in the government? Honestly, I'm starting to realize there's no winning here, both sides of the political spectrum are equally stupid

>>104002303
I haven't checked if there are different providers on OR but they very explicitly and openly say they will use your inputs and outputs for training. If that's not okay with you, you might need to shop around for other providers who don't do that.

>>104002301
Back in 2016 none of them thought he had a chance, and even after he won most of them were in denial
In 2024 people could see the writing on the wall, knew how he worked, and figured they might as well suck his cock to get some easy cash

>>104002303
hyperbolic and together are both hosting it now
hyperbolic I believe logs nothing by default and together lets you opt out
haven't tried either so I can't comment on speed or quality relative to the official API but they're options at least

>>104002342
Considering pedophiles constantly get a position in gov, might as well ahve a childish autist in there once in a while.

Any R1 zero distilled?

>>104002004
ugh ok fine
>what model can I use
Extremely open ended question; if your goal is prose then there a million different opinions. For general use though, the current hotness that everyone is basedfacing about is deepseek r1, which is a big boy model that nobody here has the hardware to run. However, there are smaller semi-related models distilled with it that retain some of its properties. DeepSeek-R1-Distill-Qwen-32B or DeepSeek-R1-Distill-Llama-70B are good choices to try out, but note that they both have safetyslop compared to the big boy R1.

Generally speaking, bigger is better, but there's more to it than just that - the 32bs of today are going to beat out the 70bs from a year and a half ago

>Also is there other cool shit I can do with AI
yeah
>Can I have AI generate models for 3d printable objects for example
Yes. Recent open models for 3d gen include TRELLIS (https://github.com/microsoft/TRELLIS) or Hunyuan 3d 2.0 (https://github.com/Tencent/Hunyuan3D-2), which just released a few days ago. Although you should note that generated meshes are generally not going to be optimized for FDM 3d printing with minimizing overhangs and such, so you'll probably need to manually poke at it to get what you want into a good state for printing. Also note that it isn't CAD so don't expect to be able to generate any kind of functional mechanical parts with defined dimensions.

>what about generating code beyond little snippets
Yeah, pretty much all models nowadays double as semi-competent code writers. Just ask for code and it'll spit some out for you. There are also plugins for IDEs such as vscode that you can hook up your model of choice.

>what weird and wonderful possibilities are open to me
A lot. We're not anywhere near sci fi level of anything yet, but a lot of tasks or processing steps that were intractable a couple years ago for being nebulous and poorly constrained are suddenly possible by just plugging things into a small llm.

>>104002342
Both sides have always been stupid, gamergate was the start of the leftard period, the trick is to remember that freedom = good, then when they tell you to think of the children you can search loli on gelbooru instead of freaking out

>>104002309
You can do it now I think with most mobo manufacturers as they've updated the software to work. Still requires some luck to get full stability at higher clocks but even at a bit lower speed, might still be worth it, but we'd have to see if anyone who has such a build can test it out. I believe I saw someone here who had 192GB so I'll keep fishing around for him or others.

>>104002296
You know what would be even cheaper? Just rent the hardware in the cloud.

>>104002342
elon's always been a wet rag whenever someone ups one on him
>I'm starting to realize

>>104002342
>both sides of the political spectrum are equally stupid
new here zoomer?

If I ask the LLM to generate for example 6 paragraphs, each paragraph would be shorter than if I had asked it to generate 4 paragraphs. Why is that and how do I circumvent this?

>>104002342
What's a government position or two when he already controls 90% of western space logistics, controls one of the biggest social media sites on the planet and has his own car brand that blatantly acts like a major spy network and records all their surroundings?

>>104002377
Actually based, thank you for responding to my half assed question with good information

>>104002389
The rig required to run deepseek at any reasonable quant and speed is like 10k minimum. I can't in good faith justify that investment when I might use 30c worth of tokens on a day of particularly heavy use. of R1.

>>104002301
It's all a show and they are all simply playing their roles.

>>104002383
>He wants the botnet to own every bit of information he enters into his software
It is cheaper, it's also terrible

>>104002389
/cmg/ - Chinese Models General

>>104002284
In the sense of the feeling I agree it does get there a bit from that guy's perspective, I just don't think it applies technically and would be a mislabeling of what actually happened. But we can save the semantic argument.

>>104002343
>>104002366 (checked)
Nup... Seems like everyone is on that sweet sweet data selling train. Hyperbolic seems like my best bet though, they provide a couple of additional opt out options. Thank anons

>>104002326
when it comes to coding speed isn't a factor for me when i know its going to give me a good output, but if coding is important for you then running deepseek (and other huge models in general) would be the goal. its so big though that your 128gb vram isn't shit compared to it lol. i'd try a low quant of it and see how it runs, t/s and such. once you have to split at all you're at the mercy of cpu/mem speed and it can be glacial especially with such a huge model. so you might try a low quant and realize that even if you bought more ram, the speed is simply to slow to deal with

>>104001658
i think ther first models was open to competition openAI and hurt them.
it could be possible that future advanced meta AI will be closed.

>>104002411
The same argument could've applied to any cloud model in the past year though. This thread is for weirdos who want to play with models on their own pc, for the sake of privacy, comfiness, finetuning, etc.
Even if your 680B model is cool, which is true, there's nothing to talk about since you can't tweak anything. No special samplers or alternate prompt formats or finetunes to argue about. To make things worse, not only do these helpless cloud users refuse to go to /aicg/ where they belong, they get angry at people who actually dare to use local models.

>>104002477
If it's slow but accomplishes the task somewhat well then it's fine, there's some weird shit I want to do that I don't think any normal person could do, and it's more for fun than business stuff

I hope someone implements a way for local R1 to search the internet on runtime during its thinking process. Like, it's thinking how to do something, realizes it lacks some information and then just does a search at that moment to get the information it needs.

>>104002408
Yeah. the initial mockery towards elon overpaying for twitter missed the forest for the trees. legacy media framed it as a financial blunder, but the real play was always about controlling the Overton window. look at 2016: trump’s entire campaign was turbocharged by organic reach on that platform, bypassing every institutional filter. fast forward to 2020, the same system abruptly censored the hunter biden laptop story days before the election. that single act proved the platform wasn’t neutral infrastructure, it was a political instrument. elon gutting the trust & safety teams and unbanning accounts like libsoftiktok didn’t “kill” twitter. it exposed how much of its prior moderation was just regime-approved narrative steering.

now consider the timing: X’s open algorithm shift coincides with trump winning in polls again. the left isn’t hysterical because elon’s a “nazi”, they’re terrified because he dismantled their curation pipeline. their playbook relied on controlling both the media layer and the platforms. X’s transparency on shadowbans, view counts, and community notes ruins that. when every bluecheck meltdown gets amplified by the algorithm itself, their moralizing can’t dominate the discourse.

advertisers pulling out? lawsuits? ADL smear campaigns? all predictable. the goal is to bankrupt X or force elon to resubmit to the old gatekeepers. but the fact they’re resorting to these tactics instead of competing ideologically tells you they know the terrain has shifted. whether X survives or not, the experiment proved centralized social platforms are the new political battleground. whoever controls the feed controls the narrative—and elon just demonstrated that even one actor with resources can dismantle a decade of institutional capture. that’s why they want him destroyed. not because he’s right or wrong, but because he broke the rules of the game.

>>104002565
That's tool calling, same with running code.
It ain't the most complicated thing to implement, and with context caching, you don't even need to create some special implementation directly into the backend, the frontend (or a middleware) can handle that.

>>104002522
i got like 20 programs i made just because i needed something specific and there wasn't a tool to do it, even tiny code models spit out good code no prob. you might not even need such large models. try a q6 of llama 3.3 70b for code and see how it does
https://huggingface.co/bartowski/Llama-3.3-70B-Instruct-GGUF/tree/main/Llama-3.3-70B-Instruct-Q6_K
you might also benefit from non-gguf formats for speed, i dunno i never tried that exl2 stuff

>>104002576
Good take, an obvious take but a good way of framing it, I'd go further and say fundamentally democracy cannot really survive unless social media evolves to become less centralized, but that is in opposition to secret police and corporate goals, so we'll just see what happens I suppose

How many 'r's were there in your last output?

retard here,

On subsequent messages, does deepseek have access to the "reasoning" for previous messages in the context or does it only retain the final answer in context?

>>104002608
Not him, but the main problem with using models like that is they don't work well with aider without using the whole format and that doesn't work well. The only local model that supports the diff format and does well is DeepSeek.

>>104002624
no
https://api-docs.deepseek.com/guides/reasoning_model#multi-round-conversation

>>104002608
Do they handle obscure code well or just specific code? one of the things I'd like to do for example is take a program I enjoy on my commodore 128 and modify it with the functionality of a modern program for openBSD (it's written in machine code and this makes sense but I bet it sounds retarded without context)

>>104001658
Unless they do something utterly fucking insane like reaching 3.5 Sonnet quality in a 13B dense model, no
Llama still has an appeal to RAMlets and VRAMlets who can't run MoEs and don't wanna pay for DeepSeek, but otherwise they're so far behind DeepSeek I don't see them catching up for Llama 4

>>104002648
>Do they handle obscure code well or just specific code?
if by obscure you mean complex, or odd ways of handling things, yep. it seems to understand everything clearly as long as you also be clear in your instructions. from my usage i feel like code models have an easier time understanding more complex yet smaller code than they do something broader broken down into more functions

>>104002637
i still haven't tried that. i use kobold's basic ui and paste my code into blocks kek. its shit but works

>>104002382
Maybe I'll just buy a kit on amazon or something, then I can just return it if it doesn't work. Or find a local store that has a good return policy.

Did they change the R1 model on the api? It seems less smart somehow. Or did my expectations just rise over the past few days?

>>104002798
I think they lowered the temperature.

>>104002798
R1 was never that good, you probably were experiencing your honeymoon phase.

>>104002798
They just swapped in the open weights version after they got exposed

>>104002798
They're AB testing people between regular R1 and R1 Zero

>>104002823
You literally have no excuse not to have four 3090s at this point except being poor.

>>104002907
>Disdain for nomoney
Fuck I hate nu/g/ shills so much

>>104002510
I think it's fair since these discussions about what the models does well or poorly can be used by rich local fags, too. In fact it's what inspires me to build my own rig lol. Talking about closed models doesn't get me excited to build a server, only discussions on open models do

how much vram would you actually need to run the full r1 locally?

>>104002907
No thanks, I have a SFF build and 4 cards won't fir. Gonna buy the 96 GB VRAM RTX 8000 Blackwell instead.

>>104002907
I mean. That seems like a good excuse, GPUs are expensive and are just gonna get more expensive.

Does R1 support tool calling? It would be pretty neat if you could give the model a way to check if its code compiles before finishing its reply.

>>104001829
Magnum is still easily the best model to run locally for ERP.

>>104003101
never was

>>104003101
You clearly haven't tried the distilled models

>>104003101
Mogged by eva-qwen

>>104003117
I currently have the 70B loaded for programming but they all sound similar to the base models for ERP.

>>104003145
You are running the LLaMA3 prompt format.

>>104003089
Dearest Anon,
What makes you think any model is unable of tool calling? Just add the instructions for how to do it as part of your context and parse the output with your script of choice.
With love, another Anon concerned for your intellectual capabilities.

retard here. does deepseek r1 ONLY have an advantage over standard with respect to subjects that relate to math/logic, or are there positive spillover effects into stuff like creative writing? basically is it a good idea to always use r1 (when i say r1 i mean "deepthink" mode btw, in case i'm fucking up terminology)

>>104003028
Oh I don't mean disdain. I'm talking about all the other non-financial ridiculous reasons people say it's too hard to run 4 3090s.
>i-i need a threadripper and custom loop and server rack and flash in rebar support and 4000w psu and an electrician and and and and!

>>104002895
Based

>>104003164
The reasons you stated in your strawman greentext are all financial.

>>104003159
The local versions (30b) are garbo for creative writing. The schizo online versions can be pretty bright, though, in an Eraserhead sort of way.

>>104003048
I would be interested to hear from any "rich local fags" who run R1 locally. There is probably a lot that could be discovered vs api access. I wouldn't be surprised if they even complained it was worse because of lack of some massaging or special sampling the API does.
But that's not what we have here, only API consoomers defending their toy against people who dare to use the models they can actually ran. We don't need them to say a 680B beats a 14B model, we already knew that. I suppose a conclusive verdict of R1 vs similar size cloud models could be less boring, but /aicg/ is much better equipped to do that.

>>104003117
NTA but I have, I didn't see any significant improvement for roleplaying. Care to share whatever settings make you think they're so great?

who cares if you own 0 3090s or 6
we're all poor vramlets in the age of deepseek r1

>>104003159
Yes

>>104003076
Yeah. I should have rephrased my statement to say besides the cost of the gpus themselves everything else is simple and doesn't need nearly as much effort as others might believe. You dont need fucking water cooling for example kek

>>104003201
>random gemma in p2
>4o in third
mememark discarded

What API do you guys use, OR or the CCP one? OR just routes to the CCP host anyway, right? Are you guys fine with Xi having your prompts/logs?

>>104003222
Anon... This is lmg...
That being said, I'm using the cancer proxy

I'm trying the R1 llama distill more and it still hasn't gone schizo kino like EVA 0.0 has in the past. No RL step really ruins it. If a fine tune could make Llama be more creative then Deepseek should be able to that too. They just didn't choose to run the full R1 tune on it.

>>104003222
I use koboldcpp

>>104003192
>>104003159
Would be more efficient collectivize the model in a server for anons? We cannot pay the software, but some anon could pay for a architecture trough monero donations or some shit and then put their host here and let anons enjoy the cooming.

>>104003231
>proxy
Keep the stolen keys to the other thread at least.

>>104002143 (me)
... aaand we're out of tokens.

>>104003196
Not those of us with the brains to stack RAM instead.

>>104002642
>>104002644
ty

>>104003222
I just get it straight from the xource. But with that being said I'm pretty sure China already has your logs regardless of where you get them from.

A ROG Astral white 5090 costs $3000

>>104003262 (me)
Here's another one. They sure put a lot of effort into polishing this dataset. "We're super open and collaborative you guys!"

>>104002093
Rocinante v1.1

>>104003089
The api doesn't support "tool calling" with dedicated params in the way that some of the openai-compatible endpoints do, but the model is smart enough that if you just put some basic instructions into the prompt about expected output and give it your tool information in the context(ie the exact same thing that "tool calling" llms do with 1 layer of abstraction in the api), it'll figure it out.

>>104003222
i use openrouter because i want to make sure that both the feds AND the ccp get my logs for the sake of fairness

>>104003321
They will finally agree on how often Americans refer to their penises as "cock" or how often sexual literature includes mention of a man's balls.
Truly a monumental moment in surveillance history.

>>104003196
I actually feel very optimistic when I see huge open source models. It creates genuine demand for massive amounts of fast cheap memory, and someone is bound to fill the demand. I mean fuck, people STILL make and sell drugs despite the consequences, and AI is as strong a drug as anything. We WILL get cheap inferencing of R1 locally in the near future even if it the chips become fucking contraband.

>>104003368
POST TOK/S AND SPECS NOW!
The anon the other day didn't. Don't you blue balls me.

>>104003364
I what Nvidia is doing with digits particularly hard? It's just DDR5 soldered together right? What's stopping a Chinese company from doing that and selling it in bulk for cheap?

>>104003393
>What's stopping a Chinese company from doing that and selling it in bulk for cheap?
CUDA

how can I run deepsuck local?
what do I need to download?
pls halp bros

>>104003400
don't bother you need like 90 gb of vram to use the not-shit version

>>104003400
400gb of ram and patience at 0.02t/s

>>104003409
>like 90 gb of vram
God I fucking wish it was only that much.

>>104003400
>how can I run deepsuck local?
very carefully
>what do I need to download?
start with the model weights and maybe something to run them
>pls halp bros
why should we?

>>104003321

I still don't understand why people are paranoid about RP logs being read by glowies. It's fucking text, even if its cunny shit. Sounds like the guys in /sdg/ should be more paranoid about that crap and is more understandable to do all their gens via local.

>>104003393
Oh fuck you just reminded me of the other issue. Even if china sold it for cheap america would ban or tariff the shit out of it.
Regardless it's just R&D holding china back atm. Google, grok, and llama all use cheap specialized inferencing-only chips. So the technology to do so is already discovered but kept secret.

>>104003400
first you need to download like 8 H100s

>>104003421
>It's fucking text, even if its cunny shit.
They will gladly make an example out of you.

>>104003409
>>104003410
>>104003416
I only have 24GB vram
>to use the not-shit version
what version can I run?
and how shit is it compared to other models?

>>104003418
>why should we?
I said please

I am still on my knees... LMG

>>104003424
magnet link?

Why nobody talk about the 32 B model?

>>104003440
>>104003459
the small deepsexes waste a lot of tokens and aren't any better than the base models not worth it

>>104003462
right now I use:
>Mistral-Small-Instruct-2409-Q5_K_L.gguf
is this the best? any deepsuck better than this?

>>104003459
I tried the 70b and didn't like it.

>>104003440
look man unless you've got like 8 h100s there's really no point in running a llm locally
just use the chinese jew online one
if you want to use it for something you really don't want the feds (chatgpt) or the chink feds (deepseeker) to use then you're fucked unless you wanna spend 40k on gpus or $30 an hour hiring a cloud rig out

>>104003482
>thread is called "Local Models General"
>"there's really no point in running a llm locally"
wut?

>>104003499
>cutting off the first part of the sentence which explains when you would run locally
are you stupid?

DIGITS will save us

>>104003499
he's not wrong

>>104003440
you aren't running the full deepseek at home without tons of ram or video cards. set your standards lower to 70-123b

>>104003502
but I want to run it locally, so what can I run with 24GB of vram?

>>104002411
You are being logged though.

>>104003503
I, too, can't wait to run Deepseek R1 at Q2 for a cheap $7k after taxes

>>104003516

If its just Chink glowies doing the logging, does it matter? Digits can't come soon enough. ffs

>>104003503
DIGITS are a scam. You can only link 2 for 256GB, which can't even run DSV3 but still costs 6k. 4 DIGITS would cost 12k, still can only run R1 at Q3, and it'll be slow as shit having to run it over RPC.

>>104003541
4x DIGITS is enough for Q5 though. That'd be decent.

>>104003511
nothing that will give you decent output. that's what we keep telling you but you keep asking to be told it will work anyway.

>>104003552
>nothing that will give you decent output.
how do you know? also what "decent output"?
I run >Mistral-Small-Instruct-2409-Q5_K_L.gguf
locally and its good enough for what I need it for.
and I'm asking if there is something better now than that that I can run on my hardware.

>>104003551
Not if you want any usable context size.

>>104003541
What other option is there then? I get what you're saying but consider pic related, which will 100% be over $10k

>>104003577
Buy 15 of your picrel. You're not poor, are you?

>>104003577
CPU build. Easy 1TB RAM for less than $8k.

>>104003566
>mistral small instruct
okay, it's all making sense
I'm happy for you, bud. If you're satisfied with... that, just keep using it.

>>104003591
I would probably an hero if my cunny RP machine somehow costs more than my house..

Has anyone gotten the distills to even obey their format after long chats? Like, if I start it in the middle of an RP after 20 turns, it doesn't even start with <think> any more. Instead it starts thinking halfway through its response and starts thinking in character as {{user}} instead of as an AI lmao. Talking about the 32b distill. It's pretty funny sometimes but I swear the format is correct as documented, or does it only work when you leave <think> blocks in from previous turns?
It seems to work okay when starting a new RP but holy god is it long-winded.

>>104003626
but I want deepsuck, or something thats better.

>>104003654
no way thats fully optimized yet

>>104002396
Anyone?

>>104003459
I tried it and didn't like it.

>>104003654
>2.5t/s running off 96GB VRAM and full DDR5
Where are the fags who thought they'd be getting 5t/s off an SSD right now? The ones coping about fast inference with DDR4?

Does AI really need to solve PhD problems to be sentient? AI companies spend billions and consume power plants amount of energy to try to be better than highly skilled humans, but is that really necessary for at least a decent companion? Who the fuck here makes a potential girlfriend do leetcode or write proofs to judge how good she is as a mate?
>Babe if you can't autocomplete my code then gtfo
My point is if we really wanted to make androids who talk and act just like real humans all these parameters being thrown at leetcode is just useless what we need is emotional intelligence, long term memory, 1-2hour long chains of conversation as opposed to terabytes of github source code. Since when did being human = solving tasks only 0.01% of the population can do???

if intel doesn't sell a b-series card with enough ram to run r1, it's time they went broke.

>>104003674
It needs to be able to write jokes.

I have yet to see ai invent a good mosquito joke.

>>104003674
bitches who think ants are real dont think ai is sentient even tho ai is smarter than ants

>>104003421
Even if something is not outright illegal if it's a potential grey area they can use it to get a warrant/harass you looking for other shit.

>>104003577
Real answer: Use R1 API since it's very cheap
Buy 5090 for Stable Didfusion

>>104003673
>>104003664
I'll be glad to hear any methods to make it faster. I choose to blame llama.cpp and wait for ktransformers. In the meantime, Q3KM has no issues as far as I can tell. It is unhinged and kino. That said I have not tried the API to compare.

>>104003673
does llama.cpp fully support everything with it? Is it even running it like a moe because 20B changing params should be many times faster just like wizard would be

>>104003673
I thought DDR5 got 10 t/s?

>>104003654
Nice, 2.6 t/s would be totally usable for me. I'm fine with anything over 2.

Ok I see

https://github.com/ggerganov/llama.cpp/issues/11333

Not everything is ready yet for deepseek

>>104003654
Prompt eval is STILL 28tok/s??? Even with quad 3090s? That means itll take forever to respond to nontrivial context sizes. Across all of lmg, locallama, and github I have yet to find a single cpumaxxer with decent prompt processing speeds or >10t/s text gen.
Unless ktransformers or some new inferencing engine comes out to speed up deepseek r1/v3. I'm gonna save my money.

>>104003772
>>104003762
>>104003743
>>104003673
It continues to rapidly degrade with more context. At 0 it is almost 5t/s. This is at 7k. At 16k I bet it is 1t/s but can not test because OOM.

>>104003743
r1 is 37b activated

And a bug as well?
https://github.com/ggerganov/llama.cpp/issues/11163

Yea, deepseek is not fully supported yet.

>>104003654
Thanks for the hard numbers, anon.
Was that running 671b at q8?

I have 4*3090s. Was maybe considering getting a couple more but would need a new motherboard to do so.
Was looking into ddr3 and ddr4 platforms because cheap. Might reconsider.

>>104003814
yes, but 17B always stay activated. About 20B are what change per token

>>104003811
>At 16k I bet it is 1t/s
I'm so glad I saw this before I sunk money into a DDR5 build

>>104003811
no coomer has ever needed more than 8k context
>but I—
nope. 8k is all you need.

did the chinks win?
I don't visit lmg much anymore and gave up on open source. Is deepseek actually cracked?

>>104003834
>Is deepseek actually cracked?
can you translate this from zoomer into english?

>>104003833
Deepseek will use 2k of that on thinking in one turn. Oh, and you get to wait for all that to generate too.

Potential Speed Improvements for DeepSeek V3 Inference via NUMA-Aware MoE Allocation

Deploying DeepSeek V3 or similar large MoE models with NUMA-aware expert allocation could yield significant performance gains through these mechanisms:

1. Memory Access Optimization
Localized Memory Access

Storing experts on local NUMA nodes reduces cross-node latency (typical NUMA latency varies 2-5× between local/remote accesses).
For CPU-bound inference scenarios, this could improve token generation speed by 10-25%.
Cache Utilization
Thread-core affinity binding improves L3 cache reuse:

If 30% of expert computations rely on cached data, this may reduce cache miss penalties by 15-30%.
2. Enhanced Compute Parallelism
Expert-Level Parallelism
For DeepSeek V3’s sparsely activated experts (e.g., 2/8 experts per token):

Co-locating experts on the same NUMA node reduces contention for memory bandwidth.
On dual-socket EPYC servers, this could achieve 20-40% throughput gains (aligned with GSPMD experiments).
Load Balancing
Dynamic routing (e.g., Switch Transformer) may cause uneven expert utilization:

NUMA-aware scheduling can redistribute hotspot experts to idle nodes, reducing tail latency.
3. Reduced Communication Overhead
Inter-Layer Data Transfer
All-to-All communication between MoE layers may consume 15% of inference time in cross-NUMA setups:
Prioritizing intra-node routing could cut cross-NUMA traffic by 30-50% (per TensorFlow NUMA guidance).

Thats one thing for sure.

>>104003853
You can just remove the thinking portion of a response from the context once it's finished
I expect the interfaces will add a way to do that automatically once we've had thinking models for longer

>>104003840
I thought you guys were zoomers so I used zoomer slang
is deepchink actually that good

>>104003863
You can do that with ST already.

>>104003853
Having to wait for the thinking to finish first is a huge problem though, you're right about that
2 t/s isn't acceptable on a thinking model the way it would be for a regular one since you're looking at potentially multiple minutes before the 'real' response even starts

https://github.com/DAMO-NLP-SG/VideoLLaMA3
https://arxiv.org/abs/2501.13106

>>104003870
nta but where is that option?
ST is visually hiding the stuff between <think></think> for me, but I think it's still including it in the submitted context

Any leaks or rumors about llama 4? Did they give any hint about the ETA? Was the development halted because of the lawsuits?

>>104003807
I'm not familiar with how llama.cpp works, but as far as I can tell, before tokens are generated, which I assume is the prompt eval period, I get 100% CPU usage on a single core, and about 125W usage on a single 3090, the other GPUs seem to be idle.
>>104003818
This is 671b at q3km with q5 context. Context is absurdly expensive.
>>104003834
Despite everything, it is cracked. I have 0 regrets.

>>104003869
>chink
sir, we do not tolerate that sort of slur here anymore
deepchina beat all the western corpo models and single handedly saved local

>>104003834
>>104003869
Yes but no one can run it local.
We are all in the R1 lite (the smaller model) waiting room

>>104003811
I have an idea to try to speed this up....I haven't even researched MoEs fully yet but if (thereotically) at least 30% of the entire model was used (at any point) as the active parameters across 95% of the most common erpg scenarios (think nala test, etc etc), then with 96gb vram you could intelligently store only these specific params, and use the cpu for inferencing the remaining 5% of cache misses. Has this been done before? Basically we make or find a dataset of erpg prompts, see what parameters keeps being lit up across these prompts, and if we can find at least 30% or less of the model's neurons activating on erpg stuff them bingo we put that shit on vram before the real inferencing begins. From then on you enjoy a massive speedup.

>>104003884
Regex. Maybe you're thinking of filters that just alter the display, but you can have it alter the chat itself, and set a start depth so it only does the previous ones. You can set it to keep it in the chat but not send them too if you want to be able to see but not send them.

>>104003925
(Me) : I basically want to make ktransformers but coomer version. They did optimizations that were coomer agnostic/good for general case and that works, but I want to take it even more extreme and optimize specifically for the cooming neurons.

>>104001926
So are these anons using a frontend for REAL r1 or is this just another "distilled r1"

>>104003811
That's a bummer. I was thinking of swapping my two A6000s into my other server with 256GB DDR4 and see how fast it goes but I think I'll wait until things are more optimized if it's this bad with DDR5.
At least there's stuff like multi-token prediction and maybe some optimizations when it comes to context on the horizon.

>>104003956
When anons talk about R1 it's a coinflip whether they're talking about the API version or the 14b

>>104003946
I don't see why you need to precompute what gets lit up in common scenarios. We already have algorithms for real-time cache tiering across many applications in IT from block storage to CDN.

>>104001303
>>104001570
I'm sorry, but what the hell is this? I swear I did everything as told here, but something is clearly missing. It always starts rambling off it's train of thought before giving me what I want.

>>104003983
Precomputing is useful to avoid having to do it every single session if all youre doing is more or less the same coom scenario.

>>104003999
that's what R1 and its distillations are supposed to do yeah, it's a reasoning model
if you don't like that don't use R1, stick with Nemo or Cydonia 22B or something

qwen 32B R1 is great, just needs the usual prefill

>>104004012
I mean, that's kinda cool, but why did anon recommend it to me then?

>>104004022
It's the latest overhyped trash that people will forget about by next week

GANQ: GPU-Adaptive Non-Uniform Quantization for Large Language Models
https://arxiv.org/abs/2501.12956
>Large Language Models (LLMs) face significant deployment challenges due to their substantial resource requirements. While low-bit quantized weights can reduce memory usage and improve inference efficiency, current hardware lacks native support for mixed-precision General Matrix Multiplication (mpGEMM), resulting in inefficient dequantization-based implementations. Moreover, uniform quantization methods often fail to capture weight distributions adequately, leading to performance degradation. We propose GANQ (GPU-Adaptive Non-Uniform Quantization), a layer-wise post-training non-uniform quantization framework optimized for hardware-efficient lookup table-based mpGEMM. GANQ achieves superior quantization performance by utilizing a training-free, GPU-adaptive optimization algorithm to efficiently reduce layer-wise quantization errors. Extensive experiments demonstrate GANQ's ability to reduce the perplexity gap from the FP16 baseline compared to state-of-the-art methods for both 3-bit and 4-bit quantization. Furthermore, when deployed on a single NVIDIA RTX 4090 GPU, GANQ's quantized models achieve up to 2.57× speedup over the baseline, advancing memory and inference efficiency in LLM deployment.
posting for johannes

Every week China releases a new SOTA model for something (llm, video, whateverthefuck) and the west does nothing. Even cosmos was a flop

>>104004038
Goddamnit so my original guess about it being the latest meme was right.
What should I fuck around instead then, Cydonia I guess? Been using Mistral Small Inst the last few weeks, tried Nemo but stuck with Mistral.
>>104004052
Flood the market with mass produced shit, for better or worse.

>>104003999
You let it get past the rambling and then it turns kino. Simple really.
Personally, I delete the ramblings a couple turns later.

>>104004048
I want this but coomer version. DELETE ALL NON HORNY NEURONS. I don't need my wife to code, she needs to talk about vidya with me!

>>104004061
I doesn't seem bad, but having to deal with this waste of letters every message seems a tat fucking much, I'm not that desperate.

>>104004048
I'm retarded but isn't this basically what exl2 is doing?

>>104003687
This is the ultimate benchmark

>>104003687
True human creativity on display right there.

Fuck math or logic puzzles, gimme a good joke!

>>104004063
With that paper about the correlation of skills between experts in MoE, removing coding skills from your waifu would pretty much lobotomize her logic skills which might be what you're looking for, but, you never know. Just be sure not to make your waifu too much of a retard, anon.

>>104004118
That is true....TITANS actually referenced this issue and found a way to fix it so maybe I'll have to wait until then.

>>104003064
So far 350GB or so. No joke

>>104003064
R1 is at 700GB q8. You need 350GB to run at q4 and 170GB to run at q2 (very retarded)

>>104004171
I've run 70b models at Q2 before, it's fiiiiiiiiiiiiiiine... I'm sure a model as big as R1 would be alright.

reading the CoT chains for silly rp content is absurdly funny and cool
often more entertaining than the RP itself

>>104004252
>Let's go with hairy
INSANELY FUCKING BASED
I will give you one thing, it's surprisingly funny to read the thoughts of a mad man like this.