Origin Stories, RAG Rants, and the $40K Home AI Rig

Tim Williams: Hello and welcome to another Rubber Duck Radio I am your host with the most Tim Williams, and here with me again Is Paul Mason. Paul Mason: Hey Tim, and hey audience. Ok, at this point though can we call me a host too? I feel like I've earned the title by this point. Tim Williams: I don't disagree, I think a solo show with just me might bore people. Anyway, now that we're a few episodes in I thought it'd be a good idea to give the audience a little more back story. Paul Mason: I guess that makes sense, after all, why listen to us, right? Tim Williams: Exactly. I'll go first. My title is 'lead developer,' but I guess what I do is functionally similar to what other companies call 'staff developer.' Anyway, I started my software development journey around 2010. Tim Williams: I decided to make a career change because I was caught up in the 2008 financial crisis. At the time I was a manager for Starbucks and basically had a dead end retail job. Tim Williams: It paid fine, but I hated it. I'm an analytical person, a problem solver, so I knew I could do a lot more, and probably be a lot more fulfilled with my life if I did something that used those strengths. Tim Williams: So. I Decided to go back to school for computer science. The thing is, I was already really passionate about this idea. My girlfriend at the time was sidelining as a photographer. Tim Williams: She was never super serious about it, so we couldn't justify paying someone to make a website for her. I thought "I can do this." Then I just set about learning. Tim Williams: So while I was taking classes I was burning the candle at both ends. I built her a website, and I had a TON of fun doing it. That's when I knew I could do this as a career and that gave me enough juice to make it happen. Tim Williams: So from that I decided I could help fund school with this new skill, and that was the decision that really got things moving. Tim Williams: I started making small static HTML websites for some local clients. I was charging next to nothing, but it was still more money than I was pulling in from Starbucks. Tim Williams: Then I found a mentor. Haven't talked to him in years, but I bet he's doing awesome work still. He was running his own development agency at the time he called 'The Web Hounds.' Tim Williams: I was able to leverage his client work to get involved in some more serious larger applications. This is where the schooling and practical application really helped me get to another level. Tim Williams: At that time we were helping a huge range of different clients. I got to work on a big project for EA Games, a large website project for a local farmers organization. A full e-commerce system for a local wine bar. Tim Williams: I'll tell you something, as a contract developer you really end up touching a lot of technologies. To me, no technology was off the plate. Tim Williams: If it was already part of the stack and I had no experience in it, I'd go out and teach myself enough to start applying it, then learn it through practical application. Tim Williams: I was learning the real technology used in the field. MySQL, jQuery, raw JavaScript, Apache, WordPress. Those were the simple days. Tim Williams: This will date me a little, but back then we were using SVN as our source management solution. Man, now I completely take Git for granted, we have it so easy now. Tim Williams: Anyway, eventually I got hired by a local business. Not a tech company exactly, I'd say the name, but most people wouldn't recognize it. It was my next taste of "this is the right path for me." Tim Williams: I kept the we dev agency going, but not really taking on many new clients because my day job had me learning all kinds of new tech. Tim Williams: My first day there my body plopped a 200 page manual on my desk and said something like "this is Clickability. This is the CMS solution the company uses, good luck!" Tim Williams: Anyway, I'm still with this company but we've grown into a much more mature development shop than we were back in the day. One thing that hasn't changed is we still use a huge range of technologies to accomplish everything we need. Tim Williams: Nowadays I'm using Cursor and working in AWS stacks. Full continuous integration pipelines from scratch. Building native applications and novel solutions for kiosk systems for my company's onsite events. Tim Williams: I am working heavily with AI, and heavily implementing AI for various systems. Custom RAG systems, custom MCP servers, sentiment analysis, custom customer management solutions. Tim Williams: I still sideline on the weekends every once in a while also. I just recently had the opportunity to build a really cool AI driven therapy tool for one of my long time clients. Tim Williams: Anyway, to the point, I am a technology generalist. I'm mostly based in web tech, but never afraid to pick up new technologies when they are the right solution for the problem. A true 'full stack developer' if you will. Paul Mason: Well, that was exhaustive. Some things in there I didn't know. It's my turn I guess. Tim Williams: Feel free to be as wordy as me. Paul Mason: I feel like my story is a little less interesting. I get to touch a lot of the same types of technology. I got my CS degree back in 2015 so I got into the field at a fantastic time. Paul Mason: I didn't do agency work directly like you did, but I did work for a small web development agency for a couple years to get my start. We did mainly custom WordPress solutions. Paul Mason: Much like you though, we took on clients with diverse solutions already in place so I was able to dabble in dot net, custom Salesforce development and all kinds of crazy things. Paul Mason: Not too long after that start I was picked up by one of the big companies, and that's where I'm at. Not sure I'm allowed to disclose who I work for on a podcast like this without approval, but let's just say it's one of the FAANG companies. Tim Williams: I think that'll be enough for folks, what kinds of projects are you working on these days? Paul Mason: Similar to you I'm on a pretty small team, but we're doing a lot of custom AI integration work for partners. Tim Williams: I guess that does position us as decent voices on the software development and AI space, don't you agree? Paul Mason: Yeah, absolutely. I think a lot of developers especially Senior and staff developers are learning a lot of the same things right now. The landscape of software development with AI is still in its infancy, though it feels like it's radically different already. Tim Williams: I know. Last year I was surprised how good my autocompletes were with Github Co-Pilot, and this year I am embracing agentic work. I feel like my understanding of what AI is and how to use it has increased exponentially. Paul Mason: It has to. To keep up with this industry, you've gotta be extremely flexible and adaptive. Even then you might end up one of the fourteen thousand laid off AWS workers. Tim Williams: tell me about it. What a nightmare. We went from an extreme hiring boom in 2020, to a slumping market as it corrected downwards for the next couple of years, to 2025 where the extreme voices are saying Software Developer will be extinct by the end of the year. Paul Mason: When you put it in that context, we have a pretty middling opinion about it. I'm a little more skeptical of AI than you, but I can see it for what it is, a pretty useful new tool. Tim Williams: Exactly. AI is a step change to how we approach the work, but it's not completely different than the innovations like the IDE or frameworks. In fact, I would say open source frameworks have had a MUCH larger impact on the industry than AI has to date. Paul Mason: I agree. It's hard to estimate, but open source tools and frameworks underpin a massive amount of what we do, and absolutely make up the foundation of the web as we know it today. Tim Williams: Think about this. Without all of the code open source has made available, and specifically, all of the well written, "peer reviewed" code if you will. Would training AI models on code have even been possible? Paul Mason: I guess we're not close enough to the source to know the answer to that, but I would guess you're right. Where else can you find such a huge trove of quality training data? Tim Williams: So this actually tees up something I've been thinking a lot about lately, and I'm curious where you land on it. I'm starting to feel like AI is basically as smart as it needs to be. Like… we're past the point where "smarter" is the thing that matters. Tim Williams: What's next is enabled AI — where the strength isn't the raw intelligence, it's the interaction layer. MCP servers, specialized models, tighter toolchains. We don't need the AI to be a genius; we need it to be precise. Paul Mason: Okay, but hold on, because I know what you mean, and I even agree to an extent, but you can't just hand-wave the raw model quality. Paul Mason: If you look at the AI benchmark results — the LMSys arena, the GAIA stuff — generalist models still absolutely wipe the floor with the "specialists." The big frontier models are the ones making the real breakthroughs. So if anything, I'd argue the opposite: we need more intelligence, not less. Tim Williams: See, I don't think so. I think we've hit the "human-level ceiling," but not in the sci-fi way — in the practical way. The models are now "smart enough" that the bottleneck is us. It's our tools, our context windows, our interfaces, our orchestration layers. Tim Williams: When I build something with AI today, the model isn't the problem. The problem is the glue code. The memory. The tooling. Honestly, the biggest breakthroughs I see are from good MCP servers, not from adding 500B more parameters. Tim Williams: Have you seen how good GPT-OSS 120b is? I can run that thing on my laptop and I swear it codes better than the frontier models of a year ago. I built a couple MCPs for it and now I rarely need to use ChatGPT. Paul Mason: But Tim, you say that while also using the smartest models you can get your hands on. Every day. You're not doing your RFP agent on a 7B. You're not writing your therapy system on a tiny instruction model. Paul Mason: You use the big ones because they're better. And that matters. Because when the generalist model gets better, everything downstream — your tools, your MCP calls, your RAG — all of it levels up. Tim Williams: Sure, but only to a point. What I'm saying is: we're hitting diminishing returns on raw IQ. And the reason is simple — your LLM doesn't live in a vacuum. It has to do work. Tim Williams: It has to touch APIs, transform data, interpret structure, handle edge cases. It's the same pattern we saw with programming languages. People think language X replaced language Y, but that's not what happened. They converged. Same thing here: the models are converging on "smart enough to reason effectively." That's the baseline. But the future? The future is infrastructure. Paul Mason: Infrastructure matters, yeah, but the generalist models are the ones that learn how to use the infrastructure. Look at the recent agent benchmarks. Tim Williams: Oh, come on Paul, you know they game those benchmarks. Paul Mason: Yeah, butwhen they test tool use, the biggest models still dominate. Even if you had the perfect MCP servers, a tiny model isn't going to magically become an expert surgeon just because you gave it access to a medical tools API. The intelligence still matters. Tim Williams: I'm not saying "tiny model everywhere." I'm saying "specialized model where it counts." Look at what happened with GPUs — we don't use one giant card for everything. We use accelerators for specific operations. Same thing will happen with models. Tim Williams: Give me a moderately-sized generalist to plan, and then route to a specialist that knows exactly how to analyze an ffmpeg pipeline, or exactly how to restructure a SQL schema, or exactly how to draft a secure authentication flow. Then glue that all together with tools that are permissioned, observable, and tightly scoped. Tim Williams: Imagine how much less compute power that would use? Now you've optimized the quality AND the compute requirements. Paul Mason: But now you've increased the coordination complexity. You've traded raw intelligence for orchestration overhead. And historically — in distributed systems, in computing in general — more moving parts means more failure points. If one giant model can do the planning and the execution, isn't that simpler? Tim Williams: Only until the bill shows up. One frontier-sized model doing everything is like hiring a PhD to load the dishwasher. Sure, they can do it, but why? You don't want a generalist performing tasks that a specialist can do more efficiently, more safely, and more predictably. Tim Williams: Frontier models will always exist, but they'll be like the "root brain." Most real work will be done by small, cheap, highly-trained executors — all given context through tools, memories, and MCP layers. Paul Mason: I don't know, man. The benchmarks don't agree with you yet. When they put models head-to-head — planning tasks, coding tasks, reasoning tasks — the generalists are still way ahead. It's not even close. If anything, specialized models tend to collapse outside their narrow training set. Tim Williams: Yeah, today. But look at early GPUs — the benchmarks didn't predict CUDA either. Or early web frameworks — nobody predicted that JavaScript would eat the entire industry. The shift doesn't show up in benchmark suites. Tim Williams: It shows up in developer workflow. And right now the workflow is screaming for determinism, for precision, for guardrails, for reproducibility. Raw LLM horsepower doesn't fix any of that. But tool-enabled AI does. Paul Mason: So your take is basically: intelligence plateau, tools skyrocket? Tim Williams: Exactly. We don't need smarter models to build better software. We need smarter systems that use models. That's the future. Paul Mason: Alright, well — I'll take the devil's advocate seat for now, but this feels like one of those arguments where we revisit it five years from now and either you look like a prophet or I look like a dinosaur. Tim Williams: So either way, a pretty typical Tuesday. Paul Mason: Fair enough. Tim Williams: Anyway, neither of us can tell the future. What I do know is that AI is here to stay. I can't wait for the hype to die down though so I can start to focus on the real value without all the noise. Paul Mason: You and I are one hundred percent aligned on that one. You know that meme of the guy poking the thing with a stick? I could see that with the caption "c'mon OpenAI, burst already" Tim Williams: We must be in the same Reddit circles, I can see the exact image in my head. Side note, I had this interesting idea and I wanted to bounce it off of you. Paul Mason: Ok shoot. Tim Williams: This one occurred to me as I was building a RAG dataset for one of my little side projects. There are certain datasets that are largely static. Meaning, you can basically curate them and distribute them. Tim Williams: If we came up with a standard schema, you could use that with SQLite's pretty darn powerful ability as a single file, self contained vector database with the source data's metadata alongside it. Paul Mason: Oh, I think this is a brilliant idea. Tim Williams: Essentially you could make portable knowledge bases. With a standardized MCP you could just swap in a knowledgeable and have a local vector powered tool you could plug into any AI system. Paul Mason: I think you could use that for all kinds of things. Going back to your example, you could make small knowledge bases for things like frameworks. Tim Williams: I didn't even think about that. My use case its a database of falconry knowledge. It's pretty obscure, but will be super useful for people who just have simple questions and want to be pointed in the right direction. Paul Mason: I do think that idea could be powerful. Based on the data you've curated for your dataset you could fine tune the vector overlap, size, quality to match the source material. You'll likely have better results than a generalized RAG system. Tim Williams: Yeah, exactly. And I think that's where this whole RAG conversation gets kind of muddled, because "RAG" has become the junk drawer term for anything that involves documents and an embedding model. Tim Williams: Like, when somebody says "we're doing RAG," nine times out of ten they just mean "we glued a vector DB to an LLM and called it a day." Paul Mason: Yeah, classic RAG is basically "LLM-powered search with citations." You chunk some documents, embed them, do a cosine dance, stuff the top five chunks into the prompt and ask the model to play librarian. Paul Mason: Which is fine, that solves a real problem, but that's the shallowest version of what retrieval can do. Tim Williams: Right. And the problem is that people use the same word "RAG" for that simple search use case and for these super complex orchestrated systems with multi-step reasoning, tool calls, long-term memory, all of it. Tim Williams: So when one team says "RAG didn't work for us," what they usually mean is "we tried naive search and pray" — not that the entire paradigm is dead. Paul Mason: Exactly. So maybe the issue isn't RAG itself, it's that we've overloaded the term to the point of uselessness. It's like calling everything with a lambda "microservices." Paul Mason: There are a bunch of different problems hiding under the same umbrella and they behave really differently under stress. Tim Williams: Yeah, maybe the way to think about it is: traditional RAG is that first bucket — question-in, relevant docs out, one-shot answer. Super useful for FAQs, policy docs, manuals, developer docs. Tim Williams: But once you start doing orchestration — planning multiple calls, refining queries, updating state over time — you're not just doing "RAG," you're building an actual system with retrieval as one of the core primitives. Paul Mason: And even within retrieval you have different jobs. There's "help me search this big static corpus," there's "remind me what I said last week," there's "pull the right ticket or RFP that this email is about." Paul Mason: Those are all solved with some combination of embeddings and metadata lookup, but they are radically different from a product and architecture standpoint. Tim Williams: Totally. I'd almost split it, like, first category is what you just described: grounded search. That's the classic "I have a knowledge base, answer questions against it, don't hallucinate." Tim Williams: Then you've got working memory — the short-term stuff that lives for a session or a task, where retrieval is more like giving the model a scratchpad than like indexing a wiki. Paul Mason: And then there's long-term memory, which I think people also incorrectly bucket under RAG. That's stuff like "Tim prefers dark mode, hates long emails, and is terrified of surprise AWS bills." Paul Mason: That's not the same as "retrieve subsection 4.2.1 of the S3 pricing docs." It might use some of the same machinery, but conceptually it's a different problem. Tim Williams: Yeah, and your little falconry DB is yet another flavor. That's what I'd call a portable domain corpus. It's mostly static, super curated, and can be shipped as a file alongside the app. Tim Williams: That's a wildly different design constraint from, say, a support bot pulling from a constantly changing Notion workspace. Paul Mason: So maybe instead of saying "we do RAG," we should be forced to answer two questions. One: am I doing search over static-ish reference data, or am I doing dynamic state retrieval? Two: am I just grounding answers, or am I orchestrating multi-step work? Paul Mason: Because once you admit you're orchestrating, you're in agent land whether you like that term or not. Tim Williams: Yeah, and orchestration is where it gets interesting. Once you let the model say, "hey, my first retrieval pass wasn't good enough, let me refine the query," or "let me summarize these results and store a new artifact," you've gone way past the "LLM plus vector DB" diagram we've all seen a thousand times. Tim Williams: You're effectively building a tiny OS for knowledge instead of a fancy search bar. Paul Mason: And that's where the term RAG really starts to fall apart, because it collapses that entire stack into one acronym. No wonder half the industry thinks it's overrated; they're arguing about completely different things under the same label. Paul Mason: One person is complaining that "RAG doesn't handle reasoning," and the other is like "my orchestrated RAG stack runs our entire internal documentation system." Tim Williams: Exactly. It's like if we used one word for "SQLite on your laptop" and "Google Spanner cluster" and then argued about whether "databases scale." The answer is "it depends what you actually built." Tim Williams: RAG as a term is trying to describe everything from "I added a context window to my chatbot" to "we built a multi-agent retrieval pipeline with feedback loops." Paul Mason: So what's the fix, do you think? Because we're not going to get the whole industry to stop saying RAG — that ship has sailed. Paul Mason: But there's probably some discipline we can bring to how we, as developers, talk about it. Tim Williams: I think, at minimum, we should stop treating "RAG" as an architecture and treat it as a capability. The architecture is: retrieval layer, memory layer, tools layer, orchestration layer. Tim Williams: RAG is just one way the retrieval layer gets filled — embeddings and similarity search — instead of, say, purely symbolic queries or direct SQL. Paul Mason: Yeah, that's a good way to put it. RAG is a verb, not a noun. "We rag against this corpus as part of our workflow," not "we built The RAG trademark symbol." Paul Mason: And then on top of that you call out, okay, this system has: short-term scratchpad, long-term memory, static knowledge base, and a planner that knows how to bounce between them. Tim Williams: Exactly. And once you frame it that way, your portable knowledge bases make way more sense. They're not "a RAG system" on their own — they're plug-in retrieval substrates that any orchestrated system can load. Tim Williams: The MCP is just the adapter that says "hey model, here's how you talk to this particular blob of curated knowledge." Paul Mason: Yeah, and that's where I think the future gets exciting. Not just "RAG for your docs," but an ecosystem of swappable, high-quality corpora with well-defined access patterns. Paul Mason: At that point, arguing about whether RAG "works" is like arguing whether "APIs" work. It's the wrong level of abstraction. Tim Williams: Exactly. So, to summarize the rant: RAG as a term is trying to do too much. "LLM-powered search" is one tiny slice of the pie, and orchestration is where all the real leverage lives. Tim Williams: If you're building with AI in 2025, stop asking "should I use RAG?" and start asking "what kinds of retrieval and memory does this system actually need?" Paul Mason: And maybe, just maybe, stop shipping the LangChain quick-start example to production and calling it a platform. Tim Williams: Yeah, that too. Tim Williams: So speaking of swapping in local knowledge and running your own corpora… that reminds me of the research we did the other day on home AI rigs. Because man, that rabbit hole got expensive fast. Tim Williams: I swear, I went into it thinking, "Alright, maybe five, six grand," and suddenly I'm pricing out dual RTX 6000 Pro Blackwells and 256 gigs of RAM like I'm outfitting a small research lab. Paul Mason: Yeah, dude, you basically specced a personal data center. Did you hear about PewdiePie's 'council' rig with a bunch of 4090s? Tim Williams: I saw an article about that. I've seen a bunch of people building similar things, and wow, collectively they're pushing up the price of components to an insane high. Paul Mason: I remember you sent me that part list and I thought you were joking. Two RTX 6000 Pros? Like… those are literal enterprise GPUs for post-production studios and ML labs, not "Tim's Weekend Science Fun Kit." Tim Williams: Look, in my defense, I thought maybe the prices would come down. They didn't. Tim Williams: For context for anyone listening: one RTX 6000 Pro Blackwell is around — what — eight grand? Nine grand? And that's per card. So dual GPUs alone are pushing twenty grand. Paul Mason: And that's just the GPUs. Then you needed a motherboard that could actually power them without melting, which is another thousand. Paul Mason: And 256 gigabytes of DDR5 ECC RAM — that's like two grand right there. Then the power supply that could run a small village. And don't forget the case that looks like a monolith from 2001: A Space Odyssey just to house all that cooling. Tim Williams: Yeah, yeah, yeah. And the kicker? Tim Williams: After we put all that together, we found a prebuilt workstation with almost the exact same specs from one of the AI server vendors… and it cost more than the DIY build by like ten grand. Tim Williams: So suddenly my "unhinged" build was actually the economical option. Paul Mason: "Economical option," he says, while holding a $40,000 shopping cart. Paul Mason: Meanwhile I'm over here running Qwen on my MacBook like, "This is fine." Tim Williams: Hey, hey — to be fair, if you want to run big models locally — not 7Bs, not 14Bs, but those 120B-class open models like GPT-OSS 120B or the new Kimi models — you actually do need serious hardware. The MacBook is great for mid-range stuff, but it's not going to chew through a 120B in real time. Paul Mason: No, absolutely. And that's where the cost breakdown gets interesting. Because once we actually listed the numbers out, you realized something important: Paul Mason: If you want to play in the frontier-model tier at home, you're entering "used Tesla" pricing. Tim Williams: Yeah, that's the line. Tim Williams: Running 7B or 14B? Any modern PC or a Mac Studio will do. Tim Williams: Running 70B? Now you need at least a 4090 or a 6000 Ada. Tim Williams: Running 120B? Now you're in multi-GPU territory. Tim Williams: Running full-context 200B+? You're basically competing with cloud inference farms. At that point you're going to have an electrician install a second breaker box. Paul Mason: Which, by the way, you actually had to do. Paul Mason: That was maybe the funniest moment of the whole thing — when you realized your home office wiring physically couldn't run two 600-watt GPUs and an AI-grade CPU at the same time without tripping the breaker. Tim Williams: Listen, the guy who installed my EV charger didn't judge me. He just said, "You're doing a home AI project? You're not the first one." Tim Williams: Which honestly scared me more. That means there are other lunatics out there melting drywall for local inference. Paul Mason: This is where I'll defend cloud inference for a second. People really underestimate how cheap it actually is at scale. Paul Mason: Like, if you're running occasional jobs, or even steady workloads for a small project, paying OpenAI or Anthropic or NVIDIA for API access might cost you 200 bucks a month. That's thirty years of API calls before you hit your GPU rig's price tag. Tim Williams: For sure. The economics only make sense if you're doing: Tim Williams: — high volume inference Tim Williams: — with large models Tim Williams: — consistently Tim Williams: — AND you don't want rate limits Tim Williams: — AND you're optimizing for total cost over time Tim Williams: — AND you enjoy the smell of toasted PCIe slots Paul Mason: And let's be honest, you do. Tim Williams: Look, it's a lifestyle. Paul Mason: But the real twist — and this is what surprised even me — is that running your own hardware isn't really about saving money. Paul Mason: It's about freedom. Paul Mason: No rate limits, no sharing GPUs with a thousand other people, no waiting for API providers to support new architectures or longer contexts or new tool chains. Tim Williams: Yep, that's exactly it. If you want experimental features, weird local workflows, custom fine-tunes, MCP servers running directly against a giant model without any API intermediary — local wins. Tim Williams: But if you're just building apps for users? Cloud wins. 99% of the time. Paul Mason: So basically the same rule as homelab culture: Paul Mason: No one builds a Kubernetes cluster in their garage because it's cheaper. Tim Williams: They build it because it's fun. Paul Mason: And because deep down, every developer wants to feel like Tony Stark booting up a server rack that glows ominously in the dark. Tim Williams: Yeah. And if the lights flicker in the neighborhood, that's just part of the experience. Tim Williams: So maybe this is a good segue into something we both noticed while going down the hardware rabbit hole — and I think a lot of people don't realize this — consumer GPUs still absolutely annihilate enterprise GPUs when it comes to price-to-performance. Tim Williams: Like, the 4090 may be "just a gaming card," but for local AI, dollar for dollar, it's still the champ. Paul Mason: Yeah, and that's the irony, right? NVIDIA basically sells three tiers of the same silicon. There's the gaming card, the prosumer "creative workstation" card, and then the enterprise card with ECC memory and a warranty that lets you rack it in a data center without getting dirty looks from an auditor. Paul Mason: But compute-wise? Raw tensor throughput? The gaming cards punch way above their weight. Tim Williams: Exactly. A 4090 is what — two and a half grand? Maybe less now? And it runs a 70B model comfortably with quantization. Tim Williams: Meanwhile an enterprise card with slightly more VRAM costs ten times as much. Ten times. And doesn't necessarily run that 70B any "better," it just runs it with fewer memory errors and a more stable thermal profile. Paul Mason: And that's the thing — enterprise hardware is built for predictability, not speed. It's about uptime guarantees, memory correction, long-term thermals, validated drivers. Paul Mason: But if you're in your spare bedroom training a LoRA on your cat photos? You don't need your GPU to be blessed by the Church of SOC 2. Tim Williams: No, you don't. And that's why most home AI rigs are "hilariously overclocked gaming PCs." Tim Williams: If you're not running a 24/7 production cluster, the consumer cards give you 90% of the performance at 10% of the cost. Tim Williams: The only reason I ended up eyeing enterprise GPUs was because I wanted 128GB+ VRAM per card. Not because they were faster — because I wanted to brute-force context windows the way God and Sam Altman intended. Paul Mason: Yeah, frontier-scale context windows on local hardware is like deciding to build a Formula 1 engine out of Ikea furniture. You can do it, but you're really pushing the spirit of the product. Paul Mason: But seriously, if you benchmark consumer vs enterprise cards on actual inference workflows, the difference is usually like 5–10%. But the cost difference is thousands of dollars. Tim Williams: And I think that's the key message for people listening who are starting to experiment with local models: Tim Williams: Don't assume you need some monster enterprise GPU to run good LLMs. A single 4090 or 4080 Super can run most of the good open-weight models today with shockingly good performance. Tim Williams: Even the Mac Studio with the M4 Ultra can run mid-range models really well. Paul Mason: Yeah — and I say this all the time — the only time enterprise GPUs "make sense" at home is if you're: Paul Mason: A) doing extreme context lengths, Paul Mason: B) doing heavy fine-tuning, Paul Mason: C) experimenting with multiple big models simultaneously, Paul Mason: or D) you just enjoy the smell of burning money. Tim Williams: Or E) you're like me and you want to say the phrase "my inference cluster" unironically. Paul Mason: That too. Paul Mason: But seriously, for the typical developer or hobbyist, the consumer GPU path is still the sweet spot. Spend two grand, get world-class performance, skip the enterprise markup. Tim Williams: And honestly, the push from the open-weights community is accelerating that trend. These new 120B-ish models running on consumer cards? Two years ago that sounded like fantasy. Now it's a Tuesday. Paul Mason: Yeah, that's the part that blows my mind. The pace of optimization — quantization, speculative decoding, CUDA kernels — it's all moving faster than the hardware. Paul Mason: The software is improving so fast that consumer GPUs keep getting more capable without you spending another dime. Tim Williams: So yeah, for local inference: Tim Williams: Consumer cards = maximum chaos and performance Tim Williams: Enterprise cards = maximum stability and wallet pain Paul Mason: Put that on a T-shirt. Tim Williams: Alright, let's wrap this thing. Tim Williams: Today we covered a lot — our backgrounds, the whole "enabled AI vs frontier intelligence" debate, the overloaded meaning of RAG, portable knowledge bases, the cost of going full homelab scientist, and why gaming cards still reign supreme for local AI. Tim Williams: If you're listening to this and you're somewhere between "I want to run my own model" and "I just priced out a rack-mounted water-cooled monstrosity," just know: we've been there. It's a slippery slope. Paul Mason: Very slippery. Wallet-shaped slope. Paul Mason: But seriously, thanks for hanging out with us. We love doing these deep dives, and apparently we love oversharing our hardware mistakes. Paul Mason: If you have questions about building your own AI rig or you want us to do a "local AI starter kit" episode, let us know. Tim Williams: Yeah, we'll happily talk you out of — or into — spending way too much money. Tim Williams: Until next time, this has been Rubber Duck Radio. Go build something weird. Paul Mason: And maybe back up your breaker panel first. Tim Williams: Always.