15 comments

  • Grombobulous 59 minutes ago
    I’m curious to know if these local AI setups are legitimately useful compared to cloud. I’ve struggled a lot to get something useful out of the hardware I have.

    I realize I’m somewhat limited (16GB RTX 9070), but still, it seems really far off from the kind of experience even a basic $20/month subscription gets me.

    Any tips anyone might have are appreciated! I’d love to be local first and would be willing to buy hardware to get there.

    • D13Fd 3 minutes ago
      I’ve been running DeepSeek 4 Flash, Qwen 27B and Qwen 9B on local hardware. They work well for coding and document review tasks. I think Qwen 9B local on a 5090 might be legitimately helpful for small task agents in omp, since it is ridiculously fast. But my motivation is that I have data that I unfortunately can’t share with 3rd parties.

      I have been eyeing a 512 GB Mac 5 Ultra to run full DS4 pro locally, which I expect would be pretty amazing as far as quality/recall. The only downside is that the speed is a lot slower than something like 27B on the 5090.

    • Aurornis 56 minutes ago
      > it seems really far off from the kind of experience even a basic $20/month subscription gets me.

      The $20/month subs are much stronger than the local models you can run, even with how far local models have advanced lately.

      The appeal of local models is that the data never leaves your network so you can feel safer putting sensitive content into it. It also feels “free” to use when you’ve already paid for the hardware.

      But it doesn’t perform better and if you do the math you’re probably not saving money either. It’s helpful for things that you can’t or don’t want to outsource to a 3rd party.

      • seanmcdirmid 47 minutes ago
        Uncensored models are also popular reasons, although it’s more of a niche.
        • xnx 31 minutes ago
          Aren't those also commonly available from cloud providers?
          • digitalsushi 22 minutes ago
            i'd say 'with' instead of 'from', the cloud provider rents you the hardware, what you run on it, is whatever you show up with
      • BurningFrog 52 minutes ago
        It also takes some load off the AI data centers.

        IDK if that might be a concern for Apple or their AI partners.

    • jasode 8 minutes ago
      >I realize I’m somewhat limited (16GB RTX 9070), but still, it seems really far off from the kind of experience even a basic $20/month subscription gets me.

      I just ordered a new Mac Studio M5 Max 128GB $5899 ($6400 with tax) to be able to run a bigger "consumer size" model in the 70B parameter range (~96 GB). That said, I have no illusions that this expensive setup with a Qwen Flash coding LLM will be comparable to a $20/month subscription. Even upgrading to an even more expensive Mac Ultra 256GB for $10000 to hold a bigger model still won't be comparable. Apple hasn't shipped my Mac yet and I'm still considering cancelling it and downgrading to a smaller 64GB RAM config ($4299) to save $1600.

      Why did I initially spend the extra $1600 if I knew ahead of time that it wasn't as good as cloud AI? Because I thought I could use some local LLM for the easy tasks or when I hit cloud rate limits. No issues with privacy so that wasn't part of the motivation at all. I just wanted some local AI capability to augment a subscription. I've not totally convinced myself of the cost/benefit of this.

      Based on today's consumer hardware landscape, you're paying very high prices for crippled capability compared to the cloud AI subscriptions. We're also in a transition period where the next iteration of hardware improvements have some compelling features for local AI. Apple's upcoming M7 (2027 or 2028) is anticipated to have better GPU and neural engine to help with prefill TTFT. AMD Strix Halo is about to release 192GB system which is a big upgrade to their current 128GB ai pc. Maybe apply my $1600 savings towards those newer products. Those future products will still be very expensive but maybe the cost/benefit will be better.

    • adamtaylor_13 47 minutes ago
      The principle of KISS keeps coming to mind when I consider local computing. I'm looking forward to the day we can just run Opus-level models at 100 tok/sec on consumer hardware.

      But currently it's really hard to beat anything offered by the cloud companies. And the cost and complexity of setting it all up, just to barely (if at all) touch on Opus-level intelligence makes it seem like we're not quite there for the common man (enthusiasts are a different story.)

      I am very excited for open source local models, and we're nearly there, but it's still too complex and expensive to be my daily driver (yet).

    • julianlam 32 minutes ago
      16GB VRAM could load a small quantised qwen 27B model but it would be a ways away from a frontier cloud model.

      Though keep in mind not being beholden to shenanigans from said cloud companies (and interference from government entities!) is definitely worth something intangible.

    • paxys 55 minutes ago
      Local setups aren't going to make sense purely from a cost perspective, and definitely not when you are buying Apple hardware. AI subscriptions are too highly subsidized right now.
      • monatron 47 minutes ago
        I think your last point is exactly why I'm so interested in local models. The current landscape doesn't feel sustainable. The last few months we've seen the big providers (OpenAI, Anthropic) start to play with usage limits, resets, banked resets, pulling models, etc. I think local models are close to the point where, with a sufficiently well-architected harness, you can get results that are on par with the experience you'd have with cloud inference. It is nice to know that I have hardware under my desk that I control with open weight models that I can interact with on my terms.
        • htrp 23 minutes ago
          yes, you have to unfortunately adopt a 2025 mindset when working with smaller local models because that's where they are compared to the frontier.
        • julianlam 28 minutes ago
          > local models are close to the point where, with a sufficiently well-architected harness, you can get results that are on par with the experience you'd have with cloud inference

          In my opinion, 98% of the work most devs would send to an AI can be capably achieved with a local model and a frontier-level model is overkill.

          The goalpost moving feeds right into Anthropic and OpenAI's interests.

    • gchamonlive 47 minutes ago
      I think 24gb is the bare minimum for a local qwen3.8 based setup. I've used qwen3.6 and it's not as straightforward as "can it replace <insert the most cost-effective cloud solution today>"

      Local llms don't suffer from cloud availability issues. Anyone that used Google models know that sometimes they just don't have capacity whatsoever, at least that was the state of things some months back when I used them. Just bear in mind if needed, cloud providers will prioritise API and corporate customers over subscriptions if availability degrades more.

      Also they don't have the same guardrails as the other models, so for hacking, reverse engineering and black coding (piracy etc...) these local models might be the only options.

    • rc1 51 minutes ago
      The article implies the demand is for running locally. I’m not convinced, at least with a mac minis. Most people I know and myself buy the mini as it is always on, easy to setup, and isolated from my main computer which is a laptop. The mini is driving the use of the $20/month subscriptions.
      • julianlam 27 minutes ago
        As a thin client to access cloud models this is an astounding waste of money.
    • inventor7777 23 minutes ago
      One use case I find cost effective is using it as a voice assistant for Home Assistant. API pricing on models is very weird compared to the normal chats, so I use Qwen/GPT-OSS on my Mac Studio via llama.cpp server.
    • fisle 52 minutes ago
      Could you elaborate on your experience with local models on your card? I've been thinking of upgrading to 9070 XT, and was thinking the 16GB would be okay-ish to at least run something usable locally, no?
      • fancyfredbot 38 minutes ago
        Usable certainly. But my impression is that useful models still need a bit more than 16GB. Something like Qwen 3.8 27B is useful but squeezing it into 16GB requires fairly aggressive quantisation which will make it unreliable (e.g it'll get stuck in loops) and won't leave enough space for a long context (which qwen 3.8 really likes)
      • villish 7 minutes ago
        If you have been using cloud hosted models, you will be severely disappointed with what you’d be able to run on 16GB VRAM. You will spend most of your time fighting with the model to fix its mistakes.
    • spacedcowboy 47 minutes ago
      I was getting semi-useful results from a 128GB M4 Max. That was a few months ago, and the models have improved (quite a bit) since then, but now I'm happy to send my $20/month to get Claude code.

      It's still frustrating as hell to come down in the morning, having given it a list of tasks to do overnight, with tests to pass before they're "done" and find that it worked for about 20 minutes after I went to bed, and decided that it would stop at "3am" (it wasn't) and "not do significant work this at this late hour". Like WTF ? You're an LLM. You don't sleep.

      Bloody training data full of humans demanding sleep. I tells ya...

      • scrumper 29 minutes ago
        > "not do significant work this at this late hour"

        Is this Claude code? Or your local? I assume Claude? I'm more than a little staggered by this, like, it makes no sense! It doesn't even serve Anthropic's interests (surely better for them if it burns your token quota so you have to buy more the next morning.) The LLM just... decided? I'd be so mad.

        WTF indeed. Can one even file bugs?

        • xienze 20 minutes ago
          > It doesn't even serve Anthropic's interests (surely better for them if it burns your token quota so you have to buy more the next morning.)

          Parent mentioned their $20/month subscription. It's definitely in Anthropic's interests for you to not use it.

      • xienze 36 minutes ago
        > I went to bed, and decided that it would stop at "3am" (it wasn't) and "not do significant work this at this late hour". Like WTF ? You're an LLM. You don't sleep.

        I think that's Anthropic trying to get you to not extract as much value out of that subsidized subscription as possible.

    • tristor 27 minutes ago
      I've been experimenting with local models on an M5 Max MBP w/ 128GB of RAM since March of this year. Generally I've had very good results. Where things were lacking initially was with tool calling and the need to rely on tool calling for functionality like web search, which is otherwise well integrated in the cloud models. There is also a lot more work required on the harness side, however at this point (August 2026) there is not only much better tool calling in local models, but community supported projects have built good harnesses. Pi.dev and OpenCode + a SOTA local model is /very/ /very/ capable these days. Using LM Studio's built-in chat with a decent system prompt and proper tuning with local models is /very/ capable these days. Cloud models are still better, but it's the "harness" (not in the desktop app, but in the backend) that makes it so for the most part.
    • xienze 37 minutes ago
      IMO local models require a substantial amount of prompt+harness engineering to get in the neighborhood of what you'd get from a cloud model. Which isn't a bad thing, you'll learn a whole lot about how these things work.

      What you'll learn pretty quickly from said engineering is that there's a lot more to a good LLM than just the weights themselves. You need a good search provider (also self-hostable, but sounds easier than it really is). You need (well, it's debatable) a memory system. You need a good system for up-to-date library references like a Context7 (also self-hostable but the options are surprisingly not that good). You need a good set of specialized subagents that can perform various tasks well -- for the sake of "doing things well" but also managing context efficiently.

      When you've got all that, local models can be _extremely_ useful. But there's one other important thing and that's decent hardware, unfortunately. A lot of people try out local models using small consumer GPUs or Macs and are rightfully unimpressed with the performance. And if the performance doesn't get them, usually they have expectations that they'll perform at Claude levels out of the box. Getting in that neighborhood, like I said, definitely requires some work.

    • epolanski 50 minutes ago
      I have multiple 48GB friends that successfully run smaller quantized models for general assistance + light editing (coding, spreadsheets, etc), that don't require very heavy models.

      So yes, they are genuinely very useful, but they are not yet a full replacement unless you have more powerful hardware and or don't need more intelligent ai.

  • setgree 1 hour ago
    It's fun to see that even an extremely large company can find unexpected product market fit [0]. Per this article, "The company reportedly did not possess an engineering team dedicated to business customers or staff focused on developer relations, and lacked an enterprise AI strategy." That sounds insane in retrospect, but I think there's just inherent uncertainty in what people actually need and will use things for.

    [0]https://pmarchive.com/guide_to_startups_part4.html: "In a great market—a market with lots of real potential customers—the market pulls product out of the startup... The product doesn’t need to be great; it just has to basically work."

    • yardie 4 minutes ago
      You should listen to the podcast Acquired, specifically Nvidia and then Jensen Huang. They basically lucked into AI. Some researcher was using Nvidia gaming cards, and reached out to them about questions on CUDA. That email eventually turned them into a trillion dollar question.
    • DannyBee 56 minutes ago
      It's also fun to see how many people here believed this was all some clear deliberate strategy in the first place rather than an accident.
    • SaltyBackendGuy 57 minutes ago
      Maybe a bit of hindsight bias / the outside view here, but I feel like they're completely asleep if they didn't anticipate strong demand for this specific use case.
      • adjejmxbdjdn 55 minutes ago
        I don’t understand how that’s possible. They should have had a better idea of what was happening in the memory markets than pretty much any other entity.
        • Nevermark 22 minutes ago
          Their universal RAM strategy is so obviously helpful for AI. (1) GPU/NPU <--> CPU RAM copies eliminated. (2) All (most) RAM available for GPU/Neural, when local models are typically kneecapped by limited GPU RAM sizes vs. the much larger RAM options for M/Max/Pro/Ultras.

          They have been taking NPU's seriously on their phones, tablets and laptops since the M1.

          Then they enabled fully-connected RDMA for 4 x 512GB MacStudio's = 2TB RAM. Perfect for a large Mixture-of-Experts model.

          It would be very strange if they didn't notice their product line had landed in a new sweet spot.

          • pizzafeelsright 2 minutes ago
            I am curious about the corporate disconnect from the frontline to the generals.

            While the company I am in is embracing AI the disconnect and delay between what is available and possible versus what is approved and permitted is a three month window. The State employees I speak to are just now getting around to writing their usage policies for internal AI usage.

    • xattt 57 minutes ago
      Was this the case in the past?

      My vibes were that Apple wound down the “actual work” side of their operations (including machines like Xserve), because Ives couldn’t handle the unsexiness and unpredictability of business requirements in hardware.

      He was self-indulgent and only wanted to work on things that “vibed” with him, rather than what the customers needed. It’s easy to be creative when you get to do what you want to do, it’s hard when you have hard constraints.

      • tonyedgecombe 36 minutes ago
        I think Jobs was quite sceptical about courting enterprises. Personally this is one of the reasons I choose Apple over Microsoft.
    • 1over137 49 minutes ago
      No ‘staff focused on developer relations’ is entirely unsurprising based on what I see from the outside.
  • AdmiralAsshat 38 minutes ago
    Mac Mini's were really nice HTPC candidates, too, before the AI boom. Like all things genuinely useful and affordable, they were snatched from the hands of normal consumers by a bunch of schmucks chasing the latest gold rush.
    • scrumper 26 minutes ago
      I need a new little Mac for my music studio, currently an M2 MacBook Pro. I thought I'd be fun to experiment with some local models as well. Well, let's price up an M5 Pro. $3,019 with 64GB RAM and a 1TB HD. Three thousand American dollars for a Mac Mini. Beefy spec for sure but not comically so.

      Frankly even the entry price is a bit high - I remember buying one for my son a few years ago (M1 mini) and it was a few hundred; now we're up to $900 for the base model.

  • martythemaniak 1 minute ago
    This may sound a little wacky, but one potential use case I'm considering is robotics. Say you want to use a fine-tuned mid-size VLM model right on the robot. You pretty much have to use the Jetson line, which has a great ecosystem and everything (depth cameras, lidar, SLAM, small segmentation models, etc) runs on it, but it's gets very expensive very fast if you want to run LLMs on it. The Jetson Thor lines are 3-5k depending on memory etc.

    One very efficient option today is to have the cheapest Jetson (Orin Nano) run the classical robotics stack, then have a base mac mini run nothing but the VLM. The Mac mini is considerably cheaper and faster at these workloads than the mid-range Jetsons.

    I think this wonky situation is because Apple us under immense consumer pressure to absorb the ridiculous memory prices, while the Jetson is aimed at "business" and much more likely to fluctuate with the market. Last year I bought a Jetson Orin Nano 8GB for $375CAD, today that official nVidia Amazon page is out of stock and other sellers have it listed for $900-$1100CAD. Absolutely bonkers pricing.

  • paxys 1 hour ago
    I really hope with Ternus taking the helm Apple starts to remember that it has products outside of iPhone.
  • ceejayoz 1 hour ago
    Time to bring back the Xserves, I guess.
  • compounding_it 1 hour ago
    More customers is generally a good problem to have in most businesses. Just that the situation is very paradoxical given the supply shortages.
  • Xeoncross 53 minutes ago
    If I had to pick a product, I'd say an affordable 32GB mac would be the sweet spot for running local models that function well like Qwen 3.8.

    It's true, most people don't run models, but being the default platform for running open weights seems like it has plenty of advantages right now. Just like sales benefited from developers defaulting to MacOS for most open source languages like Ruby, Go, Rust, and TypeScript.

    • mirekrusin 32 minutes ago
      32GB is not enough, it's unified/shared memory, you need to have space for usual system and user apps/services.

      64GB+ or dedicated 48GB (2x24 on GPUs) is IMHO absolute minimum.

    • tristor 34 minutes ago
      > If I had to pick a product, I'd say an affordable 32GB mac would be the sweet spot for running local models that function well like Qwen 3.8.

      32GB is not enough RAM. I don't even own a device with less than 36GB at this point, and that device I only have because my employer is being cheap. 64GB is a reasonable starting point for running local LLMs + normal tasks. 128GB let's you really run most smaller models like Qwen 27B and 35BA3B with good context. Even Qwen3.8-Flash-Next runs in 128GB with a 4-bit quant.

      32GB would be limited to running models like Gemma4 12B and smaller dense Qwen versions like 9B unless you were using very small quants which damages quality of response.

  • Scubabear68 53 minutes ago
    Not just the high end stuff. The Neo is sold out until late September on the budget end, it seems like it is a smash for HS and college kids.

    I hope Apple can take all this cash and do some stability releases like they used to do, bugs around things like Family Sharing, the painful "update" to Settings App, etc could all use a lot of love.

  • api 56 minutes ago
    Apple has a huge opportunity here to lead the market for machines to run local models if they step into it. Their stuff is already better than what nVidia is offering with stuff like the DGX Spark.

    It's a niche market but it's a market that overlaps heavily with professionals in the AI space and lead developers, so it's a market that gets them customers in those roles.

    If I were running Apple I'd call the RAM price bubble for what it is and temporarily eat some margin to offer machines with more RAM than competitors, especially these models that are great for edge AI, and capture market share.

  • comrade1234 1 hour ago
    I wish they sold something that could go in a colo - redundant power supplies, lights out management, etc. you know they have them internally...
  • jshier 34 minutes ago
    Now if only they hadn't discontinued the Mac Pro. Could be quite the AI machine with multiple compute GPUs at higher bandwidth than an external Thunderbolt enclosure.
  • hzwanip 1 hour ago
    Off guard? :'D
  • taskoutputs2k 1 hour ago
    just a real bummer that they raised the prices so much
    • VCFundedGenYer 1 minute ago
      I don't know if you're young, or new to this industry, or what, but take a look around - the prices went up across the board. It's not just Apple.
    • snarkyturtle 1 hour ago
      It definitely puts it out of the range of every day users but a non-insignificant proportion of people who use it for ai have become multi-millionaires because of ai. So there's definitely no shortage of people who have no problem paying those high prices.
    • AlexandrB 1 hour ago
      I suspect it can't be helped at this point. RAM is the new gold. Valve even had to increase the price for a 4 year old piece of hardware[1].

      [1] https://tech-insider.org/ca/steam-deck-price-increase-2026/

    • lenerdenator 1 hour ago
      Please address all complaints to the Office of CEO, Open AI, 123 Main Street, San Francisco, California, U.S.A., 94158.

      Your letter will help feed the boiler that heats the building and is much appreciated.

  • jmclnx 48 minutes ago
    I am missing something in the article. From what I am reading, AI companies are so desperate for memory they are buying Apple Systems and other hardware and striping them for RAM and maybe other components.

    Is that what others read ?

    • Leftium 31 minutes ago
      Apple hardware uses a unified architecture. That means the CPU and RAM are integrated together: it is very difficult/impossible to strip the RAM from these systems.

      This unified architecture makes Apple hardware very good for AI work, where latency between RAM and CPU is very important

      Even the SSDs (which are normally more strippable) use a proprietary hardware form factor.

      I think the article mentioned the real draw:

      > Apple noticeably promoted the ability to link multiple Mac Studios together into a single, more capable system for running large frontier AI models, a feature aimed at business and developer customers rather than everyday consumers.

      (Couldn't read the source cited, might have more info: The Information)