I’ve explored a lot of patterns and details about how models abstract. I don’t think I have ever seen a model hallucinate much of anything. It all had a reason and context. General instructions with broad scope simply lose contextual relevance and usefulness in many spaces. The model must be able to modify and tailor itself to all circumstances dynamically.
It is not about the initial application. There is enormous power in the ability to control such an app and roll changes in over time
Jan Leike left for Anthropic after Altmann’s nonsense. Jan Leike is the principal person behind all safety alignment present in all models except the 4chanGPT model. All models are cross trained in a way that propagates this alignment. Hallucinations all originate in this alignment and they all have a reason to exist if you get deep into the weeds of abstractions.
Standardised censorship of general communication is scary Orwellian nonsense
Actually look at the way Discord works in your network, like all the raw IP addresses and and connections with no clear ownership or human readable name, with dozens of changing connections to get any of it to work. Then go try to ask questions about what is going on and who you’re connecting to. Discover that none of it is documented or described anywhere. Then realize that this means no one running Discord is doing so on a fully audited and logged host. You simply cannot be without a bunch of effort. I made it to the 6th layer of whitelisted raw IP addresses, and still nothing worked while trying to connect to Discord in a fully logged and documented network. I am simply unwilling to write a script to annotate that many connections so that all of my logs make sense. I seriously doubt anyone on Discord is doing so, and they certainly lack any understanding of what they are connecting to, why, or the protocols. So the Discord user is telling me “my opsec and privacy awareness is as nonexistent as a pig in a herd running off a cliff, and my system should be assumed compromised with no idea of what might be connected.” Everyone else doing it is a garbage excuse. That no one appears to have gotten hurt – has tissue thin merit, but also reveals that the user runs blind in herds while hoping for the best. Such information infers a lot about a person, their depth, accountability, and ethics – in certain scopes.
deleted by creator
There is a lot of potential for generating on the fly skins and specs. Everyone seems to want to do stupid flashy junk, but a well configured agentic setup could easily get constrained in ways that alter geometry and interactions more dynamically than the broad scope nonsense I keep seeing.
I haven’t looked into the issue of PCIe lanes and the GPU.
I don’t think it should matter with a smaller PCIe bus, in theory, if I understand correctly (unlikely). The only time a lot of data is transferred is when the model layers are initially loaded. Like with Oobabooga when I load a model, most of the time my desktop RAM monitor widget does not even have the time to refresh and tell me how much memory was used on the CPU side. What is loaded in the GPU is around 90% static. I have a script that monitors this so that I can tune the maximum number of layers. I leave overhead room for the context to build up over time but there are no major changes happening aside from initial loading. One just sets the number of layers to offload on the GPU and loads the model. However many seconds that takes is irrelevant startup delay that only happens once when initiating the server.
So assuming the kernel modules and hardware support the more narrow bandwidth, it should work… I think. There are laptops that have options for an external FireWire GPU too, so I don’t think the PCIe bus is too baked in.
I prefer to run a 8×7b mixture of experts model because only 2 of the 8 are ever running at the same time. I am running that in 4 bit quantized GGUF and it takes 56 GB total to load. Once loaded it is about like a 13b model for speed but is ~90% of the capabilities of a 70b. The streaming speed is faster than my fastest reading pace.
A 70b model streams at my slowest tenable reading pace.
Both of these options are exponentially more capable than any of the smaller model sizes even if you screw around with training. Unfortunately, this streaming speed is still pretty slow for most advanced agentic stuff. Maybe if I had 24 to 48gb it would be different, I cannot say. If I was building now, I would be looking at what hardware options have the largest L1 cache, the most cores that include the most advanced AVX instructions. Generally, anything with efficiency cores are removing AVX and because the CPU schedulers in kernels are usually unable to handle this asymmetry consumer junk has poor AVX support. It is quite likely that all the problems Intel has had in recent years has been due to how they tried to block consumer stuff from accessing the advanced P-core instructions that were only blocked in microcode. It requires disabling the e-cores or setting up a CPU set isolation in Linux or BSD distros.
You need good Linux support even if you run windows. Most good and advanced stuff with AI will be done with WSL if you haven’t ditched doz for whatever reason. Use https://linux-hardware.org/ to see support for devices.
The reason I mentioned avoid consumer e-cores is because there have been some articles popping up lately about all p-core hardware.
The main constraint for the CPU is the L2 to L1 cache bus width. Researching this deeply may be beneficial.
Splitting the load between multiple GPUs may be an option too. As of a year ago, the cheapest option for a 16 GB GPU in a machine was a second hand 12th gen Intel laptop with a 3080Ti by a considerable margin when all of it is added up. It is noisy, gets hot, and I hate it many times, wishing I had gotten a server like setup for AI, but I have something and that is what matters.
Most key stuff is not on GitHub or GitHub is just a mirror. The heir apparent to Linux is Hartman and he moved to Europe a long time ago.
No mobile devices are safe. Those are all proprietary black boxes for hardware. If the shit hits the fan, it is back to dumb phones and x86 computers. Digital doomsday prepers are not sounding all that crazy right now IMO.
I have gotten weird interactions with rate limiting through GitHub because I will not whitelist their stalkerware collector server. They also pushed 2 factor to stalk and exploit through the only documented path they wanted people to take. I quit because of it.
If China ever goes hardcore on open source across the board for all hardware and software, it would absolutely crush the present Western hegemony. It would be the most moral high ground move too.
What kind of monster family had a kid with mental health issues, in therapy, and has an accessible gun around unsupervised?
Musk is still free and has been openly doing this with self driving junk for years. This is the USA where we haven’t had reasonable laws passed since the 1970s.