Today is a technical day: one detailed survey of where LLM architecture is headed, and a cluster of conceptual pushback from Dwarkesh Patel on standard AI capability narratives — covering science, power, and why pretraining is still harder than it looks.

Sebastian Raschka surveys four recent open-weight releases — Gemma 4, Laguna XS.2, ZAYA1-8B, and DeepSeek V4 — and identifies a unifying signal: every major architecture decision right now is downstream of long-context memory pressure. The question isn't how to make models smarter per token; it's how to make them cheaper at 128K, 1M token contexts.

The approaches diverge meaningfully. Gemma 4's smallest models (E2B, E4B) take the most conservative route: cross-layer KV sharing, where later transformer layers reuse key-value projections computed in earlier layers rather than generating their own. In Gemma 4 E2B's 35-layer stack, only the first 15 layers compute their own KV; the remaining 20 reuse them. This saves approximately 2.7 GB of KV cache memory at 128K context (bfloat16 precision). The trade-off is reduced model capacity — each layer still computes its own queries, so attention patterns vary, but the expensive KV state is shared. According to the cross-layer attention paper, the accuracy impact is minimal for small models.

Gemma 4 E2B also adds per-layer embeddings (PLE), a subtler trick. Rather than making the transformer stack wider or deeper, PLE gives each layer a small token-specific embedding slice computed once from a shared lookup table. The "E" in E2B denotes "effective": 2.3B effective parameters for the transformer compute path, but 5.1B total when embedding tables are counted. The logic is that embedding parameters are cheaper to use than expanding attention or FFN layers — they're largely lookup operations that can be cached. Raschka notes we have to take Google's word that this is worth the architectural complexity; no ablation comparing E2B to a plain 2.3B or 5.1B model appears in the release.

Poolside's Laguna XS.2 takes a different angle. Rather than compressing what's stored, it varies how much attention each layer is allowed to spend. Full-attention layers use 6 query heads per KV head; sliding-window layers get 8 — the inverse of what you might expect. The logic holds up: global attention is already expensive because it scans the full context, so giving it fewer query heads per KV head is a reasonable budget trade. The per-layer `num_attention_heads_per_layer` config key makes this explicit.

ZAYA1-8B from Zyphra goes further with Compressed Convolutional Attention (CCA), which is conceptually adjacent to DeepSeek's multi-head latent attention (MLA) but architecturally distinct. MLA compresses the KV representation stored per token while keeping one entry per token. CCA performs the entire attention operation inside a compressed latent space, reducing both KV cache size and attention FLOPs during prefill and training. The "convolutional" component mixes local context into the compressed Q and K representations before attention scores are computed — partially compensating for expressiveness lost to compression. According to the standalone CCA paper (October 2025), CCA outperforms MLA under comparable compression settings, though the comparison is from the authors' own evaluation.

DeepSeek V4 is the most aggressive and complex. At 1M-token context, DeepSeek V4-Pro uses 27% of the single-token inference FLOPs and 10% of the KV cache of DeepSeek V3.2; the Flash variant drops to 10% FLOPs and 7% KV cache. Two mechanisms drive this. Compressed Sparse Attention (CSA) groups tokens into compressed KV entries with sparse selection; Heavily Compressed Attention (HCA) summarizes every 128 tokens into 1 KV entry and attends densely over those fewer entries. CSA preserves more detail but uses sparse selection; HCA keeps far fewer entries and can afford dense attention over them. They're complementary enough that DeepSeek V4 interleaves rather than picks one.

DeepSeek V4 also introduces manifold-constrained hyper-connections (mHC), which replaces the single transformer residual stream with multiple interacting parallel streams. The "manifold-constrained" part keeps the mixing matrices doubly stochastic (non-negative, rows and columns summing to 1), preventing signal amplification or collapse across deep stacks. The mHC paper reports reaching baseline performance in roughly half the training tokens, with only 6.7% additional training time overhead for 4 residual streams. This is the rare architecture change that touches the residual pathway rather than attention or FFN — notable because those have been relatively static.

Raschka's meta-observation is worth sitting with: the transformer is still dominant, but code complexity has probably 10x'd since GPT-2. The basic architecture is recognizable. The attention subsystem now encompasses MQA, GQA, MLA, sliding-window, sparse, CCA, CSA, HCA, and their various layer-by-layer combinations. Each tweak reduces runtime cost while adding conceptual surface area. His bottom line: qualitative modeling performance is still largely driven by data quality and training recipes; architecture is now primarily the arena where inference efficiency is won or lost.

Dwarkesh Patel's most substantive piece argues that reinforcement learning from verifiable rewards (RLVR) — the mechanism behind AI dominance in coding and math — is structurally mismatched with genuine scientific discovery. The argument isn't about AI capability in general; it's about the epistemology of science not resembling the epistemology of a coding test suite.

The core problem is verification lag, and the historical examples Patel marshals are more precise than the usual hand-waving. Heliocentrism required stellar parallax measurement to decisively confirm — Copernicus published in 1543, parallax was measured in 1838. But more damaging to the "science is verifiable" claim: Copernicus's heliocentric model was actually less accurate than Ptolemy's geocentric model in 1543. Ptolemy had accumulated millennia of correcting epicycles; Copernicus, rejecting Ptolemy's equant trick on Platonic aesthetic grounds, had to add more epicycles to compensate. The heliocentric framing only looked like progress retroactively, once Kepler's laws (1619) and Newton's gravity (1686) made it dramatically more productive. An RL agent evaluating predictive accuracy in 1543 would have kept the geocentric model.

The Neptune/Vulcan asymmetry is the sharpest case. Both Neptune and the hypothetical planet Vulcan were predicted as explanations for orbital anomalies — one turned out to be a real planet, the other required Einstein's general relativity to resolve. You couldn't have distinguished those two research programs ex ante; they look identical under falsificationism until decades of failed observation accumulate. A proper Newtonian, Patel notes, would have kept modifying the Vulcan hypothesis — bigger telescope, cosmic dust, magnetic interference — and each step would have looked like normal scientific caution, not epicycle-stacking.

The implication for AI is specific: big conceptual breakthroughs require unreasonable long-horizon commitment to research programs that look unproductive by any short-term verification standard. Prout's hypothesis that atomic weights are integers (1815) accumulated anomalies for nearly a century before isotopes vindicated it. Darwin needed Lyell's geology (1830) to give him deep time; Wallace independently arrived at evolution at the same time, suggesting intellectual footholds matter as much as individual genius. The pattern of parallel discovery in science and technology suggests these ideas weren't individually brilliant — they were latent in the available conceptual infrastructure, waiting for enough ancillary intuition pumps.

Patel's two conclusions: first, you can't easily train an RL loop for big conceptual breakthroughs, because the reward signal is decades away and surrounded by noise. Second, a "society of AI scientists" would still need instances with idiosyncratic, persistent research commitments — the scientific equivalent of being unreasonably stubborn about an idea that looks wrong by contemporary standards but turns out to be progressive.

In a separate and shorter piece, Patel untangles a conceptual confusion that runs through a lot of AI safety thinking. If you define intelligence as "the ability to achieve goals across a wide range of domains," that's functionally the same definition as power — and on that definition, Trump and Xi are more "intelligent" than the physicists. Stalin, by that measure, was the most intelligent person who ever lived.

The alternative definition — "comprehend and build atop abstract concepts" — is clearly what we mean when we think about scientific or technological superintelligence. But the correlation between extreme power and this kind of intelligence is weak. The physicists are not running the world. George III went mad halfway through his reign; Britain still defeated Napoleon and built a global empire.

The AI we're building today is getting smarter through coding, reasoning, and scientific assistance — capabilities with modest correlation to the institutional authority, social trust, and coordination capacity that actually generate power. Patel's argument: the right frame for thinking about AI's societal impact is competitive advantage in normal capitalist dynamics, not a single AI out-scheming humanity. Like national IQ effects (modestly correlated with individual income, strongly correlated with national outcomes), AI intelligence will multiply the capacity of organizations that adopt it, generating spillover effects through coordination — not by concentrating power in the AI itself. The people holding the reins don't need to be brilliant; they just need to sit atop the right institutional structures.

Patel also surfaces detailed technical notes on why large pretraining runs fail — a useful counterpoint to any sense that training at scale is now solved engineering.

2 main culprits: broken causality and compounding bias. In mixture-of-experts (MoE) routing, expert-choice routing (each expert selects its preferred tokens rather than each token choosing its top-k experts) can break causality: whether token N gets processed by a given expert can depend on what token N+K would have chosen, information unavailable at inference. The circulating explanation for Llama 4's underperformance is that this causality violation during training was the culprit. Token dropping — where oversubscribed experts simply drop lower-priority tokens — has the same problem.

The GPT-4 original training bug is a cleaner illustration of compounding bias: FP16 all-reduce operations on gradients. FP16 distributes floating-point precision logarithmically, so at large magnitudes (1024+), the representable intervals span multiple whole numbers. Summing 10,000 small gradients into a large accumulator creates systematic rounding errors at each step — the computed sum ends up 10x off the real value. Bias compounds; variance averages out. The bug is also nearly invisible — the values look plausible, just wrong.

On the question of whether AI can automate kernel writing: Patel relays skepticism from someone who argues it's closer to an AGI-complete problem than it appears. The counterargument (that kernel optimization is highly verifiable, therefore RL should reach superhuman performance) runs into the practical observation that Nvidia's best engineers took a long time to optimize for Blackwell. Verifiability is necessary for RL but not sufficient; the search space and engineering judgment involved may exceed what short-horizon RL can navigate.

The FSDP/pipeline parallelism notes (relayed from a lecture by Horace He) have one key insight: fully sharded data parallelism's actual communication overhead is only ~50% more than vanilla data parallelism's, not the order-of-magnitude more it naively appears. The crossover point where comms time exceeds compute time — and FSDP starts to crater mean FLOPs utilization (MFU) — moves left as models become sparser. Sparse MoE models do less compute per token while comms volume stays constant, creating a specific tension between MoE efficiency gains and training parallelism scaling.

TL;DR - Long-context efficiency is now the primary driver of LLM architecture — Gemma 4, Laguna XS.2, ZAYA1-8B, and DeepSeek V4 all tackle KV cache and attention compute pressure through different compression strategies, with DeepSeek V4 achieving 10% of DeepSeek V3.2's KV cache at 1M tokens. - RLVR is structurally mismatched with scientific discovery because the verification loop in real science runs decades to centuries, and the "better" theory often makes worse predictions until ancillary conceptual infrastructure catches up. - Defining intelligence as goal-achievement conflates it with power, and the AI being built today (good at coding, reasoning, scientific assistance) has weak correlation with the institutional authority and social coordination that actually generates power. - Pretraining failures trace to causality violations in MoE routing and bias accumulation in mixed-precision operations — systematic, hard-to-detect errors that compound rather than average out.