Comparison of Contemporary Large Language Models

 This blog presents a concise structural comparison of four prominent large language models: GPT, Claude, Gemini, and xAI. Although all are built on Transformer-based foundations, they differ markedly in mathematical design, alignment strategy, training dynamics, and multimodal architecture. GPT (OpenAI) follows a scaling-law paradigm using a Transformer backbone enhanced by sparse Mixture-of-Experts layers. Claude (Anthropic) preserves the same basic architecture but introduces Constitutional AI, an alignment method that incorporates explicit behavioral constraints. Gemini (Google) adopts a unified multimodal Transformer that represents text, images, audio, and video within a single token sequence. xAI's Grok retains the Transformer form but is trained on a non-stationary, continuously shifting real-time data stream, giving it distinct temporal behavior. 

As a disclaimer, I am not a practitioner in this area; my background is primarily theoretical, so some evaluations may not be fully precise from an engineering perspective.

Basics of the Transformer Architecture in LLMs

Large language models operate on text by first decomposing it into a sequence of discrete units known as tokens. A vocabulary of approximately 50,000 tokens provides the basic alphabet from which all text processed by the model is formed. Each token is mapped to a continuous vector representation \(x_i \in \mathbb{R}^d\), called an embedding, which encodes semantic and contextual information. Transformers take a sequence of such embeddings and compute how each token should integrate information from all other tokens.  Because the self-attention mechanism by itself is insensitive to token order, each embedding \(x_i\) is augmented with a positional encoding \(p_i \in \mathbb{R}^d\), producing a position-aware vector \(\tilde{x}_i = x_i + p_i.\) All attention computations described below are applied to these position-augmented embeddings \(\tilde{x}_i\). For notational simplicity, the symbol \(x_i\)  will hereafter refer to the position-augmented embedding \(\tilde{x}_i\).

Let the encoder input sequence be represented by the position-augmented vectors \[x_1, x_2, \dots, x_n \in \mathbb{R}^d,\] which can be arranged into a matrix \[X = \begin{bmatrix}x_1^\top \\x_2^\top \\ \vdots \\ x_n^\top \end{bmatrix} \in \mathbb{R}^{n \times d}.\] The self-attention mechanism computes new representations \[ h_1, \dots, h_n \] where each \(h_i\) reflects a context-dependent aggregation of information from all tokens in the sequence. This is carried out using three learned projection matrices \[W_Q \in \mathbb{R}^{d \times d_k}, \qquad W_K \in \mathbb{R}^{d \times d_k}, \qquad W_V \in \mathbb{R}^{d \times d_v},\] which produce the query, key, and value matrices \[Q = XW_Q, \qquad K = XW_K, \qquad V = XW_V.\] The \(i\)-th rows \(q_i, k_i, v_i\) correspond to the query, key, and value of token \(i\). The similarity between tokens \(i\) and \(j\) is measured by the scaled dot-product \[s_{ij} = \frac{q_i k_j^\top}{\sqrt{d_k}},\] and collecting all such scores yields \[Score= \frac{QK^\top}{\sqrt{d_k}} \in \mathbb{R}^{n \times n}.\] Applying a row-wise softmax transforms these scores into attention weights \[ \alpha_{ij}=\frac{\exp(s_{ij})}{\sum_{m=1}^n \exp(s_{im})},\] and the new representation of token \(i\) is obtained by \[h_i = \sum_{j=1}^n \alpha_{ij} v_j.\] In matrix form, the self-attention module is expressed concisely as \[\mathrm{Attention}(X)=\mathrm{softmax}\!\left(\frac{XW_Q (XW_K)^\top}{\sqrt{d_k}} \right)XW_V.\]This operation forms the computational core of each Transformer encoder layer.

The decoder processes a partial output sequence autoregressively. Let the encoder output be \[E = \begin{bmatrix}e_1^\top \\ \vdots \\ e_m^\top\end{bmatrix} \in \mathbb{R}^{m \times d},\] and let the decoder input sequence be \[Y =\begin{bmatrix} y_1^\top \\ \vdots \\ y_t^\top \end{bmatrix} \in \mathbb{R}^{t \times d}.\] As in the encoder, the vectors \(y_i\) are understood to already include appropriate positional encodings. The decoder first applies masked self-attention to \(Y\). It forms \[Q = YW_Q^{(d)}, \qquad K = YW_K^{(d)}, \qquad V = YW_V^{(d)},\] and then adds a causal mask to prevent access to future positions. This mask is defined by \[M_{ij} = \begin{cases} 0 & j \le i, \\ -\infty & j > i, \end{cases} \] and yields the masked similarity matrix \[ S_{\text{masked}} = \frac{QK^\top}{\sqrt{d_k}} + M. \] The masked self-attention output is \[ H^{\text{self}} = \mathrm{softmax}(S_{\text{masked}})\, V.\]

After masked self-attention, the decoder attends to the encoder outputs through cross-attention. Queries originate from \(H^{\text{self}}\), while keys and values originate from the encoder output \(E\): \[ Q = H^{\text{self}} W_Q^{\text{(cross)}}, \qquad K = E W_K^{\text{(cross)}}, \qquad V = E W_V^{\text{(cross)}}. \] The cross-attention result is \[ H^{\text{cross}} = \mathrm{softmax}\!\left( \frac{QK^\top}{\sqrt{d_k}} \right) V. \] This allows each decoder position to obtain relevant information from the encoder sequence. A position-wise feed-forward mapping is applied to \(H^{\text{cross}}\), and normalization completes the computation of a decoder layer.

The encoder–decoder architecture therefore combines unmasked self-attention, masked self-attention, and cross-attention, each relying on the same mathematical structure of queries, keys, and values but differing in their constraints and sources. Through repeated stacking of such layers, the model constructs hierarchical contextual representations used for prediction or generation.

GPT: Scaling Laws and Sparse Expert Models

GPT is a standard autoregressive Transformer. For an input sequence \((x_1,\dots,x_n)\), it models the joint distribution via \[P(x_1,\dots,x_n)=\prod_{t=1}^n P(x_t \mid x_{<t}),\]and is trained by minimizing the negative log-likelihood \[\min_\theta \sum_{t=1}^n -\log P_\theta(x_t \mid x_{<t}).\]  Self-attention layers use \[\mathrm{Attention}(Q,K,V)    = \mathrm{softmax}\!\left( \frac{QK^\top}{\sqrt{d_k}} \right)V.\]

Recent GPT models employ a sparse Mixture-of-Experts (MoE) architecture:\[f(x)=\sum_{k=1}^K g_k(x)\,f_k(x),\]where \(g_k(x)\) is a softmax-based gating function and \(f_k\) are expert networks. GPT also uses reinforcement learning from human feedback (RLHF) for alignment; a reward model \(R_\phi\) guides updates according to \[\nabla_\theta J = \mathbb{E}\!\left[\nabla_\theta \log \pi_\theta(y\mid x)\,R_\phi(y)\right].\] This approach yields strong reasoning, coding, and symbolic performance. The advantages of this design include high compute efficiency due to sparse expert activation, stable autoregressive training, and strong controllability enabled by reward-model–based RLHF. Its drawbacks include potential routing imbalance in MoE layers, propagation of reward-model biases, and limitations in global coherence for very long sequences due to strictly autoregressive structure.

Claude: Constraint-Guided Optimization via Constitutional AI

Claude maintains the Transformer architecture but adopts a distinct alignment strategy. Rather than relying primarily on reward models (as in GPT with RLHF), Claude uses the model itself to evaluate and revise its own outputs according to a set of written principles (“the constitution”) that guide optimization. Let \(C=\{c_1,\dots,c_m\}\) denote these rules, which specify desirable behaviors such as avoiding harmful advice and maintaining helpfulness. Given an initial output \(y^{(0)}\), Claude performs iterative self-critique:\[y^{(t+1)} = y^{(t)} - \alpha \nabla L\!\left(y^{(t)}, C\right),\] where \(L\) measures deviation from the constitutional constraints. Repeated critique–revision steps gradually refine the output until it conforms to the prescribed principles.

This process induces constitution-mediated shifts in attention and feed-forward computations, which can be abstractly represented as: \[\mathrm{Attention}(Q,K,V)   = \mathrm{softmax}\!\left( \frac{QK^\top}{\sqrt{d_k}} + B \right)V,\] with \(B\) encoding constitution-dependent modifications. Claude is further optimized for exceptionally long context windows using sparse-attention variants and memory-efficient representations, improving stability and coherence over extended sequences. The advantages of this approach include reduced dependence on reward models, greater transparency in alignment objectives, and more consistent behavior across long contexts. Its drawbacks arise from the rigidity of rule-based constraints, the possibility of embedding unintended normative assumptions, and the additional complexity introduced by constraint-driven bias adjustments.

Gemini: A Unified Multimodal Transformer

Gemini is built on the idea of a single transformer that ingests text, images, video, audio, and other modalities within one unified token space. This allows flexible cross-modal reasoning, unified representation learning, and strong performance on tasks requiring joint understanding of video and text or audio and text. However, it also introduces a key structural weakness: token budget explosion. When a single model allocates a shared token space to video, audio, and text, the statistical capacity available for pure language modeling becomes diluted, making its fine-grained linguistic reasoning weaker than that of a dedicated text-only model.

Formally, Gemini embeds all modalities into one sequence, \[ Z = [E_{\text{text}},\,E_{\text{image}},\,E_{\text{audio}},\,E_{\text{video}},\dots],\] and processes this sequence with a single Transformer, yielding \(Z' = \mathrm{Transformer}(Z)\). This structure enables direct cross-modal interactions within the attention mechanism. Alignment combines RLHF with contrastive vision–language objectives, \[L = L_{\text{text}} + L_{\text{vision}} + \lambda L_{\text{align}},\] where \(L_{\text{align}}\) enforces consistency between perceptual and linguistic embeddings.  The unified multimodal design enables integrated perception–reasoning tasks within a single model, but it also introduces several drawbacks: large multimodal token sequences increase computational cost, training requires extensive multimodal datasets, and modality imbalance can make optimization more difficult. These structural factors contribute to higher hallucination rates: cross-modal attention introduces additional variance into hidden states, and multimodal training objectives can conflict with the requirements of stable long-form reasoning. As a result, Gemini may show less reliable logical consistency and more frequent hallucinations than similarly scaled text-only language models.

xAI Grok: Real-Time Non-Stationary Training Dynamics

In standard LLM training, as used in GPT, Claude, and Gemini, it is assumed that all samples are drawn from a fixed underlying distribution \(x \sim p_{\text{train}}.\) Under this stationarity assumption, stochastic gradient descent approximates minimization of a single objective function \(L(\theta) = \mathbb{E}_{x \sim p_{\text{train}}}\!\left[\ell(\theta; x)\right].\)

By contrast, Grok departs from this paradigm by incorporating data drawn from a time-varying distribution. If \(p_t\) denotes the distribution at time \(t\), then \(p_t \neq p_{t+1},\) so the model is trained on a stream of non-stationary data. This enables rapid adaptation to shifting real-world information, particularly from high-frequency sources such as social media streams.

When the data distribution changes with time, the instantaneous loss becomes time-dependent: \( L_t(\theta) = \mathbb{E}_{x_t \sim p_t}\!\left[\ell(\theta; x_t)\right].\)The parameter update rule takes the form \( \theta_{t+1} = \theta_t - \eta \nabla_\theta L_t(\theta_t),\) so each update is optimized with respect to different data. The gradient direction therefore follows a moving target, and classical convergence guarantees for stochastic gradient descent no longer apply. The model behaves more like an online learner than a static optimizer. Non-stationarity offers several benefits: the model adapts rapidly to new information, incorporates short-term trends, and specializes to evolving linguistic or social patterns. This responsiveness is particularly important in applications where the underlying data distribution shifts over time.

However, these advantages come with significant challenges. Because the distribution \(p_t\) changes, gradients may point in inconsistent directions, leading to increased variance and potential instability. The model is also susceptible to catastrophic forgetting: if \(p_{t+1}\) places little weight on regions emphasized by \(p_t\), previously learned information may be overwritten. The model updates its parameters in response to a time-dependent objective, enabling real-time learning at the cost of increased complexity in optimization dynamics.