The Model as One Rater Among Several: Measuring Political Positions in Data-Sparse Regions with a Language-Model Panel

1. Introduction

Ask where a Lebanese party sits relative to an Egyptian one, or whether Tunisia's Ennahda in 2024 stands where Egypt's Muslim Brotherhood stood in 2012, and the tools built for this kind of comparison mostly stop at the edge of Europe. The spatial tradition that treats politics as movement along defined dimensions is old and well established[2][1][3], and the manifesto-coding and expert-survey programs that put it to work are among the most cited tools in the field[4][13]. They were built for a different region. Many of the most important actors in the Middle East and North Africa never publish a Western-style manifesto, never contest a competitive election, or commit one position to a founding charter and the opposite one in office. The result is a gap in coverage that goes well beyond a few missing rows.

This gap matters beyond area studies. Quantitative comparative politics increasingly uses position estimates as variables in models of coalition formation, party competition, conflict onset and democratic backsliding, so the narrow reach of these tools limits what the field can say. A field that can place every party in the Bundestag to a tenth of a point, while it cannot place the actors of a twenty-state region at all, is in effect deciding which questions get asked.

The obvious move today is to point a large language model at the problem and read the answer off. A fast-growing literature reports that frontier models match or outperform crowd workers, and sometimes trained coders, on text-annotation tasks[21][22], which makes it tempting to treat the model as a measurement device that has simply read more than any human could. I consider this the wrong stance. A model's fluent and confident placement of a party is one reading, produced by one system, and it carries biases from a training corpus the analyst did not choose and cannot inspect. If the model is treated as an oracle, those biases end up laundered into apparent fact. The alternative is to treat it as a rater, that is, a judge whose output is pooled with others and whose disagreements are kept and reported. A rater of this kind is something an existing methodology already knows how to handle. For forty years, expert surveys have built valid measures out of fallible human judges by pooling them and studying their agreement[13][14]. This paper puts the language model in the same role, with the same tools and the same skepticism, and asks the questions that methodology is designed to answer. Do the raters agree? If they do, is it because they are right or because they share a mistake?

The instrument I build to make this concrete, Tayyar (تيّار, "current"), is a positional dataset of the Middle East and North Africa. It places parties and public figures, current and historical, on a set of defined and regionally anchored axes, using a panel of raters that is, for now, mostly language models. I use it throughout as a worked example, though the contribution of the paper lies elsewhere, in a method and a stance: language models used as raters in data-sparse measurement, with their reliability taken seriously and kept apart from validity. The main design choices follow from that stance, namely applicability rules, a declared-versus-behavioral lens, a synthesis that keeps disagreement, and an inclusion gate on aggregation.

Several results follow from this design. First, I set out the model-as-rater stance and the panel behind it, including the measurement reasons to keep a blank distinct from a zero and a stated position distinct from a revealed one. Second, a built-in experiment isolates the effect of the rubric by holding a definition-free round fixed. Written definitions both move scores and improve agreement between raters, and this measurable rubric effect rules out the simplest form of the steering worry, according to which the rubric merely imposes an arbitrary direction on a compliant model. Third, I report Krippendorff's α under both interval and ordinal metrics, so the result does not depend on an arguable equal-spacing assumption, and I press the point that high agreement among models trained on overlapping data falls short of validity. The pattern of disagreement turns out to be informative as well. The panel's widest split, on an actor's stance toward the state's foundational order, traces to a referent ambiguity that no rubric can resolve, and coding errors account for little of it. Throughout, I try to be explicit about what the instrument cannot yet do, so that a reader can weigh it fairly.

If the stance holds, the field gains a measurement where it currently has none: the actors of a data-sparse region placed on comparable, defined axes, with their disagreement reported, by raters that are available at scale today. The design is organized around its own central risk, that capable models can be capably wrong in the same direction, and it tries to measure that risk and keep it in view, leaving room for the human validation that would close it.

It is worth noting at the outset that the reliability figures below cannot stand in for an external check. The usual external benchmarks, such as a manifesto corpus, an expert survey like the Chapel Hill placements or party-level democracy scores, barely reach these actors, which is the very gap the method exists to address. The validity test therefore has to come from purpose-built human anchoring.

2. Related work

Tayyar sits where several literatures meet, and it tries to borrow the discipline of each while avoiding the blind spot that tends to come with it. I take the traditions one at a time, since most of the method's design choices are easiest to understand as responses to a specific limitation in each.

Spatial models of politics. The idea that political actors can be located as points in a low-dimensional issue space runs from Hotelling and Downs[1][2] through the roll-call scaling of Poole and Rosenthal[3], and it remains the common language of quantitative comparison. Tayyar keeps the geometry and drops an assumption that suits two-party legislatures far better than this region, namely that a single recovered dimension carries most of the signal. In the Middle East and North Africa, a party's stance on the role of religion in the state, its position on the regional axis of resistance versus normalization, and its posture toward the Palestinian question are empirically distinct, and they do not fold back into a left–right line. Collapsing them too early discards the structure the measurement was meant to recover. The instrument hence keeps a full position vector and treats the familiar two-axis "compass" as a default view of it.

Manifesto coding and text scaling. The Comparative Manifesto Project and its MARPOR successor[4][6][5] established the practice of estimating positions from a party's own published texts, hand-coded sentence by sentence against a fixed scheme. Its automated descendants, "words as data"[7], the Wordfish scaling model[8] and the later refinements of scaling from coded text[9], dropped the human coder at the price of needing a large reference corpus in a language the method has seen at scale. Two points matter here. First, the coding is noisier than its wide use suggests. Mikhaylov, Laver and Benoit show non-trivial misclassification even among trained human coders of manifestos[10], a sobering baseline for any automated substitute and a reminder that the human coder is one more rater, with no claim to being a gold standard. Second, and decisively for this setting, bag-of-words scaling is tied to its corpus and its language. Arabic is comparatively under-served by the pretrained representations these methods rely on[36], and Hebrew more so. A Wordfish estimate comparing an Arabic charter to a Hebrew platform compares two thinly modeled languages through a method that assumes a dense one. Grimmer and Stewart are candid that automated text methods amplify careful reading and cannot replace it[11][12]. Tayyar keeps document grounding as the spine of its declared reading and replaces word-frequency scaling with rubric-anchored judgment, which a multilingual model applies more evenly across Arabic, Hebrew and English than a model of one language's word counts can.

Expert surveys and the aggregation of judgment. The Chapel Hill Expert Survey[13][15] and Benoit and Laver's expert placements[14] ask informed people to place parties directly and take the mean expert as the estimate. Their validity rests on inter-expert agreement and on the expectation that idiosyncratic error washes out across a panel. The Varieties of Democracy project goes furthest in this direction, aggregating many country experts through an explicit item-response measurement model that estimates both the latent positions and each rater's reliability and thresholds[16][17]. This is the tradition Tayyar belongs to most closely, and the analogy is close to literal, since the panel is an expert survey in which most of the "experts" are language models. It inherits the tradition's main strength, a principled way to build a measure out of fallible judges, together with the vulnerability V-Dem's own methodologists take care over. A panel that agrees for a shared wrong reason looks, from inside the agreement statistics, exactly like one that agrees because it is right. Human expert panels are assembled to be diverse, whereas a panel of language models can be far more correlated than its size suggests, a point I return to at length.

Cross-cultural incomparability. Placing actors from very different systems on one scale raises the problem that the scale may not mean the same thing in each. King and colleagues formalized this as differential item functioning and proposed anchoring vignettes to correct it[18]. The Aldrich–McKelvey family of methods and its Bayesian extensions recover latent positions while estimating and removing each rater's distortion of the common scale[19][20]. For Tayyar the problem is live in two ways. Across actors, an axis can simply fail to be commensurable. "West alignment" names an existential strategic baseline for an Israeli party and a contested ideological choice for an Arab one, so the same number encodes different facts on the two sides of that divide. Across raters, different models may anchor the −10 to +10 scale in different places. The approach here manages incomparability by anchoring the rubric in worked regional examples that fix what the poles mean, by scoping axes to where they apply, and by flagging the axes whose comparability across the divide is weakest. These defenses are partial, and §8 limitations returns to what they leave open.

Language models as annotators. The neighboring literature finds that frontier models are strong text annotators. Gilardi, Alizadeh and Kubli report ChatGPT outperforming crowd workers on several political-text tasks[21], Törnberg reports much the same for annotating the ideology of political messages[22], and Ziems and colleagues survey the broader promise for computational social science while cataloging where it can go wrong[23]. A parallel strand insists that LLM annotations be validated against human labels and shows that unvalidated automated coding can be confidently wrong[24]. Tayyar takes both messages seriously. It uses models as annotators because, in a region where trained coders and reference corpora are scarce, they are the most capable raters available at scale, and it builds validation in as planned human anchoring.

Language models as position estimators. Closest to this paper is a young literature that uses language models to place actors directly in a policy or ideological space. Wu, Nagler, Tucker and Messing scale US senators on a liberal–conservative dimension by asking a model for pairwise comparisons, and recover a scale that correlates strongly with NOMINATE[25]. Le Mens and Gallego take the approach furthest. They ask an instruction-tuned model where a text stands on a focal dimension and average its responses, recovering manifesto and roll-call benchmarks at correlations above 0.9 across several languages, and they warn that the method has to be re-validated in each new setting[26]. Tayyar shares the premise that a model can read a position and departs from this work in four places, which the rest of the paper develops. First, it treats the models as a panel of distinct raters and keeps their disagreement, since the disagreement is itself diagnostic (§7 reliability). Second, it works in a region with no large validated reference corpus, so the benchmark correlations that make averaging look safe in well-served party systems are unavailable, and reliability has to be argued from the panel itself, with the threat of a shared prior addressed directly. Third, it gates applicability and declines to score an axis that does not apply. Fourth, it takes Le Mens and Gallego's warning, that a high correlation or a high α can hide a shared error, as its organizing idea, and places the distinction between reliability and validity at the center of the method.

The political leanings and non-independence of models. A panel of models cannot be read like a panel of independent experts, for the simple reason that models are not independent. A growing body of work documents that language models carry measurable, and often shared, political tendencies. Santurkar and colleagues show whose opinions models reflect and whose they leave out[27], Durmus and colleagues measure how some national viewpoints are over-represented on global-opinion tasks[28], Atari and colleagues argue that the representations skew toward Western, educated and industrialized populations[29], and Feng and colleagues trace political bias from pretraining data through to downstream behavior[30]. Several studies find a consistent left-of-center tilt across widely used models[31][32], with the deeper diagnosis that the training corpora, and the people who built them, shape the prior[35]. A related literature on "silicon sampling", which uses models as stand-ins for survey respondents, finds the simulations convincing and unreliable at once, faithful to some subpopulations and badly off for others[33][34]. Is a model therefore a useless rater? It seems not, though its agreement is suspect in a specific and measurable way. Models drawn from overlapping corpora may share a prior, and about a region like the Middle East that prior is plausibly an Anglophone-media one. The reliability analysis of §7 confronts this threat directly, and only the planned human anchoring can settle it.

Reliability theory. The standard coefficients are Cohen's κ for chance-corrected categorical agreement[39] and, for ordered or interval scales on which not every rater scores every item, Krippendorff's α[37][38], which handles the missing data a partially filled panel always produces. I lead with α, and following the advice to report a coefficient under the measurement assumption one is prepared to defend, I report it under both an interval and an ordinal difference function.

Regional scholarship. The frame and its exemplars draw on the area-studies literature, including Lynch on the post-2011 uprisings[40], Hinnebusch on Ba'athist Syria[41] and Wickham on the evolution of the Muslim Brotherhood[42]. Anchoring the axes in this literature lets the rubric carry the meaning of the scale, which would otherwise default to each rater's own folk theory of the region.

3. The instrument: frame, applicability, and lenses

Axes

The dataset runs on sixteen active axes (Table 1). Each has two named poles, a written definition that governs how it is scored, and a scope that decides where it applies at all. Eight are general dimensions of political contestation. The others are regional or context axes, meant to catch cleavages a Western frame misses or misreads, such as pan-Arabism, the posture toward Iran and the stance on confessional power-sharing. The default two-axis compass uses the economic and social axes, to stay close to the political-compass tradition, while the clustering and comparison analyses use the full sixteen-dimensional vector. This is one defensible way to divide the region's politics among several, and §8 limitations returns to what the choice leaves out.

Table 1. The axis frame. Each of the sixteen active axes carries a written definition and worked regional anchors that fix its poles; the definitions govern scoring, and the labels only name the poles. Each axis links to its full definition.

Applicability, and why a blank is not a zero

Most axes are universal, though a handful are scoped, and the handling of scope turns out to be a measurement decision with consequences. Pan-Arabism is simply not a meaningful question for a Jewish-Israeli party. The stance on confessional power-sharing applies only where such a system exists, as in Lebanon's Taif arrangement or Iraq's muhasasa, and context axes apply only to their countries. When an axis does not apply to an actor, Tayyar leaves the cell blank, and both scoring and rendering skip it.

In a spatial model, a score of zero places the actor at the midpoint of a contested dimension, whereas an inapplicable axis means the actor is not on the dimension at all. Coding the second as the first injects a phantom centrist position that drags every aggregate over the axis toward the middle. A region's mean "sectarianism" score, with the inapplicable cells set to zero, ends up measuring mostly how many actors the question does not concern. Being outside a question and being at its center are different facts, and a measurement that conflates them misleads any downstream model that averages, clusters or correlates over the axis. In a region whose cleavages reach so unevenly, I see no workable alternative to keeping the blank distinct from the zero.

The lens system: declared, behavioral, composite

For many political actors, a single number papers over the gap between what they profess and what they do. For this reason, Tayyar models a position as up to three readings. The declared lens is the stated position, read from platform, charter and dated speech, and it is the reproducible spine of the measurement, since two raters reading the same document can be asked to converge. The behavioral lens is what the actor does with power, meaning its legislative record and its conduct in office, and it serves to test the declared position. The composite is the published synthesis.

Each cell is assigned to the construct that fits it. The declared lens is the default. The behavioral lens is scored selectively, for actors who hold or have held power, on the conduct axes of liberal democracy, civil liberties and press freedom, where a manifesto promise is cheap and the governing record is the better evidence. A record in office speaks to these three axes more directly than to the others. How a party treats courts, dissent and the press shows in its conduct in a way its pan-Arab sympathies or its abstract stance toward the regime's legitimacy do not, and an actor that has never governed leaves no such record, so extending the behavioral lens to the other axes would mostly manufacture missing data. The approach borrows from the behavioral turn that distinguishes what regimes say from what they do[16]. An actor whose character changed across eras is modeled as separate era-stamped entities, so that one cell never has to hold the contradiction; the founding 1947 Ba'ath of Aflaq's pan-Arab charter is a different entity from the Assad regime that governed in its name. The split also removes a tension that would otherwise corrupt the reliability statistics. A good share of the cells on which the panel disagrees most turned out to be declared and behavioral readings that an undifferentiated score had been averaging together, and separating the constructs turns that spurious disagreement into two well-posed measurements.

4. Data and provenance

Table 2. Coverage in the v1.0 release. Parties are scored in fifteen of the twenty framed countries; five Gulf states have no parties on record.

The unit of analysis is the party or the public figure. Current and dissolved actors are both included and era-stamped, so historical comparison is supported directly. Scored positions are grounded, where grounding exists, in primary documents such as party platforms and charters, dated speeches, official statements and founding constitutions, while secondary sources serve only to corroborate biographical facts. For non-English originals, each source records the language it was written in, a detail that matters because a model's reading of a translated text and of the original need not agree. Verification is explicit and runs through three states, unverified, tarek-verified and externally-verified, and every record carries its state.

The provenance bears on how the reliability evidence should be read, so I state it plainly. The dataset is largely the work of one principal investigator with model assistance. The verification ladder, the document-grounding layer and the planned human anchoring exist to keep that provenance legible and improvable. A reader should, in my view, treat the current dataset as a well-instrumented pilot, whose internal agreement is real and whose external validation has not yet been done.

5. The rater panel and its synthesis

The panel

A position in Tayyar comes out of a rater panel. The choice everything else depends on is to treat each language model the way an expert survey treats a single expert[13][14][17], as a judge whose worth shows once the judgments are pooled and studied, and who can simply be wrong. The analogy breaks down in one place that matters, and I return to it in §7 reliability. A human expert panel is assembled for a spread of training and background. These raters share web-scale corpora, and two of them, Gemini and Gemma, come from the same builder. Whatever epistemic diversity the panel has is thus something to test, and nothing in its composition guarantees it.

Each cell can carry ratings from up to four kinds of rater. An author-baseline hand-coding seeds the panel as an explicit prior; it is the one rating in the system that does not come from a language model, and the panel is tracked against it. The language models number nine: Claude, Gemini, Grok, GPT and Gemma, built in the United States, and Kimi, DeepSeek, Qwen and MiniMax, built in China, drawn from eight laboratories (Appendix A lists the version each model ran). Each scores every cell on its own, in a fresh session, from a blind template that shows the axis definition and poles and none of the other raters' answers. Models join as they become available. A model that joins partway through scores from that point on, and both the synthesis and the reliability statistics tolerate a rater with partial coverage. A tenth model was trialed and dropped because it could not complete the scoring instrument reliably. Human anchors, scored by people, are a planned round whose purpose is the validity check the model panel cannot run on itself (§7 reliability); they are not in the data yet. Two smaller layers complete the picture. In a grounded pilot, one model scored directly from the cited documents, and in an early re-coding probe, two language-model re-coders scored a sample of 63 cells. Appendix B describes both.

Synthesis, and the preservation of disagreement

Each rater's ratings for a cell collapse to that rater's median, and the published score is the median across distinct raters, so a model that scored a cell in three rounds counts once. I use the median because, in a panel of nine, one model that badly misreads a cell should not be able to drag the published number, and the median, which resists outliers, is the choice an expert-survey methodology would make in any case. The published median draws on the with-definitions model round (r2), the author's prior, the grounded pilot, which is folded into Claude's value so that Claude still counts once, and, on their 63 cells, the two re-coders. The definition-free rounds never enter it, since they are the baselines held back for the experiment of §6 rubric.

The synthesis keeps disagreement visible. Dispersion is summarized by the median absolute deviation (MAD) of the raters' scores about the published median, on the −10 to +10 scale, and each cell receives a status from fixed cutoffs. A cell is settled when it has at least two raters whose MAD about the median is within tolerance (MAD ≤ 1.5). It is contested when they diverge sharply (MAD ≥ 3), in which case the cell is rendered and released as a range. It is provisional otherwise, which covers a single rater so far or a spread between the cutoffs. The thresholds are pragmatic, and the underlying MAD travels with each cell so that a reader can redraw them. A companion agreement figure, 1 − MAD/10, accompanies every synthesized cell as a convenience for the live tool; it rescales dispersion onto [0,1] for display and has no statistical standing of its own. Every rating, with its rater, round, score, and rationale, remains visible on the rater panel page.

The v1.0 release holds 12,867 ratings. Of its 1,108 active composite cells, 803 (72%) are settled, 53 (5%) are contested and 252 (23%) remain provisional, and 998 (90%) carry at least two independent model raters. The panel fills country by country, so these proportions shift with each scoring session and should be read as interim.

An inclusion gate on aggregation

A cell's status governs that cell and nothing more. Field-level quantities, such as which axis divides the region most, how two party families compare, or how the axes correlate, are a second layer of inference, and a sparse axis can quietly distort all of them. With only a handful of actors scored on an axis, a single outlier swings the mean, a high variance reads as "contested" when the axis is merely thin, and a spurious correlation can end up at the top of a table. To guard against this, Tayyar admits an axis into field-level aggregates only once its coverage clears a floor, currently thirty scored entities. The figure is a convention I chose, though it falls in an empty band of the coverage distribution. No axis has between roughly twenty and sixty-five scored cells (its per-axis n in Table 4), so the split between core and periphery comes out the same for any floor in that range. Coverage is the structural gate, because coverage is what gives an average its meaning. Cross-rater dispersion is reported alongside it as a separate signal, so a well-covered and genuinely contested axis is flagged and stays eligible. Axes below the floor are reported in full on their own and never folded into a headline. In the current data, this separates a core of well-covered dimensions from a periphery of scoped axes (sectarian power-sharing, Iran posture, press freedom and gender) whose small cell counts make them informative about individual actors and not yet about the field. The gate is unglamorous, and I doubt the dataset would work without it, since most of the ways a positional dataset misleads at the macro level come down to reading thin coverage as a finding.

6. The rubric effect: a built-in experiment

The standard objection to a language model as a rater is that the rubric does the work. Hand a compliant model a definition with a built-in direction and it will echo that direction back, so any apparent agreement measures how suggestive the prompt was. The objection is fair, and the round protocol is designed to answer it with a number.

Ratings are labeled by round. The first round (r1) ran on two models, Gemini and Grok, without the axis definitions. Each model received the poles and placed the actor from its own understanding, and the round was then locked. Every later round runs with the sharpened definitions and the construct rule of §3 frame. Holding r1 fixed makes the worry measurable. The movement from r1 to r2 on the cells scored in both is the effect of introducing the rubric, and the shape of that movement shows whether the rubric steered the raters or corrected them.

Two findings come out of this. The first concerns movement within each rater. Across the 376 rater-cell pairs scored in both rounds, the mean absolute shift is 1.79 points on the 21-point scale, and r1 and r2 scores correlate at r ≈ 0.90 (Figure 2a). In other words, the rubric leaves the central tendency of most placements in place, as one would expect if the models started from reasonable priors, but it matters in the tail, where roughly one cell in five (19%) moves by three points or more. On inspection, the large moves read as corrections. In a fairly typical case, a model placed Likud at +4 on the liberal-democracy axis without definitions, taking "holds elections" to mean "committed to liberal democracy," and at −3 once the rubric specified that the axis concerns the conduct of power as well as the holding of elections. The move brought an outlier back in line with the rest of the panel and with the behavioral record.

The second finding is that the rubric improves agreement between raters. On the 187 cells both models scored in both rounds, the definitions narrowed the mean absolute gap between Gemini and Grok from 2.81 to 2.50 points and raised their correlation from 0.81 to 0.89 (Figure 2b). The steering objection struggles with this. A rubric that imposed an arbitrary direction would shift the scores without necessarily bringing two independent raters closer about where an actor sits, and the definitions did both, which rules out a purely arbitrary or orthogonal steer. On its own, the result still cannot separate convergence toward correctness from convergence toward an anchor the rubric supplies. Only an external record, such as the behavioral evidence in the Likud correction, and eventually the planned human-anchor round can make that separation. Evidently, the experiment shows something narrower than one might want, and it holds: the first-order effect of the rubric is to turn vaguer individual readings into a more reproducible shared one.

Two raters are a thin base for a general claim, and the feature that makes the test clean, the same two models scoring the same cells with and without definitions, is also its limit. A larger replication, paired less perfectly, helps with generality. The definition-free round was run across the panel on a fixed, randomly drawn anchor of 204 cells (the persistent inter-rater sample) and compared with the rubric round on the same cells. Mean pairwise disagreement between raters falls from 2.43 to 2.12 points, in the same direction and of comparable size, now across eight raters from seven laboratories. GPT is left out of this comparison, because the import of its definition-free ratings returned entity keys that did not match its rows (Appendix B). The convergence does not depend on which two models happened to run the first round.

The design carries beyond this dataset. Any rubric-based use of a model rater can run the same fixed, definition-free baseline and report what its rubric does.

7. Reliability

The reliability question for a panel instrument is whether the same frame, applied by different raters, lands in the same place. The figures here cover the 80 parties scored so far, across fifteen countries; positions for individual public figures come in a later round. They are interim and will move as the panel fills, so they describe a measurement still in progress.

Agreement across the model panel

On the 998 cells that carry at least two independent, blind model scores, the panel agrees closely. Krippendorff's α across the model raters is 0.86 on an interval difference function. As in the synthesis (§5 panel), each rater contributes one value per cell, its within-rater median, so the coefficient is computed over distinct raters and is not inflated by repeated rounds from the same model. The steps of the −10 to +10 scale may not be equally spaced, since whether the distance from −2 to −1 equals the distance from 0 to +1 is an assumption. I also report α on an ordinal difference function, which assumes only that the values are ordered and weights a disagreement by how much of the observed distribution lies between the two ranks. The ordinal α is 0.87, essentially the same as the interval figure. Because the two agree, the reliability claim does not depend on treating the scale as evenly spaced, the assumption much interval reliability reporting leaves unstated and the one a methods reviewer is most entitled to question. The interval coefficient held at 0.86 as the panel grew from five raters to nine, which offers some comfort against one narrow worry, that the early figure was an artifact of a few near-identical raters. It cannot speak to a prior shared across all nine, which a wider panel would simply carry along.

Each model's average agreement with the others is high (Table 3). Every model's mean correlation with the rest of the panel lies between 0.85 and 0.90, with mean absolute gaps around two points. The panel as a whole also tracks the author's baseline, the one comparison available today that is not itself a language model. That tracking is worth only so much. The baseline is my own hand-coding, written by the same person who wrote the axes, the anchors and the rubric the models receive, so agreement with it is what one would see if the rubric had simply passed my prior on to the models. It is a coherence check on the instrument, and external validation can only come from raters independent of the frame. A separate internal probe had two language-model re-coders score a stratified sample of 63 cells, drawn as three blind rounds of 21 (Appendix B). Their agreement with each other is α ≈ 0.95. Both re-coders apply the same framework, which reproduces a score almost by construction, so the figure is a consistency floor and I do not treat it as a reliability result; the relative pattern across axes is the useful part.

Table 3. Per-model agreement. Each model's agreement with the rest of the panel on the r2 cells, ordered by mean pairwise correlation. r̄ is its mean Pearson correlation with each other model over shared cells, mean |Δ| the matching mean absolute gap on the −10 to +10 scale, and "n cells" the cells it shares with the rest of the panel. Every r̄ falls in 0.85–0.90, and build country does not order the column: the highest and the lowest are both US-built, with the China-built models between them. Below the rule, the panel median is set against the author's hand-coded prior, the one comparison here that is not a language model, and the two language-model re-coders against each other on their cells, a consistency floor.

Where the panel converges, and where it frays

Agreement is far from uniform across the frame, and the per-axis pattern (Table 4) is the actionable part, since it points to the rubric anchors that most need tightening. The panel converges most where a position is loudly and repeatedly declared, as on gender, the Palestinian question, and the social and state-and-religion dimensions, where two readers of the same texts land in the same place. Iran posture and press freedom look comparably tight in Table 4, though they rest on thin coverage (n = 17 and n = 19), and I would not lean on their agreement yet. The reasons the panel frays elsewhere deserve naming, mostly because they differ from the obvious guess.

Table 4. Per-axis agreement. Mean pairwise absolute gap across the panel, tightest to loosest, on the −10 to +10 scale. Scoped axes have small cell counts and are noisier, so each row should be read with its n; the inclusion gate (§5 panel) keeps the thinnest out of field-level aggregates.

Take the single widest disagreement in the data. On regime stance, Hayat Tahrir al-Sham draws scores across the entire scale. Five models place it near the revolutionary floor (three at −10, two at −9) and four near the incumbent ceiling (between +7 and +8). This is no coding error, and a knowledge gap explains it only in part. In the eighteen months before this writing, HTS went from the most heavily sanctioned jihadist franchise in the Levant to the government in Damascus, and one might guess that the split separates the models that learned of the transition from those that did not. The rationales show otherwise. Raters at both poles state the same facts, that HTS toppled the Ba'athist order and now governs Syria, and they divide on which order "the foundational order" denotes. A rater scoring the actor's posture toward the order it overthrew reads it as anti-regime (−10), and a rater scoring its posture toward the order it is now building reads it as pro-regime (+8). I call this a referent problem. The axis assumes a single stable order to take a stance toward, and an actor that has just exchanged one order for another breaks that assumption. Ennahda splits the same way across fifteen points, opposing Kais Saied's incumbent order while defending the 2011 constitutional order it helped write, and so do the Syrian Ba'ath, the Sudanese Communist Party and Yemen's Islah, each an actor whose country offers more than one plausible "regime" to be for or against. A sharper version is referent absence. In a state mid-collapse, with no settled order to support or oppose, the axis is ill-posed. The applicability rule (§3 frame) asks whether an axis applies to an actor in principle and does not check whether a stable referent exists at a given moment, so it misses this case. Leaving such cells blank on the order-dependent axes, which the rule currently renders as a contested range, is a refinement it should absorb.

To test whether this is the mechanism, three language-model coders, working blind and independently, classified the source of disagreement on the twenty-two most divided regime-stance cells, agreeing on 76% of their pairwise judgments. A clear majority traced to interpretation. Roughly two-thirds (64%) turned on which order is the referent, or on how to weigh an actor's posture toward it, against 18% genuine factual conflict and 18% definitional confusion. Written definitions are the rubric's instrument for the definitional kind, and on this axis they have little to work with. The panel divides over which regime an actor is scored against, and fixing that referent for a contested actor is a political judgment the instrument should not make. Accordingly, the synthesis marks these cells contested and keeps the insurgent and the incumbent readings apart, and the dataset flags the column to be read with that limit in mind. Each cluster mixes US-built and China-built raters, so the split follows interpretation and does not track build country.

This is the kind of finding I hoped the panel would catch. A single-number method would average the two clusters into a meaningless midpoint, while keeping the split exposes a genuine ambiguity in the question. For an actor that has just replaced one order with another, "stance toward the regime" has no single answer, and a confident zero would be the wrong one.

A second cluster sits on sectarian power-sharing, the loosest axis in the table, and it exposes a definitional trap. Hezbollah, the Amal Movement and the Syrian Ba'ath each draw spreads of thirteen to fourteen points, with raters scattered from pole to pole. The axis is meant to measure a stated position on the confessional quota, that is, whether an actor would abolish muhasasa or entrench it. A model that hears "Hezbollah" and scores a sociologically sectarian actor answers a different question from one that reads the party's own anti-confessional rhetoric and scores the platform. Both readings are defensible, and they answer different questions. The lens system (§3 frame) exists to resolve this kind of construct conflation.

The same pattern runs across the frame. The panel agrees about what actors say and frays over what their position is once it has to be inferred. This holds on the conduct axes (liberal democracy and civil liberties), where a charter and a record pull apart, and on the scoped regional axes (federalism, regime stance, pan-Arabism and above all sectarian power-sharing), where raters differ about what the axis asks before they reach an answer. The axes that define the region's conflicts are among the least settled, which is roughly the diagnosis a reliability program should produce. It directs the next round of rubric work, and it tells the reader which columns of the dataset to lean on and which to treat with caution.

This creates a real tension. The panel is tightest on the universal, Western-legible axes, the social, economic and state-and-religion core, and loosest on the regional axes (pan-Arabism, sectarian power-sharing and regime stance) that justify its existence, so its reliability is lowest exactly where the instrument claims to add the most. Admittedly, this could be read as a refutation of the method. I read it as the diagnosis of a problem. The regional, evaluatively loaded constructs carry the interpretive disagreement analyzed above, and an instrument that reported them as settled would be hiding that problem. The implication for use holds either way, and the inclusion gate of §5 panel enforces it: the regional columns inform claims about individual actors and stay out of field-level aggregates until their coverage and agreement can support the weight.

Reliability is not validity

Everything in this section concerns reliability, the reproducibility of a reading, and nothing in it establishes validity, its correctness. The distinction matters more than almost anything else in the method. Most of the panel's raters are language models trained on overlapping data, and nine models agreeing amount to nine correlated readings[23]. They can share a single prior, and for the politics of the Middle East, a prior shaped by Anglophone media coverage is exactly the kind of shared error I would most want to rule out. A high α is fully consistent with an instrument that is reliable and systematically wrong, and no statistic computed over the models alone can tell the two apart. The defenses lie outside the panel: the author's baseline that the panel tracks (a single non-model anchor, useful and insufficient alone), the document-grounding layer that ties scores to cited text, and the one that matters most and is not yet collected, human anchors scored by people.

One internal check makes the gap concrete. The archived grounded pilot, in which a single model scored from the cited documents, overlaps the parametric panel on 117 cells. There, its grounded scores correlate only r = 0.65 with its own parametric scores and differ from them by a mean of 3.7 points on the 21-point scale. The gap exceeds any model's mean disagreement with the rest of the panel (1.9 to 2.3 points in Table 3), and 44% of the cells move by three points or more. Re-reading the documents moves the same model more than swapping one model for another does. This is one model on a small, early sample, and it settles neither reading, though it points the same way as the rest of this section. The panel's close agreement is agreement about a shared parametric reading, and grounding that reading in cited text is the validity work that remains, which adding more models cannot do.

One further check can run inside the panel. It mostly comes up empty, and I report it for the record. If the shared prior were specifically a US or Anglophone one, models built in the United States and models built in China should disagree wherever that prior drives the score more than the text does. I measure this per axis as the mean absolute gap between US-built and China-built raters on the cells both blocs scored (the between-bloc gap), set against the mean gap among raters of the same build country (the within-bloc gap, a five-rater quantity on the US side and a four-rater one on the China side). On no axis does the between-bloc gap exceed the within-bloc gap by more than a third of a point (Table 5). The largest difference is on regime stance (2.74 → 3.02, +0.29), and the next, on federalism and Iran posture, never exceeds +0.25. Of the two axes a naive geopolitical-bias story would target first, pan-Arabism shows almost no difference, and sectarian power-sharing's +0.18 rests on eleven cells. The split is small and does not single out the China-built models.

Table 5. The provenance contrast is small. Mean absolute gap between raters within build-country blocs and across them, on the −10 to +10 scale, for the five axes with the largest between-minus-within difference and for pan-Arabism (the rest are smaller or negative). The largest, on regime stance, is under a third of a point. Of the two axes a geopolitical-bias account would target first, pan-Arabism shows almost none, and sectarian power-sharing rests on eleven cells. Build country is a coarse proxy for corpus independence (see text). Differences are computed on unrounded values and may differ from the rounded columns by 0.01.

I would not lean on this result heavily. Build country is a crude proxy for what actually matters, corpus independence: Google built two of the nine raters, "US-built" groups corpora that share little, and four raters are a thin base for a within-bloc dispersion. A null result here also cannot rule out a prior shared across all nine models whatever their origin, which is the harder and probably the likelier worry. A fuller treatment of the provenance contrast is left to companion work. The test that would settle the question lies outside the models altogether, and the validity round that would supply it is the method's main unfinished business.

8. Limitations

• Correlated model error, and the missing validity check. The panel consists mostly of language models trained on overlapping corpora, so their agreement is evidence of reliability and says little about independence (§7 reliability). A contrast of models by national provenance, developed in companion work, tests whether a shared prior is doing the work, though detecting a bias is a different task from removing it. Human anchors, scored by people blind to the panel, would turn reliability into validity. That round is planned and not yet collected, and until it is, the gap applies to every reliability figure in this paper. It will also close unevenly, for reasons built into the data. Human anchoring is feasible for present-day, reachable actors and hardest exactly where the data are thinnest, for defunct parties and for actors in active conflict zones where local expertise is scarce or unsafe to solicit. For those cases, the document-grounding layer, which ties a score to cited text and needs no living expert, is the validity path that has to carry the weight, so validity will arrive unevenly across the dataset.
• Stated versus revealed. The declared/behavioral lens manages the oldest problem in positional measurement and does not make it disappear. Which object a number refers to is a choice the analyst makes and has to declare, and a reader who draws that line elsewhere should read some cells differently.
• Ungrounded model rounds, and why grounding is the more valuable work. The current rounds score from the models' parametric knowledge plus the rubric, and only the pilot scores from cited documents. They are best read as "what well-informed models believe, given clear definitions," with document-grounded scoring as a separate verification layer still under construction[7][24]. Reading parametric knowledge means the rounds may be extracting the shared prior itself, so the reliability statistic and the scoring construct can agree for the same reason, both reading the prior when they should be reading the actor. Grounding the score in cited text (§7 reliability) would close that gap, and it is the unfinished work I would put first.
• Recency, and the moving target. Politics in the region moves fast, and a model's knowledge of the latest turns is bounded by its training cutoff. The prompt dates the question ("score the actor as it stands in 2026"), yet an actor mid-transition, such as Hayat Tahrir al-Sham between insurgency and government, or a party turned out of office between one model's cutoff and the next, can still split the panel across the full scale when the models disagree about a fact only some of them have learned. The design treats such splits as signal. The cells are flagged contested, and the synthesis does not average an insurgent with an incumbent. The blind triple-coding of the widest regime-stance cells (§7 reliability) is one attempt to tell the two sources apart. It attributed 18% of that disagreement to factual conflict, the category that would contain any cutoff-driven split, against 64% to interpretation, so cutoff-driven splits exist there and are in the minority. Recency is a hard ceiling on any model panel, and the only real remedy is fresh scoring once events settle.
• The frame is one decomposition among several. Every axis is a defensible choice and also an editorial call. The West-alignment axis in particular means different things for Israeli and Arab actors, so comparisons across that divide require care[18]. I do not yet report how much the headline placements move when a given axis is dropped or redefined, and a sensitivity analysis of that kind, which the regime-stance case (§7 reliability) argues for, is a necessary next step.
• An uneven, drifting panel. Not every cell is scored by the same set of models. Per-model coverage varies, so the reliability figures pool across cells whose panels differ in composition. A commercial model is also a moving target, because a rater label means whichever version answered in that session, and recording model identifiers only partly pins that drift down. The recording is itself incomplete, since the version field was left empty for several models' main-round sessions (Appendix A).
• Imports. Ratings returned from chat sessions were imported by key, and three of these imports came back with keys that did not match their rows. In the main round, 154 of GPT's ratings sat on the wrong parties until the October 2026 correction reassigned them to the parties their rationales name. In the definition-free round, 483 rows from two imports could not be reassigned and are flagged and excluded (Appendix B). Every figure in this version is computed from the corrected release. For any panel that collects ratings through chat interfaces, the practical lesson is to validate the returned keys against the template before import.
• Positionality. An instrument that places contested political actors is no view from nowhere. The choice of axes, the anchors that fix their meaning and the applicability rules reflect a particular reading of the region, written out in full so that it can be argued with. They are also one investigator's choices: the active axes, the regional anchors, the applicability rules and the seed baseline are all single-authored for now. I do not think this bottleneck should stand, and the planned human-anchor and external-expert rounds are meant to widen the judgment behind the frame as well as the scoring within it. This is a real limit on cultural validity. The frame embeds contestable theory, since treating resistance versus normalization as a primary regional cleavage is itself a choice, and no single author holds the range of expertise that axes from Lebanese confessionalism to the Gulf–Iran rivalry demand. Arguably, the external-expert round has to involve participatory co-design of the frame, beyond sign-off on the scores produced under it, and the frame will need revision as the region changes. A reader who would draw a line elsewhere is using the instrument as intended.

9. Reproducibility and access

The instrument is built so that its outputs can be traced to their sources. The live tools, the rubric and a methodology document read from the same tables as the figures reported here, so the public interface and the published numbers cannot drift apart unnoticed. The dataset is kept as versioned releases, that is, CSV and JSON exports of every primary table with a manifest and checksums, so a citation resolves to a fixed state while the data keep growing. Every figure in this version is computed from the v1.0 release, which is public at tarekgara.com/tayyar/dataset under CC BY-NC-SA 4.0 (source texts under fair use, with a documented takedown route), with SHA-256 checksums for every file and archived under DOI 10.5281/zenodo.23704154. The scripts that recompute every figure in this paper from the release files are available from the author on request.

Because the synthesis keeps disagreement visible, each cell is exported with its median, its dispersion (MAD), its settled, contested or provisional status and, where contested, a range. A downstream analysis should carry that uncertainty forward. The median serves where a model needs a point estimate. The dispersion can enter directly as measurement-error variance in an errors-in-variables or Bayesian hierarchical model, a contested range can be multiply imputed or entered as bounds, and a fixed rule, such as dropping contested cells or down-weighting by the agreement figure, keeps the decision visible. Treating a contested cell as a confident point would reintroduce the false precision the synthesis exists to avoid, and whichever rule a study adopts is a choice it should state.

@misc{gara_tayyar_2026,
  author        = {Gara, Tarek},
  title         = {The Model as One Rater Among Several: Measuring Political Positions in Data-Sparse Regions with a Language-Model Panel},
  year          = {2026},
  eprint        = {2606.23042},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CY},
  doi           = {10.48550/arXiv.2606.23042},
  url           = {https://arxiv.org/abs/2606.23042}
}

Gara, T. (2026). The Model as One Rater Among Several: Measuring Political Positions in Data-Sparse Regions with a Language-Model Panel. arXiv:2606.23042. https://arxiv.org/abs/2606.23042

10. Conclusion

This paper argues that the right way to use a language model in positional measurement is the way the discipline already uses a fallible human expert, as one rater in a panel, pooled and studied, with its agreements scrutinized and its disagreements kept. The stance converts a set of vague anxieties about "using AI for social science" into answerable questions. The first is whether the rubric steers the model or sharpens it, and the built-in experiment answers part of it. The rubric moves scores in the tail and tightens agreement between raters, which rules out a merely arbitrary steer, while the question whether that convergence leads toward correctness or toward an anchor the rubric supplies remains for the external behavioral checks and the planned human-anchor round. The second question is whether the raters agree once the equal-spacing assumption is dropped, and they do, on both an interval and an ordinal metric. The third is whether that agreement reflects correctness or a shared prior, and the agreement alone cannot tell, since no figure the models produce about themselves settles it. Where the panel does come apart, most sharply over which constitutional order an actor should be scored against, the disagreement turns out to be interpretation, a limit that no sharpening of the rubric can remove.

What remains is the validity check the method is built to receive and has not yet undergone, namely human coders scoring blind against the same rubric. That round and the document-grounding layer are the next steps, and they would allow the instrument's agreement to be read as more than reproducibility. Where these actors do appear in an existing instrument, as a few do in cross-national democracy indices or in expert codings made for other projects, those coordinates offer an obvious external check, and I would welcome their use against the released data. The overlap is thin precisely because the region is under-served, which is why purpose-built human anchoring is the validity work that counts.

Here lies the contribution of the paper: a regionally anchored frame, an applicability rule that respects the difference between a blank and a zero, a lens system that separates word from deed, a synthesis that keeps disagreement, and an experimental, skeptical treatment of model raters that keeps their fluency from passing for correctness. For a field that can place every party in a handful of well-documented democracies and almost none elsewhere, this discipline makes the rest of the map approachable. The running example is the Middle East and North Africa, though the gap is general, since the comparative apparatus goes quiet wherever the standard sources run out. The proposal is general as well, and it asks only that a language model brought in as a rater be held to the standards the discipline already applies to its human experts.

Appendix A. The rater panel: models and versions

Table 6 lists the version of each model in the with-definitions (r2) panel, with its laboratory, build country and cell coverage in the v1.0 release. A commercial model keeps changing, so the panel stores whichever version identifier answered in a given session (§8 limitations), and the table lists those identifiers. The version was written to each rating for GPT, Gemma, MiniMax, Qwen and most of Claude's sessions; for the other models it was noted at the time of scoring. All r2 scoring took place in June 2026.

Table 6. The r2 model panel. The version each model ran, its builder and build country, and its r2 cell coverage in the v1.0 release.

Appendix B. Data provenance by round

Ratings are kept round by round (Table 7). The published median draws on the with-definitions round (r2), the author's prior, the grounded pilot, folded into Claude's value, and the re-coding probe on its 63 cells, while the reliability statistics use r2 alone. The definition-free rounds supply the rubric baselines of §6 rubric and never enter the median.

Three imports came back with entity keys that did not match their rows. In r2, 154 of GPT's 961 ratings had been attached to the wrong parties, 112 of them to entity identifiers that exist in no table and the rest to three real parties that carried another party's ratings. Each block's rationales name the party it was meant for, which is how the rows were reassigned in October 2026. Where two parties' rows had collided on one key, one was lost, so GPT has no ratings for two parties and only two for a third. In r1b, all 204 of GPT's rows and 279 rows of Gemini's first import could not be reassigned, since many carry no rationale. They remain in the release with an import_flag and are excluded from every figure. This is why the 204-cell anchor of §6 rubric covers eight raters.

Table 7. Ratings by round. Every recorded rating, reconciling to the 12,867 of Table 2. Cells are distinct (actor, axis) pairs within a round.

References

1. Hotelling, H. (1929). Stability in Competition. The Economic Journal, 39(153), 41–57.
2. Downs, A. (1957). An Economic Theory of Democracy. Harper & Brothers.
3. Poole, K. T., & Rosenthal, H. (1997). Congress: A Political-Economic History of Roll Call Voting. Oxford University Press.
4. Budge, I., Klingemann, H.-D., Volkens, A., Bara, J., & Tanenbaum, E. (2001). Mapping Policy Preferences: Estimates for Parties, Electors, and Governments, 1945–1998. Oxford University Press.
5. Klingemann, H.-D., Volkens, A., Bara, J., Budge, I., & McDonald, M. (2006). Mapping Policy Preferences II: Estimates for Parties, Electors, and Governments in Eastern Europe, European Union and OECD 1990–2003. Oxford University Press.
6. Volkens, A., Burst, T., Krause, W., Lehmann, P., et al. (2021). The Manifesto Project Dataset (MARPOR), Version 2021a. WZB Berlin Social Science Center.
7. Laver, M., Benoit, K., & Garry, J. (2003). Extracting Policy Positions from Political Texts Using Words as Data. American Political Science Review, 97(2), 311–331.
8. Slapin, J. B., & Proksch, S.-O. (2008). A Scaling Model for Estimating Time-Series Party Positions from Texts. American Journal of Political Science, 52(3), 705–722.
9. Lowe, W., Benoit, K., Mikhaylov, S., & Laver, M. (2011). Scaling Policy Preferences from Coded Political Texts. Legislative Studies Quarterly, 36(1), 123–155.
10. Mikhaylov, S., Laver, M., & Benoit, K. (2012). Coder Reliability and Misclassification in the Human Coding of Party Manifestos. Political Analysis, 20(1), 78–91.
11. Grimmer, J., & Stewart, B. M. (2013). Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts. Political Analysis, 21(3), 267–297.
12. Grimmer, J., Roberts, M. E., & Stewart, B. M. (2022). Text as Data: A New Framework for Machine Learning and the Social Sciences. Princeton University Press.
13. Bakker, R., de Vries, C., Edwards, E., Hooghe, L., Jolly, S., Marks, G., et al. (2015). Measuring Party Positions in Europe: The Chapel Hill Expert Survey Trend File, 1999–2010. Party Politics, 21(1), 143–152.
14. Benoit, K., & Laver, M. (2006). Party Policy in Modern Democracies. Routledge.
15. Hooghe, L., Marks, G., & Wilson, C. J. (2002). Does Left/Right Structure Party Positions on European Integration? Comparative Political Studies, 35(8), 965–989.
16. Coppedge, M., Gerring, J., Knutsen, C. H., Lindberg, S. I., Teorell, J., et al. (2024). V-Dem Codebook v14. Varieties of Democracy Project.
17. Pemstein, D., Marquardt, K. L., Tzelgov, E., Wang, Y.-T., Krusell, J., & Miri, F. (2018). The V-Dem Measurement Model: Latent Variable Analysis for Cross-National and Cross-Temporal Expert-Coded Data. V-Dem Working Paper 21 (3rd ed.).
18. King, G., Murray, C. J. L., Salomon, J. A., & Tandon, A. (2004). Enhancing the Validity and Cross-Cultural Comparability of Measurement in Survey Research. American Political Science Review, 98(1), 191–207.
19. Aldrich, J. H., & McKelvey, R. D. (1977). A Method of Scaling with Applications to the 1968 and 1972 Presidential Elections. American Political Science Review, 71(1), 111–130.
20. Hare, C., Armstrong, D. A., Bakker, R., Carroll, R., & Poole, K. T. (2015). Using Bayesian Aldrich-McKelvey Scaling to Study Citizens' Ideological Preferences and Perceptions. American Journal of Political Science, 59(3), 759–774.
21. Gilardi, F., Alizadeh, M., & Kubli, M. (2023). ChatGPT Outperforms Crowd Workers for Text-Annotation Tasks. Proceedings of the National Academy of Sciences, 120(30), e2305016120.
22. Törnberg, P. (2023). ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning. arXiv:2304.06588.
23. Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., & Yang, D. (2024). Can Large Language Models Transform Computational Social Science? Computational Linguistics, 50(1), 237–291.
24. Pangakis, N., Wolken, S., & Fasching, N. (2023). Automated Annotation with Generative AI Requires Validation. arXiv:2306.00176.
25. Wu, P. Y., Nagler, J., Tucker, J. A., & Messing, S. (2023). Large Language Models Can Be Used to Estimate the Latent Positions of Politicians. arXiv:2303.12057.
26. Le Mens, G., & Gallego, A. (2025). Positioning Political Texts with Large Language Models by Asking and Averaging. Political Analysis (arXiv:2311.16639).
27. Santurkar, S., Durmus, E., Ladhak, F., Lee, C., Liang, P., & Hashimoto, T. (2023). Whose Opinions Do Language Models Reflect? Proceedings of the 40th International Conference on Machine Learning (ICML).
28. Durmus, E., Nguyen, K., Liao, T. I., Schiefer, N., Askell, A., et al. (2024). Towards Measuring the Representation of Subjective Global Opinions in Language Models. arXiv:2306.16388.
29. Atari, M., Xue, M. J., Park, P. S., Blasi, D. E., & Henrich, J. (2023). Which Humans? PsyArXiv preprint.
30. Feng, S., Park, C. Y., Liu, Y., & Tsvetkov, Y. (2023). From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), 11737–11762.
31. Rozado, D. (2024). The Political Preferences of LLMs. PLOS ONE, 19(7), e0306621.
32. Motoki, F., Pinho Neto, V., & Rodrigues, V. (2024). More Human than Human: Measuring ChatGPT Political Bias. Public Choice, 198, 3–23.
33. Argyle, L. P., Busby, E. C., Fulda, N., Gubler, J. R., Rytting, C., & Wingate, D. (2023). Out of One, Many: Using Language Models to Simulate Human Samples. Political Analysis, 31(3), 337–351.
34. Bisbee, J., Clinton, J. D., Dorff, C., Kenkel, B., & Larson, J. M. (2024). Synthetic Replacements for Human Survey Data? The Perils of Large Language Models. Political Analysis, 32(4), 401–416.
35. Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (FAccT), 610–623.
36. Antoun, W., Baly, F., & Hajj, H. (2020). AraBERT: Transformer-based Model for Arabic Language Understanding. Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT4).
37. Krippendorff, K. (2019). Content Analysis: An Introduction to Its Methodology (4th ed.). Sage.
38. Hayes, A. F., & Krippendorff, K. (2007). Answering the Call for a Standard Reliability Measure for Coding Data. Communication Methods and Measures, 1(1), 77–89.
39. Cohen, J. (1960). A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement, 20(1), 37–46.
40. Lynch, M. (2016). The New Arab Wars: Uprisings and Anarchy in the Middle East. PublicAffairs.
41. Hinnebusch, R. (2001). Syria: Revolution from Above. Routledge.
42. Wickham, C. R. (2013). The Muslim Brotherhood: Evolution of an Islamist Movement. Princeton University Press.

References are to the established comparative-politics, content-analysis, and computational social-science literatures the method builds on. Region-specific primary sources, party platforms, charters, and speeches, are cited per entity in the dataset itself rather than here.