A. Innateness and (Large) Language Models

• A.1 Nativism and (Large) Language Models
• A.2 Small Language Models
• A.3 Embodied and Multi-Modal Language Models
• A.4 Language Models and Impossible Languages
• A.5 Testing Language Model Competence

---

A.1 Nativism and (Large) Language Models

The striking improvements that we have seen, over the last decade or so, in the capacities of artificial systems, centrally Large Language Models, pertaining to the production of well-formed, coherent, and conversationally appropriate linguistic behavior have been seen by many to be highly significant to the question of the innate structures and representations needed to acquire language. This has already generated a massive literature (see Futrell & Mahowald 2025 for a recent overview).

The core idea animating these discussions is that Large Language Models, prior to training, do not have language-specific knowledge or biases, but after training they display behaviors relevantly similar to adult language users. They are thus argued to present a possibility proof that the relevant features of competent human speakers do not require language-specific innate biases, but can be extracted from the data. If this is correct, it undermines some of the stronger statements made by linguistic nativists about the requirements for an innate advantage in language acquisition. But there is much to unpack here.

Warstadt and Bowman (2022) helpfully spell out the relevance of Language Models (LMs) to debates about language acquisition, focusing as does most of this literature on syntax. They identify the conditions under which the success of an LM provides a proof of concept that some fact about language is learnable, by focusing on ablation experiments. Ablation experiments provide responses to a specific kind of PoS argument, which involves claiming that exposure to some specific kind (or quantity) of linguistic data is necessary in order for a learner to genuinely learn how to correctly use some specific linguistic phenomenon or construction, and denying that such data is available to human language learners, thus requiring the learner to have knowledge of the relevant kind prior to learning. Ablation experiments involve training an LM on a data set constructed carefully so as to exclude the relevant data. If the LM can, nonetheless, acquire mastery of the relevant construction, this undermines the argument that these data were indeed necessary for learning, and thus the nativist claim that innate contributions must fill this gap. Misra and Mahowald (2024) provides a nice case of just such reasoning.

As Warstadt and Bowman point out, several assumptions are needed in order to make these arguments work, and they can license claims of different strengths, suitable for rebutting different kinds of nativist claims. Firstly, such arguments function primarily as possibility proofs: they show that the relevant linguistic knowledge can be acquired on the basis of data, not that human beings in fact learn them from their linguistic experience. To go further than this, it must be shown that LMs do not make up for the lack of exposure to the relevant data with some other sort of advantage that they have over the human learners.

Much of the discussion of the relevance of LMs to the nativism/empiricism debate has thus hinged on this issue. Large Language Models are trained on massively more data than human children. In the course of their linguistic development, children are exposed to tens or hundreds of millions of words (Gilkerson et al. 2017 estimate that children are exposed to between 2 and 7 million words per year). In comparison, the most recent LLMs (e.g., Meta’s Llama) are trained on tens of trillions of tokens! That these systems are trained on data sets many orders of magnitude greater than those humans are exposed to makes them poor examples from which to build a case against empirical arguments that humans require an innate boost in their language learning capabilities.

A.2 Small Language Models

In response to this, in line with Warstadt and Bowman’s suggestions, many researchers have aimed to test the capacities of LMs on much smaller, and more ecologically plausible, data sets. For the last few years, the BabyLM challenge has been held to see how much linguistic knowledge can be acquired by LMs trained on just 10 or 100 million tokens, a number much closer to that encountered by children (although note that children’s exposure to words is not straightforwardly comparable to LMs exposure to tokens). Challengers here have developed models with altered learning mechanisms, types of input, schedules of training, and more, and have achieved interesting and important results showing that the massive data sets used by LLMs are not always required for acquisition of some kinds of linguistic capacities (see the papers collected at the BabyLM Challenge). But the results here are still preliminary, and offer encouragement both to empiricists and to nativists.

A.3 Embodied and Multi-Modal Language Models

Another crucial difference between traditional LLMs and child language learners is that LMs are exposed only to text, whereas children’s learning occurs in a multi-modal environment, in concert with other agents who are responsive to the child’s behavior. In one sense, this makes the job of the LM harder, as they lose out on information which seems crucial for language acquisition concerning the situations in which a word is uttered, the timing of which can be highly precise and serves as a cue both for semantic meaning and syntactic category. This thus furthers the anti-nativist import of LM successes: these systems can acquire linguistic knowledge despite not knowing the contexts in which the language they encounter was produced, and so adding this information should only make the child’s job easier, and thus the child in less need of innate advantages. However, one should be careful in describing this as an unmitigated benefit for the child over the LM. Linguistically relevant information is useful, but only if it can be functionally processed and combined with whatever else the child knows. Thus, providing more information can make the child’s job harder (c.f. Trueswell et al. 2013 on the ways in which children ignore information). More data requires more hypotheses and more powerful ways of choosing between them.

Recent work has aimed to test the role of multi-modal data in linguistic acquisition by using multi-model LMs. Vong, Wang, Orhan, and Lake (2024) trained a neural network on data gathered from a camera mounted on a child over the course of about a year and a half, correlated with audio recordings of the language being spoken to the child. Visual and linguistic inputs are encoded by separate neural networks, and then the correlations between the two inputs (e.g., the word ‘ball’ correlating with the visual image of a ball) are used to map vectors from each modality-specific network into a shared vector space, such that vector similarity is increased between correlated linguistic and visual inputs. In a testing task modified from child lexical acquisition studies, they presented the trained network with a selection of novel images, not drawn from the original recording, and a label which correctly applied to just one such image. The vectors generated in response to the label were then compared to those generated in response to each image, and the most similar vector was treated as the system’s “guess” as to which image the label correctly labeled. Despite being trained on a much smaller, and more naturalistic, data set, this system performed comparably to CLIP, a (then) state-of-the-art image classifier (Radford et al. 2021), when given the same task. While, predictably, there are some remaining gaps in the performance of this system, for example doing less well at categorizing images of objects with high visual variability such as toys, and it is yet to be seen how models of this sort generalize to the more abstract components of linguistic knowledge, these results again suggest that more can be extracted from the experience of a developing child than some nativists have claimed.

In addition to ensuring that LMs and children are exposed to the same (magnitude and type) of data, Warstadt and Bowman identify the assumption that LMs are not themselves biased, perhaps in subtle ways, towards the relevant linguistic hypotheses as necessary to leverage LM successes into arguments against nativism. To show that some linguistic phenomena can be mastered without innate linguistic biases, by appeal to an LM which masters them, we must demonstrate that the LM does not itself have such language-specific biases. This assumption is a bit more difficult to verify. Although engineers can ensure that they do not ‘hard code’ any such preferences into the learning architectures they develop, the size and complexity of these systems raises the possibility that they are somehow implicitly found in the systems, prior to training. And of course, commonly used machine learning (ML) systems, such as Transformer architectures, are explicitly designed to process linguistic inputs, with attention mechanisms which function to identify dependencies between non-adjacent elements. It is possible that such architectures do in fact encode something like the bias towards hierarchical structures appealed to in some classic PoS arguments. If this is so, this would undermine some forms of anti-nativist skepticism building on the capacities of these systems. Warstadt and Bowman propose a method for quantifying advantages of this sort, building on the idea that a biased system is one which is particularly likely to adopt a specific hypothesis across a range of possible environments. They cite work by McCoy, Frank, and Linzen (2020) which shows that several different kinds of neural networks do not display a bias towards hierarchically structured representations of linguistic input.

A.4 Language Models and Impossible Languages

This raises a question which has been prominent in these debates: exactly how powerful should an ML system be if it is to raise problems for the nativist? Prominent work (e.g., Moro, Greco, & Cappa 2023) has argued that a major result of modern linguistic theory consists in the identification of impossible languages, languages which, though in principle functional for communicative purposes, are not naturally learnable by developing humans. Powerful pattern-finding engines like LLMs, however, are capable of learning both possible human languages and impossible ones. This, they argue, undermines the utility of LLMs as tools for theoretical linguistics, as understanding human language consists equally in knowing how it can, and how it cannot, work. If LLMs can indeed learn languages that humans cannot, this indicates deep differences both in their internal representations of these languages, and more deeply, in the learning strategies they adopt. Kallini, Papadimitriou, Futrell, Mahowald, and Potts (2024), however, challenge these findings, showing that LLMs do display a preference for some kinds of linguistic system over others, specifically for languages compatible with known rules of human languages over rules unattested in human languages such as reversing the string or moving all even-numbered expressions string-initially.

While these debates about the learning power of LLMs are ongoing, it is interesting to note a tension here. On the one hand, advocates for the relevance of LLMs in theoretical linguistics, like Kallini et al., are keen to show that the natural boundaries around possible languages are reflected in the possible learning outcomes of LLMs, even given atypical inputs. This, it is argued, shows that what they end up learning is more akin to human language than skeptics like Moro and Chomsky suggest. On the other hand, as Warstadt and Bowman point out, if LLMs are to be taken as possibility proofs against nativist theories of language acquisition, it is important that they lack the kinds of innate biases that nativists posit in humans. In order to thread this needle, it seems that it must be shown both that LLMs are better suited to learning human than impossible languages, but that this bias is not language specific but instead stems from more general constraints on pattern recognition and information compression, as discussed in section 6 and section 7 of the main entry.

A.5 Testing Language Model Competence

With these general constraints in place, we can ask: just how good are LLMs at acquiring human language, and what resources do they require in doing so? Millière (2026) identifies three major experimental paradigms used to gauge the extent to which (L)LMs have indeed identified the relevant structural facts about natural language: behavioral studies, probing studies, and interventional studies. Behavioral studies most straightforwardly reflect the way that linguists have traditionally identified the linguistic capacities of human language users, and involve presenting language models with linguistic stimuli and gauging their response. This need not be simply prompting them to generate acceptability judgements, but will instead typically involve measuring the probability they assign to continuations of a string. Linzen, Dupoux, and Goldberg (2016) and Goldberg (2019) applied such a technique to test LMs’ mastery of subject-verb agreement, with the latter finding BERT, an early Transformer-based system which has served as the object of much work in this literature, to perform nearly perfectly, despite complex confounds created by ‘distractors’ intervening between the subject and the main verb which must agree in number. Probing experiments instead focus on interpreting the internal representational states of the system directly, by seeing whether a second machine learning model can be trained directly on the activation patterns of the LM in response to a range of classified stimuli, so as to correctly predict the properties of the former on the basis of the latter. Successful development of such a model is then taken to indicate that the system is indeed identifying the relevant linguistic properties of the stimulus (although see Millière for methodological concerns). Interventional experiments build on probing experiments by manipulating the patterns identified by the probe system in the original language model, and then seeing what effect this has on the system’s behavior. If, it is argued, the probe has correctly identified the LM’s (distributed) representational vehicles for representing some linguistic phenomenon (e.g., number agreement), then altering these vehicles should influence the LM’s proficiency in tasks involving this phenomenon. Successful interventions of this sort show not only that the states of the LM correlate with the target linguistic property, but that they are causally involved in its responses to it, in line with standard philosophical accounts of the conditions under which a system represents some distal state or property (e.g., Shea 2018). There are now massive numbers of studies of these sorts aimed at uncovering the ways that (L)LMs represent linguistic patterns. I refer the reader to Millière (2026) for a recent and thorough overview.

Returning to our main theme, the significance of these results is that if LLMs are to be viewed as challenging nativist claims about language acquisition, it is not enough that they generate the same linguistic outputs as humans (modulo the difference in medium). They must also acquire the same knowledge of language as humans, that is represent the language in the same way. PoS arguments are concerned primarily with what humans know about language, not what they say. If LLMs are capable of generating human-like output, on the basis of training on linguistic data, but these data are not sufficient for showing how humans come to know what they know about language, then the PoS arguments still go through. That is, merely showing that exposure to the relevant data can enable a system to, for example, correctly classify sentences as well-formed or not with respect to the ordering of subjects and auxiliaries, is not sufficient to rebut PoS arguments. What is needed is to show that these data are sufficient to learn the rules and structures that humans use when interpreting and producing sentences with inverted subjects and auxiliaries. In traditional terms, PoS arguments are concerned with the development of linguistic competence (what language users know), not performance (what they say). The hope of probing and interventionist methods is that they can go beyond assessments of the performance of these machines and identify their competence, as it is only by doing so that their relevance to PoS debates can be assessed.

While the matter is an empirical one, Dupre (2021b, 2024) has offered arguments for skepticism about the convergence of human grammars and LM representations of language. The core idea is that machine learning systems fundamentally function to identify and compactly capture patterns in the data, but that compact ways of representing, and even extrapolating from, a data set are unlikely to be the best systems for capturing the language-specific rules known by competent human language users. The reasoning again stems from the competence/performance distinction. The data LMs are trained on reflect all of the myriad influences on linguistic performance, conflating distinct interacting linguistic systems (e.g., syntax, morphology, phonology…) as well as various communicative pressures, the beliefs and intentions of language producers, and more. A grammar, as conceived of by the linguist, on the other hand, is a theory of just one of these inputs. But there is no guarantee that the best way of representing the patterns in the broader, messy, and confounded data set LMs are trained on will incorporate the linguist’s best guess as to the specifically grammatical rules relevant to determining the legitimate and illegitimate constructions of the language. Further, there are various forms of evidence, such as cross-linguistic compatibility, developmental data, explanatory power, and potentially psychological and neurobiological experimental results, which play a role in linguistic theory choice, but which play no role in LM training. To the extent that these are indeed relevant to selection of a grammar, we ought be skeptical that LMs are likely to acquire human-like grammars, even if they become very adept at producing human-like text. They may simply do so in different ways, thus undermining the stronger inferences one can draw concerning PoS arguments from LLM successes.

A radically opposed, and highly influential, line of thinking is pressed in Piantadosi (2024), who argues that the sheer impressiveness of LLMs, specifically their shockingly low rate of grammatical errors, undermines the empirical credentials of traditional (specifically, generative) linguistic theories (although see Katzir 2023 and Volenec & Reiss 2025 for critical responses). If, Piantadosi argues, we judge our best theory by its predictive power, then we should view LLMs not merely as replicating human linguistic behavior, but as providing the best theory of human language (see also Ambridge & Blything 2024). Taking LLMs seriously as theories of language then motivates rejecting a number of assumptions core to generative linguistics, such as the stark differentiation between linguistic domains (e.g., syntax from semantics), the role of innate knowledge/structure in acquisition, and so on. This entry is not the place to evaluate these radical claims, but just to note that they do conform to the overall structure of innateness debates as we have described them: Piantadosi’s arguments involve denying that humans do learn the relevant linguistic facts as assumed by standard PoS arguments, replacing these with the gradable and integrated picture of language we get from interpreting LLMs, which are more plausibly learnable from the PLD. If this is correct, we are owed an account of the data which originally motivated theoretical linguists to posit such structures (e.g., island constraints), and one which is similarly capable of explaining why the data are the way they are (perhaps in line with the above-discussed results on impossible languages within LLMs). More work is needed in showing how LLMs can provide such explanations.