Scale matters: Large language models with billions rather than millions of parameters better match neural representations of natural language

Could LLM AI Technology Be Leveraged in Corpus Linguistic Analysis?

The temporal progression of voltage topographies for all ERPs is presented in Figure S2. To verify that the effects were not driven by one group per duplet type condition, we ran a mixed two-way ANOVA for the average activity in each ROI and significant time window, with duplet type (Word/Part-word) as within-subjects factor and familiarisation as between-subjects factor. Future studies should consider a nlp semantic analysis within-subject design to gain sensitivity to possible interaction effects. Although this is a rich language stimulus, naturalistic stimuli of this kind have relatively low power for modeling infrequent linguistic structures (Hamilton & Huth, 2020). While perplexity for the podcast stimulus continued to decrease for larger models, we observed a plateau in predicting brain activity for the largest LLMs.

Some researchers may be tempted to propose a third step in which they ask the AI to analyze the quantitative results of the coding and report whether the ordinary meaning of “landscaping” includes non-botanical elements. For us, this is a step too far in the direction of Snell-like AI outsourcing—a step toward robo-judging. It would violate our principles of transparency, replicability, and empiricism. And it would outsource crucial decisions about what ordinary meaning is, how much evidence is enough to decide that non-botanical elements are included, and how the data should be used and weighted as part of answering the larger question about meaning. Then, data was segmented from the beginning of each phase into 0.5 s long segments (240 duplets for the Random, 240 duplets for the long Structured, and 600 duplets for the short Structured).

In each experiment, two different structured streams (lists A and B) were used by modifying how the syllables/voices were combined to form the duplets (Table S2). Crucially, the Words/duplets of list A are the Part-words of list B and vice versa any difference between those two conditions can thus not be caused by acoustical differences. Participants were randomly assigned and balanced between lists and Experiments. To control for the different embedding dimensionality across models, we standardized all embeddings to the same size using principal component analysis (PCA) and trained linear encoding models using ordinary least-squares regression (cf. Fig. 2). Scatter plot of max correlation for the PCA + linear regression model and the ridge regression model. For the GPT-Neo model family, the relationship between encoding performance and layer number.

Encoding model performance across electrodes and brain regions

We found that model-brain alignment improves consistently with increasing model size across the cortical language network. However, the increase plateaued after the MEDIUM model for regions BA45 and TP, possibly due to already high encoding correlations for the SMALL model and a small number of electrodes in the area, respectively. Natural Language Processing (NLP) is a rapidly evolving field in artificial intelligence (AI) that enables machines to understand, interpret, and generate human language. NLP is integral to applications such as chatbots, sentiment analysis, translation, and search engines. Data scientists leverage a variety of tools and libraries to perform NLP tasks effectively, each offering unique features suited to specific challenges. Here is a detailed look at some of the top NLP tools and libraries available today, which empower data scientists to build robust language models and applications.

LLMs, such as GPT, use massive amounts of data to learn how to predict and create language, which can then be used to power applications such as chatbots. A simple NLP model can be created using the base of machine learning algorithms like SVM and decision trees. Deep learning architectures include Recurrent Neural Networks, LSTMs, and transformers, which are really useful for handling large-scale NLP tasks. Using these techniques, professionals can create solutions to highly complex tasks like real-time translation and speech processing. Second, we observed no obvious advantage for the linguistic dimension in neonates. This mechanism gives them a powerful tool to create associations between recurrent events.

Building a Career in Natural Language Processing (NLP): Key Skills and Roles

It provides robust language analysis capabilities and is known for its high accuracy. Transformers by Hugging Face is a popular library that allows data scientists to leverage state-of-the-art transformer models like BERT, GPT-3, T5, and RoBERTa for NLP tasks. Judges don’t need to be told the ordinary meaning of a word or phrase—by a human or a computer. They need empirical evidence of how words and phrases are commonly used so they can discern the ordinary meaning of the law by means that are transparent and empirical.

Furthermore, the complexity of what these models learn enables them to process natural language in real-life contexts as effectively as the human brain does. Thus, the explanatory power of these models is in achieving such expressivity based on relatively simple computations in pursuit of a relatively simple objective function (e.g., next-word prediction). As we continue to develop larger, more sophisticated models, the scientific community is tasked with advancing a framework for understanding these models to better understand the intricacies of the neural code that supports natural language processing in the human brain. LLMs, however, contain millions or billions of parameters, making them highly expressive learning algorithms. Combined with vast training text, these models can encode a rich array of linguistic structures—ranging from low-level morphological and syntactic operations to high-level contextual meaning—in a high-dimensional embedding space.

B. For MEDIUM, LARGE, and XL, the percentage difference in correlation relative to SMALL for all electrodes with significant encoding differences. The encoding performance is significantly higher for the bigger models for almost all electrodes across the brain (pairwise t-test across cross-validation folds). Maximum encoding correlations for SMALL and XL for each ROI (mSTG, aSTG, BA44, BA45, and TP area).

The best lag for encoding performance does not vary with model size

Recent work has argued that the “size” of these models—the number of learnable parameters—is critical, as some linguistic competencies only emerge in larger models with more parameters (Bommasani et al., 2021; Kaplan et al., 2020; Manning et al., 2020; Sutton, 2019; Zhang et al., 2021). For instance, in-context learning (Liu et al., 2021; Xie et al., 2021) involves a model acquiring the ability to carry out a task for which it was not initially trained, based on a few-shot examples provided by the prompt. This capability is present in the bigger GPT-3 (Brown et al., 2020) but not in the smaller GPT-2, despite both models having similar architectures. This observation suggests that simply scaling up models produces more human-like language processing.

For each electrode, a p-value was computed as the percentile of the non-permuted encoding model’s maximum value across all lags from the null distribution of 5000 maximum values. Performing a significance test using this randomization procedure evaluates the null hypothesis that there is no systematic relationship between the brain signal and the corresponding word embedding. This procedure yielded a p-value per electrode, corrected for the number of models tested across all lags within an electrode. To further correct for multiple comparisons across all electrodes, we used a false-discovery rate (FDR). This procedure identified 160 electrodes from eight patients in the left hemisphere’s early auditory, motor cortex, and language areas. A. Participants listened to a 30-minute story while undergoing ECoG recording.

Thus, they should compute a 36 × 36 TPs matrix relating each acoustic signal, with TPs alternating between 1/6 within words and 1/12 between words. With this type of computation, we predict infants should fail the task in both experiments since previous studies showing successful segmentation in infants use high TP within words (usually 1) and much fewer elements (most studies 4 to 12) (Saffran and Kirkham, 2018). If speech input is processed along the two studied dimensions in distinct pathways, it enables the calculation of two independent TP matrices of 6×6 between the six voices and six syllables. These computations would result in TPs alternating between 1 and 1/2 for the informative feature and uniform at 1/5 for the uninformative feature, leading to stream segmentation based on the informative dimension. Finally, we would like to point out that it is not natural for a word not to be produced by the same speaker, nor for speakers to have statistical relationships of the kind we used here. Neonates, who have little experience and therefore no (or few) expectations or constraints, are probably better revealers of the possibilities opened by statistical learning than older participants.

Devised the project, performed experimental design, and critically revised the article. Ten patients (6 female, years old) with treatment-resistant epilepsy undergoing intracranial monitoring with subdural grid and strip electrodes for ChatGPT App clinical purposes participated in the study. Two patients consented to have an FDA-approved hybrid clinical research grid implanted, which includes standard clinical electrodes and additional electrodes between clinical contacts.

The design was orthogonal for the Structured streams of Experiment 2 (i.e., TPs between voices alternated between 1 and 0.5, while between syllables were evenly 0.2). The random streams were created by semi-randomly concatenating the 36 tokens to achieve uniform TPs equal to 0.2 over both features. The semi-random concatenation implied that the same element could not appear twice in a row, and the same two elements could not repeatedly alternate more than two times (i.e., the sequence XkXjXkXj, where Xk and Xj are two elements, was forbidden). Notice that with an element, we refer to a duplet when it concerns the choice of the structured feature and to the identity of the second feature when it involves the other feature.

A word-level aligned transcript was obtained and served as input to four language models of varying size from the same GPT-Neo family. For every layer of each model, a separate linear regression encoding model was fitted on a training portion of the story to obtain regression weights that can predict each electrode separately. Then, the encoding models were tested on a held-out portion of the story and evaluated by measuring the Pearson correlation of their predicted signal with the actual signal. Encoding model performance (correlations) was measured as the average over electrodes and compared between the different language models. Critically, there appears to be an alignment between the internal activity in LLMs for each word embedded in a natural text and the internal activity in the human brain while processing the same natural text. You can foun additiona information about ai customer service and artificial intelligence and NLP. If infants at birth compute regularities on the pure auditory signal, this implies computing the TPs over the 36 tokens.

Concepts like probability distributions, Bayes’ theorem, and hypothesis testing, are used to optimize the models. Mathematics, especially linear algebra and calculus, is also important, as it helps professionals understand complex algorithms and neural networks. To use our landscaping example, researchers could train a chatbot to apply the framework we developed for the study in our draft article for coding each instance of “landscaping” on whether the language was used to refer to botanical elements, non-botanical elements, or both. Again the chatbot’s performance on a sample could be evaluated for accuracy against the standard set by human coders who applied the framework to the same sample. The coding framework and prompting language could then be refined with the goal of improving the accuracy of the AI.

LLMs And NLP: Building A Better Chatbot

In fact, adults obtained better results for phoneme structure than for voice structure, perhaps because of an effective auditory normalisation process or the use of a writing code for phonemes but not for voices. It is also possible that the difference between neonates and adults is related to the behavioural test being a more explicit measure of word recognition than the implicit task allowed by EEG recordings. In any case, results ChatGPT show that even adults displayed some learning on the voice duplets. As cluster-based statistics are not very sensitive, we also analysed the ERPs over seven ROIS defined on the grand average ERP of all merged conditions (see Methods). Results replicated what we observed with the cluster-based permutation analysis with similar differences between Words and Part-words for the effect of familiarisation and no significant interactions.

B. Lag with best encoding performance correlation for each electrode, using SMALL and XL model embeddings. Only electrodes with the best lags that fall within 600 ms before and after word onset are plotted. In two experiments, we compared STATISTICAL LEARNING over a linguistic and a non-linguistic dimension in sleeping neonates. We took advantage of the possibility of constructing streams based on the same tokens, the only difference between the experiments being the arrangement of the tokens in the streams. We showed that neonates were sensitive to regularities based either on the phonetic or the voice dimensions of speech, even in the presence of a non-informative feature that must be disregarded.

The same statistical structures were used for both Experiments, only changing over which dimension the structure was applied. The 10 short structured streams lasted 30 seconds each, each duplet appearing a total of 200 times (10 × 20). The same random stream was used for both Experiments, and it lasted 120 seconds. Prior to encoding analysis, we measured the “expressiveness” of different language models—that is, their capacity to predict the structure of natural language. Perplexity quantifies expressivity as the average level of surprise or uncertainty the model assigns to a sequence of words.

These results show that, from birth, multiple input regularities can be processed in parallel and feed different higher-order networks. A more detailed investigation of layerwise encoding performance revealed a log-linear relationship where peak encoding performance tends to occur in relatively earlier layers as both model size and expressivity increase (Mischler et al., 2024). This is an unexpected extension of prior work on both language (Caucheteux & King, 2022; Kumar et al., 2022; Toneva & Wehbe, 2019) and vision (Jiahui et al., 2023), where peak encoding performance was found at late-intermediate layers. Moreover, we observed variations in best relative layers across different brain regions, corresponding to a language processing hierarchy.

Our results show an N400 for both Words and Part-words in the post-learning phase, possibly related to a top-down effect induced by the familiarisation stream. However, the component we observed for duplets presented after the familiarisation streams might result from a related phenomenon. While the main pattern of results between experiments was comparable, we did observe some differences.

AI-based systems can provide 24/7 service, improve a contact center team’s productivity, reduce costs, simulate human behavior during customer interactions and more. Over the past several years, business and customer experience (CX) leaders have shown an increased interest in AI-powered customer journeys. A recent study from Zendesk found that 70% of CX leaders plan to integrate AI into many customer touchpoints within the next two years, while over half of respondents expressed their desire to increase AI investments by 2025. In turn, customer expectations have evolved to reflect these significant technological advancements, with an increased focus on self-service options and more sophisticated bots. Syntax, or the structure of sentences, and semantic understanding are useful in the generation of parse trees and language modelling. NLP is also being used for sentiment analysis, changing all industries and demanding many technical specialists with these unique competencies.

• Scatter plot of max correlation for the PCA + linear regression model and the ridge regression model.
• That said, we see two means of leveraging LLM AIs’ advantages while minimizing these risks.
• Major preprocessing steps include tokenization, stemming, lemmatization, and the management of special characters.
• We compute PCA separately on the training and testing set to avoid data leakage.
• This resulted in a distribution of 5000 values, which was used to determine the significance for all electrodes.

We used electrocorticography (ECoG) to measure neural activity in epilepsy patients while they listened to a 30-minute naturalistic audio story. We fit electrode-wise encoding models using contextual embeddings extracted from each hidden layer of the LLMs to predict word-level neural signals. In line with prior work, we found that larger LLMs better capture the structure of natural language and better predict neural activity. We also found a log-linear relationship where the encoding performance peaks in relatively earlier layers as model size increases.

For these reasons, we believe this approach has too much AI and not enough corpus linguistics. But we are intrigued by the attempt to make corpus linguistics more accessible and user-friendly. In a practical sense, there are many use cases for NLP models in the customer service industry. For example, a business can use NLP-based bots to enable seamless agent routing. When a customer submits a help ticket, your NLP model can easily analyze the language used to divert the customer to the best agent for the task, accelerating issue resolution and delivering better service.

Future work investigating the neural networks involved should implement a within-subject design to gain statistical power. To test this hypothesis, we used electrocorticography (ECoG) to measure neural activity in ten epilepsy patient participants while they listened to a 30-minute audio podcast. Invasive ECoG recordings more directly measure neural activity than non-invasive neuroimaging modalities like fMRI, with much higher temporal resolution. We found that larger language models, with greater expressivity and lower perplexity, better predicted neural activity (Antonello et al., 2023).

The OPT and Llama-2 families are released by MetaAI (Touvron et al., 2023; S. Zhang et al., 2022). For Llama-2, we use the pre-trained versions before any reinforcement learning from human feedback. All models we used are implemented in the HuggingFace environment (Tunstall et al., 2022).

Parsing based on statistical information was revealed by steady-state evoked potentials at the duplet rate observed around 2 min after the onset of the familiarisation stream and by different ERPs to Words and Part-words presented during the test in both experiments. Despite variations in the other dimension, statistical learning was possible, showing that this mechanism operates at a stage when these dimensions have already been separated along different processing pathways. Our results, thus, revealed that linguistic content and voice identity are calculated independently and in parallel. Using near-infra-red spectroscopy (NIRS) and electroencephalography (EEG), we have shown that statistical learning is observed in sleeping neonates (Flo et al., 2022; Fló et al., 2019), highlighting the automaticity of this mechanism. We also discovered that tracking statistical probabilities might not lead to stream segmentation in the case of quadrisyllabic words in both neonates and adults, revealing an unsuspected limitation of this mechanism (Benjamin et al., 2022). Here, we aimed to further characterise the characteristics of this mechanism in order to shed light on its role in the early stages of language acquisition.