|
on Artificial Intelligence |
| By: | Polachek, Solomon (Binghamton University, New York); Romano, Kenneth (Binghamton University, New York); Tonguc, Ozlem (State University of New York) |
| Abstract: | Do large language models (LLMs)—such as ChatGPT 3.5, ChatGPT 4.0, and Google's Gemini 1.0 Pro—simulate human behavior in the context of the Prisoner's Dilemma (PD) game with varying stake sizes? This paper investigates this question, examining how LLMs navigate scenarios where self-interested behavior of all players results in less preferred outcomes, offering insights into how LLMs might "perceive" human decision-making. Through a replication of Yamagishi et al. (2016) "Study 2, " we analyze LLM responses to different payoff stakes and the influence of stake order on cooperation rates. LLMs demonstrate sensitivity to these factors, and some LLMs mirror human behavior only under very specific circumstances, implying the need for cautious application of LLMs in behavioral research. |
| Keywords: | Prisoner's Dilemma, cooperation, payoff stakes, artificial intelligence |
| JEL: | D01 C72 C90 |
| Date: | 2024–12 |
| URL: | https://d.repec.org/n?u=RePEc:iza:izadps:dp17521 |
| By: | Chevalier, Arnaud (Royal Holloway, University of London); Orzech, Jakub (University of London); Stankov, Petar (University of London) |
| Abstract: | Grading and providing feedback are two of the most time-consuming activities in education. We developed a randomised controlled trial (RCT) to test whether they could be performed by generative artificial intelligence (Gen-AI). We randomly allocated undergraduate students to feedback provided either by a human instructor, ChatGPT 3.5, or ChatGPT 4. Our results show that: (i) Students treated with the freely accessible ChatGPT 3.5 received lower grades in subsequent assessments than their peers in the control group who always received human feedback; (ii) No such penalty was observed for ChatGPT 4. Separately, we tested the capacity of Gen-AI to grade student work. Gen-AI grades and ranks were significantly different than human-generated grades. Overall, while the newest LLM helps learning as well as a human, its ability to grade student work is still inferior. |
| Keywords: | feeback, grading, Artificial Intelligence, learning with Gen-AI |
| JEL: | A22 C93 I23 I24 |
| Date: | 2024–12 |
| URL: | https://d.repec.org/n?u=RePEc:iza:izadps:dp17511 |
| By: | Manuel Hoffmann; Sam Boysel; Frank Nagle; Sida Peng; Kevin Xu |
| Abstract: | Recent advances in artificial intelligence (AI) technology demonstrate considerable potential to complement human capital intensive activities. While an emerging literature documents wide-ranging productivity effects of AI, relatively little attention has been paid to how AI might change the nature of work itself. How do individuals, especially those in the knowledge economy, adjust how they work when they start using AI? Using the setting of open source software, we study individual level effects that AI has on task allocation. We exploit a natural experiment arising from the deployment of GitHub Copilot, a generative AI code completion tool for software developers. Leveraging millions of work activities over a two year period, we use a program eligibility threshold to investigate the impact of AI technology on the task allocation of software developers within a quasi-experimental regression discontinuity design. We find that having access to Copilot induces such individuals to shift task allocation towards their core work of coding activities and away from non-core project management activities. We identify two underlying mechanisms driving this shift - an increase in autonomous rather than collaborative work, and an increase in exploration activities rather than exploitation. The main effects are greater for individuals with relatively lower ability. Overall, our estimates point towards a large potential for AI to transform work processes and to potentially flatten organizational hierarchies in the knowledge economy. |
| Keywords: | generative artificial intelligence, digital work, open source software, knowledge economy |
| JEL: | H40 O30 J00 |
| Date: | 2024 |
| URL: | https://d.repec.org/n?u=RePEc:ces:ceswps:_11479 |
| By: | Voraprapa Nakavachara; Tanapong Potipiti; Thanee Chaiwat |
| Abstract: | Generative AI technologies such as ChatGPT, Gemini, and MidJourney have made remarkable progress in recent years. Recent literature has documented ChatGPT’s positive impact on productivity in areas where it has strong expertise—attributable to extensive training datasets—such as the English language and Python/SQL programming. However, the literature is still limited regarding ChatGPT’s performance in areas where its capabilities could still be further enhanced. In this paper, we asked participants to perform writing analysis tasks in a non-English language (specifically, Thai) and math & data analysis tasks using a less frequently used programming package (specifically Stata). The findings suggest that, on average, participants performed better using ChatGPT in terms of scores and time taken to complete the tasks. However, a detailed examination reveals that 34% of participants saw no improvement in writing analysis tasks, and 42% did not improve in math & data analysis tasks when employing ChatGPT. Further investigation indicated that higher-ability participants, as proxied by their econometrics grades, were the ones who performed worse in writing analysis tasks when using ChatGPT. We also found evidence that participants with better digital skills performed better with ChatGPT. This research provides insights on the impact of generative AI. Thus, relevant parties can make informed decisions regarding appropriate strategies, policies, and educational systems. It also highlights the critical role of human skills in addressing and complementing the limitations of AI. |
| Keywords: | ChatGPT; Generative AI; Large Language Models; Labor Productivity |
| JEL: | A20 D24 J24 O33 |
| Date: | 2025–01 |
| URL: | https://d.repec.org/n?u=RePEc:pui:dpaper:229 |
| By: | David J. Deming; Christopher Ong; Lawrence H. Summers |
| Abstract: | This paper explores past episodes of technological disruption in the US labor market, with the goal of learning lessons about the likely future impact of artificial intelligence (AI). We measure changes in the structure of the US labor market going back over a century. We find, perhaps surprisingly, that the pace of change has slowed over time. The years spanning 1990 to 2017 were less disruptive than any prior period we measure, going back to 1880. This comparative decline is not because the job market is stable today but rather because past changes were so profound. General-purpose technologies (GPTs) like steam power and electricity dramatically disrupted the twentieth-century labor market, but the changes took place over decades. We argue that AI could be a GPT on the scale of prior disruptive innovations, which means it is likely too early to assess its full impacts. Nonetheless, we present four indications that the pace of labor market change has accelerated recently, possibly due to technological change. First, the labor market is no longer polarizing— employment in low- and middle-paid occupations has declined, while highly paid employment has grown. Second, employment growth has stalled in low-paid service jobs. Third, the share of employment in STEM jobs has increased by more than 50 percent since 2010, fueled by growth in software and computer-related occupations. Fourth, retail sales employment has declined by 25 percent in the last decade, likely because of technological improvements in online retail. The post-pandemic labor market is changing very rapidly, and a key question is whether this faster pace of change will persist into the future. |
| JEL: | E24 J21 |
| Date: | 2025–01 |
| URL: | https://d.repec.org/n?u=RePEc:nbr:nberwo:33323 |
| By: | Venkat Ram Reddy Ganuthula |
| Abstract: | This paper introduces Agency-Driven Labor Theory as a new theoretical framework for understanding human work in AI-augmented environments. While traditional labor theories have focused primarily on task execution and labor time, ADLT proposes that human labor value is increasingly derived from agency - the capacity to make informed judgments, provide strategic direction, and design operational frameworks for AI systems. The paper presents a mathematical framework expressing labor value as a function of agency quality, direction effectiveness, and outcomes, providing a quantifiable approach to analyzing human value creation in AI-augmented workplaces. Drawing on recent work in organizational economics and knowledge worker productivity, ADLT explains how human workers create value by orchestrating complex systems that combine human and artificial intelligence. The theory has significant implications for job design, compensation structures, professional development, and labor market dynamics. Through applications across various sectors, the paper demonstrates how ADLT can guide organizations in managing the transition to AI-augmented operations while maximizing human value creation. The framework provides practical tools for policymakers and educational institutions as they prepare workers for a labor market where value creation increasingly centers on agency and direction rather than execution. |
| Date: | 2024–12 |
| URL: | https://d.repec.org/n?u=RePEc:arx:papers:2501.01448 |
| By: | Yuval Rymon |
| Abstract: | AI is transforming human labor at an unprecedented pace - improving 10$\times$ per year in training effectiveness. This paper analyzes how society can adapt to AI-driven human-labor automation (HLA), using Bernardi et al.'s societal adaptation framework. Drawing on literature from general automation economics and recent AI developments, the paper develops a "threat model." The threat model is centered on mass unemployment and its socioeconomic consequences, and assumes a non-binary scenario between full AGI takeover and swift job creation. The analysis explores both "capability-modifying interventions" (CMIs) that shape how AI develops, and "adaptation interventions" (ADIs) that help society adjust. Key interventions analyzed include steering AI development toward human-complementing capabilities, implementing human-in-the-loop requirements, taxation of automation, comprehensive reorientation of education, and both material and social substitutes for work. While CMIs can slow the transition in the short-term, significant automation is inevitable. Long-term adaptation requires ADIs - from education reform to providing substitutes for both the income and psychological benefits of work. Success depends on upfront preparation through mechanisms like "if-then commitments", and crafting flexible and accurate regulation that avoids misspecification. This structured analysis of HLA interventions and their potential effects and challenges aims to guide holistic AI governance strategies for the AI economy. |
| Date: | 2024–12 |
| URL: | https://d.repec.org/n?u=RePEc:arx:papers:2501.03092 |
| By: | Lennart Ante; Aman Saggu |
| Abstract: | Following an analysis of existing AI-related exchange-traded funds (ETFs), we reveal the selection criteria for determining which stocks qualify as AI-related are often opaque and rely on vague phrases and subjective judgments. This paper proposes a new, objective, data-driven approach using natural language processing (NLP) techniques to classify AI stocks by analyzing annual 10-K filings from 3, 395 NASDAQ-listed firms between 2011 and 2023. This analysis quantifies each company's engagement with AI through binary indicators and weighted AI scores based on the frequency and context of AI-related terms. Using these metrics, we construct four AI stock indices-the Equally Weighted AI Index (AII), the Size-Weighted AI Index (SAII), and two Time-Discounted AI Indices (TAII05 and TAII5X)-offering different perspectives on AI investment. We validate our methodology through an event study on the launch of OpenAI's ChatGPT, demonstrating that companies with higher AI engagement saw significantly greater positive abnormal returns, with analyses supporting the predictive power of our AI measures. Our indices perform on par with or surpass 14 existing AI-themed ETFs and the Nasdaq Composite Index in risk-return profiles, market responsiveness, and overall performance, achieving higher average daily returns and risk-adjusted metrics without increased volatility. These results suggest our NLP-based approach offers a reliable, market-responsive, and cost-effective alternative to existing AI-related ETF products. Our innovative methodology can also guide investors, asset managers, and policymakers in using corporate data to construct other thematic portfolios, contributing to a more transparent, data-driven, and competitive approach. |
| Date: | 2025–01 |
| URL: | https://d.repec.org/n?u=RePEc:arx:papers:2501.01763 |
| By: | Flavio Calvino; Luca Fontanelli |
| Abstract: | In this work we characterise French firms using artificial intelligence (AI) and explore the link between AI use and productivity. We distinguish AI users that source AI from external providers (AI buyers) from those developing their own AI systems (AI developers). AI buyers tend to be larger than other firms, but this relation is explained by ICT-related variables. Conversely, AI developers are larger and younger beyond ICT. Other digital technologies, digital skills and infrastructure play a key role for AI use, with AI developers leveraging more specialised ICT human capital than AI buyers. Overall, AI users tend to be more productive, however this is related to the self-selection of more productive and digital-intensive firms into AI use. This is not the case for AI developers, for which the positive link between AI use and productivity remains evident beyond selection. |
| Keywords: | technology diffusion, artificial intelligence, digitalisation, productivity |
| JEL: | D20 J24 O14 O33 |
| Date: | 2024 |
| URL: | https://d.repec.org/n?u=RePEc:ces:ceswps:_11466 |
| By: | Bruno Deffains (Université Paris Panthéon Sorbonne; CRED); Frédéric Marty (Université Côte d'Azur, France; GREDEG CNRS) |
| Abstract: | The implementation of generative artificial intelligence in legal services offers undeniable efficiency gains, but also raises fundamental issues for law firms. These challenges can be categorised along a broad continuum, ranging from changes in business lines to changes in the competitive environment and the internal organisation of law firms. This paper considers the risks that law firms face in terms of both the quality of the services they provide and perceived competition, both horizontally and vertically, considering possible relationships of dependency on suppliers of large language models and cloud infrastructures. |
| Keywords: | generative artificial intelligence, legal services, accountability, competition, vertical relationships |
| JEL: | L42 L86 |
| Date: | 2025–01 |
| URL: | https://d.repec.org/n?u=RePEc:gre:wpaper:2025-01 |
| By: | Cova, Joshua; Schmitz, Luuk |
| Abstract: | The emergence of generative AI models is rapidly changing the social sciences. Much has now been written on the ethics and epistemological considerations of using these tools. Meanwhile, AI-powered research increasingly makes its way to preprint servers. However, we see a gap between ethics and practice: while many researchers would like to use these tools, few if any guides on how to do so exist. This paper fills this gap by providing users with a hands-on application written in accessible language. The paper deals with what we consider the most likely and advanced use case for AI in the social sciences: text annotation and classification. Our application guides readers through setting up a text classification pipeline and evaluating the results. The most important considerations concern reproducibility and transparency, open-source versus closed-source models, as well as the difference between classifier and generative models. The take-home message is this: these models provide unprecedented scale to augment research, but the community must take seriousely open-source and locally deployable models in the interest of open science principles. Our code to reproduce the example can be accessed via Github. |
| Date: | 2024–12–20 |
| URL: | https://d.repec.org/n?u=RePEc:osf:osfxxx:r3qng |
| By: | Galasso, Vincenzo (Bocconi University); Nannicini, Tommaso (European University Institute); Nozza, Debora (Bocconi University) |
| Abstract: | Understanding individuals' beliefs, preferences, and motivations is essential in social sciences. Recent technological advancements—notably, large language models (LLMs) for analyzing open-ended responses and the diffusion of voice messaging— have the potential to significantly enhance our ability to elicit these dimensions. This study investigates the differences between oral and written responses to open-ended survey questions. Through a series of randomized controlled trials across three surveys (focused on AI, public policy, and international relations), we assigned respondents to answer either by audio or text. Respondents who provided audio answers gave longer, though lexically simpler, responses compared to those who typed. By leveraging LLMs, we evaluated answer informativeness and found that oral responses differ in both quantity and quality, offering more information and containing more personal experiences than written responses. These findings suggest that oral responses to open-ended questions can capture richer, more personal insights, presenting a valuable method for understanding individual reasoning. |
| Keywords: | survey design, open-ended questions, large language models, beliefs |
| JEL: | C83 D83 |
| Date: | 2024–11 |
| URL: | https://d.repec.org/n?u=RePEc:iza:izadps:dp17488 |
| By: | Aromí J. Daniel; Heymann Daniel |
| Abstract: | We propose a method to generate “synthetic surveys” that reveal policymakers’ perceptions and narratives. This exercise is implemented using 80 time-stamped Large Language Models (LLMs) fine-tuned with FOMC meetings’ transcripts. Given a text input, fine-tuned models identify highly likely responses for the corresponding FOMC meeting. We demonstrate the value of this tool in three different tasks: measurement of perceived economic conditions, evaluation of transparency in Central Bank communication and extraction of policymaking narratives. Our analysis covers the housing bubble and the subsequent Great Recession (2003-2012). For the first task, LLMs are prompted to generate phrases that describe economic conditions. The resulting outputs show policymakers informational advantage. Anticipatory ability increases as models are prompted to discuss future scenarios and financial conditions. To analyze transparency, we compare the content of each FOMC meeting minutes to content generated synthetically through the corresponding fine-tuned LLM. The evaluation suggests the tone of each meeting is transmitted adequately by the corresponding minutes. In the third task, LLMs produce narratives that show policymakers’ views on their responsibilities and their understanding of main forces shaping macroeconomic dynamics. |
| JEL: | E58 E47 |
| Date: | 2024–11 |
| URL: | https://d.repec.org/n?u=RePEc:aep:anales:4707 |
| By: | Leek, Lauren Caroline (European University Institute); Bischl, Simeon |
| Abstract: | Although central bank communication is a core monetary policy and accountability tool for central banks, little is known about what shapes it. This paper develops and tests a theory regarding a previously unconsidered variable: central bank independence (CBI). We argue that increases in CBI alter the pressures a central bank faces, compelling them to address these pressures to maintain their reputation. We fine-tune and validate a Large Language Model (Google's Gemini) to develop novel textual indices of policy pressures regarding monetary policy communication of central banks in speeches of 100 central banks from 1997 to 2023. Employing a staggered difference-in-differences and an instrumental variable approach, we find robust evidence that an increase in independence decreases monetary pressures and increases financial pressures discussed in monetary policy communication. These results are not, as generally is assumed, confounded by general changes in communication over time or singular events, in particular, the Global Financial Crisis. |
| Date: | 2024–11–25 |
| URL: | https://d.repec.org/n?u=RePEc:osf:socarx:yrhka |
| By: | Li, Chao; Keeley, Alexander Ryota; Takeda, Shutaro; Seki, Daikichi; Managi, Shunsuke |
| Abstract: | We create a large language model with high accuracy to investigate the relatedness between 12 environmental, social, and governance (ESG) topics and more than 2 million news reports. The text match pre-trained transformer (TMPT) with 138, 843, 049 parameters is built to probe whether and how much a news record is connected to a specific topic of interest. The TMPT, based on the transformer structure and a pre-trained model, is an artificial intelligence model trained by more than 200, 000 academic papers. The cross-validation result reveals that the TMPT’s accuracy is 85.73%, which is excellent in zero-shot learning tasks. In addition, combined with sentiment analysis, our research monitors news attitudes and tones towards specific ESG topics daily from September 2021 to September 2023. The results indicate that the media is increasing discussion on social topics, while the news regarding environmental issues is reduced. Moreover, towards almost all topics, the attitudes are gradually becoming positive. Our research highlights the temporal shifts in public perception regarding 12 key ESG issues: ESG has been incrementally accepted by the public. These insights are invaluable for policymakers, corporate leaders, and communities as they navigate sustainable decision-making. |
| Keywords: | ESG; News; Natural Language Processing; Pre-trained Transformer; Data Mining; Machine Learning |
| JEL: | G0 H0 M1 |
| Date: | 2024–11 |
| URL: | https://d.repec.org/n?u=RePEc:pra:mprapa:122757 |