|
on Big Data |
| By: | Antonicelli, Margareth; Drago, Carlo; Costantiello, Alberto; Leogrande, Angelo |
| Abstract: | This study examines income inequality across Italian regions by integrating instrumental variable panel data models, k-means clustering, and machine learning algorithms. Using econometric techniques, we address endogeneity and identify causal relationships influencing regional disparities. K-means clustering, optimized with the elbow method, classifies Italian regions based on income inequality patterns, while machine-learning models, including random forest, support vector machines, and decision tree regression, predict inequality trends and key determinants. Informal employment, temporary employment, and overeducation also play a major role in influencing inequality. Clustering results confirm a permanent North-South economic divide and the most disadvantaged regions are Campania, Calabria, and Sicily. Among the machine learning models, the highest income disparities prediction accuracy comes with the use of Random Forest Regression. The findings emphasize the necessity of education-focused and digitally based policies and reforms of the labor market in an effort to enhance economic convergence. The study portrays the use of a combination of econometric and machine learning methods in the analysis of regional disparities and proposes a solid framework of policy-making with the intention of curbing economic disparities in Italy. |
| Date: | 2025–06–03 |
| URL: | https://d.repec.org/n?u=RePEc:osf:osfxxx:tk87m_v1 |
| By: | Md. Yeasin Rahat; Rajan Das Gupta; Nur Raisa Rahman; Sudipto Roy Pritom; Samiur Rahman Shakir; Md Imrul Hasan Showmick; Md. Jakir Hossen |
| Abstract: | The prediction of foreign exchange rates, such as the US Dollar (USD) to Bangladeshi Taka (BDT), plays a pivotal role in global financial markets, influencing trade, investments, and economic stability. This study leverages historical USD/BDT exchange rate data from 2018 to 2023, sourced from Yahoo Finance, to develop advanced machine learning models for accurate forecasting. A Long Short-Term Memory (LSTM) neural network is employed, achieving an exceptional accuracy of 99.449%, a Root Mean Square Error (RMSE) of 0.9858, and a test loss of 0.8523, significantly outperforming traditional methods like ARIMA (RMSE 1.342). Additionally, a Gradient Boosting Classifier (GBC) is applied for directional prediction, with backtesting on a $10, 000 initial capital revealing a 40.82% profitable trade rate, though resulting in a net loss of $20, 653.25 over 49 trades. The study analyzes historical trends, showing a decline in BDT/USD rates from 0.012 to 0.009, and incorporates normalized daily returns to capture volatility. These findings highlight the potential of deep learning in forex forecasting, offering traders and policymakers robust tools to mitigate risks. Future work could integrate sentiment analysis and real-time economic indicators to further enhance model adaptability in volatile markets. |
| Date: | 2025–06 |
| URL: | https://d.repec.org/n?u=RePEc:arx:papers:2506.09851 |
| By: | Dominik Stempie\'n; Robert \'Slepaczuk |
| Abstract: | This research systematically develops and evaluates various hybrid modeling approaches by combining traditional econometric models (ARIMA and ARFIMA models) with machine learning and deep learning techniques (SVM, XGBoost, and LSTM models) to forecast financial time series. The empirical analysis is based on two distinct financial assets: the S&P 500 index and Bitcoin. By incorporating over two decades of daily data for the S&P 500 and almost ten years of Bitcoin data, the study provides a comprehensive evaluation of forecasting methodologies across different market conditions and periods of financial distress. Models' training and hyperparameter tuning procedure is performed using a novel three-fold dynamic cross-validation method. The applicability of applied models is evaluated using both forecast error metrics and trading performance indicators. The obtained findings indicate that the proper construction process of hybrid models plays a crucial role in developing profitable trading strategies, outperforming their individual components and the benchmark Buy&Hold strategy. The most effective hybrid model architecture was achieved by combining the econometric ARIMA model with either SVM or LSTM, under the assumption of a non-additive relationship between the linear and nonlinear components. |
| Date: | 2025–05 |
| URL: | https://d.repec.org/n?u=RePEc:arx:papers:2505.19617 |
| By: | Dominik Stempie\'n; Janusz Gajda |
| Abstract: | We compare traditional approach of computing logarithmic returns with the fractional differencing method and its tempered extension as methods of data preparation before their usage in advanced machine learning models. Differencing parameters are estimated using multiple techniques. The empirical investigation is conducted on data from four major stock indices covering the most recent 10-year period. The set of explanatory variables is additionally extended with technical indicators. The effectiveness of the differencing methods is evaluated using both forecast error metrics and risk-adjusted return trading performance metrics. The findings suggest that fractional differentiation methods provide a suitable data transformation technique, improving the predictive model forecasting performance. Furthermore, the generated predictions appeared to be effective in constructing profitable trading strategies for both individual assets and a portfolio of stock indices. These results underline the importance of appropriate data transformation techniques in financial time series forecasting, supporting the application of memory-preserving techniques. |
| Date: | 2025–05 |
| URL: | https://d.repec.org/n?u=RePEc:arx:papers:2505.19243 |
| By: | Syoiti Ninomiya; Yuming Ma |
| Abstract: | A new deep-learning neural network architecture based on high-order weak approximation algorithms for stochastic differential equations (SDEs) is proposed. The architecture enables the efficient learning of martingales by deep learning models. The behaviour of deep neural networks based on this architecture, when applied to the problem of pricing financial derivatives, is also examined. The core of this new architecture lies in the high-order weak approximation algorithms of the explicit Runge--Kutta type, wherein the approximation is realised solely through iterative compositions and linear combinations of vector fields of the target SDEs. |
| Date: | 2025–05 |
| URL: | https://d.repec.org/n?u=RePEc:arx:papers:2505.03789 |
| By: | Maximilian Andres (University of Bremen); Lisa Bruttel (University of Potsdam, CEPA) |
| Abstract: | While an influential body of economic literature shows that allowing for communication between firms increases collusion in oligopolies, so far we have only anecdotal evidence on the precise communication content that helps firms to coordinate their behavior. In this paper, we conduct a primary-data meta-study on oligopoly experiments and use a machine learning approach to identify systematic patterns in the communication content across studies. Starting with the communication topics mentioned most often in the literature (agreements, joint benefit, threat of punishment, promise/trust), we use a semi-supervised approach to detect all relevant topics. In a second step, we study the effect of these topics on the rate of collusion among the firms. We find that agreements on specific behavior are decisive for the strong positive effect of communication on collusion, while other communication topics have no effect. |
| Keywords: | collusion, communication, machine learning, meta-study, experiment |
| JEL: | C92 D43 L41 |
| Date: | 2025–05 |
| URL: | https://d.repec.org/n?u=RePEc:pot:cepadp:88 |
| By: | Drago, Carlo; Costantiello, Alberto; Savorgnan, Marco; Leogrande, Angelo |
| Abstract: | This article investigates macroeconomic factors that support the adoption of Artificial Intelligence (AI) technologies by large European Union (EU) enterprises. In this analysis, panel data regression is combined with machine learning to investigate how macroeconomic variables like health spending, domestic credit, exports, gross capital formation, and inflation, along with health spending and trade openness, influence the share of enterprises that adopt at least one type of AI technology (ALOAI). The results of the estimations—based on fixed and random effects models with 151 observations—show that health spending, inflation, and trade and GDP per capita have positively significant associations with adoption, with significant negative correlations visible with and among domestic credit, exports, and gross capital formation. In adjunct to this, the regression of machine learning models (KNN, Boosting, Random Forest) is benchmarked with MSE, RMSE, MAE, MAPE, and R² measures with KNN performing perfectly on all measures, although with some concerns regarding data overfitting. Furthermore, cluster analysis (Hierarchical, Density-Based, Neighborhood-Based) identifies hidden EU country groups with comparable macroeconomic variables and comparable ALOAI. Notably, those with characteristics of high integration in international trade, access to credit, and strong GDP per capita indicate large ALOAI levels, whereas those with macroeconomic volatility and under-investment in innovation trail behind. These findings suggest that securing the adoption of AI is not merely about finance and infrastructure but also about policy alignment and institutional preparedness. This work provides evidence-driven policy advice by presenting an integrated data-driven analytical framework to comprehend and manage AI diffusion within EU industry sectors. |
| Keywords: | Artificial Intelligence Adoption, Macroeconomic Indicators, Panel Data Regression, Machine Learning Models, EU Policy and Innovation. |
| JEL: | C23 C45 E22 L86 O33 |
| Date: | 2025–06–08 |
| URL: | https://d.repec.org/n?u=RePEc:pra:mprapa:124973 |
| By: | Jiahao Yang; Ran Fang; Ming Zhang; Jun Zhou |
| Abstract: | In high-frequency trading (HFT), leveraging limit order books (LOB) to model stock price movements is crucial for achieving profitable outcomes. However, this task is challenging due to the high-dimensional and volatile nature of the original data. Even recent deep learning models often struggle to capture price movement patterns effectively, particularly without well-designed features. We observed that raw LOB data exhibits inherent symmetry between the ask and bid sides, and the bid-ask differences demonstrate greater stability and lower complexity compared to the original data. Building on this insight, we propose a novel approach in which leverages the Siamese architecture to enhance the performance of existing deep learning models. The core idea involves processing the ask and bid sides separately using the same module with shared parameters. We applied our Siamese-based methods to several widely used strong baselines and validated their effectiveness using data from 14 military industry stocks in the Chinese A-share market. Furthermore, we integrated multi-head attention (MHA) mechanisms with the Long Short-Term Memory (LSTM) module to investigate its role in modeling stock price movements. Our experiments used raw data and widely used Order Flow Imbalance (OFI) features as input with some strong baseline models. The results show that our method improves the performance of strong baselines in over 75$% of cases, excluding the Multi-Layer Perception (MLP) baseline, which performed poorly and is not considered practical. Furthermore, we found that Multi-Head Attention can enhance model performance, particularly over shorter forecasting horizons. |
| Date: | 2025–05 |
| URL: | https://d.repec.org/n?u=RePEc:arx:papers:2505.22678 |
| By: | Pierre Brugi\`ere; Gabriel Turinici |
| Abstract: | Option pricing theory, such as the Black and Scholes (1973) model, provides an explicit solution to construct a strategy that perfectly hedges an option in a continuous-time setting. In practice, however, trading occurs in discrete time and often involves transaction costs, making the direct application of continuous-time solutions potentially suboptimal. Previous studies, such as those by Buehler et al. (2018), Buehler et al. (2019) and Cao et al. (2019), have shown that deep learning or reinforcement learning can be used to derive better hedging strategies than those based on continuous-time models. However, these approaches typically rely on a large number of trajectories (of the order of $10^5$ or $10^6$) to train the model. In this work, we show that using as few as 256 trajectories is sufficient to train a neural network that significantly outperforms, in the Geometric Brownian Motion framework, both the classical Black & Scholes formula and the Leland model, which is arguably one of the most effective explicit alternatives for incorporating transaction costs. The ability to train neural networks with such a small number of trajectories suggests the potential for more practical and simple implementation on real-time financial series. |
| Date: | 2025–05 |
| URL: | https://d.repec.org/n?u=RePEc:arx:papers:2505.22836 |
| By: | Queiroz, Rafael L.; Martins, Joberto S. B. Prof. Dr. (Salvador University - UNIFACS) |
| Abstract: | Artificial intelligence and machine learning are widely utilized and offer significant benefits in various fields of knowledge, including healthcare. However, there is an important barrier to disseminating machine learning among healthcare professionals, which primarily stems from their unfamiliarity with programming and computing concepts. The development strategy known as 'low-code, ' when applied to software development, encompasses frameworks and tools that, in short, make application development more accessible to professional communities. The low-code strategy simplifies the software development process. This strategy is particularly relevant for smart cities that seek to develop approaches that enhance the efficiency, humanity, and sustainability of cities, thereby contributing to the achievement of the United Nations' Sustainable Development Goals (SDGs). This article positions the low-code strategy, implemented through the PyCaret framework, as a key element of innovation and contribution to the development of health systems utilizing machine learning in smart cities. The paper presents the low-code strategy through a case study that evaluates the incidence and correlation of occupational activities with the occurrence of cancer in Brazil using an anomaly detection algorithm. The article's contributions include positioning the low-code strategy as an element of innovation in smart cities and presenting a case study that serves as a reference for developing applications with machine learning in the healthcare sector. The case study presented, in turn, presents a differentiated approach to detecting cancer using an anomaly detection algorithm and reiterates correlations between types of cancer and occupational activities. |
| Date: | 2024–10–01 |
| URL: | https://d.repec.org/n?u=RePEc:osf:socarx:9wsqn_v1 |
| By: | Geyue Sun; Xiao Liu; Tomas Williams; Roberto Samaniego |
| Abstract: | We construct a novel event-level Capital Control Measures (CCM) dataset covering 196 countries from 1999 to 2023 by leveraging prompt-based large language models (LLMs). The dataset enables event study analysis and cross-country comparisons based on rich policy attributes, including action type, intensity, direction, implementing entity, and other multidimensional characteristics. Using a two-step prompt framework with GPT-4.1, we extract structured information from the IMF's Annual Report on Exchange Arrangements and Exchange Restrictions (AREAER), resulting in 5, 198 capital control events with 27 annotated fields and corresponding model reasoning. Secondly, to facilitate real-time classification and extension to external sources, we fine-tune an open-source Meta Llama 3.1-8B model, named CCM-Llama, trained on AREAER change logs and final status reports. The model achieves 90.09\% accuracy in category classification and 99.55\% in status prediction. Finally, we apply the CCM dataset in an empirical application: an event study on China, Australia, and the US. The results show that inward capital control measures significantly reduce fund inflows within one month, and restrictive policies tend to have stronger effects than liberalizing ones, with notable heterogeneity across countries. Our work contributes to the growing literature on the use of LLMs in economics by providing both a novel high-frequency policy dataset and a replicable framework for automated classification of capital control events from diverse and evolving information sources. |
| Date: | 2025–05 |
| URL: | https://d.repec.org/n?u=RePEc:arx:papers:2505.23025 |
| By: | Lucien Chaffa; Martin Trépanier; Thierry Warin |
| Abstract: | This study investigates the potential of machine learning (ML) methods to enhance the estimation of the gravity model, a cornerstone of international trade analysis that explains trade flows based on economic size and distance. Traditionally estimated using methods such as the Poisson Pseudo Maximum Likelihood (PPML) approach, gravity models often struggle to fully capture nonlinear relationships and intricate interactions among variables. Leveraging data from Canada and the US, one of the largest bilateral trading relationships in the world, this paper conducts a comparative analysis of traditional and ML approaches. The findings reveal that ML methods can significantly outperform traditional approaches in predicting trade flows, offering a robust alternative for capturing the complexities of global trade dynamics. These results underscore the value of integrating ML techniques into trade policy analysis, providing policymakers and economists with improved tools for decision-making. Cette étude examine le potentiel des méthodes d'apprentissage automatique (ML) pour améliorer l'estimation du modèle de gravité, une méthode clé de l'analyse du commerce international qui explique les flux commerciaux en fonction de la taille de l'économie et de la distance. Traditionnellement estimés à l'aide de méthodes telles que l'approche du pseudo-maximum de vraisemblance de Poisson (PPML), les modèles de gravité ont souvent du mal à saisir pleinement les relations non linéaires et les interactions complexes entre les variables. En s'appuyant sur les données du Canada et des États-Unis, l'une des relations commerciales bilatérales les plus importantes au monde, cet article effectue une analyse comparative des approches traditionnelles et des approches par apprentissage automatique. Les résultats révèlent que les méthodes de ML peuvent être nettement plus performantes que les approches traditionnelles pour prédire les flux commerciaux, offrant ainsi une alternative robuste pour saisir les complexités de la dynamique du commerce mondial. Ces résultats soulignent la valeur de l'intégration des techniques de ML dans l'analyse de la politique commerciale, fournissant aux décideurs politiques et aux économistes des outils améliorés pour la prise de décision. |
| Keywords: | Gravity Model, PPML Machine Learning, International Trade, Trade Policy Analysis, Modèle de gravité, PPML, apprentissage automatique, commerce international, analyse de la politique commerciale |
| JEL: | F10 F14 C13 C45 |
| Date: | 2025–05–20 |
| URL: | https://d.repec.org/n?u=RePEc:cir:cirwor:2025s-14 |
| By: | Chung I Lu; Julian Sester; Aijia Zhang |
| Abstract: | We propose a novel distributionally robust $Q$-learning algorithm for the non-tabular case accounting for continuous state spaces where the state transition of the underlying Markov decision process is subject to model uncertainty. The uncertainty is taken into account by considering the worst-case transition from a ball around a reference probability measure. To determine the optimal policy under the worst-case state transition, we solve the associated non-linear Bellman equation by dualising and regularising the Bellman operator with the Sinkhorn distance, which is then parameterized with deep neural networks. This approach allows us to modify the Deep Q-Network algorithm to optimise for the worst case state transition. We illustrate the tractability and effectiveness of our approach through several applications, including a portfolio optimisation task based on S\&{P}~500 data. |
| Date: | 2025–05 |
| URL: | https://d.repec.org/n?u=RePEc:arx:papers:2505.19058 |
| By: | Jasper Rou |
| Abstract: | Option pricing often requires solving partial differential equations (PDEs). Although deep learning-based PDE solvers have recently emerged as quick solutions to this problem, their empirical and quantitative accuracy remain not well understood, hindering their real-world applicability. In this research, our aim is to offer actionable insights into the utility of deep PDE solvers for practical option pricing implementation. Through comparative experiments in both the Black--Scholes and the Heston model, we assess the empirical performance of two neural network algorithms to solve PDEs: the Deep Galerkin Method and the Time Deep Gradient Flow method (TDGF). We determine their empirical convergence rates and training time as functions of (i) the number of sampling stages, (ii) the number of samples, (iii) the number of layers, and (iv) the number of nodes per layer. For the TDGF, we also consider the order of the discretization scheme and the number of time steps. |
| Date: | 2025–05 |
| URL: | https://d.repec.org/n?u=RePEc:arx:papers:2505.05121 |
| By: | Kizilirmak, Jasmin M. (German Center for Neurodegenerative Diseases, Göttingen, Germany); Peter, Frauke |
| Abstract: | The race to professorship is influenced by many factors, ranging from individual achievements to systemic structures. Prior research has underscored the significance of aspects such as academic productivity, institutional affiliations, and personal choices in shaping academic careers, but comprehensive analyses in specific contexts remain limited. The last two decades have seen strategic shifts within the German academic system, most notably with the introduction of junior and tenure-track professorships intended to offer a more secure career path. Yet, uncertainties persist regarding the determinants of career trajectories to tenured university professorship. Addressing this gap, this paper uses unique and novel longitudinal data of German professors to elucidate the various pathways to professorship. Based on a sample of individuals who successfully secured full university professorship, we estimate comprehensive models using machine learning to determine relevant factors that predict the time needed to reach this position. Key correlates are obtaining a habilitation, having held positions with leadership responsibilities, and disciplinary fields. |
| Date: | 2025–05–16 |
| URL: | https://d.repec.org/n?u=RePEc:osf:socarx:khfgj_v1 |
| By: | Agam Shah; Siddhant Sukhani; Huzaifa Pardawala; Saketh Budideti; Riya Bhadani; Rudra Gopal; Siddhartha Somani; Michael Galarnyk; Soungmin Lee; Arnav Hiray; Akshar Ravichandran; Eric Kim; Pranav Aluru; Joshua Zhang; Sebastian Jaskowski; Veer Guda; Meghaj Tarte; Liqin Ye; Spencer Gosden; Rutwik Routu; Rachel Yuh; Sloka Chava; Sahasra Chava; Dylan Patrick Kelly; Aiden Chiang; Harsit Mittal; Sudheer Chava |
| Abstract: | Central banks around the world play a crucial role in maintaining economic stability. Deciphering policy implications in their communications is essential, especially as misinterpretations can disproportionately impact vulnerable populations. To address this, we introduce the World Central Banks (WCB) dataset, the most comprehensive monetary policy corpus to date, comprising over 380k sentences from 25 central banks across diverse geographic regions, spanning 28 years of historical data. After uniformly sampling 1k sentences per bank (25k total) across all available years, we annotate and review each sentence using dual annotators, disagreement resolutions, and secondary expert reviews. We define three tasks: Stance Detection, Temporal Classification, and Uncertainty Estimation, with each sentence annotated for all three. We benchmark seven Pretrained Language Models (PLMs) and nine Large Language Models (LLMs) (Zero-Shot, Few-Shot, and with annotation guide) on these tasks, running 15, 075 benchmarking experiments. We find that a model trained on aggregated data across banks significantly surpasses a model trained on an individual bank's data, confirming the principle "the whole is greater than the sum of its parts." Additionally, rigorous human evaluations, error analyses, and predictive tasks validate our framework's economic utility. Our artifacts are accessible through the HuggingFace and GitHub under the CC-BY-NC-SA 4.0 license. |
| Date: | 2025–05 |
| URL: | https://d.repec.org/n?u=RePEc:arx:papers:2505.17048 |
| By: | Francesco Morri; H\'el\`ene Le Cadre; Pierre Gruet; Luce Brotcorne |
| Abstract: | We characterize zonal ancillary market coupling relying on noncooperative game theory. To that purpose, we formulate the ancillary market as a multi-leader single follower bilevel problem, that we subsequently cast as a generalized Nash game with side constraints and nonconvex feasibility sets. We determine conditions for equilibrium existence and show that the game has a generalized potential game structure. To compute market equilibrium, we rely on two exact approaches: an integrated optimization approach and Gauss-Seidel best-response, that we compare against multi-agent deep reinforcement learning. On real data from Germany and Austria, simulations indicate that multi-agent deep reinforcement learning achieves the smallest convergence rate but requires pretraining, while best-response is the slowest. On the economics side, multi-agent deep reinforcement learning results in smaller market costs compared to the exact methods, but at the cost of higher variability in the profit allocation among stakeholders. Further, stronger coupling between zones tends to reduce costs for larger zones. |
| Date: | 2025–05 |
| URL: | https://d.repec.org/n?u=RePEc:arx:papers:2505.03288 |
| By: | Jiaxiang Chen; Mingxi Zou; Zhuo Wang; Qifan Wang; Dongning Sun; Chi Zhang; Zenglin Xu |
| Abstract: | Financial decision-making presents unique challenges for language models, demanding temporal reasoning, adaptive risk assessment, and responsiveness to dynamic events. While large language models (LLMs) show strong general reasoning capabilities, they often fail to capture behavioral patterns central to human financial decisions-such as expert reliance under information asymmetry, loss-averse sensitivity, and feedback-driven temporal adjustment. We propose FinHEAR, a multi-agent framework for Human Expertise and Adaptive Risk-aware reasoning. FinHEAR orchestrates specialized LLM-based agents to analyze historical trends, interpret current events, and retrieve expert-informed precedents within an event-centric pipeline. Grounded in behavioral economics, it incorporates expert-guided retrieval, confidence-adjusted position sizing, and outcome-based refinement to enhance interpretability and robustness. Empirical results on curated financial datasets show that FinHEAR consistently outperforms strong baselines across trend prediction and trading tasks, achieving higher accuracy and better risk-adjusted returns. |
| Date: | 2025–06 |
| URL: | https://d.repec.org/n?u=RePEc:arx:papers:2506.09080 |
| By: | Lijie Ding; Egang Lu; Kin Cheung |
| Abstract: | We introduce a fast and flexible Machine Learning (ML) framework for pricing derivative products whose valuation depends on volatility surfaces. By parameterizing volatility surfaces with the 5-parameter stochastic volatility inspired (SVI) model augmented by a one-factor term structure adjustment, we first generate numerous volatility surfaces over realistic ranges for these parameters. From these synthetic market scenarios, we then compute high-accuracy valuations using conventional methodologies for two representative products: the fair strike of a variance swap and the price and Greeks of an American put. We then train the Gaussian Process Regressor (GPR) to learn the nonlinear mapping from the input risk factors, which are the volatility surface parameters, strike and interest rate, to the valuation outputs. Once trained, We use the GPR to perform out-of-sample valuations and compare the results against valuations using conventional methodologies. Our ML model achieves very accurate results of $0.5\%$ relative error for the fair strike of variance swap and $1.7\% \sim 3.5\%$ relative error for American put prices and first-order Greeks. More importantly, after training, the model computes valuations almost instantly, yielding a three to four orders of magnitude speedup over Crank-Nicolson finite-difference method for American puts, enabling real-time risk analytics, dynamic hedging and large-scale scenario analysis. Our approach is general and can be extended to other path-dependent derivative products with early-exercise features, paving the way for hybrid quantitative engines for modern financial systems. |
| Date: | 2025–05 |
| URL: | https://d.repec.org/n?u=RePEc:arx:papers:2505.22957 |
| By: | Zikun Ye; Hema Yoganarasimhan |
| Abstract: | Large Language Models (LLMs) are increasingly used in systems that retrieve and summarize content from multiple sources, such as search engines and AI assistants. While these models enhance user experience by generating coherent summaries, they obscure the contributions of original content creators, raising concerns about credit attribution and compensation. We address the challenge of valuing individual documents used in LLM-generated summaries. We propose using Shapley values, a game-theoretic method that allocates credit based on each document's marginal contribution. Although theoretically appealing, Shapley values are expensive to compute at scale. We therefore propose Cluster Shapley, an efficient approximation algorithm that leverages semantic similarity between documents. By clustering documents using LLM-based embeddings and computing Shapley values at the cluster level, our method significantly reduces computation while maintaining attribution quality. We demonstrate our approach to a summarization task using Amazon product reviews. Cluster Shapley significantly reduces computational complexity while maintaining high accuracy, outperforming baseline methods such as Monte Carlo sampling and Kernel SHAP with a better efficient frontier. Our approach is agnostic to the exact LLM used, the summarization process used, and the evaluation procedure, which makes it broadly applicable to a variety of summarization settings. |
| Date: | 2025–05 |
| URL: | https://d.repec.org/n?u=RePEc:arx:papers:2505.23842 |
| By: | Stephen J. Lee; Cailinn Drouin |
| Abstract: | We present a novel framework for high-resolution forecasting of residential heating and electricity demand using probabilistic deep learning models. We focus specifically on providing hourly building-level electricity and heating demand forecasts for the residential sector. Leveraging multimodal building-level information -- including data on building footprint areas, heights, nearby building density, nearby building size, land use patterns, and high-resolution weather data -- and probabilistic modeling, our methods provide granular insights into demand heterogeneity. Validation at the building level underscores a step change improvement in performance relative to NREL's ResStock model, which has emerged as a research community standard for residential heating and electricity demand characterization. In building-level heating and electricity estimation backtests, our probabilistic models respectively achieve RMSE scores 18.3\% and 35.1\% lower than those based on ResStock. By offering an open-source, scalable, high-resolution platform for demand estimation and forecasting, this research advances the tools available for policymakers and grid planners, contributing to the broader effort to decarbonize the U.S. building stock and meeting climate objectives. |
| Date: | 2025–05 |
| URL: | https://d.repec.org/n?u=RePEc:arx:papers:2505.22873 |
| By: | Hinrichs, Nicolás (Max Planck Institute for Human Cognitive and Brain Sciences); Hartwigsen, Gesa; Guzman, Noah |
| Abstract: | EEG hyperscanning offers valuable insights into neural synchrony during social interactions, yet traditional node-based network metrics may overlook critical topological features. This perspective paper introduces Forman-Ricci curvature, a novel edge-based geometric metric, to characterize time-varying inter-brain coupling networks. Unlike conventional methods, Forman-Ricci curvature provides a quantitative measure of information routing, i.e. capturing how neural network structures expand or contract during dynamic interactions. We outline how this method can be implemented for the analysis of task-specific dual-EEG data; by constructing dynamic networks via a sliding window approach the evolution of network states through changes in curvature distributions is enabled. We propose Forman-Ricci network entropy, a scalar metric derived from the Shannon entropy of curvature distributions, to detect phase transitions in neural dynamics. Additionally, we propose a framework to simulate biophysically realistic dual-brain activity to validate results and optimise algorithm selection for source-space estimation. Our method effectively extends the two-person neuroscience framework by enabling its real-time implementation in multimodal experimental paradigms, offering a novel perspective on information routing within interactive neural systems. By capturing dynamic shifts in inter-brain network states, this approach enables further understanding of the neurobiological process that underlie the reciprocity of social interaction. |
| Date: | 2025–06–04 |
| URL: | https://d.repec.org/n?u=RePEc:osf:osfxxx:abx8u_v1 |
| By: | Daniel F. Villarraga; Ricardo A. Daziano |
| Abstract: | Discrete choice models (DCMs) are used to analyze individual decision-making in contexts such as transportation choices, political elections, and consumer preferences. DCMs play a central role in applied econometrics by enabling inference on key economic variables, such as marginal rates of substitution, rather than focusing solely on predicting choices on new unlabeled data. However, while traditional DCMs offer high interpretability and support for point and interval estimation of economic quantities, these models often underperform in predictive tasks compared to deep learning (DL) models. Despite their predictive advantages, DL models remain largely underutilized in discrete choice due to concerns about their lack of interpretability, unstable parameter estimates, and the absence of established methods for uncertainty quantification. Here, we introduce a deep learning model architecture specifically designed to integrate with approximate Bayesian inference methods, such as Stochastic Gradient Langevin Dynamics (SGLD). Our proposed model collapses to behaviorally informed hypotheses when data is limited, mitigating overfitting and instability in underspecified settings while retaining the flexibility to capture complex nonlinear relationships when sufficient data is available. We demonstrate our approach using SGLD through a Monte Carlo simulation study, evaluating both predictive metrics--such as out-of-sample balanced accuracy--and inferential metrics--such as empirical coverage for marginal rates of substitution interval estimates. Additionally, we present results from two empirical case studies: one using revealed mode choice data in NYC, and the other based on the widely used Swiss train choice stated preference data. |
| Date: | 2025–05 |
| URL: | https://d.repec.org/n?u=RePEc:arx:papers:2505.18077 |