News
-
Oct. 7, 2024:
We are excited to announce the release of our benchmark, FAMMA v1.0. Our accompanying paper FAMMA: A Benchmark for Financial Domain Multilingual Multimodal Question Answering and dataset have also been published.
- May. 27, 2023: Glad to share our paper WeaverBird: Empowering Financial Decision-Making with Large Language Model, Knowledge Base, and Search Engine, an intelligent dialogue system designed specifically for the finance domain.
Dataset Statistics
Our dataset consists of 1,758 meticulously collected multimodal questions. The questions encompass three heterogeneous image types - tables, charts and text & math screenshots - and span eight subfields in finance, comprehensively covering topics across major asset classes. Additionally, all the questions are categorized by three difficulty levels — easy, medium, and hard - and are available in three languages — English, Chinese, and French. Furthermore, the questions are divided into two types: multiple-choice and open questions.
Distribution of question subfields across difficulty levels.
Distribution of question in different languages across difficulty levels.
Distribution of question types across difficulty levels.
Submission
Please follow the Submission Guideline (below) and contact famma.bench@gmail.com for test evaluation. Ususally, we will return your results in 10 days!
Citation
@article{xue2024famma,
title={FAMMA: A Benchmark for Financial Domain Multilingual Multimodal Question Answering},
author={Siqiao Xue and Tingting Chen and Fan Zhou and Qingyang Dai and Zhixuan Chu and Hongyuan Mei},
journal={arXiv preprint arXiv:2410.04526},
year={2024},
url={https://arxiv.org/abs/2410.04526}
}