"description": "<p><strong>Introduction</strong></p> <p>The European Open Source AI index is an EU-based community-driven public resource on open-source generative AI systems, created for the purposes of cataloguing and scrutinizing systems which claim to be open or open-source. This upload catalogues the index at a specific point in time (2025-05-12).</p> <p>The index is hosted at the Centre of Language and Speech Technology at Radboud University at <a href='osai-index.eu'>osai-index.eu</a>, and is maintained by a small team of academics and community members.</p> <p>The index is based largely off the papers <a href='https://dl.acm.org/doi/abs/10.1145/3571884.3604316'>Opening up ChatGPT: Tracking openness, transparency, and accountability in instruction-tuned text generators</a> and <a href='https://dl.acm.org/doi/abs/10.1145/3630106.3659005'>Rethinking open source generative AI: open-washing and the EU AI Act</a>. To this end, evaluation is done largely based on fourteen key openness criteria.</p> <p> </p> <p><strong>Open-source criteria</strong></p> <p>For this index, we consider as an open-source model any model which describes itself as either 'open-source' or 'open', or for which their positioning in the open-source space makes the model's open-source nature implicit (e.g. <a href='https://huggingface.co/NousResearch/DeepHermes-3-Llama-3-8B-Preview'>DeepHermes</a>). We thus rely largely on a model's own claims to determine whether open-source status is sought. For each model for which a claim to openness is made, we use our evaluation criteria to determine to what degree open-source values are achieved in practice.</p> <p> </p> <p><strong>File contents</strong></p> <p>For each model, the yaml files in our index collect:<br /> (1) some general information about the model,<br /> (2) some information about the organization behind it, and<br /> (3) about 14 dimension of openness.</p> <p>The below list spells out what each of our features contains, as well as openness criteria for openness dimensions.</p> <p>- <strong>System:</strong><br /> - <em>name:</em> Name of the model including eventual version number or size indication, e.g. Llama 3.1 or Olmo-7B-instruct<br /> - <em>link:</em> Link to official model publisher website or, if that does not exist, platform hosting the model.<br /> - <em>type:</em> Model type in one word, e.g. text, video, audio. Multiple keywords possible.<br /> - <em>performanceclass:</em> <a href='https://osai-index.eu/news/performance-classes'>The performance class of the model.</a><br /> - <em>basemodelname:</em> If applicable, name of base model ('foundation model') that was used.<br /> - <em>endmodelname:</em> Name of the model the enduser interacts with.<br /> - <em>endmodellicense:</em> License that applies to enduser interaction with the model.<br /> - <em>releasedate:</em> Earliest release date of the model through any offical source, in YYYY MMM format, e.g. 2024 NOV.<br /> - <strong>Organisation:</strong><br /> - <em>name:</em> The organisation that released the model. Usually synonymous with the model builder.<br /> - <em>link:</em> Link to offical source of information about model release, e.g. offical website or blog.<br /> - <strong>Datasources Basemodel:</strong> Whether data sources for training the base model are comprehensively documented and freely made available.<br /> - <em>closed:</em> Training data sources of base large language model are not open for inspection or shared.<br /> - <em>partial:</em> Some of the training data sources of base large language model are open for inspection or shared.<br /> - <em>open:</em> All training data sources of base large language model are not open for inspection or shared.<br /> - <strong>Datasources Endmodel:</strong> Whether datasources for training the model that the end-user interacts with are comprehensively documented and freely made available.<br /> - <em>closed:</em> Training data sources of the end model are not open for inspection or shared.<br /> - <em>partial:</em> Some of the training data sources of end large language model are open for inspection or shared.<br /> - <em>open:</em> All training data sources of end large language model are not open for inspection or shared.<br /> - <strong>Weights basemodel:</strong> Whether the weights of the base models are made freely available.<br /> - <em>closed:</em> Weights of the base model are not shared.<br /> - <em>partial:</em> Weights of the base model are partially/not fully shared.<br /> - <em>open:</em> Weights of the base model are shared.<br /> - <strong>Weights endmodel:</strong> Whether the weights of the model that the end-user interacts with are made freely available.<br /> - <em>closed:</em> Weights of the user-facing end model are not shared.<br /> - <em>partial:</em> Weights of the user-facing end model are partially/not fully shared.<br /> - <em>open:</em> Weights of the user-facing end model are shared.<br /> - <strong>Training Code: </strong> Whether the source code of datasource processing, model training and tuning is comprehensively and freely made available.<br /> - <em>closed:</em> No source code available.<br /> - <em>partial:</em> Some source code is open.<br /> - <em>open:</em> Project source code openly available and fully open available for inspection.<br /> - <strong>Code Documentation:</strong> Whether the source code of datasource processing, model training and tuning is comprehensively documented.<br /> - <em>closed:</em> Code documentation not available.<br /> - <em>partial:</em> Some components of the system features code documentation, but not every step of base and/or end model training and tuning is documented (irrespective of whether these components are shared).<br /> - <em>open:</em> All components of the system features a comprehensive code documentation.<br /> - <strong>Hardware Architecture Documentation:</strong> Whether the hardware architecture used for datasource processing and model training is comprehensively documented.<br /> - <em>closed:</em> System architecture and model training setup are not documented.<br /> - <em>partial:</em> System architecture and model training setup is partially documented.<br /> - <em>open:</em> System architecture and model training setup is fully documented.<br /> - <strong>Preprint:</strong> Whether archived preprint(s) are available that detail all major parts of the system including datasource processing, model training and tuning steps.<br /> - <em>closed:</em> No archived preprint(s) available.<br /> - <em>partial:</em> Archived preprint(s) that detail some parts of the system including datasource processing, model training and tuning steps.<br /> - <em>open:</em> Archived preprint(s) are available that detail all major parts of the system including datasource processing, model training and tuning steps.<br /> - <strong>Paper:</strong> Whether peer-reviewed scientific publications are available that detail all major parts of the system, including datasource processing, model training and tuning steps.<br /> - <em>closed:</em> No peer-reviewed paper(s) available.<br /> - <em>partial:</em> Peer-reviewed paper(s) detail parts of the software including base models, fine-tuning, or RLHF components.<br /> - <em>open:</em> Peer-reviewed paper(s) are available that cover all parts of the software including base models, fine-tuning, and RLHF components.<br /> - <strong>Model card:</strong> Whether a model card is available in standardized format that provides comprehensive insight on model architecture, training, fine-tuning, and evaluation.<br /> - <em>closed:</em> Model card(s) not available.<br /> - <em>partial:</em> Model card(s) that provide partial insight on model architecture, training, fine-tuning, and evaluation are available.<br /> - <em>open:</em> Model card(s) are available that provide comprehensive insight on model architecture, training, fine-tuning, and evaluation are available.<br /> - <strong>Datasheet:</strong> Whether a datasheet as defined in <a href='https://doi.org/10.1145/3458723'>'Datasheets for Datasets'</a> (Gebru et al. 2021) is available.<br /> - <em>closed:</em> Datasheet(s) are not available.<br /> - <em>partial:</em> Datasheet(s) that provide partial insight on data collection and curation are available.<br /> - <em>open:</em> Datasheet(s) are available that provide comprehensive insight on data collection and curation are available following the standards defined in <a href='https://doi.org/10.1145/3458723'>Datasheets for Datasets</a> by Gebru et al. (2021)<br /> - <strong>Package:</strong> Whether a packaged release of the is model available on a software repository (e.g. a Python Package Index, Homebrew).<br /> - <em>closed:</em> No index software package is available.<br /> - <em>partial:</em> User-oriented code or web-interface is available but not as a versioned package.<br /> - <em>open:</em> A packaged release of the model available on a software repository is available (e.g. a Python Package Index, Homebrew).<br /> - <strong>API:</strong> Whether an API is available that provides unrestricted access to the model (other than security and CDN restrictions).<br /> - <em>closed:</em> No API access.<br /> - <em>partial:</em> Commerial or restricted-access user API is available.<br /> - <em>open:</em> An API available that provides unrestricted access to the model (other than security and CDN restrictions).<br /> - <strong>Licenses:</strong> Whether the project is fully covered by Open Source Initiative (OSI)-approved licenses, including all data sources and training pipeline code.<br /> - <em>closed:</em> The project is not licensed clearly or does not use an Open Source Initiative (OSI)-approved license.<br /> - <em>partial:</em> Only parts of the model and data sources are released under an Open Source Initiative (OSI)-approved license, such as model weights.<br /> - <em>open:</em> The project is fully covered by Open Source Initiative (OSI)-approved license, including all data sources and training pipeline code.</p> <p><strong> </strong></p> <p><strong>Inclusion criteria</strong></p> <p>The index aims to include any instruct-tuned generative AI system or model that is described by the responsible organisation or builder as 'open-source' or 'open', or that is marketed as such by official outlets of the responsible organisation or builder. Generally, the index aims to:</p> <p>- Refer to models by their most recent version and generally focus on the largest available version within a model family. While this approach helps streamline comparisons, it does overlook some nuances. For example, differences in licensing, architecture, or openness between model sizes or incremental updates.</p> <p>- The index is periodically updated by our small team of researchers and community contributors. Updates reflect newly released models and improvements in documentation or licensing information over time. For instance, a model entry may be revised when a preprint becomes available or when a license changes.</p> <p>- Models that span multiple modalities (e.g., text, image, video) may appear in more than one modality category, resulting in multiple entries in the index.</p> <p><strong> </strong></p> <p><strong>Collection</strong></p> <p>Models are collected through a combination of manual curation and community contributions. Sources include:</p> <p>- GitHub issues submitted by users<br /> - Model lists maintained by platforms like Ollama<br /> - Leaderboards such as LLM Arena<br /> - Hugging Face’s most-liked models<br /> - Ongoing individual monitoring of the generative AI landscape.</p> <p>Once a model is added, it is associated with its parent organisation. New models from tracked organisations are continuously monitored, particularly through their Hugging Face profile(s) or official repositories.</p> <p><strong> </strong></p> <p><strong>Uses</strong></p> <p>The Open Source AI Index provides a structured overview of the state of openness in the generative AI ecosystem. It is intended for researchers, policymakers, developers, and advocates seeking to assess how open various AI systems truly are beyond surface-level claims.</p> <p>By cataloguing openness across different dimensions, the index supports more informed debates around transparency, reproducibility, and public accountability in AI. It can also serve as a foundation for further work in responsible AI, dataset documentation, and policy evaluation. We stand open to our data being used in future projects in the field.</p> <p><strong> </strong></p> <p><strong>Maintenance</strong></p> <p>Responsibility for maintenance lies primarily with the Centre for Language and Speech Technology (CLST) at Radboud University. However, the project is designed to be community-driven, and contributions from the wider open-source AI community are encouraged. We aim to move towards a more decentralized maintenance model over time, enabling transparency and shared ownership of the index.</p> <p><strong> </strong></p> <p><strong>Introduction</strong></p> <p>The European Open Source AI index is an EU-based community-driven public resource on open-source generative AI systems, created for the purposes of cataloguing and scrutinizing systems which claim to be open or open-source. This upload catalogues the index at a specific point in time (2025-05-12).</p> <p>The index is hosted at the Centre of Language and Speech Technology at Radboud University at <a href='osai-index.eu'>osai-index.eu</a>, and is maintained by a small team of academics and community members.</p> <p> </p> <p><strong>Composition</strong></p> <p>The index consists of a series of YAML files, containing evaluations of a subset of the most relevant open AI models. We evaluate each model on a variety of criteria. See also the README in our GitHub.</p> <p>The index is self-contained, and is endeavored to be kept fully up-to-date. Periodically, new releases of the index are published to ensure persistent copies at set points in time are made available.</p> <p> </p> <p><strong>Collection</strong></p> <p>We collect models through a combination of <a href='https://github.com/Language-Technology-Assessment/main-database/issues'>community suggestions through GitHub issues</a>, <a href='https://ollama.com/library'>Ollama's model list</a>, LLM arena evaluations at <a href='https://openlm.ai/chatbot-arena/'>openlm.ai</a>, and <a href='https://huggingface.co/models?sort=likes'>a list of HuggingFace's most liked models</a>. Additionally, we endeavor to monitor recent developments in generative AI technologies manually to ensure that all popular models are tracked.</p> <p>For each model we collect, we record its parent organization. New models published by this organization are automatically tracked through the organization's HuggingFace page(s).</p> <p> </p> <p><strong>Uses</strong></p> <p>The primary use case of our index is keeping track of openness in the latest developments in generative AI. We stand open to our data being used in future projects in the field.</p> <p> </p> <p><strong>Maintenance</strong></p> <p>Responsibility for maintenance is primarily held by the Centre of Language and Speech Technology at Radboud University. We endeavor to make the project community-led, meaning that maintenance will eventually be done in a more decentralized manner.</p>",
0 commit comments