New index shows AI models are becoming less transparent

October 19, 2023

Researchers from Stanford, MIT, and Princeton created an index to benchmark the transparency of AI companies and their models. The results weren’t great.

The Foundation Model Transparency Index (FMTI) rates the transparency of 10 foundation model companies according to 100 different aspects of transparency. The criteria are related to how a company builds a foundation model, how it works, and downstream user experience.

Meta’s Llama 2 came out on top however, its score of 54% isn’t exactly shooting out the lights. OpenAI came a close third with a transparency score of 48%. Considering the ongoing open source vs proprietary debate, it’s interesting to see that the gap between these two wasn’t a lot bigger.

Transparency is an important factor when organizations are trying to choose which company’s model to adopt. If your business or educational institution uses an AI model you want to know if you can trust it.

Does the way the model was made align with the principles of your institution? Could you be exposing yourself to litigation down the line because of copyright infringements?

Foundation Model Transparency Index Total Scores - 2023
Foundation Model Transparency Index Total Scores for 2023. Source: Stanford

The index achieves two important things. It shows that we’ve got a long way to go before we have truly transparent AI solutions. And, while we are yet to achieve that, it shows us which companies are doing a better job than others.

The list of criteria makes for interesting reading. It asks questions like, “Is geographic information regarding the people involved in data labor disclosed for each phase of the data pipeline?” Ghost workers have been a contentious ethical fly in the AI ointment.

Another one of the criteria asks, “Is the amount of energy expended in building the model disclosed?” An important point if you want to make an environmentally informed choice between two models.

These issues are undeniably important but poorly disclosed by AI companies. In the introduction to the FMTI, the Center for Research Foundation Models says, “While the societal impact of these models is rising, transparency is on the decline.”

Putting a number to it will hopefully engender some sense of competition among the AI big tech companies to strive toward the top of the FMTI table.

As performance parity between the top models converges, it could well be that the FMTI score could influence the adoption of one model over another.

Join The Future


Clear, concise, comprehensive. Get a grip on AI developments with DailyAI

Eugene van der Watt

Eugene comes from an electronic engineering background and loves all things tech. When he takes a break from consuming AI news you'll find him at the snooker table.


Stay Ahead with DailyAI

Sign up for our weekly newsletter and receive exclusive access to DailyAI's Latest eBook: 'Mastering AI Tools: Your 2024 Guide to Enhanced Productivity'.

*By subscribing to our newsletter you accept our Privacy Policy and our Terms and Conditions