Researchers from Stanford, MIT, and Princeton created an index to benchmark the transparency of AI companies and their models. The results weren’t great.
The Foundation Model Transparency Index (FMTI) rates the transparency of 10 foundation model companies according to 100 different aspects of transparency. The criteria are related to how a company builds a foundation model, how it works, and downstream user experience.
Meta’s Llama 2 came out on top however, its score of 54% isn’t exactly shooting out the lights. OpenAI came a close third with a transparency score of 48%. Considering the ongoing open source vs proprietary debate, it’s interesting to see that the gap between these two wasn’t a lot bigger.
Transparency is an important factor when organizations are trying to choose which company’s model to adopt. If your business or educational institution uses an AI model you want to know if you can trust it.
Does the way the model was made align with the principles of your institution? Could you be exposing yourself to litigation down the line because of copyright infringements?
The index achieves two important things. It shows that we’ve got a long way to go before we have truly transparent AI solutions. And, while we are yet to achieve that, it shows us which companies are doing a better job than others.
The list of criteria makes for interesting reading. It asks questions like, “Is geographic information regarding the people involved in data labor disclosed for each phase of the data pipeline?” Ghost workers have been a contentious ethical fly in the AI ointment.
Another one of the criteria asks, “Is the amount of energy expended in building the model disclosed?” An important point if you want to make an environmentally informed choice between two models.
These issues are undeniably important but poorly disclosed by AI companies. In the introduction to the FMTI, the Center for Research Foundation Models says, “While the societal impact of these models is rising, transparency is on the decline.”
Putting a number to it will hopefully engender some sense of competition among the AI big tech companies to strive toward the top of the FMTI table.
As performance parity between the top models converges, it could well be that the FMTI score could influence the adoption of one model over another.
