Who do you think are the users of AI product CO2e measurement?

[caohongliu]

Understanding who are our users is very important when designing SCI4AI.

[bokelley]

I think that AI app companies will be very hesitant to show CO2 numbers to end users. I think developers will find this to be somewhere between interesting and exciting, because who doesn't like to brag about cutting compile times by 30% or carbon by 40%? I think the corporate user is the most important, especially B2B companies that want to sell to sustainability-first clients or companies that have public sustainability commits themselves.

[jawache]

So a few thoughts here. I agree they will be very hesitant, what are the solutions?

• 1: You don't need an app company to "disclose" their SCI score for someone to be able to compute the SCI score for the app company. This is effectively what researchers have been doing in papers for a while, based on things we can observe, hints that have been published, what can we induce about the emissions. The Impact Framework tool we've developed effectively makes this 100 times easier. You can take what you can observe and then using open source models try to induce the emissions. In parallel to this we are working to build a community of citizen scientists who can measure the emissions of software products, even ones they don't own or manage.

• 2: For example the Tomlinson et al 24 paper models ChatGPT at 1.35g per "Request". That however is a terrible example, and it's been widely criticized. We have an alternative model, in an as yet unpublished paper, which is closer to 20g. But that also is a great example of why open models are so important, we can critique each other and come up with better solutions.

• 3: Most SCI numbers that are currently being calculated are done based on open source models not on disclosed data. For instance cloud carbon footprint is a fully open source tool to measure your GCP/Azure/AWS footprint. You don't need Azure/AWS/GCP to disclose their numbers, you can model it based on usage data. There is an argument to be made about which is more accurate a disclosed number from an organization or a modeled number from an open source model - I'm not so sure it's clear cut that the disclosed number is more accurate.

• 4: The spec can contain information regarding how to compute an end user score (and others, this doesn't have to be an either/or the spec can have multiple functional units). The spec doesn't need to worry about whether people are incentivized to compute it. The ones that are motivated to compute an end user score will find a spec with details of how to compute one very useful.

• 5: Regulations, the EU AI Act specifically has provision for a methodology to be developed to measure AI emissions. To be honest what we are developing here would not be able to be adopted outright in the EU AI Act - the methodology needs to be developed through an EU internal process, but what we discuss and agree here could heavily influence that methodology. The EU might essentially make something like this mandatory.

• 6: Competition. Microsoft was the first organization to come out with a carbon dashboard, then Google and now Amazon. Nothing organizations do better than compete with each other. So perhaps we just need a few companies to start, before there is enough pressure for the rest to follow.

[noahcb]

I agree that in the absence of some incentive companies will be hesitant to show CO2 numbers (particularly raw emissions numbers), but I still think that the end user is as important as any other stakeholder on this list.

As we move towards a world where AI performance is commoditized (particularly at the model level), organizations will look for other areas to compete and there is significant data showing an increase in the number of consumers who want to make decisions based on the environmental impact of products. There is an argument that application or query level CO2e might be too granular to motivate consumer behavior, but I still think we have to be focused on the user because that is what will ultimately motivate organizations to move to large scale adoption (even if they only report aggregate numbers).

[jawache]

I wrote a big reply to @bokelley similar point above, but I agree on the competition front.

My personal thesis is that organizations are basically excellent at competition, that's their function - to compete with their competitors but it has to be driven from customer demand. So customers have to demand an SCI for AI score. If there is demand and one competitor starts disclosing it, it forces the other competitors to start disclosing theirs.

What we seem to have lost in recent years is that competition around sustainability, it used to feel a lot more vibrant several years ago with every company trying to one-up each other on the sustainability front. I would chat to Googlers in mirror roles to the one I had in Microsoft and say what a beautiful competition we are in, if we (Microsoft) loses, it doesn't matter he world wins.