It's a bit like the old David Foster Wallace quote where he asks to fish. “How's the water boys?” and gets the response, “what the hell is water?” In the same way, it's easy to overlook the server sitting somewhere in a data center connected by ethernet and fiber links. Linked together by switches and arranged in racks, all of which have been carefully designed and engineered to enable so much of our digital lives.

Today I have Lisa and DJ with me who will tell us all about the Open Compute project, how it helps not just us, but the whole industry move faster with hardware and crucially how it helps lower emissions. DJ and Lisa, welcome to the Meta Tech Podcast.

DJ: Thank you for having us.

Pascal: Absolutely. So I want to first talk a little bit about you. Maybe you could both tell me how long you've been at Meta and what you did before. And Lisa, can we start with you?

Lisa: Yeah, sure. Hi, I am Lisa. I'm a systems engineer at Meta. I've been at Meta for about six years, and I currently work on the data infrastructure that supports our fleet sustainability goal. So my role involves developing methods and model to calculate it, hardware emission, estimating the environmental impact of new hardware design and identifying lower impact alternatives when possible.

Pascal: That is quite a lot and we will get more into this, but DJ, how long have you been here and what did you do before you joined Meta and the Open Compute Project?

DJ: Uh, absolutely. It's funny, I just, uh, celebrated my Meta anniversary yesterday. I've been at Meta a little over seven years. Seven years in one day. I joined the infrastructure team and my role was to support growth of open ecosystem, to develop technologies which go into our data centers, around the world.

And that's pretty much the role I have been, um, undertaking over the last seven years. The scope and responsibilities have grown, uh, just with the growth of our ecosystem, growth of our infrastructure, and I'm happy to share more about it as we go through, uh, in the discussion.

Pascal: Right. I think we should probably start with the elephant in the room because most of the people listening to this will probably be software engineers and might not know what the Open Compute Project or OCP is. So can one of you maybe give us a quick rundown of what the project encompasses?

DJ: Yeah, I, I'm happy to do that. So, Open Compute Project, or OCP as it's commonly known in the industry now, was a project which was launched in 2011 by Meta. At that time known as Facebook, working with Intel, Rackspace, Goldman Sachs, and few other companies. And at that time, the goal for Facebook or Meta was to democratize its design, which going to the data center.

And we had just launched our initial designs in Prineville data center and our power usage, efficiency numbers were much better than industry and Meta in its spirit to open source. Many of these designs decided to launch this, uh, open source project, uh, and called it, uh, Open Compete Project. So it's genesis is in our spirit to share design and knowhow and to spread best practices, uh, within the data center industry.

And now we have been up and running for over the last 14 years.

Pascal: That is a fairly long time, and it feels like a very clear parallel to the continued open sourcing efforts that we've had within Facebook and now Meta where a lot of the software that we use to power our internal infrastructure is also open sourced, and not just infrastructure, also mobile apps. Many, many things out there.

Okay, so can you talk a bit about the scale? Because as I understand it, it's not quite the same kind of narrow scale anymore that existed 14 years ago. So what does it look like today?

DJ: Yeah, I can certainly give some more color on that as well. In fact, pre OCP, period. Our entire data centers had no more than 20, 30, 40,000 servers at the most, and today that number has gone by multiple orders of magnitude and that actually is one of the motivations behind building these kind of open source technologies because when you're building things at such a large scale, you need to have standard way of approaching the infrastructure.

You cannot have a proprietary bespoke custom designs, uh, because they don't scale as well as standard things. So our scale, just when we thought, has grown immensely. We are nowhere close to where we are going next, in the next, five, 10 years with the growth of AI, so every year, the scale continues to grow and the future continues to surprise us in terms of the demand. And we don't see any end to it any time soon.

Pascal: Got it. So. Before we started talking about this, I had this idea in my head that OCP basically means we share some sketches of servers that fit really nicely into a certain rack, and that makes it very nicely standardized, but that is not anywhere near what it actually covers today. Can you give us a bit of an idea of what the entire ecosystem of OCP covers today in terms of different components?

DJ: Absolutely. And years spot on. Actually, if you look at the name Open Compute Project, we started with the notion of open compute, and our first project in 2011 was opening up the server design and that was it. The server project was the first project launched in OCP. Over the last 14 years, we have grown, because more servers go into the data hall. Today we have at the top level, 12 top level projects, and they cover other areas like networking, storage, uh, data center facilities, liquid cooling, and so on and so forth. Sustainability, et cetera. So the scope has grown. Uh, you ask a question about the ecosystem. The open compute project covers the entire gamut of the players in the ecosystems spanning from silicon providers to system builders to rack integrators to data center operators. So every one of the players in each of these verticals, and there are hundreds of companies in each of these verticals if not more, they come and join with the community in open compute. To solve a certain piece of problem, uh, which basically goes on to build a value chain, uh, of the systems which get delivered to the end users, uh, in the data center.

So you would have companies building networking silicon, or compute silicon or GPU silicon. You would have companies building one view server designs, uh, or networking designs, uh, and entire rack and cooling systems. And then companies, which are delivering, deploying. And then companies which are also using all of this gear for running certain workloads.

Meta actually interestingly, now has grown to the point where it plays a role pretty much across all of these, we have our own silicon design. We build systems, we have liquid cooling projects, and we are also the consumer of the devices and systems which go into our data center.

Pascal: Yeah, it's pretty amazing. I've been here now for nine years and it was very different. The stages at which we were involved in this back then compared to now where our own silicon is being designed and deployed in our data centers. DJ briefly touched on the requirements that you get once you reach a certain scale where you need uniformity and you can't have little one-off components to slot into your data center.

But can you go a bit deeper into the differences between open and closed hardware ecosystems and why we ended up choosing the former?

DJ: This is again a very common question and you have to look at it from the mental model of a strategic approach which companies take. With this kind of scale, companies have two major strategies: you can choose to go down the path of open, or you can choose to go and build things on a proprietary, uh, basis. Regarding technologies, which are very important, but which are not differentiating, and infrastructure has a lot of such technologies.

It makes sense to actually build things in open because it allows you to have, uh, ability to deploy at scale. It allows you to have the ability to service technologies in a standardized fashion rather than in a full fashion when something goes wrong. And we also want to deploy these technologies globally and most other companies do.

So in such cases where the proprietary of the technology is not that critical to the business needs and standardization has value. Openness is always a right choice, and this is the reason why most of the companies now are looking at standard open approach as a strategic play to deploy their capital.

Open compute is gaining momentum because of this precise shift across the industry. Meta identified this decades before, other companies kind of followed, and now we can see the whole industry shifting in this direction.

Pascal: Right, which is really exciting, pushing things into that particular direction. And there's one additional benefit, and that is that we can actually talk about this stuff that you have been designing or. Been involved in, which is fun because so often there are internal core projects that are very hard to publicly discuss.

So can you talk about a few examples of hardware projects that have recently come out of OCP?

DJ: Absolutely, so one of my roles at Meta is to work across engineering teams where brilliant folks have ideas to go and build something in open, and my role is to guide them to land those ideas in the most effective way. In 2018 Meta launched Open Accelerator Module Project working with Nvidia and Microsoft.

And this has become a defacto standard for accelerator form factor, which gets deployed in open source fashion within OCP gear. So this is a very clear example. Uh, a while back before even AI boom happened, uh, we also launched during 2019 timeframe. Uh, chiplet project, which was to build, uh, standard interfaces for silicon chiplets so companies can mix and match, which has become a big project within OCP.

And it has also led to industry projects such as UCIE, very recent example at Global Summit, which was held two weeks back, middle of October this year, we launched ethernet for scale up networking as a major, effort working with other companies. And our goal, again is to standardize approaches in ethernet to deliver efficient scale up networking.

There are other projects beyond hardware. In 2019, 2020 Meta was instrumental in launching sustainability as a major effort within OCP. Before that, there was no such effort in OCP and now Meta is deeply involved in this effort. Lisa, my colleague here, is deeply involved and she represents Meta on the sustainability side.

So there are many such examples. There are many other Metamates, as we call them, involved in leading various project work streams within OCP. Liquid cooling is another example of area. OCP Nick 3.0 is another example. So there multiple examples. Mt. Diablo is another project which Meta and Microsoft work together to disaggregate a power rack.

Those are some of the immediate examples which come to my mind.

Pascal: And the fact that this is an industry-wide effort, especially in sustainability, being able to shift potentially all the players that are in the space at the same time is really exciting. Lisa, can you talk about maybe a project that came out of the sustainability space to highlight what it exactly is that you're doing there?

Lisa: Of course. Thank you. One thing that DJ mentioned is the AI boom. And so with AI it's both a challenge and a catalyst for sustainability because, the problem is to make the AI boom happen, we need to purchase a lot of hardware. And this hardware has a carbon footprint. So what we're trying to work on is a set of rules to understand how we can account for Scope Three Emissions of our IT hardware. I think OCP helped a lot because Scope Three Emission is something that is for the whole value chain of the industry. So having all the players together in a room discuss how we can reduce our carbon footprint is the most important thing.

And OCP has two different work streams trying to tackle this challenge.

So I've always been following OCPs innovation, um, since I've been at meta looking at what, uh, innovation is coming from that very big forum. And what I really like is that it's a space where all the different companies are sharing innovation and align all on the same topics. So I think that is very important.

And on the topic I specifically work on, uh, sustainability, it's very important to have alignment across the industry on different aspect of sustainability. So this is, for me, the real motivation. Uh, I think it's even stronger than that. We cannot achieve any sustainability goals if the industry is not aligned together.

So OCP is the perfect forum, um, to reach that kind of alignment.

Pascal: That makes a lot of sense. DJ, what is, was it for you? You obviously came to this a little earlier than Lisa, but what motivated you to join OCP?

DJ: So I have been involved in OCP since 2012 in my previous company. I was instrumental in bringing them on board to join Open Complete Project. I do believe in power of open source and the community driven projects over long run. Linux is a very clear example, and Meta's history is grounded in Lamp Stack, which was instrumental in launching Meta's first website.

So I was at a point in my career where I wanted to find way to give back. I also wanted to use my technical background to get involved in areas where I can have an impact, which is much wider than just one company. And when Meta reached out, uh, because of my involvement in OCP for an opportunity here, it was the most exciting thing I could ask for.

Uh, having ability to have impact at such a large scale. Within an open source environment is one of the key factors for me joining Meta. Every day I work with absolutely smartest people within the company and across the industry where each one is trying to bring some important changes or move the industry in some meaningful fashion forward.

Enabling this in some small way is a very satisfying experience. I also get the opportunity to take very different technical domains on a day-to-day basis. Someday I'm working on networking. Some, I'm working on sustainability. Another day I am working on silicon and chiplets, compute…the list is endless.

So this keeps me very excited because I am a perennial learner about things and this role and this opportunity also gives me the ability to learn on a daily basis from a wide community. So the ability to give back as well as to learn from the community in an open fashion and see everybody succeed is very gratifying.

The journey is often, uh, not a straight line. It is windy, but the destination is always, beautiful and that's what motivates me on a day-to-day basis.

Pascal: I found this very relatable. I joined Meta because I wanted to work on a specific open source project that was about to start, and obviously much smaller scale, especially when you talk about something like specific UI framework designs for Android devices and so on than what you are tackling here, but still the idea of changing something.

The way people approach a certain problem in our industry and trying to shift the entire way people think about it, I think is really fascinating. And there are very few places where you can have that sort of impact. But now I want to talk a bit about sustainability. There is obviously a lot of talk at the moment about the need to grow a physical footprint to serve our growing, growing user base and new demands specifically for AI. So I wanted to ask, how do we balance this with our existing sustainability commitments?

Lisa: So basically one thing that is a given is that the growth, especially for AI, is inevitable. So that's something that we also had discussions about, can we actually purchase less for sustainability? It's better but this is actually a given that we cannot do that.

So what we're trying to do is, given that we're going to grow our footprint, what can we do to find the right balance and create good design? So the sustainability impact or the environmental impact is minimal. So we are working on the full value chain from the mineral extraction to how our users are actually, using our system and how our software is running efficiently to reduce our carbon footprint from the whole the whole chain.

And we have little projects here and there that is actually connecting the dots to make sure that we can have a balance and the end goal is to reduce our impact. So I can give some examples here. Some things that we're trying to do is reuse some of our hardware or extend the life of other pieces of hardware we have, purchase more sustainable components and run our inferences, our different types of algorithm, more efficiently so we can make sure that we purchase exactly what we need and we're not in over consumption of hardware.

Pascal: Right, and I feel like there is an interesting shift in this discussion to what is mostly discussed in the mainstream, which is electricity. And there's a lot of really interesting work happening in this space. A few years ago, during Earth Week, we had a few episodes about this and that, for instance, involves matching electricity generation with consumption in different places across the country or the world. I want to talk about hardware because to me it's really interesting. Can you just discuss a little why electricity does not cover the whole picture when it comes to carbon emissions?

Lisa: Uh, yeah, sure, so electricity is really important, especially, uh, the energy we use in general. But what is really important in terms of sustainability is understanding our whole footprint on the whole value chain. And the hardware itself has a significant carbon footprint because we extract mineral to actually manufacture this hardware, which is then going to be transported across the world, hopefully not to our data centers.

So when you look at the full lifecycle, you realize that embodied emission, this is how we call the emissions that are being produced by the manufacturing process of any goods we purchase are actually very complex to estimate because it comes from extracting the minerals from manufacturing what the energy used by the manufacturer to create the component that we're then using is something that we are as a company responsible for.

So it means that in terms of sustainability, electricity is important, but everything that actually contributed to creating a piece of hardware that we purchased and was creating for our need has to be taken into account and is very hard actually to estimate.

DJ: I would like to add to that as well, and this was something I shared with OCP community when I gave a keynote in 2022, introducing sustainability as the fifth tenet within OCP. We had four guiding principles and we added sustainability then, and in that keynote, there are three things I pointed out which were important for community to focus on.

So as Lisa mentioned, embodied carbon is very important and this is actually guided through the Paris Accord and trying to lower the temperatures. That is all related to the carbon emission in the atmosphere. So these things are all tied and having a proper handle on what is the carbon in your equipment and in your system and data centers is important and that goes beyond electricity.

The second piece, which I talked about was also circularity. And circularity relates to how can we increase the cycle time of the equipment because in many cases you can find productivity of equipment even beyond where the original use case was, and that can help you make better sustainable choices.

And then the third piece, which we talked about in that, discussion was measurements. And I gave example of PUE, which was a standard method to measure data center, which is actually not grounded in principles of sustainability. PUE is grounded in principles of being efficient and everybody's going to be efficient.

When I leave this room, I want to turn the electricity off in this room. That's sustainable, but that's essentially a common sense in being efficient, right? So these are the three things which go and transcend beyond electricity. Electricity generally is considered as a proxy, and that's why it gets the biggest signal because it's a envelope which encompasses everything which goes underneath.

So if you are deploying a one gigawatt of power. There is lot of things which go underneath inside the data center to support that one gigawatt and that one gigawatt number in some course manner captures that notion of the effort, which is done. But there are granular parts within the sustainability effort within OCP where we actually go and create impact for the industry.

Pascal: I've always found it, always found it curious that we standardize now on just calling data centers by gigawatts because it, it is so hard to actually get a grasp on what this actually means, but in the end, having some sort of standardized language to talk about the size, um, is probably still quite helpful side now I think.

Maybe we can get into some of the more media details, because you already talked about common sense approaches that everybody does, but I think first we need to do a quick recap because Lisa, you already mentioned different scopes of emissions, and I'll just give it a try and you can correct me. But scope one is basically everything that we do directly, so like heating in our offices or fuel used on sites, backup generators, and that kind of stuff.

Scope two would then be indirect emissions from electricity that we use to power things. And scope three is where it gets really interesting because that's basically everything else across the value chain. And you already mentioned that this might be a hardware product that we purchase from somebody where we have no direct involvement in making it, but somebody else emits covered emissions along the value chain.

Is that about right?

Lisa: Yes, it’s exactly that. And this is where OCP is very important because when we purchase something from someone else, we need this alignment and transparency to be able to understand how we can work on a sustainable way.

Pascal: Got it. And our goal as a company remains that we want to be net zero by 2030, so that means across all scopes we want to have all the emissions canceled out. So now there is some interesting work going on about standardizing scope three emission accounting. So why is that even necessary? Why is there not a standard in place already for measuring exactly what happens in scope three?

Lisa: Um, that's a very good question. We do have some standards, but the problem is we have a lot of different rules and standards to the point that we don't know what we should use when we are talking about scope three emission in the context of the IT industry. So there is a GHT protocol that creates a methodology defining what is a lifecycle assessment.

Lifecycle assessment is the calculation method we use to understand for each stage of the hardware lifecycle the emission produced to create one of the steps of the life of this component. So the issue we have is because there is a lack of official standard in the IT industry where when we're trying to understand our emission to be able to reduce them, we’re facing a lot of challenges and one of them is the difference in data quality that we can get from different suppliers when we're trying to collect information from their own scope three or scope one and two emissions.

So to give a little bit of context, Meta is reporting in a voluntary way their emission in every year and we're being audited for that. So you can find our scope 3, 1, 2, and 3 emission in the sustainability report that we publish every year. And this is done very rigorously. We follow the GHG protocol but what we're lacking is really understanding of where the emission is coming from.

If we look at the full value chain. So this is a work that we have done. Two or three last years trying to understand what is the best way for us to understand our scope three emission and work and we actually act. And the previous commonly used method in the industry was to use a spend based methodology. So how it works is you have, uh, an invoice of the product you have purchased, and there is a set of coefficient that, um, the, I need to look exactly which one, uh, is creating the standard. So I don't say any issue, any problem, but so there is a set of coefficient called the environmentally extending input output coefficients that are being created by the Environmental Protection Agency.

And so how it works is we use this coefficient to transform a bill. Invoice into Scope three emission. The problem that is facing is double. The first one is that it's giving a false incentive that the only way we have to reduce our emission is to spend less money, which is not true. And the second issue is that it's actually very inaccurate and subjected to market fluctuations, so, which was previously a commonly used method, doesn't work anymore. And especially in the era of AI because the AI components are very, very expensive due to, um, like primary resources shortage, and it doesn't translate very well into what is the scope three emission.

So this is why we had to move away, the whole industry is actually moving away from this previously commonly used method and trying to move back to what the GHG protocol is advising, which is using lifecycle assessment. And the problem is conducing lifecycle lifecycle assessment is not easy.

It's very long. So we need to create other methodologies that allows us to move away from this EIO method. And be able to leverage the knowledge we have from lifecycle assessment. And this is what we're working on at OCP in the OCP product category rule work stream. We're trying to create a set of very precise rule of what should be accounted for for any type of hardware.

For instance CPU, GPU, memory, what should be included in the scope three emission calculation and what is the calculation method we should use and do it that in a way that we can scale it because we have a lot of hardware and we need to be able to aggregate all this information from the tiny screw to the whole fleet and be able to have actionable information on that.

Pascal: That is a lot, and I'm a big fan of LCAs or lifecycle assessments because they're fantastic when you're having discussions and people are like, oh, but this wind turbine is gonna just as much as if you dug out the coal and burned it. It's like, no, no it doesn't. You can look at this here and it will exactly break down when the lines cross basically between the emissions of a wind turbine and some core.

So you already broke this down by a few components that you find in different systems. Can you talk a bit about what components in a server actually have the highest emissions and how you work with suppliers to lower them?

Lisa: Yeah, absolutely. So what our data has shown us, and this was kind of a surprise was that semiconductor was really, really, really very important in terms of carbon footprint for our servers. So we do have a lot of mechanical component. For instance, the rack frame, the chassis, but semiconductor are around 85% of the scope three emission of a rack, which means that all of our efforts shifted from trying to act a little bit everywhere to trying to really influence the industry as a whole to reduce their carbon footprint on the semiconductor category in particular. And one component in particular is memory - the DRAM is because we have a lot of these components so it's very dense in a rack. The scope three emission of DRAM in general is very, very high.

Pascal: That, that seems counterintuitive to me, or at least I would've never thought that this is why the carbon emissions are so concentrated. Do you have an explanation for why memory is so outsized? Uh, has such an outsized role in the carbon emissions.

Lisa: So it comes from the manufacturing process and in particular the way the wafers are being manufactured, it's very, very, very complex process that actually emits a lot of different chemicals. And this is where we're trying to act working with our manufacturing partners to reduce the emission produced by their manufacturing process.

There are also techniques to reduce the carbon footprint, uh, chemical actually, um, chemical processes, uh, called abatement that will, um, that will make sure that the, um, the emissions that are being created by this manufacturing process can be eliminated or reduced. Um, and, and so basically the. The gas coming out of this process is being cleaner.

Pascal: Got it. Okay. So I've seen that there are two slightly competing approaches for reducing emissions. One is DFE - Designed for Environment, and then there's DFS - Designed for Sustainability. Can you talk a bit about how these two differ and which one we've settled on?

Lisa: Yeah, sure. So DFE - Designed for Environment and Design for Sustainability are very related. The approach we take at Meta is Designed for Sustainability is broader because it includes DFE. So DFE is focusing on the environmental impact of material selection and energy efficiency. And for our Design for Sustainability process, we try to embed all of the possible things we can do to reduce our carbon footprint, including circularity that DJ mentioned earlier.

So making sure that through the all hardware lifecycle, supply chain manufacturing, we can actually act, and in-house what we can do is reusing our hardware, extending the useful life of our servers and using greener materials. So this is all the area we're trying to cover.

Pascal: I think the reuse aspect is really interesting because that is, again, it sounds like common sense. Why did we not consider this particular aspect before?

Lisa: Yeah, that's a very good question. So the why is because it does have a lot of complexity to actually reuse a component. It's not just, you know, taking a memory DRAM out of a rack and putting it back into a new rack. We need to fully erase what the data was in there, making sure that the DRAM can work and has a good reliability performance and making sure that when we select the DRAM, it's compatible with the new rack that we're actually designing. So I'm talking about the DRAM here because that's, uh, that's a use case that we are actually currently working on and, and is, uh, plan of record in our fleet. Um, so I would say the complexity of reusing component is what stopped us.

Uh, but the benefit really, really outgrows all the risks that we have identified, especially for this specific component. So it's now something that we do plan of record, uh, on all of our new rack. And this is part of the design for sustainability process we're working on For every new rack we're considering, what is the list of component we can reuse, uh, and do we have enough supply for, uh, this specific rack?

DJ: And Pascal, I would like to add, uh, one other, uh, dimension to this. Uh, Lisa mentioned how, uh, ga uh, protection is very important, uh, and that requires complex uh, handling. There's also the other element where. You have to reach a certain scale for us reusability to make sense. Uh, you want to address the biggest, uh, problem or biggest bottleneck, so you reach certain scale for DRAM use, uh, with the growth of our infrastructure, at which point it becomes meaningful to put all of these other complex processes in place to address, uh, reusability. So there are these two elements. You know, if it's something very small, then it's not contributing in an impactful manner, then you want to go look somewhere else.

But when something becomes big enough and it becomes impactful for your carbon measurements, that's when you decide. And this is always, uh, shifting over time based on how the infrastructure is scaling. So tomorrow we could be looking at reasonability in some other areas when that piece becomes bigger piece, even though it may be complicated, but it's big enough now to address it and it's worth undertaking that complexity.

Lisa: I completely agree. And one last point, uh, that adds to the complexity is the technology breaks, because sometimes we have a lot of component available, but for the new design, there is a new technology emerging, and then we have a lot of questions about does it make sense to shift to this new version of that specific component, or can we push a little bit the transition of this new generation and reuse some component?

And there is a lot. And this is why it's always a very, very cautious process that we take to review all the different aspects of design, so sustainability, of course, but when we consider reusing component, we also make sure we are reaching good performance for our new racks.

Pascal: Yeah, that is fascinating. I'm sure there are tons of considerations in there because new hardware often comes with improved efficiency, so I am glad we are having a whole team that looks at all the coefficients in the equation that goes into this, and there would be so much more that we could discuss.

I know there is green steel being used using cool electric arc furnaces now in our data centers to remove emissions from that process as well. But unfortunately, we are out of time, so at this point, I can only thank you for making our data centers as efficient as possible and for joining me here on the Meta Tech Podcast.

DJ: Thank you, Pascal, for having us as well and big shout out to the entire Infra teams which are involved in this. Lisa and myself, we represent the voice of thousands upon thousands of engineers and, uh, you know, minds which are working within Meta and perhaps even across Meta, so thank you, everyone.

Pascal: Absolutely. Thank you, everyone.