Imagine controlling your Mixed Reality headset, AR glasses or computer with just a flick of the wrist or possibly just by intending to make a movement or while your hand is comfortably by your side. These aren't just pie-in-the-sky ideas, they are part of actual prototypes that not just Metamates have had their hands or in this case wrists on.

In this episode we'll explore how non-invasive interfaces differ from previous entrants in this space, how AI is helping to generalize these so they don't require personalized training and the implications for accessibility.

To help us navigate this fascinating landscape I'm joined by Jesse, Lauren and Sean. We'll discuss the challenges and breakthroughs in developing these technologies and what they mean for the future of interaction. But now without further ado here is my conversation with Jesse, Lauren and Sean.

Today we're talking about generic neuromotor interfaces. I know that's quite a mouthful and sounds like some purely theoretical research but it's actually something that exists in a real prototype today as part of the Orion AR glasses. These were demoed at last year's Meta Connect and were even called by MKBHD the coolest input device he's tried in a long time.

To help me understand exactly what a neuromotor interface is, what the opportunities are and why the generic part is so challenging, I have three brilliant guests with me today. Lauren, Jesse and Sean, welcome to the Meta Tech podcast.

Jesse: Hey, Pascal, great to be here.

Pascal: Fabulous. Before we jump into the topic, let's introduce each of you to our audience. Jesse, how about we start with you? How long have you been at Meta? What did you do before? What is your background?

Jesse: My name is Jesse Marshall. I'm a research scientist here at Meta. I've been here for about three and a half years really working at the intersection of electromyography and computer vision.

Before coming to Meta, I was a researcher in academia where I studied the links between our brain and how our body moves.

Pascal: Fascinating. Okay, Lauren, can I pass it on to you?

Lauren: Hey Pascal, I'm Lauren.

I have been at Meta for a little over four years now. I joined the EMG research team. Before that, I was working at a company called SRI International, which is like a research for hire company, I'd say.

Doing a lot of diverse types of research in brain machine interface, but also for me, radar and scanning microscopy development. I have a background as a research scientist in neuroscience and biomedical engineering, so all over the map, but here I'm super, super happy to be working on neuromotor interfaces.

Pascal: Giving me huge imposter syndrome here that I'm even qualified to interview you and asking you questions, but okay.

Moving swiftly on, last but not least, Sean, what about you?

Sean: Hey, yeah, my name's Sean Bittner. I'm a research scientist here at Meta for the last four years working at the intersection of EMG and text entry, so prototyping different EMG methods for text entry. Before that, I finished up my PhD at Columbia studying theoretical neuroscience.

Pascal: Super exciting. I definitely want to get to the handwriting part at some point later, but we usually also ask you to talk a bit about your team. Can you tell me what the mission statement of your team is?

Lauren: Yeah, I can speak to that.

The mission statement for our entire EMG engineering and research team, I'll read it. Extend human output and revolutionize interaction for the next platforms by creating the first usable, useful human computer neuromotor interface. So really, it's about creating the future, as we say, and designing a neuromotor interface for public consumption.

Pascal: Yeah, I think now we need to actually talk about the elephant in the room and talk about what a neuromotor interface actually is.

Jesse: Yeah, I can speak to that. So, you know, some listeners might be familiar with the idea of a brain machine or brain computer interface.

So this is something that interfaces directly with the electrical activity your brain makes by implanting electrodes or with electrodes on the surface of your head and uses this to control a computer or other device. So you can simply imagine, say, moving a cursor on a computer and watch it move without making any physical movement. A neuromotor interface is something very similar, but instead of reading out brain activity, you're reading out activity from muscles.

And so much like the brain, you know, our muscles emit electrical activity when they contract. These can be very small, minute impulses. These can be very large activations.

And they also are controlled by the brain, almost directly controlled by the brain. And so in a sense, by reading out the electrical activity of the muscles, you're also reading out the neural signals that are passing to these muscles.

Pascal: Okay, that makes a lot of sense. And now we're actually here because excitingly, you have a new paper published in Nature that is called, and now I need to read this off, a generic non-invasive neuromotor interface for human-computer interaction. And that, again, sounds pretty abstract, I think, but is really cool applied science. So can you give us a high level description of what's in the paper?

Jesse: Yeah, so this is work really at the intersection of brain-computer-brain-machine interfaces and human-computer interactions, where the goal was to create one of these neuromotor interfaces that would work across people.

So the idea is you could put this computing device on any person and have them be able to use it to control a computer, to enter text, to control a cursor, or to, say, play video games. And, you know, there's historically been a huge problem in the field of these computer interfaces, which is generalization, that you can use either electrodes in the brain, or you could use, say, electrodes put on your arm to control a computer. You could, you know, you could get a lot of training data from an individual person, say, have them imagine 1,000 cursor movements, or have them perform 1,000 pinches of their index finger.

And you could build a model that would understand those electrical signals from the body and how they predict the cursor or the gesture movement, but they wouldn't generalize to new people. So you would take this interface, you would try to put it on another person, have them control the computer, have them, you know, input that little gesture, and they simply wouldn't work. And so the idea behind the paper is, you know, we develop a strategy that allows that generic control, something that works across anyone, and we do it also in a way that is non-invasive, so you don't have to implant electrodes into your brain or into any other part of your body.

Pascal: I guess we'll get to what this enables in terms of accessibility later, but if we're talking, like, mass-market consumer devices, the idea that you get some sort of new mouse, basically, and you would first have to train it for a few days before you can successfully make any movements with it on your machine, I think that's kind of unimaginable, right? So the generic part here is so crucial for a success of this in something that reaches a wide audience. Can you also talk a bit about the differences between this kind of non-invasive neuromotor interface and other input modalities? You just talked about the brain-computer interfaces, for instance.

Jesse: Yeah, yeah, that's a great question.

And so our neuromotor interface relies on a technique called surface electromyography, so surface, meaning it's recorded on the surface of your body, and then electromyography, meaning it's reading out the electrical signals from your muscles, so electromyography. And, you know, as you alluded to, there's a huge range of devices you can use to input information to a computer, right? So there's mouses, there's keyboards, there's hand-tracking systems you can use with cameras, all the way to these brain-computer interfaces where you're, you know, even implanting electrodes. And I think the distinguishing feature here is so, you know, the neuromotor interface that we describe in this paper is a wristband.

You just put it on your wrist, and you're done. And so compared to something like a keyboard or a mouse, you know, you don't have to assume the computing position, right? You don't have to go and physically engage with another device. You can use any sort of natural hand gesture that you need to control this input.

And so you can use this in mobile settings, right? If you're controlling a pair of smart glasses as you're walking around, you couldn't bring a mouse along with you. Cameras might not work to see your hand when it's by your side.

Pascal: Yeah, or you're on the way to the train station, and you have a suitcase in one hand, and maybe a map or something in the other.

Okay, map. A little anachronistic, but, you know, there are plenty of things you might be carrying around. You also don't have the space to even get your phone out or, you know, pinch the fingers and the exact right gesture to interact with some hand-tracking things.

So this is a real game changer in how you can interact with something that is kind of constantly on you in this way. So, you know, basically described the whole problem space with creating a generic interface and how important it is. But what exactly was the solution? How did you get to something that actually works for ordinary people that you find without hours or days of calibration for that specific person?

Jesse: Yes, that's a great question. And, you know, the answer was actually related to modern developments in artificial intelligence and large language models. So, you know, some people might be familiar with the idea of a scaling law for an AI model. This idea that people have just sort of discovered that, like, there is a predictable increase in the performance of modern foundation models, large language models, as you increase the amount of data you put into the model, as you increase the amount of parameters in the model.

And so what we found is that same type of scaling law applies to these neuromotor interfaces, that if you take a model to, say, predict a gesture from a person wearing this interface, if you train it on 10 people, or if you train it on 100 people, or if you train it on 1,000 people's data, you will see a very predictable power law curve emerge that predicts much performance improvements you get. And so, you know, the key discovery was this AI scaling law that you can use to guide the design of these interfaces. But, of course, a critical part of that, which we can talk about more, is, well, then how do you, you know, build a system that can actually collect data from 100 or 1,000 people, which is a huge amount of hardware and software engineering, and then, of course, the AI modeling that you ultimately need to process all that.

Pascal: Right. When you designed the actual wristband, so you may actually look at this today, there are tons of YouTube videos of how this was introduced in the beginning, and I know a lot of the focus was on the actual glasses, but I honestly found the little wristband just as exciting. It is really small and unobtrusive.

I'm sure this also brought up certain design constraints for you, because bigger is often better in terms of the signals you can capture. So, what were the trade-offs you had to consider by making something small and still powerful enough to allow you to capture all the data that you need?

Jesse: Yeah, that's a great question. And, you know, I should say this is a novel type of computing interface, right? There have been efforts to do it before.

No one has really done it successfully, and no one really knew what it would take to build, or if it was even possible, to measure EMG from this, you know, a device that has such a compelling form factor, right? So, people do electromyography in clinical settings with wires planted in the skin. Other research will use, you know, really high precision devices that have like wet gel that you put all over your forearm. And you can look at the evolution of the bands made by the team, first as a startup called Control Labs, then acquired, you know, when they started, it was sweatbands with wires poking out, right, to where it is with the Orion device today, where you have this, you know, very slim factor.

And, you know, it was iteration. And, you know, it's hard for me to speak to the exact brilliance of our, you know, hardware and industrial engineers, but like continuous iteration cycles to reduce the electrode size and to really find that fit. And then, you know, the careful electronics engineering that allows you to record these signals with really, really impressive fidelity, allowing you to record, you know, really the single, like the quanta of muscle activity, the motor unit action potential.

Pascal: I love the alternate future you're describing there, where everybody needs to regularly refresh the wet gel on their wrists in order to interact with their headsets. I'm glad we're not going down that route.

Jesse: Something out of a Neil Stevenson book, yeah.

Lauren: We, we also considered, you know, tattoos.

Can I jump in? I think one other major design consideration for the wristband was comfort.

This is something that we want people to wear all day long, and it has to be comfortable enough to wear. It has to look nice. It has to be easy to come on and off, and the battery life has to be long.

So I think these are all super important for making a consumer product.

Pascal: Yeah, I was surprised how consumer-friendly or, I don't know, market-ready it already looked, even though it was at this point, I think, a prototype. So I guess things can only get better from here.

How exciting. But can you talk about how well these models work in the kind of post-training stage when you use them online?

Lauren: Yeah, I'll speak to that. Ultimately, how it feels when somebody actually uses the device is what really matters.

You know, when we train a model, it's an ML system, we have a ton of metrics that we can look at. Things like that most people who work in AI are familiar with, false negatives, false positives, F1 score. We always try to optimize that, but in the end, it matters how it works online.

So we have a different range of ways that we assess it, and a lot of our assessment is done in a lab facility that Jesse mentioned that we use for data collection. So we've invested a lot in developing protocols or ways of getting people to do a task that mimics the product use case. What we're looking for, of course, is something that's generic, so it works for a diverse set of the population.

So this is one of the really key aspects here, is making sure that we're measuring on a diverse set, a diverse group of people, and making sure that people can perform tasks. So like just Sean will talk about the handwriting task, or the input tasks that we've talked about to just navigate through menus, things like that. Part of the generic use case, I think, is evaluating learning.

So this is a consumer, we hope it's a consumer device, that will work just out of the box. So we don't want somebody spending three days learning how to do EMG gestures. So some of the things that we assess is how fast somebody can pick up the band and just start using it immediately.

This is like critical in our assessments. And then I think the final piece of that is linking those in-lab online assessments to our product partners, because in the end, people aren't using our in-lab tasks. They're actually making phone calls, or using the product in the way it's intended to be used.

So we want to make sure that the signal that we're getting in the lab is matched to the signal that we're getting from product partners.

Pascal: What is the feedback cycle for you? Like when you see that something did not work as intended, and I guess people will be very creative in how they use it. So how do you adjust your models to work with the day-to-day feedback you receive?

Lauren: I think that this is one of the questions that makes the job so interesting, is because we don't always get it right the first time in designing the online task or designing the in-lab task that we're trying to assess.

The development cycle is one of the most interesting things that we're doing right now. So we are trying to take feedback from multiple sources, like we have surveys, we have ways to file bug reports. And what we do is we adapt our metrics.

We try to take that feedback and develop a metric so that we could get a much larger signal and to see if we notice any trends and to see if we can adjust those trends. So we'll adapt our evaluation, also our data collection and our modeling strategy. So it's basically a full cycle in terms of responding to feedback.

Pascal: In the beginning, you mentioned that LLMs are actually useful for the work that you do. To me, as a non-AI researcher, that sounds mildly confusing because what you're describing, especially the detecting of muscle activity, sounds very far removed from language. So how does that work? How is the approach that we see in generative AI and LLMs influencing the work you do in EMG?

Sean: Right. So a lot of the most recent advances in GenAI, LLMs, speech recognition technology can be brought to bear on our challenging research for neural interfaces for text entry. So to describe those briefly, we've published some research in the past on keystroke recognition, so two-handed touch typing, where people are wearing two of these EMG wristbands, or handwriting recognition more recently, where users will wear a wristband on one arm and write characters in their own handwriting style, and these machine learning models will detect these characters. And so our early research benefited immensely from close collaboration with world class experts in AI speech here at Meta, as well as from FAIR to take the techniques that we use for architectures, losses, data augmentation, data preparation, etc., to really push the performance of these models that recognize human behavior and produce text.

Pascal: How should I think about handwriting recognition in general with my wristband on? How does it work? How do you imagine people interact with something like, let's say, AR glasses again and handwriting?

Sean: Right, so the vision is that you're using WhatsApp or Messenger and you want to write a message to your friend, and so you can see the text box and the characters that you're entering into the system, and maybe you're writing on your opposite hand or on a surface in front of you or on the surface of your leg, and you're just writing in your natural handwriting style. The system is recognizing the characters that you're writing, and we're using language modeling technology, perhaps from LLMs, to infer the text that you wish to send to your friend and complete those messages and then send them.

Pascal: Yeah, that's amazing, because there will still always be environments where speech to text is not the ideal way to communicate.

If I'm sitting on the tube here and I want to send a somewhat private message to my partner, I wouldn't want to shout it into my glasses or my phone for that matter.

Jesse: Can I also just add, it is still crazy to me that this works. You know, I think when you, when I first came here, you know, I looked at the, like, raw signals that come out of these wristbands, you know, it looks like total nonsense.

And if you looked at the academic literature, right, at that point in time, like, the idea that you could do a, you know, 26-character recognition task using these signals, especially on this, like, very constrained hardware, was like, just like, it didn't make any sense to me. It seemed like just like a fanciful idea. So I think it's really incredible, you know, what that team is producing.

I mean, you could see this in the paper as far as, you know, really driving these high precision models for a really, really challenging problem.

Pascal: I might be your end boss in this matter, because my handwriting is absolutely awful. And even if I write on an iPad with a pencil, it struggles to figure out what I'm writing there.

So if you want to send me a prototype at some point, I'm happy to help.

Sean: We've certainly gained an appreciation for the immense behavioral variability that occurs in handwriting in natural populations. And we'll take that on as a key challenge.

Pascal: Okay. Can you talk a bit more about how you actually work with the product teams here? Because you're all researchers, but the way you've been describing it, this gets into the hands of real people already. And we've talked about Orion, how it's made its way into at least broadly available prototypes across the company and some selected people on the outside.

So how does that work on the day-to-day for you? How do you work with people who then turn this into real hardware?

Lauren: Yeah. So I mentioned a little bit about our evaluations in the lab, but our evaluations in the product scenario is one of the key ways that we interface with the product teams. So we're always looking for feedback from product developers testing the devices who are trying to figure out how to put the interactions that we develop as the EMG team into a product so that it actually is usable and fun to use.

We also interact with them by sometimes suggesting new interactions. So maybe a product developer, like Jesse said, would never have imagined that handwriting is possible. So one of the things that we do is try to push the boundaries of what's possible with EMG to think of new and creative ways that one might incorporate it into a product.

Yeah. And finally, actually, maybe Jesse could speak a little bit more, but thinking about accessibility and other kind of applications of the EMG and products that could be used in different populations.

Pascal: Oh, yeah. I definitely want to hear more about this because so far we've talked about just kind of the sci-fi vision for us. And it's really exciting, even for me as an able-bodied person, to think about using this at some point. But I feel like the implications for somebody with special accessibility needs are completely different.

So what's the story there?

Jesse: Yeah, definitely. I think the goal for us, for Meta broadly, is to build devices that can be used by everyone. And this is also an area, I think, of personal interest for many on the team who trained in the field of brain-computer interfaces, which is really directly geared at enabling movement or input in the cases when it's lost.

It's an area of personal interest for me. I have close family members who really rely on accessibility devices to interface with computers. And we really believe that these neuromotor interfaces could be a really huge win and a very powerful tool for accessibility.

And they can do this by restoring the ability to perform input when you can't move at all, when you move too much, and by allowing people to kind of personalize their style of movement. And I can talk a little bit more about each of those. So we have a collaboration with CMU, where we've actually used these devices on people with spinal cord injury, so on limbs where they can't move the limb at all.

But they retain some residual muscle activity. And you can use these devices to still control a computer, to play games, to probably make phone calls. We've also shown in other work that in people with tremor, where even in these excessive movement, that you can still detect these gestures.

The power of the generic model is that it can generalize to these cases of excessive movement, really speaking to the power of modern AI to generalize. And then in the last one, this is a case where we think that these models, by detecting intent and being able to detect intent in a really arbitrary generic way, coupled with state-of-the-art machine learning, that we can take individuals who have a limited range of motion, who have limited strength, and might not be able to move a mouse, might not be able to type on a keyboard, might not be able to even perform the normal gestures that we're asking to perform, and then to be able to personalize an input device specifically to them with the remaining degrees of freedom they have left.

Pascal: Yeah, for sure. And you talked about some of the alternatives that are out there that are either really invasive, take a long time to customize to their needs, and we have seen how much of a difference it makes for something to be very easily available. I just recently saw a report, I think it was on The Verge, about people using the Ray-Ban Meta glasses as an accessibility feature. And one of the reasons is that they are comparatively cheap to the options that exist in the medical space that perform similar tasks, and that itself can often be a complete game changer, in addition to all the benefits that you've highlighted there.

Jesse: Yeah, you know, 50 percent of content on Netflix is watched with closed captioning, and so I think you see there's a huge need for accessibility and customization, and I think a huge opportunity for AI, you know, generally to be able to provide some of those inputs.

Pascal: For sure. And I would also like to talk a bit about how you feel like having an organization like Meta behind you is helping you achieve your goals.

You've already talked about aspects like working with product teams, of course, but can you expand a little on other aspects that have helped you work here?

Jesse: Yeah, I think we can all speak to that. I mean, I could start, you know, maybe with reference to this paper, you know, so this paper we just published in Nature has 200 authors on it. You know, they come from a huge range of expertise in, you know, from software engineering, hardware development, neuroscience, software engineering, machine learning, and AI.

It takes a very special place and a special moment in time to bring all those people under one roof, so to speak, and I think it's also to make the bet required, right? This is a risky technology. This isn't like, say, you know, building a phone. It's like, you know, something you're not sure it's going to work, and so, you know, to have a company that can both put those people in the same room and then also believe in them enough to, like, see it through, you know, all the inevitable hurdles that occur.

Sean: You benefit so much from the research proximity to world-class experts in language modeling and speech, whether it's methodological algorithmic lessons that we learn from these collaborators or techniques for efficient training on large-scale GPU clusters. That's the core benefit that I've experienced while being here.

Pascal: Yeah. And Lauren, wanna add your two cents?

Lauren: Yeah, I'd add that the models that we're developing are working in concert with the hardware production team, the teams that are developing firmware, and I mean, I've talked a lot already about the product, but all of this has to come together, and every time we make a release or we make an update to the product, and I think that this really speaks to the excellence around production at Meta and how smooth that has been.

I think I've learned certainly a lot from folks at this company.

Pascal: Even that little part, updates, it's so easy to get this wrong. Having some people who have expertise and just updating hardware devices over the air and ensuring you don't brick them as you do this, incredibly helpful.

One thing I also wanted to quickly highlight is that as part of your paper release, you also published a bunch of open-source repositories. Can you tell us a bit about what's in there?

Sean: Yeah, we're really excited about the recent release of some of these EMG datasets on the order of 100 users for some of our core interactions like two-handed typing, handwriting, etc. It's been exciting to see a dataset that we just published at NeurIPS six months ago.

People were already writing papers and submitting conference papers, making advances on that technology already. It's an exciting way to solicit more involvement and collaboration from a broader field of academic contributors.

Pascal: Yeah, really exciting to see this continuing, this open spirit as we're approaching these fields.

All right, and as we are slowly getting close to time, I just wanted to throw a broader question your way, and that's what's next for you and your team now? What are you focused on right now?

Jesse: Yeah, I can start and other people can jump in, but I'll say it's still very early for EMG. As you mentioned, we released a prototype with Orion. There's no consumer device that we've yet released.

As I alluded to earlier, we have this little museum of old EMG wristbands, starting from the sweatbands and wires to the prototype we released there. There's still plenty of room at the bottom, so to speak, to continue to miniaturize and refine. We're still in the early innings of AI and AI interfacing with biology and the human body.

I think we're going to continue to be able to build new interactions, new ways of leveraging EMG in our devices. As Sean mentioned, we've already seen in six months people building on top of some of the open source research that we've published. I think we're going to continue to see unexpected and new gains coming from all of these different directions.

Pascal: Exciting. Lauren, do you have something to add as well?

Lauren: I'm personally excited to see where the products go or where the consumer products go that are built on this research.

I think that the, I mentioned the mission at the very top of this podcast, is to revolutionize human input. So we're trying to really change things for how you input information into devices. I think that this is just the beginning.

Pascal: All right. And I'll leave it with you, Sean, to wrap us up.

Sean: I agree with Lauren. I'm just fascinated to see how people enjoy using this technology, what they love, what they don't love, and how we can create better experiences for people.

Pascal: Well, I can't wait to get my hands on one myself at some point. But for now, I can only thank you all for contributing your research, making so much of it open access and paving the way for a gel-free input device future for us all.

And of course, for joining me here on the Meta Tech Podcast.

Jesse: Thanks, Pascal.

Lauren: Thank you.

Pascal: And that was my interview with Jesse, Lauren, and Sean. If you have thoughts or even terrible handwriting samples to share, do give us a shout on Threads or on Instagram @MetaTechPod. And if you're keen on joining us on this journey, pop over to metacareers.com for opportunities.

And that's it for another episode of the Meta Tech podcast. Until next time, toodle-loo!