Why is it so important?

First considerations

I'm passionate about virtual worlds - especially in deeply immersive video games - web development and 3D audio. You might have guessed it, this tool is precisely at the crossroad of these, and it is addressing something that, I think, was missing there. Especially with the rapid growth of metaverses, and the increasing concern for accessibility.

Improbable has rightly shown it, through The Otherside successful stress tests, and powered by their M² technology. The next generation of virtual worlds - metaverse(s) or video games - is on the Web.

There are increasingly more tools to build 3D environments on the Web, including Three.js. Some are easy to load and discover, some require more extensive graphic resources. With the support of cloud gaming, this has become significantly more accessible.

Meanwhile, immersive audio continues to grow, but, it is tough to tell the difference between real breakthroughs and marketing campaigns.

Immersive audio solutions for headphones (Binaural) have been investigated for decades (at least). The last few years have witnessed the appearance of 3D soundbars (with sound reflecting off the walls), or dual-speakers that reproduce 3D sound in an impressive way, mostly based on crosstalk cancellation.

While this is becoming more and more accessible, headphone listening is still the most popular and the most affordable.

So, what is the point of immersive audio?

It is colossal, even though it needs to be summarized here.

These concerns have not been neglected in video games. 3D sound can convey information (e.g. Fortnite, sound being an indicator of the location of chests, or enemies), atmosphere, and enhance the consistency & credibility of the space.

Consider this : a Zoom meeting, which is exhausting, and not particularly immersive. Bring it into a virtual space, 2D or 3D, expanded. The participants can move around, switch locations and atmospheres, and have different conversations with different people. Voices are spatialized, separated, more comprehensible. Plenty of people, regular people, have Zoom/Google Meet calls, and are quite familiar with these concerns.

I believe the metaverse should be at the crossroads between a video game and a virtual meeting/call, and it's crucial that immersive audio be part of the development process for such spaces. This would enhance the envelopment of these environments, improve intelligibility in vocal exchanges, extract informative value from sound spatialization. And a lot more.

This is precisely what I have been investigating for the last few years. If you are interested, and if you speak French, here is the link to my paper. It is exploring the applications of immersive audio in virtual worlds, especially in the metaverse. Sorry, it is not available in English yet. For the sake of this research, I built a basic 3D virtual space with Three.js, to integrate immersive audio. I'm still a novice developer, so the code might look messy or inaccurate at some points, but you can check it if you're interested.

Here is a link to the live demo. Please let me know what you think about this (optimization, features, sound spatialization...). And go wild! I've been working a lot on this project, I'd be more than happy to get ANY kind of feedback. :)

I hope you'll be more confident about the value of immersive audio in the metaverse, or at least in virtual worlds. But, what if you want to integrate immersive audio into a virtual environment.

How to integrate immersive audio on the Web?

If you're using a video game engine, such as Unreal Engine or Unity, go for it. There are some 3D sound APIs, such as Steam Audio, that can help you achieve more advanced immersive audio integration than the basic tools. It also provides an API written in C.

If you'd rather use JavaScript - probably for the Web - follow me.

You've most likely already heard of Omnitone, or Resonance-Audio. They are excellent libraries, worth a try. The latter can be integrated with video game engines, and the source code is in C++.

With these, you can definitely integrate 3D, or 360°, audio into a scene. But maybe you'll run into the same problems I had : sound crackling/stuttering when the camera is moving, because of the position of the sound sources struggling to update. The only library I haven't had this problem with is Atmoky. And it is also the only one currently in active development. That's why I want to praise them, and testify about its efficiency and ease of use.

First, let's try to be a little more specific.

What will you need to use?

By immersive audio, we mean 3D (or at least 360°) audio. It is not always mandatory, as you can have an immersive experience without 3D audio, but it is a key feature. And we want to go past the usual left/right spatialization.

Libraries such as Atmoky allow the use of Binaural sound, which is a way for the user to experience a 3D sound sphere around them, with only headphones. This lets you reduce the audio stream to only 2 channels, though it typically requires several channels/speakers.

This effect is mainly achieved through filters that reproduce the way sound objects are perceived, simulating the shape of the head, ears, hair... (e.g. A sound coming from the back will be, roughly, filtered in the high frequencies, because of the resistance of the skull/ears). But there is more!

This set of filters, called HRTF (Head-Related Transfer Functions), can be customized to precisely match your head, ears, etc. This way, with the proper (eventually custom-made) HRTF, you can perfectly and accurately perceive sound in 3D with headphones.

This being discussed, let's get back to the point.

The Atmoky case

With such a library, all this tedious process is integrated, so you can straight away place the sources and the listener on a 2D or 3D grid. You should also be able to customize HRTF with your parameters.

There is a demo provided by the company, which will be more explanatory than any words.

The integration process is pretty straightforward. A lot of the development process is abstracted, so you can focus on what matters. The documentation is clear and comprehensive, and you can quickly get started, as well as use the examples that are provided.

After trying out the demo, I provided some feedback to the team. Their response was swift, and the documentation was updated based on that feedback, which is quite nice to see.

It is very well suited to libraries like Three.js, or any A-frame framework. You just need to set the position & rotation relative to the camera/character (see code sample).

renderer.listener.setPosition(42, 10, 3.1415);
renderer.listener.setRotation(0.2, 0, 0);

If possible, I would recommend using streaming audio, rather than preloading (see code sample).

const createSource = (url, obj) => {
  const audioElem = document.createElement('audio');
  audioElem.src = url;
  audioElem.crossOrigin = 'anonymous';
  audioElem.preload = 'none';
  audioElem.load();
  audioElem.loop = true;

  const audioElemSrc = audioContext.createMediaElementSource(audioElem);

  let source = renderer.createSource();
  source.setInput(audioElemSrc);
  source.setPosition(obj.position.x, obj.position.y, obj.position.z);

  audioElem.play();

  return source;
};

There is not much to add, the best would be to experiment and discover by yourself. You can request a free license, that is restricted to non-commercial and testing purpose.

The documentation is detailed enough, and the team caring enough, to get by!

Special thanks to Clemens who demonstrated the demo, provided access and corresponded with me afterwards.

Thank you for reading me so far. Honestly, I really appreciate it.

Please, if you have any comment, advice, question about this, or if you believe I made a mistake, feel free to contact me. If you want to discuss immersive audio, Web development, the metaverse... Do the same! I'll be happy to chat. You can find me on Twitter (@0xpolarzero), or just email me if you like it better (0xpolarzero@gmail.com).

If you found this post instructive, feel free to share it! You can also find it as a Twitter thread. :)

The word "metaverse" comes with so many misconceptions, that we no longer know what it means. Major brands use it for marketing purposes, creators automatically associate a NFT collection to a potential future metaverse.

Where does this term come from?

The word metaverse appeared for the first time in the novel Snow Crash, by Neal Stephenson in 1992, to pick up a concept that had already existed in sci-fi for a few decades. Video games appeared quickly, doing pioneering work in the creation of a metaverse, as early as 1985, with Habitat, which represents one of the first large-scale multiplayer video games. Other publishers have contributed to the evolution of this technology, but the most important is probably Second Life. This virtual world, available for free since 2003, allows users to embody an avatar and design the world around them (infrastructure, clothes, objects, animations, sounds...). They can also acquire plots in the game and use the virtual currency made available by the company that created it (Linden Labs). In the general idea, a metaverse allows users to work, interact, exchange, travel and access educational content in a three-dimensional space, possibly via a virtual or augmented reality interface. This space requires the use of cryptocurrencies and blockchains, to testify to the authenticity and ownership of the objects within it, and to protect exchanges between users. By definition, there should be one global metaverse and not several. In reality and with the decentralized direction we are taking, there will necessarily be several metaverses. The most important thing is that these metaverses should be interoperable and interconnected.

Oh yes, decentralization! Welcome Web3!

Given the link that has formed between Web3 and the metaverse (or metaverses ?), we can define the latter as a decentralized platform built on a blockchain, which incorporates virtual objects and properties, and in which users embody a virtual identity, usually through an avatar ¹. A metaverse is often defined as a three-dimensional space. However, some have proven their ability to go beyond this definition, which should no longer be so restrictive. It will therefore be necessary to admit the possible qualification of a 2D space as a metaverse ², even if the vast majority of these environments are in 3D.

The categorization of a space as a metaverse is frequently made publicly and implies in most cases a commercial aspect that cannot be neglected. Sometimes, it can come out of a shift from a more classical category, such as video games, during an event (e.g. the concert series in Fortnite two years ago) or permanently. This distinction must force us to redefine the notion of metaverse, especially since a large part of the spaces classified as such are not directly built on a blockchain. Rather, they interact with this protocol, which is radically different since these platforms are not entirely decentralized. This first definition, which is very restrictive, certainly ensures that an environment is a metaverse. However, it does not correspond to reality, in the minds of users and in the applications that we find.

So, what is a metaverse, then?

In this case, it is necessary to borrow elements of definition from the concept of virtual world, which is very close to the definition of metaverse. In fact, it can be established that a metaverse is a virtual world. This last term is defined by the following:

“Virtual worlds are persistent online computer-generated environments where multiple users in remote physical locations can interact in real time for the purposes of work or play. Virtual worlds constitute a subset of virtual reality applications, a more general term that refers to computer-generated simulations of three-dimensional objects or environments with seemingly real, direct, or physical user interaction.” ³

Does it sound familiar?

This definition fits quite well to spaces that claim to be metaverses (The Sandbox, Decentraland, Roblox in some way, Cryptovoxels...). In addition, the characteristics established at the beginning (interaction with the blockchain, virtual identity and avatar) can ensure a safer classification. Recent applications of this concept systematically integrate NFTs and cryptocurrencies as a determining part of the environment. In reality, it is more cautious to identify a virtual space as a metaverse according to these criteria, but the classification is increasingly tolerant.

Don't they look more like video games?

They do! And I would say they are in some ways.

Indeed, the distinction between video game and metaverse is always narrower, and some video games can be considered as metaverses, and vice versa. A video game is defined as "an electronic game with a user interface that involves interaction to generate visual feedback" allowing a user (or several) "to have an impact on the game and to perceive the consequences of their actions on a virtual environment" ⁴. This definition presents characteristics that can be found in most metaverses. Conversely, some video games, especially MMORPGs, have the same characteristics as the virtual worlds we defined earlier. They require a considerable involvement from the players, encourage interaction within communities, facilitate the incorporation between the player and their avatar or virtual identity, have a vast and persistent environment that evolves according to the players' actions... Actually, these video games can be considered as metaverses. Most of them would be, for many users, their first experience of a social network on a global scale: in some spaces, players discuss and exchange constantly, but also within guilds, channels dedicated to trade, to battle... Such an extension of reality allows Internet users to meet again virtually, in a persistent universe, with particular rules, mixing various communities while bypassing geographic boundaries.

But, given that link with video games, shouldn't this especially affect young people?

Correct! Such spaces result in deeply invested users, embodied by avatars and a virtual identity, which is increasingly complex and consistent. Indeed, the young people of Gen Z (late 1990s - 2000s) spend increasingly more time in virtual worlds and connect even more to their virtual identity. Around half of this population feels a greater freedom in video games and considers this virtual identity as a real expression of their personality ⁵. Several factors can explain this particular link, notably a great receptivity to microtransaction systems: avatar and cosmetics purchases on video games, mobile games, recurring streaming subscriptions... This link is reinforced and permanently renewed, as major brands that are established "in" the metaverse ⁶ (e.g. Nike or Adidas), are particularly popular to this generation.

I get it! The metaverse actually already exists then.

Not exactly, I would say. A distinction must be made, since these spaces, which represent a new field of communication for brands, may indeed suggest that the metaverse already exists. It would match the way these new generations interact, think, act, and spend their money, in video games, and other virtual events. As a matter of fact, if we consider the metaverse as a world, or a set of interoperable and persistent virtual worlds, which should give rise to new interactions, experiences, and economic flows, these current spaces would rather be various micro-metaverses ⁷. These do indeed take the form of private virtual worlds, as with Fortnite, but gaming is not the only branch of the metaverse, although it is currently the most attractive. Above all, it represents an excellent way to study the evolution of the metaverse, since it is obviously the most evolved part of it.

So, the metaverse is not a video game, it doesn't exist yet either... What is it, then?

Well, I believe it will be necessary to accept popular conceptions of the metaverse in its definition. Inevitably, its configuration is intimately linked to the way the public perceives it. The confusion between virtual world, video game and metaverse is less and less inaccurate, and some extremely popular groups, probably among the closest to delivering a global, accessible, persistent, but also approved metaverse, are rightly taking the MMORPG model for their conception ⁸. The Sandbox, most likely one of the most popular and successful metaverses so far, is produced by a subsidiary of Animoca Brands, whose core business is originally video games. This is the same company that invested in the metaverse mentioned in the previous note, Otherside, alongside the giant Andreessen Horowitz, whose huge influence can only prompt creators to exploit video game standards to produce a successful virtual world.

Therefore, it's not surprising that metaverses look a lot like video games. I'm confident that it will continue to be the case, and that the line between the two will totally fade away. The exception would be that metaverses will most likely have much more influence on reality, that this fact will be acknowledged, and that they will be intimately linked to every new technologies of "Web3".

Great! I hope I will be able to join the metaverse(s)!

I hope so. If this environment, tightly related to NFTs, is for now mainly intended for a wealthy public, it will certainly take less exclusive and more accessible forms in a near future. Moreover, the connection between user and virtual world has been widely reinforced during the pandemic between 2020 and 2022; the success of virtual events (concerts or festivals) in such spaces made it obvious. Some plots of land in main metaverses host live conference broadcasts, and other daily events that are gradually going virtual, partially or entirely. Also, it should be mentioned that a metaverse does not necessarily have as much of a commercial purpose as the most popular ones have. Even though, this aspect is almost systematically the most prominent, mainly because of the chase for "utility". A virtual environment, which has the characteristics of a metaverse, but "simply" offers a sensitive, sensory, and/or social experience, could be accepted in the definition. We can imagine, for example, an exhibition place integrating an immersive visual and sound system, without its access being so restrictive!

In any case, I am optimistic about this evolution. I think the definition is not as crucial as to build this environment together and to stay aware of what is real and what is around us, especially what/who we care about.

And above all, not to repeat the same mistakes with Web3 that we have made in the last decades.

Thank you so much for reading! I hope to have clarified this question, even a little.

Please tell me if this is the case, or if you have any suggestion, question, or if I made any mistake. You can always get in touch with me, I'd be glad to talk about this!

My Twitter

References

1. Hao, C., Gu, E., & Jiang, Y. (2018). Metaverse, White Paper, Gravity version (v3.0). ↩

2. The WorldWide Webb for instance. ↩

3. Dionisio, J. D. N., Burns III, W. G., & Gilbert, R. (2013). 3D Virtual Worlds and the Metaverse: Current Status and Future Possibilities, ACM Computing Surveys 45, 3, Article 34. ↩

4. Wikipédia, "Video Game". ↩

5. Adams, P. (2022, April 19). What Gen Z wants to see from brands as metaverse attachments grow, Marketing Dive. ↩

6. Nike has acquired a plot in the Roblox game, called "Nikeland," a kind of virtual store-and-show, which allows users to try on products virtually, and play games with the community. This space attracted about 7 million users between November 2021 and March 2022. Sutcliffe, C. (2022, March 22). Nearly 7 million people have visited Nike's metaverse store. The Drum. Since the end of 2021, Adidas has been building relationships with leading brands in the NFT scene (e.g., by purchasing a BAYC, whose value exceeded 100 ETH at the end of January 2022, which at that time corresponded to approximately $260,000). The brand is gradually strengthening its presence in the metaverse with virtual collections, avatars, and offers to "enter the metaverse." Adidas. Section "metaverse". These brands are far from being the only ones, most of them manage to link part of their market to the metaverse, or even thoroughly change their communication on social networks to integrate into it (Puma, Dolce & Gabana, Prada, Gucci, Louis Vuitton...; Hello Kitty, which has a space in the Roblox editor, and which counts 47.6 million visits in a little less than a month, between April 4 and May 9, 2022.) Heng, E. (2022, Feb. 17). A rundown of the fashion brands that have joined the metaverse. Vogue. ↩

7. Ffiske, T. (2022, April 20). The Immersive Wire, VR/AR and metaverse analysis every Wednesday and Sunday, Newsletter.

   ”The common conception of the metaverse is long into the future, where interoperable worlds can connect together and will open new doors when it comes to economics, social interactions, and experiences. At most, we have various micro-metaverses where people play in private worlds (such as Fortnite); but it's a stretch to say we are already in the metaverse.” ; “While gaming will certainly be part of the metaverse, it further confused the data to the gaming side, when it will be so much bigger.” ↩

8. Among all the companies that claim to build a metaverse, Yuga Labs, which has some of the most popular NFT collections (BAYC, CryptoPunks, Meebits), launched a collection of terrains usable in their future metaverse (Otherside) in April 2022. They raised on this collection almost $600 million in 24 hours, in addition to the support of a large Venture Capitalist, Andreessen Horowitz, which led a campaign that brought them $450 million in March 2022. The creators explain, about their platform, that it will be :

   "[...] a gamified, interoperable metaverse [...] The game blends mechanics from massively multiplayer online role playing games (MMORPGs) and web3-enabled virtual worlds. Think of it as a metaRPG where the players own the world, your NFTs can become playable characters, and thousands can play together in real time.” ↩

   ---