It all started back in 2018 when Facebook came out with the initial introduction of 3D photos - a feature that takes the help of in-depth data to create images that can be seen from different angles, all within the implications of virtual reality headsets. Previously desktop users of Facebook needed to have a depth map file to enjoy the feature or smartphones like Samsung Galaxy S10 or iPhone 11 but now with recent changes in place, anyone having an iPhone 7 or mid-range Android device can get access to 3D photos.
The credit for this expansion goes to the state of the art machine learning techniques of Facebook as the newly deployed AI models create the 3D images without the need for depth data and irrespective of the age of images and origins. Hence users can experience decades-old family photos in a completely new way by converting them to 3D and the best part is that the feature also works with selfies, paintings, and other sceneries.
As soon as you post a 3D photo, any friend of yours or user, if the profile is public, can see the image and an icing on the cake can be VR through the Oculus browser on Oculus Go or Firefox on the Oculus Rift. You can also share the 3D photos via Facebook stories but just like any other story, it will disappear after 24 hours. However, when 3D photos are shared on Facebook’s newsfeed, you can see who’s viewed, reacted to, and responded to them for a relatively long time.
There are certain restrictions in place as well as 3D photos can’t be edited nor you can add multiple photos to a post while sharing 3D photos. If you plan onto post a 3D photo on a page, remember you won’t be able to boost it.
The 3D photo team chose a convolutional neural network and then trained it with millions of pairs of 3D images and the associated depth map. After that, they took the help of blocks inspired by FBNet to set up the model for more mobile devices.
Furthermore, for optimal architecture configuration, the 3D photos team had set up an automated process based on an algorithm called ChamNet. Developed by Facebook AI Research, ChamNet regularly takes sample points from a search space in order to train the accuracy predictor - the one that accelerates the search for a model to maximize accuracy and satisfy the limitation of resources. Moreover, Facebook told that the search for the model to underpin new 3D photos took almost three days when the company used 800 Nvidia Tesla V100 graphics cards.
In order to lessen down the number of bytes that were required to be transferred to the mid-range devices, the 3D photos team quantized the weights and activities to 8 bit. This helped to limit the drop in quality as compared to the original larger models.
Facebook is also planning to apply these techniques in-depth for videos as well taken with mobile devices as well. But this won’t definitely be easy as in videos each frame depth must be consistent with the next one. Nonetheless, these new experiences can help Facebook better understand 3D scenes which eventually can also assist robots to navigate and interact with the real world around them.
Read next: Facebook’s Android App Might Be Getting Features From Google Photos
The credit for this expansion goes to the state of the art machine learning techniques of Facebook as the newly deployed AI models create the 3D images without the need for depth data and irrespective of the age of images and origins. Hence users can experience decades-old family photos in a completely new way by converting them to 3D and the best part is that the feature also works with selfies, paintings, and other sceneries.
As soon as you post a 3D photo, any friend of yours or user, if the profile is public, can see the image and an icing on the cake can be VR through the Oculus browser on Oculus Go or Firefox on the Oculus Rift. You can also share the 3D photos via Facebook stories but just like any other story, it will disappear after 24 hours. However, when 3D photos are shared on Facebook’s newsfeed, you can see who’s viewed, reacted to, and responded to them for a relatively long time.
There are certain restrictions in place as well as 3D photos can’t be edited nor you can add multiple photos to a post while sharing 3D photos. If you plan onto post a 3D photo on a page, remember you won’t be able to boost it.
How Data Science Made 3D Photos Possible
Facebook had to overcome a lot of challenges to make 3D photos possible for a wide range of users. Their struggles majorly included training the model about correctly guessing how images might look from different angles and to make them run smoothly on different mobile processors within seconds.The 3D photo team chose a convolutional neural network and then trained it with millions of pairs of 3D images and the associated depth map. After that, they took the help of blocks inspired by FBNet to set up the model for more mobile devices.
Furthermore, for optimal architecture configuration, the 3D photos team had set up an automated process based on an algorithm called ChamNet. Developed by Facebook AI Research, ChamNet regularly takes sample points from a search space in order to train the accuracy predictor - the one that accelerates the search for a model to maximize accuracy and satisfy the limitation of resources. Moreover, Facebook told that the search for the model to underpin new 3D photos took almost three days when the company used 800 Nvidia Tesla V100 graphics cards.
In order to lessen down the number of bytes that were required to be transferred to the mid-range devices, the 3D photos team quantized the weights and activities to 8 bit. This helped to limit the drop in quality as compared to the original larger models.
Facebook is also planning to apply these techniques in-depth for videos as well taken with mobile devices as well. But this won’t definitely be easy as in videos each frame depth must be consistent with the next one. Nonetheless, these new experiences can help Facebook better understand 3D scenes which eventually can also assist robots to navigate and interact with the real world around them.
Read next: Facebook’s Android App Might Be Getting Features From Google Photos
