Successful navigation and interactions with objects in a complex, three-dimensional (3D) world are essential to survival, yet studies of the neural basis of visual perception and action often treat the world as a flat, two-dimensional (2D) canvas. Here, we review recent discoveries about how the visual systems of different mammals have adapted under the constraint of a 3D world. Many neurons in primate visual cortex are sensitive to 3D depth cues constructed from input differences to the two eyes (binocular disparities). Similar neuronal selectivity has recently been characterized in tree shrews and mice, though constrained anatomically and functionally. Disparity-sensitive receptive field structures can now be assessed noninvasively across human visual cortex and functionally linked to depth perception. At later stages in the nonhuman primate (NHP) ventral (“what”) pathway, supporting object recognition, complex 3D objects are represented as spatial configurations of geometric parts. In studies of the dorsal (“where”/“how”) pathway of NHPs, underlying visuospatial processing, the inclusion of 3D structure in visual tasks has established tighter links between neuronal mechanisms and perception. This has also revealed neuronal representations of 3D object position and orientation that are tolerant to 3D gaze location, providing a basis for robust perception and action. The cross-species analysis presented here provides new perspectives on neural coding mechanisms underpinning the richness of 3D visual experience. Elucidation of these mechanisms will be essential for understanding the neural basis of cognition and behavior, which in turn will enable the development of therapies and neuroprosthetics for those suffering from visual impairments.