The Two-Century Quest to Quantify Our Senses

One day in the year 1840, a man opened his eyes and couldn’t see. This was it, the “final blow,” as he later wrote in his diary. It was as if the man, a renowned German medical doctor turned professor of physics, had inexplicably gone blind overnight. But his condition was not new. It was the dramatic culmination of months of unexplainable symptoms that had befallen this scientist: bursts of light in the eyes, headaches, nausea, lack of appetite, insomnia, and neurosis. Little did the scientist know, however, that his dire situation would eventually result in something remarkable — a startling revelation that would forever change our understanding of the human senses and how they would come to interact with machines.

No one quite knew why Gustav Fechner fell ill. Burnout, perhaps, brought on by too much work, like partially writing and editing a 7,000-page, eight-volume encyclopedia? Or turning himself into a human guinea pig in the name of science, damaging his eyesight? Fechner had stared too long into the sun using glasses with only colored filters as he explored the perceptual phenomena of afterimages — the images that stay on the retina long after one stops gazing at a light source. This series of experiments seemed to throw him into a searing, never-ending “light chaos” that he would constantly experience, even with closed eyes. He even painted his bedroom black to stop light from leaking in.

“Close to insanity,” Fechner nevertheless began to slowly recover from his malady. Instead of gradually adjusting his eyes to faint light, he took the brute force route: sudden and intensive short-term exposure to the brightness of the everyday, quickly closing his eyes before the light caused intense pain. He resumed eating, consuming such odd delicacies as raw ham soaked in wine and lemon juice, as well as sour berries and drinks. Although he still experienced “disagreeable sensations” in his head, he finally spoke again.

One October afternoon, Fechner wandered into his garden as he occasionally had done during his illness. This time, however, he took a gigantic step to reintegrate into the visual world. He removed the thick bandages covering his eyes. The light spilled in. As he glanced into his garden, the scientist experienced a miraculous sight. He saw the flowers “glowing.” They seemed to speak to him. In this ecstatic moment, Fechner came to an astonishing realization — plants must also have souls.

Fast-forward 180 years. In the digital haze of pandemic newsfeeds, you are clicking through pages on LinkedIn. Dozens of jobs in new professions with strange sounding titles appear: vision engineer, applied perception scientist, visual experience researcher, color scientist, and neural interface engineer, the job description of which is to “help us unleash human potential by eliminating the bottlenecks between intent and action.”

One career in particular catches your eye: an applied perception scientist, working for Oculus, a once-small start-up that manufactured a lightweight VR headset, which Facebook bought in 2014 for $2 billion dollars. The job announcement asks for expertise in visual perception, the “computational modeling of vision,” and “experimental and/or modeling approaches” that “help us inform AR/VR display requirements and architectures.” This new career in applied perception science also has another thing in common with the other LinkedIn jobs — it asks for knowledge in an obscure sounding discipline called psychophysics.

What does a scientist undergoing a mysterious illness in mid-19th-century Germany have in common with 21st-century engineers seeking to plumb the depths of human perception? In the early morning hours of October 22, 1850, just seven years after his malady subsided and the encounter with the flowers in his garden, Gustav Fechner, physicist, philosopher, and believer in the ever-lasting consciousness of souls, plants, and the earth itself, had another burst of inspiration. He came to the realization that there must be a relationship between spiritual and physical energy, a measurable correspondence between the world external to our sense perception and the internal world of our brain processes.

But Fechner needed to prove his theory scientifically. He thus invented the almost mystical-sounding discipline that he christened psychophysics — a “theory of the relations between body and mind” that aimed to establish a measurable connection between two spheres that had long remained separate: the material, physical universe and the mental, psychological one. In Fechner’s formulation, psychophysics would be an “exact science, like physics” and “rest on experience and the mathematical connection of those empirical facts that demand a measure of what is experienced.” He asserted, in other words, that we can measure and calculate how we sense the world using mathematics, forever changing how we view sensing and perception in relationship to man-made machines.

Psychophysics set the European scientific world on fire. It helped advance the newly emerging discipline of experimental psychology, in which there was already a mad rush to translate human thoughts into numbers. The rising hybrid scientists of the period — psychologists, philosophers, mathematicians, and physicists — were eager to escape a nonscientific (e.g., unmeasurable) understanding of how the senses and the mind worked, and Fechner supplied them the ammunition. These scientists began to develop theories to demonstrate mathematical connections between physical phenomena, what are called stimuli, and the sensory experience of such phenomena, labeled sensation or perception. But in the process, they also sought to eliminate the experiencing, subjective self doing the sensing, replacing human sensory experience with “objective” formulas and equations.

Fechner’s ideas would also quickly be materialized in the newly appearing sensing machines of his time — instruments with strange sounding names like kymographion, tachistoscope, or chronoscope, which measured or graphically represented things like blood pressure, the speed of vision, or response time. In the words of the 19th-century French physiologist Étienne-Jules Marey, a major inventor of such devices, these new instruments sought to reveal the hidden “language of nature.” Such human sensory measuring devices were to be found in a novel kind of experimental scientific environment: the emerging experimental psychology laboratories in Europe and the United States, whose goal was to create a new kind of human being: quantifiable, calculable, and predictable.

We might assume that psychophysics died a dusty death, relegated to the history books of psychology and the crumbling sets of abandoned scientific instruments that fill up university collections. But as our LinkedIn search reveals, psychophysics is very much alive in the most unimagined of places. In the labyrinths of behavioral research at Facebook Reality Labs, for example, scientists with PhDs in neuroscience, applied perception research, robotics, and computer science still draw (albeit with updates) on the quantitative modeling of sensation, stimuli, and perception that Fechner discovered in the late 19th century in their 21st century aims to create VR, AR, and XR experiences that are both exceedingly real and, at the same time, completely artificial.

As one group of cognitive and computer science researchers claims, “VR can be seen as a continuation of a long psychophysical tradition that attempts to interfere with our perception in order to clarify its underlying mechanisms.” The game testing cubicles of Electronic Arts and the perception laboratories of various universities are united around a similar aim. They use the sensing machines of our time — networks of sensors, statistical modeling, machine intelligence, and computing infrastructures, human labor and the earth’s resources — to capture, calculate, model, and simulate human sense perception beyond the wildest dreams of 19th-century scientists and, in the process, create a wholly new relationship between these sensing machines and us.

Unlike the sensors inside the Oculus Quest that can instantiate whether a perceivable change happens in the frame rate of an image, Gustav Fechner in 1860 had little access to sensing devices to experimentally prove his theories. His “sensors” were cruder: the perceptual abilities of human beings who, under psychophysical tests, would generate verbal data about what they experienced, which then could be calculated to come up with measurements.

In other words, as mathematically rigorous as they were, Fechner’s psychophysical methods still relied on human scientists, physiologists and psychologists who would “subjectively” report what they had perceived from test subjects during an experiment. The experimenter could not control whether the subject’s report would be correct or even accurate.

To complement Fechner’s psychophysics, 19th-century scientists therefore turned to newly emerging technologies to better measure sensorial responses: new sensors to prove their new theories. These researchers invented instruments designed to capture and measure the human (and also animal) senses. From the ophthalmoscope to the acoustic whistle, the olfactometer, chronoscope, aesthesiometer, and photographic gun that enabled the emerging practice of chronophotography, these instruments became, in effect, de facto senses.

Not only were they the earliest versions of the sensing machines on offer today, but these sensing instruments also played a fundamental role in the construction of a vast new domain of knowledge about the human sensorium called sensory physiology, which understood the senses as key to the development of psychological and physiological knowledge.

Utilizing scientific observation, experimental procedures, and emerging instruments, sensory physiologists studied a broad range of phenomena, including spatial perception in hearing and seeing and the speed of neuronal firings or sensory quanta: tiny measurements in the form of thresholds and differences of stimuli intensities, mainly derived from Fechner’s psychophysics.

Sensory physiology took the body and the senses directly into the technological loop; there was not merely a chance or accidental relationship with the technologies of measurement and analysis that would soon proliferate in the first research laboratories dedicated to experimenting upon and analyzing the senses of living bodies. Not only were the senses reconceived as technologies in and of themselves, but also, like our VR and AR headsets, instruments became increasingly integrated into animal and human senses. In other words, the senses became sensors, and sensors assumed the role of sensing.

This “extension” of the senses into instruments seemed par for the course in the early to mid-19th century. Already devices such as the stethoscope and the thermometer were replacing the human senses. The overall effect of these new experimental technologies and the laboratories where they were deployed was that machines not only increasingly regulated the bodies and senses of the subjects being studied but also shaped the senses of researchers themselves. In other words, researchers became data analyzers.

Indeed, for those scientists working in the shared space between physiology, psychology, and medicine, instruments became essential partners in revealing the invisible forces fluxing through bodies, forces that were inaccessible to the human senses. There is no clearer expression of this sentiment than the words of 19th-century French physiologist Étienne-Jules Marey, who stated, “How little our senses tell us, so that we are constantly obliged to use apparatuses in order to analyze things.”

Those scientists sought to turn the messy, imprecise senses into something externally readable through an early form of data visualization — what Marey and the mathematician, philosopher, and sensory physiologist Hermann von Helmholtz called the graphic method. For Marey, Helmholtz, and other scientists of the time, the possibilities of visualizing hidden forces opened up a new chapter into the analysis of the human body’s inner life.

The age of quantifying living bodies took off. But it increasingly demanded ever-stranger sensing machines to advance its scientific cause: primitive electrodes; pneumatic tubes and mechanical harnesses that could be attached to the limbs, arms, wings, feet, and legs of unfortunate humans and animals, such as Marey’s air pantographe, which was used to study live birds in flight; recording devices like pneumographs, which graphically represented throat movements produced during vocalization; or tachistoscopic apparatuses that measured how visual sensory impressions could affect consciousness within specified time intervals.

Despite Marey’s belief in such instruments rendering visible the hidden language of nature, however, these instruments were eventually criticized as imprecise. In fact, not only was the graphic method imprecise, it was actually considered deceitful. Although there was a fervent belief in the all-powerful ability of these “self-registering” instruments — devices that automatically inscribed or self-recorded their data without human intervention — there was still a need for a human interpreter between the body and the instrument. The human eye had to read and understand the scales of data appearing on the drums of the kymographion or other machines. Wrote one French researcher, “The registering apparatus does nothing but to inscribe undulating lines that fall on our senses; but once it comes to interpreting the traces, the graphic method has no more certitude than direct observation.”

Things now have changed. While the curves of Marey look suspiciously like those Fitbit app or Apple Health app curves on your smartphone, there are major differences. Direct observation replaced by automated algorithms and rows of network servers that store the statistical analysis of the world’s sense data have become our new sensory physiologists. Now sensing machines capture, read, and analyze signals produced by human bodies — blood pressure, glucose, breathing, nerves — with even less recourse to human observers. Precise electronics, digital signal processing, statistical models, and the automation of computation are erroneously believed to have eliminated the imprecision of both the senses and mechanical instruments. If the data is imprecise, the algorithm can always be tweaked. The same goes for such technologies when they are revealed to actually exhibit cultural, gender, or racial bias. The technological solution is to identify the problem and quickly fix it, rather than recognizing the fundamental flaw in design assumptions in the first place.

The experimental researchers of the end of the 19th century who sat with pen, paper, and instrument, ready to measure reaction to stimuli, have instead moved away from the scientific lab where they originally sought new knowledge about the human senses through sensors and into the reality labs of Facebook or Apple’s secret “exercise lab.” This secret facility in a bland Cupertino, California, building employs not only 13 exercise physiologists and 29 nurses and medics but also an army of machines to log tens of thousands of hours of subjects’ physiological data as benchmarks to test the sensor-embedded products of the world’s most valuable corporation. Apple is proud of its instruments. According to the lab’s director, it has “collected more data on activity and exercise than any other human performance study in history.”

Comparing the emergence of sensing machines in the 19th century with today therefore reveals both a historical continuity and a radical break. In the 2020s, every individual who dons a fitness tracker, smart watch, biometric shirt, or wearable sensor engages in a process of transformation: turning oneself into a self-monitoring test subject without the intervention of the human psychologist or physiologist.

The myriad of sensing-measuring gadgets we now take for granted were still laboratory-bound in the 19th century. Instruments that logged physiological signals didn’t leave the sites of experimental science for the gym or the office as they do now; they were instead parts of larger scientific apparatuses.

Moreover, there is a fundamentally different understanding of the human in relationship to our technologies of digitization. In fact, even if human bodies then were at the whim of instruments that abstracted their senses into graphically plotted signals with 19th-century tools, there was still a connection between the person that produced the data and the resulting numbers. One could glance at the squiggly marks on the soot-covered surface of a drum after an experiment and claim, “that is me.”

But the computational automation of mathematics and statistics has changed this. The way we understand the temporal role of sensing now is radically different. With the kymographion and sphygmograph (blood pressure) instruments, time was recorded graphically at different scales on the physical surface of a rotating drum or on a paper surface, by manually speeding up or slowing down the mechanical instrument. The visualized curves generated by Fitbits and Apple Watches are different. They are the by-products of statistical processes: the size of a window through which you see only part of a longer and continuous signal, or derived from statistical techniques. In other words, the curves that are output represent an already computationally processed artificial time.

Like the big data world they are part of, in which meaning is dependent on the right mathematics to find patterns and meaning in a sea of randomness, our new psychophysicians and physiologists also believe that the truth of the senses can be found in the numbers — in statistical techniques that measure and predict the future based on the past.

Perhaps most importantly, the context and purpose of sensory measurement itself has radically transformed since Fechner announced his psychophysics. The physiologists and psychophysicists of the past, who turned to instruments to technologize themselves and their test subjects, now go an extra step. They now automate the almost-two-century-old scientific technique called psychophysics to design the next generation of perception machines. With today’s sensing machines in our clothing, cars, houses, games, stores, theaters, and galleries, measurement thus goes hand in hand with design and creation. The need to probe the human senses with instruments and machines is not only about gaining knowledge about how these senses work; it is applying that knowledge to designing and perfecting systems that produce and anticipate new connections between our perception and those instruments and machines, where both expand each other.

In contrast to that which came before, our new sensing machines more accurately capture and analyze the microtime and microspace of our breath, heartbeat, brainwaves, muscle tension, or reaction times. But they do this for another reason. Our sensing machines now conceive and create techniques that aim to fulfill that long sought-after dream of those forgotten 19th-century researchers like Fechner and Marey: to become one with what Fechner called the animated substance of the technological world itself.

Chris Salter is an artist, Professor of Immersive Arts and Director of the Immersive Arts Space at the Zurich University of the Arts (ZHdK) and Professor Emeritus, Design and Computation Arts, Concordia University. He is the author of “Entangled: Technology and the Transformation of Performance,” “Alien Agency: Experimental Encounters with Art in the Making,” and “Sensing Machines: How Sensors Shape Our Everyday Life,” from which this article is adapted.