Most meetings happen over video now, but almost nobody knows whether the people on the call are actually paying attention. You can see faces on your screen, but that doesn’t tell you who’s following along and who checked out ten minutes ago. Facial attention analysis is built to answer that question, using computer vision to read the cues people give off on camera without realizing it.
What It’s Actually Looking At
The system works off the camera feed you already have running; nothing extra to install or set up. It tracks a few things at once:
- Where someone’s eyes are pointed, and how often their gaze drifts off-screen
- Whether their head is turned toward the conversation or angled toward something else
- General posture and how much they’re moving, speaking, or reacting
None of these on their own tell you much. Someone can look away for a second and still be completely engaged. It’s the pattern over the length of a call that matters, not any single moment.
How Webcam Eye Tracking and Head-Pose Estimation Separate Fixations from Drift
In human-computer interaction, video-based eye tracking extracts eye position from camera frames to measure either the point of gaze or the motion of an eye relative to the head. As foundational visual-attention research—originating with Louis Émile Javal’s 1879 observations and expanded through the non-intrusive film studies of Guy Thomas Buswell and Alfred L. Yarbus—established, human vision does not sweep smoothly across a display. Instead, visual attention alternates between short stops known as fixations, where visual information is actively processed, and rapid ballistic movements called saccades that reposition the gaze between targets.
During a live video conference, a participant reading a dense slide deck, inspecting a shared architecture diagram, or checking speaker notes exhibits frequent saccades and brief off-axis glances. If an algorithm evaluated gaze direction without accounting for head orientation, a participant using a dual-monitor setup would appear disengaged whenever they looked at the secondary screen where the presentation is displayed. By jointly estimating head rotation and eye position relative to the head, the computer vision pipeline distinguishes transient visual scanning and dual-screen viewing from sustained off-screen disengagement.
Reading Facial Expression and What Comes Out of It
Expression is the other big piece. The system picks up on small movements around the eyes, brows, and mouth to tell whether someone’s focused, confused, bored, or actively following along. A furrowed brow during a complicated explanation usually means someone’s concentrating hard, not losing interest. A face that stays flat and unchanged for a few minutes straight usually does mean they’ve checked out.
People blink, glance away, and shift around constantly during a normal conversation, so a single frame can be misleading on its own. Tracking eye movement, expression, and posture together over time is what actually shows a real pattern.
The result is an attentiveness read for each person on the call, built up across the whole meeting rather than judged from one moment. It can show which part of a presentation lost the room, or which attendees tend to drift in longer sessions no matter what’s being discussed.
How the Facial Action Coding System (FACS) Quantifies Nonverbal Meeting Cues
To translate subtle facial movements into structured data without relying on subjective guesswork, modern computer vision pipelines draw on the Facial Action Coding System (FACS). Originally developed by Swedish anatomist Carl-Herman Hjortsjö in 1969 and formalized by Paul Ekman and Wallace V. Friesen in 1978 (with a major revision by Ekman, Friesen, and Joseph C. Hager in 2002), FACS deconstructs facial expressions into anatomical Action Units (AUs) corresponding to contractions or relaxations of specific facial muscles, alongside Action Descriptors (ADs) for broader head and gross behavioral movements.
Each detected Action Unit is scored for intensity on a five-point letter scale—A (trace), B (slight), C (marked or pronounced), D (severe or extreme), and E (maximum)—along with laterality modifiers such as R (right side), L (left side), U (unilateral), and A (asymmetric). In affective computing—an interdisciplinary field pioneered by Rosalind Picard’s 1995 paper and 1997 MIT Press book Affective Computing—machine learning models combine these anatomical codes across multiple modalities so that a single expression is interpreted in context. As documented in affective computing research, a furrowed brow (AU 4, driven by the corrugator supercilii and depressor supercilii) paired with tightened eyelids (AU 7) and fixed on-screen gaze (AU 69) typically indicates concentrated cognitive effort rather than negative sentiment.
| FACS Code | FACS Name & Anatomical Basis | Meeting Attention & Temporal Signal Interpretation |
|---|---|---|
| AU 0 | Neutral face (baseline muscular state) | Normal resting baseline; prolonged, unvarying AU 0 combined with drifted gaze signals passive tuning out. |
| AU 1 / AU 2 | Inner & outer brow raiser (frontalis, pars medialis & lateralis) | Tracks sudden visual orientation, interest, or surprise (when combined with upper lid raiser AU 5B and jaw drop AU 26). |
| AU 4 + AU 7 | Brow lowerer (corrugator supercilii) & lid tightener (orbicularis oculi pars palpebralis) | Distinguishes active cognitive absorption or analytical confusion during complex technical explanations from boredom. |
| AU 6 + AU 12 | Cheek raiser (orbicularis oculi pars orbitalis) + lip corner puller (zygomaticus major) | Identifies a sincere, involuntary Duchenne reaction versus a polite, voluntary Pan-Am smile (AU 12 contraction alone). |
| AU 41 / AU 43 / AU 45 | Lid droop, eyes closed & blink (levator palpebrae superioris relaxation) | Separates normal conversational blinking (AU 45) from sustained eyelid droop (AU 41) associated with meeting fatigue. |
| AU 51–58 / AD 84–85 | Head turn (51–52), up/down (53–54), tilt (55–56), forward/back (57–58), shake (84) & nod (85) | Captures head orientation toward the active conversation, forward leaning (AU 57), and affirmative backchannel nodding (AD 85). |
| AU 61–64 / AU 69 | Eyes turn left/right/up/down (61–64) & eyes positioned toward other person (69 / M69) | Measures whether eye position remains anchored on the conversational partner (AU 69) or repeatedly drifts off-screen. |
| AU 70–74 | Visibility codes (brows/forehead, eyes, lower face, or entire face not visible; 74 unscorable) | Flags camera occlusion, poor lighting, or hand-on-chin obstruction so unscorable frames are excluded rather than penalized as low attention. |
Crucially, including the FACS visibility codes (AU 70–74) prevents a common failure mode in naive webcam analytics: when an attendee rests their chin on their hand (occluding the lower face, AU 72), wears reflective glasses (AU 71), or steps out of frame (AU 73), the system marks the interval as unscorable (AU 74) rather than falsely logging a drop in engagement.
Where This Actually Gets Used
A few examples of where it’s useful in practice:
- Training and onboarding – seeing which sections people actually absorb versus tune out of, so the material can be reworked
- Sales calls – catching the moment a prospect’s interest starts to fade, which is easy to miss when you’re focused on your pitch
- Remote classrooms – giving instructors a way to gauge how a group is doing when there are too many faces to watch individually
Translating Session-Level Attention Patterns into Operational Decisions
Across all three environments, the operational value comes from correlating aggregate attention trajectories with session timestamps rather than policing individual participants:
- Curriculum and onboarding optimization: When a cohort’s aggregate gaze fixation (AU 69) and affirmative head-nod rate (AD 85) drop consistently during a specific 15-minute compliance or architecture module, instructional designers have objective evidence that the segment needs interactive exercises or clearer visual pacing.
- Sales discovery and demo pacing: Product demonstrations frequently lose prospects when presenters spend too long on slide narratives before showing live workflows. Real-time shifts from concentrated brow lowering (AU 4) and forward head lean (AU 57) to sustained off-screen gaze (AU 61/62) and neutral expression (AU 0) prompt the host to pause, ask a direct diagnostic question, or transition to the relevant use case.
- Large-cohort virtual instruction: When an instructor is sharing a screen to 30 or more remote learners, standard video grids only display a handful of thumbnails. Aggregating group-level expression and posture shifts alerts the instructor when a complex derivation has caused widespread confusion (AU 4 + AU 7) versus passive fatigue (AU 41).
Using Computer Vision the Right Way
The solution works best as something that helps a person make a judgment call, not something that replaces one. Looking at attention scores across a group or a whole session tells you something real. Reading too much into one person’s score in isolation doesn’t, since people have off days, distractions, and reasons for looking away that have nothing to do with engagement. It also only works fairly if people know it’s running and how their video is being handled, since this is more personal than most data companies collect.
Privacy, Transparency, and Human-in-the-Loop Governance
Because facial telemetry processes sensitive behavioral cues, enterprise deployments should align with established trustworthiness standards such as the NIST Artificial Intelligence Risk Management Framework (AI RMF 1.0). The NIST AI RMF organizes responsible AI deployment around four core functions—Govern, Map, Measure, and Manage—which translate directly into practical guardrails for meeting analytics:
- Informed consent and clear disclosure (Govern & Map): Attendees must be explicitly informed when attention analysis is active, what telemetry is extracted, and how session summaries are used.
- Ephemeral feature extraction over raw video retention (Manage): Rather than storing raw video recordings of participants’ faces on central servers, privacy-preserving pipelines convert frames into numerical Action Unit and head-pose vectors in memory and discard the underlying video frames immediately.
- Cohort-level aggregation and human oversight (Measure): Reporting session-level and segment-level engagement trends rather than ranking individual employees prevents neurodiversity bias, camera-angle bias, and automated disciplinary misuse, keeping human facilitators in control of the interpretation.
None of it requires changing how meetings are run. The camera’s already there, and the facial attention analysis just puts it to use.
Interested in building something like this for your own team? Get in touch with Agiliway to talk through what facial attention analysis could look like for your use case.


















