Introduction

Sightcorp is a specialist in AI-driven real-time face analysis. Using any 2D camera, Sightcorp's DeepSight technology can collect both aggregated audience data and individual metrics for everything from age and gender to views, impressions, and dwell time.

With the DeepSight Toolkit Interface Asset, Intuiface experiences can access this information and pair it with contextual data about the weather, the environment, and other potential influencers of attention and preference. The combination of these data sets through the use of Intuiface Analytics delivers a rich understanding of content effectiveness for both traditional, non-interactive signage and interactive signage. In addition, the demographic data can be used to trigger the display of personalized content in real-time, ensuring the presentation of relevant content at the display-level.

Acquiring and configuring the DeepSight Toolkit

Acquiring the DeepSight Toolkit

The DeepSight Toolkit can be evaluated - and used with Intuiface - for seven days at no cost.

• To request the trial, head to https://sightcorp.com/get-free-trial/

Toolkit configuration

The DeepSight Toolkit is a software-only solution that can run on any Windows or Ubuntu Linux device with a 2D USB camera. Intuiface Player (or Composer) can run on the same Windows PC as the toolkit or on any other device with a Player-supported operating system.

Toolkit configuration involves three steps:

• On the "Licensing" tab: Enter a license key
  
• On the "Output" tab:
  • Identify the local HTTP port of the Windows or Ubuntu device
  • In the Raw Data section
    • Ensure the "Enable pull" option is selected and specify the HTTP port
      
  • In the Aggregated Data section
    • Ensure the "Enable pull" option is selected and specify the HTTP port
    • Specify the length of time - the aggregation period - over which data should be collected for the "Historical Data, Aggregated" data set.
      You can choose between 30sec, 1min, 5 min, and 10min
      

Installation and configuration of the interface asset

Interface Asset installation

Download and install the DeepSight Toolkit Interface Asset on the PC you'll be using to run Composer: 

Make sure Composer is not running, unzip the file you downloaded, and copy all of the resulting files (.ifd, .js, .png) into the folder [Drive]:\Users\[UserName]\Documents\Intuiface\Interface Assets. You can optionally place those files into a folder of any name to separate them from files for other interface assets.

Now, when you launch Composer, the DeepSight Toolkit IA will appear in the list of available interface assets.

Interface Asset configuration

Add the DeepSight Toolkit IA into your Intuiface experience then open its X-Ray panel.

On the Properties tab

• Specify the server host and port for the device on which DeepSight is running.
  Remember, DeepSight could be on the same device as Player or on a different device. Both scenarios are supported.
  

  Determining the server and port names:

  • DeepSight is on the same device as Player
    Leave the default values of 'localhost' for the server host and '27080' for the server port
  • DeepSight is on a different device from Player
    Enter the IP address of the device running DeepSight. Only change the port number from '27080' if you changed it in the DeepSight Toolkit itself.
• Specify the face mask threshold
  Consider this property to represent the level of confidence above which DeepSight will consider a detected person to be wearing a mask. Enter a value between 0 and 100.
  It is recommended that you don't use a value below 70.
• Specify whether to collect session data per detected person
  Every time session data is collected by DeepSight, it is aggregated across all detected people. By checking this box, you are requesting that session data should also be collected for each detected person. Each detected person is assigned a unique ID. (Note, because people are anonymously detected by DeepSight, if a person is detected then leaves the field of view then returns and is detected again, DeepSight considers that to be two different people.) 

Data collected by DeepSight - for Real-Time Triggering

The DeepSight Toolkit reports two classes of data for triggering Intuiface actions - like displaying particular content - in real-time.

See the section about retrieval details to learn how to access this data for real-time triggering.

Real-time data

A summary of the people (or person) in front of the camera for the current video frame analyzed by DeepSight Toolkit.

Use this data to trigger the display of content created for a specific combination of audience metrics like age and gender.

• People count
  Total number of people detected, regardless of whether or not they looked at the content. In the analytics industry, this value is known as the "opportunity to see", or OTS.
• Male count
  Total number of people detected during the session who were male
• Male percentage
  Percentage of "People count" that is male.
• Female count
  Total number of people detected during the session who were female
• Female percentage
  Percentage of "People count" that is female.
• Average age
  Average age of people detected
• Average smile
  Average smile value across all detected people. Smile, on a scale of 0 to 100, is an indicator of interest.
• Masked people count
  Total number of people on whom a mask is detected.
• Viewer count
  Total number of people who are viewing the content
• Male viewer count
  Total number of men who are viewing the content
• Male viewer percentage
  Percentage of "People count" who is male.
• Female viewer count
  Total number of women who are viewing the content
• Female viewer percentage
  Percentage of "People count" who is female.
• Average viewer age
  Average age of people who are viewing content
• Average viewer smile
  Average smile value for all people who are viewing content. Smile, on a scale of 0 to 100, is an indicator of interest.
• Masked viewer count
  Total number of masked people who are viewing content.

Session data, aggregated

A complete set of data collected for all people detected during a session. A session is defined as the period between a call to the "Start collecting session data" and "Stop collecting session data" actions associated with the DeepSight Toolkit IA. 

Use this data to trigger content based on the audience detected during the previous session. Assumes there is some consistency about the audience during the previous session and the audience in front of the screen 

• People count
  Total number of people detected, regardless of whether or not they looked at the content. In the analytics industry, this value is known as the "opportunity to see", or OTS.
• Male count
  Total number of people detected during the session who were male.
• Male percentage
  Percentage of "People count" that is male.
• Female count
  Total number of people detected during the session who were female.
• Female percentage
  Percentage of "People count" that is female.
• Uncertain percentage
  Percentage of "People count" with unknown gender.
• Count of people with impression
  Total number of people with at least one impression during the session
• Count of people with view
  Total number of people with at least one view during the session
• Average age
  Average age of people detected during the session.
• Average smile while viewing
  Average smile value across all people who viewed content during the session. Smile, on a scale of 0 to 100, is an indicator of interest.
• Total detection time
  Cumulative number of seconds across all people detected by DeepSight during a session.
• Total impression time
  Cumulative number of seconds across all detected people during which those people were considered to be in the impression state. In the analytics industry, this value is known as the "dwell time".
• Total viewing time
  Cumulative number of seconds across all detected people during which those people were considered to be in the viewing state. In the analytics industry, this value is known as the "attention time".
• Male total impression time
  Cumulative number of seconds across all detected men during which those people were considered to be in the impression state.
• Female total impression time
  Cumulative number of seconds across all detected women during which those people were considered to be in the impression state.
• Male total viewing time
  Cumulative number of seconds across all detected men during which those people were considered to be in the viewing state.
• Female total viewing time
  Cumulative number of seconds across all detected women during which those people were considered to be in the viewing state.
• Count of people with mask
  Total number of people on whom a mask is detected during the session.
• Count of people without mask
  Total number of people on whom a mask is not detected during the session.

Data collected by DeepSight - for Analytics

(The following is a summary of what can be found in the detailed user guide for the DeepSight Toolkit Interface Asset.)

The DeepSight Toolkit collects four classes of data that are collected primarily to facilitate the creation of informative charts using Intuiface Analytics.

Historical Data, Aggregated

A complete set of data collected for all people detected during a preset aggregation period, independent of what is being displayed onscreen. The aggregation period is specified in the DeepSight Toolkit and can vary from the past 30 seconds to the past 10 minutes. This data includes all people who have entered and left the frame.

See the section about retrieval details to learn how to access this data for analytics.

• People count
  Total number of people detected, regardless of whether or not they looked at the content. In the analytics industry, this value is known as the "opportunity to see", or OTS.
• Male count
  Total number of people detected during the session who were male.
• Male percentage
  Percentage of "People count" that is male.
• Female count
  Total number of people detected during the session who were female.
• Female percentage
  Percentage of "People count" that is female.
• Uncertain percentage
  Percentage of "People count" with unknown gender.
• Count of people with impression
  Total number of people with at least one impression during the session
• Count of people with view
  Total number of people with at least one view during the session
• Average age
  Average age of people detected
• Average smile
  Average smile value across all detected people. Smile, on a scale of 0 to 100, is an indicator of interest.

Session Data, Aggregated

A complete set of data collected for all people detected during a session. A session is defined as the period between a call to the "Start collecting session data" and "Stop collecting session data" actions associated with the DeepSight Toolkit IA.

See the section about retrieval details to learn how to access this data for analytics.

• People count
  Total number of people detected, regardless of whether or not they looked at the content. In the analytics industry, this value is known as the "opportunity to see", or OTS.
• Male count
  Total number of people detected during the session who were male.
• Male percentage
  Percentage of "People count" that is male.
• Female count
  Total number of people detected during the session who were female.
• Female percentage
  Percentage of "People count" that is female.
• Uncertain percentage
  Percentage of "People count" with unknown gender.
• Count of people with impression
  Total number of people with at least one impression during the session
• Count of people with view
  Total number of people with at least one view during the session
• Average age
  Average age of people detected during the session.
• Average smile while viewing
  Average smile value across all people who viewed content during the session. Smile, on a scale of 0 to 100, is an indicator of interest.
• Total detection time
  Cumulative number of seconds across all people detected by DeepSight during a session.
• Total impression time
  Cumulative number of seconds across all detected people during which those people were considered to be in the impression state. In the analytics industry, this value is known as the "dwell time".
• Total viewing time
  Cumulative number of seconds across all detected people during which those people were considered to be in the viewing state. In the analytics industry, this value is known as the "attention time".
• Male total impression time
  Cumulative number of seconds across all detected men during which those people were considered to be in the impression state.
• Female total impression time
  Cumulative number of seconds across all detected women during which those people were considered to be in the impression state.
• Male total viewing time
  Cumulative number of seconds across all detected men during which those people were considered to be in the viewing state.
• Female total viewing time
  Cumulative number of seconds across all detected women during which those people were considered to be in the viewing state.
• Count of people with mask
  Total number of people on whom a mask is detected during the session.
• Count of people without mask
  Total number of people on whom a mask is not detected during the session.

Final Aggregated Data, per ID

Collected for each individual person (i.e. for each person "id") when they are no longer detected.

See the section about retrieval details to learn how to access this data for analytics.

• At least one impression
  Did this person ever have the opportunity to view the onscreen content, defined as ever having a face angled within 45° of the screen center (either left or right) during their detection time.
  Can only be 1 or 0.
• At least one view
  Did this person view the onscreen content. The face angle will vary depending on distance from the camera but it will be smaller than the "Impressions" angle.
  Can only be 1 or 0.
• ID
  Unique integer representing a face detected by DeepSight - i.e. a face within 90° of the screen center. As ID's are determined anonymously, if a person is detected, then detection is lost, and finally is detected again, the result would be two different IDs.
• Gender
  Values are "m", "f", or "u" (for unknown)
• Age
  The estimated age of a person, represented by an integer. If unknown (e.g. with extreme head angles), -1.
• Max smile while viewing
  Max smile value (from 0 to 100) when viewing the content. Think of this as the size of the smile.
• Max smile
  Max smile value (from 0 to 100) when detected, whether or not they viewed the content.
• Total detection time
  Number of seconds a person's face is detected by DeepSight. If a person is detected, then detection is lost, and finally is detected again, the result would be two different IDs.
• Total impression time
  Number of seconds a person was considered to be in the impression state. In the analytics industry, this value is known as the "dwell time".
• Total viewing time
  Number of seconds a person was considered to be in the viewing state. In the analytics industry, this value is known as the "attention time".
• Total number of views
  The number of viewing sessions for a person. A new viewing session starts when a person looks away then back at the camera.
• Average viewing time
  Total viewing time divided by the number of views.
• Wearing a mask
  Binary value of 0 or 1, where 1 means a mask is detected. Use the "Face mask threshold" property of the DeepSight Interface Asset to specify the level of confidence (from 0 to 100) necessary to consider a mask is present. It is recommended not to go below 70.
• Timestamp
  The time at which this person was first detected

Session Data, per ID

Collected for each individual person (i.e. for each ID) detected during a session. Only collected if the "Aggregate session data per ID" property of the DeepSight Toolkit IA is selected.
A session is defined as the period between a call to the "Start collecting session data" and "Stop collecting session data" actions associated with the DeepSight Toolkit IA.

See the section about retrieval details to learn how to access this data for analytics.

• ID
  Unique integer representing a face detected by DeepSight - i.e. a face within 90° of the screen center. As ID's are determined anonymously, if a person is detected, then detection is lost, and finally is detected again, the result would be two different IDs.
• Age
  The estimated age of a person, represented by an integer. If unknown (e.g. with extreme head angles), -1.
• Gender
  Values are "m", "f", or "u" (for unknown)
• At least one impression
  Did this person ever have the opportunity to view the onscreen content, defined as ever having a face angled within 45° of the screen center (either left or right) during the session.
  Can only be 1 or 0.
• At least one view
  Did this person view the onscreen content. The face angle will vary depending on distance from the camera but it will be smaller than the "Impressions" angle.
  Can only be 1 or 0.
• Wearing a mask
  Binary value of 0 or 1, where 1 means a mask is detected. Use the "Face mask threshold" property of the DeepSight Interface Asset to specify the level of confidence (from 0 to 100) necessary to consider a mask is present. It is recommended not to go below 70.
• Max smile while viewing
  Max smile value (from 0 to 100) when viewing the content. Think of this as the size of the smile.
• Max smile
  Max smile value (from 0 to 100) when detected, whether or not they viewed the content.
• Total detection time
  Number of seconds a person's face is detected by DeepSight. If a person is detected, then detection is lost, and finally is detected again, the result would be two different IDs.
• Total impression time
  Number of seconds a person was considered to be in the impression state.
• Total viewing time
  Number of seconds a person was considered to be in the viewing state.
• Timestamp
  The time at which this person was first detected

Toggling data collection

To control the amount of data collected by a running experience, you can toggle whether particular data sets should be collected. This is achieved by using a variety of Start and Stop data collection actions.

Data Set	Start Action	Stop Action
Real-time Data	"Start requesting real-time data"	"Stop requesting real-time data"
Session Data, Aggregated	"Start collecting session data"	"Stop collecting session data"
Session Data, per ID	[same as above]	[same as above]
Final Aggregated Data, per ID	"Start requesting historical data, per ID"	"Stop requesting historical data, per ID"

Working with sessions

Sessions are used to distinguish one audience from another. The typical use of a session is to associate it with the display of a particular piece of content. Start the session when the content is first shown, stop the session when the content is no longer shown, and the resulting session-based data can tell you who viewed the content (e.g. age, gender) and how effective it was (e.g. views, impressions).

Only one session can be run at a time.

• Start a session
  Call the "Start collecting session data" action of the DeepSight Toolkit IA
• Stop a session
  Call the "Stop collecting session data" action of the DeepSight Toolkit IA

Retrieving data collected by DeepSight Toolkit

Accessing all data sets but "Historical Data, Aggregated"

Every data set created by the DeepSight Toolkit with one exception - "Historical data, aggregated" - has an associated trigger in the DeepSight Toolkit IA that is called when an updated data set is ready.

Data Set	Trigger	When Triggered?
Real-Time Data	"Real-time data received, aggregated"	Every 0.5 seconds
Session Data, Aggregated	"Session data received, aggregated"	When the latest session is stopped
Session Data, per ID	"Session data received, per ID"	When the latest session is stopped
Final Aggregated Data, per ID	"Final aggregated data received, per ID"	When a person (i.e. unique ID) is no longer detected

Create one or more trigger/action pairs using the DeepSight Toolkit IA for the appropriate trigger, assign conditionals if targeting particular "per ID" characteristics, and the experience will wait for the trigger to be called.

Data is stored as properties of these triggers, accessible to any action parameter via binding. For example, here are the properties of the trigger "Final aggregated data received, per ID":

How to report this data for use in Intuiface Analytics? Call the "Log Event" action, recording the desired data as parameters of an event you specify.

Accessing data for "Historical data, aggregated"

• Create a timer at the experience level that calls the "Collect historical data, aggregated" action.
  Note: Your experience timer should be shorter than the aggregation frequency set for the DeepSight Toolkit; otherwise, your experience may fail to collect some data.
  
  
  
  
• Every use of the "Collect" action will set off the "Historical data received, aggregated" trigger of the DeepSight Toolkit IA. As a result, create a trigger/action pair using the DeepSight Toolkit IA for "Historical data received, aggregated" and it will wait for updated data. Data is stored as properties of the trigger, accessible to any action parameter via binding.
  
  
  
  You could use this information to trigger the display of content based on the historical trends of your audience:
  
  

Sample experiences using Sightcorp DeepSight Toolkit

There are multiple experiences in the Intuiface Marketplace that are preconfigured to work with the Sightcorp DeepSight Toolkit.

• Face Mask Detection
• Demographic-Triggered Ads with Audience Metrics
• Automated Ad Playlist with Audience Metrics

To use these experiences you must first install and run the DeepSight Toolkit on the same PC as Composer or Player.