Simon Reis
Narcissurveilance
In the digital era everyone is watching/ being watched. I am moved to seek alternate technologies that allow the freedom to track audience members but also log them as contributors to the artwork. This idea of tracking the audience was born out of an earlier interest I had as part of the Bishop & Reis collaboration. We were looking for ways to manipulate the human-art installation interaction. For a long time, many of our works had had an interactive element, some as simple as detecting someone was in the room; to other methods that attempted to get the viewer into 'the right position' to view a single point perspective. These generally failed pretty consistently. For example - the Infra-Red sensor that failed to work when more than 2 people were in the room, such as on opening night, the super accurate laser beam, set up to trigger a sequence in the Mnemonia Room (Bozo_Ink et al., 2007) that never worked because it was mounted to tree that was swaying in the wind, and so on. At some point we thought it a good idea to make the triggering sequence more cryptic, as part of x is not the same as x (Bishop & Reis, 2010) we expected the audience to look through a hidden spy-hole for 20 seconds before triggering the next reveal in the installation. No one found this spy-hole undirected, and one thing became obvious – we had been over-estimating the attention span of our audience. At some point I had started to read about pixel integration and manipulation from webcams and thought that this would solve all our problems, but it turned out to be much more difficult to implement that I originally thought. Then I undertook my Master’s and started to play with reflections of the audience, and again through failure, started to really nut out what it was about my art works that got me excited; it was the interaction the audience had with the work, not so much with the work itself but with their contribution to the work, be it their reflection or some digital representation of themselves that seemed to excite them the most. Often this representation was set up to relocate the viewer into another scenario, but that was generally over-looked; their focus was on themselves.
Computer vision uses a webcam connected to a computer running some flavor of software, to capture a scene or view and then interrogates and manipulates that scene, pixel by pixel. Computer vision programming is generally performed using a library, or a collected set of functions, that sits within a preferred programming language. It is just one ‘ingredient’ of many that make up the software. In this research I chose to teach myself the Python programming language and
OpenCV (Open Computer Vision library) as they are both open source software packages and have a fantastic community supporting them both. In facial detection OpenCV uses a process called the Viola-Jones algorithm. This algorithm works by breaking down an image, a still image or a frame from video, into smaller rectangles contained within a scan window, these smaller rectangles are looking for features that are defined within the program library in what is called a classifier. First the image is converted to black and white and then these smaller rectangles are indexed across the image within the scan window, comparing areas of dark and light in the image to what is in the classifiers’ definition. The classifier is something that is trained via machine learning, that is it is shown a lot positive images, with faces and a lot of negative images, without faces. The resultant features are extracted and applied as an expectant black/white pixel ratio. This ratio is then applied to each rectangle for each feature across the image until those facial features are located (or not). Typically, at this point you instruct the software to draw a rectangle around the face to indicate that a face has been found. The video below is a example of how OpenCV scans an image looking for a face.
OpenCV Face Detection: Visualized, 2010, Adam Harvey
American artist Adam Harvey has gone to a lot of effort to understand the Viola-Jones algorithm, this is so he can develop ways in which to fool, trick and hide from it. Since 2010, he has been working his project CV Dazzle (Harvey, 2010-2017), to develop hair styles, makeup and clothing to avoid detection by facial recognition software. This idea of hiding behind a dazzle camouflage was originally used in the first world war on battleships. Developed by the English artist Norman Wilkinson, the dazzle camouflage was not intended to hide or conceal the battleship from the enemy but make it difficult to detect the ships distance, heading and speed. Thus given battleship warfare is based on positional prediction, a target that could not be predicted was no longer a target. In the same way, Harvey is working on ideas that individuals can embrace that, although not hidden, are not seen by the algorithms in facial detection software. He sees this as a necessity for his own privacy in the digital era.
Dazzle-painting, 1918, Norman Wilkinson [Public domain], via Wikimedia Commons
Once the facial recognition software has detected a face within the frame of image, the positon of the face is accessible to the software. The image is defined by the pixel count of the resolution, or as Steyerl argues: ‘resolution defines the visibility of a picture’(Steyerl, 2013). Either way, the
arrangement of the pixels results in an (x,y) coordinate grid of the image. With the position of the face now located within the image, by applying a simple equation, the centre of the face can be defined. This is true for still images and individual frames of a video. Now that the centre of face is dimensionally known, a predefined rectangle of the image can be extracted out of the source image and saved as a secondary file for further manipulation.
I set up some trials in the front windows of RMIT’s Building 50; to test how the facial detected software would respond in a location I was investigating as a possible exhibition space. I had a vague plan to attempt to align a mirror to the audience’s face and track their reflection using the facial recognition software, in much the same way a camera tracking system moves in two axes to maintain a subject in the centre of frame. Prior to these trials, I had only ever tested the software at home, using myself as the subject. Setting up the webcam in the window, I initially started at an angle perpendicular to the flow of the public. I quickly discovered that, when taken in the context of the cameras field of view, most people walk quite fast and are only in the view of the webcam for a small period of time. Needless to say, the software rarely detected any faces; further reading after the event highlighted that the software was searching for a front facial aspect, not the side profile I was presenting to the camera. Testing continued with the camera at 45 degrees to the perpendicular and this proved to be more successful with more faces being detected. I still did note that the time duration the individuals spent in the cameras view was quite short, 1-1.5 seconds on average. What I concluded from this short test was that I will require some sort of distraction/ attraction to get people’s attention and in doing so, hopefully slow them down and get them to turn their heads, promoting a more frontal facial exposure to the camera, increasing the chances that the software will be able to detect their face.
Footage from Face Detection Trials, Building 50, 2017
This idea was beginning to feel like a return to earlier Bishop & Reis works, where the audience were was expected to jump through hoops to access the work – never over-estimate the attention span of your audience. I needed to strip back this work and understand what was working and what was not. The conceived work was small, mechanical and complicated, and more importantly, easily overlooked in the public space.
Running in parallel was the systematic rolling out of CCTV's across much of the western world. London’s CCTV network has received its fair share of publicity over the years as the most watched city, yet now the CCTV has become fairly ubiquitous in most developed cities. Last year I started to become aware of reports of the use of facial recognition software being used to analyse the footage of CCTV’s. Prior to this, 1 hour of footage required 1 hour of labour to analyse, making the reported 5.9 million CCTV cameras in the UK (BBC, 2015) seem far less evasive given it was just impossible to monitor them all. However, that all changes when there is some form of AI watching every second of footage, analysing, documenting and reporting on what is sees. This is where the forth industrial revolution takes to our public spaces with unknown consequences. What will our governments do with information such as the names of all those that attended a protest march, sat in during the Occupy Movement, or simply stood next to a ‘person of interest’ at the traffic lights. Not necessarily now, but on record for a life time.
ePassport gates in Heathrow Airport (Terminal 4),
Home Office [CC BY 2.0 (https://creativecommons.org/licenses/by/2.0)], via Wikimedia Commons
All this personal information including your biometric identity, that is the facial features that make you identifiable as you, will be stored on a national database managed by the Department of Home Affairs. This data base makes use of government held biometric information about individuals collected from driver’s licenses and passports photos.
Adam Molnar, a Lecturer in Criminology at Deakin University argues that 'if a biometric database is hacked, the information can potentially be abused by criminals over your entire life' (Molnar, 2015). Yet the government insists that the capability entails 'strong privacy safeguards'. Paul Virilio, a French theorist of technology argues that 'It is the duty of scientists and technicians to avoid the accident at all costs…In fact, if no substance can exist in the absence of an accident, then no technical object can be developed without in turn generating 'its' specific accident: The accident is thus the hidden face of technical progress…' (Redhead, 2006).
The Chinese are several steps ahead of the West in terms of implementing these national databases. In 2012 the government launched its ‘social credit system’, a growing database of its population, each with a rating score based on a range of behavior from shopping habits to online speech (Wang, 2017). Tapping into this and other national databases are smart sunglasses being trialed by police (see video below). These sunglasses have facial recognition software built into them and, assumingly, some sort of heads up display. What this all means it that the wearer can look at an individual member of the public and the glasses will compare what they see with a remote database and instantly inform the wearer whether or not the person in front of them is required for questioning. China doesn’t have a great track record when it comes to respecting privacy, Bloombergs have reported that in the countries west, facial recognition software alerts the authorities when certain minority groups leave a certain predefined ‘safe area’, essentially providing an invisible fence around a population (Bloomberg, 2018).
China's New Facial Recognition Gadget, 2018, TRT World
The one story about China and facial recognition that really caught my attention was titled ‘Chinese authorities use facial recognition, public shaming to crack down on jaywalking, criminals’ (Xu and Xiao, 2018). The article explained how, in the southern Chinese city of Shenzhen, authorities had managed to link CCTV footage to facial recognition software, and run the results through the Government database of known individuals. In doing all this, the authorities were able to track the pedestrians movement within a public intersection – detect that an individual has entered a roadway against control signage, jaywalked – then identify them within the national data base and then issue them with a fine before they have walked across the roadway. Complementing this system, is a large screen positioned on the side of the roadway, that publicly names, and shames, the offender. As the Shenzhen blogger, William Long, states in the article ‘there is no legal basis for the police to publish people's identifying information’, suggesting that the authorities are becoming a law onto themselves. Countering this approach to law and order, in September 2018, the European Court of Human Rights ruled that ‘the UK laws enabling mass surveillance violates the right to privacy and freedom of expression’ (2018b). How will this situation play out in Australia; the CCTV infrastructure is well established, the technology readily available, is it only a matter of time?
In September 2017 Bishop & Reis were invited to attend an information session for a commission of large scale public artwork in Geelong. What we conceived (and were short-listed for) was a very visual work, titled Citizen Act (in three parts: colour, movement and consciousness) (Bishop & Reis, 2017). The concept was to include panels animated through both physical mechanics and lighting; but it was the mechanisms that drove that animation that we believed were unique. We had conceived of the idea to tap into masses of data that surround our lives and to abstract it. The idea was to use live measurements of the atmosphere and compare them to historical records of the area, to act as a type of barometer of the social and environmental life of the city, playing this comparison out by altering the panels movement, speed and pattern. Connecting the work to the structure that supported it (Geelong’s Performing Arts Centre), the lighting system would tap into the performative energy of the building; using sound level measurement devices and video tracking, the sounds and movement of the internal spaces would be represented through light intensity and colour.
Proposal Video for Citizen Act (in three parts, colour, movement and consciousness), 2018, Bishop & Reis
The element of this proposal that influenced the next stage of the research was the concept of data collection. Having an engineering background has meant that I have always had an interest in the data amassed by systems; this data provides a means to an end, diagnostic logs tracking machine and operator movements, crash logs recording a system’s state - to the data produced by the very production of manufacturing - uptime, downtime, dimensional results, repeatability, accuracy, and on and on the list goes. This collection of data in industry even has its own acronym – SCADA (Supervisory Control And Data Acquisition) and every major control gear supplier has their own version that can be implemented into a production environment. From all this data, the next industrial revolution has sprung – the forth revolution. Building on the third revolutions use of electronics and automation, the forth is set to change the world. Often referred to as 'Industry 4.0', the Australian Government(2018a) states that this revolution refers to the current trend of:
-
advanced automation and robotics (including collaborative robots or ‘cobots’)
-
machine-to-machine and human-to-machine communication
-
artificial intelligence and machine learning
-
sensor technology and data analytics
This ‘revolution’ is changing engineering, manufacturing and the world at a rapid rate. According to the Founder and Executive Chairman of the World Economic Forum, Klaus Schwab, ‘We stand on the brink of a technological revolution that will fundamentally alter the way we live, work, and relate to one another. In its scale, scope, and complexity, the transformation will be unlike anything humankind has experienced before’(Schwab, 2016).
However, collecting data from a machine was all very interesting from an engineering perspective but it left a little to be desired in the creative realm. As noted, China is collecting data on its population, as are many other countries, no doubt. I took a punt here and assumed that accessing government collected data would be very unlikely, if not impossible. I was looking for a way to play up to this data collection, abstract it, reimagine it, invert it, but how does one do that without the data?
The Cambridge Analytica scandal (Ram et al., 2018) brought to the forefront of the media’s attention just what could be done with that data. Yet the data in question was sourced from Facebook, a social media giant, and was provided by its members voluntarily. Facebook believe that they shared the information of up to 87 million people with Cambridge Analytica (Schroepfer, 2018).
I had a passing interest in Facebook once. I was checking my emails in an internet café in Rotorua, N.Z, and had received an email from someone I had meet while travelling, inviting me to join up – so we could stay in touch. Sounded like a great idea for remaining in contact with family and friends, sharing photos of holidays, etc. I embraced this idea and started searching for others I might know using Facebook. I had amassed a grand total of 16 friends when it dawned on me one day, what I shared or discussed with one friend was probably not of interest to some or all of my other friends. For whatever reason, my real live physical friends didn’t all sit in one large group of similar likes and dislikes but were broken up from a multitude of social situations that often didn’t overlap. It was this behaviour in others that started to irk me; from the brag book approach, sharing everything that was wonderful about their life, to the pointless posts that really should not be of any interest to anyone and just provide digital noise, and this was coming from my friends, people I shared some common ground with. I was out, no more friend requests from me and nor would I accept any, my digital presence on Facebook stopped that day, sometime in 2008, mostly. Of course, the Facebook platform is now omnipresent; often to have some sense of connection to a community there is no choice but to access that community through Facebook; even as part of this Masters, the main collective and two-way form of communication is through the Facebook group.
So, I had concluded that accessing any data or information about the public, collected by state or government organisations was not an option, accessing the data and information collected by social media platforms seem to offer some possibilities, but those possibilities were quickly disappearing as Facebook and other social media platforms were busy rewriting their privacy policies in April 2018 to comply with the European Commission’s General Data Protection Regulation (GDPR).
In his 1973 work, Television delivers People (Serra, 1973), American artist Richard Serra argues that ‘you are consumed, you are the product – of television’. Written in response to television advertising, this phrase has never been truer than in our current digital world. Why are all these wonderful services free - Facebook, Google, Instagram? I am sure most people are aware that their data, their spending patterns and online browsing history is the currency of the day. We have all heard of the ‘algorithm’ that see and knows all about us. Social media has certainly cemented itself into our culture relatively quickly but what I am interested in is how we feed these algorithms certain information about ourselves, and what might be done with that information to turn it into a commercial product. Does a juxtaposition exist between the surveillance of society and societies' narcissistic obsession with social media? The governments of the world that are interested in deploying large scale CCTV systems want to do so to control the public space; but in the private world individuals are busy uploading images of themselves, sprouting their political beliefs and opinions; and the data that these actions generate becomes a tradable commodity. There is blurred line between what many argue is an invasion of public privacy and a voluntary presentation of the privacy of the public.
Television Delivers People, 1973, Richard Serra
Narcissism is a personality trait that has long interested psychoanalysts, Austrian neurologist Sigmund Freud described it as ‘a set of attitudes a person has towards oneself, including self-love, self-admiration and self-aggrandizement’(Raskin and Terry, 1988). How better to satisfy all your urges for self-admiration than to take a ‘selfie’ and post it on your favorite social media platform. Researches argue that social network sites ‘are attractive to those high in narcissism because they consist of a large network of shallow, impersonal relationships and give narcissists autonomy over how they choose to present themselves, often in self-enhancing ways’ (Buffardi and Campbell, 2008). So here was the driving motivation for certain sections of the public to feed the system.
When I first considered the Instagram hashtag ‘#selfie’ as a means to gauge the extent of narcissistic activity on social media, there were 347,991,294 posts, 4 months later that number is sitting at 362,529,638 posts, a 15 million post increase. In April 2018, as a result of the Cambridge Analytica scandal, Instagram and Facebook locked down a lot of their API’s (Application Programming Interface) that 3rd party software and app developers used to access the data on these platforms. This was all in an effort to increase the security surrounding personal information and the sharing of it; to reassure their users that their information was safe with them. This was all good, well intended stuff, but it really wasn’t helping my plan to appropriate this very data for my own creative ideas.
Somewhere in amongst all this talk of revolutions, data mining and privacy breaches, I was still holding onto ideas of the mirrored reflection and ways that this might tie in with facial detection software. I had been seeking a method that allowed the audience to really engage with their own reflection, yet subtly subvert the experience. Was it possible to remove someone from a reflection through mechanical distortion? Could they be isolated? Replicated? The Korean born, German philosopher Byung-Chul Han argues that digital media has 'destroyed our ability to gaze and we are therefore, staring past each other' (Han, 2017). I wanted to make use of the very technology that Han was being critical of, to re-engage the audience with themselves; develop an idea that would enable people to regain their gaze, with themselves and with others. This idea of the digital gaze, or lack of, made me think back to David Foster Wallace’s book – Infinite Jest; set in the not too distant future (from 1997), he introduces the videophone, a telephone with a video feed of your conversation partner. Wallace suggests that ‘it turned out that there was something terribly stressful about visual telephone interfaces… Good old traditional audio-only phone conversations allowed you to presume that the person on the other end was paying complete attention to you while also permitting you not to have to pay anything even close to complete attention to her’(Wallace, 1997). Could Wallace have predicted the disconnect that Han refers to and was it possible to play up to this uncomfortableness creatively?
I started to consider my previous works that had an uncomfortable element to them and the obvious one was ‘Down by His Luck’. This work addressed issues that were uncomfortable for many - the homeless situation in Melbourne. I was sensitive to how the work was perceived - middle class, middle aged, white guy makes art out homeless people, it doesn’t read very well, and the current state of homelessness wasn’t my primary focus. I was interested in exploring the idea of post-humanism homelessness, developing some sort of automaton that wandered the street, a type of mobile surveillance device – lost and disillusioned yet always performing its primary task of surveillance. At what point will an automaton have any sense of home; therefore, be capable of being homeless? About this time, I came across the American multimedia artist Tony Oursler, and his exploration of facial recognition software and surveillance. As part of the exhibition b0t / fl0w - ch@rt (Oursler, 2017) in the Galerie Forsblom Gallery, Stockholm, he arranged several glass works comprising small flat screens imbedded within a glass, head like sculptures; on the screens were human facial features – eyes and a mouth – loosely positioned in the correct anatomical position. What resulted was something identifiable as a head, but it made hints at an attempt to humanise the robot, to soften it with human features. His intention with these works was to question where artificial intelligence will take the humane race – to its doom or to a utopian world (Oursler, 2017). Oursler used pre-recorded footage of the human features in his work; would it possible to use facial detection software to extract these features from the audience? Could these facial features be appropriated into the work – displayed on similar small screens on the face of an automaton in a public space?
In discussing these ideas with supervisors and peers, a red flag began to appear; it was the homelessness element, how did it add to my research, if at all? I concluded that it was an effort to tie the idea back to a previous work, perhaps, preventing me from moving forward. Thus, the idea of a humanoid/ automaton housing this idea was scrapped, however, I still wanted to pursue the concept; but what was the alternative? Could the uncomfortable come from the interaction with a device, not the approach? The idea was to capture and display the publics face back to themselves, I was stealing their image, could this act make someone uncomfortable? This idea needed to be housed in a physical structure; one both familiar and functional in the public space, providing a sense of security that this structure was built with purpose, yet not intimidating. Innocuous. Ultimately, this device is a human image capture interface, more than just a camera, it must communicate with the public about its purpose and intent as well as provide a medium to present to the public their captured representation. Most importantly, the device must be autonomous in its function for it is representing the hardware we embrace in our lives, smart phones, tablets, computers and anything else sitting in the realm of the Internet of Things (IoT). The IoT is the network of physical objects that collect and respond to data shared over the internet, embraced by the hacker community and made possible by cheap electronics, this image steeling camera needed to suggest it was part of something much larger.