Writing Facial Recognition Software
IP/USB/File based video with real time Facial 3D Reconstruction, Recognition and Gallery Matching
Multiple FR Servers, Remote Operation, Management and FR Gallery Synchronization
Facial Recognition Gallery Management, Manual FR Matching, Multiple Appearances Per Person, Person Groups and Group Events
A video stream management dialog demonstrating the rich set of controls available per stream
Between the years 2015 and 2021, I was a Sr. Software Scientist for CyberExtruder, Inc., a globally leading developer of facial recognition software. Joining CE’s staff was the continuation of a multi-year relationship that began in 2006, when I originally contacted them to license their 3D Reconstruction software for use in my Personalized Advertising startup, Flixor. After licensing their deep learning based 3D Reconstruction software, I embedded it into an automated digital double creation pipeline, and then integrated that into high compute cluster backed a REST API, a web app, an eCommerce platform and quite a bit more. As I developed my company’s software, CyberExtruder, who was my company’s business partner, kept making me job offers. When my failed company was closing, I finally accepted their offer.
I became the Principal Engineer of their flagship product’s desktop server application, Aureus3Dv. CyberExtruder delivered software in two forms at that time: as an SDK for integration into other company's software, or as a standalone desktop server application. I was the Principal Engineer who developed the desktop server application. CE divided their engineering into two “teams” of one individual each: the CTO, a brilliant man with a statistics PhD, wrote the trained algorithms for facial recognition as well as the FR database creation, FR Matching, and remote synchronization while the “other team” (me) wrote the rest of the application and related subsystems. My responsibilities included:
- A fault tolerant application – meaning if the server were to crash for some reason, it automatically restarts and picks up doing whatever it was doing prior to the crash, with minimal if any loss of work.
- A real time multi-window application with windows supporting:
- FR Gallery review and maintenance,
- IP camera / USB camera / Media File sourced real time video windows with real time FR analysis,
- External data inputs for Remote FR Gallery Management
- External System Notifications configuration
- Additional remote instances of the application and their sync status with us.
- A universal export notification system able to deliver FR Match Notifications to arbitrary external software applications through a variety of communication methods.
- A REST API for external control of remote Aureus instances, including remote FR gallery, video stream configuration and management.
To support these capabilities I wrote, in C++ for both Windows and Linux platforms:
- An optimized for video analysis FFMPEG playback library supporting 295 different video formats, agnostic use of IP video, USB video or media files, with single core CPU decoding.
- This included a unique-to-CyberExtruder patch to FFMPEG related to dropped IP video streams and how FFMPEG handles such events.
- A fault tolerant wxWidgets based desktop server application supporting real time multi-windowed management of local video and still image sources, FR gallery compares against those image and video sources, the notifications generated by those compares, as well as remote operations of all of these through the REST administration of remote Aureus instances.
- The Export Connector Universal Notification System that delivers fully customized FR Match Notifications through a range of external means, including email, custom REST APIs, Server-Side Events (SSE), and more. When necessary, custom exports were implemented integrating Aureus notifications with dependent client systems, such as automated FR secured doorway access.
- Because wxWidgets lacks a modern looking user interface, I created a modern UI looking toolbar and window chrome for the multiple server application windows. Each window reacts to user mouse and key input in a similar fluid manner as web applications, with reactive simple animation.
- Additionally, I embedded the Chrome Browser Rendering Engine as the Server-Side Event Notification Visualizer. This provided a media rich scrolling window beside each FR Video window displaying the real time status and event notifications generated by that video stream.
I also wrote a large number of supporting libraries and applications to fill out a complete Enterprise FR Software Suite.