Big Brother gets a new pair of glasses

Closed-circuit video surveillance systems usually involve nothing more than security personnel watching a bank of monitors that display realtime video from cameras positioned elsewhere on gamblers at casino tables, travelers winding through endless lines at airport security checks, or the commotion at Occupy movement protests.

It’s a tedious, monotonous job. Most operators have an attention span of less than 30 minutes. Then their minds wander; their eyes may be open but their mind is at the ball game or the movies or already at lunch. There’s a reasonable chance they won’t notice the satchel left unattended by the propane tanks or a suspected terrorist trying to blend in with the crowd.

What if the video camera was “smarter” than the operator in the sense that it could stitch the images from a dozen cameras to one monitor instead of 12 and detect aberrant behavior or unusual, threatening activity? What if the camera was smart enough to know that there should never be an unattended satchel left near the propane tanks, and that when it “sees” a package or suitcase where it logically shouldn’t be it would automatically notify the operator through some alerting mechanism?

There’s a smart camera like that at Boston’s Logan International Airport right now, bolted to the ceiling of Terminal A.

Big Brother’s new eyes

Smart cameras, based on video analytic technology, are emerging in various security venues. One smart camera security system has been watching passengers at Logan International for the past year or so. It’s a test to see if the system performs as designed.

The new video security surveillance system, named Imaging System for Immersive Surveillance, or ISIS, was conceived at the Lincoln Laboratory at the Massachusetts Institute of Technology, and launched by the Pacific Northwest National Laboratory for the Department of Homeland Security’s Science and Technology Directorate. The version installed at Logan, called Spiral 1, sees the entire concourse like a large fisheye lens, but that’s where the similarity with most conventional surveillance systems ends.

This cutting edge technology is years ahead of the current commercial offerings, according to Pacific Northwest National Laboratory (PNNL).

Whereas a typical fisheye lens distorts the image and can only provide limited resolution, ISIS video is perfectly detailed from edge-to-edge. That’s because the video is made from a series of individual cameras stitched into a single, live view, like a high-res video quilt.

“Wide-area views generated by typical closed-circuit television systems are often chopped up into different monitors, so the security professional has to look at a bunch of different screens to get overall perspective,” says PNNL’s Doug MacDonald, ISIS project manager. “With ISIS, everything is now displayed on one large monitor.”

Capturing finely detailed, sweeping coverage requires an extremely high pixel count. ISIS has a resolution capability of 100 megapixels. That’s as detailed as 50 full-HDTV movies playing at once, with optical detail to spare. ISIS also has a zooming capability for getting up close and personal without losing clarity.

Each operator works independently with his or her own monitor and has access to any part of the 360-degree coverage. Further, each operator can review realtime video or access archive video taken moments or days before without affecting the operational performance of the system. Operators are able to use monitors at their desks, typically 24-inch flat screens, but at the same time have access to clear, high-resolution video never before provided. ISIS can see clearly up to 160 yards away.

When officials need to identify a suspect from among the throngs of passengers mulling about the airport, ISIS provides a clear enough image that operation of the entire terminal would not have to be interrupted.

In the event that a terrorist attack has occurred, forensic investigators can pore over the most recent video, using pan, zoom and tilt controls to reconstruct who did what and when. Because the controls are virtual, different regions of a crime scene can feasibly be studied by separate investigative teams simultaneously.

A stitch in real time

Stitching technology isn’t exactly cutting-edge magic. For years creative photographers have used low-cost stitching software to create interesting high-res panoramic images, such as the famous image of the National Mall taken on Inauguration Day 2009. But those are still images, created after the scene was shot with several cameras. ISIS quilts video in real time, while allowing operators to maintain the full field of view, even while a focal point of choice is magnified.

MacDonald says in the commercial world there are many software systems that can automatically identify suspicious activity, but they are proficient only in scarcely populated or unpopulated areas, where it’s easy to spot a lone person leaving a parcel unattended.

“If someone comes and sets a bag down and walks away, these systems will pick that up, but our goal is to do that in highly crowded environments,” he says.

Other ISIS features, many of which are commercially available, are provided by a suite of software applications called video analytics. One app lets operators define restricted areas, for which ISIS provides an alert the moment the area is breached. ISIS is designed for use in any environment where surveillance of large open areas is required.

Another feature lets operators select a specific target, such as a suspicious person, package or pickup. The system then tags the subject and follows it, automatically panning and tilting as needed. Video analytics at high resolution across the universal field of view, coupled with the ability to follow objects against a cluttered background, provides high-quality situational awareness as incidents unfold.

“What we’ve found in the past is the typical security professional watching a bank of monitors has an attention span of about 20 minutes,” MacDonald says. “What we wanted to do is give these professionals a tool that can continuously monitor an area, and then provide operator alerts to abnormal events or locate intruders in off-limits areas.”

Lights, camera, controversy

Many of these surveillance advances are troubling to those as interested in preserving freedom and civil liberties as they are in combating crime or detecting terrorists. They believe government dragnet surveillance activities are not directed solely at suspected terrorists and criminals, but they are instead directed at everyone, and therefore tend to erode the right to privacy and the freedoms of speech, association and religion.

The ACLU website states that “increasingly, the government is engaged in warrantless surveillance that vacuums up sensitive information about innocent people. And this surveillance takes place in secret, with little or no oversight by the courts, by Congress, or by the public.”

Steven Surfaro, Axis Communications’ industry liaison, says “the benefits of public surveillance in crime reduction have been studied and documented extensively and outweigh privacy concerns, as long as best practices are adhered to.”

Surfaro says any erosion of privacy or civil liberty in video surveillance are overcome when a system is designed and used according to established best practices.

ISIS Spiral 2

The ISIS Spiral 1 system installed at Logan International has worked well enough to be exhibited to visiting representatives of local law enforcement, branches of the military, as well as state and federal homeland security agencies. Several commercial entities with a stake in protecting the nation’s infrastructure have also shown interest.

A second generation prototype of ISIS, called Spiral 2, more than doubles the resolution to 200 megapixels and is currently undergoing operational testing. The results will be used to refine the technology before Spiral 2 is installed at Logan, probably sometime in Spring 2012.

The partnership between Logan International and DHS began in December 2009, allowing potential Homeland Security end-users the opportunity to evaluate ISIS technology. Beyond the potential for enhancing security at the nation’s airports, if successful, the current testing at Logan could pave the way for the eventual deployment of ISIS to protect other critical venues.

ISIS Spiral 2 will use custom sensors and video boards, longer range cameras, higher resolution, a more efficient video format, and a discreet, chandelier-like frame no bigger than a basketball. Eventually, DHS plans to develop a version of ISIS that will use infrared cameras to detect events that occur at night.

“We’re continuing to advance the video analytic capabilities of the software package, and incorporate features as requested by potential users that have been part of the demonstrations,” MacDonald says.

Many of the ISIS capabilities originated in technology previously developed at MIT’s Lincoln Laboratory for military applications. The current ISIS system was built with commercial off-the-shelf cameras, computers, image processing boards and software.

ISIS was nominated for the Christopher Columbus Homeland Security Award for cutting edge technology that could be applied to the protection of the nation’s assets.


Douglas Page writes about science and technology from Pine Mountain, Calif. Reach him at