Kepler’s Automated Human Activity Recognition in Care Homes: Past, Present, and Future

Our company, Kepler Vision Technologies, has now been in business for 9 years. Our software is used at more than a hundred care locations. We have zero churn, meaning that expiring software agreements get extended for three to five years. This means that if you buy our software, you will have a long-term relationship with us. Therefore, if you are considering engaging with us, I believe you’d be interested first in learning more about Kepler’s past, present, and future. That is what this article is about.

Kepler Vision’s Background

About fifteen years ago, two Dutch AI professors and I had spun off a computer vision company from the University of Amsterdam. A few years later, the company was acquired by San Diego-based multinational Qualcomm [1]. Our intellectual property adviser at the time had successfully licensed his peer-to-peer patent portfolio to Apple, Microsoft, and Spotify [2]. All of us, the professors, the IP advisor, the University, and I, then pooled together some of the proceeds and started a new University spinoff company, Kepler Vision Technologies.

Our plan was to bring a new emerging field of AI, called human activity recognition, to the market. Our plan was also not to waste our careers on, for instance, keeping someone on a phone app to increase the chances of clicking on an advertisement. Instead, we wanted to have a meaningful impact in the years to come.

Demographics are good predictors of what will happen decades from now. Demographic projections indicate that populations in Europe, China, and Japan will age. This means that the demand for care increases. At the same time, the availability of caregivers decreases. For the Netherlands, where I am from, the number of long-term care workers per 100 people aged 65+ fell by more than 20% in the last ten years [3]. This shortage will increase rapidly in the years to come. Furthermore, a third of the elderly fall every year [4], making falls the second leading unintentional injury death after road traffic injuries [5].

Our dream, therefore, was to build a calm, intelligent layer of awareness of human activities that could protect elderly, vulnerable people. The layer should help understaffed care teams respond earlier when needed, thereby giving caregivers more peace of mind. The intelligent layer would depend critically on the latest computer vision research, which is the specialty of my fellow founders and me.

The Challenge of Starting a Company Serving Care Homes

The main uncertainty at the time was whether care homes would accept cameras as sensors in residents’ rooms. There were indicators to suggest that this may be the case. Cameras that look like smoke detectors were being sold to Dutch long-term care facilities. Using geofencing, an area in front of the door or the bed could be manually outlined. If many pixels in these areas changed intensity, an alarm would be triggered.

My main concern with this rudimentary “smart” sensor, when we started Kepler, was that a geofencing-based motion detector was outdated technologyat the first location with 20 beds, but only if the system integrator has capacity; otherwise, the rollout needs to be delayed by 1 or 2process everything. It would lead to unnecessary false alarms, for instance, when a pillow falls from the bed or a foot sticks out. In addition, the solution was unreliable. If the cleaners in a care facility accidentally move the bed, the move invalidates the geofenced area, resulting in missed fall detections. Modern AI solves all these problems, and this insight led to our first product, which we called the Kepler Night Nurse.

In the early years, selling our solution to care homes proved to be difficult, though. Carehomes are not military organizations, where decisions are taken at the top and executed at the lower levels. In care homes, the number of decision-makers and decision-influencers is endless: family representatives, caregivers, IT managers, board members, the innovation manager, the operations manager, and the Chief Financial Officer all need to be involved.

Furthermore, such a decision process takes time. My experience of how the introduction of new technology in care homes works:

• It takes one or a few years to make the decision.
• The product is rolled out in the first location with 20 beds, but only if the system integrator has capacity; otherwise, the rollout needs to be delayed one or two years.
• It takes a few more years before the last location is rolled out.

This means that for a startup company to survive the first, say, three to seven years, it needs to be very well funded. It will take a long time before any money comes in. There are no shortcuts for the startup company. Offering a discount does not expedite the outlined decision process, and hiring more sales staff doesn’t either.

What also does not help is that, in the care industry, knowledge is passed on through a master-apprentice relationship. Already during the study, nurses and doctors have extensive clerkship periods. The master has no experience with AI, so innovations trickle down very slowly.

Finally, there were also the technical challenges. AI needs training data. As it turned out, the smoke detector-like camera commonly used in Dutch care homes was a fisheye camera, which is fundamentally different from the dome camera on your smartphone.There are no fisheye datasets available for training. In addition, no datasets were available on people falling. We spent significant time collecting training data, and we didn’t know whether building a reliable, computer-vision-based fall detector would be possible.

Further complications were that care homes don’t want their data processed in the cloud, demanding that we do all processing on-premises. Such on-premise processing brings with it its own technical challenges, which are not impossible to build but which still take up time and costs: High upfront investments in hardware, dealing with hardware failures, building automated redundancy, responsibility for updates and security fixes, the list goes on.

Finally, there is the new issue of geopolitics. To bypass the on-premises issues, we started by developing our software to run embedded on cameras. The first chipset we ported to was from the Chinese manufacturer Hisilicon. Porting took six months of time. During this time, to our amazement, for the first time, American export restrictions took effect, meaning the camera with the chipset was never brought to market.

Where We Are Today, at the End of the Early Days of Computer Vision-Based Fall Detection

We have overcome all of these outlined difficulties. We have data on falls in abundance to train our neural networks with. We have rolled out our software, on-premise and embedded, at more than 100 care locations.Our software quietly looks after 15,000 residents 24/7 in many different countries, from Europe to Japan to the US.

Today, we help these care organizations create safer, calmer, panic-free environments through intelligent monitoring. Our technology turns smart sensors into reliable care support, helping staff notice important situations earlier, reduce unnecessary alarms, and focus attention where it matters most. Our technology secures the availability of care.

This success did not go unnoticed. We saw and still see “me-toos” come and go.

Developing AI-driven fall detection is similar to developing self-driving car technology. It took Silicon Valley celebrity George Hotz a month to build a self-driving car in his garage [6]. But it takes a generation to make a self-driving car that you can fully trust.

This is also true for fall detection: It is easy to build a prototype, it is very difficult to build a reliable solution. You can easily verify this for yourself. There are plenty of well-conditioned demonstration rooms where newcomers showcase their fall detectors. However, over the past seven years, during which we have demonstrated our fall detection solution in the wild at the Zorg & ICT trade shows, I have never seen another vendor’s live demonstration. The newcomers tell fairy tales through PowerPoints and prerecorded videos at trade shows, but their software simply still does not work in the wild.

The good thing about competitors, of course, is that they help create the market and support the category’s future, as care homes now have something to choose from.But today, the newcomers elevate the risk for care homes of relying on an untrusted supplier. To outline a few of these risks:

1. The competitor company is bankrupt by the time the rollout is planned. This is due to the multi-year decision period and the lengthy go-live process. This is difficult for a start-up company to survive. Except for in the bible, little David usually gets killed quickly.
2. Failing installations. A proper installation requires a series of dominoes to fall, including positioning the sensor in the room, configuring the camera parameters and software settings, and connecting to the nurse call system. This often goes wrong with premature suppliers.
3. Malfunctioning software. Often, newcomers lack access to training data and therefore train their neural networks on synthetically generated imagery. Then, after the installation of the newcomer’s software, nightly rounds are omitted, as hopefully the same level of care can now be provided with fewer staff. Then a resident falls, but the immature software fails to detect it, having never seen real falls, only synthetic data. The staff and family are very upset, and the care home’s management regrets its choice.
4. In a growing field like computer vision-based fall detection, it is becoming increasingly difficult to steer clear of other people’s patents. As the pie grows, a new dynamic emerges: patent holders seeking their share. The exposure applies to the whole chain: from the me-too software vendor, to the distributor distributing the software, to the system integrator installing the software, and finally to the care home operator using the software.

Lance Good: “It is nearly impossible to write any non-trivial software that is free from patent encumbrances.” [7]

Future: What Will Tomorrow’s World Look Like?

The demand for care will grow exponentially in the years to come. For the Netherlands, without major changes, by 2040, one in four workers would need to work in healthcare, compared with about one in six today [11]. Thus, the demand for technology that helps caregivers perform their work will only rise.

To that end, alternative technologies are developed. But they are still significantly inferior to vision-based methods. There are a few: radar, time-of-flight, and depth. All of these claim to provide better privacy than vision-based methods. This is not true. Automated vision-based methods have become so reliable that visual verification, that is, looking at the video stream in case of an alarm, is simply not needed.Computer-vision-based technologies perform much better in practice, too [15].

In the coming years, top firms will add new functionality that goes beyond just fall detection, such as decubitus prevention. In addition, computer vision-based human activity recognition will move out of:

• Professional care homes into the communal rooms and into private homes;
• The hospital’s clients’ room into the operating room and into the waiting room.

We know for sure that in the coming years, care demand will keep rising because of aging, more chronic diseases, and more elderly people with health problems. If we don’t act, staff shortages will make it impossible to continue providing all forms of care as currently provided [12]. Still, I am optimistic about the future, and Kepler’s capacity to provide meaningful, necessary products for care homes:

• Our company’s software has been installed 15,000 times so far.
• We have zero churn.
• We have our intellectual property secured, with 86 granted patents, so we will be able to continue providing our services in the years to come.
• We continue to realize significant improvements in quality. By continuing to improve our AI, we went from 1 missed fall per month per 277 client rooms to 1 missed fall per 416 rooms in just 18 months [13]. That is a 44% improvement.
• Being a University spin-off, we are well-positioned to catch more of the current tailwind generated from computer vision and AI research.

Technology will help make elderly care available in the years to come.