Tumbling during implementation
What two things have EUDAMED, an app for managing a specific cardiac condition and an injection pen for lay use in common?
- They had huge issues with the implementation.
- They are not in (official) use.
EUDAMED is being used, but not yet officially applicable. It should have been applicable since May 2020 and it appears it is delayed again. The struggle with EUDAMED is widely published, the other two products are only known to a limited group of people. Those two are failed products. Just like EUDAMED, they were highly developed, sophisticated pieces of technology. And just like EUDAMED the failure became clear when there was real life interaction with humans.
App with too much hunger for data
A drug manufacturer thought it would be a good idea to develop an app that would manage certain data of their patients, so they could carefully control the dosage of their drug. Instead of a physician deciding on the dose on a weekly basis, it became now possible to calculate that dosage on a daily basis. The problem was that in order to make that big leap in dosage control, the app needed a lot of extra data. Patients would have to spend a significant amount of time every day on their phone to feed the program. And who wants to spend more than 15 minutes per day on their phone…? This app really worked very well; the health and quality of life of the patients improved significantly. But when this app left the research environment and entered the real world, it quickly started to generate ‘low data’ warnings and users started to develop serious side effects as a result of incorrect dosages. This app failed in the implementation.
Injection pen: by engineers, for engineers
It was a great idea to develop an injection pen for a specific drug that users could use in their home environment. It was intended to replace a setup where a user would have to connect a needle to a syringe, draw up the required amount, replace the needle and throw two needles, a syringe and the rest of the medication away twice per day. Too many things went wrong there so these patients were seen by a specialized nurse twice per day. Apart from the costs, this had a big impact on their quality of life. Therefore the manufacturer decided to develop an injection pen that would hold an estimated amount for a full week. The user would have to indicate the required amount, attach a fresh needle, inject, remove the needle and store the pen in the fridge. In an attempt to avoid the reuse of a needle the device was equipped with a switch that would be loaded by removing the old needle and triggered by applying a new one. The needle also had a cap that moved sideways, so it would cover the needle again when removing. In that way the needle could be thrown away in normal household waste, without a risk of needle stick incidents. That cap had to be on one side of the pen to allow for injecting. Each prefilled pen was packed with a limited number of needles. Setting the dose had to be done before applying the needle. Every now and then the patient by mistake would apply the cap in the wrong direction, or forgot to set the dose. In both cases the needle would have to be removed and discarded. Frequently patient ran out of needles before the pen was empty. This resulted in patients trying to ‘work’ this system in ways that were not safe. This device was designed by engineers, for engineers; usability was a big issue. This injection pen failed in the implementation.
EUDAMED – the frontline between MDR and real life
The European medical devices database EUDAMED is a newly designed next generation from EUDAMED I that first applied in 2010 (see Commission Decision 2010/227/EU), later followed up by EUDAMED II. This ‘old’ version of EUDAMED was intended to be used by the authorities only, so they were also made responsible for data entry. This has resulted in unreliable data. Therefore, the MDR/IVDR EUDAMED had to be different: all economic operators must enter their own data and keep them updated. This implied there would be a wide range of users, with a very wide range in training or even in relevant language skills. Work on this version started in 2016, well before the Regulations became applicable. To ensure user involvement the European Commission created a Steering Committee of people with an interest and expertise in the subject. I was one of those committee members. In 2017 several working groups were setup, where stakeholders could also provide feedback, I also joined several of these working groups. The European Commission did their best to get user involvement, which resulted in many meetings. In these meetings results of data analysis were discussed. Obviously, that was essential. But that does not mean users could fully understand how all these detailed structures would function in real life.
EUDAMED was delayed from May 2020 to May 2023 and in July 2022 the European Commission added an extra year. An important factor in this delay has been attributed by user tests in the Playground by several beta testers. When this selected group of users started to confront this concept with the reality of clinical investigations and incident reporting, it became clear there were underlying problems that would require additional time to resolve.
With the explicit requirement to have EUDAMED ready by May 2020, it looked obvious to prioritize development over implementation. As a result, working group members could only start interacting with the user interface, and assess how the whole system would be working, in a late stage. On hindsight a different strategy, with users interacting with an interface, without the real database behind it. This would have resulted in a lot of handwork on the background, but it would have provided an opportunity to see the system interact with real life input. Unfortunately, this did not happen and EUDAMED appears for now failing in the implementation.
Implement first, develop later
There is a way to limit the risk of stumbling in the implementation of any product, procedure, system etc. This is an approach I started to use when studying for industrial design engineer at the University of Delft and I have further developed this in my professional career. This concept goes even further than usability evaluation, because it investigates user interaction before the product exists. It is the principle of ‘implement first, develop later’. What you basically do is assume the item you want to develop is already there, and then you make potential users interact with it. First they only have to do the ‘easy’ tasks, later they are allowed to throw their own complex issues at it. For example, you want to create an app to monitor a disease. In the first phases you just ask users to list certain characteristics on a form. The results on that form have to be reviewed by a physician or other expert and provided back to the patient by a note or message. If certain data are difficult to capture, or a result is difficult to understand, this will show. Step by step, the form can be changed into a user interface for data entry, and the feedback can be provided by another interface. Only once the interfaces are developed far enough to understand the data input and output requirements, the app can be fully developed. For the particular patients taking part in the development of this device it will feel as if the device has been implemented long before it was actually developed. It is also obvious that a similar approach for EUDAMED would have resulted in a slower start of the project, and probably a much faster final development stage.
The lesson learned should be that usability is not something to postpone until the end, just to get a box checked in a V&V checklist. It takes courage for management to start testing an interface before there is even proof the product or service is actually working. In my career I have seen only very few examples of this approach, but I have also seen that this approach often resulted in success. If you want to exchange views on device development, please check this service.
