The Approach to Effectiveness Evaluation and User-Centred Design

Prev Next

Comments from leading digital health apps at HETT 2019

In January 2019, the NHS Long Term Plan was released and Chapter 4 read ‘Digitally-enabled care will go mainstream across the NHS’.

Over the next ten years, the healthcare industry looks towards a landscape of digitalised care. This vision describes clinicians accessing secure patient records remotely, communicating with their patients within digital channels and patients self-managing their health and wellbeing through digital platforms. Digital Therapeutics (DTx) that optimise care pathways and successfully improve patient outcomes is becoming the latest exciting and challenging project for digital health innovators. 

Navigating the current digital health space can be difficult. What makes for a successful digital health app? How should we evaluate? What is the roadmap to being listed by the NHS Apps library, becoming mainstream? And is there one? 

The pathway of the NHS Digital Assessment Questionnaire (DAQ) 


Below is the pathway for being assessed by the NHS as a digital health app, involving providing evidence for effectiveness and adherence to user-centred design, among many other requirements.

HETT 2019 brought together some of the top ORCHA registered apps, including My IBD Care and My Arthritis. Some of these are currently registered on the NHS Apps Library, and others are still working towards this. Here, we spoke to leaders in the field who shared comments on their approach to:

  • Effectiveness evaluation – ‘Available Evidence on Outcomes’
  • User-centred design – ‘Accessibility and Usability’


Speakers on a panel discussion at HETT discussed what ‘good’ evidence for effectiveness evaluation looks like. Speakers included representatives from NHSX, ORCHA, Public Health England, MedCity and Closed Loop Medicines. These bodies advised digital health innovators to:

  • ‘Invest in your team – up-skill them in academic evaluation
  • Build your evidence base before you drive your product forward
  • Outline your evaluation goals from the start – this will differentiate you and guide you at every stage of progression’

The DAQ asks app developers to share evidence of any clinical, behavioural or economical benefits of the product, demonstrating improvements in patient reported outcomes. Digital health innovators can view the hierarchy of types of evidence here – which lists Systematic reviews of RCTs at Level 1a (producing the strongest evidence) and Expert opinion pieces at Level 5. 

How do ORCHA listed apps approach effectiveness evaluation?

Smoke free – Quit smoking now

Smoke Free is the world’s most popular stop-smoking app. The app incorporates the best available evidence applying psychological theory and Behaviour Change Techniques (BCTs) which have been shown to be effective in promoting smoking cessation.

We asked David Crane, Founder of Smoke Free – ‘What does a good effectiveness evaluation method look like to you?’

‘Randomised Controlled Trials (RCTs) are obviously the gold standard of effectiveness evaluation. The two elements are randomisation and control. So you could randomly allocate versions of the apps – including different BCTs – you could do this using AB testing. It’s good to start by internally testing your intervention – as this helps you iterate your product. You might find that for this population – this BCT or collection of BCTs does not seem to be effective. You could take this out or change the way you implement it within the app. Only once you feel confident in your product, is it recommended to move onto an RCT.’

Smoke Free is allied with academic leads within the UCL Department of Behavioural Science and Health, and such academic guidance can be invaluable in the design of robust effectiveness evaluation. Furthermore, by using BCTs as the building blocks to designing and evaluating an app, it can be easier to determine what the mechanisms of change are (what is leading to a change in behaviour and patient reported outcomes). This intricate level of understanding of your product’s effectiveness adds further value to the evidence you present.

Equoo – PsycApps

eQuoo is an emotional fitness game to help support young people’s development of emotional and psychological skills. A recent 5 week Randomised Control Trial found that eQuoo significantly increased wellbeing metrics and decreased measures of anxiety. 

We asked Silja Litvin, Clinical Psychologist and Founder of PsycApps, eQuoo – What is your best advice for becoming part of NHS Apps library?

‘I would recommend using ORCHA and having a long breath! We started the process in June 2018 and we are just at the last stages of this. We are waiting for someone to “check the box” currently.’ 

eQuoo have an additional RCT beginning in September, which will test the effects of the app on depression within a larger student population. If an app can complete the DAQ highlighting evidence from trials within larger samples of relevant user-groups, the strength of evidence will be considered more robust.

Sleepio – Big Health  (PART OF THE NHS APPS LIBRARY)

Sleepio is a DTx solution to improve sleep and in turn, mental health outcomes of users. Cognitive Behavioural Therapy for Insomnia underlies the structure of the DTx solution and Sleepio has the world’s largest evidence base for any DTx, including 8 published RCTs. Sleepio continue to light the way, demonstrating the importance of conducting robust effectiveness evaluations.

In conversation with Charlotte Lee, UK Director of Big Health, we talk about the importance of a skilled Research and Development team and within this, a leader for evaluation. Resources within smaller digital health organisations can be limited, however sufficient funding and leadership needs to be allocated to evaluation. Apps in this field should be able to produce an evaluation plan that stands up ‘ethically, commercially and academically’.

Sleepio share their goal:

‘Our goal is to become mainstream in the NHS. We have so much clinical evidence. The next step for us is to test out that implementation model, build up our business case and see if we can really genuinely save the NHS money as well as improve clinical outcomes’.

User-centered design

The Usability and Accessibility questions of the DAQ request adherence to user-centered design. Sam Shah, Director of Digital Development within NHSX, is featured in a HETT 2019 video speaking about the importance of addressing clinical, emotional and practical needs when creating any digital app or wearable. It is important to focus on the UX of digital platforms for both the user/patient group as well as addressing challenges of implementation and UX within the NHS – how does this product fit into an integrated system of healthcare?

How do ORCHA listed apps approach user-centered design?

My IBD Care, My Arthritis – Ampersand Health

Ampersand Health are focused on improving patient outcomes for those with long term inflammatory conditions such as IBD and Arthritis. The apps are free for patients and the companion product, the clinician portal, is licensed to hospitals. This means as well as patients being able to self-manage their lifestyle and condition through the app, users can also be digitally connected to their clinical team. Alongside this, Ampersand Health’s Behavioural Science team are currently developing digital therapeutic lifestyle programmes to support the improvement of patient reported outcomes.

Ampersand Health are committed to involving both Clinicians and Patients in the design of their solutions. Running co-production user workshops and partnering with national charities such as Crohn’s and Colitis UK and National Rheumatoid Arthritis Society – Ampersand Health value input from their user community. 

Kishan Wadhia, UX and Design Lead at Ampersand Health –

‘The Usability and Accessibility section of the DAQ highlights the need for understanding the user profile and needs. I always strive to provide a high-quality, user-centered product and I find co-creative user-workshops necessary in all that I do. Furthermore, the DAQ asks how we collect feedback post-release. One of the ways we do this is by our new feedback feature. The great thing about this feature, is that we limit obstacles in the user experience as users no longer need to exit the app and head to their app store to send us immediate and relevant feedback. This enables us to act upon and consider user feedback swiftly and focus on delivering a product that is best for them.’

As Sam Shah notes, consideration of successful integration within the wider NHS system is also crucial. Collaborating with MHabitat, King’s and Bart’s, Ampersand Health apply the NASSS framework to ensure successful implementation and great UX of their clinician portal.

Brain in Hand

Brain in Hand offers access to personalised digital self-management tools and human support. It is used by those with neurodiverse mental health difficulties and reduces demand on carers and support services. Users are set up with personal coping plans so that when they are distressed, they can reach for their Brain in Hand and navigate the situation they are in. 

We asked Paul Allis, Business Development Director at Brain in Hand ‘How do the features within the app demonstrate a sound understanding of your user group?’

‘My ongoing vision is to help more people achieve what they want to do in their day to day life, free of panic and stress. This is what drives me. Personalisation is really key in our solution. The fact that the BiH responder can be alerted and see what has happened in that person’s day that has overwhelmed them, means that they can offer personalised advice and support in that moment. This is something that our users really value, alongside the opportunity to sit down with a BiH specialist to set up a coping plan.’ 

A deep understanding of the needs of the user is what underlies a product with high user acceptability. The more targeted the digital solution is to the user, the greater scope for positive impact on their health and wellbeing.

Calm Harm, Clear Fear – Stem 4PART OF THE NHS APPS LIBRARY 

Stem 4 is a teenage mental health charity aimed at stemming common mental health problems through the clinician developed apps. They incorporate the principles of Dialectical Behaviour Therapy and Cognitive Behavioural Therapy within their co-produced apps – Calm Harm and Clear Fear. 

We asked Nihara Krause, CEO of Stem 4 and Consultant Clinical Psychologist – ‘How do you keep your users at the centre of your design?’

‘It is our priority that we keep young people at the forefront of our mind when designing for them. The apps are colourful and host recognisable animated characters which are linked to certain activities. We run user-groups to co-produce something they will love. Our in-app activities do not require wifi which is something we think is important for our user group – this means they can whip out their app on the tube and other places where they do not have signal – and still access coping tools and guidance.’

It is one thing basing the app off sound scientific evidence, validated therapies and theory and another thing designing a tool that the user-group will actually engage with and enjoy. Digital health innovators should do our utmost best to align with and understand the needs and context of the user in order to produce solutions that will successfully fill gaps in care pathways.

A closing comment from Rob Daly, Data Scientist at ORCHA

‘On our scoring system we have a threshold of what ORCHA would describe as a ‘good’ app after our assessment process. When we have app developers that go over that threshold, we often advise and encourage them to approach the NHS. We do partner with the NHS services, we help them develop the Digital Assessment Questionnaire – DAQ. Something we are working on at the moment is researching UX, what actually constitutes good UX within an app.’

ORCHA were recently awarded a grant by Innovate UK and are inviting app developers to take part in their study alongside ORCHA and key NHS leaders to shape future standards of UX.

A closing comment from Ampersand Health

Raghu Ram Nair, Product Manager at Ampersand Health –

‘We are currently in the process of bringing together the elements necessary for the NHS DAQ. Our goal, whilst facilitating a switch from existing standard of care to a digitally-enabled care pathway, is maintaining clinical safety, while saving costs to the system. 

Alongside other apps working towards being listed by NHS, there are certain challenges associated with evidencing effectiveness such as:

(1) Sufficiency of data (quantitative – number of on-boarded patients plus period of monitoring)

(2) Quality of data  (accuracy, fidelity)

(3) Ability to foster and maintain a healthy patient engagement 

These challenges are being addressed through a clinical trial we are conducting – which helps generate clinical and health economics data. As Kishan mentioned, Ampersand Health regularly connects with patients through workshops and digital channels to generate awareness and promote engagement.

Beyond demonstrating effectiveness and excellent user-centered design, many digital health innovators face challenges with areas of the DAQ regarding data security and protection. We have setup policies and processes towards this and good practice in the security of our patient records is an ongoing exercise for us.’

Recognition of our progress so far:

  • CE marked, regulated by the MHRA and has been assessed by NICE and Orcha.
  • Recognised in the Digital Health Global 100, winner of the 2017 NHS Ideas Lab and the 2018 NHS Hackfest and an HSJ Award for Patient Participation; and was accepted into PWC’s 2018 Scale Health accelerator and the NHS’s 2019 DigitalHealth.London Accelerator.


To view the other brilliant apps that were within the Orcha Zone at HETT 2019, visit here.

For more information on Ampersand Health’s DTx solutions, contact

Post on Facebook Tweet Post on LinkedIn Email

My IBD Care

My Arthritis