Downers is a term Wellies used to describe somebody who doesn't take their Joy. In We Happy Few you play as several Downers: Arthur Hastings (an everyday man fairly well known in Wellington Wells), Sally Boyle (a famous chemist) and Ollie Starkey (a retired war hero who is feared by everyone; red-suited bobbies will aggro immediately if he's seen). They all have major roles within the TBA story. You can find a corpse of a Downer by a broken Joy Detector being investigated by Bobbies and an act of a Downer breaking into a home and killing several Wellies that occurs in an Event within Hamlyn.
Bobbies and Wellies do not take kindly to Downers. In fact, committing any act that is not considered proper will get you labelled as a Downer. These acts are not limited to refusing Joy, but also stealing, assault and being caught in a suspicious area (such as bridge gates and crime scene investigations).
Wastrels are the downers who live in the Garden District; either by choice or through force.