Methodology for data anonymization
The data given to us is aggregated and kept private through a mathematical privacy framework that bounds the posterior belief of a third party relative to their prior belief. What this means is that if an observer has some prior belief—"person x is in the data set with confidence y"—then their confidence after seeing the data can only improve by a bounded amount: less than or equal to a factor times y.
There are many frameworks that rigorously define privacy with this guarantee: ε-differential privacy1, the Pufferfish framework2, and the Blowfish framework3 are just a few examples that rigorously define privacy. These methods have been battle-tested in machine learning4, telemetry collection5, and even the US census6.
is-there-a-drought.info implements a fully connected, global Blowfish model with ε = 10. The following techniques are all used to deliver this guarantee of user-level privacy:
- Suppression—removing records based on heuristics
- Sampling—picking values based on weighted probabilities close to the true values
- Noise addition—adding small amounts of noise to mask released values
- Synthetic data generation—data added to the aggregations to protect against distinguishing events while maintaining accuracy
Attack confidence example
We know ε = 10, meaning that an adversary's confidence in their prediction of whether a specific record is in the data set can only improve by a factor of ≈22,000x. While this may sound like a weak privacy guarantee, we can run through some rough numbers to get a better picture of adversarial confidence:
We'll use the 45,000 weekly visitors of r/dataannotationtech subreddit as a rough estimate for the total amount of workers on the platform. If the starting belief of an adversary trying to identify whether your data has been reported is 1/45,000, then in this situation they could at most be 33% confident after seeing the aggregated data.
TO CLARIFY, this rough analysis does not account for any other prior beliefs by the adversary that would increase their starting beliefs. Though in reality, an adversary's starting beliefs are most likely much smaller, as these calculations are for neighboring data sets (data sets that differ by one record). Starting without even a sample size—we do not publish this data—makes assembling an attack like this exceedingly difficult. Decimal values are truncated to two places as an extra precaution.
For more resources about privacy methods, please check the resources down below.
Potential biases and skewness of data
The analysis we perform relies on voluntary submissions from any potential user. While adversarial and mistaken reporters are heuristically removed, there may be some bias in who actually reports. The data shown on this website is very prone to self-selection7 and response8 biases. Without having an accurate estimate of the larger population of users, it is difficult to compute confidence for the released data.
In spite of this, we still believe that it can be a useful tool—just don't interpret it as gospel!