From Julius Volz, co-founder or prometheus.io and PromLabs, 6 mistakes to avoid if you are new, or even not so much, to Prometheus:

– High Cardinality, aka the number of unique time series as stored in the db;

– Aggregating too many labels/dimensions – usage of the aggregation operators such as by;

– Unscoped Metric Selectors – this would cause conflicts if we do not target the specific endpoint or service we are interested in;

– Missing for Durations in Alerting Rules – to protect against data that might present gaps;

– Short Rate windows – to calculate a trend Prometheus needs at least 2 samples, therefore too short windows might not provide you with the required samples;

– Using rate with the wrong metric type – For example, rate, irate and increase only work with counters!

https://www.youtube.com/watch?v=NEMsO1qeI1s