It's actually changing very drastically when a single person votes, and the amount changed will drop off over time as more people vote. But again, they are weighted so that it is not absolutely a huge amount of influence for one person.

The algorithm for this is called a Bayesian Average which you can read about here: https://www.algolia.com/doc/guides/managing-results/must-do/custom-ranking/how-to/bayesian-average/ but the gist is that the ratings of a small group of people are trusted less than a larger group of ratings. If you have 2 items, one rated 5 stars and one rated 3 stars but the 5 star rated item was only rated by 1 person vs the 3 star rated one that was rated by five hundred people, which rating would you trust more.

I however, am not a statistician so the values I used to do this calculation (25% average rating with a confidence of 15) is very definitely flawed. More data is required before I can better tune these numbers to find a spot that works for what is best for all users and the site.