This is not caching but a bug with the display of scores. The easiest way to see it show up is when there is new activity on the votes. Note the yellow highlight in the reputation event - that means it is a new event (something you have not seen yet when visiting your reputation page). By comparison the items with grey background are already seen.
To clarify new items (yellow background) have a problem where the calculation is incorrect. As far as I can tell, the calculation is always wrong. However, the result might coincide with a correct calculation. Furthermore, the calculation is consistently incorrect. You would not get random results - the calculated value will always be the same. The calculation is therefore taking into account factors it should now. By comparison, old items (grey background) are always correctly calculated.
Furthermore, the problem is in the "Time" view for reputation changes (toggle buttons on the right side above the reputation list:

When the issue is encountered, a refresh will "fix" it but only because the item changes status to already seen, thus using the correct calculation logic, instead of the one for being unseen.
You can force the bug by appending the StartDate
parameter to the URL:
https://rpg.stackexchange.com/users/75095/groody-the-hobgoblin?tab=reputation&StartDate=2023-01-12+07:00:00Z&sort=time
Link
See comparison of the results with and without the parameter:
With the StartDate parameter set to 2023-01-12 07:00:00Z:

Some highlights:
- There are eight events shown for today (2023-01-13).
- The total for today is +95 reputation, yet somehow one of the entries is for +90. If the rep changes are summed up the result is way off to what the total is +10 +10 +10 +10 +15 +10 +90 +10 = 165 vs 95.
- One of the new entries for yesterday (2023-01-12) is an upvote event for +170 yet the post it relates to sits at only +10 / -2 votes at this point in time.
When the StartDate parameter is omitted:

Some highlights:
- There are six events shown for today (2023-01-13) compared to the eight shown before.
- The total for today is +95 reputation and the events do add up to exactly this number: +10 +10 +20 +30 +15 +10 = 95.
- The events do show completely sensible values. The upvote entry for the post which had at least one +170 before is just a +110 which is supported by all other site metrics: the post history page shows the post got 11 upvotes yesterday and 1 today. Thus the post resulted in 110 reputation from yesterday.
This issue has been reported multiple times on Meta Stack Exchange and Meta Stack Overflow over the past year. Most likely more Meta sites, which I did not really follow.
The earliest report I can find is Ghost (nonexistent) votes in reputation tab which is from 31st of March 2022. There was a bug from 23rd of February 2022 previous to that which was related with the display of reputation history events after the profile pages were updated see here and here for examples. That was fixed. Presumably the behaviour I described above is also related to the profile updates.
It has been nine months since the earliest report and so far, the responses I have seen from Stack Exchange to the reports are that the bug cannot be reproduced or no response at all. Thus I cannot even give the usual six to eight weeks estimation for a fix.