Sitecore reporting server getting lots of requests?

Reporting server issues in Sitecore 9.x

As part of an upgrade process,  we installed a new Sitecore (9.2) instance on PAAS. No custom code was deployed to it yet. Rather soon, we noticed lots of requests to our reporting server in the logs. As we started importing packages with content, the number of requests seemed to go up until a level where it became troublesome as our logs were really flooded with those requests. With a little help from Sitecore Support we found the issue and I documented it on Sitecore Stack Exchange: https://sitecore.stackexchange.com/q/23923/237

However, recently Chris Auer send this tweet:

Have you checked your reporting database lately? There might be a chance it is chewing up CPU ticks for little reason. One of our clients was seeing 100% every hour on the hour. #Sitecore 9.1 - 9.3 KB fix. https://t.co/hnoNbfWBWy pic.twitter.com/6701paBpLJ

— Chris Auer (@ChristopherAuer) February 26, 2020

This made me realize that more symptoms might be found for actually the same "issue".  A blog post might help to capture those in one place...

Symptoms

What are the symptoms you might encounter:

• rebuild of the sitecore_suggested_test_index is triggered every hour (this is actually the root cause, you should find this in your logs and probably think that is ok)
• High peaks on the reporting server or database as described by Chris
• Logs flooded with requests for: POST /~/v75/reporting/remotedatasourceproxy/

The symptoms will come in batches of an hour, as seen in this view of our Application Insights logs for the POST request to the reporting server:

The root cause

The root cause is actually the sitecore_suggested_test_index which triggers those POST requests. 

The issue is described on the Sitecore knowledge base, and appears in versions 9.1.1+ (reference number is 336911 - as soon as we see this in the release notes, it is fixed). Sitecore mentions this can increase resource consumption - meaning flooded logged and/or possible resource peaks.

The fix 

There is no real fix to stop the requests as those are apparently needed - but you can change the schedule as mentioned in the KB article: change the schedule info on /sitecore/system/Tasks/Schedules/Content Testing/Rebuild Suggested Tests Index to "20140101|99990101|127|1.00:00" for 1 day instead of 1 hour - or go for even less (7 days would be "20140101|99990101|127|7.00:00")  if you don't use the functionality.

Suggested test index

The suggested test index is described on the Sitecore documentation site. 

The Suggested Test index provides storage for computed values for all of the content items in the content tree and is used to suggest the items that should be optimized.

This index drives the suggested test list in the Experience Optimization and inside the Experience Editor. It is also used in the Sitecore client UI to show when a test has ended.

This might help you determine a good period for the rebuild schedule.