I have a process which checks and processes a large number of domains for various stats. It involves http calls and is run multithreaded. The bottleneck is clearly CPU, the IO is not the bottleneck.

I am using Task.Run() and SemaphoreSlim to manage the threading. Each thread makes a http call and then awaits the response before carrying on processing which keeps threads open a while, in some cases up to 60 secconds.

The process takes about 30 days to run. I want to get that down under 7 days so I need to improve my code but also balance the load across several servers.

Can anyone suggest a pattern I should be following?

I am not sure if .net automatically optimizes the async/await threads or if what I am doing is poor performance and there is ways to gain performance boosts.

I need to get this process running over multiple servers but I am really not sure how to do that. Only way I can think to do it is to duplicate my API on multiple servers and share a database. These feels messy though and more work to maintain. I realise I could use App Services and scale up the instance but that would be outside of my budget.

Is there some way to setup some sort of .net core processing cluster that I can hook up a load of servers to? I feel like it must exist I just lack the terminology to find it on Google.

Any help appreciated.