Scalability Testing Citrix Federated Authentication Service (FAS)
- alynpeden
- Oct 6, 2022
- 3 min read
Updated: Jan 8
Citrix FAS Multi-Threaded Script and Load Testing
Having worked on a large Citrix Federated Authentication Service (FAS) deployment of 100k+ users we had decided to pre stage user certificates once a week to ensure FAS\CA was not a bottle neck during logon storms. As the user base was geographically spread the largest Home Datacenter location would consist of around 25,000 users which meant pre staging 50,000 certificates (2 per user for HA) each week via PowerShell.
During testing of the standard FAS pre-generation script (linked at end of doc) I found that its not quite going to be able to service 50k certs in any sort of acceptable timeframe as it effectively does 1 at a time 50,000 times. During testing it took around 1 hour to generate 1000 certs and since we need to generate 50,000 certs the timing was not going to be acceptable. To solve this issue we created a multi-threaded script and managed to get the 50,000 certificates generated in under 5 minutes by splitting out the retrieved UPNs into multiple variables then running a separate job for each variable.
The script can be found here and in this case it does not look up an OU but instead uses an AD group for querying the users. I did find that Get-ADGroupMember has a limit of 5000 results and that can only be increased by editing a config file on the DC so instead I used Get-ADGroup that actually runs in half the time to retrieve the users when tested against Get-AdGroupMember with the increased limit.
This script was ran on a remote host and invoke can be removed if running on the FAS server itself.
Load Testing
Now that I had solved my performance issue by increasing threads it posed another couple of questions;
What is the actual impact if the script fails and 50,000 users (100k requests) login on a Monday morning?
Does pre staging certificates really matter that much?
Lab Environment
2 x Citrix FAS Servers on Server 2019 running 2203 with 8GB RAM, 4vCPU 100GB SSD
1 x Microsoft CA Server 2019 with 8GB RAM, 4vCPU 100GB SSD
Lab Environment Runs Nutanix CE on AHV
Load Test 1
Setting the script up with 16 threads where it divides the users into 16 different variables and splits them between each Job. Job was run directly on the FAS server.
User Accounts; 50,000 = 100k requests
Run Rime; 3 hours 17Mins
Average; 8.5 certs a second
Average FAS Server CPU utilisation; 50%
Average CA Server CPU utilisation; 15%
Prism output during runtime;
Load Test 2
Setting the script up with 32 threads and 1 main difference that it was run on a dedicated PowerShell host as it represented how I would be doing it on my project.
User Accounts; 50,000 = 100k requests
Run Rime; 1 hour 4 Mins
Average; 26 certs a second
Average FAS Servers CPU utilisation; 80% with spikes 100%
Average CA Servers CPU utilisation; 44%Average PS Host; 18%
PowerShell Host
Citrix FAS Server 1
Citrix FAS Server 2
Microsoft Certificate Authority
In summary the answers to my questions are;
Not as much as I first thought
No
This is good news as the main reason for testing was based around the question, what’s the impact if the pre generation script fails on a Sunday night? the answer is not much. Its worth being aware that the FAS Certificate is requested when the user logs into Storefront and not when the user launches the desktop so I don’t believe there is actually any performance gain in pre staging certificates even at scale but having said that I still think its a very worth while thing to be doing from a resiliency perspective to protect against CA outages or others that would impact FAS’ ability to request a certificate.
Something else to be aware of is when you are using Storefront and the FAS certificate generation fails then so will the session launch so its worth going that extra mile to protect that from happening and pre generating certificates helps. This is not so much an issue with Workspace Service as the user will just be prompted again for user name and password.
You will also notice on the screenshots the FAS Servers the Software Key Storage Provider is consuming a large amount of resources. If you plan to use a Hardware based Key Storage Provider then be aware you will need to consider the performance impact. I believe this to be more if a consideration if you plan to store user keys there and not only the Registration Authority key.
There are some great posts out there for how to configure FAS and design it, always follow the Citrix Recommended best practices for FAS.
helpful Links;
댓글