Runspaces Simplified (as much as possible)

Last year, I was looking into multi-threading in PowerShell and, with the help of Dr. Tobias Weltner, Boe Prox and Martin Pugh ultimately decided on runspaces.

Then, I presented at psconf.eu about using runspaces to speed up SQL Server/CSV imports. Runspaces took me from about 90,000 rows per second to 230,000 rows per second on average.

rps

Runspaces can be intimidating. I had heard about them, took a look at the code and was like “Ah, that looks complex. I’ll learn that later.” Because of this, I wanted to ease the audience into runspaces and repeatedly went over the bulk insert code to familiarize the audience with the functionality that I was eventually going to multi-thread.

It seems like that approach worked. The audience wasn’t overwhelmed (or didn’t admit to it ;)) — mission accomplished!

All of the code and the PowerPoint seen in the video can be downloaded in my directory on PSConfEU’s GitHub Repository.

Runspace Template

Ultimately, when I want to add a runspace to a command, I download code.zip from my presentation and copy/paste/edit 6-runspace-concept.ps1. When I add multi-threading to my scripts, I just copy and paste the code below, then modify. Understanding everything that’s going on isn’t immediately necessary.

If you’re using Runspaces as an end-user, try Boe Prox’s PSJobs module instead. But if you don’t want dependencies, you can paste the code below.

So that’s basically it. Adding multithreading honestly just requires a bunch of copy/pasting. I generally modify Steps 2 and 3.

Step-by-Step

  • BLOCK 1: Create and open a runspace pool. You don’t have to, but it increases performance so I always just leave it in. CreateRunspacePool() accepts min and max runspaces.

    I’ve found that 1 and number of processors+1 (thanks to Steffan for informing me why 5 was always my quad processor’s sweet spot.) Then I play with MTA and STA and see which one works better.

    I also create a runspaces array to keep track of the runspaces. Unfortunately, you can’t do like $pool.runspaces to get the collection, so you have to make your own.

  • BLOCK 2: Create reusable scriptblock. This is the workhorse of the runspace. Think of it as a function.

  • BLOCK 3: Create the runspace and add to runspace pool.

    If you write a lot of PowerShell functions, it should be apparent what it’s doing. Basically AddScript($scriptblock) is the function name, then AddArgument($connstring), .AddArgument($datatable) and AddArgument($batchsize) are the parameters.

    Note that you may find yourself passing a lot of parameters because the runspace will be mostly unaware of the variables that exist outside of the $scriptblock.

  • BLOCK 4: Add runspace to runspaces collection and start it

  • BLOCK 5: Wait for runspaces to finish

  • BLOCK 6: Clean up

  • BLOCK 7: Look at $results to see any errors or any other return

Up next

Now there are a couple Runspace commands in v5 but there’s no New-Runspace so this code still applies to v5. I generally code for v3 so this script won’t be changing much in the near future.

In the next blog post, I’ll detail a slightly different runspace and immediately outputs the results of the runspace to the pipeline.

Edit: Actually, I just finished that blog post and decided to paste the code here. It’s good for repetition and shows another simplified runspace, this time without comments. If you’d like more info on this runspace, check out the post.

Chrissy is a Cloud and Datacenter Management & Data Platform MVP who has worked in IT for over 20 years. She is the creator of the popular SQL PowerShell module dbatools, and holds a number of certifications, including those relating to SQL Server, Linux, SharePoint and network security. You can follow her on Twitter at @cl.

Posted in PowerShell
8 comments on “Runspaces Simplified (as much as possible)
  1. Daniel says:

    Great article! Runspaces are really amazing :)

    But I’m afraid I don’t understand why it has to be $env:NUMBER_OF_PROCESSORS+1, could you please explain this?
    I’ve re-read the article and watched your presentation but I don’t see the reason of the +1

    Thanks!

    Daniel.

    • Anon says:

      $pool = [RunspaceFactory]::CreateRunspacePool(1, [int]$env:NUMBER_OF_PROCESSORS + 1)

      You need to set the “Minimum” and “Maximum” threads, so here1 is the minimum. The maximum threads is “$env:NUMBER_OF_PROCESSORS” , which is the number of processors the current PC has plus 1. You could add plus whatever but to be safe +1 will not hurt anything.

  2. Sreejith says:

    Hi, Runspaces are great! but I have a question, how do I check if there are any errors in the result?

  3. Rich Harris says:

    Chrissy, I read your article and was wondering if you could try and help me figure out why I keep getting a System.OutOfMemory exception thrown when I try and run an import process for an excel file ? I know this is an open ended question but due to the amount of stuff that’s going on I did not want to spend much time writing this up if you would not have the time to go through it. I can email you with my code and all the steps that I’m trying to do if you are willing.
    Thanks

    • Chrissy LeMaire says:

      I can see why that would happen, for sure. What you’d want to do is add [System.GC]::Collect() in that $scriptblock. You can watch it go down every now and then in Task Manager. It slows it down a bit, but so far, we all assume it’s a memory leak in PS.

      • Rich Harris says:

        I have an SSIS script task that is running a powershell script on a remote server on a different domain and over time I get an out of memory exception error thrown. Here is the script task code with my domain user and PW changed, do you see anything wrong with the way that I went about doing this ??

        public void Main()
        {
        string URL = Dts.Variables[“User::URL”].Value.ToString();
        string ServerName = Dts.Variables[“User::ServerName”].Value.ToString();
        //string ServerName = “QAWEB1”;
        string SourceFile = Dts.Variables[“User::SourceFile”].Value.ToString();
        string ArchiveFile = Dts.Variables[“User::ArchiveFile”].Value.ToString();
        //int IsCrossDomain = int.Parse(Dts.Variables[“User::IsCrossDomain”].Value.ToString());
        string DomainName = Dts.Variables[“User::DomainName”].Value.ToString();
        string DomainUserName = Dts.Variables[“User::DomainUserName”].Value.ToString();
        string DomainPwd = Dts.Variables[“User::DomainPwd”].Value.ToString();

        try
        {

        string shellUri = “http://schemas.microsoft.com/powershell/Microsoft.PowerShell”;
        PSCredential remoteCredential = new PSCredential(“myDomain\\MyUser”, ToSecureString(“MyPW”));
        WSManConnectionInfo connectionInfo = new WSManConnectionInfo(false, ServerName, 5985, “/wsman”, shellUri, remoteCredential);
        //connectionInfo.AuthenticationMechanism = AuthenticationMechanism.Negotiate;
        connectionInfo.AuthenticationMechanism = AuthenticationMechanism.Credssp;
        using (Runspace runspace = RunspaceFactory.CreateRunspace(connectionInfo))
        {
        runspace.Open();
        using (var pipeLine = runspace.CreatePipeline())
        {
        // You can write as much lines of script as yo want by using pipeline.commands.AddScript
        string PSScript = “If(!(Get-Module RLF_Custom_CMDLets)){ “;
        PSScript += “Import-Module ‘C:\\Program Files\\WindowsPowerShell\\Modules\\RLF\\RLF_Custom_CMDLets.ps1’} “;
        PSScript += “RLF-ImportExternalCertificates \”” + SourceFile.Replace(“\\\\” + ServerName, “C:”) + “\” \”” + URL + “\””;
        pipeLine.Commands.AddScript(PSScript);
        try
        {
        var results = pipeLine.Invoke();
        if(pipeLine.Error.Count > 0)
        {
        var err = pipeLine.Error.Read();
        if(err!=null)
        {
        errMessage = err.ToString();
        Dts.Events.FireError(0, “Error Importing External Certificates for ” + SourceFile + “. “, errMessage, String.Empty, 0);
        Dts.TaskResult = (int)ScriptResults.Failure;
        }
        }

        }
        catch (Exception ex)
        {
        Dts.Events.FireError(0, “Error Importing External Certificates for ” + SourceFile + “. “, ex.Message, String.Empty, 0);
        Dts.TaskResult = (int)ScriptResults.Failure;
        }
        }
        runspace.Close();
        runspace.Dispose();
        }
        }
        catch (Exception ex)
        {
        Dts.Events.FireError(0, “Error Importing External Certificates for ” + SourceFile + “. “, ex.Message, String.Empty, 0);
        Dts.TaskResult = (int)ScriptResults.Failure;
        }
        Dts.Variables.Unlock();
        bool fireAgain = true;
        Dts.Events.FireInformation(0, “Script Task Code”,
        “This is the script task raising an event.”, null, 0, ref fireAgain);
        Dts.TaskResult = (int)ScriptResults.Success;

        }

Leave a Reply

Your email address will not be published. Required fields are marked *

*