Asked ChatGPT and other models to write a simple script
Tags: programming, ai, coding, chatgpt
Published: 2023-06-09 12:43:05

Hello! I continue my journey in the world of deep learning, ai and language models. 

I was once tasked to write a simple PowerShell script, that gets all groups from Azure Active Directory and extracts all the members from the groups. The trick here, is that there might be nested groups! 

For an average system administrator or coder it's a trivial task: just write a recursive function, then get all the groups and process via that function. I was interested in how well our language models can accomplish this basic task. 

For all models I was using default settings and the same prompt. I removed all the comments generated by model and added mine, evaluating the solution. The results are not so obvious I must say! Let me present THE PROMPT:

> Please write a PowerShell script which gets all groups from Azure AD, then iterates over all of the groups and extracts all members. Some of the groups can have nested groups. Members of those should be extracted as well.

Let's now check the outputs of our models: 

## GPT-4

```powershell
Import-Module AzureAD

# That's good, a separate function to get group members recursively is nice
Function Get-GroupMembers ($GroupId) {
    $members = Get-AzureADGroupMember -ObjectId $GroupId | Where-Object {$_.ObjectType -eq "User"} # Correct! ObjectType is the property we are looking for, though no -All $true! Bad!

    foreach($member in $members){
        Write-Host "Member DisplayName: $($member.DisplayName), UserPrincipalName: $($member.UserPrincipalName)"
    }

    $nestedGroups = Get-AzureADGroupMember -ObjectId $GroupId | Where-Object {$_.ObjectType -eq "Group"}

    foreach($nestedGroup in $nestedGroups){
        Write-Host "Nested Group: $($nestedGroup.DisplayName)"
        Get-GroupMembers -GroupId $nestedGroup.ObjectId
    }
}

# Although usually I'm already connected, it's nice that GPT-4 suggested to handle the exception. Yet it could make it better because right now it gives you this which is funny:
# Get-AzureADUser : You must call the Connect-AzureAD cmdlet before calling any other cmdlets.
try {
    if(-not (Get-AzureADUser)) { Connect-AzureAD }
} catch {
    Write-Host "Failed to connect to Azure AD" -ForegroundColor Red
    exit
}

$groups = Get-AzureADGroup # It forgot to add -All $true, without it we get only first 100. Bad!

# Iterate over each group
foreach($group in $groups){
    Write-Host "Group: $($group.DisplayName)"
    Get-GroupMembers -GroupId $group.ObjectId
}
```
Overall, very good! Almost a working solution from the very first attempt. 

Pros: the logic and cmdlets are correct

Cons: it didn't get ALL groups and ALL members although I specified this in the prompt. Also it just printed out the info. It could be nice to have an array which I could turn into CSV later, but that's fine, didn't ask for it in the prompt. Actually, it did tell me after writing the script: This script simply writes the details to the console. If you want to save the details to a file or do something else with them, you will need to modify the script accordingly.


## GPT-3.5

```powershell
# Nice! That's a simplier solution comparing to one provided by gpt-4. At least it works!
Connect-AzureAD

# It specified all! Already better than gpt-4
$groups = Get-AzureADGroup -All $true

# I hate this param stuff in PS but the logic is correct
function Get-NestedGroupMembers {
    param (
        [Parameter(Mandatory=$true)]
        [string]$GroupId
    )

    $nestedGroupMembers = Get-AzureADGroupMember -ObjectId $GroupId # forgot all here

    foreach ($nestedGroupMember in $nestedGroupMembers) {
        if ($nestedGroupMember.ObjectType -eq "Group") {
            Get-NestedGroupMembers -GroupId $nestedGroupMember.ObjectId # yep, that's okay
        } else {
            $nestedGroupMember
        }
    }
}


foreach ($group in $groups) {
    Write-Host "Group Name: $($group.DisplayName)"
    Write-Host "Members:"

    $groupMembers = Get-NestedGroupMembers -GroupId $group.ObjectId

    foreach ($groupMember in $groupMembers) {
        Write-Host "- $($groupMember.UserPrincipalName)"
    }

    Write-Host
}

Disconnect-AzureAD
```
I must say, it's already quite better that gpt-4 for some reason. At least we got ALL groups (but not all members)

Pros: it connects you to azure ad correctly and the logic is correct

Cons: again, not writing to any array and not getting all the members 


## Koala 13B

```powershell
# Wrong from the start. We are in cloud dude, this cmdlet is for on-prem active directory
$groups = Get-ADGroup -Filter *

foreach ($group in $groups) {
    $members = Get-ADGroupMember -Identity $group.Name
    
    foreach ($member in $members) {
        # why do you need to check if the user is external or not?
        if ($member.MembershipType -eq "External") {
            $email = $member.EmailAddress
            
            # I don't understand the logic at all. It creates an array, then add something weird and on line 15, it checks whether the array is empty or not. What?
            $isNested = @()
            $isNested += $member.GroupName -like "$($group.Name)*"
            if ($isNested) {
                $nestedGroups = Get-ADGroupMember -Identity $member.GroupName | Where-Object {$_.MembershipType -eq "External"}
                foreach ($nestedGroup in $nestedGroups) {
                    $email += ",$($nestedGroup.EmailAddress)"
                }
            }
            
            Write-Output "$email"
        }
    }
}
```
Very bad in general. I will not write pros and cons as I really don't understand, what it's trying to do.


## Nous-Hermes 13B

```powershell
$groups = Get-ADGroup -Filter * | Select-Object Name | Sort-Object Name 
    
    foreach ($group in $groups) {
        $members = Get-ADGroupMember -Identity $group.Name -Recursive | Where-Object {($_.ObjectClass -eq 'user')} | Select-Object SamAccountName
        
        Write-Output "$group.Name has the following members: $(($members -join ', '))"
    }
```
Again, Active Directory is used not Azure Active Directory. But I checked, Get-ADGroupMember has a parameter -Recursive, I can't try it out but probably it works. Anyway, not for my needs


## StarChat-beta

```powershell
# good, getting aaaaal the groups
$groups = Get-AzureADGroup -All $true 

foreach ($group in $groups) {
    Write-Host "Processing group: $($group.DisplayName)"

    $members = (Get-AzureADGroupMember -ObjectId $group.ObjectID).Value # I don't know what it does, the Value is empty
    
    foreach($member in $members){
        if(-not $processedMembers.Contains($member)){
            $processedMembers += $member # this variable was never declared nor initialized, so I assume it's just null, not sure

            $isGroupMember = ((Get-AzureADUser -ObjectId $member).memberof | Where-Object{$_ -like "*$($group.displayName)*"} ) # so it assumes that nested group should have a similar name? 
            if($isGroupMember){
                Process-NestedGroups -GroupId $member
            }else{

                $outputMembers += $member
            }
        }
    }
}


function Process-NestedGroups([string]$groupId){
    $nestedGroups = (Get-AzureADGroupMember -ObjectId $groupId ).value 

    foreach ($ngroup in $nestedGroups) {
        if (-not $processedGroups.contains($ngroup)) {
            $processedGroups += $ngroup
            Process-NestedGroups -GroupId $ngroup
        }
    }

    $nestedGroupMembes= get-azureadgroupmember -objectid $ngroup.objectId | select -ExpandProperty id
    $outputMembers+=$nestedGroupMemebrs
}

Write-Output "Total Number of Groups:" $groups.count 
Write-Output "Number of unique members:" ($outputMembers| Measure-Object -Unique).Count # the script fails here, no unique property
write-host "List of Unique Members"
$outputMembers | sort-object -unique # even if you get there, you will get emptiness
```
Overall, the logic seems to be almost there, but the script is overcomplicated and doesn't actually achieve what it's supposed to


## Anthropic Claude

```powershell
# why in the world you forget to add -all
$groups = Get-AzureADGroup

foreach ($group in $groups) {
    $members = Get-AzureADGroupMember -ObjectId $group.ObjectId

    foreach ($member in $members) {
        if ($member.ObjectType -eq "Group") {
            $nestedMembers = Get-AzureADGroupMember -ObjectId $member.ObjectId
            $members += $nestedMembers
        }
    }

    Write-Host "Group: $($group.DisplayName)"
    $members | Format-Table

    $members = $null # why?
}
```

A very neat solution! It worked from the very first run.

Pros: less code, easy-to-read

Cons: not all groups, it didn't write to the variable, just printed to the screen

## Conclusion

First of all, commercial models are way better at this. Secondly, sometimes even smaller and less powerful models can produce a more readable and better code. 

Thank you for reading! If you like this post, please check my other two posts on how to work with LLaMa-based models. See you!