Analysis of Administrator Propaganda: Story of a web service troubleshoot, without touching the code.
Tags: criticalsection, authentication, iis
Published: 2020-02-07 09:09:44

Hooray! It's Friday (Weekend in bloody middle east) and time to write a blog post. And wish you are good.

# Problem

In a troubleshoot problem we had a cute experience yesterday. And It was about '403 Forbidden Error' from company's server to a third party business's API.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/p14wwkeqvpmi4br667s7.png)

Everything was fine until this week. And no code was changed, nothing in network was moved, no new firewall policy was applied, and everything was totally same as previous weeks.

But *Server B* (which requests last image from *Server A*), was accepting 'HTTP Error 403' and everyone was shocked. Out **IIS** logs also confirmed that it was our side.

We just reviewed *App 1* and *App 2* codes. The idea behind *App 1* is to download images from cameras, using `ffmpeg`, convert them and keep a copy of that for archive and store last image in a directory, with a specific name, so *App 2* (Which is a `asmx` web service also), can wait for a request from *Server B* to send image byte array as response.

# Troubleshoot steps:

1- Double-checked network architecture and gained permission to review network engineers reports: nothing.

2- Double-checked `git` changes: nothing.

3- Double-checked the *SOAP API* changes with third-party business: No changes was applied, and We were the only contractor who had this error.

4- Reviewed the code, database was OK, and only thing which is not under our control is **File System** who is the accused 🔍🎩 !

5- Reviewed the `GetFiles()` function [documentation](https://docs.microsoft.com/en-us/dotnet/api/system.io.directory.getfiles?view=netframework-4.8) from Microsoft website. Took a look in **Exceptions** section and Voilà!

> **UnauthorizedAccessException**
> The caller does not have the required permission.

6- Definitely it was `GetFiles()` but why? Maybe when `ffmpeg` was downloading and converting image (*APP 1*), *App 2* was requesting the file because of *critical section* we web service was broken. That was also wrong because `ffmpeg`'s has a different behavior that creates the file at the end, and instantly release it.

7- Key question: **Which user is running App 1?**: Administrator. And **which user is running App 2?**: IIS_IUSRS! that's it!

8- **App 1** is triggered from *Windows Scheduler* every one minute to take a snapshot, and we just simply set user as *IIS_IUSRS* in scheduler tasks from `security options`>`Change User or Group`.

# Conclusion.

- Not always it's because of code. Don't blame developers!

- Maybe IIS has a list in cache which when a file creates in it's website directory, that list keeps permission details about that file, so when `ffmpeg` creates the file and then updates the permissions after logic end, IIS does not update the file details instantly after modification. And this will cause exceptions in higher layers, especially when folder gets heavier in time.

- Use `powershell` more. All those stuff could be simply handled using `.bat` and basic Windows commands. Complexity makes troubleshooting difficult.

- Comment any code, any where, even if it is as simple as a `foreach thing in GetFiles()`

- When it comes to a code which is going to run million times a day, check each obvious line of code again, and Copy-Paste potential errors from documentation to code comments with link and details.

Hope you will let me know if there is any comments. Thank you!