Repository: pytorch
Issue: Integrate Libtorch into Unreal Engine 4: _ivalue_INTERNAL ASSERT FAILED
Labels: oncall: jit,Stale

## 🐛 Bug

I'm trying to integrate Libtorch into Unreal Engine 4 using dynamic library DLLs. However, I get runtime errors, the class c10::Error catch the next error when my model's function: at::Tensor output = module.forward(inputs).toTensor(); is executed. Te error:

_ivalue_INTERNAL ASSERT FAILED at "..\\..\\torch\\csrc\\jit\\api\\object.cpp":19, please report a bug to PyTorch. 
Exception raised from _ivalue at ..\..\torch\csrc\jit\api\object.cpp:19 (most recent call first):
00007FFBD73CA29200007FFBD73CA230 c10.dll!c10::Error::Error [<unknown file> @ <unknown line number>]
00007FFBD73C9CAE00007FFBD73C9C60 c10.dll!c10::detail::torchCheckFail [<unknown file> @ <unknown line number>]
00007FFBD73A79B900007FFBD73A79B0 c10.dll!c10::detail::torchInternalAssertFail [<unknown file> @ <unknown line number>]
00007FFBD1D6112600007FFBD1D610F0 torch_cpu.dll!torch::jit::Object::_ivalue [<unknown file> @ <unknown line number>]
00007FFBD16EEA9C00007FFBD16EEA80 torch_cpu.dll!torch::jit::Object::type [<unknown file> @ <unknown line number>]
00007FFBD1D6175E00007FFBD1D61730 torch_cpu.dll!torch::jit::Object::find_method [<unknown file> @ <unknown line number>]
00007FFBD1D5BA2500007FFBD1D5B9B0 torch_cpu.dll!torch::jit::Module::forward [<unknown file> @ <unknown line number>]
00007FFBCA294FEF00007FFBCA294DD0 LibTorchUE4.dll!torch_ue4_forward [D:\PyTorch\LibTorchUE4\LibTorchUE4\LibTorchUE4.cpp @ 28]
00007FFBCA2559D900007FFBCA251EA0 UE4Editor-MyProject.dll!InitializeModule [<unknown file> @ <unknown line number>]
00007FFBCA2543F200007FFBCA251EA0 UE4Editor-MyProject.dll!InitializeModule [<unknown file> @ <unknown line number>]
00007FFBCA2581B200007FFBCA251EA0 UE4Editor-MyProject.dll!InitializeModule [<unknown file> @ <unknown line number>]
00007FFBF34F898B00007FFBF34F8940 UE4Editor-Core.dll!FRunnableThreadWin::Run [D:\Build\++UE4\Sync\Engine\Source\Runtime\Core\Private\Windows\WindowsRunnableThread.cpp @ 86]
00007FFBF34F174700007FFBF34F1590 UE4Editor-Core.dll!FRunnableThreadWin::GuardedRun [D:\Build\++UE4\Sync\Engine\Source\Runtime\Core\Private\Windows\WindowsRunnableThread.cpp @ 27]
00007FFC70E6703400007FFC70E67020 KERNEL32.DLL!BaseThreadInitThunk [<unknown file> @ <unknown line number>]
00007FFC7162265100007FFC71622630 ntdll.dll!RtlUserThreadStart [<unknown file> @ <unknown line number>]

**In order to reproduce the error**:

1. Run the this code in pthyton for save the model (is the basic example of libtorch):

```
import torch
import torchvision

# An instance of your model.
model = torchvision.models.resnet18()

# An example input you would normally provide to your model's forward() method.
example = torch.rand(1, 3, 224, 224)

# Use torch.jit.trace to generate a torch.jit.ScriptModule via tracing.
traced_script_module = torch.jit.trace(model, example)

traced_script_module.save("D:\\PyTorch\\Model\\resnet18.pt")

print('model created')
```
2. Download Libtorch: Stable (1.10), Windows, LibTorch, C++/Java and CPU (release mode), and Extract the zip in your PATH (D:\PyTorch\CPU\libtorch-win-shared-with-deps-1.10.0+cpu\libtorch)

3. Create a project in Visual Studio 2019 named LibTorchUE4 of type Dynamic-Link Library (DLL) with the configuration Release and x64. In the properties Project -> LibTorchUE4 indicates the following:

C/C++ -> General -> Additional Include Directories:
D:\PyTorch\CPU\libtorch-win-shared-with-deps-1.10.0+cpu\libtorch\include\torch\csrc\api\include
D:\PyTorch\CPU\libtorch-win-shared-with-deps-1.10.0+cpu\libtorch\include

Linker -> General -> Additional Library Directories:
D:\PyTorch\CPU\libtorch-win-shared-with-deps-1.10.0+cpu\libtorch\lib

Linker -> Input-> Additional Dependencies:
torch_cpu.lib;c10.lib;torch.lib;

4. Add to project the files LibTorchClass.h and LibTorchUE4.cpp.

The code for LibTorchClass.h:

```
#pragma once

extern "C" __declspec(dllexport) void torch_ue4_load();
extern "C" __declspec(dllexport) void torch_ue4_forward(char* ptrStr);
```

The code for LibTorchUE4.cpp:

```
#include "pch.h"
#include <iostream>
#include <torch/script.h>

#include "LibTorchUE4.h"

torch::jit::script::Module module;
std::vector<torch::jit::IValue> inputs;

void torch_ue4_load() {
	torch::device(torch::kCPU);
	try {
		torch::jit::script::Module module = torch::jit::load("D:\\PyTorch\\Model\\resnet18.pt");
	}
	catch (const c10::Error& e) {
		std::cout << e.what() << std::endl;
	}
}

void torch_ue4_forward(char* ptrStr) 
{
	inputs.push_back(torch::ones({ 1, 3, 224, 224 }));
	try {
		// Execute the model and turn its output into a tensor.
		at::Tensor output = module.forward(inputs).toTensor();
		strcpy(ptrStr, "OK");
		//std::cout << output.slice(/*dim=*/1, /*start=*/0, /*end=*/5) << '\n';
	}
	catch (const c10::Error& e) {
		strcpy(ptrStr, e.what());
	}
}

```

The code for pch.cpp:

```
#ifndef PCH_H
#define PCH_H
#define NOMINMAX
#include <algorithm>
namespace Gdiplus
{
	using std::min;
	using std::max;
}
// add headers that you want to pre-compile here
#include "framework.h"
#endif //PCH_H
```

5. Build the DLL with Build -> Buil Solution. You created the files LibTorchUE4.dll and LibTorchUE4.lib in the path of your project (D:\PyTorch\LibTorchUE4\x64\Release).

6. Create a project in Unreal Engine 4 named MyProject of type ThirdPerson and of type C++ (the default project is in mode Development Editor).

7. In Visual Studio 2019 open the file MyProject.Build.cs and add the lines of code shown in this code:

```
using UnrealBuildTool;

public class MyProject : ModuleRules
{
	public MyProject(ReadOnlyTargetRules Target) : base(Target)
	{
		PCHUsage = PCHUsageMode.UseExplicitOrSharedPCHs;

		PublicDependencyModuleNames.AddRange(new string[] { "Core", "CoreUObject", "Engine", "InputCore", "HeadMountedDisplay" });

		//bUseRTTI = true;
		bEnableExceptions = true;
		bIgnoreUnresolvedSymbols = true;

		PrivateIncludePaths.Add("D:\\PyTorch\\LibTorchUE4\\LibTorchUE4");
                PublicAdditionalLibraries.Add("D:\\PyTorch\\LibTorchUE4\\x64\\Release\\LibTorchUE4.lib");
	}
}
```

8. In the file MyProjectCharacter.h add the following lines of code #include "LibTorchUE4.h" and virtual void Tick(float DeltaSeconds) override;.

```
#include "LibTorchUE4.h"

// Copyright Epic Games, Inc. All Rights Reserved.
#pragma once

#include "CoreMinimal.h"
#include "GameFramework/Character.h"
#include "MyProjectCharacter.generated.h"

UCLASS(config=Game)
class AMyProjectCharacter : public ACharacter
{
	GENERATED_BODY()
...
public:
	...
	virtual void Tick(float DeltaSeconds) override;
...
};
```
9. In the file MyProjectCharacter.cpp add the following lines of code:

```
// Copyright Epic Games, Inc. All Rights Reserved.
#include "MyProjectCharacter.h"
...
#include "GameFramework/SpringArmComponent.h"

int count_p = 0;
char strResult[5000];

void printString(const FString& message, float timeToDisplay)
{
	GEngine->AddOnScreenDebugMessage(-1, timeToDisplay, FColor::Cyan, message);
}
//////////////////////////////////////////////////////////////////////////
// AMyProjectCharacter

AMyProjectCharacter::AMyProjectCharacter()
{
...
	torch_ue4_load();
}

void AMyProjectCharacter::Tick(float DeltaSeconds)
{
	Super::Tick(DeltaSeconds);

	if (count_p > 300) {
		auto Result = Async(EAsyncExecution::Thread, []() {
			torch_ue4_forward(strResult);
			printString(strResult, 60);
		
                	}//Fin de la llaves Async
		);//Fin de los parentesis de Async
		count_p = 0;
	}
	count_p++;
}

```
10. Paste your dynamic library LibTorchUE4.dll in the path of your UE4 project (D:\Unreal Projects\MyProject\Binaries\Win64). 

11. In the same way, paste your dynamics librarys: asmjit.dll, c10.dll. fbgemm.dll, libiomp5md.dll, torch.dll, torch_cpu.dll and uv.dll, in the path of your UE4 project.

12. Buil your project Build -> Buil solution (Default configuration: Developer Editor and Win64)

13. When your Editor is open, push the button of Play. You see in the screen the error message. The error occurs int the line: at::Tensor output = module.forward(inputs).toTensor(); and c10::Error catch the exception.

**Environment**:

Windows 10
Visual Studio 2019
Unreal Engine 4 (4.26.2)
Libtorch 1.10 versión CPU release mode