Repository: umbraco/Umbraco-CMS

Issue #10362: SQL timeout when saving document type
Created: 2021-05-31T15:37:23Z
Which Umbraco version are you using?
8.13.0
Bug summary
When saving document types, running usync import or manuallt rebuilding the databse nucache table we get a Sql timeout exception.
Specifics

System.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.

The statement has been terminated. ---> System.ComponentModel.Win32Exception (0x80004005): The wait operation timed out

   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)

   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)

   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)

   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)

   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)

   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)

   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)

   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()

   at Umbraco.Core.Persistence.FaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func) in D:\a\1\s\src\Umbraco.Core\Persistence\FaultHandling\RetryPolicy.cs:line 172

   at NPoco.Database.ExecuteNonQueryHelper(DbCommand cmd)

   at NPoco.Database.Execute(String sql, CommandType commandType, Object[] args)

   at Umbraco.Web.PublishedCache.NuCache.PublishedSnapshotService.RebuildContentDbCacheLocked(IScope scope, Int32 groupSize, IEnumerable`1 contentTypeIds) in D:\a\1\s\src\Umbraco.Web\PublishedCache\NuCache\PublishedSnapshotService.cs:line 1508

   at Umbraco.Web.PublishedCache.NuCache.PublishedSnapshotService.RebuildContentDbCache(Int32 groupSize, IEnumerable`1 contentTypeIds) in D:\a\1\s\src\Umbraco.Web\PublishedCache\NuCache\PublishedSnapshotService.cs:line 1493

   at Umbraco.Web.Editors.NuCacheStatusController.RebuildDbCache() in D:\a\1\s\src\Umbraco.Web\Editors\NuCacheStatusController.cs:line 36

   at lambda_method(Closure , Object , Object[] )

   at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.<>c__DisplayClass6_2.<GetExecutor>b__2(Object instance, Object[] methodParameters)

   at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ExecuteAsync(HttpControllerContext controllerContext, IDictionary`2 arguments, CancellationToken cancellationToken)

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at System.Web.Http.Controllers.ApiControllerActionInvoker.<InvokeActionAsyncCore>d__1.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at System.Web.Http.Filters.ActionFilterAttribute.<ExecuteActionFilterAsyncCore>d__5.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at System.Web.Http.Filters.ActionFilterAttribute.<ExecuteActionFilterAsyncCore>d__5.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at System.Web.Http.Filters.ActionFilterAttribute.<ExecuteActionFilterAsyncCore>d__5.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at System.Web.Http.Filters.ActionFilterAttribute.<ExecuteActionFilterAsyncCore>d__5.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at System.Web.Http.Controllers.ActionFilterResult.<ExecuteAsync>d__5.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at System.Web.Http.Filters.AuthorizationFilterAttribute.<ExecuteAuthorizationFilterAsyncCore>d__3.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at System.Web.Http.Filters.AuthorizationFilterAttribute.<ExecuteAuthorizationFilterAsyncCore>d__3.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at System.Web.Http.Filters.AuthorizationFilterAttribute.<ExecuteAuthorizationFilterAsyncCore>d__3.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at System.Web.Http.Controllers.ExceptionFilterResult.<ExecuteAsync>d__6.MoveNext()

ClientConnectionId:96f0230e-0d10-49c0-bbaa-1b16d4e70807

Error Number:-2,State:0,Class:11

ClientConnectionId before routing:2ecd8493-60f2-472b-89bb-e76ab2625dbb

Routing Destination:cd9f5bc88fe6.HS2.tr8110.westeurope1-a.worker.database.windows.net,11019

We run on Azure SQL with a S2 database, but scaling to a S3 database has not worked either.
Steps to reproduce
Possibly reproducible by having a document type using multiple compositions and having a lot of content (2k+ pages)
Expected result / actual result
No response

Duplicate issue #11023: Process hangs when updating "Allowed child node types"
Created: 2021-09-03T03:16:18Z
Which exact Umbraco version are you using? For example: 8.13.1 - don't just write v8
9.0.0-rc002
Bug summary
The application hangs (does not crash, but spins indefinitely and becomes unresponsive) if "Allowed child node types" is updated for a Document Type that allows itself as a child node and this relationship is used by existing content in the tree.
Specifics
No response
Steps to reproduce

Create a Document Type with Template and allow creation at root
Add itself under "Allowed child node types" and save
Create a second Document Type with Template
Create a Content item of the first type at the root, then create another content item of the same type under it as a child
Attempt to add the second Document type to the first type's "Allowed child node types" and save OR attempt to remove the first type from itself in "Allowed child node types"
Clicking save hangs the process

Only after deleting the child node and emptying the recycle bin can the recursive Document Type's "Allowed child node types" property be modified and saved.
Expected result / actual result
I expected to either get a validation error or have the Document Type save successfully. Instead, the process hangs with the spinner replacing the Save button remaining indefinitely. The app must be restarted to recover.

Closing comment (2021-09-04T13:48:20Z): Thanks for taking a look, Sebastian. I'm able to reproduce this in most cases using the same steps shown in your video, but it is proving to be inconsistent and I haven't been able to determine what makes it happen sometimes and not others.
As I've continued development, I find saving document types hangs up in other situations as well when there is no recursive allowance. So it seems likely there are multiple different things going on here. Sometimes I get a SQL timeout, which is probably the same issue reported in #10362. Here's an example of the log entry for that scenario:
{"@t":"2021-09-04T01:13:58.9382775Z","@mt":"Exception ({InstanceId}).","@l":"Error","@x":"System.Data.SqlClient.SqlException (0x80131904): Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.\r\n ---> System.ComponentModel.Win32Exception (258): The wait operation timed out.\r\n   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action'1 wrapCloseInAction)\r\n   at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action'1 wrapCloseInAction)\r\n   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)\r\n   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)\r\n   at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()\r\n   at System.Data.SqlClient.SqlDataReader.get_MetaData()\r\n   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)\r\n   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds)\r\n   at System.Data.SqlClient.SqlCommand.ExecuteScalar()\r\n   at Umbraco.Cms.Infrastructure.Persistence.FaultHandling.RetryPolicy.ExecuteAction[TResult](Func'1 func)\r\n   at NPoco.Database.ExecuteScalarHelper(DbCommand cmd)\r\n   at NPoco.Database.ExecuteScalar[T](String sql, CommandType commandType, Object[] args)\r\nClientConnectionId:91a9ef42-3c36-4440-ae5c-24f9cc9e9e3c\r\nError Number:-2,State:0,Class:11","InstanceId":"4f54826c","SourceContext":"Umbraco.Cms.Infrastructure.Persistence.UmbracoDatabase","ActionId":"1ea2a5a6-ae65-4247-9f11-17e916a607f8","ActionName":"Umbraco.Cms.Web.BackOffice.Controllers.ContentTypeController.AllowsCultureVariation (Umbraco.Web.BackOffice)","RequestId":"80000045-0002-fe00-b63f-84710c7967bb","RequestPath":"/umbraco/backoffice/umbracoapi/contenttype/AllowsCultureVariation","ProcessId":18608,"ProcessName":"iisexpress","ThreadId":261,"AppDomainId":1,"AppDomainAppId":"7c5b9bae3645fe004b4e13efb0d8d8e0181edb1e","MachineName":"BODHI-WINDOWS","Log4NetLevel":"ERROR","HttpRequestId":"8895c775-da53-434a-a2be-233deb31356f","HttpRequestNumber":3,"HttpSessionId":"432d7fff-bd34-d419-0b2b-913ffc6d7ad4"}
And sometimes I get a routing error, which is odd because why would my front-end controller be getting invoked at all when saving a Document Type on the back end (and this controller works fine on the front end).
{"@t":"2021-09-02T02:29:33.4217431Z","@mt":"An unhandled exception has occurred while executing the request.","@l":"Error","@x":"Microsoft.AspNetCore.Routing.Matching.AmbiguousMatchException: The request matched multiple endpoints. Matches: \r\n\r\nMemoryUmbraco9.Controllers.ContentPageController.Index (MemoryUmbraco9)\r\nMemoryUmbraco9.Controllers.ContentPageController.Index (MemoryUmbraco9)\r\n   at Microsoft.AspNetCore.Routing.Matching.DefaultEndpointSelector.ReportAmbiguity(CandidateState[] candidateState)\r\n   at Microsoft.AspNetCore.Routing.Matching.DefaultEndpointSelector.ProcessFinalCandidates(HttpContext httpContext, CandidateState[] candidateState)\r\n   at Microsoft.AspNetCore.Routing.Matching.DefaultEndpointSelector.Select(HttpContext httpContext, CandidateState[] candidateState)\r\n   at Microsoft.AspNetCore.Routing.Matching.DefaultEndpointSelector.SelectAsync(HttpContext httpContext, CandidateSet candidateSet)\r\n   at Microsoft.AspNetCore.Routing.Matching.DfaMatcher.SelectEndpointWithPoliciesAsync(HttpContext httpContext, IEndpointSelectorPolicy[] policies, CandidateSet candidateSet)\r\n   at Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware.<Invoke>g__AwaitMatch|8_1(EndpointRoutingMiddleware middleware, HttpContext httpContext, Task matchTask)\r\n   at SixLabors.ImageSharp.Web.Middleware.ImageSharpMiddleware.Invoke(HttpContext context)\r\n   at Microsoft.AspNetCore.Diagnostics.StatusCodePagesMiddleware.Invoke(HttpContext context)\r\n   at StackExchange.Profiling.MiniProfilerMiddleware.Invoke(HttpContext context) in C:\\projects\\dotnet\\src\\MiniProfiler.AspNetCore\\MiniProfilerMiddleware.cs:line 119\r\n   at Umbraco.Cms.Web.Common.Middleware.UmbracoRequestMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)\r\n   at Umbraco.Cms.Web.Common.Middleware.UmbracoRequestMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)\r\n   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass6_1.<<UseMiddlewareInterface>b__1>d.MoveNext()\r\n--- End of stack trace from previous location ---\r\n   at Umbraco.Cms.Web.Common.Middleware.PreviewAuthenticationMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)\r\n   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass6_1.<<UseMiddlewareInterface>b__1>d.MoveNext()\r\n--- End of stack trace from previous location ---\r\n   at Umbraco.Cms.Web.Common.Middleware.UmbracoRequestLoggingMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)\r\n   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass6_1.<<UseMiddlewareInterface>b__1>d.MoveNext()\r\n--- End of stack trace from previous location ---\r\n   at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)","EventId":{"Id":1,"Name":"UnhandledException"},"SourceContext":"Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware","RequestId":"800000ab-0001-fd00-b63f-84710c7967bb","RequestPath":"/","ProcessId":10728,"ProcessName":"iisexpress","ThreadId":9,"AppDomainId":1,"AppDomainAppId":"d4f2f2734ea1c1fb088b9ea84437b7045f08ac11","MachineName":"BODHI-WINDOWS","Log4NetLevel":"ERROR"}
And in both of these cases, the user experience differs from time to time. Sometimes I get an error displayed and sometimes it just spins for a very very long time (several minutes) before finally succeeding. Sorry this is a bit all over the place. I'll continue to observe the behavior and if I can pin it down a bit more I'll update here.