Skip to content

[BUG]: Random failures during CopyFilesV2, DeleteFilesV1, ArchiveFilesV2 tasks #21433

@dan-engler-arcadis

Description

@dan-engler-arcadis

New issue checklist

Task name

CopyFiles@V2

Task version

2.256.0

Issue Description

Summary

When performing a deployment onto Windows Server environments running a self-hosted Azure Pipeline Agent, the CopyFilesV2, DeleteFilesV1, and ArchiveFilesV2 tasks are failing on random files with the message "Exit code 1 returned from process". Pipeline tasks will have successfully copied/deleted/archived 10 or 100 or 1000 files in the same directory when the errors occur, and there is no consistency to which file or which task causes the failure.

Observations

  • The same pipelines have been used for more than two years, and worked as expected when they were last used in January 2025
  • Failures began immediately upon trying to use the pipelines in September 2025
  • Failures have occurred on multiple servers running multiple versions of Windows Server
  • We have verified that none of the files in the impacted directories are locked by running applications or competing filesystem operations; the files that triggered the error can be copied/deleted immediately after the failure
  • There are many gigabytes of available space on the system drive
  • There is plenty of CPU and memory overhead on the server
  • The servers are fully patched with the latest updates for Windows Server

Attempted Solutions

Errors persist despite taking the following steps:

  • Completely disabling third-party security software running on the server
  • Running "Update targets" from the DevOps console to ensure the latest stable build of the agent is installed
  • De-registering the agent using .\config remove, deleting all existing files from the agent install directory, and running the registration script to ensure a clean install

Environment type (Please select at least one enviroment where you face this issue)

  • Self-Hosted
  • Microsoft Hosted
  • VMSS Pool
  • Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

Windows Server 2022, Windows Server 2016 Datacenter

Relevant log output

2025-11-04T01:01:53.9518748Z ##[error]Exit code 1 returned from process: file name 'C:\azagent\A1\externals\node20_1\bin\node.exe', arguments '"C:\azagent\A1\_work\_tasks\CopyFiles_5bfb729a-a7c8-4a78-a7c3-8d717bb7c13c\2.256.0\copyfiles.js"'.
2025-11-04T01:01:53.9535705Z ##[debug]Microsoft.VisualStudio.Services.Agent.Util.ProcessExitCodeException: Exit code 1 returned from process: file name 'C:\azagent\A1\externals\node20_1\bin\node.exe', arguments '"C:\azagent\A1\_work\_tasks\CopyFiles_5bfb729a-a7c8-4a78-a7c3-8d717bb7c13c\2.256.0\copyfiles.js"'.
   at Microsoft.VisualStudio.Services.Agent.Util.ProcessInvoker.ExecuteAsync(String workingDirectory, String fileName, String arguments, IDictionary`2 environment, Boolean requireExitCodeZero, Encoding outputEncoding, Boolean killProcessOnCancel, InputQueue`1 redirectStandardIn, Boolean inheritConsoleHandler, Boolean keepStandardInOpen, Boolean highPriorityProcess, Boolean continueAfterCancelProcessTreeKillAttempt, CancellationToken cancellationToken) in D:\a\_work\1\s\src\Agent.Sdk\ProcessInvoker.cs:line 389
   at Microsoft.VisualStudio.Services.Agent.ProcessInvokerWrapper.ExecuteAsync(String workingDirectory, String fileName, String arguments, IDictionary`2 environment, Boolean requireExitCodeZero, Encoding outputEncoding, Boolean killProcessOnCancel, InputQueue`1 redirectStandardIn, Boolean inheritConsoleHandler, Boolean keepStandardInOpen, Boolean highPriorityProcess, Boolean continueAfterCancelProcessTreeKillAttempt, CancellationToken cancellationToken) in D:\a\_work\1\s\src\Microsoft.VisualStudio.Services.Agent\ProcessInvoker.cs:line 329
   at Microsoft.VisualStudio.Services.Agent.Worker.Handlers.DefaultStepHost.ExecuteAsync(String workingDirectory, String fileName, String arguments, IDictionary`2 environment, Boolean requireExitCodeZero, Encoding outputEncoding, Boolean killProcessOnCancel, Boolean inheritConsoleHandler, Boolean continueAfterCancelProcessTreeKillAttempt, TimeSpan sigintTimeout, TimeSpan sigtermTimeout, Boolean useGracefulShutdown, CancellationToken cancellationToken) in D:\a\_work\1\s\src\Agent.Worker\Handlers\StepHost.cs:line 85
   at Microsoft.VisualStudio.Services.Agent.Worker.Handlers.NodeHandler.RunAsync() in D:\a\_work\1\s\src\Agent.Worker\Handlers\NodeHandler.cs:line 284
   at Microsoft.VisualStudio.Services.Agent.Worker.TaskRunner.RunAsyncInternal() in D:\a\_work\1\s\src\Agent.Worker\TaskRunner.cs:line 474
   at Microsoft.VisualStudio.Services.Agent.Worker.TaskRunner.RunAsync() in D:\a\_work\1\s\src\Agent.Worker\TaskRunner.cs:line 79
   at Microsoft.VisualStudio.Services.Agent.Worker.StepsRunner.RunStepAsync(IStep step, CancellationToken jobCancellationToken) in D:\a\_work\1\s\src\Agent.Worker\StepsRunner.cs:line 299

Full task logs with system.debug enabled

tasklog_6.log

Repro steps

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions