Network Automation Software We are currently migrating this forum
over to our HelpSystems domain. Please
post all new threads in our new
HelpSystems Community Portal.
Post to the HelpSystems Forum
You are not currently logged on. You must be logged on in order to post. Log on
Or Create a new account
AutoMate Discussion
Decrease font size
Increase font size
Topic Title: Hung tasks pile up
Topic Summary:
Created On: 06/14/2010 08:40 AM
Status: Read Only
Linear : Threading : Single : Branch
Search Topic Search Topic
Topic Tools Topic Tools
View similar topics View similar topics
View topic in raw text format. Print this topic.
 06/14/2010 08:40 AM
User is offline View Users Profile Print this message

Author Icon
JoelWilliams
Apprentice (20-49)

Posts: 25
Joined: 08/04/2005

I am running 7.1.3.0 on Windows 7 x86. I was running 7.1.1.0 without issue. After upgrading to 7.1.2.0 I began to have issues with my tasks hanging. The task that hangs is random and changes; but the behavior is always the same. When a task hangs all the other tasks behind it queue up until the task list has 50 entries. If I try to view the event monitor, I receive an error like the one attached. Tasks will run fine for a while, but at various times throughout the day, they will begin to queue up again. I have used MS Process Explorer to kill the latest AM7TASK.exe and afterwards the other 49 queued up tasks will "complete". They are marked as running successfully, but don't actually perform the intended commands. I have installed a fresh copy of Automate on a different Win 7 x86 machine (now running 7.1.3.0) and I'm still experiencing the same issues. Any ideas?
Thanks.

Update: The additional attachment shows one of the jobs hanging on Step 4. This step is the last step in that particular task and is just an End If. The other tasks in the event monitor just piled up. Upon ending the hung task, the other tasks ended, were listed as successful, but never performed the intended jobs. I believe, in all cases, the hung job is hanging on an End If.

 Category Survey


Click for full image
Click for larger image
Click for full image
Click for larger image



Edited: 06/14/2010 at 11:13 AM by JoelWilliams
 06/14/2010 05:00 PM
User is offline View Users Profile Print this message

Author Icon
Rick Johnson
AUTOMATE EMPLOYEE

Posts: 2757
Joined: 07/14/2008

Originally posted by: JoelWilliams

I am running 7.1.3.0 on Windows 7 x86. I was running 7.1.1.0 without issue. After upgrading to 7.1.2.0 I began to have issues with my tasks hanging. The task that hangs is random and changes; but the behavior is always the same. When a task hangs all the other tasks behind it queue up until the task list has 50 entries. If I try to view the event monitor, I receive an error like the one attached. Tasks will run fine for a while, but at various times throughout the day, they will begin to queue up again. I have used MS Process Explorer to kill the latest AM7TASK.exe and afterwards the other 49 queued up tasks will "complete". They are marked as running successfully, but don't actually perform the intended commands. I have installed a fresh copy of Automate on a different Win 7 x86 machine (now running 7.1.3.0) and I'm still experiencing the same issues. Any ideas?

Thanks.



Update: The additional attachment shows one of the jobs hanging on Step 4. This step is the last step in that particular task and is just an End If. The other tasks in the event monitor just piled up. Upon ending the hung task, the other tasks ended, were listed as successful, but never performed the intended jobs. I believe, in all cases, the hung job is hanging on an End If.


Hi Joel,

We will perform some testing on this issue and let you know the results. In the meantime, could you attach a task that frequently exhibits this problem?




 Category Survey


-------------------------
- Rick Johnson
Network Automation
 06/18/2010 09:05 AM
User is offline View Users Profile Print this message

Author Icon
JoelWilliams
Apprentice (20-49)

Posts: 25
Joined: 08/04/2005

I could post just about any task because I've noticed it hanging on several. Nevertheless, here's one.

I noticed one more behavior. After I kill the hung am7task.exe and the rest of the am7task.exe appear to process, the event monitor still shows one task working when I hover over the icon.

 Category Survey



 06/18/2010 09:27 AM
User is offline View Users Profile Print this message

Author Icon
Rick Johnson
AUTOMATE EMPLOYEE

Posts: 2757
Joined: 07/14/2008

Originally posted by: JoelWilliams

I could post just about any task because I've noticed it hanging on several. Nevertheless, here's one.



I noticed one more behavior. After I kill the hung am7task.exe and the rest of the am7task.exe appear to process, the event monitor still shows one task working when I hover over the icon.


Hi Joel,

What do you have your task isolation set to? If you are not using high isolation, could you try setting the isolation to high and see if it changes anything?

How many tasks are intended to be runinng at once, and how frequently do tasks run? You may also want to have the Windows Task Manager open to the Processes tab and take note on the Private Working Set memory, Handles and Threads to see if any of those levels are continuously rising.

I will evaluate your task as well.

 Category Survey


-------------------------
- Rick Johnson
Network Automation

Edited: 06/18/2010 at 04:49 PM by Rick Johnson
 06/18/2010 10:12 AM
User is offline View Users Profile Print this message

Author Icon
JoelWilliams
Apprentice (20-49)

Posts: 25
Joined: 08/04/2005

I have set the isolation to "always" on all the tasks. I am not usually running more than one task at a time, but I do run many of them frequently (as much at every 5 min).

 Category Survey
 06/18/2010 04:50 PM
User is offline View Users Profile Print this message

Author Icon
Rick Johnson
AUTOMATE EMPLOYEE

Posts: 2757
Joined: 07/14/2008

Hi Joel,

What else is running on this machine, and how much RAM does it have?

Did you get a chance yet to monitor the Task Manager?

 Category Survey


-------------------------
- Rick Johnson
Network Automation
 06/19/2010 04:44 PM
User is offline View Users Profile Print this message

Author Icon
JeffTysco
Artisan (200-499)

Posts: 319
Joined: 07/30/2009

Originally posted by: JoelWilliams I have set the isolation to "always" on all the tasks. I am not usually running more than one task at a time, but I do run many of them frequently (as much at every 5 min).

I believe the option referenced can be found under System --> Options on the System Tab.  At the bottom there is a button to "Change Task Isolation Level".  Setting this to High resolved my very similar issues (Tasks shown as still running on an end if step).

 



 Category Survey


-------------------------
?? . Your Total IT Solutions Provider
?? . Business Class Hosting Services
 06/21/2010 01:59 PM
User is offline View Users Profile Print this message

Author Icon
JoelWilliams
Apprentice (20-49)

Posts: 25
Joined: 08/04/2005

I have monitored the task manager and it has not showed any large usage of physical or virtual memory. The machine has 2GB of RAM. I'm only running the following apps:
MS Office 2007 w/sp2
Adobe Flash
PGP Desktop 10.0.2.13
VNC Enterprise 4.5.1
WinRAR

Jeff...thanks for the info on task isolation.

 Category Survey
 06/21/2010 04:42 PM
User is offline View Users Profile Print this message

Author Icon
Rick Johnson
AUTOMATE EMPLOYEE

Posts: 2757
Joined: 07/14/2008

Originally posted by: JoelWilliams

I have monitored the task manager and it has not showed any large usage of physical or virtual memory. The machine has 2GB of RAM. I'm only running the following apps:

MS Office 2007 w/sp2

Adobe Flash

PGP Desktop 10.0.2.13

VNC Enterprise 4.5.1

WinRAR


Hi Joel,

How much memory is in use immediately before, during and after the task run?


 Category Survey


-------------------------
- Rick Johnson
Network Automation
 06/22/2010 08:01 AM
User is offline View Users Profile Print this message

Author Icon
JoelWilliams
Apprentice (20-49)

Posts: 25
Joined: 08/04/2005

The entire system is never using more than about 500 MB of RAM. This morning, even after setting the tasks to High Isolation, I had 50 tasks piled up with one of them hanging up the rest. Before I cleared things up this morning, the system was using 330MB of RAM. Please advise.

 Category Survey
 06/22/2010 09:17 AM
User is offline View Users Profile Print this message

Author Icon
JeffTysco
Artisan (200-499)

Posts: 319
Joined: 07/30/2009

Originally posted by: JoelWilliams The entire system is never using more than about 500 MB of RAM. This morning, even after setting the tasks to High Isolation, I had 50 tasks piled up with one of them hanging up the rest. Before I cleared things up this morning, the system was using 330MB of RAM. Please advise.

 

Just out of curiosity, do any of your tasks use the terminal functionality?

 

Also, do you have the ability to run more than 1 task at a time?  (System --> Options --> Priorities)



 Category Survey


-------------------------
?? . Your Total IT Solutions Provider
?? . Business Class Hosting Services
 06/22/2010 09:36 AM
User is offline View Users Profile Print this message

Author Icon
JoelWilliams
Apprentice (20-49)

Posts: 25
Joined: 08/04/2005

I do have one task that uses a terminal; I'd be happy to disable it if you think that might be the culprit. I can run 50 simultaneous tasks.

 Category Survey
 06/22/2010 10:05 AM
User is offline View Users Profile Print this message

Author Icon
Rick Johnson
AUTOMATE EMPLOYEE

Posts: 2757
Joined: 07/14/2008

Originally posted by: JoelWilliams

I do have one task that uses a terminal; I'd be happy to disable it if you think that might be the culprit. I can run 50 simultaneous tasks.


Hi Joel,

Do you see any errors in the Windows Event Viewer when the tasks start hanging and piling up?

 Category Survey


-------------------------
- Rick Johnson
Network Automation
 06/22/2010 12:33 PM
User is offline View Users Profile Print this message

Author Icon
JeffTysco
Artisan (200-499)

Posts: 319
Joined: 07/30/2009

Originally posted by: JoelWilliams I do have one task that uses a terminal; I'd be happy to disable it if you think that might be the culprit. I can run 50 simultaneous tasks.
If it's only the 1 task that uses the terminal functionality then I doubt its the same issue I was experiencing. I was just asking as I had a similar issue (posted here if you want to review: http://forums.networkautomation.com/forum/messageview.cfm?catid=34&threadid=7356&enterthread=y)

 Category Survey


-------------------------
?? . Your Total IT Solutions Provider
?? . Business Class Hosting Services
 06/22/2010 01:09 PM
User is offline View Users Profile Print this message

Author Icon
JoelWilliams
Apprentice (20-49)

Posts: 25
Joined: 08/04/2005

There are no errors recorded when the tasks begin to hang and pile up.

 Category Survey
 06/22/2010 04:47 PM
User is offline View Users Profile Print this message

Author Icon
Rick Johnson
AUTOMATE EMPLOYEE

Posts: 2757
Joined: 07/14/2008

Originally posted by: JoelWilliams

There are no errors recorded when the tasks begin to hang and pile up.


Hi Joel,

How are these tasks being triggered?

I'm having difficulty causing the problem to happen in the test labs.

 Category Survey


-------------------------
- Rick Johnson
Network Automation
 06/29/2010 12:28 PM
User is offline View Users Profile Print this message

Author Icon
JoelWilliams
Apprentice (20-49)

Posts: 25
Joined: 08/04/2005

The tasks are started various ways. Most of them are scheduled.
I went ahead and reinstalled Automate on a WinXP SP3 machine. So far so good.

 Category Survey
 06/29/2010 01:43 PM
User is offline View Users Profile Print this message

Author Icon
Bob Stein
Journeyman (50-99)

Posts: 54
Joined: 06/17/2005

I think I'm having the same problem. I just upgraded from AutoMate 6 to AutoMate 7.1.3.0 to see if it fixed the problem. Same as you. The hang is random. I have one task that runs every 5 minutes and about 3 times a week, it will hang. I'll be interested in your solution.

 Category Survey
 06/29/2010 04:51 PM
User is offline View Users Profile Print this message

Author Icon
Rick Johnson
AUTOMATE EMPLOYEE

Posts: 2757
Joined: 07/14/2008

Hi Bob,

Please let us know if using the High Isolation helps your issue.

Thanks!

 Category Survey


-------------------------
- Rick Johnson
Network Automation
 06/30/2010 07:39 AM
User is offline View Users Profile Print this message

Author Icon
JoelWilliams
Apprentice (20-49)

Posts: 25
Joined: 08/04/2005

Bob,

My hung task issue has gone away now that I am back running Automate on a Win XP SP3 machine. I simply ran out of patience in dealing with the hung tasks.

Joel

 Category Survey
 06/30/2010 08:11 AM
User is offline View Users Profile Print this message

Author Icon
AndrewWalther
Expert (100-199)

Posts: 102
Joined: 09/22/2008

I came in this morning and currently experienceing some what the same issue. The tasks are hanging at the final step, so I have all the task queued up as running stuck on the last step. Even the "end" commands have no affect. I did restart the service and the task ran good for awhile, then the issue occured again. Task isolation was always set at "High" and running 7.1.1.0 . Any solutions will be great since this is affecting half of our production.



 Category Survey
 06/30/2010 08:32 AM
User is offline View Users Profile Print this message

Author Icon
AndrewWalther
Expert (100-199)

Posts: 102
Joined: 09/22/2008

I checked our event viewer and found this .Net error which is the reason for evering thing going sour. Attached is a screenshot

Originally posted by: AndrewWalther I came in this morning and currently experienceing some what the same issue. The tasks are hanging at the final step, so I have all the task queued up as running stuck on the last step. Even the "end" commands have no affect. I did restart the service and the task ran good for awhile, then the issue occured again. Task isolation was always set at "High" and running 7.1.1.0 . Any solutions will be great since this is affecting half of our production.

 



 Category Survey


Click for full image
Click for larger image

 06/30/2010 09:08 AM
User is offline View Users Profile Print this message

Author Icon
Rick Johnson
AUTOMATE EMPLOYEE

Posts: 2757
Joined: 07/14/2008

Hi Andrew,

You should update to version 7.1.3 as we have improved product stability over the last couple of updates.

 Category Survey


-------------------------
- Rick Johnson
Network Automation
 06/30/2010 10:19 AM
User is offline View Users Profile Print this message

Author Icon
AndrewWalther
Expert (100-199)

Posts: 102
Joined: 09/22/2008

Upgraded to 7.1.3.0, and running for an hour now, so far so good. Will update if the issue comes up again.

 

Originally posted by: Rick Johnson (N.A.) Hi Andrew, You should update to version 7.1.3 as we have improved product stability over the last couple of updates.



 Category Survey
 07/02/2010 03:32 PM
User is offline View Users Profile Print this message

Author Icon
Rick Johnson
AUTOMATE EMPLOYEE

Posts: 2757
Joined: 07/14/2008

Hi Andrew,

I'm glad thing ssee to have improved, thank you for the update!

 Category Survey


-------------------------
- Rick Johnson
Network Automation
 08/04/2011 08:28 AM
User is offline View Users Profile Print this message

Author Icon
AdamKaplan
Apprentice (20-49)

Posts: 21
Joined: 06/02/2006

We are running 7.1.3 and have the same hanging issue reported in this thread. It happens randomly daily, then stops for a while and then starts again. There is no rhyme or reason and it's not on the same tasks.

We have no terminal tasks.

We do have a lot of SQL tasks and more often than not - one of the SQL tasks is hung. We can have 10-15 tasks running at one. Most are file triggered. Some are time triggered and loop until certain conditions are met.

We are running on a Windows 2003 Enterprise R2 box with 8GB of RAM.

There are no windows event log messages.

We have noticed that even though some of the tasks show as "hung" - they actually complete. Others do not. Very frustrating!

 Category Survey
 08/04/2011 02:27 PM
User is offline View Users Profile Print this message

Author Icon
Rick Johnson
AUTOMATE EMPLOYEE

Posts: 2757
Joined: 07/14/2008

Hi Adam,

In some cases, the machine can be rendered in such a stat and services need to be restarted or the machine rebooted.

Do you ever notice if the server is running low on RAM or maxed out in CPU usage?
What else runs on the server other than AutoMate? It may be possible to relocate some applications to anpther machine should that be the case that other applications hinder performance.

 Category Survey


-------------------------
- Rick Johnson
Network Automation
 04/04/2012 08:10 PM
User is offline View Users Profile Print this message

Author Icon
DaveScotese
Student (5-19 posts)

Posts: 6
Joined: 09/23/2010

We have been having a similar problem. I'm migrating tasks from an old server to a new one (Both Windows Server 2003 Enterprise w/ SP2). The new one has twice the ram (6GB) and is faster (2.39 GHz vs 2.00GHz). We have Automate 7.1.3 on the new one and 6.2.1 on the old one. The new server shows several odd behaviors:
* A Task starts and completes but is till listed in "Running Tasks" (several more also - they are scheduled by time and interval)
* I was able to select all the tasks in the "Running Tasks" window and End them, or so it looked, but when I closed that window, the tray icon turned blue for a moment (no tasks running) and then immediately back to green. The first task was in there again, and its PID was "(hosted)". So I decided to stop the service and restart it.
* The Automate Service stopped and restarted, and the Running Tasks window no longer showed the task. Ibut the next time I tried to stop it, it "did not respond in a timely manner". When I tried to reboot, it indicated that the program was not ending, and would I like to End it? I ended it.
* After the reboot, one task ran and ended properly several times, but another task ran properly, and then finished - the blue box that comes up in the lower left corner of the screen disappeared - but the AM7Task.exe was still running in Task Manager.
* The renegade task, after reboot, was disabled, but I enabled it. It is set to "when the task is late, do not run at all," so it automatically scheduled itself for the next time slot, 7:58:26pm. I didn't want to wait, so I updated the trigger. It then ran at the earlier time, but th AM7Task process didn't end. Additionally, at the overwritten schedule's time (7:58:26pm), the "Messages" window in Task Administrator shows "EAccessViolation".

After compiling the odd behaviors above, I theorized that the logic in Automate v.7.1.3 for updating triggers fails to properly clean up the object it uses to trigger things. Such an un-cleaned-up object might prevent the AMTask.exe process from exiting, which would explain why they stack up. Forgive me for theorizing from such a distance and through such a small pipe, but it has proven valuable in the past :-).

 Category Survey
 04/05/2012 02:45 AM
User is offline View Users Profile Print this message

Author Icon
JeanDelfosse
Supreme Guru (1000+)

Posts: 8883
Joined: 03/30/2005

Dave,
Usually it is not necessary to reboot the system, and stopping the Automate service (both from the services console, and if it does not stop properly, again from the windows task manager), then restarting it would be enough.

Did you try that instead of rebooting ? And did it prove unsuccessful ?

Those stopping/restarting operations of AutoMate can be achieved with an Automate task (aml file), run either from a shortcut to AMTask.exe or even scheduled by a Windows scheduled task.

Unfortunately Automate 7 is no longer maintained and only workarounds can be found.
There were indeed instances found of tasks never completely ending, or never really declared as ended in the Task Administrator, and the only escape is to restart AutoMate as I said above, hopefully without the need to reboot the system.

 Category Survey
AutoMate 7 version:
Windows version:


-------------------------
Jean Delfosse - Automate Beta Tester
Win7 Pro 64 bits -Win XP Pro SP3 32 bits - AM8, AM7, BPA10, BPA9, BPA8, BPA7, AM6 Enterprise, AM4
 04/05/2012 12:25 PM
User is offline View Users Profile Print this message

Author Icon
DaveScotese
Student (5-19 posts)

Posts: 6
Joined: 09/23/2010

I narrowed down the offending task step to a script.

I then narrowed down the offending script code to the line
Debug.Clear

After removing that code from the script, it stopped hanging and does everything we want it to do.

It takes about 18 seconds for Automate to get the task running and display the blue window that shows it's running, but the blue window now shows up for the whole time the task is running, and when it ends, the message shows it was successful, and the process goes away. Yay!

Hope it helps someone.



 Category Survey
 04/06/2012 03:07 AM
User is offline View Users Profile Print this message

Author Icon
JeanDelfosse
Supreme Guru (1000+)

Posts: 8883
Joined: 03/30/2005

Dave,
nice you found that debug.clear issue.
It is really amazing.

 Category Survey
AutoMate 7 version:
Windows version:


-------------------------
Jean Delfosse - Automate Beta Tester
Win7 Pro 64 bits -Win XP Pro SP3 32 bits - AM8, AM7, BPA10, BPA9, BPA8, BPA7, AM6 Enterprise, AM4
 04/06/2012 07:12 AM
User is offline View Users Profile Print this message

Author Icon
Leonard Amabile
HelpSystems

Posts: 5454
Joined: 05/15/2007

Hi Dave,

This is a known issue in AM7. The Debug.Clear is causing a EAccessViolation and puts the service in an unresponsive state until restarted. This problem does not occur in AM8 or AM9.




 Category Survey
AutoMate 7 version:
Windows version:


-------------------------

Leonard Amabile | Director, Cross Platform Support
AutoMate | InterMapper | Skybot | Divisions of HelpSystems
T: +1.213.738.1700 | F: +1.213.738.7665
 01/08/2013 11:17 AM
User is offline View Users Profile Print this message

Author Icon
ArunG
Expert (100-199)

Posts: 158
Joined: 10/29/2009

The hung tasks issue has continued to plague us in Automate 7. We have had several debugging sessions with the support staff, and implemented some of the recommendations around task isolation, task priority, sql connections etc, but have not had a resolution yet. We do not have the Debug.clear step or the related issue.

Now just hoping that an upgrade to AM9 would get rid of the issue

 Category Survey
 01/08/2013 07:53 PM
User is offline View Users Profile Print this message

Author Icon
JeanDelfosse
Supreme Guru (1000+)

Posts: 8883
Joined: 03/30/2005

Arun,
I understand the discomfort of being let only to "hope".

 Category Survey
AutoMate 7 version:
Windows version:


-------------------------
Jean Delfosse - Automate Beta Tester
Win7 Pro 64 bits -Win XP Pro SP3 32 bits - AM8, AM7, BPA10, BPA9, BPA8, BPA7, AM6 Enterprise, AM4
Statistics
18258 users are registered to the AutoMate Discussion forum.
There are currently 0 users logged in.
The most users ever online was 5551 on 01/08/2018 at 11:11 AM.
There are currently 1639 guests browsing this forum, which makes a total of 1639 users using this forum.

FuseTalk Enterprise Edition v4.0 - © 1999-2020 FuseTalk Inc. All rights reserved.

Sitemap Network Automation Software Blog