Network Automation Software We are currently migrating this forum
over to our HelpSystems domain. Please
post all new threads in our new
HelpSystems Community Portal.
Post to the HelpSystems Forum
You are not currently logged on. You must be logged on in order to post. Log on
Or Create a new account
AutoMate Discussion
Decrease font size
Increase font size
Topic Title: Automate Randomly Stops Working
Topic Summary:
Created On: 06/30/2010 08:38 AM
Status: Read Only
Linear : Threading : Single : Branch
Search Topic Search Topic
Topic Tools Topic Tools
View similar topics View similar topics
View topic in raw text format. Print this topic.
 06/30/2010 08:38 AM
User is offline View Users Profile Print this message

Author Icon
JamieBenton
Journeyman (50-99)

Posts: 56
Joined: 02/17/2010

hi,

 

We are running Automate 7 and every so often tasks stop triggering, restarting the Automate 7 service does not resolve this so we have to refer to restarting the server.

does anybody know what could cause this?

also, as we don't notice for about an hour after Automate stops, is there anyway of setting up a notification that could email me to let me know the tasks have stopped triggering?

thank you,



 Category Survey
 06/30/2010 09:15 AM
User is offline View Users Profile Print this message

Author Icon
Rick Johnson
AUTOMATE EMPLOYEE

Posts: 2757
Joined: 07/14/2008

Hi Jamie,

If you have not yet done so, updating to AutoMate v7.1.3 may help with some of those issues.

In thge System Options, you can set up an email notification for task errors.

Do any specific tasks, or steps with specific actions seem to be repeatedly causing issues?

 Category Survey


-------------------------
- Rick Johnson
Network Automation
 06/30/2010 09:29 AM
User is offline View Users Profile Print this message

Author Icon
JamieBenton
Journeyman (50-99)

Posts: 56
Joined: 02/17/2010

Rick,

 

We have the task failure notification enabled.

Nevermind, its not essential I have the notifications of failed task triggering.

I will try upgrading to 7.1.3

thanks!



 Category Survey
 07/09/2010 05:03 AM
User is offline View Users Profile Print this message

Author Icon
JamieBenton
Journeyman (50-99)

Posts: 56
Joined: 02/17/2010

hi,

we have upgraded our Automate to 7.1.3.0

The problem still arises where the Automate tasks randomly stop triggering. This has already happened 4 times and i am concerned that this has become a common problem which is occurring more and more...

are there any known issues as to why a piece of software, that should be reliable and not crash at all is freezing or stops the tasks triggering?

Every time Automate stops working, we have to reboot the server to get it going again.

thank you,



 Category Survey
 07/09/2010 08:33 AM
User is offline View Users Profile Print this message

Author Icon
C.Bowler
n00b (>6 posts)

Posts: 4
Joined: 05/09/2006

Has this issue been resolved? I logged in here to see if there was a fix because we are also experiencing this same problem. Using various priority types on the task properties did not seem to matter. The task could not stop the previous task and would not run.

Looking at "System / Show Running Tasks", it shows some tasks that should run in a couple of minutes that have been running for over 24 hours. When we highlight a task and click "End Task" it does nothing. When we click "End All" it clears the screen, but closing and reopening the screen shows the same tasks still running the same steps. The PID shown is not in the Taskmgr list of PIDs.

In our case, however, we did find that starting and stopping the Automate 7 service cleared the tasks and allowed them to be triggered again,. So following the examples from http://www.pctipsbox.com/how-s...ces-from-a-batch-file/ we created a batch file to stop and start the Automate 7 service.


 Category Survey
 07/09/2010 09:02 AM
User is offline View Users Profile Print this message

Author Icon
Rick Johnson
AUTOMATE EMPLOYEE

Posts: 2757
Joined: 07/14/2008

Originally posted by: JamieBenton

hi,




we have upgraded our Automate to 7.1.3.0




The problem still arises where the Automate tasks randomly stop triggering. This has already happened 4 times and i am concerned that this has become a common problem which is occurring more and more...




are there any known issues as to why a piece of software, that should be reliable and not crash at all is freezing or stops the tasks triggering?




Every time Automate stops working, we have to reboot the server to get it going again.




thank you,



Hi Jamie,

What steps are the tasks which hang getting stuck on? Are the tasks set to Always Isolate and the system prefrence for Isolation set to High?

 Category Survey


-------------------------
- Rick Johnson
Network Automation
 07/09/2010 09:04 AM
User is offline View Users Profile Print this message

Author Icon
Rick Johnson
AUTOMATE EMPLOYEE

Posts: 2757
Joined: 07/14/2008

Originally posted by: C.Bowler

Has this issue been resolved? I logged in here to see if there was a fix because we are also experiencing this same problem. Using various priority types on the task properties did not seem to matter. The task could not stop the previous task and would not run.



Looking at "System / Show Running Tasks", it shows some tasks that should run in a couple of minutes that have been running for over 24 hours. When we highlight a task and click "End Task" it does nothing. When we click "End All" it clears the screen, but closing and reopening the screen shows the same tasks still running the same steps. The PID shown is not in the Taskmgr list of PIDs.



In our case, however, we did find that starting and stopping the Automate 7 service cleared the tasks and allowed them to be triggered again,. So following the examples from http://www.pctipsbox.com/how-s...ces-from-a-batch-file/ we created a batch file to stop and start the Automate 7 service.


Hi C.Bowler,

Are you on the latest version of AutoMate (version 7.1.3.0)?
Are you able to determine which steps are causing the hangs?

 Category Survey


-------------------------
- Rick Johnson
Network Automation
 07/09/2010 11:11 AM
User is offline View Users Profile Print this message

Author Icon
JacobusPieters
Artisan (200-499)

Posts: 434
Joined: 07/24/2006

I'm seeing a similar problem on 7.1.3.0. It seems to occur more when the machine is under load (more than 5 task/minute), but it's not consistent either. In each case, the AM7 service (AM7TS) cannot be terminated using the Windows Services window and have to be killed with Windows Task Manager.

During this occurence, all task scheduling stops, no CPU are consumed by AM7TS and no tasks are running. Killing and restarting the service resolves the problem. It would seem as if all active tasks run to completion, but nothing new is scheduled. Task Adminstrator still works normally. Isolation is set to 'Always Isolate' for each task and to 'High' in System Options.

No errors are reported in the Automate logs or the Windows Event log.

The hang seems to occur about once a day. Is there any way of taking dumps that may assist in tracking down this problem?

 Category Survey
 07/09/2010 12:58 PM
User is offline View Users Profile Print this message

Author Icon
C.Bowler
n00b (>6 posts)

Posts: 4
Joined: 05/09/2006

Hi Rick,

Early April this years we installed version 7.1.2.0 build 301081. The testing went smoothly and only in the past 5-6 weeks have we begun running production tasks on the new software. We are planning to migrate our current tasks as time permits and have only created new tasks on version 7. Currently there are 6 tasks enabled in Automate 7 which runs on its own server. About 3 weeks ago we began having this problem and have only discovered this week what I shared in my earlier post.

Below are the steps of the 4 tasks identified in the "Show Running Tasks" window





On %CurrDay%, %CurrDate% we processed the following %NumOfFiles% files for this interface:

%FileSpec%












 Category Survey
 07/09/2010 04:09 PM
User is offline View Users Profile Print this message

Author Icon
Dustin Snell
AUTOMATE EMPLOYEE

Posts: 1656
Joined: 03/30/2005

Originally posted by: JacobusPieters

I'm seeing a similar problem on 7.1.3.0. It seems to occur more when the machine is under load (more than 5 task/minute), but it's not consistent either. In each case, the AM7 service (AM7TS) cannot be terminated using the Windows Services window and have to be killed with Windows Task Manager.



During this occurence, all task scheduling stops, no CPU are consumed by AM7TS and no tasks are running. Killing and restarting the service resolves the problem. It would seem as if all active tasks run to completion, but nothing new is scheduled. Task Adminstrator still works normally. Isolation is set to 'Always Isolate' for each task and to 'High' in System Options.



No errors are reported in the Automate logs or the Windows Event log.



The hang seems to occur about once a day. Is there any way of taking dumps that may assist in tracking down this problem?


Which actions do your tasks use?

 Category Survey


-------------------------
Dustin Snell
 07/09/2010 04:19 PM
User is offline View Users Profile Print this message

Author Icon
Rick Johnson
AUTOMATE EMPLOYEE

Posts: 2757
Joined: 07/14/2008

Hi Jacobus,

In addition to the actions in use, could you also describe the running environment? CPU/RAM, OS, and what else the machine may be doing?

 Category Survey


-------------------------
- Rick Johnson
Network Automation
 07/09/2010 04:20 PM
User is offline View Users Profile Print this message

Author Icon
Rick Johnson
AUTOMATE EMPLOYEE

Posts: 2757
Joined: 07/14/2008

Originally posted by: C.Bowler

Hi Rick,



Early April this years we installed version 7.1.2.0 build 301081. The testing went smoothly and only in the past 5-6 weeks have we begun running production tasks on the new software. We are planning to migrate our current tasks as time permits and have only created new tasks on version 7. Currently there are 6 tasks enabled in Automate 7 which runs on its own server. About 3 weeks ago we began having this problem and have only discovered this week what I shared in my earlier post.



Below are the steps of the 4 tasks identified in the "Show Running Tasks" window


Hi C.Bowler,

In the future, please use the Attach Code feature to post steps! I was able to find them and put them in a code window for this reply.

Version 7.1.3.0 does improve issues where tasks appeared to be stuck in the running state.

To dig any deeper we would need to know more about the environment such as CPU/RAM, OS, how many tasks run - and do they run at the same time, what other things are running on the machine.

 Category Survey



Task Steps: (Right click, Select All, Copy to copy to the clipboard. Steps can then be pasted into the AutoMate Task Builder)


-------------------------
- Rick Johnson
Network Automation
 07/12/2010 01:22 AM
User is offline View Users Profile Print this message

Author Icon
JamieBenton
Journeyman (50-99)

Posts: 56
Joined: 02/17/2010

Rick,

there are no specific tasks that are getting stuck. Automate just stops triggering all together at random times every couple of days. We have not had this problem before with Automate 6. When i try to restart the Automate 7 service its status just gets stuck in 'stopping' forcing me to reboot the server.

thank you,



 Category Survey
 07/12/2010 01:47 AM
User is offline View Users Profile Print this message

Author Icon
JeanDelfosse
Supreme Guru (1000+)

Posts: 8883
Joined: 03/30/2005

Jamie,
when you are in such a situation, it means that the service is in "Perpetual stopping state". You cannot stop nor restart the service from the services console.
But you can kill the process itself (from Windows task manager).
Then you can restart the service, which hopefully will work without rebooting the system.

 Category Survey
AutoMate 7 version:
Windows version:


-------------------------
Jean Delfosse - Automate Beta Tester
Win7 Pro 64 bits -Win XP Pro SP3 32 bits - AM8, AM7, BPA10, BPA9, BPA8, BPA7, AM6 Enterprise, AM4
 07/12/2010 11:22 AM
User is offline View Users Profile Print this message

Author Icon
JacobusPieters
Artisan (200-499)

Posts: 434
Joined: 07/24/2006

The symptoms is exactly is Jamie describes. No specific rhyme or reason to the tasks running at the time.

The server is a dedicated Automate server, running just Automate 6 and Automate 7. O/S is Windows 2003 SP2, 4GM memory, 4 x 2.4 Ghz Xeon processor. Only about 25% of memory ever in use, CPU usage peaks at about 50%, average about 10%. Automate mainly does FTPs and ODBC queries, but once again no patterm to the task active when scheduling stops.

I would again like to point out that all active tasks run to successful completion, but no new scheduling will occur whether using file triggers or time triggers. Similar to what other users reported, AM7 cannot be stopped via the service stop but can be killed with Task manager. After restarting the service, all scheduling continue normally.

My feeling on this is that this is probably more of a scheduling problem - maybe some locks not cleared and AM7 gets stuck behing a lock. My reasoning behind this is the fact that all running tasks complete sucessfully and using the 'Show running tasks' confirm this fact. In addition the logs also contains all the termination entries for the tasks that was active when the 'hang' occurred.

I get about 1 hang/day and I have to continue using Automate 6 for any critical tasks - which is a pity since Automate 7 contains new functionality that I really need.

 Category Survey
 07/12/2010 11:24 AM
User is offline View Users Profile Print this message

Author Icon
JacobusPieters
Artisan (200-499)

Posts: 434
Joined: 07/24/2006

The symptoms is exactly is Jamie describes. No specific rhyme or reason to the tasks running at the time.

The server is a dedicated Automate server, running just Automate 6 and Automate 7. O/S is Windows 2003 SP2, 4GM memory, 4 x 2.4 Ghz Xeon processor. Only about 25% of memory ever in use, CPU usage peaks at about 50%, average about 10%. Automate mainly does FTPs and ODBC queries, but once again no patterm to the task active when scheduling stops.

I would again like to point out that all active tasks run to successful completion, but no new scheduling will occur whether using file triggers or time triggers. Similar to what other users reported, AM7 cannot be stopped via the service stop but can be killed with Task manager. After restarting the service, all scheduling continue normally.

My feeling on this is that this is probably more of a scheduling problem - maybe some locks not cleared and AM7 gets stuck behing a lock. My reasoning behind this is the fact that all running tasks complete sucessfully and using the 'Show running tasks' confirm this fact. In addition the logs also contains all the termination entries for the tasks that was active when the 'hang' occurred.

I get about 1 hang/day and I have to continue using Automate 6 for any critical tasks - which is a pity since Automate 7 contains new functionality that I really need.

 Category Survey
 07/12/2010 12:14 PM
User is offline View Users Profile Print this message

Author Icon
Dustin Snell
AUTOMATE EMPLOYEE

Posts: 1656
Joined: 03/30/2005

Originally posted by: JacobusPieters
Automate mainly does FTPs and ODBC queries, but once again no patterm to the task active when scheduling stops.


This one raised a red flag for me although it's just a shot in the dark. We have seen instances where poorly written (including some from MS) OLEDB/ODBC drivers can cause this kind of behavior. This is also indicated because you said your other tasks do not cause this behavior (but, of course are affected because the engine is being affected).

What driver are you using? Perhaps you can send us a sample of your connection string sans the credentials.



 Category Survey


-------------------------
Dustin Snell
 07/12/2010 01:28 PM
User is offline View Users Profile Print this message

Author Icon
JacobusPieters
Artisan (200-499)

Posts: 434
Joined: 07/24/2006

Originally posted by: Dustin Snell (N.A.)

Originally posted by: JacobusPieters

Automate mainly does FTPs and ODBC queries, but once again no patterm to the task active when scheduling stops.





This one raised a red flag for me although it's just a shot in the dark. We have seen instances where poorly written (including some from MS) OLEDB/ODBC drivers can cause this kind of behavior. This is also indicated because you said your other tasks do not cause this behavior (but, of course are affected because the engine is being affected).



What driver are you using? Perhaps you can send us a sample of your connection string sans the credentials.


I use various drivers, but all tasks that were running last night when the scheduling stopped were doing ODBC queries (both retrieval and update) to MYSQL 5.0 using the MySQL ODBC Driver 3.51.

As far as I can determine, all the ODBC queries for the tasks that were running at the time the scheduling stopped, worked normally.

Attached is the SQL queries that were executed, the UPDATE statement the last one that was done on the system. After the update, no further AM7 scheduling occurred.
Retrievals and updates continued normally from AM6 using the same database and drivers.

Maybe it's also worthwhile to mention that I found that in Automate 7 the 'Query times out after nnn seconds' does not work (for any driver, not limited to MySQL - not used on the sample statements). A task that uses a ODBC query will just wait forever if a response is not returned. It workes fine on Automate 6. This was not a problem last night, since all tasks terminated normally.




 Category Survey



Task Steps: (Right click, Select All, Copy to copy to the clipboard. Steps can then be pasted into the AutoMate Task Builder)
 07/13/2010 11:21 AM
User is offline View Users Profile Print this message

Author Icon
Rick Johnson
AUTOMATE EMPLOYEE

Posts: 2757
Joined: 07/14/2008

Hi Jacobus,

Both of those connect strings are specifying the MSDASQL.1 driver, which is the Microsoft OLE DB ODBC driver. Did you have a separate OLE DB ODBC driver for MySQL?

 Category Survey


-------------------------
- Rick Johnson
Network Automation
 07/13/2010 12:34 PM
User is offline View Users Profile Print this message

Author Icon
JacobusPieters
Artisan (200-499)

Posts: 434
Joined: 07/24/2006

Originally posted by: Rick Johnson (N.A.)

Both of those connect strings are specifying the MSDASQL.1 driver, which is the Microsoft OLE DB ODBC driver. Did you have a separate OLE DB ODBC driver for MySQL?


Yes, I'm using the standard MySQL ODBC 3.51 driver (downloadable from the Web). Connection were predefined using 'ODBC Data Source Administrator'.

Statement were created by:

Database --> SQL Query --> Connection String 2nd Icon --> Connection --> Use data source name --> Select from dropdown of System Data Sources

The connection itself is predefined using the "ODBC Data Source Adminsitrator" as a System Data Source' in the System DSN tab. If you look in the statements included in the previous post, you will notice that there is a field "Data Source=FTPMASTER", which refers to a previously defined connection.

Any ODBC or stored procedure call in Automate that uses a predefined connection always fill in the provider as MSDASQL.1 (both AM6 & AM7). I assume that MSDASQL.1 (Micosoft OLE DB Provider for ODBC Drivers) act as middelware between ODBC drivers and Automate, since it is even included for ODBC calls to MS SQL Server.

 Category Survey
 07/14/2010 03:51 PM
User is offline View Users Profile Print this message

Author Icon
Rick Johnson
AUTOMATE EMPLOYEE

Posts: 2757
Joined: 07/14/2008

Hi Jacobus,

Can you attach your ATL file to a reply please?

 Category Survey


-------------------------
- Rick Johnson
Network Automation
 07/15/2010 10:50 AM
User is offline View Users Profile Print this message

Author Icon
C.Bowler
n00b (>6 posts)

Posts: 4
Joined: 05/09/2006

Below I have pasted the system info from our Automate server. As I mentioned before, there are only 6 enabled tasks running on the server. Typical run time ranges from 20 seconds to a few minutes.

Currently there is no overlap of tasks being triggered with the exception of a send/receive process where the sending task actually starts a separate receive task.

I just looked and the send task, of the send/receive process I just mentioned, was listed as still running a step which is just a standard FTP upload. The file it was/is transferring is only 1K.

The task doesn't fail and just never completes.


Specifications
VMware, Inc.
System Model: VMware Virtual Platform
BIOS Version: Phoenix Technologies LTD 6.00


Operating System
Microsoft(R) Windows(R) Server 2003, Standard Edition
Version: 5.2.3790
Service Pack: 2.0
Location: C:\WINDOWS
PID: 69712-640-6718391-45115
Hot Fix: XpsEPSC


Memory (RAM)
Capacity: 2 GB


Processor
Intel(R) Xeon(R) CPU 5160 @ 3.00GHz
Version: x86 Family 6 Model 15 Stepping 6
Speed: 2993 MHz


Local Disk
Total Capacity: 71.49 GB
Sum of Hard Disks: (C: E: F: )
Used: 47.95 GB

Free: 23.54 GB


 Category Survey
 07/15/2010 11:05 AM
User is offline View Users Profile Print this message

Author Icon
Rick Johnson
AUTOMATE EMPLOYEE

Posts: 2757
Joined: 07/14/2008

Originally posted by: C.Bowler

Below I have pasted the system info from our Automate server. As I mentioned before, there are only 6 enabled tasks running on the server. Typical run time ranges from 20 seconds to a few minutes.

Currently there is no overlap of tasks being triggered with the exception of a send/receive process where the sending task actually starts a separate receive task.

I just looked and the send task, of the send/receive process I just mentioned, was listed as still running a step which is just a standard FTP upload. The file it was/is transferring is only 1K.

The task doesn't fail and just never completes.


Hi C.Bowler,

The 7.1.3.0 update does include some fixes that are likely to correct the issues that you are encountering.
Would you be able to update?

 Category Survey


-------------------------
- Rick Johnson
Network Automation
 07/16/2010 01:09 AM
User is offline View Users Profile Print this message

Author Icon
ChristopherGaudet
n00b (>6 posts)

Posts: 5
Joined: 06/23/2010

Hello, we are currently running AutoMate 7.1.3.0 on our production servers. I'm new to AutoMate 7, but all of our testing went well. We have 26 simple tasks that we run on our server, many of which run every one minute. I have attached the tasks so you'll see what I mean. Nothing fancy, just checking to see if a file exists and if so, a command runs. Just after 2:52am ET today, after our tasks were completing successfully from outside of AutoMate 7, AutoMate still showed them all as "Running (1)". I tried to end the tasks and I got some eaccess violation error message. I checked task manager on the server to see if any of the related intoolx.exe's that are called by the tasks were running and none were. The solution for me was to disable task triggering, restart the AutoMate 7 service and re-enable task triggering. Everything went back to normal after that. Problem is it happened in the middle of the night and an admin had to be woken up to troubleshoot the issue. We've only been live in production with AutoMate 7 for 6 days now. Should we be expecting this to happen regularly? I can tell you that when we setup AutoMate 7, based on recommendations we set the system option for Task Isolation to High. At the task level, everything is set to Default. Please let me know if you have any further questions and/or if you have any suggested fixes for this issue. Thanks.

By the way, our server is running Windows Server 2003 DataCenter Edition, SP2. It is an IBM X Series 3950_M2_A1. It has 8 - quad core processors with 32 GB of RAM. Plenty of horsepower here.



 Category Survey



 07/16/2010 03:49 PM
User is offline View Users Profile Print this message

Author Icon
Rick Johnson
AUTOMATE EMPLOYEE

Posts: 2757
Joined: 07/14/2008

Hi Christopher,

What else is this sever doing other than running AutoMate?
How many tasks are you expecting to run at once per minute?
Is the AutoMate processing running at the main console of the server, or are there any virtual machine or remote desktop sessions involved?
If you watch the resource monitor, do the Private Working Set memory, Thread or Handle values climb without ever dropping - even over the course of multiple days?
Are there any errors logged to the Windows Event Viewer?

 Category Survey


-------------------------
- Rick Johnson
Network Automation
 07/18/2010 01:16 AM
User is offline View Users Profile Print this message

Author Icon
ChristopherGaudet
n00b (>6 posts)

Posts: 5
Joined: 06/23/2010

Originally posted by: Rick Johnson (N.A.) Hi Christopher, What else is this sever doing other than running AutoMate? How many tasks are you expecting to run at once per minute? Is the AutoMate processing running at the main console of the server, or are there any virtual machine or remote desktop sessions involved? If you watch the resource monitor, do the Private Working Set memory, Thread or Handle values climb without ever dropping - even over the course of multiple days? Are there any errors logged to the Windows Event Viewer?

Rick, The server also hosts our Document Imaging application, ImageNow. The server is never more than 40% utilized even during peak production hours, although at the time we had our issue, it was more like the server was only 5-10% utilized. I believe that there are approximately 16 tasks that all run every one minute. That was one of the main reasons we bought AutoMate 7, which was so we could stagger our task start times 5-10 secs. So, instead of 16 tasks all firing up every one minute at the top of the minute, they fire off every 1 minute all with different intervals, :05, :10, etc... The AutoMate process runs directly on the server console. There are no remote desktop sessions or VM's involved here. There were no errors in the Windows Event Viewer nor in the AutoMate Logs. Funny thing is that AutoMate 7 continued to queue up the tasks for the Overdue Task runs, however, since the original tasks never "completed", they never got a chance to run. I'll have to check the resource monitor as you request and get back to you.

Thanks.



 Category Survey
 07/18/2010 02:06 AM
User is offline View Users Profile Print this message

Author Icon
ChristopherGaudet
n00b (>6 posts)

Posts: 5
Joined: 06/23/2010

Rick, Since we were down for a maintenance window, I went ahead and captured some infor for you. I'll continue to spot check it as well. I was able to check the three items you were looking for info on and both the number of handles and the private working set seem to grow and not go back down. Threads go up when a task runs but always seem to go back down. Since we were down, I restarted the AutoMate 7 service and found that after a fresh restart, handles is 326. Before I restarted it, it was at 369. At that point, the service had been running only about 2 days. Let me know if you have any other questions.

Thanks.



 Category Survey
 07/19/2010 09:22 AM
User is offline View Users Profile Print this message

Author Icon
JacobusPieters
Artisan (200-499)

Posts: 434
Joined: 07/24/2006

Originally posted by: Rick Johnson (N.A.)

Hi Jacobus,



Can you attach your ATL file to a reply please?


File attached. I had however reverted most of my tasks back to Automate 6, so I don't know what the ATL file file show.





 Category Survey
 07/19/2010 09:32 AM
User is offline View Users Profile Print this message

Author Icon
JacobusPieters
Artisan (200-499)

Posts: 434
Joined: 07/24/2006

Originally posted by: JacobusPieters

Originally posted by: Rick Johnson (N.A.)



Hi Jacobus,







Can you attach your ATL file to a reply please?




File attached. I had however reverted most of my tasks back to Automate 6, so I don't know what the ATL file file show.

Doesn't look like the Attach file work? Trying again.




 Category Survey
 07/19/2010 10:44 AM
User is offline View Users Profile Print this message

Author Icon
Rick Johnson
AUTOMATE EMPLOYEE

Posts: 2757
Joined: 07/14/2008

Originally posted by: ChristopherGaudet

Rick, Since we were down for a maintenance window, I went ahead and captured some infor for you. I'll continue to spot check it as well. I was able to check the three items you were looking for info on and both the number of handles and the private working set seem to grow and not go back down. Threads go up when a task runs but always seem to go back down. Since we were down, I restarted the AutoMate 7 service and found that after a fresh restart, handles is 326. Before I restarted it, it was at 369. At that point, the service had been running only about 2 days. Let me know if you have any other questions.




Thanks.



Hi Christopher,

The .NET framework is supposed to clean up the freed resources, but it does so according to an internal process to .NET, and since that machine has a lot of free RAM, the cleanups might be few and far between. I'm going to review your data with the developers to see if it sheds any light on the issue. One thing I have noticed is that if you were to temporarily disable triggering for a short time, the resource usage levels drop back to their baseline values.

 Category Survey


-------------------------
- Rick Johnson
Network Automation
 07/20/2010 04:51 PM
User is offline View Users Profile Print this message

Author Icon
Rick Johnson
AUTOMATE EMPLOYEE

Posts: 2757
Joined: 07/14/2008

Hi Jacobus,

We're not seeing anything obvious in your ATL file, would you be able tt allow us to do a remote session to further troubleshoot? Please send an email to support@networkautomation.com with your availibility.

 Category Survey


-------------------------
- Rick Johnson
Network Automation
 07/22/2010 08:28 AM
User is offline View Users Profile Print this message

Author Icon
JacobusPieters
Artisan (200-499)

Posts: 434
Joined: 07/24/2006

Originally posted by: Rick Johnson (N.A.)

Hi Jacobus,



We're not seeing anything obvious in your ATL file, would you be able tt allow us to do a remote session to further troubleshoot? Please send an email to support@networkautomation.com with your availibility.

Unfortunately the servers are in a secure network that cannot be accessed externally. I'm trying to recreate the problem on a development server that would be visible.
But I don't think a remote session will help. I think there is a bug in Automate, since this problem does not seem to be dependent on any particular task, just load/frequency related.



 Category Survey
 07/22/2010 03:44 PM
User is offline View Users Profile Print this message

Author Icon
Rick Johnson
AUTOMATE EMPLOYEE

Posts: 2757
Joined: 07/14/2008

Hi Jacobus,

Unless you are running out of memory, which did not seem to be the case, then the cause for your problems is new and something we do not see in our testing labs.

We could remote your machine, then if possible remote from your station to the server. It will greatly assist in troubleshooting if we were able to get a firsthand look at what was going on there.

 Category Survey


-------------------------
- Rick Johnson
Network Automation
 07/26/2010 11:03 AM
User is offline View Users Profile Print this message

Author Icon
JacobusPieters
Artisan (200-499)

Posts: 434
Joined: 07/24/2006

Sorry, that will not be possible due to confidentiality issues.

Also, the problem does not occur on demand. Sometime it may happen twice a day, other times several days may pass before the problem occurs.

I'm still attempting to set up a set of jobs on a test server to recreate the problem. Unfortunately due to the unpredictability of the problem, it may take some time to find the right combination.

 Category Survey
 08/06/2010 06:21 AM
User is offline View Users Profile Print this message

Author Icon
Sher
Journeyman (50-99)

Posts: 59
Joined: 08/11/2006

I am having a problem reading all the text in this topic because it is cut off on the right so I don't know if anyone posted this information.

I was having a similar problem and have a simple solution. What I found is that I can't have automatic updates set on the Automate machine. It seems to clobber the memory and my tasks would no longer trigger. I had to disable all automatic Windows/ Virus updates. I install the updates manually and then reboot the machine. This seems to have fixed the problem.

Sherri


 Category Survey
 08/06/2010 09:27 AM
User is offline View Users Profile Print this message

Author Icon
Rick Johnson
AUTOMATE EMPLOYEE

Posts: 2757
Joined: 07/14/2008

Hi Sherri,

Thank you for the feedback - Nobody else to date has cured the issue in that method, so this is somehting else that other people can try to do to improve system stability.

 Category Survey


-------------------------
- Rick Johnson
Network Automation
 09/19/2010 09:41 PM
User is offline View Users Profile Print this message

Author Icon
GeoffKing
Apprentice (20-49)

Posts: 25
Joined: 03/22/2010

Not sure if this is relevant or not.

 

We have been experiencing similar problems lately.

 

The Automate Audit Log around the time of the latest incident has:

Taskname: N/A

Event: Disconnected

Followed by:

Taskname: N/A

Event: Disconnection from service. (127.0.0.1,0)

both timstamped 1:51:44PM

 

Checking messages in the Windows Event Viewer has, at 1:52:36PM:

Source: Automate7

Message: Attempt to wait and establish pipe from AMEM to Service timed out with code 121

 

When these incidents occur we usually end up rebooting the machine.

 

 



 Category Survey
 09/21/2010 10:25 AM
User is offline View Users Profile Print this message

Author Icon
Rick Johnson
AUTOMATE EMPLOYEE

Posts: 2757
Joined: 07/14/2008

Hi Geoff,

Do you know how much uptime the server has when this happens? How frequently are tasks running on this machine?

 Category Survey


-------------------------
- Rick Johnson
Network Automation
 09/21/2010 06:12 PM
User is offline View Users Profile Print this message

Author Icon
GeoffKing
Apprentice (20-49)

Posts: 25
Joined: 03/22/2010

Uptime varies. Might go for over a week, then it might happen the day after a previous instance.

 

Load varies. Between 3am and 10am there might be anything up to 20 jobs running at one time. Afternoon, maybe 5. Overnight about 2 an hour



 Category Survey
 09/21/2010 06:18 PM
User is offline View Users Profile Print this message

Author Icon
GeoffKing
Apprentice (20-49)

Posts: 25
Joined: 03/22/2010

Also, the machine Automate is on is only used to run Automate tasks.

 

The tasks are set up to log activity information to a SQLServer db table.

 

Nothing else is running on the machine (Windows Server 2008)



 Category Survey
 10/05/2010 07:56 PM
User is offline View Users Profile Print this message

Author Icon
Leonard Amabile
HelpSystems

Posts: 5454
Joined: 05/15/2007

Originally posted by: GeoffKing

Also, the machine Automate is on is only used to run Automate tasks.




 




The tasks are set up to log activity information to a SQLServer db table.




 




Nothing else is running on the machine (Windows Server 2008)



Hi Geoff,

Are you using the SQL Native Client driver in your SQL queries?

 Category Survey
AutoMate 7 version:
Windows version:


-------------------------

Leonard Amabile | Director, Cross Platform Support
AutoMate | InterMapper | Skybot | Divisions of HelpSystems
T: +1.213.738.1700 | F: +1.213.738.7665
 10/07/2010 11:39 AM
User is offline View Users Profile Print this message

Author Icon
Jacob
Apprentice (20-49)

Posts: 46
Joined: 05/17/2005

We have experienced the same problem with 7.1.3.0; I don't know what causes it but automate 6 on the same machine never stops working Automate 7.1.3.0 is better than previous versions but it is still very unreliable.

 Category Survey


-------------------------
Jacob M.
AM 6.2.8 / AM 7.1.2.0
 11/01/2010 06:41 PM
User is offline View Users Profile Print this message

Author Icon
ScottFinkle
Apprentice (20-49)

Posts: 23
Joined: 11/06/2009

Jacob,

Have you had any luck resolving your problem with AutoMate hanging.  I have the same problem.  Currently have 8 remote AutoMate run-times at client sites and they hang for no rhyme or reason.  I have never been able to stablize the system.  I"m running the latest version of AutoMate 7.1.3.  I have gone to great lengths to build alert systems that tell me when the system has stopped processing.  All I can do is logon to the remote machine using LogMeIn and reset it.  I've given up on it ever getting fixed and am resigned to having to maintain remote access so I can stand the thing back up when it falls.

If you figured something out please let me know.

Scott



 Category Survey
 11/01/2010 10:29 PM
User is offline View Users Profile Print this message

Author Icon
JeanDelfosse
Supreme Guru (1000+)

Posts: 8883
Joined: 03/30/2005

I fully agree on the reliability problem of AM7 triggering.
At a certain period, for critical tasks, I was using AM6 triggers, and the tasks were just running an AMRUN step invoking the AM7 Task.exe.
This made me benefit of the AM6 reliability and AM7 actions.

 Category Survey
AutoMate 7 version:
Windows version:


-------------------------
Jean Delfosse - Automate Beta Tester
Win7 Pro 64 bits -Win XP Pro SP3 32 bits - AM8, AM7, BPA10, BPA9, BPA8, BPA7, AM6 Enterprise, AM4
 11/15/2010 02:13 PM
User is offline View Users Profile Print this message

Author Icon
Jacob
Apprentice (20-49)

Posts: 46
Joined: 05/17/2005

I have not been able to resolve. I opened a support call today so let's see if that helps.

 Category Survey


-------------------------
Jacob M.
AM 6.2.8 / AM 7.1.2.0
 11/15/2010 03:35 PM
User is offline View Users Profile Print this message

Author Icon
Leonard Amabile
HelpSystems

Posts: 5454
Joined: 05/15/2007

Originally posted by: Jacob

I have not been able to resolve. I opened a support call today so let's see if that helps.


Hi Jacob,

Please let us know if the patch corrects your problem.

 Category Survey
AutoMate 7 version:
Windows version:


-------------------------

Leonard Amabile | Director, Cross Platform Support
AutoMate | InterMapper | Skybot | Divisions of HelpSystems
T: +1.213.738.1700 | F: +1.213.738.7665
 11/22/2010 02:10 PM
User is offline View Users Profile Print this message

Author Icon
Jacob
Apprentice (20-49)

Posts: 46
Joined: 05/17/2005

No, although we didn't have it lock up for about a week we had the same issue over the weekend.

 Category Survey


-------------------------
Jacob M.
AM 6.2.8 / AM 7.1.2.0
 11/22/2010 09:45 PM
User is offline View Users Profile Print this message

Author Icon
JeanDelfosse
Supreme Guru (1000+)

Posts: 8883
Joined: 03/30/2005

Leonard,
which patch do you refer to ?

 Category Survey
AutoMate 7 version:
Windows version:


-------------------------
Jean Delfosse - Automate Beta Tester
Win7 Pro 64 bits -Win XP Pro SP3 32 bits - AM8, AM7, BPA10, BPA9, BPA8, BPA7, AM6 Enterprise, AM4
 11/24/2010 02:25 PM
User is offline View Users Profile Print this message

Author Icon
Rick Johnson
AUTOMATE EMPLOYEE

Posts: 2757
Joined: 07/14/2008

Hi Jean,

It was a patch for the File Trigger.

 Category Survey


-------------------------
- Rick Johnson
Network Automation
 12/22/2010 03:27 PM
User is offline View Users Profile Print this message

Author Icon
ScottFinkle
Apprentice (20-49)

Posts: 23
Joined: 11/06/2009

Anything further on this thread.  I'm still dieing a slow death trying to keep 15 remote sites up and running.  The task service is constantly hanging and I have angry customers.  Waiting for v8 to become "stable".  Would like to resolve in v7 still if possible.

My tasks are heavy into copying/moving files and querying IBM iSeries (aka AS/400) databases.  I've gone from ODBC to OLEDB without any change.  I also read from XML files quite often.

I remain desparate.  It's been 1 full year now that I've been wrestling with the same issues.

Thanks in advance for any help.



 Category Survey
 12/22/2010 05:24 PM
User is offline View Users Profile Print this message

Author Icon
JacobusPieters
Artisan (200-499)

Posts: 434
Joined: 07/24/2006

I've changed my SQL queries/updates from Automate commands to VB scripts run in Automate. This seems to improve my problems and I haven't had a hang since then.
But my SQL queries was quite simple and easy to change, normally inserting/updating/returning 1 row only. May be more difficult with large result sets.

 Category Survey
 01/17/2011 11:34 AM
User is offline View Users Profile Print this message

Author Icon
MetodGantar
Student (5-19 posts)

Posts: 16
Joined: 10/29/2007

I had similar problem. Try disabling logmein.



 Category Survey
 01/20/2011 11:02 AM
User is offline View Users Profile Print this message

Author Icon
Rick Johnson
AUTOMATE EMPLOYEE

Posts: 2757
Joined: 07/14/2008

Hi Metod,

Thank you for that tip. Would you happen to recall which version of LogMeIn was causing the conflict on your system?

 Category Survey


-------------------------
- Rick Johnson
Network Automation
 01/22/2011 11:22 PM
User is offline View Users Profile Print this message

Author Icon
MetodGantar
Student (5-19 posts)

Posts: 16
Joined: 10/29/2007

It was trial version of LogMeIn Pro 4.1.0.1578

Originally posted by: Rick Johnson (N.A.) Hi Metod, Thank you for that tip. Would you happen to recall which version of LogMeIn was causing the conflict on your system?



 Category Survey
 01/25/2011 03:18 PM
User is offline View Users Profile Print this message

Author Icon
Rick Johnson
AUTOMATE EMPLOYEE

Posts: 2757
Joined: 07/14/2008

Thanks Metod,

We'll look out for that in case someone has a similar scenario.

 Category Survey


-------------------------
- Rick Johnson
Network Automation
 02/07/2011 12:59 AM
User is offline View Users Profile Print this message

Author Icon
GerardRajaratnam
n00b (>6 posts)

Posts: 1
Joined: 02/02/2011

I am also experiencing this issue. Every few weeks I come in to find the system 'hung'. The issue is as described above, no task triggering, viewing runnign tasks shows a number which have been runing for hours (not fully completed) and no new tasks running.

We are also on a secure environment so I cant share this.

We are runing 7.1.2 and have about 15 tasks which trigger every few minutes (sometime up to 5 run concurrently).

The platform is 2k3 server, dual core 3GH with 4GB RAM fully Microsoft patched and not running anythign else. We are not using VM's, also I access the machine via Remote Desktop.

The processor occasionally peaks at 100% but only for a second or so, the memory is hardly touched.

One of the most processor intensive actions we perform is to 'read' larg (10MB) XML files and write certain items to SQL. All the other actions are moving files to and from local and UNC Path directories.

The issue occurs at any time, usually at the weekend when no one is accessing the box and workflow is light.

One similarity from others is that we are accessing a MS SQL server via the standard MS ODBC, that said we are only doing this for a few tasks, yet the issue affects all tasks.

The only way to get back runing is to restart the AM7 service or reboot the machine.

This issue appears to occur across multiple OS platforms, multiple SQL products (if that is indeed the issue) etc 



 Category Survey
 02/07/2011 03:21 PM
User is offline View Users Profile Print this message

Author Icon
Rick Johnson
AUTOMATE EMPLOYEE

Posts: 2757
Joined: 07/14/2008

Hi Gerard,

Are there any entries in the Windows Event Viewer which would correspond with the problem that you encountered? If so could you post them in a reply to thus message?

 Category Survey


-------------------------
- Rick Johnson
Network Automation
 02/13/2011 04:35 PM
User is offline View Users Profile Print this message

Author Icon
FrankMaio
Student (5-19 posts)

Posts: 6
Joined: 09/23/2008

Hi Guys,

  Is the Task Engine much the Same in AM8,  we also get this happening on one of our servers.  Plenty of Ram and CPU available at the time of the Hangs.  According to the logs All teh Tasks have finished successfully.   One Task seems to alway be one of the last but it runs very frequently so not really surprising.

 Wondering if we have to go to BPA to get this stable...

 

 

Cheers :^)



 Category Survey
 02/14/2011 09:32 AM
User is offline View Users Profile Print this message

Author Icon
Rick Johnson
AUTOMATE EMPLOYEE

Posts: 2757
Joined: 07/14/2008

Originally posted by: FrankMaio

Hi Guys,




  Is the Task Engine much the Same in AM8,  we also get this happening on one of our servers.  Plenty of Ram and CPU available at the time of the Hangs.  According to the logs All teh Tasks have finished successfully.   One Task seems to alway be one of the last but it runs very frequently so not really surprising.



Hi Frank,

Version 8 does include many stability and performance ehancements, have you tried the trial version out yet?

 Category Survey


-------------------------
- Rick Johnson
Network Automation
 03/01/2013 02:16 PM
User is offline View Users Profile Print this message

Author Icon
HiroshiGoto
n00b (>6 posts)

Posts: 2
Joined: 03/01/2013

I have similar issue.

I am running 7.1.3.0 on Windows 2003. I found that all task did not run from 9:10PM last night this morning. I opened AutoMate 7 Administrator on the server, but it does not show the task list. It seemed AutoMate 7 service was hang-up. I checked Windows task manager and I found that AM7TA.exe and AM7TS.exe there but I cannot find AM7EM.exe. I tried to restart Automate 7 service on Windows Service manager, but it returned time out. So, I killed AM7TS.exe on Windows task manager process and start AutoMate 7 service. After restarting AutoMate 7 service, AutoMate 7 Administrator worked properly and all tasks run properly.

 

I checked AutoMate 7 log but there was no errors. On the Windows Event Viewer, I found following error:

Event Type:     Error
Event Source:   AM7TS.exe
Event Category: None
Event ID:       0
Date:           2/28/2013
Time:           9:10:09 PM
User:           N/A

Description:
The description for Event ID ( 0 ) in Source ( AM7TS.exe ) cannot be found.
The local computer may not have the necessary registry information or message DLL files
to display messages from a remote computer. You may be able to use the /AUXSOURCE= flag
to retrieve this description; see Help and Support for details. The following information
is part of the event: .

 

I use AutoMate 7 on this environment for few years. This is a second time for me about AutoMate 7 hang-up. The last time was in last month without any error message.

 

If you have any ideas, please advice.



 Category Survey


Edited: 03/01/2013 at 02:32 PM by HiroshiGoto
 03/04/2013 04:27 PM
User is offline View Users Profile Print this message

Author Icon
Leonard Amabile
HelpSystems

Posts: 5454
Joined: 05/15/2007

Hiroshi,

Do you have a lot of file triggers or at least added more tasks and/or triggers prior to the service hanging? What version of Windows are you running?



 Category Survey
AutoMate 7 version:
Windows version:


-------------------------

Leonard Amabile | Director, Cross Platform Support
AutoMate | InterMapper | Skybot | Divisions of HelpSystems
T: +1.213.738.1700 | F: +1.213.738.7665
 03/04/2013 05:07 PM
User is offline View Users Profile Print this message

Author Icon
HiroshiGoto
n00b (>6 posts)

Posts: 2
Joined: 03/01/2013

I have several task on AutoMate 7. Some are scheduled, some are folder watch. When the hang-up happened, one or two scheduled tasks could be running. On AutoMate 7 log, two tasks finished log are missing. However, according the job program log, those tasks were finished properly.

The operation system is Windows 2003 Server R2 32bit.  



 Category Survey
 03/06/2013 04:14 PM
User is offline View Users Profile Print this message

Author Icon
Liz Casale
AUTOMATE EMPLOYEE

Posts: 940
Joined: 05/23/2011

Hi Hiroshi,

Thank you for your reply. Can you please apply the following patch (attached) to your installation? The .zip contains a readme.txt for how to apply it.

 Category Survey


hotfixambpa713100.zip
hotfixambpa713100.zip  (4521 KB)



-------------------------
Liz Casale
Technical Support Representative
(213) 738-6966
Network Automation, Inc.

--------------------------------------------------------------------------------

AM5 5.5.5.1; AM6 6.2.8.0; AM7 7.1.3.0; AM8 8.0.9.0; AM9 9.0.3.4
BPA7 7.1.3.0; BPA8 8.0.9.0; BPA9 9.0.3.4
Statistics
18258 users are registered to the AutoMate Discussion forum.
There are currently 0 users logged in.
The most users ever online was 5551 on 01/08/2018 at 11:11 AM.
There are currently 1239 guests browsing this forum, which makes a total of 1239 users using this forum.

FuseTalk Enterprise Edition v4.0 - © 1999-2020 FuseTalk Inc. All rights reserved.

Sitemap Network Automation Software Blog