Network Automation Software We are currently migrating this forum
over to our HelpSystems domain. Please
post all new threads in our new
HelpSystems Community Portal.
Post to the HelpSystems Forum
You are not currently logged on. You must be logged on in order to post. Log on
Or Create a new account
AutoMate Discussion
Decrease font size
Increase font size
Topic Title: AMTS.exe process has high memory consumption; possible memory leak?
Topic Summary:
Created On: 09/15/2009 11:57 AM
Status: Read Only
Linear : Threading : Single : Branch
Search Topic Search Topic
Topic Tools Topic Tools
View similar topics View similar topics
View topic in raw text format. Print this topic.
 09/15/2009 11:57 AM
User is offline View Users Profile Print this message

Author Icon
NamNgo
Expert (100-199)

Posts: 131
Joined: 02/10/2009

BPA 7.1 (Build 208261)

Windows Server 2003

On one of our test machines, we seemed to notice that the memory consumption of AMTS.exe only grows. So we decided to turn off everything except for a workflow that just calls a DB stored procedure every 30 seconds (the SProc just executes and completes without a return value). Anyway, here is one day's worth of memory tracking:

AMTS.exe (Memory Usage in KB) on 9/14/2009
- 11am:  206,432
- 11:02am:  206,480
- 11:10am:  206,588
- 12N:   207,196
- 12:38pm:  207,592
- 2:11pm:  208,976
- 5:07pm:  211,028

I was wondering, why is the memory usage so high in the first place?

Could you shed any light on why this occurs with this workflow? This workflow should not be memory intensive at all.



 Category Survey
 09/17/2009 12:19 PM
User is offline View Users Profile Print this message

Author Icon
NamNgo
Expert (100-199)

Posts: 131
Joined: 02/10/2009

I ended up turning off all of the workflows, and the memory usage still went up!

IDIAuto02 AMTS.exe (Memory Usage in KB) on 9/16/2009
- 10:56am:  223,688 (with no workflows running!)
- 11:12am:  223,744 (with no workflows running!)
- 11:29am:  223,824 (with no workflows running!)
- 11:42am:  223,872 (with no workflows running!)
- 1:07pm:  224,140 (with no workflows running!)
- 1:23pm:  224,188 (with no workflows running!)
- 1:55pm:  224,328 (with no workflows running!)
- 2:27pm:  224,456 (with no workflows running!)

IDIAuto02 AMTS.exe (Memory Usage in KB) on 9/17/2009
- 9:13am:  229,388 (with no workflows running!)
- 9:43am:  229,516 (with no workflows running!)
- 10:25am:  227,684 (with no workflows running!)
- 11:24am:  227,908 (with no workflows running!)
- 12:07pm:  228,104 (with no workflows running!)
- 12:37pm:  228,168 (with no workflows running!)
- 12:55pm:  228,252 (with no workflows running!)
- 2:56pm:  228,864 (with no workflows running!)

The reason why this is brought up is that if the memory consumption gets too high, both of the services stop and we get the associated error (see screenshot). Even if we have the Server and Agent services restart themselves after failure, it seems as though the Agent either doesn't fail or it doesn't properly communicate with the server after recovery. The only way to solve this is to manually restart the Agent AFTER the Server is- then it fails to connect (exactly like in this thread: http://forums.networkautomation.com/forum/messageview.cfm?catid=33&threadid=6270&enterthread=y 

And at the rate of use, we would have to manually go in and restart the services to clear out the memory at least once a week. But we don't want to manually go in and do anything like that, because we wanted to automate the entire process- hopefully not having to look at it for months at a time. 

Should AutoMate even be consuming more memory if all that is running are the services (no workflows enabled at all)?

Should AutoMate consume this much memory?

When does it release this memory, if ever?

Does the agent know when the server fails? Will it know to restart itself in the event of a server failure? If it does, does it ensure that the server is successfully done restarting before it tries to communicate with it?

How can we have the services restart automatically and have the agent properly communicating with the server (without having to manually go in and do it)?



 Category Survey
 09/17/2009 12:49 PM
User is offline View Users Profile Print this message

Author Icon
Hamid Noorbakhsh
HelpSystems

Posts: 454
Joined: 07/29/2008

Nam,
Could you please let me know what DataStore backend you are using and give some specifics on the set-up you have witho OS involved. e.g. Server on 2003 - Agent on XP, SP3 or Server/Agent on same machine 2008. If you are using another machine for DataStore, please include that information also.

Edit: Could you please also list the 'Provider' for the FileAutomation_Core connection string you have listed in your Task.

Thank you.

 Category Survey
AutoMate BPA Server 7 version:
Windows version:


-------------------------
Thank you for choosing AutoMate

Hamid Noorbakhsh | Senior Software Engineer
AutoMate | A Division of HelpSystems
HelpSystems, LLC | T: +1.213.738.1700
 09/17/2009 01:32 PM
User is offline View Users Profile Print this message

Author Icon
NamNgo
Expert (100-199)

Posts: 131
Joined: 02/10/2009

Machine 1: Windows 2003 Server with AM Server and AM Agent

Machine 2: SQL Server 2008 on Windows 2003 Server for our DataStore and DB for our business model.

Provider for our SQL Connection: SQLOLEDB.1

Hopefully that helps.



 Category Survey
 09/17/2009 03:30 PM
User is offline View Users Profile Print this message

Author Icon
Hamid Noorbakhsh
HelpSystems

Posts: 454
Joined: 07/29/2008

Nam,
We are currently looking into this and since this is a time-based issue, it takes a bit longer to investigate and hopefully we will have somethng for you soon.

Quick small questions in the meanwhile while we are testing this:
1- In SMC under 'options\Default Properties\Task Isolation' what setting are you using?
If you are using Medium, could you switch to High and see if you still get the same.
2- Your 'IDI_UpdateDatafeedQueueTask' in its Task Isolation Property is it using the Default setting?

Thanks for your time and patience.

 Category Survey
AutoMate BPA Server 7 version:
Windows version:


-------------------------
Thank you for choosing AutoMate

Hamid Noorbakhsh | Senior Software Engineer
AutoMate | A Division of HelpSystems
HelpSystems, LLC | T: +1.213.738.1700
 09/18/2009 10:19 AM
User is offline View Users Profile Print this message

Author Icon
NamNgo
Expert (100-199)

Posts: 131
Joined: 02/10/2009

1) We indeed had it set to Medium. I have changed it to High.

2) It was using the Default inherited Task Isolation Property.

After this I ended up just restarting the services to get fresh memory values. With all the workflows disabled, the memory doesn't go up any more.

However, I turned on one workflow (the minimal IDI_UpdateDatafeedQueueWorkflow) and it seems like the memory consumption rises again. It's only been running for about an hour, so I won't have numerical results of that until much later- but this is observable quicker if we turn on more workflows concurrently.



 Category Survey
 09/18/2009 11:25 AM
User is offline View Users Profile Print this message

Author Icon
Hamid Noorbakhsh
HelpSystems

Posts: 454
Joined: 07/29/2008

Nam,

After looking into the AMTS memory increase over time and more research (and a reminder from one of the developer that we had seen this before and had looked into it), I can report that slight increase over time is normal as it's the .NET Garbage Collector using COM. This memory creep is unavoidable as, according to Microsoft, it's normal. Eventually the memory will go back down as the system needs it or as the lifetime of the COM objects expires.

Please continue to report if this pattern is not followed.

 Category Survey
AutoMate BPA Server 7 version:
Windows version:


-------------------------
Thank you for choosing AutoMate

Hamid Noorbakhsh | Senior Software Engineer
AutoMate | A Division of HelpSystems
HelpSystems, LLC | T: +1.213.738.1700
 09/22/2009 01:37 PM
User is offline View Users Profile Print this message

Author Icon
NamNgo
Expert (100-199)

Posts: 131
Joined: 02/10/2009

When you say "slight increase over time" what might be an acceptable range of values for this increase over time?

I think we'll agree that if there are zero workflows running, memory should not increase at all (and it doesn't when nothing is running- of course this is useless).

If I was to have 3 workflows (2 that do nothing but call stored procedures in a DB that return no value, and one that basically continuously does many various file transfers) running, what the relatively absolute maximum that I should see AMTS.exe memory go up in a given time frame and be concerned? 10MB/day, 50MB/day, 100MB/day?



 Category Survey
 09/22/2009 02:00 PM
User is offline View Users Profile Print this message

Author Icon
Hamid Noorbakhsh
HelpSystems

Posts: 454
Joined: 07/29/2008

Nam,
I tried to mimic your situation last time and these are the results I got:

I monitored the memory for a bit under 16 hours with a workflow continously running the same stored procedure with no return value for every 30 seconds. Here is what I got after this period:

On a Quad Core Intel CPU with 8.0 GB memory running Vista Business 64-bit OS, the 'Working Set (Memory)' went up 81,192K and the Memory (Private Working Set) went up 80,540K.

This memory usage increase, per Microsoft as we reported to them and they confirmed it, is normal and it is released when the system demands more physical memory. On systems that have memory far beyond their normal usage, it seems like this memory increase is amplified since there is excess unused memory that is not reclaimed by the Garbage Collector until needed.

I hope this helps explain this increase as well as give you some numbers to compare.

 Category Survey
AutoMate BPA Server 7 version:
Windows version:


-------------------------
Thank you for choosing AutoMate

Hamid Noorbakhsh | Senior Software Engineer
AutoMate | A Division of HelpSystems
HelpSystems, LLC | T: +1.213.738.1700
 09/23/2009 10:12 AM
User is offline View Users Profile Print this message

Author Icon
NamNgo
Expert (100-199)

Posts: 131
Joined: 02/10/2009

I'm running 3 workflows. I'm taking the last 24 hour period I have recorded (yesterday around noon to today around noon) the AMTS memory usage and it goes from 166,552 KB to 202,652 KB.

Can that be considered normal? That seems like a rather large leap of memory usage in 24 hours. That's over a 1,500 KB/hour that it goes up and stays up. I have never seen AMTS memory usage drop without restarting the services (even if I stopped all the workflows- memory usage just stays the same).

Anyway, in v7.1.1, were there any changes to how the services (agent and server) restarted after failure/interruption and interacting with each other?



 Category Survey
 09/23/2009 10:33 AM
User is offline View Users Profile Print this message

Author Icon
Hamid Noorbakhsh
HelpSystems

Posts: 454
Joined: 07/29/2008

Nam,
Actually your memory usage did not increase as much as mine. Yours went up 36,100K in 24 hours vs my observation of about 80,000K in 16 hrs.
The amount of memory available to system plays a critical role. If you have worked with SQL server or Exchange 2007 (or you can read about their behavior), you will see that even if you have 100GB of memory, they will slowly claim all even when not doing anything. They will release it however as system needs it. I guess the point I want to make is that the amount of free memory is what will stop this climb and make it get reclaimed by the system, not time.

On "Anyway, in v7.1.1, were there any changes to how the services (agent and server) restarted after failure/interruption and interacting with each other?", I would have to say that we have not changed anything in that part.

 Category Survey
AutoMate BPA Server 7 version:
Windows version:


-------------------------
Thank you for choosing AutoMate

Hamid Noorbakhsh | Senior Software Engineer
AutoMate | A Division of HelpSystems
HelpSystems, LLC | T: +1.213.738.1700

Edited: 09/23/2009 at 10:51 AM by Hamid Noorbakhsh
 09/23/2009 01:31 PM
User is offline View Users Profile Print this message

Author Icon
NamNgo
Expert (100-199)

Posts: 131
Joined: 02/10/2009

All right then. It's just that the memory issue was the cause of this error that great halts the "automation" part of keeping the services on (which we seemed to get about once a week at current normal usage, but we planned on putting a bigger load on the system: about 3000% more). So, I guess my issue goes back to the real problem: how can we get this error to gracefully start the services again so that they can recognize each other?

So when I recieve the aforementioned error (JIT Debugger.jpg), it seems as though that only the server fails. Am I corrrect? I have both services (Server and Agent) set to "Restart upon failure." When this error occurs, it seems that the server just restarts itself (ServiceWasRestarted.jpg).

I have heard from your support specialists and engineers that the Agent should "automatically detect" the server again when it starts back up. 

Today as a test, I forced this error to occur very quickly (setting a complicated workflow to happened every 1 second) to test this in v7.1.1 and upon receiving this error, I checked everything. The services were running (server had just started itself, and yes I did refresh). If you look at ServicesReconnectError.jpg, I tried running the workflow IDI_UpdateDatafeedQueueWorkflow and it failed because "the agent is not connected." So, is there a way around this so that if this fails in our production environment that it will start back up by itself AND communicate with the Agent properly? Our IT department is having headaches trying to keep this thing running and won't implement more than a few jobs because this is a huge reliability liability for them and they can't always manually watch everything to make sure that it starts back up again- and we have business processes that require timely file transfers (down to the half hour).

Is there anything that can be done about this?



 Category Survey
 09/23/2009 01:47 PM
User is offline View Users Profile Print this message

Author Icon
Hamid Noorbakhsh
HelpSystems

Posts: 454
Joined: 07/29/2008

Nam,

A restart of service by Agent does not immediately connect it to server. Agent polls in short intervals to check to see if the server is up and running and then connect. It does the same if it gets disconnected.

I think since your workflow runs every 30 seconds it might catch this 'not connected' period and hence the error you see.

May I suggest you go to 'Options\Server Settings\Load Management' and turn on 'Do not run workflows after Server startup for a period of time' and set a reasonable time you see fit (look in SMC after server comes up to see how long it takes for agent to show connected under Agents tab, looking for change in icon). Then try your load test and see what the results are. Hopefully this will help to synchronize the connection so this problem does not occur.

 Category Survey
AutoMate BPA Server 7 version:
Windows version:


-------------------------
Thank you for choosing AutoMate

Hamid Noorbakhsh | Senior Software Engineer
AutoMate | A Division of HelpSystems
HelpSystems, LLC | T: +1.213.738.1700
 09/24/2009 10:35 AM
User is offline View Users Profile Print this message

Author Icon
NamNgo
Expert (100-199)

Posts: 131
Joined: 02/10/2009

I have attempted the experiment you proposed. I set it up for 5 minutes. That should be enough time for the server to be "up and ready."

I tried it and the server and agent seem to pick back up again and run workflows (however, I had trouble turning off the Agent service after that and it required a restart of the machine as the Agent would end up in a state of "Stopping" basically forever).

Then my co-worker brings up a good point: this still doesn't solve the underlying issue- which is the crash. Because if it crashes during a transfer, it will still fail, but we won't necessarily notice it failing immediately and timely files might not get sent- which is VERY, VERY bad.



 Category Survey
 09/24/2009 11:18 AM
User is offline View Users Profile Print this message

Author Icon
NamNgo
Expert (100-199)

Posts: 131
Joined: 02/10/2009

Repeated the experiment again and the Agent and Server didn't properly communicate with each other this time.



 Category Survey
 09/24/2009 03:12 PM
User is offline View Users Profile Print this message

Author Icon
Leonard Amabile
HelpSystems

Posts: 5454
Joined: 05/15/2007

Hi Nam,

Can you send an email to support@networkautomation.com with your availability? We would like to schedule a remote session with you to troubleshoot on this machine, if possible.

 Category Survey


-------------------------

Leonard Amabile | Director, Cross Platform Support
AutoMate | InterMapper | Skybot | Divisions of HelpSystems
T: +1.213.738.1700 | F: +1.213.738.7665
 02/09/2010 08:41 AM
User is offline View Users Profile Print this message

Author Icon
NamNgo
Expert (100-199)

Posts: 131
Joined: 02/10/2009

What ended up happening with this issue? I am experiencing it again in v7.1.2 and I can't find the trouble ticket that I knew I had opened. I couldn't even find this issue in the forum search. Luckily I had saved the forum link.



 Category Survey
 02/09/2010 09:27 PM
User is offline View Users Profile Print this message

Author Icon
Leonard Amabile
HelpSystems

Posts: 5454
Joined: 05/15/2007

Hi Nam,

I have gone through your tickets as well and have not found any documentation on this issue. Perhaps we got sidetracked with your other open tickets that this issue fell through the cracks. Let's schedule another remote session once we get your other tickets cleared and hopefully clear this issue as well.

 Category Survey


-------------------------

Leonard Amabile | Director, Cross Platform Support
AutoMate | InterMapper | Skybot | Divisions of HelpSystems
T: +1.213.738.1700 | F: +1.213.738.7665
Statistics
18257 users are registered to the AutoMate Discussion forum.
There are currently 0 users logged in.
The most users ever online was 6686 on 11/01/2020 at 01:24 AM.
There are currently 1218 guests browsing this forum, which makes a total of 1218 users using this forum.

FuseTalk Enterprise Edition v4.0 - © 1999-2021 FuseTalk Inc. All rights reserved.

Sitemap Network Automation Software Blog