Network Automation Software We are currently migrating this forum
over to our HelpSystems domain. Please
post all new threads in our new
HelpSystems Community Portal.
Post to the HelpSystems Forum
You are not currently logged on. You must be logged on in order to post. Log on
Or Create a new account
AutoMate Discussion
Decrease font size
Increase font size
Topic Title: Perpetual Memory Gain
Topic Summary: BPA executables continually growing in size until system failure
Created On: 08/27/2012 11:04 AM
Status: Read Only
Linear : Threading : Single : Branch
Search Topic Search Topic
Topic Tools Topic Tools
View similar topics View similar topics
View topic in raw text format. Print this topic.
 08/27/2012 11:04 AM
User is offline View Users Profile Print this message

Author Icon
BobMorke
Journeyman (50-99)

Posts: 73
Joined: 12/08/2010

AutoMate Version: BPA 8.0.5.1
Windows Version: Server 2008 R2 Enterprise 64-bit, SP1 -v6.1.7601
Error Message: None
Reproduction Instructions: Unknown

We recently changed/upgraded servers and we're running into a problem we've never seen before.

In the Windows Task Manager, we find that BPAS_MAN.exe and BPA_TS.exe just continue to grow over time. Upon server reboot, we find that the Memory usage on the server is around 1.6GB. Then, after a week, it's up around 6GB or 7GB until everything just stops working. (as that's closing in on the total 8GB of memory alotted)

When this happens, we find the only program that's taking an outrageous amount of memory is the BPAS_MAN.exe and BPA_TS.exe. When that service is stopped, or if the server is rebooted things go back to normal.

For example, the BPA_TS.exe will start out around 30,000k memory usage... after a few days, it's up to 790,000k.

Has anyone seen this before?


 Category Survey
 08/27/2012 02:08 PM
User is offline View Users Profile Print this message

Author Icon
Leonard Amabile
HelpSystems

Posts: 5454
Joined: 05/15/2007

Hi Bob,

Version 8.0.5.1 was never released publicly and I believe this may be a build you received from us when we were troubleshooting other issues at that time. Are you able to upgrade to v8.0.9.0 and see if the problem persists?




 Category Survey
AutoMate BPA Server 8 version:
Windows version:


-------------------------

Leonard Amabile | Director, Cross Platform Support
AutoMate | InterMapper | Skybot | Divisions of HelpSystems
T: +1.213.738.1700 | F: +1.213.738.7665
 08/28/2012 04:31 AM
User is offline View Users Profile Print this message

Author Icon
BobMorke
Journeyman (50-99)

Posts: 73
Joined: 12/08/2010

That may be difficult. There have been some recent changes that make an upgrade like that amazingly complex.

We ran this same version on an older, smaller, slower server and didn't have this problem at all. It's only since moving it to the better, newer server.

I can check to see how painful it might be to upgrade, but until then, I'm happy to hear any other theories about how this .exe can keep on growing.

 Category Survey
 08/29/2012 01:17 PM
User is offline View Users Profile Print this message

Author Icon
Ricardo Castaneda
HelpSystems

Posts: 3424
Joined: 01/11/2008

Hi Bob,

Please keep us informed if and when you are able to upgrade to version 8.0.9.0. If any help is needed to upgrade to that version please do not hesitate to ask.

 Category Survey


-------------------------

Ricardo Castaneda | Principal Support Analyst
AutoMate | Division of HelpSystems
HelpSystems, LLC | T: +1.213.738.1700
 09/10/2012 11:10 AM
User is offline View Users Profile Print this message

Author Icon
JeanDelfosse
Supreme Guru (1000+)

Posts: 8883
Joined: 03/30/2005

Bob,
you could restart BPA without rebooting, and hopefully get the same benefit.
For instance you could setup a task, launched periodically by Windows scheduler, that stops all the services and restart them one after the other.
The proper sequence to restart the services is 1-execution, 2-management, 3-Agent (EMA sequence)

 Category Survey
AutoMate BPA Server 8 version:
Windows version:


-------------------------
Jean Delfosse - Automate Beta Tester
Win7 Pro 64 bits -Win XP Pro SP3 32 bits - AM8, AM7, BPA10, BPA9, BPA8, BPA7, AM6 Enterprise, AM4
 09/11/2012 08:40 AM
User is offline View Users Profile Print this message

Author Icon
AndrewHoeschen
Expert (100-199)

Posts: 162
Joined: 11/03/2010

I don't believe this is sufficient, as it appears one of the processes that burns massive resources is LSASS.exe, which is used for windows security/authentication if memory serves, and this cannot be restarted.

I am a coworker of Bob's.

 Category Survey
 09/11/2012 08:42 AM
User is offline View Users Profile Print this message

Author Icon
AndrewHoeschen
Expert (100-199)

Posts: 162
Joined: 11/03/2010

I am trying to find where to download 8.0.9, not seeing it? When I view options under my account, which has our licenses/etc linked, I only see v9 files. If I go to the 8.0.9 page and click download, it goes back to my profile.

 Category Survey
 09/11/2012 08:46 AM
User is offline View Users Profile Print this message

Author Icon
AndrewHoeschen
Expert (100-199)

Posts: 162
Joined: 11/03/2010

I had upgraded our licenses to v9, as were contemplating an upgrade, so maybe that's why I cannot see the v8 downloads, even though I can still see the license codes?

 Category Survey
 09/11/2012 09:05 AM
User is offline View Users Profile Print this message

Author Icon
Leonard Amabile
HelpSystems

Posts: 5454
Joined: 05/15/2007

Hi Bob and Andrew,

Just to be clear, is it lsass.exe or the BPA processes that are consuming the memory? There are a lot of references in Microsoft's knowledgebase referencing lsass.exe issues:

http://support.microsoft.com/kb/971265
http://support.microsoft.com/kb/976618

http://wasthatsohard.wordpress...-2008-r2-file-server/




 Category Survey
AutoMate BPA Server 8 version:
Windows version:


-------------------------

Leonard Amabile | Director, Cross Platform Support
AutoMate | InterMapper | Skybot | Divisions of HelpSystems
T: +1.213.738.1700 | F: +1.213.738.7665
 09/11/2012 10:42 AM
User is offline View Users Profile Print this message

Author Icon
AndrewHoeschen
Expert (100-199)

Posts: 162
Joined: 11/03/2010

All of the processes are using much more memory than expected.

BPAS_MAN.exe and BPA_TS.exe are WAY higher than one would expect, enough to start crippling the machine.

LSASS.exe seems to go up with it to some extent, getting sometimes near 1GB of memory, when it usually uses <100MB.

 Category Survey
 09/11/2012 10:48 AM
User is offline View Users Profile Print this message

Author Icon
AndrewHoeschen
Expert (100-199)

Posts: 162
Joined: 11/03/2010

Your one article talks about NTLM authentication... there anything in BPA that uses NTLM?

 Category Survey
 09/12/2012 07:56 AM
User is offline View Users Profile Print this message

Author Icon
Leonard Amabile
HelpSystems

Posts: 5454
Joined: 05/15/2007

Hi Andrew,

BPA does not use NTLM. I gave those links as possible solutions. Do you have a lot of file trigger/conditions?




 Category Survey
AutoMate BPA Server 8 version:
Windows version:


-------------------------

Leonard Amabile | Director, Cross Platform Support
AutoMate | InterMapper | Skybot | Divisions of HelpSystems
T: +1.213.738.1700 | F: +1.213.738.7665
 09/12/2012 08:09 AM
User is offline View Users Profile Print this message

Author Icon
AndrewHoeschen
Expert (100-199)

Posts: 162
Joined: 11/03/2010

We have seven file watchers, one job under three of them, four are empty. We've not had great luck with those, so we don't use them thoroughly. We do have 52 time/date schedules though.

 Category Survey
 09/12/2012 03:59 PM
User is offline View Users Profile Print this message

Author Icon
Ricardo Castaneda
HelpSystems

Posts: 3424
Joined: 01/11/2008

Hi Andrew,

Can you please confirm what process is taking the memory?

As a test would it be possible to disable the file triggers?

 Category Survey


-------------------------

Ricardo Castaneda | Principal Support Analyst
AutoMate | Division of HelpSystems
HelpSystems, LLC | T: +1.213.738.1700
 02/05/2015 12:54 PM
User is offline View Users Profile Print this message

Author Icon
SamVarghese
Student (5-19 posts)

Posts: 7
Joined: 02/05/2015

Hello,

My name is Sam, I work with Andrew and Bob.  We are still having issues, but unfortunately, haven't had the time to troubleshoot them.  We were getting around this issue by rebooting the server every other day.  We are at a point where we can't reboot the server as frequently--so this memory leak issue has risen to the top of my list.

I have attached two snapshots of the proceses that are leaking memory:

BPA_TS.exe ~500mb

lsass.exe ~200mb

If you take a look, you will notice a lot of open handles that are tokens between the two processes. ~15,000 conn.  My suspicion is Automate is opening these conections and not reusing them or closing them.  Could you please take a look.

We are running 8.0.5.1, I realize there may be a push to upgrade, but doing so will cause more disruption than desired.



 Category Survey
AutoMate BPA Server 8 version:
Windows version: Windows 2008 R2


Click for full image
Click for larger image
Click for full image
Click for larger image

 02/05/2015 01:06 PM
User is offline View Users Profile Print this message

Author Icon
Leonard Amabile
HelpSystems

Posts: 5454
Joined: 05/15/2007

Sam,

There was never a general release of v8.0.5.1. Please check your version information again and update this thread. Unfortunately, there are no further maintenance updates for v8 outside of v8.0.9.0. There is not enough information to be able to provide any other troubleshooting procedures out side of updating to the latest version. The task service opening multiple handles could be due to the opening of multiple SQL connections that are never closed or there is a task that is not able to logon or unlock the workstation when executed. How many simultaneous tasks are running on the machine when you see this issue?

 Category Survey
AutoMate BPA Server 8 version:
Windows version:


-------------------------

Leonard Amabile | Director, Cross Platform Support
AutoMate | InterMapper | Skybot | Divisions of HelpSystems
T: +1.213.738.1700 | F: +1.213.738.7665
 02/05/2015 01:16 PM
User is offline View Users Profile Print this message

Author Icon
SamVarghese
Student (5-19 posts)

Posts: 7
Joined: 02/05/2015

Thanks for the prompt reply! We were given a a build by Network Automation to fix other issues that we were facing.

At any given time of the day, we are running anywher from 0 to 30 tasks.  Granted, when NOTHING is running we dont see any issue, but as soon as something runs the memory usage grows and and rarely settles.  We rebooted on monday where we started with 30mb of mem usage.  Now we are at 700 MB usage and growing.

As far as SQL connections and lock/unlock executions, is there a way to ensure sql connections and logons get closed?



 Category Survey
AutoMate BPA Server 8 version:
Windows version: Windows 2008 R2
 02/06/2015 02:36 PM
User is offline View Users Profile Print this message

Author Icon
Leonard Amabile
HelpSystems

Posts: 5454
Joined: 05/15/2007

Hi Sam,

Unfortunately, in order to determine if SQL connections are being closed is to edit all of your tasks that contain SQL connections and make sure there is an Open SQL Connection and Close SQL Connection step. This also applies to logon/unlock properties. These properties need to be visually verified.



 Category Survey
AutoMate BPA Server 8 version:
Windows version:


-------------------------

Leonard Amabile | Director, Cross Platform Support
AutoMate | InterMapper | Skybot | Divisions of HelpSystems
T: +1.213.738.1700 | F: +1.213.738.7665
 02/09/2015 09:01 AM
User is offline View Users Profile Print this message

Author Icon
SamVarghese
Student (5-19 posts)

Posts: 7
Joined: 02/05/2015

In Default Properties, under Logon, if we are running everything as a Background User, how do we ensure that these logon/unlock/logoff get accomplished?



 Category Survey
AutoMate BPA Server 8 version:
Windows version: Windows 2008 R2
 02/09/2015 03:51 PM
User is offline View Users Profile Print this message

Author Icon
Mary Manzano
HelpSystems

Posts: 280
Joined: 03/10/2014

Hello Sam,

 

If you have all your tasks set to background user there is no need for the ctrl alt delete. Only interactive tasks that are set to run as logged on user and log on/unlock when it’s logged off/unlocked.



 Category Survey
AutoMate BPA Server 8 version:
Windows version:


-------------------------
Mary Manzano | Technical Consultant
AutoMate | Division of HelpSystems
HelpSystems, LLC | T: +1.213.738.1700 | F: +1.213.738.7665
 02/10/2015 08:21 AM
User is offline View Users Profile Print this message

Author Icon
SamVarghese
Student (5-19 posts)

Posts: 7
Joined: 02/05/2015

Thanks for the reply Mary! 

I guess that means we still have nothing to try to troubleshoot this issue.  Since we are logging in via Background User, and we dont have any tasks that utilize SQL, we don't really know why we are making  > 30,000 tokens of connections from the agent through LSASS.exe to essentially the server itself.



 Category Survey
AutoMate BPA Server 8 version:
Windows version: Windows 2008 R2
 02/11/2015 03:47 PM
User is offline View Users Profile Print this message

Author Icon
Ricardo Castaneda
HelpSystems

Posts: 3424
Joined: 01/11/2008

Hi Sam,

Would it be possible to confirm what version of BPA Server 8 you are running? I would like to run through the change log and check for any outstanding issues.

You can right click on the A icon (in the sys tray) and select About.

 Category Survey


-------------------------

Ricardo Castaneda | Principal Support Analyst
AutoMate | Division of HelpSystems
HelpSystems, LLC | T: +1.213.738.1700
 02/11/2015 05:16 PM
User is offline View Users Profile Print this message

Author Icon
SamVarghese
Student (5-19 posts)

Posts: 7
Joined: 02/05/2015

I understand that this isn't a standard build.  It's Version 8.0.5.1 (see attached)



 Category Survey
AutoMate BPA Server 8 version:
Windows version: Windows 2008 R2


Click for full image
Click for larger image

 02/12/2015 12:45 PM
User is offline View Users Profile Print this message

Author Icon
Ricardo Castaneda
HelpSystems

Posts: 3424
Joined: 01/11/2008

Hi Sam,

I would highly suggest that you upgrade to the latest version of BPA Server 8. Would it be possible to do so?

 Category Survey


-------------------------

Ricardo Castaneda | Principal Support Analyst
AutoMate | Division of HelpSystems
HelpSystems, LLC | T: +1.213.738.1700
 02/12/2015 03:07 PM
User is offline View Users Profile Print this message

Author Icon
SamVarghese
Student (5-19 posts)

Posts: 7
Joined: 02/05/2015

HI Ricardo,

Thanks for the suggestion, unfortunately we cannot upgrade automate any further without having to modify the tasks and/or workflows.  Are there any other steps we could take?

 

Thanks



 Category Survey
AutoMate BPA Server 8 version:
Windows version: Windows 2008 R2
 02/13/2015 01:44 PM
User is offline View Users Profile Print this message

Author Icon
Alex Escalante
HelpSystems

Posts: 871
Joined: 02/26/2014

Hello Sam,

In a previous post, Leonard mentioned:

in order to determine if SQL connections are being closed (we need) to edit all of your tasks that contain SQL connections and make sure there is an Open SQL Connection and Close SQL Connection step.

Has this been verified?

 Category Survey


-------------------------

Alex Escalante | Support Analyst
AutoMate | Division of HelpSystems
HelpSystems, LLC | T: +1.213.738.1700 | F: +1.213.738.7665
 02/26/2015 07:12 AM
User is offline View Users Profile Print this message

Author Icon
SamVarghese
Student (5-19 posts)

Posts: 7
Joined: 02/05/2015

Hi Alex,

We have verified that all our SQL connections and Sharepoint sessions are being closed for each task.  Is there anything else we should check?



 Category Survey
AutoMate BPA Server 8 version:
Windows version: Windows 2008 R2
 02/27/2015 03:40 PM
User is offline View Users Profile Print this message

Author Icon
Ricardo Castaneda
HelpSystems

Posts: 3424
Joined: 01/11/2008

Hi Sam,

There isn't much more we can do at this point. I would highly suggest that you upgrade to the latest version of BPA 8 version 8.0.9.

 Category Survey


-------------------------

Ricardo Castaneda | Principal Support Analyst
AutoMate | Division of HelpSystems
HelpSystems, LLC | T: +1.213.738.1700
Statistics
18258 users are registered to the AutoMate Discussion forum.
There are currently 0 users logged in.
The most users ever online was 5551 on 01/08/2018 at 11:11 AM.
There are currently 1627 guests browsing this forum, which makes a total of 1627 users using this forum.

FuseTalk Enterprise Edition v4.0 - © 1999-2020 FuseTalk Inc. All rights reserved.

Sitemap Network Automation Software Blog