Network Automation Software We are currently migrating this forum
over to our HelpSystems domain. Please
post all new threads in our new
HelpSystems Community Portal.
Post to the HelpSystems Forum
You are not currently logged on. You must be logged on in order to post. Log on
Or Create a new account
AutoMate Discussion
Decrease font size
Increase font size
Topic Title: BPAS_EXEC v9.0.5.4 saturating the CPU
Topic Summary:
Created On: 06/11/2014 09:00 AM
Status: Read Only
Linear : Threading : Single : Branch
Search Topic Search Topic
Topic Tools Topic Tools
View similar topics View similar topics
View topic in raw text format. Print this topic.
Answer This question was answered by JeanDelfosse, on Monday, June 30, 2014 11:36 AM

Answer:
After three days with the patch, no CPU spike at all.
I therefore consider the fix is OK.
 06/11/2014 09:00 AM
User is offline View Users Profile Print this message

Author Icon
JeanDelfosse
Supreme Guru (1000+)

Posts: 8883
Joined: 03/30/2005

I came (I don't know how it happened) into a situation where BPAS_EXEC was saturating the CPU, by itself plus invoking services.exe, do that the system just got thrashing. BPA was doing nothing at all.
See process explorer picture 1

I had to kill the BPA9 execution server process (stopping the service did not work) to recover my system.
Then BPAS_EXEC restarted automatically and the situation became normal, with little BPAS_EXEC CPU, and the system fluid again.
See Process Explorer picture 2

This situation was discovered a few hours after a BPA_EXEC warning in Windows event log (attached below), and the very same day when BPA9.0.5.4 was installed.



 Category Survey
AutoMate BPA Server 9 version:
Windows version:



Task Steps: (Right click, Select All, Copy to copy to the clipboard. Steps can then be pasted into the AutoMate Task Builder)


-------------------------
Jean Delfosse - Automate Beta Tester
Win7 Pro 64 bits -Win XP Pro SP3 32 bits - AM8, AM7, BPA10, BPA9, BPA8, BPA7, AM6 Enterprise, AM4
 06/11/2014 11:42 AM
User is offline View Users Profile Print this message

Author Icon
JeanDelfosse
Supreme Guru (1000+)

Posts: 8883
Joined: 03/30/2005

After one or two hours, the system turned again into thrashing, 100% CPU, that could be resolved only by killing again BPAS_EXEC.
There is definitely a problem with 9.0.5.4 in this area.

 Category Survey
AutoMate BPA Server 9 version:
Windows version:


-------------------------
Jean Delfosse - Automate Beta Tester
Win7 Pro 64 bits -Win XP Pro SP3 32 bits - AM8, AM7, BPA10, BPA9, BPA8, BPA7, AM6 Enterprise, AM4
 06/12/2014 03:00 AM
User is offline View Users Profile Print this message

Author Icon
DanielGrip
Master (500-999)

Posts: 569
Joined: 11/25/2007

Jean, thats really bad news.
I'm having issues with this release as well.

Wondering if using v8 is the overall better option.

 Category Survey


-------------------------
Daniel Grip
 06/12/2014 05:15 AM
User is offline View Users Profile Print this message

Author Icon
JeanDelfosse
Supreme Guru (1000+)

Posts: 8883
Joined: 03/30/2005

Daniel,
I see in your other post that you get problems migrating from v8 to 9.0.5.4.
The CPU saturation I outlined here is after updating from 9.0.4.5 to 9.0.5.4

 Category Survey
AutoMate BPA Server 9 version:
Windows version:


-------------------------
Jean Delfosse - Automate Beta Tester
Win7 Pro 64 bits -Win XP Pro SP3 32 bits - AM8, AM7, BPA10, BPA9, BPA8, BPA7, AM6 Enterprise, AM4
 06/12/2014 07:48 AM
User is offline View Users Profile Print this message

Author Icon
Leonard Amabile
HelpSystems

Posts: 5454
Joined: 05/15/2007

Jean,

As you can see from the event log that we have built exception handling into v9.0.5. When the high CPU usage returned, did it result in similar error messages in the event log?




 Category Survey
AutoMate BPA Server 9 version:
Windows version:


-------------------------

Leonard Amabile | Director, Cross Platform Support
AutoMate | InterMapper | Skybot | Divisions of HelpSystems
T: +1.213.738.1700 | F: +1.213.738.7665


Edited: 06/12/2014 at 09:44 AM by Leonard Amabile
 06/12/2014 08:45 AM
User is offline View Users Profile Print this message

Author Icon
JeanDelfosse
Supreme Guru (1000+)

Posts: 8883
Joined: 03/30/2005

Leonard,
I don't quite understand what you ask me to look for.
But hereunder the events from last time it happened and I had to kill it.

 Category Survey
AutoMate BPA Server 9 version:
Windows version:



Task Steps: (Right click, Select All, Copy to copy to the clipboard. Steps can then be pasted into the AutoMate Task Builder)


-------------------------
Jean Delfosse - Automate Beta Tester
Win7 Pro 64 bits -Win XP Pro SP3 32 bits - AM8, AM7, BPA10, BPA9, BPA8, BPA7, AM6 Enterprise, AM4
 06/12/2014 09:52 AM
User is offline View Users Profile Print this message

Author Icon
Leonard Amabile
HelpSystems

Posts: 5454
Joined: 05/15/2007

Hi Jean,

What are the system specs for your machine?

1. CPU
2. RAM
3. Hard Drive space - total and available
4. Windows version



 Category Survey
AutoMate BPA Server 9 version:
Windows version:


-------------------------

Leonard Amabile | Director, Cross Platform Support
AutoMate | InterMapper | Skybot | Divisions of HelpSystems
T: +1.213.738.1700 | F: +1.213.738.7665
 06/12/2014 12:30 PM
User is offline View Users Profile Print this message

Author Icon
JeanDelfosse
Supreme Guru (1000+)

Posts: 8883
Joined: 03/30/2005

1. CPU Intel Core2 Duo T9600 (2x2.8 GHZ)
2. RAM 6GB
3. Hard Drive space - total 280GB and available 87GB
4. Windows version 7 Pro

The problem with BPAS_EXEC turns CPU to 100% for ever and the system gets very very slow. I did not try waiting a full day, because I need the system for other purposes, but always killed it after 15-20 minutes, always at 100% CPU.
I don't know which intent developers pursued in that phenomenon, but clearly it is not sustainable.

Killing BPAS_EXEC immediately returns the CPU to a regular 15-25%, even when BPAS_EXEC restarts spontaneously, until next time.


 Category Survey
AutoMate BPA Server 9 version:
Windows version:


-------------------------
Jean Delfosse - Automate Beta Tester
Win7 Pro 64 bits -Win XP Pro SP3 32 bits - AM8, AM7, BPA10, BPA9, BPA8, BPA7, AM6 Enterprise, AM4
 06/12/2014 12:59 PM
User is offline View Users Profile Print this message

Author Icon
Ricardo Castaneda
HelpSystems

Posts: 3424
Joined: 01/11/2008

Hi Jean,

I would like to turn on our Metrics reports. In order to do so you to create a file called "BPAS_EXEC.Debug.xml" and "BPASMAN_Debug_MetricsProfiling.xml" and place it in the in the following directory (C:\Program Files\AutoMate BPA Server 9). This will give us some insight on what's happening. Please note once you create the file you will need to restart your BPA server.

If you are running 64 bit it will be in the following location;

C:\Program Files (x86)\AutoMate BPA Server 9

The Output files will be in
C:\ProgramData\Network Automation\AutoMate BPA Server 9


Please complete this when you have some time to allow the CPU to spike for awhile.

We look forward to hear from you.

 Category Survey


-------------------------

Ricardo Castaneda | Principal Support Analyst
AutoMate | Division of HelpSystems
HelpSystems, LLC | T: +1.213.738.1700

Edited: 06/16/2014 at 09:41 AM by Ricardo Castaneda
 06/12/2014 09:23 PM
User is offline View Users Profile Print this message

Author Icon
JeanDelfosse
Supreme Guru (1000+)

Posts: 8883
Joined: 03/30/2005

Ricardo,
I created the files, and log is underway.
I stopped and restarted the services, but this time the A icon did not re-appear in the tray icon. SMC looks nice, but no triggering. BPAEM is missing.
I shall reboot.

EDIT AFTER REBOOT:
The A icon is still missing, and triggering does not work.
BPAEM did not start spontaneously upon reboot and I had to start it manually. Has this to do with the Debug_Metrics ?
After starting BPAEM, A icon is OK, and triggering as well.

 Category Survey
AutoMate BPA Server 9 version:
Windows version:


-------------------------
Jean Delfosse - Automate Beta Tester
Win7 Pro 64 bits -Win XP Pro SP3 32 bits - AM8, AM7, BPA10, BPA9, BPA8, BPA7, AM6 Enterprise, AM4


Edited: 06/12/2014 at 11:37 PM by JeanDelfosse
 06/13/2014 02:23 AM
User is offline View Users Profile Print this message

Author Icon
JeanDelfosse
Supreme Guru (1000+)

Posts: 8883
Joined: 03/30/2005

Ricardo,
I found at 10:50 the CPU at 100% (I don't know when it began the 100%).
I restarted BPAS_EXEC service at 10:56 then CPU is Ok by 25%
I attach the Metrics file covering the full span before/during/after the spike.
At first glance nothing in the file catches the eye.

 Category Survey
AutoMate BPA Server 9 version:
Windows version:


BPASMANMetrics.zip
BPASMANMetrics.zip  (63 KB)



-------------------------
Jean Delfosse - Automate Beta Tester
Win7 Pro 64 bits -Win XP Pro SP3 32 bits - AM8, AM7, BPA10, BPA9, BPA8, BPA7, AM6 Enterprise, AM4
 06/13/2014 04:21 AM
User is offline View Users Profile Print this message

Author Icon
JeanDelfosse
Supreme Guru (1000+)

Posts: 8883
Joined: 03/30/2005

The strange thing is that BPAS_EXEC by itself does not spike to high CPU (just 10-20% by doing nothing, which is anyhow too much) but some other system process(es) do, like PCDoctor, or svchost.exe, or services.exe, or some other one(s)of which Skype (!), not always the same, with some 30-40%, leaving no available capacity.
And when killing BPAS_EXEC, all those overhead/parasitic processes consumptions just vanish, until the mess returns some time later.
It looks like BPAS_EXEC process gets corrupt after some time and it induces strange churn in other system processes.
I had never seen that with 9.0.4.5

 Category Survey
AutoMate BPA Server 9 version:
Windows version:


-------------------------
Jean Delfosse - Automate Beta Tester
Win7 Pro 64 bits -Win XP Pro SP3 32 bits - AM8, AM7, BPA10, BPA9, BPA8, BPA7, AM6 Enterprise, AM4


Edited: 06/13/2014 at 07:56 PM by JeanDelfosse
 06/15/2014 12:03 PM
User is offline View Users Profile Print this message

Author Icon
JeanDelfosse
Supreme Guru (1000+)

Posts: 8883
Joined: 03/30/2005

To N/A:
if you have no more data to ask me, I shall switch back to 9.0.4.5 on Monday night.
9.0.5.4 is too painful due to the subject problem.

 Category Survey
AutoMate BPA Server 9 version:
Windows version:


-------------------------
Jean Delfosse - Automate Beta Tester
Win7 Pro 64 bits -Win XP Pro SP3 32 bits - AM8, AM7, BPA10, BPA9, BPA8, BPA7, AM6 Enterprise, AM4
 06/16/2014 08:09 AM
User is offline View Users Profile Print this message

Author Icon
BenjaminKorper
Artisan (200-499)

Posts: 231
Joined: 06/18/2009

Hi guys,

I'd like to bump this topic. I'm about to do a big rollout of 3 BPA server and several agents at a new customer and this would not be a good first encouter with BPA. Have you guys found the error yet?

Regards,

Benjamin

 Category Survey


-------------------------

 06/16/2014 08:31 AM
User is offline View Users Profile Print this message

Author Icon
JeanDelfosse
Supreme Guru (1000+)

Posts: 8883
Joined: 03/30/2005

Benjamin,
IMHO, if you still have 9.0.4.5 and your workflows are OK with it, I would advise that you go along with it.
You will at some point in time be able to update to some later and more reliable 9.0.5 release than the current ailing 9.0.5.4

 Category Survey
AutoMate BPA Server 9 version:
Windows version:


-------------------------
Jean Delfosse - Automate Beta Tester
Win7 Pro 64 bits -Win XP Pro SP3 32 bits - AM8, AM7, BPA10, BPA9, BPA8, BPA7, AM6 Enterprise, AM4
 06/16/2014 09:44 AM
User is offline View Users Profile Print this message

Author Icon
Leonard Amabile
HelpSystems

Posts: 5454
Joined: 05/15/2007

Jean,

Version 9.0.5 is a reliable release based on patches that have been provided to many customers over the past year. The issue is isolated to your installation at this point and we need to review your metric files as well as all other potential conflicts on your machine to determine what is causing the issue.



 Category Survey
AutoMate BPA Server 9 version:
Windows version:


-------------------------

Leonard Amabile | Director, Cross Platform Support
AutoMate | InterMapper | Skybot | Divisions of HelpSystems
T: +1.213.738.1700 | F: +1.213.738.7665
 06/16/2014 09:47 AM
User is offline View Users Profile Print this message

Author Icon
Leonard Amabile
HelpSystems

Posts: 5454
Joined: 05/15/2007

Originally posted by: BenjaminKorper

Hi guys,



I'd like to bump this topic. I'm about to do a big rollout of 3 BPA server and several agents at a new customer and this would not be a good first encouter with BPA. Have you guys found the error yet?



Regards,



Benjamin


Benjamin,

I would not recommend v9.0.4 for any future installations. Though Jean is experiencing issues, his issues are isolated. Version 9.0.5 is a roll-up of patches that were provided to v9.0.4 users. I strongly recommend v9.0.5 for your deployments.




 Category Survey
AutoMate BPA Server 9 version:
Windows version:


-------------------------

Leonard Amabile | Director, Cross Platform Support
AutoMate | InterMapper | Skybot | Divisions of HelpSystems
T: +1.213.738.1700 | F: +1.213.738.7665
 06/16/2014 11:07 AM
User is offline View Users Profile Print this message

Author Icon
JeanDelfosse
Supreme Guru (1000+)

Posts: 8883
Joined: 03/30/2005

Leonard,
so what ?
What can I do to help diagnose ?
I never have a stable 9.0.5.4 more than hour before it jeopardizes the system.

I attach the Metrics file covering the 3 last days

 Category Survey
AutoMate BPA Server 9 version:
Windows version:


BPASMAN_Metrics.zip
BPASMAN_Metrics.zip  (847 KB)



-------------------------
Jean Delfosse - Automate Beta Tester
Win7 Pro 64 bits -Win XP Pro SP3 32 bits - AM8, AM7, BPA10, BPA9, BPA8, BPA7, AM6 Enterprise, AM4


Edited: 06/16/2014 at 11:40 AM by JeanDelfosse
 06/16/2014 03:20 PM
User is offline View Users Profile Print this message

Author Icon
Leonard Amabile
HelpSystems

Posts: 5454
Joined: 05/15/2007

Hi Jean,

We have sent the metrics over to our development group for further research. Typically, major versions of our software, in a side by side environment, are installed from oldest to newest as far as version numbers. In my case, I installed v5, then v6, etc. I have version 5, 6, 7, 8 and 9 installed on my machine. If I need to update version 8, I will uninstall versions 9 and 8, then install version 8 and then version 9. At least in your case, you had v10 installed after a v9 installation. You then uninstalled v9 to install a newer version 9 but with v10 still installed. This might be a side by side issue with the ordering of uninstall and install. We are trying to reproduce this internally. In the meantime, I would recommend uninstalling both v9 and v10 and then install just v9 and let me know if the problem persists.





 Category Survey
AutoMate BPA Server 9 version:
Windows version:


-------------------------

Leonard Amabile | Director, Cross Platform Support
AutoMate | InterMapper | Skybot | Divisions of HelpSystems
T: +1.213.738.1700 | F: +1.213.738.7665
 06/16/2014 10:15 PM
User is offline View Users Profile Print this message

Author Icon
JeanDelfosse
Supreme Guru (1000+)

Posts: 8883
Joined: 03/30/2005

Leonard,
I will try that and let you know.

EDIT: June 17, 11:30
I uninstalled both BPA 9.0.5.4 and BPA10
Cleared folders in Program Files, and Registry keys.
Then installed BPA 9.0.5.4.
It persists that BPAEM does not start spontaneously and I had to do it manually.
I will watch BPA 9.0.5.4 behavior for one full day before installing again BPA10.

EDIT: June 17, 14:00
Now running BPA9.0.5.4 WITHOUT BPA10.
I just experienced again the 100% CPU syndrome, again "resolved" by killing BPAS_EXEC.
This seems demonstrating it would have nothing to do with BPA10 coexistence.
I reactivated the Metrics, and will submit it to you again after some time.

EDIT: June 17 4PM
I have subjectively the impression that BPA_EXEC malfunction begins on the occasion of some process starting (not as the result of any task but when I manually run some application).
I "felt" that this happened at times when I started one Firefox session, or one Outlook session, which are my most frequently used apps. Not each time, but sometimes. But it also happened while the system was idle (in my absence, but logged on).
I will try and correlate (just by feeling) and report when it happens again.

 Category Survey
AutoMate BPA Server 9 version:
Windows version:


-------------------------
Jean Delfosse - Automate Beta Tester
Win7 Pro 64 bits -Win XP Pro SP3 32 bits - AM8, AM7, BPA10, BPA9, BPA8, BPA7, AM6 Enterprise, AM4


Edited: 06/17/2014 at 07:20 AM by JeanDelfosse
 06/17/2014 09:52 PM
User is offline View Users Profile Print this message

Author Icon
JeanDelfosse
Supreme Guru (1000+)

Posts: 8883
Joined: 03/30/2005

System saturation due to BPAS_EXEC keeps happening even when the system is idle and unattended.
I cannot keep it going. I switch back to BPA 9.0.4.5
I attach the latest BPASMAN_Metrics file.

My last contribution on 9.0.5.4 is the following :
After some time of normal operation, BPA 9.0.5.4 looks like tying undue links to some other components of the system, and then induces irrelevant activity that increases the CPU consumption of BPA_EXEC itself as well as of some other components. This leads to 100% CPU consumption and unresponsiveness of the system.
Killing BPAS_EXEC immediately corrects the problem, even when BPAS_EXEC gets restarted, for less than one hour and the problem happens again.

 Category Survey
AutoMate BPA Server 9 version:
Windows version:





-------------------------
Jean Delfosse - Automate Beta Tester
Win7 Pro 64 bits -Win XP Pro SP3 32 bits - AM8, AM7, BPA10, BPA9, BPA8, BPA7, AM6 Enterprise, AM4


Edited: 06/17/2014 at 10:15 PM by JeanDelfosse
 06/18/2014 04:38 AM
User is offline View Users Profile Print this message

Author Icon
RobertoTopo
Student (5-19 posts)

Posts: 16
Joined: 07/09/2008

We have the same problem.
After some time of normal operation, BPA 9.0.5.4 induces activity that increases the CPU consumption.
Killing BPAS_EXEC immediately corrects the problem.

I switch back to BPA 9.0.4.5 at Server and 22 agent.

I attach the CPU/past month % usage report.
you can see the 3 restart of past week.

i'm waiting new (stable) version.

Regard,
Roberto


 Category Survey


Click for full image
Click for larger image

 06/18/2014 09:08 AM
User is offline View Users Profile Print this message

Author Icon
JeanDelfosse
Supreme Guru (1000+)

Posts: 8883
Joined: 03/30/2005

Roberto,
I commiserate, but I'm anyhow happy somebody else did the same finding.
I was feeling lonely vs N/A's denial.

BTW, which tool do you use to produce the nice chart you attached ?

 Category Survey
AutoMate BPA Server 9 version:
Windows version:


-------------------------
Jean Delfosse - Automate Beta Tester
Win7 Pro 64 bits -Win XP Pro SP3 32 bits - AM8, AM7, BPA10, BPA9, BPA8, BPA7, AM6 Enterprise, AM4
 06/18/2014 09:55 AM
User is offline View Users Profile Print this message

Author Icon
Michael Priddy
Artisan (200-499)

Posts: 256
Joined: 09/09/2005

So what is the consesus here? Is this release stable, unstable, sometimes stable or something else?

Your findings Jean and Roberto are unsettling to say the least.
While I did not encounter any issues during my testing, I am preparing to deploy to production.

Can I get a clear answer from the user community as well as N/A please?

Jean and Roberto, what would you advice?
N/A what would be your argument to proceed with the update?

Thank you!

 Category Survey
AutoMate BPA Server 9 version: 9.0.5
Windows version: Windows Server 2008 R2


-------------------------
Michael Priddy - AutoMate Beta Tester
Windows Server 2003 / XP SP3, 7/ 8/10 Server 2008/2012/2014
BPA Version : 10.5.0.56
 06/18/2014 11:24 AM
User is offline View Users Profile Print this message

Author Icon
JeanDelfosse
Supreme Guru (1000+)

Posts: 8883
Joined: 03/30/2005

Michael,
I'm not entitled to give you a "clear answer".
And I am afraid that so far nobody can give a 100% secure position, since instability just begins to surface.
My own position remains the one I gave earlier in this thread to Benjamin Korper : wait.
Nobody has yet found the root cause of that instability.

 Category Survey
AutoMate BPA Server 9 version:
Windows version:


-------------------------
Jean Delfosse - Automate Beta Tester
Win7 Pro 64 bits -Win XP Pro SP3 32 bits - AM8, AM7, BPA10, BPA9, BPA8, BPA7, AM6 Enterprise, AM4
 06/19/2014 03:08 AM
User is offline View Users Profile Print this message

Author Icon
RobertoTopo
Student (5-19 posts)

Posts: 16
Joined: 07/09/2008

Originally posted by: JeanDelfosse


BTW, which tool do you use to produce the nice chart you attached ?



Jean,

The server is a virtual machine.
Chart tool is a embedded features of VMWare platform.



 Category Survey
 06/19/2014 03:09 AM
User is offline View Users Profile Print this message

Author Icon
RobertoTopo
Student (5-19 posts)

Posts: 16
Joined: 07/09/2008

Michael,

if you didn't encounter any issues, i wish every success to deploy to production.


 Category Survey
 06/19/2014 04:03 AM
User is offline View Users Profile Print this message

Author Icon
JeanDelfosse
Supreme Guru (1000+)

Posts: 8883
Joined: 03/30/2005

Thank you, Roberto. I wish there is something similar in Windows 7.
That may explain the "smooth" ramping up of the CPU, while mine goes straight from 20% to 100% in a couple of seconds.

 Category Survey
AutoMate BPA Server 9 version:
Windows version:


-------------------------
Jean Delfosse - Automate Beta Tester
Win7 Pro 64 bits -Win XP Pro SP3 32 bits - AM8, AM7, BPA10, BPA9, BPA8, BPA7, AM6 Enterprise, AM4
 06/19/2014 08:28 AM
User is offline View Users Profile Print this message

Author Icon
Ricardo Castaneda
HelpSystems

Posts: 3424
Joined: 01/11/2008

Originally posted by: RobertoTopo

Michael,



if you didn't encounter any issues, i wish every success to deploy to production.


Hi Robert,

Are you running BPA Server 10 and BPA Server 9 side by side?



 Category Survey


-------------------------

Ricardo Castaneda | Principal Support Analyst
AutoMate | Division of HelpSystems
HelpSystems, LLC | T: +1.213.738.1700
 06/19/2014 01:57 PM
User is offline View Users Profile Print this message

Author Icon
DeborahWells
Journeyman (50-99)

Posts: 63
Joined: 07/19/2011

I echo the concern over this. We are also looking at a very visible promotion of the 9.05 version into production this weekend. We were excited to finally be 'current', which is always the first thing N/A asks if you encounter a problem; however, I am increasingly uncomfortable based on this thread. We did not encounter any issues in our TEST environment, but then again our TEST environment does not have the full volume or load of our production environment.

N/A - do you have any type of conclusions based on Jean's metrics? We have to make a go 9.05/go 9.04 decision tomorrow.

Thanks.


 Category Survey
AutoMate BPA Server 9 version: 9.0.3
Windows version: Windows Server 2003 R2
 06/19/2014 11:55 PM
User is offline View Users Profile Print this message

Author Icon
RobertoTopo
Student (5-19 posts)

Posts: 16
Joined: 07/09/2008

Originally posted by: Ricardo Castaneda (N.A.)

Hi Robert,


Are you running BPA Server 10 and BPA Server 9 side by side?


Runs only version 9.

v10 is a beta version.
We have an important platform in production, we can not consider release beta or too young. Normally expect at least two-three months before considering the upgrade.

 Category Survey
 06/20/2014 08:16 AM
User is offline View Users Profile Print this message

Author Icon
Ricardo Castaneda
HelpSystems

Posts: 3424
Joined: 01/11/2008

Hello,

We are currently working on reproducing this issue internally.

Does anyone have exclusions set on their workflows?

 Category Survey


-------------------------

Ricardo Castaneda | Principal Support Analyst
AutoMate | Division of HelpSystems
HelpSystems, LLC | T: +1.213.738.1700
 06/20/2014 09:53 AM
User is offline View Users Profile Print this message

Author Icon
Leonard Amabile
HelpSystems

Posts: 5454
Joined: 05/15/2007

Roberto & Jean,

Please check the thread count for BPAS_EXEC in Task Manager when the CPU spikes again.




 Category Survey
AutoMate BPA Server 9 version:
Windows version:


-------------------------

Leonard Amabile | Director, Cross Platform Support
AutoMate | InterMapper | Skybot | Divisions of HelpSystems
T: +1.213.738.1700 | F: +1.213.738.7665
 06/20/2014 10:41 AM
User is offline View Users Profile Print this message

Author Icon
JeanDelfosse
Supreme Guru (1000+)

Posts: 8883
Joined: 03/30/2005

Leonard,
I'm afraid I can't because I reverted to 9.0.4.5
But every time I had to kill BPAS_EXEC 9.0.5.4, the process was alone of its kind.

 Category Survey
AutoMate BPA Server 9 version:
Windows version:


-------------------------
Jean Delfosse - Automate Beta Tester
Win7 Pro 64 bits -Win XP Pro SP3 32 bits - AM8, AM7, BPA10, BPA9, BPA8, BPA7, AM6 Enterprise, AM4
 06/20/2014 10:42 AM
User is offline View Users Profile Print this message

Author Icon
JeanDelfosse
Supreme Guru (1000+)

Posts: 8883
Joined: 03/30/2005

Ricardo,
I don't have any exclusion on my workflows.

 Category Survey
AutoMate BPA Server 9 version:
Windows version:


-------------------------
Jean Delfosse - Automate Beta Tester
Win7 Pro 64 bits -Win XP Pro SP3 32 bits - AM8, AM7, BPA10, BPA9, BPA8, BPA7, AM6 Enterprise, AM4
 06/23/2014 12:05 AM
User is offline View Users Profile Print this message

Author Icon
RobertoTopo
Student (5-19 posts)

Posts: 16
Joined: 07/09/2008

Leonard,
Even for me the same thing about Jean.
Now running 9.0.4.5
BPAS_EXEC process was one alone.

 Category Survey
 06/23/2014 09:47 AM
User is offline View Users Profile Print this message

Author Icon
Leonard Amabile
HelpSystems

Posts: 5454
Joined: 05/15/2007

Originally posted by: RobertoTopo

We have the same problem.

After some time of normal operation, BPA 9.0.5.4 induces activity that increases the CPU consumption.

Killing BPAS_EXEC immediately corrects the problem.



I switch back to BPA 9.0.4.5 at Server and 22 agent.



I attach the CPU/past month % usage report.

you can see the 3 restart of past week.



i'm waiting new (stable) version.



Regard,

Roberto


Roberto,

Please post your Windows event logs from the CPU spike time periods.



 Category Survey
AutoMate BPA Server 9 version:
Windows version:


-------------------------

Leonard Amabile | Director, Cross Platform Support
AutoMate | InterMapper | Skybot | Divisions of HelpSystems
T: +1.213.738.1700 | F: +1.213.738.7665
 06/23/2014 03:34 PM
User is offline View Users Profile Print this message

Author Icon
Leonard Amabile
HelpSystems

Posts: 5454
Joined: 05/15/2007

Roberto & Jean,

Attached is a patch .dll file that I would like you both to test in your 9.0.5 installations to verify if the issue persists. Please let me know as soon as possible if this corrects the issue.

After installation, shutdown the BPA9 services and overwrite the AutoMate.BPAServer.Execution.dll in the installation directory with the one in the .zip file. Restart the services and use the application.




 Category Survey
AutoMate BPA Server 9 version:
Windows version:





-------------------------

Leonard Amabile | Director, Cross Platform Support
AutoMate | InterMapper | Skybot | Divisions of HelpSystems
T: +1.213.738.1700 | F: +1.213.738.7665
 06/24/2014 08:21 AM
User is offline View Users Profile Print this message

Author Icon
Michael Priddy
Artisan (200-499)

Posts: 256
Joined: 09/09/2005

What specific issues does this dll address Leonard? Is there a contention somewhere?

Thank you!

 Category Survey
AutoMate BPA Server 9 version: 9.0.5
Windows version: Windows Server 2008 R2


-------------------------
Michael Priddy - AutoMate Beta Tester
Windows Server 2003 / XP SP3, 7/ 8/10 Server 2008/2012/2014
BPA Version : 10.5.0.56
 06/24/2014 08:39 AM
User is offline View Users Profile Print this message

Author Icon
JeanDelfosse
Supreme Guru (1000+)

Posts: 8883
Joined: 03/30/2005

Leonard
I'm out of office for a few days.
Hopefully I can try the patch next Friday before being caught by the expiration date of June 1st.

 Category Survey
AutoMate BPA Server 9 version:
Windows version:


-------------------------
Jean Delfosse - Automate Beta Tester
Win7 Pro 64 bits -Win XP Pro SP3 32 bits - AM8, AM7, BPA10, BPA9, BPA8, BPA7, AM6 Enterprise, AM4
 06/24/2014 08:58 AM
User is offline View Users Profile Print this message

Author Icon
Leonard Amabile
HelpSystems

Posts: 5454
Joined: 05/15/2007

Originally posted by: Michael Priddy

What specific issues does this dll address Leonard? Is there a contention somewhere?



Thank you!


Hi Michael,

There may be a threading issue with BPAS_EXEC.EXE. If you are running BPA9 v9.0.5, I recommend this file for you as well. I want to make sure that this addresses the CPU spikes as well. If the spikes are still present then something else is going on that we need to continue to investigate.







 Category Survey
AutoMate BPA Server 9 version:
Windows version:


-------------------------

Leonard Amabile | Director, Cross Platform Support
AutoMate | InterMapper | Skybot | Divisions of HelpSystems
T: +1.213.738.1700 | F: +1.213.738.7665
 06/24/2014 08:59 AM
User is offline View Users Profile Print this message

Author Icon
Leonard Amabile
HelpSystems

Posts: 5454
Joined: 05/15/2007

Originally posted by: JeanDelfosse

Leonard

I'm out of office for a few days.

Hopefully I can try the patch next Friday before being caught by the expiration date of June 1st.


Hi Jean,

Version 10 expires on June 30th and should not affect version 9's functionality.





 Category Survey
AutoMate BPA Server 9 version:
Windows version:


-------------------------

Leonard Amabile | Director, Cross Platform Support
AutoMate | InterMapper | Skybot | Divisions of HelpSystems
T: +1.213.738.1700 | F: +1.213.738.7665
 06/24/2014 10:28 AM
User is offline View Users Profile Print this message

Author Icon
Michael Priddy
Artisan (200-499)

Posts: 256
Joined: 09/09/2005

Sounds like a good idea. I will deploy this to Test today.

 Category Survey
AutoMate BPA Server 9 version: 9.0.5
Windows version: Windows Server 2008 R2


-------------------------
Michael Priddy - AutoMate Beta Tester
Windows Server 2003 / XP SP3, 7/ 8/10 Server 2008/2012/2014
BPA Version : 10.5.0.56
 06/25/2014 03:27 AM
User is offline View Users Profile Print this message

Author Icon
JeanDelfosse
Supreme Guru (1000+)

Posts: 8883
Joined: 03/30/2005

You are right Leonard.
Sorry, my mistake.

 Category Survey
AutoMate BPA Server 9 version:
Windows version:


-------------------------
Jean Delfosse - Automate Beta Tester
Win7 Pro 64 bits -Win XP Pro SP3 32 bits - AM8, AM7, BPA10, BPA9, BPA8, BPA7, AM6 Enterprise, AM4
 06/26/2014 12:04 PM
User is offline View Users Profile Print this message

Author Icon
JeanDelfosse
Supreme Guru (1000+)

Posts: 8883
Joined: 03/30/2005

I uninstalled 9.0.4.5
Installed 9.0.5.4
Substituted AutoMate.BPAServer.Execution.dll with the patch
Stopped and restarted the services.
Everything is working.
Waiting and seeing if the CPU problem comes back.
Will report findings in the coming days.

 Category Survey
AutoMate BPA Server 9 version:
Windows version:


-------------------------
Jean Delfosse - Automate Beta Tester
Win7 Pro 64 bits -Win XP Pro SP3 32 bits - AM8, AM7, BPA10, BPA9, BPA8, BPA7, AM6 Enterprise, AM4
 06/26/2014 12:29 PM
User is offline View Users Profile Print this message

Author Icon
Ricardo Castaneda
HelpSystems

Posts: 3424
Joined: 01/11/2008

Hi Jean,

We look forward to hear from you.

 Category Survey


-------------------------

Ricardo Castaneda | Principal Support Analyst
AutoMate | Division of HelpSystems
HelpSystems, LLC | T: +1.213.738.1700
 06/26/2014 12:36 PM
User is offline View Users Profile Print this message

Author Icon
BenjaminKorper
Artisan (200-499)

Posts: 231
Joined: 06/18/2009

Great work Jean! I've also installed 9.0.5.4 (fresh install) at one of my customers. I will apply the patch if neccessary.

 Category Survey


-------------------------

 06/27/2014 05:40 AM
User is offline View Users Profile Print this message

Author Icon
JeanDelfosse
Supreme Guru (1000+)

Posts: 8883
Joined: 03/30/2005

Ricardo,
so far so good.
Patch applied 20 hours ago.
No CPU spike so far, with my regular activity. And a few Reboots and hibernates independent of BPA.
Before the patch it never lasted more than one hour without the CPU spiking. Overall base CPU now remains in the 15-25% range.
I will keep reporting every day if stiil OK, earlier in case it spikes again. Wish you have solved it !

 Category Survey
AutoMate BPA Server 9 version:
Windows version:


-------------------------
Jean Delfosse - Automate Beta Tester
Win7 Pro 64 bits -Win XP Pro SP3 32 bits - AM8, AM7, BPA10, BPA9, BPA8, BPA7, AM6 Enterprise, AM4
 06/27/2014 06:31 AM
User is offline View Users Profile Print this message

Author Icon
BenjaminKorper
Artisan (200-499)

Posts: 231
Joined: 06/18/2009

Hi Jean,

Is that CPU level normal for your machine? Did the latest version impact CPU usage a lot?

Thanks,

Benjamin

 Category Survey


-------------------------

 06/27/2014 07:27 AM
User is offline View Users Profile Print this message

Author Icon
JeanDelfosse
Supreme Guru (1000+)

Posts: 8883
Joined: 03/30/2005

Benjamin,
that is the normal CPU level on my system, due to many servers (outside BPA) running on it 24/24 7/7. Some 120 services running.
With BPA 9.0.5.4 patched, BPA processes (when idle, no task running) typically consume about 5% in total, unchanged vs 9.0.4.5.


 Category Survey
AutoMate BPA Server 9 version:
Windows version:


-------------------------
Jean Delfosse - Automate Beta Tester
Win7 Pro 64 bits -Win XP Pro SP3 32 bits - AM8, AM7, BPA10, BPA9, BPA8, BPA7, AM6 Enterprise, AM4
 06/30/2014 11:36 AM
User is offline View Users Profile Print this message

Author Icon
JeanDelfosse
Supreme Guru (1000+)

Posts: 8883
Joined: 03/30/2005

Answer Answer
After three days with the patch, no CPU spike at all.
I therefore consider the fix is OK.

 Category Survey
AutoMate BPA Server 9 version:
Windows version:


-------------------------
Jean Delfosse - Automate Beta Tester
Win7 Pro 64 bits -Win XP Pro SP3 32 bits - AM8, AM7, BPA10, BPA9, BPA8, BPA7, AM6 Enterprise, AM4
 07/01/2014 12:01 AM
User is offline View Users Profile Print this message

Author Icon
RobertoTopo
Student (5-19 posts)

Posts: 16
Joined: 07/09/2008

Good news.
Patch applied yesterday morning, so far so good.
I attach the CPU usage report.

Thanks,
Roberto

 Category Survey


Click for full image
Click for larger image

 07/01/2014 12:53 AM
User is offline View Users Profile Print this message

Author Icon
JeanDelfosse
Supreme Guru (1000+)

Posts: 8883
Joined: 03/30/2005

Great, Roberto ! Happy it also works for you.
We may never know what your and my cases had so special, but the important thing is that it is now OK.

 Category Survey
AutoMate BPA Server 9 version:
Windows version:


-------------------------
Jean Delfosse - Automate Beta Tester
Win7 Pro 64 bits -Win XP Pro SP3 32 bits - AM8, AM7, BPA10, BPA9, BPA8, BPA7, AM6 Enterprise, AM4
 09/09/2015 11:56 AM
User is offline View Users Profile Print this message

Author Icon
AdamConner
Journeyman (50-99)

Posts: 55
Joined: 02/20/2014

First, I just want to say that the .DLL patch resolved high CPU I saw in v9.0.5.4 as well. Thanks Leonard.

Separately, while I know this is an older post I did note Jean's comment about BPAEM.exe not starting automatically in v9.0.5.4 and did not see a direct response.

I ran into the same issue after updating from v9.0.4.5 to v9.0.5.4 but didn't encounter the issue on ALL of my Agent servers. What I found was that BPAEM.exe started fine on servers that had both the Agent AND Dev Tools installed ... while servers with JUST the Agent installed would (as Jean said) not start up automatically.

For comparison, all Agent's were installed manually on the server (rather than deployed from SMC) using the main BPA Server installer. I verified that installing the Dev Tools with the Agent on servers that BPAEM.exe wouldn't automatically start on DID resolve the startup issue (even though Dev Tools couldn't be used ... I removed the program icon later from the Start Menu to avoid confusion for other users). So ... I've accepted this as my BPAEM.exe startup fix for now as it resolved the issue without any unwanted side effects.

 

Originally posted by: JeanDelfosse Leonard, I will try that and let you know. EDIT: June 17, 11:30 I uninstalled both BPA 9.0.5.4 and BPA10 Cleared folders in Program Files, and Registry keys. Then installed BPA 9.0.5.4. It persists that BPAEM does not start spontaneously and I had to do it manually. I will watch BPA 9.0.5.4 behavior for one full day before installing again BPA10. EDIT: June 17, 14:00 Now running BPA9.0.5.4 WITHOUT BPA10. I just experienced again the 100% CPU syndrome, again "resolved" by killing BPAS_EXEC. This seems demonstrating it would have nothing to do with BPA10 coexistence. I reactivated the Metrics, and will submit it to you again after some time. EDIT: June 17 4PM I have subjectively the impression that BPA_EXEC malfunction begins on the occasion of some process starting (not as the result of any task but when I manually run some application). I "felt" that this happened at times when I started one Firefox session, or one Outlook session, which are my most frequently used apps. Not each time, but sometimes. But it also happened while the system was idle (in my absence, but logged on). I will try and correlate (just by feeling) and report when it happens again.



 Category Survey
AutoMate BPA Server 9 version: 9.0.5
Windows version: Windows Server 2008 R2


Edited: 09/09/2015 at 12:06 PM by AdamConner
Statistics
18258 users are registered to the AutoMate Discussion forum.
There are currently 0 users logged in.
The most users ever online was 5551 on 01/08/2018 at 11:11 AM.
There are currently 947 guests browsing this forum, which makes a total of 947 users using this forum.

FuseTalk Enterprise Edition v4.0 - © 1999-2020 FuseTalk Inc. All rights reserved.

Sitemap Network Automation Software Blog