Network Automation Software We are currently migrating this forum
over to our HelpSystems domain. Please
post all new threads in our new
HelpSystems Community Portal.
Post to the HelpSystems Forum
You are not currently logged on. You must be logged on in order to post. Log on
Or Create a new account
AutoMate Discussion
Decrease font size
Increase font size
Topic Title: Strange behaviour after BPA Server upgrade to version 7.1.3.0
Topic Summary:
Created On: 03/06/2011 08:34 PM
Status: Read Only
Linear : Threading : Single : Branch
Search Topic Search Topic
Topic Tools Topic Tools
View similar topics View similar topics
View topic in raw text format. Print this topic.
 03/06/2011 08:34 PM
User is offline View Users Profile Print this message

Author Icon
MartynLeung
n00b (>6 posts)

Posts: 1
Joined: 03/06/2011

We recently upgraded our BPA server from version 7.0.7.0 to 7.1.3.0 and are noticing some strange behaviour on some of our tasks.

The task has not been modified and has been working for a couple of weeks then all of a sudden it fails.  On further investigation, I noticed that a variable was changed and had the number 1 appended to the end of it.  In other tasks, I have seen a semi-colon inserted into the condition in an IF expression!

In another task, an FTP upload, there was a leading space for the destination path and hence the file was not uploaded to the FTP server.

Has anyone encountered any strange issues after upgrading to BPA Server 7.1.3.0?



 Category Survey
 03/07/2011 09:23 AM
User is offline View Users Profile Print this message

Author Icon
Rick Johnson
AUTOMATE EMPLOYEE

Posts: 2757
Joined: 07/14/2008

Hi Martyn,

It is not expected and I think also not previously encountered by other users.

It is also strange that workflows were runing as expected for some time then the behavior changed.

Can you post any before/after examples of your step changes?

 Category Survey


-------------------------
- Rick Johnson
Network Automation
 03/09/2011 06:52 PM
User is offline View Users Profile Print this message

Author Icon
KarinaC
n00b (>6 posts)

Posts: 4
Joined: 04/13/2008

I have just upgraded the Automate BPA from 7.0.7 to 7.1.3. Before upgrading to 7.1.3, I dont have any problems with the schedulers . After upgraded to 7.1.3, I found few problems on some of the jobs arent executed where it supposes.

I have enabled the Execution status so I can watch the Workflow. I found some of the jobs are stuck. The jobs are scheduled to run every 15 minutes. On some random time, the jobs can get stuck without any reasons. I have tried to put logging i.e when the job runs, write a file. The file has never written which I assume the job has never been executed.

Any other people experience this problems? any bugs with this version?

Thanks.



 Category Survey
 03/10/2011 08:40 AM
User is offline View Users Profile Print this message

Author Icon
Rick Johnson
AUTOMATE EMPLOYEE

Posts: 2757
Joined: 07/14/2008

Hi Karina,

Are there any BPA errors listed in the Windows Event Viewer? If there are could you copy/paste them into a reply?

A few additional troubleshooting questions:
    How many jobs are running at once?
    How long do they usually take to complete?
    How long has BPA been up and running?


 Category Survey


-------------------------
- Rick Johnson
Network Automation
 03/10/2011 02:38 PM
User is offline View Users Profile Print this message

Author Icon
KarinaC
n00b (>6 posts)

Posts: 4
Joined: 04/13/2008

Hi Rick, there are no BPA errors listed in the windows event viewer. Around 25 to 40 jobs that can run at once. Most of them are FTP jobs... so they normally finish averaging around 1 to 2 minutes. The BPA server always get restarted every weekend.

When I look the logs in "C:\Documents and Settings\All Users\Application Data\Network Automation\AutoMate BPA Agent" - the job that stuck are never written into the logs. So I assume the job has never been executed.



 Category Survey
 03/10/2011 04:50 PM
User is offline View Users Profile Print this message

Author Icon
Rick Johnson
AUTOMATE EMPLOYEE

Posts: 2757
Joined: 07/14/2008

Hi Karina,

Confirming that you set the Logging Output (from Preferences under Options) to show all events and checked the box to show triggers for disbled workflows.

Does the server experience any problem-free operation after a reboot, and if so how long until problems begin?
Does the problem only appear to be affecting specific workflows, or do the stuck schedules not follow any consistent pattern?

 Category Survey


-------------------------
- Rick Johnson
Network Automation
 03/10/2011 07:25 PM
User is offline View Users Profile Print this message

Author Icon
KarinaC
n00b (>6 posts)

Posts: 4
Joined: 04/13/2008

Rick,

Confirming that you set the Logging Output (from Preferences under Options) to show all events and checked the box to show triggers for disabled workflows.  (THIS IS CORRECT).

The problem affects only 4 specific workflow so far (we have in total 60 workflows). 2 of the workflows are using the same scheduler.

I've posted few months back in regards to scheduler trigger setting #1145. Does this issue affect this problem?

http://forums.networkautomation.com/forum/messageview.cfm?catid=33&threadid=7576&forumid=1



 Category Survey
 03/14/2011 11:57 AM
User is offline View Users Profile Print this message

Author Icon
Rick Johnson
AUTOMATE EMPLOYEE

Posts: 2757
Joined: 07/14/2008

Hi Karina,

The issue from the other thread which you references pertained to using non-US date/time formats. There were also error messages which indicated there was an invalid format. Unless you do have these errors and are using non-US formats, you are likely encountering a separate issue.

Do you ever see either of those 2 workflows attempt to start? I'm trying to pinpoint if the problem is getting the trigger to start, the workflow to start, or the task.

 Category Survey


-------------------------
- Rick Johnson
Network Automation
 03/14/2011 02:17 PM
User is offline View Users Profile Print this message

Author Icon
KarinaC
n00b (>6 posts)

Posts: 4
Joined: 04/13/2008

Hi Rick, I have seen many times that the 2 workflow attempt to start and they can start. However there can be few tasks inside the workflow (not all of them) that can be stucked i.e not doing anything. It says "running" but they dont



 Category Survey
 03/14/2011 03:08 PM
User is offline View Users Profile Print this message

Author Icon
Rick Johnson
AUTOMATE EMPLOYEE

Posts: 2757
Joined: 07/14/2008

Hi Karina,

That confirms my suspicion that the triggering is working to get the workflows started up.

Are the tasks which get stuck only executing FTP operations, or are there other processing actions executing as well?

Which operating system are you using, and what are the CPU and RAM of that machine?

 Category Survey


-------------------------
- Rick Johnson
Network Automation
 01/16/2012 10:45 AM
User is offline View Users Profile Print this message

Author Icon
TomGriffiths
Student (5-19 posts)

Posts: 16
Joined: 06/15/2011

We are having a similiar problem on our Automate server (7.1.3 also).    Some workflows trigger and execute successfuly 90% of the time but about once a week, the workflow does not execute. Even though  it shows running on Execution Status tab, if never does anything and does not show up in the BPATaskEvent.txt file.    If I run the Workflow manually when this happens, it executes without an issue.    The workflow consists of a schedule, a get task (triggered by the schedule) and  a put task (triggered by the successful completion of the get)   Sometimes the put is triggered even if the get is not triggered.



 Category Survey
 01/19/2012 06:58 AM
User is offline View Users Profile Print this message

Author Icon
Leonard Amabile
HelpSystems

Posts: 5454
Joined: 05/15/2007

Hi Tom,

Would it be possible to export your workflow and attach the .ampkg file? We will need to review the contents in order to troubleshoot this further. Do you have any prioritiy settings for the tasks within the workflow? Does it appear to encounter the problem are around the same time or day?




 Category Survey


-------------------------

Leonard Amabile | Director, Cross Platform Support
AutoMate | InterMapper | Skybot | Divisions of HelpSystems
T: +1.213.738.1700 | F: +1.213.738.7665
 01/24/2012 11:23 AM
User is offline View Users Profile Print this message

Author Icon
TomGriffiths
Student (5-19 posts)

Posts: 16
Joined: 06/15/2011

Leonard,

This zip file contains 7 workflows 5 of which consistently fail despite being rebuilt and re-scheduled.    The workflows that begin with WAH do not fail, the others do.  Any help you can give would be greatly appreciated.

Tom Griffiths



 Category Survey
 01/24/2012 04:22 PM
User is offline View Users Profile Print this message

Author Icon
Leonard Amabile
HelpSystems

Posts: 5454
Joined: 05/15/2007

Hi Tom,

Nothing looks out of the ordinary from what I imported. What is your current data store for BPA configured for? Is it set to .xml? Also, what are your default properties for priorities?





 Category Survey


-------------------------

Leonard Amabile | Director, Cross Platform Support
AutoMate | InterMapper | Skybot | Divisions of HelpSystems
T: +1.213.738.1700 | F: +1.213.738.7665
 01/30/2012 09:54 AM
User is offline View Users Profile Print this message

Author Icon
TomGriffiths
Student (5-19 posts)

Posts: 16
Joined: 06/15/2011

Leon,

 

the data store under Server Settings  has Embedded Database checked and the Priority under Default Properties has "Always run task" checked.

 

Tom Griffiths



 Category Survey
 01/30/2012 10:48 AM
User is offline View Users Profile Print this message

Author Icon
Leonard Amabile
HelpSystems

Posts: 5454
Joined: 05/15/2007

Hi Tom,

Depending on your operating system, please provide the size of the following file:


Windows Vista, Windows 2008, Windows 7:
C:\ProgramData\Network Automation\AutoMate BPA Server\BPA Local Database.vdb3

Windows XP, Windows 2003:
C:\Documents and Settings\All Users\Application Data\Network Automation\AutoMate BPA Server\BPA Local Database.vdb3


I do not recommend using the embedded database or XML data stores but instead recommend you use a SQL database instead.

 Category Survey


-------------------------

Leonard Amabile | Director, Cross Platform Support
AutoMate | InterMapper | Skybot | Divisions of HelpSystems
T: +1.213.738.1700 | F: +1.213.738.7665
 01/30/2012 01:31 PM
User is offline View Users Profile Print this message

Author Icon
TomGriffiths
Student (5-19 posts)

Posts: 16
Joined: 06/15/2011

Leonard,

We have a Windows 2003 server and the size of the BPA Local Database.vdb3 is 768,414 KB.

Tom Griffiths



 Category Survey
 01/30/2012 01:48 PM
User is offline View Users Profile Print this message

Author Icon
Leonard Amabile
HelpSystems

Posts: 5454
Joined: 05/15/2007

Hi Tom,

From the SMC, click Options > Server Settings > Log Management. Click on the Clear Execution Events button and then click OK at the next prompt. You will be notified once the event logs have been cleared. Let me know if the problem persists.




 Category Survey


-------------------------

Leonard Amabile | Director, Cross Platform Support
AutoMate | InterMapper | Skybot | Divisions of HelpSystems
T: +1.213.738.1700 | F: +1.213.738.7665
 01/30/2012 07:54 PM
User is offline View Users Profile Print this message

Author Icon
TedViens
n00b (>6 posts)

Posts: 3
Joined: 03/22/2011

Tom,

  Thank you for continuing to pursue this.  I opened up a support ticket months back, using one of our annual support tickets, and worked with NetworkAutomation Support; however, this was unsuccessful.  The ticket was closed unresolved.  It seemed as if I was doing all of the research on this anyway, although I provided everthing asked.

I am not sure what is going on.  This is a PROD issue and can impact patient care and billing.  That is why am grateful that you made this post.

The workflows do not complete and at time trigger.  Even logs do not get generated when the issue occurs.  Additionally, the tasks must be killed in the Execution Tab and manually rerun when this happens.

Questions:

- What is the issue?  How can this be determined?

- Why would SQL Server make a difference?  Why was this not broght forward?  In previous conversations with Support SQL and ODBC conncectivity the first thing to be removed as a variable.

- If I engage my DBA's and purchase a SQL Server, how can I be assured without the understanding of the cause by NetworkAutomation or internal staff know I will not be wasting departmental dollars.

I tried working with support previously, but in the end this interaction got us no closer to the source of the issue.



 Category Survey
 01/31/2012 07:19 AM
User is offline View Users Profile Print this message

Author Icon
Leonard Amabile
HelpSystems

Posts: 5454
Joined: 05/15/2007

Originally posted by: TedViens

Tom,




  Thank you for continuing to pursue this.  I opened up a support ticket months back, using one of our annual support tickets, and worked with NetworkAutomation Support; however, this was unsuccessful.  The ticket was closed unresolved.  It seemed as if I was doing all of the research on this anyway, although I provided everthing asked.




I am not sure what is going on.  This is a PROD issue and can impact patient care and billing.  That is why am grateful that you made this post.




The workflows do not complete and at time trigger.  Even logs do not get generated when the issue occurs.  Additionally, the tasks must be killed in the Execution Tab and manually rerun when this happens.




Questions:




- What is the issue?  How can this be determined?




- Why would SQL Server make a difference?  Why was this not broght forward?  In previous conversations with Support SQL and ODBC conncectivity the first thing to be removed as a variable.




- If I engage my DBA's and purchase a SQL Server, how can I be assured without the understanding of the cause by NetworkAutomation or internal staff know I will not be wasting departmental dollars.




I tried working with support previously, but in the end this interaction got us no closer to the source of the issue.



Ted,

The ticket closed because you were unresponsive. We requested a remote session to be conducted and you never responded to that request other than stating that the problem still existed. The issue cannot be determined without getting more information as I am doing with Tom. It has always been the recommendation for all versions of BPA to use an external data store such as SQL or mySQL. The included data stores of XML or embedded database are out of the box solutions and are not meant to be used in in a scalable environment. You can just as easily use SQL Express if you do not want to incur an additional cost. But I do recommend that you either open a new support ticket or respond to the closed ticket so we can troubleshoot further.

 Category Survey


-------------------------

Leonard Amabile | Director, Cross Platform Support
AutoMate | InterMapper | Skybot | Divisions of HelpSystems
T: +1.213.738.1700 | F: +1.213.738.7665
 01/31/2012 08:09 AM
User is offline View Users Profile Print this message

Author Icon
TedViens
n00b (>6 posts)

Posts: 3
Joined: 03/22/2011

Tom and Leonard,

  We can set up a WebEx to look at our environment further.  One was set up with NetworkAutomation, but there were no findings.

 

Leonard,

  We REALLY need to get these issues resolve.  Any assistance you can provide us to stabalize the product would be appreciated.  Tom is an excellent resource on my team for this.

What do you need to help get this resolved at this time.  Is there a list of recommened actions?

Thank you,

  Ted



 Category Survey
 01/31/2012 01:28 PM
User is offline View Users Profile Print this message

Author Icon
Leonard Amabile
HelpSystems

Posts: 5454
Joined: 05/15/2007

We need to move you away from the embedded database and the size of 768mb is too large of a file that can cause instability in BPA. Clearing the execution events will increase perfomance but unfortunately the size does not compact as it should. Migration to a SQL database would be our best recommendation and as a test we can install SQL Express on the same workstation and migrate the data over until we/you decide if pursuing a SQL Server implementation would be beneficial. Please send an email to support@networkautomation.com with your availability and reference this thread so we can schedule a remote session as soon as possible.




 Category Survey


-------------------------

Leonard Amabile | Director, Cross Platform Support
AutoMate | InterMapper | Skybot | Divisions of HelpSystems
T: +1.213.738.1700 | F: +1.213.738.7665
 01/31/2012 07:55 PM
User is offline View Users Profile Print this message

Author Icon
TedViens
n00b (>6 posts)

Posts: 3
Joined: 03/22/2011

Thank you.  I will speak with Tom and other members of the team to discuss this and layout a timeline.

As I understand it we need to take the following recommended actions to improve the stability of the system:

- Install SQL Express or make use of a SQL Server Instance

- Migrate the data from the Embedded DB to the SQL DB.  //Need NWAutomaiton assistance here.

- Setup a WebEx or GoTo with NetworkAutomation Support ASAP


Ted



 Category Survey
 02/01/2012 12:04 PM
User is offline View Users Profile Print this message

Author Icon
Leonard Amabile
HelpSystems

Posts: 5454
Joined: 05/15/2007

Hi Ted,

Utilizing an external database will improve performance and stability and is highly recommended.




 Category Survey


-------------------------

Leonard Amabile | Director, Cross Platform Support
AutoMate | InterMapper | Skybot | Divisions of HelpSystems
T: +1.213.738.1700 | F: +1.213.738.7665
 03/22/2012 06:03 AM
User is offline View Users Profile Print this message

Author Icon
TomGriffiths
Student (5-19 posts)

Posts: 16
Joined: 06/15/2011

Leonard, we change our data stor yesterday to use SQL DB with NW Automate's assistance and this workflow failed to trigger again last night. When trying to look at the initial "get" task I received an error that says:

Untitled-AutoMate BPA Agent Task Builder

X EAccessViolation


One of my coworkers does not get the same error, but I have tried repeatedly and still get the same error. Any ideas?

A screen shot of the actual error is attached.

 Category Survey
 03/22/2012 04:28 PM
User is offline View Users Profile Print this message

Author Icon
Leonard Amabile
HelpSystems

Posts: 5454
Joined: 05/15/2007

Hi Tom,

Is it just this task that can't be opened? Can Task Builder be opened from the Start Menu without any issues?





 Category Survey


-------------------------

Leonard Amabile | Director, Cross Platform Support
AutoMate | InterMapper | Skybot | Divisions of HelpSystems
T: +1.213.738.1700 | F: +1.213.738.7665
Statistics
18258 users are registered to the AutoMate Discussion forum.
There are currently 0 users logged in.
The most users ever online was 5551 on 01/08/2018 at 11:11 AM.
There are currently 464 guests browsing this forum, which makes a total of 464 users using this forum.

FuseTalk Enterprise Edition v4.0 - © 1999-2020 FuseTalk Inc. All rights reserved.

Sitemap Network Automation Software Blog