Hyperion Essbase Application Log file formatter

In this blog, I am going to talk about a custom log file formatter that I have designed for parsing the Hyperion Essbase application log files into columnar format. I was working last week on resolving a couple of issues where I had to keep checking the log file to see exactly what was going wrong and that is when I felt I could design something to make this searching process better. 

The below snapshot is the view of the utility that I have designed.

You can basically select any Essbase application log file by using the Browse button.

Once you have selected that, the application will automatically update the parsed log file field name.

The name of the parsed application log file is as follows:- <ApplicationName>_PARSED.log

Below is the screenshot of the same.

You now need to hit the format button to format the file into a tab separated file.

Sample output is attached in the below snapshot.

I am now working on adding a database push functionality as well. But this will require some work since, I will have to code the entire relational logic in Java and do it. Will keep you posted on the same.

Also, I will be moving my code to a source code management system, mostly GitHub, so that anyone can download and use these utilities…Will keep you updated on this as well…

Hyperion Essbase Application Black Box

In this blog, I will be talking about the Hyperion Essbase applications black box, namely, the Essbase application log files. Well if I check Wikipedia for what it means to be a black box, it is, I quote “an electronic device placed in an aircraft for the purpose of facilitating the investigation of aviation accidents and incidents”. Hyperion Essbase application log file also does the same thing but thankfully, we do not have to deal with accidents only incidents and being able to read an application log file gives a wealth of information about the overall nature of the system that we have. 

Well, the first thing about the Essbase application log file, is that these files can be huge. I have personally worked with Essbase application log files that ran in gigabytes and working with them trying to access information in a production system about outage, where you can have multiple users connecting to the system and at the same time have data loads running on a daily basis is particularly time consuming.

Also, information is not visible exactly like the format we are looking for and some parsing is required to work with this. The best thing that Hyperion system administrators can do is archive the log files on a weekly, monthly or quarterly basis so that the parsing effort is minimized. 

Now, the Essbase application log file generally will look as shown in the below snapshot:-

The general format (speaking in a computer science parlance) would be in the following syntax:-

[Timestamp information][Essbase application/database operation high level details along with code]

[Essbase application/database operation in details]

The timestamp information is usually in this format : - <Day of the week> <Month> <day of the month> <time in 24 hour format> <year>

Another thing to note is that there can be multiple entries for a single timestamp. This is generally the nature of any logging software since the system would be doing multiple operations simultaneously. For example, in the above snapshot, most of the entries are for November 16 2015 11:31:55.

The Essbase application/database operation high level details basically contains the following information:-

<Hardcoded value: Local>/<Application name>/<Database name>/<User name>/<opcode>/<operation logging details>

The first field has a value of “Local” in most of the log files that I have seen. If anyone of you has ever seen any different value here, do mail me a snapshot of the same so I can analyze it.

Since every Essbase application has a single log file, the Essbase application high level details include information about the application name and cube name to distinguish among different databases within the application.

In addition it also includes the user name who performed the operation.

Opcode is basically a number that I could not find any significance for since it appears to be some kind of internal code.

The operation loggings details basically indicates if the operation caused an error, warning or was executed successfully and has the following three values that I have commonly seen:-

·         Warning

·         Info

·         Error

The below snapshot has all three types of information shown in a single file:-

I have observed that the database name and user name entries are optional and can be blank in the log file.

Also note that the each of the operation comes with a corresponding message code tagged to it.

The next section is the operation details with a detailed message on what exactly the Essbase system did. Some examples are as shown below:-

·         Essbase needs to retrieve [1] Essbase Kernel blocks in order to calculate the top dynamically-calculated block.

·         The Dyn.Calc.Cache for database [Plan1] can hold a maximum of [22] blocks.

·         The Dyn.Calc.Cache for database [Plan1], when full, will result in [allocation from non-Dyn.Calc.Cache memory].

·         Writing Parameters For Database [Plan1]

·         Reading Parameters For Database [Plan1]

I am currently working on designing a parsing system to get useful information out of this log file and build a reporting application around these files. For the time being I have managed to get the data to be loaded into an Oracle database in a format that has some meaning. Below is a snapshot of the parsed log file details:-

I am now trying to design a system that can actually sift through this data and generate some useful statistics.

Will keep you updated on the same…Do watch out my blog for more…

Creating timestamped folders in Windows using batch scripting

In this blog, I am going to show you how to create a date/timestamp folder in Windows using batch scripting… This is particularly useful when you need to take periodic backups of Hyperion applications (generally LCM backups/level zero data export of Planning applications before application refresh) as all system administrators have to do sometime or the other. 

In Linux/Unix like systems, manipulating date values and time stamp values is pretty straight forward…However, in Windows, it requires some hard work since batch scripting gives the users a very basic set of features and functionalities to manipulate and work with…

The script that will create it is as shown in the below snippet..I will explain the working later…

set datevalue=temp

set timevalue=temp

echo "Displaying the temporary values here"

echo %datevalue%

echo Displaying the values of date and time here

date /t

time /t

for /f "tokens=2-4 delims=/ " %%a in ('date /t') do set datevalue=%%a_%%b_%%c

for /f "tokens=1-3 delims=: " %%a in ('time /t') do set timevalue=%%a_%%b_%%c

echo %datevalue%_%timevalue%

mkdir .\%datevalue%_%timevalue%

pause

I save this script as a batch file as shown in the below snapshot.

The set commands in batch are equivalent to assignment statements in programming languages. 

The date command will display the system date as shown in the below snapshot.

The time command will display the system time as shown in the below snapshot.

The default behavior of the date and time command is to output the system date and time respectively and then ask the user to enter a new date and time...which is weird since I would not want to do it every time…In order to overcome this, we use the /t flag which notifies the command prompt that the system is to just output the system date and time.

All the action takes place in the two for statements… The first creates a datevalue variable and the second creates a timevalue variable

The first for statement in above snapshot on line number 11 of the script can be explained as follows:-

For the second, third, fourth token (tokens=2-4) that is found in the output of ‘date /t’ command(date /t'), where delimiters are forward slash and space (delims=/ ), assign the tokens to positional parameters starting as %a, %b and so on. Set datevalue basically creates an environment variable called datevalue which now has the tokens %a, %b and %c ( Month, date and year respectively ) separated by an underscore character.

The timevalue is created similarly the only difference being that for time we take the first, second and third token and this is separated by a colon and space.

An important point to note in the above scenario is that %a and %A positional parameters are different in Windows batch scripting. This is the only exception that I have found where batch scripting is case sensitive as of now.

In the line number 17 of the script I created a date timestamp folder in the current directory from where I am running the script.

The output of running the batch file is as shown in the below snapshot.

The timestamp folder that I created is as shown below:-