Tuesday, August 24, 2010

ClearCase Deliver with "unable to compute base" Error

Problem Symptoms:

1. Rebase and deliver operation could not finish on one particular file. Error message:
>>> Operation started: 8/17/2010 3:29:45 PM
Error from VOB database: "\MidTier".
Element "xxxx\InquiriesRequestVOFactory.java", unable to compute base (to \main\Mainline_i\CBFE17.5_THS\djogo_CBFE17.5_THS\29 from \main\Mainline_i\CBFE17.5_THS\36).
Skipping "xxxx\InquiriesRequestVOFactory.java".

Other related errors.
1. The version tree of the file could not be retrieved from GUI with the following error message. But cleartool lsvtree command ran without error.
"failed to retrieve version history of the element"

2. db_server_log error message.
% more db_server_log
2010-08-16T09:17:10-04:00 db_server(15416): Ok:
2010-08-17T12:47:07-04:00 db_server(11335): Error: Database identifier 724419 not found in "../db__obj.c" line 740.
2010-08-17T12:55:13-04:00 db_server(11335): Error: Database identifier 724419 not found in "../db__obj.c" line 740.
2010-08-17T12:57:38-04:00 db_server(11336): Error: Database identifier 724419 not found in "../db__obj.c" line 740.
2010-08-17T13:02:54-04:00 db_server(12862): Error: Database identifier 724419 not found in "../db__obj.c" line 740.
2010-08-17T13:06:08-04:00 db_server(12862): Error: Database identifier 724419 not found in "../db__obj.c" line 740.
2010-08-17T13:06:49-04:00 db_server(12862): Error: Database identifier 724419 not found in "../db__obj.c" line 740.

3. could not merge the file.
>cleartool merge -to InquiriesRequestVOFactory.java -version \main\Mainline_i\CBFE17.5_THS\36
cleartool: Error: Error from VOB database: "\MidTier".

But the file could be merge with only drawing merge arrows.
>cleartool merge -to InquiriesRequestVOFactory.java -ndata -version \main\Mainline_i\CBFE17.5_THS\36
Recorded merge of "InquiriesRequestVOFactory.java".

By drawing an merge arrow as above, we can skip the element for deliver/rebase. but it doesnot resolve the problem.

1. run command dbcheck.
/etc/utils/dbcheck -r1 -a -k -p32767 vob_db
Processing data file: vob_db.d02(3), total of 1302988 records
Problems at record 1298822:
* key field OBJ_DBID(23) error: has a missing key
1 error was encountered in 1 record/node

1. Stop ClearCase (or untag and unregister the VOB, stop ClearCase and then start ClearCase)
2. Backup (very import!). Copy the db directory
3. Copy the keybuild utility to the VOB db directory
4. run: keybuild vob_db under the VOB db directory
5. Exit out of the db directory after keybuild completes
6. Start ClearCase
7. Reformat the VOB. Run the following command:
cleartool reformatvob

If these steps are completed successfully and without errors, your VOB should now be healthy. If they did not complete or you received errors, restore the VOB from backup.

For my VOB with 7.4GB in size and 5 years in history, it took about 3 hours to finish the procedure.

Thursday, June 10, 2010

Report Server rsInternalError Error

When accessing all the TFS reports from report server, the following message showed up in browser.
An internal error occurred on the report server. See the error log for more details. (rsInternalError)

Further investigation found more information in the Reporting Service log file and dump log file.

ERROR , SQLDUMPER_UNKNOWN_APP.EXE, AdjustTokenPrivileges () failed (00000514)

w3wp!processing!5!06/07/2010-08:48:03:: a ASSERT: Assertion failed! Call stack:
Microsoft.ReportingServices.ReportProcessing.Persistence.IntermediateFormatReader.Assert(Boolean condition)
Microsoft.ReportingServices.ReportProcessing.Persistence.IntermediateFormatReader.Initialize(Stream stream)
Microsoft.ReportingServices.ReportProcessing.Persistence.IntermediateFormatReader..ctor(Stream stream)
Microsoft.ReportingServices.ReportProcessing.ReportProcessing.DeserializeReport(GetReportChunk getChunkCallback, ReportItem parent, Hashtable& definitionObjects)

These reports used to work, and there was no known changes to the system. I do notice that the size of chunkData table in the reportserver database became 0 around the same time the problem started.

To solve the problem, we need to restore the reports. From SQL Server Management Studio, run

use reportserver

itemID, Path, Name, CAST(CAST(Content AS varbinary(max)) AS xml) ReportXML
where content is not null

The ReportXML column is the report definition. Save it to a name.rdl file, and upload the file to the report site to overwrite the report.

This can not be the best solution, as I have 900+ reports to recover. But it is the only working one I found so far.

Wednesday, June 2, 2010

Hoxfix KB957196 Install procedure for database-tier mirrored TFS sites

This procedure also applies to the installation of TFS upgrades SP1 on database-tier mirrored sites.

1. before start, notify the users and make sure nobody is using TFS.
2. backup Report Server encryption key. Ship the key to site2. The key will not be used in the procedure. It will be used if the primary site needs to recover.
3. open SQL Server Management Studio, connect to site1 and site2.
4. in Management Studio, full database + transaction backup for site1. Ship the backups to site2.
6. in Management Studio, stop mirroring from site1.
7. install hotfix for 750-smart-bld1.
8. backup full database + transaction log. TFSIntegration database is modified during hotfix install.
9. ship the backup set to site2.
10. in Management Studio, restore database (and transaction log) to open status for site2.
11. in Management Studio, set up mirroring from site1.
13. full backup again for site1.

Monday, May 31, 2010

Mostly Recently Updated Table for SQL Server

I found a useful query for finding out the most recently accessed/updated table in a SQL Server database (I forgot the original resource). I used the script to find out a table back sceen of Team Foundation Server (TFS), so I could modify the content to configure a particular URL.

use TfsIntegration     
sys.dm_db_index_usage_stats i JOIN
sys.tables t ON (t.object_id = i.object_id)
database_id = db_id()

SQL Server database hang in "in recovery" status

Problem. After a failed setting mirror operation (initiated from SQL Server Management Studio from GUI), SQL Server Database stay in "in recovery" status for prolonged period of time (overnight). The database is about 7GB in size. From the primary site, the database showed normal. From the secondary site, the database showed "in recovery".

There is little that can be done for the secondary database. One option is to stop the SQL services, rename the data files (.mdf & .ldf), start the SQL database again, drop the existing database, rename the data files back, and attached the data files with same database name. After that, we can try to restore database backup and set up mirror.

On a closer look from primary, it seems that the mirror is broken. However, from the output of some database operation in the secondary database, it seems that the mirror has not be broken yet. Further more, there is no entry in the event log to indicate the progress of the recovery operation.

Let's try this approach. From the primary site, to break the mirror run the following command.
After the command finished successfully, the database on the secondary site changed to (Restoring…) status immediately. The following operations are straight forward -- copy the full database backup and log, and restore with norecovery option, set mirror for the database.
This is the preferred approach, since only the problem db is touched, no need to re-set the mirror for all other databases.

There are occasional complains from the internet about this prolonged (or hanged) SQL Server database "in recovery" status. It may worth to take a look whether the database actually started the "recovery", or still in a previous status due to unknown reasons.

Script to Monitor ClearCase license

Here are the scripts mentioned in my previous post regarding monitoring ClearCase license usage.

re: http://doublepaddle.blogspot.com/2008/02/monitoring-clearcase-license-usage.html Apperently I am not good at following up my previous posts -- Sorry for the delay.

The collection script is as simple as this.

date >> /home/ccadmin/logs/license.log
/opt/rational/clearcase/bin/clearlicensegrep "Current active users" >> /home/ccadmin/logs/license.log

The script is scheduled in cron jobs for every 10 mintues. This script can be easily converted to a Windows batch file and added to scheduler.

Then a perl script is used to load the data and generate the report in excel format. The script can be added as part of the Windows scheduler, or run at the time when you want to view the report. The nice chart in the previous post is also part of the excel report.

#Auther: Li Qin
#Date: 2008.02.05
#Function: generate ClearCase license usage report in excel format

use Spreadsheet::WriteExcel;
use Win32::OLE qw(in with);
use Win32::OLE::Const 'Microsoft Excel';

$Win32::OLE::Warn = 3; # die on errors...

$file = "C:\\Li\\SCM\\ClearCase\\report-licns.xls";
$workbook = Spreadsheet::WriteExcel->new("$file");
$sheet = $workbook->addworksheet("data");

my $format1 = $workbook->add_format();

my $format2 = $workbook->add_format();

my $format3 = $workbook->add_format();

$sheet->merge_range('C2:G2', "ClearCase Usage Report", $format1);

$sheet->set_column(3, 2, 40);
$sheet->set_column(3, 3, 20);
$sheet->set_column(3, 4, 20);
$sheet->write(3, 2, "Date", $format2 );
$sheet->write(3, 3, "Usage", $format2 );
$sheet->write(3, 4, "Max", $format2 );

open(FILE,"license.log")  || die "couldn't open file a.out";
$row = 4;
my @user1, @user2;
$max = 35;
while ($date= ) {
chomp $date;

$line1 = ;
chomp $line1;
while ( !($line1 =~ m/Current/) ) {
$user1[$row-4] = 0;

$sheet->write($row+1, 2, $date, $format3);
$sheet->write($row+1, 3, $user1[$row-4], $format3);
if ( $date =~ m/Fri Feb 1 17/ ) {
$max = 40;
$sheet->write($row+1, 4, $max, $format3);
$row = $row +1;
$date = $line1;
$line1 = ;
chomp $line1;
$pos = rindex $line1, ":";
$user1[$row-4] = substr($line1,$pos+1);

$sheet->write($row+1, 2, $date, $format3);
if ( $date =~ m/Fri Feb 1 17/ ) {
$max = 40;
$sheet->write($row+1, 3, $user1[$row-4], $format3);
$sheet->write($row+1, 4, $max, $format3);
$row = $row +1;

# #create chart
my $Excel = Win32::OLE->GetActiveObject('Excel.Application') || Win32::OLE->new('Excel.Application', 'Quit');

# open Excel file
my $Book = $Excel->Workbooks->Open("$file");
# select worksheet of data

$count = 6;
my $sheet = $Book->Worksheets("data");
my $value = $sheet->Cells($count,4)->{'Value'};
my $len = length($value);

while ( $len >0 ) {
$count = $count +1;
my $value = $sheet->Cells($count,4)->{'Value'};
$len = length($value);
#printf "At ($count, 4) the value is %s. length is %d\n", $value, $len;
print "last row is $count.";

$lastRow = $count-1;
my $Range = $sheet->Range("C6:E$lastRow");

my $Chart2 = $Excel->Charts->Add($sheet);
$Chart2->{ChartType} = xlLine;
$Chart2->SetSourceData({Source => $Range, PlotBy => xlColumns});

$Chart2->{HasTitle} = 1;
$Chart2->ChartTitle->{Text} = "ClearCase License Usage\n";
$Chart2->Axes(xlValue)->{HasTitle} = 1;
$Chart2->Axes(xlValue)->AxisTitle->{Text} = "License Usage";
$Chart2->Axes(xlCategory)->{HasTitle} = 1;
$Chart2->Axes(xlCategory)->AxisTitle->{Text} = "Date/Time";

$Chart2->{HasLegend} = 'False';
$Chart2->{Name} = "Chart";
$Chart2->{HasLegend} = 1;
$Chart2->Legend->{Position} = xlBottom;
$Chart2->SeriesCollection(1)->{Name} = "Usage";
$Chart2->SeriesCollection(2)->{Name} = "Max";

# save and clean up
# End of create charts

We have upgraded our monitoring using SQL Server Reporting Service, a nice-to-have.

Monday, May 17, 2010

Change check-in email notification to TFS Web Access

There are some very useful articles to change TFS email notifications to link to Team System Web Access:

Changing TFS emails to link to Team System Web
Changing TFS emails to link to Team System Web Part 2

The configurable links include:
- Work Item notification mails
- Build notification mails
- Check-in notification mails
- Mails sent by Team Explorer

However, for check-in notification mails, if your TFS web access is not installed as the default website, and contains a virtual directory as part of the URL, the tfsadminutil tool cannot process the URL correctly, at least not until TFS 2008 SP1. For example, if your TFS web access URL is http://myTFSserver:8080/tswa, tfsadminutil tool will set up the links as http://myTFSserver:8080/ instead.

Here we provide a workaround until tfsadminutil tool has that improved. The workaround requires direct modification of the TFS database. Please note that it is NOT recommended by Microsoft.

Prerequisite: TFS 2008 + SP1 + TFS web access + Hotfix KB957196

Step 1. run the command on the TFS server:
C:\Program Files\Microsoft Visual Studio 2008 Team Foundation Server\Tools> tfsadminutil configureconnections /TSWAUri:http://myTFSserver:8080/tswa

Step 2. connect to TFS database through SQL Server Management Studio. Run the following query.
use TfsIntegration
select * from tbl_service_interface

8 WorkItemEditor
8 ChangesetDetail
8 Difference
8 ViewItem

Step 3. Modify the URLs in the table for the above items.
Update tbl_service_interface
Set url='http://myTFSserver:8080/tswa/wi.aspx'
Where name='WorkItemEditor'

Update tbl_service_interface
Set url='http://myTFSserver:8080/tswa/cs.aspx'
Where name='ChangesetDetail'

Update tbl_service_interface
Set url='http://myTFSserver:8080/tswa/diff.aspx'
Where name='Difference'

Update tbl_service_interface
Set url='http://myTFSserver:8080/tswa/view.aspx'
Where name='ViewItem'

Step 4. After the TFS process is recycled, your check-in email subscription will generate emails linking to TFS web access.

Thursday, February 4, 2010

No Views Show up in ClearCase Explorer

One developer got problem accessing his view. ClearCase used to work fine on his desktop. Since two days ago, after refreshing the ClearCase Explorer view shortcuts, he found that there were no views showing up on the view panel. He had no problem using ClearCase Project Explorer, except for view related operations, such as list the view property.

When trying to start the view manually, it showed the following error.

H:\>cleartool startview patelsus_VER6.1.5_int
view_contact call failed: RPC: Unable to receive; errno = [WINSOCK] Connection reset by peer
cleartool: Error: Error trying to contact view_server for view Tot75xrdtw1027q:C:\CCShare\patelsus_VER6.1.5_int.vws: No such file or directory
cleartool: Error: Couldn't set view tag patelsus_VER6.1.5_int: No such file or directory

There were errors in the system's event log too, complaining view server died on start up, etc.

After rule out other causes, run winmsd from Start -> Run, under the network -> protocal, the first entry should be "MSAFD Tcpip [TCP/IP]" and the second should be "MSAFD Tcpip [UDP/IP]". However, the protocal showed: "SSH Capture over [MSAFD Tcpip [TCP/IP]]" and "SSH Capture over [MSAFD Tcpip [UDP/IP]]".

It turned out that the developer installed SSH Tectia, which conflicts with ClearCase client. After uninstallation, the problem is resolved.