I have 2 cases out to EMC at the moment, slow disk speeds and ghost single fan fault errors. EMC has come back and said both are hardware replacement fixes. I recently unfroze about 100+ virtual tapes in our VTL and all of the expired ones went to the scratch pool. 4 days later, there are about 10 frozen virtual tapes now. EMC is recommending that I replace cables and BCC's from 3 enclosures now, update the Frumon on all BCCs.
Of course I expect a 2 hours down time while I work with the Unisys people to replace these things. If this doesn't fix my speed issue, it just added another terabyte or 2 or data that I'm behind on duplication. I was hoping for more of a plausible explanation for the speed issue rather than a hardware problem, since we've been having speed issues before I took over this DL700 and they haven't been having hardware issues all the time.
the insane ramblings of a systems admin about the products he works on...
Thursday, December 20, 2007
Wednesday, December 19, 2007
Fun with EMC DL700
If you've ever had to use Powerlink for something you'll know how slow it is. Have you ever used the search function on there? Another painful process. I administer a first gen DL700, and recently I've been looking into why we have such slow duplication to physical tape from it. I've got it to the point where I can't think of much else besides the library is operating at hilariously slow speeds. A simple speed test on the virtual tapes reveals 120mb/sec beginning performance and 8-15mb/sec middle and ending performance. Random speeds are all around 5-9mb/sec. For a raid 3 array it is simply unacceptable. It's current flare code is at 2.19 and the latest I can find on Powerlink is 2.26.
Now I opened a case with EMC on Powerlink and sent them some SP collects, I hope to see a response from them that was better than the CX300 case I opened from them. You see I opened a case for the CX300 performing poorly and they sent Unisys out to collect 2000 logs plus more on our switches and then after not hearing anything from them, they simply closed the case a month later. While the CX300 is slow, it is passable for our VMware environment for now, the DL700/CX700's speeds are not for backup. I'm putting more and more data to it, yet speeds seem to be slower if anything. I hope my experience with EMC this time around is better but if they can't fix this speed issue, I am going to push, and push hard, to change SAN vendors. I won't be the only one pushing for a different storage vendor either, several other SAN admins do not like EMC much either.
It is past my bedtime and the last 2 hours spent searching for possible tweaks yielded nothing on Powerlink, I await EMC's response.
Now I opened a case with EMC on Powerlink and sent them some SP collects, I hope to see a response from them that was better than the CX300 case I opened from them. You see I opened a case for the CX300 performing poorly and they sent Unisys out to collect 2000 logs plus more on our switches and then after not hearing anything from them, they simply closed the case a month later. While the CX300 is slow, it is passable for our VMware environment for now, the DL700/CX700's speeds are not for backup. I'm putting more and more data to it, yet speeds seem to be slower if anything. I hope my experience with EMC this time around is better but if they can't fix this speed issue, I am going to push, and push hard, to change SAN vendors. I won't be the only one pushing for a different storage vendor either, several other SAN admins do not like EMC much either.
It is past my bedtime and the last 2 hours spent searching for possible tweaks yielded nothing on Powerlink, I await EMC's response.
Tuesday, December 18, 2007
Windows Server 2008 Hyper-V
Microsoft today had a webcast to announce what many people in the industry already knew, Windows Server 2008 Hyper-V. This is not an updated version of Virtual Server 2005 but a hypervisor from Microsoft. My boss expects it take over VMWare in a couple of year but I'm not sold yet. I am surprised at how developed (at least on paper) it already seems to be. Microsoft says it has support for Windows product lines and Linux (Novell now, Redhat in January) in their presentation. You can deploy it with 2008 Server Core as well. Hyper-V costs $28 which seems insanely cheap for Microsoft, the presentation said you can spend $28 and consolidate your Windows 2003 machines on it. I will have to download 2008 RC1 with Hyper-V and check out the features more in depth.
Things that stood out in the presentation.
Hyper-V server costs $28!
Low overhead design from the beginning (for Microsoft).
Server Core.
MMC 3.0 UI.
1/3 cost of VMware (Says Microsoft)
I tend to believe that last point since people are already familiar with the Windows world but not VMware when they are virtualizing. Of course if you are virtualizing mostly none Windows machines I'm not sure if Hyper-V will work. It does seem fatter as the hypervisor / Server Core both need to be (I think) in order to provide all functionality and it needs more hardware (memory) than VMware.
Links: Here are some links for Hyper-V.
Download Hyper-V
Hyper-V Overview
Things that stood out in the presentation.
Hyper-V server costs $28!
Low overhead design from the beginning (for Microsoft).
Server Core.
MMC 3.0 UI.
1/3 cost of VMware (Says Microsoft)
I tend to believe that last point since people are already familiar with the Windows world but not VMware when they are virtualizing. Of course if you are virtualizing mostly none Windows machines I'm not sure if Hyper-V will work. It does seem fatter as the hypervisor / Server Core both need to be (I think) in order to provide all functionality and it needs more hardware (memory) than VMware.
Links: Here are some links for Hyper-V.
Download Hyper-V
Hyper-V Overview
Netbackup VSP tmp file locks
I work on a Netbackup 5.0MP6 Windows Master Server and sometimes on clients the drives fill up with VSP tmp files. They use every single bit of space and the official fix from Veritas is to reboot the server and then delete the files. Not a very good solution if you have any kind of real server environment. Here is how you can get around that.
The symptoms: Symantec Netbackup creates tmp files on the root drives to backup open files on Windows clients, these are suppose to be deleted after the backup but they often do not ending up filling the drives to capacity. They look like these files.
_vxfiVspCacheFile_1.tmp
_vxfiVspCacheFile_2.tmp
_vxfiVspCacheFile_0.tmp
The cause: Virus scanners, including Symantec's own Norton antivirus puts locks on the VSP (Veritas open file backup) tmp files and they do not get deleted. The only sure fire way to delete them is to reboot the server.
The solution: I have heard this does not always work but I've never had a problem with it on Windows 2000 or 2003 servers. We can use a little program called Process Explorer from Microsoft (System Internals) to unlock the file and delete it. First stop your antivirus from its' real time scanning. Then start up Process Explorer and put in the Vsp tmp file name that shows up on your drives (they could be hidden). Process Explorer will fine the handle locking them and you can right click and close the handle locking these files. I recommend that you try Process Explorer on something else first if you've never used it since you can close any handle and blue screen your box if you don't know what you are doing. Once the handle is closed, you can delete the tmp files, turn antivirus back on and pick up your fat raise... yeah right.
Now how do we prevent it? No good way really, turn off virus scanner? yeah right. Turn off open file backups in VSP? If you don't need it. Don't use Netbackup, haha. With Windows 2003 you can use VSS (Windows built in Volume Shadow Copy Service or Volume Snapshot Service or VSS... Microsoft...) in the newer version of Netbackup (6.5 for sure) but I haven't tried this yet. Maybe it will work better since VSS lets you set max amount of disk space to use.
Links: I used the following in my original search to solve this problem.
MCSE forums
Bob's Blog
Download Process Explorer
The symptoms: Symantec Netbackup creates tmp files on the root drives to backup open files on Windows clients, these are suppose to be deleted after the backup but they often do not ending up filling the drives to capacity. They look like these files.
_vxfiVspCacheFile_1.tmp
_vxfiVspCacheFile_2.tmp
_vxfiVspCacheFile_0.tmp
The cause: Virus scanners, including Symantec's own Norton antivirus puts locks on the VSP (Veritas open file backup) tmp files and they do not get deleted. The only sure fire way to delete them is to reboot the server.
The solution: I have heard this does not always work but I've never had a problem with it on Windows 2000 or 2003 servers. We can use a little program called Process Explorer from Microsoft (System Internals) to unlock the file and delete it. First stop your antivirus from its' real time scanning. Then start up Process Explorer and put in the Vsp tmp file name that shows up on your drives (they could be hidden). Process Explorer will fine the handle locking them and you can right click and close the handle locking these files. I recommend that you try Process Explorer on something else first if you've never used it since you can close any handle and blue screen your box if you don't know what you are doing. Once the handle is closed, you can delete the tmp files, turn antivirus back on and pick up your fat raise... yeah right.
Now how do we prevent it? No good way really, turn off virus scanner? yeah right. Turn off open file backups in VSP? If you don't need it. Don't use Netbackup, haha. With Windows 2003 you can use VSS (Windows built in Volume Shadow Copy Service or Volume Snapshot Service or VSS... Microsoft...) in the newer version of Netbackup (6.5 for sure) but I haven't tried this yet. Maybe it will work better since VSS lets you set max amount of disk space to use.
Links: I used the following in my original search to solve this problem.
MCSE forums
Bob's Blog
Download Process Explorer
Subscribe to:
Posts (Atom)