Mike Says Meh The blog of Sys Admin Mike Kauspedas

2May/150

The mother of all disk benches! 4 disk SSD array, server SSD array, storage spaces

I built a new disk array for my Hyper-V lab at home. Overkill? Yeah, but I got tired of waiting for VM's to boot, or build from deployment. When I study or learn something new I need efficiency! Plus there was a sale on 256GB drives and I found a little 4 disk hot swap box. For comparison I have a SSD array in a server, some single disks, and a 4 disk old SATA300 using Windows 8.1 storage spaces (software RAID).

My PC: Intel Core i5 4670K | 32GB DDR1600
I'm using Crystal disk mark for testing.

*Note: I use RAID 0, there is a write penalty but these are SSDs. Yes, data isn't important so redundancy, well, who cares. Although I do back it up with Backblaze which allows me to backup meaningless VHDs.
4 disk RAID 0 SSD Array. Sandisk SDSSDHP256G 256GB SATA600 SSD
Sequential Read (Q= 32,T= 1) : 1578.833 MB/s
Sequential Write (Q= 32,T= 1) : 1272.100 MB/s
Random Read 4KiB (Q= 32,T= 1) : 349.640 MB/s [ 85361.3 IOPS]
Random Write 4KiB (Q= 32,T= 1) : 318.789 MB/s [ 77829.3 IOPS]
Sequential Read (T= 1) : 1419.964 MB/s
Sequential Write (T= 1) : 1146.151 MB/s
Random Read 4KiB (Q= 1,T= 1) : 125.328 MB/s [ 30597.7 IOPS]
Random Write 4KiB (Q= 1,T= 1) : 100.152 MB/s [ 24451.2 IOPS]

Notice the increased read performance between the array and a single SSD disk, but how large the write penalty is with RAID 0.

Single SSD SATA600 OCZ-Agility 256GB
Sequential Read (Q= 32,T= 1) : 386.957 MB/s
Sequential Write (Q= 32,T= 1) : 209.348 MB/s
Random Read 4KiB (Q= 32,T= 1) : 262.283 MB/s [ 64033.9 IOPS]
Random Write 4KiB (Q= 32,T= 1) : 211.379 MB/s [ 51606.2 IOPS]
Sequential Read (T= 1) : 187.703 MB/s
Sequential Write (T= 1) : 211.817 MB/s
Random Read 4KiB (Q= 1,T= 1) : 18.627 MB/s [ 4547.6 IOPS]
Random Write 4KiB (Q= 1,T= 1) : 131.864 MB/s [ 32193.4 IOPS]

Single SATA Hybrid SATA600 ST750LX003-1AC154 750GB
Sequential Read (Q= 32,T= 1) : 104.197 MB/s
Sequential Write (Q= 32,T= 1) : 105.733 MB/s
Random Read 4KiB (Q= 32,T= 1) : 1.317 MB/s [ 321.5 IOPS]
Random Write 4KiB (Q= 32,T= 1) : 1.199 MB/s [ 292.7 IOPS]
Sequential Read (T= 1) : 104.002 MB/s
Sequential Write (T= 1) : 106.107 MB/s
Random Read 4KiB (Q= 1,T= 1) : 0.663 MB/s [ 161.9 IOPS]
Random Write 4KiB (Q= 1,T= 1) : 1.214 MB/s [ 296.4 IOPS]

Storage Spaces two-way mirror (RAID 10) 4 disk array SATA300 ST3750640AS 750GB
Sequential Read (Q= 32,T= 1) : 75.898 MB/s
Sequential Write (Q= 32,T= 1) : 24.826 MB/s
Random Read 4KiB (Q= 32,T= 1) : 0.649 MB/s [ 158.4 IOPS]
Random Write 4KiB (Q= 32,T= 1) : 0.284 MB/s [ 69.3 IOPS]
Sequential Read (T= 1) : 75.491 MB/s
Sequential Write (T= 1) : 26.843 MB/s
Random Read 4KiB (Q= 1,T= 1) : 0.651 MB/s [ 158.9 IOPS]
Random Write 4KiB (Q= 1,T= 1) : 0.284 MB/s [ 69.3 IOPS]

RAID 5 with 6 disk + hot spare. PERC H710P Mini SATA600
Sequential Read (Q= 32,T= 1) : 2855.447 MB/s
Sequential Write (Q= 32,T= 1) : 2016.346 MB/s
Random Read 4KiB (Q= 32,T= 1) : 349.066 MB/s [ 85221.2 IOPS]
Random Write 4KiB (Q= 32,T= 1) : 115.260 MB/s [ 28139.6 IOPS]
Sequential Read (T= 1) : 2185.178 MB/s
Sequential Write (T= 1) : 1794.224 MB/s
Random Read 4KiB (Q= 1,T= 1) : 60.030 MB/s [ 14655.8 IOPS]
Random Write 4KiB (Q= 1,T= 1) : 59.278 MB/s [ 14472.2 IOPS]

I'm going to have to work on the random reads and writes on this server. That should blow my home array out of the water.

Filed under: Tech No Comments
8Apr/150

Open your hosts file in notepad as administrator from command prompt

Working for a web hosting company I edit my hosts file a LOT. And forever I did it the hard way, right click on notepad and then open as administrator then open the hosts file. Unacceptable! I need my hosts file open and ready in a split second. Time is money, money is time, time is infinite, money is power, power corrupts, illuminati! But I digress. Here are the steps.

1. Open command prompt as administrator. Right click on it and open as administrator. Or go to your start screen and search for cmd then hold down CTRL + SHIFT + ENTER. I have classic shell installed because over the years I grew to love and admire the start menu and I refuse to give it up despite everyone's love of the new start tile metro screen.

Note: You'll know it's open as Administrator because the top of the window will say so and the default path will be C:\Windows\System32>. If you open command prompt as your normal user it will open to a default path of your user profile. IE C:\Users\user.name>

2. Type "notepad C:\Windows\System32\drivers\etc\hosts". Without the "" into the command prompt.

3. Now the hosts file is open and notepad is running as administrator.

. Yay!

Filed under: Tech No Comments
9Jan/151

How to remove the AMD PCI Express (3GIO) Filter Driver

tl;dr:

  1. Go to device manager.
  2. Find AMD PCI Express (3GIO) Filter Driver.
  3. Update the driver.
  4. Browse my computer.
  5. Point to the folder full of intel chipset drivers.
  6. Voila.
  7. Reboot.

My Mom's HP laptop just died, the night before she is scheduled to fly up to Canada for an extended visit. She needs the laptop, it's her livelihood. So I do what I think is the easiest approach and swap the drive into my laptop. Even knowing it's Windows 7 and I'm going from AMD to Intel based chipsets it should work. And it did. Except for PoS AMD PCI Express (3GIO) Filter Driver.

After swapping the drive I got all the drivers installed and showing good except my Intel HD Graphics 4000 driver. It had a big fat yellow ! and an error stating "This device cannot find enough free resources that it can use. (Code 12)". I made sure all the AMD drivers were uninstalled, cleaned the registry of their existence, and even searched for their old brethren ATI. After some quick searching through the rest of the drivers in device manager I found the last remaining AMD driver, AMD PCI Express (3GIO) Filter Driver. Right clicked, uninstall and check the box to uninstall the driver software. Turns out that is the PCI bus and when you uninstall the PCI bus it reinstalls everything else. So I patiently waited and rebooted thinking it would simply disappear and my HD 4000 would work.

Nope, because the AMD PCI Express (3GIO) Filter Driver is full of hate and demons and terrorizes laptops like a suicide bomber. Except it's one that keeps coming back and exploding and you never die but you feel the pain over and over and over again. Sorry, it's 2am right now ...

Anyway. After reading about other peoples' woes (all prior to 2012, and it's 2015 now) I discovered that AMD in their infinite wisdom had renamed the pcisys driver. This replaced the PCI bus driver. So what if I just tried to update the driver for the AMD PCI Express (3GIO) Filter Driver in device manager? Hrm? So I did but I pointed it to my folder full of intel chipset drivers and BAM, it reverted to PCI bus, rebooted, and voila.

Filed under: Tech 1 Comment
7Oct/140

Dell DSET report default password

The default password is "dell" without the "".

I recently ran a DSET report on an older server. Dell DSET is their debug utility that pulls hardware specs but more importantly hardware logs. So when your old PowerEdge has a blinking amber light you can run that report and figure out what's going on. When you run DSET it places a zip file on your desktop. This has a small HTML app that has the report, it looks a lot like OpenManage. In the report you will find the hardware logs and what is going on with the server. When you extract the file use the password dell.

dell dset

Filed under: Tech No Comments
6Oct/140

How to fix “Windows NT user or group servername\Administrators not found” in MSSQL

Trying to add the local administrators group to a SQL server with sys admin (sa) server roles? Getting the error below? I have a very simple fix.

Windows NT user or group 'COMPUTERNAME\Administrators' not found. Check the name again. (Microsoft SQL Server, Error: 15401)

sql admins sa

Instead of adding "COMPUTERNAME\Administrators" change it to "BUILTIN\Administrators" and it will work just find. Just erase your computer/server name and replace with BUILTIN.

sql builtin

This fix should work for SQL Server 2005, 2008, 2012, and 2014.

Filed under: Tech No Comments
1Oct/140

How to fix – WordPress Upgrade Download failed. SSL certificate problem: unable to get local issuer certificate

Getting this error when trying to upgrade WordPress?

Download failed.: SSL certificate problem: unable to get local issuer certificate

Do this.

1. Download the cert.pem file from here. Right click on the link and click save link as.

cert

2. Place the cert.pem file in your php folder.
IE: C:\Program Files (x86)\PHP\v5.5\cacert.pem

3. Edit the php.ini and search for "curl". Edit the curl.cainfo line with the following.
curl.cainfo = "C:\Program Files (x86)\PHP\v5.5\cacert.pem"
Save the php.ini file.

php ini

4. Now try your upgrade again. I didn't have any issues after performing those steps.

Filed under: Tech No Comments
29Apr/140

What events to search for to find a server reboot

With virtualization on the rise we sys admins find ourselves managing a lot more server than normal. Gone are the days of managing a couple racks of pizza boxes. Instead one of those pizza boxes may hold a hundred virtual servers itself. And with so many servers, and clients doing the same fun things, we find ourselves looking into the random "my server rebooted why?" question.

When investigating a reboot you can search the system event log for the event ID's below. Each one corresponds to a reboot and will help determine why. And after you find the actual reboot you can check the rest of the events around that time to see if anything lead to or caused it. For example windows updates, or a BSOD.

1074
The process Explorer.EXE has initiated the restart of computer SERVER01 on behalf of user SERVER01\UserName for the following reason: Other (Planned)
Reason Code: 0x85000000
Shutdown Type: restart
Comment: Server updates
6006
The Event log service was stopped.
6005
The Event log service was started.
109
The kernel power manager has initiated a shutdown transition.
19
Installation Successful: Windows successfully installed the following update: Definition Update for Windows Defender - KB2267602 (Definition 1.173.438.0)22
Restart Required: To complete the installation of the following updates, the computer will be restarted within 15 minutes:
13
The operating system is shutting down at system time.12
The operating system started at system time.
I just copy and paste this line into the event filter.
12,13,19,22,109,1074,6005,6006
Reboot Events
Filed under: Tech No Comments
25Apr/142

How to fix Event ID 2213 for DFSr

I work a lot with DFSr because we use it to keep some web farm replicated and some of our customer's private farms. I can tell you it sucks, it always breaks, and it's very hard to maintain. Although I'll caveat that by saying we probably shouldn't use it for web farms with millions of little files. Seems to work fine for AD. Anyway, this is the most common issue you will run into with DFSr, the unexpected crash or shut down. Both the nodes this occurred on did not crash, in fact they didn't even reboot or shut down. But that doesn't matter, DFSr still crashed. Below is just one example and the fix for it. It's obvious from the event what you need to do, but lets review anyway.

The one thing you HAVE to remember is to leave it alone. Do not touch it after you resume replication. That's the #1 mistake I see people making with troubleshooting DFSr. Either rebooting the server or restarting the server. DFSr keeps a journal (database) of all the changes to the replicated folders. You can't just restart the service or reboot the server to fix this. That's like trying to restart SQL to recover a corrupted database. Instead you need to recover that journal, which fortunately Microsoft tells you exactly how to do in the event log.

To get to the event log go to Control Panel --> Administrative Tools --> Event Viewer --> Applications and Services Logs --> DFS Replication.

Event ID 2213
The DFS Replication service stopped replication on volume C:. This occurs when a DFSR JET database is not shut down cleanly and Auto Recovery is disabled. To resolve this issue, back up the files in the affected replicated folders, and then use the ResumeReplication WMI method to resume replication.
Recovery Steps
1. Back up the files in all replicated folders on the volume. Failure to do so may result in data loss due to unexpected conflict resolution during the recovery of the replicated folders.
2. To resume the replication for this volume, use the WMI method ResumeReplication of the DfsrVolumeConfig class. For example, from an elevated command prompt, type the following command:
wmic /namespace:\\root\microsoftdfs path dfsrVolumeConfig where volumeGuid="32A74A78-0B49-11E2-93EE-806E6F6E6963" call ResumeReplication

You will need to run the command given in step two from the event in command prompt as administrator to resume replication. Remember that each node in the DFSr replication group has a different GUID. Get the command from event viewer on each node and run it. Example below.

wmic /namespace:\\root\microsoftdfs path dfsrVolumeConfig where volumeGuid="32A74A78-0B49-11E2-93EE-806E6F6E6963" call ResumeReplication

After you run it you will see Event ID 2212 in the log.

The DFS Replication service has detected an unexpected shutdown on volume C:. This can occur if the service terminated abnormally (due to a power loss, for example) or an error occurred on the volume. The service has automatically initiated a recovery process. The service will rebuild the database if it determines it cannot reliably recover. No user action is required.

You may also see Event ID 2218

The DFS Replication service is in the second step of replication database consistency checks after an unexpected shutdown. The database will be rebuilt if it cannot be recovered. No user action is required.

Now you just need to wait for the database to recover. Depending on the amount of files and how long it has been down for it can take a few minutes, several hours, or even days. You MUST leave it alone. Do not reboot the server or restart DFSr. That will simply start the process all over again.

Once it is fully recovered you will see event ID 2214.

The DFS Replication service successfully recovered from an unexpected shutdown on volume C:.This can occur if the service terminated abnormally (due to a power loss, for example) or an error occurred on the volume. No user action is required.

Once you see that event you are good to go. More info in this MS KB.

You may also want to see this list of hotfixes for DFSr for Windows 2008 and 2008 R2.

Filed under: Tech 2 Comments
22Apr/141

Simple Powershell script to create local user and generate password

This is a very simple script to add a local administrator. You always want a backup Admin to get into a computer, because the "Administrator" should be disabled after all. This is a simple script I use to create one that also generates a password I can store somewhere, like keepass. Here is what it does.

  1. Generates a 32 character complex password.
  2. Creates a local user.
  3. Adds the user to local Administrators group.
  4. Sets the password to never expire.
  5. Spits out the password to the console window so you can copy/pasta to keepass.

Add-Type -Assembly System.Web
$pass=[Web.Security.Membership]::GeneratePassword(32,0)
NET USER username "$pass" /ADD /y
NET LOCALGROUP "Administrators" "username" /add
WMIC USERACCOUNT WHERE "Name='username'" SET PasswordExpires=FALSE
Write-Host "$pass" -foregroundcolor red -backgroundcolor yellow

Make sure to replace all the "username" (highlighted in red) with the username you wish to create. I use this at work to create a standard backup admin user for servers. It's always the same username with a different password for each server.

username script

 

**UPDATE

I realized this doesn't work with the execution policy set to restricted. So I made a bat file that runs it from your desktop after setting the execution policy to unrestricted. What I do is copy the two files to the desktop of the server (you can do this in RDP for any server 2008 or greater). Then right click on the bat file and "run as administrator".  Here is the script for the bat.

Update deuce. Per reditor's suggestion I took the command to change the execution policy and instead bypass it.

@echo off
powershell -ExecutionPolicy Bypass -file %USERPROFILE%\Desktop\name-of-your-ps1-file.ps1
pause

Filed under: Tech 1 Comment
9Apr/140

How to force a shrink of the MSSQL TempDB

In a bind for disk space on a MSSQL server that you cannot restart? Here's a way to force a shrink of the TempDB if that's your issue. Be aware this can negatively impact performance since you will be clearing the execution cache for the server. The cache that SQL uses to store execution plans after they are compiled. Meaning they have to be recompiled. But working for a small host with shared database servers that CANNOT go down during the day I've been in this bind. This is the script I found to fix the issue.

DBCC FREEPROCCACHE
GO
use tempdb
GO
-- Shrink tempDB data file
DBCC SHRINKFILE ('tempdev' , 1 )
go
-- Shrink tempdb log file
dbcc shrinkfile ('templog' ,1 )

This won't always work the first time, so just keep executing it until the TempDB goes down. I usually have to run this up to 10 times before the TempDB gets down to less than a 100MB. By default the TempDB should be around 7MB when it starts. Also. restarting the SQL server should empty the database and re-intialize it so to speak. Each time you start SQL server the TempDB should be about 7MB. Of course you should probably figure out what is filling your TempDB, but when you have 700 databases on one MSSQL server and you don't control any of them or know the developers, that's easier said than done.

Filed under: Tech No Comments