The Professional DBA

Once upon a year ago, when Sun Microsystems acquired MySQL, there were many bloggers who theorized that Big Red, who had a long-running, close partnership with Sun, was pulling some strings in this deal.  The people who endorsed the idea that Oracle couldn’t put the kibosh on MySQL without a PR headache (But Sun could), were dismissed as crazy, conspiracy-theory people.  My only surprise so far is the courteous lack of ‘I told you so’ popping up in the expert blogs.

So where do we go from here?  Oracle doesn’t have any experience in hardware.  If they keep most of Sun’s staffing, and continue to fund their innovation efforts, we may continue to see excellent products from them.  But will they retain their stellar brand identity?  Will they abandon the Sparc chip architecture and adopt x86?  It seems the best solution here looks more like a tightly coupled partnership rather than a merging of the two companies.

From Oracle’s whitepaper on the decision, MySQL’s fate seems a little less promising:

“MySQL will be an addition to Oracle’s existing suite of database products, which already includes Oracle Database 11g, TimesTen, Berkeley DB open source database, and the open source transactional storage engine, InnoDB.”

This doesn’t sound like Oracle is poised to grow MySQL and allow it to flourish.  At this time MySQL 6.0 is in Public Alpha, and has added the Falcon transactional engine as an advanced alternative to InnoDB, and SolidDB.  Looking at the architecture, this engine brings some industrial-grade caching, recovery, and long transaction support to MySQL.  Couple this with the real deal disaster recovery 6.0 is bringing to the table, and you have a free multi-platform database that rivals everything an Oracle database can offer outside of Enterprise Edition, and soundly trounces the latest Microsoft SQLServer offering.

But will Oracle put the resources toward MySQL, to allow it to be all it can be?  Personally, I don’t see it happening, but I hope I am very, very wrong.

Sid

The quarterly Oracle CPU hit the streets on Tuesday, 14 April, and patches 16 vulnerabilities in the Oracle RDBMS, including a remotely accessible exploit of the listener without authentication.  Oddly this only scored a 5.0 on the CVSS v2.0.  There was an 8.5  CVSS-scored vulnerability in Resource manager that was patched.  It has been speculated that this vulnerability could be exploited by SQL Injection, but the high score seems odd.  I’ll keep looking for details on this item.

Have you ever run into this situation: You are happily scripting out or designing a new capability, performing maintenance, or providing support. Perhaps you are eating lunch, or are home in bed, soundly sleeping at 3:00AM.

And then it happens.

Something broke somewhere, and it is database-related. No, it is not something you’ve built, maintained, or even seen - It is something from another business area, and their help is not available.

When you arrive, you are greeted by the ever-present group of concerned stake-holders, and a terminal. Will you staunch the flow of money they may be hemorrhaging? Will you bring back the data they may have lost? Will you be able to restore their system to service?

What you don’t want to do is flounder because they don’t have your favorite management software, your preferred shell, or your favorite OS.

Learn to speak the native languages!

There are 3 skill sets every good data storage professional should keep current at all times, outside of their core RDBMS interface languages:

• Bourne Shell (bash)
• vi (Unix/inux text editor)
• CMD Shell

I guarantee that any Linux system you log into will have bash and vi. I personally prefer the korn shell for navigation, and the c shell for scripting - but the bourne shell is on every system. Same with vi - Except, I really prefer vi to anything else.

This means no matter what Linux or Unix server you are presented with, you can become effective immediately.

I’ve included Microsoft Windows command shell is included because it fits in with a parallel reason for learning the native language - you can proactively increase survivability in your data storage and management systems by using the tools and utilities you KNOW will be available - Even if libraries are unavailable, even if interpreters and frameworks are lost/broken.

If the operating system can boot, you can be sure the bourn shell or CMD shell is available for use.

Knowing that, you should consider scripting the most vital system functions using the available shell script, and initiating them with the operating system’s integral scheduling tool (crontab/Scheduled Tasks). This way you can ensure that if the OS is up, your vital scripts will be executed!

And who doesn’t want that?

I was conversing with a colleague of mine who was working with some Oracle DBAs who were deciding to abandon Oracle’s Recovery Manager and replace it with a 3rd party disk-imaging ‘backup’ solution. Not augment RMAN, but replace it entirely.

I was really surprised. Really, REALLY surprised!

After mulling over all the concerns, I put together some items you may want to consider before heading down this path:

• Are you operating in ARCHIVELOG mode? If you are not, YOU WILL LOSE DATA.

• If you are in ARCHIVELOG mode – What happens to the old archivelogs? Deleting the old ones before the next RMAN level zero renders the ones you have useless (except for logmining).

• If you are in NOARCHIVELOG mode, how far back can you troubleshoot unauthorized data modification or application error? How quickly do your redo logs switch? – Multiply that by the number of groups you have, and you have your answer.

• How do you address block corruption (logical AND physical) without RMAN? With a RMAN-based DR solution, block recovery takes ONE command. No data loss, no downtime. If you take a snapshot using 3rd party tools – Your backups now have that same block corruption. Where do you go from there?

• If disk space is an issue, do you use the AS COMPRESSED BACKUPSET argument to reduce backup size? Do you pack the archivelogs into daily level ones? I’ve found ways to optimize our Oracle RMAN backups so we can cover 2 weeks with the same disk space that used to cover 2 days.

• How do you monitor for block corruption? (Waiting for something to break is not valid instrumentation) I check for block corruption automatically, every day, by using RMAN and building it into my daily database backup scripts.

NOTE: Logical corruption happens. Even on a SAN, even on a VM. VMs can crash, power can be lost. I’ve experienced 2 incidents with block corruption in the recent quarter. Of course, since I built the Disaster Recovery system around RMAN – We caught the corruption the next day and fixed it with ZERO downtime and ZERO data loss.

Point-in-Time-Recovery (PITR) is enabled by RMAN - ALL disk imaging backup solutions lack this capability. If you are relying solely on a snapshot backup, you will lose all the data since the last snapshot.

Without tablespace PITR, you have to roll ALL the data in the database back. If you have multiple instances and are using a server snapshot with no RMAN, ALL the databases on that server will lose data! This is usually not acceptable.

Lastly, How much testing have you done with the snapshot solution? REAL TESTING. Have you taken a snapshot during continuous data change? We tried snap-shotting the database server using 3 different pieces of software. NONE took a consistently consistent and usable snapshot of the database. Sometimes it did. If we were lucky, and the DB was quiet. Is it acceptable to sometime get your client’s/company’s data restored?

Remember, the key is a multi-layered DR strategy (where disk imaging and snap-shotting IN CONJUNCTION with RMAN is incredibly effective!) and continuous REAL WORLD testing.

As a parting shot, in case you were wondering, The ‘DBAs’ had decided to rely soley on a disk imaging backup solution, not because they felt it had more to offer, or because it was tested to be more effective. But because they felt RMAN was difficult to use…

Brian Fedorko

The Oracle October Critical Product Update (CPU) was released yesterday - it includes 15 security fixes for the core RDBMS, including a fix for a vulnerability allowing DB access without authentication.

Despite the high impact, that particular vulnerability only scored a 4.0 in the Common Vulnerability Scoring System v2.0 (CVSS v2). The vulnerability allows for successful a buffer overflow in the Apache Connector component (mod_weblogic) of a Weblogic Server to be exploited for running external code. This vulnerability effects a broad spectrum of WebLogic Server versions (6.1-10.0MP1), however Oracle had addressed this, along with providing guidence for a workaround, back in July with CVE2008-3257.

Another point of interest - A new post-installation script, catbundle.sql, is available with Critical Patch Updates for Oracle Database 11.1.0.6 and 10.2.0.4 on Microsoft Windows. This script replaces catcpu.sql and catcpu_rollback.sql. For more information, see OracleMetaLink Note# 605795.1, Introduction to catbundle.sql. For UNIX/LINUX Critical Patch Updates, catbundle.sql was released with CPUJUL2008.

Remember, Oracle CPUs are cumulative, so even if you have never applied one to your system, you can catch up on all the bug and security fixes entirely with the application of the latest CPU!

Next scheduled CPU will be released on 13 January 2009

It is time once again to eliminate bugs and increase the security posture of our Oracle databases. The Advisories and Risk Matrices can be found on Oracle Technology Network. The full availability information is found at Oracle Metalink under DocID# 579278.1

Points of Interest:

• This CPU contains 11 security fixes for the Oracle Enterprise Database Server
• None of the security holes for the Enterprise DBMS are remotely exploitable without authentication
• Oracle Application Express requires no security fixes (This product continues to impress me)
• ALL Windows platforms running Oracle Enterprise DB Server v10.2.0.3 will have to wait until 22-July-2008 for their CPU
• Support for Solaris 32-bit and Oracle Enterprise DB Server v10.2.0.2 seems to have been pulled! There’s no CPU for these, and none planned for the October 2008 Critical Product Update as per Oracle Metalink DocID# 579278.1.

Don’t forget to read the patch notes, test thoroughly, and check to make sure you’re using the proper version of OPatch!

Next CPU: 14-October2008

Brian Fedorko