Outages of data centers are no fun, whether they are planned or unplanned. Many teams and people are involved when an outage occurs. The bigger the company the more teams are needed.

 

The single most important ingredient of a successful outage is … you guest it: PLANNING.

 

No matter how many times you went through the database shutdown and startup steps, each and every single outage must be planned.

If you plan properly the scheduled outages then the unexpected ones, become much more easier to handle.

When I say you must plan the outage, I mean you must have a written step by step PLAN, not just a plan in your head.

 

You might say “Diana, a full outage (all servers) is not my responsibility to plan, I am only a DBA”. And I would agree partially with you.

The full data center outage is not your responsibility to plan in totality, but you do have a major role in planning it out. The DBA steps sure are your responsibility and so is the understanding of how the database ties in with the rest of the applications.

 

The second most important thing for an outage to be successful, is DOCUMENTATION.

Remember, “A GOOD DBA ALWAYS HAS A DEPLOYMENT PLAN, aka DOCUMENTATION”.

You must have all the DBA steps documented, step by step, not missing a thing. And when I say documented, you must write down in detail, every single step, even if you think it’s minor and unimportant.

Sample steps that might get forgotten, because people consider it unimportant:

• backup crontab for oracle and grid user
• disable crontab for the oracle and grid user
• stop the oracle agent
• disable backup job and so on

The above steps might not seem critical, and thus could be easily overlooked. But for a successful outage, you must complete all the steps.

 

Remember, when a step is on paper, it gets executed.

 

Now that you understand the importance if planning and documenting, I recommend, to open a spreadsheet (it is easier to follow), with at least two tabs, one for the shutdown and one for the startup.

 

Sit down and think about all the steps you follow when you must shutdown the databases on a particular server.
Create 4 columns in the spreadsheet with the following headings:

• server name
• executing user
• step description
• command.

For each step you must document the above details.

Write down everything that comes to your mind for the shutdown process. Once everything is on paper, review it and put it in a logical order (ie: you will not be disabling backup jobs after the database is shutdown, that step must come first).

 

Once you have the shutdown steps in the proper order, go ahead and work on the startup steps, usually in reverse order from the shutdown.

 

Do the above process for all the servers that have databases that you are responsible for. Save the document in a central location where other team members, other DBAs, have access to.

Now you have one of the most precious documentations for yourself and your team.

 

No matter when you must shutdown or startup a database, or all of them, you have a well documented process and you are prepared.

 

This is your homework now! Go ahead and document the shutdown and startup process if you have no documentation, or review the document you already have!