Monitoring Response Engine

[benthehutt]

So, at work I've been recruited to write a "Network Monitoring Response Engine."  I wrote a few scripts to monitor various services on the network and it's all updated about every 5 minutes in an SQL database.  But now we need to DO something with all that.  My question is thus:

How?

I'm asking more for theory, rather than coding help, but I really don't know how to begin.  For instance, we need to be able to say, "If this service is down for 5 minutes, first try this, then this, then this..."  That's all fine and dandy, but how do I do that?  Should I just have a script continually scouring the database for services that have been down for 5 minutes?  When I run something like that, surely it'll just eat up unnecessary resources.  I don't need to check EVERYTHING all the time.  but I don't know how to go about it at all...  Is any of this making sense?

Je ne sais pas...

[Metgod]

Okay, I'm not sure I follow everything, but here's some thoughts.

Firstly, the question of using resources when not necessary or if it's overloaded or .... whatever.

If you take that view, then nothing will get done in general. You could apply that to any kind of computing process, or even non computing ideas. That doesn't mean you have to be constantly checking, but you do have to check things in general -- that is the problem you're solving afterall, right ?

So, firstly, define the parameters to the problem. In fact, define the problem in full. On paper (or screen), and make a plan/outline for your scripts. I have learned it the hard way, but that does help improve efficiency and does help design it. Do I always do that ? No.. but I find it helps me when I do.

The hard question is, how to check if the processes are dead etc. What sort of processes do you have to check anyway ? That may be the crucial question that needs to be answered before I can help (if I can help).

[Metgod]

Oh, and perl is hideous. Useful yes, but ugly.

Just thought you'd like to know.

[benthehutt]

First off: your opinion is wrong--perl is awesome.

Secondly, I will describe in more detail, the system I'm spawning (that sounds gross, here's something grosser: my creative loins bursting in twain to expunge my fruitful perl scripts).

For example, I've got a script that simply pings things and puts an entry in an SQL database like so:

Device Name        Status (1=up, 0=down)        Current Time        Last Successful Time

Then I've got a few that check server services, authenticate to AD, check the proxy server, VPN, IMAP, SMTP, POP3, et cetera.  If it's up, it gets a 1, down it gets a 0.  Then there are various "Summary" values (example - if all the Email services are up, summary gets a 1).

Now I need to write a script that checks this database and takes actions based on predetermined user input.  For example, I could specify that if the IMAP services on \\myServer were down for 5 minutes, restart the service, if it's down for another 3 call me, another 2, explode.  That's the idea.  My question is how do I do that?  Do I write a script that just continually looks through the database to find 0's or what?  Also, I plan on storing the user defined actions in the database as well.

My idea was to have just a tiny little spider going that runs all up over the database--when something IS down it delegates--starts a new thread to run a script that checks if it's already working on fixing that problem, starts a new thread to read actions from the database, etc.  Thus, the spider would be super fast, so it could run maybe every 30 seconds or so.  Whaddya think of my idea?

[Metgod]

First off: I never said perl isn't awesome; I said it's UGLY. As in the looks. And it IS imo (and others too; but like everything, there are people that like it and those that don't).

Now...

I would have suggested cron as a possibility, but from  the \\myServer I suppose it's Windows ? (I guess that makes sense since you were talking about windows login scripts the other day -- now that was hideous! but forget that ).

So I guess that's off the table...

Your way may work. I was actually pondering threads, but of course you always have to be careful with threads. Also, I'm a bit more cautious with that because of the kinds of things I program, threads can be quite troublesome). Anyway... that's a possibility. And not necessarily a bad one, either.

I would say yes, you (obviously) do need to 'poll' the services (since you have to see if they're up or down--I can think of other ways but they're not as reliable), and you can sort it out how often you think it should. Frankly, I'm not sure I would write a script that uses a database just to see if something is up, then use a script that checks the database (why not just condense it is my thinking, although perhaps you didn't say something that would have persauded me, or there is something I'm missing) but your way could suffice.

The best way to find out is to do some test runs and keep an eye out (yes, that's right; I'm calling you a cyclops!). If it doesn't work out well, you can always modify your script(s) to work differently.

[benthehutt]

The main reason it's in a database is because even the people that don't know how to write a script like that need the info.  The web-development people need to be able to look at what's running, the MIS people, etc.  So, centralized info is the best way to go.


I suppose I'll try the thread way first...  Thanks.

[benthehutt]

My latest research on Perl and threads:


You know what sucks?  The following modules:

win32::process
async
threads

also, fork sucks as well--at least, the Perl emulation on a pc.

I've tried all the stupid tricks in the book and they all suck.  Admittedly, alot of the problem is that they all use the same module (win32::process), which isn't the most flexible to begin with.  Okay, also, part of it might be that I've never messed with threads or anything like that.  SO how bout some theory?

So, child threads share the same memory as the parents right?  And processes exist in a seperate memory space.  I really don't care if I can communicate with the child processes, so I tried just spinning off processes the main perl script never looks at again--but even that wasn't working!

To test if two processes were running at the same time, I had luck with only one implementation of perl threads.  I had a subprogram as follows:

Code Select Expand

sub threadSub{
     for(my $i = 0; $i < 100; $i++){
          print $i . "\n";
     }
}

then I called it right before another loop just like that using threads.  It printed something like this:

1
2
3
1
2
4
3
5
6
4
....etc.

so I was pleased.  However, it really sucks cause, say I want to start 300 of these.  I have to name all of the threads differently.  I'm not really sure how to do that...

So now I'm stuck at how to dynamically name variables--something I've never had to deal with (except in actionscript, which is quite a bit different).  My next few days of research will hopefully yield fruit...