Matahari: Systems Management and Monitoring APIs

I have been working at Red Hat for a few weeks now and have started getting some real work done. I wanted to share what I’m currently working on, and that is Matahari. Matahari is a cross-platform (Linux and Windows so far) collection of APIs accessible over local and remote interfaces for systems management and monitoring. What the heck does that mean? Read on, dear friends.

Architecture

I mentioned that Matahari is a collection of APIs. These APIs are accessible via a few different methods. The core of the functionality that we are implementing is done as C libraries. These can be used directly. However, we expect and intend for most users to access the functionality via one of the agents we provide. A Matahari Agent is an application that provides access to the features implemented in a Matahari Library via some transport. We are currently providing agents for D-Bus and QMF.

D-Bus is used quite heavily as a communications mechanism between applications on a single system. QMF, or the Qpid Management Framework, is used as a remote interface. QMF is a framework for building remote APIs on top of AMQP, an open protocol for messaging.

The agents are generally thin wrappers around a core library, so other transports could be added in the future if the need presents itself.

Current Features

So, what can you do with Matahari?

Matahari is still under heavy development, but there is already a decent amount of usable functionality.

• Host – An agent for viewing and controlling hosts
• View basic host information such as OS, hostname, CPU, RAM, load average, and more.
• Host control (shutdown, reboot)
• Networking – An agent for viewing and controlling network devices
• Get a list of available network interfaces and information about them, such as IP and MAC addresses
• Start and stop network interfaces
• Services – An agent for viewing and controlling system services
• List configured services
• Start and stop services
• Monitor the status of services
• Sysconfig – Modify system configuration
• Modify system configuration files (Linux)
• Modify system registry (Windows)

More things that are in the works can be found on the project backlog.

Use Cases

An example of a project that already utilizes Matahari is Pacemaker-cloud, which is also under heavy development. Pacemaker-cloud utilizes both the Host and Services agents of Matahari. Being able to actively monitor and control services on remote hosts is a key element of being able to provide HA in a cloud deployment.

In addition to providing ready-to-use agents, we also provide some code that makes it easier to write a QMF agent so that third-parties can write their own Matahari agents. One example of this that already exists is libvirt-qmf, which is a Matahari agent that exposes libvirt functionality over QMF.

Join Us

If Matahari interests you, follow us on github and join our mailing list.  Thanks!

Taking On New Challenges

I began working on the Asterisk project in 2004.  My work on Asterisk has led to an exciting career in open source software engineering.  At the end of July 2011, I will be leaving Digium to take on some new challenges.  Specifically, I will be joining the Cloud Infrastructure team at Red Hat as a Principal Software Engineer where I will be working on projects related to clustering, high availability, and systems management.  Additionally, I will be moving back to Charleston, SC to be closer to my family.

While I will no longer be working with Asterisk full time, I still plan to participate in the open source community.  I am excited to watch both Asterisk and Asterisk SCF continue to evolve and grow.  The engineering team at Digium, as well as the global Asterisk development community are as strong as they have ever been and will continue to accomplish big things.

I have met many great people from all over the world in my time with Asterisk.  Thank you all for making the past seven years so memorable.

Best Regards,

–
Russell Bryant

 

Related Posts:

• blogs.digium.com
• Asterisk Daily News

Debugging the Asterisk Dialplan with Verbose()

Leif Madsen and I are working on a new book, the Asterisk Cookbook. One of the recipes that I am working on this morning is a method of adding debug statements into the Asterisk dialplan. I came up with a GoSub() routine that can log messages based on log level settings that are global, per-device, or per-channel. Here’s a preview. I hope you find it useful!

Channel logging GoSub() routine.

• ARG1 – Log level.
• ARG2 – The log message.

Channel logging using this routine will be sent to the Asterisk console at verbose level 0, meaning that they will show up when you want them to regardless of the current “core set verbose” setting. This routine uses a different method, values in AstDB, to control what messages show up.

AstDB entries:

• Family: ChanLog/ Key: all
• If the log level is less than or equal to this value the message will be printed.
• Family: ChanLog/ Key: channels/
• This routine will also check for a channel specific debug setting. It will actually check for both the full channel name as well as just the part of the channel name before ‘-’. This allows setting a debug level for all calls from a particular device. For example, a SIP channel may be “SIP/myphone-0011223344″. This routine will check:
• Family: ChanLog/ Key: channels/SIP/myphone
• Family: ChanLog/ Key: channels/SIP/myphone-0011223344

Example Dialplan Usage:

exten => 7201,1,GoSub(chanlog,s,1(1,${CHANNEL} has called ${EXTEN}))

Example of enabling debugging for a device from the Asterisk CLI:

*CLI> database put ChanLog SIP/myphone 3 

chanlog routine implementation:

[chanlog]

exten => s,1,GotoIf($[${DB_EXISTS(ChanLog/all)} = 0]?checkchan1)
    same => n,GotoIf($[${ARG1}  n(checkchan1),Set(KEY=ChanLog/channel/${CHANNEL})
    same => n,GotoIf($[${DB_EXISTS(${KEY})} = 0]?checkchan2)
    same => n,GotoIf($[${ARG1}  n(checkchan2),Set(KEY=ChanLog/channel/${CUT(CHANNEL,-,1)})
    same => n,GotoIf($[${DB_EXISTS(${KEY})} = 0]?return)
    same => n,GotoIf($[${ARG1}  n(return),Return() ; Return without logging
    same => n(log),Verbose(0,${ARG2})
    same => n,Return()

Asterisk 1.10 Update

I just posted an update on the development of Asterisk 1.10 to the asterisk-dev mailing list. Here is the content:

Greetings,

Shortly after the release of Asterisk 1.8, we had a developer meeting
and discussed some of the projects that people would like to see in
Asterisk 1.10 [1]. We discussed the schedule there a bit, as well. Now
that Asterisk 1.8 has settled down and we are well into the development
cycle for Asterisk 1.10, it is a good time to revisit the plans for the
next release.

At Digium, the biggest thing we have been working on for 1.10 so far is
replacing the media infrastructure in Asterisk. Most of the critical
and invasive plumbing work is done and has been merged into trunk. Next
we’re looking at building up some features on top of that, such as
adding more codecs, enhancing ConfBridge() to support additional
sampling rates (HD conferencing), adding features that exist in
MeetMe() but not ConfBridge(), and enhancing codec negotiation.

Of course, many others have been working on new developments as well. I
would encourage you to respond if you’d like to provide an update on
some new things that you’re working on.

We would like to release Asterisk 1.10 roughly a year after Asterisk
1.8. This will be a standard release, not LTS [2]. To have the release
out in the October time frame, we need to branch off 1.10 (feature
freeze) at the end of June. At that point we will begin the beta and RC
process. If you’re working on new development projects that you would
like to get into Asterisk 1.10, please keep this timeline in mind.

As always, comments and questions are welcome.

Thanks,

[1] https://wiki.asterisk.org/wiki/display/AST/AstriDevCon+2010
[2] https://wiki.asterisk.org/wiki/display/AST/Asterisk+Versions

–
Russell Bryant

Asterisk: The Definitive Guide – Call for Technical Review

We are finally wrapping up our book, “Asterisk: The Definitive Guide”. The contents have been available on a web site for a while, but now that the content is complete, we’re looking for a lot of review over the next couple of weeks. We would really appreciate your input!  Here is a post from Leif on the asterisk-doc mailing list from earlier today:

Hey all!

We’re getting VERY close to having the first draft of the next Asterisk book, Asterisk: The Definitive Guide ready to be sent off to production. We’re very close to meeting our target dates, but our review timeline is very tight. Only about 2 weeks!

Each morning we’re continuing to work on the book, taking in your comments, reviewing chapters, testing dialplan and installation steps, and all that good stuff.

However, we’ve been looking at this book since May 2010 and our eyes are starting to get glazed We’d love for the community to have a look at the book and offer some constructive criticism.

It’s far too late to take requests for things to cover. What we have is what we’re going to get in for this edition. After we finish this book though we plan on continuing to update it, so there will be a chance to take suggestions again soon.

For now, head on over to http://ofps.oreilly.com and check out the book (updated this morning). There are a couple of bugs in the OFPS software which are causing comments to not be available after chapter 8, but we’re hoping to have those resolved by Friday. However, we do have this fancy mailing list that we can use.

Update: This issue has been resolved

Russell, Jim and myself will be monitoring this list for comments, and we’ll try and get all of them satisfied before publication. If there is a particular area we’re covering that you’re an expert in, we’d love to have you focus on that chapter. You can email me back directly for more information on what we might be looking for in that type of situation.

We do have editors to help with grammar and spelling, but pointing anything out is certainly useful. The best use of your time though is testing the dialplan snippets, the installation instructions for both Ubuntu and CentOS (we’re covering two Linux distributions this time around, which increases the testing load significantly), and making sure anything we’re explaining is concise, relates to what we’re talking about, and makes sense. The goal is to build an Asterisk system from scratch, so following through our dialplan via the chapters to make sure it all continues to build on itself would be ideal.

Additionally, if you see any sections which say, “see chapter XXX for more information” that are not links, please let us know, as those are meant to be placeholders until the chapters existed and we could link back to them. Now that all chapters are created, we should be linking to the appropriate locations. If you’re reading a section and notice a good spot to reference another part of the book (for example, lets say we’re talking about database functionality in one of the other chapters, and there is an appropriate spot in the Database Integration chapter to link to), then let us know!

Thanks for your interest! Books should be shipping sometime between March and April. Pre-orders are available now at http://oreilly.com/catalog/9780596517342. And yes, we’ll be releasing under a Creative Commons license like the last two books, so you’ll have access to the book at any time online.

This book has been pretty much written from the ground up, and is well over 600 pages of content. It’s been a lot of work, but we hope you like it!

Thanks!
Russell, Jim and Leif.

Open Source Telephony @ FOSDEM 2011

FOSDEM (Free and Open Source software Developers European Meeting) is an amazing conference held each year in Brussels, Belgium. I have been lucky enough to attend in 2009 and 2010. Both times I was very impressed with the number of attendees and the quality of the talks.

For FOSDEM 2011, I am proud to take part in the conference by organizing a day of talks on open source telephony. The talks will take place on Sunday, February 6th. Without any further delay, here are the talks that are scheduled for the open source telephony dev room at FOSDEM 2011.

• “Introduction to Asterisk Developement”, Russell Bryant
• “Digital PSTN Connectivity with Asterisk”, Jakub Klausa
• “Mobicents 2.0, The Open Source Java Communication Platform”, Jean Deruelle
• “Scaling location services in large SIP networks with Kamailio”, Henning Westerholt, Marius Zbihlei
• “Unifying SIP and Web worlds with Lua (Kamailio)”, Daniel-Constantin Mierla
• “XiVO IPBX OpenHardware”, Xavier Carcelle
• “Unified Communications - Future (Yate and YateClient)”, Diana Cionoiu
• “Asterisk SCF (Scalable Communications Framework)”, Kevin P. Fleming
• “Developing rich VoIP applications with SIPSIMPLE SDK”, Saúl Ibarra Corretgé
• “SIP Communicator: Building a Multi-Protocol Multi-OS Communications Client”, Emil Ivov

I hope to see you there!

Automated Testing Update

Greetings,

A while back, I posted a message about an effort to improve automated testing in the Asterisk project. I wanted to give an update on how that project has progressed for those that have not been following along very closely.

We started using Bamboo as a continuous integration tool, which you can find running at http://bamboo.asterisk.org/. Note that some of the pass/fail statistics on there are a bit skewed, as the Bamboo server was just rebuilt and things were failing as everything was put back together.

A lot of really good automated test cases have been developed, and more are constantly being added. There are currently 85 test cases that run against Asterisk trunk after every change to the code. While some tests are small in scope, many of them cover significant call scenarios, such as various methods of doing transfers and call parking.

I apologize for the previous flood of Bamboo emails to the -dev list. I now have a new mailing list created for those that would like to subscribe to those messages.

http://lists.digium.com/mailman/listinfo/test-results

Additionally, one of the latest updates to our Bamboo setup is automated testing code coverage analysis. It will tell us exactly what code ran as a result of our automated test cases. It provides a good metric to start using to help identify areas of Asterisk that are in need of more test cases. You can find the code coverage reports for the latest builds of Asterisk trunk and 1.8 on Linux in the artifacts tab when viewing the details of a build.

http://bamboo.asterisk.org/browse/AST-TRUNK/latest

I’m proud of the progress we have made so far and am excited to continue aggressive development of automated test cases for Asterisk. The tests we have are already catching problems on a regular basis. The resulting quality improvements make the job of the development team easier, as well as result in a better experience for end users.

If you’re looking for a way to contribute to Asterisk and you are more comfortable writing scripts instead of C code, then the external test suite is a great way to get involved and help out.

Thank you all for your continued support of Asterisk!

Best Regards,

–
Russell Bryant
Digium, Inc. | Engineering Manager, Open Source Software
445 Jan Davis Drive NW – Huntsville, AL 35806 – USA
jabber: rbryant@digium.com -=- skype: russell-bryant
http://www.digium.com -=- http://www.asterisk.org -=- blogs.asterisk.org